Fibre Channel SAN Architecture and Configuration

# Fibre Channel SAN Architecture and Configuration

Introduction

Fibre Channel (FC) SAN is a dedicated high-speed network technology for block-level storage access. It remains the dominant technology for enterprise block storage, particularly for databases, virtualization, and mission-critical applications. FC provides low latency, high reliability, and predictable performance.

This guide covers FC SAN architecture, components, zoning, LUN masking, multipathing, and configuration.

You will learn:

• FC SAN architecture and components
• FC topology and addressing (WWN, WWNN, WWPN)
• Zoning (hard vs soft, initiator vs target)
• LUN masking on storage arrays
• Multipathing configuration (MPIO, DM-Multipath)
• VSANs and inter-VSAN routing
• Performance and troubleshooting

Technical Background

FC SAN Architecture

**FC SAN Components:** 1. **Initiators:** Servers with FC HBAs (Host Bus Adapters) 2. **Targets:** Storage arrays with FC ports 3. **Fabric:** FC switches connecting initiators and targets 4. **Cabling:** Optical fiber (multi-mode, single-mode) or copper

**FC Addressing:**

• **WWNN (World Wide Node Name):** Unique identifier for a node (server or storage controller)
• **WWPN (World Wide Port Name):** Unique identifier for a port on a node
• **FCID (Fibre Channel ID):** Dynamic 24-bit address assigned by fabric

**FC Speeds:** 1, 2, 4, 8, 16, 32, 64, 128 Gb/s

FC Topologies

| Topology | Description | Max Nodes | Use Case | |----------|-------------|-----------|----------| | Point-to-Point | Direct connection between two devices | 2 | Simple, legacy | | Arbitrated Loop (FC-AL) | Shared loop, one device communicates at a time | 126 | Legacy, low cost | | Switched Fabric | Multiple devices connected via switches | Millions | Enterprise standard |

FC Port Types

| Port Type | Description | |-----------|-------------| | N_Port | Node port (server or storage device) | | F_Port | Fabric port (switch port connected to N_Port) | | E_Port | Expansion port (switch-to-switch connection) | | TE_Port | Trunking expansion port | | FL_Port | Fabric loop port | | NL_Port | Node loop port |

Zoning

Zoning controls which initiators can see which targets at the switch level.

**Zoning Types:**

• **Initiator-Target Zoning:** One initiator, one or more targets (recommended)
• **Initiator-Initiator Zoning:** Multiple initiators (not recommended, can cause issues)
• **Target-Target Zoning:** Multiple targets (not recommended)

**Zoning Enforcement:**

• **Hard Zoning:** Enforced at hardware level (ports), cannot be bypassed
• **Soft Zoning:** Enforced at name server level (WWN), can be bypassed by knowledgeable users

**Best Practice:** Use initiator-target hard zoning.

LUN Masking

LUN masking controls which LUNs a host can access at the storage array level.

**Purpose:**

• Prevent hosts from seeing unauthorized LUNs
• Prevent multiple hosts from writing to same LUN (data corruption)
• Organize LUN access by host or cluster

Multipathing

Multipathing provides redundant paths between servers and storage.

**Components:**

• Multiple HBAs in server
• Multiple paths through fabric
• Multiple storage ports
• Multipathing software (MPIO, DM-Multipath, VMware NMP)

**Benefits:**

• Redundancy (path failure doesn't lose storage access)
• Load balancing (I/O distributed across paths)
• Performance improvement

FC SAN Architecture Diagram

graph TD
    subgraph "Server Rack"
        Server1[Server 1]
        Server2[Server 2]
        HBA1[HBA 1]
        HBA2[HBA 2]
        Server1 --> HBA1
        Server1 --> HBA2
        Server2 --> HBA1
        Server2 --> HBA2
    end

    subgraph "FC Fabric"
        SwitchA[FC Switch A]
        SwitchB[FC Switch B]
        SwitchA -->|ISL| SwitchB
    end

    subgraph "Storage Array"
        ControllerA[Controller A]
        ControllerB[Controller B]
        Disks[(Disk Drives)]
        ControllerA --> Disks
        ControllerB --> Disks
    end

    HBA1 -->|FC| SwitchA
    HBA2 -->|FC| SwitchB
    SwitchA -->|FC| ControllerA
    SwitchB -->|FC| ControllerB
    SwitchA -->|FC| ControllerB
    SwitchB -->|FC| ControllerA

    Server1 -.->|Path A| ControllerA
    Server1 -.->|Path B| ControllerB
    Server2 -.->|Path A| ControllerA
    Server2 -.->|Path B| ControllerB

Prerequisites

Hardware Requirements

• Servers with FC HBAs (QLogic, Emulex, Brocade, Marvell)
• FC switches (Brocade, Cisco MDS)
• Storage array with FC ports
• FC cables (optical, LC connectors)
• SFP/ SFP+ transceivers (matched to speed)

Software Requirements

• FC HBA drivers
• Multipathing software (DM-Multipath, MPIO, VMware NMP)
• Storage array management software
• FC switch management software

Network Requirements

• Dedicated FC SAN fabric (separate from LAN)
• Zoning configuration
• VSAN configuration (if using Cisco MDS)

Step-by-Step Implementation

Step 1: Physical Installation

**Installation Steps:** 1. Install FC HBAs in servers 2. Install FC switches in rack 3. Connect servers to switches with FC cables 4. Connect storage array to switches 5. Connect inter-switch links (ISLs) between switches 6. Connect power to redundant PDUs 7. Power on all components

**Verification:**

• All ports show link light
• Switches see each other via ISL
• Storage array controllers detected

Step 2: Configure FC Switches

**Brocade Switch Configuration:**

# Login to switch
ssh admin@switch-ip

# Show switch status
switchshow

# Show fabric
fabricshow

# Show zones
zoneshow

# Create zone (initiator-target)
zonecreate "zone_server1_storage1", "10:00:00:00:c9:2e:40:10; 10:00:00:00:c9:2e:50:20"

# Create zone configuration
cfgcreate "SAN_CFG", "zone_server1_storage1,zone_server2_storage1"

# Enable configuration
cfgenable "SAN_CFG"

# Save configuration
cfgsave

**Cisco MDS Switch Configuration:**

# Login to switch
ssh admin@switch-ip

# Enter configuration mode
conf t

# Create VSAN
vsan database
 vsan 100 name Production
 vsan 200 name Development

# Create zone set
zone set name SAN_CFG vsan 100
 zone name zone_server1 vsan 100
  member pwwn 10:00:00:00:c9:2e:40:10
  member pwwn 10:00:00:00:c9:2e:50:20

# Activate zone set
zone set activate name SAN_CFG vsan 100

# Save
copy running-config startup-config

**Verification:** Zones active, fabric shows all devices.

Step 3: Configure LUN Masking on Storage Array

**HPE 3PAR Example:**

# Create host
createhost -i "10:00:00:00:c9:2e:40:10" -t VMWare ESXi_Host1

# Create host volume set
createhvs -a ESXi_Host1 VV_DB_Data,VV_DB_Log

# Export with LUN IDs
exportvvlun -a ESXi_Host1 VV_DB_Data 1
exportvvlun -a ESXi_Host1 VV_DB_Log 2

**Dell PowerStore Example:**

# Create host
storagemask create -name ESXi_Host1 -type VMware -initiators "10:00:00:00:c9:2e:40:10"

# Create masking view
storagemask create-view -name MV_ESXi -host ESXi_Host1 -volumes Vol1,Vol2

**Verification:** Host can see only its assigned LUNs.

Step 4: Configure Multipathing

**Linux DM-Multipath:**

# Install
apt install multipath-tools  # Debian/Ubuntu
yum install device-mapper-multipath  # RHEL/CentOS

# Create initial config
mpathconf --enable --with_multipathd y

# Start service
systemctl enable multipathd
systemctl start multipathd

# View multipath devices
multipath -ll

# Example output:
# mpath0 (36001438005dea...) dm-0 DGC,RAID 0
# size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
# |-+- policy='service-time 0' prio=50 status=active
# | `- 1:0:0:1 sdb 8:16 active ready running
# `-+- policy='service-time 0' prio=10 status=enabled
#   `- 2:0:0:1 sdc 8:32 active ready running

**VMware ESXi:**

• VMware Native Multipathing (NMP) enabled by default
• Configure PSP (Path Selection Policy):
• VMW_PSP_MRU (Most Recently Used) - Active/Passive
• VMW_PSP_RR (Round Robin) - Active/Active
• VMW_PSP_FIXED - Fixed path

# Check path status
esxcli storage nmp device list

# Set path policy
esxcli storage nmp device set --device naa.6001438... --psp VMW_PSP_RR

**Windows MPIO:** 1. Install MPIO feature (Server Manager or PowerShell) 2. Enable MPIO for iSCSI or FC 3. MPIO claims devices automatically 4. Verify in MPIO console

**Verification:** Multiple paths visible, failover tested.

Step 5: Verify Connectivity and Performance

**Linux Verification:**

# Check multipath status
multipath -ll

# Check FC HBA
systool -c fc_host -v

# Check LUNs
lsblk | grep dm-

# Test read/write
fio --name=san-test --rw=readwrite --bs=4K --size=50G --numjobs=4 --iodepth=32 --runtime=60

**Verification:** Multiple paths active, I/O distributed across paths.

Configuration Examples

Complete FC SAN Setup (Brocade + Linux)

# === ON BROCADE SWITCH ===
# Create zones
zonecreate "zone_esxi1_storage1", "10:00:00:00:c9:2e:40:10; 10:00:00:00:c9:2e:50:10; 10:00:00:00:c9:2e:50:11"
zonecreate "zone_esxi2_storage1", "10:00:00:00:c9:2e:40:20; 10:00:00:00:c9:2e:50:10; 10:00:00:00:c9:2e:50:11"

# Create configuration
cfgcreate "SAN_PROD", "zone_esxi1_storage1,zone_esxi2_storage1"

# Enable
cfgenable "SAN_PROD"
cfgsave

# === ON LINUX SERVER ===
# Install multipathing
apt install multipath-tools
mpathconf --enable --with_multipathd y
systemctl start multipathd

# Rescan for new LUNs
echo "- - -" > /sys/class/scsi_host/host0/scan
echo "- - -" > /sys/class/scsi_host/host1/scan

# Check multipath
multipath -ll

# Create filesystem
mkfs.ext4 /dev/mapper/mpath0

# Mount
mount /dev/mapper/mpath0 /mnt/san

VSAN Configuration (Cisco MDS)

# Create VSANs
conf t
vsan database
 vsan 10 name Production_SAN
 vsan 20 name Dev_SAN
 exit

# Assign interfaces to VSAN
interface fc1/1
 switchport mode F
 switchport vsan 10
 no shutdown

interface fc1/2
 switchport mode F
 switchport vsan 20
 no shutdown

# Create zones per VSAN
zone set name Prod_Zones vsan 10
 zone name zone_db1 vsan 10
  member pwwn 10:00:00:00:c9:2e:40:10
  member pwwn 10:00:00:00:c9:2e:50:10

zone set activate name Prod_Zones vsan 10

copy running-config startup-config

Multipath Configuration File

# /etc/multipath.conf
defaults {
    user_friendly_names yes
    find_multipaths yes
    polling_interval 10
    path_checker tur
    path_grouping_policy group_by_prio
    prio alua
    failback immediate
    no_path_retry 5
}

blacklist {
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z]"
}

devices {
    device {
        vendor "DGC"
        product "RAID 0"
        path_grouping_policy group_by_prio
        prio alua
        path_selector "service-time 0"
        hardware_handler "1 alua"
        failback immediate
    }
}

Verification

Fabric Verification

# Brocade
switchshow
fabricshow
# Check: All switches in same fabric, no segmented ports

# Cisco
show vsan
show zoneset active
show flogi database
# Check: All initiators and targets logged in

Host Access Verification

# Check FC HBA
systool -c fc_host -v | grep -E "(port_state|speed|fabric)"

# Check multipath
multipath -ll
# Check: All paths active/ready

# Check LUNs
lsblk
# Check: All expected LUNs visible

Failover Verification

# Test path failure
# Disable one switch port or unplug one cable
# Verify I/O continues via remaining path
multipath -ll
# Check: Remaining path active, failed path disabled

# Reconnect
# Verify failback (if configured)

Troubleshooting

| Scenario | Symptom | Possible Cause | Diagnostic | Corrective Action | Verification | |----------|---------|----------------|------------|-------------------|--------------| | No link on FC port | Port down, no light | Bad cable, wrong SFP, port disabled | `switchshow`, check cable/SFP | Replace cable/SFP, enable port | Link light on | | Host cannot see storage | LUN not in `lsblk` | Zoning issue, LUN masking, HBA not logged in | `zoneshow`, `showhost`, `flogi` | Fix zoning, map LUN, rescan HBA | LUN visible | | Single path only | One path in multipath | Cable, SFP, switch port, HBA port | `multipath -ll`, switch port status | Replace cable/SFP, enable port | Multiple paths active | | Slow performance | High latency | Congested ISL, wrong path policy, overloaded array | `showstats`, `top` on array | Upgrade ISL, adjust path policy, balance load | Latency reduced | | Fabric segmented | Switches in separate fabrics | ISL misconfiguration, different VSANs | `fabricshow`, `show vsan` | Fix ISL, align VSANs | Single fabric | | Zoning not working | Host sees wrong storage | Wrong zone members, soft zoning bypass | `zoneshow`, check zoning type | Use hard zoning, correct members | Correct access only | | Path flapping | Paths repeatedly fail/recover | Bad cable, SFP, port, HBA | Switch port logs, HBA logs | Replace faulty component | Stable paths | | LUN masking error | Host sees another host's LUN | Masking misconfigured | Storage array masking view | Fix masking, remove unauthorized access | Only own LUNs visible | | ISL overutilized | Slow inter-switch traffic | Insufficient ISL bandwidth | Switch port utilization | Add ISLs, upgrade switch speed | Utilization normal |

Best Practices

1. **Use initiator-target zoning** — One initiator per zone with its targets 2. **Use hard zoning** — Port-based zoning is more secure 3. **Implement dual fabrics** — Two independent FC fabrics for redundancy 4. **Connect each HBA to different fabric** — Server HBA 1 to Fabric A, HBA 2 to Fabric B 5. **Use multipathing** — MPIO/DM-Multipath for redundancy and load balancing 6. **Document zoning** — Critical for troubleshooting and changes 7. **Use VSANs (Cisco)** — Isolate production, development, and backup traffic 8. **Monitor buffer credits** — FC performance depends on buffer-to-buffer credits 9. **Keep firmware updated** — HBA, switch, and array firmware 10. **Test failover regularly** — Quarterly path failover tests

Conclusion

Fibre Channel SAN provides high-performance, reliable block storage for enterprise workloads. Success depends on proper zoning, LUN masking, multipathing, and redundancy. While iSCSI and NVMe/FC are gaining ground, FC remains the standard for mission-critical block storage due to its predictable performance and mature ecosystem.

Key decisions: topology (switched fabric), zoning strategy, multipathing software, and redundancy design.

References

• T11 Fibre Channel Standards: https://www.t11.org/
• Brocade SAN Administration Guide: https://www.broadcom.com/
• Cisco MDS Configuration Guide: https://www.cisco.com/c/en/us/support/storage-area-networking/
• Linux DM-Multipath: https://www.kernel.org/doc/html/latest/multipathing.html
• VMware NMP: https://docs.vmware.com/
• SNIA FC SAN Basics: https://www.snia.org/