Multipathing Configuration and Troubleshooting
Complete guide to storage multipathing configuration and troubleshooting on Linux and Windows including DM-Multipath, MPIO, failover, and load balancing.
# Multipathing Configuration and Troubleshooting
Introduction
Multipathing provides multiple physical paths between servers and storage devices, enabling redundancy, load balancing, and improved performance. If one path fails (cable, HBA, switch port), I/O continues via remaining paths without application disruption.
This guide covers multipathing configuration and troubleshooting on Linux (DM-Multipath) and Windows Server (MPIO), including FC, iSCSI, and SAS storage.
You will learn:
- Multipathing concepts and benefits
- Linux DM-Multipath configuration
- Windows Server MPIO configuration
- Path selection policies
- Failover testing
- Troubleshooting common multipathing issues
Technical Background
Multipathing Concepts
**Paths:**
- Physical connection between server HBA/NIC and storage port
- Each path is independent (different HBA, switch, cable, storage port)
**Multipath Device:**
- Virtual block device representing all paths to same LUN
- I/O sent through multipath device, software selects underlying path
**Path States:**
- **Active:** Path can process I/O
- **Standby:** Path available but not used (active/passive)
- **Failed:** Path unavailable
**Path Selection Policies:**
- Determine which path handles each I/O operation
- Affects performance and load distribution
Active/Passive vs Active/Active
| Mode | Description | Use Case | |------|-------------|----------| | Active/Passive | One path active, others standby | Storage arrays with ALUA active/passive | | Active/Active | All paths can process I/O | Most modern storage arrays | | Asymmetric Logical Unit Access (ALUA) | Standard for path priorities | Enterprise storage |
Multipathing Components
**Linux (DM-Multipath):**
- `multipathd` daemon
- `multipath` command
- `/etc/multipath.conf` configuration
- `/dev/mapper/mpathN` devices
**Windows (MPIO):**
- MPIO feature
- DSM (Device Specific Module)
- `Get-MSDSMDevice` PowerShell cmdlet
- MPIO console
Linux DM-Multipath
Installation
# Debian/Ubuntu
apt install multipath-tools
# RHEL/CentOS
yum install device-mapper-multipath
# Enable service
systemctl enable multipathd
systemctl start multipathdConfiguration
**Generate initial config:**
mpathconf --enable --with_multipathd y**Edit `/etc/multipath.conf`:**
defaults {
user_friendly_names yes
find_multipaths yes
polling_interval 5
path_checker tur
path_grouping_policy group_by_prio
prio alua
failback immediate
no_path_retry 5
max_fds 8192
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
# Exclude local drives
devnode "^(sda|sdb)$"
}
devices {
device {
vendor "DGC"
product "RAID 0"
path_grouping_policy group_by_prio
prio alua
path_selector "service-time 0"
hardware_handler "1 alua"
failback immediate
}
device {
vendor "NETAPP"
product "LUN"
path_grouping_policy failover
prio ha
hardware_handler "1 netapp"
}
}
multipaths {
multipath {
wwid 36001438005dea...
alias db_lun1
path_grouping_policy failover
}
}**Apply changes:**
systemctl restart multipathd
multipath -v2Multipath Management
# View multipath devices
multipath -ll
# View concise view
multipath -l
# Reload configuration
multipath -v2
systemctl reload multipathd
# Disable a path
multipathd disable path /dev/sdb
# Enable a path
multipathd enable path /dev/sdb
# Fail back paths
multipathd failback mpath0
# Flush multipath device
multipath -f mpath0Example Output
mpath0 (36001438005dea0e50000000000000001) dm-0 DGC,RAID 0
size=500G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|+-+- policy='service-time 0' prio=50 status=active
| |- 1:0:0:1 sdb 8:16 active ready running
| `- 2:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 1:0:1:1 sdd 8:48 active ready running
`- 2:0:1:1 sde 8:64 active ready running**Path status:**
- `active ready running` — Path is active and processing I/O
- `enabled ready running` — Path is available but not primary
- `failed faulty` — Path has failed
Filesystem on Multipath Device
# Create filesystem
mkfs.ext4 /dev/mapper/mpath0
# Mount
mkdir /mnt/storage
mount /dev/mapper/mpath0 /mnt/storage
# Add to fstab (use /dev/mapper/mpath0, not underlying device)
echo "/dev/mapper/mpath0 /mnt/storage ext4 defaults,_netdev 0 0" >> /etc/fstabWindows Server MPIO
Installation
# Install MPIO feature
Install-WindowsFeature -Name Multipath-IO -IncludeManagementTools
# Restart required
Restart-ComputerConfiguration
# Enable MPIO for iSCSI
Enable-MSDSM -BusType iSCSI
# Enable MPIO for FC
Enable-MSDSM -BusType FibreChannel
# Enable MPIO for SAS
Enable-MSDSM -BusType SAS
# Verify MPIO status
Get-MSDSMDevice
Get-MSDSMOptimizedBalanceParametersDevice Management
# View MPIO devices
Get-MSDSMDevice
# View paths for a device
Get-MSDSMDevice -Name "Disk 1" | Get-MSDSMPath
# Set path policy
Set-MSDSMDevice -Name "Disk 1" -PathSelectionPolicy RoundRobin
# Available policies:
# Fixed, LeastBlocks, RoundRobin, LeastQueueDepth**Verification:** Multiple paths with status "Active".
Path Selection Policies
| Policy | Description | Best For | |--------|-------------|----------| | Round Robin | Distribute I/O across all active paths | Active/Active arrays, sequential I/O | | Fixed | Use one path until it fails | Active/Passive arrays | | Least Queue Depth | Send to path with fewest pending I/O | Mixed workloads | | Least Blocks | Send to path with fewest blocks | Large sequential I/O | | Service Time | Based on path response time | Performance-sensitive | | ALUA | Use array-reported path priorities | ALUA-compliant arrays |
Linux Path Selector Configuration
# In /etc/multipath.conf
path_selector "service-time 0"
# Or:
path_selector "round-robin 0"
# Or:
path_selector "queue-length 0"Windows Path Selection Policy
# Set round robin
Set-MSDSMDevice -Name "Disk 1" -PathSelectionPolicy RoundRobin
# Set fixed
Set-MSDSMDevice -Name "Disk 1" -PathSelectionPolicy Fixed
# Verify
Get-MSDSMDevice -Name "Disk 1" | Select-Object PathSelectionPolicyFailover Testing
Linux Failover Test
# View current paths
multipath -ll
# Simulate path failure
# Method 1: Disable network interface
ip link set eth1 down
# Method 2: Disable HBA port (via vendor utility)
# Method 3: Unplug cable
# Verify multipath status
multipath -ll
# Expected: Failed path marked, remaining paths active
# Restore path
ip link set eth1 up
# Verify failback
multipath -ll
# Expected: Failed path returns to active (if failback configured)Windows Failover Test
# View paths
Get-MSDSMDevice | Get-MSDSMPath
# Disable one network adapter
Disable-NetAdapter -Name "iSCSI 2" -Confirm:$false
# Verify MPIO status
Get-MSDSMDevice | Get-MSDSMPath
# Re-enable
Enable-NetAdapter -Name "iSCSI 2"
# Verify failback
Get-MSDSMDevice | Get-MSDSMPathTroubleshooting
Linux DM-Multipath Issues
| Issue | Symptom | Diagnostic | Fix | |-------|---------|------------|-----| | No multipath device | `multipath -ll` empty | `find_multipaths` disabled, blacklisted | Edit multipath.conf, restart multipathd | | Path shows failed | Path status `failed` | Cable, HBA, switch port, target port | Replace hardware, check links | | Multipathd not running | `systemctl status multipathd` failed | Config error, service issue | Check journal, fix config, restart | | I/O errors on multipath device | Filesystem errors | All paths failed, queue_if_no_path | Restore path, check storage | | Performance poor | Low throughput | Wrong path policy, path grouping | Adjust path_selector, path_grouping_policy | | Device name changes | `mpath0` becomes `mpath1` | WWN not stable, reboot | Use `user_friendly_names yes`, persistent names |
Windows MPIO Issues
| Issue | Symptom | Diagnostic | Fix | |-------|---------|------------|-----| | MPIO not claiming disk | Disk shows as basic | MPIO not enabled for bus type | `Enable-MSDSM -BusType iSCSI` | | Path shows offline | Path status Offline | Cable, adapter, target port | Check hardware, event logs | | MPIO not installed | `Get-MSDSMDevice` not recognized | Feature not installed | `Install-WindowsFeature Multipath-IO` | | Path policy not applying | Policy unchanged | DSM doesn't support policy | Check DSM documentation |
General Multipathing Issues
| Issue | Symptom | Diagnostic | Fix | |-------|---------|------------|-----| | All paths failed | No storage access | Storage array down, fabric down | Check storage, switches, cables | | Path flapping | Paths repeatedly fail/recover | Bad cable, SFP, port, HBA | Replace faulty component | | Load imbalance | One path handles most I/O | Wrong path policy | Change to Round Robin or Service Time | | Failback not working | Failed path doesn't return | `failback` disabled, manual only | Enable `failback immediate` in multipath.conf | | No load balancing | All I/O on one path | Active/Passive array, wrong policy | Check array mode, adjust policy |
Configuration Examples
Complete Linux Multipathing for iSCSI
#!/bin/bash
# Setup multipathing for iSCSI storage
# Install packages
apt install -y open-iscsi multipath-tools
# Configure iSCSI
iscsiadm -m discovery -t st -p 10.0.2.100
iscsiadm -m node -T iqn.2024-01.com.example:storage.target1 -p 10.0.2.100 -l
iscsiadm -m node -T iqn.2024-01.com.example:storage.target1 -p 10.0.3.100 -l
# Enable multipathing
mpathconf --enable --with_multipathd y
systemctl restart multipathd
# Wait for multipath devices
sleep 5
# View devices
multipath -ll
# Format multipath device (NOT underlying sdX)
mkfs.ext4 /dev/mapper/mpath0
# Mount
mkdir /mnt/storage
mount /dev/mapper/mpath0 /mnt/storage
# Add to fstab
echo "/dev/mapper/mpath0 /mnt/storage ext4 defaults,_netdev 0 0" >> /etc/fstabComplete Windows MPIO for iSCSI
# Install MPIO
Install-WindowsFeature -Name Multipath-IO -IncludeManagementTools
Restart-Computer
# Enable MPIO for iSCSI
Enable-MSDSM -BusType iSCSI
# Configure iSCSI initiator
New-IscsiTargetPortal -TargetPortalAddress 10.0.2.100
New-IscsiTargetPortal -TargetPortalAddress 10.0.3.100
# Connect to targets
Get-IscsiTarget | Connect-IscsiTarget -IsMultipathEnabled:$true
# Verify MPIO devices
Get-MSDSMDevice
# Set path policy
Get-MSDSMDevice | Set-MSDSMDevice -PathSelectionPolicy RoundRobin
# Initialize and format disks
Get-Disk | Where-Object {$_.BusType -eq "iSCSI"} | Initialize-Disk -PartitionStyle GPT
Get-Disk | Where-Object {$_.BusType -eq "iSCSI"} | New-Partition -UseMaximumSize | Format-Volume -FileSystem NTFSVerification
Linux Verification Checklist
- [ ] `multipath -ll` shows multipath devices
- [ ] Multiple paths visible per device
- [ ] Path status: `active ready running` or `enabled ready running`
- [ ] Filesystem mounted on `/dev/mapper/mpathX` (NOT `/dev/sdX`)
- [ ] Failover test passed (path failure doesn't lose access)
- [ ] fstab uses multipath device
Windows Verification Checklist
- [ ] `Get-MSDSMDevice` shows MPIO devices
- [ ] Multiple paths per device
- [ ] Path status: Active
- [ ] Disk initialized and formatted via MPIO disk (not physical path)
- [ ] Failover test passed
Best Practices
1. **Never format underlying sdX devices** — Always use `/dev/mapper/mpathX` on Linux 2. **Use friendly names** — `user_friendly_names yes` makes management easier 3. **Blacklist local drives** — Prevent multipathing from claiming OS drives 4. **Test failover regularly** — Quarterly path failure tests 5. **Monitor path status** — Alert on path failures 6. **Use consistent path policy** — Same policy across all multipath devices 7. **Document path layout** — Which HBA connects to which switch and storage port 8. **Keep firmware updated** — HBA, switch, and storage firmware 9. **Use redundant fabrics** — Two independent paths through different switches 10. **Configure failback** — `failback immediate` to restore preferred paths
Conclusion
Multipathing is essential for storage redundancy and performance. Linux DM-Multipath and Windows MPIO provide robust multipathing for FC, iSCSI, and SAS storage. Success depends on proper configuration, regular failover testing, and monitoring.
Key principles: configure before production, test failover, monitor paths, and never manage underlying devices directly.
References
- Linux DM-Multipath: https://www.kernel.org/doc/html/latest/multipathing.html
- multipath.conf man page: `man multipath.conf`
- Windows MPIO: https://docs.microsoft.com/en-us/windows-server/storage/
- SNIA Multipathing: https://www.snia.org/
- ALUA Standard: https://www.t10.org/