Network Troubleshooting Using Linux Tools

# Network Troubleshooting Using Linux Tools

Introduction

Network troubleshooting is an essential skill for system administrators and network engineers. Linux provides a rich set of built-in and open-source diagnostic tools for identifying connectivity issues, measuring performance, capturing traffic, and analyzing network behavior.

This tutorial covers the essential Linux network troubleshooting toolkit, including connectivity testing (ping, traceroute, mtr), routing analysis (ip route, netstat, ss), interface diagnostics (ethtool, ip link), packet capture (tcpdump), DNS troubleshooting (dig, nslookup), and systematic troubleshooting methodology.

By the end of this guide, you will be able to diagnose and resolve common network issues including connectivity failures, DNS problems, routing issues, interface errors, and performance degradation.

Technical Background

OSI Model Troubleshooting Approach

Effective troubleshooting follows a layered approach:

| Layer | Name | Tools | Common Issues | |-------|------|-------|---------------| | 1 | Physical | ethtool, ip link, ifconfig | Cable unplugged, bad SFP, interface down | | 2 | Data Link | ip link, arp, bridge | VLAN mismatch, STP blocking, duplex mismatch | | 3 | Network | ip route, ping, traceroute | Wrong IP, subnet mask, gateway, routing table | | 4 | Transport | ss, netstat, nmap | Port not listening, firewall blocking | | 5-7 | Application | dig, curl, telnet, nc | DNS resolution, service not responding |

Key Networking Concepts

• **ICMP**: Used by ping and traceroute for connectivity testing. May be blocked by firewalls.
• **TCP/UDP**: Transport protocols tested with ss, netstat, nc, telnet.
• **ARP**: Maps IP to MAC addresses. Issues cause local connectivity failures.
• **DNS**: Resolves hostnames to IPs. Misconfiguration causes "network" issues that are actually DNS.
• **Routing table**: Determines where packets are sent. Wrong routes cause unreachable networks.

Prerequisites

• Linux system with root or sudo access
• Network connectivity to test
• Basic understanding of TCP/IP networking
• Permission to capture packets (tcpdump may require root)

Architecture

Troubleshooting Methodology Flow

graph TD
    A[User reports issue] --> B{Can ping IP address?}
    B -- No --> C{Interface up?}
    C -- No --> D[Physical/Data Link fix]
    C -- Yes --> E{Ping gateway works?}
    E -- No --> F[Local routing/ARP issue]
    E -- Yes --> G{Can ping remote IP?}
    G -- No --> H[Routing or firewall issue]
    G -- Yes --> I{Can ping hostname?}
    I -- No --> J[DNS issue]
    I -- Yes --> K{Application port open?}
    K -- No --> L[Service or firewall issue]
    K -- Yes --> M[Application layer issue]

Step-by-Step Implementation

Step 1: Check Interface and Link Status

**Purpose**: Verify the network interface is physically and logically up.

**Commands**:

# Show all interfaces
ip link show

# Show brief interface status
ip -brief link show

# Show detailed interface statistics
ip -s link show eth0

# Check interface speed and duplex
ethtool eth0
ethtool -k eth0  # Show offload features
ethtool -S eth0  # Show NIC statistics

**Configuration files**: Interface configuration in /etc/netplan/ (Ubuntu), /etc/sysconfig/network-scripts/ (RHEL), or /etc/network/interfaces (Debian).

**Explanation**: `state UP` means the interface is administratively enabled. `LOWER_UP` means the physical link is active. Both must be present for connectivity.

**Expected output**: Interface shows `state UP LOWER_UP`; ethtool shows `Link detected: yes`.

**Verification**:

ip link show eth0
# Expected: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000

Step 2: Verify IP Addressing

**Purpose**: Confirm the interface has correct IP address, subnet mask, and gateway.

**Commands**:

# Show IP addresses
ip addr show eth0

# Show routing table
ip route show

# Show default gateway
ip route show default

# Show all routes
ip route show table all

**Explanation**: The interface needs an IP address in the correct subnet and a default route for off-subnet communication.

**Expected output**: IP address assigned; default route via gateway.

**Verification**:

ip addr show eth0
# Expected: inet 10.1.10.50/24 brd 10.1.10.255 scope global dynamic eth0
ip route show default
# Expected: default via 10.1.10.1 dev eth0 proto dhcp metric 100

Step 3: Test Basic Connectivity (ping)

**Purpose**: Verify ICMP reachability to local and remote hosts.

**Commands**:

# Ping localhost
ping -c 4 127.0.0.1

# Ping default gateway
ping -c 4 10.1.10.1

# Ping remote host
ping -c 4 8.8.8.8

# Ping with specific interface
ping -I eth0 -c 4 10.1.10.1

# Continuous ping with timestamps
ping -D -c 4 8.8.8.8

**Explanation**: Successful ping confirms Layer 3 connectivity. Failed ping could mean ICMP is blocked (firewall) or there is a routing issue.

**Expected output**: Packets transmitted/received with round-trip times.

**Verification**:

ping -c 4 8.8.8.8
# Expected: 4 packets transmitted, 4 received, 0% packet loss

Step 4: Trace Network Path (traceroute/mtr)

**Purpose**: Identify where packets are dropped along the path.

**Commands**:

# traceroute to remote host
traceroute 8.8.8.8
traceroute -I 8.8.8.8  # Use ICMP
traceroute -U -p 53 8.8.8.8  # Use UDP port 53

# mtr combines ping and traceroute
mtr -rwc 10 8.8.8.8
mtr --report 8.8.8.8

**Explanation**: Each hop shows latency and packet loss. High loss at an intermediate hop indicates that router is dropping or rate-limiting ICMP, not necessarily end-to-end failure.

**Expected output**: List of hops with IP addresses and latency per hop.

**Verification**:

mtr -rwc 10 8.8.8.8
# Look for consistent loss pattern across all hops after a certain point

Step 5: Check Listening Ports and Connections

**Purpose**: Verify services are listening and connections are established.

**Commands**:

# Show listening ports (modern)
ss -tlnp
ss -ulnp

# Show all connections
ss -tanp
ss -uanp

# Legacy netstat
netstat -tlnp
netstat -anp

# Check specific port
ss -tlnp | grep 443

**Explanation**: `ss` is the modern replacement for `netstat`, faster and more detailed. `-t` = TCP, `-u` = UDP, `-l` = listening, `-n` = numeric, `-p` = show process.

**Expected output**: List of listening ports with associated processes.

**Verification**:

ss -tlnp | grep 443
# Expected: LISTEN 0 128 *:443 *:... users:(("nginx",pid=1234,fd=6))

Step 6: Packet Capture (tcpdump)

**Purpose**: Capture and analyze network traffic for detailed diagnosis.

**Commands**:

# Capture on interface
tcpdump -i eth0 -c 50

# Capture specific host
tcpdump -i eth0 host 10.1.10.50

# Capture specific port
tcpdump -i eth0 port 443

# Capture and save to file
tcpdump -i eth0 -w capture.pcap -c 100

# Read capture file
tcpdump -r capture.pcap

# Capture with verbosity
tcpdump -i eth0 -vvv port 53

**Explanation**: tcpdump shows packet headers in real time. Use `-w` to save for analysis in Wireshark. Requires root or CAP_NET_RAW capability.

**Expected output**: Packet summaries with source, destination, protocol, and flags.

**Verification**:

tcpdump -i eth0 -c 5 host 8.8.8.8
# Should show ICMP echo request/reply packets

Step 7: DNS Troubleshooting

**Purpose**: Diagnose DNS resolution issues.

**Commands**:

# Query DNS server directly
dig @8.8.8.8 google.com
nslookup google.com 8.8.8.8

# Query system resolver
dig google.com
nslookup google.com

# Reverse DNS lookup
dig -x 8.8.8.8

# Check DNS configuration
cat /etc/resolv.conf
systemd-resolve --status  # systemd systems
resolvectl status  # newer systemd

**Explanation**: Many "network" issues are actually DNS failures. Test with IP address first to isolate DNS from connectivity.

**Expected output**: DNS response with answer section containing IP addresses.

**Verification**:

dig google.com +short
# Expected: 142.250.x.x (or similar IP)

Configuration Examples

/etc/resolv.conf

nameserver 10.1.10.10
nameserver 8.8.8.8
search ityantra.co.in domain.local
options timeout:2 attempts:3

systemd-resolved Configuration

# /etc/systemd/resolved.conf
[Resolve]
DNS=10.1.10.10 8.8.8.8
Domains=ityantra.co.in
DNSOverTCP=yes

Netplan Interface Configuration (Ubuntu)

# /etc/netplan/01-netcfg.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      nameservers:
        addresses: [10.1.10.10, 8.8.8.8]

ifcfg File (RHEL/CentOS)

# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.1.10.50
NETMASK=255.255.255.0
GATEWAY=10.1.10.1
DNS1=10.1.10.10
DNS2=8.8.8.8
ONBOOT=yes

Verification

Connectivity Checklist

[ ] ip link show eth0 -> state UP LOWER_UP
[ ] ip addr show eth0 -> correct IP and subnet
[ ] ip route show default -> default gateway present
[ ] ping -c 4 <gateway> -> 0% packet loss
[ ] ping -c 4 8.8.8.8 -> 0% packet loss
[ ] ping -c 4 google.com -> DNS resolves and responds
[ ] dig google.com +short -> returns IP address
[ ] ss -tlnp | grep <port> -> service listening
[ ] traceroute <remote> -> path reaches destination

Expected Results

• All interfaces UP with correct IP configuration
• Gateway reachable with low latency
• Public IP reachable (internet connectivity)
• DNS resolves hostnames correctly
• Expected services listening on expected ports
• No excessive packet loss on any network path

Troubleshooting

| Symptom | Possible Cause | Diagnostic Command | Corrective Action | Verification | |---------|---------------|-------------------|-------------------|--------------| | No IP on interface | DHCP failed or static misconfigured | `ip addr show eth0` | Restart networking or set static IP | IP address assigned | | Interface down | Cable unplugged or admin down | `ip link show eth0` | Connect cable; `ip link set eth0 up` | State UP LOWER_UP | | Can ping gateway but not remote | Missing default route or firewall | `ip route show default` | Add default route; check firewall | Remote ping succeeds | | Can ping IP but not hostname | DNS misconfiguration | `cat /etc/resolv.conf`; `dig` | Fix DNS servers in resolv.conf | Hostname resolves | | Port not listening | Service not running or wrong interface | `ss -tlnp \| grep <port>` | Start service; check bind address | Port shows LISTEN | | Connection refused | Firewall blocking or service down | `ss -tlnp`; `iptables -L` | Start service; open firewall port | Connection succeeds | | High latency | Congestion or bad route | `mtr <remote>` | Contact ISP; optimize routing | Latency normalized | | Packet loss | Bad cable, interface errors, congestion | `ip -s link show`; `ethtool -S` | Replace cable; fix interface errors | Loss drops to 0% | | Duplex mismatch | Auto-negotiation failed | `ethtool eth0` | Force same duplex on both ends | Duplex matches; no collisions | | ARP issues | Duplicate IP or ARP table stale | `ip neigh show` | Clear ARP cache: `ip neigh flush all` | ARP table correct | | tcpdump shows no traffic | Wrong interface or no traffic generated | `tcpdump -i <correct-interface>` | Use correct interface; generate test traffic | Packets captured |

Best Practices

1. **Start at Layer 1**: Verify physical link before testing IP connectivity. 2. **Test with IP first**: Use IP addresses to isolate DNS from connectivity issues. 3. **Compare working and non-working hosts**: Identify what differs between hosts that can and cannot connect. 4. **Document baseline performance**: Record normal latency and throughput for comparison during issues. 5. **Use mtr for intermittent issues**: Run mtr over time to identify intermittent packet loss. 6. **Capture packets selectively**: Use filters to avoid capturing excessive data. 7. **Check both directions**: Connectivity issues may be asymmetric (outbound works, inbound doesn't). 8. **Verify firewall rules**: Check host firewalls (iptables/nftables/ufw) and network firewalls. 9. **Use non-ICMP tests**: If ICMP is blocked, test with TCP (nc, telnet, curl). 10. **Keep a troubleshooting log**: Record commands, outputs, and resolutions for future reference.

Conclusion

Linux provides powerful network diagnostic tools that cover every layer of the network stack. From interface status (ip link, ethtool) to connectivity (ping, traceroute), routing (ip route), services (ss), and packet capture (tcpdump), these tools enable systematic diagnosis of virtually any network issue. The key to effective troubleshooting is a methodical approach: verify physical connectivity, then IP addressing, then routing, then DNS, then application-layer services. Combined with good documentation and baseline measurements, these tools form a complete network troubleshooting toolkit.

References

• Linux iproute2 documentation (https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
• tcpdump man page and documentation (https://www.tcpdump.org/)
• dig (BIND) documentation (https://www.isc.org/bind/)
• ss and netstat comparison (https://man7.org/linux/man-pages/man8/ss.8.html)
• ethtool documentation (https://www.kernel.org/doc/Documentation/networking/ethtool.txt)
• mtr documentation (https://www.bitwizard.nl/mtr/)