How to Troubleshoot ECC Memory Errors in Enterprise Servers

3 Nov 2026, 17:4316 min readby ITYantra Admin

Complete guide to diagnosing and resolving ECC memory errors in enterprise servers including CE, UE counts, DIMM identification, and replacement procedures.

# How to Troubleshoot ECC Memory Errors in Enterprise Servers

Introduction

ECC (Error Correction Code) memory errors are one of the most common hardware issues in enterprise server environments. While ECC can automatically correct single-bit errors, uncorrected errors cause system crashes, data corruption, and unplanned downtime. This guide provides a complete methodology for diagnosing ECC memory errors, identifying the failing DIMM, and executing replacement procedures safely.

You will learn:

The difference between correctable (CE) and uncorrectable (UE) errors
How to read ECC error logs from BIOS, iLO, and OS
How to identify the specific failing DIMM
Safe DIMM replacement procedures
When to replace vs. when to reseat
Preventive monitoring strategies

Technical Background

ECC Error Types

| Error Type | Abbreviation | Description | OS Impact | Action Required | |------------|--------------|-------------|-----------|-----------------| | Correctable Error | CE | Single-bit error, automatically corrected by ECC | None (transparent) | Monitor, plan replacement if increasing | | Uncorrectable Error | UE | Multi-bit error, cannot be corrected | System crash (MCE/kernel panic) | Immediate DIMM replacement | | Corrected Error Overflow | CE Overflow | Too many CEs in short period | Performance degradation, potential UE | Immediate investigation |

Machine Check Architecture (MCA)

Modern x86 servers use Machine Check Architecture to report hardware errors:

**MCE (Machine Check Exception):** Fatal hardware error, causes kernel panic/BSOD
**MCA (Machine Check Architecture):** Error reporting infrastructure
**CMCI (Corrected Machine Check Interrupt):** Interrupt for corrected errors
**MCI (Machine Check Individual):** Per-error reporting registers

EDAC (Linux Error Detection and Correction)

Linux uses the EDAC subsystem to report memory errors:

`/sys/devices/system/edac/mc/mc*/ce_count` — Corrected error count per memory controller
`/sys/devices/system/edac/mc/mc*/ue_count` — Uncorrectable error count
`/sys/devices/system/edac/mc/mc*/csrow*/ce_count` — Per-chip-select row corrected errors

Error Localization

ECC errors can be localized to:

Memory controller (MC)
Chip select row (CSROW)
Specific DIMM slot
Memory channel

Error Rate Thresholds

| Error Count | Severity | Recommended Action | |-------------|----------|-------------------| | CE = 0, UE = 0 | Normal | Continue monitoring | | CE = 1-10, UE = 0 | Low | Monitor, note baseline | | CE = 11-100, UE = 0 | Moderate | Investigate, plan replacement | | CE > 100, UE = 0 | High | Replace DIMM soon | | CE any, UE > 0 | Critical | Replace immediately | | CE rising rapidly | Critical | Replace immediately, possible UE imminent |

Prerequisites

Hardware Requirements

Server with ECC memory and EDAC support
Access to server console (physical or iLO/DRAC/iDRAC)
Replacement DIMM (matched type, speed, capacity)
Anti-static wrist strap
Server documentation for DIMM slot locations

Software Requirements

Operating system with EDAC support (Linux) or Event Viewer (Windows)
iLO/iDRAC/management interface access
`edac-util`, `dmidecode`, `mcelog` (Linux)
HPE SSA or Dell OMSA (for vendor-specific diagnostics)

Knowledge Requirements

Understanding of ECC memory architecture
Familiarity with server hardware maintenance
Basic Linux command line or Windows administration

Step-by-Step Implementation

Step 1: Detect ECC Errors

**Purpose:** Confirm ECC errors are occurring and determine severity.

**Linux — Check EDAC counts:**

# Check corrected errors per memory controller
for mc in /sys/devices/system/edac/mc/mc*; do
    echo "=== $mc ==="
    echo "CE count: $(cat $mc/ce_count)"
    echo "UE count: $(cat $mc/ue_count)"
done

# Check total system-wide
edac-util --status

# Check kernel log for memory errors
dmesg | grep -i -E "(edac|machine check|memory error|ecc)"

# Check mcelog daemon (if running)
mcelog --client --errors

**Expected Output:** CE count increasing over time indicates a failing DIMM. UE count > 0 indicates immediate action needed.

**Windows — Check Event Viewer:** 1. Open Event Viewer (eventvwr.msc) 2. Navigate to Windows Logs → System 3. Filter by Source: "WHEA-Logger" or "Microsoft-Windows-Kernel-Power" 4. Look for Event ID 18 (corrected) or 19 (uncorrected)

**Verification:** ECC errors confirmed and quantified.

Step 2: Localize the Failing DIMM

**Purpose:** Identify which specific DIMM slot is producing errors.

**Linux — Detailed EDAC analysis:**

# Check per-CSROW error counts
for mc in /sys/devices/system/edac/mc/mc*; do
    echo "=== $mc ==="
    for csrow in $mc/csrow*; do
        if [ -f "$csrow/ce_count" ]; then
            echo "$csrow CE: $(cat $csrow/ce_count)"
            echo "$csrow UE: $(cat $csrow/ue_count)"
        fi
    done
done

# Map CSROW to physical DIMM slot
dmidecode -t memory | grep -E "(Locator|Bank Locator|Size|Speed)"

# Cross-reference CSROW with physical location using server manual

**iLO/iDRAC — Hardware Event Log:** 1. Log into management interface 2. Navigate to Hardware Event Log / System Event Log 3. Filter by "Memory" or "ECC" events 4. Note the DIMM slot or chassis location mentioned

**SSA/Diagnostic Utilities:**

HPE: Run Smart Storage Administrator or Inspect Diagnostics
Dell: Run OpenManage Server Administrator or Lifecycle Controller
Memory test: Boot memtest86+ and run overnight

**Verification:** Failing DIMM slot identified with confidence.

Step 3: Assess Severity and Plan Action

**Purpose:** Determine whether to replace immediately or monitor.

**Decision Matrix:**

| Condition | Action | Timeline | |-----------|--------|----------| | UE > 0 | Replace immediately | Immediately | | CE > 100 and rising | Replace soon | Within 24-48 hours | | CE 11-100, stable | Plan replacement | Within 1-2 weeks | | CE 1-10, stable | Monitor | Continue monitoring | | CE = 0, UE = 0 | No action | Continue monitoring |

**Considerations:**

Production criticality of the server
Availability of replacement DIMM
Maintenance window availability
Redundancy (can the server tolerate a DIMM failure?)

Step 4: Prepare for DIMM Replacement

**Purpose:** Gather replacement parts and plan maintenance window.

1. Order replacement DIMM matching existing specifications:

Same type (RDIMM/LRDIMM/UDIMM)
Same speed (DDR4-2933, DDR4-3200, DDR5-4800)
Same capacity (16GB, 32GB, 64GB, etc.)
Same rank (1R, 2R, 4R)
Same voltage (1.2V standard)

2. Schedule maintenance window:

Server will be offline during DIMM replacement
Notify stakeholders
Prepare rollback plan

3. Prepare tools:

Anti-static wrist strap
Flashlight
Server documentation for DIMM locations

**Verification:** Replacement DIMM ordered and matches specifications.

Step 5: Replace the Failing DIMM

**Purpose:** Safely replace the identified failing DIMM.

1. **Power down the server** (or use hot-swap if supported)

Most servers require power down for DIMM replacement
Some enterprise servers support memory hot-add/hot-remove

2. **Discharge static electricity**

Wear anti-static wrist strap connected to chassis
Touch chassis ground before handling components

3. **Open server chassis**

Remove cover per server documentation

4. **Locate the failing DIMM**

Use documentation from Step 2
DIMM slots are typically labeled (A1, A2, B1, B2, etc.)

5. **Remove the failing DIMM**

Press down on DIMM slot latches
DIMM will pop up at ~45 degrees
Remove carefully, place in anti-static bag

6. **Install replacement DIMM**

Align notch with slot key
Insert at 45-degree angle
Press down until latches click
Verify DIMM is fully seated

7. **Close chassis and power on**

**Warning:** Never install or remove DIMMs while server is powered unless hot-swap is explicitly supported.

**Verification:** Server completes POST with full memory detected.

Step 6: Verify Replacement

**Purpose:** Confirm new DIMM is working and errors are resolved.

1. **BIOS/UEFI Check:**

Full memory detected
No memory errors during POST

2. **OS Verification:**

   # Linux - Verify memory detected
   free -h
   
   # Linux - Check EDAC counts (should be stable or zero)
   for mc in /sys/devices/system/edac/mc/mc*; do
       echo "$mc CE: $(cat $mc/ce_count)"
       echo "$mc UE: $(cat $mc/ue_count)"
   done
   
   # Linux - Monitor for new errors
   watch -n 10 'cat /sys/devices/system/edac/mc/mc*/ce_count'

3. **Run Memory Test:**

Boot memtest86+ for at least 2 passes
Any errors = replacement DIMM may also be faulty

4. **Monitor for 24-48 hours:**

Check EDAC counts periodically
Verify CE count is not increasing

**Verification:** CE count stable at zero (or baseline), no new UE errors.

Step 7: Update Documentation

**Purpose:** Record the replacement for future reference.

1. Update asset tracking:

Server name
DIMM slot replaced
Old DIMM serial number
New DIMM serial number
Date of replacement
Reason (CE/UE count, error type)

2. Update maintenance log:

Maintenance window start/end
Downtime duration
Technician name
Parts used

**Verification:** Documentation updated and stored.

Configuration Examples

Continuous ECC Monitoring Script

#!/bin/bash
# ecc-monitor.sh - Monitor ECC errors and alert on thresholds
# Run via cron every 5 minutes

LOG_FILE="/var/log/ecc-monitor.log"
ALERT_EMAIL="admin@ityantra.co.in"
CE_THRESHOLD=50
UE_THRESHOLD=0

timestamp=$(date '+%Y-%m-%d %H:%M:%S')
alert=0
message=""

for mc in /sys/devices/system/edac/mc/mc*; do
    mc_name=$(basename "$mc")
    ce_count=$(cat "$mc/ce_count" 2>/dev/null || echo "0")
    ue_count=$(cat "$mc/ue_count" 2>/dev/null || echo "0")
    
    if [ "$ue_count" -gt "$UE_THRESHOLD" ]; then
        message+="$mc_name: UNCORRECTABLE ERRORS ($ue_count)\n"
        alert=1
    fi
    
    if [ "$ce_count" -gt "$CE_THRESHOLD" ]; then
        message+="$mc_name: HIGH CORRECTABLE ERRORS ($ce_count)\n"
        alert=1
    fi
done

if [ $alert -eq 1 ]; then
    echo -e "$timestamp - ECC ALERT:\n$message" | tee -a "$LOG_FILE"
    echo -e "$timestamp - ECC ALERT:\n$message" | mail -s "ECC Memory Alert: $(hostname)" "$ALERT_EMAIL"
else
    echo "$timestamp - ECC check: OK (CE/UE within thresholds)" >> "$LOG_FILE"
fi

mcelog Configuration (Linux)

# Install mcelog
apt install mcelog    # Debian/Ubuntu
yum install mcelog    # RHEL/CentOS

# Enable mcelog daemon
systemctl enable mcelog
systemctl start mcelog

# Check mcelog output
mcelog --client --errors

# View mcelog log
cat /var/log/mcelog
# Or on systemd systems:
journalctl -u mcelog

DIMM Slot Mapping (HPE ProLiant DL380 Gen10)

DIMM Slot Layout (front view, 24 slots):

CPU 1          CPU 2
Ch 0  Ch 1     Ch 0  Ch 1
 A1   A2       G1   G2
Ch 2  Ch 3     Ch 2  Ch 3
 B1   B2       H1   H2
Ch 4  Ch 5     Ch 4  Ch 5
 C1   C2       I1   I2

           ... (continues for all channels)

CSROW 0 -> Channel 0 DIMM 1
CSROW 1 -> Channel 0 DIMM 2
CSROW 2 -> Channel 1 DIMM 1
...

*(Note: Actual mapping varies by server model. Consult server QuickSpecs for exact mapping.)*

Verification

Post-Replacement Verification Checklist

[ ] Server completes POST without memory errors
[ ] Full memory capacity detected in BIOS
[ ] OS detects all memory (`free -h` matches expected)
[ ] EDAC CE count stable (not increasing)
[ ] EDAC UE count = 0
[ ] memtest86+ passes 2+ cycles without errors
[ ] iLO/iDRAC hardware log shows no new memory errors
[ ] Application/services running normally
[ ] Documentation updated

Ongoing Monitoring Verification

# Daily check script output should show:
$ cat /var/log/ecc-monitor.log | tail -5
2024-01-15 08:00:00 - ECC check: OK (CE/UE within thresholds)
2024-01-15 08:05:00 - ECC check: OK (CE/UE within thresholds)
2024-01-15 08:10:00 - ECC check: OK (CE/UE within thresholds)

Any "ALERT" entries require investigation.

Troubleshooting

| Scenario | Symptom | Possible Cause | Diagnostic | Corrective Action | Verification | |----------|---------|----------------|------------|-------------------|--------------| | CE count rising after replacement | New DIMM also producing errors | Replacement DIMM faulty, slot issue, CPU memory controller issue | EDAC per-CSROW, test new DIMM in different slot | Test DIMM in different slot, replace DIMM or motherboard | CE count stabilizes | | Cannot identify failing DIMM | EDAC shows errors but no CSROW mapping | EDAC not fully supported, incomplete error reporting | iLO hardware log, memtest86+, swap DIMMs systematically | Replace DIMMs one at a time, test after each | Errors stop after specific DIMM replaced | | Server won't boot after DIMM replacement | POST memory error, beep codes | DIMM not seated, incompatible DIMM, wrong slot | Reseat DIMM, check population rules, verify compatibility | Reseat, use correct DIMM type, follow population sequence | Server boots successfully | | CE count spikes during heavy load | Errors only under memory stress | Marginal DIMM, signal integrity issue | memtest86+ extended run, stress test with memtester | Replace marginal DIMM, check DIMM seating | No errors under load | | Multiple DIMMs showing errors | CE counts on multiple CSROWs | Memory controller issue, motherboard problem | EDAC across all MCs, test DIMMs in known-good server | Replace motherboard or CPU, test DIMMs individually | Errors isolated to single component | | ECC disabled in BIOS | No EDAC interface, errors not detected | BIOS setting disabled ECC | BIOS memory configuration | Enable ECC in BIOS (if supported) | EDAC interface available | | Intermittent UE errors | Random system crashes | Failing DIMM, power issue, overheating | Correlate crashes with thermal logs, EDAC history | Replace DIMM, check power and cooling | System stable after replacement | | Cannot find replacement DIMM | Exact DIMM unavailable | Discontinued model, limited availability | Check vendor compatibility list | Use qualified equivalent (same type, speed, rank) | System stable with replacement |

Best Practices

1. **Monitor CE counts trending upward** — A rising CE count predicts imminent DIMM failure. Replace before UE occurs.

2. **Keep replacement DIMMs on hand** — For critical servers, stock spare DIMMs matched to your configuration.

3. **Document everything** — Record error counts, DIMM serial numbers, and replacement dates.

4. **Test replacement DIMMs** — Run memtest86+ on new DIMMs before installing in production.

5. **Schedule maintenance windows** — DIMM replacement requires server downtime (unless hot-swap supported).

6. **Use identical replacement DIMMs** — Match type, speed, capacity, rank, and voltage.

7. **Check both CE and UE counts** — CE indicates degradation, UE indicates immediate failure.

8. **Correlate with system crashes** — If UE errors match crash times, the DIMM is definitely the cause.

9. **Consider server age** — Servers with multiple DIMM failures may have broader memory subsystem issues.

10. **Verify after replacement** — Always confirm errors are resolved before returning server to production.

Conclusion

ECC memory error troubleshooting requires a systematic approach: detect errors, localize the failing component, assess severity, replace safely, and verify resolution. The key insight is that correctable errors (CE) are warning signs — they indicate a DIMM is degrading and should be replaced before uncorrectable errors (UE) cause downtime.

By implementing proactive ECC monitoring and maintaining spare DIMMs, infrastructure teams can minimize memory-related downtime and ensure server reliability.

References

Linux EDAC Documentation: https://www.kernel.org/doc/html/latest/edac/index.html
Intel Machine Check Architecture: https://www.intel.com/content/www/us/en/developer/articles/technical/machine-check-architecture.html
AMD Processor Programming Reference: https://www.amd.com/en/technologies/epyc
HPE ProLiant Memory Troubleshooting: https://www.hpe.com/info/proliant-troubleshooting
Dell ECC Memory Troubleshooting: https://www.dell.com/support/memory-troubleshooting
memtest86+ Official: https://memtest.org/

#ecc-memory#troubleshooting#server-hardware#memory-errors#edac#infrastructure