Incident Response and Disaster Recovery
Complete guide to incident response planning, disaster recovery, playbooks, containment, eradication, recovery, and post-incident activities.
# Incident Response and Disaster Recovery
Introduction
Security incidents are inevitable. Whether it's a ransomware attack, data breach, insider threat, or natural disaster, organizations must be prepared to respond effectively. Incident response (IR) is the organized approach to addressing and managing the aftermath of a security breach or cyberattack. Disaster recovery (DR) focuses on restoring business operations after any disruptive event.
This tutorial provides a comprehensive guide to incident response planning, playbooks, disaster recovery, business continuity, and post-incident activities. The content aligns with NIST SP 800-61, ISO 22301, and industry best practices.
Learning Objectives
By the end of this tutorial, you will be able to:
- Build and staff an incident response team (CSIRT/CIRT)
- Develop incident response plans and playbooks for common scenarios
- Execute the incident response lifecycle (preparation, detection, containment, eradication, recovery, post-incident)
- Design disaster recovery architecture with RTO and RTO targets
- Conduct tabletop exercises and red/blue team simulations
- Integrate incident response with SIEM, backup, and business continuity
- Meet regulatory breach notification requirements
Technical Background
Incident Response Lifecycle (NIST SP 800-61)
1. Preparation → Policies, tools, training, team, communication
2. Detection & Analysis → Monitoring, alerting, triage, validation, scoping
3. Containment, Eradication & Recovery → Isolate, remove threat, restore
4. Post-Incident Activity → Lessons learned, improve defenses, reportingIncident Severity Classification
| Severity | Description | Example | Response Time | |----------|-------------|---------|---------------| | SEV-1 (Critical) | Active breach, data exfiltration, widespread ransomware | APT intrusion, ransomware deployment | 15 minutes | | SEV-2 (High) | Successful attack, privilege escalation, data exposure | Phishing compromise, SQL injection | 1 hour | | SEV-3 (Medium) | Suspicious activity, policy violation, malware detected | Malware on single endpoint, failed brute force | 4 hours | | SEV-4 (Low) | Informational, minor policy issue | Port scan, outdated software | 24 hours |
Incident Response Team Roles
| Role | Responsibility | |------|----------------| | Incident Commander | Overall coordination, decision authority | | Technical Lead | Forensic analysis, containment, eradication | | Communications Lead | Internal/external notifications, stakeholder updates | | Legal Counsel | Regulatory compliance, breach notification, liability | | HR Representative | Insider threat, employee-related incidents | | IT Operations | System restoration, backup recovery | | Public Relations | Media statements, customer notifications |
Disaster Recovery Metrics
| Metric | Definition | Example | |--------|------------|---------| | RTO (Recovery Time Objective) | Maximum acceptable downtime | 4 hours for critical systems | | RPO (Recovery Point Objective) | Maximum acceptable data loss | 15 minutes for critical databases | | MTD (Maximum Tolerable Downtime) | Total time before irreparable harm | 24 hours | | WRT (Recovery Window) | Time between disaster declaration and recovery start | 1 hour |
Prerequisites
- Asset inventory with criticality classification
- Business impact analysis (BIA) completed
- Communication tree and contact lists
- Backup and restoration procedures documented
- Legal and regulatory requirements understood
Hardware and Software Requirements
| Component | Requirement | |-----------|-------------| | IR Platform | ServiceNow, Jira, TheHive, Cortex XSOAR, Demisto | | SIEM | For detection and evidence collection | | Backup | Verified backup solution with tested restores | | DR Site | Hot, warm, or cold site; cloud DR | | Communication | Out-of-band phone, email, chat (not dependent on compromised systems) | | Forensics | Write blockers, forensic imaging tools, evidence storage |
Step-by-Step Implementation
Phase 1: Build Incident Response Capability
#### 1.1 Incident Response Policy
Incident Response Policy (ITYantra)
1. Purpose
Establish the framework for responding to security incidents.
2. Scope
Applies to all information systems, employees, contractors, and third parties.
3. Authority
The CISO has authority to declare incidents and direct response activities.
The Incident Commander has tactical authority during active incidents.
4. Incident Classification
SEV-1 through SEV-4 based on business impact.
5. Reporting
All employees must report suspected incidents to security@ityantra.co.in
or the security hotline within 1 hour of discovery.
6. Response Team
CSIRT consists of: Incident Commander, Technical Lead, Communications,
Legal, HR, IT Operations.
7. External Reporting
Breach notifications per applicable regulations (GDPR: 72 hours,
PCI DSS: immediate, state laws: varies).#### 1.2 Incident Response Playbooks
Create playbooks for common scenarios:
Playbook 1: Ransomware
1. Detect: File encryption, ransom note, backup deletion
2. Contain: Isolate infected hosts, disable network access
3. Eradicate: Remove ransomware, restore from clean backups
4. Recover: Validate restored data, monitor for re-infection
5. Post-incident: Identify initial access vector, close gaps
Playbook 2: Data Breach
1. Detect: Unauthorized data access, exfiltration alerts
2. Contain: Revoke access, secure affected systems
3. Eradicate: Remove attacker persistence, reset credentials
4. Recover: Restore data integrity, notify affected parties
5. Post-incident: Breach assessment, regulatory notification
Playbook 3: Phishing Compromise
1. Detect: User reports, unusual email forwarding, compromised account
2. Contain: Disable account, reset password, revoke sessions
3. Eradicate: Remove forwarding rules, scan for malware
4. Recover: Re-enable account, user retraining
5. Post-incident: Phishing campaign analysis, awareness training
Playbook 4: Insider Threat
1. Detect: Unusual data access, after-hours activity, policy violations
2. Contain: Restrict access, preserve evidence
3. Eradicate: Remove access, HR involvement
4. Recover: Assess data exposure, restore trust
5. Post-incident: Access review, DLP tuningPhase 2: Preparation
#### 2.1 Incident Response Toolkit
Forensic Kit:
- Write blockers (USB, SATA, NVMe)
- Forensic imaging software (FTK Imager, DD, Guymager)
- Evidence bags, labels, chain-of-custody forms
- Clean laptop with forensic software
- External storage for evidence
Response Kit:
- Laptop with administrative credentials
- Network cables, USB drives
- Contact lists (team, vendors, law enforcement)
- Playbooks and procedures (printed)
- Incident report templates#### 2.2 Communication Plan
Internal Communication:
- Incident Commander → CSIRT: Secure chat (Signal, Teams)
- CSIRT → IT Ops: Ticketing system (ServiceNow, Jira)
- CSIRT → Leadership: Email + phone bridge
- All employees: Email, intranet banner
External Communication:
- Customers: Email, website notice, call center script
- Regulators: Formal breach notification
- Law enforcement: FBI IC3, local police (if criminal)
- Media: Prepared statement, single spokesperson
Out-of-Band Communication:
- Personal phone numbers (not in corporate directory)
- Personal email accounts
- Alternate conference bridge#### 2.3 Legal and Regulatory Requirements
Breach Notification Timelines:
- GDPR: 72 hours to supervisory authority
- PCI DSS: Immediately to card brands, acquiring bank
- HIPAA: 60 days to affected individuals, HHS
- State laws (US): Varies, typically 30-45 days
- SEC (public companies): 4 days for material cybersecurity incidents
Data to Collect for Notification:
- Date of discovery, date of breach
- Types of data exposed
- Number of affected individuals
- Steps taken to remediate
- Contact information for questionsPhase 3: Detection and Analysis
#### 3.1 Incident Detection Sources
| Source | Detection Capability | |--------|---------------------| | SIEM | Correlation rules, anomaly detection | | EDR | Malware detection, suspicious process behavior | | Firewall | Blocked connections, unusual outbound traffic | | IDS/IPS | Known attack signatures | | DLP | Data exfiltration attempts | | User reports | Phishing emails, suspicious activity | | Threat intel | Known IOC matches |
#### 3.2 Triage and Validation
Triage Process:
1. Receive alert or report
2. Verify legitimacy (false positive check)
3. Assign severity (SEV-1 to SEV-4)
4. Create incident ticket
5. Notify Incident Commander (SEV-1/2)
6. Begin investigation
Validation Questions:
- Is this a real attack or a false positive?
- What systems are affected?
- What data may be exposed?
- Is the attack ongoing?
- What is the business impact?#### 3.3 Initial Containment
Short-term containment (immediate):
- Isolate affected hosts from network
- Disable compromised accounts
- Block malicious IPs at firewall
- Preserve volatile memory (if applicable)
Evidence preservation:
- Create forensic images before any changes
- Document chain of custody
- Record system state (running processes, network connections)
- Collect logs from SIEM, firewall, endpointsPhase 4: Containment, Eradication, and Recovery
#### 4.1 Strategic Containment
Long-term containment:
- Implement temporary fixes (WAF rules, ACL changes)
- Segment affected network
- Deploy additional monitoring
- Prepare clean systems for recovery
Decision tree:
- Can we contain without taking systems offline? → Yes: Apply temporary controls
- Must we take systems offline? → Yes: Plan maintenance window
- Is data actively being exfiltrated? → Yes: Immediate isolation#### 4.2 Eradication
Eradication steps:
1. Identify all affected systems (IOC scanning)
2. Remove malware and attacker tools
3. Close exploited vulnerabilities (patch, reconfigure)
4. Remove persistence mechanisms (scheduled tasks, services, registry)
5. Reset all potentially compromised credentials
6. Re-image critically affected systems from known-good media#### 4.3 Recovery
Recovery process:
1. Restore systems from clean backups (verify backup integrity first)
2. Apply security patches and hardening
3. Implement additional monitoring on restored systems
4. Gradually restore network access (pilot group first)
5. Validate application functionality
6. Confirm no signs of attacker presence
7. Return to normal operationsPhase 5: Post-Incident Activity
#### 5.1 Lessons Learned
Post-incident review meeting (within 5 business days):
Attendees: Incident Commander, Technical Lead, Legal, IT Ops, Management
Agenda:
1. Timeline reconstruction (detection → containment → recovery)
2. What worked well
3. What needs improvement
4. Root cause analysis
5. Control gaps identified
6. Action items with owners and due dates
7. Playbook updates#### 5.2 Incident Report Template
Incident Report
Incident ID: IR-2024-001
Date/Time Detected: 2024-01-15 09:30 UTC
Date/Time Contained: 2024-01-15 14:15 UTC
Date/Time Recovered: 2024-01-16 02:00 UTC
Severity: SEV-2
Incident Commander: [Name]
Technical Lead: [Name]
Summary:
[Brief description of incident]
Timeline:
- 09:30 - SIEM alert: multiple failed logins
- 09:35 - Triage: confirmed brute force attack
- 09:45 - Containment: blocked source IP at firewall
- 10:00 - Investigation: identified compromised account
- 10:30 - Eradication: reset credentials, removed persistence
- 14:15 - Recovery: restored access, monitored for recurrence
Root Cause:
[Primary cause]
Impact:
[Systems affected, data exposed, business impact]
Lessons Learned:
[What went well, what needs improvement]
Action Items:
- [Action] - Owner - Due DatePhase 6: Disaster Recovery Planning
#### 6.1 Business Impact Analysis (BIA)
| Business Process | RTO | RPO | Criticality | Dependencies | |------------------|-----|-----|-------------|--------------| | Order processing | 4 hours | 15 min | Tier 1 | Database, payment gateway, email | | Customer support | 8 hours | 1 hour | Tier 2 | CRM, phone system | | Payroll | 24 hours | 24 hours | Tier 2 | HR system, bank interface | | Marketing website | 24 hours | 24 hours | Tier 3 | Web server, CDN |
#### 6.2 DR Site Options
| Site Type | Description | Cost | Recovery Time | |-----------|-------------|------|---------------| | Cold site | Empty space with power/cooling, no equipment | Low | Days to weeks | | Warm site | Equipment pre-installed, data restored from backup | Medium | Hours to days | | Hot site | Fully operational mirror, real-time replication | High | Minutes to hours | | Cloud DR | Cloud resources provisioned on demand | Medium | Hours |
#### 6.3 DR Procedure
Disaster Declaration:
1. Assess situation (scope, duration, impact)
2. Declare disaster (Incident Commander + management)
3. Activate DR site
4. Notify employees, customers, vendors
Recovery Steps:
1. Restore network connectivity at DR site
2. Restore critical infrastructure (DNS, DHCP, directory)
3. Restore Tier 1 applications from backup
4. Validate application functionality
5. Redirect users (DNS update, VPN)
6. Monitor for issues
7. Plan return to primary sitePhase 7: Business Continuity
#### 7.1 Business Continuity Plan (BCP)
Scope: Critical business functions during extended disruption
Manual Workarounds:
- Order processing: Phone orders, paper forms
- Customer support: Mobile phones, personal email
- Payroll: Manual calculation, check writing
- IT operations: Out-of-band management, console access
Alternate Work Locations:
- Primary: Office building
- Alternate: DR site, remote work, partner facility
- Last resort: Employee homes, temporary facilities#### 7.2 Crisis Communication
Stakeholder Communication Matrix:
Stakeholder | When | Channel | Owner | Message
------------|------|---------|-------|---------
Employees | Immediate | Email, SMS, intranet | HR | Status, instructions, safety
Customers | Within 4 hours | Email, portal, call center | Customer Success | Service impact, workaround
Regulators | Per legal requirements | Formal notification | Legal | Breach details, remediation
Board | Within 24 hours | Executive briefing | CISO | Impact, response, cost
Media | As needed | Press release, statement | PR | Public statement (approved)Verification and Expected Outputs
| Check | Method | Expected Result | |-------|--------|-----------------| | IR plan tested | Tabletop exercise | Team knows roles and procedures | | Playbook validated | Red/blue team exercise | Detection and containment work | | Backup restoration | Restore test | RTO and RTO targets met | | DR drill | Full site failover | Recovery within RTO | | Communication tree | Call tree test | All contacts reachable within 1 hour | | Breach notification | Regulatory checklist | All notifications within legal deadlines |
Troubleshooting Scenarios
| Issue | Diagnosis | Resolution | |-------|-----------|------------| | IR team not responding | Contact info outdated | Update contact list, test quarterly | | Backup restoration fails | Corrupted backup, missing keys | Use alternate backup, verify encryption keys | | Communication systems down | Dependent on compromised infrastructure | Use out-of-band communication | | DR site not ready | Equipment not pre-positioned | Regular DR site maintenance, cloud DR as alternative | | Legal notification missed | Unclear regulatory requirements | Legal review of incident response plan | | Evidence not admissible | Chain of custody broken | Document all evidence handling, use write blockers |
Security Best Practices
- **Assume breach:** Operate with the assumption that a compromise will occur
- **Practice regularly:** Tabletop exercises quarterly, full DR drills annually
- **Out-of-band communication:** Never rely on compromised systems for IR communication
- **Preserve evidence:** Forensic integrity is critical for legal and insurance
- **Separate IR from daily ops:** IR team should have dedicated time and authority
- **Document everything:** Timeline, actions, decisions, evidence
- **Test backups before you need them:** A backup is only valid if you can restore it
- **Plan for worst case:** Total site loss, complete data encryption, key personnel unavailable
- **Update plans continuously:** After each incident, test, or infrastructure change
- **Include third parties:** Vendors, MSPs, cloud providers in IR planning
Enterprise Considerations
- **Cyber insurance:** Policy may require specific IR capabilities and response times
- **Regulatory overlap:** Multiple regulations may apply (GDPR + state laws + industry rules)
- **Third-party breaches:** Vendor compromise affecting your data
- **Insider threats:** Employee malicious action requires HR and legal coordination
- **Supply chain:** Software supply chain attacks (SolarWinds-style)
- **Geographic distribution:** Multiple data centers, cloud regions, employee locations
- **Language and culture:** Multi-national organizations need localized IR plans
- **Time zones:** 24/7 coverage for global operations
Conclusion
Incident response and disaster recovery are not about preventing all incidents—they're about minimizing damage when incidents occur. A well-prepared organization with tested plans, trained teams, and reliable backups can contain incidents quickly, recover operations within RTO, and learn from each event to strengthen defenses. The difference between a minor disruption and a catastrophic breach is preparation.
Frequently Asked Questions
**Q: What is the difference between incident response and disaster recovery?** A: Incident response focuses on security incidents (breaches, attacks). Disaster recovery focuses on restoring operations after any disruptive event (including natural disasters, hardware failure, and security incidents).
**Q: How often should we update the IR plan?** A: At least annually, and after any significant incident, infrastructure change, or regulatory change.
**Q: Should we pay ransomware demands?** A: Law enforcement advises against payment. It doesn't guarantee decryption, funds criminals, and may violate sanctions. Reliable, tested backups are the alternative.
Official Technical References
- NIST SP 800-61 Rev. 2: Computer Security Incident Handling Guide: https://csrc.nist.gov/publications/detail/sp/800-61/rev-2/final
- ISO 22301:2019 Business Continuity Management: https://www.iso.org/standard/75106.html
- NIST SP 800-34 Rev. 1: Contingency Planning Guide: https://csrc.nist.gov/publications/detail/sp/800-34/rev-1/final
- SANS Incident Handler's Handbook: https://www.sans.org/white-papers/33973/
- CISA Incident Response: https://www.cisa.gov/incident-response
Related ITYantra Tutorials
- Enterprise Cybersecurity Fundamentals (cybersecurity)
- Backup Security and Ransomware Protection (cybersecurity)
- SIEM, Security Logging and Monitoring (cybersecurity)
- Vulnerability Assessment and Patch Management (cybersecurity)
- Firewall Architecture and Network Security (cybersecurity)