Monitoring, Alerting, and Incident Response
Monitoring for the Big Three
In SOC 2, monitoring isn't just about uptime; it's about proving that your Security, Availability, and Confidentiality controls are working in real-time. You must detect deviations from your security baseline across these three specific Trust Services Criteria (TSC).
Welcome to our deep dive into SOC 2 monitoring. To pass an audit, your monitoring strategy must go beyond simple health checks to cover the 'Big Three' criteria. Click each criterion to see what auditors expect you to monitor. For Security, focus on CC7.1 and 7.2. You need to detect unauthorized access, like credential stuffing or 'impossible travel'—where a user logs in from two distant locations within a timeframe that is physically impossible. Availability is about performance and capacity. Auditors look for evidence that you monitor latency spikes, 5xx error rates, and resource exhaustion that could threaten your service level agreements. Confidentiality monitoring focuses on data protection. You must alert on unauthorized data exfiltration or any non-admin user querying a production database containing PII.
- Security (CC7.1, CC7.2): Detect unauthorized access or 'impossible travel'.
- Availability: Monitor performance, capacity, and SLA-impacting errors.
- Confidentiality: Watch for unauthorized access to sensitive PII data stores.
Creating Actionable Alerts
A common audit failure is alert fatigue. SOC 2 requires that alerts are actionable and traceable, forming an unbroken chain from detection to resolution.
Auditors hate noise because noise leads to ignored alerts. Instead of alerting on every single 403 error, you should set high-fidelity triggers. Notice how we only alert on a sustained spike, which suggests a real attack rather than a typo. To maintain compliance, you need an 'unbroken chain.' When a threshold is crossed in Datadog, it must automatically trigger a ticket in PagerDuty or Jira. This ensures every alert is documented and tracked.
- Use High-Fidelity triggers to reduce noise.
- Ensure alerts are traceable via automated workflows.
- Connect monitoring tools (Datadog/Prometheus) directly to incident systems (PagerDuty/Jira).
Scenario: The Leaked Key
Walk through the Incident Response Plan (IRP) for a leaked AWS access key. Follow the steps to ensure audit-ready documentation.
Let's practice a real-world scenario. A developer accidentally commits an AWS key to a public repo. First, our secret scanner triggers an immediate alert. Click the alert to begin the triage. The on-call engineer acknowledges the alert. This must happen within the timeframe defined in your Incident Response Plan to stay compliant. Now, choose the remediation action. Correct. We revoke the key and rotate it using a secrets manager. Finally, we update the ticket with timestamps for detection, acknowledgement, and resolution to provide the 'unbroken chain' for the auditor.
- Detection via automated secret scanning.
- Triage within the defined IRP timeframe.
- Remediation through revocation and rotation.
- Full documentation of timestamps and actions.
The Compliant Post-Mortem
The Post-Mortem (or Root Cause Analysis) is critical audit evidence. It proves you improve the system rather than just fixing symptoms.
After an incident is resolved, you must document a post-mortem. This isn't just a summary; it's a formal record. You need a precise timeline, a clear root cause, and specific corrective actions tracked in your ticketing system.
- Timeline: Precise timestamps of the incident lifecycle.
- Root Cause: Technical explanation of 'why' it happened.
- Corrective Actions: Tracked tickets to prevent recurrence.
- Metadata: List of people involved and systems affected.
Incident Room: PII Leak
A monitoring alert indicates a non-admin user is querying the Production PII Database. Respond to the Security Lead to manage the incident compliantly.
An alert has fired: 'Unauthorized PII Access Detected'. You are the lead engineer on call. Respond to the Security Lead and walk through the remediation steps. Remember: auditors will look for formal tracking, not just Slack messages.
- Identify the scope of the potential breach.
- Follow the formal Incident Response Plan.
- Ensure all decisions are moved from chat to the formal ticket.
Common Pitfalls in Audits
Avoid these two major traps that lead to audit exceptions: 'Slack-only' responses and missing the 'Population of Incidents'.
Finally, let's look at why developers fail audits. First is the 'Slack-only' trap. If the resolution isn't in a formal ticket, it didn't happen as far as the auditor is concerned. Second is the tagging failure. If you don't tag your tickets consistently, you can't generate the 'Population of Incidents' report for the last year.
- Slack is for coordination; tickets are for evidence.
- Consistent tagging (e.g., 'Security-Incident') is required for reporting.
- Auditors will ask for a list of ALL incidents over the last 12 months.