Incident Response: Preparing for AI-Caused Accidents
Risk management prepares "on the premise accidents happen." Have procedures ready for AI-caused incidents (info leakage, wrong output, runaway).
Assumed Incidents
- Inputting confidential/personal info into external AI
- Putting AI's wrong output straight to a customer
- Agent/integration runaway or misoperation
- Information leakage via prompt injection
Skeleton of the Response Procedure
- Detect/report: a contact where whoever notices can report immediately
- Initial action: stop damage spread (halt integration, revoke keys, take offline)
- Preservation: preserve logs/history without deleting
- Identify scope: what, to whom, how far
- Communicate: stakeholders, and if needed customers/authorities
- Prevent recurrence: reflect the cause into rules/design
