In short
A reworked keymaster check in Snapchat 12.4 was not caught by our automated diff because the change was in a native library, not in Java classes. Manual characterisation took 34 hours. 1,847 customer accounts were lost. All were credited and rebuilt at a cost of roughly $91,000. We added native-library diffing and continuous keymaster fuzzing to the canary pipeline.
What happened
Snapchat 12.4.1 shipped on 14 November 2025 at 09:12 UTC. Our crawler picked it up at 09:19. The automated diff completed at 10:04 and reported nothing significant — 11 modified Java classes, all in unrelated packages.
The change was in libsnapattest.so, a native library. Our diff pipeline at the time compared decompiled Java and string tables but did not disassemble native libraries. So we saw nothing, and we did not start manual investigation until 21:40 when ban reports started arriving from customers.
Timeline
| Time (UTC) | Event |
|---|---|
| 14 Nov 09:12 | Snapchat 12.4.1 published |
| 14 Nov 09:19 | Crawler picks it up |
| 14 Nov 10:04 | Automated diff reports no significant change |
| 14 Nov 21:40 | First customer ban reports |
| 14 Nov 23:15 | Incident declared; manual investigation begins |
| 15 Nov 14:20 | Native library identified as the changed surface |
| 16 Nov 08:50 | New keymaster check characterised |
| 16 Nov 19:30 | Patch built and regression-tested |
| 16 Nov 22:15 | Fleet-wide OTA complete |
Impact
- 1,847 customer Snapchat accounts banned across 312 customers.
- No other network affected. No data loss. No downtime.
- All affected device-months credited under Ban Shield; all identities rebuilt free of charge.
- Direct cost to PhoneCloudify: approximately $91,000 in credits and engineering time.
Root cause
The diff pipeline had a coverage gap. We had built it in 2023 when attestation logic lived in Java, and we never revisited the assumption after Snap moved theirs into native code in early 2025. That is a straightforward engineering failure and it is ours.
What we changed
- Native library disassembly added to the automated diff, covering all 27 tracked apps.
- Continuous keymaster fuzzing in the canary pipeline — we now probe our own attestation surface every 4 hours against a corpus of known check patterns.
- Ban-report threshold alerting: 5 reports of the same shape within 60 minutes now pages the on-call engineer directly, rather than routing through support triage.
- Ban Shield guarantee extended from Scale to Growth, because the customers hit hardest were on Growth and were not covered at the time. We backdated the credit anyway.
With native diffing in place, we re-ran the v3 release against the current pipeline. Detection would have occurred at T+1:10 and the patch would have shipped in approximately 18 hours.
We publish the bad numbers because the alternative is asking you to trust the good ones on faith.