Incident history
Last 90 days · every S1 gets a written post-incident reviewA regional inverter vendor API began returning malformed payloads, which filled the ingest retry queue and delayed live generation data for roughly 8% of monitored assets. Readings were buffered, not lost, and backfilled within 40 minutes of the fix. We have added payload validation at the edge and a per-vendor circuit breaker.
A slow query on the document index pushed p95 latency on document reads to 3.2s. No writes were affected and no data was at risk. The query has been rewritten and an index added; p95 is now under 220ms.
Our SMS provider throttled traffic on one route, delaying job-assignment messages in three states. Email and in-app notifications were unaffected. We now fail over to a secondary route automatically after 90 seconds of queueing.
A retry bug delivered a small number of payment webhooks out of sequence. No payment was taken twice and no milestone was released incorrectly. Affected customers were contacted directly; ordering guarantees are now enforced by sequence number.
Scheduled maintenance
Warehouse index rebuild
Sunday 9 August, 02:00–04:00 IST. Reporting and exports may be delayed. No impact to the application, API or telemetry ingest.
Database engine upgrade
Sunday 12 July, 02:00–03:10 IST. Completed 50 minutes ahead of the window with no customer-visible downtime.