Incident INC-0412 · EU data plane degraded
Meta Lead Ads webhooks are timing out, causing a backlog on the webhooks queue and p95 latency of 386 ms in Frankfurt. Leads are queued, not lost. Investigating since 09:40 CET · owner Arjun Rao.
🌐Control plane
Global · ap-south-1 Mumbai
Operational
84ms
p95 API latency
Uptime 30d
99.99%
Tenants routed
8
Postgres 16 · primary + 2 replicas
🇮🇳India · IN
Data plane · ap-south-1 Mumbai
Operational
112ms
p95 API latency
Uptime 30d
99.98%
Tenants
3
Postgres 16 · 1.8 TB
🇪🇺Europe · EU
Data plane · eu-central-1 Frankfurt
Degraded
386ms
p95 API latency
Uptime 30d
99.91%
Tenants
3
Postgres 16 · 0.6 TB
🇦🇪UAE
Data plane · me-central-1 Dubai
Operational
131ms
p95 API latency
Uptime 30d
99.97%
Tenants
2
Postgres 16 · 0.4 TB

Queue depths

BullMQ · all planes combined

Redis 7 · 41% mem
workflows
1.2k/min processed
142
Waiting
18
Active
40
Delayed
3
Failed
sequences
640/min processed
1,860
Waiting
32
Active
6,200
Delayed
11
Failed
ai-jobs
310/min processed
420
Waiting
48
Active
0
Delayed
27
Failed
webhooks
2.4k/min processed
1,310
Waiting
24
Active
120
Delayed
64
Failed

AI provider latency

p95 seconds · last 24 h

ClaudeOpenAIGemini
Claude errors
0.2%
1.9M calls
OpenAI errors
0.9%
1.1M calls
Gemini errors
0.4%
0.4M calls

Failed jobs

Exhausted or erroring retries, last hour

JobCompanyErrorAttemptsWhen
meta.leadgen.ingest
webhooks · job_8f21c
VDVan Dijk Makelaarsplane EUETIMEDOUT after 30s (graph.facebook.com)5/54 min ago
call.summarise
ai-jobs · job_8f1a9
ANAl Noor Motors LLCplane UAE429 rate_limit_error from provider3/59 min ago
n8n.lead.created
webhooks · job_8f0d2
BABrightlane Agencyplane EU502 Bad Gateway (customer endpoint)5/512 min ago
whatsapp.send.template
sequences · job_8ef77
SRSkyline Realtyplane INTemplate "site_visit_v3" not approved1/128 min ago
deal.stage.changed
workflows · job_8ee40
NWNordwind Software B.V.plane EUWorkspace in grace: write blocked2/541 min ago
lead.score.batch
ai-jobs · job_8ed13
MDMedPlus Diagnosticsplane INContext length exceeded (213k)5/51 h ago

Integration error rates

Outbound calls, 24 h

Catalog
Meta Lead Ads4.8%
48.2k
WhatsApp Cloud API1.1%
126k
Gmail / Google Workspace0.3%
72.4k
Microsoft 3650.4%
18.9k
Twilio Voice2.2%
9.1k
Funda / Bayut portals0.9%
6.3k

Maintenance windows

Affected companies get an in-app banner 72 h and 1 h before.

Postgres minor upgrade (16.4 to 16.6)
Plane IN · Sun 8 Nov · 02:00 to 03:00 IST
Read-only for up to 10 min · notified: 3 companies
In progress
Redis failover drill
Plane EU · Sat 14 Nov · 01:00 to 01:30 CET
No downtime expected · notified: –
Draft
ESC
↑↓navigate↵open⌘Btoggle sidebar

Schedule maintenance

Creates a status-page entry and an in-app banner for affected companies.

Scope
Notify company owners by email
Pause sequences and workflows during the window
Affects 3 companies in EU: Nordwind Software B.V., Van Dijk Makelaars, Brightlane Agency. Local time 01:00 CET.