Troubleshooting
Map 401, 403, and 503 failures back to the exact gateway API key checks that produced them.
When a gateway key fails, Athena usually tells you which stage failed. This page maps those failures back to the runtime implementation.
Fast triage
Common failures
| Symptom | Likely cause | What to check |
|---|---|---|
API key required | enforcement is on and no key was presented | X-Athena-Key, client override, global enforcement |
Invalid API key | bad format, unknown public_id, or secret mismatch | plaintext key shape, creation output, copy/paste errors |
Inactive API key | is_active = false | PATCH /admin/api-keys/{id} state |
Expired API key | expires_at is in the past | expiration timestamp and clock skew |
API key client mismatch | key bound to another client | client_name on the key and X-Athena-Client on the request |
Insufficient API key rights | missing required right | granted rights, wildcard behavior, route requirement |
IP blocked | blacklist match | global blacklist and per-key blacklist |
IP not allowed | whitelist exists and caller is not in it | global whitelist, per-key whitelist, virgin lock-in outcome |
Client IP required | IP rules exist but Athena cannot resolve the caller IP | X-Real-IP, trusted X-Forwarded-For, reverse proxy config |
API key validation unavailable | auth store lookup failed | auth client connectivity, pool saturation, fail mode |
API key policy unavailable | enforcement lookup failed | auth client connectivity, api_key_config access, fail mode |
Inspect the stored record
curl "http://localhost:4052/admin/api-keys" \
-H "X-Athena-Admin-Key: $ATHENA_KEY_12"Confirm:
public_idmatches the presented key prefixclient_namematches the request clientis_activeis trueexpires_atis still in the future- the required rights are granted
Inspect IP policy and learning state
curl "http://localhost:4052/admin/api-keys/$API_KEY_ID/ip-policy" \
-H "X-Athena-Admin-Key: $ATHENA_KEY_12"
curl "http://localhost:4052/admin/api-keys/$API_KEY_ID/ip-seen?limit=100" \
-H "X-Athena-Admin-Key: $ATHENA_KEY_12"Use these responses to answer:
- Is the key still unresolved in
virgin_mode? - Which whitelist and blacklist entries currently apply?
- Did Athena lock in more or fewer hosts than expected?
- Are the learned rows marked
locked_in?
Inspect enforcement configuration
curl "http://localhost:4052/admin/api-key-config" \
-H "X-Athena-Admin-Key: $ATHENA_KEY_12"
curl "http://localhost:4052/admin/api-key-clients" \
-H "X-Athena-Admin-Key: $ATHENA_KEY_12"Remember that:
- global enforcement lives in
api_key_config - client overrides live in
api_key_client_config - enforcement decisions are cached briefly, so changes may take about two seconds to settle
Use the auth logs
Athena writes request-level auth outcomes into api_key_auth_log. This is the
best place to confirm the internal failure reason code, granted rights, and
whether the request was authenticated versus authorized.
Useful fields include:
api_key_public_idclient_namerequired_rightsgranted_rightsauthenticatedauthorizedenforcedfailure_reasonremote_addr
503-specific guidance
If you see 503s with auth-related messages, check:
- the Postgres pool for
gateway.auth_client - whether
gateway.auth_clientandgateway.logging_clientpoint at the same saturated database - whether
gateway.api_key_fail_modeis stillfail_closed
If the errors coincide with pool timeouts, also read Gateway Pool Starvation And Auth Failures.