auth-providers: OIDC callback hardening - #10931
MichaelUray wants to merge 9 commits into
Conversation
|
Connected to Huly®: UBERF-16534 |
…500) Production-blocking: every OIDC callback (bogus or real code) returned 500 Internal Server Error, even with valid sessions. No log was emitted between `try auth via openid` and the 500 — passport strategy threw silently before `handleProviderAuth` could log. Per Codex plan-review amendments: - Switch to explicit-callback passport.authenticate variant so strategy errors surface as err/info/status args instead of being swallowed. - Privacy-safe diagnostic shape (no raw code/state/tokens/user). Fields: stage, errMessage/Name/Stack, hasUser, infoSummary, statusCode, hasSession + sessionKeys (names only), cookie + state + code PRESENCE (lengths only), host, forwardedProto. - Permanent invalid-callback guard: ALL failures → 302 to /login with ?error=oidc_callback_failed, never 500. Same wrap covers handleProviderAuth throws. Image-tag: hardcoreeng/account:integration-wac-2026-06-21-codex-r7. Next step (P0-T2): user attempts login, we capture the actual error from the diagnostic, then ship a surgical fix in r8. Signed-off-by: Michael Uray <michael.uray@gmail.com> Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Codex plan-review Critical: server/account/src/admin.ts:15-18 built the ADMIN_EMAILS set without normalizing, AND isAdminEmail() didn't normalize the input. Today value is clean so isAdmin works, but a future env-value like ADMIN_EMAILS=" Michael@Uray.io " would silently break admin detection (Set lookup would miss the lowercased input). Mongo backend already has this hardening (collections/mongo.ts:1141-1147). Postgres + this helper didn't. Now BOTH sides normalize: env split -> trim -> toLowerCase -> filter non-empty + warn on no-@ entries (Codex Optional: warn not reject -- too sharp for a latent fix). Lookup also trim().toLowerCase(). Standalone upstream-PR-able per Codex Q-I3. Tests cover whitespace, case-insensitive env, case-insensitive lookup, invalid-shape warn, empty/unset env, multi-entry. Signed-off-by: Michael Uray <michael.uray@gmail.com> Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…amendment) Codex E4 amendment (optional, low-priority): hasCookieHeader is sufficient to diagnose missing session-cookie issues; the cookieHeaderLen byte count adds no operational value and is one more PII-adjacent surface to keep an eye on. Drop it from both diagnostic paths in pods/authProviders/src/openid.ts. Signed-off-by: Michael Uray <michael.uray@gmail.com> Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…TH-2) Bei jedem fehlgeschlagenen OIDC-Callback wurde ein umfangreicher Diag-Block (errStack/sessionKeys/host/…) auf error-Level geloggt -> anonymes Log- Flooding via wiederholter invalider Callbacks. Normalbetrieb loggt jetzt nur eine knappe Fehlerzeile; der volle Diag-Block nur mit OIDC_DEBUG=true. Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
… allowed (L-AUTH-3) Ein Tippfehler wie ADMIN_EMAILS=admin,michel@... legte bislang einen unbeabsichtigten Admin-Identifier an (Eintrag ohne @ wurde still behalten). Jetzt werden @-lose Eintraege per Default verworfen (fail-closed) und nur bei ADMIN_EMAILS_ALLOW_LOGIN_ID=true als Login-ID akzeptiert; Warnung stets. Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…L-AUTH-4) handleProviderAuth liefert '' bei einem nicht aufloesbaren Login (z.B. kein Account + Signup disabled). Bisher erfolgte dann kein Redirect und next() lief ins Leere -> haengende/leere Antwort. Jetzt fail-closed: explizites 302 auf /login?error=oidc_no_account. Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
The account service registered the OIDC passport strategy only once at
startup via a single fire-and-forget Issuer.discover(). A transient issuer
outage in that one boot moment left the 'oidc' strategy permanently
unregistered, producing 500 "Unknown authentication strategy" on every
OpenID login until a manual container restart (observed: ~10 days).
Make registration self-healing:
- Extract registerOidcStrategyWithRetry(), which retries the ENTIRE
registration chain (discover -> new Client -> new Strategy -> passport.use)
with capped exponential backoff (5s, x2, cap 60s), unbounded. The loop only
returns after passport.use succeeds, so a failure at client or strategy
construction (e.g. temporarily incomplete issuer metadata during an IdP
upgrade) is retried too instead of permanently disabling OIDC. discover and
sleep are injectable for tests; passport is a structural { use } interface.
It never rejects: the success log runs failure-safe AFTER passport.use and
returns immediately, so a throwing logger can never re-enter the loop and
register the strategy a second time; the failure warn is likewise guarded so
a throwing sink cannot break the never-rejects contract.
- Gate log flooding via shouldWarnOnAttempt(): warn on the first 5 attempts
and every 60th thereafter (~24 warn lines/day at the 60s cap instead of
~1440), following the branch's L-AUTH-2 rationale; error stacks only under
OIDC_DEBUG. Success logs once with the attempt count.
- Wire it fire-and-forget in registerOpenid() and flip an oidcReady flag on
success. The .then body is a plain boolean flip that cannot throw; a trailing
.catch(() => {}) is belt-and-suspenders so this fire-and-forget chain can
never surface an unhandled rejection. Routes stay registered synchronously.
- During the pending window, /auth/openid responds 503 with Retry-After: 5
and a short text body instead of a 500 or a redirect (the login app ignores
an error query param, and L-AUTH-4 only covers the callback path).
Add the package's first tests: the retry helper in isolation (schedule, 60s
cap, whole-attempt retry incl. client-construction failure, warn gating,
OIDC_DEBUG both ways, a throwing success log that must not re-register, and a
throwing failure warn that must not reject) and the registerOpenid wiring
(synchronous route registration, 503 pending window, exactly-once strategy
registration).
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
bfcb6c8 to
514bdc5
Compare
| const invalid: string[] = [] | ||
| const kept: string[] = [] | ||
| for (const entry of entries) { | ||
| if (entry.includes('@')) { |
There was a problem hiding this comment.
I think it can affect existing instances where admin email is specified without valid email. Perhaps some configurations can use something like ADMIN_EMAILS=admin
There was a problem hiding this comment.
Correct — the repo's own compose files ship ADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS} (dev/, tests/, ws-tests/) and tests/prepare-pg.sh creates that admin account, so the fail-closed default would have revoked admin on upgrade.
Changed in a02b7f6: entries without @ are kept by default again (original semantics, now trimmed + lowercased), and a startup warning lists them. Dropping them is an explicit opt-in via ADMIN_EMAILS_STRICT=true. The ADMIN_EMAILS_ALLOW_LOGIN_ID flag from the earlier commit is removed, so there is a single switch. Tests cover both modes, including ADMIN_EMAILS=admin,… staying admin.
One deliberate difference from the original code remains: empty entries are dropped. The old split(',') put '' into the set for ADMIN_EMAILS='', ,admin, admin, or admin,,x, which made a provider login without an email an admin. isAdminEmail('') is now always false; this is covered by tests and described in the PR body.
…t mode opt-in Review feedback on hcengineering#10931: dropping entries without "@" by default would silently revoke admin rights on upgrade for deployments that use login-id style values. The repo's own dev/tests/ws-tests compose files ship ADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS} and tests/prepare-pg.sh creates that `admin` account. Restore the original semantics as the default: every non-empty entry is kept (trimmed + lowercased), and entries without "@" are logged once at startup so a typo stays visible. The fail-closed behaviour becomes an explicit opt-in via ADMIN_EMAILS_STRICT=true. ADMIN_EMAILS_ALLOW_LOGIN_ID, introduced earlier on this branch and never released, is removed so there is a single switch. Empty entries stay dropped. This is a deliberate tightening compared to the original `new Set(ADMIN_EMAILS.split(','))`, which put '' into the set for any empty entry (ADMIN_EMAILS='', ',admin', 'admin,', 'admin,,ops@...') and so made isAdminEmail('') true. loginOrSignUpWithProvider passes '' when the provider returns no email, so such a login received an admin token. isAdminEmail('') is now always false. Tests: default keeps `admin` and warns, strict drops, only the exact value "true" enables strict mode, no warning for clean values, empty entries in every position never make an empty email an admin, trim/lowercase coverage unchanged. Each test saves, clears and restores ADMIN_EMAILS and ADMIN_EMAILS_STRICT so none depends on the outer environment. Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Summary
OIDC robustness fixes that surfaced while integrating Huly with Authentik for a production deployment. All commits are standalone and have no dependency on the forthcoming WAC v1 stack. The branch grew beyond the original three commits; the sections below cover the current state.
Commits
1.
fix(authProviders): instrument OIDC callback + redirect on fail (not 500)Problem: Every OIDC callback (bogus or real code) returned
500 Internal Server Error, even with valid sessions. No log line was emitted betweentry auth via openidand the 500 — passport-strategy errors threw silently beforehandleProviderAuthcould log them.Fix:
passport.authenticatevariant so strategy errors surface aserr/info/statusargs instead of being swallowed.302 /login?error=oidc_callback_failed, never 500. Same wrap covershandleProviderAuththrows.2.
fix(account): admin.ts trim+lowercase on both env-set AND lookup+account(admin): keep non-email ADMIN_EMAILS entries by default; strict mode opt-inProblem:
server/account/src/admin.tsbuilt theADMIN_EMAILSset without normalizing. All callers pass an already lowercased/trimmed email (cleanEmail), so an env value such asADMIN_EMAILS=" Michael@Example.com "never matched and silently produced a non-admin login.Fix: Env split → trim → toLowerCase → drop empty entries; lookup normalizes the same way. Every non-empty entry that matched before still matches, and entries with uppercase letters or surrounding whitespace now match too.
Deliberate tightening — empty entries: the original
new Set(process.env.ADMIN_EMAILS?.split(',') ?? [])put''into the set for any empty entry —ADMIN_EMAILS='', a leading comma (,admin), a trailing comma (admin,, e.g. whenPLATFORM_ADMIN_EMAILSis empty) or a double comma (admin,,ops@example.com).isAdminEmail('')was thentrue, andloginOrSignUpWithProviderpasses''when the provider returns no email, so such a login received an admin token. Empty entries are now dropped in every position andisAdminEmail('')is alwaysfalse. A deployment that relied on this would lose admin for email-less provider logins; that is intended.Entries without
@are kept by default — the repo's owndev/,tests/andws-tests/compose files useADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS}, so dropping them would revoke admin rights on upgrade. A startup warning lists such entries. Deployments that want the fail-closed behaviour opt in withADMIN_EMAILS_STRICT=true. (An intermediate commit on this branch dropped them by default behindADMIN_EMAILS_ALLOW_LOGIN_ID; that flag is gone, see the review thread.)Tests cover whitespace, case-insensitive env and lookup, kept-with-warn for no-
@entries (includingadmin,…), the strict opt-in (only the exact valuetrue;false/TRUE/1/empty keep entries), no warning for clean values, empty/unset env, empty entries in every position ('',,admin,admin,,admin,,ops@example.com) and multi-entry. Each test clears and restoresADMIN_EMAILS/ADMIN_EMAILS_STRICT, so none depends on the outer environment.3.
chore(authProviders): drop cookieHeaderLen from OIDC diagnosticsRationale:
hasCookieHeader(boolean) is sufficient to diagnose missing session-cookie issues; thecookieHeaderLenbyte count adds no operational value and is one more PII-adjacent surface. Removed from both diagnostic paths inpods/authProviders/src/openid.ts.4.
auth(oidc): gate verbose callback diagnostics behind OIDC_DEBUGEvery failed OIDC callback logged the full diagnostic block (errStack, sessionKeys, host, …) at error level, so repeated invalid callbacks from anonymous clients could flood the log. Normal operation now logs a single short error line; the full block only with
OIDC_DEBUG=true.5.
auth(oidc): redirect to /login when provider auth yields no accounthandleProviderAuthreturns''when the login cannot be resolved (e.g. no account and sign-up disabled). Previously no redirect happened andnext()ran into nothing, leaving a hanging/empty response. Now fail-closed: explicit302 /login?error=oidc_no_account.6.
auth(oidc): retry strategy registration with capped backoffThe account service registered the OIDC passport strategy once at startup via a single fire-and-forget
Issuer.discover(). A transient issuer outage at that moment left theoidcstrategy permanently unregistered (500 "Unknown authentication strategy" on every OpenID login until a restart; observed for ~10 days).registerOidcStrategyWithRetry()retries the whole chain (discover → Client → Strategy →passport.use) with capped exponential backoff (5 s, ×2, cap 60 s);discoverandsleepare injectable for tests. It never rejects.OIDC_DEBUG./auth/openidanswers503withRetry-After: 5.registerOpenidwiring.Out of scope
The OIDC name-split heuristic for IdPs that violate OIDC §5.1 (
given_name = full_name) ships separately in #10919 — Authentik will fix this upstream in v2026.8 (PR #21544), so the Huly-side patch remains defense-in-depth.DCO
All commits Signed-off-by Michael Uray.
maintainer_can_modify=true.