Skip to content

auth-providers: OIDC callback hardening - #10931

Open
MichaelUray wants to merge 9 commits into
hcengineering:developfrom
MichaelUray:feat/oidc-hardening
Open

MichaelUray wants to merge 9 commits into
hcengineering:developfrom
MichaelUray:feat/oidc-hardening

Conversation

@MichaelUray

@MichaelUray MichaelUray commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

OIDC robustness fixes that surfaced while integrating Huly with Authentik for a production deployment. All commits are standalone and have no dependency on the forthcoming WAC v1 stack. The branch grew beyond the original three commits; the sections below cover the current state.

Commits

1. fix(authProviders): instrument OIDC callback + redirect on fail (not 500)

Problem: Every OIDC callback (bogus or real code) returned 500 Internal Server Error, even with valid sessions. No log line was emitted between try auth via openid and the 500 — passport-strategy errors threw silently before handleProviderAuth could log them.

Fix:

  • Switch to the explicit-callback passport.authenticate variant so strategy errors surface as err/info/status args instead of being swallowed.
  • Privacy-safe diagnostic shape: no raw code / state / tokens / user. Logged fields are stage, errMessage/Name/Stack, hasUser, infoSummary, statusCode, hasSession + sessionKeys (names only), cookie + state + code presence (lengths only), host, forwardedProto.
  • Permanent invalid-callback guard: ALL failures → 302 /login?error=oidc_callback_failed, never 500. Same wrap covers handleProviderAuth throws.

2. fix(account): admin.ts trim+lowercase on both env-set AND lookup + account(admin): keep non-email ADMIN_EMAILS entries by default; strict mode opt-in

Problem: server/account/src/admin.ts built the ADMIN_EMAILS set without normalizing. All callers pass an already lowercased/trimmed email (cleanEmail), so an env value such as ADMIN_EMAILS=" Michael@Example.com " never matched and silently produced a non-admin login.

Fix: Env split → trim → toLowerCase → drop empty entries; lookup normalizes the same way. Every non-empty entry that matched before still matches, and entries with uppercase letters or surrounding whitespace now match too.

Deliberate tightening — empty entries: the original new Set(process.env.ADMIN_EMAILS?.split(',') ?? []) put '' into the set for any empty entry — ADMIN_EMAILS='', a leading comma (,admin), a trailing comma (admin,, e.g. when PLATFORM_ADMIN_EMAILS is empty) or a double comma (admin,,ops@example.com). isAdminEmail('') was then true, and loginOrSignUpWithProvider passes '' when the provider returns no email, so such a login received an admin token. Empty entries are now dropped in every position and isAdminEmail('') is always false. A deployment that relied on this would lose admin for email-less provider logins; that is intended.

Entries without @ are kept by default — the repo's own dev/, tests/ and ws-tests/ compose files use ADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS}, so dropping them would revoke admin rights on upgrade. A startup warning lists such entries. Deployments that want the fail-closed behaviour opt in with ADMIN_EMAILS_STRICT=true. (An intermediate commit on this branch dropped them by default behind ADMIN_EMAILS_ALLOW_LOGIN_ID; that flag is gone, see the review thread.)

Tests cover whitespace, case-insensitive env and lookup, kept-with-warn for no-@ entries (including admin,…), the strict opt-in (only the exact value true; false/TRUE/1/empty keep entries), no warning for clean values, empty/unset env, empty entries in every position ('', ,admin, admin,, admin,,ops@example.com) and multi-entry. Each test clears and restores ADMIN_EMAILS/ADMIN_EMAILS_STRICT, so none depends on the outer environment.

3. chore(authProviders): drop cookieHeaderLen from OIDC diagnostics

Rationale: hasCookieHeader (boolean) is sufficient to diagnose missing session-cookie issues; the cookieHeaderLen byte count adds no operational value and is one more PII-adjacent surface. Removed from both diagnostic paths in pods/authProviders/src/openid.ts.

4. auth(oidc): gate verbose callback diagnostics behind OIDC_DEBUG

Every failed OIDC callback logged the full diagnostic block (errStack, sessionKeys, host, …) at error level, so repeated invalid callbacks from anonymous clients could flood the log. Normal operation now logs a single short error line; the full block only with OIDC_DEBUG=true.

5. auth(oidc): redirect to /login when provider auth yields no account

handleProviderAuth returns '' when the login cannot be resolved (e.g. no account and sign-up disabled). Previously no redirect happened and next() ran into nothing, leaving a hanging/empty response. Now fail-closed: explicit 302 /login?error=oidc_no_account.

6. auth(oidc): retry strategy registration with capped backoff

The account service registered the OIDC passport strategy once at startup via a single fire-and-forget Issuer.discover(). A transient issuer outage at that moment left the oidc strategy permanently unregistered (500 "Unknown authentication strategy" on every OpenID login until a restart; observed for ~10 days).

  • registerOidcStrategyWithRetry() retries the whole chain (discover → Client → Strategy → passport.use) with capped exponential backoff (5 s, ×2, cap 60 s); discover and sleep are injectable for tests. It never rejects.
  • Warn gating: first 5 attempts and every 60th thereafter; error stacks only under OIDC_DEBUG.
  • Routes stay registered synchronously; while registration is pending, /auth/openid answers 503 with Retry-After: 5.
  • First tests for the package: the retry helper in isolation and the registerOpenid wiring.

Out of scope

The OIDC name-split heuristic for IdPs that violate OIDC §5.1 (given_name = full_name) ships separately in #10919 — Authentik will fix this upstream in v2026.8 (PR #21544), so the Huly-side patch remains defense-in-depth.

DCO

All commits Signed-off-by Michael Uray. maintainer_can_modify=true.

@huly-github-staging

Copy link
Copy Markdown

Connected to Huly®: UBERF-16534

…500)

Production-blocking: every OIDC callback (bogus or real code) returned
500 Internal Server Error, even with valid sessions. No log was emitted
between `try auth via openid` and the 500 — passport strategy threw
silently before `handleProviderAuth` could log.

Per Codex plan-review amendments:
- Switch to explicit-callback passport.authenticate variant so strategy
  errors surface as err/info/status args instead of being swallowed.
- Privacy-safe diagnostic shape (no raw code/state/tokens/user). Fields:
  stage, errMessage/Name/Stack, hasUser, infoSummary, statusCode,
  hasSession + sessionKeys (names only), cookie + state + code
  PRESENCE (lengths only), host, forwardedProto.
- Permanent invalid-callback guard: ALL failures → 302 to /login with
  ?error=oidc_callback_failed, never 500. Same wrap covers
  handleProviderAuth throws.

Image-tag: hardcoreeng/account:integration-wac-2026-06-21-codex-r7.

Next step (P0-T2): user attempts login, we capture the actual error
from the diagnostic, then ship a surgical fix in r8.

Signed-off-by: Michael Uray <michael.uray@gmail.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Codex plan-review Critical: server/account/src/admin.ts:15-18 built the
ADMIN_EMAILS set without normalizing, AND isAdminEmail() didn't normalize
the input. Today value is clean so isAdmin works, but a future env-value
like ADMIN_EMAILS=" Michael@Uray.io " would silently break admin
detection (Set lookup would miss the lowercased input).

Mongo backend already has this hardening (collections/mongo.ts:1141-1147).
Postgres + this helper didn't.

Now BOTH sides normalize: env split -> trim -> toLowerCase -> filter
non-empty + warn on no-@ entries (Codex Optional: warn not reject --
too sharp for a latent fix). Lookup also trim().toLowerCase().

Standalone upstream-PR-able per Codex Q-I3.

Tests cover whitespace, case-insensitive env, case-insensitive lookup,
invalid-shape warn, empty/unset env, multi-entry.

Signed-off-by: Michael Uray <michael.uray@gmail.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…amendment)

Codex E4 amendment (optional, low-priority): hasCookieHeader is
sufficient to diagnose missing session-cookie issues; the
cookieHeaderLen byte count adds no operational value and is one more
PII-adjacent surface to keep an eye on. Drop it from both diagnostic
paths in pods/authProviders/src/openid.ts.

Signed-off-by: Michael Uray <michael.uray@gmail.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…TH-2)

Bei jedem fehlgeschlagenen OIDC-Callback wurde ein umfangreicher Diag-Block
(errStack/sessionKeys/host/…) auf error-Level geloggt -> anonymes Log-
Flooding via wiederholter invalider Callbacks. Normalbetrieb loggt jetzt nur
eine knappe Fehlerzeile; der volle Diag-Block nur mit OIDC_DEBUG=true.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
… allowed (L-AUTH-3)

Ein Tippfehler wie ADMIN_EMAILS=admin,michel@... legte bislang einen
unbeabsichtigten Admin-Identifier an (Eintrag ohne @ wurde still behalten).
Jetzt werden @-lose Eintraege per Default verworfen (fail-closed) und nur
bei ADMIN_EMAILS_ALLOW_LOGIN_ID=true als Login-ID akzeptiert; Warnung stets.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…L-AUTH-4)

handleProviderAuth liefert '' bei einem nicht aufloesbaren Login (z.B. kein
Account + Signup disabled). Bisher erfolgte dann kein Redirect und next()
lief ins Leere -> haengende/leere Antwort. Jetzt fail-closed: explizites
302 auf /login?error=oidc_no_account.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
The account service registered the OIDC passport strategy only once at
startup via a single fire-and-forget Issuer.discover(). A transient issuer
outage in that one boot moment left the 'oidc' strategy permanently
unregistered, producing 500 "Unknown authentication strategy" on every
OpenID login until a manual container restart (observed: ~10 days).

Make registration self-healing:

- Extract registerOidcStrategyWithRetry(), which retries the ENTIRE
  registration chain (discover -> new Client -> new Strategy -> passport.use)
  with capped exponential backoff (5s, x2, cap 60s), unbounded. The loop only
  returns after passport.use succeeds, so a failure at client or strategy
  construction (e.g. temporarily incomplete issuer metadata during an IdP
  upgrade) is retried too instead of permanently disabling OIDC. discover and
  sleep are injectable for tests; passport is a structural { use } interface.
  It never rejects: the success log runs failure-safe AFTER passport.use and
  returns immediately, so a throwing logger can never re-enter the loop and
  register the strategy a second time; the failure warn is likewise guarded so
  a throwing sink cannot break the never-rejects contract.

- Gate log flooding via shouldWarnOnAttempt(): warn on the first 5 attempts
  and every 60th thereafter (~24 warn lines/day at the 60s cap instead of
  ~1440), following the branch's L-AUTH-2 rationale; error stacks only under
  OIDC_DEBUG. Success logs once with the attempt count.

- Wire it fire-and-forget in registerOpenid() and flip an oidcReady flag on
  success. The .then body is a plain boolean flip that cannot throw; a trailing
  .catch(() => {}) is belt-and-suspenders so this fire-and-forget chain can
  never surface an unhandled rejection. Routes stay registered synchronously.

- During the pending window, /auth/openid responds 503 with Retry-After: 5
  and a short text body instead of a 500 or a redirect (the login app ignores
  an error query param, and L-AUTH-4 only covers the callback path).

Add the package's first tests: the retry helper in isolation (schedule, 60s
cap, whole-attempt retry incl. client-construction failure, warn gating,
OIDC_DEBUG both ways, a throwing success log that must not re-register, and a
throwing failure warn that must not reject) and the registerOpenid wiring
(synchronous route registration, 503 pending window, exactly-once strategy
registration).

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Comment thread server/account/src/admin.ts Outdated
const invalid: string[] = []
const kept: string[] = []
for (const entry of entries) {
if (entry.includes('@')) {

@ArtyomSavchenko ArtyomSavchenko Sep 25, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it can affect existing instances where admin email is specified without valid email. Perhaps some configurations can use something like ADMIN_EMAILS=admin

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — the repo's own compose files ship ADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS} (dev/, tests/, ws-tests/) and tests/prepare-pg.sh creates that admin account, so the fail-closed default would have revoked admin on upgrade.

Changed in a02b7f6: entries without @ are kept by default again (original semantics, now trimmed + lowercased), and a startup warning lists them. Dropping them is an explicit opt-in via ADMIN_EMAILS_STRICT=true. The ADMIN_EMAILS_ALLOW_LOGIN_ID flag from the earlier commit is removed, so there is a single switch. Tests cover both modes, including ADMIN_EMAILS=admin,… staying admin.

One deliberate difference from the original code remains: empty entries are dropped. The old split(',') put '' into the set for ADMIN_EMAILS='', ,admin, admin, or admin,,x, which made a provider login without an email an admin. isAdminEmail('') is now always false; this is covered by tests and described in the PR body.

…t mode opt-in

Review feedback on hcengineering#10931: dropping entries without "@" by default would
silently revoke admin rights on upgrade for deployments that use login-id
style values. The repo's own dev/tests/ws-tests compose files ship
ADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS} and tests/prepare-pg.sh creates
that `admin` account.

Restore the original semantics as the default: every non-empty entry is
kept (trimmed + lowercased), and entries without "@" are logged once at
startup so a typo stays visible. The fail-closed behaviour becomes an
explicit opt-in via ADMIN_EMAILS_STRICT=true. ADMIN_EMAILS_ALLOW_LOGIN_ID,
introduced earlier on this branch and never released, is removed so there
is a single switch.

Empty entries stay dropped. This is a deliberate tightening compared to
the original `new Set(ADMIN_EMAILS.split(','))`, which put '' into the set
for any empty entry (ADMIN_EMAILS='', ',admin', 'admin,', 'admin,,ops@...')
and so made isAdminEmail('') true. loginOrSignUpWithProvider passes '' when
the provider returns no email, so such a login received an admin token.
isAdminEmail('') is now always false.

Tests: default keeps `admin` and warns, strict drops, only the exact value
"true" enables strict mode, no warning for clean values, empty entries in
every position never make an empty email an admin, trim/lowercase coverage
unchanged. Each test saves, clears and restores ADMIN_EMAILS and
ADMIN_EMAILS_STRICT so none depends on the outer environment.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
@MichaelUray MichaelUray changed the title auth-providers: OIDC callback hardening (3 commits) auth-providers: OIDC callback hardening Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants