docs(opensearch): add the ES → OpenSearch migration runbook for Support and Cloud

[fabrizzio-dotCMS]

Closes #37102

Proposed Changes

New — docs/backend/OPENSEARCH_MIGRATION_RUNBOOK.md. A step-by-step migration guide: a Support or Cloud engineer is handed a customer instance and this walks them from "I have not touched anything" to "the customer is on OpenSearch". It assumes no Java and no knowledge of the internals.

It is a guide, not a test plan — verification steps exist only to confirm a step landed before you move to the next one. QA coverage stays where it lives, in OPENSEARCH_MIGRATION_TESTER_GUIDE.md and OPENSEARCH_MIGRATION_TEST_PLAN.md.

Structure — the procedure leads, the reference follows.

• Before you begin. What you are about to do (one diagram), the three rules that prevent almost every incident (a phase change never copies data; going up needs a restart, coming down does not; the readiness endpoint is the only truth), the standard-vs-fast-path decision, and the access checklist.
• Part 1 — the procedure. Six stages, each with a goal, an elapsed estimate, the risk to the customer, numbered steps, and a gate you must tick before continuing.
  • Stage 0 — Assess the customer. Measure the corpus, grep the VTL for esSearch/esRaw, grade every OSGi bundle Level 1/2/3 with the unzip/grep commands, inventory Site Search, choose the path.
  • Stage 1 — Prepare the OpenSearch server. Version and separation, TLS, the least-privilege service account with its exact permission set (including the cluster:monitor/main gap from spike Spike: Validate dotCMS OpenSearch client requests against a non-admin OS 3.x user #35922), the three permission checks that catch the index-pattern-vs-cluster-id trap ([QA-G16] OpenSearch Migration — Limited (Non-Admin) OpenSearch User (TC-053–TC-058) #36222), and the OS_* settings.
  • Stage 2 — Rehearse on a clone. A dry run of the migration on a copy of the customer, to get the measured reindex duration and to find out whether their templates and plugins survive Phase 2.
  • Stage 3 — Turn on dual-write (0 → 1). Readiness role, baseline, set the phase, rolling restart, confirm the startup log, confirm the .os rows exist, confirm one write reaches both engines by document, run the full reindex, crawl Site Search, confirm with the readiness report, soak.
  • Stage 4 — Move reads to OpenSearch (1 → 2). safeToAdvance, phase, rolling restart, walk the customer's site (the moment the Stage 0 risks materialise, with a symptom→cause table), watch for read fallbacks, soak.
  • Stage 5 — Cut over (2 → 3). Final readiness check, OpenSearch availability posture, phase, fail-loud rolling restart, confirm, cooling-off then decommission.
  • Fast path. Straight to Phase 3 for a small customer, as its own sequence, with the degraded-search window and the two-window rollback stated up front.
• Part 2 — when something goes wrong. Go back from where you are (downgrade table by cost, the 3 → 2 → reindex → 1 sequence, build rollback during dual-write, phase flip mid-reindex OpenSearch: phase rollback during in-flight reindex wipes OS index store; partial orphan indices later adopted, causing silent incomplete search results #36471, the pre-migration backup index trap), emergency stop, and a symptom table that links straight to the step that fixes it.
• Part 3 — reference (R1–R15). Concepts, why four phases, the mirror strategy and drift, the neutral domain objects that replaced the Elasticsearch SDK types, VTL/viewTools, Lucene and the search endpoints, OSGi in detail, OpenSearch security in detail, the readiness report field by field, the phase setting and restart rule, clustering, Site Search rules, configuration, log lines, commands. The steps link into it; you do not have to read it.

Fixed — OPENSEARCH_MIGRATION_TESTER_GUIDE.md was stale. It still described OS_MIGRATION_INDEX_VISIBILITY_ROLE_KEY as the role that reveals .os indices in the index portlet and in /api/v1/esindex. That stopped being true in #36360: MigrationIndexVisibility is now purely phase-based — hidden in Phases 0/1/2 for everyone, shown in Phase 3, consulting no user or role — and the role key gates only the readiness endpoint. Corrected in the config table, the §6 callout, the "which phase am I in" bullets, and the FAQ, pointing readers at the readiness endpoint instead.

Cross-links. OPENSEARCH_MIGRATION.md and the tester guide now point to the guide.

Checklist

• Tests — n/a, docs only
• Translations — n/a
• Security Implications Contemplated — the guide documents the OpenSearch least-privilege service account and the readiness endpoint's role gate; it introduces no code and no new surface. Credentials shown are the published local-lab values already in docker/docker-compose-examples/single-node-os-migration; no customer or production data appears anywhere in it.

Additional Info

Every claim was verified against the code on this branch rather than carried over from the older docs — the readiness model and its derived driftPercent / esIndexedPercent / osIndexedPercent fields, the startup validator's halt-vs-fail-loud split by phase, MigrationPhase and the system-table precedence over the environment variable, MigrationIndexVisibility being phase-only, the OpenSearch role's permission set in opensearch.py, the /v1/esindex and /v1/system-table endpoint shapes, and the ESContentTool method surface.

All 92 internal anchors and all sibling-document links resolve.

Docs-only: no code, no schema, no API contract change. Nothing for CI to test and nothing for QA to exercise.

[claude]

Claude finished @fabrizzio-dotCMS's task in 5m 14s —— View job

---

Review — ES→OpenSearch migration runbook

Docs-only PR. I verified the load-bearing technical claims against the code on this branch rather than trusting the prose, since a runbook's whole value is being right. The large majority check out exactly: the readiness endpoint path (MigrationReadinessResource @Path("/v1/index/migration") + /readiness), the dual CMS-admin and role gate → 403, @Hidden from OpenAPI, the report fields (driftPercent, esIndexedPercent/osIndexedPercent measured against the DB, verdict.{safeToAdvance,safeToRollback,outOfSyncCount,summary,blockers}, docCount: -1, MISSING_COUNTERPART/COUNT_DRIFT), Phase-3 forcing safeToAdvance=true, the MigrationPhase enum names, OS_CONNECTION_ATTEMPTS=24/RETRY_SLEEP=5, SITE_SEARCH_CRAWL_MIN_CONTENT_INDEXED_PERCENT=0/DOTCMS_SHADOW_WRITE_LOG_LEVEL=WARN, the opensearch.py permission set (incl. cluster:monitor/main and the *-pattern read-only group), the /v1/esindex reindex/switch=true/delete-active-400 behavior, the /v1/system-table POST + DELETE /{key}, the doc-id identifier_languageId_variantId (→ _DEFAULT), and the #36360 phase-only visibility model that the tester-guide fix documents. That fix is correct.

One factual error stands out, and it is the kind that bites during an incident.

New Issues

• 🟠 High: docs/backend/OPENSEARCH_MIGRATION_RUNBOOK.md:245 (and repeated at :852, :1989, :1992, :1509, :1543) — the "system table wins over the environment variable" claim is inverted for env vars. MigrationPhase.current() resolves the phase via Config.getIntProperty("FEATURE_FLAG_OPEN_SEARCH_PHASE", 0). In Config.getIntProperty (dotCMS/.../util/Config.java:634), if the key is present as an actual DOT_-prefixed environment variable (environmentSetKeys, populated in readEnvironmentVariables() at Config.java:321-327), it returns the env value and never consults the system table (line 638). So the real precedence is: env var > system table > properties file / in-memory writes. The system table only wins over the properties file and over dotCMS's own in-memory writes — which is why the adjacent sub-claim that the emergency Phase-0 reset becomes a no-op is correct (that reset writes to the in-memory store). But the runbook's Option A presents the phase primarily as DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE, the standard containerized form — and on any node where that env var is set, setting the phase via the system table (Option B) silently does nothing, and an emergency-stop that clears the system table will not change the phase. That is the opposite of the "one call, all nodes" promise and directly undermines the rollback procedure. Recommend restating as "the system table overrides the properties file and in-memory config, but an explicit DOT_-prefixed environment variable overrides the system table" and adjusting Emergency-stop step 3 / the troubleshooting row accordingly. Fix this →

• 🟡 Medium: docs/backend/OPENSEARCH_MIGRATION_TESTER_GUIDE.md:284 — dangling cross-reference. The line points at "OPENSEARCH_MIGRATION_RUNBOOK.md §16 for how to read the report," but the runbook has no §16 — the readiness report is section R9 ("The readiness report, field by field"); the runbook numbers reference sections R1–R15 and Start-here §1–8. The PR's anchor-resolution check wouldn't catch this because it's prose, not a Markdown anchor link. Change "§16" → "R9".

No other inaccuracies surfaced across the claims I could verify against code. The internal-anchor sweep the author reports I couldn't reproduce here (the sandbox blocked scripting), but the anchors I spot-checked — including the double-hyphen em-dash stage anchors — resolve correctly.