Write the definitive ES → OpenSearch migration runbook for Support and Cloud

[fabrizzio-dotCMS]

Description

We have architecture and QA material for the ES → OpenSearch migration (docs/backend/OPENSEARCH_MIGRATION.md, OPENSEARCH_MIGRATION_TESTER_GUIDE.md, OPENSEARCH_MIGRATION_TEST_PLAN.md, OPENSEARCH_CLIENT_CONFIGURATION.md), but nothing that answers the question a Support or Cloud engineer actually has: "I have been handed a customer instance. How do I migrate it, start to finish, without breaking them?"

The tester guide is written for someone exercising the migration in a lab. The architecture doc is written for someone changing the code. Neither is a procedure. Today that gap is filled by tribal knowledge, and the same operational traps keep costing us round-trips:

• Enabling Phase 1 without a restart, so the OpenSearch indices are never created and every dual-write is silently swallowed.
• Assuming a phase change backfills OpenSearch (it never does — only a full reindex does).
• A missing cluster:monitor/main permission on the OpenSearch role, which dotCMS cannot distinguish from an unreachable cluster, so the migration silently halts back to Phase 0 (spike Spike: Validate dotCMS OpenSearch client requests against a non-admin OS 3.x user #35922).
• An OpenSearch role whose index pattern does not match the cluster ID, producing a reindex that returns HTTP 200 and creates nothing ([QA-G16] OpenSearch Migration — Limited (Non-Admin) OpenSearch User (TC-053–TC-058) #36222).
• Running a Site Search crawl before the content reindex in Phases 2/3, producing a permanently truncated index that reindexing afterwards does not repair ([QA-G17] OpenSearch Migration — Site Search Functional Validation (Phases 0–3, $sitesearch viewtool) #36360).
• Not knowing that /api/v1/index/migration/readiness exists, or that it needs both CMS Admin and the migration support role.
• OSGi plugins and VTL templates still bound to Elasticsearch SDK types, which fail at Phase 2 and not before (#37870, #37559, Aggregation return-type change breaks existing VTL templates accessing $results.aggregations #36026).

We need one document that a Support or Cloud engineer can follow from zero, without reading Java.

Separately, OPENSEARCH_MIGRATION_TESTER_GUIDE.md still describes OS_MIGRATION_INDEX_VISIBILITY_ROLE_KEY as the role that reveals .os indices in the index portlet. That stopped being true in #36360: MigrationIndexVisibility is now purely phase-based (hidden in Phases 0/1/2 for everyone, shown in Phase 3) and the role key only gates the readiness endpoint. The stale text should be corrected in the same change so the doc set does not contradict itself.

Acceptance Criteria

• A new docs/backend/OPENSEARCH_MIGRATION_RUNBOOK.md written for a Support / Cloud audience, assuming no knowledge of the internals
• Explains the migration concepts and vocabulary, and why four phases exist rather than a single cutover
• Explains the mirror strategy for the active index: what dual-write does, what it does not do (no retroactive backfill), and every source of drift
• Explains how one dotCMS supports two engines: the single routing layer, and the replacement of Elasticsearch SDK types with dotCMS domain objects
• Covers the impact surface: VTL viewTools, Lucene queries and the search REST endpoints, Site Search, and OSGi plugins — including how to audit a customer's bundles and what to do when a plugin bug is found
• Covers OpenSearch server prerequisites and security criteria: version, endpoint separation, TLS, auth, and the least-privilege service account with its exact permission set
• Covers what happens when dotCMS cannot reach OpenSearch, per phase (safe degradation in 1/2, fail-loud in 3), with the log lines to recognise
• Covers migration strategy and customer selection criteria: number and size of indices, contentlet volume, measured reindex duration, plugin exposure
• Documents rehearsing the migration on a clone of the customer instance before touching the real environment
• Documents the fast path — going straight to Phase 3 for a small customer whose reindex fits in a window — with its cost stated explicitly
• States clearly when a node restart is required after changing the migration phase, and when it is not
• Covers clustered instances: node phase consistency, rolling restarts, and traffic direction during the change
• Covers downgrading to the previous phase and what to watch out for, including coming back from Phase 3
• Documents GET /api/v1/index/migration/readiness as the source of truth, the role that protects it, and how to read the report
• OPENSEARCH_MIGRATION_TESTER_GUIDE.md corrected: the visibility role no longer gates the index portlet ([QA-G17] OpenSearch Migration — Site Search Functional Validation (Phases 0–3, $sitesearch viewtool) #36360)
• The four existing OpenSearch docs cross-link to the runbook

Priority

Medium

Additional Context

Docs-only change. No code, no schema, no API surface — nothing to test in CI and nothing for QA to exercise.

The runbook is derived from the existing architecture doc plus the operational findings accumulated across #35922, #36222, #36360, #36471, #36983, #37870 and the QA epic #35476. It is intended to become the entry point of the OpenSearch doc set: the architecture doc explains why, the runbook explains how, the tester guide explains how to exercise it in a lab.

[github-actions]

PRs linked to this issue

• docs(opensearch): add the ES → OpenSearch migration runbook for Support and Cloud by @fabrizzio-dotCMS