Implement the foundation for migration engine by warwickschroeder · Pull Request #5894 · Particular/ServiceControl

warwickschroeder · 2026-09-15T06:36:55Z

What this adds

The foundation for moving an error instance's data from RavenDB to SQL Server or PostgreSQL. Nothing is copied yet, and the migration is not switched on anywhere.

A read-only RavenDB source. RavenMigrationSource opens both RavenDB databases, embedded or external, and refuses every write at the client (OnBeforeRequest, non-tracking sessions). It can describe itself and count every collection, but cannot read categories yet.
--migration-source-report. Prints the source's RavenDB version, where it is, both database names with the setting each came from, and a row count per collection. It reads the instance's own RavenDB settings; ServiceControl/Migration/SourcePersistenceType defaults to RavenDB.
Two persisters in one process. PersistenceFactory.CreateMigrationSource loads the source persister in its own AssemblyLoadContext, beside the configured one.
The migration engine, against fakes. Store-neutral seams in ServiceControl.Persistence/DataMigration (IMigrationSource, IMigrationTarget, IMigrationCheckpointStore, the category registry) and MigrationEngine, which copies a category in batches with checkpoint resume, ordering between categories, body read retries, throttling, skip reasons and a halt threshold. There is no SQL target yet; it runs against in-memory fakes.
Docs. docs/migration/ holds the overview, a system design diagram, and short instructions for what can be run today, linked from the README.

Tests

ServiceControl.UnitTests/Migration: engine, registry, options and the fakes.
ServiceControl.Persistence.Tests.RavenDB/DataMigration: the read-only source lifecycle.
ServiceControl.Migration.Tests (new): the source report command, and both persisters loaded in one process.

…d ServiceControl migration

…n, clarifying strategies, goals, and migration limitations.

…on source reporting - Implemented MigrationEngineFailurePathTests to validate behavior during write failures and halts. - Created MigrationEngineHaltTests to ensure categories halt correctly on systemic failures. - Added MigrationEngineOptionsTests to verify default and environment variable configurations. - Developed MigrationEngineOrderingTests to check execution order of migration categories. - Introduced MigrationEngineResumeTests to confirm no duplicates or gaps after restarts. - Added MigrationEngineRunCategoriesTests to ensure all categories run in specified order. - Implemented MigrationEngineSkipReasonTests to validate skip reasons and their aggregation. - Enhanced MigrationEngineThrottleTests to verify pause behavior for optional categories. - Updated MigrationSourceReportCommand for improved output and clarity. - Modified Help.txt and HostArguments.cs for better command descriptions. - Refined Settings.cs and PersistenceFactory.cs for clearer migration source configuration.

…r SQL migration from RavenDB

rbev

Partial review - adding as comments for now.

rbev · 2026-09-15T08:25:55Z

+| `ServiceControl/RavenDB/ClientCertificatePath` or `ServiceControl/RavenDB/ClientCertificateBase64`, with `ServiceControl/RavenDB/ClientCertificatePassword` | `SERVICECONTROL_RAVENDB_CLIENTCERTIFICATEPATH` and so on | A secured external server's client certificate |
+| `ServiceControl/ErrorRetentionPeriod` | `SERVICECONTROL_ERRORRETENTIONPERIOD` | Required. Don't change it during the move |
+
+`ServiceControl/Migration/SourcePersistenceType` defaults to `RavenDB` and needs no setting.


So given the abstractions does this technically allow two way migration? or will that be missing the migrationsource implementation?

If this defaults to the only permissible value should it even be a configuration knob?

The abstraction is designed to allow for possible "any direction" migrations, but there will not be an implementation for SQL as the source or Raven as the target for this version.

Yep - I'll remove that setting as it is redundant at the moment. I did think about removing this..

rbev · 2026-09-15T08:33:29Z

+1. Upgrade ServiceControl as normal, still on RavenDB.
+2. Set four things in configuration: the new `PersistenceType`, its connection string, `MigrationMode=true`, and which [optional data](#data-to-be-migrated-categories) they want copied.
+3. Run `--setup` to create the SQL schema. It fails against a SQL Server instance without Full-Text Search installed.
+4. Run the [dry run](#dry-run). It reports what it resolved as a source, what each category holds, and an estimate of how long ServiceControl will be closed. Read [what the dry run reports](#dry-run) before booking an outage around its estimate.


Should make sure that dry-run checks that the sql schema has been created, just for completeness given item 3

Part of the design is that there will be a bunch of source and target checks (implemented at that persistence layer) that run before migration starts. Checking the schema can be part of the SQL target checks.

rbev · 2026-09-15T08:41:38Z

+- `UniqueMessageId` keeps its value, but converts type: the source holds a string and the target column is a `uniqueidentifier`. It is the primary key, the ServicePulse URL, the retry correlation key and the body lookup key at once.
+- `StatusChangedAt` is reconstructed from `@expires` for resolved and archived messages, which is the only place RavenDB sets it. Unresolved and retry-issued messages have no `@expires`, so the copier uses the newest processing attempt's timestamp. The column is `NOT NULL`, so it cannot be left empty, but the value is harmless for those two: the retention sweep only considers resolved and archived rows, so an unresolved message never ages out whatever is written here.
+- Message bodies go through `IBodyStoragePersistence`, which owns the compression threshold and the choice of filesystem, Azure Blob or S3. The separate 102,400-byte inline threshold is not there: it lives on the ingestion path, so the copier has to apply it rather than inherit it.
+- Throughput rows are written directly rather than through the collector, and the write sets each day's count rather than adding to it.


Is throughput copied after SC starts up?
Overwrite rather than merge seems problematic here if it is, if not then it needs to be called out

rbev · 2026-09-15T08:50:35Z

+
+## The one point you can go back
+
+While ServiceControl is closed and the required copy is running, nothing except the copier has written to SQL, and the migration has written nothing to RavenDB, which is still authoritative. RavenDB's own expiration still runs, though: unless you disabled it, it keeps deleting expired failed messages and event log items, as [Goals](#goals) describes. Back up both RavenDB databases, or disable expiration on them, before you start. If you need your instance back, set `MigrationMode=false`, point `PersistenceType` back at RavenDB, and start. You lose the copy, not your data, and you can start again later.


How does the count work when ravenDB is actively deleting rows from underneath you? Do they just end up in a "skipped" bucket with an unknown reason because you ran off the end of the work with less rows than you expected?

rbev · 2026-09-15T09:00:16Z

+public interface IMigrationTarget
+{
+    /// <summary>How many rows to read per batch for this category. The target picks it because its own database sets the limits.</summary>
+    int BatchSizeFor(MigrationCategory category);


Should this be MaximumBatchSize so it's explicit that it's a limit rather than a target to the consumer?

rbev · 2026-09-15T09:01:14Z

+    /// <summary>How many rows to read per batch for this category. The target picks it because its own database sets the limits.</summary>
+    int BatchSizeFor(MigrationCategory category);
+
+    /// <summary>Saves the batch's rows and checkpointAfterBatch in one go, so progress never gets ahead of the data. Save the checkpoint exactly as given, without adding counts to it.</summary>


Suggested change

/// <summary>Saves the batch's rows and checkpointAfterBatch in one go, so progress never gets ahead of the data. Save the checkpoint exactly as given, without adding counts to it.</summary>

/// <summary>Saves the batch's rows and checkpointAfterBatch in one transaction, so progress is consistent with the data. Save the checkpoint exactly as given, without adding counts to it.</summary>

rbev · 2026-09-15T09:08:40Z

+
+                var result = await target.Write(category, batchToWrite, checkpointAfterBatch, cancellationToken);
+
+                checkpoint = checkpointAfterBatch with


If the Write call writes checkpointAfterBatch as given (as described in the interface specification) then how do these extra counts that are being added on from the result survive a crash/reboot before the next batch commits?

rbev · 2026-09-15T09:10:10Z

+                processedThisRun += bodySkips + result.Copied + result.Skipped + result.AlreadyPresent;
+                skippedThisRun += bodySkips + result.Skipped;
+
+                if (HaltThreshold.Exceeded(skippedThisRun, processedThisRun, options.HaltThresholdPercent, options.HaltThresholdMinimum))


Halt threshold is based on percent of progress rather than percent of total, does this cause a problem if there are a small number of invalid records at the start of the migration that would otherwise be ok?

warwickschroeder added 7 commits September 11, 2026 08:58

Add migration overview documentation for RavenDB to SQL transition

c878f32

Refine migration overview documentation for RavenDB to SQL transition

c0ccc15

Clarify documentation on RavenDB source requirements for containerize…

2e57527

…d ServiceControl migration

Enhance migration overview documentation for RavenDB to SQL transitio…

f76ce9f

…n, clarifying strategies, goals, and migration limitations.

Implement the foundation for migration engine

b045ec3

Clarify migration overview by detailing additional skip conditions fo…

d25e12b

…r SQL migration from RavenDB

internalautomation Bot assigned warwickschroeder Sep 15, 2026

warwickschroeder mentioned this pull request Sep 15, 2026

[DRAFT] Raven to SQL migration engine #5851

Closed

PR review fixes

b58e559

warwickschroeder requested review from abparticular, johnsimons and rbev September 15, 2026 08:17

rbev reviewed Sep 15, 2026

View reviewed changes

Navigation Menu

Uh oh!

Implement the foundation for migration engine - #5894

Implement the foundation for migration engine#5894
warwickschroeder wants to merge 8 commits into
masterfrom
warwick/migration-engine-1

warwickschroeder commented Sep 15, 2026

Uh oh!

rbev left a comment

Uh oh!

rbev Sep 15, 2026

Uh oh!

warwickschroeder Sep 15, 2026 •

edited

Loading

Uh oh!

rbev Sep 15, 2026

Uh oh!

warwickschroeder Sep 15, 2026

Uh oh!

rbev Sep 15, 2026

Uh oh!

rbev Sep 15, 2026

Uh oh!

rbev Sep 15, 2026

Uh oh!

rbev Sep 15, 2026

Uh oh!

rbev Sep 15, 2026

Uh oh!

rbev Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		## The one point you can go back

		While ServiceControl is closed and the required copy is running, nothing except the copier has written to SQL, and the migration has written nothing to RavenDB, which is still authoritative. RavenDB's own expiration still runs, though: unless you disabled it, it keeps deleting expired failed messages and event log items, as [Goals](#goals) describes. Back up both RavenDB databases, or disable expiration on them, before you start. If you need your instance back, set `MigrationMode=false`, point `PersistenceType` back at RavenDB, and start. You lose the copy, not your data, and you can start again later.

	/// <summary>Saves the batch's rows and checkpointAfterBatch in one go, so progress never gets ahead of the data. Save the checkpoint exactly as given, without adding counts to it.</summary>
	/// <summary>Saves the batch's rows and checkpointAfterBatch in one transaction, so progress is consistent with the data. Save the checkpoint exactly as given, without adding counts to it.</summary>


		var result = await target.Write(category, batchToWrite, checkpointAfterBatch, cancellationToken);

		checkpoint = checkpointAfterBatch with

Uh oh!

Conversation

warwickschroeder commented Sep 15, 2026

What this adds

Tests

Uh oh!

rbev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

warwickschroeder Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

warwickschroeder Sep 15, 2026 •

edited

Loading