[Flix6x]

Closes steps 1 and 2 of #2377. Shipped in v1.1.0.

If you maintain your own indexes on timed_belief

This is what the v1.1.0 changelog points here for. Two things change for hand-added indexes on that table:

1. Any index on a prefix of the new primary key is now redundant. The key is reordered to (sensor_id, source_id, event_start, belief_horizon, cumulative_probability), and a btree serves a leading-column lookup just as well as a dedicated index does, so the primary key already answers everything such an index answers.

The migration does not drop these. They are not part of FlexMeasures' schema — only the deployment that created one knows what it was for — so b7e5a2c40f18 names the ones it finds in the upgrade output and leaves the decision to you. Keep that output. Dropping one afterwards, without locking the table:

DROP INDEX CONCURRENTLY <name>;

The report is deliberately strict about what it names: partial, expression, unique, constraint-backing, non-btree and INCLUDE indexes are all excluded, as is anything with a non-default sort order or operator class, since the primary key cannot substitute for those. It is silent when there is nothing to name, which is the case for any database whose schema FlexMeasures created.

2. The single-column indexes on event_start and sensor_id are dropped for you. Those two are project schema — timely-beliefs declared index=True on both before 4.2.0 — and each is fully covered by a composite index the same library declares:

single-column index	covered by
(event_start)	timed_belief_search_session_idx (event_start, sensor_id, source_id) INCLUDE (belief_horizon)
(sensor_id)	timed_belief_search_session_singleevent_idx (sensor_id, event_start)

Whether your database carries one, both or neither depends on its age and on the timely-beliefs version that first created the table, so the migration reports exactly which ones it dropped. They are matched structurally rather than by name — single-column, non-unique, non-partial, not backing a constraint — so this works whatever naming convention your indexes were created under, and is a no-op where they are already gone. On a downgrade both are recreated, so a database that carried only one of them gains the other; dropping that one again is safe.

Both migrations run online, so neither needs a maintenance window. See Migrations below for what that costs and what a failed run leaves behind.

What

Pins the column order of timed_belief's primary key with an explicit PrimaryKeyConstraint, and reorders it to:

(sensor_id, source_id, event_start, belief_horizon, cumulative_probability)

The set of key columns is unchanged, so uniqueness semantics are identical — this is purely a reordering. timely_beliefs passes index_elements to on_conflict_do_update as a set, so upserts are unaffected.

Also drops the two redundant single-column indexes described above, and moves the timely-beliefs floor to >=4.2.0 in the same commit — deliberately, since dropping them while the ORM still declared them would recreate exactly the create_all()-vs-migrated divergence this PR set out to fix.

Why pin it

TimedBeliefDBMixin marks five columns primary_key=True and leaves the order to SQLAlchemy, which collects attributes declared on the subclass before the mixin's. So the key's shape is an accident of which columns a host happens to redeclare, and it silently changes if anyone adds a column override.

That is not hypothetical: FlexMeasures' source_id override already moves it to the front, and a schema built by create_all() therefore disagrees with a migrated one. Nothing catches that today. The new tests do.

Why this order

• sensor_id first — virtually every query filters on a single sensor. A key that does not lead with it cannot serve those queries at all, which is exactly why deployments end up hand-adding a sensor-leading composite index.
• source_id second — this makes (sensor_id, source_id, event_start, belief_horizon) a prefix of the key, so such a hand-added index becomes redundant. On the dataset profiled this was the largest and most-scanned index on the table, and the queries that used it get faster, since they are now served by a key that no longer needs a second structure maintained alongside it on every insert.
• cumulative_probability last — it is very nearly a constant (0.5 for every deterministic belief), so it contributes no selectivity wherever it sits.
• sensor_id and source_id adjacent — both are 4-byte integers, so keeping them together avoids the alignment padding that separating them forces into every index tuple. Measured at roughly 15% of the index's size, which was the most surprising result of the profiling.

Migrations

d4a7c1e93b52 swaps the primary key. It builds the replacement with CREATE UNIQUE INDEX CONCURRENTLY inside an autocommit_block, then promotes it with ADD CONSTRAINT ... USING INDEX. Building the index is the slow part and does not block reads or writes; only the promotion takes an ACCESS EXCLUSIVE lock, and that is catalog-only (milliseconds), because the index already exists by then. This rebuilds an index but does not rewrite the table — no heap pages are touched, and no other index is affected.

b7e5a2c40f18 drops the two redundant single-column indexes, CONCURRENTLY as well, and then reports the prefix-redundant indexes it will not touch. Reporting it there rather than in d4a7c1e93b52 keeps the list to indexes FlexMeasures leaves alone, since the two it drops itself are gone by that point. Both migrations print() as well as log, because FlexMeasures' logging setup does not surface the alembic logger and the upgrade output is the only record a deployment gets.

The trade for staying online is that the concurrent steps are not transactional: a run that fails partway can leave an unused — possibly invalid — timed_belief_pkey_new behind. Both migrations clean up before they start, so a retry is safe.

Both resolve timed_belief's schema once from the catalog with pg_table_is_visible, rather than trusting search_path, so a same-named table in another schema is never touched and the checks cannot disagree with the DDL that acts on them.

Downgrades restore the previous key order and recreate both single-column indexes.

Known trade-off

After the reorder no index leads with source_id, so deleting a data_source — whose foreign key is NO ACTION — falls back to a sequential scan. Profiled and discussed in a comment below: it costs a single ~20-second statement for an operation no CLI or API path performs, against a permanent index costing ~12% of what this PR reclaims. Accepted as such; easily reversed with a CREATE INDEX CONCURRENTLY if source deletion ever becomes routine.

Dropping the sensor_id index, by contrast, is safe for the ON DELETE CASCADE on its foreign key: PostgreSQL only needs to find rows by sensor_id, and timed_belief_search_session_singleevent_idx leads with exactly that column. A test pins that property.

Verification

Driven through a real MigrationContext transaction against PostgreSQL, so autocommit_block behaves as it does under env.py:

• upgrade reorders the key and drops both redundant single-column indexes; re-running is a no-op; the downgrades restore everything with rows intact; no invalid indexes are left at any point; an injected stale timed_belief_pkey_new is cleaned up on retry.
• decoys survive: an INCLUDE variant of the composite, partial indexes on event_start and on sensor_id, an expression index, a unique index on event_start, a constraint-backed unique on sensor_id, and a single-column index on another column.
• in a two-schema database, another schema's timed_belief indexes are left alone.
• against a database built by running the migration chain with a hand-added prefix index present, plus three near misses (partial, DESC, wrong column order): only the prefix index is named, and none is dropped.

Six new tests (test_timed_belief_schema.py, no database needed) pin the key's order, assert sensor_id leads it, assert the two integer columns stay adjacent, assert timely_beliefs' own indexes survive the __table_args__ override, assert no single-column index is declared again, and assert both formerly-indexed columns still lead a composite.

Full flexmeasures/data suite passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JkJ55eNSD5PMUbURUNBSNQ

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34338388 | 📁 Comparing 93f080e against latest (fecf13a)

  🔍 Preview build  

93 files changed · + 12 added · ± 81 modified

+ Added

• _autodoc/COUPLING.html
• _autodoc/COUPLING_COEFFICIENT.html
• _autosummary/flexmeasures.api.v3_0.utils.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.planning.highspy_optimization.html
• _autosummary/flexmeasures.data.models.planning.scheduling_problem.html
• _autosummary/flexmeasures.data.schemas.automations.html
• _autosummary/flexmeasures.data.services.automations.html
• _autosummary/flexmeasures.utils.sentry_utils.html
• concepts/flexibility-configuration.html
• and 2 more...

± Modified

• changelog.html
• configuration.html
• genindex.html
• getting-started.html
• http-routingtable.html
• index.html
• py-modindex.html
• _autodoc/CONSUMPTION_CAPACITY.html
• _autodoc/PRODUCTION_CAPACITY.html
• _autodoc/RELAX_CAPACITY_CONSTRAINTS.html
• and 71 more...

[Flix6x]

One consequence I should flag rather than leave for review to find.

timed_belief.source_id has a foreign key to data_source.id with no ON DELETE clause, so it is NO ACTION: deleting a data source makes PostgreSQL check for referencing beliefs. Under the old key order that check was indexed, because source_id led the primary key. After this reorder no index leads with source_id — not the new key, not either of the timely_beliefs search indexes — so that check becomes a sequential scan of the beliefs table.

I do not think this blocks the change:

• there is no CLI or API path that deletes an individual data source; the only bulk delete is in data_gen.py, i.e. the development reset path;
• a source that does have beliefs cannot be deleted anyway (the FK rejects it) — the scan just makes the rejection slow;
• an index dedicated to source_id would cost roughly what one of the redundant single-column indexes being removed in step 2 costs, which would eat a good part of the saving for a rare administrative operation.

So my inclination is to accept it as a deliberate trade rather than add an index for it. But it is a real regression in that one path, and it should be a conscious decision rather than something noticed later — if we do start deleting data sources routinely, an index on source_id is the fix.

Worth noting the sensor side is fine: sensor_id leads the new key, so the ON DELETE CASCADE from sensor stays index-driven, and in fact gets faster.

[Flix6x]

Added step 2 in 9ce56acf, now that SeitaBV/timely-beliefs#244 is released in 4.2.0.

New migration b7e5a2c40f18 (separate file, revising d4a7c1e93b52) drops the redundant single-column indexes on event_start and sensor_id. Each was fully covered by a composite index timely_beliefs declares leading with the same column, so neither offered anything a query could use — they only occupied space on our largest table and slowed every write that had to maintain them.

The floor moves to timely-beliefs>=4.2.0 in the same commit deliberately: dropping the indexes while the ORM still declared them would recreate exactly the create_all()-vs-migrated divergence the first commit set out to fix.

The migration matches structurally rather than by name — single-column, non-unique, not backing a constraint, on one of the two columns — so it works whatever naming convention a deployment's indexes were created under, and is a no-op where they are already gone.

Verified against a real PostgreSQL table carrying both redundant indexes plus four deliberate decoys: a unique index on event_start, a constraint-backed unique on sensor_id, a single-column index on another column, and both composites. Exactly the two intended indexes were dropped, everything else survived, the downgrade recreated them, and a repeated upgrade was a no-op.

Safety of dropping the sensor_id index: the ON DELETE CASCADE from sensor only needs to find rows by sensor_id, and timed_belief_search_session_singleevent_idx leads with that column. A new test pins that, so a future reordering of those composites cannot quietly remove the coverage.

Tests: flexmeasures/data gives 249 passed against tb 4.2.0, which is the previous baseline plus the two new tests. The two test_closest_sensor failures are pre-existing on main and environmental (my sandbox lacks the cube/earthdistance extensions).

One thing to look at: uv.lock shows ~3500 changed lines, but the only dependency change is timely-beliefs 4.0.1 -> 4.2.0. The rest is my local uv (0.10.11) normalising the file by adding upload-time to every entry. uv lock --upgrade-package timely-beliefs produces the identical churn, so it is not avoidable at my end. If you would rather keep the lock in its current format, regenerate it with whichever uv produced the committed one and force-push over my version — or say the word and I will drop the lock from the commit and let CI regenerate it.

[Flix6x]

Profiled the source_id foreign-key regression I flagged above, so the trade is on record rather than a guess. Measured against a realistic dataset with the full post-reorder index set in place and a real NO ACTION foreign key — not a synthetic query.

scenario	FK check plan	check	actual DELETE
after the reorder, no source_id index	Seq Scan	~15 s	~20 s
plus a dedicated source_id index	Index Scan	0.04 ms	0.3 ms
the old source_id-leading key	Index Scan on the key	0.04 ms	2.4 ms

So the regression is large in relative terms — roughly four orders of magnitude on the delete — but its absolute cost is a single ~20-second statement.

Two details worth knowing:

• The worst case is deleting an unused source. The check is SELECT 1 … WHERE source_id = $1 LIMIT 1, so when a source has no beliefs nothing matches and the scan has to read the entire table. A source that does have beliefs stops at the first match — about 4 s before raising the expected FK violation. Counterintuitively, the source you care least about is the slow one to delete.
• It is not an outage. No exclusive lock is taken; it holds ROW SHARE and pins a transaction open for the duration, which matters more on a busy ingestion path than the wall clock suggests, but it does not block reads or writes.

Recommendation: accept it, don't add the index

The mitigation costs a permanent index — on the order of 12% of the space this PR reclaims — to speed up an operation that no CLI or API path performs. The only bulk delete of data sources is in data_gen.py, i.e. the development reset, which is already tearing everything down.

Paying that forever to turn a rare 20-second admin command into an instant one is the wrong way round.

What would change the answer: if deleting data sources ever becomes routine — a UI action, a cleanup job, pruning of versioned sources — add the index then. It is a one-line CREATE INDEX CONCURRENTLY, fully online, with no migration-ordering concerns, so the decision is cheaply reversible in both directions.

Worth noting the old key was never free here either. It happened to cover this case as a side effect of an ordering that costs us on every read and every write. This change makes that implicit subsidy explicit, which seems the right way round — but it is a deliberate trade, so it should be someone's decision rather than a discovery later.

[Copilot]

🟡 Not ready to approve

The migrations have correctness issues (downgrade restores the wrong prior PK order, and redundant-index detection isn’t schema-scoped and may target the wrong table in multi-schema databases).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Pins and standardizes the timed_belief primary key column order (and removes redundant indexes) to ensure ORM-generated schemas (create_all()) and migrated schemas stay aligned, while improving index usefulness and footprint for common query patterns.

Changes:

• Add an explicit PrimaryKeyConstraint on TimedBelief to pin PK column order to (sensor_id, source_id, event_start, belief_horizon, cumulative_probability).
• Add Alembic migrations to reorder the PK and drop redundant single-column indexes on timed_belief.
• Add schema-level tests to assert PK order and index expectations, and bump timely-beliefs to >=4.2.0.

File summaries

File	Description
pyproject.toml	Bumps timely-beliefs floor to >=4.2.0 to match ORM/index expectations.
flexmeasures/data/models/time_series.py	Pins timed_belief PK order via explicit PrimaryKeyConstraint while preserving mixin indexes.
flexmeasures/data/tests/test_timed_belief_schema.py	Adds metadata-only tests to enforce PK order and index presence/absence.
flexmeasures/data/migrations/versions/d4a7c1e93b52_reorder_timed_belief_primary_key.py	Rebuilds PK in the new order and conditionally drops a now-redundant composite index.
flexmeasures/data/migrations/versions/b7e5a2c40f18_drop_redundant_timed_belief_indexes.py	Drops redundant single-column indexes on event_start and sensor_id based on structural detection.
documentation/changelog.rst	Adds changelog entries describing the PK reorder and index drops.

Review details

Suppressed comments (1)

flexmeasures/data/migrations/versions/d4a7c1e93b52_reorder_timed_belief_primary_key.py:76

• The downgrade primary-key column order (OLD_ORDER) does not match the pre-upgrade order created by FlexMeasures migrations (see 04f0e2d2924a, which created (event_start, belief_horizon, cumulative_probability, sensor_id, source_id)). As written, a downgrade will not restore the prior migrated schema shape.

# The order SQLAlchemy produced before the primary key was pinned explicitly.
OLD_ORDER = [
    "source_id",
    "event_start",
    "belief_horizon",
    "cumulative_probability",
    "sensor_id",
]

• Files reviewed: 6/7 changed files
• Comments generated: 6
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

There are a couple of concrete migration robustness/conventions issues (docstring line-breaking convention and unsafe identifier quoting in a DROP INDEX statement) that should be fixed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

flexmeasures/data/migrations/versions/d4a7c1e93b52_reorder_timed_belief_primary_key.py:33

• Module docstring breaks a line mid-phrase (ends with "inside an"), but this repo’s docstring convention is to only break lines after punctuation (see .github/instructions/docstrings.instructions.md:38). Please reflow these lines so each line ends with punctuation.

The swap is done online, so this does not need a maintenance window.
Building the replacement index is the slow part, and it runs ``CONCURRENTLY`` inside an
``autocommit_block``, so reads and writes continue throughout.

flexmeasures/data/migrations/versions/b7e5a2c40f18_drop_redundant_timed_belief_indexes.py:88

• The migration claims it quotes index identifiers safely, but wrapping the name in double quotes without escaping embedded quotes is not equivalent to PostgreSQL’s quoting and can make the migration fail on oddly-named indexes. Escape embedded double quotes before interpolating the identifier.

    for name in names:
        # Quote the identifier the same way PostgreSQL would, in case of odd naming.
        with op.get_context().autocommit_block():
            op.execute(f'DROP INDEX CONCURRENTLY IF EXISTS "{name}"')

• Files reviewed: 6/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Flix6x]

Round 2 addressed in e93ee02f. Both findings were valid.

Unsafe identifier quoting. The DROP INDEX statement wrapped the index name in double quotes itself, which is not equivalent to PostgreSQL's quoting — a name containing a double quote produced a syntax error rather than a valid statement. I verified that rather than taking it on trust: against an index named weird"quoted_es_idx, the old form fails with syntax error at or near "quoted_es_idx".

Fixed by letting PostgreSQL do the quoting. The lookup now returns quote_ident(schema) || '.' || quote_ident(index), so the identifier arrives ready to interpolate and is correct whether it needs quoting, contains quotes, or is mixed case. Tested against weird"quoted_es_idx and Mixed Case sid idx — both found and dropped.

Docstring line breaks. Reflowed. Copilot flagged one; I found a second by mechanically checking every added line against the convention rather than re-reading them, which is the better guard.

Re-verified after the change: the migrations still pass the full end-to-end run (upgrade, both downgrades, no-op re-run, decoy and other-schema survival, no invalid indexes at any point, stale-index retry), and flexmeasures/data gives 249 passed with the two pre-existing test_closest_sensor failures unchanged.

[Copilot]

🟡 Not ready to approve

The main changelog entry currently violates the repository’s one-sentence changelog entry convention and should be reflowed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

documentation/changelog.rst:52

• Main changelog entries are expected to be one sentence (no period at the end). This new entry is split into multiple sentences with periods ("...redundant." / "Requires..." / "The migration...") which breaks the established format in this file.

* Speed up sensor data queries and free up disk space by reordering the ``timed_belief`` primary key to lead with ``sensor_id``, and dropping three indexes it makes redundant. Requires ``timely-beliefs`` 4.2.0. The migration runs online, so no maintenance window is needed, but building the new index can take a while on a large database [see `PR #2378 <https://www.github.com/FlexMeasures/flexmeasures/pull/2378>`_]

• Files reviewed: 6/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Flix6x]

Round 3 addressed in e04694471 — valid again, and documented: .github/instructions/changelog.instructions.md asks for one sentence, no trailing period, written for end users with technical details abstracted away.

The entry is now:

Speed up sensor data queries and free up disk space by reordering the timed_belief primary key to lead with sensor_id and dropping three indexes it makes redundant, in a migration that runs online and so needs no maintenance window (though it can take a while on a large database)

I dropped the timely-beliefs version from it — that is a packaging detail the migration docstring and this PR already record — and kept the part hosts actually act on: no maintenance window needed.

That is three rounds of Copilot review; all findings across them were valid, and two were substantive (the missing schema constraint, and the identifier quoting). Handing back to you from here rather than looping further.

[Copilot]

🟡 Not ready to approve

The migrations’ index-matching logic can drop unintended non-redundant indexes (e.g., superset/partial/expression indexes) due to overly broad catalog checks, which should be tightened before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

flexmeasures/data/migrations/versions/d4a7c1e93b52_reorder_timed_belief_primary_key.py:90

• The redundant-index guard uses indexdef LIKE '%(sensor_id, source_id, event_start, belief_horizon)%', which can also match indexes that start with those columns but have extra key columns or INCLUDE columns. In that case, the migration would drop a non-redundant index despite the comment claiming it only drops an exact structural match. Consider switching to a pg_index-based check that verifies: (a) exactly 4 key columns, (b) no INCLUDE columns, (c) no predicate/expression, and (d) the exact key-column name sequence.

 WHERE schemaname = current_schema()
   AND tablename = 'timed_belief'
   AND indexname = :name
   AND indexdef LIKE '%(sensor_id, source_id, event_start, belief_horizon)%'
"""

flexmeasures/data/migrations/versions/b7e5a2c40f18_drop_redundant_timed_belief_indexes.py:71

• FIND_REDUNDANT_INDEXES will also match and drop partial or expression indexes on event_start/sensor_id (e.g. a deployment-specific partial index), because it only checks indnatts = 1 and the leading attribute name. Since the intent is to remove the plain single-column indexes previously created by timely-beliefs (which are non-partial and non-expression), it’s safer to exclude indexes with predicates (indpred) or expressions (indexprs).

   AND n.nspname = current_schema()
   AND x.indnatts = 1
   AND NOT x.indisunique
   AND NOT x.indisprimary
   AND a.attname = ANY(:columns)

• Files reviewed: 6/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[nhoening]

I tried this locally and the migration(s) worked.

I would like to make explicit what the benefits are. I understand that it makes our system more predictable, as sqlachemy updates can change the order unexpectedly.

And we save space as we drop an index, as well as time to maintain this index with each db update.

But are there probably speedups in queries to be expected right away?
Or maybe per definition none, and the only drawback has already been pointed out (deleting sources)?

To show you what happened on my dev db (might not resemble production). For me, only one index was dropped, and I did not have the single-column index on ``sensor_id`:

"timed_belief_event_start_idx" btree (event_start)

Maybe I didn't have the one(s) that would be dropped on production.

This is my before and after:

flexmeasures_prod=> \d timed_belief
                            Table "public.timed_belief"
         Column         |           Type           | Collation | Nullable | Default 
------------------------+--------------------------+-----------+----------+---------
 source_id              | integer                  |           | not null | 
 event_start            | timestamp with time zone |           | not null | 
 belief_horizon         | interval                 |           | not null | 
 cumulative_probability | double precision         |           | not null | 
 event_value            | double precision         |           | not null | 
 sensor_id              | integer                  |           | not null | 
Indexes:
    "timed_belief_pkey" PRIMARY KEY, btree (source_id, event_start, belief_horizon, cumulative_probability, sensor_id)
    "timed_belief_event_start_idx" btree (event_start)
    "timed_belief_search_session_idx" btree (event_start, sensor_id, source_id)
    "timed_belief_search_session_singleevent_idx" btree (sensor_id, event_start)
Foreign-key constraints:
    "timed_belief_sensor_id_sensor_fkey" FOREIGN KEY (sensor_id) REFERENCES sensor(id) ON DELETE CASCADE
    "timed_belief_source_id_data_source_fkey" FOREIGN KEY (source_id) REFERENCES data_source(id)

flexmeasures_prod=> select * from alembic_version ;
 version_num  
--------------
 3bc1e29ca1f4
(1 row)

flexmeasures_prod=> \d timed_belief
                            Table "public.timed_belief"
         Column         |           Type           | Collation | Nullable | Default 
------------------------+--------------------------+-----------+----------+---------
 source_id              | integer                  |           | not null | 
 event_start            | timestamp with time zone |           | not null | 
 belief_horizon         | interval                 |           | not null | 
 cumulative_probability | double precision         |           | not null | 
 event_value            | double precision         |           | not null | 
 sensor_id              | integer                  |           | not null | 
Indexes:
    "timed_belief_pkey" PRIMARY KEY, btree (sensor_id, source_id, event_start, belief_horizon, cumulative_probability)
    "timed_belief_search_session_idx" btree (event_start, sensor_id, source_id)
    "timed_belief_search_session_singleevent_idx" btree (sensor_id, event_start)
Foreign-key constraints:
    "timed_belief_sensor_id_sensor_fkey" FOREIGN KEY (sensor_id) REFERENCES sensor(id) ON DELETE CASCADE
    "timed_belief_source_id_data_source_fkey" FOREIGN KEY (source_id) REFERENCES data_source(id)

On production db we run right now, the indexes look indeed different:

[Flix6x]

Thanks for the approval and the comment, Nicolas — the wording fix is in 966513ef (replied on the thread).

That same commit also merges main, since a fair amount landed there while this sat open, including the v1.0.0 release. Two mechanical notes on the merge:

• The changelog conflict was because this branch's entry had landed inside what is now the released v1.0.0 section; moved it into the new v1.1.0 section instead.
• uv.lock is regenerated on top of main's own lock (upgrading only timely-beliefs) rather than hand-merged; most of that diff is marker-string reformatting from whichever uv version wrote it, not a real dependency change.

One thing worth flagging rather than leaving for someone to discover in CI: flexmeasures/data now shows 3 failures instead of 2 on this branch (test_closest_sensor ×2, plus test_sensor_data_sources_and_data_source_sensors_load_fast). I checked the new one rather than assuming it was this PR's fault — it reproduces identically against plain origin/main with none of this branch's changes (131.5 ms there against its 100 ms budget, vs. 115-120 ms on this branch), so it is a pre-existing, machine-speed-dependent test, not something this PR introduces or worsens.

[Flix6x]

Thanks for running the migration against a real database — that is the part I could not check myself.

On the benefits. You have the two real ones; I would rather not oversell a third.

• Predictability, as you say. The key's order is now pinned and tested, so a SQLAlchemy or timely-beliefs change cannot reshape it silently and leave a create_all() schema disagreeing with a migrated one.
• Space and write cost: the dropped index(es), plus the reorder itself — putting the two 4-byte integer columns next to each other removes 8 bytes of alignment padding from every tuple of the largest index in the database.
• Query speed: essentially none on day one, and I would rather say that plainly than imply otherwise. The dominant read path (one sensor, a time range) was already served by timed_belief_search_session_singleevent_idx (sensor_id, event_start), which is smaller than the primary key, so I expect the planner to keep choosing it. What the reorder really changes is that the primary key stops being an index our main access pattern can never use: leading with source_id, it was maintained on every write and read by almost nothing. I have not measured a day-one speedup on ordinary queries and am not claiming one — name a specific query and I will measure that one.

There is a genuine cost in the other direction, which is the drawback you are circling: after the reorder no index leads with source_id, so anything that looks up beliefs by source alone has to scan. Deleting a data source is the visible case, since timed_belief_source_id_data_source_fkey is not ON DELETE CASCADE and PostgreSQL has to prove no belief references the row. That is a trade I took deliberately. The sensor_id side is fine, for the opposite reason: that FK is ON DELETE CASCADE, and search_session_singleevent_idx leads with sensor_id, which is exactly what PostgreSQL needs for it. #2382 then removes the other big source-first reader — "which sensors did this source write to" — by keeping a small summary table instead of asking timed_belief.

On only one index being dropped. That is the migration working as intended, not a case it failed to handle. It does not drop by name; it matches by shape — single-column, non-unique, not backing a constraint, not partial, not an expression index — on event_start or sensor_id, and it skips whatever is absent. Which of the two a database actually has depends on when its timed_belief was first created and by which timely-beliefs version, so dev, staging and production legitimately differ, as your two outputs show. Yours had timed_belief_event_start_idx and no sensor_id equivalent, so exactly one drop is the correct outcome there.

One asymmetry I would like your opinion on, since your database is precisely the case that exposes it: downgrade() recreates both indexes unconditionally, so downgrading yours would leave it with a sensor_id index it never had. I chose that because the pre-migration canonical schema (timely-beliefs < 4.2.0) declared both, so it restores the definition rather than the accident — but making it faithful would mean recording what was actually dropped. Happy to change it if you would prefer that.

Two housekeeping notes since you approved:

• I merged main (966513ef), as v1.0.0 shipped while this sat open and the changelog entry had landed inside what is now the released section; it is in v1.1.0 now.
• That merge left CI red, now fixed in 71dc1bad. Main moved black 24.8.0 → 26.3.1 in the meantime, and pre-commit only checks the conflicted files when it runs as part of a merge commit, so two migration files never got reformatted — CI's --all-files caught what my commit hook did not. Two blank lines, no code change. Checks are green again (3.10 still running as I write this).

[nhoening]

So after downgrading, everything works but I got actually two new indexes:

• "idx_tb_sensor_source_event_horizon" btree (sensor_id, source_id, event_start, belief_horizon)
• "timed_belief_sensor_id_idx" btree (sensor_id)

We have to decide if people need to be sheltered from this - users who deliberately used less or other indexes.

I see two options:

1. print a warning, both in upgrade and downgrade, with info on what is about to happen, letting the user opt out of it.
2. record the previous state in some way (maybe the create-index commands, in a text file, and attempt to restore from that file instead, when downgrading.

Let me know if there are any caveats or complexity issues with 2), because then I advise we go for 1) and move on.

[Flix6x]

Caveats with 2), yes — enough that I agree with going for 1). But looking into it turned up something that makes 1) smaller than expected, so it is worth a moment.

On 2). Capturing the state is the easy half: pg_get_indexdef() hands back replayable CREATE INDEX statements, partial predicates and all. Persisting it is where it falls over. A text file does not survive our own deployment story — we document Docker/compose, so db upgrade runs in one container and db downgrade in another, and the file is gone; and since one checkout often drives dev, staging and production, a single file can restore the wrong database's index set, which is a correctness hazard rather than an annoyance. A table would work, but it is permanent new machinery for a path used almost only in dev, it travels through pg_dump, and it needs consume-on-downgrade logic or a second up-down cycle records the already-polluted state and makes the surprise permanent. It also makes downgrade state-dependent instead of a deterministic inverse, which nothing else here is.

The thing worth changing anyway. Your two indexes are not the same case, and once separated, one of them needs no warning at all. Both upgrades already take real care to touch only what is present — matching by shape, checking the catalog, refusing to drop a same-named index that covers something else — and then both downgrades threw that away and created unconditionally. Making the downgrades as careful as the upgrades splits by whether this project declares the index:

• timed_belief_event_start_idx and timed_belief_sensor_id_idx are declared by the project: timely-beliefs declared index=True on both before 4.2.0, so a database on the version you are downgrading to would have had them. Restoring them is restoring schema the code declares, so that stays — but it now says so, and points out that a database carrying only one of them gains the other and can safely drop it again.
• idx_tb_sensor_source_event_horizon is declared nowhere in this project; no model or migration creates it. I manually added it to our production db back in 2025. Recreating it on downgrade handed the index to every database that downgrades, including the ones that never carried it. It is no longer recreated; the downgrade prints the CREATE INDEX statement instead, and the upgrade says up front that it will not come back.

Both migrations now report what they did to the database in front of them rather than what they can do in general, since the upgrade output is the only record you get of what was dropped.

Reproduced your case end to end to check, against a database built by running the chain rather than assembled by hand — starting from timed_belief_event_start_idx and neither of the other two: upgrade drops exactly one index and names it, downgrade recreates both single-column indexes with the explanation and does not create idx_tb_sensor_source_event_horizon, and the primary key is back to the old order. (That fresh database had a third primary key column order, different again from your dev one — a decent advert for pinning it.)

One deliberate trade to flag: a database that did carry idx_tb_sensor_source_event_horizon loses it across an up-down cycle. I think not creating schema the project does not define is the right side to err on, and both migrations print the statement, but it is a behaviour change — say if you would rather it came back, or want a -x flag to opt in.

Also added a .. warning:: to the v1.1.0 changelog covering which indexes the upgrade drops and what a downgrade puts back, and fixed a broken PR link the changelog had been carrying (separate commit).

[Flix6x]

Went a step further than my last comment, in a768a046 — I think the right answer to your question is to take the whole case off the table.

idx_tb_sensor_source_event_horizon is not part of FlexMeasures' schema: no model or migration creates it, it exists because someone added it by hand, and the only database we know of that carries it is one we administer. So the migration was probing every host's catalog for an index they do not have, and the changelog was explaining our own private schema choice to them. That is the wrong layer, and it is exactly the ground your question was on.

It is gone now: the constant, the catalog match, the conditional drop, the reporting, and the changelog paragraph about not recreating it. The b7e5a2c40f18 migration is untouched, because those two indexes are ours — timely-beliefs declared index=True on both before 4.2.0 — so dropping them, recreating them on downgrade, and saying so is squarely our business. That leaves your original concern in a much smaller place: the only way to gain an index you did not have is to downgrade a database that carried just one of event_start / sensor_id, and the migration now tells you when that happens.

Two claims went out with it that should have gone regardless. The docstring called dropping that index "typically the single largest win here" — true of one database, and overselling the migration to everyone else. It now states the general fact instead: the reordered key leads with (sensor_id, source_id, event_start, belief_horizon), so any index on a prefix of that is redundant afterwards and can be dropped, and FlexMeasures does not drop it for you because it cannot know which ones you meant to keep. The changelog warning says the same.

Verified up and down against a database built by running the chain, with the hand-added index present: the upgrade drops the two project-declared single-column indexes, names them, and leaves the hand-added one alone; after dropping that one by hand, the downgrade does not bring it back and the primary key returns to its old order.

So no need for either of your options for that index — no warning, and nothing to record. Option 1's warning stays for the event_start/sensor_id pair, where it is genuinely about our own schema.

[nhoening]

and FlexMeasures does not drop it for you because it cannot know which ones you meant to keep.

But we could print the suggestion to drop them in the migration output?

[Flix6x]

Good idea — done in 750f44c.

The migration now looks for indexes whose columns are a prefix of the new primary key and names them, without touching them. On a database that has none (i.e. any schema FlexMeasures created itself) it stays quiet; on one that has such an index it prints:

The reordered primary key on timed_belief has made 1 further index(es) redundant, which this
migration leaves in place because they are not part of FlexMeasures' schema:
public.idx_tb_sensor_source_event_horizon. Each covers a prefix of (sensor_id, source_id,
event_start, belief_horizon, cumulative_probability), so the primary key already serves the
queries they serve. To drop one without locking the table: DROP INDEX CONCURRENTLY <name>;

Two things I was careful about:

The suggestion has to be conservative, because a human is going to act on it. A naive prefix match would name indexes that are still pulling weight, so the query excludes partial and expression indexes (they answer queries the primary key does not), unique and constraint-backing indexes, non-btree indexes, indexes with INCLUDE columns, and anything with a non-default sort order or operator class — the primary key's plain ascending columns cannot substitute for a DESC index. I tested it against a database built by running the migration chain, with a hand-added prefix index plus three near misses (partial, DESC, columns in the wrong order): only the prefix index was named.

The primary key's column list is read from the catalog rather than repeated in the query, so this stays correct if we ever revise that order again.

It reports from b7e5a2c40f18, not d4a7c1e93b52, even though the reordering is what causes the redundancy. If it reported from the reordering migration it would also name timed_belief_sensor_id_idx, which the very next migration drops by itself — so the operator would be told to consider dropping something FlexMeasures is about to drop anyway. Reporting one migration later keeps the list to exactly the indexes we will not touch.

Nothing about the migration's behaviour varies by database — only its output does, which was the property that made the last round work out.

[nhoening]

I tried it locally and am happy :)