CRUD for automations in the API and UI - #2294
Open
Flix6x wants to merge 142 commits into
Open
Conversation
Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
/assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS
- Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
The scheduling job creators accept an optional trigger dict (stored as job meta data), like the forecasting pipeline already does. The API trigger endpoint records origin API; the CLI and automations follow in the next commit. The status page's 'Created Via' column picks this up automatically. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations now also support the 'schedules' type: - `flexmeasures add automation --type schedules` validates the parameters as a schedule trigger message (per the AssetTriggerSchema, as accepted by the API trigger endpoint, without the asset id). The schedule 'start' may be omitted, in which case each run schedules from the run time (floored to the message's resolution, if given) — a fixed start draws a warning. - The runner dispatches schedules automations to the same job creators as the API trigger endpoint (sequential or simultaneous), recording trigger meta data (origin automation) on the queued jobs; `flexmeasures add schedule --as-job` now records origin CLI. - Job stats for schedules automations are counted from the scheduling job cache (asset-level wrap-up jobs and per-sensor device jobs). - The UI automations page's Schedules tab is now enabled, with automations filtered by type per tab. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- New endpoints on assets: POST /automations (create, validating parameters by automation type), PATCH /automations/<id> (name, cron string, activation status) and DELETE /automations/<id>. Managing automations requires the same principals that may delete the asset (account admins and consultants). - The UI automations page gets a 'New automation' modal and per-row (de)activate and delete actions, shown to users with management rights. - Creation, update and deletion logic (incl. audit log records) moved into the automations service, shared by the CLI commands and the API endpoints. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Documentation build overview
154 files changed ·
|
…essage format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Merge current main, resolve the shared forecasting and documentation changes, regenerate the lockfile, and move the automation migration after the current migration head. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl>
…esolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - Only uv.lock conflicted. Both sides listed flexmeasures' own dependencies, main having added limits and this branch croniter and cron-descriptor. - The two lockfiles were also written by different uv versions, which normalise environment markers differently, so the sides disagreed on nearly every line rather than only on those three packages. Change: - Took main's uv.lock and regenerated it with uv 0.10.9, the version that wrote it (see #2451, which pins this and will later move everything to 0.12.7). That adds cron-descriptor and nothing else: 16 insertions, no deletions, and no package re-versioned. - Resolving it by hand was not viable: keying on the package name drops the Python 3.10 halves of version-split entries such as pint 0.24.4, and keying on the whole line keeps both marker spellings of every package. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl>
…edule-automations Context: - Picking up the review changes on #2290: the scheduling_cursor column became cursor, automation "occurrences" became "runs", and the automations documentation moved to its own page. Change: - Kept this branch's prepare_schedule_trigger_message and dropped its private _asset_and_ancestor_ids, which #2290 replaced with the shared asset_and_ancestor_ids in data/queries/generic_assets. - Kept the pointer that features/forecasting.rst now holds, including this branch's cross-reference to automating_schedules. - Merged the changelog entries: dropped the #2396 entry that #2290 folded into its own, and carried "computing forecasts or schedules" into the fuller CLI wording. - Regenerated the OpenAPI specs rather than merging the generated file by hand. Signed-off-by: F.N. Claessen <felix@seita.nl>
…ons-crud Context: - Picking up the review changes from #2290 and #2293, in which scheduling_cursor became cursor, "occurrences" became "runs", and the automations documentation moved to its own page. Change: - Updated this branch's update_automation service, which git merged without a conflict because the two sides touched different regions, but which still called get_initial_scheduling_cursor and assigned to automation.scheduling_cursor. Both would have failed at runtime. - Kept the edit logic in the service, so cli/data_edit.py neither imports get_initial_cursor nor rebases the cursor itself. - Kept both automation imports in cli/data_delete.py, one for deleting an automation and one for warning which automations use a sensor. - Said "run" instead of "occurrence" in the update_automation docstring and the automations UI template. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl>
…sts' into HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>
…into HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x
added a commit
that referenced
this pull request
Sep 1, 2026
* feat: add Automation data model Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: CLI commands to manage and run automations - `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on forecasting jobs how they were created Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: API endpoints to list an asset's automations GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: UI page listing an asset's automations /assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: cover automations CLI, API and UI Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: document automations Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: render cron descriptions in 24-hour format regardless of locale CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS * fix: address code review findings for automations - Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: address code review findings for automations (remaining files) Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: assert on the cron validation failure without pinning click's message format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * data/schemas: restrict automations to five-field cron Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/schemas/tests: cover automation cron field count Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/jobs: retain automation guard after queueing failure Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover partial automation queue failure Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: normalize YAML forecasting option files Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover automation YAML option files Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/services: redact inaccessible automation provenance Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover automation provenance authorization Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: distinguish automation load failures Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation load error state Check that the automations page renders the warning target and hides the table when loading fails. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/docs: preserve standalone asterisks in RST conversion Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/tests: cover RST cron wildcard conversion Verify cron wildcards remain unchanged while ordinary italic markup is still converted. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/forecasting: clarify automation execution contract Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation API and runner contract Record the automation endpoints, authorization-aware provenance, and five-field runner behavior. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/docs: show job creation provenance Include the created_via field in the asset jobs OpenAPI example. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: keep forecast CLI stub compatible with job provenance Add the trigger method required by the forecasting CLI to the regressor parsing test double. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: require valid automation generators Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover automation generator retention Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: constrain forecast automation outputs Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover forecast automation output scope Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain forecast automation ownership rules Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: merge automation and main migration heads Join the automation and main Alembic branches so installations have a single database upgrade target. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: let data generators report their input and output sensors Context: - Review of #2290 asked for a data generator property listing the sensors it reads from and writes to, so that automations can link to those sensors (and, later, check the creating user's permissions on them) Change: - Added input_sensors and output_sensors to DataGenerator (empty by default), implemented for Forecaster from its regressors and target sensor - Added the same properties to Automation, resolved from its data generator configured with the automation's own parameters - Added get_automations_feeding_sensor to look up automations by output sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models/forecasting: only announce a pipeline run when actually running it Context: - 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline' for every automation, while it only queues the cycles as jobs Change: - Log that line at debug level when running with as_job, where the workers running the cycles log their own start Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: default the automation recurrence to daily, and reject options that --source already determines Context: - Review of #2290: --cron should not be required, and --forecaster/--config are redundant with --source, whose data generator attributes already hold both (get_data_generator silently ignores them when a source is given) Change: - Default --cron to '0 0 * * *' (daily at midnight) - Abort when --source is combined with --forecaster, --config or any of the forecaster configuration options, naming the conflicting options Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: report an automation's input and output sensors Context: - The automation details modal should link to the sensors an automation feeds Change: - Added input_sensors and output_sensors (id and name each) to GET /assets/<id>/automations/<automation_id> Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: add an endpoint for one data source Context: - The sensor page should be able to show the full record of a data source, including the attributes in which data generators store their configuration Change: - Added GET /sources/<id>, with the same access rules as listing sources - Let _serialize_source optionally include the attributes and unset fields Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs Context: - The specs are generated from the endpoint docstrings by a pre-commit hook Change: - Regenerated after adding the data source endpoint and the automation's input and output sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: link an automation's details to its sensors, and make the listing sortable Context: - Review of #2290: the details modal should link to the sensors an automation feeds, with its data source pre-selected there, and the listing was not sortable Change: - Show the input and output sensors in the details modal, linking to /sensors/<id>?source=<generator id> - Enabled ordering, sorting the rendered columns on separate values (the ISO timestamp, the activation status and the cron string), newest first Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: show a sensor's data source record and the automations feeding it Context: - Review of #2290: the sensor page should be able to show all details of a data source, and list the automations that write data to the sensor Change: - Added an info button next to the source selector, opening a modal with the full data source record - Pre-select the source given in the source query parameter, so that links from an automation land on its own source - List the automations feeding the sensor (those the user may read), linking to the automations page of their asset - Added user_can_read to the UI's permission helpers Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the automation and data source review follow-ups Context: - New behaviour from the #2290 review needs regression coverage Change: - CLI: the daily default recurrence, the --source conflict, and an automation's input and output sensors - API: an automation without a data generator reports no sensors; the new data source endpoint, its access rules and its 404 - UI: the source query parameter reaches the page, and automations feeding a sensor are listed on it Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: describe the automation and data source follow-ups Context: - The #2290 review changed user-facing CLI, API and UI behaviour Change: - Documented the daily default recurrence and reusing a forecaster via --source - Documented the links between automations and the sensors they feed - Added API change log entries for the automations and data source endpoints - Extended the changelog entry of #2290 Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs after merging Context: - The merge combined endpoint docstring changes from both sides Change: - Regenerated the specs Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: only reject configuration options that were actually given with --source Context: - The guard added on this branch compared against the assembled config, which always holds the schema defaults of the list-valued options, so any use of --source was rejected Change: - Detect the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort - Name the conflicting options in the error message Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: only consider automations that could feed a sensor Context: - Listing the automations feeding a sensor sets up a data generator per candidate, which does not need to happen for every automation in the database Change: - Narrowed the candidates to automations on the sensor's asset or an ancestor, which is where an automation writing to it must live Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the merged automation behaviour Context: - Automations now always have a data generator, and the --source guard also covers the forecaster configuration options Change: - Assert the sensors an automation with a generator reads from and writes to - Cover a configuration option conflicting with --source Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: keep mypy happy about click 8 attributes types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself. Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0, so mypy rejected the --source conflict detection and pre-commit failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * cli: keep the automation help focused on the automation `add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help, burying the options that describe the automation itself. Accept those options still, but hide them, and let the parameters file supply a field the schema requires, so --sensor no longer has to be repeated on the command line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/models: count a source-filtered regressor as an input sensor SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor, which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors. The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved, so the wrapped sensor counts as an input just like a plain sensor ID does. This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks for automations administered through the API, where a missing input sensor means a missing permission check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/services: do not report no sensors when an automation's sensors are unknown Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all. That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks for automations administered through the API, where no sensors reads as nothing to check, so a broken automation would pass every check on the sensors it involves. Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown, while get_automation_sensors keeps reporting none for display. The broad exception handler is narrowed to the failures that can actually occur here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Feat automation timezones catchup (#2396) * feat: add timezone-aware automation catch-up Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover timezone-aware automation scheduling Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain automation timezone and catch-up semantics Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation timezone and catch-up support Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: link automation catch-up changelog to PR Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * fix(data/schemas): reject cron expressions without dates Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): isolate invalid recurrences and stale claims Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/v3_0): protect automation sensor details Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(cli): cover impossible recurrence input Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover resilient automation claims Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): cover private automation dependencies Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: name an automation's cursor after what it points at Context: - Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all. - "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with. - The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden. Change: - Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column. - Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success. - Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key. - Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the automation cursor rename Context: - Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs". Change: - Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading. - Renamed the coalescing and spring-forward test cases to speak of runs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain the automation cursor, and say "run" instead of "occurrence" Context: - Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all. - The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing. Change: - Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got. - Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it. - Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations. - Linked issue #2393 where the docs referred to "durable run records". - Applied the suggested wording for the "add automation" command summary. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: give the automation API changes their own version section Context: - Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry. Change: - Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section. - Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at. - Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises. - Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: restore the v3.0-32 underline to full length Context: - This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short. Change: - Set the underline to exactly the title length. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: address review findings on the automations service Context: - Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees. Change: - run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently. - Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one. - Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models: index the automation asset foreign key Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: work out an automation's sensors once when they cannot be resolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: warn which automations a sensor deletion would break Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the sensor deletion warning, and stop depending on caplog Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: record the sensor deletion warning Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: give automations their own page Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/migrations: index the automation asset FK without adding a revision Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>
Context: - #2290 was squash-merged, so main now carries its 78 commits as one. This branch already contains those commits individually, but git cannot see that across the squash, and a normal merge conflicted in 15 files. - Checked before resolving: main's tree is byte-identical to #2290's final commit (06bf14b), that commit is an ancestor of this branch, and so is main's pre-squash tip. So this branch already contains everything main has, and main brought nothing of its own. Change: - Recorded the merge while keeping this branch's tree, rather than replaying 40 hunks of a conflict that only exists because the parent was squashed. - Verified afterwards that no file has content on main which is missing here; the differences are this branch's own work, such as generalising the job cache lookup beyond forecasts. Signed-off-by: F.N. Claessen <felix@seita.nl>
…into HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>
Resolve the test_run_automations conflict by keeping main's freeze_server_now fixture, and rejoin the migration branches with a merge revision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>
The scheduling chapter now points to the automations chapter, the way the forecasting chapter already does, and the details of what a schedule automation stores live next to the rest of the automations documentation. The changelog entry is merged with the one for forecasting automations, into a single entry on automations. That entry had stayed in the v1.0.0 section although PR #2290 was merged after v1.0.0 was tagged, so the merged entry moves to v1.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>
Queue names and job types call these tasks "forecasting" and "scheduling", so the automation types now do too: 'forecasts' becomes 'forecasting' and 'schedules' becomes 'scheduling'. An automation's type is now the name of the queue its jobs go to, so the runner no longer maps one to the other. Automations were merged after v1.0.0 was tagged, so no released CLI or API surface changes. A migration renames the values of existing rows, in both directions, and recreates the check constraint that requires a data generator for forecast automations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>
Like the automations entry it accompanied, it was written while v1.0.0 was the open section, but PR #2290 was merged on 2026-09-01, after the v1.0.0 tag of 2026-08-26. Neither the v1.0.0 nor the v1.0.0rc5 tag contains it, and it is the last entry in that section that postdates the tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>
…2464) * Make the Scheduler a DataGenerator, and give every automation a generator A scheduler's data source recorded only its class, version and author, so one source described every schedule that scheduler ever made, whatever it computed. It now also records the flex config the scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a schedule can be traced back to the configuration that produced it. `Scheduler` therefore subclasses `DataGenerator`, with a config of the asset and its serialized flex-model and flex-context. Timing stays out of it: start, end and resolution differ from run to run, which is what `DataGenerator._clean_parameters` already says about parameters. The config is snapshotted while still serialized, because a deserialized flex config holds sensors, quantities and time series which do not survive a round trip. `resolve_flex_config` returns the config as passed, and `StorageScheduler` overrides it to merge in what the asset tree stores, so a scheduler which does not read the asset tree keeps describing exactly what it was given. One scheduling request stays one data source: `create_sequential_scheduling_job` resolves the request's source once and hands it to each device job, so a schedule can still be retrieved per device from the request's job, rather than each device job resolving a source from its own slice of the flex-model. A schedule automation now points at such a source, so `generator_id` is required for every automation and the constraint requiring it only for forecasts is gone. That generator is derived rather than chosen: the scheduler follows from the asset and the config from the asset tree, so the runner resolves it again on every run and moves the automation when either has changed. For the same reason, a schedule automation's flex config may only describe the site and its devices: a field fixing a moment, such as `soc-at-start` or a `soc-targets` entry with a `datetime`, is refused when the automation is created, naming the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Address review of the scheduler data generator Keep values that describe one moment out of a scheduler's data source. A `soc-at-start`, or a `soc-targets` entry at a given datetime, differs on every trigger, so recording it would have made every schedule the work of a brand new data source. Only what the site and its devices can do now tells one scheduler source from another. The rule that already refused such fields on a schedule automation is the same one, so both now live next to the config schema they are about. That schema moves back in with the other scheduling schemas, into a submodule of its own. The circular import which had pushed it out is fixed at its root instead: three scheduling schema modules imported `Sensor` and `Asset` from the `flexmeasures` package root, which is still initialising while they load, so they now import from the modules that define them. The migration relaxing the generator constraint and the one restoring it cancel out, and both were unreleased, so they are gone, along with the merge revision that only existed to rejoin the relaxing one. `generator_id` keeps the NOT NULL it was created with, the type rename no longer has a constraint to recreate, and the automation feature arrives in one migration rather than three. Also, fail with the source's id when a job is told to record under a data source that no longer exists, rather than an AttributeError one frame later, and reflow the comments and docstrings which broke mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: say what a second schedule source means for reading a sensor The warning read as if summing two schedules of one period were a thing anyone would want to do, which made it sound like a defect rather than a change in provenance. It now says what actually changes: the newer schedule used to supersede the older one, and now both are kept, so a chart draws both and the asset's KPIs total both, because they deliberately report what the chart draws. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Keep a scheduler's data source identity stable Two ways the recorded config could differ while describing the same configuration, both from the Copilot review. A caller which deserialized the flex config first hands over sensors and assets, which were recorded by how they print. A sensor prints as its name, so renaming one described a different configuration, and two sensors sharing a name described the same one. They are now recorded by their id, which is what the serialized flex config names them by. A single moment may be written as one mapping or as a list of them, and stripping the momentary values left a null behind in the first case and an empty list in the second, so the same configuration looked like three different ones depending on how it was written. A field that stripping empties is now left out altogether, which is what leaving it out of the trigger message does too. A field that arrived empty stays as it was given, and a list with static entries beside momentary ones keeps them. The snapshot no longer deep-copies. It goes straight through the JSON-safe encoder, which yields plain structures, where the copy used to carry sensors along and be read after a commit had expired them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Send the request's configuration to the device jobs, not a data source id FlexMeasures does not auto-commit the session of the request that enqueues a scheduling job (see `flexmeasures.data.transactional`), and the schedule trigger endpoint does not commit either. A data source resolved while enqueueing therefore lives in a transaction the workers never see, so handing its id to the device jobs of a sequential schedule would have failed to find it. The jobs now carry the request's configuration instead, and each worker resolves the data source from it and commits, as `make_schedule` already does for everything else it writes. The device jobs of one request still describe one configuration, so they still share one source. A test pins that enqueueing writes no data source of its own, and that the device jobs carry the same configuration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: clear Redis after the test that only queues jobs The new sequential test never runs what it queues, so it left the jobs, the queue's deferred registry and the job cache behind. The job cache was the one that bit: a scheduling job's id is derived from what it schedules, so the next test's identical request was skipped as already made, and it saw no jobs queued. Also reflow a docstring line that ended mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs: reflow two lines that broke mid-phrase Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: fetch each queued job once Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> --------- Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Three conflicts, all from main's automation work landing beside this branch's. The changelog entries were added to the same list on both sides, so both sets are kept. `add automation` gained a dry-run guard on main, in the place this branch now splits by automation type. The guard goes before the split: `--dry-run` reaches the command whatever the automation computes, and popping it is what keeps it out of a schedule trigger message, which would otherwise reject it as an unknown field. Its message covers both types now, rather than naming forecasts. A test pins that, for each type; it was written because moving the guard into the forecasting branch first looked right and broke schedule automations. The automations page gained a Run now button on main, in the block this branch rewrote to list one table per automation type. Main's handler is kept, without its copy of the DataTables error-mode line, which this branch already sets higher up. Finally, the type rename and main's table cleanup branched from the same revision, so a merge revision rejoins them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WS6V8nZyRzMxsvqGnNpUTk Signed-off-by: F.N. Claessen <claessen@seita.nl>
The base branch renamed the automation types from 'forecasts' and 'schedules' to 'forecasting' and 'scheduling', matching the queue and job names used everywhere else. This brings the rest of this branch in line, ahead of merging, so that the merge itself stays about the CRUD endpoints. Two places needed more than a renamed literal: - The CLI singularised the type by chopping its last character, which turned 'forecasting' into 'forecastin'. The types no longer pluralise the result they compute, so the noun is now looked up in Automation.RESULT_NOUNS. - The Automations page builds its table selectors from the type name, so the markup ids had to move with it, or both tables would have silently stayed empty. test_asset_crud covers those ids, and was updated to match. The type-to-queue mapping in `flexmeasures jobs run-automations` became an identity map once the names lined up, so it is gone.
…ons-crud The base branch had moved 29 commits ahead, mostly through #2464 (schedulers as data generators) and #2460 (running one automation on demand). Endpoints, tests and page handlers from both sides simply coexist: this branch's POST/PATCH/DELETE on /assets/(id)/automations next to the base's POST .../automations/(automation_id)/trigger. Two resolutions carry a decision: - create_automation() stays the one place an automation is created, so the logic the base had grown in the CLI moved into it: a scheduling automation now resolves its scheduler's data source there, like a forecast automation resolves its forecaster's, and after the permission check for the same reason (a refused request should leave nothing behind). _check_schedule_automation_parameters is gone; a flex config that pins a moment now raises RecurringScheduleFixesAMoment, which the CLI still reports as a usage error. - Every automation has a generator again, so this branch's nullable generator_id and its 'forecast_generator' check constraint are obsolete. The base had already deleted that migration in #2464, and its deletion is taken here. Keeping it would have been worse than redundant: the constraint tested type != 'forecasts', a value the rename no longer stores, so it would have silently stopped enforcing.
Defining an automation and running it are the same act seen from two sides: both exist to write data under the asset. They were gated differently, though — creating, editing and deleting needed the principals that may delete the asset, while triggering a run needed create-children — so a user could run an automation but not fix the one that kept failing. Both now follow create-children, which widens managing automations from organisation admins and consultants to every user in the organisation. The Automation ACL moves with the endpoints, so the model and the API agree. What an automation may touch is unchanged, and is the check that actually contains this: its creator still needs read access to the sensors it reads and permission to record data on the sensors it writes to.
The main changelog folds the CRUD endpoints into the consolidated automations entry, the way #2293 was folded in, rather than restoring the four separate entries git offered — those sat in v1.0.0, which has since been released. The API change log entry had been filed under v3.0-32 (August 11), also long released, and moves to a v3.0-35 section of its own.
The trigger tests arrived from the base branch asking for add_battery_assets and db, which are module-scoped. This file's add_automations had meanwhile become function-scoped on fresh_db, and a test holding both hangs rather than fails: fresh_db drops the tables that the module-scoped session still has open, and Postgres waits on the lock. They now take add_battery_assets_fresh_db and fresh_db, like the rest of the file.
BelhsanHmida
added a commit
that referenced
this pull request
Sep 8, 2026
…s-ui-polish Follows the automation type rename that came up the stack, in `_job_cache_refs`, `get_asset_automations_job_stats` and the Automations page. Kept this branch's refactor of the job-stats lookup, which already covers reporting, over the inline version it was extracted from. Kept the *Run now* button, which arrived from PR #2460 by way of the base, alongside this branch's batched job counts: the counts now come with the listing, so the cell is filled in directly rather than fetched per row. Dropped the changelog entries this branch carried for PRs #2290, #2293, #2294 and #2297, which the base states in consolidated form, including the API entry for the automation CRUD endpoints, whose permission wording PR #2294 has since settled. This branch's own entry no longer claims the UI edit action: PR #2294 has since shipped it, so what is left here is the batched job counts and loading an automation's details only when they are opened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Flix6x
added a commit
that referenced
this pull request
Sep 9, 2026
* feat: add Automation data model Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: CLI commands to manage and run automations - `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on forecasting jobs how they were created Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: API endpoints to list an asset's automations GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: UI page listing an asset's automations /assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: cover automations CLI, API and UI Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: document automations Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: render cron descriptions in 24-hour format regardless of locale CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS * fix: address code review findings for automations - Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: address code review findings for automations (remaining files) Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on scheduling jobs how they were created The scheduling job creators accept an optional trigger dict (stored as job meta data), like the forecasting pipeline already does. The API trigger endpoint records origin API; the CLI and automations follow in the next commit. The status page's 'Created Via' column picks this up automatically. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: schedules as automations Automations now also support the 'schedules' type: - `flexmeasures add automation --type schedules` validates the parameters as a schedule trigger message (per the AssetTriggerSchema, as accepted by the API trigger endpoint, without the asset id). The schedule 'start' may be omitted, in which case each run schedules from the run time (floored to the message's resolution, if given) — a fixed start draws a warning. - The runner dispatches schedules automations to the same job creators as the API trigger endpoint (sequential or simultaneous), recording trigger meta data (origin automation) on the queued jobs; `flexmeasures add schedule --as-job` now records origin CLI. - Job stats for schedules automations are counted from the scheduling job cache (asset-level wrap-up jobs and per-sensor device jobs). - The UI automations page's Schedules tab is now enabled, with automations filtered by type per tab. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for schedule automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: assert on the cron validation failure without pinning click's message format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * data/schemas: restrict automations to five-field cron Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/schemas/tests: cover automation cron field count Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/jobs: retain automation guard after queueing failure Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover partial automation queue failure Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: normalize YAML forecasting option files Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover automation YAML option files Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/services: redact inaccessible automation provenance Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover automation provenance authorization Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: distinguish automation load failures Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation load error state Check that the automations page renders the warning target and hides the table when loading fails. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/docs: preserve standalone asterisks in RST conversion Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/tests: cover RST cron wildcard conversion Verify cron wildcards remain unchanged while ordinary italic markup is still converted. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/forecasting: clarify automation execution contract Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation API and runner contract Record the automation endpoints, authorization-aware provenance, and five-field runner behavior. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/docs: show job creation provenance Include the created_via field in the asset jobs OpenAPI example. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: keep forecast CLI stub compatible with job provenance Add the trigger method required by the forecasting CLI to the regressor parsing test double. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: require valid automation generators Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover automation generator retention Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: constrain forecast automation outputs Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover forecast automation output scope Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain forecast automation ownership rules Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: merge automation and main migration heads Join the automation and main Alembic branches so installations have a single database upgrade target. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: allow schedule automations without generators Context: - Schedule automations do not use a data generator, but the reviewed forecast automation schema required one. Change: - Make the foreign key nullable while retaining a database check that forecasts always have a generator. - Add a forward migration without weakening data-source deletion semantics. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover schedule automation validation Context: - Review uncovered untested forecast-only options, invalid durations, and DST start calculation. Change: - Add CLI and trigger-preparation regressions while retaining forecast sensor validation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/tests: cover schedule automation dispatch Context: - Persistence, inherited flex configuration, descendant statistics, and job counts lacked realistic coverage. Change: - Move automation service tests under the fresh-database fixture and add end-to-end schedule regressions. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover schedule job provenance Context: - API provenance was not asserted across asset and sensor schedule jobs. Change: - Verify API trigger metadata on sequential descendants, wrap-up jobs, and sensor jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation type tabs Context: - The asset automation page tests still targeted the former single table. Change: - Assert type-specific tables, error rendering, filters, and hidden-tab column adjustment. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * scheduling: harden automation dispatch Context: - Minimal asset schedules lost stored defaults, invalid timing could execute, provenance was incomplete, and descendant jobs were miscounted. Change: - Validate and floor fixed durations safely, inherit stored flex configuration, preserve provenance, and count the jobs actually dispatched. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: reject forecast options for schedule automations Context: - Schedule automation creation silently accepted forecaster settings that could never affect scheduling. Change: - Detect supplied forecast-only options and return a user-facing usage error. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: resize automation tables on tab changes Context: - DataTables initialized in the hidden schedule tab could render with stale column widths. Change: - Add tab accessibility state and adjust initialized table columns when a tab becomes visible. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: clarify schedule automation inputs Context: - The feature guide linked only to the API root and omitted fixed-start and duration constraints. Change: - Document canonical fields, runtime start behavior, timing validation, and generator-free schedule automations. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover malformed automation YAML Context: - Manual testing exposed raw parser exceptions for malformed automation files. Change: - Require a user-facing usage error for invalid YAML in both config and parameter files. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: report malformed automation YAML Context: - PyYAML parser errors escaped automation creation without a useful CLI message. Change: - Translate malformed config and parameter files into a normal Click usage error. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/tests: cover stored schedule flex configuration Context: - Manual execution showed minimal automations failing for a one-device asset tree. Change: - Exercise both simultaneous and sequential dispatch using realistic flex configuration stored on the child asset. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * scheduling: load stored flex config for minimal triggers Context: - A one-device asset tree collapsed to sensor scheduling without a sensor, and sequential dispatch could not resolve its stored output. Change: - Preserve asset-triggered flex models as a list and resolve sequential device sensors from stored consumption or production outputs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/scheduling: describe trigger propagation Context: - Scheduling service docstrings did not distinguish single-job and sequential provenance behavior. Change: - Document where trigger metadata is stored and correct the simultaneous return description. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: let data generators report their input and output sensors Context: - Review of #2290 asked for a data generator property listing the sensors it reads from and writes to, so that automations can link to those sensors (and, later, check the creating user's permissions on them) Change: - Added input_sensors and output_sensors to DataGenerator (empty by default), implemented for Forecaster from its regressors and target sensor - Added the same properties to Automation, resolved from its data generator configured with the automation's own parameters - Added get_automations_feeding_sensor to look up automations by output sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models/forecasting: only announce a pipeline run when actually running it Context: - 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline' for every automation, while it only queues the cycles as jobs Change: - Log that line at debug level when running with as_job, where the workers running the cycles log their own start Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: default the automation recurrence to daily, and reject options that --source already determines Context: - Review of #2290: --cron should not be required, and --forecaster/--config are redundant with --source, whose data generator attributes already hold both (get_data_generator silently ignores them when a source is given) Change: - Default --cron to '0 0 * * *' (daily at midnight) - Abort when --source is combined with --forecaster, --config or any of the forecaster configuration options, naming the conflicting options Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: report an automation's input and output sensors Context: - The automation details modal should link to the sensors an automation feeds Change: - Added input_sensors and output_sensors (id and name each) to GET /assets/<id>/automations/<automation_id> Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: add an endpoint for one data source Context: - The sensor page should be able to show the full record of a data source, including the attributes in which data generators store their configuration Change: - Added GET /sources/<id>, with the same access rules as listing sources - Let _serialize_source optionally include the attributes and unset fields Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs Context: - The specs are generated from the endpoint docstrings by a pre-commit hook Change: - Regenerated after adding the data source endpoint and the automation's input and output sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: link an automation's details to its sensors, and make the listing sortable Context: - Review of #2290: the details modal should link to the sensors an automation feeds, with its data source pre-selected there, and the listing was not sortable Change: - Show the input and output sensors in the details modal, linking to /sensors/<id>?source=<generator id> - Enabled ordering, sorting the rendered columns on separate values (the ISO timestamp, the activation status and the cron string), newest first Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: show a sensor's data source record and the automations feeding it Context: - Review of #2290: the sensor page should be able to show all details of a data source, and list the automations that write data to the sensor Change: - Added an info button next to the source selector, opening a modal with the full data source record - Pre-select the source given in the source query parameter, so that links from an automation land on its own source - List the automations feeding the sensor (those the user may read), linking to the automations page of their asset - Added user_can_read to the UI's permission helpers Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the automation and data source review follow-ups Context: - New behaviour from the #2290 review needs regression coverage Change: - CLI: the daily default recurrence, the --source conflict, and an automation's input and output sensors - API: an automation without a data generator reports no sensors; the new data source endpoint, its access rules and its 404 - UI: the source query parameter reaches the page, and automations feeding a sensor are listed on it Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: describe the automation and data source follow-ups Context: - The #2290 review changed user-facing CLI, API and UI behaviour Change: - Documented the daily default recurrence and reusing a forecaster via --source - Documented the links between automations and the sensors they feed - Added API change log entries for the automations and data source endpoints - Extended the changelog entry of #2290 Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs after merging Context: - The merge combined endpoint docstring changes from both sides Change: - Regenerated the specs Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: only reject configuration options that were actually given with --source Context: - The guard added on this branch compared against the assembled config, which always holds the schema defaults of the list-valued options, so any use of --source was rejected Change: - Detect the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort - Name the conflicting options in the error message Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: only consider automations that could feed a sensor Context: - Listing the automations feeding a sensor sets up a data generator per candidate, which does not need to happen for every automation in the database Change: - Narrowed the candidates to automations on the sensor's asset or an ancestor, which is where an automation writing to it must live Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the merged automation behaviour Context: - Automations now always have a data generator, and the --source guard also covers the forecaster configuration options Change: - Assert the sensors an automation with a generator reads from and writes to - Cover a configuration option conflicting with --source Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: keep mypy happy about click 8 attributes types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself. Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0, so mypy rejected the --source conflict detection and pre-commit failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * cli: keep the automation help focused on the automation `add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help, burying the options that describe the automation itself. Accept those options still, but hide them, and let the parameters file supply a field the schema requires, so --sensor no longer has to be repeated on the command line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/models: count a source-filtered regressor as an input sensor SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor, which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors. The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved, so the wrapped sensor counts as an input just like a plain sensor ID does. This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks for automations administered through the API, where a missing input sensor means a missing permission check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/services: do not report no sensors when an automation's sensors are unknown Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all. That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks for automations administered through the API, where no sensors reads as nothing to check, so a broken automation would pass every check on the sensors it involves. Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown, while get_automation_sensors keeps reporting none for display. The broad exception handler is narrowed to the failures that can actually occur here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Feat automation timezones catchup (#2396) * feat: add timezone-aware automation catch-up Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover timezone-aware automation scheduling Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain automation timezone and catch-up semantics Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation timezone and catch-up support Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: link automation catch-up changelog to PR Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * data/migrations: rejoin the two automation migration branches Merging the timezone and catch-up work alongside the schedule automations left two alembic heads, one adding an automation's timezone and scheduling cursor and one allowing a schedule automation without a data generator, so flexmeasures db upgrade refused to run and the Docker image build failed. The two touch different columns, so the merge point has nothing of its own to do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/schemas): reject cron expressions without dates Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): isolate invalid recurrences and stale claims Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/v3_0): protect automation sensor details Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(cli): cover impossible recurrence input Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover resilient automation claims Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): cover private automation dependencies Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): resolve schedule automation sensors Prepare the scheduling configuration through the same scheduler collection path used before queueing, then derive declared input and output sensors so generator-free schedule automations expose their stored flex dependencies and participate in sensor relationships. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover schedule sensor resolution Exercise a minimal schedule that inherits its flex model and context from the asset tree, confirming that price inputs, schedule outputs, and the sensor-to-automation relationship are all reported from stored configuration. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): expose schedule dependency details Create a generator-free schedule with stored price and output sensors and assert that the details endpoint returns both dependency sets instead of reporting empty arrays. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: refuse forecaster options that were given, not merely ones that differ from their default The guard against combining forecaster options with --type schedules compared the forecaster against its default, so naming the default forecaster explicitly passed silently and the automation was created, leaving the impression that the option had applied to a schedule automation. Ask which options were actually given on the command line instead, the way the --source conflict check already does, and name the offending options in the error rather than listing every option it could have been. Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com * data/models: name an automation's cursor after what it points at Context: - Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all. - "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with. - The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden. Change: - Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column. - Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success. - Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key. - Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the automation cursor rename Context: - Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs". Change: - Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading. - Renamed the coalescing and spring-forward test cases to speak of runs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain the automation cursor, and say "run" instead of "occurrence" Context: - Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all. - The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing. Change: - Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got. - Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it. - Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations. - Linked issue #2393 where the docs referred to "durable run records". - Applied the suggested wording for the "add automation" command summary. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: give the automation API changes their own version section Context: - Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry. Change: - Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section. - Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at. - Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises. - Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: restore the v3.0-32 underline to full length Context: - This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short. Change: - Set the underline to exactly the title length. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: address review findings on the automations service Context: - Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees. Change: - run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently. - Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one. - Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models: index the automation asset foreign key Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: work out an automation's sensors once when they cannot be resolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: warn which automations a sensor deletion would break Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the sensor deletion warning, and stop depending on caplog Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: record the sensor deletion warning Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: give automations their own page Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/migrations: index the automation asset FK without adding a revision Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: document schedule automations in the automations chapter The scheduling chapter now points to the automations chapter, the way the forecasting chapter already does, and the details of what a schedule automation stores live next to the rest of the automations documentation. The changelog entry is merged with the one for forecasting automations, into a single entry on automations. That entry had stayed in the v1.0.0 section although PR #2290 was merged after v1.0.0 was tagged, so the merged entry moves to v1.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Name automation types after the task, like the rest of the codebase Queue names and job types call these tasks "forecasting" and "scheduling", so the automation types now do too: 'forecasts' becomes 'forecasting' and 'schedules' becomes 'scheduling'. An automation's type is now the name of the queue its jobs go to, so the runner no longer maps one to the other. Automations were merged after v1.0.0 was tagged, so no released CLI or API surface changes. A migration renames the values of existing rows, in both directions, and recreates the check constraint that requires a data generator for forecast automations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: move the data source inspection entry to v1.1.0 Like the automations entry it accompanied, it was written while v1.0.0 was the open section, but PR #2290 was merged on 2026-09-01, after the v1.0.0 tag of 2026-08-26. Neither the v1.0.0 nor the v1.0.0rc5 tag contains it, and it is the last entry in that section that postdates the tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Schedulers as data generators, and a generator on every automation (#2464) * Make the Scheduler a DataGenerator, and give every automation a generator A scheduler's data source recorded only its class, version and author, so one source described every schedule that scheduler ever made, whatever it computed. It now also records the flex config the scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a schedule can be traced back to the configuration that produced it. `Scheduler` therefore subclasses `DataGenerator`, with a config of the asset and its serialized flex-model and flex-context. Timing stays out of it: start, end and resolution differ from run to run, which is what `DataGenerator._clean_parameters` already says about parameters. The config is snapshotted while still serialized, because a deserialized flex config holds sensors, quantities and time series which do not survive a round trip. `resolve_flex_config` returns the config as passed, and `StorageScheduler` overrides it to merge in what the asset tree stores, so a scheduler which does not read the asset tree keeps describing exactly what it was given. One scheduling request stays one data source: `create_sequential_scheduling_job` resolves the request's source once and hands it to each device job, so a schedule can still be retrieved per device from the request's job, rather than each device job resolving a source from its own slice of the flex-model. A schedule automation now points at such a source, so `generator_id` is required for every automation and the constraint requiring it only for forecasts is gone. That generator is derived rather than chosen: the scheduler follows from the asset and the config from the asset tree, so the runner resolves it again on every run and moves the automation when either has changed. For the same reason, a schedule automation's flex config may only describe the site and its devices: a field fixing a moment, such as `soc-at-start` or a `soc-targets` entry with a `datetime`, is refused when the automation is created, naming the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Address review of the scheduler data generator Keep values that describe one moment out of a scheduler's data source. A `soc-at-start`, or a `soc-targets` entry at a given datetime, differs on every trigger, so recording it would have made every schedule the work of a brand new data source. Only what the site and its devices can do now tells one scheduler source from another. The rule that already refused such fields on a schedule automation is the same one, so both now live next to the config schema they are about. That schema moves back in with the other scheduling schemas, into a submodule of its own. The circular import which had pushed it out is fixed at its root instead: three scheduling schema modules imported `Sensor` and `Asset` from the `flexmeasures` package root, which is still initialising while they load, so they now import from the modules that define them. The migration relaxing the generator constraint and the one restoring it cancel out, and both were unreleased, so they are gone, along with the merge revision that only existed to rejoin the relaxing one. `generator_id` keeps the NOT NULL it was created with, the type rename no longer has a constraint to recreate, and the automation feature arrives in one migration rather than three. Also, fail with the source's id when a job is told to record under a data source that no longer exists, rather than an AttributeError one frame later, and reflow the comments and docstrings which broke mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: say what a second schedule source means for reading a sensor The warning read as if summing two schedules of one period were a thing anyone would want to do, which made it sound like a defect rather than a change in provenance. It now says what actually changes: the newer schedule used to supersede the older one, and now both are kept, so a chart draws both and the asset's KPIs total both, because they deliberately report what the chart draws. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Keep a scheduler's data source identity stable Two ways the recorded config could differ while describing the same configuration, both from the Copilot review. A caller which deserialized the flex config first hands over sensors and assets, which were recorded by how they print. A sensor prints as its name, so renaming one described a different configuration, and two sensors sharing a name described the same one. They are now recorded by their id, which is what the serialized flex config names them by. A single moment may be written as one mapping or as a list of them, and stripping the momentary values left a null behind in the first case and an empty list in the second, so the same configuration looked like three different ones depending on how it was written. A field that stripping empties is now left out altogether, which is what leaving it out of the trigger message does too. A field that arrived empty stays as it was given, and a list with static entries beside momentary ones keeps them. The snapshot no longer deep-copies. It goes straight through the JSON-safe encoder, which yields plain structures, where the copy used to carry sensors along and be read after a commit had expired them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Send the request's configuration to the device jobs, not a data source id FlexMeasures does not auto-commit the session of the request that enqueues a scheduling job (see `flexmeasures.data.transactional`), and the schedule trigger endpoint does not commit either. A data source resolved while enqueueing therefore lives in a transaction the workers never see, so handing its id to the device jobs of a sequential schedule would have failed to find it. The jobs now carry the request's configuration instead, and each worker resolves the data source from it and commits, as `make_schedule` already does for everything else it writes. The device jobs of one request still describe one configuration, so they still share one source. A test pins that enqueueing writes no data source of its own, and that the device jobs carry the same configuration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: clear Redis after the test that only queues jobs The new sequential test never runs what it queues, so it left the jobs, the queue's deferred registry and the job cache behind. The job cache was the one that bit: a scheduling job's id is derived from what it schedules, so the next test's identical request was skipped as already made, and it saw no jobs queued. Also reflow a docstring line that ended mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs: reflow two lines that broke mid-phrase Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: fetch each queued job once Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> --------- Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Automations can now be created, updated and deleted through the API and the UI, not only from the
CLI.
API.
[POST] /assets/(id)/automations,[PATCH] /assets/(id)/automations/(automation_id)and[DELETE] /assets/(id)/automations/(automation_id). They require the same rights as deleting theasset — account admins and consultants — which matches the automation's own access rules: whoever
may read an asset may read its automations, and whoever may delete it may change them. A
PATCHcovers the name, the recurrence, the timezone and the activation status; the parameters are
deliberately not editable, so the sensors an automation involves stay the ones its creator was
checked against.
UI. The asset's Automations page gains a creation modal and per-row actions for those users.
Only sensors the creator can access. An automation administered this way may only involve
sensors its creator can access themselves: read access to the sensors it reads from, and
create-children on the sensors it writes to, which is the permission the API already requires for
recording data on a sensor. A refused request gets a
403naming the sensor and the action. Thissits behind a flag that the API passes; the CLI creates automations without a user and stays
unrestricted.
Which sensors those are depends on the type:
config and parameters it will run with, so a regressor that filters on sources counts too even
though it is a sensor reference rather than a plain sensor.
tree are included, and the outputs are then taken from the fields that name where results are
recorded: a device's power sensor, its state of charge, consumption and production sensors, and
the flex-context's aggregates. Everything else the parameters refer to counts as an input.
Timezone through the API. An automation carries the timezone its recurrence is interpreted in.
Creation and update now accept it, so automations administered through the API or the UI are no
longer stuck on the server's timezone. Changing it rebases the scheduling cursor, exactly as the CLI
does.
Refusals leave nothing behind. The data source holding a forecaster's configuration is set up
only once the automation is allowed, so a refused request adds nothing. The check on where a
forecast may be recorded runs after the access check, so a sensor the caller may not read is refused
as forbidden rather than described as being outside the asset.
Consolidation. Creating, updating and deleting live in
flexmeasures/data/services/automations.py,so the CLI and the API share one implementation rather than each building automations inline.
documentation/changelog.rstLook & Feel
An account admin gets a New automation button and per-row Edit, Activate /
Deactivate and Delete actions:
The same page, for the same asset, as a plain user of the same organisation. The automations and
their details are still readable; nothing that would change them is offered:
Creating one asks for the type, the recurrence and the timezone it is interpreted in, and the
parameters — forecast parameters, or a schedule trigger message:
How to test
See the manual test walkthrough in the PR comments.
Coverage includes a forecast on another organisation's sensor, a forecast whose source-filtered
regressor is another organisation's sensor, a schedule aggregated onto another organisation's
sensor, and the timezone roundtrip through creation and update. Each was verified to fail without
the check it covers.
Further improvements
reading the fields that name where results go. That agrees with the scheduler's own resolution for
a device's
sensor,consumptionandproduction, and errs towards reporting more rather thanfewer. Checking outputs against what a scheduler actually returns at run time is tracked in Check the sensors a scheduler actually writes to, instead of predicting them when an automation is created #2421.
means deleting it and creating a new one, which keeps the access check honest but is blunt.
Related items
Closes #2372. Part of the automations story #2334. Stacked on #2293.
Sign-off