refactor: validated config types and pipeline controller architecture#794
Closed
adwk67 wants to merge 14 commits into
Closed
refactor: validated config types and pipeline controller architecture#794adwk67 wants to merge 14 commits into
adwk67 wants to merge 14 commits into
Conversation
These dependencies are required by the framework module that will be added in subsequent commits: - regex: used by the attributed_string_type! macro for compile-time validated string types (e.g. RFC 1035/1123 name validation) - uuid: used for generating unique owner references in cluster resource management No code changes — dependencies are not yet referenced. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the attributed_string_type! macro which generates strongly-typed wrappers around strings with compile-time validation attributes: - RFC 1035 label names, RFC 1123 label/subdomain names - Label value constraints, length limits, regex patterns - Automatic FromStr, Display, Serialize/Deserialize, JsonSchema impls - Compile-time constant checks (IS_RFC_1035_LABEL_NAME, MAX_LENGTH, etc.) Also adds a small constant! helper macro for validated const values. The NameIsValidLabelValue trait is included in framework.rs as the macro references it for label-value-safe types. These macros are the foundation for the validated type system that replaces stringly-typed names throughout the operator. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce strongly-typed wrappers for Kubernetes and operator names: - types/common: Port type with range validation - types/kubernetes: ConfigMapName, ServiceName, StatefulSetName, ListenerName, NamespaceName, Uid — each with appropriate RFC constraints and max_length limits - types/operator: ProductName, ClusterName, RoleName, RoleGroupName, ControllerName, OperatorName — with operator-specific length budgets ensuring the combined <cluster>-<role>-<rolegroup> fits within StatefulSet name limits (52 chars) Operator-specific values (max_length, examples) are flagged with REVIEW comments for parameterisation when this module moves to operator-rs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add label construction functions that accept validated types (ProductName, OperatorName, ClusterName, etc.) instead of raw strings, eliminating the possibility of label value constraint violations at the call site. Provides: recommended_labels, role_group_selector_labels, prometheus_labels, and prometheus_annotations. The HasName trait is added to framework.rs as it is required by the label functions to extract names from cluster objects. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add type-safe builder functions for constructing Kubernetes resources using validated types instead of raw strings: - builder/meta: ObjectMeta with validated names, labels, owner refs - builder/pdb: PodDisruptionBudget with role-aware max_unavailable - builder/pod/container: container construction with ports, env vars, volume mounts, resource limits, probes - builder/pod/volume: config map and empty dir volume helpers - builder/statefulset: StatefulSet with pod template, service name, replicas, restart policy The HasUid trait is added to framework.rs as it is required by the meta builder for constructing owner references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete the framework module with: - role_utils: role config validation, merging, and iteration with validated types — replaces stringly-typed role/rolegroup handling - role_group_utils: QualifiedRoleGroupName with compile-time length assertions ensuring <cluster>-<role>-<rolegroup> fits within StatefulSet, Service, ConfigMap, and Listener name limits - controller_utils: config merging for role/rolegroup hierarchies, producing validated config per rolegroup - cluster_resources: typed wrapper around operator-rs ClusterResources - product_logging: validated logging config types (ValidatedContainerLogConfigChoice, VectorContainerLogConfig) Modules marked #[allow(dead_code)] are not yet referenced by the controller — they will be wired in when the controller pipeline is replaced in a later commit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Required so AirflowRole can be used as a BTreeMap key in the new controller's validated cluster type. No behavioural change — the enum already derives Eq/Hash, and the variant ordering is stable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The function only calls rolegroup_ref.object_name() which is available on any RoleGroupRef<T: Resource>. Making it generic allows the new controller to call it with RoleGroupRef<ValidatedAirflowCluster>. No behavioural change — existing callers continue to work. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the &ExecutorConfig parameter with a plain bool (vector_agent_enabled) — the function only ever read config.logging.enable_vector_agent from it. This decouples the function from the ExecutorConfig type, allowing the new controller to call it with a pre-validated bool without needing to reconstruct an ExecutorConfig. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the monolithic reconcile_airflow with a five-stage pipeline: dereference → validate → build → apply → update_status. The build stage is infallible — all validation and fallible operations (config generation, PodBuilder/ContainerBuilder usage, logging validation) happen in the validate stage. Services and PDBs are now built inline rather than delegating to service.rs and operations/. Also updates product_logging.rs and controller_commons.rs to accept pre-validated types (ValidatedContainerLogConfigChoice) instead of raw Logging<C>/ContainerLogConfig, making those functions infallible. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete the operations/ module (graceful_shutdown, pdb) — its functionality is now handled inline by the validate and build stages. Remove dead functions from service.rs (service building is now inline in the controller build stage) and crd/mod.rs (build_recommended_labels replaced by framework::kvp::label functions). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the monolithic airflow_controller.rs into controller/ with submodules: dereference, validate, build (+ build/role_group_builder), apply, and update_status. This is a purely structural refactor — no logic changes. Each pipeline stage now lives in its own file for easier navigation and review. Rename airflow_controller module to controller in main.rs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RoleGroupName has a max_length of 16 characters. Rename test role groups to fit within this limit: - resources-from-role (19) → from-role (9) - resources-from-role-group (25) → from-rg (7) - resources-from-pod-overrides (28) → from-overrides (14) - automatic-log-config (20) → auto-log-cfg (12) - custom-log-config (17) → custom-log-cfg (14) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove all // REVIEW: comments that were added to aid code review. These served their purpose during the review process and are no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
Author
|
Closing in favour of #795 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
These PRs are primarily for validating the refactoring process and can be ignored, merged, closed etc. as we see fit.
Summary
frameworkmodule with validated string types (ClusterName,RoleName,RoleGroupName, etc.), type-safe label builders, PDB/StatefulSet builders, and product logging utilities — designed for eventual migration to operator-rsairflow_controller.rswith a 5-stage pipeline architecture:dereference → validate → build → apply → update_status, where the build stage is infallibleValidatedAirflowClusteras a central validated type holding all data needed by the build stage, with compile-time length assertions viaattributed_string_type!macrooperations/,service.rs, andcrd/mod.rsRoleGroupNamelimitCommit structure
Each commit is designed to be reviewed in isolation:
chore: add regex and uuid dependencies— dependency additions onlyfeat: add framework macros—attributed_string_type!macrofeat: add framework validated types—ClusterName,RoleName, etc.feat: add framework KVP module— type-safe label buildersfeat: add framework builders— PDB, StatefulSet, meta, pod buildersfeat: add framework utilities— role utils, logging, controller utilsrefactor: derive Ord/PartialOrd on AirflowRole— prep for BTreeMap keysrefactor: make stateful_set_service_name generic— decouple from AirflowClusterrefactor: simplify build_airflow_template_envs signature— reduce parameter countrefactor: replace monolithic controller with pipeline architecture— the big transformationrefactor: remove dead code— clean up operations/, service.rs, crdrefactor: extract controller into submodules— split into dereference/validate/build/apply/update_statusrefactor: shorten test role group names— fit 16-char RoleGroupName limitchore: remove REVIEW comments— clean up review markersTest plan
cargo test)GIT_CREDENTIALS_SECRET_PROPERTY)🤖 Generated with Claude Code