[chore](conf) Remove deprecated/unused FE/BE configs and stale session variables#63744
[chore](conf) Remove deprecated/unused FE/BE configs and stale session variables#63744jacktengg wants to merge 1 commit into
Conversation
|
/review |
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
…n variables
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Doris has accumulated a number of FE configs, BE configs and session
variables that are either explicitly marked `@Deprecated`, or simply have
no remaining references anywhere in the codebase (production sources,
regression tests, docs, or tools). They add noise to the user-facing
configuration surface and to the source tree without changing any
behavior. This commit removes them in three coordinated parts and adds
the minimum amount of backward-compatibility plumbing required to keep a
BE-then-FE rolling upgrade safe.
1. FE configs (fe/fe-common/.../Config.java)
Remove 13 `@Deprecated` fields with no remaining production
references: `meta_publish_timeout_ms`, `backup_plugin_path`,
`max_unfinished_load_job`, `use_new_tablet_scheduler`,
`enable_concurrent_update`, `cbo_max_statistics_job_num`,
`max_cbo_statistics_task_timeout_sec`,
`cbo_concurrency_statistics_task_num`, `cbo_default_sample_percentage`,
`finish_job_max_saved_second`, `enable_array_type`,
`period_analyze_simultaneously_running_task_num`,
`maximum_parallelism_of_export_job`.
`ConfigBase.java` gains a small `warnDeprecatedKeys()` helper. After
`setFields()` runs, if any of the legacy keys above is still listed in
`fe.conf`, FE logs a single WARN line so operators notice the stale
entry on upgrade. FE startup is not affected: the FE config loader
iterates over Java fields, so unknown keys in `fe.conf` were already
silently ignored before this change.
2. BE configs (be/src/common/config.{cpp,h}, be/src/cloud/config.{cpp,h})
Remove 24 configs that have zero references in the BE source tree:
`doris_scanner_max_run_time_ms`, `multi_get_max_threads`, `num_disks`,
`read_size`, `load_max_wg_active_memtable_percent`,
`jdbc_connection_pool_cache_clear_time_sec`, `big_column_size_buffer`,
`small_column_size_buffer`, `max_fragment_start_wait_time_seconds`,
`enable_workload_group_for_scan`,
`workload_group_scan_task_wait_timeout_ms`,
`variant_use_cloud_schema_dict_cache`,
`variant_threshold_rows_to_estimate_sparse_column`,
`variant_nested_group_max_depth`, `enable_ttl_cache_evict_using_lru`,
`file_cache_background_ttl_gc_batch`,
`kerberos_refresh_interval_second`, `enable_debug_log_timeout_secs`,
`enable_index_compaction`, `schema_dict_cache_capacity`,
`test_s3_resource`, `meta_service_use_load_balancer`,
`meta_service_rpc_timeout_ms`, `delete_bitmap_rpc_retry_times`.
The BE config loader (`be/src/common/config.cpp` `init()`) also
iterates over the registered field map, so unknown keys in `be.conf`
were already silently ignored — hard removal is safe on a rolling
upgrade. None of the removed options affect on-disk data format or
the BE↔BE / BE↔FE wire format, so the standard BE-first rolling
upgrade ordering is preserved.
3. Session variables
(fe/fe-core/.../qe/SessionVariable.java, .../qe/VariableMgr.java)
Remove 16 stale Doris-specific session variables along with their
public name constants and any getter/setter methods:
`enable_jdbc_oracle_null_predicate_push_down`, `use_v2_rollup`,
`rewrite_count_distinct_to_bitmap_hll`,
`enable_variant_access_in_original_planner`, `extract_wide_range_expr`,
`auto_broadcast_join_threshold`, `runtime_filters_max_num`,
`disable_inverted_index_v1_for_variant`, `enable_infer_predicate`,
`limit_rows_for_single_instance`, `nereids_star_schema_support`,
`enable_cbo_statistics`, `enable_eliminate_sort_node`,
`drop_table_if_ctas_failed`, `trace_nereids`,
`enable_sync_mv_cost_based_rewrite`.
Selection criteria for each removed variable: zero references in
`fe/fe-core/src/main/java` and `fe/fe-common/src/main/java` outside
`SessionVariable.java` itself (after expanding the search to cover
`get`/`set`/`is`-prefixed accessors of the field name, the public
name constant, the underlying lowercase string literal, the body of
`toThrift()`, and the body of `setForwardedSessionVariables()`), and
zero hits in `regression-test/`, `docs/`, `tools/`, and `be/test/`.
Upgrade compatibility is the critical concern here. Unlike fe.conf and
be.conf, unknown session-variable names cause
`VariableMgr` to throw `ERR_UNKNOWN_SYSTEM_VARIABLE` at four sites
(`setVar`, `setVarForNonMasterFE`, `fillValue`, `getValue`). A hard
delete would therefore break existing user scripts and JDBC client
connection init paths that still issue `SET` on the old names during
a BE-then-FE rolling upgrade. To stay safe, `VariableMgr` now keeps
an explicit `REMOVED_SESSION_VAR_NAMES` set listing the 16 removed
lowercase names; each of the four lookup sites short-circuits on this
set so that `SET <removed_var> = ...` silently no-ops and
`SELECT @@<removed_var>` returns an empty string. The same pattern is
already used for `mysql_compat_var_whitelist`.
The legacy FE configs and BE configs listed above have been removed. The
following session variables have been removed; `SET` and `SELECT @@`
against them now silently succeed (with empty/no-op behavior) instead of
returning the variable's previous default:
`enable_jdbc_oracle_null_predicate_push_down`, `use_v2_rollup`,
`rewrite_count_distinct_to_bitmap_hll`,
`enable_variant_access_in_original_planner`, `extract_wide_range_expr`,
`auto_broadcast_join_threshold`, `runtime_filters_max_num`,
`disable_inverted_index_v1_for_variant`, `enable_infer_predicate`,
`limit_rows_for_single_instance`, `nereids_star_schema_support`,
`enable_cbo_statistics`, `enable_eliminate_sort_node`,
`drop_table_if_ctas_failed`, `trace_nereids`,
`enable_sync_mv_cost_based_rewrite`. If any of the removed FE keys is
still present in `fe.conf` on upgrade, FE logs a WARN line on startup.
dca393f to
344906a
Compare
There was a problem hiding this comment.
Requesting changes for one compatibility regression in the session-variable cleanup.
Critical checkpoint conclusions:
- Goal/test proof: The PR removes deprecated/unused FE/BE configs and stale session variables. Most removals are guarded by warning/no-op compatibility paths, but
enable_jdbc_oracle_null_predicate_push_downis still actively consumed by the JDBC connector and is not covered by tests here. - Scope/focus: The config cleanup is generally focused, but this session variable is not stale while connector code still depends on it.
- Concurrency/lifecycle: No new shared mutable concurrency or non-intuitive lifecycle issue was identified.
- Configuration/session compatibility: Blocking issue found. Existing users can no longer enable Oracle NULL predicate pushdown because
SETis ignored and the session map omits the key. - Incompatible changes/storage formats: No storage-format or FE-BE protocol compatibility issue was found in the reviewed config removals.
- Parallel code paths: BE/FE deprecated config warning paths were both added; no missing corresponding config warning was identified.
- Test coverage: No test covers the still-active JDBC connector session variable path, which is why this regression can slip through.
- Performance/observability: No new runtime hot-path performance issue was identified in the current PR diff.
- User focus: No additional user-provided review focus was supplied.
| } | ||
|
|
||
| // Form map from variable name to its field in Java class. | ||
| static { |
There was a problem hiding this comment.
Removing this session variable breaks the JDBC Oracle NULL predicate pushdown path. JdbcQueryBuilder still reads enable_jdbc_oracle_null_predicate_push_down from the session properties map, but after this change SET enable_jdbc_oracle_null_predicate_push_down=true is silently ignored and VariableMgr.toMap() no longer includes the key, so oracleNullPredicatePushDown always falls back to false. Please keep this variable until the connector no longer depends on it, or replace it with another forwarded setting and update the connector at the same time.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Doris has accumulated a number of FE configs, BE configs and session variables that are either explicitly marked
@Deprecated, or simply have no remaining references anywhere in the codebase (production sources, regression tests, docs, or tools). They add noise to the user-facing configuration surface and to the source tree without changing any behavior. This commit removes them in three coordinated parts and adds the minimum amount of backward-compatibility plumbing required to keep a BE-then-FE rolling upgrade safe.FE configs (fe/fe-common/.../Config.java)
Remove 13
@Deprecatedfields with no remaining production references:meta_publish_timeout_ms,backup_plugin_path,max_unfinished_load_job,use_new_tablet_scheduler,enable_concurrent_update,cbo_max_statistics_job_num,max_cbo_statistics_task_timeout_sec,cbo_concurrency_statistics_task_num,cbo_default_sample_percentage,finish_job_max_saved_second,enable_array_type,period_analyze_simultaneously_running_task_num,maximum_parallelism_of_export_job.ConfigBase.javagains a smallwarnDeprecatedKeys()helper. AftersetFields()runs, if any of the legacy keys above is still listed infe.conf, FE logs a single WARN line so operators notice the stale entry on upgrade. FE startup is not affected: the FE config loader iterates over Java fields, so unknown keys infe.confwere already silently ignored before this change.BE configs (be/src/common/config.{cpp,h}, be/src/cloud/config.{cpp,h})
Remove 24 configs that have zero references in the BE source tree:
doris_scanner_max_run_time_ms,multi_get_max_threads,num_disks,read_size,load_max_wg_active_memtable_percent,jdbc_connection_pool_cache_clear_time_sec,big_column_size_buffer,small_column_size_buffer,max_fragment_start_wait_time_seconds,enable_workload_group_for_scan,workload_group_scan_task_wait_timeout_ms,variant_use_cloud_schema_dict_cache,variant_threshold_rows_to_estimate_sparse_column,variant_nested_group_max_depth,enable_ttl_cache_evict_using_lru,file_cache_background_ttl_gc_batch,kerberos_refresh_interval_second,enable_debug_log_timeout_secs,enable_index_compaction,schema_dict_cache_capacity,test_s3_resource,meta_service_use_load_balancer,meta_service_rpc_timeout_ms,delete_bitmap_rpc_retry_times.The BE config loader (
be/src/common/config.cppinit()) also iterates over the registered field map, so unknown keys inbe.confwere already silently ignored — hard removal is safe on a rolling upgrade. None of the removed options affect on-disk data format or the BE↔BE / BE↔FE wire format, so the standard BE-first rolling upgrade ordering is preserved.Session variables (fe/fe-core/.../qe/SessionVariable.java, .../qe/VariableMgr.java)
Remove 16 stale Doris-specific session variables along with their public name constants and any getter/setter methods:
enable_jdbc_oracle_null_predicate_push_down,use_v2_rollup,rewrite_count_distinct_to_bitmap_hll,enable_variant_access_in_original_planner,extract_wide_range_expr,auto_broadcast_join_threshold,runtime_filters_max_num,disable_inverted_index_v1_for_variant,enable_infer_predicate,limit_rows_for_single_instance,nereids_star_schema_support,enable_cbo_statistics,enable_eliminate_sort_node,drop_table_if_ctas_failed,trace_nereids,enable_sync_mv_cost_based_rewrite.Selection criteria for each removed variable: zero references in
fe/fe-core/src/main/javaandfe/fe-common/src/main/javaoutsideSessionVariable.javaitself (after expanding the search to coverget/set/is-prefixed accessors of the field name, the public name constant, the underlying lowercase string literal, the body oftoThrift(), and the body ofsetForwardedSessionVariables()), and zero hits inregression-test/,docs/,tools/, andbe/test/.Upgrade compatibility is the critical concern here. Unlike fe.conf and be.conf, unknown session-variable names cause
VariableMgrto throwERR_UNKNOWN_SYSTEM_VARIABLEat four sites (setVar,setVarForNonMasterFE,fillValue,getValue). A hard delete would therefore break existing user scripts and JDBC client connection init paths that still issueSETon the old names during a BE-then-FE rolling upgrade. To stay safe,VariableMgrnow keeps an explicitREMOVED_SESSION_VAR_NAMESset listing the 16 removed lowercase names; each of the four lookup sites short-circuits on this set so thatSET <removed_var> = ...silently no-ops andSELECT @@<removed_var>returns an empty string. The same pattern is already used formysql_compat_var_whitelist.The legacy FE configs and BE configs listed above have been removed. The following session variables have been removed;
SETandSELECT @@against them now silently succeed (with empty/no-op behavior) instead of returning the variable's previous default:enable_jdbc_oracle_null_predicate_push_down,use_v2_rollup,rewrite_count_distinct_to_bitmap_hll,enable_variant_access_in_original_planner,extract_wide_range_expr,auto_broadcast_join_threshold,runtime_filters_max_num,disable_inverted_index_v1_for_variant,enable_infer_predicate,limit_rows_for_single_instance,nereids_star_schema_support,enable_cbo_statistics,enable_eliminate_sort_node,drop_table_if_ctas_failed,trace_nereids,enable_sync_mv_cost_based_rewrite. If any of the removed FE keys is still present infe.confon upgrade, FE logs a WARN line on startup.What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)