[ci] refactor testcoverage config#4630
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors the autotest “test coverage” configuration and CI wiring by moving environment-specific path settings into a single autotest/env_paths.yml, switching evaluation to a unified preset resolver, and introducing/normalizing many per-model config matrices under autotest/configs/**. It also updates several CI workflows to use a DEPS_PROFILE matrix for pinned dependency rows and adjusts a few benchmark/test utilities accordingly.
Changes:
- Centralize environment path configuration in
autotest/env_paths.ymland update autotest fixtures to load config viaget_config(). - Route API evaluation preset selection through
get_eval_preset_config(...)(enabling per-modelgen_configoverrides while keeping scalar preset limits). - Add/normalize a large set of per-model autotest config YAMLs and update CI workflows (checkout version bumps + new
DEPS_PROFILEstrategy).
Reviewed changes
Copilot reviewed 144 out of 144 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| autotest/utils/constant.py | Adds clarifying comments and switches “legacy” selection logic to DEPS_PROFILE-based detection. |
| autotest/utils/benchmark_utils.py | Adds --trust-remote-code for restful profiling and refactors prefix-cache throughput command construction. |
| autotest/tools/common_case_config.py | Updates fallback test model choices (notably for Turbomind MLLM fallback). |
| autotest/interface/restful/test_restful_completions_v1.py | Adds extensive debug print(...) output in RESTful completions tests. |
| autotest/interface/restful/test_restful_chat_completions_v1.py | Relaxes empty-content assertions to accept falsy content. |
| autotest/interface/pipeline/test_pipeline_longtext_func.py | Refactors longtext tests: adds spawn exitcode check, adds response assertions, introduces stream-merge helpers, updates model list. |
| autotest/evaluate/test_mllm_api_evaluate.py | Uses get_eval_preset_config(..., mllm=True) instead of direct constants lookup. |
| autotest/evaluate/test_api_evaluate.py | Uses get_eval_preset_config(...) instead of direct constants lookup. |
| autotest/evaluate/eval_config_chat.py | Reduces evaluation batch size (1024 → 256). |
| autotest/env_paths.yml | New: central per-environment paths for autotest runs. |
| autotest/conftest.py | Switches session config loading to get_config() (env_paths-driven). |
| autotest/configs/zai-org/GLM-5-FP8.yml | New per-model matrix entry for zai-org/GLM-5-FP8. |
| autotest/configs/zai-org/GLM-4.7-Flash.yml | New per-model matrix entry for zai-org/GLM-4.7-Flash. |
| autotest/configs/unsloth/gpt-oss-20b-BF16.yml | New per-model matrix entry for unsloth/gpt-oss-20b-BF16. |
| autotest/configs/THUDM/glm-4v-9b.yml | New per-model matrix entry for THUDM/glm-4v-9b. |
| autotest/configs/THUDM/glm-4-9b-chat.yml | New per-model matrix entry for THUDM/glm-4-9b-chat. |
| autotest/configs/THUDM/cogvlm2-llama3-chinese-chat-19B.yml | New per-model matrix entry for THUDM/cogvlm2-llama3-chinese-chat-19B. |
| autotest/configs/THUDM/cogvlm-chat-hf.yml | New per-model matrix entry for THUDM/cogvlm-chat-hf. |
| autotest/configs/THUDM/chatglm2-6b.yml | New per-model matrix entry for THUDM/chatglm2-6b. |
| autotest/configs/test/test_vl_tp8.yml | New test matrix config for test_vl_tp8. |
| autotest/configs/test/test_vl_tp2.yml | New test matrix config for test_vl_tp2. |
| autotest/configs/test/test_vl_tp2_int4.yml | New test matrix config for test_vl_tp2_int4. |
| autotest/configs/test/test_vl_tp2_gpqa.yml | New test matrix config for test_vl_tp2_gpqa. |
| autotest/configs/test/test_vl_tp1.yml | New test matrix config for test_vl_tp1. |
| autotest/configs/test/test_vl_tp1_pytorch.yml | New test matrix config for test_vl_tp1_pytorch. |
| autotest/configs/test/test_vl_dpep8.yml | New test matrix config for test_vl_dpep8. |
| autotest/configs/test/test_vl_dpep16.yml | New test matrix config for test_vl_dpep16. |
| autotest/configs/test/test_vl_cp2tp8.yml | New test matrix config for test_vl_cp2tp8. |
| autotest/configs/test/test_tp8.yml | New test matrix config for test_tp8. |
| autotest/configs/test/test_tp2.yml | New test matrix config for test_tp2. |
| autotest/configs/test/test_tp2_int4.yml | New test matrix config for test_tp2_int4. |
| autotest/configs/test/test_tp2_gpqa.yml | New test matrix config for test_tp2_gpqa. |
| autotest/configs/test/test_tp1.yml | New test matrix config for test_tp1. |
| autotest/configs/test/test_tp1_pytorch.yml | New test matrix config for test_tp1_pytorch. |
| autotest/configs/test/test_dpep8.yml | New test matrix config for test_dpep8. |
| autotest/configs/test/test_dpep16.yml | New test matrix config for test_dpep16. |
| autotest/configs/test/test_cp2tp8.yml | New test matrix config for test_cp2tp8. |
| autotest/configs/test/test_allkind.yml | New test matrix config for test_allkind. |
| autotest/configs/Qwen/Qwen3.5-397B-A17B.yml | New per-model matrix entry for Qwen/Qwen3.5-397B-A17B. |
| autotest/configs/Qwen/Qwen3.5-397B-A17B-FP8.yml | New per-model matrix entry for Qwen/Qwen3.5-397B-A17B-FP8. |
| autotest/configs/Qwen/Qwen3.5-35B-A3B.yml | New per-model matrix entry for Qwen/Qwen3.5-35B-A3B. |
| autotest/configs/Qwen/Qwen3.5-35B-A3B-FP8.yml | New per-model matrix entry for Qwen/Qwen3.5-35B-A3B-FP8. |
| autotest/configs/Qwen/Qwen3.5-35B-A3B-Base.yml | New per-model matrix entry for Qwen/Qwen3.5-35B-A3B-Base. |
| autotest/configs/Qwen/Qwen3.5-2B-Base.yml | New per-model matrix entry for Qwen/Qwen3.5-2B-Base. |
| autotest/configs/Qwen/Qwen3.5-27B.yml | New per-model matrix entry for Qwen/Qwen3.5-27B. |
| autotest/configs/Qwen/Qwen3.5-122B-A10B.yml | New per-model matrix entry for Qwen/Qwen3.5-122B-A10B. |
| autotest/configs/Qwen/Qwen3.5-0.8B.yml | New per-model matrix entry for Qwen/Qwen3.5-0.8B. |
| autotest/configs/Qwen/Qwen3-VL-8B-Instruct.yml | New per-model matrix entry for Qwen/Qwen3-VL-8B-Instruct. |
| autotest/configs/Qwen/Qwen3-VL-32B-Instruct.yml | New per-model matrix entry for Qwen/Qwen3-VL-32B-Instruct. |
| autotest/configs/Qwen/Qwen3-VL-30B-A3B-Instruct.yml | New per-model matrix entry for Qwen/Qwen3-VL-30B-A3B-Instruct. |
| autotest/configs/Qwen/Qwen3-8B.yml | New per-model matrix entry for Qwen/Qwen3-8B. |
| autotest/configs/Qwen/Qwen3-8B-Base.yml | New per-model matrix entry for Qwen/Qwen3-8B-Base. |
| autotest/configs/Qwen/Qwen3-4B.yml | New per-model matrix entry for Qwen/Qwen3-4B. |
| autotest/configs/Qwen/Qwen3-32B.yml | New per-model matrix entry for Qwen/Qwen3-32B. |
| autotest/configs/Qwen/Qwen3-32B-FP8.yml | New per-model matrix entry for Qwen/Qwen3-32B-FP8. |
| autotest/configs/Qwen/Qwen3-30B-A3B.yml | New per-model matrix entry for Qwen/Qwen3-30B-A3B. |
| autotest/configs/Qwen/Qwen3-30B-A3B-GPTQ-Int4.yml | New per-model matrix entry for Qwen/Qwen3-30B-A3B-GPTQ-Int4. |
| autotest/configs/Qwen/Qwen3-30B-A3B-FP8.yml | New per-model matrix entry for Qwen/Qwen3-30B-A3B-FP8. |
| autotest/configs/Qwen/Qwen3-30B-A3B-Base.yml | New per-model matrix entry for Qwen/Qwen3-30B-A3B-Base. |
| autotest/configs/Qwen/Qwen3-235B-A22B.yml | New per-model matrix entry for Qwen/Qwen3-235B-A22B. |
| autotest/configs/Qwen/Qwen3-235B-A22B-Thinking-2507.yml | New per-model matrix entry for Qwen/Qwen3-235B-A22B-Thinking-2507. |
| autotest/configs/Qwen/Qwen3-235B-A22B-Thinking-2507-FP8.yml | New per-model matrix entry for Qwen/Qwen3-235B-A22B-Thinking-2507-FP8. |
| autotest/configs/Qwen/Qwen3-1.7B.yml | New per-model matrix entry for Qwen/Qwen3-1.7B. |
| autotest/configs/Qwen/Qwen3-0.6B.yml | New per-model matrix entry for Qwen/Qwen3-0.6B. |
| autotest/configs/Qwen/Qwen2.5-VL-7B-Instruct.yml | New per-model matrix entry for Qwen/Qwen2.5-VL-7B-Instruct. |
| autotest/configs/Qwen/Qwen2.5-VL-3B-Instruct.yml | New per-model matrix entry for Qwen/Qwen2.5-VL-3B-Instruct. |
| autotest/configs/Qwen/Qwen2.5-VL-32B-Instruct.yml | New per-model matrix entry for Qwen/Qwen2.5-VL-32B-Instruct. |
| autotest/configs/Qwen/Qwen2.5-7B-Instruct.yml | New per-model matrix entry for Qwen/Qwen2.5-7B-Instruct. |
| autotest/configs/Qwen/Qwen2.5-72B-Instruct.yml | New per-model matrix entry for Qwen/Qwen2.5-72B-Instruct. |
| autotest/configs/Qwen/Qwen2-57B-A14B-Instruct-GPTQ-Int4.yml | New per-model matrix entry for Qwen/Qwen2-57B-A14B-Instruct-GPTQ-Int4. |
| autotest/configs/Qwen/Qwen1.5-MoE-A2.7B-Chat.yml | New per-model matrix entry for Qwen/Qwen1.5-MoE-A2.7B-Chat. |
| autotest/configs/OpenGVLab/Mini-InternVL-Chat-2B-V1-5.yml | New per-model matrix entry for OpenGVLab/Mini-InternVL-Chat-2B-V1-5. |
| autotest/configs/OpenGVLab/InternVL3-8B.yml | New per-model matrix entry for OpenGVLab/InternVL3-8B. |
| autotest/configs/OpenGVLab/InternVL3-38B.yml | New per-model matrix entry for OpenGVLab/InternVL3-38B. |
| autotest/configs/OpenGVLab/InternVL3-2B-Instruct.yml | New per-model matrix entry for OpenGVLab/InternVL3-2B-Instruct. |
| autotest/configs/OpenGVLab/InternVL3-1B-Instruct.yml | New per-model matrix entry for OpenGVLab/InternVL3-1B-Instruct. |
| autotest/configs/OpenGVLab/InternVL3_5-8B.yml | New per-model matrix entry for OpenGVLab/InternVL3_5-8B. |
| autotest/configs/OpenGVLab/InternVL3_5-38B.yml | New per-model matrix entry for OpenGVLab/InternVL3_5-38B. |
| autotest/configs/OpenGVLab/InternVL3_5-30B-A3B.yml | New per-model matrix entry for OpenGVLab/InternVL3_5-30B-A3B. |
| autotest/configs/OpenGVLab/InternVL3_5-2B.yml | New per-model matrix entry for OpenGVLab/InternVL3_5-2B. |
| autotest/configs/OpenGVLab/InternVL2-Llama3-76B-AWQ.yml | New per-model matrix entry for OpenGVLab/InternVL2-Llama3-76B-AWQ. |
| autotest/configs/OpenGVLab/InternVL2_5-8B.yml | New per-model matrix entry for OpenGVLab/InternVL2_5-8B. |
| autotest/configs/OpenGVLab/InternVL2_5-1B.yml | New per-model matrix entry for OpenGVLab/InternVL2_5-1B. |
| autotest/configs/openai/gpt-oss-20b.yml | New per-model matrix entry for openai/gpt-oss-20b. |
| autotest/configs/openai/gpt-oss-120b.yml | New per-model matrix entry for openai/gpt-oss-120b. |
| autotest/configs/moonshotai/Kimi-K2-Instruct-0905.yml | New per-model matrix entry for moonshotai/Kimi-K2-Instruct-0905. |
| autotest/configs/mistralai/Mixtral-8x7B-Instruct-v0.1.yml | New per-model matrix entry for mistralai/Mixtral-8x7B-Instruct-v0.1. |
| autotest/configs/microsoft/Phi-4-mini-instruct.yml | New per-model matrix entry for microsoft/Phi-4-mini-instruct. |
| autotest/configs/meta-llama/Meta-Llama-3.1-8B-Instruct.yml | New per-model matrix entry for meta-llama/Meta-Llama-3.1-8B-Instruct. |
| autotest/configs/meta-llama/Meta-Llama-3.1-70B-Instruct.yml | New per-model matrix entry for meta-llama/Meta-Llama-3.1-70B-Instruct. |
| autotest/configs/meta-llama/Meta-Llama-3-8B-Instruct.yml | New per-model matrix entry for meta-llama/Meta-Llama-3-8B-Instruct. |
| autotest/configs/meta-llama/Meta-Llama-3-1-8B-Instruct.yml | New per-model matrix entry for meta-llama/Meta-Llama-3-1-8B-Instruct. |
| autotest/configs/meta-llama/Meta-Llama-3-1-8B-Instruct-AWQ.yml | New per-model matrix entry for meta-llama/Meta-Llama-3-1-8B-Instruct-AWQ. |
| autotest/configs/meta-llama/Meta-Llama-3-1-70B-Instruct.yml | New per-model matrix entry for meta-llama/Meta-Llama-3-1-70B-Instruct. |
| autotest/configs/meta-llama/Llama-4-Scout-17B-16E-Instruct.yml | New per-model matrix entry for meta-llama/Llama-4-Scout-17B-16E-Instruct. |
| autotest/configs/meta-llama/Llama-3.2-3B-Instruct.yml | New per-model matrix entry for meta-llama/Llama-3.2-3B-Instruct. |
| autotest/configs/meta-llama/Llama-3.2-1B-Instruct.yml | New per-model matrix entry for meta-llama/Llama-3.2-1B-Instruct. |
| autotest/configs/meta-llama/Llama-2-7b-chat-hf.yml | New per-model matrix entry for meta-llama/Llama-2-7b-chat-hf. |
| autotest/configs/liuhaotian/llava-v1.6-vicuna-7b.yml | New per-model matrix entry for liuhaotian/llava-v1.6-vicuna-7b. |
| autotest/configs/JetLM/SDAR-30B-A3B-Sci.yml | New per-model matrix entry for JetLM/SDAR-30B-A3B-Sci. |
| autotest/configs/internlm/internlm3-8b-instruct.yml | New per-model matrix entry for internlm/internlm3-8b-instruct. |
| autotest/configs/internlm/internlm3-8b-instruct-awq.yml | New per-model matrix entry for internlm/internlm3-8b-instruct-awq. |
| autotest/configs/internlm/Intern-S1.yml | New per-model matrix entry for internlm/Intern-S1. |
| autotest/configs/internlm/Intern-S1-Pro-FP8.yml | New per-model matrix entry for internlm/Intern-S1-Pro-FP8. |
| autotest/configs/internlm/Intern-S1-Pro-BF16.yml | New per-model matrix entry for internlm/Intern-S1-Pro-BF16. |
| autotest/configs/internlm/Intern-S1-mini.yml | New per-model matrix entry for internlm/Intern-S1-mini. |
| autotest/configs/google/gemma-7b-it.yml | New per-model matrix entry for google/gemma-7b-it. |
| autotest/configs/google/gemma-3-27b-it.yml | New per-model matrix entry for google/gemma-3-27b-it. |
| autotest/configs/google/gemma-3-12b-it.yml | New per-model matrix entry for google/gemma-3-12b-it. |
| autotest/configs/google/gemma-2-9b-it.yml | New per-model matrix entry for google/gemma-2-9b-it. |
| autotest/configs/google/gemma-2-27b-it.yml | New per-model matrix entry for google/gemma-2-27b-it. |
| autotest/configs/deepseek-ai/DeepSeek-V3.1.yml | New per-model matrix entry for deepseek-ai/DeepSeek-V3.1. |
| autotest/configs/deepseek-ai/DeepSeek-V2-Lite-Chat.yml | New per-model matrix entry for deepseek-ai/DeepSeek-V2-Lite-Chat. |
| autotest/configs/deepseek-ai/deepseek-moe-16b-chat.yml | New per-model matrix entry for deepseek-ai/deepseek-moe-16b-chat. |
| autotest/configs/codellama/CodeLlama-7b-Instruct-hf.yml | New per-model matrix entry for codellama/CodeLlama-7b-Instruct-hf. |
| autotest/configs/codellama/CodeLlama-7b-hf.yml | New per-model matrix entry for codellama/CodeLlama-7b-hf. |
| autotest/configs/bigcode/starcoder2-7b.yml | New per-model matrix entry for bigcode/starcoder2-7b. |
| autotest/configs/baichuan-inc/Baichuan2-7B-Chat.yml | New per-model matrix entry for baichuan-inc/Baichuan2-7B-Chat. |
| autotest/configs/baichuan-inc/Baichuan2-13B-Chat.yml | New per-model matrix entry for baichuan-inc/Baichuan2-13B-Chat. |
| autotest/config_testascend.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_test.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_legacy.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_h800.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_h_legacy.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_ascend.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_5080.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_5080_legacy.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_3090.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/config_3090_legacy.yml | Removes legacy flat config (superseded by env_paths + per-model matrices). |
| autotest/benchmark/test_prefixcache_performance.py | Expands prefix-cache performance coverage by attempting to add prefix-caching variants. |
| .github/workflows/pr_ete_test.yml | Updates checkout action version. |
| .github/workflows/mllm_api_eval.yml | Updates checkout action version. |
| .github/workflows/evaluate.yml | Updates checkout action version. |
| .github/workflows/docker_nightly.yml | Updates checkout action version and extends nightly test image build steps. |
| .github/workflows/daily_ete_test_5080.yml | Refactors test matrix to use DEPS_PROFILE instead of legacy env tags and adjusts install steps. |
| .github/workflows/daily_ete_test_3090.yml | Refactors test matrix to use DEPS_PROFILE instead of legacy env tags and adjusts install steps. |
| .github/workflows/benchmark.yml | Switches benchmark container tag default to nightly-test-cu12.8. |
| .github/workflows/api_eval.yml | Updates checkout action version. |
| .github/scripts/eval_base_config.py | Reduces OpenCompass dataset import set and summary output configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| timeout-minutes: 150 | ||
| strategy: | ||
| matrix: | ||
| deps_profile: ['', 'transformers==4.57.6'] |
There was a problem hiding this comment.
Can transformers v5.6.0 work?
| quantization: | ||
| pytorch: | ||
| - kvint4 | ||
| - kvint8 |
There was a problem hiding this comment.
We can remove its quantization test
| - quantization | ||
| quantization: | ||
| turbomind: | ||
| - awq | ||
| - kvint4 | ||
| - kvint8 |
There was a problem hiding this comment.
We can remove lagecy models' quantization test
| RUN python3 -m pip install --no-cache-dir -r /tmp/requirements/lite.txt && \ | ||
| python3 -m pip install --no-cache-dir -r /tmp/requirements/test.txt && \ | ||
| pip install --no-cache-dir ${{ env.OFFLINE_REQUIREMENTS }} | ||
| pip install --no-cache-dir ${{ env.OFFLINE_REQUIREMENTS }} && \ |
There was a problem hiding this comment.
Could you let us know which pypi package downgrade the "nvidia-nccl-cu12"?
| @@ -0,0 +1,18 @@ | |||
| # HF model id: deepseek-ai/deepseek-moe-16b-chat | |||
There was a problem hiding this comment.
deepseek-ai/deepseek-moe-16b-chat can be eliminated
| tp: 8 | ||
| extra: | ||
| session-len: 65536 |
There was a problem hiding this comment.
Can we also add dp+ep case?
| @@ -0,0 +1,30 @@ | |||
| # HF model id: internlm/internlm3-8b-instruct-awq | |||
There was a problem hiding this comment.
We can remove internlm3 series models
| deps: | ||
| transformers: 4.57.6 |
| - name: pytorch | ||
| communicators: | ||
| - nccl |
There was a problem hiding this comment.
pytorch engine doesn't support cp. Therefore, please remove it
| pytorch: | ||
| - kvint8 |
There was a problem hiding this comment.
It can be removed safely
| @@ -0,0 +1,25 @@ | |||
| # HF model id: meta-llama/Meta-Llama-3.1-70B-Instruct | |||
There was a problem hiding this comment.
Can it be merged with "Meta-Llama-3-1-70B-Instruct.yml"?
| @@ -0,0 +1,28 @@ | |||
| # HF model id: meta-llama/Meta-Llama-3.1-8B-Instruct | |||
There was a problem hiding this comment.
Can we merge it into 'Meta-Llama-3-1-8B-Instruct'?
| @@ -0,0 +1,33 @@ | |||
| # HF model id: Qwen/Qwen1.5-MoE-A2.7B-Chat | |||
There was a problem hiding this comment.
We can remove this test case
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Please describe the motivation of this PR and the goal you want to achieve through this PR.
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist