Automate backend functionality and model compatibility verification across Annif releases #906

Activity

1. juhoinkinen

   added

   maintenance

   github_actionsPull requests that update GitHub Actions codePull requests that update GitHub Actions code

   on Oct 27, 2025
2. juhoinkinen commented on Oct 27, 2025

   

   juhoinkinen

   on Oct 27, 2025

   MemberAuthor

   More actions

   Chatting with Copilot on agent mode gave this plan:

   1. Model Compatibility Check (Backward Compatibility)

      Goal: Ensure models trained with the previous Annif version can be loaded and evaluated with the current development version, and that evaluation results do not change significantly.
      Process:
      Store models trained with the previous release in tests/compat_models/prev_version/.
      For each model:
      Load and evaluate using the current development version.
      Compare evaluation metrics to those obtained with the previous version (stored in a results file, e.g., prev_eval_results.json).
      Allow for a small, configurable threshold for metric differences.

   2. Model Reproducibility Check (Forward Consistency)

      Goal: Ensure models trained with the current development version produce the same evaluation results as models trained with the previous version, when using the same data and configuration.
      Process:
      Train new models with the current development version using the same configs/data as before.
      Evaluate these new models.
      Compare their evaluation metrics to those from the previous version’s models (again, using prev_eval_results.json).
      Flag significant differences.
3. juhoinkinen commented on Oct 27, 2025

   

   juhoinkinen

   on Oct 27, 2025

   MemberAuthor

   More actions

   Continuing the chat and the initial implementation gave these ideas:

   • the annif eval [OPTIONS] PROJECT_ID [PATHS] command has the option --metrics-file <FILENAME> to specify file in order to write evaluation metrics in JSON format, use that.
   • for permanent storage it is best to use Hugging Face Hub:
     • utilize annif upload [OPTIONS] PROJECT_IDS_PATTERN REPO_ID and annif download [OPTIONS] PROJECT_IDS_PATTERN REPO_ID commands for models
     • utilize Hugging Face Model card for evaluation results, possibly via the eval_results attribute
4. juhoinkinen

   self-assigned this

   on Oct 29, 2025