This project tests three machine-learning models to predict whether an auto insurance policyholder will file a claim:
- Best model for finding claims: Logistic Regression
- Claims found: 58.5% of actual claims (recall)
- F1 score: 0.1546
- ROC-AUC: 0.6215
The Decision Tree and Random Forest models have higher accuracy (about 85.6%),
but they miss about 87%–89% of claims. Accuracy is misleading here because
only about 6.4% of policies in the dataset have a claim. The current results
come from the 20% holdout split and are saved in
results/metrics.csv.
| Model | Accuracy | Precision | Recall | F1 | ROC-AUC | Missed claims |
|---|---|---|---|---|---|---|
| Logistic Regression | 59.0% | 8.9% | 58.5% | 0.1546 | 0.6215 | 41.5% |
| Decision Tree | 85.6% | 7.4% | 10.9% | 0.0883 | 0.5079 | 89.1% |
| Random Forest | 85.7% | 8.5% | 12.7% | 0.1018 | 0.5761 | 87.3% |
These are results for the supplied data and configuration, not a promise of future performance. Rerun the pipeline after changing the data, configuration, random seed, or library versions.
- What this project does
- Project structure
- Run the project
- Understand the pipeline
- Output files
- Configuration
- Run the tests
- Troubleshooting
The input is a CSV file containing insurance policy information. The model
predicts the binary is_claim column:
0: no claim1: claim
The project compares:
- Logistic Regression
- Decision Tree
- Random Forest
Because claims are uncommon, the project reports more than accuracy: precision, recall, F1, ROC-AUC, average precision, and false-negative rate. These measures show how many real claims the models find and how many they miss.
data/
├── train_data.csv # labeled training data
├── truncated_train_data.csv # smaller data for quick checks
└── test_data.csv # unlabeled data for predictions
src/
├── main.py # command-line entry point
├── config.py # application and model settings
├── pipeline.py # training, evaluation, and exports
├── preprocessor.py # feature conversion and encoding
├── data_loader.py # CSV loading
├── model_factory.py # model creation
├── models.py # model adapters and metrics
├── sampling_strategy.py # class-balancing methods
├── scaler_factory.py # feature scaling
└── visualizer.py # chart generation
tests/ # unit and end-to-end tests
results/ # CSV and JSON results
visualizations/ # generated charts
Run commands from the repository root. The default training file is
data/train_data.csv. It must contain a binary is_claim column.
Choose Docker or a local Python environment. You only need one option.
Install Docker Desktop or Docker Engine with Docker Compose, then run:
docker compose up --buildThe first build may take several minutes. Later runs can use:
docker compose upDocker writes results/ and visualizations/ to the repository, so the files
remain available after the container stops. Stop the container with:
docker compose downUseful Docker commands:
# Run without creating charts
docker compose run --rm ml-pipeline python3 ./src/main.py --no-viz
# Train the models and score the supplied unlabeled data
docker compose run --rm ml-pipeline python3 ./src/main.py \
--predict-data data/test_data.csvInstall Python 3.12 or newer and uv, then run:
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -r requirements.txt
python src/main.pyInstall Conda, then create the environment:
conda env create -f environment.yml
conda activate prediction-model-compare-prod
python src/main.pyIf the environment already exists, update it instead:
conda env update -f environment.yml --prune
conda activate prediction-model-compare-prodOptional commands:
python src/main.py --no-viz
python src/main.py --viz-dir reports/visualizations
python src/main.py --predict-data data/test_data.csv
python src/main.py --helpEach run follows these steps:
- Load and validate the training CSV.
- Convert
Yes/No, numeric, ordinal, and categorical features. - Remove the policy identifier.
- Split the data into an 80% training set and a 20% holdout set.
- Apply SMOTE to the training set to give the rare claim class more examples.
- Scale the features.
- Train the three models.
- Evaluate them on the untouched holdout set.
- Save metrics, predictions, and charts.
The split is stratified and uses random seed 11, so the results are
repeatable. SMOTE is applied only to the training data; the holdout keeps the
original claim rate for a fair evaluation.
The pipeline also checks different probability thresholds to find a threshold that improves F1. This analysis is reported in the results, but the default predictions file uses each model's normal prediction behavior.
After a normal run, check these files first:
results/
├── metrics.csv # metrics in a spreadsheet-friendly format
├── metrics.json # same metrics in JSON format
├── predictions.csv # holdout labels, predictions, and probabilities
└── inference_predictions.csv # created only with --predict-data
The visualizer creates PNG reports in:
visualizations/
├── 01_data/ # class balance, correlations, distributions
├── 02_performance/ # metric comparisons and confusion matrices
├── 03_discrimination/ # ROC/PR curves and threshold analysis
├── 04_features/ # feature importance charts
└── 05_diagnostics/ # calibration and classification reports
predictions.csv is for the labeled holdout set and includes actual labels.
inference_predictions.csv is different: it is created only when an unlabeled
file is supplied with --predict-data.
The main performance chart is
visualizations/02_performance/performance_comparison.png.
Most settings are in src/config.py. Important defaults are:
| Setting | Default | Meaning |
|---|---|---|
data_path |
data/train_data.csv |
Training CSV |
test_size |
0.2 |
Holdout fraction |
random_state |
11 |
Repeatability seed |
oversampling_method |
smote |
random, smote, adasyn, or none |
scaler_type |
standard |
standard, minmax, robust, or none |
validation_method |
holdout |
Holdout or cross-validation |
threshold_metric |
f1 |
f1, recall, or precision |
export_results |
True |
Write result files |
The models and their hyperparameters are also defined in ModelConfig in
src/config.py. Edit the configuration and run the command
again to compare a different setup.
The canary Conda environment includes pytest:
conda env update -f environment-canary.yml --prune
conda activate prediction-model-compare-canary
pytest -qThe tests use the supplied sample data and do not need an external service.
- Dataset not found: run the command from the repository root and check
that
data/train_data.csvexists. - Cannot write results: make sure your user can write to
results/andvisualizations/. docker: command not found: install Docker using the link in the Docker section.- Docker permission error on Linux: add your user to the
dockergroup, start a new login session, and check withdocker info. Do not make/var/run/docker.sockworld-writable. - Slow first Docker run:
docker compose up --builddownloads the Conda environment. Later runs can usedocker compose up. - Conda environment name conflict: remove the old environment with
conda env remove -n prediction-model-compare-prod, then create it again.