Skip to content

About

This is a part University's Assignment. The project aims to compare multiple models to gauge their performance in predicting the likelihood of an Insurance claim.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Insurance Claim Prediction

TL;DR

This project tests three machine-learning models to predict whether an auto insurance policyholder will file a claim:

  • Best model for finding claims: Logistic Regression
  • Claims found: 58.5% of actual claims (recall)
  • F1 score: 0.1546
  • ROC-AUC: 0.6215

The Decision Tree and Random Forest models have higher accuracy (about 85.6%), but they miss about 87%–89% of claims. Accuracy is misleading here because only about 6.4% of policies in the dataset have a claim. The current results come from the 20% holdout split and are saved in results/metrics.csv.

Model Accuracy Precision Recall F1 ROC-AUC Missed claims
Logistic Regression 59.0% 8.9% 58.5% 0.1546 0.6215 41.5%
Decision Tree 85.6% 7.4% 10.9% 0.0883 0.5079 89.1%
Random Forest 85.7% 8.5% 12.7% 0.1018 0.5761 87.3%

These are results for the supplied data and configuration, not a promise of future performance. Rerun the pipeline after changing the data, configuration, random seed, or library versions.

Contents

What this project does

The input is a CSV file containing insurance policy information. The model predicts the binary is_claim column:

  • 0: no claim
  • 1: claim

The project compares:

  1. Logistic Regression
  2. Decision Tree
  3. Random Forest

Because claims are uncommon, the project reports more than accuracy: precision, recall, F1, ROC-AUC, average precision, and false-negative rate. These measures show how many real claims the models find and how many they miss.

Project structure

data/
├── train_data.csv             # labeled training data
├── truncated_train_data.csv   # smaller data for quick checks
└── test_data.csv              # unlabeled data for predictions
src/
├── main.py                    # command-line entry point
├── config.py                  # application and model settings
├── pipeline.py                # training, evaluation, and exports
├── preprocessor.py            # feature conversion and encoding
├── data_loader.py             # CSV loading
├── model_factory.py           # model creation
├── models.py                  # model adapters and metrics
├── sampling_strategy.py       # class-balancing methods
├── scaler_factory.py          # feature scaling
└── visualizer.py              # chart generation
tests/                         # unit and end-to-end tests
results/                       # CSV and JSON results
visualizations/                # generated charts

Run commands from the repository root. The default training file is data/train_data.csv. It must contain a binary is_claim column.

Run the project

Choose Docker or a local Python environment. You only need one option.

Docker

Install Docker Desktop or Docker Engine with Docker Compose, then run:

docker compose up --build

The first build may take several minutes. Later runs can use:

docker compose up

Docker writes results/ and visualizations/ to the repository, so the files remain available after the container stops. Stop the container with:

docker compose down

Useful Docker commands:

# Run without creating charts
docker compose run --rm ml-pipeline python3 ./src/main.py --no-viz

# Train the models and score the supplied unlabeled data
docker compose run --rm ml-pipeline python3 ./src/main.py \
  --predict-data data/test_data.csv

Local Python with uv

Install Python 3.12 or newer and uv, then run:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -r requirements.txt
python src/main.py

Local Python with Conda

Install Conda, then create the environment:

conda env create -f environment.yml
conda activate prediction-model-compare-prod
python src/main.py

If the environment already exists, update it instead:

conda env update -f environment.yml --prune
conda activate prediction-model-compare-prod

Optional commands:

python src/main.py --no-viz
python src/main.py --viz-dir reports/visualizations
python src/main.py --predict-data data/test_data.csv
python src/main.py --help

Understand the pipeline

Each run follows these steps:

  1. Load and validate the training CSV.
  2. Convert Yes/No, numeric, ordinal, and categorical features.
  3. Remove the policy identifier.
  4. Split the data into an 80% training set and a 20% holdout set.
  5. Apply SMOTE to the training set to give the rare claim class more examples.
  6. Scale the features.
  7. Train the three models.
  8. Evaluate them on the untouched holdout set.
  9. Save metrics, predictions, and charts.

The split is stratified and uses random seed 11, so the results are repeatable. SMOTE is applied only to the training data; the holdout keeps the original claim rate for a fair evaluation.

The pipeline also checks different probability thresholds to find a threshold that improves F1. This analysis is reported in the results, but the default predictions file uses each model's normal prediction behavior.

Output files

After a normal run, check these files first:

results/
├── metrics.csv                 # metrics in a spreadsheet-friendly format
├── metrics.json                # same metrics in JSON format
├── predictions.csv             # holdout labels, predictions, and probabilities
└── inference_predictions.csv   # created only with --predict-data

The visualizer creates PNG reports in:

visualizations/
├── 01_data/             # class balance, correlations, distributions
├── 02_performance/      # metric comparisons and confusion matrices
├── 03_discrimination/   # ROC/PR curves and threshold analysis
├── 04_features/         # feature importance charts
└── 05_diagnostics/      # calibration and classification reports

predictions.csv is for the labeled holdout set and includes actual labels. inference_predictions.csv is different: it is created only when an unlabeled file is supplied with --predict-data.

The main performance chart is visualizations/02_performance/performance_comparison.png.

Configuration

Most settings are in src/config.py. Important defaults are:

Setting Default Meaning
data_path data/train_data.csv Training CSV
test_size 0.2 Holdout fraction
random_state 11 Repeatability seed
oversampling_method smote random, smote, adasyn, or none
scaler_type standard standard, minmax, robust, or none
validation_method holdout Holdout or cross-validation
threshold_metric f1 f1, recall, or precision
export_results True Write result files

The models and their hyperparameters are also defined in ModelConfig in src/config.py. Edit the configuration and run the command again to compare a different setup.

Run the tests

The canary Conda environment includes pytest:

conda env update -f environment-canary.yml --prune
conda activate prediction-model-compare-canary
pytest -q

The tests use the supplied sample data and do not need an external service.

Troubleshooting

  • Dataset not found: run the command from the repository root and check that data/train_data.csv exists.
  • Cannot write results: make sure your user can write to results/ and visualizations/.
  • docker: command not found: install Docker using the link in the Docker section.
  • Docker permission error on Linux: add your user to the docker group, start a new login session, and check with docker info. Do not make /var/run/docker.sock world-writable.
  • Slow first Docker run: docker compose up --build downloads the Conda environment. Later runs can use docker compose up.
  • Conda environment name conflict: remove the old environment with conda env remove -n prediction-model-compare-prod, then create it again.

About

This is a part University's Assignment. The project aims to compare multiple models to gauge their performance in predicting the likelihood of an Insurance claim.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages