Hierarchical IPC Patent Classification

This repository contains a PyTorch implementation of hierarchical multi-label patent classification by fine-tuning sentence transformers (BERT-based models). This codebase classifies patent abstracts into the International Patent Classification (IPC) hierarchy with parent-child constraints between classification levels.

Features

• Hierarchical Classification: Three-level classification (Section → Class → Subclass) with automatic parent-child constraint enforcement
• Multi-label Support: Patents can belong to multiple categories at each level
• SciBERT Integration: Uses domain-specific SciBERT model for better patent text understanding
• Mixed Precision Training: FP16 support for faster training on compatible GPUs
• Soft/Hard Masking: Flexible constraint application during training and inference
• valuation: Multi-level metrics including F1, precision, and recall

Architecture

The model uses a hierarchical approach where:

1. Section predictions are made independently
2. Class predictions are constrained by confident section predictions
3. Subclass predictions are constrained by confident class predictions

This ensures taxonomically valid predictions that respect the IPC hierarchy.

Structure

.
├── config.py                 # Configuration settings
├── data_processing.py        # Data loading and preprocessing
├── model.py                  # Hierarchical classifier architecture
├── train.py                  # Training script
├── predict.py                # Inference script
├── training_job.sh           # SLURM job script for training
├── prediction_job.sh         # SLURM job script for prediction
├── pyproject.toml            # Project metadata and dependencies
├── conda-env.yaml            # Conda environment specification
├── .pre-commit-config.yaml   # Pre-commit hooks configuration
└── data/
    ├── train.csv            # Training data
    ├── val.csv              # Validation data
    ├── test.csv             # Test data
    └── full_ipc_combined.csv # IPC hierarchy metadata

Getting Started

Prerequisites

• Python 3.10+
• PyTorch 2.0+
• Transformers 4.53+
• CUDA-capable GPU

Installation

1. Clone the repository

git clone https://github.com/rcgsheffield/ipc-classifier.git
cd ipc-classifier

2. Create a conda environment

conda env create -f conda-env.yaml
conda activate ai-innovation

3. Install additional dependencies

pip install transformers datasets

4. Install optional developer dependencies:

pip install -e .[dev]
pre-commit install

Data Format

• The CSV files (train.csv, test.csv, val.csv) should have the following columns:

• appln_abstract: Patent abstract text

• section: List of section codes (e.g., "['A', 'B']")

• class: List of class codes (e.g., "['A01', 'B32']")

• subclass: List of subclass codes (e.g., "['A01B', 'B32B']")

The IPC metadata file (full_ipc_combined.csv) should contain:

• section_code: Section identifier
• class_code: Class identifier
• subclass_code: Subclass identifier

Training

Local training:

python train.py

Submit a job on the HPC

sbatch training_job.sh

Configuration

Key parameters in config.py:

• model_name: Pre-trained model (default: "allenai/scibert_scivocab_uncased")
• batch_size: Training batch size (default: 32)
• learning_rate: Learning rate (default: 1e-5)
• epochs: Number of training epochs (default: 20)
• parent_confidence_threshold: Minimum confidence for parent constraints (default: 0.2)
• prediction_threshold: Level-specific prediction thresholds
• use_fp16: Enable mixed precision training (default: True)

Inference

Pretrained weights

The trained model is archived on Figshare (version 2 onwards) at 10.15131/shef.data.34014453. Download these files:

• scibert_hf1_0.389_best_model.pt: fine-tuned SciBERT weights (hierarchical F1 0.389 on validation data)
• scibert_hf1_0.389_processor.pkl: IPC label encoders and hierarchy mappings (required to decode predictions)
• SHA256SUMS: checksums to verify the downloads (sha256sum -c SHA256SUMS)

Place both model files in a checkpoints/ folder at the repository root. predict.py loads them by these exact filenames.

Security note: .pt and .pkl files can execute arbitrary code when loaded. Only use files from the Figshare deposit above.

Run predictions

python predict.py --input your_data.csv --output predictions.csv

The input CSV must contain a full_description column. The first column (or appln_id / gtr_proj_id, if present) is used as the ID. Without --input, data/test.csv is used. Two files are written: predictions_threshold.csv and predictions_topk.csv.

Submit a job on the HPC

sbatch prediction_job.sh

The script will:

1. Load the best saved model from ./checkpoints/
2. Process test data with hierarchical constraints
3. Save predictions to predictions.csv

Output Format

The predictions CSV contains:

• index: Sample index

• abstract: Patent abstract text

• For each level (section/class/subclass):

  • true_{level}: Ground truth labels
  • {level}_predicted: Predicted labels
  • {level}_probabilities: Prediction confidence scores

Code Quality

• This project uses pre-commit hooks for code quality:

• Black: Code formatting

• isort: Import sorting -Pylint: Code linting