ADMETBench

Rigorous Molecular Property Prediction Benchmark Suite

ADMETBench is a small, reproducible benchmark for evaluating molecular property prediction models on ADMET endpoints — Absorption, Distribution, Metabolism, Excretion, and Toxicity — the properties that determine whether a promising compound survives drug development. It ships scaffold-aware data splitting, two baseline model families (classical and graph neural), and uncertainty calibration metrics, so that reported numbers reflect genuine generalization rather than train/test leakage or overconfident point estimates.

---

Table of Contents

• Motivation
• Features
• Installation
• Quick Start
• Tech Stack
• Repository Structure
• Leaderboard
• Datasets
• Contributing
• Citation
• License

---

Motivation

Published molecular property prediction benchmarks are routinely undermined by three quiet failure modes:

1. Random splits leak information. Structural analogs of a training molecule frequently end up in the test set too, so a model can "memorize" a chemical scaffold rather than learn a generalizable structure-property relationship. Reported accuracy is inflated and collapses on truly novel chemotypes.
2. Classical baselines are under-reported. A Morgan fingerprint plus a Random Forest is cheap to train and frequently matches or beats a poorly tuned graph neural network on small ADMET datasets — but papers proposing new architectures don't always include it as a baseline.
3. Point estimates hide overconfidence. ADMET predictions inform irreversible, expensive decisions (which compound to synthesize, which to advance). A model that is confidently wrong is more dangerous than one that is honestly uncertain, yet most benchmarks report only accuracy/RMSE and never calibration.

ADMETBench addresses all three: every model is evaluated with a Bemis-Murcko scaffold split, a strong classical baseline is included by default, and every result is accompanied by an Expected Calibration Error (ECE) (or its regression analogue) alongside standard metrics.

Features

• Five curated ADMET endpoints: hERG cardiotoxicity, aqueous solubility, CYP3A4 inhibition, Caco-2 permeability, and hepatic clearance.
• Scaffold-split utilities (utils/scaffold_split.py) implementing Bemis-Murcko scaffold extraction and balanced train/valid/test partitioning, so leaderboard numbers reflect out-of-distribution generalization.
• Two baseline model families, both implementing a common ADMETModel interface (models/base.py):
  • Morgan Fingerprint + Random Forest (models/fingerprint_rf.py) — the classical cheminformatics baseline, with RDKit ECFP4 fingerprints and scikit-learn.
  • Message Passing Neural Network (MPNN) (models/mpnn.py) — a directed message-passing graph neural network built in pure PyTorch (no PyTorch-Geometric dependency), with MC-Dropout uncertainty estimation.
• Uncertainty calibration metrics (utils/metrics.py): Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) for classification; an analogous predictive-interval calibration error for regression.
• Reproducible CLI pipeline: evaluate.py trains/evaluates a (model, dataset) pair with a single command; leaderboard.py aggregates results into a Markdown/CSV leaderboard.
• Offline-friendly: datasets fall back to a chemically valid, synthetic offline generator when DeepChem/MoleculeNet downloads are unavailable, so the full pipeline (including tests) runs without network access.
• Test suite covering scaffold splitting correctness, calibration metric correctness, and both baseline models end-to-end.

Installation

git clone https://github.com/islamomar/admetbench.git
cd admetbench

python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate

pip install -r requirements.txt
pip install -e .                # installs the admetbench package + CLI entry points

Requirements: Python 3.9+. RDKit and DeepChem are the two heaviest dependencies; RDKit wheels are available on PyPI for all major platforms, and DeepChem is optional in the sense that ADMETBench degrades gracefully to an offline synthetic-data generator if MoleculeNet downloads are unavailable.

Quick Start

Run a single baseline on a single dataset:

python evaluate.py --dataset herg --model morgan_rf
python evaluate.py --dataset solubility --model mpnn --epochs 40

Run every baseline on every dataset and build the leaderboard:

python evaluate.py --all --force-synthetic   # drop --force-synthetic once you have network access to MoleculeNet
python leaderboard.py

Or explore interactively:

jupyter notebook notebooks/01_quickstart.ipynb

Minimal Python usage:

from data.datasets import load_dataset
from utils.scaffold_split import scaffold_split
from models import MorganFingerprintRF
from utils.metrics import classification_metrics

df = load_dataset("herg")
smiles, y = df["smiles"].tolist(), df["y"].to_numpy()

train_idx, valid_idx, test_idx = scaffold_split(smiles, 0.8, 0.1, 0.1, seed=42)
model = MorganFingerprintRF(task_type="classification")
model.fit([smiles[i] for i in train_idx], y[train_idx])

preds, uncertainty = model.predict([smiles[i] for i in test_idx])
print(classification_metrics(y[test_idx], preds))

Tech Stack

Layer	Library
Molecular featurization	RDKit (Morgan fingerprints, Bemis-Murcko scaffolds, atom/bond descriptors)
Dataset loading	DeepChem / MoleculeNet, with offline synthetic fallback
Graph neural network	PyTorch (custom MPNN, no PyG dependency)
Classical baseline	scikit-learn (Random Forest)
Data wrangling	pandas, NumPy, SciPy
Testing	pytest

Repository Structure

admetbench/
├── README.md
├── LICENSE
├── requirements.txt
├── setup.py
├── .gitignore
├── evaluate.py                # CLI: train + evaluate a (model, dataset) pair
├── leaderboard.py              # CLI: aggregate results into a leaderboard table
├── data/
│   ├── __init__.py
│   ├── datasets.py             # dataset registry + loaders (MoleculeNet / offline synthetic)
│   └── raw/                    # cached CSVs land here (gitignored)
├── models/
│   ├── __init__.py
│   ├── base.py                 # ADMETModel abstract interface
│   ├── fingerprint_rf.py       # Morgan fingerprint + Random Forest baseline
│   └── mpnn.py                 # PyTorch MPNN baseline with MC-Dropout uncertainty
├── utils/
│   ├── __init__.py
│   ├── scaffold_split.py       # Bemis-Murcko scaffold splitting
│   └── metrics.py              # ECE / MCE / regression calibration + standard metrics
├── notebooks/
│   └── 01_quickstart.ipynb
├── tests/
│   ├── test_scaffold_split.py
│   ├── test_metrics.py
│   └── test_models.py
└── results/                    # evaluate.py / leaderboard.py output (gitignored)

Leaderboard

Reference numbers below were produced with python evaluate.py --all --force-synthetic using the bundled offline chemically-valid demo data (see Datasets) — they demonstrate that the pipeline runs end-to-end and calibration metrics behave sensibly, but they are not a substitute for benchmarking on curated, literature-sourced ADMET data. Regenerate this table on your own data with:

python evaluate.py --all
python leaderboard.py

hERG (cardiotoxicity, classification) — ranked by AUROC

Rank	Model	Accuracy	AUROC	ECE	MCE
1	mpnn	0.803	0.783	0.110	0.273
2	morgan_rf	0.590	0.699	0.137	0.176

CYP3A4 (enzyme inhibition, classification) — ranked by AUROC

Rank	Model	Accuracy	AUROC	ECE	MCE
1	morgan_rf	0.767	0.938	0.162	0.332
2	mpnn	0.900	0.852	0.152	0.356

Solubility (logS, regression) — ranked by RMSE

Rank	Model	MAE	RMSE	R²	Calibration Error
1	mpnn	1.100	1.415	0.848	0.317
2	morgan_rf	2.671	3.291	0.177	0.233

Caco-2 (permeability, regression) — ranked by RMSE

Rank	Model	MAE	RMSE	R²	Calibration Error
1	mpnn	0.767	0.983	0.633	0.404
2	morgan_rf	1.512	1.916	-0.394	0.223

Clearance (hepatic clearance, regression) — ranked by RMSE

Rank	Model	MAE	RMSE	R²	Calibration Error
1	mpnn	0.831	1.029	0.087	0.468
2	morgan_rf	0.862	1.106	-0.056	0.263

Lower ECE/MCE/Calibration Error is better; these small demo splits (~60 test molecules) make the numbers noisy — treat them as a pipeline sanity check, not a scientific conclusion. Interestingly, the Random Forest is competitive on some classification tasks and less well-calibrated on regression tasks in this run, illustrating exactly why ADMETBench reports calibration alongside accuracy: neither baseline dominates uniformly.

Datasets

Endpoint	Task type	Description
herg	Classification	hERG potassium channel inhibition (cardiotoxicity liability)
solubility	Regression	Aqueous solubility, log mol/L (Delaney/ESOL-style)
cyp3a4	Classification	CYP3A4 enzyme inhibition (drug-drug interaction risk)
caco2	Regression	Caco-2 cell monolayer permeability, log Papp
clearance	Regression	Hepatic clearance, log(mL/min/kg)

data/datasets.py first attempts to load each dataset via DeepChem's MoleculeNet loaders. If that fails (no network, DeepChem not installed, or no native loader exists for that endpoint), it transparently falls back to an offline synthetic generator: RDKit-valid SMILES drawn from a curated pool of drug-like fragments, with labels derived as a noisy function of molecular descriptors (LogP, molecular weight, H-bond donors/acceptors). This keeps the full pipeline — including CI and the test suite — runnable with zero network access, while making it a one-line change (use_cache=False, force_synthetic=False, or dropping a curated CSV into data/raw/<name>.csv) to switch to real, literature-sourced ADMET data for production benchmarking.

Contributing

Contributions are welcome — new baselines, new datasets, better calibration diagnostics, and bug fixes are all in scope.

1. Fork the repository and create a feature branch: git checkout -b feature/my-improvement
2. Set up your environment: pip install -r requirements.txt && pip install -e .
3. Add tests for any new functionality in tests/. All new models should include at least a small fit/predict smoke test (see tests/test_models.py for the pattern).
4. Run the test suite before opening a PR: pytest tests/ -v
5. Adding a new baseline: implement the ADMETModel interface in models/base.py (fit and predict), register it in models/__init__.py's MODEL_REGISTRY, and it will automatically be picked up by evaluate.py --all and leaderboard.py.
6. Adding a new dataset: add a DatasetSpec entry to DATASET_REGISTRY in data/datasets.py, including a synth_fn offline fallback so the pipeline remains runnable without network access.
7. Style: format with black and keep functions documented with docstrings in the existing style.
8. Open a pull request describing the change, the motivation, and (for new baselines/datasets) any reference results.

Please open an issue first for larger changes (new architectures, breaking API changes) so the design can be discussed before implementation work begins.

Citation

If you use ADMETBench in your research, please cite this repository, and please also cite the paper that established the scaffold-split evaluation methodology and directed-MPNN architecture this benchmark builds on:

@article{yang2019analyzing,
  title={Analyzing Learned Molecular Representations for Property Prediction},
  author={Yang, Kevin and Swanson, Kyle and Jin, Wengong and Coley, Connor
          and Eiden, Philipp and Gao, Hua and Guzman-Perez, Angel and
          Hopper, Timothy and Kelley, Brian and Mathea, Miriam and
          Palmer, Andrew and Settels, Volker and Jaakkola, Tommi and
          Jensen, Klavs and Barzilay, Regina},
  journal={Journal of Chemical Information and Modeling},
  volume={59},
  number={8},
  pages={3370--3388},
  year={2019},
  publisher={American Chemical Society},
  doi={10.1021/acs.jcim.9b00237}
}

@software{admetbench2026,
  title={ADMETBench: Rigorous Molecular Property Prediction Benchmark Suite},
  author={Omar, Islam},
  year={2026},
  url={https://github.com/islamomar/admetbench}
}

License

Released under the MIT License. Copyright (c) 2026 Islam Omar.

ADMETBench