MIAF

MIAF is a Modular Infrastructure-Aware Fusion framework for malicious domain and URL detection. It is designed for research and reproducible experiments where domain strings and infrastructure metadata must be studied separately and together.

The framework keeps five components separated:

1. String encoder: character models, PLM/URLBERT-style encoders.
2. Metadata encoder: infrastructure feature projections.
3. Fusion operator: concat, gated, and gated-add metadata fusion.
4. Optimization objective: cross-entropy, focal loss, logit-adjusted loss, class weights, and samplers.
5. Interpretation module: prediction explanations, counterfactual metadata analysis, and export utilities.

This repository is a clean package migration from an earlier experimental codebase. The current focus is a clear CLI and reusable modules rather than a direct copy of legacy scripts.

Status

MIAF provides a working research framework with:

• canonical dataset registry
• Hugging Face dataset download patterns
• local split verification
• dataset inspection CLI
• lightweight training dry-runs
• real training MVP for meta-only and char-cnn
• metadata fusion through the clean component interfaces
• interpretation and export utilities
• focused tests for migrated functionality

The roadmap is to keep extending the same clean interfaces for PLM checkpoint workflows, full experiment sweeps, and richer interpretation reports.

Technologies & Tools Used

MIAF relies on the following core frameworks and libraries:

• Deep Learning Framework: PyTorch & PyTorch Lightning for neural string encoders, metadata fusion architectures, and reproducible training loops.
• Pretrained Language Models & Hub: Hugging Face Transformers & huggingface_hub for transformer-based encoders (e.g. PLM / URLBERT) and dataset distribution.
• Machine Learning & Data Processing: scikit-learn for data preprocessing, tabular feature scaling, and metrics; Pandas & NumPy for dataset manipulation and numeric computing.
• Metrics & Experiment Tracking: TorchMetrics for model evaluation and Weights & Biases (W&B) for run logging and visualization.
• Testing & Quality Assurance: PyTest for unit and integration testing.

Installation

For development:

pip install -e .

Optional dataset download support:

pip install -e '.[datasets]'

Optional training extras:

pip install -e '.[training]'

Datasets

The canonical dataset source is the Hugging Face dataset repository:

ahmedBargady/DomMalPhi_Datasets_Infra

Each registered dataset contains:

• train.csv
• dev.csv
• test.csv
• train_ip.csv
• dev_ip.csv
• test_ip.csv

Set a default local dataset root with:

export MIAF_DATA_ROOT=/path/to/DomMalPhi_Datasets_Infra

Or pass --root to each command.

List registered datasets:

miaf datasets list

Resolve and verify a local dataset path:

miaf datasets path hranicky-malware --root /path/to/DomMalPhi_Datasets_Infra
miaf datasets verify hranicky-malware --root /path/to/DomMalPhi_Datasets_Infra

Download one registered dataset with narrow Hugging Face allow_patterns:

miaf datasets download hranicky-malware --root data/DomMalPhi_Datasets_Infra

Preview download patterns without network access:

miaf datasets download hranicky-malware --dry-run

Inspect split sizes, columns, labels, and metadata columns:

miaf inspect data hranicky-malware --root /path/to/DomMalPhi_Datasets_Infra
miaf inspect data hranicky-malware --root /path/to/DomMalPhi_Datasets_Infra --json

Training CLI

Build a normalized training configuration without importing heavy training dependencies or starting training:

miaf train --dataset hranicky-malware --encoder char-cnn --fusion concat --dry-run
miaf train --print-config --encoder meta-only --metadata all --no-wandb

Run the first real-training MVP paths:

miaf train \
  --dataset hranicky-malware \
  --encoder meta-only \
  --metadata all \
  --epochs 5 \
  --no-wandb \
  --run-dir runs/meta_only

miaf train \
  --dataset hranicky-malware \
  --encoder char-cnn \
  --metadata all \
  --fusion concat \
  --loss cross-entropy \
  --epochs 5 \
  --no-wandb \
  --run-dir runs/char_cnn

Use --data-dir to point directly at any directory containing train.csv, dev.csv, and test.csv.

Validation

Run the test suite:

python -m compileall miaf scripts tests
python -m pytest tests
python -m miaf.cli --help
python -m miaf.cli datasets list

Training smoke tests are skipped automatically when optional training dependencies are unavailable.

Repository layout

miaf/data        split loading, metadata column selection, PyTorch datasets
miaf/datasets    dataset registry, download patterns, verification
miaf/components  metadata encoders, fusion operators, objectives
miaf/models      string encoders and PLM wrappers
miaf/training    builders, Lightning module, runner, artifacts
miaf/interpret   online interpretation and visualization utilities
scripts          compatibility entry points
tests            focused migration tests

License

MIAF is released under the MIT License. You may use, modify, and distribute it for research and projects, including derivative work, subject to the license terms.

Citation

If MIAF supports your research, please cite the associated paper or repository once a formal citation is available.