Reproducibility Assessment

Execute a query on arXiv and Semantic Scholar, download the matching papers, and manually quantify whether they are reproducible.

Quick Start

Docker

Everything (Python scraping/review as well as the R/LaTeX plot build) can be run inside a container, so nothing needs to be installed locally besides Docker itself.

make dev

Builds the image (Python, R, and a LaTeX toolchain) if needed and drops you into an interactive shell in the container, with the repository mounted. From there you can run make scrape, make continue-scan, make repro, etc. as usual.

make repro_docker

Builds the image and runs make repro inside the container non-interactively, reproducing the plots purely from the checked-in results/*.csv files (no scraping/manual review involved).

Both targets generate a .env file (via env.sh) so the container user maps to your host UID/GID, and files created in the mounted repository aren't owned by root.

Make Targets

Target	Description
make venv	Create .venv and install requirements.txt.
make scrape	Run the full query + download + text scan + interactive manual review.
make continue-scan	Resume the manual review CLI on already-fetched results, without re-querying/downloading.
make cleanup-pdfs	Remove PDFs from paper_pdfs that are no longer referenced in the bib data.
make repro	Build the plots from results/*.csv (delegates to plots/Makefile, requires R/Rscript and lualatex).
make dev	Build (if needed) and start an interactive Docker container with Python/R/LaTeX.
make repro_docker	Build the Docker image and run make repro inside a container.
make clean	Remove paper_pdfs and generated plot artifacts, keeps results/*.csv.
make distclean	clean plus removes .venv.

Scrape

make scrape

(equivalent to .venv/bin/python -m src.main -dd paper_pdfs -r results)

Downloads all PDFs that correspond to the query configuration, puts them into the paper_pdfs directory, and conducts a text-based search in each PDF for keywords that indicate a reproduction package (see configuration params in src/main.py).

Alongside, a BibTeX file references.bib is created, based on a journal reference if one is provided, otherwise on the arXiv DOI. references.csv contains a list of the bib keys, together with the arXiv/Semantic Scholar ID and further metadata.

The current content of references{.csv,.bib} was obtained with the following query for arXiv:

(cat:quant-ph) AND (((ti:"quantum computing" OR ti:"quantum_computer") AND (ti:"algorithm" OR ti:"software") AND (ti:"NISQ")) OR ((abs:"quantum computing" OR abs:"quantum_computer") AND (abs:"algorithm" OR abs:"software") AND (abs:"NISQ"))) AND submittedDate:[20210101 TO 20260327] ANDNOT (ti:"survey" OR ti:"review" OR abs:"survey" OR abs:"review")

and the following parameters for the Semantic Scholar API call:

"query": '(("quantum computing" | "quantum_computer") + ("algorithm" | "software") + ("NISQ")) + (- "survey" + -"review"),
"publicationDateOrYear": '2021-01-01:2026-03-27',
"fieldsOfStudy": 'Physics,Mathematics,Computer Science,Engineering',
"publicationTypes": 'JournalArticle,CaseReport,Conference,Study',

Running the text filter for the reproduction keywords (specified in main.py) on these queries gives:

218 out of 424 potentially experimental papers have some indication for a reproduction package.

127 out of 249 potentially experimental and peer-reviewed papers have some indication for a reproduction package.

66 out of 127 potentially experimental and peer-reviewed papers that were found on both arXiv and Semantic Scholar have some indication for a reproduction package.

42 out of 66 manually scanned papers do actually contain some kind of reproduction package.

31 out of 42 do actually contain code, of which 28 have some kind of documentation and 20 have some kind of environment specification.

The result data and metadata are stored in the results_data.csv and results_meta.csv files.

Want to continue the manual scan with existing results?

make continue-scan

(equivalent to .venv/bin/python -m src.main -s none -l -dd paper_pdfs -r results)

What happens when calling main.py?

1. The query above is executed on arxiv, semantic_scholar, or both (see the -s argument below). This step is skipped when -s none is passed, e.g. because the query has already been run and the PDFs, bib file (references.bib), and bib metadata (references.csv) already exist.

2. Duplicates in the bib data are removed based on matching title and authors; entries with a non-arXiv DOI are prioritised. With -c, duplicate PDFs are deleted too.

3. A text-based search is conducted on the remaining PDFs. Papers are classified as "experimental" if certain keywords occur in the text (see EXPERIMENTAL_PAPER_KEYWORDS in main.py), as "review" if certain keywords occur in the title or in the first 1500 characters (see REVIEW_PAPER_KEYWORDS), or as "unknown" otherwise.

4. The same search also looks for keywords indicating a reproduction package, such as known platforms (PLATFORMS in main.py) or code keywords (CODE_KEYWORDS).

5. Two CSV files are created:

   • results_meta.csv, with columns:
     • bib_key: key of the bib entry in references.bib.
     • id: arXiv or Semantic Scholar ID.
     • source: "arxiv", "semantic_scholar", or "both".
     • downloaded_pdf: whether a PDF was downloadable (currently always True, since closed-access papers aren't stored).
     • has_doi: whether the paper has a non-arXiv DOI (indicates peer-review).
     • paper_class: "experimental", "review", or "unknown" (see step 3).
     • has_repro_hint: whether a reproduction-package hint was found (see step 4).
   • results_data.csv, with columns:
     • bib_key, id, source, has_doi: same as above. Note that this file can contain multiple rows per paper — one per matching hint.
     • repro_hint_source: "platform" or "keyword" (False if none was found).
     • repro_hint: the exact platform or keyword that matched (False if none was found).
     • repro_hint_line: line number of the hint in the PDF (False if none was found).
     • repro_hint_text: the full line of text containing the hint (False if none was found).

6. Based on the papers classified "experimental" and the has_doi / has_repro_hint flags, an interactive shell opens for a manual scan, filtering out false positives (e.g. an unrelated GitHub repository mentioned in the references, or a paper misclassified as "experimental" when it's actually conceptual). Options:

   • [y] yes — there is a reproduction package (a link in the hint text is saved automatically).
   • [n] no — there is no reproduction package.
   • [u] unclear — opens the PDF for a closer look, then repeats the prompt.
   • [q] quit — stops the scan; intermediate results are kept.
   • [l] yes, manually enter link — reproduction package exists but the link in the hint text is partial or missing.
   • [i] ignore paper — skip to the next paper.
   • [c] re-classify paper — if the paper isn't actually "experimental" (e.g. "conceptual", "thesis").

   Results are stored in results_manual.csv. Previously scanned results are loaded by default; use -os to start from scratch.

7. A further CLI prompt asks, per reproduction repository, whether it has: code available, an environment spec, documentation, a version/release tag, a Docker image, hardware specification (experimental_params), and an executable. Options: [y] yes / [n] no / [p] partially / [u] unknown / [na] not applicable.

Arguments

(Output of .venv/bin/python -m src.main -h)

usage: ArXiv & SemanticScholar Scraper [-h] [-rn REF_NAME] [-dd DOWNLOAD_DIR] [-r RESULTS_DIR] [-mr MAX_RESULTS] [-s {arxiv,semantic_scholar,both,none}] [-nd] [-c] [-l] [-rd RESULT_DATA_FILE] [-res RESULT_META_FILE] [-os] [-rs RESULT_MANUAL_FILE]
                                       [-or] [-rr RESULT_REPRO_FILE]

Fetches bib entries and PDFs from arXiv and SemanticScholar.

options:
  -h, --help            show this help message and exit
  -rn, --ref_name REF_NAME
                        Bib and corresponding CSV file name (without file ending) defaults to 'references', which ends up as 'references.bib' and 'references.csv'.
  -dd, --download_dir DOWNLOAD_DIR
                        Directory to which to store PDFs, defaults to 'paper_pdfs'.
  -r, --results_dir RESULTS_DIR
                        Directory to which to store result CSV/BIB, defaults to 'results'.
  -mr, --max_results MAX_RESULTS
                        Maximum results per query. Default is unlimited (only applies to ArXiv)
  -s, --source {arxiv,semantic_scholar,both,none}
                        Source from which to download.
  -nd, --no_download    If provided, no PDFs are downloaded.
  -c, --cleanup_pdfs    If provided, PDFs are deleted that are not in bib data.
  -l, --load_existing_results
                        Whether to load result data from existing result files.
  -rd, --result_data_file RESULT_DATA_FILE
                        File to which result data get stored. Defaults to results_data.csv.
  -res, --result_meta_file RESULT_META_FILE
                        File to which meta and summary data of the results get stored. Defaults to results_meta.csv.
  -os, --overwrite_manual_results
                        Whether to overwrite already existing manually scan results for reproduction packages
  -rs, --result_manual_file RESULT_MANUAL_FILE
                        File to which results for the manual scan are stored.Defaults to results_manual.csv.
  -or, --overwrite_repro_results
                        Whether to overwrite already existing repository scan results for reproduction packages
  -rr, --result_repro_file RESULT_REPRO_FILE
                        File to which results for the specific repository repro scan are stored. Defaults to results_repro.csv.