feat: add PyPI package metadata collector - #3165
Closed
ChrisJr404 wants to merge 1 commit into
Closed
ChrisJr404 wants to merge 1 commit into
ChrisJr404 wants to merge 1 commit into
Conversation
Adds a collector that reads pypi purls from a collect data source and fetches each package's metadata from the PyPI JSON API (https://pypi.org/pypi/<name>/json), emitting one document per package. Mirrors the deps.dev collector: supports a one-shot run or a poll loop, dedupes packages already fetched, and skips non-pypi sources. Introduces a DocumentPyPI document type for the emitted documents. The processor/parser side is left as a follow-up; this change is the collector only. Tested with an httptest server standing in for the PyPI API against testdata, covering purl and bare-name inputs, dedupe, non-pypi skips, and a missing package. Refs guacsec#208 Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
ChrisJr404
force-pushed
the
collector-pypi
branch
from
August 18, 2026 05:30
1719477 to
5cd1cb3
Compare
mihaimaruseac
approved these changes
Sep 1, 2026
Member
|
Hi @ChrisJr404, closing this PR since this is not a change planned in the project roadmap. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a PyPI collector so GUAC can pull package metadata from the PyPI JSON API and feed it into the supply-chain graph, closing the gap for pypi purls that the deps.dev collector only covers indirectly.
It reads pypi purls from a collect data source and fetches each package's metadata from
https://pypi.org/pypi/<name>/json, emitting oneprocessor.Documentper package. I modeled it on the deps.dev collector:RetrieveArtifactsruns once or loops on an interval whenpollis set, packages already fetched are deduped across poll cycles, and non-pypi sources are skipped with a warning. Bare package names work too, not just purls. A newDocumentPyPIdocument type tags the emitted docs.Scope here is the collector only. The processor/parser side (registering a document parser + guesser for
DocumentPyPI) is a natural follow-up and I'm happy to send that separately.Tests spin up an
httptestserver standing in for the PyPI API and serve fixtures fromtestdata/, so they run without network. Cases cover a single purl, a bare name, dedupe of a repeated package, a non-pypi purl getting skipped, and a missing package emitting nothing.go build ./...,go vet, andgo test ./pkg/handler/collector/pypi/...all pass.Refs #208