Skip to content

commands.md

Latest commit

 

History

History
548 lines (402 loc) · 21 KB

commands.md

File metadata and controls

548 lines (402 loc) · 21 KB
Raw

Commands Reference

Current reference for the cleaned git-drs CLI.

Navigation: Getting Started -> Commands Reference -> Troubleshooting

Core Setup

git drs install

Install global Git filter configuration for git-drs.

git drs install

This sets the global filter.drs.* entries used by Git clean/smudge/filter operations.

git drs init

Initialize or repair repo-local git-drs wiring in the current repository.

git drs init [flags]

Common flags:

  • --transfers <n>: concurrent transfers
  • --upsert: enable upsert behavior for push/register flows
  • --multipart-threshold <mb>: multipart threshold in MB
  • --enable-data-client-logs: enable lower-level client logging

Use this when you want explicit initialization or to repair repo-local hooks/config. For normal onboarding, git drs remote add ... now bootstraps repo-local setup automatically when it is missing.

Remote Configuration

Repositories may commit canonical remote defaults in .git-drs/drs-policies.yaml. Clone-local drs.remote.<name>.* Git configuration overrides its endpoint and selection preference; local Globus routing is never accepted from the shared file. Before credentials are sent to an authenticated committed endpoint, confirm it with:

git config --local --add drs.trusted-endpoint https://drs.example.org

See Client Access-Method Selection Policy for the strict versioned schema.

git drs remote add [name] <endpoint-or-alias> [flags]

Add a DRS server with the unified remote command. The built-in aliases are calypr, terra, synapse, and cgc; inspect their non-secret defaults with git drs preset list or git drs preset show <alias>.

Calypr/Gen3 and Terra currently have operational runtime adapters. Synapse and CGC are catalog-only; execution commands reject them explicitly rather than constructing a partially authenticated client.

git drs remote add cgc --credential env:CGC_TOKEN
git drs remote add synapse --credential helper:synapse
git drs remote add terra --scope my-billing-project/my-workspace
git drs remote add https://drs.example.org --provider ga4gh --auth none
git drs remote add research https://gen3.example.org \
  --provider gen3 --scope PROGRAM/PROJECT \
  --auth provider-helper:gen3-profile --credential profile:research

For a preset, its alias becomes the default local name; for a URL, the name is derived from the host. Use the two-argument form to choose a name explicitly. --credential accepts env:VARIABLE, file:PATH, helper:NAME, or profile:NAME; it never accepts an inline secret. Other options are --scope, --auth, --provider, --storage, and --checkout. Checkout mode is pointers or hydrate.

Provider values are auto, ga4gh, gen3, terra, cgc, and synapse. Built-in aliases resolve auto to their catalog provider; a raw HTTPS endpoint must specify a concrete --provider before any repository state is modified. Authentication values are auto, none, bearer, basic, google-adc, provider-helper, or provider-helper:<name>. Presets are expanded once and the resolved endpoint, provider, authentication method, and catalog version are saved, so a later release cannot silently redirect an existing remote. Only HTTPS endpoints without embedded credentials are accepted.

git drs remote add gen3 [remote-name] <organization/project> remains a supported Gen3 setup command. It accepts --cred <credentials.json> or --token <access-token> and creates the local Gen3 profile. The local and terra command shapes are deprecated, hidden compatibility forms.

git drs preset list / git drs preset show <alias>

Display the presets embedded in this release. list shows each alias, provider, authentication default, endpoint, and catalog version. show displays the same details vertically and, when present, the registry service ID. Presets contain no credential or secret values.

Alias Provider Authentication default
calypr Gen3 Gen3 profile provider helper
terra Terra Google Application Default Credentials
synapse Synapse bearer token
cgc Cancer Genomics Cloud bearer token

The provider-specific remote add gen3 form remains available for Gen3 workflows. The remote add local and remote add terra forms remain only as hidden, deprecated compatibility commands.

git drs remote list

List configured git-drs remotes.

git drs remote list

git drs ping [remote-name]

Show the effective git-drs remote configuration and verify that the remote responds.

git drs ping
git drs ping anvil

What it checks:

  • prints the selected remote, remote type, endpoint, scope, bucket, storage prefix, and auth mode
  • runs a health check against the selected remote
  • for Gen3 and Terra DRS remotes, pings the GA4GH DRS service-info endpoint at <endpoint>/ga4gh/drs/v1/service-info
  • for Terra/AnVIL TDR-hosted data in production, use https://data.terra.bio as the endpoint; Terra also uses DRSHub for DRS URI resolution, but DRSHub is a resolver service rather than the GA4GH DRS service-info host
  • for scoped Syfon-style remotes, verifies that the configured organization/project and bucket are visible and readable

git drs auth globus [login|status|logout]

Manage the Globus Transfer API credential used for globus:// access methods. The Globus CLI is not required; git-drs uses globus-go-sdk directly.

export GIT_DRS_GLOBUS_CLIENT_ID='<native-application-client-id>'
git drs auth globus login
git drs auth globus status
git drs auth globus logout

login uses OAuth authorization code with PKCE, requests a refresh token, and stores the resulting credentials in the user configuration directory. Access tokens are refreshed automatically. Supply collection-dependent consent scopes when required:

git drs auth globus login --scope \
  'https://auth.globus.org/scopes/<collection-id>/data_access'

For noninteractive automation, GIT_DRS_GLOBUS_TRANSFER_TOKEN supplies a non-persistent access token and takes precedence over stored credentials.

Example Terra preset configuration and ping:

git drs remote add anvil terra --scope my-billing-project/my-workspace
git drs ping anvil

Terra credential configuration:

  • The terra preset selects google-adc, which means Application Default Credentials from the local Google Cloud environment.
  • Do not put Google access tokens, refresh tokens, service-account JSON, or other secrets in repo-local Git configuration or in --credential.
  • Configure ADC outside git-drs, for example with gcloud auth application-default login for an interactive user credential or by setting GOOGLE_APPLICATION_CREDENTIALS to a service-account key path in automation.
  • At runtime, Terra-aware resolution obtains Google credentials from the ADC provider chain; credential material is not written to committed repository files.
  • git drs ping checks the public DRS service-info endpoint without sending the ADC credential. Authenticated object resolution and access requests still use ADC.

A successful Terra ping includes:

remote: anvil (default)
type: terra
endpoint: https://data.terra.bio
health: ok

Troubleshooting:

  • no remote configuration found: run git drs remote list and pass an existing remote name, or add one with git drs remote add anvil terra.
  • endpoint errors: remove the remote and add it again with the terra preset or a complete HTTPS URL.
  • terra DRS service-info returned ...: verify the server is up and that the base endpoint is correct. You can test the exact URL with curl -i https://data.terra.bio/ga4gh/drs/v1/service-info.
  • network, DNS, or TLS errors: check VPN/proxy/firewall settings and retry with GIT_CURL_VERBOSE=1 git drs ping <remote-name> for additional HTTP diagnostics from Git-adjacent workflows.

For developers, the live Terra ping integration test is intentionally behind the integration build tag because it reaches the public Terra DRS service:

go test -tags=integration ./cmd/ping -run TestIntegrationPingTerraDRSServer -count=1

Set GIT_DRS_TERRA_DRS_ENDPOINT to point the test at a different Terra DRS deployment. Terra's DRSHub resolver URL follows the https://drshub.dsde-<env>.broadinstitute.org/api/v4/drs/resolve pattern used by terra-notebook-utils, but git drs ping needs the GA4GH DRS service base URL that exposes /ga4gh/drs/v1/service-info; for Terra production that service base URL is https://data.terra.bio.

git drs remote remove <remote-name>

Remove a configured git-drs remote.

git drs remote remove <remote-name>
git drs remote rm <remote-name>

This removes git-drs remote config, not normal Git remotes.

git drs remote set <remote-name>

Set the default git-drs remote.

git drs remote set production

Remote roles: primary and source authorities

A repository can have multiple git-drs remotes. The configured default remote, drs.default-remote, or an explicit command-level remote is the primary remote for repository-scoped operations such as push, registration, checksum lookup, and provider inspection.

A source DRS authority/resolver is the DRS service named by a drs://... URI or by source metadata recorded for a reference-first object. For git drs add-ref, the client resolves the input drs://... URI against that source authority/resolver using source credentials. The primary remote does not act as a proxy for other source DRS servers.

When the source authority is also configured as one of the repository remotes, git-drs can use that remote's endpoint and credentials for the source request. When the source authority is not configured as a named remote, the source URI itself remains the retrieval identity; adding the reference still does not create or mutate a primary-remote DRS record.

Tracking and Local Inventory

git drs track <pattern>

Track files or globs with git-drs pointer behavior.

git drs track "*.bam"
git drs track "data/**"

Stage .gitattributes after changing tracked patterns.

git drs untrack <pattern>

Stop tracking a pattern.

git drs untrack "*.bam"

git drs ls-files [path...]

List visible entries from the current directory or from the requested paths. Directory paths list their immediate contents. File paths list the file itself. The default listing includes untracked files and directories and omits hidden entries.

git drs ls-files
git drs ls-files results/
git drs ls-files --all
git drs ls-files --all UMB/
git drs ls-files -l
git drs ls-files --drs
git drs ls-files -I "*.bam"
git drs ls-files -n results/**

Important behavior:

  • default mode lists the current directory and does not query Git or DRS
  • path operands can name files or directories
  • hidden entries are omitted from directory listings
  • -a, --all lists every tracked DRS/Git-LFS pointer in the project using the original object ID, localization status, and path format
  • --pointers remains an alias for the pointer inventory
  • a directory operand in pointer mode includes tracked pointers beneath it
  • * means localized/hydrated in the worktree
  • - means the worktree still contains a pointer
  • --drs selects pointer inventory and adds DRS registration checks
  • pointer inventory flags such as --long, --json, -I, -n, and remote options also select pointer mode

Common flags:

  • -I, --include <pattern>: include filter; may be repeated
  • -a, --all: list every tracked DRS/Git-LFS pointer
  • -l, --long: show full object IDs in pointer mode
  • -n, --name-only: path-only output
  • --json: structured output
  • --drs: include DRS lookup details
  • -r, --git-remote: select the Git remote for pointer inventory
  • -d, --drs-remote: select the DRS remote for lookup
  • -h, --help: show command help

Hydration and Push

git drs pull

Hydrate tracked pointer files already present in the current checkout.

git drs pull
git drs pull -I "*.bam"
git drs pull -I data/
git drs pull -r -I data/
git drs pull -I "data/**" -I "results/*.txt"
git drs pull --dry-run -I "results/**"
git drs pull research --access-method globus

Directory includes select tracked DRS/Git-LFS files directly inside the directory. Use -r / --recursive to include files in its subdirectories. Include paths and glob patterns are relative to the current directory. Existing repository-relative paths are also accepted when there is no local match. Plain pull recursively selects tracked DRS/Git-LFS files beneath the current directory; from the repository root it selects the whole checkout.

Access-method policy:

  • auto is the default. It prefers HTTPS, then Globus among implemented data-plane handlers. S3 or GS access IDs may resolve to HTTPS.
  • --access-method <type> strictly requires that type for this pull. It does not fall back.
  • GIT_DRS_ACCESS_METHOD=prefer:<type> prefers a type but falls back to another ready method.
  • GIT_DRS_ACCESS_METHOD=require:<type> requires a type for commands that do not expose the flag as well as for pull.
  • A repository can set a per-remote default with git config --local drs.remote.<name>.access-method prefer:<type>.

The precedence is command flag, GIT_DRS_ACCESS_METHOD, legacy GIT_DRS_TRANSFER_PROVIDER, remote Git configuration, then auto. Bare method names in environment and Git configuration remain compatible aliases for prefer:<type>.

Important behavior:

  • git drs pull does not run git pull
  • it only hydrates tracked pointer files already present in the checkout
  • include matching is against repo-relative paths
  • access-method selection does not change pointer paths, cache paths, include matching, or checkout behavior
  • /access resolution is planning, so a preferred candidate may fall back when resolution or destination routing fails
  • git-drs does not switch providers after the first byte request or Globus task submission

See Client Access-Method Selection Policy for readiness states and diagnostics.

git drs push [remote-name]

Run the managed git-drs push path.

git drs push
git drs push production

What it does:

  • resolves local pointer/object metadata
  • discovers all newly reachable LFS/DRS pointer objects in the Git object graph
  • compares them with the remote refs/git-drs/synced/* acknowledgment ref
  • registers only missing scoped metadata
  • uploads only missing local payload bytes
  • advances the synchronization acknowledgment only after Git and DRS work succeeds
  • completes the Git push flow

Notes:

  • this is the normal command for tracked data changes
  • there is no provider-specific push command; Globus access methods do not change tracking, registration, upload, or Git push behavior
  • plain git push does not trigger git-drs registration or upload behavior
  • synchronization is history-derived; pointers deleted from the tip remain covered while reachable from Git history
  • the remote acknowledgment ref allows a later git drs push from another clone to recover after plain git push
  • unreachable-object cleanup is separate from normal push and is not performed implicitly

Provider/Object Reference Workflows

For details on pointer file formats and lifecycle state, see Pointer Files and Reference State.

git drs add-url <object-url-or-key> [path]

Create a pointer plus local DRS metadata for an object that already exists in provider storage.

git drs add-url path/to/object.bin data/from-bucket.bin --scheme s3
git drs add-url s3://my-bucket/path/to/object.bin data/from-bucket.bin
git drs add-url s3://my-bucket/path/to/object.bin data/from-bucket.bin --sha256 <hex>
git drs add-url globus://<collection-id>/project/sample.bam data/sample.bam
git drs add-url globus://<collection-id>/project/ data/project
git drs add-url 'globus://<collection-id>/project/**/*.bam' data/bam --dry-run

Notes:

  • object-key mode resolves against the configured bucket scope
  • explicit provider URL mode remains supported
  • --scheme is required for object-key mode
  • when --sha256 is omitted, the pointer uses a derived local/cache OID and source URL metadata remains the retrieval identity
  • globus:// URLs are detected automatically; a directory or wildcard import does not require --recursive or --manifest
  • --recursive is deprecated and accepted only for compatibility
  • quote wildcard URLs; *, ?, [...], and recursive ** are supported
  • Globus files without SHA-256 use a marked placeholder derived from URL, size, and last_modified; hydration validates size rather than treating it as a content hash
  • --manifest optionally supplies authoritative path, size, and sha256 values for a directory or wildcard selection
  • --dry-run prints planned Globus members without changing the worktree; --remote selects the DRS registration scope
  • collection import writes one DRS pointer and local DRS object per member, plus one destination/** read-only rule in .gitattributes
  • registration happens later on git drs push

See Adding Globus Objects for wildcard and identity details.

git drs add-ref <drs-id> <path>

Add a local pointer file for an existing DRS object. If the source DRS object has a SHA256 checksum, the pointer can use that checksum; otherwise the pointer preserves the source drs://... URI directly so hydration can resolve by DRS identity instead of checksum lookup.

git drs add-ref drs://example/object-id data/object.bin

add-ref also adds an exact read-only filter=drs rule for the destination to .gitattributes. Stage that file together with the pointer so local inventory commands such as git drs ls-files can discover the reference.

git drs query <drs-id>

Query a DRS object by ID or checksum.

git drs query drs://example/object-id
git drs query --checksum <sha256>

Delete and Copy Workflows

git drs rm <path>...

Remove tracked git-drs files from the worktree and index.

git drs rm data/sample.bam
git drs rm data/sample1.bam data/sample2.bam

What it does:

  • validates that each path is tracked as a git-drs pointer-managed file
  • runs git rm for those paths
  • leaves the remote DRS record and stored payload unchanged

Remote behavior on push:

  • git drs push removes the pointer from the Git tip but does not delete the DRS record or payload
  • historical DRS objects remain available while their pointers are reachable from Git history
  • use the explicit git drs delete command when destructive DRS deletion is intended

git drs copy-records [source-remote] <target-remote> <organization/project>

Copy Syfon metadata records from one configured remote to another for one scope.

git drs copy-records prod HTAN_INT/BForePC
git drs copy-records dev prod HTAN_INT/BForePC
git drs copy-records @local prod HTAN_INT/BForePC
git drs copy-records @local dev HTAN_INT/BForePC --include-path META --include-path CONFIG
git drs copy-records local dev HTAN_INT/BForePC --include-path META --include-path CONFIG

Behavior:

  • with one remote arg:
    • source defaults to the configured default remote
    • the arg is treated as the target remote
  • with two remote args:
    • first is source
    • second is target
  • copies metadata only, not object bytes
  • @local explicitly means the current repository's local records
  • the legacy local alias means repository-local only when no configured remote is named local
  • --include-path limits a copy to records matching a file or directory path; repeat it to select multiple paths
  • for indexd/Syfon sources, paths are resolved through the current repository's tracked files and source records are selected by SHA-256

Merge behavior for existing target records:

  • match by DID first, then by checksum where applicable
  • union controlled_access
  • union access_methods
  • preserve existing target metadata otherwise

Use --overwrite-existing when the source metadata must replace existing target metadata. This mode requires a Syfon target that supports PUT /index/bulk/overwrite; it matches a checksum sibling only inside the requested target project and preserves that target record's DID.

Bucket Mapping Commands

These are typically steward/admin setup commands, not normal day-to-day end-user commands.

git drs bucket add

Declare bucket credentials for a remote.

git drs bucket add-organization

Map an organization to a bucket path.

git drs bucket add-organization production \
  --organization HTAN_INT \
  --path s3://cbds/htan-int

git drs bucket add-project

Map a project to a bucket path.

git drs bucket add-project production \
  --organization HTAN_INT \
  --project BForePC \
  --path s3://cbds/htan-int/bforepc

Version

git drs version

Display version information.

git drs version

Removed Legacy Commands

These commands are gone from the cleaned CLI:

  • git drs fetch
  • git drs list
  • git drs upload
  • git drs download

If older notes mention them, treat those references as stale.