-
Notifications
You must be signed in to change notification settings - Fork 678
feat: introduce SNYK_REQUEST_CONCURRENCY for dependency request parallelism #6756
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -111,6 +111,28 @@ export type FailOn = 'all' | 'upgradable' | 'patchable'; | |
| export const RETRY_ATTEMPTS = 3; | ||
| export const RETRY_DELAY = 500; | ||
|
|
||
| const DEFAULT_REQUEST_CONCURRENCY = 10; | ||
| const MIN_REQUEST_CONCURRENCY = 1; | ||
| const MAX_REQUEST_CONCURRENCY = 50; | ||
|
|
||
| /** | ||
| * Returns the maximum number of in-flight Snyk dependency-test or | ||
| * dependency-monitor HTTP requests permitted at once. Override with the | ||
| * SNYK_REQUEST_CONCURRENCY environment variable; values are clamped to | ||
| * [MIN_REQUEST_CONCURRENCY, MAX_REQUEST_CONCURRENCY]. | ||
| */ | ||
| export function getRequestConcurrency(): number { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion: There is already a similar configuration value in the Golang CLI. While this is currently not publicly documented, I think we should inject it in the Legacy CLI and have the Golang configuration a single source of truth for this. We can also expose the config value but would probably want to frame it around
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the pointers. I didn't know about this config value. I agree having a single source of truth here makes sense. Before I plumb it through, I want to flag a semantic concern.
What this PR controls is HTTP request concurrency — how many in-flight Binding HTTP concurrency to A few options I see:
Am I understanding things correctly? What are you preferences from a CLI architecture perspective?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think semantically and technically you are right and point 3 would make sense. Besides this, there is a usability question, as a user specifying Architecturally Application configuration via Environment Variables and configuration handling is done through the Golang CLI and its configuration package. This way, the decision on where values are coming from (Env var, config file, cmd arg ..) is left to the application. Making this available in the Typescript CLI via internal environment variables makes sense though.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there an option to separate concerns and decouple the configurability and the increase of the default value? |
||
| const raw = process.env.SNYK_REQUEST_CONCURRENCY; | ||
| if (!raw) { | ||
| return DEFAULT_REQUEST_CONCURRENCY; | ||
| } | ||
| const parsed = parseInt(raw, 10); | ||
| if (!Number.isFinite(parsed) || parsed < MIN_REQUEST_CONCURRENCY) { | ||
| return DEFAULT_REQUEST_CONCURRENCY; | ||
| } | ||
| return Math.min(parsed, MAX_REQUEST_CONCURRENCY); | ||
| } | ||
|
|
||
| /** | ||
| * printDepGraph writes the given dep-graph and target name to the destination | ||
| * stream as expected by the `depgraph` CLI workflow. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thinking: Generally changing a default without visibility in the behaviour, we currently don't have any metrics on CPU consumption, makes me wonder if we should be more careful. The impact is quite huge as it affects all SCA and Container scans.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think there'll be significant CPU impact. Node executes JS on a single thread, and the code path this PR touches spends nearly all its time waiting on HTTP responses, not in JS execution.
On the "affects all SCA and Container scans" concern, I agree. A few things that make me comfortable with the bump:
Other options:
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense for a single CLI run. A lot of our user actually run the CLI highly concurrent and already experience rate limiting. In the combination in the best case, the increase of request parallelism and rate limiting might just equal out, in the worst case the rate limiting will eventually exceed the retry limit and fail the CLI.