Improve S3 performance for listing objects in transfer tasks#10293
Open
Improve S3 performance for listing objects in transfer tasks#10293
Conversation
This improves the rate at which we can list objects for S3
transfer tasks such as recursive download, sync, and s3 to s3
copies. In high compute environments this has become one of
the main bottlenecks affecting the transfer of a large number
of objects, particularly when using the CRT transfer client.
We aren't able to queue work fast enough. To speed things up
I added three changes.
The first is an improvement in parsing the `ListObjectsV2` response. We
were previously double-parsing the `LastModified` member, which is
mostly a historical artifact of when the CLI had differing behavior for
parsing timestamps than botocore. As a result of this custom parsing
being left in place in the bucket lister we were parsing the timestamps
twice. To minimize the scope of changes, we keep the existing
local-timezone datetime parsing in the bucket lister, but we set the
botocore parser used in the bucket lister client to be a noop. This
does make the code slightly more complicated as we only plumb through
this behavior for bucket lister so we need new client factory methods
for that, so we should decide if it's worth trying to make this
behavior the default for all of the S3 client creation used in the
CLI.
The remaining changes are related to moving the bucket listing off of
the main thread and over to a producer/consumer model, with the main
thread now pulling objects off of a shared queue.
The producer thread is further broken down into this "quick page"
feature where alternating threads are used to retrieve subsequent
pages with an SAX based XML parser being used to do a first pass scan
to extract the `NextContinuationToken`. This allows the network IO
work to continue as soon as possible while botocore finishes the
standard XML parsing of the response body, and the subsequent "page
drain" of processing the S3 key names and queueing files over to the
CRT layer.
As for rollout, I've added a new `bucket_lister` config option under
S3, with the default being the existing single threaded behavior.
Users can opt-in via:
```
s3 =
bucket_lister = threaded
```
The idea would be that this will flip to the default behavior after
some period of bake time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This improves the rate at which we can list objects for S3 transfer tasks such as recursive download, sync, and s3 to s3 copies. In high compute environments this has become one of the main bottlenecks affecting the transfer of a large number of objects, particularly when using the CRT transfer client. We aren't able to queue work fast enough. To speed things up I added three changes.
The first is an improvement in parsing the
ListObjectsV2response. We were previously double-parsing theLastModifiedmember, which is mostly a historical artifact of when the CLI had differing behavior for parsing timestamps than botocore. As a result of this custom parsing being left in place in the bucket lister we were parsing the timestamps twice. To minimize the scope of changes, we keep the existing local-timezone datetime parsing in the bucket lister, but we set the botocore parser used in the bucket lister client to be a noop. This does make the code slightly more complicated as we only plumb through this behavior for bucket lister so we need new client factory methods for that, so we should decide if it's worth trying to make this behavior the default for all of the S3 client creation used in the CLI.The remaining changes are related to moving the bucket listing off of the main thread and over to a producer/consumer model, with the main thread now pulling objects off of a shared queue.
The producer thread is further broken down into this "quick page" feature where alternating threads are used to retrieve subsequent pages with an SAX based XML parser being used to do a first pass scan to extract the
NextContinuationToken. This allows the network IO work to continue as soon as possible while botocore finishes the standard XML parsing of the response body, and the subsequent "page drain" of processing the S3 key names and queueing files over to the CRT layer.As for rollout, I've added a new
bucket_listerconfig option under S3, with the default being the existing single threaded behavior. Users can opt-in via:The idea would be that this will flip to the default behavior after some period of bake time.