feat(Storage)!: Enable full object checksum validation for resumable uploads - #15769
krishnamd-jkp merged 15 commits into
Conversation
product-auto-label
Bot
added
the
api: storage
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the upload validation mechanism in the Google Cloud Storage V1 client. It introduces UploadValidationMode.RejectAndThrow to allow the server to reject mismatched objects before creation, deprecates UploadValidationMode.DeleteAndThrow, and removes UploadValidationMode.ThrowOnly along with the UploadValidationException class. It also introduces a HashingStream within CustomMediaUpload to calculate CRC32C hashes on the fly and inject the x-goog-hash header. The feedback highlights that removing ThrowOnly and UploadValidationException are breaking changes for existing client applications, and recommends restoring them with an [Obsolete] attribute to maintain backward compatibility.
I am having trouble creating individual review comments. Click here to see my feedback.
apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/UploadValidationMode.cs (29-35)
Removing ThrowOnly from the UploadValidationMode enum is a breaking change for existing client applications. To maintain backward compatibility, please restore ThrowOnly and mark it as [Obsolete], mapping it to the new validation behavior.
/// <summary>
/// Obsolete. Use <see cref="RejectAndThrow"/> instead.
/// Previously, the hash of the data was computed while uploading, and if the resulting object had a different hash, an exception was thrown, but the object remained present in Storage.
/// </summary>
[Obsolete("ThrowOnly is deprecated. Use RejectAndThrow instead, as the server now rejects the object before creation.")]
ThrowOnly = 1,
/// <summary>
/// Obsolete. Use <see cref="RejectAndThrow"/> instead.
/// Previously, the object was uploaded and then deleted if the hash mismatched.
/// The server now rejects mismatched objects automatically before creation.
/// </summary>
[Obsolete("DeleteAndThrow is deprecated. Use RejectAndThrow instead, as the server now rejects the object before creation.")]
DeleteAndThrow = 2,apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/UploadValidationException.cs (23-26)
Removing the public class UploadValidationException is a breaking change for existing client applications that catch this exception.
To maintain backward compatibility, please restore this class and mark it as [Obsolete]. This allows existing code to compile with a warning rather than failing with a compilation error.
mahendra-google
changed the title
|
Let's keep this one in draft until we have updated the core libraries to the newest dependency. |
8160401 to
4b0d135
Compare
- Add integration tests to verify ArgumentException propagation through Google.Apis ResumableUpload when resuming an upload from the intermediate offset.
amanda-tarafa
left a comment
amanda-tarafa
left a comment
There was a problem hiding this comment.
Because of the breaking changes you need to update the next version of the library in pipeline-state.json. Add a new property for Storage "nextVersion" with value "5.0.0" and next time we release we'll automatically bump major.
…ry in pipeline-state.json - Assign value 5.0.0 to the nextVersion propery so that next storage release will be major release
…76.0.4250 in apis.json. - Removed Google.Apis 1.76 version dependency from Google.Cloud.Storage.V1 project file - Regenerated project file using generateprojects.sh
- Throw an argument exception as soon as gap is found while resuming upload from an intermeditae offset - Maintain -position value up to date in the CustomMediaUpload - Add static method GetWrappedSourceStream to wrap source stream based on upload validation mode - Modify upload object integration tests - Create an unit test file for CustomMediaUpload test. - Modify UploadObjectOptionsTest file.
New property for Storage |
…e offset retry. Removed test for CustomMediaUpload with intermediate offset retry.
amanda-tarafa
left a comment
amanda-tarafa
left a comment
There was a problem hiding this comment.
LGTM with a small change request and a question. Thanks!!
amanda-tarafa
added
the
allow breaking changes
|
(Also, remember to squash in a single commit before merging). |
…rom Google.Cloud.Storage.V1 file
…Cloud.Storage.V1 file
### New features - Enable full object checksum validation for resumable uploads (#15769) Librarian-Release-Library: Google.Cloud.Storage.V1 Librarian-Release-Version: 5.0.0 Librarian-Release-ID: release-20260921T190154Z
3 participants
This PR transitions upload object checksum integrity validation from client-side to the server-side.
Previously,
CRC32Cvalidation occurred after the upload is completed, requiring the client to delete the object (usingDeleteAndThrowupload validation mode) or leave a corrupted object in the bucket (usingThrowOnlyupload validation mode) upon hash mismatches. By leveraging the newLastRequestExecutingevent ingoogle-api-dotnet-clientcore library, this implementation calculatesCRC32Cincrementally during streaming and injects thex-goog-hash: crc32c=...header on the final chunk. If a checksum mismatch occurs, the server rejects the upload with an HTTP400 Bad Request, ensuring invalid objects are never created in the bucket.