fix(aws-lambda): force placeholder resource so the Lambda Extension drops dd-tracer-serverless-span#11313
Conversation
…rops dd-tracer-serverless-span The Datadog Lambda Extension filters the placeholder invocation span by comparing `span.resource == "dd-tracer-serverless-span"` (see `bottlecap/src/traces/trace_processor.rs::filter_span_from_lambda_library_or_runtime`). The intent is that the extension drops the placeholder so the inferred `aws.lambda` span (which the end-invocation handshake gives the same `span_id`) is the surviving record under that key in the trace store and remains parented to the inferred apigateway root. In practice the HTTP/JAX-RS instrumentation overwrites the placeholder's resource with the request route (e.g. "POST /") at `HTTP_FRAMEWORK_ROUTE` priority during the invocation, so the extension's filter no longer matches at end-of-invocation. Both records (the placeholder with `parent_id=0`, and the inferred `aws.lambda` with `parent_id=apigateway.span_id`) reach the backend under the same `(trace_id, span_id)` key. The trace store keeps the placeholder, and the rest of the trace is detached from the inferred apigateway root. Fix: in `LambdaHandlerInstrumentation.exit`, force the resource name back to the literal placeholder marker right before `span.finish()`, using `ResourceNamePriorities.MANUAL_INSTRUMENTATION` so the override beats whatever HTTP-framework priority the in-flight instrumentation has written. Verified end-to-end on Quarkus 3.15.4 / Java 21 / Lambda Extension v96: pre-fix the placeholder leaks to the backend with `parent_id=0` and the trace store reports orphans; post-fix zero `dd_tracer_serverless_span` rows reach the backend and the trace store reports a single 11-span chunk with no orphans. Refs: SLES-2837
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf170244a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
rithikanarayan
added
inst: aws lambda
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f07b5b0aac
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
gh-worker-dd-mergequeue-cf854d
Bot
merged commit 3bca1f0
into
master
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
zarir/sles-2837-lambda-placeholder-resource
branch
3 participants
Summary
The Datadog Lambda Extension drops the placeholder invocation span by comparing
span.resource == "dd-tracer-serverless-span"(bottlecap/src/traces/trace_processor.rs::filter_span_from_lambda_library_or_runtime). The dedup is what lets the inferredaws.lambdaspan (which the end-invocation handshake gives the samespan_id) be the surviving record under that key in the trace store, parented to the inferred apigateway root.In practice the HTTP/JAX-RS instrumentation overwrites the placeholder's resource with the request route ("POST /") at
HTTP_FRAMEWORK_ROUTEpriority during the invocation, so by end-of-invocation the extension's filter no longer matches. Both records — the placeholder withparent_id=0and the inferredaws.lambdawithparent_id=apigateway.span_id— reach the backend under the same(trace_id, span_id)key. The trace store keeps the placeholder'sparent_id=0, detaching the rest of the trace from the inferred apigateway root.Fix
In
LambdaHandlerInstrumentation.exit, force the resource name back toINVOCATION_SPAN_NAME("dd-tracer-serverless-span") right beforespan.finish(), usingResourceNamePriorities.MANUAL_INSTRUMENTATION(6) so the override beats whatever priority the in-flight HTTP/JAX-RS instrumentation has written (HTTP_FRAMEWORK_ROUTEis 3).Files touched:
LambdaHandlerInstrumentation.java(+16/-0): importResourceNamePriorities, set the resource name in the exit advice.Test plan
LambdaHandlerInstrumentationTestwith a case that simulates HTTP-framework instrumentation overwritingresourceand asserts the final serialized span has resource"dd-tracer-serverless-span".parent_id=0(/api/v1/trace/<id>shows orphans,operation_name:dd_tracer_serverless_spanrows present in the spans index). Post-fix the spans index returns 0 placeholder rows and the trace store reports a single 11-span chunk with no orphans.Refs
SLES-2837
Notes