Repository navigation
proc: fix issues when attaching to a non-Go process that has already loaded a Go shared library - #4456
Open
nacx wants to merge 4 commits into
Open
proc: fix issues when attaching to a non-Go process that has already loaded a Go shared library#4456nacx wants to merge 4 commits into
nacx wants to merge 4 commits into
Conversation
The goroutine cache only looked up runtime.allgs and runtime.allglen in the executable, and only when the target was created. When the Go runtime lives in a shared library (buildmode=c-shared) loaded by a non-Go program, the variables are not in the executable, and the library is usually loaded after the target is created (with dlopen, or before the shared library list is read when attaching). Listing goroutines then fails with "could not find goroutine array", which breaks the goroutines command and makes DAP clients report a single "Dummy" thread, even though breakpoints in the library are hit. Search all the loaded images for the runtime variables, and retry the lookup when goroutines are requested if they were not found yet.
The Go-specific setup done when the first Go image is loaded by a non-Go executable (panic, fatal throw and plugin.Open breakpoints) runs from the shared library load callback. When attaching to a process that has already loaded a Go shared library, no load event is received for it, so the setup never ran: those breakpoints were never created, and the next library loaded by the process (e.g. an NSS module during a host lookup) stopped the target with StopSharedLibLoaded. DAP reports that stop as a breakpoint, so users see the debugger stop at an unexpected location. Move the setup to Target.InitGoImage, and call it from the native Linux Attach once the shared libraries are known.
On linux/amd64 the current goroutine of a thread is read from the G pointer in thread local storage, at an offset computed only for the executable. When the Go runtime is in a shared library (buildmode= c-shared) loaded by a non-Go program, the G pointer is in the TLS block of the library, accessed with the initial-exec model: its offset is in a GOT entry filled by the dynamic linker through a R_X86_64_TPOFF64 relocation, and depends on where the TLS block was placed at load time. Delve then read an unrelated value as the G pointer, and could not associate the thread that hit a breakpoint with its goroutine: clients showed every goroutine parked (e.g. in runtime.gopark) instead of the one stopped at the breakpoint. When loading a Go shared library for a non-Go executable on x86_64, find the GOT entry of the runtime.tlsg relocation and read the offset from the target memory. arm64 is not affected, as the G pointer is kept in a register.
nacx
force-pushed
the
proc-goroutines-cshared
branch
from
October 9, 2026 21:04
aee53eb to
1a765c9
Compare
Signed-off-by: Ignasi Barrera <nacx@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4455
Context
We hit both problems debugging Go filters that run inside Envoy through its dynamic modules extension: a C++ process that
dlopens a Go c-shared library. We attached withdlv attach <envoy pid>and connected from VS Code.The tetratelabs/built-on-envoy#657 pull request already contains the patches in this PR to make debugging Go Envoy extensions possible, although that is a workaround until support for it lands in Delve.
Note
I used AI to help develop this fix. Claude with Opus 5.5
This fixes three problems when attaching to a non-Go process that has already loaded a Go shared library built with
-buildmode=c-shared. Launching such a process works since #4263. Each fix is in its own commit.1. Goroutines cannot be listed
DAP clients hit the same error: the
threadsrequest fails withUnable to retrieve goroutines: could not find goroutine array, so VS Code shows a single "Dummy" thread and cannot show where the program stopped, even though the breakpoint was hit.Cause: the goroutine cache looks up
runtime.allgsandruntime.allglenonly inbi.Images[0], and only once, when the target is created. Here the runtime variables are in a different image, which is eitherdlopened later or, when attaching, discovered only after the target is initialized.Fix: search all loaded images for the runtime variables, and retry the lookup in
getRuntimeAllgwhile the addresses are unknown. Regular Go executables behave as before, since the executable is searched first.2. The first library loaded after attaching stops the target
sharedLibCallbackruns the Go image setup (the unrecovered-panic, fatal-throw andplugin.Openbreakpoints) the first time a Go image appears. When attaching, the Go library is already loaded, so that callback never fires for it. As a result:StopSharedLibLoaded, with only non-Go frames, for example an NSS module loaded during a hostname lookup.DAP reports that stop as a breakpoint, so users see the debugger "stop at a different location" than the breakpoint they set.
Fix: add
Target.InitGoImage, shared bysharedLibCallback. The native LinuxAttachcalls it after the shared libraries are read. It does nothing for Go executables, which already create these breakpoints when the target is created.3. On linux/amd64, the goroutine that hit a breakpoint is not found
On linux/amd64 the current goroutine of a thread is read from the G pointer in thread local storage, at an offset computed only for the executable. In a Go shared library, the G pointer is in the library's TLS block and accessed with the initial-exec model: its offset is in a GOT entry filled by the dynamic linker (
R_X86_64_TPOFF64relocation ofruntime.tlsg), and depends on where the block was placed at load time. Delve read an unrelated value as the G pointer, so it could not associate the stopped thread with its goroutine: clients show every goroutine parked, e.g. inruntime.gopark, instead of the one stopped at the breakpoint.Fix: when loading a Go shared library for a non-Go executable on x86_64, find the GOT entry of the
runtime.tlsgrelocation and read the offset from the target memory (the existinggStructOffsetIsPtrmechanism). arm64 is not affected, as the G pointer is kept in a register.Tests
Both tests use the existing
godlopenGo library. Each fails on master and passes with this change.TestNonGoBinaryWithGoDlopenGoroutineslaunches the C host, stops inmain.GoFunction, and lists goroutines. On master it fails withcould not find goroutine array.TestNonGoBinaryWithGoDlopenAttachuses the new_fixtures/godlopenattachC host, which loads the Go library and then, for each line on stdin,dlopens another library and calls into Go. The test attaches after the Go library is loaded, sets a breakpoint inmain.GoFunctionand continues. On master it stops withStopSharedLibLoadedinstead.TestNonGoBinaryWithGoDlopenGoroutinesalso checks that the goroutine of the stopped thread is the one inmain.GoFunction, which covers the third fix on linux/amd64.go test ./pkg/proc/... ./service/dapon linux/arm64 gives the same results as master. The only failure,TestCoreCGOAssert, also fails on master in that environment.