Skip to content

proc: fix issues when attaching to a non-Go process that has already loaded a Go shared library - #4456

Open
nacx wants to merge 4 commits into
go-delve:masterfrom
nacx:proc-goroutines-cshared
Open

nacx wants to merge 4 commits into
go-delve:masterfrom
nacx:proc-goroutines-cshared

Conversation

@nacx

@nacx nacx commented Oct 9, 2026 •

Copy link
Copy Markdown

Fixes #4455

Context

We hit both problems debugging Go filters that run inside Envoy through its dynamic modules extension: a C++ process that dlopens a Go c-shared library. We attached with dlv attach <envoy pid> and connected from VS Code.

The tetratelabs/built-on-envoy#657 pull request already contains the patches in this PR to make debugging Go Envoy extensions possible, although that is a workaround until support for it lands in Delve.

Note

I used AI to help develop this fix. Claude with Opus 5.5

This fixes three problems when attaching to a non-Go process that has already loaded a Go shared library built with -buildmode=c-shared. Launching such a process works since #4263. Each fix is in its own commit.

1. Goroutines cannot be listed

(dlv) goroutines
could not find goroutine array

DAP clients hit the same error: the threads request fails with Unable to retrieve goroutines: could not find goroutine array, so VS Code shows a single "Dummy" thread and cannot show where the program stopped, even though the breakpoint was hit.

Cause: the goroutine cache looks up runtime.allgs and runtime.allglen only in bi.Images[0], and only once, when the target is created. Here the runtime variables are in a different image, which is either dlopened later or, when attaching, discovered only after the target is initialized.

Fix: search all loaded images for the runtime variables, and retry the lookup in getRuntimeAllg while the addresses are unknown. Regular Go executables behave as before, since the executable is searched first.

2. The first library loaded after attaching stops the target

sharedLibCallback runs the Go image setup (the unrecovered-panic, fatal-throw and plugin.Open breakpoints) the first time a Go image appears. When attaching, the Go library is already loaded, so that callback never fires for it. As a result:

  • the panic and fatal-throw breakpoints are never created;
  • the next library the process loads stops the target with StopSharedLibLoaded, with only non-Go frames, for example an NSS module loaded during a hostname lookup.

DAP reports that stop as a breakpoint, so users see the debugger "stop at a different location" than the breakpoint they set.

Fix: add Target.InitGoImage, shared by sharedLibCallback. The native Linux Attach calls it after the shared libraries are read. It does nothing for Go executables, which already create these breakpoints when the target is created.

3. On linux/amd64, the goroutine that hit a breakpoint is not found

On linux/amd64 the current goroutine of a thread is read from the G pointer in thread local storage, at an offset computed only for the executable. In a Go shared library, the G pointer is in the library's TLS block and accessed with the initial-exec model: its offset is in a GOT entry filled by the dynamic linker (R_X86_64_TPOFF64 relocation of runtime.tlsg), and depends on where the block was placed at load time. Delve read an unrelated value as the G pointer, so it could not associate the stopped thread with its goroutine: clients show every goroutine parked, e.g. in runtime.gopark, instead of the one stopped at the breakpoint.

Fix: when loading a Go shared library for a non-Go executable on x86_64, find the GOT entry of the runtime.tlsg relocation and read the offset from the target memory (the existing gStructOffsetIsPtr mechanism). arm64 is not affected, as the G pointer is kept in a register.

Tests

Both tests use the existing godlopen Go library. Each fails on master and passes with this change.

  • TestNonGoBinaryWithGoDlopenGoroutines launches the C host, stops in main.GoFunction, and lists goroutines. On master it fails with could not find goroutine array.
  • TestNonGoBinaryWithGoDlopenAttach uses the new _fixtures/godlopenattach C host, which loads the Go library and then, for each line on stdin, dlopens another library and calls into Go. The test attaches after the Go library is loaded, sets a breakpoint in main.GoFunction and continues. On master it stops with StopSharedLibLoaded instead.

TestNonGoBinaryWithGoDlopenGoroutines also checks that the goroutine of the stopped thread is the one in main.GoFunction, which covers the third fix on linux/amd64.

go test ./pkg/proc/... ./service/dap on linux/arm64 gives the same results as master. The only failure, TestCoreCGOAssert, also fails on master in that environment.

@nacx nacx changed the title Proc goroutines cshared proc: fix issues when attaching to a non-Go process that has already loaded a Go shared library Oct 9, 2026
nacx added 3 commits October 9, 2026 23:04
The goroutine cache only looked up runtime.allgs and runtime.allglen in
the executable, and only when the target was created. When the Go
runtime lives in a shared library (buildmode=c-shared) loaded by a
non-Go program, the variables are not in the executable, and the
library is usually loaded after the target is created (with dlopen, or
before the shared library list is read when attaching). Listing
goroutines then fails with "could not find goroutine array", which
breaks the goroutines command and makes DAP clients report a single
"Dummy" thread, even though breakpoints in the library are hit.

Search all the loaded images for the runtime variables, and retry the
lookup when goroutines are requested if they were not found yet.
The Go-specific setup done when the first Go image is loaded by a non-Go
executable (panic, fatal throw and plugin.Open breakpoints) runs from
the shared library load callback. When attaching to a process that has
already loaded a Go shared library, no load event is received for it,
so the setup never ran: those breakpoints were never created, and the
next library loaded by the process (e.g. an NSS module during a host
lookup) stopped the target with StopSharedLibLoaded. DAP reports that
stop as a breakpoint, so users see the debugger stop at an unexpected
location.

Move the setup to Target.InitGoImage, and call it from the native Linux
Attach once the shared libraries are known.
On linux/amd64 the current goroutine of a thread is read from the G
pointer in thread local storage, at an offset computed only for the
executable. When the Go runtime is in a shared library (buildmode=
c-shared) loaded by a non-Go program, the G pointer is in the TLS block
of the library, accessed with the initial-exec model: its offset is in
a GOT entry filled by the dynamic linker through a R_X86_64_TPOFF64
relocation, and depends on where the TLS block was placed at load time.

Delve then read an unrelated value as the G pointer, and could not
associate the thread that hit a breakpoint with its goroutine: clients
showed every goroutine parked (e.g. in runtime.gopark) instead of the
one stopped at the breakpoint.

When loading a Go shared library for a non-Go executable on x86_64,
find the GOT entry of the runtime.tlsg relocation and read the offset
from the target memory. arm64 is not affected, as the G pointer is kept
in a register.
@nacx
nacx force-pushed the proc-goroutines-cshared branch from aee53eb to 1a765c9 Compare October 9, 2026 21:04
Signed-off-by: Ignasi Barrera <nacx@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proc: attaching to a non-Go process with a Go c-shared library: goroutines can't be listed, and the next dlopen stops the target

1 participant