Repository navigation
postgresql_user / postgresql_database fail under Chef 19 (Habitat) #833
Description
When running Chef 19 packaged as a Habitat package, the postgresql_user and postgresql_database resources fail with:
FATAL: LoadError: pg's C extension failed to load:
cannot load such file -- 3.4/pg_ext
libpq.so.5: cannot open shared object file: No such file or directory - /hab/pkgs/core/ruby3_4/3.4.8/.../gems/pg-1.6.3/lib/pg_ext.so
Root Cause
Since v11.0.0, this cookbook installs the pg gem to back postgresql_user and postgresql_database. The native extension pg_ext.so links against libpq.so.5 at runtime.
Habitat packages Chef with library isolation: the Ruby process is started with a scoped LD_LIBRARY_PATH pointing only to Habitat-internal paths, and the binary appears to be compiled with -z nodefaultlib, which disables both the ldconfig cache and default system library paths (/lib, /usr/lib). System-installed libpq.so.5 (from libpq5) is therefore invisible to the dynamic linker even when correctly installed.
Setting ENV['LDFLAGS'] or ENV['DLDFLAGS'] before the gem builds does not help:
LDFLAGScontrols executable linking in mkmf, not.solinking.DLDFLAGSis initialised by mkmf fromRbConfig::CONFIG, not fromENV.
Adding system paths to LD_LIBRARY_PATH causes a crash (__nptl_change_stack_perm symbol error) because the system libc.so.6 gets loaded alongside Habitat's own bundled libc.
Environment
- OS: Ubuntu 24.04 (noble)
- Chef version: 19.3.15 (Habitat)
- sous-chefs/postgresql: 13.0.5 (reproduced on all v11+ versions)
- PostgreSQL: 16 (PGDG)
Workaround
Downgrade to Chef 18 (Omnibus). Omnibus uses the system linker and resolves system libraries normally — the issue does not occur.
Activity
damacus commentedon Aug 11, 2026
I've got a partial fix here, but you are better off leveraging your Progress Support contract here.
Given we're debugging Habitat issues I cannot guarantee any fix that lands can actually work.
damacus commentedon Aug 11, 2026
Thanks for the detailed report. We have confirmed that this is specific to the Chef 19 Habitat runtime, but this repository cannot install or exercise that runtime in CI. A cookbook workaround for native pg packaging is not something we can support or release without repeatable CI coverage of the affected environment.\n\nWe are therefore waiting on Progress/Chef to provide either a supported way to reproduce Chef 19 Habitat builds in CI or an upstream fix for the Habitat package and native-library boundary. Please raise this through Progress Support and link any support case or upstream issue here so we can track it.\n\nWe will keep this issue open under the chef-19, habitat, and waiting labels. We are closing #834 rather than shipping a runtime-specific workaround that we cannot continuously verify.
damacus commentedon Aug 11, 2026
If you're still looking at v19, you might be better off going for Cinc in the mean time.
Add cspell words for pg gem libpq kitchen test
Fix opensuse package name and add postgresql cookbook kitchen coverage
mwrock commentedon Aug 17, 2026
Hi, thanks for the detailed report. I dug into this and can confirm the root cause, and it's not -z nodefaultlib. I couldn't find that flag referenced anywhere in Habitat's core/glibc , or core/gcc , and testing confirms the actual mechanism is different.
Root cause
The pg gem's extconf.rb bakes the path passed via --with-pg-lib into the compiled pg_ext.so as a DT_RUNPATH dynamic tag. DT_RUNPATH is only used to resolve the extension's own direct dependency ( libpq.so.5 ) - it is explicitly not inherited by that library's own transitive dependencies ( libssl.so.3 , libldap.so.2 , libkrb5.so.3 , etc., per the ELF spec).
Chef Infra Client, when packaged via Habitat, runs its bundled Ruby under Habitat's own private core/glibc / ld-linux-x86-64.so.2 (a distinct PT_INTERP from the host's system loader). This is a vanilla, non-Debian-patched glibc build, so it lacks the Debian/Ubuntu multiarch default search directories ( /usr/lib/x86_64-linux-gnu ) compiled into its system_dirs , and there's no LD_LIBRARY_PATH set to compensate. The result: libpq.so.5 itself resolves fine (direct dependency, covered by DT_RUNPATH ), but one hop down, libssl.so.3 (and friends) fail to resolve, which is exactly the confusing part of the original error, since it doesn't complain about libpq.so.5 at all.
The fix
Passing --with-ldflags=-Wl,--disable-new-dtags to the pg gem's native extension build forces the linker to emit the legacy DT_RPATH tag instead of DT_RUNPATH . Critically, DT_RPATH is inherited by everything subsequently resolved down the dependency chain, so it also fixes libpq.so.5 's own dependency resolution without needing to touch Habitat's LD_LIBRARY_PATH at all.
Verified via readelf -d before/after:
baseline (broken):
0x000000000000001d (RUNPATH) Library runpath: [.../ruby3_4/.../lib:/usr/lib/x86_64-linux-gnu]
fixed:
0x000000000000000f (RPATH) Library rpath: [.../ruby3_4/.../lib:/usr/lib/x86_64-linux-gnu]
Same paths, different tag — and require 'pg' goes from failing with the libssl.so.3 LoadError to succeeding.
Validation
I reproduced this both at the gem level (bare hab pkg exec chef/chef-infra-client ruby -e "require 'pg'" ) and end-to-end via a real chef-client -z convergence of postgresql_user / postgresql_database resources against a local PostgreSQL server, using the Habitat-packaged chef-infra-client :
• Without the fix: convergence fails during postgresql_user[...] with LoadError: pg's C extension failed to load: ... libpq.so.5: cannot open shared object file .
• With the fix: convergence completes successfully, and the role/database are created as expected.
I've opened a fix that updates pg_gem_build_options in libraries/sql/_connection.rb to add --with-ldflags=-Wl,--disable-new-dtags on both the RHEL/Amazon and Debian branches. While in there, I also corrected a separate (non-blocking) bug on the Debian branch, where --with-pg-lib was accidentally passed the include directory instead of the actual multiarch lib directory (this didn't break anything in practice because pg 's extconf.rb falls back to pg_config regardless, but it was still wrong).