Skip to content

Remove nginx per-repo concurrency workaround now that RDF4J federation deadlock is fixed #155

Open

Description

@ashleycaselli

The RDF4J federation deadlock that motivated our nginx workaround has been fixed upstream, so the workaround is now likely removable.

Background

We serialize requests per repo in nginx to avoid the RDF4J deadlock described in rdf4j#5120 / rdf4j#5623:

https://github.com/knowledgepixels/nanopub-query/blob/main/nginx.conf#L31-L35

# ----------
# To avoid deadlock problem with rdf4j-workbench:5.0.2:
limit_conn uri 1;
#limit_req zone=one burst=100;
# ----------

Upstream PR rdf4j#5571 ("GH-5120: Fix SPARQL federation deadlock") was merged on 2025-12-15 and shipped in RDF4J 5.2.2. We are already on 5.3.1 (rdf4j.version in pom.xml, and eclipse/rdf4j-workbench:5.3.1-tomcat in docker-compose.yml), so the fix is in our runtime. See #5 for the version-update history.

Why it matters

limit_conn uri 1 caps each repo endpoint at a single in-flight request. Every concurrent query to the same repo is queued or rejected (503), which is a hard ceiling on read throughput regardless of how much the triple store could handle. It was an acceptable price for avoiding hangs; it should not be a permanent one.

Proposed work

  • Remove the limit_conn uri 1; block from nginx.conf (and the now-unused limit_conn_zone $uri zone=uri:10m; at the top, plus the commented-out limit_req lines if we don't intend to use them).
  • Remove the "Deadlock Problem Workaround" section from README.md.
  • Revisit the -Dorg.eclipse.rdf4j.client.http.*Timeout=10000 JAVA_OPTS in docker-compose.yml — those were the other half of the workaround. They may still be worth keeping as defensive timeouts; that's a separate call.

Verification needed before merging

The removal should not be a blind revert. The original symptom only appeared after hours or days of sustained federated load, so:

  • Run a concurrent-load test against a repo (many parallel SPARQL reads, including SERVICE federation) on 5.3.1 with the limit removed, and confirm no hangs.
  • Ideally soak it on a staging deployment for a meaningful period before rolling to production, given the latency of the original failure mode.

Also worth noting

RDF4J 5.3.2 was released 2026-06-29. Bumping from 5.3.1 is routine maintenance and independent of this issue, but could be picked up alongside it.

Activity

  1. tkuhn commented on Aug 4, 2026

    @tkuhn
    Contributor

    I removed this temporarily on the live instances but then (possibly related) issues popped up after a few weeks, so now the nginx conf workaround is on again. Let's leave this open until we're certain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions