Vertical scaling issues with Java Client #1046

Cadence Java client exhibits severe vertical scaling limitations that prevent it from utilizing additional CPU cores and threads beyond a certain point. Despite increasing the number of pollers, executor threads, and concurrency settings, throughput plateaus due to bottlenecks.

Potential problems:

1. SynchronousQueue Bottleneck in PollTaskExecutor
   Location: /src/main/java/com/uber/cadence/internal/worker/PollTaskExecutor.java:54-59

The PollTaskExecutor uses a SynchronousQueue with zero capacity for task handoff:

taskExecutor = options.getExecutorWrapper().wrap(
    new ThreadPoolExecutor(
        0,
        options.getTaskExecutorThreadPoolSize(),
        1,
        TimeUnit.SECONDS,
        new SynchronousQueue<>()));

2. Per-Workflow Lock Serialization
   Location: /src/main/java/com/uber/cadence/internal/worker/WorkflowWorker.java:197-201

Issue:

Lock runLock = null;
if (!Strings.isNullOrEmpty(stickyTaskListName)) {
    runLock = runLocks.getLockForLocking(task.getWorkflowExecution().getRunId());
    runLock.lock();
}

3. Synchronized Throttler Contention
   Location: /src/main/java/com/uber/cadence/internal/worker/Throttler.java:93

public synchronized void throttle() throws InterruptedException {
    long now = System.currentTimeMillis();
    // ... entire method is synchronized
}

Could you please investigate this ?