refactor(tracing): one trace pipeline behind a minimal provider interface by laipz8200 · Pull Request #44673 · langgenius/dify

laipz8200 · 2026-10-23T17:22:19Z

Summary

Ops tracing ran four overlapping paths: the legacy queue, "unified" trace, the Agent trace ledger and enterprise telemetry. They shared one loosely typed payload. Configuration was decrypted on the request path, and every provider package reached into the database, Redis and process-global SDK state. This PR replaces them with one pipeline and gives every provider the same minimal interface.

Provider interface (`api/providers/trace/trace-sdk`, imported as `dify_trace_sdk`)

This is the lowest common denominator of the 10 supported providers. Each provider implements its own send:

class TraceProvider[ConfigT: ProviderConfig]:
    id: ClassVar[str]
    config_model: ClassVar[type[ProviderConfig]]
    capabilities: ClassVar[ProviderCapabilities]

    def verify(self, config: ConfigT) -> ConfigT: ...          # check credentials, fill derived fields
    def project_url(self, config: ConfigT) -> str | None: ...  # no I/O
    def send(self, config: ConfigT, trace: Trace) -> SendResult: ...

send receives a provider-neutral Trace: a parent-first tree of spans with OpenTelemetry IDs and typed LLM usage and cost.
Each failure is classified as retryable, rejected or outcome-unknown.
All network access goes through an injected Egress, which applies Dify's existing outbound HTTP policy.
Providers are discovered through dify.trace_providers entry points, so a deployment can install any subset.
Ported providers: Langfuse (ingestion REST API), LangSmith, Opik, Weave, Arize, Phoenix, Aliyun, MLflow, Databricks and Tencent.
- Langfuse, Opik, Weave, Arize and Phoenix no longer use vendor SDKs.
- LangSmith and MLflow still use their SDKs, routed through the egress.
IDs are deterministic, so a resend updates the same records instead of duplicating them.

Pipeline

Producers emit typed events through TraceScope.emit after their records are committed. A scope exists only when the app traces or enterprise telemetry is on, and emitting never raises or blocks the request.
tasks.trace_delivery_task runs on the existing ops_trace queue:
- reloads the records through the event's workspace;
- feeds enterprise telemetry and the Agent collector;
- builds the trace for the app's current configuration and calls send.
A retryable failure stores only the built trace and resends it, up to OPS_TRACE_EXPORT_MAX_ATTEMPTS attempts.
The Agent trace ledger keeps its durable path and exports through send. It still reads runs captured before the upgrade.
Removed:
- the per-process queue and timer;
- Redis parent-context polling;
- the nested-workflow coordination, since a nested workflow is now its own trace linked by attributes.

Configuration

Secrets are write-only. GET /apps/<id>/trace-config returns the non-secret fields plus configured_secrets.
- A blank or omitted secret keeps its stored value.
- Changing a destination field (host, endpoint or URL) requires entering the secrets again.
- The console form follows this.
Migration 53c10126d551 keeps one trace_app_config row per (app, provider) and adds a unique index.
Settings:
- Removed: OPS_TRACE_UNIFIED_ENABLED, OPS_TRACE_RETRYABLE_DISPATCH_MAX_RETRIES, OPS_TRACE_RETRYABLE_DISPATCH_DELAY_SECONDS, OPS_TRACE_PARENT_CONTEXT_TTL_SECONDS.
- Added: OPS_TRACE_EXPORT_MAX_ATTEMPTS (6) and ENABLE_TRACE_PAYLOAD_SWEEP_TASK (daily cleanup of orphaned payloads).

Removed

api/core/ops/**;
services/app_tracing_config_gateway.py;
the legacy provider dispatch and transports;
the per-provider *_trace.py and agent_trace.py exporters;
the opik, weave, langfuse and arize-phoenix-otel dependencies.

Dropping those SDKs exposed runtime dependencies the API never declared. They are now direct dependencies at their already-locked versions:

pydantic-settings, which configs uses;
sentry-sdk;
the OTLP gRPC and HTTP exporters.

Architecture

The tracing slice follows the backend architecture guide linked from api/AGENTS.md, including the domain boundary described in #44957:

Domain (api/domain/tracing): an Agent trace run is the aggregate TraceRunAgg, with its invocations, batches and receipts as child entities.
- Its methods own the run's transitions: freezing the destination and capture policy, collecting batches, capturing the terminal, claiming, accepting, deferring and retrying exports.
- Callers pass in the current time, IDs and export tokens; the package performs no I/O.
- Recovering a dropped terminal is a domain service of named functions over the run and its source's committed state.
- Domain errors are separate from the application's and are translated at the boundary, so error codes are unchanged.
Application services (TraceScopes, TraceDeliveryService, the provider configuration gateway and the Agent trace collector) receive ports and plain settings at construction. They no longer open sessions, read configuration, or reach Celery, storage or the encrypter themselves. Their errors share one base in services/tracing/errors.py with no HTTP status, and the handlers map them.
Records are immutable values. Repositories map rows to records before their sessions close, and database failures surface as TracingStoreUnavailableError. Trace record assembly moved to repositories/trace_record_repository.py.
Agent trace runs are loaded and saved whole by ApplicationTraceRepository, which holds the run's row lock from load to save. The unique source key settles concurrent starts.
Adapters in services/tracing/adapters.py and registry.py implement the remaining ports: the Celery emitter and retry queue, the entry-point provider catalog, workspace secret encryption, enterprise telemetry and Agent export scheduling.
Composition root: extensions/application_services/tracing.py builds every tracing service from configuration. Celery tasks and handlers get them from application_services().tracing, and the Agent hooks are bound only when Agent tracing is on.
Import contracts:
- domain-boundary keeps api/domain free of application, persistence, framework, configuration, queue, transport and legacy kernel code.
- tracing-application-boundary, tracing-controller-boundary and tracing-repository-boundary keep the services free of Flask, SQLAlchemy, models, configs and Celery, keep the handlers off persistence and the domain, and keep the repositories on the contracts they implement.
Tests use in-memory port implementations and real repositories instead of patching module internals.

Behaviour changes worth reviewing

Trace and span IDs are now deterministic per Dify record, so traces sent before and after the upgrade won't merge in vendor UIs.
Span names and structure come from one shared builder. Some vendor-specific attributes and the Tencent metrics are gone.
MLflow OSS export needs an MLflow server at version 3.4 or later with a database-backed store.
tasks.ops_trace_task.process_trace_tasks remains only to drain messages a pre-upgrade API enqueues during a rolling deploy.
With a caller-supplied trace ID, suggested-question spans still form their own trace, as before. That request does not know the caller's trace ID.

Screenshots

Checklist

This change requires a documentation update, included: Dify Document
I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
I've updated the documentation accordingly.
I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

Verification:

api/tests/unit_tests together with api/tests/e2e_tests/contracts: 30,032 passed, 4 skipped.
Provider packages: 690 passed.
pyrefly: 0 findings across production code and both test configurations.
ruff, import-linter (79 contracts) and the getattr, controller SQLAlchemy, session-mock, base-model, test-double and spec-mock guards: all clean against main.
Web tracing specs: 136 passed.

Not verified against live vendor backends.

From Claude Code

github-actions · 2026-10-23T17:23:56Z

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	79.95%	80.91%	+0.95%
Strict coverage	79.64%	80.61%	+0.97%
Typed symbols	76,880	76,554	-326
Untyped symbols	19,352	18,132	-1,220
Modules	3984	3975	-9

github-actions · 2026-10-23T17:23:59Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-10-28 21:36:04.517419304 +0000
+++ /tmp/pyrefly_pr.txt	2026-10-28 21:35:49.694277666 +0000
@@ -1742,14 +1742,10 @@
    --> tests/e2e_tests/contracts/core/workflow/nodes/test_if_else.py:453:19
 ERROR `Literal['{invalid-json-schema']` is not assignable to attribute `json_schema` with type `dict[str, Any] | None` [bad-assignment]
    --> tests/e2e_tests/contracts/core/workflow/nodes/test_start_node_json_object.py:209:28
-ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
-  --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:73:9
-ERROR Object of class `ModuleType` has no attribute `TraceTask` [missing-attribute]
-  --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:74:9
 ERROR Argument `_StubToolRuntime` is not assignable to parameter `runtime` with type `ToolNodeRuntimeProtocol` in function `graphon.nodes.tool.tool_node.ToolNode.__init__` [bad-argument-type]
-   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:125:17
+   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:112:17
 ERROR Argument `Iterator[ToolRuntimeMessage]` is not assignable to parameter `messages` with type `Generator[ToolRuntimeMessage]` in function `graphon.nodes.tool.tool_node.ToolNode._transform_message` [bad-argument-type]
-   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:141:18
+   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:128:18
 ERROR `-` is not supported between `None` and `datetime` [unsupported-operation]
   --> tests/e2e_tests/contracts/libs/_human_input/test_models.py:64:21
 ERROR `-` is not supported between `None` and `datetime` [unsupported-operation]
@@ -2811,87 +2807,87 @@
 ERROR No matching overload found for function `str.join` called with arguments: (list[AgentScratchpadUnit.Action | str]) [no-matching-overload]
    --> tests/unit_tests/core/agent/output_parser/test_cot_output_parser.py:249:25
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:121:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:143:29
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:124:5
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:146:5
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:129:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:151:9
 ERROR Missing required key `action` for TypedDict `ActionDict` [bad-typed-dict-key]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:157:44
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:179:44
 ERROR Missing required key `action_input` for TypedDict `ActionDict` [bad-typed-dict-key]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:157:44
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:179:44
 ERROR Key `invalid` is not defined in TypedDict `ActionDict` [bad-typed-dict-key]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:157:45
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:179:45
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:211:55
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:233:55
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:224:55
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:246:55
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:248:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:270:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:250:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:272:33
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:251:61
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:273:61
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:269:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:291:33
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:272:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:294:29
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:275:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:297:9
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:291:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:313:33
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:294:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:316:29
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:305:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:327:35
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:309:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:331:9
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:321:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:343:29
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:326:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:348:9
 ERROR `Literal[1]` is not assignable to attribute `prompt_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:332:32
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:354:32
 ERROR `Literal[1]` is not assignable to attribute `completion_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:333:36
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:355:36
 ERROR `Literal[2]` is not assignable to attribute `total_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:334:31
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:356:31
 ERROR `Literal[1]` is not assignable to attribute `prompt_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:340:32
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:362:32
 ERROR `Literal[1]` is not assignable to attribute `completion_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:341:36
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:363:36
 ERROR `Literal[2]` is not assignable to attribute `total_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:342:31
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:364:31
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:370:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:392:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:382:19
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:404:19
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:402:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:424:9
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:410:16
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:432:16
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:428:25
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:450:25
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:448:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:472:9
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:456:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:480:33
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:458:25
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:482:25
 ERROR `list[str]` is not assignable to attribute `_agent_scratchpad` with type `list[AgentScratchpadUnit]` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:488:36
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:513:36
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:510:55
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:536:13
 ERROR Object of class `FunctionType` has no attribute `assert_called` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:514:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:541:9
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:580:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:607:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:594:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:621:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:608:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:635:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:623:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:650:35
 ERROR `list[DummyTool]` is not assignable to attribute `_prompt_messages_tools` with type `Sequence[PromptMessageTool]` [bad-assignment]
   --> tests/unit_tests/core/agent/test_cot_chat_agent_runner.py:49:37
 ERROR `DummyGenerateEntity` is not assignable to attribute `application_generate_entity` with type `AgentChatAppGenerateEntity` [bad-assignment]
@@ -2909,79 +2905,79 @@
 ERROR `in` is not supported between `Literal['Question: What is Python?']` and `None` [not-iterable]
    --> tests/unit_tests/core/agent/test_cot_completion_agent_runner.py:180:16
 ERROR `Literal[0]` is not assignable to attribute `prompt_price` with type `Decimal` [bad-assignment]
-  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:51:26
+  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:53:26
 ERROR `Literal[0]` is not assignable to attribute `completion_price` with type `Decimal` [bad-assignment]
-  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:52:30
+  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:54:30
 ERROR `Literal[0]` is not assignable to attribute `total_price` with type `Decimal` [bad-assignment]
-  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:53:25
+  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:55:25
 ERROR Missing argument `session` in function `core.agent.base_agent_runner.BaseAgentRunner.__init__` [missing-argument]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:194:37
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:216:37
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:227:5
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:249:5
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:238:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:260:9
 ERROR Argument `None` is not assignable to parameter `query` with type `str` in function `core.agent.fc_agent_runner.FunctionCallAgentRunner._organize_user_query` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:339:46
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:361:46
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:407:58
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:429:58
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:455:19
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:477:19
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:512:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:534:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:514:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:536:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:516:33
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:538:33
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:517:33
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:539:33
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:518:16
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:540:16
 ERROR Object of class `FunctionType` has no attribute `assert_called` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:523:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:545:9
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:525:23
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:547:23
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:531:19
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:553:19
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:548:57
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:570:57
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:554:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:576:9
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:570:57
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:592:57
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:578:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:600:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:580:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:602:35
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:586:46
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:608:46
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:589:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:611:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:591:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:613:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:593:16
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:615:16
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:609:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:631:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:620:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:642:35
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:636:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:658:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:638:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:660:35
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:653:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:677:9
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:658:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:682:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:667:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:691:35
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:673:25
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:698:25
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:677:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:702:9
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:689:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:714:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:692:29
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:717:29
 ERROR Missing argument `description` in function `core.agent.plugin_entities.AgentStrategyEntity.__init__` [missing-argument]
    --> tests/unit_tests/core/agent/test_plugin_entities.py:292:32
 ERROR Argument `dict[str, dict[str, str | dict[str, float | list[str]]]]` is not assignable to parameter `config` with type `AppModelConfigDict` in function `core.app.app_config.easy_ui_based_app.model_config.manager.ModelConfigManager.convert` [bad-argument-type]
@@ -2997,115 +2993,105 @@
 ERROR Expected a mapping, got object | None [invalid-argument]
   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_config_manager.py:41:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:225:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:261:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:236:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:272:22
 ERROR Object of class `object` has no attribute `files` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:337:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:368:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:380:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:424:33
 ERROR Object of type `Generator[Mapping[str, Any] | str] | Mapping[str, Any]` has no attribute `resumed` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:384:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:428:16
 ERROR Object of class `object` has no attribute `single_iteration_run` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:449:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:493:16
 ERROR Object of class `object` has no attribute `extras` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:450:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:494:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:502:18
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:546:18
 ERROR Object of class `object` has no attribute `single_loop_run` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:512:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:556:16
 ERROR Object of class `object` has no attribute `extras` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:513:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:557:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:612:32
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:656:32
 ERROR Cannot index into `Generator[Mapping[str, Any] | str]` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:615:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:659:16
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:619:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:663:16
 ERROR Argument `str | None` is not assignable to parameter `s` with type `bytearray | bytes | str` in function `json.loads` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:624:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:668:27
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:749:13
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:793:13
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:791:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:835:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:796:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:840:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:842:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:886:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:847:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:891:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:904:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:948:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:909:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:953:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:913:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:957:33
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:977:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1021:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:982:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1026:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1045:27
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1089:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1050:25
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1094:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1101:31
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1145:31
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1111:41
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1155:41
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1149:35
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1193:35
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1159:45
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1203:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1223:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1267:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1228:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1272:21
 ERROR Object of class `NoneType` has no attribute `show_retrieve_source` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1309:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1340:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1450:33
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1459:33
 ERROR Object of type `Generator[Mapping[str, Any] | str] | Mapping[str, Any]` has no attribute `ok` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1453:16
-ERROR Object of class `NoneType` has no attribute `app_id` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1457:16
-ERROR Object of class `NoneType` has no attribute `user_id` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1458:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1462:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1498:33
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1516:33
 ERROR Object of type `Generator[Mapping[str, Any] | str] | Mapping[str, Any]` has no attribute `ok` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1501:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1519:16
 ERROR Argument `Literal['advanced-chat', 'agent', 'agent-chat', 'channel', 'chat', 'completion', 'rag-pipeline', 'workflow']` is not assignable to parameter `mode` with type `AppMode | SQLCoreOperations[AppMode]` in function `models.model.Conversation.__init__` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1595:14
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1614:14
 ERROR Argument `Literal['normal']` is not assignable to parameter `status` with type `ConversationStatus | SQLCoreOperations[ConversationStatus]` in function `models.model.Conversation.__init__` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1601:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1620:16
 ERROR Argument `Literal['web-app']` is not assignable to parameter `invoke_from` with type `InvokeFrom | SQLCoreOperations[InvokeFrom | None] | None` in function `models.model.Conversation.__init__` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1602:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1621:21
 WARN Identity comparison `True is False` is always False [unnecessary-comparison]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1721:42
-ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1731:5
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1761:42
 ERROR Class member `_StubToolNode._get_error_strategy` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1763:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1792:9
 ERROR Class member `_StubToolNode._get_retry_config` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1766:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1795:9
 ERROR Class member `_StubToolNode._get_title` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1769:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1798:9
 ERROR Class member `_StubToolNode._get_description` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1772:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1801:9
 ERROR Class member `_StubToolNode._get_default_value_dict` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1775:9
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1983:37
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1804:9
 ERROR `+` is not supported between `list[str]` and `Generator[Mapping[str, Any] | str]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1993:12
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2053:12
 ERROR `+` is not supported between `list[str]` and `Mapping[str, Any]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1993:12
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2030:37
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2053:12
 ERROR `+` is not supported between `list[str]` and `Generator[Mapping[str, Any] | str]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2040:12
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2096:12
 ERROR `+` is not supported between `list[str]` and `Mapping[str, Any]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2040:12
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2096:12
 ERROR Argument `list[object]` is not assignable to parameter `value` with type `Sequence[VariableBase]` in function `models.workflow.Workflow.conversation_variables` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner.py:62:39
 ERROR Argument `object` is not assignable to parameter `variable` with type `VariableBase` in function `models.workflow.ConversationVariable.from_variable` [bad-argument-type]
@@ -3120,158 +3106,136 @@
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:130:16
 ERROR Cannot index into `str` [bad-index]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:131:16
-ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:106:45
+ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:130:23
+ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:142:33
 ERROR `SimpleNamespace` is not assignable to attribute `_base_task_pipeline` with type `BasedGenerateTaskPipeline[Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:108:36
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:161:36
 ERROR Argument `Literal[core.app.entities.app_invoke_entities.InvokeFrom.WEB_APP]` is not assignable to parameter `invoke_from` with type `models.enums.InvokeFrom | SQLCoreOperations[models.enums.InvokeFrom | None] | None` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:125:21
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:244:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:246:23
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:298:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:300:23
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:375:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:377:23
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:442:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:444:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:178:21
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:482:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:330:45
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:504:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:352:45
 ERROR Argument `Literal['session']` is not assignable to parameter `type` with type `EndUserType | SQLCoreOperations[EndUserType]` in function `models.model.EndUser.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:606:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:490:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:611:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:495:23
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:617:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:501:33
 ERROR Argument `Literal[0]` is not assignable to parameter `message_unit_price` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:634:28
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:518:28
 ERROR Argument `Literal[0]` is not assignable to parameter `message_price_unit` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:635:28
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:519:28
 ERROR Argument `Literal[0]` is not assignable to parameter `answer_unit_price` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:638:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:522:27
 ERROR Argument `Literal[0]` is not assignable to parameter `answer_price_unit` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:639:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:523:27
 ERROR Argument `Literal[0]` is not assignable to parameter `total_price` with type `Decimal | SQLCoreOperations[Decimal | None] | None` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:642:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:526:21
 ERROR Argument `Literal[core.app.entities.app_invoke_entities.InvokeFrom.WEB_APP]` is not assignable to parameter `invoke_from` with type `models.enums.InvokeFrom | SQLCoreOperations[models.enums.InvokeFrom | None] | None` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:644:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:528:21
 ERROR Argument `Literal['api']` is not assignable to parameter `from_source` with type `ConversationFromSource | SQLCoreOperations[ConversationFromSource]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:645:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:529:21
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:733:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:617:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:741:60
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:625:60
 ERROR Argument `SimpleNamespace` is not assignable to parameter `publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._listen_audio_msg` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:786:57
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:670:57
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:801:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:685:65
 ERROR `(**kwargs: Sequence[NodeExecutionSnapshot] | str) -> Literal['started']` is not assignable to attribute `workflow_start_to_stream_response` with type `(*, task_id: str, workflow_run_id: str, workflow_id: str, reason: WorkflowStartReason, node_execution_snapshots: Sequence[NodeExecutionSnapshot] = ...) -> WorkflowStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:816:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:700:83
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_publishes_stop._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:850:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:734:66
 ERROR `SimpleNamespace` is not assignable to attribute `queue_manager` with type `AppQueueManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:851:54
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:735:54
 ERROR `(outputs: Unknown) -> list[dict[str, str]]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:864:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:748:79
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:867:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:751:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:877:64
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:761:64
 ERROR `(outputs: Unknown) -> list[@_]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:884:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:768:79
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:885:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:769:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:895:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:779:52
 ERROR `(outputs: Unknown) -> list[@_]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:901:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:785:79
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:902:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:786:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:914:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:798:52
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:915:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:799:52
 ERROR `(**kwargs: QueueIterationStartEvent | str) -> Literal['iter_start']` is not assignable to attribute `workflow_iteration_start_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueIterationStartEvent) -> IterationNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:922:93
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:806:93
 ERROR `(**kwargs: QueueIterationNextEvent | str) -> Literal['iter_next']` is not assignable to attribute `workflow_iteration_next_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueIterationNextEvent) -> IterationNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:925:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:809:92
 ERROR `(**kwargs: QueueIterationCompletedEvent | str) -> Literal['iter_done']` is not assignable to attribute `workflow_iteration_completed_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueIterationCompletedEvent) -> IterationNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:926:97
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:810:97
 ERROR `(**kwargs: QueueLoopStartEvent | str) -> Literal['loop_start']` is not assignable to attribute `workflow_loop_start_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueLoopStartEvent) -> LoopNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:929:88
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:813:88
 ERROR `(**kwargs: QueueLoopNextEvent | str) -> Literal['loop_next']` is not assignable to attribute `workflow_loop_next_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueLoopNextEvent) -> LoopNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:930:87
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:814:87
 ERROR `(**kwargs: QueueLoopCompletedEvent | str) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:931:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:815:92
 ERROR `(**kwargs: RuntimeState | int | str | None) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(*, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: RuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:993:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:877:84
 ERROR `(**kwargs: QueueWorkflowPausedEvent | RuntimeState | str) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(*, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: RuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:994:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:878:83
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:999:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:883:65
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['node_finish']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1025:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:909:89
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1060:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:944:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1066:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:950:47
 ERROR Argument `TestAdvancedChatGenerateTaskPipeline.test_handle_text_chunk_event_tracks_streaming_metrics._Publisher` is not assignable to parameter `tts_publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1066:68
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:950:68
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_message` with type `MessageQueueMessage | WorkflowQueueMessage | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1066:96
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:950:96
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_appends_token._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1087:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:971:66
 ERROR `(**kwargs: Unknown) -> Literal['replace']` is not assignable to attribute `message_replace_to_stream_response` with type `(answer: str, reason: str = '') -> MessageReplaceStreamResponse` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1118:78
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1002:78
 ERROR `(**kwargs: QueueHumanInputFormFilledEvent | str) -> Literal['filled']` is not assignable to attribute `human_input_form_filled_to_stream_response` with type `(*, event: QueueHumanInputFormFilledEvent, task_id: str) -> HumanInputFormFilledResponse` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1131:92
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1015:92
 ERROR `(**kwargs: QueueHumanInputFormTimeoutEvent | str) -> Literal['timeout']` is not assignable to attribute `human_input_form_timeout_to_stream_response` with type `(*, event: QueueHumanInputFormTimeoutEvent, task_id: str) -> HumanInputFormTimeoutResponse` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1132:93
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1016:93
 ERROR `() -> Literal['end']` is not assignable to attribute `_message_end_to_stream_response` with type `() -> MessageEndStreamResponse` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1191:52
+    --> tests/unit_tests/core/app/apps\n\n... (truncated) ...

codecov · 2026-10-23T17:26:04Z

Codecov Report

❌ Patch coverage is 97.69585% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.47%. Comparing base (430d9af) to head (0160fb5).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...nix/src/dify_trace_arize_phoenix/_arize_project.py	91.57%	5 Missing and 3 partials ⚠️
.../unit_tests/src/dify_trace_aliyun/test_provider.py	95.39%	6 Missing and 1 partial ⚠️
...ests/src/dify_trace_arize_phoenix/test_provider.py	97.96%	6 Missing and 1 partial ⚠️
api/domain/tracing/normalization.py	92.10%	3 Missing and 3 partials ⚠️
api/extensions/tracing.py	92.00%	2 Missing and 2 partials ⚠️
...e/app/apps/advanced_chat/generate_task_pipeline.py	72.72%	3 Missing ⚠️
...e-phoenix/src/dify_trace_arize_phoenix/provider.py	98.08%	2 Missing and 1 partial ⚠️
api/controllers/console/agent/tracing.py	71.42%	2 Missing ⚠️
api/core/telemetry/gateway.py	85.71%	1 Missing and 1 partial ⚠️
api/controllers/inner_api/agent/tracing.py	90.00%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #44673      +/-   ##
==========================================
- Coverage   89.54%   89.47%   -0.08%     
==========================================
  Files        6009     6001       -8     
  Lines      376984   370187    -6797     
  Branches    76320    75598     -722     
==========================================
- Hits       337559   331208    -6351     
+ Misses      33428    33189     -239     
+ Partials     5997     5790     -207

Flag	Coverage Δ
api	`89.26% <97.69%> (-0.13%)`	⬇️
cli	`94.28% <ø> (ø)`
dify-ui	`92.65% <ø> (ø)`
web	`89.50% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…face Replace the legacy ops trace queue, the unified trace dispatcher and the per-provider SDK exporters with one pipeline and one provider contract. - dify_trace_sdk (api/providers/trace/trace-sdk) defines the provider interface: verify(config), project_url(config) and send(config, trace). A provider-neutral Trace (parent-first spans, OpenTelemetry ids) and classified errors (retryable, rejected, outcome unknown). All network access goes through an injected Egress that enforces the SSRF policy. - Each provider package implements its own send and is discovered through dify.trace_providers entry points: Langfuse, LangSmith, Opik, Weave, Arize, Phoenix, Aliyun, MLflow, Databricks and Tencent. - Producers emit typed events through a TraceScope after their records commit. tasks.trace_delivery_task assembles records within the event's workspace, feeds enterprise telemetry and the Agent collector, and sends the canonical trace to the app's current provider, retrying only that send. - Tracing secrets are write-only: the console receives configured_secrets instead of masked values, blank secrets keep their stored value, and a changed destination requires entering secrets again. - Migration 53c10126d551 keeps one trace_app_config row per app and provider and adds a unique index. - Remove core/ops, the legacy gateway and dispatch modules, and the Opik, Weave, Langfuse and Arize/Phoenix SDK dependencies; declare the runtime dependencies they used to pull in transitively. - Move the tracing tests onto the mirrored test layout and drop the suites for deleted modules; the touched tracing components use logical Tailwind classes.

…behind ports Align the tracing slice with the layered application-service architecture. - Application services receive ports and plain settings at construction: TraceScopes, TraceDeliveryService, the provider configuration gateway and the Agent trace collector no longer open sessions, read configuration, or reach Celery, storage or the encrypter themselves. The module-level service factories are gone; extensions/application_services/tracing.py builds every tracing service and is reachable as application_services().tracing. - Trace records and Agent collection records are plain values. Repositories map rows to records before their sessions close: trace record assembly moved to repositories/trace_record_repository.py, the Agent trace ledger returns run, invocation and source records, and the tracing configuration repository answers the pipeline's policy and destination queries. Database failures surface as TracingStoreUnavailableError. - Runtime adapters implement the ports: the Celery emitter and retry queue, the entry-point provider catalog, workspace secret encryption, enterprise telemetry and Agent export scheduling. - Tracing errors live in services/tracing/errors.py without HTTP statuses; the Agent tracing handlers map them, and handlers and Celery tasks obtain tracing services from the composition root. - Hooks are bound only when Agent tracing is enabled, so no tracing code reads its configuration outside the composition root. - Import contracts keep the tracing application modules framework and persistence neutral, the tracing handlers off persistence, and the tracing repositories on the contracts they implement. - Tests use in-memory port implementations and real repositories instead of patching module internals.

…factor-plan-1d5fc4 # Conflicts: # api/tests/unit_tests/core/app/apps/workflow/test_app_generator.py

…factor-plan-1d5fc4 # Conflicts: # api/tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py # api/tests/unit_tests/core/app/apps/workflow/test_app_generator.py

Follow the backend architecture guide (#44957). An Agent trace run is now TraceRunAgg in api/domain/tracing, with its invocations, batches and receipts as child entities. Terminal recovery is a domain service of named functions, and domain errors are translated at the application boundary so error codes stay unchanged. The repository loads and saves whole runs under the run's row lock. A domain import contract keeps the package free of application, persistence, framework and legacy kernel code.

…them A workflow run started by a workflow tool node now joins the trace of the run that started it, hanging from that tool node, when both apps export to the same destination. Otherwise it keeps a trace of its own that names its outer run. Workflow tool nodes pass the chain of outer runs down to the runs they start, so a run nested at any depth joins the outermost trace it shares a destination with. Delivery reads the outer runs' committed records and applies a pure domain placement rule. An Agent-collected outer run lends its frozen trace once collection has started. Trace units can now hang from any span of their trace. LangSmith hangs them from the trace root, because its run order needs the full ancestry, and MLflow still sends them as traces of their own.

A workflow tool node passes its run's chained parent trace context to the tool it invokes. Exercising that hand-off keeps the KnowledgeFS changed-glue coverage gate above its minimum.

This PR keeps the trace pipeline refactor as its goal. Revert the nested workflow placement (dfbccb8) and the test that covered its runtime hand-off (34286bc). A nested workflow run keeps its own trace, linked to its outer run by attributes, as before.

laipz8200 requested review from a team October 23, 2026 17:22

laipz8200 requested review from JohnJyong, MRZHUH, QuantumGhost, crazywoola, hyoban, iamjoel, lyzno1 and snakevash as code owners October 23, 2026 17:22

github-actions Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. web This relates to changes on the web. labels Oct 23, 2026

laipz8200 added 2 commits October 28, 2026 01:20

laipz8200 force-pushed the claude/ops-tracing-refactor-plan-1d5fc4 branch from fb3c16a to d519b46 Compare October 27, 2026 17:36

laipz8200 added 5 commits October 28, 2026 22:11

Merge remote-tracking branch 'origin/main' into claude/ops-tracing-re…

c215dc0

…factor-plan-1d5fc4 # Conflicts: # api/tests/unit_tests/core/app/apps/workflow/test_app_generator.py

Merge remote-tracking branch 'origin/main' into claude/ops-tracing-re…

6c86b08

…factor-plan-1d5fc4 # Conflicts: # api/tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py # api/tests/unit_tests/core/app/apps/workflow/test_app_generator.py

test(workflow): cover the parent trace context handed to workflow tools

34286bc

A workflow tool node passes its run's chained parent trace context to the tool it invokes. Exercising that hand-off keeps the KnowledgeFS changed-glue coverage gate above its minimum.

laipz8200 enabled auto-merge October 28, 2026 21:05

Navigation Menu

Repository navigation

refactor(tracing): one trace pipeline behind a minimal provider interface - #44673

refactor(tracing): one trace pipeline behind a minimal provider interface#44673
laipz8200 wants to merge 8 commits into
mainfrom
claude/ops-tracing-refactor-plan-1d5fc4

laipz8200 commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

laipz8200 commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Provider interface (api/providers/trace/trace-sdk, imported as dify_trace_sdk)

Pipeline

Configuration

Removed

Architecture

Behaviour changes worth reviewing

Screenshots

Checklist

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyrefly Type Coverage

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyrefly Diff

Uh oh!

codecov Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

laipz8200 commented Oct 23, 2026 •

edited

Loading

Provider interface (`api/providers/trace/trace-sdk`, imported as `dify_trace_sdk`)

github-actions Bot commented Oct 23, 2026 •

edited

Loading

github-actions Bot commented Oct 23, 2026 •

edited

Loading

codecov Bot commented Oct 23, 2026 •

edited

Loading