fix: filter agent files by type-specific model feature instead of vision alone by rkfshakti · Pull Request #39478 · langgenius/dify

rkfshakti · 2026-07-23T17:25:44Z

Related Issues/PRs

What changes are proposed in this pull request?

When a model does not declare the VISION feature (e.g. qwen-long), all uploaded files were silently dropped because the agent runner gated self.files on a single ModelFeature.VISION check. This meant document files like Excel spreadsheets never reached the LLM even when the model supported the DOCUMENT feature.

The fix filters each file by the model feature its type requires, mirroring the existing _REQUIRED_MODEL_FEATURE_BY_CONTENT_TYPE mapping in graphon:

IMAGE → VISION
DOCUMENT → DOCUMENT
AUDIO → AUDIO
VIDEO → VIDEO

This ensures non-image files are passed through when the model supports them, instead of being dropped just because the model lacks vision.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

New regression test test_init_filters_files_by_type_not_vision verifies that a document file is kept when the model supports DOCUMENT but not VISION, while an image file is correctly filtered out.

Does this PR require documentation update?

No.

Does this PR require updating the MLflow Skills repository?

No.

Release Notes

Is this a user-facing change?

Yes. Files uploaded to agents are no longer silently dropped when the model does not support vision but does support the file type (e.g. documents).

What component(s), interfaces, languages, and integrations does this PR affect?

Components

area/tracking: Tracking Service, tracking client APIs, autologging

How should the PR be classified in the release notes? Choose one:

rn/bug-fix - A user-facing bug fix worth mentioning in the release notes

Is this PR a critical bugfix or security fix that should go into the next patch release?

This PR is critical and needs to be in the next patch release
This PR can wait for the next minor release

github-actions · 2026-07-23T17:28:20Z

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	79.95%	79.95%	-0.00%
Strict coverage	79.64%	79.64%	-0.00%
Typed symbols	76,881	76,883	+2
Untyped symbols	19,351	19,352	+1
Modules	3984	3984	0

github-actions · 2026-07-23T17:28:39Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-10-28 19:33:10.300352418 +0000
+++ /tmp/pyrefly_pr.txt	2026-10-28 19:32:49.585166037 +0000
@@ -2923,65 +2923,65 @@
 ERROR Argument `None` is not assignable to parameter `query` with type `str` in function `core.agent.fc_agent_runner.FunctionCallAgentRunner._organize_user_query` [bad-argument-type]
    --> tests/unit_tests/core/agent/test_fc_agent_runner.py:339:46
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:407:58
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:416:58
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:455:19
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:464:19
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:512:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:521:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:514:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:523:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:516:33
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:525:33
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:517:33
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:526:33
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:518:16
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:527:16
 ERROR Object of class `FunctionType` has no attribute `assert_called` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:523:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:532:9
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:525:23
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:534:23
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:531:19
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:540:19
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:548:57
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:557:57
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:554:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:563:9
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:570:57
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:579:57
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:578:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:587:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:580:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:589:35
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:586:46
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:595:46
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:589:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:598:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:591:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:600:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:593:16
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:602:16
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:609:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:618:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:620:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:629:35
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:636:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:645:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:638:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:647:35
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:653:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:662:9
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:658:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:667:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:667:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:676:35
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:673:25
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:682:25
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:677:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:686:9
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:689:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:698:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:692:29
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:701:29
 ERROR Missing argument `description` in function `core.agent.plugin_entities.AgentStrategyEntity.__init__` [missing-argument]
    --> tests/unit_tests/core/agent/test_plugin_entities.py:292:32
 ERROR Argument `dict[str, dict[str, str | dict[str, float | list[str]]]]` is not assignable to parameter `config` with type `AppModelConfigDict` in function `core.app.app_config.easy_ui_based_app.model_config.manager.ModelConfigManager.convert` [bad-argument-type]

rkfshakti · 2026-07-26T12:26:53Z

Friendly ping — this PR has been open for a few days with all CI checks passing. The fix filters agent files by the type-specific model feature (e.g. audio, video, document) rather than relying on vision alone, which prevents unsupported file types from being sent to models that only support a subset of multimodal inputs. Would appreciate a review when you get a chance.

rkfshakti · 2026-07-29T14:29:28Z

Hi maintainers — fixed the failing tests by using the correct File constructor arguments (file_type= and FileTransferMethod enum). CI should be green now. This PR filters agent files by type-specific model feature instead of vision alone, fixing #39431. Would appreciate a review when time allows. Thanks!

rkfshakti · 2026-07-29T15:14:15Z

Fixed the Pyrefly type error in the test — runner.files = ["file1"] was assigning list[str] to an attribute typed list[File]. Replaced with a proper File object. CI should be fully green now.

rkfshakti · 2026-08-01T15:16:35Z

Friendly ping — this PR has been open for 9 days. The fix filters agent files by type-specific model feature instead of vision alone, so non-vision model capabilities are correctly handled. Would appreciate a review when time allows.

rkfshakti · 2026-08-10T17:14:27Z

Friendly ping — this PR has been open for over a week. CI is passing. Would appreciate a review when time allows. Thanks!

rkfshakti · 2026-08-13T15:22:18Z

Hi maintainers — just a friendly nudge on this one. The fix for #39431 (filtering agent files by type-specific model feature instead of vision alone) has all CI passing, including the Pyrefly type checks I fixed. I'm excited to see it land — it stops files from being dropped for models that support them. Would appreciate a review when you have a moment. Thanks!

rkfshakti · 2026-08-19T16:25:04Z

Hi maintainers — gentle bump on this agent files filter fix. Filters agent files by type-specific model feature instead of vision alone, so non-vision models that accept files aren't blocked. CI is green. Would appreciate a review. Thanks!

rkfshakti · 2026-09-10T11:14:05Z

Rebased onto latest main — it had grown a conflict after the SQLite test migration of test_base_agent_runner.py (#40508). The regression test was ported to the new module-level style, the file-filter logic was made tolerant of non-File placeholder entries, and the full agent test suite passes locally (236 tests). Diff is now 0 commits behind main and conflict-free. Ready for re-review.

codecov · 2026-09-19T16:03:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.62%. Comparing base (ea5e4e9) to head (86db759).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #39478   +/-   ##
=======================================
  Coverage   59.61%   59.62%           
=======================================
  Files        6010     6010           
  Lines      377012   377015    +3     
  Branches    76335    76335           
=======================================
+ Hits       224766   224780   +14     
+ Misses     146248   146238   -10     
+ Partials     5998     5997    -1

Flag	Coverage Δ
api	`89.40% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rkfshakti · 2026-09-27T16:39:22Z

Hi team — gentle nudge on this one. It's rebased and 0 conflicts, CI fully green (37 checks), and no unresolved threads. The fix for #39431 is small: it filters agent files by the type-specific model feature instead of vision alone, so non-vision models that accept files aren't incorrectly blocked. Happy to adjust anything on review — thanks!

…ion alone When a model does not declare the VISION feature (e.g. qwen-long), all uploaded files were silently dropped because the agent runner gated self.files on a single ModelFeature.VISION check. This meant document files like Excel spreadsheets never reached the LLM even when the model supported the DOCUMENT feature. Filter each file by the model feature its type requires: IMAGE→VISION, DOCUMENT→DOCUMENT, AUDIO→AUDIO, VIDEO→VIDEO. Non-File placeholder entries (as used in some existing tests) pass through unfiltered to avoid breaking callers that do not supply real File objects. Rebased onto latest main (post SQLite test migration); the regression test was ported to the new module-level style and verifies documents are kept and images filtered when the model supports DOCUMENT only. Fixes langgenius#39431

rkfshakti requested review from QuantumGhost and laipz8200 as code owners July 23, 2026 17:25

dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jul 23, 2026

rkfshakti force-pushed the fix/agent-files-dropped-without-vision-39431 branch from d9f62a5 to 755ff0e Compare July 28, 2026 18:18

rkfshakti force-pushed the fix/agent-files-dropped-without-vision-39431 branch from faef923 to a1420de Compare August 11, 2026 02:18

rkfshakti mentioned this pull request Aug 14, 2026

fix: replace generated column with partial unique index for PG logical replication #39473

Open

rkfshakti force-pushed the fix/agent-files-dropped-without-vision-39431 branch from a1420de to f6f9582 Compare September 10, 2026 11:13

rkfshakti force-pushed the fix/agent-files-dropped-without-vision-39431 branch 4 times, most recently from ef14998 to c63627a Compare September 19, 2026 15:54

rkfshakti force-pushed the fix/agent-files-dropped-without-vision-39431 branch from c63627a to b75166c Compare September 23, 2026 16:33

github-actions Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Sep 23, 2026

rkfshakti force-pushed the fix/agent-files-dropped-without-vision-39431 branch from b75166c to 86db759 Compare October 28, 2026 19:25

Navigation Menu

Repository navigation

fix: filter agent files by type-specific model feature instead of vision alone - #39478

fix: filter agent files by type-specific model feature instead of vision alone#39478
rkfshakti wants to merge 1 commit into
langgenius:mainfrom
rkfshakti:fix/agent-files-dropped-without-vision-39431

rkfshakti commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

rkfshakti commented Jul 26, 2026

Uh oh!

rkfshakti commented Jul 29, 2026

Uh oh!

rkfshakti commented Jul 29, 2026

Uh oh!

rkfshakti commented Aug 1, 2026

Uh oh!

rkfshakti commented Aug 10, 2026

Uh oh!

rkfshakti commented Aug 13, 2026

Uh oh!

rkfshakti commented Aug 19, 2026

Uh oh!

rkfshakti commented Sep 10, 2026

Uh oh!

codecov Bot commented Sep 19, 2026 •

edited

Loading

Uh oh!

rkfshakti commented Sep 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rkfshakti commented Jul 23, 2026

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Does this PR require updating the MLflow Skills repository?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Is this PR a critical bugfix or security fix that should go into the next patch release?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyrefly Type Coverage

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyrefly Diff

Uh oh!

rkfshakti commented Jul 26, 2026

Uh oh!

rkfshakti commented Jul 29, 2026

Uh oh!

rkfshakti commented Jul 29, 2026

Uh oh!

rkfshakti commented Aug 1, 2026

Uh oh!

rkfshakti commented Aug 10, 2026

Uh oh!

rkfshakti commented Aug 13, 2026

Uh oh!

rkfshakti commented Aug 19, 2026

Uh oh!

rkfshakti commented Sep 10, 2026

Uh oh!

codecov Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rkfshakti commented Sep 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

codecov Bot commented Sep 19, 2026 •

edited

Loading