feat: fetch-url supports inline images (PNG/JPEG/GIF/WebP) - #2149
feat: fetch-url supports inline images (PNG/JPEG/GIF/WebP)#2149bj456736 wants to merge 8 commits into
Conversation
When vis-server binds to 0.0.0.0 or :: (all interfaces), the startup banner and CLI output now display the actual LAN IP addresses that other devices on the same network can use to connect. This enables lan-range remote control from phones, tablets, or other machines. Changes: - Add isAllInterfaces() and getLocalNetworkAddresses() to config.ts - Add lanUrls field to StartedVisServer in start.ts - Update formatStartupBanner() to list LAN URLs when applicable - Update vis CLI handler to print LAN access URLs - Add unit tests for isAllInterfaces and getLocalNetworkAddresses
- Extend UrlFetchResult with new 'image' kind and imageUrl field - LocalFetchURLProvider detects image content-types, reads body as arrayBuffer, converts to base64 data URL via existing image-format-policy helpers - FetchURLTool returns image as ContentPart[] (text + image_url) for LLM consumption - Covers both agent-core and agent-core-v2 - Added tests for image conversion, MIME normalization, and oversized rejection
🦋 Changeset detectedLatest commit: ceed3fb The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ceed3fb52f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| isError: false, | ||
| output: [ | ||
| { type: 'text', text: content }, | ||
| { type: 'image_url', image_url: { url: imageUrl } }, |
There was a problem hiding this comment.
When FetchURL returns an image, this constructs a content part with image_url, but the repository ContentPart shape uses imageUrl, and the existing media tools/provider encoders read part.imageUrl.url. With this shape, fetched images will not serialize as valid image parts for the next model request; the legacy copy in packages/agent-core/src/tools/builtin/web/fetch-url.ts needs the same fix.
Useful? React with 👍 / 👎.
| // Handle accepted image formats directly — convert to base64 data URL | ||
| // so the model can view them inline. | ||
| const normalizedMime = normalizeImageMime(contentType); | ||
| if (MODEL_ACCEPTED_IMAGE_MIMES.has(normalizedMime)) { |
There was a problem hiding this comment.
When an upstream response labels unsupported or non-image bytes as an accepted Content-Type such as image/png, this header-only check accepts it and builds a data:${normalizedMime} image result. Existing media policy sniffs the effective MIME because providers decode bytes and reject formats such as AVIF/HEIC even when mislabeled, so this can persist a bad image part in session history; buffer the body, sniff/canonicalize the MIME, and refuse unsupported bytes before returning kind: 'image' here and in the legacy provider.
Useful? React with 👍 / 👎.
1 participant
Summary
Extends the "FetchURL" tool to support fetching image URLs and returning them inline to the LLM via ContentPart[] (text + image_url).
Changes
Why this approach
Minimal, backward-compatible change: existing text/HTML fetching is untouched; image URLs simply get a new "kind" and the tool output shape adapts to what the LLM already understands (ContentPart[] with image_url type).