fix: tolerant Konva hit-detection for ±1 getImageData color drift (#9810)

[sbatchelder]

This pull request fixes a bug where clicking a resize handle or a region in the image editor frequently misses - the click falls through and instead draws a new box, drags the underlying annotation, or does nothing, and the resize cursor never appears. The cause seems to be that some under canvas implementations, getImageData rounds color channel values by ±1 per channel, which breaks Konva's exact color-key hit detection. The fix makes that hit lookup tolerant of ±1 drift, and only does extra work when Konva's exact lookup has already missed.

Bug fix:

• Added konva-hit-tolerance.ts, which wraps Konva.Layer.prototype._getIntersection so that, on an opaque-pixel miss, it searches the 26 color keys within ±1 RGB of the read-back pixel and resolves to that shape. Exact hits are untouched.
• Installed the patch once in ImageView.jsx so it applies to the image editor's Konva stage.

Test:

• Added konva-hit-tolerance.test.ts, which drives the patched lookup against a minimal Konva stand-in (no canvas/GPU needed) and covers the exact-match, ±1-drift, no-neighbor, transparent-pixel, channel-boundary, idempotency, and missing-internals cases.

Fixes #9810.

Reason for change

Problem. In the image editor, clicking a Transformer resize handle or clicking a region to select it frequently misses: the pointer "clicks through" the target and instead starts a new selection box or drags the underlying rectangle, and the resize cursor never appears. On an annotation with several regions, some of the 8 resize anchors work and others don't, and which ones work changes every time the same box is re-selected. It is worst on large images. It is possibly worse also when there are 100+ annotations or highly zoomed in.

See #9810 for a video of the misbehaving handles.

Root cause. Konva identifies the shape under the pointer by drawing every shape onto an offscreen hit canvas in a unique color key, then reading that one pixel back with getImageData and looking the exact color up in Konva.shapes. This depends on getImageData returning the exact bytes that were written. On some canvas implementations (reproduced on Chrome/Chromium/Brave with hardware-accceleration enabled), getImageData rounds solid colors by ±1 per channel. The exact-color lookup then misses and the click falls through. I speculate that small targets, like resize handles, are more strongly affected by rounding-effect, and non-integer stage scale (zoom) makes the miss rate worse. Konva's own recovery mode (a spiral search to neighbouring pixel positions) doesn't help, because the whole anchor footprint reads the same drifted color.

Solution. Wrap Konva.Layer.prototype._getIntersection so that, when Konva's exact lookup returns no shape for an opaque pixel, it searches the 26 color keys within ±1 RGB of the read-back value and resolves to that shape. The true shape's key is by definition one of those neighbours (the read-back drifted from it by ≤1). Color keys are spread across 16.7M values, so the odds a ±1 neighbour is a different registered shape are negligible (~0.02% worst case). The patch:

• runs the extra search only on a miss, so healthy exact hits pay nothing;
• only considers fully-opaque pixels, leaving real antialiased edges alone;
• bails out safely (no-op) if Konva's internals aren't shaped as expected (e.g. after a version upgrade), so it can never break hit detection;
• is idempotent. Installing it more than once does nothing.

This is complementary to, but distinct from, the "handles are too small" framing in #4558, #4452, and PR #9700: enlarging a handle's screen-space hit area doesn't help when the entire enlarged footprint reads a uniformly-drifted color and still returns no key.

Screenshots / recording

The change is behavioral. The canvas looks identical. The recording below shows, on a large image with RectangleLabels regions:

• Before: grabbing resize handles is unreliable; the resize cursor often doesn't appear and the click drags/redraws instead. Which handles work changes on each re-select. See video from [bug] Resize handles and region selection often click-thru #9810
• After: every handle grabs on the first click, consistently, across repeated re-selects, and region-selection clicks land reliably.

2026-07-08.20-31-44.mp4

Rollout strategy

No feature flag or environment variable. The patch is inert on canvases that don't drift (Konva's exact lookup succeeds and the extra path never runs), and it only ever adds a hit where Konva would otherwise have returned none. It never removes or changes an existing hit. It ships on by default. If the team would prefer it behind a flag_set guard for a staged rollout, it's a one-line wrap to add.

Testing

Acceptance criteria (QA):

• When a user selects a region on a large image and clicks any of its 8 resize handles, then that handle grabs and resizes on the first click consistently; including across repeated re-selects, and regardless of browser hardware-acceleration or zoom.
• When a user clicks empty canvas, then no region is selected (no spurious hits are introduced).

Automated tests. Added web/libs/editor/src/utils/__tests__/konva-hit-tolerance.test.ts, driving the patched lookup against a minimal Konva stand-in (no real canvas/GPU required). It covers:

• exact color-key matches are left untouched;
• a pixel that drifts by ±1 per channel resolves to the correct shape;
• an opaque pixel with no ±1 neighbor registered does not invent a hit;
• transparent / partial-alpha pixels (real edges) are ignored;
• the channel search clamps at the 0/255 boundaries;
• the patch is idempotent (no double-wrapping);
• it no-ops safely when Konva's internals are missing.

Manual verification. Reproduced the miss on a recent Chromium build on Linux with a large image and confirmed the fix restores reliable handle grabs and region selection.

Risks

• Correctness: low. The patch only runs on a miss and can only add a hit that Konva would otherwise have missed; it never changes an exact hit. The one theoretical downside is a ~0.02%-worst-case single-pixel mishit to an adjacent color key, which is far less disruptive than the current dropped clicks.
• Performance: negligible. Healthy hits are untouched; the ≤26-key search runs only when the exact lookup already failed, i.e. on the pixel under a single pointer event.
• Security: none.
• Maintainability: the patch monkey-patches a Konva internal (_getIntersection), so a future Konva upgrade could change its shape. The code guards against that by feature-detecting and no-oping rather than throwing.

Reviewer notes

• The patch is scoped to the image editor by installing it from ImageView.jsx; it wraps the prototype once and is idempotent, so import order and re-mounts are safe.
• The monkey-patch approach was chosen because the drift is in the browser's getImageData, below Konva's public API — there is no prop or option to make the color-key lookup tolerant. The wrap is deliberately minimal and defensive.
• Complementary to PR feat: Scale resize handle hit area for small bounding boxes #9700; this addresses the underlying color-key miss rather than handle geometry.

General notes

• Background on Konva's hit detection: shapes are drawn to an offscreen hit canvas in unique color keys and looked up by reading back the pixel under the pointer. The whole class of "clicks fall through on GPU canvases" bugs stems from getImageData not returning the exact bytes written.
• The konva-hit-tolerance.ts module docstring documents the mechanism, the ±1 drift, and why the fix is safe, for future maintainers.

[netlify]

👷 Deploy request for label-studio-docs-new-theme pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	0806de2

[netlify]

👷 Deploy request for heartex-docs pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	0806de2

[netlify]

✅ Deploy Preview for label-studio-storybook ready!

Name	Link
🔨 Latest commit	0806de2
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-storybook/deploys/6a4eec9256fa2d0009b8e3fb
😎 Deploy Preview	https://deploy-preview-9813--label-studio-storybook.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[netlify]

✅ Deploy Preview for label-studio-playground ready!

Name	Link
🔨 Latest commit	0806de2
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-playground/deploys/6a4eec92b4bc96000970f07e
😎 Deploy Preview	https://deploy-preview-9813--label-studio-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.