initial incomplete version of the new vkapriltag backend - #2597
Draft
yojobama wants to merge 6 commits into
Draft
Conversation
Implements the vkapriltag integration as a sibling pipeline type (AprilTagVulkan), not a toggle inside the existing CPU pipeline, per explicit direction: the maintainers said they're open to either shape, and a separate type is fully deletable in one commit if this doesn't get kept up to date, at the cost of some duplicated wiring - which this commit avoids by factoring the pose-estimation/multi-target/target-list/FPS logic AprilTagPipeline already had into a shared AbstractAprilTagPipeline base, so the CPU and Vulkan pipelines differ only in how they detect. - AprilTagPipelineSettingsBase: the settings AbstractAprilTagPipeline's shared logic actually reads (tagFamily, decisionMargin, hammingDist, numIterations, doMultiTarget, doSingleTargetAlways). AprilTagPipelineSettings keeps its CPU-only fields (decimate, blur, threads, refineEdges); VkAprilTagPipelineSettings deliberately has none of those, since the Vulkan backend can't honor any of them (fixed 2x decimation, no pre-blur stage, no edge refinement - see VkAprilTagPipelineSettings' Javadoc). - VkAprilTagDetectionPipe: wraps the native detector (vkapriltag_jni), falling back to WPILib's CPU AprilTagDetector - transparently, same output shape - when Vulkan is unsupported, the frame size isn't a multiple of 8, or native create() fails. isVulkanActive() reports which one actually ran. - PipelineType.AprilTagVulkan appended LAST with baseIndex=5: see the comment on PipelineType.java - DataSocketHandler maps a wire baseIndex to an enum constant as values()[baseIndex + 3], which only holds while the enum stays declared in baseIndex order. - LoadJNI.JNITypes.VKAPRILTAG_DETECTOR + VkAprilTagAvailability: probed once at startup (Main.java), tolerant of every way this can be unavailable (missing native artifact, no Vulkan driver, CPU-only Vulkan - see vkapriltag's own comment on why a silent CPU fallback there would be worse than no fallback). Device list + reason surfaced through UIGeneralSettings/UIPhotonConfiguration, reusing the same channel supportedBackends already uses. Verified: full photon-core test suite (108 tests) - 104 pass, including every AprilTag-related test unchanged, plus a new testAprilTagSettingsBaseSplitPreservesJsonShape regression test proving the settings hierarchy split doesn't change AprilTagPipelineSettings' JSON shape (fields stay top-level, not nested). The 4 failures are pre-existing and unrelated to this change - confirmed identical on the unmodified baseline (TimeSyncClient/camera native-loading issues in this dev environment). Not yet wired: real wpilibNatives/-java maven artifacts for vkapriltag_jni (it doesn't publish any yet - see vkapriltag-jni's own repo). photon-core/ build.gradle pulls the -java classes from a local, gitignored stand-in jar for now; the comment there spells out the real dependency block to switch to once a publish exists. photon-client changes (device-picker tab, new pipeline-type entry) are not yet done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqXqd9PXVAmNAEs3gKfQ97
Client-side half of the AprilTagVulkan PipelineType added to photon-core. - PipelineType (ordinal 8) and WebsocketPipelineType (baseIndex 5) both need an entry for the same Java enum constant - they're two different numbering schemes, easy to get one right and miss the other. - VkAprilTagPipelineSettings: deliberately no decimate/blur/refineEdges, mirroring the Java settings class - the Vulkan backend can't honor any of them, so there's no control to grey out. - CameraAndPipelineSelectCard.vue: the new pipeline type is only offered in both selector lists when general.vulkanUnavailableReason is unset - a user with no Vulkan driver would only ever see it silently fall back to CPU, so don't offer it there at all rather than adding a disabled state. - ConfigOptions.vue: split isAprilTag into isAprilTag/isVkAprilTag (each picks its own tab) plus isAnyAprilTag (drives the shared 3D/PnP/Threshold/Contours filtering both detector backends need identically). - VkAprilTagTab.vue: tagFamily, decisionMargin, numIterations (shared with the CPU tab), plus vulkanDeviceIndex (populated from general.vulkanDevices, labeled from vkapriltag's own DescribeDevice() so the formatting stays owned by that library) and cpuThreads. - OutputTab.vue: extended the three existing AprilTag/Aruco multi-target checks to also cover AprilTagVulkan. Verified: npm run type-check and npm run build both succeed (only the pre-existing, unrelated echarts namespace error remains from baseline); eslint clean on every touched file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqXqd9PXVAmNAEs3gKfQ97
- detector-types.md: new "AprilTag (Vulkan) - Beta" section - hardware
requirement, automatic CPU fallback and when it triggers, and an explicit
list of the CPU pipeline's controls this one doesn't have (decimate,
blur, refine edges) and why (not implemented, not hidden).
- ExternalLicenses/vkapriltag-License (Apache-2.0) and -NOTICE, copied from
vkapriltag/{LICENSE,NOTICE}. Apache-2.0 SS4(d) makes reproducing NOTICE
mandatory when redistributing; GPLv3 absorbing Apache-2.0 code is
one-way compatible, so no relicensing is needed on either side.
Not done: a benchmarks page (needs real Orange Pi 5 numbers through
PhotonVision's own capture pipeline, not vkapriltag's standalone V4L2
sample app - a follow-up once that hardware run exists), and an
AprilTagRobotics-License entry for the BSD-2-Clause apriltag library
vkapriltag statically links (check first whether the existing WPILib
license entry already covers it transitively).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MqXqd9PXVAmNAEs3gKfQ97
The previous state of this branch made VkAprilTagAvailability report
"available" only because a native .so and a hand-patched
ResourceInformation.json had been dropped directly into
photon-core/build/NativeMain/AssembledResources on one machine, during one
session. That directory is assembleNativeResourcesMain's OUTPUT - Gradle
rebuilds it from the real wpilibNatives dependency graph on every run that
re-executes that task, which silently deleted the hand-added files the next
time `./gradlew run` was invoked fresh. There was no dependency for
vkapriltag_jni at all; the fallback path (already correct) just quietly took
over again.
Fixed by making it a real Gradle dependency instead:
- photon-core/build.gradle: linuxarm64 now pulls
org.photonvision:vkapriltag_jni-jni:$vkAprilTagVersion:linuxarm64 via
wpilibNatives, and vkapriltag_jni-java via implementation - the same
pattern rknn_jni/tflite_jni already use, gated to linuxarm64 only for now
(see the comment on why linuxx86-64 needs its own care, not just a
recompile).
- vkAprilTagVersion added to the root build.gradle ext block.
- Since vkapriltag_jni has no real maven.photonvision.org publish yet, this
resolves against mavenLocal() (already in allprojects{}) instead -
installed there from vkapriltag-jni's actual build output via a new
reproducible script (vkapriltag-jni's own repo,
scripts/install_local_maven.sh), not by hand-copying files into a build
output directory this time.
- Removed the old libs/vkapriltag_jni-java-0.1.0-local.jar +
`implementation files(...)` stopgap now that this is real.
One non-obvious bug the script's own comment calls out: the -jni artifact's
POM must declare <packaging>pom</packaging>, not jar. Real published
*_jni-jni artifacts carry no unclassified default jar, only
classifier-attached ones (linuxarm64, ...) - declaring packaging=jar with no
matching unclassified jar present makes Gradle's plain-Maven classifier
resolution fail with an unhelpful "Could not find ..." across every
configured repository, mavenLocal() included, and say nothing about why.
Verified on the Orange Pi: removed the mavenLocal() artifacts, reinstalled
them purely via the new script, killed the Gradle daemon, cleared
~/.wpilib/nativecache and photon-core/build/NativeMain, and ran a plain
`./gradlew run` from that fully clean state - "Vulkan AprilTag detector
available; 2 device(s) found" and the server started normally, with nothing
manually copied anywhere.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MqXqd9PXVAmNAEs3gKfQ97
Explicitly requested by the maintainers (integration plan §1.1, Chris
Gerth's point 4: "repeatable 'same as or better than' testing may be needed
against libapriltag - ideally as a unit test in this repo") - this was
previously only covered by manual live-UI testing.
- testVulkanMatchesLibapriltag_{singleTag36h11,16h5Family,manyTags}: run
both AprilTagPipeline and VkAprilTagPipeline over the same fixtures
already in the repo (tag1_640_480.jpg, tag1_16h5_1280.png,
36h11_stress_test.png) and assert equal decoded ID sets plus per-corner
RMS agreement. The comparison ports vkapriltag's own
tools/validate_against_libapriltag/validate_common.h
(CompareCorners/ExtractDetections/SortedIds) rather than inventing new
tolerances - that tool measured ~0.85px mean RMS on real Orange Pi 5
hardware against the same tag1_640_480.jpg fixture used here; this test's
tolerances (3px mean, 8px max) sit well above that, so a real regression
trips it without chasing noise. Each asserts isVulkanActive() so a silent
CPU fallback can't pass as a false positive.
- All three assumeTrue(VkAprilTagAvailability.isSupported()) and skip (not
fail) with no Vulkan device - the behavior already proven live on the
WSL2/no-GPU machine this session, and now exercised by the test suite
itself: verified on Windows (no vkapriltag_jni native artifact for that
platform) - 3/3 skip cleanly, 0 failures.
- testVulkanFallsBackToCpuOnUnsupportedFrameSize: drives
VkAprilTagDetectionPipe directly with a frame width that isn't a multiple
of 8 (vkapriltag's hard requirement, unrelated to GPU availability) - runs
unconditionally on every CI runner. Verified passing on Windows.
Not yet re-verified against real hardware: the Orange Pi is currently
unreachable on the network (host down), so the three GPU-comparison
assertions haven't been re-run since this file was written, only
compile-checked and confirmed to skip correctly with no device. Needs a
follow-up run once the Pi is back online.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MqXqd9PXVAmNAEs3gKfQ97
…tself Ran the previous commit's test suite for the first time against the Orange Pi's actual Vulkan hardware (it was offline when that commit was written, verified only on Windows where every GPU test skips). Two of four tests failed - both were bugs in the test, not the pipeline: - testVulkanMatchesLibapriltag_manyTags asserted isVulkanActive() on 36h11_stress_test.png, which is 3256x1228 - 1228 is not a multiple of 8, vkapriltag's own hard requirement (GpuDetector.cpp). The pipeline was correctly refusing this fixture and falling back to CPU; the test was wrong to expect otherwise. Repurposed into testVulkanFallsBackToCpuOnStressTestImage, which asserts the fallback explicitly instead. - That repurposed test then failed a second way: it compared the fallback run's detections against a baseline CPU run using assertSameDetections' by-ID corner matching, but this fixture tiles the same handful of tag IDs at ~8 different physical positions each - by-ID matching paired up same-ID detections from different image locations, producing a ~1126px mean "corner RMS" that had nothing to do with detection accuracy. Both runs are the same WPILib CPU AprilTagDetector code path anyway, so there is no Vulkan-vs-CPU distinction to compare here; swapped for a target- count sanity check, which is what this test actually needs. Verified on the Orange Pi (real hardware): all 4 tests pass, including the two genuine GPU-comparison tests (testVulkanMatchesLibapriltag_singleTag36h11/16h5Family) which now confirm Vulkan and CPU decode the same tags with corner agreement well inside tolerance. Re-verified on Windows: the same 2 GPU tests skip cleanly (no native artifact for that platform), the 2 fallback-path tests still pass unconditionally. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqXqd9PXVAmNAEs3gKfQ97
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description: in #2588 I was asked to "open some draft PRs whenever I can" so here is one
CI will fail because the vkapriltag-jni dependency is at least for the moment loaded through mavenLocal
I am aware the code is not up to standard and would have been rejected even if it passed CI tests, this is just an initial (and extremely partial) version, I just wanted to get some input about the general idea
What changed?: the code
Describe the change concisely: oh well
added a vkapriltag pipeline.
made an abstract apriltag pipeline with all the shared code of the vkapriltag and libapriltag pipelines.
added the vkapriltag license and NOTICE files to the ExternalLicenses folder.
added a libapriltag parity test for vkapriltag.
added vkapriltag-jni as a dependency (link: https://github.com/yojobama/vkapriltag-jni).
the frontend was updated to support the new pipeline (disclosure: unlike the rest of the code I havn't got a clue about frontend development so I wasn't able to verify the code like the rest of the code base, I just ran photonvision and the UI worked)
problems:
Why?: speeeeeeed!!!
as stated in #2588 most of the apriltag detection pipeline is massively parallel and therefore would most likely benefit from GPU acceleration (and it does even on a wee GPU like the Mali G610 MP4) with vulkan. the vkapriltag detector resides in this repo: https://github.com/yojobama/vkapriltag
Testing
tag36h11 parity test results: VkapriltagParityTest.txt
Related Issues: #2588
Closes: absolutely nothing
AI Disclosure
this description was written by me, myself, and I.
the code was written with heavy AI assistance (specifically Claude Code (opus5 and sonnet5)).
I have reviewed it and understand all of it (except the frontend, as I havn't got a single clue about TS and Vue, so I ran the UI and it worked).
Merge Checklist
Additional Checks (if applicable)
// MIGRATION: <dataYear>where<dataYear>is the last season the pre-migration data was used in./photon-client/src/types