Sensors Addon

[whuang37]

Refactor all third person sensing into a new addon called sensors

• SensorsManager manages all sensor lifecycles to enforce time alignment, dependencies, and caching
• Shifted all sensing to a unified frame-based caching which allows multiple scripts to query for the same sensing output
• Sensors<T> allow easy extension of new sensors
• Scheduling mode for sensors to allow for variable performance hits

New Sensors

• ScreenshotSensor captures screenshots of the raw world
• ScreenshotXR captures. screenshot of the world + XR
• ScreenshotSOM adds annotations over XR objects for referencing by AI
• VisibilitySensor traces what objects are in the user's viewport
• SemanticMapSensor formats visible sensors into plaintext descriptiosn
• DepthSensor depth maps
• TargetingSensors tracks what objects the hand is pointing at or colliding with
• ProprioceptionSensors tracks user transforms

GeminiManager Changes

• Movd the screenshot captures out of geminiManager and made it run from sensors instead.

New sample showing data at sample/sensors_debug

[dli7319]

We have this XRSystems object which is supposed to hold everything auto-created by XR Blocks (except depth mesh right now): src/core/components/XRSystems.ts.
Instead of marking everything as isInternal, can we have isInternalHelper check if the object is a descendent of XRSystems or instanceof xb.DepthMesh?

[whuang37]

We have this XRSystems object which is supposed to hold everything auto-created by XR Blocks (except depth mesh right now): src/core/components/XRSystems.ts. Instead of marking everything as isInternal, can we have isInternalHelper check if the object is a descendent of XRSystems or instanceof xb.DepthMesh?

Awesome! Never saw that since it wasn't exposed as an export. I just converted the internal helper over to that

[whuang37]

New signatures for sensors: Purely a caching + alignment on embodied sensing

import {
  sensors,
  UserViewSensor,
  DepthSensor,
  DeviceCameraViewSensor,
} from 'xrblocks/addons/sensors/index.js';

const view = await sensors.capture(UserViewSensor);
const depth = await sensors.capture(DepthSensor);
const camera = await sensors.capture(DeviceCameraViewSensor, {
  width: 320,
  height: 180,
});

Force a fresh piece of data

const userView = await sensors.capture(UserViewSensor, {
  overlayOnCamera: true,
  updateMode: 'sync',
  forceRefresh: true,
  cacheWindowMs: 0,
});

Batch capture

import {
  sensors,
  ProprioceptionSensor,
  VisibilitySensor,
  UserViewSensor,
  DepthSensor,
} from 'xrblocks/addons/sensors/index.js';

const values = await sensors.captureAll({
  pose: ProprioceptionSensor,
  visible: VisibilitySensor,
  view: [UserViewSensor, {overlayOnCamera: true}],
  depth: DepthSensor,
});

Lenient batch capture

const result = await sensors.tryCaptureAll({
  view: UserViewSensor,
  camera: DeviceCameraViewSensor,
});

console.log(result.values.view);
console.log(result.values.camera); // null if unavailable
console.log(result.errors.camera);

[dli7319]

That api looks good to me.
I'm not a huge fan of importing singleton instances but we're already doing it with XR Blocks so you can do it if you like.

[dli7319]

Any reason to use number[][]?
This involves a bunch of allocations vs a flat array.
Also, I'd recommend keeping as much info as possible including the format and extrinsics and intrinsics.