feat: account JVM UDF Arrow allocations in Spark task memory

[peterxcli]

Which issue does this PR close?

Closes #4174.

Rationale for this change

CometScalaUDFCodegen, currently the only CometUDF implementation, creates output Arrow vectors from the process-wide CometArrowAllocator. These JVM allocations bypass Spark’s task memory accounting.

FFI input vectors only wrap native-owned buffers that are already accounted by Comet, so they must not be charged again. Output buffers, however, are created on the JVM and may remain alive through the FFI release callback after the Java vector closes.

What changes are included in this PR?

• Add a per-task Arrow child allocator backed by a non-spillable Spark MemoryConsumer.
• Pass the allocator through the generic CometUDF interface and codegen output path.
• Keep imported native buffers on the root allocator to avoid double-accounting.
• Release Spark accounting with the task while retaining allocator state until the final FFI buffer release.

How are these changes tested?

Added CometUdfBridgeTest covering allocation accounting, FFI retention, task completion, and final cleanup. The focused test passes with Spark 4.1 and Spark 3.5/Scala 2.12.

Also ran few scala script with spark shell to verify:

Check	main	branch
Small output task peak	0	262,144 bytes
32 MiB output task peak	0	50,397,184 bytes
Peak increase	0	50,135,040 bytes
256 MiB batch under 256 MiB pool	Succeeds, bypassing Spark	Rejected by TaskMemoryManager
Same workload with batch size 32	—	Succeeds, 268,435,456 bytes processed

[peterxcli]

@sunchao would you like to take a look at this? TIA!

[sunchao]

Two P2 findings in task-allocator concurrency, detailed inline. Validated with targeted JVM reproductions against this exact head and Arrow 18.3.0. The native producer/cleanup paths were source-traced, not run end-to-end.

[sunchao]

@peterxcli can you check the CI failures?

[peterxcli]

@peterxcli can you check the CI failures?

@sunchao I opened a fix for it

#5439

[sunchao]

Ah thanks! Approved the PR and will merge soon