Improve performance for roundrobin shuffle writes for nested schemas

[comphead]

What is the problem the feature request solves?

Noticed dramatic slowdown for tasks with shuffle writes for RR repartition comparing to Spark, 2-3x to be more precise.

Specifically for the nested schemas, there is some inefficiency when native shuffle deals with wide/complex fields. its invisible for TPC-* benchmarks because of historical flat schema. But in real user world this is pretty much downside.

Describe the potential solution

No response

Additional context

No response

[comphead]

MIght be related to obtain correct shuffle metrics #5370

[comphead]

Spark

Comet

[comphead]

I'm experimenting with

• xxhash-rust, as looks like hashing tooks lots of time and we have to hash the entire heavyweight row
• vectorized shuffle, when counter based distribution not by row, but by arrow batch

[comphead]

with vectorized shuffle pretty good results