Switch sort order for metadata table entries

[ddanielr]

Currently, when a table is created it uses a one-up counter for its table name. This means that a workload that creates new tables end up putting all that work on the same metadata tablet.

Describe the solution you'd like
Switch the sort order of the metadata table to use the least significant bit of the tableID.

This would spread the new tables metadata entries across multiple metadata tablets while still allowing each table section to be ordered.

[phrocker]

Is table creation the load or the fate ops / zk round trips, lock acquisition, fsyncs or is the pressure downstream at metadata splits? if scattering is the goal could you get the same goal by doing it at id assignment instead of in the metadata sort order?

[dlmarion]

Would this mean that we need to rewrite the metadata table on the upgrade to 4.0? If the issue is the one-up number creating hot spots in the metadata table, then let's change that instead.

[ctubbsii]

I think this may have come from a passing comment I made to @ddanielr a few weeks ago. If that's the case, then the idea I had in mind was to simply serialize the table IDs in the metadata table in reverse character order, so that they naturally would spread out more (table 123 would be next to table 43 and c3, but table 124 would be next to a4 and 94). That way, two tables created in succession would be less likely to share the same metadata tablets, but all the table's metadata would still be adjacent for any given table.

Implementing this would require rewriting metadata on upgrade, or storing some extra information in ZK and the metadata table so we knew which format a particular table's metadata entries were using. Both of those seemed undesirable, so I kind of dismissed the idea shortly after I said it.

I'm also not really sure that this would really improve things much, because if the number of tables is sufficiently dense and table "busy-ness" is about the same (or at least, hard to predict), placing one table's metadata next to an older table's metadata may not actually be better than placing it next to the table that was most recently created. And that's another reason I kind of dismissed the idea.