Flekschas/faster beddb

[flekschas]

Description

What was changed in this pull request?

I implemented a tile-based indexing strategy for beddb which can speed up queries by up to 20x at the expense of increasing the file size by a factor of ~2.5x

To avoid adding the burden of having to handle another format to the end-user I decided to mark this indexing using an appended t to the version number. I.e., version 3 is the normal version while 3t is the tile-index version

To create a tile-indexed beddb file use clodius aggregate bedfile with --tile-index.

Why is it necessary?

The range-based rtree indexing is getting slow with >5mio intervals (i.e., 0.5s for a query) while the tile-based index remains fast with ~0.025s.

Checklist

• Unit tests added or updated
• Updated CHANGELOG.md
• Run black .

[pkerpedjiev]

The speedup looks phenomenal. I think the space might be able to be reduced. See the inline comments. And how about some tests?

[pkerpedjiev]

Couldn't you put the tile_inserts.append() call from below here and save yourself a while loop?

[flekschas]

I don't think this works. The reason why is the following:

tile_counts only counts the first time an interval is inserted but not the subsequent times. Hence, this while loop is only called once, while the one below is called in higher zoom levels as well.

Here's a short example: say we set the max number of intervals per tile to 2. The first interval will be inserted at tile 0.0 and its counter will increase. However, we want to make sure that the most important tile is always shown in tile with a higher zoom level so we will insert tile-id<>interval-id pairs for higher zoom levels as well. However, we will not increase the counter for those tiles as otherwise we would never see more details as we zoom in in some cases. Does this somehow make sense? :)

This replicates the existing behavior of the beddb format.

[pkerpedjiev]

Cool, thanks for the responses!