feat: use block matrix to reduce peak memory usage for matmul

[badGarnet]

This PR targets the most memory expensive operation in partition pdf and images: deduplicate pdfminer elements. In large pages the number of elements can be over 10k, which would generate multiple 10k x 10k square double float matrices during deduplication, pushing peak memory usage close to 13Gb

This PR breaks this computation down by computing partial IOU. More precisely it computes IOU for each 2000 elements against all the elements at a time to reduce peak memory usage by about 10x to around 1.6Gb.

The block size is configurable based on user preference for peak memory usage and it is set by changing the env UNST_MATMUL_MEMORY_CAP_IN_GB.

[christinestraub]

What do you think of cutting a new release at this point?

[ctrahey]

Do you have timing observations for this?
If it is a significant regression in time, we can consider processpool...

Nice work!

[christinestraub]

What do you think of creating an experiment PR to check metrics changes in core-product before merging this PR? I think it would worth to have an experiment PR in core-product first for all the features that could impact metrics scores.

[ctrahey]

Do you have timing observations for this? If it is a significant regression in time, we can consider processpool...

Nice work!

Thinking on this more... there is nuance because I suppose if the reduction in memory is linear with time, then it's not anything gained to do it in processpool... unless each thread working on a block is faster with the smaller overall "job size". Then it would be a net win.

[badGarnet]

Do you have timing observations for this? If it is a significant regression in time, we can consider processpool...

Nice work!

it is actually slightly faster because of the counter intuitive way cpu works. This reduces cpu cache trips actually (computation can fit into cache easier). One can even try to optimize against specific cpu by tweaking the block size smaller.

[badGarnet]

What do you think of creating an experiment PR to check metrics changes in core-product before merging this PR? I think it would worth to have an experiment PR in core-product first for all the features that could impact metrics scores.

this change doesn't change any change logic at all; it is purely a different procedure to compute the same thing

[ctrahey]

Do you have timing observations for this? If it is a significant regression in time, we can consider processpool...
Nice work!

it is actually slightly faster because of the counter intuitive way cpu works. This reduces cpu cache trips actually (computation can fit into cache easier). One can even try to optimize against specific cpu by tweaking the block size smaller.

Heck yeah - I was hoping you'd say that!