sort: Rework merge batching logic

[karlmcdowall]

Fixes #6944
Rework the way batching is done with sort such that it doesn't open more input files than necessary. Previously, the code would always open one extra input file which causes problems in ulimit scenarios. Add additional test case.

[sylvestre]

Cool
seems that some jobs are failing

[karlmcdowall]

I fixed the windows compilation issue hopefully (I don't have a Windows setup to test on though), but the cargo deny advisory failure is a mystery to me.
I see the cargo deny issue when I checkout the mainline too so I don't think that's an issue with my changes - if anyone has some insight into that then please let me know!

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

[sylvestre]

Any idea why it doesn't fix the GNU tests about this ?
thanks

2024-12-15T08:38:54.0704735Z FAIL: tests/sort/sort-merge-fdlimit
2024-12-15T08:38:54.0704803Z ===================================
2024-12-15T08:38:54.0704807Z 
2024-12-15T08:38:54.0704869Z 11
2024-12-15T08:38:54.0704931Z 1
2024-12-15T08:38:54.0704995Z 6
2024-12-15T08:38:54.0705053Z 3
2024-12-15T08:38:54.0705160Z 12
2024-12-15T08:38:54.0705269Z 13
2024-12-15T08:38:54.0705384Z 17
2024-12-15T08:38:54.0705504Z 4
2024-12-15T08:38:54.0705626Z 5
2024-12-15T08:38:54.0705781Z 14
2024-12-15T08:38:54.0705892Z 15
2024-12-15T08:38:54.0706004Z 8
2024-12-15T08:38:54.0706121Z 16
2024-12-15T08:38:54.0706280Z 10
2024-12-15T08:38:54.0706397Z 7
2024-12-15T08:38:54.0706522Z 9
2024-12-15T08:38:54.0706631Z 2
2024-12-15T08:38:54.0706893Z sort: failed to open temporary file: Too many open files
2024-12-15T08:38:54.0707133Z FAIL tests/sort/sort-merge-fdlimit.sh (exit status: 1)

[karlmcdowall]

Unfortunately there are several issues that need to be fixed before sort-merge-fdlimit will pass - there are several places in the code that aren't optimal in their use of file-descriptors.

I've written a new test case as part of this PR to demonstrate the issue that this PR fixes - the test case would previously fail but now passes.

What's the best way for me to proceed with fixing these issues? I can understand that on one hand it would be nice to have a single PR that can fix the sort-merge-fdlimit test (i.e. this would confirm that my changes are legit and not a waste of time). On the other hand, if there are several distinct issues then the fixes will be easier to code-review as separate PRs.

Let me know how you want me to queue this stuff up 👍

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

[sylvestre]

Let me know how you want me to queue this stuff up 👍

nope, this is fine :)
thanks