read: check that sizes are smaller than the file when reading

[symphorien]

related to #204

added a regression test case which used to attempt to allocate 5 exabytes of memory.

test files added in gimli-rs/object-testfiles#12

[philipc]

Thanks! I think this is a good fix.

What's the motivation for the second test? It's fine to have it, but it appears to work prior to this fix.

Can you fix the CI failures and update the submodule to the merged commit.

[philipc]

Note that ReadCache is a bit of a hack. It's better to use memmap if you can. (This might be worth documenting.)

[symphorien]

Can you fix the CI failures and update the submodule to the merged commit.

done

What's the motivation for the second test? It's fine to have it, but it appears to work prior to this fix.

my initial implementation had an off by one error and get_buildid would always return Err() even on more legitimate binaries, the test is to ensure ReadCache can succeed.

[symphorien]

Note that ReadCache is a bit of a hack. It's better to use memmap if you can. (This might be worth documenting.)

If that's the case yes please document it, and document preferred alternatives. My use case is to find the buildid of thousands of elf files in parallel. When I consulted the documentation of object, it seemed I needed to call File::parse so needed an implementation of ReadRef:

This leaves me with three options:

• mmap: Most mmap rust wrappers come with huge unsafe warnings in the readme
• reading the whole file in memory, which seemed wasteful at best
• ReadCache, which is builtin to object

You can imagine that the third option seemed better at the time.

After using ReadCache in my code, I noticed memory usage was very spikey because of read cache so actually gave a try to a mmap implementation. I quickly abandoned the experiment, and if my memory serves me right here are the reasons:

• it was slower
• mmap would often (1 file out of ten) yield unexpected error codes like EINVAL (I don't remember exactly which)
• spooky unsafe (please don't SIGBUS my process!)

I would be happy to discuss potential solutions or compare things more seriously and report the result if you are interested. (But maybe not on this PR)

[philipc]

I default to using mmap and have never had problems with it. I sometimes use fs::read for small things that don't care about performance.

it was slower

I guess that might be true if you are only using this to read the build ID, where the total amount that you need to read is small.

mmap would often (1 file out of ten) yield unexpected error codes like EINVAL (I don't remember exactly which)

I haven't encountered that, and I don't know what could cause it. I'm interested in learning how to reproduce this.

spooky unsafe (please don't SIGBUS my process!)

Yes there's technically ways that this can be unsound, but mmap has a long history of being a useful tool. In isolation you'll never have a problem. The problem occurs when other code or other processes modify the file at the same time. I don't think that's a reason not to use it though, as long as you are aware of this limitation and evaluate how likely it is to occur for your use case. If you do know that the file will be modified, you'll have to take steps to prevent it happening at the same time you read, such as using locks.

[symphorien]

I tried again and ReadCache is 47% faster for my use case. I found the source of the EINVAL: among all the files there were a quite large proportion of empty files, and mmaping empty files results in EINVAL.

[symphorien]

Also about getting sigbus in case of concurrent modification of the file: I suppose that if you reimplement a cli tool the difference between sigbus and getting a real error is just the quality of the error message, but I'm implementing a server (debuginfod) and crashing the whole long lived process because of a single "wrong" file is quite annoying in this case. I suppose I could avoid that by forking and using the child process as a sacrificial victim just in case, but the increase in complexity in off-putting.

[symphorien]

would it be possible to get a new release with this change?

[philipc]

Published 0.33.0