objcopy --redefine-sym

[jayvdb]

indygreg/PyOxidizer#183 contains a lot of boilerplate code to write a object file, substantially copied from https://github.com/gimli-rs/object/blob/master/examples/objcopy.rs .

It would be nice if there was a simple API to achieve the same, with high-level transform operations selected and then the reconstruction iteration process left for object & friends to perform.

In my case the transform I need is renaming of one sym from PyInit_* to PyInit_foo_baz. This is the same as objcopy --redefine-sym old=new, and it is a bit similar to the Mangling use-case.

[philipc]

Definitely open to adding something like this, but I don't have much idea of what the ideal API would look like. The problem is there is a wide range of possible transforms, so it would be nice if the API was flexible enough to handle unforseen needs, instead of limiting to a small number of options.

There's a similar problem for the DWARF read/write support in gimli, and for that I've written functions to convert from the read data structures to the write data structures, but currently it doesn't have support for transforming during conversion. Applying transforms is something we do need to support though.

[jayvdb]

Im new to Rust, so Im just getting familiar with the limitations. Ideally for symbol name rewriting, a function/closure/lambda could be provided to take the old name (and other args such as target object format & mach arch) and provide the new desired sym name. That could cover the mangling use-case as well. As functions are not first class entities in Rust, this might mean a closure is the only very flexible option. If that doesnt work, a table of match->rewrite-pattern would also work, and be almost as good.

[philipc]

If you want to do the transform via callbacks, then you probably want to do this via a trait that has a method for each possible callback. My concern with this is it restricts the control flow of the objcopy, which means it may not be flexible enough. It would be fine as a starting point until something better is needed though.

As functions are not first class entities in Rust, this might mean a closure is the only very flexible option.

The usual way is to use one of the Fn/FnMut/FnOnce traits, and then either functions or closures can be used.

That could cover the mangling use-case as well.

The mangling use-case is a bit different. It's not intended for objcopy, because the symbols in the source object file will already be mangled. Instead it's for the initial codegen, where the name of the symbol will be from the source language, and it needs to be mangled depending on which file format it is outputting to.

[philipc]

The objconv utility may give some ideas for the sort of transformations that are useful to support.

[david-boles]

I ran into a use-case for this feature, it looks like I'll also end up substantially copying the objcopy example.

Based on just a brief glance at the docs and source, I'm wondering if it would make sense to extend the write::Object API and then incorporate something like the objcopy demo for automatically reading object files into an editable in-memory representation (e.g. write::Object) that can be written out after any desired transformations are done. That seems easier than transforming "in-flight". write::Object seems to already support some "transform"s, e.g. set_section_data

[philipc]

write::Object is currently intended only as a builder, not as an editor. While set_section_data does exist, it currently asserts that the data hasn't been set yet.

Depending on which editing operations you need, it could require require significant changes to support editing write::Object. For example, reordering sections would be hard to implement, since currently SectionIds are expected both to never change, and to match the order.

If you are able to elaborate more on your use case then that would be helpful.

[wolfv]

We are interested in reimplementing (parts of) patchelf and install_name_tool in Rust for integration into rattler-build (for the conda package ecosystem). It is of similar interest in the Nix community (that also relies on those two tools).

In the conda ecosystem we need patchelf to change the encoded rpath to something that includes the $ORIGIN variable so that the rpath is relative and relocation is easier for these shared libs. Same idea is used for dylibs on macOS.

I am wondering if object would provide a good base to start from for this? Would love to get some insights.

[philipc]

So that would be https://github.com/NixOS/patchelf and https://github.com/opensource-apple/cctools/blob/master/misc/install_name_tool.c (or https://github.com/tpoechtrager/cctools-port/blob/986-ld64-711/cctools/misc/install_name_tool.c).

It would be possible to use object as a basis for this, but still significant effort. From the sound of it, I would implement them as patch utilities that overwrite small portions of the file, not as full copy utilities like objcopy.

[wolfv]

Thanks for your thoughts @philipc – do you think it would be more adequate to try to use goblin or object? I think goblin doesn't help much in terms of writing the ELF / Dylib files.

[philipc]

Since you only need to patch the file, they should both work. (If you needed to write the whole file then object has better support currently.) The main difference between them is that goblin copies everything it reads, and parses more up front, while object is zero copy and lazier. The result is that object can potentially be more efficient and flexible, but it can be more fiddly.

[spinpx]

Hi @wolfv , I've created Rust bindings for the patchelf code at patchelf-rs. I hope this will be useful for you.

[wolfv]

Awesome @spinpx – the API looks nice! I think this can be useful also for testing the potential Rust implementation :)

[philipc]

I've started work on converting the elfcopy example into a general ELF modification utility. It might take me a while, but just a heads up so that work isn't duplicated.

I've decided not to support this kind of thing in objcopy, because that is inherently limited on the files that it can potentially work for, so it's better to do something that can eventually work for everything.

[philipc]

#618 provides the ELF support for this, including library APIs and a CLI for much of the patchelf functionality.