zig ar: a drop-in llvm-ar replacement

[kubkon]

Since we are putting a lot of effort into implementing our linker for all supported targets (#8726), we should also put some effort into adding our own implementation of a static archiver to replace llvm-ar. While in general llvm-ar is working well on the host platform when targeting the host platform, in cross-compilation settings things can get wonky when the archiver will produce native static archive headers for foreign file formats possibly tripping the linker upon trying to use it.

This is a great issue for any new contributor as it allows you to create an archiver as a completely standalone program (in its own repo like zld for instance) and then upstream it into Zig once it's ready.

Also, with this issue closed, we will be able to offer zig ar as a subcommand that does not rely on llvm-ar in any way.

[moosichu]

I'm interested in giving this a look-in! Where would be a good place to start in understanding the problem?

[kubkon]

If I were tackling this problem, I would most likely create a fresh repo with sole purpose of building a drop-in replacement for llvm-ar, much like zld is for lld. You can then build it out-of-tree which means you don't have to worry about passing Zig tests at this stage and you significantly cut down on build times. I would also focus on just one file format in the beginning say ELF, or Mach-O. The idea here is that if you were to pick any large C/C++ codebase (or whatnot) in the wild, you could pass the Zig archiver as a replacement for the default system one or llvm-ar, that is you'd tweak the CMake/Make invocation like so:

AR=zig-ar CC=... CXX=... cmake ../

or the same but with make. If the build process succeeds, then success!

Afterwards, you might wanna consider dropping the archiver as a direct replacement for llvm-ar in the Zig upstream either by calling directly to a precompiled binary or putting the sources in-tree (the latter is the end-goal actually). The relevant source where this should/could happen is in src/link.zig#L668:

pub fn linkAsArchive(base: *File, comp: *Compilation) !void {
    //...
    const llvm = @import("codegen/llvm/bindings.zig");
    const os_type = @import("target.zig").osToLLVM(base.options.target.os.tag);
    const bad = llvm.WriteArchive(full_out_path_z, object_files.items.ptr, object_files.items.len, os_type);
    if (bad) return error.UnableToWriteArchive;
    //...
}

[moosichu]

Will give that a crack!

[wqweto]

JFYI, https://github.com/TinyCC/tinycc/blob/mob/tcctools.c has an extremely hacky impl for Elf support on Windows.

[moosichu]

Yeah - I've made a little bit of progress with this. Just still getting comfortable with zig so it's going to be a slightly idiosyncratic start. But it feels like a doable project.

https://github.com/moosichu/zar/

I just have a little program that can parse a very simple archive file and then prints out all the "filenames" of the files it contains.

I'm just building things up slowly step-by-step, with a focus on reading archives generated by llvm-ar to begin with.

The goal will be to make it a drop-in replacement, but will figure-out the order in which I do things as I go for now (still going to be very experimental early on).

I think what will probably end up happening is that I will experiment with parsing increasingly interesting archive files. And then I will loop back around and implement the command-line interface for the program. And then just incrementally work on each piece of functionality testing against the results of llvm-ar.

Will then probably make some kind of framework for testing those as well I think.

[g-w1]

You might want to look at https://github.com/SuperAuguste/zarc . It can parse ars, but it cannot create them. So maybe just using that and then adding features to create tars would be good?

[kubkon]

Yeah, the ultimate goal of zar should be generating static archives. Adding parsing logic is a good first step to figuring out how it works though. One thing to pay particular attention to is the differences in generated ar structure between linux and macos - I believe there is a difference in at least the header format but maybe more. Also, I've been reached out to by multiple people expressing interest in helping out so @moosichu are you fine taking charge on this one and potentially collaborating with others? If so, I'll send them your way (to your fresh repo, etc.).

[moosichu]

Yes - very happy to collaborate and I’ve started reading up and putting sources together on the differences in the formats for different platforms! I’ve linked to some of them in the repo - but will flesh that out properly tomorrow as well for others hoping to contribute as well.

[iddev5]

I have also been doing my own independent attempt at this issue, here https://github.com/iddev5/zig-ar

My ar can create basic files so far, and it is compatible with llvm-ar and ranlib too.

If it works out, I can try merge it with zar as discussed above...

[kubkon]

Great progress! Since you have two repos it might make sense to split the focus a little. For example, @iddev5 could focus on linux and @moosichu on macos, etc., and afterwards merge both together as zar or otherwise. How's that for a plan?

[iddev5]

This sounds great! Just to let you know that I am okay with either plans.
On my repo, I have already got reading and writing common-style archives done (without symbol table and string table, ofc)