feat(guest): replace musl with picolibc by andreiltd · Pull Request #831 · hyperlight-dev/hyperlight

andreiltd · 2025-08-28T11:48:24Z

This patch removes custom musl implementation used by the guest and replace it with picolibc v1.8.11.

Note: The picolibc submodule uses sparse checkout to exclude GPL/AGPL-licensed test and script files that are not needed for building. Only BSD/MIT/permissive-licensed source files are included.

jprendes

I love this PR!

src/hyperlight_guest_bin/src/host_bridge.rs

src/hyperlight_guest_bin/Cargo.toml

src/hyperlight_guest_bin/src/host_bridge.rs

jsturtevant · 2025-08-28T21:07:02Z

will help with #282

src/hyperlight_host/tests/sandbox_host_tests.rs

src/hyperlight_guest_bin/build.rs

jsturtevant · 2026-03-03T01:24:30Z

src/hyperlight_guest_bin/c/clock.c

did we write this file? is there attribution that is needed?

If this is really necessary, I think @jprendes just wrote a Rust implementation for this in hyperlight-js. maybe we could use it?

Yes, those are ours. We cannot use the implementation from hyperlight-js directly, it serves a bit different purpose. That being said I rewrote picolibc stubs to rust. The tradeoff is that some of the types and defines that you normally get from libc headers have to be redifined in Rust with compatible ABI. I think that is OK -- they have to be backwards compatible and I don't see them change anytime soon.

src/hyperlight_guest_bin/c/clock.c

jsturtevant · 2026-03-03T01:29:47Z

src/hyperlight_guest_bin/src/host_bridge.rs

+        Some(vec![ParameterValue::ULong(count as u64)]),
+        ReturnType::VecBytes,
+    ) {
+        Ok(bytes) => {
+            let n = bytes.len();
+            unsafe {
+                core::ptr::copy_nonoverlapping(bytes.as_ptr(), buf as *mut u8, n);


should not rely on the host provided length. nonoverlapping could write past the buffer lenghth. We are in the guest here so it would just cause corruption but might be tough to track down I think

src/hyperlight_guest_bin/src/libc.rs

jsturtevant · 2026-03-03T01:44:43Z

NOTICE.txt

-obstacle to adoption, that text has been removed.
+Note: The picolibc submodule uses sparse checkout to exclude
+GPL/AGPL-licensed test and script files that are not needed for
+building. Only BSD/MIT/permissive-licensed source files are included.


Just wanted to surface this. We should include it in the PR description I think

Where is the sparsity pattern that makes this actually happen?

That would be here:
https://github.com/hyperlight-dev/hyperlight/pull/831/changes#diff-99cd76bae1f874496398d422f5fcc07ece8599f9f26cc81bb1cb154d0cf0003fR338

dblnz

Great work, Tomasz!! This is one hell of a diff and I like it 😆

dblnz · 2026-03-04T08:16:28Z

src/hyperlight_guest_bin/c/clock.c

If this is really necessary, I think @jprendes just wrote a Rust implementation for this in hyperlight-js. maybe we could use it?

src/hyperlight_guest_bin/src/lib.rs

src/hyperlight_guest_bin/build.rs

src/hyperlight_guest_bin/build_files.rs

Justfile

jprendes · 2026-02-25T11:19:37Z

src/hyperlight_guest_bin/c/clock.c

+
+extern void _current_time(uint64_t *ts);
+
+int gettimeofday(struct timeval *__restrict tv, void *__restrict __tz) {


Thought: If we were running bindgen on picolib headers, then we could easily implement this one in rust instead of C. IIUC, the main issue is the definition of timeval, clockid_t, and timespec in Rust, that we don't want to get wrong.

This file is no longer in the diff.

If we were running bindgen on picolib headers, then we could easily implement this one in rust

That would be correct yes, but I would rather much have a single c file defining the interface or redefine types in Rust with correct layout. It's hard to justify running the whole bindgen machinery for few u64.

src/hyperlight_guest_bin/src/host_bridge.rs

syntactically · 2026-03-05T13:41:28Z

Is the licensing approval finally done for this, making it ready for review/merge?

syntactically · 2026-03-05T13:52:22Z

src/tests/c_guests/c_simpleguest/main.c


 #define GUEST_SCRATCH_SIZE (0x40000) // default scratch size
 #define MAX_BUFFER_SIZE (1024)

+#define printf_f(fmt, ...)                          \


Does it make sense to see if we can either build picolibc in a configuration without stdio buffers, or to automatically do a flush on the guest-function-call-complete path?

I feel like the buffered-I/O-by-default regime will be very confusing to users---where, for example, if you are doing the "call a function and then restore the sandbox" pattern might often not see any output at all.

Yeah, I think we can disable posix console feature: https://github.com/picolibc/picolibc/blob/main/doc/build.md#stdio-options

Fuzzy on the details but IIRC that will change to char by char sending to host which I thought was insane.

Yeah, that makes sense... char-by-char is definitely unfortunate. Flush-on-guest-function-return would be nice if we could coalesce the flush exit and the function-return exit, but I'm not sure how easy that is?

As Jorge suggested: #831 (comment) we can perhaps add flushing in next PR?

syntactically · 2026-03-05T13:56:32Z

src/hyperlight_guest_bin/src/host_bridge.rs

+
+#[unsafe(no_mangle)]
+pub extern "C" fn _current_time(ts: *mut u64) -> c_int {
+    let bytes = call_host_function::<Vec<u8>>("CurrentTime", Some(vec![]), ReturnType::VecBytes)


Similarly, I think I might like to see this fail at least by default? As with my comments on #1173, I'm against exposing a clock, especially a wall clock, to the guest by default.

This is very common dependency, for example spidermonkey crashes immediately if we don't provide some time interface. Do we really want to error or at least return an unix epoch?

Interesting... I didn't realise so many things were using wall-clock time. Do you know what we were doing for spidermonkey before? I don't think we did actually have the real time?

I wouldn't be against having it always be the epoch personally (at least by default), but I'll let others chime in here as well.

at some point we had a fallback where we took some epoch (2000?) and added 1s or 1ms for each call

syntactically · 2026-03-05T13:56:46Z

src/hyperlight_guest_bin/src/libc.rs

+static CURRENT_TIME: AtomicU64 = AtomicU64::new(0);
+
+#[unsafe(no_mangle)]
+pub extern "C" fn read(fd: c_int, buf: *mut c_void, count: usize) -> isize {


Where are the fds referenced in these functions allocated? I don't see an open, so are these always either 0/1/2 for stdin/out/err? I think it might make sense to make the behaviour a bit more limited (maybe reads from fd0 always return eof; writes to fd1 are HostPrint, and writes to fd2 are debug_print, or something like that?)

That's almost what you have here, right? except that:

fd2 also goes to HostPrint instead of debug_print (I didn't know debug_print was a thing!).

fd0 first tries HostRead, if that's not present returns -1 and sets errno = EIO (instead of EOF, which would be returning 0 and no errno)

any other fd is EBADF

So, do you suggest these changes?

fd2 to use debug_print instead of HostPrint

fd0 to return 0 (and untouched errno) in case of no HostRead

I don't care much about the fd2---it was just the first thing that sprang to mind.

Unless we have a compelling use case, I think I would prefer for read from fd0 to unconditionally be eof, without probing for a host function which seems a bit surprising and magical. If we ever have C code that needs to be able to read() from stdin, we could add some hyperlight_guest_bin API to let a little Rust wrapper code install its own read function.

I'm happy to go with that logic. @jprendes any objections?

you can call fflush(NULL) and that flushes all open FDs.
Can we leave buffering and add that on VM exit (or similar) in an upcoming PR?

also I think our current _putchar impl can go away, right?

I got rid of function probing and replaced current_time with unix epoch increasing 1s each call -- this should be enough to get some sort of ordering working. The read implementation now return eof immediately. The behavior is documented in docs/picolibc.md

As Lucy said if we need to give user control over this we can expose hooks in the guest-bin crate.

dblnz

Great work, Tomasz!
I have nothing else to add.

src/hyperlight_guest_bin/build.rs

src/hyperlight_guest_bin/build_files.rs

docs/picolibc.md

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

cargo install already skips the installation if the tool is the newest version and updates otherwise. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

ludfjig

I think this is awesome!

ludfjig · 2026-03-23T20:05:37Z

src/hyperlight_guest_bin/build.rs


-        if cfg!(windows) {
-            unsafe { env::set_var("AR_x86_64_unknown_none", "llvm-ar") };
+    // TODO: this is wrong, it should generate config file to out_dir/include


Do we have a plan to fix this?

I'm not sure how we fix this. Polluting source tree with generated files is obviously bad but we need a stable location for c guests that have no access to hashed rust target dir.

One way of improving this would be to use the existing HYPERLIGHT_GUEST_TOOLCHAIN_ROOT mechanism for c guests. We could copy generated headers to HYPERLIGHT_GUEST_TOOLCHAIN_ROOT/include and set the env when building c guests. Then we can prepend the headers from that dir to clang calls.

I believe a proper fix would be to replace just script and cargo with polyglot build system.

ludfjig · 2026-03-23T20:07:49Z

src/hyperlight_guest_bin/build.rs

+    Command::new("git")
+        .args([
+            "-C",
+            "third_party/picolibc",
+            "sparse-checkout",
+            "init",
+            "--no-cone",
+        ])
+        .status()?;
+
+    Command::new("git")
+        .args([
+            "-C",
+            "third_party/picolibc",
+            "sparse-checkout",
+            "set",
+            "**",
+            "!test/**",
+            "!scripts/**",
+            "!COPYING.GPL2",
+        ])
+        .status()?;


I think we should check .success() on these commands too

ludfjig · 2026-03-23T20:10:40Z

src/hyperlight_guest_bin/src/libc.rs

+}
+
+#[unsafe(no_mangle)]
+pub extern "C" fn _current_time(ts: *mut u64) -> c_int {


does this need a null check too given below unsafe?

ludfjig · 2026-03-23T20:13:28Z

src/hyperlight_guest_bin/build.rs

+    let has = if detect_cc_feature(code)? { 1 } else { 0 };
+    writeln!(file, "#define __HAVE_COMPLEX {has}")?;
+
+    // Static undefs


How did you decide whether to enable/disable some of these? I'm thinking mostly about

__IO_C99_FORMATS disabled, __IO_WCHAR enabled, __OBSOLETE_MATH disabled, __IO_LONG_LONG disabled

And if a particular guest wants different picolibc settings, is that possible? I can imagine that some settings are specific to hyperlight itself, and are not allowed to be changed by guest, but maybe certain guests need more control?

picolibc is using meson as build system -- most of those defines are default settings that meson generates. The defines are gathered from generated by meson picolibc.h file and compile_command.json file. You can have a look at all the switches that we enable/disable for picolibc here:

https://github.com/andreiltd/picoforge/blob/main/picolibc.just

and the docs for all the options:

https://github.com/picolibc/picolibc/blob/main/doc/build.md

I understand, I was more curious if there some specific reason for these enables/disables since I think some are not default and since it's not immediately clear to me what the correct option should be. In addition, would it be valuable to allow guests to override these for example via environment variables?

The non-defaults serve the hyperlight guest model: single-threaded, no TLS, no semihosting, no libc malloc since we provide rust allocator, verbose asserts for better debuggability, and some I/O performance tweaks discussed above like io buffering.

There are some knobs like wchar support that were needed to allow building baremetal libc++ against picolibc.

We can expose build settings but I would rather do that trough cargo features not env variables. That being said I'm not sure if we should proactively expose all the build tweaks or wait and see if any of this turns out to be problematic?

Justfile

andreiltd requested review from danbugs, dblnz, devigned, jprendes, ludfjig, marosset, simongdavies and syntactically as code owners August 28, 2025 11:48

andreiltd marked this pull request as draft August 28, 2025 11:48

andreiltd force-pushed the libc-takeover branch from 617d834 to 3f8ffcf Compare August 28, 2025 12:01

andreiltd added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Aug 28, 2025

andreiltd force-pushed the libc-takeover branch 6 times, most recently from eaf1d54 to b18a374 Compare August 28, 2025 13:44

jprendes reviewed Aug 28, 2025

View reviewed changes

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

src/hyperlight_guest_bin/Cargo.toml Show resolved Hide resolved

jprendes reviewed Aug 28, 2025

View reviewed changes

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

jsturtevant mentioned this pull request Aug 28, 2025

[License Exception Request] [Hyperlight] [Older BSD like licenses] cncf/foundation#1117

Open

12 tasks

andreiltd force-pushed the libc-takeover branch 5 times, most recently from 499c59d to 2ae0a3a Compare August 29, 2025 18:42

jprendes reviewed Sep 4, 2025

View reviewed changes

src/hyperlight_host/tests/sandbox_host_tests.rs Outdated Show resolved Hide resolved

andreiltd force-pushed the libc-takeover branch from 2ae0a3a to d3826af Compare September 4, 2025 11:29

andreiltd mentioned this pull request Sep 5, 2025

Use mimalloc as a guest allocator #867

Draft

andreiltd force-pushed the libc-takeover branch from d2043ac to 2bd1ff4 Compare September 9, 2025 09:31

jsturtevant reviewed Mar 3, 2026

View reviewed changes

src/hyperlight_guest_bin/build.rs Outdated Show resolved Hide resolved

jsturtevant reviewed Mar 3, 2026

View reviewed changes

src/hyperlight_guest_bin/c/clock.c Outdated Show resolved Hide resolved

jsturtevant reviewed Mar 3, 2026

View reviewed changes

src/hyperlight_guest_bin/c/clock.c Outdated Show resolved Hide resolved

jsturtevant reviewed Mar 3, 2026

View reviewed changes

src/hyperlight_guest_bin/src/libc.rs Show resolved Hide resolved

jsturtevant reviewed Mar 3, 2026

View reviewed changes

dblnz reviewed Mar 4, 2026

View reviewed changes

jprendes reviewed Mar 5, 2026

View reviewed changes

syntactically reviewed Mar 5, 2026

View reviewed changes

andreiltd force-pushed the libc-takeover branch 2 times, most recently from cacc739 to 2fbc721 Compare March 5, 2026 14:36

dblnz previously approved these changes Mar 10, 2026

View reviewed changes

src/hyperlight_guest_bin/build.rs Show resolved Hide resolved

src/hyperlight_guest_bin/build_files.rs Show resolved Hide resolved

docs/picolibc.md Show resolved Hide resolved

andreiltd added 7 commits March 16, 2026 09:51

feat(guest): replace musl with picolibc

60c0288

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

bump picolibc to 1.8.11

6c600c8

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: address code review comments

aa5c525

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: fmt with different rust version

63fa043

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: apply clippy fixes

caf84ec

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: clarify usage of no-red-zone flag

3ad7116

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: get rid of host function probe

20195de

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd dismissed dblnz’s stale review via 20195de March 16, 2026 09:38

andreiltd force-pushed the libc-takeover branch from 3680b58 to 20195de Compare March 16, 2026 09:38

andreiltd added 3 commits March 23, 2026 09:32

fix: use cargo-hyperlight from crates.io

24dfcd8

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Merge branch 'main' into libc-takeover

9d56fc1

fix: always run cargo install

4daa971

cargo install already skips the installation if the tool is the newest version and updates otherwise. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the libc-takeover branch from 01f8cb1 to 4daa971 Compare March 23, 2026 18:22

ludfjig approved these changes Mar 23, 2026

View reviewed changes


		extern void _current_time(uint64_t *ts);

		int gettimeofday(struct timeval __restrict tv, void __restrict __tz) {

Conversation

andreiltd commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant commented Aug 28, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dblnz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

syntactically commented Mar 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreiltd Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreiltd commented Aug 28, 2025 •

edited

Loading

andreiltd Mar 5, 2026 •

edited

Loading