Eliminate the POSIX API layer

[LemonBoy]

The os namespace (to be renamed posix) is tackling the problem of providing a cross-platform abstraction over the operating system facilities from a, in my opinion, wrong angle. In this essay I will briefly explain why.

The aim here is having a single level of abstraction that works well enough both for direct consumers (users of std.os namespace) or indirect consumers (eg. using std.fs abstractions built on std.os) without leaking too many details about the underlying implementation. The current approach tries to clump everything under the posix name, a name that carries a heavy baggage of do's and dont's that other platforms (mainly Windows) may not agree with.

A few examples:

• rename doesn't follow the posix semantics on Windows, unless you have a very recent Windows 10 version and FILE_RENAME_FLAG_POSIX_SEMANTICS is used.
• open accepts a well-defined set of options that's not even shared across all the posix-compatible and for Windows this means we're pinky-swearing to faithfully translate all of them into something equivalent. What if there's no direct equivalent? If you promise posix compatibility I'd expect something akin to mingw.
  • What we're interested in is getting a handle on a given file, with a given set of flags.
• stat doesn't exist on Windows, GetFileAttributesEx or equivalents are used and the resulting infos are chopped into a posix stat structure. It's a clear case of square peg in a round hole as many fields are unix-specific.
  • What we're interested in is getting a basic set of infos about the file, platform-specific extra stuff may be retrieved/modified with a companion function.
• preadv/pwritev are not available on Windows and, at the moment at least, you get a nice compile error trying to use them.
  • What we're interested in is a {write,read}Many function, implemented using p{write,read}v or ReadFileScatter/WriteFileGather.
  • iovec is a posix thing, I just want to write/read a [][]u8!
• copy_file_range is a Linux-only syscall with a fallback on pwrite/pread that in turn falls back on ReadFile/WriteFile on Windows.
  • IMO this doesn't even belong to os as it can be safely implemented somewhere in fs, calling copy_file_range or other platform-specific APIs as needed.
  • It's not portable! It's not part of the posix standard!
  • copy_file_range is all about efficiency as it's done at the FS level, if the fallback path is triggered you get a disappointing read/write pair (not even a loop!)

The point is that we should aim at breaking free from the posix rules and write our own, a full-blown posix compatibility layer is not something that belongs to the stdlib. I see Rust took this very same approach (I'm looking at the filsystem-related part), their API surface is small and comprises all the common bits required by the users (external ones and ones working on the stdlib). A small note lets the user know what platform-specific method is used, but that's it.

The gist of this proposal is:

• No more posix compatibility layer
• os becomes the home of all the cross-platform native interactions with the OS. No fallbacks, no posix names, no posix guarantees.
  • No fallbacks really means no half-assed fallbacks, if something can be implemented across all the different platforms with a consistent set of characteristics it belongs to os, otherwise down the platform-specific namespace it goes.
    • ✔️ writeMany implemented with WriteFileGather and pwritev
    • ❌ copy_file_range cannot be implemented as zero-copy on Windows! (On other posix-compatible systems sendfile may be used, but the gains are really small unless you're copying a lot of data (I'm using it to copy whole files in std: Make file copy ops use zero-copy mechanisms #6516))

Thanks for watching.

[kprotty]

a note on the pread/pwrite bit is that WriteFileGather on windows looks like it only supports page aligned userspace buffers + require null terminating its "iovec equivalent" structure. There exist general purpose vectored IO on windows for sockets using WSASend & friends but the generic "HANDLE" equivalent doesn't seem to be as flexible.

Another point on vectored IO is that having the function take in a generic iovec (which for windows sockets, could be WSABUF) would alleviate the os function from allocating its own in order to represent [][]u8 as the layout of slices is currently both undefined or not matching with the layout of socket iovec types, making casting to and from them probably not correct.

I would like to propose taking the idea of restructuring std.os even further and to get rid of it entirely (or at least hide it from the user), similar to what the Rust stdlib does. Replace the exposed functions by having them in their own higher level structs that they act upon (File/Dir, Socket, Pipe, Process, Thread, Random, IoPoll, Memory, .etc) and not requiring everything to go through a posix-based API as that can restrict and otherwise prohibit certain functionality from non-posix systems like Windows:

Make an std.os.posix which acts like the std.os.linux or partially the current std.os but only for posix systems while windows sticks to using std.os.windows. Then design the apis of the higher level structs based on the greatest common functionality for all stdlib supported OS instead of through the lens of posix only.

[ikskuh]

@kprotty i fully support this! Nothing more to add

[katesuyu]

I completely agree with this issue. Even Rust isn't a great example of how to do things, although we do fairly well at avoiding the most glaring flaw of Rust's OS abstractions at the moment: not supporting widechar APIs for Windows. I'm not sure we do this everywhere but I've seen Z and W functions enough to be fairly satisfied.

Also, relying on the existence of a given compatibility layer on every platform, even if you restrict it to things you can currently shim over with reasonable precision, has proven to be a huge mistake for stabilized language ecosystems. For example, the introduction of WASI as a target has made the Rust standard library shim over a huge amount of details in a quite inefficient manner, because their APIs and API consumers are hard-dependent on absolute paths and/or being able to access files and directory paths without an existing directory handle. Zig mostly works with directory and file handles already, and supports iterating over WASI preopens instead of pretending paths like /abc or ./abc are even remotely coherent on WASI.

In essence, although plenty of things can be reasonably supported in a cross-platform manner, you cannot commit yourself to such a specific standard like POSIX and expect it to work flawlessly everywhere and fit every use case. std.os should be native rather than trying to stretch itself over unnatural targets, and features elsewhere in the standard library that depend on OS-specific details should not promise features that are impossible to implement correctly for every target. They should be exposed, just not with the proposition that they will work anywhere. An perfect example of this is std.fs.cwd(), which makes the correct choice by emitting a compile error on WASI instead of attempting to shim a concept that is wholly inapplicable to WASI.

[jayschwa]

I think the standard library has too much if wasi, else if windows, else posix logic sprinkled throughout it. I can't tell if this proposal would help or hurt that.

My half-baked hot take: a namespace like std.fs shouldn't know what an operating system is. Instead, it should provide a conservative interface for file-like objects that other namespaces (OS or otherwise) can satisfy. io.Reader and io.Writer are good (albeit simpler) examples of what I mean.

This could be layered. If I'm writing a package that allows Zig to target "Jayschwa OS", I could choose to satisfy the std.fs interface directly. Or instead, I could choose to satisfy the lower-level POSIX interface (how BYOS kind of works now) and get its support for std.fs (and probably other interfaces) for free.

[katesuyu]

@jayschwa I agree that std.fs should allow overriding the basic file-like objects directly, however I disagree that the switches are bad. Referring to specific concrete APIs, as appropriate, massively improves the readability and navigability of the codebase. Deferring to a duck-typed platform abstraction layer is the real issue. Switches avoid forcing readers of the code to sift through a heap of files to find how each platform implements the same (usually simple) function, and avoid code duplication when multiple platforms are handled by the same switch arm.

Sticking the implementation of everything in separate platform-specific modules is what makes the Rust standard library (and many Rust crates that copy the standard library's organization) horrific to navigate, and the more layers of this you have, the more you end up like our present day std.os (which has already fallen for this trap: see const system and all the conditional usingnamespace declarations).

BYOS should be split up into more discretely implementable interfaces, but there should be no internal abstraction layer: std.fs should keep its switch statements, but allow its structures and functions to be overriden on a case-by-case basis. And specific APIs should not have implementations on platforms where this is impractical!

[Rocknest]

I like the idea of a zig-style posix layer that unifies error codes, checks if arguments are correct, takes care of version prefixed/extended/safe apis. This is must have until zig's own higher level apis provide all possible interactions with the os.
However POSIX as a fit-any-os apis is obsolete, existing "posix-compatible" oses continue to diverge from it, some posix abstraction prevent efficient software so most new oses are not compatible with posix.

I think std.os.posix should exist mostly how it is now, however without any fallbacks, so if i want to use a posix api i must check first if it exists: std.os.posix.has("name"). Maybe we should have a common place for fallbacks that std uses, named std.os.@"$internal$" (or other not trivially accessible name), for example copy_file_range would be moved there.

[squeek502]

I've run into confusion regarding this as well, e.g. with realpath (#4658 (comment)):

(note that fs.realpath is just an alias for os.realpath so the following applies to os.realpath)

Right now, the status quo is:

• On Linux, std.fs.realpath resolves symlinks before ... This matches the system behavior as far as I can tell (cat link/../file will output the contents of linked/../file and fail if it doesn't exist), although there seem to be some edge cases (from my testing, if link is a symlink to linked, then cd link/../dir takes you to ./dir if it exists [ignoring the symlink], otherwise it will take you to linked/../dir [resolving the symlink before ..]).

• On Windows, std.fs.realpath resolves .. before symlinks. This matches the system behavior as far as I can tell (cd link\..\dir will never take you to linked\..\dir; if .\dir does not exist, it will fail with "The system cannot find the path specified.". Same deal with type link\..\file). Please correct me if I'm wrong on this, but I'm not even sure if there exists a function in the Windows API that resolves symlinks before ... daurnimator has mentioned that Zig might need something like RtlDosPathNameToRelativeNtPathName_U_WithStatus but from using the test code at the bottom of this article, that function does not resolve symlinks before .. either. If there is precedence for Linux-like symlink resolution on Windows, it would be helpful to get that information added to std.fs.realpath bugs/inconsistencies on Windows #4658

It's unclear to me what behavior is 'correct,' and currently it feels like both std.fs and std.os are not very clear about how/if they should handle platform-specific behavior.

[qgcarver]

If we go through with this proposal, are we making a decision on #1840 ?

Writing platform code for Windows is pretty shaky right now, and if we're renaming it to posix or not is related. #5037 #4426

Edit: to clarify, #1840 'prefers' ntdll, but #4426 is on shakier ground, no proposal has been accepted.

[Rocknest]

@squeek502 in my opinion all platform dependant behaviour in std.fs and other high level apis is a bug. For example in the case of realpath, on windows it should be emulated with few more syscalls, however if some behaviour is unemulatable then it should return error.Unsupported.

[ron-wolf]

I arrived here via the Zig English Telegram group. Here’s a related essay (found on Lobste.rs) about Go’s somewhat similar issues when compared with Rust: “Early Impressions of Go from a Rust Programmer”

[lukewilson2002]

@squeek502 in my opinion all platform dependant behaviour in std.fs and other high level apis is a bug. For example in the case of realpath, on windows it should be emulated with few more syscalls, however if some behaviour is unemulatable then it should return error.Unsupported.

IMO only common behavior which is known to function fine on most operating systems should be exposed in a high-level interface. There shouldn't be a need for an error.Unsupported when the high-level wrapper over OS functions implements interfaces to common OS functions like File.open(self, URI, mode), File.close(self) etc. OS equivalence should be chosen at compile time, to provide compile-time errors for impossible instructions.

When a user (of the OS library) requires an OS-specific API, they can get it from that OS-specific library. Fits with Zig zen of communicating intent precisely and preferring compile errors opposed to runtime crashes.