I don't really like Nix but it's probably fine for this - you have files that come in, a build step, and output files. Your derivations (build recipes) are functions that can take arguments to e.g. provide a set of patches to apply. There are recipes for building a kernel in nixpkgs already which you could fork.
The main annoyance I'd have in your case is probably the number of file copies of large images it may do to keep things isolated, but most hermetic build systems will do something similar.
Nix is very awkward (and slow) to use for fine-grained caching, which is price of entry for a decent build system.
It also comes (practically) shackled to nixpkgs which is a barely-documented cataclysm of packages that extensively bake in assumptions that you are also going to use Nix as your runtime environment.
The principles behind Nix are nice but the implementation is sorely lacking in my experience.
> Nix is very awkward (and slow) to use for fine-grained caching, which is price of entry for a decent build system.
This is sadly true, I've tried setting up a bazel-esque C++ build system in Nix, and the requirement for 1 derivation per .o file makes it very inefficient
> It also comes (practically) shackled to nixpkgs
This isn't true, and I think it is a shame that it seems true. Nixpkgs is both awful, and completely optional. It would be great if there were more resources on how to use Nix without Nixpkgs, or as is commonly requested, a smaller, cleaner and better engineered alternative
Every single time Claude has confused me and I've asked what it's talking about, it's because it's got something completely wrong and has managed to obfuscate the wrongness behind never-introduced terminology, poor analogies and (what I can only assume is) exposure to wording it's used in its chain of thought reasoning.
A single question is enough for it to retract the error and correct itself. Suggesting that not understanding some of these messages is a lack of human comprehension rather than the agent being flat out wrong is...a bold take.
Typically a feature of good human technical communication is the ability to concisely explain key ideas so one can quickly identify any divergences between understanding. Opus 5 is dreadful at this.
The single saving grace is the intuition that if I don't understand it's probably wrong.
I've recently been going down the rabbit hole of creating a "fast start dev env" and it's interesting to see how this article differs from other approaches (codesandbox has some fantastic blogs, the fly.io blog on sprites has interesting pointers, e2b and daytona are related open source tools). Everyone has a different solution based on their tradeoffs.
I thought the memory snapshotting part in particular was clever since most container based systems don't bother (VM/firecracker based ones can use UFFD and call it a day), but by having emulated syscalls you can actually do single-process restore pretty well.
I am a bit dubious of the use of fuse (though it clearly works well!), and I wonder if ublk (what I ended up using) might alleviate some of the pain/magic in fuse tuning. I'd personally also be looking at forking gvisor to take a memfd which you enable UFFD on for the page loading (I have some firecracker patches where I do the same). It's nice because you can optimistically push pages, rather than waiting for the requests to come in. The series of three codesandbox blog posts are good background reading.
Our system somewhat predates ublk; at the time we wrote it, FUSE was the most reasonable option. Moving over to ublk would require re-architecting around block devices rather than filesystems, but is indeed something we're looking at!
I'm super interested to hear more on anything you can share about your projects, or the niche of gov projects you're aware of - I've been doing some work with gov and haven't seen this requirement yet, so want to be prepared if it does come up.
One annoying part of using chroot if you're creating them on the fly is teardown - you have to manually invoke umount, and also take care to get this right for partially created chroots (maybe you detected an error after mounting proc, in the process of getting other files in place).
This was my original motivation in creating machroot (mentioned elsewhere in this thread) and having it use namespaces.
As a number of comments have noted, there are a bunch of different axes that chroot could be 'better' on - e.g. security and sandboxing.
I wrote https://github.com/aidanhs/machroot (initially forked from bubble wrap) a while ago to lean into the pure "pretend I see another filesystem" aspect of chroot with additional conveniences (so no security focus). For example, it allows setting up overlay filesystems, allows mounting squashfs filesystems with an overlay on top...and because it uses a mount namespace, means you don't need to tear down the mount points - just exit the command and you're done.
The codebase is pretty small so I just tweaked it with whatever features I needed at the time, rather than try and make it a fully fledged tool.
(honestly you can probably replicate most of it with a shell script that invokes unshare and appropriate mount commands)
Out of curiosity, what would you say the current state of the art is for full compilable decompilation? This is something I have a vague interest in but I'm not involved enough in the space to be on top of the latest and greatest tooling.
Echoing IDA but its pricing is a huge PITA if you’re using it in a hobbyist capacity i.e. you don’t have an employer willing to pay for it. Could opt for the home version but that’s a yearly cost and you have to use their cloud decompiler. Ghidra’s your best bet if you want something FOSS and community-driven although not as great at decompilation.
Not only the pricing by itself, every story that I've heard about normal people trying to actually give them money is that they actually don't want to sell it to anyone other than big players
Hexrays used to be difficult to deal with if you want to purchase IDA Pro for the first time, due to their software getting leaked online.
They have eased the procedure to buy from them, but from time to time they'll ask you to fill out your info with national ID/passport (they say its because they don't want to sell their software to individuals under sanctions). This is despite them being based in Belgium (not the US).
For any serious work IDA Pro is highly suitable (the customization and scripting, loader examples and processor plugins...etc), on the other hand for side projects and basic security research Binary ninja and ghidra can go along way.
Most decompilers do not strive for recompilability. [1] I believe there are (or were) some academic projects that aimed for recompilation as a core feature, but it is a hard problem.
On the commercial side, IDA / HexRays [2] is very strong for C-like decompilation. If you're looking at Go, Rust, or even C++ it is going to be a little bit more messy. As other commenters have said, you'll work function-by-function and it is expensive, though the free version does have decompilation (F5) for x86 and x64 (IIRC).
Binary Ninja [3] (no affiliation) is the coolest IMO, they have multiple intermediate representations they lift the assembly through. So you get like assembly -> low level IL -> medium level IL -> high level IL. There are also SSA forms (static single assignment) that can aid in programmatic analyses. The high level IL is very readable but makes no effort to be compilable as a programming language. That being said, Binary Ninja has implemented different "views" on the HLIL so you can show it as pseudo-C, Rust, etc. There is a free online version and the commercial version is cheaper than IDA but still expensive. Good Python API, good UI.
Ghidra [4] is the RE framework released by NSA. It is free and open source. It supports a ton of niche architectures. This is what most people use. I think the UI is awful, personally. It has a decompiler, the results are OK. They have an intermediate representation (P-Code) and plugins are in Java (since it is written in Java). I haven't worked much with it.
Most online decompilations you see for old games are likely using Ghidra, some might be using IDA. This is largely a manual process of doing a function at a time and building up the mental map of the program and how things interact.
Also worth mentioning are lifters. There were a few projects that aimed to lift assembly to LLVM IR (compiler framework's intermediate representation), with the idea being that then all your analyses could be written over LLVM IR as a lingua franca. Since it is in LLVM IR, it would be also recompilable and retargetable. [5][6]
It's not clear to me why that is so. An LLM trained on IR for the purpose of compilation is not quite what we're looking for here but it is in the same territory.
Looking at an individual function, IDA hex-rays output is often recompilable as-is (or with minor modifications), but it won't necessarily be idiomatic, especially if you don't have symbol information.
All the emulation of desktop machines in WASM I've seen so far have been for x86 - do you think there are significant additional hurdles for x86_64? Or is it just a matter of time?
Separately, one bit of feedback - it's cool that webvm is open source, but I think it's fair to ask you to be upfront that cheerpx itself is not (which is fine!) in the blog post itself where you talk about webvm licensing. If I wasn't already familiar with the wasm emulation space I would have felt rather misled.
I cannot speak for other VMs, in the case of CheerpX there is nothing fundamental preventing emulation/JIT-ting of 64-bit platforms, but it is an inefficient choice due to the current limitations of Wasm, in particular on the front of how much memory can be used in total. In the best case scenario it would be 4GB, but it's unlikely this can be achieved on all devices in the real world.
64-bit code by its nature consume more memory, each pointer is twice the size, which makes the memory limitation even more pressing for no advantage. Please note, that this is unrelated to the work on WebAssembly Memory64. The issue is not the lack of address space, but rather on the actual memory that can be allocated in practice.
For this public deployment of WebVM we have chosen to limit the maximum memory to 700MBs, which makes the demo work fine on the vast majority of devices, including mobile ones. This said, we do plan to support the 64-bit ISA in the future with the main use case of supporting all the existing docker images available on Docker Hub and similar platforms.
Appreciate your feedback on licensing and we will take it into account, but please notice that this article is specifically about WebVM, that is indeed FOSS. A separate article dedicated to the CheerpX 1.0 release will be published soon, and it will of course be very clear that the latter is proprietary.
Interesting, that's helpful, thanks - so with the eventual arrival of memory64 and assuming I only wanted to target desktop systems and assuming browser implementations permit large allocations (e.g. 8GB) - large 64bit apps could work fine. I have a use case for this I've been poking at for a bit, but implementing my own version of cheerpx would be a lot of work, maybe I'll just wait!
On open source - I can only give you feedback as an outside fresh pair of eyes :) I incorrectly interpreted that it was full stack OSS based on the overall blog post 'vibe' and had to deliberately double check because I was aware of cheerpx beforehand. Perhaps it's just me. I look forward to the cheerpx blog post!
The main annoyance I'd have in your case is probably the number of file copies of large images it may do to keep things isolated, but most hermetic build systems will do something similar.
reply