Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.
Firecracker is an alternative to QEMU that is purpose-built for running serverless
functions and containers safely and efficiently, and nothing more. Firecracker is
written in Rust, provides a minimal required device model to the guest operating
system while excluding non-essential functionality (only 5 emulated devices are
available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal
keyboard controller used only to stop the microVM). This, along with a streamlined
kernel loading process enables a < 125 ms startup time and a < 5 MiB memory
footprint.
Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".
Can you be specific about what those hardware things are, and why they matter for serverside workloads? Yes: the premise of Firecracker is that the hypervisor only supports the virtio devices, and doesn't emulate real hardware. Where does that end up being problematic?
I'm not sure what you mean by your "CPU and memory" being "limited", either.
- Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized app? Not gonna work... IPSec/IKE VPN? May not work (iirc this requires some specific paravirt features? I might be misremembering). Have some advanced network controller that transparently handles mesh networks, or some other fancy shmancy system designed to manage complex interactions between containers and networks? Probably not gonna work due to assumptions about what lies between the layers, what components use what tricks to handle advanced routing (Netfilter, eBGP, sidecars, etc). To say nothing of link-level differences (what if your network isn't Ethernet?). And all traffic is copied from an I/O thread of an emulated network device to a host TAP device, before it makes it to the real device; my guess is network latency and maximum PPS were not a priority.
- The exposed CPU is based on what KVM supports exposing to guests; presumably QEMU supports a much wider selection. Clocksources, of which I know absolutely nothing (:D), are only allowed as kvm-clock.
- All I/O is rate-limited by a custom scheme built by the Firecracker authors. I'm sure this is fine for most use-cases, but some weird high-performance outlier is gonna hit some sort of bottleneck with this thing I'm sure.
- Firecracker emulated block devices are backed by files on the host. Ergo, any app that wants to control a disk directly, or use some fancy shmancy SAN directly, etc is out of luck.
- The guest requires a balloon driver to use balloon support, which means the guest needs this special software, and compromising the guest driver could be a serious issue. I don't know if Kata does this differently.
- Aarch64 support has a bunch of errata currently, so x86_64 is the only fully-supported platform; I dunno if Kata does any better, but this is a real hardware limitation, esp. if you're trying to buy a shit-ton of cheap powerful machines.
- This is not hardware-specific, but Firecracker seems limited to specific kernel releases; right now the latest it supports is 5.10, according to https://github.com/firecracker-microvm/firecracker/blob/main... They emphasize that they want your host and guest to run "a supported kernel", even if it's possible to run different ones. To me that says that there's the potential for Firecracker-specific bugs in newer (or older?) guest kernels. From that page: "Firecracker represents a component in a larger stack, one in which it is tightly coupled with the guest and host kernels on which it is run."
I would add that containers are used in far more settings than server-side, and would be great to have on the edge, in IoT, and on desktops, if they were a little less... funky. In general, containers requires a lot of extra steps to be "usable" as general purpose applications, and access to hardware will absolutely be a barrier we need to cross in order for "general purpose containerized apps" to become commonplace.
I'm not trying to be argumentative, but rather just to clear things up for you:
All serverside cloud-style VMs get tap devices. That's not a Firecracker thing.
IPSEC works just fine from within a VM. Firecracker doesn't care; it's just a hypervisor.
I'm not sure what you're trying to say about the CPU thing; Firecracker is a hypervisor, not an emulator. Linux QEMU VMs are KVM, too.
SANs work fine from within VMs. The point of a SAN is that the disk isn't attached. Firecracker talks to host block devices the same way other hypervisors do.
Here's `uname -a` from a Firecracker:
Linux 0a581153 5.12.2 #1 SMP Thu Jun 30 19:35:04 UTC 2022 x86_64 GNU/Linux
Unless you want to PCI pass-through an SRIOV VF into a guest? If you're paying out the ass for Cisco cloud gear you probably want the guest to have direct access to a card...
Again, I don't remember the specifics, but depending on the underlying networking (on the host etc), there may be some issues with IPSec (but I could be wrong). From some random searching: https://linux-ipsec.org/wp-content/uploads/slides/2018/quest...
> SANs work fine from within VMs. The point of a SAN is that the disk isn't attached. Firecracker talks to host block devices the same way other virtualizers do.
But you may want to talk to an attached HBA? Firecracker's docs (https://github.com/firecracker-microvm/firecracker/blob/main...) say: "Firecracker emulated block devices are backed by files on the host. To be able to mount block devices in the guest, the backing files need to be pre-formatted with a filesystem that the guest kernel supports."
Based on everything else in the Firecracker docs, I don't see any vHBA support provided, so the guest cannot access an HBA on the host. And a virtual device backed by a file is going to have crap performance.
Yep, PCI pass-through is 100% something you can't do with Firecracker. To me, that's a perfect example of a feature that's not really there to support cloud-style serverside workloads, but that might just be my bias.
I think the other way to look at it is there's just different workloads that need different solutions. Firecracker is perfect for Lamba-style workloads.
But some companies (who don't want to manage datacenter resources, but do want the large scale and flexibility of cloud computing) want a whiz-bang multi-tenant enterprise containerized solution (say, using K8s as orchestrator). And maybe one of their critical applications has certain requirements that necessitate an HBA, deep packet inspection, an HSM, a high-precision clock, etc. They need strong isolation guarantees, so they want a VM, but they want K8s to manage it.
Kata Containers seems to fit the latter, as it can change hypervisors, supports more architectures, advanced networking features, device assignment, etc. Seems to align most with on-prem cloud as it's maintained by the OpenStack people.
I'll bet you it's more like 85%. AWS has a good amount of hardware designed to pass through into the VM, various custom guest drivers, some advanced high-performance networking stuff, and the GPU support like you mentioned. Customers probably pay a hefty premium, so we shouldn't discount it from either a business or technical perspective.
>"All serverside cloud-style VMs get tap devices."
I'm having trouble understanding this sentence. Specifically "serverside cloud-style VMs." Does "serverside" mean from the host OS that's running the Hypervisor? Like if I were logged into that host an "ifconfig" would show me tap devices?
