An exokernel design would allow for that sort of thing pretty nicely- it strips the kernel down to essentially just the drivers. Everything else could be added either to the kernel in a fork or to user space.

Linux graphics drivers actually already use an interface pretty similar to an exokernel, and new APIs like Mantle and Metal are even closer.