Skip to content

About

Collection of resources about Virtualization

Topics

Resources

Contributing

Stars

2.0k stars

Watchers

81 watching

Forks

Latest commit

 

History

212 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

Awesome Virtualization Awesome

Resources for people who build, break and introspect hypervisors: CPU virtualization internals, hypervisor development, virtual machine introspection and hypervisor security.

Management platforms, desktop apps, plain containers, sandboxes and CPU emulators are out of scope; see Related lists.

Contents

Start here

Write your first hypervisor:

  1. Read Hardware and Software Support for Virtualization for the concepts (Books).
  2. Keep the Intel SDM volume 3C and the VMCS layout open as references (Specifications).
  3. Follow Hypervisor From Scratch or Hypervisor 101 in Rust end to end (Tutorials).
  4. Read SimpleVisor, then HyperPlatform or hvpp, to see complete small implementations (Teaching hypervisors).
  5. Read MMU Virtualization via Intel EPT before adding memory virtualization (Tutorials).
  6. Compare with a production codebase such as Firecracker or NOVA (Production VMMs and research hypervisors).

Start in hypervisor security:

  1. Watch Lessons Learned from Eight Years of Breaking Hypervisors for the bug classes (Attacking hypervisors).
  2. Read Hypervisor Vulnerability Research: State of the Art for the attack surface of each hypervisor (Attacking hypervisors).
  3. Work through one full exploit write-up: An EPYC escape for KVM or the Quarkslab series for Xen (Attacking hypervisors).
  4. Read First Steps in Hyper-V Research to set up a debugging environment (Hyper-V).
  5. Read the HyperCube and Morphuzz papers, then run kAFL (Fuzzing).
  6. Learn what the hardware-isolated threat model changes with SEVered and Heckler (Confidential VMs).

Chronology

Date Vendor Milestone
2005-11-13 Intel VT-x released on Pentium 4 (models 662 and 672)
2006-05-23 AMD AMD-V released on Orleans and Windsor processors
2007-09-10 AMD Barcelona adds RVI (Rapid Virtualization Indexing), a.k.a. Nested Page Tables, AMD's SLAT
2008-11 Intel Nehalem adds EPT (Extended Page Tables, Intel's SLAT) and VPID (Virtual Processor ID)
2010-01-07 Intel Westmere adds unrestricted guest (real mode support)
2013-06-04 Intel Haswell adds VMCS Shadowing, VMFUNC, #VE and EPTP switching
2017-06-21 AMD EPYC adds Secure Encrypted Virtualization (SEV)
2017 AMD Documents Encrypted State (SEV-ES)
2019 AMD Documents Secure Nested Paging (SEV-SNP)
2019-08-01 Intel Ice Lake adds EPT SPP (EPT-Based Sub-Page Write Protection) and virtualization of the Intel Processor Trace output buffer using EPT
2020-03 Intel Documents Hypervisor-Managed Linear Address Translation (HLAT)
2021-03 AMD EPYC Milan ships SEV-SNP
2021-03 Arm Announces the Confidential Compute Architecture (CCA) and its Realm Management Extension (RME) with Armv9-A
2023-01 Intel Sapphire Rapids ships Trust Domain Extensions (TDX)
2024 Linux KVM gains SEV-SNP host support (6.11)
2025 Linux KVM gains TDX host support (6.16)

Foundations

Specifications

Books

Courses

Papers

Hypervisor internals

KVM

Xen

QEMU

Hyper-V

VMware

VirtualBox

Building hypervisors

Tutorials

  • Hypervisor From Scratch - Sina Karvandi. Eight-part series building a VT-x hypervisor for Windows: VMX entry, VMCS setup, EPT, virtualizing a running system and hidden hooks.
  • 5 Days to Virtualization - Daax Rynd. Six-part series from environment setup to a VM-exit handler with CPUID emulation.
  • MMU Virtualization via Intel EPT - Daax Rynd. Follow-up series on paging structures, EPT setup and EPT violations.
  • Hypervisor 101 in Rust - Satoshi Tanda. One-day course that builds a fuzzing hypervisor for Intel and AMD, with exercises.

Teaching hypervisors

Small enough to read in full. Many are no longer maintained and may not build on current toolchains; the code is still the best reference.

  • Blue Pill - Invisible Things Lab, 2007. Source of New Blue Pill, the hypervisor rootkit that moves a running OS into a VM.
  • SimpleVisor - Alex Ionescu, 2016. Minimal VT-x hypervisor for Windows and UEFI, written to be read.
  • HyperPlatform - Satoshi Tanda, 2016. VT-x platform for Windows kernel monitoring. Archived.
  • SimpleSvm - Satoshi Tanda, 2017. The AMD SVM counterpart to SimpleVisor.
  • MiniVisorPkg - Satoshi Tanda, 2020. VT-x hypervisor built as both a UEFI and a Windows driver.
  • hvpp - Petr Beneš, 2018. VT-x hypervisor in modern C++ with EPT hooking examples. Unmaintained since 2021.
  • ksm - Ahmed Samy, 2017. VT-x hypervisor for Windows and Linux using EPT, #VE and VMFUNC. Unmaintained since 2021.
  • gbhv - Gbps, 2018. Heavily commented VT-x hypervisor with EPT hooking. Unmaintained since 2023.
  • NoirVisor - Zero Tang, 2018. Supports both VT-x and AMD-V, with nested virtualization.
  • kHypervisor - Kelvin Chan, 2016. Nested VMM built on HyperPlatform. Unmaintained since 2021.
  • HyperBone - DarthTon, 2016. Minimal VT-x hypervisor with hooks. Unmaintained since 2019.
  • hypervisor-for-beginners - Rohan, 2017. Step-by-step VT-x samples. Unmaintained since 2017.
  • zpp_hypervisor - Eyal Z, 2019. Simple VT-x hypervisor in C++ written as a learning exercise.
  • rustyvisor - Ian Kronquist, 2016. Early VT-x hypervisor in Rust. Unmaintained since 2022.
  • mythril - Type-1 VT-x hypervisor in Rust, 2019. Unmaintained since 2022.
  • illusion-rs - UEFI blue-pill hypervisor in Rust with EPT hooks, by memN0ps. Archived.
  • matrix-rs - Windows kernel blue-pill hypervisor in Rust, by memN0ps. Archived.
  • HOSS - Don Porter, 2014. Extends MIT's JOS teaching OS into a hypervisor, with guided lab exercises.

Special-purpose hypervisors

  • ramooflax - Airbus, 2011. Pre-boot hypervisor for remotely debugging a whole machine. Unmaintained since 2018.
  • VivienneVMM - Stealthy hardware-breakpoint debugging through a VT-x hypervisor, by changeofpace. Unmaintained since 2020.
  • MemoryRanger - Igor Korkin, 2019. Runs kernel drivers in separate EPT enclaves to stop data-only attacks.
  • applepie - Brandon Falk, 2018. Fuzzing hypervisor built on Bochs and the Windows Hypervisor Platform. Unmaintained since 2019.
  • orange_slice - Brandon Falk, 2019. Research kernel and hypervisor in Rust aiming at deterministic execution. Unmaintained since 2021.
  • Barbervisor - Cisco Talos, 2020. Bare-metal VT-x hypervisor in Rust for snapshot fuzzing. Unmaintained since 2020.
  • vbh - Intel, 2019. Thin hypervisor for virtualization-based hardening of a Linux kernel. Archived.

Production VMMs and research hypervisors

  • Firecracker - AWS, 2018. Minimal KVM-based VMM in Rust behind Lambda and Fargate.
  • Cloud Hypervisor - 2019. Rust VMM for cloud workloads on KVM and Microsoft Hypervisor.
  • crosvm - Google, 2017. The ChromeOS and Android VMM, where the Rust VMM ecosystem started.
  • rust-vmm - 2018. Shared Rust crates (KVM bindings, virtio, guest memory) used by the three VMMs above.
  • OpenVMM - Microsoft, 2024. Rust VMM that also runs as OpenHCL, the paravisor used for Azure confidential VMs.
  • uhyve - 2019. Minimal hypervisor for the Hermit unikernel.
  • Android pKVM - Google. Protected KVM on Arm64, which removes the host kernel from the guest's trusted computing base.
  • NOVA - Udo Steinberg, 2010. Microhypervisor with a small trusted computing base for x86 and Arm.
  • Hedron - Cyberus Technology, 2020. NOVA derivative. Unmaintained since 2023.
  • BitVisor - University of Tsukuba, 2008. Thin hypervisor that mediates I/O for a single guest. (GitHub mirror)
  • Xvisor - 2010. Monolithic type-1 hypervisor for Arm, RISC-V and x86.
  • Jailhouse - Siemens, 2013. Linux-based static partitioning hypervisor. No commits since 2024.
  • Bao - 2020. Static partitioning hypervisor for Arm and RISC-V mixed-criticality systems.
  • ACRN - Intel, 2018. Type-1 hypervisor for embedded and IoT devices.
  • Bareflank - Assured Information Security, 2015. SDK for building custom hypervisors. No commits since 2024.
  • Boxy - Assured Information Security, 2018. Bareflank-based hypervisor for running guest VMs, the predecessor of MicroV. Unmaintained since 2020.
  • MicroV - Assured Information Security, 2020. Micro-VM hypervisor built on Bareflank. No commits since 2024.
  • GiantVM - Shanghai Jiao Tong University, 2017. Distributed hypervisor that spans one VM across several hosts.
  • napoca - Bitdefender, 2020. Type-1 hypervisor designed to host memory introspection. Archived.
  • Intel HAXM - Intel, 2017. VT-x accelerator for QEMU on Windows and macOS. Archived.
  • NEMU - Intel, 2018. Stripped-down QEMU fork, superseded by Cloud Hypervisor. Archived.

MicroVMs and container isolation

Projects that put a virtualization boundary around containers and single processes. Firecracker, Cloud Hypervisor and crosvm, listed above, are the VMMs most of them build on.

  • My VM is Lighter (and Safer) than your Container - Manco et al., SOSP 2017. Shows that VMs can boot in milliseconds once the toolstack and guest are trimmed.
  • Kata Containers - 2017. Container runtime that runs each pod inside its own lightweight VM.
  • gVisor - Google, 2018. Application kernel that intercepts container system calls in user space, with an optional KVM platform; a sandbox, not a hypervisor.
  • microvm machine type - QEMU. Minimal x86 machine model with virtio-mmio devices and no PCI, inspired by Firecracker.
  • libkrun - 2020. Library that embeds a minimal KVM or Hypervisor.framework VMM in a process, used by crun and Podman.
  • go-microvm - Stacklok, 2026. Go framework that runs OCI container images as microVMs through libkrun.

Introspection and debugging

Tools

  • LibVMI - The reference VMI library: read and write guest memory and receive hardware events on Xen and KVM.
  • libmicrovmi - Rust library exposing one low-level VMI API across Xen, KVM, VirtualBox and memory dumps.
  • KVM-VMI - KVM and QEMU patches that add an introspection API, with setup instructions.
  • DRAKVUF - Agentless dynamic malware analysis on Xen using altp2m and LibVMI.
  • HVMI - Bitdefender's commercial introspection engine, open-sourced in 2020. Archived.
  • HyperDbg - Hypervisor-based kernel and user-mode debugger for Windows with EPT hooks and scripting.

Papers and talks

Security

Attacking hypervisors

KVM

Xen

VMware

VirtualBox

Hyper-V

Fuzzing

Confidential VMs

VM detection and malware analysis

Related lists

About

Collection of resources about Virtualization

Topics

Resources

Contributing

Stars

2.0k stars

Watchers

81 watching

Forks

Releases

Packages

Used by

Contributors