feat: implement hypervisor extension

cartesi · Oct 10, 2023 · 72c94d9 · 72c94d9
1 parent cc52626
commit 72c94d9
Show file tree

Hide file tree

Showing 30 changed files with 5,082 additions and 468 deletions.
diff --git a/hypervisor.md b/hypervisor.md
@@ -0,0 +1,129 @@
+# Hypervisor usage instructions
+
+The hypervisor feature is still at a highly experimental stage.
+
+In this document, we will use the term _host_ (or _host machine_) to address a cartesi machine instance that hosts a hypervisor, and the term _guest_ (or _guest machine_, or _virtual machine_) to address a machine running inside a hypervisor.
+
+## Quick start
+
+### Prepare the artifacts
+
+To test the hypervisor without building everything from scratch, you could download the rootfs and the kernel from google drive:
+- [rootfs](https://drive.google.com/file/d/1Hy9VibEf6SZU4qtqqb8n5CzHq-eN7uO7/view?usp=share_link)
+- [kernel](https://drive.google.com/file/d/1Wc9yAJLpxxg14aFDPvYcQQmrA7JPt1ZC/view?usp=share_link)
+
+### Build the emulator with a hypervisor support
+
+To build the emulator with hypervisor support you have to use the `hypervisor` [branch](https://github.com/cartesi-corp/machine-emulator/tree/feature/hypervisor) and follow the [regular build instructions](https://github.com/cartesi-corp/machine-emulator/blob/develop/README.md). After you execute `make install`, the emulator with hypervisor support will be installed in the `/opt/cartesi-hp` folder.
+
+### Run the virtual machine
+
+First of all, you have to make sure that you are using the correct emulator version. There is a special `set_lua_env_hp.sh` script for this in the repo root you may want to use. The command `source set_lua_env_hp.sh` will do the job.
+
+#### Booting a host
+
+To launch a new cartesi machine instance, use the following command:
+
+```bash
+/opt/cartesi-hp/bin/cartesi-machine \
+    --ram-image=/path/to/downloaded/opensbi.bin \
+    --rom-image=/opt/cartesi/share/images/rom-v0.13.0.bin \
+    --ram-length=1024Mi \
+    --flash-drive=label:root,filename:/path/to/downloaded/rootfs-v0.15.0-dirty.ext2 \
+    -i -- "/bin/sh"
+```
+
+This will give you a command prompt inside a host.
+
+#### Booting a guest
+
+All the hypervisor stuff you need is located inside the `/hp` folder on the host. To execute a virtual machine follow these steps:
+1. load the `kvm` kernel module: `cd /hp && insmod kvm.ko`;
+1. start the virtual machine with the provided script: `./start_machine.sh`.
+
+At this point, you should get a command prompt inside a virtual machine.
+
+### Looking around
+
+#### Testing network
+
+Inside both host and guest file systems you will find a `sender.py` and a `receiver.py` Python scripts. For the host, the scripts are located inside the `/hp` folder; for the guest look for them inside the `/opt` folder. These scripts could be used to test a network connection between the host and the guest.
+    - `receiver.py <address to listen> <port to listen>`: listens for the incoming data on the given address, and prints the data as soon as it is received;
+    - `sender.py <address to use> <port to use> <data>`: sends the `data` to the given address.
+
+#### Guest startup script
+
+The `init` process inside the guest executes an `/opt/start.sh` script. You may want to modify this script to customize the guest startup behavior.
+
+#### Modifying a guest file system
+
+At some point, you may want to persist changes in the guest file system. The corresponding `.ext2` file is located inside the host file system: `/hp/rootfs-virt.ext2`. You can mount this file and make any changes to the corresponding file system:
+
+```bash
+$ sudo mount /path/to/downloaded/rootfs-v0.15.0-dirty.ext2 /mnt
+$ sudo mount /mnt/hp/rootfs-virt.ext2 /mnt-virt
+...
+$ sudo umount /mnt-virt
+$ sudo umount /mnt
+```
+
+## Bootstrapping from scratch
+
+### Build the emulator with a hypervisor support
+
+Please, refer to [this section](https://github.com/cartesi-corp/machine-emulator/blob/hypervisor/hypervisor.md#build-the-emulator-with-a-hypervisor-support).
+
+### Build a kernel
+
+The hypervisor extension support for the RISC-V architecture is available only in the Linux kernel v6.0.9+. So, the first step you have to do is to clone the `cartesi-corp/linux` repo and checkout the corresponding branch:
+
+```bash
+$ git clone [email protected]:cartesi-corp/linux.git
+$ git checkout update/linux-6.0.9-ctsi-y
+```
+
+To build the kernel you have to use the [correct config](https://github.com/cartesi-corp/image-kernel/blob/hypervisor-config/configs/kvm-linux-config). Do not forget to copy it to your Linux kernel repo root directory.
+
+You also have to use `opensbi` to boot the kernel to have the compatible SBI interface version:
+
+```bash
+$ git clone [email protected]:cartesi-corp/opensbi.git
+$ git checkout feature/cartesi-legacy
+```
+
+Now you are ready to build a kernel using the Cartesi toolchain:
+
+```bash
+$ docker run -v /path/to/linux:/linux -v /path/to/opensbi:/opensbi -it cartesicorp/toolchain:0.14.0
+$ export ARCH=riscv; export CROSS_COMPILE=/opt/riscv/riscv64-cartesi-linux-gnu/bin/riscv64-cartesi-linux-gnu-; make Image
+$ export FW_PAYLOAD_PATH=/linux/arch/riscv/boot/Image; export PLATFORM=cartesi; make
+```
+
+You should have three files as output:
+1. host kernel: `/path/to/opensbi/platform/cartesi/firmware/fw_payload.bin`;
+1. guest kernel: `/path/to/linux/arch/riscv/boot/Image`;
+1. kvm kernel module: `/path/to/linux/arch/riscv/kvm/kvm.ko`.
+
+### Build root file systems
+
+To be able to work with a hypervisor, you need root user permissions. The current root file system build does not provide this capability, so you have to use the version from the `hypervisor` [branch](https://github.com/cartesi-corp/image-rootfs/tree/hypervisor) to fix this. Other aspects are not different from the [regular build process](https://github.com/cartesi-corp/image-rootfs/blob/develop/README.md). Just keep in mind that after the host rootfs is compiled, you will have to copy the hypervisor-related files to it (KVM kernel module, lkvm tool, guest kernel, guest root file system, and any supporting scripts), so there should be enough free space.
+
+The same rootfs build may be used both for the host and the guest.
+
+### Build the LKVM tool
+
+To run a virtual machine [we need](https://github.com/kvm-riscv/howto/wiki/KVM-RISCV64-on-Spike#4-add-libfdt-library-to-cross_compile-sysroot-directory) lkvm tool. Here are the steps to build it (should be executed inside the toolchain container):
+
+```bash
+$ git clone git://git.kernel.org/pub/scm/utils/dtc/dtc.git
+$ cd dtc
+$ export ARCH=riscv; export CROSS_COMPILE=/opt/riscv/riscv64-cartesi-linux-gnu/bin/riscv64-cartesi-linux-gnu-; export CC="${CROSS_COMPILE}gcc -mabi=lp64d -march=rv64gc"
+$ SYSROOT=$($CC -print-sysroot)
+$ make libfdt
+$ make NO_PYTHON=1 NO_YAML=1 DESTDIR=$SYSROOT PREFIX=/usr LIBDIR=/usr/lib64/lp64d install-lib install-includes
+$ cd ..
+$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git; cd kvmtool; make lkvm-static
+$ ${CROSS_COMPILE}strip lkvm-static
+```
+
+The above commands will create `kvmtool/lkvm-static` that you need to copy to your host root file system.
diff --git a/lib/grpc-interfaces b/lib/grpc-interfaces
diff --git a/set_lua_env_hp.sh b/set_lua_env_hp.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+export LUA_CPATH="/opt/cartesi-hp/lib/?.so;/opt/cartesi-hp/lib/lua/5.3/?.so;/opt/cartesi-hp/lib/lua/5.3/cartesi-hp/?.so;/usr/lib/lua/5.3/?.so"
+export LUA_PATH="/opt/cartesi-hp/bin/?.lua;/opt/cartesi-hp/share/lua/5.3/?.lua;/usr/share/lua/5.3/?.lua"
diff --git a/src/clua-i-virtual-machine.cpp b/src/clua-i-virtual-machine.cpp
@@ -138,6 +138,25 @@ IMPL_MACHINE_OBJ_READ_WRITE(stval)
 IMPL_MACHINE_OBJ_READ_WRITE(satp)
 IMPL_MACHINE_OBJ_READ_WRITE(scounteren)
 IMPL_MACHINE_OBJ_READ_WRITE(senvcfg)
+IMPL_MACHINE_OBJ_READ_WRITE(hstatus)
+IMPL_MACHINE_OBJ_READ_WRITE(hideleg)
+IMPL_MACHINE_OBJ_READ_WRITE(hedeleg)
+IMPL_MACHINE_OBJ_READ_WRITE(hie)
+IMPL_MACHINE_OBJ_READ_WRITE(hip)
+IMPL_MACHINE_OBJ_READ_WRITE(hvip)
+IMPL_MACHINE_OBJ_READ_WRITE(hgatp)
+IMPL_MACHINE_OBJ_READ_WRITE(henvcfg)
+IMPL_MACHINE_OBJ_READ_WRITE(htimedelta)
+IMPL_MACHINE_OBJ_READ_WRITE(htval)
+IMPL_MACHINE_OBJ_READ_WRITE(vsepc)
+IMPL_MACHINE_OBJ_READ_WRITE(vsstatus)
+IMPL_MACHINE_OBJ_READ_WRITE(vscause)
+IMPL_MACHINE_OBJ_READ_WRITE(vstval)
+IMPL_MACHINE_OBJ_READ_WRITE(vstvec)
+IMPL_MACHINE_OBJ_READ_WRITE(vsscratch)
+IMPL_MACHINE_OBJ_READ_WRITE(vsatp)
+IMPL_MACHINE_OBJ_READ_WRITE(vsie)
+IMPL_MACHINE_OBJ_READ_WRITE(vsip)
 IMPL_MACHINE_OBJ_READ_WRITE(ilrsc)
 IMPL_MACHINE_OBJ_READ_WRITE(iflags)
 IMPL_MACHINE_OBJ_READ_WRITE(htif_tohost)
@@ -588,6 +607,25 @@ static const auto machine_obj_index = cartesi::clua_make_luaL_Reg_array({
     {"read_sscratch", machine_obj_index_read_sscratch},
     {"read_stval", machine_obj_index_read_stval},
     {"read_stvec", machine_obj_index_read_stvec},
+    {"read_hstatus", machine_obj_index_read_hstatus},
+    {"read_hideleg", machine_obj_index_read_hideleg},
+    {"read_hedeleg", machine_obj_index_read_hedeleg},
+    {"read_hie", machine_obj_index_read_hie},
+    {"read_hip", machine_obj_index_read_hip},
+    {"read_hvip", machine_obj_index_read_hvip},
+    {"read_hgatp", machine_obj_index_read_hgatp},
+    {"read_henvcfg", machine_obj_index_read_henvcfg},
+    {"read_htimedelta", machine_obj_index_read_htimedelta},
+    {"read_htval", machine_obj_index_read_htval},
+    {"read_vsepc", machine_obj_index_read_vsepc},
+    {"read_vsstatus", machine_obj_index_read_vsstatus},
+    {"read_vscause", machine_obj_index_read_vscause},
+    {"read_vstval", machine_obj_index_read_vstval},
+    {"read_vstvec", machine_obj_index_read_vstvec},
+    {"read_vsscratch", machine_obj_index_read_vsscratch},
+    {"read_vsatp", machine_obj_index_read_vsatp},
+    {"read_vsie", machine_obj_index_read_vsie},
+    {"read_vsip", machine_obj_index_read_vsip},
     {"read_word", machine_obj_index_read_word},
     {"read_x", machine_obj_index_read_x},
     {"read_f", machine_obj_index_read_f},
@@ -637,6 +675,25 @@ static const auto machine_obj_index = cartesi::clua_make_luaL_Reg_array({
     {"write_sscratch", machine_obj_index_write_sscratch},
     {"write_stval", machine_obj_index_write_stval},
     {"write_stvec", machine_obj_index_write_stvec},
+    {"write_hstatus", machine_obj_index_write_hstatus},
+    {"write_hideleg", machine_obj_index_write_hideleg},
+    {"write_hedeleg", machine_obj_index_write_hedeleg},
+    {"write_hie", machine_obj_index_write_hie},
+    {"write_hip", machine_obj_index_write_hip},
+    {"write_hvip", machine_obj_index_write_hvip},
+    {"write_hgatp", machine_obj_index_write_hgatp},
+    {"write_henvcfg", machine_obj_index_write_henvcfg},
+    {"write_htimedelta", machine_obj_index_write_htimedelta},
+    {"write_htval", machine_obj_index_write_htval},
+    {"write_vsepc", machine_obj_index_write_vsepc},
+    {"write_vsstatus", machine_obj_index_write_vsstatus},
+    {"write_vscause", machine_obj_index_write_vscause},
+    {"write_vstval", machine_obj_index_write_vstval},
+    {"write_vstvec", machine_obj_index_write_vstvec},
+    {"write_vsscratch", machine_obj_index_write_vsscratch},
+    {"write_vsatp", machine_obj_index_write_vsatp},
+    {"write_vsie", machine_obj_index_write_vsie},
+    {"write_vsip", machine_obj_index_write_vsip},
     {"write_x", machine_obj_index_write_x},
     {"write_f", machine_obj_index_write_f},
     {"replace_memory_range", machine_obj_index_replace_memory_range},

diff --git a/src/clua-machine-util.cpp b/src/clua-machine-util.cpp
@@ -540,6 +540,24 @@ CM_PROC_CSR clua_check_cm_proc_csr(lua_State *L, int idx) try {
         {"satp", CM_PROC_SATP},
         {"scounteren", CM_PROC_SCOUNTEREN},
         {"senvcfg", CM_PROC_SENVCFG},
+        {"hstatus", CM_PROC_HSTATUS},
+        {"hedeleg", CM_PROC_HEDELEG},
+        {"hideleg", CM_PROC_HIDELEG},
+        {"hip", CM_PROC_HIP},
+        {"hie", CM_PROC_HIE},
+        {"hvip", CM_PROC_HVIP},
+        {"hgatp", CM_PROC_HGATP},
+        {"htimedelta", CM_PROC_HTIMEDELTA},
+        {"htval", CM_PROC_HTVAL},
+        {"vsepc", CM_PROC_VSEPC},
+        {"vsstatus", CM_PROC_VSSTATUS},
+        {"vscause", CM_PROC_VSCAUSE},
+        {"vstval", CM_PROC_VSTVAL},
+        {"vstvec", CM_PROC_VSTVEC},
+        {"vsscratch", CM_PROC_VSSCRATCH},
+        {"vsatp", CM_PROC_VSATP},
+        {"vsip", CM_PROC_VSIP},
+        {"vsie", CM_PROC_VSIE},
         {"ilrsc", CM_PROC_ILRSC},
         {"iflags", CM_PROC_IFLAGS},
         {"clint_mtimecmp", CM_PROC_CLINT_MTIMECMP},
@@ -755,6 +773,24 @@ static void push_cm_processor_config(lua_State *L, const cm_processor_config *p)
     PUSH_CM_PROCESSOR_CONFIG_CSR(satp);
     PUSH_CM_PROCESSOR_CONFIG_CSR(scounteren);
     PUSH_CM_PROCESSOR_CONFIG_CSR(senvcfg);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hstatus);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hideleg);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hedeleg);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hip);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hvip);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hie);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(hgatp);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(htimedelta);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(htval);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vsepc);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vsstatus);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vscause);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vstval);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vstvec);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vsscratch);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vsatp);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vsie);
+    PUSH_CM_PROCESSOR_CONFIG_CSR(vsip);
     PUSH_CM_PROCESSOR_CONFIG_CSR(ilrsc);
     PUSH_CM_PROCESSOR_CONFIG_CSR(iflags);
 }
@@ -1079,6 +1115,24 @@ static void check_cm_processor_config(lua_State *L, int tabidx, cm_processor_con
     p->satp = opt_uint_field(L, -1, "satp", def->satp);
     p->scounteren = opt_uint_field(L, -1, "scounteren", def->scounteren);
     p->senvcfg = opt_uint_field(L, -1, "senvcfg", def->senvcfg);
+    p->hstatus = opt_uint_field(L, -1, "hstatus", def->hstatus);
+    p->hideleg = opt_uint_field(L, -1, "hideleg", def->hideleg);
+    p->hedeleg = opt_uint_field(L, -1, "hedeleg", def->hedeleg);
+    p->hip = opt_uint_field(L, -1, "hip", def->hip);
+    p->hvip = opt_uint_field(L, -1, "hvip", def->hvip);
+    p->hie = opt_uint_field(L, -1, "hie", def->hie);
+    p->hgatp = opt_uint_field(L, -1, "hgatp", def->hgatp);
+    p->htimedelta = opt_uint_field(L, -1, "htimedelta", def->htimedelta);
+    p->htval = opt_uint_field(L, -1, "htval", def->htval);
+    p->vsepc = opt_uint_field(L, -1, "vsepc", def->vsepc);
+    p->vsstatus = opt_uint_field(L, -1, "vsstatus", def->vsstatus);
+    p->vscause = opt_uint_field(L, -1, "vscause", def->vscause);
+    p->vstval = opt_uint_field(L, -1, "vstval", def->vstval);
+    p->vstvec = opt_uint_field(L, -1, "vstvec", def->vstvec);
+    p->vsscratch = opt_uint_field(L, -1, "vsscratch", def->vsscratch);
+    p->vsatp = opt_uint_field(L, -1, "vsatp", def->vsatp);
+    p->vsie = opt_uint_field(L, -1, "vsie", def->vsie);
+    p->vsip = opt_uint_field(L, -1, "vsip", def->vsip);
     p->ilrsc = opt_uint_field(L, -1, "ilrsc", def->ilrsc);
     p->iflags = opt_uint_field(L, -1, "iflags", def->iflags);
     lua_pop(L, 1);