Resources
Videos
- Kernel Recipes 2014 - The Linux Kernel, how fast it is developed and how we stay sane doing it
- LPC2019 - Reflections on kernel quality, development process and testing
- virtio
Quickstart
We are going to download the Git repositories of QEMU and the Linux kernel. We are not going to cover what dependencies are required. You should read the reference links for each section to help you get the compilers and other build tools you will need.
Building the Kernel
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git checkout v5.8
# Copy your system's config into the kernel source tree to use as your config
# You can modify it too if you want but you can be pretty sure this will give
# you a working kernel without having to edit it
cp $(ls /boot/config-* | head -n 1) .config
make olddefconfig
# Build just the bzImage needed to boot the VM. If you want to build all the
# modules too, just leave off the last argument (bzImage)
make -j $(($(nproc)*4)) bzImage
References
Building QEMU
git clone https://git.qemu.org/git/qemu.git
cd qemu
git checkout v5.1.0
git submodule init
git submodule update --recursive
mkdir build
cd build
../configure --target-list=x86_64-softmmu
make -j $(($(nproc)*4))
References
Running your VM
Put the following script in a file called run-vm.sh. It will be the way you
run your development Kernel in a virtual machine. You can modify it as you wish.
Here's how you download it, make it executable, and run it.
curl -o run-vm.sh -fSL https://gist.github.com/johnandersen777/a0dc3a58b4651dc3761bee65a198a80d/raw/da2f456c9ecbb56bf84ee30c8c83a2762e86fb43/run-vm.sh && \
echo "eb5c49fb0aff6b3293ad6f4bd8c7a9c32df97f40d3c8c4fe404b72c1e9c283b44e714be493ce88b5f22e5bb717b8f71d run-vm.sh" | sha384sum -c - && \
chmod 755 run-vm.sh
./run-vm.sh
Tools
- Grab patches from mailing list
Exploitation
SMEP disable via ROP, KASLR bypasss via DMESG
Hacking With VIM
https://stackoverflow.com/questions/33676829/vim-configuration-for-linux-kernel-development
Build In Tree Modules
make -j $(($(nproc)*4)) M=arch/x86/kvm/ modules
Install Kernel To chroot
Here's how you install a kernel to a chroot. Or tar it up and send it over to another system for use there.
# Edit your config
make olddefconfig
export INSTALL_MOD_PATH=/path/to/your/chroot
export INSTALL_PATH=${INSTALL_MOD_PATH}/boot
mkdir -p "${INSTALL_PATH}"
# Use number of cores times 4, this usually is about how many
# threads your system can run
make -j $(($(nproc)*4))
make install
make modules_install -j $(($(nproc)*4))
Chroot
Instead of doing a regular chroot use systemd-nspawn which gives you a more fully featured chroot.
https://wiki.archlinux.org/index.php/Systemd-nspawn
Debugging With GDB
If you are lucky enough to be working on something that's in a VM you can get GDB working!
You need to run the kernel without KASLR (on the cmdline that's nokaslr).
If you don't disable KASLR gdb can't set breakpoints or display the source to you
Then you run QEMU with -s -S.
-ssays enable GDB on port1234-Ssays wait to start the VM until GDB connects.
Run GDB with gdb ./vmlinux
Now run
(gdb) target remote 127.0.0.1:1234
To view the source code, type layout src
To focus back on the typey type window, type focus cmd
┌──arch/x86/kernel/machine_kexec_64.c─────────────────────────────────────────────────────────────────┐
│353 if (result) │
│354 return result; │
│355 │
│356 /* update purgatory as needed */ │
│357 result = arch_update_purgatory(image); │
│358 if (result) │
│359 return result; │
│360 │
│361 return 0; │
│362 } │
│363 │
│364 void machine_kexec_cleanup(struct kimage *image) │
│365 { │
│366 free_transition_pgtable(image); │
│367 } │
│368 │
│369 /* │
│370 * Do not allocate memory (or fail in any way) in machine_kexec(). │
│371 * We are past the point of no return, committed to rebooting now. │
│372 */ │
│373 void machine_kexec(struct kimage *image) │
B+>│374 { │
│375 unsigned long page_list[PAGES_NR]; │
│376 void *control_page; │
│377 int save_ftrace_enabled; │
│378 │
│379 #ifdef CONFIG_KEXEC_JUMP │
│380 if (image->preserve_context) │
│381 save_processor_state(); │
│382 #endif │
│383 │
│384 save_ftrace_enabled = __ftrace_enabled_save(); │
│385 │
│386 /* Interrupts aren't acceptable while we reboot */ │
│387 local_irq_disable(); │
│388 hw_breakpoint_disable(); │
│389 │
│390 if (image->preserve_context) { │
│391 #ifdef CONFIG_X86_IO_APIC │
│392 /* │
│393 * We need to put APICs in legacy mode so that we can │
│394 * get timer interrupts in second kernel. kexec/kdump │
│395 * paths already have calls to restore_boot_irq_mode() │
│396 * in one form or other. kexec jump path also need one. │
└─────────────────────────────────────────────────────────────────────────────────────────────────────┘
remote Thread 1.1 In: machine_kexec L374 PC: 0xffffffff810635f0
(gdb) b *machine_kexec
Breakpoint 1 at 0xffffffff810635f0: file arch/x86/kernel/machine_kexec_64.c, line 374.
(gdb) c
Continuing.
Breakpoint 1, machine_kexec (image=0xffff888006f96800) at arch/x86/kernel/machine_kexec_64.c:374
(gdb)
Core dumps and kgdb: https://elinux.org/images/f/f0/Bingham.pdf
KVM
https://lwn.net/Articles/658511/
Nested and VMCS
Contributions
In your patches, write all about what you're doing. The kernel doesn't seem to like comments very much. Instead they rely on people writing insanely detailed commit messages describing what they are changing.
Every time you change the kernel you're changing something that has been working fine for someone for X amount of long time. As such, you need to be convincing for why your change should be accepted! Make your coverletter and commit messages very detailed. The kernel is a big place. It may have been a long time since someone reading your patch series has looked at the place you're working. Be courtious and remind them of all the moving pieces involved in what you're doing, and why and how you're changing them.
Etiquette
- You can say
patchorpatchsetin your coverletter. Just not in the commit messages themselves.
KVM
VMX
When a VM enters VMX root mode (aka "I'm gonna run some VMs mode") using
the VMXON instruction, it sets up what's called a VMCS for each VM
/ virtualized processor it wants to run. All VMX related instructions
operate on whatever VMCS we pointed to via the VMPTRLD instruction.
Nested
L0 == The host
L1 == The guest
L2 == The guest of the guest
vmx/nested.c is responsible for emulating VMX for L1. When L1 does a
VMLAUNCH or VMRESUME, enter_guest_mode is called, which means that
is_guest_mode will now return true.
Terminology
Root mode = L0 when not running a virtual machine
Non-root mode = L0 when running a virtual machine L(n)
Assembly
What is jmp 1f?