Booting

Learning Objectives

Understand what is required to boot a system.
Understand what steps are being performed to get into a higher level language, such as C++ or Rust.

Booting is a term that is given to the procedure of doing something after the CPU hands control to you. When a computer is first started, it goes through some sort of setup routine. In consumer computers, such as your laptop, it goes through something called a BIOS (basic input/output system) or EFI (extensible firmware interface).

The BIOS and EFI will run some setup specific to that computer, such as setting voltages, training the DRAM, and so forth. However, after these routines are finished, the firmware will jump to a particular location in RAM. This location better be your software.

In most embedded systems, the boot stages tend to lead to some sort of NAND flash or SD card being loaded into a particular memory address. In RISC-V, this is typically the memory address 0x80000000.

Initialization

The point of our bootloader is to get us into a higher level language, such as C, C++, or even Rust. So, we need to set up all of the things required to run at a higher language.

In most multi-core systems, we have a CPU lottery in which every CPU core runs at a given memory address. It is our job to pick a core and make it the bootstrap core. Intel and AMD CPUs do this for us during BIOS/EFI initialization.

So, now we've selected the core. Now what? Our job is now to set up certain control registers so that we know exactly what the CPU is doing. However, one of the most overlooked things to do is to set up a stack pointer. When we make a call to a C, C++, or Rust function, it will usually need the stack right away. If we don't set up the stack properly, your boot loader will crash as soon as you make the call to your C, C++, or Rust entry point.

Supervisor Binary Interface (OpenSBI)

The operating system generally sits at the supervisor level of a machine. The RISC-V architecture has three modes that we will use: (1) machine mode, (2) supervisor mode, and (3) user mode.

Applications generally sit at the user mode, where many of the privileged instructions and access to control registers is disabled. This leaves our operating system at the supervisor mode. Here, the OS can control all supervisor level control registers (and below). However, the machine mode is what is in charge of the actual hardware itself. We will be able to control this hardware in supervisor mode, but the CPUs cores (called hardware threads or harts in RISC-V) need to be initialized and a method for initializing these cores needs to be available.

This is where the OpenSBI sits. This is essentially a mini-OS that will help us control the machine itself. So, the boot process will install OpenSBI at the entry point, or 0x80000000. This OpenSBI will initialize the cores, set up the control registers and hand off controls. I have pre-built your opensbi binary to handle a payload. This payload is the Universal Bootloader, or u-boot.

OpenSBI is operating at the machine mode, which is the most privileged mode in RISC-V. Our u-boot allows us to load our operating system over and over again without having to reflash the NAND storage. This helps us prevent wearing it out, and it makes it easier to update our kernel and boot it.

For our simulator, OpenSBI will run at address 0x80000000, and then it will hand off control to 0x80020000. This is where our kernel will take control. So, the very first instruction we place under the _start label will be at memory address 0x80020000.

Going to Rust

Since OpenSBI will set up the machine for us, our job is just to get to a high level language, such as C++ or Rust as soon as possible. Since we take control at 0x80020000, we need to make sure that everything that is necessary for the high level language is set up. This usually means that the global pointer is set up as well as the stack pointer.

The global pointer is a register that holds where global variables and constants are stored in memory. Our operating system will have a lot of resident data structures, so we use static memory, which is just a fancy term for global memory. So, having a global pointer is almost a must.

Usually the first thing to take place when a function is invoked is the prologue, where the stack is allocated and certain registers are saved on it. If our stack pointer is invalid, we will immediately crash our operating system.

.section .text.init
.global _start
_start:
.option push
.option norelax
    la      gp, __global_pointer$
    la      t3, _bss_start
    la      t4, _bss_end
.option pop
    # Clear BSS
1:
    bge     t3, t4, 1f
    sd      zero, 0(t3)
    addi    t3, t3, 8
    j       1b
1:
    la      sp, _stack_end
    li      t0, (1 << 8) | (1 << 5) | (1 << 13)
    csrw    sstatus, t0
    la      t1, kinit
    csrw    sie, zero
    csrw    sepc, t1
    sret
4:
    wfi
    j       4b

You will see that we start in section .text.init. This is important because we need to make sure that the linker doesn't place what should be at memory location 0x80020000 somewhere else. Our entry point is _start, where we first set up the global pointer, gp. The symbol __global_pointer$ comes from the linker script which is set to the top of the global memory section. Here's that portion in our linker script.

.text : {
    PROVIDE(_text_start = .);
    *(.text.init) *(.text .text.*)
    PROVIDE(_text_end = .);
  } >ram AT>ram :text

  . = ALIGN(8);
  PROVIDE(__global_pointer$ = .);

.rodata : {
    PROVIDE(_rodata_start = .);
    *(.rodata .rodata.*)
    PROVIDE(_rodata_end = .);
  } >ram AT>ram :text

You can see our text section (cpu instructions) starts at the top of the memory field. Inside here, you can see that the section .text.init comes first, followed by .text and finally .text.*. The wildcard means anything that matches .text.something. Our global pointer is then pointing to the top of the .rodata (read-only) section, where global constants, including string literals will be stored.

When we're done storing cpu instructions and globals, we will have the rest of the memory to work with. In this truncated view of memory, we set the bottom of it to the heap and the top of it to the stack as you can see in the linker script.

  PROVIDE(_memory_start = ORIGIN(ram));
  PROVIDE(_stack_start = _bss_end);
  PROVIDE(_stack_end = _stack_start + 16K);
  PROVIDE(_heap_start = _stack_end);
  PROVIDE(_heap_size = _memory_end - _heap_start);

Notice that when we boot, we invoke la sp, _stack_end, which is the bottom of the stack allocation. Recall that the stack grows from the end to the start. This is why we set the stack pointer to the bottom of the stack.

You will notice that we are setting the register zero into each portion of the BSS section. Recall that the BSS section stores global, uninitialized values. High-level languages, such as Rust and C++ require that this memory section be initialized to 0. In fact, if I do something like int i = 0; where i is a global, the compiler will put int i in the BSS section.

You can see at the end that we load the memory address of kinit, which is the entry point to the high-level language, shown below. We put this into the sepc (Supervisor Exception Program Counter) register. When we execute sret, the status register updates, and it jumps to the memory address in sepc, which is our kinit function.

#[no_mangle]
extern "C" fn kinit(arg0: usize) {
}

The Rust function above is not mangled, so we can call it directly by its name (kinit). We also say that the function needs to follow "C calling conventions", which is what extern "C" is all about.

The sstatus register is what we use to enable interrupts, enable the floating point unit, and change processor modes. In our case, we're setting bits 5, 8, and 13, which are the interrupt enable bit (5), the supervisor mode bit (8), and the floating-point enable bit (13). We will look at the sstatus register more in depth to see what is happening here. However, the sstatus register will take our updates after the sret instruction is invoked.

High Level Language

Now that we're in a high-level language, we will only have to touch assembly for very limited reasons. A few of those reasons include trap handlers, switching modes, and starting a process. Furthermore, the OpenSBI requires us to use system calls at the supervisor level, which will require us to put values in proper registers before invoking the ecall instruction. Luckily, we can use the inline assembly in Rust to do this. I will cover this in a different lecture.

For now, welcome to high-level language! There are still a few things you must consider now that we're in a baremetal environment. In Rust, you don't have access to the standard library, hence the #![no_std], but luckily there is a core library we can use and alloc library we can use for collections and strings.