💻 Module 01: OS Basics & Core Architecture

Focus: What an OS is, Hardware/Kernel abstraction, Dual-Mode CPU Operation, System Calls under the hood, Interrupts & Traps, and the Boot Sequence.

📑 Table of Contents

1. What is an Operating System?
2. Types & Evolution of Operating Systems
3. Kernel Architectures: Monolithic vs Microkernel
4. Dual-Mode CPU Operation & Hardware Protection
5. System Calls Under the Hood
6. Interrupts, Traps, and Exceptions
7. The Booting Sequence (From Power Button to systemd)
8. Common Mistakes & Gotchas
9. Real-World Engineering Deep Dive (strace & Security Rings)
10. Practice Problems with Solutions
11. High-Yield Interview Questions & Answers

1. What is an Operating System?

An Operating System (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs. It acts as an intermediary between the user/application and the raw bare-metal hardware.

The Two Primary Roles of an OS:

1. Resource Allocator: Manages hardware resources (CPU cores, physical memory frames, disk blocks, network bandwidth) fairly and efficiently among competing processes.
2. Control Program: Governs execution of user programs to prevent errors, hardware misuse, security breaches, and resource starvation.

Real-World Analogy:

Think of an OS as a Government or Airport Air Traffic Control.

• It doesn't fly the planes (applications do the work).
• It ensures runways (CPUs) and gates (RAM) are allocated orderly so planes don't crash into each other (isolation and memory protection).

2. Types & Evolution of Operating Systems

1. Batch Operating Systems

• Mechanism: Jobs with similar needs were batched together and fed via punch cards/magnetic tape. The operator ran them without interactive user intervention.
• Limitation: CPU was idle for 80-90% of the time waiting for slow mechanical I/O devices (card readers, printers).

2. Multiprogrammed Systems

• Mechanism: Keeps multiple jobs in memory simultaneously. When Job A issues an I/O request, the OS switches the CPU to execute Job B instead of staying idle.
• Key Metric: Maximizes CPU Utilization.
• Limitation: Non-interactive; user could not interact with running jobs.

3. Time-Sharing (Multitasking) Systems

• Mechanism: Logical extension of multiprogramming. Uses a hardware timer to preempt the CPU rapidly between processes (e.g., every 10–50 ms), giving users the illusion that all programs run concurrently.
• Key Metric: Minimizes Response Time.

4. Distributed Operating Systems

• Mechanism: Manages a collection of independent networked computers, presenting them to the user as a single coherent system (e.g., Kubernetes, Apache Mesos concepts).

5. Real-Time Operating Systems (RTOS)

• Mechanism: Used when strict time constraints exist between an event and its processing.
• Hard RTOS: Missing a deadline causes total system failure or catastrophe (e.g., pacemakers, anti-lock brakes, missile guidance).
• Soft RTOS: Missing a deadline degrades Quality of Service (QoS) but doesn't destroy the system (e.g., video streaming, audio playback, online gaming).

3. Kernel Architectures: Monolithic vs Microkernel

The Kernel is the core component of an OS that always resides in memory and executes with highest hardware privileges.

4. Dual-Mode CPU Operation & Hardware Protection

To prevent user applications from corrupting memory, crashing the CPU, or bypassing security, modern CPU hardware enforces Dual-Mode Execution via a physical bit in the CPU's status register (e.g., CR0 / EFLAGS on x86).

x86 Privilege Rings

x86 CPUs support 4 rings of privilege (0 to 3), although modern OSes (Linux, Windows) primarily utilize Ring 0 (Supervisor/Kernel) and Ring 3 (User):

Privileged vs Non-Privileged Instructions

• Privileged Instructions (Only allowed when Mode Bit = 0):
  • Direct hardware I/O commands (in, out, cli, sti).
  • Modifying page table registers (mov %eax, %cr3).
  • Setting the CPU interrupt descriptor table (lidt).
  • Halting the CPU (hlt).
• What happens if User Mode executes a Privileged Instruction?
  • The CPU hardware immediately intercepts it, generates a General Protection Fault (#GP) exception, flips into Kernel Mode, and kills the offending process with SIGSEGV or Illegal Instruction.

5. System Calls Under the Hood

A System Call (Syscall) is the programmatic gateway through which a user-space application requests a service from the operating system kernel.

The Life of a Syscall (write() to stdout)

When you call printf("Hello\n") in C:

1. printf() formats the string and calls the C Standard Library POSIX wrapper write(1, "Hello\n", 6).
2. The write() wrapper loads:
   • Syscall number into a CPU register (e.g., RAX = 1 for sys_write on x86_64 Linux).
   • Arguments into registers: RDI = 1 (stdout), RSI = &buffer, RDX = 6 (length).
3. The wrapper triggers the special CPU instruction: syscall (or sysenter / int 0x80).
4. Hardware Transition:
   • Hardware flips the Mode Bit from 1 (User) to 0 (Kernel).
   • CPU switches to the kernel stack.
   • Saves User %RIP (Program Counter) and %RSP (Stack Pointer) into hardware registers (%RCX and %R11).
   • Jumps to the kernel's system call entry handler address stored in the Model Specific Register (MSR_LSTAR).
5. Kernel Dispatch:
   • The kernel looks up RAX in the sys_call_table[].
   • Invokes sys_write(fd, buf, count).
   • The kernel interacts with the console device driver.
6. Return:
   • Kernel places the return value (e.g., number of bytes written) into RAX.
   • Kernel executes sysret (or iret).
   • CPU restores user registers, flips Mode Bit back to 1 (User), and returns execution back to the user program.

Essential System Calls Categorization

6. Interrupts, Traps, and Exceptions

Interrupt Vector Table (IVT) / Interrupt Descriptor Table (IDT)

The IDT is an array of memory pointers initialized at boot time. Each index (vector number 0–255) contains the address of the corresponding Interrupt Service Routine (ISR):

• Vector 0: Division by Zero (#DE)
• Vector 14: Page Fault (#PF)
• Vector 32: Programmable Interval Timer Tick (IRQ 0)
• Vector 128 (0x80): Legacy Linux 32-bit System Call Trap

I/O Communication Models: Polling vs Interrupts vs DMA

7. The Booting Sequence (From Power Button to systemd)

Step-by-Step Breakdown:

1. Power-On Reset:
   • Power supply stabilizes. The motherboard asserts the POWER_GOOD signal.
   • CPU initializes in 16-bit Real Mode; Program Counter (%CS:%IP) is hardwired to 0xFFFF0 (Reset Vector).
2. BIOS / UEFI:
   • Executes POST (Power-On Self-Test) to verify RAM, keyboard, graphics, and storage devices.
   • Selects boot device according to boot order configured in CMOS NVRAM.
3. Bootloader (e.g., GRUB2, systemd-boot):
   • Reads the Master Boot Record (MBR / Partition Table) or the EFI System Partition (ESP).
   • Switches the CPU from 16-bit Real Mode to 32-bit Protected Mode, then to 64-bit Long Mode.
   • Loads the compressed Linux kernel image (vmlinuz) and initial RAM disk (initramfs) into memory.
4. Kernel Initialization:
   • Decompresses itself in RAM.
   • Initializes CPU control registers, Page Tables (MMU), IDT, memory allocators (Buddy/Slab), device drivers, and mounts root filesystem (/).
5. User-Space Init:
   • Spawns the very first user-space process: PID 1 (/sbin/init or systemd).
   • Spawns background daemons, network services, login managers (sshd, getty, desktop UI).

8. Common Mistakes & Gotchas

[!CAUTION]

1. Thinking printf() is a System Call

printf() is a user-space C standard library function that buffers formatted output. It only invokes the actual write() syscall when the buffer flushes (e.g., on \n for line-buffered stdout, or when the buffer fills up).

[!WARNING]

2. Assuming System Calls are "Free"

System calls are significantly more expensive than regular function calls:

• Normal function call: ~2-5 CPU clock cycles (push args, call, ret).
• System call: ~50-300+ CPU clock cycles due to register flushing, kernel stack switch, TLB invalidations/mitigations (KPTI for Meltdown), and security checks.

[!NOTE]

3. Confusing Interrupts with Polling

Polling is synchronous and driven by CPU software loops. Interrupts are asynchronous electrical signals triggered by hardware pins.

9. Real-World Engineering Deep Dive (strace & Security Rings)

1. Tracing Syscalls with strace

To see every single kernel interaction of a program:

Sample Output:

2. How Web Browsers (Chrome Sandbox) Use Syscall Filtering

Chrome tabs run untrusted JavaScript from random websites. To prevent compromised renderer processes from accessing the filesystem or webcam, Chrome uses Linux seccomp-bpf (Secure Computing Mode):

• At startup, the renderer tells the kernel: "Block all syscalls except read, write, futex, and exit".
• If malware attempts open("/etc/passwd"), the kernel immediately sends SIGSYS and kills the process.

10. Practice Problems with Solutions

Problem 1: Syscall Overhead Calculation

Question: A server application performs $1,000,000$ individual 1-byte write() system calls. Each system call takes $200\text{ ns}$ of kernel transition overhead. Alternatively, the application buffers the data and executes a single write() call for the entire $1\text{ MB}$ payload.

1. What is the total time spent in syscall overhead for the 1-byte writes?
2. What is the speedup factor for the buffered approach in terms of syscall overhead?

Solution:

1. Individual writes time $= 1,000,000 \times 200\text{ ns} = 200,000,000\text{ ns} = 0.2\text{ seconds } (200\text{ ms})$.
2. Buffered write time $= 1 \times 200\text{ ns} = 0.0002\text{ ms}$.
3. Speedup factor $= \frac{200\text{ ms}}{0.0002\text{ ms}} = 1,000,000\times$ reduction in transition overhead. (This is why standard I/O libraries use user-space buffering like BufferedOutputStream in Java or FILE* in C).

Problem 2: Mode Switch Identification

Question: For each of the following events, indicate whether it involves a mode switch (User $\rightarrow$ Kernel $\rightarrow$ User):

1. Computing a fibonacci number in a local loop.
2. Allocating memory with malloc(32) when heap space is already available in the runtime arena.
3. Reading data from an open network socket using recv().
4. Handling a timer tick interrupt to preempt a running process.
5. Accessing an unmapped memory address (causing a Page Fault).

Solution:

1. No — Pure user-space CPU computation.
2. No — malloc manages its own user-space free list; only calls brk/mmap when the arena runs out.
3. Yes — Involves sys_recvfrom syscall to read from OS socket buffer.
4. Yes — Hardware timer generates interrupt, vectoring CPU to kernel scheduler ISR.
5. Yes — CPU hardware generates Exception vector 14 (#PF), switching to kernel page fault handler.

11. High-Yield Interview Questions & Answers

Q1: What is the exact difference between a Trap, an Interrupt, and an Exception?

Answer:

• Interrupt (Hardware Interrupt): An asynchronous signal generated by an external physical hardware device (keyboard, network card, disk controller, timer). It is independent of the currently executing CPU instruction.
• Trap (Software Interrupt): A synchronous, intentional software instruction triggered by the user program to invoke a system call (e.g., syscall, int 0x80, trap).
• Exception (Fault / Abort): A synchronous, unintentional error condition caused directly by the executing instruction (e.g., division by zero, invalid opcode, page fault, stack overflow).

Q2: Why does the CPU need a hardware mode bit? Why can't the OS just be written in pure software?

Answer: Without a hardware-enforced mode bit, any user application could overwrite the operating system code in RAM, disable interrupts, or read private memory belonging to other processes. The hardware mode bit physically disables privileged machine instructions (like cli, in/out, mov %cr3) at the transistor/microcode level when the mode bit is 1 (User Mode).

Q3: What is the difference between a Monolithic kernel and a Microkernel? Which one is faster and why?

Answer:

• In a Monolithic kernel (Linux, Windows NT core), all OS services (scheduling, virtual memory, file systems, device drivers, network stacks) run together inside Ring 0 in a single shared address space.
• In a Microkernel (Mach, QNX, seL4), only minimal primitives (IPC, low-level scheduling, memory mapping) run in Ring 0. Everything else (file systems, device drivers) runs as isolated user-space processes.
• Speed: Monolithic kernels are substantially faster because OS components communicate via standard in-memory C function calls. Microkernels require expensive inter-process communication (IPC) and multiple user-to-kernel mode context switches for basic operations like reading a file block.

Q4: What are the steps performed during a System Call transition?

Answer:

1. User application places the syscall number in %rax and parameters in registers (%rdi, %rsi, %rdx).
2. Program triggers syscall instruction.
3. Hardware saves %rip and %rflags, flips mode bit from 1 (User) to 0 (Kernel), and switches to the kernel stack.
4. CPU jumps to the address in MSR_LSTAR (kernel entry point).
5. Kernel saves all general-purpose registers onto the kernel stack.
6. Kernel validates arguments and indexes sys_call_table[%rax] to execute the handler.
7. Return code is placed in %rax, user registers are restored from the kernel stack, and sysret instruction flips mode bit back to 1 and resumes user execution.

Q5: What is DMA and why is it crucial for high-performance operating systems?

Answer: Direct Memory Access (DMA) is a specialized hardware mechanism that allows I/O devices (like NVMe SSDs or 100GbE NICs) to transfer data blocks directly to and from RAM across the system bus without passing every byte through the CPU registers. Without DMA, the CPU would be trapped in programmed I/O loops copying bytes one by one. With DMA, the CPU simply issues a command descriptor to the DMA controller and is free to execute other computation until a single interrupt arrives upon transfer completion.

Proceed to Module 02: Process Management & IPC!