03-nodejs.md

Module 03: Node.js Deep Dive

Goal: Understand how Node.js works under the hood and master its core modules. Time: 3-4 days of focused study Prerequisites: Module 01 & 02 (JavaScript Fundamentals + Advanced)

Table of Contents

  1. What is Node.js?
  2. Node.js Architecture — V8 + libuv
  3. The Node.js Event Loop (In Detail)
  4. Core Modules Overview
  5. The fs Module — File System
  6. The path Module
  7. The events Module — EventEmitter
  8. Streams — The Power Feature
  9. Buffers — Binary Data
  10. The http Module — Building Servers
  11. The crypto Module
  12. NPM — Package Management
  13. Environment Variables & Configuration
  14. Child Processes & Worker Threads
  15. Clustering — Multi-Core
  16. Debugging Node.js
  17. Practice Problems
  18. Interview Questions

1. What is Node.js?

Node.js is a JavaScript runtime built on Chrome's V8 engine. It lets you run JavaScript outside the browser — on servers, command-line tools, IoT devices, etc.

Key Characteristics

┌──────────────────────────────────────────────────────────────┐ │ Node.js │ ├──────────────────────────────────────────────────────────────┤ │ ✅ Single-threaded (main event loop) │ │ ✅ Non-blocking I/O (asynchronous by default) │ │ ✅ Event-driven architecture │ │ ✅ Cross-platform (Windows, macOS, Linux) │ │ ✅ Huge ecosystem (npm — largest package registry) │ │ ✅ Great for I/O-intensive tasks (APIs, real-time apps) │ │ ❌ Not ideal for CPU-intensive tasks (unless using workers) │ └──────────────────────────────────────────────────────────────┘

What Node.js is NOT

  • Not a language — JavaScript is the language, Node.js is the runtime
  • Not a framework — Express, Fastify, Koa are frameworks built ON Node.js
  • Not multi-threaded — The main event loop is single-threaded (but it CAN use threads)

2. Node.js Architecture — V8 + libuv

┌─────────────────────────────────────────────────────────────────┐ │ Your JavaScript Code │ ├─────────────────────────────────────────────────────────────────┤ │ Node.js Bindings (C++) │ │ (Bridge between JS and C++ world) │ ├─────────────────┬───────────────────────────────────────────────┤ │ V8 Engine │ libuv │ │ (by Google) │ (by Node.js team) │ │ │ │ │ • Parses JS │ • Event loop │ │ • Compiles to │ • Thread pool (4 threads by default) │ │ machine code │ • Async I/O (file, network, DNS) │ │ • Executes │ • Timers │ │ • GC │ • Child processes │ │ │ • Cross-platform abstraction │ ├─────────────────┴───────────────────────────────────────────────┤ │ Operating System │ │ (file system, network, threads, etc.) │ └─────────────────────────────────────────────────────────────────┘

V8 Engine

  • Parses JavaScript into an Abstract Syntax Tree (AST)
  • Compiles to machine code (JIT — Just-In-Time compilation)
  • Executes the machine code
  • Manages memory (heap allocation, garbage collection)

libuv

  • Provides the event loop
  • Manages a thread pool (default 4 threads) for heavy I/O operations
  • Handles async file system operations, DNS lookups, and some crypto
  • Provides cross-platform support (Windows IOCP, Linux epoll, macOS kqueue)

What Uses the Thread Pool?

Thread Pool (libuv): │ OS-level Async (no thread pool): • fs operations │ • Network I/O (TCP, UDP, HTTP) • DNS lookups │ • Pipes • Crypto (pbkdf2, scrypt) │ • Signals • Zlib (compression) │ • Child processes • Some C++ addons │
// You can increase the thread pool size: process.env.UV_THREADPOOL_SIZE = 8; // Must be set before any I/O // Default: 4, Max: 1024 // This matters when you have many concurrent file operations: // With 4 threads and 100 file reads, only 4 run simultaneously

3. The Node.js Event Loop (In Detail)

We covered the general event loop in Module 02. Here's the Node.js-specific detail:

┌───────────────────────────┐ ┌─►│ timers │ ← setTimeout, setInterval │ └─────────────┬─────────────┘ │ ┌─────────────▼─────────────┐ │ │ pending callbacks │ ← Some I/O callbacks deferred from previous loop │ └─────────────┬─────────────┘ │ ┌─────────────▼─────────────┐ │ │ idle, prepare │ ← Internal use only │ └─────────────┬─────────────┘ │ ┌─────────────▼─────────────┐ ┌───────────────┐ │ │ poll │◄─────│ incoming │ │ │ (wait for I/O events) │ │ connections, │ │ └─────────────┬─────────────┘ │ data, etc. │ │ ┌─────────────▼─────────────┐ └───────────────┘ │ │ check │ ← setImmediate │ └─────────────┬─────────────┘ │ ┌─────────────▼─────────────┐ │ │ close callbacks │ ← socket.on('close'), etc. │ └─────────────┬─────────────┘ └────────────────┘ Between EVERY phase transition: 1. process.nextTick() callbacks (highest priority) 2. Promise microtask callbacks

The Key Insight — Why Node.js Is Fast

// Node.js doesn't create a thread for each request (unlike Apache/PHP) // Instead, it uses ONE thread for all requests via the event loop // Traditional server (Apache + PHP): // Request 1 → Thread 1 (blocks while reading DB) → Response 1 // Request 2 → Thread 2 (blocks while reading DB) → Response 2 // Request 3 → Thread 3 (blocks while reading DB) → Response 3 // 10,000 requests = 10,000 threads = HUGE memory usage // Node.js: // Request 1 → Start DB read (async) → Move to next request // Request 2 → Start DB read (async) → Move to next request // Request 3 → Start DB read (async) → Move to next request // DB read 1 completes → Callback processes Response 1 // DB read 3 completes → Callback processes Response 3 // DB read 2 completes → Callback processes Response 2 // 10,000 requests = 1 thread = LOW memory usage!

4. Core Modules Overview

// Node.js comes with many built-in modules (no npm install needed) const fs = require('fs'); // File system const path = require('path'); // Path utilities const http = require('http'); // HTTP server/client const https = require('https'); // HTTPS const events = require('events'); // EventEmitter const stream = require('stream'); // Streams const crypto = require('crypto'); // Cryptography const os = require('os'); // Operating system info const url = require('url'); // URL parsing const util = require('util'); // Utility functions const child_process = require('child_process'); // Run external commands const cluster = require('cluster');// Multi-process const worker_threads = require('worker_threads'); // Multi-thread const readline = require('readline'); // CLI input const zlib = require('zlib'); // Compression (gzip, deflate) const dns = require('dns'); // DNS lookups const net = require('net'); // TCP/IPC networking const querystring = require('querystring'); // Parse URL query strings const assert = require('assert'); // Testing assertions

5. The fs Module — File System

Synchronous vs Asynchronous vs Promises

const fs = require('fs'); const fsPromises = require('fs').promises; // Or: require('fs/promises') // 1. Synchronous (BLOCKS the event loop — avoid in servers!) try { const data = fs.readFileSync('file.txt', 'utf8'); console.log(data); } catch (err) { console.error(err); } // 2. Asynchronous with Callbacks fs.readFile('file.txt', 'utf8', (err, data) => { if (err) { console.error(err); return; } console.log(data); }); // 3. Promise-based (RECOMMENDED for modern code) async function readFile() { try { const data = await fsPromises.readFile('file.txt', 'utf8'); console.log(data); } catch (err) { console.error(err); } }

Essential File Operations

const fs = require('fs/promises'); // ---- READING ---- // Read entire file const content = await fs.readFile('data.json', 'utf8'); const parsed = JSON.parse(content); // Read file as buffer (binary) const buffer = await fs.readFile('image.png'); // No encoding = Buffer // ---- WRITING ---- // Write file (creates or OVERWRITES) await fs.writeFile('output.txt', 'Hello, World!', 'utf8'); // Write JSON await fs.writeFile('config.json', JSON.stringify(data, null, 2)); // Append to file await fs.appendFile('log.txt', `${new Date().toISOString()} - Event occurred\n`); // ---- DIRECTORIES ---- // Create directory await fs.mkdir('my-folder'); // Single level await fs.mkdir('path/to/deep/folder', { recursive: true }); // Nested // Read directory contents const files = await fs.readdir('.'); console.log(files); // ['file1.txt', 'file2.js', 'folder1'] // Read with file types const entries = await fs.readdir('.', { withFileTypes: true }); for (const entry of entries) { console.log(`${entry.name} - ${entry.isDirectory() ? 'dir' : 'file'}`); } // ---- FILE INFO ---- const stats = await fs.stat('file.txt'); console.log(stats.isFile()); // true console.log(stats.isDirectory()); // false console.log(stats.size); // Size in bytes console.log(stats.mtime); // Last modified time console.log(stats.birthtime); // Created time // ---- CHECK IF EXISTS ---- try { await fs.access('file.txt'); console.log('File exists'); } catch { console.log('File does not exist'); } // ---- DELETE ---- await fs.unlink('file.txt'); // Delete file await fs.rmdir('empty-folder'); // Delete empty directory await fs.rm('folder', { recursive: true, force: true }); // Delete recursively // ---- RENAME / MOVE ---- await fs.rename('old.txt', 'new.txt'); await fs.rename('file.txt', 'subfolder/file.txt'); // Move // ---- COPY ---- await fs.copyFile('source.txt', 'dest.txt'); // ---- WATCH for changes ---- const watcher = fs.watch('.', { recursive: true }); for await (const event of watcher) { console.log(`${event.eventType}: ${event.filename}`); }

Real-World: File-based Logger

const fs = require('fs'); const path = require('path'); class FileLogger { constructor(logDir = './logs') { this.logDir = logDir; if (!fs.existsSync(logDir)) { fs.mkdirSync(logDir, { recursive: true }); } } #getLogFile() { const date = new Date().toISOString().split('T')[0]; // 2024-01-15 return path.join(this.logDir, `${date}.log`); } #formatMessage(level, message, meta = {}) { return JSON.stringify({ timestamp: new Date().toISOString(), level, message, ...meta }) + '\n'; } info(message, meta) { fs.appendFileSync(this.#getLogFile(), this.#formatMessage('INFO', message, meta)); } error(message, meta) { fs.appendFileSync(this.#getLogFile(), this.#formatMessage('ERROR', message, meta)); } warn(message, meta) { fs.appendFileSync(this.#getLogFile(), this.#formatMessage('WARN', message, meta)); } } const logger = new FileLogger(); logger.info('Server started', { port: 3000 }); logger.error('Database connection failed', { host: 'localhost', retry: 3 });

6. The path Module

const path = require('path'); // ---- JOINING PATHS (handles OS separators) ---- path.join('/users', 'alice', 'documents', 'file.txt'); // Windows: '\users\alice\documents\file.txt' // Linux: '/users/alice/documents/file.txt' path.join(__dirname, '..', 'config', 'app.json'); // Resolves relative paths: /project/src/../config/app.json → /project/config/app.json // ---- RESOLVING ABSOLUTE PATHS ---- path.resolve('file.txt'); // Returns absolute path: /current/working/directory/file.txt path.resolve('/foo', 'bar', 'baz'); // /foo/bar/baz // ---- PARSING PATHS ---- const parsed = path.parse('/home/user/docs/report.pdf'); // { // root: '/', // dir: '/home/user/docs', // base: 'report.pdf', // ext: '.pdf', // name: 'report' // } // ---- EXTRACTING PARTS ---- path.basename('/home/user/file.txt'); // 'file.txt' path.basename('/home/user/file.txt', '.txt'); // 'file' (without extension) path.dirname('/home/user/file.txt'); // '/home/user' path.extname('/home/user/file.txt'); // '.txt' // ---- IMPORTANT GLOBALS ---- __dirname // Directory of the current file (absolute path) __filename // Full path of the current file // In ES Modules (no __dirname): import { fileURLToPath } from 'url'; import { dirname } from 'path'; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename);

7. The events Module — EventEmitter

The EventEmitter is the backbone of Node.js. Many built-in modules (HTTP, Streams, etc.) extend it.

const EventEmitter = require('events'); // ---- BASIC USAGE ---- const emitter = new EventEmitter(); // Register a listener emitter.on('order:created', (order) => { console.log(`Order ${order.id} created! Total: $${order.total}`); }); // Register one-time listener emitter.once('server:ready', () => { console.log('Server is ready (this only fires once)'); }); // Emit events emitter.emit('order:created', { id: 1, total: 99.99 }); emitter.emit('server:ready'); emitter.emit('server:ready'); // Nothing — once listener is removed // ---- REAL-WORLD: Event-Driven Architecture ---- class OrderService extends EventEmitter { async createOrder(data) { // Validate and save order const order = { id: Date.now(), ...data, status: 'created' }; // Emit event — other services react to it this.emit('order:created', order); return order; } async cancelOrder(orderId) { // Update order status this.emit('order:cancelled', { orderId, cancelledAt: new Date() }); } } const orderService = new OrderService(); // Different services react to order events independently orderService.on('order:created', (order) => { console.log(`[Email] Sending confirmation for order ${order.id}`); }); orderService.on('order:created', (order) => { console.log(`[Inventory] Reserving items for order ${order.id}`); }); orderService.on('order:created', (order) => { console.log(`[Analytics] Tracking order ${order.id}`); }); orderService.on('order:cancelled', ({ orderId }) => { console.log(`[Inventory] Releasing items for order ${orderId}`); }); // Create an order — all listeners fire orderService.createOrder({ item: 'Laptop', total: 999 }); // [Email] Sending confirmation for order 1705... // [Inventory] Reserving items for order 1705... // [Analytics] Tracking order 1705... // ---- USEFUL METHODS ---- emitter.listenerCount('order:created'); // Number of listeners for event emitter.eventNames(); // Array of event names with listeners emitter.removeAllListeners('event'); // Remove all listeners for event emitter.setMaxListeners(20); // Default is 10, warns if exceeded // Error handling — ALWAYS add an 'error' listener // If no 'error' listener exists, Node.js CRASHES on emit('error') emitter.on('error', (err) => { console.error('Something went wrong:', err.message); }); emitter.emit('error', new Error('Database connection lost'));

8. Streams — The Power Feature

Streams process data piece by piece instead of loading everything into memory. Essential for handling large files, network data, or any large dataset.

Why Streams Matter

// WITHOUT streams — loads ENTIRE file into memory const fs = require('fs'); // If file is 2GB, you need 2GB of RAM! const data = fs.readFileSync('huge-file.csv', 'utf8'); processData(data); // ❌ Out of memory for large files // WITH streams — processes data in chunks const readStream = fs.createReadStream('huge-file.csv', 'utf8'); readStream.on('data', (chunk) => { // chunk is a small piece (default: 64KB) processData(chunk); // ✅ Only 64KB in memory at a time }); readStream.on('end', () => { console.log('Done processing'); });

Four Types of Streams

┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Readable │ │ Writable │ │ Duplex │ │ Transform │ │ │ │ │ │ │ │ │ │ Source of │ │ Destination │ │ Both read │ │ Duplex that │ │ data │ │ for data │ │ and write │ │ transforms │ │ │ │ │ │ │ │ data │ │ Examples: │ │ Examples: │ │ Examples: │ │ Examples: │ │ • fs.read │ │ • fs.write │ │ • TCP socket │ │ • zlib │ │ • http req │ │ • http res │ │ • WebSocket │ │ • crypto │ │ • process. │ │ • process. │ │ │ │ • custom │ │ stdin │ │ stdout │ │ │ │ parsers │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘

Readable Streams

const fs = require('fs'); const readable = fs.createReadStream('large-file.txt', { encoding: 'utf8', highWaterMark: 1024 // Chunk size in bytes (default: 64KB) }); // Event-based (flowing mode) readable.on('data', (chunk) => { console.log(`Received ${chunk.length} bytes`); }); readable.on('end', () => { console.log('No more data'); }); readable.on('error', (err) => { console.error('Error:', err); }); // Paused mode (manual reading) readable.on('readable', () => { let chunk; while ((chunk = readable.read()) !== null) { console.log(`Read ${chunk.length} bytes`); } });

Writable Streams

const fs = require('fs'); const writable = fs.createWriteStream('output.txt'); // Write data writable.write('Hello, '); writable.write('World!\n'); writable.write('Another line\n'); // Signal that we're done writing writable.end('Final line\n'); // Optionally write last chunk writable.on('finish', () => { console.log('All data written'); }); // Handling backpressure // write() returns false when the internal buffer is full // You should stop writing and wait for 'drain' event const bigWritable = fs.createWriteStream('big-output.txt'); function writeData(writable, data) { let i = 0; function write() { let ok = true; while (i < data.length && ok) { ok = writable.write(data[i]); i++; } if (i < data.length) { // Buffer is full — wait for drain writable.once('drain', write); } else { writable.end(); } } write(); }

Piping — The Most Powerful Pattern

const fs = require('fs'); const zlib = require('zlib'); // Pipe: connect readable → writable // Data flows automatically with backpressure handling // Copy a file (efficient) fs.createReadStream('input.txt') .pipe(fs.createWriteStream('output.txt')); // Compress a file fs.createReadStream('large-file.txt') .pipe(zlib.createGzip()) .pipe(fs.createWriteStream('large-file.txt.gz')); // Decompress a file fs.createReadStream('large-file.txt.gz') .pipe(zlib.createGunzip()) .pipe(fs.createWriteStream('large-file-restored.txt')); // Chain multiple transforms fs.createReadStream('data.csv') .pipe(csvParser()) // Parse CSV .pipe(transformData()) // Transform .pipe(jsonStringify()) // Convert to JSON .pipe(fs.createWriteStream('output.json')); // Modern way: pipeline (handles errors properly!) const { pipeline } = require('stream/promises'); await pipeline( fs.createReadStream('input.txt'), zlib.createGzip(), fs.createWriteStream('input.txt.gz') ); console.log('Pipeline complete'); // If any stream errors, pipeline rejects and cleans up

Custom Transform Stream

const { Transform } = require('stream'); // Transform stream that converts text to uppercase class UpperCaseTransform extends Transform { _transform(chunk, encoding, callback) { // Push the transformed data this.push(chunk.toString().toUpperCase()); callback(); // Signal that we're done with this chunk } } // Usage: process.stdin .pipe(new UpperCaseTransform()) .pipe(process.stdout); // Type "hello" → outputs "HELLO" // Real-world: CSV line counter class LineCounter extends Transform { constructor() { super({ objectMode: true }); // Emit objects instead of buffers this.count = 0; this.buffer = ''; } _transform(chunk, encoding, callback) { this.buffer += chunk.toString(); const lines = this.buffer.split('\n'); this.buffer = lines.pop(); // Keep incomplete line for (const line of lines) { this.count++; this.push({ lineNumber: this.count, content: line }); } callback(); } _flush(callback) { if (this.buffer) { this.count++; this.push({ lineNumber: this.count, content: this.buffer }); } callback(); } }

9. Buffers — Binary Data

Buffers represent fixed-length sequences of bytes. They're used when dealing with binary data (files, network packets, images, etc.).

// ---- CREATING BUFFERS ---- const buf1 = Buffer.alloc(10); // 10 bytes, filled with zeros const buf2 = Buffer.alloc(10, 0xFF); // 10 bytes, filled with 0xFF const buf3 = Buffer.from('Hello'); // From string (UTF-8) const buf4 = Buffer.from([72, 101, 108, 108, 111]); // From array of bytes const buf5 = Buffer.from('48656c6c6f', 'hex'); // From hex string // ---- READING BUFFERS ---- const buf = Buffer.from('Hello, World!'); buf.toString('utf8'); // "Hello, World!" buf.toString('hex'); // "48656c6c6f2c20576f726c6421" buf.toString('base64'); // "SGVsbG8sIFdvcmxkIQ==" buf.length; // 13 (bytes, not characters!) buf[0]; // 72 (ASCII for 'H') // ⚠️ String length vs Buffer length: const emoji = '👋'; emoji.length; // 2 (JavaScript string length) Buffer.from(emoji).length; // 4 (actual bytes in UTF-8) // ---- MANIPULATING BUFFERS ---- const buf6 = Buffer.alloc(5); buf6.write('Hi'); // Write string to buffer buf6.fill(0); // Fill with zeros // Copy const source = Buffer.from('Hello'); const target = Buffer.alloc(5); source.copy(target); // Copy source → target // Concat const combined = Buffer.concat([Buffer.from('Hello'), Buffer.from(' World')]); combined.toString(); // "Hello World" // Compare Buffer.compare(Buffer.from('abc'), Buffer.from('def')); // -1 (abc < def) buf.equals(Buffer.from('Hello, World!')); // true // Slice (returns a VIEW, not a copy!) const slice = buf.subarray(0, 5); slice.toString(); // "Hello" // ⚠️ Modifying slice modifies original buffer too!

When You'll Encounter Buffers

// 1. Reading files without encoding const fs = require('fs'); const buffer = fs.readFileSync('image.png'); // Returns Buffer console.log(buffer); // <Buffer 89 50 4e 47 0d 0a 1a 0a ...> // 2. Network data const http = require('http'); http.createServer((req, res) => { const chunks = []; req.on('data', (chunk) => chunks.push(chunk)); // chunk is Buffer req.on('end', () => { const body = Buffer.concat(chunks).toString(); console.log('Body:', body); }); }); // 3. Crypto operations const crypto = require('crypto'); const randomBytes = crypto.randomBytes(32); // Returns Buffer console.log(randomBytes.toString('hex')); // 4. Base64 encoding/decoding const encoded = Buffer.from('Hello, World!').toString('base64'); // "SGVsbG8sIFdvcmxkIQ==" const decoded = Buffer.from(encoded, 'base64').toString('utf8'); // "Hello, World!"

10. The http Module — Building Servers

Basic HTTP Server

const http = require('http'); const server = http.createServer((req, res) => { // req = IncomingMessage (readable stream) // res = ServerResponse (writable stream) console.log(`${req.method} ${req.url}`); console.log('Headers:', req.headers); // Set response headers res.setHeader('Content-Type', 'text/plain'); res.setHeader('X-Custom-Header', 'Hello'); // OR set status code and headers together: res.writeHead(200, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store' }); // Send response body res.end(JSON.stringify({ message: 'Hello, World!' })); }); server.listen(3000, () => { console.log('Server running at http://localhost:3000'); });

Handling Different Routes and Methods (Without Express)

const http = require('http'); const url = require('url'); const server = http.createServer(async (req, res) => { const parsedUrl = new URL(req.url, `http://${req.headers.host}`); const pathname = parsedUrl.pathname; const method = req.method; // Parse request body let body = ''; for await (const chunk of req) { body += chunk; } if (body) { try { body = JSON.parse(body); } catch (e) { /* not JSON */ } } // Router const sendJSON = (statusCode, data) => { res.writeHead(statusCode, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(data)); }; if (method === 'GET' && pathname === '/') { sendJSON(200, { message: 'Welcome to the API' }); } else if (method === 'GET' && pathname === '/users') { sendJSON(200, { users: [{ id: 1, name: 'Alice' }] }); } else if (method === 'POST' && pathname === '/users') { sendJSON(201, { message: 'User created', data: body }); } else if (method === 'GET' && pathname.match(/^\/users\/\d+$/)) { const id = pathname.split('/')[2]; sendJSON(200, { user: { id: parseInt(id), name: 'Alice' } }); } else { sendJSON(404, { error: 'Not found' }); } }); server.listen(3000); // This is tedious! That's why Express exists (Module 04).

Making HTTP Requests (Client)

// Modern way: fetch (built-in since Node.js 18) async function fetchUsers() { const response = await fetch('https://jsonplaceholder.typicode.com/users'); const users = await response.json(); return users; } // With error handling and options async function createUser(userData) { const response = await fetch('https://api.example.com/users', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer YOUR_TOKEN' }, body: JSON.stringify(userData), signal: AbortSignal.timeout(5000) // 5 second timeout }); if (!response.ok) { throw new Error(`HTTP ${response.status}: ${response.statusText}`); } return response.json(); } // Using the http module directly (lower level) const http = require('http'); function httpGet(url) { return new Promise((resolve, reject) => { http.get(url, (res) => { let data = ''; res.on('data', chunk => data += chunk); res.on('end', () => resolve(JSON.parse(data))); }).on('error', reject); }); }

11. The crypto Module

const crypto = require('crypto'); // ---- HASHING (one-way, irreversible) ---- // Common algorithms: 'sha256', 'sha512', 'md5' (avoid md5 for security) const hash = crypto.createHash('sha256') .update('Hello, World!') .digest('hex'); // '315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3' // File hash (checksum) const fs = require('fs'); const fileHash = crypto.createHash('sha256'); const stream = fs.createReadStream('file.txt'); stream.on('data', (chunk) => fileHash.update(chunk)); stream.on('end', () => { console.log('File hash:', fileHash.digest('hex')); }); // ---- HMAC (Hash-based Message Authentication Code) ---- // Used to verify data integrity AND authenticity const hmac = crypto.createHmac('sha256', 'secret-key') .update('message to authenticate') .digest('hex'); // This is used in JWT signatures, webhook verification, etc. // ---- RANDOM VALUES ---- const randomBytes = crypto.randomBytes(32); // Cryptographically secure random bytes const randomHex = crypto.randomBytes(16).toString('hex'); // Random hex string const randomUUID = crypto.randomUUID(); // 'f47ac10b-58cc-4372-a567-0e02b2c3d479' // ---- ENCRYPTION / DECRYPTION (two-way, reversible) ---- const algorithm = 'aes-256-gcm'; const key = crypto.randomBytes(32); // 256-bit key const iv = crypto.randomBytes(16); // Initialization vector // Encrypt function encrypt(text) { const cipher = crypto.createCipheriv(algorithm, key, iv); let encrypted = cipher.update(text, 'utf8', 'hex'); encrypted += cipher.final('hex'); const authTag = cipher.getAuthTag(); return { encrypted, iv: iv.toString('hex'), authTag: authTag.toString('hex') }; } // Decrypt function decrypt(encryptedData) { const decipher = crypto.createDecipheriv( algorithm, key, Buffer.from(encryptedData.iv, 'hex') ); decipher.setAuthTag(Buffer.from(encryptedData.authTag, 'hex')); let decrypted = decipher.update(encryptedData.encrypted, 'hex', 'utf8'); decrypted += decipher.final('utf8'); return decrypted; } const encrypted = encrypt('Secret message'); console.log('Encrypted:', encrypted); const decrypted = decrypt(encrypted); console.log('Decrypted:', decrypted); // 'Secret message' // ---- PASSWORD HASHING (use bcrypt or argon2 in production — see Module 05) ---- // Node's built-in scrypt: function hashPassword(password) { const salt = crypto.randomBytes(16).toString('hex'); return new Promise((resolve, reject) => { crypto.scrypt(password, salt, 64, (err, derivedKey) => { if (err) reject(err); resolve(`${salt}:${derivedKey.toString('hex')}`); }); }); } function verifyPassword(password, storedHash) { const [salt, hash] = storedHash.split(':'); return new Promise((resolve, reject) => { crypto.scrypt(password, salt, 64, (err, derivedKey) => { if (err) reject(err); resolve(crypto.timingSafeEqual( Buffer.from(hash, 'hex'), derivedKey )); }); }); }

12. NPM — Package Management

Essential Commands

# Initialize a project npm init # Interactive npm init -y # Quick (all defaults) # Installing packages npm install express # Add dependency (saved to package.json) npm install nodemon --save-dev # Add dev dependency npm install -g npm-check-updates # Install globally # Short forms npm i express # Same as npm install npm i -D nodemon # Same as --save-dev # Remove packages npm uninstall express # Remove from node_modules and package.json # Update packages npm update # Update all within version constraints npm outdated # Show outdated packages npx npm-check-updates # Show available major updates # Run scripts (defined in package.json) npm start # Runs "start" script npm test # Runs "test" script npm run dev # Runs custom "dev" script # Security npm audit # Check for vulnerabilities npm audit fix # Auto-fix vulnerabilities

Understanding package.json

{ "name": "my-api", "version": "1.0.0", "description": "My awesome API", "main": "src/index.js", "type": "module", "scripts": { "start": "node src/index.js", "dev": "nodemon src/index.js", "test": "jest", "lint": "eslint src/", "build": "tsc" }, "dependencies": { "express": "^4.18.2", "jsonwebtoken": "~9.0.0" }, "devDependencies": { "nodemon": "^3.0.1", "jest": "^29.7.0" }, "engines": { "node": ">=18.0.0" } }

Version Ranges

"express": "4.18.2" → Exact version only "express": "^4.18.2" → >=4.18.2 and <5.0.0 (minor + patch updates) "express": "~4.18.2" → >=4.18.2 and <4.19.0 (patch updates only) "express": ">=4.0.0" → Any version 4.0.0 or higher "express": "*" → Any version (dangerous!) ^ (caret) = default, allows minor updates (safe for most packages) ~ (tilde) = conservative, only allows patch updates

package-lock.json — Why It Exists

package.json says: "express": "^4.18.2" Your machine installs: express@4.18.2 Three months later, a teammate runs npm install: Without lock file: express@4.19.1 (newer version, might break things!) With lock file: express@4.18.2 (exact same version — deterministic!) ALWAYS commit package-lock.json to version control!

13. Environment Variables & Configuration

// ---- ACCESSING ENVIRONMENT VARIABLES ---- console.log(process.env.NODE_ENV); // "development" or "production" console.log(process.env.PORT); // "3000" (always strings!) console.log(process.env.DB_URL); // "mongodb://localhost/mydb" // ---- USING .env FILES (with dotenv package) ---- // npm install dotenv // .env file (DO NOT commit to git!): // PORT=3000 // DB_URL=mongodb://localhost/mydb // JWT_SECRET=super-secret-key-12345 // NODE_ENV=development // At the top of your entry file: require('dotenv').config(); // Or in Node.js 20+: node --env-file=.env app.js // ---- CONFIGURATION PATTERN ---- // config.js const config = { port: parseInt(process.env.PORT, 10) || 3000, db: { url: process.env.DB_URL || 'mongodb://localhost/mydb', poolSize: parseInt(process.env.DB_POOL_SIZE, 10) || 10, }, jwt: { secret: process.env.JWT_SECRET, expiresIn: process.env.JWT_EXPIRES_IN || '1h', }, redis: { url: process.env.REDIS_URL || 'redis://localhost:6379', }, isProduction: process.env.NODE_ENV === 'production', isDevelopment: process.env.NODE_ENV === 'development', }; // Validate required env vars at startup const required = ['JWT_SECRET', 'DB_URL']; for (const name of required) { if (!process.env[name]) { console.error(`Missing required environment variable: ${name}`); process.exit(1); } } module.exports = config;

The process Object — Global Info

// process is available everywhere in Node.js (no require needed) process.env // Environment variables process.argv // Command-line arguments process.cwd() // Current working directory process.pid // Process ID process.ppid // Parent process ID process.platform // 'win32', 'linux', 'darwin' (macOS) process.arch // 'x64', 'arm64', etc. process.version // Node.js version ('v20.10.0') process.versions // Versions of dependencies (v8, openssl, etc.) process.memoryUsage() // { rss, heapTotal, heapUsed, external, arrayBuffers } process.uptime() // Seconds since process started process.hrtime.bigint() // High-resolution time in nanoseconds process.exit(0) // Exit with success code (1 = error) // Command-line arguments // node app.js --port 3000 --verbose console.log(process.argv); // ['node', '/path/to/app.js', '--port', '3000', '--verbose'] // stdin / stdout / stderr (streams) process.stdout.write('Hello\n'); // Same as console.log but no newline auto process.stderr.write('Error!\n'); // Signal handling process.on('SIGINT', () => { console.log('\nGraceful shutdown...'); server.close(() => process.exit(0)); }); process.on('SIGTERM', () => { console.log('Received SIGTERM, shutting down...'); server.close(() => process.exit(0)); });

14. Child Processes & Worker Threads

Child Processes — Running External Commands

const { exec, execFile, spawn, fork } = require('child_process'); // 1. exec — run a command, buffer the output (for small output) exec('ls -la', (error, stdout, stderr) => { if (error) { console.error('Error:', error.message); return; } console.log('Output:', stdout); }); // Promise version const { promisify } = require('util'); const execAsync = promisify(exec); async function run() { const { stdout } = await execAsync('node --version'); console.log('Node version:', stdout.trim()); } // 2. spawn — for long-running processes with streaming output const child = spawn('node', ['heavy-script.js']); child.stdout.on('data', (data) => { console.log(`stdout: ${data}`); }); child.stderr.on('data', (data) => { console.error(`stderr: ${data}`); }); child.on('close', (code) => { console.log(`Child process exited with code ${code}`); }); // 3. fork — special spawn for Node.js scripts (has IPC channel) // parent.js const child = fork('./worker.js'); child.send({ task: 'compute', data: [1, 2, 3, 4, 5] }); child.on('message', (result) => { console.log('Result from child:', result); }); // worker.js process.on('message', (message) => { if (message.task === 'compute') { const sum = message.data.reduce((a, b) => a + b, 0); process.send({ sum }); } });

Worker Threads — True Multi-Threading

const { Worker, isMainThread, parentPort, workerData } = require('worker_threads'); if (isMainThread) { // MAIN THREAD console.log('Main thread'); const worker = new Worker(__filename, { workerData: { numbers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] } }); worker.on('message', (result) => { console.log('Sum:', result); // 55 }); worker.on('error', (err) => { console.error('Worker error:', err); }); worker.on('exit', (code) => { console.log('Worker exited with code:', code); }); } else { // WORKER THREAD const { numbers } = workerData; // CPU-intensive work that won't block the main thread const sum = numbers.reduce((a, b) => a + b, 0); parentPort.postMessage(sum); } // When to use Worker Threads vs Child Processes: // Worker Threads: Share memory (SharedArrayBuffer), lighter, same process // Child Processes: Isolated, can run any command, heavier

Real-World: CPU-Intensive Task with Worker Pool

// worker-pool.js const { Worker } = require('worker_threads'); const os = require('os'); class WorkerPool { #workers = []; #queue = []; #activeWorkers = 0; constructor(workerScript, poolSize = os.cpus().length) { this.workerScript = workerScript; this.poolSize = poolSize; } execute(data) { return new Promise((resolve, reject) => { this.#queue.push({ data, resolve, reject }); this.#processQueue(); }); } #processQueue() { if (this.#queue.length === 0 || this.#activeWorkers >= this.poolSize) return; const { data, resolve, reject } = this.#queue.shift(); this.#activeWorkers++; const worker = new Worker(this.workerScript, { workerData: data }); worker.on('message', (result) => { resolve(result); this.#activeWorkers--; this.#processQueue(); }); worker.on('error', (err) => { reject(err); this.#activeWorkers--; this.#processQueue(); }); } } // Usage: const pool = new WorkerPool('./heavy-computation.js', 4); // Process 100 tasks using 4 workers const tasks = Array.from({ length: 100 }, (_, i) => pool.execute({ taskId: i })); const results = await Promise.all(tasks);

15. Clustering — Multi-Core

Node.js is single-threaded, but servers have multiple CPU cores. Clustering creates multiple Node.js processes to utilize all cores.

const cluster = require('cluster'); const http = require('http'); const os = require('os'); const numCPUs = os.cpus().length; if (cluster.isPrimary) { // PRIMARY PROCESS — creates workers console.log(`Primary ${process.pid} is running`); console.log(`Forking ${numCPUs} workers...`); for (let i = 0; i < numCPUs; i++) { cluster.fork(); } // Restart workers that die cluster.on('exit', (worker, code, signal) => { console.log(`Worker ${worker.process.pid} died (${signal || code}). Restarting...`); cluster.fork(); }); } else { // WORKER PROCESS — handles requests http.createServer((req, res) => { res.writeHead(200); res.end(`Hello from worker ${process.pid}\n`); }).listen(3000); console.log(`Worker ${process.pid} started`); } // Output: // Primary 1234 is running // Forking 8 workers... // Worker 1235 started // Worker 1236 started // ... (8 workers, all sharing port 3000) // Requests are distributed across workers (round-robin on Linux, random on Windows)

In production, use PM2 instead of manual clustering:

npm install -g pm2 pm2 start app.js -i max # Starts one worker per CPU core pm2 status # View status pm2 reload app # Zero-downtime restart

16. Debugging Node.js

# 1. Built-in debugger node --inspect app.js # Opens debugger on ws://127.0.0.1:9229 # Open Chrome → chrome://inspect → click "inspect" # 2. Break on first line node --inspect-brk app.js # 3. VS Code debugging # Add to .vscode/launch.json: # { # "type": "node", # "request": "launch", # "name": "Debug App", # "program": "${workspaceFolder}/app.js" # } # Then press F5 # 4. Console debugging (quick and dirty) console.log('value:', value); console.dir(obj, { depth: null }); # Deep inspect object console.time('operation'); // ...expensive operation... console.timeEnd('operation'); # "operation: 123.456ms" console.trace(); # Print stack trace console.table([{ a: 1 }, { a: 2 }]); # Pretty table

17. Practice Problems

Problem 1: Build a Simple File-Based Database

// Implement a JSON file-based database with CRUD operations // TRY IT YOURSELF FIRST! const fs = require('fs/promises'); const path = require('path'); class FileDB { constructor(filePath) { this.filePath = filePath; } async #read() { try { const data = await fs.readFile(this.filePath, 'utf8'); return JSON.parse(data); } catch (err) { if (err.code === 'ENOENT') return []; throw err; } } async #write(data) { await fs.writeFile(this.filePath, JSON.stringify(data, null, 2)); } async create(record) { const data = await this.#read(); const newRecord = { id: Date.now().toString(), ...record, createdAt: new Date() }; data.push(newRecord); await this.#write(data); return newRecord; } async findAll(filter = {}) { const data = await this.#read(); return data.filter(record => Object.entries(filter).every(([key, value]) => record[key] === value) ); } async findById(id) { const data = await this.#read(); return data.find(r => r.id === id) || null; } async update(id, updates) { const data = await this.#read(); const index = data.findIndex(r => r.id === id); if (index === -1) return null; data[index] = { ...data[index], ...updates, updatedAt: new Date() }; await this.#write(data); return data[index]; } async delete(id) { const data = await this.#read(); const filtered = data.filter(r => r.id !== id); if (filtered.length === data.length) return false; await this.#write(filtered); return true; } } // Test: const db = new FileDB('./users.json'); const user = await db.create({ name: 'Alice', email: 'alice@test.com' }); console.log(user); console.log(await db.findAll({ name: 'Alice' }));

Problem 2: Build a Stream-Based CSV Parser

// Parse a CSV file using streams (handle large files efficiently) const { Transform } = require('stream'); const fs = require('fs'); class CSVParser extends Transform { constructor(options = {}) { super({ objectMode: true }); this.headers = null; this.delimiter = options.delimiter || ','; this.buffer = ''; } _transform(chunk, encoding, callback) { this.buffer += chunk.toString(); const lines = this.buffer.split('\n'); this.buffer = lines.pop(); // Keep incomplete last line for (const line of lines) { if (!line.trim()) continue; const values = line.split(this.delimiter).map(v => v.trim()); if (!this.headers) { this.headers = values; continue; } const record = {}; this.headers.forEach((header, i) => { record[header] = values[i] || ''; }); this.push(record); } callback(); } _flush(callback) { if (this.buffer.trim() && this.headers) { const values = this.buffer.split(this.delimiter).map(v => v.trim()); const record = {}; this.headers.forEach((header, i) => { record[header] = values[i] || ''; }); this.push(record); } callback(); } } // Usage: fs.createReadStream('data.csv') .pipe(new CSVParser()) .on('data', (record) => { console.log(record); // { name: 'Alice', age: '25', city: 'NYC' } }) .on('end', () => { console.log('Done parsing'); });

18. Interview Questions

Q1: How does Node.js handle concurrent requests if it's single-threaded?

Answer: Node.js uses an event-driven, non-blocking I/O model. When an I/O operation (database query, file read, HTTP request) is initiated, Node.js delegates it to the OS or libuv's thread pool and continues processing other requests. When the I/O completes, its callback is placed in the event queue and executed when the call stack is empty. This allows thousands of concurrent connections with a single thread.

Q2: What is the difference between process.nextTick() and setImmediate()?

Answer: process.nextTick() executes its callback before ANY I/O or timer, immediately after the current operation completes (before the event loop continues). setImmediate() executes in the "check" phase of the event loop, after I/O events. nextTick has higher priority but can starve I/O if used recursively.

Q3: What are streams? Why are they important?

Answer: Streams are objects for reading or writing data piece-by-piece (in chunks) instead of all at once. They're important for memory efficiency — you can process a 10GB file with only 64KB of memory. Node.js has four types: Readable, Writable, Duplex, and Transform. The pipe() method connects streams and handles backpressure automatically.

Q4: Explain the difference between spawn, exec, execFile, and fork.

Answer:

  • exec — Runs a command in a shell, buffers the output, good for small outputs
  • execFile — Like exec but runs a file directly without a shell (more efficient, safer)
  • spawn — Streams stdout/stderr, doesn't buffer, good for long-running processes
  • fork — Special spawn for Node.js scripts, creates an IPC channel for message passing between parent and child

Q5: What's the difference between dependencies and devDependencies?

Answer: dependencies are packages needed for the app to run in production (express, mongoose). devDependencies are only needed during development (jest, nodemon, eslint). When deploying, npm install --production skips devDependencies.

Q6: How would you handle a CPU-intensive task in Node.js without blocking the event loop?

Answer: Options include: (1) Worker threads for CPU-bound work within the same process, (2) Child processes to offload to separate processes, (3) Clustering to distribute across CPU cores, (4) Breaking the task into smaller chunks using setImmediate() to yield to the event loop, (5) Offloading to a job queue (Bull/BullMQ with Redis) for background processing.

Next Module: 04 - Express.js & REST APIs — Building production-ready web servers.