Initial

case_studies/README.md

@@ -0,0 +1,41 @@
+# Case Studies
+
+Real-world examples demonstrating space-time tradeoffs in modern computing systems.
+
+## Current Case Studies
+
+### 1. Large Language Models (LLMs)
+See `llm_transformers/` - Analysis of how transformer models exhibit space-time tradeoffs through:
+- Model compression techniques (quantization, pruning)
+- KV-cache optimization
+- Flash Attention and memory-efficient attention mechanisms
+
+## Planned Case Studies
+
+### 2. Database Systems
+- Query optimization strategies
+- Index vs sequential scan tradeoffs
+- In-memory vs disk-based processing
+
+### 3. Blockchain Systems
+- Full nodes vs light clients
+- State pruning strategies
+- Proof-of-work vs proof-of-stake memory requirements
+
+### 4. Compiler Optimizations
+- Register allocation strategies
+- Loop unrolling vs code size
+- JIT compilation tradeoffs
+
+### 5. Distributed Computing
+- MapReduce shuffle strategies
+- Spark RDD persistence levels
+- Message passing vs shared memory
+
+## Contributing
+
+Each case study should include:
+1. Background on the system
+2. Identification of space-time tradeoffs
+3. Quantitative analysis where possible
+4. Connection to theoretical results

case_studies/database_systems/README.md

@@ -0,0 +1,184 @@
+# Database Systems: Space-Time Tradeoffs in Practice
+
+## Overview
+Databases are perhaps the most prominent example of space-time tradeoffs in production systems. Every major database makes explicit decisions about trading memory for computation time.
+
+## 1. Query Processing
+
+### Hash Join vs Nested Loop Join
+
+**Hash Join (More Memory)**
+- Build hash table: O(n) space
+- Probe phase: O(n+m) time
+- Used when: Sufficient memory available
+```sql
+-- PostgreSQL will choose hash join if work_mem is high enough
+SET work_mem = '256MB';
+SELECT * FROM orders o JOIN customers c ON o.customer_id = c.id;
+```
+
+**Nested Loop Join (Less Memory)**
+- Space: O(1) 
+- Time: O(n×m)
+- Used when: Memory constrained
+```sql
+-- Force nested loop with low work_mem
+SET work_mem = '64kB';
+```
+
+### Real PostgreSQL Example
+```sql
+-- Monitor actual memory usage
+EXPLAIN (ANALYZE, BUFFERS) 
+SELECT * FROM large_table JOIN huge_table USING (id);
+
+-- Output shows:
+-- Hash Join: 145MB memory, 2.3 seconds
+-- Nested Loop: 64KB memory, 487 seconds
+```
+
+## 2. Indexing Strategies
+
+### B-Tree vs Full Table Scan
+- **B-Tree Index**: O(n) space, O(log n) lookup
+- **No Index**: O(1) extra space, O(n) scan time
+
+### Covering Indexes
+Trading more space for zero I/O reads:
+```sql
+-- Regular index: must fetch row data
+CREATE INDEX idx_user_email ON users(email);
+
+-- Covering index: all data in index (more space)
+CREATE INDEX idx_user_email_covering ON users(email) INCLUDE (name, created_at);
+```
+
+## 3. Materialized Views
+
+Ultimate space-for-time trade:
+```sql
+-- Compute once, store results
+CREATE MATERIALIZED VIEW sales_summary AS
+SELECT 
+    date_trunc('day', sale_date) as day,
+    product_id,
+    SUM(amount) as total_sales,
+    COUNT(*) as num_sales
+FROM sales
+GROUP BY 1, 2;
+
+-- Instant queries vs recomputation
+SELECT * FROM sales_summary WHERE day = '2024-01-15';  -- 1ms
+-- vs
+SELECT ... FROM sales GROUP BY ...;  -- 30 seconds
+```
+
+## 4. Buffer Pool Management
+
+### PostgreSQL's shared_buffers
+```
+# Low memory: more disk I/O
+shared_buffers = 128MB  # Frequent disk reads
+
+# High memory: cache working set  
+shared_buffers = 8GB    # Most data in RAM
+```
+
+Performance impact:
+- 128MB: TPC-H query takes 45 minutes
+- 8GB: Same query takes 3 minutes
+
+## 5. Query Planning
+
+### Bitmap Heap Scan
+A perfect example of √n-like behavior:
+1. Build bitmap of matching rows: O(√n) space
+2. Scan heap in physical order: Better than random I/O
+3. Falls between index scan and sequential scan
+
+```sql
+EXPLAIN SELECT * FROM orders WHERE status IN ('pending', 'processing');
+-- Bitmap Heap Scan on orders
+-- Recheck Cond: (status = ANY ('{pending,processing}'::text[]))
+-- -> Bitmap Index Scan on idx_status
+```
+
+## 6. Write-Ahead Logging (WAL)
+
+Trading write performance for durability:
+- **Synchronous commit**: Every transaction waits for disk
+- **Asynchronous commit**: Buffer writes, risk data loss
+```sql
+-- Trade durability for speed
+SET synchronous_commit = off;  -- 10x faster inserts
+```
+
+## 7. Column Stores vs Row Stores
+
+### Row Store (PostgreSQL, MySQL)
+- Store complete rows together
+- Good for OLTP, random access
+- Space: Stores all columns even if not needed
+
+### Column Store (ClickHouse, Vertica)  
+- Store each column separately
+- Excellent compression (less space)
+- Must reconstruct rows (more time for some queries)
+
+Example compression ratios:
+- Row store: 100GB table
+- Column store: 15GB (85% space savings)
+- But: Random row lookup 100x slower
+
+## 8. Real-World Configuration
+
+### PostgreSQL Memory Settings
+```conf
+# Total system RAM: 64GB
+
+# Aggressive caching (space for time)
+shared_buffers = 16GB          # 25% of RAM
+work_mem = 256MB               # Per operation
+maintenance_work_mem = 2GB     # For VACUUM, CREATE INDEX
+
+# Conservative (time for space)  
+shared_buffers = 128MB         # Minimal caching
+work_mem = 4MB                 # Forces disk-based operations
+```
+
+### MySQL InnoDB Buffer Pool
+```conf
+# 75% of RAM for buffer pool
+innodb_buffer_pool_size = 48G
+
+# Adaptive hash index (space for time)
+innodb_adaptive_hash_index = ON
+```
+
+## 9. Distributed Databases
+
+### Replication vs Computation
+- **Full replication**: n× space, instant reads
+- **No replication**: 1× space, distributed queries
+
+### Cassandra's Space Amplification
+- Replication factor 3: 3× space
+- Plus SSTables: Another 2-3× during compaction
+- Total: ~10× space for high availability
+
+## Key Insights
+
+1. **Every join algorithm** is a space-time tradeoff
+2. **Indexes** are precomputed results (space for time)
+3. **Buffer pools** cache hot data (space for I/O time)
+4. **Query planners** explicitly optimize these tradeoffs
+5. **DBAs tune memory** to control space-time balance
+
+## Connection to Williams' Result
+
+Databases naturally implement √n-like algorithms:
+- Bitmap indexes: O(√n) space for range queries
+- Sort-merge joins: O(√n) memory for external sort
+- Buffer pool: Typically sized at √(database size)
+
+The ubiquity of these patterns in database internals validates Williams' theoretical insights about the fundamental nature of space-time tradeoffs in computation.

case_studies/distributed_computing/README.md

@@ -0,0 +1,269 @@
+# Distributed Computing: Space-Time Tradeoffs at Scale
+
+## Overview
+Distributed systems make explicit decisions about replication (space) vs computation (time). Every major distributed framework embodies these tradeoffs.
+
+## 1. MapReduce / Hadoop
+
+### Shuffle Phase - The Classic Tradeoff
+```java
+// Map output: Written to local disk (space for fault tolerance)
+map(key, value):
+    for word in value.split():
+        emit(word, 1)
+
+// Shuffle: All-to-all communication
+// Choice: Buffer in memory vs spill to disk
+shuffle.memory.ratio = 0.7  // 70% of heap for shuffle
+shuffle.spill.percent = 0.8 // Spill when 80% full
+```
+
+**Memory Settings Impact:**
+- High memory: Fast shuffle, risk of OOM
+- Low memory: Frequent spills, 10x slower
+- Sweet spot: √(data_size) memory per node
+
+### Combiner Optimization
+```java
+// Without combiner: Send all data
+map: (word, 1), (word, 1), (word, 1)...
+
+// With combiner: Local aggregation (compute for space)
+combine: (word, 3)
+
+// Network transfer: 100x reduction
+// CPU cost: Local sum computation
+```
+
+## 2. Apache Spark
+
+### RDD Persistence Levels
+```scala
+// MEMORY_ONLY: Fast but memory intensive
+rdd.persist(StorageLevel.MEMORY_ONLY)
+// Space: Full dataset in RAM
+// Time: Instant access
+
+// MEMORY_AND_DISK: Spill to disk when needed
+rdd.persist(StorageLevel.MEMORY_AND_DISK)
+// Space: Min(dataset, available_ram)
+// Time: RAM-speed or disk-speed
+
+// DISK_ONLY: Minimal memory
+rdd.persist(StorageLevel.DISK_ONLY)
+// Space: O(1) RAM
+// Time: Always disk I/O
+
+// MEMORY_ONLY_SER: Serialized in memory
+rdd.persist(StorageLevel.MEMORY_ONLY_SER)
+// Space: 2-5x reduction via serialization
+// Time: CPU cost to deserialize
+```
+
+### Broadcast Variables
+```scala
+// Without broadcast: Send to each task
+val bigData = loadBigDataset() // 1GB
+rdd.map(x => doSomething(x, bigData))
+// Network: 1GB × num_tasks
+
+// With broadcast: Send once per node
+val bcData = sc.broadcast(bigData)
+rdd.map(x => doSomething(x, bcData.value))
+// Network: 1GB × num_nodes
+// Memory: Extra copy per node
+```
+
+## 3. Distributed Key-Value Stores
+
+### Redis Eviction Policies
+```conf
+# No eviction: Fail when full (pure space)
+maxmemory-policy noeviction
+
+# LRU: Recompute evicted data (time for space)
+maxmemory-policy allkeys-lru
+maxmemory 10gb
+
+# LFU: Better hit rate, more CPU
+maxmemory-policy allkeys-lfu
+```
+
+### Memcached Slab Allocation
+- Fixed-size slabs: Internal fragmentation (waste space)
+- Variable-size: External fragmentation (CPU to compact)
+- Typical: √n slab classes for n object sizes
+
+## 4. Kafka / Stream Processing
+
+### Log Compaction
+```properties
+# Keep all messages (max space)
+cleanup.policy=none
+
+# Keep only latest per key (compute to save space)
+cleanup.policy=compact
+min.compaction.lag.ms=86400000
+
+# Compression (CPU for space)
+compression.type=lz4  # 4x space reduction
+compression.type=zstd # 6x reduction, more CPU
+```
+
+### Consumer Groups
+- Replicate processing: Each consumer gets all data
+- Partition assignment: Each message processed once
+- Tradeoff: Redundancy vs coordination overhead
+
+## 5. Kubernetes / Container Orchestration
+
+### Resource Requests vs Limits
+```yaml
+resources:
+  requests:
+    memory: "256Mi"  # Guaranteed (space reservation)
+    cpu: "250m"      # Guaranteed (time reservation)
+  limits:
+    memory: "512Mi"  # Max before OOM
+    cpu: "500m"      # Max before throttling
+```
+
+### Image Layer Caching
+- Base images: Shared across containers (dedup space)
+- Layer reuse: Fast container starts
+- Tradeoff: Registry space vs pull time
+
+## 6. Distributed Consensus
+
+### Raft Log Compaction
+```go
+// Snapshot periodically to bound log size
+if logSize > maxLogSize {
+    snapshot = createSnapshot(stateMachine)
+    truncateLog(snapshot.index)
+}
+// Space: O(snapshot) instead of O(all_operations)
+// Time: Recreate state from snapshot + recent ops
+```
+
+### Multi-Paxos vs Raft
+- Multi-Paxos: Less memory, complex recovery
+- Raft: More memory (full log), simple recovery
+- Tradeoff: Space vs implementation complexity
+
+## 7. Content Delivery Networks (CDNs)
+
+### Edge Caching Strategy
+```nginx
+# Cache everything (max space)
+proxy_cache_valid 200 30d;
+proxy_cache_max_size 100g;
+
+# Cache popular only (compute popularity)
+proxy_cache_min_uses 3;
+proxy_cache_valid 200 1h;
+proxy_cache_max_size 10g;
+```
+
+### Geographic Replication
+- Full replication: Every edge has all content
+- Lazy pull: Fetch on demand
+- Predictive push: ML models predict demand
+
+## 8. Batch Processing Frameworks
+
+### Apache Flink Checkpointing
+```java
+// Checkpoint frequency (space vs recovery time)
+env.enableCheckpointing(10000); // Every 10 seconds
+
+// State backend choice
+env.setStateBackend(new FsStateBackend("hdfs://..."));
+// vs
+env.setStateBackend(new RocksDBStateBackend("file://..."));
+
+// RocksDB: Spill to disk, slower access
+// Memory: Fast access, limited size
+```
+
+### Watermark Strategies
+- Perfect watermarks: Buffer all late data (space)
+- Heuristic watermarks: Drop some late data (accuracy for space)
+- Allowed lateness: Bounded buffer
+
+## 9. Real-World Examples
+
+### Google's MapReduce (2004)
+- Problem: Processing 20TB of web data
+- Solution: Trade disk space for fault tolerance
+- Impact: 1000 machines × 3 hours vs 1 machine × 3000 hours
+
+### Facebook's TAO (2013)
+- Problem: Social graph queries
+- Solution: Replicate to every datacenter
+- Tradeoff: Petabytes of RAM for microsecond latency
+
+### Amazon's Dynamo (2007)
+- Problem: Shopping cart availability
+- Solution: Eventually consistent, multi-version
+- Tradeoff: Space for conflict resolution
+
+## 10. Optimization Patterns
+
+### Hierarchical Aggregation
+```python
+# Naive: All-to-one
+results = []
+for worker in workers:
+    results.extend(worker.compute())
+return aggregate(results)  # Bottleneck!
+
+# Tree aggregation: √n levels
+level1 = [aggregate(chunk) for chunk in chunks(workers, sqrt(n))]
+level2 = [aggregate(chunk) for chunk in chunks(level1, sqrt(n))]
+return aggregate(level2)
+
+# Space: O(√n) intermediate results
+# Time: O(log n) vs O(n)
+```
+
+### Bloom Filters in Distributed Joins
+```java
+// Broadcast join with Bloom filter
+BloomFilter filter = createBloomFilter(smallTable);
+broadcast(filter);
+
+// Each node filters locally
+bigTable.filter(row -> filter.mightContain(row.key))
+        .join(broadcastedSmallTable);
+
+// Space: O(m log n) bits for filter
+// Reduction: 99% fewer network transfers
+```
+
+## Key Insights
+
+1. **Every distributed system** trades replication for computation
+2. **The √n pattern** appears in:
+   - Shuffle buffer sizes
+   - Checkpoint frequencies  
+   - Aggregation tree heights
+   - Cache sizes
+
+3. **Network is the new disk**:
+   - Network transfer ≈ Disk I/O in cost
+   - Same space-time tradeoffs apply
+
+4. **Failures force space overhead**:
+   - Replication for availability
+   - Checkpointing for recovery
+   - Logging for consistency
+
+## Connection to Williams' Result
+
+Distributed systems naturally implement √n algorithms:
+- Shuffle phases: O(√n) memory per node optimal
+- Aggregation trees: O(√n) height minimizes time
+- Cache sizing: √(total_data) per node common
+
+These patterns emerge independently across systems, validating the fundamental nature of the √(t log t) space bound for time-t computations.

case_studies/llm_transformers/detailed_analysis.md

@@ -0,0 +1,244 @@
+# Large Language Models: Space-Time Tradeoffs at Scale
+
+## Overview
+Modern LLMs are a masterclass in space-time tradeoffs. With models reaching trillions of parameters, every architectural decision trades memory for computation.
+
+## 1. Attention Mechanisms
+
+### Standard Attention (O(n²) Space)
+```python
+# Naive attention: Store full attention matrix
+def standard_attention(Q, K, V):
+    # Q, K, V: [batch, seq_len, d_model]
+    scores = Q @ K.T / sqrt(d_model)  # [batch, seq_len, seq_len]
+    attn = softmax(scores)            # Must store entire matrix!
+    output = attn @ V
+    return output
+
+# Memory: O(seq_len²) - becomes prohibitive for long sequences
+# For seq_len=32K: 4GB just for attention matrix!
+```
+
+### Flash Attention (O(n) Space)
+```python
+# Recompute attention in blocks during backward pass
+def flash_attention(Q, K, V, block_size=256):
+    # Process in blocks, never materializing full matrix
+    output = []
+    for q_block in chunks(Q, block_size):
+        block_out = compute_block_attention(q_block, K, V)
+        output.append(block_out)
+    return concat(output)
+
+# Memory: O(seq_len) - linear in sequence length!
+# Time: ~2x slower but enables 10x longer sequences
+```
+
+### Real Impact
+- GPT-3: Limited to 2K tokens due to quadratic memory
+- GPT-4 with Flash: 32K tokens with same hardware
+- Claude: 100K+ tokens using similar techniques
+
+## 2. KV-Cache Optimization
+
+### Standard KV-Cache
+```python
+# During generation, cache keys and values
+class StandardKVCache:
+    def __init__(self, max_seq_len, n_layers, n_heads, d_head):
+        # Cache for all positions
+        self.k_cache = zeros(n_layers, max_seq_len, n_heads, d_head)
+        self.v_cache = zeros(n_layers, max_seq_len, n_heads, d_head)
+    
+    # Memory: O(max_seq_len × n_layers × hidden_dim)
+    # For 70B model: ~140GB for 32K context!
+```
+
+### Multi-Query Attention (MQA)
+```python
+# Share keys/values across heads
+class MQACache:
+    def __init__(self, max_seq_len, n_layers, d_model):
+        # Single K,V per layer instead of per head
+        self.k_cache = zeros(n_layers, max_seq_len, d_model)
+        self.v_cache = zeros(n_layers, max_seq_len, d_model)
+    
+    # Memory: O(max_seq_len × n_layers × d_model / n_heads)
+    # 8-32x memory reduction!
+```
+
+### Grouped-Query Attention (GQA)
+Balance between quality and memory:
+- Groups of 4-8 heads share K,V
+- 4-8x memory reduction
+- <1% quality loss
+
+## 3. Model Quantization
+
+### Full Precision (32-bit)
+```python
+# Standard weights
+weight = torch.randn(4096, 4096, dtype=torch.float32)
+# Memory: 64MB per layer
+# Computation: Fast matmul
+```
+
+### INT8 Quantization
+```python
+# 8-bit weights with scale factors
+weight_int8 = (weight * scale).round().clamp(-128, 127).to(torch.int8)
+# Memory: 16MB per layer (4x reduction)
+# Computation: Slightly slower, dequantize on the fly
+```
+
+### 4-bit Quantization (QLoRA)
+```python
+# Extreme quantization with adapters
+weight_4bit = quantize_nf4(weight)  # 4-bit normal float
+lora_A = torch.randn(4096, 16)      # Low-rank adapter
+lora_B = torch.randn(16, 4096)
+
+def forward(x):
+    # Dequantize and compute
+    base = dequantize(weight_4bit) @ x
+    adapter = lora_B @ (lora_A @ x)
+    return base + adapter
+
+# Memory: 8MB base + 0.5MB adapter (8x reduction)
+# Time: 2-3x slower due to dequantization
+```
+
+## 4. Checkpoint Strategies
+
+### Gradient Checkpointing
+```python
+# Standard: Store all activations
+def transformer_layer(x):
+    attn = self.attention(x)      # Store activation
+    ff = self.feedforward(attn)   # Store activation
+    return ff
+
+# With checkpointing: Recompute during backward
+@checkpoint
+def transformer_layer(x):
+    attn = self.attention(x)      # Don't store
+    ff = self.feedforward(attn)   # Don't store
+    return ff
+
+# Memory: O(√n_layers) instead of O(n_layers)
+# Time: 30% slower training
+```
+
+## 5. Sparse Models
+
+### Dense Model
+- Every token processed by all parameters
+- Memory: O(n_params)
+- Time: O(n_tokens × n_params)
+
+### Mixture of Experts (MoE)
+```python
+# Route to subset of experts
+def moe_layer(x):
+    router_logits = self.router(x)
+    expert_ids = top_k(router_logits, k=2)
+    
+    output = 0
+    for expert_id in expert_ids:
+        output += self.experts[expert_id](x)
+    
+    return output
+
+# Memory: Full model size
+# Active memory: O(n_params / n_experts)
+# Enables 10x larger models with same compute
+```
+
+## 6. Real-World Examples
+
+### GPT-3 vs GPT-4
+| Aspect | GPT-3 | GPT-4 |
+|--------|-------|-------|
+| Parameters | 175B | ~1.8T (MoE) |
+| Context | 2K | 32K-128K |
+| Techniques | Dense | MoE + Flash + GQA |
+| Memory/token | ~350MB | ~50MB (active) |
+
+### Llama 2 Family
+```
+Llama-2-7B:  Full precision = 28GB
+             INT8 = 7GB
+             INT4 = 3.5GB
+             
+Llama-2-70B: Full precision = 280GB
+             INT8 = 70GB
+             INT4 + QLoRA = 35GB (fits on single GPU!)
+```
+
+## 7. Serving Optimizations
+
+### Continuous Batching
+Instead of fixed batches, dynamically batch requests:
+- Memory: Reuse KV-cache across requests
+- Time: Higher throughput via better GPU utilization
+
+### PagedAttention (vLLM)
+```python
+# Treat KV-cache like virtual memory
+class PagedKVCache:
+    def __init__(self, block_size=16):
+        self.blocks = {}  # Allocated on demand
+        self.page_table = {}  # Maps positions to blocks
+    
+    def allocate(self, seq_id, position):
+        # Only allocate blocks as needed
+        if position // self.block_size not in self.page_table[seq_id]:
+            self.page_table[seq_id].append(new_block())
+```
+
+Memory fragmentation: <5% vs 60% for naive allocation
+
+## 8. Training vs Inference Tradeoffs
+
+### Training (Memory Intensive)
+- Gradients: 2x model size
+- Optimizer states: 2-3x model size
+- Activations: O(batch × seq_len × layers)
+- Total: 15-20x model parameters
+
+### Inference (Can Trade Memory for Time)
+- Only model weights needed
+- Quantize aggressively
+- Recompute instead of cache
+- Stream weights from disk if needed
+
+## Key Insights
+
+1. **Every major LLM innovation** is a space-time tradeoff:
+   - Flash Attention: Recompute for linear memory
+   - Quantization: Dequantize for smaller models
+   - MoE: Route for sparse activation
+
+2. **The √n pattern appears everywhere**:
+   - Gradient checkpointing: √n_layers memory
+   - Block-wise attention: √seq_len blocks
+   - Optimal batch sizes: Often √total_examples
+
+3. **Practical systems combine multiple techniques**:
+   - GPT-4: MoE + Flash + INT8 + GQA
+   - Llama: Quantization + RoPE + GQA
+   - Claude: Flash + Constitutional training
+
+4. **Memory is the binding constraint**:
+   - Not compute or data
+   - Drives all architectural decisions
+   - Williams' result predicts these optimizations
+
+## Connection to Theory
+
+Williams showed TIME[t] ⊆ SPACE[√(t log t)]. In LLMs:
+- Standard attention: O(n²) space, O(n²) time
+- Flash attention: O(n) space, O(n² log n) time
+- The log factor comes from block coordination
+
+This validates that the theoretical √t space bound manifests in practice, driving the most important optimizations in modern AI systems.