All sorts of famous Attention Layers

Unknown — Sat, 18 Jul 2026 00:00:00 +0000

Well, we had a bomb of Kimi K3 launch, and I writing this after going through it. So, we will trace this pokemon evolution.

Basics</a></h2>
Before Transformers, models like RNNs and LSTMs processed text sequentially, word by word. This process was slow and models struggled to remember information from the distant past, creating a long-range dependency problem. The attention mechanism solved this by allowing the model to look at all parts of the input sequence simultaneously and assign importance (attention) scores to each word, creating a rich context vector. It abandoned sequential processing entirely, enabling parallel computation and providing the model with a direct, weighted memory of its entire input.
I mean, if you’re reading my blogs, most probably you know what the attention is, but for reiteration, the standard softmax based attention equation looks like :

Attention(k, Q, V) = softmax(QK^T^ / d^1/2^)V </blockquote>
The attention process :
B, T, C = x.size() # batch size, sequence length, embedding dimensionality (n_embd) # calculate query, key, values for all heads in batch and move head forward to be the batch dim q, k, v = self.c_attn(x).split(self.n_embd, dim=2) k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) # manual implementation of attention att = (q @ k.transpose(-2, -1)) (1.0 / math.sqrt(k.size(-1))) att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf')) att = F.softmax(att, dim=-1) att = self.attn_dropout(att) y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs) y = y.transpose(1, 2).contiguous().view(B, T, C) # re-assemble all head outputs side by side # output projection y = self.resid_dropout(self.c_proj(y)) return y</code></pre> Once the final hidden-state matrix is produced, the language model head maps it into vocabulary logits. During autoregressive decoding, only the logits at the final position are needed to select the next token.
There is a variety!</a></h2>
Linear Attention is not a variant of Self-Attention. I used to think that so clarifying here, some papers blur the line. MLA (DeepSeek) is still softmax-based but uses low-rank projections. The hybrid models like Kimi K3 or Qwen3 mix both.
it’s a fundamentally different formulation that replaces the softmax based attention mechanism entirely.

Attention = softmax(QK^T / √d) V # O(n²) - must compute full n×n matrix Attention = φ(Q)(φ(K)^T V) # O(n) - associativity trick </blockquote>
By removing softmax and using a kernel function φ, you can change the order of operations. instead of (QK^T^)V which requires the n×n matrix, you compute K^T^ V first (d×d matrix), then multiply by Q. This is the kernel trick that makes it linear.
Attention Mechanisms ├── Softmax-based Attention (Quadratic O(n²)) │ ├── Self-Attention (vanilla) │ ├── Multi-Head Attention (MHA) │ ├── Multi-Query Attention (MQA) │ ├── Grouped-Query Attention (GQA) │ ├── Sliding Window Attention (sparse, but still softmax) │ └── Multi-Head Latent Attention (MLA) - low-rank, but still softmax │ ├── Faster Softmax Attention (same maths, better memory access via implementation optimizations) │ ├── Flash Attention - tiled SRAM computation │ ├── Flash Attention 2 - better parallelism │ ├── Flash Attention 3 - Hopper optimizations │ └── Paged Attention - dynamic KV cache allocation │ ├── Linear Attention (O(n), and no softmax) │ ├── Linear Attention (Katharopoulos 2020) - kernel trick │ ├── DeltaNet - delta rule updates │ ├── Gated DeltaNet - with gating │ ├── RetNet - retention mechanism │ └── RWKV - time-mixing │ └── State-Space Models (not attention at all) ├── S4 ├── Mamba └── Mamba-2</code></pre> The visualization makes the differences clear. Dense attention fills the entire matrix (O(n²)). Linear attention only tracks a diagonal state that grows with sequence position. Sparse attention skips positions. Flash Attention computes the same dense pattern but tiles it for better memory access. Paged Attention handles variable-length sequences with dynamic allocation. Local/Sliding Window attention restricts each token to only attend to nearby tokens.
where to look?</a></h2>
Memory is the bottleneck in inference :

For every single token generated, the model must stream billions of parameters (weights) through the memory bus. The GPU spends more time moving data in and out of memory than doing actual mathematical calculations.</li>
The KV cache stores the context and history of a prompt. Longer prompts and agentic workflows require exponentially more space, which reduces concurrent request capacity and limits batch sizes</li> </ul>
when does it even matter?</a></h3>
the attention mechanism matters less than you think (at moderate context). At 4k tokens with hidden_size=2048, the compute breakdown per transformer layer is roughly:

Feed-Forward Network (FFN): ~130 GFLOPs (two matmuls: up-proj and down-proj through a 4x expansion)</li>
Attention (including QKV projections): ~70 GFLOPs</li> </ul>
The quadratic Q@K^T attention itself is only ~2 GFLOPs (4000² × 64 per head × num_heads). That’s a tiny fraction. The rest of attention’s cost is the linear projections (Q, K, V, O), which are the same regardless of whether you use full attention, linear attention, Mamba, or anything else.
so, at sequence lengths below ~8k tokens, the attention pattern barely matters. The FFN dominates. Linear attention, sliding window, sparse patterns, they all optimize the O(n²) part which isn’t the bottleneck. The O(n × d × 4d) FFN is. This changes above ~16k tokens where the quadratic attention term starts dominating
Softmax Magic</a></h2>
Softmax is a normalizing function that depends on all elements in its input:

softmax(x_i) = exp(x_i) / Σ_j exp(x_j) </blockquote>
To compute the output for any single element, you need the sum over all elements. This is the coupling.
Applying to attention :

attention_weights_t = softmax([q_t·k_1, q_t·k_2, …, q_t·k_n]) </blockquote>
To normalize q_t’s attention over key k_5, you need to know q_t’s dot product with k_1, k_2, k_3, k_4, k_6, …, k_n. You can’t compute the attention weight for one key without knowing the scores for all keys.
# What we want (efficient): output = Q @ (K.T @ V) # compute K.T @ V first: (d×n) @ (n×d) = d×d # then Q @ result: (n×d) @ (d×d) = n×d # Total: O(n·d²) # What softmax forces: scores = Q @ K.T # n×n matrix - unavoidable weights = softmax(scores, dim=-1) # needs full row to normalize output = weights @ V # n×d # Total: O(n²·d)</code></pre> The softmax sits between the two matrix multiplications. You can’t skip computing the n×n scores matrix because softmax needs all n scores in each row to produce each normalized weight.
# Standard softmax attention (for comparison) def forward(self, x, mask=None, past_kv=None): b, t, d = x.shape d_head = d // self.num_heads h = self.num_heads qkv = self.qkv_proj(x) q = qkv[:, :, :d].view(b, t, h, d_head).transpose(1, 2) k = qkv[:, :, d:2d].view(b, t, h, d_head).transpose(1, 2) v = qkv[:, :, 2d:].view(b, t, h, d_head).transpose(1, 2) # KV cache: concat past keys/values if past_kv is not None: k = torch.cat((past_kv[0], k), dim=2) v = torch.cat((past_kv[1], v), dim=2) # O(t²) attention computation scores = (q @ k.transpose(-1, -2)) / math.sqrt(d_head) if past_kv is None: # prefill needs causal mask causal_mask = torch.triu(torch.ones(t, t, dtype=bool, device=q.device), diagonal=1) scores = scores.masked_fill(causal_mask, float('-inf')) attn = scores.softmax(-1) # this breaks associativity o = attn @ v o = o.transpose(1, 2).contiguous().view(b, t, d) return self.o_proj(o), (k, v)</code></pre>

Softmax Attention has perfect recoverability. Query k₅, get v₅. </blockquote>
Self-Attension</a></h3>
It’s at the core of transformer models. Clearly, as HBM (around 1.5 TB/s) is not the fastest thing off GPU (its not on GPU, its a chip nearby), the K,V being stored in it are problematic. So, its quadratic complexity for HBM accesses with respect to sequence length at inference is clearly bad at scale. Lots of techniques to reduce the amount of KV data transferred between the GPU and the HBM.
`Algorithm 0 - Standard Attention Implementation Require: Matrices Q, K, V e RNxd in HBM. 1: Load Q, K by blocks from HBM, compute S = QKT, write S to HBM. 2: Read S from HBM, compute P = softmax(S), write P to HBM. 3: Load P and V by blocks from HBM, compute O = PV, write O to HBM. 4: Return O.</code></pre>`
You can refer Multi-Head Attention in BERT for reference</a>.

The diagram shows the progression. In MHA every head has its own K and V. In GQA groups of heads share K and V. In MQA all heads share a single K and V. In MLA the K and V are compressed through a latent projection before being used.
Multi-Query Attention</a></h3>
It’s almost like Self-Attension. Just that Vi and Ki (i being used by each head) is not required. We can use same set of K and V across heads. So, just one K and one V tensor shared across all heads. Thus, one head is all you need!</a> So, a great optimization wrt to amount of data that would be required to be loaded via HBM. As the KV is cached as well, we need much less cache. Awesome! Less memory pressure (so you can batch more) and faster decoding on inference. But, there is a small accuracy drop as we have few params. Also, you have to train the model with MQA, can’t just a MHA trained model and use MQA on inference. And, no Tensor parallelism as then we will kinda defeat the purpose by having KV replicated across clusters.
The tradeoff is clear. MQA gives you efficiency by sharing keys and values but you lose some of the nuanced token level interactions that separate heads would capture. Each head still has its own queries so it can focus on different aspects of the input but the shared keys and values mean less diversity in what gets attended to.
You can refer Falcon 7B for reference for MQA</a>
Group Query Attension</a></h3>
Well, it’s between MHA and MQA. Just adding another hyparam to the equation, pairing up (K, V) to some heads. This gives best of both world, a nice compromise balance between speed and accuracy. 4 and 8 were quite good. Interesting thing here is that MHA models can be uptrained (not really fine-tuning, just an upgrade) to GQA. And clearly a better fit to tensor parallelism.
The way it works is that input queries get divided into groups. For each group a shared set of key and value representations is computed. Attention scores are calculated between the grouped queries and the shared key representations. The final output is a weighted sum of the shared value representations based on those attention scores. Then outputs of all query groups get combined to produce the final representation.
GQA scales more effectively with sequence length than MHA. Llama and Mistral both use it. The grouping preserves more diversity than MQA while still being more efficient than full MHA.
This can be referenced from Llama 2</a>
Sliding Window Attention</a></h3>
In vanilla attention, we compute attention score from all token, and at inference time we mask becuase we dont want decoding to look at the future. We have a triangle shaped attention mask</a> which is quadratic. What SWA does is that it limits the self attention computation to a fixed window so we get a fixed cached size. So, we can’t see more than window size from previous token. KV cache becomes a rotating buffer. So, the max context size would be window size number of layers, reducing attention complexity to linear. So, we are shortening the attention span.
Sliding window is just one type of sparse attention pattern. There are others worth knowing about. Strided attention has each token attend to every kth token which is useful for capturing periodic patterns. Reformer uses locality sensitive hashing to cluster similar queries and keys into buckets so queries only attend to keys in the same bucket giving O(n log n) complexity. BigBird combines random attention with local attention with global attention. Longformer uses local sliding window attention with task specific global attention where some tokens like CLS attend to all tokens while others use local attention.
The tradeoff with sparse patterns is loss of global context. Fixed patterns may miss long range dependencies. Choosing the right sparsity pattern is task dependent. But for long sequences the memory savings are worth it. Reformer can handle 64K tokens. Longformer works well for summarizing legal contracts or books.
You can refer Mistral 7B paper</a> and reference sliding window causal mask code</a>.
Flash Attenstion</a></h3>
As we know, HBM memory is slower to on-GPU memory. Wouldn’t it be better to run the Self-Attension computation on GPU itself (with minimal HBM accesses)? Thats exactly what flash attention does.
The key insight is understanding the GPU memory hierarchy. Small SRAM has extremely high bandwidth around 19 TB/s but limited size around 20MB. HBM has higher capacity around 40GB but lower bandwidth around 1.5 TB/s. CPU DRAM has massive capacity over 1TB but much slower bandwidth around 12.8 GB/s. Flash Attention exploits this hierarchy by keeping intermediate results in fast SRAM instead of writing them to slow HBM.
Load Q and K from HBM once Multiply Q and K, keep S in SRAM Compute P incrementally in SRAM (tiling) Materializes S = QKᵀ and P = softmax(S) and writes only the final output O</code></pre> And, parallize over batch size and number of heads. Taking N as sequence length, d as embedding length and M as size of SRAM (d<=M<=Nd), Flash Attention requires O(N^2^d^2^M^-1^) HBM accesses which still looks quadratic. But if M=N, then its O(Nd^2) HBM accesses, so linear wrt sequence length. This optimizes for forward and backward passes, so accelerate training. The tiling strategy is clever. It divides Q, K, V into blocks sized to fit GPU SRAM. The outer loop loads blocks of K and V from slow HBM into fast SRAM. The inner loop processes Q blocks against the loaded K and V blocks. This enables processing sequences up to 4x longer than conventional attention because you never materialize the full N by N attention matrix. The fused kernel execution replaces the traditional multi step attention with a unified operation. It combines matrix multiply then scaling then masking then softmax then dropout then matrix multiply into one CUDA kernel. Intermediate results like QK transpose and softmax outputs stay in SRAM and registers. Only final attention outputs get written to HBM. This reduces HBM accesses by 10 to 20x compared to standard PyTorch implementations. Later, there was FlashAttension-2 that did some rewriting to reduce number of non-matmul operations to maximize GPU throughput. Also, it optimize operations for Multi-Query Attention and Grouped-Query Attention. Even more sequence parallelism. Its over staggering 9x faster than standard attention.
Refer the FlashAttension Paper-1</a> and paper 2</a>.
Paged Attention</a></h3>
It’s a famous vLLM optimization which enables the KV cache memory grows and shrinks dynamically for each inference request</a>.
The problem with standard KV caching is that each request gets its own dedicated KV cache block. Tokens are appended sequentially forming a contiguous memory block. There is no sharing between sequences even if parts of their content are the same. This leads to memory waste due to fixed and unshared allocation and poor handling of varying prompt lengths across sequences.
Paged Attention draws inspiration from virtual memory paging in operating systems. It introduces a separation between logical KV cache blocks which is the abstract token layout that each sequence sees and physical KV cache blocks which is the actual memory pages storing keys and values that may be shared across sequences. Each token’s data is stored in small fixed size memory pages. Pages are dynamically allocated and mapped via a page table. Multiple logical sequences can share the same physical memory if their tokens match like a common prefix. This enables non contiguous memory access through logical to physical translation.
Think of it like OS virtual memory where different processes see the same memory contents using their own mappings.
During generation the attention mechanism gathers keys and values for a sequence by following its page table. Sequences with shared prefixes map to the same physical cache blocks. When a sequence diverges Paged Attention uses copy on write. The shared page is cloned. Only the diverging sequence gets the new page. Reference count for shared pages is decremented. This ensures efficient memory use while allowing flexibility in generation.
THe KV cache without pagedAttention is a rectangle with batch vs max seq length. a lot of space wasted in the rectangle, because users dont really use the seq length to its max. we wanted to improve upon this device memory issue. pagedAttention allocates blocks in GPU memory. so you first load your model and see how much space you have left, and then everything else is filled with memory blocks. when new sequence comes in, we allocate as many blocks it needs for the prompt, and slowly grow them as needed. The management of cache was kinda an old school OS problem in the hindsight. GPU memory fragmentation wastes memory and makes it difficult to increase batch size. So, Paged Attention simply divides the KV cache into fixed-size memory-aligned blocks (pages dont have memory between them), similar to virtual memory pages in operating systems and allocating pages reduces internal and external memory fragmentation.
Refer the paper</a>.
Multi-Head Latent Attention</a></h3>
Got introduced in Deepseek v2 (also used in v3). This literally avoids caching K, V altogether. A low-rank representation of K and V learned during training is cached instead (LoRA like). This gives us much less KV cache use (90%+ savings). Also, as metrix size is also reduced, a good 5-6x inference speedup is there. And interestingly higher accuracy than MHA is achieved.
The key innovation is introducing learnable latent embeddings that act as intermediaries between queries keys and values. These latent embeddings capture high level abstract patterns and enable more efficient cross token interactions. Instead of attending to all input tokens the attention focuses on these latent embeddings leading to faster computation.
The mechanics work like this. A fixed number of latent embeddings are initialized as part of the model. Input queries attend to these latent embeddings instead of the entire sequence which drastically reduces the number of pairwise interactions. The latent embeddings then attend to the original keys and values acting as intermediaries that distill context into meaningful patterns. Finally the results get projected back to the token space preserving critical token level information.
Compared to MHA which suffers from quadratic complexity MLA achieves linear or near linear complexity by compressing the attention space. Compared to MQA which uses a single shared key value pair MLA maintains diversity by having latent embeddings act as a middle layer allowing richer context capture. Compared to GQA which focuses on groups and can miss global dependencies MLA’s latent embeddings inherently capture global patterns acting as global summaries.
The tradeoff is that compressing input tokens into latent embeddings may lose fine grained details critical for some tasks. The performance also depends on how effectively the latent embeddings are initialized and trained.
Linear Magic</a></h2>
Linear attention removes the dependency that troubles Softmax.
`# Feature map applied BEFORE the dot product q_mapped = φ(Q) # element-wise, no coupling k_mapped = φ(K) # element-wise, no coupling # Now we can re-associate output = q_mapped @ (k_mapped.T @ V) # O(n·d²)</code></pre> φ (like ELU+1) is applied element-wise - each element transforms independently. No normalization across the sequence. So you can legally reorder the multiplications`
Softmax attention: q_1 ──┬── score with k_1 ──┐ ├── score with k_2 ──┼── softmax needs ALL ── weight_1 ├── score with k_3 ──┤ weight_2 └── score with k_n ──┘ weight_n Linear attention: q_1 ── φ(q_1) ──┐ ├── can compute independently k_1 ── φ(k_1) ──┘</code></pre> With softmax, the denominator Σ exp(q·k_j) couples everything. Without it, each (q, k, v) triplet can be processed independently and accumulated into a running state. The coupling in softmax gave it expressiveness. the competition between keys (via normalization) lets the model express “attend to this, not that” patterns sharply. Linear attention loses this competitive dynamic, which is why it’s less expressive but more efficient.
Linear Attention</a></h3>
As softmax applied nonlinearity after Q·K product, coupling every query to every key, we have O(n^2^), that is full N×N attention matrix before applying softmax, as I explained earlier. Linear attention trades expressiveness for efficiency. Softmax attention can express arbitrary attention patterns, any token can attend strongly to any other token. Linear attention’s feature map constrains what patterns are representable. This is why modern hybrids (Kimi K3, Qwen3) use both! softmax layers for complex reasoning, linear layers for efficient long-range propagation.
Linear attention applies a feature map φ (such as ELU+1, or simply ReLU) to Q and K separately, before the dot product:
`Softmax: Attention = softmax(QK^T / √d) V # must compute N×N first Linear: Attention = φ(Q)(φ(K)^T V) # associativity trick</code></pre> I had an obvious question, why ELU+1 is used as the feature map and how linear attention still preserves the attention contract despite removing softmax? Both softmax and linear attention preserve the same fundamental contract, they just implement it differently:`

Make QK scores non-negative - Attention weights can’t be negative (obviously). Softmax uses exp(x) which is always positive. Linear attention uses ELU+1, since ELU(x) ≥ -1 for all x, adding 1 guarantees non-negativity.</li>
Normalize by dividing by sum - We still divide by Σ φ(q)·φ(k_j) to get proper weights. This is often omitted from diagrams but it’s there.</li>
Compute weighted average of values - Same as softmax</li> </ol>
Also, ELU(x) is smooth unlike ReLU which helps gradients. It approximates the exponential kernel that softmax uses. Softmax’s exp() function creates a sharper distribution, small differences in scores become large differences in weights. ELU+1 is flatter, so the attention distribution is more diffuse. The model can’t say “attend ONLY to this token” as sharply. This is interesting kernel trick.
Its the game of association. (AB)C = A(BC) from your Linear Algebra class. With softmax, you’re forced into (QK^T^)V because softmax breaks associativity. Without softmax, you can compute (K^T^ V) first, that’s a (d×N) × (N×d) = d×d matrix, independent of sequence length. Then multiply by Q. This means the growing history of K and V vectors can be folded into a fixed D×D state matrix S:
# Recurrent form of linear attention S = 0 # d×d state matrix for t in range(seq_len): S = S + φ(k_t).outer(v_t) # update state: O(d²) o_t = φ(q_t) @ S # query state: O(d²)</code></pre> You get it right? Constant memory, constant compute per token. No KV cache that grows with context.
 # Linear attention equivalent def forward_linear(self, x, state=None): b, t, d = x.shape d_head = d // self.num_heads h = self.num_heads qkv = self.qkv_proj(x) q = qkv[:, :, :d].view(b, t, h, d_head).transpose(1, 2) k = qkv[:, :, d:2d].view(b, t, h, d_head).transpose(1, 2) v = qkv[:, :, 2d:].view(b, t, h, d_head).transpose(1, 2) # Apply feature map (ELU+1 is common) q = F.elu(q) + 1 k = F.elu(k) + 1 if state is None: state = torch.zeros(b, h, d_head, d_head, device=x.device) # For each position, update state and compute output outputs = [] for i in range(t): # S += k_t ⊗ v_t (outer product update) state = state + torch.einsum('bhd,bhe->bhde', k[:, :, i], v[:, :, i]) # o_t = q_t @ S o_t = torch.einsum('bhd,bhde->bhe', q[:, :, i], state) outputs.append(o_t) o = torch.stack(outputs, dim=2) # b, h, t, d o = o.transpose(1, 2).contiguous().view(b, t, d) return self.o_proj(o), state # state is fixed d×d, not growing KV cache</code></pre>

Notice the difference! softmax attention returns (k, v) that grow with sequence length. Linear attention returns a fixed-size state matrix. </blockquote>
If you see original paper</a>, they say `the cost per time-step for transformers scales with the square of the current sequence length</code> which might trip you up! Today we know Flash Attention makes softmax attention practical. Well the paper was released in 2020 (a different world altogether).`
# 2020-era: recompute everything, no KV cache def generate_token(model, all_previous_tokens): # Re-run full forward pass on ALL tokens q, k, v = model.qkv_proj(all_previous_tokens) # O(t × d) # Materialize full t×t attention matrix attn_matrix = q @ k.T # O(t² × d) compute, O(t²) memory attn_matrix = softmax(attn_matrix) output = attn_matrix @ v return output[-1] # only need last token's output</code></pre> Per-token decode cost was O(t²) without cache in 2020 which became O(t) with cache, and full sequence generation was O(N³) which now is O(N²). Rememeber, algorithmic complexity is not implementation complexity.

Linear Attention has no recoverability guarantee. Query k₅, get a mixture influenced by all tokens with similar keys. </blockquote>
DeltaNet</a></h3>
Look at Linear Attention State update :

S = S + φ(k_t).outer(v_t) # additive update </blockquote>
Softmax Attention (KV Cache) :
`Cache = [k₁, k₂, k₃, ..., kₙ] # each token gets its own slot [v₁, v₂, v₃, ..., vₙ] # perfect isolation, O(N) memory</code></pre> Linear attention (state matrix):`
`S = k₁⊗v₁ + k₂⊗v₂ + k₃⊗v₃ + ... + kₙ⊗vₙ # all compressed into D×D</code></pre> When you query with q_t, softmax can retrieve v₅ in isolation by attending only to k₅. Linear attention retrieves:`o_t = q_t @ S = q_t @ (k₁⊗v₁ + k₂⊗v₂ + ... + kₙ⊗vₙ)</code></pre> The information from all previous tokens is superimposed. If k₃ and k₇ are similar, their values interfere, you can’t cleanly separate them. This is called retrieval interference or memory interference. The D×D state matrix has finite capacity. With N tokens compressed into D² slots, information must overlap when N > D². Even before that limit, similar keys cause interference. DeltaNet addresses this by using the delta rule from associative memory literature, instead of pure addition. the insightful thing here is, instead of blindly adding k⊗v to the state, subtract what’s already there for that key first: # Linear attention (naive additive) S = S + k_t ⊗ v_t # DeltaNet (delta rule) S = S + k_t ⊗ (v_t - S @ k_t) # ↑ this is the "delta" - the error/correction</code></pre> The term (v_t - S @ k_t) is the delta, the difference between what we want to store (v_t) and what we’d currently retrieve for this key (S @ k_t). Why this helps: If k_t is similar to a previous key, S @ k_t already contains that value</li> We only add the correction, not the full value</li> This reduces interference from similar keys</li> It’s inspired by Hopfield networks and fast weight programmers</li> </ul> This has better recoverability. The delta rule acts like an error correcting update, it tries to make S @ k = v hold for each stored (k, v) pair </blockquote> As we see in Fast Weight Programmers</a>: “when the sequence length exceeds storage capacity, the model may end up in an overcapacity regime. To properly operate under such a regime, the model should learn to dynamically interact with the memory contents and selectively decide which key-value associations to keep and which ones to delete. The purely additive instruction may be inappropriate for this purpose… endlessly adding new associations to a memory of finite size, as in Eq. 17, inevitably will reach a limit.” </blockquote> The regime that makes linear attention attractive (N >> D) also exposes its main limitation. Once the state exceeds its effective capacity, associations begin to interfere because the update is purely additive, nothing ever leaves the cache. def forward(self, x, mask=None, cache=None): b, t, d = x.shape d_head = d // self.num_heads h = self.num_heads qkv = self.qkv_proj(x) q = qkv[:, :, :d].view(b, t, h, d_head).transpose(1, 2) k = qkv[:, :, d:2d].view(b, t, h, d_head).transpose(1, 2) v = qkv[:, :, 2d:].view(b, t, h, d_head).transpose(1, 2) q = F.normalize(F.silu(q), dim=-1) k = F.normalize(F.silu(k), dim=-1) beta = torch.sigmoid(self.w_beta(x)).view(b, 1, t, 1) # per-token write strength S = cache if cache is not None else 0.0 v_old = k @ S # read what's currently stored at this key u = beta (v - v_old) # the delta: only what's actually new S = S + k.transpose(-1, -2) @ u # write the correction o = q @ S # read with query o = o.transpose(1, 2).contiguous().view(b, t, d) return self.o_proj(o), S</code></pre> The update first asks what information the current key retrieves from the cache (v_old = k @ S). It subtracts that existing information from the value we want to store, multiplies by beta (write strength), and adds the result back. Old information is removed and new information is written in its place. Gated DeltaNet</a></h3> Gated DeltaNet</a> adds gating to control how much of the old state to retain vs. overwrite: # Gated DeltaNet β = sigmoid(gate_proj(x)) # forget gate, per-token S = β S + k_t ⊗ (v_t - β S @ k_t)</code></pre> The gate β controls: β ≈ 1 - retain most of old state (long-term memory)</li> β ≈ 0 - forget old state (fresh start)</li> </ul> This is analogous to LSTM’s forget gate. it lets the model learn when to forget rather than accumulating everything forever. The model can now selectively decide which associations to keep and which to delete, addressing the overcapacity problem Schlag identified. RetNet</a></h3> While DeltaNet was figuring out the delta rule, Microsoft was working on a different angle. RetNet</a> (Retentive Network) came out in 2023 with a bold claim, they literally called it a successor to Transformer. The core idea is the retention mechanism, which is basically linear attention with exponential decay. Instead of the ELU+1 feature map, RetNet uses a decay factor γ that makes older tokens contribute less: # RetNet retention (simplified) S_t = γ S_{t-1} + k_t ⊗ v_t o_t = q_t @ S_t</code></pre> The γ (typically 0.9-0.99) acts like a built-in forgetting mechanism. Older associations naturally fade away, which addresses the overcapacity problem without needing the delta rule’s explicit correction. What made RetNet interesting was the three computation paradigms: Parallel mode: For training, unfold the recurrence into a matrix form (like standard attention but with a decay mask)</li> Recurrent mode: For inference, O(1) per token like linear attention</li> Chunkwise mode: Hybrid for long sequences - parallel within chunks, recurrent across chunks</li> </ul> Sound familiar? This is exactly the chunking trick that DeltaNet later formalized. RetNet got there first, though with a simpler (non-delta) update rule. In practice, RetNet showed up in Microsoft’s TorchScale library and influenced later work like YOCO (You Only Cache Once). The Gated RetNet variant added gating similar to Gated DeltaNet. But RetNet never quite took off in production the way the authors hoped, the successor to Transformer claim was kinda premature. Still, the multi-paradigm formulation was influential. RWKV</a></h3> RWKV</a> is the weird one. It’s not quite attention, not quite an RNN, but somehow both. The name comes from its four main parameters, Receptance, Weight, Key, Value. The project started as a community effort by BlinkDL and grew into something surprisingly capable. RWKV-4 was the first version that really worked, scaling up to 14B parameters - the largest dense RNN ever trained at the time. The core mechanism is time-mixing with learned decay: # RWKV time-mixing (simplified) wkv_t = Σ_{i=1}^{t-1} e^{-(t-1-i)w + k_i} v_i + e^{u + k_t} v_t o_t = sigmoid(r_t) wkv_t</code></pre> The w</code> is a learned decay (like RetNet’s γ but in log-space), u</code> is a bonus for the current token, and r</code> (receptance) gates the output. It’s inspired by Apple’s AFT (Attention Free Transformer) but with crucial modifications that make it actually trainable. What’s cool about RWKV: 100% attention-free: No QK^T computation at all</li> Parallelizable training: Can be formulated as a convolution</li> RNN inference: O(1) memory and compute per token</li> Actually deployed: There’s a whole ecosystem RWKV-Runner for local inference, rwkvserve for production APIs</li> </ul> The architecture kept evolving: RWKV-5 “Eagle” and RWKV-6 “Finch”: Added matrix-valued states and dynamic recurrence</li> RWKV-7 “Goose”: Incorporated a generalized delta rule (yes, the same delta rule from DeltaNet!)</li> </ul> From the RWKV-7 paper</a>: “RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates” </blockquote> So the field is converging, RWKV started from RNNs and added delta-rule-like updates, while DeltaNet started from linear attention and added gating. They’re meeting in the middle. RWKV is actually used in production. The community has trained models up to 14B parameters, there are multilingual variants, music generation models, and even edge deployment with quantization. It’s one of the few non-Transformer architectures that has a real user base beyond research papers. The Prefill Problem</a></h4> All the code I’ve shown so far has a dirty secret - it’s sequential. Look at the linear attention loop: for i in range(t): S = S + k[:, :, i] @ v[:, :, i] # can't parallelize - each step depends on previous S o = q[:, :, i] @ S</code></pre> Each step depends on the previous state. GPUs hate this. They want big parallel matrix multiplies, not tiny sequential ones. This is why naive linear attention is slower than Flash Attention in practice despite being O(n) vs O(n²). The DeltaNet</a> paper solves this with chunking. Split the sequence into chunks of size C, then within each chunk, do normal quadratic attention (parallel, GPU-friendly) and across chunks, use the recurrent state update (sequential, but only T/C steps instead of T) S = torch.zeros(b, h, dh, dh) if cache is None else cache outs = [] for i in range(t // C): q_c = q[:, :, iC:(i+1)C] k_c = k[:, :, iC:(i+1)C] v_c = v[:, :, iC:(i+1)C] # contribution from all previous chunks (recurrent) o_prev = q_c @ S # contribution from within this chunk (parallel attention) attn = (q_c @ k_c.transpose(-1, -2)).tril() # masked o_curr = attn @ v_c o = o_prev + o_curr # update state for next chunk S = S + k_c.transpose(-1, -2) @ v_c outs.append(o) o = torch.cat(outs, dim=2)</code></pre> The cost splits into two parts: Fixed: 2Ld² for state updates (doesn’t depend on C)</li> Growing: 2LCd for the within-chunk attention matrices</li> </ul> Setting C=N recovers full O(N²) attention. Setting C=1 gives pure sequential linear attention. In practice, C=64 or C=128 works well because that’s the granularity where tensor cores (UMMA instructions) operate efficiently. Chunking DeltaNet is Harder</a></h4> The chunking trick doesn’t directly work for DeltaNet because the delta correction needs the current state: v_old = k_i @ S # need S at this exact moment u_i = beta (v - v_old)</code></pre> You can’t batch this naively. The paper’s solution is a mathematical reparameterization that rewrites the delta update as a matrix recurrence with Householder-like transition matrices. This allows computing all C deltas within a chunk simultaneously using a forward substitution trick. I won’t pretend I understood this tbh. It took me hours to grasp a little. The key insight is that the sequential dependency can be “unrolled” into a form where you solve a triangular system once per chunk, then everything else parallelizes. The full chunked DeltaNet forward pass looks something like: def chunk_delta_rule_forward(Q, K, V, beta, C): L, d = Q.shape Q, K, V = map(lambda x: x.reshape(-1, C, d), [Q, K, V]) beta = beta.reshape(-1, C) K_beta = K beta.unsqueeze(-1) V_beta = V beta.unsqueeze(-1) # Forward substitution for the correction terms T = -(K_beta @ K.t()).tril(-1) for i in range(1, C): T[i, :i] = T[i, :i] + (T[i, :, None] T[:, :i]).sum(-2) T += torch.eye(C) W = T @ K_beta U = T @ V_beta # Chunked parallel computation S = torch.zeros(d, d) O = torch.empty_like(V) for i in range(L // C): q_i, k_i, w_i = Q[i], K[i], W[i] u_i = U[i] - w_i @ S # corrections for this chunk o_inter = q_i @ S # cross-chunk contribution A_i = (q_i @ k_i.t()).tril() o_intra = A_i @ u_i # within-chunk contribution S += k_i.t() @ u_i # update state O[i] = o_intra + o_inter return O.reshape(L, d)</code></pre> The forward substitution (the T matrix computation) is the magic that makes this work. It precomputes how each position’s delta affects subsequent positions within the chunk. Kimi Delta Attention (KDA)</a></h4> The core innovation is Kimi Delta Attention (KDA), which extends Gated DeltaNet with fine-grained gating. Instead of a single scalar decay β, KDA learns a separate decay value for each channel: # Gated DeltaNet: single scalar gate β = sigmoid(gate_proj(x)) # shape: (batch, seq, 1) S = β S + k_t ⊗ (v_t - β S @ k_t) # KDA: per-channel gate (fine-grained) α = sigmoid(alpha_proj(x)) # shape: (batch, seq, d_head) # each dimension of the state decays at its own rate</code></pre> The diagram shows the full KDA block. Q and K go through L2 normalization and convolutions. V goes through a convolution as well. The α and β parameters come from separate linear projections with sigmoid gates. Everything feeds into the Gated Delta Rule which is the core state update. The output goes through RMSNorm and a final linear projection. The 3:1 ratio at the bottom shows how layers are arranged in Kimi Linear and K3. Three KDA layers followed by one Gated Multi-head Latent Attention layer then repeat. Why does this matter? The D×D state matrix has limited capacity. With a single scalar decay, all channels forget at the same rate. With per-channel decay, the model can: Keep some channels as long-term memory (high α)</li> Use other channels as short-term scratch space (low α)</li> Learn which information needs persistence vs. which can be overwritten</li> </ul> Kimi Linear</a></h2> This was Moonshot AI’s approach before K3. The idea was simple, if softmax attention is expensive at long contexts and linear attention loses expressiveness, why not use both? Kimi Linear</a> made a bold claim : “We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios—including short-context, long-context, and reinforcement learning (RL) scaling regimes.” </blockquote> They used a hybrid architecture: Early layers: Linear attention (cheap, handles long-range dependencies)</li> Later layers: Softmax attention (expensive, but more expressive for final reasoning) The intuition is that early layers do broad gathering of information across the context, while later layers do precise reasoning that benefits from softmax’s sharp attention patterns.</li> </ul> Kimi Linear doesn’t use KDA alone. It interleaves KDA layers with Multi-Head Latent Attention (MLA) layers in a 3:1 ratio. The obvious question that would come to anyones mind is why MLA? MLA (from DeepSeek) is a softmax-based attention that uses low-rank projections to reduce KV cache size. By mixing KDA (linear, O(1) state) with MLA (softmax, but compressed), Kimi Linear gets: 75% reduction in KV cache compared to full MLA</li> Up to 6× decoding throughput at 1M context length</li> Better quality than pure MLA on benchmarks</li> </ul> The paper also replaces the standard MLP with Mixture-of-Experts (MoE)</a>, but that’s orthogonal to the attention innovation. The MoE Landscape</a></h3> Speaking of MoE, here’s where things get interesting. Kimi K3 isn’t just big, it’s efficiently big. The activation percentage tells you how much of the model actually runs per token: Kimi K3: 1.8% activation (104B of 2.8T parameters)</li> MiniMax M3: 3.1% activation</li> Inkling (Thinking Machines): 3.1% activation</li> Nemotron 3 Ultra (NVIDIA): 4.3% activation</li> </ul> K3 is currently the biggest open MoE model, but what’s more impressive is the small activation ratio. Lower activation means faster inference and lower memory bandwidth requirements. The 1.8% figure means K3 activates roughly half the proportion of parameters that competitors do, while still achieving competitive quality. The scaling from K2 to K3 tells the story: K2 had 1T parameters with 384 experts, activating 8 per token (2% activation). K3 scaled to 2.8T with 896 experts, activating 16 per token. The math: they nearly tripled the total parameters but only doubled the activated parameters. More experts, sparser activation, better efficiency. This is the MoE scaling playbook - grow the expert pool faster than you grow the activation budget. This worked reasonably well but had a problem: the transition between linear and softmax layers created a representation mismatch. Information compressed into the D×D state matrix had to be “unpacked” for softmax layers to use effectively. Previous hybrid approaches (like the early Kimi experiments) stacked linear and softmax layers in separate sections. Kimi Linear interleaves them throughout, which helps maintain representation compatibility. The 3:1 ratio was determined empirically, enough KDA for efficiency, enough MLA for expressiveness. Kimi K3</a></h2> K3 scales the Kimi Linear architecture to 2.8 trillion parameters with 104 billion activated (it’s an MoE model). To put that in perspective: one K3 contains roughly 22,580 GPT-2 models worth of parameters. But the interesting part isn’t the scale. It’s what they scaled. K3 uses the same KDA + MLA hybrid from Kimi Linear, but with several additions: The diagram shows the full picture. On the left, the overall architecture: 93 transformer blocks with the 3:1 KDA/MLA ratio (layers 1-3 use KDA, layer 4 uses MLA, repeat). Layer 1 is dense, layers 2-93 use LatentMoE. On the right, the two attention mechanisms side by side - Gated MLA (softmax-based with latent compression) and KDA (the delta rule with per-channel gating via α and β). Native Multimodality</a></h3> K3 is natively multimodal - vision is built into the architecture from the start, not bolted on. The linear attention layers handle the massive context that images require (a single high-res image can be thousands of tokens), while MLA layers do the cross-modal reasoning Stable LatentMoE</a></h3> Training a 2.8T parameter model is hard. K3 uses a Stable LatentMoE framework that combines: The MuonClip optimizer (from Kimi K2</a>) for training stability</li> Careful expert routing to prevent collapse</li> Latent representations that compress the expert outputs</li> </ul> 1M+ Context Window</a></h3> The combination of KDA’s O(1) state and MLA’s compressed KV cache enables a 1 million token context window that’s actually usable in practice. At 1M tokens, full softmax attention would require ~1TB of KV cache memory. K3’s hybrid approach reduces this to something that fits on a single node The Progression</a></h3> Looking at the evolution: Model</th> Attention</th> Key Innovation</th></tr></thead> Linear Attention</td> Additive state</td> O(1) memory, but interference</td></tr> DeltaNet</td> Delta rule</td> Better recoverability</td></tr> Gated DeltaNet</td> Scalar gate</td> Selective forgetting</td></tr> KDA (Kimi Linear)</td> Per-channel gate</td> Fine-grained memory control</td></tr> K3</td> KDA + MLA hybrid</td> Scale + multimodality</td></tr> </tbody></table> Each step adds capacity to address a concrete limitation in the preceding system. This isn’t “make it bigger and hope” - it’s targeted architectural improvements that compound. What happens after for text generation?</a></h2> Yeah, so the model actually did the job. The job of the model is to give out out probabilities. That’s it. Generally, GPT architectures and all sorts of modern LLM architectures are decode-only, so there is no real encoder. The inputs are the prompts, and it simply generates the probabilities. It has nothing to do with text generation. After that, we have to pick those tokens that the model has generated. The attention outputs are for the input sequence that we have, that is the prefill that we have done, first of all, stored in the KV cache</li> then we retrieve the attention output for the last token in the input sequence</li> Then, to get the output weights, we make it go through a linear layer (that is, a projection layer). After which we just simply multiply the transpose of the output weights to the attention output, and we get the logits. Then we take a softmax of those logits.</li> only after this do we decode the token, and decoding can happen in multiple ways:</li> </ul> We can do it greedily, so just picking up the token with the highest probability.</li> We can do some sort of sampling, so any top-k sampling in which we can pick the token from the k most likely tokens.</li> Any top-p decoding as well, in which we pick a token from the smallest subset of tokens such that their cumulative probability exceeds the p threshold. A fancier way of making the models’ output more creative</li> </ul> And at last, we simply use the new token as the next input</li> </ul> hybrid attention models need their kernels</a></h2> The code example I have shared shows the naive sequential loop for clarity. In practice, you’d use chunked parallel computation (which is what flash linear attention kernels do). The sequential form is pedagogically useful but would be slow without proper kernelization. K3 demonstrates that linear attention variants can scale to frontier model sizes when combined thoughtfully with softmax attention. The hybrid approach isn’t a compromise, it’s genuinely better than either pure approach for long-context workloads. The catch? You need the kernels. All of this efficiency is theoretical without optimized CUDA implementations. Moonshot open-sourced their KDA kernel and vLLM integration, which is why Kimi Linear actually achieves the claimed speedups in practice. To make this clearer, a practical titbit. Qwen3.5-2B uses a hybrid architecture, some layers are full quadratic attention (standard softmax), some are linear attention (avoiding the n² computation). Sounds great for efficiency. But without the flash-linear-attention CUDA kernels installed, the linear attention layers will fall back to a naive sequential torch loop, processing tokens one by one instead of in the efficient chunked/parallel form. The result is fuked up 5-6x worse speed loss. The linear attention layers are theoretically O(n) instead of O(n²). But the naive implementation is worse than a well-optimized O(n²) Flash Attention because Flash Attention’s tiled memory access pattern is so cache-friendly that it beats an algorithmic advantage destroyed by poor memory access patterns. algorithmic complexity means nothing without implementation quality. A well-kernelized O(n²) beats a poorly-implemented O(n) every time on real hardware. This is why Flash Attention dominates! not because quadratic is somehow better, but because Tri Dao spent years making the memory access pattern perfect for GPU cache hierarchies</a>. Conclusion</a></h2> Softmax → Linear (efficiency, but interference) → DeltaNet (correction) → Gated (forgetting) → KDA (fine-grained) → Hybrids (best of both) missing writing some concurrency in Go Unknown — Thu, 09 Jul 2026 00:00:00 +0000 It has been a long since I wrote Go. I still remember doing FOSS for CNCF folks and was deep in Go, until I got distracted by Rust ;) I just saw an awesome Go concurrency visualization</a>. So as I got some free time today, I am gonna write about some concurrency patterns in Go I saw being used in prod. Background</a></h2> When Communicating Sequential Processes</a> came, the main thing it did was making communication the foundation of concurrent programming. So, if you want to communicate with 2 processes, you should always do it using fixed defined channel, instaed of directly sharing memory. It was one of the core ideas of the paper. There were many languages born out of the paper. Go’s concurrency model is influenced by Communicating Sequential Processes</a>, where independently executing processes coordinate through communication. Go makes channels first-class values</a>, although channels are not the only synchronization mechanism available. mutexes and atomics are also valid tools. goroutine goroutine │ │ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Process P1 │ │ Channel C1 │ │ Process P2 │ │ Channel C2 │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │── Sends data ────>│ │ │ │ │ │ │ │ │── Passes data ───>│ │ │ │ │ │ │ │ │── Sends data ────>│ │ │ │ │ │<───────────────── Passes data back ───│ │ │ │ │ │ ┌──────┴───────┐ ┌──────┴───────┐ ┌──────┴───────┐ ┌──────┴───────┘ │ Process P1 │ │ Channel C1 │ │ Process P2 │ │ Channel C2 │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘</code></pre>Some basics</a></h2> There is an overhead in creation/destruction of OS threads (system calls and memory allocation) which are preemptive. They are typically ~1-2MB stack size, have expensive context switches (saves/restores full CPU registers, kernel transition) handled by the OS kernel for small work. As we are not dealing with OS threads, instead goroutines. And goroutines (green threads are incredibly lightweight, starting at ~2-4KB of stack space</a>) are multiplexed into multiple OS threads (usually equal to the number of CPU cores). Because the Go scheduler manages goroutines in user space, switching between them is significantly faster than a kernel-level OS thread context switch. Spawning a million goroutines is cheap relative to OS threads, very large numbers still consume stack and scheduler resources and can retain heap objects. Traditional OS threads have a fixed stack size. If a thread runs out of stack space, it causes a stack overflow. To prevent this, languages like Java allocate large stacks (e.g., 1MB) upfront, which limits the total number of concurrent threads you can have (usually in the thousands). Goroutines solve this with a growable/shrinkable segmented stack: They start very small (typically 2KB to 4KB).</li> If a goroutine needs more memory during execution, the Go runtime automatically allocates a larger segment and copies the old stack over, allowing a single program to run millions of concurrent goroutines simultaneously.</li> </ul> Go uses an M:N scheduler and its scheduler is implemented as 3 structs, G, M and P</a> (all heap allocated, but are never freed, so their memory remains type stable, so the runtime can avoid write barriers in the depths of the scheduler), wherein a goroutines exists only in the virtual space of go runtime and not in the OS : G (simply Goroutine): Represents the goroutine and its stack. When a goroutine exits, its G object is returned to a pool of free Gs and can later be reused for some other goroutine.</li> M (Machine): Represents an OS thread managed by the operating system, that can be executing user Go code, runtime code, a system call, or be idle. There can be any number of Ms at a time since any number of threads may be blocked in system calls.</li> P (Processor): Represents the logical resource (or context) required to execute Go code, such as scheduler and memory allocator state. The number of Ps usually equals the number of CPU cores (GOMAXPROCS).</li> </ul> The scheduler’s job is to match up a G (the code that we want to execute), an M (where we want to execute it), and a P (the rights and resources required to execute it). When an M stops executing user Go code, for example by entering a system call, it returns its P to the idle P pool. In order to resume executing user Go code, for example on return from a system call, it must acquire a P from the idle pool. Instead of the OS kernel switching between thousands of heavy threads, the Go runtime efficiently schedules active goroutines across a small pool of OS threads. If a goroutine blocks (e.g., waiting for I/O or a channel), its associated OS thread can be detached or assigned to run other ready goroutines, maximizing CPU utilization. If a goroutine blocks on system call, it blocks it’s running thread. But another thread is taken from the waiting queue of Scheduler and used for other runnable goroutines. However, if you communicate using channels in go which exists only in virtual space, the OS doesn’t block the thread. Such goroutines simply go in the waiting state and other runnable goroutine (from the M struct) is scheduled in it’s place. Don’t communicate by sharing memory, share memory by communicating. The go runtime scheduler also does cooperative scheduling, which means another goroutine will only be scheduled if the current one is blocking or done. This is so much better than pre-emptive scheduling which uses timely system interrupts to block and schedule a new thread as that may lead a task to take longer than needed to finish when number of threads increases, etc. Go through the paper</a> yourself. Worker Pools</a></h2> Now, worker pool allows us to reuse threads for multiple tasks.A worker pool creates a fixed set of worker goroutines. Its main purpose is to bound active work and resource consumption, it does not directly manage or reuse OS threads. to these we can push tasks. We can refer to it as thread pool as well. But, does this matter in Go? You do not really need a worker pool to save the system from the overhead of creating goroutines. So, this traditional pattern (often used in C++ or Java to avoid the heavy cost of spawning OS threads) is not strictly necessary for performance in Go in the same way it is in other languages. If we aren’t saving on thread creation costs, why use a worker pool? We use them as a resource management and throttling mechanism. Throttling</a></h3> Imagine you are building a service that scrapes 10,000 URLs. If you spawn 10,000 goroutines simultaneously, you will likely: Exhaust the file descriptors on your machine</li> Get banned by the target servers as you are making too many requests</li> Flood your database connections</li> </ul> A worker pool acts as a semaphore. It limits the number of tasks running in parallel to a fixed number, ensuring that even if you have 10,000 tasks, only 20 are active at any given moment. So, clearly there is a semaphore pattern coming! Controlling Resource Consumption</a></h3> Even though goroutines are cheap, the work they might not be. If each goroutine performs a memory-intensive task (like image processing or large JSON decoding or whatever), spawning too many simultaneously will lead to an OOM. A worker pool allows you to constrain memory footprint. Code Snippet</a></h3> // Task represents the work to be done. type Task int func main() { const numWorkers = 3 const numTasks = 10 // 1. Create the communication channel. // Buffered channels allow the producer to keep working even if workers // are slightly behind, up to the buffer limit. jobs := make(chan Task, numTasks) // 2. Initialize the WaitGroup to track active workers. var wg sync.WaitGroup // 3. Start the worker pool. for w := 1; w <= numWorkers; w++ { wg.Add(1) // Increment the counter for each worker started go worker(w, jobs, &wg) } // 4. Send tasks to the workers. for i := 1; i <= numTasks; i++ { jobs <- Task(i) } // 5. Close the channel to signal no more work is coming. // The range loop in the worker will terminate once the channel is // empty AND closed. close(jobs) // 6. Block until all workers have finished their tasks. wg.Wait() fmt.Println("All tasks complete. Main exiting.") } // worker is the function run by each goroutine. func worker(id int, jobs <-chan Task, wg sync.WaitGroup) { // Ensure the worker signals it is done when the function exits. defer wg.Done() // The 'range' loop will continue until the channel is closed. for task := range jobs { fmt.Printf("Worker %d started task %d\n", id, task) // Simulate heavy work time.Sleep(time.Millisecond 500) fmt.Printf("Worker %d finished task %d\n", id, task) } fmt.Printf("Worker %d shutting down\n", id) }</code></pre> close(jobs)</code>: This is the most important part of the pattern. Without it, the range loop in the workers will block forever, waiting for a new task that will never arrive (a deadlock/leak).</li> sync.WaitGroup</code>: This ensures your main function does not terminate before the workers have finished processing the last items in the queue.</li> Worker Loop: The pattern for task := range jobs is idiomatic Go. It cleanly handles the transition from active work to termination without requiring complex conditional checks.</li> </ul> This pattern is ideal for throttling. By changing numWorkers, you control exactly how many concurrent operations (API calls, file writes, database queries) are happening at once, preventing your system from consuming too many system resources or being rate-limited by external services. So, you use a worker pool in Go not to save the CPU from creating threads, but to protect your application, your database, or external APIs from being overwhelmed by your own concurrency. Thats why, its concurrency pattern. Whats leak?</a></h3> Because in worker pools we create goroutines once and reuse it, idle workers can become problematic. An idle worker is normal while its pool is expected to remain alive. It becomes a leak when the worker should have terminated but remains permanently blocked because its shutdown or cancellation path is broken. If your code leaks goroutines, meaning you have workers hanging around waiting for tasks that will never come, your application doesn’t break in a loud, obvious way. Instead, it slowly, quietly begins to wither. No standard memory dump is gonna save your ass. And this leak will slowly compound and make your life hell. Imagine you have a pool of 100 workers designed to process incoming web requests. Due to a bug in your shutdown logic, every time you deploy a new version of your service or trigger a specific event, 5 of those workers fail to exit. They don’t do anything. They aren’t consuming CPU cycles, so your monitoring dashboards stay green. They are just… sitting there. They are holding onto 2KB of stack memory each. To a modern server with gigabytes of RAM, 10KB of wasted memory is nothing really. But a month later? You have thousands of these zombie goroutines. Now, your memory usage has climbed by several hundred megabytes. Your garbage collector starts working harder to track these thousands of objects. Your application gets slower, more sluggish, and eventually, the OOM killer arrives and restarts your container without warning. Dont start panicking and looking at memory dump if this happens, use tools like pprof to visualize the goroutine blocking profile. It will show you exactly where your workers are stuck, and more importantly, it will show you that they have been sitting on that channel operation for hours, which is your indicator that they are zombies. Semaphore i.e Bounded Concurrency</a></h2> In typical usecases, we want to achieve the same throttle effect as a worker pool, but without the lifecycle management overhead (you don’t need to close channels or manage worker exit). So for that, we can use a buffered channel as a semaphore. The size of the buffer acts as max concurrency limit. There is no issue of idle workers here as we spawn goroutines for every task (this is a tradeoff), and they exit completely when done and the memory is reclaimed by GC. Its quite clean. If your application processes a constant, high-volume stream of tasks, a Worker Pool is more efficient because it avoids the minor overhead of constant goroutine creation. If your application handles periodic, bursty, or ad-hoc tasks, the Semaphore Pattern is significantly cleaner, easier to maintain, and eliminates the risk of goroutine leaks. In a traditional Worker Pool, if a worker deadlocks, you have a fixed number of stuck workers so your throughput drops to zero whihc is quite predictable and easy to monitor. In the Semaphore Pattern (where you spawn goroutines dynamically), a deadlock can be much more dangerous. New tasks will stop spawning and capacity, can lead to a complete system hang. So, never perform a task (especially one involving I/O, network, or DB calls) without a context.WithTimeout</code>. If a task hangs, the context will cancel, the execute function should return, and the defer will release the semaphore. You can also monitor semaphone depth, since the channel is a physical object in your memory. If len(semaphore)</code> stays at maxCapacity</code> for an unusually long time, you know your workers are deadlocked. You can expose this as a metric (e.g., Prometheus) to alert you that your in-flight tasks are stuck. If you seriously want to do an x-ray : Run go tool pprof http://localhost:PORT/debug/pprof/goroutine</code></li> Look for goroutines stuck in execute</li> Because you aren’t using a long-lived pool, you will see a bunch of identical goroutines all stuck at the exact same line of code in execute. If you see 100 goroutines all waiting on the same network call, you have found your deadlock immediately</li> </ul> Things in action</a></h2> In production, your service isn’t just processing a fixed list of tasks, it’s likely reading from an endless stream (like a Kafka topic). You want to process these messages in parallel to maintain high throughput, but you need to ensure that when your service receives a termination signal (like a Kubernetes SIGTERM), it doesn’t just kill the application and lose the message currently being processed. // Consumer manages the ingestion and processing flow func Consumer(ctx context.Context, workerCount int) { // 1. Channel to buffer messages between the reader and workers queueChan := make(chan string, 100) var wg sync.WaitGroup // 2. Spawn worker pool for i := 1; i <= workerCount; i++ { wg.Add(1) go func(id int) { defer wg.Done() // Workers process until the channel is closed for msg := range queueChan { fmt.Printf("Worker %d processing: %s\n", id, msg) time.Sleep(500 time.Millisecond) // Simulate work } fmt.Printf("Worker %d stopped.\n", id) }(i) } // 3. Main consumption loop // This loop pulls from the "stream" (Kafka/Queue) go func() { defer close(queueChan) // Closing allows workers to finish remaining tasks and exit for { select { case <-ctx.Done(): // Triggered by OS signal or shutdown request fmt.Println("Shutdown signal received, stopping ingestion...") return default: // Simulate reading from Kafka msg := "kafka-message-" + time.Now().Format("15:04:05") // Attempt to send to queue; blocks if buffer is full (Backpressure) select { case queueChan <- msg: case <-ctx.Done(): return } } } }() // 4. Graceful Shutdown Wait // Wait here until the context triggers and the channel is closed <-ctx.Done() wg.Wait() // Wait for all workers to finish processing current items fmt.Println("All workers finished. Clean shutdown complete.") }</code></pre> Interesting things in code : Cool Backpressure Management! The queueChan is buffered (e.g., make(chan string, 100)). So, if your workers are slower than the Kafka reader, the channel will fill up. Once full, the queueChan <- msg</code> operation will block the producer. This forces your Kafka consumer to wait, effectively applying backpressure to the upstream system. This prevents your service from running out of memory by hoarding too many messages in the RAM.</li> Graceful Degradation. In production, you never want to kill a process abruptly for sure. so when service receives a SIGTERM from k8s, ctx.Done()</code> signals the reader stops pulling new messages from Kafka, close(queueChan)</code> sends a stop signal to all workers and the hero wg.Wait()</code>, the main function waits for the WaitGroup. Because the channel is closed, the range loop in each worker will naturally finish processing the remaining items in the buffer and then exit gracefully.</li> </ul> In a real production app, you would add an errChan</code> to catch failures from execute(task). If a worker encounters a database error, it shouldn’t just die! it should log the error and perhaps send the message to a Dead Letter Queue (DLQ) so you can replay it later without crashing the entire consumer. Pipelines</a></h2> Where Worker Pools are about parallelism (doing the same task N times at once), Pipelines are about composition (doing a sequence of different tasks in stages). Each ETL stage is a separate goroutine, and they are connected by channels. Each stage has its own in-channel and out-channel. // Stage 1: Generates work func generator(done <-chan struct{}, nums ...int) <-chan int { out := make(chan int) go func() { defer close(out) for _, n := range nums { select { case out <- n: case <-done: return } } }() return out } // Stage 2: Processes work func sq(done <-chan struct{}, in <-chan int) <-chan int { out := make(chan int) go func() { defer close(out) for n := range in { select { case out <- n n: case <-done: return } } }() return out }</code></pre> This is quite a gold standard in prod. Its modular. You can test Stage 1, Stage 2, and Stage 3 independently</li> The gift of independent Scaling. You can have 1 “Extract” goroutine, 10 “Transform” goroutines (a worker pool within a stage), and 2 “Load” goroutines. You match the resource usage to the bottleneck of each specific stage.</li> Pipelining gives True Parallelism. Because stages are connected by channels, while Stage 3 is saving message A to the database, Stage 2 can be transforming message B, and Stage 1 can be fetching message C. It keeps your CPU cores busy.</li> </ul> Fan out and in!</a></h2> In production, you often need to Fan-Out (start multiple workers for a slow transformation stage i.e distribute tasks to multiple goroutines for parallel processing) and Fan-In (merge those multiple results back into one channel). This combines everything : Pipeline - The structure of stages</li> Worker Pool - Used within a stage to process data in parallel</li> Semaphore/Channel Throttling - Used to ensure the pipeline doesn’t consume more memory than allowed</li> </ul> Channels as first class primitives</a></h3> Because channels are values, you can pass them, store them, and send them over other channels. This allows you to treat channels as futures or promises. In standard concurrency, ordering is rarely guaranteed because of the Go scheduler is not deterministic. When you launch Goroutines, they complete in whatever order the CPU dictates. If you need order, you have to engineer it. Its a burning question! How do I process tasks in parallel to get speed, but output the results in the exact order they arrived? So, its awesome for ordered data streaming, but its quite fragile when managing complex state reconciliation (like k8s does) Technicalities</a></h3> The core of this pattern lies in decoupling the order of dispatch from the timing of execution. We achieve this by establishing a strict synchronization barrier using a Channel of Channels (a chan chan T</code>). First we establish sequence. The dispatcher is responsible for two distinct operations: Orchestration - It initializes a unique resultChannel for every incoming task. This channel acts as a private synchronization point for the specific task’s lifecycle.</li> Sequencing - It pushes the newly created resultChannel into the procQueueChan. Because procQueueChan is a buffered or unbuffered FIFO channel, the order in which channels are inserted is preserved. This is the Serialization Phase.</li> </ul> Each worker goroutine is decoupled from the sequencing logic. It receives its specific task and its private resultChannel. Because the runtime scheduler handles goroutine preemption and execution across M OS threads, the workers execute in parallel. The completion order is arbitrary, dependent on task complexity, I/O wait times, and scheduler state. Crucially, the worker is not responsible for ordering, it is only responsible for executing the logic and signaling completion by writing to its assigned resultChannel. Later we reestablish order. The Aggregator stage iterates over procQueueChan. Technically, it performs two levels of indirection: result := <-(<-procQueueChan)</code>. Channel Acquisition - It blocks until it receives a channel from the queue. This is the Synchronization Barrier.</li> Result Retrieval - Once it acquires the resultChannel</code>, it immediately blocks on <- resultChannel</code>.</li> </ul> The Aggregator effectively waits for the workers in the precise order they were dispatched. If the first task (Task N) takes significantly longer to execute than Task N+1, the Aggregator will remain blocked at <- resultChannel_N</code>, even if the data for Task $N+1$ has already been computed and is sitting in its own resultChannel. This creates a deterministic output stream that mirrors the input stream’s sequence. There are thinfs to be concerned about, plenty of runtime constraints. There can be memory pressure if you push things to scale. You are creating N channel objects for $N$ tasks. While Go’s heap allocation for channels is efficient, in high-throughput systems (thousands of tasks per second), this increases GC pressure. Developers must tune GOGC or consider channel reuse patterns to mitigate heap fragmentation.</li> This pattern inherently suffers from Head-of-Line (HOL) blocking. If the first task in a sequence is computationally expensive or hangs, the entire pipeline’s output is blocked, regardless of how quickly subsequent tasks are completed.</li> Each worker goroutine must have a defer block that either writes to the resultChannel or closes it. Failure to send a result will cause the Aggregator to block indefinitely, leading to a permanent stall of the pipeline.</li> </ul> Studtying k8s a bit</a></h2> Ordered Parallelism is pretty foundational concurrency pattern in k8s controller (kinda implicit Fan-Out/Fan-In though). but K8s prefers Work Queues with internal locking mechanisms</a>, where key-based locking acts as the serialization barrier to ensure that parallel workers don’t corrupt the cluster state by processing events out of order. pattern manifests as : The Queue Interface: Instead of passing channels, K8s uses an interface that keeps track of the Key (e.g., namespace/pod-name).</li> Deduplication: In a pipeline, if you have multiple events for the same resource (e.g., Pod “A” created, Pod “A” updated), a raw Fan-Out might send these to different workers simultaneously. Kubernetes Work Queues use deduplication based on an object’s key. This ensures that only one worker processes a specific object at any given time, which is the only way to guarantee consistency in a state machine. The workqueue internally ensures that if the same key (the same Pod) is added to the queue multiple times while a worker is processing it, the queue deduplicates those events.</li> Serialization: By locking on the Key (the object identifier), K8s achieves the exact same goal as Channel of Channels! It ensures that only one worker is ever processing a specific object at any given time, preserving the order of operations for that object.</li> Failure Recovery: Raw channels cannot “retry” an event easily. Kubernetes Work Queues provide a Requeue mechanism where a controller can signal that it failed to process an object, allowing the queue to put it back for later retries.</li> </ul> the secret sauce seems to be rate limited queue : Kubernetes must guarantee that if it receives Create Pod and then Delete Pod for the same object, it processes them in that order. If it processes “Delete” before “Create,” the controller state would be permanently broken.</li> To handle shit be inserted: If a specific object causes a controller to panic or error consistently, you do not want to block the entire pipeline you knww</li> Backoff Logic: The RateLimitingQueue tracks how many times an object has failed. If it fails, it is not immediately re-queued, the queue forces a delay (e.g., 5ms, 10ms, 20ms…) before allowing it to be processed again. This prevents a faulty object from creating a hot loop that consumes 100% of the controller’s CPU.</li> </ul> So, in k8s, things are implicit : Implicit Fan-Out: The workqueue doesn’t use a Channel of Channels to distribute work, instead, you, as the developer, define N worker goroutines that all call queue.Get() on the same workqueue instance. This is the Worker Pool pattern. It effectively performs the Fan-Out by allowing multiple workers to pull from the central queue in parallel.</li> No Fan-In Required: Because the workers update the Kubernetes API Server directly (the Source of Truth), there is no need to merge results back into a single channel. The API server acts as the final aggregator for the system’s state.</li> </ul> ultimately, K8s is a distributed system that must achieve eventual consistency. This concurrency pattern is chosen for two specific reasons: Isolation: If a worker processing a Pod event panics or hangs, it doesn’t stop the workers processing Node or Service events. The queue ensures the controller stays responsive.</li> Ordered Reconciliation: The cluster state is essentially a giant fucking state machine. You cannot reconcile a Pod’s status if you don’t process the events that led to its current state in order obviosuly. This pattern gives K8s the ability to scale to thousands of Pods (parallelism) while keeping the API server state consistent (ordering).</li> </ul> But dont you like Rust?</a></h2> Yea. Its philosophical. You might not have a great first impression of it, but often the second impression of Rust is nice</a>. You know what happens when a goroutine panics while holding a standard Go sync.Mutex? Go begins unwinding the call stack for that specific goroutine, executing any defer statements along the way. If you followed idiomatic Go and placed defer mu.Unlock() immediately after locking, the deferred function will execute during stack unwinding, releasing the mutex so other goroutines don’t deadlock. Unless caught, the panic will travel up to the goroutine’s root, crash the entire Go program, and print a stack trace. You like this? the recover()</code> will only works if called directly inside a deferred function. There are isolation troubles as well. a panic in one goroutine does not automatically crash other goroutines, it only crashes the entire program if the panic happens on a goroutine that doesn’t recover it (with the exception of the main goroutine). If Worker A panics and recovers using defer + recover(), Worker B and Worker C keep running uninterrupted. However, because Go doesn’t poison mutexes, if Worker A left the shared data structure in a half-modified, broken state before panicking, Worker B will successfully acquire the lock and read that corrupted data without any warning from the runtime. I dont like this. Go trusts you. If a panic happens, the lock opens, but the data is unverified. Lack of Compile-Time Thread Safety - Unlike Rust with rigorous ownership models, Go’s compiler does not prevent data races at compile time. Standard types (like maps and slices) are unprotected. Concurrency errors, such as concurrent map writes or race conditions, are frequently discovered only at runtime—either via tests, the runtime race detector, or sudden 2AM production crashes.</li> Silent Data Corruption via Unpoisoned Mutexes - GGo has no concept of mutex poisoning. A Go mutex is completely agnostic to panics. If a goroutine panics, the mutex unlocks (assuming you used defer), but it remains completely valid and available for other goroutines to lock immediately. Go assumes that if you recover from the panic, the shared data is fine, or it’s your responsibility to clean up any corrupted state</li> Goroutine Leaks - Because spawning a goroutine is as simple as prefixing a function call with the go keyword, it is easy to spawn tasks that block indefinitely on unbuffered channels or missing context cancellations. These leaked goroutines persist in memory for the lifetime of the application, gradually degrading performance without an immediate compiler or runtime warning.</li> Deadlock Vulnerabilities - While Go’s runtime can detect simple deadlocks (e.g., all goroutines sleeping forever), complex deadlocks involving channels, select statements, and external resource locks often go undetected by the runtime. Debugging blocked channels or circular dependencies in large Go codebases is fucked up</li> Abstracted Scheduling Control - Go’s M:N scheduler handles goroutine multiplexing onto OS threads automatically. While convenient, this abstraction gives developers very little direct control over OS thread pinning, CPU affinity, or low-level thread scheduling priorities, which can be a limitation for ultra-low-latency systems</li> </ul> Now that I started discussing about MUtex Poisoning</a></h3> If a thread panics while holding a standard standard library Mutex (typically wrapped in an Arc<Mutex>), Rust does not leave the mutex open and silent. When the thread panics, its stack unwinds, and the RAII MutexGuard goes out of scope and is dropped. During this drop, Rust’s Mutex detects that the thread holding the lock panicked. It automatically poisons the mutex. The mutex is marked as poisoned because the data it protects may have been left in a half-modified, inconsistent state when the panic hit. Unlike Go, where another thread can immediately grab the unlocked mutex and read potentially corrupted data, Rust refuses to let you blindly trust the data. When another thread tries to call .lock() on a poisoned mutex, it receives a PoisonError instead of the normal data. Rust forces you (via the type system and Result handling) to explicitly choose how to handle the poisoned state before you can access the inner data. You have to handle the Err variant of the lock result, preventing silent data corruption. use std::sync::{Arc, Mutex}; use std::thread; fn main() { let data = Arc::new(Mutex::new(vec![1, 2, 3])); let data_clone = Arc::clone(&data); let handle = thread::spawn(move || { let mut guard = data_clone.lock().unwrap(); guard.push(4); panic!("Something went wrong mid-modification!"); // Panic while holding lock }); let _ = handle.join(); // Wait for thread to panic // Next thread tries to acquire the lock: match data.lock() { Ok(guard) => { println!("Data: {:?}", guard); } Err(poisoned) => { // Rust caught the panic! The data might be corrupted. // We can access the inner data via the poison error if we want to recover it: let guard = poisoned.into_inner(); println!("Recovered from poisoned mutex. Potentially inconsistent data: {:?}", guard); } } }</code></pre> This shows you how languages differ philosophically. Go’s Philosophy to me is Simplicity and Trust. Go gives you lightweight primitives (goroutines, channels, sync.Mutex) and assumes you, as the developer, will write the correct orchestration code. It trusts that if a panic occurs, you’ve handled it with recover() and cleaned up your data structures. It prioritizes runtime performance and flexibility, meaning it won’t inject hidden overhead to track the health of every single lock. Rust’s Philosophy could be Fearless Concurrency via the Type System. Rust assumes that any shared mutable state is a potential bug factory, and threads will panic unpredictably. Therefore, Rust builds safety checks directly into the compiler and types (Send, Sync, MutexGuard, PoisonError). It forces you to deal with failure states (like a poisoned mutex) at compile time or via explicit error handling, making silent data corruption much harder to achieve. Rust is better in the Agentic world</a></h2> I personally feel writing Rust with agents is just a better experience, its ideal for vibe coding</a>. The compiler acts as an automated reviewer, it eliminates entire classes of bugs at compile time. AI agents frequently struggle with multi-threading synchronization because reasoning about concurrent state across multiple files is difficult for LLMs. Rust’s type system prevents the AI from accidentally creating data races or unsafe shared-state concurrency. Rust is built different!</a> i should have paid a closer look at tokenizer Unknown — Sat, 30 May 2026 00:00:00 +0000 Well, I gave an interview sometime ago, and wasn’t able to explain tokenizer properly, missed some points on tokenizer training, and got the sweet rejection I deserved (along with some great advices). But, I got an interested in tokenizer because of that, and damnn I found insights that I didn’t have maturity to see when I used tiktoken</a> for the first time in my 2nd year. So going back the basics. Everything is token in the model utopia we are living in! The API’s are priced per token. The fewer tokens required to represent an output, the faster will be the inference. Clearly it’s crucial to understand this token mumbo-jumbo. Tokenization</a></h3> Well, clearly the LLM cannot read English. ML models take in vectors, not weird shit like language. So, tokenizer is the bridge from English input to numbers which LLM can understand. It processes every punctuation, spaces, everything, and converts it into a sequence of integer (in a fixed range i.e. vocabulary) before the model can actually process it. The integers to vectors is embeddings. The jargon here is one-hot encodings. We map eg numbers from 1 to 100, to a 100-dim vector, with a 1 in the kth position, 0 everywhere else. Key intuition is that one-hot encodings let you think about each integer independently - useful when integers = labels. We are baking in structure to models. Dimensions = things that vary independently. Each input has its own dimension, so each input can be thought of independently, we don’t bake in any relation. And, the lookup table (the embedding matrix), is simply multiplying matrix with one-hot encoded vector. Softmax just converts vector to probability distribution. This conversion matters as it bakes assumptions into the LLM that cannot be undone later. If we don’t have this correctly, the LLM will waste its capacity in encoding common words into multiple tokens, so it decides the utility of the context window. It’s the process during which a piece of text is broken down into smaller pieces, tokens, by a tokenizer. These tokens are then assigned integer values (i.e. token IDs) which uniquely identify the tokens within the tokenizer vocabulary (set of all possible tokens used in the tokenizer training), they essentially indexes into the tokenizer vocabulary. Just as a side note, tokenizer training is different from neural network training. You can train your own tokenizer and restrict its token space by various parameters, including the size of the vocabulary. What do you think happens if any of the tokens in your text do not exist in the tokenizer’s vocabulary of the LLM you are trying to use? disaster. so most LLM vocabularies are pretty huge. tokenization can be of 3 main groups : word - splits text based on empty space characters and punctuation, etc, which is just too big of a vocabulary! can’t really create a separate token for every possible word in a language. and adding variations on top of it is just nightmarish. too much of vocabulary to manage.</li> character - splits text into individual characters, sometimes even punctuation. using individual characters look simple, vocabulary (256 ASCII) is tiny as well, but the problem is that this massively extends our own input sequence and doesn’t really have any structure. The model has to learn that the characters together mean a word which would burn the attention capacity. And as a transformer model has computational costs that goes quadratically with the sequence length (the attention layer compares every token with every other token), this is actually pretty expensive. Efficient-attention variants can reduce this, but the basic lesson still holds</li> subword - its the sweet spot. The common words stay whole and the rare words decompose into subword tokens that might seem like gibberish. Here, vocabulary also remains manageable, and the sequence also stays relatively short. We don’t really need to care much about the unknown tokens because any word can now be built from the sub-word pieces. LLMs use this</li> </ul> BPE, Unigram, Wordpiece, etc all sorts of them exist</a>. You can try building it</a> and play with a small model like MiniLM locally. If you play with emojis, you will see that if your token doesnt exist in the tokenizer vocabulary it gets tokenized as a special character. BPE: the hero of every LLM</a></h2> BPE was introduced for neural MT partly to handle rare and unknown words by representing them as subword sequences rather than requiring a huge word vocabulary</a>. Byte-level BPE goes further, GPT-2-style tokenizers can represent arbitrary bytes, so the failure is usually not “unknown token,” but fragmentation, inefficient representation, strange byte pieces, or distribution shift. It’s just a greedy compression algorithm which is sort of re-worked for tokenization. It starts small and builds up intelligently. It starts with initializing the vocabulary with 256 byte values. It tries to find a pattern by counting how often each pair of adjacent tokens appears in the training data and then merges the most common pairs. That is, it takes the most frequent pair and adds it to the vocabulary as a new token, and then it keeps on repeating until a desired vocabulary size is reached. This simply means, obviously, that the most common sequences get their own tokens while the rare ones remain split into multiple pieces. It really adapts to the specific training data set. It really has some esoteric shit if you seee reference_gpt2.tokenizer.vocab.items()</code>. It’s a fking headache. Whethre things begin with capital letters, or space matters. Arithmetic is total mess. length is inconsistent, common numbers appears more. I am hella impressed how GPT is even able to do addition! There are some interesting tricks in here as well : Pre-tokenization - it is possible that we get weird, awkward tokens by merging across logical boundaries. For an example, merging the period at the end with the word itself will create tokens that reduce the model’s flexibility. Pre-tokenization solves this by first splitting the text into logical units, which are usually the words or the punctuation, and then running the algorithm over these boundaries only. This gives us logical segregation of the tokens. And also this gives us a computational advantage because instead of counting the pairs across the entire corpus, now we can count how many times each unique word appears and then run the pair counting only once per that unique word. Then we could simply multiply the pair count by the word’s frequency.</li> Caching - obviously, this algorithm will be painfully slow on very large data sets. With each merge, we would require re-scanning the entire data set to update the statistics. It’s stupid for very large data sets, so we can just do some caching with HashMaps. We can have a bi-directional HashMap, which is:</li> </ul> which pairs exist in each token</li> which token contains each pair When we merge a pair, we only update the tokens containing that specific pair rather than re-scanning everything</li> </ul> Parallelization - the merging step clearly depends upon a global statistic, which is clearly hard to parallelize, but the initial counting of tokens can be very easily parallelized. By splitting the corpus into chunks and counting the token frequencies independently and then merging the counts, we can easily leverage multiple CPU cores to speed up the tokenizer training.</li> </ul> Despite all sorts of elegance, there are some quirks to be aware of as well : It might be possible that those rare words get split into so many small pieces, that it potentially loses all possible semantic coherence</li> The same word can be tokenized differently depending upon the context</li> Obviously, the model can struggle with words that were extremely rare or completely absent in the tokenizer’s training data</li> </ul> Embeddings</a></h3> Tokenizers were developed to do complicated numerical analysis of texts, mostly based on frequencies of individual tokens in a given text. What we need is context, to somehow capture the relationships between the tokens in the text to preserve the meaning of the text, and embeddings (vectors representing tokens) are better for that. Embeddings are byproduct of transformer training and are actually trained on the heaps of tokenized texts. Embeddings are what is actually fed as the input to LLMs when we ask it to generate text. Both the encoder and decoder accept embeddings as their input and the output of the encoder are also embeddings which are then passed into the decoder’s cross-attention head which plays a fundamental role in generating (predicting) tokens in the decoder’s output. The token IDs are used to fetch the embeddings from the embeddings matrix which are then assembled into a tensor which is then fed to the input of the transformer. The Model training chronicles</a></h3> You give it a bunch of text, and train it to predict the next token, autoregressive magic. raw text ↓ tokenizer token IDs ↓ embedding table lookup token embeddings + positional embeddings ↓ transformer blocks contextual hidden states ↓ output projection / LM head logits (one for each input token) over vocabulary ↓ softmax probability distribution over next token ↓ convert to token and then append this to the input + run again to generate more text</code></pre> in code : import torch import torch.nn as nn import torch.nn.functional as F batch_size = 4 seq_len = 8 vocab_size = 50_000 d_model = 768 token_ids = torch.randint(0, vocab_size, (batch_size, seq_len)) token_embedding = nn.Embedding(vocab_size, d_model) position_embedding = nn.Embedding(seq_len, d_model) positions = torch.arange(seq_len) x = token_embedding(token_ids) + position_embedding(positions) # Each position produces a distribution over the next token. # transformer processes this x print(token_ids.shape) # [4, 8] print(x.shape) # [4, 8, 768] lm_head = nn.Linear(d_model, vocab_size) hidden_states = x # pretend this came out of the transformer logits = lm_head(hidden_states) print(logits.shape) # [4, 8, 50000] # For training, the target is usually the same sequence shifted left: input_ids = token_ids[:, :-1] target_ids = token_ids[:, 1:]</code></pre> Importantly, if you give a model 100 tokens in a sequence, it predicts the next token for each prefix, ie it produces 100 predictions. This is kinda weird but it’s much easier to make one that does this. And it also makes training more efficient, because you can 100 bits of feedback rather than just one. weirdness is that 99 should be trivial, as you have already ttold what the next tokens are. the magic is because of causal attention, the representation at position 2 cannot look at tokens after position 2. During training, we pass the whole sequence in parallel, but the causal mask prevents cheating. The core thing is that it can only move information forwards in the sequence. The prediction of what comes after token 50 is only a function of the first 50 tokens, not of token 51. For attention weights : Q expresses what each token needs to know from others</li> K encodes the information that each token brings</li> V stores the actual content that each token shares when attended to</li> </ul> QK^T^ is the attention score that tells how well each token matches the keys of the other tokens (similarity). And then we attach corresponding V based on relevance. So, if two attention score are pretty similar, we attach more V. This is attention weights. Later the attention weights are multiplied with feed forward weight matrix so that the model has a chance to capture additional interactions across the sequence. In multi-head, just divide the embedding dimensions with number of heads. All of them see the full input sequence, but there subset of embedding dimensions, with its own Q,K,V weights. So, the attension calculation is paralleled among heads. Nothing to do with more cores or something. it’s just that each each can focus on subset of embedding space, picking up specific patterns and relationships. Tokenizer: text → integers Embedding: integers → vectors Transformer: vectors → contextual vectors LM head: contextual vectors → token probabilities</code></pre> Conclusively, transformers are sequence operation models, they take in a sequence, do processing in parallel at each position, and use attention to move information between positions (prior positions in the sequence to the current token)! once attention has moved relevant information to a single position in the residual stream, MLPs can actually do computation, reasoning, lookup information, etc. what is going on inside MLPs?! See Toy Model of Superposition Paper</a> for more on why this is hard. as said by Neil nanda, linear map → non-linearity -> linear map is the most powerful force in the universe and can approximate arbitrary functions. Idk man it just works!! The tokenizer is an architecture decision</a></h2> A tokenizer feels external because it is trained before the model. But after training starts, it becomes part of the model contract. The embedding table is indexed by token IDs. The model learns patterns over those IDs. The context window is measured in those IDs. The training budget is consumed by those IDs. The inference bill is charged by those IDs. So tokenizer choice affects: training efficiency</li> inference cost</li> effective context length that the model can handle</li> memory usage</li> Generalization to unusual or out-of-vocabulary words</li> numeric reasoning</li> code modeling</li> RAG chunk sizes</li> retrieval behavior</li> safety filters</li> debugging complexity</li> </ul> Changing the tokenizer is therefore not a harmless preprocessing change. It changes the model’s input language. tokenizer training data, vocabulary size, and pre-tokenization regex can affect compression, generation speed, effective context size, memory usage, and downstream code performance</a>. Tokenization also decides who pays more</a></h2> Tokenization is not uniform across languages</a>. The same amount of meaning can require very different token counts depending on the language, script, and tokenizer training data. That affects cost, latency, and sometimes quality. For API users, this means tokenization can create hidden pricing differences. For model builders, it means tokenizer training data should be evaluated across the languages and scripts the model is expected to serve. Your RAG depends on it actually</a></h2> In the RAG pipeline, the text is first tokenized, then its embeddings are obtained for each token via ID, then assemble the embedding tensor, then fed into the transformer where the attention magic happens. Earlier, I used to think about RAG pipelines from the embeddings, from the chunking, but I never used to think about tokenizers. Now, you should easily be able to see, missing words in the tokenizer vocabulary can produce undesirable tokens, which has implications on RAG. In old word-level systems, out-of-vocabulary words were a direct problem. In modern subword and byte-level tokenizers, the usual problem is different: the tokenizer can still represent the string, but it may represent it badly. A rare word, weird Unicode sequence, emoji, typo, long number, or domain-specific identifier may explode into many tokens, get normalized strangely, or produce a token sequence that the model rarely saw during training. Easiest example is of emojis when tokenizers don’t handle it well. Even if you add contrasting emojis to the same sentence, when you embed it and then display and then see the embedding matrices along with the text where we replace the emojis with textual descriptions, you will see that the embeddings for both the emojis, even though they may mean very different things, are very close. Another case could be of misspelled words being picked correctly, or managing date and time like “It was delivered some-time ago”, who knows that sometime ago? the models generally handle cases like these with the help of additional context and chunking with metadata properly, but if your agent doesn’t confirm the specific date, i.e. any sort of time context is missing, all the best. you can literally try introducing typos into the dates or even empty space characters and wreak more havoc. Sooo, a little bit of cleaning of input text actually go a long way, standardise the format your dates so they’re consistent throughout your embeddings, remove trailing spaces wherever you can, the same goes for any other numerical data like prices in different currencies, whatever. bloody hell there can be adversarial attacks based on word perplexities! tokenizer catches you off-guard</a></h2> A tokenizer looks like preprocessing, but in a real large-scale model training pipeline it is closer to infrastructure actually. It sits before the model, before the loss, before the dataloader emits tensors, before evaluation, and before inference. If it is wrong, the rest of the stack can be perfectly engineered and still behave strangely. it is part of the model architecture, the data pipeline, the compute budget, and sometimes the failure mode. it can catch you off-guard. When a training run fails at the same step again and again, and the instinct is often to look at the optimizer, checkpoint, distributed setup, GPU memory, mixed precision, dataloader, etc. But if skipping one exact data step avoids the crash, the search space collapses. Now it is not “the training run is broken.” It is “this example, or this batch, or this transformation of the data is broken.” That transformation includes the tokenizer. so, when a training run fails deterministically at the same data step, we do not need to debug the whole universe, just isolate : Can the same checkpoint train on the next batch?</li> Can the same batch tokenize offline?</li> Does the crash disappear if the tokenizer is swapped?</li> Does the crash disappear if the raw document is removed?</li> Does the crash disappear if multiprocessing tokenization is disabled?</li> Does the tokenized sequence have extreme length, weird special tokens, replacement characters, or abnormal numeric runs?</li> </ol> The raw document is not what the transformer sees. The transformer sees token IDs. Before that happens, the system performs normalization, pre-tokenization, subword segmentation, special-token insertion, truncation, packing, batching, and tensorization: raw text → normalization → pre-tokenization → subword segmentation → token IDs → packing/truncation → batch tensors → model A pathological raw string can become a pathological tokenization problem. Extremely long numeric sequences, repeated characters, weird Unicode, broken markup, logs, base64 blobs, corrupted JSON, or domain-specific identifiers can produce unusually long token sequences or trigger slow paths in tokenizer implementations. At small scale this looks like a bad row, but at pretraining scale it looks like a deterministic crash that costs hefty real money to reproduce. This is also why tokenizer choice affects model quality, not only runtime. The tokenizer decides the atomic symbols the model learns over. If the tokenizer represents numbers, code, dates, identifiers, or non-English text poorly, the model has to spend capacity learning around that representation. the tokenizer size, pre-tokenization regex, and tokenizer training data can materially affect generation speed, effective context size, memory usage, and downstream performance</a>. Before launching an expensive run, I would want a tokenizer report over a representative data sample: tokens per character</li> tokens per byte</li> tokens per document</li> p50/p95/p99/max tokenized length</li> longest numeric run</li> longest repeated-character run</li> Unicode normalization changes</li> replacement-character count</li> unknown-token count, if the tokenizer can emit unknowns</li> special-token leakage</li> encode/decode round-trip failures</li> tokenization latency p50/p95/p99/max</li> documents that timeout or tokenize abnormally slowly</li> compression rate by domain, language, file type, and data source</li> examples with extreme token/character ratios</li> </ul> At small scale, tokenizer bugs are annoying. At large scale, they are hella expensive. A failed experiment is not just a stack trace, it is GPU time, queue time, researcher time, and uncertainty. The expensive part is often not the bug itself, but the size of the search space. Was it the data? tokenizer? packing? distributed loader? checkpoint? optimizer? precision? kernel? scheduler? The right move is to reduce the search space as aggressively as possible. If changing the data fixes it, debug the data path. If swapping the tokenizer fixes it, debug the tokenizer. If disabling packing fixes it, debug sequence construction. If the same raw sample reproduces the failure offline, turn that sample into a regression test. Generally I suspect the tokenizer when: training crashes deterministically on the same data step</li> skipping one batch avoids the failure</li> tokenization hangs or is much slower for a few documents</li> loss is worse than expected after everything else looks normal</li> evals regress after changing the tokenizer, normalizer, or pre-tokenization regex</li> numeric, code, multilingual, emoji, or log-heavy data performs unusually badly</li> context length seems shorter than expected in real text</li> a domain-specific corpus becomes much longer in tokens than in characters</li> a model repeats or mishandles special tokens</li> encode/decode round-trip tests fail</li> tokenizer multiprocessing behaves differently from single-process tokenization</li> </ul> A tokenizer with a good average can still have awful tails. The tails are what crash training, waste context, inflate cost, or poison batches. there have been real tokenizer implementation issues involving hangs in subprocess/dataloader settings, so tokenizer latency and multiprocessing behavior are not purely theoretical concerns</a> Numbers are not just text</a></h2> Numbers look simple to humans, but tokenizers often represent them in unnatural ways. One model may split a number digit by digit. Another may group it into two-or three-digit chunks. Another may tokenize from left to right, which is awkward for arithmetic because carries usually propagate from the right. This matters because the model does not receive the number as a number. It receives a sequence of token IDs. The representation can change the difficulty of the task. A date, price, timestamp, account ID, version string, latitude-longitude pair, or long integer can become many tokens with weak numerical structure. So for numeric-heavy domains, tokenizer evaluation should include numeric stress tests as well. Obviously, right-to-left number tokenization can substantially improve arithmetic performance compared with standard left-to-right chunking</a>. Conclusion</a></h2> The tokenizer is easy to ignore because it sits before the glamorous part of the model. But the model never sees text. It sees token IDs. That means tokenizer mistakes become model mistakes, training inefficiencies, retrieval misses, cost inflation, safety gaps, and sometimes deterministic crashes. So the practical lesson is simple: when debugging model behavior, do not stop at the neural network. Look at the thing that decides what the neural network is allowed to see. cutting that inference cost Unknown — Thu, 23 Apr 2026 00:00:00 +0000 Who doesn’t want lightning fast AI responses? You don’t really want to serve responses slower than human typing speeds. Inference is hard. And, it’s not a model problem, it doesn’t differentiate between 1 and 1M requests, its scheduling problem, revolving back to depths of engineering. When the model is downloaded, we get list of artifacts, and using those ingredients and recipe we make inference out of it. Depending on what inference engine we use (vLLM, SGLang, TensorRT), they have different ways to load and serve the model. The biggest file (atleast in case of Gemma 4 that I downloaded) was model.safetensors</a> which actually holds the model weights (it’s a bloody large JSON file). The config.json has models entire architecture (like number of attention heads, number of layers, what kind of attention, size of vocabulary, etc). The inference engine takes the artifacts and put them to GPU (cudamemcpy shit). so lamma is pretty fast. vLLM can take some minutes as it compiles the model, a large import overhead and also a much heavier initialization for better scheduling and concurrency (important for the pre-filled decoding and the serving part). vLLM is pretty cool as it has PagedAttension which is adopted from OS</a> so it boasts nearly zero memory waste. In lamma.cpp, they use mmap (OS manages the memory by holding weights in SSD and keeping track of its pointer location in RAM). When weights are needed by inference engine, it’s loaded lazily. So suppose we have Gemma 4 (its 15gb in size when I saw it), and we have a 32 GB RAM, then over PCIe (7 GB/s), so you can imagine loading will be in sec latency. GPU does most matmul, tensor stuff, so RAM needs to push the weights higher up memory hierarchy which is generally faster. Modern generative models are decoder-only</a> as : Next-Token Prediction - Decoder-only models are structurally optimized for autoregressive text generation</li> Seamless Context - Instead of using an encoder to process a prompt and a decoder to respond, decoder-only models handle both by treating your prompt as the beginning of the sequence and simply letting the model continue writing</li> Better Scaling - Empirical evidence and industry scaling laws (like those that built GPT models) have shown that decoder-only models scale incredibly well in performance as you increase their parameters and training data</li> </ul> Typical GPT-2 decoder-only architecture : tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd) x = self.transformer.drop(tok_emb + pos_emb) for block in self.transformer.h: x = block(x) x = self.transformer.ln_f(x) logits = self.lm_head(x) return logits</code></pre> A zoomed in transformer block : class Block(nn.Module): def init(self, config): super().init() self.ln_1 = LayerNorm(config.n_embd, bias=config.bias) self.attn = CausalSelfAttention(config) self.ln_2 = LayerNorm(config.n_embd, bias=config.bias) self.mlp = MLP(config) def forward(self, x): x = x + self.attn(self.ln_1(x)) x = x + self.mlp(self.ln_2(x)) return x</code></pre>Anistropy</a></h2> Soo, a decoder-only transformer’s layers serve different purposes: Early layers are for building local features syntax, positional patterns, n-gram statistics. Too raw for task-level understanding.</li> Middle layers are for semantic composition. This is where the model builds a holistic representation of “what kind of request is this”, task type, complexity, domain, language, structure. This is exactly what routing needs.</li> Final layers are for collapsing toward next-token prediction. The representation becomes increasingly specialized for predicting a specific output token, losing the broad task-level signal.</li> </ul> This is an interesting one. The loss function for decoder-only LLMs is logits = hidden_state @ W_unembed → softmax(logits) vs true next token</code>. So the final hidden state must produce useful logits when multiplied by the unembedding matrix W_unembed. This means the final layer’s hidden states are optimized to live in the row-space of W_unembed, they need to produce strong dot-products with specific vocabulary vectors. Here’s the key, W_unembed has a small number of dominant singular directions (common tokens like “the”, “is”, “\n”, punctuation dominate the output distribution). The final-layer representations get pulled toward these dominant directions because: Most training examples have high-probability next tokens that align with these common directions</li> Gradient descent moves the hidden states toward regions where they produce correct logit patterns</a></li> Over billions of training steps, this creates a systematic bias, the hidden states cluster along the principal components of W_unembed</li> </ul> two semantically very different prompts (“write Python code to sort a list” vs “translate this poem to French”) end up with final-layer hidden states that have cosine similarity ~0.7-0.9</a>, because both are being pulled toward the same dominant W_unembed directions. The “what kind of task is this” information gets overwritten by “what distribution of next tokens is likely here.” The technical name for this is anisotropy problem</a>. final-layer representations of decoder LMs cluster into a narrow cone aligned with the dominant vocabulary embedding directions. The last few layers increasingly project toward dominant eigenvectors of the unembedding matrix, see the logit lens effect</a>. This means cosine similarity between ANY two final-layer representations is high regardless of input content. Two completely different prompts will have high cosine similarity at the final layer because both representations are being pulled toward the unembedding matrix’s principal components. A linear probe on these representations loses discriminative power because the geometric spread has collapsed. Middle layers aren’t directly optimized to produce logit distributions. They serve as intermediate computational stages. Their representations are shaped by: The need to pass useful information upward to later layers</li> The residual stream accumulating information from attention + FFN blocks</li> NO direct pressure to align with W_unembed</li> </ul> So middle-layer hidden states remain more isotropic, they spread across the full d-dimensional space, preserving geometric separability. A linear probe can easily find hyperplanes that separate “coding task” from “creative writing” from “factual Q&A” because these categories occupy genuinely different regions. The narrow cone isn’t the entire d=2048 space collapsing, it’s 1-3 dimensions with enormous norms pulling everything in their direction. so we can capture variance in the non-rogue dimensions using the dispersion feature (max-pool − mean-pool across token positions at layers) captures how much the tokens disagree with each other at the semantic level, a measure of input heterogeneity/complexity that’s maximally informative in the layers where tokens have been semantically processed but haven’t yet collapsed toward the prediction bottleneck. This is beautiful insight for anyone building low latency routers (discriminative features required). The self attention itself causes this anisotropy problem. it appears even in character-level models and non-NLP transformers (vision, audio). Anisotropy is a structural property of self-attention, not just a side-effect of the training objective. Some basics of model serving</a></h2> LLM serving has two different phases with very different performance characteristics: Prompt / prefill phase: when the model reads the input tokenised prompt all available upfront (tokens are known so attention can be computed for all positions simultaneously) so the model can process them all in parallel using matmuls at full GPU throughput (its takes ms) and get those KV (which are very large so we build KV cache)</li> Mostly compute-bound (GPU working hard) so highly parallel</li> Affects time-to-first-token (TTFT)</li> </ul> Decode / generation phase: Generates the response one token at a time (new token depends on all previous tokens and becomes the part of new input) till stopping criteria is met (max length or end of sequence), doing matrix-vector multiplications, one forward pass per token, loading all the model’s weights and KV cache from VRAM.</li> involves CPU overhead, from assembling the batch to dispatching GPU kernels and reading the results. CUDA graphs eliminate the overhead by recording the full GPU execution sequence once and replaying it for batches with matching shapes</li> Mostly memory-bandwidth-bound. The weight matrices are the same size as during the pre-fill, but we are multiplying them by a single vector instead of a matrix. The GPU cores finish this simple work in microseconds and then wait for the next batch of weights to arrive from the memory. The game here is how fast we can stream the model weights from the HBM to the compute units.</li> Affects time-per-output-token (TPOT)</li> </ul> For a 31B parameter model stored in 16-bit floats, the GPU has to load around 62GB of data from memory just to generate one token fragment. The GPU spends most of its time sitting idle, waiting for data to arrive from the memory bus. This is the stutter thing you see in slow LLM responses. The problem is of arithmetic intensity (fancy word to throw around but it’s just how much math happens per byte read from memory ops:byte). Matrix-vector products often have very low arithmetic intensity incase of small batches (chunked often used). The GPU finishes the math almost instantly and then waits for the next batch of weights to arrive from memory. On modern hardware, the GPU is often less than 10% utilized during decode at small batch size! This distinction matters for production systems. Clearly, the pre-fill throughput scales with the GPU compute, as more flops mean faster pre-fill, and decode throughput scales with memory bandwidth, so the faster memory means the faster decode, and that’s exactly why NVIDIA’s H100s focused on memory bandwidth improvement over the A100s, because it directly speeds up the token generation Millions of tokens prefilling</a></h2> the request is moved from pending to prefilling when the scheduler finds enough token budget and cache space to admit the request at the first place. If the prompt is too long to fit in a single step, the scheduler splits it across multiple forward passes as processing a long prompt in a single step is expensive - because it blocks other requests from generating. Chunked prefill solves this by splitting large prefills across multiple steps. This approach enables interleaving prefill and decode operations, preventing long pauses before the first token appears. So, instead of processing millions of tokens in a single prefill operation, we process chunks of tokens, and the system can begin decoding after the first chunk completes, dramatically reducing time-to-first-token (TTFT) while maintaining full context awareness. We get awesome stall-free scheduling. The schedular can be simple FIFO, or fancier like PrefillFirstScheduler</code>. Larger chunks reduce overhead but increase initial latency. Smaller chunks improve responsiveness but add processing overhead. Typical chunking strategy is to have some overlap. Dynamic chunk size helps to optimize for both for both latency and throughput. Works quite well for streaming or low-latency settings. KV caching</a></h2> Oh, this is the base transformer inference optimization. To simply put, during the autoregressive generation, without caching, each token would require recomputing the attention keys and values for all the previous tokens. We just add a simple KV cache that stores the key-value tensors derived from the previous tokens and reuses them for later decoding steps. It is primarily helpful in the decode phase, and it’s basically mandatory for any sort of practical LLM serving. For multi-million token inputs, this cache can consume 80-90% of GPU memory, making cache management critical for latency reduction. Obviously this doesn’t work for the first token, which is why it takes longer to generate. Cache size (FP16)= 22* batch_size* seq_length* num_layers* embeddings_length</code> pretty much GBs there are some advanced optimisations like : reducing the KV cache memory by grouping multiple queries with same keys and values - Grouped Multi Query Attention</li> </ul> prefix caching</a></h2> This is KV cache for a shared prompt prefix (cross request KV reuse) and is particularly important for the LLM serving. So, during the Transformers inference, the model computes the key value states for all the prompt tokens and when many requests share the same starting prompt which is like more often than not in the case, the serving engine can use those cached key value pages instead of recomputing them. It’s awesome as it skips repeated prefill, lowers TTFT and input-token cost. vLLM also lists prefix stashing along with continuous batching and chunk prefilling as part of its LLLM serving stamp. vLLM caches the KV cache of existing queries so a new query with the same prefix can reuse it and skip computation for the shared part. this reduces query/prefill processing time but does not reduce the time needed to generate new output tokens. its easy latency reduction on multi-turn conversations. it works the best when the requests share a long stable prefix (should appear first in your prompt structure) like a shared system prompt, a tool or a function schema (often large tool definitions are identical), RAG over a common documents, multi-turn chats, agent frameworks, it is a really good tool to use for overlapping back-to-back requests. but this does not reduce the cost of generating the new output tokens. It mostly reduces the repeated prompt processing work. That’s why from formatting matters. So we should really avoid inserting timestamps, request IDs, user specific metadata, or randomized text before the common prefix, because even the small token difference can break the reuse. Continuous Batching</a></h2> Starting batching is just waiting for a batch of requests to arrive and then processing them all together, and then waiting until all of them finish before accepting the new requests. Obviously, this could be problematic because the shorter request would sit idle for the rest of the decoding step to finish. Continuous batching simply inserts the new request into the batch as soon as any request completes. The batch is then re-evaluated at every decode step, so a request that finishes quickly is immediately replaced by a waiting request. Well, we don’t always have uniform inputs, in which case the continuous batching would have matched the static batching, but often, because we have variable lengths, so continuous batching delivers improved throughput because the GPU slots never really sit idle. model pruning and compression</a></h2> Among billions of parameters, some are bound to be less important than others. Pruning (or if you wanna sound cool network sparsity) involves removing weakly important parameters from pre-trained models. Modern NNs are overparamatrized, so removeing some parameters shouldn’t harm accuracy. Lottery Ticket Hypothesis</a> suggests that a subnetwork exists for every neural networks, which when trained in isolation, reach test accuracy comparable to the original model. Identifying the winning ticket (subnetwork) is crucial, and can be derived by pruning a pre-trained network. Moderate pruning sometimes improves generalisation (we know overparamatrization is bad as it can lead to overfitting</a>) across different domain (after another round of recovery fine tuning). So, we need to be careful with this. Quantization is generally the first compression attempt. Pepole also do quantization-aware or compression-aware training (model learns under the constraints of compression), which is more complex but best in class especially for low latency real-time ML with high accuracy. Also, another concern to note is that pruning often requires fine-tuning on pruned model, which is additional training time. Another way to compress is using weight clustering. So you can map model weights to a discrete set of pre-computed or learned values. And similar weights are replaced by the shared approximate values. This also reduces the model size. But it’s often noted that it does not necessarily make infants faster because it can actually introduce a look-up overhead. So it’s a good option to keep in mind in case the storage foot print has the main concern. quantization</a></h2> Supposedly, you have some Llama model of 70 billion parameters, and each parameter is a 16-bit floating-point number that is like 140 GB. A single typical A100 has 80 GB of VRAM, so I can’t even load weights on that, let alone run inference on a single GPU. I need multiple such A100s to just serve one model. If you observe closely, 16 bits per parameter is actually quite wasteful. Most weights in a neural network cluster near zero, and the full dynamic range of FP16 is almost entirely unused. If you measure the actual distribution of weights in the Llama model, most of them (I think more than 90% of them) will fall between -0.1 and 0.1. Basically, we are burning 16 bits to represent values that could be stored in simple 4. That’s where quantization comes into play. It replaces these high precision numbers with low precision ones. And the whole trade-off is of accuracy, because every bit you remove destroys some information, so the question is how much accuracy we can lose. It’s a trade-off, right? A well-quantized INT4 model retains almost 95 to 99% of the original quality on most benchmarks, but a naive quantization to INT4 can destroy the model entirely, and the difference is of techniques. So, quantization means storing and/or computing model weights (model.safetensors) and activations in lower precision to get faster memory reads. Lot of nasty stuff here like RTN, AWQ, FP8, GGUF, bits and bytes, etc. FP16/BF16 is baseline production inference, but FP8 reduces model memory by about 2x and improving throughput up to about 1.6x with minimal accuracy impact. You can take entire tensor (that is the standard method), or you can do this per channel or per group, whatever. This is what RTN (Round to Nearest) approach is. Below 4-bit, accuracy drops if we dont do post training quantization</a>.</li> AWQ (Activation-aware Weight Quantization)</a> preserves important weights at higher precision. The key observation is that not all weights contribute equally to output quality. Protect the weights/channels that matter most for model outputs. Quantize the rest aggressively. AWQ identifies salient channels using activation statistics and protects those channels through scaling, while still using hardware-friendly low-bit weight quantization. The AWQ paper states that protecting only about 1% of salient weights/channels can substantially reduce quantization error. AWQ does not literally keep random important individual weights in high precision at runtime in a naive mixed-precision way. That would be hardware-inefficient. AWQ uses activation-aware scaling so the post-training quantized representation preserves important channels better.</li> </ul> All about the glories of quantization, but there are some clear failure modes as well. First of all quantization can introduce range mismatch problems. So if the quantized weights or the activations do not match the ranges of the original model, the accuracy often deviates. So most probably we need to check whether the quantized weights and activations remain within the expected ranges compared with the original model. Second quantization can create a model compression artifact that are not captured by ordinary aggregate metrics. So, actually we should avoid compressing at the final output layer</a>. By the way, just a side note, not everything in a model tolerates quantization equally. There is a sensitivity hierarchy: Weights are the most robust of them because they change slowly during the training and follow a roughly Gaussian distribution centered near zero, so they quantize pretty well. I mean that an int8 weight with per-channel scales produces nearly lossless results. Int4 requires a more sophisticated method, but it works.</li> Activations are moderately sensitive. They are intermediate values flowing through the network during the inference, so they have a bit wider dynamic range than the weights and contain some outliers as well. A single attention head might produce activation values that are 100x or something like larger than the mean. These outliers are critical for the model quality, actually, so quantizing them naively destroys a lot of information. In case of that, we keep the outlier channels in high precision to save them.</li> KV cache is very sensitive. It stores the attention states for all the previous tokens, so at long context length the KV cache dominates the memory. Quantizing the KV cache saves massive memory, but any error compounds across all the future attention computations, so the quality impact scales with sequential length.</li> Attention logits are the most sensitive. The softmax and attention are highly sensitive to small changes in their input. A quantization error of even 0.01 in a pre-softmax logit can shift the attention distribution meaningfully. Most quantization schemes keep attention computation in high precision even when everything else is quantized.</li> </ol> btw, you can also quantize the kbcache so you can store the values in int8 or int4 format instead of fp16 and reduce the memory footprint by 2 to 4 times with minimal accuracy impact. Quantization actually hurts when you are compute bound, which is kinda counterintutive. its just a trade, you reduce memory bandwidth requirements (smaller weights = fewer bytes to load from HBM) but add dequantization compute (converting INT8 back to fp16/fp32 for the actual matmul). When you’re memory-bandwidth-bound (like decode-phase serving), this trade is massively profitable, you load half the bytes and the extra dequant math is trivially hidden by the memory latency. But when you’re already compute-bound (like 4k-token prefill), the GPU’s arithmetic units are already saturated. Adding dequant ops just piles more work onto an already-full pipeline. The diagnostic is simple, if batching doesn’t meaningfully reduce per-request latency, you’re compute-bound, and quantization will hurt. the GPU is maxing out its tensor cores, not waiting on memory. speculative decoding</a></h2> Gemma 4 pulled this off nicely, and achieved a 3x speedup in inference with zero loss in quality. Autoregressive decoding is slow because the model usually generates one token at a time at inference. Each time the model produces a token, it reads that token back as input and runs another full forward pass through all its layers to produce the next one. Each output depends on everything before it, so you cannot skip ahead or parallelize generation. You are always waiting for token N before you can start token N+1. Basically, sequential token generation with a hard dependency chain. Without any optimization, generating those token n requires recomputing attention for all the N-1 previous tokens. That would be O(N^2) per generated token or O(N^3) total for the sequence length of N. KVCache solves this exactly where after computing K and V for each token we simply store them and when generating the token N+1 we just need to compute KVCache solves this exactly, where after computing K and V for each token, we simply store them and when generating the token N plus 1, we just need to compute the query for the new token and look up the cached keys and values from all the previous tokens. KV Cache solves this exactly where after computing K and V for each token we simply store them and when generating the token N+1 we just need to compute the Q for the new token and look up the cached keys and values from all the previous tokens. This reduces per-token cost from O(N) to O(1) for KV Cache computation. Though the attention score calculation is still O(N) because we will still need to attend to all the previous positions, just that we avoid redundant matmuls on the input now. Note, generating a token from scratch is expensive, but checking whether a given token is correct is cheap. Speculative uses a cheaper smaller (each pass is cheap because the model is tiny) draft model to guess multiple future tokens, then asks the main model to verify them in parallel (one forward pass over all draft tokens simultaneously and produces a probability distribution over what each token should have been like profile - parallel, compute bound, efficient). Draft model proposes: “the answer is probably A B C D” Target model verifies in one larger pass: A accepted B accepted C accepted D rejected Then continue from the accepted prefix. If several draft tokens are accepted, we saved multiple expensive target-model decode steps. The target model remains the authority to either reject or accept the token. The draft model only proposes. If the draft is wrong at position i, tokens after position i are discarded and inference continues normally from there. If the verification algorithm i.e rejection sampling is implemented correctly, the final output distribution can be preserved, so quality should not degrade. It’s not really an approximation. There is a mathematical equivalence at play here. Obviously, the speedup from speculative decoding depends on the acceptance rate, so if its structured text like code, repetetive patterns, factual calls, drafter gets most tokens right and the acceptance rate is high. It works less well if you are creative. For most use cases, speculative decoding is better than standard inference</a>. so vLLM, SGLang, etc all support it. The verification can be parallelized as you already have the draft tokens and you can feed the entire sequence into the target model at once as a single forward pass, but generation cannot as we do not have the next token yet. To produce token N+1, the model must condition on token N. There is no way around this. This asymmetry is precisely what speculative decoding exploits. Take it as pipeline parallelism in CPUs! EAGLE-3</a> is a new speculative decoding method. It moves away from earlier feature-prediction constraints, uses direct token prediction, by training a small auto-regressive head on top of the target model’s hidden state. It operates on the target model’s own representation and not on a separate model. It achieves a higher acceptance rate with minimal extra memory. The paper reports speedups up to 6.5x in its experiments and a 1.38x throughput improvement in SGLang at batch size 64. speculative decoding mostly improves decode latency, not initial prompt prefill. So if your requests have enormous prompts and generate only 10 tokens, prefix caching and chunked prefill may matter more. a thing to note here is that speculative decoding is mathematically exact. The output distribution is identical to the target model’s distribution (not really an approximation). The verification step is there to ensure that every accepted token has exactly the same probability that the target model would have assigned. Python overhead</a></h2> At small model size, naive Python code overhead is large cuda graphs (xFormers has a great example particularly in accelerating FlashAttention and linear layers)</li> tensorrt-llm (tracing the inference + pattern matching)</li> custom kernels (fusion to reduce memory bandwidth)</li> </ul> What drives these metrics?</a></h2> Fixed Flops / Memory Bandwidth ratio = Minimal batch size B* to avoid wasting flops</li> Limited on device memory = Not trivial to reach this batch-size B</li> Python code overhead = In vLLM / TGI - not FasterTransformer but harder to deploy</li> </ul> Do the transformer inference arithmetic</a> for your needs. Open source deployment solutions are very usable, but not very optimized for small models What about routing models</a></h2> the router models lives entirely in prefill regime: we tokenize the prompt, run one forward pass through the encoder, extract hidden states, done. No KV cache, no autoregressive loop, no speculative decoding. Just one big parallel matmul party. so optimization game is different from generative models. some insights on HNSW Unknown — Thu, 16 Apr 2026 00:00:00 +0000 Suppose your RAG pipeline has embedded millions of docs. A user asks a question, and the system converts that query into a high-dimensional vector and needs to find the closest matches. The brute-force approach would be an exact kNN, just computes the distance between the query vector and every vector in the database. That’s O(N·d). With 10 million vectors at 1,536 dimensions, you’re looking at ~15 billion floating-point operations per query. Not viable at production latency. Now this exactness of kNN is enemy of speed for us in high-dim vector searches. Exact nearest neighbour search typically requires checking every point, which is O(n). ANN (approximate) methods trade a tiny fraction of precision/accuracy (recall) for a massive boost in performance by avoiding exhaustive comparisons. Instead of demanding the single mathematically perfect nearest match, ANN focuses on rapidly finding the best candidates in the right neighbourhood. This in practice is often sub-linear behaviour (like log) and you can often tune ANN to achieve 95–99% recall while cutting latency from seconds to milliseconds. HNSW has become the dominant ANN approach because it handles this trade-off really well. It combines two ideas: a Navigable Small World (NSW) graph, where vectors are connected to their neighbors like a social network, and a probabilistic skip list that stacks these graphs into multiple layers of varying density. Small world just means that in the network, we can get from one place to another in surprisingly few hops if the network has the right mix of local connections and a few long-range shortcuts. In simpler terms, HNSW builds a network of shortcuts so you can reach the right area quickly, then do a small local search to find the best match. Most vectors exist only in the bottom layer. A smaller fraction appear one layer up. An even smaller fraction appear above that. This creates the hierarchy. When we want to search, we start from top layer (tiny so quick to explore), then from our current node, we jump to the neighbour that looks closest to the query. We keep doing this until none of the neighbours is closer. We take the best node we found as our new starting point on the next layer down. Go deeper, till we are exploring a limited set of nearby candidates to produce the final nearest neighbours. Other players in ANN</a></h1> HNSW is the most used one, but different constraints call for different approaches. KD-Trees (k-dimensional tree) is used to organize data points in a multi-dimensional space, where the core structure is a decision tree for each space. At each level, the decision splits into halves and one dimension is chosen at a time. Queries traverse each tree to find candidate neighbors. It works well for low-dimensional data, but search performance can degrade as dimensionality increases. HNSW, in contrast, can maintain search speed even when the data structure gets complex.</li> LSH (Locality-Sensitive Hashing) organizes data using random hash functions to bucket similar vectors so that searches can find the right ones quickly. By hashing the query, searches can only look up vectors in a corresponding bucket. It tends to work best in cases where the data is extremely high-dimensional, or vectors are sparse. LSH is memory efficient, but recall can suffer as a result, which means HNSW is often a better approach for teams that need high accuracy, even if HNSW tends to require greater memory usage.</li> IVF (Inverted File Index) partitions the space into clusters. A search first picks the most promising clusters, then searches within those buckets. It’s often chosen when we want lower memory usage than HNSW or more explicit control over partitioning.</li> PQ (Product Quantization) compresses vectors into compact codes. This enables far larger collections to fit into memory, but with some loss of precision. It’s a common choice when memory is the bottleneck and we’re working at very large scale.</li> DiskANN keeps much of the structure on SSD rather than RAM, optimizing the layout to reduce random reads. It’s designed for datasets that simply won’t fit in memory, while still aiming for strong performance.</li> </ul> HNSW is at its most effective when we use it for large datasets, especially ones that include over one million documents and in situations when search performance and scalability are higher priorities than perfect accuracy. Due to its graph-based structure, requires much more memory than other ANN methods. With larger amounts of connections per node, HNSW can get even more memory-intensive. We need to hyperparamater tune properly. HNSW is best for in-memory, monolithic indexing, so it tends to suffer in distributed environments. HNSW is shardable, but sharding creates additional complexity and requires more memory to manage the various pieces. Quantization paradox</a></h2> HNSW latency is dominated by distance computations. If you use Scalar Quantization (SQ) or Product Quantization (PQ), you can compute distances 4x–10x faster using SIMD (AVX-512) instructions. Because distances are cheaper, you can set a much higher ef_search</code> (the number of candidates tracked during search) while staying within your latency budget. In many benchmarks (like Weaviate’s), searching a larger neighborhood with lower-precision vectors yields higher recall than searching a tiny neighborhood with high-precision vectors. So, don’t just tune ef_search on raw vectors. Quantize first, then spend your saved latency on a wider search. Tackling the silent failure</a></h2> HNSW recall isn’t a constant for obvious reasons. As your corpus grows from 10k to 10M vectors, the Small World connectivity becomes harder to maintain. A configuration that gives 99% recall at 100k vectors might drop to 80% at 10M vectors without changing a single line of code. In high-dimensional space, certain vectors naturally become “hubs” that attract a disproportionate number of incoming edges. These hubs can act as “gravity wells,” pulling search queries away from the true nearest neighbors. So clearly, we need to play with hyperparams! Tip - Before scaling your production cluster, build a Shadow Index with 10% of your production traffic being mirrored to a brute-force (flat) index. Compare the IDs. If your Truth starts drifting, it’s time to bump M or ef_construction. THe Disconnected Graph Filter Problem</a></h2> One of the biggest practical gotchas in RAG is filtered search (e.g., “Find docs about X, but only for user_id=123”). If your filter is very restrictive (e.g., only 0.1% of docs match), HNSW breaks. The search enters the graph, but most neighbors are “invalid” due to the filter. The search gets stuck in a “local island” of filtered-out nodes and can’t find the path to the valid ones. The fic is the ACORN algorithm</a> (recently adopted by Qdrant and Lucene). It modifies HNSW to perform 2-hop expansions. If your neighbors don’t match the filter, you check their neighbors. This keeps the graph navigable even when 99% of nodes are invisible to the current query. The Diversity Heuristic</a></h2> During insertion, HNSW doesn’t just pick the closest neighbors. It uses a heuristic: “Only add a neighbor if it is closer to me than it is to any neighbor I’ve already picked.” This forces the algorithm to pick neighbors in different “directions” (angular diversity). Without this, if you had a cluster of 100 very similar vectors, HNSW would connect only to that cluster and become “blind” to the rest of the world. It’s the algorithmic version of “don’t just talk to people who agree with you.” Hardware Aware Indexing</a></h2> Indexing 10M vectors on a standard CPU can take days. However, using libraries like NVIDIA cuVS (GPU-accelerated HNSW)</a>, you can drop index construction time from 28 days to 1.4 days. If your data is highly dynamic (lots of deletions/updates), HNSW fragmentation will eventually ruin your recall. Instead of trying to fix the graph, it is often more practical to just rebuild the index from scratch every week using a GPU-accelerated runner. how package managers use PubGrub for Dependency Resolution Unknown — Sat, 11 Apr 2026 00:00:00 +0000 There are all sorts of dependency resolution methods</a> used by various package managers</a>, but I am specifically writing on PubGrub as it’s used in uv package manager which I used to contribute to get some insights on how pro peps write Rust. In short, PubGrub is popular as : Tracks incompatibilities and produces human-readable conflict explanations</li> Uses CDCL (conflict-driven clause learning) like modern SAT solvers, pruning search space efficiently</li> Has intellignnt backtracking to the right decision point instead of naive retries</li> And, a lot of good quality implementations are readily available</li> </ul> You can go through it’s internals</a> and better, listen to the awesome lightening talk by its creator at DartConf</a>. The thing that nudged me to write this is something about concurreny. The PubGrub algorithm is a tight synchronous loop with complex mutable state that doesn’t fit naturally into async/await (so cant be async as its sequential). uv being clever</a></h2> What’s clever is that uv completely decouples the synchronous solver from parallel I/O using a two-thread architecture with channels: the uv-resolver (the dedicated synchronous thread) - Runs PubGrub loop normally, when it needs metadata, sends request via mpsc::channel(300) and calls wait_blocking() which blocks only the solver thread (not the entire tokio runtime). THis is sync so no async/await to reason about here.</li> Fetcher (async tokio pool) - Receives requests, fires hundreds of concurrent HTTP requests to PyPI, writes results to shared Arc cache (zero-copy deserialization, readers dont wait on mutexes) They also do some perf optimizations :</li> Prefetching - After 5 failed versions of a package, speculatively fetches the next 50 versions in parallel before the solver ever needs them</li> Conflict-priority heuristic - After 5 conflicts between packages A and B, promotes B’s priority above A’s and manually backtracks to decide B first</li> Forking resolver -Splits resolution on environment markers (e.g., python_version >= “3.11”), producing one universal lockfile for all platforms</li> </ul> SOoo, PubGrub stays simple and synchronous, but the solver can queue up to 300 requests before blocking, and those requests are served concurrently. Te network I/O get fully concurrent, biggest bottleneck (metadata fetching) gets parallelized without touching the solver logic. See the PR</a>. Others?</a></h2> The Dart’s original implementation was obviously sequential. I dont know why Poetry and pip (Python) also is sync</a> featching one package at a time. Swift does lazy fetching (doesn’t fetch metadata until after resolution decides a version, but still sequential, users report slowness</a>). Cargo (rust) is blocking during resolution</a>. yarn and npm (the javascript world) is an interesting piece over here as they use async/await throughout the entire solver code, whereas uv keeps PubGrub fully synchronous and only parallelizes I/O via a separate thread. JavaScript is single-threaded with async by default. There’s no way to write blocking code that doesn’t block the entire event loop. So they had to make everything async. It uses Single-threaded event loop + libuv thread pool (heavy lifting happens in libuv’s C++ threads). await fetchPackage(pkg) returns a Promise immediately; event loop switches to other tasks. Event loop never blocks, promises resolve later, JavaScript keeps running other code. As it uses single heap, references are naturally shared so no cross-thread synchronization needed. the longer you chat, the worse your agent's response Unknown — Thu, 19 Mar 2026 00:00:00 +0000 Just for the sake of basics, whats an agent?</a></h2> LLM is just autocomplete. No reading files, no running a query, forget about opening a browser, verifying a claim. The model will have outdated information and it will confidently say that slop. An agent is different. An agent is a system where a model operates in a loop, observing state, selecting actions, and incorporating results until a goal is met or a stopping condition is reached. An agent has three components: a model that decides, tools that execute, and a loop that connects them. The model reads the current state, system prompt, conversation history, tool results, and outputs either a tool call or a final answer. If it outputs a tool call, the tool executes, the result is appended to the state, and the model is called again. If it outputs an answer, the loop terminates. If neither, the loop continues until a stopping condition is reached. Every model has a finite context window, the maximum number of tokens for one inference call, counting both input and output. In a long agentic session with many tool calls, the conversation history can easily exhaust this limit. On a more formal note, a Markov Decision Process is a tuple ⟨S, A, P, R, γ⟩ where the system observes state (the context window (system prompt + message history + tool results)), selects an action according to a policy (respond to user, call a tool, request information, terminate), transitions to a new state (deterministic for tool execution, stochastic for LLM reasoning), and receives a reward signal (task completion (implicit) or user feedback (explicit)). The last one being discount, that is to prefer prefer shorter paths, fewer API calls, lower cost, less error accumulation. Basically, at each step, pick the action that maximizes expected future reward from the current state. The model’s system prompt, its training, and any examples in the prompt collectively approximate this strategy. The Markov property states that the next action depends only on the current state, not on how the agent got there. For an LLM agent, the current state is the context window, the system prompt, the conversation history, and every tool result so far. What is in that window is all the model knows. What is not in it does not exist to the agent. This is why memory and context management matter at scale. At the end, there are only three levers, better state (memory, tool results), better actions (more capable tools), or a better policy (prompting, fine-tuning, reasoning strategies). By now we have established that AI agents are nothing but loops. The agent loop sends HTTP requests to the Responses API, building an ever-growing JSON prompt from system instructions, tool definitions, sandbox permissions, and conversation history. Now every agent loop, regardless of framework, model, or complexity, requires exactly three components. a protocol its gonna follow, a registry, and a loop with a termination condition. The most common protocol would be ReAct</a> as it combined Chain of Thought with Action (think then act and then reason about the response you got) to avoid assumtptions that agent makes without concrete information. Feels like simple RL, but without discrete action space, if you really have open action space, will we ever converge? But LLMs are so good, there is not an issue of cold start (it has info about the world better than humans). So, only Act would be to make an observation and then sample an action space based on the observation (and the trajectory so far). ReAct just expands the action space with natural language (strong priors required, thus good models are needed). In CoT, even if the reasoning is wrong, it can still sometimes give right answer, but with ReAct, we need evidence. So CoT helps when there is domain knowledge, ReAct is better overall framework. Another common addition is of SELF-REFINE</a> which suggests to generate output however required, wheather its direct prompting, CoT, self consistency, ReAct, whatever, but once you arrive at solution, just reflect and introspect. For an example, suppose you have a code editing agent. Even if the task was to generate some code, after generating output we can agent can introspect on complexity, to refine the output even more. And the feedback can be multi-dimensional as well, like coherence, relevance, etc, and the iteration goes on until the output is solid enough. Feedback here is hard actually, because model needs to understand it. Another extension was Reflexion</a> where the self introspection happens using tools involved as well which is more practical and recovery from failure was better. But a thing to keep in mind is that , CoT prompting improves performance on many reasoning tasks, but it should not be treated as a correctness guarantee</a>. Models don’t always say what they think</a>, so prefer Faithful CoT</a> which tries to translate reasoning into symbolic or executable steps that can be checked by a deterministic solver, which is much closer to production correctness than simply asking the model to “think step by step.” So, Precise agent = controller + state ledger + tools + retrieval + verifier + compactor where, controller: decides the next action state ledger: stores objective, constraints, decisions, assumptions, artifacts tools: interact with the world retrieval: selects external evidence verifier: checks claims/actions against tests or evidence compactor: updates state without losing invariants the modern way</a></h3> originally, model had only one channel for text tokens. so reasoning was hacky with prompts like “think step by step” and by stuffing CoT text into same stream of tokens as final answer (or you manually filter it). The thought is mixed with normal tokens, the final answer and internal work is smudged together. prompt + visible text (includes reasoning and answer) → next prompt</code> these days for reasoning-focused models and APIs we have a native reasoning channel. The model produces: User-visible content: what the end user sees.</li> Reasoning content: a structured trace / scratchpad, possibly large, not meant for the user. This reasoning content - has its own token budget, can be passed back into the next call so the model remembers how it was thinking, without dumping all that into user-visible tokens, may be encrypted / hidden across providers in production prompt → {answer_channel, reasoning_channel}</code> and in next turn {new_observation, previous_reasoning_channel} → new {answer, reasoning}</code> to new {answer, reasoning}</code></li> </ul> Why this system design?</a></h2> just wait until your 2-hour agent run fails at step 99 of 100, and you will get the answer! When we are building short agents, a simple retry loop is often enough. Each agent call is isolated, cheap, and stateless. Fail, retry, done. But retrying a long-running agent spanning hours is expensive in API costs, tool calls, and wall clock time, and retries can also have side effects involving UX. Some failures that a long-running agent might hit in production are: Context window overflow (this is what I am write about)</li> trust boundary collapse (tool outputs can contain destructive things)</li> LLM provider rate limiting (really common)</li> Network timeouts for downstream calls</li> Pod eviction and rotation due to OOM that can happen anytime</li> Human-in-the-loop pauses for god knows how many hours (worse, this dont even fire alerts, just sits silently, no timeout, no escalation, no dead letter queue)</li> </ul> Classic backend engineering problems to solve, where we need checkpoint recovery, idempotency, durable event logs, distributed retries, timeout orchestration and resumable workflows. we need to build robust, fault-tolerant systems by applying hundreds of patterns. System design is for operational challenges, not just fancy multi-agent orchestration. A long-running agent is basically distributed systems engineering wearing an AI hoodie, even nastier as LLM workflows are slower, probabilistic, and dependency-heavy. The context problem</a></h2> You must have felt this! in long-running sessions, there is a point where the agent starts drifting. It forgets a constraint you shoved into its throat minutes ago. It calls the same tool again, with the same inputs. A decision from step two gets contradicted at step nine. Its a U-shaped performance curve, LLM performs best when the info occurs at beginning (primacy bias) or end (recency bias) of input context, and worst when the info is located in middle.</a>. Needle-in-a-haystack is too easy because it often rewards lexical matching. Real agents need semantic retrieval, contradiction handling, state tracking, and reasoning over noisy histories. Anthropic reported a Claude Code issue where older reasoning blocks were accidentally cleared on every turn</a>. Users saw forgetfulness, repetition, odd tool choices, and cache misses. The key lesson was that context-management bugs can look like “the model got dumber,” even when the underlying model did not change. In RAG as well, the retriever has to be good enough because the top K answers that we get should have the answers in it. But if the context window is large enough, then we could actually just have the entire document in the context itself, and we won’t need a very accurate retriever. We won’t really need fine-tuning as well, because we could include all the tasks and examples and all in the context window. Even if the retriever performance is really good in your RAG, it does not matter because the model, the agent, will not be able to see the retrieved documents in the middle. It does not really matter. Basically you are burning tokens and getting worse answers. Well, welcome to the world of context collapse. Every frontier lab is trying to push for more context windows (just to give you a sense 128k tokens is around 100 pages of doc and 1M is around 2500 pages). wherever it is, the hard limit, degradation starts long before it is reached. Better the context, better the reasoning</a> (that is, infer new assertions from a set of assertions integrating multiple knowledge sources, gettng new conclusions) will work, better the tool and MCP calls, and more, common sense. Being a monkey and just increaing the window wont help much in long term. Yet, every announcement leads with the context window. 1M tokens. 2M tokens and entire codebases in one prompt. The implied message is always the same, if the model can read more, it will reason better, which is simply misleading. Models claiming long context lengths can still degrade substantially as context grows, especially on multi-hop, aggregation, and variable-tracking tasks rather than simple needle retrieval</a>. The problem is not that context windows are short. The problem is that context is a bad substitute for state. A 1M-token context window can contain the answer and still fail if the answer is buried, contradicted, stale, or surrounded by irrelevant tool output. Long context gives the model more material to condition on; it does not guarantee retrieval, prioritization, consistency, or verification. For high-precision agents, the central engineering problem is not “how do we fit more text?” but “how do we maintain a correct working set?” Chatgpt’s memory feature gives persistent context across separate conversations. There are two distinct components working together. The first one is chat search, which lets the model retrieve relevant snippets from the past conversations using RAG. There is also memory summary thats given to models but it compresses away details the agent may need later. These burn token really fast as well as add noise. A critical architectural fact is that every API request is completely stateless. The server holds no session generally. When you’re chatting on chatgpt, the application re-sends the entire conversation history with each new turn (server-side threads, memory, prompt caches, summaries, or tool state). Turn 2 ships Turn 1 + your new message. Turn N ships all N-1 prior turns plus your message. Context grows linearly. The client manages it, not the model. This feels less agentic to me! Long context and RAG solve different problems</a>. Long context increases the amount of text the model can inspect in one call. RAG changes the working set by selecting what should be inspected. When the whole corpus is small, relevant, and affordable to pass in, long context can beat retrieval. When the corpus is large, dynamic, private, or noisy, retrieval is still a state-management primitive, not just a token-saving trick. A context window is closer to RAM than to memory. It is the model’s working set for the next forward pass. It is expensive, temporary, order-sensitive, and easily polluted. Long-term memory, retrieval indexes, files, databases, tool logs, and summaries are different layers of the memory hierarchy. A good agent does not paste everything into RAM. It pages in the right state, keeps pointers to bulky artifacts, compresses old traces, and verifies that compression did not destroy important invariants. A clean way to reason about this is : Agent layer</th> Systems analogy</th> What belongs there</th></tr></thead> Active context</td> RAM / working set</td> Current request, state ledger, relevant recent turns, selected evidence</td></tr> Short-term thread state</td> Process-local state</td> Current plan, active subtasks, unresolved questions</td></tr> Long-term memory</td> Database</td> Durable user/project facts with provenance</td></tr> Tool logs</td> Append-only event log</td> Raw observations, commands, search results, actions</td></tr> Retrieval index</td> Search engine</td> Documents, code, previous traces, policies</td></tr> Compaction summary</td> Checkpoint</td> Lossy but structured continuation state</td></tr> </tbody></table> Contact between agents</a></h3> There has to be structured output between the agents, otherwise systems becomes really fragile. Agent A: "I think the issue is probably in auth, maybe retry with the new token." Agent B: interprets that differently. Agent C: loses the uncertainty. Agent D: treats the guess as fact. -> total context collapse</code></pre> If one agent retrieves evidence, another verifies it, another plans tool calls, and another writes the final answer, they cannot pass around loose natural language and hope the next agent interprets it correctly. A handoff should specify: task_id</li> agent_role</li> input assumptions</li> output decision</li> evidence used</li> confidence</li> uncertainty</li> open questions</li> required next action</li> forbidden next actions</li> provenance</li> permissions</li> expiry / validity window</li> </ul> Structured output gives the system five things : First, type safety. The next agent knows whether it received a decision, a draft, a fact, a hypothesis, an error, or a request for clarification.</li> Second, auditability. You can inspect what each agent believed, what evidence it used, and why it made a decision.</li> Third, verification. A verifier can check fields like confidence, source, permissions, and validity instead of parsing messy prose.</li> Fourth, latency control. Structured outputs can be smaller than verbose natural-language traces and easier to route.</li> Fifth, failure isolation. If one agent produces invalid JSON or misses a required field, the orchestrator can reject the handoff before the mistake propagates.</li> </ul> In serious agentic systems, agents can use language to reason, but they should use schemas to coordinate. Free-form handoffs are where uncertainty, permissions, evidence, and state quietly disappear. Tool ambiguity is context pollution</a></h3> More tools do not always make an agent more capable. If five tools can answer the same question, the agent now has to solve a tool-selection problem before it solves the user’s problem. Overlapping tools create ambiguity, retries, inconsistent behavior, and hidden failure modes. Production agents need a small number of sharp tools with: clear purpose</li> clear input schema</li> clear failure modes</li> clear permission model</li> clear output contract</li> clear verification strategy</li> </ul> The tool-gateway pattern is generally best practice as MCP servers expose power. They may access files, databases, cloud systems, Slack, GitHub, internal metrics, customer records, or production infrastructure. The gateway gives us a central control point. whenever the agent wants to use an external tool, it has to take approval from policy engine (a simple backend servic generally with rules, context and api contracts), and if its approved, mint a JIT token and then execute the action and log for auditing purpose. Some general gateway checks are : Gateway checks: Is this user allowed to access payments deployment logs?</li> Is this tenant/project correct?</li> Is this tool allowed for this agent?</li> Are the arguments safe?</li> Is the rate limit okay?</li> Should this call be logged or redacted?</li> </ul> A tool gateway generally handles : authentication</li> authorization</li> tenant boundaries</li> tool allowlists</li> argument validation</li> secrets management</li> rate limiting</li> audit logging</li> response redaction</li> human approval for risky actions</li> tool versioning</li> tool discovery</li> policy enforcement</li> timeout/retry/circuit breaking</li> </ul> A simple flow to understand : User asks question ↓ Agent decides it needs a tool ↓ Agent runtime invokes MCP client ↓ MCP client sends a JSON-RPC tool call ↓ Optional tool gateway intercepts / routes / authorizes ↓ MCP server receives the call ↓ MCP server translates MCP call into actual backend call ↓ Backend result returns back through the same path. MCP server wraps result into MCP response. ↓ Gateway logs/redacts/validates response. ↓ MCP client returns tool result to agent runtime. ↓ Agent uses result in next reasoning step</code></pre> Conclusively, tool design is behavior design. MCP standardizes the shape of tool access. The tool gateway governs whether that access should happen. failure taxonomy</a></h3> Context failure is not just forgetting. It is misprioritization. The model may have the fact, but not treat it as the controlling fact. Failure mode</th> What it looks like in an agent</th> Engineering mitigation</th></tr></thead> Position bias</td> The model misses facts buried in the middle of a long transcript</td> Put critical state near the end or in a structured state block</td></tr> Context dilution</td> Relevant facts are present, but drowned by irrelevant text</td> Retrieve focused snippets instead of dumping whole logs</td></tr> Context pollution</td> Tool output, stale plans, or old assumptions bias the next step</td> Clear or summarize bulky tool results; use TTLs and source IDs</td></tr> Context clash</td> Old and new instructions conflict; model obeys the wrong one</td> Maintain an explicit decision log and supersession rules</td></tr> Compaction loss</td> Summary drops a tiny but crucial constraint</td> Use schema-based compaction with required fields</td></tr> Memory staleness</td> Long-term memory recalls a fact that used to be true</td> Store timestamps, provenance, and invalidation rules</td></tr> Cross-agent divergence</td> Multiple agents hold different partial views of the task</td> Prefer shared state ledgers or single-threaded control for precise tasks</td></tr> Verifier absence</td> The answer sounds plausible but no one checks it</td> Add tests, citations, deterministic tools, or external validators</td></tr> </tbody></table> As a mantra, use long context for local coherence. Use retrieval for selection. Use memory for persistence. Use verification for correctness. Cool Context Engineering</a></h2> Yeah, baby, I’m talking about managing that context window. Here the goal is to pass the right subset of conversation, state, memories, facts, and tool results. The main active context, or the working memory that is actually in the form of a prompt sent to the model for the action, should contain: the system instructions</li> the user’s current requests</li> the recent dialog inputs</li> the active task states</li> the retrieved memories</li> the retrieved documented tool results</li> the current plan or unresolved to-dos</li> the output format constraints</li> </ul> This is the most expensive memory tier because it’s a working memory tier, and everything here will be competing for attention. This is compaction. When a conversation approaches the context limit, just summarize the critical contents and restart with a compressed context, preserving architectural decisions, unresolved bugs, and important implementation details while dropping redundant tool outputs. there are three levers of compaction - (1) summarization (cheap, lossy), (2) selective truncation (drop old messages, risky), (3) model-native compaction like Codex uses (preserves latent state, best quality). In case there is user data retension problem as calling /responses/compact endpoint when the token count exceeds auto_compact_limit returning a new, smaller list of input items that represents the conversation, so data is shared to server, we can opaque encrypted_content blob that encodes the model’s latent understanding of everything that happened, and use decryption keys when required. Client integration is an interesting bit here! rather than returning a single response, prefer emitting a stream of typed events with lifecycle markers. This makes partial rendering, error recovery, and audit logging all much easier to implement. Taking example of codex, it runs the App Server inside a provisioned container, a worker checks out the workspace, launches the App Server binary, and maintains a JSON-RPC channel; the browser talks to the backend over HTTP+SSE. The App Server is long-lived process that hosts Codex core threads and exposes them to clients via a bidirectional JSON-RPC protocol over stdio. It acts as both the transport layer and the translation layer between client requests and low-level agent events. This means the agent keeps running even if the browser tab closes, and a reconnecting session can catch up from the persisted thread history. This is the answer to where the state should leave! If we run the agent in the browser tab, the tab closing kills the session. If er run it server-side, you get persistence but need a reconnect mechanism. I think codex answers it best (i am baised maybe, love gpt5.5 on codex tbh). We run server-side, stream events over SSE, persist thread history, that is exactly how I would design any long running job system (like CI pipelines, batch processing, or async workflows). Next, there could be short-term memory sessions, which could be thread-scoped. It would remember the current conversations or workflows, but it will not necessarily persist forever. It could be a checkpoint-like thread scope checkpoint. The short-term memory would be part of the agent state and would be persisted per conversation thread. There is also a long-term semantic memory, which stores durable facts like Cursor rules; it is a durable fact. There could also be episodic memories to store past experiences and not just facts. It could be in the STAR format. Episodic memories can be used to model successful interactions, and they could be preserved as learning examples. There could also be a procedural memory to store how the agents should be doing things, and maybe an archival memory for the large external searchable store. Prompt changes are not copy edits. They are behavioral changes. A one-line instruction to be shorter, stricter, or less verbose can reduce reasoning quality if it changes how the model allocates effort. Prompt caching is critical for efficiency, static content (instructions, tools) lives at the front of the prompt so each new turn gets a cache hit on all prior context. Tool ordering bugs cause expensive cache misses. We can’t treat all context as equal. If the recent chat turns, old memories, code, documentation, metrics definitions, tool outputs, and runtime observations all get thrown into the same prompt, then it’s a spaghetti! Production agents need context hierarchy. A high-precision agent should know which source wins when context conflicts: Current runtime state</li> Authoritative source-of-truth systems</li> Code and lineage</li> Curated human annotations</li> Institutional docs</li> Prior memories</li> Conversation history</li> Model prior knowledge</li> </ol> The agent is not just retrieving context. It is deciding which context is allowed to govern behavior. Bad memory systems store everything. Good memory systems store corrections that would otherwise be rediscovered painfully. the goal of memory is to retain non-obvious corrections, filters, and constraints that are critical for correctness but hard to infer from other layers. Designing memory for agentic systems</a></h2> The mistake is treating memory as storage. Memory is not storage. Memory is state selection under constraints. The hard problem is not “can we save this conversation?” The hard problem is “should this fact influence the next action?”. The memory, retrieval, verification, latency, privacy, and session boundaries all depend on structured state between agents that I discussed earlier, not vibes. Things to think about : Proper context retention strategy</a></h3> we already discussed that the active context should contain system instructions, current request, recent dialogue, active task state, retrieved memories, tool results, current plan, unresolved TODOs, and output constraints. That is good, but it needs a retention policy: what stays hot, what gets summarized, what gets archived, and what gets dropped. A production agent needs a retention strategy. Every piece of context should be assigned to one of four buckets: Hot context - Always included in the next model call.</li> Warm context - Included only when relevant to the current task.</li> Cold context - Stored externally and retrieved on demand.</li> Dead context - Dropped or expired because it is no longer useful, safe, or valid.</li> </ol> Hot context should contain: system/developer instructions</li> current user request</li> active task objective</li> current state ledger</li> latest decisions</li> open constraints</li> recent relevant turns</li> required output format</li> </ul> Warm context should contain: session summary</li> recent tool results</li> relevant prior decisions</li> unresolved subtasks</li> retrieved memories</li> </ul> Cold context should contain: old tool traces</li> historical conversations</li> previous artifacts</li> old retrieved documents</li> prior task attempts</li> </ul> Dead context should include: superseded instructions</li> stale tool outputs</li> redundant logs</li> old intermediate plans</li> failed hypotheses</li> sensitive data past its retention window</li> </ul> Retrieval strategy</a></h3> Retrieval is not just vector search. A serious agent needs hybrid retrieval, authority ranking, recency logic, and contradiction handling. The agent should not remember everything. It should retain the minimum state needed to continue the task correctly. For agent memory, retrieval has to answer four questions: What is relevant?</li> What is current?</li> What is authoritative?</li> What is safe to show this user?</li> </ol> Vector similarity only answers the first question, and even that imperfectly. A production retrieval strategy should combine: recency retrieval for recent turns and latest decisions</li> semantic retrieval for conceptually related memories</li> keyword/BM25 retrieval for exact IDs, names, errors, files, and commands</li> entity retrieval for user, project, customer, repo, ticket, or patient-specific facts</li> temporal retrieval for facts that changed over time</li> authority ranking for deciding which source wins during conflicts</li> access filtering before anything enters the prompt</li> </ul> A useful retrieval score takes care of eviction properly : score = semantic_similarity keyword_match</li> entity_match</li> recency_boost</li> source_authority</li> memory_importance</li> </ul> staleness_penalty</li> contradiction_risk</li> privacy_risk</li> token_cost</li> </ul> Retrieval should happen before context assembly, but permission checks should happen before retrieval results are exposed to the model. Latency awareness</a></h3> A production agent needs to know where latency actually lives. User query ↓ Query preprocessing (normalize the query, classify intent, extract entities, detect the current task, rewrite the query, or decide whether memory retrieval is even needed), for example, is this asking about the current session or old memory?, is this asking about the current session or old memory? should we retrieve documents, memories, tool logs, or all of them?, is this a cheap edit request that does not need retrieval?, and more. This step is usually cheap if it is rule-based. It becomes expensive if every request calls another LLM just to decide how to retrieve. A good production system should not run the full retrieval pipeline for trivial turns like. ↓ Query embedding (if semantic retrieval is needed, the query has to be embedded, hosted embedding APIs are simpler to operate, adds network latency, can have rate limits, may have variable p95/p99 latency, whereas local embedding model has lower network overhead, easier to batch, more infra complexity, consumes CPU/GPU resources. For high-throughput systems, embedding can quietly become a bottleneck, especially true when the agent performs multiple retrievals per user turn. - one embedding for the user query, one embedding for rewritten query, one embedding for entity-expanded query, one embedding for hypothetical answer retrieval, and what not. That may improve recall, but it also increases latency and cost. ↓ Vector / keyword retrieval (usually fast, but not always free. Latency depends on index size, embedding dimension, ANN setting, network distance to the vector DB, and more) ↓ Metadata filtering (pre-filtering before vector search is usually better for privacy and efficiency) ↓ Reranking (this imprved quality but iws quite costly, - In case of low-risk queries, you can actually not use any re-ranker and use the recency plus metadata plus spectra scores, if it's a medium-risk query, you can use a light-weight re-ranker, if it's a high-risk query, you can use a stronger re-ranker plus verifier) ↓ Prompt construction (Does this context improve the next action enough to justify its token and latency cost?) ↓ LLM prefill (Long memory-heavy prompts hurt prefill. Long answers hurt decode.) ↓ LLM generation ↓ Post-processing ↓ Response</code></pre> Memory is not free. Every memory layer adds latency: reading checkpoints from Postgres</li> searching a vector DB</li> running BM25</li> reranking candidates</li> fetching documents</li> decompressing summaries</li> checking permissions</li> assembling the prompt</li> computing the prefill tokens</li> </ul> A memory system that improves accuracy but adds 2 seconds to every turn may be unusable for interactive agents. A production context assembler needs a budget: 20–50 ms: load session state</li> 50–150 ms: retrieve recent/warm memories</li> 100–300 ms: hybrid search</li> 100–500 ms: reranking, only for high-value tasks</li> 0 ms: skip retrieval if the current turn can be answered from hot state</li> </ul> The agent should not run the full memory pipeline on every turn. Use Memory routing examples: “make it shorter” - Use recent turns only. No vector retrieval.</li> “continue from where we left off” - Use session summary + recent state.</li> “what did we decide last week?” - Use long-term memory + event-log retrieval.</li> “run the same analysis as before” - Use episodic memory + artifact references.</li> “what is the current status?” - Use runtime tools, not old memory</li> </ul> This connects to prefix caching as well. vLLM’s Automatic Prefix Caching reuses KV cache when requests share the same prefix, which means stable prompt layout can reduce redundant prefill computation. Latency-aware memory design also means prompt layout matters. Put stable content first: system instructions</li> tool schemas</li> durable project rules</li> stable memory block Put volatile content later:</li> retrieved snippets</li> current state</li> current user request</li> </ol> This improves prefix-cache hit rate. If you inject timestamps, random IDs, or changing retrieval blocks before the stable prefix, you destroy cache reuse. Privacy and data boundaries</a></h3> An agent memory system is dangerous if it turns private data into global context. Every memory needs scope. Memory scope should be explicit: user-scoped memory</li> session-scoped memory</li> project-scoped memory</li> organization-scoped memory</li> tenant-scoped memory</li> public memory</li> </ul> A memory should never be retrieved into a context unless the current user, session, and task are allowed to see it. Effective memory access = user permissions ∩ tenant boundary ∩ project boundary ∩ data classification policy ∩ tool permission ∩ current task need Every stored memory should carry metadata: memory_id: type: scope: owner: tenant_id: project_id: source_event_id: created_at: last_seen_at: valid_until: sensitivity: pii: true|false retention_policy: can_train_on: true|false can_cross_session_retrieve: true|false</code></pre> Memory is not just a product feature. It is a data-governance surface. Memory pruning and decay</a></h3> Long-term memory is useful only if it can forget. Without pruning, memory becomes a junk drawer: stale preferences</li> old project decisions</li> outdated APIs</li> superseded instructions</li> duplicate facts</li> temporary constraints</li> failed assumptions</li> </ul> Each memory should have a lifecycle: Candidate memory - Extracted from a conversation or tool result.</li> Validated memory - Passed checks for usefulness, scope, and sensitivity.</li> Active memory - Eligible for retrieval.</li> Decaying memory - Still stored but receives a lower retrieval score.</li> Superseded memory - Replaced by a newer fact.</li> Archived memory - Searchable only in explicit historical queries.</li> Deleted memory - Removed because it is unsafe, expired, or user-requested.</li> </ol> retrieval_priority = importance frequency_of_use</li> recency</li> source_authority</li> user_confirmation</li> </ul> age_penalty</li> contradiction_penalty</li> sensitivity_penalty</li> low_usage_penalty</li> </ul> Pruning jobs should run periodically: merge duplicate memories</li> delete low-value memories</li> downgrade stale memories</li> mark superseded memories</li> refresh memories from source-of-truth systems</li> remove memories past retention limits</li> detect contradictions between old and new facts</li> </ul> Do not let the model silently create permanent memory from every conversation. Memory write policy: ephemeral session fact: auto-write</li> project decision: write with provenance</li> user preference: write if repeated or confirmed</li> sensitive personal data: require explicit consent</li> procedural rule: require human review</li> security/medical/legal rule: never silently write</li> </ul> Sessions need hard boundaries</a></h3> A session is not just a chat window. A session defines: what the agent is trying to do</li> what state is currently active</li> which tool results are relevant</li> which temporary assumptions are valid</li> which memories are allowed to influence behavior</li> when the workflow is over</li> </ul> Session types: Single-turn session - No durable memory. Answer and forget.</li> Multi-turn chat session - Keep recent turns, active preferences, and local context.</li> Long-running workflow session - Persist checkpoints, state ledger, artifacts, tool results, and resumable progress.</li> Project session - Share selected memory across multiple workflows.</li> Cross-session user memory - Store stable user preferences and durable facts only.</li> </ol> When a session ends, the system should decide: What should be summarized?</li> What should become durable memory?</li> What should remain only in the raw event log?</li> What should be deleted?</li> What should be hidden from future sessions?</li> What should require user approval before persistence?</li> </ul> Add a concrete session closeout schema: session_closeout: objective_completed: true final_state_summary: "" durable_decisions: - "" user_preferences_detected: - preference: "" confidence: 0.72 write_policy: ask_user artifacts_created: - id: "" type: "" location: "" memories_to_create: - content: "" scope: project ttl: 90_days provenance: event_id memories_to_expire: - memory_id: "" reason: superseded unresolved_items: - ""</code></pre>Conclusion</a></h3> If someone asks, “How would you design memory for an agentic AI system?”, the answer should not be “I’ll just store the chat history.” A production memory system needs layers. I would start with short-term session memory for the current task: recent turns, active goals, open constraints, intermediate tool results, and the current state ledger. This is the agent’s working memory. Then I would add entity-scoped memory for durable facts about users, projects, organizations, repositories, tickets, customers, patients, or other domain objects. These facts should not live as vague text blobs. They should have metadata: source, timestamp, owner, tenant, confidence, sensitivity, and validity window. Then I would add retrieval-gated long-term memory. Older conversations, tool traces, decisions, summaries, and artifacts should be stored externally and retrieved only when relevant. Retrieval should use semantic search, keyword search, entity filters, recency, and source authority — not just top-k vector similarity. I would write to memory conservatively. Not every conversation turn deserves to become permanent memory. Temporary assumptions should stay in the session. Durable facts should require confidence, provenance, and the right scope. Sensitive or high-impact memories should require user approval or policy checks before being persisted. At read time, I would retrieve with strict metadata filters. Only after those boundaries are enforced would I run semantic retrieval and reranking. I might use a relevance threshold such as 0.7, but that threshold should be calibrated against evals, not chosen blindly. Below the threshold, the memory should not enter the prompt. I would also make memory decay over time. Entity facts could age on a 30-day half-life unless reaffirmed. Old preferences, stale project decisions, deprecated APIs, and superseded tool results should gradually lose retrieval priority. If a newer source contradicts an older memory, the older one should be marked superseded instead of being retrieved as if it were still true. Finally, I would namespace memory by tenant, user, project, and session at the read boundary. The agent should never retrieve memory across privacy boundaries just because the embedding search says it is similar. Memory is not just storage. It is a permissioned, time-aware, relevance-gated state system. A good agent memory system is conservative on writes, strict on reads, aware of time, aware of privacy, and optimized for the current task. It remembers enough to be useful, forgets enough to stay correct, and never lets stale or unauthorized context silently steer the agent. Do you real;y know your agent?</a></h2> A complete understanding of agents requires automated evals, production monitoring, A/B testing, user feedback, manual transcript review, and systematic human studies rather than a single metric.</a>. We need to observe state transitions, not just the final response! What context was selected? What evidence was retrieved? What tool was called? What arguments were passed? What state changed? What verifier ran? What constraint was dropped? What grader made the final judgment? An eval score without a trace is almost useless for agent engineering! We need a debugging surface, not just a score. Basically when the agent stops being predictable, and there is fall in accuracy and lots of hallucinations, it’s often a failure in the environment, or more precisely, in the system that supports the agent. Not everything is about tweaking models, at this point they are really powerful. You’re spending real money on LLM calls, your LangSmith dashboard is filling up with traces, but still you can’t confidently answer basic questions about where your budget is actually going. Which calls are wasteful? Which prompts are bloated? Is that expensive run the norm or an outlier? You have hundreds of traces and gigabytes of span data, but there’s no easy way to query across it. Agent evals are not just benchmarks or worse a leaderboard number, they are observability systems. A useful agent eval should produce a structured trace: what context the model saw, which memories were retrieved, which tools were called, what arguments were used, what state changed, what verifier ran, what grader scored the output, and which exact version of the model/prompt/tool schema produced the result. The question is not only “did the agent pass?” The question is: Did it see the right evidence?</li> Did it use the evidence?</li> Did it call the right tool?</li> Did the tool return the right state?</li> Did compaction drop a constraint?</li> Did memory inject stale information?</li> Did the grader reject a valid solution?</li> Did infra noise change the result?</li> </ul> eval-driven dev maybe? task-specific evals, logging everything so logs can become eval cases, continuous evaluation, and calibrating automated scoring with human judgment. Telemetry schema I used in one of my past projects : eval_run: run_id: "" eval_suite: "" dataset_version: "" task_id: "" task_family: "" difficulty: "" created_at: "" system_under_test: model: "" model_version: "" prompt_version: "" tool_schema_version: "" retrieval_index_version: "" memory_policy_version: "" compaction_policy_version: "" agent_harness_version: "" sampling: temperature: 0 top_p: 1 max_tokens: 0 context_trace: input_tokens: 0 output_tokens: 0 context_window: 0 system_prompt_tokens: 0 tool_definition_tokens: 0 memory_tokens: 0 retrieval_tokens: 0 conversation_tokens: 0 summary_tokens: 0 truncated_items: [] retrieved_items: - id: "" source: "" score: 0 included: true reason: "" compaction_events: - before_tokens: 0 after_tokens: 0 preserved_fields: [] dropped_fields: [] agent_trace: turns: 0 tool_calls: - call_id: "" tool: "" args_hash: "" status: success|error|timeout latency_ms: 0 side_effect_summary: "" handoffs: [] retries: 0 stop_reason: "" state_trace: initial_state_hash: "" final_state_hash: "" state_diff: {} files_changed: [] database_changes: [] external_side_effects: [] grader_trace: graders: - type: code|model|human|state_check version: "" score: 0 pass: true rationale_summary: "" judge_model: "" rubric_version: "" human_labeler_id: "" grader_disagreement: false runtime_trace: latency_ms: 0 cost_usd: 0 provider: "" region: "" hardware: "" container_image: "" timeout_ms: 0 memory_limit: "" cpu_limit: "" infra_errors: [] outcome: pass: true failure_category: "" notes: ""</code></pre> This schema makes failures queryable. You can ask: “Show me all failures where the right document was retrieved but not cited,” “Show me all failures after compaction,” “Show me all retries caused by one tool,” or “Show me tasks where the grader and human disagreed.” OpenTelemetry is moving toward this kind of standardized instrumentation for generative AI, its GenAI conventions define spans, metrics, events, model spans, agent spans, and provider-specific conventions, and its GenAI span spec includes attributes such as operation name, provider, model, conversation ID, output type, and error type.</a> btw, agent evals are infrastructure-sensitive</a>! A model can score differently because of container memory, CPU limits, dependency installation, timeout policy, sandbox provider, cache state, or serving configuration. If the eval harness is not controlled, you may be benchmarking infrastructure headroom rather than intelligence. So, we should log resource limits, timeout policy, container image, dependency state, hardware class, cache state, model version, harness version, and retry policy, in production systems. core quality and reliability metrics : Metric</th> What it tells you</th></tr></thead> Pass rate</td> Overall success</td></tr> Pass@k / pass^k</td> Reliability across repeated attempts</td></tr> Failure category</td> What kind of thing broke</td></tr> Context tokens by source</td> Whether memory/retrieval/tool logs are bloating context</td></tr> Relevant evidence included?</td> Whether retrieval/context assembly worked</td></tr> Relevant evidence used?</td> Whether the model grounded its action</td></tr> Tool-call precision</td> Whether tool calls were necessary and correct</td></tr> Tool-call recall</td> Whether required tools were skipped</td></tr> Duplicate tool-call rate</td> Whether the agent is looping or forgetting</td></tr> State-diff correctness</td> Whether the external world ended correctly</td></tr> Compaction survival rate</td> Whether summaries preserve required constraints</td></tr> Grader disagreement</td> Whether the scoring method is unstable</td></tr> Human/LLM judge agreement</td> Whether automated grading is calibrated</td></tr> p50/p95/p99 latency</td> Whether quality changes trade off against UX</td></tr> Cost per successful task</td> Whether the agent is economically viable</td></tr> Infra error rate</td> Whether failures are model failures or system failures</td></tr> Slice performance</td> Which task families regress</td></tr> </tbody></table> context specific : Eval</th> How to run it</th> Failure detected</th></tr></thead> Position sweep</td> Move the same key fact to beginning/middle/end of context</td> Lost-in-the-middle / position bias</td></tr> Distractor injection</td> Add irrelevant but plausible tool logs</td> Context dilution</td></tr> Contradiction update</td> Old instruction says A; later valid decision says B</td> Stale-state obedience</td></tr> Compaction survival</td> Force summary, then test old constraint</td> Summary loss</td></tr> Memory staleness</td> Store old fact, then supersede it</td> Bad long-term memory invalidation</td></tr> Tool deduplication</td> Give prior tool result, see if agent repeats call</td> Forgetful loops</td></tr> Handoff test</td> Move task between agents</td> Cross-agent state loss</td></tr> Long-run pass^k</td> Run the same task repeatedly across long trajectories</td> Reliability, not one-off success</td></tr> </tbody></table> For agents, cost/request is less useful than cost/successful task. A cheaper model that loops, repeats tool calls, or fails silently can be more expensive than a stronger model that finishes once. You cannot know whether an agent improved unless you test it against known cases. For deterministic software, we write unit tests. For agents, we need evals: golden questions</li> expected tool calls</li> expected source selection</li> expected final result</li> acceptable reasoning path</li> unacceptable failure modes</li> semantic graders, not string matching</li> </ul> The eval should not only ask, “Was the final answer plausible?” It should ask: Did the agent choose the right source?</li> Did it retrieve the right context?</li> Did it use the right filter?</li> Did it avoid stale memory?</li> Did it detect bad intermediate results?</li> Did it cite the correct evidence?</li> Did it escalate when uncertain?</li> </ul> llm as a judge</a></h3> I’m not very keen on this, but it simply is like using a strong model to evaluate a weaker model’s output. In here, there is often a loss of diversity if you do LLM as a judge. Considering other things, how common it is being used, it’s more important to have a good scoring prompt than the model itself. A vague prompt like “simply rate this response” will produce a very noisy score. A structured prompt with rubrics will produce consistent, reproducible scores, but there are a lot of failure modes in this as well. Judge models often exhibit position bias, i.e., they prefer the first response in the pair-wise comparisons. There is a verbosity bias as well, so it prefers longer responses, and there are weird self-preferences like GPT 5.5 would rate GPT 5.5 higher than the equivalent Claude Opus 4.7 outputs. To minimize that, you can: use randomization of order</li> normalize for the length</li> use a different judge than the model being evaluated</li> </ul> The famous RAGAS is there as an evaluation framework specifically for RAG pipelines, which measures the faithfulness, relevance and the answer correctness? So that’s there as well if you are playing with RAG Pipelines. Interestingly, there is something called prompt foo as well, so it’s a config-driven eval for prompt engineering. What it does is it simply defines test cases in YAML and runs against multiple models and gets a pass/fail report. Often used for regression test prompts, so that a prompt change does not break the existing test cases The agent should never be more authorized than the user</a></h2> A production agent is not a magic superuser. It should inherit the user’s permissions, not bypass them. The security rule is simple: Agent access = user permissions ∩ tool permissions ∩ policy permissions.</code></pre> If a user cannot read a table, document, patient record, ticket, or customer account directly, the agent should not be able to retrieve it on their behalf. This matters because agents combine information. Without pass-through permissions, an agent can accidentally become a data exfiltration system. running harness</a></h2> codex has everything, CLI, a web app, and a macOS desktop app, all with underlying harness that is a Rust lib containing the agent loop, thread lifecycle (create/resume/fork/archive), config and auth management, and sandboxed tool execution</a>. the harness manages the full lifecycle of one conversation thread, including persisting the event history so clients can reconnect and render a consistent timeline, a stable protocol that exposes this runtime to any client, language or platform agnostic. memory systems decay</a></h2> A very subtle failure mode in agentic systems is embedding-space drift. Teams often treat embeddings as timeless. They are not. A vector is only meaningful relative to the model, preprocessing pipeline, chunking strategy, normalization method, and distance function that produced it. If the document vectors were created with one embedding model and the user query is embedded with another, search quality can silently degrade. The system may still run. The vector DB may still return results. Nothing throws an error. But recall gets worse, the wrong chunks appear near the top, and users start saying the agent “feels dumber.” OpenAI’s current embedding docs, for example, distinguish text-embedding-3-small and text-embedding-3-large, with different default vector lengths, and also support shortening embeddings using a dimensions parameter. That means model choice and dimensionality are part of the retrieval contract, not incidental implementation details. Every embedded object should carry version metadata: { "embedding_model": "text-embedding-3-large", "embedding_model_version": "2026-05-embedding-config-v2", "dimensions": 1024, "distance": "cosine", "chunker_version": "semantic-chunker-v4", "preprocessing_version": "html-cleaner-v3", "created_at": "2026-05-21T10:00:00Z", "source_doc_version": "doc_982:v17" }</code></pre> Embedding model migration should be treated like a database migration, not a casual config change. A safe embedding migration looks like this: Pin the current embedding model and index config.</li> Create a new vector collection for the new embedding model.</li> Dual-write new documents to both old and new collections.</li> Backfill old documents by re-embedding from source text.</li> Run shadow retrieval against both indexes.</li> Compare recall, MRR, nDCG, answer quality, and latency.</li> Canary the new index on a small percentage of traffic.</li> Roll forward only if evals and production metrics improve.</li> Keep rollback available until confidence is high.</li> </ol> Do not overwrite the old vectors in place! bad idea, do the canary deployment. Old and new embedding spaces should coexist during migration. evals that evolve</a></h2> The hard part of evals is rarely the harness. The hard part is building the dataset, discovering edge cases, comparing configurations, understanding failures, and continuously updating the suite as the system changes. A real eval process is iterative: collect real user questions</li> label expected behavior</li> run multiple retrieval/model/tool configurations</li> compare outputs</li> inspect low-scoring cases</li> identify failure patterns</li> add new test cases</li> improve retrieval, memory, prompts, tools, or policies</li> rerun the suite</li> repeat</li> </ol> An agentic eval should test more than the final answer. It should evaluate: retrieval quality</li> memory selection</li> tool choice</li> tool arguments</li> permission handling</li> intermediate state updates</li> structured handoffs between agents</li> verifier behavior</li> abstention behavior</li> final answer quality</li> latency and cost</li> </ul> The next frontier is evals that detect their own obsolescence. An eval suite becomes obsolete when production behavior changes but the eval set does not. If users start asking new kinds of questions, tools change, documents change, policies change, or the agent starts failing in a new way, the eval suite should notice that its coverage is no longer enough. An eval suite can become stale when: production queries no longer look like eval queries</li> new tools are added but not tested</li> new document types enter the corpus</li> user behavior changes</li> policies change</li> retrieval index changes</li> memory format changes</li> new failure modes appear in logs</li> old “golden answers” become outdated</li> </ul> A stale eval suite gives false confidence. The dashboard stays green while production quality gets worse. A modern agentic eval system should monitor: Query distribution drift - Are production questions still similar to eval questions?</li> Tool-use drift - Is the agent using tools in ways the eval suite does not cover?</li> Retrieval drift - Are retrieved sources changing? Are top-k results less relevant than before?</li> Memory drift - Are stale memories entering prompts? Are useful memories being missed?</li> Failure-cluster drift - Are new classes of failures appearing in production traces?</li> Human-feedback drift - Are users correcting the agent on issues that are absent from evals?</li> Cost/latency drift - Did a configuration improve accuracy but make the system too slow?</li> Policy drift - Did product, legal, clinical, or security requirements change?</li> </ol> A production loop could be : The eval loop should look like this: production traces ↓ failure clustering ↓ candidate eval generation ↓ human review / labeling ↓ eval suite update ↓ configuration comparison ↓ regression gate ↓ deployment canary ↓ monitoring</code></pre>Conclusion</a></h2> A production agent does not only need memory. It needs memory operations. That means: embedding-space versioning</li> index migration plans</li> retrieval quality monitoring</li> evals for retrieval and generation</li> failure clustering</li> dataset refresh workflows</li> shadow runs and canaries</li> rollback paths</li> </ul> Multi-agent systems are attractive because they look like organizations: researcher, planner, coder, reviewer, manager. But splitting work across agents also splits context. Each handoff can drop assumptions, hide tool observations, or create conflicting local plans. Multi-agent architecture helps when the task naturally decomposes and each agent has a clean contract. It hurts when correctness depends on a shared, evolving state. Maybe, don’t use multi-agents if you don’t really need it?</a> Use subagents for bounded questions, not uncontrolled ownership. A subagent can inspect a file, search a corpus, or produce a critique. But the main controller should own the state ledger, decisions, and final action. The future of reliable agents is not just bigger context windows. It is better context discipline. The agent that wins is not the one that can read the longest transcript; it is the one that knows what state matters, what evidence supports it, what assumptions are stale, what actions changed the world, and what must be verified before answering. Long context is useful. But correctness comes from state, selection, and verification. The best production agents are not prompt chains. They are context systems. A production-grade agent usually has: Layered context - It separates schema, code, docs, memory, runtime state, and user conversation instead of merging everything into one blob.</li> Source authority - It knows which source wins when two pieces of context conflict.</li> Code or system-grounded semantics - It learns meaning from the systems that produce the data, not just from documentation.</li> Closed-loop reasoning - It checks intermediate results, detects anomalies, and retries before answering.</li> Scoped memory - It stores non-obvious corrections and constraints, not the whole transcript.</li> Tool discipline - It uses fewer, clearer tools with well-defined contracts.</li> Continuous evals - It has golden cases, regression tests, semantic graders, and production canaries.</li> Pass-through permissions - It never gives the agent more access than the user already has.</li> </ol> This is why long-context alone does not solve agents. A giant context window gives the model more tokens. A production context system gives the model the right facts, in the right order, with the right authority, under the right constraints. The agent will decay unless the memory system and eval system evolve with it. RAG systems rot when embeddings, documents, user behavior, and tools change but the index and evals stay frozen. The goal is not just to build an eval suite. The goal is to build an eval system that tells you when it no longer represents reality. DB Joins on GPUs!! Unknown — Thu, 26 Feb 2026 00:00:00 +0000 I was studying for my High Performance Computing Exams sometime ago, and interestingly, I stumbled upon a slide stating that H100 GPU can do 15 queries/sec and GB200NVL72</a> can do whopping 90 queries/sec compared to just 5 queries/sec on typical x86 architectures. Never thought about this, so digged deep out of curiosity. The idea of GPU hash joins emerged in 2008 academia to harness GPUs’ massive thread parallelism and memory bandwidth for data-intensive OLAP joins, which were memory-bound on CPUs. Bin He et al. pioneered it in Relational Joins on Graphics Processors</a>, implementing the first GPU hash join kernel—up to 100x faster than CPU for 1B-row joins via parallel radix partitioning and atomic inserts. Yess, SQL joins can run very efficiently on GPUs by exploiting massive data parallelism, usually via hash joins or sort‑merge joins that are rewritten as GPU kernels. The Core Idea</a></h2> A join is just a big comparison between rows of two tables: for each key in table A, find matching keys in table B. GPUs are ideal because you can give each row (or chunk of rows) to a separate thread and run tens of thousands of these comparisons in parallel. Traditional CPU hash joins process tables serially: build a hash table from smaller table A (one key at a time), then probe larger table B sequentially. This is memory-bound—hash lookups hit DRAM cache misses constantly. GPUs flip this by launching 10k-100k+ threads simultaneously: all keys from table A hash/insert in parallel during build (using atomic operations to resolve collisions), then all keys from table B probe in parallel. Most joins are equi-joins, which use only equality comparisons in the join-predicate. Hash joins are the most important equi-join physical implemenations in the analytical world. Hash join taking advantage of GPUs would typically involves two steps: Build phase: Use many GPU threads to build a hash table from the smaller input relation in GPU memory. Load the smaller table (build input) into GPU HBM. Launch thousands of threads (e.g., 10^5+ on H100) where each hashes a key and atomically inserts into a GPU hash table (using libraries like cuCollections for synchronization). Shared memory caches frequent accesses to minimize global memory latency</li> Probe phase: Other threads scan the larger table; each row hashes its join key and probes the hash table to find matches</li> </ul> This parallelism exploits GPU’s 3 TB/s HBM bandwidth vs. CPU DRAM’s ~100 GB/s. GPU DBs</a></h2> GPU‑native databases like HeavyDB and Kinetica compile SQL queries (including equi‑joins, multi‑way joins) into GPU kernels that execute these parallel hash or sort‑merge joins over columnar data layouts. They keep hot columns compressed and resident in GPU memory when possible, and otherwise stream partitions from CPU RAM to GPU HBM while overlapping transfer and join computation. HeavyDB on a single GPU outperforms CPU warehouses (Snowflake, BigQuery) by 6-190x on geospatial joins with 100M+ rows, completing in milliseconds what takes hours on CPU due to nested loops. Kinetica supports multi-table joins, offloading to GPUs for 100x analytics speedups via NVLink data streaming. Crazyyyyy Conclusion</a></h2> GPU-accelerated hash joins shine in high-volume OLAP workloads where massive parallelism slashes query times from minutes/hours to milliseconds, but they’re unnecessary for small datasets or transactional OLTP. Just go through a comprehensive overview of GPU DBs</a> if more curious about the paradigm. Which Prompts Actually Work for your Agents? Unknown — Sun, 22 Feb 2026 00:00:00 +0000 I was wondering on a lazy sunday, which parts of my prompt actually matter? Agents (ReAct, tool-calling, multi-step reasoning) depend heavily on system prompts: role, rules, tool descriptions, few-shot examples. It’s easy to bloat them and hard to know what’s redundant. So, I found out about Saliency analysis which gives you numbers, perturb each phrase, see how much the agent’s output changes. High change → that phrase matters; low change → candidate to cut or simplify. So, the goal is to find which parts of your agent’s system prompt actually drive behaviour, then trim the rest and protect what matters. Simple, yet I don’t see a lot of people using it. This is a sensitive issue</a></h2> Recent research has quantified just how sensitive LLMs are to prompt formulation. LLMs show extreme sensitivity to subtle changes in prompt formatting</a>, even after instruction tuning and scaling. The ProSA framework</a> established that: Prompt sensitivity fluctuates unpredictably across datasets and models</li> Larger models demonstrate enhanced robustness, but not immunity</li> Few-shot examples can alleviate sensitivity issues</li> Higher model confidence correlates with increased prompt robustness</li> </ul> This means that two semantically equivalent prompts can produce dramatically different outputs, making prompt engineering a high stakes optimization problem with no clear gradient signal. No real traditional debugging to save me</a></h2> Traditional software debugging relies on: Deterministic execution - Same input → same output</li> Inspectable state - Variables, stack traces, breakpoints</li> Localized effects - Changes propagate predictably</li> </ol> LLM prompts violate all three assumptions: Outputs are stochastic (using temperature=0</code> gives more stable comparisons)</li> Internal model state is opaque (billions of parameters, no interpretable variables)</li> Token interactions are highly non-local (attention spans the entire context)</li> </ol> Perturbation-based saliency addresses this by treating the LLM as a black-box function and inferring input importance from output changes under controlled edits, without requiring model internals. The Maths behind this</a></h2> The Vector Space Model (VSM)</a>, represents text as vectors in a high-dimensional space where each dimension corresponds to a distinct term (or in our case, character n-gram). So, the texts with similar content will have similar vector representations, enabling geometric operations (distance, angle) to capture semantic relationships. The things I am gonna discuss below use character trigrams rather than word-level tokens. A character n-gram is a contiguous sequence of n characters extracted from text. Why? Character trigram overlap is effective for sentence alignment in text simplification tasks</a> Property</th> Benefit</th></tr></thead> Language-agnostic</td> Works for any language without tokenizers</td></tr> Typo-robust</td> Small character changes don’t destroy similarity</td></tr> No external dependencies</td> No NLP libraries required</td></tr> Paraphrase-tolerant</td> Captures subword patterns that survive rephrasing</td></tr> Computationally efficient</td> O(L) extraction, sparse representation</td></tr> </tbody></table> We treat Saliency as Divergence. If a phrase is important to the model’s output, removing or altering it should cause the output to change significantly. Obviously!! More formally, let : $P = [p_1, p_2, …, p_n]$ be the prompt decomposed into $n$ phrases</li> $f: \text{Prompt} \rightarrow \text{Output}$ be the LLM function</li> $P_{-i}$ denote the prompt with phrase $p_i$ perturbed (replaced, removed, or paraphrased)</li> </ul> The saliency score for phrase $p_i$ is: $$S(p_i) = 1 - \text{sim}(f(P), f(P_{-i}))$$ Interpretation: $S(p_i) = 0$: Perturbing $p_i$ causes no change → phrase is redundant</li> $S(p_i) = 1$: Perturbing $p_i$ causes complete divergence → phrase is critical</li> $S(p_i) \in (0, 1)$: Partial influence</li> </ul> This formulation treats saliency as output divergence under intervention, a causal notion that measures the counterfactual impact of each phrase. Perturbation Methods</a></h3> Method</th> What you do</th> API cost</th> When to use</th></tr></thead> Perturbation</td> Replace phrase with [...]</code></td> N+1</td> Default: fast, keeps sentence structure.</td></tr> Omission</td> Remove phrase entirely</td> N+1</td> Short prompts; you want to see effect of full removal.</td></tr> Paraphrase</td> Ask an LLM to rewrite the phrase to be vague, then run agent</td> 2N+1</td> When you care about semantic content only (slower, more API calls).</td></tr> </tbody></table> For agentic workflows, perturbation or omission is usually enough; paraphrase is for deeper semantic analysis when needed. The paraphrase method isolates semantic content from structural presence: Perturbation tests: “What if this phrase were obscured?”</li> Omission tests: “What if this phrase were absent?”</li> Paraphrase tests: “What if this phrase said nothing specific?”</li> </ul> This is the most faithful measure of information contribution because it controls for the structural role of the phrase while zeroing out its semantic payload. When to use it?</a></h3> Before shipping: audit which instructions and tool descriptions the model really uses.</li> When debugging: the agent ignores a rule or uses the wrong tool → check if that part of the prompt has low saliency. See which tool descriptions actually affect tool choice.</li> When trimming: you need a shorter system prompt without losing behaviour → prune by saliency, then re-test.</li> Adding few-shot examples: System prompt before/after adding examples. Check which examples change behaviour; drop ones with near-zero impact.</li> </ul> Why this works?</a></h2> Core idea: If a phrase matters, changing or removing it should change the output. If the output barely changes, that phrase is not pulling much weight. So we perturb one phrase at a time and measure how much the output changes (e.g. with a similarity score). That change is the phrase’s “importance” for that run. </li> Marginal contribution: This is the same idea as “leave-one-out” importance in interpretability: you’re measuring the marginal contribution of each phrase to the outcome. It’s a simple approximation to more formal notions (e.g. Shapley-like attribution) that would average over many subsets; here we only compare “full prompt” vs “without this phrase” (or “with this phrase masked”), which is cheap and usually enough for prompt tuning. </li> Comparing outputs: We need a single number for “how different is output A from output B?” Character-trigram cosine similarity is language-agnostic, has no extra dependencies, and is robust to small wording changes. So: turn both outputs into trigram frequency vectors, compute cosine similarity, then use 1 − similarity as divergence (saliency). For higher semantic fidelity you can swap in embedding-based similarity later (e.g. Sentence-BERT); the workflow stays the same. </li> Why phrases, not tokens: Phrase-level (sentence/clause chunks) gives a good balance: token-level is noisy and expensive; whole-prompt is too coarse. So we split the prompt into phrases, perturb one phrase at a time, and attribute importance to the phrase. </li> </ul> Implementation</a></h2> Assume your agent as a black box: (system_prompt, user_message) → output. Wrap your agent in one async function that takes (system_prompt, user_message)</code> and returns a string (or a metric). What to return: The full response text. If your agent uses tools, concatenate the tool calls and final answer into one string. If you care specifically about tool choice, return just the tool names/args.</li> Temperature: Set to 0 for reproducibility. Even then, outputs can drift slightly due to batching/caching—handle this with multiple runs (see below). I should think about this a bit more, will update this point later.</li> Framework doesn’t matter. LangChain, OpenAI, Anthropic, custom—just wrap it so it takes two strings and returns one string.</li> </ul> Tokenize the Prompt into Phrases</a></h3> Split the system prompt into chunks you’ll perturb one at a time. Split at sentence boundaries (.</code> !</code> ?</code> newline).</li> If a sentence is longer than ~60 characters, sub-split at commas/semicolons.</li> Accumulate sub-chunks until each is at least ~35 characters.</li> </ol> Why phrase-level? Token-level is too noisy (one token rarely matters alone) and expensive (many API calls).</li> Whole-prompt is too coarse (no granularity).</li> Phrase-level (35–60 chars) balances signal and cost.</li> </ul> For structured prompts (JSON schemas, code blocks), consider custom tokenizers that respect structure boundaries. Get the Baseline Output</a></h3> Run your agent with the full, unmodified system prompt and a representative user message. Save this output, it’s your reference for comparison. Perturb Each Phrase and Measure Divergence</a></h3> For each phrase i</code>: Perturb: Replace phrase i</code> with [...]</code> (or remove it entirely for omission method).</li> Run: Call your agent with the perturbed prompt and the same user message.</li> Compare: Measure how different the new output is from the baseline.</li> Score: saliency = 1 - similarity(baseline, perturbed_output)</code>.</li> </ol> High saliency = perturbing this phrase changed the output a lot = important phrase. Choose a Similarity Function</a></h3> You need a single number for “how similar are these two outputs?” I prefer using Sentence embeddings : Use a model like Sentence-BERT (all-MiniLM-L6-v2</code> is fast and good).</li> Encode both outputs to vectors, compute cosine similarity.</li> Captures semantic equivalence: paraphrases score high.</li> Requires sentence-transformers</code> library or an embedding API.</li> </ul> Normalise Scores</a></h3> Raw saliency scores depend on the specific outputs and similarity function. To compare across phrases: Min-max normalise to [0, 1]: (score - min) / (max - min)</code>.</li> Now 1.0 = most important phrase in this prompt; 0.0 = least important.</li> If all scores are equal, return 0.5 for all (no differentiation).</li> </ul> Batch Over Multiple User Messages</a></h3> Saliency for one user message tells you “importance for this query.” To generalize: Pick 5–10 representative user messages (cover different intents your agent handles).</li> Run saliency for each user message.</li> Average the normalised scores per phrase across all user messages.</li> </ol> Now you know which phrases matter on average, not just for one query. Multiple Runs for Stability</a></h3> Even with temperature=0</code>, outputs can vary slightly. For production: Run each perturbation K times</li> Average the scores across runs.</li> Compute 95% confidence intervals: mean ± 1.96 (stdev / sqrt(K))</code>.</li> </ol> Pruning rule: Only drop phrases where the upper bound of the CI is below your threshold (e.g. < 0.3). This ensures you’re confident the phrase is low-impact, not just noisy. Act on the Results</a></h3> Score range</th> Interpretation</th> Action</th></tr></thead> High (top third)</td> Critical phrases</td> Protect; clarify if agent misbehaves</td></tr> Low (bottom third, CI upper < 0.3)</td> Redundant or weak</td> Candidate for pruning</td></tr> Middle</td> Moderate impact</td> Keep; revisit later</td></tr> </tbody></table> Pruning workflow: Drop low-saliency phrases (where upper CI < threshold).</li> Re-run your agent on the same and new user messages.</li> Verify behaviour is unchanged (use your existing evals).</li> Iterate.</li> </ol> Detect Interactions</a></h3> Single-phrase saliency assumes independence. To catch conflicts or synergies: Perturb pairs of phrases together.</li> Compute: interaction(i, j) = saliency(i+j) - saliency(i) - saliency(j)</code>.</li> Positive = synergy (removing both hurts more than sum). Negative = conflict (removing both hurts less).</li> </ol> Cost is O(N²), so only do this for short prompts or after pruning. Limitations</a></h2> Interactions: Assumes phrases contribute independently. Conflicting instructions (“Be concise” + “Explain in detail”) can make individual scores misleading. Inspect high- and low-saliency phrases together before pruning.</li> Embedding cost: Sentence-BERT adds ~50ms per comparison. For very long outputs, chunk and average.</li> Phrase boundaries: Heuristic (sentence/clause). Domain-specific prompts (code, JSON schemas) may need custom tokenizers.</li> API cost: N+1 calls per user message per run. Keep N small by pre-pruning or using fewer user messages.</li> </ul> Edit</a></h2> Actually did the implementation</a> today! It’s a bit advanced implementation. It’s CLI based for now, maybe should make it browser based if want to post about it? The chronicles of training FBPINNs Unknown — Fri, 02 Jan 2026 00:00:00 +0000 Right now I am training FBPINNs on a supercomputer. yay!. This distributed training is a total headache, so this is a writeup about how I see am optimizing the resources I have been bestowed upon. I wanna have some bitter sweet memories of GPUs when I graduate ;) Basics</a></h3> First and foremost, I am using V100</a>. The V100 is built from multiple Streaming Multiprocessors (SMs), each executing warps (block of 32 threads) via CUDA cores (the general‑purpose ALUs that execute most integer and BF16 operations) and one instruction issued by the warp scheduler is carried out simultaneously by all active CUDA cores for that warp (SIMT style). in a SIMT fashion (Single Instruction Multiple Threads), backed by register files, small L1/texture caches, and shared memory. There are Tenor cores for specialized matrix‑multiply‑accumulate operations. Instead of doing scalar FMA, a single tensor‑core instruction multiplies small matrices (e.g., 16×16 tiles) in mixed precision (FP16/BF16/TF32 → FP32/FP16 accumulators). They sit alongside CUDA cores and are used when your kernel uses tensor/matrix instructions (e.g., GEMM, convolutions). All SMs talk to a large on‑package HBM2 memory stack through several memory controllers, providing around 900 GB/s of memory bandwidth for local tensor data (activations, parameters, wavefields, etc.). CPU and GPU communicate over the PCI Express bus (it’s not technically a bus but a point to point connection). From the perspective of software running on the CPU, these days, that communication is typically in the form of memory-mapped IO. The GPU has registers and memory mapped into the CPU address space using PCIe. A write to a particular address generates a message on the PCIe bus that’s received by the GPU and produces a write to a GPU register or GPU memory. The GPU also has access to system memory through the PCIe bus. Typically, the CPU will construct buffers in memory with data (textures, vertices), commands, and GPU code. It will then store the buffer address in a GPU register and ring some sort of “doorbell” by writing to another GPU register. The GPU (specifically, the GPU command processor) will then read the buffers from system memory, and start executing the commands. Those commands can include, for example, loading GPU shader programs into shader memory and triggering the shaders to execute those shaders. There is an on‑chip L2 cache shared across SMs that backs global memory accesses and also serves as the interface to off‑chip links like NVLink and PCIe. Each V100 has high‑bandwidth HBM2 on‑package, giving around 900 GB/s of local memory bandwidth, so local tensor math is extremely fast compared to any inter‑GPU link. Each V100 is also connected to the host via PCIe (x16 slot), which has much lower bandwidth and higher latency than NVLink. Each V100 supports up to 6 NVLink 2.0 links, at about 50 GB/s bidirectional per link, for an aggregate of up to 300 GB/s GPU‑to‑GPU bandwidth per device. This is an order of magnitude faster and lower‑latency than going GPU → CPU over PCIe and then to another GPU, so if two GPUs must frequently exchange activations, parameters, or halo regions, I would want that traffic to run over NVLink, not PCIe. The CPU has DRAM and possibly a NIC (InfiniBand/Ethernet) for off‑node traffic. Any data that must leave the node (checkpoints, distributed training gradients over nodes) typically travels GPU → PCIe → CPU memory → NIC → network. In reverse, GPU results (e.g., metrics, checkpoints) go back GPU HBM2 → L2 → PCIe → CPU DRAM. At the hardware level, NVLink attaches logically near the GPU’s L2/memory controller region, so a tensor in GPU0’s HBM2 can be read/written by GPU1 over NVLink without going through the CPU or system DRAM. So, the hardware path is: SMs on GPU0 write halo tensors to HBM2 → L2 → NVLink serdes → L2/HBM2 on GPU1, and vice versa. How it all works together : The CPU (host) calls something like cudaMalloc to reserve buffers in device (GPU) memory. The host uses an async version to DMA data over PCIe or NVLink into those device buffers. Using pinned (page‑locked) host memory allows faster transfers because the driver can DMA directly from those pages without an extra staging copy.</li> The host call submits a grid of thread blocks to the GPU, and the CPU continues immediately (kernels are asynchronous). The GPU’s work distributor assigns each thread block to a SM when that SM has enough registers, shared memory, and slots for warps. On each SM, blocks are decomposed into warps (typically 32 threads). Warp schedulers on the SM issue instructions from ready warps each cycle. Modern SMs have multiple warp schedulers; each cycle they choose ready warps and issue instructions to CUDA cores, tensor cores, and load/store units.</li> Those instructions run on CUDA cores or tensor cores, using registers and shared memory, while load/store units handle memory traffic.</li> When a warp stalls (e.g., waiting on DRAM), the SM instantly switches to another ready warp, hiding latency and keeping the execution units busy.</li> </ul> So, an SM is essentially a many‑lane vector processor with its own control (warp schedulers) and fast memories, where CUDA cores handle general math, tensor cores accelerate matrix math, and the control logic juggles thousands of threads to maximize utilization. The Tech Stack</a></h2> There are layers to this circus man! OSI model on steroid : Orchestration</a></h3> SLURM is the job scheduler I use to allocate GPUs and manage queue fairness. Unlike interactive execution, batch jobs with SLURM guarantee reproducible resource allocation, critical for performance benchmarking. Drivers and LIbraries</a></h3> CUDA 12.9 provides: GPU kernel execution runtime: Compiles and executes compute kernels on V100s (JAX’s XLA compiler generates CUDA kernels optimized for V100 Tensor Cores, more on that later)</li> cuDNN (CUDA Deep Neural Network Library): Optimized convolution and activation kernels</li> cuBLAS: GPU matrix multiplication (used implicitly by neural network layers) - for handling forward and backward passes in case of PINNs</li> cuSolver: GPU linear algebra solver</li> </ul> AI Runtime</a></h3> JAX is the computational framework. Under the hood: JAX: High-level API for autodiff and functional transformations. I mark training/inference loops with JIT to get compiled functions.</li> XLA (Accelerated Linear Algebra): LLVM-based compiler that converts JAX code to CUDA kernels. XLA traces the function, builds an HLO graph. It fuses operations where possible (e.g., elementwise stencils + activation functions) and maps them to GPU operations (call cuBLAS/cuDNN kernels, or emit custom fused kernels).</li> </ul> Those GPU operations correspond to CUDA kernels with: Grid and block dimensions (how many thread blocks, threads per block).</li> Arguments (pointers to tensors in HBM2, scalars, etc.).</li> </ul> They are enqueued into CUDA streams (a queue of operations like kernels, memcpys, events, etc that execute in order on a GPU, different streams can run concurrently if there are resources). Once a kernel is launched on a stream: The GPU’s command processor pulls it from the queue.</li> It sets up the grid of thread blocks and schedules them onto SMs.</li> Each SM instantiates multiple warps (32 threads each), uses the warp scheduler to interleave warps and hide memory latency and issues memory loads/stores to global HBM2 via L1/L2, and uses the NVLink/PCIe fabric only when accessing peer memory.</li> </ul> The streams ensure the right order (e.g., do not consume halo before copy is done), and graphs allow that whole arrangement to be replayed efficiently. Networking</a></h3> export NCCL_P2P_DISABLE=0 # Enable GPU-to-GPU P2P export NCCL_IB_DISABLE=1 # InfiniBand disabled (not used) export NCCL_MIN_NCHANNELS=32 # 32 parallel channels export NCCL_BUFFSIZE=8388608 # 8MB buffers</code></pre> NCCL handles GPU-to-GPU communication</a> across NVLink. I used: NVLink bandwidth: ~300 GB/s between the two V100s</li> Collective operations: all-reduce</code>, broadcast</code>, reduce-scatter</code></li> </ul> For FBPINNs, this enables: Multi-GPU subdomain parallelism (each GPU owns different subdomains)</li> Synchronization via hardware all-reduce</code> for combining weighted sums</li> </ul> CPU Optimizaton</a></h2> Modern scientific libraries (NumPy, BLAS, OpenBLAS, MKL, etc.) use OpenMP‑style threading to parallelize CPU operations like matrix–matrix multiplications, solvers, FFTs, and so on. When allowed to use many threads, they can: Launch as many threads as CPU cores/HT threads (e.g., 16–64 threads) for a single BLAS call</a></li> Quickly saturate CPU cores, generating high CPU usage and memory‑bandwidth pressure</a></li> </ul> After JAX offloads GPU work, the CPU is still responsible for: Data loading, batching, and small preprocessing</li> Logging and profiling</li> A few tiny BLAS‑like operations (often 1–2 ms per step)</li> </ul> If each of these tiny BLAS calls spawns many threads, things happens: CPU threads compete with JAX’s runtime and GPU driver threads for OS scheduling, delaying kernel launches and GPU context switches by tens to hundreds of microseconds.</li> Many active threads trample L1/L2/L3 caches and saturate memory bandwidth, which can indirectly slow down GPU transfers (since PCIe traffic often shares the same memory subsystem and CPU memory bus).</li> The result is increased step‑time variance and reduced GPU duty cycle, even though the GPU kernel itself is unchanged !!</li> </ul> In my FBPINN setup, the GPU is dominated by: Thousands of small PDE kernels</li> Dense matmuls</a> and autograd kernels, so step times are often 20–100 ms; tiny CPU jitters do add up over thousands of steps</li> </ul> By carefully limiting BLAS/OpenMP threading, I keep the CPU mostly idle and give JAX/GPU‑driver threads clean access to CPU time and cache # Force BLAS/NumPy to single‑threaded mode os.environ["OMP_NUM_THREADS"] = "1" os.environ["OPENBLAS_NUM_THREADS"] = "1" os.environ["MKL_NUM_THREADS"] = "1" os.environ["VECLIB_MAXIMUM_THREADS"] = "1" os.environ["NUMEXPR_NUM_THREADS"] = "1" # For JAX itself, I also restrict XLA's internal threadpool: os.environ["XLA_FLAGS"] = ( "--xla_cpu_multi_thread_eigen=false " "intra_op_parallelism_threads=1 " "inter_op_parallelism_threads=1" )</code></pre> In my high‑throughput training, this setup: Reduced CPU overhead from ~1–2 ms to ~0.2–0.5 ms per step.</li> And improved GPU duty cycle by 5–10% and reduce step‑time variance by 20–40%.</li> </ul> I did this because I have a GPU-bound workload, but if it would have been CPU‑bound workloads (heavy data loading, preprocessing, small models, etc), I would have used multiple threads (e.g., OMP_NUM_THREADS=4–8) and profile where time is spent, and then tune thread count and CPU pinning accordingly. JAX CPU threading and parallelism</a></h3> JAX’s CPU backend does provide intra‑op parallelism for some operations</a> (e.g., many BLAS/LAPACK‑like matrix operations use Eigen’s internal threadpool). However not all operations are multithreaded. for example, FFT operations on CPU historically did not use multi‑threading, and even in newer versions you may still see limited parallelism compared with hand‑tuned BLAS‑based FFTs. JAX does not provide general inter‑op parallelism: it won’t automatically schedule multiple independent operations to run in parallel across CPU cores. Because of this, JAX’s CPU backend is not optimized for maximum CPU utilization in the HPC sense. It prioritizes correctness and portability as well as integration with XLA and GPU/TPU targets rather than squeezing every FLOP out of a large CPU node. This means that, for many CPU‑heavy workloads, JAX can be slower than single‑core C++ and significantly slower than well‑optimized parallel C++/Fortran/MPI code. In practice, JAX is best treated as a convenient, compiler‑enhanced numerical stack rather than a drop‑in HPC‑optimized CPU backend</a>. JAX’s JIT compiler can be very fast in many applications, but finely‑tuned C++ can be much faster for certain problems. You can extend JAX</a>! Topology-aware Sharding</a></h2> Yay! A fancy word to throw around!!! At a high level topology‑aware sharding</a> means I place model shards on GPUs in a way that matches the communication pattern of my algorithm to the physical GPU interconnect (here, NVLink between the two V100s) so that the most chatty peers talk over the fastest links, minimizing PCIe or host‑network hops. If I ignore hardware topology and just say “I have N GPUs, shard the model arbitrarily”, the runtime might place logically adjacent model pieces on GPUs that are far apart in the physical network (e.g., across nodes or over pure PCIe). This increases: Latency for every cross‑shard communication.</li> Contention on shared links (PCIe root complex, host NIC).</li> Synchronization overhead, because each step waits on slower communication.</li> </ul> I clearly dont want this cuz I am smart ;) I first understand the graph of GPU interconnects (which GPUs have NVLink, which are only PCIe peers, which are across Infiniband), and then map logical communication groups (tensor‑parallel group, pipeline stage boundaries, domain‑decomposition neighbors) onto the fastest‑connected subset. The principle is: High‑frequency, latency‑sensitive collectives (e.g., tensor parallel</a> all‑reduces, halo exchanges every iteration) stay on strongest links like NVLink.</li> Medium‑frequency transfers (e.g., pipeline activations, subdomain checkpoint exchange) can tolerate one slower hop.</li> Low‑frequency work (e.g., global gradient all‑reduce, checkpointing) can spill over slower interconnects like Ethernet/IB.</li> </ul> But how to verify topology of nodes? I used nvidia-smi topo -m</code> to see the connectivity matrix; NVLink‑connected GPUs show as NV1/NV2/…, PCIe only as PHB or similar</li> nvidia-smi -q -d NVLINK</code> to check which NVLink links are up between the two V100s</li> nvidia-smi -L</code> or CUDA_VISIBLE_DEVICES</code> to confirm device indices and then map shard placement: e.g., keep the most chatty tensor/model‑parallel ranks on GPU0 and GPU1 of the same node</li> </ul> Once you know which GPU indices share NVLink, you can implement topology‑aware sharding at the framework level (PyTorch device_ids / process group mapping, JAX mesh layout, DeepSpeed/DTensor placement) so that heavy all‑reduce/all‑to‑all happens within those NVLink pairs and only coarser‑grained sync crosses nodes over InfiniBand. Enough hardware, how did I use this information? In the FBPINN subsurface modeling, I had a large 2D domain decomposed into lots of subdomains with partition‑of‑unity windows. Each subdomain corresponds to a small local network, but neighboring subdomains still need to exchange information (e.g., wavefield values near the overlap, gradients for the inversion) each iteration, which creates a neighbor communication pattern reminiscent of a 2D stencil. Which subdomain goes to same GPU? I grouped subdomains so that those with the heaviest cross‑coupling (e.g., central, high‑wave‑energy regions) reside on the same GPU, so their communication is purely local HBM2 traffic, which is cheapest. Less tightly coupled or boundary subdomains (small number of cross‑boundary subdomains) could be split across GPUs, with their overlapped halo data sent over NVLink each step. I designed the field exchange as batched halo exchanges. Instead of many small transfers, I pack halo tensors belonging to adjacent subdomains into contiguous buffers and send them over NVLink in fewer, larger transfers, making better use of the ~300 GB/s aggregate bandwidth. There is no reason for halo data to go to CPU DRAM or across the NIC on each iteration, that would be orders of magnitude slower and would stall the SMs waiting on PCIe round‑trips. In JAX terms, this was effectively a model‑parallel sharding on distributed infrastructure</a>. The global domain is the model, and each subdomain network is a shard. So, how it works under the hood? Let me walk through one FBPINN training step in this topology‑aware setup in simplistic way (just 2 GPUs): Before training, I decide: “these 40 subdomains on GPU0, these 35 on GPU1.” I also precompute neighbor lists and overlap regions so I know exactly which data each GPU must send/receive at each step.</li> On each V100, the bulk of compute (forward PDE solve and local backward) runs purely on local HBM2, maximizing the ~900 GB/s bandwidth and SM utilization; no inter‑GPU traffic yet. This is where JAX JIT helps; it compiles each subdomain’s work into efficient fused kernels and schedules them over the SMs.</li> For subdomains at the boundary between GPU0 and GPU1, I gather the boundary tensors (wavefields on the overlap, maybe derivative info) into contiguous buffers. I then initiate device‑to‑device copies or NCCL sends/receives across the NVLink link, which runs at up to 300 GB/s aggregate and significantly lower latency than PCIe. Because the GPUs are directly NVLink‑connected, the driver routes these as peer‑to‑peer transfers without staging through host memory.</li> After the halo exchange, I perform block‑level accumulation steps (e.g., summing overlapping contributions, enforcing partition‑of‑unity consistency) and then proceed to the next time step or optimization step. I explicitly synchronize only where necessary (e.g., before using updated halos) to avoid global barriers that would stall both GPUs.</li> The same idea extends to more complex topologies: you place the most communication‑heavy neighbors in groups that sit on NVLink or NVSwitch within a node, and only lower‑frequency communications (e.g., checkpoints, global misfit evaluation) cross node boundaries over InfiniBand or Ethernet.</li> </ul> The benefits of topology‑aware sharing is clear : Higher effective throughput: by matching communication patterns to the fastest links, you utilize the ~300 GB/s NVLink bandwidth and avoid PCIe bottlenecks.</li> Lower latency and less idle time: GPUs spend more time doing local HBM2 math and less time waiting for halo or activation data.</li> Better scalability: as you move toward multi‑node, the same principles generalize to NVSwitch vs InfiniBand vs Ethernet, which is exactly the sort of reasoning you need in a hyperscaler‑grade AI cluster.</li> </ul> But the tradeoffs are real : More complex placement logic: you must know your communication graph and your hardware topology, and implement placement heuristics or algorithms.</li> Reduced flexibility: if you tie your model layout too tightly to a specific topology (e.g., dual‑V100 with certain NVLink layout), porting to a different cluster topology may require changing the sharding plan.</li> Potential imbalance: minimizing cross‑GPU communication might conflict with load balancing; you sometimes have to trade slightly more communication for better compute balance across GPUs.</li> </ul> GPU optimizations</a></h2> Well, a lot of them actually. CUDA Graphs (Kernel Launch Overhead)</a></h3> Every CUDA kernel launch has ~10-50 microseconds of overhead. My FBPINN training loop runs thousands of PDE evaluations per step, each requiring multiple kernels. This is a problem!</a> So, I use CUDA Graphs</a> to tackle this. How it work: In the first pass, JAX records a sequence of CUDA kernels into a graph. It runs the full forward + PDE evaluation + backward once, while recording which CUDA kernels are launched, in what order, and with what arguments. This graph is a static description of the GPU workload (a DAG of kernels, memcpys, events, etc.).</li> Instead of launching 500 kernels one by one from the CPU, the runtime submits the entire pre‑recorded graph to the GPU in a single call. The GPU then orchestrates the 500 kernels internally, with minimal CPU involvement. On modern GPUs, the repeat launch overhead</a> is roughly constant and small (e.g., 1–3 μs total for the whole graph), regardless of how many kernels are inside it. So subsequent passes replay the graph with 1-2 μs overhead instead of 10-50 μs per kernel.</li> </ol> So, a training step with 500 kernels normally costs 500 × 25 μs = 12.5 ms overhead. With graphs 1 graph replay = 2 μs overhead. So, ~30-50% speedup for compute-bound workloads. Yay!! Easy And it works really well for FBPINNs! Due to fixed batch sizes we have static shapes and thus graphs can be captured (computational pattern is very repetitive, fixed batch sizes, fixed PDE domains, etc.). The structure is almost the same every iteration, only tensor values change. And due to thousands of iterations, overhead savings compound significantly. XLA Compiler Flags (V100-Specific)</a></h3> XLA performance flags</a> are very much version dependent. Use these with caution. Some flags (like latency‑hiding scheduler</a>) can increase memory usage a lot on some models! export XLA_FLAGS="--xla_gpu_enablefast_min_max \ --xla_gpu_enable_triton_gemm \ --xla_gpu_enable_latency_hiding_scheduler=true \ --xla_gpu_all_reduce_combine_threshold_bytes=134217728 \ --xla_gpu_enable_highest_priority_async_stream=true"</code></pre> Flag breakdown: --xla_gpu_enable_fast_min_max</code>: Use faster, slightly lower-precision min/max Benefit: 10% faster activation functions (ReLU, etc.)</li> Tradeoff: Negligible precision loss for PINN training</li> </ul> </li> --xla_gpu_enable_triton_gemm</code>: Replace cuBLAS GEMM with Triton-compiled kernels Benefit: 15-25% speedup on matrix multiplication</li> Triton is an open-source language for writing GPU kernels; auto-tuned for V100</li> Particularly effective for non-standard matrix shapes (pretty common in FBPINNs)</li> </ul> </li> --xla_gpu_enable_latency_hiding_scheduler</code>: Overlap memory operations with compute Before: GPU stalls waiting for memory</li> After: While GPU transfers data, it computes other operations in parallel</li> Benefit: 10-15% speedup for memory-intensive workloads</li> </ul> </li> --xla_gpu_all_reduce_combine_threshold_bytes=134217728</code>: Combine small all-reduce calls 128 MB threshold means small reductions are batched into one large reduction</li> Benefit: ~20-30% faster multi-GPU synchronization</li> Why: Fewer NCCL calls so better utilization of NVLink</li> </ul> </li> --xla_gpu_enable_highest_priority_async_stream</code>: Prioritize compute stream Benefit: Compute kernels run faster by preempting lower-priority streams</li> Minor effect on single-GPU, significant on multi-GPU</li> </ul> </li> </ol> I estimate a combined improvement of ~20% Sweet! Memory Management</a></h3> # Let JAX manage GPU memory dynamically os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false" # Allocate only 70% of VRAM to leave headroom for JIT compilation os.environ["XLA_PYTHON_CLIENT_MEM_FRACTION"] = "0.70" # Use async allocator to reduce fragmentation os.environ["TF_GPU_ALLOCATOR"] = "cuda_malloc_async" # Disable float64 (slower on Tensor Cores) os.environ["JAX_ENABLE_X64"] = "False"</code></pre> Memory tuning is critical for FBPINNs: Preallocation vs. dynamic allocation: Preallocate: JAX reserves all VRAM at startup</li> Dynamic: JAX allocates only when needed</li> For research, dynamic is better (flexibility for variable batch sizes)</li> </ul> </li> Memory fraction: V100 has 16 GB VRAM</li> 0.70 × 16 GB = 11.2 GB for arrays</li> Remaining 4.8 GB: JIT compilation, temporary buffers</li> My FBPINNs code has large network weights and grouped subdomain metadata (varies, but I hit the wall pretty fast)</li> If I set to 0.90, JIT compilation fails with OOM when recompiling</li> </ul> </li> Async allocator (cuda_malloc_async</code>): Default CUDA allocator uses a buddy-block system (causes fragmentation)</li> Async allocator uses a more efficient pool strategy</li> Benefit: ~20-30% reduction in allocation latency, fewer OOM errors</li> No downside for training (not used for inference)</li> </ul> </li> Float32 vs Float64: V100 Tensor Cores: 125 TFLOPS (float32), 4 TFLOPS (float64)</li> 32x speedup by using float32</li> For PINNs, float32 is sufficient (physics-informed loss provides regularization)</li> </ul> </li> </ol> This first of all prevents OOM (total pain in my ass) and also enables 2-3x faster math. Crazzyyy… Compilation Caching</a></h3> os.environ["JAX_ENABLE_COMPILATION_CACHE"] = "1" os.environ["JAX_COMPILATION_CACHE_DIR"] = ".jax_cache"</code></pre> How it works: In the first run JAX compiles my FBPINN update function (around 10-30 seconds)</li> Compilation artifact saved to the disk (.jax_cache/</code>)</li> And in the second run, JAX loads cached artifact, skips compilation</li> </ol> Quite simple isnt it! Run the same job twice? Without cache: 30s + 50,000 steps × 0.5s = 25030 seconds</li> With cache: 0s + 50,000 steps × 0.5s = 25000 seconds</li> Research is iterative development. Restarting job to tweak hyperparameters, I get instant speedup</li> </ul> Algorithmic optimizations due to Memory</a></h2> Grouped Subdomain Evaluation</a></h3> The training domain is split into 75 overlapping subdomains. Each subdomain has its own neural network (75 networks total). Naive approach would have been: for m in range(M): for p in domain_points: u[p] += network[m](x[p])</code></pre> Memory: O(M × P × d) where P = number of test points, d = spatial dimensions. For this setup 75 × 262,144 × 3 × 4 bytes = 300 GB (well, i dont have infinite money glitch for sure) So, I used some tricks up my sleve : # Precomputed metadata: which points belong to which subdomains g_n_idx, g_n_mask = grouped_metadata # Vectorized evaluation: all 75 subdomains in parallel us_g, ws_g, us_raw_g = vmap(_model_over_subdomains, in_axes=(f, 0, 0))( all_params, x_batch[g_n_idx], g_n_mask ) # Hardware reduction: combine contributions per point u_sum_local = jax.ops.segment_sum(us_masked, idx_flat, num_segments=num_p_total) wp_sum_local = jax.ops.segment_sum(ws_masked, idx_flat, num_segments=num_p_total) # Global weighted average u_local_norm = u_sum_local / jnp.maximum(wp_sum_local, 1e-5)</code></pre> Quirks you ask? On-the-fly indexing: Instead of creating (M, max_P, d)</code> array, index into x_batch</code> dynamically Memory: O(P × d) = 262,144 × 3 × 4 = 3 MB</li> Savings is nuts: 300 GB to 3 MB</li> </ul> </li> Vectorized evaluation via vmap</code>: Automatically distributes subdomain evaluation across GPU cores</li> JAX compiles to parallel kernels</li> </ul> </li> Hardware-accelerated reduction via segment_sum: GPU tree-reduction algorithm</li> O(log P) parallel steps instead of O(P) sequential</li> Speed: 10-20x faster than Python loops</li> </ul> </li> </ol> I get 3-5x faster than naive sequential evaluation + memory enables 262k-point test grids. Win. Chunked Processing for Large Validations</a></h3> During validation, I evaluate the FBPINN on a fine test grid (e.g., 128×128×16 = 262k points): chunk_size = 25000 # Process 25k points at a time for chunk_start in range(0, n_points, chunk_size): chunk_end = min(chunk_start + chunk_size, n_points) abs_error = jnp.abs( u_exact_flat[chunk_start:chunk_end] - u_test_flat[chunk_start:chunk_end] )</code></pre> The issue is that validation grid might not fit in VRAM (even with grouped evaluation). So, the chunks of 25k points is around ~15 MB per chunk that fits comfortably. This enables high-resolution validation without OOM, ~5-10% overhead from loop. Pipelined Updates</a></h3> My run.sh</code> requests multiple GPUs. Lets stick with simple 2 for this blog. Without careful coordination, multi-GPU training can be slow due to communication overhead. MULTI_STEP = 100 ACC_STEPS = 10 def _update_pmap_impl(aos, ap, fp, static_params_local, start_step): def _acc_block(carry, _): c_aos, c_ap, curr_step = carry def _inner_step(i_carry, _): i_ap, i_step, i_grads, i_loss = i_carry l, g = value_and_grad(FBPINN_loss, argnums=0)(...) n_grads = jax.tree.map(lambda x, y: x + y, i_grads, g) return (i_ap, i_step + 1, n_grads, i_loss + l), l # Gradient accumulation loop (10 steps) init_g = jax.tree.map(jnp.zeros_like, c_ap) (next_ap, n_step, sum_g, sum_l), _ = jax.lax.scan( _inner_step, (c_ap, curr_step, init_g, 0.0), None, length=ACC_STEPS, ) final_g = jax.tree.map(lambda x: x / ACC_STEPS, sum_g) # Single optimizer update updates, n_aos = optimiser_fn(final_g, c_aos, next_ap) n_ap = optax.apply_updates(next_ap, updates) return (n_aos, n_ap, n_step), final_l # Outer loop (10 blocks) (final_aos, final_ap, _), block_losses = jax.lax.scan( _acc_block, (aos, ap, start_step), None, length=MULTI_STEP // ACC_STEPS, ) return block_losses[-1], final_aos, final_ap</code></pre> Why this is fast: Nested loops structure: Inner scan</code>: 10 gradient accumulation steps (NO optimizer update)</li> Outer scan</code>: 10 blocks of 10 steps each = 100 total steps</li> So the benefit: 1 optimizer update per 10 steps instead of 1 per step</li> </ul> </li> Gradient accumulation: Accumulate gradients: grad_total += grad_step</code></li> After 10 steps, apply optimizer once: param -= lr * grad_total / 10</code></li> Mathematically equivalent to 10 smaller updates, but faster GPU execution</li> </ul> </li> JIT compilation: Entire 100-step block compiles to single XLA function</li> Reduces Python overhead from 100 calls to 1 call</li> 100 steps × 0.5s/step = 50s with Python overhead to 50s without (if overhead was 1%)</li> More realistic: 100 steps with overhead = 55s, without = 50s to 10% speedup</li> </ul> </li> Multi-GPU coordination: jax.pmap</code> distributes the function across 2 GPUs</li> Within the pmap, each GPU owns disjoint subdomains</li> Synchronization happens via jax.lax.pmean</code> (hardware all-reduce)</li> By processing 100 steps in one pmap call, you reduce sync frequency</li> </ul> </li> </ol> Overall 2-3x speedup with 2 GPUs Hardware-Native Subdomain Sharding</a></h3> def u_sync(x_batch): # 1. Local evaluation on each GPU outs = FBPINN_model(all_params, x_batch, takes, model_fns) # 2. Hardware sync: GPU0 and GPU1 combine results if len(takes) >= 9: u_sum, wp_sum = jax.lax.psum(outs[2:4], axis_name="devices") # 3. Global weighted average wp_total = jnp.maximum(wp_sum, 1e-5) u_global = u_sum / wp_total u_global = model_fns[4](all_params, x_batch, u_global) return u_global, ()</code></pre> How sharding works: Partition of Unity (PoU): Each subdomain has a smooth window function w_m(x)</code> that sums to 1 At any point: sum_m w_m(x) = 1</code></li> Global solution: u(x) = sum_m w_m(x) * u_m(x)</code></li> </ul> </li> GPU distribution: GPU0 owns subdomains 1-37, computes: u_sum_0 = sum_{m=1}^{37} w_m(x) * u_m(x)</code>, w_sum_0 = sum_{m=1}^{37} w_m(x)</code></li> GPU1 owns subdomains 38-75, computes: u_sum_1 = sum_{m=38}^{75} w_m(x) * u_m(x)</code>, w_sum_1 = sum_{m=38}^{75} w_m(x)</code></li> </ul> </li> Hardware synchronization: jax.lax.psum</code> all-reduces: u_sum = u_sum_0 + u_sum_1</code>, w_sum = w_sum_0 + w_sum_1</code></li> Runs on NVLink (~300 GB/s)</li> </ul> </li> Final assembly: u_global = u_sum / w_sum</code> (per-point weighted average)</li> Apply constraints (boundary conditions)</li> </ul> </li> </ol> So, a near-linear scaling (1.8-2x with 2 GPUs) NCCL tuning</a></h3> export NCCL_P2P_DISABLE=0 # Enable P2P over NVLink export NCCL_IB_DISABLE=1 # No InfiniBand export NCCL_MIN_NCHANNELS=32 # 32 channels for parallelism export NCCL_BUFFSIZE=8388608 # 8 MB buffers</code></pre> NCCL parameters explained: P2P: GPU-to-GPU direct memory access Alternative: CPU-mediated (GPU → CPU → GPU, much slower)</li> My V100s are on the same PCIe switch, so P2P is efficient</li> </ul> </li> Min channels: Multiple parallel communication paths 32 channels means 32 parallel “streams” of all-reduce operations</li> Better GPU utilization on large collectives</li> </ul> </li> Buffer size: Tradeoff between latency and memory overhead 8 MB is tuned for V100 bandwidth (~900 GB/s effective)</li> Smaller = lower latency, larger = better throughput</li> </ul> </li> </ol> Thus, ~20-30% faster multi-GPU communication Fast PRNG</a></h3> jax.config.update("jax_enable_custom_prng", True) jax.config.update("jax_default_prng_impl", "threefry2x32")</code></pre> ThreeFry is a fast parallel PRNG. Benefits: ~5-10% speedup in sampling-heavy workloads and Minimal overhead for random point sampling</li> Disables NaN checks (small overhead, but adds up over millions of ops)</li> </ul> Memory Hierarchy Optimizations</a></h2> Chunked PDE Residual Computation</a></h3> Computing PDE residuals for large batches (e.g., 10,000+ points) simultaneously creates a working set that exceeds the GPU’s L2 cache (6MB on V100). This causes cache thrashing, where data is repeatedly evicted and re-fetched from slow global memory. Research shows underutilizing L2 cache can cause 160–176% slowdown</a>! To push performance beyond standard JAX JIT compilation, I implemented three targeted optimizations designed to maximize L2 cache residency</a> and minimize expensive High Bandwidth Memory (HBM) transactions. As FBPINN residual evaluations have low arithmetic intensity (few FLOPs per byte loaded). By processing in chunks that fit in L2, we can effectively increase the operational intensity by avoiding repeated DRAM fetches for the same parameters, keeping them ‘hot’ in cache. # Naive approach: Large working set evicts L2 cache # Working Set ~= 10,000 points * (inputs + grads + activations) > 6MB residuals = compute_pde_residual(all_points) # High DRAM traffic # Optimized approach: Blocked execution for L2 residency # Working Set ~= 2,048 points < 6MB (Cache Hit!) residuals = jnp.concatenate([ compute_pde_residual(chunk) for chunk in split(all_points, 2048) ])</code></pre> This gives us speedup of about 20-30% for large batch sizes and L2 Hit Rate increases from ~50% to >85%. Another win! Subdomain Batching</a></h3> The problem is that FBPINNs evaluate multiple sub-networks (subdomains). Evaluating all 75+ subdomains in parallel loads all their unique parameters into memory at once (~15MB), blowing out the 6MB L2 cache. So, we batch the subdomains themselves. By evaluating only 20 subdomains at a time, we ensure their combined parameters remain resident in L2 for the duration of the compute kernel. # Naive: Thrashing L2 with all subdomains' params vmap(evaluate_subdomain)(all_75_subdomains) # Optimized: Tiled execution # Keeps active parameter set < L2 Cache Size for batch in batch_subdomains(all_75_subdomains, size=20): vmap(evaluate_subdomain)(batch)</code></pre> Using this I reduced by ~50% due to fewer DRAM reads, and got a speedup of 15-25% on problems with high domain counts. Spatial Sorting</a></h3> Collocation points are often sampled randomly. This leads to uncoalesced memory access—thread 0 reads address $A$, thread 1 reads address $A+10000$. This wastes memory bandwidth and causes TLB (Translation Lookaside Buffer) misses. What’s not optimized</a></h2> My code uses pmap</code> (data parallelism over GPU axis) instead of DDP because: My architecture is fundamentally model-parallel (subdomains)</li> DDP is better for replicated models, worse for domain decomposition</li> </ul> Also, no Quantization tricks as: SIREN networks are fully dense</li> Float32 is necessary for precision-sensitive PDE solving</li> </ul> Some more nerdy thoughts</a></h2> Understanding NVIDIA GPUs is necessary if you want to do scientific ML</a>. I will maybe someday write my own performant GPU kernels</a> (or maybe make an agent do that?). GPU performance issues are rarely about raw compute</a>. Modern GPUs have huge peak FLOPs, but they only reach it when: Data is already in GPU memory.</li> Kernels are queued without gaps.</li> Communication is overlapped with compute.</li> </ul> If input data, gradients, or activations arrive late (I/O, network, host scheduling), the GPU finishes its current work and then sits idle waiting for the next batch. Utilization drops, but not because the GPU is weak, because the pipeline upstream is slow or bursty. Even if FLOP count is the same, a small regression in memory layout can: Drop effective bandwidth.</li> Increase latency per kernel.</li> Cause whole pipelines to run slower, dropping utilization and increasing p95/p99 latency.</li> </ul> In distributed training or multi‑GPU jobs we use NCCL all‑reduce/all‑gather for gradients, parameters, or activations : Each rank (GPU) must participate (Collectives are synchronized)</li> If one GPU is slightly slower (data pipeline, kernel, or scheduling), others finish early and then wait at the barrier.</li> </ul> That little imbalance cascades: All GPUs idle during the slowest all‑reduce.</li> As step time increases, any latency spikes at the slowest rank show up as cluster‑wide spikes</a>.</li> </ul> So I can have perfectly fine raw compute, but a subtle change (e.g., different batch size, skewed data, contention on one node) makes one rank lag, and utilization drops everywhere. Within a node, GPUs talk over NVLink/NVSwitch. NVLink is fast, but still finite! Once a link is saturated: Communication for collectives/halo exchanges queues up.</li> Kernels that depend on that data start later.</li> GPUs wait more often on communication, so compute units are idle while the link drains.</li> </ul> Again, compute capacity didn’t change! the interconnect became the bottleneck. All of these issues—memory access regressions, collective sync stalls, NVLink congestion, CPU jitter—cause short idle periods and periodic long stalls. Even if average utilization doesn’t plummet, tail latency and throughput do! For online inference P95/P99 latency jumps because some requests hit a stalled collective or a data‑starved GPU. For training time‑per‑step grows; training runs longer. You pay for GPU‑hours where the GPUs are not fully used, inflating infra cost. Optimizing AI systems is mostly about feeding GPUs consistently (data, comms, scheduling) and coordinating them well! Raw FLOPs are rarely the limiting factor. Optimize relentellesly guys, GPUs are darn expensive</a>!!! Hell, my GPUs hate PNGs</a> ;) Continuous Profiling</a></h2> In high-performance training and inference pipelines, identifying the root cause of such issues is very challenging. When CUDA kernels underperform or Tensor Core utilization drops, engineers often reach for PyTorch Profiler or NVIDIA Nsight. But I dont feel very happy using these tools as a begineer. They operate at the level of individual processes or isolated GPU nodes, making frictionless, cluster-wide GPU visibility nearly impossible. This fragmented view means stitching together multiple tools, managing complex instrumentation, and manually correlating traces across nodes. It’s unsustainable!! Continuous profiling GPU has an observability tax. Continuous GPU profiling is essential for inference heavy pipelines because system entropy is driven by user requests. This is especially critical in disaggregated deployments that split prefill and decode phases, where performance can vary unpredictably. AI will surely come for rescue here sometime in 2026 I bet! Have to stop writing now..</a></h2> It’s fun to train such large ML models on many GPUs</a>. I am finding ML Infrastructure and overall Software Engineering much harder than the actual ML itself. What a joke! When I look back, the advancements of Deep Learning over the years is simply astounding</a>. Need to sleep now. 2:22 AM already, and I have quiz for Moral Philosophy tomorrow! The Rusty path to Secure Metrics in Ambient Mesh Unknown — Fri, 23 May 2025 00:00:00 +0000 Istio</a> is widely recognized as the most popular, powerful, and trusted service mesh</a> in the cloud native ecosystem and is extensively used in Production since it’s inception</a>. Fortunately, I got the opportunity to intern at Istio (managed CNCF and spun out of Google), and man my brain transitioned from a code-monkey to a divine architect in mere span of 3 months. What did I do?</a></h2> It feels surreal to know that the work I did during my internship will have a outsized impact on everyone using Istio’s ztunnel. Imagine, everytime someone scrapes ztunnel, they use one feature designed and implemented by me. Wowwwwww…. Before my work, ztunnel metrics were exposed in plaintext, trusting any client that could reach the port - scraping metrics followed Prometheus -> HTTP -> Ztunnel Metrics Server</code> : Direct HTTP connection</li> No encryption</li> No authentication</li> No policy enforcement</li> Simple but insecure</li> </ul> I added support for TLS (mTLS via HBONE tunneling) in the metrics endpoint in the ztunnel in Shared Proxy Mode of Istio’s Ambient Dataplane Mode</a>. Lots of buzzwords right ( ՞۝՞) Initially, I naively thought the solution would be straightforward - add a new mTLS-enabled metrics server to ztunnel. This seemed logical - we need secure metrics, so let’s add TLS. I started implementing a separate server that would: Listen on a dedicated port</li> Handle TLS/mTLS directly</li> Serve metrics over the secure connection</li> </ul> However, this approach had significant drawbacks: It would bypass ztunnel’s existing inbound logic</li> It would require duplicating certificate management code</li> It would create a separate security model from the rest of the mesh</li> And, most importantly, it would break the transparency principle - Prometheus would need to be configured specifically for mTLS</li> </ul> After I made a draft PR implementing the new mTLS-metrics server, then I realized that I should take a break and actually think a lot more. The feedback from Istio community was great, and helped me grasp the context and the sheer scale of the system that I was working on. Service mesh should provide zero-trust security by default - where no service is trusted until proven otherwise, provides security and observability without requiring changes to the application code, provides a consistent security model across all services in the mesh, treats all services uniformly acting as platform for consistent service-to-service communication, allowing progressive enhancement of service capabilities. The “aha moment” came when I realized that ztunnel itself could be treated as a workload in the mesh, so I can extend the service mesh to include ztunnel itself. This was a profound insight because: It aligned with Istio’s “everything is a workload” philosophy</li> It meant we could leverage existing mesh capabilities rather than building new ones</li> It maintained transparency for both ztunnel and Prometheus</li> </ul> Now, now its Prometheus -> HTTP -> Local Ztunnel -> HBONE -> Target Ztunnel -> Metrics Server</code> : Prometheus still makes a plain HTTP request (transparency - dosen’t need to know about TLS or HBONE)</li> The request is intercepted by iptables rules on the node where Prometheus is running (local ztunnel on the same node)</li> Ztunnel recognizes this is a request to another Ztunnel pod (target ztunnel being scraped) in the mesh</li> It automatically upgrades the connection to HBONE (HTTP/2 over mTLS)</li> This happens transparently to Prometheus - it’s still just making a regular HTTP request</li> Target ztunnel processes through standard inbound path, forwarding to internal metrics server, sending back through same tunnel</li> Full Network Policy enforcement</li> Secure but transparent to both ends</li> </ul> Every metrics request must prove its identity through mTLS, and authorization policies can control exactly which Prometheus instances can access the metrics The key components that made this possible is : Ambient Redirection - The ambient.istio.io/redirection: enabled</code> label on ztunnel pods tells the mesh that ztunnel should be treated as a workload and enables automatic traffic interception and HBONE upgrade.</li> Identity Management - Ztunnel uses Kubernetes Downward API to get its identity which is used to request certificates from Istiod. Same process as any other workload in the mesh.</li> Inbound Handler - Existing component that already handles HBONE connectionsa and processes mTLS termination.</li> </ul> No new code needed! Elegant… Fun thing - I increased the ztunnel binary size form 15mb to 16mb, which lead to CI failure. I had a fun time watching my mentor figure out what the heck just happened in Weekly Standup ( ^^)Y </blockquote> Now for a deeper look you can take a look at the Design Doc</a> for the feature. Most of my time went into thinking about the design, instead of coding it up. I also created a beautiful Internship report</a>. What did I learn?</a></h2> Due to my prior internship experience in Rust, the compiler was very friendly with me this time ■D＼(^^ ) This time, I learnt a lot of Kubernetes, Helm, low-level TCP/IP, HTTP/S, certificate management, debugging docker containers in different namespaces in k8s clusters, about service mesh and why they are adopted at the first place, and more. Learning Service Mesh</a> was very daunting for a new-comer like me! So many new terminology gets thrown here and there! This tutorial</a> on Istio’s Service Mesh helped me grasp things quite well! Life of a packet in Istio is very interesting</a> ヽ(o⌣oヾ) This project taught me several valuable lessons about service mesh: Transparency is the Key - The beauty of service mesh is that applications don’t need to know about security. Prometheus still makes plain HTTP requests and the Ztunnel’s metrics server still serves plain HTTP. The mesh handles all the security complexity</li> Configuration Over Code - Instead of writing new security code, we configured ztunnel to be a workload leveraging existing, well-tested components reducing complexity and potential bugs</li> Mesh Philosophy - Understanding that service mesh is about treating everything as a workload. Even infrastructure components like ztunnel can benefit from mesh capabilities. The mesh should be the security boundary, not individual components.</li> </ul> What’s really cool about this solution is that it demonstrates the power of service mesh architecture: We didn’t need to modify the metrics server</li> We didn’t need to configure Prometheus</li> We didn’t need to write new security code</li> We just needed to tell the mesh “hey, ztunnel is a workload too”</li> </ul> This is exactly why organizations adopt service mesh - it provides security and observability without changing applications which stay simple and focused on core functionality. No need for applications to implement complex security protocols. The fact that we could extend this to ztunnel itself shows how powerful and flexible the architecture is. So, I told the tale of my internship, but if you want to learn more, you can read further about the ztunnel internals…. But what’s Ztunnel?</a></h2> Ztunnel (Zero Trust Tunnel) is a core component of Istio Ambient Mesh</a>. It acts as a secure transport proxy running on each node. Its main job is to handle traffic redirection and establish secure HBONE tunnels</a> between different pods using mutual TLS (mTLS) based on workload identities - SPIFFE</a>. It manages traffic without requiring traditional sidecars for every application pod, receiving configuration dynamically from the Istio control plane (Istiod) via xDS</a>. It can operate in a shared mode, serving multiple pods on a node. Plenty of cool blogs from the cool folks at CNCF to learn more : Lin Sun’s</a> - blog introducing ztunnel for Istio’s Ambient Mesh</a></li> Marino Wijay’s</a> - blog with cool diagrams about it</a>.</li> </ul> Well, the architecture is explained in short in the repository</a>, but that’s just too less information for me. So, as I already intended to deep-dive into the ztunnel codebase, I am writing about it. Understanding its Architecture</a></h2> The most fundamental job of ztunnel is Traffic Interception & Routing</a>. On Linux, orig_dst_addr</a> uses the getsockopt system call</a> with IPv4/v6 socket options. These options allow a program to retrieve the original destination address of a connection that was redirected by the netfilter framework (using iptables with REDIRECT</a> or TPROXY</a> targets). For platforms other than Linux, ztunnel’s transparent interception capabilities relying on this mechanism would be limited or would require different OS-specific approaches. The Config</a> struct is the single source of truth for ztunnel’s runtime parameters. Once it’s constructed</a>, various parts of ztunnel will refer to it to make decisions. It has Shared and Dedicated Proxy Modes</a> built via ProxyFactory</a> : +-------+ | Start | +-------+ | v +-------------+ | Which Mode? | +-------------+ / \ / \ / \ Dedicated Shared | | v v +----------------+ +--------------+ | Create Default | | Get Pod's | | SocketFactory | | NetNS | +----------------+ +--------------+ | | v v +----------------+ +--------------+ | Packet Mark | | Create InPod | | Configured? | | SocketFactory| +----------------+ +--------------+ / \ | Yes No v | | +--------------+ v | | Port Reuse | +--------------+ | | Enabled? | | Wrap with | | +--------------+ | MarkFactory | | / \ +--------------+ | Yes No | | | | | | v | | | +--------------+ | | | | Wrap with | | | | | ReuseFactory | | | | +--------------+ | | | | | +------------+------------+----------+ | v +----------------------+ | Create LocalWorkload | | Information | +----------------------+ | v +----------------------+ | DNS Proxy Enabled? | +----------------------+ / \ Yes No | | v | +------------------+ | | Create DNS Server| | +------------------+ | | | v | +------------------+ | | Get Resolver | | +------------------+ | | | +--------------+ | v +----------------------+ | Main Proxy Enabled? | +----------------------+ / \ Yes No | | v | +------------------+ | | Create Connection| | | Manager | | +------------------+ | | | v | +------------------+ | | Create Proxy | | | Inputs | | +------------------+ | | | v | +------------------+ | | Create Proxy | | +------------------+ | | | +--------------+ | v +----------------------+ | Return ProxyResult | +----------------------+ | v +-----+ | End | +-----+</code></pre>Whats this HBONE thingy?</a></h3> HBONE (HTTP-Based Overlay Network Encapsulation) is Istio specific thing which uses HTTP/2 as foundational protocol (HTTP CONNECT and standard mTLS). It creates a virtual network layer on top of the existing physical network. This “overlay” is where our secure tunnels live. It takes regular application traffic (which is usually TCP traffic) and “wraps” or “encapsulates” it inside these HTTP/2 streams. HBONE wraps standard TCP traffic (what your applications speak) inside an HTTP/2 stream. This HTTP/2 stream is then secured using mutual TLS (mTLS). So all the traffic flowing through the HBONE tunnel (i.e., within the mTLS session) is encrypted. Eavesdroppers on the network cannot understand the data. Pod A and Pod B just use standard TCP. They are completely unaware of the mTLS and HTTP/2 encapsulation happening underneath by ztunnel. HTTP has a CONNECT method, traditionally used for proxying. HBONE cleverly reuses this. To establish a tunnel to a target pod, the source ztunnel sends an HTTP/2 CONNECT request to the destination ztunnel. Establishing an mTLS handshake and a new HTTP/2 connection for every single small request between two pods would be slow and inefficient. To solve this, ztunnel implements sophisticated connection pooling for HBONE connections through the WorkloadHBONEPool</a>. If an existing, healthy, mTLS-secured HTTP/2 connection to ztunnel is already in the pool and has spare capacity (HTTP/2 allows many streams on one connection), that connection is reused. A new HTTP/2 stream is created on that existing connection for the current request. The H2ConnectClient</a> represents an active, pooled HTTP/2 client connection to a remote ztunnel. The pool includes sophisticated mechanisms to manage connection lifecycle : When a new connection is needed, the pool ensures that only one task creates a connection for a given destination, even if many requests arrive simultaneously</a></li> The pool tracks how many HTTP/2 streams are active on each connection and won’t exceed the configured limits</a></li> Connections that aren’t used for a configurable period are automatically closed to free up resources</a></li> HTTP/2 connections use PING frames to verify connection health</a></li> </ul> +--------------------------+ | Outbound request needs | | HBONE | +--------------------------+ | v +--------------------------+ | Check connection pool | | for existing connection | | to destination | +------------+-------------+ | +---------+---------+ | | v Connection v No Connection / At Capacity +----------------+ +--------------------------+ | Reuse existing | | Try to acquire lock for | | connection | | destination | +----------------+ +------------+-------------+ | | | +---------+---------+ | | | | v Lock Acquired v Lock Acquisition Failed | +-----------------+ +--------------------------+ | | Create new mTLS | | Wait briefly, then retry | | | + HTTP/2 conn | | pool check (go back up) | | +-------+---------+ +------------+-------------+ | | ^ | v | | +-----------------+ | | | Add connection |----------------+ | | to pool | | +-------+---------+ | | | v Connection Ready | | +--------+--------+ | v +-----------------+ | Create new | | HTTP/2 stream | +-----------------+ | v +-----------------+ | Send HTTP/2 | | CONNECT request | +-----------------+ | v +-----------------+ | Wait for 200 OK | | response | +-----------------+ | v +-----------------+ | HTTP/2 stream | | ready for data | +-----------------+</code></pre> By wrapping application TCP traffic in mTLS-secured HTTP/2 streams, ztunnel provides strong authentication, encryption, and integrity for inter-pod communication, all without requiring any changes to the applications. Ztunnel transparently manages the creation, termination, and pooling of these secure tunnels. How does a connection work?</a></h3> The Proxy</code> struct orchestrates the different parts of ztunnel : pub struct Proxy { inbound: Inbound, // Handles inbound HBONE traffic inbound_passthrough: InboundPassthrough, // Handles inbound non-HBONE TCP traffic outbound: Outbound, // Handles outbound traffic from local applications socks5: Option<Socks5>, // Optional SOCKS5 proxy support policy_watcher: PolicyWatcher, // Watches for policy changes }</code></pre> The pool tracks connection by workload key : pub struct WorkloadKey { pub src_id: Identity, // Source workload identity pub dst_id: Vec<Identity>, // Destination workload identity/identities pub dst: SocketAddr, // Destination address pub src: IpAddr, // Source IP address }</code></pre> Outbound Traffic: When your application pod tries to send a message to another service, ztunnel intercepts this message before it leaves the node. It decides whether the outbound traffic needs to go directly via TCP, or HBONE tunneling or through Waypoint/Gateway. The outbound’s build_request</a> handles the routing by constructing a Request</a> with routing decisions made, and then passed to appropriate protocol handler</a>. </li> Inbound Traffic: When a message arrives at the node destined for one of your application pods, ztunnel also intercepts this message before it reaches the pod. If its wrapped in an HBONE tunneling (meaning it came from another ztunnel or a compatible component), it decrypts the traffic, verifies the identity of sender using Workload and Certificates, and then forwards the plaintext TCP to local pod</a> after checking the security policies via serve_connect method</a>. When building the inbound request</a>, validation checks</a> ensure security, after which we determine the actual socket address to forward the traffic to</a>. Before everything, TLS handshake happens verifying server and client identities via InboundCertProvider</a>. </li> </ul> ProxyState</a> is the knowledge base that stores information about workloads and services. ConnectionManager</a> enforces policies and tracks active inbound and outbound connections. SocketFactory</a> creates properly configured sockets for each connection type. PolicyWatcher</a> monitors for policy changes and updates the connection manager. The flow of a connection : +--------+ +--------+ +--------+ +--------+ | Pod A | | zt A | | zt B | | Pod B | | Node 1 | | Node 1 | | Node 2 | | Node 2 | +--------+ +--------+ +--------+ +--------+ | | | | | | | | | Request to B | | | |---------------->| | | | | Intercept Req | | | | Analyze | | | | Decide HBONE | | | | Establish HBONE | | | | Tunnel | | | |---------------------------------->| Establish HBONE | | | | Tunnel | | Encrypted Req | | | |---------------------------------->| | | | | Receive & | | | | Decrypt Req | | | | Verify Identity | | | | Plaintext Req | | | |---------------->| | | | | | | | | | | Process Req | | | | Plaintext Resp | | | | |<----------------| | | | Encrypt Resp | | | | Encrypted Resp |<----------------| | | |<----------------------------------| | | | | | | | Decrypt Resp | | | | | Plaintext Resp | | | | |<----------------| | | | | | | | |</code></pre> The data forwarding is handled via copy_bidirectional</a> which efficiently shuttles bytes between two connections ensuring data flows smoothly once ztunnel has decided where it should go. Workload Identities</a></h3> A Workload</a> in ztunnel represents an instance of your application, typically a pod in Kubernetes. Ztunnel helps manage cryptographic identities for the workloads running on its node using SPIFFE ID</a> - spiffe://your-trust-domain/ns/your-namespace/sa/your-service-account</code>. To prove a workload actually owns a SPIFFE ID, it gets an X.509 certificate</a> and creates a CSR with the SPIFFE ID sending to Istiod over gRPC connection</a>, receiving response and verifying the certificate</a> and returning and storing complete Workload Certificate</a> used for mTLS : +------------------------+ +-----------------------------------+ +---------------------------------+ | Pod Alpha (Workload) | | ztunnel (on Pod Alpha's Node) | | Istiod (Certificate Authority) | | (PA) | | (ZT) | | (ICA) | +------------------------+ +-----------------------------------+ +---------------------------------+ | | | | 1. Workload Starts Up | | +--------------------------------------->| [ ZT activated ] | | | | | | | | 2. Determine Pod Alpha's Identity | | | | (e.g., spiffe://cluster.local/ | | | | ns/default/sa/alpha) | | | | | | | | 3. Generate CSR & Private Key | // Note: CSR contains Pod Alpha's | | | for Pod Alpha | // public key & desired identity (SAN) | | | | | | | 4. Send CSR to Istiod | | | +-------------------------------------->| [ ICA activated ] | | | | | | | | | | 5. Validate CSR | | | | | (Authenticate ztunnel, | | | | | check policies) | | | | | | | | | | 6. Sign Certificate | | | | | (embedding Pod Alpha's | | | | | identity) | | | | | | | | 7. Send Signed Certificate back | | | | +<--------------------------------------| | | | | [ ICA deactivated ] | | | | | | | 8. Store Certificate & Private Key | // Note: Certificate is now ready | | | | // for mTLS handshakes | | [ ZT deactivated ] | | | |</code></pre> Requesting a certificate</a> from the CA for every connection would be very slow. So, ztunnel uses a SecretManager to efficiently manage and cache these certificates. The SecretManager and its supporting components: Keeps track of certificates for all workloads on the node.</li> Handles requests to fetch certificates, possibly returning a cached one if it’s still valid.</li> Schedules background tasks to refresh certificates before they expire, ensuring workloads always have a valid certificate.</li> Manages the concurrency of requests to the CA to avoid overwhelming it.</li> </ul> +------------------------+ | Certificate refresh | | timer | +------------------------+ | v Application +----------------+ +-----------------+ makes request ---> | SecretManager | No | Certificate | | Has valid |------------> | needs refresh? | | cached cert? | +-----------------+ +----------------+ | | | Yes | Yes v | +------------------+ | | Queue background | | | refresh | | +------------------+ | | | v | +------------------+ | | Worker checks | | | cert priority | | +------------------+ | | | v | +------------------+ | | Worker requests | | | cert from CA | | +------------------+ | | | v | +------------------+ | | CA validates and | | | returns cert | | +------------------+ | | | v | +------------------+ | | Store cert in | | | cache | | +------------------+ | | | v | +------------------+ |<------------------------| Notify waiting | | | applications | v +------------------+ +------------------+ | Return cached | | certificate | +------------------+</code></pre> Each workload certificate is proactively refreshed halfway through its validity period, ensuring smooth operations without disruption due to expired certificates. xDS client</a></h3> Your service mesh is not static. New application versions get deployed (new workloads), old ones are removed, services scale up or down, and security policies change. ztunnel needs to know about all these changes in real-time to make correct routing and security decisions. How is stuff of ambient mesh updated in real time? The XDS Client in ztunnel is the component responsible for communicating with Istiod to receive these dynamic configuration updates. ztunnel uses a specific, efficient version of XDS called Delta ADS</a> Aggregated: Multiple types of information (like workloads, services, policies) are sent over a single connection, which is efficient.</li> Delta: Instead of sending the entire configuration every time something small changes, Istiod only sends the differences (deltas – what’s new, what’s changed, what’s removed). This saves a lot of network traffic and processing time.</li> </ul> The xDS Client</a> connects to Istiod after checking config.xds_address</code>, tells it what kind of information ztunnel is interested in, and then receives a stream of updates. These updates are then used to keep ztunnel’s internal Proxy State Management in sync. Once connected, the AdsClient</code> sends initial DeltaDiscoveryRequest messages</a> to Istiod for each resource type it’s interested in (like ADDRESS_TYPE for workloads/services and AUTHORIZATION_TYPE for policies). This request includes a Node identifier, which tells Istiod who this ztunnel is (e.g., its IP, pod name, namespace). This helps Istiod send relevant configuration. Istiod responds with DeltaDiscoveryResponse messages</a>. The first response usually contains the current state for the subscribed resources. Subsequent responses contain only the changes (deltas). The ProxyStateUpdater</a> receives the decoded XDS resources (e.g., XdsAddress which can be a workload or service, or XdsAuthorization for policies). The xDS messages themselves are defined as Protocol Buffers (protobufs)</a>. ztunnel (XDS Client) Istiod (XDS Server) | | | | |---------------------------→| 1. Establish Secure gRPC Connection (mTLS) | Uses workload ID | | | |←---------------------------| Connection Established | | |---------------------------→| 2. Send DeltaDiscoveryRequest | Subscribe to types | | | |←---------------------------| 3. Send DeltaDiscoveryResponse | Initial mesh state | | | | ∙ | | ∙ | Ongoing Updates Loop: | ∙ | | | |←---------------------------| 4. Send DeltaDiscoveryResponse | Workload/Policy updates | | | | /-----------------\ | 5. Process updates | | Update internal | | | | ProxyState | | | \-----------------/ | | | |---------------------------→| 6. Send ACK/NACK | Update confirmation | | | | ∙ | | ∙ | Loop continues... | ∙ |</code></pre> For very large meshes, ztunnel might not want to receive all configuration for all workloads and services upfront. So, there is an on-demand XDS</a> feature which allows it to fetch only what it needs, when it needs it. The proxy state manager can request specific resources when it needs them</a>. The AdsClient listens for demand requests and sends them to Istiod</a>. When a response containing the requested resource arrives, ztunnel notifies the waiting component</a>. Even if some resources in an update fail, the valid ones are still processed. Proxy State Management</a></h3> The heart of ztunnel’s state management is ProxyState</a> which has specilized stores - records of all individual workloads, catalogs all the services, and archives all the security policies. The WorklaodStore</a> is optimized for efficient lookups of Workloads. The store even optimizes for the common case where a single IP maps to a single workload. This structure provides multiple indices to find workloads efficiently by: IP address + network</li> Unique identifier (UID)</li> Identity (for local workloads)</li> </ul> The ServiceStore</a> manages information about Services. A Service</a> is an abstraction for a group of workloads that together provide a certain functionality. Clients usually talk to a service’s stable virtual IP (VIP) and port, and the mesh routes the request to one of the healthy backing workloads. This structure allows for extremely fast lookups of services by: VIP (Virtual IP address)</li> Hostname</li> Namespace + hostname combination</li> </ul> The PolicyStore</a> holds all the Authorization policies. These policies are the RBAC rules</a> that dictate traffic flow. The policy store efficiently indexes policies by: Full key (namespace/name)</li> Namespace (for quickly finding all policies in a namespace)</li> </ul> It also includes a notification system to alert subscribers when policies change. While ProxyState organizes the data efficiently, accessing it directly would be cumbersome for most operations. The DemandProxyState wrapper provides higher-level functionality and serves functions : Concurrent access management: It safely handles the RwLock to ensure thread safety</li> On-demand resource fetching: It can request resources that aren’t in local cache</li> Metrics tracking: It maintains metrics about state operations</li> DNS resolution: It handles DNS lookups when needed</li> </ul> When an application pod (say, Pod A) sends a request to “Service B”, ztunnel (managing Pod A) intercepts this. So, the decision for routing happens : +----------------+ | Intercept | | request to | | Service B | +----------------+ | v +----------------+ | Look up | | Service B | +----------------+ | +---------------------+--------------------+ | | | | Found Not | v found v +----------------+ | +----------------+ | Get healthy | | | Try on-demand | | endpoints | | | fetch? | +----------------+ | +----------------+ | | | | v | | +----------------+ | Yes | | Request | | | | Service B |<--------+ | | from Istiod | | +----------------+ No | | | +----+ | v | | +----------------+ | | | Re-check | | | | ServiceStore | | | +----------------+ | | | | | Found | Not found | +--------------------+ | | | v v +----------------+ +----------------+ | Select endpoint| | Fail request | | Workload C | +----------------+ +----------------+ | v +----------------+ | Look up | | Workload C | +----------------+ | v +----------------+ | Get IP, port, | | protocol info | +----------------+ | v +----------------+ | Has waypoint? | +----------------+ | +-------+-------+ | | Yes No | | v v +----------------+ +----------------+ | Route through | | Route directly | | waypoint | | to Workload C | +----------------+ +----------------+</code></pre>DNS Proxying</a></h3> DNS Proxying</a> in ztunnel is an optional feature</a> where ztunnel itself helps your applications (pods) find the IP addresses of other services. It acts as a local DNS resolver</a> and forwarder</a>. This speeds up lookups for services within the mesh because ztunnel can answer directly, and it ensures consistent name resolution. When DNS Proxying is enabled: Interception: Network rules (often iptables) on the node are set up to redirect any DNS queries (on UDP port 53) sent by your local application pods to ztunnel’s DNS proxy listening port (e.g., 15053, configured by config.dns_proxy_addr</code>).</li> Local Resolution Attempt: ztunnel receives the DNS query. It first checks if the requested name belongs to a service within the mesh using its internal Proxy State Management. If it’s a known mesh service (e.g., checkout.default.svc.cluster.local</code>), ztunnel resolves it to the service’s Virtual IP (VIP) or endpoint IPs (for headless services). It then sends a DNS response directly back to the application pod. Queries for mesh services are resolved directly from the local state, avoiding network roundtrips, making it very fast.</li> Caching: The DNS responses include appropriate TTLs to allow client-side caching.</li> Forwarding: If ztunnel doesn’t know the name, it forwards the DNS query to the upstream DNS servers that are configured in its environment (typically kube-dns</a> or the node’s resolvers). If a service is not found in the mesh state, the query is seamlessly forwarded to upstream DNS.</li> Relaying Response: When the upstream DNS server responds, ztunnel relays this response back to the application pod.</li> </ul> With the help of DNS proxying, we get : Speed: Resolving mesh-internal service names locally is much faster than going to an external DNS server.</li> Accuracy: ztunnel uses its live Proxy State Management data, ensuring that DNS results for mesh services reflect the current state of the mesh.</li> Consistency: Helps ensure all pods get a consistent view of service addresses within the mesh.</li> Reduced Load on Upstream DNS: Offloads queries for internal services from kube-dns or other central resolvers.</li> Enhanced Flexibility: Supports both standard and headless services with appropriate IP addressing.</li> </ul> Shutdown</a></h3> Ztunnel constantly listens for shutdown signals. This tells the DrainTrigger to notify all DrainWatchers and then waits for them to finish with their DrainMode. The drain::new()</code> function creates the DrainTrigger and its initial DrainWatcher. This DrainWatcher can be cloned and passed to any component that needs to participate in the graceful shutdown. Observibility</a></h3> Good observability is crucial for any service mesh component, especially one handling production traffic like ztunnel. It provides comprehensive metrics across all its core functions - traffic proxying, DNS resolution, TLS certificate handling, and more, and integrate seamlessly into Prometheus, making them easy to visualize using tools like Grafana and setup AlertManager. When ztunnel starts up, it creates a central Prometheus registry that will hold all metrics</a>. The metrics::sub_registry()</a> function creates a namespace for Istio-specific metrics to avoid conflicts with other systems. Each component (Proxy, DNS, XDS, etc.) defines its own metrics in a dedicated module. For connection-oriented operations like proxying traffic, ztunnel uses a pattern where metrics are incremented at the start of an operation and then finalized when the operation completes. Ztunnel exposes its metrics through an HTTP endpoint in the Prometheus format</a> with a rich set of labels which allows you to filter and group metrics by workload, namespace, service, security policy, and more, allowing bifurcation like : Identifying which services generate the most traffic</li> Spotting security policy violations</li> Tracking error rates between specific service pairs</li> Monitoring the performance impact of mesh configuration changes</li> </ul> There are access logs for detailed event tracking as well. It’s profiled</a> and benchmarked extensively</a>. The Rusty aspect of Ztunnel</a></h3> Istio’s Ambient Mesh was specifically designed for Scalability</a>. But Rust is not the reason Ztunnel is scalable. You won’t believe me, but the creator of ztunnel wrote it in Go, but was not satisfied with the performance</a>, renamed all the files to .rs</code> and with the help of the Rust compiler did a complete rewrite in Rust (◕▽◕) Crazyyyyy… Ztunnel boasts a dual runtime</a> model : Control Plane Runtime is the main thread</a> which handles tasks that are essential for the proxy’s management and configuration but are not directly impacting the data plane. It includes xDS client fetching configuration updates from Istiod, debug interfaces and health checks</a> impacting the data plane. As the Admin tasks</a> are often sequential or don’t require high parallelism and requires less resources that could otherwise be used for data plane tasks, a single thread is easier to reason about and debug. Crucially, if something goes wrong in an admin task (e.g., a bug in xDS handling, a panic in a debug endpoint), it’s less likely to crash or stall the entire data plane that’s handling live user traffic. The user request flow remains unaffected. </li> Data Plane Runtime have the worker threads</a> which handles the actual user traffic. This is the high-performance low-latency multi-thread Tokio runtime route to handle users requests which includes accepting incoming client connections, processing request headers/data, applying policies (load balancing, retries, timeouts, security), establishing outgoing connections to upstream services (this is where the “connection pool” comes in), forwarding data between client and server. The worker thread count (defaults to 2) is configurable based on the expected load and available hardware resources, common starting point often being the number of CPU cores spreading the load of encrypting/decrypting traffic, parsing, and other request processing tasks across multiple CPU cores. Tokio’s multi-threaded runtime can efficiently manage many concurrent connections across a few OS threads by leveraging non-blocking I/O. This allows ztunnel to handle a high volume of requests simultaneously. </li> </ul> Ideally, these two runtimes operate largely independently. This separation is a robust design. The admin runtime can be busy fetching a large xDS update, or even hang momentarily, without directly stalling the forwarding of packets on the worker threads. The connection pool is managed by worker threads. When establishing connection it consults the current configuration (provided by the admin runtime) to know where to connect, thus this configuration needs to be made available to the worker threads in a thread-safe way, done using mpsc channel. Conclusion</a></h2> I urge folks who are interested to go through the ztunnel codebase and contribute to the project, its really well designed, and too big to cover in a single blog. A lot of things to learn from the codebase. Don’t forget to Star it as well! Though I have covered a lot of technical aspects in here. Grateful</a></h2> I am more than grateful to the wonderful folks at Solo.io who mentored me - Ian Rudie</a>, John Howard</a>, Ben Leggett</a> and Faseela K</a>. Forever grateful to them for volunteering their time and effort. Do I officially qualify as a Software Engineer now? Unknown — Mon, 13 Jan 2025 00:00:00 +0000 With another internship coming to end, I officially have around 1 year systems programming experience (internship experience tbh) and I am quite sad, because I still don’t seem to have much knowledge ( ͡° ʖ̯ ͡°) or I am just working with extremely smart people, who knows? I am really thinking of pursuing research in Computational Sciences. Confused… To many lucrative options. This writeup is how I am thinking of writing my Rust applications from now on. It’s not your personal project…</a></h3> Well, production software has to be written keeping the conventional wisdom in mind, i.e. Design Principles</a> along with Design Patterns</a>. When creating a simple personal project, I never took care of using abstraction much, but I saw the senior engineers using it quite a lot. I still am not a fan of unnecessary abstractions tbh. And I like things stupid-simple. I was very procedural when I started out, and now I look at OOPs as an organizational principle</a>. But don’t get on horses with proc-macro and trait magic to the point that the code is bloody incomprehensible and extremely difficult to debug. My focus is to make the code low-latency and performant, easy to traverse by new-comers i.e. very debuggable, maintainable and makes future migrations easy-peeasy. And, No over-engineering! Wisdom of programming sages</a></h3> I think of a hybrid of Hexagonal Architecture</a> and Actor Model, keeping in mind the Dependency Inversion priciple as Uncle Bob calls it - Clean Architecture</a> and closely following Data-oriented approach</a>. I don’t feel happy as a traditional OOPs guy! Onion Architecture</a> also shares similar principles, but was originally limited to OOPs. I ofcourse use objects, inheritance and all, but I despise debugging a enterprise OOPs codebase, just going through classes and classes, and those never ending classes. I favor composition over inheritance to avoid inheritance spaghetti, i.e. unexpected coupling of system components. Hahhh…. the I need to understand Data Abstractions</a> and overall Programming Paradigms in Rust</a> better Trancending OOPs</a></h3> Yes, I maintain a plugin in Jenkins, which is completely based on traditional JAVA and its OOPs concepts. But, when writing performant software in Rust, I don’t think OOPs is the best thing. Or maybe it’s just me who doesn’t like Abstract Classes which encourage tight coupling and hinder cache alignment. The problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. – Joe Armstrong, the creator of Erlang In OOPs, there are layers and layers of abstractions, and logic deep within. It almost promotes complexity ( ͠° ͟ʖ ͡°), I like things simple. Deep hierarchies often lead to derived classes depending on sibling classes through shared base class behavior. This interdependence makes it harder to refactor or optimize parts of the system independently. There are plenty of arguments against OOPs</a>, but still, OOPs is undeniably and rightfully so, a really reliable guiding principle. Poorly designed inheritance hierarchies lead to fragility and coupling, paired with overengineering (e.g., excessive patterns) detracts from solving actual problems. The developer overly focus on code instead of data which shouldn’t be the case. I am learning about Data-oriented design</a> which avoids complex reference graphs by storing data in flat, database-like structures, where the relationships are ID-based, avoiding circular referencing, and access patterns are simple. I guess, the best way forward to mingle both OOPs and DoD, combining DoD’s efficiency with OOP’s modularity. I fear over-engineering, don’t wanna fall into that trap! And DoD just feels less natural for me for some reason. Hybrid Architecture of my dreams…</a></h3> The essence is that my core business logic should be ignorant about the rest of the system. External things like API, DB, cache, filesystem, etc, should be abstracted by interfaces. Interfaces make interactions of my applications with the outside world well described and structured. Also, I can simply test these interface implementations. But I want good abstractions, not layers and layers of code. What abstractions should achieve is they should help me reason about the solution in simpler terms. In essence they should allow me to think about problem from a birds eye view. It’s really not easy to create a abstraction most often we just add layers of functions not abstractions. Good abstraction decoupled low level thinking and high level thinking. [Each actor is like a small little application, self-contained, capable of performing its tasks (often as multiple spawned threads - task is not a thread</a>) and interacting with other actors through channels](https://www.youtube.com/watch?v=fTXuGRP1ee4), mpsc or broadcast or watch, just avoid over-relying on mpsc</a>, and can be run and tested seperately using mock channels. Actors communicate by sending messages rather than sharing memory. The main application code is then just an actor coordinator - setting-up main actors, gluing them together with channels, starting them, then detecting exit conditions and/or critical failures and coordinating shutdown</a>, etc. This model is particularly powerful in building concurrent, distributed, and fault-tolerant systems, which I was generally involved in building during my internships. Managing a shared I/O resource is just better with actors. I just create a Spawn</code> proc-macro</a> for generating spawn methods for actor structs, consistent cancellation via CancellationToken for graceful shutdown</a> and uniform tracing and error handling. I love Rust macros! But tracing::instrument</code> in particular is slow! Use derive_builder</code> and the verbose stuct builder boilerplate is gone. My favourite tonic</code> uses macros for gRPC. And, I generally aim for immutability.</a> The hexagonal architecture is stupid simple not only because it makes the codebase modular, but also because it allows me to defer some decisions to a later stage in the development of a system. Btw, Quickwit uses the actor model really well!</a> and this blog</a> is a very easy to follow actor model implementation. Hilariously, Rust compiles fast software slowly!</a> In Rust, each crate is a compilation unit (not file like in C++) either compiling into a binary bin</code> or library rlib</code>, which is good as it eases global optimization across modules, and is just better! When any module in a crate is changed, the crate is recompiled, pushing us to have multiple small crates instead of a large slow crate. So, I need to think about my dependency spaghetti, avoiding as many inter-dependencies as I can and arrange the crate dependencies such that as much of my workspace can be can benefit from parallel compilation. And, I need to use my intuition to keep the ever-changing code as outside the spaghetti, as I don’t want to recompile the whole damn thing just after a simple tweak. Sometimes, just introduce an extra Arc<dyn Trait></code> somewhere just to decouple two things and avoid dependency between them, instead of both of them depending on a common interface (which changes much less frequently). I recently found about mold, a high-performance linker</a>, which is a absolute heart-throb! Who won’t appreciate faster compile time in Rust?</a> Well, we can tweak the configs for faster Rust builds</a>, though they can be fragile. One more thing to consider is that CI/CD should prioritize speed and relibility otherwise its bad for developer productivity</a>. The default dev profile enables debug symbols, slowing down CI builds, and the default release profile optimizes for runtime speed, but CI might need faster compilation instead. So using a custom build profile is just better! Cache the CI smartly</a>. [profile.ci] inherits = "release" # Inherit default release opt-3 level debug = false # Debugging is not required, reduce binary size lto = false # Disable Link-Time Optimization as it slows linking down codegen-units = 32 # Increase parallelism for faster compilation incremental = false # Disable incremental compilation for clean builds ensuring consistency</code></pre> but when you want to release the optimized binary : [profile.release] debug = false lto = true</code></pre> Faster CI builds in Rust significantly improve developer velocity.</a> Match made in Heaven</a></h3> Rust is not exactly an OOP language. Rust does not support inheritance</a> and its trait system is modular by design. I am genuienly wondering, how Rust is gaining popularity regardless of the fact that its not OOPs-centric, just like OOPs at surface (゜ロ゜). Its amazing! Maybe because it’s a general-purpose language, so you can build backends, CLIs, GUIs, and ofcourse embedded firmware. Also, cargo is just the best build+package manager that I have ever used. And also a feeling that Rust makes you confident about your program. Traits are like interfaces in Java</a>, but is inspired by Haskell’s</a> typeclass. Literally, traits + generics + lifetimes, and the chef’s kiss!!! Code-sample because I am proud of understanding lifetimes</a>, atleast naively : use std::cmp::Ordering; // Trait holding two references with distinct lifetimes trait Comparator<'a, 'b> { fn compare(&self) -> Ordering; } // By separating these lifetimes, Rust ensures that the struct's // validity depends on the shorter of the two lifetimes. struct Comparison<'a, 'b, T> where T: 'a + 'b, { left: &'a T, right: &'b T, } impl<'a, 'b, T> Comparator<'a, 'b> for Comparison<'a, 'b, T> where T: Ord + 'a + 'b, { fn compare(&self) -> Ordering { self.left.cmp(self.right) } } // A function that takes a 'static reference fn static_comparator<'a, T>(value: &'static T, other: &'a T) -> &'static T where T: Ord + 'a, { // Simply returns the `'static` reference as it outlives the other value } fn main() { // Long-lived data let long_lived = String::from("Hello, Rust!"); { // Short-lived data let short_lived = "Hello"; // Comparison of references with different lifetimes let comparison = Comparison { left: &long_lived, right: &short_lived, }; println!( "Comparison result: {:?}", comparison.compare() ); // Greater } // Demonstrating `'static` usage let static_str: &'static str = "Static reference"; let result = static_comparator(static_str, &long_lived); println!("Result from static comparator: {}", result); }</code></pre> Just adding one confusion that I faced when working on a personal project and how I solved it : let (chord_handle, mut actor) = ChordHandle::new(node_id, port, addr.clone()).await; // Spawn the actor and store its handle let actor_handle = tokio::spawn(async move { actor.run().await; // returns a future which borrowed actor rather than owning it so move captures the actor and moves ownership to the spawned task });</code></pre> when I was directly trying to spawn the actor tokio::spawn(actor.run());</code> I was getting a static lifetime issue! I went through the tokio’s docs</a> and found There is no guarantee that a spawned task will execute to completion. When a runtime is shutdown, all outstanding tasks are dropped, regardless of the lifecycle of that task.</code>. Now, in std::spawn() docs</a> its mentioned The 'static constraint means that the closure and its return value must have a lifetime of the whole program execution. The reason for this is that threads can outlive the lifetime they have been created in.</code>. So the 'static</code> can outlive the lifetime 'a</code>! async fn run() { let x = 42; tokio::spawn(async { println!("{}", x); // ❌ ERROR: `x` does not have a `'static` lifetime }).await.unwrap(); }</code></pre> x</code> is allocated on the stack of main()</code>, but the spawned thread may continue running after main()</code> exits. async fn run() { let x = 42; tokio::spawn(async move { // ✅ Move `x` into the async block println!("{}", x); }).await.unwrap(); }</code></pre> Using move, we ensure the closure takes ownership of x</code>, making sure it is available for the spawned thread. To avoid all this, we can simply use crossbeam::scope</code> which creates a temporary scope where threads are spawned. n main() { let x = 10; thread::scope(|s| { s.spawn(|_| { println!("{}", x); // ✅ Works fine, as `x` is guaranteed to live }); }); // Scope ensures all threads finish before exiting }</code></pre> Since the scope will make sure that all threads are joined before the scope ends, the closures don’t need to be 'static</code>. I cant emphasize on the fact how helpful the Rust compiler is. Once you be-friend the borrow checker, you are sorted. I use Tokio’s JoinSet</code> for task lifecycle management and Arc<RwLock<T>></code> for thread-safe shared state</a>. Futures are lazy by default, so they don’t do any work until they’re polled. This allows for patterns where computations are delayed until they’re actually needed. Also, callback chains are easy using then</code>, and_then</code> and or_else</code>.</a> I can simply use join!</code> macro to await multiple futures concurrently, and join_all()</code> for fan-out, fan-in patterns. I personally like using type-alias instead of raw primitive types. Just makes the code so much more readable. I like using custom types to model my domain.</a> Error handling in Rust is just gorgeous</a>. And stop using unwraps all around the place!</a>. Just throw the ?</code> operator</a> all around the place. When I heard about Rust, fearless concurrency was something I kept on hearing about. And stupid me thought I don’t have to worry about race conditions and deadlocks when using Rust</a> at all now (◞‿◟) I use fixtures and paramatrization when testing</a> a lot these days when writing property-based tests</a>. Also, for similar test cases, I use test_case</a> procedural macro for generating parameterized tests and PropTest</a> for generating property tests. And ofcourse follow general rust testing</a> and its organizational</a> guidelines. Also, I use cargo-bloat</a> for size profiling my binaries. async/await internals</a></h3> OS threads are never going to be fast. Rust chose Stackless coroutines which added complexity to semantics but improved performance, compared to Green threads which give same semantics as OS level threads but is a downgrade on performance as now a runtime scheduler to do preemptive multitasking</a>. Async + Lifetime is surely hard to wrap your head around. In Rust, the async/await model turns the async block into a state machine. Each await point can cause the future to be paused and resumed later, possibly on a different thread. Futures are self-contained concurrent execution units, just like threads to multi-cores. If you spawn a Future it must be Send + Sync + 'static</code></a>. For the future to be Send</a>, all the data it holds across await points must also be Send. Having a non-Send type in Future is okay (just means you can only execute it in a single-threaded runtime), but holding it across await is not! Each .await</code> introduces a new state (while it waits for something) and the code in between are state transitions (a.k.a tasks), which will be triggered based on some external event (e.g. from IO or a timer etc). Something like Rc</code> is not Send, since cloning/dropping in two different threads can cause the reference count to get out of sync, so it can’t be sent among threads. Each task gets scheduled to be executed by the async runtime, which could choose to use a different thread from the previous task. If the state transition is not safe to be sent between threads then the resulting Future is also not Send so that you get a compilation error if you try to execute it in a multi-threaded runtime. Async Rust is certainly a pleasure to work with</a> ◴_◶ Async code should never spend a long time without reaching an .await.</a> until you wanna block the thread. The primary difference between what Go does and what async Rust does is whether the scheduling is preemptive or cooperative. In async Rust, your runtime will swap out the currently running task on each worker thread whenever it reaches an .await and has to wait for IO or something, but the swap can only happen at an .await. In Go, this is different, the scheduling is preemptive, which means that a task can be swapped at any time. In Go, everything is implicitly async, which is not the case with Rust. If your main()</code> function is not marked with #[tokio::main]</code>, it is synchronous by default in Rust. The tokio’s rewrite PR</a> made the scheduler really good! I was already a fan of Go’s unique runtime scheduler</a> due to it’s combination of features like very small stacks for creating a very large number of goroutines, Colorless functions, built-in, runs without VM, integrated with garbage collector, stackful, etc. Rust did start with an M:N, work-stealing scheduler based on stackful coroutines, but was removed from the std lib</a>. The groundbreaking thing that Go did is to optimize the entire language for highly concurrent I/O-bound workloads</a>, but Rust is strictly more performant than Go. It’s using state-machine based stackless coroutines, which emulate the way that manual asynchronous implementations like Nginx. You can’t get more efficient than this along with the fact that Rust doesn’t have a GC and features more aggressive compiler optimizations. Recently I was working on a personal project, and got really confused on why even after dropping the MutexGuard</code> I can’t hold it across await!</a> (◎＿◎;) So, the issue is that the compiler can’t always track that the MutexGuard</code> is no longer present after the drop. The compiler’s analysis of what lives across await points is conservative. The state machine might still include it in its captured variables, making the future non-Send. Using blocks instead of drop()</code> is better practice because it ensures all variables in the scope are dropped, not just the ones you remember. This helps prevent similar issues and keeps the future’s state smaller. So yeah, understanding these yada yada stuff is important. The custom macros seems problematic</a></h3> Custom derives generates implementations for traits like Serialize</code> at the location of the struct definition. This ties the core domain models to serialization concerns. If I want the core to be independent of external dependencies (e.g., serde), adding such derives will violate the principle of modularity. And, Rust’s orphan rules prevent trait implementations in separate modules or crates unless either the trait or the type is local, further complicating separation of concerns. A quite normal pattern I’ve seen done by many crates is to make a feature for serde</code> and then only derive the implementation if the feature is enabled. This means that all crates which need serde</code> can enable it, but all crates which don’t can just leave the feature off. Personally I think that serialization code is in general tightly coupled to the actual underlying data, hence it makes a lot of sense that it’s implemented directly next to the data, if situation allows it. Dependencies that set the vocabulary for other crates should be allowed into the core. Maybe there needs to be a Rust-adjusted sort of a Hexagonal Architecture, where the core knows that there is some logging, some database and some user interfaces, but does not choose specific technology? Like depending on log</code>, serde_derive</code> and some UI abstraction crate implementable by clap</code>; but not on env_logger</code>, serde_json</code> or clap</code>. The core has to be built around the business model, which could evolve at leisure, be sliced and diced, or fused, and was entirely unaware of the messages/protocols involved in the outer I/O layers. A distributed application requires the ability to support multiple versions of the messages that are exchanged, and sometimes to radically switch the protocol to support new functionality, while supporting previous versions so that clients can migrate incrementally. When I serialize my core business model, I lose encapsulation (struct represents a coherent piece of the domain logic, and all its internal details are private or controlled by its API (e.g., methods) and the other parts of my application shouldn’t directly manipulate this structure or depend on its exact representation), which seriously affects my ability to evolve communications. Serialization exposes the internal representation of your core business model because it directly maps fields into a specific format (e.g., JSON, Protobuf), tying them together. The serialized format (e.g., JSON keys) is a 1-to-1 mapping of the fields in the struct (id, name, email). If you change the field names or types, you break compatibility with anything consuming this serialized data. In distributed systems, serialized data often persists in storage or is sent between systems. Changing the format can be costly or impossible without breaking clients. Using Data Transfer Object, we can add a layer of indirection. A shallow mapping between the core and the serialized representation protects your system’s modularity and flexibility. Though this is expensive, but might be good for long-term projects. Well, depending upon the complexity, this can quickly get tedious, confusing and also error prone as the project grows. So, I don’t know how to go about this :( Rust’s zero-cost abstractions enable the creation of lightweight wrapper types (e.g. Http) or thin adapters without runtime overhead. So we can have a local impl in our crate for the generic as well, and serialize that. But I don’t do anything from the get go..</a></h3> Yup, still when I write code, I want the architecture to grow organically so that I don’t end up with Clean Code and Bad Performance</a>. I don’t paste solutions to problems. It’s stupid, but guess what I am not even a Junior Engineer right now, so I am somewhat allowed to be stupid. I avoid Speculative Generality</a>. My general workflow is, get this running -> make it better -> refactor -> repeat. If any clean/hexagonal architecture is required, only then go ahead, otherwise I only care about solving the problem at hand, not the code mumbo-jumbo. I am just thinking about these design decisions these days, and will try incorporating this in my next personal project to get a feel of these things! Oh! and I feel very happy when I see ARCHITECTURE.md</code> and CONTRIBUTION.md</code> in a codebase. Just stupid enough to code, and wise enough to avoid mistakes</a> (• ͡° ͜ʖ ͡°•) Why did you write this? Gone nuts..</a></h3> I am having imposter syndrome for good. God has blessed me, as I got the opportunity to work and learn from really senior engineers (like Staff engineers and CTOs were directly mentoring me, they were GOATed). I hope 2025 will treat me better :) I went ahead and wrote about Rust a lot more than whether I should be a SWE or not. The thing is, I have improved a lot since 2024, but I still ask myself, am I good enough for anyone to hire me to build something cool? Am I that good yet? I guess I should just keep my head down and keep working hard. No point overthinking stuff. I don’t know why you are reading this, but thanks චᆽච The frenzy of building and observing seniors! Unknown — Tue, 26 Nov 2024 00:00:00 +0000 I have started to experiment with a tons of pretty projects whenever I get some free time, I just love building some stuff. I recently tried making small local projects from build-your-own-x</a>. At first it doesn’t make much sense why somebody is trying to minimally re-implementing every other cool piece of technology he sees, but it just enlightening (feels cool basically) to me (although my frenzy lost its heat after some time, just wrote too much code, burned out). Why I am doing it?</a></h2> After having deep enough technical knowledge on how systems are working in it’s core, you will be able to troubleshoot effectively find solutions to the problems due to the breadth of code that you have exposed yourself to. Also it will make debugging the software more fun and less daunting as now you know how everything is working from the very roots. I guess I am god’s chosen IC :) Understanding the outsourced technologies used by your product is the key to optimizing them for your specific business needs as well as making right decisions between many options. I don’t mean to say that you should know everything deeply, but whatever you are working on in your day job, it just becomes a lot more beautiful and elegant when you take some time out (ideally your employer gives you some help (time and resources) when you want to improve as an employee) and get a solid birds-eye view on the technology empowering the business! At least that’s what I feel. I am a student, and this is no advice, just feelings, and I can feel whatever I want :-) Some instances that come to my mind</a></h3> A lot of bitter sweet memories come to my mind when I think about this ;) Some from my own work experience, some from just reading, some from being there when some staff guy takes interview for hiring other senior folks! When I was involved with Jenkins in Google Summer of Code (I am maintaining stuff at Jenkins as well these days), I encountered a very peculiar problem related to Docker internals involving OSXFX because I was on Mac</a> and Linux System Integrity Protection which was daunting as hell. I discussed with the maintainers in the community and after long investigation, they were able to help me out. While working on the GSoC project I had to debug the jenkins plugins to the point that I was tinkering with the stapled object over HTTP exceptions (static resources sent with gzip</a>), and old JSP thingy that I hope I will never encounter in my life again! There were multiple instances when I had to step up to fix the libraries consumed by the plugin which were a real pain (those super busy maintainers just don’t respond quickly and the internship was for 3 months only, typical software engineering deadline prediction issues). These incidence helped me appreciate the importance of low level systems knowledge, I didn’t even knew what difference using different file systems in Docker could cause in performance and why (if you think, oh! its just a stupid file system, this</a> is a perfect example on why you should care about your silly file system being used to optimize performance). Like, UUIDs are commonly used, but alternatives exist as well!</a>. Thinking deeply before jumping is what I learnt! Like plan properly. Now when I look back on the patches and fixes feel less scary as I worked quite a lot on my Software Engineering fundamentals. Weird though, they don’t teach these things in degree, CS degree seems to be pretty weird for me, seems like they are preparing you to do a Masters, then a PHD and then do research/teaching, the degree is too theoretical for Software Engineering</a> roles. People can take Software Engineering degrees as well, they are better bang for the buck I feel! But nevertheless, a CS degree is really good if you wanna go the research route. Fuck it, any STEM degree is cool, I love STEM. Hilariously, I met a MIT Chemistry sophomore who wrote better Rust code than most CS fresh grads! Its quite a joke, CS seems to be the easiest STEM degree. Real chads study EE and Aerospace Engineering and ship rockets, not software. Most software is mental masturbation anyways. ◔̯◔ Interviewing Nuggets</a></h2> Its fun listening to senior/staff engineers on their interviewing experience. More fun when you get to see senior folks interviewing each other. I used to watch recordings of interviews during my intership when senior engineers were hired. Some interesting insights I got from there. I need to make sure I know the basics!</a></h4> I realized that Staff+ folks generally go very deep into basic things that people generally overlook. Even when Staff guys solve Leetcode questions, they propose multiple approaches and solutions, they discuss tradeoffs, not solutions. Ofcourse that question can be solved by Dynamic programming, but we are strictly focused on optimizing time only, so use Greedy. As everybody seems to be a Leetcode monkey these days (I have seen people solving more than 500+ LC questions, which is absolutely insane), interviewers sometimes ask unoptimized solutions, focusing on improving a specific thing. But personally, I don’t know why senior folks are asked to Leetcode, work on weekdays and Leetcode on weekends for job security sounds pretty bad to me :( Even when they are asking about HTTP and some REST</a> stuff, it can go deep (don’t underestimate basics)! Content on REST is actually confusing, but this</a> talk beats the hammer on the nail nicely. HTTP Caching</a> and optimizations are often missed by candidates! The CAP theorem is foundational to Distributed Systems, when discussing about it in an interview, PACELC coming into the picture is taken as positive point by interviewers. CAP theorem’s oversimplification has already been critiqued</a>. Also to note here is that consistency in CAP is different than consistency in ACID transactions. C</code> in ACID relates to a state of a database in general and C</code> in CAP (or distributed systems in general) is about a single data item in a database. Physics is the real bottleneck here :) I saw a lot of folks getting tripped here. Like, a Master’s student whose thesis is on Distributed Systems overlooks such stuff was surprising to me. Candidates are not giving wrong answers, but half-answers, and that’s frustrating. Just punching in, there is a CAP theory like concept in Observability as well</a>! You might think I am joking, but some people who say they do async programming, don’t even know whats async in async programming!? Please don’t be this person. Async/await is not about creating multiple threads. If that’s async, what the hell is multi-threading? Instead, it’s about asynchronous programming, where tasks can be paused and resumed without blocking the entire thread. Instead of running tasks in parallel (like threads or processes do), async frameworks use non-blocking operations and an event loop to manage tasks efficiently on a single thread. Most async systems rely on a single-threaded event loop for managing tasks. The async functions are represented as coroutines (lightweight functions that can pause and resume execution). You can also relate it to State Machines</a>! Coroutine (goroutines is just coroutines on steroids which are managed by go runtime</a> and not the OS, many goroutines are multiplexed onto a smaller number of OS threads, thus they are lighter) is not multithreading, just one thread inside user space (thus no context switching, hence lightweight)! Its not competing (thread racing), instead coordinating. Goroutine is not a thread!</a>. When you await an operation, the current task (tasks are broken into non-blocking units) gives up control (yields) to the event loop, allowing it to execute other tasks. Async is an abstraction, a non-blocking cooperative concurrency</a>, not necessarily threads or processes. Its useful to handle multiple I/O tasks concurrently without blocking the program. Multi-threading (threads and processes) is for CPU-bound tasks. CPU can run only one thread at a time. Multiple tasks are handled “as if” in parallel, but they are executed one at a time on a single thread. Async does not really have anything to do with threads. Also it helps disambiguate things to noticing that the async run time schedules “tasks” not “threads”. So as to distinguish what the async run time schedules from typical POSIX like threads. They are closely related but fundamentally separate. You can have async execution within a single thread, sometimes this is even more performant than using a thread pool. Async execution just means that the event loop can be passed off to other tasks. Learn about worker threads</a>! Threads are for working in parallel, async is for waiting in parallel. I found these</a> of concurrency patter really really cool. A very solid benchmark comparison between Go and Rust for concurrency</a> for understanding stuff practically! Go was meant to simplify concurrent programming, but when I was doing FOSS work for Kubernetes and other CNCF projects, I saw lots and lots of real world concurrency bugs</a>, which mostly arise from programmers not understanding these primitives. The typical misunderstood ones are : The cooperative non-deterministic goroutine scheduler</li> Context and Timers creating channels under the hood</li> Write mutexes having higher priority than Read/Write Mutexes</li> Non deterministic and still blocking</li> </ul> If you are on a single-core machine, using more threads is bad as they wont be running in parallel (just context switching giving illusion of parallelism). Single thread with concurrency will be better. More threads dosent magically mean more speed. Single-threaded code is often faster for pure CPU-bound tasks. But in case of I/O bound tasks like web scraping 100s of websites, the CPU will be idle due to network latency - in this case multiple threads are a viable option as it’s better to have many threads ready to take over while others are waiting for network responses - so context switching is no longer pure overhead. Async will be even better. OS threads use Preemptive scheduling - the kernel can interrupt a running thread at any time to switch to another thread, no need for thread to “agree” to be paused, which is good for CPU bound tasks where we dont trust the tasks to behave properly. Coroutines and Green threads are based on cooperative scheduling - ask must manually yield and scheduler cannot forcefully pause the task. No context switch happens unless the task explicitly says. Thus very good for I/O bound tasks. But if you forget to await (e.g., a big CPU loop without await), you can block the entire event loop! These are hard topics to understand + carry a lot of misunderstanding around them, thus become a major source of bugs. Dont get me wrong, goroutines are great, but the way they are designed, a lot of resource leaks happen if you dont use them properly. In my opinion, Rust is better. Yes you pay the price for safe coding, you become slower, but Rust is dependable. And I like dependable and maintainable software. Standard maps are not thread-safe by default in both, in Go if multiple goroutines try to read and write—or write and write—to the same map concurrently without explicit synchronization, it triggers a data race. Unlike some languages where a race results in silent data corruption, Go’s runtime includes a built-in race detector for maps. If a concurrent write is detected alongside other reads/writes, the Go runtime will actively crash your program with a fatal error (concurrent map writes). You must either wrap a standard map with explicit synchronization primitives like a sync.Mutex / sync.RWMutex, use Go’s concurrent-specific sync.Map (optimized for specific read-heavy/append-only patterns), or use channel-based actor patterns. Rust’s standard HashMap is also not thread-safe, but Rust prevents you from making this mistake at compile time rather than letting it crash at runtime, which is something I like. Rust enforces thread safety using the Send and Sync marker traits. A standard HashMap<K, V></code> does not implement Sync by default. This means the compiler will throw a hard error if you attempt to share a raw reference (&HashMap) across multiple threads simultaneously. So, Go lets you do it anyway and crashes at runtime if you mess up.</li> Rust catches it during compilation and refuses to build the code.</li> </ul> I psychologically feel safe with Rust due to its ownership and borrowing : There is only, ever, one owner of data</li> Ownership can be transferred — a move</li> Owned values can be borrowed temporarily</li> Borrowing prevents moves from occurring</li> </ul> Rust async programming is inclined towards multi-threading by default</a> causing complexity completely unrelated to the task of writing async code.</a> Just another rust vs go bashing, by the infamous Zig creator</a> for fun ಠ⌣ಠ Event Loop design and how they make web servers efficient is also a fun question often asked in backend interviews! Why Ngnix or NodeJS server better than Apache web server? In each Ngnix worker process after being forked by the master process, has an event loop. This</a> has decent explanation and this</a> paired with this</a> is gold. Event Loops are used in Redis as well, so its a good concept to know about, and it was the solution for the 1990s “c10k” problem! Most NodeJS apps are slow because devs dont understand how Promises work! There is nothing wrong with the runtime. Using the right native Promise functions help</a>. A sidenote, every engineer is habituated to look at a lots of logs (though AI is better at it)! Yet most senior engineers (these days they are expected to be DBAs as well) who know DBs aren’t able to talk about write-ahead logs, commit logs and transaction logs which is crucial in DB design. This</a> I think is one of the best writeups to learn about the abstraction called LOG! A very simple statement like optimized table</code> can create a mess in the interview if you don’t actually know how your table got optimized in MySQL. Suppose you delete lots of records from a table, and you notice that the space was not reclaimed by looking at the table status! So you use OPTIMIZE TABLE</a> command to get the space back. You should know that you only marked the records as deleted, and thus, the space is still left to be reused by other insertion. This is done to avoid unnecessary fragmentation and performance degradation. The OPTIMIZE TABLE</code> command simply temporarily creates a new table and copies over the data excluding the free space, and then swap with the original table. Thus, it locks the entire table, so the command shouldn’t be used in production during operating hours. And you know, try to avoid SELECT </code> especially in columnar DBs, its bad for performance. ORMs are for smart noobs :) Very foundational questions can trip you up in an interview. Suppose you work in a team who handled large scale payment gateway. In such case, the payment went through the gateway but failed on client’s side? What if double invoice gets generated? How will you sync stuff? You will have to roll back the committed transaction either in the client’s side or gateway or bank. How will you make sure that the transactions are committed successfully? Remember, the cute SAGA pattern? Suppose, the payment is debited, but not credited? Inconsistent reads are also an issue! Simple solution is using a read-write lock or shared lock (exclusively reading or writing), but it has resource contention. MVCC</a> is standard here, in which reading do not block writing and writing do not block reading and thus provide high throughput out of your database. Locks must be granular, as they make the program synchronous. And yeah, these terminologies are confusing, dirty read, phantom read and what not! Weird enough, a lot of candidates didn’t know that there is multi-level cache in the CPU, so we need to do more stuff with CPU than memory</a>! In multi-threaded environment, thread safety is ensured using Locks or Synchronized keyword (semaphores and mutexes under the hood). When working with Distributed Systems, Cache coherence is something you should know about when interviewing for senior roles! Having a basic understanding of caching techniques</a> helps a lot! When we want to achieve strong consistency, the distributed lock itself is a performance overhead. Cache versioning (MVCC</a>) does not guarantee 100 percentage consistency as well but its better for performance. How often do you want to update which can lead to conflict between two threads executing their transactions? - If its frequent, go Pessimistic locking which will lock the critical section the threads want to execute, as it assumes that there will exist conflicts. If rare, Optimistic locking is better due to less resource contention as no explicit resource locking is required. Getting high throughput is extremely hard! Optimistic locking allows more throughput compared to Pessimistic locking, and is just better for performance. And Caching is HARD!</a> And ports are not in hardware, they are managed by kernel and are in L4 of OSI! I saw this mistake when interns were being interviewed. Sounds silly until they ask you very damn basic things in interview. You know what, lets stoop soo low, whats the difference between IO vs CPU intensive? I/O-bound tasks spend most of their time waiting for external operations to complete—like reading from disk, database queries, HTTP requests, etc. The CPU is mostly idle during this waiting period. Whereas, CPU-bound tasks require continuous CPU processing, such as data compression, cryptographic operations, large computations, etc. They don’t wait on external resources but keep the CPU busy. And what will you do when you want to handle both of them together? You shouldn’t perform CPU intense work on tokio blocking threads i.e. worker_threads() in thread pool</a>. Tokio assumes work on those threads will be I/O bound, and so will spawn a lot of them on the assumption that individually they mostly spend their time blocking (you shouldn’t run CPU-bound tasks here because they’ll block the event loop, starving other async tasks). Place all the potentially blocking tasks into separate threads (with tokio’s spawn_blocking</code> for blocking I/O tasks controlled via max_blocking_threads()</code>) and use the async threadpool for the rest. Do not use spawn_blocking</code> for your CPU heavy tasks. Rayon is optimized for parallel CPU-bound computations. Just offload CPU-heavy tasks to Rayon (or some custom thread pool), keeping both Tokio’s worker threads and blocking threads free. Another simple thing, if you are asked, what’s the best way to store key value pairs in memory? And hashmap is your only answer and you stop after that, you are up for a wild ride my friend! First questions should be, whats the best for you? Hashmaps are not thread-safe by default and has unordered storage with hashing overhead. Yeah its O(1) I know, but what if that’s not our priority? BTreeMap is when you want ordered keys to deal with range queries and is okay with O(logn) inserts/lookups. Hashmap can blow up based on their implementation but an ordered map (BTreeMap) implementation which uses BTree</a> (even Red-Black Trees are used for this, called TreeMap, for for large in-memory datasets, BTree are more performant</a> and cache-friendly) won’t but it has logarithmic complexity in worst case which can be better in cases there are too many collisions or reshashing taking place, which will lead to O(n) worst (a recent research paper</a> has improved it to (logn)^2 and it was done by a undergrad) , whereas BTreeMap will always have O(logn). So if worst case performance matters, don’t choose HashMap. And be prepared to explain stuff like rehashing, load factor and hash functions. If you say HashMap And, use DashMap instead of Arc<RwLock<HashMap<K, V>>></code> for concurrent access. Unlike Rust’s standard HashMap<K, V></code>, which requires wrapping in a Arc<RwLock<HashMap<K, V>></code> (causing contention on the entire map), DashMap internally splits data into multiple independent shards, each shard is a smaller, independently locked HashMap<K, V></code>. Instead of one global lock, DashMap only locks the critical shard on write operations, thus allowing multiple threads to modify different shards simultaneously, increasing performance. This sharded design significantly reduces contention (only a single shared is locked when writing) compared to a global Mutex<HashMap<K, V>></code>. The reads are lock free, multiple threads can read different keys in parallel without locking. Metaprogramming is a very important paradigm used in modern languages like Rust and Nim (more powerful and flexible than Rust’s macros in some ways because they operate on full AST manipulation at compile time compared to token based AST manipulation in Rust) . The ability to have compile time code execution (unlike functions, which operate at runtime) is powerful to say the least, while DRYing up the code. Its the ultimate Boilterplate killer. Like we can have literal HTML with syntax highlighting in Rust via procedural macros</a>, i.e DSL made fun. And yet, for some weird reason, so many programmers either don’t know about it, or don’t use it in its full glory! Why? The only caveat I can think of is that it makes debugging a bit harder. Like, if you want to implement similar methods for different structs, use macros. To avoid expensive runtime time computation, use macros. But Rust macros aren’t just for DRYing!</a> Last one, in a Request-Response architecture (chain dependent, can’t move ahead without response, unlike Event-driven, where a producer fires an event, and consumers react to it asynchronously without expecting a response) when using gRPC calls</a>, how is req-res paired? We don’t use any nonce to pair them up manually? It happens via HTTP/2 multiplexing</a> which uses unique stream IDs. Forward Proxy is used to protect the clients accessing the websites via the internet. It helps with Content filtering for the client, caching frequently requested resources, hiding client IP address, client request modification. Reverse Proxy is used to guard the web servers from the clients trying to access websites via the internet. It helps with server protecting from DDoS attack, hiding original IP address, load balancing, SSL management, content management. Mentioning this stupid thing as I myself tripped in an interview when asked about Reverse proxy setup in Jenkins. I didn’t knew why I setup reverse proxy for Jenkins plugins at the first place, it was part of project so I did it without asking my mentor what purpose it served :( Oh, btw check out Nim. It’s for when you want to write low-level systems code but you don’t hate yourself and aren’t a masochist. Its a very powerful language, and can produce extremely small binaries. Zig could be the modern C, and Rust the modern C++. Nim doesn’t seem to have a purpose, its feel like a weirdly good general for everything sort of a language, which has a marketing problem as nobody uses it for some reason!</a> A very cool blog to understand logical reasoning when troubleshooting network issues!</a>. I am bad at this! If I see MongoDB in someone resume then I would expect them to atleast know about writing complex queries and aggregation, have an understanding of how indexing work internally in MongoDB and efficient way to design schema, ESR rule, sharding in MongoDB and how to implement it. I need to practice writing consensus algorithm, god knows when some interviewer asks me to do so in a 40 min interview (ー_ー；) Or worse, one senior guy I know told me that he was interviewing for some HFT Rust role, and they literally asked him to implement a lock-free queue</a> from scratch in a 40min interview!!!!! And it’s so embarrassing when someone asks you the difference between scripting language and programming language</a>, and you fumble badly ( >︹<) And yeah, if somebody is thinking, why you need to know about these CS stuff, I am just going to be a frontend engineer, knowledge of algorithms are still crucial!</a> After reading all these interview nuggest, you must be having a PTSD (@д@) but whatever I wrote, I have watched first-hand in interview recordings. So maybe we are destined to be a goose-farmer if you are dumb like me ◔_◔ Btw, even I fucked up in an interview once</a></h3> I am no god, so let me explain you a question which I messed up during a very casual interview. I was being interviewed by a Principle Engineer, and I thought, because the guy is very senior he will ask me some mind-numbing question. I couldn’t be more wrong. I was asked a very very simple question, but he made me shit my pants with his followup! set(key, value): Sets a specific key to a given value. setAll(value): Sets all keys to the same value. get(key): Returns the value of a specific key. The challenge was to optimize it so that `set_all` doesn’t take too long when you call get or set after it.</code></pre> The naive set_all</code> implementation is O(N). Ofcourse he wants me to do better so I optimized the set_all</code> like this : use std::collections::HashMap; struct OptimizedMap { map: HashMap<i32, (i32, i32)>, // key -> (value, set_version) global_value: Option<i32>, global_version: i32, // can be thought as a timestamp as well } impl OptimizedMap { pub fn new() -> Self { Self { map: HashMap::new(), global_value: None, global_version: 0, } } pub fn set(&mut self, key: i32, value: i32) { self.map.insert(key, (value, self.global_version)); } pub fn set_all(&mut self, value: i32) { self.global_version += 1; self.global_value = Some(value); } pub fn get(&self, key: i32) -> Option<i32> { if let Some(&(value, version)) = self.map.get(&key) { if version >= self.global_version { return Some(value); } else { return self.global_value; } } None } } fn main() { let mut map = OptimizedMap::new(); // version - 0 map.set(1, 10); map.set(2, 20); map.set_all(100); // version - 1 map.set(3, 30); // version - 1 println!("{:?}", map.get(1)); // ✅ Some(100) (from global_value) println!("{:?}", map.get(2)); // ✅ Some(100) (from global_value) println!("{:?}", map.get(3)); // ✅ Some(30) (explicitly set after set_all) println!("{:?}", map.get(4)); // ✅ None (never existed) }</code></pre> Now all three are O(1)! I was happy :) But he was even happier ⊂◉‿◉つ and asked me for a rollback implementation. So I did this : use std::collections::HashMap; struct OptimizedMap { map: HashMap<i32, (i32, i32)>, // key -> (value, set_version) global_history: Vec<(Option<i32>, i32)>, // (global_value, global_version) current_global_version: i32, } impl OptimizedMap { pub fn new() -> Self { Self { map: HashMap::new(), global_history: vec![(None, 0)], // Initial version current_global_version: 0, } } pub fn set(&mut self, key: i32, value: i32) { self.map.insert(key, (value, self.current_global_version)); } pub fn set_all(&mut self, value: i32) { self.current_global_version += 1; self.global_history.push((Some(value), self.current_global_version)); } pub fn get(&self, key: i32) -> Option<i32> { if let Some(&(value, version)) = self.map.get(&key) { if version >= self.current_global_version { return Some(value); } else { // Find the global value at the time the key was last updated return self.global_history .iter() .rev() .find(|&&(_, v)| v <= version) .and_then(|&(val, _)| val); } } None } pub fn rollback(&mut self, target_version: i32) -> bool { // Check if the target version exists in history if let Some(&(global_value, _)) = self.global_history .iter() .find(|&&(_, v)| v == target_version) { self.current_global_version = target_version; true } else { false } } }</code></pre> Now, I straight up thought about rollback, not whether it will be frequent or not. Don’t be like me, ask good questions to the interviewer. Rollback is O(N) here! Ofcourse he wanted better, and unfortunately I started thinking of some fancy algorithm to make the rollback faster. And when I couldn’t come up with some way, I was dead in the interview (-_-; ). Don’t be silent in any fucking interview. Speak something! The solution was simple, just use BTreeMap</code> in case of frequent rollbacks to specific versions. If rollbacks are rare, Vec</code> is just fine! But this thought just didn’t come in my mind during the interview. I passed somehow! It hard to find cracked folks, not bluffers</a></h3> This is a thing I saw when binging a lot of recordings. Stop bluffing and exaggerating. When working with engineers, its quite easy to call the bullshit! I have enjoyed working with people (in OSS and when mentoring some folks) who are cut-throat truthful about what they don’t know about! Like really, some junior engineers don’t know much (which is okay! they should not! even though I feel the tech industry borderline hates junior engineers atp) but atleast be honest, and build yourself up truthfully, and don’t brag to the moon instead work hard. Using EGO productively is under-rated! One weird thing these days in that, senior engineers are expected to have working knowledge of the whole IT department of the company :-( which is undue pressure on one individual! Avoid solving imaginary scaling problems!</a>, most problems in programming are already solved, don’t make unnecessary fuss! Often, you learn about the details of what you are using when you have a deep dive out of curiosity (re-implementing some stuff in any other language is often fruitful to learn deeply about that language as well). So, when having less workload, its wonderful to let your curiosity take you into wild directions. If you write a piece of code using AI, and you dont know how it works, my pal, you are doomed!</a> Should I be an IC or a manager when I grow up?</a></h2> Don’t forget to share what you learn though! Social skills are necessary for Geeks as well!</a>. People should know why are you suddenly refactoring all that somehow working code and deleting thousands of lines</a> :) But in technical leadership and working with people</a> (I haven’t really had a decent long chat with techno-managerial folks till now), the coding will take a back seat if you are not curious enough about whats actually happening in your team! There is no Genius Programmer</a> but there sure are egoist programmers! But nevertheless, there are joys of engineering leadership</a> as well as writing some clean code this</a> and that</a> if you continue to be an IC. Coolest thing I can imagine doing as a manager is looking at the code from a business point of view, metrics would be so important for me</a>! The idea of Centralized vision and its Decentralized execution is very cool to me! But managers have a lot of meetings, and I don’t like meetings! Where is my Girl gang …</a></h3> One sad thing that I noticed is, there are less women engineers in cutting-edge startups for some reason. I have interacted with ample of girls who went to big tech, but none who went to startups. Sad life :( Why are startups so male-dominated, weird? I guess big tech push on diversity hiring</a> is what’s driving it maybe. Man I love girlies 💅 I guess I have too much free time</a></h3> I have just ranted my thoughts here! I have an exam after some time, and I guess I am up for some bad grades :) These are just my personal experiences, observations and learning from others mistakes, not some wisdom, just some things that came to my mind at some random days Another reason writing this up is revision of crucial CS concepts when interviews come up! Working at a early-stage startup! Unknown — Fri, 25 Oct 2024 00:00:00 +0000 So, I got hired at a startup at the end of my 2nd year at IITK, and man it was an intense experience working at a high growth startup! I was working, managing my academic workload and later started doing some cryptography research project under a cool prof as well, so hectic as fuck. But I loved every single moment of it. I learned about the experience of working in a startup, and some nuggets on what it takes to actually run a high-growth startup. How was it different?</a></h2> I had some internships under by belt which were also pretty solid experience technically, but not organizationally. In internship, I used to get a mentor assigned to me, I had to simply propose a timeline of how I plan to finish up the internship work and the mentor would guide me while I do the work, great if you want to improve technically. But the startup experience is totally different. I was not an intern anymore (umm… I actually was, but my CTO just trusted me to assign a project to me completely). People were actually dependent on my work. I found out that I am not that good at timeline estimation. And a lot of things don’t go the way you thought initially, especially when you work in a team. When issues happen upstream (you are dependent on someone), shit hits the fan! People don’t reply fast, maintainers of OSS are really busy people, talking to Product Managers to reach out to devs responsible for the issue to the tech lead, getting the issue resolved, giving feedback to the managers, and then returning back to your own work. Unblocking other team memebers time to time. The hardest part is not technical, but communication and management. Nobody is there to clean up your mess - technical debts, people who left the role, sudden breaking change in god knows what lib that you were using, … Reducing your bus factor</a> is very crucial. I was not working on a personal project! I can’t add any new dependencies, use a new language like Rust :) and implement new libraries that wrap something, and show that to the team expecting appreciation. Bad idea, always discuss first with the team, and then do stuff. Don’t try to be smart, and end up being a smart-ass. Technical debt is not bad, but neglected technical debt is!</a> Talk is cheap! Show us the code</code> is a really bad idea! Maintaining the codebase is a large part of the job. Also, the amount of time you will spend in meetings in just mindnumbing. But attending meetings is not a waste of time at all. Sharing the right information and mentoring juniors is absolutely essential. A culture of sharing information avoids the situation of a star dev in the team. And supporting your manager is crucial to manage you is crucial. Senior engineers are often judged by the performance of the team they are leading, apart from their own performance. You will have to actively stay in loop on where the team is stuck. As a student I used to think it’s hard to get hired. Well, it’s way harder to find someone competent and accountable enough to show up everyday with grit and determination. But I guess, I just love the thrill of working in a startup, creating products that shoot to production fast! I am still waiting for the day when I delete my first production server as a cute intern :) The feedback loop and learning speed at a high-growth startup is just hypnotic (work life balance goes for a toss). Well, you will have bad WLB even in big tech these days, so not much of a loss. But I still wanna try Big Tech and Quants just out of FOMO once. Wait before you sign that offer big boy</a></h2> The easiest kind of offer you can aim for is an intern-to-full-time conversion offer. The intern interview process can be atleast twice easier than a full-time interview process. But observe the company very keenly before accepting that offer. A solid internship gives you levrage to get even better full-time roles if you are ready to take a risk and work hard. You don’t always have to accept the conversion offer. Although I am just a student and have no corporate 10+ years of leadership experience, still some points that I have started to consider before accepting offer to join any company : Performing good in interviews don’t have much to do with performing good in job. Those behavioural rounds are useless, repetitive crap. I have seen many companies try to sell you on their culture or ‘core values.’ But honestly, you can forget about most of that after the interview Can people in your team get shit done? Find out who gets most shit done, and learn from them as an intern especially. These technical interviews are necessary-evil.</li> Being fast in software is not the best idea. If your task is estimated to take two days and you finish in two hours, no one really cares or remembers your name. but, if a bug comes from that code, everyone will remember you. So being fast isn’t always valuable.</li> Avoid teams who have heroic coders, they just don’t have process setup properly. And that cute hero is borderline mentally harassed. Also, beware of information hoarders (happening quite a lot in big tech due to layoffs), there are manipulative people everywhere, stay sharp.</li> Work with smart folks, not smart-asses. You really underestimate humility before you meet egoists. I don’t join a team where everybody is a genius. I like working with normal human beings, not with 100x engineers. Work with folks who humbly handle the tech-debt and do boring but necessary stuff in the company, not drive impact to the moon while throwing other engineers under the bus. These smart-asses rock the interview, but are a nightmare to work with. I’d rather work with someone who isn’t a genius but is willing to collaborate and share ideas. Stay away from over-competing folks, spend more time with your girlfriend or wife, pursue hobbies, life you precious life please.</li> High turnover rate is redflag, and the company clearly isn’t able to appreciate the talent.</li> If you are interviewing for an intern position, during the interviews, ask about the kind of tasks you’ll do in the role, the manager you’ll report to, and the kind of mentorship you’ll get. If you are an intern, you shouldn’t be simply thrown into the project (ideally, but a lot of times if its a startup, be ready for some serious action). Fight for the team and mentorship, they are gonna define your internship as well as full-time conversion.</li> Don’t shy away from negotiation even for internship. Even if you don’t work for money, you have every right to be paid your worth.</li> Always have other offers. I’ve seen two friends with similar experience joining the same company for the same role, but one got $50k more a year because he had a competitive offer.</li> Focus only on the take-home money. No stocks, bonuses, hikes, yada yada. They can be gamed against you.</li> Read Glassdoor/Blind reviews to get a sense of what you’re getting into. Heck, DM employees on LinkedIn to get a feel of how the company is from inside.</li> If the only thing that your job is paying you is money, better find another job.</li> </ul> Now, I wish to work on something worthwhile, not just print money. This comes with a sense of entitlement that money is not hard for me to make. Great engineers are sought after a lot… Also, discuss everything over emails, not on telegram, discord, slack, etc. And negotiate, always! Over-confidence is better than under-confidence (not humility) most of the times. I dropped the thought of building company being in college</a></h2> Don’t trying building a company when in university! Either drop-out or take a year gap to iterate. Otherwise you will have a lot on your plate. Always work in a startup before starting one. Organizing an engineering team is really hard, and you will have to give them an optimal employee experience as well. They are an employee, you are a founder. So you cant expect them to work like a founder. Building accountability systems should be a priority. Raising VC money is extremely hard as a student. An accelerator program is not real progress (even if its YC). Do you really wanna do this, are you obsessed enough? I was not. Period. Maybe some other day, some other time. What I learnt about myself</a></h2> Atleast for the time being, research will be on my cards. I really underestimated managing people. I am the god’s chosen IC Wealth Unknown — Wed, 04 Sep 2024 00:00:00 +0000 Its hard to earn money, and harder to keep it with you. Books : The Little Book of Common Sense Investing</li> Common Stocks & Uncommon Profits</li> Big Debt Crisis</li> The Mastery of Capital</li> Lets talk Money</li> </ul> Starting to pick up Nix a bit! Unknown — Wed, 31 Jul 2024 00:00:00 +0000 Well, most modern programming languages come with package managers that help manage dependencies, and among other things they do their best to ensure that they aren’t altered without the user knowing. There are always more sources of variability than you can count. If you are dealing with systems software, the surface area increases even more as now you have to worry about native libs, compilers, OS, might be kernel as well! More often that not, figuring out what the previous state of the system was is a herculean task. Only if you are unlucky that the burden of hunting for differences can land upon you like a divine punishment. One of the core ideas behind modern dependency managers lockfiles, view of the dependencies that were locked in at some point. Among other data, this includes the package location, its precise version, and a checksum to ensure it’s not tampered with or corrupt. I wanted similar level of control on a larger scale. Thats when I stumbled upon Nix</a>. Into the rabbithole</a></h2> While investigating how repl.it used Nix to secure the access to their running containers</a>, I went down the Nix rabbit hole for a week and became a bit more enlightened</a>. Also, Nixpacks</a> piqued my interest due to its capability of being able to generate OCI complaint container images from any Nix supported dependencies directly (though its not perfect</a> and I should contribute to it maybe). Folks using Fly.io to deploy literally having love stories</a> with Nix :) Interestingly I also remembered that Saksham (one of my awesome seniors at IITK) also wrote about it here</a> which made me double down on Nix. The positive impact of Nix driven workflow</a> is clearly visible by the amount Nix community has grown! This writeup is what I hoped I had before challenging myself with this steep Nix learning curve. This is not to learn about using Nix in your workflows (though I have linked some great blogs from some super smart folks), but to serve as a collection of useful resources for anyone starting out, a headstarter! Here I am focused on Nix as a package manager (nixpkgs as a central repository of nix packages), an immutable graph database nix-store -q --tree result</code> (cool diagrams here</a>) and a build system thingy. When nix builds something, it’s stored in /nix/store, which is essentially an immutable graph database, with a hash that encodes this full dependency graph. In developer workflows, we can use mkShell</code> for using nix develop</code>. For a more wholesome explanation refer this blog</a>. The concept is simple, we just read a Nix expression and derive the build with specific build inputs. We can also fetch from the cache, the needed inputs and final derivation. The advantage of Nix is in getting your developement, CI and production in sync (Devbox in their internal dev workflow</a> suffered through this), also having the power of cross compilation, i.e consistent environment accross development, making it the single source of truth. Yeah! Things would just work… You get always-working environments with little to no duplicated effort. Basically avoid a green check mark on a PR only to realize that the production is broken. Whereas Docker (with K8s) is container-based deployment solution</a> which have become so popular, that they have set a high standard when it comes to organizing systems and automating deployment. today, in many environments, I have the feeling that it is no longer the question what kind of deployment solution is best for a particular system and organization, but rather, “how do we get it into containers and deploy it into microservices?”. Docker is partially fit for this (unless you aggressively play with versions) and this concept of Reproducible vs Repeatable builds is very well explained in this</a> talk. Docker and Nix</a></h2> Docker approached building Container Images (immutable self-contained root file systems containing all necessary files to run a program, such as binaries, libraries, configuration files etc) in multi-stage builds. As a best practice, ‘scratch’ image ( though I am casually calling it base image, its not, its totally empty, yes! not even shell is there. The way container images are constructed is that it makes use of the underlying Kernel providing only the tools and system calls that are present inside the kernel. Because in Linux everything is a file you can add any self-contained binary or an entire operating system as a file in this filesystem. This means that when creating an image from Scratch, technically refers to the Kernel of the host system and all the files on top of it are loaded. That’s why building from Scratch is no also a no-op operation and when adding just a single binary the size of the image is only the size of that binary plus a bit of overhead. The resources assigned when executing an image in a container is by leveraging the cgroups and the networking makes use of the linux network namespacing</a> technique ), starkly different from Distroless Image</a> which I often see used other than parent images when the code cannot be complied directly to a single runnable binary. Thus the container life-cycle is bound to the life-cycle of a root process, that runs in an isolated environment using the content of a Docker image as its root file system. Finally, to optimize/reduce storage overhead, Docker uses layers and a union filesystem (there are a variety of file system options for this) to combine these layers by “stacking” them on top of each other. A running container basically mounts an image’s read-only layers on top of each other, and keeps the final layer writable so that processes in the container can create and modify files on the system. Whenever you construct an image from a Dockerfile, each modification operation generates a new layer. Each layer is immutable (it will never change after it has been created) and is uniquely identifiable with a hash code, similar to Nix store paths. But if you can create Container Images with Nix dockerTools</code>, boasted by Nix on their homepage</a> with doc</a> as well as Dockerfiles, then why mess your brain up with Nix and simply not use Dockerfile? Nix v/s Docker to generate OCI complaint docker images is surely a heated question catered well by this</a>. Its universally known that smaller images with less bloat lead to a smaller attack surface and probably increased security and faster deployments. So lets compare the three options we have to get our PERFECT container image looking for build speed from caching, maintainability, security: Using traditional Multistage builds Many DockerCon talks like this</a> and this</a> covered the best practices while writing Dockerfiles, and as you can see, writing good Dockerfiles becomes increasingly difficult as you try to incorporate best practices. This</a> blog highlights the size comparison using various dockerfile techniques. You’ll see that the base image contains many files we might not need or want, things that potentially could be a security problem. It comes bundled with tools such as sh, wget, and the apk. The nix image will have no such tools - only nginx and its dependencies.</li> Using Nix dockerTools Although this method to create container image has been there for years, recent caching optimization</a> has made it a very appealing choice. The functional approach rewards with better abstraction. The Hydra CI server obviates the need for paying for (or administering a self hosted) docker registry, and avoids the imperative push and pull model. Because a docker image is just another Nix package, you get distributed building, caching and signing for free. Also, as Nix caches intermediate packages builds, building a Docker image via Nix will likely be faster than letting Docker do it. This</a> sums up the sentiment of this blog really well!</li> Using Nix as base image in multistage builds Mixing both the worlds using NixOS</a> as base image to write Dockerfiles is generally a comfortable approach. Each build will run as an unprivileged user, that do not have any write access to any directory but its own build directory and the designated output Nix store paths.The network namespace helps the Nix builder to prevent a build process from accessing the network. In Nix, only builds that are so-called fixed output derivations (whose output hashes need to be known in advance) are allowed to download files from remote locations, because their output results can be verified.</li> </ol> Why you should consider using Nix in your build systems? Well you can even achieve what Vercel calls Remote Caching</a>. Nix Flakes</a></h3> I generally think of flakes in a docker container way (flakes run native on mac btw, Docker runs over VM)! Its said that flakes are processors of Nix code</code>. This blog</a> series is very informative to start working up the way! Sharing layers between flakes works better than Docker. Flakes can basically depend on other flakes! Unlike docker, there are no cgroups or namespaces or VMs or anything with nix! An excellent resource that I found for learning Nix was Ian Hnery’s How to learn Nix</a> diary! If you write a lot of Nix, you might also need to unit test</a> it. Its also an interesting read on how Supabase has now started to use Nix</a> being a well matured startup with a huge ecosystem around it. Note that you need to enable ‘flakes’ and ‘commands’ by experimental-features = nix-command flakes</code> in ~/.config/nix/nix.conf</code> to use nix build</code>. This blog</a> shows how to start! Nix flakes with Rust :)</a></h3> Nix has a pretty bad reputation for its documentation, but Jade’s Nix guide to flakes</a> is an excellent writeup! A Flake for your Crate</a> gives a great overview on how you can start using Nix with Rust. And nix-ifying a rust project</a> discussion is wholesome. If you are also using WebAssembly in your Rust project, this blog</a> is quite helpful! There is a fantastic template</a> to use! Nix flakes with Go</a></h3> You can use gomodnix</a> and refer this</a> blog to get started! Nix flakes with OCaml</a></h3> I found opam-nix</a> with the corresponding blog</a> to be a good enough starting point. Nix has more binary packages than Homebrew does, so is generally faster as we don’t need to build much from the source, and some folks are literally using Nix as their Homebrew replacement</a>, and the only practical reasoning for it might be that its easier to setup a new machine with flake.nix</code> compared to homebrew, basically having a single flake to maintain all the packages. You can even run NixOS VM on Mac</a> now! Take it easy and practically</a></h3> To paint a simplified picture, the pieces that are useful to know are nixpkgs, and something like a Cargo.lock for Nix is flakes. By using flakes we can select a particular revision of the nixpkgs repository and every dependency we use will match what’s defined and built in it. When you use it well : You get rid of works in my machine! despite running different distributions or OS, devs will have a consistent developer environment that very rarely breaks</li> You get awesome incremental build support. Nix prevents redundant work through content-addressable storage and aggressive caching.</a> Because every dependency (from specific versions of crates to the system LLVM/glibc) is isolated into its own immutable Nix store path based on a cryptographic hash, modifying a single line of application code doesn’t force a re-download or recompilation of un-impacted third-party crates. Furthermore, teams can hook into distributed binary caches (like Cachix), if a coworker or a CI runner has already compiled a specific version of a crate or toolchain component, Nix simply downloads the pre-built binary instantly rather than burning CPU cycles compiling millions of lines of dependency code from scratch.</li> Because the package manager, compiler toolchains, and build system are tightly and declaratively integrated in Nix, tracking down regressions becomes significantly less painful than what you might be used to in traditional dev envs. As a real-life example, consider a scenario where a Rust-based project compiles cleanly under one version of the toolchain (e.g., Rust 1.75), but suddenly encounters a cryptic trait-solving error or unexpected LLVM code-generation panic when building with a newer nightly or stable release. In a traditional setup, attempting to bisect this kind of compiler or dependency regression requires manually installing multiple toolchains, altering global system paths, or wrestling with conflicting system libraries. With Nix and flakes, locking down or rolling back toolchain versions only takes a few lines of configuration. By writing a short shell script wrapper or leveraging Nix’s ability to pin specific historical commits of nixpkgs, you can automate git bisect across entire dependency trees. When paired with git bisect run, Nix handles the heavy lifting—recreating an isolated, reproducible build environment for every single step in the commit history without bleeding state from your host system. In no time at all, you can drill down past thousands of package updates or compiler changes to isolate the exact PR responsible for the regression.</a></li> Because Nix enforces strict security hardening flags globally across its package ecosystem by default, such as stack protectors, position independent executables (PIE), and FORTIFY_SOURCE checks via its standard wrapper, it frequently surfaces latent bugs or undefined behavior in projects (especially those wrapping native C/C++ libraries via cgo or FFI) that other package managers quietly bypass. Nix’s build environment (stdenv) goes out of its way to inject security hardening flags by default. It doesn’t just rely on upstream defaults; it actively wraps compilers (like GCC and Clang) to enforce these policies globally.</a>. This tendency of uncompromised safety can create friction in debugging though because these security flags are baked so deeply into Nix’s build wrappers, trying to do non-standard things, like building unoptimized code or working around strict buffer checks—requires precise configuration (hardeningDisable = [ “all” ] inside a Nix expression rather than a quick shell environment variable).</li> </ul> Nix on CI</a></h2> Since at the core of any CI, Nix is doing the same thing, there is no real vendor lock-in issue. This also means it should be possible to accurately benchmark these CIs</a>. Conclusion</a></h2> You will have to face a lot of errors when starting out with Nix, so be ready for being frustrated, like this</a> :-] You can do a lot of fancy stuff with Nix, but I just started out with Nix, like you! These are a lot of resources that I shared, and I hope you will be a Nix ninja in no time if you go through all of this, so ALL THE BEST :) Computer Science Unknown — Fri, 12 Jul 2024 00:00:00 +0000 Well, I used to hop around and read many CS books for passing my time in boring undergrad classes. It was fun, and I am not the best sort of a student. Here I am listing my favorite reads till now and, what and why would I pick them up for if I wanted to read again. They have helped me enjoy in the midst of boring lectures in IITK and I want to sincerely thank the authors for putting in the work. Also, I have shared the CS lectures I watched to grasp the concepts better so that people can have reference material and broader understanding of the interesting topics. The best way to learn these concepts is to implement some intriguing idea that you have while slowly and sensually reading these beauties! The resources are in no particular order, I just enjoyed them, and you might also love them! Intro to Programming</a></h3> Reads: Teach yourself programming in 10 years</a></li> Becoming a Better Programmer by Tightening Feedback Loops</a></li> A Programmer’s Introduction to Mathematics</a></li> Coding is Boring, Engineering isn’t</a></li> Why Machines Learn</a></li> Building Modern Computers from first principles</a></li> </ul> Lectures: How Computers Work</a></li> MIT Missing</a></li> Computational Structures</a></li> Richard Feynman Computer Science Lecture</a></li> </ul> Data Structures and Algorithms</a></h3> Reads: Introduction to Algorithms</a></li> The Algorithm Design Manual</a></li> Grokking Algorithms</a></li> </ul> Lectures: Introduction to Algorithms</a></li> Design and Analysis of Algorithms</a></li> </ul> Just go Neetcode</a> for those stupid Leetcode Medium/Hards that you are going to get in your interviews. Leetcode preparation is unnecessary if you don’t want to work in Big Tech giants, but deep knowledge of Data Structures and Algorithms is crucial for crafting clever software! If you are really into puzzles, Competitive Programming is a cool sport to be in! CP Handbook is excellent to get started</a> along with this</a> lecture series. Try ICPC :) Not required for simple DSA rounds at tech companies though! Its fun, really fun once you get the hang of it. Computer Systems</a></h3> Reads: Operating System Concepts</a></li> Linux Kernel Development</a></li> The Linux Programming Interface</a></li> Advanced Programming in UNIX Environment</a></li> What every programmer should know about Memeory!</a></li> An incremental approach to Compiler construction</a></li> What is Systems Programming</a></li> Intro to Concurrency</a> and Tree Structured Concurrency</a></li> Seven Concurrency Models in Seven Weeks</a></li> Computer Systems Security</a></li> </ul> Lectures: Programming Paradigmns</a></li> NWU CS615</a></li> NWU CS361</a></li> UCL Compilers</a></li> UMass CS377</a></li> Computer System Engineering</a></li> Concurrent Programming KAISTT CS341</a></li> Performance Engineering for Software Systems</a></li> </ul> Networking</a></h3> Reads: Computer Networks</a></li> TCP/IP Illustrated Vol 1</a></li> </ul> Lectures: NWU CS340</a></li> </ul> Database Engineering</a></h3> Reads: Seven Databases in Seven Weeks</a></li> Database Internals</a></li> Against SQL</a></li> Bitcask</a></li> Monarch</a></li> </ul> Lectures: NWU CS317</a></li> CMU CS625</a></li> CMU CS721</a></li> Seven Databases in Seven weeks</a></li> </ul> For real deep Database Enthusiasts CMU DB Group</a> is the GOLD STANDARD: Scalable Architecture and Distributed Systems</a></h3> Reads: Distributed Systems</a> and some cool texts from the TA Students guide to RAFT</a> and Instructors guide to RAFT</a> and A step by step guide to Raft Consensus Algorithm</a></li> Designing Distributed Architectures</a></li> Designing Data Intensive Applications</a></li> Building Microservices: Designing Fine-Grained Systems</a></li> Internal Consistency in Streaming Systems</a></li> Dapper</a></li> Zanzibar</a></li> Scalability! But at what COST?</a></li> Flexiraft</a></li> Scalable Statistical Root Cause Analysis on App Telemetry</a></li> Google SSO</a></li> Alea-BFT</a>, All you need is DAG</a> and BullShark</a></li> Patterns of Distributed Systems</a></li> A comprihensive review of BFT Algorithms</a></li> A Survey and Comparison of Consistent Hashing Algorithms</a></li> Systems Correctness and Practices at AWS</a></li> A Scalable CAN</a></li> Some short notes</a></li> </ul> Lectures: MIT 6.824</a> and go through assignments and labs as well here</a></li> Practical Distributed Systems</a></li> NWU CS310</a></li> Jordan’s System Design videos</a></li> A large scale analysis of hundreds of in-memory cache clusters at Twitter</a></li> Distributed Systems Reading Group</a></li> RAFT</a></li> Foundations of Blockchain Protocols</a></li> Introduction to Distributed Consensus Algorithms</a></li> Choose API</a></li> </ul> Fly.io has a very cool distributes systems challenge!</a> Applied Cryptography</a></h3> Reads: Diffie Hellman Protocol</a></li> Serious Cryptography</a></li> Elliptic Curves: Number Theory and Cryptography</a> (vitalik has a cool blogpost on it Exploring Elliptic Curve Pairings</a>)</li> Sigma Protocols</a></li> Understanding ZKP systems</a></li> ZK from SMPC</a></li> Garbled Circuits</a></li> Oblix</a></li> Path ORAM</a> and How Signal Uses ORAM practically!</a></li> Snoopy</a></li> Circom and Halo2</a></li> Plonk</a></li> Replication is not Needed</a></li> Constant-Size Commitments to Polynomials and Their Applications</a></li> ZK-Book</a></li> Succient Proofs and Linear Algebra</a></li> ZKSnarks under the hood</a></li> Concise Notes when working with Bitcoin protocols</a></li> BBS+</a></li> Groth16</a> and Pinochio</a></li> Practical Threshold Signatures</a> and a Survey on ECDSA Threshold signing</a></li> Threshold ECDSA</a></li> Hotstuff 2</a></li> Narhwal and Tusk</a> and BullShark</a> paired with Shoal</a> combined with Mysticeti</a></li> Latency Reduction in P2P networks</a></li> Obscuro</a></li> Advanced Cryptography notes</a></li> Proofs, Arguments and ZK</a></li> ZKDocs</a></li> Various crypto protocols</a></li> Spartan</a></li> Pairings for Begineers</a></li> Blink</a></li> </ul> Lectures: Introduction to General Cryptography</a> and Applied Cryptography</a></li> Advanced Topics in Cryptography</a></li> Winter School to Cryptography</a></li> ZKIAP</a></li> Programming ZKPs</a></li> Multiparty Threshold ECDSA</a></li> ZKBoo</a></li> ORAM</a></li> Pure Rust ECC</a></li> Fully Homomorphic Encryptions</a></li> </ul> AI/ML</a></h3> Lectures: Linear Algebra</a></li> Matrix Calculus for ML and beyond</a></li> Probabilistic Systems Analysis and Applied Probability</a></li> Introduction to Computational Thinking and Data Science</a></li> Intro to Machine Learning</a></li> Deep Learning</a></li> Introduction to Optimization</a></li> Convex Optimizations</a></li> Differential Privacy in Machine Learning</a></li> MLSys Seminar</a></li> AI for Sciences and Engineering</a></li> Physics Informed Machine Learning</a></li> </ul> Reads: Deep Learning</a></li> </ul> Programming Languages</a></h3> Books: Rust Atomics and Locks</a></li> Rust for Rustaceans</a></li> Concurrency in Go</a></li> A Gentle Introduction to Nim</a></li> </ul> Random</a></h3> Robust De-anonymization of Large Sparse Datasets</a></li> </ul> No you don’t have to know all this to be called a great engineer (I am not even a software engineer when writing this!). These are simply a compilation of things I got particularly interested in and simply loved consuming. All the best! Philosophy Unknown — Fri, 12 Jul 2024 00:00:00 +0000 When I am talking about Philosophy, don’t confuse it with Theology, I have a love affair with life not with beliefs! Here are some of my favorite books: The Prophet</li> One Hundred Poems of Kabir</li> Escape from Freedom</li> Sophies World</li> Love (Stendhal)</li> The Monarchy of Fear</li> Dialogues concerning Natural Religion</li> Confessions of an Economic Hitman</li> The Idiot</li> Existentialism is Humanism</li> Notes from the Underground</li> Age of Reason</li> Good Fortune and the Myth of Meritocracy</li> Stories we live by</li> The Great Escape</li> Blind Owl</li> The Mind-Body Problem</li> Labyrinths</li> No Exit</li> The Burnout Society</li> The Lost Daughter</li> The Origins of Political Order</li> The Essays</li> The Feminine Mystique</li> The Invention of Solitude</li> How to Lie with Statistics</li> A Fine Balance</li> Einsteins Dreams</li> The Upside of Irrationality</li> Famine, Affluence and Morality</li> A Life’s Work: On becoming a Mother</li> Then we came to the end</li> The Tyranny of Merit</li> The Book of Mirdad</li> Several Short Sentences About Writing</li> Too loud a Solitude</li> To the Lighthouse</li> Enlightenment Is Your Nature</li> The Lessons of History</li> The Rights of Man</li> The Surrender Experiment</li> Wild Problems</li> The Plague</li> The Brothers Karamazov</li> The Letters to a young poet</li> Mans Search for Meaning</li> The Gardener and the Carpenter</li> The Essential Rumi</li> Affective Neuroscience</li> The Cynic Philosophers: from Diogenese to Julian</li> On The Shortness of Life</li> Die with Zero</li> Penguin Nausea</li> The Myth of Sisyphus</li> Either/Or</li> All About Love</li> Infinite Jest</li> The Structure of Scientific Revolutions</li> Sun and Steel</li> Steppenwolf</li> The Sound and the Fury</li> Why I am an Atheist</li> My Experiments with Truth</li> Tuesdays with morrie</li> The Defining Decade</li> Siddhartha</li> The Stranger</li> The Catcher in the Rye</li> Courage to be Disliked</li> The case against education</li> 1984</li> The Revolt of the Elites</li> East of Eden</li> Lolita</li> Brave New World</li> The Grapes of Wrath</li> The Human Zoo</li> Atlas Shrugged</li> The Politics of Reality: Essays in Feminist Theory</li> On the Genealogy of Morality</li> Pointers to Non Duality</li> Tao Te Ching</li> The Pictures of Dorian Gray</li> On Earth We are Briefly Gorgeous</li> Flow</li> Dream Machine</li> The Tail End</li> The Tell Tail Heart</li> Crime and Punishment</li> The Life of Alexander the Great</li> Beyond Good and Evil</li> The Right to Oblivion: Privacy and the Good Life</li> </ul> These have been a wonderful reads till now and have exposed me to a new me! Some wonderful talks : Achieving your childhood dreams</a></li> Justice</a></li> Mass Immigration</a></li> Building a Life</a></li> </ul> Startups Unknown — Fri, 12 Jul 2024 00:00:00 +0000 The collection of most impactful startup execution texts I have gone through. Whether you want to work on an indie company or a scaled startups, these are some good reads for general perspectives : Only the Paranoid Survive</li> Start small, stay small</li> Remote: Office not required</li> The Managers Path</li> Company of One</li> Zero to One</li> Staff Engineers</li> GenAI and engineering teams</li> Fundraising</li> High Output Management</li> Blitzscaling</li> The Startup of You</li> 4-hour workweek</li> The Hard Thing About Hard Things</li> Shoe Dog</li> The Checklist Manifesto</li> The Lean Startup</li> Play Nice But Win</li> What got you here, wont take you there</li> No Rules Rules</li> Choosing Startup Life</li> </ul> Spent my summers exploring Bitcoin 💸 Unknown — Fri, 12 Jul 2024 00:00:00 +0000 Anyone who is into investing their hard earned money knows about Bitcoin, it’s one thing that has gained significant traction over the years due to its phenomenal returns and dependability. I was too enthusiastic about it, but wanted to accumulate some technical knowledge before pouring in my money! I dont trust, I verify :) Then I stumbled upon Summer of Bitcoin</a> which was a summer internship for university students around globe to learn about Bitcoin and earn a stipend for their OSS work after learning! Now thats what I can being at the right place at the right time! Less than 1% applications are accepted, its extremely competitive and the challenge</a> that I had to solve was mind numbing to say the least, especially because I had no knowledge of Bitcoin technical details, so I learnt on the go (Programming Bitcoin</a> and Grokking Bitcoin</a> were a big help)! Luckily, I got selected to work on an Escrow Module</a> based on paper</a> for the Fedimint</a> ecosystem, under the Summer of Bitcoin 2024, and I must say, I got really humbled and inspired looking at those Staff Software Engineers punching OSS code left, right and center! I mean the maintainers build at the speed of light :). I felt like this</a> and I am grateful for the experience! I was aware of the some plugin/module architectures due to my work at Jenkins</a>. The whole idea is to asyncronyously extend the backend without tinkering with the core, giving the devs the flexibility to empower the ecosystem! The Fedimint Primer</a> is a great starting point for anyone to have an overview! This</a> paper is also a great point to start understanding things conceptually! What’s Fedimint?</a></h3> Imagine you and a group of friends want to create a small, private digital cash system, a bit like a community bank or a shared digital piggy bank. You want it to be secure and trustworthy, even if not everyone in your group is available 24/7, or (in a more serious scenario) if one person tries to cheat. How do you all agree on: Who has how much money?</li> Which transactions are valid?</li> The order in which transactions happened?</li> </ul> This is where Federations and Consensus come in. They provide the framework for a group of participants to collectively manage a system and agree on its state. Imagine the Fedimint federation is your local community bank. The Fedimint Client is like your personal banking app for that specific bank. It’s the software you run on your phone or computer that lets you interact with the federation’s services. What’s Fedimint technically?</a></h3> Fedimint represents a sophisticated approach to scaling Bitcoin and enhancing privacy through federated chaumian e-cash</a>. Its modular architecture, robust consensus mechanism, and Lightning Network integration make it a powerful tool for building decentralized financial applications A Fedimint Federation is a group of server operators, called Guardians - trusted members of the federation, who run the Fedimint software (fedimintd). A majority of these Guardians need to be honest and reliable. The fedimintd</code> serves as the main consensus code for processing transactions and providing a REST API. It’s the heart of the Fedimint server implementation. Fedimint uses AlephBFT consensus algorithm</a> to achieve agreement among federation members about the state of system like : Which Transactions are valid</li> The order in which these transactions occurred</li> The current state of any Modules (like the digital cash module)</li> </ul> The result of this consensus process is a shared ledger. // fedimint-core/src/epoch.rs /// All the items that may be produced during a consensus epoch #[derive(Debug, Clone, Eq, PartialEq, Hash, Encodable, Decodable)] pub enum ConsensusItem { /// Threshold sign the epoch history for verification via the API Transaction(Transaction), /// Any data that modules require consensus on Module(ModuleConsensusItem), // ... other variants }</code></pre> A ConsensusItem</code> can be a Transaction or data specific to a Module (like an e-cash minting operation). How a transaction is agreed upon : +--------+ +-----------+ +-----------+ +-----------+ +----------------------+ | User | | GuardianA | | GuardianB | | GuardianC | | Consensus Algorithm | +--------+ +-----------+ +-----------+ +-----------+ +----------------------+ | | | | | | Submit Proposal (e.g., tx) | | | | -------------------------------> | | | | | Share Proposal | | | | --------------> | | | | Share Proposal | | | | -------------------------------> | | | | Share Proposal | | | | -----------------> | | | | | | | | | | | | | | <------------- Guardians broadcast proposed items to each other ---->| | | | | | | | | | [ AlephBFT Ordering & Agreement Phase ] | | <----------- Guardians exchange messages -------------> | | <--- Agree on order and validity of proposed items ---> | | | | | | Finalize a batch of ordered items = SessionOutcome | | | | | | <------------- SessionOutcome Generated --------------> | | | | | | Guardians sign SessionOutcome with threshold keys | | <------------- Sign Outcome (2/3+) -------------> | | | | | | <--- SignedSessionOutcome assembled from signatures ---> | | | | | | SignedSessionOutcome recorded by all Guardians | | ---------------- Ledger Updated ---------------- |</code></pre> On the server-side (fedimintd), a component often referred to as the “Consensus Engine” manages this whole process. It takes items submitted for consensus, works with AlephBFT to order them, and then processes the agreed-upon items. Fedimint Client is responsible for managing secrets, assets, building transactions, talking to federation, tracking operations, using modeules, etc. Fedimint has an SDK in place in fedimint-client</a> for providing high-level operations for interacting with the Fedimint federation using a Dynamic api client DynGlobalApi</code> and client side data, initialized by ClientModuleInit</code> which will be implemented by each module. It provides utilities for building and submitting transactions and uses a state machine (client state machines are for writing client side logic that the module might need so that they can be executed in idempotent steps that can get persisted in the database. This is so that everything work even if a client device is being constantly being shut-down abruptly etc. When you send e-cash or perform other operations that change ownership of assets, the client builds a Transaction. This transaction will contain: Inputs: What you’re spending (e.g., some of your e-cash notes).</li> Outputs: What’s being created (e.g., new e-cash notes for the recipient, and potentially change notes for yourself).</li> Signatures: Cryptographic proof that you authorized the spending of your inputs.</li> </ul> Typically inputs and output of a given transaction have client side state machine. Sometimes some other state machines are needed as well. Typically a transaction is submitted to federation, and client needs to do something afterward - e.g. wait for the transaction to get accepted or rejected and do some corresponding things afterwards. The states in the state machine can be thought of “snapshots” saved into the database. The SDK also handles logging and error handling. pub struct Client { // Configuration specific to this federation config: tokio::sync::RwLock<ClientConfig>, // Your client's local database db: Database, // Your unique root secret for this federation root_secret: DerivableSecret, // API for talking to the federation guardians pub(crate) api: DynGlobalApi, // Registry of client modules (e-cash, LN, etc.) pub(crate) modules: ClientModuleRegistry, // Executor for running state machines executor: Executor, // Task group for managing background tasks task_group: TaskGroup, // ... other fields }</code></pre> The modules</a> are the lifeline of fedimint architecture! The Mint module (implementing federated e-cash), Wallet module (handling on-chain deposits and withdrawing) and the Lightening module (integration with the Lightening Network allowing users to send and receive Lightning payments using their e-cash balance) serve as the core modules on which the Fedimint ecosystem functions. The developers can extend the fedimint-core</code> by adding more modules</a>, like I am doing right now :) // Add modules you want to use (e.g., e-cash, Lightning) // client_builder.with_module(MintClientInit); // For e-cash // client_builder.with_module(LightningClientInit); // For Lightning let client_handle = client_builder .join(my_root_secret, federation_config, None / api_secret /) .await?;</code></pre> This join process initializes your client for that specific federation. The finalize_and_submit_transaction</code> function is a core part of the Client. It takes a partially built transaction, works with modules to complete it (e.g., by adding e-cash inputs to pay for it and creating change outputs), and then sets up a state machine (TxSubmissionStatesSM) to manage the actual submission to the federation and track its acceptance. The ClientHandle</code> manages the lifecycle of the Client. When the last ClientHandle is dropped (goes out of scope), it triggers a shutdown process for the client, stopping its background tasks and releasing resources. This is important for clean program termination. Most of your interactions will be through this handle. // fedimint-client/src/client/handle.rs pub struct ClientHandle { inner: Option<Arc<Client>>, }</code></pre> How client interacts with Federation : +-----------+ +----------------+ +--------------+ +--------------------------+ +---------------------+ | User App | | FedimintClient | | E-cash Module| | Federation API (Guardian)| | Client Local DB | +-----------+ +----------------+ +--------------+ +--------------------------+ +---------------------+ | | | | | | I want to deposit BTC! | | | | | ----------------------> | | | | | | User wants to deposit | | | | | ----------------------> | | | | | | Request deposit address | | | | | -----------------------> | | | | | <----------------------- | Returns BTC address | | | | | | | | Got deposit address | | | | | <---------------------- | | | | | | | | | | | [Starts StateMachine to | | | | | watch deposit address] | | | | | | | | Send BTC to [address] | | | | | <---------------------- | | | | | (User sends BTC) | | | | | | | | | | | | | | | | | | | | | | | | | | | Later, after confirmation... | | | | | | | | | Has deposit confirmed? | | | | | -----------------------> | | | | | <----------------------- | Yes, here's new e-cash notes | | | | | | | | | Store e-cash notes | | | | | -----------------------> | | | | | | | | | Deposit successful | | | | | <---------------------- | | | | Deposit complete! | | | | | <---------------------- | | | | | | | | |</code></pre> Every Fedimint module has two parts that work together: ClientModule: This defines how your client application interacts with the features of that specific module. It lives within your Fedimint Client.</li> ServerModule: This defines how the federation’s guardians manage the state and rules for that module. It lives within the fedimintd server software run by the guardians.</li> </ul> The ClientModule for e-cash (often called MintClientModule) is responsible for things like: Knowing how to construct a request to get new e-cash notes.</li> Storing your e-cash notes securely in the client’s database.</li> Knowing how to select the e-cash notes you own when you want to spend them.</li> Creating the “input” part of a Transaction that proves you own the notes you’re spending.</li> Understanding how to interpret responses from the federation related to e-cash.</li> </ul> The ServerModule for e-cash (often called MintServerModule) runs on each guardian’s server. It’s responsible for: Defining the rules for valid e-cash notes.</li> Issuing the cryptographic signatures (blind signatures) that create new e-cash notes when funds are deposited.</li> Verifying the “input” part of a Transaction when someone tries to spend e-cash notes. This includes checking for valid signatures and preventing double-spends.</li> Keeping track of which notes have already been spent (managing the module’s state).</li> Participating in the Federation & Consensus process to agree on the validity of e-cash transactions.</li> </ul> // fedimint-server-core/src/lib.rs (Simplified) #[apply(async_trait_maybe_send!)] pub trait ServerModule: Debug + Sized { // ... other types ... /// This module's contribution to the next consensus proposal. async fn consensus_proposal<'a>( &'a self, dbtx: &mut DatabaseTransaction<'_>, ) -> Vec</ ... ConsensusItem type ... />; /// Process a consensus item agreed upon by the federation. async fn process_consensus_item<'a, 'b>( &'a self, dbtx: &mut DatabaseTransaction<'b>, consensus_item: / ... ConsensusItem type ... /, peer_id: PeerId, ) -> anyhow::Result<()>; /// Verify and process a transaction input related to this module. async fn process_input<'a, 'b, 'c>( &'a self, dbtx: &mut DatabaseTransaction<'c>, input: &'b / ... Input type ... /, in_point: InPoint, ) -> Result<InputMeta, / ... InputError type ... />; /// Verify and process a transaction output related to this module. async fn process_output<'a, 'b>( &'a self, dbtx: &mut DatabaseTransaction<'b>, output: &'a / ... Output type ... /, out_point: OutPoint, ) -> Result<TransactionItemAmount, / ... OutputError type ... />; // ... other methods like verify_input, audit ... }</code></pre> The spend flow : +-----------+ +----------------+ +---------------------+ +----------------------------+ +------------------------+ | Your App | | FedimintClient | | Mint Client Module | | Federation API (Guardians) | | Mint Server Module | +-----------+ +----------------+ +---------------------+ +----------------------------+ +------------------------+ | | | | | | Spend 100 sats | | | | | -----------------> | | | | | | Prepare to spend 100 | | | | | ---------------------> | | | | | | Selects e-cash notes | | | | | from local DB | | | | | | | | | Input data (note+proof) | | | | <--------------------- | | | | | | | | | | Build Transaction T (includes e-cash input) | | | | ------------------------------------------------> | | | | | | Verify e-cash input from T | | | | | -----------------------------> | | | | | Check signature + spent flag | | | | | | | | | | | | | | | +-------------------------+ | | | | | | Input Valid? | | | | | | +-------------------------+ | | | | | / \ | | | | | Yes No | | | | | | | | | | | | Mark note as Reject due | | | | | spent (consensus) to double | | | | | spend | | | | | | | | | | | | <-------------------- | | | | | Input OK Input Invalid! | | | <--------------------- | | | | Transaction Confirmed | | | | -----------------------------> | | | | Spend Successful | | | | <---------------------------- | | | | | | | | | | | | [Else Branch: Double Spend] | | | | | | | | | | Transaction Rejected | | | <------------------------------------------------ | | | Spend failed (Double Spend) | | | | -----------------------------> | | | | | | Revert note to DB | | | | ---------------------> | | | | Spend Failed (Double Spend) | | | | <---------------------------- | | | </code></pre> The module creation process is eased via the custom-module-example template</a>. You can play with the module in a nerdy way using just mprocs</code> inside the nix environment, just create some users and have fun with your module! DB</a></h3> It uses RocksDB</a> as the high-performance embedded key-value storage engine (can be easily integrated into the Fedimint application without requiring a separate database server) due to its high performance, especially for write-heavy workloads (which is crucial for a financial system like Fedimint) and fast reads, optimistic transactions and more! Fedimint has customized RocksDB options to suit its specific needs, such as setting write buffer size (performance optimization), use custom prefix-keys and recovery mode. This</a> is sweet to go deeper into RocksDB! Nix</a></h3> The first hurdle I faced before contributing was to get myself comfortable with the reproducible Nix</a> environments (due to my low storage on 256GB macbook air, I had to nix-collect-garbage -d</code> a lot, deeply painful, but Nix does make the developer experience outstanding). Nix is extensively used in the Fedimint project for managing the development environment (Flakebox) , building the project, and deploying Fedimint instances, cross-compilations in CI builds, and more ofcourse! Also, debugging rust in Nix was an exciting adventure! I wrote Starting to pick up Nix a bit</a> as a collection of resources that I loved! Monitoring and Observibility</a></h3> Fedimint implements a robust metrics collection and exposure mechanism using Prometheus. Each module has its own set of metrics. The global collection of metrics happens in a static, lazy-initialized Registery</a> using a custom macro system. It utilizes various Prometheus metric types, including Histogram, IntCounter, Gauge, and their vector counterparts. An new thing is the predefined buckets</a> for amount-related histograms, chosen to cover a wide range of Bitcoin amounts, from fractions of a satoshi to 100 million satoshis (1 BTC). Fedimint exposes its metrics through an HTTP endpoint, implemented</a> using the Axum. As soon as the the fedimintd process starts, the metrics server also starts</a>. Metrics are updated when the transaction is successful maintaining consistency between DB and metrics. It supports OpenTelemetry integration with Jaeger for distributed tracing. These can easily be visualized in Grafana! The load testing tool measures and compares performance metrics</a> across different runs, tracking performance changes over time over different configurations. Performance Analysis</a></h3> Chrome tracing</a> is implemented for integration tests (devimint</code> is used for local fedimint environments) in Fedimint, generating a JSON file named trace-{UNIX_TIME}.json</code> in the current working directory which can be opened and visualized using Perfetto</a>. This setup allows developers to perform detailed performance analysis of integration tests by visualizing the execution timeline, identifying bottlenecks, and understanding the flow of operations within the Fedimint system during test runs. Also, fedimint-load-test-tool</a> is for performance testing for federation, scalability assessment, gateway integration testing. Wasm Tests</a></h3> The Wasm tests</a> are crucial for ensuring that Fedimint’s client-side operations work correctly in a browser environment, which is essential for web-based wallets or applications using Fedimint. This is a really cool thing that I saw implemented in a Bitcoin project for the first time! It also has fuzz testing but thats quite standard, so wont write about it. Damn, the repository is sooo big, it will take me an eternity writing all about it, it would be best for you to check out the repository</a> yourself, don’t forget to star it! I got rustier :)</a></h3> Naturally, was curious on why folks are using Rust at the first place!</a> especially in blockchain</a>. Was pretty excited. Played with async Rust</a>, and explored journey of Rust to async programming</a>. I got intimidated by the codebase initially due to extremely complex and long Rust Types but this</a> tutorial helped me a ton! I went deep into visualizingn memory layout of Rust Datatypes</a> just for fun! Learnt about Static and Dynamic dispatcher! You know, its slightly harder to write some Data Structures in Rust!</a> My mentor suggested me to look into Rust Design Axioms</a> and in general better Rust patterns and techniques to improve maintainability and overall code quality for which I referred to this</a> talk. The most dreaded thing that I had to face was the Rust’s overconservative Borrow Checker</a>. Explored rust compile times</a> as well! Got exposed to conditional compilation between native and WASM targets</a> and handling runtime differences. Hah! just got gently pushed to Advanced Rust!</a> Got initiated to think about structuring my code and thinking about the system design</a>. Also got some advise on code reviews</a>. The best Rust tutorial that I could find was this</a>! The more I use Rust, the more impressed I am! Laughingly, during the internship, instead of doing Test Driven Development, I ended up doing Error Driven Development ;( Thank you!</a></h3> The fedimint community was exceptionally warm and welcoming to the new contributors. Special thanks to the maintainers KodyLow</a>, Manmeet</a>, David</a>, Justin</a> and my mentor Shaurya</a> for making the experience a memorable one! I will cherish it forever, they initiated me to Bitcoin! Bitcoin stuff I am excited about in the near future!</a></h3> Really pumped up on how FROST signature schemes</a> paired with Dynamic and Verifiable Secret Sharing</a> will make the multi-sig wallet experience buttery smooth! The most awaited would be to have backup wallets! So Cool, it will redefine the Bitcoin experience for the users while also improving security :) Also excited to see the use of Conditional Payments using Discrete Log Contract</a> suitable in cases where 2 mutually distrusting parties bet on the results! Also, BitVM</a> is a heart-throb! And work on Bitcoin L2 is also cool</a>, and I think StarkNet might very realistically scale Bitcoin with Ethereum</a>! My future generation will inherit a superior money, Bitcoin!</a> Wanna Contribute to Bitcoins Glory?</a></h3> This</a> is a solid starter advise for anyone new! Get accustomed to Bitcoin Development Philosophy</a>!

missing writing some concurrency in Go

i should have paid a closer look at tokenizer

cutting that inference cost

some insights on HNSW

how package managers use PubGrub for Dependency Resolution

the longer you chat, the worse your agent's response

DB Joins on GPUs!!

Which Prompts Actually Work for your Agents?

The chronicles of training FBPINNs

The Rusty path to Secure Metrics in Ambient Mesh

Grateful</a></h2>
I am more than grateful to the wonderful folks at Solo.io who mentored me - Ian Rudie</a>, John Howard</a>, Ben Leggett</a> and Faseela K</a>. Forever grateful to them for volunteering their time and effort.</p>

Do I officially qualify as a Software Engineer now?

The frenzy of building and observing seniors!

Working at a early-stage startup!

What I learnt about myself</a></h2>
Atleast for the time being, research will be on my cards. I really underestimated managing people. I am the god’s chosen IC</em></p>

Wealth

Starting to pick up Nix a bit!

Nix on CI</a></h2>
Since at the core of any CI, Nix is doing the same thing, there is no real vendor lock-in issue. This also means it should be possible to accurately benchmark these CIs</a>.</p>

Computer Science

Philosophy

Startups

Spent my summers exploring Bitcoin 💸

HARSH PRATAP SINGH

Living the GitHub Actions Dream

All sorts of famous Attention Layers

HARSH PRATAP SINGH

Living the GitHub Actions Dream

All sorts of famous Attention Layers

missing writing some concurrency in Go

i should have paid a closer look at tokenizer

cutting that inference cost

Python overhead</a></h2> At small model size, naive Python code overhead is large</p> cuda graphs (xFormers has a great example particularly in accelerating FlashAttention and linear layers)</li> tensorrt-llm (tracing the inference + pattern matching)</li>

What drives these metrics?</a></h2> Fixed Flops / Memory Bandwidth ratio = Minimal batch size B* to avoid wasting flops</li> Limited on device memory = Not trivial to reach this batch-size B*</li>

some insights on HNSW

how package managers use PubGrub for Dependency Resolution

the longer you chat, the worse your agent's response