Prefill Disaggregation - Understanding from first principles

` In order to generate output tokens from an input prompt, LLM inference is split into two stages: prefill and decode. Prefill runs on the input tokens, populating KV caches, before entering the decode stage that generates tokens one-by-one. While a single decode step typically runs for tens of milliseconds, prefill takes substantially longer. If run on the same devices, mixing prefill with decode degrades decode performance. In this article we explore an established solution in the form of disaggregated prefill and decode, running them on separate devices to maximize both prefill throughput and decode latencies. `

Demystifying LORA

In the world of optimizing deep neural networks, torch compile and triton are perhaps the most important tools.I had no idea about the fundamentals so I dug in using first principles aided by ChatGPT.

Designing Lightweight asynchronous AI agents

The concept of an AI agent goes back to at least 2015-16 when the term was first coined by Bengio et al. From the paper “We present an approach for learning simple algorithms such as copying, multi-digit addition and single digit multiplication directly from examples. Our framework consists of a set of interfaces, accessed by a controller. Typical interfaces are 1-D tapes or 2-D grids that hold the input and output data. For the controller, we explore a range of neural network-based models which vary in their ability to abstract the underlying algorithm from training instances and generalize to test examples with many thousands of digits. The controller is trained using Q-learning with several enhancements and we show that the bottleneck is in the capabilities of the controller rather than in the search incurred by Q-learning.”

Leetcode and it’s worth

No one likes leetcode. At least not anyone that I know. Everyone considers it a necessary evil to get past FAANG and other tech interviews. I must have worked on Leetcode with the same attitude for nearly 3 years now. I am quite shocked by my own dashboard I had never thought that I would be able solve 300 problems on leetcode. It’s only in the last year or so I realized that this number means nothing. If I randomly picked up one of the problems that I had solved and tried to solve again, I’d be stumped.

My journey with AI Code Editors

How I hacked my concentration

“To realize the value of what one has, one has to lose it”

How to design a payments system for e-commerce?

I was just thinking through some of the past problems that I have solved in my career and retrospecting about how I would solve them today. I was in a group discussion with a bunch of folks discussing a payments system design use case and it brought back memories of how I went about designing the system. Given the proprietary nature of what I did, I want to focus on what I could have done differently or better.

My three heroes

The initial premise

Around March 2024, I toyed with the idea of building an AI coach for tennis. Why tennis of all sports. Why not golf or badminton or other racket sports.I don’t have an answer to this. I found tennis to be a far more technical sport than badminton or ping pong. Also it felt way harder to learn than any other sport I have played. So I thought I will build out an app that will be a very basic version of a coach. I did some basic market research and found out that there was a play here. Based on the total number of tennis players and enthusiasts this seemed like a 10-100M$ idea at the very least.

Understanding GDP

Whats the real use case for a blockchain other than a cryptocurrency

Common web apps in the distributed world.

Distributed everything

Or decentralization, however we term it, one paradigm shift that we are witnessing is the transition to deentralized computing. Predominantly driven by “cryptocurrencies as the antidote to centralized money printing that is supposedly endangering global finance”, school of thought. With that said premise, I tend to think,

The beginning, sometime in the future

It is my belief that progress plateaus once you reach a state of satisfaction. Which is why most of the progress on this planet is driven by gree

I happened to be just browsing through Paxos made simple and trying to undersand the principles that have become the bedrock for clustering and horizontally scalable applications, including databases.

Often, in the startup world I come across this choice paradox that investors seem to face. What is the predmoinant critera that drives our investments. Do we invest in people first or market first or technology first. Sequoia proudly invests in markets first and A16Z prodouly invests in people/team. So what has driven the most tectonic shifts in technology in this century ?

This is just a place for my ramblings, acciedntal discoveries and crap that goes on in my day to day life.