LLM routing
Your router does not know your evals
Model routing is an evaluation problem. A generic router cannot know where smaller models are good enough for your workload, but a router trained on your own judgments can.
ReadDiscipline under uncertainty.
रियाज़ · प्रणाली · स्वर
Chitral Patil
AI infrastructure, open-source systems, and Hindustani classical music.
I study and build the systems behind modern AI — especially the messy economics of serving LLMs in production. I'm also a Hindustani classical vocalist trained since childhood, which probably explains the obsession with structure, timing, and improvisation.
live · vllm-cost-meterstreaming
≈ USD 1.90
~68% utilization
ताल
illustrative · cost/1M output tokens
latency is tempo · telemetry is drone
Currently
What I'm in the middle of
A snapshot, not a CV. Updated when the work moves.
- Buildingvllm-cost-meter — live, objective cost telemetry for self-hosted LLM inference.
- ResearchingHow concurrency, latency SLOs, and hardware change the real cost of serving open-weight models.
- WritingField notes on why per-token pricing hides the actual economics of inference.
- PracticingHindustani classical vocals — riyaz, the same discipline that shapes the rest.
Research
The real cost of serving LLMs
An independent, concurrency-aware look at what inference actually costs — measured, not assumed.
Featured paper · arXiv
Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation
Token price is not serving cost.
Public LLM cost calculators often reduce serving economics to a static token price or assumed utilization. This work studies how request rate, concurrency, latency SLOs, hardware, model architecture, and quantization interact to change the real effective cost of self-hosted LLM inference.
- 36.3×Underutilization penaltyeffective cost near idle vs. saturated, same hardware
- 0.21 → 15.25USD / 1M output tokenseffective cost range on identical H100 hardware
- 42Benchmark validation runsacross load and concurrency regimes
- H100 / A100Hardware validatedtwo GPU classes, measured not assumed
Open source
Build the meter, not another calculator
A read-only observer that tells operators the truth about serving cost under their own traffic.
Flagship · open source
vllm-cost-meter
Objective live telemetry + effective cost-per-million-token meter for vLLM servers.
A read-only observer for running vLLM servers that ingests Prometheus metrics and surfaces live effective LLM serving cost against the operator's actual traffic.
- Reads vLLM Prometheus metrics
- Tracks throughput, request rate, TTFT, TPOT, E2E latency, prompt / generation lengths, batch state, and KV cache
- Computes live effective cost-per-million-token visibility
- Ships benchmark reference curves from the paper
- Independent, reproducible research artifact
Systems + Sound
Two halves, one discipline
Before AI, there was riyaz. The habits transfer more than people expect.
Sound
रियाज़
Before AI, there was riyaz.
I started training in Hindustani classical music around the age of five — years of riyaz before any code. It was never a hobby I dabbled in. It's the first thing I went deep enough on to actually master, and that depth is where the rest of how I think came from: patience, structure, attention, and the ability to stay with complexity for a long time.
Systems
- rhythmconcurrency
- tempolatency
- raagstructured systems
- tanpura dronetelemetry
Improvisation inside structure, repetition without boredom, deep listening for the thing that is slightly off. The same instincts run both columns.
Writing
Field notes
Short, sharp pieces on inference economics, tools, and practice.
startups
The difference has to click
What making cover songs under unreasonable deadlines taught me about invisible effort, legible differentiation, and learning to lead the orchestra instead of playing every instrument.
Readdiscipline
The unlosable thing
Why I do the expensive version of things: the proof you can rebuild it is the only asset worth holding, the trick is controlling where you collect your dopamine, and slightly-above-average is the flywheel's ignition.
ReadArc
How the throughline runs
Not a résumé timeline — the same instinct showing up in different rooms.
- Age ~5रियाज़
Riyaz begins
Hindustani classical vocal training starts. Years of repetition, structure, and listening before any code.
- Earlier workसंकेत
Data & systems
Scrapers, ETL over SEC filings, and software systems — learning to model messy real-world data and ship.
- Productअभ्यास
Building things people use
Propael and other product experiments. Taste for the painful real problem, not the demo.
- Nowप्रणाली
LLM inference economics
A concurrency-aware cost methodology and vllm-cost-meter — measuring what serving actually costs.
Discipline under uncertainty.
If it serves tokens, it has a cost. Let’s measure it.
For LLM serving economics, open-source telemetry, research collaboration — or just to talk systems and sound.