aiprodigy.devWriting on the engineering under data, models and agents. New piece every other Tuesday.
ERP → Cloud → Machine learning → Deep learning → Agents
I've built and architected through every one of Technology Innovations of last three decades - Y2K, Web, Mobile, Big Data, Cloud and Data/AI/ML/Integration Platforms. What decided whether they shipped was never the technology - it was the engineering around it.
Fundamentals aren't obsolete; they're the biggest advantage you have, and only handful of people are talking and writing about it. This changes now.
The libraryLong-form only. Nothing under 2,000 words.
Most RAG debugging starts at the prompt and the model, when the failure usually happened earlier — at chunking. This post runs six chunking strategies against the same 400-page corpus and scores each one on whether the right passage actually made it into the context window, with the eval harness and the golden set published alongside the results.
Writing the prompt before the eval is the same mistake as writing code before the test — it feels faster and costs more later. This post makes the case for treating the eval as the spec an AI feature is built against, and shows what teams actually pay when they add it after the feature has already shipped.
Gradient descent gets used long before it gets derived, and that gap is where the intuition for tuning it never forms. This post derives it by hand — the update rule, why the learning rate behaves the way it does, and where the method breaks — so it stops being a black box borrowed from a library.
The library is empty on purpose - I'd rather publish the first piece finished than fill this page with links. Subscribe to the feed above and you'll get it the day it goes up.
AboutShort version. The long one is a post I haven’t written yet.
22 years as a BUILDER - an engineer and an architect. The first twelve were ERP and cloud systems where being wrong was expensive, slow to undo, and someone’s actual job. The last ten have added data, machine learning and AI on top.
Most people treat that first half as the part that expired. I think it’s the reason the second half worked. An agent is a distributed system. A retrieval pipeline is a search problem with a data-quality problem underneath it. An eval is a test suite. Each of those has decades of hard-won practice behind it - and most of it is being rediscovered badly, at expense, by people who were told the fundamentals didn’t apply any more.
So that’s what I write about: the fundamentals, the architecture, the engineering discipline and the design principles underneath data science, machine learning and AI - not the model of the month. The posts are long, they carry working code, the illustrations, and they show what didn’t work next to what did.