In the last post I mapped Brickbot onto the Databricks platform — ingestion here, governance there, a Genie agent for the hard questions — and then I made a promise: that I'd come back and write about what each piece looked like in practice, where it broke, what I'd change. That post was the map. The Summit was the territory, and the territory had opinions.
The best way I've found to show any of this is to stop describing the architecture and just watch a single request go through it. Every reply Brickbot gives is the visible tip of a chain — retrieval, a model, some governed tools, memory. So here's what I'll do: ask it four questions, open the trace of each one in MLflow, and follow it backward from the answer down to the raw data. And at every layer, I'll tell you what happened when the doors opened and thirty thousand people showed up at once.
"Give me a session on real-time streaming"
Start with the easy one. Brickbot searches, hands back a couple of picks, done. Now open the trace of that turn — and before we chase it anywhere, just look at its shape:
Here's the thing worth internalizing before anything else: every turn has this exact shape. Memory, prompt, model, one tool, model again. Only the tool in the middle changes from question to question. Once you see it, a chatbot stops being a mysterious box and becomes a very short, very legible pipeline — which, not coincidentally, is why I could fix so much of it mid-conference without redeploying anything. Two of these steps are the same on every single turn, so let me deal with them first, because they're also where the two best stories live.
The prompt (and the four days I spent editing it live)
The system prompt isn't in the code. It's loaded at request time from the MLflow Prompt Registry — one for anonymous visitors, one for authenticated users, an FAQ variant, each versioned, each with a @production and @local alias. To change Brickbot's behavior you don't ship anything; you cut a new version and move the alias, and the app picks it up on the next turn.
This was, without exaggeration, the most valuable thing I had all week. The authenticated prompt landed a new version nearly every day of the event, and almost every one started the same way: I'd be scrolling the live traces, catch an answer I hated, and patch the prompt from the API a few minutes later. When people started coaxing it into revealing internal details about how it worked, that was a prompt version. When it kept giving up on questions it actually had the tools to answer, that was a prompt version. The morning of the keynotes, when Databricks announced a pile of new products, we taught Brickbot about them from the audience — another version, live. None of it touched a deploy pipeline. If you take one thing from this post: your prompt is a versioned artifact with a production alias, not a string literal in your repo. That single decision is the difference between "we'll fix it next release" and "fixed, thirty seconds ago."
The model (and the Day 1 incident I don't love talking about)
The model is Claude Haiku, served through the Unity AI Gateway as a plain OpenAI-compatible endpoint. About an hour into Day 1, that endpoint started throwing 429s. Thirty thousand people is, it turns out, more concurrency than our happy path assumed, and suddenly a chunk of turns were failing in front of a live audience.
Two fixes, same afternoon. First, the boring-but-critical one: a graceful failure so a rate-limited turn degraded to an apology instead of a spinner of death. Second, the fun one — because the model is just a URL, a token, and a name, I stood up cross-account failover: if the primary account 429s, retry the exact same call against a second Databricks account. That's a few hours of work, not a rewrite, precisely because there's nothing bespoke about the model endpoint. And because everything is traced, I could actually see the failover firing and confirm it was working instead of hoping. The other thing that dashboard showed me was that flat green line: we were paying full input price on every turn because prompt caching wasn't on. Turning it on — cache_control over the static part of the system prompt — got the cached line off the floor and cut input cost by about two-thirds, which is a very good thing to fix on the exact afternoon your traffic goes vertical. The lesson generalizes to any agent: put a gateway between you and the model, treat the model as swappable from day one, and watch the caching line — because the day you need any of that is not a day you'll want to be writing a client.
The memory step at the very top — the first thing that happens, before the model even thinks — I'm going to leave sitting there for now. We come back to it in question four, and it's a better story with the setup.
Now the tool — and the thread down to the data
In this trace the tool is search_sessions, and the one call underneath it is a POST /sql/statements. The "search" is a query against an AI Search index (what Databricks called Vector Search until this year) — hybrid, so it does classic keyword (BM25) and vector similarity at once. That mattered more than it sounds: it meant hybrid search was a drop-in for the keyword search we already trusted, not a rewrite that begged us to relearn what "relevant" means.
But an index is built from a table, so the honest question is: which one, and can I trust it? Click lineage and the platform just tells you — the index came from a silver table, which came from a raw table, which came from a job. I didn't set any of this up by hand; it's there because every object is governed.
Follow that graph one hop further and you land on the job: a Lakeflow pipeline on a 15-minute schedule, bronze (the API response, stored faithfully) then silver (trimmed to what we actually serve). "Every 15 minutes" sounds like a throwaway detail until you remember conference data is alive — a room gets reassigned, a session gets pulled. In fact one of our stranger constraints lived right here: session rooms were embargoed until midnight the night before the show, so the silver layer simply withheld the room column until the clock passed it. That's the kind of rule that's a nightmare in application code and a one-line filter in a pipeline.
And what does the job call to get the data? Not a hand-rolled API client — Unity Catalog functions (get_sessions, get_speakers, get_exhibitors), each a thin, governed wrapper around http_request.
get_sessions — a governed SQL function around an HTTP call. A grantable object, not a snippet buried in a repo.Those functions all point at one registered HTTP Connection, which is where the credential lives — permissioned like any other catalog object, invisible to the code. In previous years the "connection" to our event vendor was an API key on my laptop and a bus factor of one. Now it's an object I can grant to exactly who should have it. (The part that did bite us, for the record: the vendor's API names its fields startDateTime and endDateTime, and for weeks we read startTime and quietly got null. Governed or not, you still have to read the docs.)
So one "find me a session" traced the whole way down: trace → AI Search → lineage → Lakeflow → UC function → HTTP connection → the vendor API. Every hop a governed object. That's the backbone. The next three questions just swap the tool in the middle — so they'll go faster.
"How many distinct companies are speaking, and break the sessions down by day and time"
Search is great at "find me one." It's useless at "count the distinct companies" or "group these by morning and afternoon." So the model reaches for a different tool — ask_genie — and hands the hard part to a Genie Agent, a sub-agent that writes and runs SQL over the same tables. This is the one genuinely new capability this year, and it's why Brickbot could suddenly answer a whole class of question it used to just apologize for. And it doesn't hide the hand-off — Brickbot surfaces the sub-agent's entire reasoning right in the chat: what it understood, the data it picked, its plan, and the SQL it ran.
Look at the SQL in that panel: it isn't scanning the raw tables, it's reading MEASURE(distinct_companies) straight from brickbot2026.metrics. Those are the agent's metric views, and they're the part worth copying. The questions we knew would get asked — counts, distinct companies, sessions by day and time — are defined once as measures and materialized, so the agent reads a number instead of re-deriving a join every time someone asks. It's also where hard-won semantics live: morning is before noon, afternoon after, and — because the model kept confidently inventing one — there is explicitly no evening.
Two things went sideways here and both are instructive. First, time. On Day 1 Genie was reporting session times seven hours off — it was helpfully "converting" our timestamps, which were already Pacific, into UTC. Live fix. Time zones are the tar pit of every scheduling agent; the only thing that saved us was, again, seeing it in a trace within minutes. Second, throughput. Genie has a per-workspace ceiling, and we approached it. We got the ceiling raised, and I added a small rate limiter so that if we ever went over, Brickbot would quietly fall back to plain search instead of queueing behind a slow analytical query — and I taught the prompt to keep our internal capacity numbers to itself, because a helpful bot will absolutely tell a curious attendee exactly how it's provisioned if you let it. The general pattern under all of this: when a sub-problem needs a different kind of reasoning, give it to a specialized sub-agent instead of cramming it into one prompt, and put a semantic layer in front of it so "how many customers" means the same thing every time it's asked.
"What's on my schedule?"
The first two questions were anonymous — it didn't matter who was asking. This one is different, and the difference is the whole ballgame for anyone building agents that act for real people. The tool is get_my_schedule, another governed UC function, shaped like the search ones but taking an attendee ID and calling the vendor's personal-schedule endpoint.
My rule, and I'd die on this hill: the model never touches authorization. By the time get_my_schedule runs, we've already resolved who the user is — that happens in the one piece of this whole system that isn't Databricks, a thin app that validates the attendee and injects their identity into the prompt before the model runs a single token. The LLM is handed "you are talking to attendee X" as settled fact. It never decides it, so it can never be talked out of it. If you let a language model reason about who it's allowed to act as, you've built a very charming security hole, and prompt injection will find it. Resolve identity outside the model, pass it in, and keep the model out of the trust decision entirely.
One nice touch and one ugly surprise. The nice touch: the vendor's schedule comes back thin, so we enrich it in place with a LEFT JOIN to our sessions table and hand the model real titles, rooms, and times instead of bare IDs — the kind of thing that's trivial when your operational call and your analytics live in the same place. The ugly surprise: the vendor's schedule endpoint sometimes lagged, so a freshly-favorited session would briefly return nothing, and Brickbot would cheerfully tell someone their schedule was empty. We added a model-in-the-loop retry for exactly that case. Real systems are held together with small, specific patches like this, and you only find them by watching real traffic.
"What do you know about me — and file a feature request for me"
Ask two things at once and the model just chains two tools in a single turn. Both of these are about memory, in two different senses, and together they close the loop I left open back in question one.
list_my_memories reads the managed agent memory — a durable, semantic profile of what you care about. And here's the callback: that is the exact store the trace quietly loaded at the very top of every turn. The memory step in Fig 2 that I told you to hold onto? That's Brickbot reading your profile before it thinks, so it can be useful from your first message instead of your fifth. Now we're just asking it out loud.
submit_feature_request writes — and in the trace you can watch it reach for a Postgres credential and hit Lakebase, the operational store where the app's own state lives: raw chat_turns, the feature_requests people file, a little user_ui_state so a first-time visitor gets different suggestions than a regular. This tool didn't exist when the doors opened. Halfway through Day 1 I noticed people were telling Brickbot what they wished it could do — and it was just being warmly useless about it, nodding along. So we shipped a tool to capture those as real, queryable feature requests, and taught the prompt to offer one when it hit a wall. Some of the best product feedback we got all year is sitting in that table.
The symmetry I like: the trace is how you watch the agent; the operational store is how it remembers. Two ledgers of the same conversation — one for running it, one for debugging it. Any serious agent ends up needing both, plus a semantic profile on top for the "remembers you" feeling.
One trace becomes sixty thousand
I've been opening single traces this whole time. Pull all the way back and the same experiment shows the shape of the entire event — thirty thousand attendees, north of sixty thousand conversations — a quiet trickle for weeks, then the wall of traffic when the show starts.
I want to be honest about that gap, because it's the one people gloss over. "Errors near zero" means the app almost never threw. It does not mean the answers were right. A confidently wrong session recommendation doesn't register as an error anywhere — it just quietly disappoints someone. The traces are how you go hunting for those, and every single fix in this post — the failover, the time-zone bug, the empty-schedule retry, a dozen prompt versions — started as something I noticed scrolling through them. The observability wasn't a reporting afterthought. It was the control panel I ran the conference from.
What I actually took away
The first is that governed and observable is what makes real-time possible. Everything that let me fix Brickbot while it was live in front of thirty thousand people came down to the same property: every piece was an object I could see and change on its own. A bad answer was a trace. The fix was a new prompt version and a moved alias. A rate-limit meltdown was a second endpoint and a retry. None of it was a deploy. That's not a nice-to-have when you're operating something live — it's the whole difference between a system you run and a system that runs you.
The second is how little of this is application code. The hard parts — retrieval, analytics, authorization, memory, tracing — all live in the platform as governed primitives. The thing that stitches them into a chatbot is a thin shim, a few hundred lines. Which means the shim is almost beside the point: the same primitives, wired a little differently, are a data-quality agent, or an ops copilot, or a nightly enrichment job. The chatbot is just the version of it I had to keep alive in public for four days. If you're building anything agentic, the interesting question isn't "how do I build the app" — it's "which of these primitives do I get to not build."
There's more I haven't gotten to — the memory design deserves its own post, and I still owe some thoughts on measuring semantic quality. But that's the field guide for now. If you want the fast version, the whole thing above is one trace, opened five times.
If you're building something agentic, the move is to pick one of these primitives and point it at your own problem — stand up a Genie Agent on your own tables, or point AI Search at your own docs, and see how little is left for you to build.