Six Lessons from Leading a Team of Three on Agentic AI

Let code do the maths, budget your context, build the safety layer first, make long jobs resumable and price every answer: what building enterprise agentic AI taught me.

2026-11-22
4 min read

Share

For the last stretch of my time at EY I have led a team of three building generative and agentic AI for large enterprise clients. The domain and the client are confidential, so this post keeps to the engineering. These are the lessons I would hand to anyone starting a similar system.

1. Never let the model do the arithmetic

Language models are fluent, and that makes their arithmetic mistakes easy to miss. In a batch pipeline that explains recommendations in plain English, we computed every derived number (gaps, deltas, the options that qualified) in Spark first, then gave the model a compact set of labelled fields. The model's job became explaining, not calculating. That one decision removed a whole class of errors and made the output checkable against the data.

2. Context is a budget, so spend it deliberately

Our multi-agent system hit hard payload limits when too much context was passed between agents. The fix was a tiered design:

  • a small digest for routing,
  • a bounded hand-off for the specialist agents, with a hard cap on size and on table rows,
  • and a full grounded synthesis only at the end.

The builder chooses the files relevant to each question instead of sending everything. Oversized-request failures went to zero, and answers stopped citing files that did not exist, because the model was no longer drowning in them.

3. Treat guardrails as part of the product

Before any question reaches an agent it passes a five-stage check: prompt-injection defence, redaction of personal data and secrets, read-only enforcement, and matching against the domain catalogue. After the answer, a reflection step scores how well it is grounded in the data and retries low-confidence queries. Users never see this layer, which is the point. It is also the part a client's security and risk teams will ask about first.

4. Make long jobs resumable

The batch pipeline covers more than 150,000 recommendations per run. We split the work into count- and size-aware chunks and checkpointed progress in Delta Lake, so an interrupted run resumes without resubmitting finished chunks. A single run completed 99.995% of its requests within the 24-hour window, with automated retry for the rest. Reliability here came from boring engineering, not from a better prompt.

5. Cost is a design input, not a surprise

We modelled tokens per request before the first production run and found where the money goes. Reusing the long, stable part of the prompt through caching cut input cost by about 77%. Routing simple questions down a fast path avoided paying for a multi-agent answer when one agent was enough. If you cannot say what one answer costs, you cannot defend it to a client.

6. Document the design, and keep the team small and clear

The leadership lessons are plainer than the technical ones:

  • Write the design down. Technical design documents let a team of three move in parallel, and let stakeholders challenge ideas before code exists.
  • Invite the platform teams early. Working directly with the Microsoft and Databricks technical teams meant platform constraints were designed around together rather than discovered later.
  • Validate on a small sample first. Rules for the prompt were tested on a stratified sample of 50 rows before any full run.
  • Own the architecture end to end. One person holding the whole picture makes trade-offs faster, as long as the rest of the team can see and question it.

The short version

Let code do the maths, budget your context, build the safety layer first, make long jobs resumable, price every answer, and write things down. None of it is glamorous, and all of it is the difference between a demo and a system a client can rely on.

Agentic AI
Generative AI
Multi-agent systems
Guardrails
LLM cost
Team leadership
Azure

You may also like