What Building Three Generations of an Agentic AI Platform Taught Us

Model and hardware investment in AI is running at a scale most industries have never seen. Better inference, faster chips, bigger context windows…advances arriving quarterly or even faster. But ask most enterprises how confidently they could use that to stand up their five hundredth production agent, ensure it’s doing the right thing, and keep improving it all while keeping the infrastructure bill from outpacing the value it creates, and the answer gets far less certain. That gap, not the model or the chip, is where we’ve spent the last few years and two full rewrites of our own agentic platform called Solace Agent Mesh, which as of today you can use to start building and deploying agents for free.

We’ve been building agent software for a long time now, and we’ve rewritten our own platform twice because each iteration taught us things a whitepaper never could. This is the story of those three generations: what we built, what we learned, and why the architecture we landed on and released today is not an incremental product update but a genuine step-function improvement in what agents cost to build, run, and trust.

First Generation: The Monolithic Agent

We started building our own agents for internal use in 2023, back when “agent” was barely a settled term and there was no ecosystem to lean on. There was no Model Context Protocol for reaching into enterprise data. Tool calling was a novelty most models handled inconsistently. Context windows were small enough that you thought about every token you spent. Context caching didn’t exist. Hallucination wasn’t an edge case you tested for; it was the default behavior you tried to engineer around.

Given all that, we made a deliberate choice: to write our own harness, in Python, the hard way. We wanted to actually understand the technology, not inherit someone else’s abstractions for it, and we viewed it mostly as a learning experience to build and use, not as something we would productize. At least not yet.

What we built worked, in the way early software always works. And then we got excited and kept adding to it. More functions. More tools. More use cases patched with more prompt engineering. Within a few months we had, without really deciding to, built a monolith. One large agent trying to reason about everything, with every tool and every instruction competing for the same context window and the same shot at the model’s attention, with the typical evolution challenges of a software monolith.

But this phase was very successful: we learned a great deal about how to interact with models, context and memory management, and agent scoping, and it reaffirmed things we already knew about monolithic software implementations.

These learnings became the design brief for what came next.

Second Generation: Distributed Agents

The second generation was a full rewrite, and it was organized around a key realization: these things we’d been calling “agents” were, architecturally, microservices. They just happened to have an LLM instead of business logic sitting at their core. So we treated them that way. And we called this platform Solace Agent Mesh as it was intentionally built to support a mesh of autonomous agents.

• Decomposing the Monolith: We rewrote the monolith into a framework for building many smaller, focused agents. Each would have fewer tools, a narrower job, and a correspondingly smaller context to reason over. That alone fixed a lot of the reliability problems, for the same reason it always does in traditional software: a component that does one thing well is easier to get right than one that does twenty things adequately.
• Event-Driven Agents: Naturally, we made these agents event-driven, for the same reasons event-driven architecture has been winning for two decades in every distributed system where reliability, scalability, and burst tolerance matter, because we fully expected that the agentic future would require real-time processing. In production, these agents communicated over a Solace event broker…a proven, enterprise-grade eventing backbone.
• Google ADK and A2A: We kept the implementation in Python and adopted Google’s Agent Development Kit (ADK) as a component in our stack, betting on its roadmap. We based inter-agent communication on the data model of the Agent2Agent (A2A) protocol, augmented with our own extensions, and made that communication asynchronous. Agents exposed their capabilities to the rest of the system as A2A Agent Cards, leaning into a protocol that was rapidly gaining acceptance rather than inventing a proprietary alternative. Leveraging A2A data formats internally also made it easier to support connectivity to third-party agents via the A2A protocol since we viewed heterogeneous agent interoperability as critical functionality for enterprises.
• Orchestrator: On top of that, we added an orchestrator: a coordinating agent with dynamic agent registration, able to discover what other agents in the system could do and reason about how to decompose a user’s request into a plan across them. This is the piece that made the system feel intelligent rather than merely functional, and it worked well enough that “orchestrator plus specialist agents” quietly became the reference pattern much of the industry converged on independently.
• Smart Context Management: We also introduced what we called Smart Context Management; instead of stuffing full datasets into the model’s context window, we kept the data in external artifacts and gave the LLM only a reference, a schema, and the tools to query and manipulate it. This one change had an outsized effect on both cost and reliability, because smaller contexts are cheaper to run and easier for a model to reason over correctly, and it’s a principle we carried forward unchanged into the current architecture.
• Visualizer: Alongside the runtime changes, we built a real-time visualizer that let an agent builder watch, live or in historical replay, every interaction between an agent, the models it called, the tools it invoked, and the other agents it delegated to. This helped build understanding and trust in agent processes.
• Connectors and Entrypoints: Finally, we added connectors to real systems such as MCP, SQL, MongoDB, Slack, and a growing list of others, and multiple entrypoints so agentic processing could be triggered by an event on the bus, an API call, or a conversational message from a Web, Teams, or Slack interface. And we layered in the governance, security, and audit features that any system needs before it’s allowed anywhere near production data.

This generation worked. It solved real problems, for us and for the users who adopted it, spanning everything from conversational assistants to automated back-office processing.

But running it at scale, we realized six more opportunities for improvement:

• Compute resources were not used efficiently. Each agent ran as its own deployed executable, and most agents in a real enterprise don’t run continuously. Most fire a handful of times per minute, hour, or even day. When your customers are talking about hundreds or thousands of agents at that duty cycle, paying for a dedicated running process per agent stops being a rounding error and starts being very meaningful.
• Python’s portability and resource footprint became a real operational tax once you’re deploying at that density or even on your desktop. Every agent process carries its interpreter overhead and dependency surface with it.
• The ADK framework wasn’t keeping pace with what we needed. We found ourselves working around its assumptions more often than building on top of them, which is the tell that a foundational dependency has become a liability.
• Our plugin model for custom tools wasn’t structured and needed to be more secure. We wanted a real architectural answer for letting users extend agent behavior without opening a security hole.
• We found builders needed more tools to be successful in bringing agents to production in a structured, repeatable way. Agents needed the same discipline software has always needed: building, testing, deploying, observing, improving. We developed ideas on how we could help with that.
• We wanted to lower the floor for who could build an agent at all. You shouldn’t need to be a Python developer, even an AI-assisted one, to define an agent’s role or a workflow and put it to work. If agents are genuinely useful, they should be able to help build the next generation of agents.

Those six learnings are what sent us back to the drawing board a second time.

Third Generation: An Agent Development and (Serverless) Runtime Platform

The third rewrite kept every external surface intact—same YAML configuration model, same UI concepts—while completely replacing what ran underneath. The changes focused on three things:

• A leaner, more scalable, more cost-effective runtime that could host untrusted custom tools securely
• A workflow capability that involves both agentic steps and deterministic tool steps because (non-deterministic) agentic processing isn’t the best way to solve all problems.
• A set of features for builders covering the entire agent development lifecycle to help them go from prototype to production

The crux of the rewrite was twofold:

1. We rewrote the implementation in Go. That decision alone was worth more than we expected. Go produces smaller, faster executables with better concurrency characteristics than Python, which translates directly into a more scalable runtime and a smaller deployment footprint. That footprint is just one static binary for a simple desktop install of all components and three pods in Kubernetes for scalable production use. As a secondary but genuinely surprising benefit, we’ve found that AI coding assistants tend to generate more correct, more idiomatic Go than Python for the work we are doing, which translates directly into increased efficiency and feature velocity for our engineering team who make heavy use of AI coding.
2. We completely changed the architecture: we rationalized the whole system into three event-driven components, still connected through an event broker, and adopted a serverless architecture, meaning each agent is not deployed in its own separate executable.

As seen in this diagram, the three executables are Entrypoint Runtime, Agent Workflow Executor, and Secure Tool Runtime.

Entrypoint Runtime

The Entrypoint Runtime is where agentic processing is triggered. This component handles the various technologies an agent might be invoked from, such as web chat, Slack, an inbound event, an API call, and the authentication and security decisions that belong at that boundary, before anything downstream ever sees the request.

Agent Workflow Executor

The Agent Workflow Executor (AWE) is where the actual reasoning happens, and is a key innovation in this architecture. It’s a stateless executor, scalable both vertically and horizontally, that runs the LLM loop for all agents using their system prompts, guardrails, skills, and tool definitions. That’s because in this architecture, agents are just natural-language configuration that is held in external storage, not code on their own. The model loop, context and cache management and all the hard stuff is handled by the executor.

A single executor can run many tens of agents, bound only by available CPU, so adding another agent is just adding configuration to the executor. The executor is fed work by events from the event broker: input from an entrypoint, a completed tool call, an action triggered by a workflow, or another agent, and it references the context, memory, and artifacts it needs from external storage.

You can also think of the AWE like a CPU core and the agent definitions as programs. The AWE “swaps in” an agent to run when it has work to do in a similar way to how a task is context-switched into a CPU core to execute, then swapped out while another is swapped in. Except, of course, the AWE runs many agents in parallel, which gives the architecture the serverless aspect from an agent point of view.

Secure Tool Runtime

The Secure Tool Runtime (STR) is where an agent’s tools actually execute. Solace Agent Mesh ships with a variety of built-in tools for data manipulation, charting, and connecting to MCP servers and other systems, but builders can also write their own custom tools, in Go or Python, using a Tools SDK and AI coding Skills we provide.

Every tool runs in its own sandbox, isolated from every other tool, invoked in an event-driven manner whenever an agent calls on it. We use this in our SaaS to ensure custom tools can be run securely.

The ability to develop your own custom tools in Python or Go, with AI-assisted coding if you like, puts agent builders in full control of customizing what your agents can do in a standard way, and the ability to run them in a secure sandbox ensures that critical internal security requirements are met.

Session storage, artifacts and working context are stored outside of the tool and agent runtimes. This is what allows the AWE and the STR to be stateless, which in turn allows them to scale horizontally. This, combined with event delivery retries from the event broker, also allows seamless recovery from faults as an “in progress” process can be retried by another runtime when the broker redelivers the event.

The fact that agents are now structured configuration as opposed to source code is worth calling out. This means that it is very GitOps-friendly in terms of promotion to upper environments (and backing out the change), but that is also true for source code. A key benefit of the SAM approach is that agent configuration has a standardized schema defined by SAM.

You don’t have to read code written in different ways by different developers to see that a particular connector or tool is being used, for example. That is represented the same way for all agents based on the schema, and you don’t have to be a Python developer to understand it. Similarly, automation can be written against this agent configuration to easily point out differences, enforce policies, and more, as part of your GitOps governance.

The Go rewrite alone would have been a meaningful improvement. Combined with this serverless architecture and the ability to build your own tools and run them in a secure manner, it’s a genuinely different cost, security, and operations model, not just a faster version of the old one.

Why the cost curve actually bends

Two aspects drive the cost line in agentic systems: IaaS costs to run all these agents, and token costs to support their work, and both have received a huge boost in our third generation.

IaaS and software management costs so far seem to be widely ignored when assessing agentic systems, but it becomes critical as you add more agents. In most agent runtimes, cost tracks the number of agents you deploy, because each agent is its own process, its own deployment, its own thing to monitor, consuming its own IaaS resources. Add a hundred agents and you’ve added a hundred running things, whether or not any of them are doing much work at a given moment.

In the SAM architecture, cost tracks processing, not agent count. New runtime capacity gets added only when an existing component actually runs out of CPU. There’s an upfront cost to stand up the three components, and then—because most enterprise agents run intermittently rather than continuously—you can add a meaningful number of additional agents onto that same fixed footprint before you need to scale out again. Exactly how many depends on the underlying infrastructure and how CPU-hungry a given workload is, but supporting dozens of agents on a single runtime deployment is the common case, not the exception. Here is also where the increased efficiency of Go compared to Python really matters.

It’s also worth being precise about why this beats the other obvious answer to the utilization problem, which is “just run agents as serverless functions.” Functions-as-a-service still require you to package, deploy, and monitor each one individually. They pay a cold-start penalty that’s simply unacceptable for anything a person is waiting on in a chat window, and and they get expensive fast if a workload runs continuously rather than in short bursts. That’s why we discounted that option early on. A shared, event-driven runtime avoids all three problems at once: no cold start because the runtime is already warm, no per-agent deployment because adding an agent is a configuration change, and no continuous-cost penalty because idle agents cost nothing beyond the fixed runtime they share.

The second cost aspect is token costs resulting from LLM usage. To optimize this, we carried over something we call Smart Context Management from our second generation and improved it. Many agent harnesses just take the results from the previous LLM turn and hand that back to the LLM on the next turn. Sure, token caching can help to some degree, but a leaner, more efficient context sent to the LLM allows it to reason better and produce better results, in addition to being more cost-effective.

Rather than handing the large results from a previous tool call to the LLM, Solace Agent Mesh stores the tool result in an artifact and then provides the LLM with the artifact’s reference, schema, and tools to operate on it. From there, the LLM can instruct the agent to run a tool on the artifact given its schema (e.g., sum QuantitySold and subtotal by SKU), resulting in a new artifact rather than passing all the data into the LLM via its context. This is what keeps the context window lean to what the LLM actually needs and uses regular deterministic software for what it is best at.

Three things combine to deliver a step-function improvement in operational cost savings, the kind that gets the attention of your infrastructure and cost management teams: a shared executor whose cost scales with usage, not per agent; a more efficient runtime thanks to Go rather than Python; and Smart Context Management to reduce token costs.

The Importance of Workflows

The second generation’s orchestrator was well suited to conversational use cases, where a person is in the loop and a dynamic, reasoned decomposition of a user request into a plan makes sense. But headless processes that just need to run the same business process the same way every time don’t want an agent re-deriving that plan from scratch on every run. They want a pre-defined workflow. So in the third generation we added exactly that: the ability to define workflows that follow a specific, pre-determined sequence of steps.

Those workflows can include agentic processing nodes alongside the branches, loops, and scatter/gather patterns you’d expect from any workflow engine, with the option to hand any given node to an LLM when judgment is what’s actually needed. We built this directly into the Quick Build Agent, so a builder can describe a workflow in natural language and have Agent Mesh construct it. And Not just individual agents, but the orchestration between them.

The other half of that equation is knowing when not to use an agent. Plenty of workflow steps, like data transformations, lookup, and formatted API calls, are entirely deterministic and asking an LLM to reason its way through them only adds cost, latency, and a chance of getting it wrong. So we gave builders the ability to add deterministic processing via Tools as nodes in the same workflow, right alongside the agentic ones.

These types of workflows pair perfectly with event triggers to allow agents to react to events as they happen. This is why Agent Mesh has built-in integration with Solace Event Broker, which, for many clients, carries business events that can be used to trigger agentic processing in real time.

Using LLMs where judgment and reasoning earn their keep, and ordinary software everywhere else, covers a wider range of problems with one workflow than an agents-only approach ever could.

Treating Agent Development as a Lifecycle, not a Build Step

The final aspect of this third generation has nothing to do with runtime efficiency and everything to do with helping builders go from demos to production with real ROI. Our approach was to treat agents like what they are: a new category of software that needs the same disciplined lifecycle methodology every other category of production software needs, complicated by the fact that the core component—the model—is non-deterministic in a way traditional code never was.

The framing or mental model we’ve found most useful for thinking about this is the parallel to onboarding a new junior worker or intern:

• You hire for a well-defined role,
• You onboard them into your systems and give them access to data and tools,
• You coach them to do their job well,
• You supervise closely at first,
• You get them working as part of a team,
• You keep giving feedback long after the onboarding period ends, so they can constantly improve.

Agents benefit from exactly that structure, and treating agent development as a lifecycle rather than a one-time build is, in our opinion, the single most important element of building an agent development program that scales reliably across your organization to produce ROI.

But a methodology alone is not enough. You need development tools to help implement that methodology. That’s why we built features into Solace Agent Mesh to help builders at each phase of the build, test, deploy, observe, and improve loop. Some of these include:

• Hiring: a Quick Build agent in SAM that conversationally helps you define and build agents and workflows, so your agent has a well-defined system prompt, guardrails, and Skills, all defined as configuration to make it GitOps-friendly
• Onboarding: pre-built connectors, pre-built tools, and a Tools SDK to build your own, so your agents can access whatever data or applications (off-the-shelf or proprietary) they need, and you can implement any custom behaviors, data management, or deterministic workflow steps you need. Also, pre-built Entrypoints so your agents and workflows can be conveniently triggered by an API call, a Solace event, a chat session, and more
• Coaching: an experiment lab that supports datasets and evals for initial and regression testing, a built-in agent to suggest eval datasets for your agents, visualization so you can inspect your agent as it runs or traces from previous runs
• Supervision: built-in human-in-the-loop capabilities
• Teamwork: agents working together to solve a problem, using either dynamic orchestration or pre-defined workflows, all of which are event-driven for robustness and scale
• Improvement: observability of agent and tool execution time and token usage, online evals, and, coming soon, an Agent Self-Improvement Loop, where a built-in SAM agent helps you improve the performance, correctness, and cost of your agents on an ongoing basis.

Conclusion: What All This Means

The first generation taught us how to work with models, context, and memory. We learned hard-won lessons about scoping and attention that no whitepaper could have substituted for.

The second generation turned those lessons into an actual distributed system: event-driven agents, an orchestrator, and the beginnings of the context discipline we still rely on today, and provided the basis for operational experience with our users.

The third generation is where those lessons compound into something unmistakably better. Rebuilding the runtime in Go and collapsing it into three event-driven components means agents are no longer deployed processes at all; they’re configuration that gets swapped into a shared, serverless runtime the moment there’s work to do.

That single change decouples cost from agent count: instead of paying for a hundred idle processes, you pay for the CPU that active processing actually consumes, on a footprint that can absorb dozens of new agents before it needs to scale out.

Paired with the Smart Context Management carried over and sharpened from the second generation, the result is an architecture where IaaS cost tracks usage and a leaner context reduces token costs, letting agents reason better as a side effect of the same design that makes them cheaper to run.

What makes this a platform rather than just a faster runtime is that the same rewrite also gave builders a real development lifecycle—workflows that mix agentic and deterministic steps, and the tools to hire, onboard, coach, supervise, and continuously improve an agent the way any other category of production software gets built, tested, and hardened.

Put together, the cost model, the workflow flexibility, and the lifecycle tooling are what make this release feel less like a version bump and more like the disruptive advancement we set out to build: the point where agentic AI stops being just a demo and starts driving operational and economic solutions you can deploy to production.

And as proud as we are of this moment, we’re not done. There will be more learnings, more improvements to come. Some are already in our labs and showing great promise. We can’t wait to unveil those too as they become available.

You can learn more about Agent Mesh here, or if you’re itching to try it out download it now and start building agents!