New arrivals: 14 Scout case studies

Medusa Extend · AI

Monitoring your AI agents in Medusa

An agent that rewrites an announcement serves as the common thread for building, in Medusa, a first monitoring of its runs, their estimated cost and their quality, using Langfuse as a reference.

When a business hands a task to a language model, the questions come afterwards: what the agent did, what it cost, what its text is worth. Specialised tools already answer them. This article shows how to build a first scope of this with the Medusa framework. The example is deliberately small, and the setup described is a design, not a feature shipped by Medusa.

An announcement, an agent, some questions

Papeterie Alizé, a shop imagined for this example, wants to announce an exceptional opening. In the Admin, a team member pastes their text: “Dear friends of Papeterie Alizé, we are delighted to announce that, as the holidays approach, our shop will exceptionally open its doors on Sunday 13 December, from 10 am to 1 pm and then from 2:30 pm to 6 pm. We will be delighted to welcome you to browse, choose your cards and share a warm moment.” They add an instruction: keep the exact opening hours, adopt a warm tone, do not exceed forty words.

Theagent in charge of this task combines a language model, a system prompt and a tool. The tool is local: a function that counts the words of the text, with no model call. The flow is bounded. The model proposes a rewrite; the tool counts it; if it exceeds forty words, the model receives this result and rewrites only once; the tool counts again, and the agent returns the last text with its status. There is no dialogue: one input, one output.

In the fictional run used here, the first proposal exceeds the limit and the correction meets it. Another run may still be too long after its correction: the agent then stops and returns a text marked as such, rather than trying indefinitely.

With this case, the team's questions become more precise. It wants to find the text received and the text returned, know how many model calls took place and why, and know the approximate cost of the run. Above all, it wants to check that the opening hours survived the rewrite, which no counter tells, and to compare, on the same announcements, the current prompt with the one it is considering.

Langfuse as a reference: a run and its steps

Langfuse is a tool specialised in monitoring applications built on language models. In its data model, a trace groups what happens during an operation, and each step of that operation is an observation ; an observation can contain others, which forms a tree. Langfuse distinguishes several observation types, including the generation, which represents a model call, and the tool.

Applied to the announcement, this model describes the fictional run effortlessly. The trace contains a first generation, a call to the counting tool, a second generation for the correction, then a second count. The link from each step to its parent rebuilds the tree; the order of the steps and the result of the first count explain the correction.

From here on, we userun for the trace andstep for the observation. We keep runs, their steps, usage, costs and evaluation on reference cases; Langfuse's other tools remain outside this first module.

Theinstrumentation refers to the code that records these operations. Nothing is recorded if the call does not go through it: this remark guides everything that follows.

What to record from a model call

A generation is one step of the agent; but to carry it out, the server may send several requests to the model provider. If the first one fails on a transient error and a second one goes out, two real calls have taken place, and each may have consumed usage. Each request actually sent is therefore an attempt, attached to its step.

Three situations look alike without being equivalent. The correction is a new generation decided by the agent after the count: it is a new step. The retry repeats the same call after a failure: it is one more attempt within the same step. Finally, the replay of a write happens when the server records a second time the response to a call already made, for example after a resume: no new call took place, and this record must add nothing.

The order of writes matters. Before calling the provider, the server records the attempt as started, with its identifier. On the response, it completes it: status, request identifier returned by the provider if there is one, reported usage, generated text. If the server stops between sending and saving, the attempt remains started and its usage is unknown. This gap is visible, which is better than an absence: the team knows that a call may have consumed usage. An unknown usage is never counted as zero. And no cancellation on the server side undoes a call that has already gone out, nor its billing.

Each generated text remains attached to the attempt that returned it. The fictional run thus contains two versions of the announcement; it will be necessary to know which one was returned, and which one is being judged.

Reported usage and estimated cost

For each response, the provider reports the usage consumed, usually by category: input tokens, output tokens, input tokens served from a cache. These categories must be made mutually exclusive before any calculation. Some providers include cached tokens in the input total; adding up the figures as they are would count the same consumption twice. Langfuse sets the same requirement. It accepts a cost supplied directly or calculates it from the usage and a price definition for the model, and calculates nothing without one or the other.

The chosen design calculates an estimated cost from the reported usage and a price table that is versioned: a price per category and per model, with a validity start date. Each attempt records the price table version used. When a price changes, a new version is added; costs already estimated do not change, as in Langfuse, where updated prices apply only to new records.

The cost of a run is the sum of its attempts actually sent. The counting tool, being local, adds nothing to it. An attempt whose usage is unknown makes the total incomplete, and the display says so rather than presenting a figure that is too low. This amount remains an estimate: it does not replace the provider's invoice and does not cover the cost of running the server.

The monitoring module on the Medusa foundation

Runs, steps, attempts and price tables: these facts and their rules form the agent monitoring module. Medusa provides what is needed to build it like any other module.

The data models are declared with Medusa's modelling language, the DML, which describes properties, relationships and indexes, and from which migrations are generated. The run becomes agent_run, the step agent_step, the attempt agent_attempt, and each model holds the relationship to its parent. The attempt declares its status, the provider's request identifier, one property per usage category, the estimated cost and the price table version. The usage properties are optional: a missing value means an unknown usage, distinct from a zero usage. The module's service extends MedusaService, which generates the common operations on each model. The domain rules are written in this same service. A method that completes an attempt, for example, ignores a second completion of the same attempt, calculates the cost only if the usage is known and rejects a price table version that does not exist. The module is not linked to any commerce module: the agent, the routes and the jobs resolve it from Medusa's container when they need it.

Medusa already keeps, for its part, another kind of record. Its workflow engine tracks transactions and the status of each step in order to orchestrate execution. The history of these executions is not kept by default, but its storage can be enabled and viewed in the Admin settings. This record describes coordination: which step ran, failed, was retried. It does not by itself calculate the usage consumed at the provider, its cost or the quality of the text. The two records complement each other without replacing each other.

Running the agent and recording each call

Everything starts from an Admin API route, called from the page where the team pasted its text. The route launches the agent's workflow and returns the run's identifier.

The workflow opens the run in the module, with the configuration version used. It then chains the generation, the count, the correction if needed, then the closing, which records the returned text and its status: within the limit, or too long. The limit of one correction is a configuration rule, applied by the workflow, and not an instruction left to the model's discretion.

Each generation goes through the instrumentation: a thin layer around the provider's client, which every model call passes through. It is this layer that opens the attempt before sending and completes it on the response, through the module's service. Some clients retry a failed call themselves, without the layer around them seeing these extra requests; the OpenAI Node client, for example, makes two by default for certain errors. These internal retries are therefore disabled, so that the instrumentation drives and counts each attempt. The count, for its part, is recorded as a tool step, with no attempt.

The workflow engine can retry a step after a failure, depending on its configuration; a retried step can even start while the previous one is still running. The instrumentation handles this with the rules already set out: writing a record is deduplicated by the attempt's identifier, and any request actually sent again to the provider opens a new attempt. A maximum number of attempts per generation also caps spending. The instrumentation does not detect a call that bypasses it: tests check that every call made by the agent goes through it.

Once the run is closed in the module, secondary processing can follow. An event marking the end of a run triggers a subscriber that updates the totals per agent and per version. Emitting an event does not prove that collection is durable: the truth remains in the module, and the subscriber only derives aggregates from it. A scheduled job applies the retention period: the received and generated texts, which may contain sensitive information, are deleted earlier than the metrics. No record contains a provider access key.

Judging the generated text

A run without errors can return a bad announcement. The technical status says that the agent worked; it does not say whether Sunday 13 December and its opening hours survived.

So the module adds one fact: theevaluation. It applies to a specific text, the one the agent returned, identified by its attempt, and not to the run in general, which contains two versions of the announcement. It records a criterion, a result, its author and its date. Respecting the forty words is an automatic check: the code counts, and the result is recorded as an evaluation whose author is the check. Whether the opening hours are kept is judged by a team member, who answers yes or no in the Admin. Until someone has judged, the state is “not evaluated”, not “passed”.

In this fictional example, the text returned after correction would be: “Papeterie Alizé welcomes you exceptionally on Sunday 13 December, from 10 am to 1 pm and from 2:30 pm to 6 pm. Come in to choose your cards and enjoy a warm moment.” The length check counts 33 words, as groups separated by spaces, and records a result that complies with the forty-word limit. The date, “Sunday 13 December”, and the two time slots, “10 am to 1 pm” and “2:30 pm to 6 pm”, appear in it as announced; the form will let the person confirm this criterion, and the state remains “not evaluated” until then.

Langfuse likewise attaches its scores to a trace or an observation, in numeric, categorical or boolean form.

To compare two prompts, they have to be tried on the same texts. A few fictional announcements, each with its expected opening hours, form reference cases. A configuration version identifies the choices actually used by the agent: the model, the system prompt, the version of the agent's code, that of the counter, and the limits on corrections and attempts. The team runs the cases on the current version, then on the candidate version; these runs are marked as trials so that they do not mix with everyday use. It then compares, case by case, the evaluations and the estimated costs. For the comparison to hold, both versions are judged with the same reference evaluation grid, distinct from the check built into the agent, which may itself change from one version to the next.

Langfuse versions its datasets and recommends comparing experiments with the same versions of cases and evaluators. Fixed data does not, however, make the outputs identical from one trial to the next, and a bounded correction can still fail: the final status and the evaluations show it. The module records the version used; it does not thereby become a tool for managing and deploying prompts.

In the Admin, and beyond the module

The merchant views their agents on a dedicated Admin page. Medusa lets you add UI routes with their navigation entry, built with Medusa UI components. The “AI Agents” page lists the runs, filterable by agent, version, date and status. The detail of a run shows the received text and the prompt, the tree of steps, each attempt with its known or unknown usage, the estimated cost and its price table, the returned text and its evaluations, with the form asking whether the opening hours are kept. A third view puts two configuration versions side by side on the reference cases. These screens are a design proposal, not a screenshot of a tool in service.

The /admin custom routes require authentication by default. Being logged in to the Admin does not say who can read the texts, export them or evaluate them. This policy is added in the routes, on the server side, because hiding a button in the interface protects nothing.

It remains to position this module against the tool that served as its reference. Self-hosted Langfuse relies on a web application and workers, PostgreSQL for transactional data, ClickHouse for traces, observations and scores, Redis for queues and object storage for raw events. On ingestion, the event is first stored in object storage, a reference goes into a queue, and the workers then write it to ClickHouse. This architecture separates receiving events from processing them, and entrusts analytics to columnar storage, designed to aggregate large volumes.

The module described here has neither an ingestion buffer nor an analytics engine. It writes its records as the calls happen into the application's transactional database, the one that also serves the shop, and computes its aggregates on that same database. Each record there is one more write, and an analysis over months of records competes with day-to-day work. Within a chosen scope, this is not a flaw; they are simply not the same properties.

The choice of a specialised tool therefore depends on needs more than on a threshold. The volume of records and the load of analyses on the shared database are among these needs; search across several applications, alerts, prompt management, evaluation tools or integrations are others, and can justify Langfuse from the start. The two approaches are not mutually exclusive: the instrumentation layer can also feed a specialised tool, provided the adapter that translates its records is designed.

For Papeterie Alizé, the module provides a piece of data to answer each of the initial questions: the run and its two versions of the text, the attempts and their usage, the estimated cost and its price table, the evaluation of the opening hours, the comparison of versions. A second agent can reuse the same module: it declares its configuration, records its input, its steps and its output, then routes its model calls through the instrumentation. The evaluation criteria adapt to its task. The monitoring structure remains shared.

Something to dig into? Add this article to your office hours discussion.

See my discussion