New arrivals: 14 Scout case studies

Internal exploration · Merchant agent

We are building a merchant agent to understand what happens in an online store

Traffic, advertising, orders and emails each tell part of an online store's day, and none of them tells the whole story. We are building an agent meant to query these sources together, investigate the previous day and prepare a useful review for the merchant every morning. This internal tool is still under construction: we are currently testing it on the store of a brand we work with, through investigations that we run and check ourselves.

AIE-commerceAgent under construction

Every morning, a merchant asks the same questions. How much did we sell yesterday, and is that normal? Did the campaigns bring in the right visitors? Did anything block the path to purchase? The answers are scattered across the Shopify store, the Meta and Google ad accounts, the Klaviyo email tool and the traffic measurement tool. Each interface shows its own figures, with its own definitions and its own time windows. Reconciling them takes time, attention and a good knowledge of their pitfalls. This is the work of understanding that we want to hand over to an agent, without handing over the decisions.

The work of understanding that comes back every morning

Opening five dashboards is not enough to understand a day. Revenue tells you whether the target was met, not why it was missed. An ad platform reports conversions according to its own attribution rule, which does not necessarily match the store's orders. The email tool may claim a sale because a customer opened a message a few days before buying. Traffic can seem to collapse when it is actually the data collection that stopped.

A merchant who wants an explanation therefore has to connect these signals, check that they are comparable and separate what is measured from what is assumed. On an active store, this work starts again every day, and it is often replaced by a quick look at the most visible indicators. Our agent has to do this groundwork, then give the merchant an explanation they can check.

Questions that need several sources

The questions that matter to a merchant cut across tools. “Is this campaign bringing in buyers?” needs the spend and clicks from the ad platform, the visits the store actually received, what those visitors did there, and then the orders placed. “Where did yesterday's sales come from?” needs us to find, for each order, the ad clicks, unpaid visits and email clicks that preceded it. “Why did sales drop?” needs a review of campaign delivery, traffic quality, landing pages, checkout and email sends.

That is why we do not design the agent as a list of connectors. We organise it around these merchant questions, and we only plan read-only access to the data that documents each step of the journey: advertising, the journey on the site, sales, the offer (products, promotions, stock) and customer relations. Its rules forbid it to change an ad, a budget, an email flow, the consent settings or the store itself.

What each source can really tell us

To connect the sources properly, the agent has to know their limits. We spell them out for it.

The Shopify store is the reference for orders: how many, their amount, their status, the codes used and the accounts created. The amount shown is a running total, not net revenue collected. Shopify also keeps a partial trace of the buyer's journey, such as the first and last visits it recorded.

The Meta and Google Ads accounts describe spend, delivery, clicks and the conversions each platform credits to itself. These conversions are not unique orders: two platforms can claim the same sale, and Meta can revise its figures over several days. Every reading must therefore be dated. Google also keeps an account's change history, which makes it possible to date when a campaign was paused.

Klaviyo reports sends, opens and clicks. Its own attribution can count an order after a simple open: a click followed by a visit remains stronger evidence.

Signal, the measurement platform we are developing, records site visits, page views, interactions, where visits come from and consent status. We exclude bots and our own technical tools, which must never count as customers. Unknown consent is not a refusal, and missing data is not a zero.

Finally, the store's history makes it possible to place a day against the previous 7 and 30 days. That is often what separates an incident from a usual level that is too low.

An investigation that connects the day's events

Take a November day that we examined in detail. Sales came to a little over a third of the merchant's daily target. A quick reading would have blamed the advertising or the product pages. The investigation first established that our own traffic collection had been interrupted from the early evening of the day before until mid-morning: any comparison of that morning's traffic had to be flagged as not comparable. It then dated the manual pause of a Google campaign to around midday, after which that campaign's clicks fell to zero. It found that a Meta campaign optimised for page views brought a large share of the clicks, but that none of the visits observed on the site from that campaign had added a product to the cart. It noted that a fault in the checkout journey had been fixed early in the afternoon. Finally, it pointed out that the target had been met on only one of the previous thirty days.

None of these observations explains the day on its own. Together, they let us say what comes from measurement, acquisition, the journey and the structural level of sales, and what remains uncertain. This kind of cross-reading is what we want to make a daily routine.

Signal today, other connections possible

Today our investigations rely on Signal for traffic and consent, alongside Shopify, Meta, Google Ads and Klaviyo. While reconstructing a day's orders, we also read the journeys recorded by PostHog, an analytics tool already installed on the store: it helped identify where two otherwise unexplained orders came from. That reading was part of one investigation; it is not a permanent connection for the agent.

We are designing the agent to reason about questions and levels of evidence rather than about a specific tool. Other journey sources, such as PostHog, could therefore give it the same information on another store. This is a design possibility, not a multi-tool deployment that already exists.

What exists, what we do by hand, what we are building

We distinguish three levels. What already exists is a written investigation framework, which sets the order of checks, the levels of evidence and what is forbidden; a local test bench that replays past days and checks the agent's answers, whose results are still partial; a dated log of findings, which we keep by hand; and a mock-up of the morning review, which has never been sent. A daily summary based on aggregates has also been running since 12 November; it is a separate building block from the investigation. We have carried out several real investigations on November days, manually or semi-manually, including an order-by-order reconstruction of the sixteen sales on a Black Friday launch day. What remains to be built is the autonomous daily routine, an automatic memory of open questions and the agent sending its own morning review.

Our first lessons are mostly about method. Checking the measurement before interpreting avoids false alarms. Linking orders to their journeys makes it possible to place the sales that each platform claims. A useful agent must be able to say “I don't know” exactly where the data is missing, while still taking a position on the most likely explanation. We claim no increase in sales. What we are aiming for is a merchant who understands their day in a few minutes and knows which evidence supports the decisions they are weighing.

All case studies →

Something to dig into? Add this article to your office hours discussion.

See my discussion