“That just doesn't happen”.
A quarter of the visits came from Safari on Windows. Apple stopped making Safari for Windows in 2012. That detail kicked off one of the investigations we ran with the StreamMyData AI agent. Here's what it found — and what, in my view, an AI agent has to be like for you to trust it.
The case is anonymous, as in the original on the StreamMyData website (in Russian): the client is a national construction-sector manufacturer, the contractor isn't named. I briefly showed this case at PMF; here it is in detail.
What we're building
At StreamMyData my team and I are building an AI agent for marketing management. It's connected to the client's data — ad accounts, web analytics, CRM — and answers questions about it in plain words, not spreadsheets. One of its functions is spotting traffic that doesn't look like real people.
Reports that were too good
The client judged ad contractors by behavioral metrics: time on site and pages per visit. The previous contractor missed the KPI, so the client switched. The new one launched three banner CPC sources — call them A, B and C — and within a couple of weeks the reports were exemplary: 2.4 pages per visit, 2 minutes 25 seconds on site, almost no bounces.
Three different ad networks have different placements and audiences, so their metrics usually diverge. Here they matched to a decimal. The team sent the agent a single message that ended with: “that just doesn't happen”.
The new contractor's sources (A, B, C)Rest of site traffic
Eight minutes of agent work
The agent ran the investigation in 8 minutes 4 seconds and 48 data queries. Without it, that's exporting raw analytics data, stitching 279,000 hits to visits and dozens of SQL queries by hand. What it found:
- Every visit is a new person. 80,215 clientIDs for 80,216 visits: in two weeks one visitor came back. 99.99% of new-user visits had no referrer, versus 65.3% in the rest of the traffic.
- Four device templates for 80,000 visits. With identical shares across all three sources. A quarter were Safari on Windows, while mobile Safari, the main iPhone browser, barely appeared.
- Goals on a schedule. The first goal fired at exactly the 30th second of a visit in 72% of cases — the same for all three networks.
- A shared network. A common pool of IP prefixes across the three sources, identical bursts of visits in the same minutes.
The agent's conclusion: the three ad sources were one artificial stream of visits under three labels. It assembled the evidence and prepared an appeal to Yandex's anti-fraud team.
What an AI agent you can trust looks like
What matters to me in this case isn't the evidence so much as how the agent worked. These are exactly the principles we build into the product.
- It says what it's missing. Twice during the investigation the agent asked for new data — hits, IP addresses, UTM tags — and explained why it needed them for proof.
- It separates facts from hypotheses. The agent listed separately what could be stated with confidence, what couldn't, and what data was missing. Which goal fires at the 30th second, it proposed to verify instead of guessing.
- It takes you to action. Not “here are some charts” but a ready appeal to Yandex anti-fraud plus a short version for the form.
- It's transparent. Each of the 48 queries can be opened and checked by hand. The agent searches — the human decides.
A lesson for business owners
If you pay a contractor for a metric, you'll get the metric. The client judged advertising by time on site — and time on site became perfect. Behavioral KPIs are useful, but only alongside money: leads, sales and traffic quality checks. And metrics that match across different sources to a decimal aren't success — they're a reason to check.
Full case breakdown with all the evidence on the StreamMyData website (in Russian) ↗