What you get
The dashboard, the alerts and the report all read from a data store you control. The queries behind every panel are part of the handover.
- Dashboard
- Alert rules
- Weekly report template
- Replay tooling
Services/Monitoring & Analytics for AI Systems
Know what your bot said, how fast, and how often it was right.
Once a chatbot or agent is live, the questions change: how many conversations today, how slow was the worst one, what did it cost, and how often was the answer right. GTDataWorks adds structured logging to your system and builds dashboards and alerts on top of it, for conversation volume, latency, cost, answer accuracy against a sampled rubric, and drift over time.
For a support or operations lead, that is one screen that answers how the bot is doing, and a weekly report template that pulls from the same data. For your engineers, it is session replay: pick any conversation and step through it turn by turn to see what the model saw, which tool it called and where it went wrong. Your users get the quieter benefit: regressions show up in the numbers before they show up in complaints.
The dashboard, the alerts and the report all read from a data store you control. The queries behind every panel are part of the handover.
Structured logging added to your bot or agent: one record per turn with the prompt, retrieved context, tool calls, latency, tokens, cost and the answer, written to Postgres or SQLite in your hosting, or to LangSmith or Langfuse if you already use one.
Volume, latency percentiles and cost per conversation are mechanical. Accuracy is not: a short rubric is written with your team (answered the question, cited a real source, did not promise what it cannot) and applied to a fixed sample each week, by deterministic checks where possible and by a human reviewer where not.
Grafana boards per audience, typically one for engineering and one for the support lead, and alert rules for latency, error spikes, cost ceilings and a falling accuracy score.
Session replay tooling in the style of Agent Flight Recorder: open any conversation and step through it turn by turn, with the context the model had at each step and a record of what it did with it.
A weekly report template that reads from the same tables, a walkthrough of each panel, and the query behind it. The alert rules and dashboards live in your Grafana, under your logins.
Scoped by data sources and metrics.
Ongoing reporting and dashboard changes.
Prices are in USD and are starting points. The final quote follows a scoping call.
In a Postgres or SQLite database in your own hosting, or in your LangSmith or Langfuse workspace if you already use one. Conversation data does not pass through a GTDataWorks system; the dashboards read from your store with credentials you issue and can revoke.
Whoever you give a login. The usual arrangement is one board for engineering and one for a support or operations lead, reading from the same data, with Grafana running in your hosting. Any model-provider key used for grading is yours and stays in your secrets store.
By a rubric applied to a sample. Each week a fixed number of conversations is graded against criteria written with your team, deterministically where a check can be automated and by a human reviewer where it cannot. The score is tracked over time; the trend matters more than any single week's number.
You keep the dashboard, the alert rules, the weekly report template, the replay tooling and the queries behind every panel, all in your accounts. Changes to the metrics, new data sources and ongoing reporting are available at the hourly rate on this page.
Say what is in production and where it logs today. The brief builder asks the rest, and a scoping call sets the quote.