--- title: "Monitor Every GenAI Token with Grafana and Azure Monitor" description: "See which model and tool drove your AI agent bill by sending OpenTelemetry GenAI spans to Application Insights and charting tokens in Grafana." canonical: "https://adamtheautomator.com/grafana-genai-token-monitoring/" --- # Monitor Every GenAI Token with Grafana and Azure Monitor > See which model and tool drove your AI agent bill by sending OpenTelemetry GenAI spans to Application Insights and charting tokens in Grafana. Source: https://adamtheautomator.com/grafana-genai-token-monitoring/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![Monitor Every GenAI Token with Grafana and Azure Monitor](https://adamtheautomator.com/wp-content/uploads/publisher/4065d9c85b2b8124bb8ffcfe52ab0a56/e3f2c8a929b4cd4302666c23d7e4ed4bd8e5d28c26549bc97ef7d092e7a65072.webp) # Monitor Every GenAI Token with Grafana and Azure Monitor [![](https://secure.gravatar.com/avatar/d0b9d42e21e5622713f8b693aa5c0f9244d5f7dd200ed29b8398f52dee5de337?s=192&d=mm&r=g)Adam Bertram](https://adamtheautomator.com/author/adam-bertram/)9 November 202614 min. read Categories: [IT Ops](/category/it-ops/) Tags:[Grafana](/tag/grafana/)[Azure Monitor](/tag/azure-monitor/)[AI](/tag/ai/)[Monitoring](/tag/monitoring/) Table of Contents * [What the GenAI conventions put on a span](#what-the-genai-conventions-put-on-a-span) * [The attributes](#the-attributes) * [How the telemetry reaches Grafana](#how-the-telemetry-reaches-grafana) * [What you need before you start](#what-you-need-before-you-start) * [Step 1: Instrument the agent](#step-1-instrument-the-agent) * [Step 2: Run the collector with the Azure Monitor exporter](#step-2-run-the-collector-with-the-azure-monitor-exporter) * [Step 3: Confirm spans reach Application Insights](#step-3-confirm-spans-reach-application-insights) * [Step 4: Connect Grafana to the resource](#step-4-connect-grafana-to-the-resource) * [Step 5: Build the panels](#step-5-build-the-panels) * [Token spend by model](#token-spend-by-model) * [Latency by operation](#latency-by-operation) * [Error rate by operation](#error-rate-by-operation) * [The tool drill-in](#the-tool-drill-in) * [The panel model](#the-panel-model) * [When the panels come back empty](#when-the-panels-come-back-empty) * [Check Data-Source Identity and Sampling](#check-data-source-identity-and-sampling) * [Sampling, redaction and cost](#sampling-redaction-and-cost) * [Keeping cardinality and retention under control](#keeping-cardinality-and-retention-under-control) * [FAQ](#faq) * [Is an Azure OpenAI resource required?](#is-an-azure-openai-resource-required) * [Which instrumentation package should you install?](#which-instrumentation-package-should-you-install) * [Why do token totals sometimes look inflated?](#why-do-token-totals-sometimes-look-inflated) * [Wrapping up](#wrapping-up) # Monitor Every GenAI Token with Grafana and Azure Monitor Suppose your agent’s model bill doubles this month, and the only dashboard you have shows a flat line of successful requests. So which tool made the expensive calls, and how many tokens did each one burn? Your current application performance monitoring (APM) dashboard answers fewer of those questions than you’d like. Let’s say a support agent handles 40,000 conversations a month across two models and three tools. The invoice jumps, you open your APM tool, and every span reports 200 OK in about a second. Nothing in that telemetry says `gpt-4o`, 8,400 input tokens, or `get_order_status`. Nothing you can query records what each call cost. OpenTelemetry’s GenAI semantic conventions fix that by putting the model, the token counts, the tool name and the operation on the span itself. Once the data reaches [Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/opentelemetry-options), [Grafana dashboards](https://learn.microsoft.com/en-us/azure/azure-monitor/app/grafana-dashboards) turn those spans into cost and latency panels. ## What the GenAI conventions put on a span The conventions are a published attribute vocabulary for generative AI work, and they live in their own repository, [semantic-conventions-genai](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md). The [older docs page](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) is a stub pointing at the new home. Everything there carries Development status, which is the level OpenTelemetry uses in place of “experimental”. None of it is stable, so pin the instrumentation version and review dashboard queries the way you review code. ### The attributes A small set of attributes carries almost every panel worth building. The [attribute registry](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/registry/attributes/gen-ai.md) is the authority for names and requirement levels. | Attribute | What it gives you | | --- | --- | | `gen_ai.operation.name` | The kind of work: `chat`, `execute_tool` or `invoke_agent`. This separates a tool call from a model call. | | `gen_ai.provider.name` | The model provider: `openai`, `azure.ai.openai` or `aws.bedrock`. | | `gen_ai.request.model` and `gen_ai.response.model` | The model requested and the one that answered: `gpt-4o` and `gpt-4o-2024-05-13`. | | `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` | The token counts the model reported, including cached input and reasoning output. | | `gen_ai.usage.cache_read.input_tokens` | Cached input tokens, counted inside the input count. | | `gen_ai.agent.name` and `gen_ai.agent.id` | The agent that ran. Keep the name low cardinality. | | `gen_ai.conversation.id` | The session or thread that ties separate spans into one conversation. | | `gen_ai.tool.name` and `gen_ai.tool.call.id` | Which tool the agent called, and which individual invocation it was. | `gen_ai.provider.name` is the current convention, and [PR #2046](https://github.com/open-telemetry/semantic-conventions/pull/2046) records the July 2025 replacement of deprecated `gen_ai.system`. The values moved too, so `az.ai.openai` became `azure.ai.openai` and `xai` became `x_ai`. That current target is not a guarantee about automatic instrumentation. In the demo-tested stack, the automatic OpenAI child span emitted `gen_ai.system=openai` and no `gen_ai.provider.name`, even with `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`. Keep `gen_ai.provider.name` on manually created spans, inspect the child spans your pinned instrumentor emits, and use a compatibility expression in any component-scope KQL that reads the provider: ```text | extend provider = coalesce( tostring(customDimensions["gen_ai.provider.name"]), tostring(customDimensions["gen_ai.system"]) ) ``` At workspace scope, use the same `coalesce()` expression with `Properties` in place of `customDimensions`. A panel copied from a newer or older dashboard can otherwise return an empty graph after an instrumentation change, with no query error to chase. Token math has a similar trap. The registry states that `gen_ai.usage.input_tokens` “SHOULD include cached tokens and input tokens of all modalities”, so a call with 100 text tokens, 40 of them cached, plus 200 image tokens reports 300 input tokens and 40 cache-read tokens. Add those two together and you overstate usage by 13 percent on that call, and by whatever the cache hit rate is across the fleet. Any cost panel built on that sum inflates spend on every cached call. * * * _NOTE: Metrics exist for this vocabulary, but they are Development-status and were reorganized once already. Derive token and latency numbers from spans in KQL (Kusto Query Language) until you have confirmed what your exporter actually emits._ * * * ## How the telemetry reaches Grafana Your agent emits spans in the OpenTelemetry Protocol (OTLP) format, and two ingestion routes are worth knowing. The [Microsoft OpenTelemetry distro](https://github.com/microsoft/opentelemetry-distro-python) exports straight to Application Insights from inside the process. The second route sends spans to an [OpenTelemetry Collector](https://learn.microsoft.com/en-us/azure/managed-grafana/grafana-opentelemetry-app-insights), where the `azure_monitor` exporter forwards them to the same resource. Both are generally available. The Microsoft Learn guide for this pipeline states the point outright: “The dashboards work with either path because data lands in the same Application Insights / Log Analytics tables.” Choose the collector when several processes report, or when you want one place to redact and batch before data leaves the network. The diagram below traces both hops. ![Agent telemetry pipeline](https://adamtheautomator.com/wp-content/uploads/publisher/6bd58bdf6983ec5f8dfa818a64fbf96b7572b59e8633649bc3a54234939d8c41.jpg) Inside Application Insights, GenAI spans are dependency telemetry. They land in the `dependencies` table with the AI attributes stored in the dynamic `customDimensions` bag. The Log Analytics workspace underneath names the same table `AppDependencies` and the same bag `Properties`. That’s one dataset with two column names, and the mismatch between them causes more blank panels than every authentication problem combined. ## What you need before you start * An Azure subscription with an Application Insights resource. * Python 3.10 or later in a virtual environment, with the `openai` package installed. * [Grafana 12.3 or later](https://grafana.com/docs/grafana/latest/datasources/azure-monitor/), or Azure Managed Grafana on that version. Microsoft’s guide documents 11.6 as the floor, so check that the Azure Monitor data source shows a **Traces** query type. * Docker, if you take the collector route, and an OpenAI-compatible model endpoint. Assign `Reader` at the subscription scope when Grafana must discover resources across that subscription, because [the data source reads resource metadata as well as log data](https://grafana.com/docs/grafana/latest/datasources/azure-monitor/configure/). For a narrower deployment, assign the role only on the resource group or resources Grafana must query, then verify that resource selection and Logs queries both work. This walkthrough requires Python 3.10 or later and uses the demo-tested `microsoft-opentelemetry` 1.6.0, `openai` 3.46.0 and `httpx` 0.28.x package set. Grafana 12.3 is the tested target, and the panel JSON also matches the query model that passed provisioning on Grafana’s [11.6.3 Azure Monitor data source](https://github.com/grafana/grafana/blob/v11.6.3/public/app/plugins/datasource/azuremonitor/dataquery.gen.ts). ## Step 1: Instrument the agent Install Microsoft’s newer distro rather than the older package most samples still show. `microsoft-opentelemetry` [auto-instruments `openai`, `langchain`, `agent_framework` and `semantic_kernel`](https://github.com/microsoft/opentelemetry-distro-python) when they are importable, and it reads the connection string from the environment. `azure-monitor-opentelemetry` still works and is what the Agent Framework tutorials use, so either one runs; the difference is what gets instrumented for free. ```bash python -m venv .venv source .venv/bin/activate pip install \ "microsoft-opentelemetry==1.6.0" \ "openai==3.46.0" \ "httpx>=0.28,<1" export APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://eastus.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/" ``` This is the demo-tested package set for the walkthrough. Retest the emitted model span before changing any pin. The distro handles the model call, while the agent wrapper is yours to write because only your code knows which agent ran and which conversation it belonged to. ```python from microsoft.opentelemetry import use_microsoft_opentelemetry from openai import OpenAI from opentelemetry import trace use_microsoft_opentelemetry( enable_azure_monitor=True, # reads APPLICATIONINSIGHTS_CONNECTION_STRING enable_sensitive_data=False, # disables Agent Framework sensitive data capture ) tracer = trace.get_tracer("support-agent") client = OpenAI() def answer(question: str, conversation_id: str) -> str: attributes = { "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": "SupportAgent", "gen_ai.agent.id": "support-agent-v3", "gen_ai.provider.name": "openai", "gen_ai.conversation.id": conversation_id, } with tracer.start_as_current_span( "invoke_agent SupportAgent", attributes=attributes, ) as span: response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": question}], ) span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens) span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens) return response.choices[0].message.content ``` Span naming follows the agent convention, `invoke_agent` followed by the agent name. Set the sampling attributes when the span starts, because a sampler that runs before the name is attached drops runs with no pattern you can explain later. * * * _WARNING: Never estimate token counts from message length. The registry expects the numbers the model or framework reported, and an estimate produces a cost panel that disagrees with the invoice while looking perfectly consistent._ * * * ## Step 2: Run the collector with the Azure Monitor exporter The [collector configuration below](https://learn.microsoft.com/en-us/azure/managed-grafana/grafana-opentelemetry-app-insights) accepts OTLP on gRPC and HTTP, then forwards traces, metrics and logs to Application Insights. It becomes the only component holding the connection string. ```yaml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 exporters: azure_monitor: connection_string: "InstrumentationKey=;IngestionEndpoint=https://.in.applicationinsights.azure.com/;LiveEndpoint=https://.livediagnostics.monitor.azure.com/" service: pipelines: traces: receivers: [otlp] exporters: [azure_monitor] metrics: receivers: [otlp] exporters: [azure_monitor] logs: receivers: [otlp] exporters: [azure_monitor] ``` Run the OpenTelemetry Collector from the `contrib` image, because the Azure Monitor exporter is not in the core distribution. ```bash docker run -d --name otel-collector --restart unless-stopped \ -p 4318:4318 -p 4317:4317 \ -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml \ otel/opentelemetry-collector-contrib:0.164.0 ``` That tag is the demo-tested Collector version. Retest the configuration and live forwarding before moving the pin. Then point the application at it with `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` and drop `enable_azure_monitor=True` from the Python call. The distro enables OTLP export whenever an `OTEL_EXPORTER_OTLP_*` endpoint variable is set, so the exporter moves without touching application logic. ## Step 3: Confirm spans reach Application Insights Run this in the Application Insights resource before opening Grafana. It answers one question, whether anything with GenAI attributes has arrived at all. ```text dependencies | where timestamp > ago(1h) | where customDimensions has "gen_ai.agent.name" | project timestamp, name, customDimensions | take 20 ``` Rows here mean ingestion works. An empty result means the problem is upstream of Grafana, and no amount of dashboard editing will fix it. The **Agents (Preview)** view reads the same data and gives a fast visual confirmation. Microsoft documents the minimum span contract that populates it, which is an `invoke_agent` span carrying `gen_ai.agent.name`, `gen_ai.agent.id`, `gen_ai.conversation.id` and the usage token attributes ([tutorial](https://learn.microsoft.com/en-us/azure/app-service/tutorial-ai-agent-monitoring-dotnet)). It groups runs by agent, drills into tool calls and models, and has an **Explore in Grafana** action that opens prebuilt dashboards against the same resource. A healthy run looks like the one below. ![Inspect one agent run](https://adamtheautomator.com/wp-content/uploads/publisher/80438fdd05eb5079d8aa48a41e80b7e5cecd243a3a1d7c489f0607f79d0dc019.jpg) ## Step 4: Connect Grafana to the resource The [Azure Monitor data source](https://grafana.com/docs/grafana/latest/datasources/azure-monitor/configure/) ships inside Grafana, so there’s no plugin install. Provision it as code to keep the tenant, subscription and credential out of a browser session. ```yaml # provisioning/datasources/azure-monitor.yaml apiVersion: 1 datasources: - name: Azure Monitor uid: azure-monitor type: grafana-azure-monitor-datasource access: proxy jsonData: azureAuthType: clientsecret cloudName: azuremonitor tenantId: clientId: subscriptionId: secureJsonData: clientSecret: version: 1 ``` Every panel in the next step points at the `uid` `azure-monitor`. Keep the provisioned data-source UID and every panel’s `datasource.uid` identical. If a panel is empty or reports that the data source cannot be found, compare those values before changing the KQL. Other authentication methods have their own YAML, including `clientcertificate`, `msi`, `workloadidentity` and `currentuser`. Managed identity needs Grafana on Azure infrastructure, so it is unavailable from Grafana Cloud and needs `managed_identity_enabled = true` under `[azure]` in `grafana.ini`. * * * _PRO TIP: If you have customized `forward_settings_to_plugins` under `[azure]`, add `grafana-azure-monitor-datasource` to that list. Without it, a valid managed identity returns 401 Unauthorized and sends you hunting through role assignments._ * * * ## Step 5: Build the panels The [**Traces** query type](https://grafana.com/docs/grafana/latest/datasources/azure-monitor/query-editor/) is a trace browser for Application Insights resources. It accepts an operation ID and filters on event properties; it doesn’t aggregate. Every token, latency, count and error panel belongs in a **Logs** query. Use Traces for one panel only, the drill-in that shows a single agent run end to end. | Panel | Query type | What it answers | | --- | --- | --- | | Input and output tokens per hour by model | Logs | Which model drives spend, and when it changed | | Latency p50 and p95 by operation | Logs | Whether the agent slowed down, or just the tools inside it | | Error rate by operation | Logs | Where failures cluster across chat, tool and agent spans | | Tool call breakdown | Logs | Which tool runs most, and which one fails | | One agent run | Traces | What happened inside a single bad conversation | ### Token spend by model Start with the token panel, which is where spend shows up by model. This query uses Grafana’s `$__timeFilter` and `$__interval` macros so the panel follows the dashboard time picker instead of a hardcoded window. In your environment, run the query in **Explore** before building the panel. A table result proves that your query, credentials and data source agree, and a wrong number is easier to spot in a grid than on a line chart. The walkthrough validation has a precise boundary. All four KQL queries passed directly against Azure, and the data source plus dashboard provisioning passed on Grafana 12.3.0 and 11.6.3. Authenticated query execution through Grafana was not completed because no valid Microsoft Entra service-principal credential was available. The query below is Azure-tested, but this run did not produce a successful Grafana Explore result. ```text dependencies | where $__timeFilter(timestamp) | where customDimensions["gen_ai.operation.name"] == "chat" | summarize inputTokens = sum(toint(customDimensions["gen_ai.usage.input_tokens"])), outputTokens = sum(toint(customDimensions["gen_ai.usage.output_tokens"])) by bin(timestamp, $__interval), model = tostring(customDimensions["gen_ai.request.model"]) | order by timestamp asc ``` ![Expected token query layout](https://adamtheautomator.com/wp-content/uploads/publisher/351e0c12dbea41f03b10ff69cf99df1375c364f12a49a6b51d1ac2e9fa2e73e9.jpg) ### Latency by operation Latency comes next, split by operation so a slow tool call does not hide inside a fast model call. ```text dependencies | where $__timeFilter(timestamp) | where isnotempty(customDimensions["gen_ai.operation.name"]) | summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), calls = count() by bin(timestamp, $__interval), operation = tostring(customDimensions["gen_ai.operation.name"]) | order by timestamp asc ``` ### Error rate by operation Failures belong in their own panel, because a rising error rate on `execute_tool` points at a different team than a rising error rate on `chat`. ```text dependencies | where $__timeFilter(timestamp) | where isnotempty(customDimensions["gen_ai.operation.name"]) | summarize total = count(), errors = countif(success == false), errorRate = round(100.0 * countif(success == false) / count(), 1) by operation = tostring(customDimensions["gen_ai.operation.name"]) | order by errorRate desc ``` ### The tool drill-in The tool breakdown needs an operation ID from one run, which is why it works as a drill-in rather than a time series. Take the `operation_Id` from a row in the trace view and paste it into a dashboard variable. ```text dependencies | where operation_Id == "$operationId" | where customDimensions["gen_ai.operation.name"] == "execute_tool" | project timestamp, duration, success, toolName = tostring(customDimensions["gen_ai.tool.name"]), toolType = tostring(customDimensions["gen_ai.tool.type"]), toolCallId = tostring(customDimensions["gen_ai.tool.call.id"]) | order by timestamp asc ``` ### The panel model The panel JSON below is the token chart in full. The query type is the literal string `Azure Log Analytics`, the resources list is an array, and `resultFormat` accepts `logs`, `table`, `time_series` or `trace`. ```json { "type": "timeseries", "title": "Input tokens per hour by model", "datasource": { "type": "grafana-azure-monitor-datasource", "uid": "azure-monitor" }, "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, "targets": [ { "refId": "A", "queryType": "Azure Log Analytics", "azureLogAnalytics": { "resources": [ "/subscriptions//resourceGroups//providers/microsoft.insights/components/" ], "query": "dependencies\n| where $__timeFilter(timestamp)\n| where customDimensions[\"gen_ai.operation.name\"] == \"chat\"\n| summarize totalInput = sum(toint(customDimensions[\"gen_ai.usage.input_tokens\"])) by bin(timestamp, $__interval), model = tostring(customDimensions[\"gen_ai.request.model\"])\n| order by timestamp asc", "resultFormat": "time_series", "dashboardTime": true } } ], "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] } } ``` Don’t hand-write the remaining panels. Build them in the UI, copy the JSON model from **Dashboard settings**, strip the `id`, keep the `uid`, and commit the file. ```yaml # provisioning/dashboards/default.yaml apiVersion: 1 providers: - name: 'ata-genai' orgId: 1 type: file disableDeletion: false updateIntervalSeconds: 30 allowUiUpdates: false options: path: /var/lib/grafana/dashboards foldersFromFilesStructure: true ``` The `foldersFromFilesStructure` setting requires `folder` and `folderUid` to be unset. The file above follows that contract, so Grafana derives dashboard folders from the directory structure. The finished dashboard should look roughly like the one below. ![Four-panel token dashboard](https://adamtheautomator.com/wp-content/uploads/publisher/8f59e9010b99519658b10f41b3b44cb7a811e26b1df75e26b40909b4b34186d4.jpg) ## When the panels come back empty **The query runs in the portal but not in Grafana.** This is the `customDimensions` against `Properties` problem. [`AppDependencies`](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/appdependencies) has a `Properties` column and no `customDimensions` column, so a query copied from the Application Insights resource fails silently when the data source points at the Log Analytics workspace. Change the table to `AppDependencies`, change the bag to `Properties[...]`, swap `timestamp` for `TimeGenerated`, and use `$__timeFilter(TimeGenerated)`. ```text AppDependencies | where $__timeFilter(TimeGenerated) | where Properties has "gen_ai.agent.name" | project TimeGenerated, Name, Properties | take 20 ``` `Properties` is already a dynamic property bag, so access `Properties["gen_ai.operation.name"]` directly. Calling `parse_json(Properties)` is unnecessary because Kusto passes an already-dynamic value through as-is. ### Check Data-Source Identity and Sampling **Every panel is empty and the take 20 query worked.** Check the resource URI first, then compare the provisioned data-source `uid` with every panel’s `datasource.uid`. If Grafana reports that the source cannot be found or a panel remains empty, fix that mismatch before changing the KQL. **A test run is missing entirely.** The current Microsoft OpenTelemetry Distro defaults to rate-limited sampling at five traces per second for Azure Monitor export, so test spans can be dropped. During validation, use an always-on sampler or a fixed-percentage sampler set to 1.0. Once the panels populate, restore the production sampling policy and switch aggregations from `count()` to `sum(itemCount)`, because `itemCount` carries the number of events each stored row represents. A count on sampled data understates traffic and does it consistently enough to look plausible. **The token panel is flat zero.** Check for `gen_ai.usage.input_tokens` on a raw row. A hand-written agent span often has the model call inside it but no usage attributes on the parent, so a query that filters on `invoke_agent` finds a span with no tokens. ## Sampling, redaction and cost Content capture is off by default and should stay off unless you have a reason. The conventions say instrumentations should not record instructions, inputs or outputs without an opt-in, and the registry warns that `gen_ai.input.messages`, `gen_ai.output.messages` and `gen_ai.tool.definitions` are likely to contain personal data. The sanctioned patterns are recording nothing, recording the message attributes on the span, or storing content elsewhere and recording a reference. For most regulated environments the third option is the only one that survives a review, because the prompts stay in storage you already govern. Content controls differ by instrumentation. Keep `enable_sensitive_data=False` for Agent Framework instrumentation, and leave `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` unset unless you deliberately want the OpenAI instrumentor to record message content. The tested OpenAI child span still emitted `gen_ai.system=openai` with `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`, so that semantic-convention opt-in is not a content-control substitute or proof of the current provider field. Inspect a raw exported span before assuming one switch governs every auto-instrumented library. ### Keeping cardinality and retention under control Keep `gen_ai.agent.name` low cardinality, and hold `gen_ai.tool.name` and `gen_ai.request.model` to that same rule. Never dimension a panel by `gen_ai.conversation.id`, `gen_ai.response.id` or `gen_ai.tool.call.id`, because every value there is unique per event and Grafana draws one series per request. Retention sets what the dashboard can answer. Log Analytics keeps 30 days of analytics data by default and some log tables for 90 days ([Microsoft Learn: manage data retention in a Log Analytics workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-retention-configure)). Analytics retention can extend to two years, and total retention reaches 12 years, though long-term data is reachable only through search jobs. A twelve-month cost trend is therefore not free. Your contracted rate multiplies against the token sums from each panel, and that rate differs by model and region. Hardcode a price and the chart drifts from the invoice at the next contract change, so hold the rate in a dashboard variable you own. The Grafana layer itself costs nothing extra when you use the in-portal **Dashboards with Grafana** experience in Application Insights. You pay standard Azure Monitor ingestion and query charges, which is worth knowing when someone proposes Azure Managed Grafana for a single team. ## FAQ ### Is an Azure OpenAI resource required? No Azure OpenAI resource is required. A manually created `chat` span with the attributes used by these queries is enough to test the panels. The conventions define a common target schema, but automatic provider instrumentations can emit older or different convention versions. Inspect one raw span and keep compatibility queries where the emitted names differ. ### Which instrumentation package should you install? `microsoft-opentelemetry` and `azure-monitor-opentelemetry` are both documented as current. `microsoft-opentelemetry` is newer with broader auto-instrumentation, and `azure-monitor-opentelemetry` is what most Microsoft samples still show. Pick one, because mixing them in one process duplicates spans. ### Why do token totals sometimes look inflated? The [cache counter is a subset of the input counter](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/registry/attributes/gen-ai.md), and the input counter includes it. Sum `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` only, and leave `gen_ai.usage.cache_read.input_tokens` out of the total. ## Wrapping up Go back to that support agent with the doubled bill. With `gen_ai` spans in Application Insights and four KQL panels in Grafana, the question has an answer in about ten seconds, because the token chart breaks down by model and the tool panel shows which function ran 90,000 times. The bill is still the bill, but it now has a shape you can point at, and the next regression shows up as a step change on a panel instead of a surprise on an invoice. The useful next step is alerting on the same [Logs queries](https://grafana.com/docs/grafana/latest/datasources/azure-monitor/query-editor/) rather than adding panels. A token burn rate threshold and a p95 latency rule catch the two failures that cost real money in this pipeline. Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fgrafana-genai-token-monitoring%2F&text=Monitor%20Every%20GenAI%20Token%20with%20Grafana%20and%20Azure%20Monitor)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fgrafana-genai-token-monitoring%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fgrafana-genai-token-monitoring%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/2022/04/Getting-Started-with-Grafana-Prometheus-Kubernetes-Cluster-Monitoring-1.jpg) ### [Utilizing Grafana & Prometheus Kubernetes Cluster Monitoring](/prometheus-kubernetes/) Learn how to effectively monitor your Grafana and Prometheus Kubernetes cluster in this tutorial! ![](https://adamtheautomator.com/wp-content/uploads/2022/03/Get-Started-With-Monitoring-in-this-Grafana-Tutorial.jpg) ### [Learn from this Grafana Tutorial and Make Data Monitoring Easy](/grafana-tutorial/) Learn how to set up Grafana on your machine to monitor services and applications data in this Grafana tutorial effectively! ![](https://adamtheautomator.com/wp-content/uploads/publisher/4065d9c85b2b8191a472c3296ef0d282/e680d5ac1f0cde066996077d926999a4adf6d582b7266d915c0d1cdc3e77f938.webp) ### [Why Sysadmins Must Pivot to AI Agent Orchestration](/sysadmin-ai-agent-orchestration/) Learn how sysadmins can govern AI agents with scoped identities, policy gates, approvals, telemetry, and rollback while building an orchestration career. ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)