Why OpenTelemetry Is Becoming Important Infrastructure for AI

Why OpenTelemetry Is Becoming Critical Infrastructure for AI

OpenTelemetry may sound like a rather boring piece of infrastructure.

And, in a way, it is.

Its job is to answer questions such as:

• What is my application doing?
• Where is it failing?
• How long does each operation take?
• Which service called which?
• What happened before an error appeared?

To do this, OpenTelemetry provides an open standard for generating and transporting traces, metrics, and logs.

An application can be instrumented using OpenTelemetry and then send that telemetry to whichever observability system it wants.

The architecture usually looks something like this:

Aplicación
↓
OpenTelemetry
↓
Collector
↓
Backend de observabilidad

The application generates telemetry.

The Collector can process, filter, transform, or route it.

Finally, observability, storage, or analysis tools can consume it.

The idea seems simple, but it solves an important problem: preventing every application from having to implement a different way of describing what is happening internally.

And that problem is becoming especially relevant with AI systems.

Observing AI Systems

A traditional application might generate a trace like this:

HTTP request
└── database query

An AI-based application may look more like this:

User request

└── Agent
├── LLM call
├── Retrieval
│ └── Vector database
├── Tool call
│ └── CRM
├── LLM call
└── Tool call
└── Payment API

Suddenly, we no longer want to know only how long a request took.

We also want to understand things like:

• which model was used,
• how many tokens it consumed,
• which tools an agent called,
• which documents it retrieved,
• which provider ran the model,
• how much an execution cost,
• why a generation ended,
• which steps an agent followed,
• what evaluation a response received.

This is where a seemingly small problem appears.

There is no technical reason preventing each team from recording this data however it wants.

Un equipo podría guardar el modelo como:

llm.model

Another as:

model_name

And another as:

provider_model

The three fields represent essentially the same thing.

The problem appears when we want to compare applications, switch observability providers, build tools on top of those traces, or analyze systems built with different frameworks.

Multiply this inconsistency across tokens, agents, tools, retrieval, models, evaluations, and providers.

What looked like a naming problem ends up becoming an infrastructure problem.

OpenTelemetry Is Trying to Create a Common Language for AI

This is where OpenTelemetry's GenAI Semantic Conventions come in.

Semantic conventions define a common way of representing certain operations within a trace.

Instead of every SDK, framework, or observability platform inventing its own representation of a model call, the goal is for the entire ecosystem to share the same vocabulary.

Por ejemplo, una ejecución podría conceptualmente producir una traza como:

agent.run
├── gen_ai.chat
├── retrieval
├── execute_tool
└── gen_ai.chat

And each operation can carry structured attributes about the model, provider, tokens used, or tool executed.

What matters is not exactly what each attribute is called today.

What matters is that there is a common contract.

If different applications produce telemetry following the same schema, different tools can understand it.

This makes it possible to start separating two things that have historically been too tightly coupled:

Instrumentación ≠ Backend de observabilidad

Instrument Once

Without a common standard, integrating an observability platform can end up introducing vendor-specific code throughout the application.

Por ejemplo:

vendor.start_llm_span(...)
vendor.log_tokens(...)
vendor.log_tool_call(...)
vendor.finish_span(...)

If we later want to switch platforms, part of that instrumentation also has to change.

With OpenTelemetry, the model can be different:

with tracer.start_as_current_span("generate_response") as span:
response = model.generate(prompt)

span.set_attribute("gen_ai.request.model", model_name)

The application generates a trace using a common standard.

We can then decide where to send it.

┌── Backend A
Aplicación → OTel → Collector ── Backend B
└── Data warehouse

We can even send it to multiple systems simultaneously.

This makes telemetry a layer independent of the tool that ultimately visualizes it.

And that is particularly interesting in AI, where the ecosystem changes extremely quickly.

Today an application may use one model provider; tomorrow, another.

Today it may use one agent framework and replace it six months from now.

Observability should survive those changes.

An Example: Understanding What an Agent Is Doing

Imagine an agent tasked with resolving a support request.

The user writes:

Quiero saber por qué todavía no me han devuelto
el dinero del pedido 839281.

The agent performs these operations:

agent.run
│
├── llm
│
├── tool: get_order
│
├── tool: get_refund_status
│
└── llm

A trace lets us reconstruct the complete execution.

We can discover, for example, that the first model took 800 ms, the get_order call took 120 ms, get_refund_status failed once and had to be retried, and the final generation consumed 1,200 tokens.

When an AI system fails, looking only at the HTTP 500 from the outer request tells us very little.

We want to understand what happened inside the system.

That makes traces particularly valuable for agents.

Another Example: Finding Cost Problems

Observability can also answer questions that are not strictly about errors.

Suppose a new version of an agent starts consuming twice as many tokens.

Without structured traces, we may know that the bill has increased.

With them, we can detect where:

Version 41

average input tokens: 2,140
average output tokens: 430

frente a:

Version 42

average input tokens: 5,870
average output tokens: 460

Perhaps the system is sending too much context to the model, retrieving too many documents, or a tool is returning enormous responses.

Telemetry turns a cost increase into something we can investigate.

But Observing AI Introduces a New Problem

In a traditional application, a trace usually contains information about the software: execution times, errors, endpoints, or database queries.

In an AI application, it can contain something quite different: the data the user is processing.

A prompt may include a name, an email address, medical information, or financial data. A tool call may include a customer's complete record. An RAG system may retrieve parts of internal documents.

And an agent can cause that data to pass through many systems during a single execution:

Usuario
↓
Aplicación
├── Modelo
├── Vector database
├── CRM
├── APIs externas
└── Sistema de observabilidad

The problem is not simply that sensitive data exists.

It is that copies of that data start existing in many different places.

Each copy may have its own permissions, retention policies, backups, storage regions, and external providers.

A trace created to debug an agent can accidentally become another database containing personal information.

Minimize Before Storing

The first solution is quite simple: do not store what we do not need.

Para entender el comportamiento de un agente podemos querer conservar:

model = ...
input_tokens = 1842
output_tokens = 312
tool = get_customer
latency = 842ms
status = success

sin conservar necesariamente:

Me llamo María García,
mi teléfono es +34 612 345 678 y...

But in AI this is not always enough.

Sometimes we need part of the content to understand why an agent made a decision, evaluate a response, or reproduce a failure.

This is where a second strategy comes in: transforming the data before storing it or sending it to external systems.

Por ejemplo:

María García solicitó un reembolso del pedido 839281.

puede convertirse en:

<PERSON_71C2> solicitó un reembolso del pedido <ORDER_89AD>.

The trace remains useful.

We can even keep the same pseudonymous identifier across multiple spans to reconstruct what the agent did without directly exposing the person's identity.

There is an important distinction here.

If we destroy any reasonable possibility of identifying María later, we are talking about anonymization.

If there is a private mapping that makes it possible to recover her identity, we are talking about pseudonymization.

For observability, the latter is particularly interesting because it allows correlations to be maintained without distributing the original identifier across all systems.

A Privacy Layer Before Data Leaves

Una arquitectura posible sería introducir una capa específica para inspeccionar y transformar los datos antes de exportarlos:

AI Application
↓
Instrumentation
↓
Privacy layer
│
├── Detectar PII
├── Eliminar secretos
├── Aplicar políticas
├── Pseudonimizar identificadores
└── Redactar contenido
↓
Telemetry pipeline
↓
Observability backend

The important question is where that transformation happens.

Si el backend recibe primero:

María García

and then transforms it into:

<PERSON>

the backend has already received the original data.

This is a different security property from performing the transformation within our own infrastructure and exporting only:

<PERSON_71C2>

The same logic should apply not only to prompts, but also to:

• model responses,
• tool arguments,
• tool results,
• documents retrieved through RAG,
• metadata,
• exceptions,
• logs.

In agents, tool calls and retrieved documents are probably precisely where some of the most sensitive data appears.

How to Build This Layer

There is no need to solve the problem exclusively with another LLM.

A reasonable implementation can combine several techniques.

First, deterministic detectors for information that is easy to recognize:

emails
phone numbers
IPs
credit cards
API keys
internal identifiers

Then, entity recognition models for less structured information:

people
organizations
addresses
locations
medical information

And finally, application-specific rules.

A patient identifier or an internal contract number probably cannot be discovered by a generic library. The organization itself knows the format and can define how to handle it.

A policy can then be applied to those detections:

EMAIL → eliminar
PHONE_NUMBER → eliminar
PERSON → pseudonimizar
CUSTOMER_ID → HMAC/hash estable
API_KEY → bloquear completamente
ORDER_ID → conservar

This is more useful than simply applying [REDACTED] to everything.

The goal should not be to remove as much information as possible.

It should be to retain the minimum amount necessary for the trace to remain useful.

There are already open-source components for doing this; among the more mature ones are Presidio, LLM Guard, and Guardrails AI. I have personally developed two open-source projects toward this solution (pii-firewall and gateforge).

AI Observability Is Becoming Infrastructure

AI systems are moving beyond being a single call to a model.

They increasingly resemble distributed systems made up of models, agents, tools, databases, retrieval, and external APIs.

We need to understand what happens inside them when they fail, when they start spending too much, or when an agent makes an unexpected decision.

To do that, we need observability.

But we also need to avoid having that visibility mean copying prompts, internal documents, and personal data into every system a trace touches.

That is why I think the two pieces will eventually evolve together:

Observabilidad estándar
+
Privacy by design

OpenTelemetry can provide the common language and infrastructure for observing these systems.

The next piece is deciding what information we allow to cross that boundary.

Because in AI systems, good observability infrastructure should not merely explain what happened.

It should also allow us to do so without collecting data we never needed to store.