<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Observability on Stephan Michard</title><link>https://www.stephan.michard.io/tags/observability/</link><description>Recent content in Observability on Stephan Michard</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><lastBuildDate>Sun, 22 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://www.stephan.michard.io/tags/observability/" rel="self" type="application/rss+xml"/><item><title>Tracing My Agents with MLflow</title><link>https://www.stephan.michard.io/2026/tracing-my-agents-with-mlflow/</link><pubDate>Sun, 22 Nov 2026 00:00:00 +0000</pubDate><guid>https://www.stephan.michard.io/2026/tracing-my-agents-with-mlflow/</guid><description>&lt;figure&gt;&lt;img
src="https://www.stephan.michard.io/images/posts/post_51/overview_hu_557b009cf9b0522d.webp"
srcset="https://www.stephan.michard.io/images/posts/post_51/overview_hu_1fc914ca62a3506f.webp 480w, https://www.stephan.michard.io/images/posts/post_51/overview_hu_25a7c986548a73ae.webp 800w, https://www.stephan.michard.io/images/posts/post_51/overview_hu_557b009cf9b0522d.webp 1400w"
sizes="(max-width: 42rem) 100vw, 42rem"
width="1400"
height="781"
alt="A Claude Code trace in MLflow with its tool calls and token usage - AI generated"
loading="lazy"
decoding="async" title="A Claude Code trace in MLflow with its tool calls and token usage - AI generated"&gt;&lt;figcaption&gt;
&lt;h4&gt;A Claude Code trace in MLflow with its tool calls and token usage - AI generated&lt;/h4&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;
&lt;p&gt;Over the last months, more of my work has moved to agents. Two questions keep coming back: how to set up agents in an enterprise environment, and how to work with them efficiently and securely. Hermes Agent runs in my homelab and takes care of research and routine tasks, while Claude Code and Codex write the code for small software projects. What I usually see of their work is the result. What happened in between, from the tool calls to the tokens each turn used, stays hidden in a terminal or a chat history.&lt;/p&gt;
&lt;p&gt;In my &lt;a href="https://www.stephan.michard.io/2026/from-vibe-coding-to-spec-driven-development/"&gt;last post&lt;/a&gt;
, I also claimed that my new development workflow uses fewer tokens, and I admitted that I cannot prove it yet. Tracing is how I want to change that. In this post, I describe how I collect traces from my agents in a self-hosted &lt;em&gt;MLflow&lt;/em&gt; instance in my homelab. I start with the deployment and then cover the configuration for Hermes Agent, Claude Code and Codex, as well as for model calls that go through LiteLLM.&lt;/p&gt;
&lt;h2 id="what-a-trace-shows"&gt;What a Trace Shows&lt;/h2&gt;
&lt;p&gt;MLflow started as a platform for tracking machine learning experiments. Its tracing feature records what happens inside LLM applications and agents, and the documentation lists ready-made integrations for coding agents like Claude Code, Codex and Gemini CLI, as well as for Hermes Agent.&lt;/p&gt;
&lt;p&gt;A trace covers one turn of an agent, from the request to the answer. The turn is the root span. Below it, every model call and every tool call gets its own span with its inputs and outputs, and the trace as a whole records the token usage. MLflow groups traces in experiments. I use three: one for Hermes Agent, one for coding agents, which Claude Code and Codex share, and one for LLM requests that go through LiteLLM.&lt;/p&gt;
&lt;figure&gt;&lt;img
src="https://www.stephan.michard.io/images/posts/post_51/mlflow_trace_hu_284eba1ec68ebbbe.webp"
srcset="https://www.stephan.michard.io/images/posts/post_51/mlflow_trace_hu_cf38307d095baf7b.webp 480w, https://www.stephan.michard.io/images/posts/post_51/mlflow_trace_hu_2d7a5b29c3717e96.webp 800w, https://www.stephan.michard.io/images/posts/post_51/mlflow_trace_hu_284eba1ec68ebbbe.webp 1400w"
sizes="(max-width: 42rem) 100vw, 42rem"
width="1400"
height="629"
alt="A trace in the MLflow UI with the agent span at the root, two model calls and a tool call below it, and the conversation on the right - Source"
loading="lazy"
decoding="async" title="A trace in the MLflow UI with the agent span at the root, two model calls and a tool call below it, and the conversation on the right - Source"&gt;&lt;figcaption&gt;
&lt;h4&gt;A trace in the MLflow UI with the agent span at the root, two model calls and a tool call below it, and the conversation on the right - &lt;a href="https://mlflow.org/docs/latest/genai/tracing/" target="_blank" rel="noopener noreferrer"&gt;Source&lt;/a&gt;&lt;/h4&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;MLflow accepts traces on two paths. Integrations built for MLflow write through its REST API, which is what the Claude Code and Codex integrations do. Everything else can send OpenTelemetry data to the endpoint for the OpenTelemetry Protocol (OTLP) at &lt;code&gt;/v1/traces&lt;/code&gt;, and that is the path Hermes Agent and LiteLLM take. The endpoint needs MLflow 3.6 or newer and a SQL database as backend store.&lt;/p&gt;
&lt;h2 id="deploying-mlflow-in-the-homelab"&gt;Deploying MLflow in the Homelab&lt;/h2&gt;
&lt;p&gt;MLflow runs on the setup I described in my &lt;a href="https://www.stephan.michard.io/2026/my-homelab-a-traefik-centered-self-hosting-setup/"&gt;homelab post&lt;/a&gt;
: Docker Compose on a single host, with Traefik in front for TLS and host-based routing. As a starting point, I used the Docker Compose file from the MLflow repository on &lt;a href="https://github.com/mlflow/mlflow/blob/master/docker-compose/docker-compose.yml" target="_blank" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;
and adjusted it for my setup, so that Traefik works as the reverse proxy in front of MLflow.&lt;/p&gt;
&lt;h2 id="hermes-agent"&gt;Hermes Agent&lt;/h2&gt;
&lt;p&gt;Hermes Agent runs on a dedicated host in my homelab, as described in &lt;a href="https://www.stephan.michard.io/2026/hermes-agent-a-personal-ai-that-gets-more-useful-over-time/"&gt;my post about it&lt;/a&gt;
. It does not send traces on its own. The community plugin &lt;em&gt;hermes-otel&lt;/em&gt; by Brian Caffey adds OpenTelemetry export to a long list of backends, MLflow among them. I run version 1.21.0. Installing the plugin does not activate it, so it takes two commands:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;hermes plugins install hermes_otel
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;hermes plugins &lt;span class="nb"&gt;enable&lt;/span&gt; hermes_otel
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Plugins in Hermes belong to a profile. If you run more than one profile, as I described in my &lt;a href="https://www.stephan.michard.io/2026/hermes-bot-mode-persistent-specialist-profiles/"&gt;post about bot mode&lt;/a&gt;
, each profile needs the plugin and its configuration. Otherwise, only the profile you set it up for sends traces.&lt;/p&gt;
&lt;p&gt;The plugin reads its settings from &lt;code&gt;~/.hermes/hermes_otel.yaml&lt;/code&gt;. The MLflow documentation places the file inside the plugin folder instead, but the file in the home directory takes precedence. Mine looks like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-yaml" data-lang="yaml"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;enabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;force_flush_on_session_end&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;capture_sender_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;capture_conversation_history&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;backends&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;- &lt;span class="nt"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;otlp&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;mlflow&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;endpoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;https://mlflow.home.example.com/v1/traces&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;logs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;x-mlflow-experiment-id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;2&amp;#34;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The header routes the spans to the Hermes experiment, which has the ID 2 in my instance. Metrics and logs are switched off, so MLflow only receives traces, and &lt;code&gt;force_flush_on_session_end&lt;/code&gt; exports any buffered spans when a session ends. The plugin registers 15 hooks in Hermes, covering agent turns, model and API calls, tool calls, skills, approvals, subagents and cron jobs. Each turn becomes a trace with an agent span at the root and child spans for model calls (&lt;code&gt;llm.&amp;lt;model&amp;gt;&lt;/code&gt;, &lt;code&gt;api.&amp;lt;model&amp;gt;&lt;/code&gt;), tools (&lt;code&gt;tool.&amp;lt;name&amp;gt;&lt;/code&gt;), skills and subagents.&lt;/p&gt;
&lt;h2 id="claude-code"&gt;Claude Code&lt;/h2&gt;
&lt;p&gt;For Claude Code, MLflow maintains its own plugin. It writes traces through the REST API and needs no local Python installation. The marketplace has to be added in the terminal, because the command inside Claude Code clones the whole MLflow repository and tends to time out:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;claude plugin marketplace add mlflow/mlflow --sparse .claude-plugin
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;claude plugin install mlflow-tracing@mlflow-plugins
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;After a restart of Claude Code, &lt;code&gt;/mlflow-tracing:setup&lt;/code&gt; asks for the tracking URI, the experiment and the scope, and &lt;code&gt;/mlflow-tracing:status&lt;/code&gt; shows the result. With user scope, tracing covers every project. My &lt;code&gt;~/.claude/settings.json&lt;/code&gt; holds three environment variables for it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-json" data-lang="json"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&amp;#34;env&amp;#34;&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nt"&gt;&amp;#34;MLFLOW_CLAUDE_TRACING_ENABLED&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;true&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nt"&gt;&amp;#34;MLFLOW_TRACKING_URI&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;https://mlflow.home.example.com&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nt"&gt;&amp;#34;MLFLOW_EXPERIMENT_ID&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;3&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Setting &lt;code&gt;MLFLOW_CLAUDE_TRACING_ENABLED=false&lt;/code&gt; switches tracing off for sessions started from that shell. Instead of the plugin, &lt;code&gt;mlflow autolog claude&lt;/code&gt; from the MLflow Python package, version 3.4 or newer, can write the hooks into the &lt;code&gt;.claude/settings.json&lt;/code&gt; of a single project.&lt;/p&gt;
&lt;p&gt;A Claude Code trace contains the prompts and responses, every tool call with its inputs and outputs, and the use of subagents and skills. Tool outputs include file contents and command output, so a trace holds everything the agent read. Each trace also records the token usage, including the tokens served from the prompt cache, and a cost estimate. The cost is calculated from API prices, so on a subscription plan it is a reference value for what the same tokens would cost through the API. The documentation says that traces appear when a session ends. In my setup, a trace arrives after every turn, and the traces of one session share a session ID.&lt;/p&gt;
&lt;h2 id="codex"&gt;Codex&lt;/h2&gt;
&lt;p&gt;MLflow&amp;rsquo;s Codex integration uses Codex&amp;rsquo;s notify hook, which Codex calls after every turn. The npm package &lt;code&gt;@mlflow/codex&lt;/code&gt; provides the hook and a setup command:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;npm install -g @mlflow/codex
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;mlflow-codex setup
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The setup asks for the scope, project or user, and then for the tracking URI and the experiment ID. It writes &lt;code&gt;notify = [&amp;quot;mlflow-codex&amp;quot;, &amp;quot;notify-hook&amp;quot;]&lt;/code&gt; into Codex&amp;rsquo;s &lt;code&gt;config.toml&lt;/code&gt; and stores the MLflow settings in a &lt;code&gt;mlflow-tracing.json&lt;/code&gt; file next to it. My Codex traces go to the same experiment as the ones from Claude Code:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-json" data-lang="json"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nt"&gt;&amp;#34;trackingUri&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;https://mlflow.home.example.com&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nt"&gt;&amp;#34;experimentId&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;3&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A Codex trace has an agent span at the root with model and tool calls below it. It records the token usage and, inside a Git repository, the repository, branch and commit. Codex can also export its own OpenTelemetry traces to &lt;code&gt;/v1/traces&lt;/code&gt;. According to the MLflow documentation, those show internal details like websocket connections and tool scheduling, and the hook is the better source for the conversation itself. I have not enabled the OTLP export.&lt;/p&gt;
&lt;h2 id="litellm"&gt;LiteLLM&lt;/h2&gt;
&lt;p&gt;&lt;em&gt;LiteLLM&lt;/em&gt; is a proxy that offers one OpenAI-compatible API in front of many model providers. In my homelab, I use it as the model gateway and as the registry for Model Context Protocol servers in my Open WebUI setup, which I described in my &lt;a href="https://www.stephan.michard.io/2026/my-local-ai-stack-open-webui-litellm-searxng-and-docling/"&gt;post about my local AI stack&lt;/a&gt;
. Every request that goes through it can be traced as well.&lt;/p&gt;
&lt;p&gt;I use LiteLLM&amp;rsquo;s built-in OpenTelemetry callback, which sends the spans to the same OTLP endpoint as Hermes Agent. It takes one setting in the LiteLLM &lt;code&gt;config.yaml&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-yaml" data-lang="yaml"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;litellm_settings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;callbacks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;otel&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The exporter reads its target from four environment variables in the compose file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-yaml" data-lang="yaml"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;OTEL_EXPORTER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;otlp_http&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;OTEL_ENDPOINT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;http://mlflow:5000/v1/traces&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;OTEL_HEADERS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;x-mlflow-experiment-id=${MLFLOW_LITELLM_EXPERIMENT_ID}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;OTEL_SERVICE_NAME&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l"&gt;litellm&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The endpoint uses the container name instead of the public host name. LiteLLM and MLflow share a Docker network, so the spans never pass through Traefik.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;figure&gt;&lt;img
src="https://www.stephan.michard.io/images/posts/post_51/mlflow_dashboard_hu_abcda707c23599d1.webp"
srcset="https://www.stephan.michard.io/images/posts/post_51/mlflow_dashboard_hu_816f2424b0683433.webp 480w, https://www.stephan.michard.io/images/posts/post_51/mlflow_dashboard_hu_bde303f21ada6a81.webp 800w, https://www.stephan.michard.io/images/posts/post_51/mlflow_dashboard_hu_abcda707c23599d1.webp 1400w"
sizes="(max-width: 42rem) 100vw, 42rem"
width="1400"
height="718"
alt="The overview page of an MLflow experiment aggregates requests, latency, errors and token usage across all traces - Source"
loading="lazy"
decoding="async" title="The overview page of an MLflow experiment aggregates requests, latency, errors and token usage across all traces - Source"&gt;&lt;figcaption&gt;
&lt;h4&gt;The overview page of an MLflow experiment aggregates requests, latency, errors and token usage across all traces - &lt;a href="https://mlflow.org" target="_blank" rel="noopener noreferrer"&gt;Source&lt;/a&gt;&lt;/h4&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;One MLflow instance in my homelab now collects traces from various sources. Hermes Agent and LiteLLM send OpenTelemetry spans, while Claude Code and Codex use integrations that MLflow maintains itself. LiteLLM covers every model request from Open WebUI, and its spans land in Postgres. The setup took an evening. The more interesting work starts now, with the questions the traces can answer, from the tokens a single task costs to the price of a Hermes routine.&lt;/p&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;MLflow - &lt;a href="https://mlflow.org/" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MLflow Docker Compose file on GitHub - &lt;a href="https://github.com/mlflow/mlflow/blob/master/docker-compose/docker-compose.yml" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MLflow: OpenTelemetry trace ingestion - &lt;a href="https://www.mlflow.org/docs/latest/genai/tracing/opentelemetry/ingest/" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MLflow Tracing: Hermes Agent - &lt;a href="https://mlflow.org/docs/latest/genai/tracing/integrations/listing/hermes_agent/" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MLflow Tracing: Claude Code - &lt;a href="https://mlflow.org/docs/latest/genai/tracing/integrations/listing/claude_code/" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MLflow Tracing: Codex CLI - &lt;a href="https://mlflow.org/docs/latest/genai/tracing/integrations/listing/codex/" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MLflow Tracing: LiteLLM - &lt;a href="https://mlflow.org/docs/latest/genai/tracing/integrations/listing/litellm/" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;hermes-otel on GitHub - &lt;a href="https://github.com/briancaffey/hermes-otel" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LiteLLM: OpenTelemetry integration - &lt;a href="https://docs.litellm.ai/docs/observability/opentelemetry_integration" target="_blank" rel="noopener noreferrer"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;My Traefik-centered homelab setup - &lt;a href="https://www.stephan.michard.io/2026/my-homelab-a-traefik-centered-self-hosting-setup/"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;My Local AI Stack: Open WebUI, LiteLLM, SearXNG, and Docling - &lt;a href="https://www.stephan.michard.io/2026/my-local-ai-stack-open-webui-litellm-searxng-and-docling/"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hermes Agent: A Personal AI That Gets More Useful Over Time - &lt;a href="https://www.stephan.michard.io/2026/hermes-agent-a-personal-ai-that-gets-more-useful-over-time/"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hermes Bot Mode: Persistent Specialist Profiles - &lt;a href="https://www.stephan.michard.io/2026/hermes-bot-mode-persistent-specialist-profiles/"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;From Vibe Coding to Spec-Driven Development - &lt;a href="https://www.stephan.michard.io/2026/from-vibe-coding-to-spec-driven-development/"&gt;link&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;</description></item></channel></rss>