Skip to content

feat: add telemetry support to experimental_evaluate - #21162

Merged
n1ckoates merged 8 commits into
mainfrom
nick/evaluate-telemetry
Sep 22, 2026
Merged

feat: add telemetry support to experimental_evaluate#21162
n1ckoates merged 8 commits into
mainfrom
nick/evaluate-telemetry

Conversation

@n1ckoates

@n1ckoates n1ckoates commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Background

experimental_evaluate didn't have telemetry integration

Summary

  • Added lifecycle events, callbacks, tracing-channel support, and OpenTelemetry spans for experimental_evaluate.
  • Added support to both OpenTelemetry and LegacyOpenTelemetry.
  • Used standard gen_ai.* attributes for operation, provider, model, and token usage.
  • Added opt-in ai.evaluation.state, ai.evaluation.questions, and ai.evaluation.answers attributes.
    • OpenTelemetry’s gen_ai.evaluation.result describes a score/label on an existing model response for evals
    • AI SDK evaluation differs - it's a model operation over arbitrary state deciding between multiple questions.
  • Kept experimental prefixes on unstable public extension points, while keeping emitted span names and attributes semantic.

End-to-End Verification

Ran the lifecycle-hook, OpenTelemetry, and tracing-channel examples against TypeSafe AI jev-latest.

Checklist

  • All commits are signed (PRs with unsigned commits cannot be merged)
  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

@vercel-security-reviewer

vercel-security-reviewer Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Security review details

@n1ckoates
n1ckoates marked this pull request as ready for review September 22, 2026 17:42
@n1ckoates
n1ckoates enabled auto-merge (squash) September 22, 2026 17:46
@n1ckoates
n1ckoates disabled auto-merge September 22, 2026 17:48
@n1ckoates
n1ckoates enabled auto-merge (squash) September 22, 2026 17:50
@n1ckoates
n1ckoates merged commit 31d24ce into main Sep 22, 2026
53 checks passed
@n1ckoates
n1ckoates deleted the nick/evaluate-telemetry branch September 22, 2026 17:54
@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

🚀 Published in:

Package Version
ai 7.0.111 github npm
@ai-sdk/angular 3.0.111 github npm
@ai-sdk/code-mode 1.0.68 github npm
@ai-sdk/harness 1.0.121 github npm
@ai-sdk/harness-acp 1.0.59 github npm
@ai-sdk/harness-claude-code 1.0.125 github npm
@ai-sdk/harness-cline 1.0.48 github npm
@ai-sdk/harness-codex 1.0.123 github npm
@ai-sdk/harness-cursor 1.0.34 github npm
@ai-sdk/harness-deepagents 1.0.121 github npm
@ai-sdk/harness-fx 1.0.34 github npm
@ai-sdk/harness-github-copilot 1.0.16 github npm
@ai-sdk/harness-grok-build 1.0.58 github npm
@ai-sdk/harness-opencode 1.0.123 github npm
@ai-sdk/harness-pi 1.0.123 github npm
@ai-sdk/langchain 3.0.111 github npm
@ai-sdk/llamaindex 3.0.111 github npm
@ai-sdk/otel 1.0.111 github npm
@ai-sdk/policy-opa 1.0.111 github npm
@ai-sdk/react 4.0.114 github npm
@ai-sdk/rsc 3.0.111 github npm
@ai-sdk/sandbox-just-bash 1.0.121 github npm
@ai-sdk/sandbox-vercel 1.0.121 github npm
@ai-sdk/svelte 5.0.111 github npm
@ai-sdk/tui 1.0.112 github npm
@ai-sdk/vue 4.0.111 github npm
@ai-sdk/workflow 2.0.42 github npm
@ai-sdk/workflow-harness 1.0.121 github npm

@andreiborza andreiborza mentioned this pull request Sep 24, 2026
Closed

This branch was successfully deployed

1 active deployment
Preview — 2833ab47 Deployed Sep 22, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gr2m gr2m gr2m approved these changes

Assignees

@n1ckoates n1ckoates

Labels

task-identify-harness-labels-done task-merge-tracking-no-issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@n1ckoates @gr2m