Prepared for

Braintrust

Braintrust Evals Workshop

Intro to evals, hands-on from minute one

We turned one module of your workshop into a browser-based lab: real environment, auto-checked challenges, and usage data your field team can act on.

Hands-on challenges

~75

Minutes end to end

Installs or API keys for attendees

300

Seeded production traces per learner

The lab

Challenge

, running on Instruqt.

Click around. The terminal, editor, app and Check button all work in this mock.

Real terminal.

Python, SDK and sample data preloaded.

In the browser.

No installs, no local keys.

Auto-checked.

Every challenge verifies real state.

Tracked.

Every learner, every step.

Why it matters

One lab, four jobs across the customer journey.

Pipeline

Turn workshop attendees into qualified pipeline.

Where it breaks

Attendees leave with a recording, not a working setup. Interest cools before sales follows up.

With Instruqt

Every session records who finished what, so AEs follow up on real usage.

Completions and stuck points sync to your CRM, so the follow-up references what each developer actually built.

More detail

$14M

from a single prospect-facing webinar with Instruqt

Evaluation

Get evaluators to a first score, fast.

Where it breaks

Developers have to wire keys, SDKs and data before they see a single score.

With Instruqt

A preconfigured sandbox puts them in front of their first experiment on launch.

The same lab doubles as a guided POC: your SE sends a link instead of booking a setup call.

More detail

2,000+

hands-on lab sessions in its first six weeks on Instruqt

Onboarding

Shorten the path to a production eval.

Where it breaks

New customers stall between signup and their first eval in CI.

With Instruqt

Guided labs take each team from first trace to CI gate in their own environment.

CS sees which accounts finished which challenges and where they dropped off.

More detail

10x

onboarding result

Education

Teach evals beyond the room.

Where it breaks

Live workshops cap out at room size and the instructor's calendar.

With Instruqt

The same lab runs self-paced, at events or instructor-led, with one dashboard.

Update the lab once and every channel, from docs to conference booths, gets the new version.

More detail

5000+

customer workshops delivered a year

Challenge map

Six challenges, from first trace to CI gate.

Hover or tap a card to see its check

Shown above

8 min

Set up the project

Connect the SDK to a preconfigured project.

check:

API key present and SDK import succeeds

Shown above

12 min

Log your first traces

Instrument the support bot and send spans.

check:

≥ 10 spans in project logs

Shown above

12 min

Build a dataset

Turn logged traces into a reusable eval dataset.

check:

dataset support-qa has ≥ 20 rows

Shown above

15 min

Write a scorer

Add a custom helpfulness scorer next to Factuality.

check:

scorer returns 0–1 on sample input

Shown above

15 min

Run an experiment and compare

Run prompt-v2 against baseline and read the diff.

check:

experiment prompt-v2 exists and beats baseline on factuality

Shown above

13 min

Gate on CI

Fail the build when scores regress.

check:

CI eval job passes the score threshold

Instructor view

See who's stuck before they drop off.

Live progress for every learner, by challenge, for workshops, events and self-paced cohorts.

Braintrust Evals Workshop

· Field SE cohort

Example data

All learners

Stuck

Learners

Completed

Median time

46 min

Most stuck

Ch. 04

LEARNER010203040506TIME

A. Chen

41 min

M. Okafor

46 min

J. Ruiz

33 min

S. Patel

38 min

L. Novak

27 min

R. Haddad

52 min

CompleteIn progressStuck 10+ minNot started

How it runs

What sits behind each lab

Workstation

A Linux machine with Python 3.12, uv, the bt CLI and the learn-bt repo installed, plus a terminal and code editor in the browser.

Braintrust project

Each learner gets a project of their own in a workshop org, seeded with the repo's 300 replayed traces so there's real traffic to investigate from the start.

Credentials

A Braintrust service token is created for each lab when it starts and expires when it ends. Learners never see or paste a key.

Models

The agent calls an OpenAI model through Amazon Bedrock's OpenAI-compatible endpoint, provided by the lab. No OpenAI account or Braintrust credits needed.

Checks

Scripts query the Braintrust API for the learner's traces, datasets, scorers and experiments.

Clean-up

When a lab ends, its token is revoked and its project is archived.

Our offer

We'll build it, show you how, and prove it in the field

01 · BUILD

We build the full lab from learn-bt.

All six challenges, the checks and the instructor view, running in a Braintrust workshop org.

02 · SHOW

We show your team exactly how we did it.

A walkthrough of the build, the checks and the setup, so nothing about it is a black box.

03 · MEASURE

You run it in a real session and we measure it together.

Pick one live workshop or onboarding cohort. We'll track who builds, how far they get and what they do next, against your current format.

No obligation. Just the evidence to decide with.