Pylon
Prompt evaluation
Prompt iterations rubber-banded: each engineer's fix overcorrected the last. Evals are now a merge requirement—no eval, no commit.
- 10 mins of active debugging eliminated per on-call incident
Teams couldn't manually review hundreds of daily AI hotel calls. Audio models now evaluate raw recordings, routing exceptions to humans.
An all-in-one travel and expense management platform with an AI voice agent making hundreds of calls daily to hotels worldwide to confirm bookings and deliver payment card details on behalf of travelers.
As call volume scaled to hundreds per day, the development and ops teams couldn't manually listen to each conversation to assess quality or...
“When we started to build this AI agent very quickly, what started happening was that hundreds of calls started happening on behalf of our travelers.”
All-in-one travel and expense management platform that combines corporate travel booking, expense tracking, and corporate card services.
AI observability and evaluation platform that helps developers build, test, and monitor LLM-powered applications.
Navan's Voice call evaluation is part of this use case:
Related implementations across industries and use cases
Prompt iterations rubber-banded: each engineer's fix overcorrected the last. Evals are now a merge requirement—no eval, no commit.
Repeated prompts overfitted the agent, hiding regressions. Evals score for equivalence — the same app can be built in multiple valid ways.
Missed names and numbers on noisy calls broke workflows. Now, AI accurately transcribes live speech, routing complex cases to humans.
Prompt iterations rubber-banded: each engineer's fix overcorrected the last. Evals are now a merge requirement—no eval, no commit.
Repeated prompts overfitted the agent, hiding regressions. Evals score for equivalence — the same app can be built in multiple valid ways.
Thousands of tables tracking the same metrics differently. No one knew which to trust — not humans, not AI. One governed layer changed that.
Three people, three different AI answers. Grounded data sources fixed that — finance now has years of vendor spend at its fingertips.
Tournaments running simultaneously meant an hour of manual checks each. AI agents now run them in minutes, freeing the team to be proactive.
Large AI training jobs meant fighting for preemptible slots or leaving campus. Marlowe gave any lab guaranteed multi-node access on demand.
Teams couldn't manually review hundreds of daily AI hotel calls. Audio models now evaluate raw recordings, routing exceptions to humans.
An all-in-one travel and expense management platform with an AI voice agent making hundreds of calls daily to hotels worldwide to confirm bookings and deliver payment card details on behalf of travelers.
As call volume scaled to hundreds per day, the development and ops teams couldn't manually listen to each conversation to assess quality or...
“When we started to build this AI agent very quickly, what started happening was that hundreds of calls started happening on behalf of our travelers.”
All-in-one travel and expense management platform that combines corporate travel booking, expense tracking, and corporate card services.
AI observability and evaluation platform that helps developers build, test, and monitor LLM-powered applications.
Navan's Voice call evaluation is part of this use case:
Related implementations across industries and use cases
Prompt iterations rubber-banded: each engineer's fix overcorrected the last. Evals are now a merge requirement—no eval, no commit.
Repeated prompts overfitted the agent, hiding regressions. Evals score for equivalence — the same app can be built in multiple valid ways.
Missed names and numbers on noisy calls broke workflows. Now, AI accurately transcribes live speech, routing complex cases to humans.
Prompt iterations rubber-banded: each engineer's fix overcorrected the last. Evals are now a merge requirement—no eval, no commit.
Repeated prompts overfitted the agent, hiding regressions. Evals score for equivalence — the same app can be built in multiple valid ways.
Thousands of tables tracking the same metrics differently. No one knew which to trust — not humans, not AI. One governed layer changed that.
Three people, three different AI answers. Grounded data sources fixed that — finance now has years of vendor spend at its fingertips.
Tournaments running simultaneously meant an hour of manual checks each. AI agents now run them in minutes, freeing the team to be proactive.
Large AI training jobs meant fighting for preemptible slots or leaving campus. Marlowe gave any lab guaranteed multi-node access on demand.