In this episode, we dive into AI development strategies with Nico Finelli, Founding GTM at Vellum.ai. Drawing from his experience with autonomous vehicles at M City and his work at Weights & Biases, Nico shares valuable insights on building AI agents through an incremental, test-driven approach.

TOPICS DISCUSSED:

1. Incremental AI Development
Nico draws parallels between autonomous vehicle development and AI agent building, emphasizing the importance of starting with constrained tasks and gradually expanding capabilities. He illustrates how this approach can be applied to various domains, from legal to healthcare applications.

2. AI Implementation in Healthcare
Through case studies like DeepScribe, Nico demonstrates how AI can enhance healthcare workflows while maintaining human oversight. He discusses practical approaches to evaluation and implementation in sensitive domains.

3. State of AI Development
Drawing from Vellum's comprehensive developer survey, Nico shares insights about AI implementation challenges, highlighting that only 25% of teams successfully reach production, and discusses strategies for improving these outcomes.

4. Evaluation Frameworks
The conversation explores practical approaches to evaluating AI systems, emphasizing the importance of structured testing and feedback loops in development cycles.

INSIGHTS:

- The value of constraining AI problems to build competency gradually

- Why human oversight remains crucial in sensitive AI applications

- The importance of robust evaluation frameworks in AI development

- How implicit and explicit feedback shapes AI system improvement

- The role of domain experts in defining AI system constraints

TOOLS AND TECHNOLOGIES MENTIONED:

- Vellum SDK

- Electronic Health Records (EHR) Systems

- LLM Orchestration Platforms

- SOAP Notes

- Test-Driven Development Frameworks

USEFUL LINKS:

- State of AI Report 2025:

- Vellum Case Studies:

- Vellum SDK Documentation:

CONTACT INFO:

- LinkedIn:

- Email: nico@vellum.ai

CHAPTERS

00:00 Path to a Founding GTM at Vellum.ai

07:03 Insights from Weights and Biases and Vellum.ai

14:05 Building AI Agents: Lessons from Autonomous Vehicles

21:44 The State of AI: Insights from Developer Reports

28:02 Vellum SDK: Enhancing AI Development and Evaluation

34:34 Encouragement for Aspiring AI Builders

The Step-by-Step Path to Building Better AI Agents | Nico Finelli

Listen to the AI Ketchup podcast on your favorite platform

Show Notes

Transcript