The next generation of AI agents isn't just about smarter models—it's about the infrastructure that powers truly autonomous systems. In this compelling fireside chat from the HubSpot for Startups annual AiSummit in San Francisco, three industry leaders broke down what it really takes to build scalable, cost-effective AI agent infrastructure.
Featured speakers:
- Shaown Nandi - Director, Technology, AWS (Moderator)
- Ari Morcos - CEO and Co-founder, Datology AI
- Shaunak Godbole - Field CTO, Fireworks AI
The Real Pain Points Driving AI Infrastructure Innovation
Data Quality: The Hidden Multiplier
Ari Morcos kicked off the discussion with a powerful insight that challenges conventional wisdom about AI training: "Models are what they eat." His background as an AI researcher at DeepMind and FAIR revealed a fundamental problem—most AI training assumes all data is created equal, when in reality, the vast majority of data isn't helpful for teaching models desired behaviors.
Key insights from Datology's approach:
- Training on low-quality data requires orders of magnitude more compute for far worse results
- High-quality data acts as a compute multiplier, delivering 10x more performance per dollar
- Smart data curation can reduce training costs from $50M to just a couple million dollars
The Performance-Cost-Speed Triangle
Shaunak Godbole from Fireworks AI identified three critical pain points that emerged when GPT launched:
- Hallucinations significantly impacted reliability
- Cost made proprietary models prohibitively expensive for scale
- Performance created poor user experiences with slow response times
Fireworks' solution focuses on serving 5 trillion tokens daily while enabling companies to build complex agents using hundreds of models simultaneously—without sacrificing cost efficiency.
Breaking the Traditional Trade-off Paradigm
The Enterprise vs. Consumer Model Divide
The speakers highlighted a crucial distinction that many startups miss:
Consumer use case: Models need to be "a mile wide and at least an inch deep"—capable of answering questions about anything.
Enterprise use case: Models should do one thing to five nines of reliability at the lowest possible cost.
This insight is reshaping how companies approach AI agent development, moving away from general-purpose models toward specialized, smaller language models tailored to specific workflows.
The New Economics of AI Agents
As AI agents become more complex—with some customers using hundreds of steps in their workflows—the traditional approach of using the same large model for every call becomes prohibitively expensive. A 30-minute deep research task can cost $2,000 on proprietary model providers, with equally frustrating latency issues.
The solution? Smaller, specialized models that are:
- Tailored to specific use cases
- Trained on relevant proprietary data
- Optimized for the enterprise's unique traffic patterns
Build vs. Buy: Strategic Advice for Startups
The 90/10 Rule in AI Infrastructure
Shaunak shared a critical insight for startup founders: "You can build up to 90% of what you need with 10% of effort. But the last 10% is the real magic—the real user experience and value for customers."
This is where partnerships become essential. Rather than building everything in-house, successful startups are:
- Partnering with data curation platforms for training optimization
- Leveraging specialized inference engines instead of building their own
- Focusing internal resources on the unique differentiation that drives their business
The Three Optimization Dimensions
When scaling AI applications, companies typically face trade-offs between:
- Better performance
- Faster response times
- Cheaper operational costs
The key is working with infrastructure partners who can help optimize based on your specific latency thresholds and cost budgets, rather than trying to solve these complex optimization problems internally.
The Future of AI Agent Infrastructure
Reducing Human-in-the-Loop Requirements
Looking toward 2027-2028, both companies envision a dramatic reduction in human intervention required for successful AI agents. The goal is creating unified platforms where production feedback seamlessly feeds into training ecosystems—something currently only possible at massive tech companies like Meta and Google.
Unlocking Private Data's Potential
Perhaps the most exciting opportunity lies in enterprise data. At least 99% of the world's total data is private, sitting on company servers rather than on the public internet. This represents massive untapped potential for creating more powerful, specialized models.
The vision is making continuous model updates—currently updated every four hours at companies like Meta—accessible to enterprises of all sizes, not just tech giants with incredible resources.
Bottom Line for Startup Founders
The infrastructure for AI agents is rapidly evolving beyond simple model selection. Success requires:
- Strategic partnerships with specialized data and inference platforms
- Focus on the last 10% that creates real user value and differentiation
- Understanding the enterprise vs. consumer model divide for your use case
- Planning for continuous model improvement cycles from production feedback
The companies that will dominate the AI agent space aren't necessarily those with the biggest models—they're the ones that master the infrastructure to deploy smaller, faster, cheaper models that deliver magical user experiences.
~~~~~
AI Disclaimer: AI helped us summarize the key points from these videos, and our editorial team reviewed everything to make sure it's clear and correct.
