[Skip to content](https://www.hubspot.com#main-content) [](https://www.hubspot.com/startups) - [Resources](https://www.hubspot.com/startups/resources) Show submenu for Resources - [Fundraising](https://www.hubspot.com/startups/fundraising) - [Sales & Marketing](https://www.hubspot.com/startups/sales-and-marketing) - [AI Tech & Tips](https://www.hubspot.com/startups/ai) - [Tech Stack Guide](https://www.hubspot.com/startups/tech-stacks) - [Founder & VC Insights](https://www.hubspot.com/startups/scaling-smarter) - [Podcast](https://www.hubspot.com/startups/podcast) - [Stories](https://www.hubspot.com/startups/stories) Show submenu for Stories - [Digital Magazine](https://www.hubspot.com/startups/magazine) - [DocuSeries](https://www.hubspot.com/startups/docuseries) - [Case Studies](https://www.hubspot.com/startups/stories/customers) - [Community](https://www.hubspot.com/startups/community) - [Partner](https://www.hubspot.com/startups/partners) - [Apply Now](https://app.hubspot.com/signup-hubspot/hubspot-for-startups) - [Resources](https://www.hubspot.com/startups/resources) Show submenu for Resources - [Fundraising](https://www.hubspot.com/startups/fundraising) - [Sales & Marketing](https://www.hubspot.com/startups/sales-and-marketing) - [AI Tech & Tips](https://www.hubspot.com/startups/ai) - [Tech Stack Guide](https://www.hubspot.com/startups/tech-stacks) - [Founder & VC Insights](https://www.hubspot.com/startups/scaling-smarter) - [Podcast](https://www.hubspot.com/startups/podcast) - [Stories](https://www.hubspot.com/startups/stories) Show submenu for Stories - [Digital Magazine](https://www.hubspot.com/startups/magazine) - [DocuSeries](https://www.hubspot.com/startups/docuseries) - [Case Studies](https://www.hubspot.com/startups/stories/customers) - [Community](https://www.hubspot.com/startups/community) - [Partner](https://www.hubspot.com/startups/partners) - [Apply Now](https://app.hubspot.com/signup-hubspot/hubspot-for-startups) - [Home](https://www.hubspot.com/startups) - [AI Blog](https://www.hubspot.com/startups/ai) # The Importance of Benchmarking Foundation Models: Insights from Percy Liang Stanford professor Percy Liang explains why benchmarking is crucial for AI progress and how HELM provides transparent, holistic evaluation of foundation models. Send DataSend DataSend DataSend Data Send DataSend DataSend DataSend Data [Access full video library](https://www.hubspot.com/startups/ai/aisummit-sessions)  Welcome to our coverage of the HubSpot for Startups annual AI Summit in San Francisco. In this session, we're privileged to share insights from [Percy Liang](https://www.linkedin.com/in/percy-liang-717b8a), Associate Professor at [Stanford University](https://profiles.stanford.edu/percy-liang), Director at the Stanford Center for Research on Foundation Models, and Co-founder at [Together.xyz](https://www.together.ai/).   ## The Critical Role of Benchmarks in AI Development Percy Liang opened his keynote by emphasizing how benchmarks fundamentally orient AI development. He posed thought-provoking questions: "If it weren't for ImageNet, what would the deep learning revolution look like in computer vision? If it weren't for SQUAD or SuperGLUE, what would the deep learning revolution look like in NLP?" Benchmarks serve as the North Star for the AI community, providing clear targets to aim for. However, Liang stressed that benchmarking is more than just measurement—it encodes values: - Benchmarks determine what the community prioritizes - The choice of tasks, languages, and domains shapes how technology develops - Without proper benchmarking, innovation lacks direction ## The Benchmarking Crisis in Foundation Models We're living in an era of foundation models, with new models emerging monthly. This rapid pace of innovation has created what Liang describes as a "crisis" in our ability to measure model quality. "The pace of innovation is happening so quickly that we're not able to measure," Liang noted. While benchmarks historically lasted years, foundation models are now advancing faster than our ability to evaluate them properly. ## The Challenge of Evaluating General-Purpose Models Language models represent a paradigm shift in AI—a seemingly simple box that takes text in and produces text out, yet capable of an infinite variety of applications: - Generating SQL queries - Writing emails - Revising content - Explaining jokes This versatility is precisely what makes benchmarking so challenging. How do you comprehensively evaluate systems that can do so many different things? ## What We Need from Language Model Benchmarking Liang outlined several critical requirements for effective benchmarking: - __Transparency__: We need objective understanding of what models can and cannot do - __Industry standards__: Standardized evaluation methods are essential for fair comparisons - __Grounded conversations__: Factual foundations enable informed decisions for policymakers and businesses - __Socially beneficial development__: Benchmarks should guide the creation of reliable, beneficial models Beyond just measuring accuracy, Liang emphasized the importance of evaluating: - Bias - Robustness - Calibration - Efficiency - Human values ## HELM: Holistic Evaluation of Language Models Last year, the Stanford Center for Research on Foundation Models released HELM (Holistic Evaluation of Language Models), designed with three core principles: ### 1. Broad Coverage with Recognition of Incompleteness Rather than creating an arbitrary list of tasks, HELM employs a systematic taxonomy: - Tasks - Domains - Content creators - Time periods - Languages This approach acknowledges gaps in coverage while providing a framework for expansion. ### 2. Multiple Metrics Beyond Accuracy HELM measures multiple dimensions: - __Accuracy__: Basic correctness of outputs - __Calibration__: Whether models know what they don't know - __Robustness__: Consistency when inputs change slightly - __Fairness__: Performance across demographic groups - __Bias__: Tendency to generate stereotyped content - __Toxicity__: Propensity to produce harmful content - __Efficiency__: Practical considerations for deployment ### 3. Standardization HELM systematically evaluates all models on the same scenarios, ensuring fair comparisons across a wide range of use cases: - Question answering - Information extraction - Summarization - Toxicity classification - Sentiment analysis - Various text classification tasks - Coding and reasoning tasks - Legal and medical datasets ## Models and Transparency The HELM project evaluated 30 models from organizations including AI21, Anthropic, Cohere, EleutherAI, Big Science, Google, Meta, Microsoft, NVIDIA, OpenAI, Tsinghua, and Yandex. A key commitment of HELM is full transparency: - All data tables are available online - Users can examine individual instances and model predictions - All code is open-source - The community can contribute scenarios, metrics, and models ## Beyond Text: Expanding Evaluation Liang also discussed several extensions to HELM that address broader evaluation needs: ### Multimodal Models Future benchmarking must encompass models that handle both text and images, evaluating aspects like: - Quality - Originality - Knowledge - Bias and toxicity ### Human-in-the-Loop Evaluation Since models like ChatGPT are used interactively, there's a need to evaluate human-AI interactions: - Interactive metrics don't always correlate with automatic metrics - Models might optimize for responses that are difficult for humans to work with ### Generative Search Engines Evaluating systems like BingChat and Perplexity revealed interesting findings: - Only 74% of citations actually support the generated statements - There's a trade-off between perceived utility and citation accuracy - Some models generate helpful-sounding but fabricated information ## The Path Forward Liang concluded by emphasizing that benchmarking is what orients AI development and determines its future direction. He invited the community to contribute to HELM, particularly with use cases from specialized domains like law, medicine, finance, coding, and creative content. The goal is to create a comprehensive resource that tracks progress in foundation models and provides an up-to-date dashboard for the community. Like a lighthouse, HELM aims to shine light on this rapidly evolving field. *AI Disclaimer: The insights shared in this video or audio were initially distilled through advanced AI summarization technologies, with subsequent refinements made by the writer and our editorial team to ensure clarity and veracity.* Send DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend DataSend Data ## Full AI Summit Library Would you like full access to the complete AI Summit video library, featuring over ten hours of educational content and insights? Click below. [AI Summit Library](https://www.hubspot.com/cs/c/ "AI Summit Library")   Back Close ## Popular Features - [All Products and Features All Products and Features](https://www.hubspot.com/products) - [HubSpot AEO HubSpot AEO](https://www.hubspot.com/products/aeo) - [Free Meeting Scheduler App Free Meeting Scheduler App](https://www.hubspot.com/products/sales/schedule-meeting) - [Email Tracking Software Email Tracking Software](https://www.hubspot.com/products/sales/email-tracking) - [AI Content Writer AI Content Writer](https://www.hubspot.com/products/cms/ai-content-writer) - [AI Website Generator AI Website Generator](https://www.hubspot.com/products/cms/ai-website-generator) - [Email Marketing Software Email Marketing Software](https://www.hubspot.com/products/marketing/email) - [Lead Management Software Lead Management Software](https://www.hubspot.com/products/crm/lead-management) - [AI Prospecting Agent AI Prospecting Agent](https://www.hubspot.com/products/sales/ai-prospecting-agent) - [Free Website Builder Free Website Builder](https://www.hubspot.com/products/cms/drag-and-drop-website-builder) - [Free Landing Page Builder Free Landing Page Builder](https://www.hubspot.com/products/marketing/landing-pages) - [Free Online Form Builder Free Online Form Builder](https://www.hubspot.com/products/marketing/forms) - [Free Chatbot Builder Free Chatbot Builder](https://www.hubspot.com/products/crm/chatbot-builder) - [Free Live Chat Software Free Live Chat Software](https://www.hubspot.com/products/crm/live-chat) - [Marketing Analytics Marketing Analytics](https://www.hubspot.com/products/marketing/analytics) - [Breeze Assistant Breeze Assistant](https://www.hubspot.com/products/artificial-intelligence/breeze-ai-assistant) - [Free Web Hosting Free Web Hosting](https://www.hubspot.com/products/cms/web-hosting) ## Free Tools - [See All Free Business Tools See All Free Business Tools](https://www.hubspot.com/free-business-tools) - [AEO Grader opens in a new tab AEO Grader](https://www.hubspot.com/aeo-grader) - [AI Search Sensor opens in a new tab AI Search Sensor](https://www.hubspot.com/aeo-sensor) - [Make My Persona opens in a new tab Make My Persona](https://www.hubspot.com/make-my-persona) - [Email Signature Generator Email Signature Generator](https://www.hubspot.com/email-signature-generator) - [Free Business Templates Free Business Templates](https://www.hubspot.com/resources/templates) - [Software Comparisons Library Software Comparisons Library](https://www.hubspot.com/comparisons) - [Website Templates Website Templates](https://ecosystem.hubspot.com/marketplace/website) - [Connector for Claude Connector for Claude](https://www.hubspot.com/claude) ## Company - [About Us About Us](https://www.hubspot.com/our-story) - [Careers Careers](https://www.hubspot.com/careers) - [Management Team Management Team](https://www.hubspot.com/company/management) - [Board of Directors Board of Directors](https://www.hubspot.com/company/board-of-directors) - [Investor Relations opens in a new tab Investor Relations](https://ir.hubspot.com/) - [Blog opens in a new tab Blog](https://blog.hubspot.com) - [Sustainability Sustainability](https://www.hubspot.com/sustainability) - [AI Instructions AI Instructions](https://www.hubspot.com/ai-instructions) - [Contact Us Contact Us](https://www.hubspot.com/company/contact) ## Customers - [Customer Support opens in a new tab Customer Support](https://help.hubspot.com/) - [Join a Local User Group Join a Local User Group](https://www.hubspot.com/hubspot-user-groups) ## Partners - [All Partner Programs All Partner Programs](https://www.hubspot.com/partners) - [Solutions Partner Program Solutions Partner Program](https://www.hubspot.com/partners/solutions) - [Technology Partner Program Technology Partner Program](https://www.hubspot.com/partners/app) - [HubSpot for Startups opens in a new tab HubSpot for Startups](https://www.hubspot.com/startups/partners) - [Affiliate Program Affiliate Program](https://www.hubspot.com/partners/affiliates) - [Facebook](https://www.facebook.com/hubspot) - [Instagram](https://www.instagram.com/hubspot/) - [Youtube](https://youtube.com/user/HubSpot) - [Twitter](https://x.com/HubSpot) - [Linkedin](https://www.linkedin.com/company/hubspot) - [Reddit](https://www.reddit.com/r/hubspot) - [Tiktok](https://www.tiktok.com/@hubspot) [](https://apps.apple.com/us/app/hubspot/id1107711722) [](https://play.google.com/store/apps/details?id=com.hubspot.android&hl=en_US) [](https://www.hubspot.com) Copyright © 2026 HubSpot, Inc. - [Legal Center](https://legal.hubspot.com/legal-center) - [Privacy Policy](https://legal.hubspot.com/privacy-policy) - [Security](https://legal.hubspot.com/security) - [Website Accessibility](https://legal.hubspot.com/website-accessibility) - [Manage Cookies](https://www.hubspot.com)