Surprise: Google tops Forrester’s first ranking of AI foundation models – Business
Forrester Research Inc. just released its first Wave ranking of artificial intelligence foundation models, and the findings might raise some eyebrows.
The research firm rated Google LLC as the leader in the category by a substantial margin, with Databricks Inc., Nvidia Corp., IBM Corp. and OpenAI LLC listed as “strong performers.”
Google earned the top spot despite experiencing some embarrassing early snafus with the rollout of its consumer products. That shouldn’t overshadow the search giant’s strong breadth and quality of enterprise offerings, said Rowan Curran, one of the study’s lead authors.
“A lot of Google’s hiccups have been around the consumer-facing offerings,” he said, “but when you look at what they’re doing with Gemini, the quality of the tooling around it is quite significant.”
The report called Gemini, which is Google’s flagship large language model, “uniquely differentiated in the market, especially in multimodality and context length.” Multimodality refers to the ability of AI systems to process and integrate information from multiple types of data. Context length is the span of text that an AI model looks back at to understand and generate appropriate responses.
“Google has everything it takes to lead the AI market – enormous AI infrastructure capacity, a very deep bench of AI researchers, and a growing number of enterprise customers in Google Cloud,” the report asserts.
21-point ranking
Forrester ranked AI foundation models across 21 categories, including the data quality used in training models, multilingual capabilities, cogeneration, governance, scalability, innovation and partner ecosystem.
“We focused on the questions that most matter to enterprise buyers and also an assessment of the models themselves,” Curran said. “We looked at models in an enterprise context that reflects a broad set of decisions around alignment, availability, performance and things that don’t appear in a benchmark score.”
Databricks earned high marks for its hybrid strategy that includes both its own pre-trained DBRX model and support for customers training or fine-tuning their own models. Forrester praised its “comprehensive feature-focused” vision but said Databricks needs to do more to help enterprises build model-based applications.
“Enabling customers to pre-train their own models and developing the tools to do that more efficiently was one of the most interesting [aspects of Databricks’] strategy compared to some of the others,” Curran said.
Strong performers
Nvidia won plaudits for its Nemotron family of models that can be used out of the box and offer strong multilingual capabilities. The researchers wrote that Nvidia needs to fill in more gaps on its roadmap around enterprise tooling and articulate a vision that goes beyond technical virtuosity.
IBM’s Granite models are well-suited for enterprise customers, delivering high quality with indemnification clauses and tooling that are useful in cases demanding strong governance. “The Granite family of models provides enterprise users with some of the most robust and transparent insights into the underlying training data, important for efficiently refining model behavior for specific use cases and domains and for protecting enterprises from risk from any unlicensed content in the training data,” the report says.
OpenAI is an undisputed pioneer and delivers some of the most powerful models on the market, but it needs to do a better job of delivering tooling that enterprise customers can use to build and scale applications.
Further back in the “strong performer” pack are Amazon Web Services Inc., Microsoft Corp., Cohere Inc. Anthropic PBC and Mistral AI SAS.
Amazon’s Titan models have lagged in customer adoption, although that shortcoming is fixable. “AWS has not articulated a clear vision about when customers should choose to use Titan models versus other models available in Bedrock, such as Anthropic’s Claude,” the researchers write.
Microsoft’s small Phi models are a niche play for companies seeking a small size and tightly curated training data set. However, Microsoft’s investment in OpenAI has overshadowed its model strategy.
Cohere, Anthropic and Mistral all earn praise for model quality but fall short in providing tooling and ecosystems that appeal to enterprise customers, the report concludes.
Open-source exclusion
The report doesn’t cover the many domain-specific models announced or service providers offering access to multiple models on their platforms.
“We settled on a criteria of companies that pretrain a model, make it commercially available and offer a way to sell it in the market,” Curran said. “That allowed us to narrow the list to the specific set of vendors you see in the final report.” Open-source models weren’t included because factors such as vision and strategy don’t apply, he said.
Having spent most of the last year immersed in foundation model reviews, Curran said he’s confident about a few likely developments over the next year. One is that “the cost of pretraining models will drop to a level where it’ll become a very straightforward cost for many use cases,” he said. “I think costs will come down faster than many folks were expecting, particularly in the area of domain-specific models.”
Competition over the size of context windows is likely to expand over the next year, as well as efforts to make model training more affordable. Multimodality is also likely to proliferate, with Google’s PaLM-E being a prime example. It combines language understanding with physical actions to enable more meaningful interactions with the model’s environment. “I expect by the end of the year we’ll have many more multimodal models, and I expect that we’ll have a solid handful of open-weight models that are multimodal to some degree,” he said.