As AI increasingly moves from the cloud to on-device, you can be sure that a variety of new laptops will run the AI-powered apps they produce faster than competing off-the-shelf laptops (or desktops, or all). How can we know? -in-one, for that matter? Knowing can mean the difference between waiting a few seconds and waiting a few minutes for the image to generate. It is often said that time is money.
MLCommons is an industry association that supports numerous AI-related hardware benchmark standards. wish The publication of performance benchmarks for “client systems,” or consumer PCs, makes comparisons easier.
Today, MLCommons announced the formation of a new working group, MLPerf Client, with the goal of establishing AI benchmarks for desktops, laptops, and workstations running Windows, Linux, and other operating systems. MLCommons promises that the benchmark is “scenario-driven,” focuses on real-world end-user use cases, and is “based on community feedback.”
To that end, the first benchmark of the MLPerf Client will focus on text generation models, specifically Meta’s Llama 2. According to David Kanter, executive director of MLCommons, this model is already included in his suite of other benchmarks for data center hardware at MLCommons. Meta also did extensive work on Llama 2 in collaboration with Qualcomm and Microsoft. optimize Llama 2 for Windows — brings great benefits to devices running Windows.
“AI has become an expected part of computing everywhere, and the time is ripe to bring MLPerf to client systems,” Kanter said in a press release. “We look forward to working with our members to bring the great capabilities of MLPerf to client systems and drive new features for the broader community.”
Members of the MLPerf Client Working Group include AMD, Arm, Asus, Dell, Intel, Lenovo, Microsoft, Nvidia, and Qualcomm, but not specifically Apple.
Apple is also not a member of MLCommons, and Microsoft’s Director of Engineering (Yannis minadakis) co-chairs the MLPerf Client group. As such, the company’s absence is not entirely surprising. However, the unfortunate result is that whatever AI benchmarks the MLPerf Client creates will not be tested across Apple devices, at least not in the near future.
Still, I’m curious to see what kinds of benchmarks and tools come out of the MLPerf Client, with or without macOS support. If GenAI is here to stay, and there are no signs of the bubble bursting anytime soon, I wouldn’t be surprised to see these types of metrics play an increasingly important role in device purchasing decisions. Sho.
In my best-case scenario, the MLPerf client benchmark is similar to many PC build comparison tools online, showing you what kind of AI performance you can expect from your particular machine. Given the participation of Qualcomm and Arm, both of whom are investing heavily in the mobile device ecosystem, it will likely expand to include phones and tablets in the future. It’s clearly in its early stages, but there’s hope.