Skip to content

S
sheiksandwiches
Open
Opened by Adela Baine@adelabaine0415
Report abuse New issue

If there's Intelligent Life out There


Optimizing LLMs to be good at specific tests backfires on Meta, Stability.

-. -. -. -. -. -. -

When you buy through links on our site, we might earn an affiliate commission. Here's how it works.

Hugging Face has released its 2nd LLM leaderboard to rank the finest language designs it has checked. The new leaderboard looks for to be a more challenging consistent requirement for checking open large language model (LLM) efficiency across a variety of jobs. Alibaba's Qwen designs appear dominant in the leaderboard's inaugural rankings, taking 3 areas in the leading 10.

Pumped to reveal the brand new open . We burned 300 H100 to re-run new evaluations like MMLU-pro for all major open LLMs!Some learning:- Qwen 72B is the king and Chinese open models are controling total- Previous examinations have become too easy for recent ... June 26, 2024

Hugging Face's second leaderboard tests language models across 4 tasks: understanding screening, thinking on very long contexts, complicated math capabilities, and instruction following. Six criteria are utilized to check these qualities, lespoetesbizarres.free.fr with tests consisting of fixing 1,000-word murder secrets, explaining PhD-level concerns in layperson's terms, and most challenging of all: high-school math equations. A complete breakdown of the benchmarks utilized can be discovered on Hugging Face's blog site.

The frontrunner of the new leaderboard is Qwen, Alibaba's LLM, which takes 1st, 3rd, and 10th place with its handful of variants. Also showing up are Llama3-70B, Meta's LLM, and a handful of smaller sized open-source tasks that managed to outperform the pack. Notably absent is any indication of ChatGPT; Hugging Face's leaderboard does not evaluate closed-source designs to ensure reproducibility of results.

Tests to certify on the leaderboard are run specifically on Hugging Face's own computers, which according to CEO Clem Delangue's Twitter, are powered by 300 Nvidia H100 GPUs. Because of Hugging Face's open-source and collective nature, anyone is complimentary to send new designs for testing and admission on the leaderboard, with a new voting system focusing on popular brand-new entries for testing. The leaderboard can be filtered to show only a highlighted selection of significant models to prevent a complicated glut of small LLMs.

As a pillar of the LLM area, Hugging Face has become a relied on source for LLM learning and neighborhood partnership. After its very first leaderboard was released last year as a way to compare and reproduce testing outcomes from several recognized LLMs, the board rapidly removed in appeal. Getting high ranks on the board ended up being the objective of numerous developers, small and large, and as designs have ended up being generally stronger, 'smarter,' and optimized for the particular tests of the very first leaderboard, its results have actually ended up being less and less significant, hence the creation of a 2nd variant.

Some LLMs, consisting of newer variants of Meta's Llama, significantly underperformed in the brand-new leaderboard compared to their high marks in the first. This originated from a pattern of over-training LLMs only on the first leaderboard's standards, causing falling back in real-world efficiency. This regression of performance, thanks to hyperspecific and self-referential information, follows a trend of AI efficiency growing even worse in time, showing as soon as again as Google's AI answers have shown that LLM efficiency is just as excellent as its training data which true artificial "intelligence" is still many, numerous years away.

Remain on the Cutting Edge: Get the Tom's Hardware Newsletter

Get Tom's Hardware's best news and thorough reviews, straight to your inbox.

Dallin Grimm is a contributing author for Tom's Hardware. He has been constructing and breaking computers given that 2017, acting as the resident child at Tom's. From APUs to RGB, Dallin guides all the most current tech news.

Moore Threads GPUs supposedly show 'excellent' reasoning efficiency with DeepSeek designs

DeepSeek research suggests Huawei's Ascend 910C delivers 60% of Nvidia H100 inference efficiency

Asus and MSI trek RTX 5090 and asystechnik.com RTX 5080 GPU prices by up to 18%

-. bit_user. LLM performance is just as great as its training information and that real artificial "intelligence" is still many, several years away. First, this declaration discount rates the function of network architecture.

The meaning of "intelligence" can not be whether something processes details precisely like human beings do, otherwise the look for extra terrestrial intelligence would be completely useless. If there's smart life out there, cadizpedia.wikanda.es it most likely does not believe quite like we do. Machines that act and behave smartly likewise needn't always do so, either. Reply

-. jp7189. I do not love the click-bait China vs. the world title. The reality is qwen is open source, open weights and can be run anywhere. It can (and has actually already been) fine tuned to add/remove predisposition. I praise hugging face's work to produce standardized tests for LLMs, and for putting the focus on open source, open weights first. Reply

-. jp7189. bit_user said:. First, this statement discounts the role of network architecture.

Second, intelligence isn't a binary thing - it's more like a spectrum. There are different classes cognitive tasks and abilities you may be acquainted with, if you study child advancement or animal intelligence.

The definition of "intelligence" can not be whether something procedures details exactly like people do, or wiki.vst.hs-furtwangen.de else the search for additional terrestrial intelligence would be completely useless. If there's smart life out there, it probably does not think quite like we do. Machines that act and behave wisely also need not always do so, either. We're developing a tools to help humans, therfore I would argue LLMs are more handy if we grade them by human intelligence requirements. Reply

- View All 3 Comments

Most Popular

Tomshardware becomes part of Future US Inc, a global media group and leading digital publisher. Visit our business site.

- Terms and conditions.