Mon. Dec 23rd, 2024
Transformers Probably Won't Make Ai As Smart As Humans. There

The transformer technology that powers tools such as ChatGPT may have limitations.
Florence LO/Reuters

  • ChatGPT has changed the conversation around AI.
  • However, the technology that supports this has its limits, and it may be difficult to create AI that is as smart as humans.
  • Researchers are currently considering alternatives.

of A groundbreaking effort by many Googlers Introducing Transformers to the world in 2017 — neural network that Power up today’s popular AI products.

They are Powers OpenAI’s Large-Scale Language Model (LLM) under ChatGPT. The chatbot exploded onto the scene last year, prompting Bill Gates to declare that “the age of AI has begun.”

The mission for some AI entrepreneurs is now to realize their sci-fi vision and Artificial general intelligence (AGI): AI that appears to be as intelligent as humans.

However, although a transformer can power ChatGPT, Preprint paper published by Google researchers Last month, it was suggested that they may not be able to abstract, extrapolate, and predict the way humans do, which suggests we are in AGI.

ChatGPT simply uses human-trained data to respond to user prompts with text. In its initial publication format, the chatbot had no knowledge of events beyond September 2021 and needed to recognize it every time someone asked a question about a recent topic.

Google researchers tested Transformer’s ability to move beyond data and described its “generalization even in simple extrapolation tasks” as “degraded.”

This raises the question of whether human-like AI is even possible. The other is whether different technologies can get us there.

Some researchers are testing alternatives to figure it out, and another new paper suggests a better model may be in the works.

the study Submit to open access repository ArXiv On December 1, Albert Gu, an assistant professor in the machine learning department at Carnegie Mellon University, and Tri Dao, principal scientist at Together AI, announced a model called Mamba.

Media not supported by AMP.
Tap to get the full mobile experience.

Mamba is a state-space model (SSM), and according to Gu and Dao, can defeat transformers On the performance of a series of tasks.

Note: Research submitted to ArXiv is reviewed, but not necessarily peer-reviewed. This means the public can see the research sooner, but it’s not necessarily reliable.

Similar to LLM, SSM can perform language modeling, the process by which chatbots such as ChatGPT work. However, SSM does this using a mathematical model of the various “states” that a user’s prompt can take.

Gu and Dao’s research states that “Mamba delivers state-of-the-art performance across several modalities including language, speech, and genomics.”

When it comes to language modeling, Gu and Dao said Mamba “outperforms transformers of the same size and is comparable to transformers twice its size, both in pre-training and downstream evaluation.”

In his article about X, Dao also mentioned what the unique features of SSM mean. Mamba can generate language responses five times faster than Transformers.

Media not supported by AMP.
Tap to get the full mobile experience.

In response to the, Dr. Jim Huang, a research scientist at the software company Nvidia, writes about X: “I’m always excited about new attempts to dethrone transformers. We need more of these,” he said.

He gave “kudos” to Dao and Gu, who have “promoted alternative sequence architectures for many years.”

ChatGPT was a groundbreaking cultural event that sparked the AI ​​boom. But the technology is unlikely to lead the industry to the promised land of human-like intelligence.

But if repeated tests confirm that Mamba consistently outperforms Transformer, the industry could come even closer.