"OpenAI's new generation large model Orion did not have a significant breakthrough", "Anthropic postponed the release of the new model Claude", "Google's upcoming new version Gemini did not meet expectations"...

Recently, multiple media outlets successively disclosed that AI companies are facing widespread technological upgrade bottlenecks. Words like "postponed", "questioned", and "fell short of expectations" frequently appear in reports. In the current era where AI is becoming more accessible, these AI companies seem to be struggling with upgrade dilemmas.

According to a report by BusinessInsider on November 27, advances in AI technology are slowing down. The main challenges facing this field include bottlenecks in large model performance improvements, shortage of training data, quality issues with training data, and obstacles in improving reasoning capabilities.

However, top companies such as OpenAI, $Alphabet-C (GOOG.US)$ firmly assert that AI has not encountered the so-called "barriers" and "bottlenecks". They remain optimistic about the future of AI and believe that by developing new data sources, enhancing model reasoning capabilities, and utilizing synthetic data, AI models will continue to make progress.

One of the first to speak out was OpenAI's CEO Sam Altman, who said on social media earlier this month: "There is no bottleneck". Anthropic and$NVIDIA (NVDA.US)$The CEO also stated that the progress of AI has not slowed down.

AI Dilemma

Currently, some individuals including Marc Andreessen question that the performance improvement of AI models is not significant and tends to homogenize. For the technology industry, this is a trillion-dollar value problem because if the returns from existing AI model training methods diminish, it could affect the investment boom in new startups, products, and datacenters.

According to BusinessInsider's analysis, the widespread dilemmas facing the AI field include depletion of training data and hurdles in performance improvement.

In the early stages of AI research and development, enterprises may encounter two main bottlenecks: computational power and training data. Firstly, the limited ability to acquire dedicated chips (such as GPUs) affects training of large models. Secondly, the bottleneck of training data is gradually becoming apparent, with publicly available data resources on the internet gradually depleting. Research institution Epoch AI predicts that by 2028, the data available for training may be exhausted.

Data quality has also become a major issue. In the past, researchers could tolerate lower data quality in the pre-training stage, but now there is a need for more emphasis on the quality of data, not just the quantity.

The improvement and breakthrough in inference capabilities are considered the next key direction for AI development. Former Chief Scientist of OpenAI, Ilya Sutskever, mentioned to the media this month that the scale expansion of models in the pre-training phase has reached a plateau, and everyone is looking for the next breakthrough.

Meanwhile, the upgrade costs of AI are continuously rising. As model sizes increase, computing and data processing costs significantly rise. According to the CEO of Anthropic, a complete training process in the future may require investments as high as $100 billion, which includes massive costs for GPUs, energy, and data processing.

Major companies are breaking through barriers.

Faced with skepticism, major AI companies have successively proposed their own plans to address the bottleneck of AI development.

Currently, multiple companies are exploring the use of multimodal data and private data to address the problem of insufficient public data. Multimodal data involves feeding visual and audio data into AI systems, while private data is obtained through licensing agreements with publishers. At the same time, improving data quality has become a focal point of research, with synthetic data (data generated by AI) emerging as a possible solution.

In addition,$Microsoft (MSFT.US)$Companies like OpenAI are striving to give AI systems stronger reasoning capabilities, enabling them to provide deeper analysis when facing complex problems.

• OpenAI: Through partnerships with organizations such as Vox Media and Stack Overflow, they are acquiring private data for model training. In addition, they have introduced a new model, o1, in an attempt to improve reasoning abilities through 'thought'.

• Nvidia: Overcoming supply constraints to ensure the supply of GPUs for supporting AI model training.

• Google DeepMind: The company's AI lab is adjusting its strategy, no longer simply pursuing the expansion of model scale, but focusing on specializing in specific tasks more efficiently.

• Microsoft: At the recent Ignite event, CEO Satya Nadella mentioned that they are researching a new "test time computation" mode, allowing models to spend more time addressing complex issues to enhance reasoning ability.

• Clarifai and Encord: are exploring the use of multimodal data to overcome public data bottlenecks. Multimodal data combines visual and audio information, providing AI systems with more diverse data sources.

• Aindo AI and Hugging Face: are studying synthetic data to improve data quality.

Editor/Rocky