Source: Wall Street See

In a recent interview, $Microsoft (MSFT.US)$ AI Director Mustafa Suleyman delved into the latest trends in the field of artificial intelligence. He believes that in the coming years, AI models will show a trend of both large and small models advancing together.

On the one hand, the scalability competition of large models will continue and incorporate more modal data, such as videos, images, etc. On the other hand, the technology of using large models to train small models (such as distillation) is emerging, and efficient small models will play a huge role in specific scenarios. Suleyman added that in the future, knowledge will be condensed into smaller, cheaper models, embedded in various devices, to achieve a true revolution in environmental awareness.

For entrepreneurs, Suleyman believes that understanding and utilizing prompt engineering is crucial. By providing a high-quality instruction set, entrepreneurs can guide pre-trained models to align with their own brand values and create unique products. In addition, small models present huge opportunities, as entrepreneurs can leverage their low cost and efficiency to develop applications for specific use cases.

During the interview, Suleyman also emphasized the importance of data integration. Synthetic data will become a key for training models, but further exploration is needed on how to acquire and integrate this data.

Furthermore, the microsoft AI executive also discussed the introduction of new modes, such as the integration of video and image, as well as the understanding and data collection of action trajectories across complex digital interfaces. He believes that this will bring many impressive results. For entrepreneurs, the key to future success lies in how to leverage these new trends and technologies for innovation.

Below is the full content, enjoy~ (For enhanced readability, we have briefly edited the original text)

Q: In the evolving landscape of models in the coming years, what are some things worth paying attention to?

A: The scale of models is both increasing and decreasing, a trend that is almost certain to continue.

A new method called distillation has become popular since last year. These methods use large, high-cost models to train smaller models. The supervision effect is quite good, and there is already ample evidence to support this.

Therefore, scale remains a key factor in this competition, with significant room for future development and continued growth in data volume.

At least in the next two to three years, the progress of the 'scale law' in delivering better-than-expected performance will not slow down at all.

Q: What other new patterns can be included?

A: People are also considering integrating new modes such as videos, images, and action trajectories across complex digital interfaces into models.

But what we are truly interested in are the action trajectories across complex digital interfaces, such as transitioning from browser to desktop, then to mobile, switching between different ecosystems, whether in closed gardens or open networks.

We are trying to understand these trajectories, collecting a large amount of data, using methods such as supervised learning and fine-tuning. I believe this will bring many impressive results.

Q: In terms of data, in what areas do people not think enough?

A: There are many aspects to discuss about data, the classic issue being which data can be used and its quality. I think there has been a lot of discussion online.

However, people have not spent enough time thinking about the sources of new data and how to integrate this data.

For example, synthetic data is an interesting area, where if we have such data, we can train better small and large models. How to acquire this data and ensure its integration is a key issue. However, how to acquire this data and ensure they are integrated has not been discussed enough.

Q: When handling models, what is the difference between prompts and questions?

A: A prompt is not just a question you ask a chatbot. When you ask a chatbot a question, that's a question; when you write a three-page style guide with imitation examples, that's a prompt.

A prompt is your high-quality instruction set, guiding pre-trained models to behave in a specific way. Surprisingly, models can behave very differently with just a few pages of instruction.

In order to demonstrate subtle, precise, and brand-aligned behaviors in the model, you need to show tens of thousands of good behaviors examples and fine-tune these examples into the model. This is a continuation of the pre-training process, using high-quality and accurate data.

The good news is, tens of thousands of examples are very easy to obtain for many niche or specific verticals. This is an advantage, where startups have a lot of room to fine-tune pre-trained models with high quality.

Q: What opportunities do small models bring? How can entrepreneurs use them to do something interesting and unique?

A: Small models undoubtedly represent the future.

Large models activate billions of irrelevant neural representations when processing queries. While they efficiently search and reference millions of nodes, it is not always necessary.

We will condense knowledge into smaller, more affordable models that can reside on various devices such as earbuds, wearables, earrings, plants, or sensors.

This environmental perception revolution has long been anticipated, bringing functional devices like a refrigerator magnet, which is the smallest digital device I can think of. It can greet you in the morning, inform you about the weather, tell you what may or may not be in the refrigerator, and remind you to check your calendar.

It can greet you in the morning, inform you about the weather, tell you what may or may not be in the refrigerator, and remind you to check your calendar.

This model may only have a few tens of millions of parameters. Although no one is truly pushing this yet, any team of two people can explore this area.

Q: What questions should people think about in the next two days?

A: The question is, what do technologists need to do to design a more human-centered future.

This includes thinking about how technology evolves humans, and how our emotions, passions, and compassion are expressed through our ever-changing relationship with technology.

Q: Why is this called a moment of transformation?

A: We have enough evidence to show that the major technological transformations of the past fifty years have reshaped the structure of things.

I believe this is a moment to start a company, expand a company, or even change careers. Whether you are an entrepreneur, activist, organizer, or scholar, now is the time to pay attention.

By 2050, the train will have left the station, and things will be very different. We now have the opportunity to collectively shape and influence the future, nothing is predetermined. We are very fortunate to be alive at this moment, which is both a huge responsibility and an exciting opportunity.

Editor / jayden