Source: Geek Park Author: Jesse Apple WWDC, How AI Enters iPhone and Mac. Apple has finally revealed its progress in AI. Compared to Google's I/O conference last month, which mentioned the term AI 121 times, Apple hardly mentioned AI on the eve of WWDC 24. However, this WWDC conference was already destined to focus on AI. Until the conference was half-over, Apple finally showed off its main course, Apple Intelligence. Although it can also be abbreviated as AI, Apple finally chose to name its AI as 'Apple Intelligence.' During the introduction, Cook particularly emphasized that they wanted to build 'Personal Intelligence' beyond Artificial Intelligence. Unlike other large model development concepts that 'perform miracles,' Apple has adopted a relatively cautious and more granular development model, defining each function more clearly, but also appearing less 'magical.' At the conference, Apple announced its cooperation with OpenAI. Users can directly call the GPT-4o interface through Siri for dialogue, but did not make a deeper integration of GPT with phone functions. In any case, Apple represents one billion mobile device users, the highest frequency of use scenes. This may be the beginning of AI truly becoming 'useful.'
The weather is good today The weather is good today.

Apple finally revealed its progress in the field of AI.

During the WWDC24 on June 10th, almost no AI was mentioned. Although this WWDC conference was already destined to focus on AI. Compared to Google's I/O conference last month, which mentioned the term AI 121 times.

Until the conference was half-over, Apple finally showed off its main course, Apple Intelligence. Although it can also be abbreviated as AI, Apple finally chose to name its AI as 'Apple Intelligence.' During the introduction, Cook particularly emphasized that they wanted to build 'Personal Intelligence' beyond Artificial Intelligence.

Unlike other large model development concepts that 'perform miracles,' Apple has adopted a relatively cautious and more granular development model, defining each function more clearly, but also appearing less 'magical.'

At the conference, Apple announced its cooperation with OpenAI. Users can directly call the GPT-4o interface through Siri for dialogue, but did not make a deeper integration of GPT with phone functions.

In any case, Apple represents one billion mobile device users, the highest frequency of use scenes. This may be the beginning of AI truly becoming 'useful.'

What is Apple Intelligence?

Combining AI with smartphones is not new, and Apple's approach is still similar to that of other Android manufacturers.

Simply put, mobile phone manufacturers need to transform the input end of AI from user input prompt to the model actively sensing 'everything that is happening' on the phone as a prompt. At the same time, the output end of AI should be transformed from simply outputting information to calling Apps to achieve various behavioral functions.

Apple used five words to summarize the characteristics of AI: strong performance, intuitive ease of use, function integration, personalized customization, and privacy and security.

In terms of functions, Apple mainly builds the functionality of Apple Intelligence around three modules: text, image, and interaction.

The text function is mainly focused on two scenarios: SMS and email. Users can select a piece of text and perform various AI optimizations, including syntax checking, rewriting, generating summaries, key points, lists, etc. Users can also use natural language to describe their own rewriting needs, such as making them more concise, changing the tone to friendly or casual, or more professional.

Based on this language model, Apple also made phone recordings, converted them to text, generated summaries, and automatically recognized the content of SMS and email, prioritizing important content and labeling it as 'possibly important.'

The image function mainly involves 'generating images.' Apple Intelligence can automatically extract text in various scenarios as prompts to generate images. For example, when sending a message, it can generate images and emojis (which Apple calls genmoji) based on the input text, and on the iPad, it can extract key information from the user's sketches or written text to generate images. Apple has also added a lot of 'photo smart removal' features that other phone manufacturers have already done, which can remove irrelevant passers-by in photos and automatically generate some image content to fill in the blanks.

Finally, and relatively importantly, Apple said that AI will usher in a new era of Siri. Siri will be able to understand more complex language texts and have a more natural interaction. Moreover, Siri will automatically use various information on the device to 'understand the context' and better interpret the user's needs.

This part of the function is described as being more complicated. For example, if a user forgets their ID number, they can ask Siri to retrieve the passport photo from their album, extract the number, and ask the user to fill it in. However, the information Siri can perceive will include calendars, memos, messages, emails, and more...

In addition, if a user doesn't know how to operate a certain function, they can describe it in natural language to Siri, who will help them find the corresponding operation method. This will also greatly change the way users use their phones.

Finally, Apple has also integrated ChatGPT into its intelligent system. In the process of processing text and using Siri, if users want to call on a more powerful cloud-based model, they can switch to using GPT-4o to generate information, or bind their own GPT Plus account to use other models.

Make AI finer

Since AI needs to use the user's most sensitive personal information as input, Apple naturally regards privacy protection as a top priority.

Apple's intelligent model runs locally on the device. Apple has not disclosed any information about this terminal-side model, but only the iPhone 15 Pro series equipped with the A17 Pro chip and the Mac and iPad equipped with the M1 or higher chip support Apple AI. It can be seen that it has high performance requirements for chip NPU.

However, Apple AI still cannot achieve 100% local operation. When necessary, it will also use cloud computing power to process information. Apple stated that during this part of the processing, no user data will be retained. It will only upload information when the user actively requests it, and security is ensured by a verifiable privacy mechanism. Apple calls this system "Private Cloud Compute".

As for calling GPT, Apple will confirm with the user before they choose to use it and will prompt that the results returned by GPT may not necessarily be accurate.

Therefore, Apple AI can be divided into three systems: Apple's self-developed terminal-side model, cloud-based models, and GPT. This architecture itself is not very complex, but there are still many details that need to be resolved.

The most important issue is how AI will use third-party apps. Apple has already released the first batch of AI associated development tools. Developers can use APIs to integrate their app functionality into the AI system. This development mode is similar to the past "shortcut instructions", but in reality, developers, especially large companies, may not be very enthusiastic about adaptation.

For example, it may be very convenient to use Apple's AI to search for a certain chat record or picture in iMessage from the first day of AI launch. But if a user asks Siri what time a certain colleague arranged to meet them on Feishu, WeChat, etc. while scheduling a calendar, the AI system may have difficulty retrieving this information.

Similar issues are also reflected in how this "personal information" can be transferred across devices. For example, if a user wants to ask Siri about their recent exercise status on a Mac, but the exercise and health information is stored on their iPhone. Apple has not announced any cross-device information transfer mechanism, so it may not be possible to obtain this information.

The biggest revolution of large models lies in their possession of a "complete knowledge base". Because this knowledge base is nearly comprehensive, it reflects the emergence of the capabilities of large models, which makes people feel that no matter what they ask, and how they ask it, it can always give an answer. And to bring this system to the phone, to integrate all of a user's personal information into a "knowledge base" for them to access at any time, while ensuring privacy and security, will be a huge challenge in product design and logic.

Because this knowledge base covers almost everything, it is reflected in the emergence of large model capabilities, making people feel that no matter what you ask it, how to ask, it can always give an answer. Moving this system to a mobile phone requires integrating the user's entire personal information into a "knowledge base" for users to access at any time, while ensuring privacy and security. This will be a huge challenge in product design and logic.

To break through the bottleneck of user adoption for any product to become popular, it is essential to ensure the success rate of interaction and that users get what they want. Just like in the past, the main reason why voice assistants never really became popular was because of the high failure rate of interaction. Even in the AI era, it still faces the same problem.

Apple has just laid the groundwork for AI.

What can domestic users expect in 2030?

According to information released by Apple, the first batch of Apple Intelligence will only support English, and the test version will only be pushed in the autumn. The first wave of developer versions for the new system that has already been pushed out does not yet have Apple's AI.

Apple said it will add more features, languages, and platforms next year. It is likely that in Apple's AI system, GPT can eventually be replaced just like Safari search engine. Previously, news broke that Apple is communicating with Baidu about cooperation on large-scale model interfaces, and this is probably the part that will be replaced.

Since both the core on-device model and cloud model of Apple's AI are independently developed by Apple, as long as they can pass domestic compliance processes, domestic users still have a good chance of using them smoothly.

The imagination of using the power of models to subvert the interaction interfaces of computing devices and reconstruct the personal data organization structure has existed since the birth of large-scale models. However, most companies that build models and applications find it difficult to penetrate the core system of computing devices, including privacy and security issues that make companies such as Apple have to remain extremely cautious.

But now, no matter what, this ship has already set sail.

Editor/Somer