Source: Quantum bit.
The long-awaited Apple WWDC is about to happen, and all the details of AI upgrades have been leaked!
Well-known website Apple Insider received the exact details of the new features and summed up the comments: Siri will be reborn in iOS 18.
From camera albums, calendar memos to browser emails... almost all native applications in the system will be fully armed with AI.
Not only that, Bloomberg revealed that Apple will launch the full ecosystem AI capability for iPhone, iPad, and Mac, named Apple Intelligence:
It will automatically choose between running local AI or cloud-based AI. Apple's focus is on practical daily functions, leaving fancy chatbots to OpenAI.
Many netizens exclaimed that Apple Intelligence is the smartest marketing tactic and Cook has hijacked the word AI, replacing Artificial with Apple.
In addition, the shadow of the new features can also be seen from the papers published by the Apple ML lab, such as the "one-sentence image editing" technology, which is rumored to be implemented in this update.
As usual, the summary is presented first, and specific details will be discussed later.
Camera: Take photos/videos in specific modes, one sentence is all it takes to set the countdown and switch between front and rear cameras.
Photos: Edit and organize albums, find specific people, pets, and scenery based on keywords.
Memo: Record to text, summarize speech, and identify formulas, an all-around learning note-taking tool.
Voice memo: Record, transcribe, archive, and organize files, making meeting notes more efficient.
Reminders: Create to-do lists and tasks, and label, organize and categorize them.
Mail: Compose emails, save drafts, set reminders, and automatically categorize junk mail.
Books: Navigate to specific books and chapters, flip pages and change themes with one button.
Safari: Summarize web pages, and organize tabs.
Keynote: Insert audio, video and images with one sentence, and automatically edit slides.
More intelligent camera and albums.
The new version of Siri will be able to directly control the camera, and users can complete camera mode settings and other operations through Siri, specifically including:
Switch to video recording mode.
Open the camera to a specific mode, such as photo, portrait, video, slow motion, etc.
Set the timer.
Switch to front/rear camera.
In addition to the camera, Siri will bring significant enhancements to the Photos application, allowing users to select and edit, move, and hide photos or albums through Siri.
In addition, Siri can recognize specific objects, landscapes, or people in photos through Visual Look Up, which means users can request to view photos of someone or a particular scene.
Users can also choose to edit photos, videos, screenshots, and other content—all of which can be done through Apple's virtual assistant, Siri.
In the Photos app, Siri will be able to:
Search for and open photos containing specific objects in the local and network.
Rotate, copy, move, hide, and favorite photos.
Create, open, rename albums, or add photos to them.
Create new memories, albums.
Open photos and videos for editing, apply filters, enhance photos.
In addition, Apple Photos will also make some minor UI changes, introduce new AI editing features, and launch the Clean Up tool that supports removing specific objects from images.
Manage notes and recordings with Siri.
Siri enhancements in the Notes app will make browsing, marking, and moving notes easier.
In Notes, Siri with Apple's new large model support will be able to:
Create, rename, and delete annotation folders.
Add or remove tags for notes, and filter, open, create, and delete specific tags in memos.
Open the content under a specific account (such as iCloud, Outlook, etc.) in the Memo app.
Open, move, delete, pin/unpin specific notes.
Switch between note view and folder view.
The new version of Memo app is also expected to have in-app recording, audio transcription, and summary functions supported by large models, allowing users to store recordings, summaries, and transcripts together with other materials (such as text or images) in a note.
Math Notes is another upgrade prepared by Apple for Memo app, which will integrate the new GreyParrot calculator to help users embed complex math equations and charts into personal notes.
The keyboard math prediction function is also under development, which will allow Apple software to recognize mathematical expressions as part of text input and automatically complete mathematical equations through AI.
In addition to Memo app, new features will also be launched in the Voice Memo app (recorder). Through Siri, users will be able to create, pause, delete and move individual recordings or recording folders.
With the Voice Memo app, Siri will be able to:
Search for and open personal voice memos/recordings.
Create or pause new recordings.
Create, open and delete folders in the voice memo.
Delete specific recordings by name, creation date and audio content.
The new features will make it easier for users to find the specific files they may be looking for, as Siri will be able to search for recordings by name, date, or location, and is expected to receive in-app audio transcription as well.
Process emails, make PPT, and make office work more efficient.
In addition to Memo app, Apple's Mail app will also undergo major changes, also incorporating AI function to automatically categorize emails based on text content.
Specifically, the Mail app will be able to classify emails into business, news, social, and other categories, and Siri will be able to use this information to execute different actions within the app.
For example, identifying emails with promotional codes or emails from specific companies and marking them as spam.
In the mail application, Siri will have the ability to:
write, send/cancel emails, or set timed sending
save emails as drafts or delete drafts
mark emails as spam or set reminders, block, or delete emails from specific senders/specific subjects
reply to emails or all recipients based on AI smart replies
mark emails as "read later" and set specific dates or cancel settings
unsubscribe from emails, especially marketing-related emails
enable notification settings for emails from specific senders or related to specific subjects
summarize email content, archive, move, and forward emails
Apple's mail application will also feature a "smart reply" function, which allows users to send AI-generated email responses.
In the office, it's not just email; the new version of Keynote also features AI-related functions—through Siri, users will be able to easily view Keynote presentations and make changes themselves.
Through a virtual assistant, users can insert audio, images and videos into Keynote, making PPT creation, modification, and presentation even easier:
Adding images, audio and video, text boxes or comments to specific slides
create new PPT or new pages showing specific slides
Playback and stop theme presentations
Show the activity flow of the theme presentation, display the changed content as well as the changers
Add online videos from Safari to slides
Set titles and highlights for specific slides
create a reminder list with Siri
Siri's reminder function will help users create lists such as household tasks, books to read, study materials, travel arrangements, and more. Users can also add or remove topic tags in specific reminders, and generate reminder summaries.
Using this feature, enterprise users can create reminders for the people they need to meet, while students can list the items they need for the new school year.
According to Apple Insider, reminder items are expected to be integrated with Apple's default calendar application, which means users will be able to create and edit reminders directly from the built-in calendar application, with supported operations including:
Create new reminder lists or parts thereof
Add or remove topic tags from specific reminders
Display reminder list/summary
Siri one-click summary webpage
With the upgrade of Siri, Safari has also been significantly improved.
According to rumors, Siri will introduce a new intelligent browsing function, which will enhance search and provide article summaries.
There is also a feature called Web Eraser, which is a content blocker. However, this feature has caused controversy, with the News Media Association in the UK and some French publishers complaining to Apple about it. It is reported that the feature has been built and is available, but Apple may remove it before WWDC to avoid controversy.
Other products
Siri can also open different areas of the system settings, such as brightness settings, main screen and application library settings, as well as multitasking and gesture settings. The system settings application is also expected to be re-designed in iOS 18 and macOS 15.
At the same time, Siri can scan documents in the "Files" application and move specific files to the trash. It can also edit the detailed information of contacts in the address book.
In the magnifying glass function, Siri can describe objects, identify people in the field of view, and use Apple's existing frameworks and visual lookup functions, which is similar to the recognition method of products such as Rabbit 1.
In addition, Siri can generate various images through Apple's internal Generative Playground application, but it is currently unclear whether end users can use this function.
"Project Greymatter"
It is worth mentioning that some of Siri's new features have been put into "Project Greymatter," including generating articles, emails, and summaries of messages and notifications.
A feature called Greymatter Catch-Up will provide users with a review of the latest notifications.
It is understood that Apple's artificial intelligence software can generate summaries of different levels, ranging from three key words to three sentences.
Furthermore, Siri, Spotlight, and Mail will all gain advanced search capabilities and better contextual understanding through Apple's ai software. When creating responses, Siri will be able to consider different entities, such as people and companies, places, and calendar events.
According to insiders, the generative ai software will also allow for instant creation of smart responses, which will be offered in emails, Siri, and messages.
Finally, sources from AppleInsider emphasized that not all features will be released.
The relevant news comes from individuals familiar with pre-release versions and test environments of Apple's operating system, and may not reflect the final release version.
Editor/Lambor