For developers, we believe that camera access unlocks the full potential of AR. Today, we're launching Snap OS v5.58, which includes new Lenses that showcase the power of the Spectacles camera and SnapML, as well as platform features and capabilities to help you better build and experiment with Spectacles Lenses.

Introducing Piano Tutor and Ball Games

The Piano Tutor Lens puts a personal instructor and interactive courses at your fingertips. Powered by SnapML, the Lens uses a custom machine learning model to identify and track a variety of pianos, overlaying notes onto the keys in real-time. This helps you build awareness of the keyboard layout and coordinate your movement accordingly. Using another custom ML model, the Lens can detect how well you play and provide real-time feedback.

The Ball Game Lens is a game that transforms a physical ball into a controller, making it easy and fun to practice your skills through a set of different challenges. Using a custom ML tracking model, the Lens follows the ball's movements, allowing you to interact with physical and digital elements together and master the perfect kick.

New Platform Features and APIs

To unlock the ability to experiment further in development, we're introducing new APIs that make it easy to use camera data and multi-modal Large Language Models (LLMs) hosted in the cloud. Using extended permissions developer settings and our new Fetch API, you can now build Lenses that directly interface with hosted multi-modal LLMs.

We're also introducing an Image Spatialization API, which leverages generative AI to convert common 2D image formats into 3D. Developers can now incorporate this API into their Lenses to create stunning 3D effects with such realistic depth you'll feel like you're truly in the moment.

Additional platform capabilities include:

Spatial Anchors: Anchor digital objects to specific locations within a Lens and return to them later

Basic Location: Get access to the GPS coordinates of your Spectacles to create more realistic outdoor experiences

Layout Mode: Import images from your phone using the Spectacles app and spatialize them for a captivating 3D experience

Web View: Drag and drop embed web pages directly into your Lenses

Lens Unlock: Unlock a Lens that has been submitted to Spectacles, but hasn't yet been added to a category or Lens Explorer, by simply sharing the URL

To experience these Lenses and start building using the new features, update to the latest version of Snap OS and the Spectacles App using these instructions. We can't wait to see what you create!