Google has introduced new voice and video search features for Google Lens during the I/O 2024 event back in May. Now users can easily long-press and ask questions using their voice, making the search process much simpler and more convenient.
Custom Gemini Model Powers Video Search
Currently, these features are being rolled out in Search Labs for both Android and iOS devices. However, it's important to note that the voice search capability in Lens is limited to English queries at this time. The video search function utilizes a custom Gemini model that provides an AI Overview and relevant search results according to the video's content and the user's questions. During the I/O event preview, a demonstration showed a person interested in fish at an aquarium. By holding up their phone to the exhibit and using the Google Lens app, they could long-press the shutter button.
Enhanced Interaction with Google Lens
Once Lens starts capturing video, users can pose questions about what they observe. For instance, when asked, “Why are they swimming together?” the Lens responded through Google Gemini. This video search capability allows users to present their phone with moving objects and inquire about them, enhancing the usefulness of Google Lens in various situations. To access this feature, users can participate in the “AI Overviews and more” experiment within Search Labs.
Rajan Patel, Google’s vice president of engineering, explained how the feature operates. Google captures the video as a series of image frames, applying existing computer vision techniques used in Lens. Importantly, the responses are generated by a custom Gemini model designed to interpret multiple frames in sequence. Once the frames are processed, the model pulls relevant information from the web to formulate an answer.
In conclusion, this development effectively utilizes existing technology, adding significant value to Google Lens.
