Revealed by 9to5Google in the latest version of the Insights APK, a deep dive into Insights was a feature of the Nest Hub Max called “Look and talk”. Dubbed “Blue Steel” as a reference to the Ben Stiller movie “Zoolander,” the feature has just been officially revealed at Google I/O 2022, a small part of a much larger initiative where Google hopes to make its Assistant more natural and comfortable to interact with.
Find a Nest Hub Max and talk to him
The goal is to easily start a conversation with the Google Assistant on any device and speak naturally and understand you. Our voices are becoming the fastest way to take advantage of computing and process queries, but it’s not perfect yet. For example, having to say “Hey Google” every time you want to ask the Nest Hub or phone for something – anything at all. Look and Talk solves this problem by letting you look at your Nest Hub Max from up to five feet away and activate it before you speak.
You can stop worrying about the right way to ask for something, relax and speak naturally
Sisie Hsiao, Vice President/General Manager, Google Assistant at Google
To achieve this, the assistant uses six machine learning models to process more than 100 signals from both the Camera Hub and microphone in real time. Things like the direction of your head, proximity to the device, the direction of your gaze, how your lips move, and any other contextual awareness necessary to accurately process your request.
Let me start by saying that this is totally wild. We all knew the need for an important word would one day disappear in the background of smart home use, but until now, we weren’t really sure how that would happen and we had reservations about how Google would do it in a way that was efficient and respectful of user privacy.
With regard to Look and Talk, it is considered safe and secure, all data is processed on the device, and will never be sent to Google or anyone else. It also uses both face matching and voice matching at once, so it only works if you realize that it’s really you and that you’re making the request when you look at your device, which is very clever.
The assistant will handle pausing and word filling in voice requests
One of the biggest gripes I have when trying to teach my family how to use the Google Assistant properly is that they repeatedly stop in the middle of their requests, and the assistant thinks they’re done talking. He will process the matter, come back with results and not fulfill their request, and make them frustrated. In fact, they hadn’t quite finished speaking, but instead, they were simply thinking about the appropriate word to use. This way, millions of users worry about how to use the assistant because they feel they need to master their thoughts before speaking. This is annoying and inconvenient.
To fix this, Google is building a more comprehensive neural network running on its Tensor chip as well as implementing a more courteous and patient AI assistant. Instead of assuming the user has finished speaking just because there is a bit of silence, it will process what was spoken and see if it made sense or was a complete sentence before closing the microphone. If they are not quite sure that the user has conveyed a complete idea or request (read: if they use awkward pauses or filler words like “ummm” while thinking out loud) he will gently encourage them to finish by saying “mm hmm”.
In addition, it appears that Google Assistant will also gain a greater sense of how to complete a user’s ideas on their behalf. Very polite, isn’t it? When Sissie Hsiao showed this on stage, she used the word “something”, and in its place, the assistant figured out how to complete the song title and decided to autoplay it on Spotify!
Basically, that means in the sense of “Yes? What else? Please go on…” Admittedly, I was over the moon when I saw this shown live at Google I/O 2022, and it’s probably the only update that solves most My smart home frustrations.
Quick Phrases and Real Tone Apps
Quick Phrases – Google’s first app to eliminate the need for the hot word “Okay Google” for specific tasks like setting timers, toggling smart lights and more – is being expanded to the Nest Hub Max! Finally, Real Tone, an effort to improve Google’s camera and photo products across skin tones to properly represent users of diverse backgrounds, will work with the Nest Hub Max for Look and Talk. More work on this feature is being done with the new Monk Skin Tone Scale launched today (really, it just got unlocked thanks to its creator, Harvard Professor Dr. Ellis Monk) Google AI research on skin tone aims to help improve the assessment of skin tone in Machine Learning provides a set of recommended practices for use in ML Fairness.
If you want to watch the entire Google I/O 2022 Keynote program, you can do so below. We also have plenty of coverage of everything that was announced and revealed yesterday, so please be sure to check it out! Let me know in the comments which of these Google Assistant features are you most excited about, and whether you think Google is making history or playing with fire when it comes to AI and machine learning.