By 
Android users who have grown to love their voice commands could be in for a treat: Google’s been working on some crazy improvements. Recent updates have already improved how voice search works with Google Now, tightly integrated voice commands with Android Wear, and added an always listening “Ok Google” voice command to all search screens, but perhaps the best is yet to come.
Deep inside a September 4th patent – titled “detecting the end of a user question” – we find some hidden gems that show us how Google could soon add “active watching” technology to futurize Google Search and beyond. But at what cost?
The patent starts off fairly straightforward, describing some features already available in some Android devices such as the new Moto X (2nd gen). The phone uses multiple microphones to detect the location of voices, enabling it to ignore speech input coming from unintended sources. But the patent takes an interesting turn when it mentions capturing visual indicators, and not just any visual indicators- video.
The digital capture device may be a digital video recorder, digital camera, a webcam, etc. The visual capture device may capture visuals and represent the visuals as a stream of images that may form a video.
On the surface (at least to some) this might seem innocent and run-of-the-mill: earlier this year we showed you some Galaxy S5 Tips & Tricks which include a Samsung feature called Smart Stay that keeps the screen on whenever it detects you’re looking. As depicted below, Google is suggesting combining visual indicators with audio indicators to improve Google Search functionality. It doesn’t take long for things to get a lot more interesting, though.
Google attempts to detect “deliberation” between people with audio/visual indicators, and depending on what their algorithmic statistical mojo recommends, can offer answers as if engaged in an ongoing dialogue or choose to stop actively listening altogether.
Capturing video and identifying people
This Google patent goes deeper though, not only actively watching and capturing visuals, but collecting a bunch of other information along the way.
“the visual analyzer may determine the number of people in an area represented by the visual data, the identity of the people, the vertical and horizontal angles of the heads of the people, and lip movement of the people”
Determine the identity of people? Yes… and then using audio and video together they can further:
“determine the identity of the person providing the voice input based on the lip movement of people and the acoustic characteristics of the voice”
If you thought Google only wanted to extrapolate this data to differentiate between the different speakers it was hearing, you’d be wrong… it doesn’t stop there. Not only does it watch, listen, capture information about people it successfully identifies, it then stores this data in user profiles. Directly from the patent:
“the system may analyze audio and visual data and store information in a user profile…”
Google even describes some examples of the data it may want to collect and store, further explaining the data would be used to serve content that is more relevant to the user:
- User’s social networks
- Social actions or activities
- Profession
- User preferences
- Current location
My reaction when reading this? Awesome!
What’s your reaction? Today’s Google searches at Google.com are already able to identify this type of information and store it in your Google Account history to improve performance and features, but that won’t stop some people from feeling paranoid. This patent is open ended and far reaching; I’m sure plenty of freedom fighters have already rushed to the comments in defense of our civil liberties and protection of our privacy.
Ok Google… stop reading my lips
There are often two opposing camps in the privacy debate:
- If you’re not doing anything wrong you shouldn’t have to worry
- Private life should be private. After all, what happens if Google gets hacked, or shares my info with law enforcement, or people with access to this information abuse their privileges?
Regardless to which camp you belong, Google has made it clear in their patent that these settings will be optional, allowing the user to choose whether or not their personal information is stored:
“For situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect personal information”
Notice the users “may” be provided with- it’s not necessarily guaranteed. They also mention anonymizing data so that they can still collect it in aggregate without connection to personally identifiable information.
Patents serve to protect intellectual property, not as an operating guide, so critiques in advance of a formal Google announcement should be hypothetical. Google would attempt to address privacy concerns once implementing, and if overlooked, they’d be sure to face backlash.
What type of backlash? Probably the type Microsoft faced earlier last year, leading up to the launch of the Xbox One and Kinect. The main issue? Kinect was always listening, always watching, and you couldn’t turn it off. Sound familiar?
Microsoft later backtracked on that demand and most of the Kinect privacy hysteria has subsided. Dissenters likely purchased the Playstation 4 or Nintendo Wii U instead… and the world went on.
Whether the tech from this patent sees the light of day remains to be seen. If it does, privacy will certainly be an imperative issue to discuss, and I urge you to begin that discussion in the comments below. But I’m hopeful that Google would implement it responsibly. A more interesting discussion, I believe, is what this patent could mean for the future of Android devices.
How will Google use this data?
I love Google voice commands and it’s a feature I use daily. If you don’t, you really need to try it. It’s not perfect though and can be especially irritating when you’re not the only person in the room. At the very least, Google’s hopes of improving voice commands through visual indicators is promising.
This patent could mean much more than voice search improvements and its parallels with Kinect aren’t only in the privacy department. When most people think “Android” they think “smartphone” but Google’s scope is much broader and motives more sweeping. The “detecting the end of a user question” patent may feature a mobile phone in its illustrations, but it explicitly mentions computers, web cams, and other types of video and audio equipment that can collect information from what seems like a much larger and more physically static area than is likely with your phone.
Three obvious places (beyond smartphones and laptops) where Google could awesomely employ these features to create stunning new experiences:
- Android TV
- Android Auto
- Android @Home
Television is still in the stone ages, begging to be revolutionized. Google’s initial attempt – Google TV – failed quite miserably, but they’ve since announced Android TV. In its current alpha form it’s a direct competitor to products and services like Apple TV, Amazon Fire TV, and Netflix, but it could be so much more. Advanced voice operation that smartly “lives” with you and the people in your house could make the difference between a cheap set top box and truly next generation multimedia solutions. At the ground level, consider an Android TV that greatly improves upon the voice functionality already found on Xbox and PS4.
The auto industry hasn’t changed a whole lot in the past few decades, either. Google isn’t waiting two decades for their self-driving cars to become a reality, they’re launching consumer vehicles with Android built-in later this year through Android Auto. What interesting experiences, apps, and games could Google create by knowing who is in the car, where each person is sitting, when each person is talking, and what they’re each saying? That’s a challenge I’m sure developers would love to tackle.
Then we’ve got the ever-cliche, George Jetson style “home connectivity” vision. We’ve been hearing about and seeing Android appliances since 2010, but even with the advent of Android @Home, truly connected homes have made few inroads into your typical homes. Google has shown a recommitment to home connectivity after buying Nest for $3.2 Billion. Being able to communicate with your home, hands-free and with great accuracy, could be the missing link in helping the connected home emerge as the next cultural revolution.
Creepy? Awesome? Or both?
The three main takeaways from this article (and Google’s patent):
- This could help immediately improve Google Voice Search
- Extending the idea could revolutionize voice commands across many devices
- There will be no lack of privacy concerns
Using visual queues and pairing them with audio queues is a brilliant way to improve an already wonderful product, but is capturing video, listening to voices, watching lips, identifying real people, and correlating it with personally identifiable information going further than you want your relationship with Google to go? Let us know in the comments!
Note: the term “active watching” is not used by Google in this patent. They do, however, call the existing audio functionality “active listening”. I’m using the term “active watching” for this article as a logical extension of an already understood and well accepted concept. In reality, I’d hope Google would announce this feature using a term that seems less intrusive, such as “active aware” or “always aware” (which could include both audio and video).
I’m typically one of the more lenient when it comes to Google knowing about ME- afterall I buy their goods and services with the information that I provide them (If you aren’t paying for the product, you ARE the product..)
But where I draw the line is when google starts identifying and collecting information on OTHER PEOPLE through my device. Like my wife. She has a gmail account but she doesn’t use it. She doesn’t want to get on G+ because, well, she’s just too busy… I don’t want Google using my devices to collect information on other people.
This is a great point and one I planned to make in the article. I failed to… so I appreciate you making this distinction. It’s an interesting one from a legal standpoint.
This is going to end up like Facebook’s shadow profiles. Facebook has collected and stored information on almost everyone – even people who don’t use the service – by correlating information from users who let Facebook read their contacts on their phone or in their email. It’s a big problem, especially when it cross pollinates recommendations and contacts between groups that are intended to stay separate, like personal and professional lives.
@Defenestratus:disqus
I feel the same way. I am afraid that once battery life and data storage is no longer an issue, these listening devices will stay on all the time, collecting limitless amounts of data. Essentially being another tracking tool, not only to the individual that has allowed the service, but also to the individuals he/she is conversing with. I know this sounds far fetched, but what if all this data is available through warrants?
Until these tech manufacturers can successfully encrypt the data and keep it away from anyone but myself, I don’t want any part of it. My friends, family and I have some deep political discussions, many times questioning the structure of our current systems; playing devils advocate trying to understand the issues of both sides. These discussions could easily be taken out of context if the right person got a hold of them. Just like any data, including verbal, text, and location … it can be shaped and manipulated to show what the user wants. Its a slippery slope, and I simply don’t quite trust giving out so much data without the safeguards to keep it totally anonymous and/or unaccessible.
The patent didn’t specifically say, but it sounds like Google will only able to identify them if they’ve opted in for the same voice actions/search on their device (just like how Google Now stores voice data).
That, or if they’re logged into their Google account on their Android device, Google can figure out who is in the same vicinity as another user.
First off, I’m generally very relaxed about these things, however, this was the first thing I thought of. If my phone collects information about me with my consent that’s fine, but having to worry about other people’s phones possibly collecting information about me all the time would make me move somewhere such “services” are not legal. I hope this never comes to fruition in that sense. And to make it clear, it’s not that I’m involved in anything illegal or anything along those lines. I just don’t want anyone to be able to track me and keep records of what I say and do. If I use electronics I am aware that 99% of what I write I can never take back but I don’t want to have that feeling all the time. I want to be able to be intimate with my girlfriend and not worry about her phone catching the audio or video of it because she’s missed some option, or not feel like I can talk trash with my buddies about whatever or whatever. Again, I hope this never comes to fruition in that way or anything similar.
Creepy…
…i could see your lips move.
So, we’ll be able to have our own Jarvis?
Well Google found a way to turn every consumer into a spy. With Google Glass, Android wristwatch and Android powered cellphone you can be the ultimate spy.
Skynet
I Robot comes to mind also… blest!
It began to sound great and then it took a dark twisted road. I agree with your initial points, 1 & 2.
I think that this is along the same route of the Google neck tattoo patent, it’s crossing the line.
I think the concept is pretty awesome and would be very useful but the amount of data they collect and store is crazy. Especially when you get into it recording other people around you.
I love the fact that Google can take the information I give to it, and in turn it makes my life easier, whereas when FB does it, I get nothing in return short of more spam links. See? There’s a difference. It’s worth it to me to have a Now Card pop up on my phone every Tuesday because that’s when I do my food shopping. It’ll tell me the best way to get there, and what to remember to pick up when I get there. It’ll tell me about the traffic conditions, and offer up things in the immediate area. When I get to pay for my food, tap it with my phone and move on. What exactly do I get with any of the information that FB collects? A list of people that I “should become friends with?”. If I had to come up with 1 single reason I love my Android device and the Google Ecosystem above anything else, it has to be the single fact that when used properly, Google and it’s products work for me. WIth every update and new feature, it just enhances that. At some point, Google will have the answer to the question before it’s asked. (Heck it’s pretty close now)
Agreed ! I love Google and all of their services.
id say both, and in the midst of privacy concerns and the NSA, uhm kinda like just really handing over even more data. wow. “Oh hey you gave us permission, so we did nothing wrong.” ok i guess just creepy. I doubt if any one could resist abusing it.
This makes me want to buy a piece of crap iPhone. Why does everything I do/say have to be observed/listened in on, and sold? This is getting ridiculous.
Since Apple copies nearly everything Android does, it will prob be on the iPhone 7. Honestly if you think Apple doesnt sell user data just because they say they dont, you need to start thinking for yourself.
consolidated.db
It doesn’t have to be. As they say, IF they ever deploy the feature, it’s optional.
The heck with the security concerns, I want to know whose lips those are?!
ah hah !
http://commons.wikimedia.org/wiki/File:Cherry_Lips.jpg
This is the only thing that made me click the article….lol.
Sounds awesome to me. Go Google.
I’m all for innovation but Google is crossing the line into creepy-territory. Google’s Innovations are all starting to look like; Collect Data on Everybody, all the time, & everywhere.
May = are aloud to, might = could possibly do so. NOT the same!
Exactly what I came down here to say!
google better not be watching me fap, or else ill fap furiously into the camera lol
actually you should let them watch…it’ll teach them a lesson
I really don’t need this much tracking to have a good Google experience. I’m honestly not that interested in talking to my phone or have it read my facial features or whatever. I turn off location tracking where I don’t need it, and Google Now is currently off or I minimize it’s features because they’re just not necessary in my life. I don’t need all features on, they can be cumbersome and useless often, and hopefully if Google implements this, I can turn off options piecemeal and it won’t be an all-or-nothing thing.
This is creepy. I see nothing awesome about it. But then again, they made Google Glass, didn’t they?
Theodore Kacynski aka the Unabomber:
“A technological advance that appears not to threaten freedom often turns out to threaten it very seriously later on.
“Using motorized transport as example, a walking man formerly could go where he pleased, go at his own pace without observing any traffic regulations, and was independent of technological support-systems. When motor vehicles were introduced they appeared to increase man’s freedom. They took no freedom away from the walking man, no one had to have an automobile if he didn’t want one, and anyone who did choose to buy an automobile could travel much faster than the walking man. But the introduction of motorized transport soon changed society in such a way as to restrict greatly man’s freedom of locomotion. When automobiles became numerous, it became necessary to regulate their use extensively. In a car, especially in densely populated areas, one cannot just go where one likes at one’s own pace one’s movement is governed by the flow of traffic and by various traffic laws. One is tied down by various obligations: license requirements, driver test, renewing registration, insurance, maintenance required for safety, monthly payments on purchase price. Moreover, the use of motorized transport is no longer optional. Since the introduction of motorized transport the arrangement of our cities has changed in such a way that the majority of people no longer live within walking distance of their place of employment, shopping areas and recreational opportunities, so that they HAVE TO depend on the automobile for transportation. Or else they must use public transportation, in which case they have even less control over their own movement than when driving a car. Even the walker’s freedom is now greatly restricted. In the city he continually has to stop and wait for traffic lights that are designed mainly to serve auto traffic. In the country, motor traffic makes it dangerous and unpleasant to walk along the highway. (Note the important point we have illustrated with the case of motorized transport: When a new item of technology is introduced as an option that an individual can accept or not as he chooses, it does not necessarily REMAIN optional. In many cases the new technology changes society in such a way that people eventually find themselves FORCED to use it.) ”
Of course, Kacynski was nut job — Hmmm, or was he? Was this a brief moment of perfect clarity in an otherwise disturbed mind? Much of what he wrote above, turned out to actually be true.