It listens in on any audio and transcribes it. Probably handy for podcasts, but other things are just scary.
Maybe it's OK if Google does it, I don't really know. I dislike it, it concerns me. The device would have a transcription of audio conversations I have through apps like WhatsApp. Or it could do something useful like transcribe podcasts and hand the transcription over to the owners, so that they can publish it along with their podcasts, without Google needing to dedicate their servers to it.
But if companies like Xiaomi get this feature for free on Android 15 or 16, I know what they will use this tech for. I know what Facebook would use this tech for, and I wouldn't be surprised if they finally start to sell a cheap but powerful Android device.
With offline transcription the "your device is recording me" will get so much harder to detect, as no audio will get streamed. It will become so easy to listen for keywords like "lawnmower" and count their occurrences or their proximity to phrases like "need to buy", or "is pregnant" and stuff like that.
I don't want my devices to do this.
I don't think transcribing on device and then uploading would make any sense: for something like podcasts they could just do serverside transcription (they already do for youtube videos at least).
2) you should see the volume dialog with a box with squiggly lines in it at the bottom of the volume slider
3) press that to turn it off
"Live Caption detects speech on your device and automatically generates captions.
When speech is captioned, this feature uses additional battery. All audio and captions are processed locally and never leave the device. Currently available in English only."
So note that Google does not get a copy of the audio stream. It stays local to your device only. I don't know about you, but seems like a really handy feature to me, especially for those who might have hearing difficulties.
Let's say this starts getting normal, because we've had it for 5-8 years. AI accelerators are in every device, making this functionality a non-issue in terms of battery life. It's then in TVs, in cars, in phones. It's just a toggle to turn it off, but it's software, it can turn itself on without notifying the user. Google won't do this.
How long has it taken Android to start notifying users that an app is accessing or has accessed the microphone? Notifying the user hasn't appeared to be some high priority thing.
The closed caption button in the virtual volume slider, I've played with it a couple of days ago. I was wondering what it was. Toggling it multiple times did nothing, no notification, nothing. I just got a bit annoyed that the state of the image when the diagonal line is missing appeared to leave some marks in the horizontal lines. Now I know that these are supposed to represent sentences. I thought it was just a buggy design and had no idea what this toggle was there for. Long-pressing it did not popup a tooltip indicating its function. I must have explicitly turned on that feature in the settings without remembering. But this is not the problem.
The thing that is concerning is simply that it is there, in common hardware, that there is a proof of concept that these devices are perfectly capable of doing this offline. Hardware manufacturers which tweak the UI can very well just pretend that the button is toggled off and not show the overlay, while the device is transcribing. Possibly only do it after a short FFT analysis has shown that there may be interesting content going on, like speech which doesn't come from a TV, but from a discussion among people. No detectable data stream will be noticeable over the WiFi connection.
It is in fact awesome that people with hearing problems have this solution at their disposal. I'd maybe even like to have it transcribe my phone conversations and store the text in the contacts app.