Is it not possible to just turn off the voice recognition part? I played with it 3 years ago and mostly interacted with it by typing. In fact I don't think I ever actually used the speech activation, though I occasionally asked a question verbally just to test it.
Unfortunately I can't say anything about the current options, since after a couple of weeks I decided that it did nothing that I wanted and so disabled it, turned off all of the data logging that was enabled when you turn it on (but isn't disabled when you turn it off), and haven't touched it since.
P.S. I'm glad to say that I've never seen anyone in real life speaking to it in public, or even in a private setting. I'm with you completely in that I think this would be a horrible thing to inflict on the public sphere.