So, I don't trust them to have actually done what I'm going to describe, (and honestly I've just accepted that even with everything off, they're still giving me ads based on stuff I've only talked about and never clicked or written anything), but:
The programs that recognize specific phrases(Ok Google), are always separate from normal voice recognition (and much much lighter in terms of processing). So, if they weren't Google, they might have left the "Ok Google" recognition on, but not process anything else that the mic receives.
They're probably still listening in though.