I keep seeing this claim, but never with any independent verification or technical explanation.
What exactly is listening to you? How? When?
Android and iOS both make it visible to the user when an app accesses the microphone, and they require that the user grant microphone permission to the app. It's not supposed to be possible for apps to surreptitiously record you. This would require exploiting an unpatched security vulnerability and would surely violate the App Store and Play Store policies.
If you can prove this is happening, then please do so. Both Apple and Google have a vested interest in stopping this; they do not want their competitors to have this data, and they would be happy to smack down a clear violation of policy.