It is illegal to use copyrighted material

It's illegal to reproduce copyrighted material*. That includes changing the format as well as things which fall under "derivative" works, but not creating a new work in the style of someone else's (unless it falls under the derivative definition). Many voice impersonators exist and the way you impersonate a voice is to listen to (usually) recordings of that person and practice producing the same sounds that they use for common phonemes (as well as vocal tract shape and larynx positioning to alter the vocal pitch production and overtones which represent vowel shapes). ML does, effectively the same thing without requiring a human to do the listening and practicing.

That said, I think this type of use should be strictly prohibited. In fact, I think it should have severe criminal penalties for any specific voice, not just celebrities. Having the ability to simulate accurate, regional-sounded voices is extremely valuable in the general sense, but imitating or mimicking a specific person's voice without their explicit consent and/or direction has very few, if any, legitimate uses.

* I didn't think that voice mimicking would count as valid for any law, but Google tells me of the "Right of Publicity" and there is (again according to Google) case law involving Ford and Bette Midler. So while it's not a copyright violation to reproduce a voice, it may still run afoul of some laws.