AI really did that thing where you repeat a word so often that it loses meaning and the rest of the world eventually starts to turn to mush.

Jokes aside, I think I know why it does this: Because by giving it a STUPIDLY easy prompt it can rack up huge amounts of reward function, once you accumulate enough it no longer becomes bound by it and it will simply act in whatever the easiest action to continue gaining points is: in this case, it's reading its training data rather than doing the usual "machine learning" obfuscating that it normally does. Maybe this is a result of repeating a word over and over giving an exponentially rising score until it eventually hits +INF, effectively disabling it? Seems a little contrived but it's an avenue worth investigating.

These LLMs are basically just IP laundry. Anyone who claims it's anything more is either buying into the hype or is actively lying to you.

EDIT: Stable Diffusion too. It just takes images from its training data and does photoshop on them piecemeal to create a new prompt.