Technology

The issue is that the values of the parameters don't correspond to traditional variables. Concepts in AI are not represented with discrete variables and quantities. A concept may be represented in a distributed way across thousands or millions of neurons. You can look at each individual neuron and say, oh, this neuron's weight is 0.7142, and this neuron's weight is 0.2193, etc., across all the billions of neurons in your model, but you're not going to be able to connect a concept from the output back to the behavior of those individual parameters because they only work in aggregate.

You can only know that an AI system knows a concept based on its behavior and output, not from individual neurons. And AI systems are quite like humans in that regard. If your professor wants to know if you understand calculus, or if the DMV wants to know if you can safely drive a car, they give you a test: can you perform the desired output behavior (a correct answer, a safe drive) when prompted? Understanding how an idea is represented across billions of parameters in an AI system is no more feasible than your professor trying to confirm you understand calculus by scanning your brain to find the exact neuronal connections that represent that knowledge.

Well the thing is that good AI models aren't manually tuned. There's not some poor intern turning a little knob and seeing if it's slightly more accurate, it happens on its own. The more little knobs there are the better the model is. This means essentially you have no idea how any knob ultimately effects every other knob cause there's thousands of them and any little change can completely change something else.

Look at "simple" AI for playing like Super Mario World https://youtu.be/qv6UVOQ0F44 shits already pretty complicated and this thing is stupid. It's only capable of playing the first level

"Rumors claim that GPT-4 has 1.76 trillion parameters"

https://en.m.wikipedia.org/wiki/GPT-4

I'm not sure even unlimited time would help understand what's really going on.

You could build another model to try to decipher te first, but how much could you trust it?

imagine you have a simple equation:

ax + by + cz

The machine learning part finds values for the coefficients a, b and c.

Even if you stepped through the code, you will see the equation be evaluated just fine, but you still won't know why the coefficients are the way they are. Oh and there are literally billions of coefficients.

Here is a simple video that breaks down how neurons work in machine learning. It can give you an idea about how this works and why it would be so difficult for a human to reverse engineer. https://youtu.be/aircAruvnKk?si=RpX2ZVYeW6HV7dHv

They provide a simple example with a few thousand neurons, and even then, we can't easily tell what the network is doing, because the neurons do not produce any traditional computer code with logic that can be followed. They are just a collection of weights and biases (a bunch of numbers) which transform the input in a some way that the computer decided that it can arrive at the solution. GPT4 contains well over a trillion neurons, for comparison.

No. The training output is essentially a set of huge matrices, and using the model involves taking your input and those matrices and chaining a lot of matrix multiplications (how many and how big they are depends on the complexity of the model) to get your result. It is just simply not possible to understand that because none of the data has any sort of fixed responsibility or correspondence with specific features.

This is probably not exactly how it works, I'm not a ML guy, just someone who watched some of those "training a model to play a computer game" videos years ago, but it should at the very least be a close enough analogy.