If we use Predictive Coding architecture we wouldn't need backpropogation anymore which would work well for a non deterministic system that depends on randomness. Since each neuron just activates or doesn't activate we could use the 1 bit LLM architecture and control the activations with calculated chance. This would increase efficiency and memory used with the proper stochastic hardware.
Instead of expecting AI to generate a proper output in 1 attempt we could make it constantly re prompt itself to generate outputs from the input. We could store the memory in Ram and let the AI pull the neccesary information from it to retrain its weights for that specific question until the answer is satisfied. This would also avoid catastrophic forgetting and with the increased efficiency of this proposed architecture could actually be viable.
Now I understand that using the modern hardwares for this is inefficient, so why not make a new hardware that computes non diterminestically? If we could create a way of simulating randomness in transistor level and control it then each componant of that hardware can act as a neuron. The physics of the metal itself would activate the neuron or not activate it. Technically we could use heat as a noise source that would allow this, but nobody is attempting it. The closest thing I saw to this idea for hardware is Extropic's TSU, but nobody is really attempting this idea. Why? Why are we wasting resources knowing that the AI Bubble will pop without new advancments in hardware? Scaling clearly isn't working as expected. It's just stagnating.
[link] [comments]
