AI Perception vs Human Perception

I was recently listening to Lex Fridman’s AI podcast. Lex asked Elon Musk what the harder problem in the field of Self Driving Car is, The Perception or The Control. So making AI understand what it is experiencing in the world or making calculated judgements based on those perceptions?

Elon's reply

“The hardest thing is having accurate representation of real world objects in vector space”

That got me thinking, can we ever accurately represent our real world with vectors and matrices? In other words can we reliably transmit all there is about our world and condense it into a bunch of structured numbers essentially. Well recent developments in Computer Vision will make you say of course we can. Neural Networks can reliably predict what it is seeing in an image, sometimes even better than humans. So that means the perception problem must be solved right? We have figured out a way to condense the real world information into numbers? Not Quite. Yes, computer scientists and researchers have figured out ways to represent a lot of the real world information into numbers, lets say we are looking at a 3d matrix that represents an apple and if we understand everything about that matrix we will have a pretty good structural idea about what an apple is. But looking at the apple itself adds more knowledge, or does it?

The Knowledge Argument

Time for some philosophy. There is a great thought experiment called the Mary’s Room thought experiment that kind of asks the questions whether or not conscious experience involves non-physical properties. It suggests that even if someone has complete theoretical knowledge about a certain thing, more knowledge can be acquired by experiencing that thing. Mary knows everything there is to know about colors. She knows every single property of color everything from the different pigments to the wave forms that it creates to even what kind of neurons light up when humans look at color. But, and here is the catch, Mary is color blind. She never in her entire life perceived color but knows everything about it. So now the question is will she learn anything new about color if somehow she could see color again? Will that experience hold any new information that was not captured by the theories.

In our world AI is Mary. We have condensed “everything” about perceiving a real world thing into a matrix form for the AI to understand. But without experiencing the thing can it ever really understand? Let’s forget about the philosophical aspect for a second. We perceive the world we live in, on the other hand Neural networks are just mathematical functions living inside of computer circuits. In order to truly understand what it is to perceive something in reality AI needs to understand the higher level concepts of the objects that it perceives, how the different objects interact, what their relationships are. This is what Yoshua Bengio, a renowned AI researcher calls Higher Level Cognition. Current neural networks are great at recognizing patterns in the matrices, but really doesn’t know what those patterns mean. There is a lot to uncover in the world of AI. This is just one of many reasons to be excited about the future.