I been thinking about something similar for a long time now. I think abstraction of patterns is at the core requirement of intelligence.
But whats critical, and I think is what's missing is a knowledge representation of events in space-time. We need something more fundamental than text or pixels, we need something that captures space and transformations in space itself.