On Plan9 Sam and acme... heck most programs didn't have to emulate character-based terminal emulators. They could drop into a graphical mode if they wanted to.
Amiga OS applications were similar.
One I've been using more often is built into emacs, eshell. It can quite handily use emacs functions to create windows, draw graphics, evaluate elisp I type in or interpret shell commands, pipe shell command results into buffers, etc. But it doesn't emulate VT100 consoles... if you use a program that tries to use those escape sequences to control the output device it won't work well (or at all) in eshell.
Which is why I don't think they catch on. Too many useful programs or libraries programs depend on are built around termios and VT100 escape sequences to control a character-based interface. Coming up with a standard for graphical interfaces would just add another standard to the pile. And would probably have to be VT100 compatible for people to adopt it anyway. There'd be a lot of uphill battles to fight.
update: my knowledge of Console is pretty hazy though, take with a grain of salt. Maybe someone with win32 experience remembers better than I.