I’m loving your brain right now. The “plug it in directly” is just how I’m thinking; I don’t see any reason why some darn “Central Processing Unit” unit has to get in the way of most of the graphics rendering.
Tell the HD what do display and BLAM – show it – at little delay as possible. I’m thinking a dedicated graphics HD – no other purpose but long term storage of temporary display data, of whatever form.
That way, there’s unlimited graphics memory, totally dedicated. Maybe someone’s already doing it – I dunno – but it just seems completely logical to me.
Then the game logic and the human interface can get the slower stuff.