Like many, I'm interested in improvements to the area of stereoscopic displays. On the hardware side things seem clear... some monitors support stereo separation, whereas virtual reality devices (or eyephones) like the Oculus Rift will make this even easier. But one thing unsettles me on the software part; There doesn't seem to be an architecture to support multiple video channels, the same way as done with audio.
When you're listening to stereo music, the wav / mp3 / ogg file has two different channels (left and right) encoded, each with its own data entirely. Stereo imaging however seems to rely on a single channel and stream, while I haven't heard about display servers (such as X11 in Linux) knowing to work with multiple video channels. By single channel, I mean that most videos and games work with one image stream only... and do stereo either by rendering the combined output in red-cyan separation, side-by-side via splitting the image in two halves and relying on the video player to combine them, or interlaced by associating one horizontal line of pixels to each eye and again relying on the player to do the rest.
In my opinion this is a bad approach. Just like with audio, each eye should be an unique video channel, and the video player or game engine should decide how to to handle such channels. Main advantage is that videos can be distributed in one file, rather than the author having to create a stereo and a non-stereo version and post each separately. This way, if you have hardware that supports stereo separation, you can manually enable rendering stereo from the video file in the player settings. If not and you're using a normal monitor, you can tell the player to do red-cyan separation on the run as you use 3D glasses, or render side-by-side if you're willing to cross your eyes. And if you want normal video instead, just set it to render one channel to get the existing behavior. So my three actual questions are:
Video: What video formats (if any) support encoding multiple video channels as with audio channels? mp4, avi, ogv, wmv, mkv, or others? Then what video players understand such channels, and offer options as to how to handle them? Also, what video tools and converters know to work with this system? For example, can ffmpeg compile two different png sequences into separate channels in a video file?
Gaming: Most game engines also rely on a single video stream for stereo, such as using GLSL shaders to generate the red-cyan combination as post-processing. Is it possible for games to work with actual separate video channels? Do any video backends, such as OpenGL and DirectX, understand separate image streams and know to apply them to the display device?
Hardware and ports: Headphones are designed to separate the left and right speaker... and I don't mean the headphones themselves as much as the port they're using (audio jack). But what about monitors and the interface they use? Does the architecture of VGA / DVI / HDMI / DisplayPort allow video devices to receive multiple video streams, and decide how to map them by their self? For example, if my video card was to stream two video channels through my HDMI port... plugging in a standard monitor would look as normal, whereas plugging in a stereo enabled monitor would detect and render both streams.