Interview with Big Papa of Eyetoy Dr. Rich Marks

[Siraris]

Geek Out: The Playstation Eye is Nearly Upon Us. Dr. Richard Marks Takes Us Behind the Scenes of its Birth.

Ever since we first saw back in 1999 or 2000 what would later become the EyeToy, we were intrigued by its possibilities. That interest was further piqued during a subsequent visit to Sony Computer Entertainment America R&D a couple of years later, where Dr. Richard Marks, aka the Father of the EyeToy, demonstrated the possibilities of a future EyeToy with depth perception. Imagine being able to do everything that the Wii remote's gestural controls can do--without requiring the remote--with a healthy dash of "Minority Report" on top, and you'll have an idea of where Playstation would like to go. Nevertheless, one must first take baby steps; these are represented by the EyeToy's no longer toy-like successor: Playstation Eye, which was announced last week. To get the inside dope on the PS3's newest accessory, we conducted an interview with the always-affable Dr. Marks. Here's what he had to say.

What was the philosophy behind the PlayStation Eye? How did you and Sony decide on its feature set?

This is the LONG version....

The basic idea for Playstation Eye was to create a device for interactive gaming and enhanced communication. It was designed specifically to be used with PS3.

We learned a lot of things from our experience with EyeToy. The initial design meeting for what would become Playstation Eye was called by Phil Harrison. It included myself and key designers and engineers from the EyeToy game teams of SCEE [Sony Computer Entertainment Europe.] We sought to address some of the issues of EyeToy, but also add some new capabilities as well, keeping in mind all along the PS3 as the intended platform.

For designing the camera portion, there were many, many, agonizing trade-offs. Resolution vs. light sensitivity vs. framerate vs. dynamic range. There are also field of view and bandwidth and compression issues. To guide us through this, we created several usage scenarios and scored the importance of each design criteria for that scenario. For the final design, the interactive gameplay and communication scenarios were weighted as the most important for PS3.

As I mentioned already, our previous experience with EyeToy was invaluable. Low-light performance was viewed as a key issue. Also, we felt a big factor of EyeToy's success was the responsiveness imparted by its 60 frames per second framerate, so we made that a minimum requirement. The compression block artifacts of EyeToy are visually unappealing, and they also limit some of our algorithms, so we pushed for uncompressed video. And finally, the many game ideas put forth by the designers suggested the need for two different fields of view.

What about the microphone?

For the microphone portion, we knew that to truly make a useful communication device, we needed a very good voice input solution. Also, speech recognition is a technology more and more games are incorporating, and this requires clean voice input. Following the EyeToy tradition, our primary consideration was ease of use; it should just work. My U.S. R&D colleague, Crusoe Mao, had already been working on research for voice input using microphone arrays, so his work was merged into the design. The hands-free voice input this enables may be the single most important feature of Playstation Eye.

But throughout all this, we universally agreed on one underlying design criterion: cost. Just as with EyeToy, we wanted to create an affordable device that could be accessible to all players and gain widespread popularity. We understood this was the key to allowing people to enjoy the many new experiences we were planning to create.

Can you give us some more details on the device's low light operation capabilities?

In the several years since EyeToy was created, camera technology has improved drastically. In particular, the sensitivity of camera sensors to light has greatly improved. However, this has led to a mainstream trend of making smaller and smaller pixels, but more of them, leading to little overall improvement for each pixel.

For Playstation Eye, we worked closely with Omnivision, our sensor chip partner, on a design that is counter to the mainstream trend of smaller pixels. In low-light conditions, this means that each sensor pixel is able to gather more of the available light and produce a better signal.

This simple story might help explain. A while back, late one night I was testing the first prototype camera hardware, and I discovered that just the light given off by my flat-screen monitor was enough to illuminate me well enough for the camera to image me!

When I visited SCEA R&D back in 2002 or 2003, you showed me a demonstration of array microphones. Can you give me some examples of what this will mean for games and communication?

The most important thing this means is that we can get clean voice data into the PS3.It allows the device to effectively "listen" in a certain direction. In our case, we would listen in the direction of the person playing the game. Previously, the only way to get clean voice data was using a headset worn by the player; now, without wearing a headset, players will be able to talk freely. The device is especially good at removing ambient sounds like traffic and wind noises from outside, the whirring of fans, etc. Also, there is special processing to remove the game sounds generated by the PS3 itself.

The clean voice data provided to the PS3 could be used for chat or for speech recognition, or any other use that a game might have. And it easy for the player, since it doesn't involve wearing anything or configuring anything.

The device supports both 640 x 480 resolution at 60 frames per second and 320 x 240 resolution at 120 frames per second. What are the scenarios where people might want to use one versus the other?

We expect 640 x 480, 60 frames per second to be the most common usage. This provides the best video quality, and is quite responsive. Most webcams only go up to 30 frames per second, so this is already twice as fast as the norm (achieving 640 x 480, 60 frames per second, uncompressed, required a tricky bit of engineering).

320 x 240 at 120 frames per second is a more specialized mode intended for high-speed tracking applications. Most TV display modes are limited to 60 frames per second, so the doubled framerate of the video will not be directly visible. But it means the PS3 can get twice as many video frames to process, which translates into being able to track things twice as fast, or to observing an object at twice as many points along the path it travels.

One of the most surprising and welcome aspects of PS3 is how open it is when it comes to formats, ranging from its support of SD Card and CompactFlash to MP3 and AAC. Why then does EyeCreate save files in a proprietary format rather than one that can be easily shared outside of Playstation users? Do you expect to support other formats in the future via upgrades, as with WMA support on PSP and PS3?

Future formats are still being discussed.

When should we expect a price announcement for Playstation Eye?

I don't know exactly when the price announcement for Playstation Eye will be, but please note that the model used for the rough cost design was EyeToy.

N'Gai's Blog

All of this is good. I've said for a long time that the Eyetoy could be the key to the Playstations 3 success if approached properly. If it incorporates the functionality that we saw in the youtube videos, it could easily take away the entirety of the momentum behind the Wii, and shift it to the PS3.

Very psyched for the microphone goodness, although I definitely have to say that i find it hard to hear games and hear peoples voices that I'm talking to at the same time.

[OmniCloud]

Looking good...

I hope Sony bundles a game with it to show off what it can do....

I'm very interested to see if this can really replace headsets. Sounds too good to be true honestly But Sony is showing some innovative stuff lately so I won't put it passed them.

Must-buy for me even if it doesn't come with a game. But like Ninty did with DS, Sony is going to have to heavily support this peripheral with awesome software and plenty of it.

It would be nice if you could make your XMB interactive just by having the PS Eye...

I smell a Wario Ware rip-off game in the work

[LaLiLuLeLo]

if they can get that bad boy in under 50 bucks I'll be blown away.

[Afrikan]

so we have

320 x 240 120fps
640 x 480 60fps

1080 x 720 30fps maybe? probably not but would be nice....

also, does anyone know if there are any HDwebcams that are compatible with the PS3

[Viper]

I thought it was supposed to be HD? Granted the lower resolutions have nice frame rates.

[LaLiLuLeLo]

^the picture would be prettier but it'd be pretty unresponsive.

They never said anything about hd. Haha I think it was jax or someone who thought the same thing about a week ago. But nah, that was never mentioned. Plus there ain't no way on god's green earth there's gonna be an hd webcam/peripheral camera for a reasonable price or that would be any good on internet chat. AT least not for a while.

[mario25]

Oh my god!!!...the copied xbox vision!!!!!!!1!11!one







\ that's what a friend of mine actually said to me.....

[LaLiLuLeLo]

what a retard. you shouldn't be friends with him anymore. He's gonna drag you down.

[LiquidEagle]

When PSEye was officially announced I believe it was already mentioned how friggin' expensive an HD camera/lens is. My friend just bought a HD camera that goes up to 1080i and he dropped $3500 on that bad boy.

Anyways, this is a great interview, and it's cool to see Richard Marks talking again -- he's a nice guy and good for PR since he's also very technically inclined IMO

I'm very excited to see what kind of stuff comes out using this thing, and will buying it on day one, especially if they're using the EyeToy as the cost model

[Red_Eyes]

The most important thing this means is that we can get clean voice data into the PS3.It allows the device to effectively "listen" in a certain direction. In our case, we would listen in the direction of the person playing the game. Previously, the only way to get clean voice data was using a headset worn by the player; now, without wearing a headset, players will be able to talk freely. The device is especially good at removing ambient sounds like traffic and wind noises from outside, the whirring of fans, etc. Also, there is special processing to remove the game sounds generated by the PS3 itself.

This might prove to be the most useful feature.

[Z]

Originally Posted by OmniCloud

I hope Sony bundles a game with it to show off what it can do....

how about Eye of Judgment and video/picture editing software?

I smell a Wario Ware rip-off game in the work

you mean like EyeToy: Play?

Originally Posted by LaLiLuLeLo2003

They never said anything about hd.

Sony did address it as "the HD camera" at E3 05.


as for the price, Eyetoy with a game was for $50. even if this one goes 10 or 20 dollars more, I think it would be a nice price with a game and editing software included.

Originally Posted by Red_Eyes

This might prove to be the most useful feature.

I am very impressed with the mic. all the mic with camera that I have seen are pathetic. Eyetoy does have a microphone too by the way, but it is very terrible.

this one has 4 channel sound, voice recognition, filtering background sounds, detects sound source direction. really impressive stuff. that is why the mic is made into a bar.

[Viper]

Originally Posted by Z

Sony did address it as "the HD camera" at E3 05.

Developers (Criterion mostly) were calling it HD Eye Toy for a while too. The press listed it as HD constantly as well.




http://ps3.ign.com/articles/740/740684p1.html
Funny IGN article claiming Eye of Judgment will use it's own camera and not PS Eye.

[cpiasminc]

Developers (Criterion mostly) were calling it HD Eye Toy for a while too. The press listed it as HD constantly as well.

Well, about 2 weeks ago, Richard Marks came by our office and gave a few little demos in a lunch meeting, and in the course of discussion, he did say that HD resolution came up as one of the things they considered along the way, as it would basically mean more precision in object-finding and edge-finding and so on, but being able to capture and move that much data at high framerates was problem number one. Low light performance was a deal-breaker.

Another little issue was lag. The display on an LCD or Plasma is usually around a frame or two lagging behind the game. If it's a CRT, it'll probably be two more frames than that(some CRTs have that special "Game mode" which cuts that lag, though sometimes the "Game mode" is just different brightness/contrast/color settings). If the camera itself also lags (which many of them do), then everything just compounds and you end up seeing your own image being noticeably out of synch with yourself. I think he said this one has a worst-case of 2 frames lag, even at 120 fps. Typical is one.

A lot of weight is shifted to the CPU side because this delves into an area where Cell can basically breeze through without trouble. The image that's sent at 640x480x60fps is the raw Bayer color filter array image (no standard uncompressed stream at that rate fits over USB), so you have to demosaic in software. The microphone filtering and directional "picking" is also done in software (which in turn means you have full control over it).

[Viper]

I figured cost may have been the biggest factor but expected their to be some technical reasons for its exclusion.

Will 640 x 480 limit circumstances where your self or objects in the room to just a 640 x 480 'window' on screen? Say it's adding your face to a character or simply just displaying it as an avatar of sorts. Would the image be subject to pixelation if stretched beyond the 640 x 480 'window'?

[vdo]

In the several years since EyeToy was created...the sensitivity of camera sensors to light has greatly improved. However, this has led to a mainstream trend of making smaller and smaller pixels, but more of them, leading to little overall improvement for each pixel.

For Playstation Eye, we worked closely with Omnivision, our sensor chip partner, on a design that is counter to the mainstream trend of smaller pixels. In low-light conditions, this means that each sensor pixel is able to gather more of the available light and produce a better signal.

I think the above means that they find that increased resolution brings tradeoffs that they do not want. If the camera could do a higher resolution, but images were dark and the capture was slower, is the higher resolution worth it.

I see people discussing the microphone array vs. headset and the HD resolution as though the concern is what we see on the screen from what the camera captures.

But I am more excited about not what I or other players see/hear from the Eye, but what PS3 "sees/hears". For audio, that means speech recognition and for video, that means motion tracking. The camera's lack of HD resolution is not as important if what it is doing is not creating a picture on the screen, but is tracking a person or objects for gameplay, and if it can do a better job of that than if it were higher resolution. And that is more important to me. If Sony just made a camera to record video on to the PS3 to share with others, big deal, people have camcorders for that. But if they make something that can, as the article says:

do everything that the Wii remote's gestural controls can do--without requiring the remote,

and make some games that really take advantage of that and can get public awareness raised about those types of games, they will really have something. To really push it, since it is low enough cost, they may be able to start including it with either a special SKU PS3 or even every PS3 in the future, after the other components in PS3 come down in cost, in order to help increase installed base and further drive software development.

[Viper]

do everything that the Wii remote's gestural controls can do--without requiring the remote,

I hate these statements. IT WILL NOT DO EVERYTHING THE WII REMOTE CAN DO. It's been denounced logically countless times and it's getting to the point hat the only reason it's claimed any more is out of either complete confusion of what the Wii remote can do or to start controversy.

[cpiasminc]

Will 640 x 480 limit circumstances where your self or objects in the room to just a 640 x 480 'window' on screen? Say it's adding your face to a character or simply just displaying it as an avatar of sorts. Would the image be subject to pixelation if stretched beyond the 640 x 480 'window'?

With some tech demos, we only saw a working window that was a 640x480 window on screen, but since they were demos strictly outlining techniques, there was no need to scale the window to full-screen.

In practice, there's no reason you can't scale up the result and use it based on the scaled up view. And yeah, if you stretch it, it'll be pixelated or blurry depending on your scaling method. There are ways of scaling up an image without getting pixelation at least (some of which, I've published the research on), but you won't get new detail out of it -- you'll just clear out stairstepping edges, which in a video gives the *impression* of greater detail, but that illusion is quickly lost if the objects in the video hold still.

[Viper]

Originally Posted by cpiasminc

With some tech demos, we only saw a working window that was a 640x480 window on screen, but since they were demos strictly outlining techniques, there was no need to scale the window to full-screen.

In practice, there's no reason you can't scale up the result and use it based on the scaled up view. And yeah, if you stretch it, it'll be pixelated or blurry depending on your scaling method. There are ways of scaling up an image without getting pixelation at least (some of which, I've published the research on), but you won't get new detail out of it -- you'll just clear out stairstepping edges, which in a video gives the *impression* of greater detail, but that illusion is quickly lost if the objects in the video hold still.

So would the idea of mapping your face to a character be viable in a game if your character were to have moments or free form control that would show the face/object larger than 640 x 480? Ex: Your face is mapped to a tennis player and after a point your character is showed up close. Would the pixelation become a problem in that scenario?

[Red_Eyes]

Originally Posted by Viper

So would the idea of mapping your face to a character be viable in a game if your character were to have moments or free form control that would show the face/object larger than 640 x 480? Ex: Your face is mapped to a tennis player and after a point your character is showed up close. Would the pixelation become a problem in that scenario?

That might be a problem. And that can be solve mayn different ways. Maybe taking multiple pictures and automatically stitching them together to get a bigger picture. But for low resolution stuff like the avatars in HOME, this will not be a problem.

[cpiasminc]

So would the idea of mapping your face to a character be viable in a game if your character were to have moments or free form control that would show the face/object larger than 640 x 480? Ex: Your face is mapped to a tennis player and after a point your character is showed up close. Would the pixelation become a problem in that scenario?

Ah, well for something like that you can do a few little tricks to get yourself a high-res texture. For one, the camera does have the ability to take higher res still photos, mainly through interpolation and alternate centering when doing its demosaicing. It's obviously not going to be as detailed as a true HD sensor would give you, but it's not too different from the insanely high-megapixel-count cameras that don't really have a high-res sensor.

Another trick you can apply is using multiple frames of video for a few moments, which will each be slightly jittered (as it's assumed a person cannot hold *perfectly* still) and treating them as the equivalent of unresolved MSAA samples and reconstruct a higher-resolution pic out of it. Smooth out the level sets when you recombine and run a mild sharpen filter.

Suffice to say that if you don't bother, you're going to run into pixelation issues all right. The case you bring up though, is one that allows for some freedom of doing a little extra work offline to get a better result. If you needed to map video streams directly and use the full 60 fps or something, then you're in a different boat and pixelation is something of a given. You probably would have seen it with the original Eye of Judgment demonstration.