The evolution of Kinect and the real importance of Microsoft Research

Table of contents:
That Kinect is important to Microsoft is almost a truism. The Redmond capture device goes far beyond a simple method of controlling their video game console and has become an important element of their strategy and a reference for many of their products. But it is also a tangible sample of what can be born from the combination of a company department with the Microsoft Research ideas laboratory.
The first Kinect was already an example of this. Three years later, it is that same union that has allowed the device to evolve to unsuspected limits to accompany the release of Xbox One.In all its sections Kinect 2.0 represents a significant improvement over its predecessor and during this week Microsoft has taken the opportunity to explain how it has been part of the gadget development process which is on its way to becoming a fundamental piece in the interaction between humans and machines.
Kinect 1.0
When Microsoft presented Project Natal at E3 in June 2009, many saw in it a simple response from Redmond to the undoubted success that Nintendo was reaping with the Wii and its control system. But under that project with the name of a Brazilian city was hiding the Kinect, a device that turned out to be an unquestionable bestseller and over time has ended up being much more than as expected.
Although the technology behind the first Kinect was born from the software developments of the Rare studio and the image capture technology of the Israeli company PrimeSense, it will be the combination of the Xbox team with the investigations of Microsoft Research which makes it possible to reach the market
The rod-shaped device used an infrared projector and a camera that scanned the scene and sent the information to a microchip specially prepared to capture the movement of objects and people in three dimensions. They were joined by a row of microphones capable of recognizing the user's voice. All these elements together allowed 3D motion capture along with facial, gesture and voice recognition.
For such a task the Kinect specifications weren't anything special. The camera had VGA resolution and operated at 640x480 by default, although it was capable of working at 1280x1024 pixels at the cost of a lower refresh rate. The included microchip only carried out part of the work of processing the information, leaving a good part of the task to the console itself.
One of the keys to the entire system resided in the software created by Microsoft to interpret all the information collected by the Kinect sensors.This is where Microsoft Research played and continues to play an important role, investigating the most diverse uses for Kinect and collaborating on the SDK that Microsoft has made available online since 2011 so that any developer integrates it into their products or services.
Kinect 2.0
The big difference between the new Kinect and its predecessor lies in the new main camera. The second generation of the motion capture device incorporates a high-resolution time-of-flight (TOF) camera that allows the upcoming Xbox One Kinect to capture more detail with high precision and higher resolution. The new depth mode provided by this TOF camera allows you to reproduce a scene with three times more fidelity than the first Kinect.
This is not the only advantage of using this type of camera.With it, a 60% larger field of vision is also achieved, which allows recording a larger space and enables more people to be registered at the same time and at a shorter distance from the device. With the new console, up to 6 people can appear on stage, recognizing and distinguishing all their movements. It is a significant advance over its predecessor which was only capable of recording the movement of 2.
The second big change in the new generation of Kinect comes from the hand of the new infrared sensor that manages to recognize objects and people in very low light conditions. The sensor is now so powerful that it can identify items in a completely dark room. The precision is such that it can recognize people and register bodies even without any light visible to the human eye. In low light, it recognizes the pose of the hand up to four meters away, distinguishing each of the fingers with precision.
Kinect 2.0 distinguishes the complete skeleton of the user, the orientation of its limbs, the muscles of the body, and even the beat of its heart.
The combination of the new elements makes it possible to record not only the user's silhouette, but also to distinguish their complete skeleton, the orientation of their limbs, the muscles of the body with the force and weight distribution exerted on them, and even the heartbeat. Facial recognition is also greatly improved, detecting even the smallest detail and gesture and allowing more precise identification. To get an idea of what all this means, just take a look at the following video.
All this new technology also has an improvement in the Kinect processor that allows it to cope with the huge amount of information that all the new sensors obtain. Up to 2 gigabits of data per second are collected by the device to read the environmentAll this information has to be processed and interpreted quickly and for this an obvious improvement in the machine specifications has been necessary.
But changing the components has not been enough. The powerful scanner that Kinect has become requires software capable of interpreting everything it sees, and for this it has been necessary to carry out an important evolution in the code that runs it. This is where the experience and knowledge of Microsoft Research has become more important than ever, helping the Xbox team where problems arose and providing the right solutions in a timely manner. fast and efficient. Kinect 2.0 thus became the product of a collaboration whose history demonstrates the potential that Microsoft hides in its laboratory of ideas.
The process of evolution
The Evolution of Kinect is the story of how a team of engineers tried to bring a TOF camera to Xbox One.These types of cameras emit light signals that bounce off objects and are collected back by measuring the time it takes to travel the distance. In order for them to work properly, distinguishing reflections from objects in a room and from their environment, an accuracy of up to 1/10 billion seconds is necessary. Such a level of accuracy is the only way to provide enough information to allow the shapes and contours of objects to be adequately computed.
Sounds complicated, and the problem is that reaching these levels with a consumer product is as difficult as it sounds. During the development process of the new Kinect, all kinds of problems had to be de alt with that had to be solved in a limited time. Kinect 2.0 should be ready to accompany the release of Xbox One, scheduled for late 2013.
In these circumstances is where Microsoft has an ace up its sleeve: Microsoft Research, your think tankThe team behind the Kinect used the vast knowledge and technical experience of Microsoft Research members to solve the various problems that were emerging with the new technology integrated into the device. This is where the years of investment in research and development began to bear fruit thanks to the collaboration between the different departments of the company.
The challenge was not easy. Differentiating foreground objects from background and minimizing camera blur is a daunting task. For the first, small objects had to be accurately measured in all kinds of scenarios and with all kinds of light conditions. It was necessary to work until it was possible to differentiate the fingers of the hands, preventing them from being confused with the environment. As a result of this work the new Kinect is capable of detecting objects as small as 2.5 centimeters, compared to the 7.5 centimeters of its predecessor. The blur issue required some more work and software optimization, but over time, Microsoft engineers were able to reduce motion blur from 65 milliseconds on the original Kinect to 14 milliseconds on its successor.
All of these tasks require processing a huge amount of information. Data captured by Kinect cameras is on a per-pixel basis, which means that each of the 220,000 pixels supported by the Kinect sensor collects data independentlyTo this we must add much more information collected by the rest of the sensors. The complicated issue is managing to identify and interpret all this information, separating the elements and the depth at which they are found and eliminating the noise from the image.
With Kinect, Xbox One needs to process 6.5 million pixels per second
"Xbox One needs to process 6.5 million pixels per second and only a small part of the console&39;s computing power can be dedicated to the task of interpreting the information since the most power has to be reserved for gaming, skeletal tracking, or facial or audio recognition. Very little computation was required per pixel, requiring cleanup>Without the invaluable help of Microsoft Research, the Kinect team would never have achieved their goal on time"
The effective importance of Microsoft Research
The combined work of the Kinect team with the folks at Microsoft Research has not been a purely consultative relationship. Microsoft researchers took on a lot of the work and built an entire infrastructure and software to solve the problems involved in dealing with the evolution of the device. The knowledge of the two teams in their respective areas made it possible to advance faster than separately.
The key was the speed with which they integrated and the ability to provide solutions in a short period of time. But all that work is not limited to getting a product out for sale. The added bonus is that advancements made by Redmond engineers are available to developers, allowing more view modes to work with and much cleaner data.
Kinect reveals all the potential that Microsoft hides as a company and that is revealed when its departments work in an integrated manner.Multiple Microsoft Research researchers have been active in the development of Kinect 2.0, working on a project that will have immediate market impact. For those of us who have been demanding a greater participation of Microsoft Research in Redmond products, this is good news.
Kinect is also the tangible demonstration that Microsoft Research is much more than a laboratory of ideas, it is a fundamental capital for the future of Microsoft .
Via | The Official Microsoft Blog | TechCrunch