MADRID -- Instead of dancing to music, imagine music following along to dancing. Spanish start-up EasyCode uses Microsoft's controller-free gaming technology to create and blend art and music, synchronizing sound and light symphonies.
EasyCode's Rafael Gerardo Weisz brought SmartPlanet down to his basement UtopicUs co-working office and playroom to give us a hands-on -- yet hands-off -- feel for their high-tech, creative plans. The young EasyCode team pays the bills right now by programming for local customers, but it's their weekend Kinect projects that could change the way we interact with art and music.
"Kinect and music does nothing. What we can do with the music is the music recognition and the movement," says Gerardo, speaking about the universal music language of "midi." He explains how reusable codes can be used to send midi signals, which act as chords. With each movement and body placement equaling a chord, an elbow in the air could be an E chord with a drum tone. The Kinect camera "can tell where sounds go," says Gerardo, like yelling to a crowd, "This side make noise!," and a graphic flashes on a big screen. The Jersey Shore fist pumping could become the bass of the techno club music, instead of the fist-pumping grooving to the beat of the music.
EasyCode is collaborating with extremely-alternative musician Diego Ain and his Machine. Ain began making outside-the-guitar music before Microsoft's Kinect was released. "He works with new sounds and making music with his body and voice," Gerardo says. Ain calls it "biotronic music made only with voice, body, and objects." Watching the video above more easily explains why it makes a perfect partnership, as Ain uses his live and prerecorded bodily sounds, mouth, and video to create something close to a rap-techno blend. He calls Ain's interaction with Kinect and music "fun and strange." The XBox technology can give Ain the ability to not only make his music completely touch-free, but also to make his performances adaptable to and integrated with audience participation. With Kinect, movement triggers "sounds along with the other noises and sounds he makes," Gerardo says, like making a DJ's mixing a full-body, live experience.
Ain is "always working with new ways of making music and integrating the people with the show, using stranger instruments," Gerardo says. They are looking to advance Kinect, so that "it's not only about the movement, but also with sound."
EasyCode is trying to change the way folks look at their Kinects -- there is potential past just friendly competition. Two versions of Kinect are available: one with only the sensor at an affordable 100 euros, and the other version is the gaming platform. "People usually use this technically with games, but we want to change that mindset," says Gerardo. He explains how the Kinect camera technology is simple, but the software interface can be very advanced. Kinect contains three cameras -- one normal and two infrared-- that can interface with sensors. "It's all about the software because the cameras are really simple," Gerardo says. With the two infrared cameras, Kinect can recognize the distance of the shapes, creating a tunnel-style video lens, that can scan up to seven meters. It does have its limitations, like not working well outside, but could work at concerts, specifically for DJs and emcees, who could control sounds and images.
He was very eager to show off the advantages of Microsoft's Kinect Software Toolkit, recommending that "anyone interested in designing their own Kinect software . . . should first use the STK because it is excellent at explaining the sensors."
Kinect can be programmed for half or the whole body. The technology can also detect different people and catch onto an avatar. It uses a Spiderman-like mask to track faces -- think of the security uses! -- and could be used to portray an avatar that follows someone dancing and projects a mimicking or contorting image on a concert hall screen.
Gerardo sees their development of this technology being used in interactive ad campaigns and facial recognition for entering buildings.
EasyCode is also working on connecting music with an aerial light display with quadrocopters. Treating the body like the touch-screen smartphone, Gerardo demonstrates how a person basically forms the shape of the quadrocopter and then can steer it in the air by leaning. EasyCode hopes to turn DJs into new kinds of VJs, where they can use both the Kinect music interaction and the quadrocopter controls to take videos of live audiences to blend with the music. Imagine a series of quadrocopters with cameras inexpensively videotaping an entire concert or a fleet of copters with different lights dancing through the night sky.
They are also in negotiations with renewable energy powerhouse Acciona to create a quadrocopter app to adjust solar panels and inspect for problems.