Robotic Telepresence via Skype and Face Tracking with OpenCV

A few months ago I hooked up the robotic head I built to Skype. I called up my brother in Los Angeles and had him control it with his head. Based upon where he was looking and turning his head, the robotic head would then match his head motions. By permitting the user to control the robotic head with their head, the goal has been to create a natural user interface. Furthermore, the user sees through the robots perspective, making the interaction feel natural. Connecting these two technologies, Skype and robotic telepresence, people may converse while also fully engaging in the environment.

Tutorial: How to build a Robotic Head that Follows People's Faces

Edit: See the working prototype here.

I've gotten a few requests to put up a tutorial for building a robotic head.

I'm assuming if you want to build a robotic head, you'll have some experience using an arduino, servos, and some other language that can interface between a webcam and the arduino. I used processing as my interface between the arduino and my webcam. I'm also using the OpenCV library for processing.

I'll describe the overarching structure behind the code to give the illusion that a robot is following a person's face. At a high level, there are only a few steps to have a webcam follow a person's face:

1. Detect face. Take a look at some OpenCV examples on processing's website.
2. Grab X-Y pixel coordinates of the face.
3. Calculate the pixel distance between the center of the face and center of the webcam's view. In other words, you're going to take the images from the webcam, and calculate the distance between the center of that image and the center of the face.
4. Write an algorithm that minimizes the distance between the webcam's center and the face's center.
1. This algorithm will also control the servos.

You can download my processing code here and my arduino code is based on the code found here from letsmakerobots.

Let me know if you guys have any questions. I'll try to answer them as soon as I get a chance.

The following pictures are the first iteration:

Arduino Show & Tell at the Hacktory

Johnny Four and I made it out to the arduino show and tell at The Hacktory this past Saturday. The images and videos tell the rest of the story from the event.

And maybe Johnny Four may finally have a body?

PLObject: Playful Living Object - User Testing

A PLObject is an object that is aware of the child’s frustrations, engagements, excitements, and movements. Two semesters ago I prototyped and developed applications for the Emotiv headset, and one such application was a toy and a game for children. The goal was to provide an external representation of the child's frustration in the form of towers that rebuild themselves. When the child became too frustrated, the towers rebuilt. A demonstration explaining the inner workings is here. Check out some of the videos of users testing the plobject.

Robotic Head Part 2

This is an update on a previous post, the robotic head part 1. I added eyes today and stabilized some of the jittery motions in the neck. This project was particularly inspired by Golan Levin's opto-isolator and double-taker. Watch the video for a demo.

Robotic Head Part 1

I've finally gotten around to combining opencv, processing, an arduino, servos, and a webcam. I've placed two servos perpendicular to each other, and then mounted a web cam on top of that. As of now I've got the entire system zeroing in on any face it sees and then maintains that face at the center of its field of view. I'll be adding a face very soon, with eyes that will follow a person and eye brows to make it more expressive. Watch the video for a demo.

Tutorial for how to build a robotic head using opencv, arduino, and processing.

Tutorial: Using the Emotiv headset with Processing and the Arduino

You guys might remember the Mind Controlled Crane I developed. I've also put together a brief tutorial with source code.

There are three parts to this project: the Emotiv software, the arduino software, and the processing software. The arduino code I used is found here on letsmakerobots. I used processing to then act as an interface between the arduino and the emotiv headset. My processing code can be found here. I believe there are still some bugs in my processing code, but you're more than welcome to use it. The EmoKey mapping that I used for the Emotiv Epoc can be downloaded here. Since I used the free Emotiv SDK, my EmoKey mapping is really busy with key commands being sent to processing, and my processing code is messy and lacks comments.

I will update this tutorial once I have more time to sit and write a detailed explanation. For the time being, feel free to download my code. I will also try to answer any questions as soon as I can.

A New Arduino Project

I just received my arduino mini 04 in the mail. There was some difficulty uploading code since this Arduino Mini 04 has the ATMega328. Apparently, the list of boards in the Arduino IDE doesn't have the specification for this new Mini. Fortunately, I found the solution in the forums. One needs to modify the boards.txt file. I uploaded the servo sweep example, and I'm hoping to start a new wearable computing project. I'll post pictures and videos as I have time to work on it.

Mind-Crane Game Ready for User Testing

I've got the third servo finally working and it's hooked up to my thought of "pull." So if I think "pull," it should bring the wrecking ball closer to me. I've added four more towers making the game more challenging. The video shows the rest of the story.

Arduino + Servo + Emotiv headset = A Bad Hair Day

I finally got a chance to sit down and play around with a servo, arduino, and an emotiv headset (www.emotiv.com). This is a proof of concept video, and the headset is currently set to read facial gestures. I haven't put in the time to train my thoughts with it yet. Hopefully more complex mechanisms will be built shortly. Enjoy.