Motion tracking a marker using a webcam

I have absolutely no coding experience. I am making interactive artwork for a uni project and plan on using a paintbrush's movement to compose music. It would act somewhat similar to a theremin with x coordinates being the notes and y being the dynamics.

I need to motion capture a marker on a paintbrush and get the x and y coordinates. I am also wondering if its possible to track the speed of the movement? I'd like to use this as a parameter for perhaps changing the tonality of the instrument with faster being harsher? I'm unsure at this poin in the project. I have a very limited budget so im just using a normal camera.

Any advice would be greatly appreciated, thank you!

Answer 1

[Image Analyst]

@Alysia, see attached demo where I track a green Sharpie marker in a video. You'll have to adapt it to use getsnapshot to get live video instead of a previously recorded video. You'll need lots of other customization for your particular use case so I recommend you learn MATLAB.

To learn fundamental concepts, invest 2 hours of your time here:

MATLAB Academy - Free 2 hour training

[William Rose]

This is impressive. I have learned a lot from @Image Analyst's posts and File Exchange contributions.

Answer 2

[William Rose]

Alysia,

[Edit: correct typos.]

This sounds like a great project!

Tracking the x-y coordinates of the paintbrush will be the hard part. Once you have that data, the rest will be relatively easy, including computing the velocity. As you know, a theramin has two antennas: one for volume and one for pitch. Your proposal to use x,y in a similar way makes sense. You suggest using velocity to control the "tonality of the instrument" or "harshness". No doubt you know more about music than I. Your use of "tonality" to refer to a single instrument is different than what I'm used to. I think of tonality as referring to the use, or not, of a diatonic scale or mode (major, minor, phrygian, locrian), or to the greater use of tones that are not part of a diatonic scale (chromaticism). But I think I know what you mean by harshness. Some people consider the oboe to be more harsh than the clarinet. You could try to achieve different "harshnesses" by using velocity to adjust the overtones associated with each pitch.

To track the x,y coordinates of the brush using a cell phone video, you'll want a marker on the brush that stands out so it is trackable. Maybe a white dot painted on the brush, or a small white sphere glued onto the brush. But if the canvas is white, it will still be hard to track. I suggest you have the painter wear black, darken their painting hand, and paint on a black canvas with a dark tarp hung behind the easel, so that the only bright thing in the video is the white dot on the brush. This will make it easier to track the brush marker in the video. If you or a friend is facile with electronics, you could attach a bright LED (such as this) to the brush to make it stand out in a more normal painting environment. Power the LED with 4 AA batteries in series (6V), two wires running to the LED, and a 150 ohm resistor to get ~20 mA through the LED {(6V-3.2V forward drop)/150 ohm = 0.019 amps.}

[William Rose]

@Alysia,

I made a file containing simulated x,y brush position data (brushPosition.txt, attached). The simulated brush data is sampled 30 times per second, which is the standard cell phone video sampling rate. I made a sample script that reads the data from the file and plots the sampled brush positions. The x,y position data corresponds to a video file that is about 16 seconds long. The artist starts at the lower left and finishes at the top center.

The script then uses the brush positions to make an audio signal. The script plots ampltude versus time and pitch vs. time. The vertical brush position is converted to frequency (pitch) and the horizontal position is converted to amplitude (loudness).

Finally, the script plays the audio signal.

I use a logarithmic approach to convert brush vertical position to pitch. A piano keyboard works the same way.

I use a linear map of brush horizontal position to amplitude. Computer speakers are often not very linear transducers of amplitude, and neither are our ears. Therefore you will probably want to experiment with logarithmic or other ways of converting position to amplitude.

And of course you will need to acquire a video file and extract the x,y data from it.

[William Rose]

@Alysia, The solution I have suggested above is not for live performance. Doing this live would be a much more challenging project. Even "not live" is challenging enough.

[Alysia]

This is so incredibly helpful! Thank you so much, I can definitely try using an LED marker. And yes I think i'll refrain from making this a live project. Would you mind if I used these graphs you made in my planning essay? I'd cite you of course but no worries if not. Again thank you!

[William Rose]

@Alysia, you're welcome. You are welcome to use what isposted above, and thank you for citing. If you want to discuss it further offline, click the "WR" next to my postings, then click the envelope icon to send a secure message. I have some motion capture experience.