For this project, I will be working with Adnan in our class and Gabe from a different class. Gabe will be collaborating with us for his p-comp project, while Adnan and I will be making this for this ICM project.
Our idea utilizes PoseNet to make a human Ableton machine. We will add visualizations with code in p5.
The beats will sound something like this.
The beats will visualize as single lines across the screen.
The user will have at their feet some buttons. Four buttons will map to different types of instruments, with another four mapping to the beat track in the background. Another set of three buttons will allow a user to toggle between three effects: transcend, expand and wobble.
The three effects will be manipulated by the users body. The transcend effect will require the user to go up and down in their body. The expand effect will require the user to open their arms or bring them together. And the wobble effect will require the user to shake their hips. The effect will stop changing once Pose Net detects no changed within something like 2 seconds (time span to be tested).
PoseNet
My responsibility was to get PoseNet to identify body movements, which would produce certain effects in Ableton. Here are the four effects we will build and the associated motions:
We decided to use just wrists to create the movement. The user will press a physical button, after which the program will read the user's motions for 10 seconds (the time window might be later adjusted after user testing).
I was able to get PoseNet to identify the wrists, to which we mapped frequency values that we piped to the Arduino so it could send them to Ableton.
We settled on the following design for the visualizer and physical components.
The visualizer will feature a blackened top part which projects a slightly blurred rendering of the video.