WHAT IS THE CONCEPT OF COMPUTATION JOCKEY?
As a full-time 7th grade teacher in Buffalo Public Schools, I am engaged in the task of getting youth interested in Computational Thinking. One thing many of my students have in common is that they can relate to Hip Hop or DJ culture. Computation Jockey starts with the concept of DJ (disc jockey) and VJ (video jockey) but extends the scope of range to include all data in the computational universe, not just audio or video. Since Wolfram language has so much breadth in the type of data that can be integrated within it and interfaced together, this seems like a good choice to try to use to spring such a campaign.
LEADING UP TO THE SUMMER SCHOOL
Prior to the 2017 Wolfram Summer School, I had presented on the idea of Computation Jockey quite a few times. Below are some highlights:
Mathematica Symposium at The New School in NYC (April 2, 2016): The concept of Computation Jockey is first described to students of this college which is famous for design and the arts.
2016 Wolfram Technology Conference (October 18-21, 2016): This was my first unveiling of Computation Jockey to the larger Wolfram community. At the time I was working with J/Link and Java libraries to connect the Korg KP3+ Kaossilator Pro+ to control dynamic visualizations.
http://www.wolfram.com/broadcast/video.php?c=104&p=5&v=1772 and http://blog.wolfram.com/2016/11/04/my-wolfram-tech-conference-2016-highlights/
2017 Ithaca College Educational Technology Day (March 23, 2017): I was a guest speaker for a 1-hour talk and demo of CJ concept. http://www.ithaca.edu/edtechday/docs/etd2017semgrid.pdf and http://www.ithaca.edu/edtechday/docs/etd2017semdesc.pdf
Computation Jockey has been well received at all of these talks and so I decided to improve upon the original concept for the 2017 Wolfram Summer School and build on the buzz.
WHAT WAS MY EXACT CHALLENGE FOR THE WOLFRAM SUMMER SCHOOL?
As my visions are grand for the ultimate potentials of a fully-developed CJ system, I had to choose just one manageable part to complete in the 3-week Wolfram Summer School.
It was decided to focus on the AKAI Professional APC40 MIDI controller as a further proof of concept http://www.akaipro.com/product/apc40. This controller was chosen because it has been used in the field of professional DJ culture for many years and has many physical manipulables such as 110 press buttons, 10 sliders, and 17 turn-knobs. Integrating MIDI has long been a challenge as the MIDI protocol for control signals is not currently recognized natively in Mathematica and Wolfram Language. Summer School instructor Vladimir Grankovsky was instrumental in giving me advice that instead of trying to use J/Link and java libraries, I might try to connect through RS-232 Serial Protocol. For this we had to install the Hairless Midi<->Serial connector in order to make the raw MIDI usb signals from the controller readable to Mathematica http://projectgus.github.io/hairless-midiserial/, then we also had to use ELTIMA software's Virtual Serial Port in order to send the signals as a useable COM port: https://www.eltima.com/products/vspdxp/. Note that ports are doubled so if you send MIDI to COM3 in Hairless then you need to read from COM4 in Mathematica.
Finally, we had to parse the signals being sent from the controller. When listening to the COM port in Mathematica we saw that each button or control knob sent at least three variables to be parsed. The first number had to do with where the button/slider/knob is located (ie the group), the second number is related to which exact button in the group it is and they are grouped by sequential numbers, and the other digits for sliders range from 0-127 as a range while press buttons specify whether they have been pressed or "released".
THE FIRST TWO "PILOT" MODULES: JULIA SET FRACTALS AND ELEMENTARY CELLULAR AUTOMATA
We set up two modules: One for Julia set fractals and one for elementary cellular automata. For Julia sets we can use sliders to effect: real axis of Julia seed, imaginary axis of Julia seed, "zoom", x axis offset movement, y axis offset movement. For elementary cellular automata we use sliders to effect: rule number and number of iterations. Both Julia sets and elementary cellular automata can have the display enlarged using another slider for aid in display on connected monitor. We switch between the two "modules" using button presses.
We also have the VGA output from the computer converting to analog video in order to use video processing including: Kaoss Pad Entrancer video jockey (vj) controller, a Roland V-4 video mixer (which can "feedback" on itself), and a Dave Jones MVIP color processor powered through a Doepfer Eurorack A-100 power system.
Here is a link to a video of trying out the system: https://www.youtube.com/watch?v=AG7k344K1h0
Following the success of this development project, we will continue to develop and improve upon the CJ interface. We will work on connecting more types of control interfaces such as vinyl turntables. More types of visualizations will be explored for interfacing with. More dynamical systems will be explored. Interface for 3D such as UnityLink and VR could be explored. AI assistants could be programmed to "fetch" mathematical systems, functions, etc through speech recognition. This paradigm will be explored in museums for STEM outreach shows and in classrooms as an education tool as well as promoted within academia for dynamical visualization. Ultimately, it would be nice for youth to be able to "compete" in STEM/STEAM related artistic CJ demonstrations at major venues such as stereoscopic IMAX theaters across the world, possibly powered by "high-end" CPU or GPU systems.
CONCLUSIONS IN DETAIL
This project has demonstrated the potential for using DJ control devices as a tool for manipulating Mathematica code in real-time. The potentials of MIDI controllers has been demonstrated, with their benefits being:
In conclusion, there is great potential in using MIDI controllers in order to interact with Mathematica visualizations. This could be used:
The Computation Jockey (CJ) revolution has begun!
If you are interested in the code and trying it yourself, here is a link to the GitHub for this project with files and documentation: https://github.com/Dreynoldsling/CJ1
I like to thank my advisor/mentor Matthew Szudzik for his time and advice, as well as all the other staff and students at the 2017 Wolfram Summer School.
Great presentation today, very interesting and exciting topic!