DESIGN WITH DATA
Introduction to visual programming with TouchDesigner
Authors: Katarina Bankovic - 124411 , Katharina Fuchs – 124111
Third checkpoint concept:
For this course - for our first interaction with TouchDesigner and for getting to know the possibilities of this program - we wanted to create a digital prototype of our third checkpoint of the game „Cabinets of Mischief", which we invented for the „Wunderkammer 4.0“project this winter semester.
We imagined this checkpoint as an immersive and interactive cube with measurements of 4x4x4m, which could show details in different scales about certain artifacts of the baroque universe of Gotha. We chose to display important paintings on the in- and outside of the cube using screens on its surface.
Due to the fact that colors play an important role in our game, we decided to also bring a change of them into our interactive experience.
To create interaction, we wanted it like this: When no visitor is in the room, the paintings will be shown in their original color. When someone enters, their position inside the cube would influence the ‘color filter’ applied to the input paintings.
Furthermore, our vision of the architecture of the cube consists of smaller cubes inside of the main one representing chairs, shelves and benches. These would also be used to provide the interactive experience, where a person would sit and this position would trigger the translation movement of the input visual. Through movements and sitting or standing inside the cube the visitors will influence the output / the visuals.
TouchDesigner workflow:
The first thing we needed to do was to create our ‘architecture’ inside of the 3d space. This was done in Maya. Inside Maya we created the mesh of the cube, but also arranged UV maps for textures of the cube using UV editor. In TouchDesigner we imported a fbx file from Maya that contained the mesh, textures and materials. One important step was to take the textures node out of the fbx node so that when we made changes inside Maya and imported our cube again, it would keep the changes done inside the textures node
For the input visuals that we would show on the screens of the cube, we used eight different paintings from Gotha collection as imported jpg files. Then, a displacement map was created for these visuals to achieve the effect of animated displacement of pixels. This was done by using a noise TOP node with ‘nearest pixel’ selected for input smoothness and viewer smoothness (for having distinct pixels), with speed CHOP node referenced inside its translation parameter. Then, this noise TOP, a constant TOP with grey color value and a threshold TOP (used to single out specific noise pixels by luminance) were connected to matte TOP to create the final displacement map that was applied to our input visuals through displacement TOP. The final result was referenced inside the textures node for the cube.
We connected eight different camera positions to camerablend COMP and created a keyboard interaction that allows the user to move around the scene by pressing numbers from 1 to 9 on the keyboard. This interaction was achieved by creating a keyboardin CHOP and then referencing it to CHOP execute which tells the constant CHOP to take the value of 1 when toggle OfftoOn is triggered inside keyboardin CHOP:
def onOffToOn(channel, sampleIndex, val, prev): mischief = channel.index
op('number').par.value0 = mischief
return
We added three different COMP light nodes to light the cube in a way that shows it the best visually.
We added a render TOP that uses camerablend COMP as the main camera.
Another important thing for us was to add ‘light trails’ around smaller cubes, so that we could create a safe space and we did this by creating a separate mesh inside Maya for the edges of these smaller cubes. Then, in TouchDesigner we used renderpass TOP to single out these meshes, apply the blur to them (with blur TOP) and combine them again through add TOP. We also used Screen Space Ambient Occlusion TOP on our render as a post processing tool to add the illusion of depth around the edges of the cube. The composite TOP was used to merge back the render TOP and renderpass TOP.
As we mentioned before, there are two different interactions happening inside of the cube and these are influenced by sitting and standing positions of people inside the real cube. Within our td application we used keyboard and mouse interaction to represent this.
For the three sitting positions we used keyboard interaction with keys ‘f / g / h’. These three positions influence the translation of the input visual in three different directions. This was achieved by keyboardin CHOP that changes its value from 0 to 1 while one of the keys is pressed (it stops when not pressed), then this value is connected to the math CHOP, to change the range from 0 to 1, to 0 to 0.03 and then connected to the speed CHOP. This change of value range was needed to slow down the speed. There we approached a problem, where the value inside the speed node would freeze after the input from keyboardin CHOP stops. We needed the visual to go back to its starting position, so we used CHOP execute to create a code that tells the speed CHOP to reset itself (reset pulse) when the input stream of previous nodes goes from on to off:
def onOnToOff(channel, sampleIndex, val, prev): op('speed56').par.resetpulse.pulse()
We also added the lag CHOP after the speed CHOP to control this movement of reversing positions to its initial state, so that the movement of visual isn’t too hectic. The final value is referenced inside the translation parameters of the input visuals in the textures COMP.
Another important point of this interaction is the animated human figure in the sitting position that appears on three different sitting positions when the buttons f/g/h are pressed. We imported the fbx file of animated figure downloaded from mixamo website. Then, the CHOP execute was used to create a code that tells the fbx COMP to render itself or not:
def onOnToOff(channel, sampleIndex, val, prev):
op('Sitting').par.render = 0
return
Also, three more CHOP execute nodes were used to tell the tx, ty, tz parameters inside the fbx (position parameters) the values they should use while the buttons f / g / h are pressed:
def onOffToOn(channel, sampleIndex, val, prev):
op('Sitting').par.tx = 0.5
op('Sitting').par.ty = 0
op('Sitting').par.tz = 1
op('Sitting').par.ry = 180
return
The last step was to create this ‘interactive floor’ experience where the position of a human inside the space would be mapped onto a color wheel, from which we would create a palette of three colors to use it on our input visual.
The first thing we did was to create a container COMP which would use our final render as the input. Then we connected this container to the Multi Touch In DAT node so that we could get the information about the 3d space by clicking with the mouse inside of it. After that, we connected this node to Render Pick DAT node which references our render 1 TOP and takes the information (or coordinates) from the objects appearing in this render space. (If we had more objects inside the render space and needed specific object to be referenced, we could have used the render pass node to achieve this, but for our purpose this was enough)
The tx, ty and tz values show in the table inside the render pick DAT.
Then we connected a radial color ramp (ramp TOP) to a new container COMP to create input inside the container. This input we connected to the TOPtoCHOP node to get the RGBA information out of the Ramp TOP. These values can be changed by changing Ustart and Vstart parameters inside TOPtoCHOP node, which basically means picking the point in the radial color ramp that you want to get your information from.
We went back and referenced values from the renderpick DAT to a constant CHOP, connected to a null CHOP, and then connected the null CHOP to the container COMP to create second input, this time CHOP, inside the container. *
*Here we have also encountered a problem of error on the constant CHOP that references renderpick DAT when first starting the application. To solve this, we added another constant CHOP with starting values and used override CHOP to exchange these values and at the end connected it to a null CHOP
Input CHOP was connected to a math CHOP so that we could change the range of values that were taken from the limits of the cube to the range from 0 to 1, which are then referenced to the Ustart and Vstart parameters inside TOPtoCHOP node. This way, by picking the point inside the cube with the mouse, we are picking the point on the radial color ramp and getting the RGBA values from it.
These RGBA values are referenced to a constant TOP to visualize a solid color. This works similar to color picking from the color wheel in Photoshop. We wanted to create a new linear triadic color ramp out of the picked color. We followed the triad color palette theory and used two SHV adjust TOPs to change the hue of the picked color to -120 and to 120:
We used layout TOP with alignment left to right to create a new triadic color ramp. This was connected to an OUT node.
As one of the last steps, we used select TOP to select our ramp inside the textures COMP. By reusing already existing displacement map (select TOP), we connected it to the new displacement TOP to use it on our newly created ramp. Then, we used Composite TOP to connect the input visual of the painting and the new color ramp visual and chose the ‘minimum’ for the operation parameter to create the desired overlay effect. The final result we connected to a Null TOP.
We had a human standing animation in fbx format added to our td application. We added the limit CHOPs to the last null TOP and referenced x y z values to the fbx’ tx ty and tz. This way we created the effect of the figure moving to the position on the cube floor that we picked with the mouse. We put the limits so that the figure doesn’t go through the wall, etc.
We also added two more keyboard interactions for the keys ‘s’ and ‘z’. By pressing the key ‘z’ it changes between eight imported paintings. This was achieved by using a switch TOP and a count CHOP connected to the keyboardin CHOP. By pressing and holding the key ‘s’ it shows the visual of the cube when no person is inside and when there are no color overlays. This is achieved by adding a switch TOP inside the textures COMP, but also adding a constant CHOP and a math CHOP (with range from 1 to 0), which is then referenced inside render parameter of the standing fbx. This allows the figure to dissapear.
Finally, a window COMP was connected to a container COMP with the mouse interaction and set as a perform window. The borders were turned off and the window justified to the center.
The resolution of the render and containers and windows was set as 1280 x 720px.
How we worked as a team:
When first approaching the project, we have separated the tasks of searching for examples and references, where Katharina explored camera interaction and particles system and Katarina experimented with displacement and remapping of visuals.
After concluding what we really wanted to create inside the TouchDesigner, Katarina went on to creating a cube and UV mapping inside of Maya and Katharina made cameras work with camerablend COMP.
After that, the application became too demanding for Katharina’s laptop, so we continued our work together on Katarina’s. Because of this, we completed most of the tasks inside the application while working together at the same time. The work was more separated in sense of dividing microtasks that were happening while trying to make the application work, like finding references, tutorials, or searching through td documentation.
Video link: https://vimeo.com/678856926