Synesthesia in Motion
3 Week
TIMELINE
Immersive Design
PROJECT CATEGORY
Group of 2
NEW MEDIA MODULE
Mentor
FABIN RASHEED & SUMANA SOM
MY CONTRIBUTION: Designed and developed interactions on TouchDesigner, prototyped real-time visuals, VR experience on Unity, and cymatics experiments bridging physical and digital soundscapes
The Three Phases of Experience
1. Seeing Sound
Inside a VR cave setup, movement became music. Kinect tracked body motion, transforming it into sound and generating real-time visuals in TouchDesigner, making sound visible.
2. Touching Sound
A speaker setup sent vibrations through sand and water, revealing intricate patterns that showed how invisible frequencies move through the physical world.
3. Stepping Inside Sound
A VR experience built in Unity let users enter sound itself, where audio frequencies dynamically shape the visuals and the immersive environment.
Step 1 - Process
Kinect Data Processing – Kinect captures real-time XYZ coordinates of the left and right hands, sending the data to TouchDesigner.
Sound Modulation – The hand movements map amplitude and frequency, dynamically altering the audio output.
Visual Mapping – These frequencies are directly linked to visuals, generating fluid, responsive projections.
Real-Time Projection – The processed visuals are projected into the space, creating an immersive environment.
VR Cave Setup – Large-scale projections envelop users, merging their movements with sound and visuals.
Part 2
The Physical Installation – Making Sound Tangible
In the second part, we use cymatics to make sound visible. A speaker setup vibrates sand and water at different frequencies, creating unique patterns. As the sound changes, the patterns shift in real time, showing how invisible sound waves affect physical materials.
Process
Speaker Setup – A 50W speaker was placed under a stretched membrane for sand patterns.
Water Plate Experiment – A water-filled plate on a vibrating surface visualized sound waves.
Frequency Modulation – Adjusting frequencies changed the patterns dynamically mapped to the users body movement.
Part 3
The VR Installation – Stepping Inside Sound
The final step was to create an environment where sound became a fully immersive experience. In VR, audio frequencies shaped the space, creating a reactive sonic world.
Users Enter the VR Space – They wear a VR headset and step into a sound-driven digital environment.
Sound Shapes the Environment – Floating waveforms and visuals change dynamically based on the sound’s frequency and amplitude.
A Fully Immersive Experience – The environment continuously shifts, allowing users to see and hear sound in a new, spatial form.
Final Design Showcase
Challenges & Iterations
Overcoming Technical Barriers
Latency in Motion Tracking – Kinect’s real-time tracking initially had delays, causing a lag between user movements and visual/audio response. We optimized the system to ensure seamless interaction.
Tuning Cymatics Setup – Different frequencies produced inconsistent sand and water patterns. We fine-tuned the audio output to create clearer, more defined visual formations.
VR Optimization – The spatial audio wasn’t accurately aligning with the virtual environment. We refined sound positioning and responsiveness for a fully immersive experience.
Future Outcomes & Applications
Expanding the Boundaries of Sound
Interactive Music Performances – Expanding the installation into live performances where musicians or audiences can manipulate sound and visuals through body movement.
Therapeutic Soundscapes – Applying cymatics in therapy to visualize and feel sound vibrations, potentially aiding in relaxation, meditation, and sound healing practices.
Immersive Learning – Using XR to teach sound physics, allowing students to see and experience how frequencies interact with materials in real-time.
See more projects
VR Experience
Immersive Learning
Roving the Red Planet
A VR experience enabling players to explore Mars rovers and discover their key missions and findings.
WebXR
World Building
Godrej Metaverse: Virtual Showcase
Virtual platform unifying Godrej's 14 businesses for next-gen audience engagement.
Offline AR
Product Demonstration
AR for Godrej Aerospace Machines
An offline AR app showcasing Godrej's aerospace machining capabilities for client's visualisation.
