Orchestral conducting has a long history going back to the middle ages. The goal of the
Personal Orchestra series of projects is to adapt this centuries-old interaction metaphor to digital multimedia so that everyone may enjoy the wonderful experience of conducting world-famous orchestras.
Our primary research goals for this line of research are:
- innovative interfaces for musical expression
- gesture recognition (capturing the physical user movements)
- gesture modelling and interpretation
- real-time interactive audio and video time-stretching without pitch-shifting artifacts
- time design and frameworks for time design of multimedia
News
The Betty Brinn Children's Museum in Milwaukee, USA
Our latest project in interactive conducting systems is code-named
Personal Orchestra 3. The goals of this project include: adaptive gesture recognition, improved audio quality, and improved technical framework.
'Maestro!'' at the Betty Brinn Children's Museum in Milwaukee, USA
People:
Personal Orchestra 2: You're the Conductor (2002-2003) 
The Children's Museum in Boston, USA
You're the Conductor was our second generation conducting system designed specifically for kids (but it works great for adults too!). It features "Stars and Stripes Forever" performed by the Boston Pops/Boston Symphony Orchestra.
You're the Conductor offers an improved technical framework over
You're the Conductor, especially in the area of time-stretching audio. Audio is processed in real-time using an improved phase vocoder algorithm, resulting in increased responsiveness, audio quality and tempo range over the
Virtual Conductor. Movies are stored as standard QuickTime movies with uncompressed PCM audio.
The key research contributions were:
- first conducting system specifically designed for children, featuring an audio and video recording for an orchestra.
- design and implementation of a technical framework for real-time interactive audio time stretching without pitch shifting artifacts.
People:
Personal Orchestra 1: The Virtual Conductor (1998-2000) 
The House of Music in Vienna, Austria
The
Virtual Conductor was the world's first system that allowed users to conduct an actual audio and video recording of an orchestra. It has become a highlight of the
HOUSE OF MUSIC VIENNA where it has been a permanent exhibit and museum highlight ever since the museum's opening in June 2000. It features exclusive performances from the world-famous Vienna Philharmonic Orchestra.
Users conduct the Vienna Philharmonic using a Buchla Lightning II infrared baton that sends MIDI data to a host computer performing the gesture recognition. Gestures are interpreted as simple up-down motions and translated to speed, volume and instrument emphasis which are then sent to a second computer responsible for processing the audio and video for display. Movies are stored as a multi-track QuickTime movie including a series of pre-processed, pitch-shifted tracks. By switching between these different audio tracks and varying the playback speed, the audio is time-stretched without any pitch-shifting artifacts.
The key research contribution was:
- first system to let a human conduct an actual audio and video recording.
People:
Related Publications:
2006
 | Jan Borchers,
Aristotelis Hadjakos, and Max Mühlhäuser.
MICON: A Music Stand for Interactive Conducting.
In NIME 2006 International Conference on
New Interfaces for Musical Expression, pages 254-259, Paris,
France, June 2006.
  .
|
| Eric Lee, Ingo
Grüll, Henning Kiel, and Jan Borchers.
conga: A Framework for Adaptive Conducting Gesture Analysis.
In NIME 2006 International Conference on
New Interfaces for Musical Expression, pages 260-265, Paris,
France, June 2006.
.
|
 | Eric Lee,
Henning Kiel, Saskia Dedenbach, Ingo Gruell, Thorsten Karrer, Marius Wolf,
and Jan Borchers.
iSymphony: An Adaptive Interactive Orchestral Conducting System for
Conducting Digital Audio and Video Streams.
In Extended Abstracts of CHI 2006
Conference on Human Factors in Computing Systems, pages 259 - 262,
Montréal, Canada, April 2006. ACM Press.
 .
|
| Eric Lee,
Thorsten Karrer, and Jan Borchers.
Toward a Framework for Interactive Systems to Conduct Digital Audio and
Video Streams.
Computer Music Journal, 30(1):21-36, Spring 2006.
The video for this article appears in the Computer Music Journal Sound and
Video Anthology 29(4), 2005,  .
|
2005
| Eric Lee and
Jan Borchers.
The Role of Time in Engineering Computer Music Systems.
In NIME 2005 International
Conference on New Interfaces for Musical Expression, pages 204-207,
Vancouver, Canada, May 2005. NIME.
.
|
| Eric Lee, Marius
Wolf, and Jan Borchers.
Improving orchestral conducting systems in public spaces: examining the
temporal characteristics and conceptual models of conducting gestures.
In Proceedings of the CHI 2005 conference
on Human factors in computing systems, pages 731-740, Portland,
Oregon, April 2005. ACM, ACM Press.
.
|
2004
| Eric Lee,
Teresa Marrin Nakra, and Jan Borchers.
You're the Conductor: A Realistic Interactive Conducting System for
Children.
In NIME 2004 International Conference
on New Interfaces for Musical Expression, pages 68-73, Hamamatsu,
Japan, June 2004. NIME.
.
|
| Jan Borchers,
Eric Lee, Wolfgang Samminger, and Max Mühlhäuser.
Personal Orchestra: A real-time audio/video system for interactive
conducting.
ACM Multimedia Systems Journal Special Issue on Multimedia Software
Engineering, 9(5):458-465, March 2004.
.
|
2002
2001
Hardware checklist for PO demonstrations (internal access)
Personal Orchestra Documentation
)
