A conversation with the composer and inventor at MIT Media Lab.

Question: How did being raised by a piano teacher and a computer scientist influence your growing up?

Tod Machover: Yeah, I definitely think that in my case, I come from a very close family and my parents, they’ve really had a strong influence on my growing up and especially with what I ended up doing. It’s actually kind of weird, I think sometimes.  But, they’re very different in a lot of ways and I think for some reason, it’s been very important to me to find out how to bring those cultures together.  They actually still to this day get along extremely well, but they’re so different. 

My mom grew up in upstate New York, a kind of European, intellectual environment.  She studied piano her whole growing up; she’s really good and went to Juilliard, so she was a terrific pianist.  Around the time I was born, she decided not to concertize, but to teach.  She’s a very energetic and creative person, so she really, I think, developed a personal style with teaching, both excellent piano teaching, but always cared a great deal about encouraging all of her students to make their own music and learn theory in really inventive ways. 

When I was a little kid, we used to have a big plastic staff that filled the whole living room that had a bass cleft and a treble cleft and you’d throw a bean bag on these things and either sing the note or jump to the note, it was all fun.  And she also had us do things like, I remember at the end of a piano lesion, I was sort of her guinea pig because I was the oldest of three kids, and she’d say, “Okay, lessons are over.”  There’d be three or four of us around and she’d say, I’ll give you ten minutes to go through the house and everybody bring back some object that makes an interesting sound.  Anything you want to.  So, we’d run around and bring back a book, or a pot, or a lamp, or whatever.  And then we’d sit down and she’d say, “Okay, what sound does that one make? Oh, that’s interesting.  Great.  What’s the loudest sound you can make with that?  Oh, what’s the softest sound? Oh, that’s interesting.  Okay.  What does it sound like if you play those two at the same time?  Oh, that’s interesting.  If you were going to give out a word to describe that sound, what would you say? Oh, peaceful.  Oh great.  So, now we’ve got these sounds, these words.  How can we tell a story with these?  Oh great, okay, well we want to start with that, terrific.” And so we’d make up a little story and who would go first, second and what the story was, and then you’d play this thing. And then she’d say, “Okay, for next week, when you’re home, see if you can make a picture of what we did so when you come back next week, we can try it again.” 

And in that little bit of time, so many important things were there.  First of all, the fact that music wasn’t just something that a bunch of dead people had written on published music books that you had to learn.  It was something that you could make yourself.  It wasn’t just a particular system of music that was in those notes, music existed in the world around you, and it’s people who take that and put it into a system.  That was incredibly important.  You could make music with other people.  And then the business of drawing a picture, what it basically said is notation, which is such a barrier for so many people learning to read and write the notes.  Notation is nothing more than a way of remembering what you did so you can do it again and maybe interpret it differently and vary it.  So, really powerful.  So, I think growing up with that training and her open mind about music was important. 

My dad, he’s not tone deaf, but he’s not a musician, at all.  He didn’t grow up with music, but he’s an incredibly visual person.  He grew up in the Midwest, incredibly different background.  As kind of intellectual as my mom’s background was, kind of high culture, my mom, I don’t think she heard a single piece of pop music before she went to Juilliard.  I mean literally.  She just didn’t listen to that.  My dad grew up in a very popular culture environment.  He probably didn’t hear a piece of symphonic music until he met my mom.  And he’s a cartoonist, so visually oriented, but really always made caricatures of people and was quite good, and studied engineering.  Very early on he had this idea that computers were going to be big, but the only way that they’d be useful to people is if you could interact with them in the most natural way and I think partly the field of graphics was just starting, but especially because he was a visual person, his idea of intuitive was, show me a picture.  Show me something on the screen that I can manipulate. 

And before the late ‘50’s, it was all punch cards.  I mean, you didn’t have anything to look at that shows you – you could make a picture let alone have a picture showing you anything about your data or information.  So, he was involved with one of the first companies in the New York area making graphic displays.  And I have a really different feeling about that. I found his office intriguing and really kind of strange.  It didn’t feel very natural.  It was kind of in a big old industrial building.  You’d go up in a slightly creaky elevator and there was all kinds of electronic test equipment and people building things.  And the thing that was really attractive there was they had some of the first displays, and at that point you had light pens, these pretty fragile metal things.  They were always tethered with a wire to the rest of the device.

But gosh, you’d see these luminescent line drawings and there were some of the early games, and you could change a face, and you had text on the screen.  And it was really exciting.  It’s funny, it’s not like technology now where it’s in all of our consciousness and you just know that Apple is going to come out any day with something really interesting and it’s going to be better than what just happened.  This stuff back then wasn’t obvious that it was going to go anywhere.  The academia and the military were using these things, but I didn’t really have any particular sense that this was the future, and oh my gosh.  But it was very, very interesting to see these things develop from the very beginning and see the components over here and the first tests over here.  And it was a business, so it wasn’t academia.  They were selling these things and they had to work as well as they could work at that point.  

So I think that was quite interesting.  And I think, in general, the idea of this very organic intellectual, artistic music culture and the populace visual technological kind of industrial culture, my mom who is actually, well I don’t know if elitist is actually the word, you know; knew the very best performances and knew the artists that she thought were the most interesting including contemporary ones.  Because of both of their interests, we listened to a lot of John Cage, and Morton Subotnick, and European electronic music.  And I was just interested in how different these cultures were and how I was drawn to both of them. 

It was quite a lively household I think.  We went to a lot more really interesting, edgy art events whether it was just seeing Pablo Casals playing the cello, or there was a movement in the New York area in the 1960’s called EAT, experiments in art and technology.  Bell Lab technicians and artists in the New York scene.  My dad was very involved in that so we’d got to a gallery show with you know, really wild – I remember one was something where there were mice in a cage and a little robot that was building towers with blocks and the mice were knocking it down, and the robot arm would put them back up.  And this was very early and you know, why would somebody do that?  And it was quite interesting.  And I think we were in the middle of interesting events more often than I am able to drag my teenaged girls to interesting events.  It was very interesting growing up, I think. 

Question: How did you end up at Juilliard?

Tod Machover: My mom went to Juilliard and I never had it in my mind to go to Juilliard.  I’d studied piano first and switched over to cello when I was about seven.  I played mostly chamber and solo classical music.  I got really involved with rock music when I was a teenager.  I wired up my cello.  When I was done with high school, I knew that music was really important to me and I knew I didn’t want to be a cellist, but I wasn’t really sure if I wanted to be a composer, or think about – I was just interested in the ideas behind music, I was interested in mathematics.  And the last thing I thought about was going to Juilliard. 

I also had grown up and gone to high school in New York, so I wanted to get out of the east coast.  I went to the University of California, Santa Cruz for a year, which turned out to be a really vibrant, very intensive intellectual atmosphere where you could do a lot of aspect of music without it being a conservatory.  And that’s why I went there.  There was a very good guy there doing computer music.  There was a performing ensemble doing a lot of John Cage and European avant-garde composers.  There was a guy who had been at the University of Chicago who had retired to Santa Cruz who was one of the great music historians, music theorists.  And I used to go to his house and I got him to tutor me on Beethoven piano sonatas.  So, there was nobody giving a course.  It was a wonderful place.  But I had a lot more musical training than most people there.  So after a year, I thought gee I don’t really need college anymore, which wasn’t correct, but that’s what I thought.  And I definitely, within a month or so of getting to Santa Cruz, I realized that composing was what I really wanted to do.  That my mind loved to put things together and had combined all the different things I really loved.  And I started getting even more interested in technology than I was in high school. 

So, I moved to Italy for a year because one of my former cello teachers was playing in the orchestra in Florence.  So that was really nice.  I moved there and played in the orchestra for years and studied with a very famous Italian composer named Luigi Dallapiccola.  And then, well quickly, at the end of that year, I knew that I probably should study something.  I probably wasn’t done with – I didn’t know anything, but I probably should go back and study composition.  I knew I was a composer and Pierre Boulez, who at that point was conductor of the New York Philharmonic, a great Conductor and Composer.  He’s kind of my hero.  He came to Florence with the BBC Symphony and I chased him down the block after the concert, and I think he thought I was going to mug him or something.  And finally turned around and I said, I’m a young composer, I’d love to show you my music, could we talk? 

So, he invited me the next day to rehearsal and we talked.  I showed him what I was doing and he was very, very nice.  And I said, would you teach me?  And he said, “You know, I just don’t teach.  I never have.  I don’t teach composing, I don’t teach conducting.  I just don’t do that.  I’m happy to stay in touch and follow what you are doing, but I can’t teach.”  So, I said, “Where’s the best place to go?”  And he said, the only good thing right now is to go to Juilliard and study with Elliot Carter.  So, that’s what I did.  Except it was the end of the summer.  But I went back and I convinced them to let me take the exam and got in for that September.  So, I did start at Juilliard half way through my Bachelor’s.  I stayed there for my Bachelor’s, my Masters, and started my doctorate.  The first year of my doctorate, and then the same Pierre Boulez had in between, had been invited back to France.  The Pompidou Center had opened up and they started this big music technology institute.  And Boulez invited me to go there, so I went for a year and stayed for seven and never went back. 

Question: How would you evaluate your education at Juilliard?

Tod Machover: Juilliard – is this going to be on the air?  It’s funny, music is, I guess it’s true with a lot of disciplines, but just to get the skill you need to practice the art, you know, to get your way around an instrument, to be able to imagine complex music in your mind so that you can think about it, shape it, write it down.  Be able to, if you’re a composer, hear your music played and be able to immediately have judgment about whether it’s the way you want it or not and then articulate it.  I mean many, many skills that are not really intellectual skills.  You have to train your mind and your ears, but they’re more like athletic skills.  So, part of music you just have to learn those things or you can’t practice the art. 

On the other hand, one of my interests in music has always been what it means, why it affects us the way it does?  One of the big mysteries of music is, if you take music without words, it means something to us because we know it’s about something.  It’s about something important humanly, but since there are no words, nobody knows what it’s about.  Is it about a person?   Is it about that person? Is it about some kind of story that you could put into words?  So, there’s just an incredibly rich and interesting relationship between our listening to music and the way our minds engage. 

I mention that because you can’t do any of that at Juilliard.  So, a place like Juilliard is terrific.  I got to study with Elliot Carter so this amazing, probably the greatest living composer in my view right now.  But he was never a pedagog, I mean, I was one of his very few students and the lessons – I studied with him for like six years and I’d go into a lesson, there’s a piano there, but as I remember it, for six years, we never actually lifted the lid of the piano, so the piano lid was closed.  Put the music on the piano and as I remember it, every lesson we’d stand up.  I don’t remember why we didn’t sit at the piano, and we’d look at the music and start talking about it.  And you would never kind of plunk out your piece, you know here’s how it sounds, you’d never come in with a cassette or a CD.  The understanding was, you imagined it in your head and you could look at it and talk about it and both of you were hearing it as if it were a CD playing, but it was totally silent. 

It’s almost as if he were looking at his music.  He’d look at it and say, “Oh, gee, okay, I see what’s going here.  But you know it makes that sound here, but you could do it this way, or you could do it that way and if you do it like that, it would keep somebody interested for over here and this feels too obvious, you might do --”  So, it was completely unpedagogical because he wasn’t telling me a principle, he wasn’t telling me a theory, he wasn’t saying here’s a way to think about it.  He was just thinking about it out loud as if it were his music pretty much. 

And so I had to learn how to ask him questions, especially about harmony and texture.  So, it was a very unusual pedagogical experience.  It was a wonderful way to train, listening ability, inner listening.  But most of the other courses were not so interesting and it was not a very intellectual environment.  So, I took a lot of courses.  I signed up for the general studies at Columbia while I was there.  I think now, Juilliard and Columbia actually have some reciprocal program, but then it was like just doing another, you know, in my spare time.  So, that’s where I took music theory and actually computer science and I studied German.  All these things I wanted to learn, and I took some mathematics there.  So, it’s kind of like two degrees. 

And Juilliard is – you know, the great thing about it is it’s a really high level of talent.  I mean a lot of people there who are just very good at what they do and have been doing it for a long time.  A lot of people there had decided, way younger than I did, that they wanted to be musicians.  I think part of the bad thing is that skill is emphasized so much that a lot of people, by the time they get to Juilliard, well I think they kind of forget why they got into music in the first place and if they’re performers – this is a simplification, but a lot of them are trying to win a competition and play more accurately, or better, or more beautifully, whatever can be measured, than somebody else.  And you know, even for composers, it’s to win a prize, or make a mark.  It’s not really – what do I really care about and what’s – and the only reason anybody would ever want to be a composer is because there is something so burning in side you that you need to express and think about and figure out.  I mean, it’s such an impractical profession that, in general, composers are sympathetic because they’re dealing with real issues and most of the practical things in life are kind of difficult. 

But at Juilliard, it tended to be almost like a business school.  So, I learned a lot there and I still have some friends there, but it wasn’t my favorite environment.

Question: What was the first song you composed?

Tod Machover: I don’t actually remember the first song I composed.  I think probably the first, you know, we did a lot of these creative activities growing up.  I think, I laugh because my mom is also, on one hand an incredibly creative pedagog and is very good at drawing people out too.  But she is also a pretty strong personality, so I’m thinking the probably the first piece I composed is probably when she told me, “Oh, you should probably write a piano piece.”  “Are you writing your piano piece?”  Or something like that maybe.  So, there’s probably some piece maybe when I was in middle school that I can’t remember now. 

I did take composition lessons when I was in high school, so I wrote piano pieces.  I wrote some chamber music.  I don’t think any of that was particularly interesting.  I don’t remember too much of it. 

I went to Fieldston here in New York, quite an interesting creative school.  And I actually have more close friends who have stayed friends from high school at Fieldston than I do from Juilliard, as a matter of fact.  There are a lot of like-minded spirits.  And one of my friends there was a very good writer; he is now a big screenwriter in Hollywood, and a terrific musician.  So, the two of us had a king of two-man musical group.  And I’d put headphones, there was no such thing as am amplified cello at that point, so I took headphones, turned my cello on it’s side, kind of like a gigantic bass, and put headphones and clamp them around the belly of the cello so they were like a microphone, they amplified the cello, and he played guitar and keyboard and we wrote a lot of songs together. 

So, the first songs that I really like, were actually pop songs that we did together and we had tape recorders at home and I used to go into the studio and record multiple tracks, or change the quality of the sound, things like that. 

And then, the first music that I remember really investing in that I composed was kind of the second I got to college, to the University of California, Santa Cruz.  And they started out being for a music history course, or a music theory course, or a composition course.  So, not really where you’d expect to do anything interesting, but I think the bug of writing music had just been growing in me and that was the right environment for me to say, “Oh my gosh, this is me.”  I’ve got a million ideas and I want to have people play these pieces.  And there are things I want to think about in terms of structure, or how I want people to listen.  So, I wrote quite a bit that first year at Santa Cruz and kind of kept going. 

Question: What is your process for listening to music?

Tod Machover: The way I listen to music goes in waves depending on a lot of things.  How busy I am, if I’m in between composition projects, if I’m starting a new project.  So, the only time I listen to the radio for music is with my daughter’s when I’m driving them to school, or driving them somewhere. I think they’d kill me if I put the radio on to a station that I wanted to listen to.  It’s always one of their stations.  So, I listen to a lot of, I mean, they’re 15 and 12.  I must say, just in this last year, they’re taste has evolved from – “evolved,” – it has changed from really vanilla, top 40, either totally pop or hip hop, you know those kind of two kinds of stations these days.  And but now they listen to a lot more kind of Indy music and so I listen to a lot of music that teenagers are listening to because I’m around them. 

Then I exercise – I do rowing on a rowing machine– we live on a farm right outside of, right near Boston.  And I have a big barn that I converted to my music studio, so I go there early in the morning and the first thing I do is rowing.  And that’s when I listen to a lot of music.  So, I row for about 40-45 minutes every morning and put in my iPod and it’s a huge range.  That’s when I listen to either things that I just love and know very well and just want to pay attention, it’s also where I listen to things that are new that I want to get to know.  And things that are new might be new recordings of traditional repertoire by people who I really like.  There’s a pianist name Pierre LaJollamar, who I’ve known for years who’s a specialist in contemporary music, but has been recording a lot of Bach and every time he comes out with something new I’ll listen to that very carefully.  I listen to new music by composers who are interesting to me.  I listen to some; I don’t know if I want to call it pop, but it’s some interesting artist that gets my attention, I listen to in the mornings. 

But strangely, the thing I listen to 75% of the time, when I’m exercising with my headphones on is English Tudor/Elizabethan music, so music from about 1450 to the early 1600’s.  And this is music that has attracted me for years, probably ever since I was in high school.  I love Bach, I love Beethoven, I love Mozart, I love the Beatles, I love you know, Stockhausen, I love many things.  But for some reason I come back to Elizabethan music because it’s a little bit like the Beatles.  It has – I don’t think it’s an accident that it’s English, England has had a lot of really bad periods of music, but it’s had several amazing periods where they’ve found an incredible balance, not just between music that’s a rather complex and also pretty direct.  Like the Beatles. 

Everybody likes it because the tunes are memorable, I mean, any Beatles song is perfect.  It gets to you right away.  But if you look at the orchestration and the way the voices blend, and the way the instruments are used, and if you listen carefully to subsidiary voices which are not the main baseline to the main harmony, it’s very, very – I don’t know if complex is the word, but it’s very, very rich; much more than most pop music.  So, it’s managed to combine complexity and simplicity in a very special way.  And I think it took influences from all around the world.  England’s a little isolated, so when it clicks – and Tudor and Elizabethan music like that.  It’s extremely calming.  I mean, it always takes me to another place, it’s also very, very stable and simple at the same time, you know, there are these melodic lines that do the craziest things.  Much more interesting than what people were doing in other countries.  And it’s also harmonic.  The English learned, in my view, how to use harmony much earlier than the French or the Italians, or the Germans.  So, you had these crazy lines colliding against each other whether it’s string music or vocal music.  And at the same time, the beautiful chord progressions that are very modern in a lot of ways. 

So, I just keep discovering new music from that period.  New recordings, or somebody will discover at some castle a trove of music that nobody had ever played before and for some reason I listen to that a lot. 

I almost never these days sit down with a CD or my laptop and just listen to a piece with a score.  I probably would do that while I’m exercising.

Question: Are there days you crave silence?

Tod Machover: I love silence.  And one of the paradoxes about the way I live and also about my work is that definitely one of the reasons I went into music, and especially into composing is that I love setting up an environment where I can be by myself for long periods of time and have everything as quiet as possible, either to think about sound, or to think about ideas, or just to focus on things that are important to me.  So, I do.  The barn where I work, it’s only 15 minutes or so from Harvard square, so It’s very close to the center of Boston, but it happens to be a total oasis.  It’s completely quiet in there.  You’d think you were out in the forest somewhere.  And at the same time, I work at MIT which is – you’ve got this lab which is the center of high technology and new ideas and there are people all over the place and it’s a very beautiful building actually, but it’s quite intense and there’s always people who want to talk to you about something. 

So, I kind of go back and forth.  But one of the things I like most of all is being in my study, in my barn, with absolutely no sound anywhere thinking about something.  Yeah, it’s extremely important to me. 

Question: Describe the genesis of hyperinstruments.

Tod Machover: So, in a funny way, hyperinstruments – I keep talking about the Beatles, and I think it’s maybe because Sergeant Pepper's came out when I was about 13 or 14 and that was a pretty impressionable age, and it was such a kind of radical period.  But that period of the Beatles really had a big influence on me and I think are directly related to hyperinstruments. 

So here an album like Sergeant Pepper's comes out, incredibly new sounds, layer upon layer of studio produced sound like nothing anybody had ever done before.  And it’s also the period when the Beatles realized that what they were making in the studio could never be performed.  And they had already given up on performing because there were too many screaming fans and they were playing in larger and larger venues so they couldn’t even hear what they were playing, it just wasn’t any fun any more.  At the same time, their music evolved to something that they could only put together in a multi-track recording studio.  So, I think so many things about that period in the late ‘60’s stuck with me.  The sense of – I mean, I love the cello, I love the physical sense of an instrument that’s about the size of your body that vibrates enough that even if you play an open string, you feel it.  You really feel it from your toes to your hair follicles and it really uses your – you have to put a fair amount of effort into the instrument.  I think a double bass for me would be too much effort.  But the cello, you’re really engaged and the sound is kind of right here.  So, it feels like being merged, married to an instrument.  I like that feeling and I like the idea of imagining a sound and feeling a sound and then having it come out through your body, through an instrument like that.  That’s an important way to make music. 

Then you have something like the Beatles making this great music where they say, “You know?  We just give up. We’re never going to perform this.  It’s not just that we’re not going to go on tour right now; this music has too many layers.  It’s got too many changes in the sound.  This voice is all of a sudden gets prominent and then disappears and this is its final version.”  You also have at exactly the same years – I never thought of that, but 1967, Glenn Gould, the great pianist, wrote an article called The Future of Recording, and that was exactly the period when Glenn Gould said, “I’m never going to play in public anymore.  First of all, I’m kind of shy and I don’t like – it just makes me uncomfortable to play in front of people.  Plus, I want to get my performances exactly right.  I imagine precisely the way these different lines in the Bach sonata need to be balanced and I understand exactly the relationship between this voice to that one.  And yes, I can play it, but I want to go in and adjust these things in a recording studio and that’s my new medium.  Recording is the future.” 

That’s funny.  So, Gould and Beatles did that the same year.  And I think what stuck with me at that point was, wow – I love this music and I totally get it.  And here’s Gould doing it for Bach, and the Beatles doing it with their music.  But they’re not performing any more.  And I love performing.  I think that one of the things about music is it’s supposed to be spontaneous, it’s supposed to be real human beings bouncing off of each other whether its from the stage or to the audience, or jamming with friends.  I mean, I love the idea – it’s important as a composer to sit in silence and imagine these complex musical worlds in your head, but it’s also a wonderful experience to touch your music and to hear it and hear it in the room with you and to say, you can’t have an entire orchestra there, but you’d kind of like to have the orchestra there.  You’d like musicians playing for you and you’d like to say, you know, what?  Duh.  I need ten more French horns. Or that just doesn’t work right; I need some instrument that doesn’t exist. You want to go back and forth between the sound and touching it.  

So, I think from that very early time, age 13, 14, 15, I thought, yes, this rich studio produced music is the future, but it can’t be the future to go run away into the recording studio.  How can we take that kind of complexity and richness and make it possible for people to touch it and play it live.  That’s what hyperinstruments are. 

I think the seed was planted when I was a teenager, and it took me until I got out of Juilliard.  At Juilliard I was just learning to be a composer, but I was also learning how to manipulate computers.  So, nobody at Juilliard, this was in the mid to late ‘70’s, nobody was interested in computers then.  It was kind of too late for Moog synthesizers and there were a few computer around the world in Stanford University and MIT and Princeton or Columbia University were about the only ones where you could use punch cards and you’d go in and type out your computer program to make notes appear and, this is what I learned how to do when I was a Juilliard.  I found somebody at Columbia who taught me how to do this.  I’d go to the Inner City Graduate Center here in Manhattan, type out the punch cards for maybe 30 seconds of music that I wanted to, let’s say play for a string quartet, looked at my music and thought it was completely crazy and unplayable.  And I’d say, no, no, it sound really good.  I’m going to put it in – so, you made the punch cards.  A week later, I’d go back and pick up a reel-to-reel tape and put it on the tape recorder.  And of course you couldn’t edit it, you could see the notation, you’d have this ridiculous – but I learned that at Juilliard. 

And then when I went to Paris in 1978, this Eurecom Institute that just opened up at the Pompidou Center and for the first time, there was really money around and people around to think about really bringing musicians and scientists and technologists together.  It was like – I was very lucky to be there at that time.  It was really great.  And they were just starting to invent computers that worked fast enough to give you music immediately.  So, instead of waiting a week to get your music back, you still have to write software, there was no graphics, but you’d write some software, say okay, now.  And it would play your piece and you might be able to do a carriage return and it would make it louder or softer.  It was very primitive, but it was immediate.  And that’s when I started thinking about the hyperinstruments.  Wow!  You could have the best of both worlds.  You could have layer upon layer and delicate sound and voices and instruments and sound nobody had ever heard.  You could mix them altogether, but I can put them in a laboratory – put them in a studio so that they will playback immediately and if I can touch a key and have something happen, I should be able to make my own instrument.  I should be able to make something where I can squeeze something, or the way I touch something should be able to change the sound.

So, I started thinking about either leveraging off of existing instruments, or making new instruments that would take what people do well in mastering an instrument, but multiplying the effect that you could produce with that instrument.  So, I didn’t actually build a hyperinstrument until I moved to MIT in 1985 because at that point personal computers had just come out so you could do all kinds of things with a $2,000 box.  Anyway, there were a lot of things happen in the mid-‘80’s that all of a sudden made it possible to do a lot of very quick interactive music.  And with the hyperinstrument, the idea was how to take the act of performing, how to measure what a performing was playing, but also how they were playing; what the interpretation is, and let that interpretation be the equivalent of five recording engineers in the Beatles digital recording studio, in stead of a bunch of people moving knobs up and down and pushing buttons to change that effect, and saying, “Oh, let’s have that.”  You simply do it by playing your instrument.  And if you play the downbeat of certain measure louder, or if I play this phrase building up the intensity to this section or of I lay off and it all becomes very calm.  If I chain – I’m showing as if I’m playing the cello, but the cello’s not a bad way of thinking about it because you can change all kinds of qualitative aspects of the sound.  If I’m playing a note and it goes from [making singing sound] If I change that kind of quality of sound, I started to measure those kinds of things so that that sound might mean, okay, if I make that transition, then I want to change the cello, it’s not a cello anymore, it’s a voice, or it’s something I’ve never heard before.  Or if I use this part of the bow and play with certain accents, I’m going to play a melody in and depending on the way I play it’s going to add its own harmony. 

So, the instrument – a single person playing an instrument could make the equivalent of a multi-track recording and by interpreting the music differently, in as natural way as possible, I could shape the way this sounded.  Not like being at a mixing desk in a recording studio, not like writing a computer program, not like working with the machine, but just like playing naturally.  So, that’s what I wanted a hyperinstrument to be, especially for trained musicians. 

Question: How did hyperinstruments lead to the creation of Guitar Hero?

Tod Machover: So, one of the interesting things about inventing these instruments, and actually in some ways, the more unusual the instruments are that you think of, or the sounds you think of, the more radical the technology is that you’re trying to develop for music in particular.  The more likely it is that something unexpected is going to come out of it that you didn’t expect at the beginning.  So, as a perfect case, we built this hyper cello for Yo-Yo Ma and I wanted to keep to as close a regular cello as possible, so had a regular finger board, a regular strings, Yo-Yo’s a really nice guy, but when I started talking about taping or nailing things onto his Stradivarius, he didn’t like that idea too much.  So, we made a physical instrument from scratch so we could put measurement devices and sensors right in the instrument. 

So, he played this cello, you plug it in, it can sound like a regular cello, but it’s also measuring everything that’s happening.  Then we took a regular bow, because measuring the bowing for cello is like breathing for a singer.  So, we have to measure everything about the horse hair on the string and the pressure and how fast it’s moving, and the angle.  And we found out the best way to do that was to put two computer chips on either side of the bow, to send electricity through the chips into the air, to put a little antenna on the cello.  The antenna picks up the electricity and it can tell, if you write software, which side of the bow the electricity is coming from and by doing that, you can tell where the bow is, how fast it’s moving, how much bow you’re using, which part of the bow you’re on, the angle.  Everything you need to have a kind of language of the gesture of bowing. 

So we did that, it worked well.  I wrote some pieces for Yo-Yo, he played them, and so that kind of hyper cello started taking off.  But at the same time, we were rehearsing and we found that when Yo-Yo’s bow got close to the cello, when his hand got close to the cello, the measurement went all whacky.  It wasn’t supposed to do that.  So we went back to the lab and we found out that, sure enough, his body was absorbing electricity from this circuit, which was kind of a drag because it became unpredictable. So, first we figured out how you could modify it so could predict it.  But then I started thinking, oh gee, if this electric circuit is measuring how much electricity his hand is absorbing, it actually knows where – I could tell where his hand was.  And if I could do that, I could actually throw away the cello and throw away the bow and just make an instrument that measures the way somebody moves their hands. 

And so, a light bulb went off and I started thinking, my gosh, all this sophisticated software for measuring how Yo-Yo plays, and how he moves and this technique of the bow, I should be able to use similar techniques for measuring the way anybody moves, and so somebody who is not a professional or a trained musician, I should be able to make a musical environment for them.  So, the first instrument we made after the hyper cello was a chair.  We made something called the sensor chair, and we actually made this for the magicians Penn and Teller because they were following the work and we kind of had this idea. 

So we took a chair put a piece of metal on the chair.  When you sit on the chair you’re rear end touches the metal and you’re body becomes part of this electric circuit just like the bow was.  So, electricity is streaming out of your body, you put two poles in front of the chair and the poles have sensors in them with little lights.  So when I move my arms, electricity is coming out of my arms and these sensors can tell how much electricity is in the air.  So, again, if I write software it knows exactly where my hands are and also, if I’m smart enough I can write software that knows, am I moving in a jagged way?  Am I moving smoothly?  Am I moving continuously?  Am I moving discontinuously?  There are all kinds of quality I can pick up. 

So we made actually a musical instrument and then a little opera magic trick for Penn and Teller which is based on sitting on this chair and playing melodies and harmonies and rhythms just by moving your hands.  And it was great.  And I sort of thought of it as a virtuosic instrument for anybody.  I mean everybody can move their hands, but you kind of have to sit down, learn where the sounds are, learn how to shape them, learn how to communicate this to somebody.  So, it was a real breakthrough.  And we started making a lot of instruments for the general public. 

We did a big project at the Lincoln Center for the Lincoln Center Festival called “The Brain Opera” where we made a whole orchestra of about 75 instruments, about 10 different models.  We called it “the Mind Forest.”  We filled up the lobby of the Juilliard Theater, and people would come in and you’d be surrounded with this web of instruments, like a driving game where you could drive musical notes through pathways.  Depending on how you drove the music would change.  Tabletops where you could move your fingers and change the music, big walls where you could move your whole body; same electricity system, to change big masses of sound.  Percussion instruments that looked like these rubber organic pads.  Hundreds of them all over the room.  So, the audience got to experiment with these and we put on a performance every hour at the Brain Opera, but half of it was prepared by me ahead of time, half of it we took what the audience had just done with these instruments.  So, it was a collaboration through these instruments with the audience. 

And around that time, I had a lot of really interesting students in my group, a lot of them have gone on to do really, really interesting things in all these kinds of areas.  How to make music accessible to the general public. 

A couple of these students started a company right near MIT, it’s still there.  The company is called Harmonics.  And they started taking these same, especially this idea of a driving game, where I could drive a musical note through a series of roads.  They actually took some joy sticks and had a piece of music that would play.  And depending on how you moved – if you moved the right joy stick, it would push this music to different harmonies, if you moved the left joy stick; it would make the rhythm more complex.  So, just by moving the joy sticks, anybody could modify this music.  And they did work like that for about seven or eight years.  They made some games that were kind of okay.  They were actually very good games.  They didn’t sell all that much.  And then somebody came to them and said, “Did you ever think of using – making a plastic guitar and what would it be like if you took exactly the same software and had somebody with a guitar interface instead of these joysticks."

So we all talked about that.  They made this game called Guitar Hero and nobody expected it to be a success.  And actually when Guitar Hero first came out, the big, like, Best Buy and Target and all those places bought very few of them.  But the fact that it was a guitar interface and not joysticks, so many people had imagined what it would feel like to play a guitar.  And the game play, the idea that it wasn’t just moving music around with joysticks, but you were being graded at how accurate you were and it was very easy to tell how well you were doing.  And as we know, it became a success very fast.  So, that came directly out of the work on the bow and the cello and that turning into this sensor chair and all that software went into Harmonics and is now out there as Guitar Hero and Rock Band. 

Question: You are working on the opera of the future. What will it look like?

Tod Machover: Yeah, so I never liked opera growing up.  I always liked chamber music or solo music even more than orchestral music.  But as I started writing my own music, I kept getting attracted to words and especially to – we talked before, I think, about these imaginary worlds that non-word music creates.  You’re somehow following a story, it’s some how important humanly, if it’s good music, but I fill in the details myself.  And I started realizing that one of the great things about opera is that if you make the right kind of story, you can still have this kind of abstract subliminal quality to take you on a journey, but you can root it just enough in a particular situation, a particular kind of real situation that a person might have, or a particular context in the real world.  If it’s done in the wrong way, it kind of I think puts a straight jacket on music and a lot of the power music has to take us somewhere unexpected can go away.  But if you do it in the right way, music’s subliminal power stays and it connects to your own life and to real situations more closely.  So, I’ve done a lot of operas.  I’ve probably done more different kind of operas than anybody.  I’ve been trying to explore this forum to see how to get the most out of it. 

First I had The Brain Opera, where the public helps to create it, I did science-fiction opera for the Pompidou Center in Paris, this Magic Opera with Penn and Teller.  The one I’m working on now is called Death in the Powers, and it has an original story and libretto by Robert Pinsky, the great poet.  And my original images for this opera was, I’m so tired, and in some ways technology is wonderful.  I love working with technology because it allows me to follow my imagination and to invent new things.  But a lot of the feeling of technology, you know what things sound like when they come through loud speakers, what I looks like when you go to a rock concert and you see a giant screen with Bono’s nose and on stage U-2’s about this big, and so I think in many ways, the texture of technology actually diminishes human beings.  It doesn’t augment them. 

So I started thinking, well how could we take this kind of hyperinstrument idea of measuring the way humans perform, the way they behave, they way they sing, the way they move, making that larger and more and growing it through the performance.  So, the final result wouldn’t be coming out of loud speakers and wouldn’t be just video screens, but could actually be physical things.  Could we make it so that if I move my hand here, there’s some strings that start vibrating.  There’s wind that blows through some objects, or there are multiple percussive objects that either make sound like that, or maybe much more delicate sounds. 

And visually as well.  Could we do something where flying objects and morphing forms, real physical ones combine with the music help to tell a story, even if there are no words.  So, I contacted Robert Pinsky, this poet to say, would you want to work on a crazy project like that?  Could we make a stage to tell a story?  And before we had a story, I was thinking about that.  And then we made up a story which turned out to literally be about a guy who is in his late ‘60’s, rich, powerful, successful, slightly creepy, kind of like Bill Gates meets; sorry Bill.  Bill Gates meets Howard Hughes meets Walt Disney, somebody like that.  And he’s obsessed not so much with staying alive forever, in fact, he actually wants to leave the world and he imagines a kind of better, higher level of existence.  So, he wants to go, but he wants everything about himself.  His memories, the texture of his life, his ability to still be in touch with the people he loves.  His ability to manipulate his business partners, or whatever.  He wants that to stay.  So, he uses his money and his smarts and his power to invent this thing called The System, which basically allows him – he can basically download himself into his environment. 

So in the first scene, he’s just finishing this system and his wife and his daughter and his assistants are saying, “Yep, it’s almost ready.  But are you still going – is it going to be you?  Where are you going to go?  Don’t leave.  What is this?  And he turns on the system at the end of the first scene and says, “See ya later.”  And transmigrates, and goes somewhere.  And little by little the stage comes alive.  So, the stage is like a big robot.  And his whole room, which is made up of bookcases and furniture and objects and a gigantic chandelier, which looks like a chandelier.  These all turn into him and they start to move and they start to vibrate and they start to make sound.  And they don’t literally look like him, and they don’t literally necessarily sound like him, although he does talk and sing through them in a modified form.  This becomes the future of his existence and everybody who is left has to decide, is this really him?  Is that you dad?  Is that you Simon?  They have to decide if they like this.  I mean, he’s left himself, he’s left his legacy.  This is what he wants to leave in the world.  Do they want this stuff?  Do they want to live with this Simon left behind? 

And then they have to decide if this existence – he actually want them to follow him.  So they have to decide whether what we know of as human existence is good enough.  Should we stay here?  Or is this really a promise of something better?  And that’s where the tension of the opera comes. 

So it’s very unusual.  And we have all this set that makes music and comes alive and moves and interacts with the characters.  It’s quite large.  We’re building it all from scratch at the MIT Media Lab.  We have live performers on stage as well.  We have a chorus of robots.  There are 12 robots, 7-feet tall.  They are 3 ½ feet that extend to seven feet.  They comment on the action, they’re actually slightly futuristic; they’re also being designed by this research assistant.  They’re kind of a bridge between now and the future.  And they’re a little bit mechanical.  They glide around; they don’t really understand what’s going on because they’re robots.  So, death and you know Simon’s somewhere else.  We could love – what is this stuff?  Why would you care about this?  

And then my favorite characters, I think, are the furniture.  Because wanted very unexpected parts of the set to, not exactly be human, but be able to reach out to humans, so the chairs and the tables and the sofa, all have legs that walk around and they glide around, and there’s kind of – they’re not as well developed as what we call the Opera Box, the elegant 7 foot ones.  They’re kind of strange and they’re lurching and they’re trying to be as elegant as humans.  They’re trying to imitate their voices so they learn how to talk.  And it’s just a very strange world and it’s a tug of war between the limit between a human being and what a very sophisticated technology can represent about yourself.  It’s a reflection about mortality and what you can leave behind and what you can pass on to others.  And I think it’s a reflection about being human.  Strangely, some of the most human characters was one of my themes generally, I think some of the technology especially these furniture robots are more emotional and more real, more human, certainly than you’d ever expect technology to be then some of the human.

Question: What do you predict will be the next breakthrough in music technology?

Tod Machover: We’re at a very, very interesting, exciting and a little bit scary point in music technology.  I mean, over my career, I’d say the last 25 years; we’ve gone from music and computer being for 10 people in the world to having personal computers, to now being able to do amazing things on your iPhone, or with Rock Band.  So, right now there’s enormous capability with technology in our devices that everybody has access to. 

The one obvious thing is that the devices are so good now that you can also see their limitations extremely well.  So, there are a lot of music apps on the iPhone, for instance.  There’s an ocarina, there’s music instruments you can play, there’s some rhythm instruments, there’s some editing instruments.  And it’s like the dog that can talk.  I have a dog that can talk.  Oh gee, but I can’t understand, I can only understand one or two words that he’s saying.  But, it’s a dog, that pretty good.  So, an iPhone doing what it can do with music, it’s kind of remarkable, but it’s just very crude.  The sound isn’t that refined.  I mean, I’ve got one in my pocket.  It doesn’t feel like a musical instrument.  It’s hard, it’s fragile.  You can’t squeeze it, you can’t touch it.  It’s even much more kind of – it’s fine for most of the things you do with multi-media, but even think of a piano. I mean, a piano is a machine, but you’ve got ivory and there’s weight behind the keys and you have this really – you feel the resonance in the instrument, you feel the vibration in the pedal.  I mean, these a still very crude. 

So, I think one thing that will happen is that the kind of merger of hyperinstruments, of instruments that have a real physical feel, that are sensitive, that are worthy of somebody learning and mastering will be combined with a lot of computing and software power already in these little devices.  So, I think we’ll start seeing real instruments and composing technology that is available to many people that is just a lot more sophisticated than this.  I think we’ll see something like Rock Band and Guitar Hero, which right now is a huge amount of fun, but it doesn’t really teach you about music, it doesn’t really help you listen even if you’re playing with four or five people, you’re glued to the screen making sure you’re following the visual symbols.  It doesn’t teach you about personalizing your performance.  It doesn’t rate you on “is your performance more beautiful.”  There are no adjectives involved. 

So I think that’s another frontier certainly we’ll see pretty soon, things like music games that are also creative and expressive and worthy of learning.  But I think the big, big breakthroughs really have to do with our growing understanding of what goes on in our minds and why is it that music affects us at all.  I mean, it’s still a pretty big mystery that music exists in every culture, every society that we know about, but it’s very hard to trace back why that would be.  You know, what are the biological reasons, what are the social reasons.  Why does every society seem to want to make music when it often seems like kind of a frill.  Well, it’s not a frill.  And we probably don’t have time to talk about why.  I’ll come back and we can talk about that again some other time. 

But music seems to stimulate more parts of our mind than almost every other activity.  It combines more parts of our minds.  It synchronizes our minds.  It allows people in groups to do a non-verbal immediate activity together.  You could argue that is synchronizes people’s biological – many, many biological functions for individuals when you are part of a group making music together.  There’s many reason music exists and we are beginning to no only understand that, but measure that. 

So one thing that I think we’ll see, we already starting to see people being interested in it, but this is something in the next 20 years, we’ll start seeing enough knowledge developing that we’ll be able to have some objective ways of measuring whether a particular piece of music is pleasing to you.  Whether, if you are in a particular mood and you want to keep that mood, or be in another mood, whether by listening to his particular piece of music, or by changing that music while you are listening, whether we can reinforce something, pull you somewhere else.  You can think of it – in some ways you can think of it like a perfect psycho-analysis session, or like a yoga session except the music is a medium to lead you somewhere or reinforce something.  It’s really being used as a more precise instrument than we could possibly do now.  Not just guessing, but really measuring the effect it has on you both to select it and to play it and to modify it. 

I think we’re going to start seeing a variety of music environments rather soon and developing over the next 25 years.  So music that adapts to what you need at a particular moment.   I think over the next 25 to 50 years, let me put it this way.  All music right now, when there’s a hit, the reason it’s a hit is that a composer and a performer have identified some common quality in that music, something about it, something about the way that particular melody falls that a very large number of people are going to find attractive.  I mean, that’s the definition of popularity.  Something that literally resonates with many, many people. 

So all the music we know that’s popular is actually commonly shared music that takes things that are similar about all of us.  But if you take the idea that we’re going to be able to measure more and more your particular mental structures, your particular reactions to a piece of music, your particular needs at a particular moment so I can select music for you, watch you while you’re listening and fine tune the music.  If you can imagine completely the opposite thing happening, which is that perhaps 25 to 50 years from now, I can design a piece of music, no so that it appeals to something common in millions of people, but I can design the music so that it’s exactly right for you and only you at this particular moment for your particular experience, things that have happened to you over 20 years, to you’re particular mental state right now. 

Recorded on January 14, 2010