Over the next months I built one of the first MIDI sequencing and music editing applications for the Macintosh. The program was called Total Music. Instead of recording sound directly, it recorded musical events—what note was played, when it started, how long it lasted, and how hard the key had been struck.
Those events appeared as a musical score on the screen, where they could be edited, rearranged, and played back.
A year later my wife and I were demonstrating the system in New York at Manny’s Music and Sam Ash. Musicians leaned over the keyboard watching their notes appear on the screen in real time.
The idea spread quickly. Thousands of composers and performers began using the program to write and edit music.
The most interesting part for me wasn’t the editing. It was watching how musicians actually used the system and thinking about what they needed next. They wanted better tracking of rhythm and time signatures. They wanted ways to stretch or compress passages to hit precise cues for film and television. They wanted the software to recognize riffs and phrases they played repeatedly.
To make that work, I started experimenting with ways for the program to compare groups of notes.
The idea felt strangely familiar.
Instead of comparing patterns of pixels the way the vision systems did, the program compared patterns of notes.
It looked at pitch, how long each note lasted, and how hard the key was struck. Then it calculated a kind of “closeness” between sequences—how much one phrase resembled another.
They were rarely exact matches.
It was all about resemblance.
At the time I was just trying to make the software useful for musicians. Only later did it occur to me that treating music as patterns like that was something very few people had tried yet.
The computer still didn’t understand music any more than those early text programs had understood language.
But it was beginning to do something interesting.
It was learning which patterns were more likely than others.
Looking back now, the path seems obvious. First we experimented with patterns in text. Then patterns in images. Eventually those same ideas began appearing in music.
At the time, though, none of it felt like a grand progression. Each step was just another small problem in front of us—another puzzle to solve, another machine to take apart and see how it worked.
But over time those small experiments began circling around a much larger question.
What would happen if a computer could learn patterns on its own?
By the mid-1980s researchers in several laboratories were beginning to explore that possibility using systems loosely inspired by networks of neurons in the brain. The idea itself wasn’t new—scientists had been experimenting with similar models since the 1950s—but the computers of the time were still small and slow, and the early results were uneven enough that the field drifted in and out of fashion.
The question, however, never really disappeared.
If machines could recognize patterns in language, images, and music, it seemed natural to wonder whether they might eventually learn those patterns for themselves.
And somewhere in that question lay the beginning of the next part of this story.