If only someone could write this for me… plainpicture/cultura/Lauren Devon

Got words but no melody? A machine learning system turns poetry into song by composing a pop music score to suit the lyrics it’s given.

“I was studying singing while I was doing my PhD in computer science,” says Margareta Ackerman at San Jose State University in California, who developed the system with David Loker at technology advisory firm Orbitwerks. “Over time, I started to think of computers as creative partners instead of tools, which could maybe help me write songs.”

The system, called ALYSIA, processes short lines of text and associates each syllable with a musical note. It chooses the pairing based on features including the syllable’s position in the word and how it will fit with the previous five notes.


ALYSIA can write whole accompanying scores this way, or provide musicians with a variety of melody options for each segment of lyrics, acting like a co-creator. Ackerman and Loker developed the system to produce pop tunes, but say it could be adapted different genres. The system uses two models, one focused on rhythm and the other on pitch. These were trained on the melody line and lyrics of 24 different pop songs.

They then used the system to make melodies to accompany two sets of words written by Ackerman that involved it coming up with tunes for lyrics such as “Now that you’re gone / I just realised that I’m all alone”. They also fed it the lyrics to the vaudeville classic I’m Always Chasing Rainbows to see how it could reimagine the song in the pop genre.

Getting in tune

The idea of trying to automate musical composition is not new, but David Cope at the University of California, Santa Cruz, says ALYSIA is unusual in taking lyrics as its starting point. He is impressed that the system manages to match the metre of the melody with that of the lyrics, but says the compositions show an “almost annoying” lack of harmony.

Rebecca Fiebrink, a researcher in machine learning and music at Goldsmiths, University of London, questions how useful the lyrics-to-melody approach is. “Is this really solving the compositional process for people who want to make music?” she says. “Creating a melody without additional accompaniment, like this system does, is the easiest thing to achieve.”

The songs admittedly aren’t about to win any Grammys, but Ackerman says this is just the start. She initially imagined targeting ALYSIA at the electronic music community, but is now working on repurposing it for professional songwriters with the help of classical composers.

Ultimately, Ackerman hopes to create a system capable of composing all aspects of a song on its own. “We want to design a program able to generate the music, the lyrics, and ideally even the production and the singing by itself,” she says.

Journal reference: arXiv: https://arxiv.org/pdf/1612.01058v1.pdf