Sunday, April 9, 2017

Turning Pictures Into Music

Some people use their spring break to go on vacation to New York or publish a science paper. I chose to spend mine making some bullshit on my computer and resurrecting my blog that I haven't posted on in months.

Anyways I’m back with some more nonsense. And it’s some unusually big-time nonsense. No neuroscience here, not really data science, you won't be inspired. There is literally no real purpose for this project except that we thought it would be cool.

To fully experience this post, you'll need to be able to have sound playing out of your device and be without fear of being judged by those around you.
Also I'm using this weird video format so you'll have to turn up the volume on the sketchy videos.

Image result for days since last nonsense

TLDR (too long, didn't read):

My friend and fellow graduate student, Vivek and I made a little algorithm that turns pictures into songs. For instance, I can give our algorithm a picture like this handsome logo for the Northwestern University Interdepartmental Neuroscience Program:

and it'll make a nice little song like this:


Yup. That's about it. Dozens of hours of work...

If you're still with me, this post will explain how we came up with this idea, show you some weird youtube videos, provide some examples of what our thing can do, then explain how it works. If all goes as planned, this should all amount to a substantial waste of time for both you and me. I am confident that we can achieve this goal.


A few weeks ago I was talking with Vivek about levels of meaning in music. When you listen to a song you can just enjoy the nice sounds but it’s so much cooler when you know something about what’s going on behind the curtain. If you know about music theory, or the artist who wrote the music, or the context that a song was written or performed in, it just makes the experience a little richer. For example, when you learn something like how Dark Side of the Moon synchronizes with The Wizard of Oz, or that Closing Time by Semisonic is not about a bar but about the singer's son’s birth - it just adds this whole new layer of meaning to a song that you already liked. And we thought that was an interesting concept.
I stumbled onto some really interesting youtube videos that demonstrate this idea of embedding meaning into a song in this totally new and fascinating way. In the following videos, the artists have put visual pictures into their music.

This group engineered their music so that when it is played through an oscilloscope it makes pictures that go along with the music (~3:20 is especially cool IMO):

Shrooms by Jerobeam Fenderson

So what you’re seeing in this video is literally the waveform of what is being played out of your speakers. Most songs would look like indiscernible noise if played through an oscilloscope, but they made this music with the purpose of the actual waveform being interpretable visually. I am kept perpetually amazed by the things that engineers can do. Details here:

Here’s another one where the artist inserted a picture of his face into the spectrogram of the music. And music has a very loose definition here. I don't find this 'song' to be particularly musical. But the important thing here is that he somehow engineered the song to play a series of tones at the specific amplitudes and frequencies in the middle of the song that made this picture (Skip to 5:30 for the face):

Equation by Aphex Twin

Also I just now remembered seeing this super weird piece of sheet music that someone left in the jazz band room in high school:

It's kind of the same idea of embedding visual aspects in music. Not super relevant, but whatever. It's neat and worth sharing. Don't yuck my yum.
(Info here:

Then while I was working super hard in lab and not browsing reddit I found this meme where people were making midi files out of pictures. In other words, people will take a simple line image and turn the lines into notes to put into music software like GarageBand, where the horizontal spacing is the timing of the notes and the vertical spacing is the pitch of the notes.

Drawing pictures with music - Andrew Huang

To do this, the guy in this video printed out the image he wanted, manually traced it by hand into the music production software on his computer and adjusted the notes to make it sound nice. As people who understand a small amount about computers and music theory, Vivek and I thought this was cool but realized that we could do this ~algorithmically~ because what the world needs right now is an algorithm that saves the precious time of artists who make pictures of unicorns into music. So we chose to bear this great burden. You're welcome world.


We did it.

And not only can we process an image into a song, we can tweak features of the image to make it sound a little nicer. For instance, this is what the NUIN logo sounds like if we don't edit it:


This sounds pretty bad. But you can improve bad things and make them better, like me at doing grad school. In the image above, we're playing so many notes at a time and there is no organization to the tonal quality so it sounds like shit. It's like pressing random adjacent chunks of keys on a piano erratically over time. But we can fix that.

Here's what the NUIN logo sounds like if we shift all of the notes to notes in D# major and play them with a more forgiving instrument:


And we can add chord changes so that for every 16 beats, we switch to a new key in pattern. Dare I say that this sounds almost kind of good?


Here are some Examples:

Lax in C# Major.

Doge Blues (I tried running this on a a picture that wasn't a line drawing. It wasn't great at getting the important features in the image but sounds pretty cool.)


Northwestern Wildcats Electronic Odyssey.
Powerhouse of the cell for bass

Poop in C Major for Trumpet Ensemble


And this:


Also this:

OK that's enough of that. 

Let me know if there are any images that you're dying to hear and I will consider making it for you! (or just use cool open source algorithm we made)

(All code was written in Python and can be downloaded from github at

This project seemed complicated at first but we made it easier by splitting it into just a few discrete, not-too-hard parts:

1) Get the important parts out of an image and turn it into a matrix.
2) Manipulate the matrix to clean up the picture and make the corresponding notes sound nice.
3) Turn the matrix into a midi file that can be played by a music program.

1) It is easy to take an image and turn it into a 3d matrix of RGB values for each pixel using the sk-image library. But you can imagine that if we 'played' an image like this it would sound totally noisy and crazy because there would be values at every pixel that would correspond to notes. We chose to filter the image for any prominent edges that it has and only include these in the matrix. We did this using a Sobel filter. I don't know how it works, but it does work and it works pretty well. This outputs a 2d matrix where edge intensity is coded for by a value between 0 and 1. So faint edges play softer notes, like the second half of the brain in the NUIN logo and the second N in NUIN. Also it had a little trouble with the U for some reason. We also built some parameter tuning functions into our code so that we could threshold which edges to include. I had to play with this a lot to get the doge to work. That was time well spent.

2) The next challenge was to take a matrix and manipulate it such that it maintains the visual picture, but sounds a little bit better. First we built simple functions to resize and pad the x and y dimensions of the image using scipy.signal.resample. In this way we could choose how many beats we wanted the image to play for and the range of notes that would be used to play it. The more interesting methods we made could determine the starting note and the key of the song and these could be called repeatedly to make chord changes.  

3) Lastly, we found a library that writes notes to MIDI files - a file format that is pretty much a piece of sheet music for an electronic instrument (or soundfont) to play. We could just use this function as we iterated through the picture/music matrix that we made and dump it into a midi file with a function that we wrote. Then we just dropped the MIDI into GarageBand to play the 'song.' There's probably a fancier way to programmatically make videos of the MIDI being played but whatever. We tried to do this and we started to write some fancier algorithms to change more things about the picture and music but then we realized that we had spent way, way, too much time on this 'project' and it was time to stop.

All of our code is written in Python and can be found here:

Feel free to use any of it but if you somehow find a way to make money with this then please give us some.

What did we learn?
Absolutely nothing. But I hope that you had fun and that someone you want to impress didn't see you listening to the song made by the poop emoji.

Since I posted this, a few people have pointed out other media that is related: This is a piece called Pictures at An Exhibition written by a Russian composer named Mussorgsky. He wrote this song after being inspired by an art exhibit. There are several movements in this work that each represent a work of art from the exhibit. (Thanks Sean McWeeny!)

The results of this algorithm have been likened to the work of La Monte Young, who has been called one of the greatest living composers today and who's work called into question the nature and definition of music. (Thanks Claire Chambers)

Friday, June 3, 2016

Studying Fake Busses With Science!

My last post was about my experience at this cool hackathon I went to. This post is about what I actually did there. 

All of the code that I used is written in Python and is on my github [Here]. I'd like to think it's pretty readable. You can also run all of the code and reproduce all of my figures in Binder without even having to download anything!

Look its that thing we made

For the hackathon we came up with the idea of using the city's public data to improve bus usage in San Diego. We noticed that certain bus routes were either overused, underused, or had surge times. During commute times in San Diego people actually wait in line for the bus. They get to the bus stop and some of the people don't get to even board the bus when it comes because there are too many people. This phenomenon is whack and presents lots of space for improvement. 

Whack bus line somewhere thats not in San Diego

The Plan

Here's what we set out to do:
* Build an app that the city to use to visualize and analyze bus usage data

* Identify overused, underused, and surging bus lines.
* Explore ways we could partner with rideshare companies to make busses great again (sorry)

We learned that the city of San Diego just started started installing infrared sensors on the doors to the bus to count how many people get on and off bus at each stop. This seemed like a great place to start... but we couldn't get our hands on that data over the weekend with 0 hours notice. Crap. 

Well in neuroscience if you want to study something that there is no data for, the best thing you can do is to build a simulation. And by simulation I mean a big 'ol math equation. This math equation will output realistic data by making certain assumptions about the underlying processes that generate it Then you can analyze the simulated data while you tweak the parameters to see what happens. 

Here's what I decided were the important features of one day in the life of a bus line:
1. Number of stops
2. Max capacity of the bus
3. How many passengers get on at each stop
4. The stops where the bus fills up
5. Number of surges in the bus line
6. When the surges happen

This seems like a lot of stuff to stuff into a math equation but it's actually not too bad if we assume bus demand is governed by A: some combination of random noise (riders randomly get on and off stops irrespective of the time) and B: oscillations (rider demand increases and decreases over time). I whipped up a simulation of this [here] but if that's not your thing then I guess you can just take my word for it.

Fake Busses

Using my algorithm I could create daily bus routes and change different things about them. Here are some examples: (blue line is the number of passengers on the bus at each stop, red dots are where the bus reaches capacity of 40)

If I make ridership completely random it looks like this:

And if I make ridership completely oscillatory it looks like this:

Both of those look kind of artificial but if I find the happy medium between noise and oscillations it looks a little more realistic.

The previous models all have the same number of times that they fill up. I can also change the overall demand without changing the timing of demand. 

High demand:

Low demand: 

Cool. Now we have a bunch of fake buses... Why did we do that again? 

We made these so that we can use them as templates for figuring out how to analyze them. If we only have the data, and we don't have access to the parameters that generated it, can we infer them? In other words, is there a way that I can give you numbers that characterize the noisiness, 'surginess', and demand of a bus line?


Demand is easiest so I'll talk about it first. If we want to know the total passengers who got on a bus in a given day, we just have to find the integral of the number of passengers on the route over time. There is a handy function for that called sum().

The more important feature of demand is how many times the bus filled up. If a bus line has lots of demand but doesn't fill up then thats perfect. I want to know when the bus is too full. I did this by set a threshold at the max capacity in for the bus and each time the amount of passengers on the bus reaches that number I record the times. It looks something like this:

These are two bus routes that both filled up 20 times but the way they filled up over time is totally different. The blue one filled up randomly, but the red one filled up in bursts. It's obvious to a human which is which but if we had 1000 bus lines that we didn't want to manually comb through, could we write an algorithm that would tell us if a bus line is noisy or surgey automatically?


Yes we can. Some neuroscientists were faced with a similar problem in analyzing spike times in neurons and figured out you can quantify 'burstiness' using the coefficient of variation of the inter-spike intervals (CV).

Check it out. If I take a purely surging bus line and iteratively add a little bit of noise, CV tracks this change pretty well - before it gets too noisy.


Cool. Now we can use CV automatically tell us how noisy and surgey busy bus lines are. But wouldn't it be helpful to know how many surges there are and when they happen? It would, dear reader, and we can do that pretty well using k-means clustering. If you look through my code you'll see that I tried the traditional approach of finding the optimal k using the elbow method but I couldn't get that to work so I made up my own thing. 

Here's an example of my method identifying the surge times in a bus line:

It looks like it works but I totally could have lied so here's a little more proof - here's how it does on bus lines with between 1 and 5 surges and 50% noise:

Summary, Problems, Caveats, and Future Directions

Overall I did a pretty good job. I was successfully able to quantify demand, noisiness, and surges in simulated MTS data. By doing this, I could take large amounts of ambiguous 1's and 0's and turned it into actionable information someone can use to understand and hopefully improve the system. 

However, there are some important things to acknowledge. First of all, my simulation is definitely still a little artificial. Bus surges are probably not totally sinusoidal and I don't think surges ever happen at 3:30 AM. This feature shouldn't really affect the way I analyzed the data but it makes the simulations seem a little less genuine. Moreover, as the real data comes in, it might turn out that noise/oscillations are not the best way we can describe demand. 

CV is also a pretty finicky measure. It is highly sensitive to both the length of time being analyzed and the total number of points where the busses fill up. I didn't think this would happen and I have been using this measure in real science code so I gotta go make sure it really makes sense.

There are much more sophisticated ways to characterize peaks in the presence of noise, (somebody might even publish an algorithm that does just this for neural oscillations someday). There are also several other machine learning algorithms one could use to analyze bus ridership that I didn't get to for the sake of time. For instance, longitudinal ridership could be clustered over weeks and months, and we could do outlier detection for unusual surges. 

Feel free to mess around the code and let me know if I made any mistakes!

Monday, May 23, 2016

My First Hackathon

I went to my first hackathon, The San Diego SmartCity Hackathon, this weekend and it was absolutely incredible. 

If you're interested in the actual project I did, click here and the code for it is here.
This post is more about the actual experience of attending, which I found to be really meaningful.

Here are the takeaways:
  • GO TO A HACKATHON. This entire post will be mostly be a demonstration of why you have got to go to one of these things no matter how technically incompetent you think you are.
  • I thought I would be intimidated but the environment was extremely supportive and inclusive
  • Wizards are real and they are at hackathons.
  • I joined a team, made some friends, and contributed to a project that I'm really proud of.
  • We didn't win but it was still super worth it.

I almost didn't go to my first hackathon this weekend.

On Friday afternoon I remembered that I had registered to go to some "hackaton" thing on campus that evening. The keynote speaker was someone I had heard about and I was hoping there would be free snacks and coffee. I was tired from the week and wanted to just go party but I thought I might as well check it out because I had a ticket. 

If you aren't familiar with the term, a "hackathon" is an event where nerds amass to spend a weekend building tech projects together. I had heard legendary stories of people who conceived of an idea at a hackathon in San Francisco, worked on it all weekend, and got hired to work on it full time before Monday. I didn't think this kind of thing was real.

I had no intention to participate. I just wanted to consume the coffee, snacks, and free entertainment that I was entitled to. I get to the room where this guy is going to speak and there was free beer and tons of hor d'oeuvres. Nice beer and legit hor d'oeuvres too. Nice. I stood in the corner, stuffed my face, and just kind of observed. The room was full of about 100 people, 90 dudes, 125 computers (some desktop boxes that people must have lugged from home), 10 cameras, 20 pieces of technology that I couldn't quite identify, and one big virtual reality setup taking up a corner. I started chatting with some dude as we were all ushered to the auditorium. 

This dude, Rushil, had been to a couple of these things and programs mobile apps for fun. As we walk to the auditorium, some older guy in a suit overhears our conversation about data visualization and starts telling us about his tech company. After a few minutes, he gave us his card and offered us summer internships on the spot. Woah.

Joining the hackathon

The featured speakers talk for a little bit and it's pretty cool. Then the director of the hackathon invites any teams who want to pick up additional members to present their projects. A couple people get up and talk about what they plan to work on. 

The theme of the hackathon is to help San Diego come up with ideas for how to deliver on it's Climate Action Plan. It turns out that California has a shit load of environmental problems, believe it or not. The city just released a ton of infrastructure data to the public and was interested in whether these kinds of data could somehow be used to ameliorate some of the city's environmental problems. It also turns out that people who are highly technically competent like to solve these kinds of problems just for fun on a weekend, basically for free.

The project presentations begin. They range from, "We're pretty much an early stage startup trying to build a website to gamify environmentalism and looking for anyone who wants to join", to "Wildfires are a problem. Maybe we can use drones or sensors or something?"

Rushil and I start to talk about what is good and bad about certain ideas. He's all fired up to join a team and win this thing. I'm embarrassingly buzzed from my Stone IPA and entertaining the idea of hovering around a team and checking out what this is all about.

After the presentations, we introduce ourselves to the leaders of the groups we're interested in joining. We talk to the drones/something/wildfires guy, whose name is Victor and start talking about how we could work on this. Then we start bouncing other ideas off of each other. We come up with some pretty cool projects but realize we have to think about what kinds of projects we could do with our specific skills.

These are our skills:

* Has completely programmed dozens of mobile apps
* Well-versed in business, economics, and tech entrepeneurship

* Experienced freelance full-stack web developer who can build literally anything on a computer
* Made the website for the event
* Is friends with all of the organizers and seems to know the whole SD tech community

* Claims to be a "neuroscientist" and "data scientist"
* Has no idea what's going on
* Still kind of buzzed from one beer

Seems like we're all about the same level. Nice.


So I guess we're a team now and we head into the coding room to work on ideas together. 

Holy crap. More free stuff. T-shirts, Sandwiches, pizza, high-quality granola bars, Red Bull, soda, and chips. I switch from beer to Red Bull and we start to talk about what to work on. 

We try to solve forest fires. Forest fires are complicated and none of us know how to do hardware engineering. Crap. Next we get really excited about making a fitbit for your house. You'll get points for using less energy, water, and waste. Then you can post that shit on Facebook and brag to your friends. Victor and Rushil can build it and I can do stuff with the data. Crap somebody already made that. Ugh. We think about making an online game that would optimize recycling bin locations and joke about how funny it would be if we could trick people into picking up trash off the ground to get points. Suddenly it's 12:30 AM and we have no solid idea. I'm thinking of quitting my new team. 

Then we think about if there's any way we can use the city's data to improve the bus system and maybe partner public transit with ride-sharing services like Uber or Lyft. I'm not super excited about it but I can't prove to them that it's a dumb idea not worth our time. We head out a little after 1 AM, add each other on Facebook, and message each other til about 2 AM.

I wake up to my phone alarm at 7 AM on Saturday. We had planned on meeting at 8 AM.
"What am I doing awake right now? Am I really going to go back to this thing? I don't even really know these people, our idea is super fucking dumb, and we came up with it in the middle of the night."
I somehow convince myself to get up and head back.

When I get there, I see people who seem like they had been there all night. There are people programming the crap out of some complicated looking stuff and all we have is a weak idea. Somehow the magic happens and we start to get this project rolling. Rushil and Victor start building the backend of a website we can use to visualize bus usage data. I start modeling bus use data and coming up with analyses we use to characterize it. We absolutely crush work for most of the day, not even stopping for lunch.

We meet again in the morning and put on some finishing touches. The 'finishing touches' take way longer than we expect. We frantically finish up our project and barely make the deadline - coding until the second the judges arrive to evaluate us. 

Presenting the project

Our final project kicks ass. It's a dashboard that a city controller can use to visualize bus usage on all of the city's bus routes using the new passenger counter sensors they're adding to busses. We can algorithmically label high-load and surging bus routes (post about this, code I wrote to do this). We have economic value models explaining how a partnership with Lyft could help with bus use surges to make public transit more reliable. The app looks nice.

The first judge comes over. Rushil gives the elevator pitch. Victor shows off his web platform. I begin to speak on the caveats of modeling surges in bus demand at length. I'm cut off quickly. I didn't even get to brag about clever math I used because we only have 4 minutes total to present. This is not a science conference.

Our pitch becomes better with each judge and by the end we sound pretty darn smart.

While the judges deliberate, I look at a couple of the projects and they're unreal. I was completely in awe of what these people could make in a weekend. One group was using sensors to optimize water efficiency when watering plants with hydroponics. One group had the idea of displaying the city's water supply on the treasury building using LED's so the building would look like a giant, kind of empty, cup of water.

The finalists were announced. Unfortunately, we were not chosen but the ones that were chosen were pretty awesome. Remember when I said we were joking on Friday night about making a game out of throwing trash away? The group with the virtual reality setup made a game where you throw objects either into the trash or recycling in virtual reality. They got picked. It actually looked pretty rad. Damn it. One group built a low-cost sensor that could be put in forests to detect forest fires and instantly communicate with the fire department. These sensors could also be repurposed to help 911 identify your location when you call far from a cell tower. They were not even finalists. 
One group received an honorable mention and was publicly offered a job with the city.

We didn't move on to the next round, but we received a lot of positive feedback about our project and we talked to a few people about what our next steps could be. We'll see what happens.

Go to a hackathon

Before I got there I was worried that I would be laughed out of the room for not knowing anything about tech. I was accepted with open arms and I found that I had a niche set of skills that were necessary to make our project work. Not many engineers have the skills that scientists do! Moreover, many projects were in dire need of non-techy skilled individuals like design specialists and people who could come up with projects and keep them on track.

Every person there seemed to be genuinely interested in helping other people out. Staff members and participants were frequently helping out with each other's projects. I heard the following conversation dozens of times: "Oh you're trying to do this? I know all about this. You should use that! Here's how you do it...(technical jargon for 5 minutes to half-an-hour)"

This was one of the best events I've ever been to. The breadth and diversity of technical knowledge at this small room could rival the entire Annual Society for Neuroscience Conference. The encouraging, friendly, and positive atmosphere could probably outdo most music festivals. I found myself in a group with two new friends and we built something in a day that I couldn't have done in a year on my own. 

I got a whole lot more out of this experience than the snacks, coffee, and lecture that I came for.