Tuesday, July 25, 2017

The Creator's Game: Lacrosse and Neuroscience

I've had a hard time putting my finger on exactly what makes me love lacrosse so much. I've wanted to write this down for a long time but I've had trouble organizing all of my thoughts on this topic into words until last weekend. 


My Squad

Last weekend I was at the Lake Tahoe Lacrosse Tournament with roughly over a thousand other people who have been playing this game since they were kids but just can't seem to put their sticks down. I'm not aware of any other sport that draws so many people to train and compete after the glory of high school and college is long gone. Tournaments like this, of which there are dozens, often have masters divisions (for people over 30), some have grandmasters (50+), and one even has a zenmasters division for 60 year olds+ (!!!). I'm convinced that there is something really special about this game and I'm starting to put together why that might be. This game is designed unlike any other sport in that it constantly engages myriad aspects of the players' minds. Playing lacrosse facilitates gaining a deep understanding of one's own mental processes and other peoples'. This post is going to be half love letter to lacrosse and half all neuroscientists must play lacrosse if they want to understand how our brains work. I'm not aware of anyone else who is already interested in both of these theses besides myself but I hope that you see what I'm getting at by the end of this.

Someone else's squad



While I played at a highly competitive level in college and I think that I know something about how brains work, Unfortunately I am a mere sub-sub-sub-zenmaster at lacrosse and I definitely don't completely know how brains work. I think there is a real connection between these things and I'll try my best to stitch them together with the tools that I've got. 



Lacrosse is the Creator's game 


Last weekend a teammate told me to read the story of lacrosse, which I had never been exposed to in roughly 15 years of playing. When I read the story, many of my feelings towards lacrosse fell into place. Lacrosse was invented by the Iroquois tribe somewhere around what is now upstate New York and sometime around 1100 AD. Some sources say it was invented to prepare young men for war but the way the Iroquois describe it characterizes lacrosse as something much more spiritual that connects them to their minds, nature, and each other. There's a beautiful origin story of lacrosse that you can read here (http://iroquoisnationals.org/the-iroquois/the-story-of-lacrosse/), or you can read my oversimplification below.

The first legendary game of lacrosse was between the four-legged animals and the winged birds. The Bear, The Deer, and The Turtle were the captains of the four-legged animals. The Bear was strong and could overpower any opponent physically, The Deer was quick and could cover long distances, and The Turtle was strong and could withstand blows from any opponent and still advance the ball forward. The Owl, The 
Eagle, and The Hawk, were the captains of the winged birds. The Owl could keep track of the ball, while The Eagle and Hawk were swift and agile. The four-legged animals shunned The Mouse and The Squirrel because they were small but the birds recognized their value and added them to their roster. And beautiful, epic, long story short - while the four-legged animals were physically stronger, the birds won the first game of lacrosse by appreciating each other's skills and working together.



Lacrosse was not made for war and victory, but to recognize the value of each individual and work together. The Creator made lacrosse so that he could watch all of his children enjoy the game and also as a 'medicine game' they could use to heal. I can vouch that it is certainly fun to play and its easy to imagine a Creator proudly watching his children playing with literally the best toy ever made. I can also attest to the healing aspect to this game but it is hard to explain without experiencing it. Obviously the physical exercise involved in lacrosse is good for you but there is a substantial mental health component to this game. The four-legged animals represent the physical nature of the game but the winged animals, the more successful team, use a subtle, harmonizing collection of mental skills to succeed. Playing this game well requires one to think critically about his own abilities, develop the understanding that other players' skills are often different than his, and that we can create beautiful things by combining our unique abilities together. From the outside, lacrosse looks like an aggressive, violent sport - and to some players that is unfortunately all that it is. But for me at least, and I think many others, I play to have new experiences, to develop and understand my human abilities, and learn the way other players think about the game and the abilities I don't have. Psychology calls this idea of attributing mental states to others 'theory of mind.' Practicing this skill feels both therapeutic and helps me as a scientist push my understanding of how our brains work.



Neuroscience and Lacrosse - Experience Matters


Doing neuroscience is similar to lacrosse. We observe what a brain is doing and we try to reverse engineer how that might have worked. In lacrosse you're given rich experiences of high-level brain function but in neuroscience you can simplify complicated scenarios and study their components. Psychology and neuroscience have explored a lot of the winged-birds' skills and have developed a vocabulary for these ideas. For instance, there is a lot of scientific work on how the brain adapts to allow non-human primates to integrate a tool into their body schema. When this came up in class, I instantly remembered the hundreds of hours I've spent throwing a lacrosse ball against a wall and the mental experience of actually feeling my stick become part of my body. Neuroscience dissects selective attention, the psychometrics of different components of visual acuity, and sensory integration by splitting them into their components and finding how each of these atoms work alone. When I play lacrosse, I can feel all of these components churning together in my sensorium and building my spatial representation of the world. When I lose track of the attackman I'm guarding, I understand the gaps in my perception better, how he found a way to take advantage of them, and how I can adapt. 

Not so surprisingly, the foundational work describing how our brains can connect the location where a sound originates from to an eye movement towards that location, was all found by studying juvenile owl brains. (Thanks Owl). 
Thanks Owl
see here for more if you have access: https://www.ncbi.nlm.nih.gov/pubmed/12015612

Rigorous science is necessary to understand the brain. But we do this work with controlled experiments in labs with simple circuits of neurons and explain our ideas to each other using diagrams and powerpoint slides. 

It's kind of a catch-22 where I have rich experiences combining many mental processes playing lacrosse that I can deeply intuit but don't understand mechanistically. And the only way we know how to understand how our brains do such amazing things in natural and social environments must simplify these complex, interacting systems and study them in artificial laboratory environments. Experiencing and training the complexity of our minds is central to the function of our brains in real life scenarios but I don't think this is appropriately appreciated in academic neuroscience.

I don't think that a blind person will ever be able to fully comprehend vision even if a scientifically complete explanation of it is available to them. Can this possibly be false? Taking this a step further, I'd argue that someone who spends time engaging and pushing their mental faculties at a high level in a dynamic, natural environment might have a better intuition for studying the system scientifically than someone who only studies it in a lab.


This is a strong assertion and I don't have evidence to support it besides my obviously biased experience. 
Detailed research into neural circuits is necessary to understand the brain but there is something about playing lacrosse or otherwise, fully engaging with other people using their mental skills at their limits - that facilitates gaining deep insights into cognition. Each time I play, I learn a little more about each of these processes and whether they worked in the specific situation that I used them. 

My mind is on fire when I play lacrosse. I'm using my entire brain at once and learning about it as I engage with the game - It feels amazing. When I play defense, the neurons in my brain churn through ATP, carefully modulating circuits that keep my mental representation of everything on the field updated. I use my long-term memory knowledge of the game and modify it with novel aspects of the current context. I have to regulate my emotions towards blatantly incorrect, unjust, and stupid calls from the ref and actions of other players. I'm constantly trying to read my opponent's mind and predict what will happen next in the game, while he knows I'm doing this and is trying to trick me. Then I integrate all of this information and more into multitudes of small decisions and actions that impact the game in this giant, complex, recursive, loop that I'm a part of. It's not like I'm not some sort of brainiac, I'm just one of twenty people on the field who also all have brains doing the exact same thing but each metallization flavored like a different kind of ice cream. A good lacrosse game is like a Baskin Robbins. 



Lax vs. The Creator's Game

Like every mention of lacrosse, its important to note its place in contemporary culture as a caricature of elite, east coast, overprivileged, private school, white, male culture - which is a stark contrast to its origins. I'm conflicted because I'm not sure if I would have been exposed to neuroscience otherwise. A significant admissions bump from the lacrosse coach separated me from swaths of applicants and got me into an elite college with a great neuroscience program that I was only starting to get interested in. Moreover, lacrosse has pushed me to think in new ways and has given me a rich bank of experiences I use to think about how our brains work every day at work







The elephant in the room is that one might think all sports do this. And they probably do to an extent. Also, there are obviously other ways of having formative cognitive experiences besides playing lacrosse. The critical aspect of lacrosse that makes it different from other sports and activities is that there is a way for just about everyone to participate and playing any position well requires one to develop deep insights into their own abilities as well as others. While it might be easier to be good at lacrosse if you start out as a Bear or a Deer, I see Squirrels outplay them all of the time by engaging their mind - learning, practicing, and understanding their own gifts.


 


I hope I get to see this kid out at an open tournament some day. I bet even the zenmasters might even have something to learn from him.


Lacrosse is Both Intellectual and Spiritual


I guess that the conclusion of this post/rant is that lacrosse is special to me and it is deeply intertwined with my professional work of studying our brains. I spend most of my time with people who live their lives in their head and forget that it is connected to a body that is optimized to do a lot more than pipette solutions into test tubes and type on computers. I, and I'm sure most of my lacrosse friends, are met with furrowed brows when we say we're traveling across the country for an extended weekend to play a game we're obviously too old for with other manchildren. It is a healing game. Each time we play we engage all of our cognitive abilities and learn a little more about our own and each other's. Not only is lacrosse highly intellectual, it just might just be the most intellectual activity. It requires that one use and develop a breadth of cognitive functions, unlike regurgitating the answers to a contrived science exam or writing a boilerplate five paragraph essay for an english class. I want people who study brain sciences to have these kinds of experiences and not just read about them. Using your brain is different than reading studies on individual functions of the brain conducted in contrived laboratory environments. I'm a pretty strident atheist and I have difficulty connecting with most religious ideas but learning this game with my teammates has a spiritual component that I can understand. It sometimes feels like a group meditation more than a game. Unlike other team sports that have become somewhat formulaic, the Iroquois discovered a game that is so damn good at pushing our human cognitive abilities that coaches and players constantly find new edges and spaces in lacrosse that break traditions and redefine the game nearly a millennium after it was created.





I'm happy that I found lacrosse and that it led me to think about the ideas that got me interested in neuroscience. Lacrosse played an invaluable role in finding my self and I relish seeing other players using lacrosse to discover their own gifts and apply them to their lives in their own unique way. I don't understand this game the same way that the Iroquois, the zenmasters, and lots of people at my level do. This is the flavor that I see it in, and I think that's what it's all about.


Sunday, April 9, 2017

Turning Pictures Into Music


Some people use their spring break to go on vacation to New York or publish a science paper. I chose to spend mine making some bullshit on my computer and resurrecting my blog that I haven't posted on in months.

Anyways I’m back with some more nonsense. And it’s some unusually big-time nonsense. No neuroscience here, not really data science, you won't be inspired. There is literally no real purpose for this project except that we thought it would be cool.


To fully experience this post, you'll need to be able to have sound playing out of your device and be without fear of being judged by those around you.
Also I'm using this weird video format so you'll have to turn up the volume on the sketchy videos.

Image result for days since last nonsense



TLDR (too long, didn't read):

My friend and fellow graduate student, Vivek and I made a little algorithm that turns pictures into songs. For instance, I can give our algorithm a picture like this handsome logo for the Northwestern University Interdepartmental Neuroscience Program:




and it'll make a nice little song like this:


video



Yup. That's about it. Dozens of hours of work...



If you're still with me, this post will explain how we came up with this idea, show you some weird youtube videos, provide some examples of what our thing can do, then explain how it works. If all goes as planned, this should all amount to a substantial waste of time for both you and me. I am confident that we can achieve this goal.




Inspiration

A few weeks ago I was talking with Vivek about levels of meaning in music. When you listen to a song you can just enjoy the nice sounds but it’s so much cooler when you know something about what’s going on behind the curtain. If you know about music theory, or the artist who wrote the music, or the context that a song was written or performed in, it just makes the experience a little richer. For example, when you learn something like how Dark Side of the Moon synchronizes with The Wizard of Oz, or that Closing Time by Semisonic is not about a bar but about the singer's son’s birth - it just adds this whole new layer of meaning to a song that you already liked. And we thought that was an interesting concept.
I stumbled onto some really interesting youtube videos that demonstrate this idea of embedding meaning into a song in this totally new and fascinating way. In the following videos, the artists have put visual pictures into their music.

This group engineered their music so that when it is played through an oscilloscope it makes pictures that go along with the music (~3:20 is especially cool IMO):

Shrooms by Jerobeam Fenderson

So what you’re seeing in this video is literally the waveform of what is being played out of your speakers. Most songs would look like indiscernible noise if played through an oscilloscope, but they made this music with the purpose of the actual waveform being interpretable visually. I am kept perpetually amazed by the things that engineers can do. Details here: http://www.jerobeamfenderson.net/post/101351329308/how-it-works

Here’s another one where the artist inserted a picture of his face into the spectrogram of the music. And music has a very loose definition here. I don't find this 'song' to be particularly musical. But the important thing here is that he somehow engineered the song to play a series of tones at the specific amplitudes and frequencies in the middle of the song that made this picture (Skip to 5:30 for the face):

Equation by Aphex Twin


Also I just now remembered seeing this super weird piece of sheet music that someone left in the jazz band room in high school:



It's kind of the same idea of embedding visual aspects in music. Not super relevant, but whatever. It's neat and worth sharing. Don't yuck my yum.
(Info here: http://socks-studio.com/2012/05/19/the-unplayable-score-faeries-aire-and-death-waltz-john-stump/)

Then while I was working super hard in lab and not browsing reddit I found this meme where people were making midi files out of pictures. In other words, people will take a simple line image and turn the lines into notes to put into music software like GarageBand, where the horizontal spacing is the timing of the notes and the vertical spacing is the pitch of the notes.

Drawing pictures with music - Andrew Huang

To do this, the guy in this video printed out the image he wanted, manually traced it by hand into the music production software on his computer and adjusted the notes to make it sound nice. As people who understand a small amount about computers and music theory, Vivek and I thought this was cool but realized that we could do this ~algorithmically~ because what the world needs right now is an algorithm that saves the precious time of artists who make pictures of unicorns into music. So we chose to bear this great burden. You're welcome world.






Results

We did it.

And not only can we process an image into a song, we can tweak features of the image to make it sound a little nicer. For instance, this is what the NUIN logo sounds like if we don't edit it:


video


This sounds pretty bad. But you can improve bad things and make them better, like me at doing grad school. In the image above, we're playing so many notes at a time and there is no organization to the tonal quality so it sounds like shit. It's like pressing random adjacent chunks of keys on a piano erratically over time. But we can fix that.

Here's what the NUIN logo sounds like if we shift all of the notes to notes in D# major and play them with a more forgiving instrument:



video



And we can add chord changes so that for every 16 beats, we switch to a new key in pattern. Dare I say that this sounds almost kind of good?



video



Here are some Examples:


Lax in C# Major.
video





Doge Blues (I tried running this on a a picture that wasn't a line drawing. It wasn't great at getting the important features in the image but sounds pretty cool.)

video

Northwestern Wildcats Electronic Odyssey.
video
Powerhouse of the cell for bass
video



Poop in C Major for Trumpet Ensemble

           video

This:
video
And this:

video

Also this:
video



OK that's enough of that. 

Let me know if there are any images that you're dying to hear and I will consider making it for you! (or just use cool open source algorithm we made)



Methods
(All code was written in Python and can be downloaded from github at https://github.com/torbenator/prettysounds)

This project seemed complicated at first but we made it easier by splitting it into just a few discrete, not-too-hard parts:

1) Get the important parts out of an image and turn it into a matrix.
2) Manipulate the matrix to clean up the picture and make the corresponding notes sound nice.
3) Turn the matrix into a midi file that can be played by a music program.


1) It is easy to take an image and turn it into a 3d matrix of RGB values for each pixel using the sk-image library. But you can imagine that if we 'played' an image like this it would sound totally noisy and crazy because there would be values at every pixel that would correspond to notes. We chose to filter the image for any prominent edges that it has and only include these in the matrix. We did this using a Sobel filter. I don't know how it works, but it does work and it works pretty well. This outputs a 2d matrix where edge intensity is coded for by a value between 0 and 1. So faint edges play softer notes, like the second half of the brain in the NUIN logo and the second N in NUIN. Also it had a little trouble with the U for some reason. We also built some parameter tuning functions into our code so that we could threshold which edges to include. I had to play with this a lot to get the doge to work. That was time well spent.

2) The next challenge was to take a matrix and manipulate it such that it maintains the visual picture, but sounds a little bit better. First we built simple functions to resize and pad the x and y dimensions of the image using scipy.signal.resample. In this way we could choose how many beats we wanted the image to play for and the range of notes that would be used to play it. The more interesting methods we made could determine the starting note and the key of the song and these could be called repeatedly to make chord changes.  

3) Lastly, we found a library that writes notes to MIDI files - a file format that is pretty much a piece of sheet music for an electronic instrument (or soundfont) to play. We could just use this function as we iterated through the picture/music matrix that we made and dump it into a midi file with a function that we wrote. Then we just dropped the MIDI into GarageBand to play the 'song.' There's probably a fancier way to programmatically make videos of the MIDI being played but whatever. We tried to do this and we started to write some fancier algorithms to change more things about the picture and music but then we realized that we had spent way, way, too much time on this 'project' and it was time to stop.

All of our code is written in Python and can be found here:
https://github.com/torbenator/prettysounds


Feel free to use any of it but if you somehow find a way to make money with this then please give us some.

What did we learn?
Absolutely nothing. But I hope that you had fun and that someone you want to impress didn't see you listening to the song made by the poop emoji.


Bonus:
Since I posted this, a few people have pointed out other media that is related: This is a piece called Pictures at An Exhibition written by a Russian composer named Mussorgsky. He wrote this song after being inspired by an art exhibit. There are several movements in this work that each represent a work of art from the exhibit. (Thanks Sean McWeeny!)

The results of this algorithm have been likened to the work of La Monte Young, who has been called one of the greatest living composers today and who's work called into question the nature and definition of music. (Thanks Claire Chambers)

Friday, June 3, 2016

Studying Fake Busses With Science!

My last post was about my experience at this cool hackathon I went to. This post is about what I actually did there. 

All of the code that I used is written in Python and is on my github [Here]. I'd like to think it's pretty readable. You can also run all of the code and reproduce all of my figures in Binder without even having to download anything!


Look its that thing we made


For the hackathon we came up with the idea of using the city's public data to improve bus usage in San Diego. We noticed that certain bus routes were either overused, underused, or had surge times. During commute times in San Diego people actually wait in line for the bus. They get to the bus stop and some of the people don't get to even board the bus when it comes because there are too many people. This phenomenon is whack and presents lots of space for improvement. 


Whack bus line somewhere thats not in San Diego


The Plan

Here's what we set out to do:
* Build an app that the city to use to visualize and analyze bus usage data

* Identify overused, underused, and surging bus lines.
* Explore ways we could partner with rideshare companies to make busses great again (sorry)

We learned that the city of San Diego just started started installing infrared sensors on the doors to the bus to count how many people get on and off bus at each stop. This seemed like a great place to start... but we couldn't get our hands on that data over the weekend with 0 hours notice. Crap. 


Well in neuroscience if you want to study something that there is no data for, the best thing you can do is to build a simulation. And by simulation I mean a big 'ol math equation. This math equation will output realistic data by making certain assumptions about the underlying processes that generate it Then you can analyze the simulated data while you tweak the parameters to see what happens. 

Here's what I decided were the important features of one day in the life of a bus line:
1. Number of stops
2. Max capacity of the bus
3. How many passengers get on at each stop
4. The stops where the bus fills up
5. Number of surges in the bus line
6. When the surges happen


This seems like a lot of stuff to stuff into a math equation but it's actually not too bad if we assume bus demand is governed by A: some combination of random noise (riders randomly get on and off stops irrespective of the time) and B: oscillations (rider demand increases and decreases over time). I whipped up a simulation of this [here] but if that's not your thing then I guess you can just take my word for it.



Fake Busses

Using my algorithm I could create daily bus routes and change different things about them. Here are some examples: (blue line is the number of passengers on the bus at each stop, red dots are where the bus reaches capacity of 40)


If I make ridership completely random it looks like this:



And if I make ridership completely oscillatory it looks like this:



Both of those look kind of artificial but if I find the happy medium between noise and oscillations it looks a little more realistic.





The previous models all have the same number of times that they fill up. I can also change the overall demand without changing the timing of demand. 

High demand:



Low demand: 




Cool. Now we have a bunch of fake buses... Why did we do that again? 

We made these so that we can use them as templates for figuring out how to analyze them. If we only have the data, and we don't have access to the parameters that generated it, can we infer them? In other words, is there a way that I can give you numbers that characterize the noisiness, 'surginess', and demand of a bus line?



Demand

Demand is easiest so I'll talk about it first. If we want to know the total passengers who got on a bus in a given day, we just have to find the integral of the number of passengers on the route over time. There is a handy function for that called sum().

The more important feature of demand is how many times the bus filled up. If a bus line has lots of demand but doesn't fill up then thats perfect. I want to know when the bus is too full. I did this by set a threshold at the max capacity in for the bus and each time the amount of passengers on the bus reaches that number I record the times. It looks something like this:


These are two bus routes that both filled up 20 times but the way they filled up over time is totally different. The blue one filled up randomly, but the red one filled up in bursts. It's obvious to a human which is which but if we had 1000 bus lines that we didn't want to manually comb through, could we write an algorithm that would tell us if a bus line is noisy or surgey automatically?


Noise


Yes we can. Some neuroscientists were faced with a similar problem in analyzing spike times in neurons and figured out you can quantify 'burstiness' using the coefficient of variation of the inter-spike intervals (CV).

Check it out. If I take a purely surging bus line and iteratively add a little bit of noise, CV tracks this change pretty well - before it gets too noisy.



Surges

Cool. Now we can use CV automatically tell us how noisy and surgey busy bus lines are. But wouldn't it be helpful to know how many surges there are and when they happen? It would, dear reader, and we can do that pretty well using k-means clustering. If you look through my code you'll see that I tried the traditional approach of finding the optimal k using the elbow method but I couldn't get that to work so I made up my own thing. 

Here's an example of my method identifying the surge times in a bus line:


It looks like it works but I totally could have lied so here's a little more proof - here's how it does on bus lines with between 1 and 5 surges and 50% noise:
Dope


Summary, Problems, Caveats, and Future Directions

Overall I did a pretty good job. I was successfully able to quantify demand, noisiness, and surges in simulated MTS data. By doing this, I could take large amounts of ambiguous 1's and 0's and turned it into actionable information someone can use to understand and hopefully improve the system. 

However, there are some important things to acknowledge. First of all, my simulation is definitely still a little artificial. Bus surges are probably not totally sinusoidal and I don't think surges ever happen at 3:30 AM. This feature shouldn't really affect the way I analyzed the data but it makes the simulations seem a little less genuine. Moreover, as the real data comes in, it might turn out that noise/oscillations are not the best way we can describe demand. 

CV is also a pretty finicky measure. It is highly sensitive to both the length of time being analyzed and the total number of points where the busses fill up. I didn't think this would happen and I have been using this measure in real science code so I gotta go make sure it really makes sense.

There are much more sophisticated ways to characterize peaks in the presence of noise, (somebody might even publish an algorithm that does just this for neural oscillations someday). There are also several other machine learning algorithms one could use to analyze bus ridership that I didn't get to for the sake of time. For instance, longitudinal ridership could be clustered over weeks and months, and we could do outlier detection for unusual surges. 




Feel free to mess around the code and let me know if I made any mistakes!