Tuesday, July 25, 2017

The Creator's Game: Lacrosse and The Brain

I've had a hard time putting my finger on exactly what makes me love lacrosse so much but I think I've figured it out. I love lacrosse because this game is all about the brain. From the outside it looks like manchildren running around hitting each other with sticks...and it is, but to play this game well requires a broad and deep level of cognitive abilities unlike just about every other sport or activity.


Here I'm going to argue that lacrosse is both intellectual and spiritual. This is something all experienced lacrosse players know to be true, but this idea isn't taken seriously anywhere else. The Iroquois, who invented this game nearly a millennium ago, were getting at something special about human thought. This game and the mental framework it cultivates is something that I use to think about how our brains work almost every day.


Last weekend I was at the Lake Tahoe Lacrosse Tournament with roughly over a thousand other people who have been playing this game since they were kids but just can't seem to put their sticks down. I'm not aware of any other sport that draws so many people to train and compete after the glory of high school and college is long gone. (But actually, is there any other sport that does this kind of stuff?) Tournaments like this, of which there are dozens, often have masters divisions (for people over 30), some have grandmasters (50+), and one even has a zenmasters division for 60 year olds+ (!!!). I'm convinced that there is something really special about this game and I'm starting to put together why that might be. 


My Squad



My squad in 40 years



I played at a competitive level in college and I think that I know something about how brains work, but unfortunately I am a mere sub-sub-sub-zenmaster at lacrosse and I certainly don't completely understand how our brains work. My experience playing lacrosse has facilitated thinking about my own mental processes and other peoples' in new, invaluable ways. This post is going to be half 'my gushing love letter to lacrosse' and half 'all brain scientists must play lacrosse if they want to understand how our brains work'. Team sports, and especially lacrosse, aren't taken seriously in academia and most scientists I know didn't really play team sports. I hope that maybe one person who reads this and becomes a teacher won't roll their eyes when student athletes have to miss class for sports - something I've experienced my share of.

Lacrosse is the Creator's game 


Last weekend a teammate told me to read the origin story of lacrosse, something I had never been exposed to in roughly 15 years of playing. When I read it, many of my feelings towards lacrosse fell into place. Lacrosse was invented by the Iroquois tribe somewhere around what is now upstate New York and sometime around 1100 AD. Some sources say it was invented to prepare young men for war but the way the Iroquois describe it characterizes lacrosse as something much more spiritual that connects them to their minds, nature, and each other. There's a beautiful origin story of lacrosse that you can read here (http://iroquoisnationals.org/the-iroquois/the-story-of-lacrosse/), or you can read my little oversimplification below. Also, what other sport has a mythology? How f**cking cool is that?


The first legendary game of lacrosse was a competition between the four-legged animals and the winged birds. The Bear, The Deer, and The Turtle were the captains of the four-legged animals. The Bear was strong and could overpower any opponent physically, The Deer was quick and could cover long distances, and The Turtle was strong and could withstand blows from any opponent and still advance the ball forward. The Owl, The 
Eagle, and The Hawk, were the captains of the winged birds. The Owl could keep track of the ball, while The Eagle and Hawk were swift and agile. The four-legged animals shunned The Mouse and The Squirrel because they were small but the birds recognized their unique value and added them to their roster. And beautiful, epic, long story short - while the four-legged animals were physically stronger, the birds won the first game of lacrosse with the help of their new teammates by appreciating each other's skills and working together.



Lacrosse was not designed for war and victory, but to recognize each individual's gifts and for a group to use their skills together to achieve a common goal. In Iroquois mythology The Creator made lacrosse so that he could watch all of his children enjoy the game. I can vouch that it is certainly fun to play and its easy to imagine a Creator proudly watching his children playing with literally the best toy ever made. The Creator also invented lacrosse as a 'medicine game' they could use to heal. I can also attest to the healing aspect to this game but it is hard to explain without experiencing it. Obviously the physical exercise involved in lacrosse is good for you but there is a substantial mental health component to this game. 


The four-legged animals represent the physical nature of the game but the winged animals, the more successful team, use a subtle, harmonizing collection of mental skills to succeed. Playing this game well requires one to think critically about his own abilities, develop the understanding that other players' skills are often different than his, and that we can create beautiful things by combining our unique abilities together. 


Other team sports become algorithmic. I'd argue that lacrosse is a much, much higher entropy game than any other team sport.  Clear strategies, with few degrees of freedom, emerge in basketball, where you pretty much have to be tall and football, where you pretty much have to be a freak athlete ...or a kicker. Soccer is a creative game, but lacrosse is like a parent class or superset of soccer. The fields are the same size and you can kick the ball in both, but lacrosse adds many more elements like tools and more contact which allow much more creativity. In lacrosse there are multitudes of viable strategies that different phenotypes of players can use - even Mouse and Squirrel can play.


In playing such an open ended, creative game, participants have to engage in an always updating process of self reflection as well as evaluation of other players' ideas and strategies. Psychology calls this idea of attributing mental states to others 'theory of mind.' I don't have data to prove this but thinking about how I and others think while solving flurries of novel physical and mental challenges makes me feel like I'm expanding my own cognitive toolbox.


That's all pretty abstract but here's an example. In lacrosse, an ironclad strategy has been to throw the ball with both hands on your stick ...obviously. Conversely, defensive players are taught to force attackers' hands off of their sticks so they can't pass or shoot. This is Mark Matthews. This friggin guy figured out how to do just about everything with only one hand on his stick - which totally changes how defenders have to face him. Also, by having only one hand on his stick he can score goals when his body is behind the goal by using the extra length the one-handed strategy affords him. But by doing this, he has to string his stick in an unorthodox way that limits other aspects of his game and changes how he fits into an offense. Both offensive and defensive players have to adapt their strategies to interact with a player like this. While Matthews found a pretty rare strategy, players constantly create and innovate on this game. Over the course of a game, players learn the habits of their opponents so they have to adapt and create if they want to win. The importance (or salience) of different features of the game are constantly shuffled in the players' cognitive landscape.  The mind of a good lacrosse player has to actively resonate with the game.






Brain Sciences and Lacrosse - Experience Matters


I missed a little bit of class in college because of lacrosse commitments, but I'd argue that I was using my brain more than my classmates with that time. I was exercising different cognitive skills that I don't normally get to use in school regurgitating the answers to contrived science exams or writing boilerplate five paragraph essays for english class.


In neuroscience, psychology, and cognitive science, we study brains by observing what they do and we try to reverse engineer how it might have worked. In lacrosse you're presented with rich, complex experiences of high-level brain function but in neuroscience we simplify these kinds of complicated mental phenomena and study their components. The difference is that in lacrosse, players have to exercise almost all of their cognitive abilities: sensory decoding and integration, motor coordination, language, tool/body schema interactions, emotional regulation, planning and prediction, awareness of time - I'm having trouble thinking of something our brains can do that isn't a critical component of lacrosse. In a neuroscience experiment, you pretty much have to choose part of just one winged-bird skill to study.
 Not so surprisingly, the foundational work describing how our brains can connect the location where a sound originates from to an eye movement towards that location, was all found by studying juvenile owl brains. Thanks Owl. 
Thanks Owl
see here for more if you have access: https://www.ncbi.nlm.nih.gov/pubmed/12015612

When I play lacrosse, I can feel each of these cognitive components churning together in my sensorium. I'm using my entire brain at once and learning about it as I engage with the game. My thoughts are on fire and it feels awesome. My neurons churn through ATP, carefully modulating circuits that keep my mental representation of everything on the field updated. I use my long-term memory knowledge of the game and modify it with novel aspects of the current context. I have to regulate my emotions towards blatantly incorrect, unjust, and stupid calls from the ref. I'm constantly trying to read my opponent's mind and predict what will happen next in the game, while he knows I'm doing this and is actively trying to trick me. Then I integrate all of this information and more into multitudes of small decisions and actions that impact the game in this giant, complex, recursive, loop that I'm a part of. It's not like I'm not some sort of brainiac, I'm just one of twenty people on the field who also all have brains doing the exact same thing but with each metallization flavored like a different kind of ice cream. A good lacrosse game is like a free-for-all at a Baskin Robbins. 


This presents a catch-22 for me where I can have these rich experiences of combining many mental processes playing lacrosse that I've learned to intuit but I want to understand mechanistically. And the only way we know how to study how our brains do such cool shit is to simplify and isolate these complex, interacting systems and study them in artificial laboratory environments. Rigorous, detailed science is necessary to understand the brain but we lose the 'lacrosse-ness' of our cognitive abilities by studying them in pristine experiments in labs with simple circuits of neurons and explain our ideas to each other using diagrams and powerpoint slides. 


Do you think that a blind person will ever be able to fully comprehend vision even if a scientifically complete explanation of it is available to them? Seriously. I really don't think so. There is something fundamental to the experience of a cognitive phenomenon compared to analyzing data about it. 
Taking this a step further, I'd argue that someone who spends time engaging and pushing their mental faculties at a high level in an evolving, dynamic, natural environment might have a better intuition for studying the system scientifically than someone who only studies it in a lab. I read somewhere that as you learn new words, your mental representations of the concepts they refer to become more detailed and complex. I believe that playing lacrosse might have a similar ability to expand conceptual space.


This is a strong assertion and I don't have evidence to support it besides my obviously biased experience. Focused, 
detailed research into neural circuits is necessary to understand the brain but there is something about playing lacrosse or otherwise, fully engaging with other people using their mental skills at their limits - that facilitates gaining deep insights into cognition. Each time I play, I learn a little more about each of these processes and whether they worked in the specific situation that I used them. I want other brain scientists to share these experiences and learn from their perspectives.



Lax vs. The Creator's Game


Like every mention of lacrosse, its important to note its place in contemporary culture as a caricature of elite, east coast, overprivileged, private school, white, male culture - which is just such a bizarre contrast to its origins. I'm conflicted because I'm not sure if I would have been exposed to neuroscience otherwise. A significant admissions bump from the lacrosse coach separated me from swaths of applicants and got me into an elite college with a great neuroscience program that I was only starting to get interested in. Lacrosse has pushed me, and continues to push me, to think in new ways. It has given me a rich bank of experiences I use to think about how our brains work every day at work







Lacrosse culture is strange and sometimes disappointing but I find myself constantly inspired by the incredible things people do with this game. It's definitely easier to be good at lacrosse if you start out as a Bear or a Deer but I see Squirrels outplay them all of the time by engaging their mind - learning, practicing, and understanding their own gifts.


 


I hope I get to see this kid out at an open tournament some day. I bet even the zenmasters might even have something to learn from him.



Lacrosse is Both Intellectual and Spiritual


I guess that the conclusion of this whole thing is that lacrosse is special to me and it is deeply intertwined with my professional work of studying our brains. I spend most of my time with people who live their lives in their head and forget that it is connected to a body that is optimized to do a lot more than pipette solutions into test tubes and press computer keys. I, and I'm sure most of my lacrosse friends, are met with furrowed brows when we say we're traveling across the country for an extended weekend to play a game that we're obviously too old for with other manchildren. They don't understand why this experience is so special. The Iroquois discovered a game that is so damn good at pushing our human cognitive abilities that coaches and players constantly find new spaces and edges in this game that break traditions and redefine the game nearly a millennium after it was created.


He does everything that's hard about using a lacrosse stick except it's also upside-down and backwards and there's this whole other team trying to put him on the ground.



Not only is lacrosse highly intellectual, it just might just be the most intellectual thing to do. Unlike any singular activity I can think of right now, it requires that one develop, flexibly modify, and rapidly deploy this huge breadth of cognitive functions. I want people who study brain sciences to have these kinds of experiences and not just read about them. Using your brain is different than reading about brains. I think I've beat this point into the ground but the last point I want to make is that this game is just as spiritual as it is cognitive. I'm a pretty strident atheist and I have difficulty connecting with most religious ideas but learning this game with my teammates has a spiritual component that I can understand. Not to say that this is something magical, lacrosse just sometimes feels like a group meditation more than a game.


I'm happy that I found lacrosse and that it led me to think about the ideas that got me interested in neuroscience. Lacrosse played an invaluable role in finding my self and I love seeing other players use lacrosse to discover their own abilities and apply them to their lives in a unique way. I don't understand this game the same way that the Iroquois, the zenmasters, and lots of people at my level do. This is the unique but shared perspective I bring to the game, and I think that's what lacrosse all about.





Thanks to Evan Xanthos, Derrick Kravitz, Alex Moffit, and everyone at Tahoe for the feedback and ideas!


Sunday, April 9, 2017

Turning Pictures Into Music


Some people use their spring break to go on vacation to New York or publish a science paper. I chose to spend mine making some bullshit on my computer and resurrecting my blog that I haven't posted on in months.

Anyways I’m back with some more nonsense. And it’s some unusually big-time nonsense. No neuroscience here, not really data science, you won't be inspired. There is literally no real purpose for this project except that we thought it would be cool.


To fully experience this post, you'll need to be able to have sound playing out of your device and be without fear of being judged by those around you.
Also I'm using this weird video format so you'll have to turn up the volume on the sketchy videos.

Image result for days since last nonsense



TLDR (too long, didn't read):

My friend and fellow graduate student, Vivek and I made a little algorithm that turns pictures into songs. For instance, I can give our algorithm a picture like this handsome logo for the Northwestern University Interdepartmental Neuroscience Program:




and it'll make a nice little song like this:


video



Yup. That's about it. Dozens of hours of work...



If you're still with me, this post will explain how we came up with this idea, show you some weird youtube videos, provide some examples of what our thing can do, then explain how it works. If all goes as planned, this should all amount to a substantial waste of time for both you and me. I am confident that we can achieve this goal.




Inspiration

A few weeks ago I was talking with Vivek about levels of meaning in music. When you listen to a song you can just enjoy the nice sounds but it’s so much cooler when you know something about what’s going on behind the curtain. If you know about music theory, or the artist who wrote the music, or the context that a song was written or performed in, it just makes the experience a little richer. For example, when you learn something like how Dark Side of the Moon synchronizes with The Wizard of Oz, or that Closing Time by Semisonic is not about a bar but about the singer's son’s birth - it just adds this whole new layer of meaning to a song that you already liked. And we thought that was an interesting concept.
I stumbled onto some really interesting youtube videos that demonstrate this idea of embedding meaning into a song in this totally new and fascinating way. In the following videos, the artists have put visual pictures into their music.

This group engineered their music so that when it is played through an oscilloscope it makes pictures that go along with the music (~3:20 is especially cool IMO):

Shrooms by Jerobeam Fenderson

So what you’re seeing in this video is literally the waveform of what is being played out of your speakers. Most songs would look like indiscernible noise if played through an oscilloscope, but they made this music with the purpose of the actual waveform being interpretable visually. I am kept perpetually amazed by the things that engineers can do. Details here: http://www.jerobeamfenderson.net/post/101351329308/how-it-works

Here’s another one where the artist inserted a picture of his face into the spectrogram of the music. And music has a very loose definition here. I don't find this 'song' to be particularly musical. But the important thing here is that he somehow engineered the song to play a series of tones at the specific amplitudes and frequencies in the middle of the song that made this picture (Skip to 5:30 for the face):

Equation by Aphex Twin


Also I just now remembered seeing this super weird piece of sheet music that someone left in the jazz band room in high school:



It's kind of the same idea of embedding visual aspects in music. Not super relevant, but whatever. It's neat and worth sharing. Don't yuck my yum.
(Info here: http://socks-studio.com/2012/05/19/the-unplayable-score-faeries-aire-and-death-waltz-john-stump/)

Then while I was working super hard in lab and not browsing reddit I found this meme where people were making midi files out of pictures. In other words, people will take a simple line image and turn the lines into notes to put into music software like GarageBand, where the horizontal spacing is the timing of the notes and the vertical spacing is the pitch of the notes.

Drawing pictures with music - Andrew Huang

To do this, the guy in this video printed out the image he wanted, manually traced it by hand into the music production software on his computer and adjusted the notes to make it sound nice. As people who understand a small amount about computers and music theory, Vivek and I thought this was cool but realized that we could do this ~algorithmically~ because what the world needs right now is an algorithm that saves the precious time of artists who make pictures of unicorns into music. So we chose to bear this great burden. You're welcome world.






Results

We did it.

And not only can we process an image into a song, we can tweak features of the image to make it sound a little nicer. For instance, this is what the NUIN logo sounds like if we don't edit it:


video


This sounds pretty bad. But you can improve bad things and make them better, like me at doing grad school. In the image above, we're playing so many notes at a time and there is no organization to the tonal quality so it sounds like shit. It's like pressing random adjacent chunks of keys on a piano erratically over time. But we can fix that.

Here's what the NUIN logo sounds like if we shift all of the notes to notes in D# major and play them with a more forgiving instrument:



video



And we can add chord changes so that for every 16 beats, we switch to a new key in pattern. Dare I say that this sounds almost kind of good?



video



Here are some Examples:


Lax in C# Major.
video





Doge Blues (I tried running this on a a picture that wasn't a line drawing. It wasn't great at getting the important features in the image but sounds pretty cool.)

video

Northwestern Wildcats Electronic Odyssey.
video
Powerhouse of the cell for bass
video



Poop in C Major for Trumpet Ensemble

           video

This:
video
And this:

video

Also this:
video



OK that's enough of that. 

Let me know if there are any images that you're dying to hear and I will consider making it for you! (or just use cool open source algorithm we made)



Methods
(All code was written in Python and can be downloaded from github at https://github.com/torbenator/prettysounds)

This project seemed complicated at first but we made it easier by splitting it into just a few discrete, not-too-hard parts:

1) Get the important parts out of an image and turn it into a matrix.
2) Manipulate the matrix to clean up the picture and make the corresponding notes sound nice.
3) Turn the matrix into a midi file that can be played by a music program.


1) It is easy to take an image and turn it into a 3d matrix of RGB values for each pixel using the sk-image library. But you can imagine that if we 'played' an image like this it would sound totally noisy and crazy because there would be values at every pixel that would correspond to notes. We chose to filter the image for any prominent edges that it has and only include these in the matrix. We did this using a Sobel filter. I don't know how it works, but it does work and it works pretty well. This outputs a 2d matrix where edge intensity is coded for by a value between 0 and 1. So faint edges play softer notes, like the second half of the brain in the NUIN logo and the second N in NUIN. Also it had a little trouble with the U for some reason. We also built some parameter tuning functions into our code so that we could threshold which edges to include. I had to play with this a lot to get the doge to work. That was time well spent.

2) The next challenge was to take a matrix and manipulate it such that it maintains the visual picture, but sounds a little bit better. First we built simple functions to resize and pad the x and y dimensions of the image using scipy.signal.resample. In this way we could choose how many beats we wanted the image to play for and the range of notes that would be used to play it. The more interesting methods we made could determine the starting note and the key of the song and these could be called repeatedly to make chord changes.  

3) Lastly, we found a library that writes notes to MIDI files - a file format that is pretty much a piece of sheet music for an electronic instrument (or soundfont) to play. We could just use this function as we iterated through the picture/music matrix that we made and dump it into a midi file with a function that we wrote. Then we just dropped the MIDI into GarageBand to play the 'song.' There's probably a fancier way to programmatically make videos of the MIDI being played but whatever. We tried to do this and we started to write some fancier algorithms to change more things about the picture and music but then we realized that we had spent way, way, too much time on this 'project' and it was time to stop.

All of our code is written in Python and can be found here:
https://github.com/torbenator/prettysounds


Feel free to use any of it but if you somehow find a way to make money with this then please give us some.

What did we learn?
Absolutely nothing. But I hope that you had fun and that someone you want to impress didn't see you listening to the song made by the poop emoji.


Bonus:
Since I posted this, a few people have pointed out other media that is related: This is a piece called Pictures at An Exhibition written by a Russian composer named Mussorgsky. He wrote this song after being inspired by an art exhibit. There are several movements in this work that each represent a work of art from the exhibit. (Thanks Sean McWeeny!)

The results of this algorithm have been likened to the work of La Monte Young, who has been called one of the greatest living composers today and who's work called into question the nature and definition of music. (Thanks Claire Chambers)