How ZeroMQ Makes Science Possible

0 0

I'm really excited to be here at signal conf and I'm really excited to be presenting this information with you folks today before we get started I know everyone's in the middle row but if for whatever reason you want to have a version of this presentation on your phone or on your laptop you can access it by visiting JMP /s s c dash live and you'll be able to follow along and just stare at your laptop during the whole talk I did wear jeans and a dress and like wear my hair down for this so please do look up onstage on occasion because I tried really hard to get look good do that I want to start off by posing a question to everybody which is why is twilio such a popular company why are so many people here at this conference why are so many businesses building products on top of twilio is it because it's a great product is it because of their great developers is it because of the strong leadership the company has maybe very well might be those things but I think the real reason that twilio is such a popular company is much deeper and it touches on to the core of our humanity it reaches into our desire to communicate with other people to hear their stories to hear their voice to read their words we've all felt that anticipation when we get those three blinking dots on our I messages someone's typing us a message whether it's a friend or a loved one and that visceral and primal satisfaction you feel when you get a notification from your text messaging app or email is Realty or the satisfaction of hanging up after a great video chat with your mom those are all real feelings and that's what makes twilio popular and the most satisfying form of communication in all of mankind is the narrative the ability to share a story whether it's paintings on cave walls whether it's a long funny skit that you watch on YouTube stories are fulfilling and delightful things to us as people and there's one place where stories are most important and that's in the research lab in research laboratories and industry and universities scientists are making important breakthroughs and how we understand ourselves our society or planet our universe and those discoveries affect all of us but how are scientists sharing those insights how are they sharing their stories and their process and their narrative and how are we getting access to it as a general public that's affected by these I care about this process a lot I care a lot about how people share knowledge because withholding knowledge is a form of oppression and I get to see the power that knowledge has through the work I do at my company DISA I work with small businesses and nonprofits help them with their data science strategy and analytics and I discover amazing things that knowledge can unlock for businesses and consumers all across the world if you want to learn more about all the stuff I'm learning you can check out my website which is softly a dot rocks dot rocks is an actual top-level domain that you can buy and just kind of be very full of yourself so Sophia dot rocks is where you can find out all about me now but I don't know about that I want to talk about black holes black holes are regions of space where the gravitational pull is so strong that nothing can escape now 1.3 billion with Abby years ago well the first plants were emerging here on planet Earth far far away thankfully two black holes collided into each other and formed one supermassive black hole now on Monday September 14th 2015 at 251 a.m. pacific time the waves unleashed when those black holes collided into each other's gravitational waves were detected by the laser detection system set up by the laser and barometer gravitational observatory these waves are minut disturbances in space and time thankfully we have smart people working to detect things like this and they found them that's a big date in that specific time not only did they find them and they produce a press release associated with discovery like most scientists do and then it like lands on BuzzFeed and a crazy listicle somehow they did something really big they released an interactive notebook that contained the data that they used and the signal processing code that they use to analyze the data and this is an interactive notebook that anybody with a computer and a fair chunk of curiosity can explore analyze and run we're going to do that just now because we're curious people right so I'm going to go ahead and pull the notebook produced by the wonderful folks at lag 0 if it'll sorry awkward technical difficulties awesome so this is it a steno book that LIGO shared with the world it's an interactive document it contains some information about where you can get the data they use to make the actual discovery it contains interactive code cells that you can run in this particular case we're running Python to code that they've used to read and data to do some numerical analysis and you can read about what that analysis is to create plots and this is really important because you as a citizen and you as a curious person have access to the same exact process that's those scientist 'add now if you're like me you're probably wondering what's sound due to massive black holes colliding into each other at half the speed of light releasing energy equivalent to the mass of three sons make when they collide to which I say make sure you've got the volume turned up on your headphones because we're about to hear it right now is everyone ready hear that how about now yeah it's really it's one second lon it sounds like a bloop okay you heard it awesome people heard it no one watching the video stream heard it but no worries and that's it and this is a document that you all have access to and if you come to me after the talk i'll show you how to get access to it and how to load it let's step back to our talk because that was much funner so I want us to take a moment and appreciate the awesomeness that just happened because we were just able to not only analyze the data that was used by actual scientists play around with it if we so desired and if we were really really into it we could actually validate what they did and make sure that they actually did detect gravitational waves from two colliding black holes and that's a really big deal that's the changing the way people do science it's no longer having to buy expensive journals it's no longer having to be in an elite stuffy coffee bar in seventeenth-century england it's right there on your computer available to you and what makes all of this possible what gives you the ability to view an interactive document as executable code plots rendered in line narrative text in it and the answer is zero i'm q but before i talk about the technology and what 0 and Q is I want to talk about what it helps us accomplish specifically I want to talk about the Jupiter messaging protocol Jupiter is an open source project that works to develop interactive web applications for handling documents like the one I just showed you so documents that contain executable code on documents that contain narrative tags markdown lots of fun stuff the Jupiter messaging protocol essentially outlines how a front end or a client like the one I just showed you interacts with a back-end colonel that's responsible for executing code maintaining program state really quickly we're just going to go through the workflow of what happens when you run a code cell so we start off and we've got an executable code cell on a front-facing client I'm in the particular application that i showed you in that electron app the code cell was a react component that encased on the code mirror front-end tool so you've got an executable code cell just some fancy front on the on the front and that executable code cell has an event tied to it that sends a JSON payload over to set of communication protocols that hand it over to a kernel which is actually responsible for executing that code now you're probably wondering what the communication protocols are and if you haven't picked up on my Star Trek obsession between the way Greg introduced me and some of the slides that have in my talk you now know I'm obsessed with Star Trek Voyager all the way TNG sometimes original series never really got into it so let's dive deep into those communication protocols how do they work well we can really approach them in two ways which is what do they do for the user on the front end and how are they implemented on the back end so it's a front end what communication socket do they go through so as a user I want to execute a piece of code I have an execute request I go ahead and send that executes request through a specific socket called the shots locket the shell socket is responsible for reading my request analyzing it makes you make sure that it's the right request structured properly and then passing it all for execution now let's say that that particular code that I want to execute renders a plot like the plots that we saw in the interactive document that we were looking through in that particular case we have an IO publication socket that's responsible for publishing that content to the front end and either these are important messages almond useful things like plots you need those in science graphs and stuff and let's say that that particular code that you're trying to execute is requesting some kind of input from the user maybe it's the raw input function in Python or read function or whatever fancy new programming language is hot nowadays well it goes through the standard input socket which is you might suspect is used to get input from the user that's important now let's say that we have a user and they've written tons of like fancy code in their single code so let's say it's training a Beijing classification model let's say half of it taking a really long time to run and halfway through they realize oh there's a typo in one of my parameters or there's something wrong with this code till and I want to stop executing it oh and a halt everything well for super important messages like that you have the control oh my goodness oh my good oh everything's those disastrous I just I just ruin the movie for you guys I'm sorry for critical messages like that you have the control socket and you can give the control socket as the muscle in the family it Trump's president's overall the other socket so you can say control socket I need you to kill the colonel right now and it'll do it regardless of what the shell sockets working on well any of the other sockets are doing and then to tie all of these up you want to make sure that your system is alive and well write because that's important you have the heartbeat socket and the heartbeat socket is really simple it sends a single byte string message to your front end telling it that everything's okay we're good to go captain the Borg are not coming to attack us anytime soon hopefully not for a while so I want to Zone in on those arrows that were used to connect the front end to the communication socket what kind of data is actually being transported when I click play on a cell or when I press control enter to run it and then magically there's output well so JSON payload our favorite type of Sun next to buy son okay that was an awful joke I don't expect you to laugh and the JSON payload is really an easy way for us to track when a certain message was requested who requests today in one session it was running in what kind of messages it is way for us to track metadata about the message and that's all that's transported through these sockets it's JSON messages back and forth very simple robust system and the actual sockets are implemented as zeromq using zeromq sockets I pulled this right out of the Jupiter codebase if you're interested in checking out out it's all open source and on github once you essentially do is create an application that reads in messages through a socket parses them channels them to the appropriate relays runs them if necessary returns output to the user all based on 0 mq and I sort of lied in my top title and in my talk description and like at the beginning of my talk and I've just been lying to you consistently this whole time so I apologize we're actually not going to talk a lot about 0 I'm Q because I'm running out of time and because I want to focus on what you can do with this communication protocol because that's all it really is it's a way to talk between a front end and a client exchange executable code so how can you make use of this communication protocol in your applications what cool things can you build with this I want to talk about some awesome stuff that the open-source organization interact without the eye because it's all about we is doing we can find them on github as interact with the eye and they've been doing some really great work to make it easier to engage with the stupider messaging protocol and I want to showcase two of the tools that they've open source that you can use in all your applications if you're so inspired to really super charged them with interactivity the first is a simple NPM package that allows you to use this communication protocol that I just outlined this exchange of JSON payloads with specific specifications through different sockets via zeromq back end in the very traditional sense the same way that the original Jupiter engineers implemented it but here's the thing that protocol that I just described is just a way of communicating data it's an agreement of how things are represented and what to do when you get data of a certain representation and so you could communicate it over protocol like zeromq but you can also do it over a protocol like socket i/o if you are so inclined and you can use both of those things these are all open source packages just an NPM install away you can have it in your application you can start using at the documentation super clear and concise if you want to learn more about them you can come talk to me afterwards and I can like talk your ear off but you probably just want to grab a beer after this since it's the last talk of the day and then you have the data that you exchange and order checks acute code and to send information from a kernel to a client how do you actually represent that interactive notebook document those cells that are code cells or markdown cells well for that you have another awesome package that's also open source under the interact organization called commutable which allows you to interact through JavaScript with that data representation of that notebook so those code sells those markdown cells being able to move them being able to interact with their outputs being able added to their source super awesome fun stuff and you can build awesome things with this and at 6pm so you're probably not feeling super creative and you want to be inspired so what kind of awesome things can you build well the cool thing about this is that you can have one back end so one kernel running and you can have it connected to multiple clients you can have one kernel running on a high-performance machine somewhere and you can of a rebel connected to it from a terminal you could have a text editor connected to it you can have an interactive document on your active document editor like the interaction notebook I just so do connected to it they can all be connected to that same kernel running of the high-performance compute can run awesome code and you can run code in a context because here's the thing we no longer just develop api's or develop software in a singular sense we interact with lots of different api's at the same time in the context that you use your applications in is varied and you want to be able to build code that is embedded in the documentation or embedded in its use case and with an interactive document format like the one I just showed you you have the ability to tailor different dashboards Taylor different documentation sweets to the platforms and the code that you're writing at your company if you want to get more inspired about that come talk to me afterwards I hope I'm keeping on the edge of your seats good and lastly you have the ability to build computing driven narratives when everyone in the world has a laptop on their lap you have the ability to unlock amazing things with code and indeed the world is quickly moving that way we see a lot of things like data journalism we see a lot of citizen science we see a lot of people running machine learning algorithms on their laptops to solve challenges for money or prestige that's really changing the way that people share their stories everything's compute driven now and when you tell these stories you really need to have a way for people to not only share that narrative that markdown sell you need to have an ability for them to share that code that code sell but what's the big deal with all of this this ability to execute code interactively on a front-end separate from the back end that's running on well you get to liberate the front end why do we have to write code in a plain old text editor why do we have to interact with code in a ripple why can't we build our own dynamic ways of interacting with our software and we're sharing our software with others and you get to free the narrative why should a narrative be constrained to a static research paper in a journal that no one's ever going to read when it can be Fred to people step laptops and desktops an interactive format in an engaging way and finally you can open the code I'm really excited about open source software if I like talk loaded with open source tools and all that fun stuff hasn't proven it open source is really important because it gives you the opportunity to share your creativity with the world and to inspire other developers to be just as creative and even more creative than you so when you open source your code you're not just open sourcing your idea you're open sourcing the potential of millions of others ideas so that's really important a major keys from this talk because I'm heavily inspired by DJ Khaled as well narratives are important stories are important there is important to us now as they were when we were painting pictures of donkeys and placing our hands on cables you need to liberate the context your program runs in from the actual program execution need to be sure to share and develop in the open and make sure that your one idea inspires millions of others and most importantly with all of the things I just showed you with the tools I just showed you are available I want you to go forth and help people share their story I have three minutes left which is awesome because I should give you three minutes extra to grab a beer or two talk to me hopefully thanks for coming to my talk I hope it was everything Greg promised it would be I think he overhyped it i'm captain sophia everywhere on the internet so if you want to find me on github or Twitter or Friendster or Napster or any updated technology i am captain sophia on it email me your feedback safiya at sofia rocks I want to make this talk better for potential future presentations if you have a good joke that i missed telling please do so come up to the front if you want to see the entire lego notebook and if you want to like here black holes colliding one more time on it'll get beer you