IPython Is Not Just Python Anymore

0 0

hi so I'm min Reagan Kelly I work I work on ipython and I'm going to give my talk with ipython I recently graduated with in physics at from Berkeley I still work at Berkeley now funded by the sloan Foundation I get to work on my Python all the time which is really nice to be able to work on open source software as my job so just to get a feel for the room who here has used ipython okay who here has used the ipython notebook okay who here to bring in rubyists ooh here is used pry anybody okay and does anybody actually type Python at the terminal to use an interactive Python session okay and IRB anybody actually use IRB okay so I Python came into existence I guess 11 or 12 years ago when Fernando Perez was doing his PhD and doing some Python analysis and just got fed up with the interactive Python terminal and needed a better interactive shell and so first order I python is interactive Python so i can define functions i can run them i have output i execute them to get results let's come back as they arrived it has what you expect from near active experienced as tab completion one thing that the current architecture the to process architecture by python is so my interface in this case is a web browser we have multiple input modes the interface is not actually in the same process for the code runs one of the consequences of this is that I can this is a little bit of code that will seg fault but when it segfaults it just says oh it just restarted and then I'm actually fine but you'll notice the prompt came back to one because the process came back and it came back with the in a fresh state fault is too small yeah how's that better I python is a little bit more than Python we extend the syntax a little bit for special commands so we add a question mark that inspect respects I probably need a little bit bigger int respects the the function so i just did linspace ? which brings up information about the function linspace so this is the doc string as parameters and funk your signature so it's it's useful for exploring packages and kind of learning as you go because we the people who work on my Python world scientists we're doing research which as various people have said Einstein and Verner von Braun research means room we don't know what we're doing so we open a terminal we know kind of vaguely what we want to use but we don't necessarily know how to use it and things like ? and type of completion help us get through then we also have shell interactions so if you start a line with bang then it'll execute the shell commands so you don't have to type type out OS system or sub-process call and all these things just type LS which makes it a little bit more useful for an active work because you can just look in directory you can do assignments to the output of shell commands and then the second thing we add to I Python syntax is this thing called magics so if you start if you start a line with the % then it will run what's called an I Python magic which has this kind of a cell like I can have a shell like syntax for running just simple things it's a shorthand it just it transforms an expression into a corresponding Python call and time it if you've ever written profiling code with the time at module it's a little bit verbose we have to write a little block of code and time it lets you do that in a slightly more efficiently and it automatically picks based on how long it takes it automatically picks the the number of cycles it should take and this is a useful way of saying which which approach should I take sometimes you say I want to insert I want basically the the front side version of extent of a Python list there isn't the front side version of extend and there are a variety of ways to write that in Python and with time it you can just write them all and then run them and see which one's fastest and pick that and it's a it's a really nice easy workflow to get through just figuring out which thing you want to use I'm some more magics this is just a simple couple functions that result in dividing dividing by zero which is an error so we have a debug magic which gets me a PDB prompt so i can i can look at x i can walk up the stack it can look at why you can look at the back-trace i can leave the prompt and we have a lot we have quite a few magics so this is just a magic that lists what magics are available and you can see information about magics say I want to look at auto call i can look what it does is it will add the parentheses if you get to cook too comfortable in the shell like environment you start forgetting direct your parentheses on your Python functions if you turn on auto call it will add them for you but I'm clearly not in a terminal right now this the environment is this is a web browser so in a web browser I can do more than just a plain repple i can run code and i can do more than print statements so the first thing that we add is rich display so Matt plot lib is essentially the standard plotting library and Python and it is a wonderful thing and what we do is integrating matplotlib with a special in line back end that's actually part of ipython when you make a plot it just shows up in line in the in a notebook so the plot generates B&G sends that over the network and then we embed it right and right on the page they also have la tech I don't know if these are going to be familiar equations to people who aren't physicists but here's Maxwell's equations and this is what's called a cell magic so a cell magic with the 22 parentheses means take this block of text and pass it to some Python function to do something in this case it's just display la tech and to this rich display functionality libraries so simpy is a symbolic mathematics library they have this in it printing function which tells simp I to tell ipython how to display simp I objects so I just create this expression you know X to the 10-1 and then the actual rapper when sista display hook gets called on the expression i can actually get the la tech math 4x to the 10-1 and then i can faq that which gives me another expression and it gets la tech rather than the text plain text representation in another library that that is working with ipython is Vincent which is a a Python plotting library that wraps the Vega JavaScript library so I created some Python data and I I create a Vincent bar graph which when I display that takes a moment depending on my network connection and it creates this plot actually with a JavaScript plotting library Vega which is built on d3 so in this case I just actually published the numbers the the actual data to the front end and then the front end called the appropriate JavaScript to draw the graph in the page rather than drawing with matplotlib which draws a PNG in Python and then just sends the image and the result of this is that you should be able to get a more interactive plots and things and then we have simple literals for for a shorthand for embedding images doing simple HTML as output so bringing all these extensions and rich display things together I python has extensions were so you can define and an extension is just a python module that defines a function that can modify ipython and in this case our magic loads a magic or few magics for working with the language are in a separate process so with matplotlib I can create a pretty little data and plot it with matplotlib but I can also with our with this double percent are all of this is actually in the language are and this will be passed off to dr process using our pie too and i actually send the data x and y locally send that to our as as x and y and then i can actually do plots in our and from our i even get the the plots that were generated by our come back in Thai python and back into the notebook putting this notebook we also have it's not just code it's not just code input and output we also have markdown cells you just you just write mark down and we run mathjax on the page that means you can embed math we render mark down with the marked javascript library which means it supports a certain amount of github flavored markdown with triple back Tex and pipe tables and things like that and this so it's it's all rendered browser side so now I've been working in the notebook but what actually is a notebook so save and then this is showing some of our variable expansion in a shell command so I'm just going ahead in the subprocess and i created a python variable and then I'm expanding the Python variable in the in the shell command and you see the notebook is just doc is just Jason it's just a really simple JSON structure this is the actual file on the disk in cells the first cell is a heading cell so this is our h1 and it has no has content down cells and I have a code cell which has input and then it has outputs so it's a very simple structure of encapsulating the the input and the output and the text and everything in just in a simple document and one of the reasons we have this is a simple structure is that we now have a tool called the mb convert for converting notebooks to other formats so if I just call em be convert and then to HTML I can open that and then instead of the dynamic page I just created a static HTML version that I can share so now I've just got an HTML document that I can show you all my code my input my output I can and I can share this so you can you can show your code your actual results and all your documentation about it you can share that with someone and they don't even need to have ipython in order to be able to at least read it and we have a service called NB viewer which just runs this you point it you give it any URL to a notebook on the internet and it will download it right and B convert and show you the static HTML page so which has been useful for sharing sharing work sharing sharing notebooks little demos snippets of code scientific analysis another and be nav convert can convert in multiple formats it can do la tech output for pdfs it can do restructure text markdown mr. texmarq down to pure python script and rich HTML and it's all templated with jinjja and the language the various format conversion is all done with pan dock so one of the other things is I can export this notebook to a reveal j/s slideshow so this is actually the notebook that we're looking at right now as a slideshow so you can do presentations from it this the slideshow version doesn't have interactive code so i can't execute interrupt the subprocess so what actually is ipython under the hood how does this all work and essentially when we when we started coming up with this to process model which was mostly for dealing with the segfault problem and dealing with the remote execution right we want to have the interface somewhere and the code runs somewhere else and so we had to do is since I python is basically a ripple is we had to write repple as a message protocol so we have a very simple message protocol with Jason messages of message types may have contents and different keys that is kind of RPC ish that we send over zeromq to communicate with the the colonel where the code executes and it's just a bunch of little messages that are very straightforward and easy to understand we send the notebook talks to a web server with WebSockets and then the server talks two Colonels multiple kernels with 0 and Q it's actually quite straightforward simple little messages but you'll notice that in these messages so then I connects this is a Python code cell that actually execute JavaScript in the web browser so this just grabs my kernel object and then executes Python code as a string and I can print a right so like just to prove that I wasn't lying right so I just in Python excuse from Python code that sends JavaScript that executes JavaScript that executes Python and then show the result that actually happened in the Python side and we have a public document describing this this whole message protocol there's quite a few messages and you'll realize that there's nothing actually Python specific about a ripple over the wire right it's you're sending code is text and you're getting results that are mime type output so if there's nothing Python specific about it then why don't I input implement a colonel in maybe a different language so no there's nothing Python specific about it so this I Ruby started as a project for a weekend project by a guy who had basically never written Python code just looking at our mission spec and implementing a basic Ruby kernel and then just in the last couple weeks which has helped me out here a guy Daniel mendler has taken over and added a bunch of awesome stuff to the I Ruby Colonel so now this is all Ruby and if you look at the ipod config the ipod and configuration that enables a ruby colonel instead of a python colonel it's just this when the command to start a colonel you just say I Ruby colonel and that's all you need to do and then it passes a connection file so that it knows how to the ports and everything and when I set the port so that I wouldn't keep loading things on conflicting ports and if you look at the dr gamins that started this process it is indeed it's just I Ruby colonel and the connection file and I Ruby implements a lot of what I Python does so you know when you return something it comes in with an out prompt when you have something that's actually an in a file image it just displays it in line has a display method for displaying in line HTML you can display I Ruby math and la tech because again you're just sending the la tech to the browser and mathjax is rendering it and I ruby has a table primitive which ipython doesn't this is something that rivalry has that ipython doesn't have and has these table primitives that render as tables in ipython typically if you want tables you're going to be using a tool called pandas the pry is a wonderful prize kind of like I Python for for Ruby it's a better interactive shell that is focused on debugging which adds some syntax like I can look at the methods on file and shows me a little summary of methods class methods parents and with the are magic gem this lets me create a little image and I can display it Ruby vis does some JavaScript plotting and gruff simple graph flat right so I've got a simple I've implanted a kernel it's a it's it's a simple project at the Ruby Colonel it's fairly small and another colonel this is another language called Julia which is a language for doing numerical computing mathematical computing out of MIT and they have what was really the first real full colonel for the I PI for ipython in another language and it's a it's a cool language it has basic you know input and output it has output caching just like ipython so if you address out out to that actually gives you the value that's next to the prompt can create matrices display them in a reasonable summarized format because that's a big matrix can print to standard out in standard error i can call directly printf and see this is something that actually doesn't work in Python in Orion ipython we don't capture the output of printf that will go straight to the terminal but we will capture Python print statements and with Julia there's a library called PI call that lets you call Python from Julia so I just imported syfy that optimize which is a Python package from Julia and I just called it and then pipe lot is a Julia library that actually wraps the matplotlib plotting library this is this is a really great thing because nobody should have to re-implement matplotlib matplotlib is just in a class of its own for for doing prep publication quality plots and essentially no matter what language you work in you probably want to use map Watley to generate your figures and you want that to be as easy as possible from the language you're working in and in Julia thanks to the work of Stephen Johnson that's actually remarkably easy and so I just created a plot in matplotlib and Julia and then it came back through Julia as my figure and then it has all the same you know multimedia display I can create a class an HTML type that with a literal will display and there's a difference between display and then printing okay that's kind of a summary of what I Python does what I python is and how you can you know you can write your own kernel for foreclosure for there's a scalar colonel there's a node.js colonel but those are those are really toys at this point but it's it's for certainly for an expert in any language it is it is no great undertaking to write a kernel for the ipython notebook or a kernel for ipython button and then you get all this notebook sharing capability and everything for free just by implementing them at the protocol and that's about all I have to say thank you very much questions great thank think so first there are the slides I mean is the notebook available somewhere yeah it's a min arcade / rupee group i and then i posted actually the MV convert output so that's that's the repo and then this is the output oh that's odd I trust github com that's weird oh no it's just because I start typed of stupid ass yeah so then this is the envy convert output as static served on github pages so you can you can see the notebooks if you can't even if you can't run them and you can download them just from you know okay thanks and the second thing I know that lately you've been working a lot on bridging JavaScript and Python in the notebook so I'm wondering what's the road map here can we expect maybe some better more powerful integration of JavaScript visualization Indian so that that's what work at work we're working on right now in ipython is just about ready to merge is a basically a what we call a communicator that lets you create your JavaScript objects that have a special pipe to your own Python site objects and you can send messages of your own design back and forth and I can I could do a demo of an example of that that lets you do interactive plotting in with I have a demo with flot with that and then we'll have some basic primitives for simple html5 controls that you move a slider that actually changes values in Python yeah that's our main milestone for the release in this winter and it's that the first prototypes are open PRS just about ready to merge I just have to get home so I can review them questions more questions we're sorry so it's really cool that you support all these different languages there's two languages that kind of came to my mind when is all the examples I've seen so far seem to be mostly dynamic languages is it any more difficult at sin like Haskell or something statically typed so what you would need to do the most important thing about the two about Ruby and I julia is that they're not actually by the ipython team and that's an important sustainability issue for for maintenance that we we just provide this message specification and then the Julia guys wrote I Julia and various random volunteers have written I've worked on on I Ruby so basically if you can take a message and then do something with it in a way that makes sense in your language then then that then it will work so you could you may write the colonel in a different language that like a Python or something and then you actually when you handle an execute message you do some with some capi or something called the eval in some compiled language or something you could have a seeker Nell using scythes on you could you could write a kernel that's actually see where all the code you type is actually see and then you implement the colonel in Python and then with a tool called scythe on you compile it and then you execute it you could do it really in just about any language but and some languages may have run time through some restrictions that that make it not reasonable but it's certainly easy in any in any interpreted language it that's where it makes the most sense right but so the other question is JavaScript yeah a couple years ago I had this project where I made a raffle for coffee script and it was nearly as cool as I Python notebook but when the cool things about it is you didn't really need a web server of your own so you could just log in you can connect to different services use api's whatever fleet all your group calendar events and when one command or do anything yeah so actually we have to toy JavaScript kernels one that uses nodejs and the backend which has the whole multiprocessing structure but there's also one that I wrote where I just sort circuited the the colonel JavaScript object where instead of connecting web sockets and everything Colonel dot execute just does eval and so you can actually extend the the JavaScript to do not use the remote colonel at all you can have a local kernel in just plain JavaScript that's really cool it's it's a bit wonky how you have to get that how you have to make that work but but it does work and then with the notebook server the only thing you would use the notebook server for is saving the documents that kind of stuff and but not for code execution more questions um that is we were sprinting on that this before I flew here so that should be next week it's pretty much ready now yes no I know it's yeah it's been a partially done PR since May and it's it's about ready now so look for it in the next week probably anyone else well then thank you very much for this internship you you