Rust at Sentry

27 1

so hello my name is Armand I usually do a lots and lots of Python software development and now as of a couple of years ago I also started doing a lot more rest I do most of my work as an open source work so I have lots of open source libraries and as of recently also my paint job is thankfully open source related because we are no the sauce project perfect this is why you can find my Twitter handle my github projects and the slides of the talk will be available as well on the last URL perfect so um now my work currently century and century is a project where we show you when you have an application it crashes and the reason that's relevant to this talk is because we introduced a lot of rest code into this so to get an idea roughly what the system looks like this is the user interface there's a list of issues that you can see so any problem that you have in your server app or in your browser app or whatever you have can be sent through an agent to the system so there's a processing pipeline eventually will show up in UI and so that's roughly how it looks like this is what an event looks like if you have it and so there's a lot of a wealth of information needs to be processed in this whole thing the other important part is that we are an open source project so we we do have a business model behind it which is a software service business but the core of the application and also in fact what we host for users is an open source project and this is sort of the two parts of it one of them is century IO which is the software as a service hosted version but it's running the same code as we have on 44 the open source customers so there's actually no difference between the two and this is very important for our development process because it sets some limitations in regards to how we can pick and use technologies so let's see so the most important if you take a regular and soft as a service business one of the big advantages of it is that you have a full control of your technology stack so you can pick very exotic databases for instance because you can just license it for yourself you pay whatever the other the license fee is for using that database so you have a lot of possibility in particularly choosing technology for us it's a little bit harder because we are confined within the open source environment so there are no commercial databases we can use for instance and it also means that because customers use it on Prem we actually have to support multiple databases so we have to support Postgres we have to support my sequel so and this is the environment in which we started introducing rust as an additional language so our technology spec for the most part is Python so David Kramer the CEO of century me and Matt we were the first people sort of the did development work on the century know Python for a very long time we use Python we use Python before century during service development obviously and that was sort of the place where we started with and we're very conservative in putting more different technologies into this environment and the biggest sort of services that we are running is for our own installation not for a customer's installation is Postgres for the main database we use react for storing large non index blob data we use s3 for some larger data that doesn't need to be accessed that frequently we have Redis for our main queuing and up queuing our caching system and we also use memcache but right it is a big part of our caching story and then we have rabbit and queue and but we but you can see from this is that it's fairly conservative stacked there is while there is reoccurred we don't actually use anything fancy of it would be treated as a key value store so we are very careful with introducing these authorities and yet we picked up rust as as one part of it I'm going to skip this perfect um and one of the reasons why we picked rust is we have an adverse in to some degree to service-oriented architecture it's not that videos like it we do have some services that we are you seeing from other bits and pieces of our infrastructure but we prefer internal modular code over doing more HTTP calls internally in our sect it makes bugging much easier it makes deployments much easier it also helps us get the software running on the customers own installation easier and is easy to understand from customer support ticket what's what's going on in their own prime minister Latian the fewer moving parts you have and because we are confined to some degree with our internal modular but kind of monolithic code base to to do most of our work in Python so the traditional way in which you would extend Python outside of the Python language itself is typically as the extension module and none of us are particularly big friends of writing C and C++ code for this sort of stuff but we managed to actually introduce rust like normal people would do a C or C++ extension so this is the reason why we have rust in our code bases because we're actually using it as by extension modules so how do we do others relative in the whole thing it really is because we have different models and this is our annual setting references this is not the first areas only Hydra actually why is there a spinnaker space so we didn't we didn't just decide we're going to start writing rust extension modules for Python that sort of came later the reason for this is that I actually like forests and I started writing a project for century which is a command line client and this was our first experiment with getting rust into production use so the most important thing about rust as a programming language and this is why we why we started using it is so we built this command line client and rust as a programming language if you haven't been exposed to it at all it is compiled and because it's compiled but not necessarily because it's compiles but one of the benefits of it is that it brings in the final executable that it can distribute to customers it brings all the requirements that it has to run so all you have to do is compile the executable for a platform then a user can run it and it doesn't have to there is no requirement for the runtime environment of JavaScript or Python or anything like that on a system to be in a certain state so we we were very concerned about the idea of writing the command-line utilities in Python because we knew it's very hard to guarantee that everybody's Python environment is is in a good state distributing Python command line tools is is not easy so that's why we were looking at other solutions and rust was was my favorite from my personal experience was using it before and that's kind of what we settled with and then only later on we use some of the stuff we've wrote for the command line tool and actually reuse it on a server side so it was never really an intention to start with rust on the server so let's do this so this is um if you can see this this is a print from the dependencies of century CLI so this is the Mac version I don't know how you can see it but effectively you can see that the only dynamic link library is that it depends on our system provided so we can we know that these will be there for instance if there is a security issue in curl then above all patched is hopefully in time and we don't have to do it ourselves so that the idea is that we can we we distribute executables that only use the system libraries which we can depend on and everything else is statically compiled into it so for instance what you don't see here is as a we we have live get to as part of century CLI so that is an internal statically linked library and the the idea behind sandy CLI was to some degree that a user can just do this scary curl to bash installation or a use NPM or homebrew but it will be very easy for us to make a new release it's basically one new binary for each architecture go to github and sign in the script we can fetch it from so it's very easy to get it into the build process that was our goal and that's what we achieved with rust and so the question is what alternative would there be to rust so obviously you could achieve the same thing this go with C and people those already main competitors and one of the reasons why we really like to rust was the great i/o ecosystem so rust is very modern in in the sense of of the ecosystem because it comes with a package manager called cargo and people distribute crates which is reusable rust code on crate oil and it actually turns out that there was a lot of stuff we could already reuse to the community build and turns out the quality of packages for rust crates is surprisingly high so the community is actually doing a really good job in writing trades and so there was a lot of stuff we could use and that made the development of the command-line client much easier then then Erin would have been in other languages so I was looking at gold also and there were some things that I will go into later on where you can see like what we chose rust so we rejected basically the idea of writing it in Python or JavaScript because we would have to ship the runtime and it is kind of tricky so the the best part I hope this is the correct slide yeah so the best part about rust for us was the serialization support and that was kind of why we chose it you can very easily consume restful services that expose JSON data and you only have to do is mirror the the response data or the request data in rust structs and it will generate use your license for it and they are very handy to use they have very good error handing support so that the code becomes very readable and there is a lot of really good already existing infrastructure for building very nice command-line interfaces so that was the story for the command line and we had this and one of the things the command line client was doing is the goal was that the customer will be able to check before they are submitting stuff to the API that the data that they are submitting is actually in a good state so what you have to consider here is that if you find send send a JavaScript event from a browser so imagine you have a browser application where it's an error in it and then our agent will submit that error from the JavaScript application to our server and typically JavaScript code is minified which means it's very for us to actually give you useful information so what we have to do is we have to consume source maps and sourcing a huge JSON data blobs which we can resolve and then try to build better secretaries out of this um and it turns out that it's very easy to build source maps which are not in a good format so one of the tasks that this Andrew CLI tool has was before uploading the source maps to us we would verify locally that the source maps are actually in a good state because there's information in source maps which might only be able to be processed on the client side so for instance you might have source code references in the source maps which you never upload and they're like your local files so by executing it on your computer we can then look at those files and inline the source code is necessary so that is something that we had to do on the client we couldn't do in the server so we already had this really good source of library within sentry CLI and then we had performance problems with the source and parser on the server side and so this is this was the image from the from the blog post but you can see here is a CPU use and I think there was a similar one that looked almost the same because memory use so we actually managed to get like ten times saving so to ten percent of what we had before I think by switching a specific functionality on the server side that was dealing with my parsing from Python to rust and that was that was the part where we were like this is really interesting we can actually benefit from reusing some of this stuff we do on the client on the server so we basically built a Python extension module that reused some of the bits and pieces that we built a century life for the server side so source map was the first one the next one let me just skip this so the savings were for user is quite nice because it also meant that the time to from error to using it went from 20 seconds to less than at less than a second and it also meant that the back clock on our excuse was heavily reduced we could reduce the total number of workers and that was a performance game and the reason why we actually ended up with such a big performance game just the reason why we ended up with such a big performance gain was that the vast majority of time spent in dealing with source before in Python was actually memory allocations there were so many little bits and pieces in the chasing dump that when you parse them in Python there are a lot of individual objects being created there was a lot of allocations taking place so we actually I think I don't have the numbers by hand now but I think we spent about one third of the time of source not parsing actually just freeing up the memory from the parsing because we parse it we looked up four or five frames of information and then we had to dump the filtration thing again and we also allocated nearly a gigabyte of memory for something that could have been like less than 100 so that was that was interesting and then when we already went down this path of using rust for source maps which was a java script we then also started using rust for debug symbols so debug symbols are sort of the equivalent of source map for iOS or other native code you can ship an executable to Apple it will run on people's iOS phones you will run on iOS on Android but when a native crash is happening you get very little information out of it you basically are unable to resolve the function names on this crashing device because the function names are typically not there there's a bunch of formats that help you take the debug information stored in a separate file and then resolve it on the server side and this is what we started using rust for as well some of the information is still being processed by LLVM so we have a LLVM zebra zebra sauce pining for Python but part of the informations are actually now being consumed and in a rust library and the reason why we ended up using rust for this was not that it was performance much nicer through performance wise but it actually turns out the ecosystem is so strong that there are already better libraries for rust for dealing with debug information data then there's actually for C++ not because LLVM doesn't have that or is better that LLVM it has consumers for this data format but the API is very awkward to use in comparison to how easy it is to use rust fairies so we work much quicker building this in rust and where I would have been able to receive us and the last parts where we now introduce rust is ProGuard data pro-gard is sort of the equivalent of source maps for Java so if you have an Android application you can minify function names to figure out what the original function name was there's a huge text file you can parse we actually use a regular expression engine for this and it turns out to be sufficient but the we use the rest regular expression engine which is very quick and it's very nice to use and so there's also an area where we now have for us running so the part where it's a little bit hard to do what we are doing currently is actually getting an extension while you're running for Python and the reason for this is that the reason for this is that when you compile and an extension modules of Python there are two ways in which you can do it and the most common one is the one we didn't want to do the most common one is you compile your expansion module against lip item and lip - is the internal library that comes with pipe which exposes the API of the pipe interpreter so for starters this means that you can only target see Python which is the standard 500 asian if someone were to use pi pi it doesn't work because the emulation layer is quite slow and it doesn't really do what you want to do so if your goal is performance games then a C extension is not the way to go on on pi PI and but we're not using pi PI D for us the bigger reason why we wanted to avoid lip isin is that lip PI from is not stable which means that if a new version of python comes out you actually need to recompile for this version of python and not just only for this version of python you also have to compile it for this particular ABI version of python in particular we're still using python 2 to some degree and on python 2 and it actually is that the different versions of Python for larger and smaller unicode characters and they have different abis which means that if we want developer stuff a good develop and experience with our own libraries our goal is to distribute binary versions of these Rus dependencies that everybody can just install them on the computer without having to pip install both of them having to do a local cargo build so the goal is that came from tip install which is the Python package err the dependency and it will come in a second instead of having to compile and pull in all the dependencies so by avoiding lip - we can build three versions - for Linux one for OS 10 which is good enough for this set of systems we support and if we would link against the Python we would have to do about 24 for each version of the library which redistribute for a while we did this manually and now we are moving towards this project just go back it's called whatever email it's called snake and it's an extension for setup tools where you can build pipe in extension modules that link against rust and here you can see that there is a installation requirement which means that during the setup process you want to pull in our dependency which is called snake and then this dependency extends this code so that you can run as snake modules and a first part is where it will generate in the virtual module for Python or will show up so you can import all your rest code from there and a second part is on the file system where the sources lie so in this case you have a rest folder and all the rest or design there and then you can build titan wheels what is called is kinda like char files on on java where the extension modules already pre compiled in it so all you do is download the zip file and it gets loaded directly from there so our goal is to move everything to snake we haven't done it yet we still use our own hacks but hopefully in a month or two we'll have all of our stuff in using that reuse of expansion module and then other users can do it as well yeah so the the way we which we are doing this is we actually start out with writing a rest library usually so for instance a program parser our dwarf format parser are reusable rest libraries which we can use in the sensory CLI client as well as we can use on the server side then we build a separate module which exposes the rust module Sarathi ABI so we line can consume it from any programming languages can consume see function calls which our case we use it for Python only but it would also allow us to use this from no js' or or from a C++ library anything like this so we go to the most common denominator basically and then we use snake to automatically build Python bindings for for this and the Python binding is to come out a very very low level so we don't have to build higher level wrappers for it so here you can roughly see what the what an expose see ABI looks like in rust so I'm Molly can see here is you can mark structs as being representable in the same format see so it means that this looks the same as if you would write struct point in C so there's a there's a guarantee that the layout is the same and then we also mark the function there's no mangle which means that the function name will actually show up without any additional mangling that you have in rust to see buffers and it can be consumed from the paisa side and then on the Python side you would see that you import from this module which have which was created in the setup pipe so it's like example that underscore native in this case in that module there is a there's an one object called lip and the on that lip object they're all the functions which are exposed from rust and they're all destruct on there as well as you can allocate memories for it and so forth and it becomes a fairly straightforward to use obviously there are some things you still need to do manually for instance in rust you have the concept called return values that are sort of compound objects where in case everything goes well it's an okay value in case there's an error is an error value and we have we hand wrote a layer where if then error value comes through the C API we convert it into Python exception and then raise it as a Python exception stuff like this and we would like to automate some of that stuff later on so that we can actually build higher level bindings for Python automatically because there are definitely some issues with this for instance we we had a problem we're being produced a massive memory leak because there was the the destructor in Python called the function is slightly wrong way and then it failed executing the function and because constructors in pipe more automatically capturing down the the exception we didn't actually see that they the memory allocation failure that the memory deallocation didn't happen and then within like 15 seconds the service went down because of the ones where it was deployed they just ran out of memory and that is that is a problem that you have if you manually write a memory allocation code and we would like to automatic build bindings for us where we are not at this point yet generally the things that we love in rust are this is my favorite if you mark a struct in Python and in rust you can automatically at these that standard what's called trait implementations for it so for instance here you can say please derive me as your Eliza for this struct and there's a live we called Sarah which will take this information and build a serialize and deserialize avoid it and it doesn't just build a serialize and deserialize for it it builds ones that are is so good to use for all kinds of data formats that we never run into a limitation with this which is great and that I have never had this experience before so we can can consume arbitrary JSON web service API is with these structs basically so you can tell it to rename individual fields you can tell it to use default values from function calls and stuff like this and you just write this and you get a serialize and deserialize so for free you also get this debug feature up here I think it's on there so if you add debug you can pretty print all your structs recursively with automatic indentation and everything and it's just detroy to debug that is a massive time saver then and it's typesafe which is great and then our api layer is actually super pretty all we have to do is we have a function called list deploys which gives me all the deployments for an organization of the project and then we just call healthsketch which is our standardized way of doing get request with HTTP we give it a URL you can also see that we're using the formatting here so pass arc is a wrapper around an arbitrary string which will automatically escape slashes correctly and stuff like this and then the question mark which you can see over there means please do automatic error handling which means that if the call fails with failure it will convert the error into one that's specific to this library there's a very powerful feature in rust and one we like a lot is the idea that if have a library that fails with failure for instance you use Curl Curl can fill with a specific error we can convert this curl error into a standardized error of our API so consumer of our API of our like internal API will always get a very consistent error message so the errors there is a central piece of code which runs everywhere which normalizes error values it's very powerful and say it's a good concept so the idea is if you hit this question mark and there's an error the function will return with a converted version of this error and then convert em is it well so good convert is a is a function call which will convert the response value as he realized so if you go for it to be very clean it should be another slight what maybe not perfect we can I just skip this C perfect so and this is code we use which we started to enjoy a lot the biggest one is yeah so is this a set of three grades the biggest one is is error chain error chain set of automatic error conversion for you so if you have an error coming from curl or i/o or network or something like this you can set up an automatic error conversion for this error specifically to your grade this is basically how we do most of the error handling then there's if chain if chain is just a nice way to avoid a lot of nesting in rust and chain basically method if calls into a much nicer way and the last one is lazy static which is am just very useful utility created to set up reusable global structures which are initialized on first use so these are in almost all of our rest code in there it should actually just be built into language but they're not yet um this is Sarah the basic one just gives you services and deserialization support and then there's so many specific versions of it so for instance we use we do config file parsing bisetta we do HTTP call also education we I think the even parsing Apple specific plist files with said at this point so there are a lot of these really good libraries that you can use with it and I fully standby I think that it's the best civilization utilization library I've ever used in any programming language and then these are ones that we wrote they're surprisingly named French and nothing through this conference the these are libraries that can help you build really nice-looking user interfaces for the console so for instance we have one that just those colors and has different supports for erasing lines and repainting than indicative for the value call it as doing a progress bar is similar to the ones that you have in yarn and travel script and what we got out of this is just a much nicer experience in using our command line tools because they used to spit out ridiculous amounts of output which nobody wanted to look at and now they just make nice progress person if they encounter errors they will spit out some other stuff and make it look very nice and the last REM there is giving you the basic console input yes/no questions menus and stuff like this and then this is generally I think the best way right now for a lot of people to do HTTP requests rust curl you scroll off the nice thing of rust curl is that it actually uses curl on the system it's available and if not it will automatically compile it in and then Procope message L is a library that will help you configure open SSL specifically for all environments it can find this is great because it will find the open SSL Certificates on your system in the most bizarre locations so just putting that n means it works on Alpine it works on most Linux s works on OS 10 and just mix up messes that magically work which is great yeah and then we have XML reported the pipe in elementary library to rust so we do all x ml parsing with this at the moment and then the one thing that we don't like is compile times and that's sort of my personal pet peeve especially coming from Python I disliked that everything takes so long and part of the reason for this is unfortunately that the unitive compilation in rust is a great and not an individual files which means that you recompile more stuff than looks like would be necessary for the changes needed but this is the situation we're in so what we actually want is well actually I was expecting the site to be up there saying we want incremental recompilation which will eventually come but I think what we also want is to teach people to use rust code better is a guide of things not to do because rust is a language and those are the most important thing you can take away it's very different from everything else you've used before because it gives you a very strong sense of how to deal with memory ownership and it refuses to compile a lot of stuff where you think that should work but the rest is a very good reason for why it doesn't work there's some cases where it really doesn't any compile it even though it would be sound but that's something that I think the language will improve but rust makes you reconsider a lot of stuff that you took for that is actually good code and it will tell you no that is not good code so I think to reduce the frustration that first time rust programmers have there should be a guide of all the things not to do so that they don't keep running into those walls because nothing is more frustrating than spending an hour on doing something until someone tells you afterwards you can never do this because this is not sound so I think a guide would be great there and I think that's pretty much everything I have other than this which it would be great to have a higher level abstraction layer to expose rust abis to other consumers and if you have any questions I don't know if there's still time otherwise just talk to me after the presentation you