Closing Keynote: Safe Systems Software and the Future of Computing

0 0

[Music] all right hello I'm Joe super happy to be here very lofty title so I hope that I don't disappoint so for those of you don't know me I most recently worked in Microsoft I was in charge of the languages and compilers and IDE and taking dotnet open source and and cross-platform and all those sorts of things I was actually there for 12 years a significant amount of the time was actually working on safe system software not to cut to the chase too much but to cut to the chase you probably haven't seen much about it because we actually didn't ship it and so that's actually part of the story so I left Microsoft in September to do a startup so I'm currently a CEO plumie in Seattle and we're cloud startup currently self mode but hoping to change that pretty soon so I'm gonna tell you a story and I'm hoping this story doesn't depress you too much the goal is to learn from the story I sort of went through this and emerged stronger than before and I'm hoping that I can share some of the key lessons and then talk a little bit about why I think some of what everybody in this room is doing and something some of what we were doing really tells us where the future of computing is going in particular for system software and that's a pretty broad topic pretty broad scope system software but I'll try to break that down a little bit as well and so first the story so myself my background when I was at Microsoft basically throughout the entire 2000s I was in charge of sort of threading concurrency primitives for for dotnet some of C++ so as soon as you know the Moore's law is over sort of panic set in you know we were working close to this Intel to come up with new technologies to solve that and software so that included pretty basic things like introducing thread pools and better ways of introducing concurrency better synchronization primitives pretty basic stuff that people take for granted these days some exotic things like software transactional memory so we actually add a version of the CLR with STM built into it turns out you know that membership but something like eight nine ten years left after that speculative la collision in processors and some of the hle instructions started showing up and maybe most lasting during that era was coming up with a test parallelism and better data parallelism primitives that today are sort of the underpinning of async and await as you see them in c-sharp and so back then we're working with you know the F sharp team who was really the one of the earliest pioneers of doing async await and trying to get those ideas into c-sharp and that eventually panned out but notably absent during this period was anything about safety there's a little bit of message passing so you could say that was absent as well but really the biggest thing was safety we tried with software transactional memory but it turns out that was the wrong solution to the wrong problem but the whole beauty of STM was you could slap an atomic block around a big honkin piece of code and not worry about what it does it turns out that doesn't work in practice because you end up with lock contention in places that are completely bizarre that you didn't expect and so it turns out taking a step back and thinking about safety in an entirely different way was a better path forward and it's not that STM is bad you know it's definitely got its place but the fundamental architectural principles of isolation and safety were far more important and we sort of missed the point back then and partially it was because we were we were dealing with constraints we're trying to wedge this into existing systems like you know windows the windows theme will recur so around the same time obviously you know this is a pretty abysmal situation it really hasn't gotten better over the years in fact it's gotten kind of worse you know every week you see the top of hacker news is like Oh some new buffer overrun and some new exploit some remote code execution in some critical piece of software that literally the world depends on its it is kind of ridiculous that we're in 2017 and the situation has not gotten much better thankfully you know we're all kind of jointly working on a solution to that but this was back then even back then we were looking at hey security and Windows is really terrible and it's a really bad PR thing and and this is back around the trustworthy computing initiatives if you remember that you know Bill Gates is one of his top priorities and so this is top of mind as well and so you add those together and we're kind of like hey we're screwed what do we do you know the windows had this antiquated threading model that didn't scale any of you that know comm single threaded apartments you know thread affinity there's so many kind of deeply rooted problems in the architecture that we were really running into a wall when multi-core arrived and of course the growing security threat you know what if it accelerated even further and this lack of safety at the very fundamental heart of the system became a critical and fundamental weakness what would we do we really didn't have an answer and furthermore it is more of Windows sort of trivia thing but I don't know there's like a whole way of the architecture of Windows and the dependency diagram and it looks like somebody smeared spaghetti across like a 20 foot wall and that really inhibits the ability to innovate and and you know if you don't have componentization you don't have separation of concerns you can't have separate authors for these components and so we looked at this and you know there are many promising alternatives for the more you know Kiko's arrows kyoto so these sort of more secure capability based operating systems had been around for a long time we had been working on safe at least typing memory safe languages in you know sort of the.net space the I don't know if people remember the Windows Vista experiment but you know cramming c-sharp into the heart of the operating system without anything more than just that it turns out it's not a good approach but at least we had some experience with what that looked like and so that's when we came up with the idea is this probably somewhere around 2006 2000 late 2006 to start an effort to build the new safe operating system from the ground up and so this was literally reused no code to start an architect at the right way don't focus on you know compatibility that's the big thing I don't know if folks realize how big a Microsoft compatibility was you know a breaking change was just forbidden completely even if the only way that it was allowed was it was a security fix but people really took compatibility very seriously and so we were saying hey no it's Greenfield we're gonna start from scratch and we'll see where it takes us and really this is an insurance policy we didn't go into thinking we're actually going to replace Windows we thought hey worst case we could take some of the ideas and apply them to Windows best case we don't actually need it because it's an insurance policy and so we focused on these core concepts isolation and so singularity we actually took some of the singularity ideas to start with this is a project that came out of Microsoft Research that use software isolation for safety so it was actually building on the fact that I had type in memory safety at its core to actually have lightweight isolation and this is what's necessary especially compared to Windows where the the process model is so terribly heavyweight I think you know Windows thread is roughly the cost of UNIX or Linux process and so you know Windows process is just enormous ly heavy weight and that discourages some of the message passing is sort of important architectures that we saw capability based security both in the application model and the programming model so we actually use the fact that we had typing and memory safety to actually embed capabilities into the programming model such that you an object reference really is an unfortunate in and so we actually used object references as essentially capabilities similarly there's no ambient Authority so there's no mutable statics no thread-local storage etc error model you know comparing two win32 with return codes and some exceptions but mostly return codes and you know c-sharp that had implicit any api can throw you know have no idea what it is in fact there's something called asynchronous exceptions so the garbage collector can come in and like suspend your thread and injecting exception between arbitrary instructions and so it's not even possible to know where all the errors can come from which I know that sounds ridiculous it is and so we said no no all errors are going to be explicit and we will bet on fail fast in a big way so not just fail fast for you know unhandled errors but fail fast for unexpected conditions like arithmetic overflow even out of memory and so what we ended up with was essentially free safety sort of working in harmony memory safety type safety and concurrency safety for all code with the exception of essentially some very low-level you know microkernel stuff that interface with the hardware but 99% of the system was type memory and concurrency safe code so that was sort of a system architecture and so we got to you know writing lots of code and it turns out you know we started using static analysis we started from c-sharp why don't we start from c-sharp well we're at Microsoft so that's one reason and so you know C sharps are a great language is very you know productive but it definitely has some problems like the fact that exceptions can permeate you know or emanate from any instruction there are other problems too it's garbage collected there's no way to do stack allocation there's no notion of lifetime safety there's no notion of concurrency safety and so we essentially took c-sharp and we started changing it and the goal was really - you know this is sort of a false bifurcation of the space but rough approximation you know you tend to have you know performance and safety and productivity and they're often net you know odds with one another so we're looking for the sweet spot where it was a safe and productive language while also being performant and what we did is we sort of went and looked you know I was familiar with a lot of the research in you know sort of advanced type systems I'm a big Haskell fan so we went and basically dug around you know I was worried you know cyclone and all these you know region based systems in the past and borrowing and effect types and we sort of took all those things and threw them into a blender along with c-sharp and said okay what what actually comes out and turns out what came out was something that was really a good blend of some of the productivity you haven't C sharp and the performance we needed and the safety we needed and it turns out you know this this was all happening 2007-2008 somewhere in that timeframe and it's funny because there's around the timeframe that you know some of the rust concepts started coming out and so I remember you know we would compare notes with some of the early rusts you know folks and some early go folks and folks in the research community and we ended up in a semi similar place and we even used ampersands for borrowing and then this happened and so you know we've had some wins and I'll talk about some of those wins later we were actually running this in production so we're running the Microsoft speech server on this you know so anytime you spoke in to your Windows Phone which I'm sure a lot of people did you would actually hit Midori on the back end we'd be serving up requests and that that was massively parallel because I use deep neural networks and and it was using all safe parallelism and it was it replaced the existing 1 million lines of code C++ system that had existed beforehand so better performance safety better productivity and it was great and it turns out we were trying to figure out ok so this experiment seems to be going well what are we gonna do next and so things like this started happening you know the government actually came out and said hey you shouldn't be using Internet Explorer because it's just not safe it turns out a lot of these bugs in Internet Explorer were used after free bugs basically what would happen is you'd be on a thread you'd call out to some user code run the JavaScript it would call back into the the the engine and the engine would reenter into some piece of code that wasn't expecting to be rendered it would release some critical resource and then you unwind the code and the memory would be freed and game over and this happened all the time and Internet Explorer and it's partially the way they wrote the code what is partially you know just a hard problem there's a huge C in C++ code base I should mention just as an aside even though Windows and a lot of its components are C++ it's really C with classes and so it's not very modern the C and so as we talk about some of the future improvements that's something to keep in mind so we went to Windows so we actually moved the project into the windows team and said ok how do we actually make this thing happen it turns out we bit off a lot there and we'll see the outcome wasn't wasn't awful but I think this starts to talk to some of the challenges that we have with safe systems programming in particular essentially what we found was there are a lot of guys who have been and gals who've been programming C for the last you know thirty years and not only were unwilling to accept to adopt some of the C++ things that would make their lives easier like smart pointers and everything and never mind some new fancy language that had this type and memory safety stuff like why do we need that right we seem to be doing well without all those fancy things see see previous slide with the use after free bus but that really is when you look at some of the hairiest code bases on the planet that really need this stuff it is human human challenges that are going to get in the way like has anybody talked to Lennis about whether he wants to replace parts of the Linux kernel with rust good luck and yet that's where we need to go so the outcome of this so I'll save you the the gory details of what it looked like in the end but essentially we ended up with lots of learning unfortunately not shared broadly which is shame on us we should have a bride in research papers the whole time if the project started in Microsoft of today it would have been open-source that's a sort of a big change since when I was when I was there or towards the end of my tenure there we did end up with some improvements to c-sharp so we basically said okay let's take this all the sets of things that we needed to change and dotnet and c-sharp and make you know bring those back to c-sharp so that it could be used for more System z things it turns out you know got some nice performance wins slices are coming out we'll just basically ability to take a subsection of an array without actually having to copy and allocate it's something that should have been there from day one but wasn't many of them still in the works we also launched a bunch of efforts to try to improve the state of affairs in C++ you probably hate me for that I kind of I have trouble sleeping with myself at night sometimes as well but some good things came out of it like you know better ways of writing C++ better tooling to enforce some of the the safety things you know really with modern C++ you can stick to a safer subset with smart pointers and the team tried to define you know sort of a borrowing like thing that would work with supercilious references and smart pointers some improvements in Windows not as much as we had envisioned but it turns out all this happened around the same time that you know the governor was come and saying don't use ie and so that that's actually what birthed this edge browser which was this Newburgh rebranded thing but a lot of the core tech actually made its way into that as well with some associated safety winds so you didn't have these use after free issues running rampant so I want to talk a little bit about some of the key lessons from from this overall experience like I said I hope not to be depressing so let's get on to the inspiration and innovation part so what what worked well surprisingly safety Everywhere worked I I thought that was the biggest bold bet that we were making I thought there was no chance it would work you know the idea of writing entire operating system with safe code just seemed crazy at the time everybody else told us we were crazy what was particularly interesting is that you could actually have data parallelism because we had data race free ways of introducing parallelism this is nothing new to folks you know using rust but at the time this was a pretty novel and their productivity wins as a result of that you know just not having to worry about all the classical you know buffer overruns and and standard things that just kill productivity and suck your time dry you know debugging just we're not present by construction where this phrase you know correct by construction which was a little bit of a lie because you know it's not really correct by construction safe by construction it had a nice ring to it so the magical combination of isolation security capabilities and message passing was actually really powerful actually you see this in Erlang and other you know even some programming models that you don't even need to you know change the language like octa for example it really you know most of the system was massively parallel because of this alone our web browser for example you would fire up a single page and it you know behind the scenes of spawning 32 processes and so a lot of those things were exposing latent parallelism were able to take advantage of it and so that that was actually really powerful for a long time we didn't even think we need data parallelism as a result until we started doing things like you know decoders encryption speech recognition I mean there are certainly areas where even with super cheap processes you still needed data parallelism to to really get the maximal scaling that because remember we're trying to compare ourselves against what you could get with C++ and so you know we were benchmarking continuously against the best workloads in C++ and so if somebody can introduce data races intentionally if somebody can roll their sleeves and you know really hyper optimize the thing that's a pretty high bar to jump over I'd say fail-fast was actually wonderful I think one of the funny stories I like to tell when we first ported the speech server to this model it was in C++ and he used H results and it turns out like all Taiwanese speech requests were failing with the same error code and the code just silently forgot to check the H result and so some garbled answer would come back every single time now I don't know what it says about the QA and validation that this wasn't caught before get into production but when there was poured into a fail fast model boom it just failed right away you found it it turns out you know we ran pretty aggressive stress programs you know run run one program and do as much as you can you know run multimedia stuff in the background with some web serving with some you know random just random things at compile going on in the background and see what fails and it turns out we get to the point where most of the failures were actually integer overflows because we failed fast on integer overflow it's kind of annoying but it actually pointed out like hey did you mean for this integer to overflow in some cases it's okay you know it's just a simple performance counter or something about in many cases that actually pointed out a bug in the system that for many other systems would have lain dormant for who knows you know years if not longer and the result was performance parody with Linux and Windows so we measured mostly server-side benchmarks like spec web and some internal benchmarks the speech recognition done from mark a lot of client-side benchmark so like you know some spider and octane and these sorts of things but you know our goal wasn't really to beat the performance of a lot of these systems although we ended up doing that largely because of the inherent concurrency in the process model but the fact that we even got close exceeded our expectations I mean we thought we would never get there we thought this is gonna be the thing that killed the project like we'd never get to performance parity and in the end people really didn't care about the performance that much it wasn't like a selling feature and I don't think even at 10 or 20 percent performance hit for some scenarios would have been catastrophic I think people would have accepted the benefits that you get from the safety and the productivity I would say the long-term productivity was great but I'll return to this and it was very interesting to see this morning that you know productivity is gonna be you know focus for for the next couple years because I think that's spot-on that's exactly what you need to focus on and when I say long term productivity I like to differentiate because the long term productivity was like you look over the course of the entire project you know we wrote five seven million lines of code something like that with the team of 100 people we're constantly rewriting pieces of the system as we went because we were learning as well the system was built on itself so it was fairly complicated but for a team of 100 you know we basically built you know editors web browsers compilers you know a shell file system networking stack basically everything that an opera state operating system has notepad you know everything from the bottom to the top was just you know 100 people and doing that in C and C++ like I just would not have happened it really wouldn't but here's the things that didn't work and I'm gonna spend a lot more time on these it's short term productivity and what I mean by that is like how quickly can I just write a piece of code and see what it does it turns out that was a constant struggle for us I'll talk a little bit about that in a moment people are generally unwilling to give up productivity now for a gain later on maybe maybe there's just a thing in like the United States or something everybody wants instant gratification I want to see it now but the reality is you know the business you know business happens that our breakneck pace nowadays you know startups are coming and going all the time it's you know really is innovation happens a lot faster and so anything that seems like it's an impediment to succeeding at that which frankly is the thing that my you know CEO or whatever is gonna reward me for like that's a difficult equation for a lot of people to work through people as I mentioned you know really convincing people that hey yeah you can write safe systems code safety is possible like you don't need to use C and C++ you know you can actually get a lot of benefits from using this thing it's just a lot of people are stuck in there in their ways and it's it's a new concept for a lot of people like if you've been reading research papers for last twenty years thirty years it's not a new concept but there's a lot of sort of bias that people have against even the word safety which I was really astonished by I would say another place that we really failed on was we really did try to boil the ocean and partially we didn't know like what were the pieces that we actually wanted to survive in the end and so what we ended up with was something that didn't have any incremental adoption you know we had some way of you know doing capi stuff and and you could potentially use your own build system if you wanted but really you had to buy into our whole thing and to end to get even the first little bit of benefit and that I think was ultimately the most critical failure so the type TEKsystems are wonderful like I said you know I've I've been a huge Haskell fan for a long time I should work to Simon paint Jones and a lot of the types of some stuff that we did in the dialect of c-sharp we caught hundreds of buds if not thousands I mean you can't even quantify because you catch a bug simply by the program not compiling so maybe it's even millions of bugs I don't know it's a large number of folks that we caught that never made them their way into the system people take that for granted there's not an easy way to quantify it unfortunately and even the ones that did become bugs because of defense-in-depth the fact that you've got this belt and suspenders type and memory safety thing underneath you it's it's harder to break you know basically jailbreak the whole system you can even if you find an exploit up here well turns out you're gonna turn around and try to do something over there and that's gonna you know fail fast on you so that defense-in-depth actually was very powerful as well and using the type system we actually used a lot of we use that to enable a bunch of pretty novel compiler optimizations you know it's interesting to see the Const a generic constant valuation conversations today so like one of the nice things is because we had immutability in the type system and all statics were guaranteed to be immutable we can actually evaluate all statics at compile time and so all these like giant tables that you have to build up that that cost you at runtime because you have to well in c-sharp at least it basically it does double check locking to make sure that things are initialized and and we could elide all of that it turns out that was leaner like 5% of the instructions throughout the entire system could be alighted simply because we had guaranteed frozen immutable objects which it turns out is much closer to sort of c++ and in terms of you know manual initialization in context birth and yet despite these wonderful things people didn't love it all the time I would say don't underestimate the syntactic you know a lot of people say office in text doesn't matter what matter is the is the concepts in the semantics and it's true but it's not into knots it's not the whole story the syntax is the user interface for the for the language and so if the syntax is you know scary for in whatever that's just gonna be yet another adoption blocker yet another barrier for people who could overcome and I would say some of the syntax we had especially around borrowing you know one of the interesting things I looked at C sharp is c-sharp ended up with this ultra simplistic model and similar to Java just because they wanted to save you know you having to put a star character right they don't differentiate between values and renders to values or pointers to objects and that actually turns out to me one of the critical problems that we tried to solve in c-sharp but that is actually where most of the syntactic challenges came from and it's not just in text like I said it's really the model as well but the syntax and people look at it say wow this is bizarre what does this mean rigor is not what people want when they are prototyping I talked about this earlier in fact the type system it's not just the rigor it's also you know oh I wrote this very simple code and I got this wonky error message I have no idea what it means the text system isn't mapping to my mental model this problem at all what do I do well a lot of that you can solve with better error messages so I know the mirror stuff that came in last year I guess it would have been made the borrow checker able to give out much more diagnoseable error message it was a fantastic thing but you know even if there's any places where the type system doesn't match the humans mental model it's gonna be constant friction and we we ran into this all the time and it led to a feeling people would say hey I feel like I'm just fighting the compiler I just want to get this thing work nationally see what it what it does and you know or and and they just spend all their time you know tracing they're tracking down every message in string fear oh what's going on rewriting the code weird ways contorting themselves and so that that was a problem I can't say that we ever really cracked that and even great IDs and tooling wasn't enough so I think the core problem was you know certain developers just are accustomed to not even thinking about memory management whatsoever it's like this completely opaque thing that just happens you know like Python or and c-sharp is in that category so you could argue hey we started with the wrong language but I feel like if you're gonna as soon as you try to appeal to a much broader audience this is going to be a fundamental problem and I don't have a great answer but really paying attention to this and really getting to the essence I think is critical and like I said earlier paying more upfront to save time later is a tough sell this one surprised me so reliability security even productivity didn't sell and it's nothing I didn't so like you know you would think that the chief executive of a company the size of Microsoft would care deeply about security even that that's sort of the business that they're in and I'm not saying they're they don't care but cranking up features being more competitive in the marketplace etcetera etcetera to the extent that security productivity reliability helps with that problem yes I think you get a lot of support when you try to go into a large organization you know you've got to sell the project right you got it or you can sort of go go rogue and and try to make it happen sort of as a grassroots thing and I've seen that work quite a bit but you're trying to do something as large as like hey let's get support for replacing components in Windows with rust or something like that I thought that there would be more support you know like obviously it's 2017 we should be writing systems code in a safe language if we can but that's not the prevailing opinion at least from my biased experiences and one of the challenges we rarely ran into was this notion of systems programming when we were trying to sell it to people they're like what is systems programming it's this vague term I don't a system is but you go to Wikipedia it's not I mean it a little bit it says systems programming games pretty software and software platforms which provide services to other software our performance constraint or both so I guess that narrows it down a little bit but especially today with lots and lots of distributed systems like a lot of code falls into this category and I actually think that's a good thing the only other thing I'd add is really correctness is paramount and unfortunately often at the expense of productivity which is a lengthy topic in and of itself and your mileage might vary but for us systems turned out to be a terrible branding strategy it required constant you know reiteration of the message people would say oh oh what I'm doing application development so you're telling me I can't use this no no you you can but it's just not 40 like it just set us up for a difficult conversation and it's true it's great for systems but it was you know I also I also think the world is sort of you can go is an interesting example where people are building systems but they're able to get by with sort of garbage collection and maybe less rigor and so you could argue like that's one language that works for both categories and so the more you sort of bifurcate the target audience for a language I feel like it's it makes it a tougher sell but in the end I think inertia is the biggest hurdle of all I think you know the most critical software on the planet has already been written by the definition otherwise it wouldn't be mission critical today and that's the software that's already picked a language in many cases it's gonna be see and convincing people to move away from that is is quite difficult so that's not to say we shouldn't try so what is the future well I think all of us have sort of seen a slice of the future I really do I think when I go talk with folks like deep in the bowels of Windows and talk to them about the problems that they're struggling with and you know I think Microsoft's getting better about seeing outside of the holes of that you know those buildings I think you know if you go talk to Linux contributors they probably have a better awareness of what's available in the ecosystem compared to you know windows engineers but I think neither categories really lived and breathed the experience of writing a very fundamental systems component that really demands the highest performance reliability security you name it they haven't lived through the experience of actually writing that code and rust or you know insert favorite contender for the new systems programming language of the future here and the question is how do we get them to see the value I think for me living through the experiments made me a believer but the question is how do you how do you get people to become believers when they haven't actually done it themselves I think one way is really lowering the barrier to entry and making it super easy to experiment and play around with this stuff but there are other things to this I borrowed this from Simon Paton Jones I think it's beautiful it basically tells the story of what happened our language clearly not rust but this is this is what you want to avoid you know the like hey you appeal to the geeks but not the practitioners and then over time you know people will just absolutely forget about it I would say our death was a little bit faster than than this but it happened almost overnight so how do we avoid that well the good news is their systems everywhere based on my definition earlier you know operating system kernels device drivers browsers distribute systems which alt all returned to databases you can go on and on and all of these are systems that really should be using a safe language I think there are more systems being built today than ever before I think the the trends to do more distributed programming means like you know kubernetes is clearly a system linka be it's clearly a system there are a lot of these systems components and the nice thing is people are writing lots of new code here as well and so certainly looking for areas where people are creating new things I think that's a very good recipe for for starting to sort of take over the entire space and make it the the default answer for people in that space unfortunately there is no silver bullet I wish there was but I think it's some comedy of doing big bold new things patience and hard work and being in the right place at the right time and those three things over the long run I think put you in a good position so this is a good example of big bold new work I think RIT Ops is a great shining example of really innovating and using rust to change the world in a in a big way but I think in addition to those things incrementality is absolutely essential and this is something I think I learned from my experience in the project you know we did not foster an incremental solution and every step of the way that was one of the major things that prevented people from betting on us you know you want to make it easy for folks to bet on rust or whatever component that you're building for a very targeted solution and then allow it to sort of grow from there right this whole like swallow the whole thing at once value proposition is a very tough sell in fact many large ecosystems just can't be replaced all at once like imagine you know Linux in canoe okay you can't just rewrite the whole world all at once it has to be an incremental path it's certainly easier to write new code and that's one of the traps that we fell into it's not only easier it's a lot more fun you can just go right lots of code and do great things and then you know especially today with such a vibrant open-source ecosystem that that approach works better today than ever before but really to get at the heart of the issue and you know get some of these mission critical systems on on the right path it's going to require getting messy and getting into the existing C code bases and really trying to turn them on their heads and eventually the disruption comes but you can hardly ever predict when that's going to be so one possibility this is just a thought exercise you know if I if I were to not me doing my startup this is probably something I'd be working on which is you know take canoe and Linux ecosystem and start rewriting components maintaining as much compatibility as possible so take something like this and really you know use a C ABI as sort of the outer shell and really start replacing the guts of these components one one at a time you do this over a long enough period of time next thing you know one of the most important pieces of software on the planet Linux becomes safe and then eventually you can get rid of the C API and life is beautiful I sort of liking this too a skittles analogy except it's like an inside-out skittle it's not the exterior isn't hard if the exterior is soft the inner core is hard I think also you know finding a wave to ride or making a new one is also a super successful strategy I think it's largely by accident I think but you know it's partially goes heritage as well but I see you know the correlation of go+ docker plus kubernetes like this whole ecosystem of you know cloud native components really picks go by default and largely I think you know it's it's part of the culture it's part of you know there's this you've got a cute logo but I actually go for a lot of the stuff at my new company and it's you know especially if you're trying to build a new eco open-source eco system you want to sort of go with the flow if you will and I think you know doctor and kubernetes really set the precedent there but I think finding that sort of an analogy like what is the next big system or big important technology that's really gonna take over the world and making sure that's written in rust I think that's a very compelling approach now who knows right like the go team as far as I know no didn't have much involvement and come in saying you know Solomon it's offered to write everything and go so you sort of have to be in the right place at the right timing there's a little bit of luck involved as well but that also leads me to this notion of distributed systems where you know new pieces are being written and swapped in and out and really that's that's an opportunity where people are writing new code they can pick up new technologies they're willing to do that there's an RPC boundary and so there's not this messy disgusting sea and drop stuff that you have to deal with you know to worry about memory management on both sides because if you have the CA bi is the shell you have to worry about Oh pointer came in and now it's technically unsafe but we pretend it's safe on the inside and if somebody screws up and the whole system's at risk at least here you have much stronger isolation boundaries I think this is one of the things with Midori that sort of made it work as well the entire system was sort of a distributed system in a sense all processes were sort of distributed software isolated processes and communicated using RPC and the world is sort of going this way so maybe maybe I'm wrong maybe Linux is the wrong thing to focus on maybe the right thing to focus on is the fact that you know there's this distributed operating system that's being created now out of heterogeneous polyglot components and maybe being a key player there is really the right long-term strategy that very well could be the case so returning to my original premise I think in the future all systems will be safe systems the question is you know how long until we get there and what's the path from here to there but I'm pretty convinced that the people in this room and the community around you guys are in the best position to answer that question and really make that change and so for that I'm super excited and I say let's go build a safer future [Applause]