The emoji that killed Chrome!!

0 0

[Music] yeah so I'm just going to share a little debugging story with you and I just it's it's not really maybe in the end the bug is not terribly complex but I still feel it contains all of the essence of what I find exciting about programming you know and that is the unexpected so one day a bunch of my co-workers came back from lunch and they discovered that everyone's coffee of Chrome had crashed and it would just keep crashing it was like no you could not open crash so obviously productivity plummeted to zero and and I thought this was wonderful already but but you know first principle of debugging you isolate what changed and you know so the question was what was there just like a chrome update and that killed everything but there was nothing online no evidence this was anywhere else but here so we're like well what else you know what else happened you know in our environment and the only other thing is we had bought this printer and then we got suspicious because we had named the printer an emoji we had named it printer emoji so we unplug the printer right and chrome stopped crashing and then we plugged it in again and boom all everybody's copy of Chrome on the entire network Pratt and so okay hey you know obviously we start debugging this what I have to say today says this is this is the greatest day of my life that's how excited that's how excited I got I get about issues like this so what was the first thing first approach for this so the first thing that I did is they go into gdb on on the system now this is actually only Mac OS Chrome this is a company with a lot of almost everyone was running Mac's so pretty good pretty dominant on Mac OS you would usually use ll DB but old dogs new tricks I'm sort of more comfortable with gdb so anyway we see here that like chrome is crashing in str ln the c standard library function get the length of a string and so that initially is really exciting because something terrible is probably happening right and that but we go into chrome as an open source although we'll get to more of that later but at the very least we can always we can always disassemble we can always look at the registers that's one of my favorite things to do the the registers after a program has crashed often tell such a fascinating story they tell us all sorts of places it's been but in this case actually it's a pretty boring register to dunk we can see that in particular like the stack is fine all these things are fine and RDI which is the first argument to str Ln is zero so we just have a null pointer dereference it's still exciting because we can reduce productivity in the office to zero anytime we want but it you know we can't we can't necessarily make chrome dance for us so the next step is well what's happening what is the printer doing and so we turn to an excellent tool called TCP dump and there actually is a amazing theme that just came out that's that should be available here I think maybe that will talk but will tell you all about TCP dump and how wonderful it is and and and how to use it so I won't tell it to say too much about that but I do want to mention paired with that is another tool that I really love which is TCP replay which allows you to replay packet dumps so you TCP dump spies on your network shows you what packets are going by your network interface a TCP replay lets you replay those packets right so this is a way for us to figure out what what traffic is actually causing this problem and so we apply an idea from property testing and if you don't already know what property testing is you don't already do it you should come up to me afterwards and talk to me about it because I think it's a really important idea and actually we have the author of an excellent C property testing library called theft here with us as wonderful libraries check it out but a really interesting idea from quick check and all these things is the shrinking of inputs so you have some we captured a bunch of traffic where the printer was spewing out its weaponized emoji and you know and then we we bisect it and we chop it down until TCP replay is just playing back you know just the packets that are causing this problem it turns out these two packets and then nicely we can package into a script you know Starbucks SH fast food that triggers the problem wherever we like you know it's very very convenient but so what are these packets so these are these are mdns packets which is the multicast dns and of course it's like notable way i immediately looked at the RFC oh actually I'll get back to that I would be look at the RFC and there's this sort of quote that I enjoyed that's you know the obvious elegant solution is that everything should be encoded as utf-8 and of course so we look at these packets this would be the PAX book so in mdns is the you know basically so DNS is this glue that that sort of essentially keeps the internet together mdns is this is this is a very similar protocol but just for local service discovery but something that's interesting about it is it sort of changes the kind of the inverts the relationship between servers and clients because basically unlike DNS where you're asking a server for something in mdns you're just constantly being bombarded by packets from every device on your network like this printer right but anyway so this printer is advertising itself you can see that the service name 1f v a8 downstairs printer downstairs that actually worked right but this TX TTY registers so this is some kind of Google Google Cloud Print thing that I don't understand that some new feature but that the type record here is obviously not encoded correctly and then so we look at this so something that I mean I noticed emojis have been a part of every successful talk here right so we can't deny their vast importance but you know so one of the right ways that emojis are representatives Unicode which has this large space and then you have different ways of how you actually represent Unicode on the wire so the most common way and what mdns is supposed to use is utf-8 which is actually this really elegant encoding except for the fact that there's lots of there's lots of sequences of bytes you can generate that aren't valid utf-8 right and in this case the printers is spewing out s 0 HD a 0 BD s 0 AP v6 a a so the sequence that's twice as long as what we would expect and it turns out that this is like a utf-8 encoding of a utf-16 encoding of the printer emoji right so some classic a classic kind of problem but wasn't wasn't caught in time anyway why does this cause chrome to crash so luckily of course there's a sort of an open-source version of chrome called chromium and we're able to reproduce the problem there you know and it turns out that it's like a very simple problem so we can see here that we're creating this this string and initializing it with these bytes that we received off the wire which is always always a tenuous thing I think you always want to be very careful when doing that although in this case it's not nearly as much fun as we had hoped the key thing here is we're asking for this NS utf-8 string encoding and if we actually look at the core foundation documentation this says well this returns near if record bite isn't valid utf-8 and so it isn't so that's where our null pointer dereference comes from so turns out the the fix that you know restores productivity in the office it's just this like almost a one-liner kind of thing and so yeah I said you know submitted that and unfortunately that because it was technically a security issue the the bug was sealed for a long time and it took some poking and patches before it kind of finally got fixed but you know and it's so all's well that ends well except that actually you know this printer is still lurking right it's still sending out its weaponized packets and I tried to contact HP to report this there was just no way to get in contact with anyone responsible and probably it's hard to fix anyway it's probably you know by upgrading that firmware is difficult but consider this they're they're public shaming in any case you know it was a really simple thing in the end but it just was lovely going through all these systems and having this such an unexpected huge result from such a tiny thing I think is really the the essence of you know they be unexpected which is is what's so exciting about programming so anyway I think if there's a moral to that story it's that everyone should go out and stuff every input field with emoji thank you [Applause]