How Twilio Writes Documentation

0 0

my name is jared reyes and this is andrew baker and we're on the developer education team a twilio and developer education team handles documentation a twilio a year ago this team was just starting to get serious about documentation we had we had we if you would come to signal last year and you had asked us about documentation we would've said yeah we care about it we want to you know make it awesome and the title of today's talk is called how twilio rights documentation but we don't really think of ourselves as writing documentation everyone on our team is a developer so we build documentation and if you build documentation like you would build a product you can start to really prioritize data and the developer experience and see fun things happen so let's talk about what the last year has been like for us like I said a year ago we got serious about it and so the first thing we wanted to do before we built this new product was to first thing we wanted to do and we built this product was to look at the data and to figure out what kind of people were using our products so in the last year 800 thousand users have used to Leo documentation that documentation is made up of 1036 pages and those pages are maintained by any guesses for people obviously this isn't maintainable for people managing a thousand pages for 800,000 users is not something that we're going to be able to do for very long and so one of the main things we're going to be talking about today is how we handled building a platform for documentation that was scalable and would allow us to collaborate with developers in our community and with other to leon's but before we go explain exactly what we built and how we built it let's talk about what the docs used to be a year ago if you visited the docs this is the home page you would see it's probably very familiar to is this familiar to anyone recognise this yeah this was a this was a homepage for quite a long time actually even after we had built a developer education team and the reason it works and it exists is because this is what documentation is around the world since the invention of the internet documentation the kind of working theory was take the knowledge base take what you know about your product whether you're an engineer or you're on product team and write a book explain exactly how this technology works explain how to build something with it and tell the developer how to do it the problem is it's a lot of writing right here this is the quick start page more writing we sure we had some you know links to take you to the quick starts to give you some more information but it was a lot a lot of text in fact when we looked at how developers were interacting on our pages like this how to we saw very interesting behavior we asked developers in user testing to do a basic task we'd say what's your language of choice dhp what do you want to do I want to figure out how to send an SMS all right show me how you do that on twitter docs you know what the first thing they would do they would leave twilio they'd go to google they go to the search bar they'd be like send SMS with Willie oh in PHP and then they would land on some sort of page that would explain that to them but we knew that this was not an ideal experience and the second thing we saw from them is they land on the page and they would immediately scroll down the page and they would just kind of skim it and look for something valuable the only problem is developers don't have time to read that all of those pages you just saw we're just tons and tons of texts we were explaining our concepts we're basically breaking down our products for you but we weren't showing you a lot of code and the reason and developers are too busy to read is because we're scoping points we're building products we're doing our job right we're uploading our friends on Hacker News whatever it is we are we are busy so reading a book long worth of documentation is not something that fits into our workflow but it's the way that we've done it for a long time and it's for a good reason right when we started to teach anything at the beginning of mankind go way back just kidding we're not going to go that deep but we started to tell stories to teach people things right and stories still stick in the brain it's the best way to learn something and so people who are writing documentation they love to write they love to explain the stories they love to explain how they came up with this product and why you need it to make your business stronger but that turns out not to be what you are looking for when you're in middle of the flow when you're in middle of your workflow we need to reduce the friction so pages like this that's just all friction the when you're in middle of the workflow and unu it if you do end up on this page what do you do next what do you actually need you probably need either a solution to something you're trying to fix or you need code that's going to show you how to develop the thing you are trying to build from scratch this has neither of those and it doesn't have a ton of great avenues to get there as well this was true of all of our documentation and so we focused on the how to's because the how to's were the lowest performing part of our Docs and we're like well there's four of us let's let's experiment let's get crazy let's see what we can do because we know when developers come here they're going to go straight to the code how can we put code first so the first thing we wanted to do was experiment but we also had two other problems we had a usability problem because when developers would come to our site they would say why is your nav on your right hand side like Trisha airy and abs just shouldn't be on the right hand side and that shouldn't and stuff are not very good in product building so we wanted to get the data there and it turns out that like humans just read navigate better if content is on the left hand side of the page because we read from left to right top to bottom at least in Western countries so they pointed out a right hand nav and they also said you know the hierarchy is kind of confusing like what's the action on this index page I see a beautiful quick start page but like what do I do we hide some little red links in there but it's not very obvious what to do so we had some big problems to fix but the first one we wanted to fix was how to's the way we wanted to do that was wanted to create a green field of experimentation we wanted to give ourselves free rein and complete autonomy to build something that would put code first and would integrate with developers workflows a lot better and we had some good learnings from this early user testing we knew that we wanted to build solutions that were production ready and fully scalable because we heard from you look I'm trying to build a real business I'm not trying to do it you know I'm not necessarily just trying to make a hack or a robot do something cool I'm actually trying to build a scalable business so we said well let's make our tutorials our use case applications fully scalable and deployable and the idea there was well let's integrate with Roku let's let's do something really cool where people could just deploy instantly and we also heard that we need to have code be the story code needs to be the narrative not a whole bunch of Flowery prose explaining to me why this is going to make my business better so we took a pass at that this was the first iteration of what we were calling view Soros a view source had some code front center it also had the idea that this was an application that had multiple files so you could kind of browse the files and the idea was to highlight the code as if a buddy was sitting next to you explaining what he was building or she was building so like when I was learning the best way for me to get under the skin of an application was to sit next to the person who wrote it they would tell me exactly why they wrote it this way they're thinking when they built the architecture and this actually was super valuable right we go to hackathons and we learned that kind of thing we go to school and we get the similar experience of people helping us become better developers we wanted to kind of replicate that and we knew we were going to have to do something like annotated code but this past just wasn't doing it for developers they looked at it and they said look you're kind of pretending to be an IDE but you don't have any of the features of an ID you're not full screen you don't have a file explorer this is like just a half-assed attempt at doing what I actually need so we made another pass at it and this iteration was the complete opposite direction we're like sure ID eat full screen let's give it to him we had file explorer we had all the things that they had asked for but we had taken for granted that what they had had in the first version was complete control over the page and in this version you scroll down the page a little bit and we hijacked the scroll and we went full screen it was a horrible experience because it got in the middle of the flow gotten middle of the code work floral and added friction to them getting the information they needed they were coding they hopped over to twilio this thing took over their page there wow my god I need to go get a drink like this is horrible but we learned some special thing we learned some good things from this iteration as well we did some user testing and they were like what is this thing like I like the icons it's pretty but what does it do like well what do you think it does and like nothing well you're right except it's kind of navigation and it's kind of like our thoughts around and again it came down to us wanting to present something about our product and how we thought about building her product was just friction it was just getting the way of people coating the other thing they said is look you still have things that are important on the right-hand side I'm not going to read it on the text but if it's there and I need context about the code I'm going to go to it and you put it way over here not only that but the main navigation of the thing is those to arrows and that was on the right-hand side as well we're all right we can fix that and we finally made of this iteration which we launched in march this is twilio tutorials and we launched in march and people were happy it's an IDE we put the file explorer off on the side we got rid of those things that we had really held dear we stopped explaining why we built it we stopped explaining you know the architecture of the app in a sort of diagram and we took away the deploy buttons we got them out of the interface because it turns out developers weren't using them developers all of us are constantly building things that are unique to the problem that we have to solve so giving you a solution that we think is going to be perfect for you it's just not realistic people weren't using it so we freed up the space that we had taken up with those big buttons and we just gave you more code and in fact some very very very interesting information and data came from this this experiment we saw that developers who are completing more tutorials had less to do with the actual use case we were describing and much more to do with how the tutorial was written or how the tutorial was built I took a server notifications office graph because it's it it just Dwarfs everything else but but needless to say developers love to learn how to how to send SMS when an error happens on their server by far the most popular tutorial but this graph shows you that we have other tutorials and this is their completion rate and this shows you that there's some tutorials that outperform others and we were like well why do they outperform others it turns out there was a thirty percent higher completion rate if we reduce the first step code to less than twenty lines so we were just showing what we thought was the most important thing first right we were like okay here's the controller it like takes a number it provisions that number then it sends an SMS listens for an error code it generates a twill response and people were this was just too much context switch for them they were jumping into something that was already way too deep in the weeds and the only way that they knew that was because they skimmed the page they saw a lot of code and they bailed so we reduced that and there's a thirty percent higher completion rate so instead of showing the controller we'd show a config file which a package to JSON file and we see a lot higher success the other thing we saw is that there's a twelve percent increase in completion rate if we just reduced the copy for everything if we just put less words on there people were much more likely to get through it and it's because they were looking for the code and because they were in middle of coding that was what they needed there's another graph that I go to often which is this little this pi wheel and it's not actually incredibly valuable but I love looking at it because when we first wrote tutorials it was like 90 percent Ruby I was like look Kevin it's all Ruby like everything's Ruby means like Jared there's four tutorials in three of them or Ruby oh that's right but nowadays we have one hundred percent language coverage we have over 150 tutorials they're written in all of the major languages that Tullio supports and we can look at this graph now and we can see our audience we can see the makeup of the developers that are using our product but we also can see which languages are performing better and we can see that PHP author whoever's writing those is doing a hell of a good job and nowadays it's not just the four of us writing documentation it's also a team and Ecuador that we work with and every single developer in Ecuador and every single developer at home has a different specific language that they're good at so when we see that the node tutorials are outperforming anything else or in this case PHP we can go to that author and we can take the learnings that they have and we can implement them across the board and it's kind of thinking about documentation is a product that allowed us to even start thinking of what's the data that we care about you know if you build a product you really really care about time on page and completion rates but if you're just writing documentation that's not necessarily the frame of mind you're in so a note about tools we built view saurus using nodejs for the engine so now it's an NPM package than any of our developers can check out on the first day of work they can run a simple command line and they can have an entire tutorial generating Apple commands it gives them a template and they can start building on it and it gives them the interactive experience which is then rendered by backbone another note for testing and throughout these iterations getting feedback we used Optimizely to do a/b testing so we changed that Heroku button we made it a little bit skinnier and we change the type and we saw a lot more people clicked it if we didn't say deploy to Heroku we just said deploy we also saw that that was different depending on the language and framework because some people weren't thinking about deploy Mixpanel is how we track custom events and there's a really cool feature in Mixpanel where you can see funnels so we track custom events for every view that they did every step that they viewed on the tutorial and we were able to see which steps performed better than others in every single tutorial and this is how we learned that the first step was crucial to getting completion if you get them past the first step we could get them to the completion and that was all about reducing the friction on that first step and then lastly we use user testing for doing videos and interviews and we actually were able to custom select twilio developers to do the user testing it was invaluable we've we just used this every month if we can because it gives us a ton of data so we had we're onto something we knew we could put code first and we would have better documentation for developers we reduce the friction and we get them back to work so to talk about how we're building it oh well I'll just show you some more slides because they're there yay so we had turned the least used part of the site into one of the most active parts of the site's following these principles but to tell you about what we're doing next and how we're going to get there is Andrew thanks button thanks Jared so like Jared mentioned we had some early success picking a specific type of documentation going deep on it experimenting iterating and we saw those efforts bear some fruit so after we were done with that we were ready to take off a bigger chunk of work but you know a jared before we move on to that maybe we ought to just get checked with the audience how much they agree with what we've learned what do you think about a quick game of lecture prose or code okay so right now for the next few minutes I want you to pretend that you are a member of our team you are writing documentation for twilio I've got three quick questions for you and you're going to decide how to best document it first topic is web hooks webhooks our concept near and dear to any twilio developers heart and good for web api development in general but a hard concept to explain to people who are new to programming so if you are going to explain how to use a web hook an API webhook to someone who didn't know about them how many of you would try to do it face-to-face and maybe write out a piece of documentation a diagram and how many of you would try and start with code okay all right so maybe a little heavy on code but an interesting distribution so the correct answer was code and this is a old iteration of a twilio webhook guide you can see we really didn't have much above the fold when a developer landed there we tried to step them through like a sip a six-step diagram but there was no code to be found until he got pretty far down on the page so let's go to question two let's get a little bit more specific this time and let's talk about a specific API endpoint so if you're going to explain how to use a specific API endpoint to someone would you do lecture code or pros ok yeah all right people are yelling code fantastic so yes code was the answer and we got a little bit better on this one this is our REST API lookups page we have some code there above the fold but it's just the endpoint of the API which is not especially helpful because we normally try to steer developers towards using our helper libraries let's take more interesting detour now you are a 15th century Renaissance man and you've just devised a brand-new siege weapon for your sponsors the Medici family if you are going to explain how to get the Medici families engineers to create this weapon of destruction would you choose lecture code or pros great the correct answer was code so Leonardo da Vinci did not know how to program but we imagine if he did he would have gone with code and we're really just driving the point here because this is something that we're really holding up as a big learning from the past year but it's still difficult for us to keep it front of mine when we're working on twilio documentation all of us join this team because we like to write as much as we like to code and so when we're approaching a new documentation task it's really really easy to fall into the trap of writing this beautifully constructed narrative like getting your words super concise really really precise and accurate and the truth is that all that effort most of the time is wasted unless you are showing a code sample to the developer front center early so shifting gears a little bit let's talk again about what we're gonna do next in the docs so we knew that we wanted to build a new platform to run to leo's documentation something that would serve as a really strong foundation for giving us the tools we needed to make all these big changes we had in mind so the things like fixing the information hierarchy but also the kind of more experimental features that we serve our twinkles in RI so we had some features requirements that were important to us when we started on this we wanted to try and use open source tools if we could we wanted something that would allow us to do dynamic documentation so maybe not necessarily showing the same content to two different users when they visit a certain page and we also most importantly wanted something that was easy for contributors to our documentation to collaborate and an author twilio documentation they're just four of us but there are lots and lots and lots of people within twilio who need to create and most importantly update and maintain documentation and making our Docs a good experience for them was really really important so we face the same challenge that every new software project has of do we start from scratch empty git repo or do we try and use an existing open source framework and one of the cool things about our team like Jared said is that we're all deep in different programming areas so we could cast an especially wide net in this discovery phase look in lots of different programming communities and see what good tools were out there and one of them bubbled to the top and it's this framework called wagtail wagtail is a CMS framework which runs on top of django which is a python web framework and so if you're feeling a little weirded out at the idea of using a framework on top of a framework to build a website you are not alone we were all so weirded out but wagtail had a few interesting features which really drew us in it seemed to strike the right balance between features that come out of the box but also flexibility to customize so things like saving drafts of pages keeping previous revisions of pages tightly scope permissions for our users so that we didn't have to give everyone access to edit all of twilio stocks all the time these were things that wagtail gave us out of the box and they seemed to work pretty well but because wagtail is running on top of django a modern and kind of battle-tested web framework we still have the flexibility to go and write straight-up custom web application code whenever we needed to so we decided to dig into wagtail and we found this really killer feature that made the decision for us called stream fields so in wagtail stream fields is this concept by which you can combine structured and unstructured content and we use it primarily to put code samples and other content in the middle of our documentation so for us we knew that if we're thinking about building Doc's instead of authoring docs we wanted to treat our code samples as first-class citizens that means they're going to have their own model in the database their own you know their own Django model and we're going to wire up all sorts of additional functionality on top of them which would be really really hard to do if all those code samples lived in line inside our documentation and wagtail stream field makes it really really easy for us to treat those code samples as the first-class citizens that they are but easy for authors to drop it in the middle of a document when an author wants to add a code sample to an existing document all they have to do is click that tiny little plus button this nice little menu comes up and it says do you want to drop in a code sample beta feature stuff rich text editor and mark down the author can choose whatever they're looking for for that particular piece of content and then you as the end user just see it as all one cohesive page so we really really like this flexibility it seemed great for our authors it seemed great for us for maintaining our code samples and that's what really made the wagtail a cell for us so then we rolled up our sleeves and start building didn't MVP scoped out stories just like any other project but this isn't a DevOps talk so I'll sit all instead give you the highlights I mostly spend time on the DevOps reactions gifts website two of my personal favorites fixing the bug during the deploy and perfect exception catching / handling and we just kept on truckin kept on truckin and one of the big changes we made in this first iteration of the CMS was to split all of our code samples actually out of the database and keep them in a github repository so we still store them locally for caching purposes but we wanted the source of truth for twilio as code samples to live in a public git repo so you can go right now to twilio devid / AP i-- snippets and you will see every piece of code that powers every code sample widget on / docs now the main motivation for us to take this big step so early in our development was because we wanted to start testing those code samples so we thought that hosting them externally in a git repository would be easier for us to hook in the kind of testing tools we wanted to use to make sure that we Leo's code samples are always accurate and functional and right now we are doing some like testing on those code samples so for compiled languages we make sure that they compile and for interpreted languages we make sure that they run an exit successfully but one of the fun bonuses of hosting our code samples externally on github is that we can also solicit outside contribute contributions so you right now if you spot a bug in one of our code samples you can go fork the repo open a PR as soon as one of us plus ones and merges that PR your change is going to be live on / docs so it's a really really exciting way for all of you guys our developers who care the most about accuracy to make sure that our dogs are working for everybody so we went live with the CMS at the beginning of April we did a real kind of soft launch thing we're secretly started powering four pages of the documentation and we since since ramped it up to serve all / docs so I'm gonna let Jared talk you through some of the big changes that we've made yeah so if you were at the keynote this morning you may have seen some sneak previews of this but I wanted to just highlight again how learnings that we did when we were starting to get serious about documentation really affected how we thought about our new docs on the first thing you will notice we're not a mirror oh good call command F 1 boom um so uh the first thing you will notice this whole talk we're talking about code first is there's no code on this page but I wanted to show you something else uh this API reference is the new index page so this is this is what you will see now when you visit API reference and the first thing you might notice is that this is a new design and it's not only new design but it's a responsive design the reason that it's responsive is be not because we think that everything just needs to be responsive but because we know that developers have a workflow that happens on very different devices all the time not everyone has a 30 inch apple cinema display some people are using Doc's on the road some people are using them on the bart and other people are using them next to their beautiful ide like visual studio or clips or whatever you use sometimes having to switch out of that in view the code samples isn't extremely useful so if you come to our REST API and you're using an IDE you can now just pull up your code right next to the ID you're using see the parameters that you need to access and you have a responsive design to help you do that again it's just about getting out of the way of the developer getting it back to work there's one other thing that we do here to kind of match your ID and that's having a light and dark theme switcher so we heard em user testing from a lot of people that you know not only was it a context switch but it was often a light environments which where you would just go from working in a dark or light theme and you would get the exact opposite of whatever doxy were visiting so we're gonna make sure that was not a problem to degree that we not only switch the code samples but as you can tell we switch the reading environment so that the light environment matches a little bit more what your coding environment is the other thing that we added was search because we didn't want people leaving twilio anymore to go search on Google but there's also one benefit that we had if we control search which is that Google if you search send a message with image you're probably going to get a page description if you're lucky and a link to that page but if you search send message with coolio you will now get code first code is going to be the first thing you see you can switch languages and now you can get right back to work so you just found out the code that you were looking for without having to go all over the internet without having to go to stack overflow or Google and this is something that we can control and we can grow as more features come in from our developers and the last thing I'll point out is that on these nifty little home pages we replace that right nav which was kind of a dumping ground for all of our new products so after signal last year we had video and IP messaging those went on to the right nav to live there forever and instead of having it just be a navigation on my we really wanted to introduce the idea of filtering because we know that developers not only are working in different ideas and different languages but they're working in different products of Tulio so being able to give you the ability to switch between the product that you most use and the language that you most use gives you the ability to filter down the relevant documentation pretty quickly we're very very excited about building more filters based on your feedback so if you have any ideas please get in touch with us later today or just go and give us feedback on the website because this is something we are going to be working on like tomorrow so to tell you a little bit more about the future and to give you a demo of something where it's really excited about is Andrew who's going to take over right now thanks bud cool okay so from the beginning of this talk we've been talking about we think about documentation is something that should be built not something that should be written and when you're building something you build it out of lots of different pieces and what we are really excited to do later this year is exposed the pieces that make up our documentation through an API for you guys to use and consume so later this year we're going to start hosting documentation content behind twilio API the first endpoint that we're targeting is code samples so when you hit the code samples API you will see every code sample that appears anywhere on / docs will give you some tools to search for them filter them slice them by your particular language and our first kind of low-hanging fruit that we're really excited to tackle with this one is an embedding feature so we'll add a little button to the bottom of the code sample widget which says in bed which gives you the ability to take one of our code samples and drop it wherever you want if you're building another you know another business that uses twilio you can use that to help your customers figure out how to interact with us or if you're just writing a personal blog post that uses twilio and you want to have the those code samples and there it's going to be super easy to drop in and once you drop it in it's going to have the guarantee of us continually testing and updating these code samples so that we are keeping your content fresh for you in addition to the benefit of that same code sample being implemented across all of the languages that twilio officially supports but when I think about the docs API I think about a workshop that I ran last fall with a group called Jango girls so I live in Washington DC and we did an event where we spent a day with a whole bunch of women who were new to programming the new to web development and just taught them how to use how to use Django it's a fantastic group and a fantastic international organization and it would have been really really helpful during those exercises if there was some way for our attendees to get the information they needed to finish their twilio exercises in the simplest way possible because when you're new to programming and you go out and try read docs for something you don't know about you can get lost really really fast you can feel lost and overwhelmed and so I thought it would be really fun to just write a really quick slack bot that can help us out with that so I'm going to go ahead and start a new channel here will call it signal drop that in and I'm going to go ahead and invite doc spot to the channel invite them to join cool so using the docs API that we've got in its current state I've just written up a really simple slack bot if you haven't tried using the slack API yet and you use slack in your organization I highly encourage checking it out it's a beautiful API to use and its really really easy to get started so there's a little bit of boilerplate stuff in here but the magic happens in these few lines right here so all we're doing is hitting the API with a tag that the user is going to provide in their mention of Doc's pop and we're going to bring back a code sample for the language that they're looking for so I'm going to drop in my channel name up here I'm going to skip over to my terminal start up doc spot alright connecting to slack and let's go back over here and say hey Doc spot how would you do a lookup in Python and we've got the code sample dropped in right there and of course cool thing about all of our code samples is if someone was trying to go renegade at this Jango girls workshop and maybe do some rails stuff we can drop them in just the same so that that's something we're really really looking forward to sharing with you all later this year right on so um you know we care a lot about documentation we care a lot about getting you guys back to work and making sure that developers aren't stopped by the documentation they're seeing so if you want to jam with us if you want to talk to us about how you reduce friction or if you just want to jam on documentation please find us we're going to be on the floor and then tomorrow night we're both going to be a bash wearing crazy space outfits and playing silly games so please come say hi and let us know what you thought thank you very much guys you