From Dev to DevOps

go I'm here good morning to everyone that pain is only two companies on the house it's really amazing to see all these people here and i'm here to talk to you about my experience of going from the harbor to dev ops guy so it was a really nice and bumpy ride and I mean it's a it's a really nice way to for you as a developer to get to know what's happening in what it's what is it that we do as developers that we do not understand that's causing pain to people on the other side which is like what I mostly saw when I was doing my job so going back a couple years ago when i was just about birth and if you came to me and you asked what i think about the deployment process that we have like if you ask me like why what's your perspective on how your code the code the stuff that you're producing your artifacts that you're building are going to staging go into production and getting to you the customers and yes sir you would get from me would be like I don't know who cares it's not my business right on the rubber so why should a devolver worry about the county code is getting two productions so I have no idea how the coder was producing in the suburbs building was making its way to the environment where customer travel using i would really i would write the test i would run my test suite i would like maybe try my beer like just run the browser and see what I wasn't looking and if you're looking good there are no like really like glaring box going on but I wouldn't really worry about it that much because boom sonic built it and also have the party pack also have something that I can throw out like over the wall to the people that are on the other side jobs people then I'm like it's not my job it's their job to do this kind of stuff but you know the outseam on the other side they care whenever there was a new version of the software to publish their respective would be like like no like no why is there something new why do you want to upgrade this library why do you want to upgrade the operating system like what why are you doing all of this change and do your taxes out i will be like yeah you go to my computer it works like a mirror so why is this a problem why whenever i'm trying to upgrade something on all the machines that we have like a lot of the the doses of machines that we have the obviously would be like oh my god why are you doing this did you test it out is it safe this is secure like I never worried about it but because i was running away machine it's working so it's okay through me because i'm not at all i don't have to answer the phone when something breaks so for them they had a really different perspective and what what we bought like SFL person you would get this time responding everyone the team asked people do not like change right they do not like change I change is back because whenever something changes something else breaks right whenever there's like a lot of change going on and you don't not me like process to validate everything that's happening and you have a lot of people to watch the environment to work on everything that's going on in there and you have this team of the thumpers that is just like oh let's keep on changing that let's not worry about what's happening on the other side like it's not a good position to be to be it but for developers they're just throwing cold like over the wall like it's not a problem because they're not involved in the rolling in from the golfers perspective that's what we have like I goes perfect right the test suite is green all these reduced integration testing greening like it's not a problem and then you guys with the deployment and it goes out I mean like it's really really good and when you guys are boy this is what's happened like the this is what the ops team actually gets when they go get your production like it's not pretty and it's a thing there's a button and then there would be the answer that they got from the developer is that like you know it's a small fix its assembly to change this small little line of gold and then I'm going to we're going to push this your production we don't want to wait for the test we to run because that everyone is now we just have to like push it as the way this I just raise your hand I don't want to be alone gear so like it's a really small change everybody's now let's do a small change but it's not on the test week let's publish let's break everywhere again and this is a cycle that continues to happen whenever you have the separation between the decoder building code and the people that are pushing this code to the actual environment and this was the main issue that we have like as developers we instead of working with the ops team our goal was how can we go over the off steam and or co2 production so we're not working together who are fighting against each other just because from our perspective they were being mean because they did not want the change and from their perspective we were awful people because we're bringing stuff all the time and they get paid every single night this is not good this is really not good that it's not like out the team should work and how stuff to happen so what you have this everyone is trying to eat each other like you just like you just have teams are fighting each other all the time because they're coming from different perspectives and the things that they feel are different like we have no idea the amount of weight we were like causing to the ops team and young she was just like guys just do not send you stuff let's do like one least every month right let's at least have a single mode where nothing is going to break where we're going to be a warm introduction but there's a long flight know like amongst you wanna like have a story ready I want you to but I am why don't we have continuous delivery what's preventing is like isn't that always net what every single company should be doing so until oh no no not this guy that's one so boss comes and it is like um we don't have asked people to just go to production and then I was like what does it mean and like well you will have to wait to get this project that you have just fished to actual supper all right oh and what do we have to delete you could be coming up seemingly yea or nah I'm all about her right eye I can do this like I have the power I can write the code I can do anything I want the computers because I'm one of the developers and I see would be like yeah yeah really like really and when we start actually doing the work for like okay this is not really what we were thinking hey this is not the pelvis this is hard like really artists I mean it's not like just writing the code and throw into someone else there's no someone else on the other side you have to be able to other side you catch it as well so it research me like me there's something else in here and then there's the pain of going to the bar and drinking a lot of fireballs and not knowing how to find a way back home so the when we actually started to work this project to get it to the point where like oh so that's how all fool the kind of stuff that we're doing this when we get into production that's how it looks like to actually push this out so we when we started to work on this we finally got a perspective from what the hop scene was doing and the thing we were causing them the amount of work that they had to do to ensure our sub was easy to was like possible to actually get your servers to upgrade servers to make sure all the many dependencies that we have in place we were not going to communicate the fantasy so we were running rails and many times you have to install like a dependency like a ruby 10 and this ruby gem is going to require native code but on your machine you have a lot of stuff install you don't remember that you have and when you guys production and you guys to run you're like oh I did not your soldiers depends and then it brushes and then it doesn't go and then your application doesn't go up because you just didn't think about this and we will never like going to the people in the office against a negative zombies Newton fantasy they were just figure it out they won't run the door they would like Trump unload it was salty jamjam fulfilling oh there's such offenses ever after salt so imagine how many times they had to figure out all the different defenses that we had and we were not communicating this kind of stuff sometimes we didn't even know that they where the fantasies but so we were not paying attention and this kind of stuff started to be really like it was an eye-opener for the whole team to figure out what's going on and what time is up we're doing to discuss so if we were never uncle so a lot of the issues like the problem that we saw in the application we're not problems were not problem that were like serious to the point where it would require would simplify the stuff that galaxy with zoom because we're not being no one would wake us up like overnight to fix these issues so what's happening is that if we saw something that was broken and we felt like oh it would be good to fix this we could fix it we're not paying attention to the issues that were happening in production we're taking the system's down because like it wasn't our issue and then the very first weeks when the systems were going down we're getting bays and we're fixing stuff we were like oh this just woke me up this night so what am I going to do I'm going to fix it now I'm going to have in fact so next night this one's not going to wake me up but we only got this perspective when we were home call instead of like doing this when the other people when i'm gone and we're having the issue because the issue that we're having were not being bottled up to the development team there were not being prioritized when we got an issue like when the oxy remove like oh say to the end of the back bumper fits in in the future and you know like the backlog and backlog and backlog and it never happens because it's not a pain that we as a development team or feeling so if this was a really really bad thing for for us in there and we were like in this safe once the developers were hidden from all the issues in all of the pain lure that was happening across a violence or all this up it was happening and this is not ideal developers should not like the minute they should not be protected from the stuff that they cook is cause a new production everyone's we show the managers have to be like hiding all the communist party shirt but the bigger the developers are causing in their apartment is not something that you should hire they should see what's going on they should understand what their application on are going and when I stop that that that's going on in there because otherwise they would not amortize the people they're actually going to work the people that are pushing this stuff to production they're maintaining delegation running so what you have to do what you have to really do is to make sure your developers understand what require what is required to get school on there would be that you make stuff work because you don't want them like just banging oh I just draw my code and then importance and then there's nothing else to worry about that's not how development works and that's not all about with teams should work so what we went through was this if we just we had to fix the issue and we have to do it by ourselves because we didn't have people to do the work so the developer just had to do again it was one of the best experiences I had in my life because I was never on the other side I was never paying attention to what these guys are doing and there's even like this is perspective from developers that ops people are less right there's going to be this thinking that oh it's just a linux administrator just a guy that runs baskets right it's just someone that's like on that machine is just going to run a script and it's just going to work with it doesn't require a lot of work like this guy doesn't need to know all the programming languages that I know because I'm source of the power it doesn't enjoy yourselves so it's not it's that even this kind of stuff is going even this behavior that people are going to be like oh yeah but you know they have a different in creation is like it's really different work and requires a real different tool set and requires a different understanding what's going on and there should be a lot of respect for everyone there's going offs like really a lot of its back and the more you got into it the more you understand how dominated all the species are and how hard it is to make sure that all these applications are running correctly on on all the servers that we have so how did it work for us back then first thing was teamwork like this is this is the most basic piece that you need now that you understand that you want to do under developers to be doing box there should be there should be no separation between the off scene and the levity they have to be working together you have to be really close to each other and as you're jumping like other belapur to do the center of office work they really need guidance so other than fur has a really different perspective on how stuff will be built and how stuff actually goes out there so you need ask people to sit them and show how it works so headband Mecca what the work we're doing we're using shaft and it was Reza so at least we knew the language that was being used so we could just write the recipes chap using Ruby but if you don't if you have different tool in you're back in the developers are using and the front until that's doing the seven-man management is someone else then you will require that it's going to be required by some learning on in terms of the language of seniors and the tools are being used but you should have ops people involved in this don't throw the Vela purrs like a dinner at the fire and expect them to do good work you really want them to sit with you up seen you really want to make sure that they are working and moving in the same direction as the ops team there they understand thing for structure because if I was going to play oh why can't I just change everything and make it all better where can I just use doctoring to Bernanke's and get it all running that's not how it works and developers most of the time don't get this perspective interesting like everything is easy you can just upgrade it and just run and it just works and in general that's not how we work for for actual production environments you can just change everything out of nowhere just because your developers one so make sure we had a lot of import from the obscene they were helping us a lot we're reviewing the stuff the work that we were doing to it showing us so this is how it looks for this application this is how it will work for your application like building code get it out there so make sure it is easy for developers to access of see members to do this network to to get together and to make sure everyone understands what's going on and worse it bounced so don't throw the ball very straight into production please it's not going to be pretty your databases will disappear the data will go away and the cops will not be restored so it's not safe to throw all about guard that has no idea oxley introduction about it I remember once we were taking down one of the environments that we had and we were doing it from just from one machine in one of the photos I really want to take to help take down which games as well and and the person that was doing the work was like yeah yeah go for it then you took down one of the improvement production theories we were just taking staying down so it wasn't nice and you don't want you to have scanners that happen and these people are still trying to understand what's going on so you don't want people that are still trying to understand what's going on to be able to issue commands to destroy service and production this is not ideal so make sure that as developers are growing up in there getting used to your off stuff they're allowed to make mistakes because they're not making mistakes and for those they're making me stays on the other environments that you have and this is one thing that we were really into doing we were making sure that all of the stuff that was being built but and the new son was being built was not going straight two pillars it was going to its own environment in his own separate pieces so this was something that made sure that we were not like destroying and causing troubles in production get on home developers should always be on call dunno if the developer is responsible for a project you're working on project they should be on the rotation for the specific project like the barber should not be removed from the on-call rotation because and the old conversations where you see the issues where you see downtime when you find the bugs there cause your application to this behave if the bumpers are not involved or not be like bingo like at 12am it's not going to they're not going to change it's gone I goes okay it's fine like there's there's nothing going on like I'm not getting based there's no issues the backlog is really huge because if they get on the ball the next day you're going to be like oh we really have to fix this because it's going me up right so it's really important for four people they're responsible for something this is just for everyone involved but anyone that's involved in a project they should be on the overall rotation and you should not be someone that has no idea what's going on like like as you all saw in the last presentation if you don't know how I'll javascript work and how to debug service before going to the pub back so who's the person that then we'll have the best perspective to the bug production issue which one called if not the developers involved or next to build in the other case so these people are going to be the ones that will like and they will find and figure out what's going on as fast as possible so these are the people you want you don't want to have like someone just wind up all just be just trying to figure out trying to read locks when the issue is happening on golden its initial on the data so you also know they really understand what's going on and developers should be the people to do this kind of work and when you get you work on different teams into this time different work you get an idea of all the different perspectives they're going on in the company so it's not just the developer is not just the ops team is all the different teams are getting together they're making sure the application is running the everything is working correctly and it provides a better idea for everyone involved in a project and everything on how it also work and what work needs to be done for the obligation to actually run abruptly so it's really important to make sure that you get people to provide input to provide ideas to provide details and to understand what's going on on all of the different parts of the company and as you move people around between teams and you get developers to also do ups where you can ask people that are just going off to also do the button that everyone understands what's going on on on a bigger scale so it's not just like us like someone sitting in their own like place and doing the work and not worrying about everyone else it's important for us to understand what's going on across all the different parts of the company and development and offs really have to be working together at the scan of stuff and this is like what everyone has been talking about like it's cross everything and you just make sure like you just everyone just mixed and all do the work together the difference of having a team would be booking to front end they can do design that you do often can do develop and its really really like amazing when you're barely teams where you have one guy that one team that does packet one guy one team that has grown and one team that has asked like the difference between having people from all these different perspectives working together so all the separate teams is huge because instead of waiting because we have different teams what would happen is that we would create a car once you're right we would rate a car in general who say oh we want this done from a number see this on your team already has their own priorities they have a lot of stuff that they have to work on so what's going to happen is that and the cars that were coming from 44 us developers offer like a normal rotation well they will do would be like oh he goes to the backbone unless someone sits and with the TV like okay you guys move to start out you really need this or like one of the managers comes in and says like oh we really meet you have to sort it out and have it fixed it just doesn't move it has to become a priority so when you have people on the team that can do the work they can do all the different amount of work that are that there are needed for the sins of function what's going to happen is that you get much more productive and when you remove the separation between development and ops this is one of the biggest boost you will gather in Provence given e14 is that there's not going to be a separation between only half to the point subrogation and we have to build an application building an application is going to provide you with a much better perspective on how it all works and when you get the bounder soon understand offs the coal gets better because they will now think about what's going to happen when this is running new production it's not just how am I going to get this done or it's how is this going to affect the environment that I'm running this is going to cause more issues am I going to be creating more problems when this goes out or or whatever I'm doing and I'll just your library including is it safe is it going to do a good job am I going to cause even more issues when this sub gets out there so it's really important to provide stand perspective for the powers because the amount work and the kind of work is you do when understand what's happening on the system that you have on the other side really changes your perspective on the only stuff like imagine that you're running on a day plus or a like or digital ocean or any white cloud provider that's available out there one thing that I can't say like as a digital ocean employee is that our network is a hundred percent it's going to work one hundred percent of the time but the powers when you're running on your own machine or your own environment your own idea saying you might think like oh it's really like it's only us running on this specific data center so we won't have a lot of issues with networking but when you understand that you're going to one of this like public clouds you're going to run into stuff in there you're running with a lot of people you're going to get noisy neighbors you're going to have like versus network like some people are going to using a lot you're not going fishing off and then you meet you a lot and then the network is going to fail so even the way of going to be building software is going to change because you need to understand what's happening on the other side when it's running in actual production so you're not your software is going to have to handle the error cases or the lack of network many more times going to have to handle soul network so all these different things that developers are not going to think about if they're just building now just like we're gonna everyone is talking about like distributed systems right now but as anyone here like how long has it been since the last time you ran your application at the same machine as your database how long does it mean like it's been really really long time so we were running all these years and distributed systems but why is the database not a disappearin system like your application is running less and the database is running on another and then you have gases so you don't need microservices to been building distributed systems like if you have like a mother and your mother left talk to the database option when casting talk suit like some kind of service that guy with a cloud that's going the world at the bathing service it is a distributed system already it doesn't have to be micro services for each bill distributed system but we have this thing where oops we have this day that it is just like thrown from obligations on the database we don't we don't like say that it's it's a distributed system just because we assume that they're going to be together they're going to the post each other but they're not they're suffering the running separately change the run yourself and that person owns you go on God you will see all the issues that this is going to cause you're going to see physical actions you're going to see up against that open to medical actions and are causing the database server to run out of memory you were going to find like slow queries and all of this stuff is stuff that the numbers are only going to see when they're actually being called like if they're not involved in this game is up and there's no prep privatization or making sure that all these issues when move forward and become visible that is going to be really hard for for developers to pay attention to all of this so there should be for us when this finally that when this finally ended then we got the application out there then it completely changes the dynamics on thing so instead of us waiting for us people to do the work we became of people as well so we were not waiting for like it off steam that continued to even even happened this happened but most of that a lot the department work that was needed on y'all seem to get applications out there started to be done by the development team itself so the development team was pushing new stuff was getting new applications out there was evolving our infrastructure with the ops team and doing the work like with them it's maybe maybe it would be it could be better if the op seem like just did not exist anymore and we just have one in the most women overworked but the way it was like organized continue to work and we the absolute we're still doing working developers were doing a lot of continuing to do the work that was needed to make sure applications are running and we're on call all the time so they had to fix stuff and what what happened was that instead of having just one or two people that could punch the stuff to production when another huge team of people that all knew how to work with the Tony depakote so if something bro if we wouldn't have to wait for one specific or to specific people to work on it we have anyone the development team they could just sit and say like let's fix this up and get this out there so it was really important for for us to remove this like this wall and not have separation between the teams and make sure that whatever was happening and any work that be done be done by anyone there was no separation between one specific and we go and the other specific kind of people and for me and for the other people to the velveteen it gave us a really really nice idea of all the work that needs to be done and even provided us with someone else you are so instead of just learning a new way you learn all the stuff that's required to run to something about what are the options that we have I'll do we use top and how may be used to burn IV so do we maintain all this love services so for the developer it's really exciting to enjoin something like this like doing something new so if you're not doing if anyone in here is not doing us right now is just doing development this is a really great experience for in one of you like the amount stuff that you're going to learn the amount of stuff that's going to happen and amount of education and improvement as a person and as a professional that you're going to get by doing this kind of work is really amazing and really recommend anyone to that wants to do up in developer that wants to do ops I would really recommend them to the surprise because it's a return amazing right and we Google's ah sorry look so this is like basic it wasn't available back then when this happened to us which is unfortunate but right now you should if you are like a dance or a developer you should get this book you should read this book from cover to cover like it's really going to change their perspective on a lot of this up and I've seen a lot of people say oh it's it's a lot of stuff that we already know yes it's a lot of something we already know but it's in a book in a single place in you can just look at all the different pieces and it's not specific to how Google does it work we use a lot of this new digital Woodson and we're not at Google scale at all so I would really recommend that wants to do Hoffs please get this book please read this book and is suddenly going to take a left this book was given to me by academy there was a worker of mine just raise your hand gathering I see you from here yeah so it's I would really recommend anyone that's interested on ongoing offs and to get this book and with it because it's really amazing like education on all all all this stuff works and that's it we have any questions how did you manage like the occasional help that how did you I'm humanity people know that the folks that are formally developers now serving is DevOps web some teams would be grown what's probably the centralized specialist operation resources you mean how they would actually go stop doing development work go to Oxford yeah we talked about how that would be helping guide to be better out how did you like coordinate that yeah so at the beginning it was mostly we have to do the work so they were just like just get us like and ask the ops guy all the time so it was probably a pain for or the guy that was tempered was Richard penny so if anyone knows Richard from here oh he's right there yeah so Richard knows all the questions I have I ask you ask him all the time so it was we were not really that organized on making sure this was like we have specific moment in time it was basically we have to do something and we would ask rich and do have an example we have people folks that do this you know like where we should be looking for resources so it was kind of ad hoc getting this kind of information out of it but it was basically just getting to ask the people that were responsible for the stands out before how it was supposed to be done one question so the other organizational buy-in yeah they kind of Excel cross-functional team or result from the top down no my my boss just came to me and said you have to do this because we don't have anyone else that was basically what would happen it came to me as like you know like bridge is just one person and it's not going to be able to go to work by himself so you guys have to step up and do the work so thankfully we didn't have to go of the change to make sure this was going to work with that the company itself knew that either we wouldn't do this or the project would not be delayed so get it from from it was a small company sister it's me here in filming and we didn't have a huge team didn't have a lot of people and didn't have a lot of money so it was really it was have easy for former boss which was like Travis room and back then and he said just like you guys have to it you just have to get it done then before but if you if you need like to do this at a bigger company you can just show all the adventures we set our trap we have a lot of people do the work yourself having like no sold in very specific places it improves productivity like in ways if we're not going to see in by other reply as up so it's a really good important change that I I think you should happen in every single night there should be no separation between there is one about a ten people their own house that's all the time we have Thank You Mauricio voyage in times with Johnny from then when I 10 at 10 25 we're going to take a short break