Ignite: How DevOps Antipatterns Ruined Holidays

0 0

alright so at dev updates conferences most people talk about things how things that get better with that laughs how everything is awesome I want to talk about something to put a different I want to talk about how thanks actually crumble and go awfully awfully wrong like this when you thought I swear this was not cute I swear this was not good nor was it voice-activated all right should I continue we're gonna go old-school forgetting that all right so I'm going to talk about really awful things that happen when you don't do don't use em ups this was not the horrible thing and this is an actual real story very unfortunate it's going to be a real story it's also going to be a Lego and illustrate a story and i'm going to use the word unfortunate a lot in this talk so it happened to e-commerce company during the black friday company wasn't amazon sell but they were making about five nail in a month or so and it was run by two people who are used to traditional brick-and-mortar which means they build the IT team with the f's group on one side and ab group on another side but their own hierarchy their own managers who are allowed to talk to each other so first forward to actual Black Friday as everything horrible it starts with the page right and the personal Co receives a page and sees the the performance on the side just plummet right people rush to the side that stand in the morning things are looking bad and as any good person on cold he wants to try to troubleshoot it except because of the separation of the rose he had absolutely no idea what the application does right he knew it sells widgets and whatnot but that's the extent of the knowledge and there was no monitor setup on application itself so there was really nothing to add to so what's the next step to do escalated somebody who can help unfortunately there's that word again ops were not allowed to talk to them directly they had to go to their managers who had to go to the dev managers had to go to them unfortunately it's a black friday it's a holiday everybody's off except for the people who are on coal which our own apps people so now ops manager is frantically trying to get somebody on the developer team and nobody's real available it's been two hours and developer is still not available nobody knows where they are they're out of reach and site is still down right so finally finally the manager managed to get ahold of a developer who took another 40 minutes to get to the place where the committed they had a computer and they start troubleshooting and running exactly the same problem there were no develop an instrumentation & application monitor was considered to be part of operational responsibility which yeah also they use the version control but they use it as in dumping ground for everything there was no branching no features nothing so they just commit stuff as I go along then they go through a release process they bunch it up and push it to production and of course roll back one like its enterprise mentality right you'd never roll back way there is option failure is not an option if there is a bug you put it in the next cycle you fix it and then your shipping and CD with an X rich right that's how we do it keep in mind it's been four hours companies listen to our 30-grand an hour and orcses still see Jenna cos I mean users still can't get to the site Black Friday ecommerce site down for four hours so finally finally developer with everybody screaming over his shoulder found the problem everybody can be happy right this is slide 14 of 20 we're not done yet ski Fixodent development of course right and now he has to get into production but to get on the production you have to go through all the hoops like qie authorization sign of all the checkpoints luckily with everybody CEO CTO and corey sit on the phone and yon each other it was much easier to get authorization to push something production but none of the rest are you have to do it manually so somebody had to go in manual update everything manually oh they're the report and then say we're done so after eight hours and loosen crap ton of money really and even more so an opportunity cost considering its the biggest time of the year everybody does what everybody does right has a boy full post-mortem right who's focuses and jordan says everybody's right you can put a lot of blame own operations human development team for still adhering to bad practices but first you got to blend the business themselves right you got a boy i start bullying from the top and recently I heard a lot a lot of the fact that you know it in the DevOps it's a cool you got to do a racecourse analysis and you can't just shift all your prices overnight and that's true but like everything in DevOps you can continue to improve and if you need an example to prove in your company use this because every one of these steps could have saved thousands of dollars if only implement just one of them let alone all of them so keeping in mind if you need to use it as an example by all means thank you and thank you