Security in the Delivery Pipeline

0 0

[Music] all right so who am i i work at signal sciences i also help with devops days austin I work at lynda.com I'm an author on stuff over on the DevOps DevOps fundamental stuff over there and I'm recovering from a lot of years of operations work and security work and and all that that stuff and I wanted to start up front with a I want to start with with a summary so if you're like after these next three slides if you decide like this isn't this isn't really the talk I thought I'd want to go to you're free to leave right so hopefully hopefully you'll stay but I think the the path that we're gonna try to talk about here is security is still making the journey from DevOps it's not it's not over this is still undergoing process security seize new opportunities to automate and add value I think the delivery pipeline we usually think of it kind of a small like we think of it as Jenkins or Travis or whatever but like we need to think about the pipeline in a wider breadth and I think culture and truly need to align to help us make this work and coverage for security needs to happen mainly in three areas where you inherit stuff where you build stuff and you have your runtime and I have some advice on the side like how to deal with auditors and other people that are blockers inside of your organization as you're moving forward so I'd like to start with a CI CD journey and how like as you're thinking about going with continuous delivery I wanted to give you a little story like how I I went through that how I went through that process so I've done this at three different companies and currently I'm at signal Sciences and now we do about 15 deploys a day we're moving fast we're always kind of an enabling our team to kind of go go quickly roughly we've done about 10,000 deploys in the last two and a half years kind of development and we sort of were we're getting things started and maybe more than that but we'd kind of like lost track of the early days we're not we're not exactly sure and early on I I was tasked at work it was like hey we're gonna build this we're gonna build this thing that like allows us to deploy and help push this out and it's gonna be you and you're gonna demo it in a couple weeks and I was like oh sweet like that's that's great and so I you know said a prayer and I read the continuous delivery book by John Bull and it's like alright I'm gonna make this like this happen and we're gonna try to try to make that happen so I really both jazz and Dave are really great and really appreciate their work that they did on that book it's totally worth checking out and one of the things that really was helpful for me to think about in this is like continuous delivery is how little you can deploy at a single point right it's like I want to deploy just like a one-line change and be able to walk that through the whole pipeline in the middle as little time as possible and we started talking about words like we want to optimize for total total cycle time that's the time that something goes in when you when you push it into your get your github or your version control system and it actually rolls out into production so time to push to time customers are using it and so at first we were like like five minutes or seven minutes and now I think we get it down to like let's see what it probably like two minutes or so on average depending on like if you change certain bits like there's invokes other tests sometimes and you get that'll be a three or four minute deploy and then then we built a button right we said let's build this thing so anybody in the organization can go in and once you commit code you deploy it and and kind of off it goes and and so we're also a security company and whereas the software as a service security company and we wanted to make sure that continuous delivery and all that stuff included security like this couldn't just be something that we just did and didn't didn't kind of fit with our overall our overall story and before I was in signal Sciences I also found this idea of rugged software circa like 2010 and there's this lovely presentation by Josh Corman David Rice and Jeff Williams and in it they started talking about things that that that gave us new ideas of what security really could be and so I think I pulled this out of slides that I did in like 2012 it was like security like it's always there's a lot of problems with security you're kind of judged on like absence of events like success is not really well known it's high cost it's often negative in a lot of organizations had a lot of fear uncertainty and doubt and really it's like a toxic word like you want to go into an organization and like not get promotions and not do well it's like like I'm the security person they're like okay you know go over there right and anybody is that ever happened to a friend yeah right so but rugged it was like this idea and some of the people in the security industry that had been there for a long time started thinking like well what if we started talking about this is a quality problem and we want to start verifying like we're providing the right qualities that we start adding benefits to the organization stuff that's affirming stuff that's positive and we're putting like known quantities on things and therefore there's a while there's like we should have the software should sit with like ship with like the cereal box label to tell you like you know what what stuff is included in this and all the dependencies that are built into it but at the time we were really trying to wrestle with the idea of what does this look like to do things in a different way than the way we did before and then about four or five years ago I started the gauntlet project which is which I'll talk about more later but that is a way for us to help kind of take security and DevOps and kind of merge those together in a way that groups can collaborate and I found over time like security is different in a continuous integration world and security has a dilemma and that dilemma is that it doesn't really know what it's doing doesn't really know how well it's doing and I think that in security we always kind of came up with this idea like you're we have this binary bifurcation of how we think about about security so it was always are we breached or are we unbreached are we secured or unsecured and like we didn't really resist it was just a one zero type relationship there and I don't know like that's it's not good it's actually not true a state of reality it's actually an over adduction ism of what we're trying to do it and I think that we would we would take these complex systems and just try to say like I've got a lot of firewalls and like everything's blocked and so like they're there therefore now we're okay but I think that we we start kind of moving away from that that question of whether we're breached or whether we're secure and we start thinking of how does security add value I think we're left with a different approach we're left with a different a different a different way of thinking and I think that's going to be more in line with the kind of the future as security is growing so I'll quickly summarize summarize how I think that we got to this state so agile came on the scene and everyone was really excited and it tried to remove the kind of the ability of not knowing what we're doing and how we're getting there it just said we're going to we're going to ship it right and it largely worked in a large and a large number of ways but really it just created a culture that they gave us fast delivery fast feedback loops and but it never really oh it sorry I got these slides that order that's this is my funny slide so that's about and those are about as funny as they're gonna get I got it just warned yeah like I mean I know it's afternoon alright so but but we we didn't really get a jellal the way into operations like and it sort of stopped there and I think that was that was whenever we started to see the next rise right and that's when DevOps came on the scene there's a continuation of agile operations in Tom ylim and Shelley's book he says the same thing DevOps is the application of agile methodology to system administration and then let's see in 2009 there's the famous we do ten deploys a day over at Flickr presentation of velocity this is a big system operations conference and everybody there just sort of lost it they're like you you know you do deploys once a quarter you know if you're lucky maybe once a month if you're like a real cowboy or something but you're not doing tend to plays a day that's ridiculous and and it was at that conference and a couple other things that happen around that time period what DevOps really started to take off Patrick Dubois I had the first DevOps days he coined the term and had the event here not too far away here and again and early days you know we were really good at Photoshop in the DevOps community so we were like we just sort of wanted to like illustrate our problem right it's the development and operations to two incompatible things because we kind of had this wall between us and so we're really spending a lot of times tearing down the wall I'm like if you go to DevOps today's events there's a lot of cultural talks because in a lot of organizations this is still this is still existing or still kind of in the process of going through that and we always encapsulated it with this with this presentation and or this this graphic and it was it was just like it's sort of this this picture embodies kind of all DevOps I had to train some sales guys recently and I said you know like this here's two graphics I just showed them this graphic in the previous graphic and I was like that's DevOps and they're like yeah we got it like that's it right there like that they got the emotion aspect and it's why we really spend a lot of time in the cultural pieces of DevOps early on we also found in a lot of organizations you saw like a ten to one ratio of developers to operations folks like so it's a complete order you know it's roughly different for different organizations different teams different things that you're doing but there it's an order of magnitude problem that we just weren't able to solve right away Patrick Dubois he says you know culture is the most important aspect to DevOps and for it to succeed in the enterprise and I think it was like in 2011 or 2012 there's the Qatar IT Journal and there was there was because there's early days and it's like can you do DevOps in the enterprise and I was on one of the teams that was featured in that one of my my co-workers Ernest molar and good friend but our team had just kind of at the right time like we had we'd gone cloud in 2010 with with kind of a charter and we also said hey we're gonna do be doing this thing called DevOps and it was like sure we don't we don't care let's just do it and then we started to see like how this was able to affect a lot of changes in the enterprise and it was really really fun time but it really drove me to think about culture like how are we how are we changing our culture and I think there's four keys that I would leave you to culture and I'm not an expert on this I'm not a psychologist or any of that stuff but I've always seen like you have you start kind of building a mutual understanding you build shared language shared views about how things work and then you put tooling in place that helps encourage all that collaboration and those pre things I have a I have a five-year-old daughter and when she was three or so in her rooms position so like right when you look out the windows you can see the moon as its setting and so every every time you know not all the time but whenever there's like the right right time of year and everything we and time that night when she's going to bed and we'd see it and we look at the moon and we talk about the moon and we started having like these same values and the same same understanding and then I started to figure out like a couple days later I was like I got excited like I would come home and I would tell my wife was like oh we're gonna see the moon cuz like the moon is out tonight and then then I started installing apps like moon tracking apps and like weather apps and like trying to know like what phases the moon and it's like now I could tell my three-year-old daughter and impress her with like the moon phase you know you know it's it's ridiculous right but but like all that stuff started to change right like a new thing happened for me like a new priority happened and I thought that was to me I think that kind of illustrates those four four key aspects but all this you know we're kind of here how we got here where was security and all this has security just been sitting by and and I really think they have they haven't really participated in this and they're just now sort of making the making the change part of the problem is like security has always come from a compliance driven aspect it's it's been inevitable in our industry but that was a that was the way that we kind of spent the last 10 15 years there's this really great book it's about browser security but the first chapter in the book he goes and does this talks about this history of security all over the way from like the 60s to like the modern day a web browser and where we sit and I'd like to read this quote for you he's talking about risk assessment but doing security by risk assessment introduces a dangerous fallacy that structured inadequacy is almost as good as adequacy and that underfunded security efforts plus risk management are about as good as properly funded security work and does that does that bother anybody in here does that leave you feeling uneasy yeah it made it like made me upset right I'm like what what the heck like why would we how believe that we could have this that we could just have bad bad workmanship bad quality and just say because we put enough structure around it we put enough legal policies around we bought enough insurance around it then now we're good like that that makes no sense you wouldn't do that in the real world why do we do this with software and why is that why is that idea and also we saw in organizations there's this there's another problem like it's not just ten to one it's like a hundred to one or three hundred to one or three hundred to half like like it doesn't matter what the actual numbers are but like everybody I've asked like these are these are order of magnitude type problems and security doesn't have the resources and one of the hidden secrets about security is like there's not enough people in the industry to solve this problem like if you said tomorrow we're gonna like 3x our security teams like you couldn't do it there's not enough bodies to do that in the industry and you've seen you see security people hop jobs every eighteen months because it's like recruiters keep calling them and they keep doing it's just like just like development but maybe even like hotter right now like it's great it's crazy and also security because of all that other stuff because they're they've been addicted to compliance and kind of been been sitting in the organization that way because there's the large Delta between you know how many people are working on security is the cultural outlier and a lot of organizations and they don't really want the same thing I had a friend so I've always kind of been the security guy on the team and one of my friends and he was actually like my manager at the time was like well security you guys just prefer to have a system that are powered off and unplugged and I was like well ok I guess but like that that sort of hurts my feelings you think I'm like it I like I know that if we do that like I have no job like you know I'm not I'm not an idiot right and then yeah you know I try not to do this because I've lived the pain on the other side but how many security people have ever said like those stupid developers right has anybody heard security people say that as if a friend would have said that right it happens and it's not it doesn't feel good and but I think this illustrates that is a divide between the two also have I found this lovely gym in a a white paper from these people they're doing web app firewalls and I'd like to read this to you this is every aspect of managing web app firewalls is an ongoing process that's the antithesis of a sudden and forget a technology that's the real point of this research sits to maximize value maximize value from your web app firewall you need to go in with everyone's eyes open to the effort required to get it to get and keep the laughs running productively so when they go on to tell you like you need two or three headcount to do this right a very questionable technology that's for any any effect and we don't have enough people in the industry anyways and now you're saying like now my new business is running this piece of junky technology right and I don't mention the name because I don't want to you know make fun of them publicly but Security's always taken that same approach like they put a really heavy constraint on resources it's a big bottleneck approach there is an article I guess last year in Fortune magazine the average time to deliver for corporate projects is not only as has gotten worse it's increased over the last five years which you know as a DevOps person like makes you really sad like I'm like why is that and then the next thing he says well the growth of security functions which is too poorly coordinated and resulting a proliferation of new tasks in the areas of compliance privacy and data protection is the main reason and the main culprit for that and so ok so now we now we see that security is is causing a real problem ok this is a new one it is 30 times cheaper to fix security defects and prod are in dev versus prod newsflash 2002 right and and and so I this came on this came in my mail in a in a flyer from a security company within like the last year like it was like hey we have news like it's it's more expense you gotta be kidding me right but security is feeling the same pressure like they're behind as well alright so that might have been the funniest one so I'm a little worried for the right for your sake for the rest of the slides because that was that was my best one okay well anyways Security's ineffective oh that might that's pretty funny too I like that one all right mr. robot fans but I think security knows that it needs to change or die and I think it's true in the history and I think we've seen it across across the way Stephen Bullivant he's written the firewall book that was it was really huge and he came out with a book last year or I guess a year and a half ago he says companies are spending a great deal on security but we read a massive computer related attacks and clearly something is wrong the ROO the problem is twofold we're protecting the wrong things and we're hurting productivity in the process so it's he sees that security is like in in both ways like causing a problem we're not adding value and we're protecting the wrong things of course you know you could just print a million of these slides and show like they asked security is expensive the the last DevOps the state of DevOps report last year our performers spend 50 percent less time or median security issues than low performers high-performing organizations achieve quality by incorporating security and security teams into the delivery process and here are some of the findings that they came out of this rich mogul who's a big in the security industry at RSA Conference got up and said and in front of a large large crowd and said a large percent of the companies that are hanging out in the show floor here will not be here in five years and he said you know basically everything that we know about the security industry is changing that we see that the writing is on the wall and all that stuff is happening as we go a couple years ago Zane Lackey gave this presentation about how at Etsy they deliver faster better cheaper and I watched that and I thought it's that's right on as a friend of Zane's but I also was really really excited to see that that approach all right so let's talk about SEI CD pipeline and what that looks looks like so I think for different people pipelines are gonna look look a little bit different and most people will break it down into some grouping of this like design build operate or a design build deploy operate I also like to think about inherit is like one of the steps there and I'm just going to focus on these three spots inherit build and operate and I'll explain what I mean at that here and and there's there's other stuff you can do in these pieces like you should you should do all the design pieces there's other stuff like for and in your design you know you do do I write dataflow threat models and all that business but I'm not really an expert in that so I'm not going to spend time on that and then on the deploy piece totally you should manage your secrets vault other stuff like that also not going to talk about that because that's really specific to your language your framework and all the stuff that you're using but I'd like to talk about these these three and these are the three considerations I'd like to propose and maybe maybe a couple questions that you can think about that it might help drive drive you forward as you're as you're going through here what have I bundled into my application that leaves me vulnerable and and think about not just the the couple packages that you you know install with your install with your your actual app that you called in there your any dependencies in the app but think about like what about the OS level what about the the provider that I'm using to deploy it on the talk on docker stuff this morning or this after this afternoon was great because as you think about all the stuff you're bundling in it's like do you want a bundle in 128 Meg's or five Meg's of dependencies right and you can start breaking that stuff down for build do i do my build and build acceptance tests and integration test catch security issues before release like how many people are doing security testing in their build phases couple yeah okay great but you know I think there's there's a lot more that can be done in that phase and we often just sort of skip by that and say hey we'll pick that up pick that up later and then operationally it might be an attacked right now and if I am like is it working are they actually getting any any successful foothold the company I work for we kind of play in this laugh space a little bit or we playing this last space but we ask you know we ask people all the time like do you know if you're being attacked and like you just usually blank stares right and it won't out any of our customers I'd be bad but like it's kind of like that's indicative of the industry that's indicative of what what we actually know what's going on with our system Shane and Lisa gave this talk over a DevOps days Austin and broke it up into these four phases and I liked it because really asked at each of those phases different questions and so I borrowed some of my ideas from that and so just want to make sure she had good good coverage there okay security in the delivery pipeline so inherit what you know we've already talked about this a little bit today okay open SSL right all of a sudden it's like oh I have a lot of problems why because I'm using this thing or shellshock because I'm using this you know old version of bash or whatever Gareth rush Grove did a great presentation on what's inside the container and showing how many how many containers have vulnerabilities in them there's statistics out there saying like 30 percent of docker images out there have vulnerabilities right now dr. hub you can actually see the vulnerabilities listed out with different types of containers if you're using something like Ruby you should be doing like a bundler audit to check like what gems you have going on there there's other other tools like Linnaeus that do the same thing if you're doing service there's a thing called snick which does dependency checking for service but specifically a node there's docker bench which will check for dozens of common best practices and things that you're doing for deploying and and and dealing with containers in production also a good Erland friend of mine has this library called retire Jas it checks if you're using any any JavaScript that's like no longer any good we do this for our for our sites where we're delivering things and just you never know like whenever something your keep something statically a static static website and then you know the version of jQuery or whatever you're using is no longer any good because it's got some vulnerabilities you want to be alerted to that so putting all that stuff in your build pipeline is really great yeah and I think that's kind of fits Kimber if I have any other stuff on this but you want to instrument your CI system with all the checks for all the things that you inherit and that's just as much as you possibly can if you're using containers like twistlock and aqua or you know look at some of the type stuff and Blackduck if you have some money but there's a lot of like free stuff that you can pull in I depending on your language and all that stuff you just pull those pull those in all right on the on the build side Security's function of qual and I think that when we look across all code languages this is from a white hat security report like a couple years back if you look across it doesn't matter which language you're in they all have vulnerabilities so across the top is like ASP ColdFusion net Java Perl PHP and then on this on the left hand side there is cross-site scripting and all like the OWASP top 10 all the way down and they're just showing like all languages everywhere have problems and so we can't just say like oh I'm using go like I'm good now right or I can just say oh or using I don't know yeah I don't know it's like you pick on any one of the languages here you don't want to do that so we used to go at work so I guess I'll pick on that one but also another problem with security and the way that you like instrument that inside of a build pipeline is security tools are really noisy they you run them and if you pick up like just an off-the-shelf scanner and you hook it up to your your system that you want to do some testing on it'll like try to spider it and then it'll of you know enumerate a bunch of stuff and then you'll come back like three hours later and it'll either be crashed or it'll like show you something may be useful or it'll show you a bunch of stuff that's not really true and if you've ever had as anybody had to deal with security scanners so I saw a couple chuckles so if you've had to deal with those you kind of know I'm talking about right and then in the end you're kind of left with this like giant PDF or whatever to deal with but so I sort of felt that we really needed a new way to collaborate across like developers operations and security and to see what we could do to make that happen and I was hoping to really make a language that would kind of put some of that collaboration into practice and so that's why I started working on a project called gauntlet call it is open source at MIT license comes with a lot of pre can steps that'll help you compose and in what we call attack files some statements to go ahead and like be mean to your code which is our little tagline that we like to use it doesn't install any tools but it does sit really nicely inside of your CI CD pipeline it's built on top of cucumber it's not really meant to be a peer replacement for like a BDD type thing but it functions in the same way so it has good good handling of of exit status and all that you can check it out at Gamla org but the the general idea in our very slick graphics here is that you would run your code through the gauntlet there there it is did you see it just don't need it do it again okay yeah you're gonna be like in your and your feedback forms would be like awesome graphics like I'm hoping that will come through okay that's like that's the full jokes that's all I got anyways I'll just stick to the slides now so if we look in the directory we look at the attack files and it pulls in all those attack files and you're just getting started by doing it like a simple gym install we also have docker containers which I think you have an another slide but if you read through this it sort of fits kind of an English language approach here so given I'm I'm looking for cross-site scripting and I'm gonna do it against a URL on my site and you can make it more specific so like on my login page I'm going to look for this and then then you give it a scenario with some description there and then you say given that I have the tooling that I need in place installed when I launch an attack against this URL I expect that the output should have zero issues detected if not failed failed the test and and send an alert status out I was just talking to a friend over at UnitedHealthcare and he was telling me that it's like we have saved millions of dollars using garland for the largest healthcare industry project out there right now in the United States and I was like oh man I should have figured I had to get tens of those you know but you know he's really you know saying that they were able to start taking all these pieces of their applications across their entire stack I think there's this project spans like three hundred three hundred applications he was saying it started putting in like default default checks so looking for cross-site scripting looking for sequel injection you can kind of set up some parameters and like was starting to able to bake up all these things together and now they got everybody testing kind of a low bar a low bar security threshold there I did a workshop a couple saygus about a year and a half ago with Matt Johansson and it's it's at this bitly link so if you're interested in it it's got eight labs for how to use gauntlet it's got using gauntlet for doing things like network checks if you want to like do things like that or if you want to check for application issues like cross-site scripting or sequel the sequel injection or other apses I don't know it's feeling I don't know what I was I don't know what I meant there I'm sorry about that and then handling reporting so you can you can report in two different formats you can pump out to JSON or whatever how to hook it up with environment variables and importantly showing you how to do it in a CI system and a CI setup so there's the the bitly link but you can check it out again it's all using this project inside of our repo instead of github just called gothic gauntlets - demo and I'm really sorry for the spelling like when we released the gym originally like the gym with an EE was taken so you know you got a deal with the smaller hipster gauntlet with it just a tee there and so it's pretty easy to get started we also have a starter kit that installs a virtual machine and gets you to bootstrap there and really we're just trying to help think through like creating feedback loops right we want to create a lot of feedback loops from development operations and security and that's the hope that gauntlet would do inside of those teams because now you're able to take the the knowledge strapped and and time time resource strapped people in the security the one at 110 won and they're able to implement their kind of knowledge base into the end of the system and everybody else can reference it and you don't have to be a security expert to see like oh like that test started failing whenever I checked in this new code like that's you know you may not know what the test actually does but you can now get to the root of like okay now I have cross-site scripting for some reason it wasn't wasn't sure why also gauntlet demo is also hooked inside of Travis so it runs CI builds if you're interested in checking that out and that's what those those look like a lot of teams are using Gallatin containers because you're like I don't want Ruby on it so even for our stuff inside inside of single Sciences we use call it in a docker container and it just spins up runs all the attacks and then and turns out the results for us and you can check that out at gauntlet docker other stuff you might want to check out in this space zette attack proxy is a really cool project it lets you do if your views like a web proxy but it lets you do replay of things and you're able to do both fuzz testing and just other type of attacks with it as well if you're doing static code analysis something like break man's a really cool tool I think that was mentioned a little bit earlier and that's but that's specifically for Ruby hey check that out alright I think we're in the last last bit here how's everybody doing we're hanging in ok good all right so configuration run time you know earlier today we're talking about configuration is one of the biggest problems that we have and and you know problems we fix I didn't show you some of the other slides from that white hat report but it's like problems we have today and then then come back a year later percentage of problems that got fixed and it's like yeah like you know it's like or how about how often a bug stays open at a corporation that's you know medium level or lower it's like 200 days 500 days something crazy and your do you think well how's that even possible okay so for configuration I like chef in spec you can do but you know for using chef like that's useful if you're using something else you know check them check that out for those I only know stuff that I know right but you can do audit and do CIS benchmarks on machines other stuff that you could look at for cloud providers ever do thread stack alienvault they'll look at your cloud configuration and see like did anything change in between and be able to do reporting on that I know there's other companies in the space but I've some friends at some of those places and I think those are good pretty good options to least start taking a look at there okay so for run time okay and that's that's pretty much my best gif and the whole thing okay so okay you know run time is arguably the most important place to start and I think that's the most important place to instrument because you're actually able to see what is actually happening with your application and it can inform some of the other things downstream so you want to think about am i under attack and where am i under attack you can get a lot of benefit using like something like modsecurity and pumping that over to a log stash did an elk stack and reporting out of that other things you want to look at in this space or like rasp next gen laugh or like web protection platforms which is kind of a marketing mouthful there but you should also look at other other companies I know that Ruben's company would also fit up here on this list so you definitely want to give them give them a check and the marketing team was going to see this I had to say that they're ours are the best but really like you want to be doing rasp you want to be doing something in this space to see like what is actually under attack what's going on here alright so you want to start figuring out how to detect what matters so whether that's code instrumentation in the actual app itself or whatever but you're looking for things like account takeover attempts what part of the site gets more attacks than others most likely of vectors of attack business logic flaws I had this really interesting conversation with a friend and they were saying what they saw and their runtime actually informed how they were going to prioritize their backlog because they were doing all these bug bounties and they were there you know saying okay $20,000 up for grabs if you hack our hack our site anything that came in high or critical like that got fixed like immediately anything came in medium or low just kind of went into the backlog and they paid out like a hundred bucks or whatever for the for the bounty but just never really prioritized in their their environment and then through adding runtime instrumentation they were able to say oh like we are our login page gets like these types of attacks and like do we have anything with that tag that's in our backlog oh yeah let's prioritize that like those are vectors that people are actively going against our site with and so we want to know we want to know that I should have been should have oh well if you're doing a bug bounty that's right here's a couple of bug bounty sites that are they're good these are also commercial but you can I mean you can run your own of course but these are something to kind of make it easy to easy to hit there one thing that I found is a is a problem is a lot of times people will kind of you say I want to do a pipeline developers going to push code and then they say things like you know you gotta have separation of duties and you know just like go to is considered harmful we're gonna say separation of duties is considered harmful we're not we're not people you know launching nuclear missiles right you don't need two people to turn a key to your you have like your paid staff that are engineers that are it should be allowed to be trusted with their environment and you can defend this with even in PCI I think it's like PCI 36 or can remember the exact number but there is a little wine it says like you need to have separation of duties and but if you read the guidance it just says the same account that's used for development cannot be used to use to push for production it doesn't say that the same person can't do it or whatever like a lot of enterprises and big organizations have interpreted this way I have a friend who was an auditor and she's like oh yeah we've been doing this for like decades where where people would you know just the same person that developed it you know is also pushing it small teams and you can take that same law and process an approval control process and like make it work for four big teams and she was working at a big company doing the same thing and kind of that's how they do it so you just invest in improving out your pipeline making sure that the controls are so that development and test and production accounts don't match and that there has to be some promotion going on there's a there's a toolkit there's called the DevOps audit defense toolkit it was kind of a bunch of security and compliance people that said okay we want to give people that are doing DevOps and security a way to move forward without being blocked by all the lawyers and accountants and stuff and so we they produce this 20 page PDF and it kind of goes through assists can different compliance specs and it breaks it down because system by system controlled by control and helps you map to say yeah here here's how we get over that and so just a little bit of argument if your auditor and kind of this document and you should be should be square to go if you're not I'd love to love to hear that okay so there's three things that I learned come on the journey and I think we're kind of running late on time three lessons I learned one security is not a binary event so we want to embrace feedback loops and continually kind of continually innovating there we also want to think of attack driven defense instead of compliance driven driven defense no one no one's gonna be adding value just by checking the checkbox of compliance but if you're able to actually actively know where people have an insight into where you're undergoing attacks or being able to defend against that whether in development or in run time like that's really beneficial also there's a there's a lot of movement inside if you're in security or your friend in security might suggest to them like every time their a blocker someone is going to go around them and in in the Riot Games office they have this big sign that says like if you are a blocker if security is functions as a blocker you will be routed around right it's because people are going to get the work done that they need to get done whether security says they can or not so you want to be part of adding value to the chain and you know putting as much automation in place but not not making those other things so all right so here's the summary that we went through earlier hopefully that went good like I said if you want the slides you can hit that email I should Auto respond to you and I think that's it any questions