Generating and Running 1,000,000 Selenium tests in 60 minutes

0 0

thank you very much for stopping by so I'm going to be talking about generative testing it's a little bit different from a lot of the testing that we do today I have some interesting properties as well as some interesting caveats and challenges so first off just real quick basically I'm just some guy engineer pay garden get me normal places I work a lot on functional programming stuff so we're actually a closure shop I do a lot of work in o camel as well so I'm approaching this from a very functional programming mindset and this talk has some pretty strong caveats so first of all this is mainly meant to be a thought experiment so we use a lot of these techniques ourselves but they're not necessarily things that are going to be able to graft onto your existing code base right away they take a fundamental rethink of the entire architecture in order to be able to pull off a lot of this stuff therefore it's very well suited for greenfield projects stop where you can kind of come in and lay a very strong foundation in the beginning I assume things like a functional stack and I mean functional in a functional programming kind of way so the Scala maybe closure o camel Haskell stuff that lends itself to very strong immutable persistent data structures even immutable data bases in in this case that'll come back to help us a little bit later and then this is not a basic talk I assume that if we're doing this kind of thing we all understand how we're going to have a fully parallelized set up so we're not running a billing tests sequentially in an hour there's a lot of parallel that something needs to happen here and that needs to happen at the browser level let me to be able to happen at the server level even for your testing infrastructure and then your runners in your reporters need to be able to around in parallel as well with a greenfield project in thought experiments a lot of times I think there's value in saying rather than figuring out what can we incremental e do better about our existing code base here what if we were starting from scratch and we want to kind of create a perfect setup and we will never make it a perfect set it but we want to try something totally new and you kind of see what you can achieve given no restraints and then based off of what you can do there you can figure out how you can slowly start to graph that back and back poured that onto existing code bases so with that kind of start off with what is testing for I think you guys will all agree that testing is for exploring state transition space for invalid states rights I mean it's a terrible definition but actually will help us kind of think about this a little bit later so give an example what I mean let's just imagine we have a simple cart on a site or a nap we just have a couple of steps right step one is we're going to add an item and that's going to update our states so now our item ids we have one item in there and we can take that state and we say all right is that state valid so yeah sure we remove an item still value valid and if we add a non-existent item that is not followed so this is what i mean by an example of a state transition which is an invalid we should not be able to represent this kind of thing in our code usually oftentimes this takes the form of maybe we do several steps and then we run as a in order to build up the states so for example I go to the site I log in I add several things in my cart and then I click on this thing and really what we're doing is we're building up States until that final step and we're running a test over that final transition where we say alright whenever I click checkout I should have this many things in my cart or whatever so this is kind of useful because whenever we're traversing the state graph there are times where we want actually narrow in on one specific transition so testing today is largely kind of example based where we hand write unit tests and this includes things like page objects and whatnot it also includes things where we use an IDE maybe to record our tests and it's kind of an interesting idea because we think we can kind of manual enumerate all of the important examples really is what we're saying whenever we write one of these tests right when did we write a cucumber test and we say as a user I do this and now to do this I do this and we kind of feel like we've covered a very important thing there but actually there are millions of millions of variations on the thing that we're not really capturing we're kind of hoping that we've got the general structure there in fact we're very optimistic about it because there's a componer combinatorial explosion between features so if you have one feature you can write a couple of tests and that's pretty easy if you have two features the interaction between them you now have to write a lot more tests and three the number of tests just kind of starts to explode so so for this talk I'll just go over a little simple app it's called osmium so book summarizing site you guys recognize most of the ideas so basically we can submit a book to this and we can give a summary of it and we can rate the book that's it we can log in a viewer account update our account this is about as simple as you can get for a kind of real-world ish demo app like I think we can all agree this is a very very simple app this app that's very very little and just show you like what it looks like there is it's mentally kind of emphasize how simple this is I've kind of circled all the different actions we can do from here we can login we can sign up we can view a book and we can list the books up at the top on the login page we can login we can login we can sign up we list books and then from the viewing books page we have a few more actions so like we can see already that there are a few actions but overall still very very simple app that site flow from this might be we can for example go from the listing books page to viewing a book so if we're looking at the list of books we can click on one and view it if we are not logged in from any page we should be able to sign up or login saw down the footer if we are logged in we should always be able to log out we should be able to view our account and then from the viewing our account we should be able to update our account now if we are logged in we should be able to submit a book we should be able to read a book it should be able to update a book so pretty simple still we have just kind of a very simple stateflow our site flow diagram here but actually it kind of helps if we don't think of it as a flow diagram but rather as a tree for example if we start off at the list books page what options are available from us from there so just like in the previous example we can view a book we can sign up we can log in but if we now Traverse to the next step we go to the login page we can sign up and then from the signup page we can log in or we can list books we back up we go to login page and we successfully login we can now log out we can also submit a book we can list books but now because we traverse this it actually the state is a little bit different because now from this look or listening books page we can log out we can view our accounts we can view a book and if we're viewing a book we can still log out we can list books we can view in our account we can update the description we can write a book and then backing up if we go down this branch we can view books login list books guys kind of get the idea so the idea is actually it's when or we're think of it as like a slight flow we tend to think of it as like this kind of closed loop and actually it ends up being this ever-expanding tree of state possibilities and this is incredibly difficult to enumerate in tests it actually like for each one of these branches there's an infinite number of sub-branches generally like very few of these end up terminating and so whenever we write tete or whenever we write or record tests we're actually are tracing a specific path through this tree and we're hoping that kind of we've captured the canonical example and that's going to be enough to protect us later on whenever we do some refactoring adding a feature of fixing bugs and this actually gets worse whenever we consider concurrent access to a shared resource so simple example let's say that we have two tabs open in a browser tab a logs in the tab B goes to the add books page and it fills out the form it does not yet submit it tab a logs out and then tab B submits the book normally whenever we're looking through that tree we would expect that through the linear flow you wouldn't be able to do this right because we've logged out that's no longer a thing that's accessible but actually because of the concurrent nature of this like maybe a user will do this right users are encouraged about and like the way that they will find bugs in your site and so now we have this question like what's the result in state did we create a book did the book like if we have a submitted by ID like who the user that created this this book is it null did we write a test for it did anyone think to write a test for it it's kind of the challenge there so an alternative is just not to write or record tests in fact you should generate them this is by John Hughes he's a creator of click check which is a very famous generative testing and property based testing library for both high school and Erlang so or you know alternatively as we just heard in the last talked we should automate all the things even writing the tests so the core proposition that we're going to say here is that it is a much better use or it's a better idea to use infrastructure computers rather than Cuban times human time computers are really good actually at enumerated all of the branches of possibilities much better than human beings like we tend to lose interest if we say all right what if I click login seven times and that click logout and they click login again very few people write that test but the computer one doesn't mind Peter will do it in fact if we can teach the computer to do this then if we have if we write properties about our tests for example that we say whenever the user is logged in they should always see a logout link right that now generate or we can out generalize that over sorry if we take that that's property and the ability to walk through the site and to generate all these different examples that one property node is able to expand and scale over millions and millions of tests so we get kind of constant effort no matter how many tests we run versus the linear effort that it takes if we write a million tech or if we want to run a million tests of handwritten ones we have to write a million tests so this scale is much much better for covering all the state space so they're kind of two properties of properties two classes of properties that we can assert one is about no matter where we are in the graph here we have certain properties that we want to assert about this state so for example as we're traversing it we should never see a 500 error page right the user should never be able to do anything that causes a 500 similarly they should never be able to do anything that causes a javascript error if the user is logged in they should always see a logout link and vice versa there are all these kind of very fundamental basic properties that we can assert no matter where we are on that graph and then but more interestingly we can actually target specific transitions and we can say for example that if we're at the rate book page and we've just kind of come out of it well one thing we know is that the user must have been logged in in order to do that that if we're keeping track of it then the number of ratings for a book has to increase by one the updated at timestamp should have changed so we can do tests on the properties on the state but we can also test on the page on the rendering so for example the logout link must be visible we can do things like if we've updated book then we must be at the view book URL right after we've edited the description it should send us back to viewing the book same thing with logout count sorry submit book each of them has a specific set of properties that we want to ensure that no matter where we came from whenever we are going into it and where we leave it that there is the state conforms to certain properties and the page conforms to certain properties so there is a challenge in teaching a computer to navigate our app you you can do things like for example oh so as we saw every page has an inbound kind of edge and outbound edge so all the other pages that link to it and all the pages that it links to and the page model as we saw and we've seen before can explicitly list these right so we can actually encode in the page model and we can say that this page is a larger transition to these other pages and then you can use as kind of a predicate you say that given this state here are the pages you can go to and here's how to generate the link to them but this is pretty clunky and it's actually really prone to rot I'm not a huge fan of this idea if I have to go back and maintain something and if I forget to update it then and the tests just kind of silently don't cover something then I am likely to get in trouble much much later so I want something where the computer just kind of does it on its own so in our case what we try to do is actually capture at render time we say given this input which is some immutable piece of data even the database is immutable we say all right here's the input what are the outputs that are available here so we actually rendered the page to an intermediate form and closure it's known as hiccup and each of these so rather than actually just hard coding a link inside of the the page or whatnot we use functions to say are a generative button with this action name and so we can then take that data and we actually extract that we walk over the data structure and we say all right here all the possible next transitions that we can go to based off of this input at this node we do a couple of metadata tagging and then what happens is as we generate these nodes we then walk that we keep them over to the side and we can walk down them later on and we just like to just recur and then that's how we actually are kind of teaches computer to navigate our site and then each step along it away each time we go to a new node we can run the appropriate assertions so now we the computer knows how to render a starting point so now we just have to list our starting points it knows how to figure out where I can go from there and it knows which test it should run at each node so just give me an example what this might look like this is an array of that was generated by one of the tests and it's going to just say at the start go to the homepage click on this action fill in this password with this you can see that in this test case it really liked clicking on this element for whatever reason and then it fills in the email this is the kind of test that a human probably wouldnt rights right it seems not really relevant to click on something five times but often times you'll find that there is some states then ends up changing because of submitting of a blank form over and over or whatever might be so to give you an example what a test might look like we want to know that no matter where we are on the graph we should never show an error page there should never be a case where the user sees some error and so there's a bit of setup here can kind of ignore this we're just getting a new browser and then we tear down down here but we're saying is I want you to walk the graph in steps so starting from the homepage I want you to render the page get all of the steps that are possible and I want you to walk down each of them and then walk down each of all of those subsequent ones five layers down so there's this huge explosion of number of possibilities now each time you do that I want you to run this function where you give me the the current action and also the path that we're on and I'm going to run some assertions based off of where we are so in this case it's easy I don't really care about the specific node and I just say you know I want to make sure that I don't find this string in any of these these pages and if I do I want to record the path how did we get here what was it that we did and where to trigger this error page so a quick demo this has some assumptions so we have a few fully immutable database we have a referential transparent application where we have States passed in we have the database is actually passed in we can muck about with it as much as we want and then if we pass in the same database later on we'll always get the same output and this is really important because as the computer is doing crazy things and traversing randomly through our application we want to make sure that whenever we replay that same path we get exactly the same output so in this case I'm just going to look at here all the failing paths that we have so far ooh this could be tough I might have to talk quickly all right so we have no failing pass so we're going to start our server and then we're going to run our tests and we want to walk 11 steps down the graph so we're going to go to the first page get all the children walk all those get all their children walk all of those run all of our tests and we want make sure that there are no errors at all and so you can see actually we've hit some some errors in this page we expected not to see any errors and actually we did so now we can take a look and say you know what worthy the paths and the past actually corresponds to repro cases it's literally saying if you go through and you do this you will see an error page so here are all the different ones and we just want to look at the last one which will be the longest one probably and you can see that if we just do this and you can see that it involves clicking several times here we actually see an error page and then you can take that and you can give that to a developer or you can put that in you can save that for later for unit tests or whatever but now the computers actually generated a valid test for us to make sure that like that like we would not have written almost certainly and then you can do kind of cool things where we have two different classes of tests here I have some user tests where I want to make sure that every time a user is created that's there are that is valid that always has the properties that I expect and I want to run what I'm going to do is create 250 threads up here I'm going to run that test where we walk 11 steps down the graph and then create 250 more threads at the same time walk 11 down those so 250 times about 500 times we're generating the graph and walking 500 or 11 steps down and running assertions across that the entire time so we start with no assertions yet we've a no surgeons we have no failing tests and this is set up because we're spinning up a lot of stuff this is soft labs there are some significant infrastructure challenges to running all these tests so you can see that it's actually hitting our site and there are a lot of failures in this test and i really like up here in just a moment so you can see we start to have 286 assertions have been made so far we have lots of failing pet test cases we can actually use for repro later at some point here we go from 133 to 559 is this like we literally like break the scale it's supposed to be halfway I just love this yeah so I mean this is really cool so now we actually see how many sessions did we make we made actually post a 13,000 assertions on that when we have 116 cases that failed as we can actually take this and is our repro cases from now on all right so there are a lot of challenges in generative testing though so in particular figuring out the right properties to test so this requires a very different mindset because we're very good at finding examples of what we want to test this is what user stories are all about right where we have a specific story in our mind and what you actually need to do is kind of lift that up a level and you need to abstract that what are the general properties that we're actually looking for at any given time and this is actually it takes a lot of practice to get into that mindset and to internalize it another thing here is that reproducibility is critical so it doesn't matter if you have that repro step if every time you go through the site a bunch of state changes in your database and the repo case is different every single time you need to make sure that the app is very core reproducible and flakiness is a really big deal here it as we'll see a little bit later every time the test fails once and used to fail again in the future until you fix the bug but within a single run another thing is you get really long tests so you saw that let run pretty quickly I was fed up I think three times but it's still pretty quick but some of the tests can be thousands and thousands of steps long right we only went to 11 steps in there but we're blowing up a lot of browsers in order to do that and a lot of the times the thousand long step repro case a lot of those cases aren't or a lot of the steps are not relevant a lot of time that you click on the same element 10 times doesn't really change anything and so if you give that to a developer that's all times very frustrating because as a human being what do you do you start to kind of do this binary space partition right where you you take out at one and you see does the test still fail you take out another one and another one and you kind of try to shrink it but it turns out that the generative testing people have thought of this and there's this thing called shrinking which is pretty cool I'll come to in a second the other big challenge here is the extreme scale you are probably not going to be doing this on your laptop in order to run you know that the one that we I just showed will run all rights but in order to hit a million tests per per hour you're going to have to run probably 10 like dozens or hundreds of servers in your testing infrastructure and that's not just because of the test but actually because of shrinking so freaking super cool idea so we have this very very long failing test two thousand steps which is great right we found a bug we didn't have to do anything we found a bug but the bad news super long diagnosing this bug maybe longer than trying to go and figure out how to like repro it yourself I just I love this idea of like a user reporting this bug at some point right like just for driving like a 2,000 step thing and then trying to debug it but as a human being what you would do is you take out some of the steps and you would rerun it and you'd see if it still fails and you would see like what are the relevant pieces here and it turns out that's active that exactly what shrinking does and it just repeats it until it has a minimum repro case so it can take this 2000 step case it will turn into a seven-step case where it says here is the minimum like set of steps that I found from this long list that still produces the same error and so this is why reproducibility is incredibly important right because we need to know like if you have flakiness and the test fails for a different reason then shrinking is not going to be able to do any work at all it needs to be able to actually find the same bug by running it again and then kind of stepping back again to the infrastructure if you imagine you have that 2,000 step test that's a lot of subsequent tests that you didn't have to run to figure out what are the minimum number of relevant steps so I think for tests recording has this place but it's not in the browser it's kind of in the database so one of the nice things about this functional setup is because it's in immutable database and it has structural sharing all the way through we can take a snapshot at every step along the way so as we're running this kind of crazy number of tests over the database we keep a copy of every database that was generated and a share structure so it's very very efficient and then later on we can actually go and run tests over the state of the database so rather than running super slow test in the browser we can run very efficient tests in the database so when we need to run browser based tests we want to do page rendering and that kind of thing we can do that and then whenever we want to run specifically States based tests you can do that here yeah so kind of the question is like why do we care about the stuff like a million assertions per hour and the title it's it's it's pretty cool I think first of all but really it's about if you think about how deep you want to go in these tests right you want the computer be able to test very very deep and each one of those those paths the longer each the deeper you go the longer the session you're representing you also want them to be able to travel traverse over the breath right so you want to make sure that you're not just checking all right what happens if i log in and log out log in log out a million times right you also want to try some things like what if i log in and then i submit a book and then i do this you actually want to get a breath of coverage over as well and then once you find these these bugs like the computer that needs to be able to shrink it so there's a lot of assertions that happen in this so infrastructure is a challenge here but a tractable in our case yeah we just get so gender of testing has been used in lots of areas but mainly in memory kind of context nothing that's super slow nothing that's blocking because of the huge demands it takes so Circle C is what we can use to scale out our servers and the test infrastructure we use cell slabs scale outs on the browser side and there's some pretty cool future areas here as well so quick show hands anyone notice a problem like with the generative testing and I'm going to think of one case where it might not be great where it might miss some bugs anyone back there yes so there are cases where so Erlang has a system called simple check which is actually really good about this kind of thing about it's called like linearizing the test cases so if you see these kind of successive states that depend on one another then you can then assert that there must have been some linear way of doing it or else it's a failure so there are some kind of cases where this has been thought of but another kind of interesting one or technique that's based off of this is can call accessing so it's a combination of concrete test and symbolic execution and it's a terrible terrible name like I always feel childish saying it but it is a really really cool idea so oh you mean the second let me change the color on this one of the challenges or like one of the like big cases that's generative testing might miss would be something like this where let's say that we're going we have a function that takes a user and we want to decide if they're like a super user based off of how many internet points they have and we have three cases that if they have fewer than 42 they are not and if they have more than 42 they are and if they have exactly 42 then the whole app blows up so we have to generate a very very specific value in order to trigger that third case like we can run infinite number of tests more than 42 and fewer than 42 and never hits that 42 so we can actually use something where we bring in code analysis so we code analysis is something it's a bit like sack analysis where we will take a function and we'll examine all of the code paths that are in there and so in this case we see that there are three code paths and that they're guarded by three conditions and so what we do is we then back propagate that what are those conditions and what are they reliant on in this case their reliance on solely internet points it's on the input there so then we'll take the cases and will constrain what we can generate for the inputs to make sure that we actually exercise that branch so repeat for fewer than 42 for more than 42 and then finally repeat it for exactly 42 and so we can constrain it and we can make sure that we now have three classes of values that we can generate and we can make sure that we're actually execute every bit of our code without having to teach it how to execute our code at all unfortunately so this is really cool this is like cutting edge there's just a recent this is an academia still basically there was a recent implementation of a knurling however this is really hard to translate into for example selenium test it's still really unclear how to do that because what happens is whenever you hit a page and you load up a page or a part of your app it's unclear what code you're executing so there's a lot of research left to be done on like how do we know whenever we hit a page what's what codes were hitting and then how do we change the input for a page to make sure that we hit the right code paths another really cool thing is predictive testing so the idea is just like we did the we had that kind of declarative that simple data structure that was a repro case for us we actually record the actual users site our path to our site and then if an error is triggered we automatically email the developers without repro case and then another case thing we can do with it is we take the previous 10,000 sessions and this is something I did beforehand but we take 10,000 sessions or a rolling window of the previous 10,000 sessions and before we would push any new code we'd actually take that the sin thousand and replay them against the new code and so then we would have this idea before we push this code is this new code going to break anything that our users are actually doing yeah so just basically in summary humans are really terrible at annually enumerated all possible stage face and computers are really terrible at being creative about it so but they are very good if we teach them to find edge cases so we just have to give them a little bit of help and most important thing is don't write tests generate them so I just show you a quick example of predictive testing so this is a little demo chat app where I use this same structure we're going in we're making we're changing state we're updating things and then we're going to pull up a debugger and you can see we have the same number of states here and we can just kind of scroll back and forth you guys have seen this probably with things like I don't know if the ID does this but it's basically a time traveler right where we can go back and forth but if we have those state-based repro cases we can actually take the user's session we can load it up in our browser and we can actually scrub through it and see exactly what happened whenever an error is triggered so there's a lot of possibility if we start to go into this generative testing model which requires everything to be kind of data-driven so just real quick almost all the ideas here and the work and whatnot was done by sebastian is co-worker of mine super cool guy you should follow him on twitter my company is cool with me coming out here which it was a long flights they were super nice to let me do that so I want to thank them and yeah happy to take any credit or questions so irrespective of generating test but let me just so I don't forget Oh answer the first question first so just for testing so it's not about paralyzing tests right generative testing is not about bringing test times down in fact it uses a massive amount of infrastructure like it is required that you already have this stuff there what it is about is saying I have described the general way that my my application states works and now I have some properties that I believe are true here and I want you to try to disprove me so I'm going to have the computer actually search through that and try to prove me wrong and then whenever it does find something give me the minimum case that shows that so this in no way is going in fact like you need to have all the infrastructure because in order to run a million tests per hour like you need to have a lot of setup yeah so a second question well humans are definitely not doing a million tests per hour right yeah also I think nobody's so people do this but just not in selenium as of yet so because there are big infrastructure challenges around it's a lot easier to run a million so for example a really kind of classic use case of generative testing is let's say you have an algorithm that you're implementing and you're going to implement very slow version of it but it's very correct it's really easy to read right you can you can kind of look at me or make sure that it's right now you're gonna implement simultaneously a very optimized version of it and so you're going to say that for every input into the first function the output should match the the output of the second function and so now I can actually use the first one as a test case for my second one and this is very easy to do for in memory stuff right like there's almost no latency you could also use this for refactoring of a web page for example you could say that I have an old version of the web page we're implementing it a new language or whatever it's a terrible idea but say we were anywhere I go on the old pagemaster your page and you could say try to prove me wrong it just hasn't really been done in the selenium world because there are difficulties and how do you actually get the browser to traverse your site how do you make that efficient enough so you can run a million tests per hour kind of regularly and it's not really a big deal like I think a lot of us need to move in that direction but right now everything is and then the last bit is like everyone like the status quo is to write your tests manually or to like write specific example case or to record them there's very little thought over the general properties of our application so I think very very few to know people are doing this right now no so it doesn't replace in some ways it replaces so the way we always take it is so we have for example the predictive testing right so the first thing is we have everything our users have done beforehand we run that through our new code to make sure that we're not breaking anything that they they would want we then have jenner of testing that explores a bunch of this state space and then after that if something actually makes it past that and we get an error from user so we'll get a nice email with like the repro steps exactly right and say here's how you get this user's case in fact you can load it up in your browser you can scrub through it awesome tackle take that and generate an automated like unit tests based off of that so it kind of augments it so most of the leverage is going to be in writing the generative tests but you can take the specific examples that have gone like have made up through those gauntlets and then just use this to make sure that they don't happen again say this again oh yes definitely so often cases where the thing on what level you're working at right where maybe someone logged out but actually there was a bit of state that wasn't cleaned up and that was only a parent if he did a couple of other things to the site then you submitted something over here and like you know you you as humans we always imagined when we write to that log out code it's so simple there's no way there's a bug in this code right and then if there's no way that this a bug here would affect that thing over there and so that's where a case we're having the computer I should go through and say by the way I found this bug which this is basically the mirror of having a lot of users because if you have a lot of users this will happen they will do generative testing on your site basically users do crazy things where they'll log out to know go in and do a bug and you'll have a hard time debugging it oh sorry oh so we use closure and then we use selenium so the closure is a functional Lisp on the JVM and then we use the closure webdriver yeah I don't yeah I know so the code is upon on github as well I'm happy to talk about the specific implementation like it's not as relevant as the ideas but yeah we use closure on the JVM we use selenium webdriver so someone stop me if we're running out of time I don't I don't know yeah so it took quite a long time to kind of think about like what are the properties that actual want to start about my my code like but I think just after you do it for a while you get a good sense of it it does mean that you need to be a decent developer this is not a manual QA kind of thing you can take the manual QA test and try to figure out what are they actually trying like what what's the actual property that they're testing for because underneath each example there's actually a generalized property that you are trying to prove and so you kind of learn to suss out like what's what does this thing is actually doing here but really yeah so like I said one of the common ways is to compare to alternative implementations we're trying to do a new thing and it has to match up with the old one that's really critical but you you kind of whatever reason are kind of scared to touch the old code and so you just want to make sure that the new code matches the old code so that's a really nice way to start with it and you get a sense of like how all the pieces work and then you start to figure out all right what am I generalized properties that I now want to like that's that's entry level like and then you can go to the graduate level which is I'm going to now write my own properties so the unit level is a lot easier I think because it's it's generally going to be in memory you're not going to be bringing up in a browser and tearing it down or one not so you have a look like one of the killers for development productivity is the turnaround time right so if it takes me five minutes to test something I only get to test that a couple of times per hour and it's really really like that kills my flow if it's a unit test or integration test you can probably keep that turn like that turn cycle down into the milliseconds right we hit it into it runs a million tests or a thousand desk doesn't matter and give you an answer and I would actually encourage you to start there but I think from maybe your think about a philosophical direction like how to encourage developers to do this I think it just goes to like drawing off that states diagram was a big kind of mind-blowing thing for me when I started to realize actually like I'm just scratching the surface here and I don't want to spend all of my time like with these the examples and then once you actually get a couple of test cases and it's like going from no testing to testing right where you say actually there I did have a few bugs and I'm glad I have testing when you go to generative testing and it starts to find bugs that you didn't realize we're even possible and you think I'm really glad that someone did that then you start to feel like this warm blankie of security around you so it's just kind of an experience where you have to gradually get someone in and get them to have a few success points or they feel like all right I feel good with this and then once they they feel that sense I think becomes this obvious thing where you want to invest more time in yeah so like I said this is a thought experiment right so like and it's not we actually do this but this isn't a promise of like you can use this today in your existing code base so there are ways to do it where let's say you were to for example I don't know if Oracle has this but like if you had a mock in-memory database where I provided the same API this is kind of crazy I don't think it would be a good idea at the Oracle level but then you could keep a copy of every database so we used atomic and the atomic actually provides us out of the box where every transaction is actually kept in history and we can always record like we can zoom back to another one think of it like get when go back to a previous state and we can fork the database and use a clean state from there so we can always go back to any previous state and then that helps a lot with the reproducibility because we're pointing at that point so it depends so it's a really good question actually so we actually give every every test its own database but it's database in our case is just an in-memory data structure surgeon so so everyone gets their own environment to call it right their own kind of clean pure environment unless we are testing for concurrency and we want to know what happens when we have multiple users testing on it but that's a very explicit thing versus this like we'll just randomly throw everyone all together on the same database 11 more ok so we're already doing selenium for a bunch of other stuff right we're already driving silentium through the page and we're making assertions about the rendering of it and we're trying to mimic real-world behavior via the browser or by the mobile app and kind of while we're there we might as well make these assertions and oftentimes we do we won't use necessarily selenium to verify the status code code we'll just use that like we'll look up the request ID and we'll say what was the status of that from the server point of view so you know points the effort at the right right level I guess rights you sling for what is good for and then use the server and the backend and in-memory data structures for whether it's good for all right thank you very much