From Laptop to the World - Global Deployment at Your Fingertip with Kubernetes Federation

0 0

[Music] all right cool thank you thank you well thank you for being here unfortunately most of my talks are gonna be live coded and live demo and everything and I can only it seems like there's a little problem with HDMI so they have to put my laptop in the back so I don't know I I really not sure how this is gonna work because I can either run to the back and you can just watch the screen or you can see me talk here but there will be no live coding so I'm gonna let you decide on how this is going to work all right so see me in the back and see the screen okay you don't want me to be up here that's good okay that's good that's good I can go back and play a video and you wouldn't know that was here all right let's do this all right so we got sorry for the delay we were about two minutes short I'm gonna make it quick and my name is Ray I'm a developer advocate for the Google cloud platform what that means is I'm a engineer for a very long time for the past 15 years I've been in industry and now I'd like to bring some of the latest and greatest technology to developers of the world I also love to hear about your experiences in your feedback so the best way to do that is you know contact me afterwards or talk to me after this session or just find me on Twitter and just direct message me anybody can reach me if you have any questions I'd love to travel aside from technology I love I love to travel and take photos if you like to see some other the places I Bing check out the Flickr calm serene is it ok so just a very very quick show of hands how many people here heard about Crippin that he's already oh wow everyone's here ok ok very good very good so how many people are using it ok ok some some other people are already using it for the rest of you why not let's get in there ha ha I hope hopefully you'll see more of kubernetes and eventually you'll get to use it as well so it is a container management system a container orchestration system that was developed based on the Google experience in the past you know what we had internally which was called Borg it was designed to run containers I scale are multiple multiple machines thousands of machines right so cribben Eddy's now can manage up to two three thousand nodes that's two three thousand means in a single cluster and running your containerized workload in a it's 100% open source and you can run it in the cloud on google cloud or another cloud or um frame or whatever you want I run it on Raspberry Pi I was here last year in vanilla raspberry pi and that was really fun to do very quick recap of what and how it actually works behind the scenes you start up with a container image the container image is like this fusion of your environment in your application put together into this giant binary right it will be like a giant image file but it's going to be layered so you can actually transfer the different shows of much much quicker but the key here is that once you have this environment with your application code in it you can push it right into your centralized registry somewhere and then you can write some kind of configuration to say how you want to run this app in like as far as or which image you want to run and also how many instances of this application you want to run right in then how do you do the health checking but how do you handle the restarts and all that all of those information can be captured inside of the configuration file then you push it to the kubernetes master node this is like the master and all that controls the entire cluster and behind the scenes there is a scheduler a scheduler that will be waking up periodically very very quickly and say okay how many instances do I need to run and if I see that oh I need four instances and I have none then what is this going to do is to check with other machines to see who has the capacity to run it and if you have the capacity it's gonna go ahead and download the the image down to that machine and starting it ok this is so nice right this is something that you probably have seen before quite a lot it's very easy I'm not sure why the thighs Fasching on me but but it is very easy to do right so now you can provision a cluster you can you know just every note would look exactly the same you don't have to worry about environments and how each machine's environment could be different anymore because oh the environment is now captured in your container image and you can just say go ahead and keep CTO run or keeps if you apply with the command line and you can just deploy this app so the key here really for me is that with this type of system right where we no longer really have to deal with the individual machines anymore basically every line of code that I right now is directly contributing to the application and when you do that is basically saying that every line of code that you write you can much much quickly affect the business of value or whatever that you're trying to work on rather than managing that infrastructure like we have done before I had a start-up before where I only have three machines I had a start-up with only three VMs and it took a lot of time out by own time just to configure it and at the end of the day nobody knows how everything runs because only I know how I set it up even if I documented my other two teammates in the startup sphere wouldn't be able to get it so with the service system were able to actually just make sure everything looks exactly the same people understand how to deploy and run their app and we can deploy something much much faster now the very first thing though is how do you actually get started on a local environment so I'm gonna do this very quickly I'm going to run to the back alright so the very first thing that you need to do with the let me put this down yeah very first thing you need to do to get started on that the local environment is by using something called the mini cube for example mini clip allows you to start them in urban areas cluster or communities node locally on your machine I think I really have something running here so let me just make sure this is big so I can't see it myself so I'm really meeting cube installed and to start basically I can just say mini queue start and I think I already started some glass a mini cube studies I'm gonna see if it's really rock up and running great what that actually means is that it's running kubernetes behind the scenes in a VM now we meanie cube we also have docker installed as well so rather than using your own separate dr. demon what you can do is to use the mini cubes dr. Thielen as well and to get that information in hence a mini cube taka in and that will actually give you all of the configuration you need to set in the shell and the easiest way to set this shell is by using eval so once I even evaluated these these output basically what it does is that it exported all of this environmental variable and now whenever do attacker PS I'm actually going into mini cubes darker demon and what is really good is that now whenever I go ahead and build an image it's actually going to build directly into a mini cube so what I also means is that I don't have to push my image out separately into a registry first in order to test it now I can actually test everything just locally I mean if you first I can you know build images as as quickly as I want and and redeploy it as quickly as I want I don't have to push it out separately until I'm ready okay so here I have actually a couple of applications I can deploy a couple of different services I'm going to do right now is going to do a front end and back end okay so I'm going to do the back in first and by the way this is a go app so the all these things are written go for this particular demo and I can I have a docker file here that describes the environment I want to do I want to have but basically I'm using a doctor imbued so what that means this is going to copy this go file into the container buuuut environment it's going to actually view this go file into the binary and then putting everything into a container in one shot okay so I'm gonna go out and beauty I think I have this cute up somewhere some glass a dark beer - tea and I can do this back in continue oh that was too fast that was too fast alright so when something goes too fast you just never know it's going toward right but this is actually building directly into my darker demon that is running in mini cube as well and then if I go to the front end I can also build the front end so I'm going to do that and maybe maybe you have another company in line here and there we go I just spewed the front Ian everything is local but the reason why so fast is because I built it before and it's kind of like cash so far okay and then what I can do is a mini coop it actually creates a context for you right what is the context right a context is just a way for you to connect to the kubernetes cluster so in this case I'm going to go ahead and see the config I can get context this and as you can see I have many many many clusters running that I can connect to I got so many right here but one of the the cluster that they created by default is actually the mini tube context so what I'm going to do is I can just use it so I'm going to say cube CTO config use context mini cube okay and this allows me to switch directly into the mini cube context and now whenever I say get pods I'm actually using the mini cube the mini cube kubernetes instance on the VM that's running locally on my machine I can also start for example proxy I'm gonna say start the mini few proxy sorry and then let's go in you see the visualization in this video ization tool unless we've seen that I have a cluster I have a cluster but it only has one node okay so how do we actually deploy this app well for development purposes I like to use just keep CTO run so for example I can go ahead and to keep city or run I want I can run the back end so I just give you the name I give it the the image I want to run and let's go ahead and run it there you go running mini cube running inside of mini tube right now and I can also run the front end as well there you go and I should have started in front in all of these things are now running locally on my machine and I can actually go and test one of these things so for example a white if I want to test to use the back end there are a few things I can do right for example I can say get parts and I know what the back end is and I can connect to it and they're multiple way for you to do this if you just want to testings out you can open this port forward so what that means is I can open a tunnel directly from my local machine and I can forward this into port forward there we go I can connect directly into this part or this in application instance hopefully I can say nine thousand there we go and then I can say API next and that's hitting the back end and basically what this API does is to return a random string every time I run it and it's saying stay calm don't panic which is something I need to do right now all right so we can do that but then we can also expose a low balance that we can expose services so for example I can create a service in kubernetes service is the way for you to create a load balancer in front so you can load balance across multiple instances for example so here I can go ahead and create a load balancer for the front-end it's gonna be listening on port 80 and it's going to go to the target port 80 as well go ahead and do that okay and I can also expose the the backend as well so let me see if I can pull that up so I got both the front end and the back end exposed now okay now this is cool because if I say keep CTO again this VC I can see that I got two services running but they're all having a cluster IP what that means is that these things are only accessible from within the cluster at this very moment but because I actually could post it I say a node port what I means is that if I want to access it I can actually access it from the mini cubes IP and go to the node port which is one of the ports that's exposed on the mini tube IP and the port will then forward to the report of the container this is the low balanced port there is an easy way for you to get to this port which is to use a mini cube community service and I can say let me open up the button masher back-end okay and hopefully that's going to pop this open yeah so if you look closely you actually get the IP for myself auda matically and then also determine that this is a no point I need to connect to this node on this specific port now to support is actually dynamically assigned you can assign a static port if you want to but to avoid conflicts is usually dynamically assigned and I can of course go into the API and I should be able to see everything here ok so that is how I can actually do some of the testing but then I need to test the front-end so I'm gonna go ahead and open the frontier as well front and ok so in the frontier I can see that I have the UI and there at this button I can click oh shoot huh there's nothing it's not actually calling the API I wonder what is happening let me open this up and if I go to the network and do a refresh let me see ah aha I see the issue here the problem here is that I'm actually expecting the API to be accessible from a relative directory right this is a relative URL so what that means is I'm actually expecting that API next is available underneath that the same domain but it's not there right now because I'm running the two services separately right because if you look back to this this view I have the service here I got a service error they have different IP addresses so I cannot really get to it the relative path and in kubernetes there's actually a really nice construct which is called ingress now typically what happens in this case if you're using a regular stack for example if you're using spring or spring boot you you probably need to run what we call the API gateway right something that front to all of the services it responds to a proxy directory or URL and then enforce that they the request to the actual back-end incriminates we can actually create the ingress this is going to create a real ill server load balancer it's gonna be your HTTP load balancer and for each of the different URLs you can actually map it to the actual service pack and that needs to respond to these requests okay so in mini cube I can also deploy this kind of load balancer as well but before you do it I have to enable the add-ons Ameena cube there are many many different atoms you can have for example cube DN eyes hipster ingress atom manager dashboard and such but for you to enable the add-on like ingress all you have to do is to say add-ons enable ingress and what that's going to do is to deploy a local version of engine acts that can act and listen to these ingress resources and creating these increased load balancers now I already have it enabled what I mean is that I can go ahead and say apply F and I can apply this invoice file and I should create the ingress for me and if I'm lucky today I can say getting dressed I cannot type from back here I it's going to expose the IP address and this will take like a few seconds to provision this is because he needs to sync up what will sync up on the honest on the the back end but once it is provisioned I should be able to go there and now I can actually access my app and because I proxied the API next URL so now I can also go to this ingress IP and I can go directly into this directory okay into this URL all right very good so now I'm running everything locally I'm testing everything out locally what do I need to do to deploy it into someone else's computer in this case into you into the cloud well first of all you can just create another company's cluster I have already one created and that is there's a five level cluster so I'm going to bring that up and for me to switch to a different cluster if I want to have this as a default setting I wanted to do is to again use the config I'm going to use the context and I can use my demo cluster for example but before I do I need to kind of export my configurations so what I can do is I can still keep CTO geared deployments for example and I got two different deployments here I got the back end and the front end so I can actually just get the deployment and I'm going to output as a demo file and in the X Pro Mode if I do this I can't just a pipe it oh here we go get deployments export and oh yeah mo so I can get the mo file I'm going to into my deployments IMO okay and similarly I'm going to also export my services so I'm going to get my services I'm going to get the packing service and the front end service and I'm going to do the export I'm going to om oh and then I can I'll put it into services um oh okay very good so if I open this out right it should have all of the configuration that I have done previously Oh export it and this is a really easy way to move from you know your development environment potentially into the cloud worrying into a real environment it's by bootstrapping some of these demo files because these are the configurations you need to write you can write it by hand or in my case I just like to generate it and once it's generated I can clean this up a little bit right I can remove some of these things I don't need specifically that I don't want to have and in the cloud sometimes I don't want the note port to be dynamically assigned so I'm just going to create the note port I'm going to assign the note port explicitly in this case and let me see here what else do I need to change the cluster IP ends removes that right and no poor I'm going to use a three thirty thousand and eighty in this case okay so I can do all my changes here to make sure that this is something I can deploy into the cloud by in most cases you don't really have to change anything I just decided to make the note port static just so I can expose it across multiple clusters later okay I'm going to do that and now I can switch my context to not mean EQ I'm going to go to demo okay and if I reset my visualization tool now if I go back to the visualizer I am in the cloud now I'm in a kubernetes cluster that has five nodes rather than just one and how would this actually work well I exported the yellow file the configuration file from my local environment in this case I did all my changes in settings so that I know that this is the the configuration I wanna run in multi know a cluster which you know I didn't really change much but having out done I can just go ahead and fly F I am going to say deploy the deployment and then deploy the services mo and I can deploy both and hopefully that's gonna get created and deployed into the right environment there you go okay so now I really just took whatever I was working on on my local laptop right in mini cube environment and I just exported all the settings and now I'm actually running in a multi known environment without changing which is about anything in this case right I can also go ahead and create ingress and and expose everything via the public IP everything would just work exactly the same now that will get you to the cloud that would just be now it's not bad right now you have potentially a production environment but some of the businesses actually have to deal with global traffic right how do we actually do that okay so what we have done so far is to basically take a configuration we can deploy it into a cluster of machines write effectively what we're seeing is that we're looking at the entire one data center or a pool of resources as a one view we're seeing rather than just individual machines we're seeing you know in aggregate how much CPU and memory - do I have individual machines doesn't really stand out anymore it's about the cluster so the next step is to actually you know take it a step further and say well what happens if we take the same abstractions and we say you know what I don't really care about which data center anymore all I want to do is to deploy this and deploy this globally across the world and to do that you have to deal with multiple multiple data centers original data centers for example or within multiple zones so what we have done so far is that we are able to control servers were machines through a single control point via you know via the kubernetes master and the master node exposes the API you can use the client library you can use the user interface to interact with the master node and then subsequently that affects every single machine that's under its control if we take this even a step then we can also create a control plan that we'll be able to control not the machines but control individual clusters okay because each cluster will expose the API endpoint and we control the cluster via the API so if we have another abstraction on top of that then we can create another control plan that will operate against each and each one of these clusters via the API as well so now you can deploy true binaries potentially multiple unframed clusters you can deploy kubernetes on the cloud and another web service is called I don't know what that stands for but lets you suppose you deploy down through two different cloud platforms it doesn't matter where you deploy it they all expose the API and through the control plan we can actually control the deployment across different different clusters and in many cases this is used in order to deploy this across multiple regions okay and how would that actually work well first we need to install this control plan and that is called cube that is the Federation control plan and out of all the clusters that you may want to further away in this in this view we have the u.s. east u.s. central data center in Europe in a data center in Asia right out of these four clusters we actually have to pick one of them as the host cluster the host culture is actually the one that runs the control point for the Federation because you have to run the Federation API in the Federation controller somewhere and you have to pick one of the clusters to run it now what I have actually created is I created a simple project let me see if I can open it so I can so console cloud account I created a project called cube thread world I'm gonna zoom in here a little bit so there's a the project I created in Google calling this case that's where I'm running in my clusters and if I go to continue engine in this case is the manage container service we have I actually have two clusters each cluster has three nodes in each cluster is running in a different zone so I got a cluster running in Europe right around here around Belgium area and got another cluster in United States in the central okay so I got two clusters running right now I should be able to open up the visualization around them as well so I'm going to do exactly that so I'm gonna open the visualizer so that visualizer goes to let me see here so the first of you guys are the one in the bottom goes to Europe okay and then I need to get another visualization up which is this one and this visualization tool on the top it's gonna go to US central ok so the top one is going to the US the bottom visualization is going to to Europe right now okay so how do I choose Federation to to cannot deploy pretty much the same application across multiple clusters well first I need to install the Federation control point okay now it is a little bit more involved so what I'm going to do is to use a little note here okay so I'm gonna I never really done this before you know using a lot of notes but for Federation I kind of I kind of need to do that okay so let me see here a cat step Federation so there are a few things I need to do first of all is I need to initialize this Federation control plant now in order for everything to work you actually need to have a real DNS name okay this is you know for testing purposes you can have something that doesn't really exist but that only goes as far as being able to control the deployment across multiple cluster later in this talk I'm gonna talk a little bit about cluster federated service this will be a federated service which allows you to do cross cluster discovery and if you want to do a cross cluster service discovery then you have to have a real working to me and and this is actually working right now so I actually registered this domain name okay so first I'm going to go ahead and initialize this Federation so I'm going to go ahead and deploy this for the Federation control plan what this come and do this cube fed command is pretty new they actually make the installation of Federation a lot easier what this is doing right now is to connect to the host cluster which is the US central cluster which is right the central cluster with three nodes and it's going to install the control plan into the Federation namespace so it's going to create a new Federation system in namespace and it's going to install the API server is going to install the control Federation controller and also create some basically configurations so that it can interact with my DNS now in order to interact with the DNS we actually have to give it the provider like who actually control setting us you actually need some kind of plugging for this so right now there is a support for Google's cloud in class and the the route 53 and though so that if you run this on frame you can probably look into core OS is coordinates so you can run coordinates locally in an unfree and environment now the dns is important because whenever that we create a new service from the sound forward if the service has an external IP address we can use the DNS entries to kind of point through which IP address they should be running they should be connecting to okay so he actually uses the DNS API is to update DNS entries whenever you deploy new applications into this Federation okay so it took a little time but now we have the control plane installed on my cluster to look at it I can say let me switch the contacts rather than switching the default contact missing I'll go and use the context for this specifically I'm going to specify it I'm going to say this is a cube fed us Central and can get my namespaces and I should be able to see now a federation systems namespace is created and I can get the part so I can say namespace is equal to Federation - system and we should be able to see that we have two to pass running one on this API the other wise controller manager okay so now we can actually join up into this Federation so I'm going to go back to this slide a little bit for each of the cluster you want to have participating in the Federation you have to you know explicitly join and by joining I mean that you actually have to pass the credentials that is necessary to connect to the cluster into the Federation control point okay so every of the clusters will have a context the context will point to my credentials so what I need to do then is to make sure that my credentials is actually registered with the country the Federation control plan and I can do that by using the cuvette coming online as well so in this case I can say cuvette I need to join into a federation and if you join my central cluster into this Federation so I can just do that and going to copy and paste it okay so literally what happened before the behind the scenes is I took my US central credential they actually give it to my Federation server and also registered it in the API server as well and can do very similar things for my u.s. West now hold on a second I think I saw a little problem here because the papapapa that is definitely not the right name of the context right so it's actually cube fed - EU - West I think that should be right okay so I'm going to register my new Federation EU cluster as well now once I've done that I should be able to see the status of whether these clusters are now being recognized by the Federation so what I can do is I can use my Federation context I can get clusters and as you see two clusters being registered now only one is ready the other one is unknown so actually have to wait for both of them to be ready and now now this is cool now I have two clusters one in the Europe one in the US and I can control everything on my deployment directly from these two directly from this Federation okay so let's do an example I'm gonna go back here and talk to the world remember we exported the service we exported the deployment I was able to deploy those two things into a single cluster I can now deploy literally the exact same thing into the Federation as well the only thing I may want to change is the load balancer the type I'm going to change to blow balancer in this case the only reason for this is because I want to show you a cross cluster service discovery okay in order to do that I actually have to expose my services via an external IP and the way to do that in companies is by exposing it as a type of load balancer okay so I'm gonna do that and now I can say rather than deploying into my demo cluster I can now deploy it into my Federation so if everything goes wrong I should be able to say contacts well oh shoot that's my that's my auto completion when I say apply that chef I'm going to apply services and then let's see how it works crossing my fingers I'm gonna move this to this view so you can see it okay there we go oh yeah of course so what happened is that in the Federation namespace it doesn't really understand the default that there's a namespace called default so actually have to create it so I'm going to say create default namespace yeah okay and once I'm done that the Federation server now understands all the clusters will have to have a default namespace now I can go ahead and create the service and if I'm lucky today it's not bad you can actually see that the service is actually provisioned into both clusters okay and they all have a different IP of course because there are physically different clusters right but it now is actually provisioned and I did in one shot via the Federation now if I go back to here and say guess service using the Federation context I can see that we're still waiting for the external IP addresses to provision some way for that a little bit but in the meantime what I can do is I can actually go and check out what happened in the DNS so if I go to cloudiness into networking okay so I can go into networking and go to cloudiness I have here a zone that I created for my subdomain a demo cube that world I can go in there and you can actually see that now the end trees for these services are actually being created I have a back-end and I should also see a front-end shortly as well but these are going to be just created automatically but what's interesting is that there is actually no IP addresses associated with these dns names that is because I don't have any backends running oh I did is I created the low balancer I just created that the low balancing eye services behind the scenes but there is actually no back-end application that's running okay so Federation does something really smart if you deployed the back-end into one of the region's one of the clusters then is going to recognize that this cluster can now serve this application because now I have the backend that is going to automatically register IP address into the DNS name as well and that was something you can actually entrust the server discovery to the DNS server right because you can say let me access button as a back-end in my Federation and this why should it be updated with the IP addresses that actually serves the actual requests okay so let's see if all my services can back up with this IP addresses yes I have done that okay very good I got both the front in the back in and if I do a little refresh as you see up front him back in as well yeah good so I got fronting and back in alright so let's go ahead and deploy so I had this deployment right here right it's exactly the same file I haven't cheated or anything it's exactly the same file so I'm going to say keep CTO context is equal to Federation I'm going to apply this deployment this this auto-completion thing is not liking the equal sign all right F I'm going to say deploy this all right so let's see let's see how it works today all right so I did the deployment and into my Federation and hopefully very quickly soon enough I should be able to see oh there we go not bad I check this out so now I have my back hand deployed in the top so that's the u.s. cluster but it's not in the front end is not there yet and that's fine and then I have the backhand oh I have the front end deployed in a you it's running one instance but there is no back end that's only because I only only running one instance of each things right so I can actually scale so I can also say context Federation I can scale my deployment I can scale my button masher back end I can say you know replicas equal to say 4 or a to 4 let's do 4 ok and let me scale my front end maybe the front end doesn't need as much processing so I'm going to say scaled out to 2 so again without having to go to each of the clusters individually I'm now doing this via the basically the other Federation and you can see now that the Federation server actually figured out oh I got to round 4 I need to have some kind of balance between the two clusters and so rather than running all of them on a single cluster I'm going to balance them out so now I'm running two instances in the US and running two instances in the EU now on the other side for the front-end I'm running two instances so now I have one instance of each in each of the cluster right you can actually configure the weight as well because now every cluster has the same size some clusters could be less powerful than the other so you can actually configure this through the the annotations as well you can actually say that hey you know the cluster is bigger so I'm going to put more parts there I'm going to run more instances here you can control that directly through the annotation right and I can deploy everything else I can deport the ingress and everything else I have another one that's already deploys I'm gonna go there right now and you can you can also go to this URL okay this is called the hcaaf style okay this is the same application I just have it deployed earlier you'll be kind of nice if you can go and check it out okay and in this deployment if you actually open up the URL if you press the buttons a little bit okay this is fun no one's pricing either I don't know why but if you price I should be able to see the QPS coming - okay so you can go to HDR ki sio if you press the button we'll be able to see your QP s okay and yes everybody's clicking now that's Allison that's good alright keep on clicking do not stop I'm skating but but you can see that all the traffic right now is actually going to somebody scripted it so I think somebody scripted this - to try to details my my back in but keep on doing that this gonna be over shortly alright but as you can see all of the traffic is going to just the u.s. cluster at this moment I have three clusters but all of them is going to the US that is only because I have one version one instance running okay what I can do is let me get back to my shell okay what I can do is since this the Federation I can go ahead and scale it so I'm going to say cube CTO scale my deployment I think it's called the hollow world uh sorry not the hollow world sorry button masher back-end I'm going to spin this up to six okay so I'm going to do that scale actually know what before I do that I'm going to go ahead and do you a deployment of the load balancer sorry at the low tester okay so what I can do here is I can't deploy you can use federated deployment and I can use what we call the federated name discovery what service discovery and I can just use a DNS name ok this DNS name is pretty smart what it's going to do is to resolve to the nearest services that's actually serving so nearest cluster that has the service serving so holla means is if I'm referencing to this DNS name because I only have I only have the application running in the US this is going to resolve to the United States IP address okay so I'm going to go ahead and deploy this Federation cluster applying the chef loaders deployment demo okay so what this is going to do is to spin out three instances of Milo bot each of the lowball is going to January about 1000 QPS and once this is up and running I should be able to see 3050 s being being there we go being opened up and yeah so whoever is doing the clicking I'm sorry I don't see your clicks anymore but but now we are seeing a few thousand GPS now here is the the most beautiful part because it's using cross cluster service discovery what that means again is if I'm actually having packings running Asia running the EU issues actually find its local services issue rather than coming across the the ocean to the US that should actually stick within their own data center so I can simulate that very quickly so what i can do is i can go ahead and scale my deployment this would be done in a second okay scale my deployment and i'm going to up my button my sure back end i'm going to scale this up to six instances let's do that now watch carefully watch carefully because it's gonna be done very very quickly all right so this clusters are synchronizing up so it's probably going to run two instances of each in each of the the clusters there we go six instances now watch the Lobot the low bada was using a federated you know sir discovery mechanism so as soon as there's something running locally they actually split the traffic very quickly right because now the service discovery is saying yeah you know what rather than going across the ocean just go to your own so we can see that the traffic for each of these things just split and now they are all going to their local data center which is pretty awesome right now last thing I'm going to mention is that behind the scenes I also set up by Ingres and this is pretty cool so rather than using Network load balancer I can actually create again the l7 load balancer which is what we call the ingress now with the ingress it's actually depending on where you are about on Google cloud the ingress will be created which is one single IP address okay one single IP so I can show you what that IP is do this quickly get deployments sorry get ingress okay and that's going to have an IP address here right that is actually for your heating with this URL so rather than going into the the local you know network discovery with a network load balancer I can use this ingress to and I'm going to go ahead and edit my deployment of my load BOTS I'm going to use the context Federation cluster okay keep CTO and this will be done very quickly and rather than going into my local service discovery I'm going to use HDR que esta IO and the port I'm going to use port 8080 again this is the federal deployment so it's going to update on Milo bot to use rather than the internal load balancer is going to using external one now through the ingress okay and that should be done okay so now all the traffic is coming through through the ingress now here's the beauty the beautiful part you can still see that each of the the clusters are still receiving approximately the same load even though I'm going through the same external load balancer oh there we go so now it's a little bit different okay now check this out it's gone Asia is now zero GPS we have 2000 you and one thousand US the reason for that is because through the ingress currently my low body's just running in Asia ah now I got two instances of the Lobos running in Asia and so the ingress even though it's a single IP he actually routes the traffic also to the nearest servers as well okay so this is one of the ways for you to literally deploy multi reaching deployments and you know having a global load balancer set up and now you can route the traffic depending who you are you can actually draw the traffic to the nearest data centers without having to do a lot of work okay so I know my time is up I really appreciate you uh you know attending and also apologize I was in the front but if you have any questions or if you want to see some of these things working you can actually try a lot of these things I just want to put up the last slide here so you can find out where you can find the information okay so all of the code is online you can find them on on my github you can set up mini cube just downloaded on your local machine and you can try the Federation now in my collab as well you can open the document and you can go to the collab so thank you very thank you so much for your time and again apologies for starting late so thank you Cheers