Erlang in multi-lingual systems

0 0

so yes I'm Robert willing I was originally a physicist way way way back when yeah it was a lot of fun until I discovered programming and then I wasn't wasn't a physicist anymore okay yeah but um I started working for Ericsson and eventually end up in their computer science lab will we did a lot of interesting things amongst other things we developed a like a number of us left around 1999 to form a company called blue tail which was one of the first companies outside ericsson to build products using Aling Aling had been open sourced in 98 this was in the middle of the IT boom and it was a lot of fun and really weird in many ways then the IT boom ended we'd been brought up by nortel networks and they started cutting things amongst other things cutting us ah and me and I started working for Swedish military procurement yeah that's I thought it was very strange as well to zoe's I had a lot of fun we didn't do any programming military procurement doesn't do things that buys things but I i had the pleasure of working with computer games in the military so it wasn't too bad then I four and a half years ago I started working fouling solutions I came back into the a line fold officially some of the things i'll be talking about here are things I've done during that time so the thing here is that yeah they say principle language expert it depends which card I have what I am right so I don't really know anyways so I in multilingual systems that's the whole I do it here and I think the start off is just want to say that we in the owling community especially a long developers with very great believers in things like no language is good at everything okay this is not saying you can't program everything in any language you can write but most languages are not good at everything yes you can do all this stuff and see if anyone's ever tried it you only do once and if you can do it and see of course you can do an assembler I mean why bother going up when I can do things in assembly and also the right tool for the right for the job so there there are a large number of tools out there they all tools or languages or systems or whatever you want to call them and they good at doing different things so as you use the one if you have a specific set of things you want to do pick the language that's best for that and what you find generally find is that any large system different parts of the system will have different requirements so it's not always logical or sensible to try and program everything in one language that can sometimes be conceptually difficult to convince people it's not always easy but I very much very strongly believe in this so we're going to look at a bit about our case now we can look at programming building using our line the a line system for doing different language with different languages and why you'd want to do that so we're going at a number of couple going upwards from the base system here so we have the beam which is the a-line virtual machine will look at some of the properties of that then we'll look at different levels of languages so our wood good names for like all native languages they're running on the beam and they're using the beam as the beams intended to be used and non native languages they're running on the beam but they're not doing what they're supposed to be doing and then you've got external systems and hopefully the end a little bit of discussion when you use them ok so so what is the beam first question here so it's the it's the name of the a-line virtual machine and what it is it's a virtual machine to run ali really that that's exactly what it is no more or less and like other virtual machines is designed for running something something specific I mean if you take the JVM the JVM design for java then it depends what were you coming on top of it within that base anyway so what what's what's the basis of the beam of the a line system so what are the properties that beam is trying to give you right and these the first slide here these are things about the a lanx system and all these things have direct support in the beam so none of these things are built on top of it so I mean if you're looking at our line you'll see a lot of things look like library calls but all this stuff is built into the machine at the very low level so there's support for lightweight massive concurrency their support face synchronous communication there's process isolation the error handling primitives they're all built into the Machine and directly supported in the machine continuous evolution the system which in this case means being able to do hot coal oding other people mentioned that before that again that primitive so that's built into the machine the soft real-time nature again that was somewhat that was a requirement we had that's built primitives or the basis for that's built into the Machine and support for introspection and monitoring so the things that should say mentioned with the observer the primitives for doing that they're built into the machine so there are lot of their support for introspecting into a running system which something was something very interesting for us because we would have work on these type of systems where you could not take the system down to look at it so you want to be able to look at a running system or in production and just introspect to see is it behaving strangely is this process behaving strange things like this for now most of those things are sold and what if you're going to implement another language on top of it are directly what you really wouldn't worry about except for receiving messages because we use pattern matching to select which messages you want to receive and pattern matching sort of more up at the language level right then you've got things which are more two more at the language level and again all this stuff is built into the machine support for doing this so yeah we have a mutable data of course we also direct support in the machine for doing pattern matching of course it's a functional language with support for that so the top three are just sort of basic things you'll find in functional languages nothing strange we have a predefined set of data types so one thing you cannot do is define your own data type the system machine does just doesn't support it it has its own concept of code of modules and how they work and what they can do and what they're supposed to do you cannot get around that another one there is no global data in the system at all if you there there's just the thing to get around after whilst comes extremely natural there's no problem with it but the same thing same thing with all these and these are the type of properties if you're implementing a language on top of the beam like our line is what you will exactly see so what happens if you do want to add a new data type so our line has something we call records it's just the structure with tact with name fields but we can't do it we have to fake it because there is no support in the a line machine for implement adding new data types so we fake it we use an existing data type of topple and we have a special format and we say that the first element of topple is name of this record then you've got the field straight in there so this is the way we do the new type we're just faking on top the old time talking about languages here which languages in this case basically anything they can be config files they can be yourselves they can be real and languages anything doesn't really matter what we're talking about here there is in the airline system or the o2 billing OTP there is a set of basic tools you can use to help you write languages this is nothing strange i mean what you'll find that most other languages and systems as well there's a lexical scanner generator it's call leaks it just happens to look very close to lex there's a yekke a parser generator and lalr 1 parser generator which pretty much does the same things yak but for the alling stuff and there are there are other tools for building and working with code as XML parser a couple of them and of course there's the compiler for doing a lot as well so the compiler has been designed in such a way you can you can input things and get things out of the compiler at different stages you don't have to put a file in and get a file out you can put code in at various levels and work things happen so this is quite nice so what I plan out looking first what I call the native languages right and what you end up is basically the new skins for an all ceremony if nothing else don't get anything else out of this talk and you hadn't heard that record by then it can't go listen to it it's great it's fantastic so yes so what what what what this means here is that if you're going to implement a language sort of natively using the basic properties of the beam you're going to end end up with the same type of execution at the lowest level the same type of execution as you get nailing because that is what the beam is designed to implement so the virtual machine the language go very much together here now you can put new syntax on top of it you can do different packaging but at the bottom level this is what you get and there is just no way around that in that sense the virtual machine is very unforgiving and all of the facts I mentioned the property as I mentioned they're all very much interlaced with each other so going in and changing something is really difficult to do because you'll affect everything else for example you might say well why can't I do mutable data I mean it can't be too difficult to put a fix in there to fix mutable data then you'll effect all the garbage collectors you will affect other properties of the process management and stuff like this as well too so all these things start it will affect each other once you start fiddling with one thing it spreads so I'm just going to mention two languages here that are at this level one is alexia which shows a was talking about this morning and one I call that a fee which is play with a line and both of these work in sort of this level there are skin on top of the existing system you have the hour-long virtual machine the beam at the bottom you have our line you have OTP which is which is a set of libraries for our set a design patterns how you use the language to build systems things like this they're all there and then you put your new language your new thing on top of that so you have your new language you'll have a new set of libraries you'll have whatever you might be but the base level you'll still have the same thing and yeah that's why the basic properties lender language you end up being that and you make full use of what you have there I mean there's been a lot of work into libraries to make them usable for building reliable systems and again I can also say getting back to the properties of the system and ander the language when we were designing that the language we're also also thinking about how you would actually use to build a system so the features aren't in there just because we thought they were fun that we're there because we've thought we felt that with those features you could build reliable systems on top of the language we're using the language so that it's very much integrated with what's in the language and how you intend to use the language okay so the first one I was going to mention is LFE which is my baby and the goal here was just to provide a lisp with a with all the list goodies you used to having right so truly home iconic has has a real macros I mean Lisp macros yeah and it's seamless interacts with vanilla Lang an OTP so i can use everything in there you can use it together with that with OTP so you can write modules in our life in LA for you can have modules now lying it you interact with them everything like this as many lists it has a small base core language most of the lot of things you see a macros and it's seeing is directly going down to our line it's the same speed as writing things a telling you know you're not paying anything extra for doing this that meant you had it couldn't be any of the existing lisps you cannot be cannot be common lisp or can't be skiing because they just don't have the properties or have other properties which yelling machine doesn't support and the goal was not to extend the machine so there's no mutable data for example we only have the standard data types we have records but that's nothing special there we have the we don't have say common list packages namespace packages because they they just don't work in our line we have the a-line modules instead macros are at compile time because the machine only sports a compiled code so you can't have macros there it's a lisp to enlist instead of all this one I'm not going to go in turn to you and again it's as fast as it the other major alternative on directly implement on top as a native languages is of course Alexia and the quote from the home page it's a dynamic functional language designed for building scalable and maintainable applications that's that were designed for it's for those who have seen it all those haven't seen it it's the influenced by Ruby but it's definitely not Ruby on the ellen video it has made meta programming capabilities by macros they can't really play with the syntax but otherwise have a pretty good macro package there and it has a lot of libraries and interfaces which have been standardized and rewritten so if you want to compare it to it to LFE it's a thicker skin than LFA is for ya it's also been influenced I think quite lately quite a bit by closure as well to its getting closure type properties in there as well yeah so for example with it with the data types okay it has records but I don't think they're really very much used their main i'm guessing just to interface airline necessary but it has a concept called the struct but i've just said you can't define in your data type structs they're built on top of maps and you have a special key the underscore underscore struct underscore underscore key which gives you name with the struct and that tells the tells the alexia system that this is not just a normal map it's a struct and said and gives you a set of operations for working on structs amongst other things you can get compile time checking of the keys you're using which is one very nice feature one reason why you want it yeah so so that's that's that's a fe and alexia and both have this problem that whatever you want whenever you want to make it look like at the bottom level you have this execution model then you continue on top of that you can put nice things to make it give you the properties you want on the features you want you can add new libraries you can add other features for you can of course you do all your own packaging as well that doesn't affect so yeah so the next level I was going up is okay with language is still on the on the ailing machine but I'm these the ones are call not a native languages because we're now when we're going to go beyond what the ailing machine directly gives us an dad you've add new functionality at new semantics and new data types planning for so there are different semantics different out there and not definitely not our line data types and non our lang handling and data so here we're here we're going to say yes we have these languages and they have these properties and we're going to implement these properties and these are two two options i have i've done myself there's one color log which is a prologue implementation and is loyal which is a lure implementation running on top of the yelling machine and the ER log it's a standard prologue you can go look up it was a subset of standard prologues if you go look under standard prologue booking cooking connected directly to a login and you can see everything like this but of course this has completely different semantic smelling ok thanks functional prologues logic and has completely different semantics so we have backtracking we have logical variables we have unification and always things but there is no direct support at all for this in a line so this is something you have to implement in your langdon your language implementation on top of our line to get the language to work fortunately here we have a pretty good mapping between our line and the log data types except for logical variables because they're sort of destructive but they're not and they do strange things but so here what we've done is we've kept basically kept most of our line data so the data at data handling is pretty straightforward as in our lane but we've put another semantics on top of the we have to implement that there are a number of ways you can do this or I if anyone was talk about this later but I hadn't planned to go through now but so here we're changing the semantics of the language and it's a decidedly different semantics the other ones loyal and implements lower standard straight five to lure straight out of the manual and I like Laura okay you fifa if I have to work with a non functional language I think it's a very nice language it's small it's neat it's imperative is dynamic it's lexically scoped it has mutable variables has mutable environments has global data mutable global data all the bad things right yeah after a while you get stuck honestly so I quick pushy get slightly surprised here you get so used to working with the mutual data you sort of get surprised mutable data how to actually use this in any sensible way right we well if I do start right at modifying things I'm going to i'm going to start destroying things for someone else I mean if you haven't work with the immutable data it's fantastic because I know data will never change under my feet whatever anyone else does if I have a reference to a bit a date to a block of data will always be the same and I won't no one can ruin that for me so having mutable data is really weird look that was a bit better yeah so what what we have here is the semantics the language pretty much follow a line this it's a simple functional language so you can basically compile it straight almost compile it straight down into our line the difficulty here is the data so it's quite restricted in data types there is an ill which just means nothing they've got bullion's it has true or false it's got numbers just floating point i think the latest version has integers but I'm not really certain if they are if they're real integers or just floats that behave like integers have to check that it has strings which are immutable strings to strut nothing straightforward and then it's got tables so all the all the difficult bits and the good bits are in the inning are immutable key value global mutable key value tables and they use tables for everything right it's for tables it's for rays they use lists well the kitchen sink as well everything is a table and if I have sharing a table between different parts of the program and I make a change in the table those changes are visible everywhere which is completely different from Ali so here the problem is not the semantics that's pretty straightforward here the problem is managing the data in such a way which law which law requires and this is it's a nice it's actually quite a nice little language they could do quite a lot with the tables actually it's not object oriented but you can write object-oriented code on it and define classes and class hierarchies and things like this because it has something called meta tables you can work around with so you can do a lot of stuff in there I think one of the major problems I see with it it's not there's a lot of things that aren't standardized and they've become very very and then that not to standardize them so they don't really have a lot one module that way of defining modules have different ways of defining modules we'd sort of work together well they don't depart from that's a very nice language so you have these two languages you've got the prologue you got the law and they will not run director on our line the prologue needs needs working with it with us with the semantics and the lower needs working with the data but they're running on that system the other that now we get to the third level this is external systems now we're talking things that aren't directly running on the owling machine and there are a number of ways of communicating with the outside world or communicating with an on our line world so we have we have a line ports and sometimes as well something called linkedin drivers we have nifs which are natively implemented functions we have see notes so i can actually program other other other operating system process to look like distributed airline notes and their support for that we can also use across use UDP or TCP to talk to other things as well but I'm not going to make go through that so yep starting our imports okay to need to face the outside world this is actually the oldest mechanism we have in the language for communicating with the outside world and it makes the outside world look like processes look like our line processes so I sorry I mentioned this but I mean a lying is very processed based basically do everything in processes there are asynchronous communication you send messages for communicating and sharing data and processes this ports are linked ports make the outside world look like processes so if you want to do something the outside world you send a message when something happens in the outside world you'll get a message from it so yeah alling is very reactive in that sense and we've been reactive for about 25 years right so we've been yeah so yeah I love reactive programming but I do not see that I do not see the bit but where is new and innovative to be honest so yeah we've been buzzword compliant for many years record and yet it elling ports behave like processes I can send messages to them like the error handling works on on on the ports as well to their failsafe okay in the sense that if one of the external processes crashes is not going to crash my alling system and one of the things you tend to become when you're working with a line you become tend to become very paranoid right what if I do this will that crash me we're not talking about crashing a few a long process we're talking about crashing the ailing system here alling processes are fine so yeah this is very sale safe because we're talking about either operating system processes if they crash for some reason the worst thing that will happen is the port will get closed basically you're sending byte byte streams that's how you talk with outside world's most general thing we could dream up your send sequences of bytes the outside world the stream its of its unique spy pro tcp connection socket there is support for packaging these things it's very basic you can say I want to pack up my things with 2012 or four byte lengths in front of it not just as that there is support for encoding and decoding our line data structures so there's there's a bunch of C library so I can actually send an hour-long data structure to it and from the seaside I can unpack that and work with that just as a very simple simple example but but the most trivial one I could find here this is just an external small external C program which I could start up with a port and I can communicate with the point so I would send it lines it will read the lines and think F get s and they just print the line back to the standard output so it'll get input from our line in still on each standard input and we'll send stuff back to our lying on its standard output we're just printing here and after the printer if we have to a flush of course to get the thing actually send it out and if so if our line closes support then the input then the standard input here will close and the f get S will returns will return no showing its closed then we'll just terminate this process and goes away so this is extremely simple but the simplest port process on the airline's side you could do something like this we can open it there's an open port command and we this just says born we're going to start an external operating system program the interface is a stream but we're actually putting a little bit of extra semantics is on top we've got some built-in line handling so we're gonna talk about 880 character long lines here so it will actually read lines and this is just just to make the last function of it easier so we can have an echo line function here we're give it a port and the line I want to send to the port the first line here just the port that just sends this message to the port and the hour-long system will decide the year we're sending a message and we'll pipe take that line we sent out that the stream of characters will send out and just send off the of the operating system process and then we'll just sit sit and wait for the reply okay so this is a stream so there is no guarantee that we're going to get all the returned values in a packet and the other end we're talking about the no guarantees that everything we send in one go it will get us it will get us as one one packet of data that's why we're using the f get S which actually reads until it gets a new line so here we have to we have to sit in the loop and read until we get the underline and that's what the first Clause the port does their data this is a bit of help we getting from the port driver saying here now it's got a line it's got the end of the line here and this is all the whites here then we'll just write it out if we're getting characters from the second Clause here we say well you're getting more characters will append those to accumulate until we collect everything in the whole line and then we just happen so this is just a very simple code for doing stockings but you see here that the port is it's just like a process like it and like any other process there's a protocol for communicating with us nothing strange about this it just happens to be bites okay this is good but sometimes you don't actually want to do something I want to port but I don't actually want to do it in another operating system process so there's something called linkedin drivers which just allows me to move functionality into the a-line virtual machine though this is predefined interface for doing that and I can do that and from the outlying point of view it behaves like a port I open it in the same as it would be a port but it just doesn't open the operating system thing at page does internally and it's usually what can be more efficient than normal ports because I'm actually don't have to send the data to another operating system process I don't have through context which is everything like this so it could be very good the a-line file system file interface it works through LinkedIn drivers for example the problem here is there's there's no safety now linkedin drive is there c code they're running inside the ailing emulator if they make a mistake the whole system crashes so basically no safety nets when i'm running these type of things so that's a trade-off yes it's faster but i'm offering safety so it's trade-off there for the second method are nifs the native we'll implemented functions and they allow me to implement functions in C but to call them from our long as if they were a line functions so it's a foreign function interface now because I'm going to be calling them without from our long I'm going to be calling with our lang data structures i'm expecting hear back our line data structures so there's a large library for accessing the data structures which allows me to access state alling data structures there is support for threads if I want to actually run other threads inside the operating system processes well from from the from the nifs but again there's no safety because this is just C code running in the emulator if something goes wrong there the whole thing just crashes straight down but sometimes this is very nice sometimes what I want to do is a very see based or function based interface and I have a bit of small example here this is real code that by the way this is an IT I I to see bus interface that runs on a raspberry pi this is a couple years old version of probably improved it since then but this is this is AC function which I can then call from our language I'll show you on the next slide and this is just how the interface looks here so I call it I give I have this in n which is just some form of the hour-long environment I get nog see saying how much many arguments this was called with and I have an arguing here's an array of the arguments and what I first do is I unpack this array and check am I getting three integers that's what the first if does I'm expecting to get three integers unpack the integers and put them in FD the file descriptor the address in the length that's what that does if I don't do that I'll return it and make a bad argh about to return something which generates a bad argument for need from the a lang syne then the second bit that that is just the interface the i2c drivers on the on the on the pie so I'm just setting which which address i want to talk which file just using the file descriptor saying which address I to see address i want and then i'll just read this many bytes from that address and then i'll build a binary an hour long data structure binary and then i'll return that has returned value for the function and if something goes wrong in the ioctl or the read i'll just jen again just generate a bad idea so this is a real bit of C code it's a very simple one about the best I could fit into one page for doing things for well actually does something this this this actually worked this code actually worked there is more in this package there was something to write the i2c is to open and close them as well as so how do i call this from the alling site so here we have a little loop i wrote to do these so one problem here with this I to see interface I can't do one thing at a time the buses just happens to be two pins that they're going to be flipped up and down all the time so if i try to do multiple reader writes at the same time to that then it's going to really bomb out completely so i have to make sure that i'm only doing one thing at one time in other languages you might use a mutex do that so i could just block who's doing it in a long you tend not to use mutexes and yeah so so what we have here is a loop and the first two of the reed and rights they just straight calls into it the second one I want to do a request i want to rely want to send something or off from a dress wait little bit and read but have to make sure no one else does anything in the same time so I'm blocking this I'm only having one one a long process which accesses this some we r 2 c-bus and i sit in that request and wait so I do eyes do the right I sit a wait for a bit and then do the read the last one just is even more that just I can give it a list of requests I want to do is in one atomic operation so this is just a typical way of interfacing the outside world and howling you very seldom use mutexes for locking you generally have a process you communicate with which goes through the outside bar so yeah so the final ones see notes they're writing an external OS process that behaves like a distributed our line note so the a-line distribution mechanism for them I can send using at normal airline distribution normal airline sense to do that it's failsafe because we're running on a separate operating system processes and it's a natural interface from our line because I'm sending and receiving messages for the outsides without side note and again there is a large library for interfacing this from the other side man and making it behave like like an a-line note because now expect certain things and game data I couldn't find a good small example for that but if you go to this address as colleague of mine he's written a lure interface using cenotes running the standard or so yet now very quickly so what why why would I use which one the native language is of course that gives me complete access to the Erlang of the vm everything it's definitely the fastest way to run code on the ellen video because i'm using i'm using as good as I can do it the downside is it's I'm limited to what the language can do the non native languages in this case the Earl the prologue and the lower they've got good access to the to the vm i might have to write more interface modules and stuff that they're quite good accesses it's very fast data transfer because they're still just in just inside the a line system like anything else there other they are other languages when we're not we haven't restricted ourselves here to a line in that sense but generally they'll be slower than a normal implementation of those languages outside although one guy was saying though my lure implementation actually went faster than the standard lure for some things and I'm I have to work out because that they shouldn't do that it shouldn't go fast these are costs can be costly to implement lowers a small language and the prologue is actually quite a small language as well so that's quite good but you wouldn't do something like Java in this because it's just a sheer amount of doing things in Java and that's why you do you have the external systems of course you can talk to anything it's not just language systems I can talk to other other things in the world of I might have for example big applications or large set of libraries written in certain language it's just completely infeasible to rewrite those in a line either just sheer work of it off the efficiency like doing the the mathematical computations right you just wouldn't do that knowing you can but you wouldn't therefore you don't exit your airline to access other things like Ericsson does they did they do the signal processing in specialized hardware but they're using our line to to control and manage that which is a very good split up for so you can monitor and manage systems is very good for elings very good for monitoring out the systems we sort of control freak so we like to keep control of other things but we're very good at that downsides generally slower interface and it can be safe or unsafe whether I'm doing just using a normal port which cases say forum user linkedin drive which case Beyonce Fergie and yet we do worry about safety you I mean one of the things you one of the things one reason you come to the a line sister ecosystem at all is for well it's for the concurrency and for building fault tolerant systems if we start worrying about if you can crash the whole machine nothing strange there okay that's about it that's me and I'm generally very bad as you've seen of having pictures I've got two pictures so that's that's pretty good going for my talks right and i just had to do this okay so i could unfortunately I could not fit the no cat in somewhere so yes but okay