Building Composable Abstractions

0 0

hello hello thank you thank you so I have this thing on the screen a little question that just keep in the back of your mind while I'm while the talk is going on my name is Eric Normand the title of this talk is building composable abstractions you might know me from purely functional TV that's my company my newsletter used to be called the closure Gazette please sign up I often get asked how to make abstractions like take a problem and break it down and turn it into functional code a lot of people are able to solve small problems like Fibonacci and that's kind of the typical example you see online but then when they finally want to create an app they're stuck they don't know how to take the tools that they've learned and turn them into software so that's kind of the the purpose of this talk is to develop a process to do that and also I'd like to start a discussion about how we can do that better here's sort of the map of the talk why focus on abstractions we're going to talk about the process and then go through a pretty in-depth example app or at an example abstraction and go through the process with it and then we'll conclude with some like feel-good stuff so I first have to motivate like why abstractions are important to talk about I really like this book refactoring I love the book it's really great it defined refactoring which is now like a common everyday word like this changing software systems in such a way that it does not alter the external behavior of the code now the thing I like about this is it it in like the general industry we now have this idea that there's a difference between the behavior of the code and the actual implementation the actual code itself and I've really credit for that but the thing is it's not enough because what happens when people do refactoring is they think I can just coat it and then clean it up I can just start coding and later I'll make it better so let me explain that a little more why you can't always make it better this Isaac Newton he developed a system called Newtonian mechanics it's three really elegant laws and this system is the prototype of mathematically based scientific systems it's what everything aspires to and there's actually four really simple concepts in it and if you use these you can you can say anything in Newtonian mechanics now this replaced Aristotelian physics that was the system that people thought was the truth about how the universe works for thousands of years and in this system you have these this is just a subset of the concepts you have a concept like ideal speed ideal speed is basically saying small things move faster than big things and natural place rocks sink to the bottom of the water and water it so that's its natural place and it just stops there because that's where it wants to be and then you have water that sits on top and that's where it wants to be like why doesn't the air go into the water world it's natural places on top of the water natural motion oh that's when you're doing smooth easy motion like that is generated by the thing itself and then unnatural motion that's when you get jostled right that's unnatural so you can kind of see how this might have developed like a student would ask Aristotle like well why is it that I can throw a little rock really far and fast but a big rock I can't even get it very far and he's like oh well that's called ideal speed big things just move slower than small and then the student has another user story well natural place like what why is it that rocks fall to the bottom of the ocean oh that's just called natural place and so he comes up with a little answer to that user story and then natural motion another user story so just view one user story at a time this system of physics has aggregated into something that you can kind of see it's just a big mess especially if you know Newtonian mechanics it's much cleaner and elegant so the thing is you can't refactor Aristotle into Newton there's no way to change from this system with these concepts like ideal speed into velocity and acceleration there's no way to do it so I put a tweet icon up there the reason is I like it when people tweet quotes from from a conference talk and so I'm helping you out these are these are like I've measured them they're ready to be tweeted all right so Newton he's a really good example that's why I kept him in the talk even though I might go long he had this quote if I've seen further it's by standing on the shoulders of giants there's three interpretations one is he was a humble person which is not true so we can rule that out number two was he was insulting his rival who was a short man it wasn't him it was a giant that I was standing on okay that's probably true but I actually prefer the third interpretation which is he's giving us a method he's telling us how he did what he did all you have to do is read and learn from people in the past what they've done what they did wrong what they did right how they did it and you just have to see just a little bit more than what they saw and you'll do something great because no one else is doing that thing and everyone will be impressed so there's a little visual thing the thing is over time we've added so much knowledge since his his time so if we graph this well he had to invent calculus I don't know if you knew that we all should actually have calculus for free he had to spend years like developing the notation and everything all the ideas so if you graph this giant height over time there's newton over there and we're like almost at the asymptotes we got so many people so many giants stacked on top of each other that we're able to see way farther way faster my teacher always told me to label my graphs so so we actually can write something like Newtonian mechanics everyday it's three equations we can do that on a computer not only that but we can actually run the thing we can do more calculations than Newton did in his whole lifetime in minutes and see if it works the cool thing is that because we have computers we have all we have access to all the knowledge more than knowledge than he had access to giant height is higher but we also have a system programming which encourages us to make Newtonian style systems instead of Aristotelian systems who uses an Aristotelian system at work where it feels like it's just a bunch of big ball of mud spaghetti code don't you hate it don't you wish you would be able to write something more clean and elegant so there's another there's another tweet all right so where does that put refactoring well we're gonna come back to that but I just want to say that cleaning up is really important and I'm glad that we have this idea but it's like talking about cleaning up your room it's really therapeutic it's important it helps you be more productive and everything but then it doesn't tell you like what needs to be in your room why do you have a room what what does your room have to be like okay so I've developed a process and here are the objectives for that process it has to consistently produce good abstractions and buy good abstractions I mean the Newtonian type so there are gonna be better and worse abstractions that you know have better properties and worse properties for your particular problem but this produces Newtonian style and it's an iterative process so it's gonna get you something you might not like it but you can learn from from making it how to make it better right you make it you learn something you revisit your design decisions you start again anyone can do it what I mean by this is you can do it from stuff you already know like you don't have to go study an abstract branch of mathematics to be able to do it you can start with what you've got right now and then fosters collaboration this is really important we work on teams now and so we need to be able to talk about what we're doing and so often I find that I've like just invented a new concept it's like a new abstraction and I can't talk about it and I try and it just confuses people and so we have disagreements about different things like we're not even talking about the same thing so I feel like this is a way to solve there's a way to solve this and this is actually one of the easier problems to solve this talk was and the process was largely inspired by Connell Elliott's denotational design and in the abstract I said I would be teaching that but I realized that my objectives were much different from his so I kind of went in a different direction so I apologize if that's what you wanted to learn it is very worth learning it's very cool stuff but mine is cool too so I don't feel that bad ok an example this is the example we're going to develop vector graphics system and we're gonna use the process it's a very simple vector graphic system so we can go through it quickly but when I if someone said we're gonna do a vector graphic system in quill I would think well I'm sorry enclosure I think quill and it's true it's a vector graphic system but it doesn't do what I want so let's let's see a little bit how it doesn't do exactly what I want it to do so here's a simple quill thing we do a background and it just makes the window all black if I want to draw a rectangle I have to do rect give it an XY coordinate and within a height if I want to make it red I put fill before I draw so it's kind of like I'm I'm dipping a pen in in the paint and so like the last paint color I dipped in I like a paintbrush a dip a paint brush in paint the last color I dip it in that's the color whatever I'm gonna draw next translate so I can move the move the rectangle by putting translate before I draw okay so I'm it's kind of like saying well I'm gonna move my arm over and then draw right okay now we're gonna get to some problems if I do rotate and I put it before the translate it doesn't just rotate the rectangle where it's at it actually is like rotating around the the top left of the window and so it's it's almost like like I have to move and then I turn right or I turn first and then I moved from wherever I'm at right and then I draw but if I put rotate after translate it does rotate in place around the the top-left corner of the rectangle so it's like I step so I'm translating so I step over and then I rotate and then I draw so like it's this isn't how I draw I don't like so like that it's so it's it's kind of counterintuitive to me it does have a very formal mathematical basis it's called affine transformations but it's not what I want okay so now we're gonna go through the steps three steps physical metaphor meaning construction and implementation I'm gonna go through each of the steps well basically what we're doing is hammock time we are in have two steps before I even start implementing but this is what this is thinking if someone says just think about it first well we don't know what to think it's indistinguishable from procrastination but this is a formal process that actually gives you stuff along the way so you know you're making progress alright so the physical metaphor the idea behind this is to choose a metaphor that will capture the important information in your program and this can actually become the first implementation of your abstraction so you already start with an existing implementation before you've written any code and also because it's physical we live in the physical world since we were born so like we're really good at answering questions about the physical world we'll see that all right so it has the good metaphor has to be able to answer questions and it has to be a shared experience that's another thing is we can talk about our physical experience all right so let's go through some examples of what we could use for vector graphics could be painting which we saw but maybe we have a better metaphor than the one where you you're moving around and drawing stencils clay projected light all of these are good metaphors and they answer the questions so if I want to put a green square on top of a red square what happens well if we're painting it means that the red and the green are gonna mix a little when they overlap but if we're doing clay well the clay is gonna like sit on top and so you'll see the one on top and not the one on bottom so see it answers the questions right and I would like to stress that the particulars of the vector graphic system that I'm developing they're not that important but you should be seeing the process and the thought process that's what I'm trying to express so there's there's these are all good metaphors but they have different answers to the questions so what are we gonna choose shapes in construction paper so just to elaborate the metaphor a little bit I have different colored papers and some scissors and I cut out shapes like rectangles and ellipses those are the two shapes we're gonna do and then I can move them around and put them like let's say I have a big black piece of paper and I just move them around and put them where I want and then that's my that's my graphic so the nice thing is I could explain that in pretty shortly short amount of time now you understand it and you can follow along that's the idea we're working as a team right now now you might disagree with some of the answers that I come up with from my intuition but at least we're disagreeing about the same thing right so easy to disagree about two different things and you don't get anywhere another question you could ask if it's not so obvious like kind of a prompt is how would you do it without a computer how would you do it before computers existed and very often it just means you'd do it like pen and paper right and so you'd have some system like well I would draw a line down the middle of the page and I'd draw you know some columns and some rows and I'd put the the ID on the left and then the name on the right you know you'd have some system and kind of developing an algorithm that a person would follow and so I want to say like if you don't think that your abstraction has a metaphor I've never met a good abstraction that didn't have a good metaphor I've met some Aristotelian type systems that I just couldn't figure out like what they were supposed to really do so maybe you don't have a good abstraction you know if you already have a spec and it's already a mess sorry but you might be able to find smaller abstractions within it okay so this is a summary of the first step we have this rich physical intuition it's how there are studies showing that mathematicians use physical intuition more than notation to do their thinking and so we can do that same thing they contain the answers to our questions we don't have to like just experimenting with different things in code we can ask our thing and we're gonna see that we're gonna see that in the next step they keep you grounded like one thing that happens to me a lot when I start abstracting is I make abstractions that I think might be useful but there's no real basis in the actual thing I'm implementing like just making it overly complex and this grounding you can always just say well what is that in the physical world why are you developing that like what does it correspond to keeps you grounded and then there discussable all right second step this is where you take the you take the intuition you have you ask yourself questions and you mind it for for the answers and you turn them into precise mathematical language so we're gonna be focusing on the interface right now the interface is the meaning of the thing what does it mean to call this method what does it mean to hit that API endpoint what does it mean to have an object of this type we're not saying how the object is implemented we just know that the type has some kind of meaning so we need to figure out what are the things in the interface that we want to keep and what's just an implementation detail we already have an implementation remember we already have construction paper what's important about this that we want to keep so here's a list of things that I'm gonna I'm gonna keep and we'll go over each of them okay so you notice I said precise mathematical language and now I'm showing you a closure code lambda calculus is math a subset of closure is lambda calculus so we can just use closure notation to represent our math also either there spec on here and spec is fairly well defined mathematically I'm not a mathematician so I can't say it's perfectly well defined but I feel like there's a there's an easy subset that you can see like makes a lot of sense and is well defined so we're defining two types here cut out and shape we're defining a function we're not implementing it yet and we're saying at the bottom it takes a cut out and it returns a shape so this is our accessor we can we can see what shape something is we can construct shapes so we're we have the type called color not saying what it is yet not implementing it and we have a function called rect that will return what I cut out takes a color width and height same for ellipse now preservation of shape what I mean by this is I can imagine like getting mixed up in quill and doing some weird transformation where now my rectangle isn't a rectangle anymore you know I I don't trust myself to not get into that mess so what I'm saying is if I translate it if I move it around it's not gonna change the shape so I've done that by making the spec say that the shape of the return value is the shape of the argument I pass it and I'm doing the same for rotation now preservation of color is very similar the color of the return value is the color of the argument now this is just me covering my bases because I don't trust myself when I'm implementing it right this is like TDD overlay order so I want to be able to put something on top of something else just like construction paper and so I'm making a function called overlay it takes two cutouts and it returns a new cutout but then I hope you're all seeing that this is a corner case because I said that all cutouts have a shape and a color what is it what's the color of a green rectangle on top of a red rectangle it doesn't make sense it doesn't even make sense to ask it so that's a corner case and you want to avoid corner cases especially at this stage like it's really early if you can eliminate the corner case it's gonna save you a lot of time why do you want to eliminate corner cases we're talking about making stuff that's composable composable abstractions so if you have a corner case the best case scenario is there's two branches two branches in your corner case because it's like an if statement so if you compose two if statements you now have four branches they multiply if you compose again with another one another corner case now you have eight branches you compose it again you have sixteen branches now you have a thing that came from one of those sixteen branches which one did it come from well now you have to query the value itself to kind of figure out in another if statement which branch it comes from this is how this is how messes happen and it's why we feel that some things some api's are not composable so avoid them like the plague you don't want corner cases especially this early so we're gonna redo overlay I'm gonna find a new type called a collage a collage is now a collection of things put on top of each other think the collection of cutouts and we have an overlay that takes a collage in a cutout so the collage will start empty and we'll add cutouts to it like a like of a reduction now we can define a test check property like a mathematical expression of what how it's supposed to behave and we're basically saying for all cutouts and two different I'm sorry for all collages and two different cutouts if I put a and then B it's different from putting B and then a so I'm saying the order matters okay rotation and translation independence what I mean is you saw before if I translate and then rotate it it draws a different picture from rotating translate and I don't want that to happen so I'm going to I'm going to constrain my system here so I'm saying if I translate and then rotate it's equal to rotating and then translating I'm just saying that as a constraint on my system rotation is additive all right so if I have a rectangle and I rotate it a and then B it's the same as rotating at B and then a right that makes sense so I'm just expressing that in precise mathematical notation same for translation if I move it this way and then that way is the same as moving it that way and then that way and so at this point I've been composing stuff together and you notice that I've been defining how the composition works before I've even said how the things are implemented and what they are so that's how that's kind of like the secret for making things composable as you start with the composition before you've even defined how how the thing will draw and we often do it the other way we're like oh when I see a rectangle on the screen so we draw it and then we haven't thought about composition yet all right so we forgot to draw so I just have a function that will draw a thing and the thing can either be a collage or a cutout just using spec okay so I got to this point and I realized I was very unsatisfied with all of this I just made you walk through it with me the reason I'm unsatisfied is like so we did like a kind of a TVV approach everything we did was writing tests spec we'll do tests we did test check properties and so we've defined all of the properties from the outside and we have all of the work ahead of us have actually like implementing this thing that's one approach is to have a lot of safety and all the tests but the other approach is to construct the meaning from pre-existing parts from pre-existing constructs in the language and what do I mean by that I said rotation is additive well what if I want something that says that if I rotate a and then B is the same as rotating B and then a it's just addition over numbers it already does that and that numbers in addition already have that property and like almost nothing else so we can just use addition with numbers so all we have to do is find constructs that have exactly the properties that we want in our in our meaning so let's go back let's throw it all away and start again so color I'm just gonna bottom out on a tuple of integers I really don't have anything special to say about color so I'm just gonna use the thing that everyone uses RGB but the interesting thing is that we're now defining a cut out as a function two reasons functions are turing-complete so you're never like throwing you're never like gonna get into a corner by using functions the second thing is they have this nice property that the things that change about functions are the arguments so if I don't put something in the argument list I know it won't change and we had this with two things that we want to preserve preservation of color and preservation of shape and now by not putting shape and color in the argument list I'm guaranteeing that they won't change so now that whole thing is taken care of for me I don't have to have an accessor so that I can test it you still can if you want but you don't have to all right so two things I want to point out meanings have to bottom out somewhere and if you're bottoming out in a well-defined mathematical thing you're fine and then you want to choose meanings that have the same structure right choose meanings choose constructs to express your meanings that have the same structure that of the thing you're looking for okay so now here's our constructor for a rectangle notice it returns a function of three arguments like I said it cut out is now a function of three arguments and also notice that there's some quill in there well quill is a well defined system so I'm bottoming out on quill doesn't mean like I'm stuck on quill this is just meaning it's not implementation I just want to say it's acting like this quill code so notice I don't have color and width and height in the in the arguments of the inner function so I know they're not going to change okay now it's starting to get interesting translate takes a cut out and an X and a Y and then it returns a cut out so a function of three arguments and it just adds the T X's and the T Y's so I'm saying that that it's additive that's basically vector addition and it's gonna call the cut out that you passed it with this new translation rotation is very similar rotation is additive so we're doing the R in our prime adding them up and just calling the thing with a new rotation and overlay so it takes two cutouts we don't have a collage anymore takes two cutouts and we're using the property that the thing you draw a second in quill shows up on top so I'm drawing a and then Bibi is gonna show up on top and I'm just passing in the translation in the rotation and now we can define draw as just all the cutout at the origin with zero rotation it's all zeros now the reason I like this approach is we can now run it so I make a rectangle I rotate and translate it make an ellipse and translate it and then I overlay the ellipse on top of the rectangle and I can see what it does this is this is nice it's what I want then I was playing around with it I won't show you all of them but I found one that didn't do what I wanted if I take that overlay and I rotate it the two things separate now this because I've developed the physical metaphor I see this is not what I wanted to do I know that okay this boom my intuition is like whoa something's wrong they're separated out I thought when I thought rotation of an overlay it's sort of like I grabbed both of them with my hand and I rotate them together and so they're not gonna move relative to each other but they did they separated and so I went back to the code and I saw oh I see what's happening it's like I grabbed both of them independently and turn them I'm passing are to both so they're separate they're still separate even though they're now being drawn in the same function it's like I'm grabbing both and turning them so what do I do well I it's thinking about it you know there's got to be something missing that is in my physical metaphor that I'm not putting in my code and the thing is when I when I grab the things I get to choose where I grab them right I get to put my hand here I get to put my hand here and and then when I rotate them it's around the center point like around my wrist right where my wrist is so I didn't include that I didn't include where to grab it so I want to say that like you have to revisit your physical metaphor it has the answers if it's a good metaphor it has the answer like my first instinct as a you know an experienced programmer with bad habits is to just like start trying things did you start trying oh what if I you know do something with our and transform it a little bit like that's wrong just go back and think about it some more okay so we're gonna add the CX and CY so that's the center point that's where I grabbed it and now it looks a little bit more complicated but this is because in the affine system you only rotate around the origin so what I have to do is I grab the things I move them up to the origin rotate it and then move it back and then I can draw them okay so now I have a system I've made it rotate over time so R is changing over time and so it's cool it's they're rotating and they're change not changing relative to each other I think there might be some distortion on the on the aspect ratio there okay so let's that's that's enough of step two to demonstrate the process so let's summarize you want to preserve the features you want to keep that's the interface eliminate the features you don't need that's the implementation details remember we haven't implemented anything yet I know I had code that ran but that was just the meaning I said it means the same as this code no corner cases they're multiplicative you just don't want them you want to when you're constructing your meaning you want to use constructs that already have that same structure that you're looking for and they have to be well-defined so there's a lot of stuff you know some kinds of side effects you don't want that kind of thing you don't you want to avoid those non well-defined things and then focus on composition first before you even define what the thing is alright step three is implementation we have something that runs so it's kind of actually funny to start talking about it now but what if I don't want to use quill what if I want to use SVG I can still do that right but at this point we already know what to do we're a closure programmers that's kind of a joke but it's kind of not but anyway actually that's where this goes refactoring is if we revisit the definition you're changing the code without changing the external behavior we can squint and see external behavior well that's another way to say the meaning of the code so we have the meaning already built now all we have to do is change the code so that it does what we want so for instance if I want to output SVG I can simply refactor from quill to SVG but all of my compositions and stuff those are already done if I want to be able to serialize it to disk that might be a like a implementation detail that I need that wasn't like included in the metaphor while functions can't serialize to disk so I can refactor it into a data structure DSL if I want to be able to run those tests that I had I need to have the quality between things and I don't have that with functions so a data structure DSL also is the answer to that all right I want to revisit my objective to see how well this process works so consistently produces good abstractions I think I think I I have a lot of work to do to show that this works all the time I've used it several times and it's been very useful the thing that I think might be sort of the the thing now I won't be able to do by myself is if I can just cuz I can do it doesn't mean I've elaborated all the steps right I I would like to elaborate the steps more anyone can do also yet to be seen but I think that the steps are pretty pretty easy to do and then fosters collaboration I think I have shown this for sure because y'all could follow along with you know me moving my hands around and stuff okay just to summarize so you use a physical metaphor for guidance and grounding don't want to go space-cadet on these things when you're constructing the meaning you want to define the parts and their relationships and how they compose and you need to use precise mathematical language otherwise you're kind of cheating if you can't actually express the thing that you're doing implementation you just have to refactor to get all the meta properties that aren't part of your metaphor so these are like the implementation details needs to run fast he's to export to SVG that kind of thing some corollaries for this process so for each of the steps there's kind of like an assumption so in the first step you have to know your domain if you don't know quantum physics you're not gonna be able to do this for quantum physics and that's it's kind of like an unfortunate thing but you need to I mean you just need to learn the thing you're coding know your constructs that means you have to know the language and how the the parts of your language are behave and what structures they have so that you can express your meaning in terms of them and know your refactoring so you have to know how to manipulate your code without actually changing the meaning I have a page up on my site please go there and you can download the slides there's also links to some very much better talks than this that were inspirations for this talk and for this process and also sign up for my newsletter there's a form at the bottom that you can sign up for that's it thank you