Modern Functional Programming: The Onion Architecture

283 points by Roxxik 8 years ago | 95 comments
  • slashdotdash 8 years ago
    Gary Bernhardt describes a similar architecture using a "Functional Core, Imperative Shell" in his Boundaries talk[1].

    "Purely functional code makes some things easier to understand: because values don't change, you can call functions and know that only their return value matters—they don't change anything outside themselves. But this makes many real-world applications difficult: how do you write to a database, or to the screen?"

    "This design has many nice side effects. For example, testing the functional pieces is very easy, and it often naturally allows isolated testing with no test doubles. It also leads to an imperative shell with few conditionals, making reasoning about the program's state over time much easier."

    [1] https://www.destroyallsoftware.com/talks/boundaries

    [2] https://www.destroyallsoftware.com/screencasts/catalog/funct...

    • junke 8 years ago
      At the lowest levels, the pattern can be reversed: you provide a nice functional shell around a little bit of imperative core (e.g. implementing "map" with a loop).
      • harryjo 8 years ago
        You are referring to, for example: http://clojure.org/reference/transients

        Combining the two ideas

        Transient imperative logic in the core (5%), Functional mantle (90%), Side-effecting imperative crust (5%).

        • junke 8 years ago
          Yes, because some purely functional approaches cannot beat imperative ones when it comes to resource usage.
          • rtpg 8 years ago
            the main thing being that side effects still aren't happening in the core, so the core is still referentially transparent.
            • rjbwork 8 years ago
              mmmmmmmmmmmm. Functional sandwich.
            • KirinDave 8 years ago
              This kind of abstraction layer is really challenging though unless you're totally aware of the extent of your imperative code's side effects.
              • stingraycharles 8 years ago
                Isn't this already happening, though, on a low-level (i.e. ASM) ?
            • hota_mazi 8 years ago
              I understand the value of referential transparency and how it makes "certain" things easy, but saying that it automatically makes testing functional code easy is a myth. Sometimes it does, sometimes it doesn't.

              If you want to stick to referential transparency, you can't use dependency injection: you have to pass all the parameters the function needs. None of these can be implicit or belong to a field on the class since that would mean side effects. The `Reader` monad is not dependency injection, it's dependency passing and it comes with a lot of unpleasant effects on your code.

              And because of that, functional code is often very tedious to test. Actually, in my experience, there is a clear tension between code that's referentially transparent and code that's easily testable. In practice, you have to pick one, you can't have both.

              • mattnewton 8 years ago
                > And because of that, functional code is often very tedious to test. Actually, in my experience, there is a clear tension between code that's referentially transparent and code that's easily testable. In practice, you have to pick one, you can't have both.

                In all my years of software development, I've never encountered a referentially-transparent function that was even remotely hard to test, let alone harder than one with environmental baggage. In fact, being referentially transparent opens you up to new kinds of powerful testing strategies that are nearly impossible if the function isn't, like QuickCheck. (I can't highly recommend quick check enough, it's worth the little learning curve 100x over)

                • bad_user 8 years ago
                  First of all, passing parameters in functions is "dependency injection". And what you're describing is a really good thing.

                  Lets be honest, most dependency injection frameworks and techniques are about hiding junk under the rug. But they fix the symptoms, not the disease. You see, if you find yourself having components with too many dependencies, feeling pain on initialization, the problem is that you have too many dependencies, which actually means you have too much tight coupling and not that it is hard to initialize them. At this point you should embrace that pain and treat the actual disease.

                  Also, functional programming naturally leads to building descriptions of what you want, in a declarative way. So instead of depending directly on services that trigger side-effects directly, like a DB component that does inserts or something doing HTTP requests or whatever, instead you build your application to trigger events that will eventually be linked to those side-effects triggering services.

                  There are multiple ways of doing this. For example you could have a channel / queue of messages, with listeners waiting for events on that queue. And TFA actually speaks about the Free monad. Well the Free monad is about separating the business logic from the needed side-effects, the idea being to describe your business logic in a pure way and then build an interpreter that will go over the resulting signals and trigger whatever effects you want. There's no dependency injection needed anymore, because you achieve decoupling.

                  > And because of that, functional code is often very tedious to test.

                  That hasn't been my experience at all, quite the contrary, we've had really good results and we're doing such a good job of pushing the side-effects at the edge that we no longer care to unit-test side-effecting code. And yes, I believe you've had that experience, but I think it happens often with people new to FP that try and shoehorn their experience into the new paradigm.

                  E.g. do you need a component that needs to take input from a database? No, it doesn't have to depend on your database component at all. Do you need a component that has to insert stuff into the database? No, it doesn't have to depend on your database component at all. Etc.

                  • hota_mazi 8 years ago
                    > First of all, passing parameters in functions is "dependency injection". And what you're describing is a really good thing.

                    It's only injection if the parameter is passed automatically by a framework. Otherwise, it's parameter passing.

                    And it's only a good thing if you value referential transparency over ease of testing and encapsulation. Not everybody does (and personally, sometimes I do and sometimes I don't).

                  • DanWaterworth 8 years ago
                    When you do dependency injection well, you don't inject every object; that would be horrible/impossible. What you may have noticed is that there are two kinds of objects, ones that you inject and ones that you don't. The ones you don't are things like numbers, strings and maps/sets; things that you treat as values. The other objects do things, I tend to call them services.

                    In order to do a straight-forward conversion to functional programming, I suggest leaving the values as they are and each service becomes a free monad transformer. So, instead of having a logger, you have a logging monad transformer that has a log instruction. Instead of having a database, you have a database monad transformer that has a query instruction, etc.

                    You are then free (no pun intended) to replace the interpreters of these free monads during testing with whatever mock implementation you please and the result is a more principled dependency injection inspired style.

                    Actually, I would constrain the monad type via type classes, rather than using free monads, but the approaches are equivalent.

                    • hota_mazi 8 years ago
                      I agree there is a dichotomy between objects you inject and objects you don't but I think your characterization is incorrect: what decides if an object needs to be injected is not tied to its type but to its role. Sometimes, I inject integers or strings or other primitive types. Other times, I pass them explicitly.

                      The decision is made based on whether that object is a runtime object (i.e. decided by the user or some other factor that cannot be known when the app starts) or a dependency that's decided early and won't change through the life of the app.

                      Either way, this aspect is independent of the point I was making above and which is that functional code is not inherently easier to test than procedural code.

                    • sedachv 8 years ago
                      > I understand the value of referential transparency and how it makes "certain" things easy, but saying that it automatically makes testing functional code easy is a myth. Sometimes it does, sometimes it doesn't.

                      > And because of that, functional code is often very tedious to test.

                      Your argument rests on a fundamentally wrong assumption. Expressions in functional programs do not have to be (and indeed are almost never) referentially transparent. Just consider global or module-level immutable variables. Those function names? Also not referentially transparent. This goes all the way back to free variables in the lambda calculus: https://en.wikipedia.org/wiki/Lambda_calculus#Free_variables

                      Further, dependency injection is a completely idiotic and broken pattern and IMO the worst thing to come out of object oriented programming. Once you have dynamic scoping (surprise! also not referentially transparent) everything that DI does (and much more) becomes trivial.

                      • Jweb_Guru 8 years ago
                        My own opinions on the matter aside, I don't fully understand why anyone who likes dynamic scoping would dislike dependency injection.
                      • wyager 8 years ago
                        The reader monad removes 90% of the syntactic overhead of dependency passing. That's the point.

                        If you want dependency injection as you've defined it, you can use (if we're talking about Haskell) typeclasses or, by extension, implicit parameters, to do dependency injection in the way you like.

                        It's still much safer and easier to reason about than Java-style dynamic dependency injection.

                        • hota_mazi 8 years ago
                          Actually, `Reader` adds a lot of boiler plate that's not present with traditional @Inject injection:

                          - All your functions now need to return a Reader[C,A] instead of just A

                          - You need to pass all the parameters explicitly in each method signature as opposed to passing just the ones that don't need to be injected.

                      • qznc 8 years ago
                        I like the 'weakly pure' concept in D. A function like

                          pure int frignate(database db, const config cfg);
                        
                        cannot change anything except the database object (and anything reachable from it). Can not mutate the environment. Can not mutate the config object parameter. This is finer control than pure functional programming and safer than imperative/object-oriented programming.
                        • mej10 8 years ago
                          You can achieve this level of control using monad transformers or extensible effects -- this is a standard technique in Haskell.

                              frignate :: (MonadDB m) => Config -> m Int
                              frignate cfg = do
                                db <- getDB
                                ...
                                return 1
                          
                          And it is composable, so if a function calls a function that uses one of the managed resources then the requirement propagates upward.

                          And you can swap in non-IO based instances for testing, or whatever else you want.

                          • thinkpad20 8 years ago
                            I'm not sure how D's purity system works, but that doesn't seem very pure to me. Mutating the database object is a globally visible effect, after all.
                            • buzzybee 8 years ago
                              The guarantee it's making is that it's not going to manipulate program state that is outside the scope of "db". That is a pretty big deal for a systems developer since, as I discovered while working with D, many standard library functions that you wouldn't have given a second thought about happen to do unpure things like set a processor flag. We aren't even talking about your own program.

                              From an abstracted viewpoint, it's not great, since a whole database covers potentially a lot of scope, and you may not want to care about the details of your floating point calculations in hardware, but in a concrete sense this is totally correct!

                          • VeejayRampay 8 years ago
                            I'd like to use the occasion that I'm really waiting for the video of his talk "Ideology" given at StrangeLoop 2015.
                          • asciihacker 8 years ago
                            I'm enjoying watching the boundaries talk, especially when he converts the serial code the concurrent code using actors.
                            • pmarreck 8 years ago
                              This is one of the best talks I've ever seen, highly recommend anything by Gary Bernhardt
                              • KirinDave 8 years ago
                                I was quite disappointed the author chose not to cite Bernhardt's work, as this is pretty clearly derivative of that talk and other work in the community around this design.

                                I'm certain I've heard Hickey talk about it a few years ago as well. Trying to remember where.

                                • dack 8 years ago
                                  I don't think it's derivative. He mentions the onion architecture, which is an older concept. I really liked the way Gary presented the idea, but he wasn't the originator.
                              • bad_user 8 years ago
                                > Free monads permit unlimited introspection and transformation of the structure of your program; Free monads allow minimal specification of each semantic layer, since performance can be optimized via analysis and transformation.

                                That is not true and this overselling of the Free monad is hurting the concept.

                                The Free monad is nothing more than the flatMap/bind operation, specified as a data-structure, much like how a binary-search tree describes binary search. And this means an imposed ordering of operations and loss of information due to computations being suspended by means of functions.

                                You see, if the statement I'm disagreeing with would be true, then you'd be able to build something like .NET LINQ on top of Free. But you can't.

                                • buffyoda 8 years ago
                                  Free monads can be inspected "up to the first lambda", which is always at least one operation in, and it's what enables things like purely functional mocking. However, other free structures, such as free applicatives, can be completely introspected:

                                  https://www.youtube.com/watch?v=H28QqxO7Ihc

                                  The point is that free monads allow introspection up to the information-theoretic limit (obviously you can't inspect a program whose structure depend on a runtime value), while transformers do not allow any introspection at all.

                                  • KirinDave 8 years ago
                                    Maybe John is attributing more magic to the common free monad patterns than is actually there? Certainly I had stars in my eyes when I first read:

                                    https://people.cs.kuleuven.be/~tom.schrijvers/Research/paper...

                                    > then you'd be able to build something like .NET LINQ on top of Free. But you can't.

                                    You sort of can. You could build an interpreter of Linq commands and then have an executor interpret that. Which I suppose could be argued is making Linq. :)

                                    • rtpg 8 years ago
                                      Maybe this is a bit trite, but isn't the ordering given in Free simply the order of the code? At what stage does loss of information happen.
                                      • mbrock 8 years ago
                                        Yeah, the free monad structures involve lots of lambdas. It's not so much that information is "lost", more that you cannot see beyond the next lambda abstraction.

                                        From a DSL perspective, it's like you can only inspect the program so far as to know the next statement in the "do" block. To see what the next statement will be, you need to actually evaluate the current one.

                                        Instead of free monads, if you want very analyzable structures, look at free applicative functors.

                                        "Applicative functors are a generalisation of monads. Both allow the expression of effectful computations into an otherwise pure language, like Haskell. Applicative functors are to be preferred to monads when the structure of a computation is fixed a priori. That makes it possible to perform certain kinds of static analysis on applicative values. We define a notion of free applicative functor, prove that it satisfies the appropriate laws, and that the construction is left adjoint to a suitable forgetful functor. We show how free applicative functors can be used to implement embedded DSLs which can be statically analysed."

                                        http://arxiv.org/abs/1403.0749

                                        • lmm 8 years ago
                                          The problem is that applicative functors aren't powerful enough to allow computations that depend on previous results. I suspect what we want is something like free ArrowChoice, but I'm not aware of any work in that direction.
                                        • lmm 8 years ago
                                          Monad linearizes the call graph; the effects of f(g, h) have to look the same as those of f(g(h)). In particular your interpreter can't parallelize, because it can't tell whether there's a data dependency between one effect and the next or not.
                                      • gravypod 8 years ago
                                        I'd not say this is a feature of functional programming. This is a feature of Object Oriented elements being included in fp languages.

                                        These are the same things we were going to happen when using Java and C++ years ago.

                                        You can even see similar graphics here: https://docs.oracle.com/javase/tutorial/java/concepts/object...

                                        I remember there was another in this tutorial that shared more with the image in this post. Although this is the same idea. You're just hiding Objects in Objects.

                                        • lmm 8 years ago
                                          The difference is that a "layer interpreter" in functional style is just a pure function that transforms values to values, so it's very easy to test in a very direct way. And we have laws that guarantee that composition works correctly (i.e. the interpretation of a composition is the composition of the interpretations). Whereas it's a lot fiddlier to confirm that an object that uses other objects behaves correctly (you have to mock the inner objects), and virtually impossible to determine whether composition behaves correctly since the object's internal state is opaque.
                                          • mickronome 8 years ago
                                            Thank you for writing 'functional style' and not 'functional language'. While the latter is usually, and quite naturally, better at expressing the former, there is usually a lot to gain simply by adapting the style to the issue at hand, which not _necessarily_ implies changing languages.

                                            What follows is not necessarily of high value, I'm simply a working programmer since 20 odd years that's bit weary and sad that the craft appears to be stuck in a rut by getting stuck between an unnecessarily theory-less reality and a nirvana of unrealistic purity. It's - maybe - a backdrop to explain why I felt a need to thank you for your choice of words with so many words.

                                            If we consider every problem has a shape (loose analogue for the set of constraints thet uniquely identifies a problem) , then for every shape, or class of shapes a problem embodies, there exists an in some sense - ideal - language to solve that problem. Few are however the times when you only need to solve one discrete class of problem in the same system, but in case of mismatc between language and shape, it's quite common the only solution brought forth is to change languages. Unfortunately that solution is rarely feasible for a multitude of reasons. In the fallout after having to keep working with the same non-ideal language, the entire idea that there are multiple ways - styles - to express a solution is sadly often lost. This would not necessarily happen if the idea that style matters enough that when we can't change language, we could, and would still change how we express the solution within our constraintd. Be it in any language or paradigm under the sun, we need the words from them all to be able to talk about our problems, as they are either unique or already solved.

                                            • lmm 8 years ago
                                              The functional mindset views programming languages almost as families or toolkits: solutions should be written in the language of the domain, and successively interpreted to the language of the machine. Thus you don't change language to adapt to a different domain; rather you write a new domain sublanguage. The challenge is if anything to avoid going too far in the other direction; lisp in particular is notorious for being so flexible that no two people's lisp styles end up compatible.

                                              I think there's a happy medium to be found. As my programming career has progressed I've become more and more in favour of Scala for everything - I think it gets pretty close to striking the right balance between the flexibility to express any given domain and the consistency to allow programmers to collaborate.

                                          • barrkel 8 years ago
                                            The architectural pattern of implementing a program in a language close to the domain and iteratively transforming it into something that can be consumed in an execution environment long predates OO, and is in fact the principle behind compilers.

                                            It came out in a different guise under Model Driven Architecture and executable specifications a few years back.

                                            It'll pop up again in the future under some other name. The principle is as old as computing itself, though.

                                            • eternalban 8 years ago
                                              > At the center of the application, semantics are encoded using the language of the domain model.

                                              Say pg/psql.

                                              > Beginning at the center, each layer is translated into one or more languages with lower-level semantics.

                                              ORM mapping into Java?

                                              > At the outermost layer of the application, the final language is that of the application’s environment — for example, the programming language’s standard library or foreign function interface, or possibly even machine instructions.

                                              Or maybe even html+javascript.

                                              Congratulations, you have (re-)invented the layered architecture.

                                              • ubertaco 8 years ago
                                                >> At the center of the application, semantics are encoded using the language of the domain model.

                                                > Say pg/psql.

                                                Minor nitpick: "pg/psql" is not the language of the domain model. The language of the domain model is stuff like "A car is considered 'All-Wheel Drive' if the drivetrain delivers power to any/all of the axles, not just a single axle."

                                                pg/psql requires translating that domain-model language into something (roughly) like

                                                    set @awdCars := (select * from cars where drivetrainAxles >= allAxles)
                                                • asciihacker 8 years ago

                                                      > Beginning at the center, each layer is translated into one or more languages with lower-level semantics.
                                                      ORM mapping into Java?
                                                  
                                                  An ORM would not have lower-level semantics would it?
                                                  • lmm 8 years ago
                                                    Only if you can express your domain well in SQL. For most domains you can't, you need a... domain specific language.
                                                • wtetzner 8 years ago
                                                  Well, he says this pretty early on:

                                                  > The onion architecture can be implemented in object-oriented programming or in functional programming.

                                                  • nickpsecurity 8 years ago
                                                    It started in CompSci sort of in parallel with Bob Barton's B5000 designed for ALGOL, the work of Dijkstra, McCarthy's LISP, Hamilton's USL, and so on. They each came up with abstract ways to specify or implement programs for greater correctness, readability, safety, and composition. Using provers, compilers, or manual work, these would be converted into specific code at lower levels or final, machine level in ways intended to preserve high-level properties. Dijkstra went furthest in THE where the did hierarchical layers that were loop free. In parallel, there was a group trying to figure out how to compile Monte Carlo simulations on their computers in a way that was easy to specify. Their Simula was first, OOP language. SimScript, a RAND language, was a discrete, event simulation tool that came out with many similarities to emerging Simula and OOP. The main programmer had already been exposed to ALGOL, loved the heck out of it, and took some inspiration from SimScript.

                                                    Hierarchical layering, careful interfaces, dynamic programming, functional composition, refinement, event-based simulation... all showing up before Simula was published in 1967. So, it anywhere from depended on to came after many things with properties key to OOP's effectiveness. It certainly got a powerful technique for structuring programs started but didn't happen in isolation or even necessarily ahead in many ways. It surprised me that needs of Monte Carlo apps is what led to OOP but not that ALGOL60 was involved.

                                                    http://phobos.ramapo.edu/~ldant/oop/simula_history.pdf

                                                    https://www.rand.org/content/dam/rand/pubs/research_memorand...

                                                    • munro 8 years ago
                                                      The graphics are both indeed circular which makes them look alike, but the meaning is completely different.

                                                      In the OO graphic, it's representing a single cell organism, hence the circle. This is not the complete application. It's encapsulating the state of a single process, and only through externally interacting with the organism can the state be inspected & changed, all based on time.

                                                      The circles in the FP represent domain logic, the inner most circle represents your high level business logic. This is the complete application. Then translating your logic to the next lower level domain, until the physical hardware layer is reached and your program becomes something concrete and runnable. This layering resembles an onion, which is also a circle.

                                                      • gravypod 8 years ago
                                                        Them being circular has nothing to do with what are they representing. They are showing off the idea of hiding implementation through abstracting it. This is the exact same concept that both of these photos are showing.
                                                        • munro 8 years ago
                                                          > Them being circular has nothing to do with what are they representing.

                                                          Please read my comment, that's what it says right after the first comma.

                                                          > They are showing off the idea of hiding implementation through abstracting it. This is the exact same concept that both of these photos are showing.

                                                          My comment breaks down how the hiding of implementation is completely different between the two, can you point what you think is incorrect so we're not talking past each other? Or is there anything you need clarification on?

                                                          • 8 years ago
                                                        • gclaramunt 8 years ago
                                                          As far I understand, the difference is that in OO you normally transform data as you jump from layer to layer, here what do you have assembled is a simple program that will be interpreted by the lower layer
                                                        • jackmott 8 years ago
                                                          I didn't understand a word, and I bet 99% of people who clicked the link didn't either.
                                                        • tucaz 8 years ago
                                                          Unrelated to functional programming there is Onion Architecture defined by Jeffrey Palermo in 2008 which I find a very nice and simple way to architect/organize a LoB application.

                                                          http://jeffreypalermo.com/blog/the-onion-architecture-part-1...http://jeffreypalermo.com/blog/the-onion-architecture-part-2...http://jeffreypalermo.com/blog/the-onion-architecture-part-3...http://jeffreypalermo.com/blog/onion-architecture-part-4-aft...

                                                          • munro 8 years ago
                                                            How does this architecture constrast with Rust? Has anyone built anything like this in Rust?

                                                            It makes me think of an article where the author tries to abstract the implementation of IO from the the domain logic. [1]

                                                            [1] https://blog.skcript.com/asynchronous-io-in-rust-36b623e7b96...