Constructing Large Scale Enterprise Applications With Ruby And Rails

Constructing Large Scale Enterprise Applications With Ruby And Rails A Bitcoin Exchange Implementation Karan Singh Garewal This book is for sale at http://leanpub.com/largescalesystems This version was published on 2014-01-21

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and many iterations to get reader feedback, pivot until you have the right book and build traction once you do. ©2013 - 2014 Karan Singh Garewal

Tweet This Book! Please help Karan Singh Garewal by spreading the word about this book on Twitter! The suggested hashtag for this book is ##large_scale_systems_in_rails. Find out what other people are saying about the book by clicking on this link to search for this hashtag on Twitter: https://twitter.com/search?q=##large_scale_systems_in_rails

Contents Preface . . . . . . . . . . . Design Is Good . . . . . The Focus Of This Book . Pathways In The Book . Presumed Background .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

1 2 2 3 4

Essential Object Oriented Design . . . . . . . . . . Heuristic Object Design Principles . . . . . . . . . Fundamental Principles Of Object Oriented Design Dependency Injection . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

5 6 8 14

Preface

Preface

2

Design Is Good This is a book on the design and architecture of large scale web enterprise systems with Ruby and Rails. What principles and patterns should be used to architect such systems? This book addresses itself to this question. The design of enterprise scale systems raises distinct issues that are not present in the development of small software applications. In large systems, design becomes the critical factor in ensuring the longevity and maintainability of the application. The canonical example of the implications of bad design is the automatic branch exchange system developed by Ericcson in C++. The entire project had to be abandoned after several years of development and over three million source lines of code. The entire system was then rewritten in an experimental language called Erlang. Ericsson’s ABX experience revealed that the developers saw the trees but very few could see the forest. Though Rails web applications are much smaller in magnitude, they frequently suffer from the same malady; the overall architecture is not well understood. The Ericsson lesson is transparent, large systems rarely survive bad design. We can ignore design and software architecture in small systems; but we do so at out peril in a large scale implementations. There are a number of goals when designing an enterprise scale system. The code base should be easily maintainable. The best systems are easily refactorable. In the best designed systems, refactorability is baked in from the start. I will be looking at this issue in depth. The architecture should be robust, scalable and fault tolerant. Catastrophic errors should not be allowed to propagate throughout the system. The Erlang motto is Fail Early. This exemplifies the Erlang philosophy that building highly fault tolerant systems is a preferred practise and systems should always recover from errors, even catastrophic errors. We want the implemented system to be scalable. The system should be stable under varying and high loads. Scalability has a hardware aspect to it; but nevertheless, scalability can also be designed into the software architecture. A component in the system should not be tied to any particular location. We want locational transparency so that subsystems can be located anywhere. Locational transparency has implications for fault tolerance as well as scalability. A well architected enterprise system must solve these and many other design problems.

The Focus Of This Book In this book we address the architecture of well designed enterprise scale systems. Among other things, the following concerns are addressed: • The use of design patterns which are very useful in the enterprise context • Using service objects • Using the the data context interaction pattern

3

Preface

• • • • • • •

Building a fault tolerant system Building a Rest API for a large system Using concurrency in enterprise scale systems Decoupling enterprise systems with a message bus Implementing locational transparency Implementing workflows with finite state machines Internationalizing the application

The patterns and architectural design described in this book closely emulates the implementation of a Bitcoin Exchange under development in East Asia. In going through this book, you will gain an understanding of cryptographic currencies and the Mathematics which underpins it. Along the way you should also obtain a good understanding of the Bitcoin ecosystem and Bitcoin related code. Time is precious and developers are always short of time. Therefore keeping in conformance with this constraint, this book is meant to be lucid and easily digestible. My aim is to make it an easy read. Being primarily a book on software architecture, the prime objective is to give you the big picture in large systems design.

Pathways In The Book You need not read the book sequentially. There are seevral pathways that are available to traverse the book. If you are only interested in the logical design of large systems, then you should read the following chapters: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

Basic Object Oriented Design Architecting A Large Scale Enterprise System The Service Object Pattern The Data Context Interaction Pattern The Facade Pattern The Decorator Pattern *Architecting A Restful API For The Bitcoin Exchange Implementing A Biological Finite State Machine Implementing A Message Bus And Decoupling Baking In Locational Transparency Injectable Performance Monitors Reaping The Benefits Of Architecture

4

Preface

If your interest is primarily in a survey of cryptographic currencies from the perspective of a software developer, then you should traverse the following chapters: 1. 2. 3. 4.

Bitcoin And Other Cryptographic Currencies The Mathematical Cryptographic Primer Bitcoin Mining And Transactions Architecting A Bitcoin Exchange

Developers who want to absorb not only the theory but see it’s implementation in a working Bitcoin Exchange will benefit by going through the book sequentially.

Presumed Background Since this is primarily a book on logical design, you do not need to know the Ruby language to understand the design theory. Logical design is language agnostic. In order to follow the implementation of the design principles you will need to know Ruby. If you are not acquainted with the Ruby language, I recommend the book,I Love Ruby, by A. K. Karthekeyan. It is very well written and has an easy to read style. I Love Ruby is freely available on the Net. The O’Reilly book, The Ruby Programming Language, by David Flanagan and Yukihiro Matsumoto is the definitive reference on the language. This is also a very well written technical book. For information on the Rails web framework, you can refer to Agile Web Development With Rails 4 by Sam Ruby. Experienced C++ and C developers should find the transition to Ruby and Rails web development relatively easy. A version of this book is available in Mandarin.

Essential Object Oriented Design

Essential Object Oriented Design

6

This chapter is concerned with the fundamental principles of object oriented design. These principles embody best practises in object design and are equally applicable to the design of large and small systems. Before turning to these principles, it is fruitful to enunciate some heuristic object design principles.

Heuristic Object Design Principles Enunciated belows are heuristic principles, meaning that they are only guidelines and not hard and fast rules. There is a tangible advantage in following these heuristics; they cause the developer to introspect on the design implementation and trigger refactoring.

The Object Model Must Model The Domain Space Object oriented design has it’s origins in the principles of taxonomic classification in Biology. One of the fundamental problems in Biology is the articulation of a taxonomy for the Animal Kingdom based on a coherent classification scheme. The objective is to articulate a class hierarchy which faithfully represents the animal kingdom. The class hierarchy for class Animalia includes Phyla, Genus, Chordata, Species and so forth. Similarly in software design we want to model the problem domain faithfully. In order to accomplish this, we need two things. We must understand the domain space and secondly we must have a precise articulation of the problem that is to be solved. This understanding permits us to abstract the domain space in order to solve the problem. The abstraction is the class hierarchy that represents the problem domain.

The Model View Controller Pattern The Model View Controller pattern does not address itself to modelling the problem domain. It is concerned with presenting a view of data; that is, tying persisted data to the presentation of data. This focus typically manifests itself in articulating model classes that include all of the business logic and functionality pertaining to the underlying data table. The approach produces an impedance mismatch since the model classes do not map to the underlying problem domain. In particular, model classes in MVC do not represent the class hierarchy in the problem domain. In MVC, Model classes map to data tables. Aside from violating some basic principles of object design, this impedance mismatch produces brittle and unmaneagable classes. This book pursues an alternate approach to the design of model classes.

Essential Object Oriented Design

7

The Class Hierarchy Should Be Shallow A shallow class hierarchy is easier to comprehend than a deep hierarchy. Shallow class hierarchies are preferred. In modelling the problem domain, we want to abstract everything that is required to solve the problem and disregard the rest. This points to shallow class hierarchies with sharply defined classes. This is of course a heuristic design rule.

Classes Should Be 150 SLOC This is another heuristic rule. When designing a class try to keep the source lines of code (SLOC) to about one hundred and fifty lines or less. If the SLOC goes over two hundred, start thinking about refactoring the class. Small classes are easier to comprehend, refactor and debug. This heuristic suggests that a fat class should be replaced by more than one thin classes.

Methods Should Receive No More Than Three Arguments A heuristic rule again. We are striving to keep methods as simple as possible. Methods should address a single concern only. Methods that do more than one thing are error-prone and difficult to refactor. A method that receives more than three arguments leads to a suspicion that the Single Responsibility Principle is being violated. When there are more than three arguments in a method, you should be thinking about whether the method should be refactored.

Methods Should Be No More Than Fifteen Lines (SLOC) Small methods are easier to refactor and debug. It is tangibly easier to reason about a method that contains 14 SLOC than one that contains 40 SLOC. If you have a method which is twenty or more SLOC, pause and ask yourself, should this method be refactored?

Simple Code Is Preferred It is easy to write obfuscated and complex Ruby code. It is difficult to write simple code. Simple code does not translate into the least lines of code. Simple code is written with simple constructs and feels natural. Simple code is easy to understand and debug. Even though one can write obfuscated code in C, the archetypical example of simple code construction is the C language. One often hears the saying: I Think In C; an allusion to the fact that C code can have a very natural language like feel to it. Aside from having a very small vocabulary, C maps almost perfectly into Assembly language and the mind space; an unique feat.

Essential Object Oriented Design

8

Fundamental Principles Of Object Oriented Design I will now discuss some fundamental design principles. These principles have been traditionally exposed in the context of Java which is a statically typed language. The exposition that follows is Ruby-centric. Ruby is dynamically typed. The five principles described below are considered best practises in modelling problem domains. I am going to show how the principles apply by reference to the following Ruby Bitcoin wallet class: 1

class Wallet

2 3 4 5

def balance(us_exchange_rate) num_bitcoins*us_exchange_rate end

6 7 8

def transfer(bitcoins, recipient) end

9 10 11 12 13

def print gui = Gui.new gui.output(balance) end

14 15

end

In this class, the balance method returns the value of bitcoins in a wallet in US dollars. The transfer method transfers bitcoins to a recipient. The print method outputs the balance in the wallet on some GUI device.

The Single Responsibility Principle Th Single Responsibility Principle (SRP) states that a class should only implement a single responsibility. A responsibility is best understood as some discrete action in the problem domain. For example, the following are some responsibilities in a Bitcoin exchange implementation: • • • •

maintain the account balance transfer bitcoins to a recipient receive bitcoins from a recipient print out the balance

Essential Object Oriented Design

9

To be perfectly clear, a class satisfying SRP assumes total responsibility for performing a particular responsibility. A class that conforms with this principle implements only one class for each responsibility identified in the problem domain. If you examine the Wallet class you will discern that it has three responsibilities: • maintaining the bitcoin balance in the wallet • transferring bitcoins to a recipient • printing the current balance This violates SRP. We obtain conformance by refactoring the wallet class into three classes: 1 2 3 4 5

class Wallet def balance(us_exchange_rate) num_bitcoins*us_exchange_rate end end

6 7 8 9 10 11

class BitcoinTransfer def transfer(wallet, bitcoins, recipient) end end

12 13 14 15 16 17 18

class AccountPrint def print(wallet) gui = Gui.new gui.output(wallet.balance) end end

The application of SRM leads to smaller classes which are easier to understand and debug. In addition, the principle leads to more classes being created.

The Open-Close Principle The Open-Close Principle (OCP) states that a class should be open to extension but closed to modification. Ruby does not have abstract interfaces like C++. However, the signatures (argument types and return values ) of the public facing methods of a Ruby class constitute the implicit interface of the class. This interface expresses the contract between the object and the rest of the world.

Essential Object Oriented Design

10

OCP states that the implicit interface should not change but we should be able to add methods to the implicit interface of any Ruby class. It is clear that if the signature of a public facing method changes then this will break the contract that this object has with other objects which use this class. This is undesirable behaviour. Take a look at the AccountPrint class above, it prints the wallet balance on some GUI device which is represented by the GUI class. Suppose we want to output the balance on a line printer. We will have to modify the body of the print method as well as the arguments of the print method. This will break the contract presented by this method. Making this change modifies the AccountPrint class and constitutes a violation of the OC Principle. The objective is to abstract the media upon which the wallet balance is printed. The required method modification is: 1 2 3 4

class AccountPrint def print(wallet, media) media.output(wallet.balance) end

The print method receives a media object which responds to an output method. The AccountPrint class does not hard-code the media to which the wallet balance is sent. It is only concerned that the media object received respond to the output method. A small but vital abstraction. Designing the method in this way from the outset ensures that the object will satisfy OCP.

The Interface Segregation Principle The Interface Segregation Principle (ISP) states that the interface of a class should be no larger than it needs to be. ISP requires that the class implement only the methods required to fulfil the single responsibility of the class and furthermore the implicit interface of the class should be minimized. Why should the implicit interface of a Ruby class be minimized? Because the state of an object is effected by other objects acting upon the object through it’s implicit interface. These objects then become interdependent through their interaction with the object through it’s implicit interface. This is unwanted behaviour which we want to minimize. Suppose that we had refactored the original Wallet class as follows:

Essential Object Oriented Design 1

11

class Wallet

2 3 4 5

def balance(us_exchange_rate) num_bitcoins*us_exchange_rate end

6 7 8

def transfer(bitcoins, recipient) end

9 10

end

11 12 13 14 15 16

class AccountPrint def print(wallet, media) media.output(wallet.balance) end end

Notice that the AccountPrint class does not use the transfer method. The problem with this refactoring is that the AccountPrint object can be effected by some unknown class which acts upon the transfer method. This third class changes the balance through the transfer method. The change in the wallet balance then effects the output emitted by AccountPrint. This is unwanted behaviour since it sets up an implicit dependency between the AccountPrint class and that other third class. The solution to this violation of ISP is to move the transfer method to a new class as was done previously. An anti-pattern is indicated in Ruby classes that have complex and large implicit interfaces. Next consider the circumstance where a sub-class is derived from a class with a large implicit interface. The sub-class will inherit all of the public methods of it’s superclass even though it it does not need all of these methods in order to perform it’s single responsibility. This anti-pattern will propagate down the class hierarchy; a very nefarious consequence of violating the principle. The solution to the anti-pattern is to eliminate the unwanted interfaces or re-factor the class into a number of classes with narrower interfaces.

The Liskov Substitution Principle In it’s original incarnation, the Liskov Substitution Principle (LSP) is a statement about sub-types in statically typed languages. This principle states that functionality in the derived class should be completely substitutable by the functionality in the base class. A subtype B of A should have the same semantics as A. Look at the following class derived from the Wallet class

Essential Object Oriented Design 1 2 3 4 5

12

class Wallet def balance(us_exchange_rate) num_bitcoins*us_exchange_rate end end

6 7 8 9 10

class CanadianWallet < Wallet def balance(us_exchange_rate, converter) num_bitcoins*us_exchange_rate*converter end

11 12

end

The balance method in the Wallet class returns the balance in USD. The balance method in the CanadianWallet class returns the balance in Canadian dollars by applyig a converter to the USD exchange rate. The implementation of CanadianWallet violates LSP bcause the semantics of the balance method in the sub-class have changed. The derived class has changed the signature of the balance method. What we really want from the design viewpoint is to obtain the balance in a wallet without worrying whether the object is of type Wallet or of type CanadianWallet. We can refactor Canadian Wallet by adding a method to the CanadianWallet class as follows: 1 2 3 4 5

class CanadianWallet < Wallet def cnd_balance(cnd_exchange_rate) balance*cnd_exchange_rate end end

A much better solution is to not create a sub-class at all; but define the Wallet class as follows: 1 2 3 4 5

class Wallet def balance(exchange_rate) balance*exchange_rate end end

6 7

class CanadianWallet < Wallet

8 9 10

... end

Essential Object Oriented Design

13

Here exchange rate is simply the exchange rate of a currency in USD. Now we can compute obj.balance without having to reflect upon whether obj is of type Wallet or CanadianWallet.

The Dependency Inversion Principle The Dependency Inversion Principle (DIP) states that a class should not depend on other classes, but should depend on abstractions of classes. This is a principle about decoupling classes from each other. A very important matter in object design. Take a look at the following implementation of the AccountPrint class: 1 2 3 4 5 6

class AccountPrint def print(wallet) gui = Gui.new gui.output(wallet.balance) end end

The problem with this class implementation is that the print method depends upon a concrete class : GUI. This violates DIP. We can remove this concrete dependency as follows: 1 2 3 4 5

class AccountPrint def print(wallet, media) media.output(Wallet.balance) end end

Now the print method can receive any object responds to a output method. We have decoupled the print method from the output media and the GUI class in particular.

Art and Science In Design Object-oriented design is a science as well as an art. Therefore design principles are not cast in stone but are subject to art as well as prior experience. The ultimate objective in designing systems is to speed up the development process, bake in system extensibility and reduce the complexity of debugging.

Essential Object Oriented Design

14

Dependency Injection Consider the AccountPrint class again. The implementation of the print method shows dependency injection at work. Instead of hard-coding the GUI object into the body of the print method, we are injecting the dependency at run-time. Before the actual injection at run-time, the print only cares that the injected object it receives responds to the output method. Dependency injection has four elements: • • • •

The consumer object (the AccountPrint object in our case) The injected component (the instantiated object passed as media to the consumer) The implicit interfaces required by the component in order for it to be used in the print method The injector which instantiates the component which is to be injected

Dependency injection decouples objects. The relationship between objects is determined at run-time.