Building Enterprise Blockchain-as-a-Service Applications Using Ethereum and Quorum
Ethereum is a general-purpose blockchain that is more suited to describing business logic, through advanced scripts, also known as smart contracts. Ethereum was designed with a broader vision, as a decentralized or world computer that attempts to marry the power of the blockchain, as a trust machine, with a Turing-complete contract engine. Although Ethereum borrows many ideas that were initially introduced by bitcoin, there are many divergences between the two.
The Ethereum virtual machine and smart contracts are key elements of Ethereum, and constitute its main attraction. In Ethereum, smart contracts represent a piece of code written in a high-level language (Solidity, LLL, Viper) and stored as bytecode in the blockchain, in order to run reliably in a stack-based virtual machine (Ethereum Virtual Machine), in each node, once invoked. The interactions with smart contract functions happen through transactions on the blockchain network, with their payloads being executed in the Ethereum virtual machine, and the shared blockchain state being updated accordingly.
If you are not familiar with blockchain technology reading History and Evolution of Blockchain Technology from Bitcoin article is highly recommended.
If you like to learn more Ethereum blockchain development (with Solidity programming), Blockchain Developer Guide- Introduction to Ethereum Blockchain Development with DApps and Ethereum VM, Building Ethereum Financial Applications with Java and Web3j API through Blockchain Oracles and Harness the Power of Distributed Storage IPFS and Swarm in Ethereum Blockchain Applications are highly recommended.
The Ethereum main network, and its corresponding test networks, are all open for public use. All transactions made on the networks are 100% transparent, and anybody with access to the network—which is potentially everybody—can see and access all data.
The users of these networks don’t necessarily know each other: they are mutually distrustful parties who must assume that the other participants in the network are dishonest. For this reason, public blockchain networks require a way for such parties to reach an agreement without needing to trust each other, and this is where decentralized consensus algorithms, such as proof of work (PoW), come into the picture.
There are cases, however, where transacting parties in a network are more trusting of each other, and where the use of a blockchain isn’t solely centered on allowing mutually distrustful parties to interact.
In this recipe, we will explore the main types of blockchain networks available – public and private—before looking in more detail at how blockchains can be used in business settings, and what options are currently available for keeping data private. We will then implement a very basic private network that makes use of certain privacy features.
Public versus private and permissioned versus permissionless blockchains
The Ethereum main network is public, meaning anyone is free to join and utilize the network. There are no permissions involved: not only can users send and receive transactions, they can also take part in a consensus, as long as they have the appropriate hardware to mine blocks. All parties in the network are mutually distrustful, but are incentivized to remain honest by the mechanisms involved in the PoW consensus. This is an example of a public, permissionless network. These networks offer high resistance to censorship and good data persistence, but are less performant due to the decentralized nature of consensus.
The idea of permission, when applied to blockchains, could take one of several forms: it could be explicit, as in the case of an access control list, or implicit, as in a requirement placed on users to enable them to join a network. An example of a public, permissioned blockchain, could take the form of a public proof of stake network, in which permission to participate in the network as a validator is granted in exchange for a deposit or stake.
Of more concern for this recipe is the idea of networks run by companies, either for their own internal use, or as a shared resource used by a group of companies wanting to transact. Such networks are private: they are not open to the general public, but only to those companies authorized to join. These are sometimes also referred to as consortium blockchains, and as such, networks often also impose different types of permissions for different participants. For example, different members of the network may have read, write, or access permissions, while another subset of members may take on the job of validation.
Such private networks can take advantage of permissioning to be more performant than their public and permissionless counterparts. If a set of nodes can be trusted as validators, then consensus can be reached more quickly. There is, however, a tradeoff: a set of permissioned validators must be trusted to correctly and honestly validate blocks. In the world of business, where time is money, such a tradeoff can often be worth making.
Privacy and anonymity in Ethereum
We have so far discussed the differences between public and private networks, where it is the network itself, and access to it, which is either public or private. We now turn our attention to the data—both transaction and contract data—inside a given network.
The Ethereum main network can be joined by anyone. Furthermore, all transaction and smart contract data is public, meaning all transactions between a to and from address can be seen by everybody using the network. There is no way to hide these transactions, or the addresses transacting, and as such, there’s no way for a user on Ethereum to be truly anonymous. If a way were found to link an address with a real-world identity—either at the present time, or at a point in the future—then the identity of the transacting party would be known. This might seem obvious: on public networks, all data is public. What is less obvious is that even in a private Ethereum network, data within the network is visible to all participating nodes.
Other cryptocurrencies, such as Zcash, Monero, and Dash, provide differing degrees of anonymity, and we will briefly discuss two of the techniques for doing so in this recipe, and look at how these could be applied to Ethereum.
Monero uses a type of digital signature called a ring signature, which helps anonymize the transacting addresses as well as the amount being sent. It’s possible to use a similar technique in Ethereum by using mixing services based on ring signatures, but these generally aren’t accepted as being robust and scalable methods that could be used in an enterprise.
Zcash uses a different technique, leveraging Zero-Knowledge Succinct Non-Interactive Argument of Knowledge (zk-SNARK) proofs. In this scheme, a user is able to prove possession of some data without needing to reveal what the data is, and without needing to interact with the user verifying the data. Some work has been done to incorporate zk- SNARKs with Ethereum, though at the time of writing this work is still experimental.
So, there is currently no reliable method to make addresses and transactions anonymous in a guaranteed way – what about the contents of those transactions?
The contents of a transaction, as well as the code and data associated with a smart contract, are publicly viewable and cannot be obscured. Though a smart contract’s code is compiled to bytecode, it should not be assumed that an adversary wouldn’t be able to decompile and read the code. As such, sensitive information should neither be hardcoded into a contract nor sent to it as part of a transaction.
What can be done, however, is to encrypt any sensitive data off-chain before sending it to the network. Using public-key cryptography, one method would be as follows:
- The sensitive data is encrypted with the recipient’s public key, which could have been published either on- or off-chain
- The encrypted data is sent either to a smart contract written for the purpose of receiving it, or in the data field of a normal transaction
- The received data is decrypted using the recipient’s private key
Why are privacy and anonymity important?
If we consider the case of a private or consortium blockchain run by a group of companies, then certain aspects of privacy are clearly important.
For example, companies A, B, and C create a consortium blockchain. Although the network is closed to the outside world, any transactions between A and B are visible to company C, which could be undesirable for a number of reasons, especially if the three companies are in competition with each other. In addition to any business-to-business (B2B) interactions between the companies themselves, it’s possible the network would involve some form of business-to-customer (B2C) interaction, meaning further types of confidentiality and privacy would be required.
Before we look in detail at this recipe’s project, let’s take a look at the currently available Ethereum-based business platforms.
The Ethereum Enterprise Alliance
The Enterprise Ethereum Alliance (EEA) is a nonprofit working group whose aim is to define an open, standards-based architecture for business and enterprise use of Ethereum. Group members include many large companies from the world of software and finance, such as Microsoft, Accenture, and J.P.Morgan. The group is currently working toward defining the Ethereum Enterprise Architecture Stack, which itself will help guide the development of the EEA’s overall standards-based specification.
EEA’s specification will help define a standard way in which Ethereum’s blockchain can be used for business purposes. However, given the opensource nature of Ethereum’s codebase, adhering to EEA’s specification won’t necessarily be required: Ethereum’s codebase can be used in any way desired.
Ethereum’s licensing
According to Ethereum Foundation, different parts of the Ethereum technology stack are licensed – or will be licensed—in different ways.
The core parts of the stack, including the consensus engine, networking code, and supporting libraries, haven’t yet been licensed, but are expected to be covered by either the MIT, MPL, or LGPL license. It’s important to understand that the first of these is a permissive license, while the latter two are more restrictive, and restrict a user’s ability to distribute any modifications under commercial terms, therefore potentially restricting how businesses can use the code.
Regardless of the potential future restrictions that may appear once licensing is confirmed, several enterprise-centric implementations of Ethereum have already been created, with two such implementations being Quorum, created by J.P.Morgan, and Monax. Later in the recipe we will be discussing and implementing a project based on Quorum.
Blockchain-as-a-Service
Blockchain-as-a-Service (BaaS), is a service that allows customers to create and run their own client nodes on popular public blockchain networks, or to create their own private networks for their own use and testing. The provisioned services are based in the cloud, and are analogous to the Software-as-a-Service (SaaS) model. Several of the large cloud computing providers are now offering BaaS, such as the following:
- Microsoft Azure enables users to quickly deploy and manage applications in the cloud, and has templates available for Ethereum, Hyperledger, and R3’s Corda.
- AWS Blockchain Templates is a very similar offering, and again provides templates for Ethereum and Hyperledger. Further services are offered by IBM, HP, and Oracle, all along the same lines.
Quorum
Quorum (https://www.jpmorgan.com/global/Quorum) is an enterprise-focused fork of the Ethereum codebase, and offers a private blockchain infrastructure aimed specifically at financial use cases. It was created by J.P.Morgan, and claims to address three of the topics that would make a public blockchain network unsuitable for business use:
- Privacy, of both transaction and contract data
- Higher performance and throughput Permission
- and governance
Privacy
To achieve its headline feature of making transactions and contract data private, Quorum builds on Ethereum’s existing transaction model, rather than completely redefining it.
Public transactions and public contracts are visible to everyone on the network, and make use of Ethereum’s existing infrastructure—Quorum doesn’t implement anything different in this respect. As well as these public transactions, Quorum offers the ability to mark transactions as private, making them visible only to the intended recipients.
This privacy is achieved using public-key cryptography, specifically by setting the recipient’s public key in a new, Quorum-specific transaction parameter, privateFor. This allows the transaction to be encrypted, and therefore read-only by the owner of the private key.
Higher performance and throughput
Private and consortium blockchains are only open to certain authorized parties, rather than completely open to the public. As a consequence, there is little need for the consensus algorithms that are usually used between distrustful parties, namely PoW in the case of the Ethereum main network.
Quorum ultimately aims to offer pluggable and changeable consensus algorithms, and currently there are two to choose from:
- Raft-based consensus: This provides a much faster block time—in the order of milliseconds as opposed to seconds—as well as transaction finality, meaning once a transaction is placed into a block, there comes a point where it is impossible to remove it. A further difference from PoW is that this mechanism only creates blocks when there are transactions ready to go into them. It doesn’t create empty blocks in the way that the Ethereum main network does.
- Istanbul Byzantine fault-tolerance: This is a PBFT-inspired algorithm that again includes short block times and transaction finality.
Permission and governance
The third of Quorum’s headline features is its ability to permission only chosen nodes to join a given network. This is achieved by each node in the network that has a whitelist specifying the remote nodes that are permitted for both inbound and outbound connections. We will later look in detail at a practical implementation of this.
The Quorum client
At a high level, a Quorum client consists of the following components:
- Quorum Node: This is based on the Ethereum Geth client
- Constellation: This is a general-purpose system for submitting information in a secure way, and is itself composed of Transaction Manager and Enclave subcomponents, as shown in the following diagram:
Souce: https://github.com/jpmorganchase/quorum/wiki/Quorum-Overview
Quorum Node
Quorum Node is based on Ethereum’s Geth client, but with a series of changes:
- Ethereum’s PoW consensus algorithm has been replaced. There is currently a choice of two alternatives, which were briefly discussed previously.
- The P2P protocol has been modified to allow only authorized nodes to connect, preventing connections from the wider public.
- The way Ethereum maintains and records its state has been changed to allow public and private states to be handled separately in two different Patricia Merkle trees.
- Block validation has been altered to handle private transactions as well as public transactions.
- The gas-pricing mechanism has been removed, though the concept of gas is still used.
So far, Geth has been altered in such a way to allow both public and private transactions to occur.
Constellation
Constellation could be considered a P2P system in which each node in the system can act as a distributed key server and mail transfer agent (MTA), using PGP encryption to encrypt messages. It is this part of Quorum that allows for transactions to be made private, and is composed of two parts: the Transaction Manager and Enclave.
The Transaction Manager handles transaction privacy by storing and making transaction data available, as well as exchanging transaction payloads with Transaction Manager instances on other nodes. It does not have access to any of the private keys required for encryption, which is the job of the Enclave. The Enclave’s job is to handle both key- generation and the encryption of transaction data. As its name suggests, handling these jobs eparately allows sensitive operations to be isolated.
Our project
To demonstrate Quorum’s privacy characteristics, we will set up a small private network using an online example as a basis. Through this, we will explore how the nodes communicate, and show how private transactions—in this case, in the scope of interacting with a smart contract—can be broadcast to the network while being accessible only by the intended recipients.
Prerequisites
While it would be possible to set up our network manually, it will be cleaner to do so in a separate environment that we can bring up and take down easily.
As a first step we’ll need to install the following applications:
- VirtualBox: https://www.virtualbox.org/wiki/Downloads
- Vagrant: https://github.com/hashicorp/vagrant
VirtualBox is a virtualization application that will let us run our network inside a virtual machine, while Vagrant is a tool that will allow us to configure the environment easily so that our network can be initialized and run on command. Once the two applications are installed, we can move onto installing our Quorum-related code.
We first want to download the appropriate code from Quorum’s GitHub repository. If Git isn’t already installed on your system, now would be the time to do so (see https://git- scm.com/downloads):
git clone https://github.com/jpmorganchase/quorum-examples cd quorum-examples
The quorum-examples/ directory contains a file called Vagrantfile, which will be read by Vagrant when it starts. This file defines the parameters of our VirtualBox VM, and includes the ports that will be associated with each node in our network. It also calls a script, called vagrant/bootstrap.sh, which is run to create the node software from scratch.
Running the following will create a virtual machine in VirtualBox and configure it from the
Vagrantfile file:
vagrant up
The first time this is run, it will take somewhere in the order of 5-10 minutes. Vagrantfile defines the image that should be run on our VM—in this case, an Ubuntu Xenial image—which has to be downloaded. On subsequent runs, the startup time will be much quicker. When it has finished, we will have our configured virtual machine. To interact with our new environment, we have to connect to our virtual machine using SSH on the command line:
vagrant ssh
Our instantiated VM comes with a predefined network that we’ll use for our examples, which can be found inside our virtual environment under the quorum-examples/7nodes/directory. As the name suggests, our test network will consist of seven nodes in total.
Bringing up the network
Inside the quorum-examples/7nodes/ directory are the scripts we will use to bring up our network, detailed as follows.
The first script is raft-init.sh, which is run only once and performs the following actions:
- Initializes the Quorum Node part of each node by running geth init — datadir qdata/dd<x> genesis.json, where <x> is the number of the node. This uses the same genesis file for each node, meaning they all share the same network.
- Creates the key pair for each node and places it in each node’s qdata/dd<x>/keystore directory.
From inside quorum-examples/7nodes/, run the following:
./raft-init.sh
The second script is raft-start.sh, which is used to start the network properly. This script instantiates each node by doing the following:
- Runs the Geth client with appropriate parameters, thereby starting the Quorum Node component of the node.
- Runs the constellation-start.sh script, which starts the Constellation component of each node, creating the appropriate qdata/c<x> directory, and copying the Transaction Manager keys created by the initialization script to the proper place.
From the same directory, run:
./raft-start.sh
When this completes, you will see the following message:
All nodes configured. See ‘qdata/logs’ for logs, and run e.g. ‘geth attach qdata/dd1/geth.ipc’ to attach to the first Geth node.
Having brought up the network, it will now look like the following diagram, which shows three of the seven nodes in the network. The Quorum Node portions of the nodes are connected via Geth’s P2P protocol, while the Transaction Managers are connected via Constellation:
When we need to tear down the network, we can use a third script from the same directory, stop.sh.
Interacting with the network
When we started the Geth clients during the second step, we also created IPC endpoints under each node’s data directory. Using these endpoints, we can attach to each of our nodes. First, open three new terminals, navigate to your host machine’s quorum- examples/ directory in each window, and run vagrant ssh to connect each of them to the VM.
From within the environment, using the familiar Geth commands, we can attach to three of the running nodes in our network. For example, to attach to node 1, perform the following:
geth attach qdata/dd1/geth.ipc
This provides us with a familiar JavaScript console through which to interact with the node. For this tutorial, we recommend attaching to nodes 1, 4, and 7, each in their own Terminal window. Repeat the preceding step for each of these nodes, substituting the appropriate number in the command.
Testing the network
To test that our network is working as expected, we’ll run two tests using a contract that comes prepackaged with the Quorum code, simplestorage.sol:
pragma solidity ^0.4.15;
contract simplestorage { uint public storedData;
function simplestorage(uint initVal) { storedData = initVal;
}
function set(uint x) { storedData = x;
}
function get() constant returns (uint retVal) { return storedData;
}
}
This is a very simple contract that will allow members of the network to read and write a variable on the condition that they have authorization to do so. We’ll first deploy the contract publicly, check that all nodes are able to interact with it, and then redeploy the contract in such a way that only two of the nodes are able to interact with it.
To deploy the contract, we can make use of the runscript.sh script, together with the following:
- public-contract.js
- private-contract.js
These preprepared files contain references to both the ABI and bytecode required to deploy the contracts using Web3. If we look at the two files, we can see that the ABI and bytecode are identical – we’re deploying exactly the same contract. The only difference is how the contract is deployed.
For public-contract.js, we use the following:
var simple = simpleContract.new(42, {from:web3.eth.accounts[0], data: bytecode, gas: 0x47b760}, function(e, contract) { … }
And for private-contract.js, we use this code:
var simple = simpleContract.new(42, {from:web3.eth.accounts[0], data: bytecode, gas: 0x47b760, privateFor: [“ROAZBWtSacxXQrOe3FGAqJDyJjFePR5ce4TSIzmJ0Bc=”]}, function(e, contract) {
… }
In each case, the simple variable is the reference to the newly deployed contract. The initial value passed to the constructor is 42, and the contract is deployed by the first account on node 1. The specific difference is that the private contract is deployed with the privateFor parameter, with a public key being passed as the value. The transaction being used to deploy the contract is therefore private, and intended only for the owner of the specified public key.
We can check which node this public key equates to by checking the keys held by the Constellation Transaction Manager on each node. Upon inspection, we see that this equates to node 7, meaning once the contract is deployed, only nodes 1 (the deployer) and 7 will be able to access and view the data belonging to the contract as shown here:
$ cat examples/7nodes/keys/tm7.pub ROAZBWtSacxXQrOe3FGAqJDyJjFePR5ce4TSIzmJ0Bc=
Deploying the public contract
Let’s first deploy the public contract by running the relevant scripts from the console of our main session. It’s not necessary to attach to any of the nodes as runscript.sh performs the attach for us:
$ ./runscript.sh public-contract.sh
Having deployed our public contract, we want our nodes to be able to interact with it, and to do so requires each of our three nodes to know the contract ABI and address to which it’s deployed.
Using the transaction hash returned during the preceding deployment, we can get the contract address by running the following from one of the nodes directly, from within one of the geth attach consoles:
> eth.get.transactionReceipt(“<transaction_hash>”);
From here, we can read the contractAddress value, which we can set to a variable for ease of use. Make sure you use the address output by your particular instance:
> var address = “<your_contract_address>”;
The ABI can be copied directly from the public-contract.sh script, and again assigned to a variable for convenience:
> var abi = [{“constant”:true,”inputs”:[],”name”:”storedData”,”outputs”:[{“name”:””,”ty
pe”:”uint256″}],”payable”:false,”type”:”function”},{“constant”:false,”input
s”:[{“name”:”x”,”type”:”uint256″}],”name”:”set”,”outputs”:[],”payable”:fals
e,”type”:”function”},{“constant”:true,”inputs”:[],”name”:”get”,”outputs”:[{
“name”:”retVal”,”type”:”uint256″}],”payable”:false,”type”:”function”},{“inp
uts”:[{“name”:”initVal”,”type”:”uint256″}],”type”:”constructor”}];
We can then use these values in our node consoles to generate a reference to the deployed public contract:
> var public_contract = eth.contract(abi).at(address);
These steps should be followed for each of the nodes we have attached to. Now that we have a reference to the deployed contract, we can read the value of the storedData variable that we set in the constructor by calling the contract’s get() function.
From our Terminal attached to node 1, use the following:
> public_contract.get() 42
Repeating the same command on each of the other attached nodes will yield the same result. All the nodes can read the contract because it was marked as public – or, rather, it was not marked as private—during deployment.
Deploying the private contract
Now, let’s test that by deploying a private contract if we make data visible only to those nodes or not that are authorized.
Using the same methodology as before, let’s first deploy our private contract:
> ./runscript.sh private-contract.js
Then, follow the same steps to allow each node to understand the contract:
- Get the contract address from the returned transaction hash.
- Set the address to a variable.
- Get the ABI from the .js file and set it to a variable. (Note: This is the same for both private and public contracts, so there’s no strict need to update this.)
- Get a reference to the contract using the address and ABI:
> var private_contract = eth.contract(abi).at(address);
We can now show Quorum’s privacy in action. On nodes 1 and 7, we are able to read the variable and be returned the expected value:
> public_contract.get() 42
On node 4—or any other node, if we attach to them—we are unable to read the value as shown here:
> public_contract.get() 0
Great! We have demonstrated how Quorum is able to keep data private within a network by using its privateFor option when deploying a contract.
Permissioning the network
Unlike public networks, which are open for anyone to join, Quorum includes an explicit permissioning system. This can be used to specify both the nodes that can make inbound connections to a network, as well as which nodes can be connected to from inside a network.
In our network, the whitelist is contained in the permissioned-nodes.json file, the contents of which are shown as follows (the full node identifiers have been truncated):
[
“enode://ac6b1096…@127.0.0.1:21000?discport=0&raftport=50401”, “enode://0ba6b9f6…@127.0.0.1:21001?discport=0&raftport=50402”, “enode://579f786d…@127.0.0.1:21002?discport=0&raftport=50403”, “enode://3d9ca595…@127.0.0.1:21003?discport=0&raftport=50404”, “enode://3701f007…@127.0.0.1:21004?discport=0&raftport=50405”,
“enode://eacaa74c…@127.0.0.1:21005?discport=0&raftport=50406”, “enode://239c1f04…@127.0.0.1:21006?discport=0&raftport=50407”
]
When the network was brought up, the raft-init.sh script copied the same whitelist to the data directory of each node (qdata/dd<x>/). When we then called raft-start.sh, each node was started with a –permissioned flag, meaning the permissioned- nodes.json file was parsed, and the permissioning came into effect. Every node on our network could connect to every other node.
To test the permissioning system, we will remove one of the nodes from the whitelist and check whether it can see other nodes in the network. Start by taking down the network from the original console you ran vagrant ssh on:
./stop
Next, we want to remove the original JSON file from the data directory of each node:
rm qdata/dd*/static-nodes.json
rm qdata/dd*/permissioned-nodes.json
Then edit the master copy of permissioned-nodes.json to remove node 6 using your favorite editor. Having done that, we can reinitialize the network using ./raft-init.sh, before restarting it with ./raft-start.sh.
To check that our changes have worked, attach to node 1 (geth attach qdata/dd1/geth.ipc) and check the active connections using the following:
> admin.peers
The output should show that node 6 is now excluded, meaning that it’s unable to connect to any nodes, nor be connected to. Our network permissions are now in force.
Conclusion
In this relatively short recipe, we learned how to create a private Quorum network inside a virtual environment. We then explored how transactions can be made private by using the public keys of those nodes that we want to have access to the transaction data, or in our specific case, the data associated with a smart contract. We then briefly looked how our network could be permissioned using Quorum’s whitelisting and the –permissioned flag.
If you like to explore blockchain development with an alternative platform like Hyperledger or learn about the projects of Hyperledger like Sawtooth or Iroha, visit Comprehensive Hyperledger Training Tutorials page to get the outline of our Hyperledger articles.
To conclude this recipe, we like to recommend Blockchain Hyperledger Development in 30 hours, Learn Hands-on Blockchain Ethereum Development & Get Certified in 30 Hrs, and Become Blockchain Certified Security Architect in 30 hours courses to those interested in pursuing a blockchain development career. This recipe is written by Brian Wu who is our senior Blockchain instructor & consultant in Washington DC. His books (Blockchain By Example and Hyperledger Cookbook) are highly recommended for learning more about blockchain development.