A Crash Course on Blockchains
You're excited. So excited. The buzz has finally caught you; everywhere you go you hear "blockchain" this, "distributed ledger" that. You're ready to make your mark in Changing the World Through Blockchains (tm pending). You turn on your computer, your favorite text editor springs into action. Except...you are drawing a blank as copiously empty as the temporary file you haven't yet created.
You don't have the faintest idea how to use a blockchain. Or what a blockchain is, really.
Don't worry. Almost everybody who uses "blockchain" in a sentence is as confused as you are. They're just in too deep to admit it. But never fear, we're here to help! Below is a high-level, 30-minutes-or-less crash course on public blockchains, geared toward inquisitive developers who don't have a cryptography background. We use Bitcoin as the blueprint, but with a few differences it can extend to other public blockchains.
General Byzantine's Distributed Pizzeria
A long time ago, there was a General of grand pomp and import, who garnered quite a reputation for pizazz in his station. Upon retirement, he transformed his panache for pizazz into something more palatable: pizza. Thus was General Byzantine's Distributed Pizzeria born. It was a bit old-fashioned, but it was the best pizza in antiquity...if you could get it.
Sadly, the General's pizza venture was short-lived, primarily owing to his archaic method of delivery and accounting. The General was a great believer in delegation and autonomy...to a detrimental extreme. The General's pizzeria was not one, but many restaurants, each independently taking orders, delivering pizzas, and keeping its own running tabs on customers. If one franchise received an order but ran out of ingredients, for redundancy's sake it would send requests to several other nearby restaurants to complete the order, but without any consistent reconciliation system. Orders were constantly intercepted and modified by competitors and pranksters. Some customers would receive dozens of pizzas after ordering just one; others would have their tabs called in by several franchises independently, referencing dozens of phantom pizzas they never ordered.
But the pizzas that were successfully delivered? Magnifico. Best in the land.
Owing to runaway costs and severe mismanagement, General Byzantine's Distributed Pizzeria was shuttered two weeks after its grand opening. Well, three weeks. Unsurprisingly, it was hard to communicate the closure to all the General's pizza outposts. It was a beautiful, flawed experiment.
Why the story of his failed pizzeria? A few reasons: everybody loves pizza, and the first real-world transaction to ascribe value to Bitcoin involved a pizza delivery (a cool 10,000 BTC for a single pie, worth $20 then and $6 million as of this writing).
The Building Blocks Behind Building Blockchains: A Story In Three Parts
Not to be too hard on poor General Byzantine, but explicitly understanding his pizzeria's problems will help reveal how a blockchain could mitigate them.
Problem 1: Order and Pizza Integrity
The General couldn't ensure the authenticity of the orders sent between restaurants and customers, leading to modified orders, pizzas, and customer tabs (thanks to pranksters and competitors).
Problem 2: No Consistent or Authoritative Record
When a restaurant ran out of ingredients, it would send requests to many different restaurants to fulfill its order, potentially resulting in the pizza getting delivered many times, or not at all. Customer debts might be duplicated or dropped after haphazard reconciliation. The final record was a mess, because one didn't really exist.
Problem 3: The Double Pepperoni/Spend Problem
As a consequence of both problems above, some particularly sneaky customers may attempt to deliberately pay once for multiple pizzas; especially if they knew that a restaurant was running out of ingredients, or could intercept and modify messages about their tabs. Without a consistent, authoritative record, and no guarantee of message authenticity, what's to stop these thieves?
Ownership and Transferring Value: A Generalized Distributed Pizzeria
It should come as no surprise that inconsistency, data malleability, and double-spends plague systems beyond our hypothetical General Byzantine's pizzeria. Like supply chains with dozens of different participants, or individuals wishing to exchange value amongst themselves, or—perish the thought—even banks.
The typical solution to these problems is to simply designate a central (sometimes external) party as the ultimate arbiter and record. In our pizzeria example, that might mean all orders and customer tabs flowing into a single designated restaurant; in the bank case, they rely on SWIFT and credit card companies for payments, and central banks for loan capital. This is not a fundamentally bad solution, but it creates new risks by relying on a centralized party with out-sized authority on the final record. For banks, that translates into big security risks and effective monopoly pricing power for the centralized party in control (and why remittances, credit cards, and bank fees tend to be so damned expensive).
Enter Our Plucky Hero, The Blockchain
What if we could solve all these issues without relying on a single trusted party? Where every participant could download and update their own copy of the ledger, controlling the rows where they have authority, yet none can double-spend or create conflicting orders? What if we could make General Byzantine's Distributed Pizzeria work? What if these were more than just leading questions? (We can, they can, we can, they're not)
As you likely suspect, public blockchains—like Bitcoin and Ethereum—are the answer. Public blockchains provide a practical solution to our problems, without relying on a single trusted party to arbitrate consensus and data integrity. And here's the key insight why:
A public blockchain like Bitcoin is simply a set a of rules and incentives for a public, permissionless, immutable ledger.
The "how" it manages this magic requires more exploration, which we've divided into four parts below. The concepts are presented in the order necessary to understand a blockchain from its most fundamental unit first (transactions) before graduating to more abstract concepts (blocks, a chain of blocks, the consensus mechanism/incentives). Presenting it this way also conveniently matches solving our Byzantine pizzeria problems in the order they were presented.
Part 1: Public-Key Cryptography and Transactions
Let's start at the most fundamental unit of a blockchain: its transactions. Consider Alice and Bob (and yes, before you ask, their parents were cryptographers who couldn't spare precious imagination on naming them).
Bob is the manager of one of General Byzantine's pizza shops, and Alice is a customer. Alice wants to send Bob some bitcoin in exchange for pizza. So she sends a message to an open, public network of computers that transfers bitcoin from her control to Bob's control. In the most naive case, what prevents someone from modifying this message? If it's an open ledger that anyone can download and interact with, can't someone intercept Alice's message and change it?
Thankfully not! Because this message—which is really a transaction—takes advantage of Public-Key Cryptography. You can read more about how this works on its wonderful Wikipedia page, but here's a short explanation in a nutshell:
Take a random number, selected from a very large range (like ~2^256 large, in Bitcoin and Ethereum's case). This is your "private key."
Now from this private key, you can perform a series of mathematical operations to generate a new number, called your "public key." In good cryptosystems, it's easy to generate a public key from a private key, but nigh impossible to do the reverse; these operations satisfy the properties of a one-way function. The operations are arcane and complex, and depend on the cryptography you're using...in Bitcoin and Ethereum's case, it's ECDSA over the secp256k1 curve. If that doesn't mean much to you, that's okay; you can accept that it's fancy and trust it works, but again, Wikipedia has a nice explanation for the interested reader.
Now here's where things get "taken to 11"...mathematically. With your private key, you can "sign" arbitrary data, and then anyone with the resulting data from this operation (called a "signature") and your public key can mathematically verify that you signed the data...without ever revealing your private key! It's kinda like affixing your handwritten signature to the bottom of a message/document in real life. But unlike a handwritten signature, this one is nigh impossible to forge short of stealing the private key, and signing data doesn't reveal you name, private key, or private information.
So to recap: with a public-private key pair, you can demonstrate you signed messages with a particular private key without ever revealing the private key! What does this have to do with Alice and Bob?
Well, imagine instead of Alice's and Bob's name in this shared ledger (or heaven forbid, checking/routing number) you have Alice's public key and Bob's public key, and the associated amounts of bitcoin. When Alice sends bitcoin to Bob, she can send a transaction that says "take the bitcoin associated with my public key, and instead, associate it with Bob's public key," along with a signature that validates this message. Then anyone can verify that she was the right person—or more appropriately, her private key was valid for her public key—to send it! And Bob can do the same thing for future transactions with his public key. Again, this is a simplification and the actual rules differ a bit between Bitcoin and Ethereum, but it's a fair simplification.
Now imagine that we set up a ledger that requires valid signatures derived from public-private key pairs for all transactions, and everyone agrees to run this version of the ledger. Now if someone tries to modify Alice's message without her private key, the modification would be rejected by all participants, and the ledger would not be updated.
Public-key cryptography is central to a public blockchain's permissionless and secure nature. Since the number of possible private keys is such a mind-bogglingly large number (2^256 is on the order of 1.15 * 10^77, which is close to the number of atoms estimated to exist in the Universe) you can generate one offline with a practical guarantee that nobody else will generate the same one. You can then generate a public key from this private key, copy it to an online computer, and then distribute this public key widely to accept payment. Used this way, public-key cryptography is not only vastly more secure than traditional means of payment acceptance and storage, it also enables a user to generate a practically limitless number of "accounts" offline, without ever having to ask anyone's permission to participate in the system.
Put more succinctly: if you can generate a 256-bit number, you can accept bitcoin (and ether, in Ethereum's case). Neat huh? And to get a little ahead of ourselves, since a public-private key pair is just a couple of numbers, "you" doesn't have to mean a person; machines can generate addresses to pay or send value to other humans or machines! Think of the possibilities...
But Wait, None of This Means We Have An Authoritative Record or Solves Double-Spends
Right you are, astute, idealized reader! There's nothing particularly novel about public-key cryptography in and of itself; it's a powerful, prevalent technology in today's world, but it only solves part of the blockchain puzzle. Consider our current take on the Alice and Bob pizza problem: Sure, with signed messages, you can cryptographically validate that Alice sent the message saying "give Bob my bitcoin," and Bob received it...but how do you know that Alice didn't send the same message to dozens of others, attempting to send her same "bitcoin" many times over? How does one actually reach a consistent, authoritative ledger?
To get there, we need to introduce another cryptographic construct, which will enable us to discuss how many transactions are structured into the latticework of blockchains...and in the process answer where the term "blockchain" got its name.
Part 2: Cryptographic Hashing and Blocks
With just public-key cryptography, we have verifiable messages that ensure transaction integrity and authenticity, but we cannot stop Alice (or any other participant) from attempting to double-spend transactions, irreconcilably drifting everyone's copy of the ledger into disarray. To tackle this problem, we need to introduce a new cryptographic construct into our repertoire: The Cryptographic Hash. Luckily it's simpler to understand than public-key cryptography, but it's just as critical.
Much like a signature from public-key cryptography, cryptographic hashing lets you prove data is valid...but instead of proving it's valid from a private key, you're proving it's valid from the data itself. Confused? Now that we've read that sentence back, so are we; let's build on our example to help.
Let's say I have a bundle of transactions from Bob's pizza shop. They look something like this:
Alice sent Bob 1 bitcoin.
Chris sent Bob 2 bitcoin.
Dave sent Bob 5 bitcoin.
Bob sent Eve 3 bitcoin.
Let's call this list of 4 transactions a single page in our ledger...or a "block" if you will (hint, hint). We can use a cryptographic hashing function to create a unique ID for each transaction, based on the data itself. Let's try doing that with SHA256; a cryptographic hash function used extensively in Bitcoin. Let's start with the first line:
Alice sent Bob 1 bitcoin.
We can use OpenSSL in the command line to generate the hash of the first row:
echo -n 'Alice sent Bob 1 bitcoin.' | openssl sha256
The output is a 32 byte (256-bit) hash, encoded in hexadecimal. Note that this value is unique for the message 'Alice sent Bob 1 bitcoin.' Let's change it to say 'Alice sent Bob 1.1 bitcoin' and see what happens to the hash:
echo -n 'Alice sent Bob 1.1 bitcoin.' | openssl sha256
Despite the initial data—which is also known as a "pre-image" in fancy cryptography terms—changing only slightly, the resulting hash looks very different. This is a desirable outcome for a cryptographic hash; you want the output to appear very random, and change dramatically even if the "pre-image" changes only slightly. Without knowing the initial data, the output will appear very random, and it would be very difficult to predict the input...the only way to do so is to brute-force the inputs (that is to say: guess a piece of data, run your cryptographic hashing function on this guess. If the output isn't what you're looking for, you repeat the cycle with a new guess...over and over and over again).
So let's go back to our list of transactions and convert all of our messages into SHA256 hashes; here's what our list looks like:
You don't need to take my word that this is the list; you can verify that these are the right hashes by using the command line on our original list, or a website like this.
Armed with this list of hashes, it's possible to generate a single, cryptographically verifiable hash (and ID) for this block of transactions. How? By hashing the hashes! The particular structure used by blockchains for transactions is typically a Merkle tree. Once you know how cryptographic hashes work, it's easy to understand; you just concatenate your hashes into pairs, then find the hash of each pair. With those hashes, you do the same thing...and if you have lots of hashes, you keep doing it until you have a single hash (known as the "Merkle root"). Let's try it with our list of hashes. First, we concatenate the two pairs (shortened here for readability's sake).
Then we apply SHA256 onto each of these; in case you're wondering, we pipe through xxd to convert our hex string into raw binary data. (Again, hashes shortened for readability)
echo -n '96f9ee71...' | xxd -r -p | openssl sha256
echo -n '4b1267af...' | xxd -r -p | openssl sha256
Our list went from 4 to 2 hashes:
Now we do this whole process again, resulting in our final Merkle root:
echo -n 'a4e0820e0743...' | xxd -r -p | openssl sha256
Tadaa! Now we have a single hash that represents—and validates!—our list of transactions, using the data of each transaction. Consider what would happen to this hash if you modified a single transaction's data. It would change the hash of the transaction, which would then cascade through the Merkle tree, changing all of the branches' affected and eventually modifying the root.
This is pretty much how transactions are structured in blockchains. A "block," equivalent to a page in an old-fashioned accounting ledger, contains both the data of each transaction and each transaction's cryptographic hash, along with a Merkle root representing all transactions contained therein.
While this is a bit of a simplification (transactions aren't written in English and we're missing tons of detail), it's a fair demonstration of how cryptographic hashes validate and structure a block. But even with this structure, it's unclear how this could solve the double-spend problem, or help us achieve group consensus on our ledgers. If everyone is sending transactions to each other and assembling blocks themselves, and Alice wanted to be mischievous, there's nothing about this structure that would prevent her; she'd simply have to send separate transactions to different parties assembling these blocks.
But don't fret! Now that you understand public-key cryptography, hashes, and block structure, we can solve that final riddle. And boy is the solution a doozy.
Part 3: ...Chaining Blocks, Reaching Consensus? Always. Be. Hashing.
As previously shown, a cryptographic hash function like SHA256 generates hashes that are 32 bytes (aka 256 bits) long. In essence, SHA256 takes any arbitrary data, and outputs a 256-bit number (presented in our examples above in hexadecimal).
Knowing what we know about SHA256, lets ponder a puzzle. Assuming the output of SHA256 is randomly distributed given random input data, what's the probability that the output of SHA256 on any piece of data is below 2^256? Well, it's 100%. All 256-bit numbers are below 2^256, from 0 to 2^256-1.
How about below 2^255? Remember that 2^256 / 2 = 2^255...so you are effectively halving your "success" condition for this probability. So we go from 100% to 50%. Now how about 2^254? You halve your success space again, from 50% down to 25%. You can see the pattern...the smaller your desired SHA256 output, the smaller the probability you'll find one. If you want to find a particularly low number, it may take millions, billions, trillions, quadrillions, or even quintillions of guesses!
But remarkably, it's very easy to verify that your solution is correct; all someone need do is take your data, apply the SHA256 hashing function, and verify the answer is below the amount they require. This is a desirable feature of cryptographic hashes; they are one-way functions—like deriving a public key—but in the context of cryptographic hashes, it's difficult to guess without having the pre-image/prior data, but easy to verify.
When you broadcast a pre-image and hash that meets these criteria, you're publicly demonstrating that you spent a lot of computational time guessing the hash, which anyone can verify quickly. Or more succinctly, you've shown Proof-of-Work. In a sense, Proof-of-Work (PoW) is a proxy for voting with computing power, but cryptographically tied to the data it hashed. Or to put another way, it's a way of notarizing data with computational power. "This data is valid to me, and to prove it, here's the time and money I spent on computing a hash that meets your criteria."
Aye, this is just the trick we need to conjure some blockchain magic. Let's add some new rules to our hypothetical open/shared block-based ledger:
- All transactions in a block must have valid signatures, a la public-key cryptography, and all transaction inputs must spend money from previous outputs (e.g., if Alice's public key only has 1 bitcoin from a prior transaction, she can't send 2 to someone else from this public key). Thanks to public-key cryptography, any participant can verify these are valid without revealing private keys.
- Upon receiving a transaction, participants can choose to include it in a candidate block that they're in the process of assembling.
- Every block hash must be lower than a number generated by an agreed upon difficulty formula. Some pieces of data in the block can be randomly adjusted without affecting transaction validity (to enable participants to generate new block hashes).
- The difficulty formula is self-adapting based on how quickly or slowly blocks with hashes below the difficulty threshold were found; in Bitcoin's case, the protocol adjusts the difficulty to target a 10 minute average block confirmation time.
- All blocks contain a hash of the previous block (which includes transactions, transaction hashes, the Merkle root of all transactions), thus chaining them together.
- The longest chain with the most Proof-of-Work is the authoritative ledger.
And that, good reader, is a blockchain! How exactly does this setup prevent mischievous Alice from double-spending her pizza money? Let's run through an example, where we have four participants with equal computing power; Alice, Bob, Chris, and Dave. Alice creates a transaction that sends money to Bob's public key; we'll call it 1 bitcoin for simplicity's sake. So it looks like this:
To Bob, Chris, Dave (75% of PoW computing power):
Alice sends Bob 1 bitcoin.
She broadcasts this transaction to Bob, Chris, and Dave. BUT, being sneaky, she creates a different transaction that takes this same bitcoin and sends it to another public key she controls (remember, it's very easy to create new public-private key pairs), and decides to use her computing power to attempt to find a block before everyone else. This transaction looks like this:
To Alice (25% of computing power):
Alice sends Alice 1 bitcoin.
This transaction sits in all the participants "mempools"; a staging area for transactions that haven't yet been included in blocks. They are all working in parallel to find a hash lower than an agreed upon difficulty. Assuming this simple distribution of computing power, 25% of the time, Alice would discover a hash that creates a valid block (with bitcoin sent back to her), while 75% of the time, Bob, Chris, or Dave would discover a hash that creates a valid block (sending Alice's bitcoin to Bob).
Already, the odds are against Alice; at first glance it appears she'll only be successful 25% of the time. However, it's even worse in practice. Because Bob doesn't fully trust the other participants—having been burned by Alice's cyberized five-finger discount before—he decides to wait until Alice's transaction has been "confirmed" by being deep in the blockchain; that is, by waiting until her transaction has several discovered blocks ahead of it. Consider a case where Bob says he will wait 6 blocks. Now, in order to double-spend Bob, Alice will have to "beat" the other three participants after they discover 6 more blocks; in other words, she'll have to find the same number of lucky hashes, with a lot less computing power. There's a mathematical discussion in the original Bitcoin paper that explains the probability in detail.
This Proof-of-Work consensus mechanism works marvelously, as long as at least ~51% of the participants are honest. In fact, a public blockchain with these constraints practically solves a difficult problem in computer science: the Byzantine Generals' Problem. And yes, our own hypothetical General Byzantine's eponymous name was not a coincidence.
Why Hash At All? Satoshi's One Weird Trick: Marrying Technology and Incentives
Our blockchain story is nearly complete. But one thing should probably be nagging at you...where do the "bitcoin" (or "ether") come from in this ledger? Since according to the rules listed above, you can never create more bitcoin in a transaction; you can only spend previously unspent bitcoin. But how does the bitcoin get minted into the ledger? Who has that power?
The answer is a testament to the elegance of Bitcoin's enigmatic founder, Satoshi Nakamoto. The best way to answer that question is to first ask another question: why should anyone agree to engage in Proof-of-Work for chaining blocks? It takes lots of computational power, and while it's a public service to secure the network, that's not usually a compelling incentive...but what if you incentived those who secure the network with the power of minting currency?
That's exactly what Satoshi designed! Every block contains what's known as a "coinbase" transaction, which includes freshly minted bitcoin. That is why these players in the blockchain ecosystem are known as "miners." The number of bitcoin given as a reward is coded into the software that describes the protocol; it has a known schedule, and if you attempt to violate it with a newly discovered block, it will be soundly rejected by other participants. The native token schedule differs by blockchain; for Bitcoin there will only ever be 21,000,000 bitcoin (each one sub-divisible to 10^8) making it a naturally deflationary currency. Ethereum, meanwhile, has no hard cap, but its inflationary rate decreases over time.
In addition to the "minted" bitcoin/ether (which are known as the "block reward"), anyone who discovers a block is also entitled to fees that are added to all transactions within that block. This is to incentivize including transactions, especially in Bitcoin's case, where the block reward halves every 4 years and will eventually entirely disappear.
It's this alignment of incentives and technology which marks the difference between a simple collection of preexisting cryptography, and Bitcoin/Ethereum. At the time of this writing the Bitcoin and Ethereum ecosystems are collectively worth $11bn, and are both protected by an unfathomable amount of computing power. And thanks to this incentive alignment, there are finally free, permissionless, secure, immutable ledgers that have the potential to change the world in ways we can't even imagine. Perhaps thanks to blockchains, General Byzantine's Distributed Pizzeria might actually work...well, at least the payment and supply management components. No idea how he's going to stop people stealing his delicious pizzas en route.
Take a Deeper Slice of Blockchains: Further Reading/Viewing
There is much, much more to blockchains; this is but a humble, developer-friendly introduction. But for the curious reader, there's a wealth of free information out there on the web. Here's some suggested reading/viewing to get you started, presented in recommended order:
Khan Academy Bitcoin Introduction
Khan Academy did a wonderful video series about Bitcoin which covers the topics raised here in an approachable, but technical, manner. You can start the first video here.
Original Bitcoin Whitepaper
On a fateful Halloween Day in October, a pseudonym named Satoshi Nakamoto posted their world-changing research in a succinct, surprisingly matter-of-fact manner: "I've been working on a new electronic cash system that's fully peer-to-peer, with no trusted third party." The Bitcoin Whitepaper remains one of the most cogent explanations of Bitcoin's workings, a remarkable feat for the paper that introduced the concept. Read it here.
The Bitcoin.org Developer Reference
Now that you have a basis for understanding the world's most prominent blockchain, it's time to dive into the weeds. The Bitcoin.org Developer Reference is a great place to add depth to your understanding.
Ethereum, Solidity, Smart Contracts, Oh My
Imagine someone took the clever ideas behind Bitcoin's blockchain, and said, "if it's good enough for embedding static data and transferring money, it's good enough for a general state machine." That's Ethereum. While newer and smaller than Bitcoin, in its 2 years of existence it has catapulted to the second largest blockchain by market capitalization and one of keen developer interest. We just started supporting it ourselves, and plan on expanding our API services to include many of Ethereum's advanced features. Ethereum has a number of differences compared to Bitcoin: accounts have balances of a native currency called "ether," there can be self-executing "accounts" that have fully-baked programming logic to allow exotic, conditional exchanges...hence why Ethereum calls itself a "world computer." It has much more flexibility than Bitcoin, but is also a fair bit riskier. Here are a few places to start your journey into Ethereum:
- Ethereum's Design Rationale; How It Differs From Bitcoin
- Ethereum Documentation
- Learn Solidity, Ethereum's Smart Contract Language
Now, at long last, we recommend checking out our API services. BlockCypher is like a bridge for blockchains; instead of running your own nodes, our API scales and manages connections to public blockchains so you can spend your time working on your game-changing Bitcoin/Ethereum application. Our own reference documentation is available here:
- Bitcoin, Litecoin, Dogecoin, Our Own Bitcoin-Compatible Testnet
- Ethereum and Our Own Ethereum-Compatible Testnet
No matter where your curiosity takes you, we wish you the best of luck on your journey into the wild and wonderful world of blockchains. We are still early in this experiment, and already blockchains and cryptocurreny are changing the way millions of people engage in their economies, both locally and globally. Projects taking advantage of blockchains run the gamut from helping bring the unbanked online to building partially autonomous organizations through smart contracts. A better economy is yours to build. And if you happen to successfully build a distributed pizzeria, definitely let us know; it's an idea whose time has certainly come.