Compose Database-as-a-Service Help and Documentation

Everything you need to know about Compose, Hosted or Enterprise, is here in our help system. Whether you run one database for your businesses' sole application or six different databases to support an entire corporation, we've got the information you need.

etcd Playground

What Is etcd?

At its heart, etcd is an open-source, distributed key-value storage. Any introduction to etcd would be incomplete without a brief introduction to the RAFT consensus algorithm. As intimidating an opening concept as a consensus algorithm may be, it is easier to digest if we start from the end result and then go back to the concept we used to get there.

etcd specializes in keeping a set of data consistent. Consider a scenario where you have a cluster consisting of 3 members. The cluster will only report on the contents of its data if each of the 3 members agrees upon what the data should be. A consensus algorithm is used to establish the agreement. If a request for data does not have the cluster unanimously agreeing on what the response should be, no response is delivered.

A perfect use-case for a truth-enforcing system like this would be distributed configuration files, where uniformed consistency and redundancy are paramount. Putting what we know together, we understand that etcd is a distributed version of the configuration-maintaining directory, /etc. The RAFT consensus algorithm, while daunting in initial complexity, is the intelligence keeping the distributed cluster in-sync, and afloat.

Talking To etcd

This guide assumes that you're using OS x. If you're using Linux, swap out brew for apt-get.

brew install etcd

First, we need to install etcd. We need this installed locally in order to facilitate a connection to our cluster. You can also use curl to communicate with your cluster, but the commands are significantly more verbose and complex. Installing etcd locally is a small price to pay for the convenience of etcdctl.

After you have spun up an etcd deployment, click on the 'Overview' tab within the dashboard.

Here, you'll be able to reveal your cluster's username and password within your 'Command Line' snippet. We'll use this to connect to your newly minted 'etcd' deployment.

$ etcdctl --no-sync --peers aws-us-east-1-portal8.dblayer.com:10168,aws-us-east-1-portal7.dblayer.com:10058 --username username:password ls /

Let's break down the commands and options we've used:

  • etcdctl: Makes it known we're communicating via etcd's api controller, etcdctl.
  • --no-sync: Prevents etcd from syncing cluster information before sending the request.
  • --peers: Allows you to specify the various cluster members, delineated by a comma. A default compose configuration will provide you with two haproxies to connect to.
  • --username: Username and password, separated by a colon.

Trailing the above is the command we're sending our cluster: ls /. Entering ls / when we have no keys, the cluster will respond with an affirmative '/ping' - success. We've asked our etcd deployment to list (ls) the contents of /, our root directory. All members agreed on a response, and a successful ping was returned.

Communicating With etcd

Unlike other data storage offerings you may have explored in the past, we don't connect "into" an etcd shell, but query the cluster (acting unanimously, thanks to RAFT) with a request, then await a response (if the cluster agrees on what the response should be).

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' set greeting 'Hello cluster'
> Hello cluster

We've done three new things:

  • Told etcd that we are going to 'set' a key.
  • Specified a name for our key: 'greeting'.
  • Attached a value to the key: 'Hello cluster'.

This has set data into each member of our cluster, and the cluster unanimously agrees on the contents.

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' get greeting
> Hello cluster

We've done two things:

  • Asked etcd to 'get' a key.
  • Specified a name for the key we wanted to 'get': 'greeting'.

This allowed us to get the value of our key to return.

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' rm greeting

Finally, we can DELETE keys and the values associated with them.

Directories

To demonstrate the directory-orchestrating power of etcd, let's create some.

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' set /master_configuration/secret_passcode 12345

We've made a request to set another key value and we've defined a directory structure containing two directories: master_configuration and secret_passcode.

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' ls /master_configuration
> /master_configuration/secret_passcode

etcd will automatically create the directories when you've set a value this way. Above, we've run another ls statement to retrieve the contents of the master_configuration directory. You can see the the secret_passcode directory has also been created.

Feature Exploration

We have taken a look into the basic tools and general function of etcd. You should now have an inkling of how you can use etcd to manage your configuration files and bring some sanity to multiple application/database type configurations. There are three powerful main features of etcd that you will want to explore now that you have a comfort level with the basics:

Wait For Changes

By sending GET requests with wait=true enabled, you will actively poll the queried key until the value changes. You will know immediately when values in your deployment have changed.

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' watch --recursive greeting 

Meanwhile, in another terminal, we will set a value within the greeting key.

etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' set greeting 'Hello watcher'

Now, observe the first terminal - powerful!

Time-To-Live

Directories can be created with a TTL value - a set expiry date. Working with encryption, or delicate/temporary configuration transmissions is a well-supported fit for etcd. We simply add the --ttl tag and provide a value for how many seconds the value is required to live for.

$ etcdctl --no-sync --peers https://aws-us-east-1-portal8.dblayer.com:10168,https://aws-us-east-1-portal7.dblayer.com:10058 -u 'username:password' set greeting 'Hello cluster' --ttl 60

Acid Compare-And-Swap & Compare-And-Delete

An all-or-nothing agreement is required before swapping or deleting key values - the requests and the clusters state is considered before swap and delete operations are executed. Failure to reach consensus returns an HTTP error. RAFT approved.

Try etcd

etcd is a useful and powerful tool. Hopefully, our basic introduction has given you some inspiration. By spinning up an etcd deployment with Compose, you'll have an auto-scaling, secure, stable, production grade cluster ready for whatever you'd like to throw at it.

We're excited to see what our community is able to accomplish with auto-scaling etcd clusters - if you have a unique use-case, we'd love it if you would consider submitting it to our Write Stuff program. If you're looking to chat about etcd, come have a chat with us in #compose on Freenode.


Still Need Help?

If this article didn't solve things, summon a human and get some help!

etcd Playground