Developer Notes: Setting up a Bitcoin Node on AWS

Recently I started doing some Bitcoin-related work, and needed to set up a Bitcoin node running the Testnet. For anyone interested, below is a step-by-step summary of what I did to get it running.

These notes assume some experience with AWS, Linux, and Bitcoin.

The basic flow is adapted from chapter 2 of the open-source e-book, Learning Bitcoin from the Command Line by Christopher Allen.

These notes are provided AS IS. USE AT YOUR OWN RISK.

Initial steps

• From the AWS console, navigate to EC2
• Click the Launch Instance button

Step 1: Choose an Amazon Machine Image (AMI)

• Click on the AWS Marketplace tab
• Search for Debian Stretch in the search box
• Find the AMI for “Debian GNU/Linux 9 (Stretch)” and click Select
• A box will appear with details and a list of example prices. Click Continue.

Step 2: Choose an Instance Type.

You will need an instance with 2-3 gigabytes of memory.

• Choose the t3.medium instance, which has 4GB of memory and can be connected to EBS (Elastic Block Storage) of any size.
• Click Next: Configure Instance Details.

Step 3: Configure Instance Details

There’s probably nothing you need to change here, but in my case the first time I tried launching my instance I was told that the instance type I wanted (t3.medium) wasn’t available in the us-east-1b availability zone and I had to come back here to change it to a different one, after which the launch succeeded.

• Click Next: Add Storage.

Step 4: Add Storage

The block chain is pretty big. You’ll need upwards of 19GB of storage (for the current Testnet) or 170GB of storage (for the current Mainnet).

• Click Add New Volume.
• Choose EBS, /dev/sdb for the Device column, 300 GB for the Size, and General Purpose SSD for the Volume Type.
• Select the Delete on Termination checkbox for your volume. That way when you dispose of your instance you won’t have to go through an extra step to dispose of the volume.
• Click Next: Add Tags.

Step 5: Add Tags

Adding a Name tag will make it easier to find your new instance, especially when you have more than one.

• Click Add Tag.
• Enter Name into the Key field and Bitcoin or something similar into the Value field.
• Click Next: Configure Security Group.

Step 6: Configure Security Group

• Select Create a new security group
• Enter bitcoind in the Security Group Name field.
• In the Description field enter something like, Ports and services necessary for running a Bitcoin node.

A rule allowing SSH access to your instance from any other IP address is already in place. The bitcoind client needs to talk through port 8333 for the Mainnet and 18333 for the Testnet. We’ll create rules allowing them both. We’ll also add another rule that opens the port used by the Lightning protocol.

• Click the Add Rule button.
• Select Custom UDP for the Type.
• Enter 8333 for Port Range.
• Select Anywhere for Source.
• Enter Mainnet in the Description field.

• Perform the last five steps again, but use 18333 for the Port Range and Testnet for the Description.

• Do it one more time, but use 9735 for the Port Range and Lightning for the Description.

• Click Review and Launch.

Step 7: Review Instance Launch

All the details of your instance are now presented for review. Assuming it all looks good, click Launch.

Select Key Pair

One more step. Access to your AWS instance is controlled by a public/private key pair (not covered here.) I already had an AWS key pair set up, so I simply selected Choose an existing key pair, the pair I wanted to use, and selected the box acknowledging that access to the private key file will be required to access the instance, then clicked Launch Instance.

Launching

The next page keeps you informed on the launch status. Assuming all goes well, you will receive the message Your instances are now launching and instance ID like this: i-0d881a693cb29c072.

• Click View Instances.

You should see your new Bitcoin instance in the list. Eventually the Instance State will change to Running. For awhile the Status Checks column will also say Initializing. Wait for this to change (reload the page if necessary) to 2/2 checks passed.

Log In to Your Instance for the First Time

From the View Instances page, click on your new instance in the table of instances. At the top you will find a Public IP field with an IP address like: xx.xx.xx.xx. This is an ephemeral IP address that will change if you stop and restart your instance. You can set up an EC2 Elastic IP address to keep this from happening (not covered here.)

I’m assuming you also have your .pem file with your private key.

In the terminal:

$ ssh -i /path/to/your/keypair.pem admin@xx.xx.xx.xx

Since this is the first time you’re logging in to this IP address, you’ll be asked to verify the machine’s fingerprint. Go ahead and say yes.

After logging in successfully you’ll see the Linux prompt:

$

Installing Updates

The first thing to do is update everything that needs updating since this distro was produced:

$ sudo apt-get update && apt-get upgrade

Stuff happens, you may be asked to confirm the download of updated packages. In some cases you may need to reboot your instance. Once you get back to the root prompt, set up future updates to happen automatically:

$ sudo echo "unattended-upgrades unattended-upgrades/enable_auto_updates boolean true" | debconf-set-selections
$ sudo apt-get -y install unattended-upgrades

Set Up The External Volume

The instructions in this part are adapted from Making an Amazon EBS Volume Available for Use on Linux.

Find the name of the device

lsblk shows the block devices currently attached, along with any mounted volumes they contain.

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme1n1     259:0    0  300G  0 disk 
nvme0n1     259:1    0    8G  0 disk 
└─nvme0n1p1 259:2    0    8G  0 part /

In the output of lsblk above, the device and volume mounted at / is the boot volume— don’t touch this! The other one is the raw block (unformatted) volume EBS is providing.

$ sudo file -s /dev/nvme1n1
/dev/nvme1n1: data

The fact that this responds with data means that there is currently no filesystem on the volume— it needs to be formatted:

Format the device

$ sudo mkfs -t ext4 /dev/nvme1n1
mke2fs 1.43.4 (31-Jan-2017)
...
Writing superblocks and filesystem accounting information: done

Make a directory for the volume mount point

$ sudo mkdir /data

Mount the volume on the device at the mount point

$ sudo mount /dev/nvme1n1 /data

Set permissions

Change the permissions of the data volume so it is accessible by the admin group.

$ sudo chgrp admin /data
$ sudo chmod g+w /data

Find the UUID of the external volume

$ sudo file -s /dev/nvme1n1
/dev/nvme1n1: Linux rev 1.0 ext4 filesystem data, UUID=6ba00350-4caa-41cf-bf9d-4924a74f1110 (needs journal recovery) (extents) (64bit) (large files) (huge files)

The command above shows that the UUID of /dev/nvme0n1p1 is 6ba00350-4caa-41cf-bf9d-4924a74f1110.

$ sudo ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root  80 Sep 10 06:40 .
drwxr-xr-x 6 root root 120 Sep 10 03:21 ..
lrwxrwxrwx 1 root root  13 Sep 10 06:40 6ba00350-4caa-41cf-bf9d-4924a74f1110 -> ../../nvme1n1
lrwxrwxrwx 1 root root  15 Sep 10 03:21 b524f8c0-90e7-4fc7-a842-6cb2380086c8 -> ../../nvme0n1p1

The output of the command above agrees.

Make sure the volume is mounted at boot-time

Edit /etc/fstab to add a line for the file, including the UUID you identified above.

UUID=6ba00350-4caa-41cf-bf9d-4924a74f1110       /data   ext4    defaults,nofail      0       2

Check for volume errors

$ sudo mount -a

Reboot and make sure everything is correct

$ sudo reboot

Install a Random Number Generator

You can read about haveged here.

$ sudo apt-get install haveged -y

Install Bitcoin

Make sure you’re working as admin and not root for this ($ prompt). To exit the root shell, type Ctrl-D.

$

Add Useful Aliases

Edit .bash_profile to add these useful aliases:

alias btcdir="cd ~/.bitcoin/" #linux default bitcoind path
alias bc="bitcoin-cli"
alias bd="bitcoind"
alias btcinfo='bitcoin-cli getwalletinfo | egrep "\"balance\""; bitcoin-cli getinfo | egrep "\"version\"|connections"; bitcoin-cli getmininginfo | egrep "\"blocks\"|errors"'
alias btcblock="echo \\\`bitcoin-cli getblockcount 2>&1\\\`/\\\`wget -O - http://blockexplorer.com/testnet/q/getblockcount 2> /dev/null | cut -d : -f2 | rev | cut -c 2- | rev\\\`"

Set up some shell variables

Set up two variables to make this installation more automatic.

The first variable, $BITCOIN, should be set to the current version of Bitcoin. It was 0.16.2 when I wrote this. The second will then automatically generate a truncated form used by some of the files.

$ export BITCOIN=bitcoin-core-0.16.2
$ export BITCOINPLAIN=`echo $BITCOIN | sed 's/bitcoin-core/bitcoin/'`

Download Files

$ wget https://bitcoin.org/bin/$BITCOIN/$BITCOINPLAIN-x86_64-linux-gnu.tar.gz
$ wget https://bitcoin.org/bin/$BITCOIN/SHA256SUMS.asc
$ wget https://bitcoin.org/laanwj-releases.asc

Verify Bitcoin Signature

$ /usr/bin/gpg --import laanwj-releases.asc
$ /usr/bin/gpg --verify SHA256SUMS.asc

Amongst the info you get back from the last command should be a line telling you that you have a “Good signature”. (Don’t worry about the warning.)

Verify Bitcoin SHA

Next, you should verify the Hash for the Bitcoin tar file against the expected Hash:

$ /usr/bin/sha256sum $BITCOINPLAIN-x86_64-linux-gnu.tar.gz | awk '{print $1}'
$ cat SHA256SUMS.asc | grep $BITCOINPLAIN-x86_64-linux-gnu.tar.gz | awk '{print $1}'

If those both produce the same number, it’s OK.

Install Bitcoin

$ /bin/tar xzf $BITCOINPLAIN-x86_64-linux-gnu.tar.gz
$ sudo /usr/bin/install -m 0755 -o root -g root -t /usr/local/bin $BITCOINPLAIN/bin/*
$ /bin/rm -rf $BITCOINPLAIN/

Create a symlink for the bitcoin core directory

The allows you to use the external volume for all your data.

$ mkdir /data/bitcoin
$ ln -s /data/bitcoin .bitcoin

Create Bitcoin Configuration

Finally, you should set up a bitcoin configuration file. This is the core bitcoin.conf file, which is appropriate for an unpruned testnet setup:

$ cat >> .bitcoin/bitcoin.conf << EOF
# Accept command line and JSON-RPC commands
server=1

# Set database cache size in megabytes (4 to 16384, default: 450)
dbcache=1536

# Set the number of script verification threads (-6 to 16, 0 = auto, <0 = leave that many cores free, default: 0)
par=1

# Set to blocksonly mode, sends and receives no lose transactions, instead handles only complete blocks
blocksonly=1

# Tries to keep outbound traffic under the given target (in MiB per 24h), 0 = no limit (default: 0)
maxuploadtarget=137

# Maintain at most <n> connections to peers (default: 125)
maxconnections=16

# Username for JSON-RPC connections
rpcuser=bitcoinrpc

# Password for JSON-RPC connections
rpcpassword=$(xxd -l 16 -p /dev/urandom)

# Allow JSON-RPC connections from, by default only localhost are allowed
rpcallowip=127.0.0.1

# Use the test chain
testnet=1

# Maintain a full transaction index, used by the getrawtransaction rpc call (default: 0)
txindex=1

# Make the wallet broadcast transactions (default: 1)
walletbroadcast=1
EOF

TESTNET vs MAINNET: If you want to use mainnet instead of testnet, just omit the “testnet=1” line; easy!

PRUNED vs UNPRUNED: If you want to use a pruned copy of the blockchain instead of an unpruned copy, to minimize storage requirements and loading time, do not include the txindex=1 line, but instead add a prune=550 line. txindex gives the benefit of a complete transaction index, but is not compatible with pruning, so you choose one or the other

Limit permissions to your configuration file:

/bin/chmod 600 .bitcoin/bitcoin.conf

Start the daemon

$ bitcoind -daemon
Bitcoin server starting
$

Check progress downloading the blockchain

$ btcblock
2718/1412822

Time passes…

$ btcblock
1356386/1412834

Automatically start the daemon after reboot

$ ( /usr/bin/crontab -l -u admin 2>/dev/null; echo "@reboot /usr/local/bin/bitcoind -daemon" ) | /usr/bin/crontab -u admin -
$ crontab -l
@reboot /usr/local/bin/bitcoind -daemon

Reboot and check to make sure the external volume is mounted and bitcoind is running

$ sudo reboot

A few minutes later…

$ ls /data
bitcoin  lost+found
$ btcblock
1412867/1412867

🐺

To the extent possible under law, I waive all copyright and related or neighboring rights to the text of this article.