Server Setup

This server is hosted on prgmr.com, a VPS provider based on xen virtualization technology, and targeted at advanced users. This server is installed through a custom console, which provides access to a bunch of pre-installed OS images as well as some netboot installers for common Linux distributions and the {Free,Open,Net}BSD operating systems.

Main Menu - <server>

Current status:
	<server> is running.

Wiki at http://wiki.prgmr.com
Please contact support@prgmr.com with any issues accessing your machine.

Options:
1. out of band console (press ctrl-] to escape, not resizeable)
2. create/start, opens OOB console (try this if the machine is not running)
3. shutdown (requests clean shutdown, forces off after 4 min)
4. force power off (destroy/hard shutdown)
5. reboot (shutdown + start)
6. set bootloader, rescue mode, or netboot installer
7. view/add/remove ssh authorized_keys
8. view/edit reverse dns
9. install new OS image
a. system details

0. Exit
R. refresh

enter selection>6
Set Boot Options - <server>

Configured to boot from disk.

Options:
1. Boot from disk
2. External Bootloaders
3. Linux-based Live Rescue
4. Linux netboot installers - install mode
5. Linux netboot installers - rescue mode
6. BSD installers

0. Return to main menu
R. refresh

enter selection>4
Linux netboot installers - install mode - <server>



Options:
1. Alpine 3.8.0 - 64 bit
2. Centos 6 - 32 bit
3. Centos 6 - 64 bit
4. Centos 7 - 64 bit
5. Debian Jessie - 32 bit
6. Debian Jessie - 64 bit
7. Debian Stretch - 32 bit
8. Debian Stretch - 64 bit
9. Fedora 27 - 64 bit
a. Fedora 28 - 64 bit
b. Nixos 18.03 - 64 bit
c. Ubuntu Bionic 18.04 - 64 bit
d. Ubuntu Bionic 18.04 Docker - 64 bit
e. Ubuntu Precise 12.04 - 32 bit
f. Ubuntu Precise 12.04 - 64 bit
g. Ubuntu Trusty 14.04 - 32 bit
h. Ubuntu Trusty 14.04 - 64 bit
i. Ubuntu Xenial 16.04 - 32 bit
j. Ubuntu Xenial 16.04 - 64 bit

0. Return to set boot options
R. refresh

enter selection>c

The first choice to make is what OS to install. This server is running on Ubuntu 18.04 LTS (Bionic Beaver). In the interest of simplicity and security, I installed it using the Netboot installer image, allowing for a minimal install containing only the software needed for the server, therefore reducing the potential attack surface. For the most part, the installation prompts are self-explanatory, and the defaults provided by the installer are good. However, I have deviated from the defaults in a couple of significant ways.

First of all, it is good practice to use full disk encryption. Some best practices are outlined in the OWASP Cryptographic Storage Cheat Sheet, which is outside the scope of this article. In the case of this server, the (virtual) disk is encrypted using the Linux Unified Key Setup (LUKS) with the following parameters (identifying metadata has been sanitized):

root@<server>:~# cryptsetup luksDump /dev/xvda5 
LUKS header information for /dev/xvda5

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha256
Payload offset: 4096
MK bits:        512
MK digest:      xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx 
MK salt:        xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx 
                xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx 
MK iterations:  35656
UUID:           xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Key Slot 0: ENABLED
        Iterations:             570498
        Salt:                   xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx 
                                xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx 
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

LUKS provides 8 keyslots which can use either passphrases or keyfiles or a combination of both for access. This disk is protected with a randomly generated 100 character passphrase using both upper and lowercase letters and symbols, which should provide about log2(94100) ≈ 655.46 bits of entropy assuming the 94 printable ASCII characters (not including <space>) are all equally likely to appear in each position in the passphrase. While this does not protect any of the data in transit, it does mean that the server can be powered down and the virtual disk can be cloned and stored elsewhere on the network with a good degree of security. Also, it is good practice to encrypt all data at rest.

The installer provides a simple option: Guided - use entire disk and set up encrypted LVM

partition_disks

which will not only set up the LUKS encryption container, but also use LVM2, the Linux Logical Volume Manager, which makes repartitioning easier should the need arise. This will then prompt for confirmation, while also displaying the size of the device which will act as the encrypted container.

virtual_disk_1

Since this Ubuntu installation is the only OS installed on this virtual disk, it is safe to proceed with the next step and install the bootloader, in this case GNU GRUB. This will allow the server to boot and prompt for the disk encryption password each time it is rebooted. While it is possible to set this up in such a way that the disk can be unlocked using Dropbear SSH, this server is configured such that it requires logging into the Management Console to unlock the disk. Since the box will only be rebooted as necessary (either for maintenance or kernel upgrades), this isn’t to inconvenient, although it represents a security / accessibility trade-off since a server crash will require my manual intervention since the hosting provider does not have access to the encryption key.

install_grub

The file /et/default/grub needs some edits to properly work with the out of band console. The diff is shown below:

--- /etc/default/grub.orig	2018-08-08 12:31:24.146061494 -0400
+++ /etc/default/grub	2018-08-02 17:25:23.000000000 -0400
@@ -7,8 +7,8 @@ GRUB_DEFAULT=0
 GRUB_TIMEOUT_STYLE=hidden
 GRUB_TIMEOUT=10
 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
-GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
-GRUB_CMDLINE_LINUX=""
+GRUB_CMDLINE_LINUX_DEFAULT="nosplash"
+GRUB_CMDLINE_LINUX="console=ttyS0 rootflags=barrier=0"
 
 # Uncomment to enable BadRAM filtering, modify to suit your needs
 # This works with Linux (no patch required) and with any kernel that obtains
@@ -16,7 +16,7 @@ GRUB_CMDLINE_LINUX=""
 #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
 
 # Uncomment to disable graphical terminal (grub-pc only)
-#GRUB_TERMINAL=console
+GRUB_TERMINAL="serial console"
 
 # The resolution used on graphical terminal
 # note that you can use only modes which your graphic card supports via VBE
@@ -24,12 +24,12 @@ GRUB_CMDLINE_LINUX=""
 #GRUB_GFXMODE=640x480
 
 # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
-#GRUB_DISABLE_LINUX_UUID=true
+GRUB_DISABLE_LINUX_UUID=true
 
 # Uncomment to disable generation of recovery mode menu entries
 #GRUB_DISABLE_RECOVERY="true"
 
 # Uncomment to get a beep at grub start
 #GRUB_INIT_TUNE="480 440 1"
-GRUB_TERMINAL=serial
-GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --stop=1"
+export LINUX_ROOT_DEVICE="LABEL=PRGMRDISK1"
+GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

Since the server is installed from the minimal installer, it is necessary to explicityl install the OpenSSH server:

software_selection

While the default /etc/ssh/sshd_config is a good starting point, its defaults prioritize compatibility over security. To harden its configuration, a good starting point is Mozilla’s OpenSSH security configuration. Since I am using updated software to connect to this server for administrative purposes, using modern secure defaults is the way to go. The only changes I have made to the configuration as suggested by Mozilla is PermitRootLogin yes, and delete UsePrivilegeSeparation sandbox configuration option and comments, as this option is deprecated as of OpenSSH 7.5, since it is enabled by default and will spam the server’s logs if left in the config.

# Supported HostKey algorithms by order of preference.
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

# Password based logins are disabled - only public key based logins are allowed.
AuthenticationMethods publickey

# LogLevel VERBOSE logs user's key fingerprint on login. Needed to have a clear audit track of which key was using to log in.
LogLevel VERBOSE

# Log sftp level file access (read/write/etc.) that would not be easily logged otherwise.
Subsystem sftp  /usr/lib/ssh/sftp-server -f AUTHPRIV -l INFO

PermitRootLogin yes

The server is configured for public key authentication and will reject password-based authentication. This means that a pubic-private ssh key-pair is needed for remote administration of the box. Recent versions of OpenSSH allow us to use fast elliptic curve cryptography, that provides the security of RSA. We need to generate an ssh key on the development machine such as:

user@<laptop>$ ssh-keygen -a 100 -t ed25519 -C <comment> -f $HOME/.ssh/<private-key-name> 

This specifies the number of rounds (100), that we want the key derivation function to be run to slow down brute-forcing attempts on the password of our private key should it be stolen, or accidentally committed to GitHub. Also, we are specifying that we want to use Ed25519 keys (supported as of OpenSSH 6.5), which are performant and secure. OpenSSH provides the option of giving the key a comment so that we can more easily identify it in the authorized_keys file on a remote host.

For public-key authentication to work with OpenSSH, the public key of the newly-generated ssh key needs to be present on the remote host in the .ssh/authorized_keys file of the user that we wish to login as; in this case we want to authenticate as root. One benefit of using ed25519 instead of RSA keys is that the public key is short enough that we can also type it directly into the /root/.ssh/authorized_keys file without too much trouble, if, for example, we have any trouble pasting input into the console provided by the hosting provider.

user@<laptop>$ cat $HOME/.ssh/<private-key-name>.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI<random-44-char-string> <comment>

By launching a shell in the installer before completing the installation, it is possible to confiure this to work on the first boot. Alternatively, this step can be completed from the out-of-band console provided by many hosting providers. Either way, it is important to disable password-based authentication over ssh, especially for root, at the earliest possible time after provisioning a new server.

root@<server># mkdir $HOME/.ssh 
root@<server># echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI<random-44-char-string> <comment>" \
               >>$HOME/.ssh/authorized_keys

The file system permissions on both the .ssh directory and the authorized_keys must not be too permissive, and owned by the the correct user, which can be set as follows:

root@<server># chown -R $USER:$USER .ssh &&
               chmod 0700 $HOME/.ssh &&
               chmod 0600 $HOME/.ssh/authorized_keys

Enabling a firewall is crucial for basic security. Ubuntu ships with UncomplicatedFirewall (ufw) by default. This is a simple frontend to netfilter / iptables, which simplifies basic firewall administration and provides numerous modules for common use cases / applications.

In this case, it is crucial to enable the module for OpenSSH so that remote access to the box is presrved. This is easily accomplished with the following:

root@<server># ufw allow OpenSSH && ufw enable

Since this box is going to be used to host a nextcloud instance, it is also necessary to enable access to the ports used by the Apache HTTP Server.

root@<server># ufw allow "Apache Full"

This can be verified with:

root@<server># ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Apache Full                ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Apache Full (v6)           ALLOW       Anywhere (v6)

It is also possible to save the firewall rules natively using iptables with the following redirection to a file:

root@<server># iptables-save > /some/arbitray/file

This will generate something such as:

# Generated by iptables-save v1.6.1 on Sun Aug 19 01:49:26 2018
*filter
:INPUT DROP [3424:200705]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [30:1893]
:ufw-after-forward - [0:0]
:ufw-after-input - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-before-input - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-output - [0:0]
:ufw-logging-allow - [0:0]
:ufw-logging-deny - [0:0]
:ufw-not-local - [0:0]
:ufw-reject-forward - [0:0]
:ufw-reject-input - [0:0]
:ufw-reject-output - [0:0]
:ufw-skip-to-policy-forward - [0:0]
:ufw-skip-to-policy-input - [0:0]
:ufw-skip-to-policy-output - [0:0]
:ufw-track-forward - [0:0]
:ufw-track-input - [0:0]
:ufw-track-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-user-input - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-output - [0:0]
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-forward -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-forward -j ufw-user-forward
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -j ufw-not-local
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-input -j ufw-user-input
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -j ufw-user-output
-A ufw-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] "
-A ufw-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
-A ufw-skip-to-policy-forward -j DROP
-A ufw-skip-to-policy-input -j DROP
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 22 -m comment --comment "\'dapp_OpenSSH\'" -j ACCEPT
-A ufw-user-input -p tcp -m multiport --dports 80,443 -m comment --comment "\'dapp_Apache%20Full\'" -j ACCEPT
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT
COMMIT
# Completed on Sun Aug 19 01:49:26 2018

The defaults are fairly secure, denying most incoming connections. Many people will argue in favor of changing the default port for OpenSSH (22) to avoid the logs being spammed with attempted logins. This is worth considering, but in this case login is only allowed via public key authentication, so I am not worried about someone being able to brute force the login.

With a port scan of the server using nmap, I am pretty confident that the firewall is set up securely:

user@<laptop>$ sudo nmap -O -v fog.vincible.space
Starting Nmap 7.70 ( https://nmap.org ) at 2018-08-19 17:36 EDT
Initiating Ping Scan at 17:36
Scanning fog.vincible.space (71.19.146.187) [4 ports]
Completed Ping Scan at 17:36, 0.21s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 17:36
Completed Parallel DNS resolution of 1 host. at 17:36, 0.11s elapsed
Initiating SYN Stealth Scan at 17:36
Scanning fog.vincible.space (71.19.146.187) [1000 ports]
Discovered open port 80/tcp on 71.19.146.187
Discovered open port 22/tcp on 71.19.146.187
Discovered open port 443/tcp on 71.19.146.187
Completed SYN Stealth Scan at 17:36, 11.65s elapsed (1000 total ports)
Initiating OS detection (try #1) against fog.vincible.space (71.19.146.187)
Nmap scan report for fog.vincible.space (71.19.146.187)
Host is up (0.085s latency).
Other addresses for fog.vincible.space (not scanned): 2605:2700:0:3:a800:ff:fe48:207d
Not shown: 997 filtered ports
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose
Running: Linux 3.X|4.X
OS CPE: cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:4
OS details: Linux 3.10 - 4.11, Linux 3.2 - 4.9
Uptime guess: 1.001 days (since Sat Aug 18 17:34:54 2018)
TCP Sequence Prediction: Difficulty=264 (Good luck!)
IP ID Sequence Generation: All zeros

Read data files from: /usr/bin/../share/nmap
OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 14.24 seconds
           Raw packets sent: 2059 (93.212KB) | Rcvd: 59 (5.483KB)

The scan was not able to fingerprint the OS and the only open ports are the ones configured through ufw allow $Application. It is worth noting that while port scanning is invaluable for testing the security of infrastructure that you control, you should not port scan infrastructure that is not under your control, unless you have been given explicit permission to do so.