The cover image for this post is by Dessidre Fleming
This blog post was written by Jamie.
The last 30+ months have been rather tough on most industries. With local and global lockdowns forcing companies to offer up remote working where possible, a lot of companies have had to adapt very quickly. At RJJ, we’re proud of the fact that we are a remote first company - quite literally our employees can work from wherever they want, as long as they get the work done in time for their deadlines.
But working remotely presents a number of technological challenges - along with infrastructure and process challenges, too. I’d like to talk through one of the ways that we recently boosted our connectivity whilst still remaining remote.
Being Remote
We’ve been a remote first company since we started. When asked to work on-site for our clients, we will always start with why
great advice from Simon Sinek which applies to almost anything, by the way
Usually the response is a combination of “infrastructure” and “security” reasons. I can get behind the security reasons, as soon as you let data out of your organisation it may as well be public. There’s no way to secure data outside of your organisation. I’m not talking about data which should be encrypted
of course, all data should be encrypted at both rest and in transit
I’m talking about data about the company, project plans, and industry secrets.
Letting a new person into the fold requires carefully vetting them. And if they are working on-site, you can keep a watchful eye open for the exfiltration of secrets, and do something about it when you suspect it’s happening. And this is worse still with someone who is completely external to your company: to a certain extent, you can control what your employees do. But you cannot control what an external business does when they are providing a service.
As a trivial example, take the employees of the company you contracted the work of cleaning your office space out to. Can you tell them precisely how to do their jobs? No. Their working practices are set by the policies that their employers have in place.
Anyway, all of this is to say that we are completely remote and offer only remote services to our clients.
On VPNs
You might be thinking:
But what about infrastructure? You must have to be on-site to access key infrastructure, surely.
Well, here’s where it gets just a little bit wobbly and very “upset the apple cart”-like. Because, no you don’t need to be on-site to access any infrastructure. There’s this great technology which was first explored in 1996 called a VPN - or Virtual Private Network. A VPN is a network between two or more computers which is virtual (i.e. managed entirely in software) and completely private (only those with the correct access credentials can get in).
What this means is that you can set up a VPN on a central server and have everyone connect to that, granting them access to whatever infrastructure they might need, regardless of where they are geo-located. As long as they have a stable Internet connection, that is.
The accepted protocol for granting access to resources behind the VPN is to have an LDAP
Lightweight Directory Access Protocol
such as Active Directory, that way you can have a central domain controller which controls the access to services and directories, and grants them to authenticated users.
This is great for organisations who are new to offering remote working, as they will have some on premises server infrastructure which was previously working as on-site LDAP
you were limiting access to company resources in The Before Times, right?
and offering this externally is a case of punching a hole in the company firewall
again, you definitely have one of these, right?
for the VPN to use, setting up a VPN server, and allowing users to connect to it and authenticate against it.
A simplistic solution would be to allow any VPN connections from any IP address ranges (within the countries where you have employees); but this is a recipe for disaster, because almost anyone can attempt to connect to the VPN. All that a malicious actor would require is that address of the VPN server and some leaked (or brute forced) credentials. You can shore this up in two ways, and I recommend both:
- Limiting the IP address range to known employee addresses
- Forcing good multi-factor authentication for accessing the VPN
We’ll take the latter first: good multi-factor authentication requires the use of a Time-based one-time password. This means using things like the Google or Microsoft Authenticator applications
I personally prefer authy
as these will generate a six to twelve digit one-time passcode which must be entered alongside your username and password credentials. The great thing about these is that they are only valid for a very short amount of time - the RFC for OTPs states 30 seconds, but it can be as high as 60.
But what about IP address ranges?
This is where things get tricky. Because your employees might not have a static IP address. This means that the IP address which assigned to them by their Internet Service Provider might change on a regular basis. Which means that their connection to the VPN server may suddenly stop working, and they’ll have to file a support ticket with you company’s Operations or IT Support so that someone can remove their previous IP address from the VPN server’s allowlist, and replace it with their new one. This will also happen if they move from their remote office space (perhaps their home office) to a co-working space (or a cafe); they will have another IP address and will need the new one to be added to the VPN server’s allowlist.
Remote First Solutions
Because we are remote first, we don’t have any on premises infrastructure to offer a VPN server through. So our options were a little more limited. We couldn’t set up a VPN server in someone’s work space either, as our IP addresses could change; meaning that the VPN server’s external IP address would change
and if the VPN server’s IP address changes, you have slightly bigger issues because no one will be able to connect
You’d think that having a domain for this would solve it. But the domain just resolves to an IP address (or IP address range), and when that IP address changes you need to inform te DNS servers of the world; and updating DNS records can take up to 48 hours.
We could spin up a VPN server in the cloud using something like Linode, Azure, or any of the other cloud service providers; and that would work. One thing to note is that there are also times when cloud services go down - like when Azure went down globally in 2018, or when it happened to AWS in 2021. Granted, when a cloud service goes down, they have much bigger problems than you not being able to access the company network infrastructure.
We could have used a VPN service, too. But the big issue with paid VPN services is trust
I’m a little jaded here, but if you can’t trust a paid VPN service, can you trust a free one?
As we cannot see into their logging and networking infrastructure, we cannot be sure whether they were logging data or requests as they travel into the entrance node, through their network of servers, and out of the exit node. As we deal with a lot of highly confidential projects and services, we needed to ensure that our data was kept away from prying eyes.
So What’s The Solution For Us?
Before we get into the solution that we chose, I need to let you all know about how our employees work: we each have company owned hardware, and each employee is asked whether they would prefer to work with Windows, MacOS, or a Linux distribution. Sometimes the options are limited; for instance, our design department all prefer to use MacOS, and Mark (and the other audio engineers) can only use either Windows or MacOS because the tools that they need are not available for Linux distributions. I use a distribution of Linux called PopOS - though may be moving to Fedora or NixOS for unrelated reasons - as I work with cross-platform languages like C# & .NET, Python, JavaScript & TypeScript, and Go.
So whatever solution we found would have to be cross platform out of the box. We couldn’t continue to offer the choice of hardware and OS if our hands were tied by the way we connected our devices together. I would much rather not have to have someone unable to connect because they are using an unsupported operating system. I would also prefer that getting everything up and running took minutes, if not seconds. I would also rather have a single solution from a single provider, rather than monkey patching things together with lots of different applications and solutions, because that becomes very fragile very quickly.
We looked around for a long time, and started playing around with WireGuard. WireGuard worked really well for our purposes, and has support for roaming - so when your IP address changes it deals with the change gracefully and continues to operate with no action required from the user. The problem was that it has a complicated (ish) setup, and we wanted the onboarding process for new employees to be:
- Download this app
- Authenticate with username, password, and OTP
- Done
This wasn’t exactly possible with WireGuard - at least, not immediately. We wanted something simpler; idiot-proof, you might say. After all, we need everyone to be able to get up and running with almost not trouble, in as short a time as possible, and for them to not have to remember any commands or have to troubleshoot esoteric issues or have to futz with their home router
Some ISPs in the UK block VPNs at the home router
That’s why we chose Tailscale.
I need to make this very clear: this blog post is neither promotional material for Tailscale, nor are we being paid to publish this post
Here is the entire process we had to go through for setting up two devices with Tailscale:
- I signed into Tailscale using our company SSO (Single Sign On) provider
- I downloaded the Tailscale client
- I authenticated within the client
- I had Mark do the same
- Done
Seriously, it was that simple. Since our SSO accounts used the same domain, Tailscale did all of the wiring up of everything for us. This means that we have a VPN made up of our machines, without any Tailscale infrastructure between our devices*.
*my understanding is that there is some Tailscale infrastructure which facilitates the initial connection, but after the connection is made the Tailscale infrastructure is no longer used
So we had everyone download the client for their operating system, authenticate with their company SSO and we were connected immediately.
But What Can We Do?
As we now have a virtual network consisting of RJJ hardware, we can do anything that we can with a real network: ssh, ping, rsync, shared drives, etc. Providing remote desktop-based assistance means that we never leave our VPN in order to connect - fewer hops means SLIGHTLY* less latency
*and I really means SLIGHTLY, the difference is in the tens of milliseconds, but it’s worth pointing out
It also means that we can now have servers hosted at an employee’s remote work space - Tailscale allows you to bring up to five devices of your own into the VPN by authenticating the application with your SSO on those devices. So we now have a centralised file server, and an LDAP for it - meaning that access to file server is locked behind a authentication and authorisation.
And best of all, since VPN technology is a peer-to-peer encrypted connection, everything is now encrypted in transit. Coupled with the multi-factor auth we have for our company SSO, this means that we’re protected against the majority of trivial, script-kiddie based breaches whilst also allowing our employees to work from wherever they want in the world.
There are some security considerations with using Tailscale or WireGuard
when stripped back, Tailscale uses WireGuard along with a bunch of other things, and is completely open-source
but with the source code being completely open (and it being possible to build and host your own instance), our level of trust with Tailscale was slightly higher. They also claim to not be able to see into the data that your transmit when using Tailscale - a claim which can be backed up by reading the source code.
There’s also a risk of individual machines being exploited by a malicious actor, virus, or malware. But we have policies and protections in place for those two - defence in depth is something which should be the default, but that’s a subject for another time.