Let’s pay some attention to the man behind the curtain: the Land of SysAdmins.

First described in around 1st century Anno Domini, the race for building a steam engine properly picked up in the early 18th century. The oohs and aahs as well as coins and the grants, some wicked engineers built mock engines on top of the ships, but stuffed the lower deck with rowing servants and making the oars invisible.

“It works!” — they used to shout.
“Yes, but we’re working on it.” — they used to murmur after getting caught.

Many automatons were dazzling people all around the world, from the Arab peninsula to the tall and mighty churches in Western Europe. Yet the most ingenious of them had a person inside operating them.

Welcome to the Munchkin Country in the Land of SysAdmins. Where you will most likely hear:

Pay no attention to the man behind the curtain!

We’re here to provide tools, shelter, 24/7 customer service hotline and occasional entertainment. All of this enables our innovative developers and creative designers (think of minds comparable to Brunel and Escher) to craft high level products to satisfy a wide range of request from our well known clients.

Here’s a few solutions we implemented in our infrastructure.

The technologies we use so our developers and creatives can make high lever products

All of our servers are virtual containers and virtual machines which live on a hardware running Proxmox virtualization platform set up using ZFS.

There are a few reasons we chose ZFS as our file system.

• First of all — we are developers, we like to build things just as much as break them. Taking a snapshot of a virtual environment with ZFS makes it easy and quick to roll back the changes we didn’t like or move the servers around with ease and speed.
• Secondly — we want to make use of our resources as much as possible, and the way ZFS can do it is pretty smart.

ZFS’s Cache management

Imagine a bouncer in your favorite club, where the needle is piercing your dearest black circles of Beatles, The Who or Rolling Stones. Perfect night you don’t want to miss, I know, yet picture the guest list being so mismanaged, you end up waiting for too long. Continue this ancient policy for an extended period of time and you have kids losing patience and ending up listening to papa roach and green day.

The classic cache management protocols are still the same in most of the systems as they were in the 70s. The most popular choice is one of the following:

The most obvious caveats for LRU — it ignores frequency, can be easily messed up with a whole system scan. LFU — stores data you’re no longer using, complex and slow, does not consider recency.

What ZFS does — it combines the best of both logics and keeps a ghost list for both of these types. It’s called Adaptive Replacement Cache or ARC for short. Here’s a basic overview of how it works:

1. At the beginning cache is partitioned into two equal sectors: LRU and LFU,
2. Both LRU and LFU also has a ‘ghost’ list — when the item is evicted from cache, it’s key (not it’s data) is stored there,
3. If you try to access the page and you find it in ARC — all good,
4. If the page is accessed frequently — it’s moved to LFU,
5. Depending which ghost list you hit — ARC changes the ghost list decreasing LRU and increasing LFU or the other way around,
6. This adaptive method makes sure your system is optimized for your workload.

ZFS’ Copy on write (COW)

On most of the systems new data is overwritten on the old data blocks. On ZFS, the data changes are performed on the copy of the data instead of the original data blocks. In other words, the changes are stored on a different location on a disk and then the metadata is updated in that place on the disk.

Traditional filesystems

Copy-on-write (COW). Visualisation is simplified ignoring the randomness of blocks.

This mechanism guarantees that the old data is safely preserved in case of power loss or system crash that in other cases would result in loss of data.

It also means our showcase applications and websites are always reachable, quick to recover even if unexpected happens and easy to clone to a multitude of test environments, thus making the developers more efficient and creative. And even when big scale hardware upgrades are executed, our smart infrastructure minimises downtime for staff and clients to a minimum or allow the changes to go unnoticed.

The technologies we use to control access of our internal resources

Another thing we want to do, is control access to internal resources. If a staff member needs direct access to a server — we grant or revoke it based on LDAP. You can imagine it as a list of all our employees being handed to a bouncer standing next to every server (Yes, he is back. But he’s a workaholic and has many mouths to feed, so please leave him be). The access-management server has the original list where it can add little notes to every name, telling which person is allowed to enter which server, and the list gets updated in real time. The server can read this list and either grant or deny access.

Of course, some resources are open and free for all. Just ask the interns about their wonderful experience with the coffee machine! Speaking of which — my cup is getting empty. You’re game?

Moving on to an all so familiar experience we call WFH. Once your employees try to access the same resources from outside the office, they must pass our trusted firewall, before they virtually enter an environment mimicking the office setup. This has been working so great in Elements, no one is even coming back to the office! Oh wait…

We achieve this using IP whitelisting with the help of on-premise dynamic dns server. Everyone gets a unique domain name and a unique key they can use to update the domain name if the public IP changes. Updates can be done by running a little script or entering a specific URL in a browser. Then the dynamic dns server says “Hello?”, checks if the address and the key match with what is expected, and if it does — this speed dating is labeled as successful and the public IP of our staff members is updated. Firewall is now behaving friendlier and you can march on to establish VPN connection, access our Intranet website or any other resource. All of this has more fine-grained rules within the firewall.

Let’s talk VPN

I’m curious to hear what your experience with VPN has been like? Are you issuing certificates per user? Have a similar whitelisting approach? How happy are you with the lack of complaints about WiFi and printers?
Invest in a robust and simple to use core infrastructure and the whining days may never come back! At least not until a new toy by Apple is released and you have to give that early call to Santa and provide a rectangular metal piece of happiness to your colleagues.

Enjoy life simplified by sysadmins. Even if the tidiness under the hood is not always a guarantee or even a must.

We’re working on it.

‍