SadServers is a playground where users can test their Linux troubleshooting skills on real Linux servers. Besides sharpening your troubleshooting skills, it is also suitable for practicing for SRE/DevOps interviews.

Others call it the "LeetCode for Linux". As LeetCode, there's challenges classified under difficulty level, a collection of scenarios, a description of what’s wrong and a test to check if the issue has been solved, within an allocated time length. After that time the VM is terminated.

The problems you're going to encounter range from Linux administration to database configuration to webserver troubleshooting and issues with Docker, which are things that you, as an SRE, will encounter on a daily basis. Since the problems are realistic and on the job, SadServers can also be used as a testbed for interviews where the employer checks the candidates skills against a challenge.

Here's a a few of the troubleshooting scenarios that you will encounter: 

1. "Saint John": what is writing to this log ?

Level: Easy

Description: A developer created a testing program that is continuously writing to a log file /var/log/bad. log and filling up disk. You can check for example with tail -f /var/log/bad. log.
This program is no longer needed. Find it and terminate it.

Test: The log file hasn't changed in the last 6 seconds: find /var/log/bad. log -mmin -0. 1 (You don't need to know the details of this command).

Time to Solve: 10 minutes.

2. "Salta": Docker container won't start.

Scenario: "Salta": Docker container won't start.

Level: Medium

Description: There's a "dockerized" Node. js web application in the /home/admin/app directory. Create a Docker container so you get a web app on port :8888 and can curl to it. For the solution to be valid, there should be only one running Docker container.

Test: curl localhost:8888 returns Hello World! from a running container.

Time to Solve: 15 minutes.

3. "Hong-Kong": can't write data into database.

Scenario: "Hong-Kong": can't write data into database.

Level: Hard

Description: (Similar to "Manhattan" scenario but harder). Your objective is to be able to insert a row in an existing Postgres database. The issue is not specific to Postgres and you don't need to know details about it (although it may help).

Postgres information: it's a service that listens to a port (:5432) and writes to disk in a data directory, the location of which is defined in the data_directory parameter of the configuration file /etc/postgresql/14/main/postgresql. conf. In our case Postgres is managed by systemd as a unit with name postgresql.

Test: sudo -u postgres psql -c "insert into persons(name) values ('jane smith');" -d dt

Should return:INSERT 0 1

Time to Solve: 20 minutes.

For every scenario at hand you spawn a fully blown remote Linux server (an ephemeral Virtual Machine) with root access, accessible over and SSH shell via the browser. As a constraint, DNS is available locally and you cannot reach the Internet from within the VM spawned. Also you must not interfere with services unrelated to the issue which are needed for SadServers to work properly, specifically the services running on ports :2020 and :6767.

If you are stuck or not sure what to do, you can click on the "Next Clue / Solution" button and it will display a new hint (and the previous ones) until it reveals the solution. Close the clue window and click "Next Clue / Solution" again to get a new clue. Once you think you have fixed the issue, click on the "Check My Solution" button to verify it as per the given Test.

The scenarios are limited in number for the time being but new scenarios should be getting added weekly.

The architecture that SadServers is build upon is very interesting as well. There are many components at play, including: 

• Django and Python3
• Bootstrap and plain Javascript at the front
• In front of Django there's an Nginx server and Gunicorn WSGI server
• The tasks are managed asynchronously by Celery with a RabbitMQ
• A PostgreSQL database is the permanent storage backend
• Gotty provides a terminal with a shell as HTTP(S)

Have a look at its diagram over on the project's Github repo.

All in all, Sadservers is a noteworthy project in both utility and architecture. Make sure you check it out.

More Information

SadServers - Troubleshooting Linux Servers 

Related Articles

Bash-Oneliner and GameShell Teach Unix Command Line

The Linux Upskill Challenge

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info