MinJunMaru's Blog

Server breakdown, also happy near year!

—

by

MinJunMaru
in

Happy 2023! New year, new problem, old me.

Right before the Christmas festivities, productivity in the office grinded to a halt. My office email server decided to go on an unannounced vacation, permanently.

Usually when the server goes down, a (not so quick) restart would solve any issues, but this time was different. Zimbra refused to load due to Webmin occupying port 10000. I tried to kill it with the following code, to no avail.

sudo kill -9 $(sudo lsof -t -i:10000)

sudo   - superuser
kill   - kill the process
-9     - force kill
lsof   - List Open Files(and list of related processes)
-t     - show only process ID
-i     - show only internet connections related process
:10000 - show only processes in this port number

After a few hours of tinkering, I was still unable to find the root cause. This effectively cut off our ways to communicate with clients as we heavily relied on emails. Since I’ve always wanted to build a server from scratch, why not make use of company resources to start my own project? I finally had the perfect excuse to upgrade my personal computer.

Maru’s Plan to Get Rich.

  1. Disassemble old pc
  2. Build a new server
  3. ???
  4. Profit

Jokes aside, we had to figure out a solution fast. The office had contemplated on moving to cloud solutions like Microsoft 365 or Google Workspace for the convenience and reliability. However at approximately SGD$8.00/month per user, this would balloon our costs significantly.

After mulling for a bit, we ended up building our own server, spending only $550 on parts through recycling my old computer. It was still cheaper than an annual subscription of Microsoft/Google.

The next question was what software? I pondered over the following shortlist:

  1. Zimbra Open Source
  2. Mailcow
  3. iRedMail
  4. Control Panels with email (ISPConfig/HestiaCP/Webmin+VirtualMin)
  5. Manually configuring postfix + dovecot

I was tempted to just install Zimbra and call it a day, but I had bad experiences with the old installation, as well as security concerns. Next in line was Mailcow and iRedMail. The installation and setup processes were quick and smooth, but I didn’t like SOGo’s UI when compared to RoundCube. As for control panels, I felt that the increased complexity and lack of control was not worth the effort, and ended up setting up manually using Postfix + Dovecot. It was (in my opinion) the best solution for the office due to familiarity and control over the software.

As for hardware, I recycled my old 9900k and slapped a new Solid State Drive (SSD) on it. I decided against RAID and instead opted for daily backups in a separate server. In the event of outage, I could just pull an archived copy and quickly spin up in another VM. I’ll need to improve this in near future.

We have also improved our data redundancy, having users to depend on a separate storage server (with RAID) instead of pulling documents from emails, and multiple levels of security, including OTPs, restricted user access and enforced stronger passwords. Admittedly, the server breakdown pushed me to implement tougher security as a preventative measure for data loss.

With all these in place, I could finally take a breather to enjoy my upcoming holiday in Korea. Thanks for reading and hope to see you in the next post 🙂

Tl;dr – Server kaput, new server. Right before I flew for the holidays. Yay.