How to set up a minimal server with Monit and email alerts using MSMTP

Recently, I was setting up a Debian server on an old laptop with the sole purpose of running Draytek's VigorConnect software to monitor and manage the network on my campus (which uses Draytek switches and wireless access points). There are plenty of cloud-based management AP controller options, but I wanted a local, self-hosted option because our internet has been patchy in the past, and because I don't want any monthly license fees. My aim was to have a server that requires the bare minumum of maintenance, but sends me regular alerts by email, while also minimising the number of 'moving parts' as much as possible.

I considered buying a Raspberry Pi, but then decided that an old laptop would be ideal because A. we get a lot of powercuts in our area and a laptop has a built-in UPS of sorts, in the form of a battery, and B. It doesn't need to be very fast for my purposes, and running Linux means we can make use of old hardware that would otherwise be thrown away. I first tried using an ancient Samsung NC10 but it's 32-bit and the Draytek VigorConnect software is 64-bit only.

Install Debian

There are loads of guides on how to install Debian out there, so I won't go into any detail here. I went for the Stable release for obvious reasons. See Getting Debian.

I allocated a fixed IP address for the machine from the DHCP section on the router.

Set up SSH server

I installed openssh-server. The default OpenSSH server config on Debian now automatically reads all files matching /etc/ssh/sshd_config.d/*.conf. Therefore you can customise things in there, leaving the default /etc/ssh/sshd_config untouched, so automatic updates won't trip up over changes to that file. I created a file called /etc/ssh/sshd_config.d/custom.conf and used it to change the default port number (actual port number not shown here), and set some basic options:

Port 22
LoginGraceTime 45
PermitRootLogin no
X11Forwarding no
ClientAliveInterval 60
ClientAliveCountMax 60

Prevent system suspend when laptop lid is closed

I wanted to close the laptop lid while the machine is running so I can tuck it neatly in a cupboard next to the router. I found some notes on this here. I did some experimentation, and the only thing I needed to do was edit /etc/systemd/logind.conf and change this line:

#HandleLidSwitch=suspend

to this:

HandleLidSwitch=ignore

With this in place, it looks like the screen switches off when I close the lid (which is good for saving power and prolonging the life of the screen), and the system does not go into suspund.

Set up unattended-upgrades

The purpose of unattended-upgrades is to keep the computer current with the latest security (and other) updates automatically. The official documentation for this is here. Editing /etc/apt/apt.conf.d/50unattended-upgrades, I changed the following lines (note that the comments in the file itself have more detail than the online docs).

The documentation says "You should at least uncomment the following: Unattended-Upgrade::Mail "root";, but I wanted to send mail directly to my email address rather than going to/via the root user, so I set it like this:
Unattended-Upgrade::Mail "recipient@example.com";. More on the email setup later.

Other lines I uncommented and set were:

Unattended-Upgrade::Automatic-Reboot "true"; # Allow reboot when needed after kernel updates
Unattended-Upgrade::Automatic-Reboot-Time "02:00"; # This server won't be getting much use, so it can safely reboot out of hours if a kernel update requires it.
Unattended-Upgrade::OnlyOnACPower "true"; # in case the laptop's power supply is unplugged for some reason, don't install updates.

Set up VigorConnect

I Installed Draytek's VigorConnect software to monitor Draytek switches and Access Points on the local network, automate firmware upgrades, and sync settings to all of the APs. I copied the installer to the server and unzipped it, changed to root, and then followed the instructions in the PDF that was included in the download.

In the installer I used the default install destination and started the install. Part way through, I got this error: rename VigorConnectTempt/VigorConnect /usr/local/VigorConnect: invalid cross-device link. After a bit of poking around, I realised that this was caused because the installer was running from my home directory which is on a different partition from the install location (it seems silly that the installer assume that people won't run it like this, and that it doesn't give a more meaningingful error message!). Anyway, to work around it, I copied the installer to /root/ (which is on the same partition as the install destination in /usr/), and then it completed successfully.

I left it on the default port number, which is 4433, which means that once running, I could access the VigorConnect interface by going to the IP address of the server and specifying that port number.

Set up Monit

Monit lets you set up automated monitoring on the status of various parts of the system, restart services if they fail, and send email alerts. I followed the official documentation and a helpful guide to set it up.

On Debian, in a similar way to SSHD, Monit loads config files from /etc/monit/conf.d/ and /etc/monit/conf-enabled/ so it's recommended to put files in there to set custom config options, and avoid changing the /etc/monit/monitrc file to avoid difficulties during OS upgrades. It's suggested to put general/global config options in /etc/monit/conf.d/, while definitions of things you want to monitor should go in separate files in /etc/monit/conf-enabled/. There are some examples in /etc/monit/conf-available/, and if you wanted you could edit them and make symlinks in /conf-enabled/ pointing to /conf-available/, but I just created my own files in /conf-enabled/ because the existing files were quite elaborate but I wanted to keep them for reference.

I wanted to monitor the VigorConnect service, but couldn't see any mention of this online. It's quite easy to define custom services in Monit, so after some experimentation, ended up with a custom file called /etc/monit/conf-enabled/VigorConnect, which contains:

check process VigorConnect
matching WatchVigorConnect start program = "/etc/init.d/VigorConnectd -start"
stop program = "/etc/init.d/VigorConnectd -stop"
if failed port 4433 protocol https then restart
if 5 restarts within 5 cycles then timeout

Line 2 tells Monit to check that a process is running which (loosely) matches the name WatchVigorConnect (matching can be tested with monit procmatch my-process-name). It's not documented, but from my investigation, it seems that this daemon is responsible for watching and restarting various child processes belonging to VigorConnect, so this is the main one that Monit needs to monitor and keep alive. Lines 3 and 4 refer to the file /etc/init.d/VigorConnectd which was generated by the VigorConnect installer to stop/start the service gracefully, but it took a bit of digging to find it.

In my /etc/monit/conf.d/custom file, I set some options to tell Monit to send me an email whenever an 'alert' is triggered:

set daemon 300
set mailserver localhost
set alert recipient@example.com

Note that mail will fail to send until we've done the rest of the next steps below. You might see complaints about this in /var/log/mail.log

Set up Gmail

Gmail has fancy authentication which means you can't connect to your Google account from a 'simple' client like MSMTP using your normal Google account credentials. Instead, you need to generate an app-specific password for MSMTP to use.

I created a new account specically for sending these system messages - system-messages@example.com - so that the messages don't get marked as spam due to sender and recipient being the same, it's easier to filter them, and there's no chance of my main Google account getting compromised.

First, check that your Gmail account has IMAP/SMTP enabled, and then generate an app-specific password. When prompted, I selected App: Mail and Select device: Other - 'Local laptop server - MSMTP' so I know what each password is for in case I need to revoke any in future. Make a note of the password for the next step.

Set up MSMTP

Now we need a way of sending emails from the server - without installing a full-blown mail server such as Postfix. Initialliy I tried a package called nullmailer, but I couldn't get it working, and the only substantial documentation I could find made it seem very complicated. In the end I used a package called msmtp.

At the end of the README file for msmtp in Debian (/usr/share/doc/msmtp/README.Debian), it says:

It is possible to use msmtp as a MTA (like postfix, exim, etc.) by installing
the package msmtp-mta. It will provide the basic features required to be a MTA
(/usr/{sbin,lib}/sendmail, newaliases, etc.).

In other words, if you install msmtp and msmtp-mta packages, it can present itself as a sort of basic 'mail server' on local host for local services to connect to (eg. monit and unattended-upgrades), but then it connects to an external SMTP server to send the mail (in my case, that external server is provided by Gmail).

For this, we only need the system-wide config file - /etc/msmtprc. If this doesn't exist, create the file. In my case, the file looks like this:

# find out more about the configuration here: https://marlam.de/msmtp/msmtprc.txt
# Set default values for all following accounts.
defaults
logfile /var/log/msmtp.log

# A system wide configuration file is optional.
# If it exists, it usually defines a default account.
# This allows msmtp to be used like /usr/sbin/sendmail.
account default
host smtp.gmail.com
port 587
auth on
tls on
tls_starttls on

# Set the 'from' header on emails if it's not yet set
from system-messages@example.com
# Gmail credentials - do not put your main Google password here!
user system-messages@example.com
password MyAppSpecificPassword

# Syslog logging with facility LOG_MAIL instead of the default LOG_USER
syslog LOG_MAIL

Do not put your main Gmail password here - see Set up Gmail above

There's also a config example available in the full manual.

In my example above, the password is stored in plan text, which is a bad idea, but this is only a local server, and I've set the file permissions as recommended so that other users can't read it the file. If you figure out a simple way to encrypt the password, please let me know in the comments below!

chmod 0640 /etc/msmtprc
chown root:msmtp /etc/msmtprc

This sets the file owner to root, the file group to msmtp (the system user that runs the msmtp daemon), and restricts access to it.

Next I ran the test from the Debian docs:

mail -s "test" address@email.com <<END
This is a test
END

I got the test message, so I thought I was finished at this point, but I wasn't getting any messages from Monit or unattended-upgrades. In the Monit logs (/var/log/monit.log) I saw these errors:

[2023-04-25T16:43:24+0100] error : Cannot connect to [localhost]:25 -- Connection refused
[2023-04-25T16:43:24+0100] error : Cannot open a connection to the mailserver localhost:25 -- Operation now in progress
[2023-04-25T16:43:24+0100] error : Mail: Delivery failed -- no mail server is available

I asked for help on ServerFault and someone kindly pointed out that the MTA component of MSMTP probably isn't running yet. In other words, the msmtp-mta package had installed the daemon but hadn't set it to run automatically.

I found a guide on how to define a system stop-start file manually, but then I realised that the Debian package comes with an msmtpd.service file already (which was disabled), so I simply needed to run

systemctl enable msmtpd.service

and

systemctl start msmtpd.service

Check the status of it with systemctl status msmtpd.service, then reboot and check again to make sure the service starts automatically.

Done! Now I get regular emails from monit and unattended-upgrades.