Mastodon on a Ploi VPS

Thanks to a warped concept of “fun,” I decided to try running Mastodon instance on a modest VPS. It worked, and without much drama!

Below I’ll go through the steps I took to get Mastodon running on a freshly-paved Ploi server, with things I’ve learned migrating my account and what resource usage looks like on the virtual machine.

This would look similar with Laravel Forge and a Digital Ocean Droplet, and not too different with any machine that runs Ubuntu and PostgreSQL.

I like Ploi because it provisions a web-project-focused server for me, complete with SSH key management and best practices that button things up to optimize performance and avoid minor disasters. A friendly web GUI lets me take care of common things quickly when I need to, while running on whatever hosting platform I choose. I have no affiliation with the company, I just moved from Forge a while ago and prefer it.

Overview

This isn’t exactly my wheelhouse; I haven’t touched Ruby in a long time and Ploi is geared toward PHP apps. I’m also pretty new to Mastodon in general, so take all of this for the awkward little adventure it is.

Thankfully it’s 80% following clear instructions and adjusting for the setup I chose.

1. Prepare and provision a web server that runs PostgreSQL.
2. Install Ruby, required packages, and the Mastodon source.
3. Install Mastodon and give it service connection details for Cloudflare R2¹ (storage) and Mailgun (email).
4. Configure the web server.
5. Set up system services.
6. Establish backups and uptime monitoring.
7. Migrate account from ohai.social to t00t.cloud.

Provision the Server

A few years ago I got a sweet $30/year deal on a 2 CPU, 2GB NVMe virtual machine (thanks HostHatch!). It’s a perfect little champ for projects just like this.

I moved a tiny site off that server, reinstalled Ubuntu 22.04 LTS, and used Ploi to create a “Custom server” with PostgreSQL.

After Ploi initialized the server, I followed Mastodon’s install instructions with one key difference: I’d use the existing ploi user instead of creating a mastodon one like they recommend. Every Ploi server is set up to have the ploi user responsible for web services, so I assumed that’d be the path of least resistance.

Prep the Environment

I started with three quickish steps:

1. Created a new site on the server for my t00t.cloud domain. Default settings.
2. Pointed the domain name to the server and set up a Let’s Encrypt SSL certificate.
3. Created a mastodon database.

Then I SSH’d into the VPS and became root to run the first commands:

sudo bash

I skipped the instructions for setting up system repositories, Node.js, and PostgreSQL since Ploi took care of that.

I installed these packages exactly as demonstrated, which was a little hamfisted since some are already installed:

apt install -y \
  imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core \
  g++ libprotobuf-dev protobuf-compiler pkg-config nodejs gcc autoconf \
  bison build-essential libssl-dev libyaml-dev libreadline6-dev \
  zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev \
  nginx redis-server redis-tools postgresql postgresql-contrib \
  certbot python3-certbot-nginx libidn11-dev libicu-dev libjemalloc-dev

It went fine.

I also ran these commands because they said to:

corepack enable
yarn set version classic

Next—again just following the instructions—comes installing rbenv, which needs to happen as the ploi user.

So I backed out of root:

exit

Then set up rbenv:

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
cd ~/.rbenv && src/configure && make -C src
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
exec bash
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build

Everything seemed a little too easy, but I pressed on (again as the ploi user):

RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.0.4
rbenv global 3.0.4
gem install bundler --no-document

For whatever reason, the first line above never finished running and I eventually got a broken pipe. When I SSH’d back in, however, rbenv had indeed installed Ruby 3.0.4 and I could run rbenv global 3.0.4 and continue with no problem.

Computers can be fickle, but you don’t need me to tell you that.

Next, from my ~/t00t.cloud/ directory I cloned the Mastodon source and checked out the latest stable release:

git clone https://github.com/mastodon/mastodon.git live && cd live
git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)

Can we appreciate that git checkout line together for a second? Shells are neat.

Install Mastodon

Configuring and installing Mastodon took a few more commands and some waiting:

bundle config deployment 'true'
bundle config without 'development test'
bundle install -j$(getconf _NPROCESSORS_ONLN)
yarn install --pure-lockfile

Yarn barfed on the first attempt (last line), but I deleted its lock file, installed nvm and made sure I was using Node.js 16. The second attempt worked fine.

The excitement built as I ran the installer:

RAILS_ENV=production bundle exec rake mastodon:setup

I scrambled to whip up a Mailgun domain and S3 storage credentials before my session timed out. Then I mis-entered SMTP settings about twelve times. But you won’t do that because you’ll be better prepared. (Get your SMTP settings and S3-compatible storage endpoint and keys ready, if you’re using those!)

My resulting .env.production came out like this:

LOCAL_DOMAIN=t00t.cloud
SINGLE_USER_MODE=true
SECRET_KEY_BASE=•••
OTP_SECRET=•••
VAPID_PRIVATE_KEY=•••
VAPID_PUBLIC_KEY=•••
DB_HOST=127.0.0.1
DB_PORT=5432
DB_NAME=mastodon
DB_USER=ploi
DB_PASS=•••
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
S3_ENABLED=true
S3_ENDPOINT=https://•••.r2.cloudflarestorage.com/
S3_BUCKET=mastodon
AWS_ACCESS_KEY_ID=•••
AWS_SECRET_ACCESS_KEY=•••
S3_ALIAS_HOST=files.t00t.cloud
SMTP_SERVER=smtp.mailgun.org
SMTP_PORT=587
SMTP_LOGIN=•••
SMTP_PASSWORD=•••
SMTP_AUTH_METHOD=plain
SMTP_OPENSSL_VERIFY_MODE=none
SMTP_ENABLE_STARTTLS=auto
SMTP_FROM_ADDRESS=mastodon@t00t.cloud

I replaced the sensitive parts with ••• because publishing them seems silly.

Adjust the Web Server

I realized I needed live/public/ to be the document root. So I updated that in Ploi’s control panel (site Settings → Web directory).

Mastodon does some fancy stuff that requires an update to the site’s nginx settings. I blended Ploi’s default setup with the provided nginx config, removing unneeded PHP bits. The result has worked great.

For the site, visit Manage and click NGINX configuration.

This gist is my entire config. I was careful to work around the # Ploi Webserver Configuration (...) lines. At the very least, be sure to fix the domain name if you copy and paste the whole thing.

You can apply it by clicking Save, then Test configuration, and finally Deploy.

Set Up systemd Services

Almost there!

Mastodon runs system services with systemd, which need to be configured to run and survive a system restart.

This is a matter of copying files into the right place:

sudo cp dist/mastodon-*.service /etc/systemd/system/

And then starting and enabling them:

systemctl daemon-reload
systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming

If you’re me, you skipped a big obvious part and were sad to get 500 responses from the browser.

I figured out the problem pretty quickly thanks to the troubleshooting recommendation of running journalctl -u mastodon-web to investigate.

But I could’ve just read the docs in the first place to notice this part:

If you deviated from the defaults at any point, check that the username and paths are correct

My bold decision to embrace the ploi system user instead of the recommended mastodon one is exactly what they mean by deviating from the defaults.

I needed to edit each of these files to change two things:

1. References to the /home/mastodon/live path needed to be /home/ploi/t00t.cloud/live.
2. References to the mastodon user (like User=mastodon) needed to be User=ploi.

I use nano a lot, and you don’t have to like that but I need you to accept it for now because I forget how to use vi. So as the root user (sudo bash)…

nano /etc/systemd/system/mastodon-web.service
nano /etc/systemd/system/mastodon-streaming.service
nano /etc/systemd/system/mastodon-sidekiq.service

After saving these edits, I could finally load and reload and restart the services without issue. (Which makes sense once they’re pointing to users and directories that exist.)

I could finally visit the instance in a browser and get everything configured!

Pro tip: if you change your configuration, recompile assets, or create any kind of disturbance in the force, restart the web service by running
sudo systemctl restart mastodon-web.

Monitoring and Backup

I set up a HetrixTools server monitor and uptime monitor to keep an eye on system health. HetrixTools doesn’t have a dazzling UI, but the monitoring has been rock-solid with almost no false positives in the many years I’ve been using it (after Pingdom and UptimeRobot).

And just in case I toot some really valuable jokes at some point, I used a variation of my trusty old restic + Backblaze B2 combo for incremental, encrypted, inexpensive off-site backups.

Resource Usage

I didn’t find many conversations about resource guidelines for small Mastodon servers², so I was curious how it’d do on this thing. I’ve tried apps before that needed way more than two cores and two gigabytes of memory could handle, even with a single user. (I’m looking at you, Java.)

To recap, my VPS gets access to two AMD Epyc cores, 2GB of memory, and fast NVMe storage. It doesn’t see much IO wait, and it’s much nicer than it has any right to be for the price.

The Mastodon instance has one user (👋) that’s not particularly active, and in the few days I’ve been watching it’s been perfectly stable and responsive. The entire system uses about 8GB of storage, CPU load generally bounces betwen 2.5–5% with modest spikes to ~15%. The 2GB of memory, however, is perpetually 60–75% utilized. There is some swapping, and I’m not proud of it.

I’m not sure what this means for you, but if I was spinning up my own new solo Mastodon instance I wouldn’t pick lower specs than this. One CPU core may cut it, but I doubt less than 2GB of memory would suffice.

That said, everything seems to work well and queued tasks constantly clear out quickly and successfully. No big difference in my day-to-day usage.

Migrating Accounts

I read many times that you could migrate from one Mastodon server to another with a minimal amount of pain.

Getting ready to actually do it, I learned some new things:

• There’s a cooldown period after changing servers, so you can’t do it constantly. (Fine unless my experiment goes poorly and I can’t escape a flailing server for a few weeks.)
• You can download your data, migrate your followers and redirect smoothly, but your toots and likes don’t come with you. There are projects for that you can use if you run your own server, but it gets complicated.
• If you’re using lists (I wasn’t), those don’t migrate either. But federike.social is a lovely app for organizing or re-organizing them.
• Exporting and importing your followed accounts should be straightforward, but it doesn’t seem to work with Mastodon 4.1.0. I originally couldn’t export and import my followed accounts—the export was fine but the import failed silently. Two days later, it randomly worked. 🤷‍♂️

I followed instructions from the article I linked to above and everything worked quickly and smoothly.

If you’re doing this yourself, have fun and be prepared for the particulars of that account transition! If I forgot something or made some kind of glaring mistake, kindly give me a shout.