Return to homepage

Upgrade Florence Pre-release

On Florence Mastodon v0.0.1.1 was released, and a few days later I upgraded "Bobadon.co.uk" to make use of it. This blog post outlines the steps I followed to achieve this so that if you decide to follow the "Install Florence Pre-release" tutorial I wrote last month, you will know how to upgrade your own setup to the latest version.

But before you continue, here is a reminder of that big red warning I provided last time:

WARNING: v0.0.1.1 is an early pre-release for this project. While it has proven to be stable on my own instance, it is strongly recommended that you wait until a proper v1.0.0 release before migrating across any existing servers you care about or relying on Florence for larger multi-user instances. The Florence project does not support these instructions so the community may only be able to offer limited help if you get stuck, and if you follow these steps you are doing so entirely at your own risk.

Now that is out the way, let us proceed!

Scaling up media storage

As I had almost completely filled up the hard disk on the Bobadon.co.uk server I decided to migrate the media files Florence Mastodon uses to a popular file storage host called Amazon S3. Not only is it very straightforward to configure, the service only charges 2¢ (US) per gigabyte per month. Seeing as I am already paying $10 (US) per month for the virtual server Bobadon.co.uk runs on this is a great way of keeping my costs as low as possible.

The best method I found for moving the existing media assets for my server is this handy guide from cyberspace: https://github.com/TrechNex/cybrespace-meta/blob/master/s3.md

The key difference to be mindful of is that the nginx configuration cyberspace's guide suggests to automatically redirect old image links to your S3 bucket will not work with Florence Mastodon. Instead of copying and pasting that step, you need to find this section in /etc/nginx/sites-available/florence:

location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {
  add_header Cache-Control "public, max-age=31536000, immutable";
  add_header Strict-Transport-Security "max-age=31536000";
  try_files $uri @proxy;
}

Now add this line before the last curly brace and edit it to match your own configuration:

rewrite ^/system(.*) https://s3-your-region-choice.amazonaws.com/yourbucketname$1 permanent;

Once you are finished, restart the nginx service. If everything worked then when you navigate to https://yourdomain.ext/system/media_attachments/whateveryourimageiscalled.png it should redirect to the correct file hosted by Amazon S3.

Fixing weird logging behaviour

After migrating the media files for Bobadon.co.uk to Amazon S3 I was surprised to discover that a huge amount of disk space was still being used, even after I deleted the contents of /home/florence/live/public/system/.

Initially I thought this might be the database and spent some time investigating how PostgresSQL could be better optimised and even considered switching to a "scaled up" alternative like Amazon RDS. Then it occurred to me that I should probably run the du command with some filters to confirm this assumption I was making. Much to my surprise, the database had a very small footprint and it was actually the system log file that was taking up gigabytes of space!

To resolve this problem I modified the /etc/systemd/journald.conf file and set the maximum size for system log messages:

SystemMaxUse=50M

I also set up a cron job in /etc/crontab that runs once a month and ensures the overall log size does not fill so much disk space again in future:

0 0 1 * * root journalctl --vacuum-size=200M

Upgrading Bobadon.co.uk

To achieve this you could just follow the same instructions as you would for upstream, albeit without the need to export and reimport your data: https://docs.joinmastodon.org/administration/migrating/

However there are some Florence-specific "gotchas" to watch out for, and if you followed the previous tutorial about installing the pre-release then you will need to tailor these steps to your setup. For example, these are the commands you need to stop the three services before upgrading:

$ sudo systemctl stop florence-web
$ sudo systemctl stop florence-sidekiq
$ sudo systemctl stop florence-streaming

The version tag you need is also different, and you must ensure the latest Ruby language dependencies have been installed too:

$ sudo su - florence
$ cd ~/live
$ git checkout v0.0.1.1
$ bundle install

Just like an upstream upgrade or migration you also need to recompile the Florence Mastodon code and re-build the toot feeds your users created:

$ RAILS_ENV=production bundle exec rails assets:precompile
$ RAILS_ENV=production ./bin/tootctl feeds build

If everything worked as expected, you should be able to restart all your existing Florence Mastodon services. Restarting nginx also ensures the underlying web server is using the correct settings:

$ sudo systemctl start florence-streaming
$ sudo systemctl start florence-sidekiq
$ sudo systemctl start florence-web
$ sudo systemctl restart nginx

Try Florence Mastodon yourself

You can register your interest in joining a general purpose Florence Mastodon instance with Zatnosk. They are currently in the process of planning to provision and administer a well-scaled instance and this should be available for you to use in the near future. Follow the ForkTogether fediverse account to keep up to date with Florence project announcements.