Categories
Daily Musings

How to Fix: Gitlab LetsEncrypt Renewal Failure

So my LetsEncrypt certificate for Gitlab expired, and I wanted to see if I could manually renew it, since it did not renew on its own.

I was getting an error in the console when running sudo gitlab-ctl renew-le-certs that said the response from the server was malformed.

Digging deeper, it seems that it can be really simple to get around.

Simply edit /opt/gitlab/embedded/cookbooks/letsencrypt/resources/certificate.rb and comment out the lines in the acme_certificate section as shown in the image above.

Now you should be able to force the certificate to renew by manually running sudo gitlab-ctl renew-le-certs again!

Enjoy!

Categories
Daily Musings

Restoring Gitlab from its own backups is a pain, but it's possible!

Gitlab helpfully lets you know it’s making backups of the database each time it does an upgrade, but trying to find information on where those files are stored by default on the web is an exercise in futility. Most links lead to Gitlab themselves and they sure as hell aren’t telling you.

Well I hope to help someone else (and possibly myself in the future) by listing a few useful ones here.

Step 1: Needed files

To restore from a boot / OS failure, or after a fresh install, you’ll need the following files:

  • /etc/gitlab/gitlab.rb
  • /etc/gitlab/gitlab-secrets.json
  • the entire /var/opt/gitlab/git-data folder
  • the file you want from /var/opt/gitlab/backups
  • install the exact same version of gitlab you were previously running before the failure

So hopefully you were able to rescue those files as they contain your settings, secrets, the db backup and the repositories themselves.

Step 2: Install Gitlab

Having gathered all of these things, you’ll need to first install the relevant version of Gitlab. I needed the version before current so the command in Xubuntu (using apt) is:

sudo apt install gitlab-ce=12.5.3-ce.0

12.5.3 is the version the restore process told me I needed. The -ce.0 part is important if you’re using gitlab-ce.

If you’ve already installed a prior version before coming here, you can look at the output of that command for clues on how to get it to continue despite you already having a database.

Step 3: Restore the database

Now copy the backup files from where you have them into /var/opt/gitlab/backups:

sudo cp -R <source folder goes here> /var/opt/gitlab/backups/

Now that you have the right version installed, you can restore the backup. Make sure you set the permissions on /var/opt/gitlab/backups so the user git can access it.

# Allow the git user to access the backups
sudo chown -R git:git /var/opt/gitlab/backups

# List the backups to see what we can restore!
sudo ls -alF /var/opt/gitlab/backups

You should get a listing of the different backups that were made. Usually you’ll want to pick the most recent one, which will be listed last.

Next you type in the command to actually restore the backup database. My backup file was 1576180022_2019_12_12_12.5.3_gitlab_backup.tar

We need to take the _gitlab_backup.tar part off of the end and use that as our file to restore:

sudo gitlab-backup restore BACKUP=1576180022_2019_12_12_12.5.3

If you have any problems, the fix is usually in the error output. You may run into an issue if you have the wrong version installed. Go back to Step 2: Installing Gitlab and fetch the version it tells you to.

Step 4: Restore the repositories

After the db is restored, you may still need to copy the repositories over too. Those are located in the /var/opt/gitlab/git-data folder.

sudo cp -R <source folder goes here> /var/opt/gitlab/git-data/

# Allow Gitlab to access the files
sudo chown git:git /var/opt/gitlab/git-data/

I’ve had some luck simply copying them over but sometimes that doesn’t work. So you may need to import those repositories manually. That fix is outside the scope of this short how-to, but keep at it and you’ll get things back up and running in no time!