Automate build, test and deploy of a static Jekyll site

In the last month I implemented CI/CD in multiple projects. Some of them were Javascript applications, others Python Flask REST Backends and Jekyll apps. I also used CI/CD methods to automate the generation of my LaTeX Awesome CV..

In this post I want to provide a complete tutorial on how to automatically build, test, integrate and deploy a Jekyll site to a FTP server. I describe how to use Bundle and Rake to create a virtual Ruby environment and how to use GitLab CI to execute the automation tasks.

Overall setup

In the past I developed the blog on my local machine, using jekyll serve to preview changes I made. If i had a state where I was confident that it is good enough to be published at ayeks.de, i pushed the files manually in Filezilla. Needless to say that this procedure takes some time. That was one of the reasons why I published so few posts in the past - the effort to publish changes was just to high.

Enough motivation to integrate everything in a pipeline. What is the current setup?

local GIT repository for development
Gitlab GIT repository as central repository
Gitlab CI Pipelines for build, test and deploy
ayeks.de Web and FTP server

Using a Rakefile for a Jekyll project

Just like the gitlab-ci file does task automation in GitLab, a Rakefile automates Ruby tasks. Jekyll is written in Ruby, so why don’t use it to automate it.

We will use the Rakefile as middleman between the GitLab CI script and the Jekyll codebase. We will later call Rake tasks from our command line and the gitlab-ci.yml file.

Rake and Bundle

A rakefile is code written in Ruby. We define tasks which can be called. To make everything as reproduce able as possible we will use bundle as our virtual environment for Ruby. In order to use the bundle you have to install it (eg. apt-get install bundler).

Install Rubygems with the Gemfile and Bundle

Then create a Gemfile where you list all your dependencies for the project. My Gemfile looks like that:

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
source 'http://rubygems.org'
ruby RUBY_VERSION
gem "jekyll-paginate"
gem "pygments.rb"
gem "html-proofer"
gem 'rake'
gem "jekyll", "3.4.0"

In the first two lines I define UTF8 encoding because of some strange ASCII errors which I get otherwise.. The third line defines the source for the Ruby gems. The rest of the gems are used by Jekyll and the tests. Also include rake to be able to execute the Rakefile tasks.

Install the Ruby gems with bundle install

Create the Rakefile for Jekyll

Now we are able to execute our virtual environment with bundle. Create a file calls Rakefile. I will go through the file step by step, have a look here our scroll down for the complete file.

First, we want to test if we can serve the Jekyll site with Rake. Insert the following into the Rakefile:

#encoding: utf-8
require 'jekyll'

desc 'Serve the site'
task :serve do
  puts 'Serving your website..'.bold
  Jekyll::Commands::Serve.process(livereload: true)
end

We import the Jekyll gem and create the task serve. In the task we will call the Jekyll serve command. For more information on the Rakefile have a look at the documentation. To call this task type into the command line: bundle exec rake serve

lars@lars-Inspiron-ubuntu:~/Documents/ayeksde/ayeksde$ bundle exec rake serve
** Invoke serve (first_time)
** Execute serve
Serving your website..
Configuration file: /home/lars/Documents/ayeksde/ayeksde/_config.yml
    Server address: http://127.0.0.1:4000//
  Server running... press ctrl-c to stop.

Sidenote: On my machine the live reload through rake wont work, so I call bundle exec jekyll s for the live reload. The benefit of using bundle for everything Ruby related is, that you don’t have to install the Gems with apt-get. Just use bundle for that.

Rakefile Build

require "rake/clean"
require 'jekyll'

task :clean do
  puts 'Cleaning up _site...'.bold
  Jekyll::Commands::Clean.process({})
end

desc 'Build the site'
task build: [:clean] do
  puts 'Building your website'.bold
  Jekyll::Commands::Build.process(profile: true)
  puts
end

The clean task removes generated files. The build task depends on the clean task, which will be executed before running the build script. It generates the static files and stores them to _site.

Rakefile Test

desc 'Test the site, depends on build'
task :test_html do
  puts 'Testing your website with HTMLProofer'.bold
  options = {
    :assume_extension => true ,
    :check_html => true,
    :empty_alt_ignore => false,
    :check_favicon => true,
    :check_img_http => true,
    :url_ignore => ["/dl.acm.org"]
  }
  HTMLProofer.check_directory("./_site", options).run
  puts 'Your website is now tested!'
end

desc 'Test the post structure to be sure no links to the site break'
task :test_structure do
  puts 'Testing post structure'.bold
  file = File.absolute_path('./_site/2016/12/tresorsgx/index.html')
  if !File.exists?(file)
    $stderr.puts "Error: Folderstructure has changed!".bold
    exit
  end
  puts 'The post structure is now tested!'
end

The test_html task uses HTMLProofer which checks the HTML syntax, validates the links and images.

The task test_structure tries to access a generated post. If the file is not accessible to directory structure of the posts has changed. That is bad because all external links would be broken. I don’t want that happen - therefore this check should always be true.

Complete Rakefile

#encoding: utf-8

require 'html-proofer'
require "rake/clean"
require 'jekyll'

Rake.application.options.trace = true

task :clean do
  puts 'Cleaning up _site...'.bold
  Jekyll::Commands::Clean.process({})
end

desc 'Serve the site'
task :serve do
  puts 'Serving your website..'.bold
  Jekyll::Commands::Serve.process(livereload: true)
end

desc 'Build the site'
task build: [:clean] do
  puts 'Building your website'.bold
  Jekyll::Commands::Build.process(profile: true)
  puts 'Your website has been built.'
end

desc 'Test the site, depends on build'
task :test_html do
  puts 'Testing your website with HTMLProofer'.bold
  options = {
    :assume_extension => true ,
    :check_html => true,
    :empty_alt_ignore => false,
    :check_favicon => true,
    :check_img_http => true,
    :url_ignore => ["/dl.acm.org"]
  }
  HTMLProofer.check_directory("./_site", options).run
  puts 'Your website is now tested!'
end

desc 'Test the post structure to be sure no links to the site break'
task :test_structure do
  puts 'Testing post structure'.bold
  file = File.absolute_path('./_site/2016/12/tresorsgx/index.html')
  if !File.exists?(file)
    $stderr.puts "Error: Folderstructure has changed!".bold
    exit
  end
  puts 'The post structure is now tested!'
end

desc 'Build the site and run all tests'
task ci: [:build, :test_structure, :test_html] do
    puts 'CI pipeline finished.'.bold
end

Thats the complete Rakefile of my project. For local testing I simply call bundle exec rake ci which builds and tests the Jekyll project:

lars@lars-Inspiron-ubuntu:~/Documents/ayeksde/ayeksde$ bundle exec rake ci
** Invoke ci (first_time)
** Invoke build (first_time)
** Invoke clean (first_time)
** Execute clean
Cleaning up _site...
Configuration file: /home/lars/Documents/ayeksde/ayeksde/_config.yml
           Cleaner: Removing /home/lars/Documents/ayeksde/ayeksde/_site...
           Cleaner: Nothing to do for /home/lars/Documents/ayeksde/ayeksde/.jekyll-metadata.
           Cleaner: Nothing to do for /home/lars/Documents/ayeksde/ayeksde/.sass-cache.
** Execute build
Building your website
Configuration file: /home/lars/Documents/ayeksde/ayeksde/_config.yml
            Source: /home/lars/Documents/ayeksde/ayeksde
       Destination: /home/lars/Documents/ayeksde/ayeksde/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 0.976 seconds.
 Auto-regeneration: disabled. Use --watch to enable.
Your website has been built.
** Invoke test_structure (first_time)
** Execute test_structure
Testing post structure
The post structure is now tested!
** Invoke test_html (first_time)
** Execute test_html
Testing your website with HTMLProofer
Running ["ImageCheck", "LinkCheck", "FaviconCheck", "ScriptCheck", "HtmlCheck"] on ["./_site"] on *.html...
Checking 68 external links...
Ran on 21 files!
HTML-Proofer finished successfully.
Your website is now tested!
** Execute ci
CI pipeline finished.

So now we are able to reproduce the environment for all the Ruby stuff with bundle and rake. Now we can call these tasks in GitLab CI.

GitLab CI Security preparations

For the deploy-to-FTP-part we will use GitLabs secret variables. In its documentation they state:

CAUTION: Important: Be aware that secret variables are not masked, and their values can be shown in the job logs if explicitly asked to do so. If your project is public or internal, you can set the pipelines private from your project’s Pipelines settings. Follow the discussion in issue #13784 for masking the secret variables.

We do not want to our secret variables to become public. Therefore, before anything else, disable public pipelines:

If you save credentials in your GitLab repository, everyone that has access to your GitLab account can read those credentials. Therefore I strongly recommend to activate GitLabs 2-Factor Authentication in your profile settings:

Using GitLabs secret variables

To be able to deploy to FTP we will set our credentials as secret variables. Go to your projects settings - CI / CD - Secret variables. Add three individual variables for your FTP username, password and the host.

I checked the box for protected because I want that the variables can only be used when building my protected master branch. If you want be able to use the credentials in every branch, leave the box unchecked. Read more about protected branches here.

In the end you should have created three variables:

gitlab-ci.yml file

I will describe the gitlab-ci file in detail. Scroll down or go here for the complete file. The GitLab runners will start when a .gitlab-ci.yml file can be found in the root directory of the repository. In the end, we will have a pipeline that looks like that:

gitlab-ci.yml Global settings

First of all, we define the docker image for all runners:

image: ruby:2.3

Then we define the different stages. We use the default stages:

stages:
  - build
  - test
  - deploy

We want to build the Jekyll site into the folder _site. Therefore we need to cache this folder between the different stages and runners. To do that we use the GitLab cache. We also want to store the vendor folder where all the Ruby stuff will be saved.

cache:
  paths:
    - vendor
    - _site

Before every job we want to install all the stuff required to run Jekyll and our tests. To do so, we use before_script. Bundle installs everything that is defined in the Gemfile.

before_script:
  - bundle install --path=vendor/

gitlab-ci.yml Build job

The build job generates all the static files with Jekyll through the Rakefile. build is the name and the stage of the job. In the script block you defined the commands you want to execute in the docker container, just like you execute these scripts on the local machine. The GitLab artifacts are used to store the _site folder, accessible from the outside. If anything goes wrong after the build, you can download the folder and have a look at the files. The artifacts will be deleted after 1 week to save disk space.

build:
  stage: build
  script:
  - bundle exec rake build
  artifacts:
    paths:
    - _site
    when: on_success
    expire_in: 1 week

gitlab-ci.yml Test jobs

I run two tests in my repository. Both checks will be executed in parallel in stage test. The first test, named test_html performs HTML syntax checks for the Jekyll sites.

The other test checks if the structure of the posts is still the same.

test_html:
  stage: test
  script:
  - bundle exec rake test_html

test_structure:
  stage: test
  script:
  - bundle exec rake test_structure

If all checks were successfull and the runner is executed in the master branch the deploy job will be executed. It first installs lftp using apt-get. Then it uses our secret variables to story the static HTML site at the defined FTP server.

deploy:
  stage: deploy
  script:
  - apt-get update -qq && apt-get install -y -qq lftp
  - lftp -c "set ftp:ssl-allow yes; open -u $ayeksde_user,$ayeksde_pw $ayeksde_ftphost; mirror -Rv _site/ ./  --ignore-time --parallel=10 --exclude-glob .git* --exclude .git/"
  only:
  - master

How the lftp command works in detail:

lftp -c  # Execute the following command
set ftp:ssl-allow yes;  # Try to use SSL
open -u  # open the connection with username, password hostname
$ayeksde_user,$ayeksde_pw  # my secret variables for username, password
$ayeksde_ftphost; # my secret variable for the ftphost
mirror  # we want to send data from a source to a target
-R  # R for reverse, we want to upload files
v  # use verbose output for more information
_site/  # our local directory
./  # the server directory
--ignore-time  # ignore the time when decide whether to uplodad
--parallel=10  # upload up to 10 parallel files
--exclude-glob .git* --exclude .git/  # dont upload git files

Now we put everything together:

Complete gitlab-ci.yml

image: ruby:2.3

stages:
  - build
  - test
  - deploy

cache:
  paths:
    - vendor
    - _site

before_script:
  - bundle install --path=vendor/

build:
  stage: build
  script:
  - bundle exec rake build
  artifacts:
    paths:
    - _site
    when: on_success
    expire_in: 1 week

test_html:
  stage: test
  script:
  - bundle exec rake test_html

test_structure:
  stage: test
  script:
  - bundle exec rake test_structure

deploy:
  stage: deploy
  script:
  - apt-get update -qq && apt-get install -y -qq lftp
  - lftp -c "set ftp:ssl-allow yes; open -u $ayeksde_user,$ayeksde_pw $ayeksde_ftphost; mirror -Rv _site/ ./  --ignore-time --parallel=10 --exclude-glob .git* --exclude .git/"
  only:
  - master

Have a look at the complete pipeline

Go to CI/CD overview of your project. When you commit something you will an overview similar to mine:

Both times the pipeline finished successfill. The branch cicd only has 2 stages because the deploy stage will only be executed in the master branch. The deploy stage was executed when merging the branch into master.

Thanks a lot for following through the tutorial. If you have any more questions on automating Jekyll sites with GitLab CI and Rake feel free to open an issue.

Automate build, test and deploy of a static Jekyll site.