1. Introduction
2. Prerequisites
3. Installing GitLab Runner on Windows
4. Runner Basics
5. Yml Script Example for Builds and Tests
   • (A) Variables
   • (B) Build/Test Environment Setup
   • (C) Build and Test Jobs
6. Scheduled Builds
7. Testing Your Runner
8. Useful Tips for YML Scripts
   • Using Merge Request Labels as Conditionals
9. Workarounds for Slow 'base' Repo Cloning
   • (A) Clone/Copy the Repo Beforehand
10. Runner Concurrency

Introduction

This page will show you how to set up a basic runner on Windows using the Shell Executor on Gitlab to make builds and run tests.

Prerequisites

• Maintainer access to the repository you will be setting up the runner for.
• A computer to use as the runner (our example computer has Windows, CET Developer, and Git Client.
• Administrator access to the runner computer.
• CET Developer version on the runner computer should be the latest available and logged into CET Developer with the same Email as Windows.

Installing GitLab Runner on Windows

1. Create a folder somewhere on the runner machine. We use "C:\Gitlab-Runner" for this example.
2. Download the 64-bit binary for Windows here (v11.11.0 - used for the examples in this wiki page, later versions might not work).
3. Put the downloaded binary into the folder you created in step 1. Rename it to gitlab-runner.exe
4. Register the Runner.

   1. Open an administrative command prompt.

   2. In the prompt, go to the folder you created in step 1.

      

   3. Type "./gitlab-runner.exe register:

      

   4. Our GitLab instance URL is as https://git.configura.com:

      

   5. You can find your runner registration token under "Settings -> CI/CD -> Runners" in your Gitlab project:

      

   6. The description for the runner will show up in the runner's jobs. Put anything you want, you can change it later

      

   7. For the tags, put a simple "build" tag for now. You can modify these easily later.

      

   8. When asked for the runner executor, please input shell

      

5. Install the runner as a service and start it. The default installation will install the service under the Built-in System Account. DO NOT do this. CET Developer has an issue where it cannot find the correct folders for builds if you do so! Instead, install the runner under the same WINDOWS user account that will be logged in the machine to avoid issues:

   

You can change the logon here to be perfectly sure:

Instructions in this section are taken from Gitlab Official Docs.

Please refer to them if you run into any issues! For additional configurations to your runners like concurrency, refer to here.

Runner Basics

• The CI/CD and runners on Gitlab parse the ".gitlab-ci.yml" file whenever a push is made to the Gitlab server.
• This file has to be made in the root of your repository.
• The ".gitlab-ci.yml" file describes jobs, which have their own scripts and conditions on when to run (such as only on merge requests, certain branches, etc).
• After parsing the yml, the jobs will be made on the server which will then be picked up by runners.

Yml Script Example for Builds and Tests

Below is a yml script that runs a build job and a test job when merge requests are made to the "version10.0" branch in the repo. Explanations for each section can be found further below. There is also a quick start guide that can be found here but please be aware that the quick start guide is covering a lot less than the one below.

variables:
  CUSTOM_REPO: "myBrand"
  BASE_VERSION: "version10.0"

.SetupBuildEnv:
  before_script:
    # Base has to be cloned outside the repo, because of default git clean options
    - cd ..
    - if not exist base (git clone https://gitlab-ci-token:%CI_JOB_TOKEN%@git.configura.com/cet/external/base.git)

    # Set CM environment variables + add custom to cm class_path args
    - powershell.exe -file "base/bin/createsetenv.ps1"
    - call setenv.cmd

    # Setup base branch
    - cd base
    - git checkout %BASE_VERSION%
    - git reset --hard
    - git clean -fd
    - git pull

    # Make a simulated merge for building
    # This is cleaned up in the after_script
    - cd %CI_PROJECT_DIR%
    - git checkout -b runner/%CI_COMMIT_REF_NAME%/%CI_JOB_ID%
    - git fetch origin %CI_MERGE_REQUEST_TARGET_BRANCH_NAME% 
    - git merge origin/%CI_MERGE_REQUEST_TARGET_BRANCH_NAME%

    # Make symlink in extensions/custom/myBrand to to match folder structure we usually get.
    # (extensions/custom/xxx)
    # We need this until they fix -https://gitlab.com/gitlab-org/gitlab-runner/issues/4167
    - cd ..
    - if exist "extensions" rmdir /s /q "extensions"
    - mkdir "extensions/custom"
    - mklink /D "extensions/custom/%CUSTOM_REPO%" "%CI_PROJECT_DIR%"

    # Clean write folder + BuildOutput
    - if exist write rd /s /q write
    - if exist BuildOutput rd /s /q BuildOutput

  # Always runs even script/build fails.
  after_script:
    # We have to move BuildLogs within the original repo if we want to upload it
    # (Artifacts limitation that isn't clearly documented by Gitlab :D ) 
    - cd %CI_PROJECT_DIR%/..
    - move BuildLogs %CI_PROJECT_DIR%

    # Delete local merged branch
    - cd %CI_PROJECT_DIR%
    - if exist .git/MERGE_HEAD (git merge --abort)
    - git checkout %CI_COMMIT_REF_NAME%
    - git branch -D runner/%CI_COMMIT_REF_NAME%/%CI_JOB_ID%

Build:
  tags:
    - build
  variables:
    GIT_STRATEGY: Fetch
    GIT_DEPTH: 1000
  extends: .SetupBuildEnv
  script:
    # Make operator/cop recognise the workspace
    - cd %CI_PROJECT_DIR%/..
    - powershell.exe -file "base/cm/core/test/runner/cop/Workspace/createcopcmws.ps1"

    # Build/Upload
    - cop clean
    - cop repair
    - cop make [myBrand] -minggwformat
    - cop upload [myBrand] /uploadname="runner-myBrand-[%CI_MERGE_REQUEST_IID%]-[%CI_JOB_ID%]"
 
  # Upload build logs even on job failure so we know what went wrong 
  artifacts:
    when: always
    paths:
      - BuildLogs/*

  # Only on merge requests to the branch.
  only:
    refs:
      - merge_requests
    variables:
      - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "version10.0"

Test:
  tags:
    - build
  variables:
    GIT_STRATEGY: Fetch
    GIT_DEPTH: 1000
  extends: .SetupBuildEnv
  script:
    # Make operator/cop recognise the workspace
    - cd %CI_PROJECT_DIR%/..
    - powershell.exe -file "base/cm/core/test/runner/cop/Workspace/createcopcmws.ps1"

    # Just test
    - cop clean
    - cop repair
    - cop make [myBrand TestGroup] -minggwformat

  # Upload build logs even on job failure so we know what went wrong 
  artifacts:
    when: always
    paths:
      - BuildLogs/*

  # Only on merge requests to the branch.
  only:
    refs:
      - merge_requests
    variables:
      - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "version10.0"

(A) Variables

variables:
  CUSTOM_REPO : "myBrand"
  BASE_VERSION: "version10.0"

These are just global variables across the yml script. Change "myBrand" to the name of your repo. This is the same as the top-level folder of your extension in the cm code, like "custom/myBrand". "BASE_VERSION" is the version of the 'base' repo you want builds to be made with (where the CM core code is).

Within the yml scrip, you may see variables like "CI_MERGE_REQUEST_TARGET_BRANCH_NAME", "CI_JOB_TOKEN" and other variables that start with a "CI". These are predefined environment variables that make it easier to write scripts for the runner. You may find a list of them here

(B) Build/Test Environment Setup

This is used for both jobs. We are setting up environment variables, the correct directory structure, then making a simulated merge before building and testing. After building and testing, we move the build logs/test logs to a location where Gitlab can upload them, and clean up the simulated merge.

.SetupBuildEnv:
   before_script:
     # Base has to be cloned outside the repo, because of default git clean options
    - cd ..
    - if not exist base (git clone https://gitlab-ci-token:%CI_JOB_TOKEN%@git.configura.com/cet/external/base.git)

    # Set CM environment variables + add custom to cm class_path args
    - powershell.exe -file "base/bin/createsetenv.ps1"
    - call setenv.cmd

    # Setup base branch
    - cd base
    - git checkout %BASE_VERSION%
    - git reset --hard
    - git clean -fd
    - git pull

Getting the base repo for build, setting CM environment variables, and then making sure the base repo is up to date. If the network speed of the runner machine to Gitlab servers are slow, it is highly likely that cloning the base repo will take too long and fail. To workaround this, see this section.

# Make a simulated merge for building
# This is cleaned up in the after_script
- cd %CI_PROJECT_DIR%
- git checkout -b runner/%CI_COMMIT_REF_NAME%/%CI_JOB_ID%
- git fetch origin %CI_MERGE_REQUEST_TARGET_BRANCH_NAME% 
- git merge origin/%CI_MERGE_REQUEST_TARGET_BRANCH_NAME%

We make a 'simulated merge' by first branching out to a different branch, then merging the target branch of the merge request to that branch. We do this because otherwise the builds/tests will be running on the 'detached' feature branch instead of the result of the merge.

# Make symlink in extensions/custom/myBrand to to match folder structure we usually get.
# (extensions/custom/xxx)
# We need this until they fix -https://gitlab.com/gitlab-org/gitlab-runner/issues/4167
- cd ..
- if exist "extensions" rmdir /s /q "extensions"
- mkdir "extensions/custom"
- mklink /D "extensions/custom/%CUSTOM_REPO%" "%CI_PROJECT_DIR%"

This step creates a folder symlink between "extensions/custom/myBrand" with the source code cloned by the runner from the repo. This step looks ugly, but is needed for now because we cannot change the cloning path for the runner (until Gitlab resolves the issue in a later version - it defaults to cet/external/<group_name>/<repo_name>). The correct structure for the code is important so that cm can find the packages it needs to use.

# Clean write folder + BuildOutput
- if exist write rd /s /q write
- if exist BuildOutput rd /s /q BuildOutput

We remove the write and the buildOutput folder so we have clean builds everytime.

# Always runs even script/build fails.
after_script:
    # We have to move BuildLogs within the original repo if we want to upload it
    # (Artifacts limitation that isn't clearly documented by Gitlab :D ) 
    - cd %CI_PROJECT_DIR%/..
    - move BuildLogs %CI_PROJECT_DIR%

    # Delete local merged branch
    - cd %CI_PROJECT_DIR%
    - if exist .git/MERGE_HEAD (git merge --abort)
    - git checkout %CI_COMMIT_REF_NAME%
    - git branch -D runner/%CI_COMMIT_REF_NAME%/%CI_JOB_ID%

We move buildLogs to within the original repo directory because Gitlab will not be able to upload anything outside it. The buildLogs are important to figure out what went wrong if the build fails. We also delete the new branch we made for the simulated merge.

(C) Build and Test Jobs

Build:
  tags:
    - build
  variables:
    GIT_STRATEGY: Fetch
    GIT_DEPTH: 1000
  extends: .SetupBuildEnv
  script:
    # Make operator/cop recognise the workspace
    - cd %CI_PROJECT_DIR%/..
    - powershell.exe -file "base/cm/core/test/runner/cop/Workspace/createcopcmws.ps1"

For the build/test jobs, we use "GIT_STRATEGY: Fetch" to fetch instead of clone the entire repo each job because cloning every job takes too much time.

The depth for git fetch is set to an arbitrarily large number so that we can find a common point for our simulated merges in (B).

Extends adds the 'before_script' and 'after_script' from .SetupBuildEnv to our build/test job, and is required to make sure the job works as intended.

We use the "build" tag here because we registered our runner with the "build" tag earlier.

script:
    # Make operator/cop recognise the workspace
    - cd %CI_PROJECT_DIR%/..
    - powershell.exe -file "base/cm/core/test/runner/cop/Workspace/createcopcmws.ps1"

    # Build/Upload
    - cop clean
    - cop repair
    - cop make [myBrand] -minggwformat
    - cop upload [myBrand] /uploadname="runner-%CUSTOM_REPO%-[%CI_MERGE_REQUEST_IID%]-[%CI_JOB_ID%]"

For actually making builds and running tests, we use cop, short for Commandline Operator. COP is a command-line interface for CET Operator/Developer. Find out more about COP  here.

For every job, we have to create a .cmws file for COP to find the workspace. Before building/testing, we clean and repair the workspace.

This example will build the build group "[myBrand]". To find out more about build groups and how cop works please go here.

After building, the builds will be uploaded to MyConfigura Build Center. The "/uploadname" sets the upload folder name. The example name given here adds a "runner-" prefix to your brand, then makes use of predefined environment variables to add the Merge Request ID and Job ID as a suffix. This makes it easier to find the build in Build Center.

# Upload build logs even on job failure so we know what went wrong 
   artifacts:
     when: always
     paths:
      - BuildLogs/*

# Only on merge requests to the branch.
   only:
    refs:
      - merge_requests
     variables:
      - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "version10.0"

Artifacts are what the runner will upload to Gitlab after the job completes. We upload the BuildLogs here so we can debug any build errors.

The only section makes it so that the build job only runs on merge requests to the version10.0 branch. It is possible to use ruby regex for this section. For example, the below will run on any branch that starts with 'version':

- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME =~ /^version*/

An 'except:' parameter also exists that does the opposite of only (run on every branch BUT...)

The test job is the exact same as the Build job with the exception that COP builds the tests instead of the extension. This is similar to opening CET Developer Build Central, selecting the test group, and pressing build.

- cop make [myBrand TestGroup] -minggwformat

instead of

- cop make [myBrand] -minggwformat

Scheduled Builds

Please refer to this page to see how to make scheduled builds for your GitLab repository.

Testing Your Runner

To set up a test branch and ensure your runner is working as intended, follow the steps below:

1. You want to make a test branch that includes the .gitlab-ci.yml shown in the above section that does builds and tests.
   1. Make sure you place the file in the root of your repo folder
   2. Make sure you change all the "myBrand" examples to use your brand's and your repo's names.
2. Make a merge request to the target branch specified in the yml, in our example it is "version10.0":

   # Only on merge requests to the branch.
     only:
       refs:
         - merge_requests
       variables:
         - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "version10.0"

3. If you open the Merge Request after submitting it you should be able to see "Pipeline" as shown below:

   

   Clicking on the "Circle" before the pipeline or the number (#xxxxx) after the pipeline should bring you to a page where you can see the jobs:

   

   Clicking on the jobs will bring you to a page with a long command-line style log:

   

   This is your job trace or job log. You will be using this to debug your job.

4. If everything goes well, you should be able to see a section where COP is building your extensions:

   

   Note that it will take some time for COP to build before you see the "success" or "error" messages.

5. The bottom of the log will show if your job succeeded or not.

   
   
   Note that this specific "CertPool" warning ALWAYS appears on Windows machines when uploading artifacts, and that's a harmless GitLab bug, so you can just ignore it.

   

6. The job should upload the BuildLogs, so if your build fails, you can figure out why. You can find them on the right side as "Job Artifacts"

   

Useful Tips for YML Scripts

Using Merge Request Labels as Conditionals

In some cases, you want to run builds, upload builds, and run tests on merge requests, but not on EVERY merge request. One way to add a conditional that is easy for developers to change themselves without needing to adjust the yml scripts or mess with the pipelines is to make use of merge request labels.

If we want to use the same job, but only upload if there is an "upload" merge request label, we can make the conditional in the shell like this:

- if not [%CI_MERGE_REQUEST_LABELS:Upload=%]==[%CI_MERGE_REQUEST_LABELS%] (cop upload [myBuild] /uploadname=xxxxx)
      else echo ("Not uploading to MyConfigura, there is no "upload" label set

The %CI_MERGE_REQUEST_LABELS% environment variable gives you comma-separated labels in a string. The code in the above section actually does a trick where we substitute the label we want ("Upload") with an empty string, then check if %CI_MERGE_REQUEST_LABELS% remains the same before deciding to upload builds.

Another option would be to make two separate jobs, with one job uploading if the label exists and one job uploading if it does not. You would put the check in the "only" and "except" section of the yml like this:

only:
    variables:
      - $CI_MERGE_REQUEST_LABELS =~ /(^|,)upload(,|$)/ 

except:
    variables:
      - $CI_MERGE_REQUEST_LABELS =~ /(^|,)upload(,|$)/ 

The first example only runs the job if there is an 'upload' label.

The second example only runs the job if there is no 'upload' label.

Gitlab and thus, the above examples use Ruby Regex. The example is slightly more thorough because you could have labels like 'soupload' causing problems (soupload)

Workarounds for Slow 'base' Repo Cloning

Before the job is run, the runner will first check if the repo already exists, otherwise, it will clone the repo. This can be a problem if it takes too long.

(A) Clone/Copy the Repo Beforehand

A very easy way to avoid this issue is to ensure the repo is already cloned before the job even starts. Copy or clone the repo at least one-level above where the runner clones the custom_repo. This is because, by default, every job runs a git clean which will remove the base repo if it is placed inside the custom repo.

To determine your runner's build directory, you can let it run its first job, then cancel it. The default build directory can be found somewhere within where you placed the "gitlab-runner.exe":

Copy/Clone the repo at least one level above where the custom_repo will be cloned to avoid it being deleted by git clean.

If you copied the repo, open a bash/cmd prompt in the copied repo location and do a quick 'git-status' to refresh the git index.

For subsequent jobs after the first, we overcome this by having the runner fetch instead of clone by setting "GIT_STRATEGY" to "fetch". Because we fetch and also pull in the script for every job, it also helps keep the repo up-to-date.

Runner Concurrency

By default, the number of jobs a Gitlab runner is allowed to run at a time is 1. This can be changed in the runner's "config.toml" file.

1. Go to where the gitlab-runner.exe is located and open the toml file in a suitable text editor (at least notepad++).

2. Edit concurrent to the number you want.

   

3. Restart the Gitlab Runner (needs admin cmd prompt):

   

4. If you had to clone/copy the "base" repo or other repos beforehand as a workaround to slow network speeds, make sure to do the same for the other build workspaces. The builds directory will have the number of workspaces equal to the number of concurrent jobs you set.

   

- cop make [fika] /maxWorkers=4

Other configuration settings that can be changed for the Gitlab Runner can be found in the official docs here