As a newcomer to the tooling market, “Actions” by GitHub quickly became a popular option to develop CI/CD workflows. GitHub was already one of the top code management platforms, thus these additional features were a natural segway to enable end-to-end capabilities throughout your CI/CD pipeline—from writing code to production release.
In this post we’ll introduce GitHub Actions runners and show you how to set up GitHub Actions using self-hosted AWS runners in your Cloud Volumes ONTAP deployment.
Use the links below to get started with the step-by-step walkthrough:
A GitHub Actions workflow is a collection of steps that define what should be done after a new code contribution or change is made. Naturally, those steps require an environment where they can be executed.
GitHub Actions Runners are the virtual machines or containers where that execution happens. These environments can be public (i.e., provided on-demand by GitHub) or self-hosted (i.e., provided by you). The GitHub runner agent software supports multiple operating systems: Linux, Mac OS, and Windows.
A CI/CD workflow is not meant to be executed in a developer’s local environment. Since the workflow is triggered every time a new git commit happens, the workflow requires a computational environment that can either be span up on demand or kept always-on waiting for requests. However, there might be situations, especially during the creation of the CI/CD workflow, where it would be indeed useful to run Actions in a local environment.
The public GitHub runners are non-customizable standard environments that are deployed on demand. The alternative is to create and deploy a self-hosted runner. In a self-hosted runner, the agent software can be installed and configured in a machine you provide, such as a cloud server, container, or your local development environment.
The self-hosted runner capability is not specifically meant to gain the ability to execute GitHub Action workflows in your local machine. That is however, one of the multiple possibilities it enables. The other benefits of using self-hosted runners include:
Cloud Volumes ONTAP, the NetApp data management solution, is particularly good at helping to increase reusability to your CI/CD environment, providing data protection capabilities, and enhanced storage performance. Its FlexClone® data cloning technology enables data volumes (e.g., those containing datasets for CI/CD automation) to be rapidly cloned. Cloud Volumes ONTAP’s hybrid and multicloud nature lets data be copied and moved across multiple environments such as on-premises, AWS, Google Cloud, and Microsoft Azure.
Next, we are going to show you how to create and deploy a self-hosted AWS GitHub runner with Cloud Volumes ONTAP.
Deploying Actions in GitHub is a very straightforward process.
1.1. Choose any GitHub project you have created, either public or private, and navigate to the Actions tab. You will be greeted with an option to use one of the getting started examples.
1.2. Select the “set up this workflow” option to create a “Hello World” GitHub Actions workflow.
An example pipeline code file will be generated. Before committing the file to your project repository and enabling the workflow, please note how the code is structured.
Written in YAML format, the code has a value runs-on that is currently defined as ubuntu-latest, which means that the workflow will run in a public GitHub runner with the Ubuntu OS.
1.3. Commit the example pipeline file to your project as is, without any changes. Note that there are few steps that are going to be executed. First, it will print out a simple “Hello, world!” text, and then two sentences that are defined in a multi-line script.
As you might have noticed, the first workflow execution happened in a public runner (as defined in the runs-on value.
2.1. To transform our workflow to use self-hosted runners, we start by creating and deploying a new AWS EC2 instance using the default settings. If needed, you can refer to these simple instructions on how to set up a new EC2 instance.
2.2. In the GitHub project settings, find the Actions section and select the option to add a self-hosted runner located at the end of the page.
2.3. Choose Linux as the runner operating system and copy the commands in the page.
2.4. Start an interactive SSH session with the AWS EC2 instance you deployed.
2.5. Copy-paste the instructions from the GitHub self-hosted runner setting page into the instance terminal. When bootstrapping the GitHub Actions agent, use all default values for the option prompted, such as the name of the runner, tags, and work folder.
2.6. Run the agent application in the EC2 instance.
2.7. Go to the GitHub settings page and verify that the self-hosted runner appears in the list and is idling.
Our newly created self-hosted runner is active and running, however, the GitHub Actions workflow is still not going to take it into use. To enable that, edit the workflow file you created earlier and change the runs-on value to self-hosted.
3.1. After you commit the change to the file, the workflow will be executed again.
3.2. This time, the workflow is executed in the self-hosted runner. To validate that, look at the instance SSH session. A new line in the output of the agent, stating that the workflow job was successfully completed.
To take our self-hosted runner to the next level, we can leverage Cloud Volumes ONTAP.
As a prerequisite to this, you’ll need to connect NetApp Cloud Manager with your AWS Account. If you have done this in the past, you can proceed with creating the new cloud volume. Otherwise, please refer to these step-by-step instructions on provisioning Cloud Volumes ONTAP on AWS.
Make sure this prerequisite step is fully completed before moving forward. Once your AWS account is connected to Cloud Manager, you can create a new cloud volume for GitHub Actions.
4.1. Within NetApp Cloud Manager, add a new working environment and cloud volume. These following steps will focus on provisioning an iSCSI LUN from Cloud Volumes ONTAP to take it into use the EC2 GitHub Runner we set up previously.
4.2. Proceed through the next few Cloud Manager setup wizard steps, choosing the default settings until you reach the panel that asks you to input the volume details and protocol information.
4.3. Give the volume a random name and size.
4.4. In protocol, select iSCSI and Linux as the operating system type.
4.5. Use lvm_initator_group as the value for the new initiator group.
4.6. Under host initiator (IQN) use lvm_host as the alias with the value of the initiator name in your EC2 instance.
4.7. The initiator name value can be retrieved using the command cat /etc/iscsi/initiatorname.iscsi upon installing the initiator utils via yum install iscsi-initiator-utils -y
4.8. In Cloud Manager, review and approve the new volume creation.
4.9. The creation process might take a few minutes if this is the first time you’re using it. There are a few background preparation tasks that NetApp Cloud Volumes ONTAP is doing in your AWS environment to make it operational.
4.10. Once ready, you will be able to enter the working environment.
4.11. Inside the ONTAP Environment, you can access the created volume and have the ability to change different configuration settings. Also, you can create and manage additional volumes.
4.12. In the volume settings, select “Target IQN” to view its iSCSI target information.
4.13. Take note of the target iQN information—it will be needed later to attach the volume to the EC2 instance.
4.14. Next, click the burger menu in the upper righthand corner of the screen and select “Information” from the drop down.
4.15. The information panel will give fine-grained details about the volume and access IP addresses. Take note of the data (iscsi) value—it will also be required later to attach the volume to the EC2 instance.
Now we will show you how to set up Cloud Volumes ONTAP with your EC2 instance.
5.1. With the target IQN and data (iscsi) values at hand, login into the AWS GitHub Runner instance deployed earlier.
5.2. Enable the iSCSI service to start automatically on boot, using the commands sudo systemctl enable iscsid and sudo systemctl start iscsid
5.3. Next, use the iscsiadm tool to discover and connect to the Cloud Volume deployed earlier. Replace the IP address with the data (iscsi) value retrieved in Cloud Manager and verify that the output of the commands matches the target IQN value.
The following commands will automatically discover and connect the volumes: sudo iscsiadm --mode discovery --op update --type
sendtargets --portal 172.31.9.24
sudo iscsiadm --mode node -l all
sudo iscsiadm --mode session
5.4. Use the command sudo fdisk -l to verify that the new volume is now present in the Linux system as /dev/sdb.
5.5. Take the new volume into use by formatting using sudo mkfs.ext4 /dev/sdb and mounting as the working directory for the Actions Runner agent with the command: sudo mount /dev/sdb ~/actions-runner/_work/
5.6. Change the permissions to enable the local user account to use the volume with the following command: sudo chown -R ec2-user:ec2-user actions-runner/_work/
One key advantage of using cloud volumes is that we can store datasets to be used in our CI/CD workflow.
6.1. Download an image collection dataset—in this example we’ll use the Columbia University Image Library(COIL-100), a public dataset for computer vision research—to be able to perform tasks such as conversions, test automation, and data preparation using our GitHub Actions workflow.
6.2. The following steps will download and extract the COIL-100 dataset in our cloud volume working directory: cd actions-runner/_work/
wget http://www.cs.columbia.edu/CAVE/databases/SLAM_coil-
20_coil-100/coil-100/coil-100.zip
unzip coil-100.zip
With the dataset in the runner working directory, we can now start the GitHub Actions runner by going to the agent directory cd ~/actions-runner/ using the command ./run.sh
We can now change the steps of the “Hello World” GitHub Actions workflow, that was already pointing to the self-hosted runner, to something more challenging that can make use of the image dataset that we have stored in the Cloud Volume. The next commands should be used in the GitHub Actions workflow YAML code file.
8.1. First, we make sure the ImageMagick software is installed using the command yum install ImageMagick -y
8.2. Convert all images from the PNG to the JPG format using the following command: cd ~/actions-runner/_work/coil-100 && mogrify -format jpg *.png
8.3. Finish by listing the directory content and cleaning up using the commands: cd ~/actions-runner/_work/coil-100 && ls -lh
cd ~/actions-runner/_work/coil-100 && rm -fr *.jpg
See below the image for reference on how the different commands are mapped into independent steps.
The new steps are executed upon commit of the new workflow file. The output can be verified in GitHub Actions.
All steps are successfully completed and the format conversion of all images in the Cloud Volume happens automatically.
Using self-hosted runners in GitHub is a staple cloud automation best practice to follow. Not only will it increase security and give you the ability to control infrastructure as code, you’ll also be able to unlock new opportunities and benefits.
Cloud Volumes ONTAP is a great match to GitHub Actions Runners. As we saw in this step-by-step guide, it can be enabled easily in an existing setup without disrupting the existing workflow.
With the simple introduction of Cloud Volumes ONTAP, the self-hosted runner data can now be made available across different cloud environments, benefit from high availability, cost-cutting storage efficiencies, data protection and many other enterprise grade features that make Cloud Volumes ONTAP a popular choice among DevOps teams.
Read here to learn more about the Cloud Volumes ONTAP benefits for CI/CD.