How to setup a shared filesystem in CJE

Issue

  • We have several tools/libraries that we want to share between Agents
  • We do not want to download all dependencies with each build from the repository

Environment

Resolution

We have two approaches to resolve these kind of use cases, one is to use a NFS shared filesystem mounted on every build worker and use it as volume in your Docker Templates, and the other is to use a Docker container with one or more volumes and is used by your Agents as volumes-from.

NFS shared filesystem

For this approach we suggest that we use a group id 1002 (gid) and a user id 1005 (uid), this is important because is the user and group that mount the NFS filesystem in the worker and is the user and group that run commands inside the Agent Docker container and can read/write in this shared file system, if you use another user and group to run command in your Agent Docker container, you should be sure that can read and write files in that NFS shared file system.

First of all, we have to create a folder in each build worker that will be used as mount point for the NFS shared filesystem. You can identified the build workers with the command cje run list-workers|grep build

Then enter on each build worker and create the mount point, the user and group to manage the shared filesystem and give the right permissions on that folder.

dna connect worker-3
sudo mkdir -p /mnt/mesos/sandbox/shared
sudo chmod 770 /mnt/mesos/sandbox/shared
sudo groupadd --gid 1002 shared-group
sudo useradd --shell /sbin/nologin --uid 1005 shared-user
sudo adduser shared-user shared-group
sudo chown -R root:shared-group /mnt/mesos/sandbox/shared

After that, you can mount the NFS shared filesystem in the mount point created

sudo mount -v -t nfs -o rw,vers=3.0 nfs.example.com:/nfs-shared-fs /mnt/mesos/sandbox/shared

Now, it is time to create a Docker image to work with this shared filesystem, it should have the user and group that we defined before with the same UID and GID, in order to be able to read/write files in the NFS shared filesystem, also we create a folder to mount the Bind Docker volume.

FROM openjdk

RUN groupadd --gid 1002 shared-group
RUN useradd --create-home --shell /bin/bash --uid 1005 --gid 1002 shared-user
RUN mkdir -p /mnt/shared && chown shared-user:shared-group /mnt/shared

you should build the Docker image and push it to your repository

sudo docker build -t shared-fs-agent .
sudo docker push my-repo.example.com/shared-fs-agent:1.0

once you have it done, you can test the Docker container to check that you can read/write in the NFS shared filesystem.

sudo docker run --rm --name test-shared-fs -it -v /mnt/mesos/sandbox/shared:/mnt/shared -u 1005:1002 shared-fs-agent /bin/bash
touch /mnt/shared/test-user

If every thing is ok, you should create a new Docker Template in CJE with the parameter USER set to shared-user:shared-group and a Volume set to mnt/mesos/sandbox/shared for the host path, and /mnt/shared for the container path.

Finally create a test Pipeline job to run the Agent and check the access to the filesystem.

stage('test agent'){
    node ('shared-fs'){
       echo 'Hello World'
       sh 'ls -la /mnt/shared'
       sh 'touch /mnt/shared/test-user-p'
       sh 'ls -la /mnt/shared'
    }
}

Docker container shared filesystem

This solution is based on a Docker container that only has a volume, this volume will be imported by others Docker containers and used as shared filesystem.

First of all you have to create the Docker container, and create the user and group to manage the files in it. It will have a volume where you will copy the shared files later.

FROM alpine:3.6

RUN addgroup -g 1002 shared-group
RUN adduser -D -u 1005 -G shared-group shared-user
 
RUN mkdir -p /mnt/shared && chown shared-user:shared-group /mnt/shared

VOLUME ["/mnt/shared"]

you should build the Docker image and push it to your repository

sudo docker build -t shared-docker-fs .
sudo docker push my-repo.example.com/shared-docker-fs:1.0

Then you can create the local container in each build worker

sudo docker create --name shared-docker-fs-img shared-docker-fs

And copy the shared files from a folder in the build worker to the volume in the Docker container.

sudo docker run -it --rm --volumes-from shared-docker-fs-img -v /mnt/source-files:/mnt/source-files shared-docker-fs /bin/sh
cp -R /mnt/source-files/* /mnt/shared/
chown -R shared-user:shared-group /mnt/shared

If every thing is ok, you should create a new Docker Template in CJE with the parameter volumes-from set to shared-docker-fs-img this will incorporate the volume /mnt/share to your Agent Docker container.

Finally create a test Pipeline job to run the Agent and check the access to the filesystem.

stage('test agent'){
    node ('shared-docker-fs'){
       echo 'Hello World'
       sh 'ls -la /mnt/shared'
       sh 'touch /mnt/shared/test-user-p'
       sh 'ls -la /mnt/shared'
    }
}
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.