Why is my multibranch project cloning the whole repository on the master?

Issue

  • My multibranch pipeline clones the entire repository on the master to obtain the Jenkinsfile
  • My multibranch pipeline shows a timeout error when cloning the repository to obtain the Jenkinsfile:
Branch indexing
23:43:32 Connecting to https://api.github.com using my-credentials/****** (my-credentials GitHub personal access token)
Checking out git https://github.com/org/repo.git into /var/jenkins_home/workspace/_repo_PR-123-P7DALB4WUZUT3GSAR3HPW2RCNJME43BF5FNLWOCKUK55BAHQV6SA@script to read Jenkinsfile
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/org/repo.git
 > git init /var/jenkins_home/workspace/_repo_PR-123-P7DALB4WUZUT3GSAR3HPW2RCNJME43BF5FNLWOCKUK55BAHQV6SA@script # timeout=10
Fetching upstream changes from https://github.com/org/repo.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials my-credentials GitHub personal access token
 > git fetch --no-tags --progress https://github.com/org/repo.git +refs/pull/123/head:refs/remotes/origin/PR-123 +refs/heads/master:refs/remotes/origin/master
ERROR: Timeout after 10 minutes
[...]

Environment

Related Issues

  • JENKINS-33273: Optimize Jenkinsfile loading and branch detection
  • JENKINS-43194: Lightweight checkout for PR merge jobs
  • JENKINS-42518: Provide lightweight checkout capability for bitbucket to avoid repository clone for multi-branch pipeline jobs

Resolution

Pipeline from SCM used to perform several checkout as explained in JENKINS-33273. Since version 2.13 of Pipeline Multibranch Plugin and the resolution of JENKINS-33273, Pipeline from SCM offers the ability to perform a lightweight checkout - if the SCM supports it - to retrieve the Jenkinsfile. With a lightweight checkout, no clone is needed to retrieve the Jenkinsfile anymore. The Jenkinsfile is instead retrieved using an API.

The following plugin provide support for lightweight checkout:

However in the case of merge Pull Requests only those versions of the plugins support a lightweight checkout:

In unsupported versions, plugin falls back to the previous behavior - a heavyweight checkout - which mean that a full clone is performed to retrieve the Jenkinsfile.

For such cases, there are workarounds that can be used until further support for a lightweight checkout is provided.

Workaround 1: Use a reference Repository

This workaround requires SCM API v2.2.0: Prior to that version, only the behavior of the checkout step can be adjusted - see [How to Customize Checkout for Pipeline Multibranch? - but not the behavior of the checkout carried out to obtain the Jenkinsfile.

A reference repository is a general tip to speed up the checkout/cloning of large Git repository. You can set up a reference repository on the master that will serve as a cache for your repository. It also saves disk space and reduce the traffic between Jenkins and the Git server.

First, have a look at the article Using a Git reference repository to understand how to set up a reference repository.

Then, in the configuration of the organization / multibranch project, add an SCM Trait of type Advanced Clone Behaviors and specify the path to the reference repository:

Workaround 2: Increase the Git timeout

If affected by the timeout error, it is possible to increase the Git timeout.

You can increase the timeout globally by setting the system property -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=<number of minutes> on startup. Upon restart, you can check that the timeout is properly set by running the groovy script return org.jenkinsci.plugins.gitclient.CliGitAPIImpl.TIMEOUT under Manage Jenkins > Script Console.

SCM API v2.2.0 and later

You can set the timeout per organization / multibranch. In the configuration of organization / multibranch project, add the SCM Traits of type Advanced Clone Behaviors and Advanced Checkout Behaviors and set a suitable value for the timeout (in minutes):

About SCM API v2.2.0

Since version 2.2.0 of the SCM API Plugin, it is now possible to add SCM Traits - like the git additional behaviors - at the organization / multibranch level. These traits apply to:

  • the checkout performed to retrieve the Jenkinsfile
  • the checkout scm step

SCM API v2.2+ along with GitHub Branch Source 2.2+ and Bitbucket Branch Source 2.2+ are available under the CloudBees Assurance Program since CJE 2.73.3.1.

About Multibranch Pipeline support

Multibranch Pipelines support lightweight checkout since JENKINS-33273.
It is activated by default.
There is a kill switch to deactivate it.
Simply add org.jenkinsci.plugins.workflow.multibranch.USE_HEAVYWEIGHT_CHECKOUT=true to Jenkins arguments (please refer to How to add Java arguments to Jenkins?).

Why is my Pipeline job cloning the entire repository on the master even when I select a lightweight checkout?

In order to work the lightweight checkout needs a cache repository.
This cache repository is created the first time on the master.

Why is my Multibranch job cloning the entire repository on the master ?

Like for a standard Pipeline job, lightweight checkout needs a cache repository.
This is used for instance by the branch indexing feature.

Have more questions?

1 Comments

  • 0
    Avatar
    Steven Wall

    Hello,

    I'm having trouble finding the setting to do a lightweight checkout when defining a multibranch pipeline. I see the checkbox for simple pipeline jobs. But I'm not seeing something analogous for multibranch pipeline jobs. I did double check the plugin versions and all looks good. Any ideas? Maybe a screenshot would be help?

    Best,

    Steve

Please sign in to leave a comment.