Bitbucket Webhooks Troubleshooting

Symptoms

  • Jobs are not automatically triggered when a changes are pushed to Bitbucket

Diagnostic/Treatment

  • Pre-condition: Jenkins (resp. Bitbucket) can reach out to Bitbucket (resp. Jenkins)

Non-Multibranch

In Jenkins for non-multibranch jobs, you are either using the Git Plugin or the Bitbucket Plugin.

Git Plugin

1) Endpoint

This applies to repository configured with a hook to the endpoint $JENKINS_URL/git/notifyCommit

2) Add-on (for Bitbucket Server)

This endpoint is used by the add-on Webhook to Jenkins for Bitbucket in Bitbucket Server:

3) Review Configuration

Review the articles:

4) Troubleshooting

To troubleshoot request processing on that endpoint, please add a logger to the following class:

hudson.plugins.git.GitStatus

Expected Logs

With branch and/or sha specified:

2017-03-10 08:07:09.273+0000 [id=831118]    FINE    h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Received notification for uri = https://my.bitbucket.server/myuser/myrepo.git ; sha1 = e3baa50a43479d6d0246fa2ba5d921c0e06fc707 ; branches = [develop]
2017-03-10 08:07:09.273+0000 [id=831118]    FINE    h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Branch Spec origin/develop matches modified branch develop for mybitbucket » myrepo.
2017-03-10 08:07:09.273+0000 [id=831118]    INFO    h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Scheduling folder-bitbucket » myrepo to build commit e3baa50a43479d6d0246fa2ba5d921c0e06fc707

Without branch and/or sha specified:

2017-03-13 02:39:51.532+0000 [id=106781]	FINE	h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Received notification for uri = https://my.bitbucket.server/myuser/myrepo.git ; sha1 = null ; branches = []
2017-03-13 02:39:51.532+0000 [id=106781]	INFO	h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Triggering the polling of mybitbucket » myrepo

Bitbucket Plugin

1) Endpoint

This applies to Bitbucket repositories configured with a hook to the endpoint $JENKINS_URL/bitbucket-hook/.

2) Add-on (for Bitbucket Server)

Note: The Bitbucket Plugin functionality was broken prior to version 1.1.7 of the plugin. You can find more details in JENKINS-28877.

Bitbucket Plugin integrates with Bitbucket Server via the add-on Bitbucket Server Web Post Hooks Plugin in Bitbucket server:

3) Review Configuration

Review the articles:

4) Troubleshooting

To troubleshoot request processing on that endpoint, please add a logger to the following class:

com.cloudbees.jenkins.plugins.BitbucketJobProbe
com.cloudbees.jenkins.plugins.BitBucketTrigger
com.cloudbees.jenkins.plugins.BitbucketPayloadProcessor
com.cloudbees.jenkins.plugins.BitbucketHookReceiver

Expected Logs

2017-03-13 02:32:36.138+0000 [id=127609]	FINE	c.c.j.p.BitbucketHookReceiver#doIndex: Received commit hook notification : {<payload>}
2017-03-13 02:32:36.207+0000 [id=127609]	INFO	c.c.j.p.BitbucketPayloadProcessor#processPayload: Processing new Webhooks payload
2017-03-13 02:32:36.208+0000 [id=127609]	INFO	c.c.j.p.BitbucketPayloadProcessor#processWebhookPayload: Received commit hook notification for {<simplified-payload>}
2017-03-13 02:32:36.307+0000 [id=127609]	FINE	c.c.j.plugins.BitbucketJobProbe#triggerMatchingJobs: Considering candidate job freestyle-push
2017-03-13 02:32:36.307+0000 [id=127609]	FINE	c.c.j.plugins.BitbucketJobProbe#triggerMatchingJobs: Considering to poke support-team » Bucket » CE-424 » BitbucketCloud » freestyle-push
2017-03-13 02:32:36.311+0000 [id=127609]	INFO	c.c.j.plugins.BitbucketJobProbe#triggerMatchingJobs: Triggering BitBucket job freestyle-push

Multibranch

In Jenkins, for multibranch jobs, you should be using the Bitbucket Branch Source Plugin.

1) Endpoint

This applies to repositories configured with a hook to the endpoint:

2) Add-on (for Bitbucket Server)

Bitbucket Branch Source Plugin integrates with Bitbucket Server via the add-on Post Webhooks for Bitbucket in Bitbucket server:

3) Review Configuration

Review the articles:

4) Troubleshooting

To troubleshoot request processing on that endpoint, please add a logger to the following class:

com.cloudbees.jenkins.plugins.bitbucket.hooks

Expected Logs

Registering Hooks:

2017-03-12 23:50:10.659+0000 [id=28857]	INFO	c.c.j.p.b.h.WebhookAutoRegisterListener#registerHooks: Registering hook for jenkinssupport/simple-project
2017-03-13 00:15:57.702+0000 [id=28857]	FINE	c.c.j.p.b.h.WebhookAutoRegisterListener#removeHooks: NOT removing hook for jenkinssupport/simple-project because does not exists or its used in other project

Processing Reindexing:

2017-03-13 02:33:27.697+0000 [id=132556]	INFO	c.c.j.p.b.h.PushHookProcessor#process: Received hook from Bitbucket. Processing push event on jenkinssupport/simple-project
2017-03-13 02:33:27.699+0000 [id=132556]	INFO	c.c.j.p.b.hooks.HookProcessor$1#run: Multibranch project found, reindexing jenkinssupport
2017-03-13 02:33:27.699+0000 [id=132556]	INFO	c.c.j.p.b.hooks.HookProcessor$1#run: Multibranch project found, reindexing simple-project
2017-03-13 02:33:27.700+0000 [id=132556]	INFO	c.c.j.p.b.hooks.HookProcessor$1#run: Multibranch project found, reindexing pipeline-multibranch

5) Indexing Logs

Any event received on this endpoint should trigger a Branch indexing. But only the logs of the last branch indexing are persisted. If an issue seems intermittent and it is difficult to catch the Branch Indexing that exposes it. In order to catch the issue, you can set the system property com.cloudbees.hudson.plugins.folder.computed.FolderComputation.BACKUP_LOG_COUNT to for example 10 in order to keep the logs of the last 10 branch indexing.

You can set this property on startup by adding it to Jenkins Java arguments or run the following Groovy script in Manage Jenkins > Script Console:

com.cloudbees.hudson.plugins.folder.computed.FolderComputation.BACKUP_LOG_COUNT=10

The branch indexing files are located in the File System in $JENKINS_HOME/jobs/<pathToMultibranch>/indexing/*
The folder computation files are located in the File System in $JENKINS_HOME/jobs/<pathToMultibranch>/computation/*

Troubleshooting Further

If for some reason the request is not received by Jenkins and improved logging did not help, you can troubleshoot this even further by enabling Access Logging for Jenkins.

Expected Logs

Bitbucket Cloud

192.168.0.1 - - [13/Mar/2017:13:23:28 +1000] "POST /bitbucket-scmsource-hook/notify HTTP/1.1" 200 0 "-" "Bitbucket-Webhooks/2.0"
192.168.0.1 - - [13/Mar/2017:13:23:28 +1000] "POST /bitbucket-hook/ HTTP/1.1" 200 0 "-" "Bitbucket-Webhooks/2.0"

Bitbucket Server

192.168.0.2 - - [13/Mar/2017:13:23:52 +1000] "POST /bitbucket-scmsource-hook/notify HTTP/1.1" 200 0 "-" "Atlassian HttpClient 0.23.0 / Bitbucket-4.13.0 (4013000) / Default"
192.168.0.2 - - [13/Mar/2017:13:23:53 +1000] "GET /git/notifyCommit HTTP/1.1" 200 110 "-" "Apache-HttpClient/4.5.1 (Java/1.8.0_102)"
192.168.0.2 - - [13/Mar/2017:13:23:53 +1000] "POST /bitbucket-hook/ HTTP/1.1" 500 3348 "-" "Bitbucket 4.13.0"

Most common issues

  • The hook URL is not correctly configured. See JENKINS-55649 for more information.

On this case, by enabling the custom loggers previously explained on this case, you should get the stacktrace below on Jenkins once the webhook arrives to Jenkins. The problem is that the server_url parameter is not sent by Bitbucket.

X-Bitbucket-Type header / server_url request parameter not found. Bitbucket Cloud webhook incoming.

Recall that the endpoint definition is not exactly the same for the native webhooks and the Bitbucket WebPost Hooks Plugin

  • Native webhooks in Bitbucket endpoint definition: $JENKINS_URL/bitbucket-scmsource-hook/notify?server_url=<BITBUCKET_URL>
  • Bitbucket WebPost Hooks Plugin endpoint definition: $JENKINS_URL/bitbucket-scmsource-hook/notify

An example of endpoint for the native webhooks is -> https://jenkins.example.com/teams-dse-emea/bitbucket-scmsource-hook/notify?server_url=https%3A%2F%2Fbitbucket.example.com

References

Bitbucket Integration

Was this article helpful?
0 out of 0 found this helpful

Have more questions?

2 Comments

Date Votes
  • 0
    Avatar
    K Caola

    For the multibranch example above that mentions

    This applies to repositories configured with a hook to the endpoint $JENKINS_URL/bitbucket-scmsource-hook/notify/

    Does the URL have to end in /bitbucket-scmsource-hook/notify/ or is the bitbucket-scmsource-hook supposed to be a variable like $JENKINS_URL?

  • 0
    Avatar
    Félix Belzunce Arcos

    The URL must end with `/bitbucket-scmsource-hook/notify`

Please sign in to leave a comment.