Issue

• Jenkins instance and users are in the same Timezone but Jenkins System keeps triggering builds randomly.

• On logs can be found a similar trace to the following:

2017-02-06 12:16:53.815-0600 [id=5089154]	SEVERE	s.s.stash.StashApiClient$1#call: Failing to get response from Stash PR GET/rest/api/1.0/projects/INNODELOC/repos/InNode-location-performance-ci/pull-requests
2017-02-06 12:16:53.816-0600 [id=37]	WARNING	hudson.triggers.Trigger#checkTriggers: stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run() failed for hudson.maven.MavenModuleSet@6d32c80b[Walmart-Technology-Reston/Enterprise-Location-Service-Performance-Test-CI]java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Didn't get a 200 response from Stash PR GET! Response; 'Not Found' with message; null
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:262)
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequests(StashApiClient.java:86)
	at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:72)
	at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
	at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
	at hudson.triggers.Trigger.checkTriggers(Trigger.java:273)
	at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Didn't get a 200 response from Stash PR GET! Response; 'Not Found' with message; null
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
	... 14 more
Caused by: java.lang.RuntimeException: Didn't get a 200 response from Stash PR GET! Response; 'Not Found' with message; null
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient$1.call(StashApiClient.java:234)
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient$1.call(StashApiClient.java:221)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 1 more

• On thread dumps can be found a similar trace to the following which proves that Stash pullrequest builder plugin is doing too much work in the timer trigger thread, delaying all subsequent executions.

"jenkins.util.Timer [#1]" id=37 (0x25) state=TIMED_WAITING cpu=40%
    - waiting on <0x1c6062eb> (a java.util.concurrent.FutureTask)
    - locked <0x1c6062eb> (a java.util.concurrent.FutureTask)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426)
    at java.util.concurrent.FutureTask.get(FutureTask.java:204)
    at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
    at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestMergeStatus(StashApiClient.java:150)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isPullRequestMergable(StashRepository.java:209)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:261)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
    at hudson.triggers.Trigger.checkTriggers(Trigger.java:273)
    at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@2b0a3762

Environment

• Jenkins
• CloudBees Jenkins Enterprise
• Stash pullrequest builder plugin 1.7

JENKINS

JENKINS-41336

Resolution

First, disable the Stash pullrequest builder plugin because it is blocking the scheduler and preventing all timer triggers.

Then, think about your jobs making use of other plugins which enable to build pull requests from an Atlassian Stash instance.

• Bitbucket Branch Source Plugin for Bitbucket 4.0 or later; Stash 3.x and earlier are not supported.
• Otherwise, Pipeline Multibranch Plugin