Issue

CloudBees CD is using Hikari CP to manage the server connection pools to the DB.

In some cases, Hikari CP component is not able to create new connections after some of them have exceeded maxlife time.

In this case, you will detect a big slowness in the CloudBees CD Server, with a long loading page time in the Web UI.

You can identify this scenario reviewing the commander.log file, and looking for entries with the keyword connection has passed maxLifetime similar to:

2021-05-12T04:47:54.055 | cdservername | DEBUG | qtp751321846-25102             |          |                                      |                                                                             | ApiServletImpl                 | servletRequestContext[id=1058,on 16.192.120.52:8443,from 16.192.120.62:38196,bytes=133]:
{
  "version" : "2.0",
  "sessionId" : "[PROTECTED]",
  "requests" : [ {
    "operation" : "getServerStatus",
    "parameters" : {
      "serverStateOnly" : true
    },
    "requestId" : "24"
  } ]
}

...

2021-05-12T05:11:40.738 | cdservername | DEBUG | HikariCP-001|||| HikariPool| CloudBees Flow - After cleanup  stats (total=4, active=0, idle=4, waiting=0)
2021-05-12T05:12:00.426 | cdservername | DEBUG | HikariCP-4286|||| LogThreadCallback| Starting Thread[HikariCP-4286,5,main]
2021-05-12T05:12:00.426 | cdservername | DEBUG | HikariCP-4286|||| PoolBase| CloudBees Flow - Closing connection oracle.jdbc.driver.T4CConnection@3a4e8441: (connection has passed maxLifetime)
2021-05-12T05:12:00.427 | cdservername | DEBUG | HikariCP-4287|||| LogThreadCallback| Starting Thread[HikariCP-4287,5,main]
2021-05-12T05:12:00.480 | cdservername | DEBUG | HikariCP-4287|||| HikariPool| CloudBees Flow - Added connection oracle.jdbc.driver.T4CConnection@563a9ce0
2021-05-12T05:12:05.429 | cdservername | DEBUG | HikariCP-4286|||| LogThreadCallback| Exiting Thread[HikariCP-4286,5,main]
2021-05-12T05:12:05.429 | cdservername | INFO  | HikariCP-4286||| ThreadStateCleanup| Contention: waitedCount=1 waitedTime=5000ms blockedCount=0 blockedTime=0ms

....

2021-05-12T05:17:54.056 | cdservername | DEBUG | qtp751321846-26163             |          |                                      |                                                                             | JsonApiRequestHandler          | Response for id=1058:
{
  "responses" : [ {
    "error" : {
      "where" : "",
      "message" : "Request timed out",
      "details" : "",
      "code" : "RequestTimeout"
    }
  } ]
}

Environment

• CloudBees CD (CloudBees Flow)
• System using Oracle DB

Resolution

To solve this issue, we recommend you restart the CloudBees CD Server service, as explained in this article.

After the CloudBees CD Server service restarts the system should perform properly again.