Issue
- I want to monitor Palace
- I want to gather information from Palace
Environment
- CloudBees Jenkins Enterprise (CJE)
- Palace
Resolution
Palace provides a REST API and following are useful endpoints:
- Health check (should return
RUNNING
if palace is up):
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/health
- Palace version:
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/version
- Get tasks overall stats:
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/stats
- Get the tasks lists. Queued tasks have the status
TASK_PENDING
. When Palace tried to provision an agent, you would see the statusTASK_LAUNCHING
.
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks
The response lists the task records with the following schema:
<creationTime> <taskId> : <taskState> [<stateReason>] [<killReason>] > <slaveId> : <taskHostName>
For example:
Tasks
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:21:35 UTC 2018 controller-1.eef8d99b:cc3a1950-367c-48af-8070-e08767e0c9ab : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:20:45 UTC 2018 controller-1.1868164a:eefee33a-5af9-4a46-806e-5144f95c9bc9 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
- Get the list of queued tasks:
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks?status=TASK_PENDING
- Get the status and logs of a task by task ID. You can also pass as parameter the level of logs to display.
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/$TASK_ID?level=INFO
where:
$CLUSTER_FQDN
is the domain name of palace in your cluster:PALACE_USER
is the palace usernamePALACE_PASSWORD
is the palace passwordTASK_ID
is the ID of a task
Concrete example
From the root of the CJE project, you can retrieve the different variables:
export PALACE_USER=$(cje run echo-secrets palace_username)
export PALACE_PASSWORD=$(cje run echo-secrets palace_password)
export CLUSTER_FQDN=$(awk '/domain_name/ {print $3}' .dna/project.config)
- Health check
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/health
RUNNING
- Palace Version
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/version
{
"status" : "1.9.3-SNAPSHOT:726ca9c783c3cfcfe20f8dbd6461529496d380c1"
}
- Tasks Overall Statistics
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/stats
Tasks Statistics:
TOTAL > TASK_KILLED: 3
400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com ACTIVE> TASK_KILLED: 3
- Tasks List
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks
Tasks
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:21:35 UTC 2018 controller-1.eef8d99b:cc3a1950-367c-48af-8070-e08767e0c9ab : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:20:45 UTC 2018 controller-1.1868164a:eefee33a-5af9-4a46-806e-5144f95c9bc9 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
- Tasks List filtered by status “TASK_KILLED”
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks?status=TASK_KILLED
Tasks
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-35-153-101-202.compute-1.amazonaws.com
Wed Jan 03 06:21:35 UTC 2018 controller-1.eef8d99b:cc3a1950-367c-48af-8070-e08767e0c9ab : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-35-153-101-202.compute-1.amazonaws.com
Wed Jan 03 06:20:45 UTC 2018 controller-1.1868164a:eefee33a-5af9-4a46-806e-5144f95c9bc9 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-35-153-101-202.compute-1.amazonaws.com
- Tasks status and INFO log for task ID “controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8”
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/master-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8?level=INFO
Task:
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0
SLAVE LOGS:
TASK STDERR LOGS:
I0103 06:25:45.876174 9684 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0","items":[{"action":"BYPASS_CACHE","uri":{"executable":false,"extract":false,"value":"http:\/\/cje.example.com.com\/master-1\/jnlpJars\/agent.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0\/docker\/links\/99472c9c-bf2d-4831-b311-5d51cf345524"}
I0103 06:25:45.877678 9684 fetcher.cpp:379] Fetching URI 'http://cje.example.com.com/master-1/jnlpJars/agent.jar'
I0103 06:25:45.877691 9684 fetcher.cpp:250] Fetching directly into the sandbox directory
I0103 06:25:45.877704 9684 fetcher.cpp:187] Fetching URI 'http://cje.example.com.com/master-1/jnlpJars/agent.jar'
I0103 06:25:45.877713 9684 fetcher.cpp:134] Downloading resource from 'http://cje.example.com.com/master-1/jnlpJars/agent.jar' to '/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524/agent.jar'
I0103 06:25:45.916498 9684 fetcher.cpp:456] Fetched 'http://cje.example.com.com/master-1/jnlpJars/agent.jar' to '/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524/agent.jar'
I0103 06:25:46.060079 9692 exec.cpp:143] Version: 0.28.2
I0103 06:25:46.061303 9693 exec.cpp:217] Executor registered on slave 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Jan 03, 2018 6:25:46 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: 2169a19b
Jan 03, 2018 6:25:46 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jan 03, 2018 6:25:46 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /root/.jenkins/cache/jars
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://cje.example.com.com/master-1/]
Jan 03, 2018 6:25:47 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping, OperationsCenter2]
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: ec2-1-2-3-5.compute-1.amazonaws.com
Agent port: 31845
Identity: e8:1c:85:f2:86:04:f6:ad:b3:73:b7:37:60:92:04:39
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to ec2-1-2-3-5.compute-1.amazonaws.com:31845
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: e8:1c:85:f2:86:04:f6:ad:b3:73:b7:37:60:92:04:39
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Jan 03, 2018 6:27:30 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies
WARNING: Invalid cookie header: "Set-Cookie: AWSALB=Y5Pt6w930T5u5wuS23Kje1+Li3MFzLZYj42r3ncwj2hU6Ll3GB5x4A34JRC1TUvtDjzG9UkIPO2hW5ClLmOakjZnToOkb54nRyF6gcvEIwfANWZLPWl2tMV8ak4E; Expires=Wed, 10 Jan 2018 06:27:30 GMT; Path=/". Invalid 'expires' attribute: Wed, 10 Jan 2018 06:27:30 GMT
Jan 03, 2018 6:27:30 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Jan 03, 2018 6:27:30 AM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@5fdeca47
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:671)
at hudson.remoting.Request$2.run(Request.java:370)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
... 4 more
TASK STDOUT LOGS:
--container="mesos-400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0.99472c9c-bf2d-4831-b311-5d51cf345524" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524" --stop_timeout="10secs"
--container="mesos-400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0.99472c9c-bf2d-4831-b311-5d51cf345524" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524" --stop_timeout="10secs"
Registered docker executor on ec2-1-2-3-4.compute-1.amazonaws.com
Starting task controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8
Received killTask
Shutting down
Tested product/plugin versions
The latest update of this article has been tested with:
- CloudBees Jenkins Enterprise 1.11.0
- cloudbees/pse-palace:1.10.0
0 Comments