Palace REST API

Issue

  • I want to monitor Palace
  • I want to gather information from Palace

Environment

Resolution

Palace provides a REST API and following are useful endpoints:

  • Health check (should return RUNNING if palace is up):

curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/health

  • Palace version:

curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/version

  • Get tasks overall stats:

curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/stats

  • Get the tasks lists. Queued tasks have the status TASK_PENDING. When Palace tried to provision an agent, you would see the status TASK_LAUNCHING.

curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks

The response lists the task records with the following schema:

<creationTime> <taskId> : <taskState> [<stateReason>] [<killReason>] > <slaveId> : <taskHostName>

For example:

Tasks
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:21:35 UTC 2018 controller-1.eef8d99b:cc3a1950-367c-48af-8070-e08767e0c9ab : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:20:45 UTC 2018 controller-1.1868164a:eefee33a-5af9-4a46-806e-5144f95c9bc9 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
  • Get the list of queued tasks:

curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks?status=TASK_PENDING

  • Get the status and logs of a task by task ID. You can also pass as parameter the level of logs to display.

curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/$TASK_ID?level=INFO

where:

  • $CLUSTER_FQDN is the domain name of palace in your cluster:
  • PALACE_USER is the palace username
  • PALACE_PASSWORD is the palace password
  • TASK_ID is the ID of a task

Concrete example

From the root of the CJE project, you can retrieve the different variables:

export PALACE_USER=$(cje run echo-secrets palace_username)
export PALACE_PASSWORD=$(cje run echo-secrets palace_password)
export CLUSTER_FQDN=$(awk '/domain_name/ {print $3}' .dna/project.config)
  • Health check
curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/health
RUNNING
  • Palace Version
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/version
{
  "status" : "1.9.3-SNAPSHOT:726ca9c783c3cfcfe20f8dbd6461529496d380c1"
}
  • Tasks Overall Statistics
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/stats
Tasks Statistics:
TOTAL > TASK_KILLED: 3
400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com ACTIVE> TASK_KILLED: 3
  • Tasks List
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks
Tasks
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:21:35 UTC 2018 controller-1.eef8d99b:cc3a1950-367c-48af-8070-e08767e0c9ab : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
Wed Jan 03 06:20:45 UTC 2018 controller-1.1868164a:eefee33a-5af9-4a46-806e-5144f95c9bc9 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-1-2-3-4.compute-1.amazonaws.com
  • Tasks List filtered by status “TASK_KILLED”
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/tasks?status=TASK_KILLED
Tasks
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-35-153-101-202.compute-1.amazonaws.com
Wed Jan 03 06:21:35 UTC 2018 controller-1.eef8d99b:cc3a1950-367c-48af-8070-e08767e0c9ab : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-35-153-101-202.compute-1.amazonaws.com
Wed Jan 03 06:20:45 UTC 2018 controller-1.1868164a:eefee33a-5af9-4a46-806e-5144f95c9bc9 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0 : ec2-35-153-101-202.compute-1.amazonaws.com
  • Tasks status and INFO log for task ID “controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8”
$ curl -u $PALACE_USER:$PALACE_PASSWORD -X GET http://$CLUSTER_FQDN/palace/v1/admin/task/master-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8?level=INFO
Task:
Wed Jan 03 06:25:45 UTC 2018 controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8 : TASK_KILLED AGENT_SUCCEEDED > 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0
SLAVE LOGS:

TASK STDERR LOGS:
I0103 06:25:45.876174  9684 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0","items":[{"action":"BYPASS_CACHE","uri":{"executable":false,"extract":false,"value":"http:\/\/cje.example.com.com\/master-1\/jnlpJars\/agent.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0\/docker\/links\/99472c9c-bf2d-4831-b311-5d51cf345524"}
I0103 06:25:45.877678  9684 fetcher.cpp:379] Fetching URI 'http://cje.example.com.com/master-1/jnlpJars/agent.jar'
I0103 06:25:45.877691  9684 fetcher.cpp:250] Fetching directly into the sandbox directory
I0103 06:25:45.877704  9684 fetcher.cpp:187] Fetching URI 'http://cje.example.com.com/master-1/jnlpJars/agent.jar'
I0103 06:25:45.877713  9684 fetcher.cpp:134] Downloading resource from 'http://cje.example.com.com/master-1/jnlpJars/agent.jar' to '/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524/agent.jar'
I0103 06:25:45.916498  9684 fetcher.cpp:456] Fetched 'http://cje.example.com.com/master-1/jnlpJars/agent.jar' to '/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524/agent.jar'
I0103 06:25:46.060079  9692 exec.cpp:143] Version: 0.28.2
I0103 06:25:46.061303  9693 exec.cpp:217] Executor registered on slave 400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Jan 03, 2018 6:25:46 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: 2169a19b
Jan 03, 2018 6:25:46 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jan 03, 2018 6:25:46 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /root/.jenkins/cache/jars
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://cje.example.com.com/master-1/]
Jan 03, 2018 6:25:47 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping, OperationsCenter2]
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: ec2-1-2-3-5.compute-1.amazonaws.com
  Agent port:    31845
  Identity:      e8:1c:85:f2:86:04:f6:ad:b3:73:b7:37:60:92:04:39
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to ec2-1-2-3-5.compute-1.amazonaws.com:31845
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: e8:1c:85:f2:86:04:f6:ad:b3:73:b7:37:60:92:04:39
Jan 03, 2018 6:25:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Jan 03, 2018 6:27:30 AM org.apache.http.client.protocol.ResponseProcessCookies processCookies
WARNING: Invalid cookie header: "Set-Cookie: AWSALB=Y5Pt6w930T5u5wuS23Kje1+Li3MFzLZYj42r3ncwj2hU6Ll3GB5x4A34JRC1TUvtDjzG9UkIPO2hW5ClLmOakjZnToOkb54nRyF6gcvEIwfANWZLPWl2tMV8ak4E; Expires=Wed, 10 Jan 2018 06:27:30 GMT; Path=/". Invalid 'expires' attribute: Wed, 10 Jan 2018 06:27:30 GMT
Jan 03, 2018 6:27:30 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Jan 03, 2018 6:27:30 AM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@5fdeca47
hudson.remoting.ChannelClosedException: channel is already closed
	at hudson.remoting.Channel.send(Channel.java:671)
	at hudson.remoting.Request$2.run(Request.java:370)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1$1.run(Engine.java:98)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
	... 4 more

TASK STDOUT LOGS:
--container="mesos-400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0.99472c9c-bf2d-4831-b311-5d51cf345524" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524" --stop_timeout="10secs"
--container="mesos-400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0.99472c9c-bf2d-4831-b311-5d51cf345524" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slaves/400c68cb-827b-4c9d-a0f0-6bdad1e94053-S0/docker/links/99472c9c-bf2d-4831-b311-5d51cf345524" --stop_timeout="10secs"
Registered docker executor on ec2-1-2-3-4.compute-1.amazonaws.com
Starting task controller-1.2169a19b:aafa283c-b477-4b14-88cc-97e3d25995e8
Received killTask
Shutting down

Tested product/plugin versions

The latest update of this article has been tested with:

Was this article helpful?
1 out of 1 found this helpful

Have more questions?

0 Comments

Please sign in to leave a comment.