Troubleshooting the service access to Marathon and Mesos Master

Issue

  • Marathon can’t connect with with Mesos Master
  • Mesos Master is not reachable
  • Marathon service is not reachable

Environment

Resolution

Marathon healthcheck

This section intends to provide a set of actions we could perform in order to check the health of Marathon service. The document address how to check if the Marathon service is up and running and if Marathon can be reached out inside and outside the cluster.

  • Service check

We will check if the Marathon service is up and running. For this, we need to connect to each controller in the cluster and check that the service is up and running

dna connect controller-X
sudo service marathon status
  • Marathon can be reached out inside the cluster

We will check if Marathon can be reached out inside the cluster. For this, we need to connect to each controller and reach out Marathon UI port curl localhost:8080

We should get a HTML output like the one below.

curl localhost:8080
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 </title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /. Reason:
<pre>    Unauthorized</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.z-SNAPSHOT</a><hr/>
</body>
</html>
  • Marathon can be reached out outside the cluster

We will check if Marathon can be reached out putside the cluster passing through the ELB/Load Balacer/HAProxy or whatever is in front of the cluster. For this, we need to access Marathon from your web-browser. You can get the Marathon URL from the command cje run display-outputs. The username/password can be retrieved from .dna/secrets.

  • Marathon logs

Mesos Master logs can be found by connecting into the controllers and running:

  • Ubuntu: tail -f /var/log/syslog | grep marathon
  • RedHat/CentOS: tail -f /var/log/messages | grep marathon

Mesos Master healthcheck

This section intends to provide a set of actions we could perform in order to check the health of Mesos service. The document address how to check if the Mesos service is up and running and if Mesos can be reached out inside and outside the cluster.

  • Service check

We will check if the Mesos service is up and running. For this, we need to connect to each controller in the cluster and check that the service is up and running

dna connect controller-X
sudo service mesos-master status
  • Mesos-master can be reached out inside the cluster

We will check if Mesos can be reached out inside the cluster. For this, we need to connect to each controller and reach out Mesos-Master UI port curl localhost:5050

We should get a HTML output like the one below.

curl localhost:5050
<!DOCTYPE html>
<html lang="en" ng-app="mesos">
  <head>
    <meta charset="utf-8">
    <title>Mesos</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="">
    <meta name="author" content="">

    <link href="/static/css/bootstrap-3.3.6.min.css" rel="stylesheet">
    <link href="/static/css/mesos.css" rel="stylesheet">

    <link rel="shortcut icon" href="/static/ico/favicon.ico">
  </head>
  ....
  • Mesos Master can be reached out outside the cluster

We will check if Mesos can be reached out putside the cluster passing through the ELB/Load Balacer/HAProxy or whatever is in front of the cluster. For this, we will try to access Mesos Master from your web-browser. You can get the Mesos Master URL from the command cje run display-outputs. The username/password can be retrieved from .dna/secrets.

  • Mesos Master logs

Mesos Master logs can be found connecting into the controllers and running tail -f /var/log/mesos/*

Tested products/plugins version

The latest update of this article was tested with:

Have more questions?

0 Comments

Please sign in to leave a comment.