KBEC-00503 - Understanding CD resources and agents

Problem:

Q: What’s the difference between an Agent and a Resource in the CD environment?

Solution

CD endpoints require agent software to be installed.  In order for those agents to work in the CD system, that agent installation requires that 1 resource be defined inside the CD DB that identifies a resource name and the associated endpoint IP address where that agent was installed.  However, this is not a 1:1 restriction.  You may define multiple resources inside the DB that point to the same endpoint agent.

Each resource contains a definition of a “step limit”.  By default, this value is not filled in, which translates to having an “unlimited” number of active steps.  This gives you flexibility in not setting a hard-limit, but if you find too many jobs getting assigned (which may overload the machine and essentially block the assigned work from progressing, or from progressing at a reasonable rate), then you would be better off to set a limit based on this experience.  A baseline rule-of-thumb might be to use 1 core per step-limit, but it really depends on what work is being assigned to those endpoints.  Some customer might test and confirm that 4 steps can safely run and share the same core.  So this means a 4-core machine could handle having a 16-step-limit just fine.  Others may realize that their jobSteps are memory-hungry, so core are not what defines their stepLimit sizing and instead they might define this based on a rule like….8GB for each step, so only 3 steps on my 24-GB memory agent. As you can see here, all such explorations need to be run local to your needs and equipment.

This also means that such settings should be reviewed from time to time to understand when adding additional resources may allow the system to work more effectively.

While most scenarios will see you assign 1 resource to 1 agent, it’s not uncommon to see customers requiring a 1-step-resource to be used as a locking mechanism.  So you might see 2 resources defined against the same agent in cases like that.  Another approach might be to setup a separate resource for each core, and identifying only 1-step for each resource, so in that situation you might see something like 4 resources against the same agent endpoint.  Also see the section titled “An Edge Case” in this KB article:
KBEC-00244 Controlling Resource Usage with Retain Exclusive

It’s important to realize that counting “resources” won’t necessarily match to your agent count.  Resources will always be “greater than or equal to” your agent counts.  It’s also necessary to consider total step limits whenever there is a decision to create multiple resources against the same agent. In other words, the sum of the step-limits for these agents will need to adhere to your desired maximum.  As per above, if you determined that your step-limit should not surpass 10, but you end up creating a special resource on this box for some locking purpose, then it may be necessary to be dropping the total step-limit you have for the main resource down to 9, to help ensure this special resource will get enough CPU cycles to proceed as expected when the box is otherwise under heavy usage loads.

Whenever you have multiple resources defined against the same end-point agent, it may be important to recognize that “exclusive” mode may not do what you think.  If your use of Exclusive is intended to avoid someone else modifying some interim data held on this resource, then knowing how those other resource(s) are being used may be critical to make the overall usage run smoothly.

More details on resources and various settings associated with resources can be found in the documentation here:
Resources

An understanding of how the step-scheduler works to decide about assigning a resource to a step/task is outlined here:
KBEC-00152 - CD Step Scheduler Algorithm

Q: How is a proxy-agent resource different?

A:

A proxy-agent is intended to be used for situations where you want to drive work to a proxy target host, for which there is no defined CD agent installer available.  The classic example would be a mainframe host.  As long as the proxy target is capable of allowing an SSH connection, a host which has the CD agent installation can be used to communicate with that proxy target.  This proxy-target pairing requires a Resource entry to be defined in the Server Database, but there would be no agent software installed on the actual target endpoint in this scenario.  
The step-limit setting recommendations above would still apply here.  The total number of steps that can be handled by any set of proxy-resources on such agents should be a higher-than-usual amount, since the proxy-agent will mostly act as a pass-through agent, and will therefore not perform too much heavy-lifting.  The actual limits should be tested in your environment on a case by case basis.

Have more questions?

0 Comments

Please sign in to leave a comment.