EC2-Plugin Out of Memory issue when launching slaves in AWS

Issue

We have EC2-plugin that is launching m3.large dynamic slaves in AWS. However, we are running into memory issues consistently below. Also, When we manually attach the slave to CloudBees Enterprise server using slave.jnlp the jobs run perfectly fine.

The jobs are failing because can not allocate memory at slaves:

There is insufficient memory for the Java Runtime Environment to continue. 
Native memory allocation (malloc) failed to allocate 855376 bytes for Chunk::new 
An error report file with more information is saved as: 

Error occurred during initialization of VM 
Could not reserve enough space for object heap 

Jenkins log shows errors like:

2015-12-08 15:28:33.726+0000 [id=337861]	SEVERE	h.p.ec2.win.winrm.WinRMClient#sendRequest: I/O Exception in HTTP POST
org.apache.http.conn.ConnectTimeoutException: Connect to 10.71.6.223:5985 timed out
	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:119)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:231)
	at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:202)
	at hudson.plugins.ec2.win.winrm.WinRMClient.openShell(WinRMClient.java:85)
	at hudson.plugins.ec2.win.winrm.WinRM.ping(WinRM.java:29)
	at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:117)
	at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:123)
	at hudson.plugins.ec2.win.EC2WindowsLauncher.launch(EC2WindowsLauncher.java:31)
	at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:107)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:243)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Environment

CloudBees Jenkins Enterprise

EC2 Plugin

AWS

Resolution

WinRM limits the amount of RAM available to the “remote shell”

https://msdn.microsoft.com/en-us/library/windows/desktop/ee309367(v=vs.85).aspx

the EC2 plugin launches windows slaves via WinRM.

These limits will need configuring if you want more than 1GB of RAM to be used by Jenkins and all processes it starts.

e.g. as an Administrator using an elavted command prompt execute
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="10240"}'
Note you may also require to set the number of processes as well depending on the number of executors and types of jobs run.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.