KBEC-00104 - Signs you are out of disk space and remedies to recover disk space

Background

The disk system can fill up over time. Eventually, CloudBees CD (CloudBees Flow) will fail when the disk fills up.

Symptoms

After days of CloudBees CD running uninterrupted, one of the following occurs:

1. Users cannot login to the system.

2. When restarting CloudBees CD, CloudBees CD cannot connect to the database and reports in the GUI:

   Server not running: Waiting for a valid database configuration - go to the Database Configuration page
   

   or

   Server not running: Can't connect to the database: No suitable driver - go to the Database Configuration page 
   

3. The commander.log file is no longer growing.
   1. CloudBees CD stopped writing to the log file when the disk became full.
   2. last error can be: “org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update”
4. Steps fail with “AGENT_INTERNAL_ERROR”
5. If you are using MySQL, stop and start MySQL - if MySQL is out of space, it will not restart.

Diagnosis

1. Check the CloudBees CD server’s disk space. The CloudBees CD server log files have filled up the disk - look at the size in the “logs” CloudBees CD directory.
2. Check the database server’s disk space. When using local MySQL, do the following:
   1. Look at the size of the files in the MySQL data directory.
   2. Search the MySQL error log, found in mysql/data/.err by default, for any recent errors.

Remedy

1. Stop CloudBees CD.
2. If using MySQL, stop MySQL.
3. Recover disk space by compressing or deleting CloudBees CD log files.
4. Check the contents of file database.properties found in the CloudBees CD conf directory.
   1. This file can be left corrupted, for example, it has only two lines instead of the twelve normally in the file.
   2. Restore this file from backup.
5. If using MySQL, restart MySQL.
6. Restart CloudBees CD.