When training reinforcement learning agents, the C drive is fully occupied

During the process of using the reinforcement learning toolbox to train agents, the occupancy of the C drive increased and eventually became full. I found that the virtual memory in the training process was very large (nearly 50G) by browsing the hidden files in the C disk. After restarting MATLAB, this part of memory will be released again.The training environment is built through Simscape.

I want to know what methods can be used to avoid the increasing usage of the C drive as the training duration increases. Because of this issue, I have to always terminate the training process.

Answer 1

[Shaik]

Hi Yunyi,

When training reinforcement learning agents using the reinforcement learning toolbox in MATLAB, it's important to manage the memory usage to avoid filling up the C drive. Here are a few methods you can try to mitigate the increasing disk space usage during training:

Change the temporary directory: MATLAB uses a temporary directory for storing intermediate data during training. By default, this directory is set to the system's temporary folder, which is often located on the C drive. You can change the temporary directory to a different location, such as a different drive with more available space. Use the tempdir function in MATLAB to set a different temporary directory.

newTempDir = 'D:\Temp';  % Replace with the desired temporary directory
matlab.internal.getAndSetTemporaryDirectory(newTempDir);

1. Limit the size of the experience replay buffer: The experience replay buffer can consume a significant amount of memory during training. You can reduce its size by adjusting the BufferSize parameter when creating the rlDQNAgent or rlDDPGAgent object. A smaller buffer size will require less memory but may affect learning performance.
2. Reduce the training batch size: The batch size determines how many experiences are processed in each training iteration. A smaller batch size will reduce the memory footprint but may slow down training. You can adjust the MiniBatchSize parameter when creating the agent to decrease the batch size.
3. Lower the resolution of observations: If the observations in your environment have high resolution, consider reducing their resolution. This can be achieved by downsampling or compressing the observations before feeding them into the agent. Lower resolution observations require less memory.
4. Restart MATLAB periodically: As you mentioned, restarting MATLAB releases the memory consumed during training. If it's feasible for your workflow, you can periodically restart MATLAB after a certain number of training iterations or episodes to reclaim the memory. You can use the exit function to gracefully terminate MATLAB.

These methods should help mitigate the increasing disk space usage during training. It's important to find the right balance between memory usage and training performance for your specific application.

[yunyi]

Thank you very much!

[Tobias Schindler]

It seems a bit worrisome that "restart Matlab" is a common solution to these kind's of memory leaks. Maybe some staff could provide some additional input here.