1. Messages
  2. bitbake controlling memory use

Re: bitbake controlling memory use

Chen Qi
 

I think you write a script to do all the WATCH-STOP-CONTINUE stuff.
e.g.
memwatch bitbake core-image-minimal
Options could be added.
e.g.
memwatch --interval 5 --logfile test.log bitbake core-image-minimal

This script first becomes a session leader, then forks to start the 'bitbake' command as its child process.
Then, every several seconds (say 10s by default), it checks the current memory usage, and according to its policy, determines whether to stop/continue some process or not.

For stopping the process, you can first get all its child process by simply using the 'ps' command.
e.g.
$ ps o vsize,comm,pid -s 28303 | sort -n -r
317284 emacs           12883
 28912 ps              36302
 26248 sort            36303
 21432 man             24797
 17992 bash            28303
  9852 pager           24807
   VSZ COMMAND           PID

Then skip the bitbake processes, stop the first one that uses the largest memory, record its PID.

For continuing processes, just start it according to the records. Maybe using FILO by default?

Best Regards,
Chen Qi

On 04/11/2021 11:23 PM, Gmane Admin wrote:
My build machine has 8 cores + HT so bitbake enthusiastically assumes 16. However I have (only?) 16GB of RAM (+24GB swap space).

16GB of RAM has always been more then enough with 4 core + HT, but now building certain recipes (nodejs, but rust will probably do it as well) eats up more memory then there actually is RAM causing excessive swapping.

In fact the swapping is so excessive that just this single recipe determines largely the total image build time. However restricting bitbake (or actually parallel make) to use only 4 cores + HT sounds also like a waste.

I know this has been looked at in the past, but I think it needs fundamental resolving (other then, hé, why not add just another stick of memory).

What I do manually when I run into swapping so bad my desktop becomes unresponsive is ssh from another machine and then stop (not kill) the processes that use the most memory.

These then get swapped out, but not back in, allowing the remaining tasks to complete without swapping. Then when sufficient memory becomes free again I continue the stopped processes.

Isn't this something that could be added to bitbake to automate using memory efficiently?


Join yocto@lists.yoctoproject.org to automatically receive all group messages.