Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@...
W: www.topic.nl

Please consider the environment before printing this e-mail
On 11-04-2021 17:23, Gmane Admin via lists.yoctoproject.org wrote:

My build machine has 8 cores + HT so bitbake enthusiastically assumes 16. However I have (only?) 16GB of RAM (+24GB swap space).

Been there, done that (8/16 cores with 8GB RAM).

The major cause is that bitbake will not just spawn 16 compiler threads, it will actually spawn up to 16 "do_compile" tasks which may spawn 16 compiler processes each, thus you'll be running a whopping 256 compilers. Bitbake may spawn a total of n^2 threads by default with "n" the detected number of cores.

The workaround I used was to limit the number of bitbake threads but leave the make threads at 16. Since most software these days is using parallel make, the impact on the build time of reducing the bb threads to 8 or 4 is negligible.

As for translating this into a bitbake feature, an implementation that comes to mind is to add a "weight" to each task. Most tasks would get a weight of just "1", but a (multithreaded) compile would be weighted "8" (some formula involving the number of CPUs) or so. This would stop bitbake spawning more processes early so that you don't get into the n^2 threads problem, and the number of processed that bitbake will spawn will be close to linear agin instead of quadratic as is is now.

Recipes that have excessive memory loads (usually C++ template meta-programming) can then just increase the weighting for the compile task.


--
Mike Looijmans