Re: bitbake controlling memory use


Alexander Kanavin
 

make already has -l option for limiting new instances if load average is too high, so it's only natural to add a RAM limiter too.

  -l [N], --load-average[=N], --max-load[=N]
                              Don't start multiple jobs unless load is below N.

In any case, patches welcome :)

Alex


On Sun, 11 Apr 2021 at 18:08, Gmane Admin <gley-yocto@m.gmane-mx.org> wrote:
Op 11-04-2021 om 17:55 schreef Alexander Kanavin:
> On Sun, 11 Apr 2021 at 17:49, Gmane Admin <gley-yocto@m.gmane-mx.org
> <mailto:gley-yocto@m.gmane-mx.org>> wrote:
>
>     Yes, and make project doesn't care, because make is called with -j
>     16 so
>     that is what it does.
>
>     So here's my pitch: bitbake can stop processes spawned by make, because
>     it knows that it started make on 4 recipies, each with -j 16. The
>     individual makes don't know about each other.
>
>
> And neither they should. They can simply abstain from spawning new
> compilers if used RAM is, say, at 90% total. Then bitbake does not have
> to get involved in babysitting those makes.
>
> Alex
Bitbake does a lot of babysitting anyway :-) And is pretty good at it too.

To me, fixing make et al. is more work and less effective then adding a
feature to bitbake. The only way to know how much memory the compiler
will use for each spawned compiler is to let it run. And then it's too late.

This memory issue is all over our eco system and nobody cares (kernel,
make etc.) The only thing moving is systemd's oom killer will arrive and
start killing processes. So that will just stop our builds from completing.

Yeah, I prefer a babysitter over a child murderer :-)

Ferry




Join yocto@lists.yoctoproject.org to automatically receive all group messages.