Date
1 - 7 of 7
nodejs do_compile eats all resources
|
Martin Jansa
> PARALLEL_MAKE = "-j ${@int(oe.utils.cpu_count() / 4)}" This won't be optimal in cases where you're rebuilding just one recipe in incremental build which will be finished much faster if you can unleash whole potential of dual-epycs. Try to add -l to limit overall load instead and limit -j only for the biggest offenders included in your build, you don't need to do that in .bbappend, just use pn override, e.g. my local.conf has: BB_PRESSURE_MAX_CPU = "1000" BB_NUMBER_THREADS = "8" BB_NUMBER_PARSE_THREADS = "64" PARALLEL_MAKE = "-j 70 -l 140" #PARALLEL_MAKE:pn-qtwebengine = "-j 40" #PARALLEL_MAKE:pn-webruntime = "-j 40" On 64 thread TR 3970X with 128GB ram, the restriction for qtwebruntime and webruntime isn't necessary, since I've added BB_PRESSURE_MAX_CPU I haven't seen OOMK just from the OE builds. It's very common when you build some "big" images with "big" components. Many components cannot utilize very high PARALLEL_MAKE for very long or for demanding processes. nodejs, chromium, webkit-gtk, firefox are notable exceptions which take long time to build and scale very well with PARALLEL_MAKE (while hammering the HW as much as you let them). There is now also the pressure regulation in bitbake, which might help to limit overall load on the system, when too many heavy do_compile tasks get triggered at the same time (see https://git.openembedded.org/bitbake/commit/?id=502e05cbe67fb7a0e804dcc2cc0764a2e05c014f ) See https://github.com/shr-project/test-oe-build-time and you can compare your dual-epyc with the entries there. Regards, On Tue, Sep 27, 2022 at 11:02 AM Oliver Westermann via lists.yoctoproject.org <oliver.westermann=cognex.com@...> wrote: > Am Dienstag, 27. September 2022 10:50 schrieb Alexander Kanavin <alex.kanavin@...>: |
|
|
|
Oliver Westermann <oliver.westermann@...>
Am Dienstag, 27. September 2022 10:50 schrieb Alexander Kanavin <alex.kanavin@...>:Yeah, I'm currently using PARALLEL_MAKE = "-j ${@int(oe.utils.cpu_count() / 4)}" In a .bbappend to limit it, but keep it dynamic. One question out of curiosity or lack of understanding: Bitbake offers BB_NUMBER_THREADS to limit (IMHO) the number of bitbake tasks, eg parallel do_compile() tasks. Each of these however spawn the compiler with several parallel threads oe.utils.cpu_count (PARALLEL_MAKE defaults to cpu_count()). Why isn't it more common that we end up in issues due to cpu_count^2 threads being spawned? Shouldn't (roughly) BB_NUMBER_THREADS * PARALLEL_MAKE fully load the CPU? Best regards, a curious Olli |
|
|
|
Alexander Kanavin
Do keep in mind that PARALLEL_MAKE can and should be set per recipe,
so you can make-limit only the worst items. Alex On Tue, 27 Sept 2022 at 10:34, Westermann, Oliver <Oliver.Westermann@...> wrote: Am Montag, 26. September 2022 19:33 schrieb Alexander Kanavin <alex.kanavin@...>I expected something like this. Thanks for the confirmation and I will try to play with PARALLEL_MAKE to reduce the impact. |
|
|
|
Oliver Westermann <oliver.westermann@...>
Am Montag, 26. September 2022 19:33 schrieb Alexander Kanavin <alex.kanavin@...>I expected something like this. Thanks for the confirmation and I will try to play with PARALLEL_MAKE to reduce the impact. - Olli |
|
|
|
On 9/26/22 10:20 AM, Oliver Westermann via lists.yoctoproject.org wrote:
Hey,how many cores do you have on the dual Epyc sockets. for Mmeory., usually 4GB/per-core is a safe mark for worst case scenario. nodejs build does run some target binaries using qemu-usermode eg. mksnapshot during build, which in a threaded mode will put more pressure on memory. I only observed this on the nodejs build, is there a good approach to debug this (or even a known issue)? |
|
|
|
Alexander Kanavin
Anything written in C++ tends to consume 1-2 Gb of ram per compiler
toggle quoted message
Show quoted text
process. If that lands you in OOM, you probably should limit that with PARALLEL_MAKE:pn-nodejs, but otherwise that is the sad reality of C++ builds. Alex On Mon, 26 Sept 2022 at 19:20, Oliver Westermann via lists.yoctoproject.org <oliver.westermann=cognex.com@...> wrote:
|
|
|
|
Oliver Westermann <oliver.westermann@...>
Hey,
We recently added nodejs to our images and noticed that it's do compile process is a real memory hog. Since we planned to update the recipe anyway we didn't pay much attention, but today we updated the recipe to the current version from openembedded-core (16.14.2). Still I see that it consumes nearly all resources it can get, way more than any other package in our build process (and crashing some VMs even). On my build machine (dual Epyc, 256GB of RAM) it manages to produce nearly 100% CPU load AND eats ~90GB of RAM. Some other machines fail the build due to OOM issues. I only observed this on the nodejs build, is there a good approach to debug this (or even a known issue)? Best regards, Olli |
|
|