On Mon, 2022-12-26 at 18:54 -0800, Mistyron wrote:

Hi,

I'm want to write a recipe to compile phytool but I'm getting the below
error when building it with:
$ bitbake -b phytool_2.0
ERROR: Failed to spawn fakeroot worker to run
/home/yocto/rzg_vlp_v3.0.0/build/../meta-mistysom/recipes-bringup/phytool/phytool_2.0.bb:do_install:
[Errno 2] No such file or directory:
'/home/yocto/rzg_vlp_v3.0.0/build/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo'

my bb file looks like:

[phytool_2.0.bb]
DESCRIPTION = "Recipe builds phytool utility"
# Needs to be changed to GPLv2
LICENSE = "MIT"
SRC_URI = "git://github.com/MistySOM/phytool.git"
SRCREV = "8882328c08ba2efb13c049812098f1d0cb8adf0c"
LIC_FILES_CHKSUM =
"file://../git/LICENSE;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0"
INHERIT += " pseudo"

python do_display_banner() {
    bb.plain("some message")
}

addtask display_banner before do_build
[/phytool_2.0.bb]

It appears like my INHERIT line does not satsify the requirement.
How can I resolve this?

There is no class called "pseudo" so the inherit line is wrong. Why do
you think you need to inherit such a class?

Pseudo is usually used by tasks as needed by default.

Cheers,

Richard

https://git.openembedded.org/meta-openembedded/tree/meta-networking/recipes-support/phytool/phytool.bb

Hi Marcus,

Thanks for the reference. I have since found it too and just looped it in with:

MAGE_INSTALL_append = " phytool"

That's a lot easier but I still would like to know about the pseudo error I received. I think it will come back at me at one point or another.

Ron

Am Mo, 26. Dez 2022 um 18:54:01 -0800 schrieb Mistyron <ron.eggler@...>:

Hi, I'm want to write a recipe to compile phytool but I'm getting the below error when building it with: $ bitbake -b phytool_2.0 ERROR: Failed to spawn fakeroot worker to run /home/yocto/rzg_vlp_v3.0.0/build/../meta-mistysom/recipes-bringup/phytool/phytool_2.0.bb:do_install: [Errno 2] No such file or directory: '/home/yocto/rzg_vlp_v3.0.0/build/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo' my bb file looks like: [phytool_2.0.bb] DESCRIPTION = "Recipe builds phytool utility" # Needs to be changed to GPLv2 LICENSE = "MIT" SRC_URI = "git://github.com/MistySOM/phytool.git" SRCREV = "8882328c08ba2efb13c049812098f1d0cb8adf0c" LIC_FILES_CHKSUM = "file://../git/LICENSE;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0" INHERIT += " pseudo" python do_display_banner() {     bb.plain("some message") } addtask display_banner before do_build [/phytool_2.0.bb] It appears like my INHERIT line does not satsify the requirement. How can I resolve this?

--

RON EGGLER Firmware Engineer (he/him/his) www.mistywest.com

Hi,
Op 26-12-2022 om 03:11 schreef Randy MacLeod:

On 2022-12-24 13:58, Ferry Toth via lists.yoctoproject.org wrote:

Op 18-04-2021 om 21:34 schreef Gmane Admin:

Hi,
Op 18-04-2021 om 11:59 schreef Richard Purdie:

On Sun, 2021-04-18 at 00:17 +0200, Gmane Admin wrote:

Hi,
Op 14-04-2021 om 06:59 schreef Richard Purdie:

On Tue, 2021-04-13 at 21:14 -0400, Randy MacLeod wrote:

On 2021-04-11 12:19 p.m., Alexander Kanavin wrote:

make already has -l option for limiting new instances if load average is
too high, so it's only natural to add a RAM limiter too.

     -l [N], --load-average[=N], --max-load[=N]
                                 Don't start multiple jobs unless load is
below N.

In any case, patches welcome :)

During today's Yocto technical call (1),
we talked about approaches to limiting the system load and avoiding
swap and/or OOM events. Here's what (little!) i recall from the
discussion, 9 busy hours later.

In the short run, instead of independently maintaining changes to
configurations to limit parallelism or xz memory usage, etc, we
could develop an optional common include file where such limits
are shared across the community.

I tried PARALLEL_MAKE_nodejs = "-j 1" from local.conf but that didn't work.

It would need to be:

PARALLEL_MAKE_pn-nodejs = "-j 1"

So I watched it run for a while. It compiles with g++ and as at about
0.5GB per thread, which is OK. In the end it does ld taking 4GB and it
tries to do 4 in parallel. And then swapping becomes so heavy the
desktop becomes unresponsive. Like I mentioned before ssh from another
machine allows me to STOP one of them, allowing the remaining to
complete. And then CONT the last one.

I worked around it now, by creating a bbappend for nodejs with only
PARALLEL_MAKE = "-j 2"

If that works, the override above should also work. You do need the "pn-"
prefix to the recipe name though.

And indeed it does, thanks so much for the tip.

This discussion is very old, but I just wanted to share an update. With my image (meta-intel-edison) building nodejs is the dominant time consumer. With "-j 2" the nodejs build takes multiple hours (I have 8 core + HT), all the rest is then long completed. Without "-j 2" building  nodejs_16.14.2 linking in the final stage 5 processes are started, each 4 - 5GB RAM. With my meager 16GB disk thrashing starts and it still would take hours to complete.

While I still believe we need bitbake to SIGSTOP processes that trigger thrashing, for nodejs I found a solution at the recipe level:

The yocto recipe already has:

EXTRA_OEMAKE = "\
    CC.host='${CC}' \
    CFLAGS.host='${CPPFLAGS} ${CFLAGS}' \
    CXX.host='${CXX}' \
    CXXFLAGS.host='${CPPFLAGS} ${CXXFLAGS}' \
    LDFLAGS.host='${LDFLAGS}' \
    AR.host='${AR}' \
    \
    builddir_name=./ \
"

I inserted `    LINK.host='flock /tmp ${CXX}' \` as that seemed most logical, but that didn't help a bit (the 4 calls in my log to flock were never simultaneous). However after changing it to ` LINK='flock /tmp ${CXX}' \` the linker is serialized.

This makes an enormous difference in my Yocto image build time (nodejs was the dominant factor). Now compile stage ~25min + 5min for link.

Terry,
It's nice to hear that you have an approach to make building nodejs feasible for you on your hardware.
I have been wondering if we should be trying to serialize linking across recipes (nodejs, webkitgtk, llvm, and other larger packages)
but I hadn't noticed that nodejs itself was in need of umm, *guidance*. ;-)
Are you aware of the recently added: BB_PRESSURE_MAX_[CPU|IO|MEMORY] variables?
https://docs.yoctoproject.org/bitbake/2.2/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU
That might help you a bit as well.

Nope I hadn't seen that. I immediately tried it, but of course it didn't work.

Reason is, it regulates starting tasks. But the problem with nodejs is (if I don't intervene) the task do_compile takes so long, there are not other tasks running.

Also looking at the documentation: "At this point in time, experiments show that IO pressure tends to be short-lived and regulating just the CPU can help to reduce it."

This is not true. I have just taken some measurements with htop for my case 16 threads + 16GB RAM:
- the do_compile task roughly falls appart into 2 phases, compiling (many files) and linking (1 + 5 + 5 files).

During compiling I measure PSI CPU=4%, IO = 0%, MEM = 0%, memory use 10G, swap 1.5G
During linking PSI CPU=0.1%, IO = 90%, MEM = 66%, memory use 14G, swap 8G

You can easily see CPU pressure is not the issue, all CPU time is spend on swapping (IO pressure).

This is also the reason I can get my system responding again by SIGSTOP on some of the ld processes, once they are stopped (not killed) they are swapped out, and stay out.

Basically this is proof that when I manually schedule processes I do a better job then the kernel.

Zheng and I also have a brand new, unpolished patch, to 'make' to limit builds based on PSI (the *some* line from /proc/pressure/cpu ).
If you're interested, you could give that a try by either applying a patch to the make recipe and having nodejs depend on make-native
or if you just want to try it out on a private system, build make from:

I am, but I already know this will not work, see above. CPU pressure is not the issue.

If you would be limiting IO pressure it might help, although I expect once you get an IO over-pressure it's already too late, the processes have already been started. The only way I know to resolve, is to stop them temporarily.

   https://github.com/ZhengQ2/make/tree/cpu-pressure-to-commit
save your old version of make and install the new one in in /usr/bin
and set:
PARALLEL_MAKE = " -j -z 5"
This will prevent make from starting new tasks when the CPU pressure,
based on our very simple calculation, is > 5%
We've seen that it can speed up the openssl build on a 24 core system from:
89.030 s ±  0.485 s <-- -j 24
81.727 s ±  1.139 s <-- -j -z 11
for native builds of openssl outside of bitbake.
Early times but it's an interesting data point.
Builds of the kernel didn't show any speed-up and were in fact a bit slower:
$ hyperfine --runs 10 --prepare 'make clean; make defconfig' 'make -j 24' --export-json make-j-24--linux-kernel.json
Benchmark 1: make -j 24
  Time (mean ± σ):     181.770 s ±  0.290 s    [User: 3416.715 s, System: 354.282 s]
  Range (min … max):   181.266 s … 182.231 s    10 runs
# Try -j -z 11
 hyperfine --runs 10 --prepare 'make clean; make defconfig' 'make -j -z 11 ' --export-json make-j-z-11--linux-kernel.json
Benchmark 1: make -j -z 11
  Time (mean ± σ):     185.915 s ±  1.988 s    [User: 3447.606 s, System: 343.528 s]
  Range (min … max):   184.046 s … 189.901 s    10 runs
but presumably not needing to specify the number of cores to use, could be a benefit.
PSI also has /proc/pressure/memory but we've found that for bitbake at least,
if you limit CPU usage, you can limit overall memory usage. Package builds and linking
may be a different kettle of fish and your explicit serialization would always be better
at preventing the system from swapping than PSI memory, which only provides a signal
when things have gotten out of hand already.
../Randy

Ferry

Cheers,

Richard

Hi everyone,

I have a task to add browser to my yocto project and start it immediately after it is booted. Also I have to open specific web page. Any suggestions how to do it? Any advice would be helpful. 

Thanks in advance!

Hi,

I'm want to write a recipe to compile phytool but I'm getting the below error when building it with:
$ bitbake -b phytool_2.0
ERROR: Failed to spawn fakeroot worker to run /home/yocto/rzg_vlp_v3.0.0/build/../meta-mistysom/recipes-bringup/phytool/phytool_2.0.bb:do_install: [Errno 2] No such file or directory: '/home/yocto/rzg_vlp_v3.0.0/build/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo'

my bb file looks like:

[phytool_2.0.bb]
DESCRIPTION = "Recipe builds phytool utility"
# Needs to be changed to GPLv2
LICENSE = "MIT"
SRC_URI = "git://github.com/MistySOM/phytool.git"
SRCREV = "8882328c08ba2efb13c049812098f1d0cb8adf0c"
LIC_FILES_CHKSUM = "file://../git/LICENSE;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0"
INHERIT += " pseudo"

python do_display_banner() {
    bb.plain("some message")
}

addtask display_banner before do_build
[/phytool_2.0.bb]

It appears like my INHERIT line does not satsify the requirement.
How can I resolve this?

--
RON EGGLER Firmware Engineer (he/him/his) www.mistywest.com

From: Alexandre Belloni <alexandre.belloni@...>

Signed-off-by: Alexandre Belloni <alexandre.belloni@...>
---
config.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index 298791b694f6..000d331b8eff 100644
--- a/config.json
+++ b/config.json
@@ -196,7 +196,7 @@
"extravars" : [
"RPM_GPG_SIGN_CHUNK = '1'"
],
- "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror -T machine -T toolchain-user -T toolchain-system -j 15"],
+ "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror -T machine -T toolchain-user -T toolchain-system -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
}
},
@@ -233,7 +233,7 @@
},
"step2" : {
"shortname" : "OE Selftest",
- "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15"],
+ "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
},
"step3" : {
--
2.38.1

From: Alexandre Belloni <alexandre.belloni@...>

langdale isn't currently supported on any branch of meta-agl

Signed-off-by: Alexandre Belloni <alexandre.belloni@...>
---
config.json | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/config.json b/config.json
index 27283c9f1610..45560c5cc522 100644
--- a/config.json
+++ b/config.json
@@ -890,33 +890,29 @@
}
},
"check-layer-nightly" : {
- "NEEDREPOS" : ["poky", "meta-agl", "meta-arm", "meta-aws", "meta-intel", "meta-ti", "meta-openembedded", "meta-virtualization"],
+ "NEEDREPOS" : ["poky", "meta-arm", "meta-aws", "meta-intel", "meta-ti", "meta-openembedded", "meta-virtualization"],
"step1" : {
- "shortname" : "Test meta-agl YP Compatibility",
- "EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-agl/meta-agl-core ../meta-agl/meta-netboot ../meta-agl/meta-pipewire --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python --no-auto-dependency"]
- },
- "step2" : {
"shortname" : "Test meta-arm YP Compatibility",
"MACHINE" : "qemuarm64",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-arm/meta-arm ../meta-arm/meta-arm-toolchain ../meta-arm/meta-arm-bsp ../meta-arm/meta-gem5 ../meta-arm/meta-arm-autonomy --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python ../meta-poky ../meta-openembedded/meta-filesystems ../meta-virtualization --no-auto-dependency"]
},
- "step3" : {
+ "step2" : {
"shortname" : "Test meta-aws YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-aws --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python -n --no-auto-dependency"]
},
- "step4" : {
+ "step3" : {
"shortname" : "Test meta-intel YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-intel --no-auto-dependency"]
},
- "step5" : {
+ "step4" : {
"shortname" : "Test meta-ti YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-ti/meta-ti-bsp ../meta-ti/meta-ti-extras --dependency ../meta-arm --no-auto-dependency"]
},
- "step6" : {
+ "step5" : {
"shortname" : "Test meta-oe YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-openembedded --no-auto-dependency"]
},
- "step7" : {
+ "step6" : {
"shortname" : "Test meta-virtualization YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-virtualization --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python ../meta-openembedded/meta-filesystems --no-auto-dependency"]
}
--
2.38.1

I forgot to tag this one for langdale

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

From: Alexandre Belloni <alexandre.belloni@...>

langdale isn't currently supported on any branch of meta-agl

Signed-off-by: Alexandre Belloni <alexandre.belloni@...>
---
config.json | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/config.json b/config.json
index 27283c9f1610..45560c5cc522 100644
--- a/config.json
+++ b/config.json
@@ -890,33 +890,29 @@
}
},
"check-layer-nightly" : {
- "NEEDREPOS" : ["poky", "meta-agl", "meta-arm", "meta-aws", "meta-intel", "meta-ti", "meta-openembedded", "meta-virtualization"],
+ "NEEDREPOS" : ["poky", "meta-arm", "meta-aws", "meta-intel", "meta-ti", "meta-openembedded", "meta-virtualization"],
"step1" : {
- "shortname" : "Test meta-agl YP Compatibility",
- "EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-agl/meta-agl-core ../meta-agl/meta-netboot ../meta-agl/meta-pipewire --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python --no-auto-dependency"]
- },
- "step2" : {
"shortname" : "Test meta-arm YP Compatibility",
"MACHINE" : "qemuarm64",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-arm/meta-arm ../meta-arm/meta-arm-toolchain ../meta-arm/meta-arm-bsp ../meta-arm/meta-gem5 ../meta-arm/meta-arm-autonomy --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python ../meta-poky ../meta-openembedded/meta-filesystems ../meta-virtualization --no-auto-dependency"]
},
- "step3" : {
+ "step2" : {
"shortname" : "Test meta-aws YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-aws --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python -n --no-auto-dependency"]
},
- "step4" : {
+ "step3" : {
"shortname" : "Test meta-intel YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-intel --no-auto-dependency"]
},
- "step5" : {
+ "step4" : {
"shortname" : "Test meta-ti YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-ti/meta-ti-bsp ../meta-ti/meta-ti-extras --dependency ../meta-arm --no-auto-dependency"]
},
- "step6" : {
+ "step5" : {
"shortname" : "Test meta-oe YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-openembedded --no-auto-dependency"]
},
- "step7" : {
+ "step6" : {
"shortname" : "Test meta-virtualization YP Compatibility",
"EXTRACMDS" : ["yocto-check-layer-wrapper ../meta-virtualization --dependency ../meta-openembedded/meta-oe ../meta-openembedded/meta-networking ../meta-openembedded/meta-python ../meta-openembedded/meta-filesystems --no-auto-dependency"]
}
--
2.38.1

On 2022-12-24 13:58, Ferry Toth via lists.yoctoproject.org wrote:

Op 18-04-2021 om 21:34 schreef Gmane Admin:

Hi,
Op 18-04-2021 om 11:59 schreef Richard Purdie:

On Sun, 2021-04-18 at 00:17 +0200, Gmane Admin wrote:

Hi,
Op 14-04-2021 om 06:59 schreef Richard Purdie:

On Tue, 2021-04-13 at 21:14 -0400, Randy MacLeod wrote:

On 2021-04-11 12:19 p.m., Alexander Kanavin wrote:

make already has -l option for limiting new instances if load average is
too high, so it's only natural to add a RAM limiter too.

     -l [N], --load-average[=N], --max-load[=N]
                                 Don't start multiple jobs unless load is
below N.

In any case, patches welcome :)

During today's Yocto technical call (1),
we talked about approaches to limiting the system load and avoiding
swap and/or OOM events. Here's what (little!) i recall from the
discussion, 9 busy hours later.

In the short run, instead of independently maintaining changes to
configurations to limit parallelism or xz memory usage, etc, we
could develop an optional common include file where such limits
are shared across the community.

I tried PARALLEL_MAKE_nodejs = "-j 1" from local.conf but that didn't work.

It would need to be:

PARALLEL_MAKE_pn-nodejs = "-j 1"

So I watched it run for a while. It compiles with g++ and as at about
0.5GB per thread, which is OK. In the end it does ld taking 4GB and it
tries to do 4 in parallel. And then swapping becomes so heavy the
desktop becomes unresponsive. Like I mentioned before ssh from another
machine allows me to STOP one of them, allowing the remaining to
complete. And then CONT the last one.

I worked around it now, by creating a bbappend for nodejs with only
PARALLEL_MAKE = "-j 2"

If that works, the override above should also work. You do need the "pn-"
prefix to the recipe name though.

And indeed it does, thanks so much for the tip.

This discussion is very old, but I just wanted to share an update. With my image (meta-intel-edison) building nodejs is the dominant time consumer. With "-j 2" the nodejs build takes multiple hours (I have 8 core + HT), all the rest is then long completed. Without "-j 2" building  nodejs_16.14.2 linking in the final stage 5 processes are started, each 4 - 5GB RAM. With my meager 16GB disk thrashing starts and it still would take hours to complete.

While I still believe we need bitbake to SIGSTOP processes that trigger thrashing, for nodejs I found a solution at the recipe level:

The yocto recipe already has:

EXTRA_OEMAKE = "\
    CC.host='${CC}' \
    CFLAGS.host='${CPPFLAGS} ${CFLAGS}' \
    CXX.host='${CXX}' \
    CXXFLAGS.host='${CPPFLAGS} ${CXXFLAGS}' \
    LDFLAGS.host='${LDFLAGS}' \
    AR.host='${AR}' \
    \
    builddir_name=./ \
"

I inserted `    LINK.host='flock /tmp ${CXX}' \` as that seemed most logical, but that didn't help a bit (the 4 calls in my log to flock were never simultaneous). However after changing it to ` LINK='flock /tmp ${CXX}' \` the linker is serialized.

This makes an enormous difference in my Yocto image build time (nodejs was the dominant factor). Now compile stage ~25min + 5min for link.

Terry,

It's nice to hear that you have an approach to make building nodejs feasible for you on your hardware.

I have been wondering if we should be trying to serialize linking across recipes (nodejs, webkitgtk, llvm, and other larger packages)
but I hadn't noticed that nodejs itself was in need of umm, *guidance*.  ;-)

Are you aware of the recently added: BB_PRESSURE_MAX_[CPU|IO|MEMORY] variables?

https://docs.yoctoproject.org/bitbake/2.2/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-BB_PRESSURE_MAX_CPU

That might help you a bit as well.

Zheng and I also have a brand new, unpolished patch, to 'make' to limit builds based on PSI (the *some* line from /proc/pressure/cpu ).
If you're interested, you could give that a try by either applying a patch to the make recipe and having nodejs depend on make-native
or if you just want to try it out on a private system, build make from:

   https://github.com/ZhengQ2/make/tree/cpu-pressure-to-commit

save your old version of make and install the new one in in /usr/bin

and set:

PARALLEL_MAKE = " -j -z 5"

This will prevent make from starting new tasks when the CPU pressure,
based on our very simple calculation, is > 5%

We've seen that it can speed up the openssl build on a 24 core system from:

89.030 s ±  0.485 s <-- -j 24

81.727 s ±  1.139 s <-- -j -z 11

for native builds of openssl outside of bitbake.
Early times but it's an interesting data point.

Builds of the kernel didn't show any speed-up and were in fact a bit slower:

$ hyperfine --runs 10 --prepare 'make clean; make defconfig' 'make -j 24' --export-json make-j-24--linux-kernel.json
Benchmark 1: make -j 24
  Time (mean ± σ):     181.770 s ±  0.290 s    [User: 3416.715 s, System: 354.282 s]
  Range (min … max):   181.266 s … 182.231 s    10 runs


# Try -j -z 11

 hyperfine --runs 10 --prepare 'make clean; make defconfig' 'make -j -z 11 ' --export-json make-j-z-11--linux-kernel.json
Benchmark 1: make -j -z 11
  Time (mean ± σ):     185.915 s ±  1.988 s    [User: 3447.606 s, System: 343.528 s]
  Range (min … max):   184.046 s … 189.901 s    10 runs

but presumably not needing to specify the number of cores to use, could be a benefit.

PSI also has /proc/pressure/memory but we've found that for bitbake at least,
if you limit CPU usage, you can limit overall memory usage. Package builds and linking
may be a different kettle of fish and your explicit serialization would always be better
at preventing the system from swapping than PSI memory, which only provides a signal
when things have gotten out of hand already.

../Randy

Ferry

Cheers,

Richard

--
# Randy MacLeod
# Wind River Linux

On 2022-12-24 08:38, Markus Volk wrote:

Am Sa, 24. Dez 2022 um 08:34:49 -0800 schrieb Mistyron <ron.eggler@...>:

$ grep usb path/to/package.manifest   libusb-1.0-0   usbutils   usbutils-python

lsusb is part of usbutils, so it is not explicitly listed, but should be included in your image

Oh, I think the cleanall didn't wipe the cache sufficiently, after I deleted contents in sstate-cache/ I get the following:
$ tar -tvf path/to/myimage-20221224185313.rootfs.tar.gz  | grep lsusb
lrwxrwxrwx 0/0               0 2018-03-09 04:34 ./usr/bin/lsusb -> /usr/bin/lsusb.usbutils
-rwxr-xr-x 0/0           14266 2018-03-09 04:34 ./usr/bin/lsusb.py
-rwxr-xr-x 0/0          247976 2018-03-09 04:34 ./usr/bin/lsusb.usbutils
-rw-r--r-- 0/0              43 2018-03-09 04:34 ./usr/lib/opkg/alternatives/lsusb

I would expect "$ lsusb" to work now when I copy this to my board (cannot est right now)

the package.manifest however still only lists:

libusb-1.0-0
usbutils
usbutils-python

Op 18-04-2021 om 21:34 schreef Gmane Admin:

Hi,
Op 18-04-2021 om 11:59 schreef Richard Purdie:

On Sun, 2021-04-18 at 00:17 +0200, Gmane Admin wrote:

Hi,
Op 14-04-2021 om 06:59 schreef Richard Purdie:

On Tue, 2021-04-13 at 21:14 -0400, Randy MacLeod wrote:

On 2021-04-11 12:19 p.m., Alexander Kanavin wrote:

make already has -l option for limiting new instances if load average is
too high, so it's only natural to add a RAM limiter too.

     -l [N], --load-average[=N], --max-load[=N]
                                 Don't start multiple jobs unless load is
below N.

In any case, patches welcome :)

During today's Yocto technical call (1),
we talked about approaches to limiting the system load and avoiding
swap and/or OOM events. Here's what (little!) i recall from the
discussion, 9 busy hours later.

In the short run, instead of independently maintaining changes to
configurations to limit parallelism or xz memory usage, etc, we
could develop an optional common include file where such limits
are shared across the community.

I tried PARALLEL_MAKE_nodejs = "-j 1" from local.conf but that didn't work.

It would need to be:

PARALLEL_MAKE_pn-nodejs = "-j 1"

So I watched it run for a while. It compiles with g++ and as at about
0.5GB per thread, which is OK. In the end it does ld taking 4GB and it
tries to do 4 in parallel. And then swapping becomes so heavy the
desktop becomes unresponsive. Like I mentioned before ssh from another
machine allows me to STOP one of them, allowing the remaining to
complete. And then CONT the last one.

I worked around it now, by creating a bbappend for nodejs with only
PARALLEL_MAKE = "-j 2"

If that works, the override above should also work. You do need the "pn-"
prefix to the recipe name though.

And indeed it does, thanks so much for the tip.

This discussion is very old, but I just wanted to share an update. With my image (meta-intel-edison) building nodejs is the dominant time consumer. With "-j 2" the nodejs build takes multiple hours (I have 8 core + HT), all the rest is then long completed. Without "-j 2" building nodejs_16.14.2 linking in the final stage 5 processes are started, each 4 - 5GB RAM. With my meager 16GB disk thrashing starts and it still would take hours to complete.

While I still believe we need bitbake to SIGSTOP processes that trigger thrashing, for nodejs I found a solution at the recipe level:

The yocto recipe already has:

EXTRA_OEMAKE = "\
CC.host='${CC}' \
CFLAGS.host='${CPPFLAGS} ${CFLAGS}' \
CXX.host='${CXX}' \
CXXFLAGS.host='${CPPFLAGS} ${CXXFLAGS}' \
LDFLAGS.host='${LDFLAGS}' \
AR.host='${AR}' \
\
builddir_name=./ \
"

I inserted ` LINK.host='flock /tmp ${CXX}' \` as that seemed most logical, but that didn't help a bit (the 4 calls in my log to flock were never simultaneous). However after changing it to ` LINK='flock /tmp ${CXX}' \` the linker is serialized.

This makes an enormous difference in my Yocto image build time (nodejs was the dominant factor). Now compile stage ~25min + 5min for link.

Ferry

Cheers,

Richard