On Mon, Oct 25, 2021 at 9:46 AM Steve Sakoman <steve@...> wrote:

On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:

On 10/25/21 7:07 AM, Anuj Mittal wrote:

From: Trevor Gamblin <trevor.gamblin@...>

Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.

Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.

Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'",

We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.

So this change would need some testing before we make it permanent into AB

Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.

it was dunfell primarily

And master has been running with this for q
Steve

"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",

On Mon, Oct 25, 2021 at 6:46 AM Steve Sakoman via
lists.yoctoproject.org <steve=sakoman.com@...>
wrote:

On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:

On 10/25/21 7:07 AM, Anuj Mittal wrote:

From: Trevor Gamblin <trevor.gamblin@...>

Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.

Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.

Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'",

We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.

So this change would need some testing before we make it permanent into AB

Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.

And master has been running with this for q

Accidentally hit send :-)

I intended to say that master has been running with this for quite
some time now.

Steve

"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",

On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:

On 10/25/21 7:07 AM, Anuj Mittal wrote:

From: Trevor Gamblin <trevor.gamblin@...>

Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.

Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.

Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'",

We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.

So this change would need some testing before we make it permanent into AB

Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.

And master has been running with this for q
Steve

"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",

On 10/25/21 7:07 AM, Anuj Mittal wrote:

From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'",

We have seen a lot of hung builds when we used this option internally on builders, Additionally some packages like bison started to fail intermittently which seemed like a parallel build issue but never got to see why it would happen only with -l enabled.

So this change would need some testing before we make it permanent into AB

"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",

Hello Richard,

Thanks for the quick help!

Best regards,
Matthias

From: Trevor Gamblin <trevor.gamblin@...>

Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.

Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.

Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
--
2.31.1

I am looking to selectively remove portions of 1or 2 components I will not be using from my running image

No, you got it wromg... I have a buildable image which has all the components necessary, I want to remove some overhead installed by some components that I will not be within my running application...

The super yoctoish way would be to alter the packaging of the recipe to your needs.

Let's assume recipe-a consists of the files

foo
bar

and it would package both files into recipe-a pkg.

In the actual image you only install packages into a file system, so if you don't want bar to be installed into your final image, it'd be best to package bar into a separate package.

PACKAGES_BEFORE_PN = "${PN}-bar"
FILES:${PN}-bar = "bar"

in this case the recipe would produce

recipe-a (containing just foo)
recipe-a-bar (containing just bar)

You could install now (via IMAGE_INSTALL in your image recipe) just add recipe-a and only foo would be put into the final image.

While a potential other image (IMAGE_INSTALL += "recipe-a-bar recipe-a")
would contain both.

But if you'd go down that road you have to make sure that runtime dependencies between the packages are met (using REDEPENDS:* statements), as otherwise it could turn into very hard to debug issues.
So I would propose to **not** do that via bbappends, but forks of the recipes you want to repackage.

Another option (as likely mentioned already) is to use

rem_stuff() {
rm ${IMAGE_ROOTFS}/<file you want to delete>
}

ROOTFS_POSTPROCESS_COMMAND += "rem_stuff;"

in your image recipe

Hello Steve,

Is the approach where you remove the recipes from the image an option?
Is there a reason why you want to build the recipes that you do not
want to appear in your image?

Regards,

Leon.


--
Leon Woestenberg
leon@...
T: +31 40 711 42 76
M: +31 6 472 30 372

Sidebranch Embedded Systems
Eindhoven, The Netherlands
http://www.sidebranch.com



On Mon, Oct 25, 2021 at 2:55 PM Monsees, Steven C (US) via
lists.yoctoproject.org
<steven.monsees=baesystems.com@...> wrote:

On Mon, 25 Oct 2021, Monsees, Steven C (US) via lists.yoctoproject.org wrote:

If I am building an image “image-ABC”, and it is composed of a
number recipes, and for some of these recipes I may NOT want to
install their final components within my image…

Which is the best place to modify the build with bbappend ?

Would I modify a recipe’s do_install (do_install_append-recipe_xyz)
?, or would I modify the image recipe (do_install_append-image_ABC)
I am leaning this way to avoid cases where the component recipes
might have bbappends in place already ?

i would think start with defining a minimal image, then define other
images based on that one which add more recipes. this has nothing to
do with redefining the recipes, it just means defining more than one
image.

rday

Hello,

I have a recipe that installs a simple bash script to run on startup.
As you all know, there is systemd and sysvinit.
I need to add support for both init managers.

I can do :
SRC_URI_append = " ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'file://script.service', 'file://script.sh', d)}"

and I can do the same in all the tasks.

The problem is, that I need to inherit either systemd or update-rc.d

Is it possible to inherit them all, or how can I inherit something conditionally ?

Thanks,
Talel

On Mon, 2021-10-25 at 06:50 +0000, Matthias Klein wrote:

Hello,

our dunfell based yocto no longer builds in the pkgconfig-native section. I tried it in an Ubuntu 18.04 and Debian 10 Docker container.
The build always aborts as follows:

checking thread related cflags... -D_REENTRANT

checking for pthread_create/pthread_join... no
checking for pthread_create/pthread_join in -lpthread... no
checking for pthread_create/pthread_join in -lpthread32... no
checking for pthread_create/pthread_join in -lpthreads... no
checking for pthread_create/pthread_join in -lthread... no
configure: error: I can't find the libraries for the thread implementation
posix. Please choose another thread implementation or
provide information on your thread implementation.
configure: error: ../../git/glib/configure failed for glib
NOTE: The following config.log files may provide further information.
NOTE: /work/yocto-optimeas/build/tmp/work/x86_64-linux/pkgconfig-native/0.29.2+gitAUTOINC+edf8e6f0ea-r0/build/glib/config.log /work/yocto-optimeas/build/tmp/work/x86_64-linux/pkgconfig-native/0.29.2+gitAUTOINC+edf8e6f0ea-r0/build/config.log
ERROR: configure failed
WARNING: /work/yocto-optimeas/build/tmp/work/x86_64-linux/pkgconfig-native/0.29.2+gitAUTOINC+edf8e6f0ea-r0/temp/run.do_configure.243679:1 exit 1 from 'exit 1'
ERROR: Execution of '/work/yocto-optimeas/build/tmp/work/x86_64-linux/pkgconfig-native/0.29.2+gitAUTOINC+edf8e6f0ea-r0/temp/run.do_configure.243679' failed with exit code 1

ERROR: Task (virtual:native:/work/yocto-optimeas/yocto/poky/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_configure) failed with exit code '1'
ERROR: ninja-native-1.10.0-r0 do_compile: Execution of '/work/yocto-optimeas/build/tmp/work/x86_64-linux/ninja-native/1.10.0-r0/temp/run.do_compile.319277' failed with exit code 1
ERROR: Logfile of failure stored in: /work/yocto-optimeas/build/tmp/work/x86_64-linux/ninja-native/1.10.0-r0/temp/log.do_compile.319277
Log data follows:

DEBUG: Executing shell function do_compile
ninja: fatal: posix_spawn: Operation not permitted
bootstrapping ninja...
wrote build.ninja.
bootstrap complete. rebuilding...
Traceback (most recent call last):
File "./configure.py", line 709, in <module>
subprocess.check_call(rebuild_args)
File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./ninja']' returned non-zero exit status 1.
WARNING: /work/yocto-optimeas/build/tmp/work/x86_64-linux/ninja-native/1.10.0-r0/temp/run.do_compile.319277:1 exit 1 from 'python3 ./configure.py --bootstrap'
ERROR: Execution of '/work/yocto-optimeas/build/tmp/work/x86_64-linux/ninja-native/1.10.0-r0/temp/run.do_compile.319277' failed with exit code 1

ERROR: Task (virtual:native:/work/yocto-optimeas/yocto/poky/meta/recipes-devtools/ninja/ninja_1.10.0.bb:do_compile) failed with exit code '1'


I also tried using the commits from dunfell-next in the poky repo as a test. But then the build doesn't even start and I get the following error:

ERROR: ParseError at /work/yocto-optimeas/yocto/meta-openembedded/meta-oe/recipes-extended/libimobiledevice/libplist_2.1.0.bb:9: Could not inherit file classes/python3targetconfig.bbclass | ETA: --:--:--


Does anyone have any idea what is causing this? does anyone have the same behavior?

I just backported a uninative upgrade to dunfell which should address this
issue.

Cheers,

Richard

Hello,

We also have the same issue with similar in-docker builds, a workaround for the issue is mentioned here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14519 until the fix is backported to dunfell.

Best Regards,
Ahmed Hossam

No there isn't - with the yocto project you can build your own distribution not mimic other already available distributions.

If you want to build Ubuntu you have to ask canonical.

I want to build ubuntu distro using yocto.

Is there a way for that ?