Date   

Re: [PATCH yocto-autobuilder-helper] Enable canary builds for meta-aws[aws-iot-device-sdk-cpp-v2] on x86-64 and arm64.

Richard Purdie
 

On Thu, 2021-07-22 at 16:05 -0400, elberger via lists.yoctoproject.org wrote:
Hello,

What can I do to move this patch along - or did I miss anything? 
Apologies for my ignorance, it's my first time submitting a patch to YP.
I can't see the patch in the mailing list archives:

https://lists.yoctoproject.org/g/yocto/topics?

only your reply:

https://lists.yoctoproject.org/g/yocto/topic/patch/84387478?

so I don't think the original patch made it to the list? Since I never
saw it there, I couldn't apply it!

Cheers,

Richard


Re: [PATCH yocto-autobuilder-helper] Enable canary builds for meta-aws[aws-iot-device-sdk-cpp-v2] on x86-64 and arm64.

Richard Elberger
 

Hello,

What can I do to move this patch along - or did I miss anything?  Apologies for my ignorance, it's my first time submitting a patch to YP.

Richard Elberger

On 7/20/21 7:08 AM, rpcme.github@... wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



From: Richard Elberger <elberger@...>

This patch provides the buildbot system to make the aws-iot-device-sdk-cpp-v2.
- build on poky
- meta-openembedded and meta-aws are additional repo dependencies
- build for the two dominant architectures: x86-64 and arm64.

Signed-off-by: Richard Elberger <elberger@...>
---
config.json | 17 +++++++++++++++++
1 file changed, 17 insertions(+)

diff --git a/config.json b/config.json
index 3709b40..f04fac8 100644
--- a/config.json
+++ b/config.json
@@ -350,6 +350,23 @@
"BBTARGETS": "agl-image-core-autobuilder"
}
},
+ "meta-aws" : {
+ "NEEDREPOS" : ["poky", "meta-openembedded", "meta-aws"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-aws"
+ ],
+ "step1" : {
+ "MACHINE": "qemux86-64",
+ "BBTARGETS": "aws-iot-device-sdk-cpp-v2"
+ },
+ "step2" : {
+ "MACHINE": "qemuarm64",
+ "BBTARGETS": "aws-iot-device-sdk-cpp-v2"
+ }
+ },
"bringup" : {
"BUILDINFO" : true,
"SSTATEDIR" : ["SSTATE_DIR ?= '${HELPERBUILDDIR}/sstate'"],
--
2.17.1
--
Richard Elberger
Principal Technologist
AWS IoT

+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email


Re: [poky] [PATCH] local.conf.sample: disable prelink

Alexander Kanavin
 

PIE is nowadays more or less the only available option and is expected for improved security; Yocto does not even test non-PIE builds or provide an off the shelf way to turn it off.

I also have to note that prelink does show higher RAM consumption in your tests as well (MemFree column). On the constrained systems which would benefit most from improved prelink timings that might be a bigger loss than not prelinking.

But yes, there is a timing benefit visible in the tests: 0.01s vs 0.1s.

Alex


On Mon, 19 Jul 2021 at 22:58, Robert Berger@... <robert.berger.yocto.user@...> wrote:
Hi Alex, RP, Mark,

I did some research on the subject in order to try to figure out what is
going on.

1) I come to a similar conclusion with what found, but tried to look a
bit deeper for the reason.

1.1) The reason that cross-prelink is not prelinking is, that for a
quite some time by default everything is built with PIE mode by default
and cross-prelink does not seem to be able to work on exe/libs compiled
with PIE mode. So seeing the same behavior with and without prelinking
is what I would expect as long as everything is compiled with PIE mode.

A more detailed analysis of my tests can be found on my not yet
officially published site:

https://rlbl.me/prelink-1

https://rlbl.me/prelink-2

Alex:

Can you please rebuild your test images without PIE mode and re-run the
tests?

Then we should have the 4 test cases:

prelinked-with-pie
no-prelink-with-pie
prelink-no-pie
no-prelink-no-pie

I guess then we can discuss what are the next steps.

In my opinion the current default settings, which compile close to
everything in PIE mode, but invoke also cross-prelink do not make much
sense.

The question is: "Do we want to drop cross-prelink, or do we want to
drag it along and come up more fine-grained configuration options?"

We could e.g. exclude certain files from pre-linking.

IMHO cross-prelink still works, but not on exe/libs which were compiled
in PIE mode.

Regards,

Robert


[yocto-autobuilder-helper][hardknott][PATCH] config.json: track system load with PARALLEL_MAKE

Trevor Gamblin
 

Backport for hardknott since we have been running for ~1 week with this
patch in master and the same load limiting may help reduce build
failures for hardknott itself as well as when its builds are running
alongside other branches.

This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 50 (the previous iteration of this patch both
had it set to 100 and mistakenly stated that this was a percentage,
rather than an absolute representation of system load ala top/htop).
This option is supported by both Make and Ninja. However, we also
require the "--debug=j" option to be passed to Make in order for the
latter to report perceived system load in the do_compile logs, (which we
need in order to track test values), and since this option is not
supported by Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we
can determine if the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
config.json | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index 110b0d9..5689cb9 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 50'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
@@ -57,7 +57,8 @@
"SDK_INCLUDE_TOOLCHAIN = '1'",
"BB_DISKMON_DIRS = 'STOPTASKS,${TMPDIR},1G,100K STOPTASKS,${DL_DIR},1G STOPTASKS,${SSTATE_DIR},1G STOPTASKS,/tmp,100M,100K ABORT,${TMPDIR},100M,1K ABORT,${DL_DIR},100M ABORT,${SSTATE_DIR},100M ABORT,/tmp,10M,1K'",
"BB_HASHSERVE = 'typhoon.yocto.io:8686'",
- "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'"
+ "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
+ "EXTRA_OEMAKE = ' --debug=j'"
]
},
"templates" : {
--
2.31.1


[yocto-autobuilder-helper][gatesgarth][PATCH] config.json: track system load with PARALLEL_MAKE

Trevor Gamblin
 

Backport for gatesgarth since we have been running for ~1 week with this
patch in master and the same load limiting may help reduce build
failures for gatesgarth itself as well as when its builds are running
alongside other branches.

This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 50 (the previous iteration of this patch both
had it set to 100 and mistakenly stated that this was a percentage,
rather than an absolute representation of system load ala top/htop).
This option is supported by both Make and Ninja. However, we also
require the "--debug=j" option to be passed to Make in order for the
latter to report perceived system load in the do_compile logs, (which we
need in order to track test values), and since this option is not
supported by Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we
can determine if the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
config.json | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index 5dda653..8a4343a 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 50'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
@@ -56,7 +56,8 @@
"SDK_EXT_TYPE = 'minimal'",
"SDK_INCLUDE_TOOLCHAIN = '1'",
"BB_DISKMON_DIRS = 'STOPTASKS,${TMPDIR},1G,100K STOPTASKS,${DL_DIR},1G STOPTASKS,${SSTATE_DIR},1G STOPTASKS,/tmp,100M,100K ABORT,${TMPDIR},100M,1K ABORT,${DL_DIR},100M ABORT,${SSTATE_DIR},100M ABORT,/tmp,10M,1K'",
- "BB_HASHSERVE = 'typhoon.yocto.io:8686'"
+ "BB_HASHSERVE = 'typhoon.yocto.io:8686'",
+ "EXTRA_OEMAKE = ' --debug=j'"
]
},
"templates" : {
--
2.31.1


[yocto-autobuilder-helper][dunfell][PATCH] config.json: track system load with PARALLEL_MAKE

Trevor Gamblin
 

Backport for dunfell since we have been running for ~1 week with this
patch in master and the same load limiting may help reduce build
failures for dunfell itself as well as when its builds are running
alongside other branches.

This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 50 (the previous iteration of this patch both
had it set to 100 and mistakenly stated that this was a percentage,
rather than an absolute representation of system load ala top/htop).
This option is supported by both Make and Ninja. However, we also
require the "--debug=j" option to be passed to Make in order for the
latter to report perceived system load in the do_compile logs, (which we
need in order to track test values), and since this option is not
supported by Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we
can determine if the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
config.json | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index 3cff363..8323864 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 50'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
@@ -56,7 +56,8 @@
"SDK_EXT_TYPE = 'minimal'",
"SDK_INCLUDE_TOOLCHAIN = '1'",
"BB_DISKMON_DIRS = 'STOPTASKS,${TMPDIR},1G,100K STOPTASKS,${DL_DIR},1G STOPTASKS,${SSTATE_DIR},1G STOPTASKS,/tmp,100M,100K ABORT,${TMPDIR},100M,1K ABORT,${DL_DIR},100M ABORT,${SSTATE_DIR},100M ABORT,/tmp,10M,1K'",
- "BB_HASHSERVE = 'typhoon.yocto.io:8686'"
+ "BB_HASHSERVE = 'typhoon.yocto.io:8686'",
+ "EXTRA_OEMAKE = ' --debug=j'"
]
},
"templates" : {
--
2.31.1


[meta-spdxscanner][PATCH] MAINTAINERS & README.md: Updated the email address where send patches

Marco Cavallini
 

From: Marco Cavallini <m.cavallini@...>

Was yocto@... and now is yocto@...

Signed-off-by: Marco Cavallini <koansoftware@...>
---
MAINTAINERS | 2 +-
README.md | 4 ++++
2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 23761bf..ec34c24 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1,7 +1,7 @@
This file contains a list of maintainers for the meta-spdxscanner layer.

Please submit any patches against meta-spdxscanner to the Yocto Project
-mailing list (yocto@...) with'[meta-spdxscanner]' in the
+mailing list (yocto@...) with'[meta-spdxscanner]' in the
subject.

You may also contact the maintainers directly.
diff --git a/README.md b/README.md
index eb0c389..62b59f5 100644
--- a/README.md
+++ b/README.md
@@ -69,3 +69,7 @@ Note
- scancode-tk has to install on host development under python2. So, please make sure that python2 exists on your build server.
- If you want to use scancode.bbclass, There is no need to install anything on your host.

+---
+
+Please submit any patches against meta-spdxscanner to the Yocto Project mailing list (yocto@...)
+with'[meta-spdxscanner]' in the subject.
\ No newline at end of file
--
2.17.1


[meta-spdxscanner][PATCH] fossology-rest.bbclass: Moved check if spdx already exists out of the task

Marco Cavallini
 

From: Marco Cavallini <m.cavallini@...>

With the previous code was impossible to run the do_get_report
task whenever a correspondent .spdx file was already present
in tmp/deploy/spdx.
Now the check if spdx already exists is at a higher level
therefore is possible to run : bitbake -c get_report

Signed-off-by: Marco Cavallini <koansoftware@...>
---
classes/fossology-rest.bbclass | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/classes/fossology-rest.bbclass b/classes/fossology-rest.bbclass
index 3694440..e55e859 100644
--- a/classes/fossology-rest.bbclass
+++ b/classes/fossology-rest.bbclass
@@ -84,6 +84,15 @@ python () {
create_manifest(info,sstatefile)
return

+ # if spdx already exists
+ if os.path.exists(info['outfile']):
+ bb.note(info['pn'] + "spdx file has been exist, do nothing")
+ return
+ if os.path.exists( sstatefile ):
+ bb.note(info['pn'] + "spdx file has been exist, do nothing")
+ create_manifest(info,sstatefile)
+ return
+
d.appendVarFlag('do_spdx', 'depends', ' %s:do_get_report' % pn)
bb.build.addtask('do_get_report', 'do_configure', 'do_patch' , d)
bb.build.addtask('do_spdx', 'do_configure', 'do_get_report', d)
@@ -157,15 +166,6 @@ python do_get_report () {
info['outfile'] = os.path.join(manifest_dir, info['pn'] + "-" + info['pv'] + ".spdx" )
sstatefile = os.path.join(spdx_outdir, info['pn'] + "-" + info['pv'] + ".spdx" )

- # if spdx has been exist
- if os.path.exists(info['outfile']):
- bb.note(info['pn'] + "spdx file has been exist, do nothing")
- return
- if os.path.exists( sstatefile ):
- bb.note(info['pn'] + "spdx file has been exist, do nothing")
- create_manifest(info,sstatefile)
- return
-
spdx_get_src(d)

bb.note('SPDX: Archiving the patched source...')
--
2.17.1


Re: [ptest-runner 0/5] ptest: Various memory fixes and enhancements

?ukasz Majewski
 

Hi Randy,

On 2021-07-21 5:46 a.m., ?ukasz Majewski wrote:
This patch series provides some memory management fixes and
corrected return code handling for ptest-runner2.

Adrian Freihofer (4):
git: Extend the gitignore
mem: Fix memleak for ptest_opts
mem: Simplify memory management
mem: Refactor ptest_list cleanup

Lukasz Majewski (1):
main: Do not return number of failed tests when calling
ptest-runner

.gitignore | 3 +++
main.c | 33 ++++++++++++++++++++++++++++-----
ptest_list.c | 13 ++++---------
ptest_list.h | 8 +-------
tests/ptest_list.c | 13 +++++++------
tests/utils.c | 22 +++++++++++-----------
utils.c | 11 ++++-------
7 files changed, 58 insertions(+), 45 deletions(-)
create mode 100644 .gitignore




Looks good to me.

Did you happen to check the compile with clang as well?
If not, I may do that eventually.
No, I've just used gcc.


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@...


Re: [qa-build-notification] QA notification for completed autobuilder build (yocto-3.4_M2.rc1)

Sangeeta Jain
 

Hi all,

Intel and WR YP QA is planning for QA execution for YP build yocto-3.4_M2.rc1. We are planning to execute following tests for this cycle:

OEQA-manual tests for following module:
1. OE-Core
2. BSP-hw

Runtime auto test for following platforms:
1. MinnowTurbot 32-bit
2. Coffee Lake
3. NUC 7
4. NUC 6
5. Edgerouter
6. Beaglebone

ETA for completion is next Monday, July 26.

Thanks,
Sangeeta

-----Original Message-----
From: qa-build-notification@... <qa-build-
notification@...> On Behalf Of Richard Purdie
Sent: Wednesday, 21 July, 2021 5:22 PM
To: <yocto@...> <yocto@...>
Cc: qa-build-notification <qa-build-notification@...>
Subject: [qa-build-notification] QA notification for completed autobuilder build
(yocto-3.4_M2.rc1)

A build flagged for QA (yocto-3.4_M2.rc1) was completed on the autobuilder
and is available at:


https://autobuilder.yocto.io/pub/releases/yocto-3.4_M2.rc1


Build hash information:

bitbake: 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa
meta-arm: 9efa3b5683a5dd7ebbf63ff31b889ed589ff7a9a
meta-gplv2: b3fa26bc777ec0136ce189d90123b50f6ee567b9
meta-intel: 94c097a82cf9167fa567c2af80d5e39a3fbbc11f
meta-mingw: 8f3b6e3772879dc2caec8fe249ce277fbb1aa55f
oecore: c4fc226c2e3856b942bb4f57ead21a64c3dc8c0d
poky: 1ad79313a5c3e6a453fbeb44caac5c5bbad46d3c


[RP sending manually as the original failed to get through] This is an automated
message from the Yocto Project Autobuilder
Git: git://git.yoctoproject.org/yocto-autobuilder2
Email: richard.purdie@...






Re: [ptest-runner 0/5] ptest: Various memory fixes and enhancements

Randy MacLeod
 

On 2021-07-21 5:46 a.m., ?ukasz Majewski wrote:
This patch series provides some memory management fixes and corrected
return code handling for ptest-runner2.
Adrian Freihofer (4):
git: Extend the gitignore
mem: Fix memleak for ptest_opts
mem: Simplify memory management
mem: Refactor ptest_list cleanup
Lukasz Majewski (1):
main: Do not return number of failed tests when calling ptest-runner
.gitignore | 3 +++
main.c | 33 ++++++++++++++++++++++++++++-----
ptest_list.c | 13 ++++---------
ptest_list.h | 8 +-------
tests/ptest_list.c | 13 +++++++------
tests/utils.c | 22 +++++++++++-----------
utils.c | 11 ++++-------
7 files changed, 58 insertions(+), 45 deletions(-)
create mode 100644 .gitignore
Looks good to me.

Did you happen to check the compile with clang as well?
If not, I may do that eventually.

--
# Randy MacLeod
# Wind River Linux


Re: How to pass arguments to a shell function from python task bb.build.exec_func ? #bitbake #yocto

Joshua Watt
 

AFAIK, passing arguments from a python function to a shell function is not allowed

On 7/21/21 10:45 AM, Bel Hadj Salem Talel wrote:

Hi All,

I have this example that can call a shell function from a python task:

shell_function(){
    bbwarn "This is a shell function, arg1 = $1"
}

python do_something(){
   bb.build.exec_func('shell_function', d)
}

How to pass arguments to the shell function ?

I tried this : "bb.build.exec_func('shell_function 1', d)" but it fails with the error: "shell_function 1" not found.

Thanks,
Talel



meta-security layer

Michael Nazzareno Trimarchi
 

Hi Armin

Using the meta-integrety layer and I'm able to deploy the image but I
don't have the attribute insde the ext4 and wic image I create.
Do you know what step are missing?

Michael


How to pass arguments to a shell function from python task bb.build.exec_func ? #bitbake #yocto

Bel Hadj Salem Talel <bhstalel@...>
 

Hi All,

I have this example that can call a shell function from a python task:

shell_function(){
    bbwarn "This is a shell function, arg1 = $1"
}

python do_something(){
   bb.build.exec_func('shell_function', d)
}

How to pass arguments to the shell function ?

I tried this : "bb.build.exec_func('shell_function 1', d)" but it fails with the error: "shell_function 1" not found.

Thanks,
Talel


Yocto Technical Team Minutes, Engineering Sync, for July 20, 2021

Trevor Woerner
 

Yocto Technical Team Minutes, Engineering Sync, for July 20, 2021
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner
Stephen Jolley
Armin Kuster
Jan-Simon Möller
Joshua Watt
Steve Sakoman
Trevor Gamblin
Scott Murray
Randy MacLeod
Richard Purdie
Martin Jansa (JaMa)
Michael Halstead
Bruce Ashfiled
Tony Tascioglu
Ross Burton
Alexandre Belloni
Paul Barker
Denys Dmytriyenko

== project status ==
- -m2 to be built this week (after pzstd included in buildtools tarball)
- 3.3.2 (hardknott) to QA after -m2 build
- proposal to add lz4c, zstd, and pzstd as required host commands
- architectural discussion on a change to the override syntax
- prserv still waiting on python asyncio issues
- AB-INT issues around ptest
- simpler test cases still required for multiconfig to resolve issues

== discussion ==
RP: syntax changes. we’re limited by not knowing definitively what and
what isn’t an override. leads to lots of caching and extra resources.
it would help speed things up. there are issues around backwards
compatibility, but it shouldn’t be a problem. the new character was
illegal previously (no clash). so with the backwards compatibility solved
it should be clear sailing (famous last words)
ScottM: sgtm, i like the syntax
RP: “feels” right, and it helps massively inside bitbake. how old would we
have to go back
ScottM: dunfell
RP: so if we backport this to dunfell then AGL should be helped
ScottM + J-SM: yes. we’d have to wait for a bunch of BSPs to adopt too
ScottM: can we mix and match old and new syntax?
RP: yes
ScottM: then we should be okay. AGL will be a good test case
JPEW: mix and match now, then make it required in the future?
RP: discussion ongoing
TrevorW: we started discussions on override syntax at OEDEM 2015, glad to see
it coming in
RP: my instinct is to bring it in quickly rather than have a long overlap. it
is used a lot in image filesystem code, also in tune features, qemu. the
package code relies heavily on the override syntax (package data). the tsc
generally thinks we should move forward. how long to wait before moving
forward?
AlexB: not too long. definitely before next lts and a little before
RP: which means now is a good time
PaulB: great idea. how about -m2? before or after?
RP: wanted -m2 last week, need new buildtools tarball, tempted to do it before
-m2
PaulB: ship it! (lol)
JPEW: at this point we’re just talking about converting the metadata?
RP: yes.
RP: tempted to do it for -m2.
RP: steve?
SS: sounds good, good idea to backport
RP: see discussion on oe-architecture

RP: Ross is proposing a function for bitbake-utils (layer_path()) for a layers
path to be extracted. why not a bblayersdir variable. core has this, sort
of. but maybe other layers could use this too. if you start encoding
paths like this, they get into the signatures (sigs depend on locations)
so path has to be in hash whitelisting variable. but path ends up in
signature file (for debugging). maybe have “add layer” like “add
task” that takes care of this. patches are on mailing list. we could
drop bbcollections variable that would be nice to drop (historic, predates
layers). it’d be nice to simplify some of the variables used in bitbake.
we force the priority to 5 (for no reason)
JPEW: priority levels has always been confusing
RP: this could make this go away
Denys: and there’s recipe priority
RP: and preferred, etc…
JPEW: and recipe priority only holds inside a layer
RP: one of the big issues with python2 → python3 was this sort of thing
JPEW: this could be easily backwards compatible?
RP: yes
PaulB: sounds like the right direction. layer.conf has always been confusing
(copy this boilerplate and it’ll work, don’t ask how) moving this into
bitbake would make it easier for users
Ross: i like getting rid of layer priority since this is a cause of lots
of issues since it works one day then stops a while later (after
bblayers.conf changes)
RP: behind the scenes bitbake does ugly stuff expanding layer dirs etc. so
this would get rid of that ugly code
ScottM: also backported to dunfell?
RP: more nervous backporting something like this
JPEW: probably wouldn’t be able to backport forcing the priority
PaulB: it’d be nice to have a branch of a layer support dunfell, hardknott,
and master all at the same time (without changes), but it probably won’t
be too hard
JPEW: yes, we do that too, and we go back a lot further. it’s possible
RP: i don’t want to block Ross’s patch and i think this would be good

ScottM: read-only prserv. moving the syncio loop creation. ran oe-selftest a
couple times. i don’t think it’s helping, still seeing failures with
the patches (and not without). i’ll need to double-check to make sure
these aren’t intermittent issues. do we want this feature bad enough
that doing it without the unification of the async-rpc code with the hash
equivalency is required? could we push that out to next release and just
do more minor tweaks for read-only mode for this release?
RP: i’m now worried about hash-equiv server as well. we run one on the
server but i think there’s gremlins there
JPEW: ScottM ping me on it, i think it should work, i’d like to help
ScottM: yea, i think it should work too. i think there’s a race in the
shutdown code. with enough load i think we’re running into something.
but i think it’s in the shutdown and solvable. i’m pretty sure i can
run enough load locally to mimic the behaviour of the AB.
RP: when i originally worked on this and we scaled AB, i had to do crazy load
tests. AB finds these failures
ScottM: was seeing loadavg of 800-900 on my local machine. it looked like
bitbake had shutdown during the build at one point. so it seems to match
the AB failures
PaulB: sounds like the failures i was seeing
ScottM: unfortunately the server tests don’t ever see these. they have to be
run on the whole build with a loaded build machine. so i think we should
do some for -m2 and then work on the rest for future releases/builds
RP: i’m not up-to-date on the asyncio stuff, but i've analyzed lots of
failures with bitbake shutdowns so i think i can help too
PaulB: i’ve moved off, but i’m happy to review patches or help brainstorm

ScottM: would it be possible to do a build on the AB with the auto-start
hash-equiv to see how that goes?
RP: if we put something into helper then we could try that
ScottM: we would lose some performance
JPEW: it would be a “throw-away” build
ScottM: okay, i’d like to try it to see how that would work
RP: AlexB can you set that up?
AlexB: sure
ScottB: bbhashserv = auto
RP: okay, do that and let scott know which build it is that’s got that
setting

ScottM: elc/yps?
TrevorW: working on it. we’ll do something, most likely a copy of the last
one but probably 3-days

PaulB: anyone looked at Deny’s email? (adding 5.10 kernel to dunfell)
ScottM: we’d really like to do it for AGL
J-SM: many customers have already done it so it’d be nice to do it in one place
Denys: kernel has a new lts every year, and dunfell is over a year old. this
is something coming from YP members requests. thanks for reviews. the next
yocto lts would have the same issue
PaulB: reminds me of hardware enablement kernels from ubuntu, in other words,
seems common among linux distros with long-term lts releases that last
longer than 1 year (i.e. kernel lts time frames)
Randy: sounds like this is a good and requested thing. are there any
downsides? effects on userspace?
Denys: userspace would be 5.4 but kernel would be 5.10. if we switch userspace
to 5.10 then everything has to rebuild (toolchain, glibc, which leads to
everything rebuilding)
RP: i think the goal is to not change the toolchain. the impact shouldn’t be
on SS (libc-headers causes toolchain updates). thanks Denys for getting
the ball rolling
Denys: i’d like to move it away from github and bring it under
git.yoctoproject.org
PaulB: the repos on git.yp.org are organized by headings, so maybe have a
heading for dunfell mixin layers
RP: sounds like something the tsc should think about
Denys: anyone want to do a mixin layer for gcc? gcc10 for dunfell? (i.e. a
newer toolchain)
<crickets>
Randy: sounds like a “no”. what’s the motivation?
Denys: some vendors would like to stay on dunfell (less work than moving to
master) but also test and build against newer tools
PaulB: newer toolchains always generate new warnings, errors… which leads to
lots of patches required everywhere. lots of effort to update to a newer
toolchain
Denys: true. like with all the -fcommon stuff
Randy: so vendors would have to wait for next lts and/or work towards master
PaulB: if dunfell gets extended for another 2 years then a newer u-boot mixin
layer would be useful (and we might as well make that public). but only if
dunfell is extended
Denys: to be clear: my understanding is that all these individual updates
should be done with individual mixin layers, not all done in one layer?
PaulB: yes, kernel, u-boot, are relatively self-contained, so should be able
to do as separate mixin layers

SS: any process/timeline for making that decision (dunfell extended?)
RP: comes down to funding. we’ve asked the members, but they haven’t
decided yet. i’m advocating for it and would like to see it


[ptest-runner 5/5] main: Do not return number of failed tests when calling ptest-runner

?ukasz Majewski
 

Up till now ptest-runner2 returns number of failed tests with its
exit status code. Such use case is not recommended [1] and may cause
issues when there are more than 256 tests to be executed.

To alleviate this issue the number of total tests with number of failed
ones is printed before exit. To be more specific - failure of a single
test doesn't indicate that the ptest-runner itself encounter any issue
during its execution.

One can test this change with executing:
./ptest-runner -d tests/data fail

Links:
[1] - https://www.gnu.org/software/libc/manual/html_node/Exit-Status.html

Signed-off-by: Lukasz Majewski <lukma@...>
---
main.c | 3 +++
1 file changed, 3 insertions(+)

diff --git a/main.c b/main.c
index efa21b2..9f03857 100644
--- a/main.c
+++ b/main.c
@@ -219,6 +219,9 @@ main(int argc, char *argv[])
ptest_list_remove(run, opts.exclude[i], 1);
=20
rc =3D run_ptests(run, opts, argv[0], stdout, stderr);
+ fprintf(stdout, "TOTAL: %d FAIL: %d\n", ptest_list_length(run), rc);
+ if (rc > 0)
+ rc =3D 0;
=20
ptest_list_free_all(&run);
=20
--=20
2.20.1


[ptest-runner 4/5] mem: Refactor ptest_list cleanup

?ukasz Majewski
 

From: Adrian Freihofer <adrian.freihofer@...>

Try to make memory management more robust by assigning always NULL to
struct ptest_list pointers. It's a refactoring which probably improves
the code but does not fix a concrete bug.

Signed-off-by: Adrian Freihofer <adrian.freihofer@...>
---
main.c | 9 +++++----
ptest_list.c | 13 ++++---------
ptest_list.h | 8 +-------
tests/ptest_list.c | 13 +++++++------
tests/utils.c | 22 +++++++++++-----------
utils.c | 6 +++---
6 files changed, 31 insertions(+), 40 deletions(-)

diff --git a/main.c b/main.c
index e73626c..efa21b2 100644
--- a/main.c
+++ b/main.c
@@ -116,7 +116,8 @@ main(int argc, char *argv[])
mtrace();
#endif
=20
- struct ptest_list *head, *run;
+ struct ptest_list *head =3D NULL;
+ struct ptest_list *run =3D NULL;
__attribute__ ((__cleanup__(cleanup_ptest_opts))) struct ptest_options =
opts;
=20
opts.dirs =3D malloc(sizeof(char **) * 1);
@@ -175,7 +176,7 @@ main(int argc, char *argv[])
=20
head =3D NULL;
for (i =3D 0; i < opts.dirs_no; i ++) {
- struct ptest_list *tmp;
+ struct ptest_list *tmp =3D NULL;
=20
tmp =3D get_available_ptests(opts.dirs[i], opts.timeout);
if (tmp =3D=3D NULL) {
@@ -211,7 +212,7 @@ main(int argc, char *argv[])
=20
run =3D filter_ptests(head, opts.ptests, ptest_num);
CHECK_ALLOCATION(run, (size_t) ptest_num, 1);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
=20
for (i =3D 0; i < ptest_exclude_num; i++)
@@ -219,7 +220,7 @@ main(int argc, char *argv[])
=20
rc =3D run_ptests(run, opts, argv[0], stdout, stderr);
=20
- ptest_list_free_all(run);
+ ptest_list_free_all(&run);
=20
return rc;
}
diff --git a/ptest_list.c b/ptest_list.c
index b689670..87b8c71 100644
--- a/ptest_list.c
+++ b/ptest_list.c
@@ -69,24 +69,19 @@ ptest_list_free(struct ptest_list *p)
free(p);
}
=20
-int
-ptest_list_free_all(struct ptest_list *head)
+void
+ptest_list_free_all(struct ptest_list **head)
{
- int i =3D 0;
struct ptest_list *p, *q;
=20
- VALIDATE_PTR_RINT(head);
-
- p =3D head;
+ p =3D *head;
while (p !=3D NULL) {
q =3D p;
p =3D p->next;
=20
ptest_list_free(q);
- i++;
}
-
- return i;
+ *head =3D NULL;
}
=20
int
diff --git a/ptest_list.h b/ptest_list.h
index e583d9f..949250c 100644
--- a/ptest_list.h
+++ b/ptest_list.h
@@ -36,12 +36,6 @@
x =3D NULL; \
} while (0)
=20
-#define PTEST_LIST_FREE_ALL_CLEAN(x) \
- do { \
- ptest_list_free_all(x); \
- x =3D NULL; \
- } while (0)
-
#define PTEST_LIST_ITERATE_START(head, p) for (p =3D head->next; p !=3D =
NULL; p =3D p->next) {
#define PTEST_LIST_ITERATE_END }
=20
@@ -58,7 +52,7 @@ struct ptest_list {
=20
extern struct ptest_list *ptest_list_alloc(void);
extern void ptest_list_free(struct ptest_list *);
-extern int ptest_list_free_all(struct ptest_list *);
+extern void ptest_list_free_all(struct ptest_list **);
=20
extern int ptest_list_length(struct ptest_list *);
extern struct ptest_list *ptest_list_search(struct ptest_list *, char *)=
;
diff --git a/tests/ptest_list.c b/tests/ptest_list.c
index 37d19ae..6bbc13b 100644
--- a/tests/ptest_list.c
+++ b/tests/ptest_list.c
@@ -53,7 +53,7 @@ START_TEST(test_add)
{
struct ptest_list *head =3D ptest_list_alloc();
ck_assert(ptest_list_add(head, strdup("perl"), NULL, 1) !=3D NULL);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
@@ -62,14 +62,15 @@ START_TEST(test_free_all)
struct ptest_list *head =3D NULL;
int i;
=20
- ck_assert(ptest_list_free_all(head) =3D=3D -1);
+ ptest_list_free_all(&head);
+ ck_assert(head =3D=3D NULL);
ck_assert(errno =3D=3D EINVAL);
=20
head =3D ptest_list_alloc();
for (i =3D 0; i < ptests_num; i++)
ptest_list_add(head, strdup(ptest_names[i]), NULL, 1);
=20
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
@@ -87,7 +88,7 @@ START_TEST(test_length)
ptest_list_add(head, strdup(ptest_names[i]), NULL, 1);
=20
ck_assert_int_eq(ptest_list_length(head), ptests_num);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
@@ -109,7 +110,7 @@ START_TEST(test_search)
for (i =3D ptests_num - 1; i >=3D 0; i--)
ck_assert(ptest_list_search(head, ptest_names[i]) !=3D NULL);
=20
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
@@ -141,7 +142,7 @@ START_TEST(test_remove)
ck_assert_int_eq(ptest_list_length(head), n);
=20
ck_assert(ptest_list_search(head, "busybox") !=3D NULL);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
diff --git a/tests/utils.c b/tests/utils.c
index 8df1b54..4e244fe 100644
--- a/tests/utils.c
+++ b/tests/utils.c
@@ -88,13 +88,13 @@ START_TEST(test_get_available_ptests)
for (i =3D 0; ptests_not_found[i] !=3D NULL; i++)
ck_assert(ptest_list_search(head, ptests_not_found[i]) =3D=3D NULL);
=20
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
START_TEST(test_print_ptests)
{
- struct ptest_list *head;
+ struct ptest_list *head =3D NULL;
=20
char *buf;
size_t size =3D PRINT_PTEST_BUF_SIZE;
@@ -116,14 +116,14 @@ START_TEST(test_print_ptests)
=20
head =3D ptest_list_alloc();
ck_assert(print_ptests(head, fp) =3D=3D 1);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
line =3D fgets(line_buf, PRINT_PTEST_BUF_SIZE, fp);
ck_assert(line !=3D NULL);
ck_assert(strcmp(line, PRINT_PTESTS_NOT_FOUND) =3D=3D 0);
=20
head =3D get_available_ptests(opts_directory, 1);
ck_assert(print_ptests(head, fp) =3D=3D 0);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
line =3D fgets(line_buf, PRINT_PTEST_BUF_SIZE, fp);
ck_assert(line !=3D NULL);
ck_assert(strcmp(line, PRINT_PTESTS_AVAILABLE) =3D=3D 0);
@@ -144,7 +144,7 @@ END_TEST
START_TEST(test_filter_ptests)
{
struct ptest_list *head =3D get_available_ptests(opts_directory, 1);
- struct ptest_list *head_new;
+ struct ptest_list *head_new =3D NULL;
char *ptest_not_exists[] =3D {
"glib",
};
@@ -161,8 +161,8 @@ START_TEST(test_filter_ptests)
ck_assert(head_new !=3D NULL);
ck_assert(ptest_list_length(head_new) =3D=3D 3);
=20
- ptest_list_free_all(head);
- ptest_list_free_all(head_new);
+ ptest_list_free_all(&head);
+ ptest_list_free_all(&head_new);
}
END_TEST
=20
@@ -191,7 +191,7 @@ START_TEST(test_run_ptests)
=20
rc =3D run_ptests(head, opts, "test_run_ptests", fp_stdout, fp_stderr);
ck_assert(rc =3D=3D 0);
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
=20
fclose(fp_stdout);
free(buf_stdout);
@@ -227,7 +227,7 @@ START_TEST(test_run_timeout_duration_ptest)
=20
test_ptest_expected_failure(head, timeout, "hang", search_for_timeout_a=
nd_duration);
=20
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
@@ -255,7 +255,7 @@ START_TEST(test_run_fail_ptest)
=20
test_ptest_expected_failure(head, timeout, "fail", search_for_fail);
=20
- ptest_list_free_all(head);
+ ptest_list_free_all(&head);
}
END_TEST
=20
@@ -354,7 +354,7 @@ test_ptest_expected_failure(struct ptest_list *head, =
const int timeout, char *pr
fp_stdout
);
=20
- PTEST_LIST_FREE_ALL_CLEAN(filtered);
+ ptest_list_free_all(&filtered);
}
=20
fclose(fp_stdout);
diff --git a/utils.c b/utils.c
index a23679a..4737bcd 100644
--- a/utils.c
+++ b/utils.c
@@ -110,7 +110,7 @@ get_ptest_file(char **ptest_file, struct stat *st_buf=
, const char *main_dir,
struct ptest_list *
get_available_ptests(const char *dir, int global_timeout)
{
- struct ptest_list *head;
+ struct ptest_list *head =3D NULL;
struct stat st_buf;
=20
int n, i;
@@ -212,7 +212,7 @@ get_available_ptests(const char *dir, int global_time=
out)
free(namelist);
=20
if (fail) {
- PTEST_LIST_FREE_ALL_CLEAN(head);
+ ptest_list_free_all(&head);
errno =3D saved_errno;
break;
}
@@ -282,7 +282,7 @@ filter_ptests(struct ptest_list *head, char **ptests,=
int ptest_num)
}
=20
if (fail) {
- PTEST_LIST_FREE_ALL_CLEAN(head_new);
+ ptest_list_free_all(&head_new);
errno =3D saved_errno;
}=20
} while (0);
--=20
2.20.1


[ptest-runner 3/5] mem: Simplify memory management

?ukasz Majewski
 

From: Adrian Freihofer <adrian.freihofer@...>

Removes the following warnings thrown by
make && valgrind -s --leak-check=3Dfull ./ptest-runner -d tests/data2

=3D=3D4154390=3D=3D Invalid write of size 8
=3D=3D4154390=3D=3D at 0x40360D: run_child (utils.c:357)
=3D=3D4154390=3D=3D by 0x403C5B: run_ptests (utils.c:534)
=3D=3D4154390=3D=3D by 0x402C4D: main (main.c:223)
=3D=3D4154390=3D=3D Address 0x4a66440 is 0 bytes inside a block of size =
2 alloc'd
=3D=3D4154390=3D=3D at 0x4839809: malloc (vg_replace_malloc.c:307)
=3D=3D4154390=3D=3D by 0x4035E4: run_child (utils.c:354)
=3D=3D4154390=3D=3D by 0x403C5B: run_ptests (utils.c:534)
=3D=3D4154390=3D=3D by 0x402C4D: main (main.c:223)
=3D=3D4154390=3D=3D
=3D=3D4154390=3D=3D Invalid write of size 8
=3D=3D4154390=3D=3D at 0x403618: run_child (utils.c:358)
=3D=3D4154390=3D=3D by 0x403C5B: run_ptests (utils.c:534)
=3D=3D4154390=3D=3D by 0x402C4D: main (main.c:223)
=3D=3D4154390=3D=3D Address 0x4a66448 is 6 bytes after a block of size 2=
alloc'd
=3D=3D4154390=3D=3D at 0x4839809: malloc (vg_replace_malloc.c:307)
=3D=3D4154390=3D=3D by 0x4035E4: run_child (utils.c:354)
=3D=3D4154390=3D=3D by 0x403C5B: run_ptests (utils.c:534)
=3D=3D4154390=3D=3D by 0x402C4D: main (main.c:223)
=3D=3D4154390=3D=3D
=3D=3D4154390=3D=3D Syscall param execve(argv) points to unaddressable by=
te(s)
=3D=3D4154390=3D=3D at 0x4955C2B: execve (in /usr/lib64/libc-2.32.so)
=3D=3D4154390=3D=3D by 0x40365E: run_child (utils.c:368)
=3D=3D4154390=3D=3D by 0x403C5B: run_ptests (utils.c:534)
=3D=3D4154390=3D=3D by 0x402C4D: main (main.c:223)
=3D=3D4154390=3D=3D Address 0x4a66442 is 0 bytes after a block of size 2=
alloc'd
=3D=3D4154390=3D=3D at 0x4839809: malloc (vg_replace_malloc.c:307)
=3D=3D4154390=3D=3D by 0x4035E4: run_child (utils.c:354)
=3D=3D4154390=3D=3D by 0x403C5B: run_ptests (utils.c:534)
=3D=3D4154390=3D=3D by 0x402C4D: main (main.c:223)

Signed-off-by: Adrian Freihofer <adrian.freihofer@...>
---
utils.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/utils.c b/utils.c
index bce9808..a23679a 100644
--- a/utils.c
+++ b/utils.c
@@ -351,12 +351,9 @@ read_child(void *arg)
static inline void
run_child(char *run_ptest, int fd_stdout, int fd_stderr)
{
- char **argv =3D malloc(sizeof(char) * 2);
+ char *const argv[2] =3D {run_ptest, NULL};
chdir(dirname(strdup(run_ptest)));
=20
- argv[0] =3D run_ptest;
- argv[1] =3D NULL;
-
dup2(fd_stdout, STDOUT_FILENO);
// XXX: Redirect stderr to stdout to avoid buffer ordering problems.
dup2(fd_stdout, STDERR_FILENO);
--=20
2.20.1


[ptest-runner 2/5] mem: Fix memleak for ptest_opts

?ukasz Majewski
 

From: Adrian Freihofer <adrian.freihofer@...>

make && valgrind -s --leak-check=3Dfull ./ptest-runner -d tests/data2

=3D=3D4154029=3D=3D HEAP SUMMARY:
=3D=3D4154029=3D=3D in use at exit: 20 bytes in 2 blocks
=3D=3D4154029=3D=3D total heap usage: 45 allocs, 43 frees, 42,909 bytes=
allocated
=3D=3D4154029=3D=3D
=3D=3D4154029=3D=3D 20 (8 direct, 12 indirect) bytes in 1 blocks are defi=
nitely lost in loss record 2 of 2
=3D=3D4154029=3D=3D at 0x4839809: malloc (vg_replace_malloc.c:307)
=3D=3D4154029=3D=3D by 0x40252D: str2array (main.c:70)
=3D=3D4154029=3D=3D by 0x402768: main (main.c:119)
=3D=3D4154029=3D=3D
=3D=3D4154029=3D=3D LEAK SUMMARY:
=3D=3D4154029=3D=3D definitely lost: 8 bytes in 1 blocks
=3D=3D4154029=3D=3D indirectly lost: 12 bytes in 1 blocks
=3D=3D4154029=3D=3D possibly lost: 0 bytes in 0 blocks
=3D=3D4154029=3D=3D still reachable: 0 bytes in 0 blocks
=3D=3D4154029=3D=3D suppressed: 0 bytes in 0 blocks
=3D=3D4154029=3D=3D
=3D=3D4154029=3D=3D ERROR SUMMARY: 1 errors from 1 contexts (suppressed: =
0 from 0)

With this patch valgrind reports 0 errors.

Signed-off-by: Adrian Freihofer <adrian.freihofer@...>
---
main.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/main.c b/main.c
index 467548e..e73626c 100644
--- a/main.c
+++ b/main.c
@@ -84,6 +84,25 @@ str2array(char *str, const char *delim, int *num)
return array;
}
=20
+void cleanup_ptest_opts(struct ptest_options *opts)
+{
+ for (int i=3D0; i < opts->dirs_no; i++)
+ free(opts->dirs[i]);
+
+ free(opts->dirs);
+ opts->dirs =3D NULL;
+
+ if (opts->ptests) {
+ free(opts->ptests);
+ opts->ptests =3D NULL;
+ }
+
+ if (opts->xml_filename) {
+ free(opts->xml_filename);
+ opts->xml_filename =3D NULL;
+ }
+}
+
int
main(int argc, char *argv[])
{
@@ -98,7 +117,7 @@ main(int argc, char *argv[])
#endif
=20
struct ptest_list *head, *run;
- struct ptest_options opts;
+ __attribute__ ((__cleanup__(cleanup_ptest_opts))) struct ptest_options =
opts;
=20
opts.dirs =3D malloc(sizeof(char **) * 1);
CHECK_ALLOCATION(opts.dirs, 1, 1);
--=20
2.20.1


[ptest-runner 1/5] git: Extend the gitignore

?ukasz Majewski
 

From: Adrian Freihofer <adrian.freihofer@...>

Signed-off-by: Adrian Freihofer <adrian.freihofer@...>
---
.gitignore | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 .gitignore

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..ef07e6a
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+*.o
+ptest-runner
+ptest-runner-test
--=20
2.20.1