Date   

Re: [PATCH] perf: reproducibility fixes for pmu-events.c

Bruce Ashfield
 

Ignore this.

I reused an old git-send-email and sent it to the wrong list. I've
re-send to oe-core :D

Bruce

On Thu, Mar 11, 2021 at 8:56 AM Bruce Ashfield via
lists.yoctoproject.org
<bruce.ashfield=gmail.com@lists.yoctoproject.org> wrote:


From: Bruce Ashfield <bruce.ashfield@gmail.com>

perf generates pmu-events.c as part of the build process. The
code that generates the events is doing tree walks and potentially
other non-determinstic things.

We'd rather not mess with that implementation, so we add a script
that knows how to read the pmu-events.c, sort the entries and then
copy it over the generated one.

With this, we should always have events in the same order, improving
reproducibility.

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
---
meta/recipes-kernel/perf/perf.bb | 16 ++++
.../perf/perf/sort-pmuevents.py | 93 +++++++++++++++++++
2 files changed, 109 insertions(+)
create mode 100755 meta/recipes-kernel/perf/perf/sort-pmuevents.py

diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 2beb404c03..c7653e523c 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -250,6 +250,14 @@ do_configure_prepend () {
# all the calls have YFLAGS, which contains prefix mapping information.
sed -i -e 's,$(YACC),$(YACC) $(YFLAGS),g' ${S}/scripts/Makefile.host
fi
+ if [ -e "${S}/tools/perf/pmu-events/Build" ]; then
+ target='$(OUTPUT)pmu-events/pmu-events.c $(V)'
+ replacement1='$(OUTPUT)pmu-events/pmu-events.c $(V)\n'
+ replacement2='\t$(srctree)/sort-pmuevents.py $(OUTPUT)pmu-events/pmu-events.c $(OUTPUT)pmu-events/pmu-events.c.new\n'
+ replacement3='\tcp $(OUTPUT)pmu-events/pmu-events.c.new $(OUTPUT)pmu-events/pmu-events.c'
+ sed -i -e "s,$target,$replacement1$replacement2$replacement3,g" \
+ "${S}/tools/perf/pmu-events/Build"
+ fi
# end reproducibility substitutions

# We need to ensure the --sysroot option in CC is preserved
@@ -292,6 +300,14 @@ do_configure_prepend () {
# so we copy it from the sysroot unistd.h to the perf unistd.h
install -D -m0644 ${STAGING_INCDIR}/asm-generic/unistd.h ${S}/tools/include/uapi/asm-generic/unistd.h
install -D -m0644 ${STAGING_INCDIR}/asm-generic/unistd.h ${S}/include/uapi/asm-generic/unistd.h
+
+ # the fetcher is inhibited by the 'inherit kernelsrc', so we do a quick check and
+ # copy for a helper script we need
+ for p in $(echo ${FILESPATH} | tr ':' '\n'); do
+ if [ -e $p/sort-pmuevents.py ]; then
+ cp $p/sort-pmuevents.py ${S}
+ fi
+ done
}

python do_package_prepend() {
diff --git a/meta/recipes-kernel/perf/perf/sort-pmuevents.py b/meta/recipes-kernel/perf/perf/sort-pmuevents.py
new file mode 100755
index 0000000000..5ddf0f144f
--- /dev/null
+++ b/meta/recipes-kernel/perf/perf/sort-pmuevents.py
@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+
+# perf pmu-events sorting tool
+#
+# Copyright (C) 2021 Bruce Ashfield
+#
+# SPDX-License-Identifier: MIT
+#
+
+import sys
+import os
+import re
+from collections import OrderedDict
+
+if len(sys.argv) < 2:
+ print( "[ERROR]: input and output pmu files missing" )
+ sys.exit(1)
+
+if len(sys.argv) < 3:
+ print( "[ERROR]: output pmu file missing" )
+ sys.exit(1)
+
+infile = sys.argv[1]
+outfile = sys.argv[2]
+
+if not os.path.exists(infile):
+ print( "ERROR. input file does not exist: %s" % infile )
+ sys.exit(1)
+
+if os.path.exists(outfile):
+ print( "WARNING. output file will be overwritten: %s" % infile )
+
+with open(infile, 'r') as file:
+ data = file.read()
+
+preamble_regex = re.compile( '^(.*?)^struct', re.MULTILINE | re.DOTALL )
+
+preamble = re.search( preamble_regex, data )
+struct_block_regex = re.compile( '^struct.*?(\w+) (.*?)\[\] = {(.*?)^};', re.MULTILINE | re.DOTALL )
+field_regex = re.compile( '{.*?},', re.MULTILINE | re.DOTALL )
+cpuid_regex = re.compile( '\.cpuid = (.*?),', re.MULTILINE | re.DOTALL )
+name_regex = re.compile( '\.name = (.*?),', re.MULTILINE | re.DOTALL )
+
+# create a dictionary structure to store all the structs, their
+# types and then their fields.
+entry_dict = {}
+for struct in re.findall( struct_block_regex, data ):
+ # print( "struct: %s %s" % (struct[0],struct[1]) )
+ entry_dict[struct[1]] = {}
+ entry_dict[struct[1]]['type'] = struct[0]
+ entry_dict[struct[1]]['fields'] = {}
+ for entry in re.findall( field_regex, struct[2] ):
+ #print( " entry: %s" % entry )
+ cpuid = re.search( cpuid_regex, entry )
+ if cpuid:
+ #print( " cpuid found: %s" % cpuid.group(1) )
+ entry_dict[struct[1]]['fields'][cpuid.group(1)] = entry
+
+ name = re.search( name_regex, entry )
+ if name:
+ #print( " name found: %s" % name.group(1) )
+ entry_dict[struct[1]]['fields'][name.group(1)] = entry
+
+
+# created ordered dictionaries from the captured values. These are ordered by
+# a sorted() iteration of the keys. We don't care about the order we read
+# things, just the sorted order. Hency why we couldn't create these during
+# reading.
+#
+# yes, there's a more concise way to do this, but our nested dictionaries of
+# fields make it complex enough that it becomes unreadable.
+entry_dict_sorted = OrderedDict()
+for i in sorted(entry_dict.keys()):
+ entry_dict_sorted[i] = {}
+ entry_dict_sorted[i]['type'] = entry_dict[i]['type']
+ entry_dict_sorted[i]['fields'] = {}
+ for f in sorted(entry_dict[i]['fields'].keys()):
+ entry_dict_sorted[i]['fields'][f] = entry_dict[i]['fields'][f]
+
+# dump the sorted elements to the outfile
+outf = open( outfile, 'w' )
+
+print( preamble.group(1) )
+outf.write( preamble.group(1) )
+for d in entry_dict_sorted:
+ outf.write( "struct %s %s[] = {\n" % (entry_dict_sorted[d]['type'],d) )
+ for f in entry_dict_sorted[d]['fields']:
+ outf.write( entry_dict_sorted[d]['fields'][f] + '\n' )
+
+ outf.write( "};\n" )
+
+outf.close()
+
--
2.19.1



--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


[PATCH] perf: reproducibility fixes for pmu-events.c

Bruce Ashfield
 

From: Bruce Ashfield <bruce.ashfield@gmail.com>

perf generates pmu-events.c as part of the build process. The
code that generates the events is doing tree walks and potentially
other non-determinstic things.

We'd rather not mess with that implementation, so we add a script
that knows how to read the pmu-events.c, sort the entries and then
copy it over the generated one.

With this, we should always have events in the same order, improving
reproducibility.

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
---
meta/recipes-kernel/perf/perf.bb | 16 ++++
.../perf/perf/sort-pmuevents.py | 93 +++++++++++++++++++
2 files changed, 109 insertions(+)
create mode 100755 meta/recipes-kernel/perf/perf/sort-pmuevents.py

diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 2beb404c03..c7653e523c 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -250,6 +250,14 @@ do_configure_prepend () {
# all the calls have YFLAGS, which contains prefix mapping information.
sed -i -e 's,$(YACC),$(YACC) $(YFLAGS),g' ${S}/scripts/Makefile.host
fi
+ if [ -e "${S}/tools/perf/pmu-events/Build" ]; then
+ target='$(OUTPUT)pmu-events/pmu-events.c $(V)'
+ replacement1='$(OUTPUT)pmu-events/pmu-events.c $(V)\n'
+ replacement2='\t$(srctree)/sort-pmuevents.py $(OUTPUT)pmu-events/pmu-events.c $(OUTPUT)pmu-events/pmu-events.c.new\n'
+ replacement3='\tcp $(OUTPUT)pmu-events/pmu-events.c.new $(OUTPUT)pmu-events/pmu-events.c'
+ sed -i -e "s,$target,$replacement1$replacement2$replacement3,g" \
+ "${S}/tools/perf/pmu-events/Build"
+ fi
# end reproducibility substitutions

# We need to ensure the --sysroot option in CC is preserved
@@ -292,6 +300,14 @@ do_configure_prepend () {
# so we copy it from the sysroot unistd.h to the perf unistd.h
install -D -m0644 ${STAGING_INCDIR}/asm-generic/unistd.h ${S}/tools/include/uapi/asm-generic/unistd.h
install -D -m0644 ${STAGING_INCDIR}/asm-generic/unistd.h ${S}/include/uapi/asm-generic/unistd.h
+
+ # the fetcher is inhibited by the 'inherit kernelsrc', so we do a quick check and
+ # copy for a helper script we need
+ for p in $(echo ${FILESPATH} | tr ':' '\n'); do
+ if [ -e $p/sort-pmuevents.py ]; then
+ cp $p/sort-pmuevents.py ${S}
+ fi
+ done
}

python do_package_prepend() {
diff --git a/meta/recipes-kernel/perf/perf/sort-pmuevents.py b/meta/recipes-kernel/perf/perf/sort-pmuevents.py
new file mode 100755
index 0000000000..5ddf0f144f
--- /dev/null
+++ b/meta/recipes-kernel/perf/perf/sort-pmuevents.py
@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+
+# perf pmu-events sorting tool
+#
+# Copyright (C) 2021 Bruce Ashfield
+#
+# SPDX-License-Identifier: MIT
+#
+
+import sys
+import os
+import re
+from collections import OrderedDict
+
+if len(sys.argv) < 2:
+ print( "[ERROR]: input and output pmu files missing" )
+ sys.exit(1)
+
+if len(sys.argv) < 3:
+ print( "[ERROR]: output pmu file missing" )
+ sys.exit(1)
+
+infile = sys.argv[1]
+outfile = sys.argv[2]
+
+if not os.path.exists(infile):
+ print( "ERROR. input file does not exist: %s" % infile )
+ sys.exit(1)
+
+if os.path.exists(outfile):
+ print( "WARNING. output file will be overwritten: %s" % infile )
+
+with open(infile, 'r') as file:
+ data = file.read()
+
+preamble_regex = re.compile( '^(.*?)^struct', re.MULTILINE | re.DOTALL )
+
+preamble = re.search( preamble_regex, data )
+struct_block_regex = re.compile( '^struct.*?(\w+) (.*?)\[\] = {(.*?)^};', re.MULTILINE | re.DOTALL )
+field_regex = re.compile( '{.*?},', re.MULTILINE | re.DOTALL )
+cpuid_regex = re.compile( '\.cpuid = (.*?),', re.MULTILINE | re.DOTALL )
+name_regex = re.compile( '\.name = (.*?),', re.MULTILINE | re.DOTALL )
+
+# create a dictionary structure to store all the structs, their
+# types and then their fields.
+entry_dict = {}
+for struct in re.findall( struct_block_regex, data ):
+ # print( "struct: %s %s" % (struct[0],struct[1]) )
+ entry_dict[struct[1]] = {}
+ entry_dict[struct[1]]['type'] = struct[0]
+ entry_dict[struct[1]]['fields'] = {}
+ for entry in re.findall( field_regex, struct[2] ):
+ #print( " entry: %s" % entry )
+ cpuid = re.search( cpuid_regex, entry )
+ if cpuid:
+ #print( " cpuid found: %s" % cpuid.group(1) )
+ entry_dict[struct[1]]['fields'][cpuid.group(1)] = entry
+
+ name = re.search( name_regex, entry )
+ if name:
+ #print( " name found: %s" % name.group(1) )
+ entry_dict[struct[1]]['fields'][name.group(1)] = entry
+
+
+# created ordered dictionaries from the captured values. These are ordered by
+# a sorted() iteration of the keys. We don't care about the order we read
+# things, just the sorted order. Hency why we couldn't create these during
+# reading.
+#
+# yes, there's a more concise way to do this, but our nested dictionaries of
+# fields make it complex enough that it becomes unreadable.
+entry_dict_sorted = OrderedDict()
+for i in sorted(entry_dict.keys()):
+ entry_dict_sorted[i] = {}
+ entry_dict_sorted[i]['type'] = entry_dict[i]['type']
+ entry_dict_sorted[i]['fields'] = {}
+ for f in sorted(entry_dict[i]['fields'].keys()):
+ entry_dict_sorted[i]['fields'][f] = entry_dict[i]['fields'][f]
+
+# dump the sorted elements to the outfile
+outf = open( outfile, 'w' )
+
+print( preamble.group(1) )
+outf.write( preamble.group(1) )
+for d in entry_dict_sorted:
+ outf.write( "struct %s %s[] = {\n" % (entry_dict_sorted[d]['type'],d) )
+ for f in entry_dict_sorted[d]['fields']:
+ outf.write( entry_dict_sorted[d]['fields'][f] + '\n' )
+
+ outf.write( "};\n" )
+
+outf.close()
+
--
2.19.1


Want to use nodejs 14 from nodejs 10 in yocto warrior bsp

darshak.raval@...
 

Hi,
I am currently using yocto warrior bsp as my build system.
In warrior, nodejs version 10 is running at sources/meta-openembedded/meta-oe/recipes-devtools
I want to use nodejs 14. I couldn't find yocto recipe for nodejs 14.
Is it possible to upgrade from nodejs 10 to nodejs 14 in yocto warrior ?
Thanks in advance.
Darshak


Failing to patch u-boot .dts

Jonas Vautherin
 

Hello!

I am using u-boot on a pocketbeagle which, according to "Default Device Tree for DT control" in `bitbake -c menuconfig u-boot`, uses am335x-evm, which I understand is the file in ./build/tmp/work/pocketbeagle-poky-linux-gnueabi/u-boot/1_2020.07-r0/git/arch/arm/dts/am335x-evm.dts. 

My issue is that this file ends up setting usb1 as `dr_mode = "host"` and usb0 as `dr_mode = "otg"`. I would like to use fastboot on my pocketbeagle, and therefore set them to `dr_mode = "peripheral"`.

In order to do that, I wrote a .bbappend that does the following:

```
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI += "file://am335x-evm.dts.patch"
SRC_URI += "file://logging.cfg"
```

The patch sets both usb0 and usb1 to `dr_mode = "peripheral"`, while the cfg enables logging in u-boot. Because logging is effectively enabled, I get that my .bbappend is used. And I can confirm that ./build/tmp/work/pocketbeagle-poky-linux-gnueabi/u-boot/1_2020.07-r0/git/arch/arm/dts/am335x-evm.dts is patched indeed.

However, whatever I do, I can't seem to get that modification in my u-boot device tree at runtime, as usb0 always ends up as "otg" and usb1 as "host". I have even tried a full clean build where I removed build/tmp and cache_sstate.

I cannot really tell if my patched file is actually used or not. For instance, if I `bitbake -c cleansstate u-boot`, then `bitbake -c do_patch u-boot`, then `rm -rf ./build/tmp/work/pocketbeagle-poky-linux-gnueabi/u-boot/1_2020.07-r0/git/arch/arm/dts` and finally `bitbake u-boot`, it does not complain at all about a missing dts file.

What am I missing, and how could I make sure that my patched am335x-evm.dts is the device tree being used by my u-boot install?

Best Regards,
Jonas


Re: how often would one use "VAR_someoverride_append = ..."?

Quentin Schulz
 

Hi Robert,

On Thu, Mar 11, 2021 at 04:27:46AM -0500, Robert P. J. Day wrote:
On Wed, 10 Mar 2021, Quentin Schulz wrote:
... snip ...

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n12
for example.

This is an example of a "valid" use case (not that there are invalid
ones) for VAR_foo.

Would probably a better example:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n50

So, you might decide that a machine is so much different than others
that KERNEL_FEATURES should be overridden for said machine.

Then in a bbappend, one might want to add another feature for this
machine, they'll therefore need to use KERNEL_FEATURES_foo_append.

I do not have examples at hand of VAR_foo_append except the ones
Leon sent in another mail. Which should show how rare it is :)
i will ramble for just a bit as i think i settled on a way to
perhaps explain the difference here to some students.

in the overwhelmingly common case, we normally see overrides at the
end of either assignments or appends:

VAR = "snafu"
VAR_qemux86 = "more snafu"
VAR_append_qemux86 = "qemux86_specific_fubar"

it seems that the proper way to interpret the above is that, in the
end, all you would be after is the *final* value of the single
variable VAR, and all the appending and overriding is leading you in
that direction -- whatever "VAR" contains after it's all over is all
you care about.
For that particular example, I'm not entirely sure what the actual value
for VAR would be.

Also not sure what:
VAR = "snafu"
VAR_append = " more"
VAR_qemux86 = "qemu"

would mean for VAR. I don't know exactly when _append is applied to
VAR... after or before the override is applied?

Gut feeling would say before, in which case for non-qemux86 one would
have VAR set to "snafu more" and for qemux86 it'd be "qemu".

I'd anyway use bitbake -e trick to check that, which IMO should really
be given in that section of the docs, to explain how one can debug the
assignment of the variable thanks to the history of the variable :)

on the other hand, with the example from the kernel recipe:

KBRANCH_qemuarm ?= "v5.10/standard/arm-versatile-926ejs"
KBRANCH_qemuarm64 ?= "v5.10/standard/qemuarm64"
KBRANCH_qemumips ?= "v5.10/standard/mti-malta32"
KBRANCH_qemuppc ?= "v5.10/standard/qemuppc"
KBRANCH_qemuriscv64 ?= "v5.10/standard/base"
KBRANCH_qemux86 ?= "v5.10/standard/base"
KBRANCH_qemux86-64 ?= "v5.10/standard/base"
KBRANCH_qemumips64 ?= "v5.10/standard/mti-malta64"

it seems that the right way to interpret the above is that you are,
simultaneously, creating *multiple* versions of the variable KBRANCH,
so that if you subsequently see, for example:

KBRANCH_qemux86_append = " whatever"

you're appending to the variable KBRANCH_qemux86 because the ultimate
result you're after is the *set* of KBRANCH-related variables, perhaps
because you will actually need them all at once.

and given that most situations don't require access to all of those
variations of that variable, that's why the first approach is by far
the most common.

that might be an over-simplification, but it seems to at least give
an idea of why that second approach would be used in very particular
situations.

thoughts?
I like to think that:

KBRANCH_qemux86

is for OVERRIDING, while

VAR_append_qemux86

is for CONDITIONAL appending.

In which case,

KBRANCH_qemux86_append

is UNCONDITIONALLY APPENDING to OVERRIDDEN KBRANCH for qemux86 override.

Only _append and _prepend would be a case of CONDITIONAL
appending/prepending. All others are OVERRIDING.

It's a topic I've long been interested in describing with simple words
but couldn't come with anything yet.

Because you kind of need to explain also the difference between:
KBRANCH_qemux86_class-native
and
KBRANCH_class-native_qemux86

which are completely different.

You also have many, many, many different kinds of overrides, from
pn-recipe, to class-target, virtclass-lib32, task-compile, ${PN},
remove, poky, _X.Y.Z when BBVERSIONS is set, etc...

If you omit _append and _prepend, it's simpler to explain that the
overrides happen from right to left. But once you have _append and
_prepend in the mix, it's a bit confusing and hard to explain.

IMO it would warrant some kind of drawings/schemes to explain step by
step what's happening, couldn't think of a proper one though. Thought of
using multiple FIFOs but couldn't come up with something less confusing
that what it already is :/

I'd be very happy to find a way to explain those (IMO) difficult
notions and make it part of the docs.

Quentin


[meta-security][PATCH v2] samhain: fix compile error on powerpc

kai
 

From: Kai Kang <kai.kang@windriver.com>

It fails to comile samhain for powerpc(qemuppc):

| x_sh_dbIO.c: In function 'swap_short':
| x_sh_dbIO.c:229:36: error: initializer element is not constant
| 229 | static unsigned short ooop = *iptr;
| | ^

Assign after initialization of the static variable to avoid the failure.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
...-initializer-element-is-not-constant.patch | 28 +++++++++++++++++++
recipes-ids/samhain/samhain.inc | 1 +
2 files changed, 29 insertions(+)
create mode 100644 recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch

diff --git a/recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch b/recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch
new file mode 100644
index 0000000..72cb880
--- /dev/null
+++ b/recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch
@@ -0,0 +1,28 @@
+Fix error when compile for powerpc:
+
+| x_sh_dbIO.c: In function 'swap_short':
+| x_sh_dbIO.c:229:36: error: initializer element is not constant
+| 229 | static unsigned short ooop = *iptr;
+| | ^
+
+Upstream-Status: Pending
+
+Signed-off-by: Kai Kang <kai.kang@windriver.com>
+---
+ src/sh_dbIO.c | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/src/sh_dbIO.c b/src/sh_dbIO.c
+index b547ac5..23a9621 100644
+--- a/src/sh_dbIO.c
++++ b/src/sh_dbIO.c
+@@ -226,7 +226,8 @@ static unsigned short * swap_short (unsigned short * iptr)
+ else
+ {
+ /* alignment problem */
+- static unsigned short ooop = *iptr;
++ static unsigned short ooop;
++ ooop = *iptr;
+ unsigned short hi = (ooop & 0xff00);
+ unsigned short lo = (ooop & 0xff);
+ ooop = (lo << 8) | (hi >> 8);
diff --git a/recipes-ids/samhain/samhain.inc b/recipes-ids/samhain/samhain.inc
index 6a2eb08..0148e46 100644
--- a/recipes-ids/samhain/samhain.inc
+++ b/recipes-ids/samhain/samhain.inc
@@ -18,6 +18,7 @@ SRC_URI = "https://la-samhna.de/archive/samhain_signed-${PV}.tar.gz \
file://samhain-avoid-searching-host-for-postgresql.patch \
file://samhain-add-LDFLAGS-variable-for-samhain_setpwd.patch \
file://fix-build-with-new-version-attr.patch \
+ file://samhain-fix-initializer-element-is-not-constant.patch \
"

SRC_URI[sha256sum] = "3e57574036d5055e9557ec5095818b419ea6c4365370fc2ccce1e9f87f9fad08"
--
2.17.1


Re: how often would one use "VAR_someoverride_append = ..."?

Robert P. J. Day
 

On Wed, 10 Mar 2021, Quentin Schulz wrote:
... snip ...

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n12
for example.

This is an example of a "valid" use case (not that there are invalid
ones) for VAR_foo.

Would probably a better example:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n50

So, you might decide that a machine is so much different than others
that KERNEL_FEATURES should be overridden for said machine.

Then in a bbappend, one might want to add another feature for this
machine, they'll therefore need to use KERNEL_FEATURES_foo_append.

I do not have examples at hand of VAR_foo_append except the ones
Leon sent in another mail. Which should show how rare it is :)
i will ramble for just a bit as i think i settled on a way to
perhaps explain the difference here to some students.

in the overwhelmingly common case, we normally see overrides at the
end of either assignments or appends:

VAR = "snafu"
VAR_qemux86 = "more snafu"
VAR_append_qemux86 = "qemux86_specific_fubar"

it seems that the proper way to interpret the above is that, in the
end, all you would be after is the *final* value of the single
variable VAR, and all the appending and overriding is leading you in
that direction -- whatever "VAR" contains after it's all over is all
you care about.

on the other hand, with the example from the kernel recipe:

KBRANCH_qemuarm ?= "v5.10/standard/arm-versatile-926ejs"
KBRANCH_qemuarm64 ?= "v5.10/standard/qemuarm64"
KBRANCH_qemumips ?= "v5.10/standard/mti-malta32"
KBRANCH_qemuppc ?= "v5.10/standard/qemuppc"
KBRANCH_qemuriscv64 ?= "v5.10/standard/base"
KBRANCH_qemux86 ?= "v5.10/standard/base"
KBRANCH_qemux86-64 ?= "v5.10/standard/base"
KBRANCH_qemumips64 ?= "v5.10/standard/mti-malta64"

it seems that the right way to interpret the above is that you are,
simultaneously, creating *multiple* versions of the variable KBRANCH,
so that if you subsequently see, for example:

KBRANCH_qemux86_append = " whatever"

you're appending to the variable KBRANCH_qemux86 because the ultimate
result you're after is the *set* of KBRANCH-related variables, perhaps
because you will actually need them all at once.

and given that most situations don't require access to all of those
variations of that variable, that's why the first approach is by far
the most common.

that might be an over-simplification, but it seems to at least give
an idea of why that second approach would be used in very particular
situations.

thoughts?

rday


Re: How can I create a truly minimal distribution that runs entirely from RAM?

Zoran
 

BBB example:
https://www.yoctoproject.org/pipermail/yocto/2018-July/041696.html

The line:
DISTRO_FEATURES_append = " ram"

Should be:
DISTRO_FEATURES_append = " nfs"

BSP Traces for BBB (YOCTO Warrior):
https://github.com/ZoranStojsavljevic/bbb-yocto/blob/master/bbb-releases/bbb-warrior/target-bbb-platform-traces.txt

And the ash script in the U-Boot supporting above written:
https://github.com/ZoranStojsavljevic/bbb-yocto/blob/master/custom/u-boot.ash#L3

Line 3 to 10.

Please, pay attention to line 1 as well!

Zee
_______

On Thu, Mar 11, 2021 at 12:32 AM p32 via lists.yoctoproject.org
<p32=tuta.io@lists.yoctoproject.org> wrote:

Hello everyone,

I am currently using this Yocto-based build setup provided by NXP to create a custom Linux distribution for one of the i.MX boards. My custom image is based on the core-image-minimal recipe and works fine, i.e., runs on the platform as expected. However, I have to following two issues:

Although core-image-minimal is documented as "A small image just capable of allowing a device to boot", I can tell from the running system that it contains a huge number of components that I think are not be strictly necessary to boot the device. For instance, the boot log contains entries about an FPGA manager framework, Bluetooth, Ethernet, KVM, USB, and a lot of i.MX-specific modules such as for DMA or power management. For evaluation purposes, I want to get rid of all of these and end up with a truly minimal Linux system that is able to boot, schedule its tasks, and to communicate via UART. How can I achieve this without losing the i.MX support, i.e., the generation of a bootloader and suitable device tree files?

Furthermore, I would like the minimal system to run entirely from RAM. More specifically: After being started from the SD card, U-Boot should start the Linux distribution via initramfs. I am able to generate some kind of initramfs binary using the following changes:
# local.conf
INITRAMFS_IMAGE = "recipe-name"
INITRAMFS_IMAGE_BUNDLE = "1"
# recipe-name.bb
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
PACKAGE_INSTALL = "${IMAGE_INSTALL}"

However, this does not affect the generated U-Boot, which means that U-Boot still tries to boot from an SD card partition. What is the "right way" to make use of the Image-initramfs-board.bin or the image-board.cpio.gz files that Yocto creates in this case?

Any help yould be greatly appreciated.

Kind regards!


[meta-security][PATCH] samhain: fix compile for powerpc

kai
 

From: Kai Kang <kai.kang@windriver.com>

It fails to comile samhain for powerpc(qemuppc):

| x_sh_dbIO.c: In function 'swap_short':
| x_sh_dbIO.c:229:36: error: initializer element is not constant
| 229 | static unsigned short ooop = *iptr;
| | ^

Fix it by not use local static variable.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
...-initializer-element-is-not-constant.patch | 32 +++++++++++++++++++
recipes-ids/samhain/samhain.inc | 1 +
2 files changed, 33 insertions(+)
create mode 100644 recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch

diff --git a/recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch b/recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch
new file mode 100644
index 0000000..846e132
--- /dev/null
+++ b/recipes-ids/samhain/files/samhain-fix-initializer-element-is-not-constant.patch
@@ -0,0 +1,32 @@
+Fix error when compile for powerpc:
+
+| x_sh_dbIO.c: In function 'swap_short':
+| x_sh_dbIO.c:229:36: error: initializer element is not constant
+| 229 | static unsigned short ooop = *iptr;
+| | ^
+
+Upstream-Status: Pending
+
+Signed-off-by: Kai Kang <kai.kang@windriver.com>
+---
+ src/sh_dbIO.c | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/src/sh_dbIO.c b/src/sh_dbIO.c
+index b547ac5..7516463 100644
+--- a/src/sh_dbIO.c
++++ b/src/sh_dbIO.c
+@@ -226,11 +226,11 @@ static unsigned short * swap_short (unsigned short * iptr)
+ else
+ {
+ /* alignment problem */
+- static unsigned short ooop = *iptr;
++ unsigned short ooop = *iptr;
+ unsigned short hi = (ooop & 0xff00);
+ unsigned short lo = (ooop & 0xff);
+ ooop = (lo << 8) | (hi >> 8);
+- return &ooop;
++ *iptr = ooop;
+ }
+ return iptr;
+ #else
diff --git a/recipes-ids/samhain/samhain.inc b/recipes-ids/samhain/samhain.inc
index 6a2eb08..0148e46 100644
--- a/recipes-ids/samhain/samhain.inc
+++ b/recipes-ids/samhain/samhain.inc
@@ -18,6 +18,7 @@ SRC_URI = "https://la-samhna.de/archive/samhain_signed-${PV}.tar.gz \
file://samhain-avoid-searching-host-for-postgresql.patch \
file://samhain-add-LDFLAGS-variable-for-samhain_setpwd.patch \
file://fix-build-with-new-version-attr.patch \
+ file://samhain-fix-initializer-element-is-not-constant.patch \
"

SRC_URI[sha256sum] = "3e57574036d5055e9557ec5095818b419ea6c4365370fc2ccce1e9f87f9fad08"
--
2.17.1


Re: Assign IP address at boot time

Rudolf J Streif
 


On 3/10/21 1:32 PM, jchludzinski wrote:
Create the file /etc/systemd/network/eth0.conf:

[Match]
Name=eth0
[Network]
Address=<ip>
Gateway=<ip>
DNS=<ip>

That didn't work BUT editing /lib/systemd/network/80-wired.network
[Match]
Name=en* eth*
KernelCommandLine=!nfsroot

[Match]
Name=eth0
[Network]
Address=192.168.0.101/24
#Gateway=<ip>
#DNS=<ip>

[Network]
DHCP=no

[DHCP]
#RouteMetric=10
#ClientIdentifier=mac
... did work, after:
root@arria10:~# systemctl restart systemd-networkd
Preferred method for local configuration is using /etc/systemd/network. However, the probable reason why it did not work is that the file should be eth0.network. The name of the file(s) do not matter that much. The extension does though.
-- 
Rudolf J Streif
CEO/CTO ibeeto
+1.855.442.3386 x700


How can I create a truly minimal distribution that runs entirely from RAM?

p32@...
 

Hello everyone,

I am currently using this Yocto-based build setup provided by NXP to create a custom Linux distribution for one of the i.MX boards. My custom image is based on the core-image-minimal recipe and works fine, i.e., runs on the platform as expected. However, I have to following two issues:

Although core-image-minimal is documented as "A small image just capable of allowing a device to boot", I can tell from the running system that it contains a huge number of components that I think are not be strictly necessary to boot the device. For instance, the boot log contains entries about an FPGA manager framework, Bluetooth, Ethernet, KVM, USB, and a lot of i.MX-specific modules such as for DMA or power management. For evaluation purposes, I want to get rid of all of these and end up with a truly minimal Linux system that is able to boot, schedule its tasks, and to communicate via UART. How can I achieve this without losing the i.MX support, i.e., the generation of a bootloader and suitable device tree files?

Furthermore, I would like the minimal system to run entirely from RAM. More specifically: After being started from the SD card, U-Boot should start the Linux distribution via initramfs. I am able to generate some kind of initramfs binary using the following changes:
# local.conf
INITRAMFS_IMAGE = "recipe-name"
INITRAMFS_IMAGE_BUNDLE = "1"
# recipe-name.bb
IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
PACKAGE_INSTALL = "${IMAGE_INSTALL}"

However, this does not affect the generated U-Boot, which means that U-Boot still tries to boot from an SD card partition. What is the "right way" to make use of the Image-initramfs-board.bin or the image-board.cpio.gz files that Yocto creates in this case?

Any help yould be greatly appreciated.

Kind regards!


Re: Assign IP address at boot time

jchludzinski
 

Create the file /etc/systemd/network/eth0.conf:

[Match]
Name=eth0
[Network]
Address=<ip>
Gateway=<ip>
DNS=<ip>

That didn't work BUT editing /lib/systemd/network/80-wired.network
[Match]
Name=en* eth*
KernelCommandLine=!nfsroot

[Match]
Name=eth0
[Network]
Address=192.168.0.101/24
#Gateway=<ip>
#DNS=<ip>

[Network]
DHCP=no

[DHCP]
#RouteMetric=10
#ClientIdentifier=mac
... did work, after:
root@arria10:~# systemctl restart systemd-networkd


Integrating npm into install?

Paul Wicks
 

I've got a python app that uses some javascript assets that are normally built via webpack. In a normal build, the process is as follows:

* run "npm install" to get all the javascript dependencies
* run "npm build", a custom task in my package.json file that invokes webpack to build the assets and place them where the python install can find them.
* run "python setup.py install" to build and install the python part of the app

What's the best/most correct way to integrate this into a yocto recipe? My current approach is to have the main recipe "inherit setuptools3" and then add the following:

"
do_run_npm_install() {
    bbplain "Running npm install, this may take a few minutes..."
    npm install
    npm run build
}

addtask run_npm_install before do_compile
"

However, this fails with an "npm: not found" error.

-Paul Wicks


Re: Assign IP address at boot time

Rudolf J Streif
 

Create the file /etc/systemd/network/eth0.conf:

[Match]
Name=eth0
[Network]
Address=<ip>
Gateway=<ip>
DNS=<ip>

Restart systemd-networkd

# systemctl restart systemd-networkd

or reboot.

:rjs

On 3/10/21 11:31 AM, jchludzinski wrote:

What is your startup manager? systemd or SysVInit?
systemd
--
Rudolf J Streif
CEO/CTO ibeeto
+1.855.442.3386 x700


Re: Assign IP address at boot time

jchludzinski
 

What is your startup manager? systemd or SysVInit?
systemd


[meta-security][PATCH] fscryptctl: Fix installation path

Łukasz Płachno
 

- Without the patch fscryptctl is installed in
/usr/bin/usr/local/bin instead of /usr/bin.
---
recipes-security/fscryptctl/fscryptctl_1.0.0.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/recipes-security/fscryptctl/fscryptctl_1.0.0.bb b/recipes-security/fscryptctl/fscryptctl_1.0.0.bb
index 440b4e3..df76a3d 100644
--- a/recipes-security/fscryptctl/fscryptctl_1.0.0.bb
+++ b/recipes-security/fscryptctl/fscryptctl_1.0.0.bb
@@ -15,7 +15,7 @@ SRC_URI = "git://github.com/google/fscryptctl.git"
S = "${WORKDIR}/git"

do_install() {
- oe_runmake DESTDIR=${D}${bindir} install
+ oe_runmake DESTDIR=${D} PREFIX=/usr install
}

RRECOMMENDS_${PN} += "\
--
2.17.1


Demo of abi checker hook with hashequiv

Michael Ho <Michael.Ho@...>
 

Hi all,

 

I wanted to share with the mailing list a small proof of concept / demo I created that shows how the hashequiv checksum hook can be overridden with a custom hook to avoid rebuilds where the file checksums are not exactly reproducible or change in a compatible manner. In the case of this demo it shows how an abi checker like libabigail can be used to avoid recompiling downstream applications when a recipe deploying a shared library introduces changes that still maintain the ABI (while still catching cases where the ABI breaks). Would be good to get any comments people have about this and also the general feasibility of using abi checkers in a Yocto build.

 

See: https://github.com/bmwcarit/meta-abicompat and https://github.com/bmwcarit/meta-abicompat-poky

 

I found though to make this demo that I had to modify poky to split the do_package task into two parts in order to be able to re-package without inducing a recompilation. Would be also interesting to get some comments about what people think about this.

 

Kind regards,

Michael Ho

 

-- 

BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung – Build and Release Engineering
Lise-Meitner-Straße 14
89081 Ulm

Tel.: ­+49-731-37804-071

Mobil: +49-152-54980-471
Fax: +49-731-37804-001
Mail: 
michael.ho@... 
Web: 
http://www.bmw-carit.de
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Kai-Uwe Balszuweit und
 Michael Böttrich
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------

 


Re: Assign IP address at boot time

Rudolf J Streif
 

On 3/9/21 11:34 PM, jchludzinski wrote:
To start, do you have the driver required for your network interface?
Yes, the network/Ethernet device is recognized by Linux and the appropriate driver is loaded. If I use:

$ ip add add 192.168.9.101/24 dev eth0

I’m up and running. But I want the IP assignment to happen during boot time.
What is your startup manager? systemd or SysVInit?

--
Rudolf J Streif
CEO/CTO ibeeto
+1.855.442.3386 x700


Re: [meta-security][dunfell][PATCH 0/9] Some IMA/EVM fixes to dunfell branch

Armin Kuster
 

On 3/10/21 2:31 AM, Ming Liu wrote:
Hi, akuster808:

I saw this patch set has been merged to gatesgarth, may I ask, any
plan for dunfell? I am asking because dunfell is a LTS branch and many
users are building their products based on it. Thanks!
that are being built current. I can on build on branch at a time and it
takes time to do each.  Things are looking good so I suspect they
changes will land in the next day or so.

-armin

the best,
thank you

series in build testing

-armin

On 3/2/21 6:57 AM, liu.ming50@gmail.com
<mailto:liu.ming50@gmail.com> wrote:
> From: Ming Liu <ming.liu@toradex.com <mailto:ming.liu@toradex.com>>
>
> Cherry pick some IMA/EVM fixes to LTS dunfell branch, with these
> patches applied, I could run a ima enabled image with
sysvinit/systemd
> on qemuarm/qemuarm64 and some NXP machines.
>
> Ming Liu (9):
>   ima-evm-utils: set native REQUIRED_DISTRO_FEATURES to empty
>   initramfs-framework-ima: fix a wrong path
>   ima-evm-keys: add recipe
>   initramfs-framework-ima: RDEPENDS on ima-evm-keys
>   meta: refactor IMA/EVM sign rootfs
>   README.md: update according to the refactoring in
>     ima-evm-rootfs.bbclass
>   initramfs-framework-ima: let ima_enabled return 0
>   ima-evm-rootfs.bbclass: avoid generating /etc/fstab for wic
>   ima-policy-hashed: add CGROUP2_SUPER_MAGIC fsmagic
>
>  meta-integrity/README.md                      |  4 ++-
>  meta-integrity/classes/ima-evm-rootfs.bbclass | 33
+++++++++----------
>  .../initrdscripts/initramfs-framework-ima.bb
<http://initramfs-framework-ima.bb>  |  2 +-
>  .../initrdscripts/initramfs-framework-ima/ima |  3 +-
>  .../ima-evm-keys/ima-evm-keys_1.0.bb
<http://ima-evm-keys_1.0.bb>          | 16 +++++++++
>  .../ima-evm-utils/ima-evm-utils_git.bb
<http://ima-evm-utils_git.bb>        |  1 +
>  .../ima_policy_hashed/files/ima_policy_hashed |  3 ++
>  7 files changed, 41 insertions(+), 21 deletions(-)
>  create mode 100644
meta-integrity/recipes-security/ima-evm-keys/ima-evm-keys_1.0.bb
<http://ima-evm-keys_1.0.bb>
>


Re: how often would one use "VAR_someoverride_append = ..."?

Quentin Schulz
 

Hi Robert,

On Wed, Mar 10, 2021 at 06:16:44AM -0500, Robert P. J. Day wrote:
On Tue, 9 Mar 2021, Quentin Schulz wrote:

Hi Robert,

On Tue, Mar 09, 2021 at 09:39:14AM -0500, Robert P. J. Day wrote:

bitbake manual, chapter 3, examples of conditional syntax:

https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#examples

correctly distinguishes between A_foo_append and A_append_foo, but how
often would one use that first form, anyway?

most uses of conditional appending are either just straight
appending:

VAR_append = "fubar"

or used with an override thusly:

VAR_append_x86 = "snafu"

is there an actual practical usage of, say:

VAR_x86_append = "huh"

i can't remember the last time i saw something of that form and,
while it might be worth explaining, it seems that the reader might be
warned that that form is almost certainly *not* what they want.
Yes, in 99% of the cases, you want VAR_append_foo and not VAR_foo_append.

VAR_foo_append makes sense when you want to append to VAR_foo which
is a way to override completely VAR for builds matching the foo
override. This happens in kernel-yocto recipes where branches and
defconfigs are different per machine for example.
can you point at an actual example of that? i took a look and all
the yocto kernel recipes i see use the first form. am i just looking
in the wrong place?
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n12
for example.

This is an example of a "valid" use case (not that there are invalid
ones) for VAR_foo.

Would probably a better example:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n50

So, you might decide that a machine is so much different than others
that KERNEL_FEATURES should be overridden for said machine.

Then in a bbappend, one might want to add another feature for this
machine, they'll therefore need to use KERNEL_FEATURES_foo_append.

I do not have examples at hand of VAR_foo_append except the ones Leon
sent in another mail. Which should show how rare it is :)

Cheers,
Quentin

2161 - 2180 of 54795