Date   

Re: what's the state of things with pushing the bounds on ASSUME_PROVIDED?

Richard Purdie
 

On Thu, 2021-06-24 at 07:50 -0400, Robert P. J. Day wrote:
  i asked about this once upon a time, so i thought i'd follow up ...
given the fairly stable state of recent linux distros, is there any
standard for taking advantage of what *should* be robust native tools
rather than building them? (i'm ignoring taking advantage of sstate
and building SDKs and other clever speedups for now.)

  from scratch, i did a wind river (LINCD) build of
wrlinux-image-small (and i assume it would be much the same under
current oe-core), and i notice that numerous native tools were
compiled, including such standards as cmake, curl, elfutils ... the
list goes on and on.

  so other than the tools that are *required* to be installed, if i
mention that i am currently running ubuntu 20.04, is there any
indication as to which tools i'm relatively safe to take advantage
using ASSUME_PROVIDED and HOSTTOOLS? i realize that the versions built
will probably differ from the host versions, but it seems that if
there is an incompatibility, that would be fairly obvious in short
order.

  thoughts?
Quite often things aren't as simple as they first seem:

Elfutils has a history of interesting changes between versions so having 
our builds use a consistent version is good.

Some recipes build libs as well as binaries, e.g. the compression tools.
Its relatively easy to check a binary is present, it is harder to check
the right -devel headers are present. That is a solvable problem but again, 
version consistency is good. If you require a HOSTTOOLS bin but our own
lib, you can get version mismatches.

We do patch some utilities for 'reasons' and having those patches missing
can be a pain and cause weird errors.

Reproducibility is also a concern, particularly if different versions of 
tools like flex/bison generated different code.

I also wonder who is going to support testing all these different options
and handle the resulting build failures and bugs being raised?

This list isn't definitive.


In summary, I see a lot of problems for what amounts to not much speed
gain. Particularly when we have a mechanism like sstate available
which allows binary reuse.

Cheers,

Richard


Re: Hardknott (GCC10) Compiler Issues

Zoran
 

> I have no idea if this is possible in the current YOCTO development stage:
> GCCVERSION = "11.%"
> To do the FF to GCC 11.>

WARNING: preferred version 11.% of gcc-runtime not available (for item libg2c)
WARNING: versions of gcc-runtime available: 10.2.0


For hardknott. Guess, this answers my later question.

Let us see about my very first question!

BR,
Zee
_______

INCLUDED:
WARNING: preferred version 11.% of gcc-runtime not available (for item libssp-dev)
WARNING: versions of gcc-runtime available: 10.2.0
WARNING: preferred version 11.% of gcc-runtime not available (for item libg2c-dev)
WARNING: versions of gcc-runtime available: 10.2.0
WARNING: preferred version 11.% of gcc-runtime not available (for item libssp)
WARNING: versions of gcc-runtime available: 10.2.0

Build Configuration:
BB_VERSION           = "1.50.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "fedora-33"
TARGET_SYS           = "arm-poky-linux-gnueabi"
MACHINE              = "beaglebone"
DISTRO               = "poky"
DISTRO_VERSION       = "3.3.1"
TUNE_FEATURES        = "arm vfp cortexa8 neon callconvention-hard"
TARGET_FPU           = "hard"
meta                
meta-poky            
meta-yocto-bsp       = "hardknott:74dbb08c3709fec6563ee65a3661f66fdcbb3e2f"
meta-jumpnow         = "hardknott:ac90f018ebb9de8d6ac12f22368e004aa7be69a2"
meta-bbb             = "hardknott:d838aa54e3ed81d08597c08e112fc8966aaa501d"
meta-oe              
meta-python          
meta-networking      = "hardknott:aca88908fd329f5cef6f19995b072397fb2d8ec6"
meta-qt5             = "upstream/hardknott:a00af3eae082b772469d9dd21b2371dd4d237684"
meta-socketcan       = "master:cefd86cd1def9ad2e63be527f8ce36a076d7e17c"

NOTE: Fetching uninative binary shim http://downloads.yoctoproject.org/releases/uninative/3.2/x86_64-nativesdk-libc.tar.xz;sha256sum=3ee8c7d55e2d4c7ae3887cddb97219f97b94efddfeee2e24923c0cb0e8ce84c6 (will check PREMIRRORS first)
Initialising tasks: 100% |###########################################################################################| Time: 0:00:11
Sstate summary: Wanted 1709 Local 0 Network 0 Missed 1709 Current 0 (0% match, 0% complete)
NOTE: Executing Tasks


On Fri, Jun 25, 2021 at 7:58 AM Zoran via lists.yoctoproject.org <zoran.stojsavljevic=gmail.com@...> wrote:
An interesting issue, and I think I hit it as well (my best guess).

Here is my issue:
https://github.com/mguentner/cannelloni/issues/35

> During the thud-to-hardknott upgrade process, we did nightly
> builds of the new hardknott based target image from our thud
> based SDK VM. I assumed that since GCC10 was being built
> as part of the build sysroot bootstrap process, we were getting
> a clean and consistent result irrespective of the underlying
> build server OS.

Maybe you can try the following: in your local.conf to insert the
following line:

GCCVERSION = "9.%"

for hardknott release.

I need to try this myself, I just used gcc as is (default one which
comes with the release, I guess 10).

I have no idea if this is possible in the current YOCTO development stage:

GCCVERSION = "11.%"

To do the FF to GCC 11.

Zee
_______

On Fri, Jun 25, 2021 at 6:48 AM Chuck Wolber <chuckwolber@...> wrote:
>
> All,
>
> Please accept my apologies in advance for the detailed submission. I think it is warranted in this
> case.
>
> There is something... "odd" about the GCC 10 compiler that is delivered with Hardknott. I am still
> chasing it down, so I am not yet ready to declare a root cause or submit a bug, but I am posting
> what I have now in case anyone has some insights to offer.
>
> For all I know it is something unusual that I am doing, but we have a lot of history with our
> build/dev/release methods, so I would be surprised if that was actually the case. I have also
> discussed aspects of this on IRC for the last few days, so some of this may be familiar to some
> of you.
>
> Background: We maintain a virtual machine SDK for our developers that is as close as possible to
> the actual embedded hardware environment that we target. The SDK image is our baseline Linux
> OS plus lots of the expected dev and debugging tools. The image deployed to our target devices is
> the baseline Linux OS plus the core application suite. It is also important to note that we only
> support the x86_64 machine architecture in our target devices and development workstations.
>
> We also spin up and spin down the SDK VM for our nightly builds. This guarantees strict consistency
> and eliminates lots of variables when we are trying to troubleshoot something hairy.
>
> We just upgraded from Thud to Hardknott. This means we built our new Hardknott based SDK VM
> image from our Thud based SDK VM (GCC 8 / glibc 2.28). When we attempted to build our target
> device image in the new Hardknott based SDK VM, we consistently got a segfault when any build
> task involves bison issuing a warning of some sort. I traced this down for a very long time and it
> seemed to have something to do with the libtextstyle library from gettext and the way bison used it.
> But I now believe that this to be a red herring. Bison seems to be very fragile, but in this case,
> that may have actually been a good thing.
>
> After some experimentation I found that the issue went away when I dropped down to the 3.6.4
> recipe of bison found at OE-Core:bc95820cd. But this did not sit right with me. There is no way I
> should be the only person seeing this issue.
>
> Then I tried an experiment... I assumed I was encountering a compiler bootstrap issue with such a
> big jump (GCC8 -> GCC10), so I rebuilt our hardknott based SDK VM with the 3.3.1 version of
> buildtools-extended. The build worked flawlessly, but when I booted into the new SDK VM and
> kicked off the build I got the same result (bison segfault when any build warnings are encountered).
>
> This is when I started to mentally put a few more details together with other post-upgrade issues that
> had been discovered in our lab. We attributed them to garden variety API and behavioral changes
> expected during a Yocto upgrade, but now I am not so sure.
>
> During the thud-to-hardknott upgrade process, we did nightly builds of the new hardknott based
> target image from our thud based SDK VM. I assumed that since GCC10 was being built as part of
> the build sysroot bootstrap process, we were getting a clean and consistent result irrespective of the
> underlying build server OS.
>
> One of the issues we were seeing in the lab was a periodic hang during the initramfs phase of the
> boot process. We run a couple of setup scripts to manage the sysroot before the switch_root, so it
> is not unusual to see some "growing pains" after an upgrade. The hangs were random with no
> obvious cause, but systemd is very weird anyway so we attributed it to a new dependency or race
> condition that we had to address after going from systemd 239 to 247.
>
> It is also worth noting that systemd itself was not hung, it responded to the 'ole "three finger salute"
> and dutifully filled the screen with shutdown messages. It was just that the boot process randomly
> stopped cold in initramfs before the switch root. We would also occasionally see systemd
> complaining in the logs, "Starting requested but asserts failed".
>
> Historically, when asserts fail, it is a sign of a much larger problem, so I did another experiment...
>
> Since we could build our SDK VM successfully with buildtools-extended, why not build the target
> images? So I did. After a day of testing in the lab, none of the testers have seen the boot hang up in
> the initramfs stage, whereas before it was happening about 50% of the time. I need a good week of
> successful test activity before I am willing to declare success, but the results were convincing
> enough to make it worth this summary post.
>
> I did an extensive amount of trial and error testing, including meticulously comparing
> buildtools-extended with our own versions of the same files. The only intersection point was gcc.
>
> The gcc delivered with buildtools-extended works great. When I build hardknott's gcc10 from the
> gcc in buildtools-extended, we are not able to build our target images with the resulting compiler.
> When I build our target images from the old thud environment, we get a mysterious hang and
> systemd asserts triggering during boot. Since GCC10 is an intermediate piece of the build, it is
> also implicated despite the native environment running GCC8.
>
> I will continue to troubleshoot this but I was hoping for some insight (or gentle guidance if I am
> making a silly mistake). Overall, I am at a loss to think of a reason why I should not be able to build
> a compiler from the buildtools-extended compiler and then use it to reliably build our target images.
>
> Thank you,
>
> ..Ch:W..
>
>
> P.S. For those who are curious, we started out on Pyro hosted on Ubuntu 16.04. From there we made
> the jump to self hosting when we used that environment to build a thud based VM SDK. After years of
> successful build, we are now in the process of upgrading to Hardknott.
>
> P.P.S. For the sake of completeness, I had to add the following files to the buildtools-extended
> sysroot to fully complete the build of our images:
>
> /usr/include/magic.h -> util-linux "more" command requires this.
> /usr/include/zstd.h -> I do not recall which recipe required this.
> /usr/bin/free -> The OpenJDK 8 build scripts need this.
> /usr/include/sys/* -> openjdk-8-native
> /lib/libcap.so.2 -> The binutils "dir" command quietly breaks the build without this. I am not a fan of the
>                             lack of error checking in the binutils build...
> /usr/include/sensors/error.h and sensors.h -> mesa-native
> /usr/include/zstd_errors.h -> qemu-system-native
>
> --
> "Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire
>
>
>




Re: Hardknott (GCC10) Compiler Issues

Zoran
 

An interesting issue, and I think I hit it as well (my best guess).

Here is my issue:
https://github.com/mguentner/cannelloni/issues/35

During the thud-to-hardknott upgrade process, we did nightly
builds of the new hardknott based target image from our thud
based SDK VM. I assumed that since GCC10 was being built
as part of the build sysroot bootstrap process, we were getting
a clean and consistent result irrespective of the underlying
build server OS.
Maybe you can try the following: in your local.conf to insert the
following line:

GCCVERSION = "9.%"

for hardknott release.

I need to try this myself, I just used gcc as is (default one which
comes with the release, I guess 10).

I have no idea if this is possible in the current YOCTO development stage:

GCCVERSION = "11.%"

To do the FF to GCC 11.

Zee
_______

On Fri, Jun 25, 2021 at 6:48 AM Chuck Wolber <chuckwolber@...> wrote:

All,

Please accept my apologies in advance for the detailed submission. I think it is warranted in this
case.

There is something... "odd" about the GCC 10 compiler that is delivered with Hardknott. I am still
chasing it down, so I am not yet ready to declare a root cause or submit a bug, but I am posting
what I have now in case anyone has some insights to offer.

For all I know it is something unusual that I am doing, but we have a lot of history with our
build/dev/release methods, so I would be surprised if that was actually the case. I have also
discussed aspects of this on IRC for the last few days, so some of this may be familiar to some
of you.

Background: We maintain a virtual machine SDK for our developers that is as close as possible to
the actual embedded hardware environment that we target. The SDK image is our baseline Linux
OS plus lots of the expected dev and debugging tools. The image deployed to our target devices is
the baseline Linux OS plus the core application suite. It is also important to note that we only
support the x86_64 machine architecture in our target devices and development workstations.

We also spin up and spin down the SDK VM for our nightly builds. This guarantees strict consistency
and eliminates lots of variables when we are trying to troubleshoot something hairy.

We just upgraded from Thud to Hardknott. This means we built our new Hardknott based SDK VM
image from our Thud based SDK VM (GCC 8 / glibc 2.28). When we attempted to build our target
device image in the new Hardknott based SDK VM, we consistently got a segfault when any build
task involves bison issuing a warning of some sort. I traced this down for a very long time and it
seemed to have something to do with the libtextstyle library from gettext and the way bison used it.
But I now believe that this to be a red herring. Bison seems to be very fragile, but in this case,
that may have actually been a good thing.

After some experimentation I found that the issue went away when I dropped down to the 3.6.4
recipe of bison found at OE-Core:bc95820cd. But this did not sit right with me. There is no way I
should be the only person seeing this issue.

Then I tried an experiment... I assumed I was encountering a compiler bootstrap issue with such a
big jump (GCC8 -> GCC10), so I rebuilt our hardknott based SDK VM with the 3.3.1 version of
buildtools-extended. The build worked flawlessly, but when I booted into the new SDK VM and
kicked off the build I got the same result (bison segfault when any build warnings are encountered).

This is when I started to mentally put a few more details together with other post-upgrade issues that
had been discovered in our lab. We attributed them to garden variety API and behavioral changes
expected during a Yocto upgrade, but now I am not so sure.

During the thud-to-hardknott upgrade process, we did nightly builds of the new hardknott based
target image from our thud based SDK VM. I assumed that since GCC10 was being built as part of
the build sysroot bootstrap process, we were getting a clean and consistent result irrespective of the
underlying build server OS.

One of the issues we were seeing in the lab was a periodic hang during the initramfs phase of the
boot process. We run a couple of setup scripts to manage the sysroot before the switch_root, so it
is not unusual to see some "growing pains" after an upgrade. The hangs were random with no
obvious cause, but systemd is very weird anyway so we attributed it to a new dependency or race
condition that we had to address after going from systemd 239 to 247.

It is also worth noting that systemd itself was not hung, it responded to the 'ole "three finger salute"
and dutifully filled the screen with shutdown messages. It was just that the boot process randomly
stopped cold in initramfs before the switch root. We would also occasionally see systemd
complaining in the logs, "Starting requested but asserts failed".

Historically, when asserts fail, it is a sign of a much larger problem, so I did another experiment...

Since we could build our SDK VM successfully with buildtools-extended, why not build the target
images? So I did. After a day of testing in the lab, none of the testers have seen the boot hang up in
the initramfs stage, whereas before it was happening about 50% of the time. I need a good week of
successful test activity before I am willing to declare success, but the results were convincing
enough to make it worth this summary post.

I did an extensive amount of trial and error testing, including meticulously comparing
buildtools-extended with our own versions of the same files. The only intersection point was gcc.

The gcc delivered with buildtools-extended works great. When I build hardknott's gcc10 from the
gcc in buildtools-extended, we are not able to build our target images with the resulting compiler.
When I build our target images from the old thud environment, we get a mysterious hang and
systemd asserts triggering during boot. Since GCC10 is an intermediate piece of the build, it is
also implicated despite the native environment running GCC8.

I will continue to troubleshoot this but I was hoping for some insight (or gentle guidance if I am
making a silly mistake). Overall, I am at a loss to think of a reason why I should not be able to build
a compiler from the buildtools-extended compiler and then use it to reliably build our target images.

Thank you,

..Ch:W..


P.S. For those who are curious, we started out on Pyro hosted on Ubuntu 16.04. From there we made
the jump to self hosting when we used that environment to build a thud based VM SDK. After years of
successful build, we are now in the process of upgrading to Hardknott.

P.P.S. For the sake of completeness, I had to add the following files to the buildtools-extended
sysroot to fully complete the build of our images:

/usr/include/magic.h -> util-linux "more" command requires this.
/usr/include/zstd.h -> I do not recall which recipe required this.
/usr/bin/free -> The OpenJDK 8 build scripts need this.
/usr/include/sys/* -> openjdk-8-native
/lib/libcap.so.2 -> The binutils "dir" command quietly breaks the build without this. I am not a fan of the
lack of error checking in the binutils build...
/usr/include/sensors/error.h and sensors.h -> mesa-native
/usr/include/zstd_errors.h -> qemu-system-native

--
"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire



Re: TLV320AIC3104: tlv320aic3104 #kernel #yocto

Amrun Nisha.R
 

Hi Alexandre,

Thanks for your guidance. I have updated my device tree with dummy codec as linux. Still the sound card is not properly added. When i tried to verify that using aplay, i got error message as "no sound card is found". Kindly advice.


Hardknott (GCC10) Compiler Issues

Chuck Wolber
 

All,

Please accept my apologies in advance for the detailed submission. I think it is warranted in this
case.

There is something... "odd" about the GCC 10 compiler that is delivered with Hardknott. I am still
chasing it down, so I am not yet ready to declare a root cause or submit a bug, but I am posting
what I have now in case anyone has some insights to offer.

For all I know it is something unusual that I am doing, but we have a lot of history with our
build/dev/release methods, so I would be surprised if that was actually the case. I have also
discussed aspects of this on IRC for the last few days, so some of this may be familiar to some
of you.

Background: We maintain a virtual machine SDK for our developers that is as close as possible to
the actual embedded hardware environment that we target. The SDK image is our baseline Linux
OS plus lots of the expected dev and debugging tools. The image deployed to our target devices is
the baseline Linux OS plus the core application suite. It is also important to note that we only
support the x86_64 machine architecture in our target devices and development workstations.

We also spin up and spin down the SDK VM for our nightly builds. This guarantees strict consistency
and eliminates lots of variables when we are trying to troubleshoot something hairy.

We just upgraded from Thud to Hardknott. This means we built our new Hardknott based SDK VM
image from our Thud based SDK VM (GCC 8 / glibc 2.28). When we attempted to build our target
device image in the new Hardknott based SDK VM, we consistently got a segfault when any build
task involves bison issuing a warning of some sort. I traced this down for a very long time and it
seemed to have something to do with the libtextstyle library from gettext and the way bison used it.
But I now believe that this to be a red herring. Bison seems to be very fragile, but in this case,
that may have actually been a good thing.

After some experimentation I found that the issue went away when I dropped down to the 3.6.4
recipe of bison found at OE-Core:bc95820cd. But this did not sit right with me. There is no way I
should be the only person seeing this issue.

Then I tried an experiment... I assumed I was encountering a compiler bootstrap issue with such a
big jump (GCC8 -> GCC10), so I rebuilt our hardknott based SDK VM with the 3.3.1 version of
buildtools-extended. The build worked flawlessly, but when I booted into the new SDK VM and
kicked off the build I got the same result (bison segfault when any build warnings are encountered).

This is when I started to mentally put a few more details together with other post-upgrade issues that
had been discovered in our lab. We attributed them to garden variety API and behavioral changes
expected during a Yocto upgrade, but now I am not so sure.

During the thud-to-hardknott upgrade process, we did nightly builds of the new hardknott based
target image from our thud based SDK VM. I assumed that since GCC10 was being built as part of
the build sysroot bootstrap process, we were getting a clean and consistent result irrespective of the
underlying build server OS.

One of the issues we were seeing in the lab was a periodic hang during the initramfs phase of the
boot process. We run a couple of setup scripts to manage the sysroot before the switch_root, so it
is not unusual to see some "growing pains" after an upgrade. The hangs were random with no
obvious cause, but systemd is very weird anyway so we attributed it to a new dependency or race
condition that we had to address after going from systemd 239 to 247.

It is also worth noting that systemd itself was not hung, it responded to the 'ole "three finger salute"
and dutifully filled the screen with shutdown messages. It was just that the boot process randomly
stopped cold in initramfs before the switch root. We would also occasionally see systemd
complaining in the logs, "Starting requested but asserts failed".

Historically, when asserts fail, it is a sign of a much larger problem, so I did another experiment...

Since we could build our SDK VM successfully with buildtools-extended, why not build the target
images? So I did. After a day of testing in the lab, none of the testers have seen the boot hang up in
the initramfs stage, whereas before it was happening about 50% of the time. I need a good week of
successful test activity before I am willing to declare success, but the results were convincing
enough to make it worth this summary post.

I did an extensive amount of trial and error testing, including meticulously comparing
buildtools-extended with our own versions of the same files. The only intersection point was gcc.

The gcc delivered with buildtools-extended works great. When I build hardknott's gcc10 from the
gcc in buildtools-extended, we are not able to build our target images with the resulting compiler.
When I build our target images from the old thud environment, we get a mysterious hang and
systemd asserts triggering during boot. Since GCC10 is an intermediate piece of the build, it is
also implicated despite the native environment running GCC8.

I will continue to troubleshoot this but I was hoping for some insight (or gentle guidance if I am
making a silly mistake). Overall, I am at a loss to think of a reason why I should not be able to build
a compiler from the buildtools-extended compiler and then use it to reliably build our target images.

Thank you,

..Ch:W..


P.S. For those who are curious, we started out on Pyro hosted on Ubuntu 16.04. From there we made
the jump to self hosting when we used that environment to build a thud based VM SDK. After years of
successful build, we are now in the process of upgrading to Hardknott.

P.P.S. For the sake of completeness, I had to add the following files to the buildtools-extended
sysroot to fully complete the build of our images:

/usr/include/magic.h -> util-linux "more" command requires this.
/usr/include/zstd.h -> I do not recall which recipe required this.
/usr/bin/free -> The OpenJDK 8 build scripts need this.
/usr/include/sys/* -> openjdk-8-native
/lib/libcap.so.2 -> The binutils "dir" command quietly breaks the build without this. I am not a fan of the
                            lack of error checking in the binutils build...
/usr/include/sensors/error.h and sensors.h -> mesa-native
/usr/include/zstd_errors.h -> qemu-system-native

--
"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire


Re: what's the state of things with pushing the bounds on ASSUME_PROVIDED?

Randy MacLeod
 

On 2021-06-24 7:50 a.m., Robert P. J. Day wrote:
  i asked about this once upon a time, so i thought i'd follow up ...
given the fairly stable state of recent linux distros, is there any
standard for taking advantage of what *should* be robust native tools
rather than building them? (i'm ignoring taking advantage of sstate
and building SDKs and other clever speedups for now.)

  from scratch, i did a wind river (LINCD) build of
wrlinux-image-small (and i assume it would be much the same under
current oe-core), and i notice that numerous native tools were
compiled, including such standards as cmake, curl, elfutils ... the
list goes on and on.

  so other than the tools that are *required* to be installed, if i
mention that i am currently running ubuntu 20.04, is there any
indication as to which tools i'm relatively safe to take advantage
using ASSUME_PROVIDED and HOSTTOOLS? i realize that the versions built
will probably differ from the host versions, but it seems that if
there is an incompatibility, that would be fairly obvious in short
order.

  thoughts?

Hi Robert,

I didn't remember where this was defined so I took a look:

$ git blame meta/conf/bitbake.conf | grep -A 24 ASSUME_PROVIDED | cut -c1-12,74-
34927dfa61c         2007-12-18 15:04:06 +0000 175) ASSUME_PROVIDED = "\
da52dce440a         2021-05-16 14:16:56 -0400 176)     bash-native \
34927dfa61c         2007-12-18 15:04:06 +0000 177)     bzip2-native \
40a3cf008f2         2012-10-02 14:12:21 +0100 178)     chrpath-native \
da52dce440a         2021-05-16 14:16:56 -0400 179)     diffstat-native \
43c46e9c837         2015-10-16 22:49:26 +0100 180)     file-native \
2980ac001f7         2016-02-22 16:23:49 +0000 181)     findutils-native \
b51b4f5ae26         2017-09-10 22:59:08 +1000 182)     gawk-native \
617835990ed         2012-07-10 13:26:18 +0000 183)     git-native \
34927dfa61c         2007-12-18 15:04:06 +0000 184)     grep-native \
675ff42c603         2016-01-07 13:39:39 +0200 185)     hostperl-runtime-native \
b248e55c0c3         2016-01-13 10:03:04 +0200 186)     hostpython-runtime-native \
da52dce440a         2021-05-16 14:16:56 -0400 187)     libgcc-native \
da52dce440a         2021-05-16 14:16:56 -0400 188)     patch-native \
da52dce440a         2021-05-16 14:16:56 -0400 189)     sed-native \
cb1897b8bee         2008-04-11 13:19:21 +0000 190)     tar-native \
7d3d6a35fdb         2014-08-17 23:24:02 -0700 191)     texinfo-native \
da52dce440a         2021-05-16 14:16:56 -0400 192)     virtual/crypt-native \
da52dce440a         2021-05-16 14:16:56 -0400 193)     virtual/libiconv-native \
da52dce440a         2021-05-16 14:16:56 -0400 194)     virtual/libintl-native \
0c0b07286f2         2016-01-25 13:38:00 +0000 195)     wget-native \
34927dfa61c         2007-12-18 15:04:06 +0000 196)     "
40a3cf008f2         2012-10-02 14:12:21 +0100 197) # gzip-native should be listed above?
dd1b4430b5d         2006-02-10 11:45:39 +0000 198)
^4b46c1f6e8         2005-08-31 10:45:47 +0000 199) ##################################################################


Your sort commit is one of the most recent ones:
   da52dce440 bitbake.conf: alphabetize contents of ASSUME_PROVIDED

I don't see the value in trimming the list too much especially since
you typically only have to build them once.

Of the tools you listed: 
    cmake, curl, elfutils
curl is the only one that I'd want to add to the list but
it's not a default install across all distros I assume so
why bother. I prefer to use our own cmake and elfutils.

../Randy






-- 
# Randy MacLeod
# Wind River Linux


The do_populate_sdk is finishing OK even when there are errors present in the build

Fabio Berton
 

Hi all!

I'm running some test with do_populate_sdk task and I'm seeing this on the log:

check_data_file_clashes: Package kmsxx-dbg wants to install file /home/builder/build/tmp/work/foo-poky-linux/core-image-minimal/1.0-r0/sdk/image/opt/bar/sysroots/aarch64-poky-linux/usr/bin/.debug/kmstest
But that file is already provided by package  * libdrm-dbg

I also see this kind of message with other packages.

Looking in the source code I found that the install_complementary function runs this [1] with attempt_only=True, and if attempt_only is true log above it's just a warning, as shown here [2].

This [3] comment says that "will only attempt to install these packages, if they don't exist then no error will occur."

My question is how can I force an error and not just a warning when running do_populate_sdk? 

I understand that I can change [1] to run:

  self.install(install_pkgs)

so, it'll use set attempt_only to False, that is the default, but I think this will break some use cases. 

What is the correct behaviour here, see the warning messages and fix the packages to avoid "file is already provided by package" messages, every time I create a SDK or change in some way to see an error message and stop SDK generation?

What is the correct behavior here, inspect the warning messages, and fix the packages to avoid "file is already provided by package" messages, every time I create an SDK or change it in some way to see an error message and stop the SDK generation?

Thanks!

Fabio Berton




Re: [meta-rockchip][PATCH v2] console cleanup

Trevor Woerner
 

On Thu 2021-06-24 @ 10:36:34 AM, Khem Raj wrote:
This version looks good to me.
Thanks, I appreciate your feedback.


Re: [meta-rockchip][PATCH v2] console cleanup

Khem Raj
 

On 6/24/21 5:39 AM, Trevor Woerner wrote:
Consolidate all the various console definitions to the common
conf/machine/include/rockchip-defaults.inc file and create
RK_CONSOLE_BAUD and RK_CONSOLE_DEVICE variables that can be
reused in the wks files.
The following variables were checked before and after this patch
to make sure they are sensible:
- SERIAL_CONSOLES
- RK_CONSOLE_DEVICE
- RK_CONSOLE_BAUD
A boot test was performed on the following boards to make sure
they all continue to boot to a cmdline:
- tinker-board
- rock-pi-e
- nanopi-m4-2gb
- rock64
- rock-pi-4b
This version looks good to me.

Signed-off-by: Trevor Woerner <twoerner@...>
---
Changes from v1:
- In v1 I defined RK_CONSOLE_BAUD and RK_CONSOLE_DEVICE for each MACHINE
and then redefined SERIAL_CONSOLES to be the concatenation of these two
variables. Khem pointed out this is a bad approach because I'm redefining
an oe-core-defined variable that all BSPs expect.
- In v2 I set/consolidate SERIAL_CONSOLES for each MACHINE and then generate
RK_CONSOLE_BAUD and RK_CONSOLE_DEVICE based on the first-defined
<baud>;<device> pair found in SERIAL_CONSOLES; these generated variables are
then used in the wks files.
---
conf/machine/include/nanopi-m4.inc | 2 --
conf/machine/include/rk3066.inc | 2 ++
conf/machine/include/rk3188.inc | 2 ++
conf/machine/include/rk3288.inc | 4 ++--
conf/machine/include/rk3328.inc | 2 --
conf/machine/include/rk3399.inc | 2 --
conf/machine/include/rock-pi-4.inc | 2 --
conf/machine/include/rockchip-defaults.inc | 12 +++++++++++-
conf/machine/marsboard-rk3066.conf | 1 -
conf/machine/radxarock.conf | 1 -
wic/firefly-rk3288.wks | 2 +-
wic/rock-pi-4.wks | 2 +-
wic/rock-pi-e.wks | 2 +-
wic/tinker-board.wks | 2 +-
wic/vyasa-rk3288.wks | 2 +-
15 files changed, 22 insertions(+), 18 deletions(-)
diff --git a/conf/machine/include/nanopi-m4.inc b/conf/machine/include/nanopi-m4.inc
index a14b705..8a7c1d9 100644
--- a/conf/machine/include/nanopi-m4.inc
+++ b/conf/machine/include/nanopi-m4.inc
@@ -21,5 +21,3 @@ WKS_FILE_DEPENDS ?= " \
IMAGE_BOOT_FILES ?= "\
${KERNEL_IMAGETYPE} \
"
-
-SERIAL_CONSOLES = "1500000;ttyS2"
diff --git a/conf/machine/include/rk3066.inc b/conf/machine/include/rk3066.inc
index dffbee0..5cc1024 100644
--- a/conf/machine/include/rk3066.inc
+++ b/conf/machine/include/rk3066.inc
@@ -7,5 +7,7 @@ require conf/machine/include/tune-cortexa9.inc
require conf/machine/include/soc-family.inc
require conf/machine/include/rockchip-defaults.inc
+SERIAL_CONSOLES = "115200;ttyS2"
+
KBUILD_DEFCONFIG = "multi_v7_defconfig"
KERNEL_IMAGETYPE = "zImage"
diff --git a/conf/machine/include/rk3188.inc b/conf/machine/include/rk3188.inc
index 59e65d1..6f3da93 100644
--- a/conf/machine/include/rk3188.inc
+++ b/conf/machine/include/rk3188.inc
@@ -7,5 +7,7 @@ require conf/machine/include/tune-cortexa9.inc
require conf/machine/include/soc-family.inc
require conf/machine/include/rockchip-defaults.inc
+SERIAL_CONSOLES = "115200;ttyFIQ0"
+
KBUILD_DEFCONFIG = "multi_v7_defconfig"
KERNEL_IMAGETYPE = "zImage"
diff --git a/conf/machine/include/rk3288.inc b/conf/machine/include/rk3288.inc
index 480e250..a109f26 100644
--- a/conf/machine/include/rk3288.inc
+++ b/conf/machine/include/rk3288.inc
@@ -7,11 +7,11 @@ require conf/machine/include/tune-cortexa17.inc
require conf/machine/include/soc-family.inc
require conf/machine/include/rockchip-defaults.inc
+SERIAL_CONSOLES = "115200;ttyS2"
+
KBUILD_DEFCONFIG ?= "multi_v7_defconfig"
KERNEL_IMAGETYPE = "zImage"
-SERIAL_CONSOLES = "115200;ttyS2"
-
PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot"
SPL_BINARY ?= "idbloader.img"
diff --git a/conf/machine/include/rk3328.inc b/conf/machine/include/rk3328.inc
index a4bbc5d..5b11868 100644
--- a/conf/machine/include/rk3328.inc
+++ b/conf/machine/include/rk3328.inc
@@ -19,7 +19,5 @@ TFA_BUILD_TARGET = "bl31"
UBOOT_SUFFIX ?= "itb"
UBOOT_ENTRYPOINT ?= "0x06000000"
-SERIAL_CONSOLES = "1500000;ttyS2"
-
PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot"
SPL_BINARY ?= "idbloader.img"
diff --git a/conf/machine/include/rk3399.inc b/conf/machine/include/rk3399.inc
index f6b7826..9f9f474 100644
--- a/conf/machine/include/rk3399.inc
+++ b/conf/machine/include/rk3399.inc
@@ -19,8 +19,6 @@ TFA_BUILD_TARGET = "bl31"
UBOOT_SUFFIX ?= "itb"
UBOOT_ENTRYPOINT ?= "0x06000000"
-SERIAL_CONSOLES = "115200;ttyS2"
-
PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot"
SPL_BINARY ?= "idbloader.img"
diff --git a/conf/machine/include/rock-pi-4.inc b/conf/machine/include/rock-pi-4.inc
index 9c21084..a3e60c7 100644
--- a/conf/machine/include/rock-pi-4.inc
+++ b/conf/machine/include/rock-pi-4.inc
@@ -17,6 +17,4 @@ IMAGE_BOOT_FILES ?= "\
${KERNEL_IMAGETYPE} \
"
-SERIAL_CONSOLES = "1500000;ttyS2"
-
MACHINE_EXTRA_RRECOMMENDS += "kernel-modules"
diff --git a/conf/machine/include/rockchip-defaults.inc b/conf/machine/include/rockchip-defaults.inc
index a4e2a2c..b0346c9 100644
--- a/conf/machine/include/rockchip-defaults.inc
+++ b/conf/machine/include/rockchip-defaults.inc
@@ -21,9 +21,19 @@ XSERVER = " \
"
# misc
+SERIAL_CONSOLES ?= "1500000;ttyS2"
IMAGE_FSTYPES += "ext4"
+# use the first-defined <baud>;<device> pair in SERIAL_CONSOLES
+# for the console parameter in the wks files
+RK_CONSOLE_BAUD ?= "${@d.getVar('SERIAL_CONSOLES').split(';')[0]}"
+RK_CONSOLE_DEVICE ?= "${@d.getVar('SERIAL_CONSOLES').split(';')[1].split()[0]}"
+
# boot device (sd-card/emmc)
RK_BOOT_DEVICE ??= "mmcblk0"
-WICVARS_append = " RK_BOOT_DEVICE"
+WICVARS_append = " \
+ RK_BOOT_DEVICE \
+ RK_CONSOLE_BAUD \
+ RK_CONSOLE_DEVICE \
+ "
diff --git a/conf/machine/marsboard-rk3066.conf b/conf/machine/marsboard-rk3066.conf
index 09414bc..52fd256 100644
--- a/conf/machine/marsboard-rk3066.conf
+++ b/conf/machine/marsboard-rk3066.conf
@@ -8,5 +8,4 @@
require conf/machine/include/rk3066.inc
-SERIAL_CONSOLES = "115200;ttyS2"
KERNEL_DEVICETREE = "rk3066a-marsboard.dtb"
diff --git a/conf/machine/radxarock.conf b/conf/machine/radxarock.conf
index 2036f6a..42d8848 100644
--- a/conf/machine/radxarock.conf
+++ b/conf/machine/radxarock.conf
@@ -9,5 +9,4 @@
require conf/machine/include/rk3188.inc
-SERIAL_CONSOLES = "115200;ttyFIQ0"
KERNEL_DEVICETREE = "rk3188-radxarock.dtb"
diff --git a/wic/firefly-rk3288.wks b/wic/firefly-rk3288.wks
index da0067f..7b14d1f 100644
--- a/wic/firefly-rk3288.wks
+++ b/wic/firefly-rk3288.wks
@@ -4,4 +4,4 @@
include rk3288-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root
-bootloader --ptable gpt --append="console=tty1 console=ttyS2,115200n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/rock-pi-4.wks b/wic/rock-pi-4.wks
index c6174a9..5c02e9f 100644
--- a/wic/rock-pi-4.wks
+++ b/wic/rock-pi-4.wks
@@ -4,4 +4,4 @@
include rk3399-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root
-bootloader --ptable gpt --append="console=tty1 console=ttyS2,1500000n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/rock-pi-e.wks b/wic/rock-pi-e.wks
index 97f84d1..9c10d90 100644
--- a/wic/rock-pi-e.wks
+++ b/wic/rock-pi-e.wks
@@ -1,4 +1,4 @@
include rk3328-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root
-bootloader --ptable gpt --append="console=tty1 console=ttyS2,1500000n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/tinker-board.wks b/wic/tinker-board.wks
index 5a63ce0..00ae820 100644
--- a/wic/tinker-board.wks
+++ b/wic/tinker-board.wks
@@ -5,4 +5,4 @@ include rk3288-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root
-bootloader --ptable gpt --append="console=tty1 console=ttyS2,115200n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/vyasa-rk3288.wks b/wic/vyasa-rk3288.wks
index 5db65df..5346fbd 100644
--- a/wic/vyasa-rk3288.wks
+++ b/wic/vyasa-rk3288.wks
@@ -4,5 +4,5 @@
include rk3288-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root
-bootloader --ptable gpt --append="console=tty1 console=ttyS2,115200n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"


Re: devtool deploy-target with strip option: failed to preserve ownership #devtool #dunfell

Khem Raj
 

On 6/24/21 3:11 AM, florian.amstutz@... wrote:
Hi all,
when I use "devtool deploy-target" with the --strip option in the eSDK, the ownership cannot be preserved during copying the files from "image" to "deploy-target-stripped":
devtool deploy-target --strip --no-host-check --no-preserve less root@....2
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin/less': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin/lesskey': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin/lessecho': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1/less.1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1/lesskey.1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1/lessecho.1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped': Operation not permitted
INFO: Successfully deployed /home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped
On the target, the deployed files have then the ownership of the eSDK user:
root@qemu:~# ls -lah /usr/bin/less
-rwx------    1 1002     1003      116.7K Jun 24 08:53 /usr/bin/less
"devtool deploy-target" without the --strip option works as expected. The files are owned by root.
The eSDK has been built and installed on the same machine (tested on Ubuntu 16.04, 18.04 and 20.04).
Used Yocto release: 3.1.8
Thanks in advance for any help!

Can you file a ticket in bugzilla for this as well please.

Best regards,
Florian


Yocto Autobuilder: Latency Monitor and AB-INT - Meeting notes: June 24, 2021

Randy MacLeod
 

YP AB Intermittent failures meeting - June 24, 2021, 9 AM ET
https://windriver.zoom.us/j/3696693975

Attendees: Alex, Richard, Saul, Randy, Tony, Trevor, Sakib


Summary: Things are improving somewhat on the autobuilder,
RCU stalls are still the top problem now.


Meeting Notes:
==============

1. The most common problem is still the qemu RCU hang.

Alex found the qemu machine protocol debugging commands:

{"execute": "qmp_capabilities"}

{"execute":"dump-guest-memory","arguments":{"paging":false,"protocol":"file:/tmp/vmcore.img"}}

This generates a kernel core.

Saul is going to send a patch to do that when qemu hangs.


RP investigated a few things (ACPI table alignement, etc)
and while there are differences between crashes, there's nothing
that is apparently significant so we still don't have a good
understanding of the cause of the stalls.

Richard summarized the situation here:
https://lists.yoctoproject.org/g/swat/message/177

2. We had an interesting AB failure:
Core image sato failed to start since
image copy took longer than 300 seconds:

https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3559/steps/13/logs/stdio


System load was ~ 140.
Nothing in qemu log file so qemu never started.

There was a qemu mips in the logs so qemu was running but not fast enough.


There was no AB INT log produced so the trigger was not effective.
( We may have to change out trigger to use iostat or something else. )


Can't tell from logs if the image 'start cmd' in QMP should be
put in logs. - Saul to add a log after QMP port has connected.
(maybe also add a log before?).



RP did some experiments with starting qemu in a controlled
stressful env. He wasn't able to cause the RCU hang.

- stress-ng could cause the qemu to pause itself but not crash.
The pause may have been due to running out of disk space.
qemu was running in snapshot mode initially after that
also from a tmpfs like we use for our testing.
The test generated load of ~ 3000+ and the qemu's were trying
to boot.


Any testimage failure should run Sakib's report generator.
- Sakib to send patch.



3. System load: make, ninja jobs

make:
Trevor and Randy were looking at some tools:

https://github.com/gscano/libjobserver.git
https://github.com/olsner/jobclient.git

that use the jobserver feature of make but they seemed awkward
and we weren't actually able to see them be useful.
They passed self-tests in one case but in the limited time we
used them, it seemed like a dead end.
The primary purpose of the job server feature is to enable a
single recursive make to limit the number of jobs dispatched.
It is likely possible to have independent builds using 'make'
co-operate but we have not yet figured out how to do that,
This document by Paul Smith:
http://make.mad-scientist.net/papers/jobserver-implementation/
in point "7.", explains what needs to be set to use the jobserver
but exactly how to achieve that use wasn't yet clear to us.

Luckily Trevor noticed in the make source that in 2017,
there was a patch added to make the load average calculation
more timely:
https://github.com/mirror/make/blob/master/src/job.c#L1947

https://github.com/mirror/make/commit/d8728efc80b720c630e6b12dbd34a3d44e060690

We're confirming that this actually works on a buld server and
for all versions of make in the cluster.

We can just re-use this variable:

PARALLEL_MAKE = "-j 4"
and add -l <NUM> in addition to -j <NUM>


- Randy may write to Paul Smith about the general problem
we are having since they worked together years ago.


4. ptest issues are improving.
Valgrind ptest results are getting better. Thanks Tony!


5. discussed Sakib's summary script. It's coming along.
TO DO:
- collapse compile pipeline (cc1, as, ...) to one line.
The update was just merged to master-next, let us know if
you'd like to see other info in the summary or the logs.


6. Timeouts:

- qemu-runner? timeout increase 120 -> 240
Increased qemu-runner timeout.
- ptest timeouts 300 -> 450?
Not happening.


7. The iostat output is in some of the AB logs:
https://autobuilder.yocto.io/pub/non-release/
for example:

https://autobuilder.yocto.io/pub/non-release/20210623-17/testresults/meta-oe/2021-06-23--21-51/host_stats_0_top.txt
search for "start: iostat"
it looks like the io sub-systems are 100% utilized but
we need more time, data and the summary script to be able to easily
make a general statement about AB INT issues and IO load and
to be able to identify what is typically generating the IO load.

Plans for the week:

Richard: RCU stall
Alex: qemu debugging of core
Sakib: - testimage failure dump summary.
Trevor: make job server
Tony: nothing this week for YP.
Saul: QMP logs
Randy: make job w/ Trevor, herd cats!! Here kitty,....


../Randy


[meta-rockchip][PATCH v2] console cleanup

Trevor Woerner
 

Consolidate all the various console definitions to the common
conf/machine/include/rockchip-defaults.inc file and create
RK_CONSOLE_BAUD and RK_CONSOLE_DEVICE variables that can be
reused in the wks files.

The following variables were checked before and after this patch
to make sure they are sensible:
- SERIAL_CONSOLES
- RK_CONSOLE_DEVICE
- RK_CONSOLE_BAUD

A boot test was performed on the following boards to make sure
they all continue to boot to a cmdline:
- tinker-board
- rock-pi-e
- nanopi-m4-2gb
- rock64
- rock-pi-4b

Signed-off-by: Trevor Woerner <twoerner@...>
---
Changes from v1:
- In v1 I defined RK_CONSOLE_BAUD and RK_CONSOLE_DEVICE for each MACHINE
and then redefined SERIAL_CONSOLES to be the concatenation of these two
variables. Khem pointed out this is a bad approach because I'm redefining
an oe-core-defined variable that all BSPs expect.
- In v2 I set/consolidate SERIAL_CONSOLES for each MACHINE and then generate
RK_CONSOLE_BAUD and RK_CONSOLE_DEVICE based on the first-defined
<baud>;<device> pair found in SERIAL_CONSOLES; these generated variables are
then used in the wks files.

---
conf/machine/include/nanopi-m4.inc | 2 --
conf/machine/include/rk3066.inc | 2 ++
conf/machine/include/rk3188.inc | 2 ++
conf/machine/include/rk3288.inc | 4 ++--
conf/machine/include/rk3328.inc | 2 --
conf/machine/include/rk3399.inc | 2 --
conf/machine/include/rock-pi-4.inc | 2 --
conf/machine/include/rockchip-defaults.inc | 12 +++++++++++-
conf/machine/marsboard-rk3066.conf | 1 -
conf/machine/radxarock.conf | 1 -
wic/firefly-rk3288.wks | 2 +-
wic/rock-pi-4.wks | 2 +-
wic/rock-pi-e.wks | 2 +-
wic/tinker-board.wks | 2 +-
wic/vyasa-rk3288.wks | 2 +-
15 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/conf/machine/include/nanopi-m4.inc b/conf/machine/include/nanopi-m4.inc
index a14b705..8a7c1d9 100644
--- a/conf/machine/include/nanopi-m4.inc
+++ b/conf/machine/include/nanopi-m4.inc
@@ -21,5 +21,3 @@ WKS_FILE_DEPENDS ?= " \
IMAGE_BOOT_FILES ?= "\
${KERNEL_IMAGETYPE} \
"
-
-SERIAL_CONSOLES = "1500000;ttyS2"
diff --git a/conf/machine/include/rk3066.inc b/conf/machine/include/rk3066.inc
index dffbee0..5cc1024 100644
--- a/conf/machine/include/rk3066.inc
+++ b/conf/machine/include/rk3066.inc
@@ -7,5 +7,7 @@ require conf/machine/include/tune-cortexa9.inc
require conf/machine/include/soc-family.inc
require conf/machine/include/rockchip-defaults.inc

+SERIAL_CONSOLES = "115200;ttyS2"
+
KBUILD_DEFCONFIG = "multi_v7_defconfig"
KERNEL_IMAGETYPE = "zImage"
diff --git a/conf/machine/include/rk3188.inc b/conf/machine/include/rk3188.inc
index 59e65d1..6f3da93 100644
--- a/conf/machine/include/rk3188.inc
+++ b/conf/machine/include/rk3188.inc
@@ -7,5 +7,7 @@ require conf/machine/include/tune-cortexa9.inc
require conf/machine/include/soc-family.inc
require conf/machine/include/rockchip-defaults.inc

+SERIAL_CONSOLES = "115200;ttyFIQ0"
+
KBUILD_DEFCONFIG = "multi_v7_defconfig"
KERNEL_IMAGETYPE = "zImage"
diff --git a/conf/machine/include/rk3288.inc b/conf/machine/include/rk3288.inc
index 480e250..a109f26 100644
--- a/conf/machine/include/rk3288.inc
+++ b/conf/machine/include/rk3288.inc
@@ -7,11 +7,11 @@ require conf/machine/include/tune-cortexa17.inc
require conf/machine/include/soc-family.inc
require conf/machine/include/rockchip-defaults.inc

+SERIAL_CONSOLES = "115200;ttyS2"
+
KBUILD_DEFCONFIG ?= "multi_v7_defconfig"
KERNEL_IMAGETYPE = "zImage"

-SERIAL_CONSOLES = "115200;ttyS2"
-
PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot"
SPL_BINARY ?= "idbloader.img"

diff --git a/conf/machine/include/rk3328.inc b/conf/machine/include/rk3328.inc
index a4bbc5d..5b11868 100644
--- a/conf/machine/include/rk3328.inc
+++ b/conf/machine/include/rk3328.inc
@@ -19,7 +19,5 @@ TFA_BUILD_TARGET = "bl31"
UBOOT_SUFFIX ?= "itb"
UBOOT_ENTRYPOINT ?= "0x06000000"

-SERIAL_CONSOLES = "1500000;ttyS2"
-
PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot"
SPL_BINARY ?= "idbloader.img"
diff --git a/conf/machine/include/rk3399.inc b/conf/machine/include/rk3399.inc
index f6b7826..9f9f474 100644
--- a/conf/machine/include/rk3399.inc
+++ b/conf/machine/include/rk3399.inc
@@ -19,8 +19,6 @@ TFA_BUILD_TARGET = "bl31"
UBOOT_SUFFIX ?= "itb"
UBOOT_ENTRYPOINT ?= "0x06000000"

-SERIAL_CONSOLES = "115200;ttyS2"
-
PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot"
SPL_BINARY ?= "idbloader.img"

diff --git a/conf/machine/include/rock-pi-4.inc b/conf/machine/include/rock-pi-4.inc
index 9c21084..a3e60c7 100644
--- a/conf/machine/include/rock-pi-4.inc
+++ b/conf/machine/include/rock-pi-4.inc
@@ -17,6 +17,4 @@ IMAGE_BOOT_FILES ?= "\
${KERNEL_IMAGETYPE} \
"

-SERIAL_CONSOLES = "1500000;ttyS2"
-
MACHINE_EXTRA_RRECOMMENDS += "kernel-modules"
diff --git a/conf/machine/include/rockchip-defaults.inc b/conf/machine/include/rockchip-defaults.inc
index a4e2a2c..b0346c9 100644
--- a/conf/machine/include/rockchip-defaults.inc
+++ b/conf/machine/include/rockchip-defaults.inc
@@ -21,9 +21,19 @@ XSERVER = " \
"

# misc
+SERIAL_CONSOLES ?= "1500000;ttyS2"
IMAGE_FSTYPES += "ext4"

+# use the first-defined <baud>;<device> pair in SERIAL_CONSOLES
+# for the console parameter in the wks files
+RK_CONSOLE_BAUD ?= "${@d.getVar('SERIAL_CONSOLES').split(';')[0]}"
+RK_CONSOLE_DEVICE ?= "${@d.getVar('SERIAL_CONSOLES').split(';')[1].split()[0]}"
+
# boot device (sd-card/emmc)
RK_BOOT_DEVICE ??= "mmcblk0"
-WICVARS_append = " RK_BOOT_DEVICE"

+WICVARS_append = " \
+ RK_BOOT_DEVICE \
+ RK_CONSOLE_BAUD \
+ RK_CONSOLE_DEVICE \
+ "
diff --git a/conf/machine/marsboard-rk3066.conf b/conf/machine/marsboard-rk3066.conf
index 09414bc..52fd256 100644
--- a/conf/machine/marsboard-rk3066.conf
+++ b/conf/machine/marsboard-rk3066.conf
@@ -8,5 +8,4 @@

require conf/machine/include/rk3066.inc

-SERIAL_CONSOLES = "115200;ttyS2"
KERNEL_DEVICETREE = "rk3066a-marsboard.dtb"
diff --git a/conf/machine/radxarock.conf b/conf/machine/radxarock.conf
index 2036f6a..42d8848 100644
--- a/conf/machine/radxarock.conf
+++ b/conf/machine/radxarock.conf
@@ -9,5 +9,4 @@

require conf/machine/include/rk3188.inc

-SERIAL_CONSOLES = "115200;ttyFIQ0"
KERNEL_DEVICETREE = "rk3188-radxarock.dtb"
diff --git a/wic/firefly-rk3288.wks b/wic/firefly-rk3288.wks
index da0067f..7b14d1f 100644
--- a/wic/firefly-rk3288.wks
+++ b/wic/firefly-rk3288.wks
@@ -4,4 +4,4 @@
include rk3288-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root

-bootloader --ptable gpt --append="console=tty1 console=ttyS2,115200n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/rock-pi-4.wks b/wic/rock-pi-4.wks
index c6174a9..5c02e9f 100644
--- a/wic/rock-pi-4.wks
+++ b/wic/rock-pi-4.wks
@@ -4,4 +4,4 @@
include rk3399-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root

-bootloader --ptable gpt --append="console=tty1 console=ttyS2,1500000n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/rock-pi-e.wks b/wic/rock-pi-e.wks
index 97f84d1..9c10d90 100644
--- a/wic/rock-pi-e.wks
+++ b/wic/rock-pi-e.wks
@@ -1,4 +1,4 @@
include rk3328-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root

-bootloader --ptable gpt --append="console=tty1 console=ttyS2,1500000n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/tinker-board.wks b/wic/tinker-board.wks
index 5a63ce0..00ae820 100644
--- a/wic/tinker-board.wks
+++ b/wic/tinker-board.wks
@@ -5,4 +5,4 @@ include rk3288-boot.wks

part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root

-bootloader --ptable gpt --append="console=tty1 console=ttyS2,115200n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
diff --git a/wic/vyasa-rk3288.wks b/wic/vyasa-rk3288.wks
index 5db65df..5346fbd 100644
--- a/wic/vyasa-rk3288.wks
+++ b/wic/vyasa-rk3288.wks
@@ -4,5 +4,5 @@
include rk3288-boot.wks
part / --ondisk ${RK_BOOT_DEVICE} --source rootfs --fstype=ext4 --label root

-bootloader --ptable gpt --append="console=tty1 console=ttyS2,115200n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"
+bootloader --ptable gpt --append="console=tty1 console=${RK_CONSOLE_DEVICE},${RK_CONSOLE_BAUD}n8 rw root=/dev/${RK_BOOT_DEVICE}p7 rootfstype=ext4 init=/sbin/init"

--
2.30.0.rc0


what's the state of things with pushing the bounds on ASSUME_PROVIDED?

Robert P. J. Day
 

i asked about this once upon a time, so i thought i'd follow up ...
given the fairly stable state of recent linux distros, is there any
standard for taking advantage of what *should* be robust native tools
rather than building them? (i'm ignoring taking advantage of sstate
and building SDKs and other clever speedups for now.)

from scratch, i did a wind river (LINCD) build of
wrlinux-image-small (and i assume it would be much the same under
current oe-core), and i notice that numerous native tools were
compiled, including such standards as cmake, curl, elfutils ... the
list goes on and on.

so other than the tools that are *required* to be installed, if i
mention that i am currently running ubuntu 20.04, is there any
indication as to which tools i'm relatively safe to take advantage
using ASSUME_PROVIDED and HOSTTOOLS? i realize that the versions built
will probably differ from the host versions, but it seems that if
there is an incompatibility, that would be fairly obvious in short
order.

thoughts?

rday


devtool deploy-target with strip option: failed to preserve ownership #devtool #dunfell

Florian Amstutz
 

Hi all,
 
when I use "devtool deploy-target" with the --strip option in the eSDK, the ownership cannot be preserved during copying the files from "image" to "deploy-target-stripped":
 
devtool deploy-target --strip --no-host-check --no-preserve less root@....2
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin/less': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin/lesskey': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin/lessecho': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/bin': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1/less.1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1/lesskey.1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1/lessecho.1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man/man1': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share/man': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr/share': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped/usr': Operation not permitted
cp: failed to preserve ownership for '/home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped': Operation not permitted
INFO: Successfully deployed /home/yocto/sdk/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/less/551-r0/deploy-target-stripped
 
On the target, the deployed files have then the ownership of the eSDK user:
 
root@qemu:~# ls -lah /usr/bin/less
-rwx------    1 1002     1003      116.7K Jun 24 08:53 /usr/bin/less
 
 
"devtool deploy-target" without the --strip option works as expected. The files are owned by root.
 
The eSDK has been built and installed on the same machine (tested on Ubuntu 16.04, 18.04 and 20.04).
Used Yocto release: 3.1.8
 
Thanks in advance for any help!
 
Best regards,
Florian


Re: Integration of mpg321 in yocto zeus

Ross Burton <ross@...>
 

You asked this a week ago, and I answered then too.

oe-core has mpg123. Is this not sufficient?

Ross

On Thu, 24 Jun 2021 at 10:17, Poornesh <poornesh.g@...> wrote:

Greetings !

If anyone achieved integrating the "mpg321" in yocto , requesting you to kindly share some inputs and procedure that need to be followed.

--

Thanks in advance




Re: [EXT] Re: [oe] [yocto] [meta-java] icedtea7 fetching error

Jose Quaresma
 

Hi Alejandro,

Alejandro Lozano Lozano <alejandro.lozano@...> escreveu no dia
sexta, 18/06/2021 à(s) 14:08:

Hello,



Do you know where can we fetch the correct tarballs?
This files are now on the yocto mirror
https://downloads.yoctoproject.org/mirror/sources/f89009ada191.tar.bz2




Thanks,

Alejandro



From: openembedded-devel@... <openembedded-devel@...> On Behalf Of Alexander Kanavin via lists.openembedded.org
Sent: Tuesday, June 15, 2021 1:53 AM
To: Richard Leitner - SKIDATA <Richard.Leitner@...>
Cc: Richard Purdie <richard.purdie@...>; George.Mocanu@...; Yocto-mailing-list <yocto@...>; openembedded-devel@...
Subject: [EXT] Re: [oe] [yocto] [meta-java] icedtea7 fetching error



Caution: EXT Email

On Mon, 14 Jun 2021 at 23:31, Richard Leitner - SKIDATA <Richard.Leitner@...> wrote:

Thank you very much Richard!
The openjdk8 tarballs are now hosted at https://downloads.yoctoproject.org/mirror/sources/
with their local download names.

Unfortunately I currently don't have the time and resources to provide a
patch for fixing the URLs and adding the mirror.
So help is greatly appreciated!



Unfortunately I think you mirrored the wrong tarballs. The problematic ones are specifically the six tarballs coming from http://icedtea.classpath.org/hg/

and I am not seeing them mirrored. See the original error at the start of this thread:



--2021-06-07 17:24:33-- http://icedtea.classpath.org/hg/release/icedtea7-forest-2.1/archive/f89009ada191.tar.bz2

Resolving icedtea.classpath.org (icedtea.classpath.org)... 172.104.137.120
Connecting to icedtea.classpath.org (icedtea.classpath.org)|172.104.137.120|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-06-07 16:21:05 ERROR 404: Not Found.



Alex








--
Best regards,

José Quaresma


Re: Integration of mpg321 in yocto zeus

Poornesh <poornesh.g@...>
 

Greetings !

If anyone achieved integrating the "mpg321" in yocto , requesting you to kindly share some inputs and procedure that need to be followed.

--

Thanks in advance


[poky][PATCH] models: Add a new error type for check-layer

Thomas Perrot
 

Defines a new ErrorType and ERROR_TYPE_CHOICES, in order to support this kind of
errors.

[YOCTO #14208]

Signed-off-by: Thomas Perrot <thomas.perrot@...>
---
Post/models.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/Post/models.py b/Post/models.py
index 3fa66f2198a4..b7a913c82359 100644
--- a/Post/models.py
+++ b/Post/models.py
@@ -15,6 +15,7 @@ import Levenshtein

class ErrorType(object):
RECIPE = 'recipe'
+ CHECK_LAYER = 'check-layer'
CORE = 'core'
BITBAKE_SELFTEST = 'bitbake-selftest'
OE_SELFTEST = 'oe-selftest'
@@ -26,6 +27,7 @@ class InvalidErrorType(Exception):
class Build(models.Model):
ERROR_TYPE_CHOICES = (
(ErrorType.RECIPE, 'Recipe'),
+ (ErrorType.CHECK_LAYER, 'check-layer'),
(ErrorType.CORE, 'Core'),
(ErrorType.BITBAKE_SELFTEST, 'Bitbake selftest'),
(ErrorType.OE_SELFTEST, 'OE selftest'),
--
2.31.1


[PATCH V2][yocto-autobuilder-helper] summarize_top_output.py: add script, use it and publish summary

sakib.sajal@...
 

summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

Signed-off-by: Sakib Sajal <sakib.sajal@...>
---
scripts/collect-results | 2 +-
scripts/generate-testresult-index.py | 2 +-
scripts/run-config | 1 +
scripts/summarize_top_output.py | 176 +++++++++++++++++++++++++++
4 files changed, 179 insertions(+), 2 deletions(-)
create mode 100755 scripts/summarize_top_output.py

diff --git a/scripts/collect-results b/scripts/collect-results
index 7474e36..7178380 100755
--- a/scripts/collect-results
+++ b/scripts/collect-results
@@ -19,7 +19,7 @@ if [ -e $WORKDIR/buildhistory ]; then
fi

HSFILE=$WORKDIR/tmp/buildstats/*/host_stats
-d=`date +%Y-%m-%d--%H-%M`
+d="intermittent_failure_host_data"

mkdir -p $DEST/$target/$d

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 7fdc17c..d85d606 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -154,7 +154,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
hd = []
counter = 0
# do we really need the loop?
- for p in glob.glob(buildpath + "/*/*/host_stats*top.txt"):
+ for p in glob.glob(buildpath + "/*/*/host_stats*top_summary.txt"):
n_split = p.split(build)
res = reldir[0:-1] + n_split[1]
hd.append((res, str(counter)))
diff --git a/scripts/run-config b/scripts/run-config
index 8ed88cf..82de91f 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -327,6 +327,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
if args.results_dir:
hp.printheader("Running results collection")
runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
+ runcmd([scriptsdir + "/summarize_top_output.py", args.results_dir, args.target])
sys.exit(0)

if jcfg:
diff --git a/scripts/summarize_top_output.py b/scripts/summarize_top_output.py
new file mode 100755
index 0000000..50c9b0a
--- /dev/null
+++ b/scripts/summarize_top_output.py
@@ -0,0 +1,176 @@
+#!/usr/bin/env python3
+
+import os, sys, glob
+
+# constants
+HOME = "/home/pokybuild/yocto-worker/"
+top_header = 7
+max_cols = 11
+cpu_hoggers = 5
+zombie_proc_id = "<defunct>"
+parser = "Parser"
+
+# report the following whenever they occur
+special_cmds = ["rm", "tar", "qemu"]
+
+# string substitution to make things easier to read
+subs = {
+ "/home/pokybuild/yocto-worker/" : "~/",
+ "/build/build/tmp/work/" : "/...WORK_DIR.../"
+}
+
+def usage():
+ print("Usage: " + sys.argv[0] + " <dest> <target>")
+
+def list_top_outputs(logfile):
+ # top delimiter
+ top_start = "start: top output"
+ top_end = "end: top output"
+
+ # list of top outputs
+ top_outputs = []
+
+ # flag
+ collect = False
+ with open(logfile) as log:
+ top_output = []
+ for line in log:
+ lstrip = line.strip()
+ if collect:
+ if lstrip.startswith(top_end):
+ collect = False
+ top_outputs.append(top_output)
+ top_output = []
+ else:
+ top_output.append(lstrip)
+ if lstrip.startswith(top_start):
+ collect = True
+
+ return top_outputs
+
+def summarize_top(top_outs, target):
+ summaries = []
+ kernel_summaries = []
+ zombie_summaries = []
+ short_summaries = []
+ other_builds = []
+ for top_out in top_outs:
+ summary = {}
+ kernel_summary = {}
+ zombie_summary = {}
+ short_summary = top_out[:top_header + cpu_hoggers]
+ for line in top_out[top_header:]:
+ cmd = line.split(maxsplit=max_cols)[-1]
+ if cmd.startswith(HOME):
+ b = cmd.split(HOME)[1].split("/")[0]
+ if b not in other_builds:
+ other_builds.append(b)
+ if cmd[0] == "[" and cmd[-1] == "]": # kernel processes
+ kproc = cmd[1:-1].split("/")[0]
+ if kproc not in kernel_summary:
+ kernel_summary[kproc] = 1
+ else:
+ kernel_summary[kproc] += 1
+ elif zombie_proc_id in cmd: # zombie processes
+ zproc = cmd.split()[0][1:-1]
+ if parser in zproc:
+ zproc = parser
+ if zproc not in zombie_summary:
+ zombie_summary[zproc] = 1
+ else:
+ zombie_summary[zproc] += 1
+ else: # userspace processes
+ cmd_split = cmd.split()
+ prog = cmd_split[0]
+ if prog not in summary:
+ summary[prog] = 1
+ else:
+ summary[prog] += 1
+ summary = dict(sorted(summary.items(), key=lambda item: item[1], reverse=True))
+ kernel_summary = dict(sorted(kernel_summary.items(), key=lambda item: item[1], reverse=True))
+ zombie_summary = dict(sorted(zombie_summary.items(), key=lambda item: item[1], reverse=True))
+
+ summaries.append(summary)
+ kernel_summaries.append(kernel_summary)
+ zombie_summaries.append(zombie_summary)
+ short_summaries.append(short_summary)
+ return (short_summaries, summaries, kernel_summaries, zombie_summaries, other_builds)
+
+def summarize_path(path):
+ sub = ["/recipe-sysroot-native/", "/../../libexec/", "/gcc/"]
+ p = path
+ for k, v in subs.items():
+ p = p.replace(k, v)
+ if all(x in p for x in sub):
+ rsn_spl = p.split("/recipe-sysroot-native/")
+ gcc_spl = rsn_spl[-1].split("/gcc/")
+ p = rsn_spl[0] + "/...GCC.../" + gcc_spl[-1]
+
+ return p
+
+def write_summary(short_summary, summary, kernel_summary, zombie_summary, other_build, target, logfile):
+ dirname = os.path.dirname(logfile)
+ fname = os.path.basename(logfile)
+ report_name = fname.split(".")[0] + "_summary.txt"
+ outfile = os.path.join(dirname, report_name)
+ out = "NOTE:\nProcesses that occur only once is not reported.\n"
+ out += "Program names have been shortened for better readability.\n"
+ out += "Substitutions are as follows:\n"
+ for k, v in subs.items():
+ out += (v + " = " + k + "\n")
+ out += "\n"
+
+ out += "top was invoked " + str(len(short_summary)) + " times.\n\n"
+ out += "Current build: " + target + "\n"
+ out += "Other builds:"
+ for b in other_build:
+ out += " " + b
+ out += "\n\n"
+ for i in range(len(short_summary)):
+ for l in short_summary[i]:
+ out += (l + "\n")
+
+ out += ("\nUserspace Process Summary: " + "\n")
+ if not summary:
+ out += "There were no userspace processes\n"
+ else:
+ for k, v in summary[i].items():
+ if v > 1 or any(k.startswith(x) for x in special_cmds):
+ r = summarize_path(k)
+ out += (str(v) + " " + r + "\n")
+
+ out += ("\nKernel Process Summary: " + "\n")
+ if not kernel_summary:
+ out += "There were no kernel processes\n"
+ else:
+ for k, v in kernel_summary[i].items():
+ if v > 1 or any(k.startswith(x) for x in special_cmds):
+ out += (str(v) + " " + k + "\n")
+
+ out += ("\nZombie Process Summary: " + "\n")
+ if not zombie_summary:
+ out += "There were no zombie processes\n"
+ else:
+ for k, v in zombie_summary[i].items():
+ if v > 1 or any(k.startswith(x) for x in special_cmds):
+ out += (str(v) + " " + k + "\n")
+ out += ("\n")
+
+ with open(outfile, "w") as of:
+ of.write(out)
+
+def main():
+ if len(sys.argv) != 3:
+ usage()
+ sys.exit()
+
+ dest = sys.argv[1]
+ target = sys.argv[2]
+ host_data_dir = "intermittent_failure_host_data"
+ directory = os.path.join(dest, target, host_data_dir)
+ for f in glob.glob(directory + "/*_top.txt"):
+ outputs = list_top_outputs(f)
+ short_summary, summary, kernel_summary, zombie_summary, other_build = summarize_top(outputs, target)
+ write_summary(short_summary, summary, kernel_summary, zombie_summary, other_build, target, f)
+
+main()
--
2.25.1


[PATCH] models: Add a new error type for check-layer

Thomas Perrot
 

Defines a new ErrorType and ERROR_TYPE_CHOICES, in order to support this kind of
errors.

[YOCTO #14208]

Signed-off-by: Thomas Perrot <thomas.perrot@...>
---
Post/models.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/Post/models.py b/Post/models.py
index 3fa66f2198a4..b7a913c82359 100644
--- a/Post/models.py
+++ b/Post/models.py
@@ -15,6 +15,7 @@ import Levenshtein

class ErrorType(object):
RECIPE = 'recipe'
+ CHECK_LAYER = 'check-layer'
CORE = 'core'
BITBAKE_SELFTEST = 'bitbake-selftest'
OE_SELFTEST = 'oe-selftest'
@@ -26,6 +27,7 @@ class InvalidErrorType(Exception):
class Build(models.Model):
ERROR_TYPE_CHOICES = (
(ErrorType.RECIPE, 'Recipe'),
+ (ErrorType.CHECK_LAYER, 'check-layer'),
(ErrorType.CORE, 'Core'),
(ErrorType.BITBAKE_SELFTEST, 'Bitbake selftest'),
(ErrorType.OE_SELFTEST, 'OE selftest'),
--
2.31.1

3481 - 3500 of 57419