Red alert but apparently harmless setscene errors


Michael Opdenacker
 

Hi,

Sharing this before opening a bug if needed...

I'm building the latest Poky ("core-image-minimal" plus a few extra
packages).

I'm always getting the below setscene errors after upgrading my Poky
sources:

... <more similar errors before>
WARNING: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Failed to
fetch URL
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
WARNING: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Failed
to fetch URL
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: No
suitable staging package found
ERROR: Logfile of failure stored in:
/media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/libpcre2-native/10.39-r0/temp/log.do_populate_sysroot_setscene.315121
WARNING: Setscene task
(virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-support/libpcre/libpcre2_10.39.bb:do_populate_sysroot_setscene)
failed with exit code '1' - real task will be run instead
Currently  4 running tasks (1838 of 1838/467 of 4630)  10%
|################

As expected, the error messages are highlighted in red, but they are not
critical as BitBake can always build the corresponding recipes from sources.

Two questions:

* Anything wrong with my local sstate cache? Should I erase it?
* Should these issues really be treated as errors, scaring users that
something could be wrong while the resulting build looks correct anyway?

Thanks in advance
Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Jose Quaresma
 

Hi,

I've seen the same for some time now when building core-image-minimal with yocto state mirror and hashequiv servers.

After some investigation I think the root cause is some bandwidth/connection limitation on the http yocto mirror.

Reducing the BB_NUMBER_THREADS="1" to a single core solves many of the fetch issues for me,
my config is the default one with:

  BB_NUMBER_THREADS="1"
  SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"

Jose

Michael Opdenacker <michael.opdenacker@...> escreveu no dia quinta, 13/01/2022 à(s) 17:23:

Hi,

Sharing this before opening a bug if needed...

I'm building the latest Poky ("core-image-minimal" plus a few extra
packages).

I'm always getting the below setscene errors after upgrading my Poky
sources:

... <more similar errors before>
WARNING: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Failed to
fetch URL
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
WARNING: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Failed
to fetch URL
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: No
suitable staging package found
ERROR: Logfile of failure stored in:
/media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/libpcre2-native/10.39-r0/temp/log.do_populate_sysroot_setscene.315121
WARNING: Setscene task
(virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-support/libpcre/libpcre2_10.39.bb:do_populate_sysroot_setscene)
failed with exit code '1' - real task will be run instead
Currently  4 running tasks (1838 of 1838/467 of 4630)  10%
|################

As expected, the error messages are highlighted in red, but they are not
critical as BitBake can always build the corresponding recipes from sources.

Two questions:

  * Anything wrong with my local sstate cache? Should I erase it?
  * Should these issues really be treated as errors, scaring users that
    something could be wrong while the resulting build looks correct anyway?

Thanks in advance
Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com






--
Best regards,

José Quaresma


Joshua Watt
 

On Thu, Jan 13, 2022 at 11:23 AM Michael Opdenacker
<michael.opdenacker@bootlin.com> wrote:

Hi,

Sharing this before opening a bug if needed...

I'm building the latest Poky ("core-image-minimal" plus a few extra
packages).

I'm always getting the below setscene errors after upgrading my Poky
sources:

... <more similar errors before>
WARNING: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Failed to
fetch URL
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
/media/mike/ssd/yocto/poky/build/sstate-cache
/media/mike/ssd/yocto/poky/build/sstate-cache
WARNING: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Failed
to fetch URL
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
/media/mike/ssd/yocto/poky/build/sstate-cache
/media/mike/ssd/yocto/poky/build/sstate-cache
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: No
suitable staging package found
ERROR: Logfile of failure stored in:
/media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/libpcre2-native/10.39-r0/temp/log.do_populate_sysroot_setscene.315121
WARNING: Setscene task
(virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-support/libpcre/libpcre2_10.39.bb:do_populate_sysroot_setscene)
failed with exit code '1' - real task will be run instead
Currently 4 running tasks (1838 of 1838/467 of 4630) 10%
|################

As expected, the error messages are highlighted in red, but they are not
critical as BitBake can always build the corresponding recipes from sources.

Two questions:

* Anything wrong with my local sstate cache? Should I erase it?
This usually happens when there is a sstate archive (.tar.zst), but no
corresponding siginfo file (.tar.zst.siginfo). The sstate code assumes
the siginfo file is present if archive is present, and "errors" if
not. In my experience, this usually happens when you try to clean out
the sstate cache using a mechanism that is unaware of the relationship
and deletes the siginfo and not the archive (e.g. some form of "find
-delete ...")

* Should these issues really be treated as errors, scaring users that
something could be wrong while the resulting build looks correct anyway?
As noted by the build, it will actually just execute the missing
tasks; one annoyance (depending on context) is that bitbake will
return a non-zero exit code if any ERROR occurs in the build, even
though in this case it was recovered.

This has been discussed at length before (I can't find a citation
ATM), and I believe it was left this way because this particular case
is considered an actual error that should never happen and needs
investigation on the YP Autobuilder.


Thanks in advance
Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com




Michael Opdenacker
 

Hi Jose,

Thanks for your advice!

On 1/13/22 6:37 PM, Jose Quaresma wrote:
Hi,

I've seen the same for some time now when building core-image-minimal
with yocto state mirror and hashequiv servers.

After some investigation I think the root cause is some
bandwidth/connection limitation on the http yocto mirror.

Reducing the BB_NUMBER_THREADS="1" to a single core solves many of the
fetch issues for me,
my config is the default one with:

  BB_NUMBER_THREADS="1"
  SSTATE_MIRRORS = "file://.*
http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
<http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
<http://typhoon.yocto.io:8687>"

Oops, it seems you're right. I have the same SSTATE_MIRRORS and
BB_HASHSERVE_UPSTREAM settings as you do, and adding
BB_NUMBER_THREADS="1" makes the errors disappear.

It seems I should file a bug about that. The error messages just mention
local paths, not remote ones, so it sounds like a local issue while it's
apparently not.

Thanks again
Michael.

--

Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Richard Purdie
 

On Thu, 2022-01-13 at 19:37 +0100, Michael Opdenacker wrote:
Hi Jose,

Thanks for your advice!

On 1/13/22 6:37 PM, Jose Quaresma wrote:
Hi,

I've seen the same for some time now when building core-image-minimal
with yocto state mirror and hashequiv servers.

After some investigation I think the root cause is some
bandwidth/connection limitation on the http yocto mirror.

Reducing the BB_NUMBER_THREADS="1" to a single core solves many of the
fetch issues for me,
my config is the default one with:

  BB_NUMBER_THREADS="1"
  SSTATE_MIRRORS = "file://.*
http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
<http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
<http://typhoon.yocto.io:8687>"

Oops, it seems you're right. I have the same SSTATE_MIRRORS and
BB_HASHSERVE_UPSTREAM settings as you do, and adding
BB_NUMBER_THREADS="1" makes the errors disappear.

It seems I should file a bug about that. The error messages just mention
local paths, not remote ones, so it sounds like a local issue while it's
apparently not.
That is useful to debug it but suggests there is an issue with concurrent
connections to the YP server.

I think we should have an open bug and get Michael Halstead's input on this.

Cheers,

Richard


Jose Quaresma
 


Richard Purdie <richard.purdie@...> escreveu no dia quinta, 13/01/2022 à(s) 21:53:
On Thu, 2022-01-13 at 19:37 +0100, Michael Opdenacker wrote:
> Hi Jose,
>
> Thanks for your advice!
>
> On 1/13/22 6:37 PM, Jose Quaresma wrote:
> > Hi,
> >
> > I've seen the same for some time now when building core-image-minimal
> > with yocto state mirror and hashequiv servers.
> >
> > After some investigation I think the root cause is some
> > bandwidth/connection limitation on the http yocto mirror.
> >
> > Reducing the BB_NUMBER_THREADS="1" to a single core solves many of the
> > fetch issues for me,
> > my config is the default one with:
> >
> >   BB_NUMBER_THREADS="1"
> >   SSTATE_MIRRORS = "file://.*
> > http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
> > <http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
> >   BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
> > <http://typhoon.yocto.io:8687>"
>
>
> Oops, it seems you're right. I have the same SSTATE_MIRRORS and
> BB_HASHSERVE_UPSTREAM settings as you do, and adding
> BB_NUMBER_THREADS="1" makes the errors disappear.
>
> It seems I should file a bug about that. The error messages just mention
> local paths, not remote ones, so it sounds like a local issue while it's
> apparently not.

That is useful to debug it but suggests there is an issue with concurrent
connections to the YP server.

I think we should have an open bug and get Michael Halstead's input on this.

Cheers,

Richard


During the last yocto project summit 2021.11 I have submitted a patch
to reduce the time needed for handling the sstate mirrors on startup,
that without my realizing it actually launches a DoS attack to the mirror.
This patch was rejected as expected.

After that patch, maybe by coincidence I got the idea that some measures
had been taken to mitigate this problem but if it wasn't the case then there is
a bug in concurrent connections to the YP server.

When the patch was sent there was no problem with the concurrent connections to the YP server.

Jose


Michael Opdenacker
 

On 1/13/22 10:53 PM, Richard Purdie wrote:
On Thu, 2022-01-13 at 19:37 +0100, Michael Opdenacker wrote:
Hi Jose,

Thanks for your advice!

On 1/13/22 6:37 PM, Jose Quaresma wrote:
Hi,

I've seen the same for some time now when building core-image-minimal
with yocto state mirror and hashequiv servers.

After some investigation I think the root cause is some
bandwidth/connection limitation on the http yocto mirror.

Reducing the BB_NUMBER_THREADS="1" to a single core solves many of the
fetch issues for me,
my config is the default one with:

  BB_NUMBER_THREADS="1"
  SSTATE_MIRRORS = "file://.*
http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
<http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
<http://typhoon.yocto.io:8687>"
Oops, it seems you're right. I have the same SSTATE_MIRRORS and
BB_HASHSERVE_UPSTREAM settings as you do, and adding
BB_NUMBER_THREADS="1" makes the errors disappear.

It seems I should file a bug about that. The error messages just mention
local paths, not remote ones, so it sounds like a local issue while it's
apparently not.
That is useful to debug it but suggests there is an issue with concurrent
connections to the YP server.

I think we should have an open bug and get Michael Halstead's input on this.

Done: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14690
Thanks for your advice.
Cheers
Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Jose Quaresma
 

Hi,

I have tested the state mirror locally with my own http mirror and it works perfectly.
There are no problems with the concurrent connections.
I'm more sure that the root cause is some bandwidth/connection limitation on the http YP mirror.

Jose

Michael Opdenacker <michael.opdenacker@...> escreveu no dia sexta, 14/01/2022 à(s) 19:16:


On 1/13/22 10:53 PM, Richard Purdie wrote:
> On Thu, 2022-01-13 at 19:37 +0100, Michael Opdenacker wrote:
>> Hi Jose,
>>
>> Thanks for your advice!
>>
>> On 1/13/22 6:37 PM, Jose Quaresma wrote:
>>> Hi,
>>>
>>> I've seen the same for some time now when building core-image-minimal
>>> with yocto state mirror and hashequiv servers.
>>>
>>> After some investigation I think the root cause is some
>>> bandwidth/connection limitation on the http yocto mirror.
>>>
>>> Reducing the BB_NUMBER_THREADS="1" to a single core solves many of the
>>> fetch issues for me,
>>> my config is the default one with:
>>>
>>>   BB_NUMBER_THREADS="1"
>>>   SSTATE_MIRRORS = "file://.*
>>> http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
>>> <http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
>>>   BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
>>> <http://typhoon.yocto.io:8687>"
>>
>> Oops, it seems you're right. I have the same SSTATE_MIRRORS and
>> BB_HASHSERVE_UPSTREAM settings as you do, and adding
>> BB_NUMBER_THREADS="1" makes the errors disappear.
>>
>> It seems I should file a bug about that. The error messages just mention
>> local paths, not remote ones, so it sounds like a local issue while it's
>> apparently not.
> That is useful to debug it but suggests there is an issue with concurrent
> connections to the YP server.
>
> I think we should have an open bug and get Michael Halstead's input on this.


Done: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14690
Thanks for your advice.
Cheers
Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com



--
Best regards,

José Quaresma