VS: [yocto] Make do_image_complete wait for an earlier task


Mikko Murto
 

Lähettäjä: Richard Purdie <richard.purdie@...>
Lähetetty: tiistai 18. toukokuuta 2021 9.57

On Tue, 2021-05-18 at 06:15 +0000, Mikko Murto wrote:
Hi,

I've created a task do_create_spdx that gets some data from packages
after do_package. The task is currently added with `addtask
do_create_spdx after do_package before do_packagedata` The data is
then enriched and combined in do_image_complete by adding the combining
function to IMAGE_POSTPROCESS_COMMAND.
For the most part everything works as expected, but for some packages
do_create_spdx does not complete in time for the combination step in
do_image_complete.
Is there a way for me force do_image_complete to wait for
do_create_spdx to have been completed for all packages?
This seems a little odd since image.bbclass has:

do_rootfs[recrdeptask] += "do_packagedata"

which means do_rootfs should only happen after all do_packagedata have run.
Your task is added before do_packagedata.
do_image and do_image_complete should run after do_rootfs.

I'm wondering if the issue is more to do with reuse of previous build artefacts
where sstate is involved. Does your added task account for sstate?
That is odd indeed. I need to give it another go. As for accounting for sstate, no, at least not intentionally.

I've been giving some thought to SDPX and SBOM during our build process so I
am curious what you're doing, is the work public anywhere?
Public at https://github.com/doubleopen-project/meta-doubleopen. The usual disclaimers about PoC-level spaghetti apply; all questions and comments very much appreciated! 😊

Mikko


Richard Purdie
 

On Tue, 2021-05-18 at 07:09 +0000, Mikko Murto wrote:
Lähettäjä: Richard Purdie <richard.purdie@...>
Lähetetty: tiistai 18. toukokuuta 2021 9.57

On Tue, 2021-05-18 at 06:15 +0000, Mikko Murto wrote:
Hi,

I've created a task do_create_spdx that gets some data from packages
after do_package. The task is currently added with `addtask
do_create_spdx after do_package before do_packagedata` The data is
then enriched and combined in do_image_complete by adding the combining
function to IMAGE_POSTPROCESS_COMMAND.
For the most part everything works as expected, but for some packages
do_create_spdx does not complete in time for the combination step in
do_image_complete.
Is there a way for me force do_image_complete to wait for
do_create_spdx to have been completed for all packages?
This seems a little odd since image.bbclass has:

do_rootfs[recrdeptask] += "do_packagedata"

which means do_rootfs should only happen after all do_packagedata have run.
Your task is added before do_packagedata.
do_image and do_image_complete should run after do_rootfs.

I'm wondering if the issue is more to do with reuse of previous build artefacts
where sstate is involved. Does your added task account for sstate?
That is odd indeed. I need to give it another go. As for accounting for sstate, 
no, at least not intentionally.
I don't have much to go on but guessing, if you run some test and then rerun that
test, the second build would come from sstate. If a build comes from sstate, it will 
skip tasks that precede the sstate artefact.

More specifically, if sstate exists for do_packagedata, it would skip any tasks 
preceeding that task including your do_create_spdx task.

There is no direct dependency from do_image_complete on the do_create_spdx tasks.

You can add:

do_rootfs[recrdeptask] += "do_create_spdx"

but this will have the side effect that the build will never use sstate and always
rebuild since that task isn't an sstate task.

I've been giving some thought to SDPX and SBOM during our build process so I
am curious what you're doing, is the work public anywhere?
Public at https://github.com/doubleopen-project/meta-doubleopen. The usual 
disclaimers about PoC-level spaghetti apply; all questions and comments 
very much appreciated! 😊
I did have a quick look and you're going further than I'd been thinking of, at 
least initially.

What I'm thinking of in core YP initially is to have do_packagedata generate SPDX 
data for the output packaged files in do_package/do_packagedata. I noticed you go
further and process all the input sources and I'm not sure we're ready to do that
yet.

Doing it at do_package/do_packagedata time would still access any of the sources
included from a debug perspective, hence it should correctly find the shipped 
manifest/license info without the complexity of having to scan all the sources

For your level of source scanning, I'd look at the existing do_populate_lic task
which is sstate and generates license info. I think I'd be in favour of totally 
replacing that with something generating spdx output...

I'm actually hoping we could simply what we're doing today however the more I look
at all the information you can put into SPDX, the more I worry that whilst we can
generate tons of data and huge SPDX files, I'm not sure they're actually useful
to anyone to actually use :/.

For the SBOM information, we do need to somehow make something as useful as our
normal manifest to people for this to be useful and adopted, at least from my 
perspective.

Cheers,

Richard