RFC: Package exclusion


Paul Eggleton
 

Hi all,

As part of the 1.1 feature list I suggested a review of the
INCOMPATIBLE_LICENSE and COMMERCIAL* package exclusion mechanisms we have
within Poky. Below I've outlined my ideas and would appreciate any
comments/additions/corrections.

==== Aims ====

* Make error messages clear when user/dependencies have asked for something
to be built that can't be due to restrictions

* Ensure that exclusion system is reliable

==== Proposed implementation ====

1) Ensure all documentation of LICENSE field value syntax is clear, concise
and up-to-date (wiki and manual)

2) Go through and audit all recipes LICENSE field values to ensure that they
all conform to the specifications. This includes making sure that | (package
may be used under one of a selection of licences) and & (recipe has mixed
licences that apply to the code base, so conditions of all must be observed)
are used correctly.

3) bitbake/core changes:

* LICENSE field checking must fully parse the field and understand the
difference between | and &, and must not e.g. mark Qt as being GPLv3 only.

* Make the LICENSE validity checking more strict (given recipes have been
audited and rules are clear after above)

* Don't exclude any recipes at parse time - simply record all excluded
recipes and their runtime provides in a blacklist which also includes flags
indicating the reason for blacklisting

* Ensure all excluded licences in INCOMPATIBLE_LICENSE are valid (e.g. catch
GPL3 as apposed to GPLv3) - if not, error out

* Check when calculating dependencies if anything is scheduled to be built
that is on the blacklist - if any are, gather all of them up and then stop and
list them in an error message along with reason and depchain for each one

* Check when constructing the rootfs if anything in the runtime provides
blacklist is going to be included - if so, error out


Some further possible extensions:

* Possibly apply similar logic to COMPATIBLE_MACHINE?

* Replace COMMERCIAL* with some more generic exclusion mechanism that allows
the reason to be defined as part of the exclusion list?

* As a helper for non-en_US users, fail on parse if user defines any of the
*LICENSE* variables as *LICENCE*? (we definitely don't want the build to
continue and just ignore this as the user might not realise what has happened)


Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Elizabeth Flanagan <elizabeth.flanagan@...>
 

On 05/03/2011 09:00 AM, Paul Eggleton wrote:
Hi all,

As part of the 1.1 feature list I suggested a review of the
INCOMPATIBLE_LICENSE and COMMERCIAL* package exclusion mechanisms we have
within Poky. Below I've outlined my ideas and would appreciate any
comments/additions/corrections.

==== Aims ====

* Make error messages clear when user/dependencies have asked for something
to be built that can't be due to restrictions

* Ensure that exclusion system is reliable

==== Proposed implementation ====

1) Ensure all documentation of LICENSE field value syntax is clear, concise
and up-to-date (wiki and manual)

2) Go through and audit all recipes LICENSE field values to ensure that they
all conform to the specifications. This includes making sure that | (package
may be used under one of a selection of licences) and& (recipe has mixed
licences that apply to the code base, so conditions of all must be observed)
are used correctly.

3) bitbake/core changes:

* LICENSE field checking must fully parse the field and understand the
difference between | and&, and must not e.g. mark Qt as being GPLv3 only.

* Make the LICENSE validity checking more strict (given recipes have been
audited and rules are clear after above)

* Don't exclude any recipes at parse time - simply record all excluded
recipes and their runtime provides in a blacklist which also includes flags
indicating the reason for blacklisting

* Ensure all excluded licences in INCOMPATIBLE_LICENSE are valid (e.g. catch
GPL3 as apposed to GPLv3) - if not, error out

* Check when calculating dependencies if anything is scheduled to be built
that is on the blacklist - if any are, gather all of them up and then stop and
list them in an error message along with reason and depchain for each one

* Check when constructing the rootfs if anything in the runtime provides
blacklist is going to be included - if so, error out
Most of the issues surrounding the license field is currently on my plate for M1.D

License tracking Build a parser to do license tracking more gracefully and make sure all recipes are correct. (takes ~2 weeks) 1 Accept Beth Beth M1, Sprint D

There is currently no specification on the LICENSE field values and how they should be parsed. I'm going to be starting that in the next week or so. We should sync up with each other if this work is going to overlap. Schedule some time for this week?

-b


Cheers,
Paul
--
---------------
Elizabeth Flanagan
Yocto Project
Release Engineer


Paul Eggleton
 

Hi Beth,

On Wednesday 04 May 2011 00:08:59 Elizabeth Flanagan wrote:
Most of the issues surrounding the license field is currently on my plate
for M1.D

License tracking Build a parser to do license tracking more gracefully and
make sure all recipes are correct. (takes ~2 weeks) 1 Accept Beth Beth
M1,
Sprint D

There is currently no specification on the LICENSE field values and how
they should be parsed. I'm going to be starting that in the next week or
so. We should sync up with each other if this work is going to overlap.
Schedule some time for this week?
Yes there's definitely quite an overlap here and we should coordinate efforts.
Will catch you later on IRC to organise some time.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Joshua Lock <josh@...>
 

Apologies for the tardy reply. I like where you're going, usually I have
to disagree to reply swiftly. Sorry.

On Tue, 2011-05-03 at 17:00 +0100, Paul Eggleton wrote:
Hi all,

As part of the 1.1 feature list I suggested a review of the
INCOMPATIBLE_LICENSE and COMMERCIAL* package exclusion mechanisms we have
within Poky. Below I've outlined my ideas and would appreciate any
comments/additions/corrections.

==== Aims ====

* Make error messages clear when user/dependencies have asked for something
to be built that can't be due to restrictions

* Ensure that exclusion system is reliable
Solid.


==== Proposed implementation ====

1) Ensure all documentation of LICENSE field value syntax is clear, concise
and up-to-date (wiki and manual)

2) Go through and audit all recipes LICENSE field values to ensure that they
all conform to the specifications. This includes making sure that | (package
may be used under one of a selection of licences) and & (recipe has mixed
licences that apply to the code base, so conditions of all must be observed)
are used correctly.

3) bitbake/core changes:

* LICENSE field checking must fully parse the field and understand the
difference between | and &, and must not e.g. mark Qt as being GPLv3 only.

* Make the LICENSE validity checking more strict (given recipes have been
audited and rules are clear after above)

* Don't exclude any recipes at parse time - simply record all excluded
recipes and their runtime provides in a blacklist which also includes flags
indicating the reason for blacklisting

* Ensure all excluded licences in INCOMPATIBLE_LICENSE are valid (e.g. catch
GPL3 as apposed to GPLv3) - if not, error out

* Check when calculating dependencies if anything is scheduled to be built
that is on the blacklist - if any are, gather all of them up and then stop and
list them in an error message along with reason and depchain for each one
This sounds like overlap with a task Scott Garman is set to work on
(Error handling in bitbake), let's make sure you guys are collaborating
here.
http://bugzilla.yoctoproject.org/show_bug.cgi?id=542



* Check when constructing the rootfs if anything in the runtime provides
blacklist is going to be included - if so, error out


Some further possible extensions:

* Possibly apply similar logic to COMPATIBLE_MACHINE?

* Replace COMMERCIAL* with some more generic exclusion mechanism that allows
the reason to be defined as part of the exclusion list?
Sounds reasonable, care to elaborate? Something like:

COMMERCIAL_LICENSE = "naughty packages from copyright hell"
PERL_PACKAGES = "packages I don't want to include because I dislike
Perl"
EXCLUDED_PATTERNS = "PERL_PACKAGES COMMERCIAL_LICENSE"

??


* As a helper for non-en_US users, fail on parse if user defines any of the
*LICENSE* variables as *LICENCE*? (we definitely don't want the build to
continue and just ignore this as the user might not realise what has happened)
Darn tootin!

Looks like a good start, thanks Paul!

Cheers,
Joshua
--
Joshua Lock
Yocto Build System Monkey
Intel Open Source Technology Centre


Paul Eggleton
 

On Wednesday 11 May 2011 00:20:14 Joshua Lock wrote:
* Check when calculating dependencies if anything is scheduled to be
built that is on the blacklist - if any are, gather all of them up and
then stop and list them in an error message along with reason and
depchain for each one
This sounds like overlap with a task Scott Garman is set to work on
(Error handling in bitbake), let's make sure you guys are collaborating
here.
http://bugzilla.yoctoproject.org/show_bug.cgi?id=542
I hadn't considered this - you may well be right. It depends on how Scott
proposes to attack that particular problem, I will check with him.

* Replace COMMERCIAL* with some more generic exclusion mechanism that
allows the reason to be defined as part of the exclusion list?
Sounds reasonable, care to elaborate? Something like:

COMMERCIAL_LICENSE = "naughty packages from copyright hell"
PERL_PACKAGES = "packages I don't want to include because I dislike
Perl"
EXCLUDED_PATTERNS = "PERL_PACKAGES COMMERCIAL_LICENSE"
More like:

EXCLUDED_PACKAGES += "broken1,broken2:These packages are broken by design and
not supported;"
EXCLUDED_PACKAGES += "licenserequired,clickthru,special1:These packages
require a special license agreement;"

The example syntax is horrible but you get the idea.

* As a helper for non-en_US users, fail on parse if user defines any of
the *LICENSE* variables as *LICENCE*? (we definitely don't want the build
to continue and just ignore this as the user might not realise what has
happened)
Darn tootin!
I think you meant to say "hear hear!" or maybe even "good show!"... ;)

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Joshua Lock <josh@...>
 

On Wed, 2011-05-11 at 10:06 +0100, Paul Eggleton wrote:
On Wednesday 11 May 2011 00:20:14 Joshua Lock wrote:
* Check when calculating dependencies if anything is scheduled to be
built that is on the blacklist - if any are, gather all of them up and
then stop and list them in an error message along with reason and
depchain for each one
This sounds like overlap with a task Scott Garman is set to work on
(Error handling in bitbake), let's make sure you guys are collaborating
here.
http://bugzilla.yoctoproject.org/show_bug.cgi?id=542
I hadn't considered this - you may well be right. It depends on how Scott
proposes to attack that particular problem, I will check with him.
Cool. If they don't overlap I'd be surprised.


* Replace COMMERCIAL* with some more generic exclusion mechanism that
allows the reason to be defined as part of the exclusion list?
Sounds reasonable, care to elaborate? Something like:

COMMERCIAL_LICENSE = "naughty packages from copyright hell"
PERL_PACKAGES = "packages I don't want to include because I dislike
Perl"
EXCLUDED_PATTERNS = "PERL_PACKAGES COMMERCIAL_LICENSE"
More like:

EXCLUDED_PACKAGES += "broken1,broken2:These packages are broken by design and
not supported;"
EXCLUDED_PACKAGES += "licenserequired,clickthru,special1:These packages
require a special license agreement;"

The example syntax is horrible but you get the idea.
The idea is sound, yeah.


* As a helper for non-en_US users, fail on parse if user defines any of
the *LICENSE* variables as *LICENCE*? (we definitely don't want the build
to continue and just ignore this as the user might not realise what has
happened)
Darn tootin!
I think you meant to say "hear hear!" or maybe even "good show!"... ;)
Bravo!

Joshua
--
Joshua Lock
Yocto Build System Monkey
Intel Open Source Technology Centre