Maintaining ABI Compatibility for LTS branch


Richard Purdie
 

Forwarding to the correct list address and cc the LTS maintainer.

Cheers,

Richard


Ross Burton <ross@...>
 

On Tue, 8 Feb 2022 at 23:07, Richard Purdie
<richard.purdie@...> wrote:
I was curious about how everyone is approaching this problem.
There was some attempt to bring ABI checking functionality in the past
but this has never been merged.

Is everyone rolling their own solution? or never moving forward?
Relatedly, somewhere I have a branch that uses libabigail to extract
the library ABIs in an build. The next step was doing the comparison,
but I never got that far. I wouldn't be surprised if someone has done
the same but actually finished the effort.

Ross


Steve Sakoman
 

On Tue, Feb 8, 2022 at 1:07 PM Richard Purdie
<richard.purdie@...> wrote:

Forwarding to the correct list address and cc the LTS maintainer.

Cheers,

Richard



---------- Forwarded message ----------
From: Sinan Kaya <okaya@...>
To: Richard Purdie <richard.purdie@...>, Paul Eggleton <bluelightning@...>, Yocto list discussion <yocto@...>
Cc:
Bcc:
Date: Sun, 6 Feb 2022 20:14:37 -0500
Subject: Maintaining ABI Compatibility for LTS branch
Hi Everyone,

One of the limitations of Yocto LTS branch is that there is no
guaranteed backwards compatibility. Therefore, any time we move a branch
forward to move to latest dunfell release, we are taking a risk of
breaking our customers.
I'd be interested in hearing about any cases where we've broken things!

In general I don't take version upgrades (other than bug fix/cve fix
only dot releases)

Yocto reserves the right to move a package version forward if a
security fix cannot be applied properly as an example.
These cases should be extremely rare, and the community/technical
steering committee is given an opportunity to review this before it
happens.

I'm certainly open to suggestions on how we can do better.

I'd love to see some tooling to do ABI checking!

Steve

This promise is being held true on the kernel by running kernel API
tests etc. and running test benches across different CI environments.

I was curious about how everyone is approaching this problem.
There was some attempt to bring ABI checking functionality in the past
but this has never been merged.

Is everyone rolling their own solution? or never moving forward?

Sinan


Richard Purdie
 

On Sun, 2022-02-06 at 20:14 -0500, Sinan Kaya wrote:
One of the limitations of Yocto LTS branch is that there is no
guaranteed backwards compatibility. Therefore, any time we move a branch
forward to move to latest dunfell release, we are taking a risk of
breaking our customers.
Whilst there isn't a guarantee, it is something we're doing our best to ensure.
Where there are security issues and we can't fix them as being backwards
compatible, we reserve the right to break things in preference to the security
issues. We wouldn't do that lightly.

Yocto reserves the right to move a package version forward if a
security fix cannot be applied properly as an example.
It also comes down to resources. If we have a choice between this and not fixing
the issue, we'd have to do this. If there are resources to do a more precision
backport, we'd take that option but we're often resource limited.

This promise is being held true on the kernel by running kernel API
tests etc. and running test benches across different CI environments.
We do run extensive tests on the autobuilder including things like LTP. We
admittedly lack resources for analysing the comparisions in an automated or
manual way sadly. I'd love to see API/ABI reporting added to our builds.

I was curious about how everyone is approaching this problem.
There was some attempt to bring ABI checking functionality in the past
but this has never been merged.
There have been ideas proposed but I've not seen any full solution as yet.

Is everyone rolling their own solution? or never moving forward?
I think most are just accepting some level of change and we've tried to keep it
to a minimal level.

I mentioned this in yesterdays call and these were mentioned:

https://github.com/lvc/abi-compliance-checkerĀ 
https://abi-laboratory.pro/index.php?view=tracker

There are two reasons people are interested:

a) for release stability as you mention
b) for performance as it could be tied into the hash equivalence mechanism for
artefact reuse - if A hasn't changed ABI, B dependning on it needn't rebuild.

There was a proof of concept of b) here:
https://lists.yoctoproject.org/g/yocto/message/52650

There are lots of levels it could be implemented at but it is something someone
would need to pick up and drive forward with a long term view to helping with
issues etc.

Cheers,

Richard


Khem Raj
 

On Wed, Feb 9, 2022 at 8:37 AM Steve Sakoman <steve@...> wrote:

On Tue, Feb 8, 2022 at 1:07 PM Richard Purdie
<richard.purdie@...> wrote:

Forwarding to the correct list address and cc the LTS maintainer.

Cheers,

Richard



---------- Forwarded message ----------
From: Sinan Kaya <okaya@...>
To: Richard Purdie <richard.purdie@...>, Paul Eggleton <bluelightning@...>, Yocto list discussion <yocto@...>
Cc:
Bcc:
Date: Sun, 6 Feb 2022 20:14:37 -0500
Subject: Maintaining ABI Compatibility for LTS branch
Hi Everyone,

One of the limitations of Yocto LTS branch is that there is no
guaranteed backwards compatibility. Therefore, any time we move a branch
forward to move to latest dunfell release, we are taking a risk of
breaking our customers.
I'd be interested in hearing about any cases where we've broken things!

In general I don't take version upgrades (other than bug fix/cve fix
only dot releases)

Yocto reserves the right to move a package version forward if a
security fix cannot be applied properly as an example.
These cases should be extremely rare, and the community/technical
steering committee is given an opportunity to review this before it
happens.

I'm certainly open to suggestions on how we can do better.

I'd love to see some tooling to do ABI checking!
I think its not that LTS itself but also upgrade from LTS to LTS or other
newer versions, it wouuld be good to have this tool and prerhaps a list
of API/ABI changes to help migration/upgrade to newer LTS releases.


Steve

This promise is being held true on the kernel by running kernel API
tests etc. and running test benches across different CI environments.

I was curious about how everyone is approaching this problem.
There was some attempt to bring ABI checking functionality in the past
but this has never been merged.

Is everyone rolling their own solution? or never moving forward?

Sinan


Sinan Kaya <okaya@...>
 

On 2/9/2022 11:42 AM, Richard Purdie wrote:
There are two reasons people are interested:
a) for release stability as you mention
b) for performance as it could be tied into the hash equivalence mechanism for
artefact reuse - if A hasn't changed ABI, B dependning on it needn't rebuild.
There was a proof of concept of b) here:
https://lists.yoctoproject.org/g/yocto/message/52650
There are lots of levels it could be implemented at but it is something someone
would need to pick up and drive forward with a long term view to helping with
issues etc.
What would be the minimum acceptable solution with the least investment?
in other words, do we have a list of requirements?

Our team has posted a solution. BMW folks posted a solution.
None of them got merged.

Could we take the version from BMW folks, merge and have the next person
add new features where it doesn't satisfy requirements?

or vice versa? or as Ross said some other work?

or none of the solutions are acceptable?


Richard Purdie
 

On Wed, 2022-02-09 at 13:15 -0500, Sinan Kaya wrote:
On 2/9/2022 11:42 AM, Richard Purdie wrote:
There are two reasons people are interested:

a) for release stability as you mention
b) for performance as it could be tied into the hash equivalence mechanism for
artefact reuse - if A hasn't changed ABI, B dependning on it needn't rebuild.

There was a proof of concept of b) here:
https://lists.yoctoproject.org/g/yocto/message/52650

There are lots of levels it could be implemented at but it is something someone
would need to pick up and drive forward with a long term view to helping with
issues etc.
What would be the minimum acceptable solution with the least investment?
in other words, do we have a list of requirements?
You're after our LTS to maintain ABI. In order to do that we need help, not just
with patches generating some output, but in day to day running of the project,
i.e. help comparing output before and after changes. Whenever a patch is
submitted which breaks this, it will need attention and help some someone to
explain to that submitter what the issue, why it is important and hints on how
to fix it.

The idea of "least investment" sends shivers down my spine since it sounds like
you want to do the absolute bare minimum to have this happen, rather than a more
community oriented approach.

Anyway, my point is there is more to this than just a patch. We have various
kinds of build output already and reports on test regressions - nobody helps
with them. I need some kind of a sign that ABI would be different and there are
people able to help with review on an ongoing basis, else it will just be
something else I and a small number of others become overloaded with.

Our team has posted a solution. BMW folks posted a solution.
None of them got merged.
Can you remind me of your team's please?

The BMW one is about hash equivalence so wouldn't help your ABI output problem
with the LTS. From what I remember, it predates hash equivalence and the project
needed a generic equivalance mechanism complete with server done at the runqueue
level in bitbake. This has now happened so we could revisit the idea of what is
in that layer but translating it to a hash equivalence plugin.

I'd also add that even with hash equivalence done well like we ended up with, we
have only two people interested/able to work on bugs with it, the author and
myself. For a key component of the system, this worries me a lot. Adding more
complexity without maintainer support isn't going to help anyone.

Could we take the version from BMW folks, merge and have the next person
add new features where it doesn't satisfy requirements?
See above on the BMW version. I'm a little worried you're suggesting merging
something which doesn't actually do what you need/want :(.

or vice versa? or as Ross said some other work?
or none of the solutions are acceptable?
I have to admit I can't remember what the conclusion was on your team's version
but if you remind me of the patches I can try and remember. This is a bigger
problem than just patches though sadly.

Cheers,

Richard


Sinan Kaya <okaya@...>
 

On 2/9/2022 1:41 PM, Richard Purdie wrote:
There are lots of levels it could be implemented at but it is something someone
would need to pick up and drive forward with a long term view to helping with
issues etc.
What would be the minimum acceptable solution with the least investment?
in other words, do we have a list of requirements?
You're after our LTS to maintain ABI. In order to do that we need help, not just
with patches generating some output, but in day to day running of the project,
i.e. help comparing output before and after changes. Whenever a patch is
submitted which breaks this, it will need attention and help some someone to
explain to that submitter what the issue, why it is important and hints on how
to fix it.
That's true, this will require engagement from the community. Tool could
take few iterations to perfect. Until then, I expect tool owner to be
responsible for fixing these bugs. Once stability is reached, it becomes
community maintained.

If the tool owner doesn't maintain and community has no interest, tool
dies and gets reverted. It is as simple as any open source engagement.

When stability is established, each code contributor to LTS would be
subject to addressing issues found before they get merged.

The idea of "least investment" sends shivers down my spine since it sounds like
you want to do the absolute bare minimum to have this happen, rather than a more
community oriented approach.
It depends on your taste. I believe in smaller improvements
as opposed to throwing a big project to you that no-one will use it.

Everyone has different needs. We need to find the common ones.
That's why, I'm asking if there is an existing tool that works for
large part of the community accepting that there will be some folks
that won't have their needs addressed.

I'm interested in revisiting the tooling discussion and have these gaps
addressed for the biggest audience so that we can have something to
build upon.

Anyway, my point is there is more to this than just a patch. We have various
kinds of build output already and reports on test regressions - nobody helps
with them. I need some kind of a sign that ABI would be different and there are
people able to help with review on an ongoing basis, else it will just be
something else I and a small number of others become overloaded with.
Noted. Hopefully, things will be not that volatile for the LTS branch
and tool would actually help the maintainer.

In an ideal world, change needs to be stopped before that happens and
have the patch author address it similar to how you monitor build
pipelines.

Our team has posted a solution. BMW folks posted a solution.
None of them got merged.
Can you remind me of your team's please?
https://lists.yoctoproject.org/g/yocto/topic/85279259?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,160,85279259

This was an intern project from last summer that we are interested
in expanding coverage.

The BMW one is about hash equivalence so wouldn't help your ABI output problem
with the LTS. From what I remember, it predates hash equivalence and the project
needed a generic equivalance mechanism complete with server done at the runqueue
level in bitbake. This has now happened so we could revisit the idea of what is
in that layer but translating it to a hash equivalence plugin.
I'd also add that even with hash equivalence done well like we ended up with, we
have only two people interested/able to work on bugs with it, the author and
myself. For a key component of the system, this worries me a lot. Adding more
complexity without maintainer support isn't going to help anyone.
OK, I didn't know the story behind the change.

Could we take the version from BMW folks, merge and have the next person
add new features where it doesn't satisfy requirements?
See above on the BMW version. I'm a little worried you're suggesting merging
something which doesn't actually do what you need/want:(.
Got it.

or vice versa? or as Ross said some other work?
or none of the solutions are acceptable?
I have to admit I can't remember what the conclusion was on your team's version
but if you remind me of the patches I can try and remember. This is a bigger
problem than just patches though sadly.
Sure, let's find out what everyone is doing.


Richard Purdie
 

On Wed, 2022-02-09 at 18:41 +0000, Richard Purdie wrote:

The BMW one is about hash equivalence so wouldn't help your ABI output problem
with the LTS. From what I remember, it predates hash equivalence and the project
needed a generic equivalance mechanism complete with server done at the runqueue
level in bitbake. This has now happened so we could revisit the idea of what is
in that layer but translating it to a hash equivalence plugin.

I'd also add that even with hash equivalence done well like we ended up with, we
have only two people interested/able to work on bugs with it, the author and
myself. For a key component of the system, this worries me a lot. Adding more
complexity without maintainer support isn't going to help anyone.
Sorry, I'm getting confused here with earlier work Michael Ho did at BMW. The
links here:

https://lists.yoctoproject.org/g/yocto/message/52650
https://github.com/bmwcarit/meta-abicompat

are the revised version from last year which *does* hook into hash equivalence.
I'm getting two things confused, sorry.

The nice thing about the layer above is that it is a standalone layer, we don't
have to merge it in order to use it. This shows the power of the new hash
equivalence code as it is a plugin to it. We may consider merging it at some
point but there is less of a pressing need and we need time to experiment with
it.

At this point it is a proof of concept and doesn't solve the ABI problem you're
describing in the original email. Sorry about any confusion.

The abidw recipe could be useful to your ABI issue.

Cheers,

Richard


Richard Purdie
 

On Wed, 2022-02-09 at 14:13 -0500, Sinan Kaya wrote:
On 2/9/2022 1:41 PM, Richard Purdie wrote:
There are lots of levels it could be implemented at but it is something someone
would need to pick up and drive forward with a long term view to helping with
issues etc.
What would be the minimum acceptable solution with the least investment?
in other words, do we have a list of requirements?
You're after our LTS to maintain ABI. In order to do that we need help, not just
with patches generating some output, but in day to day running of the project,
i.e. help comparing output before and after changes. Whenever a patch is
submitted which breaks this, it will need attention and help some someone to
explain to that submitter what the issue, why it is important and hints on how
to fix it.
That's true, this will require engagement from the community. Tool could
take few iterations to perfect. Until then, I expect tool owner to be
responsible for fixing these bugs. Once stability is reached, it becomes
community maintained.

If the tool owner doesn't maintain and community has no interest, tool
dies and gets reverted. It is as simple as any open source engagement.

When stability is established, each code contributor to LTS would be
subject to addressing issues found before they get merged.
I agree, this either needs input from the community in order to drive it or a
sponsor. It will be interesting to see if people are interested in doing this
and I guess we can gauge that from the replies to this thread to see if people
do want to do it. I can tell you first hand how badly the existing maintainers
are burning out with the current load so we need new people.

The idea of "least investment" sends shivers down my spine since it sounds like
you want to do the absolute bare minimum to have this happen, rather than a more
community oriented approach.
It depends on your taste. I believe in smaller improvements
as opposed to throwing a big project to you that no-one will use it.
We both agree on incremental improvements and I am fine with that. I don't want
any patch acceptance taken as a sign we're goging to add a significant process
burden though. I'd prefer we have a broad agreement of what the end objective is
architecture wise too.

Everyone has different needs. We need to find the common ones.
That's why, I'm asking if there is an existing tool that works for
large part of the community accepting that there will be some folks
that won't have their needs addressed.

I'm interested in revisiting the tooling discussion and have these gaps
addressed for the biggest audience so that we can have something to
build upon.


Anyway, my point is there is more to this than just a patch. We have various
kinds of build output already and reports on test regressions - nobody helps
with them. I need some kind of a sign that ABI would be different and there are
people able to help with review on an ongoing basis, else it will just be
something else I and a small number of others become overloaded with.
Noted. Hopefully, things will be not that volatile for the LTS branch
and tool would actually help the maintainer.

In an ideal world, change needs to be stopped before that happens and
have the patch author address it similar to how you monitor build
pipelines.

Our team has posted a solution. BMW folks posted a solution.
None of them got merged.
Can you remind me of your team's please?
https://lists.yoctoproject.org/g/yocto/topic/85279259?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,160,85279259

This was an intern project from last summer that we are interested
in expanding coverage.
"None of them got merged" isn't particularly fair here. There was a brief thread
on the yocto list, no patches were proposed or reviewed and the implementation
is a standalone layer so doesn't need to merge anyway, people could use it
already if they wanted.

If the question is whether something would be accepted into core, the answer is
possibly, it would depends. The quality of the code in that layer needs
improving for core and I'm not sure about the approach. Hooking it against
buildhistory is "easy" but as I mentioned in separate discussions earlier today,
buildhistory is getting pulled in different directions (such as a SBOM) and I'm
worried about some of the extensions to it. Certainly, the ABI saving shouldn't
really be buried in a do_install postfunc and perhaps is more of a build history
item that some others that are being proposed.

This probably does need a discussion on the architecture list and we need some
discussion and decisions about where/what buildhistory could/should do. Adding
this to buildhistory is all well and good but we don't have a meaningful
integration/monitoring of existing buildhistory issues in our
autobuilder/workflow today even before adding new things.


I have to admit I can't remember what the conclusion was on your team's version
but if you remind me of the patches I can try and remember. This is a bigger
problem than just patches though sadly.
Sure, let's find out what everyone is doing.
Ok.

Cheers,

Richard


Sinan Kaya <okaya@...>
 

On 2/9/2022 4:38 PM, Richard Purdie wrote:
This probably does need a discussion on the architecture list and we need some
discussion and decisions about where/what buildhistory could/should do. Adding
this to buildhistory is all well and good but we don't have a meaningful
integration/monitoring of existing buildhistory issues in our
autobuilder/workflow today even before adding new things.
I was hoping for free cycles. I didn't get one. This will be an intern
project.

The way I'm thinking is to have the ABI compat XML be part of the state
cache tgz file and come up with a CVE check kind of hook maybe called
"ABI check" that will start flagging problems.

Would this be a better architecture?


Richard Purdie
 

On Mon, 2022-05-02 at 17:44 -0400, Sinan Kaya wrote:
On 2/9/2022 4:38 PM, Richard Purdie wrote:
This probably does need a discussion on the architecture list and we need some
discussion and decisions about where/what buildhistory could/should do. Adding
this to buildhistory is all well and good but we don't have a meaningful
integration/monitoring of existing buildhistory issues in our
autobuilder/workflow today even before adding new things.
I was hoping for free cycles. I didn't get one. This will be an intern
project.

The way I'm thinking is to have the ABI compat XML be part of the state
cache tgz file and come up with a CVE check kind of hook maybe called
"ABI check" that will start flagging problems.

Would this be a better architecture?
I'm not sure it would.

Shared state works by computing inputs into a checksum and then using that
checksum to access the cache. Finding a "previous version" or a history to
compare to therefore isn't a good fit for it.

Buildhistory does fit much better but isn't scaling in a controlled way and I
think we need the architecture discussion and an agreed plan rather than trying
to tack things onto it. That does mean someone stepping back and considering the
various needs users have rather than just an isolated use case.

Cheers,

Richard