Yocto Technical Team Minutes, Engineering Sync, for March 30, 2021
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner, Stephen Jolley, Armin Kuster, Richard Purdie, Randy
MacLeod, Jan-Simon Möller, Trevor Gamblin, Tim Orling, Michael Halstead,
Paul Barker, Jon Mason, Joshua Watt, Steven Sakoman, Saul Wold, Scott
Murray, Khem Raj, Peter Kjellerstedt (saur), Ross Burton, Michael
Opdenacker

== notes ==
- 3.2-m3 released
- -m4 being built this week, no more update patches

== general ==
Randy: got info collection thingy working on AB thanks to help from SS and
integrated into results
RP: have you had a chance to look at my theory
Randy: not yet
RP: io-nice doesn’t change async write (I think AlexK pointed this out). we
do nice levels for all processes and io-nice for things that write, it’s
possible that this is where we’re hitting the io block. we might want to
try doing some builds under qemu (then we can enforce more nice levels) as
a means of investigating this issue


RP: the valgrind stuff seems to be testing well in master-next, but i will
defer those to -m4
RP: the -m4 build date is, technically, monday, anything else to include?
Randy: lttng modules?


Khem: i have gcc-11 patches, probably not for -m4, but if the AB has cycles
can i send them up?
RP: on the one hand i’d prefer to not do multiple things at the same time,
on the other hand i can’t stop people from sending new features
Khem: i can hold off a bit. i could send an RFC. yocto project has already
submitted gcc-11 patches and i’m looking to expand the testing on it
RP: any issues for -native recipes?
Khem: i’ve found a bunch already and upstreamed them, any that weren’t
accepted in the upstream project have been put into oe-core


Peter: what about the covered/not-covered (on runqueue) issue
RP: i have drafted a patch and is in master-next. it doesn’t appear to break
anything, but proving it fixes things is hard. have you seen the problem
after adding that patch?
Peter: no
SS: i had seen it on 3 builds, and haven’t seen them again either
RP: trying to reproduce an exact same build (wrt sstate, runqueues, etc) is
hard, so it’s hard to know if we’ve solved it


Khem: in the past we’ve seen issues with bitbake running forever due to
“make -l” issue. so we removed it. but we still see long builds with
qemarm
Randy: how much longer?
Khem: (guessing) 30%
Randy: qemuarm64 too?
Khem: no, just qemuarm. same number of CPUs, same underlying machine
Jon: qemuarmv5 as well?
Khem: no
Jon: maybe it’s a qemuarm hardfloat issue. i’ll try to reproduce it
Khem: i’m building lots of stuff, not just the basics. mostly doing world
builds, so maybe it’s a particular package
<various>: we can give it a try
RP: maybe by dropping the -l you’ve introduced something else (e.g. swapping)
Khem: but no other qemu’s are affected. most builds finish in 10 hours,
but qemuarm gets killed after 13 hours (due to a timeout) which is why i
notice it (because it gets killed)
RP: it would be interesting to look at the build stats data then compare build
times recipe-to-recipe
Randy: is there a bug filed?
Khem: not yet, i can file a tracking bug


TimO: Ross asked me about splitting meta-python out to standalone layer (from
meta-openembedded). would allow us to use dynamic layers. might help with
ptest coverage. it might might other workflows possible (i.e. move it
to gitlab and use their things). some have offered strong opinions that
it’s just splitting for splitting sake
RP: i think splitting out would be good. there’s lots of burnout and it
would be great to spread the load a bit more (instead of making one person
responsible for all meta-oe)
TrevorG: having to share the support with the rest of meta-oe is hard
TimO: Khem had asked us to move to a PR mechanism
JPEW: this subset feels like the best one to be pulled out (as a test)
TimO: true, meta-perl might be a good candidate too. in any case i would start
with dunfell, but nothing before that
Khem: have you considered this from the users point of view. obviously we’re
thinking more of the maintenance point of view but let’s not forget to
seek user input
TimO: obviously it’s going to impact users, they’ll need to change URIs
and bblayers
PaulB: just make sure the layer index gets updated and we could leave a stub
behind that points the user to the change
RP: (same)
Khem: i also think it would help if these layers have separate mailing list.
it’s hard for users to know where to send patches (especially poky). so
instead of using mete-oe as a kitchen sink
TimO: i’ve always wished for a separate mailing list. i’ve had to do lots
of filtering but it doesn’t catch everything
Khem: good to bring to TSC and community to get input
TimO: it’s fairly quick to split this out. mechanics aren’t a problem
Armin: looking at layers that depend on meta-python, do you think they would
change to dynamic layers? meta-networking and meta-multimedia depend on
meta-python, so splitting it out would require pulling it in for many
use-cases
RP: i wouldn't get too worried about the details, everything could be
solved, let’s not worry about all possibilities, when only some might
come to pass. i think more layers is good, maybe it opens up the ability
to use the AB for some of this stuff (as stuff gets smaller)
Randy: so the reasons for are:
1. burden on maintainers
2. quality and test coverage
?
TimO: also simplification (if you want some python you don’t need all of
meta-oe)
RP: i see dependency creep in external layers (since you have to pull in
meta-oe for meta-python then we might as well pull in all these other
things too that we’re already getting in meta-oe)
Randy: do we want to see everything in meta-oe split up, or other splits in
oe-core?
RP: case by case, there’s pros and cons
Randy: what about incremental transition vs a full change-over
TimO: i want to make sure all the ptests are in place first, and there’s the
question of review
Khem: why can’t we add ptests now? (i.e. in meta-oe)
TimO: i’d *like* to add more ptests, but doesn’t mean we will/should
Khem: propose it and see what the feedback is. people are making products with
this and we should make sure to move forward in a way the impacts them in
a way they’re happy with
TimO: 10 people will provide 20 opinions
RP: we can do a POC and move forward
TimO: we’ve talked about it so much over the years and we never agree on
trying a change, maybe people can’t see the benefit until they actually
try it
Randy: could this live on git.yoctoproject.org
TimO: maybe git.openembedded.org instead
Randy: would you track bugs on YP bugzilla
RP: where to host is irrelevant, but there are larger issues (such as better
AB usage, maintainer question, burnout, mailing lists)
Khem: lots of people listed as maintainers of individual packages, but little
actual maintainer help. how many maintainers actually review and help?
RP: the same problem exists in oe-core. there are some that are active, but in
the end it’s usually AlexK that does the upgrades
TimO: it could be an issue of timing, sometime you don’t get to it in a
day and someone else does it. with unlimited resources we would use yp
bugzilla, but there are consequences to the members
Randy: do you want more layers removed too?
Khem: yes, that would be great. i’m not maintainer by choice, then i could
sign up to maintain what matters to me
TrevorG: if we do it now with meta-python, then it lets us know if we should
go ahead and do it with others too
RP: yes, good experiment and we can roll it back. meta-python looks like it
would be a good test
TimO: i have tools that make it easy to do, and then easy to roll back. and
the tools are not just meta-python-specific, they could be used for any
sub-layer


TrevorW: there is a Yocto Project related virtual conference being planned:
https://www.yoctoproject.org/yocto-project-virtual-summit-2021/
TrevorW: the CfP is now open:
https://pretalx.com/yocto-project-summit-2021/cfp

On Wed, Mar 31, 2021 at 4:25 PM Trevor Woerner <twoerner@...> wrote:

Yocto Technical Team Minutes, Engineering Sync, for March 30, 2021
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner, Stephen Jolley, Armin Kuster, Richard Purdie, Randy
MacLeod, Jan-Simon Möller, Trevor Gamblin, Tim Orling, Michael Halstead,
Paul Barker, Jon Mason, Joshua Watt, Steven Sakoman, Saul Wold, Scott
Murray, Khem Raj, Peter Kjellerstedt (saur), Ross Burton, Michael
Opdenacker

== notes ==
- 3.2-m3 released
- -m4 being built this week, no more update patches

== general ==
Randy: got info collection thingy working on AB thanks to help from SS and
integrated into results
RP: have you had a chance to look at my theory
Randy: not yet
RP: io-nice doesn’t change async write (I think AlexK pointed this out). we
do nice levels for all processes and io-nice for things that write, it’s
possible that this is where we’re hitting the io block. we might want to
try doing some builds under qemu (then we can enforce more nice levels) as
a means of investigating this issue


RP: the valgrind stuff seems to be testing well in master-next, but i will
defer those to -m4
RP: the -m4 build date is, technically, monday, anything else to include?
Randy: lttng modules?


Khem: i have gcc-11 patches, probably not for -m4, but if the AB has cycles
can i send them up?
RP: on the one hand i’d prefer to not do multiple things at the same time,
on the other hand i can’t stop people from sending new features
Khem: i can hold off a bit. i could send an RFC. yocto project has already
submitted gcc-11 patches and i’m looking to expand the testing on it
RP: any issues for -native recipes?
Khem: i’ve found a bunch already and upstreamed them, any that weren’t
accepted in the upstream project have been put into oe-core


Peter: what about the covered/not-covered (on runqueue) issue
RP: i have drafted a patch and is in master-next. it doesn’t appear to break
anything, but proving it fixes things is hard. have you seen the problem
after adding that patch?
Peter: no
SS: i had seen it on 3 builds, and haven’t seen them again either
RP: trying to reproduce an exact same build (wrt sstate, runqueues, etc) is
hard, so it’s hard to know if we’ve solved it


Khem: in the past we’ve seen issues with bitbake running forever due to
“make -l” issue. so we removed it. but we still see long builds with
qemarm
Randy: how much longer?
Khem: (guessing) 30%
Randy: qemuarm64 too?
Khem: no, just qemuarm. same number of CPUs, same underlying machine
Jon: qemuarmv5 as well?
Khem: no
Jon: maybe it’s a qemuarm hardfloat issue. i’ll try to reproduce it
Khem: i’m building lots of stuff, not just the basics. mostly doing world
builds, so maybe it’s a particular package
<various>: we can give it a try
RP: maybe by dropping the -l you’ve introduced something else (e.g. swapping)
Khem: but no other qemu’s are affected. most builds finish in 10 hours,
but qemuarm gets killed after 13 hours (due to a timeout) which is why i
notice it (because it gets killed)
RP: it would be interesting to look at the build stats data then compare build
times recipe-to-recipe
Randy: is there a bug filed?
Khem: not yet, i can file a tracking bug


TimO: Ross asked me about splitting meta-python out to standalone layer (from
meta-openembedded). would allow us to use dynamic layers. might help with
ptest coverage. it might might other workflows possible (i.e. move it
to gitlab and use their things). some have offered strong opinions that
it’s just splitting for splitting sake
RP: i think splitting out would be good. there’s lots of burnout and it
would be great to spread the load a bit more (instead of making one person
responsible for all meta-oe)
TrevorG: having to share the support with the rest of meta-oe is hard
TimO: Khem had asked us to move to a PR mechanism
JPEW: this subset feels like the best one to be pulled out (as a test)
TimO: true, meta-perl might be a good candidate too. in any case i would start
with dunfell, but nothing before that

I think this is a really good discussion, but I wonder why you want to
go back in time.. I expect users of dunfell branch (which is an LTS
from YP perspectives) would be rather unhappy to see such a major
change in their stable builds. I think it's a change for master only,
so either now before the next release or a change for the Oct release,
no?

Khem: have you considered this from the users point of view. obviously we’re
thinking more of the maintenance point of view but let’s not forget to
seek user input
TimO: obviously it’s going to impact users, they’ll need to change URIs
and bblayers
PaulB: just make sure the layer index gets updated and we could leave a stub
behind that points the user to the change
RP: (same)
Khem: i also think it would help if these layers have separate mailing list.
it’s hard for users to know where to send patches (especially poky). so
instead of using mete-oe as a kitchen sink
TimO: i’ve always wished for a separate mailing list. i’ve had to do lots
of filtering but it doesn’t catch everything
Khem: good to bring to TSC and community to get input
TimO: it’s fairly quick to split this out. mechanics aren’t a problem
Armin: looking at layers that depend on meta-python, do you think they would
change to dynamic layers? meta-networking and meta-multimedia depend on
meta-python, so splitting it out would require pulling it in for many
use-cases
RP: i wouldn't get too worried about the details, everything could be
solved, let’s not worry about all possibilities, when only some might
come to pass. i think more layers is good, maybe it opens up the ability
to use the AB for some of this stuff (as stuff gets smaller)
Randy: so the reasons for are:
1. burden on maintainers
2. quality and test coverage
?
TimO: also simplification (if you want some python you don’t need all of
meta-oe)
RP: i see dependency creep in external layers (since you have to pull in
meta-oe for meta-python then we might as well pull in all these other
things too that we’re already getting in meta-oe)
Randy: do we want to see everything in meta-oe split up, or other splits in
oe-core?
RP: case by case, there’s pros and cons
Randy: what about incremental transition vs a full change-over
TimO: i want to make sure all the ptests are in place first, and there’s the
question of review
Khem: why can’t we add ptests now? (i.e. in meta-oe)
TimO: i’d *like* to add more ptests, but doesn’t mean we will/should
Khem: propose it and see what the feedback is. people are making products with
this and we should make sure to move forward in a way the impacts them in
a way they’re happy with
TimO: 10 people will provide 20 opinions
RP: we can do a POC and move forward
TimO: we’ve talked about it so much over the years and we never agree on
trying a change, maybe people can’t see the benefit until they actually
try it
Randy: could this live on git.yoctoproject.org
TimO: maybe git.openembedded.org instead
Randy: would you track bugs on YP bugzilla
RP: where to host is irrelevant, but there are larger issues (such as better
AB usage, maintainer question, burnout, mailing lists)
Khem: lots of people listed as maintainers of individual packages, but little
actual maintainer help. how many maintainers actually review and help?
RP: the same problem exists in oe-core. there are some that are active, but in
the end it’s usually AlexK that does the upgrades
TimO: it could be an issue of timing, sometime you don’t get to it in a
day and someone else does it. with unlimited resources we would use yp
bugzilla, but there are consequences to the members
Randy: do you want more layers removed too?
Khem: yes, that would be great. i’m not maintainer by choice, then i could
sign up to maintain what matters to me
TrevorG: if we do it now with meta-python, then it lets us know if we should
go ahead and do it with others too
RP: yes, good experiment and we can roll it back. meta-python looks like it
would be a good test
TimO: i have tools that make it easy to do, and then easy to roll back. and
the tools are not just meta-python-specific, they could be used for any
sub-layer


TrevorW: there is a Yocto Project related virtual conference being planned:
https://www.yoctoproject.org/yocto-project-virtual-summit-2021/
TrevorW: the CfP is now open:
https://pretalx.com/yocto-project-summit-2021/cfp