Here's something I didn't think of before. Has this occurred anywhere
else except Ubuntu 18.04?

Alex

On Sat, 4 Jun 2022 at 13:07, Richard Purdie
<richard.purdie@...> wrote:

Hi,

Occasionally we see perl do_install failures on the autobuilder. It
looks like some kind of make race. With the last failure I saved off
the build directory and have spent a lot of time staring at it but I
don't understand how what the logs say happened actually happened.

The build failure is here:

https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5315/steps/11/logs/stdio

and you can see it failed in do_install. The first error is;

| Couldn't copy cpan/podlators/blib/script/pod2text to /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/core2-64-poky-linux/perl/5.34.1-r0/image/usr/bin/pod2text: No such file or directory
| Couldn't chmod 755 /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/core2-64-poky-linux/perl/5.34.1-r0/image/usr/bin/pod2text: No such file or directory

then

installman: Can't open cpan/podlators/blib/script/pod2text: No such file or directory
| ABORTED

Looking at the build directory, cpan/podlators/blib/script/pod2text
isn't there, cpan/podlators/script/pod2text is. I went digging in the
compile log which didn't fail but is interesting in what it doesn't say
nd the ordering. The full log:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/log.do_compile.25823

or my edited down version:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/compile-cutdown.log

and note it never actually builds cpan/podlators/blib/script/pod2text.

If I go into the failed build dir, it does build it:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/manual-compile.log

I tried a few variations of deleting and rebuilding files there just to show behaviour.

The makefiles from the podlators directory:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/Makefile
https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/Makefile.PL


What really puzzles me is the "Manifying 2 pod documents" message
*before* it then generates scripts/pod2text from scripts/pod2text.PL in
the do_compile log. It appears to run the scripts/pod2text late and
never runs the $(INST_SCRIPT)/pod2text at all (hence no cp message and
the missing file).

The Makefile has default targets of pure_all and manifypods, that
message comes from manifypods but manifypods depends on pure_all.

pure_all is a double colon rule and I'm on less certain ground with how
those behave.

Are there any make experts out there who can spot the race in this
makefile?

Cheers,

Richard

On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:

Here's something I didn't think of before. Has this occurred anywhere
else except Ubuntu 18.04?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096

I'm struggling to get the data out from the old builds, one mentions
ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.

It is possible this is an ubuntu specific make issue or a make bug.

Cheers,

Richard

On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
lists.yoctoproject.org wrote:

On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:

Here's something I didn't think of before. Has this occurred anywhere
else except Ubuntu 18.04?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096

I'm struggling to get the data out from the old builds, one mentions
ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.

It is possible this is an ubuntu specific make issue or a make bug.

Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).

I noticed these patches from 2016:

https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0

I think we may want to mandate a modern make for both this class of
issues and also perhaps for better loadavg support to keep load under
control on the autobuilders.

I'm torn, on the one hand we need to test the distros people use, on
the other we do need to remove sources of intermittent issues. I think
this bug must be some issue with make itself.

Adding a make-native dependency to perl would "hurt" people on modern
distros...

Cheers,

Richard

On Sat, 2022-06-04 at 17:12 +0200, Jacob Kroon wrote:

On 6/4/22 16:55, Khem Raj wrote:

On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie
<richard.purdie@...
<mailto:richard.purdie@...>> wrote:

On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
lists.yoctoproject.org <http://lists.yoctoproject.org> wrote:
> On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
> > Here's something I didn't think of before. Has this occurred
anywhere
> > else except Ubuntu 18.04?
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
<https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096>
>
> I'm struggling to get the data out from the old builds, one mentions
> ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.
>
> It is possible this is an ubuntu specific make issue or a make bug.

Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).

I noticed these patches from 2016:

https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
<https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a>
https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0
<https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0>

I think we may want to mandate a modern make for both this class of
issues and also perhaps for better loadavg support to keep load under
control on the autobuilders.

I'm torn, on the one hand we need to test the distros people use, on
the other we do need to remove sources of intermittent issues. I think
this bug must be some issue with make itself.

Adding a make-native dependency to perl would "hurt" people on modern
distros...


Make perhaps does not have many complex dependency needs so it might not
be as bad

My master build is already building make-native due to a dependency from
glibc, since 2018:

https://git.openembedded.org/openembedded-core/commit/?id=0cd89e4af625941f8ab8c033f72f900a2979b304

Don't know if that dependency is still valid though.

It is a fair point. We may as well add it to perl/perl-native. Centos7
still has make 3.82 but I think we now already require buildtools
tarball there so we could probably drop the glibc dependency on make-
native now.

Cheers,

Richard

On Sun, 2022-06-05 at 04:26 +0200, Jacob Kroon wrote:

On Sat, 4 Jun 2022, 19:40 Richard Purdie,
<richard.purdie@...> wrote:

On Sat, 2022-06-04 at 17:12 +0200, Jacob Kroon wrote:

On 6/4/22 16:55, Khem Raj wrote:

On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie
<richard.purdie@...
<mailto:richard.purdie@...>> wrote:

     On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
     lists.yoctoproject.org <http://lists.yoctoproject.org>
wrote:
      > On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin
wrote:
      > > Here's something I didn't think of before. Has this
occurred
     anywhere
      > > else except Ubuntu 18.04?
      >
      > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
     <https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096>
      >
      > I'm struggling to get the data out from the old builds,
one mentions
      > ubuntu1604, there is an ubuntu1804 on both x86 and arm
hosts.
      >
      > It is possible this is an ubuntu specific make issue or
a make bug.

     Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).

     I noticed these patches from 2016:

   
 https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8
319c2b153cd3d6d61e2c2882895e7c3a
   
 <https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e
8319c2b153cd3d6d61e2c2882895e7c3a>
   
 https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480a
e9cb8df4878286411f178d32db14eff0
   
 <https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480
ae9cb8df4878286411f178d32db14eff0>

     I think we may want to mandate a modern make for both this
class of
     issues and also perhaps for better loadavg support to keep
load under
     control on the autobuilders.

     I'm torn, on the one hand we need to test the distros
people use, on
     the other we do need to remove sources of intermittent
issues. I think
     this bug must be some issue with make itself.

     Adding a make-native dependency to perl would "hurt"
people on modern
     distros...


Make perhaps does not have many complex dependency needs so it
might not
be as bad

My master build is already building make-native due to a
dependency from
glibc, since 2018:

https://git.openembedded.org/openembedded-core/commit/?id=0cd89e4af625941f8ab8c033f72f900a2979b304

Don't know if that dependency is still valid though.

It is a fair point. We may as well add it to perl/perl-native.
Centos7
still has make 3.82 but I think we now already require buildtools
tarball there so we could probably drop the glibc dependency on
make-
native now.

Would it be a bad idea to add make-native to DEPENDS depending on
whether the host version of make is new enough or not ? Would it
break sstate cache reuse in some way ? 

We can't have a conditional dependency like that, the task checksums as
implemented today wouldn't work and it would break ssttate reuse.

Cheers,

Richard