Re: dbus-wait git failures on the autobuilder


Michael Halstead <mhalstead@...>
 

On Thu, Feb 17, 2022 at 8:54 AM Richard Purdie <richard.purdie@...> wrote:
After the discussion in triage I tried to test some of the theories mentioned on
the call. There was this failed build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3169/steps/14/logs/stdio

from alma8-ty-1.yocto.io, so I logged in there, restored the exact same build
directory which failed as it hadn't been cleaned up yet and re-ran the test.

[pokybuild@alma8-ty-1 build]$ oe-selftest -r fetch.Fetch.test_git_mirrors
2022-02-17 16:46:05,158 - oe-selftest - INFO - Adding layer libraries:
2022-02-17 16:46:05,158 - oe-selftest - INFO -  /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta-poky/lib
2022-02-17 16:46:05,158 - oe-selftest - INFO -  /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib
2022-02-17 16:46:05,159 - oe-selftest - INFO -  /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta-yocto-bsp/lib
2022-02-17 16:46:05,159 - oe-selftest - INFO -  /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta-selftest/lib
2022-02-17 16:46:05,160 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2022-02-17 16:46:07,875 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st/conf/local.conf
2022-02-17 16:46:07,875 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2022-02-17 16:46:07,876 - oe-selftest - INFO - test_git_mirrors (fetch.Fetch)
2022-02-17 16:47:18,909 - oe-selftest - INFO -  ... ok
2022-02-17 16:47:19,960 - oe-selftest - INFO - ----------------------------------------------------------------------
2022-02-17 16:47:19,960 - oe-selftest - INFO - Ran 1 test in 73.472s
2022-02-17 16:47:19,960 - oe-selftest - INFO - OK
2022-02-17 16:47:24,497 - oe-selftest - INFO - RESULTS:
2022-02-17 16:47:24,498 - oe-selftest - INFO - RESULTS - fetch.Fetch.test_git_mirrors: PASSED (71.03s)
2022-02-17 16:47:24,502 - oe-selftest - INFO - SUMMARY:
2022-02-17 16:47:24,503 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 73.473s
2022-02-17 16:47:24,503 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)

so it worked.

This tells is that the issue is probably not a cert issue on the worker or
something worker specific but some transient issue on the network or upstream
server side of things :/.

Michael: Is there any reason git://git.yoctoproject.org/dbus-wait;branch=master
would be unreliable to clone? Are we hitting some kind of rate limiting on the
upstream git services?


There are no rate limiting rules in place on that server. There is a load balancer in front of it that shows several connections reset around the time of the error. I don't have detailed logging about the connection resets but I suspect issues are on the autobuilder data center side not on the git.yoctoproject.org mirror.
 

This being some network/server issue does fit with several selftests failing at
the same time with the same error (3 our of 4 selftests failed at the same time
for the failure above).

Cheers,

Richard



--
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engineer

Join swat@lists.yoctoproject.org to automatically receive all group messages.