Re: Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python


Georgi Georgiev
 

Another serious issue with svn fetcher, always occurring when doing incremental builds. I am putting it here because I think both issues are related. Please suggest what to do. The error we get is shown below. This time it happens always. The only workaround is to clean and build. As soon as we make modification of the recipe (PV change only) it fails:

ERROR: ww-pacman-121204-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:base_do_fetch(d)
0003:
File: '/data/home/w23698/projects/proj/build/../sources/poky/meta/classes/base.bbclass', lineno: 163, function: base_do_fetch
0159: return
0160:
0161: try:
0162: fetcher = bb.fetch2.Fetch(src_uri, d)
*** 0163: fetcher.download()
0164: except bb.fetch2.BBFetchException as e:
0165: bb.fatal(str(e))
0166:}
0167:
File: '/data/home/w23698/projects/proj/sources/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1678, function: download
1674: try:
1675: if not trusted_network(self.d, ud.url):
1676: raise UntrustedUrl(ud.url)
1677: logger.debug(1, "Trying Upstream")
*** 1678: m.download(ud, self.d)
1679: if hasattr(m, "build_mirror_data"):
1680: m.build_mirror_data(ud, self.d)
1681: localpath = ud.localpath
1682: # early checksum verify, so that if checksum mismatched,
File: '/data/home/w23698/projects/proj/sources/poky/bitbake/lib/bb/fetch2/svn.py', lineno: 150, function: download
0146: if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
0147: # Warn the user if this had externals (won't catch them all)
0148: output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir)
0149: if output:
*** 0150: if "--ignore-externals" in svnfetchcmd.split():
0151: bb.warn("%s contains svn:externals." % ud.url)
0152: bb.warn("These should be added to the recipe SRC_URI as necessary.")
0153: bb.warn("svn fetch has ignored externals:\n%s" % output)
0154: bb.warn("To disable this warning add ';externals=nowarn' to the url.")
Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment

ERROR: Logfile of failure stored in: /data/home/w23698/projects/proj/build/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121204-r0/temp/log.do_fetch.18853

-----Original Message-----
From: Georgi Georgiev
Sent: Wednesday, February 05, 2020 6:57 PM
To: Georgi Georgiev <Georgi.Georgiev@woodward.com>; Khem Raj <raj.khem@gmail.com>; yocto@lists.yoctoproject.org
Subject: Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python

Sorry Khem,
With esc character '\' before & it can't take the full path. So briefly:

Yocto build with char '\' before & in SRC_URI:
online - OK
offline - Error - svn: E170013: Unable to connect to a repository...
Yocto build without '\' in SRC_URI:
online and offline same error - do_unpack: Unpack failure for URL - "package".tar.gz is present in DL_DIR

Thanks



-----Original Message-----
From: yocto@lists.yoctoproject.org [mailto:yocto@lists.yoctoproject.org] On Behalf Of Georgi Georgiev via Lists.Yoctoproject.Org
Sent: Wednesday, February 05, 2020 5:26 PM
To: Khem Raj <raj.khem@gmail.com>; yocto@lists.yoctoproject.org
Cc: yocto@lists.yoctoproject.org
Subject: Re: [EXTERNAL] Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python

Hi Khem,

Yes, no issues with that. When I am connected to network it fetches the code with and without escape character before & in the path.
svn co " https://urldefense.proofpoint.com/v2/url?u=http-3A__cocosubversion_svn_Embedded_Valve-26Actuator_DVPII_trunk_SOCPACManEnvEngKeys&d=DwIGaQ&c=y6L7g950KfMp92YmLM0QlMdXcRn6b-Cq4AApnSJOenE&r=kHtJrQGzfH1ZmfsNkJpYuH-jtNpv8yMDkqAmsRP99mc&m=vXia-xlcxuLr2fSmBogZpMIaOntXLKWv3mkxcRgtUc4&s=UYH9lueDIeMGZndMxgbZgb76A5EfqN-MR58DBv1iriI&e= " works fine and fetches the package.

Georgi

-----Original Message-----
From: Khem Raj [mailto:raj.khem@gmail.com]
Sent: Friday, January 31, 2020 8:33 PM
To: Georgi Georgiev <Georgi.Georgiev@woodward.com>; yocto@lists.yoctoproject.org
Subject: [EXTERNAL] Re: [yocto] Yocto [thud], [zeus] do_fetch and do_unpack failures with offline/online svn build! #yocto #python

On 1/31/20 4:02 AM, georgi.georgiev via Lists.Yoctoproject.Org wrote:
Hello Community,
This is the third time I am asking for support on this issue. This
time I decided to use the web form.
In our project we have a requirement to be able to build the project
offline. E.g. on the field without any network connection. When we are
connected with the recipe mentioned below we don't have issues:

svn: E170013: Unable to connect to a repository at URL 'https://urldefense.proofpoint.com/v2/url?u=http-3A__cocosubversion_svn_Embedded_Valve-26Actuator_DVPII_trunk_SOCPACManEnvEngKeys_trunk&d=DwIGaQ&c=y6L7g950KfMp92YmLM0QlMdXcRn6b-Cq4AApnSJOenE&r=kHtJrQGzfH1ZmfsNkJpYuH-jtNpv8yMDkqAmsRP99mc&m=ocx6BoVSrDQGWUvUUVvgKtuJbT7eH7jFSjCy1Ys73Vw&s=OTunnmsi_tCUt3JytmU2Hs7i7Xnqhl8-2CMsqmzIl90&e= '
svn: E670003: Temporary failure in name resolution

can you try checking out the repo out side of fetcher and see if the machine can fetch the url you are using in SRC_URI ?

*SUMMARY = "PACMan - Parameter And Configuration MANager"* *LICENSE =
"CLOSED"* *inherit systemd useradd* *REQUIRED_DISTRO_FEATURES =
"systemd"* *# SVN revision* *PV = "121026"* *# Name of SVN project*
*PACMAN_PROJ_NAME="SOCPACManEnvEngKeys"*
*SRC_URI =
"svn://cocosubversion/svn/Embedded/Valve%5C&Actuator/DVPII/trunk/$%7BP
ACMAN_PROJ_NAME%7D;module=trunk;protocol=http;externals=allowed;rev=$%
7BPV%7D%22* *SRC_URI += "file://ww-authpacman.service%22* *SRC_URI +=
"file://ww-pacman.service%22* *S = "${WORKDIR}/trunk/"* *# ${PN}-sys:
content related to system, which goes to base rootfs (only .service
file and symlinks)* *# ${PN}:      real content which may go to
separate partition* *PACKAGES =+ " ${PN}-sys"* .........
When disconnect the network, erase sstate-cache, cache and tmp I see
log file attached (log.do_fetch.32757) and the following output:

*ERROR: ww-pacman-121026-r0 do_fetch: Fetcher failure: Fetch command
export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="11412"; export
SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/home/w23698/projects/proj_dvp2/build_dvp2/tmp/sysroots-uninativ
e/x86_64-linux/usr/bin:/home/w23698/projects/proj_dvp2/sources/poky/sc
ripts:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-n
eon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/b
in/arm-poky-linux-gnueabi:/home/w23698/projects/proj_dvp2/build_dvp2/t
mp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-
sysroot/usr/bin/crossscripts:/home/w23698/projects/proj_dvp2/build_dvp
2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/reci
pe-sysroot-native/usr/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/
tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe
-sysroot-native/usr/bin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp
/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sy
sroot-native/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/
cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-
native/bin:/home/w23698/projects/proj_dvp2/sources/poky/bitbake/bin:/h
ome/w23698/projects/proj_dvp2/build_dvp2/tmp/hosttools";
export HOME="/home/w23698"; /usr/bin/env svn --non-interactive
--trust-server-cert update --no-auth-cache -r 121026 failed with exit
code 1, output:* *Updating '.':*
*svn: E170013: Unable to connect to a repository at URL
'https://urldefense.proofpoint.com/v2/url?u=http-3A__cocosubversion_sv
n_Embedded_Valve-26Actuator_DVPII_trunk_SOCPACManEnvEngKeys_trunk-27-2
A&d=DwIGaQ&c=y6L7g950KfMp92YmLM0QlMdXcRn6b-Cq4AApnSJOenE&r=kHtJrQGzfH1
ZmfsNkJpYuH-jtNpv8yMDkqAmsRP99mc&m=ocx6BoVSrDQGWUvUUVvgKtuJbT7eH7jFSjC
y1Ys73Vw&s=Ffv1JU1QYBh4g49fmoLnsDSFgMMBc_5MbOpy59QUS18&e=
*svn: E670003: Temporary failure in name resolution*
*ERROR: ww-pacman-121026-r0 do_fetch: Fetcher failure for URL:
'svn://cocosubversion/svn/Embedded/Valve%5C&Actuator/DVPII/trunk/SOCPACManEnvEngKeys;module=trunk;protocol=http;externals=allowed;rev=121026'.
Unable to fetch URL from any source.*
*ERROR: Logfile of failure stored in:
/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-po
ky-linux-gnueabi/ww-pacman/121026-r0/temp/log.do_fetch.32757*
*ERROR: Task
(/home/w23698/projects/proj_dvp2/build_dvp2/../sources/meta-ww-dvp2/re
cipes-ww/ww-pacman/ww-pacman.bb:do_fetch)
failed with exit code '1'*

When remove the '\' character in SRC_URI, e.g. to become:

*SRC_URI =
"svn://cocosubversion/svn/Embedded/Valve&Actuator/DVPII/trunk/$%7BPACM
AN_PROJ_NAME%7D;module=trunk;protocol=http;externals=allowed;rev=$%7BP
V%7D%22*

In connected and not connected to network do_fetch() passes
successfully but I see one and same error (log.do_unpack.25226) output:

*ERROR: ww-pacman-121026-r0 do_unpack: Unpack failure for URL:
'svn://cocosubversion/svn/Embedded/Valve&Actuator/DVPII/trunk/SOCPACManEnvEngKeys;module=trunk;protocol=http;externals=allowed;rev=121026'.
Unpack command
PATH="/home/w23698/projects/proj_dvp2/build_dvp2/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/w23698/projects/proj_dvp2/sources/poky/scripts:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot/usr/bin/crossscripts:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/usr/bin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/sbin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/ww-pacman/121026-r0/recipe-sysroot-native/bin:/home/w23698/projects/proj_dvp2/sources/poky/bitbake/bin:/home/w23698/projects/proj_dvp2/build_dvp2/tmp/hosttools"
tar xz --no-same-owner -f
/home/w23698/projects/proj_dvp2/build_dvp2/downloads/trunk_cocosubvers
ion_.svn.Embedded.Valve&Actuator.DVPII.trunk.SOCPACManEnvEngKeys_12102
6_.tar.gz
failed with return value 127*
*ERROR: Logfile of failure stored in:
/home/w23698/projects/proj_dvp2/build_dvp2/tmp/work/cortexa9hf-neon-po
ky-linux-gnueabi/ww-pacman/121026-r0/temp/log.do_unpack.25226*
*ERROR: Task
(/home/w23698/projects/proj_dvp2/build_dvp2/../sources/meta-ww-dvp2/re
cipes-ww/ww-pacman/ww-pacman.bb:do_unpack)
failed with exit code '1'
*
I don't it this matters but, the build machine is baremetal Ubuntu
18.04.3 LTS. In all cases the packed tar.gz remain in downloads
directory with one name!

Cordially,
Georgi




***
The information in this email is confidential and intended solely for the individual or entity to whom it is addressed.  If you have received this email in error please notify the sender by return e-mail, delete this email, and refrain from any disclosure or action based on the information.
***

***
The information in this email is confidential and intended solely for the individual or entity to whom it is addressed.  If you have received this email in error please notify the sender by return e-mail, delete this email, and refrain from any disclosure or action based on the information.
***

Join yocto@lists.yoctoproject.org to automatically receive all group messages.