On Fri, 2021-01-29 at 13:48 -0600, Andrew Geissler wrote:

On Jan 29, 2021, at 10:54 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:

On Fri, 2021-01-29 at 08:40 -0800, Andrew Geissler wrote:

What was SDKMACHINE set to before?

Hey Richard, thanks for the quick response.

We did not have it set. It looks like this recent commit is what
changed things on us:

commit c74ec1dd7393b9dc7bec1a3ca2ed0a56fb18d8fb
Author: Ross Burton <ross@burtonini.com>
Date: Tue Dec 22 17:23:14 2020 +0000

    bitbake.conf: default SDKMACHINE to the build host architecture


Looks like I need to submit a patch to y'all to add a ppc64le.conf to
conf/machine-sdk/?

That would likely avoid the failures you're seeing, yes.

I really wish we had better support for ppc64le both for hosts and
targets and had people actively involved/maintaining the ppc support,
its in need of massive modernisation particularly on the target side.

Darn, I tried just adding this:

cat meta/conf/machine-sdk/ppc64le.conf

SDK_ARCH = "ppc64le"
ABIEXTENSION_class-nativesdk = ""

I get further but then hit a bunch of fails like this:

ERROR: /home/andrewg/Code/openbmc/meta/recipes-devtools/meson/nativesdk-meson_0.56.0.bb: Unable to determine endianness for architecture '${SDK_ARCH}'
ERROR: /home/andrewg/Code/openbmc/meta/recipes-devtools/meson/nativesdk-meson_0.56.0.bb: Please add your architecture to siteinfo.bbclass
ERROR: Failed to parse recipe: /home/andrewg/Code/openbmc/meta/recipes-devtools/meson/nativesdk-meson_0.56.0.bb
ERROR: /home/andrewg/Code/openbmc/meta/recipes-graphics/xorg-lib/libxext_1.3.4.bb: Unable to determine endianness for architecture '${SDK_ARCH}'
ERROR: /home/andrewg/Code/openbmc/meta/recipes-graphics/xorg-lib/libxext_1.3.4.bb: Please add your architecture to siteinfo.bbclass
…

I’m did some grepping around but don’t see anything obvious. Why is it not
finding the SDK_ARCH I’ve set in the new conf file?

Wondering if I should just go with my first workaround and set SDKMACHINE=x86_64.
Apparently using that default in the past hasn’t affected our use cases.

SDKMACHINE is the machine to run any SDK that is built on. If you're
not building an SDK or eSDK, it won't matter to your use case.
BUILD_ARCH is the machine you're building upon which is already being
set correctly and unchanged.

Cheers,

Richard

The issue here is that you are overwriting the DEPENDS after the inherit.
Two ways to solve this problem

Either use

DEPENDS = "python3-astropy-helpers"
PYPI_PACKAGE = "astropy"
inherit pypi


or

PYPI_PACKAGE = "astropy"
inherit pypi

DEPENDS += "python3-astropy-helpers"

python3-setuptools doesn't need to be added at all manually as this is done implicitly by setuptools3 class

I'm attempting to make a simple bitbake recipe for python3-astropy that inherits from pypi and setuptools3.  The very simple recipe is attached.
When I build I'm getting this error:

|     import pkg_resources

| ModuleNotFoundError: No module named 'pkg_resources'

| WARNING: exit code 1 from a shell command.

I don't understand this error because I'm using setuptools3 which I believe contains pkg_resources.  In desperation I added a "DEPENDS = "python3-setuptools" but that did not help either.  I'd appreciate any advice on this.
Thanks

On 2021-01-29 3:11 p.m., Randy MacLeod wrote:

Did you figure this out?

Oops, I see that Khem has replied to a later email.

--
# Randy MacLeod
# Wind River Linux

On 2021-01-04 3:43 a.m., safouane maaloul wrote:

Bonjour, j'essaye d'ajouter le aom plugin au niveau de la gstreamer1.0-plugins-bad. J'ai basculé sur gatesgarth version. Je commence à faire lebuild. J'avais un problème de lisence. Je l'ai résolu. Et maintenant j'ai un problème "no such file or directory automake-native yocto" et j'arrive pas à la résoudre ? Est-ce que vous pouvez m'aider ? y a t il quelqu'un qui a essayé d'ajouter av1 codec (aom plugins/gstreamer1.0-plugins-bad).
Cordialement,
Safouane.Maaloul

Bonjour Safouane.Maaloul,

Did you figure this out?

The Yocto email list is >99.99% English so please consider translating
before posting:

Google translate says:

Hello, I'm trying to add the aom plugin to the level of the
gstreamer1.0-plugins-bad. I switched to gatesgarth version. I start to
do a build. I had a license problem. I solved it. And now I have a "no
such file or directory automake-native yocto" problem and I can't
solve it? Can you help me ? has anyone tried adding av1 codec (aom
plugins / gstreamer1.0-plugins-bad).

I don't use gstreamer but if you answer the following questions,
you may get an answer:

What version of oe-core/poky are you using?
What's the top level commit id?
What image are you using and what changes have you made
to your local.conf file?

If you are only using oe-core or poky and have a reproducible
problem and no one helps, you can open a defect as decribed here:

http://docs.yoctoproject.org/dev-manual/common-tasks.html#submitting-a-defect-against-the-yocto-project

../Randy

--
# Randy MacLeod
# Wind River Linux

On Jan 29, 2021, at 10:54 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:

On Fri, 2021-01-29 at 08:40 -0800, Andrew Geissler wrote:

What was SDKMACHINE set to before?

Hey Richard, thanks for the quick response.

We did not have it set. It looks like this recent commit is what
changed things on us:

commit c74ec1dd7393b9dc7bec1a3ca2ed0a56fb18d8fb
Author: Ross Burton <ross@burtonini.com>
Date: Tue Dec 22 17:23:14 2020 +0000

bitbake.conf: default SDKMACHINE to the build host architecture


Looks like I need to submit a patch to y'all to add a ppc64le.conf to
conf/machine-sdk/?

That would likely avoid the failures you're seeing, yes.

I really wish we had better support for ppc64le both for hosts and
targets and had people actively involved/maintaining the ppc support,
its in need of massive modernisation particularly on the target side.

Darn, I tried just adding this:

cat meta/conf/machine-sdk/ppc64le.conf

SDK_ARCH = "ppc64le"
ABIEXTENSION_class-nativesdk = ""

I get further but then hit a bunch of fails like this:

ERROR: /home/andrewg/Code/openbmc/meta/recipes-devtools/meson/nativesdk-meson_0.56.0.bb: Unable to determine endianness for architecture '${SDK_ARCH}'
ERROR: /home/andrewg/Code/openbmc/meta/recipes-devtools/meson/nativesdk-meson_0.56.0.bb: Please add your architecture to siteinfo.bbclass
ERROR: Failed to parse recipe: /home/andrewg/Code/openbmc/meta/recipes-devtools/meson/nativesdk-meson_0.56.0.bb
ERROR: /home/andrewg/Code/openbmc/meta/recipes-graphics/xorg-lib/libxext_1.3.4.bb: Unable to determine endianness for architecture '${SDK_ARCH}'
ERROR: /home/andrewg/Code/openbmc/meta/recipes-graphics/xorg-lib/libxext_1.3.4.bb: Please add your architecture to siteinfo.bbclass
…

I’m did some grepping around but don’t see anything obvious. Why is it not
finding the SDK_ARCH I’ve set in the new conf file?

Wondering if I should just go with my first workaround and set SDKMACHINE=x86_64.
Apparently using that default in the past hasn’t affected our use cases.

Cheers,

Richard

On Fri, 2021-01-29 at 08:40 -0800, Andrew Geissler wrote:

What was SDKMACHINE set to before?

Hey Richard, thanks for the quick response.

We did not have it set. It looks like this recent commit is what
changed things on us:

commit c74ec1dd7393b9dc7bec1a3ca2ed0a56fb18d8fb
Author: Ross Burton <ross@burtonini.com>
Date:   Tue Dec 22 17:23:14 2020 +0000

    bitbake.conf: default SDKMACHINE to the build host architecture


Looks like I need to submit a patch to y'all to add a ppc64le.conf to
conf/machine-sdk/?

That would likely avoid the failures you're seeing, yes.

I really wish we had better support for ppc64le both for hosts and
targets and had people actively involved/maintaining the ppc support,
its in need of massive modernisation particularly on the target side.

Cheers,

Richard

> What was SDKMACHINE set to before?

Hey Richard, thanks for the quick response.

We did not have it set. It looks like this recent commit is what changed things on us:

commit c74ec1dd7393b9dc7bec1a3ca2ed0a56fb18d8fb
Author: Ross Burton <ross@...>
Date:   Tue Dec 22 17:23:14 2020 +0000

    bitbake.conf: default SDKMACHINE to the build host architecture


Looks like I need to submit a patch to y'all to add a ppc64le.conf to conf/machine-sdk/?

On Fri, 2021-01-29 at 06:38 -0800, Andrew Geissler wrote:

Over in OpenBMC, we utilize a mix of x86 and ppc64le machines for our
CI.

Our latest rebase of poky master
(https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/39533/) has
started failing to compile on our ppc64le machines with the below
error.

I can workaround it by setting "SDKMACHINE=x86_64" but that seems a
bit weird on a ppc64le machine?

+ bitbake obmc-phosphor-image obmc-phosphor-debug-tarball
ERROR: OE-core's config sanity checker detected a potential
misconfiguration.
    Either fix the cause of this error or at your own risk disable
the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    Specified SDKMACHINE value is not valid

What was SDKMACHINE set to before?

Cheers,

Richard

From: Matt Hoosier <matt.hoosier@garmin.com>

Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS rec=
ords.
Network access is not allowed at that point in the recipe's lifecycle.

[YOCTO #14191]

(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)

Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bitbake/lib/bb/fetch2/git.py | 44 ++++++++++++++++++++++++++++++++---
bitbake/lib/bb/tests/fetch.py | 28 +++++++++++++++-------
2 files changed, 61 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index d255afeb36..7c32eba6c3 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -378,6 +378,35 @@ class Git(FetchMethod):
if missing_rev:
raise bb.fetch2.FetchError("Unable to find revision %s e=
ven from upstream" % missing_rev)
=20
+ if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud)=
:
+ # Unpack temporary working copy, use it to run 'git checkout=
' to force pre-fetching
+ # of all LFS blobs needed at the the srcrev.
+ #
+ # It would be nice to just do this inline here by running 'g=
it-lfs fetch'
+ # on the bare clonedir, but that operation requires a workin=
g copy on some
+ # releases of Git LFS.
+ tmpdir =3D tempfile.mkdtemp(dir=3Dd.getVar('DL_DIR'))
+ try:
+ # Do the checkout. This implicitly involves a Git LFS fe=
tch.
+ self.unpack(ud, tmpdir, d)
+
+ # Scoop up a copy of any stuff that Git LFS downloaded. =
Merge them into
+ # the bare clonedir.
+ #
+ # As this procedure is invoked repeatedly on incremental=
fetches as
+ # a recipe's SRCREV is bumped throughout its lifetime, t=
his will
+ # result in a gradual accumulation of LFS blobs in <ud.c=
lonedir>/lfs
+ # corresponding to all the blobs reachable from the diff=
erent revs
+ # fetched across time.
+ #
+ # Only do this if the unpack resulted in a .git/lfs dire=
ctory being
+ # created; this only happens if at least one blob needed=
to be
+ # downloaded.
+ if os.path.exists(os.path.join(tmpdir, "git", ".git", "l=
fs")):
+ runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.c=
lonedir, d, workdir=3D"%s/git/.git" % tmpdir)
+ finally:
+ bb.utils.remove(tmpdir, recurse=3DTrue)
+
def build_mirror_data(self, ud, d):
if ud.shallow and ud.write_shallow_tarballs:
if not os.path.exists(ud.fullshallow):
@@ -473,7 +502,7 @@ class Git(FetchMethod):
if os.path.exists(destdir):
bb.utils.prunedir(destdir)
=20
- need_lfs =3D ud.parm.get("lfs", "1") =3D=3D "1"
+ need_lfs =3D self._need_lfs(ud)
=20
if not need_lfs:
ud.basecmd =3D "GIT_LFS_SKIP_SMUDGE=3D1 " + ud.basecmd
@@ -562,6 +591,9 @@ class Git(FetchMethod):
raise bb.fetch2.FetchError("The command '%s' gave output wit=
h more then 1 line unexpectedly, output: '%s'" % (cmd, output))
return output.split()[0] !=3D "0"
=20
+ def _need_lfs(self, ud):
+ return ud.parm.get("lfs", "1") =3D=3D "1"
+
def _contains_lfs(self, ud, d, wd):
"""
Check if the repository has 'lfs' (large file) content
@@ -572,8 +604,14 @@ class Git(FetchMethod):
else:
branchname =3D "master"
=20
- cmd =3D "%s grep lfs origin/%s:.gitattributes | wc -l" % (
- ud.basecmd, ud.branches[ud.names[0]])
+ # The bare clonedir doesn't use the remote names; it has the bra=
nch immediately.
+ if wd =3D=3D ud.clonedir:
+ refname =3D ud.branches[ud.names[0]]
+ else:
+ refname =3D "origin/%s" % ud.branches[ud.names[0]]
+
+ cmd =3D "%s grep lfs %s:.gitattributes | wc -l" % (
+ ud.basecmd, refname)
=20
try:
output =3D runfetchcmd(cmd, d, quiet=3DTrue, workdir=3Dwd)
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.p=
y
index 4702c99a7e..9453c90d2b 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -2046,13 +2046,14 @@ class GitLfsTest(FetcherTest):
cwd =3D self.gitdir
return bb.process.run(cmd, cwd=3Dcwd)[0]
=20
- def fetch(self, uri=3DNone):
+ def fetch(self, uri=3DNone, download=3DTrue):
uris =3D self.d.getVar('SRC_URI').split()
uri =3D uris[0]
d =3D self.d
=20
fetcher =3D bb.fetch2.Fetch(uris, d)
- fetcher.download()
+ if download:
+ fetcher.download()
ud =3D fetcher.ud[uri]
return fetcher, ud
=20
@@ -2062,16 +2063,21 @@ class GitLfsTest(FetcherTest):
uri =3D 'git://%s;protocol=3Dfile;subdir=3D${S};lfs=3D1' % self.=
srcdir
self.d.setVar('SRC_URI', uri)
=20
- fetcher, ud =3D self.fetch()
+ # Careful: suppress initial attempt at downloading until
+ # we know whether git-lfs is installed.
+ fetcher, ud =3D self.fetch(uri=3DNone, download=3DFalse)
self.assertIsNotNone(ud.method._find_git_lfs)
=20
- # If git-lfs can be found, the unpack should be successful
- ud.method._find_git_lfs =3D lambda d: True
- shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
- fetcher.unpack(self.d.getVar('WORKDIR'))
+ # If git-lfs can be found, the unpack should be successful. Only
+ # attempt this with the real live copy of git-lfs installed.
+ if ud.method._find_git_lfs(self.d):
+ fetcher.download()
+ shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
=20
# If git-lfs cannot be found, the unpack should throw an error
with self.assertRaises(bb.fetch2.FetchError):
+ fetcher.download()
ud.method._find_git_lfs =3D lambda d: False
shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
fetcher.unpack(self.d.getVar('WORKDIR'))
@@ -2082,10 +2088,16 @@ class GitLfsTest(FetcherTest):
uri =3D 'git://%s;protocol=3Dfile;subdir=3D${S};lfs=3D0' % self.=
srcdir
self.d.setVar('SRC_URI', uri)
=20
+ # In contrast to test_lfs_enabled(), allow the implicit download
+ # done by self.fetch() to occur here. The point of this test cas=
e
+ # is to verify that the fetcher can survive even if the source
+ # repository has Git LFS usage configured.
fetcher, ud =3D self.fetch()
self.assertIsNotNone(ud.method._find_git_lfs)
=20
- # If git-lfs can be found, the unpack should be successful
+ # If git-lfs can be found, the unpack should be successful. A
+ # live copy of git-lfs is not required for this case, so
+ # unconditionally forge its presence.
ud.method._find_git_lfs =3D lambda d: True
shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
fetcher.unpack(self.d.getVar('WORKDIR'))
--=20
2.20.1

From: Mauro Queirós <maurofrqueiros@gmail.com>

Function "contains_lfs" was only looking at the master branch when searching for LFS
content. LFS may be configured in specific branches only, so we need to use the
correct branch.

(Bitbake rev: 4fa67c2af830035a1ddedc14592ee25a15ebff22)

Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bitbake/lib/bb/fetch2/git.py | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 59afc32de1..d255afeb36 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -566,8 +566,15 @@ class Git(FetchMethod):
"""
Check if the repository has 'lfs' (large file) content
"""
- cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
- ud.basecmd)
+
+ if not ud.nobranch:
+ branchname = ud.branches[ud.names[0]]
+ else:
+ branchname = "master"
+
+ cmd = "%s grep lfs origin/%s:.gitattributes | wc -l" % (
+ ud.basecmd, ud.branches[ud.names[0]])
+
try:
output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
if int(output) > 0:
--
2.20.1

From: Mauro Queirós <maurofrqueiros@gmail.com>

Git-LFS objects were being fetched even when lfs=0 was not set.
This patch disables LFS smudging when lfs=0. That way, only the LFS pointers
are downloaded during checkout.

(Bitbake rev: 646d86df7de774255246a3d7051c308e43eb257d)

Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bitbake/lib/bb/fetch2/git.py | 3 +++
1 file changed, 3 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index dcecff5d38..59afc32de1 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -475,6 +475,9 @@ class Git(FetchMethod):

need_lfs = ud.parm.get("lfs", "1") == "1"

+ if not need_lfs:
+ ud.basecmd = "GIT_LFS_SKIP_SMUDGE=1 " + ud.basecmd
+
source_found = False
source_error = []

--
2.20.1

Sorry, third patch needs to be backported from master for git lfs to work
nicely on dunfell. Will send a v2.

-Mikko

Over in OpenBMC, we utilize a mix of x86 and ppc64le machines for our CI.

Our latest rebase of poky master (https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/39533/) has started failing to compile on our ppc64le machines with the below error.

I can workaround it by setting "SDKMACHINE=x86_64" but that seems a bit weird on a ppc64le machine?

+ bitbake obmc-phosphor-image obmc-phosphor-debug-tarball
ERROR: OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    Specified SDKMACHINE value is not valid

From: Matt Hoosier <matt.hoosier@garmin.com>

Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS rec=
ords.
Network access is not allowed at that point in the recipe's lifecycle.

[YOCTO #14191]

(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)

Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bitbake/lib/bb/fetch2/git.py | 44 ++++++++++++++++++++++++++++++++---
bitbake/lib/bb/tests/fetch.py | 28 +++++++++++++++-------
2 files changed, 61 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index fc14192e24..85c45d4dd4 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -378,6 +378,35 @@ class Git(FetchMethod):
if missing_rev:
raise bb.fetch2.FetchError("Unable to find revision %s e=
ven from upstream" % missing_rev)
=20
+ if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud)=
:
+ # Unpack temporary working copy, use it to run 'git checkout=
' to force pre-fetching
+ # of all LFS blobs needed at the the srcrev.
+ #
+ # It would be nice to just do this inline here by running 'g=
it-lfs fetch'
+ # on the bare clonedir, but that operation requires a workin=
g copy on some
+ # releases of Git LFS.
+ tmpdir =3D tempfile.mkdtemp(dir=3Dd.getVar('DL_DIR'))
+ try:
+ # Do the checkout. This implicitly involves a Git LFS fe=
tch.
+ self.unpack(ud, tmpdir, d)
+
+ # Scoop up a copy of any stuff that Git LFS downloaded. =
Merge them into
+ # the bare clonedir.
+ #
+ # As this procedure is invoked repeatedly on incremental=
fetches as
+ # a recipe's SRCREV is bumped throughout its lifetime, t=
his will
+ # result in a gradual accumulation of LFS blobs in <ud.c=
lonedir>/lfs
+ # corresponding to all the blobs reachable from the diff=
erent revs
+ # fetched across time.
+ #
+ # Only do this if the unpack resulted in a .git/lfs dire=
ctory being
+ # created; this only happens if at least one blob needed=
to be
+ # downloaded.
+ if os.path.exists(os.path.join(tmpdir, "git", ".git", "l=
fs")):
+ runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.c=
lonedir, d, workdir=3D"%s/git/.git" % tmpdir)
+ finally:
+ bb.utils.remove(tmpdir, recurse=3DTrue)
+
def build_mirror_data(self, ud, d):
if ud.shallow and ud.write_shallow_tarballs:
if not os.path.exists(ud.fullshallow):
@@ -473,7 +502,7 @@ class Git(FetchMethod):
if os.path.exists(destdir):
bb.utils.prunedir(destdir)
=20
- need_lfs =3D ud.parm.get("lfs", "1") =3D=3D "1"
+ need_lfs =3D self._need_lfs(ud)
=20
source_found =3D False
source_error =3D []
@@ -559,6 +588,9 @@ class Git(FetchMethod):
raise bb.fetch2.FetchError("The command '%s' gave output wit=
h more then 1 line unexpectedly, output: '%s'" % (cmd, output))
return output.split()[0] !=3D "0"
=20
+ def _need_lfs(self, ud):
+ return ud.parm.get("lfs", "1") =3D=3D "1"
+
def _contains_lfs(self, ud, d, wd):
"""
Check if the repository has 'lfs' (large file) content
@@ -569,8 +601,14 @@ class Git(FetchMethod):
else:
branchname =3D "master"
=20
- cmd =3D "%s grep lfs origin/%s:.gitattributes | wc -l" % (
- ud.basecmd, ud.branches[ud.names[0]])
+ # The bare clonedir doesn't use the remote names; it has the bra=
nch immediately.
+ if wd =3D=3D ud.clonedir:
+ refname =3D ud.branches[ud.names[0]]
+ else:
+ refname =3D "origin/%s" % ud.branches[ud.names[0]]
+
+ cmd =3D "%s grep lfs %s:.gitattributes | wc -l" % (
+ ud.basecmd, refname)
=20
try:
output =3D runfetchcmd(cmd, d, quiet=3DTrue, workdir=3Dwd)
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.p=
y
index 4702c99a7e..9453c90d2b 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -2046,13 +2046,14 @@ class GitLfsTest(FetcherTest):
cwd =3D self.gitdir
return bb.process.run(cmd, cwd=3Dcwd)[0]
=20
- def fetch(self, uri=3DNone):
+ def fetch(self, uri=3DNone, download=3DTrue):
uris =3D self.d.getVar('SRC_URI').split()
uri =3D uris[0]
d =3D self.d
=20
fetcher =3D bb.fetch2.Fetch(uris, d)
- fetcher.download()
+ if download:
+ fetcher.download()
ud =3D fetcher.ud[uri]
return fetcher, ud
=20
@@ -2062,16 +2063,21 @@ class GitLfsTest(FetcherTest):
uri =3D 'git://%s;protocol=3Dfile;subdir=3D${S};lfs=3D1' % self.=
srcdir
self.d.setVar('SRC_URI', uri)
=20
- fetcher, ud =3D self.fetch()
+ # Careful: suppress initial attempt at downloading until
+ # we know whether git-lfs is installed.
+ fetcher, ud =3D self.fetch(uri=3DNone, download=3DFalse)
self.assertIsNotNone(ud.method._find_git_lfs)
=20
- # If git-lfs can be found, the unpack should be successful
- ud.method._find_git_lfs =3D lambda d: True
- shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
- fetcher.unpack(self.d.getVar('WORKDIR'))
+ # If git-lfs can be found, the unpack should be successful. Only
+ # attempt this with the real live copy of git-lfs installed.
+ if ud.method._find_git_lfs(self.d):
+ fetcher.download()
+ shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
=20
# If git-lfs cannot be found, the unpack should throw an error
with self.assertRaises(bb.fetch2.FetchError):
+ fetcher.download()
ud.method._find_git_lfs =3D lambda d: False
shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
fetcher.unpack(self.d.getVar('WORKDIR'))
@@ -2082,10 +2088,16 @@ class GitLfsTest(FetcherTest):
uri =3D 'git://%s;protocol=3Dfile;subdir=3D${S};lfs=3D0' % self.=
srcdir
self.d.setVar('SRC_URI', uri)
=20
+ # In contrast to test_lfs_enabled(), allow the implicit download
+ # done by self.fetch() to occur here. The point of this test cas=
e
+ # is to verify that the fetcher can survive even if the source
+ # repository has Git LFS usage configured.
fetcher, ud =3D self.fetch()
self.assertIsNotNone(ud.method._find_git_lfs)
=20
- # If git-lfs can be found, the unpack should be successful
+ # If git-lfs can be found, the unpack should be successful. A
+ # live copy of git-lfs is not required for this case, so
+ # unconditionally forge its presence.
ud.method._find_git_lfs =3D lambda d: True
shutil.rmtree(self.gitdir, ignore_errors=3DTrue)
fetcher.unpack(self.d.getVar('WORKDIR'))
--=20
2.20.1

From: Mauro Queirós <maurofrqueiros@gmail.com>

Function "contains_lfs" was only looking at the master branch when searching for LFS
content. LFS may be configured in specific branches only, so we need to use the
correct branch.

(Bitbake rev: 4fa67c2af830035a1ddedc14592ee25a15ebff22)

Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bitbake/lib/bb/fetch2/git.py | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index dcecff5d38..fc14192e24 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -563,8 +563,15 @@ class Git(FetchMethod):
"""
Check if the repository has 'lfs' (large file) content
"""
- cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
- ud.basecmd)
+
+ if not ud.nobranch:
+ branchname = ud.branches[ud.names[0]]
+ else:
+ branchname = "master"
+
+ cmd = "%s grep lfs origin/%s:.gitattributes | wc -l" % (
+ ud.basecmd, ud.branches[ud.names[0]])
+
try:
output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
if int(output) > 0:
--
2.20.1

On Thu, 2021-01-28 at 16:18 +0100, Marek Belisko wrote:

Hi,

I have repo where I'm using lfs. I've added to my SRC_URI =
"git://...... ;lfs=1" and the project is fetched but the issue is that
lfs files are shown as references only (content is not fetched).
I briefly checked git fetcher in bitbake and seems it checks that repo
have lfs content but I cannot see anywhere git lfs pull to actually
pull lfs content. Am I missing something? When checked poky master it
looks like proper fetching is in place. Is there any way how to get it
to dunfell LTS release? Thanks.

There is a patch in bitbake master related to this which you may want
to test and maybe request for adding to dunfell if it helps.

https://git.openembedded.org/bitbake/commit/?id=0efac66043662e7a2027192f50e92e982db2ba1c

Cheers,

Richard

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
README | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/README b/README
index c17f74a..af3ace9 100644
--- a/README
+++ b/README
@@ -15,3 +15,9 @@ Layer Maintainer: Joshua Watt <jpewhacker@gmail.com>

Please send changes to the yocto mailing list with [meta-mingw] in the subject line,
cc'ing the maintainer.
+
+This can be configured within the repository with the commands:
+
+git config sendemail.to yocto@lists.yoctoproject.org
+git config sendemail.cc jpewhacker@gmail.com
+git config format.subjectprefix "meta-mingw] [PATCH"
--
2.27.0