On Wed, 2022-03-30 at 12:18 +0100, Ross Burton wrote:

On Wed, 30 Mar 2022 at 12:10, Richard Purdie
<richard.purdie@...> wrote:

f) Switch the problematic recipes to use shallow clones with something like:

BB_GIT_SHALLOW:pn-binutils = "1"
BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
BB_GIT_SHALLOW:pn-glibc = "1"

The challenge here is that in order to be effective, there needs to be a
PREMIRROR setup with the shallow tarballs on it. This means we couldn't do e)
above and have this have much effect unless we craft some very specific
PREMIRROR entries too.

Even without premirrors this is a lot faster for glibc:

$ time git clone git://sourceware.org/git/glibc.git
Cloning into 'glibc'...
remote: Enumerating objects: 6956, done.
remote: Counting objects: 100% (6956/6956), done.
remote: Compressing objects: 100% (2938/2938), done.
remote: Total 670093 (delta 5328), reused 4750 (delta 3932), pack-reused 663137
Receiving objects: 100% (670093/670093), 205.19 MiB | 16.39 MiB/s, done.
Resolving deltas: 100% (573265/573265), done.
Updating files: 100% (19011/19011), done.

real 1m56.255s

$ time git clone git://sourceware.org/git/glibc.git --depth 1
Cloning into 'glibc'...
remote: Enumerating objects: 18809, done.
remote: Counting objects: 100% (18809/18809), done.
remote: Compressing objects: 100% (9704/9704), done.
remote: Total 18809 (delta 8812), reused 12185 (delta 7968), pack-reused 0
Receiving objects: 100% (18809/18809), 41.79 MiB | 11.96 MiB/s, done.
Resolving deltas: 100% (8812/8812), done.
Updating files: 100% (19011/19011), done.

real 0m8.701s

A full clone fetches 200MB and takes 2 minutes (a lot of that is
actually resolving the deltas, not the fetch). A shallow clone of the
current HEAD fetches 40MB and is done in 8 seconds.

Why would we need a premirror?

The code doesn't do "--depth=1".

https://git.yoctoproject.org/poky/commit/?id=27d56982c7ba05e86a100b0cca2411ee5ac7a85e

"""
This implements support for shallow mirror tarballs, not shallow clones.
Supporting shallow clones directly is not really doable for us, as we'd need
to hardcode the depth between branch HEAD and the SRCREV, and that depth would
change as the branch is updated.
"""

Put another way, you didn't specify a revision in your clone above and if you
try, it becomes rather tricky.

To make this work we therefore need a mirror with the shallow tarballs on it.

Just for info, the binutils mirror tarball is ~1.3GB, the shallow tarball is
65MB.

Cheers,

Richard