Hi,

I have put together a recipe inheriting from go-mod. This project depends on out-of-repo modules that sit in private repos. As long as the SSH key required to pull the requirements is present as a file (under $HOME/.ssh), everything works fine. However, as soon as the SSH credentials are only available via agent and $SSH_AUTH_SOCK, the do_compile step fails. I have traced this down to the fact that the $SSH_AUTH_SOCK environment variable is not available to do_compile which is when the requirements are pulled. This is the sort of error message I get:

ERROR: mypackage-git-r0 do_compile: Execution of '[...]/mypackage/git-r0/temp/run.do_compile.20076' failed with exit code 1:
# cd .; git ls-remote https://bitbucket.org/myorg/some-requirement
Permission denied (publickey).
fatal: Could not read from remote repository.

Note, that the do_fetch step succeeds in pulling the actual repo. I tried fixing the problem by wrapping the do_compile function and providing $SSH_AUTH_SOCK from the original environment:

def origenv(d, var):
return d.getVar("BB_ORIGENV", False).getVar(var, False)

do_compile() {
if [ -n "${@origenv(d, 'SSH_AUTH_SOCK') or ''}" ]; then
export SSH_AUTH_SOCK="${@origenv(d, 'SSH_AUTH_SOCK')}"
fi
go_do_compile
}

This allows the do_compile step (and all subsequent steps) to finish successfully. However, that way, I get a bunch of errors like this (cleansstate does not help):

ERROR: When reparsing [...]/mypackage_git.bb:do_compile, the basehash value changed from eb51e4ec321c723587cec03bb9b33b94ee43e0b0939eb43b52824e3d5cfebec2 to 2bb034f43856917d6454a56b32946b1c68cf7f286b20fd7a7eaf1bfd2a92d34f. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake mypackage -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake mypackage -cdo_compile -Sprintdiff

Neither command helps to fix this. What can I do? I'm on poky yocto-3.1.5-18-gbb7747497a.

Best regards,
Sven

On Mon, May 03, 2021 at 11:25:09AM -0700, Sven via lists.yoctoproject.org wrote:

Hi,

I have put together a recipe inheriting from go-mod. This project depends on out-of-repo modules that sit in private repos. As long as the SSH key required to pull the requirements is present as a file (under $HOME/.ssh), everything works fine. However, as soon as the SSH credentials are only available via agent and $SSH_AUTH_SOCK, the do_compile step fails. I have traced this down to the fact that the $SSH_AUTH_SOCK environment variable is not available to do_compile which is when the requirements are pulled. This is the sort of error message I get:

On Ubuntu, when the password for my SSH key is in the gnome keyring, I
run:
eval $(/usr/bin/gnome-keyring-daemon --start --components=pkcs11,secrets,ssh); export SSH_AUTH_SOCK
in the terminal that will run bitbake.

No idea if it's the proper way to make it work, but for the few times I
need to fetch new revisions of private repos, it's been good enough to
me.

Hope this helps,
Cheers,
Quentin

Hi Quentin,

Thanks for your reply. Did you have success with this technique when compiling a go-mod recipe where some of the go dependencies sit in private repos? The standard git fetcher works for me with private repos, that is not the problem.

Best regards,
Sven

Hi Sven,

On Wed, May 05, 2021 at 06:37:41AM -0700, Sven via lists.yoctoproject.org wrote:

Hi Quentin,

Thanks for your reply. Did you have success with this technique when compiling a go-mod recipe where some of the go dependencies sit in private repos? The standard git fetcher works for me with private repos, that is not the problem.

No, I do not use any custom go-based recipe.

BR,
Quentin

On Mon, 2021-05-03 at 11:25 -0700, Sven via lists.yoctoproject.org wrote:
Hi,

I have put together a recipe inheriting from go-mod. This project depends on 

out-of-repo modules that sit in private repos. As long as the SSH key required 
to pull the requirements is present as a file (under $HOME/.ssh), everything 
works fine. However, as soon as the SSH credentials are only available via 
agent and $SSH_AUTH_SOCK, the do_compile step fails. I have traced this down 
to the fact that the $SSH_AUTH_SOCK environment variable is not available to
do_compile which is when the requirements are pulled. This is the sort of error 
message I get:

ERROR: mypackage-git-r0 do_compile: Execution of '[...]/mypackage/git-r0/temp/run.do_compile.20076' 

failed with exit code 1:

# cd .; git ls-remote https://bitbucket.org/myorg/some-requirement
Permission denied (publickey).
fatal: Could not read from remote repository.

Note, that the do_fetch step succeeds in pulling the actual repo. I tried fixing 

the problem by wrapping the do_compile function and providing $SSH_AUTH_SOCK from
the original environment:

def origenv(d, var):
    return d.getVar("BB_ORIGENV", False).getVar(var, False)

do_compile() {
    if [ -n "${@origenv(d, 'SSH_AUTH_SOCK') or ''}" ]; then
        export SSH_AUTH_SOCK="${@origenv(d, 'SSH_AUTH_SOCK')}"
    fi
    go_do_compile
}

This allows the do_compile step (and all subsequent steps) to finish successfully. 

However, that way, I get a bunch of errors like this (cleansstate does not help):

ERROR: When reparsing [...]/mypackage_git.bb:do_compile, the basehash value 

changed from eb51e4ec321c723587cec03bb9b33b94ee43e0b0939eb43b52824e3d5cfebec2 
to 2bb034f43856917d6454a56b32946b1c68cf7f286b20fd7a7eaf1bfd2a92d34f. The metadata 
is not deterministic and this needs to be fixed.

ERROR: The following commands may help:
ERROR: $ bitbake mypackage -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake mypackage -cdo_compile -Sprintdiff

Neither command helps to fix this. What can I do? I'm on poky yocto-3.1.5-18-gbb7747497a.

You can probably 'fix' that with:

do_compile[vardepsexclude] += "SSH_AUTH_SOCK"

however you really shouldn't be accessing the network in a compile task.
That is a wider go issue :(.

Cheers,

Richard

Hi Richard,

Unfortunately, that doesn't make the error messages go away. I agree that it's not great that go-mod fetches during do_compile but that's the way it currently is.

For completeness sake, here is the complete minimal example that _does_ compile. However, error messages à la

ERROR: When reparsing [...]/golang-test/golang-test.bb:do_compile, the basehash value changed from be5e3ca0e52fd3e3a0315fb32a2845f097bb6925940a8141b6f4d99fa2423e20 to fc952756231a898ae7d3fd611607066b349042f5313b8363d11eef18ed317ed2. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake golang-test -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake golang-test -cdo_compile -Sprintdiff

are spewed out. Here's the recipe:

LICENSE = "CLOSED"
GO_IMPORT = "bitbucket.org/sven_schwermer/golang-a"
SRC_URI = "git://git@${GO_IMPORT};protocol=ssh;branch=master"
SRCREV = "f7d820d09a5b2332737a8105b3efd9c1e8e51d32"
inherit go-mod
do_compile() {
export SSH_AUTH_SOCK="${@d.getVar('BB_ORIGENV', False).getVar('SSH_AUTH_SOCK', False)}"
go_do_compile
}

bitbucket.org/sven_schwermer/golang-a pulls in a secondary dependency (bitbucket.org/sven_schwermer/golang-b) which is not publicly available. This is golang-a's go.mod:

module bitbucket.org/sven_schwermer/golang-a
go 1.16
require bitbucket.org/sven_schwermer/golang-b v0.0.0-20210505180831-90ba3cd6391e

I have tested this also on latest poky@master (5a0679cb75) with identical results.

Adding BB_ORIGENV to do_compile[vardepsexclude] solved the issue. Thanks for your help!