#golang: go fetches dependencies in compile phase


Juergen Landwehr
 

Hi,
 
we are developing an application in go and use the "go-mod" bb-class from poky. This works fine.
 
However, the problem is, that the dependencies in go.mod are now fetched in the compile phase and not in the fetch phase.
 
This is not allowed in our project setup and kind of contradicts the Yocto paradigmn of having a fetch and a compile phase.
 
Is there a way to avoid this or some other solution that we could use?
 
Thanks,
 
Jürgen


Konrad Weihmann <kweihmann@...>
 

Well AFAIK your observation is correct, go-mod.bbclass doesn't work a in BB_NO_NETWORK-safe manner.
What you could do is to provide all the dependencies manually and add them via DEPENDS, which is grant the behavior expected.

The bad thing about go is that chances of circular dependencies of go modules are relatively high.

I've also seen a few approaches using vendoring to escape this, but I'm afraid none of the code is pretty well tested nor publicly available.

If you'd ask me, I would go the long painful road of providing each required go module as a recipe of it's own and inject it into the final recipe - this also makes it pretty visible what you're pulling into your build in terms of licensing a.s.o.

On 09.04.21 15:53, Juergen Landwehr wrote:
Hi,
we are developing an application in go and use the "go-mod" bb-class from poky. This works fine.
However, the problem is, that the dependencies in go.mod are now fetched in the compile phase and not in the fetch phase.
This is not allowed in our project setup and kind of contradicts the Yocto paradigmn of having a fetch and a compile phase.
Is there a way to avoid this or some other solution that we could use?
Thanks,
Jürgen


Juergen Landwehr
 

Hello Konrad,

thanks for your reply.

It is interesting that you mention vendoring, because we tried this approach as well. We started to develop a "go-mod-vendor.bbclass", which retrieves the
application source code in "do_fetch_append" and then calls "go mod vendor".

The big problem was, that in that phase there is not yet a go executable available for the recipe, so we downloaded our own go version, which makes this approach a bit ugly.

But there is already a go version installed on the build host, right? In "poky/meta/recipes-devtools/go" I found a "go-native_1.14.bb" recipe.
So can we use this go exectuable somehow by adding a dependency (e.g. DEPENDS="go-native") and then use a path to the go exectuable that is guaranteed to work?

The other alternative (defining a recipe for each dependent go module) sounds really painful. Especially as we have quite a few dependencies, which have dependencies as well.
I see the point, that doing it that way makes it more visible and more Yocto like, but thinking about it gives me headaches already :)

Thanks again.


Robert Berger
 

On 09/04/2021 16:53, Juergen Landwehr wrote:
Hi,
we are developing an application in go and use the "go-mod" bb-class from poky. This works fine.
However, the problem is, that the dependencies in go.mod are now fetched in the compile phase and not in the fetch phase.
Alexander (whom you cc'ed here as well) wrote a long time ago on the mailing list[1] about this stuff.

We have a couple of issues with those "modern" languages like go, which behave quite differently from good old Unix stuff. So the paradigm on which the whole Bitbake/OpenEmbedded/Yocto system was built does not quite apply for those.

Don't get me wrong, you can also do stupidities like git clone via make or cmake, but this is most likely not the way those tools should be used.

1) "Guaranteed __NOT__ reproducible builds"(C)

Go pulls in dependencies by itself, which might explain why it's in the compile phase and not it the fetch phase.

This leads to many interesting issues, like "guaranteed __NOT__ reproducible builds"(C). The problem is, that you don't have any influence on dependencies of dependencies. Someone changes somewhere a dependency on github and the party starts.

1.1) One sane solution to "please give me always the same input" is what Konrad already mentioned in combination with a local golang proxy.

2) Open Source License Compliance

If you pull in all those funny dependencies and also link things statically, like with go, Open Source License Compliance is getting very interesting.

People typically just use the license of the top level "main" entry point. In reality you would need to check all the dependency tree for licenses. You should also check if it's legally allowed to combine the licenses and in case it's allowed what are implications of GPLvx, LGPLv2, LGPLv3 without exceptions for your product.

[1] https://www.yoctoproject.org/pipermail/yocto/2017-March/035028.html

Regards,

Robert


Juergen Landwehr
 

Hi Robert,

thanks for your thoughts.

I see your point and the last thing I want is "NOT reproducable builds".

But dependency management in go is not that arbitrary as it may seem. Dependencies and their version is stored in "go.mod". To ensure reproducable builds, hashes for each dependency and version are stored in "go.sum". Both files are in git and together with a local golang proxy, this should ensure reproducable builds, right?

To ensure that licences are valid and did not change over time, we developed a little tool, that goes through all dependencies and creates the following output, which we insert into our recipe:

LICENSE = "Apache-2.0 & MIT & MIT & BSD-2-Clause & BSD-3-Clause & ...

LIC_FILES_CHKSUM = " \
file://${S}/src/${GO_IMPORT}/vendor/github.com/coreos/go-oidc/LICENSE;md5=d2794c0df5b907fdace235a619d80314 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/locales/LICENSE;md5=3ccbda375ee345400ad1da85ba522301 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/universal-translator/LICENSE;md5=2e2b21ef8f61057977d27c727c84bef1 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/godbus/dbus/v5/LICENSE;md5=09042bd5c6c96a2b9e45ddf1bc517eed \
file://${S}/src/${GO_IMPORT}/vendor/github.com/golang/gddo/LICENSE;md5=4c728948788b1a02f33ae4e906546eef \
...
This is a manual step and whenever dependencies change we have to create this list again and add it to the recipe, but it contains all the required license information and makes them visible in the recipe.

It might pollute the recipe a bit, but luckily we do not have thousands of dependencies like some npm projects. So I think it is still manageable. And is it much less work, than defining a recipe for each dependency.

So we would
* guarantee reproducable builds
* use the programming language (in our case golang) to handle dependency management
* ensure that licenses are visible and correct

I mean it is not perfect and still a compromise, but it should work without breaking reproducable builds or causing license issues?
What do you think?

Regards,
Jürgen

PS: Thanks for the link. I was not aware of this discussion (it is quite a bit to read:))


Alexander Kanavin
 

On Mon, 12 Apr 2021 at 13:47, Juergen Landwehr <juergen.landwehr@...> wrote:
But dependency management in go is not that arbitrary as it may seem. Dependencies and their version is stored in "go.mod". To ensure reproducable builds, hashes for each dependency and version are stored in "go.sum". Both files are in git and together with a local golang proxy, this should ensure reproducable builds, right?

Reproducibility means anyone can run a build at any point in the future even if the upstream repositories are gone, so all inputs must be stored in a local download cache, which is the other thing SRC_URI guarantees, in addition to verifying integrity of the inputs.

Alex


Juergen Landwehr
 

Hi Alex,

OK, understood.

If the "local download cache path" is well-known (this is not by any chance $DL_DIR?), then we could create a tar from the vendor directory (which is created when you call "go mod vendor" and contains all the downloaded dependencies) and put this tar file into the download cache.

Before actually calling "go mod vendor", we would first check, if there is a tar file for the vendor directory in the download cache and if so simply unpack the tar.

Does this make sense, or am I too naive and this is just nonsense?

Jürgen


Alexander Kanavin
 

I'd suggest you place that tarball into some artefact storage, and have it listed in SRC_URI. Then the standard Yocto mechanism for fetching, checksumming, caching and unpacking tarballs kicks in, so you only need to make sure the tree is set up correctly after unpacking, maybe with some simple post-processing.

Alex


On Mon, 12 Apr 2021 at 16:02, Juergen Landwehr <juergen.landwehr@...> wrote:
Hi Alex,

OK, understood.

If the "local download cache path" is well-known (this is not by any chance $DL_DIR?), then we could create a tar from the vendor directory (which is created when you call "go mod vendor" and contains all the downloaded dependencies) and put this tar file into the download cache.

Before actually calling "go mod vendor", we would first check, if there is a tar file for the vendor directory in the download cache and if so simply unpack the tar.

Does this make sense, or am I too naive and this is just nonsense?

Jürgen



Robert Berger
 

Hi,

My comments are inline.

On 12/04/2021 14:47, Juergen Landwehr wrote:
Hi Robert,
thanks for your thoughts.
I see your point and the last thing I want is "NOT reproducable builds".
But dependency management in go is not that arbitrary as it may seem.
... if everybody does what they are supposed to - which is not the case.

Dependencies and their version is stored in "go.mod". To ensure reproducable builds, hashes for each dependency and version are stored in "go.sum". Both files are in git and together with a local golang proxy, this should ensure reproducable builds, right?
This does not sound too bad. This would also mean, that if the outside repo dies you can still build and that you know what's on your proxy.

To ensure that licences are valid and did not change over time, we developed a little tool, that goes through all dependencies and creates the following output, which we insert into our recipe:
LICENSE = "Apache-2.0 & MIT & MIT & BSD-2-Clause & BSD-3-Clause & ...
Here you

1) Totally lost which License comes from which module and I hope 2 times MIT is just a typo.

Of course if you are really serious about licensing you should use a tool like fossology, upload all your sources there and inspect them.
You will be surprised.

There are a couple of tools out there which scan your sources and some which claim to do stuff with golang as well.

2) Do you check if the licenses are compatible?

MIT and Apache are compatible:

some googling:

"Both the Apache License and the MIT license are permissive, so incorporating MIT licensed code into your Apache licensed project is certainly allowed. Just be sure to attribute the original author for the parts your incorporated and include a copy of the MIT License terms, as required by the license."

Apache and BSD should also be OK:

some googling:

"In both of them you must:

Include copyright

But in Apache, unlike BSD you must:

Include License
State Changes
Include Notice
"

LIC_FILES_CHKSUM = " \
file://${S}/src/${GO_IMPORT}/vendor/github.com/coreos/go-oidc/LICENSE;md5=d2794c0df5b907fdace235a619d80314 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/locales/LICENSE;md5=3ccbda375ee345400ad1da85ba522301 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/go-playground/universal-translator/LICENSE;md5=2e2b21ef8f61057977d27c727c84bef1 \
file://${S}/src/${GO_IMPORT}/vendor/github.com/godbus/dbus/v5/LICENSE;md5=09042bd5c6c96a2b9e45ddf1bc517eed \
file://${S}/src/${GO_IMPORT}/vendor/github.com/golang/gddo/LICENSE;md5=4c728948788b1a02f33ae4e906546eef \
...
This is a manual step and whenever dependencies change we have to create this list again and add it to the recipe, but it contains all the required license information and makes them visible in the recipe.
really all?

You search through every single file of a go module and it's dependencies? Or just the license text, if there is one?

It might pollute the recipe a bit, but luckily we do not have thousands of dependencies like some npm projects. So I think it is still manageable. And is it much less work, than defining a recipe for each dependency.
So we would
* guarantee reproducable builds
* use the programming language (in our case golang) to handle dependency management
* ensure that licenses are visible and correct
I mean it is not perfect and still a compromise, but it should work without breaking reproducable builds or causing license issues?
What do you think?
Regards,
Jürgen
PS: Thanks for the link. I was not aware of this discussion (it is quite a bit to read:))
Regards,

Robert