Re: Dealing with go dependencies in recipes - native docker-compose
Bruce Ashfield
On Tue, Oct 11, 2022 at 2:51 PM Konstantin Kletschke <konstantin.kletschke@...> wrote:
Today I realized, that the docker-compose in the meta-virtualization
layer is the old python3 based one and a native solution is available. I
tried to make a recipe for it to get rid of python3 (no other packages
uses it in our setup) and the current python3-docker-compose is broken:
https://lore.kernel.org/yocto-meta-virtualization/49660EA1-CD22-40DB-98C7-C43F38A72CCA@.../
Adding the missing setuptools does get things working.
I tried to build the native solution proposed in
https://github.com/docker/compose/tree/v2.11.2.
I actually have a prototype recipe for this that I was working on before ELCe, but I didn't get it into meta-virtualization yet, as it had a few rough edges.
If you give me a few days, I can post it to the meta-virtualization list, but I'm on the road right now and don't have access to all my build machines.
So my current recipe docker-compose_2.11.2.bb looks like this:
LICENSE = "Apache-2.0"
inherit go
inherit go-mod
GO_IMPORT = "github.com/docker/compose"
SRC_URI = "git://${GO_IMPORT};protocol=https;branch=v2"
# v2.11.2
SRCREV = "616777eb4ad4d1101622d6727d9b7adaeb7943bb"
# S = "${WORKDIR}/git"
DEPENDS = "docker-ce"
RDEPENDS:${PN} = "docker-ce-cli"
With this I run into the issue, that go want's to download stuff while
compiling. Which is forbidden and breaks the reliable build proposal.
Totally understandable. Looks similair to this:
dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
I read about a proposal allowing this by adding do_compile[network] =
"1" to meta/classes/go.bb:
https://lore.kernel.org/all/20220228235433.3948994-1-andrei@.../
Of course, if this works, the reliable build thingy is gone, but while
Right, it works, but would never make it into core due to it bypassing many
of the fetcher properties.
trying I discovered go loads external stuff, compiles it too.
In my case (I have kirkstone with go-1.17) I run into modules demanding
a go more recent:
ASH[build k8s.io/client-go/applyconfigurations/autoscaling/v2beta2]: "file containerresourcemetricsource.go Mas4-HIX5lGBEQNTIo58\n"
# github.com/docker/compose/v2/pkg/utils
pkg/utils/slices.go:23:6: missing function body
pkg/utils/slices.go:23:14: syntax error: unexpected [, expecting (
note: module requires Go 1.19
HASH[build k8s.io/client-go/applyconfigurations/certificates/v1beta1]
HASH[build k8s.io/client-go/applyconfigurations/certificates/v1beta1]: "go1.17.13"
which could be worked around by moving to landsdale release. Which
bothers me, because I too want to keep my build reliable. And I do not
want to mess around outside our meta layer in the distribution!
How do I do this properly?
I see people pulling in each dependency by individual golang.org-x.bb
recipes, how could such a recipe look like?
There are many different discussions about how to generate go recipes, as well
as similar discussions for ruby/rust, etc, you can find them on the openembedded-core
and openembedded-architectures lists.
I also did a presentation at the yocto summit about "modern languages".
So I won't try and summarize all those discussions here, since they are archived
and do a much better job than I could here.
But to answer the specific question .. you can't really handle the individual
go dependencies as recipes. They are far too sensitive to versions and
individual git hashes. Plus you'll end up with thousands of recipes of varying
versions.
There have been several proposals about how to generate go recipes to deal with
the dependencies. Some prototypes have been posted, but there hasn't
been a lot of traffic on that topic for several months now.
You can see the approach that I take for this in the k3s and nerdctl recipes
in meta-virtualization. My new docker-compose recipe is of similar format.
I read
https://lore.kernel.org/all/8132db85-5881-636e-c091-d84c47efe4fe@.../T/
where Mike is not happy with this appraoch either and comes up with a
working recipe I don't get why this could work at all.
What am I missing here where is the missing link I did not get yet?
Also, I am jealous about the buildroot guys sometimes, how do they do
this in a 22 lines makefile including comments:
https://git.busybox.net/buildroot/tree/package/docker-compose/docker-compose.mk
If we just bypass the fetcher, offline builds, some of licensing and SBOM
and reproducible builds .. you can have a simple recipe like that as well :)
Bruce
Regards
Konstantin
--
INSIDE M2M GmbH
Konstantin Kletschke
Berenbosteler Straße 76 B
30823 Garbsen
Telefon: +49 (0) 5137 90950136
Mobil: +49 (0) 151 15256238
Fax: +49 (0) 5137 9095010
konstantin.kletschke@...
http://www.inside-m2m.de
Geschäftsführung: Michael Emmert, Ingo Haase, Dr. Fred Könemann, Derek Uhlig
HRB: 111204, AG Hannover
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
- "Use the force Harry" - Gandalf, Star Trek II