A little context before the actual questions. As a project that I have worked on grew, the number of repositories used for the application we're developing grew as well. To the point that it's gotten unbearable to keep making modifications to the codebase: when something is changed in the core library in one repository we need to make adjustments to other Python middleware projects that use this library. Whereas this sounds not that terrifying, managing this in Yocto has become a burden: on every version bump of the Python library we need to bump all of the dependant projects as well. After some thinking, we decided to try a monorepo approach to deal with the complexity. I'm omitting the logic behind choosing this approach but I can delve into it if you think this is wrong.

1) Phase 1 was easy. Just bring every component of our middleware to one repository. Right now we have a big repository with more than git 20 submodules. Submodules will be gone once we finish the transition but as the project doesn't stop while we're doing the transition, submodules were chosen to track the changes and keep the monorepo up to date. Every submodule is a Python code with a setup.py/Pipfile et al. that does the bootstrapping.

2) Phase 2 is to integrate everything in Yocto. It has turned out to be more difficult than I had anticipated.

Right now we have this:

Application monorepo

Pipfile
python-library1/setup.py
python-library1/Makefile
python-library1/python-library1/<code>


python-library2/setup.py
python-library2/Makefile
python-library2/python-library2/<code>


...
python-libraryN/setup.py
python-libraryN/Makefile
python-libraryN/python-libraryN/<code>

Naturally, right now we have python-library1_<some_version>.bb, python-library2_<some_other_version>.bb in Yocto. Now we want to get rid of this version hell and stick to monorepo versions so that we could have just monorepo_1.0.bb that is to be updated when anything changes in any part.

Right now I have the layout described below.

monorepo.inc is as follows:

SRC_URI=<gitsm://the location of the monorepo,branch=...,>
S = "${WORKDIR}/git"
SRCREV = <hash>

python-libraryN.inc:

require monorepo.inc
S = "${WORKDIR}/git/python-libraryN"
inherit setuptools

#some libraries also use entry points and the recipe needs to inherit systemd to enable some services.

This approach works. But it has a very significant drawback I can't ignore. Every time SRCREV (or PV/tag in the future) is changed every recipe downloads the whole repository that is quite big.

So, well, the question is how to structure the Yocto recipes for Python monorepo manipulation? I'm looking for a way to trick do_unpack and do_fetch to their thing on the monorepo basis.

This section of the documentation makes me think that I'm approaching this task wrong. I was trying to decipher gcc-source.inc and gcc-shared-source.inc files in the poky but they are out of my depth for now and look a little bit overcomplicated for my use-case. So basically my question boils down to how to deal with multiple recipes referencing the same source. Any structural change to either the monorepo or the recipe structure can be done now, so I would be very grateful for any feedback.

P.S. 1) The monorepo structure is not set in stone so if you have any suggestions, I'm more than open to any criticism 2) The .inc file structure might not be suitable for another reason. This stack is deployed across a dozen of different devices and some of those python-library_N.bb have .bbappends in other layers that are custom for the devices. I mean that some of them might require different components of the system installed or some configs and files/ modifications.

Any suggestion on how to deal with the complexity of a big Python application in Yocto will be welcome. Thanks in advance.