A little context before the actual questions. As a project that I have worked on grew, the number of repositories used for the application we're developing grew as well. To the point that it's gotten unbearable to keep making modifications to the codebase: when something is changed in the core library in one repository we need to make adjustments to other Python middleware projects that use this library. Whereas this sounds not that terrifying, managing this in Yocto has become a burden: on every version bump of the Python library we need to bump all of the dependant projects as well. After some thinking, we decided to try a monorepo approach to deal with the complexity. I'm omitting the logic behind choosing this approach but I can delve into it if you think this is wrong.
1) Phase 1 was easy. Just bring every component of our middleware to one repository. Right now we have a big repository with more than git 20 submodules. Submodules will be gone once we finish the transition but as the project doesn't stop while we're doing the transition, submodules were chosen to track the changes and keep the monorepo up to date. Every submodule is a Python code with a setup.py/Pipfile et al. that does the bootstrapping.
2) Phase 2 is to integrate everything in Yocto. It has turned out to be more difficult than I had anticipated.
Right now we have this:
Naturally, right now we have
Right now I have the layout described below.
This approach works. But it has a very significant drawback I can't ignore. Every time SRCREV (or PV/tag in the future) is changed every recipe downloads the whole repository that is quite big.
So, well, the question is how to structure the Yocto recipes for Python monorepo manipulation? I'm looking for a way to trick do_unpack and do_fetch to their thing on the monorepo basis.
P.S. 1) The monorepo structure is not set in stone so if you have any suggestions, I'm more than open to any criticism 2) The
Any suggestion on how to deal with the complexity of a big Python application in Yocto will be welcome. Thanks in advance.