Date
1 - 5 of 5
Pull a single file using git lfs
Andy Pont <andy.pont@...>
Hello,
I have a private git repo on our GitLab server that contains a bunch of source files that came as part of a commercial product and the final binary executable is also stored there using “git lfs”.
I’m trying to figure out how to create a bitbake recipe so that in the do_fetch() stage it just pulls the single binary file. I think the file can be got using “git lfs checkout path/to/file” but I’m not sure how to make it connect to the repo and not do a regular “git pull”.
-Andy.
I have a private git repo on our GitLab server that contains a bunch of source files that came as part of a commercial product and the final binary executable is also stored there using “git lfs”.
I’m trying to figure out how to create a bitbake recipe so that in the do_fetch() stage it just pulls the single binary file. I think the file can be got using “git lfs checkout path/to/file” but I’m not sure how to make it connect to the repo and not do a regular “git pull”.
-Andy.
Quentin Schulz
Hi Andy,
On Wed, Dec 18, 2019 at 10:46:17AM +0000, Andy Pont wrote:
I'm thinking about the subpath argument to SRC_URI git fetcher.
https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher
""subpath": Limits the checkout to a specific subpath of the tree. By
default, the whole tree is checked out."
So maybe by giving the path to the file you'll be able to get it without
the whole repo. Provided you're able to get those files with git pull
and not git lfs.
The other thing could be to use a do_fetch_append() with the call to git
lfs in there. I think the git used in that task will be the host one, so
you should then have support for git lfs provided you installed it on
the build machine.
Hopefully I'm not answering with complete non-sense, and giving you some
hints on what to test a bit more :)
Good luck,
Quentin
On Wed, Dec 18, 2019 at 10:46:17AM +0000, Andy Pont wrote:
Hello,I unfortunately don't have experience with git lfs.
I have a private git repo on our GitLab server that contains a bunch of
source files that came as part of a commercial product and the final binary
executable is also stored there using “git lfs”.
I’m trying to figure out how to create a bitbake recipe so that in the
do_fetch() stage it just pulls the single binary file. I think the file can
be got using “git lfs checkout path/to/file” but I’m not sure how to make it
connect to the repo and not do a regular “git pull”.
I'm thinking about the subpath argument to SRC_URI git fetcher.
https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher
""subpath": Limits the checkout to a specific subpath of the tree. By
default, the whole tree is checked out."
So maybe by giving the path to the file you'll be able to get it without
the whole repo. Provided you're able to get those files with git pull
and not git lfs.
The other thing could be to use a do_fetch_append() with the call to git
lfs in there. I think the git used in that task will be the host one, so
you should then have support for git lfs provided you installed it on
the build machine.
Hopefully I'm not answering with complete non-sense, and giving you some
hints on what to test a bit more :)
Good luck,
Quentin
Andy Pont <andy.pont@...>
Quentin wrote...
Unfortunately, there are other files in the same directory that can’t be checked out. Looking at that documentation, I have added the “nocheckout=1” option to the git definition in SRC_URI which is stopping the initial pull.I unfortunately don't have experience with git lfs.I'm thinking about the subpath argument to SRC_URI git fetcher.https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher
When I try to add any git commands into a do_fetch_append() then it dumps a load of Python looking stuff on the screen that ends with:
File "autogenerated", line 4
git lfs checkout build/filename
^
SyntaxError: invalid syntax
git lfs checkout build/filename
^
SyntaxError: invalid syntax
-Andy.
Andy Pont <andy.pont@...>
I wrote...
So, do_fetch() is a Python function and you can’t run shell commands in it. That is fine, I have let it do its thing to initialise the git repository and moved my custom commands into a do_configure() to replace the standard one.Unfortunately, there are other files in the same directory that can’t be checked out. Looking at that documentation, I have added the “nocheckout=1” option to the git definition in SRC_URI which is stopping the initial pull.When I try to add any git commands into a do_fetch_append() then it dumps a load of Python looking stuff on the screen that ends with:File "autogenerated", line 4
git lfs checkout build/filename
^
SyntaxError: invalid syntax
It will pull a single file out of the repo using "git checkout master <filename>” but is having difficulties with “git lfs …” commands. That though looks like an issue there not with the bitbake recipe per-se.
-Andy.
Quentin Schulz
Hi Andy,
On Wed, Dec 18, 2019 at 11:23:25AM +0000, Andy Pont wrote:
directory, in which case that might work good enough.
I'd add a task between do_fetch and do_unpack for your file if the thing
I said above does not work out. I personally don't like putting stuff in
tasks which aren't meant to be here but that's personal taste.
(configure is not for pulling files).
Let us know what works for you best so people can find this thread later
on if they have the same issue :)
BR,
Quentin
On Wed, Dec 18, 2019 at 11:23:25AM +0000, Andy Pont wrote:
Quentin wrote...I'm curious if subpath can be the path to a file directly and not aI unfortunately don't have experience with git lfs.Unfortunately, there are other files in the same directory that can’t be
I'm thinking about the subpath argument to SRC_URI git fetcher.
https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher
checked out. Looking at that documentation, I have added the “nocheckout=1”
directory, in which case that might work good enough.
option to the git definition in SRC_URI which is stopping the initial pull.Interesting one, thanks for telling me :)
When I try to add any git commands into a do_fetch_append() then it dumps a
load of Python looking stuff on the screen that ends with:
File "autogenerated", line 4
git lfs checkout build/filename
^
SyntaxError: invalid syntax
I'd add a task between do_fetch and do_unpack for your file if the thing
I said above does not work out. I personally don't like putting stuff in
tasks which aren't meant to be here but that's personal taste.
(configure is not for pulling files).
Let us know what works for you best so people can find this thread later
on if they have the same issue :)
BR,
Quentin