Pull a single file using git lfs


Andy Pont
 

Hello,

I have a private git repo on our GitLab server that contains a bunch of source files that came as part of a commercial product and the final binary executable is also stored there using “git lfs”.

I’m trying to figure out how to create a bitbake recipe so that in the do_fetch() stage it just pulls the single binary file. I think the file can be got using “git lfs checkout path/to/file” but I’m not sure how to make it connect to the repo and not do a regular “git pull”.

-Andy.


Quentin Schulz
 

Hi Andy,

On Wed, Dec 18, 2019 at 10:46:17AM +0000, Andy Pont wrote:
Hello,

I have a private git repo on our GitLab server that contains a bunch of
source files that came as part of a commercial product and the final binary
executable is also stored there using “git lfs”.

I’m trying to figure out how to create a bitbake recipe so that in the
do_fetch() stage it just pulls the single binary file. I think the file can
be got using “git lfs checkout path/to/file” but I’m not sure how to make it
connect to the repo and not do a regular “git pull”.
I unfortunately don't have experience with git lfs.

I'm thinking about the subpath argument to SRC_URI git fetcher.

https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher

""subpath": Limits the checkout to a specific subpath of the tree. By
default, the whole tree is checked out."

So maybe by giving the path to the file you'll be able to get it without
the whole repo. Provided you're able to get those files with git pull
and not git lfs.

The other thing could be to use a do_fetch_append() with the call to git
lfs in there. I think the git used in that task will be the host one, so
you should then have support for git lfs provided you installed it on
the build machine.

Hopefully I'm not answering with complete non-sense, and giving you some
hints on what to test a bit more :)

Good luck,
Quentin


Andy Pont
 

Quentin wrote...

I unfortunately don't have experience with git lfs.
 
I'm thinking about the subpath argument to SRC_URI git fetcher.
 
https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher
Unfortunately, there are other files in the same directory that can’t be checked out. Looking at that documentation, I have added the “nocheckout=1” option to the git definition in SRC_URI which is stopping the initial pull.

When I try to add any git commands into a do_fetch_append() then it dumps a load of Python looking stuff on the screen that ends with:

  File "autogenerated", line 4
    git lfs checkout build/filename
               ^
SyntaxError: invalid syntax

-Andy.


Andy Pont
 

I wrote...

Unfortunately, there are other files in the same directory that can’t be checked out. Looking at that documentation, I have added the “nocheckout=1” option to the git definition in SRC_URI which is stopping the initial pull.

When I try to add any git commands into a do_fetch_append() then it dumps a load of Python looking stuff on the screen that ends with:

  File "autogenerated", line 4
    git lfs checkout build/filename
               ^
SyntaxError: invalid syntax
So, do_fetch() is a Python function and you can’t run shell commands in it.  That is fine, I have let it do its thing to initialise the git repository and moved my custom commands into a do_configure() to replace the standard one.

It will pull a single file out of the repo using "git checkout master <filename>” but is having difficulties with “git lfs …” commands.  That though looks like an issue there not with the bitbake recipe per-se.

-Andy.



Quentin Schulz
 

Hi Andy,

On Wed, Dec 18, 2019 at 11:23:25AM +0000, Andy Pont wrote:
Quentin wrote...

I unfortunately don't have experience with git lfs.

I'm thinking about the subpath argument to SRC_URI git fetcher.

https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#git-fetcher
Unfortunately, there are other files in the same directory that can’t be
checked out. Looking at that documentation, I have added the “nocheckout=1”
I'm curious if subpath can be the path to a file directly and not a
directory, in which case that might work good enough.

option to the git definition in SRC_URI which is stopping the initial pull.

When I try to add any git commands into a do_fetch_append() then it dumps a
load of Python looking stuff on the screen that ends with:

File "autogenerated", line 4
git lfs checkout build/filename
^
SyntaxError: invalid syntax
Interesting one, thanks for telling me :)

I'd add a task between do_fetch and do_unpack for your file if the thing
I said above does not work out. I personally don't like putting stuff in
tasks which aren't meant to be here but that's personal taste.
(configure is not for pulling files).

Let us know what works for you best so people can find this thread later
on if they have the same issue :)

BR,

Quentin