[PATCH yocto-autobuilder-helper] run-docs-build: build from tags dynamically instead of static list


Michael Halstead
 

All new releases are Sphinx ready so we exclude old tags and build for
all the rest.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
---
scripts/run-docs-build | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/scripts/run-docs-build b/scripts/run-docs-build
index 910f03d..13df34a 100755
--- a/scripts/run-docs-build
+++ b/scripts/run-docs-build
@@ -7,6 +7,7 @@ ypdocs=$2/documentation/
bbdocs=$3/doc/
docs_buildtools=/srv/autobuilder/autobuilder.yoctoproject.org/pub/buildtools/x86_64-buildtools-docs-nativesdk-standalone-3.2+snapshot-20201105.sh
outputdir=$builddir/output
+excluded_tags="yocto-3.1.4 yocto-3.1.3 yocto-3.1.2 yocto-3.1.1 yocto-3.1 yocto-3.0.1 yocto-3.0 yocto-2.6.4 yocto-2.6.3 yocto-2.7.1 yocto-2.6.2 yocto-2.7 yocto-2.6.1 yocto-2.6 yocto-2.5.2 yocto-2.5.1 yocto-2.4.4 yocto-2.4.3 yocto-2.5 yocto-2.3.4 yocto-1.0.2 yocto-1.1.2 yocto-1.2.2 yocto-1.2.1 yocto-1.3 yocto-1.3.1 yocto-1.3.2 yocto-1.4.3 yocto-1.4.2 yocto-1.4.1 yocto-1.4 yocto-2.1.3 yocto-2.4.2 yocto-2.1.1 yocto-2.1.2 yocto-2.0.3 yocto-1.8.2 yocto-2.2.3 yocto-2.4.1 yocto-2.3.3 yocto-2.3.2 yocto-2.4 yocto-2.2.2 yocto-2.3.1 yocto-2.3 yocto-2.2.1 yocto-2.0.2 yocto-2.2 yocto-2.1 yocto-2.0.1 yocto-2.0 yocto-1.8.1 yocto-1.7.3 yocto-1.6.3 yocto-1.7.2 yocto-1.8 yocto-1.5.1"


cd $builddir
@@ -77,13 +78,18 @@ for branch in dunfell gatesgarth hardknott; do
done

# Yocto Project releases/tags
-for tag in 3.1.5 3.1.6 3.2 3.2.1 3.2.2 3.2.3; do
+cd $ypdocs
+for tag in $(git tag -l |grep 'yocto-' |sort); do
+ if [[ $excluded_tags =~ $tag ]]; then
+ continue
+ fi
cd $ypdocs
- git checkout yocto-$tag
+ git checkout $tag
make clean
make publish
- mkdir $outputdir/$tag
- cp -r ./_build/final/* $outputdir/$tag
+ version=$(echo $tag | cut -c7-)
+ mkdir $outputdir/$version
+ cp -r ./_build/final/* $outputdir/$version
done

# Update switchers.js with the copy from master ypdocs
--
2.30.2


Quentin Schulz
 

Hi Michael,

On April 6, 2021 10:58:20 PM UTC, Michael Halstead <mhalstead@linuxfoundation.org> wrote:
All new releases are Sphinx ready so we exclude old tags and build for
all the rest.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
---
scripts/run-docs-build | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/scripts/run-docs-build b/scripts/run-docs-build
index 910f03d..13df34a 100755
--- a/scripts/run-docs-build
+++ b/scripts/run-docs-build
@@ -7,6 +7,7 @@ ypdocs=$2/documentation/
bbdocs=$3/doc/
docs_buildtools=/srv/autobuilder/autobuilder.yoctoproject.org/pub/buildtools/x86_64-buildtools-docs-nativesdk-standalone-3.2+snapshot-20201105.sh
outputdir=$builddir/output
+excluded_tags="yocto-3.1.4 yocto-3.1.3 yocto-3.1.2 yocto-3.1.1 yocto-3.1 yocto-3.0.1 yocto-3.0 yocto-2.6.4 yocto-2.6.3 yocto-2.7.1 yocto-2.6.2 yocto-2.7 yocto-2.6.1 yocto-2.6 yocto-2.5.2 yocto-2.5.1 yocto-2.4.4 yocto-2.4.3 yocto-2.5 yocto-2.3.4 yocto-1.0.2 yocto-1.1.2 yocto-1.2.2 yocto-1.2.1 yocto-1.3 yocto-1.3.1 yocto-1.3.2 yocto-1.4.3 yocto-1.4.2 yocto-1.4.1 yocto-1.4 yocto-2.1.3 yocto-2.4.2 yocto-2.1.1 yocto-2.1.2 yocto-2.0.3 yocto-1.8.2 yocto-2.2.3 yocto-2.4.1 yocto-2.3.3 yocto-2.3.2 yocto-2.4 yocto-2.2.2 yocto-2.3.1 yocto-2.3 yocto-2.2.1 yocto-2.0.2 yocto-2.2 yocto-2.1 yocto-2.0.1 yocto-2.0 yocto-1.8.1 yocto-1.7.3 yocto-1.6.3 yocto-1.7.2 yocto-1.8 yocto-1.5.1"


cd $builddir
@@ -77,13 +78,18 @@ for branch in dunfell gatesgarth hardknott; do
done

# Yocto Project releases/tags
-for tag in 3.1.5 3.1.6 3.2 3.2.1 3.2.2 3.2.3; do
+cd $ypdocs
+for tag in $(git tag -l |grep 'yocto-' |sort); do
IIUC the man page,
git tag --list 'yocto-*' | sort
would be doing the same thing as the one command with grep above.
Discovered it recently so just wanted to share.

I guess this is something we can also do for bitbake Sphinx documentation?

Removed the git context inadvertently but, is =~ some bash built-in? I don't know what's the shebang on top but maybe we want to force it to bash since I'm not sure it's POSIX "compliant" anymore?

Reviewed-by: Quentin Schulz <foss@0leil.net>

Thanks,
Quentin


Nicolas Dechesne
 

hey!

On Wed, Apr 7, 2021 at 9:31 AM Quentin Schulz <foss@...> wrote:
Hi Michael,

On April 6, 2021 10:58:20 PM UTC, Michael Halstead <mhalstead@...> wrote:
>All new releases are Sphinx ready so we exclude old tags and build for
>all the rest.

Thanks for starting this!
 
>
>Signed-off-by: Michael Halstead <mhalstead@...>
>---
> scripts/run-docs-build | 14 ++++++++++----
> 1 file changed, 10 insertions(+), 4 deletions(-)
>
>diff --git a/scripts/run-docs-build b/scripts/run-docs-build
>index 910f03d..13df34a 100755
>--- a/scripts/run-docs-build
>+++ b/scripts/run-docs-build
>@@ -7,6 +7,7 @@ ypdocs=$2/documentation/
> bbdocs=$3/doc/
> docs_buildtools=/srv/autobuilder/autobuilder.yoctoproject.org/pub/buildtools/x86_64-buildtools-docs-nativesdk-standalone-3.2+snapshot-20201105.sh
> outputdir=$builddir/output
>+excluded_tags="yocto-3.1.4 yocto-3.1.3 yocto-3.1.2 yocto-3.1.1 yocto-3.1 yocto-3.0.1 yocto-3.0 yocto-2.6.4 yocto-2.6.3 yocto-2.7.1 yocto-2.6.2 yocto-2.7 yocto-2.6.1 yocto-2.6 yocto-2.5.2 yocto-2.5.1 yocto-2.4.4 yocto-2.4.3 yocto-2.5 yocto-2.3.4 yocto-1.0.2 yocto-1.1.2 yocto-1.2.2 yocto-1.2.1 yocto-1.3 yocto-1.3.1 yocto-1.3.2 yocto-1.4.3 yocto-1.4.2 yocto-1.4.1 yocto-1.4 yocto-2.1.3 yocto-2.4.2 yocto-2.1.1 yocto-2.1.2 yocto-2.0.3 yocto-1.8.2 yocto-2.2.3 yocto-2.4.1 yocto-2.3.3 yocto-2.3.2 yocto-2.4 yocto-2.2.2 yocto-2.3.1 yocto-2.3 yocto-2.2.1 yocto-2.0.2 yocto-2.2 yocto-2.1 yocto-2.0.1 yocto-2.0 yocto-1.8.1 yocto-1.7.3 yocto-1.6.3 yocto-1.7.2 yocto-1.8 yocto-1.5.1"
>
>
> cd $builddir
>@@ -77,13 +78,18 @@ for branch in dunfell gatesgarth hardknott; do
> done
>
> # Yocto Project releases/tags
>-for tag in 3.1.5 3.1.6 3.2 3.2.1 3.2.2 3.2.3; do
>+cd $ypdocs
>+for tag in $(git tag -l  |grep 'yocto-' |sort); do

IIUC the man page,
git tag --list 'yocto-*' | sort

sort -V is even better since it does "natural sort of (version) numbers within text", let's get ready for 3.10 ;) 
 
And using -V, how about something along these lines:

v_sphinx='yocto-3.1.5'
for v in $(git tag --list 'yocto-*'); do
    first=$(printf '%s\n%s' $v $v_sphinx | sort -V | head -n1)
    if [ "$first" = "$v_sphinx" ]; then
        echo "Yocto $v uses Sphinx!"
    fi
done

and it outputs the following when I run it locally:

Yocto yocto-3.1.5 uses Sphinx!
Yocto yocto-3.1.6 uses Sphinx!
Yocto yocto-3.2 uses Sphinx!
Yocto yocto-3.2.1 uses Sphinx!
Yocto yocto-3.2.2 uses Sphinx!
Yocto yocto-3.2.3 uses Sphinx!



would be doing the same thing as the one command with grep above.
Discovered it recently so just wanted to share.

I guess this is something we can also do for bitbake Sphinx documentation?

yes.
 

Removed the git context inadvertently but, is =~ some bash built-in? I don't know what's the shebang on top but maybe we want to force it to bash since I'm not sure it's POSIX "compliant" anymore? 

It is bash already. 
 

Reviewed-by: Quentin Schulz <foss@...>

Thanks,
Quentin


Nicolas Dechesne
 



On Wed, Apr 7, 2021 at 10:16 AM Nicolas Dechesne <nicolas.dechesne@...> wrote:
hey!

On Wed, Apr 7, 2021 at 9:31 AM Quentin Schulz <foss@...> wrote:
Hi Michael,

On April 6, 2021 10:58:20 PM UTC, Michael Halstead <mhalstead@...> wrote:
>All new releases are Sphinx ready so we exclude old tags and build for
>all the rest.

Thanks for starting this!
 
>
>Signed-off-by: Michael Halstead <mhalstead@...>
>---
> scripts/run-docs-build | 14 ++++++++++----
> 1 file changed, 10 insertions(+), 4 deletions(-)
>
>diff --git a/scripts/run-docs-build b/scripts/run-docs-build
>index 910f03d..13df34a 100755
>--- a/scripts/run-docs-build
>+++ b/scripts/run-docs-build
>@@ -7,6 +7,7 @@ ypdocs=$2/documentation/
> bbdocs=$3/doc/
> docs_buildtools=/srv/autobuilder/autobuilder.yoctoproject.org/pub/buildtools/x86_64-buildtools-docs-nativesdk-standalone-3.2+snapshot-20201105.sh
> outputdir=$builddir/output
>+excluded_tags="yocto-3.1.4 yocto-3.1.3 yocto-3.1.2 yocto-3.1.1 yocto-3.1 yocto-3.0.1 yocto-3.0 yocto-2.6.4 yocto-2.6.3 yocto-2.7.1 yocto-2.6.2 yocto-2.7 yocto-2.6.1 yocto-2.6 yocto-2.5.2 yocto-2.5.1 yocto-2.4.4 yocto-2.4.3 yocto-2.5 yocto-2.3.4 yocto-1.0.2 yocto-1.1.2 yocto-1.2.2 yocto-1.2.1 yocto-1.3 yocto-1.3.1 yocto-1.3.2 yocto-1.4.3 yocto-1.4.2 yocto-1.4.1 yocto-1.4 yocto-2.1.3 yocto-2.4.2 yocto-2.1.1 yocto-2.1.2 yocto-2.0.3 yocto-1.8.2 yocto-2.2.3 yocto-2.4.1 yocto-2.3.3 yocto-2.3.2 yocto-2.4 yocto-2.2.2 yocto-2.3.1 yocto-2.3 yocto-2.2.1 yocto-2.0.2 yocto-2.2 yocto-2.1 yocto-2.0.1 yocto-2.0 yocto-1.8.1 yocto-1.7.3 yocto-1.6.3 yocto-1.7.2 yocto-1.8 yocto-1.5.1"
>
>
> cd $builddir
>@@ -77,13 +78,18 @@ for branch in dunfell gatesgarth hardknott; do
> done
>
> # Yocto Project releases/tags
>-for tag in 3.1.5 3.1.6 3.2 3.2.1 3.2.2 3.2.3; do
>+cd $ypdocs
>+for tag in $(git tag -l  |grep 'yocto-' |sort); do

IIUC the man page,
git tag --list 'yocto-*' | sort

sort -V is even better since it does "natural sort of (version) numbers within text", let's get ready for 3.10 ;) 
 
And using -V, how about something along these lines:

v_sphinx='yocto-3.1.5'
for v in $(git tag --list 'yocto-*'); do
    first=$(printf '%s\n%s' $v $v_sphinx | sort -V | head -n1)
    if [ "$first" = "$v_sphinx" ]; then
        echo "Yocto $v uses Sphinx!"
    fi
done

and it outputs the following when I run it locally:

Yocto yocto-3.1.5 uses Sphinx!
Yocto yocto-3.1.6 uses Sphinx!
Yocto yocto-3.2 uses Sphinx!
Yocto yocto-3.2.1 uses Sphinx!
Yocto yocto-3.2.2 uses Sphinx!
Yocto yocto-3.2.3 uses Sphinx!



would be doing the same thing as the one command with grep above.
Discovered it recently so just wanted to share.

I guess this is something we can also do for bitbake Sphinx documentation?

yes.

Well, in fact no , we shouldn't, is a short answer ;) 

Here is the long answer.. 

For bitbake and YP, we currently publish: master, master-next and 'stable' branches (e.g. 1.46, 1.48, 1.50 for bitbake dunfell, gatesgarth, hardknott for YP). Then we also publish YP releases (and point releases), e.g. 3.1.5 and beyond, but  we don't publish anything for releases/point release of bitbake, At least just yet. And it's a 'bug'. Since the YP docs includes links (with intersphinx)  to bitbake docs, then when we look at a master, master-next or any branches on docs.yp.org, then the links are correct. for example if you look at

vs

this page has links to bitbake and it points respectively to:
and

Which is correct. 

[Side note: I've noticed that the hardknott page is wrong, since
I will send a patch for that]

Now if we look at a YP release, such as 
it has this link:

which is not correct, since it links to a bitbake 'branch' docs build, and not a bitbake 'release' docs build. 


 

Removed the git context inadvertently but, is =~ some bash built-in? I don't know what's the shebang on top but maybe we want to force it to bash since I'm not sure it's POSIX "compliant" anymore? 

It is bash already. 
 

Reviewed-by: Quentin Schulz <foss@...>

Thanks,
Quentin