Date   

Re: [[meta-mingw][PATCH] libgcrypt: disable amd64 optimization

Mark Hatle
 

Looking at the config.log, I only see the standard Yocto Project (nativesdk)
paths, as well as: --disable-asm --without-capabilities

I'm building on 18.04.4 LTS, I think it's a fairly standard config, no
containers or anything like that.

Hardware is Intel Xeon Gold 6130 CPU.

Using my other build machine, an 18.04.03 LTS machine, hardware is Intel Xeon
E5-2697v2.

So two different, but similar environments I get exactly the same problem. I'm
pretty sure the original patch I sent is still needed, and simply won't cause
any issues for environments where this is not reproducible. Worst case it
disables optimizations, and libgcrypt may be slightly slower (if the
optimizations even worked.)

(if anyone wants to look, I can give you access to the second build machine
where it's easily reproducible.)

--Mark

On 12/17/20 10:17 AM, Khem Raj wrote:
On Wed, Dec 16, 2020 at 4:50 PM Mark Hatle
<mark.hatle@kernel.crashing.org> wrote:

Note, this is still happening. Using meta-mingw w/ gatesgarth to build, and
including libgcrypt I get the same failure. The failure resolves with
"--disable-amd64-as-feature-detection".

Looking at the code, I don't really understand it, but there are some lines
after that which may be more indicative of the actual error:

../../libgcrypt-1.8.6/cipher/rijndael.c: In function 'do_encrypt':
../../libgcrypt-1.8.6/cipher/rijndael-tables.h:103:24: error: expected ']'
before '.' token
103 | #define encT enc_tables.T
| ^
../../libgcrypt-1.8.6/cipher/rijndael.c:775:20: note: in expansion of macro 'encT'
775 | [encT] "r" (encT)
| ^~~~
../../libgcrypt-1.8.6/cipher/rijndael.c:776:17: error: undefined named operand
'encT'
776 | : "cc", "memory", "r8", "r9", "r10", "r11");
| ^
does it have --enable-optimization switch on ? if so then its probing
the build platform
for features and detects XMM registers and enables them but perhaps
does not account
for it not being a sysv ABI based platform for right ABI calling convention
may be asm code have made some assumptions here

I can't find any difference in generated content anywhere, only that the
function (.c) that is failing seems to have a comments it's loading something
for use w/ a SysV style function:

# else
/* Call SystemV ABI function without storing non-volatile XMM registers,
* as target function does not use vector instruction sets. */
const void *key = ctx->keyschenc;
uintptr_t rounds = ctx->rounds;
uintptr_t ret;
asm volatile ("movq %[encT], %%r8\n\t"
"callq *%[ret]\n\t"
: [ret] "=a" (ret),
"+D" (key),
"+S" (bx),
"+d" (ax),
"+c" (rounds)
: "0" (_gcry_aes_amd64_encrypt_block),

As far as my configuration (all gatesgarth):

conf/bblayers.conf:
BBLAYERS ?= " \
/scratch2/mhatle/2021/sources/poky/meta \
/scratch2/mhatle/2021/sources/poky/meta-poky \
/scratch2/mhatle/2021/sources/meta-mingw \
/scratch2/mhatle/2021/sources/poky/../meta-xilinx/meta-xilinx-bsp \
/scratch2/mhatle/2021/sources/poky/../meta-xilinx-tools \
/scratch2/mhatle/2021/sources/poky/../meta-openembedded/meta-python \
/scratch2/mhatle/2021/sources/poky/../meta-openembedded/meta-oe \
"

DISTRO=nodistro MACHINE=zynqmp-generic SDKMACHINE=x86_64-mingw32 bitbake
nativesdk-libgcrypt

On 3/3/20 9:19 AM, Joshua Watt wrote:

On 2/25/20 1:32 PM, Mark Hatle wrote:

On 2/25/20 12:36 PM, Adrian Bunk wrote:
On Tue, Feb 25, 2020 at 12:02:14PM -0600, Mark Hatle wrote:
amd64 optimization when using with mingw32 sometimes results in:

| ../../libgcrypt-1.8.4/cipher/rijndael-tables.h:103:24: error: expected
']' before '.' token
| 103 | #define encT enc_tables.T
...
Please send a complete build log, this looks like a parallel build
failure where some .c is compiled before some header it uses is
fully generated (or something similar).
I'll reply with a private email to where you can get a copy of the build directory.

--- a/recipes-support/libgcrypt/libgcrypt_%.bbappend
+++ b/recipes-support/libgcrypt/libgcrypt_%.bbappend
@@ -1,3 +1,4 @@
+EXTRA_OECONF_append_mingw32 = " --disable-amd64-as-feature-detection"
...
I assume PARALLEL_MAKE = "" would also work instead?
I tried this and it didn't appear to make a difference. I only see the problem
when building for mingw32.
Did we ever figure out what was causing this? It does appear to be some
sort of race condition at the moment, and I'm surprised we haven't seen
it on the autobuilder (we have pretty good test coverage for meta-mingw
there).


I've never seen any issues for any other SDK target.

--Mark

cu
Adrian








Re: [[meta-mingw][PATCH] libgcrypt: disable amd64 optimization

Khem Raj
 

On Wed, Dec 16, 2020 at 4:50 PM Mark Hatle
<mark.hatle@kernel.crashing.org> wrote:

Note, this is still happening. Using meta-mingw w/ gatesgarth to build, and
including libgcrypt I get the same failure. The failure resolves with
"--disable-amd64-as-feature-detection".

Looking at the code, I don't really understand it, but there are some lines
after that which may be more indicative of the actual error:

../../libgcrypt-1.8.6/cipher/rijndael.c: In function 'do_encrypt':
../../libgcrypt-1.8.6/cipher/rijndael-tables.h:103:24: error: expected ']'
before '.' token
103 | #define encT enc_tables.T
| ^
../../libgcrypt-1.8.6/cipher/rijndael.c:775:20: note: in expansion of macro 'encT'
775 | [encT] "r" (encT)
| ^~~~
../../libgcrypt-1.8.6/cipher/rijndael.c:776:17: error: undefined named operand
'encT'
776 | : "cc", "memory", "r8", "r9", "r10", "r11");
| ^
does it have --enable-optimization switch on ? if so then its probing
the build platform
for features and detects XMM registers and enables them but perhaps
does not account
for it not being a sysv ABI based platform for right ABI calling convention
may be asm code have made some assumptions here

I can't find any difference in generated content anywhere, only that the
function (.c) that is failing seems to have a comments it's loading something
for use w/ a SysV style function:

# else
/* Call SystemV ABI function without storing non-volatile XMM registers,
* as target function does not use vector instruction sets. */
const void *key = ctx->keyschenc;
uintptr_t rounds = ctx->rounds;
uintptr_t ret;
asm volatile ("movq %[encT], %%r8\n\t"
"callq *%[ret]\n\t"
: [ret] "=a" (ret),
"+D" (key),
"+S" (bx),
"+d" (ax),
"+c" (rounds)
: "0" (_gcry_aes_amd64_encrypt_block),

As far as my configuration (all gatesgarth):

conf/bblayers.conf:
BBLAYERS ?= " \
/scratch2/mhatle/2021/sources/poky/meta \
/scratch2/mhatle/2021/sources/poky/meta-poky \
/scratch2/mhatle/2021/sources/meta-mingw \
/scratch2/mhatle/2021/sources/poky/../meta-xilinx/meta-xilinx-bsp \
/scratch2/mhatle/2021/sources/poky/../meta-xilinx-tools \
/scratch2/mhatle/2021/sources/poky/../meta-openembedded/meta-python \
/scratch2/mhatle/2021/sources/poky/../meta-openembedded/meta-oe \
"

DISTRO=nodistro MACHINE=zynqmp-generic SDKMACHINE=x86_64-mingw32 bitbake
nativesdk-libgcrypt

On 3/3/20 9:19 AM, Joshua Watt wrote:

On 2/25/20 1:32 PM, Mark Hatle wrote:

On 2/25/20 12:36 PM, Adrian Bunk wrote:
On Tue, Feb 25, 2020 at 12:02:14PM -0600, Mark Hatle wrote:
amd64 optimization when using with mingw32 sometimes results in:

| ../../libgcrypt-1.8.4/cipher/rijndael-tables.h:103:24: error: expected
']' before '.' token
| 103 | #define encT enc_tables.T
...
Please send a complete build log, this looks like a parallel build
failure where some .c is compiled before some header it uses is
fully generated (or something similar).
I'll reply with a private email to where you can get a copy of the build directory.

--- a/recipes-support/libgcrypt/libgcrypt_%.bbappend
+++ b/recipes-support/libgcrypt/libgcrypt_%.bbappend
@@ -1,3 +1,4 @@
+EXTRA_OECONF_append_mingw32 = " --disable-amd64-as-feature-detection"
...
I assume PARALLEL_MAKE = "" would also work instead?
I tried this and it didn't appear to make a difference. I only see the problem
when building for mingw32.
Did we ever figure out what was causing this? It does appear to be some
sort of race condition at the moment, and I'm surprised we haven't seen
it on the autobuilder (we have pretty good test coverage for meta-mingw
there).


I've never seen any issues for any other SDK target.

--Mark

cu
Adrian




Re: [[meta-mingw][PATCH] libgcrypt: disable amd64 optimization

Mark Hatle
 

Note, this is still happening. Using meta-mingw w/ gatesgarth to build, and
including libgcrypt I get the same failure. The failure resolves with
"--disable-amd64-as-feature-detection".

Looking at the code, I don't really understand it, but there are some lines
after that which may be more indicative of the actual error:

../../libgcrypt-1.8.6/cipher/rijndael.c: In function 'do_encrypt':
../../libgcrypt-1.8.6/cipher/rijndael-tables.h:103:24: error: expected ']'
before '.' token
  103 | #define encT enc_tables.T
      |                        ^
../../libgcrypt-1.8.6/cipher/rijndael.c:775:20: note: in expansion of macro 'encT'
  775 |                   [encT] "r" (encT)
      |                    ^~~~
../../libgcrypt-1.8.6/cipher/rijndael.c:776:17: error: undefined named operand
'encT'
  776 |                 : "cc", "memory", "r8", "r9", "r10", "r11");
      |                 ^

I can't find any difference in generated content anywhere, only that the
function (.c) that is failing seems to have a comments it's loading something
for use w/ a SysV style function:

# else
  /* Call SystemV ABI function without storing non-volatile XMM registers,
   * as target function does not use vector instruction sets. */
  const void *key = ctx->keyschenc;
  uintptr_t rounds = ctx->rounds;  
  uintptr_t ret;
  asm volatile ("movq %[encT], %%r8\n\t"
                "callq *%[ret]\n\t"
                : [ret] "=a" (ret),
                  "+D" (key),
                  "+S" (bx),
                  "+d" (ax),
                  "+c" (rounds)
                : "0" (_gcry_aes_amd64_encrypt_block),

As far as my configuration (all gatesgarth):

conf/bblayers.conf:
BBLAYERS ?= " \
    /scratch2/mhatle/2021/sources/poky/meta \
    /scratch2/mhatle/2021/sources/poky/meta-poky \
    /scratch2/mhatle/2021/sources/meta-mingw \
    /scratch2/mhatle/2021/sources/poky/../meta-xilinx/meta-xilinx-bsp \
    /scratch2/mhatle/2021/sources/poky/../meta-xilinx-tools \
    /scratch2/mhatle/2021/sources/poky/../meta-openembedded/meta-python \
    /scratch2/mhatle/2021/sources/poky/../meta-openembedded/meta-oe \
"

DISTRO=nodistro MACHINE=zynqmp-generic SDKMACHINE=x86_64-mingw32 bitbake
nativesdk-libgcrypt

On 3/3/20 9:19 AM, Joshua Watt wrote:

On 2/25/20 1:32 PM, Mark Hatle wrote:

On 2/25/20 12:36 PM, Adrian Bunk wrote:
On Tue, Feb 25, 2020 at 12:02:14PM -0600, Mark Hatle wrote:
amd64 optimization when using with mingw32 sometimes results in:

| ../../libgcrypt-1.8.4/cipher/rijndael-tables.h:103:24: error: expected
']' before '.' token
| 103 | #define encT enc_tables.T
...
Please send a complete build log, this looks like a parallel build
failure where some .c is compiled before some header it uses is
fully generated (or something similar).
I'll reply with a private email to where you can get a copy of the build directory.

--- a/recipes-support/libgcrypt/libgcrypt_%.bbappend
+++ b/recipes-support/libgcrypt/libgcrypt_%.bbappend
@@ -1,3 +1,4 @@
+EXTRA_OECONF_append_mingw32 = " --disable-amd64-as-feature-detection"
...
I assume PARALLEL_MAKE = "" would also work instead?
I tried this and it didn't appear to make a difference. I only see the problem
when building for mingw32.
Did we ever figure out what was causing this? It does appear to be some
sort of race condition at the moment, and I'm surprised we haven't seen
it on the autobuilder (we have pretty good test coverage for meta-mingw
there).


I've never seen any issues for any other SDK target.

--Mark

cu
Adrian



Re: Cross compiler which runs on the target architecture

Stefan Herbrechtsmeier
 

Hi Khem,

Am 09.12.20 um 19:23 schrieb Khem Raj:
On 12/9/20 4:38 AM, Stefan Herbrechtsmeier wrote:
Hi Peter and Richard,

does a follow up of this old topic (thread) regarding a cross compiler which runs on the target architecture exists?
https://www.yoctoproject.org/pipermail/yocto/2014-December/022782.html
I dont think there were further discussions. However, I think the cross-canadian approach is perhaps step in right direction. We would need to enhance it to be able to build multiple cross-canadian tuples instead of only one that we build today which is based on SDK_MACHINE and TARGET_ARCH
The cross-canadian changes the HOST variables and not the TARGET_ variables. I need to change the TARGET variables but keep the HOST variables. It looks like this isn't really intended because the HOST_PREFIX use the TARGET_PREFIX for example.

Regards
Stefan


Re: [meta-tensorflow][PATCH 1/25] openjdk-8-native: 212b04 -> 275b01

Kent Dorfman
 

Is there a project policy reason that all these patches are posted in
the main list instead of a dev list? Isn't there a list specifically
for this kind of thing, to separate questions and info sharing from
the software maintenance aspect?

On 12/16/20, hongxu <hongxu.jia@windriver.com> wrote:
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
...njdk-8-native_212b04.bb => openjdk-8-native_275b01.bb} | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
rename recipes-devtools/openjdk/{openjdk-8-native_212b04.bb =>
openjdk-8-native_275b01.bb} (78%)

diff --git a/recipes-devtools/openjdk/openjdk-8-native_212b04.bb
b/recipes-devtools/openjdk/openjdk-8-native_275b01.bb
similarity index 78%
rename from recipes-devtools/openjdk/openjdk-8-native_212b04.bb
rename to recipes-devtools/openjdk/openjdk-8-native_275b01.bb
index 788612a..dcc1780 100644
--- a/recipes-devtools/openjdk/openjdk-8-native_212b04.bb
+++ b/recipes-devtools/openjdk/openjdk-8-native_275b01.bb
@@ -3,13 +3,13 @@ builds using source code from OpenJDK project"
LICENSE = "GPL-2.0-with-classpath-exception"
LIC_FILES_CHKSUM = "file://LICENSE;md5=3e0b59f8fac05c3c03d4a26bbda13f8f"

-SRC_URI[md5sum] = "8082ad8dafec378f2a4b24cbfdb4a9a4"
-SRC_URI[sha256sum] =
"ef6a3050a1c3477a6e13c24d10ab36decad548649a260559d466467401db15de"
+SRC_URI[md5sum] = "52c1f769ab67b58b4300713fb5d46a47"
+SRC_URI[sha256sum] =
"fccaa6cc14571813dbb427ac08d5acd034782a2654e6090ad4d63e7200011ac4"
SRC_URI = " \
-
https://github.com/ojdkbuild/contrib_jdk8u-ci/releases/download/jdk8u212-b04/jdk-8u212-ojdkbuild-linux-x64.zip
\
+
https://github.com/ojdkbuild/contrib_jdk8u-ci/releases/download/jdk8u275-b01/jdk-8u275-ojdkbuild-linux-x64.zip
\
"

-S = "${WORKDIR}/jdk-8u212-ojdkbuild-linux-x64"
+S = "${WORKDIR}/jdk-8u275-ojdkbuild-linux-x64"

do_patch[noexec] = "1"
do_configure[noexec] = "1"
--
2.21.0


Re: Cross compiler which runs on the target architecture

Stefan Herbrechtsmeier
 

Hi Peter,

Am 09.12.20 um 14:32 schrieb Peter Saunderson:
The proof of concept layer is http://layers.openembedded.org/layerindex/branch/master/layer/meta-exotic/ - https://github.com/peteasa/meta-exotic/wiki.  I created a set of cross compilers for the Epiphany processor http://www.adapteva.com/epiphanyiii/ using this layer and the layer https://github.com/peteasa/meta-epiphany that builds the specific cross compilers for that processor.  It worked quite well with yocto-1.7.1 so it's quite an old bit of code, however the ideas could work in a more recent versions of Yocto.
Thanks for the links but your solution looks very complex. I will try to improve the examples from Richard.

Regards
Stefan

On Wed, 9 Dec 2020 at 12:43, Stefan Herbrechtsmeier <stefan@herbrechtsmeier.net <mailto:stefan@herbrechtsmeier.net>> wrote:
Sorry I have an error in Peter`s mail address.
Am 09.12.2020 um 13:38 schrieb Stefan Herbrechtsmeier:
> Hi Peter and Richard,
>
> does a follow up of this old topic (thread) regarding a cross
compiler
> which runs on the target architecture exists?
>
https://www.yoctoproject.org/pipermail/yocto/2014-December/022782.html
>
> Kind regards
>   Stefan
>
>
>


Re: Review request 0/25:[meta-tensorflow] upgrade tensorflow 1.13.1 -> 2.4.0.rc4

hongxu
 

This series requires to work with two OE patches which I've submitted to OE mailing list

//Hongxu

On 12/16/20 9:07 PM, hongxu wrote:
# Build and Run
## 1. Clone away
```
$ mkdir <ts-project>
$ cd <ts-project>
$ git clone git://git.yoctoproject.org/meta-tensorflow
$ git clone git://git.openembedded.org/meta-openembedded
$ git clone git://git.openembedded.org/openembedded-core oe-core
$ cd oe-core
$ git clone git://git.openembedded.org/bitbake
```

## 2. Prepare build
```
$ . <ts-project>/oe-core/oe-init-build-env <build>

# Build qemux86-64 which runqemu supports kvm.
$ echo 'MACHINE = "qemux86-64"' >> conf/local.conf

$ echo 'IMAGE_INSTALL_append = " tensorflow"' >> conf/local.conf

Edit conf/bblayers.conf to include other layers
BBLAYERS ?= " \ 
    <ts-project>/oe-core/meta \
    <ts-project>/meta-openembedded/meta-python \
    <ts-project>/meta-openembedded/meta-oe \
    <ts-project>/meta-tensorflow \
"
```
## 3. Build image                                                                                                                      
```                                                                                                                                    
cd <build>                                                                                                                             
$ bitbake core-image-minimal                                                                                                           
```

## 4. Start qemu with slrip + kvm + 5GB memory:
```
$ runqemu qemux86-64 core-image-minimal slirp kvm qemuparams="-m 5120"
```

## 5. Verify the install
```
root@qemux86-64:~# python3 -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
tf.Tensor(-3304.6208, shape=(), dtype=float32)

## 6. Run tutorial case
### Refer: https://www.tensorflow.org/tutorials
```
root@qemux86-64:~# cat >code-v2.py <<ENDOF
import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])

predictions = model(x_train[:1]).numpy()
tf.nn.softmax(predictions).numpy()
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
loss_fn(y_train[:1], predictions).numpy()

model.compile(optimizer='adam',
              loss=loss_fn,
              metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test,  y_test, verbose=2)

probability_model = tf.keras.Sequential([
  model,
  tf.keras.layers.Softmax()
])
probability_model(x_test[:5])

ENDOF

root@qemux86-64:~# python3 ./code-v2.py
2020-12-15 08:16:44.171593: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2020-12-15 08:16:44.184464: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3099995000 Hz
Epoch 1/5
1875/1875 [==============================] - 14s 7ms/step - loss: 0.4833 - accuracy: 0.8595
Epoch 2/5
1875/1875 [==============================] - 13s 7ms/step - loss: 0.1549 - accuracy: 0.9558
Epoch 3/5
1875/1875 [==============================] - 13s 7ms/step - loss: 0.1135 - accuracy: 0.9651
Epoch 4/5
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0889 - accuracy: 0.9729
Epoch 5/5
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0741 - accuracy: 0.9777
313/313 - 1s - loss: 0.0757 - accuracy: 0.9757
```
## 7. TensorFlow/TensorFlow Lite C++ Image Recognition Demo                                                                            
```                                                                                                                                    
root@qemux86-64:~# time label_image                                                                                                    
2020-12-15 08:18:34.853885: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3099995000 Hz                    
2020-12-15 08:18:41.565167: I tensorflow/examples/label_image/main.cc:252] military uniform (653): 0.834306                            
2020-12-15 08:18:41.567874: I tensorflow/examples/label_image/main.cc:252] mortarboard (668): 0.0218696                                
2020-12-15 08:18:41.568936: I tensorflow/examples/label_image/main.cc:252] academic gown (401): 0.0103581                              
2020-12-15 08:18:41.569985: I tensorflow/examples/label_image/main.cc:252] pickelhaube (716): 0.00800819                               
2020-12-15 08:18:41.571025: I tensorflow/examples/label_image/main.cc:252] bulletproof vest (466): 0.00535086                          
                                                                                                                                       
real    0m7.178s                                                                                                                       
user    0m6.101s                                                                                                                       
sys 0m0.893s                                                                                                                           
                                                                                                                                       
root@qemux86-64:~# time label_image.lite                                                                                               
INFO: Loaded model /usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite                                                            
INFO: resolved reporter
INFO: invoked
INFO: average time: 213.584 ms
INFO: 0.780392: 653 military uniform
INFO: 0.105882: 907 Windsor tie
INFO: 0.0156863: 458 bow tie
INFO: 0.0117647: 466 bulletproof vest
INFO: 0.00784314: 835 suit

real    0m0.233s
user    0m0.216s
sys 0m0.012s





[meta-tensorflow][PATCH 25/25] update LIMITATION.md

hongxu
 

Target arch only supports 32 bit arm and 32 bit x86, 64 bit arm and
64 bit x86. BSP (MACHINE) incluced in above archs should be supported.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
LIMITATION.md | 20 +++++++++-----------
1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/LIMITATION.md b/LIMITATION.md
index deda0ad..3c69100 100644
--- a/LIMITATION.md
+++ b/LIMITATION.md
@@ -1,21 +1,19 @@
# Limitation
```
-* Bazel build takes lots of time, since it like bitbake which has own rules
- and builds everything from scratch. Currently bazel could not reuse Yocto
- DEPENDS/RDEPENDS.
+* Target arch only supports 32 bit arm and 32 bit x86, 64 bit arm and
+ 64 bit x86. BSP (MACHINE) incluced in above archs should be supported.
+
+* Bazel build takes lots of time, it has own rules and builds everything
+ from scratch. Currently bazel could not reuse Yocto DEPENDS/RDEPENDS.

* In order to run tensorflow cases in a reasonable time, although it builds
- successfully on qemuarm, qemuarm64, qemumips, qemumips64, qemux86 and
- qemux86-64, only qemux86-64 with kvm for runtime test.
+ successfully on qemuarm, qemuarm64, qemux86 and qemux86-64, only qemux86-64
+ with kvm for runtime test.

-* It failed to use pre-build model to do predict/inference on big-endian platform
- (such as qemumips), since upstream does not support big-endian very well
+* It failed to use pre-build model to do predict/inference on big-endian
+ platform, since upstream does not support big-endian very well
https://github.com/tensorflow/tensorflow/issues/16364

-* Do not support 32-bit powerpc (qemuppc) since BoringSSL does not support it.
- (BoringSSL is a fork of OpenSSL used to implement cryptography and TLS across
- most of Google's products)
-
* If host(build system) is not x86_64, please add meta-java to BBLAYERS in
conf/bblayers.conf (git://git.yoctoproject.org/meta-java)
```
--
2.21.0


[meta-tensorflow][PATCH 24/25] tensorflow: split sub packages

hongxu
 

Split sub packages (libtensorflow-c libtensorflow-framework
label-image label-image-lite python3-tensorflow), then user
chould select the required one rather than install them
as a whole

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
recipes-framework/tensorflow/tensorflow_2.4.0.bb | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/recipes-framework/tensorflow/tensorflow_2.4.0.bb b/recipes-framework/tensorflow/tensorflow_2.4.0.bb
index 49c868e..6c4163e 100644
--- a/recipes-framework/tensorflow/tensorflow_2.4.0.bb
+++ b/recipes-framework/tensorflow/tensorflow_2.4.0.bb
@@ -168,7 +168,20 @@ FILES_${PN}-dev = ""
INSANE_SKIP_${PN} += "dev-so \
already-stripped \
"
-FILES_${PN} += "${libdir}/* ${datadir}/*"
+
+PACKAGE_BEFORE_PN += "libtensorflow-c libtensorflow-framework label-image label-image-lite python3-tensorflow"
+
+RDEPENDS_label-image += "libtensorflow-framework"
+RDEPENDS_python3-tensorflow += "libtensorflow-framework"
+RDEPENDS_${PN} += "libtensorflow-c libtensorflow-framework label-image label-image-lite python3-tensorflow"
+
+ALLOW_EMPTY_${PN} = "1"
+
+FILES_python3-tensorflow += "${libdir}/* ${datadir}/* ${sbindir}/*"
+FILES_libtensorflow-c = "${libdir}/libtensorflow.so ${libdir}/libtensorflow_cc.so"
+FILES_libtensorflow-framework = "${libdir}/libtensorflow.so ${libdir}/libtensorflow_framework.so*"
+FILES_label-image = "${sbindir}/label_image"
+FILES_label-image-lite = "${sbindir}/label_image.lite"

inherit siteinfo unsupportarch
python __anonymous() {
--
2.21.0


[meta-tensorflow][PATCH 23/25] BUILD.md: update quick start for tensorflow 2

hongxu
 

Refers begginers of upstream [1] [2]
[1] https://www.tensorflow.org/install/pip
[2] https://www.tensorflow.org/tutorials/quickstart/beginner

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
BUILD.md | 97 ++++++++++++++++++++++++++++++++------------------------
1 file changed, 55 insertions(+), 42 deletions(-)

diff --git a/BUILD.md b/BUILD.md
index da5a148..bd0f44a 100644
--- a/BUILD.md
+++ b/BUILD.md
@@ -41,75 +41,88 @@ $ runqemu qemux86-64 core-image-minimal slirp kvm qemuparams="-m 5120"

## 5. Verify the install
```
-root@qemux86-64:~# python3 -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"
-tf.Tensor(-604.65454, shape=(), dtype=float32)
+root@qemux86-64:~# python3 -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+tf.Tensor(-3304.6208, shape=(), dtype=float32)
```

## 6. Run tutorial case
### Refer: https://www.tensorflow.org/tutorials
```
-root@qemux86-64:~# cat >code.py <<ENDOF
+root@qemux86-64:~# cat >code-v2.py <<ENDOF
import tensorflow as tf
mnist = tf.keras.datasets.mnist

-(x_train, y_train),(x_test, y_test) = mnist.load_data()
+(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
- tf.keras.layers.Dense(512, activation=tf.nn.relu),
+ tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
- tf.keras.layers.Dense(10, activation=tf.nn.softmax)
+ tf.keras.layers.Dense(10)
])
+
+predictions = model(x_train[:1]).numpy()
+tf.nn.softmax(predictions).numpy()
+loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
+loss_fn(y_train[:1], predictions).numpy()
+
model.compile(optimizer='adam',
- loss='sparse_categorical_crossentropy',
+ loss=loss_fn,
metrics=['accuracy'])
-
model.fit(x_train, y_train, epochs=5)
-model.evaluate(x_test, y_test)
+model.evaluate(x_test, y_test, verbose=2)
+
+probability_model = tf.keras.Sequential([
+ model,
+ tf.keras.layers.Softmax()
+])
+probability_model(x_test[:5])
+
+
ENDOF

-root@qemux86-64:~# python3 ./code.py
-Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
-11493376/11490434 [==============================] - 7s 1us/step
-Instructions for updating:
-Colocations handled automatically by placer.
-Instructions for updating:
-Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
+root@qemux86-64:~# python3 ./code-v2.py
+2020-12-15 08:16:44.171593: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
+2020-12-15 08:16:44.184464: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3099995000 Hz
Epoch 1/5
-60000/60000 [==============================] - 27s 449us/sample - loss: 0.2211 - acc: 0.9346
+1875/1875 [==============================] - 14s 7ms/step - loss: 0.4833 - accuracy: 0.8595
Epoch 2/5
-60000/60000 [==============================] - 24s 408us/sample - loss: 0.0969 - acc: 0.9702
+1875/1875 [==============================] - 13s 7ms/step - loss: 0.1549 - accuracy: 0.9558
Epoch 3/5
-60000/60000 [==============================] - 26s 439us/sample - loss: 0.0694 - acc: 0.9780
+1875/1875 [==============================] - 13s 7ms/step - loss: 0.1135 - accuracy: 0.9651
Epoch 4/5
-60000/60000 [==============================] - 23s 390us/sample - loss: 0.0540 - acc: 0.9832
+1875/1875 [==============================] - 13s 7ms/step - loss: 0.0889 - accuracy: 0.9729
Epoch 5/5
-60000/60000 [==============================] - 24s 399us/sample - loss: 0.0447 - acc: 0.9851
-10000/10000 [==============================] - 1s 91us/sample - loss: 0.0700 - acc: 0.9782
+1875/1875 [==============================] - 13s 7ms/step - loss: 0.0741 - accuracy: 0.9777
+313/313 - 1s - loss: 0.0757 - accuracy: 0.9757
```
## 7. TensorFlow/TensorFlow Lite C++ Image Recognition Demo
```
root@qemux86-64:~# time label_image
-2019-03-06 06:08:51.076028: I tensorflow/examples/label_image/main.cc:251] military uniform (653): 0.834306
-2019-03-06 06:08:51.078221: I tensorflow/examples/label_image/main.cc:251] mortarboard (668): 0.0218695
-2019-03-06 06:08:51.080054: I tensorflow/examples/label_image/main.cc:251] academic gown (401): 0.010358
-2019-03-06 06:08:51.081943: I tensorflow/examples/label_image/main.cc:251] pickelhaube (716): 0.00800814
-2019-03-06 06:08:51.083830: I tensorflow/examples/label_image/main.cc:251] bulletproof vest (466): 0.00535084
-real 0m 10.50s
-user 0m 3.95s
-sys 0m 6.46s
+2020-12-15 08:18:34.853885: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3099995000 Hz
+2020-12-15 08:18:41.565167: I tensorflow/examples/label_image/main.cc:252] military uniform (653): 0.834306
+2020-12-15 08:18:41.567874: I tensorflow/examples/label_image/main.cc:252] mortarboard (668): 0.0218696
+2020-12-15 08:18:41.568936: I tensorflow/examples/label_image/main.cc:252] academic gown (401): 0.0103581
+2020-12-15 08:18:41.569985: I tensorflow/examples/label_image/main.cc:252] pickelhaube (716): 0.00800819
+2020-12-15 08:18:41.571025: I tensorflow/examples/label_image/main.cc:252] bulletproof vest (466): 0.00535086
+
+real 0m7.178s
+user 0m6.101s
+sys 0m0.893s
+
root@qemux86-64:~# time label_image.lite
-Loaded model /usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite
-resolved reporter
-invoked
-average time: 1064.8 ms
-0.780392: 653 military uniform
-0.105882: 907 Windsor tie
-0.0156863: 458 bow tie
-0.0117647: 466 bulletproof vest
-0.00784314: 835 suit
-real 0m 1.10s
-user 0m 1.07s
-sys 0m 0.02s
+INFO: Loaded model /usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite
+INFO: resolved reporter
+INFO: invoked
+INFO: average time: 213.584 ms
+INFO: 0.780392: 653 military uniform
+INFO: 0.105882: 907 Windsor tie
+INFO: 0.0156863: 458 bow tie
+INFO: 0.0117647: 466 bulletproof vest
+INFO: 0.00784314: 835 suit
+
+real 0m0.233s
+user 0m0.216s
+sys 0m0.012s
```
--
2.21.0


[meta-tensorflow][PATCH 22/25] bazel-native/bazel.bbclass: support bazel build without limitation

hongxu
 

Set BAZEL_JOBS and BAZEL_MEM with empty string to make bazel build without
limitation

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
classes/bazel.bbclass | 9 ++++++++-
recipes-devtools/bazel/bazel-native_3.7.1.bb | 6 +++---
2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/classes/bazel.bbclass b/classes/bazel.bbclass
index 8a3dadf..f5dc597 100644
--- a/classes/bazel.bbclass
+++ b/classes/bazel.bbclass
@@ -77,13 +77,20 @@ def bazel_get_flags(d):
flags += "# From TOOLCHAIN_OPTIONS\n"
flags += "build --linkopt=%s\n" % i

+ if d.getVar("BAZEL_JOBS"):
+ flags += "# From BAZEL_JOBS\n"
+ flags += "build --jobs=%s --local_cpu_resources=%s\n" % (d.getVar("BAZEL_JOBS"), d.getVar("BAZEL_JOBS"))
+
+ if d.getVar("BAZEL_MEM"):
+ flags += "# From BAZEL_MEM\n"
+ flags += "build --local_ram_resources=%s\n" % (d.getVar("BAZEL_MEM"))
+
return flags

bazel_do_configure () {
cat > "${S}/bazelrc" <<-EOF
build --verbose_failures
build --spawn_strategy=standalone --genrule_strategy=standalone
-build --jobs=${BAZEL_JOBS} --local_ram_resources=${BAZEL_MEM} --local_cpu_resources=${BAZEL_JOBS}
test --verbose_failures --verbose_test_summary
test --spawn_strategy=standalone --genrule_strategy=standalone

diff --git a/recipes-devtools/bazel/bazel-native_3.7.1.bb b/recipes-devtools/bazel/bazel-native_3.7.1.bb
index 03cff37..4f2e28b 100644
--- a/recipes-devtools/bazel/bazel-native_3.7.1.bb
+++ b/recipes-devtools/bazel/bazel-native_3.7.1.bb
@@ -30,9 +30,9 @@ inherit bazel-base
EXTRA_BAZEL_ARGS = " \
--host_javabase=@local_jdk//:jdk \
--python_path=python3 \
- --jobs=${BAZEL_JOBS} \
- --local_ram_resources=${BAZEL_MEM} \
- --local_cpu_resources=${BAZEL_JOBS} \
+ ${@oe.utils.conditional("BAZEL_JOBS", "", "", "--jobs=${BAZEL_JOBS}", d )} \
+ ${@oe.utils.conditional("BAZEL_JOBS", "", "", "--local_cpu_resources=${BAZEL_JOBS}", d )} \
+ ${@oe.utils.conditional("BAZEL_MEM", "", "", "--local_ram_resources=${BAZEL_MEM}", d )} \
"

do_compile () {
--
2.21.0


[meta-tensorflow][PATCH 21/25] add classes/bazel-base.bbclass and imporve classes/bazel-base.bbclass

hongxu
 

- Move common config settings to bazel-base.bbclass

- Improve bazel_get_flags to make each flag per line with comments

- Use BAZEL_MEM rather than hardcoded for option --local_ram_resources

- Explicitly remove dir ${BAZEL_DIR} before do_clean

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
classes/bazel-base.bbclass | 11 +++++
classes/bazel.bbclass | 47 ++++++++++++--------
recipes-devtools/bazel/bazel-native_3.7.1.bb | 6 +--
3 files changed, 41 insertions(+), 23 deletions(-)
create mode 100644 classes/bazel-base.bbclass

diff --git a/classes/bazel-base.bbclass b/classes/bazel-base.bbclass
new file mode 100644
index 0000000..a067733
--- /dev/null
+++ b/classes/bazel-base.bbclass
@@ -0,0 +1,11 @@
+export JAVA_HOME="${STAGING_LIBDIR_NATIVE}/jvm/openjdk-8-native"
+
+BAZEL_JOBS ??= "4"
+
+# Memory 4GB
+BAZEL_MEM ??= "4096"
+
+TS_DL_DIR ??= "${DL_DIR}"
+
+CCACHE_DISABLE = "1"
+
diff --git a/classes/bazel.bbclass b/classes/bazel.bbclass
index 3ace2ba..8a3dadf 100644
--- a/classes/bazel.bbclass
+++ b/classes/bazel.bbclass
@@ -3,6 +3,8 @@ DEPENDS += "bazel-native \
"
DEPENDS_append_class-target = " python3"

+inherit bazel-base
+
BAZEL_DIR ?= "${WORKDIR}/bazel"
BAZEL_OUTPUTBASE_DIR ?= "${BAZEL_DIR}/output_base"
export BAZEL_ARGS="--output_user_root=${BAZEL_DIR}/user_root \
@@ -11,8 +13,6 @@ export BAZEL_ARGS="--output_user_root=${BAZEL_DIR}/user_root \
--batch \
"

-export JAVA_HOME="${STAGING_LIBDIR_NATIVE}/jvm/openjdk-8-native"
-
BAZEL ?= "${BAZEL_DIR}/bazel"

do_prepare_recipe_sysroot[postfuncs] += "do_install_bazel"
@@ -26,56 +26,64 @@ do_install_bazel() {
def bazel_get_flags(d):
flags = ""
for i in d.getVar("CC").split()[1:]:
- flags += "--conlyopt=%s --cxxopt=%s --linkopt=%s " % (i, i, i)
+ flags += "# From CC\n"
+ flags += "build --conlyopt=%s --cxxopt=%s --linkopt=%s\n" % (i, i, i)

for i in d.getVar("CFLAGS").split():
if i == "-g":
continue
- flags += "--conlyopt=%s " % i
+ flags += "# From CFLAGS\n"
+ flags += "build --conlyopt=%s\n" % i

for i in d.getVar("BUILD_CFLAGS").split():
- flags += "--host_conlyopt=%s " % i
+ flags += "# From BUILD_CFLAGS\n"
+ flags += "build --host_conlyopt=%s\n" % i

for i in d.getVar("CXXFLAGS").split():
if i == "-g":
continue
- flags += "--cxxopt=%s " % i
+ flags += "# From CXXFLAGS\n"
+ flags += "build --cxxopt=%s\n" % i

for i in d.getVar("BUILD_CXXFLAGS").split():
- flags += "--host_cxxopt=%s " % i
+ flags += "# From BUILD_CXXFLAGS\n"
+ flags += "build --host_cxxopt=%s\n" % i

for i in d.getVar("CPPFLAGS").split():
if i == "-g":
continue
- flags += "--conlyopt=%s --cxxopt=%s " % (i, i)
+ flags += "# From CPPFLAGS\n"
+ flags += "build --conlyopt=%s --cxxopt=%s\n" % (i, i)

for i in d.getVar("BUILD_CPPFLAGS").split():
- flags += "--host_conlyopt=%s --host_cxxopt=%s " % (i, i)
+ flags += "# From BUILD_CPPFLAGS\n"
+ flags += "build --host_conlyopt=%s --host_cxxopt=%s\n" % (i, i)

for i in d.getVar("LDFLAGS").split():
if i == "-Wl,--as-needed":
continue
- flags += "--linkopt=%s " % i
+ flags += "# From LDFLAGS\n"
+ flags += "build --linkopt=%s\n" % i

for i in d.getVar("BUILD_LDFLAGS").split():
if i == "-Wl,--as-needed":
continue
- flags += "--host_linkopt=%s " % i
+ flags += "# From BUILD_LDFLAGS\n"
+ flags += "build --host_linkopt=%s\n" % i

for i in d.getVar("TOOLCHAIN_OPTIONS").split():
if i == "-Wl,--as-needed":
continue
- flags += "--linkopt=%s " % i
+ flags += "# From TOOLCHAIN_OPTIONS\n"
+ flags += "build --linkopt=%s\n" % i

return flags

-TS_DL_DIR ??= "${DL_DIR}"
-BAZEL_JOBS ??= "4"
bazel_do_configure () {
cat > "${S}/bazelrc" <<-EOF
build --verbose_failures
build --spawn_strategy=standalone --genrule_strategy=standalone
-#build --jobs=${BAZEL_JOBS} --local_ram_resources=4096 --local_cpu_resources=${BAZEL_JOBS}
+build --jobs=${BAZEL_JOBS} --local_ram_resources=${BAZEL_MEM} --local_cpu_resources=${BAZEL_JOBS}
test --verbose_failures --verbose_test_summary
test --spawn_strategy=standalone --genrule_strategy=standalone

@@ -98,8 +106,10 @@ EOF

bazel_do_configure_append_class-target () {
cat >> "${S}/bazelrc" <<-EOF
-# FLAGS
-build ${@bazel_get_flags(d)}
+# FLAGS begin
+${@bazel_get_flags(d)}
+# FLAGS end
+
build --linkopt=-Wl,-latomic

EOF
@@ -109,8 +119,6 @@ EOF

EXPORT_FUNCTIONS do_configure

-CCACHE_DISABLE = "1"
-
PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/bazel"

inherit unsupportarch
@@ -125,4 +133,5 @@ clean_bazel() {
${BAZEL} clean
fi
fi
+ rm ${BAZEL_DIR} -rf
}
diff --git a/recipes-devtools/bazel/bazel-native_3.7.1.bb b/recipes-devtools/bazel/bazel-native_3.7.1.bb
index a1575d6..03cff37 100644
--- a/recipes-devtools/bazel/bazel-native_3.7.1.bb
+++ b/recipes-devtools/bazel/bazel-native_3.7.1.bb
@@ -25,19 +25,17 @@ DEPENDS = "coreutils-native \

S="${WORKDIR}"

-TS_DL_DIR ??= "${DL_DIR}"
+inherit bazel-base

-BAZEL_JOBS ??= "4"
EXTRA_BAZEL_ARGS = " \
--host_javabase=@local_jdk//:jdk \
--python_path=python3 \
--jobs=${BAZEL_JOBS} \
- --local_ram_resources=4096 \
+ --local_ram_resources=${BAZEL_MEM} \
--local_cpu_resources=${BAZEL_JOBS} \
"

do_compile () {
- export JAVA_HOME="${STAGING_LIBDIR_NATIVE}/jvm/openjdk-8-native"
TMPDIR="${TOPDIR}/bazel" \
VERBOSE=yes \
EXTRA_BAZEL_ARGS="${EXTRA_BAZEL_ARGS}" \
--
2.21.0


[meta-tensorflow][PATCH 20/25] bazel-native: 3.1.0 -> 3.7.1

hongxu
 

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
...l-native_3.1.0.bb => bazel-native_3.7.1.bb} | 4 ++--
...e-sysroot-dir-to-the-default-Bazel-to.patch | 12 ++++++------
.../bazel/files/0001-python3.patch | 18 +++++++++---------
3 files changed, 17 insertions(+), 17 deletions(-)
rename recipes-devtools/bazel/{bazel-native_3.1.0.bb => bazel-native_3.7.1.bb} (90%)

diff --git a/recipes-devtools/bazel/bazel-native_3.1.0.bb b/recipes-devtools/bazel/bazel-native_3.7.1.bb
similarity index 90%
rename from recipes-devtools/bazel/bazel-native_3.1.0.bb
rename to recipes-devtools/bazel/bazel-native_3.7.1.bb
index 1336da8..a1575d6 100644
--- a/recipes-devtools/bazel/bazel-native_3.1.0.bb
+++ b/recipes-devtools/bazel/bazel-native_3.7.1.bb
@@ -2,8 +2,8 @@ DESCRIPTION = "Bazel build and test tool"
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"

-SRC_URI[md5sum] = "381ca27503c566ce5e489d1ba07d1d25"
-SRC_URI[sha256sum] = "d7f40d0cac95a06cea6cb5b7f7769085257caebc3ee84269dd9298da760d5615"
+SRC_URI[md5sum] = "cc255121586e849c4de2483a8a5814b6"
+SRC_URI[sha256sum] = "c9244e5905df6b0190113e26082c72d58b56b1b0dec66d076f083ce4089b0307"

SRC_URI = "https://github.com/bazelbuild/bazel/releases/download/${PV}/bazel-${PV}-dist.zip \
file://0001-HttpDownloader-save-download-tarball-to-distdir.patch \
diff --git a/recipes-devtools/bazel/files/0001-add-Yocto-native-sysroot-dir-to-the-default-Bazel-to.patch b/recipes-devtools/bazel/files/0001-add-Yocto-native-sysroot-dir-to-the-default-Bazel-to.patch
index b08c65a..7d73eb7 100644
--- a/recipes-devtools/bazel/files/0001-add-Yocto-native-sysroot-dir-to-the-default-Bazel-to.patch
+++ b/recipes-devtools/bazel/files/0001-add-Yocto-native-sysroot-dir-to-the-default-Bazel-to.patch
@@ -1,6 +1,6 @@
-From 707ba08068432262b3d02b29804c00afe7133ff6 Mon Sep 17 00:00:00 2001
+From 34c4dc1a10140addf75d3503d4b9f427303fe212 Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
-Date: Thu, 10 Dec 2020 16:12:51 +0800
+Date: Mon, 14 Dec 2020 16:45:31 +0800
Subject: [PATCH] add Yocto native sysroot dir to the default Bazel toolchain

While using the default Bazel C++ toolchain to build Yocto native tools
@@ -21,12 +21,12 @@ Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
1 file changed, 3 insertions(+)

diff --git a/tools/cpp/unix_cc_configure.bzl b/tools/cpp/unix_cc_configure.bzl
-index 84f5479..e17aa9d 100755
+index d48485b..0d297bf 100755
--- a/tools/cpp/unix_cc_configure.bzl
+++ b/tools/cpp/unix_cc_configure.bzl
-@@ -424,6 +424,9 @@ def configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools):
- ),
- )
+@@ -443,6 +443,9 @@ def configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools):
+ paths["@bazel_tools//tools/cpp:generate_system_module_map.sh"],
+ ))

+ # Customize for Yocto
+ builtin_include_directories.append(get_env_var(repository_ctx,"YOCTO_NATIVE_SYSROOT", "NOT_SET_YOCTO_NATIVE_SYSROOT"))
diff --git a/recipes-devtools/bazel/files/0001-python3.patch b/recipes-devtools/bazel/files/0001-python3.patch
index 93eee2d..c14dec7 100644
--- a/recipes-devtools/bazel/files/0001-python3.patch
+++ b/recipes-devtools/bazel/files/0001-python3.patch
@@ -1,6 +1,6 @@
-From e43263d6163f7ba1622b268e93635bf42493f758 Mon Sep 17 00:00:00 2001
+From aa31b751f9b2e5263e4510c8719f29050b8ce0de Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
-Date: Tue, 8 Dec 2020 11:09:44 +0800
+Date: Mon, 14 Dec 2020 16:43:45 +0800
Subject: [PATCH] set python3 interpreter

Since many distrobution choose python3, and drop python2,
@@ -20,28 +20,28 @@ Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java
-index 2c0ae4d..ae39cc9 100755
+index 9be9ef3..dec19e5 100755
--- a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java
-@@ -237,7 +237,7 @@ public class BazelPythonSemantics implements PythonSemantics {
+@@ -247,7 +247,7 @@ public class BazelPythonSemantics implements PythonSemantics {
PathFragment shExecutable = ShToolchain.getPathOrError(ruleContext);
// TODO(#8685): Remove this special-case handling as part of making the proper shebang a
// property of the Python toolchain configuration.
- String pythonExecutableName = OS.getCurrent() == OS.OPENBSD ? "python3" : "python";
+ String pythonExecutableName = OS.getCurrent() == OS.OPENBSD ? "python3" : "python3";
+ // NOTE: keep the following line intact to support nix builds
+ String pythonShebang = "#!/usr/bin/env " + pythonExecutableName;
ruleContext.registerAction(
- new SpawnAction.Builder()
- .addInput(zipFile)
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt
-index 59c00e8..31d29a4 100755
+index 29f043f..c83891a 100755
--- a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt
+++ b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3

- from __future__ import absolute_import
- from __future__ import division
+ # This script must retain compatibility with a wide variety of Python versions
+ # since it is run for every py_binary target. Currently we guarantee support
--
2.18.2

--
2.21.0


[meta-tensorflow][PATCH 19/25] tensorflow: clean up CROSSTOOL.tpl

hongxu
 

CROSSTOOL files are no longer supported after Bazel v0.26+

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../tensorflow/files/CROSSTOOL.tpl | 230 ------------------
.../tensorflow/tensorflow_2.4.0.bb | 3 -
2 files changed, 233 deletions(-)
delete mode 100644 recipes-framework/tensorflow/files/CROSSTOOL.tpl

diff --git a/recipes-framework/tensorflow/files/CROSSTOOL.tpl b/recipes-framework/tensorflow/files/CROSSTOOL.tpl
deleted file mode 100644
index 753898a..0000000
--- a/recipes-framework/tensorflow/files/CROSSTOOL.tpl
+++ /dev/null
@@ -1,230 +0,0 @@
-major_version: "local"
-minor_version: ""
-default_target_cpu: "same_as_host"
-
-toolchain {
- abi_version: "armeabi"
- abi_libc_version: "armeabi"
- builtin_sysroot: ""
- compiler: "compiler"
- host_system_name: "armeabi"
- needsPic: true
- supports_gold_linker: false
- supports_incremental_linker: false
- supports_fission: false
- supports_interface_shared_objects: false
- supports_normalizing_ar: false
- supports_start_end_lib: false
- target_libc: "armeabi"
- target_cpu: "armeabi"
- target_system_name: "armeabi"
- toolchain_identifier: "yocto-linux-gnueabihf"
-
- tool_path { name: "ar" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-ar" }
- tool_path { name: "compat-ld" path: "/bin/false" }
- tool_path { name: "cpp" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-cpp" }
- tool_path { name: "dwp" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-dwp" }
- tool_path { name: "gcc" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-gcc" }
- tool_path { name: "gcov" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-gcov" }
- tool_path { name: "ld" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-ld" }
-
- tool_path { name: "nm" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-nm" }
- tool_path { name: "objcopy" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-objcopy" }
- tool_path { name: "objdump" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-objdump" }
- tool_path { name: "strip" path: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/bin/%%CT_NAME%%/%%CT_NAME%%-strip" }
-
-
- cxx_builtin_include_directory: "%%YOCTO_COMPILER_PATH%%"
-
- compiler_flag: "--sysroot=%%YOCTO_COMPILER_PATH%%/recipe-sysroot"
-
- # The path below must match the one used in
- # tensorflow/tools/ci_build/pi/build_raspberry_pi.sh.
- cxx_builtin_include_directory: "/tmp/openblas_install/include/"
- cxx_flag: "-std=c++11"
- # The cxx_builtin_include_directory directives don't seem to be adding these, so
- # explicitly set them as flags. There's a query to the Bazel team outstanding about
- # why this is necessary.
- linker_flag: "-lstdc++"
-
- unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
- unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
- unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
- unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
-
- unfiltered_cxx_flag: "-no-canonical-prefixes"
- unfiltered_cxx_flag: "-fno-canonical-system-headers"
-
- # Include target pyconfig.h
- compiler_flag: "-D_PYTHON_INCLUDE_TARGET"
-
- compiler_flag: "-U_FORTIFY_SOURCE"
- compiler_flag: "-D_FORTIFY_SOURCE=1"
- compiler_flag: "-fstack-protector"
- compiler_flag: "-DRASPBERRY_PI" # To differentiate from mobile builds.
- linker_flag: "-Wl,-z,relro,-z,now"
-
- linker_flag: "-no-canonical-prefixes"
- linker_flag: "-pass-exit-codes"
-
- linker_flag: "-Wl,--build-id=md5"
-
- compilation_mode_flags {
- mode: DBG
- # Enable debug symbols.
- compiler_flag: "-g"
- }
- compilation_mode_flags {
- mode: OPT
-
- # No debug symbols.
- # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
- # even generally? However, that can't happen here, as it requires special
- # handling in Bazel.
- compiler_flag: "-g0"
-
- # Conservative choice for -O
- # -O3 can increase binary size and even slow down the resulting binaries.
- # Profile first and / or use FDO if you need better performance than this.
- compiler_flag: "-O2"
-
- # Disable assertions
- compiler_flag: "-DNDEBUG"
-
- # Removal of unused code and data at link time (can this increase binary size in some cases?).
- compiler_flag: "-ffunction-sections"
- compiler_flag: "-fdata-sections"
- linker_flag: "-Wl,--gc-sections"
- }
- linking_mode_flags { mode: DYNAMIC }
-
-}
-
-toolchain {
- abi_version: "local"
- abi_libc_version: "local"
- builtin_sysroot: ""
- compiler: "compiler"
- host_system_name: "local"
- needsPic: true
- supports_gold_linker: false
- supports_incremental_linker: false
- supports_fission: false
- supports_interface_shared_objects: false
- supports_normalizing_ar: false
- supports_start_end_lib: false
- target_libc: "local"
- target_cpu: "local"
- target_system_name: "local"
- toolchain_identifier: "local_linux"
-
- tool_path { name: "ar" path: "/usr/bin/ar" }
- tool_path { name: "compat-ld" path: "/usr/bin/ld" }
- tool_path { name: "cpp" path: "/usr/bin/cpp" }
- tool_path { name: "dwp" path: "/usr/bin/dwp" }
- tool_path { name: "gcc" path: "/usr/bin/gcc" }
- cxx_flag: "-std=c++0x"
- linker_flag: "-lstdc++"
- linker_flag: "-B/usr/bin/"
-
- # TODO(bazel-team): In theory, the path here ought to exactly match the path
- # used by gcc. That works because bazel currently doesn't track files at
- # absolute locations and has no remote execution, yet. However, this will need
- # to be fixed, maybe with auto-detection?
- cxx_builtin_include_directory: "/usr/lib/gcc/"
- cxx_builtin_include_directory: "/usr/lib64/gcc/"
- cxx_builtin_include_directory: "/usr/local/include"
- cxx_builtin_include_directory: "/usr/include"
- cxx_builtin_include_directory: "%%YOCTO_COMPILER_PATH%%/recipe-sysroot-native/usr/include"
-
- tool_path { name: "gcov" path: "/usr/bin/gcov" }
-
- # C(++) compiles invoke the compiler (as that is the one knowing where
- # to find libraries), but we provide LD so other rules can invoke the linker.
- tool_path { name: "ld" path: "/usr/bin/ld" }
-
- tool_path { name: "nm" path: "/usr/bin/nm" }
- tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
- objcopy_embed_flag: "-I"
- objcopy_embed_flag: "binary"
- tool_path { name: "objdump" path: "/usr/bin/objdump" }
- tool_path { name: "strip" path: "/usr/bin/strip" }
-
- # Anticipated future default.
- unfiltered_cxx_flag: "-no-canonical-prefixes"
- unfiltered_cxx_flag: "-fno-canonical-system-headers"
-
- # Make C++ compilation deterministic. Use linkstamping instead of these
- # compiler symbols.
- unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
- unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
- unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
- unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
-
- # Security hardening on by default.
- # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
- # We need to undef it before redefining it as some distributions now have
- # it enabled by default.
- compiler_flag: "-U_FORTIFY_SOURCE"
- compiler_flag: "-D_FORTIFY_SOURCE=1"
- compiler_flag: "-fstack-protector"
- linker_flag: "-Wl,-z,relro,-z,now"
-
- # Include native pyconfig.h
- compiler_flag: "-D_PYTHON_INCLUDE_NATIVE"
-
- # Enable coloring even if there's no attached terminal. Bazel removes the
- # escape sequences if --nocolor is specified. This isn't supported by gcc
- # on Ubuntu 14.04.
- # compiler_flag: "-fcolor-diagnostics"
-
- # All warnings are enabled. Maybe enable -Werror as well?
- compiler_flag: "-Wall"
- # Enable a few more warnings that aren't part of -Wall.
- compiler_flag: "-Wunused-but-set-parameter"
- # But disable some that are problematic.
- compiler_flag: "-Wno-free-nonheap-object" # has false positives
-
- # Keep stack frames for debugging, even in opt mode.
- compiler_flag: "-fno-omit-frame-pointer"
-
- # Anticipated future default.
- linker_flag: "-no-canonical-prefixes"
- # Have gcc return the exit code from ld.
- linker_flag: "-pass-exit-codes"
- # Stamp the binary with a unique identifier.
- linker_flag: "-Wl,--build-id=md5"
- linker_flag: "-Wl,--hash-style=gnu"
- # Gold linker only? Can we enable this by default?
- # linker_flag: "-Wl,--warn-execstack"
- # linker_flag: "-Wl,--detect-odr-violations"
-
- compilation_mode_flags {
- mode: DBG
- # Enable debug symbols.
- compiler_flag: "-g"
- }
- compilation_mode_flags {
- mode: OPT
-
- # No debug symbols.
- # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
- # even generally? However, that can't happen here, as it requires special
- # handling in Bazel.
- compiler_flag: "-g0"
-
- # Conservative choice for -O
- # -O3 can increase binary size and even slow down the resulting binaries.
- # Profile first and / or use FDO if you need better performance than this.
- compiler_flag: "-O2"
-
- # Disable assertions
- compiler_flag: "-DNDEBUG"
-
- # Removal of unused code and data at link time (can this increase binary size in some cases?).
- compiler_flag: "-ffunction-sections"
- compiler_flag: "-fdata-sections"
- linker_flag: "-Wl,--gc-sections"
- }
- linking_mode_flags { mode: DYNAMIC }
-}
diff --git a/recipes-framework/tensorflow/tensorflow_2.4.0.bb b/recipes-framework/tensorflow/tensorflow_2.4.0.bb
index af66397..49c868e 100644
--- a/recipes-framework/tensorflow/tensorflow_2.4.0.bb
+++ b/recipes-framework/tensorflow/tensorflow_2.4.0.bb
@@ -9,7 +9,6 @@ SRC_URI += " \
file://0001-support-32-bit-x64-and-arm-for-yocto.patch \
file://BUILD.in \
file://BUILD.yocto_compiler \
- file://CROSSTOOL.tpl \
file://cc_config.bzl.tpl \
file://yocto_compiler_configure.bzl \
"
@@ -71,7 +70,6 @@ ENDOF
mkdir -p ${S}/third_party/toolchains/yocto/
sed "s#%%CPU%%#${BAZEL_TARGET_CPU}#g" ${WORKDIR}/BUILD.in > ${S}/third_party/toolchains/yocto/BUILD
chmod 644 ${S}/third_party/toolchains/yocto/BUILD
- install -m 644 ${WORKDIR}/CROSSTOOL.tpl ${S}/third_party/toolchains/yocto/
install -m 644 ${WORKDIR}/cc_config.bzl.tpl ${S}/third_party/toolchains/yocto/
install -m 644 ${WORKDIR}/yocto_compiler_configure.bzl ${S}/third_party/toolchains/yocto/
install -m 644 ${WORKDIR}/BUILD.yocto_compiler ${S}
@@ -82,7 +80,6 @@ ENDOF
SED_COMMAND="${SED_COMMAND}; s#%%YOCTO_COMPILER_PATH%%#${BAZEL_OUTPUTBASE_DIR}/external/yocto_compiler#g"

sed -i "${SED_COMMAND}" ${S}/BUILD.yocto_compiler \
- ${S}/third_party/toolchains/yocto/CROSSTOOL.tpl \
${S}/WORKSPACE

${TF_CONFIG} \
--
2.21.0


[meta-tensorflow][PATCH 18/25] tensorboard: 1.12.2 -> 2.4

hongxu
 

Rebase patch to 2.4
- 0001-customize-for-Yocto.patch

Fix projector cannot find module 'd3'

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../0001-customize-for-Yocto.patch | 199 +++++++++++-------
...-projector-fix-cannot-find-module-d3.patch | 34 +++
...nsorboard_1.12.2.bb => tensorboard_2.4.bb} | 53 ++++-
3 files changed, 195 insertions(+), 91 deletions(-)
create mode 100644 recipes-framework/tensorflow/tensorboard/0001-projector-fix-cannot-find-module-d3.patch
rename recipes-framework/tensorflow/{tensorboard_1.12.2.bb => tensorboard_2.4.bb} (47%)

diff --git a/recipes-framework/tensorflow/tensorboard/0001-customize-for-Yocto.patch b/recipes-framework/tensorflow/tensorboard/0001-customize-for-Yocto.patch
index 07d8257..4055e0b 100644
--- a/recipes-framework/tensorflow/tensorboard/0001-customize-for-Yocto.patch
+++ b/recipes-framework/tensorflow/tensorboard/0001-customize-for-Yocto.patch
@@ -1,6 +1,6 @@
-From 03b1ddf5058485ce1c94e6a8dc0762ad3430b6a2 Mon Sep 17 00:00:00 2001
+From 30c3d17327b7b66e082dbb443c9e528bf7aa6ac4 Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
-Date: Thu, 25 Apr 2019 21:13:32 +0800
+Date: Mon, 14 Dec 2020 15:14:25 +0800
Subject: [PATCH] customize for Yocto

- Remove virtualenv/pip/bdist_wheel calling which Yocto does not support
@@ -11,13 +11,13 @@ Upstream-Status: Inappropriate [oe specific]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
- WORKSPACE | 6 ++++++
- tensorboard/pip_package/build_pip_package.sh | 31 +++-------------------------
- third_party/workspace.bzl | 8 +++++++
- 3 files changed, 17 insertions(+), 28 deletions(-)
+ WORKSPACE | 6 +++
+ tensorboard/pip_package/build_pip_package.sh | 57 ++------------------
+ third_party/workspace.bzl | 8 +++
+ 3 files changed, 17 insertions(+), 54 deletions(-)

diff --git a/WORKSPACE b/WORKSPACE
-index 8ab70cc..0c18f6f 100644
+index 86f5feaa..8056cfff 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -1,5 +1,11 @@
@@ -31,102 +31,139 @@ index 8ab70cc..0c18f6f 100644
+
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

- # Needed as a transitive dependency of rules_webtesting below.
+ http_archive(
diff --git a/tensorboard/pip_package/build_pip_package.sh b/tensorboard/pip_package/build_pip_package.sh
-index 754fa83..969709d 100755
+index ef8df010..085ea608 100755
--- a/tensorboard/pip_package/build_pip_package.sh
+++ b/tensorboard/pip_package/build_pip_package.sh
-@@ -23,7 +23,7 @@ else
- sedi="sed -i"
- fi
-
--run_smoke_test=1
-+run_smoke_test=0
- while [ "$#" -gt 0 ]; do
- case "$1" in
- "--no-smoke")
-@@ -71,31 +71,20 @@ from tensorboard.plugins.beholder import Beholder, BeholderHook
+@@ -12,7 +12,7 @@
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+-
++set -x
+ usage() {
+ cat <<EOF
+ usage: build_pip_package OUTPUT_DIR
+@@ -28,12 +28,6 @@ EOF
}

- set -x
--command -v curl >/dev/null
--command -v perl >/dev/null
--command -v python2 >/dev/null
--command -v python3 >/dev/null
--command -v virtualenv >/dev/null
- [ -d "${RUNFILES}" ]
+ main() {
+- if [ $# -ne 1 ]; then
+- usage 2>&1
+- return 1
+- fi
+- output="$1"
+-
+ if [ -z "${RUNFILES+set}" ]; then
+ RUNFILES="$(CDPATH="" cd -- "$0.runfiles" && pwd)"
+ fi
+@@ -41,16 +35,12 @@ main() {
+ if [ "$(uname)" = "Darwin" ]; then
+ workdir="$(mktemp -d -t tensorboard-pip)"
+ else
+- workdir="$(mktemp -d -p /tmp -t tensorboard-pip.XXXXXXXXXX)"
++ workdir="${DESTDIR}"
++ rm -rf ${workdir} && mkdir -p ${workdir}
+ fi
+ original_wd="${PWD}"
+ cd "${workdir}" || return 2

--dest=/tmp/tensorboard
-+dest=${DESTDIR}
- if [ ! -e $dest ]; then
-- mkdir $dest
-+ mkdir -p $dest
- else
- dest="$(mktemp -d -p /tmp -t tensorboard-pip.XXXXXXXXXX)"
- fi
- cd "${dest}"
+- cleanup() {
+- rm -r "${workdir}"
+- }
+- trap cleanup EXIT
+-
+ log_file="${workdir}/log"
+ build >"${log_file}" 2>&1
+ exit_code=$?
+@@ -68,20 +58,12 @@ build() (
+ sedi="sed -i"
+ fi

- cp -LR "${RUNFILES}/org_tensorflow_tensorboard/tensorboard" .
--mv -f "tensorboard/pip_package/LICENSE" .
--mv -f "tensorboard/pip_package/MANIFEST.in" .
--mv -f "tensorboard/pip_package/README.rst" .
--mv -f "tensorboard/pip_package/setup.cfg" .
--mv -f "tensorboard/pip_package/setup.py" .
- rm -rf tensorboard/pip_package
+- command -v virtualenv >/dev/null
+ [ -d "${RUNFILES}" ]

- rm -f tensorboard/tensorboard # bazel py_binary sh wrapper
--chmod -x LICENSE # bazel symlinks confuse cp
- find . -name __init__.py | xargs chmod -x # which goes for all genfiles
+ cp -LR "${RUNFILES}/org_tensorflow_tensorboard/tensorboard" .
+- mv -f "tensorboard/pip_package/LICENSE" .
+- mv -f "tensorboard/pip_package/MANIFEST.in" .
+- mv -f "tensorboard/pip_package/README.rst" .
+- mv -f "tensorboard/pip_package/requirements.txt" .
+- mv -f "tensorboard/pip_package/setup.cfg" .
+- mv -f "tensorboard/pip_package/setup.py" .
+ rm -rf "tensorboard/pip_package"

- mkdir -p tensorboard/_vendor
-@@ -117,21 +106,7 @@ find tensorboard -name \*.py |
- s/from tensorflow_serving/from tensorboard._vendor.tensorflow_serving/
- '
+ rm -f tensorboard/tensorboard # bazel py_binary sh wrapper
+- chmod -x LICENSE # bazel symlinks confuse cp
+ find . -name __init__.py -exec chmod -x {} + # which goes for all genfiles

--virtualenv venv
--export VIRTUAL_ENV=venv
--export PATH="$PWD/venv/bin:${PATH}"
--unset PYTHON_HOME
+ mkdir -p tensorboard/_vendor
+@@ -100,39 +82,6 @@ build() (
+ s/^import webencodings$/from tensorboard._vendor import webencodings/
+ s/^from webencodings/from tensorboard._vendor.webencodings/
+ ' {} +
-
--# Require wheel for bdist_wheel command, and setuptools 36.2.0+ so that
--# env markers are handled (https://github.com/pypa/setuptools/pull/1081)
--pip install -qU wheel 'setuptools>=36.2.0'
+- virtualenv -q -p python3 venv
+- export VIRTUAL_ENV=venv
+- export PATH="${PWD}/venv/bin:${PATH}"
+- unset PYTHON_HOME
-
--python setup.py bdist_wheel --python-tag py2 >/dev/null
--python setup.py bdist_wheel --python-tag py3 >/dev/null
+- # Require wheel for bdist_wheel command, and setuptools 36.2.0+ so that
+- # env markers are handled (https://github.com/pypa/setuptools/pull/1081)
+- export PYTHONWARNINGS=ignore:DEPRECATION # suppress Python 2.7 deprecation spam
+- pip install -qU wheel 'setuptools>=36.2.0'
-
- if [ "$run_smoke_test" = 1 ]; then
- smoke 2
- smoke 3
- fi
+- # Overrides file timestamps in the zip archive to make the build
+- # reproducible. (Date is mostly arbitrary, but must be past 1980 to be
+- # representable in a zip archive.)
+- export SOURCE_DATE_EPOCH=1577836800 # 2020-01-01T00:00:00Z
-
--ls -hal "$PWD/dist"
+- python setup.py bdist_wheel --python-tag py3 >/dev/null
+-
+- cd "${original_wd}" # Bazel gives "${output}" as a relative path >_>
+- case "${output}" in
+- *.tar.gz)
+- mkdir -p "$(dirname "${output}")"
+- "${RUNFILES}/org_tensorflow_tensorboard/tensorboard/pip_package/deterministic_tar_gz" \
+- "${output}" "${workdir}"/dist/*.whl
+- ;;
+- *)
+- if ! [ -d "${output}" ]; then
+- printf >&2 'fatal: no such output directory: %s\n' "${output}"
+- return 1
+- fi
+- mv "${workdir}"/dist/*.whl "${output}"
+- ;;
+- esac
+ )
+
+ main "$@"
diff --git a/third_party/workspace.bzl b/third_party/workspace.bzl
-index 083c441..24786f8 100644
+index e6a5183f..9260a8ab 100644
--- a/third_party/workspace.bzl
+++ b/third_party/workspace.bzl
-@@ -24,6 +24,7 @@ load("//third_party:polymer.bzl", "tensorboard_polymer_workspace")
- load("//third_party:python.bzl", "tensorboard_python_workspace")
+@@ -25,6 +25,7 @@ load("//third_party:python.bzl", "tensorboard_python_workspace")
load("//third_party:js.bzl", "tensorboard_js_workspace")
+ load("//third_party:rust.bzl", "tensorboard_rust_workspace")
load("//third_party:typings.bzl", "tensorboard_typings_workspace")
+load("//third_party/toolchains/yocto:yocto_compiler_configure.bzl", "yocto_compiler_configure")

- def tensorboard_workspace():
- tensorboard_fonts_workspace()
-@@ -32,6 +33,13 @@ def tensorboard_workspace():
- tensorboard_typings_workspace()
- tensorboard_js_workspace()
+ def tensorboard_workspace(name = ""):
+ """Add repositories needed to build TensorBoard.
+@@ -39,6 +40,13 @@ def tensorboard_workspace(name = ""):
+ tensorboard_js_workspace()
+ tensorboard_rust_workspace()

-+ # Point //external/local_config_yocto_compiler to //external/yocto_compiler
-+ yocto_compiler_configure(
-+ name = "local_config_yocto_compiler",
-+ build_file = str(Label("//third_party/toolchains/yocto:BUILD")),
-+ remote_config_repo = "../yocto_compiler",
-+ )
++ # Point //external/local_config_yocto_compiler to //external/yocto_compiler
++ yocto_compiler_configure(
++ name = "local_config_yocto_compiler",
++ build_file = str(Label("//third_party/toolchains/yocto:BUILD")),
++ remote_config_repo = "../yocto_compiler",
++ )
+
- http_archive(
- name = "com_google_protobuf_js",
- strip_prefix = "protobuf-3.6.0/js",
+ # Set up TypeScript toolchain.
+ ts_setup_workspace()
+
--
-2.7.4
+2.18.2

diff --git a/recipes-framework/tensorflow/tensorboard/0001-projector-fix-cannot-find-module-d3.patch b/recipes-framework/tensorflow/tensorboard/0001-projector-fix-cannot-find-module-d3.patch
new file mode 100644
index 0000000..1cab5dd
--- /dev/null
+++ b/recipes-framework/tensorflow/tensorboard/0001-projector-fix-cannot-find-module-d3.patch
@@ -0,0 +1,34 @@
+From 60f90ec3cc3e417c4b8fe2ea25ddf6e5ff99bde0 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Mon, 14 Dec 2020 14:51:36 +0800
+Subject: [PATCH] projector: fix cannot find module 'd3'
+
+[snip]
+$ bazel run tensorboard/plugins/projector/vz_projector:standalone
+tensorboard/plugins/projector/vz_projector/projectorScatterPlotAdapter.ts:16:21 - error TS2307: Cannot find module 'd3'.
+
+16 import * as d3 from 'd3';
+[snip]
+
+Upstream-Status: Submitted [https://github.com/tensorflow/tensorboard/pull/4465]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ tensorboard/plugins/projector/vz_projector/BUILD | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/tensorboard/plugins/projector/vz_projector/BUILD b/tensorboard/plugins/projector/vz_projector/BUILD
+index f173cd78..1529440d 100644
+--- a/tensorboard/plugins/projector/vz_projector/BUILD
++++ b/tensorboard/plugins/projector/vz_projector/BUILD
+@@ -63,6 +63,7 @@ tf_ts_library(
+ "//tensorboard/webapp/third_party:tfjs",
+ "@npm//@polymer/decorators",
+ "@npm//@polymer/polymer",
++ "@npm//@types/d3",
+ "@npm//d3",
+ "@npm//numeric",
+ "@npm//three",
+--
+2.18.2
+
diff --git a/recipes-framework/tensorflow/tensorboard_1.12.2.bb b/recipes-framework/tensorflow/tensorboard_2.4.bb
similarity index 47%
rename from recipes-framework/tensorflow/tensorboard_1.12.2.bb
rename to recipes-framework/tensorflow/tensorboard_2.4.bb
index a27bdea..aa97890 100644
--- a/recipes-framework/tensorflow/tensorboard_1.12.2.bb
+++ b/recipes-framework/tensorflow/tensorboard_2.4.bb
@@ -3,17 +3,22 @@ your TensorFlow runs and graphs."
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=e74df23890b9521cc481e3348863e45d"

-SRC_URI = "git://github.com/tensorflow/tensorboard.git; \
+SRC_URI = "git://github.com/tensorflow/tensorboard.git;branch=2.4; \
file://0001-customize-for-Yocto.patch \
- file://BUILD \
+ file://0001-projector-fix-cannot-find-module-d3.patch \
+ file://BUILD.in \
file://BUILD.yocto_compiler \
- file://CROSSTOOL.tpl \
+ file://cc_config.bzl.tpl \
file://yocto_compiler_configure.bzl \
"
-SRCREV = "7194c7486a0c4d107322ffad102c1ca0fcc0fc24"
+SRCREV = "4e2a918a0559514a633c3a29ac6238fed4b72ed5"
S = "${WORKDIR}/git"

-DEPENDS = "util-linux-native"
+DEPENDS = " \
+ util-linux-native \
+ python3-numpy-native \
+ python3-absl-native \
+"

RDEPENDS_${PN} += "python3 \
python3-core \
@@ -23,13 +28,40 @@ RDEPENDS_${PN} += "python3 \
python3-werkzeug \
python3-six \
python3-markdown \
+ python3-absl \
+ python3-google-auth \
+ python3-google-auth-oauthlib \
+ python3-requests \
"
inherit python3native bazel

+export PYTHON_BIN_PATH="${PYTHON}"
+export PYTHON_LIB_PATH="${STAGING_LIBDIR_NATIVE}/${PYTHON_DIR}/site-packages"
+export CROSSTOOL_PYTHON_INCLUDE_PATH="${STAGING_INCDIR}/python${PYTHON_BASEVERSION}${PYTHON_ABI}"
+
do_configure_append () {
+ if [ ! -e ${CROSSTOOL_PYTHON_INCLUDE_PATH}/pyconfig-target.h ];then
+ mv ${CROSSTOOL_PYTHON_INCLUDE_PATH}/pyconfig.h ${CROSSTOOL_PYTHON_INCLUDE_PATH}/pyconfig-target.h
+ fi
+
+ install -m 644 ${STAGING_INCDIR_NATIVE}/python${PYTHON_BASEVERSION}${PYTHON_ABI}/pyconfig.h \
+ ${CROSSTOOL_PYTHON_INCLUDE_PATH}/pyconfig-native.h
+
+ cat > ${CROSSTOOL_PYTHON_INCLUDE_PATH}/pyconfig.h <<ENDOF
+#if defined (_PYTHON_INCLUDE_TARGET)
+#include "pyconfig-target.h"
+#elif defined (_PYTHON_INCLUDE_NATIVE)
+#include "pyconfig-native.h"
+#else
+#error "_PYTHON_INCLUDE_TARGET or _PYTHON_INCLUDE_NATIVE is not defined"
+#endif // End of #if defined (_PYTHON_INCLUDE_TARGET)
+
+ENDOF
+
mkdir -p ${S}/third_party/toolchains/yocto/
- install -m 644 ${WORKDIR}/BUILD ${S}/third_party/toolchains/yocto/
- install -m 644 ${WORKDIR}/CROSSTOOL.tpl ${S}/third_party/toolchains/yocto/
+ sed "s#%%CPU%%#${BAZEL_TARGET_CPU}#g" ${WORKDIR}/BUILD.in > ${S}/third_party/toolchains/yocto/BUILD
+ chmod 644 ${S}/third_party/toolchains/yocto/BUILD
+ install -m 644 ${WORKDIR}/cc_config.bzl.tpl ${S}/third_party/toolchains/yocto/
install -m 644 ${WORKDIR}/yocto_compiler_configure.bzl ${S}/third_party/toolchains/yocto/
install -m 644 ${WORKDIR}/BUILD.yocto_compiler ${S}

@@ -39,25 +71,26 @@ do_configure_append () {
SED_COMMAND="${SED_COMMAND}; s#%%YOCTO_COMPILER_PATH%%#${BAZEL_OUTPUTBASE_DIR}/external/yocto_compiler#g"

sed -i "${SED_COMMAND}" ${S}/BUILD.yocto_compiler \
- ${S}/third_party/toolchains/yocto/CROSSTOOL.tpl \
${S}/WORKSPACE
}

do_compile () {
unset CC
+ export CT_NAME=$(echo ${HOST_PREFIX} | rev | cut -c 2- | rev)
DESTDIR=${WORKDIR}/python-tensorboard \
${BAZEL} run \
- --cpu=armeabi \
+ --cpu=${BAZEL_TARGET_CPU} \
--subcommands --explain=${T}/explain.log \
--verbose_explanations --verbose_failures \
--crosstool_top=@local_config_yocto_compiler//:toolchain \
+ --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
--verbose_failures \
//tensorboard/pip_package:build_pip_package
}

do_install () {
install -d ${D}${PYTHON_SITEPACKAGES_DIR}
- cp -rf ${WORKDIR}/python-tensorboard/* ${D}${PYTHON_SITEPACKAGES_DIR}
+ cp -rf ${WORKDIR}/python-tensorboard/tensorboard ${D}${PYTHON_SITEPACKAGES_DIR}
}

FILES_${PN} += "${libdir}/*"
--
2.21.0


[meta-tensorflow][PATCH 17/25] python3-pyasn1-modules: add version 0.2.8

hongxu
 

It is required by tensorboard

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../python/python3-pyasn1-modules_0.2.8.bb | 22 +++++++++++++++++++
1 file changed, 22 insertions(+)
create mode 100644 recipes-devtools/python/python3-pyasn1-modules_0.2.8.bb

diff --git a/recipes-devtools/python/python3-pyasn1-modules_0.2.8.bb b/recipes-devtools/python/python3-pyasn1-modules_0.2.8.bb
new file mode 100644
index 0000000..fb11941
--- /dev/null
+++ b/recipes-devtools/python/python3-pyasn1-modules_0.2.8.bb
@@ -0,0 +1,22 @@
+SUMMARY = "ASN.1 modules for Python"
+HOMEPAGE = "https://github.com/etingof/pyasn1-modules"
+DESCRIPTION = "The pyasn1-modules package contains a collection of \
+ASN.1 data structures expressed as Python classes based on \
+pyasn1 data model."
+SECTION = "devel/python"
+
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=a14482d15c2249de3b6f0e8a47e021fd"
+
+inherit pypi
+
+SRC_URI[md5sum] = ""
+SRC_URI[sha256sum] = ""
+
+RDEPENDS_${PN} += " \
+ python3-pyasn1 \
+"
+
+BBCLASSEXTEND = "native"
+
+inherit setuptools3
--
2.21.0


[meta-tensorflow][PATCH 16/25] python3-pyasn1: add 0.4.8

hongxu
 

It is required by tensorboard

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
recipes-devtools/python/python3-pyasn1_0.4.8.bb | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
create mode 100644 recipes-devtools/python/python3-pyasn1_0.4.8.bb

diff --git a/recipes-devtools/python/python3-pyasn1_0.4.8.bb b/recipes-devtools/python/python3-pyasn1_0.4.8.bb
new file mode 100644
index 0000000..7f8b718
--- /dev/null
+++ b/recipes-devtools/python/python3-pyasn1_0.4.8.bb
@@ -0,0 +1,17 @@
+SUMMARY = "ASN.1 library for Python"
+HOMEPAGE = "https://github.com/etingof/pyasn1"
+DESCRIPTION = "This is a free and open source implementation of ASN.1 types \
+and codecs as a Python package"
+SECTION = "devel/python"
+
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://LICENSE.rst;md5=a14482d15c2249de3b6f0e8a47e021fd"
+
+inherit pypi
+
+SRC_URI[md5sum] = "dffae4ff9f997a83324b3f33fe62be54"
+SRC_URI[sha256sum] = "aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba"
+
+BBCLASSEXTEND = "native"
+
+inherit setuptools3
--
2.21.0


[meta-tensorflow][PATCH 15/25] python3-google-auth-oauthlib: add version 0.4.2

hongxu
 

It is required by tensorboard

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../python3-google-auth-oauthlib_0.4.2.bb | 20 +++++++++++++++++++
1 file changed, 20 insertions(+)
create mode 100644 recipes-devtools/python/python3-google-auth-oauthlib_0.4.2.bb

diff --git a/recipes-devtools/python/python3-google-auth-oauthlib_0.4.2.bb b/recipes-devtools/python/python3-google-auth-oauthlib_0.4.2.bb
new file mode 100644
index 0000000..09e3900
--- /dev/null
+++ b/recipes-devtools/python/python3-google-auth-oauthlib_0.4.2.bb
@@ -0,0 +1,20 @@
+SUMMARY = "The oauthlib integration for Google Auth"
+HOMEPAGE = "https://github.com/googleapis/google-auth-library-python-oauthlib"
+DESCRIPTION = "This library provides oauthlib integration with google-auth."
+SECTION = "devel/python"
+
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=7b7e367c7e1664e6565ee7d16cccc58a"
+
+inherit pypi
+
+SRC_URI[md5sum] = "5d3cb6250dd05fff0b7bc0912a3e8f0a"
+SRC_URI[sha256sum] = "65b65bc39ad8cab15039b35e5898455d3d66296d0584d96fe0e79d67d04c51d9"
+
+BBCLASSEXTEND = "native"
+
+RDEPENDS_${PN} += " \
+ python3-requests-oauthlib \
+ python3-oauthlib \
+"
+inherit setuptools3
--
2.21.0


[meta-tensorflow][PATCH 14/25] python3-google-auth: 1.6.3 -> 1.24.0

hongxu
 

Move it from meta-demo to meta-tensorflow which is required by
tensorboard

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../python/python-google-auth.inc | 10 ++++++++--
.../python/python3-google-auth_1.24.0.bb | 0
2 files changed, 8 insertions(+), 2 deletions(-)
rename {meta-demo/recipes-devtools => recipes-devtools}/python/python-google-auth.inc (63%)
rename meta-demo/recipes-devtools/python/python3-google-auth_1.6.3.bb => recipes-devtools/python/python3-google-auth_1.24.0.bb (100%)

diff --git a/meta-demo/recipes-devtools/python/python-google-auth.inc b/recipes-devtools/python/python-google-auth.inc
similarity index 63%
rename from meta-demo/recipes-devtools/python/python-google-auth.inc
rename to recipes-devtools/python/python-google-auth.inc
index 61e37c0..bfee039 100644
--- a/meta-demo/recipes-devtools/python/python-google-auth.inc
+++ b/recipes-devtools/python/python-google-auth.inc
@@ -9,7 +9,13 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=86d3f3a95c324c9479bd8986968f4327"

inherit pypi

-SRC_URI[md5sum] = "0440718247cc4531422e2a8f4f012e8b"
-SRC_URI[sha256sum] = "0f7c6a64927d34c1a474da92cfc59e552a5d3b940d3266606c6a28b72888b9e4"
+SRC_URI[md5sum] = "d1f8a9284283b8db69e84c41b36db501"
+SRC_URI[sha256sum] = "0b0e026b412a0ad096e753907559e4bdb180d9ba9f68dd9036164db4fdc4ad2e"
+
+RDEPENDS_${PN} += " \
+ python3-cachetools \
+ python3-rsa \
+ python3-pyasn1-modules \
+"

BBCLASSEXTEND = "native"
diff --git a/meta-demo/recipes-devtools/python/python3-google-auth_1.6.3.bb b/recipes-devtools/python/python3-google-auth_1.24.0.bb
similarity index 100%
rename from meta-demo/recipes-devtools/python/python3-google-auth_1.6.3.bb
rename to recipes-devtools/python/python3-google-auth_1.24.0.bb
--
2.21.0


[meta-tensorflow][PATCH 13/25] classes/bazel.bbclass: clean up bazel files before do_clean

hongxu
 

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
classes/bazel.bbclass | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/classes/bazel.bbclass b/classes/bazel.bbclass
index 5a76244..3ace2ba 100644
--- a/classes/bazel.bbclass
+++ b/classes/bazel.bbclass
@@ -75,7 +75,7 @@ bazel_do_configure () {
cat > "${S}/bazelrc" <<-EOF
build --verbose_failures
build --spawn_strategy=standalone --genrule_strategy=standalone
-build --jobs=${BAZEL_JOBS} --local_ram_resources=4096 --local_cpu_resources=${BAZEL_JOBS}
+#build --jobs=${BAZEL_JOBS} --local_ram_resources=4096 --local_cpu_resources=${BAZEL_JOBS}
test --verbose_failures --verbose_test_summary
test --spawn_strategy=standalone --genrule_strategy=standalone

@@ -116,3 +116,13 @@ PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/bazel"
inherit unsupportarch

export YOCTO_NATIVE_SYSROOT="${BAZEL_OUTPUTBASE_DIR}/external/yocto_compiler/recipe-sysroot-native"
+
+do_clean[prefuncs] += "clean_bazel"
+clean_bazel() {
+ if [ -d ${S} ]; then
+ cd ${S}
+ if [ -e ${BAZEL} ] && [ -e ${S}/bazelrc ]; then
+ ${BAZEL} clean
+ fi
+ fi
+}
--
2.21.0

2141 - 2160 of 53914