What is the recommended method for debugging applications with Yocto


Patrick Doyle <wpdster@...>
 

I know I can do

$ bitbake base-image -cpopulate_sdk

and distribute the resulting SDK to developers (myself included).

But suppose I've just built and deployed an image and want to debug a
core file generated by one of the applications.

I don't see .debug directories anywhere in $TMP, so when I try to tell
gdb to use the sysroot from the application, that works, but I don't
get debugging symbols libc, nor do I get stack traces (on my MIPS
target).
(I guess it's possible that I would have found the .debug directories
if I had performed a complete build from scratch without the benefit
of the sstate-cache, but I rely upon the sstate-cache and routinely
start with a blank $TMP)

So, I can populate the sdk, and end up with a pile of .debug directories:

$ find . -name .debug -type d
./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug

Which sysroot directory is the best one at which I can point gdb for
getting debug symbols?

If I build this with an autobuilder and want to archive the sysroot
for use by gdb later, which one should I archive?

I could archive the generated SDK, but that will require that I (or
others) install the SDK just to debug a core file, which seems a
little heavy to me. But I can do that, if that's "the Yocto way".

I wonder if there is a lighter weight mechanism for accessing the
debugging symbols during development, and for archiving said symbols
for post-development support.

Any ideas? Best practices? Recommended approaches?

Thanks

--wpd


Khem Raj
 

On 5/15/20 8:25 AM, Patrick Doyle wrote:
I know I can do
$ bitbake base-image -cpopulate_sdk
and distribute the resulting SDK to developers (myself included).
But suppose I've just built and deployed an image and want to debug a
core file generated by one of the applications.
I don't see .debug directories anywhere in $TMP, so when I try to tell
gdb to use the sysroot from the application, that works, but I don't
get debugging symbols libc, nor do I get stack traces (on my MIPS
target).
(I guess it's possible that I would have found the .debug directories
if I had performed a complete build from scratch without the benefit
of the sstate-cache, but I rely upon the sstate-cache and routinely
start with a blank $TMP)
So, I can populate the sdk, and end up with a pile of .debug directories:
$ find . -name .debug -type d
./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug
Which sysroot directory is the best one at which I can point gdb for
getting debug symbols?
If I build this with an autobuilder and want to archive the sysroot
for use by gdb later, which one should I archive?
I could archive the generated SDK, but that will require that I (or
others) install the SDK just to debug a core file, which seems a
little heavy to me. But I can do that, if that's "the Yocto way".
I wonder if there is a lighter weight mechanism for accessing the
debugging symbols during development, and for archiving said symbols
for post-development support.
Any ideas? Best practices? Recommended approaches?
I would think of using IMAGE_GEN_DEBUGFS set to 1 which should generate a debug archive of image

see following for detailed steps

https://docs.windriver.com/bundle/Wind_River_Linux_Platform_Developers_Guide_9_1/page/fvn1520619692704.html

https://developer.ridgerun.com/wiki/index.php?title=Preparing_Yocto_Development_Environment_for_Debugging

Thanks
--wpd


Alexander Kanavin
 

The three debugging sections in the yocto manual should help?


Alex


On Fri, 15 May 2020 at 17:26, Patrick Doyle <wpdster@...> wrote:
I know I can do

$ bitbake base-image -cpopulate_sdk

and distribute the resulting SDK to developers (myself included).

But suppose I've just built and deployed an image and want to debug a
core file generated by one of the applications.

I don't see .debug directories anywhere in $TMP, so when I try to tell
gdb to use the sysroot from the application, that works, but I don't
get debugging symbols libc, nor do I get stack traces (on my MIPS
target).
(I guess it's possible that I would have found the .debug directories
if I had performed a complete build from scratch without the benefit
of the sstate-cache, but I rely upon the sstate-cache and routinely
start with a blank $TMP)

So, I can populate the sdk, and end up with a pile of .debug directories:

$ find . -name .debug -type d
./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug

Which sysroot directory is the best one at which I can point gdb for
getting debug symbols?

If I build this with an autobuilder and want to archive the sysroot
for use by gdb later, which one should I archive?

I could archive the generated SDK, but that will require that I (or
others) install the SDK just to debug a core file, which seems a
little heavy to me.  But I can do that, if that's "the Yocto way".

I wonder if there is a lighter weight mechanism for accessing the
debugging symbols during development, and for archiving said symbols
for post-development support.

Any ideas?  Best practices?  Recommended approaches?

Thanks

--wpd


Andreas Müller
 

On Fri, May 15, 2020 at 5:36 PM Khem Raj <raj.khem@...> wrote:

On 5/15/20 8:25 AM, Patrick Doyle wrote:
I know I can do

$ bitbake base-image -cpopulate_sdk

and distribute the resulting SDK to developers (myself included).

But suppose I've just built and deployed an image and want to debug a
core file generated by one of the applications.

I don't see .debug directories anywhere in $TMP, so when I try to tell
gdb to use the sysroot from the application, that works, but I don't
get debugging symbols libc, nor do I get stack traces (on my MIPS
target).
(I guess it's possible that I would have found the .debug directories
if I had performed a complete build from scratch without the benefit
of the sstate-cache, but I rely upon the sstate-cache and routinely
start with a blank $TMP)

So, I can populate the sdk, and end up with a pile of .debug directories:

$ find . -name .debug -type d
./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug

Which sysroot directory is the best one at which I can point gdb for
getting debug symbols?

If I build this with an autobuilder and want to archive the sysroot
for use by gdb later, which one should I archive?

I could archive the generated SDK, but that will require that I (or
others) install the SDK just to debug a core file, which seems a
little heavy to me. But I can do that, if that's "the Yocto way".

I wonder if there is a lighter weight mechanism for accessing the
debugging symbols during development, and for archiving said symbols
for post-development support.

Any ideas? Best practices? Recommended approaches?
I would think of using IMAGE_GEN_DEBUGFS set to 1 which should generate
a debug archive of image

see following for detailed steps

https://docs.windriver.com/bundle/Wind_River_Linux_Platform_Developers_Guide_9_1/page/fvn1520619692704.html

https://developer.ridgerun.com/wiki/index.php?title=Preparing_Yocto_Development_Environment_for_Debugging

Thanks

--wpd
I created [1] in meta-mortsgna. It is not recommended (bit of a hack /
nobody but me and my colleagues use it) but it

* works on the fly: Once enabled building the recipe you want to debug
is good enough
* not bound to SDK or image which tends to last
* no manual copying necessary

[1] https://github.com/schnitzeltony/meta-mortsgna/blob/master/classes/instant-sysroot-target.bbclass

Andreas


Patrick Doyle <wpdster@...>
 

Thanks folks, IMAGE_GEN_DEBUGFS was exactly what I was looking for.

--wpd