No debugging symbols found while using compilation_mode dbg

[sevan...@gmail.com]

Hi 

I followed the manual installation guide at https://github.com/google/asylo/blob/master/INSTALL.md

I built the quickstart example using below:

bazel build  --config=enc-sim --compilation_mode=dbg -s  --strip=never quickstart

(I have also tried bazel build --config=enc-sim --compilation_mode=dbg -s  --copt -g3 --strip=never quickstart)

When I try to run quickstart_host_loader under gdb, gdb says "no debugging symbols found"

I notice that the build is still using -g0 . Any pointers on how to make the build using -g3? 

SUBCOMMAND: # //asylo/examples/quickstart:quickstart_loader [action 'Compiling asylo/examples/quickstart/demo_driver.cc [for host]']

(cd /home/radhikaj/.cache/bazel/_bazel_radhikaj/6be83c60a36556d0e1d9cc3d23278920/execroot/com_google_asylo && \

  exec env - \

    PATH=/home/radhikaj/.opam/system/bin:/home/radhikaj/bin:/home/radhikaj/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \

    PWD=/proc/self/cwd \

  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/host/bin/asylo/examples/quickstart/_objs/quickstart_loader/demo_driver.d '-frandom-seed=bazel-out/host/bin/asylo/examples/quickstart/_objs/quickstart_loader/demo_driver.o' -D__CLANG_SUPPORT_DYN_ANNOTATION__ -iquote . -iquote bazel-out/host/genfiles -iquote bazel-out/host/bin -iquote external/com_google_protobuf -iquote bazel-out/host/genfiles/external/com_google_protobuf -iquote bazel-out/host/bin/external/com_google_protobuf -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -iquote bazel-out/host/bin/external/bazel_tools -iquote external/com_google_absl -iquote bazel-out/host/genfiles/external/com_google_absl -iquote bazel-out/host/bin/external/com_google_absl -iquote external/com_google_asylo -iquote bazel-out/host/genfiles/external/com_google_asylo -iquote bazel-out/host/bin/external/com_google_asylo -iquote external/com_googlesource_code_cctz -iquote bazel-out/host/genfiles/external/com_googlesource_code_cctz -iquote bazel-out/host/bin/external/com_googlesource_code_cctz -iquote external/linux_sgx -iquote bazel-out/host/genfiles/external/linux_sgx -iquote bazel-out/host/bin/external/linux_sgx -iquote external/com_github_gflags_gflags -iquote bazel-out/host/genfiles/external/com_github_gflags_gflags -iquote bazel-out/host/bin/external/com_github_gflags_gflags -isystem external/com_google_protobuf/src -isystem bazel-out/host/genfiles/external/com_google_protobuf/src -isystem bazel-out/host/bin/external/com_google_protobuf/src -isystem external/com_googlesource_code_cctz/include -isystem bazel-out/host/genfiles/external/com_googlesource_code_cctz/include -isystem bazel-out/host/bin/external/com_googlesource_code_cctz/include -isystem external/com_github_gflags_gflags/include -isystem bazel-out/host/genfiles/external/com_github_gflags_gflags/include -isystem bazel-out/host/bin/external/com_github_gflags_gflags/include -g0 -g0 -g3 -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c asylo/examples/quickstart/demo_driver.cc -o bazel-out/host/bin/asylo/examples/quickstart/_objs/quickstart_loader/demo_driver.o

Thanks

Radhika

[Keith Moyer]

Radhika,

This is because the enclave and the loader are built with different toolchains. When building both components in one build invocation, Bazel only passes the build arguments to the toolchain that builds the enclave.

To get debug symbols in the loader, you can build the enclave and the loader separately.

$ bazel build -c dbg //quickstart:quickstart_loader

$ bazel build --config=enc-sim -c dbg //quickstart:demo_enclave

$ gdb bazel-bin/quickstart/quickstart_loader

…

Reading symbols from bazel-bin/quickstart/quickstart_loader...done.

(gdb)

Note, however, that normal GDB doesn’t automatically load the debug symbols for the enclave when the enclave is loaded because it’s not loaded as a normal shared object library.  You can use the GDB from Intel’s SGX SDK if you want that behavior.

$ sgx-gdb --args bazel-bin/quickstart/quickstart_loader --enclave_path bazel-bin/quickstart/demo_enclave --message "In GDB"

...

Reading symbols from bazel-bin/quickstart/quickstart_loader...done.

(gdb) b quickstart/demo_enclave.cc:53

...

(gdb) run

…

add-symbol-file ‘.../quickstart/demo_enclave’ ...

…

Thread 1 "quickstart_load" hit Breakpoint 1, ...

53 const StatusOr<std::string> EncryptMessage(const std::string &message) {

(gdb) p message

$1 = “In GDB”

(gdb)

I hope this helps!

- Keith (Asylo team)

[sevan...@gmail.com]

Hi Keith

Thanks a lot for the instructions. I am still unable to debug the code inside a simulated enclave. As an example: I have the callstack below(Especially Frame #5). I am also unable to set a breakpoint on EncryptMessage. 

Any pointers would be very helpful. Thanks so much again.

(gdb) backtrace

#0  bridge_ocall_enc_untrusted_malloc (pms=0x7fffffffcf00)

    at bazel-out/k8-dbg/genfiles/asylo/platform/arch/sgx/untrusted/generated_bridge_u.c:1185

#1  0x000000000070a37c in do_ocall (bridge=0x6c0f94 <bridge_ocall_enc_untrusted_malloc>, ms=0x7fffffffcf00)

    at external/linux_sgx/psw/urts/linux/sig_handler.cpp:237

#2  0x0000000000705111 in CEnclave::ocall (this=0xab29f0, proc=1, ocall_table=0xa857e0 <ocall_table_bridge>, ms=0x7fffffffcf00)

    at external/linux_sgx/psw/urts/enclave.cpp:368

#3  0x000000000070433f in sgx_ocall (proc=1, ocall_table=0xa857e0 <ocall_table_bridge>, ms=0x7fffffffcf00, trust_thread=0xab2090)

    at external/linux_sgx/psw/urts/routine.cpp:83

#4  0x00000000007090ee in stack_sticker () at external/linux_sgx/psw/urts/linux/enter_enclave.S:244

#5  0x000000000070901b in __morestack () at external/linux_sgx/psw/urts/linux/enter_enclave.S:121

#6  0x000000000070a34b in do_ecall (fn=1, ocall_table=0xa857e0 <ocall_table_bridge>, ms=0x7fffffffd5b0, trust_thread=0xab2090)

    at external/linux_sgx/psw/urts/linux/sig_handler.cpp:228

#7  0x0000000000704f82 in CEnclave::ecall (this=0xab29f0, proc=1, ocall_table=0xa857e0 <ocall_table_bridge>, ms=0x7fffffffd5b0,

    is_switchless=false) at external/linux_sgx/psw/urts/enclave.cpp:318

#8  0x000000000070422a in _sgx_ecall (enclave_id=73259257167874, proc=1, ocall_table=0xa857e0 <ocall_table_bridge>, ms=0x7fffffffd5b0,

    is_switchless=false) at external/linux_sgx/psw/urts/routine.cpp:55

#9  0x000000000070427d in sgx_ecall (enclave_id=73259257167874, proc=1, ocall_table=0xa857e0 <ocall_table_bridge>, ms=0x7fffffffd5b0)

    at external/linux_sgx/psw/urts/routine.cpp:68

#10 0x00000000006c2d4a in ecall_run (eid=73259257167874, retval=0x7fffffffd628, input=0x7fffffffd6f0 "ʲ\004\004\n\002hi", input_len=8,

    output=0x7fffffffd670, output_len=0x7fffffffd678)

    at bazel-out/k8-dbg/genfiles/asylo/platform/arch/sgx/untrusted/generated_bridge_u.c:2061

#11 0x00000000006bd76d in asylo::run (eid=73259257167874, input=0x7fffffffd6f0 "ʲ\004\004\n\002hi", input_len=8, output=0x7fffffffd670,

    output_len=0x7fffffffd678) at asylo/platform/arch/sgx/untrusted/sgx_client.cc:73

#12 0x00000000006be49a in asylo::SGXClient::EnterAndRun (this=0xabd460, input=..., output=0x7fffffffd830)

    at asylo/platform/arch/sgx/untrusted/sgx_client.cc:231

#13 0x00000000006b92b4 in main ()

(gdb) 

Also note that the debugger is unable to find the function EncryptMessage after ecall_run was called.

(gdb) rbreak Encrypt

Note: breakpoint 40 also set at pc 0x72a56a.

Breakpoint 47 at 0x72a56a: file external/boringssl/src/crypto/fipsmodule/cipher/cipher.c, line 313.

int EVP_EncryptFinal_ex(EVP_CIPHER_CTX *, uint8_t *, int *);

Note: breakpoint 41 also set at pc 0x72af1f.

Breakpoint 48 at 0x72af1f: file external/boringssl/src/crypto/fipsmodule/cipher/cipher.c, line 603.

int EVP_EncryptInit(EVP_CIPHER_CTX *, const EVP_CIPHER *, const uint8_t *, const uint8_t *);

Note: breakpoint 42 also set at pc 0x72a22e.

Breakpoint 49 at 0x72a22e: file external/boringssl/src/crypto/fipsmodule/cipher/cipher.c, line 232.

int EVP_EncryptInit_ex(EVP_CIPHER_CTX *, const EVP_CIPHER *, ENGINE *, const uint8_t *, const uint8_t *);

Note: breakpoint 43 also set at pc 0x72a2b4.

Breakpoint 50 at 0x72a2b4: file external/boringssl/src/crypto/fipsmodule/cipher/cipher.c, line 244.

int EVP_EncryptUpdate(EVP_CIPHER_CTX *, uint8_t *, int *, const uint8_t *, int);

[Keith Moyer]

Radhika,

We've recently come to realize that the SGX SDK GDB plugin is not working with the latest Asylo 0.3.0 release. We have determined the issue's root cause and are working on a fix.

In the meantime, you can use Asylo 0.2.2 when debugging with GDB if that suffices. If so, there is also a small change you would need to make to the gdb_sgx_plugin.py file installed by the SDK to get it to work with 0.2.2.  Near the end of that file, there is a newobj_handler function that should be replaced with:

def newobj_handler(event):

    if newobj_handler.once:

        sgx_debugger_init()

        newobj_handler.once=False

    return

newobj_handler.once=True

At the end of this email, I've included a Dockerfile that does this to make sure all of the needed setup has been taken into account. Here is a transcript of commands showing it function (debugging the asylo-examples.tar.gz from our website, with the WORKSPACE changed to use Asylo 0.2.2 instead of 0.3.0).

$ docker build -t asylo-gdb-v0.2.2 .

…

$ ASYLO_GDB_DOCKER="docker run -it --rm --cap-add SYS_PTRACE -v bazel-cache:/root/.cache/bazel -v ${PWD}:/work -w /work asylo-gdb-v0.2.2"

$ BAZEL="${ASYLO_GDB_DOCKER} bazel"

$ GDB="${ASYLO_GDB_DOCKER} sgx-gdb"

$ ${BAZEL} build -c dbg //quickstart:quickstart_loader

…

$ ${BAZEL} build --config=enc-sim -c dbg //quickstart:demo_enclave

…

$ ${GDB} --args bazel-bin/quickstart/quickstart_loader --enclave_path bazel-bin/quickstart/demo_enclave --message "In GDB"

…

Reading symbols from bazel-bin/quickstart/quickstart_loader...done.

(gdb) b quickstart/demo_enclave.cc:54

…

(gdb) run

…

add-symbol-file ‘…/quickstart/demo_enclave’ …

…

Thread 1 "quickstart_load" hit Breakpoint 1, …

54 const StatusOr<std::string> EncryptMessage(const std::string &message) {

(gdb) p message

$1 = "In GDB"

(gdb)

I apologize for the inconvenience related to this.  We will work to make the debugging experience more straightforward.

- Keith

---Dockerfile---

FROM gcr.io/asylo-framework/asylo:buildenv-v0.2.2 AS sgxsdk

ADD https://github.com/intel/linux-sgx/archive/sgx_2.3.tar.gz /tmp/

RUN apt-get update && \

    apt-get install -y \

        autoconf \

        automake \

        gawk \

        libprotobuf-dev \

        libssl1.0-dev \

        libtool \

        ocaml \

        protobuf-compiler \

        wget && \

    tar -C /tmp -xzf /tmp/sgx_2.3.tar.gz && \

    cd /tmp/linux-sgx-sgx_2.3 && \

    gawk -i inplace \

        '/^def newobj_handler/{r=1} \

         /^$/{ \

             if(r) { \

                 print "def newobj_handler(event):"; \

                 print "    if newobj_handler.once:"; \

                 print "        sgx_debugger_init()"; \

                 print "        newobj_handler.once=False"; \

                 print "    return"; \

                 print "newobj_handler.once=True" \

            } \

            r=0 \

        } \

        {if(!r)print $0}' \

        sdk/debugger_interface/linux/gdb-sgx-plugin/gdb_sgx_plugin.py && \

    ./download_prebuilt.sh && \

    make sdk_install_pkg && \

    linux/installer/bin/sgx_linux_x64_sdk_2.3.100.46354.bin --prefix=/opt/intel

FROM gcr.io/asylo-framework/asylo:buildenv-v0.2.2

RUN apt-get update && \

    apt-get install -y \

        gdb

COPY --from=sgxsdk /opt/intel /opt/intel

ENV PATH=${PATH}:/opt/intel/sgxsdk/bin:/opt/intel/sgxsdk/bin/x64

---End of Dockerfile---

[Keith Moyer]

Radhika,

Note that with today's Asylo 0.3.1 release, GDB debugging should be fixed.  The change to gdb_sgx_plugin.py is still needed, but sgx-gdb can be used to debug applications created with Asylo 0.3.1.  I've just reconfirmed the instructions in my previous reply, but with 0.3.1 instead of 0.2.2 in the WORKSPACE and Dockerfile.

Let me know if you have any more questions or issues.

- Keith

--

Keith Moyer |

Software Engineer |

km...@google.com |

425-936-4301

[sevan...@gmail.com]

Thanks Keith. It worked. Thanks a lot for the instructions. In the docker file, I had to change this:

I just to remove this line -  # tar -C /tmp -xzf /tmp/sgx_2.3.tar.gz && \

The ADD on the second line of the docker file takes care of extracting the tar file.

Thanks a lot

Radhika

[Keith Moyer]

Interesting, Radhika. The Docker documentation says that it shouldn't extract the tar file ("If <src> is a URL and <dest> does end with a trailing slash, then the filename is inferred from the URL and the file is downloaded to <dest>/<filename>."). If you were using a locally downloaded tar.gz, it would extract it ("If <src> is a local tar archive in a recognized compression format then it is unpacked as a directory.").  It may be a difference in behavior between versions of Docker. Good to know!

Glad you got it working!

- Keith

--
Visit asylo.dev for the latest information.
---
You received this message because you are subscribed to the Google Groups "Asylo Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asylo-users...@googlegroups.com.
To post to this group, send email to asylo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/asylo-users/679635b6-2c37-4a36-b5eb-a4890961b02a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sevan...@gmail.com]

Hi Keith

Following your instructons I could successfully debug simulation enclaves with Asylo 0.3.0. However, while debugging SGX enclaves, I hit a SIG-KILL. Some details are below.

Could you please give me some suggestions on how to proceed? 

bazel build  --config=sgx -c dbg //asylo/examples/quickstart:demo_enclave

sgx-gdb --args quickstart_host_loader_elf_file_from_host --enclave_path=./demo_enclave --message="hi"

(gdb) b BuildTrustedApplication

(gdb) c

The program is not being run.

(gdb) r

Starting program: /home/radhikaj/.cache/bazel/_bazel_radhikaj/6be83c60a36556d0e1d9cc3d23278920/execroot/com_google_asylo/bazel-out/k8-dbg/bin/asylo/examples/quickstart/quickstart_host_loader_elf_file_from_host --enclave_path=./demo_enclave --message=hi

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

[New Thread 0x7ffff6b36700 (LWP 17934)]

add-symbol-file '/home/radhikaj/.cache/bazel/_bazel_radhikaj/6be83c60a36556d0e1d9cc3d23278920/execroot/com_google_asylo/bazel-out/k8-dbg/bin/asylo/examples/quickstart/demo_enclave' 

Thread 1 "quickstart_host" received signal SIGILL, Illegal instruction.

0x00007ffff024bb22 in google::protobuf::Map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, google::protobuf::Value>::InnerMap::Seed() const 

Thanks,

Radhika

[Keith Moyer]

From your output, it looks like you're getting SIGILL, not SIGKILL.  The way that SGX works is that CPUID and RDTSC are "illegal instructions", so any code that executes them are expected to get SIGILL.

However, Asylo configures the system to trap on that SIGILL emulate the instruction, and continue. GDB defaults to halting the code on any signal, but you can tell it that you don't need to be notified on SIGILLs.

handle SIGILL nostop noprint pass

You can do this before running, or after the first SIGILL is hit (and then you can continue).

[sevan...@gmail.com]

Thanks a lot Keith. That was very helpful.

[Bao Bao]

Hi Keith,

With the latest release version 0.5.3,  `bazel build -c dbg //quickstart:quickstart_loader` doesn't work anymore during our experiment. May I ask what is the current way to only build the loader?

Bao

[Keith Moyer]

In 0.5.3, we were still working on incorporating the new Bazel "transitions" feature, which simplifies the build process significantly. At HEAD, you can build with that command again (transitions have been enabled), but at 0.5.3 you need to add `--define=SGX_SIM=1` when building the loader in isolation.

Full command:

bazel build -c dbg --define=SGX_SIM=1 //quickstart:quickstart_loader

Let me know if you have any more questions!

- Keith

To view this discussion on the web visit https://groups.google.com/d/msgid/asylo-users/485c129a-1d1e-4ff7-9c43-d926730cd57do%40googlegroups.com.

[Keith Moyer]

Small clarification: when I referred to HEAD, I meant HEAD at http://github.com/google/asylo (where we have enabled transitions). That change will not be reflected in http://github.com/google/asylo-examples until we make our next release.  In the meantime, when dealing with asylo 0.5.3, that additional --define is unfortunately needed when building the loader by itself.

[Bao Bao]

That is very helpful. Thank you for the information.

Thanks,

Qinkun Bao