Browse Source

[LibOS] Use -O2 optimization level when building in non-debug mode

There was a bug in LibOS code which manifested in heisenbugs in our tests.
The bug happened because `shim_clone.c:clone_implementation_wrapper()` called
`object_wait_with_retry()/DkObjectClose()` before actually setting up a proper
in-enclave TCB. The root cause was:
1. The newly created thread has an in-enclave `FS segment register == 0`
   because Graphene-SGX doesn't set up SGX's `TCS.OFSBASGX`, so in-enclave FS
   register was equal to `%fs == TCS.OFSBASGX == 0`.
2. `object_wait_with_retry()` checks `PAL_NATIVE_ERRNO` which is implemented
   via TCB's `mov %fs:<offset>`.
3. However, `%fs == 0` at this point so there was a SIGBUS while executing
   `mov %fs:<offset>`.

The fix (in this commit) is to move `object_wait_with_retry()/DkObjectClose()`
after TCB set-up in `allocate_tls()`.
Dmitrii Kuvaiskii 4 years ago
parent
commit
8e3fe7a86f
2 changed files with 9 additions and 4 deletions
  1. 2 0
      LibOS/shim/src/Makefile
  2. 7 4
      LibOS/shim/src/sys/shim_clone.c

+ 2 - 0
LibOS/shim/src/Makefile

@@ -63,6 +63,8 @@ ifeq ($(DEBUG),1)
 CC += -gdwarf-2 -g3
 CFLAGS += -DDEBUG
 ASFLAGS += -DDEBUG
+else
+CFLAGS += -O2
 endif
 export DEBUG
 

+ 7 - 4
LibOS/shim/src/sys/shim_clone.c

@@ -122,9 +122,6 @@ int clone_implementation_wrapper(struct clone_args * arg)
 
     int stack_allocated = 0;
 
-    object_wait_with_retry(arg->create_event);
-    DkObjectClose(arg->create_event);
-
     /* We acquired ownership of arg->thread from the caller, hence there is
      * no need to call get_thread. */
     struct shim_thread* my_thread = arg->thread;
@@ -134,8 +131,14 @@ int clone_implementation_wrapper(struct clone_args * arg)
         stack_allocated = 1;
         my_thread->tcb = __alloca(sizeof(__libc_tcb_t) + PTHREAD_PADDING);
     }
-    allocate_tls(my_thread->tcb, my_thread->user_tcb, my_thread);
+    allocate_tls(my_thread->tcb, my_thread->user_tcb, my_thread); /* set up TCB */
     shim_tcb_t * tcb = &my_thread->tcb->shim_tcb;
+
+    /* only now we can call LibOS/PAL functions because they require a set-up TCB;
+     * do not move the below functions before allocate_tls()! */
+    object_wait_with_retry(arg->create_event);
+    DkObjectClose(arg->create_event);
+
     __disable_preempt(tcb); // Temporarily disable preemption, because the preemption
                             // will be re-enabled when the thread starts.
     debug_setbuf(tcb, true);