## What is Graphene's PAL Host ABI

PAL Host ABI is the interface used by Graphene to interact with its host. It is translated into
the host's native ABI (e.g. system calls for UNIX) by a layer called the Platform Adaptation Layer
(PAL). A PAL not only exports a set of APIs (PAL APIs) that can be called by the library OS, but
also acts as the loader that bootstraps the library OS. The design of PAL Host ABI strictly follows
three primary principles, to guarantee functionality, security, and portability:

* The host ABI must be stateless.
* The host ABI must be a narrowed interface to reduce the attack surface.
* The host ABI must be generic and independent from the native ABI of any of the supported hosts.

Most of the PAL Host ABI is adapted from the Drawbridge library OS.

## PAL as Loader

Regardless of the actual implementation, we require PAL to be able to load ELF-format binaries
as executables or dynamic libraries, and perform the necessary dynamic relocation. PAL needs
to look up all unresolved symbols in loaded binaries and resolve the ones matching the names of
PAL APIs. PAL does not and will not resolve other unresolved symbols, so the loaded libraries and
executables must resolve them afterwards.

After loading the binaries, PAL needs to load and interpret the manifest files. The manifest syntax
is described in [[Graphene Manifest Syntax]].

### Manifest and Executable Loading Rules

The PAL loader supports multiple ways of locating the manifest and executable. To run a program
in Graphene properly, the PAL loader generally requires both a manifest and an executable,
although it is possible to load with only one of them. The user shall specify either the manifest
or the executable to load in the command line, and the PAL loader will try to locate the other
based on the file name or content.

Precisely, the loading rules for the manifest and executable are as follows:

1. The first argument given to the PAL loader (e.g., `pal-Linux`, `pal-Linux-SGX`, `pal-FreeBSD`,
or the cross-platform wrapper, `pal-loader`) can be either a manifest file or an executable.
2. If an executable is given to the command line, the loader will search for the manifest in the
following order: the same file name as the executable with a `.manifest` or `.manifest.sgx` extension,
a `manifest` file without any extension, or no manifest at all.
3. If a manifest is given to the command line, and the manifest contains a `loader.exec` rule,
then the rule is used to determine the executable. The loader should exit if the executable file
doesn't exist.
4. If a manifest is given to the command line, and the manifest does *not* contain a `loader.exec rule`,
then the manifest *may* be used to infer the executable. The potential executable file has the same
file name as the manifest file except it doesn't have the `.manifest` or `.manifest.sgx` extension.
5. If a manifest is given to the command line, and no executable file can be found either based on
any `loader.exec` rule or inferring from the manifest file, then no executable is used for the
execution.


## Data Types and Variables

### Data Types

#### PAL handles

The PAL handles are identifiers that are returned by PAL when opening or creating resources. The
basic data structure of a PAL handle is defined as follows:

    typedef union pal_handle {
        struct {
            PAL_IDX type;
        } hdr;
        /* other resource-specific definitions */
    } PAL_HANDLE;

As shown above, a PAL handle is usually defined as a `union` data type that contains different
subtypes that represent each resource such as files, directories, pipes or sockets. The actual
memory allocated for the PAL handles may be variable-sized.

#### Numbers and Flags

`PAL_NUM` and `PAL_FLG` types represent integers and flags. On x86-64, they are defined as follows:

    typedef uint64_t      PAL_NUM;
    typedef uint32_t      PAL_FLG;

#### Pointers, Buffers and Strings

`PAL_PTR` and `PAL_STR` types represent pointers that point to memory, buffers, and strings.
On x86-64, they are defined as follows:

    typedef const char*   PAL_STR;
    typedef void*         PAL_PTR;

#### Boolean Values

`PAL_BOL` type represents boolean values (either `PAL_TRUE` or `PAL_FALSE`). This data type is
commonly used as the return value of a PAL API to determine whether the call succeeded. On x86-64,
it is defined as follows:

    typedef bool          PAL_BOL;

### Graphene Control Block

The control block in Graphene is a structure that provides static information about the current
process and its host. It is also a dynamic symbol that will be linked by the library OS and resolved
at runtime. Sometimes, for the flexibility or the convenience of the dynamic resolution, the
address of the control block may be resolved by a function (`pal_control_addr()`).

The fields of the Graphene control block are defined as follows:

    typedef struct {
        PAL_STR host_type;
        /* An identifier of current picoprocess */
        PAL_NUM process_id;
        PAL_NUM host_id;

        /***** Handles and executables *****/
        /* program manifest */
        PAL_HANDLE manifest_handle;
        /* executable name */
        PAL_STR executable;
        /* handle of parent process */
        PAL_HANDLE parent_process;
        /* handle of first thread */
        PAL_HANDLE first_thread;
        /* debug stream */
        PAL_HANDLE debug_stream;
        /* broadcast RPC stream */
        PAL_HANDLE broadcast_stream;

        /***** Memory layout ******/
        /* The range of user address */
        PAL_PTR_RANGE user_address;
        /* address where executable is loaded */
        PAL_PTR_RANGE executable_range;
        /* manifest preloaded here */
        PAL_PTR_RANGE manifest_preload;

        /***** Host information *****/
        /* host page size / allocation alignment */
        PAL_NUM pagesize, alloc_align;
        /* CPU information */
        PAL_CPU_INFO cpu_info;
        /* Memory information */
        PAL_MEM_INFO mem_info;

        /* Attestation information */
        PAL_STR attestation_status;
        PAL_STR attestation_timestamp;

        /* Purely for profiling */
        PAL_NUM startup_time;
        PAL_NUM host_specific_startup_time;
        PAL_NUM relocation_time;
        PAL_NUM linking_time;
        PAL_NUM manifest_loading_time;
        PAL_NUM allocation_time;
        PAL_NUM tail_startup_time;
        PAL_NUM child_creation_time;
    } PAL_CONTROL;

## PAL APIs

The PAL APIs contain 44 functions that can be called from the library OS.

### Memory Allocation

#### DkVirtualMemoryAlloc

    PAL_PTR DkVirtualMemoryAlloc(PAL_PTR addr, PAL_NUM size, PAL_FLG alloc_type, PAL_FLG prot);

This API allocates virtual memory for the library OS. `addr` can be either `NULL` or any valid
address aligned at the allocation alignment. When `addr` is non-NULL, the API will try
to allocate the memory at the given address and potentially rewrite any memory previously allocated
at the same address. Overwriting any part of PAL and host kernel is forbidden. `size` must be a
positive number, aligned at the allocation alignment.

`alloc_type` can be a combination of any of the following flags:

    /* Memory Allocation Flags */
    #define PAL_ALLOC_RESERVE     0x0001   /* Only reserve the memory */
    #define PAL_ALLOC_INTERNAL    0x8000   /* Allocate for PAL */

`prot` can be a combination of the following flags:

    /* Memory Protection Flags */
    #define PAL_PROT_NONE       0x0     /* Page can not be accessed */
    #define PAL_PROT_READ       0x1     /* Page can be read */
    #define PAL_PROT_WRITE      0x2     /* Page can be written */
    #define PAL_PROT_EXEC       0x4     /* Page can be executed */
    #define PAL_PROT_WRITECOPY  0x8     /* Copy on write */

#### DkVirtualMemoryFree

    void DkVirtualMemoryFree(PAL_PTR addr, PAL_NUM size);

This API deallocates a previously allocated memory mapping. Both `addr` and `size` must be non-zero
and aligned at the allocation alignment.

#### DkVirtualMemoryProtect

    PAL_BOL DkVirtualMemoryProtect(PAL_PTR addr, PAL_NUM size, PAL_FLG prot);

This API modifies the permissions of a previously allocated memory mapping. Both `addr` and
`size` must be non-zero and aligned at the allocation alignment. `prot` is defined as
[[DkVirtualMemoryAlloc|PAL Host ABI#DkVirtualMemoryAlloc]].

### Process Creation

#### DkProcessCreate

    PAL_HANDLE DkProcessCreate(PAL_STR uri, PAL_STR* args);

This API creates a new process to run a separate executable. `uri` is the URI of the manifest file
or the executable to be loaded in the new process. `args` is an array of strings -- the arguments
to be passed to the new process.

#### DkProcessExit

    void DkProcessExit(PAL_NUM exitCode);

This API terminates all threads in the process immediately. `exitCode` is the exit value returned
to the host.

### Stream Creation/Connection/Open

#### DkStreamOpen

    PAL_HANDLE DkStreamOpen(PAL_STR uri, PAL_FLG access, PAL_FLG share_flags, PAL_FLG create,
                            PAL_FLG options);

This API opens/creates a stream resource specified by `uri`. If the resource is successfully opened
or created, a PAL handle will be returned for further access such as reading or writing. `uri` is
the URI of the stream to be opened/created. The following is a list of URIs that are supported:

* `file:...`, `dir:...`: Files or directories on the host file system. If `PAL_CREAT_TRY` is given
   in `create` flags, the file/directory will be created.
* `dev:...`: Open a device as a stream. For example, `dev:tty` represents the standard I/O.
* `pipe.srv:<ID>`, `pipe:<ID>`, `pipe:`: Open a byte stream that can be used for RPC between
   processes. Pipes are located by numeric IDs. The server side of a pipe can accept any number
   of connections. If `pipe:` is given as the URI, it will open an anonymous bidirectional pipe.
* `tcp.srv:<ADDR>:<PORT>`, `tcp:<ADDR>:<PORT>`: Open a TCP socket to listen or connect to
   a remote TCP socket.
* `udp.srv:<ADDR>:<PORT>`, `udp:<ADDR>:<PORT>`: Open a UDP socket to listen or connect to
   a remote UDP socket.

`access` can be a combination of the following flags:

    /* Stream Access Flags */
    #define PAL_ACCESS_RDONLY   00
    #define PAL_ACCESS_WRONLY   01
    #define PAL_ACCESS_RDWR     02
    #define PAL_ACCESS_APPEND   04

`share_flags` can be a combination of the following flags:

    /* Stream Sharing Flags */
    #define PAL_SHARE_GLOBAL_X    01
    #define PAL_SHARE_GLOBAL_W    02
    #define PAL_SHARE_GLOBAL_R    04
    #define PAL_SHARE_GROUP_X    010
    #define PAL_SHARE_GROUP_W    020
    #define PAL_SHARE_GROUP_R    040
    #define PAL_SHARE_OWNER_X   0100
    #define PAL_SHARE_OWNER_W   0200
    #define PAL_SHARE_OWNER_R   0400

`create` can be a combination of the following flags:

    /* Stream Create Flags */
    #define PAL_CREAT_TRY        0100  /* Create file if does not exist (O_CREAT) */
    #define PAL_CREAT_ALWAYS     0200  /* Create file and fail if already exists (O_CREAT|O_EXCL) */

`options` can be a combination of the following flags:

    /* Stream Option Flags */
    #define PAL_OPTION_NONBLOCK     04000

#### DkStreamWaitForClient

    PAL_HANDLE DkStreamWaitForClient(PAL_HANDLE handle);

This API is only available for handles that are opened with `pipe.srv:...`, `tcp.srv:...`, and
`udp.srv:...`. It blocks until a new connection is accepted and returns the PAL handle for the
connection.

#### DkStreamRead

    PAL_NUM DkStreamRead(PAL_HANDLE handle, PAL_NUM offset, PAL_NUM count, PAL_PTR buffer,
                         PAL_PTR source, PAL_NUM size);

This API reads data from an opened stream. If the handle is a file, `offset` must be specified
at each call of DkStreamRead. `source` and `size` can be used to return the remote socket
address if the handle is a UDP socket. If the handle is a directory, DkStreamRead fills the buffer
with the names (NULL-ended) of the files or subdirectories inside of this directory.

#### DkStreamWrite

    PAL_NUM DkStreamWrite(PAL_HANDLE handle, PAL_NUM offset, PAL_NUM count,
                          PAL_PTR buffer, PAL_STR dest);

This API writes data to an opened stream. If the handle is a file, `offset` must be specified
at each call of DkStreamWrite. `dest` can be used to specify the remote socket address if the
handle is a UDP socket.

#### DkStreamDelete

    #define PAL_DELETE_RD       01
    #define PAL_DELETE_WR       02
    void DkStreamDelete(PAL_HANDLE handle, PAL_FLG access);

This API deletes files or directories on the host or shuts down the connection of TCP/UDP sockets.
`access` specifies the method of shutting down the connection and can be either read-side only,
write-side only, or both if 0 is given.

#### DkStreamMap

    PAL_PTR DkStreamMap(PAL_HANDLE handle, PAL_PTR address, PAL_FLG prot,
                        PAL_NUM offset, PAL_NUM size);

This API maps a file to a virtual memory address in the current process. `address` can be NULL or
a valid address that is aligned at the allocation alignment. `offset` and `size` have to be non-zero
and aligned at the allocation alignment. `prot` is defined as
[[DkVirtualMemoryAlloc|PAL Host ABI#DkVirtualMemoryAlloc]].

#### DkStreamUnmap

    void DkStreamUnmap(PAL_PTR addr, PAL_NUM size);

This API unmaps virtual memory that is backed by a file stream. `addr` and `size` must be aligned
at the allocation alignment.

#### DkStreamSetLength

    PAL_NUM DkStreamSetLength(PAL_HANDLE handle, PAL_NUM length);

This API truncates or extends a file stream to the given length.

#### DkStreamFlush

    PAL_BOL DkStreamFlush(PAL_HANDLE handle);

This API flushes the buffer of a file stream.

#### DkSendHandle

    PAL_BOL DkSendHandle(PAL_HANDLE handle, PAL_HANDLE cargo);

This API sends a PAL handle `cargo` over another handle. Currently, the handle that is used
to send cargo must be a process handle.

#### DkReceiveHandle

    PAL_HANDLE DkReceiveHandle(PAL_HANDLE handle);

This API receives a handle over another handle.

#### DkStreamAttributeQuery

    PAL_BOL DkStreamAttributesQuery(PAL_STR uri, PAL_STREAM_ATTR* attr);

This API queries the attributes of a named stream. This API only applies for URIs such as
`file:...`, `dir:...`, and `dev:...`.

The data type `PAL_STREAM_ATTR` is defined as follows:

    /* stream attribute structure */
    typedef struct {
        PAL_IDX handle_type;
        PAL_BOL disconnected;
        PAL_BOL nonblocking;
        PAL_BOL readable;
        PAL_BOL writeable;
        PAL_BOL runnable;
        PAL_FLG share_flags;
        PAL_NUM pending_size;
        struct {
            PAL_NUM linger;
            PAL_NUM receivebuf;
            PAL_NUM sendbuf;
            PAL_NUM receivetimeout;
            PAL_NUM sendtimeout;
            PAL_BOL tcp_cork;
            PAL_BOL tcp_keepalive;
            PAL_BOL tcp_nodelay;
        } socket;
    } PAL_STREAM_ATTR;

#### DkStreamAttributesQuerybyHandle

    PAL_BOL DkStreamAttributesQuerybyHandle(PAL_HANDLE handle, PAL_STREAM_ATTR* attr);

This API queries the attributes of an opened stream. This API applies to any stream handle.

#### DkStreamAttributesSetbyHandle

    PAL_BOL DkStreamAttributesSetbyHandle(PAL_HANDLE handle, PAL_STREAM_ATTR* attr);

This API sets the attributes of an opened stream.

#### DkStreamGetName

    PAL_NUM DkStreamGetName(PAL_HANDLE handle, PAL_PTR buffer, PAL_NUM size);

This API queries the name of an opened stream.

#### DkStreamChangeName

    PAL_BOL DkStreamChangeName(PAL_HANDLE handle, PAL_STR uri);

This API changes the name of an opened stream.

### Thread Creation

#### DkThreadCreate

    PAL_HANDLE DkThreadCreate(PAL_PTR addr, PAL_PTR param);

This API creates a thread in the current process. `addr` is the address of an entry point of
execution for the new thread. `param` is the pointer argument that is passed to the new thread.

#### DkThreadDelayExecution

    PAL_NUM DkThreadDelayExecution(PAL_NUM duration);

This API suspends the current thread for a certain duration (in microseconds).

#### DkThreadYieldExecution

    void DkThreadYieldExecution(void);

This API yields the current thread such that the host scheduler can reschedule it.

#### DkThreadExit

    void DkThreadExit(void);

This API terminates the current thread.

#### DkThreadResume

    PAL_BOL DkThreadResume(PAL_HANDLE thread);

This API resumes a thread.

### Exception Handling

#### DkSetExceptionHandler

    PAL_BOL DkSetExceptionHandler(void (*handler) (PAL_PTR event, PAL_NUM arg, PAL_CONTEXT* context),
                                  PAL_NUM event);

This API sets the handler for the specific exception event.

`event` can be one of the following values:

    /* arithmetic error (div-by-zero, floating point exception, etc.) */
    #define PAL_EVENT_ARITHMETIC_ERROR 1
    /* segmentation fault, protection fault, bus fault */
    #define PAL_EVENT_MEMFAULT         2
    /* illegal instructions */
    #define PAL_EVENT_ILLEGAL          3
    /* terminated by external program */
    #define PAL_EVENT_QUIT             4
    /* suspended by external program */
    #define PAL_EVENT_SUSPEND          5
    /* continued by external program */
    #define PAL_EVENT_RESUME           6
    /* failure within PAL calls */
    #define PAL_EVENT_FAILURE          7

`flags` can be a combination of the following flags:

    #define PAL_EVENT_PRIVATE      0x0001       /* upcall specific to thread */
    #define PAL_EVENT_RESET        0x0002       /* reset the event upcall */

#### DkExceptionReturn

    void DkExceptionReturn(PAL_PTR event);

This API exits an exception handler and restores the context.

### Synchronization

#### DkMutexCreate

    PAL_HANDLE DkMutexCreate(PAL_NUM initialCount);

This API creates a mutex with the given `initialCount`.

#### DkMutexRelease

    void DkMutexRelease(PAL_HANDLE mutexHandle);

This API unlocks the given mutex.

##### DkNotificationEventCreate/DkSynchronizationEventCreate

    PAL_HANDLE DkNotificationEventCreate(PAL_BOL initialState);
    PAL_HANDLE DkSynchronizationEventCreate(PAL_BOL initialState);

This API creates an event with the given `initialState`. The definition of notification events
and synchronization events is the same as the WIN32 API. When a notification event is set to the
signaled state it remains in that state until it is explicitly cleared. When a synchronization
event is set to the signaled state, a single thread of execution that was waiting for the event is
released, and the event is automatically reset to the not-signaled state.

#### DkEventSet

    void DkEventSet(PAL_HANDLE eventHandle);

This API sets (signals) a notification event or a synchronization event.

#### DkEventClear

    void DkEventClear(PAL_HANDLE eventHandle);

This API clears a notification event or a synchronization event.

### Objects

#### DkObjectsWaitAny

    #define NO_TIMEOUT ((PAL_NUM)-1)
    PAL_HANDLE DkObjectsWaitAny(PAL_NUM count, PAL_HANDLE* handleArray, PAL_NUM timeout_us);

This API polls an array of handles and returns one handle with recent activity. `timeout` is the
maximum time that the API should wait (in microseconds), or `NO_TIMEOUT` to indicate it is to be
blocked until at least one handle is ready.

#### DkObjectClose

    void DkObjectClose(PAL_HANDLE objectHandle);

This API closes (deallocates) a PAL handle.

### Miscellaneous

#### DkSystemTimeQuery

    PAL_NUM DkSystemTimeQuery(void);

This API returns the current time (in microseconds).

#### DkRandomBitsRead

    PAL_NUM DkRandomBitsRead(PAL_PTR buffer, PAL_NUM size);

This API fills the buffer with cryptographically-secure random values.

#### DkSegmentRegister

    #define PAL_SEGMENT_FS  0x1
    #define PAL_SEGMENT_GS  0x2
    PAL_PTR DkSegmentRegister(PAL_FLG reg, PAL_PTR addr);

This API sets segment register FS or GS specified by `reg` to the address `addr`. If `addr` is
specified as NULL, then this API returns the current value of the segment register.

#### DkMemoryAvailableQuota

    PAL_NUM DkMemoryAvailableQuota(void);

This API returns the amount of currently available memory for LibOS/application usage.

#### DkCpuIdRetrieve

     #define PAL_CPUID_WORD_EAX  0
     #define PAL_CPUID_WORD_EBX  1
     #define PAL_CPUID_WORD_ECX  2
     #define PAL_CPUID_WORD_EDX  3
     #define PAL_CPUID_WORD_NUM  4
     PAL_BOL DkCpuIdRetrieve(PAL_IDX leaf, PAL_IDX subleaf, PAL_IDX values[4]);

This API returns CPUID information in the array `values`, based on the leaf/subleaf.


### Memory Bulk Copy (Optional)

#### DkCreatePhysicalMemoryChannel

    PAL_HANDLE DkCreatePhysicalMemoryChannel(PAL_NUM* key);

This API creates a physical memory channel for the process to copy virtual memory as copy-on-write.
Once a channel is created, other processes can connect to the physical memory channel by using
[[DkStreamOpen|PAL Host ABI#DkStreamOpen]] with a URI `gipc:<key>`.

#### DkPhysicalMemoryCommit

    PAL_NUM DkPhysicalMemoryCommit(PAL_HANDLE channel, PAL_NUM entries, PAL_PTR* addrs,
                                   PAL_NUM* sizes);

This API commits (sends) an array of the virtual memory area over the physical memory channel.

#### DkPhysicalMemoryMap

    PAL_NUM DkPhysicalMemoryMap(PAL_HANDLE channel, PAL_NUM entries, PAL_PTR* addrs,
                                PAL_NUM* sizes, PAL_FLG* prots);

This API maps an array of virtual memory area from the physical memory channel.
