graphene/Documentation/oldwiki/PAL-Host-ABI.md

What is Graphene's PAL Host ABI

PAL Host ABI is the interface used by Graphene to interact with its host. It is translated into the host's native ABI (e.g. system calls for UNIX) by a layer called the Platform Adaptation Layer (PAL). A PAL not only exports a set of APIs (PAL APIs) that can be called by the library OS, but also acts as the loader that bootstraps the library OS. The design of PAL Host ABI strictly follows three primary principles, to guarantee functionality, security, and portability:

• The host ABI must be stateless.
• The host ABI must be a narrowed interface to reduce the attack surface.
• The host ABI must be generic and independent from the native ABI of any of the supported hosts.

Most of the PAL Host ABI is adapted from the Drawbridge library OS.

PAL as Loader

Regardless of the actual implementation, we require PAL to be able to load ELF-format binaries as executables or dynamic libraries, and perform the necessary dynamic relocation. PAL needs to look up all unresolved symbols in loaded binaries and resolve the ones matching the names of PAL APIs. PAL does not and will not resolve other unresolved symbols, so the loaded libraries and executables must resolve them afterwards.

After loading the binaries, PAL needs to load and interpret the manifest files. The manifest syntax is described in [[Graphene Manifest Syntax]].

Manifest and Executable Loading Rules

The PAL loader supports multiple ways of locating the manifest and executable. To run a program in Graphene properly, the PAL loader generally requires both a manifest and an executable, although it is possible to load with only one of them. The user shall specify either the manifest or the executable to load in the command line, and the PAL loader will try to locate the other based on the file name or content.

Precisely, the loading rules for the manifest and executable are as follows:

1. The first argument given to the PAL loader (e.g., pal-Linux, pal-Linux-SGX, pal-FreeBSD, or the cross-platform wrapper, pal-loader) can be either a manifest file or an executable.
2. If an executable is given to the command line, the loader will search for the manifest in the following order: the same file name as the executable with a .manifest or .manifest.sgx extension, a manifest file without any extension, or no manifest at all.
3. If a manifest is given to the command line, and the manifest contains a loader.exec rule, then the rule is used to determine the executable. The loader should exit if the executable file doesn't exist.
4. If a manifest is given to the command line, and the manifest does not contain a loader.exec rule, then the manifest may be used to infer the executable. The potential executable file has the same file name as the manifest file except it doesn't have the .manifest or .manifest.sgx extension.
5. If a manifest is given to the command line, and no executable file can be found either based on any loader.exec rule or inferring from the manifest file, then no executable is used for the execution.

Data Types and Variables

Data Types

PAL handles

The PAL handles are identifiers that are returned by PAL when opening or creating resources. The basic data structure of a PAL handle is defined as follows:

typedef union pal_handle {
    struct {
        PAL_IDX type;
    } hdr;
    /* other resource-specific definitions */
} PAL_HANDLE;

As shown above, a PAL handle is usually defined as a union data type that contains different subtypes that represent each resource such as files, directories, pipes or sockets. The actual memory allocated for the PAL handles may be variable-sized.

Numbers and Flags

PAL_NUM and PAL_FLG types represent integers and flags. On x86-64, they are defined as follows:

typedef uint64_t      PAL_NUM;
typedef uint32_t      PAL_FLG;

Pointers, Buffers and Strings

PAL_PTR and PAL_STR types represent pointers that point to memory, buffers, and strings. On x86-64, they are defined as follows:

typedef const char*   PAL_STR;
typedef void*         PAL_PTR;

Boolean Values

PAL_BOL type represents boolean values (either PAL_TRUE or PAL_FALSE). This data type is commonly used as the return value of a PAL API to determine whether the call succeeded. On x86-64, it is defined as follows:

typedef bool          PAL_BOL;

Graphene Control Block

The control block in Graphene is a structure that provides static information about the current process and its host. It is also a dynamic symbol that will be linked by the library OS and resolved at runtime. Sometimes, for the flexibility or the convenience of the dynamic resolution, the address of the control block may be resolved by a function (pal_control_addr()).

The fields of the Graphene control block are defined as follows:

typedef struct {
    PAL_STR host_type;
    /* An identifier of current picoprocess */
    PAL_NUM process_id;
    PAL_NUM host_id;

    /***** Handles and executables *****/
    /* program manifest */
    PAL_HANDLE manifest_handle;
    /* executable name */
    PAL_STR executable;
    /* handle of parent process */
    PAL_HANDLE parent_process;
    /* handle of first thread */
    PAL_HANDLE first_thread;
    /* debug stream */
    PAL_HANDLE debug_stream;
    /* broadcast RPC stream */
    PAL_HANDLE broadcast_stream;

    /***** Memory layout ******/
    /* The range of user address */
    PAL_PTR_RANGE user_address;
    /* address where executable is loaded */
    PAL_PTR_RANGE executable_range;
    /* manifest preloaded here */
    PAL_PTR_RANGE manifest_preload;

    /***** Host information *****/
    /* host page size / allocation alignment */
    PAL_NUM pagesize, alloc_align;
    /* CPU information */
    PAL_CPU_INFO cpu_info;
    /* Memory information */
    PAL_MEM_INFO mem_info;

    /* Attestation information */
    PAL_STR attestation_status;
    PAL_STR attestation_timestamp;

    /* Purely for profiling */
    PAL_NUM startup_time;
    PAL_NUM host_specific_startup_time;
    PAL_NUM relocation_time;
    PAL_NUM linking_time;
    PAL_NUM manifest_loading_time;
    PAL_NUM allocation_time;
    PAL_NUM tail_startup_time;
    PAL_NUM child_creation_time;
} PAL_CONTROL;

PAL APIs

The PAL APIs contain 44 functions that can be called from the library OS.

Memory Allocation

DkVirtualMemoryAlloc

PAL_PTR DkVirtualMemoryAlloc(PAL_PTR addr, PAL_NUM size, PAL_FLG alloc_type, PAL_FLG prot);

This API allocates virtual memory for the library OS. addr can be either NULL or any valid address aligned at the allocation alignment. When addr is non-NULL, the API will try to allocate the memory at the given address and potentially rewrite any memory previously allocated at the same address. Overwriting any part of PAL and host kernel is forbidden. size must be a positive number, aligned at the allocation alignment.

alloc_type can be a combination of any of the following flags:

/* Memory Allocation Flags */
#define PAL_ALLOC_RESERVE     0x0001   /* Only reserve the memory */
#define PAL_ALLOC_INTERNAL    0x8000   /* Allocate for PAL */

prot can be a combination of the following flags:

/* Memory Protection Flags */
#define PAL_PROT_NONE       0x0     /* Page can not be accessed */
#define PAL_PROT_READ       0x1     /* Page can be read */
#define PAL_PROT_WRITE      0x2     /* Page can be written */
#define PAL_PROT_EXEC       0x4     /* Page can be executed */
#define PAL_PROT_WRITECOPY  0x8     /* Copy on write */

DkVirtualMemoryFree

void DkVirtualMemoryFree(PAL_PTR addr, PAL_NUM size);

This API deallocates a previously allocated memory mapping. Both addr and size must be non-zero and aligned at the allocation alignment.

DkVirtualMemoryProtect

PAL_BOL DkVirtualMemoryProtect(PAL_PTR addr, PAL_NUM size, PAL_FLG prot);

This API modifies the permissions of a previously allocated memory mapping. Both addr and size must be non-zero and aligned at the allocation alignment. prot is defined as [[DkVirtualMemoryAlloc|PAL Host ABI#DkVirtualMemoryAlloc]].

Process Creation

DkProcessCreate

PAL_HANDLE DkProcessCreate(PAL_STR uri, PAL_STR* args);

This API creates a new process to run a separate executable. uri is the URI of the manifest file or the executable to be loaded in the new process. args is an array of strings -- the arguments to be passed to the new process.

DkProcessExit

void DkProcessExit(PAL_NUM exitCode);

This API terminates all threads in the process immediately. exitCode is the exit value returned to the host.

Stream Creation/Connection/Open

DkStreamOpen

PAL_HANDLE DkStreamOpen(PAL_STR uri, PAL_FLG access, PAL_FLG share_flags, PAL_FLG create,
                        PAL_FLG options);

This API opens/creates a stream resource specified by uri. If the resource is successfully opened or created, a PAL handle will be returned for further access such as reading or writing. uri is the URI of the stream to be opened/created. The following is a list of URIs that are supported:

• file:..., dir:...: Files or directories on the host file system. If PAL_CREAT_TRY is given in create flags, the file/directory will be created.
• dev:...: Open a device as a stream. For example, dev:tty represents the standard I/O.
• pipe.srv:<ID>, pipe:<ID>, pipe:: Open a byte stream that can be used for RPC between processes. Pipes are located by numeric IDs. The server side of a pipe can accept any number of connections. If pipe: is given as the URI, it will open an anonymous bidirectional pipe.
• tcp.srv:<ADDR>:<PORT>, tcp:<ADDR>:<PORT>: Open a TCP socket to listen or connect to a remote TCP socket.
• udp.srv:<ADDR>:<PORT>, udp:<ADDR>:<PORT>: Open a UDP socket to listen or connect to a remote UDP socket.

access can be a combination of the following flags:

/* Stream Access Flags */
#define PAL_ACCESS_RDONLY   00
#define PAL_ACCESS_WRONLY   01
#define PAL_ACCESS_RDWR     02
#define PAL_ACCESS_APPEND   04

share_flags can be a combination of the following flags:

/* Stream Sharing Flags */
#define PAL_SHARE_GLOBAL_X    01
#define PAL_SHARE_GLOBAL_W    02
#define PAL_SHARE_GLOBAL_R    04
#define PAL_SHARE_GROUP_X    010
#define PAL_SHARE_GROUP_W    020
#define PAL_SHARE_GROUP_R    040
#define PAL_SHARE_OWNER_X   0100
#define PAL_SHARE_OWNER_W   0200
#define PAL_SHARE_OWNER_R   0400

create can be a combination of the following flags:

/* Stream Create Flags */
#define PAL_CREAT_TRY        0100  /* Create file if does not exist (O_CREAT) */
#define PAL_CREAT_ALWAYS     0200  /* Create file and fail if already exists (O_CREAT|O_EXCL) */

options can be a combination of the following flags:

/* Stream Option Flags */
#define PAL_OPTION_NONBLOCK     04000

DkStreamWaitForClient

PAL_HANDLE DkStreamWaitForClient(PAL_HANDLE handle);

This API is only available for handles that are opened with pipe.srv:..., tcp.srv:..., and udp.srv:.... It blocks until a new connection is accepted and returns the PAL handle for the connection.

DkStreamRead

PAL_NUM DkStreamRead(PAL_HANDLE handle, PAL_NUM offset, PAL_NUM count, PAL_PTR buffer,
                     PAL_PTR source, PAL_NUM size);

This API reads data from an opened stream. If the handle is a file, offset must be specified at each call of DkStreamRead. source and size can be used to return the remote socket address if the handle is a UDP socket. If the handle is a directory, DkStreamRead fills the buffer with the names (NULL-ended) of the files or subdirectories inside of this directory.

DkStreamWrite

PAL_NUM DkStreamWrite(PAL_HANDLE handle, PAL_NUM offset, PAL_NUM count,
                      PAL_PTR buffer, PAL_STR dest);

This API writes data to an opened stream. If the handle is a file, offset must be specified at each call of DkStreamWrite. dest can be used to specify the remote socket address if the handle is a UDP socket.

DkStreamDelete

#define PAL_DELETE_RD       01
#define PAL_DELETE_WR       02
void DkStreamDelete(PAL_HANDLE handle, PAL_FLG access);

This API deletes files or directories on the host or shuts down the connection of TCP/UDP sockets. access specifies the method of shutting down the connection and can be either read-side only, write-side only, or both if 0 is given.

DkStreamMap

PAL_PTR DkStreamMap(PAL_HANDLE handle, PAL_PTR address, PAL_FLG prot,
                    PAL_NUM offset, PAL_NUM size);

This API maps a file to a virtual memory address in the current process. address can be NULL or a valid address that is aligned at the allocation alignment. offset and size have to be non-zero and aligned at the allocation alignment. prot is defined as [[DkVirtualMemoryAlloc|PAL Host ABI#DkVirtualMemoryAlloc]].

DkStreamUnmap

void DkStreamUnmap(PAL_PTR addr, PAL_NUM size);

This API unmaps virtual memory that is backed by a file stream. addr and size must be aligned at the allocation alignment.

DkStreamSetLength

PAL_NUM DkStreamSetLength(PAL_HANDLE handle, PAL_NUM length);

This API truncates or extends a file stream to the given length.

DkStreamFlush

PAL_BOL DkStreamFlush(PAL_HANDLE handle);

This API flushes the buffer of a file stream.

DkSendHandle

PAL_BOL DkSendHandle(PAL_HANDLE handle, PAL_HANDLE cargo);

This API sends a PAL handle cargo over another handle. Currently, the handle that is used to send cargo must be a process handle.

DkReceiveHandle

PAL_HANDLE DkReceiveHandle(PAL_HANDLE handle);

This API receives a handle over another handle.

DkStreamAttributeQuery

PAL_BOL DkStreamAttributesQuery(PAL_STR uri, PAL_STREAM_ATTR* attr);

This API queries the attributes of a named stream. This API only applies for URIs such as file:..., dir:..., and dev:....

The data type PAL_STREAM_ATTR is defined as follows:

/* stream attribute structure */
typedef struct {
    PAL_IDX handle_type;
    PAL_BOL disconnected;
    PAL_BOL nonblocking;
    PAL_BOL readable;
    PAL_BOL writeable;
    PAL_BOL runnable;
    PAL_FLG share_flags;
    PAL_NUM pending_size;
    struct {
        PAL_NUM linger;
        PAL_NUM receivebuf;
        PAL_NUM sendbuf;
        PAL_NUM receivetimeout;
        PAL_NUM sendtimeout;
        PAL_BOL tcp_cork;
        PAL_BOL tcp_keepalive;
        PAL_BOL tcp_nodelay;
    } socket;
} PAL_STREAM_ATTR;

DkStreamAttributesQuerybyHandle

PAL_BOL DkStreamAttributesQuerybyHandle(PAL_HANDLE handle, PAL_STREAM_ATTR* attr);

This API queries the attributes of an opened stream. This API applies to any stream handle.

DkStreamAttributesSetbyHandle

PAL_BOL DkStreamAttributesSetbyHandle(PAL_HANDLE handle, PAL_STREAM_ATTR* attr);

This API sets the attributes of an opened stream.

DkStreamGetName

PAL_NUM DkStreamGetName(PAL_HANDLE handle, PAL_PTR buffer, PAL_NUM size);

This API queries the name of an opened stream.

DkStreamChangeName

PAL_BOL DkStreamChangeName(PAL_HANDLE handle, PAL_STR uri);

This API changes the name of an opened stream.

Thread Creation

DkThreadCreate

PAL_HANDLE DkThreadCreate(PAL_PTR addr, PAL_PTR param);

This API creates a thread in the current process. addr is the address of an entry point of execution for the new thread. param is the pointer argument that is passed to the new thread.

DkThreadDelayExecution

PAL_NUM DkThreadDelayExecution(PAL_NUM duration);

This API suspends the current thread for a certain duration (in microseconds).

DkThreadYieldExecution

void DkThreadYieldExecution(void);

This API yields the current thread such that the host scheduler can reschedule it.

DkThreadExit

void DkThreadExit(void);

This API terminates the current thread.

DkThreadResume

PAL_BOL DkThreadResume(PAL_HANDLE thread);

This API resumes a thread.

Exception Handling

DkSetExceptionHandler

PAL_BOL DkSetExceptionHandler(void (*handler) (PAL_PTR event, PAL_NUM arg, PAL_CONTEXT* context),
                              PAL_NUM event);

This API sets the handler for the specific exception event.

event can be one of the following values:

/* arithmetic error (div-by-zero, floating point exception, etc.) */
#define PAL_EVENT_ARITHMETIC_ERROR 1
/* segmentation fault, protection fault, bus fault */
#define PAL_EVENT_MEMFAULT         2
/* illegal instructions */
#define PAL_EVENT_ILLEGAL          3
/* terminated by external program */
#define PAL_EVENT_QUIT             4
/* suspended by external program */
#define PAL_EVENT_SUSPEND          5
/* continued by external program */
#define PAL_EVENT_RESUME           6
/* failure within PAL calls */
#define PAL_EVENT_FAILURE          7

flags can be a combination of the following flags:

#define PAL_EVENT_PRIVATE      0x0001       /* upcall specific to thread */
#define PAL_EVENT_RESET        0x0002       /* reset the event upcall */

DkExceptionReturn

void DkExceptionReturn(PAL_PTR event);

This API exits an exception handler and restores the context.

Synchronization

DkMutexCreate

PAL_HANDLE DkMutexCreate(PAL_NUM initialCount);

This API creates a mutex with the given initialCount.

DkMutexRelease

void DkMutexRelease(PAL_HANDLE mutexHandle);

This API unlocks the given mutex.

DkNotificationEventCreate/DkSynchronizationEventCreate

PAL_HANDLE DkNotificationEventCreate(PAL_BOL initialState);
PAL_HANDLE DkSynchronizationEventCreate(PAL_BOL initialState);

This API creates an event with the given initialState. The definition of notification events and synchronization events is the same as the WIN32 API. When a notification event is set to the signaled state it remains in that state until it is explicitly cleared. When a synchronization event is set to the signaled state, a single thread of execution that was waiting for the event is released, and the event is automatically reset to the not-signaled state.

DkEventSet

void DkEventSet(PAL_HANDLE eventHandle);

This API sets (signals) a notification event or a synchronization event.

DkEventClear

void DkEventClear(PAL_HANDLE eventHandle);

This API clears a notification event or a synchronization event.

Objects

DkObjectsWaitAny

#define NO_TIMEOUT ((PAL_NUM)-1)
PAL_HANDLE DkObjectsWaitAny(PAL_NUM count, PAL_HANDLE* handleArray, PAL_NUM timeout_us);

This API polls an array of handles and returns one handle with recent activity. timeout is the maximum time that the API should wait (in microseconds), or NO_TIMEOUT to indicate it is to be blocked until at least one handle is ready.

DkObjectClose

void DkObjectClose(PAL_HANDLE objectHandle);

This API closes (deallocates) a PAL handle.

Miscellaneous

DkSystemTimeQuery

PAL_NUM DkSystemTimeQuery(void);

This API returns the current time (in microseconds).

DkRandomBitsRead

PAL_NUM DkRandomBitsRead(PAL_PTR buffer, PAL_NUM size);

This API fills the buffer with cryptographically-secure random values.

DkSegmentRegister

#define PAL_SEGMENT_FS  0x1
#define PAL_SEGMENT_GS  0x2
PAL_PTR DkSegmentRegister(PAL_FLG reg, PAL_PTR addr);

This API sets segment register FS or GS specified by reg to the address addr. If addr is specified as NULL, then this API returns the current value of the segment register.

DkMemoryAvailableQuota

PAL_NUM DkMemoryAvailableQuota(void);

This API returns the amount of currently available memory for LibOS/application usage.

DkCpuIdRetrieve

 #define PAL_CPUID_WORD_EAX  0
 #define PAL_CPUID_WORD_EBX  1
 #define PAL_CPUID_WORD_ECX  2
 #define PAL_CPUID_WORD_EDX  3
 #define PAL_CPUID_WORD_NUM  4
 PAL_BOL DkCpuIdRetrieve(PAL_IDX leaf, PAL_IDX subleaf, PAL_IDX values[4]);

This API returns CPUID information in the array values, based on the leaf/subleaf.

Memory Bulk Copy (Optional)

DkCreatePhysicalMemoryChannel

PAL_HANDLE DkCreatePhysicalMemoryChannel(PAL_NUM* key);

This API creates a physical memory channel for the process to copy virtual memory as copy-on-write. Once a channel is created, other processes can connect to the physical memory channel by using [[DkStreamOpen|PAL Host ABI#DkStreamOpen]] with a URI gipc:<key>.

DkPhysicalMemoryCommit

PAL_NUM DkPhysicalMemoryCommit(PAL_HANDLE channel, PAL_NUM entries, PAL_PTR* addrs,
                               PAL_NUM* sizes);

This API commits (sends) an array of the virtual memory area over the physical memory channel.

DkPhysicalMemoryMap

PAL_NUM DkPhysicalMemoryMap(PAL_HANDLE channel, PAL_NUM entries, PAL_PTR* addrs,
                            PAL_NUM* sizes, PAL_FLG* prots);

This API maps an array of virtual memory area from the physical memory channel.