# Floram docker experimentation environment

Ian Goldberg, iang@uwaterloo.ca  
Adithya Vadapalli, adithya.vadapalli@uwaterloo.ca

This repo contains scripts to run [Doerner and shelat's Floram](https://dl.acm.org/doi/10.1145/3133956.3133967) in docker containers for easy experimentation on varying the ORAM size, and network latency and bandwidth.

These scripts are in support of our paper:

Adithya Vadapalli, Ryan Henry, Ian Goldberg. Duoram: A Bandwidth-Efficient Distributed ORAM for 2- and 3-Party Computation. USENIX Security Symposium 2023. [https://eprint.iacr.org/2022/1747](https://eprint.iacr.org/2022/1747)

It is a dockerization of [Doerner and shelat's published code](https://gitlab.com/neucrypt/floram/), with two small changes:

  - Their benchmarking code (`bench_oram_read` and `bench_oram_write`) sets up the ORAM, and then does a number of read or a number of write operations.  The _time_ to set up the ORAM is included in the reported time, but the _bandwidth_ to set up the ORAM is not included in the reported bandwith. We have [a patch](bench_oram.patch) to also measure the bandwidth of the setup, and report it separately from the bandwidth of the operations.
  - We also add [a read/write benchmark](bench_oram_readwrite.oc) that does alternating reads and writes. If you ask for 128 operations, for example, it will do 128 reads and 128 writes, interleaved.

## Reproduction instructions

Follow these instructions to reproduce the Floram data points (timings
and bandwidth usage of Floram operations for various ORAM sizes and
network settings) for the plots in our paper.  See
[below](#manual-instructions) if you want to run experiments of your
choosing.

  - Build the docker image with `./build-docker`
  - Start the dockers with `./start-docker`
    - This will start two dockers, each running one of the parties.
  - Run the reproduction script `./repro` with one of the following
    arguments:
    - <code>./repro test</code>: Run a short (just a few seconds) "kick-the-tires" test.
      You should see output like the following:

            Running test experiment...
            Tue 21 Feb 2023 01:37:45 PM EST: Running read 16 1us 100gbit 2 ...
            Floram read 16 1us 100gbit 2 0.554001 s
            Floram read 16 1us 100gbit 2 3837.724609375 KiB

      The last two lines are the output data points, telling you that a
      Floram read test on an ORAM of size 2<sup>16</sup>, with a network
      configuration of 1us latency and 100gbit bandwidth, performing 2
      read operations, took 0.554001 s of time and 3837.724609375 KiB of
      bandwidth.  If you've run the test before, you will see means and
      stddevs of all of the output data points.  When you run it,
      the time of course will depend on the particulars of your
      hardware, but the bandwidth used should be exactly the value
      quoted above.

    - <code>./repro small _numops_</code>: Run the "small" tests.  These
      are the tests up to size 2<sup>26</sup>, and produce all the data
      points for Figures 7 and 8, and most of Figure 9.
      <code>_numops_</code> is the number of operations to run for each
      test; we used the default of 128 for the figures in the paper, but
      you can use a lower number to make the tests run faster.  For the
      default of 128, these tests should complete in about 4 to 5 hours,
      and require 16 GB of available RAM.

    - <code>./repro large _numops_</code>: Run the "large" tests.  These
      are the rightmost 3 data points in Figure 9. They are not
      essential to our major claims, so they are optional to run,
      and you will definitely require a larger machine to run them.
      For the default <code>_numops_</code> of 128, these experiments
      will require 9 to 10 hours to run and 540 GB of available RAM.
      Reducing <code>_numops_</code> will only slightly reduce the
      runtime (down to 8 to 9 hours) and will not change the RAM
      requirements.

    - <code>./repro all _numops_</code>: Run both the "small" and
      "large" tests.

    - <code>./repro none _numops_</code>: Run no tests. This command is
      nonetheless useful in order to parse the output logs and display
      the data points for the graphs (see below).

    - <code>./repro single _mode_ _size_ _latency_ _bandwidth_ 
      _numops_</code>: run a single manually selected test with the 
      given parameters.

    - After `small`, `large`, `all`, or `none`, the script will parse 
      all of the outputs that have been collected with the specified 
      <code>_numops_</code> (in this run or previous runs), and output 
      them as they would appear in each of the subfigures of Figures 7, 
      8, and 9.

  - When you're done, `./stop-docker`

## Manual instructions

  - `./build-docker`
  - `./start-docker`
    - This will start two dockers, each running one of the parties.

Then to simulate network latency and capacity (optional):

  - `./set-networking 30ms 100mbit`

To turn that off again:

  - `./unset-networking`

If you have a NUMA machine, you might want to pin each party to one
NUMA node.  To do that, set these environment variables before running
`./run-experiment` below:

  - `export FLORAM_NUMA_P0="numactl -N 1 -m 1"`
  - `export FLORAM_NUMA_P1="numactl -N 2 -m 2"`

Adjust the numactl arguments to taste, of course, depending on your
machine's configuration.  Alternately, you can use things like `-C 0-7`
instead of `-N 1` to pin to specific cores, even on a non-NUMA machine.

Run experiments:

  - <code>./run-experiment _mode_ _size_ _numops_ _port_ &gt;&gt; _outfile_</code>
    - <code>_mode_</code> is one of `read`, `write`, `readwrite`, or `init`
      - `init` measures setting up the database with non-zero initial values; the other three modes include setting up the database initialized to 0.  Defaults to `read`.
    - <code>_size_</code> is the base-2 log of the number of entries in the ORAM (so <code>_size_</code> = 20 is an ORAM with 1048576 entries, for example).  Defaults to 20.
    - <code>_numops_</code> is the number of operations to perform; one setup will be followed by <code>_numops_</code> operations, where each operation is a read, a write, or a read plus a write, depending on the <code>_mode_</code>. Defaults to 128.
    - <code>_port_</code> is the port number to use; if you're running multiple experiments at the same time, they must each be on a different port.  Defaults to 3000.

  - <code>./parse\_sizes _outfile_</code>
    - Parses the file output by one or more executions of `./run-experiment` to extract the number of bytes sent in each experiment.  The output will be, for each experiment, a line with the two numbers <code>_size_</code> and <code>_kib_</code>, which are the size of the experiment and the average number of KiB (kibibytes = 1024 bytes) sent per party, including both the ORAM setup and the operations.

  - <code>./parse\_times _outfile_</code>
    - Parses the file output by one or more executions of `./run-experiment` to extract the runtime of each experiment.  The output will be, for each experiment, a line with the two numbers <code>_size_</code> and <code>_sec_</code>, which are the size of the experiment and the time in seconds, including both the ORAM setup and the operations.

To see an example of how to use `./run-experiment` while varying the experiment size and the network latency and bandwidth, and using the NUMA functionality, the [`./run-readwrite-experiments`](run-readwrite-experiments) script wraps `./run-experiment`.

When you're all done:

  - `./stop-docker`