Dittosuite

Dittosuite is a work in progress collection of tools that aims at providing a high-level language called Dittolang that defines operations.

The defined Dittolang operations can be interpreted by Dittosim for simulation to provide a simulated performance measurement and quickly identify the goodness of a solution.

Specularly, Dittobench interprets the Dittolang operations and executes them on a real device, tracking the behavior and measuring the performance.

How to run

$ ./dittobench [options] [.ditto file]

To run a benchmark, a well formed .ditto file must be provided, see section How to write .ditto files In addition, these options can be set:

• --results-output=<int | string> (default: report). Select the results output format. Options: report, csv with 0, 1 respectively.
• --log-stream=<int | string> (default: stdout). Select the output stream for the log messages. Options: stdout, logcat with 0, 1 respectively.
• --log-level=<int | string> (default: INFO). Select to output messages which are at or below the set level. Options: VERBOSE, DEBUG, INFO, WARNING, ERROR, FATAL with 0, 1, 2, 3, 4 and 5 respectively.
• --parameters=string. If the benchmark is parametric, all the parameters (separated by commas) can be given through this option.

How to write .ditto files

Every .ditto file should begin with this skeleton:

main: {
  ...
},
global {
  ...
}

Optionally, it can contain init and clean_up sections:

init: {
  ...
},
main: {
  ...
},
clean_up: {
  ...
},
global {
  ...
}

global

Global section should contain general benchmark configuration. Currently available options:

• (optional) string absolute_path (default = ""). Specifies the absolute path for the files.

init

init is optional and can be used to initialize the benchmarking environment. It executes instructions similar to main, but the results are not collected in the end.

main

main is the entry point for the benchmark. It can contain a single instruction or instruction_set (also with nested instruction_set).

clean_up

clean_up is optional and can be used to reset the benchmarking environment to the initial state, e.g, delete benchmark files. Similar to init, it executes instructions like main, but results are not collected in the end.

instruction

{
  <name of the instruction>: {
    <first argument>,
    <second argument>,
    ...
  },
  <general instruction options>
}

Currently available options:

• (optional) int repeat (default = 1). Specifies how many times the instruction should be repeated.

instruction_set

{
  instruction_set: {
    instructions: {
      {
        <name of the first instruction>: {
          <first argument>,
          <second argument>,
          ...
        },
        <general instruction options>
      },
      {
        <name of the second instruction>: {
          <first argument>,
          <second argument>,
          ...
        },
        <general instruction options>
      },
      ...
    },
    iterate_options: {...}
  },
  <general instruction options>
}

Instruction set is an Instruction container that executes the contained instructions sequentially. Instruction set can optionally iterate over a list and execute the provided set of instructions on each item from the list. To use it, iterate_options should be set with these options:

• string list_name - Shared variable name pointing to a list of values.
• string item_name - Shared variable name to which a selected value should be stored.
• (optional) Order order (default = SEQUENTIAL) - Specifies if the elements of the list should be accessed sequentially or randomly. Options: SEQUENTIAL, RANDOM.
• (optional) Reseeding reseeding (default = ONCE) - Specifies how often the random number generator should be reseeded with the same provided (or generated) seed. Options: ONCE, EACH_ROUND_OF_CYCLES, EACH_CYCLE.
• (optional) uint32 seed - Seed for the random number generator. If the seed is not provided, current system time is used as the seed.

multithreading and threads

multithreading: {
  threads: [
    {
      instruction: {...},
      spawn: <number of threads to spawn with the provided instruction>
    },
    ...
  ]
}

Multithreading is another instruction container that executes the specified instructions (or instruction sets) in different threads. If the optional spawn option for a specific instruction (or instruction set) is provided, then the provided number of threads will be created for it.

Example

main: {
  instruction_set: {
    instructions: [
      {
        open_file: {
          path_name: "newfile2.txt",
          output_fd: "test_file"
        }
      },
      {
        close_file: {
          input_fd: "test_file"
        }
      }
    ]
  },
  repeat: 10
},
global {
  absolute_path: "/data/local/tmp/";
}

See more examples in example/.

Predefined list of instructions

open_file

Opens the file with a file path or a shared variable name pointing to a file path. If neither of those are provided, a random name consisting of 9 random digits is generated. Optionally saves the file descriptor which can then be used by subsequent instructions. Also, can optionally create the file if it does not already exist.

Arguments:

• (optional) string path_name - Specifies the file path.
  OR
  string input - Shared variable name pointing to a file path.
• (optional) string output_fd - Shared variable name to which output file descriptor should be saved.
• (optional) bool create (default = true) - Specifies if the file should be created if it does not already exist. If the file exists, nothing happens.

delete_file

Deletes the file with a file path or a shared variable name pointing to a file path. Uses unlink(2).

Arguments:

• string path_name - Specifies the file path.
  OR
  string input - Shared variable name pointing to a file path.

close_file

Closes the file with the provided file descriptor. Uses close(2).

Arguments:

• string input_fd - Shared variable name pointing to a file descriptor.

resize_file

Resizes the file with the provided file descriptor and new size. If the provided size is greater than the current file size, fallocate(2) is used, while ftruncate(2) is used if the provided size is not greater than the current file size.

Arguments:

• string input_fd - Shared variable name pointing to a file descriptor.
• int64 size - New file size (in bytes).

resize_file_random

Resizes the file with the provided file descriptor and a range for new size. New file size is randomly generated in the provided range and if the generated size is greater than the current file size, fallocate(2) is used, while ftruncate(2) is used if the generated size is not greater than the current file size.

Arguments:

• string input_fd - Shared variable name pointing to a file descriptor.
• int64 min - Minimum value (in bytes)
• int64 max - Maximum value (in bytes)
• (optional) uint32 seed - Seed for the random number generator. If the seed is not provided, current system time is used as the seed.
• (optional) Reseeding reseeding (default = ONCE). How often the random number generator should be reseeded with the provided (or generated) seed. Options: ONCE, EACH_ROUND_OF_CYCLES, EACH_CYCLE.

write_file

Writes to file with the provided file descriptor. For SEQUENTIAL access, the blocks of data will be written sequentially and if the end of the file is reached, new blocks will start from the beginning of the file. For RANDOM access, the block offset, to which data should be written, will be randomly chosen with uniform distribution. 10101010 byte is used for the write operation to fill the memory with alternating ones and zeroes. Uses pwrite64(2).

Arguments:

• string input_fd - Shared variable name pointing to a file descriptor.
• (optional) int64 size (default = -1) - How much data (in bytes) should be written in total. If it is set to -1, then file size is used.
• (optional) int64 block_size (default = 4096) - How much data (in bytes) should be written at once. If it is set to -1, then file size is used.
• (optional) int64 starting_offset (default = 0) - If access_order is set to SEQUENTIAL, then the blocks, to which the data should be written, will start from this starting offset (in bytes).
• (optional) Order access_order (default = SEQUENTIAL) - Order of the write. Options: SEQUENTIAL and RANDOM.
• (optional) uint32 seed - Seed for the random number generator. If the seed is not provided, current system time is used as the seed.
• (optional) bool fsync (default = false) - If set, fsync(2) will be called after the execution of all write operations.
• (optional) Reseeding reseeding (default = ONCE) - How often the random number generator should be reseeded with the provided (or generated) seed. Options: ONCE, EACH_ROUND_OF_CYCLES, EACH_CYCLE.

read_file

Reads from file with the provided file descriptor. For SEQUENTIAL access, the blocks of data will be read sequentially and if the end of the file is reached, new blocks will start from the beginning of the file. For RANDOM access, the block offset, from which data should be read, will be randomly chosen with uniform distribution. Calls posix_fadvise(2) before the read operations. Uses pread64(2).

Arguments:

• string input_fd - Shared variable name pointing to a file descriptor.
• (optional) int64 size (default = -1) - How much data (in bytes) should be read in total. If it is set to -1, then file size is used.
• (optional) int64 block_size (default = 4096) - How much data (in bytes) should be read at once. If it is set to -1, then file size is used.
• (optional) int64 starting_offset (default = 0) - If access_order is set to SEQUENTIAL, then the blocks, from which the data should be read, will start from this starting offset (in bytes).
• (optional) Order access_order (default = SEQUENTIAL) - Order of the read. Options: SEQUENTIAL and RANDOM.
• (optional) uint32 seed - Seed for the random number generator. If the seed is not provided, current system time is used as the seed.
• (optional) ReadFAdvise fadvise (default = AUTOMATIC) - Sets the argument for the posix_fadvise(2) operation. Options: AUTOMATIC, NORMAL, SEQUENTIAL and RANDOM. If AUTOMATIC is set, then POSIX_FADV_SEQUENTIAL or POSIX_FADV_RANDOM will be used for SEQUENTIAL and RANDOM access order respectively.
• (optional) Reseeding reseeding (default = ONCE) - How often the random number generator should be reseeded with the provided (or generated) seed. Options: ONCE, EACH_ROUND_OF_CYCLES, EACH_CYCLE.

read_directory

Reads file names from a directory and stores them as a list in a shared variable. Uses readdir(3).

Arguments:

• string directory_name - Name of the directory
• string output - Shared variable name to which files names should be saved.

invalidate_cache

Drops kernel clean caches, including, dentry, inode and page caches by calling sync() first and then writing 3 to /proc/sys/vm/drop_caches. No arguments.

Dependencies

Android

The project is currently being developed as part of the Android Open Source Project (AOSP) and is supposed to run out-of-the-box.

Linux

The following utilities are required to build the project on Linux:

sudo apt install cmake protobuf-compiler

Testing

Linux

A suite of unit tests is provided in the test/ directory. In Linux these tests can be run with the following commands:

mkdir build
cd build
make
cd test
ctest

Use cases

File operations performance (high priority)

Bandwidth and measurement when dealing with few huge files or many small files. Operations are combinations of sequential/random-offset read/write.

Latency in creating/deleting files/folders.

These operations should be able to be triggered in a multiprogrammed fashion.

Display pipeline

A graph of processes that are communicating with each others in a pipeline of operations that are parallely contributing to the generation of display frames.

Scheduling (low priority)

Spawning tasks (period, duration, deadline) and verifying their scheduling latency and deadline misses count.

Workflow example and implementation nits

In the following scenario, two threads are running.

T1 runs the following operations: Read, Write, Read, sends a request to T2 and waits for the reply, then Write, Read. T2 waits for a request, then Read, Write, then sends the reply to the requester.

Operations are encoded as primitives expressed with ProtoBuffers. The definition of dependencies among threads can be represented as graphs.

Initialization phase

The first graph traversal is performed at initialization time, when all the ProtoBuffer configuration files are distributed among all the binaries so that they can perform all the heavy initialization duties.

Execution phase

After the initialization phase completes the graph can be traversed again to put all the workloads in execution.

Results gathering phase

A final graph traversal can be performed to fetch all the measurements that each entity internally stored.

Post processing

All the measurements must be ordered and later processed to provide useful information to the user.

T1: INIT : [ RD WR RD ] SND RCV [ WR RD ] : END T2: INIT : RCV [ RD WR ] SND : END

Scratch notes

critical path [ READ WRITE READ ] [ READ WRITE ] [ WRITE READ ] --------------------> > < Thread1 III-XXXXXX|X-SSSSSS-XX-TTTT Thread2 III-XXXX|XXX-TTTT ^

   >       XXXXXXX    XX<
                  XXXX

READ WRITE READ

--->

vector<instr*> {read(), write(), read()}; -> start()

RECEIVE READ WRITE READ SEND ---> vector<instr*> {receive(), read(), write(), read(), send()}; start()

lock on receive()