# Dittosuite Dittosuite is a work in progress collection of tools that aims at providing a high-level language called Dittolang that defines operations. The defined Dittolang operations can be interpreted by Dittosim for simulation to provide a simulated performance measurement and quickly identify the goodness of a solution. Specularly, Dittobench interprets the Dittolang operations and executes them on a real device, tracking the behavior and measuring the performance. # How to run ``` $ ./dittobench [options] [.ditto file] ``` To run a benchmark, a well formed .ditto file must be provided, see section [How to write .ditto files](#how-to-write-ditto-files) In addition, these options can be set: - `--results-output=` (default: report). Select the results output format. Options: report, csv with 0, 1 respectively. - `--log-stream=` (default: stdout). Select the output stream for the log messages. Options: stdout, logcat with 0, 1 respectively. - `--log-level=` (default: INFO). Select to output messages which are at or below the set level. Options: VERBOSE, DEBUG, INFO, WARNING, ERROR, FATAL with 0, 1, 2, 3, 4 and 5 respectively. - `--parameters=string`. If the benchmark is parametric, all the parameters (separated by commas) can be given through this option. # How to write .ditto files Every .ditto file should begin with this skeleton: ``` main: { ... }, global { ... } ``` Optionally, it can contain `init` and `clean_up` sections: ``` init: { ... }, main: { ... }, clean_up: { ... }, global { ... } ``` ## `global` Global section should contain general benchmark configuration. Currently available options: - (optional) `string absolute_path` (`default = ""`). Specifies the absolute path for the files. ## `init` `init` is optional and can be used to initialize the benchmarking environment. It executes instructions similar to `main`, but the results are not collected in the end. ## `main` `main` is the entry point for the benchmark. It can contain a single `instruction` or `instruction_set` (also with nested `instruction_set`). ## `clean_up` `clean_up` is optional and can be used to reset the benchmarking environment to the initial state, e.g, delete benchmark files. Similar to `init`, it executes instructions like `main`, but results are not collected in the end. ## `instruction` ``` { : { , , ... }, } ``` Currently available options: - (optional) `int repeat` (`default = 1`). Specifies how many times the instruction should be repeated. ## `instruction_set` ``` { instruction_set: { instructions: { { : { , , ... }, }, { : { , , ... }, }, ... }, iterate_options: {...} }, } ``` Instruction set is an Instruction container that executes the contained instructions sequentially. Instruction set can optionally iterate over a list and execute the provided set of instructions on each item from the list. To use it, `iterate_options` should be set with these options: - `string list_name` - Shared variable name pointing to a list of values. - `string item_name` - Shared variable name to which a selected value should be stored. - (optional) `Order order` (`default = SEQUENTIAL`) - Specifies if the elements of the list should be accessed sequentially or randomly. Options: `SEQUENTIAL`, `RANDOM`. - (optional) `Reseeding reseeding` (`default = ONCE`) - Specifies how often the random number generator should be reseeded with the same provided (or generated) seed. Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. - (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, current system time is used as the seed. ## `multithreading` and `threads` ``` multithreading: { threads: [ { instruction: {...}, spawn: }, ... ] } ``` Multithreading is another instruction container that executes the specified instructions (or instruction sets) in different threads. If the optional `spawn` option for a specific instruction (or instruction set) is provided, then the provided number of threads will be created for it. ## Example ``` main: { instruction_set: { instructions: [ { open_file: { path_name: "newfile2.txt", output_fd: "test_file" } }, { close_file: { input_fd: "test_file" } } ] }, repeat: 10 }, global { absolute_path: "/data/local/tmp/"; } ``` See more examples in `example/`. # Predefined list of instructions ## `open_file` Opens the file with a file path or a shared variable name pointing to a file path. If neither of those are provided, a random name consisting of 9 random digits is generated. Optionally saves the file descriptor which can then be used by subsequent instructions. Also, can optionally create the file if it does not already exist. ### Arguments: - (optional) `string path_name` - Specifies the file path.
OR
`string input` - Shared variable name pointing to a file path. - (optional) `string output_fd` - Shared variable name to which output file descriptor should be saved. - (optional) `bool create` (`default = true`) - Specifies if the file should be created if it does not already exist. If the file exists, nothing happens. ## `delete_file` Deletes the file with a file path or a shared variable name pointing to a file path. Uses `unlink(2)`. ### Arguments: - `string path_name` - Specifies the file path.
OR
`string input` - Shared variable name pointing to a file path. ## `close_file` Closes the file with the provided file descriptor. Uses `close(2)`. ### Arguments: - `string input_fd` - Shared variable name pointing to a file descriptor. ## `resize_file` Resizes the file with the provided file descriptor and new size. If the provided size is greater than the current file size, `fallocate(2)` is used, while `ftruncate(2)` is used if the provided size is not greater than the current file size. ### Arguments: - `string input_fd` - Shared variable name pointing to a file descriptor. - `int64 size` - New file size (in bytes). ## `resize_file_random` Resizes the file with the provided file descriptor and a range for new size. New file size is randomly generated in the provided range and if the generated size is greater than the current file size, `fallocate(2)` is used, while `ftruncate(2)` is used if the generated size is not greater than the current file size. ### Arguments: - `string input_fd` - Shared variable name pointing to a file descriptor. - `int64 min` - Minimum value (in bytes) - `int64 max` - Maximum value (in bytes) - (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, current system time is used as the seed. - (optional) `Reseeding reseeding` (`default = ONCE`). How often the random number generator should be reseeded with the provided (or generated) seed. Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. ## `write_file` Writes to file with the provided file descriptor. For `SEQUENTIAL` access, the blocks of data will be written sequentially and if the end of the file is reached, new blocks will start from the beginning of the file. For `RANDOM` access, the block offset, to which data should be written, will be randomly chosen with uniform distribution. `10101010` byte is used for the write operation to fill the memory with alternating ones and zeroes. Uses `pwrite64(2)`. ### Arguments: - `string input_fd` - Shared variable name pointing to a file descriptor. - (optional) `int64 size` (`default = -1`) - How much data (in bytes) should be written in total. If it is set to `-1`, then file size is used. - (optional) `int64 block_size` (`default = 4096`) - How much data (in bytes) should be written at once. If it is set to `-1`, then file size is used. - (optional) `int64 starting_offset` (`default = 0`) - If `access_order` is set to `SEQUENTIAL`, then the blocks, to which the data should be written, will start from this starting offset (in bytes). - (optional) `Order access_order` (`default = SEQUENTIAL`) - Order of the write. Options: `SEQUENTIAL` and `RANDOM`. - (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, current system time is used as the seed. - (optional) `bool fsync` (`default = false`) - If set, `fsync(2)` will be called after the execution of all write operations. - (optional) `Reseeding reseeding` (`default = ONCE`) - How often the random number generator should be reseeded with the provided (or generated) seed. Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. ## `read_file` Reads from file with the provided file descriptor. For `SEQUENTIAL` access, the blocks of data will be read sequentially and if the end of the file is reached, new blocks will start from the beginning of the file. For `RANDOM` access, the block offset, from which data should be read, will be randomly chosen with uniform distribution. Calls `posix_fadvise(2)` before the read operations. Uses `pread64(2)`. ### Arguments: - `string input_fd` - Shared variable name pointing to a file descriptor. - (optional) `int64 size` (`default = -1`) - How much data (in bytes) should be read in total. If it is set to `-1`, then file size is used. - (optional) `int64 block_size` (`default = 4096`) - How much data (in bytes) should be read at once. If it is set to `-1`, then file size is used. - (optional) `int64 starting_offset` (`default = 0`) - If `access_order` is set to `SEQUENTIAL`, then the blocks, from which the data should be read, will start from this starting offset (in bytes). - (optional) `Order access_order` (`default = SEQUENTIAL`) - Order of the read. Options: `SEQUENTIAL` and `RANDOM`. - (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, current system time is used as the seed. - (optional) `ReadFAdvise fadvise` (`default = AUTOMATIC`) - Sets the argument for the `posix_fadvise(2)` operation. Options: `AUTOMATIC`, `NORMAL`, `SEQUENTIAL` and `RANDOM`. If `AUTOMATIC` is set, then `POSIX_FADV_SEQUENTIAL` or `POSIX_FADV_RANDOM` will be used for `SEQUENTIAL` and `RANDOM` access order respectively. - (optional) `Reseeding reseeding` (`default = ONCE`) - How often the random number generator should be reseeded with the provided (or generated) seed. Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. ## `read_directory` Reads file names from a directory and stores them as a list in a shared variable. Uses `readdir(3)`. ### Arguments: - `string directory_name` - Name of the directory - `string output` - Shared variable name to which files names should be saved. ## `invalidate_cache` Drops kernel clean caches, including, dentry, inode and page caches by calling sync() first and then writing `3` to `/proc/sys/vm/drop_caches`. No arguments. # Dependencies ## Android The project is currently being developed as part of the Android Open Source Project (AOSP) and is supposed to run out-of-the-box. ## Linux The following utilities are required to build the project on Linux: ``` sudo apt install cmake protobuf-compiler ``` # Testing ## Linux A suite of unit tests is provided in the test/ directory. In Linux these tests can be run with the following commands: ``` mkdir build cd build make cd test ctest ``` # Use cases ## File operations performance (high priority) Bandwidth and measurement when dealing with few huge files or many small files. Operations are combinations of sequential/random-offset read/write. Latency in creating/deleting files/folders. These operations should be able to be triggered in a multiprogrammed fashion. ## Display pipeline A graph of processes that are communicating with each others in a pipeline of operations that are parallely contributing to the generation of display frames. ## Scheduling (low priority) Spawning tasks (period, duration, deadline) and verifying their scheduling latency and deadline misses count. # Workflow example and implementation nits In the following scenario, two threads are running. T1 runs the following operations: Read, Write, Read, sends a request to T2 and waits for the reply, then Write, Read. T2 waits for a request, then Read, Write, then sends the reply to the requester. Operations are encoded as primitives expressed with ProtoBuffers. The definition of dependencies among threads can be represented as graphs. ## Initialization phase The first graph traversal is performed at initialization time, when all the ProtoBuffer configuration files are distributed among all the binaries so that they can perform all the heavy initialization duties. ## Execution phase After the initialization phase completes the graph can be traversed again to put all the workloads in execution. ## Results gathering phase A final graph traversal can be performed to fetch all the measurements that each entity internally stored. ## Post processing All the measurements must be ordered and later processed to provide useful information to the user. T1: INIT : [ RD WR RD ] SND RCV [ WR RD ] : END T2: INIT : RCV [ RD WR ] SND : END # Scratch notes critical path [ READ WRITE READ ] [ READ WRITE ] [ WRITE READ ] --------------------> > < Thread1 III-XXXXXX|X-SSSSSS-XX-TTTT Thread2 III-XXXX|XXX-TTTT ^ > XXXXXXX XX< XXXX READ WRITE READ ---> vector {read(), write(), read()}; -> start() RECEIVE READ WRITE READ SEND ---> vector {receive(), read(), write(), read(), send()}; start() lock on receive()