Direct I/O with Gramine

[ila]

Hi, we are benchmarking a database system on SGX, where our workload is primarily to read from disk-based (encrypted) files.

We know that there is no caching of computations output, either by the application or the OS. Only the input files are cached, in case of buffered I/O. Therefore, as a way to reduce the overhead of page swapping (incurred running many different huge queries consecutively), we decided to try direct I/O.

However, both gramine-direct and gramine-sgx do not seem to be affected by direct IO. According to the documentation, O_DIRECT is not supported.

Does this mean that there is no way in Gramine to perform direct I/O?

Is there a way to implement this syscall, or to forcefully avoid caching of I/O?

Is directed I/O a safe operation to perform inside SGX, what is the reasoning behind ignoring this flag?

Thanks in advance!

[dimakuv]

You are correct. This flag is ignored by Gramine: https://gramine.readthedocs.io/en/stable/devel/features.html#file-system-operations.

(By the way, ignoring the O_DIRECT flag is considered a legit behavior. O_DIRECT is more like a hint, but its exact semantics depend on a number of factors. See here: https://stackoverflow.com/questions/41257656/what-does-o-direct-really-mean)

This means that the host Linux will not receive this flag during opening of the file.

Is directed I/O a safe operation to perform inside SGX, what is the reasoning behind ignoring this flag?

I think there is no reasoning behind ignoring this flag. I don't remember anyone ever doing such an experiment and asking us this question. In other words: by default Gramine forbids/ignores most flags/features, just to be on the safe side. When a customer comes and asks for a particular flag/feature, then we consider implementing it. So the reason is purely "because nobody cared before".

Now that you raise this question, it seems to me that simply forwarding O_DIRECT from the app through Gramine and to the host OS is a security-unrelated choice. In the best (benign) case, the host OS will honor this flag and you'll see performance improvement. In the worst (malicious) case, the host OS will ignore this flag and you'll see bad performance. The latter case is a variant of Denial of Service, which is not in the threat model of Intel SGX and Gramine. So if I'm not mistaken, implementing O_DIRECT is simple and safe.

I will educate myself a bit on O_DIRECT and its consequences, and if all is well, I'll submit a PR to add this flag.

[ila]

Thanks a lot for this very insighful explaination :)

[dimakuv]

@ila Quick question: according to your experiments, what is the performance improvement when using O_DIRECT on your original (native) database application? Just curious to know how beneficial this feature is.

[ila]

In our native database application, running outside SGX, I am not sure it makes too much sense to evaluate this kind of behaviour - usually, database applications highly benefit from data being in cache, and buffered IO is the preferred solution. In general, I'd say the IO time is slightly slower (due to having to read blocks of fixed size, so allocating more memory and copying if the data does not align). However, in SGX, in huge workloads, a lot of page swapping occurs - the OS filesystem will cache pages that the enclave is not currently using, and the enclave will fetch them from there. This creates a lot of overhead since the pages need to be encrypted and decrypted and page faults cause CPU interrupts. We are confident IO_DIRECT is going to fix these issues - apparently the overhead of I/O is still less significant than the constant page swapping. Let me know if you have any more questions :)

just to add a bit more context - we are experiencing up to 50x slowdown of queries (60GB dataset) if we run them consecutively in an enclave, compared to running them separately in a newly-initialized enclave. We think all of this is due to page swapping - tuning parameters such as the enclave size and the available DB memory help, but still 2-3x slowdown compared to the "optimal" performance

[dimakuv]

@ila Can you explain a bit more about your scenario:

1. Are you using fs.mounts = { type="encrypted" } for your DB files that you want to apply O_DIRECT on?
2. Have you tried changing from Encrypted files to Allowed files, i.e. declaring DB files as sgx.allowed_files? Do you still observe high overhead in that case?

Update from my side: I hacked a Gramine version that applies O_DIRECT on allowed files. However, it is not currently possible to apply this to encrypted files, because encrypted files' read/write logic is more complicated and uses bounce buffers. And O_DIRECT has this specific requirement to allocate memory in file-block-aligned memory regions (which would require significant changes to Gramine codebase, and is also somehow brittle by itself).

Reading more on O_DIRECT, this flag/feature is really controversial... I'm now leaning towards not having it in Gramine, unless there are significant workloads and benchmarks that prove otherwise.

I continue reading about O_DIRECT and whether it can be applied at the level of the host OS (so we can toggle it on Linux and leave Gramine unmodified). I'll update my reply when I'm done.

[mkow]

hovwever, the database files (ie the ones containing the data itself) are in allowed files indeed - our reasoning behind this is that if these files are encrypted to begin with, it is "safe" for the application to access them, as it needs a secret key to open and modify them. Mind you, the encryption I am mentioning is entirely application based (in short, we take Parquet files and we encrypt block by block with AES). This with io_direct has been tested outside Gramine and seems to be working fine.

Sorry for a slight off-topic, but a warning here: this is most likely insecure and it's possible to modify the database from the host without the app noticing. It depends on the encryption you're using, but most likely your scheme is similar to the disk encryption ones, which are only protecting against data leaking, but not against an active attacker who can modify the data. You'd need to either reencrypt the whole file on each modification or use a merkle tree for a more localized integrity checks. What encryption scheme are you using exactly?

[ila]

We are using either AES-GCM from the OpenSSL library or the same from mbedtls (currently testing different methods - if you have any suggestions, we are open to them - we are trying to exploit Intel-based hardware instructions in Gramine). Our key can be a variable size. We are aware that data modifications would need a re-encryption of the whole database file, therefore right now our workloads are read-only (which is acceptable for analytical workloads). The data is only unencrypted when it resides in the enclave memory. If edits would be possible, we would need to 1) decrypt the whole database inside the enclave, 2) apply modifications, and 3) re-encrypt it before writing on disk. We are still working on this part, so if there are security issues here, we are happy to address them in the future.

[dimakuv]

how come the encrypted file logic is more difficult to handle? Please pardon my ignorance, but I thought that encryption/decryption happens at a higher level than read/write operations, so what do you think is the issue here?

Since you're using sgx.allowed_files for your database, the discussion becomes redundant. But just to satisfy your curiosity:

1. At least on my current Linux with ext4 filesystem, a write() from a non-aligned memory address returns with -EINVAL (invalid value error). This means that the application must allocate buffers in a special way (e.g. using posix_memalign()).
2. Recall that Gramine's Encrypted Files are transparently encrypted and decrypted inside the SGX enclave. This necessitates a "bounce buffer" -- a temporary buffer that sits in-between the user buffer (the buffer in your program) and the host-OS buffer. Given requirement above, this temporary buffer must be allocated with a correct alignment.
3. Gramine has its own memory allocator (slab allocator) that doesn't have an alignment specifier. So we would need to modify our memory allocator to satisfy the requirement above. This is extremely painful.
4. I am unclear what are all the alignment requirements -- it looks like every OS and every FS implement their own alignment requirements. So how should Gramine choose the default alignment? How can Gramine be tuned to set an alignment for a particular program? It just gets annoying with all this "O_DIRECT is not standardized".

How come the feature is controversial? I totally understand if it would require huge changes to the codebase, I am more interested about the security/performance/stability issues here

Well, O_DIRECT was never standardized from what I know. Some FSes support it, some silently ignore it, some try to support but silently fall back to buffered I/O, some have specific requirements and fail if the requirements are not fullfilled, etc. Check these (old but seem to hold true):

• https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO%27s_Semantics
• https://yarchive.net/comp/linux/o_direct.html

[mkow]

We are using either AES-GCM from the OpenSSL library or the same from mbedtls (currently testing different methods - if you have any suggestions, we are open to them - we are trying to exploit Intel-based hardware instructions in Gramine).

AES is of course good, it's mostly about the higher-level scheme you're using (and these mostly aren't directly implemented in hardware). In this case it's GCM, which is good as long as you don't want partial modifications to the file.

Our key can be a variable size. We are aware that data modifications would need a re-encryption of the whole database file, therefore right now our workloads are read-only (which is acceptable for analytical workloads). The data is only unencrypted when it resides in the enclave memory.

Ok, then it sounds good :)

If edits would be possible, we would need to 1) decrypt the whole database inside the enclave, 2) apply modifications, and 3) re-encrypt it before writing on disk. We are still working on this part, so if there are security issues here, we are happy to address them in the future.

This design is good (except performance, as you need to re-encrypt the whole database), just remember that it will still be possible to rollback the whole database to an older version (the untrusted host can save an older encrypted database and overwrite a newer state with it). But this is fine in most cases, as long as the rollback can only be done to the whole database, not e.g. only to a specific encryption block.

[ila]

Thank you so much for this explaination! It is very interesting for us to know about the potential performance and security issues (trying to write a paper about analytical workloads on SGX), so all this information is very precious to extend our knowledge and implementative choices.

[ila]

Edit: your patch compiles and builds no problem, however I am not seeing any performance difference than we had before. The steps I did are:

• uninstalled Gramine, cloned from source
• applied your changes (4 files)
• built from scratch using the instructions in the documentation
• cleaned our Gramine-generated manifest and built it again
• tested with gramine-direct

It could also be that any of these steps somehow did not work, but it is hard to tell since I did test in between steps that the gramine executable did not exist. I am using the upstream SGX drivers (already preinstalled in our Intel machine, Intel Xeon Platinum 8352Y).

To give you an example, this is the query we are running:
Running query: SELECT UserID FROM read_parquet('duckdb-parquet-encrypted/hits_encrypted.parquet', encryption_config={footer_key: 'key128'}, force_direct_io=true);
1st run: 2.82s
2nd run: 1.78s
3rd run: 1.79s

The fact that there is a difference between the 1st and the successive runs makes me guess that the IO_DIRECT flag was still ignored. Running the very same query outside Gramine gives us 3 equal results, so we don't think it's an issue on your side.

Just to clarify, the expected behaviour is 3 equal results for 3 runs, which imply that nothing is being cached, therefore our IO is not buffered. To give you a reference, this is how we are implementing IO_DIRECT in our application:
duckdb/duckdb@a358dfa
duckdb/duckdb@bf93e1c

Edit: I also tried the very same workload without file encryption, and the behaviour is still the same (1st run slower than 2nd and 3rd). So, we can rule out application-based encryption as the culprit).

I have taken a look at the file you sent, how do you make sure the behaviour is indeed correct? Should it give a different output on gramine-direct compiled with and without the IO_DIRECT hack? We are purely comparing performance here, so it could also be that the IO_DIRECT works and something else is being cached for some reason (but it seems unlikely, as this behaviour is very reproducible). If you need some example to reproduce, I can provide that as well. Have a good day :)

Another edit: this is our manifest file if needed: https://github.com/cwida/ClickBench-SGX/blob/main/benchmark.manifest.template
You can check the README and the correspondent folders to have an idea of what we are doing, although for full reproducibility a few extra steps are needed (wouldn't recommend, these are going to take some time, but can help if needed).

[ila]

Update - implemented and tested it, now the strace logs for my application inside and outside Gramine are the same, for example:

strace gramine-direct benchmark duckdb/profile_duckdb.py 2>&1 | grep O_SYNC
open("./duckdb-parquet-encrypted/hits.parquet", O_RDONLY|O_SYNC|O_DIRECT|O_CLOEXEC) = 297
open("./duckdb-parquet-encrypted/hits.parquet", O_RDONLY|O_SYNC|O_DIRECT|O_CLOEXEC) = 317
open("./duckdb-parquet-encrypted/hits.parquet", O_RDONLY|O_SYNC|O_DIRECT|O_CLOEXEC) = 280

However, performance is still unaffected. So the issue here could be:

1. either the metadata files are buffered and that causes a different performance, although I doubt that only that causes such a big improvement (1s performance difference for 14GB is quite a bit)
2. Gramine (or the OS) is caching something that we are unaware of
3. the implementation of these syscalls is problematic (or maybe our implementation is problematic)

I'll look more into it, maybe my advisor has some ideas. If you have any other suggestions, I am open :) in the meantime, again, many thanks for the help!

[dimakuv]

@ila Have you tried gramine-direct? If you see significant overhead in gramine-direct, then please debug/profile this first.

[ila]

Yes, all these logs are using gramine-direct.

Update: I actually tested with and without direct io, and it looks like the direct io is actually being applied - however there is still a bit of "noise" (200-300ms of difference between cold and hot runs), so I was wondering, is Gramine caching anything (regardless of the application) during its execution?

Also, I now tested your code with gramine-sgx (only the patch you posted here, without my modifications) and it doesn't seem to work. The error it gives me is:
Error IO Error: Could not read from file "duckdb-parquet-encrypted/hits.parquet": Invalid argument

The very same thing with gramine-direct works no problem. Did you by any chance test with SGX? I feel like there might be something missing in the code.

[dimakuv]

@ila I did test with gramine-sgx too, and it worked. At least on my dummy example.

But this feels like a problem of alignments (O_DIRECT on ext4 requires 512B alignment, iiuc). I got the same error when I used malloc() (without alignment guarantees) vs posix_memalign() (with specific alignment guarantees).

I can imagine that my patch doesn't account for some corner case, with e.g. much bigger workloads and memory regions.

Anyway, it feels like the problem is not in O_DIRECT support, but somewhere else. Could you continue your performance analysis? For example, you can do perf record on gramine-direct (built in debug_optimized mode). You can find additional documentation on this here: https://gramine.readthedocs.io/en/stable/performance.html#profiling-with-perf. Of course, start with gramine-direct since already there you observe problems.

[ila]

Hi, just a quick update - we decided to avoid hacky stuff and try moving the application part dealing with I/O outside the enclave. This should help paging as well. Will keep you posted :)