[LibOS] Emulate madvise(MADV_DONTNEED) on anonymous mappings as uncommitting pages

[kailun-qin]

Description of the feature

Currently in Gramine, madvise(MADV_DONTNEED) on anonymous mappings is emulated as "zeroing the pages":

gramine/libos/src/bookkeep/libos_vma.c

Line 1336 in 1cf1f46

memset((void*)zero_start, 0, zero_end - zero_start);

However, the manpage of madivse() states that: "after a successful MADV_DONTNEED operation, the semantics of memory access in the specified region are changed: subsequent accesses of pages in the range will succeed, but will result in ... zero-fill-on-demand pages for anonymous private mappings".

Note that Gramine's current way of implementation is due to "zero-fill-on-demand" would imply allocating on #PF, which is not implemented in Gramine at all yet, pls see #1692 (review) for detailed discussions.

W/ the pending effort of #1513, on machines w/ EDMM support, we should be able to do this on-demand allocation by first "uncommitting" the anonymous mapping pages (i.e. moving them back to "lazy alloc" state) instead of zeroing them on madvise(MADV_DONTNEED); and the subsequent accesses of pages in the range would result in lazy allocations (zero-filled) on #PF -- this follows what Linux does.

However, this may be impossible w/ the current implementation. We're holding the non-reentrant/recursive vma_tree_lock when we're in the madvise_dontneed_visitor() callback (which is invoked during VMA traversing):

gramine/libos/src/bookkeep/libos_vma.c

Line 1310 in 1cf1f46

static bool madvise_dontneed_visitor(struct libos_vma* vma, void* visitor_arg) {

And if we dynamically remove/uncommit a page on madvise(MADV_DONTNEED) in the callback, #PF can happen at any time when accessing the uncommitted pages -- our lazy allocation logic would then also try to acquire the same lock for VMA lookup in g_mem_bkeep_get_vma_info_upcall (see pal_mem_bkeep_get_vma_info() for details).

We thus propose to emulate madvise(MADV_DONTNEED) as "uncommitting pages" and address the related issues in a separte PR, see #1513 (review) for details.

Why Gramine should implement it?

This follows what Linux's semantics of madvise(MADV_DONTNEED) and may benefit some applications for saving enclave pages.

[dimakuv]

@kailun-qin Doesn't #1513 fix this issue? If yes, please add to that PR a line with Fixes #...

[kailun-qin]

Doesn't #1513 fix this issue? If yes, please add to that PR a line with Fixes #...

Yes. Done, thanks!