WIP [LibOS] Move trusted and allowed files logic to LibOS

[dimakuv]

Description of the changes

This is a draft that has all code that I wrote while working on the main target: moving trusted and allowed files logic from Linux-SGX PAL to LibOS. There are several unrelated changes that I detected while working on the main target; they will be separated into separate PR.

I'm submitting this commit as a draft PR just to have a back up and a convenient way to extract sub-PRs out of this huge pile of code.

TODOs:

• Introduce a new manifest option with SHA256 hash of the LibOS executable, so that Linux-SGX PAL reads this one instead of sgx.trusted_files. Then the LibOS hash can be removed from sgx.trusted_files completely as LibOS itself doesn't need it.
• REMOVE ALL LINUX-SGX PAL CODE FOR TRUSTED/ALLOWED FILES + FILE_CHECK_POLICY [EVIL LAUGH]!

How to test this PR?

All current tests must pass. Should be enough of testing.

---

This change is 

[dimakuv]

Reviewable status: 0 of 23 files reviewed, 6 unresolved discussions, not enough approvals from maintainers (2 more required), not enough approvals from different teams (1 more required, approved so far: Intel), "WIP" found in commit messages' one-liners

a discussion (no related file):
One more TODO:

• Remove PAL_OPTION_PASSTHROUGH, it becomes redundant.

---

---

libos/src/libos_init.c line 401 at r1 (raw file):

    RUN_INIT(init_vma);
    RUN_INIT(init_r_debug);

This is an independent fix. Submitted as a separate PR: #1814

---

libos/src/bookkeep/libos_vma.c line 1498 at r1 (raw file):

}

static bool vma_filter_needs_prot_refresh(struct libos_vma* vma, void* arg) {

This code (among a lot of other code) is extracted into this PR: #1818

---

libos/src/fs/shm/fs.c line 168 at r1 (raw file):

}

static int shm_unlink(struct libos_dentry* dent) {

This is an unrelated change (well, slightly related because with this PR, shm filesystem definitely cannot reuse chroot_unlink() as the latter uses chroot_temp_open() which checks for trusted/allowed files).

Create a separate PR for this: #1815

---

libos/test/ltp/manifest.template line 20 at r1 (raw file):

  # many LTP multi-process tests rely on shared-memory IPC via `mmap(MAP_SHARED, </dev/shm fd>)`
  { type = "untrusted_shm", path = "/dev/shm", uri = "dev:/dev/shm" },

This particular change is extracted into PR:

---

libos/test/regression/test_libos.py line 872 at r1 (raw file):

        self.assertIn('TEST OK', stdout)

    @unittest.skip('sigaltstack isn\'t correctly implemented')

This re-enabling of the sigaltstack test was extracted into this PR: #1819

[dimakuv]

Reviewable status: 0 of 23 files reviewed, 7 unresolved discussions, not enough approvals from maintainers (2 more required), not enough approvals from different teams (1 more required, approved so far: Intel), "WIP" found in commit messages' one-liners

---

python/gramine-manifest line 33 at r1 (raw file):

    template = infile.read() if infile else string
    manifest = Manifest.from_template(template, define)
    manifest.expand_all_trusted_files(chroot=chroot)

Now that gramine-direct also works with Allowed and Trusted Files, we need to call this expansion function.

There is one side-effect: now both .manifest and .manifest.sgx files have exactly the same contents (previously they only differed in not-expanded vs expanded Trusted Files). This eats memory on the hard disk, so we could symlink .manifest.sgx -> .manifest. But on the other hand, users will most typically disregard .manifest file anyway and use only .manifest.sgx, so making the latter a symlink sounds wrong.

[dimakuv]

Reviewable status: 0 of 45 files reviewed, 6 unresolved discussions, not enough approvals from maintainers (2 more required), not enough approvals from different teams (1 more required, approved so far: Intel), "WIP" found in commit messages' one-liners

---

libos/test/regression/test_libos.py line 872 at r1 (raw file):

Previously, dimakuv (Dmitrii Kuvaiskii) wrote…

This re-enabling of the sigaltstack test was extracted into this PR: #1819

Update: I removed this change from here, since something is broken on EDMM. Let's deal with that in the separate PR, here I'm reverting sigaltstack test to be skipped again.

[dimakuv]

Reviewable status: 0 of 45 files reviewed, 7 unresolved discussions, not enough approvals from maintainers (2 more required), not enough approvals from different teams (1 more required, approved so far: Intel), "WIP" found in commit messages' one-liners

---

pal/include/pal_internal.h line 296 at r2 (raw file):

int _PalDebugLog(const void* buf, size_t size);

int _PalValidateEntrypoint(const void* buf, size_t size);

This is extracted into separate PR: #1820