Question about EPC size

[ila]

Hi, I have a quick question - using is-sgx-available I see that my EPC size is 16GB, however in my BIOS I specified 64GB. This makes sense since my machine has 4 NUMA regions, so EPC is region-wise. However, if I run Gramine, will my available EPC size be 16GB or 64GB? If I pin Gramine to one NUMA core, will it use all the 64GB?

In other words - if I want Gramine to use 64GB of EPC, should I specify 256GB in the BIOS? Or 64GB?

Also would appreciate a lot any kind of documentation on this - I find it a bit confusing that Gramine does not explain the difference between enclave size and EPC size, or that the tool is-sgx-available shows per-region statistics.

[mkow]

Hmm, no idea, but I'd expect all EPC regions to be shared between CPUs, because enclave threads may be scheduled on a core from any CPU package (in case of no pinning).
@ScottR-Intel may be able to provide a more detailed answer.

Also would appreciate a lot any kind of documentation on this

Yeah, someday hopefully we'll find time for that :) We started once, but didn't finish it: https://gramine.readthedocs.io/en/latest/sgx-intro.html#term-EPC.

I find it a bit confusing that Gramine does not explain the difference between enclave size and EPC size

Not sure if that was a statement and you already know what the difference is, or maybe a question :) I'll answer anyways: EPC is only a cache, it's possible to have enclaves much larger than the EPC size, but if the working set exceeds EPC size then it will cause terribly slow swapping to non-EPC RAM.

or that the tool is-sgx-available shows per-region statistics.

That sounds like a bug. We calculate it by summing all EPC regions reported by cpuid, which I guess shows only regions local to the CPU currently executing is-sgx-available? Fixing this may be a bit painful, but should be doable.

[ila]

Yes sorry, what I meant is - I think Gramine is a starting point for people who aren't that knowledgeable about SGX internals (like me) and want to port applications, so these things sometimes are not immediate :)

Thanks a lot for the input!

[mkow]

I did some update to the docs, please check: #1827. It's still not complete, but at least better than before.

[ila]

Great, thank you so much!

[ScottR-Intel]

The easiest way to see all the EPC ranges is by looking at the dmesg log - the in-kernel driver prints each range out:

sudo dmesg | grep -i sgx

The cpuid tool can be used, also, because that's how the in-kernel driver gets them the EPC ranges, as does is-sgx-available.

[mkow]

@ScottR-Intel: Will an enclave running on a single CPU have access to EPC from other nodes as well? Or will it start swapping to RAM after exhausting its local EPC? (I think that was the original question @ila had)

[ScottR-Intel]

Ah... Yes, all ranges are available to all cores/sockets. There is some basic NUMA support in the upstreamed in-kernel driver where all requested enclave memory is placed inside a single EPC range, if possible. If not, it has to spread it out, which of course, incurs some performance penalty.

[mkow]

or that the tool is-sgx-available shows per-region statistics.

That sounds like a bug. We calculate it by summing all EPC regions reported by cpuid, which I guess shows only regions local to the CPU currently executing is-sgx-available? Fixing this may be a bit painful, but should be doable.

So, I was discussing this with @mwkmwkmwk and my answer actually doesn't make sense, all CPUs have to be aware of all EPC mappings in all NUMA regions, and this is indeed the case on a server with NUMA we have.

@ila: Could you double check your numbers? Is it really 64GB in the BIOS, but is-sgx-available reporting 16GB? We don't have any theory right now how this would be possible, except a bug in the BIOS... The calculation code in is-sgx-available is quite simple and based directly on cpuid output:

gramine/tools/sgx/is-sgx-available/is_sgx_available.cpp

Lines 130 to 145 in 4dae6b0

	// Check if there's any EPC region allocated by BIOS
	for (uint32_t subleaf = 2; subleaf >= 2; subleaf++) {
	uint32_t eax, ebx, ecx, edx;
	cpuid(0x12, subleaf, &eax, &ebx, &ecx, &edx);
	auto type = eax & 0xF;
	if (!type)
	break;
	if (type == 1) {
	// EAX and EBX report the physical address of the base of the EPC region,
	// but we only care about the EPC size
	if (ecx & 0xFFFFF000 || edx & 0xFFFFF) {
	epc_region_size_ += ecx & 0xFFFFF000;
	epc_region_size_ += (uint64_t)(edx & 0xFFFFF) << 32;
	}
	}
	}