Isolating cores in Nomad

[caiodelgadonew]

Nomad version

Nomad v1.7.7
BuildDate 2024-04-16T19:26:43Z
Revision 0f34c85ee63f6472bd2db1e2487611f4b176c70c

Operating system and Environment details

AlmaLinux release 9.3 (Shamrock Pampas Cat)
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"

Issue

Nomad shouldn't override taskset defined inside binaries in raw_exec even when the reserved cores are configured in the client stanza

Reproduction steps

Client configuration

client {
  reserved {
    cores = "1"
  }
}

In the raw_exec driver a C++ binary is run, this binary spawns multiple threads and one of them has the affinity defined by the binary itself using the sched_setaffinity

eg.:

#include <thread>
#include <sched.h>
void pin_cpu(int core) {
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(core, &cpuset);
    if (sched_setaffinity(0, sizeof(cpuset), &cpuset) < 0) {
    } else {
        std::cout << "Success!" << std::endl;
    }
}

But if we have this configuration on Nomad client, when we run the binary it overwrites the taskset leaving the thread to run only on non-reserved cores.

Expected Result

Nomad does not taskset on threads that has specific affinity in the binary code.

Actual Result

Nomad overwrites the affinity of the thread.

EXTRA DETAILS

I've experiment the core isolation feature over the following settings:

• Allocating CPU resources over reserving a whole core
• Client reserved cores
• Job CPU Resource
• CPU Environment Variables

1. Client config cores = "0,2-7" & NOMAD_CPU_CORES="0-7"
   

2. Client config cores = "1" & Job NOMAD_CPU_CORES="0-7" & resources.cores = 1
   

3. No specific Client config & Job NOMAD_CPU_CORES="0,2-7" ( THIS makes me think the NOMAD_CPU_CORES is not working as intended)