UI support for Variable Locks

[schmichael]

Proposal

Noamd 1.7.0 added Locks to Variables which are useful for a variety of use cases: workloads can implement leader election by racing to acquire a shared lock, workloads can ensure only a single instance is updating data at one time, etc.

Observing Locks

Unfortunately distributed locking is one on the most notoriously fragile components in distributed systems. Locking, as implemented, always comes down to clocks. And clocks mean "magical" state changes in the absence of an observable event (eg a TTL was hit and a lock was released without the lock holder realizing it).

Updating Locks

Not only would being able to observe locks ease debugging of a complex component, but there are many times when it's useful for a human operator to manually interact with locks.

Imagine a user deploying a workload with many instances (count > 1) that initializes an S3 bucket on first run if it doesn't exist already. This is an excellent use case for locks: the workload would use the Nomad API to try to acquire a single lock. Only one instance would "win" and therefore be able to initialize the S3 bucket without worrying about another instance overwriting it. The other instances would block until the lock is released, then all instances would see the bucket has been initialized and go about with their work.

The human operator deploying this job for the first time may want to be extra careful and ensure it was deployed successfully before allowing it to initialize S3.

Once again locks are useful! The human operator can acquire the lock via nomad var lock ... before running the job. The job will run and all instances will block before initializing S3 because the lock is already acquired. The human operator is then free to perform manual inspections of the workloads to gain confidence they will work as intended when allowed to continue. Once the operator is confident they release the lock, and the workloads race to acquire it.

Resources

Locks are implemented as additional operations and metadata on top of the Variables CLI/API. Any locking UI can build upon the existing Variable UI.

• API Docs: https://developer.hashicorp.com/nomad/api-docs/variables/locks
• CLI Docs: https://developer.hashicorp.com/nomad/docs/commands/var/lock
• Announcement: https://www.hashicorp.com/blog/nomad-1-7-improves-vault-and-consul-integrations-adds-numa-support#distributed-locks
• Internal RFC: NMD-178