Killing builds after timeout doesn't work reliably

[kpcyrd]

Especially on the debian rebuilder builds get frequently stuck during tests and the kill-after-timeout doesn't seem to work reliably.

The code that configures the timeout:

rebuilderd/worker/src/rebuild.rs

Lines 186 to 193 in 9b58682

	let opts = proc::Options {
	timeout: Duration::from_secs(timeout),
	size_limit: ctx.build.max_bytes,
	kill_at_size_limit: false,
	passthrough: !ctx.build.silent,
	envs,
	};
	let (_success, log) = proc::run(bin.as_ref(), &[input_path], opts).await?;

The code that's supposed to kill the process:

rebuilderd/worker/src/proc.rs

Lines 90 to 114 in 9b58682

	pub async fn next_wakeup(&mut self, child: &mut Child) -> Result<Duration> {
	// check if we need to SIGKILL due to SIGTERM timeout
	if let Some(sigterm_sent) = self.sigterm_sent {
	if sigterm_sent.elapsed() > Duration::from_secs(SIGKILL_DELAY) {
	if let Some(pid) = child.id() {
	warn!("child(pid={}) didn't terminate {}s after SIGTERM, sending SIGKILL", pid, SIGKILL_DELAY);
	// child.id is going to return None after this
	child.kill().await?;
	}
	}
	}
	// check if the process timed out and we need to SIGTERM
	if let Some(remaining) = self.timeout.checked_sub(self.start.elapsed()) {
	return Ok(remaining);
	} else if self.sigterm_sent.is_none() {
	// the process has timed out, sending SIGTERM
	warn!("child timed out, killing...");
	self.truncate(child, "TRUNCATED DUE TO TIMEOUT", true).await?;
	}
	// if we don't need any timeouts anymore we just return any value
	Ok(Duration::from_secs(SIGKILL_DELAY))
	}
	}

This needs investigation.

[kpcyrd]

#111 seems to improve the situation significantly. Shoutout to @stoeckmann who helped me getting this to work.