kola/rpm-ostree/replace-rt-kernel: do not hardcode kernel versions #1309

jmarrero · 2023-06-07T17:36:51Z

openshift-ci · 2023-06-07T17:36:56Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

cgwalters · 2023-06-07T17:52:41Z

Continuing this thread #1306 (comment) on this PR:

Agree eventually we should stop parsing HTML for the previous kernel test but doing that is going to require handling the "repoquery" stuff either in rpm-ostree or temporarily layering dnf or something else

As far as the --from thing for the kernel-rt override, I think today that that is actually a no-op in this case and we'll just use the latest from whatever repo - we're not repository-locking. It seems actually a bit more confusing to me to provide it because it won't do anything.

tests/kola/rpm-ostree/replace-rt-kernel

dustymabe

LGTM

dustymabe

/lgtm

openshift-ci-robot · 2023-06-08T18:33:29Z

/retest-required

Remaining retests: 0 against base HEAD 1550016 and 2 for PR HEAD 25c0b98 in total

travier

/lgtm

travier · 2023-06-09T07:00:44Z

/retest

openshift-ci-robot · 2023-06-09T10:14:07Z

/retest-required

Remaining retests: 0 against base HEAD 4874462 and 2 for PR HEAD f15d050 in total

openshift-ci-robot · 2023-06-09T19:09:00Z

/retest-required

Remaining retests: 0 against base HEAD 4874462 and 2 for PR HEAD ad4995c in total

LorbusChris · 2023-06-09T21:29:24Z

/test scos-9-build-test-qemu

openshift-ci-robot · 2023-06-10T13:28:35Z

/retest-required

Remaining retests: 0 against base HEAD c546e95 and 1 for PR HEAD ad4995c in total

LorbusChris · 2023-06-10T18:31:42Z

I tested this manually on SCOS and can't get it to fail. If this fails again might try even more memory.

I'm also thinking this is a memory issue. In the SCOS community builds we give the pods running COSA 4gb ram and 1 full CPU.

travier · 2023-06-12T10:47:22Z

Let's bump this to 4GB here for this test only, master only as we don't run SCOS on other branches: https://github.com/openshift/release/blob/master/ci-operator/config/openshift/os/openshift-os-master.yaml#L78

jmarrero · 2023-06-12T14:45:53Z

lowering to 2GB and making this RHCOS only for now. I'll add this test for SCOS to my backlog.

travier · 2023-06-12T14:54:13Z

tests/kola/rpm-ostree/replace-rt-kernel

+##   # We've seen some OOM when 1024M is used in similar tests:
+##   # https://github.com/coreos/fedora-coreos-tracker/issues/1506
+##   minMemory: 2048
+##   distros: rhcos


We don't have an scos switch here, rhcos is aliased to scos. You'll have to keep the denylist entry above but just for SCOS so c9s.

It would be nice if we could drop the denylist entry.. Maybe we use is_scos() inside the test?

If the reason it's failing is because of #1309 (comment) then we could just do the rt test on SCOS, which is still a pretty good test.

dustymabe · 2023-06-12T16:22:29Z

what failures are we seeing with SCOS? Could it be that there is nothing to override because we are now using the latest version of the kernel in c9s and it matches the already baked in kernel in the OSTree?

jmarrero · 2023-06-12T16:58:03Z

Jun 12 14:01:32 qemu0 kola-runext-replace-rt-kernel[2030]: Resolving dependencies...done
Jun 12 14:01:32 qemu0 kola-runext-replace-rt-kernel[2030]: error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.

error: Bus owner changed, aborting. is the error. If it was something with the repos I can tweak the test easily. Sadly this error I have seen it in low memory scenarios but for some reason even with 4GB I am seeing the test fail like this on SCOS.

#1315

jmarrero · 2023-06-13T15:52:53Z

/retest

LorbusChris · 2023-06-13T18:04:53Z

tests/kola/rpm-ostree/replace-rt-kernel

+runv rm -rf /etc/yum.repos.d/*
+runv curl -sSLf "${repo_url}" -o /etc/yum.repos.d/cs.repo
+runv curl -sSLf https://centos.org/keys/RPM-GPG-KEY-CentOS-Official -o /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
+runv sed -i 's|^gpgkey.*|gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official|' /etc/yum.repos.d/cs.repo


This sed invocation also replaces other unrelated keys.

Analog to https://github.com/openshift/os/blob/master/extensions/Dockerfile#L13-L17:

Suggested change

runv sed -i 's|^gpgkey.*|gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official|' /etc/yum.repos.d/cs.repo

runv sed -i 's|/usr/share/distribution-gpg-keys/centos|/etc/pki/rpm-gpg|' /etc/yum.repos.d/cs.repo

Edit: Edited for clarity.

I'm a bit confused here. This is running inside the RHCOS machine (throw away test machine). What change are you suggesting and why is it important?

To me it looks like rpm-ostree may be failing when trying to verify the repos in c9s.repo. The keys for all of them have to be available. Blanket replacing all of them with one key won't work.

hmm. but this test as written passes on RHCOS (@jmarrero, correct me if I am wrong).

According to @jmarrero it fails on SCOS though and we aren't sure why yet.

And we do have repo_gpgcheck=0 on the repos in c9s.repo. I might've pulled a false alarm here.

Correct this passes on RHCOS both on CI & locally

The problem here is that those reference distribution-gpg-keys which we don't ship in the OS by default. I think @LorbusChris 's suggestion makes sense...

But I don't understand actually why we're removing the repo files by default?

It is because this is a test for RHCOS only at the moment, I am just making sure there nothing on yum.repos.d before downloading the centos repo. When I get to the SCOS test, the plan is to just use the provided repos/keys and just turn on the rt repo if it's disabled.

see also #1316

LorbusChris · 2023-06-13T18:06:49Z

/hold

dustymabe · 2023-06-13T18:43:29Z

error: Bus owner changed, aborting. is the error.

What do the journal logs say?

LorbusChris · 2023-06-13T18:52:35Z

My previous hold was probably a false alarm, but I'm still unsure why this fails on SCOS
/hold cancel

jmarrero · 2023-06-13T19:07:03Z

I opened #1315 to investigate the SCOS failure, CI is not saving the journal for this test (unless I am looking in the wrong place, there is a link to the logs in the issue). On a SCOS VM running the test manually I could not get it to fail, so I need to build scos and run the test against it locally to see if it fails and debug further.

jmarrero · 2023-06-13T20:02:30Z

/retest

jmarrero · 2023-06-14T11:51:20Z

/test rhcos-92-build-test-qemu

openshift-ci · 2023-06-14T15:23:19Z

@jmarrero: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

cgwalters

/lgtm

openshift-ci · 2023-06-14T17:44:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, dustymabe, jmarrero, travier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters,dustymabe,jmarrero,travier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 7, 2023

jmarrero mentioned this pull request Jun 7, 2023

kola/rpm-ostree/replace-rt-kernel: do not hardcode kernel versions #1306

Closed

jmarrero marked this pull request as ready for review June 7, 2023 17:38

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2023

openshift-ci bot requested review from cgwalters and jschintag June 7, 2023 17:39

cgwalters reviewed Jun 7, 2023

View reviewed changes

tests/kola/rpm-ostree/replace-rt-kernel Outdated Show resolved Hide resolved

jmarrero force-pushed the rt-kernel-vers branch 3 times, most recently from bdb6084 to 88e659d Compare June 8, 2023 13:19

jmarrero requested a review from cgwalters June 8, 2023 13:27

cgwalters reviewed Jun 8, 2023

View reviewed changes

tests/kola/rpm-ostree/replace-rt-kernel Outdated Show resolved Hide resolved

jmarrero force-pushed the rt-kernel-vers branch from 88e659d to b428e6e Compare June 8, 2023 14:02

dustymabe reviewed Jun 8, 2023

View reviewed changes

tests/kola/rpm-ostree/replace-rt-kernel Outdated Show resolved Hide resolved

dustymabe approved these changes Jun 8, 2023

View reviewed changes

jmarrero force-pushed the rt-kernel-vers branch from b428e6e to 25c0b98 Compare June 8, 2023 15:27

dustymabe approved these changes Jun 8, 2023

View reviewed changes

openshift-ci bot assigned dustymabe Jun 8, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2023

jmarrero force-pushed the rt-kernel-vers branch from 25c0b98 to f15d050 Compare June 8, 2023 20:36

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2023

travier approved these changes Jun 9, 2023

View reviewed changes

openshift-ci bot assigned travier Jun 9, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 9, 2023

jmarrero force-pushed the rt-kernel-vers branch from ad4995c to a9ca787 Compare June 12, 2023 11:49

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 12, 2023

jmarrero force-pushed the rt-kernel-vers branch from a9ca787 to 0a2491c Compare June 12, 2023 14:44

travier reviewed Jun 12, 2023

View reviewed changes

kola/rpm-ostree/replace-rt-kernel: do not hardcode kernel versions

9f1ded5

jmarrero force-pushed the rt-kernel-vers branch from 0a2491c to 9f1ded5 Compare June 13, 2023 11:34

LorbusChris suggested changes Jun 13, 2023

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 13, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 13, 2023

cgwalters approved these changes Jun 14, 2023

View reviewed changes

openshift-ci bot assigned cgwalters Jun 14, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 14, 2023

openshift-merge-robot merged commit 1f2c0eb into openshift:master Jun 14, 2023
7 checks passed

	runv sed -i 's\|^gpgkey.*\|gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official\|' /etc/yum.repos.d/cs.repo
	runv sed -i 's\|/usr/share/distribution-gpg-keys/centos\|/etc/pki/rpm-gpg\|' /etc/yum.repos.d/cs.repo

kola/rpm-ostree/replace-rt-kernel: do not hardcode kernel versions #1309

kola/rpm-ostree/replace-rt-kernel: do not hardcode kernel versions #1309

Conversation

jmarrero commented Jun 7, 2023 • edited

openshift-ci bot commented Jun 7, 2023

cgwalters commented Jun 7, 2023

dustymabe left a comment

Choose a reason for hiding this comment

dustymabe left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Jun 8, 2023

travier left a comment

Choose a reason for hiding this comment

travier commented Jun 9, 2023

openshift-ci-robot commented Jun 9, 2023

openshift-ci-robot commented Jun 9, 2023

LorbusChris commented Jun 9, 2023

openshift-ci-robot commented Jun 10, 2023

LorbusChris commented Jun 10, 2023

travier commented Jun 12, 2023

jmarrero commented Jun 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe commented Jun 12, 2023

jmarrero commented Jun 12, 2023 • edited

jmarrero commented Jun 13, 2023

LorbusChris Jun 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LorbusChris commented Jun 13, 2023

dustymabe commented Jun 13, 2023 • edited

LorbusChris commented Jun 13, 2023

jmarrero commented Jun 13, 2023

jmarrero commented Jun 13, 2023

jmarrero commented Jun 14, 2023

openshift-ci bot commented Jun 14, 2023

cgwalters left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jun 14, 2023

jmarrero commented Jun 7, 2023 •

edited

jmarrero commented Jun 12, 2023 •

edited

LorbusChris Jun 13, 2023 •

edited

dustymabe commented Jun 13, 2023 •

edited