Replies: 3 comments
-
This part is precisely about this part afaik. You have not described where you run your Helm and how you deploy it but what you described is wha Argo CD does for example. They run "post-hook" after the pods are already started which is not what regular "helm install" or "helm upgrade" does and it's not according to specification. The "post-hook" is run after the resources have been created but there is no good reason they should wait for the pods to start-up. If you take regular Helm ugprade, the This is what helm defines:
It's not "after all resources are loaded and new containers are successfully started" - ths condition is missing in the "standard" helm chart (and regular chart does not wait for containers when you use standar Helm Chart and Kubernetes). Or at least this is how I understand that - but if you have other sources of information, I am also all ears. I also find it peculiar Argo and some others are doing it in non-standard way, so maybe there are other reasons for that. I guess you need to find out what's the equivalent approach for whatever you are using. The other options are not applicable:
Generally hooks is a good idea if you either have a regular helm chart, or whatever your Helm "hosting" solution provider suggests as an alternative if they want to delay running the post-hooks. But maybe you can indeed propose another way that will have no such drawback? We always welcome PRs, but here it seems that:
|
Beta Was this translation helpful? Give feedback.
-
Hey, I seem to have run into the same issue and came to the same conclusions as you @sbrandtb. I found helm/helm#11778 that could explain the unexpected behaviour. If I get this right, the helm behaviour varies depending on whether the I guess that would also explain the weird descrepancies between "standard" helm behaviour and ArgoCD, terraform, etc. that you mentioned @potiuk. They probably Since there is a TTL for job cleanup for a while now, I guess the helm hooks could just be dropped to have a consistent behaviour. |
Beta Was this translation helpful? Give feedback.
-
Greetings, For me, it's a simple solution. I increased the resources limits. migrateDatabaseJob:
<snipped>
resources:
limits:
cpu: 1000m #128m
memory: 1024Mi #128Mi
requests:
cpu: 100m
memory: 128Mi |
Beta Was this translation helpful? Give feedback.
-
Hey,
I want to ask this here before I create a pull request or issue, because it feels like an obvious problem to me and I do not understand why I cannot find any existing issue or discussion on that topic, so I am assuming I am doing something wrong here.
Our issue is that Airflow with default settings basically deadlocks every time a database upgrade (
airflow db migrate
) is required.By default...
webserver.waitForMigrations.enabled=true
(airflow/chart/values.yaml
Line 1297 in 0cb875b
post-install,post-upgrade
:airflow/chart/templates/jobs/migrate-database-job.yaml
Line 46 in 0cb875b
So, the migrate database job is not run until installation/upgrade is finished (that's the meaning of post-hooks, right?), which never happens because pods are waiting for the migration before they turn into "Running" status.
Am I missing anything here or are the defaults downright unusable?
I found this part in documentation that talks about disabling the hooks for ArgoCD and similar, but that does not apply here, does it? I want to use Helm hooks.
Possible solutions that come into my mind:
Make the hook
pre-install,pre-upgrade
. I know from experience that this leads to issues if hooks are using resources (ConfigMap, Secret) that are being created by the Chart itself, leading to another chicken-egg problem, but we are not using Helm-managed CMs for this very reason anyway. I would love to overwrite the hook definition, but due to the order how the merge with user defined annotations is done here, I cannot even overwrite:Disable
waitForMigrations
, which would lead to temporarily broken deployments between start of new pods and post-hooks. Plus, Airflow might not even start without migrations, leading to the same deadlock again.Run migrations manually in CI before upgrade, but I would rather not go back to writing manual shell scripts.
Run migrations in init containers and not hooks, but then again I would rather not go back to writing manual scripts.
So, again... Am I missing anything? Any suggestions?
Beta Was this translation helpful? Give feedback.
All reactions