SUSE Support

Here When You Need Us

'Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"' error when upgrading Rancher immediately after performing a local cluster Kubernetes upgrade

This document (000021315) is provided subject to the disclaimer at the end of this document.

Environment

A Rancher v2.x instance running in an RKE or RKE2 cluster

Situation

The Rancher upgrade task is launched immediately after upgrading the underlying Kubernetes cluster. The Rancher upgrade fails with the following error:
image.png

Resolution

Before running a Rancher upgrade, check that ingress-nginx Pods in the local cluster are running. If the Rancher upgrade task is executed in an automated fashion, a delay should be introduced between the cluster upgrade and the Rancher helm upgrade.

The commands below will enable you to validate that the ingress-nginx pods are running before performing the Rancher helm upgrade. The flag "timeout" ensures the command exits with an error on a timeout, rather than waiting indefinitely, e.g. --timeout=2m.

RKE1
kubectl rollout status daemonset -n ingress-nginx nginx-ingress-controller --timeout=x

kubectl wait --for=condition=Available -n ingress-nginx nginx-ingress-controller

RKE2
kubectl rollout status daemonset -n kube-system rke2-ingress-nginx-controller  --timeout=x

kubectl wait --for=condition=Available -n kube-system rke2-ingress-nginx-controller 

 

Cause

The 'helm upgrade --install` task for Rancher is invoked before the ingress-nginx pods in the cluster are in a running state, following the cluster upgrade. As a result, the ingress-nginx validating webhook cannot be satisfied, resulting in errors when helm attempts to patch the Rancher ingress resource.

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000021315
  • Creation Date: 29-Dec-2023
  • Modified Date:02-Aug-2024
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

tick icon

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

tick icon

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

tick icon

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.