Longhorn - Replica unable to start due to low available space in disk

This document (000021268) is provided subject to the disclaimer at the end of this document.

Situation

Longhorn UI shows one volume in Degraded state.

One or more of the replicas are not being scheduled and showing the following error:
"Scheduling Failure. Replica Schedule Failure".

Resolution

There are 2 conditions that must be met to successfully schedule a replica to a disk:

The actual space usage condition: (Storage Available - Actual Size) > (Storage Maximum * Minimal Available Percentage)/100)
The scheduling space condition: (Size + Storage Scheduled) <= ((Storage Maximum - Storage Reserved)* Over Provisioning Percentage / 100)

Where:

Storage Available is Storage Available of the disk in node detail page in the picture below.
Actual Size is the actual size of the Longhorn volume.
Storage Maximum is Storage Max of the disk in node detail page in the picture below.
Minimal Available Percentage is the setting https://longhorn.io/docs/1.5.3/references/settings/#storage-minimal-available-percentage
Size is the spec size of the Longhorn volume.
Storage Scheduled is Storage Scheduled of the disk in node detail page in the picture below.
Storage Reserved is Storage Reserved of the disk in node detail page in the picture below.
Over Provisioning Percentage is the setting https://longhorn.io/docs/1.5.3/references/settings/#storage-over-provisioning-percentage

If one or both of these requisites are not being fulfilled, it is expected that some replica/s can not get scheduled.
To solve this, you may try to free space, or maybe you want to reduce your Minimal Available Percentage¹, e.g., from 20 (0.2) to 10 (0.1).

After both pre-conditions are satisfied, you should be able to schedule the volume with no issues.

Cause

One potential root cause behind that error may be that the available size of the disk or the scheduling space on the affected node is not enough to schedule the replica.

Additional Information

¹Best practice: the Minimal Available Percentage value should be around 20-25% if you are making use of Longhorn in the root disk. However, if you're using a dedicated disk for Longhorn, you can lower the setting minimal available storage percentage to 10%.

You can check these pre-conditions in the Longhorn code here: https://github.com/longhorn/longhorn-manager/blob/16c09fdd296bdd893ceaaa2717456d8ea8cc7036/scheduler/replica_scheduler.go#L709-L710

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.