After upgrading a failover cluster node from Windows Server 2022 to Windows Server 2025 the cluster service failed to start with the following error:

The Cluster Service service terminated with the following service-specific error:
A specified authentication package is unknown.

According to this blog post https://jigsolving.com/failover-cluster-service-wont-start-server-2025/ the following is a workaround for this problem, however, having this setting in place is a security benefit and enabling custom SSPs and APs weakens the system.

The bad news for us was that upon deploying Server 2025 and Failover clustering, a CIS GPO setting (specifically from the LSA node introduced in 2022) now seems to think that SSP/AP used by Failover Clustering in Server 2025 is custom and therefore does not allow it to be loaded or used. Boo!

In order for clustering to work, this GPO setting MUST NOT be configured on Server 2025. I believe it is a bug that Microsoft’s own CLUSAUTHMGR.DLL file is declared as a custom package.

The GPO Element is known as:
Allow Custom SSPs and APs to be loaded into LSASS and it is set to Disabled

In the registry this GPO restriction will appear under:
HKLM\Software\Policies\Microsoft\Windows\System

With a value of:
AllowCustomSSPsAPs REG_DWORD 0

If the GPO Element is set after a cluster is formed, the Cluster Service will not start.
If the GPO Element is set BEFORE a cluster is even formed – it will seemingly hang whilst trying to form the cluster for around 15 minutes and fail.

Removing this setting or GPO (and rebooting) will resolve this specific condition.

Having the policy set to disabled may be a company requirement. You can override the setting in the local group policy editor under Computer Configuration\Administrative Templates\System\ Local Security Authority.