NOTE: See https://geekmungus.co.uk/?p=3619 for the most up to date article!
We’ve had an issue with the recent “Security Update for Microsoft Windows (KB5019964)” update, as by the looks of it a load of other people.
In our case we are running Microsoft Windows 2016 domain controllers, the NetApp filers are pretty old we’re readying to migrate off them; they only support SMB version 1.
We suddenly got reports that accessing the shares via a UNC with the hostname did not work from Microsoft Windows machines giving an error along the lines of name not found. Using the IP address within the UNC path worked fine from a Microsoft Windows machine. Additionally using the UNC with the hostname from Apple Mac workstations and Linux clients (Samba client) worked fine.
Symptoms
We first started with the name resolution, DNS was resolving fine. NETBIOS also appeared to be fine. Essentially:
\\server1 – Does not work, name is not resolved, even though DNS works fine.
\\172.16.10.10 – Does work fine.
Diagnosis
After much checking we found that the Kerberos authentication tickets were not working. Essentially when using the IP address we think that NTLM authentication is being used rather than Kerberos, hence why using the IP works while the hostname does not; that is the same for the Apple Mac and Linux clients.
Running a “klist” from affected clients we saw this:
Current LogonId is 0:0x4b7cb6d6
Cached Tickets: (2)
#0> Client: USER123 @ DOMAIN.COM
Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
Start Time: 11/16/2022 10:59:20 (local)
End Time: 11/16/2022 20:59:20 (local)
Renew Time: 11/23/2022 10:59:20 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0x1 -> PRIMARY
Kdc Called: DC1.DOMAIN.COM
#1> Client: USER123 @ DOMAIN.COM
Server: cifs/server1.domain.com @ DOMAIN.COM
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
Start Time: 11/16/2022 10:59:20 (local)
End Time: 11/16/2022 20:59:20 (local)
Renew Time: 11/23/2022 10:59:20 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: DC1.DOMAIN.COMNotice the “Session Key Type: AES-256-CTS-HMAC-SHA1-96” on the #1 key. It is AES-256-CTS-HMAC-SHA1-96 not RSADSI RC4-HMAC(NT), The NetApp is on a software version that can’t work with the later cipher type. Hence the issues across two other NetApp arrays running the same version.
This was confirmed on the NetApp logs saying an “unsupported cipher” was being used.
Resolution
It turned out the issue was down to a recent Microsoft Windows update to our domain controllers. The following patch which was applied overnight 15th november 2022 to 16th November 2022 was the cause of the issue:
November 8, 2022—KB5019964
Security Update for Microsoft Windows (KB5019964) – Installed 15/11/2022
CVE-2022-37967
So we removed the patch from all domain controllers which had been installed overnight and this resolved the issue.
The difference in behavior was visible through the tickets issues, we first ran a “klist purge” to clear the client of the TGS ticket with the incorrect cipher type and then attempting the connection to the NetApp via the UNC path with the hostname which was now successful.
The klist output of the TGS ticket for the NetApp before and after the patch was removed is given below. Firstly before (patch was removed)
#2> Client: USER123 @ DOMAIN.COM
Server: cifs/server1.domain.com @ DOMAIN.COM
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
Start Time: 11/16/2022 13:29:45 (local)
End Time: 11/16/2022 23:29:45 (local)
Renew Time: 11/23/2022 13:29:45 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: DC1.DOMAIN.COM
Then the TGS klist output after the patch has been removed:
After (patch is removed)
#1> Client: USER123 @ DOMAIN.COM
Server: cifs/server1.domain.com @ DOMAIN.COM
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
Start Time: 11/16/2022 14:12:26 (local)
End Time: 11/16/2022 23:47:44 (local)
Renew Time: 11/23/2022 13:47:44 (local)
Session Key Type: RSADSI RC4-HMAC(NT)
Cache Flags: 0
Kdc Called: DC1.DOMAIN.COMOnce the Session Key Type: RSADSI RC4-HMAC(NT) was being shown for the TGS Tickets from the NetApps the issue was resolved.
Any remaining issues should be able to be resolved by a reboot of the machine or a “klist purge” run from the command prompt to flush the incorrect cipher TGS and use a working one.
We’re then aiming to move off the NetApps posthaste to allow us to put this patch back in place in the long term to remediate the Kerberos vulnerabilities it has. This particular patch however has been known to cause other problems even with those who have AES-128 and above enabled and don’t use legacy ciphers, so waiting on Microsoft for updates on that one.
Additional Information
https://support.microsoft.com/en-gb/topic/november-8-2022-kb5019964-os-build-14393-5501-5c195bd1-91d5-402e-a973-813373ba4357
https://learn.microsoft.com/en-gb/windows/release-health/windows-message-center#2952
https://support.microsoft.com/help/5020805
https://support.microsoft.com/help/5021130
https://support.microsoft.com/help/5021131
https://support.microsoft.com/en-gb/topic/november-8-2022-kb5019964-os-build-14393-5501-5c195bd1-91d5-402e-a973-813373ba4357
https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/what-happened-to-kerberos-authentication-after-installing-the/ba-p/3696351