We had an issue where our RODCs suddenly stopped authenticating users and also would not allow administrators to logon via RDP or locally via the console giving this message:
“there are currently no logon servers available to service the logon request”
Any downstream services that were trying to authenticate via LDAP or LDAPS also failed to authenticate.
In our environment the RODCs have IPSEC VPNs from the isolated network they reside in to the internal Domain Controllers, as per Microsoft best practice to minimise the surface area for attack, effectively these RODCs can only talk to the internal Domain Controllers via an IPSEC VPN tunnel(s).
By default a RODC does not cache any credentials (other than a couple of exceptions to make the RODC actually work), so unless you specifically add AD accounts which you wish to have their credentials cached locally by the RODC (citation), the RODC by default listens for authentication requests, then verifies the username based on a cached copy of the Active Directory database, but the authentication request is passed through and processed by one of the internal Domain Controllers – if this can’t happen because the RODC can’t reach the internal DC(s) then you can’t login!
Issue
The issue we had was that the IPSEC VPN tunnels had gone down and therefore the RODC was unable to reach the internal DCs to perform authentication to either clients or for administrators.
Cause
In our case the VPN tunnels had gone down because of certificate expiry during a migration to a new Certificate Authority, these domain controllers had used auto-enrollment quite happily for a number of years which would automatically rotate the certificates and ensure IPSEC VPN connectivity was preserved and they operated correctly.
Resolution
The first challenge is how you can actually login to the RODC, being that they are domain controllers, there isn’t a local administrator (well kinda), so you must login with a domain account, but you can’t because the RODC can’t authenticate you.
The steps below were what was followed to remediate the issue and get the RODC working again, specifically in our case the local machine certificate on both the RODCs and the internal DCs had expired, so what was required was install valid certificates and ensure the Local and Group Policy (that configures the IPSEC VPNs) specified the issuing CA certificate.
Step 1 – Logon with Safe Mode Directory Services Restore Mode
Boot the RODC into Safe Mode – Directory Services Restore Mode, (hitting F8 on boot) you’ll need to use the following credentials to logon:
Username: .\Administrator (issues local administrator logon)
Password: <Directory Services Restore Mode Password>
Step 2 – Verify Certificate
When logged on to DSRM, perform the following check using the Certificate MMC Snap-In, specifying the “Computer Account”.
You then need to ensure that on the RODC (and any Internal DC) to which it attempts to establish a VPN has the following:
- A valid Machine Certificate is present (signed by the below) in the “Personal” store.
- A valid Intermediate CA Certificate is present in the “Intermediate Certification Authorities” store.
- A valid Root CA Certificate is present in the “Trusted Root Certification Authorities” store.
- A valid Management CA Certificate is present (EJBCA Specific) in the “Trusted Root Certification Authorities” store.
Once the certificates are validated, you need to ensure the VPN configuration utilises them.
Step 3 – Verify VPN Configuration and Specified Certificate
In our case the VPN configuration was managed by a Local Policy on the RODC and by Group Policy on the Internal DCs to minimise repetition of configuration, for the purposes of this they look the same irrespective of if they are Local Policy or Group Policy.
There are two CSRs (Connection Security Rules) one for outgoing and the other for incoming, both need to specify either the certificate (not recommended) or the issuing certificate, i.e. the Intermediate certificate (recommended) so that the VPN IPSEC connection can be authenticated.
Ensure that the “Certificate Store Type” is set to “Intermediate CA”, then click “Browse” to specify the correct Intermediate CA certificate that has been used to sign the machine certificate.
Once added, save and close all windows and reboot the RODC to get out of Safe Mode – Directory Services Restore Mode, the Internal DCs apply their policy periodoically so do not require a reboot if changes have been made there.
Assuming the authentication is successful you should be able to ping between the endpoints and you should see Security Associations appear in the “Security Associations” section of the Windows Firewall configuration.
Conclusion
Hopefully that is of use if you are facing something similar, a workaround can be applied if getting the IPSEC VPN up is problematic such as swapping to use pre-shared keys or use of directly allow connections between the RODC and internal DCs (not recommended). In our case the issue was just down to being midway through a migration to a new CA and some certificates expiring (because they did not auto-renew).