This one had dogged me for ages. There are a number of possible solutions out there but none helped my situation. Today I finally cracked it!
Here’s the situation:
- ISA 2006 standard edition in a perimeter network
- Published HTTPS apps (MOSS 2007 + other misc ASP.NET apps)
- Weblistener with forms based authentication configured to use LDAP with secure connection and without GC
- Internal CA used to provision certs to domain controllers
- Password management features enabled (Hence the requirement for secure connection and no GC)
When users login they get a delay of up to a minute before the authentication screen disappears and their application begins to load. I did some traffic dumps and found that this delay was occurring during the TLS handshake between the ISA server and the LDAP server (domain controller).
Immediately after the LDAP server issues the “Server Hello” there would be a delay, or what appeared to be some kind of a time-out, of about 15 seconds. This occurred several times throughout the authentication process which resulted in the long delay. I confirmed this by disabling the secure connection to the domain controller. This got rid of the delay but of course this wasn’t an option because without SSL you can’t have password management features enabled.
After much digging I eventually ran process monitor on the ISA server and found that there was a RSA machine key which the firewall service (wspsrv.exe) was trying to access which it didn’t have permission to.
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys
Every time it tried to read the key it introduced a few seconds delay as can be seen in the time column above.
The solution: Give the NETWORK SERVICE account read access on that file (not the whole folder). Of course the file name will be different for each installation so you’ll need to use ProcessMon to find out which file can’t be read by wspsrv.exe.
So this raises a few questions for me, some easier to answer than others:
- How, when and why are these machine keys are used? *update* These are private keys for certificates installed on the server. They are used to encrypt data which is later decrypted using a public key. They are called “machine keys” because they are for the local “machine” account, as opposed to a user account who’s keys would be under C:\documents and settings\username\…
- How and by what are these machine keys created? *update* When you create or import a certificate e.g Public certificate for SSL web publishing.
- Is there a specific key associated with an application? *update* The private key is associated with a certificate. Use FindPrivateKey.exe to help you match keys to certificates
- Why doesn’t ISA server set the correct permission on the machine key file during installation? *update* Not really an ISA fault – ISA runs as NETWORK SERVICE so if that doesn’t have access neither does ISA. I’m still not sure how the permissions get messed up.
- Why doesn’t ISA throw an error to the event log when it couldn’t read the file?
- Why does everything still work even though it can’t read the file! (albeit with significant delay)?
Any input would be much appreciated!
Thanks for reading.
Dans l’ordre, il faut donc signer le contrat de prêt et envoyer la facture à la CAF après achat.
I think i finally understand what´s going on:
http://blogs.technet.com/b/isablog/archive/2010/05/31/authentication-delay-for-sites-published-through-isa-server-2006-using-forms-based-authentication.aspx
is incorrect and should read:
“… Make sure that the server authentication certificate ON THE ISA SERVER does not have “Client Authentication” attribute enabled as per article http://technet.microsoft.com/en-us/library/cc514301.aspx. …”
If you follow the link it tells you how to disable mutual-auth on the DC-Client (hence ISA and NOT the DC) to NOT trying to authenticate to the DC using a Web-Publishing-Certificate with the “Client-Authentication”-Purpose.
http://msdn.microsoft.com/en-us/library/cc223502.aspx
explains exactly how LDAPS works with SSL/TLS.
http://technet.microsoft.com/en-us/library/cc514301.aspx also tells that ISA/NetworkService should NOT have access to the Private Key (which causes the delay during TLS-Nego)
rgds,
peterzapfl
P.S.: … like your blog ….. greetz from Austria
Cheers peterzapfl, And greetings from Aotearoa! During troubleshooting I did go through those articles which looked at certificate purpose but it didn’t resolve the issue.
In hindsight I really didn’t collect enough data so it’s hard to say now what was going on. If I had the chance I’d go back re-produce the scenario. But I guess I’ll probably never get around to it! I’d have to rebuild it in the LAB. Are you in the position where you can experiment? I’d be keen to hear your findings.
Cheers,
Rhys
btw. Sorry I just realised your comment was still not approved. I suspect I need more practice operating my comments queue!
@ “….I’m still not sure how the permissions get messed up. ….”
I think i know why – I think this happens when you import the certificate with the private key into your USER-Cert-Store and move it to the machine store later on using drag-and-drop in the cert-mgmt-mmc. I have to verify this, but i think it copies the file with the private key over, but does NOT set the correct access permissions for the network-service.
@ “Why does everything still work even though it can’t read the file! (albeit with significant delay)?”
I think that as long you do not have to change the password (which only works via LDAPS) it still tries to establish the LDAPS-connection but because mutual-auth fails it gives up (SSL/TLS-timeout) and continues … but that´s just my 2cent 😉
Hi, yeah that certainly sounds like a plausible scenario. I’d be interested to hear back if you do get a chance to verify. From memory we didn’t have issues with password management while the issues existed.
Cheers,
Rhys
Man, you rock. Thank you so much!!!! Once I set read allow flag to the file, FBA got a big performance boost…
Man, you simply rock. I was having this issue with all published sites with FBA after ISA recovery. Struggled for 7 months with that. Thanks a million!
Hey Marcin, Thanks for the feedback. I guess I should really raise this with MS and see if they can put out a KB.
Cheers,
Rhys
I have two ISA servers, both did have problem (RSA file access denied) only after “new” publish. Exchange 2010 and Sharepoint 2010 to be exact. Exchange 2003 did work fine. So, why does that matter? Or is it that I need to configure rule, so problem appear only after re-configure. (patch or so?)
Very interesting. Do you have a single Web Listener for Exchange 2003, 2010 and ShaerPoint 2010. All using LDAP? Is Exchange 2003 is still ok? Or is it broken after creating the Exchange 2010 / SharePoint 2010 Rule?
Hah, I was thinking this cant be true.
But this solve my problem. I have been seaching all over Internet and finally.
Thanks for your comment. This is why I blog, to share the solutions I find.
Great post! Finally I solved the problem with slow client logons in isa server! Thanks!
On to your questions: I cannot answer them, but I will post any answers here, if, or when I find them 🙂