HOWTO: Diagnose 401.x HTTP errors on IIS

One of the most common questions asked about IIS on the newsgroups as well as Microsoft Product Support is "why am I getting 401 Access Denied"?
There are many, many possible causes and variations, but from the IIS perspective, the top-level, logical categories are fixed. This information can help dramatically narrow down the scope of any investigation, but unfortunately, few people know to take advantage of this information. This is what I am going to address with this entry - how to use and diagnose the 401.x error codes on IIS.

Step 1: Determine the SubStatus Code

When you get a 401 response in the browser from IIS and you want to troubleshoot it, the first thing you should do is determine what "type" (i.e. HTTP substatus) of 401 it is.
Starting with IIS 6.0, the IIS web log files located at:

%SYSTEMROOT%\System32\LogFiles\W3SVC###\*.log

record both the HTTP status and substatus code, which when combined with the Win32 error code can aid to troubleshoot many 401 errors. The W3C log entries look like the following, with the HTTP status, substatus, and Win32 error codes highlighted.

#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2005-05-21 05:39:27
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status 
2005-05-21 05:39:27 192.168.0.101 GET /VirtualServer/VSWebApp.exe view=1 1024 WEBBROWSER\User 192.168.0.101 Mozilla/4.0+(User-Agent) 200 0 0
2005-05-21 05:39:27 192.168.0.101 GET /VirtualServer/scripts/VSScripts.js - 1024 - 192.168.0.101 Mozilla/4.0+(User-Agent) 401 2 5
2005-05-21 05:39:27 192.168.0.101 GET /VirtualServer/scripts/VSScripts.js - 1024 - 192.168.0.101 Mozilla/4.0+(User-Agent) 401 1 2148074254
2005-05-21 05:39:27 192.168.0.101 GET /VirtualServer/scripts/VSScripts.js - 1024 WEBBROWSER\User 192.168.0.101 Mozilla/4.0+(User-Agent) 304 0 0

You can also get the substatus code from the HTML response itself, but web browsers like Internet Explorer have options like "Show Friendly HTTP Errors" which obscure the detailed error response by making them "simple" and "user friendly", so if you want the real error response, you need to turn off that option.
Unfortunately, prior to IIS 6.0, the IIS web log files are useless in distinguishing 401 substatus because it does not even record it. Your only option is to figure this out from the HTML response itself, assuming the 401.x Custom Error pages for that URL's scope are configured to send different HTML pages for each type of error.

Step 2: Determine Course of Action

Once you have determined the HTTP substatus code, you can start narrowing down the types of failures and causes. The following are the fixed categories that IIS reports. I will give detailed explanation of what each means as well as some common causes/solutions (obviously not exhaustive).

401.1 Denied by Invalid User Credentials

This error indicates that IIS failed to obtain an NT user token with which to execute the request.
In a nutshell, IIS expects to have a NT user token at the end of Authentication (even anonymous authentication - see this URL for details), and if this does not happen, you get 401.1.
Some common causes include:

• The client gave the wrong username/password (including none at all). This could be from incorrect cached auto-login attempt by the browser, or from a user login dialog from the browser.
• Invalid Kerberos configuration - on IIS6, if you have a customized Application Pool Identity AND Integrated Authentication is used AND the web server is in a domain, you will mysteriously get 401.1 unless you configure SETSPN *or* change Integrated Authentication to favor NTLM. See the following URLs on Application Pool Identity, Integrated Authentication in IIS, and Constrained Delegation configuration as well as this URL on additional Kerberos-related troubleshooting for more information
• You enabled Anonymous authentication, yet you still get 401.1 for all requests. One common cause is if the configured anonymous user credentials stored in the IIS metabase configuration file is DIFFERENT than the user principle's credentials in reality (i.e. mismatched password). In all cases, the preferred solution is to manually synchronize the username/password of the anonymous user principle in IIS with that of the real user principle. I have seen many amazing variations of this cause, including:
  • For testing purposes, the user types in his/her OWN username/password as anonymous user credentials at some point in the past and forgets about it. Later, when password policy forces them to change their password, the anonymous user credentials stored in IIS configuration is now mismatched with reality. On subsequent anonymous requests, IIS fails to login and obtain a NT user token for anonymous authentication and fails with 401.1, and it looks like IIS is just plain buggy and could not even support anonymous authentication.
  • I have also seen the reverse happen - user configures IIS to use their username/password as anonymous user, and when they changed their password, web server traffic quickly causes IIS to incorrectly login with wrong user credentials too many times, causing their user account to be locked out. These users now complain that their user account is mysteriously getting locked out as soon as it is unlocked, even before they log in anywhere.
  • On upgrading from IIS 5 to IIS 6, IIS Sub Authentication (i.e. the "allow IIS to control anonymous user's password" feature) is enabled by default for compatibility. This allows IIS to log in the anonymous user principle without actually keeping the user credentials in sync, and anonymous authentication looks good while in IIS 5 Compatibility Mode. However, as soon as you switch into IIS 6 Worker Process Isolation Mode, Sub Authentication is disabled because it requires a privileged process identity like Local System (which is a known and quite unnecessary security risk for the lowly purpose of password sync). This means that IIS 6 now tries to log in the anonymous user credentials stored in the metabase, which has probably NEVER been kept in sync with reality through the upgrade... and you now get 401.1 for every single anonymous request. To a casual user, it looks like switching into IIS 6's native mode simply breaks anonymous authentication and the rest of the website.
• The server has been reconfigured to deny necessary login privileges for the authenticating user or its containing group (either anonymous or through some authentication protocol). This can be done through automated re-application of Group Policy for domain members, DCPROMO to/from Domain Controller, or static application of security templates. What ends up happening is that the server-side reconfiguration may remove Local/Remote Login rights for that user, impose new restrictions (like Login hours, Logon type), etc... preventing IIS from successfully logging in the user to execute requests and resulting in 401.1.
• Your event log could be full for some reason - see KB 832981.

401.2 Denied by Server Configuration

This error indicates that the web server is configured to require certain authentication protocols for communication, but the browser failed to use any of those authentication protocols. The corrective action should be to either configure to require an authentication protocol acceptable to the client, or use a client that satisfies the server authentication protocol requirements.

• A common cause of this issue happens with the older Netscape/Mozilla browser clients and an IIS web server configured to require Integrated Authentication. These browser clients did not understand Integrated Authentication, so when IIS required Integrated Authentication and the browsers repeatedly ignored those responses, IIS will return 401.2 indicating that the browser failed to use an authentication protocol required by the server. Newer Mozilla browsers like FireFox do not have this deficiency.
• Another possible cause is when using Integrated Authentication over the Internet. Integrated Authentication (NTLM) is a connection-based authentication protocol, meaning that an authenticated connection between a client and server is the only proof of authenticity. This works fine in Intranet scenarios, but for Internet scenarios a lot of network devices in between the client and server can either not support or mishandle NTLM (such as Proxy Server connection pooling/multiplexing), causing unexpected 401.2. Here is how it happens: since NTLM is connection-based authentication, once a client successfully authenticates using NTLM, it often re-uses its end of the connection and simply sends anonymous requests over it. Now, assume an intervening proxy server pools connections between client and server, is unaware of NTLM, and independently decides to send the client's request over *another* connection in its pool (instead of the already authenticated one) to the server. This causes the client's anonymous request to be sent to the server over a new, unauthenticated connection, and the server dutifully rejects it with a 401.2 since the server requires Integrated authentication. The 401.2 rejection is totally unexpected by the client since it thought it was re-using an authenticated connection and did not initiate any re-authentication. Yup, fun... ;-)
  Common variations of an "intervening proxy server" include:
  
  • "Web Accelerators", such as the Google Web Accelerator. These programs basically act like a local "caching proxy" such that requests for content has a higher chance of coming from your local hard drive than over the network, thus "speeding up" apparent web access.
    
  • Web Anonymizers - these programs basically disguise your own IP and other request characteristics with their proxy's IP and characteristics, and this is shared amongst all their users, thus providing anonymity through numbers.
    
  • Sniffer tools like Fiddler for IE - these programs act like "Web Accelerators" except instead of caching request/responses, it chooses to selectively capture and display those request/responses for user analysis.
    
  • Web Access Proxies for some broadband providers or preset Company proxies - these are the traditional obvious proxies.

401.3 Denied by Resource ACL

This error indicates that the web server was able to authentication and obtain SOME NT user token to process the HTTP request (you still have to determine WHICH user's token...), but that NT user token lacks the FileSystem ACLs to access the requested resource. This is the typical "access denied" due to missing file ACLs that people assume, and yes, you will likely need to adjust ACLs to resolve this issue.
However, realize that all of the OTHER 401.x errors have nothing to do with ACLs, so I recommend AGAINST tweaking resource ACLs to "Everyone: Full Control" to remove ACL issues from the picture. You should be able to determine the exact user that fails to have ACLs to the resource, and just adjust ACLs for that user on the necessary resources and resolve the issue
Common causes include:

• Wrong/Missing ACLs on the file for the authenticated user. You need to change the ACLs or change the user to an identity that has correct ACLs on the file.
• You are not authenticating with the authentication protocol you think, and thus the user principle may be unexpected. Reconfigure the authentication protocols as-appropriate so that you end up running as user identity you expect on the server.
• If your content is on a UNC share, you may have mismatched NTFS ACLs vs. UNC Share ACLs.

A useful tool to pragmatically determine access-denied to file resources is File Monitor.

401.4 Denied by Custom ISAPI Filter

This error indicates that some ISAPI Filter running on that request sent back a structured 401 response of some sort.
The reasons why the ISAPI Filter is returning such 401 responses are completely arbitrary and uncontrollable by IIS. You will need to determine WHICH ISAPI Filter is returning this response and obtain support for this ISAPI Filter to resolve the issue.

401.5 Denied by Custom ISAPI/CGI Web Application

This error indicates that some ISAPI Extension or CGI Web Application sent back a structured 401 response of some sort.
The reasons why the CGI/ISAPI are returning such 401 responses are completely arbitrary and uncontrollable by IIS. You will need to determine WHICH CGI/ISAPI is returning the response and obtain support for it.
In the case of requests that execute .DLL or .EXE requests, the CGI/ISAPI binary is clear. In the case of requests with Extensions that have Application Mapping (i.e. the .asp extension is mapped to the ASP ISAPI DLL Script Engine), you need to look up the extension and its associated Application Mapping in the URL's scope to determine the Script Engine to obtain support.
Starting with IIS6, it is also possible for Wildcard Application Mappings to execute on any request within its configured scope prior to executing the actual handler for the request. It is conceptually like how an ISAPI Filter can act on a .asp request before the ASP ISAPI handler processes the request.
A popular scenario for Wildcard Application Mapping is to implement custom authentication, so a 401.5 on a .html file may indicate presence of a Wildcard Application Mapping-based custom authentication. To the astute reader - .html is handled by the Static File Handler by default, which will only return 401.3 for access denied... so if you see a 401.5 involved with a resource like .html that is usually NOT associated with an Application Mapping which can only return 401.5 for access denied... you know either a Wildcard Application Mapping, or non-standard Application Mapping for .html is involved.
To determine which Wildcard Application Mapping is involved on a request, you have to use the IIS Manager to look up the EFFECTIVE Wildcard Application Mapping for that request, starting from the web page of the URL itself and working up the directory tree until you find the nearest Mapping definition. Yes, it is a major hassle prior to IIS7 to determine the effective Application Mapping of a request for support purposes.
Starting with IIS7, you can use Failed Request Tracing to determine all handlers that executed in sequence on a given request, which will show exactly what module / application mapping caused the 401.5. This makes it extremely easy to identify the faulty DLL without trying to calculate effective Wildcard Application Mapping configuration.

Conclusion

401.1 through 401.3 errors are associated with IIS request processing and allow the logical interpretations and assumptions that I listed above.
Meanwhile, the 401.4 and 401.5 errors are the most arbitrary to diagnose since custom ISAPI DLLs and CGI EXE can cause IIS to behave in non-obvious manners. Thus, much of the logical assumptions about 401.x do not apply.
I hope that this information has been useful in deciphering the 401.x errors from IIS. If you have additional questions, feel free to post a comment or post a private question via the "contact" link.
Recently, we have also released a tool, AuthDiag, to help troubleshoot IIS access denied issues. You can download it from this location. In particular, it has a feature to hook in to various failure points in IIS and directly troubleshoot what is failing on a given request - you need to see and try it out!
Good Luck.

Original Post: