AngryPets Blog

IIS and 403 - Oh My!!! (ReverseDOS update)

If you've ever played around with load testing a real application (by real I mean one that involves security) served up by IIS and ASP.NET, you've probably seen some interesting behavior. If you BURRY the server, it will start throwing HTTP 403 (Access Denied) Response Codes up the wazoo. My gut feeling is that it's a bit like trying to delete an Excel file (or .dll) while it's open/in-use -- it's just a no-go because the OS knows that something already has its hooks in it. I'm pretty sure that something (file system, etc.) gets overloaded when you pummel a web app in IIS, and IIS has no other recourse to tell you: sorry, I can't get serve that request, because the underlying file is currently locked/in use/something.

At any rate, that's the EXACT behavior I'm trying to emulate with ReverseDOS. I want spammers to 'see' your webserver through tinted glasses that make it look like your server is so busy that it can't handle their requests. And, of course, a 403 is also desirable because it just blocks the request.

So, IF your webserver gets severely overloaded, you WILL begin to see random 403s where they do not belong. IIS/ASP.NET is just getting its butt kicked, and when it gets scared, it just starts throwing 403s while ducking behind the couch, cowering and babbling about how things aren't fair.

Here's an example that will work on any ASP.NET site, any time. Create the following HttpModule, and throw it into your site:

public class Tester : IHttpModule
{
  public void Init(HttpApplication application)
  {
    application.BeginRequest 
      += new EventHandler(application_BeginRequest);
  }
  public void Dispose() {}
  private void application_BeginRequest(object sender, EventArgs e)
  {
    if(HttpContext.Current.Request.QueryString["crash"] == "true")
    {
      HttpContext.Current.Response.StatusCode = 401;
      HttpContext.Current.Response.End();
    }
  }
}

There's nothing tricky there. If crash=true in the querystring, the site is supposed to throw a 401 (Access Denied). Pretty simple. Then if you hammer it with the Web Application Stress tool with 3 requests against a single page - with one of the pages requesting a 'crash.' Run it at something that keeps the CPU pegged for a few seconds, then go check your logs. Sure enough, 403s start popping up where they don't belong. (And actually, 500's start showing up where they don't belong as well if you look hard enough.) Here's a snippet from my logs:

15:52:59 GET /xx_403tester/webform1.aspx - 200 15
15:52:59 GET /xx_403tester/webform1.aspx - 200 15
15:52:59 GET /xx_403tester/webform1.aspx crash=true 401 15
15:52:59 GET /xx_403tester/webform1.aspx crash=true 401 15
15:52:59 GET /xx_403tester/webform1.aspx crash=true 401 15
15:52:59 GET /xx_403tester/webform1.aspx - 200 15
15:52:59 GET /xx_403tester/webform1.aspx - 200 15
15:52:59 GET /xx_403tester/webform1.aspx - 403 15
15:52:59 GET /xx_403tester/webform1.aspx - 403 15
15:52:59 GET /xx_403tester/webform1.aspx crash=true 401 0
15:52:59 GET /xx_403tester/webform1.aspx - 200 0
15:52:59 GET /xx_403tester/webform1.aspx - 403 16
15:52:59 GET /xx_403tester/webform1.aspx crash=true 401 16
15:52:59 GET /xx_403tester/webform1.aspx - 200 16
15:52:59 GET /xx_403tester/webform1.aspx - 200 0
15:52:59 GET /xx_403tester/webform1.aspx - 200 0
15:52:59 GET /xx_403tester/webform1.aspx - 200 0
15:52:59 GET /xx_403tester/webform1.aspx crash=true 401 16
15:52:59 GET /xx_403tester/webform1.aspx crash=true 403 16
15:52:59 GET /xx_403tester/webform1.aspx crash=true 403 16
15:52:59 GET /xx_403tester/webform1.aspx - 403 16
15:52:59 GET /xx_403tester/webform1.aspx crash=true 403 31
15:52:59 GET /xx_403tester/webform1.aspx - 200 15
15:52:59 GET /xx_403tester/webform1.aspx - 200 15

Notice that I'm putting one HECK of a load on the server. ALL of these requests came in the same second (and there were a hundred or so before this same second...). But notice that not only do some of the 401s become 403s, but some of the requests that should have resulted in 200 end up as 403s. IIS/ASP.NET is just getting killed, and starts lobbing 403s like a pidgeon-feeding lunatick attacked by teenagers in central park.  

It's ALSO worth noting that if I had instructed my HttpModule to throw 500s when a 'crash' was requested, that I WOULD NOT see this problem, at ALL. 500s would happen where expected, so would 200s. There would be no 403s -- which inclines me to suspect some sort of BUG with IIS/ASP.NET when it comes to 401s/403s.)

PROBLEM:
If ReverseDOS is trying to simulate this behavior, what do I do when stray 403's start popping up? Is that IIS getting hammered? Or, is that ReverseDOS sucking and shooting out false positives? The issues that I had with ReverseDOS 2.1 were most likely (??) some sort of logic or architectural flaw introduced by me (*sniff *).

BIGGER PROBLEM:
I've addressed my 2.1 problems. There were potentially some troublesome areas around my use of a static collection that may have been the culprit. I also wanted to bring ReverseDOS more in line with the functionality that would be present in 3.0. So I've now built version 2.8. (The beauty is that it gets rid of FilterTypes. Now you just add a filter, specify the text, and let ReverseDOS know if you want it evaluated as a regex. That's it.)

At any rate, testing is a total nightmare. If I overload my test box, 403s pop up everywhere. If I just run it 'normally,' everything works fine, 200s where they belong, and 403s where they belong. Of course, that's just in a controlled, tame, environment.

Given the problems with 2.1, I'm going to do a bit more testing with 2.8 before I release it. I want to get metrics on the overalll cost of filtering. (2.8 has been completely re-designed in terms of how it filters, it should be a near magnitude faster... ) The biggest problem, of course, will be simulating reality. In fact, I doubt I'll be able to simulate it. What I'll probably do is test it some more, drop it on my site and see how it does. If it works well, and doesn't cause me problems, I'll turn it loose to a control group of others and see how it fares on their boxes. 

Existing Comments:

# re: IIS and 403 - Oh My!!! (ReverseDOS update) - Posted: 7/23/2005 6:06 PM - By: Jamie Thingelstad
   Thanks for the update! Looking forward to getting my grubby hands on 2.8!

Couple of comments:

* The machines I'm running my web sites on are way (way!) overpowered for their usage and I just looked at my Cacti and MRTG graphs of CPU usage and there were no increases when I had ReverseDOS 2.1 installed. Just pointing this out as I was getting 403's without the CPU being pegged or ASP.Net getting hammered.

* If IIS handles 500's so much better, maybe use them instead? Does it change that much the behavior on the other side? Maybe give the user the option?

# re: IIS and 403 - Oh My!!! (ReverseDOS update) - Posted: 7/23/2005 9:39 PM - By: Michael K. Campbell
   Jamie,

Thanks for hanging in there. Good to hear about the power of your box, AND the impact (none) that ReverseDOS had on performance.

I'm pretty sure that the 403s on the 2.1 version were GRATUITOUS <g>. My lament here is that they're going to be hard to test against.

Oh, and ReverseDOS does/has allowed you to specify which code to throw (check the docs page). In the end I MAY end up being forced to use 500s, but I doubt it - if IIS has problems and throws 403s when it gets overloaded, it will do that WITH or WITHOUT ReverseDOS being on the box, meaning that ReverseDOS won't be a part of the problem.

The release of 2.8 will be soonish.

Add your own comment: