Investigating performance bottlenecks in your web application - Part 1

Recently I was responsible for improving the performance of a website implemented with ASP.NET MVC. Although the performance improvement does not have a straight forward approach and it strongly depends on your application structure and physical topology, here are the steps I chose and I believe they are valid in most applications with the same structure. Before I list these steps let me describe the structure of our case study:

We are using:
2 Web Servers: Windows Server 2008 R2, ASP.NET MVC 3 hosted in IIS 7
2 App Servers: Windows Server 2008 R2, WCF services hosted in IIS 7
2 DB Server: SQL Server 2008 R2
2 Reporting Servers: Sql Server Reporting Servers (SSRS)


The preliminary load test results revealed that the system was unable to cope beyond 50 concurrent users. The CPU usage in application servers was always flat around 100%. As a first rule, when you experience the high CPU usage the first thing you should blame is the APPLICATION itself! So you have to leverage your investigation skills to find the bottlenecks in the code.

An average CPU utilization of 20–30% on a web or application server is excellent, and
50–70% is a well utilized system. An average CPU utilization above 75% indicates that a system is reaching its computational capacity; however, there's usually no need to panic if you have an average of 90–95%, as you may be able to horizontally scale out by adding an additional server.

As long as we are using Windows Communication Foundation (WCF) in the application layer we can easily leverage the WCF diagnostic option to investigate the long running requests. To turn on the tracing on WCF services you need to add the following section in web.config file.

Here is more information about WCF diagnostics in MSDN.

As you see we log the tracing results in Tracing.svclog file. To view the content of this file you have to use SvcTraceViewer.exe located in "C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin" folder.

You can easily use this powerful tool to analyse the diagnostic traces that are generated by WCF. To get a reliable result you need to run the application for a reasonable period of time while the wcf tracing is on. In this way you could find out the number of method calls along with the process duration for each method. Choose the methods with unexpected duration time and try to modify the method to achieve a better performance.

Sometimes the service operation calls the other services or hits the database to fetch or update the data. So you will need to find out witch part of the service operation causes the bottleneck. Is it the database call or the operation process itself? You can plan different approaches based on your coding structure.

Using WCF tracing will help you to nominate the potential bottlenecks in service level. What if the bottleneck is in the other layers such as Repositories. As you may know people are using repository design patterns to separate the logic that retrieves the data and maps it to the entity model from the business logic that acts on the model. Persistence ignorance is the benefit of using this pattern. Hence we need to do tracing in repository level as well. If you are familiar with Aspect Oriented Programming, known as AOP, you can implement an Interceptor to inject a custom logger into all methods of your repositories. This logger wraps the service method and calculates the duration of the process.

As an example, let's assume that we are using Castle.Windsor as an IoC container. Here is the castle Interceptor I've implemented to do the job for us:

public class FunctionalTraceInterceptor : IInterceptor
  {
     public void Intercept(IInvocation invocation)
     {
         StringBuilder logEntry = new StringBuilder();
         Stopwatch stopWatch = new Stopwatch();
         stopWatch.Start();

         invocation.Proceed();

         stopWatch.Stop();
         double duration = stopWatch.Elapsed.TotalMilliseconds;

         logEntry.AppendLine(String.Format("{0},{1},{2}", invocation.InvocationTarget.ToString(), invocation.MethodInvocationTarget.Name, duration));
         ApplicationLogger.Inst.LogInfo("TraceAllFunctionCalls", logEntry.ToString());
    }
  }

IInterceptor is an interface located in Castle.Core.Interceptor namespace. ApplicationLogger is our singleton logger that uses log4net. You can implement your own logger instead. InvocationTarget is the service and MethodInvocationTarget is the service method.

To inject this interceptor into the services you can simply define it in castle configuration file:

  
    
        
          ${FunctionalTraceInterceptor}
        
    

    
    
  

SampleRepository is the repository that has implemented the ISampleRepository interface and we want to inject the interceptor to all of its public methods. You can use this solution for all the services you registered in IoC container.

The purpose of this post was introducing two common techniques for tracing the service calls. However the application is one of the components you need to modify to get a better performance. There are other parameters in performance improvement process that I will mention in the next posts.