Introduction

As most of you might know, I am currently in the process of improving the scalability of the Paycento backend..

As a real lean adept, the idea is to optimize where it hurts... As the creator of Node.js pointed out perfectly, the biggest pain is in the disk/network access etc.. Blocking threads hurt bigtime... So I started searching for easy low-cost optimizations that would not require to much effort....

Before we begin: Phase 0 - measuring=knowing; ask Heracles

I spent a big part of last month coding a command line app called "Heracles", which is some kind of a helper app. It supports the following commands:

• checkout : downloads source and all solutions from SVN and puts them in the folders following our convention
• build: builds all the source and publishes it to the publish & publishweb folders
• db: we have a "db drop xxx" and a "db restore xxx", which downloads a backup from our ref db from the web and restores it in SQL Express..
• install: this is a simple wrapper that invokes chocolatey or downloads installer packages from the web, so we all have the same dev environment. "Heracles install *" is all one needs to have all the required prerequisites
• performancetest: restores the db, fires up the api WCF service with this db, fires up memcached and then runs all integration tests that have the category "integration" and "speed". (simple MStest invoke, capturing relevant output). We redirect trace output to the console and append performance output in logs, so we have a log that contains every single performance test we ran....

First things first: Caching

The easiest way to speed up network/disk access, is simply avoiding it, so I started with caching the part that requires scalability...

Caching is an essential part of scalable websites, and there is no need to reinvent the wheel here... I started of with a simple static in-memory dictionary to verify my hunches, and then I started considering established options... After some reading up between different caching solutions, I found it hard to decide which option to select, but luckily I found the wonderfull CacheAdapter written and maintained by Paul Glavich. It allows you to switch between different caching types by altering web/app.config. The current cache options are:

• No cache
• .Net 4.0 ObjectCache
• Asp.Net Cache
• Appfabric cache
• Memcached

This improved performance bigtime wiithout a lot of effort.

Up next: blocking IO requests hurt bigtime

After considering the option to convert our WCF service into an async one (way to much effort for now), I discovered that one can easily improve performance of parallel requests by replacing simple calls with their async counterpart and just wait for them to complete...

I wrote a simple benchmark to download my homepage 20 times using async & sync approaches, and the results were unbelievable:

The async method was 28 times faster then the sync method...

How does it work ?

It is actually quite simple; I wrote a little helper function in a static class called Wait.Async. Here is the complete sample code with the stats included, let us take a look first:

So, the code is really simple; I simple wait for the ResetEvent to be set... As this is quite repetetive code I wrote a little helper for it...

Can we use it to optimize database access ? Even Linq2SQL ?

It is a little hackerisch for LinqToSQL and it has its limitations for the queries, but it is actually quite easy to do so... As we are a big fan of the community and like to give back, we offer you the code we use to improve LinqToSQL scalability.

Unfortunately non-selects are AFAIK (close to) impossible to make async, so we currently simply opt to update the cached values directly and process the SQL on a background thread...

What's up next ?

For now, we have the tools and options in place to start optimizing the code... We applied the principles to 2 execution paths that require performance, and reached our initial performance goal. So now we need to optimize other paths as well. As my pseudo-fulltime consultancy period for Paycento is about to end next week, I can only assume it will be quite a busy week.. Fortunately, I will still be a member of the Paycento team, even though it will (for now) be more on an ad-hoc basis...