The SoftSource Community

WinDbg Extension sos2.dll v1.2 released

Prashant — Tue, 18 Nov 2008 10:28:03 GMT

Version 1.2 supports a command to dump GC Handles by type. Actually, support for this command was suggested by Ilya Ryzhenkov (ReSharper Product Manager at JetBrains) .

Download it from here

To Do List is to dump all the handles for a type along with stats with an option to specify the # of handles. For example, if there are 100 strong gchandles, you may want to print just 10 of them or so

GCHandlesByType(gcht)

————————————————
Get GC Handles Stat by Type
————————————————
!GCHandlesByType provides statistics about GCHandles by type.
Supported Types are
1. Pinned(p)
2. AsyncPinned(ap)
3. Strong(s)
4. WeakLong(wl)
5. WeakShort(ws)
6. RefCount(r)
Example Syntax
!gcht -t <type>
type specified is not case sensitive, for example command syntax to print the
stat for Strong handle type is

0:003> !gcht -t strong
||
0:003> !gcht -t s
||
0:003> !gcht -t Strong
Strong GC Handle Statistics:
Strong Handles: 15
Statistics:
      MT    Count    TotalSize Class Name
793040bc        1           16 System.Object[]
79330fb8        1           28 System.SharedStatics
79331e38        2           48 System.Reflection.Assembly
79330ec0        1           56 System.Threading.Thread
79330c30        1           72 System.ExecutionEngineException
79330ba0        1           72 System.StackOverflowException
79330b10        1           72 System.OutOfMemoryException
793310cc        1          100 System.AppDomain
793325b0        4          144 System.Security.PermissionSet
79330cc0        2          144 System.Threading.ThreadAbortException
Total 15 objects

Download it from here

ADPlus configuration file to the rescue

Prashant — Fri, 14 Nov 2008 05:43:57 GMT

ADPlus Configuration file to the rescue

Someone asked me about getting a memory dump on breakpoints in production environment.

Usually, you won't have the luxury of attaching a debugger and inserting a breakpoint in production environment. However, you can still get a memory dump under different conditions or execute a command line option using ADPlus configuration file. AdPlus does support -hang switch to take a memory dump of a process anytime but that's not good enough if you need to take a memory dump on a particular first chance exception or even when a breakpoint is hit.

ADPlus script supports configuration file with -c switch to create a memory dump of a user mode win32 process. You should read more on ADPlus configuration file on WinDbg help.

Below is the example of a ADPlus configuration file, which will create a memory dump under the following conditions

When the application throws an unhandled exception with the exception code 0x80000001, a guard page exception which occurs when you access for example a stack's guard page.
Creates a full dump, when breakpoint hits the function kernel32.dll!UnhandledExceptionFilter
Creates a mini dump. When breakpoint hits the function kernel32.dll!SetUnhandledExceptionFilter

<RunMode> CRASH </RunMode>

<Option> Quiet </Option>

</Settings>

<!--

PreCommands is included to change the symbol path for kernel32.dll, the first command .sympath sets the symbol path to c:\windows\system32(kernel32.dll location and the 2nd command reload the kernel32.dll defaulting to export symbols.

The reason for loading the export symbols has to do with setting a breakpoint in kernel32.dll functions as described in my last blog entry

-->

<Cmd> .sympath c:\windows\system32 </Cmd>

<Cmd> .reload /f kernel32.dll </Cmd>

</PreCommands>

<Actions1> MiniDump </Actions1>

<Actions2> FullDump </Actions2>

</Config>

</Exceptions>

<NewBP>

<Address> kernel32.dll!UnhandledExceptionFilter </Address>

<Actions> FullDump</Actions>

</NewBP>

<NewBP>

<Address> kernel32.dll!SetUnhandledExceptionFilter </Address>

<Actions> MiniDump </Actions>

</NewBP>

</Breakpoints>

</ADPlus>

Below is the command line snapshot to run adplus configuration file with "exception.cfg"

Breakpoint gotcha with kernel32.dll microsoft public symbols

Prashant — Thu, 13 Nov 2008 07:10:51 GMT

While debugging crash dump generation issue as described in my previous blog on Dr Watson gotcha, I noticed that you can't set a breakpoint on kernerl32 functions since microsoft symbols server gives you access to stripped public symbols only. This is one of those scenario where you would rather have export symbols.

Scenario

While doing live debugging or attaching a debugger to generate a dump when it hits a breakpoint on kernel32!SetUnhandledExceptionFilter

Steps using WinDbg

Run the following command

0:021> bm kernel32!SetUnhandledExceptionFilter

You can use bm to set a symbol breakpoint that matches the pattern.

Gotcha

If you have symbol server path set correctly pointing to microsoft public symbol server, WinDbg will display the following message and it suggest you to switch to export symbols

No matching code symbols found, no breakpoints set.

If you are using public symbols, switch to full or export symbols.

How to Switch to Export Symbols to set a breakpoint?

Run the following commands

0:000> .sympath c:\windows\system32

Symbol search path is: c:\windows\system32

This will set your symbol path to kernel32.dll which should be under your windows system folder in my case it is "c:\windows\system32"

0:000> .reload /f kernel32.dll

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Windows\system32\kernel32.dll -

Don't worry about ERROR message because this is what we want, we want it be set to export symbols. .reload command will reload the symbols for kernerl32.dll defaulting to export symbols

0:000> bm kernel32!SetUnhandledExceptionFilter

breakpoint 6 redefined

6: 7617d16f @!"kernel32!SetUnhandledExceptionFilter"

And your breakpoint is set using export symbols and of course you can use depends for all the exported symbols.

Debugging Rules

Prashant — Wed, 12 Nov 2008 11:02:25 GMT

Check out this book from David Agans on Debugging Rules (http://www.debuggingrules.com) Below is the must have poster for all Debuggers

we were done with dr watson or so you thought

Prashant — Mon, 10 Nov 2008 12:30:07 GMT

Scenario

cdb debugger(using adplus script) fails to create a crash dump although application is terminating unexpectedly. First thought, may be stack corruption.

Steps to resolution

1) Get a mini dump when it hits a breakpoint kernel32!UnhandledExceptionFilter

2) Get a mini dump when it hits a breakpoint kernel32!SetUnhandledExceptionFilter

Surprise

1) 002eebb4 6dcbc89f 00000000 0040f7f0 c0000417 kernel32!SetUnhandledExceptionFilter 002eeeec 6dc802fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xec

2) 002eebb4 6dcbc8ac 002eec14 0040f7f0 c0000417 kernel32!UnhandledExceptionFilter 002eeeec 6dc802fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xf9

It turns out CRT - C runtime library is hardcoded to use Dr Watson. You will notice that SetUnahdledExceptionFilter is called with NULL so basically it is deleting any unhandled exception filter.

Following is the stack trace

002eebb4 6dcbc8ac 002eec14 0040f7f0 c0000417 kernel32!UnhandledExceptionFilter
002eeeec 6dc802fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xf9
002eef34 6dc80366 045b0030 ffffffff 00000001 msvcr90!fread_s+0x4a
002eef50 6ef3105e 045b0030 00000001 000001f4 msvcr90!fread+0x18

Root Cause Analysis

1) Crash dump was not getting created because it was using fread with invalid runtime parameter resulting in calling hardcoded dr watson

2) Event log also shows the Exception Code = 0xC0000417, defined as #define STATUS_INVALID_CRUNTIME_PARAMETER ((NTSTATUS)0xC0000417L) in "NTSTATUS.h"

prior to windows vista, DrWatson is installed as the default post mortem debugger. But you always have an option to set WinDbg or NTSD or any other debugger to change post mortem debugger. However, you don't have Dr Watson in Windows Vista ( now you have windows error reporting service). So, We ran into an interesting issue while trying to get a crash dump for a user mode win32 process.

More Information

After googling on _invoke_watson, got more information on it from microsoft

A proposal to make Dr.Watson invocation configurable

Ruby on Rails Setup with DreamHost

Stuart Thompson — Tue, 28 Oct 2008 13:29:21 GMT

As step 1 of my latest experiment, I've started by configuring a Ruby on Rails site with DreamHost. This actually involved quite a bit of setup work, which is not unusual when remotely hosting a new website. I have documented the steps as fully as I can for anyone else curious about trying development in Rails.

Step 1 - Signup with DreamHost
You can sign up using this link or use the promo code LETMETRYRAILS and receive $50 off (i.e. no setup fee).

Step 2 - Configure your domain
The next step is to configure your domain. We'll start at the DreamHost configuration panel. Select Domains and then Manage Domains from the left-hand sidebar. This will bring up the list of domains in your account. Click the Edit button in the Web Hosting column.

This will open the settings page for that domain. Check the FastCGI and Passenger (mod_rails) checkboxes. Finally click the Change fully hosted settings now! button (note that this button is not at the bottom of the page, rather at the end of the first section).

FastCGI support allows your Rails project to run much faster. The Passenger (mod_rails) option allows your Rails application to be configured much more easily and is the preferred DreamHost method for hosting Ruby on Rails applications.

Step 3 - Configure your FTP/SSH user
We will be performing much of the setup and configuration of the website via SSH. We will need to setup a user account for this purpose. During your initial setup you should have created an ftp account for your domain, probably with the same name as that domain (if you accepted the defaults). Select the Users and then Manage Users options from the left side-bar. Click the Edit button next to your user in the Actions column.

This will open the following page where you can edit your user. Change the User Account Type to Shell account and select the /bin/bash shell type. Finally, hit the Save Changes button at the bottom of the page.

This enables the user to connect using a Telnet/SSH client, which is exactly what we are going to do next.

Step 4 - Download some tools
Next we're going to download the tools we'll need to create and configure our rails website. We'll need the latest release of Ruby, MySQL, HeidiSql, and PuTTY. HeidiSql is a free gui for maintaining MySQL databases. PuTTY is a free Telnet/SSH client that we'll use for remote configuration of our site. Note that you do not need to signup on the MySQL site to get the download. Click the No thanks, just let me download link.

Ruby 1.8.6-27 RC1 can be downloaded here.
MySQL 5.0 Community Server can be downloaded here (select the Windows ZIP/Setup.EXE (x86) option)
HeidiSQL 3.2 can be downloaded here.
PuTTY can be downloaded here.

Step 5 - Installing Rails
Now that we have the necessary tools installed it is time to setup Rails locally. Fortunately, Ruby comes with everything we need to get setup and running, including the WEBrick web server that we can use to run and test our site before deployment. When installing Ruby, be sure to check the Enable RubyGems option on the first page of the installer.

Once the installer completes, open up a command window and execute the following commands:

cd ruby/bin
gem update --system
gem install rails --version 2.1.2

This will use the gem system to install rails locally. If everything is successful, your command window should look something like this:

You may need to restart your system at this point. Ughh. This is because Windows Vista and XP sometimes struggle to detect that new environment variables were added to the $PATH. Unless you relish fully qualifying the C:/Ruby/bin directory on the next few commands, I'd advise you restart now. It's also not a bad idea since we did just install four new programs. I've found Windows seems to run more smoothly if you reboot after installing new things.

Step 6 - Create your Rails site locally
Now that we have Rails setup, let's create a simple site locally. I created a new directory on C:\ called projects in which to develop the site. Open a command prompt and type the following:

cd \
md projects
cd projects
rails MyRailsSite

This will create your rails site in C:\projects\MyRailsSite. Before we add our first view and controller, let's quickly check that everything wired up properly. To do this we need to start the WEBrick server:

cd C:\projects\MyRailsSite
ruby script/server

Vista may ask if you wish to unblock Ruby, which you do. Your command prompt should look something like the following:

You can now navigate to http://127.0.0.1:3000 to see your site. If everything is configured correctly you should see the Ruby on Rails welcome page:

We can shutdown the WEBrick server by pressing Ctrl-C in the command prompt window.

Now let's create our first controller and view. We're going to keep it simple and put up a page that displays the message "Hello World!". At the command prompt type the following:

cd projects\MyRailsSite
ruby script/generate controller HelloWorld

This will generate a new controller for you as well as a new view. If you aren't familiar with MVC development, just follow along for now until the sample becomes clear. The rest will come in time and with additional reading and practice. For now it is suffice to say that a controller will perform the work for a web page request where the view will display the results of that work.

Let's edit the controller that was generated. To do this, open up an explorer window, navigate to C:\projects\MyRailsSite\app\controllers, right-click over the hello_world_controller.rb file and select Edit from the context menu. This should open the controller file in the SciTE editor. We need to add lines to the class to respond to the index action. The index action is the default action that will be performed for a controller. To do this, edit the file to look like the following:

This controller will now set the variable @message to the literal "Hello World"; when the index action is requested. Controllers perform actions.

Save the file and close it. Now we will create a view to display the message. Navigate to the C:\projects\MyRailsSite\app\views\hello_world directory in explorer (notice how Rails created the view directory for us). Right-click in the directory and create a new file called index.html.erb. Now right-click this file and open it in your favorite editor (Notepad will suffice). Add the following to the file:

Save the file and close the editor. We're almost ready to test out our new controller and view. First, however, we need to configure a route for our site. A route is a mapping from an end-point (url) in our site to an actual controller and view that will fulfil the request. In explorer, navigate to C:\projects\MyRailsSite\config, right-click the file routes.rb and select Edit from the context-menu. This will open your routes file in the SciTE editor. Almost the entire file is commented out. Most of this file contains instructions for configuring custom routes. We're going to comment out the last two lines of the file and insert a single mapped route of our own. Edit the end of the file to look like the following:

This line connects the helloworld route to the hello_world controller for the index action. The action specification isn't actually necessary as the index action is assumed by default if none is specified, but I prefer to be explicit.

There is one final thing we'll need to do before we can browse to our page locally. We need to install and configure MySQL.

Step 7 - Install and configure MySQL
This is a very short step. Run the MySQL setup package you downloaded earlier and select the Typical installation option. Then hit Next->Next->Finish all the way to the end. When the configuration wizard runs, choose Detailed Configuration, then Developer Machine, then Multifunctional Database. When asked for a location for the Inno files, I used C:\projects\MySQL. Choose Decision Support (DSS)/OLAP then leave both boxes checked on the next dialog with 3306 as the port number. Choose Standard Character Set, then check Install As Windows Service and leave Include Bin Directory in Windows PATH unchecked. Enter a new root password, leave Create An Anonymous Account unchecked. Finally, hit Execute to finish the configuration wizard. MySQL is now installed and configured.

Step 8 - Create a database for our project
We'll need to create a database for our project. To do this we'll need to install HeidiSql. Let's run that installer now. I use the defaults from the installer except on the Select Additional Tasks dialog where I uncheck Associate .SQL Files with HeidiSQL. When the installer completes it will launch the HeidiSQL gui. Click the New button at the top of the first dialog. Name the connection MyRailsSite, click OK, and then enter the password you selected during your MySQL installation into the password box provided. The first thing we're going to do is create a new database. Right-click over root@127.0.0.1 in the left side-bar and select Create database from the context menu. Enter MyRailsSite as the database name and click ok. Now select Tools->User-Manager from the top menu. We're going to create a new user to avoid using the root user in our application. For the Username type myrailsdbuser, set From Host to localhost, and type a password for your new user. Don't use the same password you used for root. In the Allow access to list, left-click over the myrailssite database (it will turn yellow). Now click Add User. Then click Close to close the user manager dialog. Now we've created a database and a specific user for our Rails application to use.

Step 9 - Configure Rails to use the database
Finally we need to configure our Rails site to use our new database. In an Explorer window, navigate to C:\projects\MyRailsSite\config, right-click the database.yml file and edit it in your favorite editor (I use TextPad but NotePad will suffice). Edit your file to look like the following:

Step 10 - Testing the site locally
We're finally ready to test the site locally! Open up a command window are run the following to start the WEBrick server:

cd c:\projects\myrailssite
ruby script/server

Then open a browser and navigate to http://127.0.0.1:3000/. You'll notice that the Rails welcome page still displays. However, change the url in the browser to point to http://127.0.0.1:3000/helloworld. You should now see your view displaying the message "Hello World!". Let's remove the default index page now and then make your default route point to your helloworld view. To do this, open up Explorer and navigate to c:\projects\myrailssite\public. This is your site's public directory. From here, delete the index.html file. That's the welcome page gone; we don't need it anymore. Now navigate to c:\projects\myrailssite\config and open the routes.rb file again. Add the following line just before the map.connect line we added earlier:

map.root :controller=>"hello_world"

This configures the root route to point to our hello world controller. As we stated earlier, there is no need to specify the index action as it is assumed by default. Save and close the file, then browse to http://127.0.0.1:3000 once more. Your site should now be displaying your "Hello World!" message. Note how the url is just http://127.0.0.1:3000 and doesn't have helloworld at the end. This is an important point about routes. You aren't navigating a particular directory structure on the site, you are simply navigating routes and actions. The controller could be named anything you like as long as the route for a particular end-point correctly routed requests for that end-point to your controller. For now, let's stop the WEBrick server and close the command window. The final task is to get your Rails site up to DreamHost and serve your "Hello World!" request remotely.

Step 11 - Configuring DreamHost for your site
This final configuration step may seem a little confusing at first, but only if you aren't familiar with Linux and BASH. It doesn't take very long and it isn't important to understand how it works because you'll only need to do it once. We're going to install version 2.1.2 of rails to your remote host because that's the version that Passenger (mod_rails) needs in order to run your site. This is where we'll need PuTTY. Fortunately there is no installation step with PuTTY. It is simply an executable that you download and run.

When you run PuTTY, you will need to supply the username and password of the user we configured in step 3. This will connect you to the server that will host your site. The following instructions are commands to enter into the BASH shell as well as instructions on editing your ~/.bash_profile file (which is similar to environment settings for the command shell in Windows). The first part of these instructions is about editing your ~/.bash_profile file to ensure that some paths and other required settings are configured before beginning installation. These instructions have been adapted from the DreamHost wiki article on this topic.

Before we update rails, let's first create our Rails site remotely. To do this, first run PuTTY. For the Host Name you will supply your domain name in the form www.MyDomainName.com. Then click Open. I choose No from the next dialog as I don't want to store anything in the registry. You will now enter the username and password for the user we created in step 3. When you connect, type ls and hit enter. This is the BASH command to list the contents of the current directory. You should see MyDomainName.com listed in the output. We are going to navigate to that folder and then create a rails site inside that folder.

cd MyDomainName.com
rails MyRailsSite

This will create a Rails site on the DreamHost server, the same way we did it locally earlier. We're going to create the site like this and then upload our site files from our local machine to the remote site. To upload our files, we'll use an Explorer window and ftp. First, however, we need to create a copy of our database.yml and environment.rb files. This is because those files will be different on our remote host than they are locally. Create a copy of each of those files called database.yml.live and environment.rb.live now. Now edit the database.yml.live file to look like the following:

Edit your environment.rb.live file and uncomment the following line (its on line 5):

ENV['RAILS_ENV'] ||= 'production'

This is important because running in development mode under FastCGI will cause memory leaks on the DreamHost server. They will catch them and clean them up but it's better to run in full production mode and not cause problems.

One final change we need to make locally is to update the c:\projects\myrailssite\config\environments\production.rb file. Comment out the config.action_view.cache_template_loading = true line as it causes problems with the Passenger product that will host our site.

Now we're ready to install Rails 2.1.2 remotely. Switch back to your PuTTY session (or log in again if you exited). Let's first edit the ~/.bash_profile file. To do this we will be using the nano text editor. To open the ~/.bash_profile file in nano, type the following:

nano ~/.bash_profile

Now you can use the cursor keys to navigate the file. We need to add the following lines to the end of the file:

# Path settings
export GEM_HOME="$HOME/.gems"
export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8"
export PATH="$HOME/bin:$HOME/.gems/bin:$PATH"
export RUBYLIB="$HOME/lib:$RUBYLIB"

# Stops the DreamHost processes from killing the gem install process
alias gem="nice -n19 ~/bin/gem"

To save your changes press Ctrl-O then hit enter. Press Ctrl-X to exit the editor and return to the shell. You can now type cat ~/.bash_profile and the contents of the file should be printed out. This is a good way to check that your changes were saved and that you typed everything correctly.

Now we need to perform the installation. The following commands will achieve this:

cd ~
mkdir .gems

mkdir bin lib src

cd ~/src
wget http://rubyforge.org/frs/download.php/43985/rubygems-1.3.0.tgz
tar xzvf rubygems-1.3.0.tgz
cd rubygems-1.3.0
ruby setup.rb --prefix=$HOME

cd ~/bin
ln -s gem1.8 gem

For more information on what each of these steps is doing check the DreamHost wiki article. We can check that the path and gem versions are correct by typing the following:

which gem
gem -v

These commands should return /home/USERNAME/bin/gem and 1.3.0 respectively. Now we can execute the command to install Rails 2.1.2, which is:

gem install rails --version 2.1.2

What we've accomplished here is to install RubyGems and Rails 2.1.2 to our hosted server. The last thing we need to do to make everything work is to use rake to freeze our version of Rails to 2.1.2 so that Passenger will use our Rails installation when running our site. The commands to do this are as follows. Please note that you need to supply your domain name in the format MyDomainName.com and the name of your Rails site in place of MyRailsSite in the first command. This simply changes the current directory so that we're running rake in the right place.

cd ~/MyDomainName.com/MyRailsSite/
rake rails:freeze:edge TAG=rel_2-1-2

That's it! Phew. That felt like a lot of work, but at least you only have to do it once.

Step 12 - Deploying and testing your site remotely
Now that we've set up the environment with DreamHost, the only task left is to deploy the site remotely and test it out. Before we do that though, we are going to check one last configuration setting in the DreamHost control panel, just to be sure that everything is setup correctly. If you navigate to Domains->Manage Domains in the left side-bar and then click the Edit button again (see step 2), then check out the Specify your web directory setting. This is where DreamHost will send requests to MyDomainName.com. It should be set to:

/home/username/MyDomainName.com/MyRailsSite/public

By setting the directory to the public directory of your Rails site, it means that users cannot navigate outside of that directory and browse places that they aren't supposed to be.

Let's finish up and deploy our Rails site from the local machine to the remote server. We'll use Explorer and ftp to do this. Open up Explorer and type ftp://ftp.MyDomainName.com into the address bar. Now open up the MyDomainName.com and then MyRailsSite folders. You will need to copy your app directory from your local machine at c:\projects\myrrailssite\app up to the remote server. When you have copied these files, navigate to the pubilc directory on the remote server and delete the index.html file. We don't need the Rails welcome page on live anymore either! Then navigate to the config directory and copy the routes.rb, environment.rb.live and database.yml.live files up to the corresponding folder on the server. Don't forget to rename these to environment.rb and database.yml on the remote server. We only renamed them so that we could have two different files locally. Finally, navigate to the config/environments folder and copy the production.rb file that we edited up to the corresponding location on the remote server. From now on, when you deploy your site, you will only have to copy the app folder to the remote server (unless you change config settings that is).

The final step before we can test our site on live is to setup the MySQL database on the DreamHost side to match the one we set up locally. Since we didn't create any database tables, this is fairly simple to do through the DreamHost interface. Go to Goodies->Manage MySQL and repeat the same steps here as we did in step 8. You can also use HeidiSQL if you want to manage your database remotely that way. However, for now I recommend using the DreamHost web interface to do this work. All you need to ensure is that the database name and username/password for that database match the settings you specified in your database.yml file. I'm going to cover using databases in Rails in more detail in a future article. For now we just need to get an empty database and a user that has access to keep Rails happy.

Congratulations! Assuming that everything was wired up correctly, you should now be able to open a browser and navigate to http://www.MyDomainName.com to see your site being hosted remotely.

Hopefully this article will help you get up and running with a basic Rails site that is hosted remotely. The focus of this article was understanding the relationship between running a Rails site locally and hosting it on a remote server. Setting up your development in this way from day one will save a lot of headaches later when it comes to deployment time. Developing and testing locally, then deploying to a live host means that problems can be identified early and that changes can be tested locally rather than on a live site. I'm going to expand upon this article as part of my ongoing experiement with ASP.NET, Silverlight, PHP, and Ruby on Rails. Next we're going to develop a database, understand models using ActiveRecord, and dig a little deeper into Rails. For now: happy hosting!

The ASP.NET, Silverlight, PHP, Rails Experiment

Stuart Thompson — Mon, 27 Oct 2008 11:06:37 GMT

I've been working with Microsoft technologies for over ten years, specifically Microsoft's web development technologies. During that time I have built a variety sites ranging in scope from e-commerce, educational, service provision, SaaS, to dashboards, portals, internal tools, and even SAP-migration. While I praise the technology stack for what it allows me to do, it also comes with a whole host of limitations that make it hard for me to deliver the experience my clients would like. Throughout this decade of web development I have heard a lot from those around me about alternative technology stacks that promise all manner of improvements and experiences. Unfortunately there is a lot of resistance from any one technology stack to listen to and learn from the stacks around them, most of the cross-platform feedback comes in the form of "Microsoft sucks" or "PHP is for fan-boy script kiddies". The truth is that a large number of successful business are hosted on a wide variety of platforms. Each technology stack has to at least be viable for that to be true. The problem is getting at the truth without the zealous noise of "my stacks better than your stack" getting in the way.

I decided that it was time for me to branch out and become more informed on these technologies myself so that I could truly understand the differences and what made each of them unique. I'm hoping that as part of this experience I can not only become a better developer but also learn the problems that Ruby on Rails or PHP addresses. I'll like to be able to speak from an informed standpoint about the differences between the Microsoft and open-source approaches as well as better understand the problems each of those development worlds faces on a daily basis. Either way, it's going to be a heck of a lot of fun.

The experiment I have in mind is to build the same web-site using four different technologies. From the Microsoft camp I have chosen ASP.NET and Silverlight. From the open-source perspective I have chosen PHP and Ruby on Rails. I chose PHP and RoR because they are two technologies that I have always wanted to learn. Learning both of those will also require some digging into Apache and Linux, another two areas that I have too little direct experience with to effectively participate in an informed conversation.

Our horses are ready to race. In terms of hosting, I have selected DreamHost for the PHP and Rails sites, and I'm going to use StormHosts for the ASP.NET and Silverlight projects (assuming that SH can do .NET 3.5, which I believe I saw at some point). The reason I'm using remote hosting services for this experiment is that I don't believe you get the true experience of a technology stack until you host it in this way. It is very easy to get a Rails site up and running using WEBrick and a local installation of Ruby. Similarly, it is easy to get an ASP.NET site running on the personal web server that ships with Visual Studio 2008.

The first phase of the experiment is to get a "Hello World!" site up and running for each of the four environments. The specifications are that a single end-point be accessible that displays the text "Hello World!" and that this end-point is accessible in both Internet Explorer 7 and FireFox 3.

Reflector to go to RedGate

Stuart Thompson — Wed, 20 Aug 2008 07:21:13 GMT

Lutz Roeder has decided to move on from Reflector. RedGate will be taking over the project from now on. Fortunately, they have agreed to maintain the community edition of the software.

http://blog.lutzroeder.com/2008/08/future-of-net-reflector.html

NDepend Part II - Beginning CQL

Stuart Thompson — Tue, 19 Aug 2008 09:25:12 GMT

In my continuing use of NDepend on both my work and home projects, I have been investigating CQL more and more, especially from the perspective of use in continuous integration. NDepend provides some incredibly powerful analysis features and I find the idea that constraints can be applied and warnings reported as part of day 1 builds to be extremely appealing. I'm all about keeping code clean, constrained, and consistent. NDepend provides another great tool in my kit to ensure that my own code is marshalled to my standards from the very first line that is written. As David Muhondro put so eloquently in his blog post, NDepend is static analysis on steroids. When most users encounter CQL for the first time, the response is "wow, this looks really powerful, now what does it mean?". I've started by taking the stock queries that are included in a basic NDepend project and understanding what they mean and the warnings that they are expected to generate. By understanding the warnings and their purpose in the project, I began to understand how I could tweak those queries and even write my own to enforce the rules I'd like my code to live by.

Let's start by looking at an example. After creating a new NDepend project and pointing it at a list of assemblies to analyze, the default CQL queries that are included are grouped into the following categories:

Code Quality
Design
Unused Code / Dead Code
Encapsulation
Diff / Changes / Evolution
Test Coverage
Purity / Immutability / Side-Effects
Naming Conventions
.NET Framework Usage
Statistics
Samples of Custom Constraints
Constraints extracted from Source Code

I started by looking at the Unused Code / Dead Code queries, specifically at the Potentially unused methods query. By double-clicking on the query it is opened in the CQL query editor and displays the following:

 // <Name>Potentially unused methods</Name>

WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE

MethodCa == 0 AND            // Ca=0 -> No Afferent Coupling -> The method is not used in the context of this application.

!IsPublic AND                // Public methods might be used by client applications of your assemblies.

!IsEntryPoint AND            // Main() method is not used by-design.

!IsExplicitInterfaceImpl AND // The IL code never explicitely calls explicit interface methods implementation.

!IsClassConstructor AND      // The IL code never explicitely calls class constructors.

!IsFinalizer                 // The IL code never explicitely calls finalizers.

Breaking it Down
Let's break this down a little. First of all it is important to realize that this is not SQL, despite any similarities in the syntax. The first line selects the top 10 methods that satisfy the query and then raises a warning if the total number of methods is greater than 0. The reason for the seemingly superfluous top 10 selection is that if the count is greater than zero then those top 10 will be displayed as the results of the query. To avoid swamping with noise the query is restricting to only the first ten offenders. The next line is the first part of the WHERE clause and looks for methods with an CruiseControl to bring these warning and reports into the daily life of your projects. While static analysis en masse isn't for everyone (not even for me), a small list of CQL queries can help you perform necessary housekeeping without having to do the hard hunting work yourself. The inevitable question is whether or not this overlaps with FxCop. There is some overlap, but they are both just tools from which you should pluck the best of the functionality you like and strike down the cruft that gets in the way. There are some cases where writing an FxCop rule specific to your application is a good thing, however I'm finding that writing CQL queries for warnings lends itself a little better to how I think about static analysis in the first place. The key is finding that sweet spot in the middle.

Designing for Testability

Stuart Thompson — Wed, 30 Jul 2008 10:56:41 GMT

Automotive manufactures have long been on the vaunted TDD bandwagon. For years they have harnessed the amazing potential of unit testing, verifying components individually prior to the expensive and time-consuming exercise of assembling them to make a completed vehicle. Integration testing in the automotive world is very expensive and to find that a single component failure has wasted a fleet of test vehicles as well as many months of construction and fabrication is close to unacceptable. However, they have also long faced the question of when and where to design for testability; a line still not clearly defined in any industry.

It makes perfect sense that if a component is to be tested in isolation from the system it will eventually be integrated with that a certain amount of planning is necessary in order to facilitate that testing. This means that certain designs, while technically and functionally brilliant, may need to be modified if they are ever to be tested. The question is where to draw the line. All too often in software engineering the line is drawn firmly on one side or the other, rarely in a position of balance.

I can understand on a modern motor vehicle with multiple on-board computers performing a variety of related tasks that there is a need to interface with and test those computers thoroughly. I'm sure that to the component manufacturers the need for extra diagnostic access ports and programmable interfaces is a hindrance to their development; after all they are working in a very closed system. However, to the test engineers the ability to simulate all manner of failures and receive highly detailed information about everything the computer is doing is absolutely indispensible. They cannot guarantee the component functions correctly without it.

Unfortunately this desire for components to be testable can easily get out of hand. Consider that a test engineer is tasked with ensuring that the onboard computer functions at various speeds. Unforuntately his equipment is bulky and doesn't fit inside the vehicle. Even at low speeds he finds that keeping up with a car in motion requires him to run alongside pretty quickly. Should he require that the car not go faster than 8mph in order to ensure that he can completely guarantee the functionality of the computer for the legal range of speeds he is able to test? Something tells me that the manufacturer wouldn't be able to sell too many of that particular model.

Instead the test engineer must find other ways to test the car at high speeds. Custom models of the car are constructed that allow a connected rig to be tethered to the car, carrying all of the the test equipment alongside even at high speeds. At even higher speeds, motion simulations and car treadmills are employed to allow for testing that the tethered rig is unsafe for. The design of the car wasn't limited to a maximum speed of 8mph, rather a compromise was found to allow the right balance of functionality and testing.

In software engineering, we have a variety of tools and frameworks at our disposal to accomplish similar testing of software. Unit testing frameworks, mock frameworks, dependency injection containers; they are tools. Each plays a specific role and solves a particular flavor of problem, however none of them are all-encompassing and all of them require thought before use.

There is a growing epidemic of code grown out of the desire for "high coverage" unit testing. However, some of the motivations behind this testing appear to have taken an 8mph turn somewhere; for the worse. In an effort to improve code quality, the initiatives towards testing have started to build 8mph cars with 6 wheels and no doors. In terms of test coverage the cars are truly superb, but from the outside many consumers are left scratching their heads and wondering where it all went wrong.

I usually find that if a particular design decision is inhibiting the testability of a component, one of three things is happening:

The design is so tightly coupled and stable that no testing could hope to penetrate the darkness of the black box surrounding it. (i.e. a dll with one 5,000 line method inside that does "something")
The design is so abstract that is doesn't actually do anything. (i.e. a dll containing only interfaces - test that suckas!)
The testing tool being applied is the wrong one for the job.

As a software engineer, I can fix 1 and 2. Fortunately with good design and architecture we can actually prevent 1 and 2 from ever reaching QA in the first place.

There isn't much I can do about number 3. This is where I ask for the line to be moved a little with regards to the tool that is being employed. Here are a few examples of hammering screws and screwing nails.

Static methods, unless invoked through reflection, attract coupling from consumers. That is to say that if an assembly contains a static method, there is a good chance that something calling that static method has a strong reference to the assembly that exposes it. This is not a bad thing and there are many places where such a design makes perfect sense. However, in certain situations, this can cause mocking frameworks to perform more work than necessary to set up the premise for a unit test because they can't mock away the reference the exposed assembly that performs work. This is because mocking frameworks are ideal for mocking abstract implementations and poor for mocking tightly coupled components. There is nothing to mock because everything is explicitly stated. See screw, swing hammer. Static methods are perfect for unit testing thanks to the magic of reflection. Inspect the method contract (the logical contract not the physical source code - i.e. what, in English, does it do) and based upon that contract construct a suite of tests to affirm that the method is indeed doing what it promised it would.

Service assemblies, such as configuration loaders or proxies, tend to operate more upon interfaces than implementations. That is because the value they add is an operation upon some implementation of an interface rather than a concrete process defined up front. This is the provider model. We write a set of interfaces describing the kinds of things a provider can do. We also write a set of services than help those providers achieve some work. We don't write the actual providers themselves because this is a plugin model. Well, how do we test a provider that hasn't been written yet? Wait, this could be hard in unit testing. We call a method on an interface but have no way to know what the "as yet unwritten" provider will do in that method call and have no way of knowing whether the result is correct. NTemporalDisplacementUnit 1.0 hasn't been released yet so we can't do it in a state of temporal flux either...panic! But wait, in thinking about the problem further we realise that we don't care what the provider does. The contract we are providing is that we will interact with a provider in a predictable and documented fashion. As long as we interact with the interface in the correct way and at the correct time, our part of the contract is fulfilled. Woohoo! That's what a mock framework is for. We mock up a provider and check that our service correctly interacts with the provider. Mock frameworks are great for this. Finally, a nail that we can hit with our hammer.

If you find that such thinking manifests itself in dogmatic emails that state "static methods shouldn't be used because they inhibit testability" or "interfaces are bad because we can't get coverage with them." then it might be time to inspect both the situation and the tools and see that the two are being correctly paired.

The goal of testing is to improve the quality of the end product, not to turn lots of lights a pretty shade of green. 8mph cars with lots of green LEDs and no doors for great dogmatic case study's and really awful consumer responses.

Planning for the worst

Stuart Thompson — Fri, 25 Jul 2008 11:28:28 GMT

"The status page we use to report service disruptions in progress is suffering a temporary misconfiguration."

Brilliance!

Technorati Blog Claim

Stuart Thompson — Tue, 15 Jul 2008 12:19:20 GMT

I claim this blog in the name of Stuart Thompson for my new Technorati Profile.

LINQ to Coffee

Stuart Thompson — Tue, 15 Jul 2008 10:55:41 GMT

Nice! A perfect addition to the reference material you keep within reach.

http://www.cafepress.com/linq.225122905

Debugging Into .NET Source Code

Stuart Thompson — Mon, 14 Jul 2008 11:04:31 GMT

Many have “heard” that you can now step into .NET source code when debugging in Visual Studio 2008. For those who have not been doing this since the start of the year, you can find excellent instructions for configuring Visual Studio to get the necessary symbols here.

I highly recommend understanding this process and making it a part of your debugging routine. It often becomes necessary to understand why the .NET framework is behaving in a certain fashion while developing. By stepping into the source code and using the watch window, I’ve saved myself a lot of hours of head-scratching. More often than not it’s as simple as understanding why an exception was thrown and the hidden meaning behind the message. Within minutes I can usually pinpoint how I was incorrectly using one of the libraries, fix the bug, and be on my way.

LINQ'd In #1 - Behavior Injection

Stuart Thompson — Mon, 14 Jul 2008 09:51:44 GMT

As part of a continuing set of articles on LINQ, I want to first take a moment to look at how the "natural" syntax of LINQ is turned into executable code. Once we understand how those expressions and compiled we can investigate ways to extend the code that is being executed.

First of all let’s consider a very simple LINQ expression:

from num in Enumerable.Range(1,9)
where num % 2 == 0
select num

This expression selects the even numbers in the series 1–9. Let’s first break this down into the actual code that is being executed. Behind the scenes LINQ is viewing this expression as a series of lambda expressions and extension methods. Our example is actually being run as:

Enumerable.Range(1,9).Where(n => n % 2 == 0)

Enumerable.Range(1,9) yields an IEnumerable<int>. By using Reflector we can discover that the signature of the Where method looks like the following:

public static IEnumerable<TSource> Where<TSource>(
this IEnumerable<TSource> source,
Func<TSource, bool> predicate)

This is clearly an extension method on IEnumerable<TSource> that takes a Func<TSource, bool> as its only parameter. While the actual internals of the Where method are actually a little more complex due to the way in which iterators have been abstracted, we could theorize that the behavior is roughly similar to the following:

public static IEnumerable<TSource> Where<TSource>(
    this IEnumerable<TSource> source,
    Func<TSource, bool> predicate)
{
    foreach (TSource t in source)
    {
        if (predicate.Invoke(t) == true)
            yield return t;
    }
}

“For each item in the supplied IEnumerable, invoke the supplied predicate and add the item to the returned collection if that predicate returned true.”

Note: If you haven’t encountered keyword yield before, check out this post about how it works.

Now that we understand how the statement is working internally, lets look at ways in which we can extend it. To understand the following trick we need to think for a second about how extension methods are implemented, or more importantly about how they are resolved.

When the compiler encounters a statement of the form:

<type>.<method>() (e.g. IEnumerable<int>.Where(…))

for which there is no explicitly defined method, it searches the current list of scoped namespaces for an extension method that matches the implied signature. The System.Linq namespace defines the extension method we’ve been examining above. However, what if we wanted to use our own method instead of the System.Linq version of Where()?

We might want to add in some logging statements when the predicate finds a match. Let’s do this right now by adding our own extension method for the IList<T> generic. To do this we need to define a static class that contains a static Where method that has this IList<T> as its first parameter:

public static class ListExtensions
{
    public static IEnumerable<T> Where<T>(
        this IList<T> source, Func<T, bool> predicate)
    {
        foreach (T t in source)
        {
            if (predicate.Invoke(t) == true)
            {
                Console.WriteLine(
                    String.Format("Predicate indicated success for value: {0}", t));
                yield return t;
            }
        }
    }
}

This method is almost identical to the theoretical version of Where we built above. The only differences are that it extends IList<T> instead of IEnumerable<T> and that it logs when the predicate finds a match prior to yielding the matched item.

Now let’s put together a little snippet that will exercise this extension method. We can actually use the first sample we used in this post and by simply adding ToList() to the end of Enumerable.Range(1,9) we’ll be calling our extension method for Where() instead of the built-in System.Linq version. Let’s look at that sample:

from num in Enumerable.Range(1,9).ToList()
where num % 2 == 0
select num

By simply adding the ToList() to the sample above we’ve caused the compiler to call our new Where() method. This gives a pretty clear idea of how LINQ is working internally and what’s actually going on when we use the new C# 3.0 language keywords. There is a lot less voodoo involved in the process that one might first believe!

Example Listing
The following is a complete example suitable for pasting into snippet compiler:

using System;
using System.Collections.Generic;
using System.Linq;

public class MyClass
{
    public static void RunSnippet()
    {
        Console.WriteLine("Results using the IEnumerable<T>.Where() method from System.Linq.");
        var enumResults =
            from n in Enumerable.Range(1,9)
            where n % 2 == 0
            select n;
        foreach (int n in enumResults)
            Console.WriteLine(n);

        Console.WriteLine("Results using the custom IList<T>.Where() method from this sample.");
        var listResults =
            from n in Enumerable.Range(1,9).ToList()
            where n % 2 == 0
            select n;
        foreach (int n in listResults)
            Console.WriteLine(n);
    }

    #region Helper methods

    public static void Main()
    {
        try
        {
            RunSnippet();
        }
        catch (Exception e)
        {
            string error = string.Format("---\nThe following error occurred while executing the snippet:\n{0}\n---", e.ToString());
            Console.WriteLine(error);
        }
        finally
        {
            Console.Write("Press any key to continue...");
            Console.ReadKey();
        }
    }

    private static void WL(object text, params object[] args)
    {
        Console.WriteLine(text.ToString(), args);
    }

    private static void RL()
    {
        Console.ReadLine();
    }

    private static void Break()
    {
        System.Diagnostics.Debugger.Break();
    }

#endregion
}

public static class ListExtensions
{
    public static IEnumerable<T> Where<T>(this IList<T> source, Func<T, bool> predicate)
    {
        foreach (T t in source)
        {
            if (predicate.Invoke(t) == true)
            {
                Console.WriteLine(String.Format("Predicate indicated success for value: {0}", t));
                yield return t;
            }
        }
    }
}