SQL

SqlDependency - Creating a Smarter Cache

Stuart Thompson — Wed, 13 Jun 2007 22:56:15 GMT

I recently got the chance to learn about a technology that has interested me for quite some time: SQL dependencies. The premise of my encounter with SqlDependency was to create a smarter cache. The project I'm currently working on was employing a sliding expiration policy to invalidate the contents of a cache of database lookup values and I proposed some research be conducted into using SQL dependencies to automatically invalidate the cache items when the corresponding database contents were changed. I had experimented with the SQL dependency classes when .NET 2.0 was first released using a test SQL 2000 database. The example I was using employed a polling strategy to keep the contents of an ASP.NET web cache item fresh. While the technology was cool, I was less than enamored by the fact that the web server was polling the database for changes. It felt a little too similar to setting a sliding expiration value of 5 seconds on the cache and pinging the database that way.

I was, however, aware of the fact that SQL 2005 notifications could be used to invalidate the contents of a cache without the need for polling. With this in mind, I set about investigating the possibilities of SQL dependencies using .NET 2.0 and SQL 2005. While this certainly isn't a new topic, I found that very little good documentation exists on the topic and decided to write this article to hopefully bring some clarity to others who might wish to take advantage of the technology.

When is this a good solution?
The gap that this solution is attempting to fill is the short window where data can be stale with a sliding expirationon cache items. If the data in the database has changed but the sliding expiration on the cache has not yet expired then the values in the cache will be out of date but still considered valid and hence returned to the UI. Using SQL 2005 notifications will invalidate the contents of the cache almost as soon as the database contents are changed. You will need to determine for your application how long data in the cache can be stale before the usefulness of the application is compromised. If the sliding expiration of cache items can be shortened to compensate satisfactorily then that may provide a "better" solution that SQL dependencies. However, if near real-time cache invalidation is desirable then SQL dependencies using notifications could provide a good solution.

How does the solution work?
The basic mechanics of the solution involve a cache object, a rowset within SQL (I'll simplify for now and just say a table), a trigger and a notification queue. There are many moving parts under the scenes that make the technology come together, but that covers the basic list of participants. In simplified terms, a rowset is read from the database and an object representation of that data is added to a cache. A SQL dependency is established upon that rowset and a callback method is registered for that dependency such that when the rowset changes the callback method will be invoked. A trigger is used within the database to detect when the rowset changes, and a notification queue within SQL Server is used to broker a message back to cause the callback method to be invoked. The callback method invalidates the contents of the cache. The next time the data is requested from the cache, the contents are found to be invalid and the process starts again.

Creating a Sample Application
This example uses Visual Studio 2005 and SQL Server 2005 to set up a sample database and two-tier web application running on the same machine to demonstrate use of the SqlDependency class as part of a caching strategy.

Let's start by creating a sample database called SqlDependExample. Add a table to the database called Color. This table has two columns: ColorID, and ColorName. It will represent a simple lookup table that contains a list of colors within your application. Make the ColorID column an int that does not allow nulls, the primary key and set (Is Identity) to Yes. Make the ColorName column a varchar(100) that does not allow nulls. Add a couple of rows of data to the table with the following values: 1 Red 2 Blue 3 Green 4 Yellow

Create a login on the SQL Server called SqlDependExampleUser that uses SQL Server Authentication, and choose an appropriate password. Add a user called SqlDependExampleUser to the SqlDependExample database for the SqlDependExampleUser login and grant the db_datareader, db_datawriter, and db_ddladmin roles.

Now we need to configure the database to be able to use the service broker and grant some additional permissions to the user we created in order that they can create and manage objects necessary for the notification strategy to work. The following script performs each of the necessary steps to configure the database and grant the required permissions. It is important to note that the first step requires SQL Server to have an exclusive lock on the database while performing the query, so be sure to close out any unused connections in the management studio; including the dialog that was used to create the Color table.

Open a query window and paste the following script: &nbps;(be sure that the selected database is SqlDependExample, not master)
ALTER DATABASE SqlDependExample SET ENABLE_BROKER
GO

GRANT CREATE PROCEDURE TO [SqlDependExampleUser];
GO

GRANT CREATE SERVICE TO [SqlDependExampleUser];
GO

GRANT CREATE QUEUE TO [SqlDependExampleUser];
GO

GRANT REFERENCES ON CONTRACT::[http://schemas.microsoft.com/SQL/Notifications/PostQueryNotification] TO [SqlDependExampleUser];
GO

GRANT SUBSCRIBE QUERY NOTIFICATIONS TO [SqlDependExampleUser];
GO

GRANT CONTROL ON SCHEMA::[dbo] TO [SqlDependExampleUser];
GO

GRANT IMPERSONATE ON USER::DBO TO [SqlDependExampleUser];
GO

Execute the script to enable the service broker for the SqlDependExample database and grant the necessary permissions to SqlDependExampleUser.

Now, open Visual Studio and create a new project (File -> New Project).  Select Class Library as the project type, enter SqlDependExample as the solution name, and enter SqlDependExampleCache as the project name.  Add a class to the project called CacheProvider and paste in the following code:
using System;
using System.Collections.Specialized;
using System.Data;
using System.Data.SqlClient;

namespace SqlDependExampleCache {

    public class CacheProvider {

        ///

        /// Reads the list of colors from the cache.
        ///
        ///
        public StringCollection GetColorList() {
            if (this.colors == null) {
                this.colors = this.ReadColorList();
            }
            return this.colors;
        }

        ///

        /// Reads the list of colors from the database.
        ///
        /// A that contains the list of colors.
        private StringCollection ReadColorList() {
            string connStr = "Database=SqlDependExample;Server=.;User ID=SqlDependExampleUser;Password=supersecret;";
            StringCollection colors = new StringCollection();

            using (SqlConnection conn = new SqlConnection(connStr)) {
                conn.Open();
                using (SqlCommand cmd = new SqlCommand("SELECT ColorID, ColorName from dbo.Color", conn)) {
                    SqlDependency.Start(connStr);
                    SqlDependency sqlDependency = new SqlDependency(cmd);
                    sqlDependency.OnChange += new OnChangeEventHandler(sqlDependency_OnChange);
                    using (SqlDataReader reader = cmd.ExecuteReader(CommandBehavior.CloseConnection)) {
                        while (reader.Read()) {
                            colors.Add(reader["ColorName"].ToString());
                        }
                    }
                }
            }

            return colors;
        }

        ///

        /// Raised when the underlying rowset changes. Invalidates the list of colors.
        ///
        void sqlDependency_OnChange(object sender, SqlNotificationEventArgs e) {
            this.colors = null;
        }

        ///

        /// Internal list of colors.
        ///
        private StringCollection colors;

    }
}

Now add a new web site to the solution named SqlDependExampleWeb. Add it as a file system web site underneath the solution directory we created earlier that was called SqlDependExample. Open the property pages for the web site, select the References tab and add the SqlDependExampleCache project as a reference to the project. Open the Default.aspx page and add the following bulleted list definition between the main div tags in the page:

Now open the code-behind file for the page and paste the following code as the contents of the class: using System;
using System.Collections.Specialized;
using System.Web;
using System.Web.UI.WebControls;

using SqlDependExampleCache;

public partial class _Default : System.Web.UI.Page {

    protected void Page_Load(object sender, EventArgs e) {

        CacheProvider cp = new CacheProvider();
        StringCollection colors = cp.GetColorList();

        foreach (string color in colors) {
            ddlColorList.Items.Add(new ListItem(color));
        }
    }

}

Add a breakpoint to line 50 of the CacheProvider class (this.colors = null), select the SqlDependExampleWeb as the StartUp project and hit F5 to begin debugging. Answer yes that a new web.config should be created to enable debugging when prompted and the web site should run, displaying the default.aspx page. The four colors Red, Blue, Green, and Yellow that we added to the Color table earlier should be displayed in a bulleted list. Now open the Color database table in SQL Management Studio and add a new row for the color Purple. You should notice that the breakpoint on line 50 of CacheProvider is active. This is because the SQL dependency on the Color table has fired and raised the OnChange callback event. Allow execution to continue and the full list of five colors will be displayed in the page.

This short demonstration should give a good idea of the mechanism that is being employed. The cache is vastly oversimplified and documentation is lacking simply for the brevity of the example. However, the principal holds for a larger application that a rowset can be monitored by SQL server and a callback method invoked when that rowset changes thus prompting some action.

Some important points to note here are that the query syntax for rowsets that can be monitored is quite restrictive. The query used in this example was:
SELECT ColorID, ColorName from dbo.Color

The table name is qualified both with the owner and table name. This is required for the SQL dependency to work correctly with the row set. A number of additional restrictions apply to the queries that can be used with SQL dependencies.

.NET CLR in SQL Server 2005

Stuart Thompson — Fri, 04 May 2007 17:52:20 GMT

Of recent interest to me has been the ability to deploy user-defined managed functions, stored procedures, and other objects to SQL from within Visual Studio 2005. With the .NET CLR in SQL Server 2005, these managed functions can be written and deployed from within Visual Studio, even as part of an automated build process. In wanting to investigate this further, I've decided to write a few articles on how this technique is used and on how it may be useful to you. When I first heard about the ability to (essentially) deploy .NET assemblies into SQL Server and then have SQL Server make managed calls to them for use in queries I was skeptical, thinking this was yet another lost solution embarking upon a life quest for a problem to solve. However, several examples have since been drawn to my attention regarding the usefulness of this solution in exposing features of the richer .NET languages for use in advanced queries.

The first of these areas is that of regular expressions. Earlier this year, I came across an article describing how calls to the .NET 2.0 RegEx classes could be wrapped in a managed function for use in the where clause of a SQL query. My ears (eyes) perked up at this, for the idea of using regular expressions to match data in queries could add a massive amount of power to my SQL toolbox. Even better, because the call would be to the .NET classes, we already know that I'd be able to pre-compile the expressions if needed as that is a feature of that implementation. The rest of this article walks through a simple example for creating a C# Database Project in Visual Studio 2005, deploying a managed function developed in that project, and then writing a query to use that function for RegEx matching within the WHERE clause. For those interested in more information, the original article can be found here: http://msdn.microsoft.com/msdnmag/issues/07/02/SQLRegex/default.aspx.

Creating a C# Database Project in Visual Studio

First, open Visual Studio, then select File->New->Project. In the New Project dialog, expand the Visual C# node and select Database, then choose SQL Server Project from the list of templates. After the project is created, a dialog named Add Database Reference will ask for the specification of a database connection to work with for this project. This connection is used to deploy the project to SQL Server. Select a connection from the dialog or create a new one by clicking Add New Reference. I'll assume for the brevity of this article that you either have a SQL database or can create a new one and that you can establish a database connection to it. Next, right-click the new SQL Server Project and select Add->New Item. Choose User-Defined Function from the Add New Item dialog and name it whatever you like; I used the class name RegexUtilities.cs in the test project I wrote for this article. This will add a snippet of code in the class that is a template for the function. Replace the entire contents of the code window with the following:

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.Text.RegularExpressions;
using Microsoft.SqlServer.Server;

public partial class UserDefinedFunctions {

[Microsoft.SqlServer.Server.SqlFunction]
public static bool Match(SqlChars searchString, SqlString regexPattern) {
Regex regex = new Regex(regexPattern.Value, RegexOptions.Compiled | RegexOptions.IgnorePatternWhitespace | RegexOptions.Singleline);
return regex.IsMatch(new string(searchString.Value));
}

}

The code here is a very simple wrapper around the Regex.IsMatch method and is adapted from the source code supplied by David Banister is his original article. I've cut a couple of things down for simplicity. Two parameters are passed to the method named Match: the string to search and the regex pattern to use in the match. The first line of the method creates a new compiled Regex object from the supplied pattern. The second line of codes returns a Boolean that indicates whether or not the supplied searchString was matched. First build the project with Ctrl+Shift+B and then right-click over the project in the Solution Explorer pane and select Deploy. This deploys the managed function to the SQL server you specified in the database connection for the project.

Now it's time to test the function out in a query. First, let's take the Social Security Number example and check that everything made it across OK. Open SQL Server Management Studio and connect to your database, then open a New Query window. Now type the following into the new query window:

select dbo.Match( N'123-45-6789', N'^\d{3}-\d{2}-\d{4}$' )

You should receive the following error:

Msg 6263, Level 16, State 1, Line 1

Execution of user code in the .NET Framework is disabled. Enable "clr enabled" configuration option.

This is because we have not yet enabled a state within SQL server that is disabled by default for security reasons. CLR code obviously wields great power and, as such, the ability to execute that code within the SQL process gives any would be malicious code a very nice process under which to run, and in an authenticated context. To run our example then, we must first instruct SQL server to allow CLR code to execute. This is achieved by running the following SQL within a query window:

sp_configure 'clr enabled', 1
go

reconfigure
go

A confirmation message should indicate that the 'clr enabled' option was changed from 0 to 1. It indicates that RECONFIGURE should be run to install the change, hence the second SQL statement in the snippet above. Now we are ready to execute our Social Security Number example again. This time you should receive a single result for (No column name) with a value of 1. This indicates a match and was reported as a literal with no column name because of the structure of the statement we ran. Change part of the statement's left parameter to be an invalid social security number and re-run the example to see that a 0 is received (indicating false) this time instead.

Now the real purpose of this article was the use of Regex in a WHERE clause and for this pick a table that contains a string column. Now construct a SQL statement similar to the following:

select dbo.Match(C.[First Name], N'^\w{8}$'), C.[First Name]
from C Customer

The statement above displays two columns of data, the first containing the result of the regular expression call, the second containing the customer first name. The regular expression in this example has been simplified to match words of eight characters in length (an oversimplification of the actual syntax, I know, but sufficient for the purposes of this article). Any customers with a first name of eight characters in length will be accompanied by a 1 in the left-hand column, the others with a 0.

That's it for this article. A quick demonstration of how managed functions in SQL Server 2005 can open the power of the .NET class library for use within SQL.