The partition offset issue is relevant to any disk partitions created prior to Windows Server 2008. What this means is that whilst your OS may be Windows Server 2008, if the disk you are using was migrated from an older OS the chances are that it may still be exposed to the problem.
Also, certain OEM built machines may be exposed to this problem even on Windows Server 2008 as they often include hidden partitions used for e.g. server recovery date.

Partition offset

Without going into too great a level of detail on how a disk is structured both physically and logically, the partition offset is basically where Windows will start writing its first bytes to disk once the partition concerned is first used.
When a disk partition is created in Windows Server 2003 and before, the first 63 sectors are reserved for the MBR as well as disk vendor proprietary information. That means Windows Server 2003 and before (hereon referred to as just Windows) will write its first bytes of data on the 64th sector.

What’s the big deal?

These 63 sectors are invariably 512 bytes (worth checking with your SAN team/disk vendor however as some SANs may use larger sector size but present the sectors as logical 512 bytes sectors to Windows). Therefore the 64th sector will begin at a 31.5KB sector boundary. What this means is that if this is a SQL Server disk, SQL Server will write its first data page on a 31.5K sector boundary, and every nth page will be written across these boundaries.
The big deal therefore is that each of these nth IO operations will result in an additional IO to read or write the data that was on the page that crossed that boundary (the actual value of n will depend on cluster (sector) size and stripe unit size).
What this means in real (performance) terms is that a properly aligned system will yield a 20-30% performance improvement in IO latency (and therefore query execution time). All this for free, without having to analyse SQL Server configuration, indexing strategy, query plans etc etc.

How do I check my offset?

Checking the offset can be done via a wmi console (wmic) command.

The above command will show output similar to the following (if the wmic command is being executed locally, there’s no need for the /node parameter):

There will be one line of output for each disk and each disk will be identified by its disk number.
Use the Disk Administrator to identify how these drive letters map to your drive letters.
Surprisingly enough, the StartingOffset column is the key column, and will show the starting offset in bytes.
Disks 1 and 2 have 64 sector offset which lines up with e.g. legacy controllers on local disks.
Drives on default values will show a value of 32256, such as disks 0, 3 and 4 in the output above. Divide that by the 512 byte sectors and you’ll get that dreaded value of 63.
Disks 5 and 6 have a 1MB offset which should line up with most modern disk, SAN cache and cluster size scenarios.
What the above output shows is that, depending on the age of the system, there can be a myriad of offsets used which were correct according to how the sytem was configured at the time. Disks 1 and 2 were altered from the default, so at some point in this system’s history someone looked into the issue of partition misalignment and tweaked it for this drive.
So why are disks 3 and 4 on misaligned defaults? Well, have you ever had disks recut to (usually) increase storage. That usually involves blowing away partitions and recreating them. If partition offsetting was not carried out when the partition was created for whatever reason, this is what you can end up with. The newest disks 5 and 6 were created with the latest information to hand, hence the higher offset, but as this slightly contrived example shows (it’s based on real world systems) it pays to be vigilant and review systems that were previously correctly set up.

How do I fix it?

On paper, the fix is very simple; adjust the partition offset. The diskpart command line utility is used to reset the offset.
Move the offset along from the default misaligned 31.5K offset to an offset that works with your cluster size and stripe unit size; 64K, or a multiple of 64K usually works on most systems, but where a SAN is involved, it gets more complicated as stripe unit sizes will vary from vendor to vendor and rig to rig, so you’ll have to get your SAN engineer/vendor involved.
Windows Server 2008 defaults to 1024K offset.

Ideally, the offset needs to start at a point where the IO requests that SQL Server issues (which will be a multiple of 8K, depending on the IO request type being fulfilled), the cluster size, and any caching involved at disk controller/SAN level all line up, thus avoiding additional IO requests.
The downside, of course, is the partition will have to be recreated, so downtime will be involved on standalone servers that have no HA solution implemented.

Realigning the partition offset involves blowing away the disk partition in question, so if you’re not familiar with diskpart, leave it to your relevant storage/Windows team to reset the partition. If you really want to do this yourself, you’ll find step-by-step instructions in the SQLCAT team’s Disk Partition Alignment Best Practices article linked to at the bottom of this post.

The resulting 20-30% improvement in latency times have been verified by the SQLCAT team who have published them in that paper on partition alignment mentioned above, and is highly recommended reading for anyone with an interest in this subject.

Links

If you only read one further article on partition offsets, make it this one!
Disk Partition Alignment Best Practices for SQL Server

SQL Server 2000 I/O Basics

Disk performance may be slower than expected when you use multiple disks in Windows Server 2003, in Windows XP, and in Windows 2000

It’s a broad topic and all aspects of it are pretty well covered in various blogs and technical articles, but there’s not much out there that gathers it all together. The intention of this series of posts is to try consolidate some of this information and focus in on some of the functions and procedures that can be used to monitor, troubleshoot and configure SQL Server disk configuration and most, importantly, disk performance. It won’t be exhaustive and cover every conceivable option, but will be based on experience gained whilst working at Sybase and Microsoft and sprinkling of ‘real world’ database administration experience.

I’ve split the topic up into smaller sub-topics in order to make it more manageable:

1. Partition offsets
2. Wait Stats
3. SQL Server IO statistics function (fn_virtualfilestats)
4. System Monitor (Perfmon)
5. Tempdb

Kinda ambitious, I know, but I don’t blog often and it does beat writing a ten line post on e.g. backup (yawn) compression!

There are plenty of excellent articles which cover IO in some depth, so I’m going to try my best to avoid regurgitation and try keep things DBA-centric.  (I highly recommend reading the links at the end of the relevant posts to anyone wanting a deeper understanding of IO issues, from configuration to monitoring and troubleshooting).

Until solid state disk prices come down and become ubiquitous, disk access is still a painfully slow (relatively speaking) mechanical process. An IO (read or write) request is issued by SQL Server which is passed onto the OS. The OS then passes this onto your disk controller or HBA and eventually this data request translates into a mechanical arm sitting above a spinning disk which will read or write the data that SQL Server requested.

What this means in performance terms is that disk access will be measured in milliseconds.

Access times to the other major components of a DBMS that directly affect performance such as CPU and RAM can be measured in microseconds or nanoseconds.

As you can see, it’s an order of magnitude difference.

This single factor alone highlights why reviewing a SQL Server database installation’s disk configuration can be such a valuable and rewarding exercise.  In fact, I begin many of my SQL Server troubleshooting exercises by eliminating IO performance first.

Prior to SQL Server 2000 SP4 it was difficult to identify IO performance issues without a lot of digging around. The biggest clue SQL Server would give would be the occasional mysterious 17883 (or even 17884) errors, and these were not always guaranteed to be IO related, although IO was usually heavily implicated. SP4 for SQL Server 2000 introduced those ’slow IO’ messages which most DBAs will be familiar with and was a step in the right direction.

SQL Server 2005 introduced various DMVs which would give much greater insight into IO performance, and SQL Server 2008 built on this with the Performance DataWarehouse. These posts are going to cover methods that can be applied from SQL Server 2000 onwards, so will not delve deeply into features and properties only available in later versions, but focus instead on functions and features available across the last three versions of SQL Server as I know there’s still a lot of SQL Server 2000 installations out there.

This blog post intends to show that, in the right hands, the default SQL Server trace in SQL Server 2005 and SQL Server 2008 can yield very useful information.

What is the default trace?

A default installation of SQL Server 2005/2008 results in a lightweight server-side trace running continuously in the background.
The trace records a handful of events to a trace file which can be loaded up and reviewed in SQL Profiler, just the same as any other trace file.
By default, five trace files are kept in the same folder as the SQL Server error log. Each file has a size limit of 20MB before it rolls over to the next file. After the fifth file is filled, the default trace rolls over to the first file, and so on.
To confirm if the trace is enabled, run the following query:

sp_configure 'default trace enabled'


A run_value of 1 indicates the default trace is enabled and should be active.

If the trace is enabled, the following query will list the details:


select * from sys.traces where is_default = 1


Amongst the output will be confirmation that the trace is running (is_shutdown will be set to 0 if it is running, otherwise it will be 1). The path column will show the full path and name of the current default trace output file.

Interrogating the default trace

Getting information out of the default trace file is much the same as getting information out of any other SQL Profiler trace file. There are multiple options, from simply loading the file in SQL Profiler, interrogating it directly via T-SQL statements, or by loading it into a table.

I’ve used all three methods in the past. The method you use depends on your own preferences and immediate requirements.

The examples in this blog will interrogate the trace files directly, as that’s the quickest way to get up and running and view the contents of the trace.
With more experience, the trace file can be regularly polled via e.g. a stored procedure that can be incorporated into standard health-checks to automatically retrieve events of any interest; you decide what those interesting events are. More on that later.

The queries I will demonstrate in this post reference a system table called sys.trace_events. This is used to get the graphical name of the events I’m interested in (otherwise we will only see the numeric event id, as that is what the default trace records).

This table, introduced in SQL Server 2005 holds details of the all the eventclass ids a SQL trace uses, so you can use this to identify which eventclass id corresponds to which event, which is how I established eventclass 20 is a login failed event (the actual event is Audit login failed).

First things first

The first thing to do is isolate the trace file in question.
Assuming the default trace is running and it has a traceid of 1 then the following query will return the name of the trace file. The property column value of 2 relates to the value which holds the name of the output trace file.


select value from ::fn_trace_getinfo(1) where property=2


This will tell you the name and location of the current trace file. If this does not overlap with the time of the error you are trying to track down, go to the location of that trace file and identify which of the preceding trace files is from the correct time period. As a precautionary next step, make a copy of the trace file, so you have a permanent record, and also to prevent the file being overwritten during a subsequent rollover.

Example 1: Finding out who filled up tempdb (or any other database for that matter…)

So, let’s get started. A common scenario for a dba is getting called out overnight because somebody, or some thing has maxed out all the space in e.g. tempdb.
Using the fn_trace_gettable function to interrogate the trace file directly we can get some useful information about the cause. Assume there was an 1105 error just before 21:00 last night (”Could not allocate space for object ‘‘ in database ‘tempdb’”).

To find this out it’s just a question of confirming when the exact time the database full error occurred and filtering the trace from that period for events in that database. Deceptively simple isn’t it? The query below does just that.


select te.name as [event],e.textdata, e.applicationname, convert (varchar(20), object_name(e.objectid)) as object, e.spid, e.duration/1000 as [duration (ms)], e.starttime, e.endtime, e.databasename, e.filename, e.loginname, e.hostname, e.clientprocessid
from fn_trace_gettable(’C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\log_326.trc’, default) e
inner join sys.trace_events te on e.eventclass=te.trace_event_id
where databasename = ‘tempdb’
and e.starttime between ‘2009-09-24 20:30′ and ‘2009-09-24 21:00′
order by e.starttime


Query 1: Who maxed out tempdb (or any other database)?

The query tracks all events in tempdb that occurred in tempdb for about half an hour before the tempdb full error.

Fig. 1: Events leading to tempdb full

Most of the events causing this type of issue will be object creation and autogrow as can be seen above. What we will also have is what was causing them; applicationname , loginname, spid, hostname and clientprocessid . These should all be enough to identify who or what was responsible for the tempdb issue (see fig. 1).

Sometimes, the objectname column will have the names of the tables being created, and the spid column will allow you to narrow down the trace even further in case tracing events half an hour before the database full problem occurred presents too much or too little data.

Needless to say, but I’ll say it anyway, change the databasename column to the name of the database you’re interested if you’re looking for events leading to the same issue in another database.
Also, the trace can be filtered on object creation (trace_event_id = 46) or data or log file autogrow (trace_event_id = 92 or 93) events

Without this information you have nothing to go on, unless you happened to be logged in at the server at the time the issue was occurring and realised that there was a long running transaction running against tempdb which was going to cause it to fill up.
You might also notice that for the autogrow events the Duration column records how long the autogrow took - very useful in identifying potential IO performance isues (or impractical file growth settings).
The object column returns the name of the object being referenced/created. If this query is not run against the trace file on the server that the error occurred on, the chances are this column will contain NULLs, so you can select just the raw objectid instead to view the values and then manually retrieve the object names from the affected server.
Incidentally, the fn_trace_gettable function is also the function used to load a trace file into a database table.

Example 2: Isolating login failures

OK, so the default trace has some use, but what else can it do?

Suppose you get the occasional login failure on a system, you’re not too worried as that’s inevitable sometimes, but from time to time the login failure is for sa and the ip address logged in the error log doesn’t correspond to any workstation belonging to a dba who should have access. You should be worried now.


select e.name as eventclass, t.textdata, t.hostname, t.ntusername, t.ntdomainname, t.clientprocessid, t.applicationname, t.loginname, t.spid, t.starttime, t.error from fn_trace_gettable('c:\program files\microsoft sql server\mssql.1\mssql\log\log_329.trc', default) t
inner join sys.trace_events e on t.eventclass = e.trace_event_id where eventclass=20


Query 2: Isolating where a login failure came from

Fig 2: Isolating login failures

The query references the trace file from the period of the login failure and retrieves the columns containing all the information that should be needed to pin down where any login failure recorded in that file came from. I’ve already covered how to use this information in another blog posting covering how to identify the source of login failures.

Example 3: Identifying performance issues

So we’ve seen some practical applications like pinning down login failures and database or log growth issues which makes the default trace pay for itself in some ways. But wait, there’s more.

The bane of most dbas lives’ are application teams who will describe a performance problem they have: great, you say, what’s the application and SQL Server database, I’ll log on and take a look? Oh, it happened last week, some time in the afternoon the application team will reply. That’s OK, you (want to) say, I’ll just step into my time machine and go back a few days, or get in touch with my sixth-sense and do the same and carry out some sort of mind-meld with SQL Server and try and find out why it was mis-behaving a few days ago. [Disclaimer] Before you go any further I would like to make clear that none of this will be possible via the default trace. It can do quite a few things, but time-travel and mind-melds are slightly out of scope.

The default trace does offer some hope here. Because by default 5 trace files are kept (including the current trace files) there is a chance that the older trace files may cover the time period in question.

The default trace will capture some common performance affecting events, namely:

• Missing Column Statistics
• Hash Warning
• Sort Warnings
• Missing Join Predicate

So, if you still have the trace file from the period in question, a query like the following will pick out all the those events:


select v.name as [Event Class], TextData, SPID, Duration, StartTime, EndTime, DatabaseName, ObjectName, ClientProcessID,
ApplicationName, LoginName, SessionLoginName, NTUserName
from fn_trace_gettable(’C:\log_180.trc’, DEFAULT) trc
inner join sys.trace_events v on
trc.EventClass = v.trace_event_id
where
v.name in (’Missing Column Statistics’,'Hash Warning’,'Sort Warnings’,'Missing Join Predicate’)
and StartTime between ‘2009-09-05 14:00′ and ‘2009-09-06 16:00′


Query 3: Isolating who made table DDL changes

Example 4: Finding performance problem events

Let’s start by clarifiying that this is not going to tell us what the query was that generated these warnings. At the end of the day, the default trace is designed to be a low-impact trace that will not cause any performance or load issues so what it gathers is therefore restricted.
What we do have, however are details of when all these errors occcured, the PID of the offending client process, together with the hostname, application name and login name.

Armed with that information, we can then go back into that trace and zoom in on a particular spid or application. By narrowing down the spid or application, but expanding the events we’re looking at we can find out a bit more about the problem. I use a combination of the time period, spid and clientprocessid to narrow down (see my January 2009 blog on login failures on how to use this handy column to track down exactly what application is connecting).


select v.name as [event class], textdata, spid, duration, starttime, endtime, databasename, objectname, clientprocessid,
hostname, applicationname, loginname, sessionloginname, ntusername
from fn_trace_gettable(’c:\log_180.trc’, default) trc
inner join sys.trace_events v on
trc.eventclass = v.trace_event_id
where
starttime between ‘2009-09-05 14:00′ and ‘2009-09-06 16:00′
and spid = 57 and clientprocessid = 2044


Query 4: Narrowing down performance problems

Fig 3: Performance problem events

I’ve replaced figure 3 with the output of an identical query (but using different time range parameters) trace from a server where the output allowed us to identify what query in a large batch job that was running overnight needed tuning. The output showed what time the query ran, the relevant details of the client application and the names of some of the tables/indexes that were affected .

Example 5: Finding out who made changes

Perhaps one of the most useful aspects of the default trace is the fact that it records changes made to objects and server configuration changes

Suppose someone added or removed a column from a table? The following query highlights who did it:


select e.name as eventclass, t.textdata, t.objectid, t.objectname, t.databasename, t.hostname, t.ntusername, t.ntdomainname, t.clientprocessid, t.applicationname, t.loginname, t.spid, t.starttime, t.error from fn_trace_gettable('C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\log_339.trc', default) t
inner join sys.trace_events e on t.eventclass = e.trace_event_id
where eventclass=164


Query 5: Isolating who made table DDL changes

Eventclass 164 is the Object:Altered event, and the above query should show what application and login made the changes.

If the change concerned was e.g. a server configuration change like e.g. the max server memory setting, the following code will isolate the culprit:


select e.name as eventclass, t.textdata, t.hostname, t.ntusername, t.ntdomainname, t.clientprocessid, t.applicationname, t.loginname, t.spid, t.starttime, t.error from fn_trace_gettable('C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\log_339.trc', default) t
inner join sys.trace_events e on t.eventclass = e.trace_event_id where eventclass=22


Query 5a: Isolating who made server configuration changes

Eventclass 22 is the ErrorLog event, so of course the changes can also be seen in the error log, but the error log won’t give you the details of who made the change.

Disabling the default trace

The chances are, most SQL Server 2005 (and beyond) systems are running this trace in the background without anyone even being aware of them, which in itself highlights what low impact these traces have. However, if CPU or disk resources are scarce the default trace can be disabled.
To disable the default trace, run the following commands:


sp_configure 'default trace', 0
go
reconfigure with override
go


And finally…

This post was not intended to show ever possible piece of information that can be extracted from the default trace, but merely to highlight some of the things that can be found without too much effort.
Have a look at your own trace files as everyone installation and every application running against SQL Server is different - it’s a great way of pro-actively finding potential issues.

So go and explore…

Upgrading SQL Server 2005

This is where it gets unnecessarily complicated if you’re upgrading a SQL Server 2005 installation.
Basically, you have to run the install via the command line and send the setup program the relevant parameters to upgrade the relevant SQL Server components you want to upgrade.
So, assuming you want to upgrade SQL Server 2005 Standard Edition to SQL Server 2005 Enterprise Edition, you’d open a DOS prompt in the same folder as the install media and run the following command:

start /wait setup.exe ADDLOCAL=SQL_Engine INSTANCENAME=Desktop\SQL2K5EXP UPGRADE=SQL_Engine SKUUPGRADE=1 /qb

The above command line upgrades the SQL Server instance named SQL2K5EXP on machine Desktop to whatever edition the install media contains.

The multitude of parameters here is what causes all the confusion and the problems when trying to carry out something that should really be fairly straightforward, so make sure your syntax matches what is above when you’re trying the upgrade.

A compromise method involves invoking the gui component of the installation by running just the following:

start /wait setup.exe SKUUPGRADE=1

Which will start the graphical setup program and step through the appropriate options to complete the edition upgrade.

Upgrading SQL Server 2008

Fortunately, Microsoft have realised the upgrade mechanism offered for SQL Server 2005 was lacking in some areas and have addressed the issue via the gui component of the setup program, so now the same setup program you use to install SQL Server 2008 can be used to upgrade SQL Server 2008. Crazy idea, I know.

Figure 1 below shows the setup GUI with the Installation Maintenance option selected:

Clicking on Edition Upgrade will result in a couple of Setup Support Rules checks running which will check e.g. whether the WMI service is running, the OS version and whether the user running the setup application has the relevant privileges to complete the install.

Assuming you get past these checks you’ll end up at Figure 2 which is the Product Key dialog where you can select either a free edition to ‘upgrade’ to or enter the Product Key that accompanies the edition of SQL Server that the setup program was launched from.

Figure 2 below shows the setup GUI with the Product Key option selected:

After selecting the relevant option the install is pretty much the same as a normal install; accept the licence agreement, choose the instance name to be upgraded and let the install run through and complete. Simple.

Validating the edition change

The following query will confirm the edition of SQL Server:

select serverproperty('Edition')

This will return the edition, and if the install went through correctly it will confirm that SQL Server is now running the expected edition.

Troubleshooting the installation

A run-through of troubleshooting SQL Server installs is slightly out of scope and there are plenty of KBs and webcasts on the Microsoft web site that will take you through that.
All I will say is look at the summary.txt file in the setup bootstrap directory (default location is C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log) to identify the initial failure. This should provide you with a summary of the problem and further details will be found in a subdirectory off the location of the summary.txt file which will be named after the date and time of the setup run.

Useful links

How to: Install SQL Server 2005 from the Command Prompt
How to: Upgrade to a Different Edition of SQL Server 2008 (Setup)

So, the relevant listener is in the list and, as far as SQL Server is concerned, is up and running.The next step is to check connectivity. The SQL Server clients are best for this, as this will take the problem client application out of the equation (presumably you’re reading this blog because of issues connecting to SQL Server via a specific listener).
Either SQL Server Management Studio (SSMS), or the lightweight console mode SQLCMD, which is my preferred tool will do here.
To test each listener you just need to prefix the listener name abbreviation before the server name. Using SQLCMD as an example and a local instance by the name of .\SQL2K5DEV:

In case you didn’t know the ‘.’ where the host name should be prior to the instance name is a shortcut for the local machine name.
If you prefer to use SQLCMD, just put the listener prefix in front of the server name as listed in the prefix column after the -S (server name) parameter.

To verify what protocol a connected client is using, use the following query from SQL Server 2005 onwards:

select net_transport, endpoint_id, connect_time,client_net_address from sys.dm_exec_connections where session_id = @@spid;Replace @@spid with the spid (or session_id) of the connection you’re interested in.

This will output the listener the relevant connection is using as well as information like the endpoint id (which can be checked against sys.endpoints), when the connection came in and its address, as shown below (I dispensed with the column to make the output easier to read):

If you can confirm the listener is up and you can connect to it (at least locally) you know it’s not a SQL Server configuration issue and is more likely to be a firewall/network problem, so make sure the relevant ports you’re listening on are open, e.g. named pipes required port 445 to be open.

Just to prove it, go to the client machine(s) concerned and verify if the Microsoft port querying tool can connect to that server. Fortunately, there is a graphical interface for this tool. This is a very useful tool for identifying, or at least narrowing down, connectivity problems.

Happy troubleshooting!

The technique is pretty much version independent, so is not specific to any particular version, but I will assume you know how to use SQL Server Profiler and capture a trace.

Login Error 18456

A login failure will throw an 18456 error and will be accompanied by the following entry in the SQL Server error log (SQL Server 2000 tends not to display the IP address):

2009-01-15 09:40:24.55 Logon Error: 18456, Severity: 14, State: 8.
2009-01-15 09:40:24.55 Logon Login failed for user 'Domain\User'. [CLIENT: xxx.xxx.xxx.xxx]


The severity of the error indicates the seriousness of the error. A severity level of 14 indicates an error in the range described as user correctable, which is understandable for login failures.

The next item of information the error provides is the state number. Most errors have a state number associated with them which provides further information which is usually unique to the error that has been thrown. For a login error, state: 8, shown in the above example, indicates an invalid password was used.

The state number therefore provides invaluable information about the reason for the login failure and can often be enough to identify the cause of an 18456 error.

The table below illustrates what some of these state values mean:

The next item of information is the login (SQL Server or Windows) generating the failure, followed by the IP address of the host from which the login was attempted, which provides a useful cross-reference to confirm we’re looking at the right host when we’re trying to isolate the login failure.

Isolating the login failure

If the information provided in the error log for the login failure is not enough to isolate the source of the errors, the next step is to run a quick trace against SQL Server to get more information.
The easiest way to identify the process generating the login failures is via a SQL Server Profiler (SSP) trace.

If you’re running SQL Server 2005 or above and you still have the default trace enabled (which is on by default in an out-of-the-box installation) then you don’t need to start a new trace; check out my SQL Server default trace post instead.

If the default trace is disabled, or you have an earlier version of SQL Server, then read on.

Start SSP, and, using either your favourite trace template, or via a new trace template (File > Templates > New Templates…) make sure the following columns are selected:

• ClientProcessID
• Hostname
• LoginName
• NTUserName
• NTDomainName
• ApplicationName

These columns can be found in the Trace Properties dialog on the Events Selection tab. If they are not visible, tick the Show all columns checkbox. Note that SPID is always a selected column by default, and cannot be de-selected.

Under Events select the Audit Login Failed event under Security Audit. As we’re only interested in login failures this is the only event we need, and will help ensure that any performance impact from running the trace can be kept to a minimum. On a production system it’s never really advisable (imho) to run a graphical SSP trace on the server; always use a server side trace.

Figure 1 below shows a completed trace template:

It might look a bit sparse, but we are only interested in a specific error.

Step 1
Save the modified trace template and launch a new trace, specifying the saved template as the template for the new trace and wait for the login failures. Stop the trace after a login failure has been generated.

Step 2
The Hostname column should have recorded the name of the server that the invalid login emanated from and the ClientProcessID should have captured the Process ID, or PID of the offending process (or processes if there are multiple processes involved).

Step 3
Log on to the server triggering the errors, and list the PIDs of the relevant processes. This can be done using Tasklist or Task Manager.

To view the PIDs via Task Manager, start Task Manager (Shift+Ctrl+Esc), go to View > Select Columns… and tick the checkbox labelled PID (Process Identifier) and click OK.

Click on the Processes tab to bring all the processes running on that server into view (make sure Show all processes from all users is ticked) and click on the PID column heading to sort the PIDs in descending or ascending order.

Step 4
Once you’ve isolated the process responsible via the PID it should just be a matter of identifying where that process stores the credentials it uses for logging into SQL Server and verifying them.
Usually, the process will be a service, so it’s just a question of bringing up the Services plugin via Control Panel, or Start > Run > services.msc should also do the trick.

That’s it, so happy hunting!

Useful links
Troubleshooting: Login failed for user ‘x’
Understanding “login failed” (Error 18456) error messages in SQL Server 2005

Often, especially in older (pre version 7) versions of SQL Server the only option was to restart SQL Server to clear those locks.

Spids with a value of -2 are orphaned distributed transactions.

What’s an orphaned distributed transaction?

In a nutshell, it’s a distributed transaction for which the transactional state is unknown. A distributed transaction is a database transaction that involves more than one database system (usually) located on different servers.

In order to maintain transactional consistency these transactions need to be coordinated, as we can’t have one database involved in the transaction committing the data it has been sent, whilst another database fails to commit the data it has been sent; either all the data is successfully committed by all the databases involved in the transaction, or the entire transaction is rolled back and no data gets committed by any database.

The process responsible for coordinating these transactions is the Microsoft Distributed Transaction Coordinator, or MSDTC.

You can get to it via Start > Run and typing dcomcnfg.

This brings up the Component Services plugin. Click on Component Services > Computers > My Computer.

Under My Computer, click on the Distributed Transaction Coordinator folder. Under Local DTC you will find the following information about MSDTC transactions:

• Transaction List lists active distributed transactions
• Transaction Statistics reports summary statistics for previous MDTC transactions

Both items provide useful information, but in the context of this blog, it’s the Transaction List that we’re interested in. This lists all the currently active transactions that have not been successfully completed, or been successfully rolled back. The status column lists what MSDTC regards as the state of the transaction, and the Unit of Work ID column with those strange GUID type values displays a unique identifier that is assigned to that transaction; this is a very useful value to have as it will allow any MSDTC transactions executing within SQL Server, including transactions that MSDTC has lost control of and thus have a spid value of -2 to be killed by specifying the Unit of Work (UoW) ID in place of the spid number.

If the status of the transaction is ‘Active’ then the transaction is currently executing. Figure 1 below, shows an active distributed transaction when viewed in the MSDTC console (the screenshot was taken from a Windows Vista host):

Figure 1: The MSDTC Console

The problems arises when, for whatever reason, MSDTC loses track of one of these transactions. The transaction will then be marked as in-doubt. If this happens, SQL Server will honour the locks the spid bound to that transaction has left, but the spid itself will be assigned a negative value of -2.
This means it will no longer appear as an internal process, but will still cause blocking if the original transaction was accessing tables at the time it went awol, as the locks will have been retained. This is because SQL Server does not know whether it can commit or roll back the transaction as MSDTC has control of distributed transactions.

Sometimes this can happen because the MSDTC service (Distributed Transaction Coordinator) has been terminated, or is in a hung state. When MSDTC is restarted it will go through its own internal transaction log and try and regain control of transactions that were left open in order to commit them or roll them back. Usually, however, it’s because one or more of the applications involved in the distributed transaction has mis-behaved in some way and has hung or crashed, and the only option there is to try and restart the application(s) concerned and hope that it has a recovery mechanism to deal with any MSDTC transactions that were left unresolved.

From time to time, therefore, MSDTC will not be able to regain control of a transaction and it’s status will be marked as ‘in-doubt’ in MSDTC coordinator.

Once upon a time (well, before SQL Server 6.5 SP5 at any rate) the only way to resolve this was to restart SQL Server, and because this was once the only way to deal with orphaned spids, it’s become something of a myth that it remains the only way to deal with the issue.

However, since Service Pack 5 of SQL Server 6.5 (http://support.microsoft.com/default.aspx/kb/195542) it has been possible to kill an MSDTC transaction by specifying something called the UoW ID, as opposed to just the spid number.

The Unit of Work (UoW) ID

The UoW ID is a 24 character GUID assigned to each transaction issued by MSDTC, and it is the UoW ID that is used to identify and kill orphaned MSDTC transactions in SQL Server.

Killing an MSDTC transaction

So, you’ve come up against an MSDTC transaction that the MSDTC service has been unable to successfully commit or rollback.

This transaction has now left locks on your data that are associated with an orphaned spid and you’ve got no option but to kill that MSDTC transaction.

You first need to retrieve the UoW ID. This should be displayed in the MSDTC console, but to verify it is the correct ID, you can interrogate the syslockinfo system view. Specifically the req_transactionUoW column. The req_spid column in syslockinfo hosts the spid number.
So, to pull out the UoW ID for an orphaned MSDTC transaction you just need to run the following query:


select req_transactionUoW as [UoW ID] from syslockinfo where req_spid = -2


The above will work in SQL Server 2000 onward, but from SQL Server 2005, it’s more politically correct to use the DMVs, so the table and column names have changed slightly:


select request_owner_guid as [UoW ID] from sys.dm_tran_locks where request_session_id = -2

Then to kill it:

kill {'UoW ID'}

where {UoW ID} is the result of the above select statement.

However, this must only be done as an absolute last resort when other methods, such as e.g. restarting the applications/processes involved in that distributed transaction do not resolve the problem. At the end of the day, those locks will have been placed for a reason, so you don’t know if the consistency of the data is now compromised especially if syslockinfo/sys.dm_tran_locks shows the MSDTC transaction(s) concerned have placed exclusive locks on objects.

Useful MSDTC links

Troubleshooting Problems with MSDTC
New DTC Functionality in Windows Server 2003 Service Pack 1
Enable Firewall Exceptions for MS DTC

The problem is due mainly to incorrect assumptions about Microsoft’s licencing policy, combined with the prevalence of multi-core CPUs. This has led to some incorrect responses to postings about CPU licencing on the SQL Server forums, so I’ve decided it’s worth blogging about to try and clarify the matter.

This blog covers SQL Server 2005 and SQL Server 2008 editions.

Microsoft’s multicore licencing policy is based on the number of sockets on the motherboard, not the number of cores. Therefore, for licencing purposes, a dual-core CPU equates to 1 CPU, not 2. A quad-core CPU equates to 1 CPU, not 4 etc. etc.

What this means is that your free SQL Express edition can use all 4 cores in your quad core CPU and not just one core. You can verify this by interrogating the sys.dm_os_sys_info SQL Server dynamic management view (DMV), particularly. The following query lists how many CPUs a particular instance of SQL Server can see:

select cpu_count from sys.dm_os_sys_info

The sys.dm_os_sys_info DMV returns a whole host of other useful information that used to be very difficult to obtain without using xp_cmdshell or WMI, so it’s well worth exploring further. In fact, most of the DMVs have invaluable information if you know what you’re looking for, but that’s a subject area far too large for a blog; just peruse Books Online as that gives a complete listing and a good rundown of what each DMV offers.
Just remember that, although SQL Server Express will be able to see multiple CPUs and use them to improve concurrency, SQL Server Express edition is prevented from taking advantage of this to parallelise individual queries, which basically means that an individual query will not be executed across multiple CPUs.

Back to counting CPUs. To find out how many CPUs a particular instance is actually using, run the following query which is based on the sys.dm_os_schedulers DMV:

select scheduler_id,cpu_id, status, is_online from sys.dm_os_schedulers where status='VISIBLE ONLINE'

This will return how many CPUs SQL Server is using. Ordinarily, this will always equate to the number of CPUs on the system. However, if CPU affinity is being used to assign specific CPUs to SQL Server, this query will show how many CPUs SQL Server is actively using.

Run the query without the where clause to see how many CPUs are offline and not being used (their status will be VISIBLE OFFLINE). The additional rows returned will show up internally used schedulers such as the scheduler retained for the Dedicated Admin Connection, or DAC.

If this query does not list the number of CPUs you are expecting, and you’re definitely not using CPU affinity, check the edition of SQL Server that you are using against the table below.

Note: To confirm that CPU affinity is indeed being used, the cpu_id column will always be less than 255.

A summary of how many CPUs the different editions of SQL Server support is summarised in the table below:

The problem is due in part to the fact that on 32-bit systems (which are still the most prevalent) configuration changes usually have to be made both in SQL Server and at the OS level, and in part to the sprawl of documentation available on configuring SQL Server’s memory settings, as opposed to a single jumping off point which runs through all the settings and considerations that need to be made.
Add to that the black art of establishing exactly how much memory SQL Server is using (most of the obvious options will only show how much memory the buffer cache is using) and it’s easy to see why it’s such a problem area.

In this post I’ll attempt to clear some of the smog and provide what I hope will be one document which answers most of the questions that arise about configuring SQL Server’s memory usage.
This discussion will cover configuring memory for SQL Server 2000, SQL Server 2005 and SQL Server 2008 (with the exception of the Resource Governor). This blog assumes an edition of SQL Server that is not internally limited in its memory usage.

32-bit or 64-bit?

There’s a big difference between the memory configuration settings between 64-bit SQL Server and 32-bit SQL Server, so it’s not possible to start a discussion about SQL Server’s memory management without clarifying whether we are dealing with 32-bit versions or 64-bit versions of the product, as this is key to how much memory SQL Server can address, and (almost as importantly) how it addresses that memory.

Configuring 32-bit SQL Server

Until fairly recently 32-bit software was ubiquitous. The server Windows operating systems were 32-bit, your desktop, usually Windows XP was 32-bit. Therefore, I’ll be focusing a fair bit on 32-bit SQL Server as this is what requires the most configuration, and also where most of the confusion lies.

So, here goes.

The amount of memory a 32-bit process can access is 2^32 or 4294967296 bytes, or 4 GB.
On a 32-bit Windows OS this 4 GB of memory is not all addressable by a single process. Instead, it’s partitioned by the OS into two address spaces. 2 GB is kept by the OS (commonly referred to as the kernel) and the remaining 2 GB is the user mode address space, or the area each application (process) will have access to. Whilst each user mode process gets 2 GB as its addressable memory range, the kernel mode area is shared between all processes. SQL Server runs in the user mode address space and is bound by default to the 2 GB of memory limit on a 32-bit OS.
This directly addressable memory will hereon be referred to by what it is more commonly known as, the virtual address space or VAS.

SQL Server’s default out-of-the-box memory limit is deliberately set to a very high value of 2147483647 , which basically means all available memory, but as you should now know, there’s no way it can actually use anywhere near that much memory, particularly on a 32-bit platform.

64-bit operating systems have a far far bigger address space open to them; 8 TB to be exact. Before you run off to your calculator to evaluate 2^64, the answer won’t be 8TB, but 8TB is what each user mode application gets due to current hardware and OS limitations. The kernel also gets 8TB and this kernel address space is shared by all processes, just as in 32-bit Windows.

Having said all that, I should point out that no current MS Windows OS can address more than 2 TB.

What this means for 64-bit SQL Server is that out of the box, it can address all the memory on a server without any special configuration changes either within SQL Server or at the OS level. The only thing you need to look at is a cap on its memory usage; capping memory usage is covered in the ‘max server memory’ and ‘min server memory’ section which is further down.

To /3GB or not to /3GB

So, as 32-bit applications are natively restricted to a 2 GB VAS, OS configuration tweaks are required to allow access to more than 2 GB, and these are covered next.

The first modification is one that, rather ironically, should be used as a last resort. Ideally, it should be used on the advice of Microsoft Support (PSS).
I’m choosing to get it out of the way now because the /3GB setting is probably the most well known and most misused.
/3GB allows a 32-bit process to increase its VAS to 3 GB by taking away 1 GB of address space from the kernel, and this is why it’s a last resort; there’s no such thing as a free lunch as the removal of 1 GB of addressable memory from the OS can introduce instability. More on that shortly.

To allow a 32-bit process to gain a 3 GB VAS you have to add the /3GB switch to the Windows boot.ini file.

As I stated, this can introduce system instability, particularly on Windows 2000 by starving the OS of System Page Table Entries (PTEs). A discussion about PTEs is out of the scope of this blog, but its effects can be dramatic and cause blue-screens. There is scope for manoeuvre here, by adding the /USERVA switch to the boot.ini it is possible to reduce the VAS increase from a straight 3 GB to a lower user-defined amount which will give the OS room to breathe, and thus resolve any instability issues.

The only reason you will be advised by PSS to use /3GB is if you are suffering VAS starvation issues, such as a bloated procedure cache as it can only reside in VAS memory (also see the next section on MemToLeave). Only database pages can reside in the part of the SQL Server memory cache (called the buffer cache) that is utilising awe enabled memory.

MemToLeave

Because of the inherent address space limitations of a 32-bit process, a certain amount of memory has to be set aside by SQL Server on startup that SQL Server uses for overheads. This memory is set aside in case it all gets used by the buffer cache.
COM objects, extended stored procs, memory allocations exceeding 8K and the memory allocated to the threads SQL Server creates to service e.g. user connections come from a section of memory within the VAS but outside the buffer cache which is typically referred to as the MemToLeave area. This is 384 MB by default on 32-bit SQL and is derived as follows:

256 MB + (0.5 MB x max worker threads [default: 255]) = 384.

0.5 MB is the default thread stack size on 32-bit Windows. 64-bit Windows has a default stack size of 2 MB or 4 MB depending on which 64-bit flavour of Windows you are running (AMD x64 or IA64).
This is why a default 32-bit SQL Server configuration appears to be ‘only’ using 1.6 GB out of the 2 GB it is entitled to; 1.6 GB is the buffer cache, and the remainder is MemToLeave.
SQL Server 2005 and beyond uses a formula to calculate the max worker threads setting which affects the size of the MemToLeave area.
There is a SQL Server startup parameter (-g) which can be used to increase the MemToLeave area, but again, only do this if advised by PSS.

4 GB of RAM and beyond

So, we know 32-bit SQL Server can use 2 GB out of the box, and up to 3 GB (with an OS tweak that is best avoided, if at all possible).
However, 32-bit SQL Server can benefit from much more memory than 3 GB with the help of OS and SQL Server configuration modifications which will be covered next.

/PAE

To address more than 4 GB of RAM on 32-bit Windows, the OS needs to have the /PAE switch added to the boot.ini file, although if your system supports hot-swappable memory you won’t need to add this as Windows should automatically be able to see the additional memory. If you’re not sure, take a look at how much memory the OS can see via System properties; if you have more than 4 GB installed and the OS is only showing 4 GB, review your boot.ini settings. I’m not mentioning specific Windows versions because the /PAE switch applies to all current 32-bit versions of Windows.

Both /PAE and /3GB

Some systems have both /3GB and /PAE enabled.
This is fine as long as the system does not have more than 16 GB of RAM. Add any more memory and Windows will not recognise it because of the overhead required to manage the additional memory.

Clusters

No special configuration settings regarding memory settings are required for a cluster, but I thought I better mention clusters specifically because you won’t believe how many installations there are out there where there are different settings on different nodes within the same cluster.
So, make sure any OS setting changes like /3GB or /PAE are consistently applied across all nodes.

Enable AWE

After configuring the OS, you’ll need to configure SQL Server by enabling AWE (Address Windowing Extensions). AWE is in essence a technique for ‘paging’ in sections of memory beyond the default addressable range.
AWE can be enabled in SQL Server using Query Analyzer/SQL Server Management Studio (SSMS) via the following statements:

sp_configure 'show advanced options', 1
RECONFIGURE
GO
sp_configure 'awe enabled', 1
RECONFIGURE
GO

AWE enablement is not a dynamic option and will require a SQL Server restart.

‘max server memory’ and ‘min server memory’

The more memory you give SQL Server, the greater the need to set an upper limit on how much it uses. When you start SQL Server it’ll ramp up its memory usage until it has used up all the memory it can access, which will either be an internal OS limit or a SQL Server configured limit.
A 32-bit SQL Server instance will therefore grab up to 2 GB if the workload demands it and it is on default settings.
An awe enabled SQL Server instance will go on using up all the memory on the system if the workload is there and an upper limit on its memory usage is not set.

Setting a limit has the double-benefit of not starving the OS of resources and avoiding ‘Out of memory’ errors which can occur on SQL Server systems that may have a lot of memory. The latter (rather contradictory) situation can arise because SQL Server will try and allocate more memory when it is already at the system limit (if no upper limit has been set via the ‘max server memory’ setting) instead of freeing up memory it is already using.

Multiple instances

A third reason to set an upper limit is if you have more than one SQL Server instance installed on a single host, as this will stop the instances competing for memory.
Allocate a high enough ‘max server memory’ limit to each instance to allow it to do its job without running into memory starvation issues, whilst reserving the bulk of the memory for higher priority instances (if any) and the OS.
This is where benchmarking comes in handy.

To set a max server memory limit of 12 GB via Query Analyzer/SSMS:

sp_configure 'max server memory', 12288
RECONFIGURE
GO


SQL Server ramps up its memory usage because by default it is set to use no memory on startup. This is controlled by the ‘min server memory’ setting. Specifying this to a higher value has the benefit of reserving a set amount of memory for SQL Server from the off which can provide a slight performance benefit, especially on busy systems. It’s actually not uncommon to see ‘min server memory’ and ‘max server memory’ set to the same value, to reserve all of SQL Server’s memory straight away. The downside is SQL Server will take slightly longer to start up than if ‘min server memory’ was set to a low value.

SQL doesn’t really release memory

This will probably get me into a bit of trouble, as there are KBs that clearly state that SQL Server releases memory when the OS is under pressure.
True, but only when it’s under a lot of pressure (although this is getting better with each version of SQL Server), and by then it’s often too late as the system is usually in such a degraded state by that stage that a restart is necessary.
That’s why it’s vital to set an upper limit on its memory usage via the ‘max server memory’ setting.

Memory corruption

Slightly off-topic, but this is an appropriate place to bring this up.
Certain builds of Windows 2000 and Windows Server 2003 contained a potentially serious memory corruption problem which affected SQL Server more than other applications, mainly because there are few applications that run on Windows that can utilise the amount of memory SQL Server does.
It’s difficult to overstate the problems this can cause, so make sure you’re on the appropriate Windows Service Packs if you’re running SQL Server on a PAE enabled system.
Another issue that arose in SQL Server 2000 SP4 was a bug that meant SQL Server only saw half the memory on awe enabled systems, although it was identified quickly and the hotfix for this was placed alongside the SP4 download.

32-bit SQL Server on 64-bit Windows

If you 32-bit SQL Server on 64-bit Windows the SQL Server process can access the entire 4 GB VAS.

Checking SQL Server’s memory usage

This is another area where there is lot of confusion, so below is a run-through of the most common methods for confirming SQL Server’s memory usage.

Ignore Task Manager

If you have an awe enabled SQL Server instance, do not rely on Task Manager to display memory usage as it does not show the AWE memory a process is using, so the memory usage figure it presents for the SQL Server process (sqlservr.exe) will be incorrect.

DBCC MEMORYSTATUS

Running the above command outputs the memory usage of SQL Server including how that memory is allocated, so unless you need to know how and where that memory is being used, the output it generates can be a bit bewildering. The important bits of this output pertaining to SQL Server’s total memory usage are as follows:

Buffer Counts                  Buffers
------------------------------ --------------------
Committed                      3872
Target                         65536
Hashed                         2485
Stolen Potential               60972
External Reservation           0
Min Free                       64
Visible                        65536
Available Paging File          702099

The key figures in the above output are committed, target and hashed.
Committed is the amount of memory in use by the buffer cache and includes AWE pages.
Target is how big SQL Server wants the buffer to grow, so you can infer from this whether SQL Server wants more memory or is releasing memory.
There’s an excellent KB on interpreting all the output INF: Using DBCC MEMORYSTATUS to Monitor SQL Server Memory Usage for SQL Server 2000 and How to use the DBCC MEMORYSTATUS command to monitor memory usage on SQL Server 2005.
Edit (05/02/09): Remember the buffer count numbers refer to pages of memory which are 8K in SQL Server

System Monitor (perfmon)

Perfect way to get a quick reference on exactly how much memory SQL Server is using at that moment. Start System Monitor and add the SQL Server: Memory Manager: Total Server Memory (KB) counter.

Replace “SQL Server” with MSSQL$ and the name of the named instance if it’s not a default instance, e.g. MSSQL$INSTANCE1.

‘Total’ memory usage

When trying to establish exactly how much memory SQL Server is using it’s not just the buffer pool memory you have look at, but the MemToLeave area as well. The key point to bear in mind here is that it’s not only SQL Server that can make allocations from this latter area of memory but third party processes as well, which can make it impossible to precisely account for SQL Server’s absolute memory usage, contrary to some myths out there about calculating SQL Server’s memory usage via e.g. DBCC MEMORYSTATUS, as such methods can only account for SQL Server’s own memory allocations and not allocations by foreign processes.

NUMA

I couldn’t really finish this blog without at least a passing reference to NUMA, or Non-Uniform Memory Architecture. NUMA enabled systems allow memory to be partitioned between CPUs on multi-CPU systems. SQL Server can also affinitise NUMA nodes to tcp/ip ports. If you have high spec non NUMA capable hardware SQL Server can take advantage of the scalability enhancements NUMA provides by simulating NUMA-like behaviour using a technique referred to as soft-NUMA.

64-bit

I mentioned at the start of this post that all you have to worry about for 64-bit SQL Server is setting a max memory limit as SQL Server can access all the memory current Windows operating systems can support, and 8 TB in total. That’s mostly true, with the exception of a certain privilege that the SQL Server service account needs, and that’s the ‘Lock Pages in Memory’ privilege.
This privilege is vital as it prevents the OS from paging out SQL Server memory to the swap file.
With the introduction of SQL Server 2005, this right was restricted on 64-bit Windows to only take effect on Enterprise Editions of SQL Server, so if you’re wondering why your huge new multi-gigabyte multi-core 64-bit system is paging like crazy, this might be why. [Edit: This has finally been reversed for both SQL Server 2005 and SQL Server 2008 Standard Editions: http://blogs.msdn.com/psssql/archive/2009/04/24/sql-server-locked-pages-and-standard-sku.aspx
Whilst we’re on the subject of paging on 64-bit SQL Server systems, take a look at the following KB:
How to reduce paging of buffer pool memory in the 64-bit version of SQL Server 2005 which covers issues a number of issues that cause SQL Server’s (Standard or Enterprise editions) memory to be paged out.

In summary…

The table below describes how much memory SQL Server can use, and assumes an edition of SQL Server that has no internal limitations as to how much memory it can use, e.g. Express and Workgroup editions are limited to 1 GB and 3GB respectively.

When I started this blog I wanted to keep it as short and succinct as possible, but there’s a lot more to configuring SQL Server’s memory usage than simply setting a ‘max server memory’ limit. Configuring SQL Server’s memory settings can be quite a complex undertaking, especially in a 32-bit environment. It’s not easy to cover all the pertinent points without branching off and describing the different areas of its memory architecture, although I’ve tried to provide the relevant information without going into too much detail.
Hopefully, this blog has clarified how to configure SQL Server’s memory usage and provided enough information to answer most memory configuration related questions.

Useful links

How It Works: DBCC MemoryStatus Locked Pages Allocated and SinglePageAllocator Values

INF: Using DBCC MEMORYSTATUS to Monitor SQL Server Memory Usage

MemToLeave and 64-bit SQL Server