The sys.dm_io_virtual_file_stats DMV and fn_virtualfilestats function are great tools for extracting granular information about the IO performance of your database(s), right down to the individual file in each database.

It gives details of the number of physical reads/writes, amount of data read/written and how much was spent reading/writing that data. Traditionally, this has been important because IO access is the slowest part of fulfilling any database queries as RAM and CPU access takes nanoseconds or microseconds, whereas typically access to a spinning disk takes several milliseconds. In other words, disk access is thousands of times slower than CPU or RAM access, so improving disk access performance often brings about the biggest bang for the buck, in a manner of speaking.

Lately, with the uptake of SSDs increasing this is starting to become less of an issue, but if like me, some of the systems you administer are still using the ancient technology of spinning disks then getting familiar with this view will help you confirm any IO issues. Incidentally, the forerunner of this DMV was fn_virtualfilestats and goes back to (at least) SQL Server 2000 and still works in the latest SQL Server version (SQL Server 2012, as of writing this post) so this post concentrates on the fn_virtualfilestats function.

I won’t bore you with details of how to query the function as it is already well documented, and my own query which utilises it is below:

;WITH sum_table
AS
(
SELECT [dbid]
, [fileid]
, [numberreads]
, [iostallreadms]
, [numberwrites]
, [iostallwritems]
, [bytesread]
, [byteswritten]
, [iostallms]
, [numberreads] + [numberwrites] AS [total_io] FROM :: fn_virtualfilestats(db_id(), null) vf
inner join sys.databASe_files df
on vf.[fileid] = df.[file_id]
where df.[type] <> 1
)
SELECT db_name([dbid]) AS db
, file_name([fileid]) AS [file]
, CASE [numberreads]
WHEN 0 THEN 0
ELSE CAST(ROUND([iostallreadms]/([numberreads] * 1.0), 2) AS NUMERIC(5,2)) END AS [read_latency_ms]
, CASE [numberwrites]
WHEN 0 THEN 0
ELSE CAST (ROUND([iostallwritems]/([numberwrites] * 1.0), 2) AS NUMERIC(5,2)) END AS [write_latency_ms]
, [numberreads] AS [#reads]
, [numberwrites] AS [#writes]
, [total_io]
, CAST (ROUND(([bytesread]/1073741824.0),2) AS NUMERIC (12,2)) AS [gb read]
, CAST (ROUND(([byteswritten]/1073741824.0),2) AS NUMERIC (12,2)) AS [gb written]
, [iostallms] AS [#stalled io]
, CAST (ROUND((([numberreads] * 1.0 ) /[total_io]) * 100,3) AS NUMERIC (6,3)) AS [file_read_pct]
, CAST (ROUND((([numberwrites] * 1.0 ) /[total_io]) * 100,3) AS NUMERIC (6,3)) AS [file_write_pct]
, CAST (ROUND((([numberreads] * 1.0) /(SELECT sum(total_io) FROM sum_table)) * 100,3) AS NUMERIC (6,3)) AS [tot_read_pct_ratio]
, CAST (ROUND(((numberwrites * 1.0) /(SELECT sum(total_io) FROM sum_table)) * 100,3) AS NUMERIC (6,3)) AS [tot_write_pct_ratio]
, CAST (ROUND((([total_io] * 1.0) /(SELECT sum(total_io) FROM sum_table)) * 100,3) AS NUMERIC (6,3)) AS [tot_io_pct_ratio]
, (SELECT CONVERT(NUMERIC(10,2),DATEDIFF(MI, sqlserver_start_time, GETUTCDATE()) / 1440.0) from sys.dm_os_sys_info) as [uptime_days]
FROM sum_table
GROUP BY [dbid], [fileid], [total_io], [numberreads], [iostallreadms], [numberwrites], [iostallwritems], [bytesread], [byteswritten], [iostallms]
ORDER BY [total_io] desc
GO

I’ve chosen to use the fn_virtualfilestats function instead of the sys.dm_io_virtual_file_stats DMV so it should still work right back to SQL Server 2000. The main difference between this query and others I’ve found on the blogosphere is that it excludes the log file (because on an OLTP system it will get hammered and generate a lot of IO anyway), and once I’ve excluded the log file being an issue (usually from wait stats and System Monitor (perfmon) data) I’m usually more interested in how busy my data files are, and if there are any hotspots and it does this by aggregating the IO data and returning the percentage of IO that not only each file does, but the actual ratio of IO that each file does relative to the total IO against all data files within that database. Mike Lewis has tweaked the query so it scans all databases (see the first comment on this post below).

Automating the collection of this data provides a great way of baselining performance and spotting trends (hint hint). The uptime_days column shows how many days the server has been running so you can work out the average daily read/write IO.

If you want information regarding log file performance, simply comment out the
where df.[type] <> 1 line.

It’s been tested in battle and I’ve found it invaluable for highlighting high contention filegroups and IO subsystem issues. A word of caution, before looking at the output and pointing the finger at the disks/SAN it’s always worth digging behind the stats. If high latencies/reads are found, the issue can boil down to database design which can be uncovered by investigating the reason for the high number of reads/writes on the relevant filegroup(s), e.g. a lack of indexing on a highly read table will result in a lot of table scans which can account for a substantial amount of IO on the file/filegroup holding that table and adding a well chosen index or two can eliminate the scans, reducing the IO and (hopefully) the latency. If the IO goes down, but the latency does not, you have more ammunition for looking more closely at the IO subsystem provided that factors like, for example, anti-virus filter drivers etc have been investigated. Recently, the output highlighted the slow performance of one particular file in an 18 file database which showed latencies of 89ms. After some digging around I found that this particular file was the primary file and (taking an educated guess) probably had default autogrowth settings when it was originally created, and has probably gone through some shrinkfile operations in a previous life, because this one 120GB file was spread over 4,818 physical fragments on the SAN. Once this was addressed the latency went down to 25ms. Whilst 25ms is not brilliant in itself it’s still a 70% improvement on its previous performance (and I know there’s room for further improvement as the partition it’s on has an sub-optimal allocation unit size, for some strange reason). IMHO physical file fragmentation, even on SANs is a big issue and I’ll be blogging about that particular issue in an upcoming post.

As I was saying, it pays to do a little digging around behind the numbers before jumping to the most obvious conclusion.

I know there are dozens of variations of queries out there that interrogate fn_virtualfilestats or sys.dm_io_virtual_file_stats but the reason I came up with yet another variation was that, for me, the relativity was difficult to ascertain; on a system that’s been running for while the IO numbers can be vast and on a system with a lot of data files it can be difficult to spot the troublesome files/filegroups, hence the addition of percentages and percentage ratios. Another inspiration was the Performance Dashboard Reports (which don’t work on SQL Server 2008 R2 out of the box but I hear that they can be ported back to SQL Server 2008R2 from SQL Server 2012, or you can try this fix) which do offer percentages for IO, but also includes the IO on the log files which can make it difficult to get a clear picture of IO that was exclusively related to data files.

The main reason for this post was to highlight the presence and significance of this DMV and also to point out that it should not be used in isolation. Other methods for gathering performance related data, like perfmon and wait stats data are invaluable and all this information combined, together with corroborating evidence like e.g. slow IO warnings in the SQL Server error log will give a more rounded picture of IO performance.

The benefits of flash memory as a replacement for spinning drives in order to boost SQL Server performance is well documented. However, I’m not sure if one particular benefit has been fully exploited. This post covers how we reduced the average duration of our overnight reindex job by 80% with the help of just one Fusion IO card (per node).

A bit of background: in order to give a creaking and very tempdb heavy SQL Server 2008 R2 system a bit of a lift we shifted tempdb onto Fusion IO cards. The system consists of a  2-node cluster running on a SAN. We did indeed see throughput increase, as well as an improvement in concurrency (as we switched on snapshot isolation as well) but some 2 months after the cards were installed another benefit has come to light which has turned perceived wisdom of reorganising versus rebuilding indexes on its head (disclaimer: for our system running on our hardware, at any rate).

I’ve been DBA-ing for so long now that the good old 5%/30% general advice for reorganising as opposed to rebuilding an index as documented on BOL is so deeply ingrained that I’ve hardly given it a second thought and just followed it automatically, only tweaking it when performance analysis showed an issue with these thresholds. For more background info on these thresholds I strongly advise reading Paul Randal’s post on where these guidelines came from.

With tempdb on a much faster IO subsystem than the application database the SORT_IN_TEMPDB option for the ALTER INDEX command suddenly comes into play. After installing the Fusions I tweaked our reindex job to turn the SORT_IN_TEMPDB  option on, made a mental note to follow this up in a few days time, and promptly forgot about it until several weeks later when I wanted to track down indexes that were getting highly fragmented (we use Ola Hallengren’s excellent maintenance script which logs this kind of information (and more) in a table) and noticed some  indexes had huge variations in the time taken to reindex them, with a full rebuild taking a fraction of the time a reorg did, and that’s when the penny dropped.

The Fusion IO boost

I promptly turned off reorgs so now it rebuilds at 10% with every index typically getting fully rebuilt in under 5 minutes (multiple indexes are between 10-14 GB in size). Needless to say, this has had a massive impact on the total reindex time. The reindex job on this system used to take anywhere between 4 to 9 hours which was a real problem as this would often encroach well into the business day.
Since we switched off the reorgs, the reindex has never taken more then 2 hours (and the longest run at 114 minutes included 45 minutes of blocking). The graph below shows the performance improvement very er…graphically:

Fig. 1: Reindexing duration decrease

Purdy, ain’t it? As we can see, February 16th was the first night the reindex job used the rebuild option in place of reorgs, and the drop off in the run time could not be more dramatic. For the two weeks from 1st February – 14th February (inclusive) the average duration was 361 minutes, or 6 hours. After the SORT_IN_TEMPDB tweak on the 16th, the average (from the 16th up until today (March 11th) has been just 78 minutes.
Incidentally, the Fusion IO cards in question are 785GB ioDrive2 Mono MLC PCIe Solid State Storage Cards, and yes, technically MS don’t support this tempdb configuration on a pre SQL 2012 cluster, but lots of people are doing it.

The benefits

For us, modifying the reindex job to drop the reorgs in favour of rebuilds on this particular system was a no-brainer. The reindexing finishing in a quarter of the time taken previously on a consistent basis every night has eliminated the uncertainty and inconvenience caused when the original approach caused the index to collide with subsequent maintenance jobs or encroach into the working day of the system.
Furthermore, the database was in simple recovery so we didn’t have to worry about transaction log backups (although a reorg can have a greater impact on transaction log backups).

This is either a benefit that’s so obvious that no-one has bothered blogging about it, or it’s something that’s gone under the radar thus far (I’m obviously hoping it’s the latter). I’ve seen posts hint at it by mentioning index creation is faster etc, and I’ve even read a MS whitepaper analysing tempdb on Fusion IO on a SQL 2008R2 system (which did a reindexing test with the SORT_IN_TEMPB option on, but left the reindexing thresholds on defaults) but no-one, as far as I can tell, has gone the whole hog with regard to adapting their approach to reindexing and just rebuilding indexes in favour of the reorg option to fully exploit their investment in Fusion IO cards (or they just kept quiet about it, whereas I couldn’t help blabbing about it!); I can think of a number of scenarios and previous systems where the improvement in reindex job performance alone would justify this investment.

Are reorgs redundant?

If you have your tempdb on Fusion IO or similar technology, with the application database(s) on a slower IO subsystem with the reindexing strategy left on original settings its an option well worth exploring.

The things to look out for are the bandwidth from host to SAN as on this system we do briefly get close to maxing it out during the reindex now as it reads in the index pages and writes out the rebuilt index. Personally, I’d rather fully exploit our infrastructure than have it burbling along at the fraction of its capacity, which is what it used to be like when the most of the indexes were being reorganised.

Another consideration is the latency and throughput difference between the storage subsystem the application dbs are stored on and the Fusion IO cards; against spinning disk SAN based systems there’s no competition, the Fusion IOs will outperfrom the SAN in every category, but if you have e.g. SSDs already in the SAN the difference may not be that substantial, so the switch may not bring about such a dramatic boost. On this particular system, in our environment, there was a huge disparity between SAN performance and the Fusions which has made the switch so beneficial for us.

If alternative HA strategies like log-shopping have been implemented the impact will have to be measured as the costs may outweigh the benefits (thanks Paul!).

A final thing to bear in mind is that if you are fortunate enough to have some older/smaller Fusions lying around (perhaps you’ve had to swap them out as the application dbs have got bigger) it’s a great way of recycling these cards on other systems to give them a new lease of life.

So are reorgs redundant? If you have access to this kind of technology, it’s easy enough to prove on your system.

In summary

“It depends”, and “your mileage may vary” have become well worn cliches, but they are so true. Every system is different and there are a number of factors to consider, as (I hope) the post outlines, but the potential benefits warrant exploring this if you have not already done so.

Fortunately, this has all now changed. Gone are the calls to undocumented extended procs or other home-brewed solutions because now (for those of us working on SQL Server 2012 onward), we can just run a straight t-sql function:

SELECT SERVERPROPERTY('INSTANCEDEFAULTDATAPATH') as [Default_data_path], SERVERPROPERTY('INSTANCEDEFAULTLOGPATH') as [Default_log_path];

I found this entirely by chance whilst tracing SSMS in order to update a server build script.
The SERVERPROPERTY function has been around for a while and provides really useful information about that SQL Server instance, but for some reason the SQL Server 2012 documentation has not been updated to reflect these two new additions to this function, which is a bit of a shame so I’ve decided to plug it in this post, as well as submitting a connect bug on this, as I’m hoping this is just an oversight and it’s not been deliberately left as an undocumented command.

Incidentally, if you’re wondering how to change the default data path or the default log path, connect to the server in SSMS, right click on the server in Object Explorer and select Properties. On the Server Properties dialog box look for Database Settings in the object list and make the necessary  modifications under Database default locations.

(EDIT: I did get a response the same day I logged the connect bug and this post, but it got short shrift and closed as a “Won’t fix” which I cannot understand, so if more people vote on it maybe they’ll change their mind…)

Over the years some of these have been made public – the most famous of which is PSSDIAG on codeplex, which was released with some fanfare (at least within the SQL community).

However, there’s one which is (arguably) even more useful as it doesn’t just apply to SQL Server. It’s called err.exe and was originally produced by the MS Exchange team. Essentially all it is is a command line tool to which the relevant error code is passed and it returns all the info it can find on it. Err.exe will look up just about any MS error code. It will dig out the full error message and which application/OS libraries it came from. It’s saved me from a lot of digging around and head-scratching in the past.

Download it. Extract it. Add the folder you’ve extracted it into to your path, and you’re done.

To use it, open a command prompt type “err <error code>” and hit return as per the following screenshot:

Output of Err for 0x8009030c error code

I’ve searched for the 0x8009030c error (the dreaded “SSPI handshake failed with error code 0x8009030c” message) and the real beauty of this little tool is that it will accept either a decimal or hexadecimal value and it will show you the error message and which application header file it found the message in along with any additional information available in the header as the above example shows.

The next example shows a search for decimal code 170456 and the multiple places this error can be returned from:

Output of check 170456 error

I’ve found it extremely useful when looking up error codes in the SQL Server error log and the Windows event logs, and I’m sure it’ll save y’all time too…

At some point every dba will have had to look at the current SQL Server error log using their favourite text editor.

However, as the default error log file is literally just called errorlog (with no file extension), there’s the awkward process of selecting the application with which to open the file every time you need to view it, with no option to associate errorlog with your favourite text editor thanks to the lack of file extension. If it’s a 3am callout and the server is down, you can’t use Management Studio to view and filter the log so your only resort is to manually load the file and anything that makes this easier is always helpful.

Like all good solutions, the answer is deceptively simple and I use it whenever I build a SQL Server instance (and it also has an added benefit I’ll describe later); change the default error log filename to have an .txt or .log extension. It’s an old trick I learnt from my Sybase days and is still very much relevant to the latest and greatest versions of SQL Server (2008 R2, as of the time of writing).

It’s achieved by editing the command line in SQL Server Configuration Manager (SSCM) and editing the default error log location hard-coded in that command line to something more user-friendly. Double-click the SQL Server instance you want to edit in SSCM, under the SQL Server Services folder to bring up the properties, click on the Advanced tab and the command line can be edited under Startup Parameters. Before you do anything, save the current parameters to a text file.

The default SQL Server command line has 3 basic components: the name and location of the master database primary data file, the name and location of the master database transaction log file, and the name and location of the error log. These are represented by the -d (data file) -l (log file) and -e (error file) parameters in the command line. There are other command line options like -E for large page extents and -T for trace flags, as well as options for starting in single user mode and specifying named instances etc etc but they’re all well outside the scope of this post (I’m not mentioning them to show off what I know about the command line, but just to highlight the most likely additional parameters you may come across on any of your servers that you want to make this modification on).

I know most dbas are terrified of playing around with the SQL Server command line, but all we’re interested in is modifying the -e parameter, which by default will be something like ‘-eC:Program FilesMicrosoft SQL ServerMSSQL10_50.MSSQLSERVERMSSQLLogERRORLOG;’. All we’re interested in is modifying the bit between -e and the semi-colon. Simply adding .LOG or .TXT (see Fig. 1) will make your life so much easier when you have to peruse the log file contents of either the current log or rolled-over logs as you can bypass SSMS and just use your preferred text editor.

Inevitably, a restart will be required for the change to take effect, so don’t do this on production systems until you’ve tested on development and staging systems first.

I also prefix the name of the host/instance to the error log name as it makes it so much easier to identify the server concerned if you copy log files down to your desktop machine (as I tend to do quite often).

That added benefit? Not only can you alter the filename of the error log, but you can also alter the path, which means you can keep the SQL Server error logs, SQL Server Agent error logs and default trace files in a separate location, as it is this folder that you specify for the error log that dictates where these other files get written to.

Disclaimer: Do this entirely at your own risk. I’ve carried these instructions out on production systems on places I have worked out and am comfortable with making these changes. Test them on development/staging servers first if you’re doing this for the first time or are unsure about making changes.

There’s also a Connect enhancement request to give the error log an extension by default as well as to add the instance name to the name of the log, so feel free to vote for it on the Microsoft Connect site.

Combined with the output of another column in this DMV called estimated_completion_time we can create a query which neatly displays when a backup or restore operation started, how long SQL Server expects it to take and therefore when it will complete, with a percentage completion column thrown in for good measure.

Here’s the query I put together whilst waiting for my backup performance tests to complete:


select
session_id,
convert(nvarchar(22),db_name(database_id)) as [database],
case command
when 'BACKUP DATABASE' then 'DB'
when 'RESTORE DATABASE' then 'DB RESTORE'
when 'RESTORE VERIFYON' then 'VERIFYING'
when 'RESTORE HEADERON' then 'VERIFYING HEADER'
else 'LOG' end as [type],
start_time as [started],
dateadd(mi,estimated_completion_time/60000,getdate()) as [finishing],
datediff(mi, start_time, (dateadd(mi,estimated_completion_time/60000,getdate()))) - wait_time/60000 as [mins left],
datediff(mi, start_time, (dateadd(mi,estimated_completion_time/60000,getdate()))) as [total wait mins (est)],
convert(varchar(5),cast((percent_complete) as decimal (4,1))) as [% complete],
getdate() as [current time]
from sys.dm_exec_requests
where command in ('BACKUP DATABASE','BACKUP LOG','RESTORE DATABASE','RESTORE VERIFYON','RESTORE HEADERON')


This will return the following:

The database name has been converted to varchar so the output of the query looks ok when result output is set to text mode, otherwise the database name gets padded with blank characters up to the width of the sysname datatype.
It comes in rather useful when running a backup or restore operation where the WITH STATS option has not been used – I’m actually finding that this script is preferrable to the output the WITH STATS option provides, as it can be run on demand and the estimate completion time will get updated with the latest predicted time whenever it is run.
Just bear in mind that if instant file initialisation is off the first few minutes of a new restore will be spent initialising the database files so the predicted end time will be wildly pessimistic. Once the file initialisation is complete the predicted end time will return more accurate information.
I have also found that during a large restore the end time is not accurate for the first 5 minutes or so.

What are waitstats?

In short wait stats are statistics gathered on what resources SQL Server is waiting on internally whilst executing queries.

Why are they useful?

They can often provide very quick and objective evidence of a performance bottleneck by showing cumulative wait times that SQL Server spent on getting access to the CPU, memory or disk access as well as numerous other internal resources.
This post is only going to concentrate on IO related waits.

Gathering this information on at least a daily basis can provide invaluable data on a system’s performance over time and can reveal sudden changes in behaviour caused by e.g. a recent change to an application. Speaking from personal experience, using historically captured wait stats and contrasting them with sudden changes to the topmost wait types is an invaluable way of proving that recent application changes were the trigger for a change in behaviour.

How do I gather these wait stats?

That’s fairly straightforward – it’s a T-SQL statement. First, the SQL Server ‘version independent’ method:

DBCC SQLPERF (WAITSTATS)


The sample output from my local SQL Server 2005 Express installation is below:

Wait Type                        Requests      Wait Time     Signal Wait Time
-------------------------------- ------------- ------------- ----------------
MISCELLANEOUS                    0             0             0
PAGEIOLATCH_NL                   0             0             0
PAGEIOLATCH_KP                   0             0             0
PAGEIOLATCH_SH                   178           327           0
PAGEIOLATCH_UP                   16            436           0
PAGEIOLATCH_EX                   14            0             0
PAGEIOLATCH_DT                   0             0             0
IO_COMPLETION                    173           1279          0
ASYNC_IO_COMPLETION              1             1294          0
CHKPT                            1             202           0
BACKUPIO                         0             0             0
DISKIO_SUSPEND                   0             0             0
IMPPROV_IOWAIT                   0             0             0
WRITELOG                         29            639           15

From SQL Server 2005, the wait stats data has been split into two separate dynamic management views (DMVs) sys.dm_latch_wait_stats and sys.dm_os_wait_stats. The latch wait stats are counters for internal waits within the SQL Server database engine and the OS wait stats accumulate data on waits for external resources such as CPU, disk and memory.

As we’re concentrating on IO waits, its the sys.dm_os_wait_stats we’re interested in.

The following query will grab the IO wait types output:

select * from sys.dm_os_wait_stats
where
wait_type like 'PAGEIO%' or
wait_type like '%IO_COMPLETION' or
wait_type like 'DISKIO%' or
wait_type like 'BACKUPIO%'or
wait_type like 'WRITE%'
order by wait_time_ms


wait_type                                                    waiting_tasks_count  wait_time_ms         max_wait_time_ms     signal_wait_time_ms
------------------------------------------------------------ -------------------- -------------------- -------------------- --------------------
PAGEIOLATCH_SH                                               20892941             110133998            17895                332331
PAGEIOLATCH_EX                                               392948               7270457              2395                 10353
IO_COMPLETION                                                719058               1061552              916                  6929
PAGEIOLATCH_UP                                               4332                 81307                3635                 1243
BACKUPIO                                                     3998                 16393                240                  18
ASYNC_IO_COMPLETION                                          133                  14771                805                  0
PAGEIOLATCH_DT                                               0                    0                    0                    0
PAGEIOLATCH_NL                                               0                    0                    0                    0
PAGEIOLATCH_KP                                               0                    0                    0                    0
DISKIO_SUSPEND                                               0                    0                    0                    0

Incidentally, these queries are only meant to show how to grab the data, if you hunt around on the web you’ll find some elegant queries for processing this data.

This KB from MS discusses these wait types (and more).

SQL Server 2008 R2 has even more granular wait-types to isolate IO related waits, but the above query has been limited to allow it to work from SQL Server 2005.

wait_time_ms is the total accumulated wait time (since that SQL Server instance was last restarted) spent on accessing that resource.
max_wait_time_ms is the peak wait time for one of these requests.
signal_wait_time_ms signal wait time is how long was spent waiting to get access to the runnable queue for that resource, and can indicate CPU pressure if this is high.

If these IO related wait types are high in the list of overall waits (see the Waits and Queues article from Microsoft lower down), then you have an IO issue.
This is either down to the performance of the IO subsytem, or the physical design of the database. The design issues could be due to the file/filegroup layout of the database concerned, but also related to the layout of tempdb (in a previous role it never ceased to amaze me how much thought went into the layout of an application database at various customer sites, whilst tempdb was left e.g. at it’s 8MB starting size with (just one) data file and log file sharing the same drives and/or sharing application database drives).

Take a close look at the particular wait type concerned. If the WRITELOG wait type is dominant, you know it’s the log drive if this is backed up by sys.dm_io_virtual_file_stats. If it’s BACKUPIO, we can infer that it’s the backup drive(s).
A word of caution, don’t talk the numbers at face value.  For example, if the PAGEIOLATCH_SH wait type is high in the list, we know it’s the data drive(s), but that does not necessarily mean there is an issue with the disk subsystem, merely that a lot of IO requests are being issued and this could point to database design issues – if some of the tables had better indexes the query optimizer can come up with a more efficient query plan and retrieve the same data with far fewer IOs. Needless to say (almost), but corroborating evidence from the DMVs or perfmon should always be gathered whenever possible.

These are just a few examples, if you gather this data regularly a clear picture can be built up of what performance issues are affecting your systems, and by picking upon trends you can get advance warning of impending issues.

For further information on wait types and what all the output columns mean, there is an excellent Waits and Queues document on the Microsoft website which goes through them and is highly recommended reading.

Thomas Kejser also has an excellent bottleneck analysis script.

The most telling output can be from identifying IO stalls.
SQL Server 2000 had fn_virtual_filestats and this has now been incorporated into sys.dm_io_virtual_file_stats which is coming up in the next post…

The partition offset issue is relevant to any disk partitions created prior to Windows Server 2008. What this means is that whilst your OS may be Windows Server 2008, if the disk you are using was migrated from an older OS the chances are that it may still be exposed to the problem.
Also, certain OEM built machines may be exposed to this problem even on Windows Server 2008 as they often include hidden partitions used for e.g. server recovery date.

Partition offset

Without going into too great a level of detail on how a disk is structured both physically and logically, the partition offset is basically where Windows will start writing its first bytes to disk once the partition concerned is first used.
When a disk partition is created in Windows Server 2003 and before, the first 63 sectors are reserved for the MBR as well as disk vendor proprietary information. That means Windows Server 2003 and before (hereon referred to as just Windows) will write its first bytes of data on the 64th sector.

What’s the big deal?

These 63 sectors are invariably 512 bytes (worth checking with your SAN team/disk vendor however as some SANs may use larger sector size but present the sectors as logical 512 bytes sectors to Windows). Therefore the 64th sector will begin at a 31.5KB sector boundary. What this means is that if this is a SQL Server disk, SQL Server will write its first data page on a 31.5K sector boundary, and every nth page will be written across these boundaries.
The big deal therefore is that each of these nth IO operations will result in an additional IO to read or write the data that was on the page that crossed that boundary (the actual value of n will depend on cluster (sector) size and stripe unit size).
What this means in real (performance) terms is that a properly aligned system will yield a 20-30% performance improvement in IO latency (and therefore query execution time). All this for free, without having to analyse SQL Server configuration, indexing strategy, query plans etc etc.

How do I check my offset?

Checking the offset can be done via a wmi console (wmic) command.

The above command will show output similar to the following (if the wmic command is being executed locally, there’s no need for the /node parameter):

There will be one line of output for each disk and each disk will be identified by its disk number.
Use the Disk Administrator to identify how these drive letters map to your drive letters.
Surprisingly enough, the StartingOffset column is the key column, and will show the starting offset in bytes.
Disks 1 and 2 have 64 sector offset which lines up with e.g. legacy controllers on local disks.
Drives on default values will show a value of 32256, such as disks 0, 3 and 4 in the output above. Divide that by the 512 byte sectors and you’ll get that dreaded value of 63.
Disks 5 and 6 have a 1MB offset which should line up with most modern disk, SAN cache and cluster size scenarios.
What the above output shows is that, depending on the age of the system, there can be a myriad of offsets used which were correct according to how the sytem was configured at the time. Disks 1 and 2 were altered from the default, so at some point in this system’s history someone looked into the issue of partition misalignment and tweaked it for this drive.
So why are disks 3 and 4 on misaligned defaults? Well, have you ever had disks recut to (usually) increase storage. That usually involves blowing away partitions and recreating them. If partition offsetting was not carried out when the partition was created for whatever reason, this is what you can end up with. The newest disks 5 and 6 were created with the latest information to hand, hence the higher offset, but as this slightly contrived example shows (it’s based on real world systems) it pays to be vigilant and review systems that were previously correctly set up.

How do I fix it?

On paper, the fix is very simple; adjust the partition offset. The diskpart command line utility is used to reset the offset.
Move the offset along from the default misaligned 31.5K offset to an offset that works with your cluster size and stripe unit size; 64K, or a multiple of 64K usually works on most systems, but where a SAN is involved, it gets more complicated as stripe unit sizes will vary from vendor to vendor and rig to rig, so you’ll have to get your SAN engineer/vendor involved.
Windows Server 2008 defaults to 1024K offset.

Ideally, the offset needs to start at a point where the IO requests that SQL Server issues (which will be a multiple of 8K, depending on the IO request type being fulfilled), the cluster size, and any caching involved at disk controller/SAN level all line up, thus avoiding additional IO requests.
The downside, of course, is the partition will have to be recreated, so downtime will be involved on standalone servers that have no HA solution implemented.

Realigning the partition offset involves blowing away the disk partition in question, so if you’re not familiar with diskpart, leave it to your relevant storage/Windows team to reset the partition. If you really want to do this yourself, you’ll find step-by-step instructions in the SQLCAT team’s Disk Partition Alignment Best Practices article linked to at the bottom of this post.

The resulting 20-30% improvement in latency times have been verified by the SQLCAT team who have published them in that paper on partition alignment mentioned above, and is highly recommended reading for anyone with an interest in this subject.

Links

If you only read one further article on partition offsets, make it this one!
Disk Partition Alignment Best Practices for SQL Server

SQL Server 2000 I/O Basics

Disk performance may be slower than expected when you use multiple disks in Windows Server 2003, in Windows XP, and in Windows 2000

It’s a broad topic and all aspects of it are pretty well covered in various blogs and technical articles, but there’s not much out there that gathers it all together. The intention of this series of posts is to try consolidate some of this information and focus in on some of the functions and procedures that can be used to monitor, troubleshoot and configure SQL Server disk configuration and most, importantly, disk performance. It won’t be exhaustive and cover every conceivable option, but will be based on experience gained whilst working at Sybase and Microsoft and sprinkling of ‘real world’ database administration experience.

I’ve split the topic up into smaller sub-topics in order to make it more manageable:

1. Partition offsets
2. Wait Stats
3. Tracking SQL Server IO performance
4. Physical Fragmentation
5. System Monitor (Perfmon)
6. Tempdb

Kinda ambitious, I know, but I don’t blog often and it gives me something to aim for!

There are plenty of excellent articles which cover IO in some depth, so I’m going to try my best to avoid regurgitation and try keep things DBA-centric.  (I highly recommend reading the links at the end of the relevant posts to anyone wanting a deeper understanding of IO issues, from configuration to monitoring and troubleshooting).

Until solid state disk prices come down and become ubiquitous, disk access is still a painfully slow (relatively speaking) mechanical process. An IO (read or write) request is issued by SQL Server which is passed onto the OS. The OS then passes this onto your disk controller or HBA and eventually this data request translates into a mechanical arm sitting above a spinning disk which will read or write the data that SQL Server requested.

What this means in performance terms is that disk access will be measured in milliseconds.

Access times to the other major components of a DBMS that directly affect performance such as CPU and RAM can be measured in microseconds or nanoseconds.

As you can see, it’s an order of magnitude difference.

This single factor alone highlights why reviewing a SQL Server database installation’s disk configuration can be such a valuable and rewarding exercise.  In fact, I begin many of my SQL Server troubleshooting exercises by eliminating IO performance first.

Prior to SQL Server 2000 SP4 it was difficult to identify IO performance issues without a lot of digging around. The biggest clue SQL Server would give would be the occasional mysterious 17883 (or even 17884) errors, and these were not always guaranteed to be IO related, although IO was usually heavily implicated. SP4 for SQL Server 2000 introduced those ‘slow IO’ messages which most DBAs will be familiar with and was a step in the right direction.

SQL Server 2005 introduced various DMVs which would give much greater insight into IO performance, and SQL Server 2008 built on this with the Performance DataWarehouse. These posts are going to cover methods that can be applied from SQL Server 2000 onwards, so will not delve deeply into features and properties only available in later versions, but focus instead on functions and features available across the last three versions of SQL Server as I know there’s still a lot of SQL Server 2000 installations out there.

The default SQL Server trace in SQL Server 2005 onwards is a background trace that runs continuously and records event information that, despite the adverse comments on the web about its value, can be useful in troubleshooting problems. Not only that, but it’s there out-of-the box so you don’t have to do put any effort into enabling or managing it.

This blog post intends to show that, in the right hands, the default SQL Server trace in SQL Server 2005 and SQL Server 2008 can yield very useful information.

What is the default trace?

A default installation of SQL Server 2005/2008 results in a lightweight server-side trace running continuously in the background.
The trace records a handful of events to a trace file which can be loaded up and reviewed in SQL Profiler, just the same as any other trace file.
By default, five trace files are kept in the same folder as the SQL Server error log. Each file has a size limit of 20MB before it rolls over to the next file. After the fifth file is filled, the default trace rolls over to the first file, and so on.
To confirm if the trace is enabled, run the following query:

sp_configure 'default trace enabled'


A run_value of 1 indicates the default trace is enabled and should be active.

If the trace is enabled, the following query will list the details:


select * from sys.traces where is_default = 1


Amongst the output will be confirmation that the trace is running (is_shutdown will be set to 0 if it is running, otherwise it will be 1). The path column will show the full path and name of the current default trace output file.

Interrogating the default trace

Getting information out of the default trace file is much the same as getting information out of any other SQL Profiler trace file. There are multiple options, from simply loading the file in SQL Profiler, interrogating it directly via T-SQL statements, or by loading it into a table.

I’ve used all three methods in the past. The method you use depends on your own preferences and immediate requirements.

The examples in this blog will interrogate the trace files directly, as that’s the quickest way to get up and running and view the contents of the trace.
With more experience, the trace file can be regularly polled via e.g. a stored procedure that can be incorporated into standard health-checks to automatically retrieve events of any interest; you decide what those interesting events are. More on that later.

The queries I will demonstrate in this post reference a system table called sys.trace_events. This is used to get the graphical name of the events I’m interested in (otherwise we will only see the numeric event id, as that is what the default trace records).

This table, introduced in SQL Server 2005 holds details of the all the eventclass ids a SQL trace uses, so you can use this to identify which eventclass id corresponds to which event, which is how I established eventclass 20 is a login failed event (the actual event is Audit login failed).

First things first

The first thing to do is isolate the trace file in question.
Assuming the default trace is enabled and running the following query will return the current trace file and its location:

select path from sys.traces where is_default = 1


If the current trace does not overlap with the time of the error you are trying to track down, go to the location of that trace file and identify which of the preceding trace files is from the correct time period. As a precautionary next step, make a copy of the trace file, so you have a permanent record, and also to prevent the file being overwritten during a subsequent rollover.

Example 1: Finding out who filled up tempdb (or any other database for that matter…)

So, let’s get started. A common scenario for a dba is getting called out overnight because somebody, or some thing has maxed out all the space in e.g. tempdb.
Using the fn_trace_gettable function to interrogate the trace file directly we can get some useful information about the cause. Assume there was an 1105 error just before 21:00 last night (“Could not allocate space for object ” in database ‘tempdb’”).

To find this out it’s just a question of confirming when the exact time the database full error occurred and filtering the trace from that period for events in that database. Deceptively simple isn’t it? The query below does just that.


select te.name as [event],e.textdata, e.applicationname, convert (varchar(20), object_name(e.objectid)) as object, e.spid, e.duration/1000 as [duration (ms)], e.starttime, e.endtime, e.databasename, e.filename, e.loginname, e.hostname, e.clientprocessid
from fn_trace_gettable('C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLLOGlog_326.trc', default) e
inner join sys.trace_events te on e.eventclass=te.trace_event_id
where databasename = 'tempdb'
and e.starttime between '2009-09-24 20:30' and '2009-09-24 21:00'
order by e.starttime


Query 1: Who maxed out tempdb (or any other database)?

The query tracks all events in tempdb that occurred in tempdb for about half an hour before the tempdb full error.

Fig. 1: Events leading to tempdb full

Most of the events causing this type of issue will be object creation and autogrow as can be seen above. What we will also have is what was causing them; applicationname , loginname, spid, hostname and clientprocessid . These should all be enough to identify who or what was responsible for the tempdb issue (see fig. 1).

Sometimes, the objectname column will have the names of the tables being created, and the spid column will allow you to narrow down the trace even further in case tracing events half an hour before the database full problem occurred presents too much or too little data.

Needless to say, but I’ll say it anyway, change the databasename column to the name of the database you’re interested if you’re looking for events leading to the same issue in another database.
Also, the trace can be filtered on object creation (trace_event_id = 46) or data or log file autogrow (trace_event_id = 92 or 93) events

Without this information you have nothing to go on, unless you happened to be logged in at the server at the time the issue was occurring and realised that there was a long running transaction running against tempdb which was going to cause it to fill up.
You might also notice that for the autogrow events the Duration column records how long the autogrow took – very useful in identifying potential IO performance isues (or impractical file growth settings).
The object column returns the name of the object being referenced/created. If this query is not run against the trace file on the server that the error occurred on, the chances are this column will contain NULLs, so you can select just the raw objectid instead to view the values and then manually retrieve the object names from the affected server.
Incidentally, the fn_trace_gettable function is also the function used to load a trace file into a database table.

Example 2: Isolating login failures

OK, so the default trace has some use, but what else can it do?

Suppose you get the occasional login failure on a system, you’re not too worried as that’s inevitable sometimes, but from time to time the login failure is for sa and the ip address logged in the error log doesn’t correspond to any workstation belonging to a dba who should have access. You should be worried now.


select e.name as eventclass, t.textdata, t.hostname, t.ntusername, t.ntdomainname, t.clientprocessid, t.applicationname, t.loginname, t.spid, t.starttime, t.error from fn_trace_gettable('c:program filesmicrosoft sql servermssql.1mssqlloglog_329.trc', default) t
inner join sys.trace_events e on t.eventclass = e.trace_event_id where eventclass=20


Query 2: Isolating where a login failure came from

Fig 2: Isolating login failures

The query references the trace file from the period of the login failure and retrieves the columns containing all the information that should be needed to pin down where any login failure recorded in that file came from. I’ve already covered how to use this information in another blog posting covering how to identify the source of login failures.

Example 3: Identifying performance issues

So we’ve seen some practical applications like pinning down login failures and database or log growth issues which makes the default trace pay for itself in some ways. But wait, there’s more.

The bane of most dbas lives’ are application teams who will describe a performance problem they have: great, you say, what’s the application and SQL Server database, I’ll log on and take a look? Oh, it happened last week, some time in the afternoon the application team will reply. That’s OK, you (want to) say, I’ll just step into my time machine and go back a few days, or get in touch with my sixth-sense and carry out some sort of mind-meld with SQL Server and try and find out why it was mis-behaving a few days ago. [Disclaimer] Before you go any further I would like to make clear that none of this will be possible via the default trace. It can do quite a few things, but time-travel and mind-melds are slightly out of scope.

The default trace does offer some hope here. Because by default 5 trace files are kept (including the current trace files) there is a chance that the older trace files may cover the time period in question.

The default trace will capture some common performance affecting events, namely:

• Missing Column Statistics
• Hash Warning
• Sort Warnings
• Missing Join Predicate

So, if you still have the trace file from the period in question, a query like the following will pick out all the those events:


select v.name as [Event Class], TextData, SPID, Duration, StartTime, EndTime, DatabaseName, ObjectName, ClientProcessID,
ApplicationName, LoginName, SessionLoginName, NTUserName
from fn_trace_gettable('C:log_180.trc', DEFAULT) trc
inner join sys.trace_events v on
trc.EventClass = v.trace_event_id
where
v.name in ('Missing Column Statistics','Hash Warning','Sort Warnings','Missing Join Predicate')
and StartTime between '2009-09-05 14:00' and '2009-09-06 16:00'


Query 3: Isolating who made table DDL changes

Example 4: Finding performance problem events

Let’s start by clarifiying that this is not going to tell us what the query was that generated these warnings. At the end of the day, the default trace is designed to be a low-impact trace that will not cause any performance or load issues so what it gathers is therefore restricted.
What we do have, however are details of when all these errors occcured, the PID of the offending client process, together with the hostname, application name and login name.

Armed with that information, we can then go back into that trace and zoom in on a particular spid or application. By narrowing down the spid or application, but expanding the events we’re looking at we can find out a bit more about the problem. I use a combination of the time period, spid and clientprocessid to narrow down (see my January 2009 blog on login failures on how to use this handy column to track down exactly what application is connecting).


select v.name as [event class], textdata, spid, duration, starttime, endtime, databasename, objectname, clientprocessid,
hostname, applicationname, loginname, sessionloginname, ntusername
from fn_trace_gettable('c:log_180.trc', default) trc
inner join sys.trace_events v on
trc.eventclass = v.trace_event_id
where
starttime between '2009-09-05 14:00' and '2009-09-06 16:00'
and spid = 57 and clientprocessid = 2044


Query 4: Narrowing down performance problems

Fig 3: Performance problem events

I’ve replaced figure 3 with the output of an identical query (but using different time range parameters) trace from a server where the output allowed us to identify what query in a large batch job that was running overnight needed tuning. The output showed what time the query ran, the relevant details of the client application and the names of some of the tables/indexes that were affected .

Example 5: Finding out who made changes

Perhaps one of the most useful aspects of the default trace is the fact that it records changes made to objects and server configuration changes

Suppose someone added or removed a column from a table? The following query highlights who did it:


select e.name as eventclass, t.textdata, t.objectid, t.objectname, t.databasename, t.hostname, t.ntusername, t.ntdomainname, t.clientprocessid, t.applicationname, t.loginname, t.spid, t.starttime, t.error from fn_trace_gettable('C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLLOGlog_339.trc', default) t
inner join sys.trace_events e on t.eventclass = e.trace_event_id
where eventclass=164


Query 5: Isolating who made table DDL changes

Eventclass 164 is the Object:Altered event, and the above query should show what application and login made the changes.

If the change concerned was e.g. a server configuration change like e.g. the max server memory setting, the following code will isolate the culprit:


select e.name as eventclass, t.textdata, t.hostname, t.ntusername, t.ntdomainname, t.clientprocessid, t.applicationname, t.loginname, t.spid, t.starttime, t.error from fn_trace_gettable('C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLLOGlog_339.trc', default) t
inner join sys.trace_events e on t.eventclass = e.trace_event_id where eventclass=22


Query 5a: Isolating who made server configuration changes

Eventclass 22 is the ErrorLog event, so of course the changes can also be seen in the error log, but the error log won’t give you the details of who made the change.

Disabling the default trace

The chances are, most SQL Server 2005 (and beyond) systems are running this trace in the background without anyone even being aware of them, which in itself highlights what low impact these traces have. However, if CPU or disk resources are scarce the default trace can be disabled.
To disable the default trace, run the following commands:


sp_configure 'default trace', 0
go
reconfigure with override
go


And finally…

This post was not intended to show ever possible piece of information that can be extracted from the default trace, but merely to highlight some of the things that can be found without too much effort.
Have a look at your own trace files as everyone installation and every application running against SQL Server is different – it’s a great way of pro-actively finding potential issues.

So go and explore…