Troubleshooting 0x48 CANCEL_STATE_IN_COMPLETED_IRP

The Debugging Tools for Windows are required to analyze crash dump files. If you do not have the Debugging Tools for Windows installed or dump files are not being generated on system crash, see this post for installation/configuration instructions:
http://mikemstech.blogspot.com/2011/11/windows-crash-dump-analysis.html

0x00000048 CANCEL_STATE_IN_COMPLETED_IRP is a fairly uncommon blue screen of death on the Windows platform (Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, Windows Server 2008 R2, and Windows 8). This bug indicates that the cancel routine was called for an IRP after it was already completed. This is analogous to attempting to stop payment on a check after it has been cashed/deposited and processed... At this point, there is nothing to cancel because the money has already entered/left the bank (depending on the perspective). 

The documentation for the Windows Driver Kit states that this is often caused by more than one driver accessing/processing the same packet, rather than a single buggy driver. An example might be two drivers that believe that they "own" an IRP, one completes the IRP with a call to  IoCompleteRequest and the other driver calls IoCancelIrp on the packet. If both drivers had called IoCompleteRequest, the bug check MULTIPLE_IRP_COMPLETE_REQUESTS would have been thrown. The example with a single buggy driver might occur if a programmer mistakenly creates a IoCancelIrp call after the IoCompleteRequest call in a path of execution.

Debugging a minidump with this stop error isn't always the most informative. I have only come across one example,

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CANCEL_STATE_IN_COMPLETED_IRP (48)
This bugcheck indicates that an I/O Request Packet (IRP) that is to be
cancelled, has a cancel routine specified in it -- meaning that the packet
is in a state in which the packet can be cancelled -- however, the packet
no longer belongs to a driver, as it has entered I/O completion.  This is
either a driver bug, or more than one driver is accessing the same packet,
which is not likely and much more difficult to find. The cancel routine
parameter will provide a clue as to which driver or stack is the culprit.
Arguments:
Arg1: 950fc008, Pointer to the IRP
Arg2: 8fb66ff1, Cancel routine set by the driver.
Arg3: 00000000
Arg4: 00000000

Debugging Details:
------------------


CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR:  0x48

PROCESS_NAME:  w3wp.exe

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 81c82297 to 81c6e72f

STACK_TEXT:  
8f5d1be4 81c82297 950fc008 95027dd8 00000000 nt!IoCancelIrp+0x73
8f5d1c18 81e38234 8800a168 88ffad90 00000000 nt!IopCancelIrpsInFileObjectList+0xb3
8f5d1c74 81e3310a 88ffad90 8800a168 00100081 nt!IopCloseFile+0x409
8f5d1cc4 81e32f9a 88ffad90 0041ed40 00100081 nt!ObpDecrementHandleCount+0x146
8f5d1d14 81e32cad 9e33fa70 9bc6d500 88ffad90 nt!ObpCloseHandleTableEntry+0x234
8f5d1d44 81e33530 88ffad90 a7a89901 a7a89901 nt!ObpCloseHandle+0x73
8f5d1d58 81c9997a 00000280 0019f9cc 76f79a94 nt!NtClose+0x20
8f5d1d58 76f79a94 00000280 0019f9cc 76f79a94 nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0019f9cc 00000000 00000000 00000000 00000000 0x76f79a94


STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt!IoCancelIrp+73
81c6e72f 8a442414        mov     al,byte ptr [esp+14h]

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  nt!IoCancelIrp+73

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  48d1b7e8

FAILURE_BUCKET_ID:  0x48_nt!IoCancelIrp+73

BUCKET_ID:  0x48_nt!IoCancelIrp+73

Followup: MachineOwner
--------- 

In this particular example, the IRP is not dumped,

0: kd> !irp 950fc008
950fc008: Could not read Irp 

Using the ln (list nearest symbol debugger command) on the cancel routine address indicates the driver that set the cancel routine.

0: kd> ln 8fb66ff1
(8fb66ff1)   rdbss!RxCancelRoutine   |  (8fb670b6)   rdbss!WPP_SF_qZLL 

In this case, the "Redirected Drive Buffering SubSystem Driver" registered the cancel routine. Advanced troubleshooting of this issue requires the use of a kernel dump or remote debugger to identify the relevant drivers involved in the device stack (based on the information gathered from the !irp output in the kernel memory dump or from the live remote debugging session). See the !devstack command in the debugger help for more information. Try updating the involved drivers or contacting the vendor(s) for support.

See Also,
Windows Crash Dump Analysis