Comprehensive OSPF Cost Example

The Open Shortest Path First routing protocol is a critical piece of knowledge for any networking professional working in an enterprise environment. Most major networking certifications covering routing and switching including the Cisco Certified Network Associate (CCNA), Cisco Certified Network Professional (CCNP), and Cisco Certified Internetwork Expert (CCIE) extensively test OSPF knowledge and skills. OSPF is the most popular dynamic routing protocol used in complex enterprise networks as an interior gateway protocol (IGP). This post will provide a comprehensive example of the OSPF metric calculation and how different configurations impact the metric for type-1 and type-2 routes. The OSPF lab is configured in Dynamips/GNS3 using 5 Cisco c3725 routers laid out in the topology below. For more information on how the OSPF metric is determined, see OSPF Cost/Metric Calculation.

As I described in a previous post, there are really three main types of metrics considered in OSPF:

• Intra-area and summary cost
  
• External/NSSA type 1 cost
  
• External/NSSA type 2 cost
  

In the topology below, we have a backbone area and two non-backbone areas (one a regular area and one a totally not so stubby [totally NSSA] area).

We have two routing domains running the routing information protocol (RIP) and the routing domain running Open Shortest Path First (OSPF). To get our external routes into OSPF, we redistribute RIP into OSPF on R1 and R2. The RIP routes from R1 are propagated through the OSPF domain (until R02) and the RIP routes from R2 are only propagated through the no-summary NSSA area as N1/N2 routes, then they become E1/E2 routes in area 0 and beyond.

For the initial part of the lab, get the basic OSPF configuration and RIP redistribution set up. In this instance, I use a route map on R1 and R2 to set the following:

172.16.1.0/24	Metric type 1, initial cost 100
172.16.2.0/24	Metric type 2, cost 100
192.168.1.0/24	Metric type 1, initial cost 100
192.168.2.0/24	Metric type 2, cost 100

There are multiple ways to achieve this configuration, but I use a route map and prefix lists. Here are the relevant configuration commands for R1

!
router ospf 1
 redistribute rip subnets route-map set-ospf-metric-type
 network 10.0.0.0 0.255.255.255 area 1
!
ip prefix-list match-172-16-1 seq 5 permit 172.16.1.0/24
!
ip prefix-list match-172-16-2 seq 5 permit 172.16.2.0/24
!
route-map set-ospf-metric-type permit 10
 match ip address prefix-list match-172-16-1
 set metric 100
 set metric-type type-1
!
route-map set-ospf-metric-type permit 20
 match ip address prefix-list match-172-16-2
 set metric 100
 set metric-type type-2
!

And for R2:
!
router ospf 1
 log-adjacency-changes
 area 2 nssa no-summary
 redistribute rip subnets route-map set-ospf-metric
 network 10.0.3.0 0.0.0.255 area 2
!
ip prefix-list match-192-168-1 seq 5 permit 192.168.1.0/24
!
ip prefix-list match-192-168-2 seq 5 permit 192.168.2.0/24
!
route-map set-ospf-metric permit 10
 match ip address prefix-list match-192-168-1
 set metric 100
 set metric-type type-1
!
route-map set-ospf-metric permit 20
 match ip address prefix-list match-192-168-2
 set metric 100
 set metric-type type-2
!

Looking at the Area Border Router (ABR) routing tables, it is clear that the E1/N1 cost increases as it is propagated through the network, but the E2/N2 cost remains what it was initially set to.

From R01:

R01#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     172.16.0.0/24 is subnetted, 2 subnets
O E1    172.16.1.0 [110/110] via 10.0.2.1, 00:12:14, FastEthernet0/1
O E2    172.16.2.0 [110/100] via 10.0.2.1, 00:12:14, FastEthernet0/1
     10.0.0.0/24 is subnetted, 4 subnets
C       10.0.2.0 is directly connected, FastEthernet0/1
O IA    10.0.3.0 [110/30] via 10.0.0.2, 01:57:47, FastEthernet0/0
C       10.0.0.0 is directly connected, FastEthernet0/0
O       10.0.1.0 [110/20] via 10.0.0.2, 01:57:47, FastEthernet0/0
O E1 192.168.1.0/24 [110/130] via 10.0.0.2, 01:57:47, FastEthernet0/0
O E2 192.168.2.0/24 [110/100] via 10.0.0.2, 01:57:49, FastEthernet0/0

From R02:

Gateway of last resort is not set

     172.16.0.0/24 is subnetted, 2 subnets
O E1    172.16.1.0 [110/130] via 10.0.1.1, 00:00:02, FastEthernet0/0
O E2    172.16.2.0 [110/100] via 10.0.1.1, 00:00:02, FastEthernet0/0
     10.0.0.0/24 is subnetted, 4 subnets
O IA    10.0.2.0 [110/30] via 10.0.1.1, 00:00:02, FastEthernet0/0
C       10.0.3.0 is directly connected, FastEthernet0/1
O       10.0.0.0 [110/20] via 10.0.1.1, 00:00:02, FastEthernet0/0
C       10.0.1.0 is directly connected, FastEthernet0/0
O N1 192.168.1.0/24 [110/110] via 10.0.3.2, 03:44:00, FastEthernet0/1
O N2 192.168.2.0/24 [110/100] via 10.0.3.2, 03:44:00, FastEthernet0/1

For internal routes, it is easy to see that the cost increases as the advertisement propagates from the source.

OSPF Cost for Summary Routes

Summary external routes can be created on autonomous system border routers (ASBRs) using the OSPF summary-address router configuration command. External summary route cost is determined according to the following rules:

• If all summary components are E2/N2 routes, the summary is considered E2/N2 and is advertised with the lowest metric (cost) of the summarized routes
• If any of the summarized routes are E1/N1 routes, the summary is considered E1/N1 and is initially advertised with the lowest cost/metric of any of the summarized routes. As the LSA is propagated, the new metric follows regular E1/N1 rules.  

For routes within the OSPF routing domain that are summarized at the area border routers (ABRs) using the area range command, the metric advertised with the summary is the lowest metric of any of the summarized routes.

See Also,
The Road to the CCIE
OSPF Cost/Metric Calculation

OSPF Cost/Metric Calculation

The Open Shortest Path First routing protocol is a critical piece of knowledge for any networking professional working in an enterprise environment. Most major networking certifications covering routing and switching including the Cisco Certified Network Associate (CCNA), Cisco Certified Network Professional (CCNP), and Cisco Certified Internetwork Expert (CCIE) extensively test OSPF knowledge and skills. OSPF is the most popular dynamic routing protocol used in complex enterprise networks as an interior gateway protocol (IGP). This example will demonstrate the concepts and configuration involved with metrics in an OSPF network. I utilize a Cisco c7200 in Dynamips/GNS3 to provide syntax examples in the post below.

Understanding OSPF Cost

Internal Route OSPF Cost

OSPF uses a value called cost for its metric when determining the metric for a particular routing prefix. OSPF defines the cost for a particular prefix according to the following formula:

OSPF cost = cost from LSA + (incoming interface bandwidth/reference bandwidth)

OSPF uses 100 Mb/s for its default reference bandwidth on Cisco routers, but on routers that use Gigabit Ethernet or 10 Gigabit Ethernet, this value should be changed both on the router and everywhere else in the OSPF network. This can be changed using the auto-cost reference-bandwidth  command:

R1(config-router)#auto-cost reference-bandwidth ?
  <1-4294967>  The reference bandwidth in terms of Mbits per second

Note that this is the configured bandwidth on the interface, not the actual bandwidth. The following table shows the OSPF cost for different link types and reference bandwidths:

Link Type	100M (default) Reference Bandwidth	1G Reference Bandwidth	10G Reference Bandwidth
56k serial	1785	17857	178571
64k serial	1562	15625	156250
T1 (1.544 Mbps serial)	64	647	6476
E1 (2.048 Mbps serial)	48	488	4882
Ethernet	10	100	1000
Fast Ethernet	1	10	100
Gigabit Ethernet	1	1	10

The OSPF cost can also be manually set for an interface with the ip ospf cost interface subcommand.

R1(config-if)#ip ospf cost ?
  <1-65535>  Cost

The OSPF cost of an outgoing interface is not added to an outgoing LSA, rather the receiving router adds its own interface cost to the cost that was advertised in the LSA (and propagates that cost to other routers). In this way, the total cost is the cost of each outgoing interface on each router between a particular router and the router connected to a specific prefix.

The OSPF cost can be set for connected networks being advertised into OSPF using the ip ospf cost interface command. It is also possible to set the OSPF cost for a connected route using a route map and redistributing connected networks, though this causes the route to be considered an external route and the metric is determined initially by the set clause of the route map and propagated according to whether it is considered a type 1 external route or a type 2 external route (type 2 is default on the Cisco IOS platform).

It should also be noted that the bandwidth interface configuration command (which has no actual effect on available bandwidth) can modify the OSPF cost for an interface by overriding the bandwidth inferred from the interface type.

Inter-Area and External Route OSPF Cost and Path Selection

Inter-area routes add the cost of the ABR to reach a particular network with the cost to reach an ABR.

External routes redistributed into OSPF are either considered type 2 or type 1 external routes. Type 2 is default and the cost used to determine the shortest path to the advertised network is solely the cost advertised by the ASBR for the prefix (not by the cost to reach any intermediate ABRs or other routers). For type 1 routes, routers add the cost to get to the ASBR to the cost advertised by the ASBR to determine the metric for the route as well as the cost to reach any intermediate ABRs or other routers.

Most of the examples that show routing differences between type 1 and type 2 external routes are theoretical. There aren't very many drivers from a network design standpoint to implement multiple interior gateway routing protocols in a new network or in a network that is being re-engineered or transitioning between routing protocols (ex. RIP -> OSPF or EIGRP -> OSPF). The drivers that do exist mainly involve migrations due to mergers and acquisitions or in high complexity service provider scenarios (mainly interior to a service provider's network or involved with layer 3 carrier MPLS solutions). For VoIP design, EIGRP and OSPF can both be tuned to have subsecond convergence assuming the correct level of redundancy exists in the network and redistribution is not typically required.  

When thinking about OSPF costs, it is important to think about how OSPF chooses routes because the rules may supercede cost in a few different scenarios:

• OSPF prefers intra-area routes to inter-area routes, regardless of cost
• OSPF routes across area zero without routing across a non-backbone area if at all possible
• Finally, OSPF routes to the destination without traversing area 0

This creates challenges with design and troubleshooting because flows may take a non-intuitive path if two non-backbone areas are directly connected with a lower cost than the cost across a backbone area, or if there is a high cost route through the backbone area and a low cost route that traverses the backbone area and a non-backbone area. Special care should be taken with regard to link failures and different external routing scenarios.

See Also
The Road to the CCIE

OSPF Concepts: Areas and LSA Types

OSPF (Open Shortest Path First) is the most popular interior gateway protocol (IGP) used in private networks today. Because of its widespread use, it is one of the most important topics for individuals pursuing any of the mainstream networking certifications available. Both Cisco and Juniper extensively test on OSPF concepts and configuration for their exams at the entry level, intermediate level, and advanced level. Since I have more Cisco background than Juniper, I will describe OSPF concepts and configuration from the standpoint of the Cisco Internetwork Operating System (IOS). Since I am pursuing the second highest Cisco certification, the Cisco Certified Internetwork Expert (CCIE), I will develop both simple and advanced labs and examples with Dynamips/GNS3 involving basic and advanced configuration scenarios. For now, I will describe OSPF version 2 which is currently used for IPv4 networks. At a future time, I will develop a similar set of posts that describes OSPFv3 and IPv6.

Open Shortest Path First is a link state protocol, meaning that it propagates information about individual links (and their connected subnets) and each router builds a complete view of the attached network (really the OSPF area, but more on this later...). Each router analyzes the information in the OSPF link state database and makes its own decision on the best way to route traffic to a particular destination. Compare this with a distance vector protocol such as RIPv2 where each router has no view of the topology beyond the first hop, but only knows the distance (for RIP this is hop count) to reach a subnet that was advertised by the neighboring router. No computation is necessary beyond knowing which next hop has the lowest hop count to a specific destination.

OSPF Areas

Areas are the name given to a set of routers that has a complete view of the link states in any given area. OSPF uses a two layer hierarchy of consisting of a backbone area (area 0) and one or more non-backbone areas. For small networks, it is possible that only a single area is used. Each interface/subinterface can be part of a single OSPF area. All areas other than the backbone area must connect directly to the backbone area (or connect via an OSPF virtual link). Since the routers in a single area have a complete view of the topology, adding more routers to an area increases the size of the OSPF database in memory and increases the time that the shortest path first (SPF) algorithm takes to run. Specifics with each type of area: normal area, stub area, totally stubby area, not so stubby area (NSSA), and totally not so stubby area will be discussed in their own posts.

OSPF LSA Types

Routers in an OSPF area propagate reachable subnets via link state advertisements (LSAs). Within an individual area Type 1 (link) and type 2 (transit network) LSAs help routers develop a complete topology view. Between areas type 3 (summary) and type 4 (autonomous system border router summary [ASBR summary]) LSAs are propagated. Type 5 LSAs are generated and propagated for external routes (those completely outside of OSPF). Type 7 LSAs are used to represent external routes in not so stubby areas (NSSAs). Each of these LSA types and their appearance in the OSPF database will be described in more detail and demonstrated in future configuration examples.

See Also,
The Road to the CCIE

Troubleshooting 0x7E SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

The Debugging Tools for Windows are required to analyze crash dump files. If you do not have the Debugging Tools for Windows installed or dump files are not being generated on system crash, see this post for installation/configuration instructions:
http://mikemstech.blogspot.com/2011/11/windows-crash-dump-analysis.html

0x0000007E SYSTEM_THREAD_EXCEPTION_NOT_HANDLED is a common bug check (blue screen of death) on the Windows platform (Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, Windows Server 2008 R2, and Windows 8). This bug check also occurs with an exception code of 0x1000007E. This error indicates a general kernel mode exception that the error handler did not catch. Parameter 1 identifies the exception code that will provide more insight into the cause of the issue. A couple of the common ones are given below:

Exception Code	Description
0x80000002: STATUS_DATATYPE_MISALIGNMENT	This exception code indicates that an object was not properly aligned with its pointer. This often occurs if a programmer incorrectly calculates the pointer address of an object in an array or other data structure. The error code lookup tool shows the following for this error: {EXCEPTION} Alignment Fault A datatype misalignment was detected in a load or store instruction.
0x80000003: STATUS_BREAKPOINT	This error, if encountered outside of development, indicates a really sloppy software release management process by the driver's developers. Software developer use breakpoints to examine the state of an application at a specific point of execution. Often this is to identify the contents of the variables associated with a specific program at a point of execution. This exception itself indicates that a programmer was working on an issue in the code, but left a breakpoint (which generates an exception to stop execution and pass control to the debugger) in the code that was encountered by the system. The error code lookup tool shows the following for this error: {EXCEPTION} Breakpoint A breakpoint has been reached.
0xC0000005: STATUS_ACCESS_VIOLATION	This indicates that memory corruption occurred at some level. This is typically due to a driver corrupting the memory/system state and another driver or the system kernel identifying the issue at a later time. The FAULTING_MODULE in WinDbg is not reliable for this exception code. The error code lookup tool shows the following for this error: The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
0xC0000006: STATUS_IN_PAGE_ERROR	This error indicates an I/O error that possibly points to a hardware issue. The error code lookup tool shows the following for this error: The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".

Troubleshooting SYSTEM_THREAD_EXCEPTION_NOT_HANDLED is fairly straightforward. For error codes other than 0xc0000005 (STATUS_ACCESS_VIOLATION), the faulting module indicated by kd/Windbg reports the driver (or possibly a related driver in the case of generic drivers like netio.sys and ndis.sys) that needs to be upgraded/downgraded/changed.

The following examples will give troubleshooting ideas for 0xC0000005 and 0xC0000006.

0xC0000006 STATUS_IN_PAGE_ERROR

STATUS_IN_PAGE_ERROR indicates that a memory page(s) were not written to disk or read from the disk due to an IO error. There are various causes for IO errors, but the memory and hard drive should be examined for issues. Below is an example analysis of a dump involving this substatus:

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M (1000007e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000006, The exception code that was not handled
Arg2: 8c3c1532, The address that the exception occurred at
Arg3: 9b2a5398, Exception Record Address
Arg4: 9b2a4f70, Context Record Address

Debugging Details:
------------------


OVERLAPPED_MODULE: Address regions for 'ZTEusbmdm6k' and 'USBSTOR.SYS' overlap

EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at 0x%p referenced memory 

                                        at 0x%p. The required data was not placed 

                                        into memory because of an I/O error status 

                                        of 0x%x.

FAULTING_IP: 
nvlddmkm+3a5532
8c3c1532 8b1f            mov     ebx,dword ptr [edi]

EXCEPTION_RECORD:  9b2a5398 -- (.exr 0xffffffff9b2a5398)
ExceptionAddress: 8c3c1532 (nvlddmkm+0x003a5532)
   ExceptionCode: c0000006 (In-page I/O error)
  ExceptionFlags: 00000000
NumberParameters: 3
   Parameter[0]: 00000000
   Parameter[1]: 85e40000
   Parameter[2]: c0000010
Inpage operation failed at 85e40000, due to I/O error c0000010

CONTEXT:  9b2a4f70 -- (.cxr 0xffffffff9b2a4f70)
eax=00000002 ebx=00000000 ecx=92544000 edx=002b0c70 esi=9b2a54bc edi=85e40000
eip=8c3c1532 esp=9b2a5460 ebp=9b2a546c iopl=0         nv up ei ng nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010286
nvlddmkm+0x3a5532:
8c3c1532 8b1f            mov     ebx,dword ptr [edi]  ds:0023:85e40000=????????
Resetting default scope

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at 0x%p referenced memory 

                                    at 0x%p. The required data was not placed 

                                    into memory because of an I/O error status 

                                    of 0x%x.

EXCEPTION_PARAMETER1:  00000000

EXCEPTION_PARAMETER2:  85e40000

EXCEPTION_PARAMETER3:  c0000010

IO_ERROR: (NTSTATUS) 0xc0000010 - The specified request is not a valid operation 

                                  for the target device.

BUGCHECK_STR:  0x7E

EXCEPTION_STR:  0xc0000006_c0000010

FOLLOWUP_IP: 
+3a5532
85e40000 ??              ???

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: hardware_disk

IMAGE_NAME:  hardware_disk

DEBUG_FLR_IMAGE_TIMESTAMP:  0

STACK_COMMAND:  kb

FAILURE_BUCKET_ID:  0x7E_IMAGE_hardware_disk

BUCKET_ID:  0x7E_IMAGE_hardware_disk

Followup: MachineOwner
---------
 

In this particular error, the IO operation failed with 0xC0000010 (STATUS_INVALID_DEVICE_REQUEST: The specified request is not a valid operation for the target device).

0xC0000005 STATUS_ACCESS_VIOLATION

For most bug check codes, 0xC0000005 indicates that the memory and system state have been corrupted (resulting in a crash when the memory corruption is detected). For the majority of crashes with substatus of 0xC0000005, the issue occurs when the memory is corrupted, but the system crashes when the corruption is detected by another driver or the system memory manager. This typically results in another driver (or the kernel itself) getting blamed for the issue (as shown below). Empirically, there is a high probability of a video driver (ATI or Nvidia) being blamed (though this may or may not be true from the explanation above). Below is a typical analysis of a dump showing substatus 0xC0000005 where the error is blamed on the kernel (nt):

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M (1000007e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 828d1fb0, The address that the exception occurred at
Arg3: 8a743b48, Exception Record Address
Arg4: 8a743720, Context Record Address

Debugging Details:
------------------


EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced 

                                 memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
nt!IopGetFileObjectExtension+f
828d1fb0 8b448104        mov     eax,dword ptr [ecx+eax*4+4]

EXCEPTION_RECORD:  8a743b48 -- (.exr 0xffffffff8a743b48)
ExceptionAddress: 828d1fb0 (nt!IopGetFileObjectExtension+0x0000000f)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 5a035b09
Attempt to read from address 5a035b09

CONTEXT:  8a743720 -- (.cxr 0xffffffff8a743720)
eax=00000001 ebx=84ffff80 ecx=5a035b01 edx=00000000 esi=00000800 edi=85676020
eip=828d1fb0 esp=8a743c10 ebp=8a743c10 iopl=0         nv up ei pl nz na po nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010202
nt!IopGetFileObjectExtension+0xf:
828d1fb0 8b448104        mov     eax,dword ptr [ecx+eax*4+4] ds:0023:5a035b09=????????
Resetting default scope

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced 

                       memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1:  00000000

EXCEPTION_PARAMETER2:  5a035b09

READ_ADDRESS: GetPointerFromAddress: unable to read from 829a8848
Unable to read MiSystemVaType memory at 82987e40
 5a035b09 

FOLLOWUP_IP: 
nt!IopGetFileObjectExtension+f
828d1fb0 8b448104        mov     eax,dword ptr [ecx+eax*4+4]

BUGCHECK_STR:  0x7E

LAST_CONTROL_TRANSFER:  from 828ccc12 to 828d1fb0

STACK_TEXT:  
8a743c10 828ccc12 00000001 00000000 00000800 nt!IopGetFileObjectExtension+0xf
8a743c24 82a7092d 84ffff80 848ac3c0 84ffff68 nt!IoGetRelatedDeviceObject+0x50
8a743c6c 82a61601 84ffff80 84ffff80 84ffff68 nt!IopDeleteFile+0x32
8a743c84 828b7d40 00000000 000c0000 00000000 nt!ObpRemoveObjectRoutine+0x59
8a743c98 828b7cb0 84ffff80 82a66fe1 85b64b18 nt!ObfDereferenceObjectWithTag+0x88
8a743ca0 82a66fe1 85b64b18 85b64b40 829aa980 nt!ObfDereferenceObject+0xd
8a743ccc 828a0f04 85b64b18 00000000 00000000 nt!MiSegmentDelete+0x191
8a743d28 828a1225 848b4020 00000000 00000000 nt!MiProcessDereferenceList+0xdb
8a743d50 82a47fda 00000000 abe63408 00000000 nt!MiDereferenceSegmentThread+0xc5
8a743d90 828f01d9 828a115e 00000000 00000000 nt!PspSystemThreadStartup+0x9e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  nt!IopGetFileObjectExtension+f

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrpamp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  4e02a389

STACK_COMMAND:  .cxr 0xffffffff8a743720 ; kb

FAILURE_BUCKET_ID:  0x7E_nt!IopGetFileObjectExtension+f

BUCKET_ID:  0x7E_nt!IopGetFileObjectExtension+f

Followup: MachineOwner
--------- 

For issues involving STATUS_ACCESS_VIOLATION, troubleshooting usually starts with these steps:

• Rule out a hardware issue with the memory or hard drive
• Enable driver verifier and analyze the dumps after the system crashes again
• Examine the loaded modules (run the "lm nt" debugger command) and BIOS (!sysinfo machineid) and look for older versions that need to be upgraded
• Finally, if the system is under warranty/support, contact the manufacturer as it might be a known issue with a resolution provided by the manufacturer
  

See Also
Windows Crash Dump Analysis

Kerberos Password Policies Made Easy

Password policies are typically part of an organization's larger security policy and dictate items such as:

• Minimum password length
  
• Number of character classes
• Which character classes are used
• Use of dictionary words
  
• Minimum password age
• Maximum password age
• Number of invalid password attempts
• Lockout and duration of lockout

The last two items of the list actually apply to an internal security control known as a clipping level. A clipping level (such as the maximum number of invalid password attempts before an account is locked out) greatly reduces the feasibility of a brute force password attack against a given authentication source because it increases the risk of detection by the user who owns the account and greatly reduces the number of password attempts that can occur during a given period of time.

Here is an example of how clipping levels reduce the number of password attempts in a given period of time. Say the organization only allows 3 invalid password attempts and locks a principal from authenticating for 10 minutes. Assume the attacker can carry out 1000 authentication attempts per minute without the clipping level. To achieve the same 1000 authentication attempts with the clipping level, the account would be locked out for around 3330 minutes (~2.31 days). This results in an average rate 0f 0.3 authentications per minute. As the length and complexity of a password increase, the average number of attempts increases significantly to the point that an attacker is effectively forced to use other methods to obtain a password (such as social engineering or key logging). Assuming that the correct preventative and detective controls are in place, an attacker will be forced to try to find another way to compromise a system/network.

The two main Kerberos distributions in use, MIT Kerberos V and Microsoft Active Directory, both allow password complexity and clipping levels to be set by policy. Microsoft Active Directory password policies can be set by default in the Default Domain Controllers Group Policy Object (GPO) and can be set in the Local Security Policy administrative tool for workgroup (non-domain) systems. In Active Directory, the default GPO precedence does not allow password policies to be set in the Default Domain Policy GPO and policies defined in the Default Domain Policy GPO will be overwritten.

For MIT Kerberos V KDCs, the password policies are stored as part of the database (and are propagated via slave KDC propagation) and are associated with principals. Creating a policy is relatively straightforward using the add_policy command in kadmin/kadmin.local.

For older versions (1.6.3 in this example) of MIT Kerb, no automatic unlock feature was implemented:

kadmin.local:  add_policy
usage; add_policy [options] policy
        options are:
                [-maxlife time] [-minlife time] [-minlength length]
                [-minclasses number] [-history number]

For newer versions (1.10 in this example), the feature is implemented as part of the password policy:

kadmin.local:  addpol
usage; add_policy [options] policy
        options are:
                [-maxlife time] [-minlife time] [-minlength length]
                [-minclasses number] [-history number]
                [-maxfailure number] [-failurecountinterval time]
                [-lockoutduration time]

By default, no policy is applied when a principal is created:

kadmin.local:  ank burrm
WARNING: no policy specified for burrm@MIKESBLOG.LAN; defaulting to no policy
Enter password for principal "burrm@MIKESBLOG.LAN":
Re-enter password for principal "burrm@MIKESBLOG.LAN":
Principal "burrm@MIKESBLOG.LAN" created.

If you create a policy named "default" then it will be applied to all new principals (but not applied to existing ones):

kadmin.local:  addpol -minlength 12 -minclasses 2 default
kadmin.local:  add_principal burrm2 
NOTICE: no policy specified for burrm2@MIKESBLOG.LAN; assigning "default"
Enter password for principal "burrm2@MIKESBLOG.LAN":
Re-enter password for principal "burrm2@MIKESBLOG.LAN":
Principal "burrm2@MIKESBLOG.LAN" created.
kadmin.local:  getprinc burrm2
Principal: burrm2@MIKESBLOG.LAN
Expiration date: [never]
Last password change: Thu Mar 08 14:24:03 MST 2012
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Thu Mar 08 14:24:03 MST 2012 (burrm/admin@MIKESBLOG.LAN)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 1, Triple DES cbc mode with HMAC/sha1, no salt
Key: vno 1, DES cbc mode with CRC-32, no salt
Attributes:
Policy: default

The principal created before is not affected by the creation of the new default policy:

kadmin.local:  getprinc burrm
Principal: burrm@MIKESBLOG.LAN
Expiration date: [never]
Last password change: Thu Mar 08 14:09:25 MST 2012
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Thu Mar 08 14:09:25 MST 2012 (burrm/admin@MIKESBLOG.LAN)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 1, Triple DES cbc mode with HMAC/sha1, no salt
Key: vno 1, DES cbc mode with CRC-32, no salt
Attributes:
Policy: [none]

A dictionary file can be used globally and is specified by the dict_file option in the [realms] section of the kdc.conf file. It should be noted that if a principal is created with a policy, the password has to conform to the policy. Also, if a principal  To set a weaker password (sometimes used by tier 1 staff when a user forgets his/her password), the policy must be cleared with the modprinc -clearpolicy command before a weak temporary password can be set. A way for a tier 1 help desk technician to reset a password could be the following:

kadmin:  modprinc -clearpolicy burrm2
Principal "burrm2@MIKESBLOG.LAN" modified.
kadmin:  cpw -pw "temp123" burrm2
Password for "burrm2@MIKESBLOG.LAN" changed.
kadmin:  modprinc -policy default -pwexpire yesterday burrm2
Principal "burrm2@MIKESBLOG.LAN" modified.

This clears the policy temporarily, changes the password to something easily communicated, then reinstates the policy and forces the user to change password at next use. There is a high probability that this should just be incorporated into a helpdesk script because it may be difficult for tier 1 staff to remember and properly execute all of the steps. Additionally the steps may be undesirable because the tier 1 staff would need somewhat elevated permissions with kadmin to make the changes. If policies are assigned, users/admins will likely encounter the following errors when changing passwords:

Not long enough:

kadmin.local:  cpw burrm2
Enter password for principal "burrm2":
Re-enter password for principal "burrm2":
change_password: Password is too short while changing password for "burrm2@MIKESBLOG.LAN".

Not complex enough:

kadmin.local:  cpw burrm2
Enter password for principal "burrm2":
Re-enter password for principal "burrm2":
change_password: Password does not contain enough character classes while changing password for "burrm2@MIKESBLOG.LAN".

Previously used:

kadmin.local:  cpw burrm2
Enter password for principal "burrm2":
Re-enter password for principal "burrm2":
change_password: Cannot reuse password while changing password for "burrm2@MIKESBLOG.LAN".

Password policies are pretty simple to set up and serve as a valuable tool to help reduce the risk of compromise due to brute force attacks. Use them wisely, because users will tend to do insecure things with overly complex passwords and passwords that need to be changed frequently including writing them on post-it notes, whiteboards, and pads of paper. Some people may tape them to a monitor or place them in a drawer or under a keyboard. In many ways, an overly harsh password policy is worse than not having a password policy and actually serves to hurt organizational information security instead of helping it.

See Also,
Deploying a Kerberos KDC in Ubuntu 11.10 or Fedora 15

Cisco Frame Relay Switching Lab: Fully Meshed PVCs

This post is somewhat a continuation of my previous post that provided an introduction to frame relay switching and acts as an alternate post to my lab that shows switching for partially meshed PVCs. This post will aim to show the logic around configuring fully meshed permanent virtual circuits (PVCs) for 4 customer sites using a single provider Frame Relay switch. This Cisco frame relay lab is configured in Dynamips/GNS3 with 5 Cisco c3725 routers utilizing WIC-2T cards.

The Provider Frame Relay Switch Configuration

If you have been following along with the two-site and three-site example, then this should be fairly straightforward as we are only increasing the number of PVCs. As the number of PVCs increase, the focus shifts to ensuring that DLCIs are being translated accurately as frames pass through the provider network (in this case FRS). If a company truly wants a full mesh, then it will require n(n-1)/2 PVCs where n is the number of sites, and this quickly becomes very expensive for many sites. Most companies settle on a partial mesh topology or a combination of frame relay with other WAN technologies.  

! Note: Some output omitted
!
frame-relay switching
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 102 interface Serial0/1 201
 frame-relay route 103 interface Serial0/2 301
 frame-relay route 104 interface Serial0/3 401
!
interface Serial0/1
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 201 interface Serial0/0 102
 frame-relay route 203 interface Serial0/2 302
 frame-relay route 204 interface Serial0/3 402
!
interface Serial0/2
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 301 interface Serial0/0 103
 frame-relay route 302 interface Serial0/1 203
 frame-relay route 304 interface Serial0/3 403
!
interface Serial0/3
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 401 interface Serial0/0 104
 frame-relay route 402 interface Serial0/1 204
 frame-relay route 403 interface Serial0/2 304
!

The Customer Perspective

The customer has an ever-growing number of choices of how to configure IP connectivity as the number of sites increase. As I presented in the three site example, the customer needs to choose between point-to-pojnt and multipoint subinterfaces based on how they are planning and using IP addresses within the network. Point-to-point subinterfaces require a unique IP network to be created for the devices at each end of a PVC while multipoint subinterfaces allow the same IP network to be used for all of the connected devices. The table below shows how quickly the number of required IP networks increases as the number of sites increases:

Number of Sites (Full Mesh)	IP Networks (PtP Subinterfaces) = n(n-1)/2	IP Networks (MP Subinterfaces)
2	1	1
3	3	1
4	6	1
5	10	1
6	15	1
7	21	1
8	28	1
9	36	1
10	45	1
11	55	1

Most companies and network engineers still think in terms of classful networks and use class C (/24) networks for simplicity. Given that a Class C (/24) network can only be subnetted to 64 point to point networks (/30 network mask), multiple class C networks would need to be planned for a mesh of 12 or more sites. As the number of sites increase, the value of multipoint subinterfaces becomes far more apparent. Unless routes are being summarized, it is more desirable from a routing protocol standpoint to have multipoint subinterfaces defined because it limits the memory and processing capability that is required by routing updates (or limits the effort required to maintain static routes).  For this reason, I will only continue with a discussion of multipoint subinterfaces. See my 3 site example for a discussion of point-to-point subinterfaces.

Configuration from the customer perspective is fairly straightforward. For this example, all of the customer edge devices participating in the frame relay network will have IP addresses in the 192.168.1.0/28 range.

CE1 Serial 0/0.1	192.168.1.1
CE2 Serial 0/0.1	192.168.1.2
CE3 Serial 0/0.1	192.168.1.3
CE4 Serial 0/0.1	192.168.1.4

Since we are using multipoint subinterfaces, frame-relay map commands need to be used to map IP addresses to DLCIs (since the LMI messages exchanged between the edge devices and the frame-relay switch have no knowledge of anything at or above layer 3). The relevant configuration sections for each customer edge router are shown below:

CE1

!
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 multipoint
 ip address 192.168.1.1 255.255.255.240
 frame-relay map ip 192.168.1.2 102
 frame-relay map ip 192.168.1.3 103
 frame-relay map ip 192.168.1.4 104
 frame-relay interface-dlci 102
 frame-relay interface-dlci 103
 frame-relay interface-dlci 104
!

CE2

!
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 multipoint
 ip address 192.168.1.2 255.255.255.240
 frame-relay map ip 192.168.1.1 201
 frame-relay map ip 192.168.1.3 203
 frame-relay map ip 192.168.1.4 204
 frame-relay interface-dlci 201
 frame-relay interface-dlci 203
 frame-relay interface-dlci 204
!

CE3

!
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 multipoint
 ip address 192.168.1.3 255.255.255.240
 frame-relay map ip 192.168.1.1 301
 frame-relay map ip 192.168.1.2 302
 frame-relay map ip 192.168.1.4 304
 frame-relay interface-dlci 301
 frame-relay interface-dlci 302
 frame-relay interface-dlci 304
!

CE4

!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 clock rate 2000000
!
interface Serial0/0.1 multipoint
 ip address 192.168.1.4 255.255.255.240
 snmp trap link-status
 frame-relay map ip 192.168.1.1 401
 frame-relay map ip 192.168.1.2 402
 frame-relay map ip 192.168.1.3 403
 frame-relay interface-dlci 401
 frame-relay interface-dlci 402
 frame-relay interface-dlci 403
!

Verification of the above topology relies on using ICMP echo (ping). See the note below for configuring broadcast capability.

Broadcasts on Frame-Relay Links

The map commands in the configuration snippets above do not allow broadcast reliant protocols such as CDP to run. To allow these types of protocols to run properly and send broadcasts on the frame-relay PVCs, the frame-relay map command needs to be specified with the map command. Here are examples of modifying the above configurations to allow broadcasts between CE1 and CE2,

CE1(config-subif)#no frame-relay map ip 192.168.1.2 102
CE1(config-subif)#frame-relay map ip 192.168.1.2 102 broadcast

CE2(config-subif)#no frame-relay map ip 192.168.1.1 201
CE2(config-subif)#frame-relay map ip 192.168.1.1 201 broadcast

Now, broadcasts can be sent between CE1 and CE2, and other protocols that use broadcasts at layer 2 can run over the multipoint subinterface. Note that Cisco Discovery Protocol (CDP) is not supported over multipoint subinterfaces.

See Also,
Introduction to Frame Relay Switching
Cisco Frame Relay Switching Lab: Partially Meshed PVCs

Cisco Frame Relay Switching Lab: Partially Meshed PVCs

This post is somewhat a continuation of my previous post that provided an introduction to frame relay switching and acts as an alternate post to my lab that shows switching for fully meshed PVCs. This post will aim to show the logic around configuring partially meshed permanent virtual circuits (PVCs) for 3 customer sites using a single provider Frame Relay switch. This frame relay lab is configured in Dynamips/GNS3 with 4 Cisco c3725 routers utilizing WIC-2T cards.

Configuration of the Frame-Relay Switch

The configuration for this scenario is fairly straightforward on the frame relay switch. The main detail is to remember that DLCIs are only significant on the link between a frame relay endpoint and the first hop in the frame relay cloud. To illustrate this, CE2 and CE3 both reference their PVCs to CE1 with a DLCI value of 100. Here are the important pieces of the configuration for the frame relay switch (in the provider's cloud):

!    Frame Relay Switch Configuration
!    Note that lines not directly related to frame relay switching
!    or interface configuration have been removed.
frame-relay switching
!
interface Serial0/0
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 102 interface Serial0/1 100
 frame-relay route 103 interface Serial0/2 100
!
interface Serial0/1
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 100 interface Serial0/0 102
!
interface Serial0/2
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 100 interface Serial0/0 103
Remember that the frame-relay route command specifies how the DLCI is rewritten for frames received on that interface. This command specifies what the DLCI should be rewritten for the next frame relay hop and also specifies the outgoing interface (or Tunnel pseudo-interface or frame relay multilink bundle).

Customer Edge Device Configuration

The customer's networking team has a couple of decisions to make around how to configure CE1. The main decision is whether to use point-to-point or multipoint subinterfaces. If the team wants to maintain full connectivity between sites and use point-to-point subinterfaces, they will need a unique layer 3 network for each pair of subinterfaces involved in sending data over a PVC. Since this is a connection between 2 IP systems, it is possible to use IPv4 networks with the 255.255.255.252 (/30) subnet mask. If the team wants all of the connections over the frame relay cloud to use a single layer 3 network, then a multipoint subinterface is required for the CE1 router. For CE2 and CE3, there is only need for point-to-point subinterfaces because each router connects with only 1 router using PVCs across the connected serial interface. I will show both configurations.

Point-to-Point Subinterfaces

For the first example, I will show the scenario where the customer's team decided to use point-to-point subinterfaces. I will use the 10.0.0.0/30 network for the subinterfaces mapped to the PVC between CE1 and CE2 and I will use the 10.0.0.4/30 network for the subinterfaces mapped to the PVC between CE1 and CE3. In addition to creating and configuring the subinterfaces, I will need to add a static route on CE2 and CE3 to ensure full connectivity.

Here is the relevant configuration for CE1:
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 point-to-point
 ip address 10.0.0.1 255.255.255.252
frame-relay interface-dlci 102
!
interface Serial0/0.2 point-to-point
 ip address 10.0.0.5 255.255.255.252
frame-relay interface-dlci 103
!

CE2 is also fairly straightforward. The main item to note is the static route to ensure connectivity to 10.0.0.4/30:

interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 point-to-point
 ip address 10.0.0.2 255.255.255.252
 frame-relay interface-dlci 100
!
ip route 10.0.0.4 255.255.255.252 10.0.0.1

CE3 follows similar lines to CE2:

interface Serial0/0
 no ip address
 encapsulation frame-relay
 clock rate 2000000
!
interface Serial0/0.1 point-to-point
 ip address 10.0.0.6 255.255.255.252
 snmp trap link-status
 frame-relay interface-dlci 100
!
ip route 10.0.0.0 255.255.255.252 10.0.0.5

Verification of connectivity between CE1 and it's connected routers is straightforward using the techniques presented in my previous post. The main items to check in this configuration are end-to-end connectivity between CE2 and CE3 and verify the routing tables on CE2 and CE3:

CE2#ping 10.0.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/21/60 ms
CE2#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/30 is subnetted, 2 subnets
C       10.0.0.0 is directly connected, Serial0/0.1
S       10.0.0.4 [1/0] via 10.0.0.1

From CE3:

CE3#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/25/76 ms
CE3#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/30 is subnetted, 2 subnets
S       10.0.0.0 [1/0] via 10.0.0.5
C       10.0.0.4 is directly connected, Serial0/0.1

Multipoint Subinterfaces

With multipoint subinterfaces, we can keep all of the interfaces connected to the frame-relay topology assigned to the same layer IP network. For this example, I will use 10.0.0.0/24 (specifically .1 for CE1, .2 for CE2, and .3 for CE3). In this case we need to map next-hop IP addresses to DLCIs on CE1 so that we can achieve full connectivity. Unlike the point-to-point subinterface example above, static routes are not needed. BElow are the relevant configuration sections for CE1, CE2 and CE3.

On CE1:
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 multipoint
 ip address 10.0.0.1 255.255.255.0
 frame-relay map ip 10.0.0.3 103
 frame-relay map ip 10.0.0.2 102
 frame-relay interface-dlci 102
 frame-relay interface-dlci 103
!

On CE2:
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 point-to-point
  ip address 10.0.0.2 255.255.255.0
  frame-relay interface-dlci 100

On CE3:
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.1 point-to-point
  ip address 10.0.0.3 255.255.255.0
  frame-relay interface-dlci 100
!

Verification should be performed to ensure that CE2 and CE3 can successfully communicate:

CE2#ping 10.0.0.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/27/92 ms

CE3#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/25/104 ms

It is also possible to verify the next hop IP addresses mapped to DLCIs using the show frame-relay map command and also determine if they are statically defined or discovered dynamically using inverse ARP:

CE1#show frame-relay map
Serial0/0.1 (up): ip 10.0.0.2 dlci 102(0x66,0x1860), static,
              CISCO, status defined, active
Serial0/0.1 (up): ip 10.0.0.3 dlci 103(0x67,0x1870), static,
              CISCO, status defined, active


See Also:
The Road to the CCIE
Introduction to Frame Relay Switching

Introduction to Frame Relay Switching

Introduction

Frame Relay is an important exam topic in the CCNA (mostly ICND2 exam), CCNP, and CCIE certifications for the routing and switching track. Frame Relay is not as popular for WAN connectivity strategy because of the rise in newer technologies like Multiprotocol Label Switching (MPLS) and other VPN strategies that create a private communication link over a public network (like IPSec tunneling). Many organizations continue to use traditional point to point dedicated lines (T1, T3, etc).

Frame Relay switching is a possible way for a service provider to switch frames between a customer's site. Frame Relay is considered a Layer 2 technology (compare with other layer 2 technologies like Ethernet and Asynchronous Transfer Mode [ATM]). Frame Relay operates differently from other protocols in that the destination is addressed from the sender's perspective instead of the recipients perspective. To clarify this, consider how Ethernet switching works:  A station sends a frame to the destination MAC address of another system. The MAC address uniquely identifies the system within the layer 2 switching topology (which can traverse multiple switches).

For Frame Relay, the switch on the service provider's network notifies a router of the permanent virtual circuits (PVCs) that are available to be switched and the data link connection identifiers (DLCIs) that need to be used to transmit on each PVC. This notification happens using the Local Management Interface (LMI) messages that are passed from the frame relay switch to the routers. In this case, the DLCIs only need to be unique between the router and the frame-relay switch. In the example below, R1 and R2 could both use DLCI 100 and have a functional data path between them because the DLCI only needs to be unique between the router and the first hop switch on the service provider portion of the network.

Below I use different DLCIs for clarity (and in practice some customers use the globally unique DLCI addressing practice instead of relying on the local significance property of DLCIs). When I write DLCI 101, this really means "the PVC that connects R2 and R1 that is addressed from R2's perspective."

In this Cisco frame relay lab, I show the easiest topology that can be created with three routers (2 acting as customer edge devices and one acting as a Frame Relay switch). I built the lab in GNS3/Dynamips using 3 Cisco c3725 routers with WIC-2T cards. A single PVC is created between routers R1 and R2 and basic IP connectivity is established. Understanding this is the first step to understanding how to configure a frame relay cloud.

Configuration of the Frame Relay Switch

Configuration of the frame relay switch is fairly straightforward. In global configuration mode, the frame-relay switching command is needed.

FRS(config)#frame-relay switching

For the serial interfaces, a couple of pieces need to be done. The encapsulation type needs to be set to frame-relay and the frame-relay interface types need to be set to DCE. The frame-relay route command needs to be used to identify how to switch DLCIs on the interfaces that they are received on. The format of the command is:

frame-relay route <received_dlci> interface <outgoing_interface> <outgoing_dlci>

Options also exist to route frames across Frame-Relay Multilink and tunnel interfaces. The configurations for Serial 0/0 (connecting to R1) and Serial 0/1 (Connecting to R2) are shown below:

interface Serial0/0
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 100 interface Serial0/1 101

interface Serial0/1
 no ip address
 encapsulation frame-relay
 clock rate 2000000
 frame-relay intf-type dce
 frame-relay route 101 interface Serial0/0 100

From the configurations, any frames received on Serial 0/0 with a DLCI of 100 are sent out Serial 0/1 with a new DLCI of 101. Frames received on Serial 0/1 with DLCI 101 are sent out Serial 0/0 with DLCI 100. The main command that is useful for verification on the frame-relay switch is the show frame-relay route command:

FRS#show frame-relay route
Input Intf      Input Dlci      Output Intf     Output Dlci     Status
Serial0/0       100             Serial0/1       101             inactive
Serial0/1       101             Serial0/0       100             inactive

Since our customer edge routers are not configured yet, the switching status shows as inactive. It should be noted that no IP configuration is necessary on the frame-relay switch (since Frame Relay is a layer 2 protocol).

Configuration of the Customer Edge Devices

Configuration of the customer edge devices is fairly straightforward. Since we only have a single PVC between R1 and R2, a point-to-point subinterface on each router (for the respective output DLCI) is all that is required. IP addresses need to be configured to achieve layer 3 connectivity. The relevant configurations for R1 and R2 are shown below:

On Customer Router R1:
interface Serial0/0
 no ip address
 encapsulation frame-relay
!
interface Serial0/0.100 point-to-point
 ip address 10.0.0.100 255.255.255.0
 frame-relay interface-dlci 100

On Customer Router R2:
interface Serial0/1
 no ip address
 encapsulation frame-relay
!
interface Serial0/1.101 point-to-point
 ip address 10.0.0.101 255.255.255.0
 frame-relay interface-dlci 101

On subinterfaces, the frame-relay interface-dlci command is necessary because the subinterface configuration is not known by the frame relay switch and the subinterface configuration is not part of the LMI messages exchanged between the service provider and customer edge devices. The network engineers for the customer are responsible for determining higher layer configuration (such as IP connectivity) and depending on the PVC layout, it may make more sense to use point-to-multipoint subinterfaces (these will be demonstrated in the next post on Frame Relay that shows provider and customer configurations for three sites).

Verification of the Solution

Verification is fairly straightforward, for this example I will demonstrate a top-down approach (layer 3 -> layer 2 -> layer 1).

Layer 3 (IP Connectivity)

Can R1 ping R2's S0/1.101 interface?
R1#ping 10.0.0.101

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.101, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/20/44 ms

Can R2 ping R1's S0/0.100 interface?
R2#ping 10.0.0.100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.100, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/20/52 ms

We should also only see a single hop in a traceroute (since there is no IP configuration on the frame-relay switch and it is not a routed connection from the provider's perspective)...

R1#traceroute 10.0.0.101

Type escape sequence to abort.
Tracing the route to 10.0.0.101

  1 10.0.0.101 44 msec *  44 msec

R2#traceroute 10.0.0.100

Type escape sequence to abort.
Tracing the route to 10.0.0.100

  1 10.0.0.100 40 msec *  44 msec

Layer 2 Verification (Frame Relay)

From the frame relay switch, the most useful command is show frame-relay route. This shows which switching paths are actually active.

FRS#show frame-relay route
Input Intf      Input Dlci      Output Intf     Output Dlci     Status
Serial0/0       100             Serial0/1       101             active
Serial0/1       101             Serial0/0       100             active

Running a show frame-relay pvc command from the switch also shows some useful information. :

FRS#show frame-relay pvc

PVC Statistics for interface Serial0/0 (Frame Relay DCE)

              Active     Inactive      Deleted       Static
  Local          0            0            0            0
  Switched       1            0            0            0
  Unused         0            0            0            0

DLCI = 100, DLCI USAGE = SWITCHED, PVC STATUS = ACTIVE, INTERFACE = Serial0/0

  input pkts 45            output pkts 43           in bytes 10976
  out bytes 10328          dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 0         out bcast bytes 0
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 0 bits/sec, 0 packets/sec
  switched pkts 45
  Detailed packet drop counters:
  no out intf 0            out intf down 0          no out PVC 0
  in PVC down 0            out PVC down 0           pkt too big 0
  shaping Q full 0         pkt above DE 0           policing drop 0
  pvc create time 01:29:17, last time pvc status changed 00:28:25

PVC Statistics for interface Serial0/1 (Frame Relay DCE)

              Active     Inactive      Deleted       Static
  Local          0            0            0            0
  Switched       1            0            0            0
  Unused         0            0            0            0

DLCI = 101, DLCI USAGE = SWITCHED, PVC STATUS = ACTIVE, INTERFACE = Serial0/1

  input pkts 43            output pkts 46           in bytes 10328
  out bytes 11300          dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 0         out bcast bytes 0
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 0 bits/sec, 0 packets/sec
  switched pkts 43
  Detailed packet drop counters:
  no out intf 0            out intf down 0          no out PVC 0
  in PVC down 0            out PVC down 0           pkt too big 0
  shaping Q full 0         pkt above DE 0           policing drop 0
  pvc create time 01:28:13, last time pvc status changed 00:29:01

Items that can quickly be seen from this command are misconfigured interface types, excessive dropped frames, attempts to switch on nonexistent/down PVCs, and PVCs marked inactive or deleted (that should be active or should exist).

On the customer side, the show-frame-relay route command is not applicable and the show frame-relay pvc command shows PVC information gathered from the LMI messages passed between the provider switch and the customer edge device. For brevity, I will only show the output from R1,

Nothing here because we are not switching on R1:
R1#show frame-relay route
Input Intf      Input Dlci      Output Intf     Output Dlci     Status

This shows that we only have 1 PVC defined on the switch for this interface (specifically the PVC with DLCI 100 that goes to R2). This also shows that we configured S0/0.100 for the communication on the PVC.:
R1#show frame-relay pvc

PVC Statistics for interface Serial0/0 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          1            0            0            0
  Switched       0            0            0            0
  Unused         0            0            0            0

DLCI = 100, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0.100

  input pkts 52            output pkts 55           in bytes 13244
  out bytes 14216          dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 40        out bcast bytes 12960
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:39:20, last time pvc status changed 00:37:30

Also worth noting is that Cisco Discovery Protocol (CDP) does not show the provider switch, it only shows the router on the other end of the PVC:

R1#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
R2               Ser 0/0.1          152         R S I     3725      Ser 0/1.1


R2#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
R1               Ser 0/1.1          140         R S I     3725      Ser 0/0.1

Layer 1 Verification

Verification of layer 1 (and layer 2) is possible by examining the output from the show interface command. I will show the output from R1 for the Serial 0/0 and Serial 0/0.100 interfaces:

R1#show int s0/0
Serial0/0 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  LMI enq sent  271, LMI stat recvd 272, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/64, broadcasts sent/dropped 46/0, interface broadcasts 0
  Last input 00:00:08, output 00:00:09, output hang never
  Last clearing of "show interface" counters 00:45:28
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     331 packets input, 19121 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     336 packets output, 19736 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

R1#show int s0/0.100
Serial0/0.100 is up, line protocol is up
  Hardware is GT96K Serial
  Internet address is 10.0.0.100/24
  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY
  CRC checking enabled
  Last clearing of "show interface" counters never

Here we clearly see that the interface is in an up/up status. We also see other useful information such as statistics regarding LMI messages, LMI type, and general interface statistics.

This post mainly detailed basic switching between two customer sites. Future posts will go into more complex customer/provider scenarios and will eventually get into the QoS and routing protocol complexities associated with frame relay.

See Also:
The Road to the CCIE
Cisco Frame Relay Switching Lab: Partially Meshed PVCs
Cisco Frame Relay Switching Lab: Fully Meshed PVCs

The Road to the CCIE

The Cisco Certified Internetwork Expert (CCIE) designation is considered by many to be one of the most difficult IT certifications to achieve. The certification consists of two parts, a written multiple choice exam and a practical exam.  By all indications, the written exam is not difficult compared to the practical exam. Many book sections and anecdotes on the Internet indicate that the practical exam is almost always required to be taken twice (or more). A lot of people even say that the available IT certification training courses for the CCIE are not sufficient training to pass the exam.

Based on what I've observed so far, candidates really need to have the equivalent knowledge of a Cisco Certified Network Professional to really begin studying for the CCIE exams. Individuals with the Cisco Certified Network Associate (CCNA) certification likely do not have the required background knowledge in routing and switching to meaningfully dig into the exam certification guide (and some of the recommended references for the CCIE).

I have not taken the exam yet, but I am planning to take both parts over the next 6-12 months and this series of posts will chronicle the progress with studying. My chosen method of study consists of trying to build a strong theoretical background using Cisco books and configuration guides and building virtual topologies in Dynamips/GNS3 that demonstrate working configurations with all of the stated exam topics and the related topics that are not specifically listed with the exam objectives on the Cisco Learning site. A check on myself that I have developed sufficient knowledge is that I will be able to clearly and coherently teach others about the technologies and explain them in detail, and that is my aim with this series of posts. A caution to the reader is important here because I explain the topics in an order that is different than most of the books on the subject. I start with WAN technologies (such as Frame Relay, ATM, and MPLS), then move to Ethernet switching topics, and then provide a discussion routing. Afterwards I look at how everything works together in different security, Quality of Service (QoS), and routing scenarios (IPv4, IPv6, Multicast).

The full list of posts is below. Note that many of the posts have working GNS3 labs and/or instructions available for download.

• General Study for the Certification
• Why Should a CCIE Read Standards?
• Layer 2 Ethernet Tools
• Setting Up a CCIE Switching Lab 
• Frame Relay for the WAN
• Introduction to Frame Relay Switching
• Cisco Frame Relay Switching Lab: Partially Meshed PVCs
• Cisco Frame Relay Switching Lab: Fully Meshed PVCs
• Frame Relay Switching Lab Using Multiple Switches (Tunnels)
• Multiprotocol Label Switching (MPLS) for the WAN
• Layer 2 Virtual Private Networks (VPNs)
• Layer 3 Virtual Private Networks (VPNs)
• Routing Protocols (IPv4)
• Routing Information Protocol version 2 (RIPv2)
• Open Shortest Path First version 2  (OSPF)
• OSPF Concepts: Areas and LSA Types 
• OSPF Cost/Metric Calculation  
• Comprehensive OSPF Cost Example
• Single Area OSPF Routing Example 
• The OSPF Stub Area 
• The OSPF Totally Stubby Area
• The OSPF Not-so-Stubby Area (NSSA) 
• The OSPF Totally Not-so-Stubby Area (NSSA)
• OSPF Virtual Links 
• Enhanced Interior Gateway Routing Protocol (EIGRP)
• Intermediate System to Intermediate System (IS-IS)
• Border Gateway Protocol (BGP)
• On-Demand Routing
• Configuring Cisco On-Demand Routing (ODR)
• Routing Protocols (IPv6)
• Routing Information Protocol Next Generation (RIPng)
• Open Shortest Path First version 3 (OSPF)
• Border Gateway Protocol with Multiprotocol Extensions
• IPv6 Coexistence
• Dual Stack
• Tunneling
• Network Address Translation (NAT) strategies between IPv4 and IPv6
• Multicast Routing
• Quality of Service (QoS)
• Network Device Management 
• Scripting the IOS Platform
• Miscellaneous
• Interface Order of Operations
• Router on a Stick with GNS3
• Emulating a Managed Switch With Dynamips/GNS3
• Connecting GNS3 and Microsoft Hyper-V