T1 Troubleshooting on Cisco Routers with t1 cards
Pt to pt 1 troubleshooting:
Some point to point T1's still need both CPE equipments to have
clock set to line as the network provides the clock. The signal strength on the t1 line should be around 0 or a few db negative. A mismatch of clocking between our hardware and the central office crossover equipment could cause a signal to be high, such as -17db or so. The normal CO tester won't look at this sort of thing; they only do the loopback pattern testing. You will need to ask them about the signal strength of your line is you suspect this sort of problem.
DEBUGGING T1 issues
Before troubleshooting any aspect of a connectivity issue (e.g., ISDN, CAS, modem) you should always verify the physical integrity of the T1 line. You should always check the status of the T1 controllers and verify you are not receiving any errors. show controller T1 x will give the snapshot of the T1 physical layer status. There should not be any framing errors, Slips, or line code violations.
Q. The Telco says the line is clean, but I am still having problems. Could it be the router?
A. Yes, but its not as likely as the line itself still having problems. The basic tests (sending all 1s) that a telco will do upon getting a trouble ticket do not always expose the problem very well unless the circuit is clearly damaged or a smart jack is losing singal for whole seconds at a time and the t1 is bouncing up and down. You often will have to ask (demand) that the telco run longer tests, with test patterns that can simulate data transfer better. I have personally observed a t1 circuit that passed all basic telco pattern tests but when actual data was passed across the line, the errors would accumulate fast and the line provided only a few K of actual thruput due to the heavy errrors. The actual problem was a bad fiber to copper repeater that would intermittently misbehave. If you can take the t1 out of service, you can send a cisco ping across the line with a pattem and ask the telco to watch the line for several hours. This may be the best way to convince the telco there is an actual problem. If you can swap out the router, t1 wic, and patch cable to the smartjack while testing this is also a good idea. Being able to tell the telco that you have swapped ALL of your gear and verified that your in-house demarc extension is not at fault by hooking a test router up directly to the back of the smart jack enclosure is the best path towards showing the telco the issue is on their end.
Here is an example of a ping across a serial line that will send a data pattern to be verfied:
Target IP address: 192.168.1.2
Repeat count : 100
Datagram size :
Timeout in seconds : 1
Extended commands [n]: y
Source address or interface: 192.168.1.1
Type of service :
Set DF bit in IP header? [no]:
Validate reply data? [no]: yes
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size :
Sweep max size : 1420
Sweep interval :
Type escape sequence to abort.
Sending 138500, [36..1420]-byte ICMP Echos to 192.168.1.2, timeout is 1 seconds:
Packet sent with a source address of 192.168.1.1
Reply data will be validated
Using show commands:
Following is the sample output of show controller T1 0 and what to look at:
show controllers t1 0
T1 0 is up.
Applique type is Channelized T1
Cablelength is long gain36 0db
No alarms detected.
Version info of slot 0: HW: 4, Firmware: 16, PLD Rev: 0
Manufacture Cookie Info:
EEPROM Type 0x0001, EEPROM Version 0x01, Board ID 0x42,
Board Hardware Version 1.32, Item Number 73-2217-5,
Board Revision B16, Serial Number 09356930,
PLD/ISP Version 0.0, Manufacture Date 18-Jun-1998.
Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.
Data in current interval (8 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
The main items to look at in the above output is
* The status of the line
* Linecode and Pathcode violations
* Slip Secs
The line status will tell us if the T1 is either up, down, or administratively down. The Alarms section is very important and it will tell us what type of problem maybe present on the line. The presence of any alarms indicates a serious problem on the line.
It is recommended whenever you encounter a T1 that is in an alarm state that you verify the framing and linecoding parameters are configured correctly. Please refer to the show controller t1 commands in the Command Reference to find out all the possible values for the alarm state.
A common message you will see in the alarm field is "receiver has loss of frame." Some routers will also report a 'loss of frame' even when it should be a "loss of signal." So, make sure whenever you receive these errors that the T1 signal is present and the framing is correct.
Another message you might receive is "receiver is getting AIS." This means the receiver is getting an alarm indication signal (blue alarm). This is a framed or unframed all-ones signal in both SF and ESF formats transmitted to maintain transmission continuity. This is typically seen when the far-end CSU has lost its terminal side equipment. The "receiver has remote alarm" indicates the presence of a yellow alarm. This means the downstream CSU is in a loss-of-frame or loss-of-signal state. Therefore, the remote CSU has a red alarm.
The "transmitter is sending remote alarm" indicates that the local CSU has detected either a loss-of-frame or loss-of-signal condition. This indicates that the local controller has a red alarm. This message will be accompanied by a "receiver has loss of frame." Always verify framing and T1 signal when troubleshooting this problem.
If any of the above highlighted fields doesn't contain zeros, than here are some of the possibilities what might causing the physical problem. Following is the brief explanation of these fields.
Line Code Violations
Indicates the occurrence of either a Bipolar Violation (BPV) or Excessive Zeros (EXZ) error event.
A BPV error event for an AMI-coded signal is the occurrence of a pulse of the same polarity as the previous pulse.
A BPV error event for a B8ZS is the occurrence of a pulse of the same polarity as the previous pulse without being a part of the zero substitution code. An EXZ is the violation of the pulse density requirement.
Path Code Violations
Indicates a frame synchronization bit error in the D4 and E1-noCRC formats, or a CRC error in the ESF and E1-CRC formats.
Line Errored Seconds (LES)
A Line Errored Second, according to T1M1.3, is a second in which one or more Line Code Violation error events were detected.
In the T1M1.3 specification, near end Line Code Violations and far end Line Errored Seconds are counted. For consistency, we count Line Errored Seconds at both ends.
A Controlled Slip Second is a one-second interval containing one or more controlled slips.
Errored Seconds (ES)
For ESF and E1-CRC links an Errored Second is a second with one or more Path Code Violations OR one or more Out of Frame defects OR one or more Controlled Slip events OR a detected AIS defect.
For D4 and E1-noCRC links, the presence of Bipolar Violations also triggers an Errored Second.
This is not incremented during an Unavailable Second.
Bursty Errored Seconds (BES)
A Bursty Errored Second (also known as Errored Second type B) is a second with fewer than 320 and more than 1 Path Coding Violation error events, no Severely Errored Frame defects and no detected incoming AIS defects. Controlled slips are not included in this parameter.
This is not incremented during an Unavailable Second.
Severely Errored Seconds (SES)
A Severely Errored Second for ESF signals is a second with 320 or more Path Code Violation Error Events, one or more Out of Frame defects, or a detected AIS defect.
For E1-CRC signals, a Severely Errored Second is a second with 832 or more Path Code Violation error events or one or more Out of Frame defects.
For E1-noCRC signals, a Severely Errored Second is a 2048 LCVs or more.
For D4 signals, a Severely Errored Second is a count of one-second intervals with Framing Error events, or an OOF defect, or 1544 LCVs or more.
Controlled slips are not included in this parameter. This is not incremented during an Unavailable Second.
Severely Errored Framing Second (SEFS)
An Severely Errored Framing Second is a second with one or more Out of Frame defects or a detected AIS (Alarm Indication Signal) defect.
A Degraded Minute is one in which the estimated error rate exceeds 1E-6 but does not exceed 1E-3 (see G.821 ).
Degraded Minutes are determined by collecting all of the Available Seconds, removing any Severely Errored Seconds grouping the result in 60-second long groups and counting a 60-second long group (a.k.a., minute) as degraded if the cumulative errors during the seconds present in the group exceed 1E-6. Available seconds are merely those seconds which are not Unavailable, as described below.
Unavailable Seconds (UAS)
Unavailable Seconds (UAS) are calculated by counting the number of seconds that the interface is unavailable. The DS1 interface is said to be unavailable from the onset of 10 contiguous SESs, or the onset of the condition leading to a failure (see Failure States). If the condition leading to the failure was immediately preceded by one or more contiguous SESs, then the DS1 interface unavailability starts from the onset of these SESs. Once unavailable, and if no failures present, the DS1 interface becomes available at the onset of 10 contiguous seconds with no SESs. Once unavailable, and if a failure is present, the DS1 interface becomes available at the onset of 10 contiguous seconds with no SESs, if the failure clearing time is less than or equal to 10 seconds. If the failure clearing time is more than 10 seconds, the DS1 interface becomes available at the onset of 10 contiguous seconds with no SESs, or the onset period leading to the successful clearing condition, whichever occurs later. With respect to the DS1 error counts, all counters are incremented while the DS1 interface is deemed available. While the interface is deemed unavailable, the only count that is incremented is UASs.