T1 Troubleshooting on Cisco Routers with t1 cards

Pt to pt 1 troubleshooting:

Some point to point T1's still need both CPE equipments to have
clock set to line as the network provides the clock. The signal strength on the t1 line should be around 0 or a few db negative. A mismatch of clocking between our hardware and the central office crossover equipment could cause a signal to be high, such as -17db or so. The normal CO tester won't look at this sort of thing; they only do the loopback pattern testing. You will need to ask them about the signal strength of your line is you suspect this sort of problem.

DEBUGGING T1 issues

Before troubleshooting any aspect of a connectivity issue (e.g., ISDN, CAS, modem) you should always verify the physical integrity of the T1 line. You should always check the status of the T1 controllers and verify you are not receiving any errors. show controller T1 x will give the snapshot of the T1 physical layer status. There should not be any framing errors, Slips, or line code violations.

Following is the sample output of show controller T1 0 and what to look at:

show controllers t1 0

T1 0 is up.

Applique type is Channelized T1

Cablelength is long gain36 0db

No alarms detected.

Version info of slot 0: HW: 4, Firmware: 16, PLD Rev: 0

Manufacture Cookie Info:

EEPROM Type 0x0001, EEPROM Version 0x01, Board ID 0x42,

Board Hardware Version 1.32, Item Number 73-2217-5,

Board Revision B16, Serial Number 09356930,

PLD/ISP Version 0.0, Manufacture Date 18-Jun-1998.

Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.

Data in current interval (8 seconds elapsed):

0 Line Code Violations, 0 Path Code Violations

0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

The main items to look at in the above output is

* The status of the line
* Alarms
* Linecode and Pathcode violations
* Slip Secs

The line status will tell us if the T1 is either up, down, or administratively down. The Alarms section is very important and it will tell us what type of problem maybe present on the line. The presence of any alarms indicates a serious problem on the line.

It is recommended whenever you encounter a T1 that is in an alarm state that you verify the framing and linecoding parameters are configured correctly. Please refer to the show controller t1 commands in the Command Reference to find out all the possible values for the alarm state.

A common message you will see in the alarm field is "receiver has loss of frame." Some routers will also report a 'loss of frame' even when it should be a "loss of signal." So, make sure whenever you receive these errors that the T1 signal is present and the framing is correct.

Another message you might receive is "receiver is getting AIS." This means the receiver is getting an alarm indication signal (blue alarm). This is a framed or unframed all-ones signal in both SF and ESF formats transmitted to maintain transmission continuity. This is typically seen when the far-end CSU has lost its terminal side equipment. The "receiver has remote alarm" indicates the presence of a yellow alarm. This means the downstream CSU is in a loss-of-frame or loss-of-signal state. Therefore, the remote CSU has a red alarm.

The "transmitter is sending remote alarm" indicates that the local CSU has detected either a loss-of-frame or loss-of-signal condition. This indicates that the local controller has a red alarm. This message will be accompanied by a "receiver has loss of frame." Always verify framing and T1 signal when troubleshooting this problem.

If any of the above highlighted fields doesn't contain zeros, than here are some of the possibilities what might causing the physical problem. Following is the brief explanation of these fields.
Line Code Violations

Indicates the occurrence of either a Bipolar Violation (BPV) or Excessive Zeros (EXZ) error event.

A BPV error event for an AMI-coded signal is the occurrence of a pulse of the same polarity as the previous pulse.

A BPV error event for a B8ZS is the occurrence of a pulse of the same polarity as the previous pulse without being a part of the zero substitution code. An EXZ is the violation of the pulse density requirement.
Path Code Violations

Indicates a frame synchronization bit error in the D4 and E1-noCRC formats, or a CRC error in the ESF and E1-CRC formats.
Line Errored Seconds (LES)

A Line Errored Second, according to T1M1.3, is a second in which one or more Line Code Violation error events were detected.

In the T1M1.3 specification, near end Line Code Violations and far end Line Errored Seconds are counted. For consistency, we count Line Errored Seconds at both ends.
Slip Seconds

A Controlled Slip Second is a one-second interval containing one or more controlled slips.
Errored Seconds (ES)

For ESF and E1-CRC links an Errored Second is a second with one or more Path Code Violations OR one or more Out of Frame defects OR one or more Controlled Slip events OR a detected AIS defect.

For D4 and E1-noCRC links, the presence of Bipolar Violations also triggers an Errored Second.

This is not incremented during an Unavailable Second.
Bursty Errored Seconds (BES)

A Bursty Errored Second (also known as Errored Second type B) is a second with fewer than 320 and more than 1 Path Coding Violation error events, no Severely Errored Frame defects and no detected incoming AIS defects. Controlled slips are not included in this parameter.

This is not incremented during an Unavailable Second.
Severely Errored Seconds (SES)

A Severely Errored Second for ESF signals is a second with 320 or more Path Code Violation Error Events, one or more Out of Frame defects, or a detected AIS defect.

For E1-CRC signals, a Severely Errored Second is a second with 832 or more Path Code Violation error events or one or more Out of Frame defects.

For E1-noCRC signals, a Severely Errored Second is a 2048 LCVs or more.

For D4 signals, a Severely Errored Second is a count of one-second intervals with Framing Error events, or an OOF defect, or 1544 LCVs or more.

Controlled slips are not included in this parameter. This is not incremented during an Unavailable Second.
Severely Errored Framing Second (SEFS)

An Severely Errored Framing Second is a second with one or more Out of Frame defects or a detected AIS (Alarm Indication Signal) defect.
Degraded Minutes

A Degraded Minute is one in which the estimated error rate exceeds 1E-6 but does not exceed 1E-3 (see G.821 [15]).

Degraded Minutes are determined by collecting all of the Available Seconds, removing any Severely Errored Seconds grouping the result in 60-second long groups and counting a 60-second long group (a.k.a., minute) as degraded if the cumulative errors during the seconds present in the group exceed 1E-6. Available seconds are merely those seconds which are not Unavailable, as described below.
Unavailable Seconds (UAS)

Unavailable Seconds (UAS) are calculated by counting the number of seconds that the interface is unavailable. The DS1 interface is said to be unavailable from the onset of 10 contiguous SESs, or the onset of the condition leading to a failure (see Failure States). If the condition leading to the failure was immediately preceded by one or more contiguous SESs, then the DS1 interface unavailability starts from the onset of these SESs. Once unavailable, and if no failures present, the DS1 interface becomes available at the onset of 10 contiguous seconds with no SESs. Once unavailable, and if a failure is present, the DS1 interface becomes available at the onset of 10 contiguous seconds with no SESs, if the failure clearing time is less than or equal to 10 seconds. If the failure clearing time is more than 10 seconds, the DS1 interface becomes available at the onset of 10 contiguous seconds with no SESs, or the onset period leading to the successful clearing condition, whichever occurs later. With respect to the DS1 error counts, all counters are incremented while the DS1 interface is deemed available. While the interface is deemed unavailable, the only count that is incremented is UASs.

Cisco T1 PRI Troubleshooting

Contents

Introduction

Using the show isdn status Command

Using the debug q921 Command


Introduction

When troubleshooting a Primary Rate Interface (PRI), ensure that the T1
is running properly on both ends. If Layer 1 problems have been resolved,
look for problems on Layers 2 and 3. Use the show controller t1
command to verify that the configuration of the line matches that of the
remote end. Ensure that the framing, line coding, and clock source are
configured correctly.

Using the show isdn status Command

The show isdn status command displays a summary of all ISDN interfaces.
It also displays the status of Layers 1, 2, and 3. Complete the following
steps to check the status of the layers:

  1. Verify that Layer 1 is in the ACTIVE state. The status of Layer 1 should
    always be ACTIVE unless the T1 is down.

  2. If the show isdn status command output indicates that Layer
    1 is DEACTIVATED, then there is a problem with the physical connectivity
    of the T1 line. If the line is administratively down, use the no shutdowncommand to restart the interface.

  1. Ensure that Layer 2 is in the MULTIPLE_FRAME_ESTABLISHED state. This is
    the desired state for Layer 2, indicating that Layer 2 frames are being
    exchanged and Layer 2 initialization has finished.

  2. If Layer 2 is not in the MULTIPLE_FRAME_ESTABLISHED state, use the
    show controller t1
    EXEC command to diagnose the problem. For more information,
    see the T1
    Alarm Troubleshooting
    document.

    Since the show isdn status command displays a summary of the
    current status, it is possible that Layer 2 is bouncing up and down despite
    indicating a MULTIPLE_FRAME_ESTABLISHED state. Use the debug isdn q921
    command to verify that Layer 2 is stable.

    Following is an example of show isdn status output:

      maui-nas-03#show isdn status Global ISDN Switchtype = primary-5ess
      ISDN Serial0:23 interface dsl 0, interface ISDN Switchtype = primary-5essLayer 1 Status: ACTIVE Layer 2 Status: TEI = 0, Ces = 1, SAPI =
      0, State = MULTIPLE_FRAME_ESTABLISHED Layer 3 Status: 5 Active Layer
      3 Call(s) Activated dsl 0 CCBs = 5 CCB:callid=7D5, sapi=0, ces=0, B-chan=9,
      calltype=DATA CCB:callid=7D6, sapi=0, ces=0, B-chan=10, calltype=DATA CCB:callid=7DA,
      sapi=0, ces=0, B-chan=11, calltype=DATA CCB:callid=7DE, sapi=0, ces=0,
      B-chan=1, calltype=DATA CCB:callid=7DF, sapi=0, ces=0, B-chan=2, calltype=DATA
      The Free Channel Mask: 0x807FF8FC ISDN Serial1:23 interface dsl
      1, interface ISDN Switchtype = primary-5ess Layer 1 Status: ACTIVE
      Layer 2 Status: TEI = 0, Ces = 1, SAPI = 0, State = TEI_ASSIGNED
      Layer
      3 Status: 0 Active Layer 3 Call(s) Activated dsl 1 CCBs = 0 The Free Channel
      Mask: 0x807FFFFF Total Allocated ISDN CCBs = 5

    Notice that T1 0 (whose D channel is Serial 0:23) has Layer 1 as ACTIVE
    and Layer 2 as MULTIPLE_FRAME_ESTABLISHED indicating that the signaling
    channel is functioning correctly and is exchanging Layer 2 frames with
    the telco switch. The D channel (Serial1:23) for T1 1 has Layer 1 ACTIVE,
    but Layer 2 is TEI_ASSIGNED. This indicates that the PRI is not exchanging
    Layer 2 frames with the switch. Use the show controller t1 x
    command to troubleshoot. Refer to the T1
    Troubleshooting
    flowchart for more information

Using the debug q921 Command

The debug isdn q921 command displays data link layer (Layer 2) access
procedures that are occurring at the router on the D-channel.

Ensure you are configured to view debug messages by using the logging
console
or terminal monitor command.

Note: In a production environment, verify that console logging
is disabled by using the show logging command. If logging is enabled,
the access server might intermittently stop working when the console port
is overloaded with log messages. Enter the no logging console command
to disable logging.

Note: If debug isdn q921 is turned on and you do not receive
any debug outputs, place a call or reset the controller to get debug outputs.

Complete the following steps to ensure that the data link layer access
procedures are occurring at the router on the D-channel:

  1. Verify that Layer 2 is stable by looking for messages in the debug output.
    If the line is bouncing up and down, output similar to the following will
    appear:
  2. Mar 20 10:06:07.882: %ISDN-6-LAYER2DOWN: Layer 2 for Interface Se0:23, TEI 0 changed to down
    Mar 20 10:06:09.882: %LINK-3-UPDOWN: Interface Serial0:23, changed state to downMar 20 10:06:21.274: %DSX1-6-CLOCK_CHANGE: Controller 0 clock is now selected as clock source
    Mar 20 10:06:21.702: %ISDN-6-LAYER2UP: Layer 2 for Interface Se0:23, TEI 0 changed to up
    Mar 20 10:06:22.494: %CONTROLLER-5-UPDOWN: Controller T1 0, changed state to up
    Mar 20 10:06:24.494: %LINK-3-UPDOWN: Interface Serial0:23, changed state to up
  3. Verify that only service access point identifier (SAPI) messages appear
    on both the transmit (TX) and receive (RX) sides. For example:
  4. Mar 20 10:06:52.505: ISDN Se0:23: TX -> RRf sapi = 0 tei = 0 nr = 0
    Mar 20 10:06:52.505: ISDN Se0:23: RX <- RRf sapi = 0 tei = 0 NR = 0
    Mar 20 10:07:22.505: ISDN Se0:23: TX -> RRp sapi = 0 tei = 0 NR = 0
    Mar 20 10:07:22.509: ISDN Se0:23: RX <- RRp sapi = 0 tei = 0 NR = 0
    Mar 20 10:07:22.509: ISDN Se0:23: TX -> RRf sapi = 0 tei = 0 NR = 0
    Mar 20 10:07:22.509: ISDN Se0:23: RX <- RRf sapi = 0 tei = 0 NR = 0
  5. Verify that asynchronous balanced mode extended (SABME) messages do not
    appear. These messages indicates that Layer 2 is trying to reinitialize.
    The messages usually appear when poll requests (RRp) are transmitted and
    there is no response from the switch (RRf), or vice versa. Following are
    examples of SABME messages:
  6. Mar 20 10:06:21.702: ISDN Se0:23: RX <- SABMEp sapi = 0 tei = 0
    Mar 20 10:06:22.494: ISDN Se0:23: TX -> SABMEp sapi = 0 tei = 0

    If SABME messages appear, complete the following steps:

    1. Use the show running-config command to ensure that isdn switch-type
      and pri-group timeslots are configured correctly. Contact your Service
      Provider for the correct values.
    2. To change the isdn switch-type and pri-group settings, enter
      the following commands:
    3. maui-nas-03#configure terminal
      maui-nas-03(config)#isdn switch-type primary-5ess
      maui-nas-03(config)#controller t1 0
      maui-nas-03(config-controlle)#pri-group timeslots 1-24
  7. Ensure the D-channel is up using the show interfaces serial number:23
    command, where the number is the interface number.

  8. If the D-channel is not up, use the no shutdown command to
    bring it up. For example:

    maui-nas-03(config)#interface serial 0:23
    maui-nas-03(config-if)#no shutdown
  9. Ensure encapsulation is PPP. If not, use the encapsulation ppp command
    to set encapsulation. For example:
  10. maui-nas-03(config-if)#encapsulation ppp
  11. Ensure the interface is in loopback mode. Loopback should be set only for
    testing purposes. Use the no loopback command to remove loopbacks.
    For example:
  12. maui-nas-03(config-if)#no loopback
  13. Power cycle the router.