Troubleshooting Guide for Cisco Unified Customer Voice Portal ...

Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

January 2011

Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0833

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCBs public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED "AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at http://www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R) Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental. Copyright 2011 Cisco Systems, Inc. All rights reserved.

Table of Contents Preface ...........................................................................................................................................................1 Purpose .....................................................................................................................................................1 Audience ....................................................................................................................................................1 Organization ..............................................................................................................................................1 Related Documentation .............................................................................................................................2 Conventions................................................................................................................................................3 Obtaining Documentation and Submitting a Service Request...................................................................4 Documentation Feedback...........................................................................................................................4 1. Troubleshooting Overview...........................................................................................................................5 Problem Solving Process...........................................................................................................................5 Guidelines to Assist in Isolating Problems.............................................................................................5 Problem Solving Tasks...............................................................................................................................6 Troubleshooting Checklist.........................................................................................................................11 2. Symptoms and Solutions..........................................................................................................................13 Overview...................................................................................................................................................13 Call Server................................................................................................................................................13 VXML Server............................................................................................................................................38 Reporting Server......................................................................................................................................54 Audio........................................................................................................................................................60 Transfer and Connect...............................................................................................................................77 Unified CM................................................................................................................................................91 Unified ICME............................................................................................................................................91 Content Services Switch..........................................................................................................................98 General System Issues............................................................................................................................98 H.323 Troubleshooting............................................................................................................................105 SIP Troubleshooting................................................................................................................................109 SNMP Troubleshooting...........................................................................................................................117 Installation Troubleshooting....................................................................................................................118 Gateway Troubleshooting.......................................................................................................................124 Cisco Unified Presence Server..............................................................................................................128 Operations Console (OAMP)..................................................................................................................129 Cisco Security Agent (CSA)...................................................................................................................130 Support Tools..........................................................................................................................................131 Backup and Restore...............................................................................................................................131 Appendix A. Error Messages......................................................................................................................135 Infrastructure Error Messages................................................................................................................135 IVR Error Messages...............................................................................................................................141 VMS Error Messages.............................................................................................................................145 Reporting Server Error Messages..........................................................................................................149 ICM Error Messages...............................................................................................................................151 SIP Error Messages...............................................................................................................................154 Appendix B. Event and Cause Codes.........................................................................................................157 IVR Service Error Codes........................................................................................................................157 Unified ICME Error Codes......................................................................................................................159 SIP Codes..............................................................................................................................................161 Appendix C. Troubleshooting Radvision Errors...........................................................................................163 Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

i

Appendix D. Accessing and Interpreting SNMP Data.................................................................................165 Appendix E. Internal Interfaces Description................................................................................................169 Appendix F. Trace Bits Description..............................................................................................................171 Appendix G. Debugging..............................................................................................................................175 SIP IOS Debugging................................................................................................................................175 SIP IOS Troubleshooting Commands................................................................................................176 H.323 IOS Debugging............................................................................................................................176 Index ...........................................................................................................................................................179


ii

Preface Purpose This manual describes how to troubleshoot common issues for the Cisco Unified Customer Voice Portal (Unified CVP) product. The document also explains the tools and techniques that can assist in the troubleshooting process.

Audience This document is intended for Contact Center managers, Unified CVP system managers, VoIP technical experts, and IVR Service application developers.

Organization The document is divided into the following chapters. Chapter

Description

Chapter 1, Troubleshooting Overview

Describes various methods of approaching troubleshooting tasks.

Chapter 2, Symptoms and Solutions

Describes particular issues, any associated error messages, the cause of the issue, as well as the solution.

Appendix A, Error Messages

Provides information on Unified CVP error messages.

Appendix B, Event and Cause Codes

Provides information on Unified CVP error codes.


1

Preface Related Documentation

Chapter

Description

Appendix C, Troubleshooting Radvision Issues

Describes how to isolate Unified CVP errors from those of Radvision. This appendix aims at distinguishing these errors, along with an explanation of each error code, and the appropriate action to take before involving Radvision TAC.

Appendix D, Accessing and Interpreting SNMP Data

Provides information on SNMP severity levels and their descriptions.

Appendix E, Internal Interfaces Description

Summarizes the major interfaces between the Unified CVP product components and other Cisco products.

Appendix F, Trace Bits Description

Lists the common trace bits, their corresponding values and the description for each Unified CVP subsystem type. Each value is unique within each Unified CVP subsystem.

Appendix G, Debugging

Describes SIP troubleshooting for some problems that may occur, and describes some useful command line commands for H.323 IOS troubleshooting.

Related Documentation Note: Planning your Unified CVP solution is an important part of the process in setting up Unified CVP. Cisco recommends that you read the Cisco Unified Customer Voice Portal Release 7.x Solution Reference Network Design (SRND) guide before configuring your Unified CVP solution. With Unified 7.x, the Planning Guide for Cisco Unified Customer Voice Portal has been incorporated into the SRND guide. Unified CVP provides the following documentation: • Cisco Security Agent Installation/Deployment for Cisco Unified Customer Voice Portal provides installation instructions and information about Cisco Security Agent for the Unified CVP deployment.We strongly urge you to read this document in its entirety. • Cisco Unified Customer Voice Portal Release 7.x Solution Reference Network Design (SRND) provides design considerations and guidelines for deploying contact center voice response solutions based on Cisco Unified Customer Voice Portal (Unified CVP) 7.x releases. • Configuration and Administration Guide for Cisco Unified Customer Voice Portal describes how to set up, run, and administer the Cisco Unified CVP product, including associated configuration. • Element Specifications for Cisco Unified CVP VXML Server and Unified Call Studio describes the settings, element data, exit states, and configuration options for Elements. • Installation and Upgrade Guide for Cisco Unified Customer Voice Portal describes how to install Unified CVP software, perform initial configuration, and upgrade. • Operations Console Online Help for Cisco Unified Customer Voice Portal describes how to use the Operations Console to configure Unified CVP solution components. • Port Utilization Guide for Cisco Unified Customer Voice Portal describes the ports used in a Unified CVP deployment.


2

Preface Conventions

• Programming Guide for Cisco Unified CVP VXML Server and Unified Call Studio describes how to build components that run on the Cisco Unified VXML Server. • Reporting Guide for Cisco Unified Customer Voice Portal describes the Reporting Server, including how to configure and manage it, and discusses the hosted database. • Say It Smart Specifications for Cisco Unified CVP VXML Server and Unified Call Studio describes in detail the functionality and configuration options for all Say It Smart plugins included with the software. • User Guide for Cisco Unified CVP VXML Server and Unified Call Studio describes the functionality of Call Studio including creating projects, using the Call Studio environment, and deploying applications to the Unified CVP VXML Server. For additional information about Unified ICME, see the Cisco web site Cisco web site (http:/ /www.cisco.com/en/US/products/sw/custcosw/ps1001/tsd_products_support_series_home.html) listing Unified ICME documentation.

Conventions This manual uses the following conventions: Convention

Description

boldface font

Boldface font is used to indicate commands, such as user entries, keys, buttons, and folder and submenu names. For example: • Choose Edit > Find. • Click Finish.

italic font

Italic font is used to indicate the following: • To introduce a new term. Example: A skill group is a collection of agents who share similar skills. • For emphasis. Example: Do not use the numerical naming convention. • A syntax value that the user must replace. Example: IF (condition, true-value, false-value) • A book title. Example: See the Cisco CRS Installation Guide.

window font

Window font, such as Courier, is used for the following: • Text as it appears in code or that the window displays. Example: Cisco Systems,Inc.


3

Preface Obtaining Documentation and Submitting a Service Request

Convention

Description

< >

Angle brackets are used to indicate the following: • For arguments where the context does not allow italic, such as ASCII output. • A character string that the user enters but that does not appear on the window such as a password.

Obtaining Documentation and Submitting a Service Request For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.

Documentation Feedback You can provide comments about this document by sending email to the following address: mailto:[email protected] We appreciate your comments.


4

Chapter 1 Troubleshooting Overview When troubleshooting a telephony or IP network environment, it is important to define the specific symptoms, identify all potential problems that could be causing the symptoms, and then systematically eliminate each potential problem from most likely to least likely until the symptoms disappear. This chapter contains the following topics: • Problem Solving Process, page 5 • Problem Solving Tasks, page 6 • Troubleshooting Checklist, page 11

Problem Solving Process The following steps provide some guidelines to assist in isolating a particular issue.

Guidelines to Assist in Isolating Problems Step 1

Analyze the problem and create a clear problem statement. Define symptoms and potential causes.

Step 2

Gather the facts that you need to help isolate possible causes.

Step 3

Consider possible causes based on the facts that you gathered. • Was anything recently added, removed, or modified? • Is it a reproducible event? • Does it occur at a particular time of day, or day of week?


5

Chapter 1: Troubleshooting Overview Problem Solving Tasks

• Have there been any changes made to the domain, network, or security policies? Step 4

Create an action plan. Begin with the most likely problem and devise a plan in which you manipulate only one variable at a time.

Step 5

Implement the action plan, performing each step while testing to see whether the symptom disappears.

Step 6

Analyze the results to determine whether the problem has been resolved. If it has, the process is complete.

Step 7

If the problem has not been resolved, create an action plan based on the next most probable cause on your list, or contact the Cisco Technical Assistance Center (TAC), or your Cisco Partner.

Note: Only change one variable at a time. If that does not resolve the issue, undo that change and move on to the next step of your plan.

Problem Solving Tasks In addition to the process discussed above, there are other areas to consider if a problem arises within the Unified CVP environment. The information below provides some additional information to assist in isolating or resolving a particular issue. • Verify that the software versions comply with the Hardware and System Software Specification for Unified Customer Voice Portal document. A version incompatibility might not necessarily be the cause of the issue, but can be investigated and considered as a possibility. This document is available at: http://www.Cisco.com/en/US/products/sw/custcosw/ps1006/ prod_technical_reference_list.html • Review the Unified CVP logs for possible error or warning events. Use the Support Tools utility since it has the ability to retrieve and process trace logs from most components, and to set or reset trace levels on those components. To access the Support Tools, open the Operations Console, and click Support Tools from the Tools menu. The Configuration and Administration Guide for Cisco Unified Customer Voice Portal and Cisco Support Tools User Guide for Cisco Unified Software provide additional information about Support Tools. Note: Users can execute some UNIX equivalent commands, such as more, grep, diff, tail, tee on Unified CVP servers during troubleshooting. These commands are part of the Support Tools installation. The PATH environment variable allows these commands to be accessible via command line. Unified CVP captures two different types of logs: events and errors. When a Unified CVP component generates an event, it is logged in the Unified CVP log. When a log statement is written to the log file and the level is at ERROR or above, it is also written to error log file.


6


Note: This file is created on demand. If there are no ERROR level (or above) logs, you do not see this file. Below are the locations of the log files for Unified CVP. In the listings below, CVP_HOME is the location that was selected for the installation of your Unified CVP software, such as C:\Cisco\CVP. – CVP_HOME\logs\ Call Server and/or Reporting Server logs. – CVP_HOME\logs\OAMP\ Operations Console logs. – CVP_HOME\logs\VXML\ VXML Server logs. – CVP_HOME\logs\SNMP\ SNMP Agent logs. – CVP_HOME\logs\ORM\ Unified CVP Resource Manager logs. Note: The Operations Console Resource Manager (ORM) is a service that is installed during the installation of the Operations Console. When installing the Operations Console, the two Windows services that are installed are Unified CVP OpsConsoleServer and Unified CVP Resource Manager. The Resource Manager resides on each Unified CVP server and it allows the Operations Console to communicate with that server and perform functions on that server, such as starting, stopping, or monitoring. Virtually all of the Unified CVP logs start a new log file at 12:00AM every night, with the date as part of the filename. The VXML Server logs do not necessarily start a new file at 12:00AM. The format of the date is yyyy-mm-dd. These logs also start a new log file when they reach the predefined size limit of 10 MB and have a number as part of the filename extension. The number indicates which log it was for that day. When the log directory size reaches the maximum number of megabytes (20,000MB) for disk storage for log files, the old files are purged as necessary. Both the Log File Size and Log Directory Size are configurable in the device Infrastructure tab of the Operations Console. • Check that the network is functioning properly. For the initial connectivity tests, the following command line tools may be helpful: – ping - tests whether a host is reachable across an IP network. The ping command offers several parameters or flags that can be set to produce various outputs. To view all of the available parameters, type ping /? at the command prompt. C:\> ping 10.86.132.150


7


Pinging 10.86.132.150 with 32 bytes of data: Reply from 10.86.132.150: bytes=32 time=5ms TTL=122 Reply from 10.86.132.150: bytes=32 time=4ms TTL=122 Reply from 10.86.132.150: bytes=32 time=4ms TTL=122 Reply from 10.86.132.150: bytes=32 time=4ms TTL=122 Ping statistics for 10.86.132.150: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milliseconds: Minimum = 4ms, Maximum = 5ms, Average = 4ms C:\> – traceroute - determines the packet route that is taken to the target host across an IP network. The command in a Windows environment is: tracert. The tracert command offers several parameters or flags that can be set to produce various outputs. To view all of the available parameters, type tracert /? at the command prompt. C:\> tracert 10.1.1.6 Tracing route to 10.1.1.6 over a maximum of 30 hops --------------------------------------------------1 2 ms 3 ms 2 ms 10.1.2.1 2 25 ms 83 ms 88 ms 192.168.11.1 3 25 ms 79 ms 93 ms 10.1.1.6 Trace complete. – netstat - lists the active incoming and outgoing network connections on a system. The netstat command offers several parameters or flags that can be set to produce various outputs. To view all of the available parameters, type netstat /? at the command prompt. C:\>netstat -a -n Active Connections Proto Local Address Foreign Address State TCP 0.0.0.0:42 0.0.0.0:0 LISTENING TCP 0.0.0.0:88 0.0.0.0:0 LISTENING TCP 0.0.0.0:135 0.0.0.0:0 LISTENING TCP 0.0.0.0:389 0.0.0.0:0 LISTENING


8


TCP 0.0.0.0:445 0.0.0.0:0 LISTENING TCP 0.0.0.0:593 0.0.0.0:0 LISTENING TCP 0.0.0.0:1038 0.0.0.0:0 LISTENING TCP 0.0.0.0:1041 0.0.0.0:0 LISTENING TCP 0.0.0.0:1048 0.0.0.0:0 LISTENING TCP 0.0.0.0:1723 0.0.0.0:0 LISTENING TCP 0.0.0.0:3268 0.0.0.0:0 LISTENING TCP 10.99.99.1:53 0.0.0.0:0 LISTENING TCP 10.99.99.1:139 0.0.0.0:0 LISTENING TCP 10.99.99.1:389 10.99.99.1:1092 ESTABLISHED TCP 10.99.99.1:1092 10.99.99.1:389 ESTABLISHED TCP 10.99.99.1:3604 10.99.99.1:135 TIME_WAIT TCP 10.99.99.1:3605 10.99.99.1:1077 TIME_WAIT UDP 0.0.0.0:135 *:* UDP 0.0.0.0:445 *:* UDP 0.0.0.0:1087 *:* UDP 10.99.99.1:53 *:* UDP 10.99.99.1:137 *:* UDP 10.99.99.1:138 *:* • Using the Windows Services control panel, verify that all appropriate services are running for that server. To view the Service control panel, go to Start and select Run. In the Run window, type services.msc and click OK. • Check the Application and System logs in the Windows Event Viewer for warning or error events. To start the Event Viewer, click Start and select Run. In the Run window, type eventvwr.exe and click OK. This launches the Event Viewer console, which displays the three logging options (Application, Security, and System). • A device may be in some state other than Up, such as Down, or Not Reachable. Check the Operations Console to verify the status of a particular device. To check the status: – From the Operations Console, select System > Control Center. – Click the Device Pool tab in the left frame of the page.


9


– In the right frame of the page, click the General tab if it is not already selected. – Locate the device you are interested in and check the status of that device in the Status column. • Check the call statistics from the Operations Console to try to determine any abnormalities. The reported values from the various components might provide some information that will assist you in isolating the issue. Note: The Operations Console online help describes the device status fields in detail. From the console, choose System > Control Center and then select the Device Pool tab and choose a device pool. You can also view a particular type of device by selecting the Device Type tab and choosing a device type. The status of some devices such as, IOS devices, Unified CM, ICM servers, and SIP proxy servers display as N/A (Not Applicable) because the Operations Console does not monitor these device types. You can access call statistics either by: – Choosing System > Control Center, selecting a Call Server, clicking Statistics icon in the toolbar, and then selecting the various tabs (Unified ICME, SIP, IVR, or Infrastructure). – Choosing Device Management > CVP Call Server, selecting a Call Server, clicking the Statistics icon in the toolbar, and then selecting the various tabs (Unified ICME, SIP, IVR, or Infrastructure). The examples above list the procedure for viewing the statistics for a Call Server. The statistics for other devices, such as the Reporting Server, VXML Server, and so forth can be viewed in the same manner. Below are some of the initial areas to check in the Infrastructure tab for abnormal values or thresholds. – Check for licensing issues in the Infrastructure tab. The Licensing Statistics column is broken up into Realtime, Interval, and Aggregate sections. Check the value being reported for Port Licenses Available against the value of Maximum Port Licenses Used. – Check the level of the Current Port Licenses State. This reports on the threshold level of port license usage. There are four levels: safe, warning, critical and failure. An administrator may set the required percentage of port licenses in use needed to reach a given threshold level, with the exception of the failure level which is reached when the number of ports checked out is equal to the total number of licenses ports. – Check the Uptime statistic, as well as the memory and thread pool statistics for possible issues. If the Uptime statistic is low, then that indicates that the component was recently unavailable for some period of time. High memory or thread pool statistics may indicate a low available resource issue. Below are some of the initial areas to check in the SIP tab for abnormal values or thresholds.


10

Chapter 1: Troubleshooting Overview Troubleshooting Checklist

– In the Interval Statistics section, check the value of the Failed SIP Transfers (Pre-Dialog). This is the total number of failed transfers on the first Unified CVP transfer in the last statistics aggregation interval. A SIP dialog is established after the first Unified CVP transfer is completed. The metric does not include rejections due to the SIP service being out of service. The metric includes failed transfers that were made after a label was returned from the Unified ICME in a CONNECT message. – Check the value of the Failed SIP Transfers (Post-Dialog). The total number of failed transfers on second and subsequent Unified CVP transfers in the last statistics aggregation interval. After SIP dialog is established, Unified CVP performs SIP transfers using re-INVITE messages. The metric does not include rejections of SIP REFER transfers. The metric includes failed transfers that were made after a label was returned from the Unified ICME in a CONNECT message. Below are some of the initial areas to check in the IVR tab for abnormal values or thresholds. – Check the Active Calls statistic. This is the number of active calls being serviced by the IVR service currently. Refresh the page to determine if the calls are increasing. – Check the Average, Maximum, and Minimum Call Latency statistics. These report the amount of time in milliseconds it took for the IVR Service to complete the processing of a New Call Request or a Request Instruction Request during the time interval. Long durations may indicate a network or load problem. Also, verify that your network environment has been correctly configured. The table below shows the speed and duplex settings for Unified CVP. Ethernet Switch Speed Capability

Server/Gateway NIC Speed Switch Port Speed/Duplex Server/Gateway NIC Capability Setting Speed/Duplex Setting

1000 Mb

1000 Mb

Auto / Auto

Auto / Auto

1000 Mb

100 Mb

100 Mb / Full

100 Mb / Full

100 Mb

100 Mb

100 Mb / Full

100 Mb / Full

100 Mb

1000 Mb

100 Mb / Full

100 Mb / Full

If you have exhausted all the common causes and actions including those outlined in this document or others that you have identified in your environment, contact the Cisco Technical Assistance Center (TAC) or your Cisco Partner.

Troubleshooting Checklist Complete this checklist to assist in isolating the issue, or to provide information to your support partner or Cisco Technical Support. 1. What is the version of Unified CVP that is currently running? Include any Hotfix or Service Release (SR) information. 2. Is this a new installation or an upgrade?


11

Chapter 1: Troubleshooting Overview Troubleshooting Checklist

3. If this is an upgrade, what version was previously installed? 4. When did the problem occur? 5. What are the observed symptoms, and the conditions under which these symptoms occur? 6. Was anything changed or updated in hardware, software, or network components prior to the first occurrence of the observed symptoms? 7. Describe the related call flow. Some examples include: Public Switched Telephone Network (PSTN) originated or IP Phone originated. If it is a transfer, is it Unified CVP dequeue or agent originated? Is the call inbound ACD, inbound IVR Service treatment, inbound queue treatment, outpulse transfer, or agent transfer? 8. Is the problem reproducible? 9. What Unified CVP deployment model used for deployment? 10. What is the call transfer method used? 11. Are you able to capture a screen shot of the error or failure? If Yes, save it to a file and attach to a case. Network Topology Complete this checklist to assist in isolating the issue, or to provide information to Cisco Technical Support. 1. Has auto-negotiate been disabled on all PCs, routers, and switch ports? Note: Duplex/speed mismatch between a device and its corresponding port on the switch is the single most common problem for network latency. 2. Is a network topology diagram available? 3. Which type of IP Gateway is being used in this Unified CVP solution? 4. Save the output from a show tech, show stack, and a show Gatekeeper endpoint from the Gateway. 5. Which type of Gatekeeper is being used in this Unified CVP solution? 6. On which server are the recorded media files located, and what is the path to those files? 7. Are Content Service Switches (CSS) used for failover? 8. Collect and provide versions of IOS, Digital Signal Processor (DSP) , applications, and Engineering Special (ES)/patch levels in the environment.


12

Chapter 2 Symptoms and Solutions Overview This chapter contains problem descriptions and steps to diagnose and resolve each problem. Problem descriptions are grouped by Unified CVP component.

Call Server This section contains problems associated with the Call Server.

Device Becomes Corrupted Symptom: A device has become corrupted. The user receives a specific error message that indicates what device has been corrupted.

Message: "Device with IP Address: {1} and Hostname: {2} is in an inconsistent state. Please Save and Deploy the properties of this Server again to fix the problem. If problem persists, please contact your administrator for more details." "Device with IP Address: {1} and Hostname: {2} is not ready at this time. Please retry the operation at a later time." "Device with IP Address: {1} and Hostname: {2} is in an inconsistent state and cannot be properly deleted at this time. Please retry the operation later or try deleting again for forceful delete."


13

Chapter 2: Symptoms and Solutions Call Server

Cause: The most likely cause is that the Unified CVP software was reinstalled.

Action: The following steps outline the process to resolve the issue of software being reinstalled on a server. 1. Identify the affected devices from the Control Center of the Operations Console. The affected devices are: - devices with the same IP addresses, and - devices associated to the devices identified with the same IP address. 2. Go to the Edit page of each device and click Save&Deploy. 3. Restart all affected devices from Control Center in the Operations Console. If the procedure above does not resolve the issue, then you must clean up the device by performing the following steps. Device Clean Up 1. Delete the device from the Operations Console. 2. Reinstall the software on that device. 3. Recreate the device from the Operations Console. Each Operations Console Server will be identified by a unique identifier. The identifier is generated at the time of creating a first device through that Operations Console Server. Once generated, the identifier will then be stored in the Operations Console Server database and in the Resource Manager Configuration file on every device that includes the Operations Console Server. The purpose of this unique identifier is to prevent management of a device from multiple Operations Consoles. The unique identifier gets imported whenever the import functionality of an Operations Console database is invoked. Importing a database which was exported from a different Operations Console Server is not a supported feature unless the Operations Console is newly installed or has never been used before. If a user uses an Operations Console Server that is already managing at least one device to import a database (from a different Operations Console Server or an empty database which was never used to create a device) the following side effects will occur: User may not be able to manage devices which were managed by the same Operations Console server prior to the import. This is because the unique identifiers will now be different between the Operations Console Server and the previously managed device. User may not be able to


14


create an IOS device such as Gatekeeper(s), Gateway(s), and CSS(s) from the Operations Console. An Operations Console Server that is managing at least one device must not import an empty database that was previously exported from any Operations Console Server. This will cause a mismatch in the unique identifiers used since the Operations Console database and the Operations Console Server have a unique identifier and the empty database has no identifier. If a customer requires replacing the existing database with an empty database, the recommended way is to reinstall the Operations Console Server software. If this step is not followed, the customer can no longer manage devices that were previously managed by the same Operations Console Server.

SSL Recovery Symptom: The Operations Console is unable to communicate with the Operations Console Resource Manager (ORM) service on a device.

Message: Call Server device with IP Address: and Hostname: operation failed. Device could not be reached because of a failure in negotiating security certificates. Please refer to documentation for configuring security before trying the operation again.

Cause: If the user checks the secure communications checkbox in Operations Console device configuration without first configuring security certificates as outlined in Chapter 6 of the Configuration and Administration Guide for Cisco Unified Customer Voice Portal, the Operations Console will not be able to communicate with the Operations Console Resource Manager on the device.

Action: The solution is to edit the orm_jmx.properties in the conf directory of the device. Change the property com.sun.management.jmxremote.ssl to false to put the device back into non-secure mode. Next, uncheck the checkbox in the Operations Console and restart the Operations Console Resource Manager service on the device. This will put both the Operations Console and the device Resource Manager back into a non-secure mode. The user can then configure security certificates before checking the secure communications checkbox. Note: Changes in security settings require that the Operations Console service and the Resource Manager service be restarted.

Resource Manager Service Fails to Start Symptom:


15


Since the default log setting is ERROR for the Resource Manager Wrapper, the log file should provide exception details of the failure. The wrapper log file does indicate there is a failure in Resource Manager service startup, but the details of the event are not logged.

Message: None.

Cause: The wrapper log was set to the wrong level.

Action: The change is unable to be made from Support Tools, so the change requires manual steps to change this setting. The log file to look for the exception is %CVP_HOME%\logs\orm_wrapper.log. 1. Stop the 'Cisco CVP Resource Manager' service. 2. Edit %CVP_HOME%\conf\wrapper.conf file to change the property 'wrapper.logfile.loglevel' value from 'ERROR' to 'INFO'. 3. Save and close the file. 4. Start the 'Cisco CVP Resource Manager' service. If the Resource Manager service fails to start, the ORM logs (orm_wrapper.log) collected from Support Tools will show the detailed information of the event.

Call Server Is Not Reachable Symptom: After a forced delete of the Reporting Server, the Call Server state did not change from the Down state to the Partial or Up state.

Message: Opsconsole Control Center: Call Server status is down.

Cause: The Call Server messageAdapter.properties file is corrupt.

Action: 1. From the command line, run %CVP_HOME%\bin\tac\reimage.bat on the Call Server. 2. Restart the Operations Console Resource Manager (ORM) service.


16


3. Log in to the Operations Console, and choose Device Management > CVP Call Server. 4. Click Save & Deploy to save the changes and apply them to the Call Server. Note: All co-located devices—including VXML Server, Reporting Server, and Video Media Server—must be deployed after reimage.bat is run. You must repeat these steps for each co-located device. Note: Optionally, you can reinstall the CVP device.

Call Server Cannot Communicate with Co-located Reporting Server Symptom: The environment: Call Server 1 (primary) & Reporting Server 1. Call Server 2 (secondary) In the above deployment, when the Operations Console Resource Manager (ORM) on Call Server 1 is down, the ORM on Call Server 2 is unable to communicate with the Reporting Server.

Message: None.

Cause: This is not a supported model for deployment.

Action: A co-located Reporting Server can never scale to support multiple Call Servers and must not be configured in that manner.

Call Server or VoiceXML Service Cannot Be Restarted Symptom: After applying license files, the Call Server service or the VoiceXML service are unable to be restarted. Both services can be stopped, but receive a warning message when trying to restart them.

Message:


17


Windows could not start the Unified CVP Call Server/VXML Server on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 0.

Cause: The problem is due to the side effect of the first version (1.0) of the MS06-040 patch (KB921883):

Action: Apply the new version of the Microsoft security patch KB921883. See Also Refer to the Microsoft Security Bulletin KB921883 that is available on the Microsoft web site. Go to: http://support.microsoft.com/

Connection with PIM Dropped, Call Server Going Out of Service Symptom: The TCP connection between ICM VRU PG and CVP CallServer ICM subsystem was dropped. • Calls are dropped during load conditions. • Wireshark can not capture H.323 messages when a call processing load is running on the Call Server.

Message: The following errors are written to the CVP error log file when either the connection between the VRU PIM and CVP drops for an unknown reason, or the VRU PIM is intentionally shut down for maintenance: • LOGMSG_ICM_SS_PIM_SHUTDOWN: Lost socket connection to VRU PIM. Transitioning to partial service. • LOGMSG_ICM_SS_STATE: Waiting for VRU PIM Connection.

Cause: The TCP Chimney feature is an offload engine that was introduced in Windows Server 2003 Scalable networking Pack.

Action: Disable the TCP Chimney feature.


18


1. Open Registry Editor, and locate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 2. Set the EnableTCPChimney parameter to 0 . 3. Reboot the machine (required).

H.323 Service Cannot Access Call Server Symptom: Some HTTP requests from the Unified CVP Voice Browser to the Call Server are being blocked.

Message: Blocked by port blocking rule VoiceBrowser.exe Prevent IRC communication. Blocked by port blocking rule tomcat5.exe Prevent IRC communication.

Cause: VirusScan is blocking ports in the range of 6666-6669, which are used in IRC communications. The application does not have to be listening on these ports for it to be blocked. If the application is assigned a client port in that range, the attempt to connect to a server using that port is blocked by VirusScan on the server machine. The connect attempt fails.

Action: Check the VirusScan logs for error messages. access the logs,VirusScan Console -> Task -> View Log You might find entries like these: 10/10/2006 3:15:01 AM Blocked by port blocking rule VoiceBrowser.exe Prevent IRC communication 10.86.129.138 10/10/2006 8:49:01 PM Blocked by port blocking rule tomcat5.exe Prevent IRC communication 10.86.129.211. To view VirusScan logs 1. Go to Start > Programs > Network Associates > VirusScan Console. 2. Click on the Task menu and select View Log. You might find entries such as: Blocked by port blocking rule VoiceBrowser.exe Prevent IRC communication. Blocked by port blocking rule tomcat5.exe Prevent IRC communication. INFO: Retrying communication with Call Server localhost:8000 INFO: Retrying communication with Call Server localhost:8000 Voice Browser to Call Server localhost:8000 connection unsuccessful.


19


Voice Browser cannot access any Call Servers and is no longer accepting calls. Voice Browser is out of service. Exclude the Unified CVP processes from port blocking when using McAfee VirusScan on a system running Unified CVP Call Server. If a different anti-virus product is used, do the equivalent exclusions for the port blocking rules for that product. Instructions for McAfee VirusScan on Unified CVP Call Server. 1. Open the VirusScan Console. 2. Click Access Protection. 3. Click Properties under the Task menu. 4. Click the Port Blocking tab. 5. Check both Prevent IRC Communication rules. 6. Click the first Prevent IRC Communication rule and click Edit. 7. Enter VoiceBrowser.exe,tomcat5.exe to the Excluded Process field. 8. Click OK. 9. Click the second Prevent IRC Communication rule and click Edit. 10. Enter VoiceBrowser.exe,tomcat5.exe to the Excluded Process field. 11. Click OK.

Resetting System Clock Stops IVR Service Requests Symptom: Resetting the system clock on the Call Server causes the IVR Service to stop functioning.

Message: None

Cause: Changing the time of the system clock on the Unified CVP Call Server causes the IVR Service to stop accepting calls.

Action:


20


Do not reset the Windows system clock on a machine running Unified CVP. Resetting the Windows system clock is not supported on a Call Server.

IVR Service CALL_RESULT Returns Error Code Other than 0 Symptom: The IVR Service sends a CALL_RESULT error code other than a zero (0) to the Call Server log.

Message: Below is an example of one of the messages: 4765560: 10.86.129.211: Aug 18 2007 06:09:59:484 -0400: %TEMP:CVP7.0_IVR-7-TRACE_CALL: {Thrd=http-8000-3} VBServlet:service: Request from 10.86.129.20: {CALL_ID=979679B9-2DD811DB-B72D0014-6944B762, MSG_TYPE=CALL_RESULT, CALL_SEQ_NUM=2, ERROR_CODE=32}

Cause: The problem is due to an error with the Run Script Request from Unified ICME.

Action: Check the Error Code that was received in the Event Codes and Reasons appendix of this document, and take the appropriate action. For additional information and a complete list of the cause codes, refer to the Cause Codes and Debug Values section of the Cisco IOS Voice Troubleshooting and Monitoring Guide.

Call Server Is Unreachable Symptom: The Call Server is unreachable.

Message: UNREACHABLE

Cause: A configured component is unreachable in any one of the following three conditions: • Operations Console is unable to connect to the Operations Console Resource Manager that is co-located with the Call Server. For example, the Operations Console Resource Manager is down. • The Operations Console Resource Manager on the Call Server has not received a state event from the controller of that component.


21


• The Operations Console Resource Manager is unable to connect to the Call Server so it is not receiving state events from the central controller. For example, the Call Server is down.

Action: In this example, all three services (Unified ICME, IVR, and SIP) have been configured. The central controller reports that IVR Service and SIP are IN_SERVICE, but it does not report the status of the Unified ICME Service to the Operations Console for some reason. When this occurs, Operations Console reports the status of Unified ICME as UNREACHABLE. Operations Console aggregates the status of the various components of the device to arrive at device status. It sees that two of the components (IVR Service and SIP) are in SERVICE, but Unified ICME is UNREACHABLE. Operations Console shows the status of the Call Server as UNREACHABLE even though IVR Service and SIP are actually in service. Check the network environment for connectivity, as well as the actual state of the server.

Received ERROR_SCRIPT_NAME = 6 on IVR Service Symptom: The ERROR_SCRIPT_NAME = 6 error code is received on the IVR Service.

Message: ERROR_SCRIPT_NAME = 6

Cause: The VRU Script Name data that is passed from the Unified ICME to the IVR Service does not contain the expected components. These components are micro-application name, media file name, and media file type.

Action: In the Unified ICME Network VRU Script List tool, locate the appropriate script and resolve the configuration error. See Also For more information, refer to the Micro-Application configuration section of the Unified CVP Configuration and Administration Guide.

Invalid Path or File Name Symptom: Note: This is a Call Server issue that is found within the Video Media Server (VMS), which is a part of the Call Server.


22


Sometimes, a file is read from the VMS movies directory with a name that does not validate to the standard accepted character set. These files are marked with the "File could not be read" status.

Message: File could not properly be read from the file system.

Cause 1: An invalid status for a media file stating that the "file could not be read", generally implies that the file has an invalid path or file name.

Action 1: Verify and correct the path or file name.

Cause 2: An invalid status for a video media file (that is a 3gp or an XML file) stating that the "file could not be properly read", generally implies that the file has an invalid file name or presents within an invalid path.

Action 2: Verify and correct the path or file name. The file names can contain alphanumeric characters, underscore, spaces, and the following special characters: !$&().,'-;@[]^_`{}~+=. File names cannot lead with a period (.) or a space. Only path names with alphabetical, numeric, spaces, and !$&().,'-;@[]^_`{}~+= characters are permitted; path names cannot lead or end with a period (.) or a space.

“Dialogue Failure Event” Error in the Call Server Log Files Symptom: Callers hear critical error media and “Dialogue Failure Event” errors appear in the Call Server log files.

Message: None.

Cause: If a call reaches the end of ICM script processing without being queued or released, the Unified ICME sends a “Dialogue Failure Event” message to the Call Server. The Call Server, in turn, sends an error message to the H.323 Service.

Action:


23


The author of the script must ensure that each path in the script ends with one of the following Script Editor nodes: Release, Label, Skill Group, or Queue to Skill Group.

Call Server “Variable Data Is Invalid” Error Symptom: The Call Server is reporting an error.

Message: Variable Data is Invalid.

Cause: The ICM script might truncate values with leading zeros or zeros after decimals or rounds values.

Action: Put quotes around numbers in a Script Editor Set node so it is processed as a string. This is especially important if: • Leading zeros are present (example: dates). • Trailing zeros are present after a decimal point (example: currency). • The number is very large (example: a number normally expressed through exponential notation).

Unable Add IOS Device Because Device Is Already Managed Symptom: When attempting to add a device to the Operations Console, an error message is received.

Message: Gateway device with IP Address: and Hostname: cannot be created. The device is already managed by another Operations Console.

Cause: This occurred because a database was imported from a different Operations Console. Each Operations Console Server is identified by a unique identifier. The identifier is generated at the time of creating a first device through that Operations Console Server. Once generated, the identifier will then be stored in the Operations Console Server database and in the Resource Manager Configuration file in every device that includes the Operations Console Server. The


24


purpose of this unique identifier is to prevent multiple Operations Console Servers from configuring the same device. The side effects of importing a database to an Operations Console Server that is managing at least one device, if the database to be imported is an exported database from a different Operations Console Server or is an empty database which was never used to create a device are: 1. User may not be able to manage devices which were managed by the same Operations Console server prior to the import. 2. User may not be able to create an IOS device such as Gatekeeper(s), Gateway(s), and CSS(s) from the Operations Console. Importing a database which was exported from a different Operations Console Server is not a supported feature. In this case the import may succeed but will have various side effects including the above two side effects.

Action: Importing a database which was exported from a different Operations Console Server is not a supported feature. The solution is to never import an empty database to an Operations Console Server, if the server is managing at least one device prior to import. If customers require replacing the existing database with an empty database, the customer is to reinstall the Operations Console Server software. Doing so, the customer can no longer manage devices that were previously managed by the same Operations Console Server.

VRU Application Error in Call Server Log Symptom: The call might not go through the Call Server, so a prompt is not played. An error or warning message might appear in the Call Server log.

Message: VRU APPLICATION ERROR: Assigning the tag 7 to the non-existing ECC variable “user.microapp.error_code”.

Cause: The ECC Variable is not configured on the Unified ICME and/or NAM software, or the defined length is not the same on both devices.

Action: Add the identical ECC variable definition to the Unified ICME and/or NAM.


25


ECC Variable Contents Truncated after Passing through Call Server Symptom: The contents of an ECC Variable configured in Script Editor are truncated after passing through the Call Server. For example, the Text-To-Speech (TTS) text that is spoken to the caller is not the complete text that was configured. Another example of this behavior is where there are media fetch failures and the URL to the media file is only a subset of the expected URL.

Message: None.

Cause: The cause of this problem may be that the length of the ECC Variable value that was set in Script Editor is longer than the maximum length of the ECC Variable configured at setup time.

Action: The solution is to make the maximum length of the ECC Variables longer. This is done using the Unified ICME Configuration Manager. If you do alter the maximum length of ECC Variables, you need to restart the Call Server after making the change. Note: In a NAM/ICM environment, the length needs to be identical on all NAM and Unified ICME environments or the variable will not pass.

"Have You Configured Your Call Server?" Error Symptom: The Call Server is running, but the an error message is logged in the log file.

Message: Have you configured your Call Server?

Cause: The Call Server was started from the Services Control Panel, but it was not configured from the Operations Console.

Action: Configure the Call Server from the Operations Console. After completing the configuration, restart the Call Server from the Services Control Panel.


26


Erratic Behavior When Everything Is Configured Properly Symptom: General erratic behavior when everything is configured properly. Sometimes TTS plays, sometimes it does not, or the call stops when the VXML Server is running in script node in Unified ICME. Other behaviors include calls succeed in Unified ICME, but no TTS plays, or the caller hears silence, but no errors are reported in the log. Unavailable TTS errors are reported.

Message: None.

Cause: If everything appears to be configured properly, and behavior is generally erratic, check the configuration of your Ethernet link.

Action: Do the following: • Check the Gateway log files. • Check the VXML Server application logs. • Verify the network settings. The chart below shows the speed and duplex settings for Unified CVP. Table 1: Speed and Duplex Settings

Ethernet Switch Speed Capability


1000 Mb

1000 Mb

Auto / Auto

Auto / Auto

1000 Mb

100 Mb

100 Mb / Full

100 Mb / Full

100 Mb

100 Mb

100 Mb / Full

100 Mb / Full

100 Mb

1000 Mb

100 Mb / Full

100 Mb / Full

IVR Subsystem Goes into Disabled State When Call Server Service Starts Symptom: When the Call Server service is started, the IVR subsystem goes into a Disabled state.

Message: Cause 1:


27


The Unified CVP license in use is a non-expiring license (no expiration date).

Action 1: Use a Unified CVP license that has an expiration date

Cause 2: If the Unified CVP license in use has an expiration date, the Unified CVP Call Server service need to be recycled.

Action 2: Recycle the Unified CVP Call Server Service. After recycling, check to see if the IVR subsystem is not in a Disabled state. If it is disabled, the process needs to be repeated a few times.

IVR Subsystem Returns a 500 Rejection on VXML Fetch from Gateway Symptom: IVR Subsystem returns a 500 rejection on VXML fetch from the gateway as seen in the gateway or CVP logs.

Message: Cause: Incorrect setting of parameter "Max Length of DNIS" on ICM service configuration page of Operations Console.

Action: Set parameter to the same length as the DNIS part of the label returned by ICM.

Full Video Call Disconnected When Routed Through IVP Symptom: Full video call is disconnected when it is routed through IVP.

Message: Cause: Access number DN prefixes are not added to the iCONTACT application and/or the iCONTACT xml configuration.

Action:


28


See Chapter 17 in the Configuration and Administration Guide for Cisco Unified Customer Voice Portalfor information on modifying the Access Number in the iCONTACT xml file.

SIP Code 488 or 603 Returned on Full Video Call Rejection Symptom: SIP code 488 or 603 is returned on a full video call rejection from IVP. A Fast Busy is heard.

Message: Cause: MCU or MVP/EMP unit on IVP is not registering or is in service.

Action: Follow the steps in the Radvision documentation to place the MCU or MVP in service.

Full Video Warm Transfer with Queuing Not Working Symptom: SIP trunk on outgoing call to VRU label is not getting routed to IVP.

Message: Cause: Action: Set the SIP trunk on the route pattern for the ICME VRU label to point to IVP instead of CVP.

CVP Warm Transfer With Queuing Get Disconnected Symptom: CVP warm transfer with queuing gets disconnected after a few seconds when the transfer is complete. CUCM logs and/or sniffer trace show a reinvitation collision scenario. This only occurs with IP originated callers, not TDM callers.

Message: Cause: This problem is visible when CVP and CUPS are using proxy and UDP on the outgoing direction, and the CUCM SIP trunk is using TCP on the outgoing direction.

Action: Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

29


Set UDP on the CUCM SIP trunk security profile or set CVP and CUPS to use TCP.

Hold Movie not Visible to Caller on Hold Symptom: Note: This tip applies to full video only. The caller does not see a hold movie when agent places the caller on hold.

Message: Cause 1: Mis-configuration of ECC variables that control the hold movie to be played to caller.

Action 1: Make sure the user.microapp.media_server and user.cvpmovies_bg_media ECC variable are configured correctly via the "Set Variable" node in the ICM script: for example, user.microapp.media_server = "rtsp://10.86.129.233" user.cvpmovies_bg_media="app/hold.3gp"

Cause 2: The default hold movie is not configured in the /home/rv/ivp/applications/iCONTACT/conf/iCONTACTConfiguration.xml file.

Action 2: Verify if a default hold movie is configured in the /home/rv/ivp/applications/iCONTACT/conf/iCONTACTConfiguration.xml file as follows: agent rtsp:///en-us/app/hold.3gp

Specify the RTSP URL (as shown above) or as a file:// URL (if file stored locally on IVP box). (e.g. file:///home/rv/movies/local/hold.3gp). 1

Full Video Calls Drop upon Dialing the DNIS Symptom: 1)

A Linux server in which the IVP software from Radvision is loaded is refered to as an IVP system. IVP systems are used as part of the Unified CVP Release 7.x solution.


30


Upon dialing the Dialed Number Identification Service (DNIS), full video calls are immediately dropped.

Message: none

Cause 1: The connection between Unified CVP and the Interactive Video Platform (IVP) system from Radvision (used as part of the Unified CVP solution) may not be established.

Action 1: Verify the following requirements: • The Call Server status is functioning (up) in the OAMP control center. • A valid connection exists between Unified CVP and IVP via Diag Servlet. To verify, click on Dump IVR subsystem and look for the following ContactClient statements: ContactClientManager: video enabled: true ContactClient count: 1 ContactClient [192.168.150.170]:9981, connectionId:iCONTACT-Connection-1 • Look for following log message in CVP log: 2472: 192.168.150.173: Sep 24 2007 11:35:25.500 -0400: %CVP_7_0_IVR-1-VIDEO_CONNECTION_UP: Connection to iContact server [192.168.150.170]:9981 is UP [id:3100]

Cause 2: The access numbers and/or prefix numbers have not been configured for iCONTACT in IVP Manager and/or iCONTACT configuration file.

Action 2: 1. Configure the access numbers and/or prefix numbers for iCONTACT using IVP Manager. 2. Configure the same access numbers and/or prefix numbers in the iCONTACT configuration file. 3. Restart IVP after making these changes.

Undecipherable Message Is Displayed When the VMS Subsystem Is Down Symptom:


31


When playing a video to a caller, the DarwinStreamingServer.exe is terminated. The video stops playing and an error message is recorded in the VMS Error Log.

Message: 340: 10.30.30.97: Oct 25 2007 16:59:46.267 -0400: %CVP_7_0_VMS-3-VMS_SUBSYSTEM_STREAMING_SERVER_DOWN: The Streaming Server State: UNKNOWN. [id:11512]

Cause: Note: There are three internal states outputted by the VMS Darwin Streaming Server keep-alive thread which help drive the overall state of the VMS. These are internal states, not necessarily representative of the state of the VMS as a whole. The appropriate error message for an agent to see when VMS detects Darwin is down is "VMS Subsystem is down". You may see a different error message if the streaming server poller is in between cycles when the VMS Subsystem goes down. This error message is accurate as it is the output from the keep-alive thread, which tracks the VMS Subsystem through three states: IN_SERVICE, PARTIAL_SERVICE and UNKNOWN. These are not the states that the VMS Subsystem uses.

Action: To see the appropriate error message, wait for one full polling cycle (default: 30 sec) to ensure that the proper error message appears.

Connection between Unified CVP and IVP cannot be established when setting up TLS Symptom: Note: This tip applies to full video only. When the Transport Layer Security (TLS) is being set up, the connection cannot be established between Unified CVP and IVP

Message: SSL session handshake failed (is the server SSL enabled?). Log messages constantly show connection being made and then broken.

Cause 1: The TLS setting is enabled on the Call Server but not in iContact. Enable TLS setting on Call Server and IVP are not in sync.

Action 1:


32


Verify that both Call Server and iContact are configured for TLS. Call Server configuration is enabled via OAMP. iContact configuration is via iContactConfiguration.xml. Both Call Server and IVP must be restarted if their settings for enabling/disabling TLS are changed.

Cause 2: The required procedures to import/export keys between Call Server and IVP has not completed or is not accurate.

Action 2: Verify that the proper keys are imported and exported. Refer to the TLS configuration documentation for procedural Call Server/IVP information.

Only One Call Server Is Receiving Calls Symptom: Note: This tip applies to full video only. When multiple Call Servers are connected to a single iCONTACT, only one of the Call Servers is receiving calls.

Message: None

Cause: iCONTACT configuration may not be set to distribute calls to Call Servers via round-robin mechanism.

Action: Open the iCONTACT configuration file and verify that the notification type is set to round robin i.e Round-robin.

Calls Are Disconnecting Symptom: There are a number of abnormally disconnected calls.

Message: Calls are disconnecting.

Cause:


33


The Radvision IVP Linux interface, as well as Radvision MVP/MCU, does not autonegotiate interface link speed/duplex mode properly.

Action: On the Radvision IVP Linux server, use the ethtool utility to manually set the current interface link speed (MB) and duplex mode. 1. At the command prompt type, ethtool -s eth0 speed 100 duplex full 2. To permanently change the interface link speed/duplex mode settings (that is, to survive a reboot), in the /etc/sysconfig/network-scripts directory add the following line to the ifcfg-eth0 file: ETHTOOL_OPTS="speed 100 duplex full autoneg off "

Non-English Video Text Overlay is Garbled in Full Video Symptom: Message: Cause 1: The "user.micropp.charset" ECC variable is not set correctly for the target language.

Action 1: Verify that the correct value is set in the "user.microapp.charset" ECC variable for the target language in the ICM script. The supported values values are specified in the Configuration and Administration Guide for Cisco Unified Customer Voice Portal.

Cause 2: The Radvision MCU is not configured for the target language.

Action 2: Upgrade the Radvision MCU to the target language so that the proper fonts are loaded. If the MCU is already configured for the target language, restart the MCU and EMP

Movie Media of Video Micro-Application Cannot be Interrupted Symptom: Note: This tip applies to full video only. Movie media of video micro-app cannot be interrupted by a DTMF barge-in or when the agent becomes available.


34


Message: None.

Cause: This can happen if the micro-application movie media only has an audio-stream and does not have any video stream.

Action: In the Network VRU script micro-app configuration, change the Show/Display option in the Configuration to A for audio-only.

Media Poller timing issues Symptom: Metadata files are modified from within the administrative web UI or from within the file system while the media poller is polling in the background.

Message: none

Cause: The media poller feature is used to automatically add new video resources from the file system, but can result in some rare timing issues. The media poller begins its polling and cycles through the c:\foo directory. At this time, if an administrator adds and saves a new file c:\foo\bar.xml using the administrative web UI, the administrator (and possibly agents) see bar.xml displaying in their web UI. At this point the media poller cycle ends as c:\foo\bar.xml was not present when that directory was polled. This is because c:\foo\bar.xml is considered to be deleted by the media poller and is removed from memory. Subsequent refreshes of the administrative or agent web UIs show c:\foo\bar.xml as missing. The following media poller cycle picks up c:\foo\bar.xml more permanently.

Action: This is a subtle, timing issue. In the end, the media poller performs its function and gets back to synchronizing with the file system after another polling cycle. The VMS media poller process will never delete or modify existing files on the system. It will only poll for their existence and current status.

Invalid Thumbnail Images on VMS Server Symptom: Invalid images in the VMS movie library causing unintended consequences on the VMS system.


35


Message: none

Cause: none

Action: Be sure to adhere to the following guidelines for thumbnail images on the VMS system. Valid thumbnail image pixel size is 120 x 80 (width x height).Thumbnail images must have the same name as the XML file, with the exception of the file extension. Supported file extensions are: .jpg, .gif, and .png. Note: Large images use more network bandwidth and will impact Video Media Server performance when they are displayed to agents. Images that are larger or smaller than120 x 80 will be resized for display only. The content on the image itself does not change on the file system.

CALL Server Cannot Connect to IVP Server Symptom: Note: This tip applies to full video only. Call Server cannot get a connection to the IVP server.

Message: The following is seen in the iCONTACT log: 2007-11-16 18:41:09,320 (iContactStartupHandler) - Can not start iContact application server. The following is seen in the Call Server log CVP_7_0_Infrastructure-3-SOCKET_SETUP: Failed getting socket for host: 10.86.129.104 port: 9981 Exception: java.net.ConnectException: Connection refused: connect

Cause: The most likely cause is that the iCONTACT XML configuration file has an XML syntax error.

Action: Fix any XML syntax errors in the iCONTACT configuration file and restart the IVP server using the IVP Manager.

VMS Administrator Replaces Video Content While Agent is Previewing Symptom: Note: An administrator should be aware that deleting/replacing/moving video files while calls are being served can result in unpredictable behavior. VMS is limited in that it cannot control


36


how the Darwin Streaming Server or the IVP server will react to file changes. However, the results are very minor. At worst, the currently playing video stops, which is not fatal to the system. VMS Administrator replaces a .3gp video file with new content, but retains the same filename while at the same time a VMS Agent is Previewing the initial video. The video displaying in the VMS Agent QuickTime window freezes.

Message: none

Cause: none

Action: VMS Agent must close the QuickTime video window, reselect the video, and then Preview. The new video content will now be available.

VMS Administrator Replaces Video Content While Agent is Pushing Initial Video to Caller Symptom: Note: An administrator should be aware that deleting/replacing/moving video files while calls are being served can result in unpredictable behavior. VMS is limited in that it cannot control how the Darwin Streaming Server or the IVP server will react to file changes. However, the results are very minor. At worst, the currently playing video stops, which is not fatal to the system. VMS Administrator replaces a .3gp video file w\new content, but retains the same filename while a VMS Agent is Pushing the initial video to a Caller. The video displaying in the Callers window stops.

Message: none

Cause: none

Action: VMS Agent must reselect the video and then Push it to the Caller. The new video content will display to the Caller.


37

Chapter 2: Symptoms and Solutions VXML Server

Unable to Process Playback Request Symptom: An agent may see this error in the Cisco Unified CVP Video Desktop if the agent attempts a playback request after a call has ended. However, if the agent sees this during an active call, it means that the Video Desktop is still trying to send requests for a previous call.

Message: Unable to process playback request. The call has been lost.

Cause: Action: To be able to send playback requests for the current call: 1. Close the Video Desktop browser window. 2. Click the Video Desktop button in CTIOS to open the Video Desktop with the new call information. 3. Playback requests should now work correctly.

VXML Server Ended Session Request Error Symptom: This error message appears when an HTTP request initiated by the root document is made by the browser for a session that the VXML Server has already ended. The way the VXML Server knows what session the request is asking for is by looking at the cookie included in the request for the session’s ID. A request from the root document indicates either a VXML error occurred or the caller hung up. In a typical call, all requests made for a call session come in series and so there is never a request that comes after the VXML Server believes the call has ended. This error message would appear if a request telling the VXML Server to end the session is followed by a new request initiated by the root document referencing the same cookie.

Message: SERVER ERROR: There has been an error with the http session: a request was made due to an event caught in the root document that refers to a session that has already ended.

Cause 1:


38


This is a very rare situation as normal operation would not cause any further requests to be made after the request that ended the session (it is expected that the session-ending request is the last one). Note that for this error to appear, the new request must come from the root document, in many cases indicating an error encountered by the H.323 Service. There could be several rare situations where this could occur. The first begins with the end of call event activated by the final request to VXML Server taking a long time to run. This would cause the browser to make a new request timing out the session. This next request would have to be made before the initial request returned, but after the end of call event completed, causing the VXML Server to end the session before the new request was received. In order for this situation to occur, the timing would have to be very precise.

Action 1: For the first situation, the developer must ensure that their end of call events do not have the possibility of taking longer to execute than the H.323 Service’s fetch timeout VoiceXML property. Another possible solution if the end of call event is a Java class would be to execute the class in a separately spawned thread. The problem with this would be that the spawned thread should not attempt to access content in the session due to the possibility that it could be invalidated while the thread was executing.

Cause 2: Another situation that could cause this would be some network overhead or slowdown causing the final response for a call to fail to arrive at the browser, precipitating a time out to occur. Since VXML Server functioned without problem and ended the session, this new request would cause the error to occur. Note that some Call Servers and operating systems under high HTTP load can queue HTTP requests and potentially even lose requests if the queue is too large. The administrator should investigate this queue if their system is expected to support high load (for example, on Apache Tomcat, this is known as acceptcount).

Action 2: For the second situation, it may be an indication of excessive network, operating system, or Call Server load. For the network, perform diagnostics to ensure proper operation. For the operating system, check the CPU and memory utilization to ensure they are not maxed out. And for the Call Server, check the configuration to ensure the HTTP request queue is not too small (for example, on Apache Tomcat, the acceptcount setting can be configured in the conf/server.xml configuration file). Finally, the administrator may decide to direct less load on the system. The administrator may also choose to ignore these issues if they occur rarely since there are no bad consequences for the call session that encounters this error. Severity: Due to the fact that this error message appears only in very rare situations, as well as the fact that even if it does occur, the caller does not encounter any issues with their call, this error message is benign.

Ending Session Request Error Symptom:


39


This error message appears when an HTTP request initiated by the root document is made by the browser for a session that VXML Server has started the process of ending but that process is not complete. This is similar to the previous error, but where in that situation the request was made after the session had completed, this situation involves the request occurring while the VXML Server was ending the session but before it completed.

Message: SERVER ERROR: An HTTP request was made due to an event caught in the root document to a session that has been slated for invalidation.

Cause: One very rare situation that could cause this error to occur under standard call volume involves a perfect timing of events. This error would not be produced if the extra request arrives while an end of call event was executing. However if it did take a long time to execute, prompting the browser to time out the request by having the root document make a new request, and that request came at exactly the time the end of call event completed but before VXML Server was able to finish the session ending process, this error would occur. Since this process normally occurs rapidly, the chances that this would occur during this period is very small. This error would also be encountered if the network, operating system, or Call Server on which VXML Server is installed is sufficiently overloaded such that either HTTP requests are being queued up too much, HTTP requests are being dropped or the Call Server is so overloaded that the normally trivial process of ending a session takes a non-trivial amount of time. Severity: The severity for this error is very low because there is no effect on the caller since this occurs after the call is complete. Additionally, the situations in which this can occur during normal operation are extremely rare.

Action: For the first situation, the developer must ensure that their end of call events do not have the possibility of taking longer to execute than the H.323 Service’s fetch timeout VoiceXML property. Another possible solution if the end of call event is a Java class would be to execute the class in a separately spawned thread. As long as the end of call class completed its execution before the session invalidation delay specified in the conf/global_conf.xml of the installation directory, there would be no risk of the class referring to data within the session after it was invalidated. For the second situation, it may be an indication of excessive network, operating system, or Call Server load. For the network, perform diagnostics to ensure proper operation. For the operating system, check the CPU and memory utilization to ensure they are not maxed out. And for the Call Server, check the configuration to ensure the HTTP request queue is not too small (for example, on Apache Tomcat, the acceptcount setting can be configured in the conf/server.xml configuration file). Finally, the administrator may decide to direct less load on the system. The administrator may also choose to ignore these issues if they occur rarely since there are no bad consequences for the call session that encounters this error.


40


Session Does Not Contain Call Information Symptom: This is a rare error message that is displayed after end session request error. In this situation, VXML Server has completed its process of ending a session, however the Call Server has not completed its process for ending the session when a new HTTP request is received referencing that session. This would yield a legitimate session from the Call Server’s standpoint but an invalid session by VXML Server’s standpoint.

Message: SERVER ERROR: The session is not associated with a Server Configuration.

Cause: The situations where this would be encountered are identical to those listed in the previous error condition except the timing of the second request would occur after the standard end of call process taken by VXML Server completed but before the Call Server actually removed the session. The timing for this situation is actually tighter than the previous situation and therefore would occur even more rarely. While there is a chance that this could occur under standard call volume, it would more likely occur only under extreme load to the network, operating system, or network. Severity: The severity for this error is very low because there is no effect on the caller since this occurs after the call is complete. Additionally, the situations in which this can occur during normal operation are extremely rare.

Action: For the first situation, the developer must ensure that their end of call events do not have the possibility of taking longer to execute than the H.323 Service’s fetch timeout VoiceXML property. Another possible solution, if the end of call event is a Java class would be to execute the class in a separately spawned thread. As long as the end of call class completed its execution before the session invalidation delay specified in the conf/global_conf.xml of the installation directory, there would be no risk of the class referring to data within the session after it was invalidated. For the second situation, it may be an indication of excessive network, operating system, or Call Server load. For the network, perform diagnostics to ensure proper operation. For the operating system, check the CPU and memory utilization to ensure they are not maxed out. And for the Call Server, check the configuration to ensure the HTTP request queue is not too small (for example, on Apache Tomcat, the acceptcount setting can be configured in the conf/server.xml configuration file). Finally, the administrator may decide to direct less load on the system. The administrator may also choose to ignore these issues if they occur rarely since there is no bad consequences for the call session that encounters this error.


41


Rapid Multiple Sequential Applications Warning Symptom: This warning message appears when an HTTP request that is not initiated by the root document is made by the browser for a session that VXML Server has started the process of ending, and a built-in error-correcting mechanism within the VXML Server has attempted to resolve the issue by ignoring the cookie hence issuing a warning rather than an error. While the scenario is similar to previous errors, this is a warning, not an error since the call itself was handled without the caller encountering any error messages. The warning message is logged to inform the administrator of a situation that could indicate high call volume, which could cause other more severe issues.

Message: SERVER ERROR: Warning: An HTTP request was made to a session that has been slated for invalidation. It is treated as if not associated with a session.

Cause: The likeliest situation that would cause this warning message to appear involves a setup where a VXML Server application is configured to return to Unified ICME, that after some processing promptly visits a second application within the same call. Due to the fact that these two applications are visited in the same call, the cookie created in the first application and stored by the browser is referenced in the first HTTP submit to the second application. This is not normally an issue because the first application’s session ends before the next application’s first submit is made, especially if the ICM script performed time-consuming tasks before visiting the second VXML Server application. However under high volume, the process for ending a session may not take a trivial amount of time. Additionally, if the ICM script completes in a very short period of time, basically a shorter amount of time than it takes the Call Server to end a session, the second application’s first request would arrive before the first application’s session ended and with the cookie referenced in the header, would make the VXML Server believe that the new request was made for the first application. VXML Server has a mechanism to recover from this exact situation. It waits for a certain period of time for the first application’s session to end and when it does, continues with the call to the second application normally. It logs this warning as a note to the administrator that the volume may be reaching its upper limits. Severity: Very low since no call was adversely affected (any problem was averted by the error-correcting mechanism in the VXML Server).

Action: If this warning appears very rarely, the administrator can safely ignore these warnings. If they appear more frequently, this would be an indication of high volume and the administrator should attempt to reduce the call volume to the machine. Should it be found that the ICM script executed in between visits to VXML Server applications runs too quickly, a workaround to use would be to put a slight pause in this script to allow the first application’s session to end. Performing load tests on the system yields the optimal pause length. Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

42


Session Loss During a Call Error Symptom: The bad_url error message is generally displayed when VXML Server receives an HTTP request it does not expect or understand. This occurs when a request is made to an application that does not exist (or the application name is misspelled or has case-sensitive differences). The error message includes the URL that caused the error, including any HTTP GET arguments. The message, however, does not report any HTTP POST arguments due to their potential size, so when the URL “/CVP/Server” is displayed, this indicates that there were either only HTTP POST arguments passed to the URL or no arguments passed at all. The initial URL for an application as well as the URL requesting the root document includes GET arguments (that is “/CVP/Server?application=MyApp”), however all subsequent requests made by the VXML Server use HTTP POST arguments to the /CVP/Server URL. Therefore, this error message is most likely due to an HTTP submit made for a standard request within a call that has somehow lost its session or the cookie that references the session. This error would likely occur within a call and so the caller would hear an error message.

Message: SERVER ERROR: bad_url: /CVP/Server

Cause: There are several situations where this can be seen: 1. An H.323 Service or person accesses VXML Server via an incorrectly configured URL “/CVP/Server” without passing any arguments. Since the VXML Server does not know what to do, it returns a VoiceXML page with an error message and logs the error. 2. Some network device involving either the H.323 Service or some entity in the network such as a load balancer, loses or removes the cookie associated with the call session. In an environment in which a load balancer distributes call load across multiple VXML Server instances, it is necessary for the load balancer to be “sticky”. This means that once a call is sent to a server, all subsequent requests for the same session must be sent to the same server. If the load balancer fails to do so, it sends a request with a cookie referring to an unrecognizable session to a server prompting it to return an error VoiceXML page and record this exact error. A similar situation occurs if some other network device acts to remove cookies from HTTP requests or modifies them for its own purposes; these would become unrecognizable by the VXML Server. 3. An issue with developer-specified code. If the developer uses an element that manually invalidates the call session, but that element contains VoiceXML that requests a new page, this error would occur because once the session is invalidated, any further requests would refer to an unavailable session. In another situation, if the developer creates a hot event that overrides an existing event caught in the VXML Server root document, this error would be logged if the event occurs after the call session ended either as a result of a hangup or other event.


43


4. Some process that handles HTTP requests causes a request to be delayed so that it arrives out of order with other requests to the same session. This would only cause the error to occur if the request that was delayed was followed by a request by the H.323 Service to end the session. When the delayed request finally arrives, the session it referred to would have ended by then and the error would be logged. Note that if this delayed request was initiated from the root document, one of the previous errors listed in this appendix would occur. This error message would occur only if the delayed request was for a standard page within the application. This unusual situation would most likely occur when the system is experiencing high load. Additionally, if the H.323 Service and VXML Server were connected via the Internet as opposed to via a LAN, there could be a greater chance that an HTTP request could be delayed. Severity: Situation 1 is most likely due to an error by the administrator or developer and should never occur in a production system. Situation 2 should also be discovered during the testing and integration stage and should not occur in production, however there still exists a possibility with a bug in a load balancer or a glitch in the network setup that would cause this in a live system. Situation 3 should also be managed with proper testing that would focus on when the VoiceXML event is thrown. The last situation is the most severe as it could occur in a production system that is overloaded. Note: Since this error would most likely fall within a call, the caller would hear an error message and most likely their call would be terminated. As a result, this error does have consequences with the caller and should be considered more severe.

Action: For the first two situations, ensuring the system is properly tested before moving to production is essential. For the third situation, there should be good justification for defining a hot event that overrides default functionality. Eliminating this hot event may resolve the issue. For the final situation, it may be an indication of excessive network, operating system, or Call Server load. For the network, perform diagnostics to ensure proper operation. For the operating system, check the CPU and memory utilization to ensure they are not maxed out. And for the Call Server, check the configuration to ensure the HTTP request queue is not too small (for example, on Apache Tomcat, the acceptcount setting can be configured in the conf/server.xml configuration file). Finally, the administrator may decide to direct less load on the system. Additionally, a preferred setup would involve keeping the H.323 Service on a local network with the VXML Server machine. That way, all communication would be as fast as possible, without any pitfalls with using the Internet as the communications medium.

Session Ending within an Element Error Symptom: This error is displayed when during the process of producing a response to a browser request, the session is invalidated. This is slightly different than the previous situations. In those situations, the session was already invalidated when the browser makes the request. Here, the session is


44


invalidated while VXML Server is in the process of handling a request. This occurs within a call, precipitating an error message to be played to the caller.

Message: SERVER ERROR: An error occurred because the HttpSession has been invalidated. This could have been caused by an action element whose execution time exceeded the VoiceXML gateway's time limit for VoiceXML page requests.

Cause: There is one specific situation where this can occur under standard load. An element that takes a long time to complete, more time than the H.323 Service’s fetch timeout property (typically around 30 seconds) would hold up the response while it worked. The browser would time out the request, causing a new request to be made to end the session with an error. When the element finally completes, any subsequent work done by the VXML Server would encounter a null session and log this error. Under correct conditions, elements should execute very quickly. Likely causes for delays would be when the element is accessing external resources such as databases or, less likely, the element is performing a very complex calculation. While the error message references an action element (which is usually the preferred element to make calculations or access databases), this can occur within any kind of element, both standard and configurable. Another situation where this could occur would be if the system is under very high call volume. In this case, the operations of the system may cause normally rapid operations to slow down to non-trivial lengths of time. This, coupled with an element that performs a slow operation, could cause this error to occur. Severity: The severity of this error is high since the caller would encounter an error message during the call and could be an indication that the system is overloaded.

Action: When designing an element that has the potential for taking a long time, the developer may wish to implement mechanisms for handling a slowdown such as implementing internal timeouts, or using backup systems if the primary systems fail. If the application designer knows that an element can take a long time to run, they can consider increasing the fetch timeout VoiceXML property just for that element, write the element to run in a separate thread so the call can continue without delay, or provide audio to play while the caller is waiting. The administrator may also wish to analyze the system to see if there is excessive load on it and take measures to reduce the load if so.

Empty Error Message Symptom: This error message is displayed with no message at all other than saying there is a server error.

Message: Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

45


SERVER ERROR:

Cause: This error message appears when a Java exception that itself does not define an error message occurs. Most of the time, this is a Java NullPointerException, which defines no error message. Note that all VXML Server errors contain descriptive error messages and only very few Java exceptions have no message. Note that if this exception occurred within a custom Java class written by a developer, the Java stack trace of the exception is displayed in the error log. This stack trace provides the developer information on where this error occurred. If there is no stack trace, this could be an indication of a bug in the VXML Server, though it is possible that an incorrectly configured element or Java class could cause some object to be null that should not be, causing the error to occur within VXML Server code. Severity: This message should be considered severe because a NullPointerException only occurs when Java objects that should have a value do not.

Action: If this error occurs in a custom Java class, the developer should debug it to find out the issue by using the stack trace. If the error occurs without a stack trace, under standard call volume, the developer may first wish to debug the problem to see if there is any custom code that could be setting up data incorrectly. If the options are exhausted, this would be a situation where Cisco customer support should be contacted.

VXML Server Displays Information of an Old Version Symptom: When starting the Call Server, the VXML Server Call Services displays information of an old version.

Message: Information displayed about an older version.

Cause: A prior version than 4.0 of the VXML Server software is still installed.

Action: If a prior version of VXML Server software was installed on the same Call Server, it is likely that the old expanded Web application still resides in the Call Server. Either delete the old expanded directory or use the Call Server console to decommission the old Web application and load the new Web application archive (WAR) file.


46


VXML Server Console Displays No Information Symptom: When starting the Call Server, the console displays no information on VXML Server Call Services at all.

Message: No message displayed

Cause: This is most likely caused by not having the VXML Server Call Services Web application archive file, audium.war, in the correct directory.

Action: This could have been due to not installing VXML Server Call Services or choosing the wrong directory in the VXML Server installer for the Call Server’s Web application archive directory.

VXML Server Sample Application Displays Page Not Found Symptom: When running the Hello World sample application to test if VXML Server Call Services is running, the page could not be found.

Message: Page not found.

Cause: This URL would be valid only if the Web browser were running on the same machine as Call Services.

Action: The URL to be entered to obtain the Hello World application is: http://localhost:PORT/Audium/Server?audium_application= HelloWorld

This URL would be valid only if the Web browser were running on the same machine as Call Services. If it is a different machine, the appropriate IP address should be used instead of “localhost”. Additionally, the port on which the Call Server is listening must be used. Use the netstat command to determine the port number.


47


VXML Server Sample Application Displays Technical Difficulties Symptom: When running the Hello World sample application to test if the VXML Server Call Services is running, a VoiceXML page appears indicating that there are technical difficulties.

Message: VoiceXML pages indicate that there are technical difficulties.

Cause: Several possibilities exist for this error as outlined below.

Action: There are several possibilities that exist for this error: • The application name was not typed in exactly as it appears in the URL shown in the previous option. All application names do not have spaces • There is no HelloWorld application deployed on Call Services. Check the INSTALLATION_PATH\CallServices\applications folder. If it does not contain a folder named HelloWorld, this application has to be newly deployed from Call Studio. • When the Call Server was launched, Call Services encountered an error. See the troubleshooting options in this section for resolutions.

Excessive Unavailable ASR Errors in VXML Server Log Symptom: Excessive unavailable ASR errors in VXML Server application error log.

Message: SERVER ERROR:A VXML exception occurred:error.com.Cisco.media.resource.unavailable.asr

Cause: The default MRCP timeout value of 3 seconds needs to be increased.

Action: The default MRCP timeout values need to be increased to 5 seconds on the gateway as follows: mrcp client timeout connect 5mrcp client timeout message 5


48


VXML Server Loggers Not Functioning Symptom: When deploying a new application on a VXML Server, you receive an error message.

Message: There was a problem instantiating the logger class com.Cisco.CVP.vxml.datafeed.DatafeedLogger.

Cause: The VXML Server and the Call Server are not communicating.

Action: The application cannot be deployed successfully until the VXML Server and Call Server are communicating. Check the network connectivity between the two servers. Traceroute and Ping are some of the initial network testing utilities that are available to verify the network integrity. Be sure to test the network connectivity in both directions. Also verify that the network cards are set to the correct settings. These settings are provided in the General Systems Issues section of this document.

VXML Server Is Unreachable Symptom: The VXML Server shows as Unreachable.


Cause: An UNREACHABLE status is reported when the Operations Console Resource Manager does not get a valid number for active calls after running status script.

Action: The Operations Console bypasses the Operations Console Resource Manager on the Call Server to which the VoiceXML service is connected and directly talks to the Operations Console Resource Manager that is co-located with the VXML Server. The reason for this is that the status obtained through the central controller on the Call Server is incorrect if no calls have been placed through the VXML server. It reports and displays the status as STOPPED. The status of VXML server is obtained in a different manner than that for Reporting Server.


49


VXML Server Does Not Start After Patch Symptom: Patch installer and Explorer errors display during the application of the CVP 7.0.2 patch.

Message: "Unable to delete directory: C:\Cisco\CVP\VXMLServer\Tomcat\webapps\CVP". Please try to delete this directory manually before restarting the system.

Cause: The directory was not deleted after upgrade.

Action: • Delete the %CVP_HOME\VXMLServer\Tomcat\Webapps\CVP directory. • Restart the VXML Server.

VXML Server Fails After CVP 7.0 Upgrade Symptom: The VXML Server application fails following an upgrade to CVP 7.0.

Message: None.

Cause: Customer audio files that reside in the .war expansion directory (CVP_HOME/VXMLServer/Tomcat/webapps) are not preserved upon upgrade to CVP 7.0.

Action: Redeploy custom audio files to the VXML Server, but not to the .war expansion directory because the contents of this directory are not preserved at upgrade time.

VoiceXML Scripts Fail Symptom: VoiceXML scripts fail with a CALL_ERROR error message.


50


Message: CALL_ERROR; http://10.8.7.70:8080/CVP/Server?application=HelloWorld at line 5: Undeclared ECMAScript variable audium_vxmlLog

Cause: The Cisco Gateway is configured with the 'no http client cookie' statement.

Action: Enable the HTTP client to send and receive cookies on the Cisco Gateway when using VoiceXML applications by entering the following command: http client cookie

This command is enabled by default. You do not need to enter it unless you have previously disabled cookie support by using the 'no http client cookie' command. Use the 'show running-config' command to verify the HTTP configuration information. If the defaults are used, HTTP configuration commands are not shown. If changes have been made to the HTTP defaults, the configuration is shown. For example, the following output shows the HTTP client cookie parameter configured to a non-default value: no http client cookie

See Also For more information on the 'HTTP Client Cookie' feature, see the Cisco IOS Tcl IVR and VoiceXML Application Guide.

Call Start Element Deleted From VXML Server Symptom: Call Start element is deleted from the VXML Server application.

Message: None.

Cause: Usually user error.

Action: Add the new Page Entry element to the call flow, then right-click on it and choose Start Of Call -> Yes. This element is automatically used as the first element - start of call. You can then manually rename it to Call Start.


51


VoiceXML Application Fails to Access Another Resource Symptom: VoiceXML application fails to access another resource, for example, to play a media .wav fail, and the IOS log contains SERVER ERROR: A VoiceXML exception occurred: error.badfetch.http.-n, where n is a negative error code.

Message: SERVER ERROR: A VoiceXML exception occurred: error.badfetch.http.-n

Cause: error.badfetch.http. has a positive error code when a connection to the HTTP server is successful, for example; 200 (OK), 400 (Bad Request), or 404 (Not Found). When a gateway encounters an internal error, including server connection failure, it uses the following negative error codes to differentiate the error from a server response code: -1 Client error -2 Timeout error -3 Message decoding error -4 Message encoding error -5 Message transmission error -6 Client write queue full -7 Connection error -8 Client unsupported -9 Client socket receiving error

Action: Make sure the resource you are attempting to access is running. For example, if the VoiceXML application fails to access a .wav file from a media server, make sure that the media server's World Wide Web Server service is running. Also make sure that the path to the .wav file is correct.

Error 500: Error Initializing Velocity Symptom: In trying to access the link listed below, an error message is received. Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

52


http://:7000/CVP/Licensing

Message: Error 500: Error initializing Velocity: java.lang.Exception: Unable to configure AvalonLogSystem: java.security.AccessControlException: Access denied (java.io.FilePermission /opt/WebSphere/AppServer/profiles/AppSrv01 read)

Cause: This is due to the Use Java 2 security to restrict application access to local resources being enabled. When enabled, it prevents the VXML Server to run.

Action: Modify the setting via the WebSphere Application Server (WAS) Integrated Solutions Console. Accessing the Integrated Solutions Console 1. From a web browser enter the following link:http://:9060/ibm/console (http://:9060/ibm/console) 2. Click the Security link in the left frame after you log in. 3. Click Secure administration, applications, and infrastructure. 4. In the Java 2 security section uncheck Use Java 2 security to restrict application access to local resources. 5. Click Apply.

Keytool Error: Failed to Establish Chain from Reply Symptom: Receiving the error message.

Message: "keytool error: Failed to establish chain from reply"

Cause: This indicates that the keystore was not set up properly and keytool cannot find the a path from your certificate down to a root certificate through the intermediate certificate.

Action: Check the configuration of the keystore.


53

Chapter 2: Symptoms and Solutions Reporting Server

VXML Server Statistics Not Available Error Message Symptom: An error message from the Operations Console indicates that VXML Server statistics are not available.

Message: VXML Server statistics are not available.

Cause: VXML Server requires at least one Studio application script that enables Datafeed Logger to report statistics. This error occurs when Datafeed Logger has been removed or is missing in either of the following conditions: • The HelloWorld script that is installed and deployed by the installer does not contain the Datafeed Logger. • The Studio script(s) developed prior to Release 4.x have been upgraded and does not contain the Datafeed Logger.

Action: Add the Datafeed logger to the script(s).

Reporting Server Reporting Server Association Error Symptom: A warning popup is produced with an error message and the associated Call Servers changes are reverted.

Message: Associate Call Servers can only be changed when deploying to device.

Cause: When a user creates a new Reporting Server they are required to associate a Call Server. If neither device is created yet the user must perform a Save, instead of a Save & Deploy. If the user decides to change, or add to the list of associated Call Servers and perform a Save, a warning


54


popup is produced with the message "Associate Call Servers can only be changed when deploying to device." and the associated Call Servers changes are reverted.

Action: In order to proceed past this warning the user has two options: 1. Install the Unified CVP software and perform a Save & Deploy. 2. Delete the Reporting Server device, read it with the changes to the associated Call Server list and Save.

No Informix Services Found on Reporting Server Symptom: After installing the Reporting Server, the Informix Services with the Unified CVP database name is not found.

Message: None.

Cause: The .bat files were not run.

Action: none

Starting Informix Database Causes Client Locale Not Set Error Symptom: When starting Informix from the command prompt generates an error message.

Message: CLIENT_LOCALE is not set. The database cannot be accessed.

Cause: The database was tried to be started from the command line instead of the Services control panel.

Action:


55


Verify that the database has stopped. Go to the Services Control Panel and start the Informix service with the appended database name.

Reporting Server Not Receiving Messages From Message Bus Symptom: Reporting server does not receive the messages from the message bus. There is no indication of an error in the error log.

Message: None.

Cause 1: Check if there are other Reporting Servers that are listening on the same port.

Action 1: Open a command prompt and type: netstat -a Stop the offending Reporting Server that is listening on the same port. Once that server is stopped, restart the Reporting Server that was not receiving the messages.

Cause 2: Another possibility is that the Reporting Server was added from the Operations Console after the Call Server was started.

Action 2: Restart the Call Server and the Reporting Server.

Reports Are Slow Symptom: The reports are slow to generate.

Message: None.

Cause: The database may be very large, or report filters may be too wide.

Action:


56


Contact your database administrator. Request that the database undergo a backup followed by an off-hours purge of the database.

Queries Timing Out Under Load Symptom: Queries are timing out under load.

Message: No message displays.

Cause: This is typical response when the database is under heavy load.

Action: Consider querying the database when the database is not under heavy load. You might also try to reduce contention with active processes using the following statements: • SET LOCK MODE WAIT 30 the WAIT option causes the session to suspend the process until the lock releases or until the waiting period ends. • SET ISOLATION DIRTY READ this option instructs the session not to place locks on the data when it is attempting to read, and potentially read data ignoring existing locks.

Reporting Server Cannot Connect to Database Symptom: The Reporting Server is unable to connect to the database.

Message: None.

Cause: cvp_db user password has expired.

Action:


57


Have the Administrator reset the database password.

Reporting Server Goes into Partial Service Mode Symptom: The Reporting Server shows that it is in a partial service mode.

Message: Partial Service.

Cause: The Informix database services are not running.

Action: Go to Start > Administrative Tools > Computer Management. Click Services and Applications. Click Services. Check that the following services are started: • Informix Dynamic Server Message Service • Informix IDS - cvp_db_ Start services if they are not running.

Reporting Server Cannot Initialize JDBC Connection Symptom: The Reporting Server is unable to initialize the Java Database Connectivity (JDBC) connection.

Message: RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connection java.sql.SQLException: com.informix.asf.IfxASFException: Attempt to connect to database server (cvp_db_) failed.

Cause: The database is disconnected and is unable to reconnect possibly due to a bad password.

Action: Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

58


Check the database password from the Operations Console. From the Device Management menu select CVP Reporting Server. Select the Reporting Server, then select Database Administration -> Change User Passwords. Click Database User and change the password.

Reporting Server Is Unreachable Symptom: The Reporting Server is Unreachable.


Cause: Reporting server is UNREACHABLE in any one of the following three conditions: • Operations Console is unable to connect to the Operations Console Resource Manager that is co-located with the Call Server. For example, the Operations Console Resource Manager is down. • The Operations Console Resource Manager on the Call Server has not received a state event from the central controller for the Reporting Service. • The Operations Console Resource Manager is unable to connect to the Call Server and it is not receiving state events from the central controller. For example, the Call Server is down. The state of the Reporting Server is obtained from the Operations Console Resource Manager that is co-located with the Call Server to which the Reporting Server is connected. If Call Server1 and Reporting Server1 are two different machines and both have Operations Console Resource Managers running on them, Operations Console talks to the Operations Console Resource Manager on Call Server1 to get status of the Reporting Server because that is where the central controller reports the status of the Reporting Server.

Action: Configure the Reporting Server if the Operations Console Resource Manager on the Reporting Server is running, but Operations Console shows the status as unreachable.

Errors During Reporting Server Database Purge Symptom: If you attempt to run a database purge on the reporting server when another purge is already in progress, or if the reporting server is down, the following messages appear in the log.

Message:


59

Chapter 2: Symptoms and Solutions Audio

createClientSocket -- Couldn't get I/O streams for the socket. Error creating the client socket. Error sending a Start message to Reporting Server. Error sending an End message to Reporting Server.

Cause: The reporting server could not open a socket connection to itself, either because it was down or a purge process was in progress.

Action: No action necessary. This condition and the associated messages do not prevent the back-end task from completing the purge process.

Audio Incoming Audio-Only Calls to IVP Are Not Rejected Symptom: Incoming audio-only calls to IVP are not rejected.

Message: None.

Cause: Audio-only or basic video calls are routed to IVP. The ivp.xml configuration file on the IVP machine is not set to reject audio-only calls.

Action: Add to /home/rv/ivp/conf/ivp.xml, and then restart the IVP machine. Note: Enabling this parameter prohibits audio-only agents from performing warm consultative transfer back into IVP. Audio-only calls must be routed through the CVP Call Server.

Caller Hears Nothing but No Errors Logged Symptom:


60


The caller does not hear any voice audio, and Unified ICME and Unified CVP logs do not indicate any problems.

Message: None.

Cause: There may be a problem with explicitly appending a .wav extension to the Network VRU Script Name.

Action: In Configure ICM, remove the .wav extension from the Network VRU Script name. For example, if the name is PM,welcome.wav, change it to PM,welcome.

Caller Does Not Hear Prompt or Expected Event Does Not Occur Symptom: When a call is made, the caller does not hear a prompt or an expected event (such as a transfer) does not take place.

Message: None.

Cause 1: There has been a misconfiguration on the Unified ICME.

Action 1: Do the following: 1. In Configure ICM, under Network VRU Scripts, note the Network VRU for the Network VRU script in the ICM Run VRU Script node. 2. In Configure ICM, under Labels for the Network VRU that is running the ICM script, click Set Target, click Network VRU and select the Network VRU noted in Step 1. Note: When using VRU Types 2 or 5, try running Call Tracer.

Cause 2: An error has occurred in Unified CVP.

Action 2:


61


Do the following: 1. Use Monitor mode in the ICM Script Editor to see if the call is following the expected path (that is, it does not travel through an X branch). 2. Check the Call Server logs to see what the error or warning messages have been generated.

Caller Does not Hear Multicast Music on Hold Symptom: The system is configured to use multicast MOH when callers are placed on hold, but Caller does not hear multiocast MOH when placed on hold and the call is going through a SIP trunk . Additionally, the Unified Communications Manager Realtime Monitor indicates that mMOH is active.

Message: None

Cause: N/A

Action: Add the global command: ccm-manager music-on-hold on the Gateway.

One-Way Audio on Voice Gateway Symptom: One-way audio on a AS5350 or AS5400 voice gateway after an IP transfer.

Message: None.

Cause 1: One-way audio after a remote H.323 device sends the VoIP-Gateway a HOLD or TRANSFER message.

Action 1: Disable the voice-fastpath command on the Gateway, using the syntax: no voice-fastpath disable. Note: This command is enabled by default and the syntax is hidden.


62


Cause 2: The Gateway and Cisco Unified Communications Manager are in different subnets and no IP route defined in the Gateway.

Action 2: Ping from the Gateway to the Unified CM and also from the Unified CM to the Gateway. If there is a problem with pinging, add an IP route for the different subnet in the Gateway. For example, specify the command ip route 0.0.0.0 0.0.0.0 10.86.137.1 where 10.86.137.1 is the default routing gateway for the Gateway. This command enables routing to all subnets. After entering this command, try a ping session again. If that is successful then you should be able to listen to the two-way voice.

GetSpeech Micro-Application With Datatype "Currency" Does Not Work with Negative Currencies Symptom: The GetSpeech Micro-Application with datatype set to "currency" does not work with negative currencies.

Message: None.

Cause: The ASR-TTS server is Nuance 8.5(ASR). This version of Nuance does not support negative currencies for the VXML Server built-in "currency" grammar.

Action: Write your own grammar for dealing with currency or contact Nuance support.

A Prompt File From Media Server Did Not Run Symptom: A prompt file from Media Server did not run. Other symptoms of this issue may include: • There is an error message in the Unified CVP logs stating a problem with invalid encoding. • There is an error message in the Unified CVP logs stating the audio file has a bad header. • There is an error message in the Unified CVP logs stating that the payload type in audio file does not match the payload.


63


Message: Invalid encoding ERROR: Audio file had a bad header. ERROR: Payload type in audio file does not match the payload.

Cause 1: Codec mismatch between the Voice Browser configured codec and Audio file codec.

Action 1: Complete the following steps: 1. Convert the prompt file codec to the H.323 Service configured codec (g711Alaw64K or g711Ulaw64K). 2. Verify that the codec is using the sCodec command from VBAdmin and check the prompt file properties to verify that codec matches the VB configured codec.

Cause 2: The audio file is not a valid .wav file.

Action 2: Re-record or convert the audio file in standard .wav format.

Long Pauses between Prompts Symptom: Caller experiences long pauses between prompts.

Message: None.

Cause: There are a couple of possible causes for this issue. 1. There is a WAIT node in the ICM script. 2. There is delay in the network or one of the components in the solution.

Action:


64


Below are the possible solutions to the causes mentioned above. 1. WAIT nodes are not supported with Unified CVP. Check for WAIT nodes in the ICM script and remove them. 2. The point of latency within the network must be isolated and resolved.

Caller Hears System Error Message Symptom: A caller hears a system error message.

Message: The caller hears the default audio message that indicates a system error has occurred.

Cause: When a caller hears a system error message, this usually indicates an ICM scripting or configuration error.

Action: Complete the following steps: 1. If the caller is experiencing the error before hearing any other prompts, in Unified ICME software make sure the dialed number is associated with a call type and the call type is associated with a script. 2. In the ICM script, be certain that all external paths out of Run External Nodes have some default action (transfer to default LABEL, Queue to Skill Group, and so forth) . 3. In the ICM script, make sure that the last node in the script at the logical end of the call terminates gracefully, such as with a RELEASE or LABEL. Otherwise, the system error is played. 4. Unified CVP might have timed out waiting for a response from Unified ICME software. This is typically caused by an Application Gateway data base dip that is taking longer than the Unified CVP Call Server timeout settings.

Prompts Not Played Intermittently, Random Prompts, Random ICM Scripts Symptom: The callers are encountering problems such as prompts only being played intermittently or are not being played at all. Other issues include random prompts, and random ICM scripts.


65


Message: None.

Cause: Media file caching is not configured correctly on the gateway.

Action: If media caching is not enabled on the gateway, the gateway is forced to load every prompt for each call. This can sometimes create a situation that overloads the gateway and can cause prompt play failures. For procedures on how to configure gateway caching, refer to the Configuration and Administration Guide for Cisco Unified Customer Voice Portal.

No Ring Tone When Transfer to Unified CCE Agent Symptom: No ring tone is heard when transferred to Unified Contact Center Enterprise agent.

Message: None

Cause: Cisco Unified CM has an incorrectly configured service parameter.

Action: In CCMAdmin > Service > ServiceParameters > Unified CM, check the value of H323 parameter Send H225 User Info Message. It must be set to User Info for Ring Back Tone.

Garbled Prompts or Voice Symptom: Garbled prompts or voice

Message: N/A

Cause 1: There is a PIX firewall between the gateway and Unified CVP or between gateway and Unified CM.

Action 1:


66


PIX firewalls can sometimes cause issues where the prompts or voice communications become garbled. In an attempt to determine if the firewall is the issue, try to temporarily circumvent the firewall to see if that resolves the issue. If it does appear to be a firewall issue, then review PIX Firewall documentation for additional information.

Cause 2: Another possibility is that there is a latency issue in the network that is causing IP phone jitter or bad quality prompts.

Action 2: The network should be analyzed to determine if jitter conditions are present. If jitter is detected, then the IP packets are getting delayed somewhere on the network.

Caller Hears Prompt then Dead Air, Fast Busy, or Pause, then Disconnected Symptom: Caller hears prompt and then dead air, or fast busy, or a pause, and is then disconnected.

Message: None

Cause: If this is Comprehensive or VRU-Only, and the last ICM script node executed was a PM (Play Media), a dummy dial-peer and translation rule may be missing from the gateway config.

Action: Play Media VXML contains a "dummy" transfer to 987654 that is meant to fail. However, it is possible that the gateway contains a dial-peer that tries to send a call to 987654. Add the following lines exactly as they appear below to the gateway to prevent that dial-peer match from occurring. Note: 987654 is the actual number you type. voice translation-rule 1 rule 1 /987654/ // ! voice translation-profile block translate called 1 ! dial-peer voice 987654 voip


67


translation-profile incoming block incoming called-number 987654

Apply the translation-profile incoming block to all VoIP dial-peers

One-way Voice after Agent Transfer Symptom: When a phone call is established from an IP station through a Cisco IOS voice gateway or router, and the agent transfers the call, only one of the parties receives audio (one-way communication).

Message: None.

Cause: No IP connectivity between the phone and the gateway.

Action: To summarize the more common ones: • Check Basic IP Routing: Basic IP reachability should always be checked first. As RTP streams are connectionless (transported over UDP), traffic may travel successfully in one direction, but get lost in the opposite direction. • Enable IP Routing: Make sure your router has IP routing enabled, that is, does not have the global configuration command "no IP routing". To enable IP routing, simply type the following global configuration command in your Cisco IOS gateway: gateway(config)# conf t gateway(config)# ip routing

• Cut-through Two Way Audio Early: In some cases it is necessary to establish a two-way audio path as soon as the RTP channel is opened. In other words, before the connect message is received. To achieve this, type the voice rtp send-recv global configuration command: gateway(config)# conf t gateway(config)# voice rtp send-recv

• Disable voice-fastpath: The Cisco IOS command voice-fastpath enable is a hidden global configuration command for the AS5350 and AS5400, which is enabled by default. To disable it, use: gateway(config)# conf t


68


gateway(config)#no voice-fastpath enable.

See Also For a more comprehensive list of possible solutions, go to: http://www.Cisco.com/en/US/tech/ tk652/tk698/technologies_tech_note09186a008009484b.shtml

Micro-Application May Not Always Create Correct VoiceXML Symptom: The execution of Play Data (PD) Micro-Application may not always create VoiceXML as per the current VoiceXML and SSML built-in grammar types. This may lead to variations on expected spoken results when using TTS.

Message: The TTS server bases its rendering of speech interpretation based on how we output these values, and it may not produce the exact output we are predicting or desire.

Cause: Configuration not in alignment with VoiceXML standards.

Action: Change the say-as element and the interpret-as attribute to align them to the following standards in order to hear the desired output: Voice Extensible Markup Language (VoiceXML) Version 2.0 (Specifically Appendix P): http:/ /www.w3.org/TR/voicexml20/#dmlABuiltins Speech Synthesis Markup Language (SSML) 1.0:http://www.w3.org/TR/speech-synthesis/ Speech Recognition Grammar Specification (SRGS) 1.0: http://www.w3.org/TR/ speech-grammar/ Semantic Interpretation for Speech Recognition (SISR) - W3C Working Draft 1 April 2003: http://www.w3.org/TR/2003/WD-semantic-interpretation-20030401/

Large Number of No Matches for Speech Recognition Symptom: There are a large number of no matches for speech recognition.

Message: None


69


Cause: Speech recognition confidence threshold is set too high.

Action: When the Speech recognition server recognizes voice input, it determines a confidence score that signifies how confident the server is that it correctly matched caller input. This confidence score is anywhere from 0 to 1.0 where 1.0 is 100% confidence that the grammar match is correct. By default, Unified CVP tells the Speech recognition server to throw an invalid entry error for all confidences below 0.7 or 70% for "GetDigits" and "Menu" and 0.5 or 50% for "GetSpeech". For some applications, this value is too high and results in false invalid entry errors. Consult the ASR vendor to determine if the default confidence thresholds used by Unified CVP (0.7 for GetDigits and Menu, 0.5 for GetSpeech) is incorrect for your applications. If the default values are indeed incorrect and need to be changed, the Unified CVP templates need to be adjusted. Contact your support contact for instructions on template modification.

Same Prompt Repeats Multiple Times Symptom: Same Prompt Repeats Multiple Times

Message: None

Cause: When using ASR/TTS, this symptom is caused when the gateway cannot connect to the ASR or TTS server.

Action: In Operations Console > Call Server > IVR tab there is a field called Number of tries for ASR/TTS server. The default is zero. If the gateway cannot connect to the ASR or TTS server, the value configured determines the number of times it tries to connect. When that happens, the prompt repeats on each subsequent try. Also, on the gateway the customer can specify a backup ASR/TTS server (look for “ip host asr-en-us-backup” or “tts-en-us-backup” in the gateway ‘show run’ config). If specified, the gateway tries the backup server it uses the value configured to determine the number of times it tries to connect. The end result is a condition where the caller hears the prompt repeat if the value is greater than zero. The solution is to resolve the connection to the ASR/TTS server.

Cannot Deploy Newly Added Speech Server Symptom:


70


An error messages is received after the Unified CVP Operations Console Server, and the Operations Console Resource Manager on a Nuance Speech Server was installed, and added the Speech Server in Operations Console and saved configuration.

Message: " device with IP Address: and Hostname: operation failed. Device could not be reached. The device may have multiple NIC cards enabled. To resolve the issue please keep only one NIC card with required IP address enabled. In addition to the NIC cards change also restart the Resource Manager service on the device."

Cause: In some instances, when starting the RMI Server, the RMI server remote stubs contain the IP address of the local host (127.0.0.1) instead of the real IP address. Since the 'Resource Manager service' on each machine uses JMX (RMI Server), in the situation where remote stubs containing local host (127.0.0.1) the Operations Console Server cannot communicate with that specific 'Resource Manager' service. It is because the Operations Console Server uses the real IP address to communicate with JMX (RMI) Server in the Resource Manager Service where as remote stubs contain local host ip address, so the connection fails.

Action: As indicated in the error message that is displayed in the user interface (UI), perform the steps below: 1. Check the server where Resource Manager service is running to determine if it has multiple network interface cards (NIC) enabled. If so, disable all of the NIC cards, except the one that has the required IP address for Unified CVP communication. 2. Open the Services control panel, and locate the Unified CVP Resource Manager service. 3. Stop the service. 4. After the service has stopped, start the service.

SpeechWorks Does Not Work with Unified CVP Symptom: ScanSoft OSSServer.cfg, and Baseline.xml files require changes.

Message: The default configuration for SpeechWorks Media Server (SWMS) 3.19, OpenSpeech Recognizer (OSR) 3.09, RealSpeak 4.0.10 from ScanSoft requires changes to work with Unified CVP.

Cause: Default configuration still in place.


71


Action: The default configuration for SpeechWorks Media Server (SWMS) 3.19, OpenSpeech Recognizer (OSR) 3.09, RealSpeak 4.0.10 from ScanSoft requires changes to work with Unified CVP. The following parameters need to be set: OSSServer.cfg: The OSSServer.cfg file is for MRCP server and webserver application configuration. On a Scansoft Server it can be found at the following path location: C:\Program Files\SpeechWorks\MediaServer\server\config

The Name, Type and Value entries should be configured as shown below. Values should be changed, lines un-commented or entries added in order to have the OSSServer.cfg file reflect these entries. Be careful to search for each one first because most exist already and just need changes to the "Value". server.session.ossrec.useWidecharStringGrammers VXIInteger 0 server.transport.port VXIInteger 554 server.transport.dtmfPayloadType VXIInteger 101 server.session.ossweb.machine VXIString server.session.mrcpdefaults.nbest_list_length VXIString 1 server.session.mrcpdefaults.no-input-timeout VXIString 10000 server.session.mrcpdefaults.completetimeout VXIString 1000 server.session.mrcpdefaults.incompletetimeout VXIString 1000 server.session.mrcpdefaults.fetch-timeout VXIString 5000 server.transport.dtmfTriggerLeading VXIInteger 0 server.transport.dtmfTrailingEdgeTimeout VXIInteger 2000 ## All DiagTag values should be set to "0" to disable debug and improve performance. server.log.diagTag.2000 VXIInteger 0 server.log.diagTag.2001 VXIInteger 0 server.log.diagTag.3000 VXIInteger 0 server.log.diagTag.3001 VXIInteger 0 server.log.diagTag.6000 VXIInteger 0


72


server.log.diagTag.6001 VXIInteger 0 server.log.diagTag.8000 VXIInteger 0 server.log.diagTag.9000 VXIInteger 0 server.log.diagTag.10000 VXIInteger 0 server.log.diagTag.10001 VXIInteger 0 server.log.diagTag.79999 VXIInteger 0

Baseline.xml: In the Baseline.xml file, there is a parameter called, swirec_extra_nbest_keys that specifies which ScanSoft keys to return. The default setting of this parameter: SWI_meaning SWI_literal SWI_grammarName Below are the results of the default code: Code: reebar reebar mygrammar {SWI_literal:reebar} The following are changes that need to be made to the Baseline.xml file to in order to remove grammar keys in the XML result. Below are the required changes to the Program Files\SpeechWorks\OpenSpeech Recognizer\config\Baseline.xml. 1. Open the Baseline.xml file. 2. Change the swirec_extra_nbest_keys parameter from: param name="swirec_extra_nbest_keys">


73


SWI_meaning SWI_literal SWI_grammarName

to: param name="swirec_extra_nbest_keys">

3. Save the Baseline.xml file. 4. Restart the server. 5. With the changes in place, the XML result will now be: Code: reebar

Hearing English System Critical Message When Locale Not Set to en-us Symptom: The caller is hearing English system critical media when the locale is set to something other than en-us.

Message: None.

Cause: The language specific prompts were not copied to the H.323 Service.

Action: You need to copy the language specific prompts to the H.323 Service.


74


Speech and DTMF Keystroke Recognition Take Long Time to Complete Symptom: Any speech and DTMF keystroke recognition seem to take a long time to complete, and initial words or keystrokes are missed.

Message: None.

Cause: Using M, GD or GS Micro-Applications with input_type set to 'B' (both).

Action: Virus scanning can severely degrade recognition speed if you have it running in your recognition servers. Check with your recognition service provider for their input on how to configure virus scanning for maximum protection and minimum performance impact.

Speech Recognition Does Not Recognize Wide Character String Grammars Symptom: Scansoft Speech Recognition Server does not recognize wide character string grammars.

Message: None.

Cause: ScanSoft server not configured for Wide Character String Grammars.

Action: Set the following: In the \ScanSoft\SpeechWorks OpenSpeech Media Server\Server\config directory, modify the OSSserver.cfg file to add server.session.ossrec.useWidecharStringGrammars VXIInteger 0

In the OSSserver.cfg, set: server.transport.dtmfPayloadType VXIInteger 101


75


Large Number of Invalid Entries for Speech Recognition Symptom: Large number of invalid entries for speech recognition

Message: None.

Cause: Unified CM has an incorrectly configured service parameter.

Action: When the Speech recognition server recognizes voice input, it determines a confidence score that signifies how confident the server is that it correctly matched caller input. This confidence score is anywhere from 0 to 1.0 where 1.0 is 100% confidence that the grammar match is correct. By default, Customer Voice Portal tells the Speech recognition server to throw an invalid entry error for all confidences below 0.7 or 70% for GetDigits and Menu and 0.5 or 50% for GetSpeech. For some applications, this value is too high and results in false invalid entry errors. Consult the ASR vender to determine if the default confidence thresholds used by Customer Voice Portal (0.7 for GetDigits and Menu, 0.5 for GetSpeech) is incorrect for your applications. If the default values are indeed incorrect and need to be changed, the Customer Voice Portal templates needs to be adjusted. Contact Cisco TAC for instructions on template modification.

DTMF Barge-in Not Detected Symptom: Start of speech not flagged in Scansoft configuration file causing DTMF barge-in not to be detected.

Message: None.

Cause: Change made to Scansoft OSSserver.cfg file under ScanSoft\SpeechWorks OpenSpeech Media Server\Server\config.

Action: Set the following: server.session.ossrec.mrcp.startOfSpeechOnDTMF VXIInteger 1


76

Chapter 2: Symptoms and Solutions Transfer and Connect

Clicking Noise Heard During Scansoft TTS on IP Phones Symptom: A clicking noise is heard during Scansoft TTS on IP phones.

Message: None.

Cause: Change made to Scansoft OSSserver.cfg file under ScanSoft\SpeechWorks OpenSpeech Media Server\Server\config.

Action: Set the following: server.transport.audio.playerPacketDataSize VXIInteger 160

Transfer and Connect Agent Does Consult Transfer to CVP and Completes Consult While Connected to IVR Symptom: The agent does a consult transfer to CVP and then completes the consult while still connected to the IVR, instead of waiting until finally being connected to the agent.

Message: Cause: To avoid this issue, do a single step transfer instead of a consult transfer to get the caller transfered to the CVP IVR. In the event you still want to use a consult transfer into CVP for some reason (for example, an agent desktop is customized for only one flavor of transfer), you can create a separate SIP trunk that has MTP enabled. This SIP trunk will service the route patterns destined for CVP. The DNIS you are using is different than the warm transfer DNIS, so there should be no problem using separate SIP trunks. It happens for SIP only (H323 is fine). Future enhancement to the VXML gateway will support reinvitations on the IVR leg, in which case MTP will not be required on this particular call flow.

Action:


77


The set up includes adding an additional SIP trunk with MTP, allowing MTP to be allocated only on the warm transfer legs. 1) Create the secondary SIP trunk, call it "Outbound SIP trunk-Warm Xfers" to differentiate it with the incoming SIP trunk that handles calls from CUPS or CVP. 2) Apply MTP and point it to the CVP IP address. 3) Create the Route Pattern for the VRU label and set the outbound sip trunk.

Agent Available and Transfer Does Not Happen Symptom: When an agent becomes available during prompt-play in a Queue to Skill Group script, the transfer never happens.

Message: None.

Cause: The Interruptible checkbox in a Network VRU Script configuration is not checked. (This is the default setting). The Unified ICME software does not interrupt a VRU script to perform a transfer if this feature is not enabled.

Action: Select the Interruptible checkbox on all Network VRU script nodes that might be run while a caller is waiting in queue for an agent.

Problem with IP Transfer Symptom: Problem with IP Transfer.

Message: None.

Cause 1: Unified CM is not registered with the Gatekeeper because the Gatekeeper information in Unified CM is incorrect.

Action 1: Change the Gatekeeper host name used during configuration to an IP address.

Cause 2:


78


Unified CM is not registered with the Gatekeeper because the Gatekeeper information in Unified CM is incorrect.

Action 2: Do the following: 1. In ccmadmin, select Device > Gatekeeper. 2. Add the Gatekeeper Device Information and select gateway as the Terminal Type. 3. Click Update, then Reset Gatekeeper. 4. Finally click, Reset to apply the changes.

Cause 3: Unified CM in not registered with Gatekeeper because the Reset and Restart buttons was not clicked after the Gatekeeper information was updated.

Action 3: Do the following: 1. In Operations Console, select Device > Gatekeeper. 2. Add the Gatekeeper Device Information and select gateway as the Terminal Type. 3. Click Update, then Reset Gatekeeper. 4. Click Reset to apply the changes.

Cause 4: Unified CM in not registered with Gatekeeper because a Gatekeeper registration port is not explicitly configured in Gatekeeper.

Action 4: Do the following: 1. Check the Gatekeeper configuration to verify that port 1719 is explicitly defined for the local Gatekeeper zone. For example: zone local gk1 Cisco.com 10.86.137.146 1719. 2. Verify the Unified CM registration in Gatekeeper by entering the show Gatekeeper endpoints command in the Gatekeeper CLI.

Cause 5:


79


If the Unified CM is an egress Gateway, the Voice Browser is not defined as a Gateway in the Unified CM configuration.

Action 5: If the H.323 Service displays the message: ERROR: IP Transfer - Destination endpoint at IP Address refused connection for phone number with reason code Unreachable Destination (where: is the Unified CM IP address and is the label where you want the call to be transferred), do the following:1.2. 1. In Operations Console on the Unified CM machine by selecting Device > Gateway. 2. Click Find and verify that there is an entry and is configured.

Cause 6: The call is getting routed elsewhere.

Action 6: If the H.323 Service displays the message: ERROR: IP Transfer - Destination endpoint at IP Address refused connection for phone number with reason code Unreachable Destination (where is not the Unified CM IP address that’s expected and is the label where you want the call to be transferred), then there is a problem with Gatekeeper routing. From the Gatekeeper CLI, enter show Gatekeeper gw-type-prefix and examine the Gatekeeper routing table.

Agent Cannot Transfer Calls to Unified CVP or Another Agent Symptom: Agent cannot transfer calls back to the Unified CVP or to another agent.

Message: None.

Cause: The Unified CM SingleStepTransferViaRedirect setting is incorrect. Other symptoms: The OPCTest list indicates that the initial call has not cleared; the Agent Desktop displays the initial call.

Action: In the Unified CM, modify the SingleStepTransferViaRedirect setting to 1.


80


Agent Cannot Perform a Consult Transfer and Conference to a Second Agent Symptom: An agent cannot perform a consult transfer and conference to a second agent if NetworkTransferEnabled is set to 1.

Message: None.

Cause: The Unified ICME router cannot distinguish between an agent pressing the consult button vs. the blind transfer button. Therefore, for Unified ICME, the routing behavior is the same whether the agent types "1234" or presses consult or blind transfer.

Action: Uncheck the Network Transfer Preferred box on the Unified CM PIM in PG Explorer. • Set NetworkTransferEnabled to 1 if the agent is available, or to 0 if the agent is not available. • If the agent is not available, they still need a Translation Route to VRU node to make a call queue in the type 2 VRU.

When a Call Is IP Transferred, the Caller Is Disconnected Symptom: When a call is IP Transferred, the caller is disconnected.

Message: None.

Cause 1: Gateway and Gatekeeper cannot find each other on the network.

Action 1: Make sure that the Gateway can ping the Gatekeeper and vice versa: ping from Gateway ping from Gatekeeper


81


Make sure that Gateway and Gatekeeper can resolve each other’s names to IP address.

Cause 2: Configuration on the Gateway might be incorrect.

Action 2: Verify that the Gateway configuration is correct. Turn on debug in the Gateway with the following commands: debug h225 asn1 debug cch323 ras term mon

If the Gateway is configured properly, it should send a RAS message to register with Gatekeeper. Note: Turn off debug in the Gateway using the following command: no debug all

A Call Placed from an Analog Phone on an FXS Port Fails Symptom: A call placed from an analog phone on an FXS port does not appear to trigger the new_call.vxml application.

Message: None.

Cause: There is no application session command defined on the POTS dial-peer for the telephone’s voice port. Note: This command is necessary to enable phones on FXS ports to run a VXML Server script.

Action: On the POTS dial-peer, model your configuration after the following: dial-peer voice 1000 pots application session destination-pattern 12016881000 port 1/0/0


82


Calls Dropped When a Caller Tries to Invoke Supplementary Services Symptom: Calls are dropped when a caller tries to invoke supplementary services like Call Hold, consult or conference.

Message: ERROR: Supplementary service invoked by Caller. This feature is not supported.

Cause: Caller is trying to invoke supplementary services like Call Hold, consult or conference. This is an unsupported feature.

Action: On Unified CM (with supported Unified CVP version), turn on MTP (Media Termination Point).

Call is Disconnected during IP Transfer and the Gatekeeper Is Configured Symptom: The call is disconnected during IP transfer and the Gatekeeper is configured.

Message: None.

Cause: The configured H.323 Gatekeeper is not running or Gatekeeper is configured incorrect.

Action: Follow these steps: 1. Make sure the H.323 Gatekeeper is running on the IP address configured in the Unified CVP H.323 Service. Verify that the correct IP address (and not host name) of H.323 Gatekeeper is configured by using the sGK command from Voice Browser Admin. 2. Verify that the IP network connectivity between the Unified CVP H.323 Service and H.323 Gatekeeper.


83


Agent Desktop Gets a "Call Failed Due to Network Interruption" Error Symptom: Agent desktop gets an error message stating, “call failed due to network interruption.” Unified CM trace log contains the message, “Calls are dropped in H225 with a CV=41 which shows a ‘temporary failure’.”

Message: Call failed due to network interruption.

Cause: IP Security policies on your H.323 Service or Call Server machines can prevent connections, thereby causing calls to fail.

Action: View the current IP Security policy on the machine on which the Call Server is installed: 1. Select Start--> Run--> mmc-Console. 2. Click Add/Remove Snap-in. 3. Click Add. 4. ClickIP Security Policy Management. 5. Click Finish and OK. 6. In the left pane, click IP Security Policies on Local Machine. 7. Either disable the policy or edit the policy to allow the subnet from which calls are trying to connect.

Call Back For IVR Service Treatment or Subsequent Network Transfer Fails Symptom: Taking a call back for IVR Service treatment or a subsequent network transfer does not work.

Message: None.

Cause 1: The VRU PIM that connects to the Unified CVP is not set to Type 5.


84


Action 1: In Unified ICME Configuration Manager, Network VRU Explorer, change the Type to 5.

Cause 2: The ICM script does not include a Set node setting the Call.NetworkTransferEnabled variable to 1.

Action 2: In any script that might perform VRU Network Transfer or Takeback for IVR Service treatment, add a Set node at the beginning of the script setting the Call.NetworkTransferEnabled variable to 1.

Call Disconnected As Soon As Agent Answers Symptom: Call disconnected as soon as agent answers.

Message: N/A

Cause: There is a codec incompatibility between the phone and the codec capabilities that Unified CVP is sending.

Action: Do the following: 1. Find the Region in which the phone resides. In CCMAdmin, check the Device Pool the phone is in and the Region the Device Pool is in. 2. In VBAdmin, do a ‘ShowLocationsBasedCAC’. –

If set to onGateway, look at the device pool (and subsequently Region) for the originating gateway.

–

If set to offGateway, look at the device pool (and subsequently Region) for the Unified CVP H.323 Service.

3. You now have two regions: the phone region and the originating region. In CCMAdmin > System > Region, check what codec is to be used between those two regions. Both caller and agent need to be able to speak that codec. If they cannot, a transcoding resource needs to be allocated on Unified CM.


85


4. If the originating endpoint is an IOS gateway, most probably the codec is incorrectly configured. IOS gateways can speak both g711 and g729. On the gateway: –

conf t

–

voice class codec 1 codec preference 1 g711ulaw codec preference 2 g729r8 Note: For outside of North America, it is typically g711alaw.

–

Assign that voice-class codec to the Unified CVP voip dial-peer

5. If the originating endpoint is an IP phone (or a call initiated from Cisco Unified Outbound Agent), codec renegotiation is not possible. Either the device region needs to be changed or a transcoding resource needs to be allocated in Unified CM.

Call Immediately Disconnected and Caller Hears Nothing Symptom: Caller is immediately disconnected and does not hear anything.

Message: None.

Cause: There is a gateway misconfiguration on the IVR Service leg of the call.

Action: If the call is clearly getting to the ICM script, but failing on the ‘Send to VRU’ node or on the first Run External node if there is no explicit ‘Send to VRU’ node, then the gateway configuration should be checked.

Agent Goes Ready, Queued Caller Never Delivered to Agent Symptom: Agent is ready but caller in queue never delivered to agent

Message: None Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

86


Cause 1: The Interruptible box on the queuing Network VRU script is not checked.

Action 1: In the Unified ICME Network VRU Script Explorer, look at the Network VRU script that is queuing the caller. It must be marked Interruptible for the Unified ICME software to be able to interrupt the script and deliver the caller to the available agent.

Cause 2: The agent Device Target does not have a label associated with the Unified CVP routing client.

Action 2: In ICM Device Target explorer, look at the labels for the agent device target. The label must be associated with the Unified CVP routing client. If is not associated with a Unified CVP routing client, then one must be added.

Cause 3: The NetworkTransferEnabled flag has not been set to 1 in the ICM script.

Action 3: At the beginning of the first Unified ICME caller entry script, and on every post-route script that may be invoked later in the call, the NetworkTransferEnabled variable must be set to the value of 1.

Cause 4: The Gatekeeper routing is not configured properly and the Unified CVP H.323 Service cannot find the correct Unified CM to which to send the agent transfer.

Action 4: When the H.323 Service gets a device target label from the Unified ICME, it must look up that label on the Gatekeeper in order to know to what IP address to send the call. Configure the Gatekeeper with the correct routing information to resolve this issue.

Post-Route Transfers Do Not Work Symptom: Post-route transfers do not work.

Message: None.


87


Cause: There is an Unified ICME configuration error.

Action: • Make sure the following are defined for the post-route number the agent is transferring to: – Dialed number plan – dialed number – call type – script • Verify that post-routing is enabled on the Unified CVP Peripheral Gateway. Open the PG Explorer utility and check under the Peripheral tab. • Make sure NetworkTransferEnabled variable is set to 1 in the ICM script that occurred before the post routing script. It is important to note that the NetworkTransferEnabled variable applies to future transfers, so it must be set in the script before the post-route. If the agent is going to do more post-routes, it must also be set at the beginning of the current post-route script. • The transfer label that is returned in the post-route script must be defined in the Unified CVP routing client.

No Ringback on Blind Transfer Symptom: Caller does not hear ringing once an agent performs a blind transfer call to a second agent. At that point, the caller only hears silence.

Message: None

Cause: Caller is transferred to ICM script and hears a prompt. The caller is transferred to an agent and that agent performs a Single Step Transfer to another agent. At that point, the first agent is out of the call, and the caller does not hear any ringing.

Action: In CCMAdmin under Service > ServiceParameters, select your server and the Unified CM Service. Then under H323 Parameters, set "Send H225 UserInfo Message" to "H225 Info for Call Progress Tone".


88


RONA and Router Requery Not Working as Expected Symptom: Ring On No Answer (RONA) and router requery are not working as expected.

Message: None.

Cause: RONA timers are not set correctly.

Action: Check the RONA timers and reconfigure times. There are three system timers that affect RONA behavior: • Unified CVP - In VBAdmin, SetRNATimeout and ShowRNATimout. H.323 Service must be restarted for a setting change to take effect. • Agent Desktop settings Ring no Answer Time. • Unified CM Forward No Answer Timer

Call Disconnected as Soon as Agent Answers Symptom: As soon as the agent answers, the call drops.

Message: None

Cause: There is a codec incompatibility between the phone and the codec capabilities that Unified CVP is sending.

Action: Do the following: 1. Find the Region in which the phone resides. In CCMAdmin, check the Device Pool the phone is in and the Region the Device Pool is in.


89


2. In VBAdmin, do a ‘ShowLocationsBasedCAC’. –

If set to on, look at the device pool and Region for the originating gateway.

–

If set to off, look at the device pool and Region for the Unified CVP H.323 Service.

3. There are two regions: the phone region and the originating region. In CCMAdmin >System > Region, check what codec is being used between those two regions. Both caller and agent need to be using the same codec. If are not, then a transcoding resource needs to be allocated on Unified CM. 4. If the originating endpoint is an IOS gateway, the codec may be incorrectly configured. IOS gateways can be configured for both g711 and g729. On the gateway: –

type: conf t

–

type: voice class codec 1 codec preference 1 g711ulaw codec preference 2 g729r8

–

Assign that voice-class codec to the Unified CVP voip dial-peer

5. If the originating endpoint is an IP phone or a call initiated from Cisco Unified Outbound Agent, then codec renegotiation is not possible. Either the device region needs to be changed or a transcoding resource needs to be allocated in Unified CM.

DTMF Tones not Recognized Symptom: Dual-Tone Multi-Frequency (DTMF) tones are not recognized.

Message: None

Cause 1: If using ASR, and the caller is coming from the PSTN, the gateway dial-peers may be incorrectly configured.

Action 1: Possible Solution: In Unified CVP, there are two VoIP dial-peers involved in the call: One to go from the gateway to the Unified CVP H.323 Service and one to go from the Unified CVP H.323 Service to the IVR gateway. Each of these dial-peers must have the following attributes to be recognizabled correctly by DTMF:


90

Chapter 2: Symptoms and Solutions Unified CM

• dial-peer voice 800 voip • destination-pattern 8......... • session target ipv4:10.64.228.100 • dtmf-relay rtp-nte h245-signal h245-alphanumeric • codec g711ulaw no vad

Cause 2: If using Scansoft ASR, the DTMF payload type is incorrectly configured in Scansoft.

Action 2: In Scansoft config file C:\Program Files\ScanSoft\SpeechWorks OpenSpeech Media Server\Server\config\OssServer.cfg, change the following value from 96 to 101: server.transport.dtmfPayloadType VXIInteger 101

Unified CM ASP Error Occurs on Unified CM Symptom: An .asp error occurs on the Unified CM and it does not function normally.

Message: None.

Cause: If the Call Server is installed on the same machine as the Unified CM, and you uninstall the Call Server, some files the Unified CM needs might be deleted.

Action: Reinstall Unified CM.

Unified ICME Unified ICME Lookup Requests Show 0 Symptom: Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

91

Chapter 2: Symptoms and Solutions Unified ICME

When reviewing the Unified ICME Lookup Requests in the Realtime Statistics area of the Operations Console, the value typically reports zero requests.

Message: ICM Lookup Requests = 0

Cause: This request is too transient, unlike that of some of the other statistics, such as Active Calls. In a 60 second call the ICM Lookup Request may only last a few hundred milliseconds or less. The probability of catching a real time snapshot of this event is extremely low. As a result, the reported value for that statistic will typically be zero.

Action: No action required. The value will show zero most of the time for the Realtime Statistic of ICM Lookup Requests.

Unified ICME Timeout Occurs and No Error Found Symptom: In a NAM/CICM environment, an Unified ICME timeout occurs and there is no indication of an error in the Call Server logs.

Message: None.

Cause: If the time to retrieve the first script request (or connect) takes longer than the NAM’s remote ICM timeouts, then default routing occurs.

Action: Locate and correct any network or script delay, or increase the NAM remote ICM timeouts.

Call Receives Default Route Label From Unified ICME Symptom: A new call arriving at the Customer Voice Portal receives a default route label from the Unified ICME. You do not see an Call Server error for this type of ICM timeout.

Message: None.


92


Cause: The Unified ICME is taking a long time to handle new call requests.

Action: Ensure there are no wait nodes in the ICM script prior to the Run VRU Script node. For more information, see the Cisco ICM Software Scripting and Media Routing Guide.

ICM Script Fails on Run VRU Script Node Symptom: The ICM script fails on a Run VRU Script node and there is no indication of a failure in the Call Server log.

Message: None.

Cause: The Dialed Number, Network VRU Script or Network VRU might be associated with the wrong Customer.

Action: Use Configure ICM to ensure that the: • Dialed Number and Network VRU Script are associated with the correct Customer. • Customer is associated with the correct Network VRU.

Running Out of ECC Variable Space Symptom: Running out of ECC variable space.

Message: None.

Cause: IVR Service scripts are complex and require many ECC variables.

Action:


93


Some amount of ECC variable space can be freed up by doing the following: • As of Unified ICME 6.0 SR1, there are a number of Unified CVP ECC variables that are no longer needed. You can save about 170 bytes by deleting them from your Unified ICME configuration. They are: – user.num.steps.executed – user.session.handled – user.task.id – user.connect.script.config – user.connect.script.name – user.connection.type – user.last.redirected.address In addition, if you have Unified CVP ECC variables which you are not using and have set them to a length of 1, they can now be eliminated as well, as of Unified ICME 6.0 SR1. That may save a few more bytes. • You may want to switch to VXML Server scripting. All of the ECC-heavy IVR Service speech-enabled scripting that would normally require lots of ECC variables can now be confined to the VXML Server.

VRU Script Returns Incorrect Result Symptom: When prompted, the caller enters digits, but the script returns incorrect results.

Message: None.

Cause: One of the most likely possibilities is that this is a scripting issue. For example, setting user.microapp.play_data = 167.8900, but when the audio is played, customer hears "one hundred sixty seven point eight nine". The "zero zero" from the value of 167.8900 is missing. This is because of missing double quote signs to make the passed data "167.8900" as a string. Without these double quote signs around the value, it is treated as a number instead of a string.

Action: Add the double quote signs around the Micro-Application value.


94


Incorrect Number for Play Data Micro-applications Symptom: If you enter an integer value that has a max value of 0x7fffffff (2147483647) in the ICM script it does not play the required prompt.

Message: 33: 10.77.60.174: Apr 20 2007 16:50:43.062 +0530: %CVP_4_0_IVR-3-MICROAPP_FORMAT_ERROR: [VARIABLE_DATA(13)] CALLGUID=0EB8F53B-10000112-69665ACE-0A4D3CAE: [VARIABLE_DATA(13)] A number value is expected in '-1e+006' {VRUScriptName: PD,Number,8 ConfigParam: 9,Y } [id:3016]"

Cause: Large numbers must be used as strings (enclosed in quotes) in ICM scripts used by Play Data Micro-applications.

Action: If you are using integers that are larger than nine digits, enclose the value in quotation marks, so it will be treated as a string.

Caller Is Restarted at Beginning of ICM Script Symptom: Caller experiences a situation where they are restarted at beginning of an ICM script.

Message: None.

Cause: The Call Restart feature is enabled in the Unified CVP H.323 Service.

Action: There is an optional feature which can be enabled in the Unified CVP H.323 Service that restarts the caller at the beginning of the ICM script in lieu of disconnecting the caller. Normally, without this optional feature enabled, when some critical system error occurs, the caller hears the system error message and is disconnected. This feature is enabled by doing SetNewCallOnly on in VBAdmin. Do a ShowNewCallOnly in VBAdmin to see if the feature is enabled.


95


Numeric data Is not Playing in the Expected Format Symptom: Numeric data is not playing in the expected format.

Message: None.

Cause: The ICM script might be: • Truncating values with leading 0’s or 0’s after decimals. • Rounding values.

Action: Put quotes around numbers in a Script Editor Set node so it is processed as a string. This is especially important if: • Leading 0’s are present (example: dates). • Trailing 0’s are present after a decimal point (example: currency). • The number is very large (example: a number normally expressed through exponential notation).

Unified ICME Service Control Only Displays H.323 Service After Core Software Install Symptom: After rebooting the Call Server, the Unified CVP H.323 Service, and Unified CVP Resource Manager should all start automatically. The issue is that only the H.323 Service is listed in the ICM Service Control.

Message: None

Cause: The Unified ICME service control is for the legacy purposes that the Unified CVP H.323 Service carries.


96


Action: The user is required to check the All checkbox in the lower left corner of the Unified ICME Service Control panel in order to see the Unified CVP Call Server and Unified CVP Resource Manager services. All three are not displayed by default for the Unified ICME Service Control list. Unified CVP Services should be started and stopped using the Operations Console.

Error Code 31, 32, 33 Returned to ICM Script Symptom: CALL_RESULT returns an error code 31, 32, or 33 to the ICM script.

Message: Error code 31, 32, or 33 received.

Cause: If using ASR/TTS, the gateway is unable to connect to the ASR or the TTS server.

Action: To resolve this issue, the connection to the server must be restored.

Error Code 9 Returned to ICM Script Symptom: ICM script receives an error code 9.

Message: Error code 9 received.

Cause: The gateway or Unified CVP H.323 Service cannot connect to the media server or the prompt does not exist.

Action: To resolve this issue, the connection to the server must be restored.


97

Chapter 2: Symptoms and Solutions Content Services Switch

Content Services Switch Content Services Switch Shows Call Server Out of Service but IVR Service Is up Symptom: Content Services Switch (CSS) shows Call Server out of service but IVR Service is up.

Message: None

Cause: The keepalive retry period is set too low.

Action: The CSS polls the Unified CVP Call Server with a special Unified CVP script that is manually placed on the CSS. The keep alive retry period must be greater than the retry period specified in the script. Make sure the CSS keep alive retry period is set to 6. Also do a show script on the CSS to verify that the script exists. service vxml1 keepalive maxfailure 1 keepalive retryperiod 6 ip address 10.86.129.22 keepalive type script "10.86.129.31" active

General System Issues Intermittent Timeout Issues Symptom: ICM timeouts in the Unified CVP Call Server log.

Message: N/A Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

98

Chapter 2: Symptoms and Solutions General System Issues

Cause: This may be due to your network settings not being set properly.

Action: The chart below shows the speed and duplex settings for Unified CVP. Ethernet Switch Speed Capability


1000 Mb

1000 Mb

Auto / Auto

Auto / Auto

1000 Mb

100 Mb

100 Mb / Full

100 Mb / Full

100 Mb

100 Mb

100 Mb / Full

100 Mb / Full

100 Mb

1000 Mb

100 Mb / Full

100 Mb / Full

Windows OS Application Log Fills up Symptom: Windows Operating System Application log fills up, and clearing the log only resolves the issue temporarily.

Message: None

Cause 1: Event log settings are not set to roll over.

Action 1: Change the Event setting so that the Application Log rolls over by doing the following: 1. Select Start > Programs > Administrative Tools > Event Viewer. 2. Right-click Application Log and select Properties from the menu. 3. Select the Overwrite events as needed option. 4. Click Apply and OK.

Cause 2: Some unnecessary protocols might have been installed. For example, NetBEUI has been known to fill the event log.

Action 2:


99


Uninstall protocols that are not necessary in this application.

Error Message: Operation Failed. Device Could Not Be Reached Symptom: Message: " device with IP Address: and Hostname: operation failed. Device could not be reached"

Cause: Action:

Error When Adding a Dial-Peer Symptom: When adding a dial-peer, an error message is issued and the dial-peer is not added.

Message: Could not add peer.

Cause: This message implies that the amount of memory required to add the dial-peer is not available.

Action: Use the show memory command to verify the available free memory. If the available free memory is > 0, then the logging buffer is taking up all available memory. In this case, adjust the amount of memory used by the log buffer by changing the value of logging buffered . The recommended value for this setting is between 10 - 20% of the total Gateway memory.

"Java is Not a Recognized Program" Error Symptom: "Java is not a recognized program" error message is displayed when trying to run the .bat files in the Unified CVP Server/admin directory ( or the .bat files in the application specific admin directory) after a Windows upgrade.

Message: Java is not a recognized program


100


Cause: The environment variable containing the java install path was overwritten during the upgrade.

Action: Possible Solution 1: (Reboot the machine.) VXML Server requires the use of an environment variable that is created by the installer. In order for this environment variable to take effect, the machine must be rebooted. If the machine was not rebooted by the installer, reboot the machine manually. Before rebooting, close all running application and save any unsaved work. Possible Solution 2: (Reset the Java Path system variable.) 1. From Start, select Programs > Administrative Tools > Computer Management. 2. From the Computer Management menu bar, select Action > Properties. 3. Select Advanced Tab > Environment Variables Settings. 4. In the System Variables Path, click Edit and add the path to where the java.exe file is installed.

"Password Does Not Meet the Operating System Security Requirements" Error Symptom: If an attempt is made to change an existing user password before the domain password policy's minimum password age is reached, the user receives an error.

Message: Password does not meet the operating system security requirements.

Cause: This condition is caused by attempting to change an existing user password before the domain password policy minimum password age has been reached.

Action: The user password should not be changed until the minimum password age specified in the password policy for the domain has been exceeded.

Password Lost or Forgotten Symptom: Password has been lost or forgotten.


101


Message: None.

Cause: For some reason, a password has been lost or forgotten.

Action: Contact your Cisco Partner or Cisco Technical Support for assistance.

Server IP Address Has Been Changed Symptom: The IP address of a Unified CVP server has been changed and is now listed as Not Reachable.

Message: Not Reachable

Cause: The IP address of a Unified CVP server has been changed locally instead of from the Operations Console.

Action: If you have already changed the IP address of the server, you may be able to delete it from the Operations Console, but the properties files on the server will not be deleted. This is due to the Operations Console not being able to access the server using original IP address. Adding the server with the new IP address to the Operations Console will add new property entries, but will not clean up the original property entries. The original property files will need to be cleaned up using a manual process. Contact your Cisco Partner or Cisco Technical Support for assistance when removing property files. Below is the correct procedure for changing server IP addresses. 1. Create the new device from the Operations Console. Use the original device as template. From the menu bar of the device, select the "Use As Template" option. Assign the new IP address to the new device and click Save. Do not select Save and Deploy since you have not changed the physical server to the new IP address. 2. Delete the device from the Operations Console before changing the IP address of the server. 3. Configure the new IP address on the local server. 4. From the Operations Console, select Save and Deploy of the new device.


102


5. Restart the server.

Unable to Open Command Prompt Window or Other Applications Symptom: It fails when trying to open a command prompt window, or some other application.

Message: None.

Cause: There are too many Windows threads in use on the system.

Action: If the total number of calls configured in Call Server and VBAdmin is too high, the system can run out if threads. • 1. Open the Task Manager on the Call Server. 2. Click the Processes tab. 3. Click the View menu. 4. Click the Select Columns option. 5. Check the Thread Count box. 6. Click the Threads heading to sort by decreasing thread count. If the thread count starts approaching 2000, then this is most likely the problem. In VBAdmin, the ShowMaxTotalCalls should never exceed 400.

When Hyperthreading Is Enabled, Unified CVP Performance Is Degraded Symptom: When hyperthreading is enabled, the Unified CVP performance is degraded.

Message: None.

Cause:


103


When hyperthreading is enabled in Windows Operating System on the Call Server, the number of concurrent calls handled decreases by approximately ten percent.

Action: Disable hyperthreading.

Users Cannot Login Symptom: Users, such as: Informix, cvp_dbadmin, and cvp_dbuser are unable to log in, even when using their assigned password.

Message: Login failed message

Cause: The password for this user has expired. This can occur if the user has not logged in for the number of days specified by the MaximumPasswordAge security policy, system calendar has been set forward, or security policy has been modified.

Action: Log in as administrator and change the password for the user.

Errors Reported by Various Unified CVP Components Symptom: Errors reported by various Unified CVP components. ICM script fails calls from Play Media node, and many calls abnormally disconnect.

Message: None.

Cause: Running Symantec virus scan software with a heavy call load (greater than 2 calls per second) can cause these types of errors.

Action: If your call load is heavier than two calls per second, disable Symantec virus scan software and use McAfee 8.0i virus scan software instead.


104

Chapter 2: Symptoms and Solutions H.323 Troubleshooting

Other Areas to Check Symptom: General failures.

Message: None.

Cause: Software version incompatibilities, hotfixes not installed or installed incorrectly, firewalls, network slowness.

Action: • Verify that the your software versions comply with the Unified CVP compatibility matrix in the Hardware and System Software Specification for the Cisco Unified Customer Voice Portal. • Determine if there are any firewalls in the solution. Firewalls can introduce network latencies and misrouted messages if it is not configured correctly.

H.323 Troubleshooting H.323 Service Process Restarts Unexpectedly Symptom: H.323 Service process restarts unexpectedly

Message: None

Cause 1: An internal memory error occurred in the H.323 Service.

Action 1: Search the H.323 Service logs for an occurrence of the word EXCEPTION around the time of the failure. If found, this indicates a memory corruption issue. Contact your support provider for additional information.


105


Cause 2: Another possibility is the user accidentally hung the H.323 Service console window by highlighting text on the screen or using the scroll bar to view the contents of the console window.

Action 2: Check the H.323 Service logs for an occurrence of the phrase H.323 Service will be restarted in 1800 seconds. If no EXCEPTION errors are found around that time, it indicates that the user inadvertently hung the H.323 Service process.

H.323 Service Out of Service Symptom: The H.323 service is out of service. Although it is out of service, the H.323 Service allows calls in progress to complete, but does not accept new calls.

Message: None.

Cause 1: H.323 Service is not registered to a Gatekeeper.

Action 1: In VBAdmin, verify the Gatekeeper IP address by typing showGatekeeper. On the Gatekeeper, type show Gatekeeper endpoints

Cause 2: There are no in-service Call Servers. The Call Server may take itself out of service when Unified ICME response times become too long.

Action 2: Within the Operations Console, go to Device Management > Unified CM. Go to Engine > Engine Configuration. Change New Call Throughput Upper Threshold and Call Event Throughput Upper Threshold values to 30000.

Cause 3: A user has administratively taken the H.323 Service out of service.

Action 3: A user can administratively take the Call Server out of service by selecting Go Out of Service from AppAdmin > Engine. To resolve this issue, start the service.


106


Cause 4: The H.323 Service has gone out of service due to an internal processing error.

Action 4: Consult your support provider for additional information regarding this type of event.

H.323 Service Log Indicates Wrong Number of Licensed IVR Service Ports Symptom: The H.323 Service log file indicates that the maxIVRPorts = 50, but the CVP.license file has 150 ports enabled.

Message: H323CallMgr::getRegistrySettings: maxIVRPorts = 50

Cause: It appears that the remaining ports are not being recognized. However, the Unified CVP License file applies to the Call Server when using the SIP Protocol. The Unified CVP H.323 Service is limited by the MaxTotalCalls registry value, so there is no correlation between the Unified CVP VoiceBrowser MaxTotalCalls and the CVP.license file.

Action: The maxTotalCalls registry value is based on the capacity for the Unified CVP H.323 Service and should not be changed. In this case, you need to provision more than one Unified CVP H.323 Service.

H.323 Service Must be Registered with Gatekeeper to Perform IP Transfers Error Symptom: Error message in the H.323 Service logs states “ERROR: H.323 Service must be registered with Gatekeeper to perform IP transfers” during a call.

Message: H.323 Service must be registered with Gatekeeper to perform IP transfers.

Cause: The Unified CVP Call Server is not successfully registered with the H.323 Gatekeeper.

Action:


107


Set the Gatekeeper IP address from the VBAdmin using setGK . Restart the Unified CVP Call Server after changing the H.323 Gatekeeper.

Unexpected Event for the Gatekeeper Registrations Error Symptom: Error Message in the H.323 Service Logs States “Unexpected event for the Gatekeeper registration. Please make sure H.323 Gatekeeper is up and running.”

Message: Unexpected event for the Gatekeeper registration. Please make sure H.323 Gatekeeper is up and running.

Cause: The configured H.323 Gatekeeper is not running or Gatekeeper is not configured correctly.

Action: Follow these steps: • Make sure the H.323 Gatekeeper is running on the IP address configured in the Unified CVP Call Server. Verify that the correct IP address of H.323 Gatekeeper is configured. • Verify that the IP network connectivity between the Unified CVP Call Server and H.323 Gatekeeper.

VBadmin Command Fails Symptom: An attempt to run VBAdmin from the command window fails. The browser disappears, the window does not remain open long enough to accept a setGK command, and the following error message appears.

Message: "trace: PIMcom:Init - Failure connection to pipe \\.customer\voicebrowser\VBcmdpipe "

Cause: Service Pack 1 or later of Windows 2003 is not installed on the system.

Action: Install Windows 2003 Service Pack 1 or later.


108

Chapter 2: Symptoms and Solutions SIP Troubleshooting

SIP Troubleshooting SIP to SIP Calls Do Not Play Ring Ringback Tone Symptom: The ringback tone is not heard when a SIP to SIP call is placed.

Message: None.

Cause: In high load situations, the playback of ringtone for many calls might over utilize the CPU processing of the gateway.

Action: In this case, it is necessary to use load balancing or configure the gateways to optimally distribute media termination of calls. The following are some facts regarding ringtone: • Local ringback in SIP is performed with a 180 response process before the call is established. • After the call has been established, ringtone is played back to the caller using an intermediate transfer mechanism. This is accomplished by a transfer to a ringtone playback voip dialpeer on the Gateway. Once the agent answers the call, the caller is then reinvited to the agent. At that point, the ringtone dialog is dropped. • Troubleshooting ringtone issues consists of configuring the dial-peer to match an incoming voip call to the dialed number (DN) that is configured in the SIP Service for Ringtone. For additional information for SIP configuration, refer to the Configuration and Administration Guide for Cisco Unified Customer Voice Portal Guide. • Ringtone playback is not the same as music on hold (MOH) on the Unified CM. For example, when a caller is put on hold at Unified CM phone they hear music. The ringtone playback is separate from this feature. • Necessary items for ringtone playback: – Ringback.wav on the Gateway. – Ringtone service and TCL configured on the Gateway. – Dial-peer associated with ringtone service. – Static routes to route calls from Unified CVP to the Gateway. Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

109


Assigning DN Auto Answer Settings for Shared Lines Symptom: The Auto Answer DN setting is not compatible with Cisco TelePresence on shared lines.

Message: None.

Cause: Unified CVP calls routed through a shared SIP trunk line causes problems when using Cisco.

Action: Do not enable the Auto Answer DN setting for shared Directory Numbers (DNs) when using Cisco TelePresence.

SIP Calls Experience Long Delay or Fast Busy Symptom: During the making of a SIP call, the caller experiences a long delay or hears a fast busy condition.

Message: None.

Cause: If the calls go through, but there is a long delay or a fast busy, it could be the result of a Domain Name Server (DNS) problem in the proxy.

Action: Verify that the correct nameserver and domain information was configured in the /etc/resolv.conf file. If DNS is not being used, then remove any entries in that file.

SIP Calls Receives 500 Internal Server Error "Routing Failed" Event Symptom: A SIP call fails and a 500 Internal Server Error "Routing Failed" event is received.

Message: 500 Internal Server Error "Routing Failed".


110


Cause: A SIP Trunk is not configured to receive or send calls to the IP Address from the source of the SIP INVITE event.

Action: The solution is to configure a SIP Trunk to the IP address of the sending endpoint, CUPS, Gateway, or Unified CVP.

SIP Call Receives 489 Bad Event Message Symptom: A 489 Bad Event message is received from a Unified CM or Proxy Server when a SIP call is made.

Message: 489 Bad Event

Cause: Logs indicate 489 Bad Event messages are being received from either the Unified CM or the Proxy Server. This event is used to indicate that the server did not understand the event package that was specified in the Event header field. The problem is that Key Press Markup Language (KPML) is trying to be used, and is currently not supported in Unified CVP.

Action: The solution is to only use the inband Dual-Tone Multi-Frequency (DTMF) method setting (RFC 2833) on the SIP Trunk that is associated with the Unified CM or the SIP Proxy Server. This is done by selecting the last checkbox on the SIP Trunk configuration page of the Unified CM Administration page.

Back to Back User Agent (B2BUA) Sending 503 Server Not Available Symptom: The Back-To-Back User Agent (B2BUA) is sending a 503 Server Not Available response before the call is even established.

Message: 503 Server Not Available

Cause 1: This is most likely caused by the SIP service not running. The ICM VRU PIM needs to connect to the ICM Subsystem for it to go into service. When that happens, SIP can then go into service.


111


If the ICM VRU PIM connection is lost, then the SIP Service goes into partial service and rejects calls.

Action 1: Enable the ICM Subsystem to allow the SIP Subsystem to go into service.

Cause 2: This is a license depletion issue, which typically equates to a high call rate. When all the licenses are in use, the service goes into a partial state and rejects calls. When the active call number drops and licenses are available again, then the SIP Subsystem will start accepting calls.

Action 2: Monitor the call statistics from the Operations Console to determine if additional licenses are needed.

REJECTED WITH 401 - Unauthorized Reason: Q.850;cause=57 Symptom: SIP call failure.

Message: %CVP_4_0_SIP-3-SIP_CALL_ERROR: GUID:24FCDB65-129111DB-82A20014-6944B792 - LEGID: 115289260464070 - OUT - DsSipInvitation ;tag=11296519 REJECTED WITH 401 - Unauthorized Reason: Q.850;cause=57

Cause: Unified CM SIP Trunk Security Profile has authentication turned on.

Action: Clear the digest and other authentication check boxes on the Unified CM SIP Trunk Security Profile associated with the SIP Trunk that is being used with Unified CVP B2BUA.

MTP Resource Allocated Even Though MTP is Disabled Symptom: Media Transfer Protocol (MTP) Resource is allocated for each call coming into SIP trunk even though MTP disabled is selected on the trunk.

Message: None.


112


Cause: Failure to negotiate DTMF method occurs for the call, according to the DTMF method supported by the endpoints involved in the call.

Action: Set the endpoints of the call to negotiate the same DTMF method, that is RFC2833 for Unified CVP calls. Note: Out of Band (OOB) Key Press Markup Language (KPML) is not supported for the Unified CVP 4.0 and 4.1 releases.

SIP Call Fail on SIP Trunk on Unified CM to Back to Back User Agent Symptom: SIP calls cannot be made out on the SIP Trunk on the Unified CM to the Back to Back User Agent (B2BUA).

Message: None.

Cause: The route pattern is not defined, in use, or has the wrong wildcards configured. For example, a pattern of 7xxx routes any calls with a DN beginning with a 7 and has 3 more digits after it. Using something like 7* does not work.

Action: Correct the route pattern on the Unified CM.

SIP Call Is Cleared From Unified CVP with Code 127 Symptom: SIP call is cleared from Unified CVP with code 127.

Message: None.

Cause: Call was detected as a phantom or zombie call, which may have happened as a result of a network down scenario.

Action:


113


Check the server network connectivity.

Critical Error Audio Is Not Heard on an Abnormal Disconnect SIP Call Symptom: A critical error wav file is not heard on an abnormal disconnect call. Instead the caller hears a fast busy condition, or the call just drops.

Message: None.

Cause: When using REFER transfers, including the REFER used to play back critical error.wav for abnormal disconnects, an outbound voip dial peer is necessary on the ingress gateway. This outbound dial peer is needed because when the REFER message comes into gateway from Unified CVP Call Server, it needs to match an outbound dial peer in order for the call to succeed, or else a 503 rejection occurs if no dial peers match the REFER-TO header URI. Dial peer destination targets must match the labels in the REFER-TO SIP URI, that is @ and other labels that may be used in the Unified ICME routing label.

Action: Configure an outbound voip dial peer on the ingress gateway. For example: dial-peer voice 1050 voip destination-pattern 9....... voice-class codec 1 session protocol sipv2 session target sip-server session transport tcp dtmf-relay rtp-nte h245-signal h245-alphanumeric no vad

SIP Calls Survivability Script Failing Symptom: Survivability script does not seem to work.

Message:


114


None.

Cause: SIP calls from the IOS gateway are getting REFER to the ERROR DN, and critical error wav file is heard (or not heard). SIP calls originating from gateway do not contain "--CVP" in the remote party id header display name of the INVITE.

Action: The pots dial peer must match the incoming called number and be associated with the survivability service. For example: dial-peer voice 1000 pots service cvp-survivability incoming called-number 8T direct-inward-dial

482 Loop Detected Error from CUPS SIP Proxy Symptom: 482 loop detected error is received from the CUPS SIP Proxy. Ringing is heard by caller, but the call is never answered.

Message: 482 Loop Detected

Cause: When using SRV with a CUPS farm for failover, the SIP URL from Unified CVP contains the SRV domain name. Proxy loops that call back to itself because it does not know about itself as the SRV domain.

Action: Under Service Parameters section in CUPS admin, add the domain name in the "SRV cluster name" editbox.

SIP Call Fails with 404 Not Found Error Message Symptom: Sending call to Unified CM with a domain name like an SRV cluster name fails with 404 not found rejections.


115


Message: 404 Not Found.

Cause: The cluster fully qualified domain name (FQDN) is missing.

Action: Go to System -> Enterprise Parameter -> Cluster Fully Qualified Domain Name on the Unified CM and add the Cluster FQDN.

H.323 Gateway/SIP Call with IVR Service is Terminated with Reason Code: Q.850;Cause=38 Symptom: Any failure of the bootstrap VXML Server fetches to Call Server causes the SIP IVR Service leg of call to be terminated by the recovery handoff TCL scripts on VXML Server Gateway with a Q850 code of 38. The same error also occurs on the H.323 gateway; however, the failure occurs less often.

Message: Q.850;Cause=38

Cause: If the Unified CVP loses network connectivity, the VXML Server Gateway is not able to get information from the IVR Service, and as a result a code 38 rejection is generated in the Gateway logs.

Action: Be sure that the network interface configuration is as follows: ip route-cache same-interface ip route-cache cef ip route-cache ip mroute-cache no cdp enable

If specified, remove the following line from the network interface: keepalive 1800


116

Chapter 2: Symptoms and Solutions SNMP Troubleshooting

SIP Call Drops When Mobile Agent's Phone (Remote CTI Port) is Busy Symptom: If you are using Mobile Agent with the CUPS SIP Proxy and CUPS is configured with multiple static routes for the same Destination Pattern to Unified CM nodes, calls may be dropped when the Mobile Agent's phone rings busy.

Message: Multiple 180 Ringing messages and the 403 Forbidden message are sent to CVP. ICM GED 125 Dialog Failures occur between CVP and ICM.

Cause: By default, CUPS attempts to reach the Mobile Agent's phone using each of the configured static routes.

Action: In the CUPS Service Parameter Configuration Window, add 403 to the "4xx Contact Failover Exception List".

SNMP Troubleshooting SNMP Notification: authenticationFailure Symptom: The SNMP log shows frequent connect and disconnect events from the Reporting Server.

Message: Received SNMPv2c Trap: Community: public From: sysUpTime.0 = 00:09:54.02 snmpTrapOID.0 = authenticationFailure

Cause: Missing SNMP Community Strings.


117

Chapter 2: Symptoms and Solutions Installation Troubleshooting

Action: You need to manually import the previously configured community strings from the windows SNMP Service on that machine. Complete the following steps. Importing Previously Configured SNMP Community Strings (windows) 1. To see a list of the previously configured Windows SNMP V1 community strings: –

a. Open Windows Services viewer.

–

b. Right-click SNMP Service and select Properties.

–

c. Select the Security tab. This tab lists the accepted V1 community strings and the access granted for each string. It also lists the hosts from which SNMP packets are accepted. Note: The accepted hosts apply to all community strings, whereas the Operations Console gives you more granularity, allowing you to specify accepted hosts on a community string basis.

2. Configure these community string from the Operations Console. –

a. Open the Operations Console and select SNMP | V1/V2C | Community String.

–

b. For each community string discovered above that has not already been configured in the Operations Console, add it via Add New. c. Enter the community string exactly as it was listed in the step above. d. Choose V1 as the version. e. For windows community strings with permission other than Read Only, select Read Write in the Operations Console. f. Be sure to select the device that the community string was reported.

Installation Troubleshooting Increase Size of the Virtual Memory Paging File Symptom: Windows Virtual Memory is too low. Event Viewer (System log) shows that system is running out of virtual memory.


118


Message: Application popup: Windows - Virtual Memory Minimum Too Low: Your system is low on virtual memory.

Cause: Windows is increasing the size of your virtual memory paging file. During this process, memory requests for some applications may be denied.

Action: To change the size of the virtual memory paging file, perform the following steps. 1. From the Windows Start menu, click Start --> Settings --> Control Panel. 2. Double-click System. 3. On the Advanced tab under Performance, click Settings. 4. On the Advanced tab under Virtual memory, click Change. 5. Under Paging file size for the selected drive, click Custom size and type a new paging file size in megabytes in the Initial size (MB) and Maximum size (MB) boxes, and then click Set. Note: The Initial and Maximum size should be set to the same value.

InstallShield Engine Cannot Be Installed Symptom: During Unified CVP installation, the following error message appears:

Message: “The InstallShield Engine(iKernel.exe) could not be installed. Ikernel.exe could not be copied to C:\Program Files\Common Files\InstallShield\Engine\6 \Intel 32.”

Cause: Multiple instances of the Install Program are being run. Note: InstallShield (the install program for the Unified CVP) allows only one ikernel.exe to be running at a time.

Action: Stop all install programs and run the Unified CVP installation program.


119


File in Use Message When Uninstalling Symptom: While trying to uninstall Unified CVP, a dialog box displays with the message a file is in use and the software cannot be uninstalled.

Message: File is in use and the software cannot be uninstalled.

Cause: H.323 Service, and dumplog use the files msgs2.dll, Icrmsgs.dll, cat2.dll, and Icrcat.dll. Sometimes the filter files appl2.flt, custvis2.flt, object2.flt, and primpos2.flt. If you try to uninstall while one of the other processes is using these files, the Uninstall procedure is not able to remove them.

Action: Before running Uninstall, verify that: • All Unified CVP services are stopped. • The Unified ICME Service Control Panel is closed. • The dumplog utility is not running. • All file accessing programs (such as Microsoft Word or Windows Explorer) are closed.

Error Uninstalling WebSphere Application Server Symptom: You encounter an error condition when you attempt to uninstall IBM WebSphere Application Server.

Message: Several possible.

Cause: Several possible.

Action: See the Unified CVP Installation log for details and instructions.


120


See Also IBM WebSphere Application Server documentation and support website.

Password Issues Symptom: Passwords must meet complexity requirements.

Message: N/A

Cause: This security setting determines whether passwords must meet complexity requirements.

Action: If this policy is enabled, passwords must meet the following minimum requirements: • Not contain all or part of the user's account name. • Be at least twelve characters in length. • Contain characters from three of the following four categories: – English uppercase characters (A through Z) – English lowercase characters (a through z) – Base 10 digits (0 through 9) – Non-alphabetic characters (for example, !, $, #, %) • Complexity requirements are enforced when passwords are changed or created.

Reporting Password Error During Installation Symptom: When you specify the Reporting password during Reporting installation, you encounter the following message:

Message: Error 2245 while running: changing default password for user Informix.

Cause: Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

121


The password security policy on the machine on which you are installing Reporting conflicts with Unified CVP Reporting password requirements. Most likely, the machine password security policy has a minimum password age greater than 0 or the minimum password length is greater than 13 characters.

Action: On the machine on which you are installing Reporting, abort the installation process. Open the local Security Policy control panel (Control Panel > Administrative Tools > Local SecurityPolicy) and examine the Password Policy (Security Settings > Account Policies > PasswordPolicy). If the password policy on your machine conflicts with Unified CVP Reporting password requirements, make the appropriate adjustments. Restart the installation process.

Install Radvision Package in Linux Symptom: MySQL service will not start.

Message: Starting MySQL../etc/init.d/mysql: line 151: kill: (15147) - No such process [FAILED]

Cause: The problem is caused by "selinux" or Security Enhanced Linux which is commonly installed with linux.

Action: Disable it by editing /etc/selinux/config and changing SELINUX=enforcing to disabled. Then reboot.

Dialog Box During Reporting Uninstallation Symptom: If you are removing the Unified CVP Reporting component, the following message displays when component removal is complete:

Message: Could not delete product files and/or directories.

Cause: This dialog box is a present in this release solely due to a Informix product limitation, and doesn't indicate a problem with removing Reporting.


122


Action: No action necessary.

Install with Non-US English Region/Locale Settings Symptom: ICM subsystem not starting.

Message: Cause: CVP software was installed using non-US English region and locale settings for OAMP and Call Server.

Action: Reinstall the system with US English locale.

Call Server CPU spikes to 90%-100% Symptom: During the installation process, the Call Server CPU shows high utilization adn spikes to 90%-100%.

Message: Error logs on the outprocess service indicating terminateConnection:Transport Down every 5-6 seconds.

Cause: This occurs if there is an outprocess service (for example a VXML Server, REP Server or ORM) which does not have the correct version of cvp-common.jar. This can happen, if the CallServer or any other component is uninstalled and the CVP_HOME folder is not deleted after the uninstall. Re-installing the component, does not delete the cvp.war file or overwrite it.

Action: Ensure all components have the correct versions installed and the cvp-common.jar file is consistient on all components. Re-installing the correct build on all services is the right way to correct this problem.


123

Chapter 2: Symptoms and Solutions Gateway Troubleshooting

Gateway Troubleshooting Ringing Stops on IP Phone Calling TDM Symptom: Ringback tone is clipped when an outbound call is placed from TNP IP phones to a TDM endpoint. For IP originated calls whose destination is a TDM endpoint, and the call goes through Unified CVP, this shows up when the call originates from TNP phones (7971, 7970, 7961, 7941). Customer experience will be something like that given in the call flow example below: • Dial 1234 from a TNP phone (7941, 7961, 7970, 7971) • Call goes out a SIP trunk to Unified Cisco Voice Portal • Unified CVP sends call to a Voice Gateway (for example, AS5400) to receive IVR treatment • Caller hears a prompt, and so forth • Call then gets transferred to a destination on the TDM side (for example, outbound call to endpoint with extension 5678) • On the IP phone side, user hears the first ring but then nothing while the destination end point continues to ring

Message: None.

Cause: The appropriate settings were not made on the Gateway that is placing the outbound call.

Action: Apply the following setting on the POTS dial-peer on the Gateway that is placing the outbound call to the TDM destination: GW(config-dial-peer)#progress_ind alert strip

Gateway Cannot Connect to ASR/TTS Server Symptom: Gateway cannot connect to ASR/TTS server.

Message: Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

124


None.

Cause: There is a connectivity problem between the gateway and the ASR/TTS server or there are not enough ASR/TTS licenses available.

Action: Check that the Gateway has the correct IP addresses for the ASR and TTS servers, as well as the ASR and TTS backup servers. Run a series of ping tests to each of the IP addresses of the ASR and TTS servers and backup server. Verify that there is network connectivity to each of the servers. Check that the ASR and TTS services are running. Check that the IVR Service configuration is correct.

Pressing Digit 9 Causes No-Match Symptom: In DTMF-only adapter, pressing digit 9 while using options causes no-match after 10 seconds.

Message: None.

Cause: This delay is caused by the following link grammar in the root document for the DTMF-only adapter. 99999 Since the options are created with modal=false, the link grammar remains active in the Gateway DTMF-browser. On pressing 9, the Gateway starts to match this grammar and waits for the next input instead of returning a 'no-match' immediately.

Action: When writing a VXML Server script for a DTMF-only adapter that uses the Get Digits, Forms, or n_Option_Menu elements, follow this procedure to avoid delay in processing the digit 9 when the script is executed.


125


1. In the VXML Server script, insert a hotlink to an inline grammar. When configuring the hotlink’s grammar, the values of the digits must not match: –The value of any digit that is expected in response to the Get Digits element. –The value of an option number used in any n_Option_Menu element. 2. If the inline grammar in Step 1 contains more than one digit, set the interdigittimeout in the root document properties to 5 seconds. For example: interdigittimeout 5s Examples: If the VXML Server script uses 3_Option_Menu, where 1, 2, and 9 are valid options, then the inline grammar can have the value 8, but cannot have the value 1, 2, or 9. Similarly, if the VXML Server script uses 9_Option_Menu, where digits 1 through 9 inclusive are valid options, then the inline grammar can have the value 99, and the interdigittimeout must be set to 5s. If you are using a GetDigits element and have set the follow range: min=2, max=4, then the link grammar can be 99999, and you must set the interdigittimeout to 5 seconds. If the script contains an Option Menu as well as GetDigits elements, you only need to set the link grammar once.

Receiving Bootstrap Warning Error Symptom: When adding the bootstrap service to the gateway you get bootstrap warnings.

Message: config-app)# service bootstrap flash:bootstrap.tcl (config-app-param)# paramspace english language en (config-app-param)# paramspace english index 0 (config-app-param)# paramspace english location flash (config-app-param)# param CVPcallserver-primary 10.3.3.14 Warning: parameter CVPcallserver-primary has not been registered under bootstrap namespace (config-app-param)# paramspace english prefix en (config-app-param)# param CVPcallserver-port 8000


126


Warning: parameter CVPcallserver-port has not been registered under bootstrap namespace (config-app-param)# param CVPcallserver-backup 10.3.3.14 Warning: parameter CVPcallserver-backup has not been registered under bootstrap namespace

Cause: Verify that the service is configured correctly.

Action: When this error is received, it is usually a benign warning event. Verify that the service has been configured correctly. To view the configuration, type the following command: router># show run

The output should still show that it has been configured, as the example below shows. service bootstrap140 flash:bootstrap.tcl param CVPcallserver-backup 10.3.3.14 paramspace english language en paramspace english index 0 paramspace english location flash paramspace english prefix en param CVPcallserver-primary 10.3.3.14 param CVP-port 8000

Switch Transfer Is Not Going to the Gateway Defined in the Gatekeeper Zone Prefix Command Symptom: Switch Transfer is not going to the Gateway defined in the Gatekeeper zone prefix command.

Message: None.

Cause: The IP address of the outbound Gateway, which is registered with a DNS name, is used in the zone prefix.


127

Chapter 2: Symptoms and Solutions Cisco Unified Presence Server

Action: Use the same name of the outbound Gateway in zone prefix. To verify the outbound Gatekeeper priority list, do the following: 1. Enter the show Gatekeeper gw-type-prefix command in the Gatekeeper CLI. 2. Enter the show Gatekeeper end command in the Gatekeeper CLI and verify that H323 ID used by egress gateway/endpoint to register with Gatekeeper is used in the Zone prefix configuration of Gatekeeper.

Cisco Unified Presence Server Installation Keeps Failing in Network Connectivity Validation Page Symptom: During the installation of the Cisco Unified Presence Server, the installation fails at the Network Connectivity Validation screen.

Message: None

Cause 1: Wrong hostname is configured in the server.

Action 1: Make sure the CUPS hostname is configured correctly and it matches the name specified in the Call Server page of the CM Publisher.

Cause 2: Wrong CM Publisher hostname or password.

Action 2: Make sure the CM Publisher's hostname and password are entered correctly. If you have DNS server setup, make sure the CM Publisher's hostname is resolvable.

The Install Complains About Hardware Check Failure Symptom:


128

Chapter 2: Symptoms and Solutions Operations Console (OAMP)

During the installation of the Cisco Unified Presence Server, the installation complains about a hardware check failure.

Message: None

Cause: Using non-compatible MCS server hardware.

Action: Verify that you are using a supported server.

Call Fails When Transferred to Agent Symptom: A call is transferred to an agent, and the agent is reserved, but call transfer fails.

Message: None.

Cause: Cisco Unified Presence Server (CUPS) trunk on the Unified CM is not configured.

Action: Configure the CUPS trunk on your Unified CM. Refer to the Configuration and Administration Guide for Cisco Unified Customer Voice Portal for configuration information.

Operations Console (OAMP) OAMP Functionality Is Limited Symptom: User is able to log in to the Operations Console, but cannot navigate the drop-down menus. Drop-down menus do not respond to mouse clicks or key stroke combinations.

Message: No message displays.

Cause:


129

Chapter 2: Symptoms and Solutions Cisco Security Agent (CSA)

Javascript is disabled in Internet Explorer.

Action: Enable JavaScript in Internet Explorer.

Cisco Security Agent (CSA) CSA Agent is Disabled; You Cannot Re-enable It Symptom: CSA agent is disabled on the majority of boxes and cannot be re-enabled in the usual ways.

Message: An Application Event error is recorded as follows: Event Type: Information Event Source: CSAgent Event Category: Local Event Mgr Event ID: 256 Description: The Agent Rule file hash has been modified. Shims will be disabled, and NO security policies will be enforced.

Cause: This happens when the new standalone agent is installed over the current one.

Action: 1. Navigate to C:\Program Files\Cisco Systems\CSAgent\cfg\ 2. Delete the agent.state file 3. Restart the agent Note: If this problem persists, uninstall the agent and re-install the agent again.


130

Chapter 2: Symptoms and Solutions Support Tools

Support Tools Unable to Login to Support Tools Symptom: Attempts to login to the Support Tools fails.

Message: Login failed.

Cause: The Support Tools server was not installed in a domain.

Action: Log into the Support Tools using the following format: .\

Backup and Restore Reporting Server Purging Daily Symptom: Purge events are occurring on a daily basis on the Reporting Server.

Message: None.

Cause: The retention times might be set too high.

Action: Back up the database, then change the retention times. To configure Reporting Database purge settings: 1. From the Operations Console, choose Device Management > CVP Reporting Server.


131

Chapter 2: Symptoms and Solutions Backup and Restore

2. In the Reporting Server window that opens, click Edit. 3. Select the Database Administration menu in the toolbar, then select Data Purge. 4. Change the data retention time for each category of data. 5. Enter your Database Administrator Password and click Save & Deploy. For a complete list and description of ECC variables, see the Configuration and Administration Guide for Cisco Unified Customer Voice Portal.

Changing Backup Server Requires Restart Symptom: Changing backup server requires a restart.

Message: None.

Cause: Changing primary or backup Call Server updates the Call Server messageAdapter.properties file and informs the user to restart all the affected devices.

Action: Restart all of the affected devices.

"Retrying ASR Server. Last tried BACKUP asr-en-US-backup. primary tried:0, backup tried:1" Error Symptom: Missing parameters in the nuance-resources.txt file.

Message: Retrying ASR Server. Last tried BACKUP asr-en-US-backup. primary tried:0, backup tried:1

Cause: Missing parameters in the nuance-resources.txt file.

Action: If you are using Nuance speech recognition to recognize DTMF digits, be sure the following parameters are in your nuance-resources.txt file on the recognition server machine:


132


dtmf.SuppressNotifications=FALSE


133



134

Appendix A

Error Messages This appendix provides detailed information on error messages.

Infrastructure Error Messages Table 2: Infrastructure Error Messages

Title

Description

Cause

Resolution

EMERGENCY STATE_MANAGER_ STARTUP_FAILURE

StateManager did not start.

The specific cause will be The Call Server must be noted in the log message. restarted to resolve this issue.

STARTUP

start up failed.

The specific cause will be The Call Server must be noted in the log message. restarted to resolve this issue.

START

No Services have been started. The Call Server has not been setup through the Operations Console.

SHUTDOWN

Unable to shutdown the Web Application Server (WAS).

Login to the Operations Console, and create a Call Server with the IP address of this server.

The specific cause for the Before attempting to restart shutdown will be noted in the server, manually shut it the log message. down with the command: CVP_HOME\bin\runcvp stopservice call

If this fails, you must force the process to stop with Windows Task Manager. Once it has been stopped, then restart the server.


135

Error Messages Infrastructure Error Messages

Title

Description

Cause

Resolution

PORT_THRESHOLD

The Port License Checkout Threshold is at the threshold level.

The volume of calls on the Call Server is approaching the maximum capacity allowed in the license.

When the call volume decreases, a message stating that the threshold has returned to the safe level will be logged. If the system is often running at or near licensed capacity you may wish to contact Cisco or your Cisco Partner to discuss the increasing of your system capacity.

ALERT PORT_THRESHOLD

The Port License Checkout The volume of calls on the No steps need to be taken. Threshold has returned to the Call Server is no longer safe level. near the maximum capacity.

SNMP_CONNECTION_ FAILURE

Could not bind serviceability There is a non-Unified manager to . Retrying in CVP process running minutes. which is bound to the port(s) listed.

(AIX) SNMP_ INIT_FAILURE

Could not start SNMP Agent. The specific cause will be Resolve the reason for start noted in the log message. up failure, and then restart the SNMP Agent. To restart the SNMP Agent, type:

If this message is consistently being logged, the processes bound to the listed port(s) should be stopped, and then restart Unified CVP.

CVP_HOME\bin\snmp-wrapper. sh start

at a command prompt. Note: CVP_HOME is the location where Unified CVP was installed. (AIX) SNMP_ FORWARDER_INIT _FAILURE

Could not initialize SNMP Forwarder.

The specific cause will be The Unified CVP application noted in the log message. that logged this error must be restarted to enable the SNMP services.

CRITICAL CAH_STARTUP _FAILURE The Common Admin Handler The server has been will not run properly. The misconfigured. The service underlying service is absent. noted in the log message failed to start.


136

Try restarting the server. If this message appears during the restart, then the server must be removed from the Operations Console. Once it has been removed, you can


Title

Description

Cause

Resolution

recreate it from the Operations Console. HEARTBEATS_ STOPPED Heartbeats from stopped. Unilaterally setting state to PARTIAL_SERVICE. Sending update.

LICENSE_ EXPIRATION

The service noted in the log message has failed or lost connectivity to the Call Server.

If the noted service has been manually shutdown, then this message is normal and expected. If it was not manually shutdown, then check if it has failed. If it failed, attempt to restart it. Verify that the server on which the noted service should be running has a valid network connection to the Call Server. Once everything is running and connected you will see the following message: "STATE_ MANAGEMENT". This message notes that the "Heartbeats from resumed" where the service ID's match.

The license for The license on the system has expired, has expired and the system gracefully shutting down the will shutdown gracefully. system.

The system will not start with an expired license. Contact Cisco or your Cisco Partner to obtain a new license for your server.

ERROR LOGMSG_SET_ PROP_TYPE_ERROR

Set property operation failed for the specified Property.

This occurs when Property Restart CallServer, OAMP, Type is not a String. and ORM. Reset the property through OAMP only.

LOGMSG_SOCKET_ SETUP

Errors were encountered during Network issues could cause Ensure all servers are in the socket setup. problems while creating same domain. If this message sockets. occurs repeatedly, restart the relevant server (that is, callserver/vxml/reporting/iContact application).

START

Could not start due to: {reason}

SHUTDOWN

{shutdown type} shutdown of Either a forced or graceful [{subsystem id}] failed due to: shutdown (as noted) of the {reason} subsystem failed due to the reason noted.

The subsystem noted in the Restart the Call Server. log message failed to start up due to the reason noted. If the Call Server successfully shuts down, then this message can be safely ignored. If the Call Server failed to shutdown (as reported by Windows


137


Title

Description

Cause

Resolution

Services) you must stop it manually with the command: CVP_HOME\bin\runcvp stopservice call.

If this fails, you must force the process to stop with Windows Task Manager. SECURITY

Could not configure secure connection in Tomcat's server.xml.

During the installation, modification of the server.xml file failed.

Key size of {size} illegal, Java During setup of the security security policies allow system, local security maximum keysize of {size}. policies caused key creation to fail.

(AIX) SERVER_SETUP

SNMP Agent Server setup failed because {reason}.

Make sure to completely uninstall Unified CVP, reboot, and then re-install. First, make sure your system is running a valid copy of Windows which is setup with the correct locality. Next, make sure you do not have a limited security version of the Java Runtime Environment (JRE) on your system that is overriding the one installed with Unified CVP. Finally, if this message was seen during install, you must completely uninstall Unified CVP. Once that has been completed, re-install Unified CVP. Otherwise Unified CVP needs to be restarted.

The SNMP service failed Restart the SNMP service. At to start due to the reason a command prompt, type: noted in the log message. CVP_HOME\bin\snmp-wrapper. sh restart

CVP_HOME is the location where Unified CVP was installed. WARN DUPLICATE_ RETRIEVAL Second or more attempt at getting the CommonAdminHandler


138

Although operations will If the Operations Console not be affected, the system operations on this server are has been misconfigured. failing, then remove the server from the Operations Console. Once removed, reconfigure it from the Operations Console.


Title

Description

Cause

Resolution

INVALID_MSG

Message processing aborted in ({service}) due to Error: {error}. Message was: {message details}.

This may be caused by connecting together incompatible versions of Unified CVP.

If you have recently upgraded some Unified CVP components, make sure all necessary components have been upgraded. Please send this error message in it's entirety to Cisco Technical Support, or your Cisco Support Partner.

IO_ERROR

Properties may not have been There was a problem Make sure all properties files {loaded/saved} due to an IO accessing a properties file in the CVP_HOME\conf Exception. on the disk. directory are read/write enabled, and are not currently opened in an editor.

LICENSING

{license type} license already The system was started expired on {date}. with an expired license.

Please contact Cisco or your Cisco Partner for a new license.

SNMP_CONNECTION_ FAILURE

Connection to the SNMP Subagent on port {port(s)} failed. Retrying next port in {time} minutes.

In Windows, the SNMP service can be started from the services control panel. Select the Cisco CVP SNMP Management service, and click Start.

If this message is seen consistently, the local SNMP service may not be running.

In an AIX environment, the SNMP service can be started by typing: CVP_HOME\bin\snmp-wrapper. sh start

at a command prompt. NOTICE SUBSYSTEM_NOT _SHUTDOWN

Normal: Subsystem [{subsystem id}] is in [{operation state}] state.

During a graceful shutdown, the externally connected subsystem noted in the log message that it was still running. This is normal.

RECEIVED_STATE _MSG StateManager: Subsystem The subsystem noted has [{subsystem id}] reported moved to a different state change to state [{state}] due to for the reason noted. [{reason}].

If you wish to have all services associated with the Call Server that was just shutdown, as noted in these log messages, this can be accomplished via the Operations Console Control Center. These messages are expected during the operation of Unified CVP, especially during start up and shutdown. If subsystems are frequently going into PARTIAL_SERIVE due to


139


Title

Description

Cause

Resolution

"dependency state unknown" this may indicate an over-used link between the server and the Unified ICME VRU PG. STARTUP

MessageBus Started.

Expected during Call Server start up.

No action required.

SHUTDOWN

MessageBus Stopped.

Expected during Call Server shutdown.

No action required.

SERVLET_STARTUP

Servlet Startup Error: {error} This Web Application No action required. Server (WAS) initialization error is benign and automatically corrected.

START

All Subsystems have been started.


No action required.

BAD_LOG_ MESSAGE_XML

The CVPLogMessages.xml file was corrupt or could not be found. Default log message and settings will be used.

CVPLogMessages.xml file is either missing from CVP_HOME\conf or has been incorrectly modified. This will not affect the operation of the server, but default logging will take the place of any customized logging that may have been configured.

Restore CVPLogMessages.xml file with a correct version from the Operations Console.

[AUDIT]

The system has started.


No action required.

[AUDIT]

The system has completely shutdown.

Expected during Call Server shutdown.

No action required.

[VERSION]

{product} version {version Version information number}({version type}) built expected during start up. on {date}.

No action required.

CAH_STARTUP

The {subsystem id} subsystem Expected during start up. will start the Common Admin Handler.

No action required.

STATE_MANAGEMENT

Heartbeats from {subsystem id} resumed. Current state: {last known state}. Reason: {reason}. Requesting current state of {subsystem id}...

[AUDIT]

Log Level now set to [{level}]. The log level of the No action required. subsystem which logs this message has been set to the level noted.

INFO


140

This message will be seen No action required. when a subsystem that lost it's connection to the Call Server has reconnected.

Error Messages IVR Error Messages

Title

Description

Cause

Resolution

(AIX) SERVER_SETUP

SNMP Agent Server setup on Expected during start up of No action required. port {port}. the SNMP service on AIX.

[AUDIT]

Maximum log file size: {size} Informational log messages No action required. MB. Maximum logging regarding the current (as of directory size: {size} MB. log time) logging file and directory size, expected at least at the beginning of every log file.

[AUDIT]

"{subsystem id}" is at Log Informational message Use the Support Tools from Level {level} with trace mask: regarding the current (as of the Operations Console to {mask} log time) log level and modify log levels. trace mask of the subsystem noted.

STATS

{subsystem id/service} {stat type} statistics: {comma delineated statistics}

SNMP_CONNECTION

SNMP service has successfully Expected after start up. connected to the local SNMP Agent.

No action required.

Description

Resolution

Information message with Use the Operations Console the current (as of log time) to view statistics on demand. statistics for the subsystem/service noted.

IVR Error Messages Table 3: IVR Error Messages

Title

Cause

ALERT IVR_VIDEO_ CONNECTION_UP

Socket connection between CVP (IVR subsystem) and Radvision IVP (iContact Application) is established successfully.

Occurs when socket Not applicable. connection between IVR and iContact is established.

IVR_VIDEO_ CONNECTION_DOWN

Socket connection between CVP (IVR subsystem) and Radvision IVP (iContact Application) is down.

Occurs when socket connection between IVR and iContact is broken.

By default, IVR will reattempt to connect to iContact every two seconds. If connection stills fails after a few attempts, ensure that IVP is up and running using the IVP Manager. In IVP Manager, ensure that the iCONTACT application is loaded and activated. If the iCONTACT application is okay, ensure both sides are configured to use the same


141


Title

Description

Cause

Resolution

protocol (that is, either both sides use TCP or both sides use TLS). If both sides are using the same protocol, check IVP box with netstats to see if port 9981 (normal iCONTACT port) is opened. If it is opened, check the iCONTACT log for errors of not being able to establish a listen server socket. In the worse case scenario, users may be required to perform a hard reboot of IVP. IVR_VIDEO_ALERT

Read the specific error messages. If it says that the IVR service timed-out waiting for a connect response from iCONTACT, then the cause of this error occurs after the TCP connection is already established. This connection request is an iCONTACT api call after the TCP connection is already established. This should not happen unless something is wrong in iCONTACT.

If the error says that the IVR service timed-out waiting for a connect request response from iCONTACT, then:

Occurs if someone manually tampers with or moves the file. This error may also occur if file is not in Java properties file format. It is unlikely that anyone will make changes to this file unless they are providing writing to the Call Server VPD (Video Play Data) localization API.

Ensure the custom grammar properties file (customGrammar.properties) is located under \conf\customgrammar.

IVR_CANNOT_ This error occurs if someone is If a custom localizer class INSTANTIATE_CUSTOM_ developing a custom VPD is being developed, the LOCALIZER_CLASS (Video Play Media) possible causes are: localization class.

1. Ensure the name of the custom localizer class in customgrammar

IVR_CUSTOM_ LOCALIZER_FILE_ MISSING_OR_ UNREADABLE

Errors are encountered that prevent video call processing. Action must be taken immediately.

This error may occur if someone is developing a custom Video Play Media (VPD) localization class. Custom Grammar properties file may have been moved or removed, corrupted or not defined correctly.


142

1. Check the IVP box. If necessary, restart the IVP box using the IVP Manager. 2. If the error persists, reboot the IVP machine.

Ensure that any changes to the file are compliant with Java properties file format. Restart CallServer after this issue is fixed.


Title

Description

Cause

Resolution

1. The custom localizer class name may not match the actual class name. 2. The custom localizer class may not be in the class path of the Call server. 3. The custom localizer class may not be implementing the Localizer interface.

properties matches the actual class name. 2. Place the custom localizer class or jar file in the classpath of the Call Server as specified in the VPD localization located in the Configuration and Administration Guide for Cisco Unified Customer Voice Portal guide. 3. Ensure that the class implements the Localizer interface. Restart Call Server after any of the above issues are fixed.

IVR_LOCALIZER_ The VPD custom localizer INITIALIZATION_FAILED class returns 'false' from it's init( ) method.

This error usually only occurs if someone is developing a custom VPD (Video Play Media) localization class. It is caused when the custom localizer class returns a boolean 'false" from it's init( ) method.

IVR_API_ALERT

Investigate the custom Fix problem and restart Call VPD localizer class to Server if it is necessary to determine why this error is reload the class. occurring. Potential problems could be that some needed resource cannot be found or there is an internal error.

These errors are only logged by VPD localization classes. They are provided as a hook for custom localization classes to log errors that require immediate attention.

1. Check the custom localizer class code to see why it's returning 'false' from it's init( ) method. 2. Fix problem and restart CallServer to reload the class.

CRITICAL IVR_API_CRITICAL

These errors are only logged by VPD localization classes. They are provided as a hook for custom localization classes to log errors that may not require immediate attention but are critical to it's function.

Investigate the custom Fix problem and restart Call VPD localizer class to Server if it is necessary to determine why this error is reload the class. occurring. Potential problems could be that some needed resource cannot be found or there is an internal error.


143


Title

Description

Cause

Resolution

ERROR IVR_VIDEO_ERROR

Errors were encountered during There are many possible call processing. call processing errors. The log message has the details of the specific error. Examples include media not found, receiving error status codes from iCONTACT, no response from iCONTACT and so on.

Resolution depends on the specific error logged. For example, if the error is media not found, then ensure that the movies are located where they should be. If the error is caused by an error status code from iCONTACT, then investigate iCONTACT logs to see possible causes.

IVR_VIDEO_CALL_ TIMED_OUT

Error encountered when IVR A request to ICM from the Ensure that ICM PIM is up times out waiting for ICM subsystem has taken and running and ICM/IVR instructions from ICM. longer than expected. subsystems are in full service. Investigate possible network issues between CVP and ICM that may have caused the delay.

IVR_VIDEOSERVLET_ ERROR

Error logged by the video servlet component.

Most likely cause is that a request was received by the video servlet for action on a call session that does not exist.

IVR_VIDEOSERVLET_ INVALID_REQUEST

Error logged by the video servlet component when an invalid request is made.

The video servlet received Not applicable. a request that is not valid. For instance, this error is logged if a request is made to fast forward or rewind a live stream video which is not possible.

Not applicable. The call session may have ended before the request was received.

IVR_NO_AUDIO_LIST_ The VPD custom localizer did OR_TEXT_FROM_LOCALIZE not return localized text nor an audio file list to the Call Server for a VPD.

No audio list or text output Fix problem and restart Call received from localizer for Server if it is necessary to a particular data type for a reload the class. specific data.

IVR_API_ERROR

Investigate the custom Fix problem and restart Call VPD localizer class. Server if it is necessary to Potential problems include reload the class. that the custom localizer class could not return localized text for a VPD or could not generate the list of audio files for a VPD.

These errors are only logged by VPD localization classes. They are provided as a hook for custom localization classes to log errors during call processing.


144

Error Messages VMS Error Messages

VMS Error Messages Table 4: VMS Error Messages

Title

Description

Cause

Resolution

EMERGENCY VMS_SUBSYSTEM_INIT_ Unable to instantiate the media The internal media library Restart the Call Server MEDIA_LIBRARY_INVALID library. was not initialized in service. Check that the memory. cvp-vms-backend.jar is not corrupted. VMS_SUBSYSTEM_INIT_ Unable to create the search CREATE_INDEX_FAILURE index.

The search index was unable to be created.

VMS_SUBSYSTEM_ The CVP_HOME environment The subsystem could not PROPERTY_CVP_HOME_ variable value is missing or retrieve the value of the FAILURE invalid. CVP_HOME environment variable.

VMS_SUBSYSTEM_ PROPERTY_FAILURE

If VMS does not recover gracefully from this issue, go to the VMS index directory on the file system and delete all files within this directory. Determine if the environment variable CVP_HOME exists and its value is equivalent to the root CVP directory. Restart the Call Server service.

The defined VMS property is The listed property does not Verify the property exists missing or invalid. exist, has no value, or has within vms.properties and is an invalid value. associated with a valid value. ALERT

VMS_SUBSYSTEM_INIT_ The schema file property is SCHEMA_FILE_FAILURE missing or invalid.

The VMS schema file is missing or invalid.

Check in the VMS schema directory for the presence of the XML schema file. Determine if the contents of the schema file are valid XML.

VMS_SUBSYSTEM_INIT_ Unable to start the media START_POLLER_FAILURE poller.

The media poller was unable to be started.

Check that the user-selected VMS movies directory is present on the file system. Restart the Call Server service.

VMS_SUBSYSTEM_INIT_ Unable to create the media SETUP_POLLER_FAILURE poller.

The media poller was Check that the user-selected unable to be initialized in VMS movies directory is memory. present on the file system. Restart the Call Server service.

VMS_SUBSYSTEM_INIT_ Unable to create the media The media poller could not Check that the user-selected SETUP_POLLER_FACTORY_ poller because metadata factory be initialized. VMS movies directory is FAILURE creation failed. present on the file system. Check that the VMS XML schema exists on the file


145


Title

Description

Cause

Resolution

system and is a valid XML schema. Restart the Call Server service. VMS_SUBSYSTEM_INIT_ Unable to create the login SETUP_LOGIN_MANAGER_ manager. FAILURE

The login manager could not be initialized.

Check that the cvp-vms-backend.jar is present and not corrupted. Make sure that port 2100 is not being used by another application.

VMS_SUBSYSTEM_INIT_ An unexpected error occurred The subsystem failed to Restart the Call Server UNEXPECTED_ during VMS subsystem start due to an unexpected service. EXCEPTION initialization. error. VMS_SUBSYSTEM_INIT_ Unable to start the media poller The media poller manager Check for the existence and POLLER_MANAGER_ manager. failed to initialize. validity of the VMS XML FAILURE schema on the file system. Check in the logs that the media poller was started successfully. Check that the user-selected VMS movies directory is present. Restart the Call Server service. VMS_SUBSYSTEM_ UNSUPPORTED

VMS version is currently is unsupported.

An unsupported device Verify that all CVP attempted to interface with components that are VMS subsystem. communicating with VMS are of the supported version.

VMS_MEDIA_POLLER_ MANAGER_RESTART_ FAILURE

Unable to restart the media poller after X number of attempts.

The media poller failed to Restart the Call Server stop appropriately. service.

VMS_MEDIA_POLLER_ MANAGER_DEAD

The media poller is dead.

The media poller manager Check the logs if the media detected that the media poller manager restarted the poller is dead. media poller successfully after this message. Restart the Call Server service if needed.

VMS_SUBSYSTEM_ PROPERTY_MEDIA_ HOME_FAILURE

The media home directory is invalid.

The media home property Verify that could not be retrieved. VMS.darwin.home, VMS.darwin.app, and VMS.darwin.config are present with valid values in the vms.properties file. Restart the Call Server service.

VMS_SUBSYSTEM_ PROPERTY_DARWIN_ HOME_FAILURE

Unable to retrieve the Darwin The Darwin media media directory location. directory property could not be retrieved.


146

Verify that VMS.darwin.home, VMS.darwin.app, and VMS.darwin.config are present with valid values in the vms.properties file.


Title

Description

Cause

Resolution

Verify that the CVP Darwin Streaming Server directory is present. Verify that the Darwin Streaming Server streamingserver.xml contains an XML element pointing to the VMS movies directory. Restart the Call Server service. VMS_SUBSYSTEM_ LOCAL_IP_FAILURE

The local IP address could not The local IP address could Verify that the machine has be determined. not be determined. a valid network interface with an assigned IP address. Re-add the VMS device via OAMP. Restart the Call Server service.

VMS_SUBSYSTEM_ LOCAL_IP_INVALID

The local is invalid.

The local IP address is not Verify that the OAMP valid. configured IP address is a valid IP address or that the local IP address on local network interfaces are valid. Restart the Call Server service.

ERROR VMS_SUBSYSTEM_INIT_ Unable to start the Darwin START_DARWIN_FAILURE Streaming Server.

Not applicable.

Not applicable.

VMS_SUBSYSTEM_INIT_ Darwin Streaming Server has Not applicable. DARWIN_PARTIAL_ started, but is in partial service. SERVICE

Not applicable.

VMS_SUBSYSTEM_ RELOAD_MEDIA_ POLLER_FAILURE

The media poller could not be The media poller failed to reloaded. reload during a VMS administration reload process.

Verify the VMS movies directory is present. Verify the VMS XML schema is present and valid. Restart the Call Server service.

VMS_SUBSYSTEM_ RELOAD_MEDIA_ POLLER_MANAGER_ FAILURE

The media poller manager could not be reloaded.

Verify the VMS movies directory is present. Verify the VMS XML schema is present and valid. Restart the Call Server service.

VMS_MEDIA_POLLER_ PROCESS_FOLDER_ FAILURE

Failed to create folder media: Not applicable. .

VMS_MEDIA_POLLER_ PROCESS_MEDIA_ WITH_XML_READ_ FAILURE

Failed to read media: . metadata file could not be present on the file system and read from the file system. validates to the schema. Verify the VMS movies directory is present. Verify

The media poller manager failed to reload during a VMS administration reload process.

Not applicable.


147


Title

Description

Cause

Resolution

the VMS XML schema file is present and valid. Restart the Call Server service. VMS_MEDIA_POLLER_ Failed to create generic media. Not applicable. PROCESS_MEDIA_ONLY_ FAILURE

Not applicable.

VMS_MEDIA_POLLER_ PROCESS_XML_ONLY_ READ_FAILURE

Failed to read media.

Not applicable.

Not applicable.

VMS_SUBSYSTEM_ STREAMING_SERVER_ DOWN

The Streaming ServerState is Not applicable. down.

Not applicable.

VMS_ACTION_ EXCEPTION_CAUGHT

VMS caught an unhandled exception: .

Verify there are no errors in the Call Server Tomcat logs. Check for exception details in the CVP VMS logs. Restart the Call Server service.

An unhandled exception was caught via a web action.

VMS_ERROR_INVALID_ Got an invalid value for media The media home property Verify that VMS.darwin.app, MEDIA_HOME home from the subsystem is invalid or missing. VMS.darwin.config, and VMS.darwin.home are present in vms.properties with appropriate values. Verify CVP Darwin Streaming Server directory is present. Verify DSS streamingserver.xml contains valid reference to the user-selected movies directory. Restart the Call Server service. VMS_ERROR_COULD_ NOT_PARSE_MEDIA_ HOME

Could not parse the value for The Darwin Streaming Check for the Darwin media home. Server config file is invalid. Streaming Server streamingserver.xml configuration file presence. Verify the XML config file is properly formatted XML. Verify DSS streamingserver.xml contains valid reference to the user-selected movies directory. Restart the Call Server service.

VMS_ERROR_COULD_ Could not parse the value for The Darwin Streaming Check for the Darwin NOT_PARSE_RTSP_PORT the rtsp port. Server config file is invalid. Streaming Server streamingserver.xml configuration file presence.


148

Error Messages Reporting Server Error Messages

Title

Description

Cause

Resolution

Verify the XML config file is properly formatted XML. Verify DSS streamingserver.xml contains valid reference to the RTSP port. Restart the Call Server service. VMS_ERROR_NO_ RESOURCES_PARAM

Could not find entry in web.xml for

The resources parameter in Verify the web.xml is present the Tomcat web in the VMS web deployment deployment descriptor for in the WEB-INF directory. VMS is missing or invalid.

VMS_MISC_EXCEPTION An exception was caught: An exception was caught method - type by VMS. - , message -

Turn on TRACE_HANDLED_EXCEPTION tracing and troubleshoot the exception. Restart the Call Server service.

Reporting Server Error Messages Table 5: Reporting Server Error Messages

Emergency

Description

Cause

Resolution

ALERT Alert

Unable to start subsystem: {ss} The cause will be noted in The configuration must be the log message. fixed and call server must be restarted.

Alert

Reporting Server is not licensed

The license file is wrong

Check if the license file presences are correct. Connect to Cisco.

Alert

License file is invalid or does The license file is wrong not exist

Check if the license file presences are correct. Connect to Cisco.

Alert Alert Alert Alert Alert Alert Alert Alert Alert Alert Alert


149

Error Messages Reporting Server Error Messages

Emergency

Description

Cause

Resolution

Alert Alert Alert Alert ERRORS Error

Unable to stop: {message}

Error

Message adapter admin object Message adapter manager Check the infrastructures cannot be found can't find the message adapter admin object by SS name

Error

Unable to get Subsystem state: Any problem while trying Check the infrastructures {error_msg} to get SS state

Error

Can't fetch Subsystem Name from {msg}

Problem while trying to Check the infrastructures fetch SS name from tabular message

Error

Can't fetch Subsystem Type from {msg}

Problem while trying to Check the infrastructures fetch SS type from tabular message

Error

Can't fetch Message Bus Name Problem while trying to Check the infrastructures from {msg} fetch Message Bus Name from tabular message

Error

No Call Descriptor found for Can't find the call {msg} descriptor for message

N/A

Error

No message handler for {msg} Can't find the handler for this tabular message type

N/A

Error

The The Thread has been ReportingMessageProcessor: interrupted. {this} has been interrupted.

Restart the reporting server service.

Error

Error creating the client socket. Error creating the client socket with ReportingServer.

Check if ReportingServer service runs and do netstat, to see if the server socket opened for port 5503.

Error

Error sending a Start message Error sending a Start to Reporting Server. message to Reporting Server.


Error

Error sending an End message Error sending an End to Reporting Server. message to Reporting Server.


Error

Error closing the client socket. Error closing the client socket.

N/A


150

The cause will be noted in N/A the log message.

Error Messages ICM Error Messages

Emergency

Description

Cause

Resolution

Error Error Error Error Error Error Error Error

ICM Error Messages Table 6: ICM Error Messages

Title

Description

Comment

Resolution

Critical LOGMSG_ICM_SS_MSGBUS_SHUTDOWN Message Bus Connection is Down.

ICM subsystem received a Restart the CVP Call Server. plug-in down message from the message bus. Message bus may have shutdown due to an internal error.

LOGMSG_ICM_SS_PIM_SHUTDOWN Lost socket connection to VRU PIM. Transitioning to partial service.

ICM subsystem lost Check VRU PIM logs for connection with the VRU reason of disconnect. PIM. PIM may have been shutdown on purpose or a failure occurred in PIM. Secondary PIM should re-connect.

Alert LOGMSG_ICM_SS_STATE

VRU PIM connection is down. Transitioned to partial service.

ICM subsystem lost Check VRU PIM logs for connection with the VRU reason of disconnect. PIM. PIM may have been shutdown on purpose or a failure occurred in PIM. Secondary PIM should re-connect.

LOGMSG_ICM_SS_STATE

Waiting for VRU PIM connection.

ICM subsystem ready to accept connection from VRU PIM.

LOGMSG_ICM_SS_STATE

ICMSubsystem An error was encountered Check ICM SS configuration init() failed in initializing the ICM and restart Call Server subsystem. Most likely due to invalid configuration.

Make sure VRU PIM is configured to connect to call server on the same port.


151


Title

Description

Comment

Resolution

LOGMSG_ICM_SS_STATE

Transitioning to A critical thread No manual intervention partial service. deadlocked or terminated required. ICM SS will Reason = unexpectedly. transition to partial service and restart the critical thread.

LOGMSG_ICM_SS_STATE

Transitioned to Graceful shutdown partial service requested. due to graceful shutdown request.

Verify that graceful shutdown was intentional.

LOGMSG_ICM_SS_STATE

Transitioned to Graceful shutdown stopping due to requested. graceful shutdown request.


LOGMSG_ICM_SS_STATE

Transitioned to Graceful shutdown stopped due to requested. graceful shutdown request


LOGMSG_ICM_SS_STATE

Transitioned to Forced shutdown stopping due to requested. forced shutdown request.

Verify that forced shutdown was intentional.

LOGMSG_ICM_SS_STATE

Transitioned to Forced shutdown stopped due to requested. forced shutdown request

Verify that forced shutdown was intentional.

Error LOGMSG_ICM_SS_UNSUPPORTED_MSG %s Received Unsupported Message, Type=%s

Unknown/unhandled message received from ICM.

CVP should be connected to a supported ICM version. Check the CVP BOM for supported versions.

LOGMSG_ICM_SS_HEARTBEAT_FAILURE Message Bus Critical thread Message Listener missed Bus Listener either dead a heartbeat. locked or was terminated either unexpectedly or due to a normal shutdown request.

No manual intervention required. ICM SS will transition to partial service and restart the critical thread.

LOGMSG_ICM_SS_HEARTBEAT_FAILURE Message Bus Publisher missed a heartbeat.



152

Critical thread Message Bus Publisher either dead locked or was terminated either unexpectedly or due to a normal shutdown request.


Title

Description

Comment

Resolution

LOGMSG_ICM_SS_HEARTBEAT_FAILURE PIM Listener missed a heartbeat.

Critical thread PIM Listener either dead locked or was terminated either unexpectedly or due to a normal shutdown request.


LOGMSG_ICM_SS_HEARTBEAT_FAILURE PIM Publisher missed a heartbeat.

Critical thread PIM Publisher either dead locked or was terminated either unexpectedly or due to a normal shutdown request.


LOGMSG_ICM_SS_HEARTBEAT_FAILURE Admin Monitor Critical thread Message missed a Bus Listener either dead heartbeat. locked or was terminated either unexpectedly or due to a normal shutdown request.


LOGMSG_ICM_SS_HEARTBEAT_FAILURE Garbage Collector missed a heartbeat

Critical thread Garbage Collector either dead locked or was terminated either unexpectedly or due to a normal shutdown request.


LOGMSG_ICM_SS_EXCEPTION

An unexpected exception Forward logs to TAC/DE for occurred in the subsystem. analysis.

Stack trace

LOGMSG_ICM_SS_MSGLAYER_ERROR %s Unable to publish, receive or process message, Message=%s

There was an error publishing a message to the ICM PIM plug-in via the messaging layer

In most cases, these errors are normal since a connection may be terminating while messages are still being published for example. Restart call server. Forward logs to TAC/DE if errors are occurring while all connections are up and calls should be processed normally.

ICM_CALL_ERROR

ICM subsystem has run out of trunks. In most cases, this is due ICM SS not getting the proper CALL_STATE_EVENT from SIP/IVR/VXML subsystems to release the trunk it occupies when sending a NEW_CALL or REQUEST_INSTRUCTION message to ICM

This usually requires a DEBUG trace with Call trace level turned on in ICM subsystem to debug this issue to understand the call event flow. In a non-productions system, use diagnostic servlet to dump ICM calls to determine which calls are still active in the system. Capture a screen shot of this to track some of the call GUIDs in the

*** NO TRUNKS ARE AVAILABLE ***


153

Error Messages SIP Error Messages

Title

Description

Comment

Resolution

log. Depending on the call flow, also turn on Call Debug tracing for SIP/IVR/VXML subsystems to track the same call end to end to determine what’s happening in terms of event flow in the other subsystems for that call. ICM_CALL_ERROR

Removing call: due to exception. Exception:

ICM subsystem ran out of This is an internal error. trunks or ICM subsystem Forward logs to TAC/DE for threw an exception analysis. processing a call state event or run script result for a call and needs to remove the call from the call table.

ICM_CALL_ERROR

Current state of ICM service is . Rejecting new ICM label lookup request

ICM subsystem is not In Service and will reject the ICM label lookup request or a New Call request until it’s back In Service.

This message will most likely appear due to a timing issue where ICM SS is no longer connected to a VRU PIM and while it transitions into partial service and the state bubbles up to the dependent services such as SIP/IVR/VXML, some new calls and label lookup requests from VXML service may still get through and ICM SS will reject these requests. Make sure VRU PIM is configured to connect to call server on the same port. Check VRU PIM logs for reason of disconnect.

SIP Error Messages Table 7: SIP Error Messages

Title

Description

Comment

SIP_INTERNAL_ERROR

Failed to verify outbound proxy hostname with DNS Server: + {PROXY NAME}

Could not resolve Change the hostname on CVP Outbound Proxy with DNS. or else configure your DNS server to resolve the hostname.

SIP_INTERNAL_ERROR

No local static routes configured.

Non-proxy mode is Add local routes. OAMP selected but no local routes should not allow this to are configured on CVP. happen anyway.


154

Resolution


Title

Description

Comment

Resolution

SIP_INTERNAL_ERROR

B2BUA is not configured with B2BUA cannot route to the Add a route to the Error DN a route for making CVP Error DN. in OAMP. ERROR [+{ERROR DN HERE}+"] playback calls. Please add this route."

SIP_INTERNAL_ERROR

B2BUA is not configured with B2BUA cannot route to the Add a route to the Ringtone a route for making CVP Ringtone DN. DN in OAMP. RINGTONE [+{RINGTONE DN HERE}+"] playback calls. Please add this route."

SIP_INTERNAL_ERROR

B2BUA is not configured with B2BUA cannot route to the Add a route for this particular a route for making calls to returned label. label. [+{ICM LABEL HERE}+"]. Please add this route."

SIP_CALL_ERROR

{THE CALLGUID HERE} + No route configured for ": Destination URL is null, label. cannot make the transfer."

Add a route for this particular label.

SIP_CALL_ERROR

{THE CALLGUID HERE} + ": Aborting XFER. The call to the RINGTONE dial peer is exceeding 20 msecs. Check ringtone gateway(s), OR the thread pool for B2BUA is getting backlogged. Check STATS_RT_RUNNING_POOL_THREADS or increase thread pool size."

Confirm the routing in CUPS Proxy server is correct for Ringtone calls.

SIP_CALL_ERROR

{THE CALLGUID HERE} + No route for REFER label Add a route for this particular ": Destination URL is null, is found. label. cannot make the REFER."

SIP_CALL_ERROR

{THE CALLGUID HERE} + REFER operation was not ": Refer failed with "+sipcode successfully performed. +"-"+ DsSipResponseCode.getReasonPhrase(sipcode)+". May be a problem with DN Routes or Gateway Dail-Peers."

Check that the endpoint for the transfer is configured on the gateway or call manager. Perhaps an incoming or outgoing dial-peer is needed.

SIP_CALL_ERROR

{THE CALLGUID HERE} + Dialog Failure on the ICM " - DIALOGUE_FAILURE script was encountered and from ICM Router sends 500 B2BUA is ending the call. rejection to call."

The ICM script returned a dialog failure. If it is a timeout, confirm that the call is able to be sent to the VXML gateway in timely manner. Confirm that the Microapp is fetching correctly with the IVR service.

Thread to perform Ringtone during the transfer has waited 20 seconds and timed out.


155



156

Appendix B

Event and Cause Codes Event codes and cause codes provide information on the status of a particular operation within the Unified CVP environment. H.323 and SIP standard cause codes reflect the nature of the associated issue. For each of the issues, an ITU-T Q.850 release cause code is also assigned and maps the new standard categories with the Q.850 release cause code and description that is assigned to each category.

IVR Service Error Codes Table 8: IVR Service Error Codes

Error Code Name

Error Code Error Code Description Value

ERROR_NONE

0

No error.

ERROR_HANGUP

1

N/A

ERROR_NETWORK

2

Failure of an IP network connection.

ERROR_SYSTEM

3

Unexpected failure of an Unified CVP component.

ERROR_SCRIPT_TYPE

4

N/A

5

Micro-Application name passed from the Unified ICME to the Call Server did not exist on the Call Server.

6

VRU Script Name data passed from the Unified ICME to the Call Server does not contain the expected components (Micro-Application name, media file name, media file type, uniqueness value).

7

Data passed from the Unified ICME to the Call Server is not consistent with what the micro-application requires for processing.

8

An ECC variable was set to a value the Application Server did not recognize. ECC variable definitions must be the same in Unified ICME and Unified CVP.

ERROR_UNKNOWN_UAPP ERROR_SCRIPT_NAME

ERROR_CONFIG_PARAM

ERROR_MISCONFIG_ECC


157

Event and Cause Codes IVR Service Error Codes

Error Code Name

ERROR_MEDIA_FILE


9

Media file name passed from the Unified ICME to the Call Server did not exist on the Media Server.

10

N/A

11

The VoiceXML Interpreter did not recognize a VoiceXML format passed from the Call Server.

12

The VoiceXML Interpreter did not recognize a VoiceXML element passed from the Call Server.

13

The variable data passed was not valid for the script type being processed.

ERROR_NO_VAR_DATA

14

N/A

ERROR_FORMAT

15

N/A

16

Caller was unsuccessful in entering digits during each of the tries allowed by the Micro-Application.

17

Caller did not enter digits in response to the prompt for each of the tries allowed by the Micro-Application.

18

Unable to perform video-related request due to resource limitations.

19

Unable to start the recording or some type of recording failure.

20

N/A

21

Caller did not enter digits in response to the prompt in the time allowed by the Micro-Application.

ERROR_CALLED_HANGUP

22

N/A

ERROR_NO_ANSWER

23

No answer

ERROR_BUSY

24

Busy

ERROR_GEN_TRANSFER

25

General transfer error

ERROR_INVALID_EXTN

26

Invalid extension

ERROR_HANGUP_FORCED

27

Called party hung up

ERROR_AFTER_TRANS_ESTAB

28

Error after transfer established

30

The VoiceXML Interpreter did not recognize the locale passed from the Call Server.

31

TTS service was unable to perform a request due to resource limitations.

32

TTS service was unable to perform a request due to resource limitations. However, it may also indicate that an unsupported locale was specified.

33

N/A

40

Returned if the Call Server is unavailable (shutdown, network connection disabled, and so forth)

ERROR_SEMANTIC ERROR_VXML_FORMAT ERROR_VXML_ELEMENT ERROR_VARIABLE_DATA

ERROR_MAX_INVALID ERROR_MAX_NO_ENTRY MEDIA_RESOURCE_VIDEO RECORDING_FAILED ERROR_DATA_RANGE ERROR_TIMED_OUT

ERROR_UNSUPPORTED_LANGUAGE ERROR_MEDIA_RESOURCE_ASR ERROR_MEDIA_RESOURCE_TTS

ERROR_GENERAL_ASR_TTS ERROR_CVP_SYSTEM_UNAVAILABLE


158

Event and Cause Codes Unified ICME Error Codes

Error Code Name


ERROR_CVP_APP_ERROR ERROR_CVP_APP_HANGUP ERROR_CVP_APP_SUSPENDED

41

Returned if some VXML Server-specific error occurs (For example, java exception).

42

Returned if a hangup element is used, instead of subdialog return element, to send a Cisco application.

43

Returned if VXML Server application is suspended.

44

Returned when an emergency error occurs (for example, an application is called that has not been loaded in the VXML Server application).

45

Returned when the VXML Server encounters a bad fetch situation. This code is returned when either a wav file or an external grammar file is not found.

ERROR_CVP_NO_SESSION_ERROR

ERROR_CVP_BAD_FETCH

Unified ICME Error Codes Table 9: Unified ICME Error Codes

Error Code Name

Error Code Value

Error Code Description

NO_ERROR

0

No error occurred.

INVALID_VERSION

1

The version number requested by the Peripheral Gateway is not supported by the VRU.

SESSION_ALREADY_ACTIVE

2

The VRU already has an active communication session with a Peripheral Gateway.

VRU_OFFLINE

3

The VRU is unavailable.

SESSION_NOT_ACTIVE

4

No session is currently active.

INVALID_DIALED_NUMBER

5

The dialed number specified is not known to the Unified ICME.

EVENTS_NOT_SUPPORTED

6

The VRU does not support the Event Data Feed.

POLLING_NOT_SUPPORTED

7

(Obsolete)

ROUTING_NOT_SUPPORTED

8

The VRU does not support the Call Routing Interface.

TIME_SYNCH_NOT_SUPPORTED

9

The VRU does not support the Time Synchronization Interface.

TIMEOUT

10

A time-out has occurred.

PG_OFFLINE

11

The Peripheral Gateway is off-line.

REQUEST_REFUSED

12

The request was refused due to a temporary condition.

ROUTING_NOT_AVAILABLE

13

The Unified ICME routing service is not available.


159

Event and Cause Codes Unified ICME Error Codes

Error Code Name

Error Code Value


ROUTE_NOT_ACCEPTED

14

The VRU did not accept the supplied route.

UNSPECIFIED_FAILURE

15

An unspecified error occurred.

INVALID_INVOKEID

16

An invalid InvokeID has been specified in a result or response message.

SERVICE_CTRL_NOT_SUPPORTED

17

A request was made to initialize the Service Control Interface to a VRU that does not support the interface.

NO_SCRIPT

18

An ICM script has not been scheduled for the call type which the VRU is seeking instruction.

CALL_VARIABLE1

19

The VRU could not process a request due to invalid value in CallVariable1.

CALL_VARIABLE2

20


CALL_VARIABLE3

21


CALL_VARIABLE4

22


CALL_VARIABLE5

23


CALL_VARIABLE6

24


CALL_VARIABLE7

25


CALL_VARIABLE8

26


CALL_VARIABLE9

27


CALL_VARIABLE10

28


INVALID_SCRIPT

29

The Script ID specified in the RUN_SCRIPT_REQ message was invalid on the VRU.

INVALID_CALLID

30

The CallID specified in a Service Control request message not valid on the VRU.

DUPLICATE_DIALOGUEID

31

The dialogue identifier specified in either a NEW_CALL or REQUEST_INSTRUCTION message was not unique.


160

Event and Cause Codes SIP Codes

Error Code Name

Error Code Value


INVALID_MESSAGE

32

This error code may be sent in response to any message received with missing required floating fields.

INVALID_DIALOGUEID

33

The DialogueID specified in a request message is no longer valid.

OPERATION_CANCELLED

34

The specified operation was successfully cancelled by a cancel request.

OPERATION_NOT_CANCELLED

35

The specified operation could not be cancelled by a cancel request.

SIP Codes SIP Codes: SIP codes are available on the Cisco web site. To view the codes, go to the following link: http://www.cisco.com/en/US/docs/ios/12_2t/12_2t11/feature/guide/ftmap.html ( http:// www.cisco.com/en/US/docs/ios/12_2t/12_2t11/feature/guide/ftmap.html)


161

Event and Cause Codes SIP Codes


162

Appendix C

Troubleshooting Radvision Errors This appendix lists issues that would require Radvision support personnel to examine prior to involving Cisco TAC. Potential Radvision issues are as follows: • User Interface issues with IVP componets (for example, IVP Manager, MCU Admin GUI) • Alarms and Events in IVP Manager for various Radvision components • High CPU usage on IVP components • Recording anomalies or unexpected behaviors in recorded files • Error-notification messages in CVP logs


163

Troubleshooting Radvision Errors

– After an error-notification is sent by iCONTACT to CVP, iCONTACT will terminate the call. • Error-response messages seen in CVP logs with certain error codes If the error-code is one of internal-timeout, general-error, then this is most likely a Radvision internal issue. For example:


164

Appendix D

Accessing and Interpreting SNMP Data The Unified CVP SNMP agent allows customers and partners to integrate with their existing SNMP Network Management System (NMS) to provide instantaneous feedback on the health of their Unified CVP system. The Call Server, VXML Server, and Reporting Server can be configured to send SNMP traps and statistics to any standard SNMP Network Management System. You can configure a link to the administration web page for an SNMP monitoring tool and then access it by selecting SNMP Monitor from the Tools menu of the Operations Console. To add a URL link to a tool: 1. Choose Tools > Configure from the main Operations Console menu bar. The Configure Tools window opens, listing the current URL configured for each tool listed on the Tools menu. 2. In the Enter New URL text box for the tool you want to configure, enter the URL for each tool to launch. The web page indicated by this URL is launched when you select the tool from the Tools menu. 3. Click Save to save the URLs. Once you have configured the administration web page for your NMS, you can launch the SNMP monitoring tool from the Operations Console. The procedure below describes how to launch the tool. To launch an SNMP monitoring tool: 1. Choose Tools > SNMP Monitor from the main Operations Console menu.


165

Accessing and Interpreting SNMP Data

The table below describes the available severity levels and their descriptions for Unified CVP events. The severity level for an event can be set by editing the CVPLogMessages.xml file on the server that generated that particular event. Severity

Name

Description

0

Emergency

System or service unusable. Examples: • Service repeatedly fails to start up • System ran out of disk space while disk space is essential for this system to operate • Application requires root privileges to run but does not have them.

1

Alert

Action must be taken immediately. Examples: • Application is about to run out of license. • Application is about to run out of disk space. • Too many unauthorized access attempts detected. • Denial of service attack is detected.

2

Critical

Critical condition. Similar to alert, but not necessarily requiring an immediate action. Examples: • Received an invalid authentication request. • Service crashed due to an error that could not be handled, like an out of memory condition, (provided it has a watchdog process to restart it, it does not necessarily require immediate action). • Unexpected code error that could not be handled.

3

Error

An error condition, which does not necessarily impact the ability of the service to continue to function. Examples:


166


Severity

Name

Description

• Problem parsing/processing a particular request which does not prevent the application from handling other request. • Unexpected, but handled, code exception. 4

Warning

A warning about some condition, which is not necessarily an error. Examples: • Lost network connection to some resource. • Timed out waiting for a response.

5

Notice

Notifications about interesting system-level conditions, which are not error conditions. Examples: • Configuration was updated (not audit level information). • Process has halted. • Process is shutting down gracefully on request.

6

Informational

Informational messages are distinguished from notification in that they provide information for internal flows of the application or per-request information instead of system-wide notifications. Informational messages are used for troubleshooting by users who are familiar with the basic flows of the application. Examples: • Request received. • Request was parsed successfully. • Request being processed. • Response sent back. • Acknowledgment received. • Detailed audit information.


167


Severity

Name

Description

7

Debug

Debugging messages are similar to informational messages, but provide more detail and they require the user to have better knowledge of system internals. These messages are typically reserved for very advanced users or Cisco technical support. Examples: • Complete details for a request packet. • Internal state machine state changes. • Internal profiling statistics. • Internal events.

See Also For additional information, refer to the Configuration and Administration Guide for Unified Customer Voice Portal.


168

Appendix E

Internal Interfaces Description The following table summarizes the major interfaces between the Unified CVP product components and other Cisco products. Table 10: Major Internal Interfaces

Interface

Interface Characteristics

Between This Component... ...and This Component

Interface Type

Selection Determined By

Gateway (Ingress or Egress) Call Server

SIP

SIP Proxy Server or dial-peer configuration

H.323 Service

H.323

Gateway dial peers or Gatekeeper, based on H.323 Service availability and proximity

Gateway

Gatekeeper

H.323

Gateway configuration

H.323 Service

IVR Service

HTTP (URL with VXML response). H.323 Service client, IVR Service server.

H.323 Service configuration

Media Server

HTTP

URL is given in VXML generated by the IVR Service from information specified in Unified ICME.

Unified CM/IP Telephone Gateway Unified CM/IP Telephone

Normal web access (DNS, distributors, and so forth) is used to resolve it. Note: The H.323 to Media Server interface exists ONLY in legacy H.323 with Media


169

Internal Interfaces Description

Interface

Interface Characteristics

Termination call flow models. IVR Service

Unified ICME

ICM VRU Messaging

CVP configuration

Gateway

VXML Server

HTTP (URL with VXML response)

Unified ICME script configuration (though this does not apply for VXML Server (standalone))

Unified ICMH

Unified ICME

Proprietary

Customer identified by calling information; Unified ICMH configuration determines Unified ICME

Unified ICME

Customer database

SQL Gateway

Unified ICME Configuration

IVR Service

Unified ICME Configuration

Reporting Server

Informix DB

JDBC

Operations Console

VXML Server

Call Server

Proprietary

Operations Console, VoiceXML script, and Unified ICME script configuration

VMS

Call Server

HTTPS

MediaServer variable as configured in the ICM script or Call Server IVR tab.

VMS

IVP

RTSP/RTP

MediaServer variable as configured in the ICM script or Call Server IVR tab.


170

Appendix F

Trace Bits Description This appendix lists the common trace bits, their corresponding values and the description for each Unified CVP subsystem type. Each value is unique within each Unified CVP subsystem. Type

Trace Bits

Value

Description

Globally Reserved Trace Settings

GLOBAL RESERVED 3

0x00000200

Reserved for future use

GLOBAL RESERVED 4

0x00000400

GLOBAL RESERVED 5

0x00000800

GLOBAL RESERVED 6

0x00001000

GLOBAL RESERVED 7

0x00002000

GLOBAL RESERVED 8

0x00004000

GLOBAL RESERVED 9

0x00008000

Message Layer Trace MSGLAYER_MESSAGE Level Settings MSGLAYER_METHOD

0x0001000000L Trace mask for message values 0x0002000000L Trace mask for method values

MSGLAYER_HANDLED_EXCEPTION 0x0004000000L Trace mask for exception values MSGLAYER_PARAM

0x0008000000L Trace mask for param values

MSGLAYER_GLOBAL_EVENT 0x0010000000L Trace value from the old messaging layer code MSGLAYER_EXTERNAL_EVENT 0x0020000000L Trace value for external event MSGLAYER_STATIC_FIELD

0x0040000000L Trace value for static field

MSGLAYER_EXTERNAL_STATE 0x0080000000L Trace value for external state MSGLAYER_INTERNAL_STATE 0x0100000000L Trace value for internal state MSGLAYER_CODE_BRANCH 0x0200000000L Trace value for code branch MSGLAYER_CODE_MARKER 0x0400000000L Trace value for code marker MSGLAYER_CLASS_DUMP

0x0800000000L Trace value for class dump

MSGLAYER_LOCAL_DUMP

0x1000000000L Trace value for local dump


171

Trace Bits Description

Type

ORM Trace Level Settings

Subsystem Video Media Trace Level Settings

OAMP Trace Level Settings

Trace Bits

Value

Description

INPROCESS_MESSAGE

0x00100000L

Trace value for external event

INPROCESS_METHOD

0x00200000L

Trace value for static field

INPROCESS_HANDLED_EXCEPTION 0x00400000L

Trace value for external state

INPROCESS_PARAM

0x00800000L

Trace value for internal state

OUTPROCESS_MESSAGE

0x01000000L

Trace value for message traces

OUTPROCESS_METHOD

0x02000000L

Trace value for method input/output traces

OUTPROCESS_HANDLED_EXCEPTION 0x04000000L

Trace value for message handling exceptions

OUTPROCESS_PARAM

0x08000000L

Trace value for message parameters

TRACE_ORM_INFRA

0x00010000

Traces related to ORM startup, utilities, infrastructure

TRACE_ORM_FT

0x00020000

Traces related to File Transfer (both FTP and HTTP)

TRACE_ORM_IOS

0x00040000

Traces related to a IOS low level code logging

TRACE_ORM PC

0x00080000

Traces related to CVP services

RACE_ORM_PC_MSGBUS

0x00100000

Traces related to CVP Service messages sent across through messagebus

TRACE_ORM_PC_MSGBUS_HEARTBEA 0x00200000

Traces related to CVP Services heartbeat

TRACE_ORM_PC_CONTROLCENTER_STATS 0x00400000

Traces related to CVP Service Control Center Stats

Subsystem Init

0x00010000L

VMS subsystem initialization trace

Subsystem Shutdown

0x00020000L

VMS subsystem shutdown trace

Media Poller

0x00040000L

Media poller operations

Media

0x00080000L

Media modification operations

Search

0x00100000L

Search operations

Metadata Factory

0x00200000L

Metadata factory operations

VMS Web

0x01000000L

Outputs debug information from the VMS web code

Subsystem Updates

0x00800000L

Outputs debug information from the VMS Subsystem (other than startup/shutdown), such as state machine changes.

Streaming Server Poller

0x00400000L

Outputs debug information for the streaming server keep-alive thread (a thread that polls Darwin to make sure it's up and running).

MISC_CONFIGURATION

0x10000000L

Misc logging

DEVICE_CONFIGURATION

0x01000000L

Device logging

ST_CONFIGURATION

0x02000000L

ST logging

BULK_OPERATIONS

0x00100000L

Bulk operations


172


Type

Infrastructure Trace Level Settings

Trace Bits

Value

Description

BULK_EXCEPTION_STACKTRACE 0x00200000L

Bulk exception stack traces

DATABASE_MODIFY

0x00010000L

DB update, insert, delete logging

DATABASE_SELECT

0x00020000L

DB select logging

DATABASE_PO_INFO

0x00040000L

Persistent Object ID and class details

MANAGEMENT

0x00001000L

MGMT logging

LOAD_SUBSYSTEM

0x00010000

Used when loading subsystems

TRACE_THREAD

0x00020000

All Infrastructure thread operations

TRACE_MSG

0x00040000

Information about messages sent and received

TRACE_MSG_DETAIL

0x00080000

Detailed information about messages sent and received

TRACE_MESSAGE_HANDLING 0x00100000

Information about the handling of the messages

TRACE_TIMER

0x00200000

Information about when a timer expires

TRACE_STATE

0x00400000

Information about state changes and events

TRACE_SECURITY

0x00800000

Information about all security operations

TRACE_LICENSING

0x01000000

Information about all licensing operations

TRACE_STARTUP

0x02000000

Information about detailed startup information

TRACE_SHUTDOWN

0x04000000

Detailed shutdown information

TRACE_STATS

0x08000000

Detailed information about the Stats Manager operations

TRACE_SNMP

0x10000000

SNMP forwarder logging statistics

0x00010000

Traces related to a JDBC processing

0x00040000

Traces related to a CallRegistry processing

0x00800000

Traces related to load processing

0x00000001

Traces related to a call/call processing

TRACE_METHOD

0x00000002

Traces when entering/exiting a method

TRACE_PARAM

0x00000004

Traces for any parameters (not just method arguments)

TRACE_LOW_LEVEL

0x00000008

Traces for bits and bytes

TRACE_CLASSDUMP

0x00000010

All information available about a class

TRACE_HEARTBEAT

0x00000020

Traces related to heartbeats, heartbeat threads, or heartbeats sent/received

Reporting Service TRACE_JDBC Trace Level Settings TRACE_CALL_REG TRACE_BATCH Common Trace Bits TRACE_CALL

TRACE_HANDLED_EXCEPTION 0x00000040

Traces related to the exception and how it was handled

TRACE_OOOQUEUE

Traces related to OoOQueue tracing

0x00000080

TRACE_GARBAGE_COLLECTOR 0x00000100

Traces related to GarbageCollector tracing


173


Type

Trace Bits

Value

Description

TRACE_MESSAGE

0x00000200

General debug details of incoming/outgoing messages

TRACE_DEBUG_STATS

0x00000400

Enables printing of debug statistics


174

Appendix G

Debugging This appendix discusses various debugging facilities that are available to help you diagnose Unified CVP problems.

SIP IOS Debugging The Session Initiation Protocol (SIP), is an application-layer control (signaling) protocol for creating, modifying, and terminating sessions with one or more participants. These sessions include Internet telephone calls, multimedia distribution, and multimedia conferences. SIP is a simple, ASCII-based protocol that uses requests and responses to establish communication among the various components in the network and to ultimately establish a conference between two or more endpoints. Users in a SIP network are identified by unique SIP addresses. A SIP address is similar to an e-mail address and is in the format of sip:[email protected]. The user ID can be either a user name or an E.164 address. Users register with a registrar server using their assigned SIP addresses. The registrar server provides this information to the location server upon request. When a user initiates a call, a SIP request is sent to a SIP server (either a proxy or a redirect server). The request includes the address of the caller (in the From header field) and the address of the intended called party (in the To header field). Over time, a SIP end user might move between end systems. The location of the end user can be dynamically registered with the SIP server. The location server can use one or more protocols (including finger, rwhois, and LDAP) to locate the end user. Because the end user can be logged in at more than one station and because the location server can sometimes have inaccurate information, it might return more than one address for the end user. If the request is coming through a SIP proxy server, the proxy server tries each of the returned addresses until it locates the end user. If the request is coming through a SIP redirect server, the redirect server forwards all the addresses to the caller in the Contact header field of the invitation response.


175

Debugging H.323 IOS Debugging

SIP IOS Troubleshooting Commands The table below lists several debug commands that are useful for troubleshooting problems with SIP. Use the no form of the command to disable the debug output. For example, no debug ccsip all Debug Command

Description

debug ccsip all

Displays all Session Initiation Protocol (SIP)-related debugging output.

debug ccsip calls

Displays all Session Initiation Protocol (SIP) Service Provider Interface (SPI) call tracing information.

debug ccsip error

Shows all Session Initiation Protocol (SIP) Service Provider Interface (SPI) errors.

debug ccsip events

Enables tracing of events that are specific to the service provider interface (SPI).

debug ccsip info

Enable tracing of general information for the service provider interface (SPI).

debug ccsip media

Enables tracing of the Session Limitation Protocol (SIP) call media streams.

debug ccsip messages

Displays all Session Initiation Protocol (SIP) Service Provider Interface (SPI) message tracing information.

debug ccsip preauth

Enables the diagnostic reporting of authentication, authorization, and accounting (AAA) preauthentication for Session Initiation Protocol (SIP) calls.

debug ccsip states

Displays all Session Initiation Protocol (SIP) Service Provider Interface (SPI) state tracing information.

show sip status

Displays the Session Initiation Protocol (SIP) user agent listener status.

show sip statistics

Displays the Session Initiation Protocol (SIP) user agent statistics.

For additional information on these and other Cisco IOS commands, refer to the Cisco IOS Debug Command Reference (http://www.cisco.com/en/US/products/sw/iosswrel/ps5187/ products_command_reference_book09186a008017cf4d.html) Note: This section assumes that you possess basic proficiency in the use of Cisco IOS software commands.

H.323 IOS Debugging The Cisco Internet Operating System (IOS) is very complex, with many commands and configuration possibilities. Many of these have no direct relationship to Unified CVP. There


176


are some basic IOS configuration settings and tools that may be helpful. Below are some of the more common and useful commands. Debug Command

Description

show run

Allows the user with EXEC privileges to view the running configuration. To gain EXEC privileges, use the enable EXEC command. For additional information on the enable EXEC command, refer to the Cisco IOS Command Reference document.

show ver

Allows the user to obtain the current IOS version and specific hardware information.

show proc cpu hist

The command displays an ASCII graphical form of the total CPU usage on the device over a period of time.

show Gatekeeper endpoints

Displays the status of all the registered endpoints for a Gatekeeper.

show proc cpu sorted

Displays detailed CPU utilization statistics (CPU use per process) that are sorted by percentage.

show memory statistics

Displays a view of the memory information.

copy tftp flash

Run this command to copy a file from a tftp server into flash.

copy flash tftp

Run this command to copy files from flash to a tftp server.

show Gatekeeper gw-type-prefix

Use this command to check the priority of the gateways registered with the Gatekeeper. Check to see if any are marked as "out-of-resources."

show standby

Displays which Gatekeeper is primary and which is standby.


177



178

Index

Cannot connect....36

Index

IVR error messages....141 media poller....35

audio problems troubleshooting....60

movie media....34 Overview

Call Server

Troubleshooting overview....5

device state....16

Radvision Errors....163

troubleshooting....13

Reporting Server

Cisco Unified Presence Server troubleshooting....128 Content Services Switch troubleshooting....98 Debugging....175

troubleshooting....54 Reporting Server Error Messages....149 SIP SIP IOS Debugging....175 troubleshooting....109

H.323 IOS....175

SIP Error Message....154

SIP IOS....175

SNMP

device status....10

accessing and intrepreting SNMP data....165

DNIS....31


dropping calls full video....31 Error Messages infrastructure....135

Support Tools troubleshooting....131 Symptoms and Solutions Troubleshooting Solutions....13

IVR....135

TLS....32

VMS....135

Trace bits....171

Event and Cause Codes....157 H.323

Troubleshooting Audio Problems....60

H.323 IOS Debugging....176

Backup problems....131


Call Server....13

hold movie....30

checklist....11

ICM Error Messages....151

Cisco Unified Presence Server....128

Infrastructure error messages....135

Connect problems....77

Installation

Content Service Switch....98


Gateway ....124

internal interfaces....169

General system problems....98

invalid images....35

H.323 ....105

IVP Server

Reporting Server....54


Index 179

Index

Restore problems....131 SIP ....109 SNMP....117 Support Tools ....131 Transfer problems....77 Unified CM....91 Unified ICME....91 VXML Server....38 Unified CM troubleshooting....91 Unified ICME troubleshooting....91 video content while agent previewing....37 while agent pushing....37 video termination....32 VMS error messages....145 VXML Server troubleshooting....38


Index 180