mySAP CRM Monitoring

Best Practice: mySAP CRM Monitoring

mySAP CRM Monitoring Best Practice for Solution Management Version Date: November 2002 The newest version of this Best Practice can always be obtained through the SAP Solution Manager CRM 3.0

Contents Applicability, Goals and Requirements _________________________________________________________3

2

1.1

Goal of Using this Service ___________________________________________________________3

1.2

Staff and Skills Requirements _______________________________________________________3

1.3

Example of a Business Scenario ______________________________________________________3

Best Practice Procedure and Verification ___________________________________________________6 2.1 Preliminary Tasks _________________________________________________________________6 2.1.1 Monitoring Concepts____________________________________________________________6 2.2 Procedures ______________________________________________________________________10 2.2.1 R/3 Back-End System Monitoring ________________________________________________10 2.2.2 CRM Server Monitoring ________________________________________________________15 2.2.3 Field Sales Specific Monitoring __________________________________________________23 2.2.3.1 CRM Server________________________________________________________________23 2.2.3.2 Communication Station _______________________________________________________25 2.2.3.3 Mobile Client Monitoring _____________________________________________________28 2.2.4 Interaction Center Specific Monitoring_____________________________________________29 2.2.4.1 CRM Server________________________________________________________________29 2.3 Verification _____________________________________________________________________31 2.3.1 Middleware Portal – SMWP _____________________________________________________32 2.3.2 Monitoring CRM Outbound Queues – SMQ1________________________________________33 2.3.3 QOUT Scheduler – SMQS ______________________________________________________36 2.3.4 Monitoring CRM Inbound Queues – SMQ2 _________________________________________37 2.3.5 Scheduler Status – SMQR _______________________________________________________39 2.3.6 TRFC Monitoring – SM58 ______________________________________________________40 2.3.7 Monitoring the Middleware Message Flow Statistics – SMWMFLOW ____________________41 2.3.8 Check Flow Definitions – SMO8FD _______________________________________________42 2.3.9 Middleware Trace _____________________________________________________________43 2.3.10 Monitor Request – R3AR3 ______________________________________________________43 2.3.11 Monitor Initial Load – R3AM1 ___________________________________________________43 2.3.12 Monitoring the Replication & Realignment Queues – SMOHQUEUE ____________________44 2.3.13 Queue Information for CRM Mobile Sites – SMWMQUEUES __________________________45 2.3.14 Communication Monitor – SMWMCOMM _________________________________________46 2.3.15 Message Recovery – CMWQ ____________________________________________________47 2.3.16 DCOM Connector Monitor ______________________________________________________47 2.3.17 Communication Station Log File: TransferService.Log ________________________________48 2.3.18 Windows Performance Monitor __________________________________________________49 2.3.19 SAP Connect Monitor – SCOT ___________________________________________________50 2.3.20 SAP Phone Administration – SPHB _______________________________________________51

© 2002 SAP AG

1

Best Practice: mySAP CRM Field Sales Monitoring 2.3.21 2.3.22 2.3.23 2.3.24 2.3.25 2.3.26 2.3.27 2.3.28 2.3.29 2.3.30 2.3.31

Internet Communication Manager Monitor – SMICM _________________________________52 Gateway Monitor – SMGW _____________________________________________________53 R/3 Buffer Monitor – ST02 ______________________________________________________53 Workload Monitor – ST03N _____________________________________________________53 Database Performance Analysis – ST04 ____________________________________________54 Operating System Monitor – ST06 ________________________________________________55 Local and System-Wide Work Process Overview – SM50 / SM66 _______________________55 Display Statistical Records – STAD _______________________________________________56 ABAP Dump Analysis – ST22 ___________________________________________________56 System Log – SM21 ___________________________________________________________57 Update Error Monitor – SM13 ___________________________________________________58

2.4 Troubleshooting__________________________________________________________________59 2.4.1 Problems during Data Exchange __________________________________________________59 2.4.1.1 Error Detection in Delta Download______________________________________________59 2.4.1.2 Error Detection in Initial Download _____________________________________________65 2.4.1.3 Error Detection in Upload _____________________________________________________68 2.4.2 Problems with E-Mail Sending (Outbound Direction) _________________________________71 2.4.3 CRM Server Performance problems or high I/O load due to excessive traces and logs being written 72 2.4.4 Mass changes of data (creating, changing) on the OLTP system leads to reduced system performance__________________________________________________________________________74 2.4.5 Performance problems due to statistics updates on tRFC/qRFC tables_____________________74 2.4.6 System performance degrades as the size of the tRFC/qRFC tables increases _______________74 2.4.7 Problem situation 1 ____________________________________________________________75 2.4.8 Problem situation 2 ____________________________________________________________75 2.4.9 Problem situation 3 ____________________________________________________________75 2.4.10 Problem situation 4 ____________________________________________________________75 2 ______________________________________________________________________________________77 Feedback and Questions ________________________________________________________________77

© 2002 SAP AG

2

Best Practice: mySAP CRM Field Sales Monitoring

Applicability, Goals and Requirements A Best Practice may not be applicable in some situations. To ensure that this Best Practice is the one that is needed in your situation, consider the following goals and requirements.

1.1 Goal of Using this Service The goal of this Best Practice is to document and to recommend procedures for: •

Identifying system conditions that can lead to business process standstill, loss of production or inefficient system operations

•

Facilitating good system operations practices

•

Troubleshooting system problems

The general monitoring goals are: •

Detect documents in an error status

•

Refer to reliable procedures for error handling

• Measure performance of data exchanges Monitoring objects and their attributes are usually displayed in the alert monitors. However, some interface points must be checked using generic tools and procedures. These are also described in this document.

1.2 Staff and Skills Requirements System administrators with experience for your installed my SAP CRM-Scenarios like e.g. mySAP CRM Field Sales, mySAP CRM Field Service solutions, mySAP Interaction Center solutions and CRM Online.

1.3 Example of a Business Scenario The example below describes a business process typical for companies with a field force organization and covers the basic tasks in the daily work of a sales representative like e.g. entering new sales orders. The business process flow for Sales Order Processing in the Field Sales environment is depicted on the following page. Business Process Flow Description 1. Sales person creates an order for one of his customers. 2. Once a day the sales person connects to the CRM Server to upload his sales orders. 3. The sales orders are transferred to the CRM Server and passed through the validation process, which then decides to either update the Consolidated Database (CDB) or to forward rejections. In case of an update, all subscribed receivers are provided with the order. One of the receivers can be e.g. the R/3 back-end system. 4. If the R/3 back-end system is subscribed to these data, an SD sales order is created based on the information provided by the CRM Server. 5. During this process within the R/3 back-end system, the sales order is created or rejected, e.g. because of the exceeding the credit limit. 6. The R/3 back-end system sends confirmation or refusal information to the CRM Server.

© 2002 SAP AG

3

Best Practice: mySAP CRM Field Sales Monitoring 7. The CRM Server updates the CRM Online database and after replication to the CDB puts the information to update the mobile client’s database into the client’s outbound queue on the CRM Server. 8. The mobile client picks up its data from the CRM Server.

SAP CRM Mobile

SAP CRM

SAP R/3

Create inquiry Create inquiry Create quotation

Create sales order

Print sales order

Create quotation

Create sales order Create sales order

Mark for upload

Save sales order to CDB

Perform upload (connTrans)

Perform download (connTrans)

IPC

Replicate to mobile clients

IPC

Configure product Configure product Determine price Determine price

Technical Process Flow Overview The business process described above is technically represented by the message flow used for transferring data from the mobile client to the CRM Server and to the R/3 back-end system. The flow is based on the standard SAP business process steps of the “Standard Sales Order Processing” Scenario. This general description can be applied to most communications between the mobile clients and the R/3 back-end and/or CRM Servers. “Sales orders” are represented by the corresponding BDoc message type being transferred between the mobile client and the CRM Server. BDoc types can also transfer all types of data between the mobile client and the CRM Server (such as system information and administrative information) and then to all other receivers. Message Flow Within CRM In all CRM scenarios, message flow is used to supply all components of the CRM landscape with necessary information in accordance with subscription rules. The figure below shows message flow for CRM Field Sales scenario.

© 2002 SAP AG

4


R/3 Adapter

Mobile Client

R/3 External interfaces (XML - SOAP, IDoc)

XIF Adapter

Both XML/SOAP and IDocs can be generated from the ABAP complex data type used to define the external interface

Sync BDoc Flow

Mapping: Sync BDoc -> Msg BDoc Msg BDoc Flow

CRM Adapter

CRM Server Application

CRM Adapter Routing (Simple Replication)

XIF Adapter R/3 Adapter Mobile Bridge: Msg BDoc -> Sync BDoc

External interfaces (XML – SOAP, IDoc) R/3 Sync BDoc Flow CDB Service Replication/ Realignment Outbound Adapter

Mobile Client

CRM Middleware Adapters and Services •

The CRM Adapter is called by the message flow to pass inbound BDoc messages to the CRM Server application for validation. In case of a successful validation by the CRM Server applications, the content of the BDoc message is written (from the extension part) to the corresponding tables of the CRM database. If the validation was not successful, the BDoc message is sent back to the sender.

•

The Replication and Realignment Service determines whether a replication and/or a realignment is necessary or not. If a realignment has to be performed for a BDoc message, this message is copied into a separate realignment queue for further processing. If realignment is not required, the receiving sites for a BDoc message are determined.

•

The Mapping Service maps synchronization BDoc types to messaging BDoc types. The reverse direction is mapped using the Mobile Bridge. The Mobile Bridge takes a messaging BDoc type and creates one or more synchronization BDoc types (1:n relationship). It also takes one or more synchronization BDoc message and produces exactly one messaging BDoc message of one predefined type (n:1 relationship).

•

The CDB Service saves the content of a synchronization BDoc message in the corresponding CDB tables.

Note: Mobile Bridge and CDB Service are only active in Field Sales (=MSA) or Field Service (=MSE) Scenarios. (CDB = Consolidated Database on the CRM Server, MTS = Message Transfer Service. MTS client is on the Mobile Client, MTS server - on the Communication Station)

© 2002 SAP AG

5


2 Best Practice Procedure and Verification 2.1 Preliminary Tasks Before applying this Best Practice, ensure that you perform the following preliminary tasks or checks in the system: •

Complete all installation and post-installation actions and procedures including customizing.

•

Ensure that the initial load has been successfully executed.

•

Apply all SAP recommendations from SAP Service Sessions and any SAP recommendations resulting from customer problem messages.

•

Implement all current SAP Support Packages upon availability.

2.1.1 Monitoring Concepts Monitoring in the mySAP CRM can be divided into the following areas: •

Monitoring the end-to-end message flow in the CRM Server

•

Monitoring transactional RFC requests

•

Monitoring the queues located on the respective server systems

•

Operating system performance monitoring

•

Database system performance monitoring

•

Error analysis

Detected errors

Actions

Undetected errors

Prevention by regular monitoring

Automatically generated

Use of processing monitors

Standard R/3 tools

Computing Center Management System

E.g. message to system administrator

è

Middleware Portal

è

Display BDoc Messages

è

Monitoring of queued RFC

è

Middleware Trace

è

RFC log files

The following table briefly lists the Software Components and the monitoring functions associated with them:

© 2002 SAP AG

6


Server Types

Monitoring Function

Scenario

Mobile Client

1. Monitor the outbound queue of the mobile client

Only MSA/ MSE

2. Monitor the inbound queue of the mobile client 3. Monitor database 4. Monitor the operating system 5. Monitor network throughput 6. Message Recovery Communicatio n Station

Only MSA/ MSE

1. Monitor message flow from all mobile clients to the CRM Server 2. Monitor network throughput 3. Monitor operating system

R/3 Back-End System

all, if R/3Back end is connected

1. Monitor outbound queue 2. Monitor R/3 Basis System 3. Monitor database 4. Monitor operating system 5. Monitor network throughput

CRM Server

1. Start with the Middleware Portal. In case of an error go to the appropriate transactions. 2. Monitor outbound queue, data transfer from the CRM Server to mobile clients 3. Monitor inbound queue, data transfer from mobile clients to the CRM Server 4. Monitor outbound queue, data transfer from the CRM Server to the R/3 back-end system 5. Monitor inbound queue, data transfer from the R/3 back-end system to the CRM Server 6. Monitor replication and realignment queues (only MSA/ MSE) 7. Monitor Middleware message flow 8. Monitor flow control 9. Monitor the Middleware log 10. Monitor status of BDoc types 11. Monitor initial/delta load / requests from the R/3 back-end system to the CRM or from the CRM to the CDB (only MSA/ MSE) In addition to the Middleware monitors, the SAP basis has to be monitored: 1. Monitor R/3 Basis System 2. Monitor SAP Connect interface for mailing and/or fax (if used in CRM Online or Interaction Center Scenarios) 3. Monitor CTI interface (if used in Interaction Center scenario) 4. Monitor database

© 2002 SAP AG

7


Server Types

Monitoring Function

Scenario

5. Monitor operating system 6. Monitor network throughput

© 2002 SAP AG

8

Best Practice: mySAP CRM Monitoring

Business Process Monitoring in the CRM – MSA Server Landscape Example DCOM Connector Monitor TransferService.log

Comm. Station

CRM MW System Outbound Queue

Inbound Business Processing Queue Outbound Queue

SMQ2

SMQ1/SMQS

QMTCHECK

Mobile Client(s)

SMQ2 SMQR SMWMQUEUES

DCOM Connector

Inbound Queue

R/3 System Inbound Queue Outbound Queue

SAP Business Processing

Replication and Realignment Queue

DB & Log

Log file

Flow / Process Control Engine & other services R/3 Application Database CDB

MS Windows NT Performance Monitor


SMWMFLOW SMO8FT SMW01/SMW02 SMOHQUEUE SM58

SMQ1 / SMQS ST03 ST06 SM58 ST22

Non CRM-specific SAP System monitoring transactions: ST02, ST04, SM21, SMGW, SM13, STAD © 2002 SAP AG 9

DB02 SM50 / SM66 SMGW


2.2 Procedures 2.2.1 R/3 Back-End System Monitoring After the CRM Server has been installed and the initial load has been successfully performed, system load (data exchange) to and from the R/3 back-end system can be categorized as follows: R/3 Back-End System à CRM Server •

Delta load (business data and conditions)

•

Request of specific data

•

Synchronization load (customizing data only)

•

“Regular” communications using RFC calls

CRM Server à R/3 Back-End System •

Delta uploads from CRM via outbound adapter

•

“Regular” communications using RFC calls

The following diagram shows the main points of interface on the R/3 back-end system. The transactions used to monitor these interface points are listed along with a brief label describing their function.

© 2002 SAP AG

10


Monitoring R/3 System SMQ2

Mobile Client(s)

Comm. Station

R/3 System

Outbound Queue


CRM MW System

DCOM Connector

Inbound Queue

Inbound Queue

Outbound Queue



DB & Log

Log file


SMQ1 / SMQS ST03N ST06 SM58 ST22

© 2002 SAP AG

11

DB02 SM50 / SM66 SMGW

Best Practice: mySAP CRM Field Sales Monitoring The table below lists transactions for monitoring the R/3 back-end system: Middleware Specific Monitoring R/3 Back-End System Monitoring Queue Monitoring Activities / Functions

Transaction Code

Monitoring Frequency Periods and Events

qRFC Outbound Queue Monitor:

SMQ1

•

Several times a day depending on the business process

•

Monitor data exchange from the R/3 back-end system to the CRM Server

•

Queues should be relatively short and quickly processed

•

In case of an error message

•

Check, if the qRFC Version is up to date (see SAP Note 438015)

•

During mass updates

•

Use Alert Monitoring

•

In case of a status error

•

Use Alert Monitoring

•

To prevent data inconsistencies, you need to monitor the interfaces regularly for aborted or stopped data transfer Status of Queue Scheduler •

SMQS

Monitor status of the QOUT Scheduler

Basis Monitoring: SAP Performance Monitors R/3 Back-End System General Monitoring Activities / Functions

Transaction Code


R/3 System Buffer Monitor:

ST02

•

Weekly

•

In case of performance problems

•

Daily

•


•

Daily

•


•

Monitor memory resource usage for specific R/3 application servers

•

Implement parameter recommendations for memory management per EarlyWatch analysis

•

Use “normal” R/3 system monitoring practices

R/3 System Workload Analysis: •

Monitor RFC response time statistics for CRM

•

Monitor the Dialog response time for online transactions

Database Performance Analysis:

•

Monitor database statistics

•

Monitor the buffer cache hit ratio

© 2002 SAP AG

ST03N

ST04

12


R/3 Back-End System General Monitoring Activities / Functions

Transaction Code


Database Performance Monitor

DB02

•

Daily

•


•


•


•

Monitor the growing of tables and indices, especially on tRFC- and qRFC-tables

Operating System Monitor: •

ST06

Monitor hardware load during high RFC transmission times

Basis Monitoring: General System Checks R/3 Back-End System General Monitoring Activities / Functions

Transaction Code

Monitoring Frequency Periods and

Local Work Process Overview:

SM50

•


•

Daily

•

In case of performance problems or error messages

•


•


•


•


•

Daily

•


•

Daily

•


•


•

Monitor current state of individual work processes

•

Ensure that enough work process capacity is available at peak times

System-wide Work Process Overview: •

Similar to SM50 but for system-wide statistics

ABAP Dump Analysis: •

SM13

Check for entries that have “ERR” status

Display Statistical Records: •

SM21

Check for general system errors

Update Errors: •

ST22

Analyze aborted programs

System Log: •

SM66

Determine the performance of single statistical records

© 2002 SAP AG

STAD

13


R/3 Back-End System Parameter Settings The parameters for data exchange with an R/3 back-end are defined in tables CRMCONSUM and CRMRFCPAR in the R/3 back-end. These tables are customized during system installation and are usually not changed during system operations. CRMRFCPAR controls the RFC traffic between the R/3 back-end system and the CRM Server. The following table gives an example how to fill the CRMRFCPAR table. The actual CRMRFCPAR entries are client dependent and significantly larger than this example. If you want to, for example, provide a CRM system with data, you have to create the following entries: Example for CRMRFCPAR Settings Parameter Name

Value

Description

RFC QUEUE

CRM

Consumer that uses the CRM plug-in functions as a data receivers (e.g. CRM)

OBJNAME

*

Object name * = valid for all objects

RFCDEST

CRM__

Specifies the RFC destination of the CRM Server.

DOWNLOAD

*

Restricts CRMRFCPAR entries to the initial or delta load. Possible values are: * valid for all kinds of data transfer I only for initial data transfer D for delta data transfer R for request With initial transfers, data can be sent to two destinations but with delta transfers data can only be sent to one destination. Standard value: *

USE IN Q

X

Controls whether qRFC inbound queues are used on the CRM Server

SEND XML

M

“M”* – mixed mode (recommended) “space”* - no XML-conversion. This is only possible if the sending and receiving CPUs have the same architecture (homogeneous system landscape). “X” -XML will be used. To improve performance in data exchange and further information on this setting, refer to note 442277 in the SAP Service Marketplace

© 2002 SAP AG

14


Scheduling Background Jobs There are no CRM-specific background jobs to schedule on the R/3 back-end system.

2.2.2 CRM Server Monitoring The following diagram shows the interfaces on the CRM Server. The transactions used to monitor these interface points are listed along with a brief label describing their function.

© 2002 SAP AG

15


Monitoring CRM Middleware Server

SMQ2 SMQR SMWMQUEUES SMQ1/SMQS

Mobile Client(s)

Comm. Station

Outbound Queue


CRM MW System

DCOM Connector

R/3 System Inbound Queue

Inbound Queue

Outbound Queue



DB & Log

Log file


SMWMFLOW SMO8FT SMW01/SMW02 SMOHQUEUE SM58

© 2002 SAP AG

16


The table below lists transactions used for monitoring the CRM Server: CRM Server Central Monitoring Activities / Functions

Monitoring Type


Middleware Portal

SMWP

•


•

After implementing transports

•


•

Daily

Monitoring the CRM Middleware Portal with transaction SMWP can be subdivided into the following areas: •

Generation Information containing: BDocs Types: Generation of structures BDocs Types: Generation of other runtime objects Replication objects Publications: Missing Indexes, Status of Delta/Initial Generation, Flow Definitions

•

Runtime Information containing: Adapter Status Information Replication and Realignment Queues BDoc Messages in the Flow

qRFC Outbound Queue Monitor: •

Monitor data transfer between the R/3 back-end and the CRM Server and between the CRM Server and mobile clients and other connected systems

•

Queues destined for the R/3 back-end and other stady connected systems should be relatively short and quickly processed, queues to destination NONE too

•

Entries destined for the mobile clients and BW remain in the queue until each receiver fetches its data

•

When the queue entries for a client reach 10,000, the queue should be closely monitored (e.g. issue warning). When the entries reach 100,000, severe problems may occur and performance will be affected. Administrative actions must be taken in these cases, any deletion of queues causes data inconsistencies

•

If a queue that is “in use” between a mobile client and the CRM Server is deleted, it will cause data inconsistency between the CRM

© 2002 SAP AG

SMQ1

17


CRM Server Central Monitoring Activities / Functions

Monitoring Type


SMQS

•

Daily

•


SMQ2

•


SMQR

•

Daily

•


•

Several times a day depending on the b i

Server and the mobile client.

•

In severe cases when a client cannot be manually rebuilt, it can be brought back into a consistent state by rebuilding the client data from the CDB (AC_EXTRACT)

QOUT Scheduler

•

Ensure that all destinations are registered. Only destination NONE must be excluded. See also SAP Note 400330

qRFC Inbound Queue Monitor: •

Monitor data transfer between the R/3 back-end system and the CRM Server and between mobile clients and the CRM Server and all other queues, which have to be stored in the CRM Online DB

•

Messages in the inbound queue are processed according to the capacity of the CRM Server.

•

High number of entries in the inbound queue can indicate insufficient capacity on the CRM Server

QIN Scheduler Status •

This transaction runs the scheduler to check the inbound queues on the CRM Server

•

Ensure that all inbound queues are registered (type R). Register them, if necessary, using the transaction SMQR

•

If a queue is registered and has not been processed for a long time, the administrator has to check the reason for it. Possible a scheduler problems may exist.

•

Check the maximum processing time of the inbound queues.

•

Set the maximum processing time of a queue ("Max-time") to 60 seconds for all queues during normal operations

TRFC Monitoring •

Monitor all transactional RFCs

© 2002 SAP AG

SM58

18


CRM Server Central Monitoring Activities / Functions

Monitoring Type

Monitoring Frequency Periods and Events business process

(tRFCs) processed on the CRM Server (e.g. workflow) SM58 is linked to the Replication and Realignment queues in error status in the EXTRACT and/or AC_EXTRACT queue (click on the error symbol) Message Flow Statistics • Collects statistical data about the workload on the CRM Server caused by BDocs messages • Use this as a starting point for analyzing performance problems • The message flow is a central function in the CRM Server and uses BDocs • Display statistics of the message flow within the CRM Server • Ensure that the middleware message flow statistics are switched on

SMWMFLOW Only for power users

• •

Daily In case of an error message

BDoc Messages / Summary • Application error analysis • SMW01: Display the data of a BDoc message and possible errors • SMW02: Display BDocs message summary in dependency on the site ID

SMW01 / SMW02

•

Several times a day depending on the business process In case of an error message

Middleware Trace • Developer Trace

SMWT

Summary of Unprocessed BDoc Messages

SMW03

Check Flow Definitions Only after changes in the customizing • Consistency Check for Flow Definitions

SMO8FD

© 2002 SAP AG

•

•


•

After BDoc type changes or changes in services or in the flow are made In case of an error message

•

19


CRM Server R/3 Adapter / Load Monitoring Activities / Functions

Monitoring Type


Monitor Load Status

R3AM1

•

•

Checks, whether the initial load was successfully completed

•

Object status and actions: •

RED: Refer to SAP Notes 429423 and 350176 for initial load problem analysis

•

YELLOW: initial download must still be done / is running

•

GREEN: OK

Monitor Request • Used in exceptional cases to bring the database into a consistent state after a data lost for instance in the CDB

•

R3AR3

Basis Monitoring: General System Checks CRM Server General Monitoring Monitoring Type Activities / Functions R/3 System Workload Analysis: •

Monitor RFC response time statistics for CRM

•

Monitor the dialog response time for Online transactions (e.g. CIC0)

Local Work Process Overview: •

Monitor current state of individual work processes

•

Ensure that enough work process capacity is available at peak times

System-wide Work Process Overview: •

•

•

In case of an error message, if the databases are not consistent and a request from the R/3 back-end, the CRM or the CDB is necessary

ST03N

Monitoring Frequency Periods and Events • Daily • In case of an error message

SM50

• •

Hourly In case of an error message or performance problems

SM66

•

Several times a day depending on the business process In case of an error message or performance problems Upon error Daily

Similar to SM50, but for system-wide statistics

Internet Communication Manager (ICM) Monitor

In case of an error messages during/after initial load Depending on the object status

•

SMICM

• •

Monitor current state of ICM and services

© 2002 SAP AG

20


CRM Server General Monitoring Activities / Functions •

•

Monitor memory resource usage for specific R/3 application servers

•

Use “normal” SAP System monitoring practices

Operating System Monitor:

•

Monitor the active connections to other servers

•

Display gateway trace

•

Check the “SAP return code” and “CPI-C return code” values for errors

Update Errors:


DB02

• •

Daily In case of an error message

ST04

• •

ST22

• •

SM21

• •

SMGW

•

Daily In case of performance problems Daily In case of an error message Hourly In case of an error message In case of an error message

SM13

•

Check for entries that have an “ERR” status

Display Statistical Records: •

•

Check for general system errors

Gateway Monitor / Connection List:

•

ST06

Analyze aborted programs

System Log: •

Weekly

Monitor the growth of tables and indices

ABAP Dump Analysis: •

•

Monitor indices on tRFC and aRFC tables

Database Performance Analysis: •

ST02

Monitor hardware load during high dialog user load / RFC transmission times

Database Performance Monitor: •


ICM administration

Buffer Monitor:

•

Monitoring Type

• STAD

Determine the performance of statistical records

•

Several times a day depending on the business process In case of an error message In case of an error message

CRM Server Parameter Settings For data exchange with an R/3 back-end the parameters are defined in the CRMCONSUM table on the CRM Server side. Parameter Name

© 2002 SAP AG

Value

Description

21


Parameter Name

Value

Description

Consumer

CRM / CDB / EXT / ONL

Consumer of adapter functionality

Active

X

Flag if application is activated

Q_Prefix

CDB

Prefix for queue names in QRFC

R3A EXT ONL

The table SMOFPARSFA contains control parameters of the CRM Server. Other settings of the parameters depend on the business scenario you are using, see CRM Installation Guide. Current, experience-based parameter settings are checked during each SAP Service Session (EarlyWatch, GoingLive, etc.). The SAP recommendations should be applied according to the instructions given in the SAP Service report. The following parameters are important for CRM Field Sales and are provided here as an example: Parameter key

Parname

Description

CRMGENERAL

TRACE-LEVEL

ENV=G

CRMGENERAL

TRACE-LEVEL

ENV=*

CRMGENERAL

TRACE-LEVEL

ENV=I

RRS_COMMON

MAX_PACKAGE_SIZE

Note 453882 (only MSA/MSE)

RRS_COMMON

RRQUEUE_PARALLEL

Note 453882 (only MSA/MSE)

FLOW

USE_INQUEUE_ALWAYS

Note 529764 (for distributed systems)

Scheduling Batch Jobs Middleware Trace Reorganization For tracing, you can use various standard settings or your own settings; see above in section settings in table SMOFPARSFA. Depending on the selected trace level, the system writes entries into trace tables. Such entries must be deleted in regular intervals to prevent these tables from becoming too large. Some possibilities to handle this would be to keep trace information (especially errors) in the log for e.g. 1 day, or 1 week and to delete the data afterwards. You have to schedule reorganization jobs, called SMO6_REORG (SAP Note 206439) and MW_REORG (SAP Note 487915) Middleware Portal To be able to use the centralized status monitoring for the generation steps, you must call up the Middleware Portal (transaction SMWP) and activate the background job for status processing by clicking on Scheduled Background Job. Note that status monitoring will be available only during the next day.

© 2002 SAP AG

22

Best Practice: mySAP CRM Field Sales Monitoring BDoc Flow Statistics Activation Ensure that the middleware kernel application statistics are switched on. Call transaction SMWMFLOW. Execute Goto -> Activate Statistics. Select “message flow statistics”. Activate monitoring of the middleware message flow. To activate the application statistics, select “Kernel application statistics”, select the change mode, mark the field “MW” (Middleware Message Hub Statistics”. Save. Communication Monitor Collector Activation For activating transaction SMWMCOMM, the Conntrans Performance monitoring, you have to choose “Environment”-> “Run Collector”

2.2.3 Field Sales Specific Monitoring

qRFC Inbound

R/3 Adapter

qRFC Inbound

XIF Adapter

qRFC Inbound

Synchronization Flow

Mapping Service: Sync BDoc -> Msg BDoc

Inbound and Outbound Queue Monitoring

CRM Adapter

Messaging Flow Routing (Simple Replication)

Display BDoc Messages, Middleware Trace

XIF Adapter

qRFC Outbound

R/3 Adapter

qRFC Outbound

Mobile Bridge: Msg BDoc -> Sync BDoc

Replication & Realignment Queues

Synchronization Flow CDB Service Replication/ Realignment Outbound Adapter

qRFC Outbound

In this section, all CRM Middleware monitors specific for Mobile scenarios are described.

2.2.3.1 CRM Server Mobile Bridge settings: For all BDoc types that you are planning to use in MSA/ MSE scenario you have to enable the Mobile Bridge: You have to set field ACTIVE to ‘X’ in table SMW3FDCUST according SAP Note 430392.

© 2002 SAP AG

23


Monitoring Activities / Functions

Monitoring Type


Queue Information for CRM Client Sites

SMWMQUEUES

Only for error tracking in seldom cases

Replication Queues Monitor • Displays information about the status and contents of the replication and realignment queues in the clients defined in the CRM Server • The EXTRACT queue shows to which clients a specific BDoc type will be distributed • To check the status of the queues • Deselect the flag: Display current client only • Press the Refresh button • Search for queues with • Status = Hold • number of entries > 0 and a • flash icon button in the right-most column • Correct the error situation for the indicated queues (error handling in SM58) Note: Do not press the trash icon button if it appears next to an entry.

SMOHQUEUE

•


•

Daily

Communication monitor • Communication monitor: monitoring individual sessions, statistics of the data exchange for each site

SMWMCOMM Middleware à Monitoring à Mobile Client à Communication Monitor

•

Daily

•


Mobile Client Message Recovery Unprocessed message recovery: reporting messages informing the CRM Server about errors during the import on the mobile clients Operating System / Gateway The SAP system statistic collector daemons, SAPOSCOL and RFCOSCOL, run on the Communication Station and gather hardware resource consumption data. Complementary programs run on the CRM Server and collect and display statistical data.

CMWQ

•

Daily

•

Daily

•


•

Display all mobile sites defined in the CRM Server together with the name of the queue assigned to each of these sites

© 2002 SAP AG

Monitoring -> Mobile Client -> Message Recovery OS07

24



Monitoring Type


While the Communication Station is running, data is continually being collected via existing connections, and system data is also collected for subsequent evaluation. This data is called up periodically by the CRM Server and can be displayed and analyzed there using monitoring tools. The gateway, which is installed on the Communication Station, is used to call up the collected data. Windows performance monitor

2.2.3.2 Communication Station In the CRM Field Sales implementation, the monitoring possibilities for the Communication Station can be divided into three areas: •

DCOM Connector monitor

•

Communication Log File: TransferService.Log

•

Windows NT Performance Monitor / WIN 2000 Perfmonitor

Communication Station Monitoring Activities / Functions CRM Transfer Service The SAP CRM Queued Transfer Service component logs the communication sessions between the mobile clients and the CRM Server in the TransferService.log file.

Monitoring Type

TransferService.log


•

Daily

The default file path is as follows: C:\WINNT\TransferService.log QmtCnfg.exe, a Release 3.0 tool, which can be used to view the current trace level and log file location

QmtCnfg.exe

Tracing the data transfer on the Communication Station is not required, only for troubleshooting.

© 2002 SAP AG

25


Communication Station Monitoring Activities / Functions DCOM Connector: Use the following menu path to access the DCOM Connector component monitor: Start > Programs > Middleware > DCOM Connector. Start the application and choose Monitor. The following tables explain the different areas of the monitor.

Monitoring Type


•

On demand

The following diagram shows the monitoring areas in respect to the Communication Station: The Communication Station (CommStation) is the link between the CRM Server and the mobile clients.

© 2002 SAP AG

26


DCOM Connector Monitor TransferService.log

Mobile Client(s)

Comm. Station

CRM MW System Outbound Queue


Monitoring Communication Station

DCOM Connector

R/3 System Inbound Queue

Inbound Queue

Outbound Queue



DB & Log

Log



© 2002 SAP AG

27


2.2.3.3 Mobile Client Monitoring When a traveling sales person has data to up- or download from his laptop, he must establish a connection to the CRM Server via the Communication Station. This may be done at the customer site using a telephone line. The Communication Station maintains a connection to the CRM Server that in turn maintains a connection to the R/3 back-end system (LAN configuration). The CONTRANS.EXE program is executed on the mobile client to establish the connection. Data from the outbound queue of the mobile client is transferred through the Communication Station to the CRM Server. This data can be customer orders and/or customer data, such as newly created customers on the mobile client. For monitoring the mobile clients, you have to consider four different areas: •

The operating system

•

The local IDES database

•

Trace files

•

Data transfer log file

Mobile Client Monitoring Activities / Functions

Monitoring Type

Queued Transfer Service

QmtCnfg.exe

The QmtCnfg program displays the connection status between the mobile client and the Communication Station.

TransferService.log

Client Console, metadata check, generation, comparison of BDoc structures

Trace.log

•

Perfmon.exe


Data bound from the CRM outbound queue to a specific mobile client is copied to the inbound queue of that mobile client.

Monitoring Frequency Periods and Events • In case of an error message in the data transfer phase

In case of an error message in the import phase

Data from the mobile client outbound queue is transferred to the inbound queue on the CRM Server. The inbound and outbound queues of the mobile client can be displayed using the Client Console. Operating system Windows NT Performance Monitor Database You can monitor the SQL-Server database on the Mobile client with the remote SQL-Server Monitor

© 2002 SAP AG

SAP Note 358507 SAP Note 433401 SAP Note 530317

28


2.2.4 Interaction Center Specific Monitoring R/3 Adapter

qRFC Inbound / Outbound

XIF Adapter

qRFC Inbound / Outbound

Inbound and Outbound Queue Monitoring

E-Mail Server SAP Connect

CRM Server Application

SAPphone

Telephony Gateway In ST03, Create Transaction Detail Profile for CIC0

Monitor Interfaces to External Components

In this section, CRM Middleware monitors specific for Interaction Center scenarios are described.

2.2.4.1 CRM Server Monitor interfaces to the external components If you are using telephony / E-mail functionality in your Interaction Center scenario, you have to monitor the corresponding SAPphone and SAP connect interfaces. Monitoring Activities / Functions

Monitoring Type


Monitor outgoing e-mails / faxes

SCOT

•

•

Nodes: List of all nodes defined for your System and their status.

View > Nodes or

•

•

Jobs: List of all jobs defined for sending mails of faxes from your system.

•

Routing: List of all routes defined for your system.

•

Send orders: overview of the send orders (e-mails, faxes) in a given time

© 2002 SAP AG

Upon error Daily

View > System status View > Routing Utilities > overview of send orders

29



Monitoring Type


Monitor SAPphone Interface

SPHB

•

Upon error

•

Trace Files: switch the trace on or off and display the trace files.

Utilities > Trace > internal Trace

•

Alert Monitor: starting the collection method and view the alert monitor data (Option: Integration of the Alert Monitor into CCMS)

Utilities > Alert Monitor > Start collection method or Display

•

SAPphone version: get the SAPphone version

Utilities > SAPphone version •


period and/or send status.

Transaction Detail Profile for CIC0 • monitor transaction CIC0 by using the transaction detail profile

© 2002 SAP AG

ST03N

30


2.3 Verification After adapting the recommendations described in this Best Practice, you can monitor your mySAP CRM solution using the transactions described in this section. Many CRM-specific transactions are not available in the R/3 back-end system. Those common to both the R/3 back-end system and the CRM Server are described together. Those handled or interpreted differently are explained separately.

© 2002 SAP AG

31


2.3.1 Middleware Portal – SMWP

© 2002 SAP AG

32


2.3.2 Monitoring CRM Outbound Queues – SMQ1 Functional Description The CRM outbound queues contain all objects, which should be sent to the R/3 back-end, the mobile clients and other connected systems. •

Each queue contains single tRFCs in a specific sequence, which is called qRFC.

•

Each client has a dedicated queue number.

•

It is normal that some entries exist in the queues for the individual mobile clients. These have to connect asynchronously to the CRM-system and “pick up” their messages from the dedicated queue, which waits in SMQ1 (only MSY/MSE).

The R/3 back-end system, the APO and the BW system should always be connected (except during the downtime for backup) The following table shows the different prefixes used for the different queues: Queue Name

Systems

Description

R3AI

R/3 → CRM

Initial. BAPIs destined for the R/3 back-end

R3AD

R/3 → CRM

Delta

R3AR_

R/3 → CRM

Request

R3AU

R/3 → CRM

Upload

CRM_SITE

CRM-> Mobile Client

Synchronization BDoc types from the CRM Server going to the mobile client

CSA_

CRM-> CRM (simple replication)

Outbound for simple replication of BDocs to receivers

BW..

CRM-> BW

Queues to BW systems

EXT..

CRM -> extern

Queues to external systems

CSA_MASS_

When mass updates are carried out (e.g. application data updates) in the R/3 back-end system, the data is shipped via the outbound queue to the CRM Server. This data is controlled via qRFC settings. Choose Information → Version in transaction SMQ1 to determine the current qRFC version. For a detailed description of qRFC, refer to SAP Notes 193515 and 438015. Usage Menu path: Middleware → Monitoring → Queues → Display Outbound RFC Queues Transaction: SMQ1 Handling Procedure •

Deleting a queue will lead to data inconsistencies between the sender and the receiver. E.g. deleting a queue that is “in use” between a mobile client and the CRM Server will lead to data inconsistencies between the CRM Server and the mobile client.

© 2002 SAP AG

33

Best Practice: mySAP CRM Field Sales Monitoring In this case, you must bring the client back to a consistent state. In some cases, you can do this by extracting the data from the CRM Server and sending it to the client. This restores the client state, as it is known on the server. Data that was not yet successfully transferred from the client to the CRM Server must be manually reconstructed. Local data that is not usually sent to the server may be also lost and you may need to reconstruct it manually. •

If you accidentally delete data from a queue that is not assigned to a mobile client, data may be lost and cannot be recovered (e.g. in APO, BW, R/3 System).

•

In case of error, first try to solve the problem. If all other attempts to fix the problem fail, you may need to delete an entry in a queue. In this case, you must bring the different databases (R/3, CRM Server, CRM Mobile Client) manually into a consistent state. To do so, you must be familiar with the technical and business logic of the process involved.

Problems and errors can occur during data exchange (such as program crash, lost connection between R/3 back-end and the CRM Server, and so on). Queues for the R/3 back-end should not wait too long. Possible causes for waiting can be e.g.: •

Error in the first BAPI in queue,

•

Resource bottlenecks in the R/3 System,

•

Logical sequence.

Check the status. If it is SYSFAIL or CPICERR, an error has occurred for the corresponding queue (most likely on the CRM Server). Double-click on the queue name and look at the status text of the first entry in the queue: you will find a short error message describing the error. After correcting the error, you can restart the queue. SAP Note 443900 contains useful information on how to analyze the error situations during the data exchange.

Monitoring CRM Outbound Queues - SMQ1 – Queue Details Functional Description In the Details view of the outbound queue entries you can see the oldest queue entry. This gives you information about when the queue was running last time. For MSA/MSE this gives you an estimation, how long a Mobile Client was not connected. It is possible that clients are not used anymore. The end user for such clients should be contacted and the site deleted or the client connected for running Conntrans. Note: If the queue is deleted for the mobile client without deleting the mobile client, an inconsistency can occur. If data is sent to this mobile client later, it does not have the data that was deleted from the queue. The outbound queues information is stored in the following tables: Table Name TRFCQOUT ARFCSSTATE

Description Queue Information (Header of the Queue)

ARFCSDATA

Data

Status table of the LUWs in the tRFC/qRFC target system (Header of the record)

To display the function modules of the queue, double-click on the line. The following table describes the different information shown for a specific queue: Field Name

© 2002 SAP AG

Description

34


Field Name

Description

Client

Client

Queue name

Queue name

Destination

Logical destination (specified in function call)

Entries

Number of logical units of work (LUWs)

Status

Normal status READY: The first entry of the queue is ready for sending RUNNING: The first entry of the queue is just processing EXECUTED: The first entry of the queue is already processed, just waiting for confirmation of the receiver system NOSEND: no active sending, receiver has to fetch the records, deleting the record only after confirmation of the receiver

Error status SYSLOAD: No dialog work process free, the record will be automatically reprocessed by a background job SYSFAIL: error on receiver, look for a short dump there CPICERR: network or communication error, record will be automatically reprocessed by a background job Wait status STOP: Application locks the first entry of the queue temporarily WAITSTOP: Locked because of dependencies to other queues, because the other queue is just locked WAITING: locked because of dependency to another queues, there is the linked record not the first entry in the queue WAITUPDA: First record in the queue contains update function, for which the queue is just waiting NOSENDS: Queue is just waiting for debugging MODIFY: The data of the first LUW are just been modified Detailed information about the status can be found in note 378903. 1. Date

Oldest date of a queue entry

1. Time

Time of oldest queue entry

NxtDate

Most recent date of queue entry

NxtTim

Time of most recent date of queue entry

Wait for queue

Only used for linked queues, not in CRM

Handling Procedure

© 2002 SAP AG

35

Best Practice: mySAP CRM Field Sales Monitoring If you recognize many entries in one or more queues, take care of the following points: •

Only MSA/MSE: The corresponding client of the queue has not been connected to the CRM Server for a long time. You can check this in the column 1. Date. You should avoid situations when the mobile client has not synchronized his data for a long time. Many queues with a large amount of data can lead to a rapid growth of communication tables (e.g. ARFCSDATA) and have an impact on the system performance. The client should regularly connect to the CRM Server, in exceptional cases at least every two weeks.

•

Only MSA/MSE: To reduce the amount of data the subscriptions and publications for a client should be as few as possible necessary from a business point of view.

•

For all other queues not assigned to a client errors should be monitored hourly at least several times per day. Errors should be resolved and the queue restarted afterwards.

Monitoring CRM Outbound Queues - SMQ1 – Transaction Details Functional Description Only the first and last entries are displayed. To display the full queue you can double-click on the dots in the mid of the screen. To get detailed data about qRFC, double-click on a line and by selecting a client or user or queue name column. By selecting the destination you are linked to transaction SM59. Double-click on the function module column shows you the coding of the function.

2.3.3 QOUT Scheduler – SMQS Functional Description This transaction monitors the scheduler to process the outbound queues per destination (see SMQ1). Usage Here you can register, deregister or exclude the destinations from scheduling. You can stop all outbound queues in transaction SMQS. Menu path: > Edit>Registration/ Deregistration Transaction: SMQS Handling Procedure For registration of destination NONE refer to SAP Note 496826 dependent on your QRFC version. Setup for Optimizing the Performance. You can configure the “Max_time” parameter per queue by deregistering the queue and registering them again. Please do not set it too small (edit>change AS-group. Refer to SAP Note 400330 for getting more information about the QOUT Scheduler.

© 2002 SAP AG

36


2.3.4 Monitoring CRM Inbound Queues – SMQ2 Functional Description The transmission of delta information is performed from the R/3 back-end system or other systems to the CRM Server. When the delta information arrives at the CRM Server, the data is forwarded to the inbound queue. •

The delta data can be found for instance in the R3AD* queues.

•

The inbound queues are normally empty or small, if the CRM Server has enough resources and no errors occur.

•

Mark an entry and press Change View (F8) to detect stopped or hanging queues.

•

Double-click on the queue to display the LUWs belonging to the queue.

It is strongly recommended to avoid deleting the queue or queue entries, because this can lead to data inconsistencies! The qRFC Inbound Queue Monitor, SMQ2, is not a critical monitor on the R/3 back-end system because there is normally no data to be displayed. Instead, monitor the outbound queue (SMQ1 or R3AM1) on the CRM Server. When the CRM Server tries to send data to the R/3 back-end and the system is not “up”, the data stays in the outbound queue of the CRM Server in status CPICERR. As soon as the data is in the R/3 back-end in-queue, it gets processed through to the respective application or to its appropriate end location. Data is not deposited in the R/3 back-end’s in-queue and picked up later. It is processed immediately. The inbound queue information is stored in the following tables: Table Name TRFCQIN TRFCQSTATE TRFCQDATA ARFCRSTATE

Description Status inbound Queue - Header of queue QRFC call condition Data of records Status table of the LUWs in the tRFC/qRFC target system (Header of the records)

Usage Menu path: Middleware > Monitoring > Queues > Display Inbound RFC Queues Transaction: SMQ2 1) Select the entry you want to view. 2) To view the details, press the Change View button (F8) and check for stopped or hanging queues.

© 2002 SAP AG

37


Field Names

Descriptions

Client

Client

Queue name

Queue name

Entries

Number of logical units of work (LUWs) Normal status READY: The first entry of the queue is ready for sending RUNNING: The first entry of the queue is just processing Error status SYSFAIL: error on receiver, look for a short dump there CPICERR: network or communication error, record will be automatically reprocessed by a background job ARETRY: Temporary error on receiver, record will be automatically reprocessed by a background job

Status Options

ANORETRY: Fatal error, processing by QRFC manager stopped, check together with application Wait status STOP: Application locks the first entry of the queue temporarily WAITSTOP: Locked because of dependencies to other queues, because the other queue is just locked WAITING: Locked because of dependency to another queues, there is the linked record not the first entry in the queue NOEXEC: Queue is waiting for debugging MODIFY: The data of the first LUW are just been modified Read SAP Note 378903 for further information related to the status.

1. Date

Oldest date of a queue entry

1. Time

Time of oldest queue entry

n. Date

Most recent date of queue entry

n. Time

Time of most recent date of queue entry

Source ID

Destination of the sender

Waiting for queue

Only used for linked queues, not in CRM

Handling Procedure An error has occurred, if during an object load with status Running (light yellow) the date, time and the number of blocks remain constant over the time or grows, if new entries are written to the end of the queue. No queue entries are processed here. General procedure: In the case of stopped queues, first search for the error causes and then fix them. Here are some possible error situations: •

If the status of the queue is SYSFAIL or CPICERR, an error is occurred.

© 2002 SAP AG

38


•

-

By double clicking on the queue name and looking at the status text of the first entry in the queue, you will find a short error message describing the error reason. After the correction of the error, you can restart the queue.

-

Check the short dumps (transaction ST22) on the receiving system, here the CRM system. An error in the queue data processing generates a short dump. Here you can find information about the cause and location of the error.

-

Check further information about errors in the system log (transaction SM21) on the CRM Server. Have a look at the CRM Server Log.

The input queue on the CRM Server has status Ready, but the number of entries does not decrease:

Transactions cannot be performed because the system is overloaded. Check if enough system resources are available and check queue registration in SMQR. If the queue cannot be activated again, check the application log for possible application error messages. •

•

Data arrive from the R/3 back-end system but is still hanging in the inbound queue of the CRM Server: •

Check that the scheduler is set to active on the CRM Server (transaction SMQR)

•

Follow-up problems can be found in the tRFC (transaction SM59) or in case of short dumps in transaction ST22.

If no delta data arrives from the R/3 back-end (no data in the inbound queue of the CRM Server), or the delta queue has the status STOP you can proceed as follows: •

At least one object has load status Running or Abort and has not finished. Wait until the object is finished or you have to force termination.

2.3.5 Scheduler Status – SMQR Functional Description This transaction runs the scheduler to activate the inbound queues if tRFC resources are free on the CRM Server. It also controls the runtime of a queue. Using transaction SMQR you can register or deregister the inbound queues “to be scheduled” by the scheduler. If you have more than one client in your system, per client one scheduler is running. You can stop all inbound queues in the CRM Server by de-registering all queues in transaction SMQR. Only registered queues are processed. These are registrations with any generic queue name with type R (R = registered, U = unregistered). So an entry must exist in transaction SMQR for the queue names, to get the queues processed by the scheduler in SMQR. In the upper part of the display, you find the scheduler status, which is either INACTIV, ACTIVE or WAITING, STARTING (or SYSFAIL or CPICFAIL in error case). If the scheduler status is INACTIV: This status means that the qRFC scheduler has no work list at the moment. Therefore no registered Inbound queues in the READY status. The scheduler is ready for queues are to be periodically checked for scheduling. The period can be set by the parameter “Max_time” per queue. The scheduler runs also if a new queue comes in (see SAP Note 369007). Usage Menu path: Monitoring -> Queues -> Register/Deregister Queues

© 2002 SAP AG

39

Best Practice: mySAP CRM Field Sales Monitoring Transaction: SMQR Handling Procedure You can configure the “Max_time” parameter per queue by deregistering the queue and registering them again. Please do not set it too small (edit>change AS-group.

2.3.6 TRFC Monitoring – SM58 Functional Description Monitor all transactional RFCs (tRFC) processed on the CRM Server (such as workflow or replication/realignment for MSA/MSE). Execution generates a selection screen where you can choose the display period, the user name, function, destination and status of transactional RFCs. Almost any combination (single or several values, ranges, and exclusions) of these parameters is possible. If a system or application exception is raised during the processing of a tRFC LUW the target system will return this error to the sending system. The status of this LUW will be updated with the exception error message (red background color of status message). You get information about the caller, function module, target system, date, time and status text. Additionally you get information about the transaction ID, Host, current transaction code, program, client and number of attempts establishing the connection. Usage Menu path: Middleware à Monitoring à Transactional RFC à Display Transactional RFC Transaction SM58 Handling Procedure When error messages appear in the SM58 transaction, the error condition(s) must be resolved before re-execution of the tRFC procedure. When the problem is solved, the tRFC should be manually re-executed / re-scheduled. If the error is fixed, the entry does no longer appear in the SM58 display. Do not delete an entry before resolving the problem because this will lead to data inconsistency.

© 2002 SAP AG

40


2.3.7 Monitoring the Middleware Message Flow Statistics – SMWMFLOW Functional Description With this function you can display statistics on the BDoc message flow within the CRM Server. The statistics provided by this function are divided into two main areas: •

BDoc type/ service kernel application statistics: Application statistics from the R/3 kernel for BDoc types and services.

•

BDoc type/ site / queue statistics: Statistics for BDoc types, sites and queues.

Usage Menu Path: Middleware → Monitoring → Message Flow Transaction: SMWMFLOW Activation To activate the statistics, choose the transaction SMWMFLOW and execute Goto → Activate statistics. •

If you want to check whether the application statistics from the R/3 kernel have been activated, choose Kernel application statistics. The checkbox for Middleware Message Hub Statistic must be active. So that the statistics are actually activated, you must have first scheduled the background job SAP_COLLECTOR_FOR_PERFMONITOR.

•

Choose Middleware Message Flow Statistics. You can toggle the creation of statistics on and off with On/Off. When On is selected, the background job RSMWM_BSTAT_COLLECTOR is automatically scheduled (status: Collector: On).

Workload from database Choose WorkloadàWorkload from database. Select one or all (TOTAL) R/3 instance(s) and specify the required time period. This statistics records the progress of BDoc messages in the CRM Server (including the retention period in inbound queues). The retention period in the outbound queues is not recorded. The single records are directly written to the message flow before the completion of processing. In contrast to the application statistics, these statistics also contain data from the message header of the BDoc messages concerned. The report RSMWM_CHECK_STATISTIC_RECORDS displays single records of the application statistic. It displays related statistical data records together (i.e. a single BDoc type with its services). You can also view the entire statistical data record here by double clicking. Just as in case of transaction ST03N, the single records created in this way are available at least for the current day or the last 2-3 days, depending on settings in ST03N.and can also be displayed. These records are automatically aggregated by the CCMS data collector to form the following statistics (the sum of the total and detailed response times, as well as the average total and detailed response times): •

By BDoc type (including the times for the services)Per service (for all BDoc types)Per BDoc type and service. Just as in the case of ST03N these statistics are available for days, weeks and months. Column Records displays the numbers of flow runs of the different BDoc types.

Alternatively, in SMWMFLOW, choose Workload from database àPer service Here you can get the workload for individual services per BDoc type. You can also display the workload per BDoc type (Detail).

© 2002 SAP AG

41

Best Practice: mySAP CRM Field Sales Monitoring This overview does not include the processing time for the following: •

Inbound adapter

•

R/3 inbound adapter

•

Flow itself

Therefore, the sum of the total response times of all services can be significantly smaller than the total response time of the BDoc type you have chosen before. Last minutes workload Choose Workloadà last minutes workload (Set time interval: Analyze last 15 min for this Instance) Here you can get statistics about the current workload of the message flow in your system (for BDoc types and for CRM services). To display statistics for the Middleware services, you can also choose Goto à Service statistics. In the service statistics you can see Detail statistics for individual services as well as the workload for individual services per BDoc type. In all detail statistics, to display individual statistics for the selected BDoc type or service, choose Single steps Alternatively, in SMWMFLOW, choose Last minutes workload à Per service Message-flow statistics Press the Message flow statistics button. The Data Collector starts automatically every hour to provide the required data. If you need the latest data for your analysis, you can manually start the data collector at any time by choosing Run Collector. The statistics relate to one week and to different instances. The header data shows the display and your selection criteria. The results show the single records that the statistics refer to. For each record the BDoc type, client, site, queue and name of the CRM Server are displayed, as well as the retention time of the corresponding BDoc type on the CRM Server.

2.3.8 Check Flow Definitions – SMO8FD Functional Description Consistency Check for Flow Definitions Usage Menu path: Middleware → Message Flow → Display and Check Flow Definitions Transaction: SMO8FD Handling procedure You developed one or several new BDoc types. You generated the flow definition for these BDoc types. •

Flow definition errors may occur using newly developed BDoc types.

© 2002 SAP AG

42

Best Practice: mySAP CRM Field Sales Monitoring •

Search for SAP Notes that solve the problem. If there is no solution, create a customer message.

Note: Use SMO8FD after a new BDoc definition has been transported.

2.3.9 Middleware Trace Functional Description The table SMWT-TRC stores all BDOC trace information and is linked to transaction SMW01. Usage This information is only necessary for error handling in transaction SMW01. Handling Procedure Trace level is important: during the implementation phase should be high. In the production system should be reduced, otherwise you may get performance problems. Refer to note 206439, see setting in table SMOFPARSFA. The trace should be used withSMW01/SMW02 to check the content and find the error and fix it.

2.3.10 Monitor Request – R3AR3 Functional Description Data requests can be defined and triggered manually from an R/3 back-end or from the CRM database using the transactions R3AR2 and R3AR4. The request monitor then monitors these requests. If there are data in the CRM Server or CDB or R/3 back-end missing or incomplete, in exceptional cases, you can use the request to bring your database into a consistent state after a data loss in one the databases. Usage Menu path: Middleware à Monitoring àData Exchange-> Monitor Requests Transaction: R3AR3

2.3.11 Monitor Initial Load – R3AM1 Functional Description Checks whether the initial load was successfully completed. Usage Menu path: Middleware à Monitoring à Data Exchange-> Monitor Objects Transaction: R3AM1 Call transaction R3AM1 (Monitor Objects) on the CRM Server and search for objects with a status other than green Handling Procedure •

RED: If the status monitor displays objects with a status other than green, then problems occurred during the initial load

•

YELLOW: If the display is empty - initial load hat not been performed yet

© 2002 SAP AG

43


GREEN: Status OK

If the status symbol for an object is RED, refer to SAP Note 429423 for initial load problem analysis.

2.3.12 Monitoring the Replication & Realignment Queues – SMOHQUEUE Functional Description This transaction must only be monitored for MSA/MSE. Displays information about the status and contents of the replication and realignment queues in the Mobile Sites defined on your CRM Server. •

To stop or start replication queues manually, click on the traffic light: The light changes between red and yellow or green.

•

Normally all queues should be running or waiting: Particular processes are automatically triggered to process queue entries when such entries are entered into the queue.

•

The Number of entries displays the number of entries that are currently in the queue. This number should be continuously decreasing, unless new entries are entered into the queue at the same time. Double-click on the field number to view the entries in the respective queue.

•

If you interrupt queue processing, the processing of the current entry is completed and then the queue is set to status Hold.

Usage Menu path: Middleware à Monitoring à Queues à Monitor R&R Queues Transaction SMOHQUEUE Field Names

Descriptions

Name

Name of queue:

•

AC_EXTRACT

Represents those extract jobs triggered directly from the Administration Console, for example for a data reload to a mobile client to recover the local SQL database

•

EXTRACTBLK

•

EXTRACT

Contains extract jobs broken down by BDoc types (the jobs are created by realignment)

•

REALIGN

Contains realignment jobs for individual BDoc types whose responsibilities must be checked

•

SUBCHECK

Contains extract jobs broken down by BDoc types of the type bulk.

Contains subscription checker jobs (these are generated in the Administration Console, if a subscription is changed)

Number of entries

Current number of entries (messages) in the queue

Active tasks

Current number of active tasks

By setting the status icon in the status column you can:

© 2002 SAP AG

44


Release queues for processing by setting their status to Released (yellow light)

•

Reset released queues to Hold (red light)

•

Interrupt queue processing (Status Running – green light) by setting the status to Hold.

Handling procedure Normal condition: all lights green or yellow, number of queue entries is decreasing, if no new queue entries are created. •

If an error occurs during processing the Queue demon automatically sets the queue to hold. An error symbol appears in the last column. You can select it to display the error message. When you release the queue again, the error status is reset.

•

If you release a queue after the Queue demon has started, the status of the queue changes to Running, provided it contains entries for processing. If all entries have been processed (number of entries is zero), the status changes to Released again.

If the number of entries in a queue remains constant over a period of time, it may be necessary to manually trigger the processing jobs by clicking the green flag corresponding to the queue. If a queue contains entries, you can display details of the individual entries (such as the number of sites affected) by clicking on the number of entries. You can delete individual entries by selecting the checkbox in the first column of the record and choosing Delete. To update the information choose Refresh, but: Try to avoid deleting entries; this leads to inconsistencies of the mobile client local database. Refer to note 429627 for implementing the authorization checks necessary to avoid this kind of situations.

2.3.13 Queue Information for CRM Mobile Sites – SMWMQUEUES Functional Description Display all mobile sites defined in the CRM System together with the name of the queue assigned to each site. This transaction collects some queue information to Mobile Sites. Call this transaction: •

If you want to know which queue is assigned to a specific mobile client site

•

If you want to check whether this queue has been registered for the qRFC scheduler

•

If you want to check when data was last entered into this queue.

Field Names

Descriptions

Client

Displays the respective CRM Server client

Site Name

Site name assigned to the mobile client

Queue Name

Queue name assigned to the mobile client

Status

Displays the queue status In the case of inbound queues, a yellow symbol indicates that the queues have either been stopped or are waiting for another queue to be processed, while a red symbol indicates an error.

Entries

© 2002 SAP AG

Indicates the number of entries currently in this queue.

45


Registered

This column is displayed only in the case of inbound queues. It indicates, if a queue has been registered (R), has never been registered (empty) or has been deregistered again (U). A red symbol indicates that there are entries in the respective queue, that the queue is however not registered or that there are problems with the Queue Scheduler (for example status SYSFAIL or CPICFAIL). A yellow symbol refers to the same problems, but indicates that there are no entries in the queue. A green symbol indicates that there are no problems.

First Date / First Time

Informs you when the oldest entry was placed in the queue.

Last Date / Last Time

Informs you of when an entry was last placed in this queue.

Site description

Describes the respective site and was defined when this site was created in the Administration Console.

Usage Menu path: Middlewareà Monitoring à Queues à Display Mobile Site Queue Information Transaction SMWMQUEUES Handling Procedure Shows the last time of connection. Here you can see whether a client did connect for a longer time (also shown in transaction SMQ1), Assignment of site name and queue name (also shown in transaction SMOEACSMOEAC).

2.3.14 Communication Monitor – SMWMCOMM Functional Description This transaction is only for MSA/MSE-monitoring. You can use this for checking the performance of the Conntrans sessions of the Mobile Sites. The transfer rates, MW server’s answer time and the time consumed by the database read and writes during a session can be monitored using the transaction SMWMCOMM. The session monitoring functions will be used for the logging the time intervals. The module SMWM_START_SESSION will be called soon after the client attempts an authentication with the CRM server. Then after the transfer completes all the relevant session data and the session completion will be sent to the CRM server using the DCOM Connector interface (The Gateway installed on the Communication Station is not needed to execute this operation). A session ID, which is a 32-byte GUID generated at the time of instantiation of the server component will be passed in all the above session monitoring function module calls as one of the parameters. These data is reorganized within the middleware reorganization job. Usage Menu path: Middleware à Monitoring à Mobile Client à Commstation Monitor Handling Procedure

© 2002 SAP AG

46

Best Practice: mySAP CRM Field Sales Monitoring You can monitor the Conntrans performance per site, queue or Commstation. Otherwise you can check the performance per time range, for instance, day or week. The statistic you get shows upload and download times and transferred mount of data. It is also possible to get information about failed session between Commstation and CRM server.

2.3.15 Message Recovery – CMWQ Functional Description

The Mobile Client Message Recovery displays the status of BDoc messages sent from the CRM Server to the Mobile Clients (inbound processing) and allows their reprocessing: Usage Menu path: Middleware à Monitoring à Mobile Client >Message Recovery Handling Procedure Processed messages have successfully been processed from the inbound queue into the database of the Mobile Client. Unprocessed messages could not be saved on the client database. These BDoc messages are deleted by the system, a reporting message is sent to the CRM Server. The corresponding BDoc messages can be sent again to the Mobile Client by calling the transaction Message Recovery. The complete information on the BDoc message status on the Mobile Client can be accessed via the Client Console.

2.3.16 DCOM Connector Monitor Usage To access the DCOM Connector component monitor choose Startà ProgramsàMiddleware à DCOM Connector. Start the application and choose Monitor. The following tables explain the different areas of the monitor. View: Active Processes Shows the active processes of the DCOM Connector Column Headings

Descriptions

Process ID State

Either active or terminated

Up since

Time since process started

Page faults

Number of page faults since startup

Working Set

Size of the process’ actual working set in KB (kilobytes)

Peak Working Set

Maximum working set of the process since startup

Virtual Memory

Set actual size of the page file usage in KB

Peak Virtual Memory

Maximum usage of the page file since startup in KB

View: Open Connections

© 2002 SAP AG

47

Best Practice: mySAP CRM Field Sales Monitoring Shows the open connections of the DCOM Connector Column Headings

Descriptions

Destination State

Initial: created but not yet allocated Connecting: connecting to server Active: allocated by an object Pool: resource was returned to pool without cleanup Cleaning: R/3 context is being released Clean: resource was returned to pool in a clean state Closed: resource is closed and not available

Sys

Name of application server

Host

Hostname of application server

ID

System number (port number is 33## and 32##)

SAP User

User ID in R/3

Rem. Vers.

Version on application server

Rem. Codep.

Code page on application server.

Proc.

Process ID

hRfc

Handle to RFC connection

Loc. Vers.

Version of DCOM Connector

Loc. Codep.

Code page of DCOM Connector

Conv. ID T

Conversation ID like in SAP gateway monitor Trace

Calls

Number of calls since startup

Last Function

Name of last function being called

2.3.17 Communication Station Log File: TransferService.Log Functional description The SAP CRM Queued Transfer Service component logs the communications between the mobile clients and the CRM Server in the TransferService.log file. Usage The default file path is as follows: \TransferService.log The TransferService.log file supplies the following data for each session. This data is stored and can be displayed in text format on the local Communication Station server. Session header data •

Name and/or IP address of the DCOM Server

•

Date and time when the session was started (in UTC time)

•

Date and time when the session was closed

•

PID of the DCOM process

•

TID (thread ID) of the current session (PID and TID together are unique IDs for session)

© 2002 SAP AG

48


Object ID (session ID)

•

Site name

•

Queue name

•

Name of the PC from which the session was opened

•

Name of the Windows user

•

Session status

Session performance data •

Number of transactions sent to the R/3 System

•

Number of table entries sent to R/3

•

Number of transactions received from R/3

•

Number of table entries received from R/3

•

Compression rate for data received from client

•

Compression rate for data sent to client

•

Processing time on the DCOM Server for data received from R/3 and sent to the client

•

Processing time on the DCOM Server for data received from the client and sent to R/3

•

Session status

•

Detailed information for the functions Read, ReadConfirm, Send and SendConfirm:

•

Number of calls to R/3

•

R/3 response time

•

Network time between Mobile Client and DCOM Server

•

Response time of the client database

•

Transmission rate for communication of DCOM Server with R/3

•

Transmission rate for RFC communication between client and DCOM server

•

Access rate in the client database (bytes per second)

2.3.18 Windows Performance Monitor Functional description The Microsoft Windows Performance Monitor (PERFMON) is a performance-monitoring tool on Windows NT / 2000 systems. PERFMON can be used on the Communication Station to analyze performance problems due to: •

Total response time

•

Network time

•

Browser generation time

•

CPU and Memory Usage

When analyzing network conditions, PERFMON should be run locally on the Communication Station rather than from a remote server. When running PERFMON from a remote server,

© 2002 SAP AG

49

Best Practice: mySAP CRM Field Sales Monitoring network statistics can be affected due to the network load caused by PERFMON itself. Other network-based applications, such as PCAnywhere, also create a significant network load that may invalidate any network statistics measured while connected. Usage •

Verify that the Performance Monitor is installed.

•

Set up the counters and the log file (modify/set the log file and chart settings)

•

Ensure that no other services or programs are running that may impact the measurement (such as programs causing network or CPU load).

•

Perform the measurement

•

Extract the relevant counters (export them to a file)

•

Calculate the relevant quantities

•

Interpret the results

For further details, go to www.microsoft.com à Support Knowledgebase and see the White Paper Measuring performance-relevant data using PERFMON on Windows NT. In Windows NT/2000 start PERFMON by choosing Start à Run and enter PERFMON. Handling Procedures Set up PERFMON as follows. 1. Measure the CPU utilization on the server or PC. From the PERFMON menu, choose Edit à Add to Chart / Add Item Select in PERFMON: Object, Processor, Counter, % Processor Time 2. Determine the size of dataset transferred for analysis. Select in PERFMON: Object, Network Interface, Counter, Bytes Received/sec Note: Depending on the network connection, the name of the category and item of the abovementioned examples may differ. When selecting the network interface counter it is possible that various instances (= lines) are offered. If you are not sure on which network line the traffic takes place, choose all instances. You may have to start processes at the NT level monitoring network traffic. 3. To start your analysis, monitor during the peak time when Mobile Clients are uploading to the CRM Server. To save the performance data and transfer it into a spreadsheet program, choose File à Export Chart. 4. Evaluate your analysis.

2.3.19 SAP Connect Monitor – SCOT Functional description

SAP connect provides a standard interface for external communication, which supports sending using telecommunication services, such as faxing, pagers (SMS), Internet and X.400, as well as sending to printers and between several SAP Systems. It enables external communication components to be connected to the SAP System.

© 2002 SAP AG

50

Best Practice: mySAP CRM Field Sales Monitoring As of Release 6.10, SAP connect also provides a direct connection to the Internet using the SMTP plug-in of the Web application server. This enables you to send messages directly to Internet, fax, and paging addresses without using an additional external communication system. Transaction SCOT can be used for SAP connect administration. Handling Procedures

SAP connect administration provides various views of your communications environment. Each view shows the environment from a certain viewpoint: •

Nodes / System Status: You see all nodes defined for your System and their status when you choose View > System status or View > Node. Choose the DISPLAY Button to view the node data.

•

Jobs: You see all Jobs defined for sending mails of faxes from your System. The list shows for each job the time for the next run and the last runs that transmitted mails or faxes. When you mark one entry and chose the DISPLAY button you jump into the job details where you can find the job logs.

•

Routing: The view of routing (View > Routing) shows, for each communication method, which address areas are processed by which node. The display also tells you if a communication method is not available in your system or is processed using another communication type.

•

Alert Monitor: Choose Utilities > Alert monitor > Start data collection method to start the data collection for the monitor and then Utilities > Alert monitor > Display to see the results of this collection. (Option: Integration of the Alert Monitor into CCMS)

•

Send orders: to get an overview of the send orders choose Utilities > overview of send orders. You can specify a time period, the send status or the sender.

Field Names

Descriptions

Status

Status of send process (i.e. transmitted, waiting, send, errors,…)

Trans. Method

Transmission Method (i.e. via Telefax, via Remote SAP, via Internet,…)

Doc. Title

Document Title

Sender

Sender

Recipient

Recipient

Send date

Send date

Send time

Send time

Msg.

Message: Link to further details like status text, long text and transmission history

(Double-click on the entries to see more details.) For error analysis, you can use the trace functionality by choosing Utilities > Trace > Internal Trace. Here you can set the trace for inbound or outbound direction and display results.

2.3.20 SAP Phone Administration – SPHB Functional description

SAPphone integrates telephony functions in SAP applications. This allows data to be exchanged between computer and telephone processes.

© 2002 SAP AG

51

Best Practice: mySAP CRM Field Sales Monitoring Third-party manufactures can program a direct connection to the RFC interface or use the SAPphone server delivered with the SAP system. The SAPphone server serves as a nexus between the SAPphone RFC interface and the TAPI standard interface that Microsoft has defined for telephone integration solutions. If the SAPphone server is used, manufacturers of TAPIcompatible telephony products do not have to program an individual connection to the SAPphone RFC interface. The functions for starting and monitoring the telephony environment are available in transaction SPHB. The existing work centers are displayed there for each telephony server in the navigation tree. You can also display other information about each telephony server. In the standard view, the telephony servers are displayed with name and description, the work centers with extension and user name of the work center user. Handling Procedures

To enable communication the SAP System and the SAPphone server (or the self-programmed gateway software) you need a configured RFC connection between them. The following settings are required: •

For outbound calls, an RFC destination should be maintained in transaction SM59

•

For inbound calls, an RFC destination should be maintained in the Saprfc.ini or the gateway software

You can see the name of RFC connection for the corresponding telephony server in transaction SPHB in tab strip ‘Server attributes’. By double-clicking on the destination name you can jump into the destination settings in SM59 and perform e.g. connection test or maintain required settings. •

Trace Files: Choose Utilities > Trace > internal Trace. Here you can switch the trace on or off and display the trace files.

•

Alert Monitor: Choose Utilities > Alert Monitor > Display or start collection method. After starting the collection method you can view the alert monitor. (Option: Integration of the Alert Monitor into CCMS)

•

SAPphone version: Choose Utilities > SAPphone version to get the SAPphone version

2.3.21 Internet Communication Manager Monitor – SMICM Functional description

Use this transaction to monitor and administrate the ICM (Internet Communication Manager). The ICM receives and sends requests (i.e., in a server role, incoming HTTP requests) from or to the internet. The SAP Web Application Server can serve both as a (WEB) server and as a client. As a server, ICM accepts incoming HTTP requests and forwards them to the Internet Communication Framework for processing. As a client, requests (i.e. SMTP e-mails) are sent from the SAP server via ICM to any Internet server (i.e. Mail server). Handling Procedures

Status: you can check the status of Internet Communication Manager by calling transaction SMICM. The ICM status should be "runs" with green lights. Services: Choose Goto > Services to obtain the service list (e.g. SMTP, HTTP(S)) with corresponding ports. The column ‘Active’ shows you whether the service is running. Trace Files: Choose Goto > Trace file or Trace level. Here you can display or reset the trace file and set the trace level (values between 0 and 3 are permitted, default is 1). For normal operation, © 2002 SAP AG

52

Best Practice: mySAP CRM Field Sales Monitoring the ICM trace level should be set to 1. Higher trace levels should only temporarily be used during active error monitoring.

2.3.22 Gateway Monitor – SMGW Functional description

Analyzing the gateway trace gives you information about the RFC connection between the CRM Server and the Communication Station. Handling Procedures

Check if the connection is broken or still existing and check for CPIC errors.

2.3.23 R/3 Buffer Monitor – ST02 Functional Description The Buffer Monitor shows the current status and the memory resource usage for a specific R/3 application server. As part of the SAP Basis, the Buffer Monitor does not display any CRM specific monitoring data. It should be analyzed using the same procedures as those used in a non-CRM system environment. General Threshold Values - ST02 Name

Threshold value

Hit ratio (PXA, TABLP, EIBUF) Swaps (PXA) Swaps (other than PXA) Roll area Extended memory

>= 98% Details by double-clicking on the corresponding transaction name. Analyzing RFC Profiles In all CRM Scenarios, large part of processing in CRM is done by RFC. The RFC statistics should be therefore analyzed in detail. Choose: RFC Profiles, you can check client (sender) and server (receiver) records in detail. Keep in mind, that QRFCs and TRFCs are processed as function ARFC_DEST_SHIP. All other RFCs are displayed by function module in this statistics. You can double –click to get details of the call type. Select the function module and look into SE37 to get the short description. •

Check to see which functions are time consuming or very often executed or transport high mount of data.

2.3.25 Database Performance Analysis – ST04 Functional Description

© 2002 SAP AG

54

Best Practice: mySAP CRM Field Sales Monitoring The Database Performance Monitor does not display any CRM specific monitoring data. It should be analyzed using the same procedures as those used in a non-CRM system environment. Exception: Check the indices on the tRFC and aRFC tables (ARFC* and TRFC*). For MSA/MSE: monitor the disk I/O in high-load phases to find possible bottlenecks. See also transaction DB02. Refer to R/3 Monitoring Best Practice.

2.3.26 Operating System Monitor – ST06 Functional Description Transaction ST06 for monitoring the available swap space in the host system. Compare the hardware load for times of high RFC transmissions with the times of “normal” operations. You can detect the following bottlenecks from the following values: •

CPU bottleneck by CPU idle time of less than 5% or a CPU load greater than 3% A very high load and a CPU idle time of about 50% normally indicates an I/O bottleneck.

•

Memory bottleneck

•

UNIX page-out rate greater than 60 000 pages/hour

•

NT page-in rate greater than 60 0000 pages/hour

•

High file system utilization and wait times can indicate a disk bottleneck.

•

A high number of collisions can indicate network problems on your shared Ethernet network. General Threshold Values

Name

Threshold Value

CPU idle (UNIX) page-out rate (NT) page-in rate

>= 20%, or better >= 35% 40% of all WP (database problem?) > 20% of all WP (exp. SQL or excl. lock waits?)

Work processes in database action Work processes reading the same table at same time

© 2002 SAP AG

55

Best Practice: mySAP CRM Field Sales Monitoring By monitoring work processes in the SAP System (transaction SM50) or from outside the system (program DPMON at operating system level), you can determine the status of the work processes in relation to the PRIV mode. If work processes are often switched to the PRIV mode, you must increase the extended memory and/or adjust the limit for the extended memory). In this case, you must consult with SAP. •

You can use transaction SM66 to evaluate this information for all the work processes.

•

You can determine the swap space requirements at R/3 and operating system level.

•

Many monitoring tools are operating system dependent.

Transaction SM66 shows the SAP System work process overview for the local application server and for all other SAP application servers that are connected to the same SAP database (such as all application servers connected to the database and central SAP instance). Usage With help of this transaction you can detect long running transactions. Monitor periodically, and also check if there are sudden performance problems. Handling Procedures SM66 – check for ARFC programs running Make sure that there are enough work processes free for online users, if all processes are blocked, check the RFC parameters. Are there work-processes that are stopped or locked by CPIC to determine whether there are enough work processes configured for the system? In SM50, ARFC jobs show up as entries for the background work processes of rescheduled tRFCs..

2.3.28 Display Statistical Records – STAD Functional Description Shows who has done what and when. Expansion •

RFC statistic

•

Time analysis

Check whether it says “no subrecords” Check “RFC subrecords” to see who has done what.

2.3.29 ABAP Dump Analysis – ST22 Refer to partner systems for short dumps.

© 2002 SAP AG

56


2.3.30 System Log – SM21 Check the syslog for error messages referring to RFC and gateway with message number Q23 and Q22. Example: Deletion of single message in the queue (here in the outbound queue) (Q23, Q26)

Delete a single message, with TID 0A1089630DEC3D491F782EA9 The following system log entry is created 15:32:09 DIA 3 800 D036792 SMQ1 Q23 Missng:TSL1TE,Q23):D036792&0A1089630DEC3D491F782EA9 Error in SM21: Q23, user deleting and from which transaction as well, which client If something is deleted in system log

Technical details File 170136 Position 0000329040 Entry type Message ID Q2 3 Variable parts D036792&0A1089630DEC3D491F782EA9&CRM_SITE_000000000000100,CRM_SI

© 2002 SAP AG

57

Best Practice: mySAP CRM Field Sales Monitoring Example: Deletion of whole queue (here from the inbound queue) (Q22, Q25)

If a whole queue is deleted, this information is displayed (the TID are not shown anymore): 15:40:14 DIA 3 800 D036792 Q22 Missng:TSL1TE,Q22):D036792&CRM_SITE_000000000000100 Example: Deletion of a single message from the R&R Queues (CM1)

Example: deletion of all messages from the R&R Queues (CM1)

Client information is missing on this screen, but it is in the process. Example: Release and hold R&R Queues (CM0)

2.3.31 Update Error Monitor – SM13 Functional description The Update Error Monitor does not display any CRM specific monitoring data. It should be analyzed using the same procedures as those used in a non-CRM system environment.

© 2002 SAP AG

58


2.4 Troubleshooting This chapter describes problems that can arise in the CRM environment and helps handling and solving these problems. The possible problem solving strategies are described either as flow diagrams or as problem descriptions.

2.4.1 Problems during Data Exchange Problems and errors during data exchange can occur at several points in the CRM landscape. Unsuccessful data transfers may have one of the following causes: •

System capacity overload

•

Program crashes due to database problems, e.g. tablespace overflow

•

RFC connection problems (network related)

•

Incorrect middleware specific configuration settings

•

Incorrect application specific configuration settings

Whatever the reason, the exact cause of the problem must be determined and the problem fixed before the data transfer can be properly completed. The download or upload of objects should only be restarted after the errors have been found and repaired. The following problem analysis diagrams should be used as guides in correcting errors occurring during the initial download, during delta downloads and in troubleshooting upload problems. The basic recommendation is to first ensure that the download works properly before trying to correct any upload problems. Middleware specific situations are covered in the following flow diagrams - the CRM application settings may need to be handled by the application personnel. 2.4.1.1

Error Detection in Delta Download

If you experience problems with delta load of some objects, you can refer to the flow diagram below. This is a logical sequence of checks to analyze the problems and helps to determine and to eliminate the causes of the errors. For further information, refer to: •

SAP Note 430980 and/or to

•

SAP Library CD – Error Detection in Delta Download.

© 2002 SAP AG

59


THE DELTA DOWNLOAD OF SOME OBJECTS IS NOT

Error Detection in

SUCCESSFUL -

R3AM1 on the CRM server shows objects with - Status RUNNING - Date, Time, and Block Number remaining constant for longer than 5 minutes

NO

Delta Download Note 430980

OLTP System?

YES

Event Control

NO

Active? ➼ Table TBE11 Parameter: BC_MID, ACTIVE: X Parameter: NDI, ACTIVE: X

Parameter Settings OK?

➼ Table CRMCONSUM CONSUMER: CRM (or consumer application, e.g. BBP) ACTIVE: X TEXT: CRM (or a descriptive text) ➼ Table CRMRFCPAR CONSUMER: CRM (see table CRMCONSUM) OBJNAME: * RFCDEST: DOWNLOAD: * INACTIVE: DISCARDDAT: USE_IN_Q: X, SEND_XML: X HOLD_DATA:

(CONTINUED ON THE NEXT PAGE)

© 2002 SAP AG

60

Best Practice: mySAP CRM Field Sales Monitoring (Continued from the previous page)

OLTP


Check RFC Communication Queues

CRM

➼ SMQ2 Check R3AD* queues for errors

➼ SMQ1 Check R3AD* queues for errors

Short Dumps Exist?

YES?

➼ CRM System – ST22

NO?

➼ OLTP System – ST22 Analyze and fix problems

STOP Entries in the CRM Inbound Queue?

NO?

YES?

➼ SMW01: Analyse FLOW information on CRM system

Error messages from CRM Services in FLOW?

YES?

Refer analysis to application people

NO?


© 2002 SAP AG

61



Delta data passed-on from OLTP to CRM System?

NO?

➼ OLTP – SMQ1 ➼ CRM – SMQ2 Restart the queues manually

YES?

➼ CRM – SMW01 Release queue (manually unblock queue)

Initial Download completed successfully?

NO

➼ CRM - R3AM1 Ensure that all objects have GREEN status

YES

➼ OLTP – SMQ1 Ensure that all initial download objects are completed

Events for all application relevant objects activated?

YES

NO

➼ CRM - R3AC4 Activate the missing entries manually by using change mode and then save


© 2002 SAP AG

62


R3AD* Delta queues registered for processing?


NO

➼ CRM – SMQR

YES

Do LOCK entries exist for inbound queue entries in table SMO8_LQU?

All R3AD* queues must have “R“ in Type column

YES

➼ CRM – SM30 – table SMO8_LQU

NO

Delete the lock entries ➼ CRM – SMQ2 Manually restart the queue(s) (ACTIVATE) ➼ OLTP – SMQ1 Check for possible follow-on problems ➼ ST22 Analyze possible short-dumps

Do FILTER criteria exist for objects that have not been downloaded?

YES

➼ CRM – R3AC1 NO

Delete unwanted filters that may be preventing an object to download from OLTP Check with the application people before deleting any filters!


© 2002 SAP AG

63



Does the CRM Inbound queue stop after processing each object?

YES

➼ CRM – SMQ2 Check for errors ➼ CRM – Production client – table SMOFPARSFA Check for BREAK-POINTS (parameter BREAK_POINT) Delete the BREAK POINTS or deactivate them by entering a non-existent user-name for each

NO

Has a Message Flow been generated for the application relevant

NO

➼ SMO8FD Check whether the Message Flow has been generated. ➼ GNRWB Generate the Message Flow for all objects

YES

Are there still Delta Download problems?

YES

Call SAP Support NO

DONE

© 2002 SAP AG

64


2.4.1.2

Error Detection in Initial Download

References: •

SAP Notes 429423 and 350176

•

SAP Library CD Error Detection and Correction

THE INITIAL DOWNLOAD WAS NOT FINISHED SUCCESSFULLY -

R3AM1, Download Monitor, shows non-GREEN status for some objects

-

R3AM1, Download Monitor, shows GREEN status but no Objects are downloaded

-

SMW01, BDoc Display, shows error segments for downloaded objects

Tablespace problems?


YES

➼ CRM – SM21 the system log shows the affected tablespaces

NO

➼ Extent tablespace

Are the newest Support Packages installed - R/3 Plug-In? OLTP System? CRM System?

NO ➼ Check SAPNet for newest Support Package releases HTTP://Service.SAP.com/swcenter-main Upgrade to newest versions

YES

OLTP System Parameter Settings OK? YES

NO ➼ Table CRMCONSUM CONSUMER: CRM (or consumer application, e.g. BBP)) ACTIVE: X TEXT: CRM (or a descriptive text) ➼ Table CRMRFCPAR CONSUMER: CRM (see table CRMCONSUM) OBJNAME: * RFCDEST: DOWNLOAD: * INACTIVE: DISCARDDAT: USE_IN_Q: X, SEND_XML: X HOLD_DATA:

(CONTINUED ON THE NEXT PAGE) © 2002 SAP AG

➼ have you maintained the logical system of R/3- and CRM-System

65


Does the RFC connection between systems work?


NO?

➼ Test RFC connection between the systems OLTP System çè CRM System CRM System çè OLTP System

YES?

➼ CRM - SM58 check for errors in tRFC

Are there R3AI* queues waiting on the OLTP system?

YES

➼ R/3 - SMQ1 check for errors, restart queue in case of capacity overload

NO

Are there R3AI* (R3AMATERIAL) inbound queues on the CRM system?

NO

➼ CRM- ST22 check for short dumps

YES

YES ➼ CRM – SMQ2 check for errors, often there are customizing errors ➼ CRM - SMW01 analyze errors, correct the customizing settings

Are there Entries with error messages?

NO

➼ CRM – SMQ2 restart the queue ➼ CRM - SMQR check registration of queue

➼ CRM – SMQ2 Restart the processing of the queue ➼ Check note 309734

(CONTINUED ON THE NEXT PAGE) © 2002 SAP AG

66



Are there still Initial Download problems?

YES

Call SAP Support NO

DONE

Furthermore, there are several other problems related to the initial download: - R3AM1: the Download Manager shows that some objects remain in status WAIT: The reason is that this object has a parent object, and the initial load of the parent object was not yet completed successfully. Whether or an object has a parent object can be seen in transaction R3AM1 by the "HasPar" indicator. When it is set to "X", then the respective object has parent objects. These parent objects can be seen in table SMOFOBJPAR. - R3AM1: the Download Manager shows only few Objects in status RUNNING: The maximum allowed number of loads and requests is already being processed. This number can be set in table SMOFPARSFA via the MAX_PARALLEL_PROCESSES parameter (default = 5). If the number of running loads and the number of running requests in total is as much as or higher than this number, the remaining loads have to wait until a process is free again. - The R3AM1 transaction shows that an object from the initial download has a status of RED The initial load has not finished successfully. Analysis Errors occurred in the CRM System during the initial download and as a result the data was not posted into the CRM database. In order to be able to restart the data from the queue after elimination of the error an exception was probably triggered. Normally, the problem is caused by missing/incorrect customizing of the CRM online application. Handling: See SAP notes 429423, 350176 The reason why the initial load was not successful must be determined. Error messages and traces should be examined to see where the problem(s) might be. Use transactions SMW01 (BDoc Viewer) or SM08FT (Flow Trace) to view the error messages Correct the error based on the error message text Use transaction SMQ2 to restart the queue Use a similar procedure for subsequent errors

© 2002 SAP AG

67

Best Practice: mySAP CRM Field Sales Monitoring Do not a complete restart of the initial download. The download will resume from the point where it stopped after the error has been corrected. Customizing of the CRM online application as specified in the “SAP CRM Setup and Load Guide” may have to be done to avoid subsequent errors.

2.4.1.3

Error Detection in Upload

References: •

SAP Note 431345

•

SAP Library CD Error Detection in Upload

© 2002 SAP AG

68


THE UPLOAD OF SOME OBJECTS HAS NOT FINISHED SUCCESSFULLY -

SMW01 shows YELLOW or RED status for some objects

-

SMW01 shows GREEN Status, but the upload causes no changes


NO Does the Delta Download ➼ R/3 change i.e. the address of a business partner

YES

➼ Analyze errors using Error Detection in Delta Download

Are there error messages?

NO

YES

➼ CRM – SMW01 analyze all entries marked red or yellow find out which steps the message has already run through ➼ CRM - SMQ1, SMQ2 check for dependant queue entries ➼ R/3 – SMQ1, SMQ2 check for dependant queue entries ➼ CRM + R/3 – ST22 check for short dumps ➼ CRM – SMQ1 Debug LUW (Note 337753)


© 2002 SAP AG

69



NO

Did the upload cause changes?

➼ CRM – SMW01 analyze relevant entries

YES

➼ CRM – SMQ1 analyze similar message by using Debugging (Note 337753)

Are there still Upload problems?

YES

Call SAP Support

NO

DONE

© 2002 SAP AG

70


2.4.2 Problems with E-Mail Sending (Outbound Direction) NO MAILS GOING OUTSIDE

SMTP Plugin active?

Error Detection for sending Mails via SMTP

NO?

➼ Check the profile parameters – RZ10:

YES?

•

icm/plugin_x: PROT=SMTP,PLG=/usr/sap//SYS/exe/run/smt pplugin.dll

•

Icm/server_port_x: PROT=SMTP,PORT=8025 (default Port) ➼ Check path for the Plugin (AL11)

SMTP node active?

NO?

➼ Check the settings for the SMTP node – SCOT:

YES?

Send job running?

•

Mail host and Mail port correct

•

Address area correct

•

Node active

NO?

➼ Check the send job – SCOT: •

View > Jobs

YES?

© 2002 SAP AG

71


Error messages in CRM?

YES?

➼ Analyze errors -SCOT: •

Alert Monitor, send Orders (see above)

NO?

Are there still problems while sending?

YES?

➼ External Software: Look for error messages on the mail server) ➼ Call SAP Support

NO?

DONE

2.4.3 CRM Server Performance problems or high I/O load due to excessive traces and logs being written The performance of the CRM server is poor and may have some of the following symptoms •

Slow response times

•

High I/O load

•

Disk space increases relatively fast

•

Great data quantities are written to the database logs

•

Database is in a standstill condition

•

The Middleware Log and Flow control Trace tables are larger than 100 MB (transaction DB02 can be used to check the size of the tables)

Affected tables:

© 2002 SAP AG

72

Best Practice: mySAP CRM Field Sales Monitoring SMW3_BDOC, SMW3_BDOC1, SMW3_BDOC2, SMW3_BDOC4, SMW3_BDOC5, SMW3_BDOC6, SMW3_BDOC7 (Tables for Administration of BDoc messages) SMWT_TRC (Table for Middleware Trace). Analysis Possible causes •

High trace level settings are activated on the CRM server

•

The log level may be set incorrectly

Handling Check the relative sizes of the above listed tables using transaction DB02 è Detailed Analysis è Tables and Indexes Check the following traces, optimize their settings, and schedule a periodic reorganization of the tables as described in note 206439: As of Release 3.0: Set default trace level '1' for all trace environments except for 'G' (generation) of the Middleware trace. For this purpose, choose 'Middleware' -> 'Administration' -> 'Define Middleware Parameters' (R3AC6) from the SAP Menu and check entries with the following structure in the displayed table (SMOFPARSFA): Column

Value

Key

CRMGENERAL

ParamName

TRACE-LEVEL

ParamName2

ENV=m

Param.Wert

LEVEL=n

Where m is the trace environment and n the trace level that can be a number between 1 and 4. If this is higher than 1 in certain entries, except for those where the 'ParamName2' column contains the string 'ENV=G' (trace environment generation), set the default value 1 and then save the changes. The default value of the trace level for the trace environment generation (the entry where the ‘ParamName2' column contains the string 'ENV=G') is 4. Scheduling the reorganization Reference SAP Note 206439 Create a background job MW_REORG using variant SAP_MW_REORG and schedule the SMO6_REORG program as periodic job for execution every night. When you use the variant SAP_MW_REORG all trace and log data that is older than a week will be deleted. If you want to keep the data for a longer or shorter period create a variant for the SMO6_REORG report via Transaction SE38 and use it instead of SAP_MW_REORG for the job scheduling.

© 2002 SAP AG

73


2.4.4 Mass changes of data (creating, changing) on the OLTP system leads to reduced system performance Due to all of the dialog and background work processes becoming occupied the OLTP system has poor performance. Analysis: The outbound queue (SMQ1) of the OLTP system shows that there are between several hundred and several thousand different queues waiting to be processed. A large amount of the changed data from the OLTP system is intended for the delta download to the CRM Middleware Server and is therefore stored in the outbound queue of the OLTP before being transferred to the middleware server. For performance reasons qRFC tries to transfer as many queues as possible, as fast as possible. The result is that many parallel queues (with different queue names) are generated. The total number of queues that can be generated for parallel processing is limited by the possible names that can be assigned to the queues. Handling See SAP Note: 356228.

2.4.5 Performance problems due to statistics updates on tRFC/qRFC tables This applies only to SAP Systems using the Oracle database. Performance problems on the tRFC and qRFC tables ARFCSSTATE ARFCRSTATE TRFCQIN TRFCQOUT

ARFCSDATA TRFCQDATA TRFCQINS TRFCQSTATE

Analysis The size of the tRFC and qRFC tables can vary for applications (such as CRM) that frequently use this function. For databases that use a cost-based optimizer and do not keep the table statistics current, incorrect access paths may be chosen. This can cause a high database load or lead to excessive runtimes. Handling Update statistics should be switched-off (and deleted) for the database tables listed below Transaction DB21 should be used. For details see SAP Notes 330059 and 371068.

2.4.6 System performance degrades as the size of the tRFC/qRFC tables increases There is a high system interface load because 1 or more of the tRFC/qRFC tables have become too large. Affected tables:

© 2002 SAP AG

74

Best Practice: mySAP CRM Field Sales Monitoring ARFCSSTATE ARFCRSTATE

ARFCSDATA

TRFCQIN

TRFCQOUT

TRFCQSTATE

TRFCQDATA

Analysis 1 or more of the tRFC and qRFC tables have become bigger than 500 MB. The large tRFC and qRFC tables affect the performance of the tRFC and the qRFC interfaces. Handling Use transaction DB02 – Detail Analysis to check the relative sizes of the tables. Reorganize these tables regularly to avoid future performance problems. Reorganization will decrease the sizes of the tables. Refer to the SAP Note 375566 for details concerning administration of these tables Note: Never delete the contents of this tables otherwise data inconsistencies might occur.

2.4.7 Problem situation 1 Roll-Out of new mobile clients with productive mobile client. Organization of the Roll-Out timeframe to avoid performance problems. No parallelization of the R&R is possible.

2.4.8 Problem situation 2 Do not restart a new Initial load

2.4.9 Problem situation 3 BDOC Fehlerhandling procedure, consistency of data

2.4.10 Problem situation 4 Mobile Brdge activation and deactivation

© 2002 SAP AG

75


Checklists SAP Basis (CRM Server and R/3 Back-end Server) SMGW ST02 ST03N ST04 DB02 ST06 SM50 SM66 ST22 SM21 SM13 STAD

R/3 Back-end System SMQ1 SMQS BGD

--

CRM Server SMWP SMQ1 SMQS SMQ2 SMQR SM58 SMWMFLOW SMW01/SMW02 SMWT SMW03 SMO8FD R3AM1 R3AR3 SMWMQUEUES SMOHQUEUE SMWMCOMM CMWQ

© 2002 SAP AG

76


Feedback and Questions Send any feedback by composing an SAP customer message to component SV-GST. You can do this at http://service.sap.com/message. © Copyright 2001 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation. IBM®, DB2®, OS/2®, DB2/6000®, Parallel Sysplex®, MVS/ESA®, RS/6000®, AIX®, S/390®, AS/400®, OS/390®, and OS/400® are registered trademarks of IBM Corporation. ORACLE® is a registered trademark of ORACLE Corporation. INFORMIX®-OnLine for SAP and Informix® Dynamic ServerTM are registered trademarks of Informix Software Incorporated. UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA® is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, SAP Logo, R/2, RIVA, R/3, ABAP, SAP ArchiveLink, SAP Business Workflow, WebFlow, SAP EarlyWatch, BAPI, SAPPHIRE, Management Cockpit, mySAP.com Logo and mySAP.com are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other products mentioned are trademarks or registered trademarks of their respective companies. Disclaimer: SAP AG assumes no responsibility for errors or omissions in these materials. These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party web pages nor provide any warranty whatsoever relating to third party web pages.

© 2002 SAP AG

77