Blackboard Learn Performance Optimization Guide - Behind the ...

Release 9.1

Performance Optimization Guide

© 2 0 1 0 B l ac k b o a r d I n c . P r o p r i e t a r y a n d C o n f i d e n t i a l

Publication Date: June 11, 2010 Worldwide Headquarters

International Headquarters

Blackboard Inc.

Blackboard International B.V.

650 Massachusetts Avenue NW Sixth Floor Washington, DC 20001-3796

Dam 27 2nd Floor 1012 JS Amsterdam The Netherlands

+1 800 424 9299 toll free US & Canada +1 202 463 4860 telephone

+31 20 5206884 (NL) telephone

+1 202 463 4863 facsimile

+31 20 5206885 (NL) facsimile

www.blackboard.com

www.blackboard.com

Copyright © 1997-2010. Blackboard, the Blackboard logo, BbWorld, Blackboard Learn, Blackboard Transact, Blackboard Connect, the Blackboard Outcomes System, Behind the Blackboard, and Connect-ED are trademarks or registered trademarks of Blackboard Inc. or its subsidiaries in the United States and other countries. U.S. Patent Numbers: 6,988,138; 7,493,396; 6,816,878. Apache is a trademark of The Apache Software Foundation. Microsoft, SQL Server, and Windows are registered trademarks of Microsoft Corporation in the United States and other countries. NetApp is a registered trademark of NetApp, Inc. in the U.S. and other countries. Oracle is a registered trademark of Oracle Corporation and its affiliates. Quest and Foglight are registered trademarks or trademarks of Quest Software, Inc. in the United States and other countries. Sun, Java, Java EE, J2SE, and JVM are registered trademarks or trademarks of Sun Microsystems, Inc. in the United States and other countries. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Ltd. Other product and company names mentioned herein may be the trademarks of their respective owners. No part of the contents of this manual may be reproduced or transmitted in any form or by any means without the written permission of the publisher, Blackboard Inc.

Blackboard Learn Performance Optimization Guide ©2010 Blackboard Inc. Proprietary and Confidential

Page 2

Contents Introduction ...............................................................................................................................5 An Enterprise Application Built for Performance .............................................................. 5 Optimizing Performance..................................................................................................6 Optimizing Application Tier Performance .................................................................. 7 Optimizing backlog="0" enableLookups="false" minThreads="1" maxThreads="40" protocol="org.apache.jk.server.JkCoyoteHandler" soTimeout="20000" tomcatAuthentication="false" URIEncoding="UTF-8" useBodyEncodingForURI="true" useURIValidationHack="false" /> 4. Save and close the file. 5. Restart the Blackboard Learn application. Note

The server.xml file will be overwritten each time the PushConfigUpdates command is run. To make changes permanent, follow the steps in the next section.


Page 31

To permanently enable Tomcat thread queuing 1.

Open the server.xml.bb file in the following directory: UNIX: %/usr/local//config/tomcat/conf Windows: C:\\config\tomcat\conf

2.

If Tomcat clustering is in use, open the server.xml.template file in the following directory: UNIX: %/usr/local//config/tomcat/cluster/ conf Windows: C:\\config\tomcat\cluster\conf\

3. In both files, change the value of the backlog attribute to at least 10 to increase the number of requests that will be placed on the Tomcat queue if no Tomcat threads are available to process a request from the web server. You change increase this value as needed. 4. Save and close the files. 5. Update the configuration by running the PushConfigUpdates command: •

UNIX: %/usr/local/tools/admin/ PushConfigUpdates.sh

•

Windows: C:\\tools\admin\ PushConfigUpdates.bat


Page 32

6. If Tomcat clustering is in use, update each cluster node by running the UpdateCluster command: • UNIX: %/usr/local/tools/admin/ UpdateCluster.sh •

Windows: C:\\tools\admin\ UpdateCluster.cmd


Page 33

Using Tomcat Clusters Tomcat Application Clusters consist of multiple Blackboard Learn java application (JVM) server instances running simultaneously (on the same physical server) and working together to provide increased scalability. Scalability is an application's ability to support a growing number of users. If it takes one application 10 milliseconds to respond to one request, how long does it take to respond to 10,000 requests? If a user logs on at 3 AM do they experience the same responsiveness as they do at 3 PM? Tomcat Application Clusters ensure that Blackboard Learn remains responsive by adding server instances to a cluster without interruption of service. Note

Cluster nodes cannot be modified. If a change is required, the node must be removed and then added again with the updated information.

Tomcat Application Clusters are not meant as a replacement for traditional load balance configurations, but as a complementary scheme that provides improved scalability and failover capabilities. Traditional load-balancing is a physical distribution of server instances across multiple servers. Tomcat Application Clusters are logical instances of the Java application components that reside on a single server, or are distributed across multiple servers. There are two types of server clustering: horizontal and vertical: •

Horizontal clustering allows server instances to be deployed across multiple physical servers. This method of clustering is not implemented by Blackboard because Blackboard already supports load-balancing.

•

Vertical clusters, also known as multi-home clusters, allow multiple server instances to be run on a single machine. This method takes full advantage of the processing power of a single server. Vertical clusters are load-balanced by the web server.

Tomcat Clustering runs on all Blackboard platforms (Windows, Linux, and Solaris).

Installing One or More Tomcat Clusters This section includes all of the procedures needed to install and configure Tomcat Application Clusters. These procedures assume an Administrator has full administrative rights as either administrator or root. Before Installing a Cluster The server must be configured to support clustering prior to installing a Tomcat Cluster node. Clustering is disabled by default and must be enabled. Installing a cluster requires that the application instance be shutdown and restarted. Configuring in Windows: 1. Navigate to the following directory: \config 2. Open the bb-config.properties file. 3. Search for the bbconfig.tomcat.cluster.enable variable, and modify the default value from 'FALSE' to 'TRUE'. 4. Open the context.xml file and uncomment the option to disable session persistence across Tomcat restarts.


Page 34

5. Within the context.xml file, the listening port assigned to the Tomcat nodes must be different for each node. 6. Navigate to the following directory: \tools\admin 7. Launch the Push Config Updates script by running the PushConfigUpdates.bat file. 8. Create a new ServerGroup in Blackboard Learn a. Navigate to the following directory: \tools\admin b. Create a ServerGroup with a specified port number by running the following command: ServerGroupManager.bat -c -n -p

Configuring in UNIX: 1. Navigate to the following directory: /config 2. Open the bb-config.properties file. 3. Search for the bbconfig.tomcat.cluster.enable variable, and modify the default value from 'FALSE' to 'TRUE'. 4. Open the context.xml file and uncomment the option to disable session persistence across Tomcat restarts. 5. Within the context.xml file, the listening port assigned to the Tomcat nodes must be different for each node. 6. Navigate to the following directory: /tools/admin 7. Launch the Push Config Updates script by issuing this following command: ./PushConfigUpdates.sh 8. Create a new ServerGroup in Blackboard Learn a. Navigate to the following directory: /tools/admin b. Create a ServerGroup with a specified port number by issuing the following command: ./ServerGroupManager.sh -c -n -p Note

In a Windows or UNIX load-balanced environment, the PushConfigUpdates script must be run on each load-balanced server. Each Node must have its own ServerGroup.


Page 35

How to Install These procedures assume an Administrator has full administrative rights as either administrator or root. Upon creating an individual cluster node, the Blackboard services will need to be shutdown and restarted in order to reset the configuration with the new clustered node. Pro p erty

Desc ri p ti o n

Cluster ID

Must be a unique name with no spaces between any of the letters. Best practice is to identify one naming convention for all nodes and increment numerically to differentiate servers and nodes. Example: Blackboard1 or Blackboard2.

Server Shutdown Port

Must be a unique port in order to connect to the JVM for the clustered node and shut it down. By default the primary application server installed with Blackboard runs on port 8005, therefore a new value must be supplied to avoid port socket contention. Best practice is to increment the current server shutdown port by 1000.

JK Connector Port

Must be a unique port in order for the web server to communicate to each clustered node. The default port is 8009. Best practice is to increment the current jk connector port by 1000.

Cluster Listener Port

Must be a unique port for the cluster to replicate session information across the wire. The default port is 4000. Best practice is to increment the current listen port by 1000.

Installing in Windows: 1. Launch the Install Cluster script by running the InstallCluster.bat file. After running the file, you will be prompted for the following information. Configure the settings according to the previous table. a. Cluster ID. b. Server shutdown port. c. JK Connector port. d. Cluster Listener port. 2. The cluster is now created. 3. Stop your server by running the following command: ServiceController.bat services.stop 4. Start your server by running the following command: ServiceController.bat services.start

Installing in UNIX: 1. Launch the Install Cluster script by issuing ./InstallCluster.sh After running the command, you will be prompted for the following information. Configure the settings according to the previous table. a. Cluster ID. b. Server shutdown port.


Page 36

c. JK Connector port. d. Cluster Listener port. 2. The cluster is now created. 3. Stop your server by issuing the following command: ./ServiceController.sh services.stop 4. Start your server by issuing the following command: ./ServiceController.sh services.start 5. Run the jps command from your JAVA_HOME to see the bootstrap processes. You should see one Tomcat Process, and the number of Cluster Process IDs.

Tip

UNIX customers should increase the Apache MaxClients value as they add nodes. Blackboard recommends multiplying the initial value by the number of total JVMs. For example, if MaxClients is 500 for a non-clustered configuration, when 2 JVMs are added, this setting should increase to a minimum of 1500.

Files that are Affected After a cluster is installed, there are a number of files and directories that are produced. The following table highlights some of these files and directories. Di recto ry o r F i l e

Desc ri p ti o n

/apps/to mcat/ cluster/

A new directory that is created when the first node (other than root) is installed.

/apps/to mcat/cluster/

A new directory that is created for every node added. Each directory contains files affecting that node: …/conf/server.xml and log files.

/apps/to mcat/ conf/jk/workers.properties

This file exists before adding a node but changes whenever a node is added to include load-balancing information for that node. For more information about the workers.properties file, see http://tomcat.apache.org.

server.xml

A file that allows you to configure Tomcat nodes through the use of XML descriptors.

Troubleshooting Installation Issues This topic contains three steps to take if the Tomcat Cluster is not functioning as expected. 1. Check that the following value is present and set to true in the bb-config.properties file located in the following /config directory: bbonfig.tomcat.enable=true If it is set to false, then run PushConfigUpdates.bat to change. For more information, see Before installing a cluster.


Page 37

2. Check that the workers.properties file contains the correct information for all of the nodes installed in the cluster. For more information about the workers.properties file, see Files that are affected. 3. Watch the Java processes running on your application server. There should be a Java process running for each node installed and one for the root node. CPU processing should be distributed across the nodes.

Enabling Cache Replication By default, Blackboard Learn will install with cache replication in a cluster disabled. To enable replication, manually configure the setting and restart the application servers. Replication in Windows: 1. Navigate to the following directory: \config\internal/ 2. Open the ehcache.xml file. 3. Search for the two cacheManagerxxx elements, and remove the comments. 4. Additional information is located within the ehcache.xml file, specific to the elements. 5. Stop your server by running the following command: ServiceController.bat services.stop 6. Start your server by running the following command: ServiceController.bat services.start

Replication in UNIX: 1. Navigate to the following directory: /config/internal/ 2. Open the ehcache.xml file. 3. Search for the two cacheManagerxxx elements, and remove the comments. 4. Additional information is located within the ehcache.xml file, specific to the elements. 5. Stop your server by issuing the following command: ./ServiceController.sh services.stop 6. Start your server by issuing the following command: ./ServiceController.sh services.start After restarting, all caches which are marked as bbconfig.cache..needsclusterinvalidation=true in cache-settings.properties will send invalidation notifications to all nodes in the system when entries are removed/flushed/updated.


Page 38

Removing a Cluster Node These procedures assume an Administrator has full administrative rights as either administrator or root. Upon removing an individual cluster node, the Blackboard services will need to be shutdown and restarted in order to reset the configuration without the clustered node. Removing in Windows: 1.

Navigate to the following directory: \tools\admin

2.

Stop your server by running the following command: ServiceController.bat services.stop

3.

Launch the Remove Cluster script by running the following command: RemoveCluster.bat

4.

Specify the cluster to be deleted when prompted.

5.

Start your server by running the following command: ServiceController.bat services.start

Removing in UNIX: 1.

Navigate to the following directory: /tools/admin

2.

Stop your server by issuing the following command: ./ServiceController.sh services.stop

3.

Launch the Remove Cluster script by issuing the following command: ./RemoveCluster.sh

4.

Specify the cluster to be deleted when prompted.

5.

Start your server by issuing the following command: ./ServiceController.sh services.start

6.

Verify the cluster was successfully removed by running the jps command from your JAVA_HOME to see the bootstrap processes.


Page 39

Optimizing Microsoft SQL Server When using Blackboard Learn 9.1 in a Windows environment, the deployment must use Microsoft SQL Server 2005 to serve data. Note

Future versions of Blackboard Learn may use a newer version of Microsoft SQL Server. For up to date information about supported configurations, go to the Blackboard Maintenance Center by logging into Behind the Blackboard (https://behind.blackboard.com).

The recommendations in this section come from experience in the Blackboard Performance Engineering Laboratory configuring a high-performance Microsoft SQL Server production environment. The information in the following sections can be used to optimize the SQL Server 2005 database: •

Pre-Installation Recommendations

•

Post-Installation Recommendations

•

Monitoring Key Areas


Page 40

Pre-Installation Recommendations This section provides best practices for configuring the hardware environment in which Microsoft SQL Server 2005 will reside. Before continuing, read the following information: •

Guidance provided by Microsoft (http://msdn.microsoft.com/en-us/library/ms143719(SQL.90).aspx)

•

SQL Server I/O Basics (http://www.microsoft.com/technet/prodtechnol/sql/2005/iobasics.mspx)

Pre-Allocating LUNs When setting up the SQL Server 2005 environment, Blackboard recommends creating multiple logical unit numbers (LUNs) for specific purposes to reduce the amount of unnecessary paging. Blackboard recommends creating a LUN for each of the following: •

The Windows Operating System Note

Size this LUN to handle the full installation of the Operating System only.

•

The paging file (optional)

•

The SQL Server 2005 product installation

•

The TEMPDB space

After Blackboard Learn has been installed, configure additional LUNs for the Blackboard schemas. Create LUNs for each schema or create a subset of LUNs based on data file growth and usage. Data files and transaction logs should always reside on separate LUNs. Blackboard recommends that creating an additional LUN for each of the following: •

The BBLEARN (legacy: BB_BB60) schema data file

•

The BBLEARN (legacy: BB_BB60) schema transaction log

•

The remaining Blackboard schema data files

•

The remaining Blackboard schema transaction logs

In total, eight LUNs would be pre-allocated: •

LUN 1: Operating System

•

LUN 2: Windows Paging File (Optional)

•

LUN 3: SQL Server 2005 Product Installation

•

LUN 4: TEMPDB Files

•

LUN 5: BBLEARN (legacy: BB_BB60) Data File

•

LUN 6: BBLEARN (legacy: BB_BB60) Transaction Log

•

LUN 7: Remaining Blackboard Schema Data Files

•

LUN 8: Remaining Blackboard Schema Transaction Log


Page 41

Configuring the Page File The paging file is a hidden file that resides on the boot partition of a server. The Windows Operating System uses the paging file as additional random access memory (RAM). The paging file and physical memory constitute the virtual memory. The default paging file size is equal to three times the total RAM. The primary reason for configuring such a large paging file is to have a file large enough to handle a SQL Server memory dump during a software failure. When the paging file resides on the boot partition, Windows must perform disk read and write requests to both the system folder and the paging file. Placing the paging file on a different partition reduces competition between read and write requests. To enhance performance, Blackboard recommends placing the paging file on a different partition as well as on a different physical hard disk drive so that Windows can handle multiple I/O requests quickly.


Page 42

Post-Installation Recommendations This section provides best practices for configuring Microsoft SQL Server 2005. Before continuing, read the following information: •

Administering the Database Engine (http://msdn.microsoft.com/en-us/library/ms191470(SQL.90).aspx)

•

Troubleshooting SQL Server Performance (http://www.microsoft.com/technet/prodtechnol/sql/2005/tsprfprb.mspx)

Configuring SQL Server Settings For the most part, SQL Server is a fairly independent, self-tuning system that adjusts to the workloads of the user community. To further improve performance, modify the following values using the SQL Server Management Studio: •

Max Degree of Parallelism – The Max Degree of Parallelism (MAXDOP) is a configuration indicating how the SQL Server optimizer will use the CPUs. This is a server-wide configuration that by default uses all of the CPUs to have the available portions of the query executed in parallel. The default MAXDOP configuration for SQL Server is 0; however, Blackboard recommends lowering this setting to lower than n – 1, where n is equal to the number of CPUs. In the Blackboard Performance Engineering Laboratory, the MAXDOP is often to set to a value of 1. Rarely has there been a performance difference by setting the value from 1 to any number higher than 1. Query parallelism can become a CPU resource saturation problem when the database optimizer incorrectly distributes the query across multiple CPUs. To determine whether an unlimited MAXDOP setting is causing high synthetic CPU utilization, look for the wait event CXPACKET (Wait Event Category of Parallelism).

•

Address Windowing Extensions (AWE) – AWE allows 32-bit operating systems to access large amounts of memory. It is exposed by the operating system and is implemented with slight differences in Microsoft Windows 2000 Server and Windows Server 2003. To run locked pages in memory, the SQL Server operating system user account must have administrative permissions. After enabling and configuring AWE from the SQL Server Administration console, SQL Server will now be able to take advantage of more than 2GB of memory. For more information, see http://support.microsoft.com/kb/274750. Note

•

AWE can be enabled and configured on a 64-bit operating system. For more information, see http://blogs.msdn.com/slavao/archive/2005/04/29/ 413425.aspx.

Maximum Worker Threads – Maximum Worker Threads are SQL Server threads that execute user or batch requests. A worker is bound to a batch until it completes, so the maximum number of workers limits the number of batches that execute concurrently. Blackboard recommends using the default setting of 0 because it will spawn the appropriate number of Worker Threads based on need. For more information, see http://support.microsoft.com/kb/319942.


Page 43

•

Fiber Mode – Avoid using Fiber Mode. A fiber is a lightweight process that is similar to a thread issued by the Operating System. Use fiber mode only when Microsoft recommends doing so in feedback from a support incident. For more information, see http://msdn.microsoft.com/en-us/library/aa175385.aspx.

Configuring READ_COMMITTED_SNAPSHOT Isolation Level The READ_COMMITTED_SNAPSHOT database option determines “what locks SQL Server takes when accessing data and, therefore, by extension, they determine the level of concurrency 2 and consistency that statements and transactions experience.” READ_COMMITTED_SNAPSHOT specifies that data read by any statement in a transaction will be the transactionally consistent version of the data that existed at the start of the transaction. The transaction can recognize only those data modifications that were committed before the start of the transaction. Data modifications made by other transactions after the start of the current transaction are not visible to statements executing in the current transaction. The effect is that the statements in a transaction get a snapshot of the committed data as it existed at the start of the transaction. Blackboard recommends setting READ_COMMITTED_SNAPSHOT to ON for the BBLEARN (legacy: BB_BB60) schema: ALTER DATABASE BBLEARN SET READ_COMMITTED_SNAPSHOT ON GO To disable READ_COMMITTED_SNAPSHOT, change the value to OFF: ALTER DATABASE BBLEARN SET READ_COMMITTED_SNAPSHOT OFF GO

Working with the TEMPDB The TEMPDB is one of the most critical areas for optimizing performance of SQL Server. Most complex query operations take place in this region of SQL Server. When configured incorrectly, performance can be affected dramatically. By default, the space is not meant to shrink (nor do we recommend auto-shrinking the space), but a penalty does exist for this space growing. Size this space carefully based on the needs of the application. The TEMPDB is often sized to 8MB by default. It continues to grow as long as the SQL Server is online. Restarting the SQL Server process will force the TEMPDB to return to its default size. This is why users often complain about the overall responsiveness of the system after maintenance. Part of the problem could be that the TEMPDB needs to grow, but another factor could be the flushing of the procedure cache during the restart operation. The culprit of slow performing queries on a system with a poorly configured TEMPDB is often latency caused by TEMPDB data file growth.

2

Quotation taken from Craig Freedman's SQL Server Blog: “Read Committed Isolation Level” (http://blogs.msdn.com/craigfr/archive/2007/04/25/read-committed-isolation-level.aspx). For more information about the option, see http://msdn.microsoft.com/en-us/library/ms173763.aspx.


Page 44

Sizing the TEMPDB Blackboard recommends the following when sizing the TEMPDB: •

To determine the actual TEMPDB size needed, pre-size the TEMPDB to 2GB and let it grow over several days of high usage.

•

To determine the data file size for each CPU, divide the actual TEMPDB size by the number of CPUs available.

•

To enable the last file to grow if additional storage is needed, disable auto-growth for each of the data files except the last one. For example, suppose that an 8 CPU server needs roughly 4GB of actual TEMPDB space. Configure eight data files, each pre-sized to 512MB. Disable auto-growth for the first seven files, and enable auto-growth for the final file.

Using the SQL Server Performance Dashboard Microsoft provides an open source diagnostic tool called SQL Server Performance Dashboard. The Performance Dashboard reports enable SQL Server Administrators and other users to monitor the health of the SQL Server and diagnose issues. Although it relies upon Reporting Services definition files (.rdl), Reporting Services does not need to be installed to use the Performance Dashboard Reports. This custom report set relies on SQL Server’s dynamic management views (DMVs) as a data source, which provides the wealth of data that the dynamic management views contain but insulates the user from the views and underlying structures. Accessing and using this storehouse of performance information does not require additional sources, data capture, or tracing. Other obvious benefits of using these prefabricated views include the constant availability of the information that they contain and their inexpensive nature (from the tandem perspective of collection and querying) as a source of server monitoring. The Performance Dashboard provides roughly one hour of system history. It is an effective tool for performing immediate forensics when a problem is reported, but it cannot be used for longterm monitoring. To download the tool from Microsoft, see http://www.microsoft.com/downloads/details.aspx? FamilyId=1d3a4a0d-7e0c-4730-8204-e419218c1efc&displaylang=en. For more information about getting the most out of the Performance Dashboard, see Appendix B.

Tracking Wait Events and Queues Waits and queues can be used to identify performance improvement areas within SQL Server. Wait events indicate where SQL Server is spending time waiting instead of servicing a request. When a request is waiting, the response time experience of a user is affected directly. The longest wait events highlight the most important or relevant queues (that is, Performance Monitor counters and other data). The wait event information can be extracted easily from the Performance Dashboard. For information about the methodology and each of the wait events, see http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/performance_tuning_ waits_queues.mspx.


Page 45

Monitoring Key Areas In its Troubleshooting Performance Problems in SQL Server 2005 document (http://www.microsoft.com/technet/prodtechnol/sql/2005/tsprfprb.mspx), Microsoft recommends monitoring the key areas and counters in SQL Server listed in the following table. Area Name: Counter Name

Description

Memory: Pages/sec

Rate at which pages are read from or written to disk during hard page faults As the memory pages are transferred to and from the relatively slow disk storage, the system will experience slow performance. Although it may spike occasionally, this counter should average 20 or less. If the counter is too high on a regular basis, add memory to the server.

Memory: Available Bytes

Amount of physical memory available for allocation At least 5MB of RAM should be available. If the amount available is less than 5MB, consider adding RAM to the server.

Network Interface: Bytes Total/sec

Per-second rate at which data is passing though the network interface card (NIC)

Physical Disk: % Disk Time

Percentage of time that the disk is busy Although it may spike occasionally, this counter should be less than 55% during any sustained period. If the counter is too high, consider doing one or more of the following: • Add drives to the array to increase the spindles and spread the load. • Add additional channels. • Change the RAID version to a higher performance version (for example, RAID 5 to RAID 0+1).

Physical Disk: Avg. Disk Queue Length

Number of disk requests that are waiting in the queue According to queuing theory, this counter should be less than 2 over any sustained period or the queue could become backlogged. If the counter is too high, consider doing one or more of the following: • Add drives to the array to increase the spindles and spread the load. • Add additional channels. • Change the RAID version to a higher performance version (for example, RAID 5 to RAID 0+1).


Page 46

Area Name: Counter Name

Description

Processor: % Processor Time

Time that the processor is doing actual work This value is calculated by measuring the percentage of time that the processor is idle and subtracting that from 100 to get the time the processor is busy doing work. Although it will spike occasionally, this counter should be less than 80% over any sustained period. If this counter is too high, consider adding or upgrading the processors on the server.

System: Processor Queue Length

Number of threads in the processor queue Even on computers with multiple processors, a single queue exists for processor time. If this exceeds 2 per processor (that is, 4 on a 2-processor server or 8 on a 4-processor server), consider adding or upgrading the processors on the server.


Page 47

Optimizing Oracle Database When using Blackboard Learn Release 9.1 in a UNIX environment, use Oracle Database 10g R2 to serve data. Note

Future versions of Blackboard Learn may use a newer version of Oracle Database. For up to date information about supported configurations, go to the Blackboard Maintenance Center by logging into Behind the Blackboard (https://behind.blackboard.com).

The recommendations in this section come from experience in the Blackboard Performance Engineering Laboratory configuring a high-performance Oracle production environment. Oracle performance tuning is one of the more discussed subjects on the Internet. Hundreds of books have been written about this subject. By no means is this guide meant to replace any of these resources. Before continuing, review the Oracle Databse 10g R2 Performance Tuning Guide (http://download.oracle.com/docs/cd/B19306_01/server.102/ b14211/toc.htm), which includes information about designing for performance and scalability, techniques for optimization, and an introduction to managing a growing system for capacity. The information in the following sections can be used to optimize the Oracle Database: •

Pre-Installation Recommendations

•

Post-Installation Recommendations

•

Configuring Temporary Tablespaces

•

Setting Oracle Initialization Parameters

•

Using the Oracle Wait Interface


Page 48

Pre-Installation Recommendations This section provides best practices for configuring the hardware environment in which Oracle will reside. Note

Most hardware and storage vendors provide specific guidance for designing a high performance Online Transaction Processing (OLTP) Oracle environment as part of the installation and set-up activities. Before installing Oracle, contact the vendor to obtain the best practices for configuring the storage sub-system.

Configuring the Storage Configuration From a product installation and data file perspective, Oracle has the flexibility to be deployed in a variety of storage configurations. Oracle Blackboard Learn deployments can use any of the following configurations: •

Network File System (NFS)

•

ISCSI (IP-SAN)

•

Fiber-Channel SAN

•

Local disks (SCSI/JBOD)

The major hardware and storage vendors have published many performance benchmarks regarding which storage platform provides the best overall performance. Blackboard leaves it up to each client to make the best decision with regard to maintainability, growth, and cost. Note

NetApp™ is a storage vendor in the NFS space that also has ISCSI and Fiber-Channel SAN deployments. For more information about storage configuration options, see their white paper (http://media.netapp.com/documents/tr-3423.pdf) describing performance differences across the three storage configurations.


Page 49

Post-Installation Recommendations Forecasting Memory Utilization Oracle offers automatic tuning capabilities that simplify the performance management process for database administrators. To address memory management, regularly measure the following four aspects: •

Host Operating System

•

System Global Area (SGA)

•

High-Watermark of Connections

•

Program Global Area (PGA)

Blackboard recommends allocating roughly 10% to 20% of available memory to the operating system, and then calculating the connection high-water mark to identify the demands of PGA. When not using the Oracle multi-threaded server, each Oracle connection can use up to 2MB of RAM. Additionally, factor in the SORT_AREA_SIZE and HASH_AREA_SIZE, which are session-controlled variables.

Sizing the System Global Area Oracle offers the ability to auto-size the System Global Area (SGA) using an automatic tuning mechanism. By setting the SGA_TARGET to a particular memory allocation and setting STATISTICS to TYPICAL, Oracle will be able to decide when to move memory between the database cache and the shared pool. If a dedicated DBA is not available to study SGA utilization and make appropriate forecasting changes, this makes sense as a viable setting. For guidance on whether to use an automatic tuning approach and the best approach to doing so, see http://www.oracle.com/technology/products/manageability/database/index10g.html. The following table provides a short summary of the key SGA components. If manually tuning the SGA region, it is imperative that the DBA forecast the memory needs of the system. This requires a set of exercises to characterize the workload, as well as account for unexpected utilization increases when demand in not necessarily transparent. Blackboard recommends accounting for a 20% spike. SGA Component

Areas Of Influence

Initialization Setting

Description

Shared Pool: Library Cache

SHARED_POOL_SIZE

Shared SQL areas Private SQL areas PL/SQL procedures and packages Various control structures

Oracle needs to allocate and deallocate memory as SQL or procedural code is executed based on the individual needs of users' sessions and in accordance with the LRU algorithm.

Shared Pool: Dictionary Cache

SHARED_POOL_SIZE

Row cache Library cache

Highly-accessed memory structures that provide information about object structures to SQL statements being parsed.


Page 50

SGA Component

Areas Of Influence

Initialization Setting

Description

Java Pool

JAVA_POOL_SIZE

Data in JVM

Memory available for the Java memory manager to use for all things Java.

Streams Pool

STREAMS_POOL_SIZE

Stream activity

Memory available for stream processing.

Redo Log Buffer

LOG_BUFFER

Redo Entries

Holds changes made to data and allows for the reconstruction of data in the case of failure.

Database Buffered Cache

DB_2K_CACHE_SIZE DB_4K_CACHE_SIZE DB_8K_CACHE_SIZE DB_16K_CACHE_SIZE DB_32K_CACHE_SIZE DB_KEEP_CACHE_SIZE DB_RECYCLE_CACHE_SIZE

Write list LRU list

Holds copies of data requested by SQL and reduces requests to disk by having data in memory. Many different buffer caches can help segregate on usage patterns.

Large Pool

LARGE_POOL_SIZE

Shared server I/O server processes Backup and restore

For large memory allocations.

Sizing the Program Global Area The Program Global Area (PGA) is a private memory region containing data and control information for a server process. Access to it is exclusive to that server process and is read and written only by the Oracle code acting on behalf of it. Oracle Database 10g introduced automatic PGA memory management to simplify and improve the way in which PGA memory is allocated. By default, PGA memory management is enabled. In this mode, Oracle dynamically adjusts the portion of the PGA memory dedicated to work areas based on 20% of the SGA memory size. The minimum value is 10MB. Sizing this area depends heavily on factors such as the amount of RAM available on the system, the number of processes/connections accessing the system (high-water mark), and the degree of sort and hash area space for query operations. For more information about determining how much PGA is needed, see http://www.scribd.com/doc/6469805/How-to-Determine-if-PGA-Size-is-Set-Properly.


Page 51

Configuring Temporary Tablespaces Configuring the TEMP Tablespace Oracle provides temporary tablespaces that are optimized for the storage of transient data (data available only for the duration of the session). Sort operations are the primary user of temporary tablespaces. For example, if two large tables are joined as part of a query, and Oracle cannot do the sort in memory, space will be allocated in a temporary tablespace for performing the sort operation. The following SQL operations, all of which can be found in Blackboard Learn, might require disk sorting: •

CREATE INDEX

•

ANALYZE

•

SELECT DISTINCT

•

ORDER BY

•

GROUP BY

•

UNION

•

SORT-MERGE

SORT_AREA_SIZE Initialization Parameter Sorting is managed by the SORT_AREA_SIZE initialization parameter, which is managed on a per session basis. For example, if the SORT_AREA_SIZE is set to 16KB and there are 1000 sessions, the maximum size would be 16MB. The sort space identified by this parameter is maintained in PGA memory. Sort operations are attempted first in the PGA sort area, just as well as hash join operations are attempted first in the PGA hash area. There are different tuning philosophies with regard to pre-allocating space to this region or making use of automatic PGA memory management mode. Although larger sort and hash areas can dramatically improve the performance of a particular operation, they come at the cost of greater memory utilization. When the size of the work area is smaller than desired, the response time of the request increases. Ideally this work area space is configured for nearly 100% of all single pass online transaction processing (OLTP) operations.

SORT SEGMENT REQUEST Wait Event Sorting can be measured using the SORT SEGMENT REQUEST wait event. The wait event suggests that a user process tried to acquire a sort segment and had to wait until that segment was obtained. As noted above, the query will attempt to perform the sort operation in memory first. If the memory allocated for the user’s sort space is insufficient, the process must acquire space in the TEMP tablespace before performing its sort on disk. This space is called a sort segment. Although the latency for performing a SORT SEGMENT REQUEST is small (often in the milliseconds), every opportunity for latency to be applied to the service time of a request adds up. Therefore, it is important to measure the effects of sorting on a regular basis. For more information, see http://www.dbazine.com/blogs/blog-cf/chrisfoot/blogentry.2006-0414.0082403220.


Page 52

Configuring the REDO Tablespace The REDO tablespace is used as a temporary storage location prior to Oracle performing changes at the data file layer. Enable the REDO advisor using the FAST_START_MTTR initialization parameter. Checking messages in the alert log is an easy way to determine how fast Oracle is filling and switching logs. The specific message to watching for is "Checkpoint not complete." To improve performance, increase the REDO tablespace size so that log switching can occur at Oracle’s recommended interval of 15 to 30 minutes. To do so, identify the current size of the REDO log members from V$LOG, record the number of log switches per hour, and then increase the size of the log to allow Oracle to switch at the recommended rate of one switch per 15 to 30 minutes.

Configuring the UNDO Tablespace The UNDO tablespace is used for read consistency, data recovery, and rollback. A database running in automatic UNDO management mode transparently creates and manages UNDO segments. Oracle recommends using automatic undo management because it simplifies operational management and removes the need for manual sizing of UNDO using one or more rollback segment. Use the UNDO advisor to properly size the UNDO space. Be aware of the state and size of the UNDO tablespace is important when trying to eliminate the ORA-1555: snapshot too old messages that identify a poorly sized UNDO space from the Oracle alert log. For more information, see http://www.oracle.com/technology/oramag/oracle/05-jul/o45tuning.html.


Page 53

Setting Oracle Initialization Parameters Oracle initialization parameters influence the performance characteristics of a running instance of Oracle. In general, a very simple initialization file with few settings is appropriate in most situations, and the initialization file should not be the main focus for performance tuning. Performance improvement projects should involve the analysis of under-performing SQL and the causes of latency, which could be related to physical database design, stale database statistics, system resource saturation, or poorly configured instance configuration.

Understanding the Importance of Optimizer Statistics Optimizer statistics provide detail about a database and its objects and can be used to select the best execution plan for each SQL statement. Physical and logical system statistics also can be maintained. The combination of these statistics, whether managed automatically or manually, can directly affects the performance or service time of a query. The following table lists the initialization parameters that directly affect the way statistics are used and subsequently how they shape the decision of the Optimizer. Note

These settings are recommendations and should be tested out over time. Oracle has put together a very informative guide about achieving the best performance from the Optimizer. For more information, see http://www.oracle.com/technology/products/ bi/db/10g/pdf/twp_bidw_optimizer_10gr2_0208.pdf.

Parameter Name

Description

Recommended Setting

OPTIMIZER_MODE

Mechanism for result set retrieval As of 10g R2, only a subset of values are available: ALL_ROWS, FIRST_ROWS_1, FIRST_ROWS_10, FIRST_ROWS_100, and FIRST_ROWS_1000

OPTIMIZER_DYNAMIC_ SAMPLING

Controls the level of dynamic sampling performed by the Optimizer This parameter can introduce overhead because the setting forces the CBO to perform read-ahead scans of tables to capture more accurate estimates. Set this parameter to 0 (no sampling) to 10. By default, this parameter is set to 2, meaning that the Optimizer will dynamically sample all unanalyzed tables. The number of blocks sampled is two times the default number of dynamic sampling blocks.

1 or 2

OPTIMIZER_FEATURES_ ENABLE

Umbrella parameter for enabling a series of Optimizer features based on an Oracle release number

Do not change


ALL_ROWS

Page 54

Parameter Name

Description

Recommended Setting

OPTIMIZER_INDEX_ CACHING

Adjusts the behavior of cost-based optimization to favor nested loops joins and IN-list iterators Setting this parameter to a higher value makes nested loops joins and IN-list iterators look less expensive to the Optimizer. As a result, it will be more likely to pick nested loop joins over hash or sort-merge joins and to pick indexes using IN-list iterators over other indexes or full table scans. By default, this parameter is set to 0.

80–100

OPTIMIZER_INDEX_ COST_ADJ

Tunes the Optimizer behavior for access path selection to be more or less index friendly— that is, makes the Optimizer more or less prone to selecting an index access path over a full table scan

10–30


Page 55

Using the Oracle Wait Interface The Oracle Wait Interface provides insight and quantification to the causes of latency experienced when executing SQL. The 10046 trace event is the primary mechanism for studying wait events at a statement level. The Blackboard Performance Engineering team uses 10046 tracing to investigate the causes of latency combined with 10053 tracing to understand the characteristics of the Oracle Cost-Based Optimizer. For more information, see the article Exploring the Oracle Database 10g Release 1 Wait Interface (http://www.oracle.com/technology/pub/articles/schumacher_10gwait.html). The following two books are excellent resources for learning about wait events and 10046 tracing: •

Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning (http://www.amazon.com/Oracle-Wait-Interface-Performance-diagnostics/dp/ 007222729X)

•

Optimizing Oracle Performance (http://www.amazon.com/Optimizing-Oracle-Performance-Cary-Millsap/dp/059600527X)


Page 56

Appendix A: bb-config.properties File To edit the bb-config.properties file 1. Open the bb-config.properties file in the following directory: •

UNIX: %/usr/local//config

•

Windows: C:\\config

2. Edit the parameters using the information provided in the appropriate section in this document or in aggregated table below. 3. Save and close the file. 4. Update the configuration by running the PushConfigUpdates command: •

UNIX: %/usr/local//tools/admin/ PushConfigUpdates.sh

•

Windows: C:\\tools\admin\PushConfigUpdates.bat

Optimization Parameters in bb-config.properties Parameter Name

Description

Recommended Setting

Apache HTTP Server bbconfig.unix.httpd. minspareservers

Dynamic mechanism for keeping a minimum number spare of servers available to handle simultaneous requests from a browser

5

bbconfig.unix.httpd. maxspareservers

Dynamic mechanism for keeping a maximum number of spare servers available to handle simultaneous requests from a browser

50

bbconfig.unix.httpd. startservers

Number of servers to start initially

10


Page 57

Recommended Setting

Parameter Name

Description

bbconfig.unix.httpd. maxclients

Maximum number of servers allowed to be running This limits the number of clients that can connect at the same time. Blackboard Learn is compiled to support as many as 5,000 clients. To avoid keepalive issues, this parameter can be set to a value higher than bbconfig.appserver.maxthreads. For more information, see Configuring Database Connections. Note For Tomcat cluster configurations, Apache HTTP Server acts as the main load balancing component. As more instances of Tomcat are deployed, increase this value for each instance in the cluster.

400-1024

bbconfig.unix.httpd. maxrequestsperchild

Number of requests each child process is allowed to process before the child dies Note Setting this parameter to a higher value reduces delays that are caused by spawning new Apache HTTPD child processes. Prolonged use is not an issue for Blackboard Learn.

1024-2048

bbconfig.unix.httpd. keepalivetimeout

Number of seconds to wait for the next request from the same client on the same connection

15

bbconfig.unix.httpd. keepalive

Whether to allow persistent connections (more than one request per connection)

On

bbconfig.unix.httpd. MaxKeepAliveRequests

Maximum number of requests to allow during a persistent connection

100

bbconfig.webserver.c ompression

Enables HTTP compression via GZip

Yes

Microsoft IIS 1

bbconfig.webserver. keepalive

Whether to allow persistent connections (more than one request per connection) Always make sure to review the setting with your hardware load balancer vendor.

bbconfig.webserver. keepalivetimeout

Number of seconds to wait for the next request from the same client on the same connection

15

bbconfig.webserver.c ompression

Enables HTTP compression via GZip

Yes


Page 58

Parameter Name

Description

Recommended Setting

Apache Tomcat Database Connections bbconfig.appserver. minthreads

Minimum number of threads created for Tomcat

bbconfig.appserver. maxthreads

Maximum number of threads allocated to Tomcat for 32-bit 1.4 GB RAM in the heapspace This setting limits the number of requests that can be simultaneously processed by Tomcat. This number should match the bbconfig.unix.httpd.maxclients setting for Apache on UNIX platforms. For more information, see Sizing the Maximum Thread Count.

200-250

bbconfig.appserver. maxthreads

Maximum number of threads allocated to Tomcat for 64-bit 4 GB RAM in the heapspace This setting limits the number of requests that can be simultaneously processed by Tomcat. This number should match the bbconfig.unix.httpd.maxclients setting for Apache on UNIX platforms. For more information, see Sizing the Maximum Thread Count.

400-800

bbconfig.min.heap size.tomcat

Minimum heap size allocated to the Tomcat JVM

1/2 of or equal to max.heap size.tomcat

bbconfig.max.heap size.tomcat

Maximum heap size allocated to the Tomcat JVM for 32-bit configurations

1024-1400m

bbconfig.max.perm size.tomcat

Maximum perm space allocated to the Tomcat JVM

bbconfig.max.stack size.tomcat

Stack size for each thread in the Tomcat JVM

bbconfig.jvm.options .extra.tomcat

Additional settings used to configure the JVM See Using Non-Standard (-XX) JVM Arguments.


100

256m 170-300k User defined

Page 59

Recommended Setting

Parameter Name

Description

bbconfig.database.bb learn.db.maxpoolsize

Maximum number of connections waiting to serve database requests The BBLEARN (legacy: BB_BB60) connection pool is the primary database connection pool interface for the Blackboard schema in Tomcat. Blackboard recommends that this value be sized conservatively based on need and usage. Start with a maximum value of 100, and then increase the maximum size as the Tomcat application threads increase. Optimally, this value should be equal to, but not greater than, the bbconfig.appserver.maxthreads parameter setting. Note JDBC connection consumption is a symptom of a performance issue or degradation. Increasing the JDBC connection count will not always solve a performance issue. Performance issues should be diagnosed with care and accuracy and identify bottlenecks at their source.

bbconfig.database.bb learn.db.minpoolsize

Number of connections waiting to serve database requests

10

bbconfig.database.bb learn.db.mssql. initpoolsize

Initial number of connections waiting to serve database requests for SQL Server

10

bbconfig.database.bb learn.db.timeout

Time window in which database transactions must be processed before a timeout occurs

60-300

300-1024

Recommended JVM Tuning Set Each release of Blackboard Learn has its own recommended tuning set. For more information, see the Release Notes for your particular release on Behind the Blackboard (https://behind.blackboard.com).


Page 60

Appendix B: Related Resources Apache HTTP Server for UNIX •

General Information about Apache Server (http://www.apache.org)

•

General Documentation About Apache Server 1.3 (http://httpd.apache.org/docs/1.3)

•

Performance Notes for Apache Server 1.3 (http://httpd.apache.org/docs/1.3/misc/perftuning.html)

•

Apache Server Status (http://httpd.apache.org/docs/1.3/mod/mod_status.html)

•

Apache Bench (http://httpd.apache.org/docs/1.3/programs/ab.html)

•

Quest Foglight (http://www.quest.com/foglight)

Microsoft IIS for Windows •

General Documentation about IIS (http://www.iis.net)

•

IIS 6.0 Documentation (http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/848968f3 -baa0-46f9-b1e6-ef81dd09b015.mspx)

•

IIS 7.0 Documentation (http://technet.microsoft.com/en-us/library/cc753734(WS.10).aspx)

•

Optimizing Performance (http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/e621190 d-1015-40c2-a5ec-0dcb32c98286.mspx)

•

Controlling Memory Usage (http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/ad56540c -2323-4316-b981-7ebb70352baa.mspx)

•

Monitoring Threads in IIS (http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/a63ee1c2 -04d6-44dc-b4d6-678eb3117bf9.mspx)

•

Monitoring with Performance Counters (http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/b6fd16ee -1192-4b8b-9d02-af307879fdb7.mspx?mfr=true)

Apache Tomcat Application Container •

Java Performance Documentation (http://java.sun.com/docs/performance)

•

Java Performance Tuning (http://www.javaperformancetuning.com)

•

J2SE 5.0 Troubleshooting and Diagnostic Guide (http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf)

•

Analyzing Stack Traces (http://java.sun.com/developer/onlineTraining/Programming/JDCBook/stack.html)

•

Profiling Java Using JProbe (http://www.quest.com/jprobe)


Page 61

•

A Brief History of Garbage Collection (http://www.ibm.com/developerworks/java/library/j-jtp10283)

•

Garbage Collection Ergonomics Overview (http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html)

•

Java Garbage Collection Tuning (http://www.folgmann.com/en/j2ee/gc.html)

•

Tuning Garbage Collection in the JVM (http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html)

•

Shortcuts to Tuning Garbage Collection (http://www.petefreitag.com/articles/gctuning)

•

Diagnosing a Garbage Collection Problem (http://java.sun.com/docs/hotspot/gc1.4.2/example.html)

•

Java HotSpot Garbage Collection (http://java.sun.com/javase/technologies/hotspot/gc/index.jsp)

•

Java Hotspot VM Options (http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp)

Microsoft SQL Server •

Microsoft SQL Server 2005 (http://www.microsoft.com/sqlserver/2005/en/us/default.aspx)

•

Guidance provided by Microsoft (http://msdn.microsoft.com/en-us/library/ms143719(SQL.90).aspx)

•

SQL Server I/O Basics (http://www.microsoft.com/technet/prodtechnol/sql/2005/iobasics.mspx)

•

Administering the Database Engine (http://msdn.microsoft.com/en-us/library/ms191470(SQL.90).aspx)

•

Microsoft Pre-deployment I/O Practices (http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/pdpliobp.mspx)

•

How to Configure SQL Server to Use More than 2 GB (http://support.microsoft.com/kb/274750)

•

How to Determine Proper SQL Server Configuration Settings (http://support.microsoft.com/kb/319942)

•

Slava Oks’s WebLog: “Be Aware: Using AWE, locked pages in memory, on 64 bit” (http://blogs.msdn.com/slavao/archive/2005/04/29/413425.aspx)

•

The Perils of Fiber Mode (http://msdn.microsoft.com/en-us/library/aa175385.aspx)

•

Craig Freedman's SQL Server Blog: “Read Committed Isolation Level” (http://blogs.msdn.com/craigfr/archive/2007/04/25/read-committed-isolation-level.aspx)

•

SET TRANSACTION ISOLATION LEVEL (Transact-SQL) (http://msdn.microsoft.com/en-us/library/ms173763.aspx)

•

Configuring SQL Server 2005 TempDB for Performance (http://searchsqlserver.techtarget.com/tip/0,289483,sid87_gci1180698,00.html)

•

Optimizing TEMPDB Performance (http://technet.microsoft.com/en-us/library/ms175527(SQL.90).aspx)

•

Flushing of the Procedure Cache (http://www.sql-server-performance.com/articles/per/data_cache_p1.aspx)


Page 62

•

Concurrency Enhancements for the TEMPDB Database (http://support.microsoft.com/kb/328551)

•

Working with TEMPDB in SQL Server 2005 (http://www.microsoft.com/technet/prodtechnol/sql/2005/workingwithtempdb.mspx)

•

Managing Extent Allocations and Free Space (http://msdn.microsoft.com/en-us/library/ms175195.aspx)

•

Tony Rogerson's Blog: "Put TEMPDB on a RAM Drive" (http://sqlblogcasts.com/blogs/tonyrogerson/archive/2006/08/24/958.aspx)

•

Download SQL Server 2005 Performance Dashboard Reports (http://www.microsoft.com/downloads/details.aspx?FamilyId=1d3a4a0d-7e0c-4730-8204e419218c1efc&displaylang=en)

•

Black Belt Administration: Performance Dashboard for Microsoft SQL Server, Part I (http://www.databasejournal.com/features/mssql/article.php/10894_3665281_2)

•

Black Belt Administration: Performance Dashboard for Microsoft SQL Server, Part II (http://www.databasejournal.com/features/mssql/article.php/10894_3671736)

•

Making the Most Out of the SQL Server 2005 Performance Dashboard (http://www.sql-serverperformance.com/articles/per/bm_performance_dashboard_2005_p1.aspx)

•

SQL Server 2005 Waits and Queues (http://technet.microsoft.com/en-us/library/cc966413.aspx)

•

Troubleshooting SQL Server Performance (http://www.microsoft.com/technet/prodtechnol/sql/2005/tsprfprb.mspx)

Oracle Database •

Oracle Database 10g R2 Performance Tuning Guide (http://download.oracle.com/docs/cd/B19306_01/server.102/ b14211/toc.htm)

•

Network Appliance Performance Report: Oracle10g RAC on Linux (http://media.netapp.com/documents/tr-3423.pdf)

•

Oracle Database Manageability (http://www.oracle.com/technology/products/manageability/database/index10g.html)

•

Sizing the Program Global Area (http://www.scribd.com/doc/6469805/How-to-Determineif-PGA-Size-is-Set-Properly)

•

SORT SEGMENT REQUEST Wait Event (http://www.dbazine.com/blogs/blog-cf/chrisfoot/blogentry.2006-04-14.0082403220)

•

Tuning UNDO Tablespace (http://www.oracle.com/technology/oramag/oracle/05-jul/o45tuning.html)

•

Upgarding from Oracle Database 9i to 10g: What to expect from the Optimizer (http://www.oracle.com/technology/products/bi/db/10g/pdf/twp_bidw_optimizer_10gr2_02 08.pdf)

•

Exploring the Oracle Database 10g Release 1 Wait Interface (http://www.oracle.com/technology/pub/articles/schumacher_10gwait.html)


Page 63

•

Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning (http://www.amazon.com/Oracle-Wait-Interface-PerformanceDiagnostics/dp/007222729X)

•

Optimizing Oracle Performance (http://www.amazon.com/Optimizing-Oracle-Performance-Cary-Millsap/dp/059600527X)


Page 64

Blackboard Learn Performance Optimization Guide - Behind the ...

Blackboard Learn Performance Optimization Guide - Behind the ...

Suggest Documents

Blackboard Learn Performance Optimization Guide - Behind the ...

Mobile User's Guide - Behind the Blackboard

Blackboard Transact Client Support Guide - Behind the Blackboard!

Mobile User's Guide - Behind the Blackboard

Blackboard Learn

Blackboard Learn

Blackboard IM Quick Guide - Getting Started - Blackboard Learn

Release Notes - Behind the Blackboard

PDF/ePUB Behind the Blackboard - Google Sites

Blackboard Learn Course Conversion Tool

A2 Level 2 - Blackboard Learn

Blackboard Learn Hardware Sizing Guide for Dell Deployments

QUICK START GUIDE TO USING BLACKBOARD LEARN 9.1 ...

Blackboard Learn Hardware Sizing Guide for Dell Deployments

Blackboard Collaborate IM User's Guide - Blackboard Help

Blackboard Learn 9.1 User Guide 4.2 (PDF) - Wimba

Import Blackboard 9 Chapter Test Banks into Blackboard Learn ...

Blackboard Collaborate IM User's Guide - Blackboard Help

What's in Mobile Learn 18DEC2010 - Blackboard

Download Environment the Science Behind the Stories: Blackboard ...

[PDF] Behind the Blackboard: Selections on the ... - Google Sites

Blackboard Learn 9.1 - Help for Instructors

Blackboard Learn Text Widget - Regroup Mass Notification

4.30pm HAMILTON CENTRE - Blackboard Learn - Brunel University