10 Questions to Ask Before Buyingan Optimization Solution - FICO

»» insights

10 Questions to Ask Before Buying an Optimization Solution Learn how you can avoid common pitfalls and gain up to 20% profit improvement

Number 47—December 2010

The adage “choice is the enemy of decision” certainly holds true for optimization solutions. As banks move towards optimizing customer decisions, the many choices available make it difficult to know what works best. While some optimization solutions are purely software-based “solvers,” others offer more, such as modeling services or domain expertise within a given decision area. Moreover, choice is not the only challenge. Historically, many solutions are highly academic and not built to meet real-world requirements. So, what are the right criteria to evaluate an optimization solution?

Know how to evaluate the optimization solution that works best for your business. Banks that follow these guidelines typically see 5%–20% profit improvement.

This paper outlines the 10 questions you should ask to help you understand the requirements—and avoid the pitfalls—of decision optimization. These reflect lessons learned from FICO’s extensive work with clients successfully using optimization today.

www.fico.com

Make every decision countTM

10 Questions to Ask Before Buying an Optimization Solution

»» insights

»» 10 Essential Questions

With so many optimization solutions in the market today, how do banking institutions evaluate which one will work best for their business? Start by asking these 10 questions, which explore the criteria needed to evaluate alternative solutions and make a determination.

When Evaluating an Optimization Solution

1. How does the solution incorporate the sensitivities of consumers into various actions? In order to optimize the actions that you take on your customers and prospects, you first need to understand the reactions of each consumer to the possible treatments you can take. Optimizing based on incorrect assumptions about these “action-effect” dynamics will render the optimization ineffective (at best) or grossly incorrect (at worst). Incorrect models of consumer behavior—whether on the dimension of response to a product offer, risk from a line increase or revenue from a pricing action—lead to incorrect optimization results. In other words, “garbage in equals garbage out.” Consider Figure 1, which shows three curves representing the relationship between expected profit and different levels of loan price for a particular consumer. The orange curve represents the “true” relationship between profit and different levels of loan price. Note that a consumer’s price sensitivity is typically not known a priori, for several reasons— for instance, you may not have tried every possible combination of price on this consumer, actions other than price are taking place simultaneously or the environment is changing. This true relationship becomes apparent only after the actions are taken, and balances the likelihood of taking up the loan with the potential revenue and loss. The red line represents a “good” estimate, since it estimates the true relationship closely. The green line shows an incorrect estimate that is a bit off, perhaps due to data biases.

Figure 1: Incorrect optimization results have significant impact on profit 105

Good Estimation

95 EXPECTED PROFIT ($)

Optimal Actions

True Profitability

100

Incorrect Estimation

90 85 80

$4 Difference in Profit

75 70 65 60 6.49

6.99

7.49

7.99

8.49

8.99

9.49

9.99

10.49

10.99

11.49

11.99

12.49

12.99

13.49

LOAN PRICE (%)

www.fico.com

page 2


»» insights Without constraints, the optimal action is determined by the top point on the curve. If a lender took action based on the incorrect estimate, it would offer this consumer a loan price of 9.49%, while an action based on the good estimate would offer 10.49%. Looking at these points on the true profit curve, the good estimate would increase profit by about $4 per account. Multiplied over hundreds of thousands or millions of accounts, the effect of the error on profit would be significant. Action-effect models are typically more complex than the examples above, since they are influenced by multiple factors. The modeling techniques required to accurately understand consumers’ sensitivities are very different from traditional scorecard approaches. Ensure that your vendor is fluent in decision modeling techniques, can validate its assumptions and approach and can deal with limitations in data.

2. Does the optimization solution address data limitations? Getting the action-effect relationship right depends on both data and business expertise. Data will never be perfect, and any vendor who claims you must invest 6–12 months in running experimental designs to gather perfect data is mistaken. Leveraging your existing data is important and can tell you a lot about your customers and their response dynamics. While utilizing your data assets is important, ensure that the solution does not blindly rely on historical data, but instead has procedures in place to tease out casual relationship despite biases in the data. The ability to identify and address these holes and/or biases in data is an important component of your solution, since these can lead to incorrect assumptions about consumer behavior as illustrated in Figure 1. In the rare cases where no data is available, there are still ways to move forward today, including: 1) developing expert models by encoding business expertise into the assumptions and relationships and 2) performing smart, limited testing to quickly gather the data to inform your models. In cases where you need to rely heavily on expertise, more stress-testing may be required, which will be discussed further in question 7. Choose an optimization solution that explicitly identifies and addresses data limitations and can incorporate business expertise.

3. Can you leverage your existing analytic assets within the solution? You have invested in gathering data, building predictive models and perhaps even building action-effect models to drive your optimization solution. The optimization framework that you choose should allow you to incorporate these models in a way that you’re confident that all relevant customer data is utilized to make the best decision. Your data elements and models can be used as inputs or decision keys, and in many cases, can be used directly in the underlying action-effect models themselves.

4. Does the solution provide insight into your key business trade-offs to facilitate the selection of an optimal operating point? Your business will need to optimize decisions subject to many constraints. These range from policy constraints, such as who you lend to or what offers certain customers are eligible for, to portfolio-level goals, such as “reduce losses by 5%” or “ensure marketing budget is less than $10 million.” While some optimization solutions focus on finding the “one” solution to a problem you specify, it is always important to understand the impact business constraints have on your bottom line and your strategy. Exploring these trade-offs helps to properly set those constraints.

www.fico.com

page 3


»» insights

Figure 2: Efficient frontier adds critical insight into key business trade-offs Balancing profit and market share trade-off 100,000

Current operating strategies Efficient frontier/optimized strategies

C

D

80,000

PROFIT ($)

60,000

B

E

A

40,000

20,000

F

0 0

10

20

30

40

50

60

70

FUNDED RATE (%)

A key tool for quantifying these trade-offs is an “efficient frontier,” as seen in Figure 2. For example, if you want to increase both profitability and market share, you may look at the rate at which new loans are “funded” (approved and opened). If today you are operating at point A, you can increase both profit and market share by moving to the efficient frontier (e.g., move up and to the right to point C). If you are operating at point B, the trade-off between business goals really comes into play. You may decide to maintain your current market share and increase profit (by moving up to point D). Or you may decide to maintain or even sacrifice profit to gain additional market share (by moving to points E or F). Similar trade-offs can be made between profit and loss, volume, attrition, exposure or any other business metrics of interest. A trusted advisor with relevant business expertise can help you identify the optimal operating point. Make sure your optimization vendor can provide someone you can work with to identify options and make a choice on strategies to deploy.

www.fico.com

page 4


»» insights 5. Does the solution allow you to establish and evaluate a baseline of “business as usual” or what you’re doing today? There are many reasons why you should be able to measure your current strategies in the context of the optimization solution. First and foremost, it provides a Return on Investment (ROI) estimate of the optimized solution. It is important for this estimate of lift to be made within the optimization solution to ensure an “apples-to-apples” comparison. It also provides a sanity check that your decision model is able to project the results of the historical strategy. Another reason is to tie the various outcomes and constraints to what you’re experiencing today. For instance, “reduce attrition by 10%” will only be meaningful if you tie it to the current outcome. In some cases, the “business as usual” baseline is best represented directly from the data. These historical actions can be passed directly through the simulation. This option tends to work best when multiple strategies (e.g., champion/challenger) are being used. In other cases, it is helpful to represent your baseline strategy as a set of rules or a decision tree, particularly when this strategy is not represented in the data. A good optimization solution will provide both options for simulating your current strategy or even a strategy you are considering.

6. Does the solution provide a mechanism for business users to verify that the optimized solution is valid? Optimization results are only as good as the assumptions and inputs on which they were built. As such, it is imperative to evaluate the optimization results from both a technical and business point of view. A mechanism to investigate optimization results allows the business user to impute business judgment into a process that is otherwise a pure mathematical exercise.

Figure 3: Optimization results must make business sense Example 1: Overly optimistic profiles Customer Profile (average) Credit Bureau Utilization

Reject

Accept Accept (low line of credit) (high line of credit)

5%

42%

7%

Credit Bureau Balance

$1,800

$4,100

$3,505

Debt-to-Income Ratio

65%

58%

71%

FICO® Score

680

705

690

Revenue Score

140

160

172

58

74

67

Length of Credit History (months) Example 2: Reasonable profiles Customer Profile (average) Credit Bureau Utilization

Reject

Accept Accept (low line of credit) (high line of credit)

5%

33%

43%

Credit Bureau Balance

$637

$4,305

$6,368

Debt-to-Income Ratio

78%

47%

23%

FICO® Score

655

710

744

Revenue Score

140

176

190

58

74

106

Length of Credit History (months)

Let us look at an example for offering a new line of credit. Figure 3 compares customer profiles generated by an optimization solution: Customers who are rejected, customers who are accepted with a low line of credit, and customers who are accepted with a high line of credit. The goal is to see whether the more favorable treatments will generate the profit improvements that the solution projects. Business experience suggests that Example 1 may be too optimistic. For instance, the “accept with high line of credit” group is still fairly risky (judging by FICO® Score and debt-to-income ratio) and their credit usage is relatively low (37% utilization). It seems unlikely that the solution would generate many gains from this treatment. In Example 2, the solution identifies less risky accounts with higher credit needs for the more favorable treatment. Therefore, it is more likely to generate profit.

www.fico.com

page 5


»» insights The ultimate question is: Can the business user justify the treatments based on his/her business knowledge? In other words, is the optimization driven by unreasonable modeling assumptions? To help the business user validate the reasonableness of the approach, the optimization solution should have reporting functionality that provides visibility and hands-on review. “Swap set” reports can also be useful to compare different optimized strategies, or an optimized strategy to what you’re doing today. Ideally, the business user could create his/her own diagnostic reports in the tool and drill down, where necessary, to get comfortable with the optimization.

7. Can the optimization solution help you better prepare for potential changes in your environment? Despite investing in good data, software and expertise, unexpected events can happen to change the business world. The economy changes, the competitive environment changes, the way consumers react changes. This means the outcomes expected from the strategy—profit, revenue, loss, market share, to name a few—will also change. Ideally, the optimization solution captures the interaction between various macro-economic conditions and underlying model estimates—at a minimum, for the risk models. When done, you will be able to evaluate the impact of various exogenous effects. This will help in choosing strategies that will fare well under a variety of market conditions. This is not only important from a regulatory perspective (as required by the Basel Accords), but it also makes good business sense. Thus, an optimization solution must provide a facility to perform “what-if” scenario analysis that goes beyond simply changing your constraints. It should be able to simulate the impact of changes, such as external factors (e.g., the economy) or consumer behavior (e.g., price sensitivities or risk levels). The best solutions will allow you to: 1) Automate stress-testing to perform many scenario investigations at once, 2) Assume that changes may apply differently to different segments of your population and 3) Provide a range of insightful diagnostics and support to inform your choice of optimal strategy, in light of what the future may hold.

8. Can the optimization framework be easily adopted for new decision areas across the enterprise? Some optimization frameworks require the user to formulate the mathematical expressions corresponding to the optimization problem. While applications made for a mathematically oriented “super user” allow flexibility, the trade-offs are that it can be more difficult to use, further from the business problem and often very error-prone. Using this methodology, it is much harder to validate mathematical expressions than to validate simple business logic. In addition, it is much more difficult for the business user to understand and use the framework and may require specialized training or additional resources. Conversely, if the framework is too restrictive, you will not be able to leverage it in other applications and decision areas, which is a common goal for many building an optimization practice. For example, you might want to optimize your marketing decisions after you tackle your line increase decisions. Therefore, it is critical to balance flexibility within skill set requirements when choosing an optimization solution, and have a tool that both meets analytic requirements and communicates to the business user. This will allow your organization to scale your optimization efforts from a common methodology and platform, and develop a repeatable process that can grow as your business grows.

www.fico.com

page 6


»» insights 9. Can the optimization solution be readily deployed in your existing decision-making platform? After choosing the optimal scenario to be deployed for your portfolio, the next step is to bring those optimal actions or rules to your current decision execution platform. In financial services and elsewhere, this is most often in the form of a strategy tree that can assign treatments to current and future customers. However, in certain circumstances, it may mean calling the optimization from within an application to optimize realtime. If you are deploying to a rules management system, it is highly beneficial if the output of your optimization can be directly accepted as an input into your decision-making platform, without the need to manually recode. This eliminates the errors that often happen when trying to manually enter the optimized strategy, that can be much more complex than a typical strategy. The key to deployment is flexibility and the ability to translate optimal actions to be used by your system.

10. Does the optimization solution have sophisticated optimization software designed to solve business problems of financial services? Optimizing decisions on millions of customers with multiple business constraints—as is done in financial services—requires an optimization algorithm that can find optimal solutions efficiently. Your optimization algorithm must solve for the best (i.e., true optimum) solution and exploit the structure of your decision problem to do so efficiently. Questions to ask your vendor include: • Is your algorithm solving for the true customer-level optimum, or merely approximating it? • How quickly can you solve problems with millions of customers and dozens of constraints? • How much do you invest in Research and Development and improvement of the software and algorithms? • Do you have any performance benchmarks for your optimization technology solving challenging optimization problems?

Figure 4: Capabilities needed for best-in-class optimization Da t a

Design

Model

Optimize

Simulate

Deploy

Identify data sources

Identify objectives and constraints

Build out models

Program constraints

Generate financial projections for multiple what-if scenarios

Build strategy tree and deploy as rules or code

Gather

Identify available and required models

Import and manage models

Run scenarios

Drill down to understand scenario trade-offs

Import

Design decision model

Diagnose decision model

Generate efficient frontiers

Understand drivers of optimal action

Integrate with FICO tools, such as FICOTM Blaze Advisor® system or FICO® TRIAD® Customer Manager

Improving customer decisions and portfolio profitability requires an optimization solution that includes more than just solving the optimization problem. This chart illustrates the full capabilities required for an effective decision optimization methodology, from data management and decision modeling through simulation and deployment.

Many optimization software vendors try to circumvent the problem by optimizing at the segment level or using an approximate search algorithm—relaxation of the optimization problem, as it is known in academia. The best optimization algorithms can exploit the structure of a particular decision problem, while still solving for the true, rather than approximate, solution. In financial services, it is reasonable to assume a problem size of millions of consumers or accounts, with dozens of business constraints (in practice, 2–5 portfolio constraints are most typically applied), and dozens to hundreds of possible treatments. Depending on your computing power and number of constraints, this size problem should take a few minutes to a few hours to solve. Finally, choose a vendor that continually invests in its optimization algorithm to improve performance and efficiency.

www.fico.com

page 7


»» insights

»» High-Value Optimization

Evaluating an optimization solution using the guidelines above can help you avoid potential pitfalls and ensure that your solution delivers true business value. These questions were crafted using best practices from FICO’s more than 100 optimization projects across different countries, portfolios and applications. Recent client results include: • 12% profit improvement in credit line management, while controlling losses. • 45% profit improvement among new loan applicants by improving pricing and loan amount decisions, while meeting regulatory requirements. • $10 increased profit per eligible customer through better targeting of marketing offers within the same marketing budget and risk profile. Banks that are most successful with decision optimization combine sound methodology, deep domain expertise and software designed to address the mathematics and business implications of optimization. Depending on the constraints imposed, these companies typically see 5%–20% profit improvement..

Figure 5: FICO optimization portfolio Create a specific decision model using FICO experts

Best-in-class analytic services METHODOLOGY, SERVICES AND TOOLS

FICO™ Custom Decision Optimization Decision Modeling • Exploration • Stress Testing • Pre-Configured Applications

Pre-developed solutions for quick adoption Optimal strategy design software

TOOLS

Optimization tools for operational research experts Optimal strategy deployment

Solve your business problem with a tailored solution

Frameworks and bundled solutions Product Offers • Pricing • Line Management • Loan Originations • “Starter Kits” Develop your own optimized customer strategies

FICO™ Decision Optimizer Optimization Workflow • Simulation • Exploration • Reporting Solve a wide range of optimization problems

FICO™ Xpress Optimization Suite Solvers • Modeling and Programming • Visual Development Environment Automate and deploy optimal decisions

FICO™ Blaze Advisor® and FICO applications Rules Management • Decision Simulation

FICO provides a wide range of optimization offerings, from tools to methodologies and services. Contact us for more information.

The Insights white paper series provides briefings on research findings and product development directions from FICO. To subscribe, go to www.fico.com/insights. For more information

US toll-free +1 888 342 6336

International +44 (0) 207 940 8718

email [email protected]

web www.fico.com

FICO, Blaze Advisor, Triad and “Make every decision count” are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners. © 2010 Fair Isaac Corporation. All rights reserved. 2743WP 12/10 PDF