Applied Computational Economics

Applied Computational Economics Mario J. Miranda The Ohio State University

and

Paul L. Fackler North Carolina State University

Contents Preface

ii

1 Introduction

1

2 Linear Equations

6

1.1 Some Apparently Simple Questions . . . . . . . . . . . . . . . 1.2 An Alternative Analytic Framework . . . . . . . . . . . . . . .

2.1 2.2 2.3 2.4 2.5 2.6

L-U Factorization . . . . Gaussian Elimination . . Rounding Error . . . . . Ill Conditioning . . . . . Special Linear Equations Iterative Methods . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

3 Nonlinear Equations 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8

. . . . . .

Bisection Method . . . . . . . . . Function Iteration . . . . . . . . . Newton's Method . . . . . . . . . Quasi-Newton Methods . . . . . . Problems With Newton Methods Choosing a Solution Method . . . Complementarity Problems . . . Complementarity Methods . . . .

4 Finite-Dimensional Optimization

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

1 4

7 10 12 13 15 17

24 26 28 29 33 38 41 43 47

55

4.1 Derivative-Free Methods . . . . . . . . . . . . . . . . . . . . . 57 4.2 Newton-Raphson Method . . . . . . . . . . . . . . . . . . . . 62 4.3 Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . 63 i

ii

CONTENTS

4.4 Line Search Methods . . . . . . . . . . . . . . . . . . . . . . . 68 4.5 Special Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.6 Constrained Optimization . . . . . . . . . . . . . . . . . . . . 73

5 Integration and Dierentiation 5.1 5.2 5.3 5.4 5.5 5.6 5.7

Newton-Cotes Methods . . . . . Gaussian Quadrature . . . . . . Monte Carlo Integration . . . . Quasi-Monte Carlo Integration . Numerical Dierentiation . . . . An Integration Toolbox . . . . . Initial Value Problems . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

6 Function Approximation 6.1 6.2 6.3 6.4 6.5 6.6 6.7

Interpolation Principles . . . . . . . . . . . . . . Polynomial Interpolation . . . . . . . . . . . . . Piecewise Polynomial Splines . . . . . . . . . . Multidimensional Interpolation . . . . . . . . . Choosing an Approximation Method . . . . . . An Approximation Toolkit . . . . . . . . . . . . Solving Functional Equations . . . . . . . . . . 6.7.1 Cournot Oligopoly . . . . . . . . . . . . 6.7.2 Function Inverses . . . . . . . . . . . . . 6.7.3 Linear First Order Dierential Equations

7 Discrete State Models

7.1 Discrete Dynamic Programming . . . . . 7.2 Economic Examples . . . . . . . . . . . . 7.2.1 Mine Management . . . . . . . . 7.2.2 Deterministic Asset Replacement 7.2.3 Stochastic Asset Replacement . . 7.2.4 Option Pricing . . . . . . . . . . 7.2.5 Job Search . . . . . . . . . . . . . 7.2.6 Optimal Irrigation . . . . . . . . 7.2.7 Bioeconomic Model . . . . . . . . 7.3 Solution Algorithms . . . . . . . . . . . . 7.4 Dynamic Simulation Analysis . . . . . . 7.5 Discrete Dynamic Programming Tools .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

84

85 88 91 93 94 102 106

119

120 123 128 136 139 142 147 147 151 153

160

161 163 163 165 166 167 168 170 171 172 175 178

iii

CONTENTS

7.6 Numerical Examples . . . . . . . . . . . 7.6.1 Mine Management . . . . . . . . 7.6.2 Deterministic Asset Replacement 7.6.3 Stochastic Asset Replacement . . 7.6.4 Option Pricing . . . . . . . . . . 7.6.5 Job Search . . . . . . . . . . . . . 7.6.6 Optimal Irrigation . . . . . . . . 7.6.7 Bioeconomic Model . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

8 Continuous State Models: Theory 8.1 8.2 8.3 8.4

Continuous State Dynamic Programming . . . . Euler Equilibrium Conditions . . . . . . . . . . Linear-Quadratic Control . . . . . . . . . . . . . Economic Examples . . . . . . . . . . . . . . . . 8.4.1 Asset Replacement . . . . . . . . . . . . 8.4.2 Industry Entry and Exit . . . . . . . . . 8.4.3 Option Pricing . . . . . . . . . . . . . . 8.4.4 Optimal Growth . . . . . . . . . . . . . 8.4.5 Renewable Resource Problem . . . . . . 8.4.6 Nonrenewable Resource Problem . . . . 8.4.7 Feedstock Problem . . . . . . . . . . . . 8.4.8 A Production-Adjustment Problem . . . 8.4.9 A Production-Inventory Problem . . . . 8.4.10 Optimal Growth with Debt . . . . . . . 8.5 Rational Expectations Models . . . . . . . . . . 8.5.1 Lucas-Prescott Asset Pricing Model . . . 8.5.2 Competitive Storage Under Uncertainty 8.6 Dynamic Games . . . . . . . . . . . . . . . . . . 8.6.1 Risk Sharing Game . . . . . . . . . . . . 8.6.2 Marketing Board Game . . . . . . . . .

9 Continuous State Models: Methods 9.1 9.2 9.3 9.4

Traditional Solution Methods . . . . . Bellman Equation Collocation Methods Euler Equation Collocation Methods . Dynamic Programming Examples . . . 9.4.1 Optimal Stopping . . . . . . . . 9.4.2 Stochastic Optimal Growth . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

181 181 183 186 189 191 194 196

206

207 211 214 216 216 217 218 219 221 223 224 226 227 229 232 233 234 237 239 241

253

255 257 263 268 268 270

iv

CONTENTS

9.4.3 Renewable Resource Problem . . . 9.4.4 Nonrenewable Resource Problem . 9.5 Rational Expectation Collocation Methods 9.5.1 Example: Asset Pricing Model . . . 9.5.2 Example: Commodity Storage . . . 9.6 Comparison of Solution Methods . . . . . 9.7 Dynamic Analysis . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

10.1 Introduction . . . . . . . . . . . . . . . . . . . 10.1.1 Stochastic Models with Ito Processes . 10.1.2 The Feynman-Kac Equation . . . . . . 10.1.3 Arbitrage Based Asset Valuation . . . 10.2 Probability Distributions for Ito Processes . . 10.2.1 Transition Distributions . . . . . . . . 10.2.2 Long-Run (Steady-State) Distributions 10.3 End Notes . . . . . . . . . . . . . . . . . . . . 10.3.1 Bibliographic Notes . . . . . . . . . . . 10.3.2 References . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

10 Continuous Time Mathematics

11 Continuous Time Models: Theory

11.1 Stochastic Control . . . . . . . . . . . . . . 11.1.1 Relation to Optimal Control Theory 11.1.2 Boundary Conditions . . . . . . . . . 11.1.3 Choice of the Discount Rate . . . . . 11.1.4 Examples . . . . . . . . . . . . . . . 11.2 Free Boundary Problems . . . . . . . . . . . 11.2.1 Impulse Control . . . . . . . . . . . . 11.2.2 Barrier Control . . . . . . . . . . . . 11.2.3 Discrete State/Control Problems . . 11.2.4 Stochastic Bang-Bang Problems . . . 11.3 End Notes . . . . . . . . . . . . . . . . . . . 11.3.1 Bibliographic Notes . . . . . . . . . . 11.3.2 References . . . . . . . . . . . . . . .

12 Continuous Time Models: Methods

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

272 274 276 276 276 278 281

285

285 286 292 294 299 299 301 310 310 311

316

316 319 320 322 324 337 341 351 354 364 378 378 379

391

12.1 Partial Dierential Equations . . . . . . . . . . . . . . . . . . 392 12.1.1 Finite Dierence Methods for PDEs . . . . . . . . . . . 393

v

CONTENTS

12.1.2 Method of Lines for PDEs . . . . . . . . 12.1.3 Collocation Approaches to Solving PDEs 12.1.4 Variable Transformations . . . . . . . . . 12.2 Solving Stochastic Control Problems . . . . . . 12.2.1 Free Boundary Problems . . . . . . . . .

A Mathematical Background A.1 A.2 A.3 A.4

Normed Linear Spaces Matrix Algebra . . . . Real Analysis . . . . . Markov Chains . . . .

B Computer Programming

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

400 401 401 404 407

425

425 428 431 432

435

B.1 Computer Arithmetic . . . . . . . . . . . . . . . . . . . . . . . 435 B.2 Data Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 B.3 Programming Style . . . . . . . . . . . . . . . . . . . . . . . . 439

Preface Many interesting economic models cannot be solved analytically using the standard mathematical techniques of Algebra and Calculus. This is often true of applied economic models that attempt to capture the complexities inherent in real-world individual and institutional economic behavior. For example, to be useful in applied economic analysis, the conventional Marshallian partial static equilibrium model of supply and demand must often be generalized to allow for multiple goods, interegional trade, intertemporal storage, and government interventions such as taris, taxes, and trade quotas. In such models, the structural economic constraints are of central interest to the economist, making it undesirable, if not impossible, to \assume an internal solution" to render the model analytically tractable. Another class of interesting models that typically cannot be solved analytically are stochastic dynamic models of rational, forward-looking economic behavior. Dynamic economic models typically give rise to functional equations in which the unknown is not simply a vector in Euclidean space, but rather an entire function de ned on a continuum of points. For example, the Bellman and Euler equations that describe dynamic optima are functional equations, as often are the conditions that characterize rational expectations and arbitrage pricing market equilibria. Except in a very limited number of special cases, these functional equations lack a known closed-form solution, even though the solution can be shown theoretically to exist and to be unique. Models that lack closed-form analytical solution are not unique to economics. Analytically insoluble models are common in biological, physical, and engineering sciences. Since the introduction of the digital computer, scientists in these elds have turned increasingly to numerical computer methods to solve their models. In many cases where analytical approaches fail, numerical methods are often used to successfully compute highly accurate apvi

PREFACE

vii

proximate solutions. In recent years, the scope of numerical applications in the biological, physical, and engineering sciences has grown dramatically. In most of these disciplines, computational model building and analysis is now recognized as a legitimate subdiscipline of specialization. Numerical analysis courses have also become standard in many graduate and undergraduate curriculums in these elds. Economists, however, have not embraced numerical methods as eagerly as other scientists. Many economists have shunned numerical methods out of a belief that numerical solutions are less elegant or less general than those obtained from algebraic models. The former belief is a subjective, aesthetic judgment that is outside of scienti c discourse and beyond the scope of this book. The generality of the results obtained from numerical economic models, however, is another matter. Of course, given an economic model, it is always preferable to derive an explicit algebraic solution|provided such a solution exists. However, when essential features of an economic system being studied cannot be captured neatly in an algebraically soluble model, a choice must be made. Either essential features of the system must be ignored in order to obtain an algebraically tractable model, or numerical techniques must be applied. Too often Economists chose algebraic tractability over Economic realism. Numerical economic models are often unfairly criticized by economists on the grounds that they rest on speci c assumptions regarding functional forms and parameter values. Such criticism, however, is unwarranted when strong empirical support exists for the speci c functional form and parameter values used to specify a model. Moreover, even when there is some uncertainty about functional forms and parameters, the model may be solved under a variety of assumptions in order to assess the robustness of its implications. Although some doubt will persist as to the implications of a model outside the range of functional forms and parameter values examined, this uncertainty must be weighed against the lack of relevance of an alternative model that is algebraically soluble, but which ignores essential features of the economic system of interest. We believe that it is better to derive economic insights from a realistic numerical model of an economic system than to derive irrelevant results, however general, from an unrealistic, but tractable algebraic model. Despite the resistance placed by the economics profession as a whole, an increasing number of economists are becoming aware of the potential bene ts of numerical economic model building and analysis. This is evidenced

PREFACE

viii

by the recent introduction of journals and an economic society devoted to the sub-discipline of computational economics. The growing popularity of computational economics, however, has been impeded by the absence of adequate textbooks and computer software. The methods of numerical analysis and much of the available computer software have been largely developed for non-economic disciplines, most notably the physical, mathematical, and computer sciences. The scholarly literature can also pose substantial barriers for economists, both because of its mathematical prerequisites and because its examples are unfamiliar to economists. Many available software packages, moreover, are designed to solve problems that are speci c to the physical sciences. This book attempts to address, in a number of ways, the diÆculties typically encountered by economists attempting to learn and apply numerical methods. First, this book emphasizes practical numerical methods, not mathematical proofs, and focuses on techniques that will be directly useful to economic analysts, not those that would be useful exclusively to physical scientists. Second, the examples used in the book are drawn from a wide range of sub-specialties of economics and nance, both in macro- and microeconomics, with particular emphasis on problems in agricultural, nancial, environmental, and macro- economics. And third, we include with the textbook a library of computer utilities and demonstration programs to provide interested researchers with a starting point for their own computer models. We make no attempt to be encyclopedic in our coverage of numerical methods or potential economic applications. We have instead chosen to develop only a relatively small number of techniques that can be applied easily to a wide variety of economic problems. In some instances, we have deviated from the standard treatments of numerical methods in existing textbooks in order to present a simple consistent framework that may be readily learned and applied by economists. In many cases we have elected not to cover certain numerical techniques when we regard them to be of limited bene t to economists, relative to their complexity. Throughout the book, we try to explain our choices clearly and to give references to more advanced numerical textbooks where appropriate. The book is divided into two major sections. In the rst seven chapters, we develop basic numerical methods, including root nding, complementarity, nite-dimensional optimization, numerical integration, and function approximation methods. In these chapters, we develop appreciation for basic numerical techniques by illustrating their application to partial equilibrium

PREFACE

ix

and optimization models familiar to most economists. The last ve chapters of the book are devoted to methods for solving and estimating dynamic stochastic models in economic and nance, including dynamic programming, rational expectations, and arbitrage pricing models in discrete and continuous time. The book is aimed at both graduate students, advanced undergraduate students, and practicing economists. We have attempted to write a book that can be used both as a classroom text and for self-study. We have also attempted to make the various sections reasonably self-contained. For example, the sections on discrete time continuous state models are largely independent from those on discrete time discrete state models. Although this results in some duplication of material, we felt that this would increase the usefulness of the text by allowing readers to skip sections. Although we have attempted to keep the mathematical prerequisites for this book to a minimum, some mathematical training and insight is necessary to work with computational economic models and numerical techniques. We assume that the reader is familiar with ideas and methods of linear algebra and calculus. Appendix A provides an overview of the basic mathematics used throughout the text. Furthermore, in an attempt to make the book modular in organization, some of the mathematics used in studying speci c classes of dynamic models is developed in the text as needed. Examples include the basic theory of Markov processes, dynamic programming, and, for continuous time models, Ito stochastic calculus. One barrier to the use of numerical methods by economists is lack of access to functioning computer code. This presents an apparent dilemma to us as textbook authors, given the variety of computer languages available. On the one hand, it is useful to have working examples of code in the book and to make the code available to readers for immediate use. On the other hand, using a speci c language in the text could obscure the essence of the numerical routines for those unfamiliar with the chosen language. We believe, however, that the latter concern can be substantially mitigated by conforming to the syntax of a vector processing language. Vector processing languages are designed to facilitate numerical analysis and their syntax is often simple enough that the language is transparent and easily learned and implemented. Due to its facility of use and its wide availability on university campus computing systems, we have chosen to illustrate algorithms in the book using Matlab and have provided an extensive library of Matlab utilities and demonstration programs to assist interested readers develop their own

PREFACE

x

computational economic applications. In the future, we plan to make available these programs available in other popular languages, including Gauss and Fortran. Our ultimate goal in writing this book is to motivate a broad range of economists to use numerical methods in their work by demonstrating the essential principles underlying computational economic models across subdisciplines. It is our hope that this book will help broaden the scope of economic analysis by helping economists to solve economic and nancial models that heretofore they were unable to solve within the con nes of traditional mathematical economic analysis.

Chapter 1 Introduction 1.1 Some Apparently Simple Questions Consider the constant elasticity demand function

q=p

0:2 :

This is a function because for each price p there is an unique quantity demanded q . Given a hand-held calculator, any economist could easily compute the quantity demanded at any given price. An economist would also have little diÆculty computing the price that clears the market of a given quantity. Flipping the demand expression about the equality sign and raising each side to the power of 5, the economist would derive a closed-form expression for the inverse demand function

p = q 5: Again, using a calculator any economist could easily compute the price that will exactly clear the market of any given quantity. Suppose now that the economist is presented with a slightly dierent demand function

q = 0:5 p

0:2

+ 0:5 p

0:5 ;

one that is the sum a domestic demand term and an export demand term. Using standard calculus, the economist could easily verify that the demand function is continuous, dierentiable, and strictly decreasing. The economist once again could easily compute the quantity demanded at any price using 1

CHAPTER 1.

INTRODUCTION

2

a calculator and could easily and accurately draw a graph of the demand function. However, suppose that the economist is asked to nd the price that clears the market of, say, a quantity of 2 units. The question is well-posed. A casual inspection of the graph of the demand function suggests that its inverse is well-de ned, continuous, and strictly decreasing. A formal argument based on the Intermediate Value and Implicit Function Theorems would prove that this is so. An unique market clearing price clearly exists. But what is the inverse demand function? And what price clears the market? After considerable eort, even the best trained economist will not nd an answer using Algebra and Calculus. No apparent closed-form expression for the inverse demand function exists. The economist cannot answer the apparently simple question of what the market clearing price will be. Consider now a simple model of an agricultural commodity market. In this market, acreage supply decisions are made before the per-acre yield and harvest price are known. Planting decisions are based on the price expected at harvest:

a = 0:5 + 0:5 Ep: After the acreage is planted, a random yield y~ is realized, giving rise to a supply

q = a y~: The supply is entirely sold at a market clearing price

p = 3 2q: Yield is exogenous and distributed normally with a mean of 1 and a variance of 0.1. Most economists would have little diÆculty deriving the rational expectations equilibrium of this market model. Substituting the rst expression into the second, and then the second into the third, the economist would write

p = 3 2(0:5 + 0:5 Ep) y~: Taking expectations on both sides

Ep = 3 2(0:5 + 0:5 Ep);

CHAPTER 1.

INTRODUCTION

3

she would solve for the equilibrium expected price Ep = 1. She would conclude that the equilibrium acreage is a = 1 and the equilibrium price distribution has a standard deviation of 0.4. Suppose now that the economist is asked to assess the implications of a proposed government price support program. Under this program, the government guarantees each producer a minimum price, say 1. If the market price falls below this level, the government simply pays the producer the dierence per unit produced. The producer thus receives an eective price of max(p; 1) where p is the prevailing market price. The government program transforms the acreage supply relation to

a = 0:5 + 0:5 E max(p; 1): Before proceeding with a formal mathematical analysis, the economist exercises a little economic intuition. The government support, she reasons, will stimulate acreage supply, raising acreage planted. This will shift the equilibrium price distribution to the left, reducing the expected market price below 1. Price would still occasionally rise above 1, however, implying that the expected eective producer price will exceed 1. The dierence between the expected eective producer price and the expected market price represents a positive expected government subsidy. The economist now attempts to formally solve for the rational expectations equilibrium of the revised market model. She performs the same substitutions as before and writes

p = 3 2(0:5 + 0:5 E max(p; 1)) y~: As before, she takes expectations on both sides

Ep = 3 2(0:5 + 0:5 E max(p; 1)): In order to solve the expression for the expected price, the economist uses a fairly common and apparently innocuous trick: she interchanges the max and E operators, replacing E max(p; 1) with max(Ep; 1). The resulting expression is easily solved for Ep = 1. This solution, however, asserts the expected market price and acreage planted remain unchanged by the introduction of the government price support policy. This is inconsistent with the economist's intuition. The economist quickly realizes her error. The expectation operator cannot be interchanged with the maximization operator because the latter is

CHAPTER 1.

INTRODUCTION

4

a nonlinear function. But if this operation is not valid, then what mathematical operations would allow the economist to solve for the equilibrium expected price and acreage? Again, after considerable eort, our economist is unable to nd an answer using Algebra and Calculus. No apparent closed-form solution exists for the model. The economist cannot answer the apparently simple question of how the equilibrium acreage and expected market price will change with the introduction of the government price support program.

1.2 An Alternative Analytic Framework The two problems discussed in the preceding section illustrate how even simple economic models cannot always be solved using standard mathematical techniques. These problems, however, can easily be solved to a high degree of accuracy using numerical methods. Consider the inverse demand problem. An economist who knows some elementary numerical methods and who can write basic Matlab code would have little diÆculty solving the problem. The economist would simply write the following elementary Matlab program: p = 0.25; for i=1:100 deltap = (.5*p^-.2+.5*p^-.5-2)/(.1*p^-1.2 + .25*p^-1.5); p = p + deltap; if abs(deltap) < 1.e-8, break, end end disp(p);

He would then execute the program on a computer and, in an instant, compute the solution: the market clearing price is 0.154. The economist has used Newton's root nding method. Consider now the rational expectations commodity market model with government intervention. The source of diÆculty in solving this problem is the need to evaluate the truncated expectation of a continuous distribution. An economist who knows some numerical analysis and who knows how to write basic Matlab code, however, would have little diÆculty computing the rational expectation equilibrium of this model. The economist would replace the original normal yield distribution with a discrete distribution that has identical lower moments, say one that assumes values y1 ; y2 ; : : : ; yn

CHAPTER 1.

INTRODUCTION

5

with probabilities w1 ; w2 ; : : : ; wn. After constructing the discrete distribution approximant, which would require only a single call to a library routine, call it qnorm, the economist would code and execute the following elementary Matlab program:1 [y,w] = qnwnorm(10,1,0.1); a = 1; for it=1:100 aold = a; p = 3 - 2*a*y; f = w'*max(p,1); a = 0.5 + 0.5*f; if abs(a-aold)tol; d = d/2; if s == sign(f(x)) x = x+d; else x = x-d; end end

In this implementation of the bisection algorithm, d begins each iteration equal to the distance from the current root estimate x to the boundaries of the bracketing interval. The value of d is cut in half, and the iterate is updated by increasing or decreasing its value by this amount, depending on the sign of f(x). If f (x) and f (a) have the same sign, then the current x implicitly becomes the new left endpoint of the bracketing interval and x is moved d units toward b. Otherwise, the current x implicitly becomes the new right endpoint of the bracketing interval and x moved d units toward a. The Matlab toolbox accompanying the textbook includes a function bisect that computes the root of a univariate function using the bisection method. The following script demonstrates how bisect may be used to compute the cube root of 2, or, equivalently, the root of the function f (x) = x3 2: f = inline('x^3-2'); x = bisect(f,1,2)

Execution of this script produces the result x = 1.2599. In this example, the initial bracketing interval is set to [1; 2] and the root is computed to the default tolerance of 1:5 10 8 , or eight decimal places. The sequence of iterates is illustrated in Figure 3.1. The subroutine bisect is extensible in that it allows the user to override the default tolerance and to pass additional arguments for the function f ; the subroutine also checks for input errors. The Matlab operation inline is used here to de ne the function whose root is sought.

CHAPTER 3.

28

NONLINEAR EQUATIONS

Computing Cube Root of 2 by Bisection 6

5

4

3

2

1

0

−1

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2

Figure 3.1

3.2 Function Iteration Function iteration is a relatively simple technique that may be used to compute a xed-point, g (x) = x, of a function from kf (x + dx)k. Backstepping prevents

CHAPTER 3.

NONLINEAR EQUATIONS

40

Newton and quasi-Newton methods from taking a large step in the wrong direction, substantially improving their robustness. Backstepping, however, has the tendency in some applications to begin taking very short steps as the iterations approach the root. One way to prevent this from happening is to employ a `safeguarding' strategy. A simple safeguarding strategy is to seek a reduction in the function norm by repeatedly cutting the Newton step in half, but to stop if the norm begins to rise. This prevents the procedure from getting stuck near the root. The maximum number of allowable backsteps should be no more than, say, 30, which implies a minimum step size that is 2 30 or approximately 1e 9 times the Newton step. The following Matlab script computes the root of a function using a safeguarded Newton's method. It assumes that the user has speci ed a maximum number maxit of Newton iterations, a maximum number maxsteps of backstep iterations, and a convergence tolerance tol, along with the name of the function f and an initial value x: for it=1:maxit [fval,fjac] = f(x); fnorm = norm(fval); if fnorm 0 and xi > i 0. The consumer wants to maximize his utility subject to the budget constraint 3 X

i=1

pi xi I

CHAPTER 4.

FINITE-DIMENSIONAL OPTIMIZATION

78

where pi > 0 denotes the price of xi , I denotes income, and I P3 i=1 pi i > 0: (a) Write the Karush-Kuhn-Tucker necessary conditions for the problem. (b) Verify that the Karush-Kuhn-Tucker conditions are suÆcient for optimality. (c) Derive analytically the associated demand functions. (d) Derive analytically the shadow price and interpret its meaning. (e) Prove that the consumer will utilize his entire income. 3. Derive and interpret the Karush-Kuhn-Tucker conditions for the classical transportation problem: min

s:t:

n X m X {=1 |=1 n X x{| {=1 m X |=1

x{| x{|

d| | = 1; : : : ; m

x{| s{ { = 1; : : : ; n

x{| 0

{ = 1; : : : ; n; | = 1; : : : ; m

State suÆcient conditions for the transportation problem to have an optimal feasible solution. 4. Demand for a commodity in regions A and B is given by:

Region A : p = 200 2q Region B : p = 100 4q Supply is given by:

Region A : p = 20 + 8q Region B : p = 10 + 6q: The transportation cost between regions is $10 per unit.

CHAPTER 4.


79

Formulate an optimization problem that characterizes the competitive spatial price equilibrium. Derive, but do not solve, the Karush-KuhnTucker conditions. Interpret the shadow prices. 5. Consider a vector of n random assets with expected return n1 and variance nn . Formulate a quadratic program whose solution is the Markowitz E-V eÆcient portfolio xn1 whose expected return is at least r . Derive the Karush-Kuhn-Tucker conditions for the program. Interpret the Lagrangian multiplier and explain its relation to the risk aversion parameter of the objective function 0 x x0 x of Freund's portfolio choice model. 6. Consider the nonlinear programming problem

maxx ;x x22 2x1 x21 s:t: x21 + x22 1 x1 0; x2 0: 1

2

(a) Write the Karush-Kuhn-Tucker necessary conditions for the problem. (b) What points satisfy the Karush-Kuhn-Tucker necessary conditions. (c) Are the Karush-Kuhn-Tucker conditions suÆcient for optimality? (d) How do you know that problem possesses an optimum? (e) Determine the optimum, if any. 7. A tomato processor operates two plants whose hourly variable costs (in dollars) are, respectively,

c1 = 80 + 2:0x1 + 0:001x21 c2 = 90 + 1:5x2 + 0:002x22 ; where xi is the number of cases produced per hour at plant i. In order to meet contractual obligations, he must produce at a rate of at least 2000 cases per hour (x1 + x2 2000.) He wishes to do so at minimal cost.

CHAPTER 4.


80

(a) Write the Karush-Kuhn-Tucker necessary conditions for the problem. (b) Verify that the Karush-Kuhn-Tucker conditions are suÆcient for optimality. (c) Determine the optimal levels of production. (d) Determine the optimal value of the shadow price and interpret its meaning. 8. Consider the problem of allocating a scarce resource, the total supply of which is b > 0, among n tasks with separable rewards:

f1 (x1 ) + f2 (x2 ) + : : : + fn (xn ) s:t: x1 + x2 + : : : + xn b x1 0; x2 0; : : : ; xn 0:

max

x1 ;x2 ;:::;xn

Assume each fi is strictly increasing and dierentiable but not necessarily concave. (a) (b) (c) (d) (e)

How do you know that problem possesses an optimum? Write the Karush-Kuhn-Tucker necessary conditions. Prove that the scarce resource will be completely utilized. Interpret the shadow price associated with the resource constraint. Given a marginal increase in the supply of the resource, to which task(s) would you allocate the additional amount.

9. Consider a one-output two-input production function

y = f (x1 ; x2 ) = x21 + x22 : Given the prices of inputs 1 and 2, w1 and w2 , the minimum cost of producing a given level of output, y, is obtained by solving the constrained optimization problem min C = w1 x1 + w2 x2

x1 ;x2

s:t:

f (x1 ; x2 ) y:

Letting denote the shadow price associated with the production constraint, answer the following questions:

CHAPTER 4.


81

(a) Write the Karush-Kuhn-Tucker necessary conditions. (b) Find explicit expressions for the optimal x1 , x2 , and C . (c) Find an explicit expression for the optimal and interpret its meaning. (d) Dierentiate the expression for C to con rm that @C @ y = . 10. A salmon cannery produces Q 1-lb. cans of salmon according to a technology given by Q = 18K L , where capital K is xed at 16 units in the shortrun and labor L may be hired in any quantity at a wage rate of w dollars per unit. Each unit of output provides a pro t contribution of 1 dollar. 1 4

1 3

(a) Derive the rm's shortrun demand for labor. (b) If w = 3, how much would the rm be willing to pay to rent a unit of capital. 11. Consider the nonlinear programming problem min s:t:

x1 ;:::;x4

x01:25 x03:50 x04:25 x1 + x2 + x3 + x4 4 x1 ; x2 ; x3 ; x4 0:

(a) What can you say about the optimality of the point (1; 0; 2; 1)? (b) Does this program possess all the correct curvature properties for the Karush-Kuhn-Tucker conditions to be suÆcient for optimality throughout the feasible region? Why or why not? (c) How do you know that problem possesses an optimal feasible solution? 12. Consider the non-linear programming problem min s:t:

x1 ;x2

2x21 12x1 + 3x22 3x1 + x2 12 x1 + x2 6 x1 ; x2 0:

18x2 + 45

The optimal solution to this problem is: x1 = 3 and x2 = 3.

CHAPTER 4.


82

(a) Verify that the Karush-Kuhn-Tucker conditions are satis ed by this solution. (b) Determine the optimal values for the shadow prices 1 and 2 associated with the structural constraints, and interpret 1 and 2 . (c) If the second constraint were changed to x1 + x2 5, what would be the eect on the optimal values of x1 , x2 , 1 , and 2 ? 13. Suppose that the probability density function of a non-negative random variable, y , is exp( yi =i)=i where i = Xi for some observable data Xi . (a) Show that the rst order conditions for the maximum likelihood estimator of can be written as X Xi > yi X Xi > Xi = : (Xi )2 (Xi )2 (b) Use this result to de ne a recursive algorithm to estimate . (c) Write a Matlab function of the form [beta,sigma]=example(y,X) that computes the maximum likelihood estimator of and its asymptotic covariance matrix . The function should be a standalone procedure (i.e., do not call any optimization or root- nding solvers) that implements the recursive algorithm. (d) Show that the recursive algorithm can be interpreted as a quasiNewton method. Explain fully. 14. Write a Matlab function that is passed a vector of observations (of positive numbers) and returns the maximum likelihood estimates of and their covariance matrix for the two-parameter gamma function:

f (x; ) =

2 x 1

1

1 e 2 x

: (1 ) Hint: Formulate the problem as a maximization of the log-likelihood. Note that the rst and second derivatives of the log of the function are the psi and trigamma functions. The Matlab toolbox contains procedures to evaluate these special functions.

CHAPTER 4.


83

15. Continuing in the vein of the last problem, reformulate the likelihood function of the two-parameter Gamma distribution in terms of 1 and = 1 =2 . (a) Solve explicitly for the optimal , and express the likelihood function in terms of 1 and the data alone. (b) Write a Matlab function that maximizes the resulting univariate likelihood function using algorithm golden provided in the toolbox. (c) Write a Matlab function that maximizes the resulting univariate likelihood function using algorithm newton provided in the toolbox. (d) The maximum likelihood Pn estimator of depends on the data 1 only through Y1 = n i=1 xi , the arithmetic mean, and Y2 = Pn 1 exp( n i=1 ln(xi )), the geometric mean (Y1 and Y2 are known as suÆcient statistics for ). Plot 1 as a function of ln(Y1 =Y2 ).

Chapter 5 Numerical Integration and Dierentiation In many computational economic applications, one must compute the de nite integral of a real-valued function f with respect to a \weighting" function w over an interval I of 0 and > 0 (the model is scaled to eliminate excess parameters; you should determine how many scaling dimensions the model has). Although admittedly a simple model, it captures some of the essential features of the relationship. First the prey population grows at rate when there are no predators present and the greater the number of predators, the slower the population grows and declines when the predator population exceeds . The predator population, on the other hand declines if it grows too large unless prey is plentiful. Determine the equilibria (there are two) and draw the phase diagram [hint: this model exhibits cycles]. 4. Demand for a commodity is given by q = 2p 0:5 . The price of a good falls from 4 to 1. Compute the change in consumer surplus: (a) (b) (c) (d)

analytically using Calculus; numerically using a 10 interval trapezoid rule; numerically using a 10 interval Simpson rule; numerically using a 10 point Gauss-Legendre rule.

5. For z > 0, the cumulative probability function for a standard normal random variable is given by

F (z ) = 0:5 + p12

Rz 0 exp

f

x2 2

g dx:

(a) Write a short Matlab program that will estimate the value of F (z ) using Simpson's rule. The program should accept z and the number of intervals n in the discretization as input; the program should print F (z ). (b) What values of F (z ) do you obtain for z = 1 and n = 6, n = 10, n = 20 n = 50, n = 100? How do these values compare to published statistical tables? 6. Using Monte Carlo integration, estimate the expectation of f (X~ ) = 1=(1 + X~ 2 ) where X~ is exponentially distributed with CDF F (x) = 1 exp( x) for x 0. Compute an estimate using 100, 500, and 1000 replicates.

CHAPTER 5.

INTEGRATION AND DIFFERENTIATION

116

7. A government stabilizes the supply of a commodity at S = 2, but allows the price to be determined by the market. Domestic and export demand for the commodity are given by: D = ~1 P 1:0 X = ~2 P 0:5 ; where log ~1 and log ~2 are normally distributed with means 0, variances 0.02 and 0.01, respectively, and covariance 0.01. (a) Compute the expected price Ep and the ex-ante variance of price V p using a 6th degree Gaussian discretization for the demand shocks. (b) Compute the expected price Ep and the ex-ante variance of price V p using a 1000 replication Monte Carlo integration scheme. (c) Repeat parts (a) and (b) assuming the log of the demand shocks are negatively correlated with covariance -0.01. 8. Consider the commodity market model of Chapter 1, except now assume that log yield is normally distributed with mean 0 and standard deviation 0.2. (a) Compute the expectation and the variance of price without government support payments. (b) Compute the expectation and the variance of the eective producer price assuming a support price of 1. 9. Consider a market for an agricultural commodity in which farmers receive a government de ciency payment whenever the market price p drops below an announced target price p. In this market, producers base their acreage planting decisions on their expectation of the eective producer price f = max(p; p); speci cally, acreage planted a is given by:

a = 1 + (Ef )0:5 : Production q is acreage planted a times a random yield y~, unknown at planting time:

q = a y~;

CHAPTER 5.


117

and quantity demanded at harvest is given by

q=p

0:2 + p 0:5 :

Conditional on information known at planting time, log y is normally distributed with mean 0 and variance 0.03. For p = 0, p = 1, and p = 2, compute: (a) (b) (c) (d) (e)

the expected subsidy E [q (f p)]; the ex-ante expected producer price Ef ; the ex-ante variance of producer price V f ; the ex-ante expected producer revenue E [fq ]; and the ex-ante variance of producer revenue V [fq ].

10. Suppose acreage planted at the beginning of the growing season is given by a = (Ep; V p) where p is price at harvest time and E and V are the expectation and variance operators conditional on information known at planting time. Further suppose that p = (ay ) where yield y is random and unknown at planting time. Develop an algorithm for computing the acreage planted under rational expectations. 11. Professor Jones, a well-known econometrician, argues that the best way to approximate a real-valued function with no closed-form expression over an interval is to (1) evaluate the function at n equally-spaced points and then (2) t an m-degree polynomial to the points, using ordinary least squares to compute the coeÆcients on the xi terms, i = 0; 1; 2; : : : ; m. To improve the approximation, he further argues, increase n until the standard errors are tolerably close to zero. Is Jones's approach sensible? If not, what method would you recommend? Justify your method using language that Jones is capable of understanding. 12. Professor Sayan, a regional economist, maintains a large deterministic model of the Turkish economy. Using his model, Professor Sayan can estimate the number of new jobs y that will be created under the new GATT agreement. However, Dr. Sayan is unsure about the value of one critical model parameter, the elasticity of labor supply x. A

CHAPTER 5.


118

recent econometric study estimated the elasticity to be x and gave an asymptotic normal standard error . Given the uncertainty about the value of x, Dr. Sayan wishes to place a con dence interval around his estimate of y . He has considered using Monte Carlo methods, drawing pseudo-random values of x according to the published distribution and computing the value of y for each x. However, a large number of replications is not feasible because two hours of mainframe computer time are needed to solve the model each time. Do you have a better suggestion for Dr. Sayan? Justify your answer.

Chapter 6 Function Approximation In many computational economic applications, one must approximate an analytically intractable real-valued function f with a computationally tractable function f^. Two types of function approximation problems arise often in computational economic applications. In the interpolation problem, one is given or otherwise uncovers some properties satis ed by the function f and then must choose an approximant f^ from a family of `nice', tractable functions that satis es those properties. The data available about f is often just its value at a set of speci ed points. The data, however, could include rst or higher derivatives of f at some of the points. Interpolation methods were originally developed to approximate the value of mathematical and statistical functions from published tables of values. In most modern computational economic applications, however, the analyst is free to chose what data to obtain about the function to be approximated. Modern interpolation theory and practice is concerned with ways to optimally extract data from a function and with computationally eÆcient methods for constructing and working with its approximant. In the functional equation problem, one must nd a function f that satis es Tf = g where T is an operator that maps a vector space of functions into itself and g is a known function in that space. In the equivalent functional xed-point problem, one must nd a function f such that T f = f: 119

CHAPTER 6.

FUNCTION APPROXIMATION

120

Functional equations are common in dynamic economic analysis. For example, the Bellman equation that characterizes the solutions of a dynamic optimization model is a functional xed-point equation. The Euler equation and the fundamental asset pricing dierential equation are also functional equations. Functional equations are diÆcult to solve because the unknown is not simply a vector in p:

j =

a if j 1 : b if j p

Additionally, at the endpoints we set the terms

B0k

1;

Bnk

1;

= 0: 1 1 k n+1 n k+1 Given a B-spline representation of a spline, the spline can easily be differentiated by computing simple dierences, and can be integrated by computing simple sums. Speci cally: =

dBjk; (x) k = Bk dx j j k j and

Z x a

Bjk; (z )dz

=

n X i i=j

1; 1 (x)

j +1

k Bk j +1 k j

1;

(x)

i k k+1; Bi+1 (x): k

Although these formulae appear a bit complicated, their application in computer programs is relatively straightforward. First notice that the derivative of a B-spline of order k is a weighted sum of two order k 1 B-splines. Thus,

CHAPTER 6.


130

the derivative of an order k spline is an order k 1 spline with the same breakpoints. Similarly, the integral of a B-spline can be represented in terms of two B-splines of order k + 1 splines. Thus, the antiderivative of an order k spline is an order k + 1 spline with the same breakpoints. Two classes of splines are often employed in practice. A rst-order or linear spline is a series of line segments spliced together to form a continuous function. A third-order or cubic spline is a series of cubic polynomials segments spliced together to form a twice continuously dierentiable function. Linear spline approximants are particularly easy to construct and evaluate in practice, which explains their widespread popularity. Linear splines use line segments to connect points on the graph of the function to be approximated. A linear spline with n evenly spaced breakpoints on the interval [a; b] may be written as a linear combination

f^(x) =

n X i=1

ci i(x)

of the basis functions:

j (x) =

1 0

jx j j if jx w

j j w otherwise

Here, w = (b a)=(n 1) is the distance between breakpoints and j = a+(j 1)w, j = 1; 2; : : : ; n, are the breakpoints. The linear spline basis functions are popularly called the \hat" functions, for reasons that are clear from Figure 6.5. This gure illustrates the basis function for twelve-degree, evenly spaced breakpoint linear splines on the interval [0; 1]. Each hat function is zero everywhere, except over a narrow support element of width 2w. The basis function achieves a maximum of 1 at the midpoint of its support element. One can x the coeÆcients of an n-degree linear spline approximant for a function f by interpolating its values at any n points of its domain, provided that the resulting interpolation matrix is nonsingular. However, if the interpolation nodes x1 ; x2 ; : : : ; xn are chosen to coincide with the spline breakpoints 1 ; 2 ; : : : ; n , then computing the basis coeÆcients of the linear spline approximant becomes a trivial matter. If the interpolation nodes and breakpoints coincide, then i (xj ) equals one if i = j , but equals zero otherwise. That is, the interpolation matrix is simply the identity matrix and the interpolation equation reduces to the trivial identity c = y where y is

CHAPTER 6.

131


Linear Spline Basis Functions 1

1

0

1

0 0

1

1

0 0

1

1

0

0 0

1

1

1

1

1

1

1

0

1

0

1

1

0 0

1

0 0

1

0

0 1

0 0

1

0 0

1

0

0 1

0 0

1

Figure 6.5 the vector of function values at the interpolation nodes. The linear spline approximant of f when nodes and breakpoints coincide thus takes the form

f^(x) =

n X i=1

f (xi )i (x):

When interpolation nodes and breakpoints coincide, no computations other than function evaluations are required to form the linear spline approximant. For this reason linear spline interpolation nodes in practice are always chosen to be the spline's breakpoints. Evaluating a linear spline approximant and its derivative at an arbitrary point x is also straightforward. Since at most two basis functions are nonzero at any point, only two basis function evaluations are required. Speci cally, if i is the greatest integer less than 1 + (x a)=w, then x lies in the interval [xi ; xi+1 ]. Thus, f^(x) = (ci+1 (x xi ) + ci (xi+1 x))=w

CHAPTER 6.


132

and

f^0 (x) = (ci+1

ci )=w:

Higher order derivatives are zero, except at the breakpoints, where they are unde ned. Linear splines are attractive for their simplicity, but have certain limitations that often make them a poor choice for computational economic applications. By construction, linear splines produce rst derivatives that are discontinuous step functions and second derivative that are zero almost everywhere. Linear spline approximants thus typically do a very poor job of approximating the rst derivative of a nonlinear function and are incapable of approximating its second derivative. In some economic applications, the derivative represents a measure of marginality that is of as much interest to the analyst as the function itself. In other applications, the rst and maybe second derivative of the function may be needed to solve for the root of the function using Newton-like method. Cubic spline approximants oer a higher degree of smoothness while retaining much of the exibility and simplicity of linear spline approximants. Because cubic splines possess continuous rst and second derivatives, they typically produce adequate approximations for both the function and its rst and second derivatives. The basis functions for n-degree, evenly spaced breakpoint cubic splines on the interval [a; b] are generated using the n 2 breakpoints j = a+w(j 1), j = 1; 2; : : : ; n 2, where w = nb a3 . Cubic spline basis function generated with evenly spaced breakpoints are nonzero over a support element of width 4w. As such, at any point of [a; b], at most four basis functions are nonzero. The basis functions for twelve-degree, evenly spaced breakpoint cubic splines on the interval [0; 1] are illustrated in Figure 6.6. Although spline breakpoints are often chosen to be evenly spaced in most applications, this need not be the case. Indeed, the ability to distribute breakpoints unevenly and to stack them on top of one another adds considerably to the exibility of splines, allowing them to accurately approximate a wide range of functions. In general, functions that exhibit wide variations in curvature are diÆcult to approximate numerically with entire polynomials of high degree. With splines, however, one can often nesse curvature diÆculties by concentrating breakpoints in regions displaying the highest degree of curvature.

CHAPTER 6.

133


Cubic Spline Basis Functions 1

1

0

1

0 0

1

1

0 0

1

1

0

0 0

1

1

1

1

1

1

1

0

1

0

1

1

0 0

1

0 0

1

0

0 1

0 0

1

0 0

1

0

0 1

0 0

1

Figure 6.6 To illustrate the importance of breakpoint location, consider the problem of forming a cubic spline approximant for Runge's function 1 f (x) = for x 2 [ 5; 5]: 1 + 25x2 Figure 6.7 displays two cubic spline approximations, one using thirteen evenly spaced breakpoints, the other using thirteen breakpoints that cluster around zero (the breakpoints are indicated by `x' symbols). Figure 6.8 shows the associated approximation errors (note that the errors for the unevenly spaced approximation have been multiplied by 100). In Figure 6.7 the unevenly spaced breakpoints approximation lies almost on top of the actual function, whereas the even spacing leads to signi cant errors, especially near zero. The gures clearly demonstrates the power of spline approximations with good breakpoint placement. The placement of the breakpoints can also be used to aect the continuity of the spline approximant and its derivatives. By stacking breakpoints on top

CHAPTER 6.

134


Runge’s Function with Spline Approximations 1.2

Runge Even Spacing Uneven Spacing

1

0.8

y

0.6

0.4

0.2

0

−0.2 −5

−4

−3

−2

−1

0

1

2

3

4

5

x

Figure 6.7 of one another, we can reduce the smoothness at the breakpoints. Normally, an order k spline has continuous derivatives to order k 1 at the breakpoints. By stacking q breakpoints, we can reduce this to k q continuous derivatives at this breakpoint. For example, with two equal breakpoints, a cubic spline possesses a discontinuous second derivative at the point. With three equal breakpoints, a cubic spline possesses a discontinuous rst derivative at that point, that is, it exhibits a kink there. Stacking breakpoints is a useful practice if the function is known a priori to exhibit a kink at a given point, a not uncommon occurrence in practice. Regardless of the placement of breakpoints, splines have several important and useful properties. We have already commented on the limited domain of the basis function. This limited support implies that spline interpolation matrices are sparse and for this reason can be stored and manipulated as sparse matrices. This property is extremely useful in high-dimensional problems for which a fully expanded interpolation matrix would strain any computer's memory. Another useful feature of splines is that their values

CHAPTER 6.

135


Approximation Errors for Runge’s Function 0.6

Even Spacing Uneven Spacing 0.4

0.2

y

0

−0.2

−0.4

−0.6 Uneven spacing errors 100x −0.8 −5

−4

−3

−2

−1

0

1

2

3

4

5

x

Figure 6.8 are bounded, thereby reducing the likelihood that scaling eects will cause numerical diÆculties. In general, the limited support and bounded values make spline basis matrices well-conditioned. If the spline interpolation matrix must be reused, one must resist the temptation to form and store its inverse, particularly if the size of the matrix is large. Inversion destroys the sparsity structure. More speci cally, the inverse of the interpolation matrix will be dense, even though the interpolation matrix is not. When n is large, solving the sparse n by n linear equation using sparse L-U factorization will generally be less costly than performing the matrix-vector multiplication required with the dense inverse interpolation matrix.

CHAPTER 6.


136

6.4 Multidimensional Interpolation The univariate interpolation methods discussed in the preceding sections may be extended in a natural way to multivariate functions through the use of tensor products. To illustrate, consider the problem of approximating a bivariate real-valued function f (x; y ) de ned on a bounded interval I = f(x; y)y j ax x bx; ay y by g in 0,

P (s0js; x)

1 s0 = 0 (death is permanent) 0 otherwise;

8 px qx > > < = p(1x (1 p )qx ) > x > :

0

s0 = min(s; s 1 + ex ) (survive, nds food) s0 = s 1 (survive, no food) s0 = 0 (does not survive) otherwise.

The reward function is

f (s; x) = 0: Here, s = 0 is an absorbing state that, once entered, is never exited. More to the point, an animal whose energy stocks fall to zero dies, and remains dead. The reward function for periods 1 through T is zero, because there is only one payo, surviving to procreate, and this payo is earned in period T + 1.

CHAPTER 7.

DISCRETE STATE MODELS

172

The value function

Vt (s) = probability of procreating, given energy stocks s in period t must satisfy Bellman's equation

Vt (s) = max fpxqx Vt+1(min(s; s 1 + e)) + px(1 qx )Vt+1(s 1)g; x2X

for t 2 1; : : : ; T , with Vt (0) = 0, subject to the terminal condition

VT +1 (s) =

0 s=0 1 s>0

7.3 Solution Algorithms Below, we develop numerical solution algorithms for stochastic discrete time, discrete space Markov decision models. The algorithms apply to deterministic models as well, provided one views a deterministic model as a degenerate special case of the stochastic model for which the transition probabilities are all zeros or ones. To develop solution algorithms, we must introduce some vector notation and operations. Assume that the states S = f1; 2; : : : ; ng and actions X = f1; 2; : : : ; mg are indexed by the rst n and m integers, respectively. Let v 2 i f(i,k) = -inf; else f(i,k) = irrben(k) + recben(i-k+1);

CHAPTER 7.


195

end end end

Here, a reward matrix element is set to negative in nity if the irrigation level exceeds the available water stock, an infeasible action. Next, one constructs the transition probability matrix: P = []; for k=1:m Pk = sparse(zeros(n,n)); for i=1:n; j=i-k+1; j=max(1,j); j=min(n,j); Pk(i,j) = Pk(i,j) + 0.4; j=j+1; j=max(1,j); j=min(n,j); Pk(i,j) = Pk(i,j) + 0.6; end P = [P;Pk]; end

One then packs the essential model data into a structured variable model: model.reward model.transition model.horizon model.discount

= = = =

f; P; inf; delta;

To solve the in nite horizon model via policy iteration, one issues the command: [v,x] = ddpsolve(model);

To solve the in nite horizon model via function iteration, one issues the command: [v,x] = ddpsolve(model,'func');

Upon convergence, v will be n vector containing the value function and x will be n vector containing the optimal irrigation policy. Once the optimal solution has been computed, one may plot the optimal value and irrigation policy functions:

CHAPTER 7.

196


figure(1); plot(S,X(x)); xlabel('Stock'); ylabel('Optimal Irrigation'); figure(2); plot(S,v); xlabel('Stock'); ylabel('Optimal Value');

Suppose one wished to compute the steady-state stock level. One could easily do this by calling markov to compute the steady state distribution and integrating: pi = markov(pstar); avgstock = pi'*S; fprintf('\nSteady-state Stock

%8.2f\n',avgstock)

To plot expected water level over time given that water level is currently zero, one would issue the commands figure(3) nyrs = 20; s1=ones(10000,1); st = ddpsimul(pstar,s1,nyrs,x); plot(1:nyrs,mean(S(st))); xlabel('Year'); ylabel('Expected Water Level');

Here, we use the function ddpsimul to simulate the evolution of the water level via Monte Carlo 10000 times over a 20 year horizon. The mean of the 10000 replications is then computed and plotted for each year in the simulation. The expected path, together with the optimal value and policy functions are given in Figure 7.5.

7.6.7 Bioeconomic Model Consider the bioeconomic model with three foraging areas, predation survival probabilities p1 = 1, p2 = 0:98, and p3 = 0:90, and foraging success probabilities q1 = 0, q2 = 0:3, and q3 = 0:8. Also assume that successful foraging delivers e = 4 units of energy in all areas and that the procreation horizon is 10 periods. The rst step required to solve the model numerically is to specify the model parameters and to construct the state and action spaces:

CHAPTER 7.

197

DISCRETE STATE MODELS 75

1

0.9

70

0.8

65

60

Optimal Value

Optimal Irrigation

0.7

0.6

0.5

0.4

55

50

0.3

45 0.2

40

0.1

0

0

0.5

1

1.5

2

2.5

35

3

0

0.5

1

1.5

2

2.5

3

Stock

Stock 3

Expected Water Level

2.5

2

1.5

1

0.5

0

0

2

4

6

8

10

12

14

16

18

20

Year

Figure 7.5: Solution to Optimal Irrigation Problem T = 10; eadd = 4; emax = 10; S = 0:emax; n = length(S); X = 1:3; m = length(X);

% % % % % % %

foraging periods energy from foraging energy capacity energy levels number of states foraging areas number of actions

There is no need to explicitly de ne an action space since actions are represented by integer indices. Next, one constructs the reward and transition probability matrices: f = zeros(n,m); p = [1 .98 .9];

% predation survival prob.

CHAPTER 7.


198

q = [0 .30 .8]; % foraging success prob. P = []; for k=1:m Pk = zeros(n,n); Pk(1,1) = 1; for i=2:n; Pk(i,min(n,i-1+eadd)) = p(k)*q(k); Pk(i,i-1) = p(k)*(1-q(k)); Pk(i,1) = Pk(i,1) + (1-p(k)); end P = [ P ; Pk ]; end

Note that the reward matrix is zero because the reward is not earned until the post-terminal period. Upon the reaching the post-terminal period, either the animal is alive, earning reward of 1, or is dead, earning a reward of 0. We capture this by specifying the terminal value function as follows v = ones(n,1); v(1) = 0;

% terminal value: survive % terminal value: death

One then packs the essential model data into a structured variable model: model.reward model.transition model.horizon model.discount model.vterm

= = = = =

f; P; inf; delta; v;

To solve the nite horizon model via backward recursion, one issues the command: [v,x] = ddpsolve(model);

Upon convergence, v will be n by 1 matrix containing the value function and ix will be n by 1 matrix containing the indices of the optimal foraging policy for all possible initial energy stock levels. Once the optimal solution has been computed, one may print out the survival probabilities (see Table 7.4):

CHAPTER 7.

199


Table 7.4: Survival Probabilities Period 1 2 3 4 5 6 7 8 9 10

0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1 0.59 0.59 0.64 0.64 0.64 0.64 0.72 0.72 0.72 0.72

2 0.71 0.77 0.77 0.77 0.77 0.85 0.85 0.85 0.85 1.00

3 0.80 0.80 0.80 0.80 0.88 0.88 0.88 0.88 1.00 1.00

Stock of Energy 4 5 6 0.82 0.83 0.85 0.82 0.83 0.92 0.82 0.91 0.92 0.90 0.91 0.92 0.90 0.91 0.92 0.90 0.91 1.00 0.90 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

7 0.92 0.92 0.92 0.92 1.00 1.00 1.00 1.00 1.00 1.00

8 0.93 0.93 0.93 1.00 1.00 1.00 1.00 1.00 1.00 1.00

9 0.93 0.93 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

10 0.93 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

fprintf('\nProbability of Survival\n') disp(' Stock of Energy') fprintf('Period ');fprintf('%5i ',S);fprintf('\n'); for t=1:T fprintf('%5i ',t);fprintf('%6.2f',v(:,t)');fprintf('\n') end

A similar script can be executed to print out the optimal foraging strategy (see Table 7.5).

Exercises 1. Consider a competitive price-taking rm that wishes to maximize the present value sum of current and future pro ts from harvesting a nonrenewable resource. In year t, the rm earns revenue pt xt where pt is the market price for the harvested resource and xt is the amount harvested by the rm; the rm also incurs cost x t , where and are cost function parameters. The market price takes one of two values, p1 or p2 , according to the rst-order Markov probability law: Pr[pt+1 = pj jpt = pi ] = wij :

CHAPTER 7.

200


Table 7.5: Optimal Foraging Strategy Period 1 2 3 4 5 6 7 8 9 10

0 1 1 1 1 1 1 1 1 1 1

1 3 3 3 3 3 3 3 3 3 3

2 3 3 3 3 3 3 3 3 3 1

Stock of Energy 3 4 5 6 7 3 2 2 2 2 3 2 2 2 2 3 2 2 2 2 3 2 2 2 2 2 2 2 2 1 2 2 2 1 1 2 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1

8 2 2 2 1 1 1 1 1 1 1

9 10 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Assuming an annual discount factor of Æ , and that harvest levels and stocks must be integers, formulate the rm's optimization problem. Speci cally, formulate Bellman's functional equation, clearly identifying the state and action variables, the state and action spaces, and the reward and probability transition functions. 2. Consider a timber stand that grows by one unit of biomass per year. That is, if the stand is planted with seedlings at the beginning of year t, it will contain t0 t units of biomass in year t0 . Harvesting decisions are made at the beginning of each year. If the stand is harvested, new seedlings are replanted at the end of the period (so the stand has biomass 0 in the next period). The price of harvested timber is p dollars per unit and the cost of harvesting and replanting is c. The timber rm discounts the future using a discount factor of Æ . (a) Set up the decision problem (de ne states, controls, reward function, transition rule). (b) Formulate the value function and Bellman's recursive functional equation. (c) For parameters values Æ = 0:95, p = 1 and c = 5, determine the optimal harvesting policy.

CHAPTER 7.


201

3. Suppose that a new machine costs c and that the net pro t contribution of a machine of age i is pi for i = 0; 1; : : : ; n, where 0 = pn < pn 1 < pn 2 < : : : < p0 . Formulate the rm's pro t maximization problem. Speci cally, formulate Bellman's functional equation, clearly identifying the state and action variables, the state and action spaces, and the reward and probability transition functions. 4. Suppose that a new machine costs $50 and that the net pro t contribution of a machine is: age 0 1 2 3 4+

net profit 50 45 35 20 0

What is the optimal replacement policy for this machine. 5. Suppose that a new machine costs $75 and that its net pro t contribution in a given year

f (a; n) = (50 2:5a 2:5a2 ) (1 (a n 1)=4) depends both on its age a at the beginning of the year and number of times n that it has been serviced. At the beginning of the year, one must decide whether to keep and service the machine, keep but not service the machine, or replace the machine (one does not service a new machine). It costs $10 to service a machine. Assuming a discount factor of 0.9, what is the optimal replacement-maintenance policy for the machine? 6. A rm operates in an uncertain pro t environment. At the beginning of each period t, the rm observes its potential short-run variable pro t t , which may be negative, and then decides whether to operate, making a short-run variable pro t t , or to temporarily shut down, making a short-run variable pro t of zero. Although the rm faces no xed costs or shut-down costs, it incurs a start-up cost c if it reopens after a period

CHAPTER 7.

202


of inactivity. The short-run variable pro t t follows a stationary rstorder Markov process. Speci cally, short-run variable pro t assumes ve values p1 , p2 , p3 , p4 , and p5 with stationary transition probabilities Pij = Pr(t+1 = pj jt = pi ). (a) Formulate the rm's in nite horizon pro t maximization problem. Speci cally, formulate Bellman's functional equation, clearly identifying the state and action variables, the state and action spaces, and the reward and probability transition functions. (b) In the standard static model of the rm, a previously open rm will shut down if its short-run variable pro t pt is negative. Is this condition suÆcient in the current model? (c) In the standard static model of the rm, a previously closed rm will reopen if its short-run variable pro t pt exceeds the start-up cost c. Is this condition necessary in the current model? 7. Consider the preceding problem under the assumption that the startup cost is c = 0:8, the discount factor is Æ = 0:95, and the short-run variable pro t assumes ve values p1 = 1:0, p2 = 0:2, p3 = 0:4, p4 = 1:2, and p5 = 2:0 with stationary transition probabilities:

from

p_1 0.1 0.1 0.1 0.2 0.3

p_1 p_2 p_3 p_4 p_5

p_2 0.2 0.3 0.5 0.1 0.2

to p_3 0.3 0.2 0.2 0.3 0.2

p_4 0.4 0.2 0.1 0.2 0.1

p_4 0.0 0.2 0.1 0.2 0.2.

(a) Compute the optimal operation-closure policy. (b) What is the value of the rm? (c) In the long-run, what percentage of the time will be rm be closed? 8. Consider the problem of optimal harvesting of a nonrenewable resource by a competitive price-taking rm: P

t max E 1 t=0 Æ [pt xt s.t. st+1 = st xt

x t ]

CHAPTER 7.


203

where Æ = 0:9 is the discount factor; = 0:2, = 1:5, are cost function parameters; pt is the market price; xt is harvest; and st is beginning reserves. Develop a Matlab program that will solve this problem numerically assuming stock and harvest levels are integers, then answer the following questions. (a) Graph the value function for p = 1 and p = 2. (b) Graph the optimal decision rule for p = 1 and p = 2. (c) Assuming an initial stocks of 100 units, graph the time path of optimal harvest for periods t = 0 to t = 20, inclusive; do so for both p=1 and p=2. (d) Under the same assumption as in (c), graph the shadow price of stocks for periods t = 0 to t = 20. Do so both in current dollars and in year 0 dollars. 9. Consider the preceding problem, but now assume that price takes one of two values, p = 1 or p = 2 according to the following rst-order Markov probability law: Pr[pt+1 = 1jpt = 1] Pr[pt+1 = 2jpt = 1] Pr[pt+1 = 1jpt = 2] Pr[pt+1 = 2jpt = 2]

= = = =

0:8 0:2 0:3 0:7

Further assume that the manager maximizes the discounted sum of expected utility over time, where utility in year t is

ut = expf (pt xt

x t )g

where = 0:2 is the coeÆcient of absolute risk aversion. (a) Write a Matlab program that solves the problem. (b) Graph the optimal decision rule for this case and for the risk neutral case on the same graph. (c) What is the eect of risk aversion on the rate of optimal extraction in this model?

CHAPTER 7.


204

10. Consider the article by Burt and Allison, \Farm Management Decisions with Dynamic Programming," Journal of Farm Economics, 45(1963):121-37. Write a program that replicates Burt and Allison's results, then compute the optimal value function and decision rule if: (a) the annual interest rate is 1 percent. (b) the annual interest rate is 10 percent. 11. Consider Burt and Allison's farm management problem. Assume now that the government will subsidize fallow land at $25 per acre, raising the expected return on a fallow acre from a $2.33 loss to a $22.67 pro t. Further assume, as Burt and Allison implicitly have, that cost, price, yield, and return are determinate at each moisture level: (a) Compute the optimal value function and decision rule. (b) Derive the steady-state distribution of the soil moisture level under the optimal policy. (c) Derive the steady-state distribution of return per acre under the optimal policy. (d) Derive the steady-state mean and variance of return per acre under the optimal policy. 12. At the beginning of every year, a rm must decide how much to produce over the coming year in order to meet the demand for its product. The demand over any year is known at the beginning of the year, but varies annually, assuming serially independent values of 5, 6, 7, or 8 thousand units with probabilities 0.1, 0.3, 0.4, and 0.2, respectively. The rm's cost of production in year t is 10qt + (qt qt 1 )2 thousand dollars, where qt is thousands of units produced in year t. The product sells for $20 per unit and excess production can either be carried over to the following year at a cost of $2 per unit or disposed of for free. The rm's production and storage capacities are 8 thousand and 5 thousand units per annum, respectively. The annual discount factor is 0.9. Assuming that the rm meets its annual demand exactly, and that production and storage levels must be integer multiples of one thousand units, answer the following questions: (a) Under what conditions would the rm use all of its storage capacity?

CHAPTER 7.


205

(b) What is the value of rm and what is its optimal production if its previous year's production was 5 thousand units, its carryin is 2 thousand units, and the demand for the coming year is 7 units? (c) What would be the production levels over the subsequent three years if the realized demands were 6, 5, and 8 units, respectively?

Chapter 8 Discrete Time Continuous State Dynamic Models: Theory We now turn our attention to discrete time dynamic economic models with state variables that may assume a continuum of values. Three classes of discrete time, continuous state dynamic economic models are examined. One class includes models of centralized decisionmaking by individuals, rms, or institutions. Examples of continuous state decision models admitting a continuum of choices include a central planner managing the harvest of a natural resource so as to maximize social welfare, an entrepreneur planning production and investment so as to maximize the present value of her rm, a consumer making consumption and savings decisions so as to maximize his expected lifetime utility. Examples of continuous state dynamic decision models requiring dichotomous or binary choices include a nancial investor deciding when to exercise a put option, a capitalist deciding whether to enter or exit an industry, and a producer deciding whether to keep or replace a physical asset. A second class of discrete time continuous state dynamic model examined includes models of strategic gaming among a small number of individuals, rms, or institutions. Dynamic game models attempt to capture the behavior of a small group of dynamically optimizing agents when the policy pursued by one agent directly aects the welfare of another. Examples include a two national grain marketing boards deciding quantities of grain to sell on world markets and two individuals deciding how much to work and invest in the presence of an income risk-sharing arrangement. A third class of discrete time continuous state dynamic economic model 206

CHAPTER 8.

CONTINUOUS STATE MODELS: THEORY

207

examined includes partial and general equilibrium models of collective, decentralized economic behavior. Dynamic equilibrium models characterize the behavior of a market, economic sector, or entire economy through intertemporal arbitrage conditions that are enforced by the collective action of atomistic dynamically optimizing agents. Often the behavior of agents at a given date depends on their expectations of what will happen at a future date. If it is assumed that agent's expectations are consistent with the implications of the model as a whole, then agents are said to possess rational expectations. Examples of rational expectations models include arbitrage pricing models for nancial assets and physical commodities. Dynamic optimization and equilibrium models are closely related. The solution to a continuous state dynamic optimization may often be equivalently characterized by rst-order intertemporal equilibrium conditions obtained by dierentiating Bellman's equation. Conversely, many dynamic equilibrium problems can be \integrated" into equivalent optimization formulations. Whether cast in optimization or equilibrium form, most discrete time continuous state dynamic economic models pose in nite-dimensional xed-point problems that lack closed-form solution. This chapter provides an introduction to the theory of discrete time continuous state dynamic economic models. The subsequent chapter is devoted to numerical methods that may be used to solve and analyze such models.

8.1 Continuous State Dynamic Programming A discrete time, continuous state Markov decision model involves a reward function that depends on a state variable that may assume any one of an in nite number of values contained in a closed convex set. Such models may be classi ed according to whether the action space is also a continuum or whether it is a nite set. We treat these two classes of decision models separately, given that they may be analyzed and solved numerically in ways that are similar in some respects, but dissimilar in other respects. When discussing continuous state, discrete action Markov decision models, we limit our attention to models with binary choices, without signi cant loss of generality. The discrete time, continuous state and action Markov decision model has the following structure: In every period t, an agent observes the state of an economic process st , takes an action xt , and earns a reward f (st ; xt ) that

CHAPTER 8.


208

depends on both the state of the process and the action taken. The state space S 2 0:

CHAPTER 8.


220

Assuming u0 (0) = 1 and f (0) = 0, the solution to Bellman's equation will always be internal, and the shadow price function, which gives the shadow price of stock, satis es the Euler equilibrium conditions:

u0(s x) ÆE [( x + f (x)) ( + f 0 (x))] = 0 (s) = u0(s x):

Thus, along the optimal path,

u0t = ÆEt u0t+1 ( + t+1 ft0 )

where u0t is marginal utility and t+1 ft0 is the ex-post marginal product of capital. That is, on the margin, the utility derived from a unit of good today must equal the discounted expected utility derived from investing the good and consuming it and its product tomorrow. The certainty-equivalent steady-state is obtained by xing at its mean 1. The certainty-equivalent steady-state stock of good s , investment level x , and shadow price are characterized by the nonlinear equation system

u0(s

x ) = Æ ( + f 0 (x ))

= u0 (s

x )

s = x + f (x ): The certainty-equivalent steady-state conditions imply the golden rule: 1 + r = f 0 (x ). That is, in deterministic steady-state, the marginal product of capital equals the capital depreciation rate plus the interest rate. Totally dierentiating the equation system above with respect to the interest rate r: @s 1 + r = 00 < 0 @r f @x 1 = 0: @r f That is, a permanent rise in the interest rate will reduce the deterministic steady-state supply and investment, and will raise the shadow price.

CHAPTER 8.


221

8.4.5 Renewable Resource Problem A social planner wishes to maximize the discounted sum of net social surplus from harvesting a renewable resource over an in nite horizon. For year t, let st denote the resource stock at the beginning of the year, let xt denote the amount of the resource harvested, let ct = c(xt ) denote the total cost of harvesting, and let pt = p(xt ) denote the market clearing price. Growth in the stock level is given by st+1 = g (st xt ). What is the socially optimal harvest policy? This is an in nite horizon, deterministic model with time t 2 f0; 1; 2; : : :g measured in years. There is one state variable,

st = stock of resource at beginning of year t st 2 [0; 1); and one action variable,

xt = amount of resource harvested in year t, subject to the constraint 0 xt st : The reward earned by the optimizing agent is Z xt 0

p( ) d

c(xt ):

State transitions are governed by

st+1 = g (st

xt ):

The value function, which gives the net social value of resource stock, satis es Bellman's equation Z x

V (s) = 0max f xs

0

p( ) d

c(x) + ÆV (g (s x))g:

Assuming p(0) = 1 and g (0) = 0, the solution to the optimization problem embedded in Bellman's equation will be internal. Under these assumptions the shadow price function satis es the Euler conditions, which stipulate that for every stock level s > 0 there is a harvest level x such that

p(x) = c0 (x) + Æ(g (s x))g 0(s x)

CHAPTER 8.


222

(s) = Æ(g (s x))g 0 (s x): Thus, along the optimal path

pt = c0t + t t = Æt+1 gt0 where pt is the market price, c0t is the marginal harvest cost, and gt0 is the marginal future yield of stock in t. Thus, the market price of the harvested resource must cover both the marginal value of the unharvested resource and the marginal cost of harvesting it. Moreover, the value of one unit of resource today equals the discounted value of its yield tomorrow. The steady-state resource stock s , harvest x , and shadow price solve the equation system

p(x ) = c0 (x ) + Æ g 0 (s = Æ g 0(s s = g (s

x )

x )

x ):

These conditions imply g 0 (s x ) 1 = r. That is, in steady-state, the marginal yield equals the interest rate. Totally dierentiating the equation system above: @s 1 + r = 00 < 0 @r g

@x r = 00 < 0 @r g @ (c00 p0 )r = < 0: @r g 00 That is, as the interest rate rises, the steady-state stock, the steady-state harvest, and the steady-state shadow price all fall.

CHAPTER 8.


223

Figure 8.4.5 Steady-state optimal harvest of a renewable resource.

8.4.6 Nonrenewable Resource Problem A social planner wishes to maximize the discounted sum of net social surplus from harvesting a nonrenewable resource over an in nite horizon. For year t, let st denote the resource stock at the beginning of the year, let xt denote the amount of the resource harvested, let ct = c(xt ) denote the total cost of harvesting, and let pt = p(xt ) denote the market clearing price. What is the socially optimal harvest policy? This is an in nite horizon, deterministic model with time t 2 f0; 1; 2; : : :g measured in years. There is one state variable, st = stock of resource at beginning of year t st 2 [0; 1); and one action variable, xt = amount of resource harvested in year t, subject to the constraint 0 xt st : The reward earned by the optimizing agent is Z xt 0

p( ) d

c(xt ):

State transitions are governed by st+1 = st xt : The value function, which gives the net social value of resource stock, satis es Bellman's equation Z x

V (s) = 0max f xs

0

p( ) d

c(x) + ÆV (s x)g;

s 0:

Assuming p(0) = 1 and g (0) = 0, the solution to the optimization problem embedded in Bellman's equation will be internal. Under these assumptions, the shadow price function satis es the Euler conditions, which stipulate that for every stock level s > 0 there is a harvest level x such that p(x) = c0 (x) + Æ(s x)

CHAPTER 8.


224

Figure 8.4.6 Optimal harvest path of a norenewable resource.

(s) = Æ(s x): Thus, along the optimal path

pt = c0t + t t = Æt+1 where pt is the market price and c0t is the marginal harvest cost at t. That is, the market price of the harvested resource equals the marginal value of the unharvested resource plus the marginal cost of harvesting it. Also, the price of the harvested resource grows at the rate of interest. The steady-state, which occurs when stock is s = 0, is an uninteresting case.

8.4.7 Feedstock Problem An animal weighing s1 pounds in period t = 1 is to be fed up to period T +1, at which time it will be sold at a price of p dollars per pound. The cost of increasing the animal's weight by an amount xt during period t is given by c(st ; xt ) where st is the animal's weight at the beginning of t. What feeding strategy maximizes the present value of pro t? This is a nite horizon, deterministic model with time t 2 f1; 2; : : : ; T g measured in feeding periods. There is one state variable,

st = weight of animal at beginning of period t st 2 [0; 1); and one action variable,

xt = weight gain during period t; subject only to a nonnegativity constraint. The reward earned by the hog farmer in feeding periods is

c(st ; xt ): State transitions are governed by

st+1 = st + xt :

CHAPTER 8.


225

Figure 8.4.7 Feedstock problem dynamics. The value function, which gives the value of animal weighing pounds s in period t, satis es Bellman's equation

Vt (s) = max f c(s; x) + ÆVt+1 (s + x)g; x0 subject to the terminal condition

VT +1 (s) ps: The shadow price function, which measures the price of animal mass, satis es the Euler conditions, which stipulate that for each decision period t and weight level s > 0, the optimal weight gain x satis es the complementarity conditions

x0 Æt+1 (s + x) cx (s; x)le0 x > 0 =) Æt+1 (s + x) cx (s; x) = 0 t (s) = cs (s; x) + Æt+1 (s + x): For the post-terminal period,

T +1 (s) = p: Thus, along an optimal path, assuming an internal solution, we have:

Æt+1 = cx(st ; xt ) cs(st ; xt ) = t

Æt+1 :

In other words, the marginal cost of feeding the animal this period must equal the discounted value of the additional body mass obtained the following period. Also, the marginal value of body mass declines at the same rate at which it weight gains become increasingly more costly.

CHAPTER 8.


226

8.4.8 A Production-Adjustment Problem The output price faced by a competitive rm follows a rst-order autoregressive process:

j j < 1; t i.i.d.

pt+1 = + pt + t+1 ;

The cost of producing qt units in period t is c(qt ) plus an adjustment cost of a(qt qt 1 ). The rm cannot store the commodity because it is perishable. Assuming pt is known at the time the period t production decision is made, what production policy maximizes the sum of current and expected future pro ts? This is an in nite horizon, stochastic model with time t 2 f0; 1; 2; : : :g measured in years. There are two state variables:

qt 1 = past production; pt = current market price: There are one action variable:

qt = current production subject to a nonnegativity constraint. The reward earned by the optimizing agent is

pt qt

c(qt ) a(qt

qt 1 ):

Price state transitions are governed by

pt+1 = + pt + t+1 where

t = price process innovation in year t: The transition rule for qt is trivial. The value function, which gives the value of the rm, satis es Bellman's equation

V (q 1 ; p) = max fpq 0q

c(q ) a(q

q 1 ) + ÆEV (q; + p + )g:

CHAPTER 8.


227

Assuming a positive production level, the Euler conditions require that for every state vector (q 1 ; p), there is a production level q such that

p c0 (q ) a0 (q

q 1 ) + ÆEVq (q; + p + ) = 0

Vq (q 1 ; p) = a0 (q

q 1)

Along the optimal path,

pt = c0t + (a0t

ÆEa0t+1 ):

Thus, marginal revenue equals the marginal production cost plus the net marginal adjustment cost. The certainty-equivalent deterministic problem is obtained by assuming p is xed at its longrun mean =(1 ). If a0 (0) = 0, then the certaintyequivalent steady-state production is constant and implicitly de ned by the short-run pro t maximization condition:

p = c0 (q )

8.4.9 A Production-Inventory Problem The output price faced by a competitive rm follows a rst-order autoregressive process:

pt+1 = + pt + t+1 ;

j j < 1; t i.i.d.

The cost of producing qt units in period t is c(qt ). The rm may store across periods at a constant unit cost k. Assuming pt is known at the time the period t production-inventory decision is made, what production-inventory policy maximizes the sum of current and expected future pro ts? This is an in nite horizon, stochastic model with time t 2 f0; 1; 2; : : :g measured in years. There are two state variables:

bt = beginning inventories; pt = current market price: There are two action variables:

qt = current production

CHAPTER 8.


228

xt = ending inventories subject to the constraints

qt 0 xt 0 xt qt + bt ; that is, production, inventories, and deliveries must be nonnegative. The reward earned by the optimizing agent is

pt (qt + bt

xt ) c(qt ) kxt :

State transitions are governed by

pt+1 = + pt + t+1 where

t = price process innovation in year t and

bt+1 = xt : The value function, which gives the value of rm given inventories b and price p satis es Bellman's equation

V (b; p) =

max

fp(q + b x) c(q) kx + ÆEV (x; + p + )g:

0q;0xq+b

The shadow price function

(b; p) = Vb (b; p) = marginal value of inventories satis es the Euler conditions, which require that for every beginning inventory level b and price p, there is a production level q , ending inventory level x, and material balance shadow price such that

x0 ÆE(b; + p + ) p k

0

CHAPTER 8.


x > 0 =) ÆE(b; + p + ) p k

229

=0

q0 p c0 (q ) 0 q > 0 =) p c0 (q ) = 0 0 q+b x0 > 0 =) q + b x = 0 (b; p) = p Along the optimal path, if deliveries and storage are positive,

ÆEt pt+1

pt

k=0

pt = c0t : That is, marginal revenue equals the marginal production cost and the discounted expected future price equals the current output price plus the cost of storage. The certainty-equivalent deterministic problem is obtained by assuming p is xed at its longrun mean =(1 ). The certainty-equivalent steady-state inventories are 0 and production is constant and implicitly de ned by the short-run pro t maximization condition:

p = c0 (q ):

8.4.10 Optimal Growth with Debt Reconsider the optimal growth problem when the central planner can carry an external debt load dt whose unit cost 0 + 1 qt rises with the debt to asset ratio qt = dt =st . This is an in nite horizon, stochastic model with time t 2 f0; 1; 2; : : :g measured in years. There are two state variables:

st = stock of good at beginning of year t st 2 [0; 1)

CHAPTER 8.


230

and

dt = debt load at beginning of year t dt 2 ( 1; 1): Here, dt < 0 implies that the economy runs a surplus. There are two action variables:

xt = amount of good invested in year t ct = amount of good consumed in year t; both subject to nonnegativity constraints. The reward earned by the optimizing agent is

u(c) = social utility in t: Supply state transitions are governed by

st+1 = xt + t+1 f (xt ) where

t = productivity shock in year t: Debt state transitions are governed by

dt+1 = dt + bt ; where

bt = ct + xt + (0 + 1 dt =st ) dt

st ;

indicates net borrowing in year t. The value function, which gives the sum of current and expected future social welfare, satis es Bellman's equation

V (s; d) = xmax fu(c) + ÆEV ( x + f (x); d + b)g 0;c0

where b = x + c + (0 + 1 d=s) d s) is net borrowing. Assuming u0 (0) = 1 and f (0) = 0, the solution to Bellman's equation will always be internal, and the shadow price and cost functions @V (s; d) = (s; d) = shadow price of stock @s

CHAPTER 8.


231

and

@V (s; d) = shadow cost of debt @d satisfy the Euler equilibrium conditions, which stipulate that for every stock level s > 0 and debt level d, (s; d) =

u0(c) + ÆE( x + f (x); d + b) = 0 ÆE [( x + f (x); d + b) ( + f 0 (x))] + ÆE( x + f (x); d + b) = 0 (s; d) = ÆE [( x + f (x); d + b) (1 + 1 q 2 )] (s; d) = ÆE [( x + f (x); d + b) (1 + 0 + 21 q )] where q = d=s is the debt to asset ratio. The certainty-equivalent steady-state is obtained by assuming = 1 with probability 1. The certainty-equivalent steady-state stock of good s , debt load d , debt-asset ratio q = d =s , investment level x , consumption level c , stock shadow price , and debt shadow cost solve the equation system

u0(c) + Æ = 0 Æ ( + f 0 (x )) + Æ = 0 = Æ (1 + 1 q 2 ) = Æ (1 + 0 + 21 q ) s = x + f (x ) s = x + c + (0 + 1 q )d q = d =s : These conditions imply a steady-state optimal debt load q = (r 0 )=(21 ), which increases with the discount rate r but falls with the base cost of debt 0 .

CHAPTER 8.


232

8.5 Rational Expectations Models By de nition, agents in rational expectations models take into account how their actions will aect them in the future and form expectations that coincide with those implied by the model as a whole. Most discrete time rational expectation models take the following form: At the beginning of period t, an economic system emerges in a state st . The agents in the economic system observe the state of the system and, by pursuing their individual objectives, formulate a collective behavioral response xt . The economic system then evolves to a new state st+1 that depends on the current state st and response xt , and an exogenous random shock t+1 that is realized only after the agents respond at time t. More formally, the behavioral responses of economic agents and the state transitions of the economic system are governed by a structural law of the form

f (st ; xt ; Et xt+1 ) = 0; and the dynamic law

st+1 = g (st; xt ; t+1 ): The stipulation that only the expectation of the subsequent period's behavioral response is relevant to the current response of agents is more general than rst appears. By introducing new accounting variables, the current response can be made to depend on the expectation of any function of future states and responses, including states and responses more than one period into the future. The state space S 2 0, > 0, > 0, and > 0 are given. Formulate the rm's problem of maximizing the present value of net revenue over an in nite time horizon. Speci cally: (a) Set up the decision problem (de ne states, actions, reward function, transition rule). (b) Formulate the value function and Bellman's recursive functional equation. (c) Assuming an internal solution, derive and interpret the rst order conditions for optimality. What does the derivative of the value function represent? (d) What eect does an increase in the base tax rate, , have on output in the long run. (e) What eect does an increase in the discount factor, Æ , have on output in the long run. 7. Consider the Optimal Growth example in this Chapter. Find and sign @s , @x , and @ . @ @ @

CHAPTER 8.


247

8. Consider the Optimal Growth with Debt example in this Chapter. Find the golden rule of growth with debt. Also, perform comparative statics analysis with respect to the steady state. 9. Consider the Renewable Resource example in this Chapter. However, now assume that the renewable resource is entirely owned by a pro tmaximizing monopolist. Will the steady-state harvest and stock levels be greater for the monopolist or for the social planner? Give conditions under which a \regular" steady-state will exist. What if these conditions are not satis ed? 10. Hogs breed at a rate . That is, if a farmer breeds xt hogs during period t, there will be (1 + )xt hogs at the beginning of period t + 1. At the beginning of any period, hogs can be marketed for a pro t p per hog. Only the hogs not sent to market at the beginning of the period are available for breeding during the period. A farmer has H hogs at the beginning of period 0. Find the hog marketing strategy that maximizes the present value of pro ts over a T -period horizon. 11. A rm has a contractual obligation to deliver Q units of its product to a buyer rm at the beginning of period T ; that is, letting xt denote inventories on hand at the beginning of period t, the rm must produce suÆcient quantities in periods 0; 1; 2; : : : ; T 1 so as to ensure that xT Q. The cost of producing qt units in period t is given by c(qt ), where c0 > 0. The unit cost of storage is k dollars per period; due to spoilage, a proportion of inventories held at the beginning of one period do not survive to the following period. The rm's initial inventories are x0 where 0 < x0 < Q. The rm wishes to minimize the present value of the cost of meeting its contractual obligation; assume a discount factor Æ < 1. (a) Identify the state and decision variables, the payo function, and the equation of motion associated with this problem. (b) Write Bellman's recursive equation. What does the value function represent? (c) Derive the rst order conditions for optimality and interpret them. What does the derivative of value function represent? (d) Assuming increasing marginal cost, c00 > 0, qualitatively describe the optimal production plan.

CHAPTER 8.


248

(e) Assuming decreasing marginal cost, c00 < 0, qualitatively describe the optimal production plan. 12. A subsistence farmer grows and eats a single crop. Production, yt , depends on how much seed is on hand at the beginning of the year, kt , according to

yt = kt where 0 < < 1. The amount kept for next year's seed is the dierence between the amount produced and the amount consumed, ct :

kt+1 = yt

ct :

The farmer has a time-additive logarithmic utility function and seeks to maximize T X t=0

Æ t ln(ct ):

subject to having an initial stock of seed, k0 . What is the farmer's optimal consumption-investment policy? (a) Set up the decision problem (de ne states, decisions, objective function, transition equation). (b) Formulate the value function and Bellman's recursive functional equation. (c) Derive and interpret the rst order conditions for optimality. (d) Show that the value function is time invariant and has the form V (kt ) = A + B ln(kt ) and that the optimal decision rule for this problem is kt+1 = Cyt ; nd the values for A, B , and C .

CHAPTER 8.


249

13. A rm competes in a mature industry whose total pro t is a xed amount X every year. If the rm captures a fraction pt of total industry sales in year t, it makes a pro t pt X . The fraction of sales captured by the rm in year t is a function pt = f (pt 1 ; at 1 ) of the fraction it captured the preceding year and its advertising expenditures the preceding year, at 1 . Find the advertising policy that maximizes the rm's discounted pro ts over a xed time horizon of T years. Assume p0 and a0 are known. (a) Set up the decision problem (de ne states, decisions, objective function, transition equation). (b) Formulate the value function and Bellman's recursive functional equation. (c) Derive and interpret the rst order conditions for optimality. (d) Assuming an in nite horizon, what conditions characterize the steady-state optimal solution? 14. A corn producer's net per-acre revenue in year t is given by

ct = pt yt

ct xt

wt lt

where pt is the unit price of corn ($/bu.), yt is the corn yield (bu./acre), ct is the unit cost of fertilizer ($/lb.), xt is the amount of fertilizer applied (lbs./acre), wt is the wage rate ($/man-hour), and lt is the amount of labor employed (man-hours/acre). The per-acre crop yield in year t is a function

yt = f (lt ; xt ; st ) of the amount of labor employed and fertilizer applied in year t and the level of fertilizer carryin st from the preceding year. Fertilizer carryout in year t is a function

st+1 = f (xt ; st ) of the amount of fertilizer applied and the level of fertilizer carryin in year t. Assume that future corn prices, fertilizer costs, and wage rates are known with certainty. The corn producer wishes to maximize the expected present value of net revenues over a nite horizon of T years. Formulate the producer's optimization problem. Speci cally,

CHAPTER 8.


250

(a) Set up the decision problem (de ne states, decisions, objective function, transition equation). (b) Formulate the value function and Bellman's recursive functional equation. (c) Derive and interpret the rst order conditions for optimality. (d) Assuming an in nite horizon, what conditions characterize the steady-state optimal solution? 15. The role of commodity storage in intertemporal allocation has often been controversial. In particular, the following claims have often been made:

Competitive storers, in search of speculative pro ts, tend to hoard a commodity|that is, they collectively store more than is socially optimal. A monopolistic storer tends to dump a commodity at rst in order to extract monopoly rents in the future|that is, he/she stores less than is socially optimal.

Explore these two propositions in the context of a simple intraseasonal storage model in which a given amount Q of a commodity is to be allocated between two periods. Consumer demand is given by pi = a qi for periods i = 1; 2, and the unit cost of storage between periods is k. There is no new production in period 2, so q1 + q2 = Q. Speci cally, answer each of the following: (a) Determine the amount stored under the assumption that there are a large number of competitive storers. (b) Determine the amount stored under the assumption that there is a single pro t- maximizing storer who owns the entire supply Q at the beginning of period 1. (c) Taking expected total consumer surplus less storage costs as a measure of societal welfare, determine the socially optimal level of storage. Address the two comments above. (d) Consider an Economist who rejects net total surplus as a measure of social welfare. Why might he/she still wish to nd the level of storage that maximizes total surplus?

CHAPTER 8.


251

To simplify the analysis, assume that the discount factor is 1 and that the storer(s) are risk neutral and possess perfect price foresight. 16. Consider the problem of maximizing the present value of social welfare for an aggregate economy consisting of single composite good. Each year t begins with a predetermined amount of the good st , of which an amount ct is consumed and the remainder xt is retained as capital. The social welfare derived from consumption in year t is u(ct ) where u is the aggregate utility function. The amount of good available in year t + 1 is st+1 = f (xt ) where f is the aggregate production function. The utility and production functions exhibit standard curvature properties and the discount rate r is positive. (a) How will an increase in the interest rate r aect the long-run levels of consumption and capital stock? Use analytic mathematical methods to structure your argument. (b) Suppose now that st+1 = yt+1 f (xt ) where yt is a positive i.i.d. production shock. How must the question in (a) be modi ed to remain meaningful in a stochastic setting? What techniques would you use to assess, say, whether an increase in the interest rate would raise or lower the long-run variability of consumption. 17. Consider an industry of identical price taking rms. For the representative rm, let st denote beginning capital stock, let xt denote newly purchased capital stock, let qt = f (st + xt ) denote production, let k denote the unit cost of new capital, and let > 0 denote the survival rate of capital. Furthermore, let pt = p(qt ) be the market clearing price. Find the perfect foresight competitive equilibrium for this industry. 18. Water from a dam can be used for either irrigation or recreation. Irrigation during the spring bene ts farmers, but reduces the dam's water level during the summer, damaging recreational users. Speci cally, if st is the stock of water in the dam at the beginning of year t and an amount xt is released for irrigation, farmer bene ts in year t will be f (xt ) and recreational user bene ts will be u(st xt ). Water levels are replenished during the winter months by i.i.d. random rainfalls t , giving rise to the water stock transition relationship st+1 = st xt + t+1 . As a social planner, you wish to nd the irrigation policy that max-

CHAPTER 8.


252

imizes the expected discounted sum of farmer and recreational user bene ts over an in nite time horizon. (a) Formulate and interpret Bellman's equation. (b) Assuming an internal solution, derive and interpret the Euler conditions. (c) What conditions characterize the certainty equivalent steady-state?

Chapter 9 Discrete Time Continuous State Dynamic Models: Methods This chapter discusses numerical methods for solving discrete time continuous state dynamic economic models, with emphasis on Markov decision and rational expectations models. Continuous state dynamic economic models give rise to functional equations whose unknowns are entire functions de ned on an interval of Euclidean space. For example, the unknown of a Bellman equation

V (s) = max ff (s; x) + ÆE V (g (s; x; ))g; x2X (s)

s 2 S;

is the value function V (). The unknown of the Euler conditions

fx (s; x(s)) + ÆE [(g (s; x(s); )) gx (s; x(s); )] = 0; fs (s; x(s)) + ÆE [(g (s; x(s); )) gs(s; x(s); )] = (s);

s 2 S; s 2 S;

are the shadow price and policy functions () and x(). And the unknown of a rational expectations intertemporal equilibrium condition

f (s; x(s); Ex(g (s; x(s); )));

s 2 S;

is the response function x(). 253

CHAPTER 9.

CONTINUOUS STATE MODELS: METHODS

254

In most applications, the functional equations that arise in dynamic economics lack analytic closed form solution and can only be solved approximately using computational methods. A variety of methods are available for computing approximate solutions to these equations. Linear-quadratic approximation and space discretization historically have been popular among economists. However, in most applications, these methods either provide unacceptably poor approximations or are computationally ineÆcient. Only recently have economists begun to employ Galerkin techniques, which have been used among computational physical scientists for decades. Among the various versions of the Galerkin technique, the collocation method is clearly the most useful for solving dynamic models in Economics and Finance. The collocation method is exible, accurate, and numerically eÆcient and can be developed directly from basic numerical integration, approximation, and root nding methods. Collocation methods may be used to solve discrete and continuous choice Markov decision models and rational expectations models. Bounds and general constraints on variables can also be handled using the method. The collocation method employs the following general strategy for solving a functional equation:

Approximate the unknown function with a nite linear combination of n known basis functions whose coeÆcients are to be determined.

Require the approximant to satisfy the underlying functional equation at n prescribed points of the domain, called the collocation nodes.

The collocation strategy eectively replaces the functional equation with a nite-dimensional nonlinear equation problem that can be solved using basic nonlinear equation techniques. If the basis functions and nodes are chosen wisely, the collocation method will be numerically consistent; that is, the approximation error can be made arbitrarily small by increasing the number of basis functions and nodes. The collocation method is a solution strategy rather than a speci c technique. When applying the collocation method, the analyst still faces a number of computational modeling decisions. For example, the analyst must choose the basis function and collocation nodes. Numerical approximation theory oers guidance here, suggesting a Chebychev polynomial basis coupled with Chebychev collocation nodes, or a spline basis coupled with equally

CHAPTER 9.


255

spaced nodes will often be good choices. Also, the analyst must chose an algorithm for solving the resulting nonlinear equation. Standard choices include Newton, quasi-Newton, and function iteration methods. A careful analyst will often try a variety of basis-node combinations, and may employ more than one iterative scheme in order to assure the robustness of the results. Although the collocation method is general in its applicability, the details of implementation vary with the functional equation being solved. Below, the collocation method is developed in greater detail for Bellman equations, Euler conditions, and rational expectations equilibrium conditions.

9.1 Traditional Solution Methods Before discussing collocation methods for continuous state Markov decision models in greater detail, let us brie y examine the two numerical techniques that historically have been popular among economists for computing approximate solutions to such models: space discretization and linear-quadratic approximation. Space discretization calls for the continuous state Markov decision problem is to be replaced with a discrete state discrete action Markov decision problem that closely resembles it. To \discretize" the state space of a continuous state Markov decision problem, one partitions the state and action spaces S into nitely many regions, S1 ; S2 ; : : : ; Sn. If the action space X is also continuous, it too is partitioned into nitely many region s X1 ; X2 ; : : : ; Xm . Once the space and action spaces have been partitioned, the analyst selects representative elements, si 2 Si and xj 2 Xj , from each region. These elements serve as the state and action spaces of the approximating discrete Markov decision problem. The transition probabilities of the discrete problem are computed by integrating with respect to the density of the random shock:

P (si0 jsi; xj ) = Pr[g (si ; xj ; ) 2 Si0 ]: If the model is deterministic, then the state is assumed to migrate from state si to si0 when decision xj is taken, if g (si; xj ) 2 Si0 . When the state and action spaces are intervals, say, S = [smin ; smax ] and X = [xmin ; xmax ], it is often easiest to partition the spaces so that the nodes are equally-spaced and the rst and nal nodes correspond to the endpoints of the intervals. Speci cally, we set si = smin + (i 1)ws and xj = xmin + (j 1)wx, for i = 0; 1; : : : ; n and j = 0; 1; : : : ; m, where

CHAPTER 9.


256

ws = (smax smin )=(n 1) and wx = (xmax xmin )=(m 1). If the model is stochastic, the transition probabilities of the approximating discrete state decision model are given by P (si0 jsi; xj ) = Pr[si0

ws=2 g (si; xj ; ) si0 + ws =2]:

If the model is deterministic, the state is assumed to migrate from state si to si0 when decision xj is taken, where si0 is the state element nearest g (si; xj ). Another popular method for solving dynamic optimization models is linear-quadratic approximation. Linear-quadratic approximation calls for the transition function g and objective function f to be replaced with linear and quadratic approximants, respectively. Linear-quadratic approximation is motivated by the fact that an unconstrained Markov decision problem with linear transition and quadratic objective has a closed-form solution that is relatively easy to derive. Typically, the linear and quadratic approximants are constructed by forming the rst- and second-order Taylor expansions around the certainty-equivalent steady-state. When passing to the linearquadratic approximation, all inequality constraints of the original problem, including nonnegativity constraints, must be discarded. The rst step in deriving an approximate solution to a continuous state Markov decision problem via linear-quadratic approximation is to compute the certainty-equivalent steady-state. If denotes the mean of , the certaintyequivalent steady-state state s , optimal action x , and shadow price are characterized by the nonlinear equation system:

fx (s ; x ) + Æ gx(s ; x ; ) = 0 = fs (s ; x ) + Æ gs (s ; x ; ) s = g (s ; x ; ): The nonlinear equation may be solved for the steady-state values of s , x , and using standard nonlinear equation methods. The second step is to solve the linear-quadratic control problem whose transition function g^ and objective function f^, are the rst- and second-order Taylor series approximants of g and f , respectively: f (s; x) f^(s; x) = f + fs (s s ) + fx (x x ) + 0:5(s s )0 fss (s s ) (x x ) + 0:5(x x )0 f (x x ) +(s s )0 fsx xx

CHAPTER 9.

g (s; x; )


257

g^(s; x) = g + gs(s s ) + gx(x x ):

Here, f and g are the values of f and g ; fs , fx , gs , and gx are the rst , and f are the second partial partial derivatives of f and g ; and fss , fsx xx derivatives of f ; all evaluated at the certainty-equivalent steady-state. The shadow price and optimal policy functions for the linear-quadratic control problem will be linear. Speci cally:

(s) = + s (s s ) x(s) = x + Xs (s s ) where 0

0

0

0

0

][Æg g + f ] 1 [Æg g + f ] [Ægs sgx + fsx x s x xx x s s sx 0 +Ægs s gs + fss 0 0 ] 1 [Æg 0 g + f 0 ] Xs = [Ægx sgx + fxx x s s sx

s =

The rst of these two conditions characterizes the slope s of the approximate shadow price function as a xed-point of a nonlinear map. The slope can be computed using by either function iteration, typically with initial guess s = 0, or by applying the quadratic formula, if the problem is one dimensional. Given the slope s, the slope Xs of the approximate optimal policy function may be directly computed from the second condition. = If the problem has one dimensional state and action spaces, and if fss fxx fsx2 , a condition often encountered in economic problems, then the slope of the shadow price function may be computed analytically as follows: s = [fss gx2

g g + f g 2 2fss fxx s x xx s

=Æ ]=g 2 fxx x

9.2 Bellman Equation Collocation Methods Consider Bellman's equation for an in nite horizon discrete time continuous state dynamic decision problem:

V (s) = max ff (s; x) + ÆE V (g (s; x; ))g x2X (s)

s 2 S:

CHAPTER 9.

258


To compute an approximate solution to Bellman's equation via collocation, one employs the following strategy: First, one approximates the unknown value function V using a linear combination of known basis functions 1 ; 2 ; : : : ; n whose coeÆcients c1 ; c2 ; : : : ; cn are to be determined:

V (s)

n X j =1

cj j (s):

Second, the basis function coeÆcients c1 ; c2 ; : : : ; cn are xed by requiring the approximant to satisfy Bellman's equation, not at all possible states, but rather at n states s1 ; s2 ; : : : ; sn, called the collocation nodes. Many collocation basis-node schemes are available to the analyst, including Chebychev polynomials and nodes, and spline functions and uniform nodes. The best choice of basis-node scheme is application speci c, and typically depends on the curvature properties of the value function. The collocation strategy replaces the Bellman functional equation with a system of n nonlinear equations in n unknowns. Speci cally, to compute the approximate solution to the Bellman equation, or more precisely, to compute the n basis coeÆcients c1 ; c2 ; : : : ; cn in the basis representation of the value function approximant, one solves the equation system X j

cj j (si ) = max ff (si ; x) + ÆE x2X (si )

n X j =1

cj j (g (s; x; ))g

i = 1; 2; : : : ; n;

which may be compactly expressed in vector form as the collocation equation: c = v (c): Here, , the collocation matrix, is the n by n matrix whose typical ij th element is the j th basis function evaluated at the ith collocation node ij = j (si ) and v , the conditional value function, is a function from typical ith element is

vi (c) = max ff (si ; x) + ÆE x2X (si )

n X j =1

cj j (g (si ; x; ))g:

0 the process is mean reverting, with a long run tendency to return to the value . In the limit as ! 1 the process

uctuates randomly around . Second moments can be found by combining this approach with Ito's Lemma. Consider the special case of (10.3) with a constant variance term

dx = ( x)dt + dz: To determine the variance of this process note that

V ar (xt ) = E x2

(E xt )2

. Using Ito's Lemma

dx2 = [2( x)x + 2 ]dt + 2xdz: The time derivative of the expectation of x2 is therefore

dEx2 = 2 2Ex2 + 2Ex: dt The time derivative of the square of Ex is d(Ex)2 = 2e (t ) (x )Ex: dt Combining these expressions yields dV ar (xt ) = 2 2Ex2 + 2Ex + 2e dt = 2 2(Ex2 (Ex)2 )) = 2 2V ar (xt ):

(t ) (x

)Ex

CHAPTER 10.

292

CONTINUOUS TIME MATHEMATICS

This is an ordinary dierential equation with the boundary condition that V ar (x ) = 0, which is solved by 1 e 2(t ) 2 : 2 In the limit as ! 0 this expression yields the familiar result that the conditional variance grows linearly in time: 1 e 2(t ) = t : lim !0 2 On the other hand, the limit as ! 1 demonstrates that the process becomes degenerate for large , with the probability distribution being concentrated at the point . The long-run distribution of the process is found by letting t ! 1), thereby demonstrating that the process has a long-run mean and variance (, 2 =2 ).

V ar (xt ) =

10.1.2 The Feynman-Kac Equation Control theory in continuous time is typically concerned with problems which attempt to choose a control that maximizes a discounted return stream over time. It will prove useful, therefore, to have an idea of how to evaluate such a return stream for an arbitrary control. Consider the value

V (St ; t) = Et

Z T t

e

( t) f (S

)d

+e

(T t) R(S ) T

;

where

dS = (S )dt + (S )dz: An important theorem, generally known in economics as the Feynman-Kac Equation, but also known as Dynkin's Formula, states that V (S ) is the solution to the following partial dierential equation4 V (S; t) = f (S ) + Vt (S; t) + (S )VS (S; t) + 21 2 (S )VSS (S; t);

4 The partial dierential equation of this theorem has a linear parabolic form. Parabolic

PDEs are ones that can be expressed in terms of the rst time derivative and the second (and possibly lower) space derivatives. The term comes from the equation for a parabola y = a + bx + cx2 , substituting dt for y and dx for x. Other common forms of second order PDEs are hyperbolic and elliptic, both of which involve second order derivatives in both space and time.

CHAPTER 10.


293

with V (S; T ) = R(S ). The function R here represents a terminal value of the state, i.e., a salvage value.5 By applying Ito's Lemma, the Feynman-Kac Equation can be expressed as:

V (S; t) = f (S ) + E [dV ]=dt:

(10.4)

(10.4) has a natural economic interpretation. Notice that V can be thought of as the value of an asset that generates a stream of payments f (S ). The rate of return on the asset, V , is composed of two parts, f (S ), the current income ow and E [dV ]=dt, the expected rate of appreciation of the asset. Alternative names for the components are the dividend ow rate and the expected rate of capital gains. A version of the theorem applicable to in nite horizon problems states that

V (St ) = Et

Z 1 t

e

f (S )d

;

is the solution to the dierential equation

V (S ) = f (S ) + (S )VS (S ) + 21 2 (S )VSS (S ): Although more general versions of the theorem exist (for example see DuÆe for a version with a state dependent discount rate), these will suÆce for our purposes. As with any dierential equation, boundary conditions are needed to completely specify the solution. In this case, we require that the solution to the dierential equation be consistent with the present value representation as S approaches its boundaries (often 0 and 1 in economic problems). Generally economic intuition about the nature of the problem is used to determine the boundary conditions; we will discuss this issue more presently.

Example: Geometric Brownian Motion

Geometric Brownian motion is a particularly convenient stochastic process because it is relatively easy to compute expected values of reward streams. If S is governed by

dS = Sdt + Sdz;

5 The terminal time T need not be xed, but could be a state dependent. Such an

interpretation will be used in the discussion of optimal stopping problems (Section 11.2.3).

CHAPTER 10.


294

the expected present value of a reward stream f (S ) is the solution to

V = f (S ) + SVS + 21 2 S 2 VSS : As this is a linear second order dierential equation, the solution can be written as the sum of the solution to the homogeneous problem (f (S ) = 0) and any particular solution that solves the non-homogeneous problem. The homogeneous problem is solved by

V (S ) = A1 S + A2 S ; 1

2

where the i are the roots of the quadratic equation 1 2 2 (

1) +

=0

and the Ai are constants to be determined by boundary conditions. For positive , one of these roots is greater than one, the other is negative: 1 > 1, 2 < 0. Consider the problem of nding discounted expected values of powers of S , i.e., where f (S ) = S (assuming, momentarily, that it exists). It is easily veri ed that a particular solution is

V (S ) = S =(

1 2 2 (

1)):

(10.5)

All that remains, therefore, is to determine the value of the arbitrary constants A1 and A2 that ensure the solution indeed equals the expected value of the reward stream. This is a bit tricky because it need not be the case that the expectation exist (the integral may not converge as its upper limit of integration goes to 1). It can be shown, however, that the present value is well de ned for 2 < < 1 , making the numerator in (10.5) positive. Furthermore, the boundary conditions require that A1 = A2 = 0. Thus the particular solution is convenient in that it has a nice economic interpretation as the present value of a stream of returns.

10.1.3 Arbitrage Based Asset Valuation An important use of continuous time methods results from powerful arbitrage conditions that can be derived in a simple and elegant fashion. Originally developed by Fisher Black and Myron Scholes, as well as by Robert Merton, to solve option pricing problems, arbitrage arguments apply much more broadly. Any assets that are based on the same underlying risks have values

CHAPTER 10.

295


that are related to one another in very speci c ways. Although this clearly applies to nancial derivatives such as options, it also applies more generally. Consider two assets which have values V and W , both of which depend on the same random process S . Suppose that S is an Ito process, with6

dS = S dt + S dz: Under suitable regularity conditions, this implies that V and W are also Ito processes, with

dV = V dt + V dz dW = W dt + W dz: Suppose further that the assets generate income streams (dividends), which are denoted by ÆV and ÆW . One can create a portfolio consisting of one unit of V and n units of W , the value of which is described by

dV + ndW = [V + nW ]dt + [V + nW ]dz: This portfolio can be made risk free by the appropriate choice of n, speci cally by setting the dz term to 0:

n = V =W : Because it is risk-free the portfolio must earn the risk-free rate of return. Therefore the capital appreciation on the portfolio plus its income stream must equal the risk free rate times the investment cost:

V

V dt + ÆV W W

V Æ dt = r V W W

Divide by V dt and rearrange to conclude that V + ÆV rV W + ÆW rW = = (S; t): V W

V W dt W

6 The following notational conventions are used. , and Æ represent drift, diusion and payouts associated with random processes; subscripts on these variables identify the process. V and W represent asset values, which are functions of the underlying state variables and time; subscripts refer to partial derivatives.

CHAPTER 10.


296

In other words, there is a function, , which depends on S and t, that is common to all assets whose values depend on S . can be interpreted as the market price of the risk in S . To avoid arbitrage opportunities, any asset with value V that depends on S must satisfy

V + ÆV = rV + V This is a fundamental arbitrage condition that is interpreted as saying that the total return on V , V + ÆV , equals risk free return plus a risk adjustment, rV + V . Ito's Lemma provides a way to evaluate the V and V terms. Speci cally,

V = Vt + S VS + 21 S2 VSS and

V = S VS : Combining with the arbitrage condition and rearranging yields

rV = ÆV + Vt + (S

S )VS + 21 S2 VSS :

(10.6)

This is the fundamental dierential equation that any asset derived from S must satisfy, in the sense that it must be satis es by any frictionless economy in equilibrium. It is worth exploring the market price of risk function, , more carefully. is the market price of risk in S and therefore does not depend on the speci c terms of any asset derived from S . This arbitrage framework is consistent, but more general than, the any speci c market equilibrium such as the Capital Asset Pricing Model (CAPM). In the CAPM all assets have expected excess return (over the risk free rate) that is proportional to the expected excess return on the so-called market portfolio. The factor of proportionality, called the beta, is equal to the covariance of the excess returns on the asset and the market portfolio divided by the covariance of the market excess return: 2 : S = SM =M

Thus in the CAPM the following relationship holds

S + ÆS

rS = S (M

rM ):

CHAPTER 10.


297

(this assumes that the market portfolio is payout protected so ÆM = 0). If we de ne the market price of risk on the market portfolio to be M = (M rM )=M , then in the CAPM the market price of risk on S will equal

S = SM M ;

where SM SM =S M is the correlation between S and M . Thus, in the CAPM, there is a single market price of risk, M , and the market price of any speci c risk is M times the correlation between the speci c risk and the market risk. It is important to note that, in general, S may or may not be the price of a traded asset. If it is the price of a traded asset then the arbitrage condition applies to S itself, so

S

S = rS

ÆS :

Furthermore, the value of any asset, V , which is derived from S , satis es the partial dierential equation

rV = ÆV + Vt + (rS

ÆS )VS + 12 S2 VSS :

On the other hand, if S is not the price of a traded asset, but there is a traded asset or portfolio, W , that depends only on S , then the market price of risk, , can be inferred from the behavior of W : +Æ rW ; (S; t) = W W W where ÆW is the dividend ow acquired by holding W .

Example: Black-Scholes Formula

Consider a non-dividend paying (or payout protected) stock (Æ = 0), the price of which follows

dS = Sdt + Sdz; where and are constants, so S follows a geometric Brownian motion (sometimes denoted dS=S = dt + dz ). The log dierences, ln(S (t + t)) ln(S (t)), are normally distributed with mean ( 12 2 )t and variance 2 t. A derivative asset is de ned such that its value, V (S; t), is a function of the state variable S implying that

rV = Vt + rSVS + 21 2 S 2 VSS :

CHAPTER 10.


298

Suppose that the boundary condition is that V (S; T ) = max(0; S K ), for some constant K and time T . This is the boundary condition for a European call option on S with a strike price of K . A call option has a payout at time T of S K if S > K and 0 otherwise. It can be shown that

V (S; t) = S (d) e where = T

d=

p

r K

d

t,

ln(S=K ) + r 1 p p + 2 ;

and is the standard normal CDF: Z x 1 1 (x) = p e 2 z dz: 2 1 Some tedious algebra will demonstrate that 2

VS = (d) ; VSS =

(d) p S

and

S (d) p Vt = 2 where

re

r K

d

p

; 2

1 x2

e 2 (x) = 0 (x) = p

2 (in the industry these are known as the delta, gamma and theta of the call option and are used in hedging portfolios of stocks). Using these expressions it is straightforward to verify that the partial dierential equation above, including the boundary condition, is satis ed.

CHAPTER 10.


299

10.2 Probability Distributions for Ito Processes 10.2.1 Transition Distributions Obtaining a policy function (optimal control feedback function) is only one of the goals of dynamic analysis. a topic taken up in the next chapter. Another important goal is to characterize the probability distributions of the state and control variables. For this we will need to use some results from the probability theory of stochastic processes. In particular, we will use the Kolmogorov forward equation, which describes the evolution of the probability distribution as it moves forward in time from some initial condition. This gives the transition density for any time horizon. By letting the time horizon go to in nity (assuming the process is stationary) we will obtain the long-run transition density. Consider an arbitrary Ito process

dS = (S; t)dt + (S; t)dz: We are interested in knowing

P rob[ST

bjSt ] =

Z b

1

f (ST ; T ; St ; t)dS:

Thus f represents the time t probability density function associated with S at time T , given the time t value of S is St . The Kolmogorov forward equation is a partial dierential equation that the transition probability density function must satisfy:

@f (S; T ) @(S; T )f (S; T ) 1 @ 2 2 (S; T )f (S; T ) + 2 @T @S @S 2 2 = fT + (S S SS )f + ( S )fS 12 2 fSS :

0 =

From the de nition of the transition density function, f must have a degenerate distribution at T = t, i.e.,

f (St ; t; St ; t) = Æ (St ); where Æ () is the Dirac function which concentrates all probability mass on a single point. It is often the case, however, that one wants only the long-run stationary distribution (assuming it exists), in which case the fT term equals zero and the Kolmogorov equation reduces to a dierential equation in S .

CHAPTER 10.

300


To illustrate the use of the Kolmogorov equation, consider processes that have Gaussian (normal) transition densities with mean M (T ) and variance V 2 (T ). The Gaussian density is 1 f (S; T ) = p exp 2V (T )

(S

M (T ))2 : 2V 2 (T )

with associated the partial derivatives: S M fS = f V2 1 (S M )2 1 f fSS = 2 V V2 0 (S M )2 V S M 0 fT = 1 + M f: V2 V V2 Substitute these de nitions into the Kolmogorov equation and rearrange terms: 0 (S M )2 V (S; T )2 S M 0 1 (M (S; T ))+S (S; T ) = 0: (10.7) V2 V 2S 2 V2 (10.7) can be satis ed when the drift term is linear in S and the variance term is constant in S :

(S; t) = 0 (t) + 1 (t)S (S; t) = (t): Substituting these expressions into (10.7) yields

(S

0 V

M )2

1

V2

V

2 (T )

2V 2

0 @

(S

M) S V2

M 0 0 (T ) 1 (T )

1

1A 1 (T ) = 0:

If M satis es M = (M 0 0 (T ))=1 (T ), or equivalently, M 0 = 1 (T )M + 0 (T ), this simpli es to

(S

M )2

V2

1

0 V

V

2 (T ) 2V 2

1 (T ) = 0:

CHAPTER 10.


301

Notice that the second term in () does not involve S ; setting it equal to zero and rearranging yields 2V V 0 V 2 0 = 2 (T ) + 21 (T )V 2 :

The Kolmogorov equation will therefore be satis ed if M (t) and V 2 (t) are selected to satisfy the linear dierential equations

M 0 = 1 (T )M + 0 (T ) V 2 0 = 21 (T )V 2 + 2 (T );

with boundary conditions that M (t) = St and V 2 (t) = 0. It can be readily veri ed (see Appendix ??) that the solutions to these dierential equations are

M (T ) = V 2 (T )

=

RT t

( )0 ( )d + (t)St (T )

RT 2 2 t ( ) ( )d ; 2 (T )

where

(T ) = exp

Z T

1 ( )d :

10.2.2 Long-Run (Steady-State) Distributions It is often not possible to nd a closed form solution for the transition density. When the drift and diusion terms are not functions of t, however, there may be a long-run density that represents the limit of the transition density as T ! 1. We say \may be" because many Ito processes, including absolute and geometric Brownian motion, do not have long-run densities. For example, the mean, variance or other moments of the transition densities may become in nite in the long-run and hence no stationary density exists. Such is the case for absolute and geometric Brownian motion. On the other hand, some processes reach an absorbing barrier with positive probability and will have either discrete probability distributions (if they are absorbed with probability one) or a mixture of discrete and continuous distributions.

CHAPTER 10.


302

If a well de ned long-run density exists, it will depend on S alone (not on T ) and Kolmogorov forward equation becomes d(S ) (S ) d2 2 (S ) (S ) = 2 dS 2 dS where is the long-run density function. Integrating both sides and rearranging terms yields (S ) d 2 (S ) (S ) = 2 2 dS: 2 (S ) (S ) (S ) Integrating both sides again, taking the exponential of both sides and rearranging terms yields Z

S (s) c (S ) = 2 exp 2 ds ; (S ) 2 (s) where c is chosen to ensure that integrates to 1.7 To illustrate, consider again the process in which (S ) = (m (S ) = . The long run distribution is then equal to

Z S

(10.8)

S ) and

(m s) (S ) = c exp 2 ds 2 2mS S 2 = c exp 2 2 2 = c~ exp (S m) ; 2 2 which is recognizable as the normal distribution with mean m and variance 2 =2. Suppose instead that we are interested in an Ito process for which the log of S has a N (m; v 2 =2) long-run distribution. The density would then have the form 1 2 (S ) 2m ln(S ) ln (S ) / exp vS v2

7 See Merton (1975), Appendix B for further discussion. Merton discusses further regularity conditions on and (e.g., they are continuous and (0) = (0) = 0). He points

out that there is another solution to the Kolmogorov equation but that it must be zero when the probability of the boundaries of the state space is zero. This discussion is also related to the Feller classi cation of boundary conditions in the presence of singularities (see Bharucha-Reid, sec. 3.3 and Karlin and Taylor, chap.14).

CHAPTER 10.


303

To match terms within the exponential we must have Z S

(S ) 2 2m ln(S ) : ds = ln( S ) 2 (S ) v2 Dierentiating both sides, this implies that m ln(S ) (S ) =2 2 : 2 2 (S ) v S 2

(10.9)

Furthermore 2 (S ) must be proportional to S to satisfy the term outside the exponential. Setting

(S ) = (m ln(S )) and

2 (S ) = v 2 S satis es (10.9). Thus, the Ito process

p

dS = (m ln(S )) dt + v Sdz has a log-normal long-run distribution with ln(S ) N (m; v 2 =2).8 The expression for the mode of the long-run distribution is easily found from Equation (10.8). The mode, which maximizes the expression in (10.8) can equivalently be found by maximizing Z S

(s) ds ln ( (S )) : 2 (s)

8 It should be noted, however, that dx = (m x)dt + vdz and the log of S , where

p

dS = (m ln(S ))dt + v Sdz do not have the same transition densities, even though they have the same long-run density. To see this apply Ito's Lemma to the transformation y = ln(S ) to obtain

dy = [(m y)

1y 1 2 y 2 v ]e dt + ve 2 dz;

the transition density of which is not known.

CHAPTER 10.


304

Dierentiating both sides and setting the result equal to zero yields the condition d 2 (S ) = 0: (S ) 12 dS The value of S that solves this can be found either analytically or numerically. In economics, state variables are often constrained to be positive and to exhibit mean reversion. A useful class of processes with these characteristics has variance term 2 S 2 and a drift term of the form

Sf (S=m); where f is a decreasing function with f (1) = 0 and with lim zf (z ) = 0:

z !0

For such processes S = 0 is an absorbing barrier; once achieved the process remains at zero. An absorbing barrier at zero is a feature of many economically interesting processes, including stock values, where a zero value can be thought of as bankruptcy, and stocks of renewable resources, where a zero value indicates extinction. Examples of the graph of the instantaneous mean for such processes of the form

dS = 1

S m

!

S dt + S dz

were displayed in Figure 11.1 (page 328) for = 1; 0:5; :::; 2. Figure 10.1 displays the corresponding long run probability distributions. Notice that, although S = 0 is an absorbing barrier, the probability of achieving the barrier is 0. Figure 10.2 displays the mean and mode for alternative values of . Table 10.1 displays long-run density functions, expected values and modes for several families of non-negative, mean reverting processes.

Example: Long-Run Sustainable Harvest

It is sometimes of interest to consider the long run consequences of control policies using the stationary distribution. Consider a renewable resource such as a shery, the stock of which evolves according to dS = (1 (S=m) )S q (S ) dt + Sdz;

1

S) 1 (m S )

+( m

S

S S m

S

S

p (S ) S S

GB 2

2

1

1

2

GG

1 1 ; m; ; 2 1

+ 1

1 2 ; ( ) m; 1

(S ) LN (m; ) LN (ln(m) ; )

B

1

1

1 2

1

2

(1

1

m

2 ) m

maxS (S ) exp(m 2 ) exp( 2 )m

1 ; 12 1 2 m 1 1 1 1+2 ; 2 +

1

2

1

2

B

1

2

E [S1 ] exp(m + =2) exp( = 2)m ( ) m

1

1

( + )

Notes: = 2 =2 N (; 2 ) denotes the normal (Gaussian) distribution LN (; 2 ) denotes the log-normal distribution, i.e., ln(S ) is N (; 2 ) is the Gamma function with (a + 1) = a (a) B is the Beta function: B(a; d) = (a) (d)= (a + d) ac (S=b)c ) Generalized Gamma: GG(S ; a; b; c) = c(S=b) b exp( (a) ac S=b)c ) a d Generalized Beta-2: GB 2(S ; a; b; c; d) = c(S=b) b(1+( B(a;d) The Generalized Gamma distribution requires that < 1; otherwise the process goes to zero with probability one. With = 1, E [S1 ] = (1 )m (the 2-parameter Gamma distribution). The Generalized Beta-2 distribution requires that < 1. With = 1, E [S1 ] = (1 )m (the Generalized Pareto distribution).

(S ) (m ln(S )) ln( mS )S

Table 10.1: Long-Run Densities For Selected Ito Processes

CHAPTER 10. CONTINUOUS TIME MATHEMATICS

305

CHAPTER 10.

306


PDFs with Alternative Shape Parameters (α=0.05, σ=0.1) 1.6 β = −1 1.4

β = −0.5 β=0 β = 0.5

1.2

β=1 β = 1.5

1

f(S)

β=2 0.8

0.6

0.4

0.2

0 0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

S

Figure 10.1 where q (S ) is the harvest rate. The biological growth process (with q (S ) = 0) is the mean reverting process discussed in the example on page 324. For regulatory purposes it is of interest to know what harvest rate maximizes the long-run average harvest level. This would require maximization over the function q (S ); a simpler problem is to nd the maximizing constant proportional harvest rate (C (S ) = cS ), which requires maximization over the constant c. This is facilitated by noting that, for this harvest function, the long run average harvest is c times the long-run average stock level. The mean term for the process can be put in the form

~ ~(S ) = 1 by setting ~ = (

S m ~

!

S

c) and m ~ = (1

c=)1= m. Using Table 10.1, the

CHAPTER 10.

307


Mean and Modal Stock Levels (α=0.05, σ=0.1) 1

0.95

Mean 0.9

0.85

Mode 0.8

0.75 −1

−0.5

0

0.5

β

1

1.5

2

Figure 10.2 long-run average harvest is

2 2

1

2( c) 2 2 2( c) 1 2 2

cm:

Maximizing this with respect to c is equivalent to the following problem max ln c

2( c) 22

ln

2( c) 22

1 + ln(c):

CHAPTER 10.


The FOC for this problem can be written9 2 2

2( c) 22

2( c) 22

1

308

1 + = 0; c

which can be solved for c using a standard root- nding algorithm (see Section ??). In the special case that = 1 the solution has the particularly simple form10 2 c= : (10.10) 2 4 It is useful to note that for 2 > 2 the resource has a non-zero probability of extinction even if no extraction occurs. Thus, in cases in which the possibility of extinction occurs, the optimal sustainable (constant catch rate) policy is to not catch at all.11 Values of the maximum sustainable average harvest rate for alternative and are shown in Figure 10.3. The fact that the catch rate increases as decreases is explainable by recalling from Figure 11.1 that lower values of result in a stock that recovers from low levels more quickly and dies o from high stock levels more slowly.

Code Box 10.1: Maximal Sustainable Harvest Rate

9 The psi function (x) is the derivative of the log of the gamma function; see Abro-

mowitz and Stegun for details. The function le PSI.M is provided to evaluate this function. 10 It is helpful to note that (x) (x 1) = 1=(x 1). 11 Solutions for other values of are possible, though tedious. For example, it can be shown that for = 21 p 1 c= 16 32 642 242 + 34 12 In the limiting case as ! 0, the mean process approaches S ln(m=S ) cS and the mean harvest rate approaches

e

2 c 4

cm

which is maximized at c = . It can also be shown that as ! 0, c approaches =(1 + ).

CHAPTER 10.

309


Maximum Sustainable Harvest Rates (α=0.05) 0.05 σ=0

0.045

σ = 0.05 σ = 0.1

0.04

σ = 0.15 σ = 0.2

0.035

c

0.03

0.025

0.02

0.015

0.01

0.005

0 0

0.5

β

Figure 10.3

1

1.5

CHAPTER 10.


310

10.3 End Notes 10.3.1 Bibliographic Notes Many books contain discussions of Ito stochastic calculus. A useful reference with nance related applications is Hull; this has a particularly good discussion of arbitrage conditions. At a more advanced level see DuÆe; the discussion of the Feynman-Kac formula draws heavily on this source. A brief but useful discussion of steady-state distributions is found in Appendix B of Merton (1985). For more detail, including discussion of boundary issues, see Karlin and Taylor, chapter 15. Early work in this area is contained in several papers by Feller. [INCOMPLETE]

CHAPTER 10.


311

10.3.2 References Cox, D.R. and H.D. Miller. The Theory of Stochastic Processes. John Wiley and Sons, New York. 1965. Feller, William. \Two Singular Diusion Problems." Annals of Mathematics. 54(1951): 173-182. Merton, Robert C. \An Asymptotic Theory of Growth Under Uncertainty." Review of Economic Studies. 42(1975):375-393. [INCOMPLETE]

Exercises 1. Consider the continuous time optimal control problem of determining the investment policy that maximizes the discounted stream of income: Z 1

maxx(t)

0

f (x(t); t)

dk(t) = x(t) k(t) dt x(t) 0 k(0) = k0

s:t:

where, in period t, k(t) is the level of capital stock, x(t) is the optimal rate of investment, f (k; t) is the income generated from a capital stock k, and is the continuous rate of capital decay. Formulate the Hamiltonian associated with the optimal control problem. Give an explicit economic interpretion the costate variable and the Hamiltonian. Apply the K-K-T theorem to the maximization problem embedded in Pontryagin's Maximum Principle to show that for an optimal path, x(t) (t) = 0 for all t. In light of the assertion just proved, state and interpret the costate equation. 2. Consider an economy of price taking rms facing adjustment costs and a downward-sloping demand curve for their identical products. The competitive rational expectations equilibrium for this economy is characterized by the capital accumulation path that solves the following dynamic optimization problem: max I ()

Z 1 0

exp( rt)fW (K ) pI

1 2 aI gdt 2

CHAPTER 10.


312

K_ = I K K (0) = K0

s:t: where

K (t) = capital stock at time t I (t) = capital investment rate at time t (may be negative) W (K ) = D(Q) F (K ) a r p

= = > > > >

Z F (K ) 0

D(Q)dQ is consumer surplus

is the inverse demand function (D(Q) > 0; D0(Q) < 0) is the production function (F (0) = 0; F 0(K ) > 0) 0 is the cost of adjustment coeÆcient 0 is the interest rate 0 is the unit price of new capital 0 is the depreciation rate of capital

Perform a comparative dynamics analysis of the steady-state equilibrium, using a phase diagram to illustrate your discussion. Linearize about the steady state, deriving and interpreting the eigenvalues and eigenvectors corresponding to the linear approximation. How does the steady-state level of capital stock change as a, r, p, or increase? How does the speed of adjustment to steady-state change as a, r, p, or increase? 3. Suppose that a variable is governed by

dS = Sdt + Sdz: a) Show that

E0 [St ] = S0 et : b) Use (a) to show that

E0

Z t 0

e

S d

= (1 e t )

S0

:

CHAPTER 10.


313

4. Pricing Bonds De ne P (t; T ) to be the current (time t) price of a pure discount bond maturing at time T , i.e., a bond that pays $1 at time T . The price of a bond of any maturity depends on the instantanous interest rate, r. It can be shown that

P (r; t; T ) = E^ exp

Z T t

r( )d

;

where the expectation is taken with respect to the risk adjusted process governing the instanteous interest rate. Assuming that this process is

dr = (r; t)dt + (r; t)dz an extended version of the Feyman-Kac Formula implies that P is the solution to

rP = Pt + (r; t)Pr + 21 2 (r; t)Prr ; subject to the boundary condition that P (r; T; T ) = 1. Suppose that the instantaneous interest rate process is

dr = (m r)dt + dz: Show that P has the form

P (r; t; T ) = A(t; T ) exp( B (t; T )r) and, in doing so, determine the functions A and B . 5. Given the setting of the previous problem, suppose we take the instantaneous interest rate process to be

p

dr = (m r)dt + rdz: Verify numerically that P has the form

P (r; t; T ) = A(t; T ) exp( B (t; T )r)

CHAPTER 10.


314

with

2 e( +a)(T t)=2 A(t; T ) = ( + a)(e (T t) 1) + 2

2ab=2

and 2(e (T B (t; T ) = ( + a)(e (T

t) t)

p

1) ; 1) + 2

where = a2 + 2 2 . 6. A futures contract maturing in periods on a commodity whose price is governed by

dS = (S; t)dt + (S; t)dz can be shown to satisfy

V (S; ) = (rS

Æ (S; t))VS (S; ) + 12 2 (S; t)VSS (S; )

subject to the boundary condition V (S; 0) = S . Here Æ is interpreted as the convenience yield, i.e., the ow of bene ts that accrue to the holders of the commodity but not to the holders of a futures contract. Suppose that the volatility term is

(S; t) = S: In a single factor model one assumes that Æ is a function of S and t. Two common assumptions are

Æ (S; t) = Æ and

Æ (S; t) = ÆS: In both cases the resulting V is linear in S . Derive explicit expressions for V given these two assumptions.

CHAPTER 10.

315


7. Continuing with the previous question, suppose that the convenience yield is

Æ (S; t) = ÆS where Æ is a stochastic mean-reverting process governed by

dÆ = (m Æ )dt + Æ dw; with Edzdw = Æ . Furthermore, suppose that the market price of the convenience yield risk is a constant . Then the futures price solves

V = (r

Æ )SVS + ((m Æ ) ) VÆ + 21 2 S 2 VSS + Æ SVSÆ + 12 Æ2 VÆÆ ;

with V (S; 0)=S. Verify that the solution has the form V = exp(A( ) doing so derive expression for A( ) and B ( ).

B ( )Æ )S and in

8. Suppose that

dS = dt + dz; where and are constants. Show that the transition probability distribution is normal and determine it mean and variance. 9. Suppose that

dS = (m S )dt + dz (in the notation of Section 10.2.1 0 = m and 1 = ). Show that the transition probability distribution is normal and determine it mean and variance. Take limits as T ! 1 to determine the long-run distribution. 10. Suppose that

dS = e

t dz:

Show that the transition distribution is normal and determine it mean and variance. 11. Verify the solutions for maximum average sustainable harvest rates for = 1, 21 and 0 given in Equation (10.10) and Footnote 11.

Chapter 11 Continuous Time Dynamic Models: Theory 11.1 Stochastic Control On an intuitive level, continuous time optimization methods can be viewed as simple extensions of discrete time methods. In continuous time one replaces the summation over time in the objective function with an integral evaluated over time and the dierence equation de ning the state variable transition function with a dierential equation. For non-stochastic models, the optimization problem is1 max

Z T

x(S;t) 0

e t f (S; x)dt + e

T R(S (T ));

s.t. dS = g (S; x)dt;

where S is the state variable (the state), x the control variable (the control), f is the reward function, g the state transition function and R is a terminal period \salvage" value. The time horizon, T , may be in nite (in which case R has no meaning) or it may be state dependent and must be determined endogenously (see Section 11.2.3 on optimal stopping). For non-stochastic problems, optimal control theory and its antecedent, the calculus of variations, have become standard tools in economists mathematical toolbox. Unfortunately, neither of these methods lends itself well to extensions involving uncertainty. The other alternative for solving such 1 We cover here the more common discounted time autonomous problem. The more

general case is developed as an exercise.

316

CHAPTER 11.

317

CONTINUOUS TIME MODELS: THEORY

problems is to use continuous time dynamic programming. Uncertainty can be handled in an elegant way if one restricts oneself to modeling that uncertainty using Ito processes. This is not much of a restriction because the family of Ito processes is rather large and can be used to model a great variety of dynamic behavior (the main restriction is that it does not allow for jumps). Furthermore, we will show that for deterministic problems, optimal control theory and dynamic programming are two sides of the same coin and lead to equivalent solutions. Thus, the only change needed to make the problem stochastic is to de ne the state variable, S , to be a controllable Ito process, meaning that the control variable, x, in uences the value of the state:2

dS = g (S; x)dt + (S )dz: To develop the solution approach on an intuitive level, notice that for problems in discrete time, Bellman's equation can be written in the form

1 V (S; t) = max f ( S; x ) t + E [V (St+t ; t + t)] : x 1 + t t Multiplying this by (1 + t)=t and rearranging:

Et [V (St+t ; t + t) V (S; t)] : V (S; t) = max f ( S; x; t )(1 + t ) + x t Taking the limits of this expression at t version of Bellman's equation:

! 0 yields the continuous time

Et dV (S; t) V (S; t) = max f ( S; x; t ) + : x dt

(11.1)

If we think of V as the value of an asset on a dynamic project, Bellman's equation states that the rate of return on V (V ) must equal the current income ow to the project (f ) plus the expected rate of capital gain on the asset (E [dV ]=dt), both evaluated using the best management strategy (i.e., the optimal control). Thus, Bellman's equation is a kind of intertemporal arbitrage condition.3 2 A more general form would allow x to in uence the diusion as well as the drift term; this can be handled in a straightforward fashion but makes exposition somewhat less clear. 3 It is important to note that the arbitrage interpretation requires that the discount rate, , be appropriately chosen (see Section 10.1.3 for further discussion).

CHAPTER 11.


318

By Ito's Lemma

dV = [Vt + g (S; x)VS + 21 (S )2 VSS ]dt + (S )VS dz: Taking expectations and dividing by dt we see that the term Et dV (S; t)=dt can be replaced, resulting in the following form for Bellman's equation in continuous time:4

V = max f (S; x) + Vt + g (S; x)VS + 21 2 (S )VSS : x

(11.2)

The maximization problem is solved in the usual way by setting the rst derivative equal to zero:

fx (S; x) + gx (S; x)VS = 0:

(11.3)

Combining this with

V = f (S; x) + Vt + g (S; x)VS + 12 2 (S )VSS

(11.4)

results in two functional equations that must be solved for to yield the two functions: the value function V (S; t) and the optimal policy function x (S; t).5 If a solution to the maximization problem can be found of the form

x = x(S; VS ) it may be useful to form the concentrated Bellman equation:

V = f (S; x(S; VS )) + Vt + g (S; x(S; VS ))VS + 12 2 (S )VSS :

(11.5)

Notice that the concentrated Bellman equation is non-linear whereas the Bellman equation is linear in the value function and its partial derivatives. The usefulness of the concentrated Bellman Equation will depend on whether it is easier to solve a single nonlinear PDE or a linear PDE combined with a functional equation not involving derivatives. Notice that Bellman's Equation is not stochastic; the expectation operator and the randomness in the problem have been eliminated by using Ito's Lemma. As with discrete time versions the state transition equation is incorporated in Bellman's equation. This eectively transforms a stochastic 4 Also known as the Hamilton-Jacobi-Bellman equation. 5 It may be puzzling why the max operator is dropped from 11.4 until it is noted that 11.3 must be satis ed simultaneously, i.e., the optimized value of x(S; t) is used in 11.4.

CHAPTER 11.


319

dynamic problem into a deterministic one. If there are additional constraints on the state variables they typically can be handled in the usual way (using Lagrange multipliers and, for inequality constraints, Karush-Kuhn-Tucker type conditions). Constraints on the control on somewhat more problematic (they are discussed in the inventory example in Section 11.2.3). In nite time horizon problems, the value function is a function of time. In in nite time horizon problems, however, the value function becomes time invariant, implying that V is a function of S alone and thus Vt = 0. Thus the Bellman's Equation simpli es to

V = max f (S; x) + g (S; x)VS + 21 2 (S )VSS : x

11.1.1 Relation to Optimal Control Theory It is worth spending some time relating the dynamic programming approach to optimal control theory. As stated previously, optimal control theory is not naturally applied to stochastic problems but it is used extensively in deterministic ones. The Bellman equation in the determinstic case is

V = max f (S; x) + Vt + g (S; x)VS ; x where x is evaluated at its optimal level. Suppose we totally dierentiate the marginal value function with respect to time: dVS dS = VSt + VSS = VSt + VSS g (S; x): dt dt Now apply the Envelope Theorem to the Bellman equation to determine that

VS = fS (S; x) + VtS + g (S; x)VSS + VS gS (S; x): Combining these expressions and rearranging yields dVS = VS fS VS gS : (11.6) dt This can be put in a more familiar form by de ning = VS . Then (11.6), combined with the FOC for the maximization problem and the state transition equation can be written as the following system 0 = fx (S; x) + gx (S; x)

CHAPTER 11.


320

d = fS (S; x) gS (S; x) dt and

dS = g (S; x): dt These relationships are recognizable as the Hamiltonian conditions from optimal control theory, with the costate variable representing the shadow price of the state variable (expressed in current value terms).6 The message here is that dynamic programming and optimal control theory are just two approaches to arrive at the same solution. It is important to recognize the distinction between the two approaches, however. Optimal control theory leads to three equations, two of which are ordinary dierential equations in time. Optimal control theory therefore leads to expressions for the time paths of the state, control and costate variables as functions of time: S (t), x(t) and (t). Dynamic programming leads to expressions for the control and the value function (or its derivative, the costate variable) as functions of time and the state. Thus dynamic programming leads to decision rules rather than time paths. In the stochastic case, it is precisely the decision rules that are of interest, because the future time path, even when the optimal control is used, will always be uncertain. For deterministic problems, however, DP involves solving partial dierential equations, which tend to present more challenges than ordinary dierential equations.

11.1.2 Boundary Conditions The Bellman's equation expresses the optimal control in terms of a dierential equation. In general, there will be many solutions, many of which are useless to us. Furthermore, from a numerical point of view, without boundary conditions imposed on the problem, it will be luck as to whether the derived solution is indeed the correct one. Unfortunately, the literature on this topic is incomplete and boundary conditions are often justi ed by economic rather than mathematical reasoning. For example, consider a case in which one is extracting a resource with a stochastic price. Suppose also that the price has an absorbing barrier at P = 0 (e.g., dP = (m P )P dt + P dz ). The value of the inventory is a function of the level of the inventory and the price: 6 See Kamien and Schwartz, pp. 151-152 for further discussion.

CHAPTER 11.


321

V (I; P ). The reward function is P q , where dI = qdt, so the control q is the rate of extraction. It is obvious that the stream of pro ts generated by selling from an inventory will be zero if the price is zero because, once zero is reached, the price is zero forever and the inventory is therefore worthless. Also, if the inventory reaches zero it is worthless. We see, therefore, that V (I; 0) = V (0; P ) = 0: We would still need to determine upper boundaries, which we discuss further in the example on page 324. Many problems in economics specify a reward function that has a singularity at an endpoint. Typical examples include utility of consumption functions for which zero consumption is in nitely bad. The commonly used constant relative risk aversion family of utility functions

U (c) = (c

1)=

(with ln(c) when = 0) is a case in point. Again, economic reasoning would suggest that if consumption is derived from a capital or resource stock and that stock goes to zero, consumption must also go to zero and hence the value of a zero stock, which equals the discounted stream of utility from that stock must be 1. Furthermore, the marginal value of the stock when the stock gets low becomes quite large, with VS = 1 as S ! 0. Although this reasoning makes good sense from an economic perspective, it raises some diÆculties for numerical analysis. As a rule of thumb, one needs to impose a boundary condition for each derivative that appears in Bellman's equation. For a single state problem, this means that there are two boundary conditions needed. In a twodimensional problem with only one stochastic state variable, we will need two boundary conditions for the stochastic state and one for the non-stochastic one. For example, suppose Bellman's equation has the form

V = f (S; R; x) + g (S; R; x)VR + (S )VS + 21 2 S 2 VSS : To completely specify the problem we could impose a condition at a point R = Rb , e.g. V (S; Rb ) = H (S ) and conditions at S = S and S = S , say VSS (S ) = VSS (S ) = 0. Like all rules of thumb, however, there are exceptions. The exceptions tend to arise in singular problems, when the variance term vanishes at a

CHAPTER 11.


322

boundary. For example, it may not be necessary to impose explicit boundary conditions when the state variable is governed by

dS = (S; x)dt + Sdz; where (0; 0) = 0 and x is constrained such that x = 0 if S = 0. Zero is a natural absorbing state for this process, meaning that if S (t) = 0 then S ( ) = 0 for all > t. In this case, it may not be necessary to impose conditions on the boundary at S = 0. An intuitive way to think of this situation is that a second order dierential equation becomes eectively rst order as the variance goes to zero. We may, therefore, not need to impose further conditions to achieve a well de ned solution. Feller devised a classi cation scheme for diusion processes with singular boundaries (see discussion by Bharucha-Reid, sec. 3.3, and Karlin and Taylor, Chap 15.). Although the literature is incomplete on this issue, a rule of thumb is that, if a boundary is inaccessible, meaning that the probability is zero that the process (naturally) will achieve the boundary, no boundary condition need be imposed. Several examples we will discuss have singular boundary conditions. It is perhaps clear that in continuous time stochastic problems, de ning the appropriate dierential equation is the easy part and specifying the correct boundary conditions is the tricky part.

11.1.3 Choice of the Discount Rate The choice of the appropriate discount rate to use in dynamic choice problems has been a topic of considerable discussion in the corporate nance literature. The arbitrage theory discussed in the previous chapter has been fruitfully applied to this issue. In particular, there is an equivalence between the choice of a discount rate and the price of risk assigned to the various sources of risk aecting the problem. In general, if there is a market for assets that depend on a speci c risk, S , then arbitrage constrains the choice of the discount rate. If an inappropriate discount rate is used, a potential arbitrage opportunity is created by either overvaluing or undervaluing the risk of the project. To see this note that the concentrated Bellman's equation for a dynamic project can be written

V = ÆV + Vt + S VS + 21 S VSS ;

CHAPTER 11.


323

where ÆV = f (S; x ; t) and x = g (S; x ; t). To avoid arbitrage, however, (10.6) must hold. Together these relationships imply that

= r + S VS =V = r + V =V

(11.7)

In practice we can eliminate the need to determine the appropriate discount rate by using the risk-free rate as the discount rate and acting as if the process S has instantaneous mean of either

^S = S

S S

^S = rS

ÆS :

or Which form is more useful depends on whether it is easier to obtain estimates of the market price of risk for S , S , or income stream generated by S , ÆS . The latter, however, is only possible if S is itself the value of an asset, whereas the former can be estimated (in principle) if there is some traded asset the value of which depends on S . Even if the project involves a non-traded risk, it may be easier to guess the market price of that risk than to de ne the appropriate discount rate. For example, if the risk is idiosyncratic and hence can be diversi ed away, then a well-diversi ed agent would set the market price of risk to zero. An appropriate discount rate is particularly diÆcult to select when there are multiple source of risk (state variables) because the discount rate becomes a complicated function of the various market prices of risk. Having said that, there may be cases in which the appropriate discount rate is easier to set. For rm level capital budgeting, the discount rate is the required rate of return on the project and, in a well functioning capital market, should equal the rm's cost of capital. Thus the total return on the project must cover the cost of funds:

V = ÆV + V = rV + S V : The cost of funds, , therefore implicitly determines the market price of risk (using 11.7). Summarizing, there are three alternative cases to consider: 1. S is a traded asset for which

S

S = rS

ÆS

CHAPTER 11.


324

2. S is not a traded asset but there is a traded asset the value of which, W , depends on S and the market price of risk can be determined according to

= (W + ÆW

rW )=W

3. S represents a non-priced risk and either or must be guessed When S is a controllable Ito process, the payment stream, Æ (S; t), becomes f (S; x; t) and the drift term, (S; t), becomes g (S; x; t). There are three forms of Bellman's equation:

ÆS ) + 21 VSS 2 (S; t)

A)

rV = max f (S; x; t) + Vt + VS (rS x

B)

rV = max f (S; x; t) + Vt + VS (g (S; x; t) (S; t)) + 12 VSS 2 (S; t) x

C)

V = max f (S; x; t) + Vt + VS g (S; x; t) + 21 VSS 2 (S; t) x

Any of the three forms can be used when S is a traded asset, although (A) and (B) are preferred in that they rely on market information rather than on guesses concerning the appropriate discount rate. When S is not a traded asset but represents a risk priced in the market, (B) is the preferred form although (C) can be used. If S represents a non-priced asset then either form (B) or (C) may be used, depending on whether it is easier to determine appropriate values for or for .

11.1.4 Examples Example: Optimal Renewable Resource Extraction

Pindyck (1984) discusses the optimal extraction rate and in situ rents of a renewable resource. Suppose that the stock of a resource, S , is governed by the controlled stochastic process

dS = (B (S ) q )dt + Sdz; where B (S ) is a biological growth function and q is the harvest rate of the resource. Typically, there will be a value, K , such that B (K ) = 0. Also we require that B (0) = 0, so 0 is an absorbing barrier of the process (there is

CHAPTER 11.


325

no return from extinction) and that B 0 > 0 for S 2 (0; K ) and B 0 < 0 for S > K . K can be thought of as an environmental carrying capacity; the resource tend to shrink once it becomes greater than K . Suppose that marginal costs depend only on the stock of the resource with the speci c functional form

C (q ) = c(S )q: The total surplus (consumer plus producer) is

f (S; q ) =

Z q 0

D 1 (z )dz

c(S )q

With a discount rate of , the Bellman Equation for this optimization problem is

V = max q

Z q 0

D 1 (z )dz

c(S )q + (B (S ) q ) VS + 21 2 S 2 VSS :

The FOC for the optimal choice of q is

D 1 (q ) c(S ) VS (S ) = 0; or

q = D(c(S ) + VS ): Notice that the FOC implies that marginal surplus of an additional unit of the harvested resource is equal to the marginal value of an additional unit of the in situ stock:

fq (S; q ) D 1 (q ) c(S ) = VS (S ): To make further progress towards speci c solutions we must parameterize the demand, cost and growth functions. Following Pindyck, we assume that the demand for the harvested resource is iso-elastic,

q = D(p) = bp ; the cost function is

c(S ) = cS ;

CHAPTER 11.


326

and the biological growth function is of the form

B (S ) = S 1

S K

!

(the limiting case as ! 0 is B (S ) = S ln(K=S )). The biological growth function is mean reverting with S < K resulting in expected increases in the stock and S > K resulting in expected decreases in the stock. The parameters and determine the speed of the mean reversion. Other things being equal, increasing causes the size of positive changes to be greater and negative changes to be smaller when the stock is far from K (for stock levels near K the value of has little eect). These features are illustrated in Figure 11.1, which shows the mean function for alternative values of (shown here with = 0:05 and K = 1). Features of the this process are discussed further in Section 10.2.2 on page 304. [Note that the mean growth rate is maximized at S = (1 + ) 1= , which goes to 1=e as ! 0]. The model is summarized in Example Box 1. The concentrated Bellman's Equation using these functional forms becomes7 b V = cS + VS 1 + S 1 (S=K ) VS + 12 2 S 2 VSS :(11.8) 1 The boundary conditions for the problem require that the marginal surplus must become in nite at S = 0 and be zero as S goes to in nity: lim VS (S ) = 1 and lim VS (S ) = 0:

S !0

S !1

As the stock gets small and hence the catch rate declines, the value of an additional unit of the resource becomes in nitely high because of the form of the demand function. On the other hand, as the stock gets very large, an additional unit of the resource has no value at all because, after a point, the harvest of additional units does not justify the cost and those units merely contribute to the crowding and hence the mortality of the resource. Dierential equations of the form (11.8) generally have no closed-form solution and therefore it is necessary to solve them numerically. In the special 7 Following Pindyck, we ignore the technicality that the integral de ning the surplus

does not converge when the lower limit of integration is 0. The lower limit is treated as a constant and ignored.

CHAPTER 11.


327

Example Box 11.1: Optimal Harvest of a Renewable Resource Problem:

max q

Z 1 0

e t f (S; q )dt

s.t.

dS = (B (S ) q )dt + Sdz Variables: q harvest (consumption) rate (control: quantity per period) S resource stock (state: quantity) Parameters: B (S ) D(p) c(S ) f (S; q )

biological growth function: B (S ) = S (1 (S=K ) )= demand function: D(p) = bp marginal cost function: c(S ) = cS surplus function: f (S; q ) = D 1 (z )dz c(S )q discount rate

with B (0) = 0, B (K ) = 0, B 0 (S ) > 0 for 0 < S < K and B 0 (S ) < 0 for S>K Bellman's Equation:

V = max q = max q

Z q 0

D 1 (z )dz

b1= 1 1= q cS q 1 1= + S 1 (S=K ) q VS + 21 2 S 2 VSS

Boundary Conditions:

VS (0) = 1 VS (1) = 0 Optimal Harvest Function:

q = b(cS

c(S )q + (B (S ) q ) VS + 21 2 S 2 VSS

+ VS ) :

CHAPTER 11.

328


Mean Functions with Alternative Shape Parameters 0.05 β = −1

0.04

β = −0.5 β=0

0.03

β = 0.5 0.02

β=1 β = 1.5

µ(S)

0.01

β=2

0

−0.01

−0.02

−0.03

−0.04

−0.05 0

0.5

1

1.5

S

Figure 11.1 case that = 1 + and = 1=(1 + ), however, an analytic solution is possible. Speci cally, 1 V (S ) = + S K where solves

and

1+

c = 0; 1+

+ 2 : = b 1 + 2b It is straightforward to solve for using a standard root nding solver (see Section ??) and for some values of an explicit solution is possible. Table

CHAPTER 11.


329

11.1 provides three special cases discussed by Pindyck that have closed form solutions, including the limiting case as ! 0.

Table 11.1: Known Solutions to the Optimal Harvesting Problem 1 0 1 2

B (S ) 1 2 S (1 S=K ) 2 1 1 S ln(K=S ) q K 2 12 2S 1 S

where

1 = 2 1 2 =

3 = c

b + 2

2

c2 +

s

@1 +

b( + ) ln 1 + ( + )c

1 2

3

0

s

V (S ) VS (S ) 1 S1 + K S b b +ln(S ) + p2 (+)S p p 3 S + K S

1+c

+ b

2 2

q (S )

pc + S

1 b(+) b+(+)c S b S (c 21 3 )2

1 A

ln(K ) 21 1 b+ +

!

2b + + 2 =8

Example: Stochastic Growth

Cox, Ingersoll and Ross (1985) develop a growth model in which the productivity of capital, K , depends, both in mean and variance, on an exogenous technology shock, denoted Y . Y is governed by p dY = (aY b)dt + Y dz: With c denoting current consumption (the control), the capital stock dynamics are p dK = ( KY c)dt + K Y dz; where the same Brownian motion, dz , that drives the technology shocks also causes volatility in the productivity of capital. The social planner's optimization problem is to maximize the present value of the utility of consumption,

CHAPTER 11.


330

taken here to be the log utility function, using discount rate . The problem is summarized in Example Box 2.

Example Box 11.2: Optimal Growth Problem:

max c

Z 1 0

e

t ln(c)dt

s.t.

dY = (aY

p

b)dt + Y dz

dK = ( KY

p

c)dt + K Y dz

Variables:

c consumption rate (control: quantity per period) Y technology shock (state) K capital stock (state: quantity) Parameters: ln(c) a; b; ;

utility function technology shock dynamics capital productivity parameters discount rate

Bellman's Equation:

V = max ln(c) + VK ( KY c +

2 2 1 2 VKK K Y

Optimal Consumption: c = K

c) + VY (aY

b)

+ 21 VY Y 2 Y + VKY KY

Before discussing the solution it is useful to consider the form of the technology assumed here. The expected growth rate in capital, ignoring consumption, is aÆne in the capital stock and depends on the size of the technology shock. The technology shock, in turn, has an expected growth

CHAPTER 11.

331


pattern given by

dEY = (aEY

b)dt:

This dierential equation can be solved for the expected value of Y :8

E0 Yt = (Y0

b=a)eat + b=a:

Roughly speaking, this implies that, for a given capital stock, the productivity of capital is expected to grow at a constant rate (a) if Y is greater than b=a and to shrink at the same rate when Y is less than b=a (the model might make more sense if we take a and b to be negative parameters; this would imply that b=a is a stable point rather than an unstable one). Let us guess that the solution is one with consumption proportional to the capital stock

c = K: The FOC condition associated with the Bellman equation (see Example Box 2) tells us that the optimal c satis es 1=c = VK : If our guess is right it implies that V (K; Y ) = ln(K )= + f (Y ), where f (Y ) is yet to be determined. To verify that this guess is correct, substitute it into the Bellman equation:

ln(K ) + f (Y ) = ln(K ) + Y

1 + f 0 (Y )(aY

b)

2 Y 1 00 + f (Y ) 2 Y: 2 2

Collecting terms and simplifying, we see that = and that f (Y ) solves a certain second order dierential equation. Rather than try to solve f (Y ) directly, however, a more instructive approach is to solve for the value function directly from its present value form. If our guess is correct then

V (K; Y ) = E

Z 1 0

e

t ln(K )dt

Z 1 ln() + e t E [ln(K )] dt(11.9) = 0

8 This is the same functional form as equation 10.3 on 290.

CHAPTER 11.


332

The only diÆculty presented here is to determine the time path of E [ln(K )]. Using Ito's Lemma and c = K p dK 1 2 Y dt = 21 2 Y dt + Y dz: d ln(K ) = 2 K Taking expectations and using the previously obtained result for EY yields

dE [ln(K )] = 12 2 E [Y ] dt = 12 2 Y0 ab eat + ab dt = [c0 aeat + c1 ] dt; where

c0 =

a

1 2 2 Y

0

b a

b 12 2 : a Integrating both sides and choosing the constant of integration to ensure that, at t = 0, the expected value of E [ln(Kt )] = ln(K0 ) produces an expression for E [ln(K )] when c = K :

c1 =

E [ln(K )] = ln(K0 ) c0 + c0 eat + c1 t: One step remains; we must use the formula for E [ln(K )] to complete the derivation of the present value form of the value function. Recalling (11.9)9 Z 1

V (K; Y ) =

Z0 1

=

0

e t E [ln(K )] dt + (ln(K0 )

ln(K0 )

=

c0

+

c0 ) e c0

a

ln()

t + c e(a )t 0

+

+ c1 te

c1 ln() + : 2

9 If the third line is problematic for you it might help to note that Z

te

t dt

=

e

t

t+

1 :

t dt +

ln()

CHAPTER 11.

333


Substituting in the values of c0 and c1 and rearranging we obtain an expression for the value function ln(K ) + V (K; Y ) = (

1 2 2 Y

1 + ln() a)

b( 12 2 ) ( a)

!

1 :

(the subscripts on K and Y are no longer necessary). Notice that this does indeed have the form ln(K )= + f (Y ), with f (Y ) a linear function of Y . We have therefore satis ed the essential part of Bellman's equation, namely verifying that c = K is an optimal control. We leave as an exercise the task of completing the veri cation that Bellman's equation is satis ed by our expression for V (K; Y ). Let's review the steps we took to solve this problem. First, we guessed a solution for the control and then used the rst order conditions from Bellman's equation to determine a functional form for V (K; Y ) that must hold for this to be an optimal control. We then evaluated the present value form of the value function for this control, thereby obviating the need to worry about the appropriate boundary conditions on Bellman's equation (which we have seen is a delicate subject). We were able to obtain an expression for the value function that matched the functional form obtained using the rst order conditions, verifying that we do indeed have the optimal control. This strategy is not always possible, of course, but when it is, we might as well take advantage of it.

Example: Portfolio Choice The previous examples had a small number of state and control variables. In the example we are about to present, we start out with a large number of both state variables and controls, but with a speci c assumption about the state dynamics, the dimension of the state is reduced to one and the control to two. Such a reduction makes a problem from that is essentially impossible to solve in general into one that is much closer to being solved. If a speci c class of reward functions is used, the problem can be solved explicitly (we leave this as an exercise). Suppose investors have a set of n assets from which to invest, with the per unit price of these assets generated by an n dimensional Ito process

dP = (P )dt + (P )dz;

CHAPTER 11.


334

Example Box 11.3: Portfolio Choice Problem:

V = max U (C ) + W w> C VW + 21 W 2 w>wVW W ; C;w

s.t.

P

i wi

= 1 and

dP = P dt + dz

dW = W w> C dt + W w>dz: wi = Ni Pi =W: Variables:

p W C w

price (underlying n-dimensional state ) wealth (state) consumption (control) share of weath in asset i (n-dimensional control)

Optimality Conditions:

U 0 (C ) = VW ; W VW + W 2 VW W w 1 = 0; and

X i

wi = 1;

Solution:

w = + (W );

(11)

where 11 = > 1 ; 1 1

=

1

and

(W ) =

1> 1 1 1> 1 1

VW ; W VW W

CHAPTER 11.


335

where (P ) is an n k matrix valued function (i.e., : dt + w>dz; i.e., that W is itself a geometric Brownian motion process. This means that portfolio decisions can be expressed in terms of wealth alone, without reference to the prices of the underlying assets in the portfolio. Geometric Brownian motion, therefore, allows for a very signi cant reduction in the dimension of the state (from n to 1). Consider an investor who draws o a ow of consumption expenditures C . The wealth dynamics are then dW = W w> C dt + W w>dz: Suppose the investor seeks to maximize the discounted stream of satisfaction derived from consumption, where utility is given by U (C ) and the discount rate is . The Bellman's Equation for this problem is11 V = max U (C ) + W w> C VW + 21 W 2 w>wVW W ; C;w

10 the case in which a riskless asset is available is treated in an exercise. 11 If prices were not geometric Brownian motion the coeÆcients and would be func-

tions of current prices and the Bellman's Equation would have additional terms representing derivatives of the value function with respect to prices, which would make the problem considerably harder to solve.

CHAPTER 11.

s.t.

P


336

i wi

= 1. The FOC associated with this maximization problem are

U 0 (C ) = VW ;

(12a)

W V W + W 2 VW W w and

X i

1 = 0;

wi = 1;

(12b) (12c)

where is a Lagrange multiplier introduced to handle the adding-up constraint on the wi . A bit of linear algebra applied to (12b) and (12c) will demonstrate that the optimal portfolio weight vector, w, can be written as a linear combination of vectors, and , that are independent of the investor's preferences:

w = + (W );

(13)

where

11 = > 1 ; 1 1

=

1

1> 1 1 1> 1 1

and

VW ; W VW W This has a nice economic interpretation. When asset prices are generated by geometric Brownian motion, a portfolio separation result occurs, much like in the static CAPM model. Only two portfolios are needed to satisfy all investors, regardless of their preferences. One of the portfolios has weights proportional to 1 1, the other to 1 ( ( >)1). The relative amounts held in each portfolio depend on the investor's preferences, with more of the rst portfolio being held as the degree of risk averse rises (for smaller values of (W )). This is understandable when it is noticed that the rst portfolio is the minimum risk portfolio, i.e., solves the problem (W ) =

min > ; s.t. > 1 = 1:

CHAPTER 11.


337

Furthermore, the expected return on the minimum risk portfolio is > ; hence the term ( > )1 can therefore be thought of as an \excess" return vector, i.e., the expected returns over the return on the minimum risk portfolio. The problem is therefore reduced to determining the two decision rule functions for consumption and investment decisions, C (W ) and (W ), that satisfy:

U 0 (C (W )) = VW (W ) and

VW (W ) : W VW W Notice that the two fund separation result is a result of the assumption that asset prices follow geometric Brownian motions and not the result of any assumption about preferences. Given the enormous simpli cation that it allows, it is small wonder that nancial economists like this assumption. (W ) =

11.2 Free Boundary Problems We have already seen how boundary conditions are needed to determine the solution to dynamic models in continuous time. Many important problems in economics, however, involve boundaries in the state space the location of which must be determined as part of the solution. Such problems are known as free boundary problems. The boundary will either mark the location where some discrete decision is made or will represent a location at which some transition takes place.12 Table 11.2 contains a classi cation of dierent free boundary problems that have appeared in the economics literature. The most important distinction, both in understanding the economics and in solving the problem numerically, is whether the boundary can be crossed. If the control is such that it maintains a stochastic process within some region de ned by the free boundary, the problem is a barrier problem and we will solve a dierential equation in this region only. For example, the stock of a stochastic renewable 12 In the physical sciences free boundary problems are also known as Stefan problems. A

commonly used example is the location of the phase change between liquid and ice, where the state space is measured in physical space coordinates.

CHAPTER 11.


338

resource can be harvested in such a way as to keep the stock level below some speci ed point. If the stock rises to this point, it is harvested in such a way as to maintain it at the boundary (barrier control) or to some point below the boundary (impulse control). In barrier controls problems, the barrier de nes a trigger point at which, if reached, one maintains the state at the barrier by exactly osetting any movements across the barrier. Typically, such a control is optimal when there are variable costs associated with exerting the control. In such a situation it is only optimal to exert the control if the marginal change in the state osets the marginal cost of exerting the control. In impulse control problems, if the barrier is reached one takes an action that instantaneously moves the state to a point inside the barrier. An (s; S ) inventory control system is an example of an impulse control in which the state is the level of inventory, which is subject to random demand. When the inventory drops to the level s, an order to replenish it to level S is issued. Typically such controls are optimal when there is a xed cost associated with exerting the control; the control is exerted only when the bene t from exerting the control covers the xed cost. The other major type of free boundary problem arises when, in addition to one or more continuous state variables, there is also a state that can take on discrete set of values. In this case, boundaries represent values of the continuous states at which a change in the discrete state occurs. For example, consider a rm that can either be actively producing or can be inactive (a binary state variable). The choice of which state is optimal depends on a randomly uctuating net output price. Two boundaries exist that represent the prices at which the rm changes from active to inactive or from inactive to active (it should be clear that the latter must be above the former to prevent the rm from having to be continuously changing!). An important special case of the discrete state problem is the so-called optimal stopping problem; the exercise of an American option is perhaps the most familiar example. Stopping problems arise when the choice of one of the discrete state values is irreversible. Typically the discrete state takes on two values, active and inactive. Choosing the inactive state results in an immediate one time payout. An American put option, for example, can be exercised immediately for a reward equal to the option's exercise price less the price of the underlying asset. It is optimal to exercise when the underlying asset's price is so low that it is better to have the cash immediately and reinvest it than to wait in hopes that the price drops even lower.

Action Change state Switch between control extrema

Action Jump from boundary Move along boundary

Boundary Condition i j ij V (S ) = V (S ) F 0 1 V (S ) = V (S ); 0 1 V ( S ) = V (S ) S S

Boundary Condition V (S ) = V (S + J ) F VS (S ) = mc

Optimality Condition j i V (S ) = V (S ) S S 0 1 V SS (S ) = VSS (S )

Optimality Condition VS (S ) = VS (S + J ) mc VSS (S ) = 0

Notes: represents a point on an arbitrary boundary where an action is taken S S represents a point on an optimally chosen boundary where an action is taken and J represent arbitrary and optimal jump sizes in impulse control J F and mc are xed cost and marginal cost, respectively In impulse and barrier control: use +mc if the boundary is approached from above use mc if approached from below (for cost minimization problems reverse the signs) In the discrete state problem the switch is from state 0 to state 1

TRANSITIONAL BOUNDARIES: Problem Discrete states Bang-bang

BARRIERS: Problem Impulse control Barrier control

Table 11.2: Types of Free Boundary Problems

CHAPTER 11. CONTINUOUS TIME MODELS: THEORY

339

CHAPTER 11.


340

Another important special case is the so-called stochastic bang-bang problem. Such problems arise when it is optimal to exert a bounded continuous control at either its maximum or minimum level. Eectively, therefore, there is a binary state variable that represents which control level is currently being exerted. The free boundary determines the values of the continuous variables at which it is optimal to change the binary state. A couple points should be mentioned now and borne in mind whenever considering free boundary problems. First, it is useful to distinguish between the value function of a problem at an arbitrary choice of a boundary and the optimal choice of the boundary. The value function (the present value of the return stream) using an arbitrary barrier control is described by a second order partial dierential equation subject to the appropriate boundary conditions; this is the message of the Feynman-Kac equation (see Section 10.1.2). The optimal choice of the boundary must then add additional restrictions that ensure its optimality. We therefore distinguish in Table 11.2 between a point, S , on an arbitrary boundary and a point, S , on the optimal boundary. As we shall see in the next chapter, this distinction is particularly important when using a strategy to nd the free boundary that involves guessing its location, computing the value function for that guess, and evaluating a condition that should hold at the boundary. Related to this is an understanding the number of boundary conditions that must be applied. Here are some rules that should help you avoid problems. First, any non-stochastic continuous state will have one partial derivative and will require one boundary condition. On the other hand, any stochastic state variable will have second order derivatives and will generally need two boundary conditions.13 These statements apply to both arbitrary and optimal controls. For optimality we will require an additional boundary condition for every parameter needed to de ne the control. Thus if the control is de ned by a single point in the state space, we need one additional constraint to de ne the location of that point. If the control is a curve in a 2-dimensional state space, we will need a single functional constraint, that will take the form b(S; V; VS ; VSS ) = 0 (this will become a lot clearer after you've read through some examples). The additional constraints can be derived formally by maximizing the value 13 The exception to this rule of thumb involves processes that exhibit singularities at

natural boundaries, which can eliminate the need to specify a condition at this boundary

CHAPTER 11.


341

function for an arbitrary barrier with respect to the location of the barrier, which for single points means solving an ordinary maximization problem and for functional barriers means solving an optimal control problem. In all of these cases one can proceed as before by de ning a Bellman's Equation for the problem and solving the resulting maximization problem. The main new problem that arises lies in determining the region of the state space over which the Bellman's Equation applies and what conditions apply at the boundary of this region. We will come back to these points so if they are not clear now bear with us. Now let us consider each of the main types of problem and illustrate them with some examples.

11.2.1 Impulse Control Impulse and barrier control problems arise when the reward function includes the size of the change in a state variable caused by exerting some control. Such problems typically arise when there are transactions costs associated with exerting a control, in which case it may be optimal to exert the control at an in nite rate at discrete selected times. In addition, the reward function need not be continuous in S . The idea of an in nite value for the control may seem puzzling at rst and one may feel that it is unrealistic. Consider that in many applications encountered in economics the control represents the rate of change in a state variable. The state is typically a stock of some asset measured in quantity units. The control is thus a ow rate, measured in quantity units per unit time. If the control is nite, the state cannot change quickly; essentially the size of the change in the state must grow small as the time interval over which the change is measured gets small. In many situations, however, we would like to have the ability to change the state very quickly in relation to the usual time scale of the problem. For example, the time it takes to cut down a timber stand may be very small in relation to the time it takes for the stand to grow to harvestable size. In such situations, allowing the rate of change in the state to become in nite allows us to change the state very quickly (instantaneously). Although this makes the mathematics somewhat more delicate, it also results in simpler optimality conditions with intuitive economic interpretations. Consider the single state case in which the state variable governed by

dS = [(S ) + x]dt + (S )dz

CHAPTER 11.


342

and the reward function that is subject to xed and variable costs associated with exerting the control: 8 if x < 0 < r (S ) c ( S ) F 0 if x = 0 f (S; S; x) = r (S ) : + r (S ) c+ (S ) F + if x > 0 with c (0) = c+ (0) = 0. In this formulation there are xed costs, F and F + , and variable costs, c and c+ , associated with exerting the control, both of which depend on the sign of the control. Typically, we would assume that the xed costs are positive. The variable costs, however, could be negative; consider the salvage value from selling o assets. To rule out the possibility of arbitrage pro ts (when the reward is increasing in the state: rS 0), we require that F + + c+ (z ) + F + c ( z ) > 0 for any positive z ; thereby preventing in nite pro ts to be made by continuous changes in the state. With continuous time diusion process, which are very wiggly, any strategy that involved continuous readjustment of a state variable would become in nitely expensive and could not be optimal. Instead the optimal strategy is to change the state instantly in discrete amounts, thereby incurring the costs of those states only at isolated instants of time. An impulse control strategy would be optimal when there are non-zero xed costs (F +; F > 0). Barrier control strategies (which we discuss in the next section) arise when the xed cost components of altering the state are zero. With impulse control, the state of the system is reset to a new position (a target) when a boundary is reached (a trigger). It may be the case that either or both the trigger and target points are endogenous and need to be determined. For example, in a cash management situation, a bank manager must determine when there is enough cash-on-hand (the trigger) to warrant investing some of it in an interest bearing account as well as how much cash to retain (the target). Alternatively, in an inventory replacement problem, an inventory is restocked when it drops to zero (the trigger), but the restocking level (the target) must be determined (restocking occurs inatantaneously so there is no reason not to let inventory fall to zero). A third possability arises in an asset replacement problem, where the age at which an old machine is replaced by a new one must be determined (the trigger), but the target is known (the age of a new asset).

CHAPTER 11.


343

In any impulse control problem, a Feynman-Kac Equation governs the behavior of the value function on a region where control is not being exerted. The boundaries of the region are determined by value matching conditions that equate the value at the trigger point with the value at the target point less the cost of making the jump. Furthermore, if the trigger is subject to choice, a smooth pasting condition is imposed that the marginal value of changing the state is equal to the marginal cost of making the change. A similar condition holds at the target point if it is subject to choice. For those wishing a rigorous discussion and veri cation of these points see Appendix A.

Example: Asset Replacement (Cows revisited)

In Chapter ?? we examined a discrete time and state problem concerning the optimal age to replace an asset. The speci c example involved the number of production cycles after which a milk cow should be replaced by a one-year old cow. The value of the cow depends on her current and future yield potential, which is described by y (A), where A is the state variable representing the age of the cow. The value also depends on the net price of milk, P , and the net cost of replacing the cow, c. This is a deterministic problem in which the state dynamics are simply dA = dt. The reward function is y (A)P . Thus the Bellman equation is

V (A) = y (A)P + V 0 (A): This dierential equation is solved on the range A 2 [1; A ], where A is the optimal replacement age. The boundary conditions are given by the value matching condition:

V (1) = V (A ) + c and the optimality (smooth pasting) condition:

V 0 (A ) = 0 The smooth pasting condition may not be obvious, but it is intuitively reasonable if one considers that a cow above the age A should always be immediately replaced. Once past the age of A , therefore, the value function is constant: V (A) = V (A ) = V (1) c, for A A . Also, no optimality condition is imposed at the lower boundary (A = 1) because we are not free to pick the age of the new cow.

CHAPTER 11.


344

The Bellman equation is a rst order linear dierential equation and hence can be solved analytically. In the speci c case that y (A) is quadratic in A the Bellman equation can be written as V 0 = V P (a0 + a1 A + a2 A2 ) which has solution V (A) = keA + 0 + 1 A + 2 A2 ; where k is a constant to be determined by the boundary conditions and the i can be veri ed to satisfy the recursive conditions14 2 = P a2 =

i = P ai = + (i + 1) i+1 =; i = 1; 0: To compute the values of the constant of integration, k, and the optimal replacement age A we impose the value matching and smooth pasting conditions. Although there are two unknowns here (k and A ) it is easy to eliminate the k term: Pn i A 1 +c i=0 i k= e eA to obtain the single root condition

eA

n X i=1

!

i

(Ai

1) + c +

e

eA

n X i=1

i i Ai 1 = 0;

which can be solved using any univariate root nding algorithm (see Section ??). Figure 11.2 displays the value function, with the star representing A . For values above A the value function is at: a cow that old would be immediately replaced by a one year old and hence the value function equals V (1) less the replacement cost c. The dashed curve for values A > A represents the continuation of the value function but has no meaningful interpretation. It is included to make clear that V (A) reaches a minimum at A , and therefore V 0 (A ) = 0. 14 It is straightforward to verify that if y(A) were an nth order polynomial, the solution would have the same form with n = P an= and i = P ai = + (i + 1) i+1 = for i = n 1; ::; 0:

CHAPTER 11.

345


Value of Cows 15.4

15.2

Value (1000$)

15

14.8

14.6

14.4

14.2

14

13.8 1

2

3

4

5

6

7

8

9

10

Age

Figure 11.2 Before leaving this example, a potentially misleading interpretation should be discussed. Although we have referred to the value function as representing the value of a cow of age A, this is not quite correct. The value function represents the value of the assets used to milk a cow of age A. The distinction lies in the fact that the particular cow will be replaced at age A , at which point a new cow will be used. The current cow has value equal to the discounted stream of returns it generates: Z A A 0

et P y (A + t)dt;

but the value function is

V (A) =

Z A A 0

e t P y (A + t)dt + e

(A A) V (A )

CHAPTER 11.


346

Thus the cow at age A has value

V (A) e

(A A) V (A ):

Example: Timber Harvesting

In the previous example we examined an asset replacement problem in which the asset generated a continuous stream of net returns. In some cases, however, the returns are generated only at the replacement time. Consider a forest stand that will be clear-cut on a date set by the manager. The stand is allowed to grow naturally at a biologically determined rate according to

dS = (m S )dt: The parameter m represents a biological equilibrium point. When the stand is cut, it is sold for a net return of pS . In addition, the manager incurs a cost of c to replant the stand, which now has size S = 0. The decision problem is to determine the optimal cutting/replanting stand size, using a discount rate of . The Bellman equation is

V = (m S )V 0 (S ); for S 2 [0; S ], where S is determined by boundary conditions and

V (S ) = V (0) + pS V 0 (S ) = p

c value matching smooth pasting.

If the stand starts at a size above S it is optimal to cut/replant immediately. Clearly the marginal value of additional timber when S > S is the net return from the immediate sale of an additional unit of timber. Hence, for S > S , V (S ) = V (S ) + p(S S ) and V 0 (S ) = p: The Bellman equation can be rewritten in the form V V0 = ; m S the solution to which is easily veri ed to be

V = k(m S )

= ;

CHAPTER 11.


347

where k is a constant of integration to be determined by the boundary conditions. There are, therefore, two unknowns to be determined, k and S . The value matching condition allows us to solve for k in terms of S : pS c k (S ) = (m S ) = m = The optimal S can be found using the smooth-pasting condition: k(S ) (m S )=+1 = 0; p

which can be solved with any one-dimensional root nding solver (see Section ??). The method is illustrated in Code Box 2. This code sets = p = m = 1, c = 0:15 and = 21 . The optimal harvest stand size is 0.4291, indicating that it is optimal to harvest the stand when it is less than one half its mature size (m). The value function for this problem is shown in Figure 11.3. The starred point represents the optimal harvest size. For stand sizes above this point the value function is linear. The dashed lines extend the two pieces of the value function beyond their domains to illustrate the value matching and smooth pasting conditions.

Code Box 11.2: Timber Harvesting It is convenient to normalize by setting = p = m = 1, which amounts to picking scales for time, stand size and money. When the parameters thus normalized, the economically relevant range for the cost parameter is [0; 1]; for values greater than 1 it would never be optimal to harvest because the revenues thus generated would not cover the replanting costs.15 The discount rate can be greater or less than the maximal timber growth rate, , however. Comparative static exercises, therefore, need only examine the problem for normalized parameter values of (; c) on the interval [0; 1) [0; 1].16 Figure 11.4 is a contour plot of S illustrating the behavior of the optimal harvest stand size (as a fraction of the carrying capacity, m. It can be shown that as = gets large (i.e., either the future is heavily discounted or the timber grows very slowly), the optimal harvest size declines, reaching a lower bound 15 This may not be true if the initial stand is larger than m, in which case it is optimal

CHAPTER 11.


348

Value Function for Tree Harvesting Problem 1.4

1.3

1.2

V

1.1

1

0.9

0.8

0.7 0

0.1

0.2

0.3

0.4

0.5

S

Figure 11.3 of c=p. This lower bound represents the point at which the returns from cutting just oset the cost of replanting. In this case the value function becomes identically zero. On the other hand, the optimal harvest size increases as c=p increases. Intuitively, when replanting costs are high, it pays to delay harvest. Similarly, as timber revenue increases, it pays to cut sooner. Notice that, for c = 0, we to harvest immediately if S > c=p. 16 With some algebraic manipulation one can show that, for the normalized problem, V (0) = (1 S ) V (S ) and V (S ) = (1 S )=:

CHAPTER 11.

349


Contours of Optimal Harvest Stand Sizes 0.9

0.9

0.8 0.8 0.7

0.7

c/π

0.6

0.5 0.6 0.4 0.5 0.3 0.4 0.2 0.3 0.1

0

0.2 0.1 0.5

1

ρ/α

1.5

2

2.5

Figure 11.4 get the curious result that S = 0, i.e., that timber is harvested as soon as it is planted. In this case the value function is equal to pm=. Essentially, when replanting costs are zero, one harvests continuously a crop that grows at rate m; the discounted present value of this continuous harvesting is thus 1= times the instantaneous return pm. The timber harvesting problem with replanting is known as the Faustmann problem. A related problem, known as the Fisher problem, is the determination of the optimal harvest time when the stand is abandoned after being clear-cut (i.e., it not replanted). The Fisher problem is an optimal stopping problem and it solution diers only in the boundary conditions. The smooth-pasting condition is, in fact, the same but the value matching condition is simply

V (S ) = pS :

CHAPTER 11.

350


We leave asan exercise the veri cation that the solution in this case is 1 m S = 1 1 + = For the parameters used in the numerical example, this yields an optimal cutting size of 2=3, which is larger than the optimal cutting size when replanting will occur. A comparison of the value functions are shown in Figure 11.5. With these parameters, the value of the stand with abandonment is less than half the value with replanting, indicating that replanting is the preferred management mode in this case. With other parameter values, however, this need not be true, especially if the replanting cost, c, is high. Tree Harvesting: Replanting vs. Abandonment 1.8

1.6

1.4

V

1.2 Replanting

1

0.8

0.6

Abandonment

0.4

0.2 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

S

Figure 11.5 The growth function (m S ) used above was \nice" in allowing a simple solution. It is somewhat limited in its ability to represent biological phenomena. Simple extensions, however, do not necessarily have simple solutions

CHAPTER 11.


351

and we will need to resort to numerical methods. This is also true if randomness is added to the growth process. For example, let us complicate the growth process by de ning the timber stand to be governed by

dS = (m S )(b + S )dt + (S )dz: It is reasonable to require that (0) = 0, thereby preventing p negative stand sizes. An example is the square root process (S ) = S . Furthermore, with b > 0, the instantaneous mean is quadratic with a positive root, m, and a negative root b; if b < m the mean is increasing for S < (m b)=2, whereas if b > m the mean is decreasing for all positive S . The Bellman's equation is 0

V = (m S )(b + S )V (S ) + 21 2 SV 00 (S ); with the same boundary conditions as in the deterministic problem.17 We defer discussion of numerical solution of this problem to the next chapter (Section ??).

11.2.2 Barrier Control In barrier control problems it is optimal to maintain the state within a region by keeping it on the region's boundary whenever it would otherwise tend to move outside of it and to do nothing when the state is in the interior of the region. This, of course, assumes that the state is suÆciently controllable so that such a policy is feasible. Barrier control problems can be thought of as limiting cases of impulse control problems as the size of any xed costs go to zero. When this happens, the size of the jump goes to zero, so the trigger and target points become equal. This represents something of a dilemma because the value matching condition between the target and jump points becomes meaningless when these points are equal. The resolution of this dilemma is to shift the value matching condition to the rst derivative and the smooth pasting to the second derivatives.

Example: Capital Investment

Consider an investment situation in which a rm can add to its capital stock,

17 It might seem odd that introducing the second order term, VSS , in the stochastic

case does not require an additional boundary condition. This is a case, however, in which a singularity exists at the boundary S = 0 because (0) = 0 (see discussion in Section 11.1.2).

CHAPTER 11.


352

K , at a cost of c per unit. The capital produces output at rate q (K ) and the net return on that output is P . Hence the reward function facing the rm is f (K; P; I ) = P q (K ) cI: K is clearly a controllable state, with dK = Idt: P , on the other hand, is stochastic and is assumed to be governed by dP = P dt + P dz; (geometric Brownian motion). Using a discount rate of , the Bellman equation for this problem is

V (K; P ) = P q (K ) cI + IVK (K; P ) + P VP (K; P ) + 21 2 P 2 VP P (K; P ): There are, however, no constraints on how fast the rm can add capital and hence it is reasonable to suppose that, when it invests, it does so at an in nite rate, thereby keeping its investment costs to a minimum. The optimal policy, therefore, is to add capital whenever the price is high enough and to do so in such a way that the price remains on or below a curve P (K ). Below this curve no investment takes place and the value function therefore satis es

V (K; P ) = P q (K ) + P VP (KP ) + 21 2 P 2 VP P (KP ): This is a simpler expression because, for a given K , it can be solved more or less directly. It is easily veri ed that the solution has the form

V (K; P ) = A1 (K )P + A2 (K )P + 1

2

P q (K )

where the i solves 12 2 ( 1) + = 0. It can be shown that 2 < 0 < 1 < 1 . For the assumed process for P , 0 is an absorbing barrier so the term associated with the negative root must be forced to equal zero by setting A2 (K ) = 0 (we will henceforth drop the subscripts on A1 (K ) and 1 ). At the barrier, the marginal value of capital must just equal the investment cost:

VK (K; P ) = c:

(14)

CHAPTER 11.


353

Consider now the situation in which the rm nds itself above the barrier (for whatever reason). The optimal policy is immediately to invest enough to bring the capital stock to the barrier. The value of the rm for states above the barrier, therefore, is equal to the value at the barrier (for the same P ) less the cost of the new capital: V (K; P ) = V (K (P ); P ) c(K (P ) K ) where K (P ) is the inverse of P (K ). This suggests that the marginal value of capital when the state is above the barrier equals c and hence does not depend on the current price. Thus, in addition to (14), it must be the case that

VKP (K; P ) = 0: (15) The barrier conditions (14) and (15) can be solved to show that c( ) P (K ) = ( 1)q 0 (K ) and 1 1 q 0 (K ) 0 : A (K ) = c Notice that to determine A(K ) and therefore to completely determine the value function, we must solve a dierential equation. The optimal policy, however, does not depend on knowing V , and, furthermore, we have enough information now to determine the marginal value of capital for any value of the state (K; P ). Examples of the optimal trigger price curve are displayed in Figure 11.6 using the parameters = 0 = 0:2 = 0:05 c = 1 and two alternative speci cations for q (K ): q (K ) = ln(K + 1) p q (K ) = K:

CHAPTER 11.

354


Barrier Control Investment Trigger Curve 0.5 q(k)=ln(k+1) 0.45

0.4

0.35

0.3

P

q(k)=sqrt(k) 0.25

0.2

0.15

0.1

0.05

0 0

1

2

3

4

5

6

7

8

9

10

K

Figure 11.6

11.2.3 Discrete State/Control Problems We turn now to problems involving transitional boundaries. These can arise because there are discrete states and the optimal time to move from one state to another must be determined. For example, the investment problem described above as an optimal stopping problem can be made more complicated and realistic by allowing disinvestment. The problem then becomes one of determining the value of the project if active, given that one can deactivate it, together with the value of the project if inactive, given that it can be activated. The state here is a stochastic variable that determines the return stream of the activated project. The solution involves two boundaries, one which determines when the project should be activated (given that it is currently inactive), the other when it should be deactivated (given that it is

CHAPTER 11.


355

currently active).18 The hallmark of transitional boundary problems is that there is a distinct value function on either side of the boundary and there are conditions that must apply to both of these functions at the boundary. Thus the boundary and the value functions on both sides must all be simultaneously determined. For arbitrary speci cations of the boundary, we require that the two value functions are equal at the boundary (value matching) and for the optimal boundary, we require that their derivatives are equal at the boundary (smooth-pasting or high contact).

Example: Entry/Exit

Let us develop in more detail the entry/exit problem mentioned above. A rm can either be not producing at all or be actively producing q units of a good per period at a cost of c per unit . The state variable in this case is the return per unit of output, P , which is a geometric Brownian motion process:

Pt = P dt + P dz: We assume there are xed costs of activating and deactivating of I and E , with I + E 0 (to avoid arbitrage opportunities). The value function is

V (P; Æ ) = E

Z 1 0

e

t Æ (P

c)dt

the discounted costs of switching states;

where Æ = 1 if active, 0 if inactive. For positive transition costs, it is reasonable that such switches should be made infrequently. Furthermore it is intuitively reasonable that the optimal control is to activate when P is suÆciently high, P = Ph , and to deactivate when the price is suÆciently low, P = Pl . It should be clear that Pl < Ph , otherwise in nite transactions costs would be incurred. The value function can therefore be thought of as a pair of functions, one for when the rm is active, V a , and one for when it is inactive, V i . The former is de ned on the interval [Pl ; 1), the latter on the interval [0; Ph]. On the interior of these regions the value functions satisfy the Feynman-Kac equations

V a = P c + P VPa + 2 P 2 VPaP : V i = P VPi + 2 P 2 VPi P

(16)

18 More accurately, there is an additional, binary, state variable that is 0 if the project

is inactive and 1 if it is active. The control is binary, being equal to 0 if the project is active and 1 if it is inactive.

CHAPTER 11.


356

At the upper boundary point, Ph , the rm will change from being inactive to active at a cost of I . Value matching requires that the value functions dier by the switching cost: V i (Ph ) = V a (Ph ) I . Similarly at the point Pl the rm changes from an active state to an inactive one; hence V i (Pl ) E = V a (Pl ). Value matching holds for arbitrary choices of Pl and Ph. For the optimal choices the smooth pasting conditions must also be satis ed:

VPi (Pl ) = VPa (Pl ) and

VPi (Ph ) = VPa (Ph ): For a geometric Brownian motion process the solution is known for arbitrary levels of Pl and Ph . The general form of the solution is

V a = Aa1 P + Aa2 P + P=( ) c= 1

2

V i = Ai1 P + Ai2 P 1

2

where the four A terms will be pinned down by the boundary conditions and the solve 1 2 2 (

1) +

= 0:

It can be shown that, for > 0, one of the is negative and the other is greater than one; de ne 1 > 1 and 2 < 0. (It is easy to verify that these solutions solve (16)). Two of the unknown constants can be eliminated by considering the boundary conditions at P = 0 and P = 1. At P = 0 only V i is de ned and the geometric Brownian motion process is absorbed; hence V i (0) = 0, which requires that Ai2 = 0. For large P , only V a is de ned and the probability of deactivation becomes vanishingly small; hence the value function would approach P=( ), requiring that Aa1 = 0. We still have two unknown constants to determine, Ai1 and Aa2 (we shall henceforth refer to these as A1 and A2 , as there is no possible confusion concerning which function they belong to). The value matching conditions require that,

V a (Ph) I = A2 Ph + Ph=( ) c= I = A1 Ph = V i (Ph) 2

1

CHAPTER 11.


357

and

V a (Pl ) = A2 Pl + Pl =( ) c= = A1 Pl 2

E = V i (Pl ) E:

1

The optimality conditions on Pl and Ph are that the derivatives of V a and V i are equal at the two boundary locations:

VPa (P ) = 2 A2 P

2

1

+ 1=( ) = 1 A1 P

1

1

= VPi (P )

at P = Pl and P = Ph . Taken together, the value matching and smooth pasting conditions yield a system of four equations in four unknowns, A1 , A2 , Pl and Ph . The optimal value functions for a numerical example are illustrated in Figure 11.7. Exogenous parameter values are

I E c 0 0.2 0.05 5 6 1 The endogenous parameters are computed to be

1 2 A1 A2 Pl Ph 2.1583 -1.1583 3.6299 2.2546 0.4182 2.1996 The optimal boundaries are noted in Figure 11.7 by dashed vertical lines. At Pl , V i is greater than V a by the amount E (the cost of switching from active to inactive), whereas, at Ph , V a exceeds V i by the amount I (the cost of activating). While less obvious the slopes of the value functions are equal at these two points, as required by the smooth pasting condition. The gure includes extensions of the value functions beyond the ranges for which they are de ned. Thus, although, illustrated, V a for points less than Pl are meaningless in the context of the problem. Similarly, for points above Ph , the function V i is unde ned. We show them to make the smooth pasting conditions easier to see and to point out that the extensions would not satisfy the natural boundary conditions on the problem at P = 0 and P = 1. Another useful way to view the problem is to graph the function G(P ) = a V (P ) V i (P ) (including the extensions of these functions). Several alternative G(P ) functions are displayed in Figure 11.8. The one labeled e = 1 corresponds to the parameters used to generate Figure 11.7. The smooth pasting conditions ensure that the points P = Pl and P = Ph are stationary points, which occur at the local minimum and maximum values of G(P ); the value matching conditions ensure that G(Pl ) = E and G(Ph) = I .

CHAPTER 11.

358


Value Functions for Entry/Exit Problem 30 Pl

Ph

25

Va

20

V(P)

15

i

10

V

5

0

−5

−10 0

0.5

1

1.5

2

2.5

P

Figure 11.7 Consider now what happens as the switching costs get small, say by multiplying I and E by a smaller and smaller number e. The eect on G(P ) is shown in Figure 11.8. As e ! 0 the values of Pl and PH collapse towards each other and the local minimum and maximum values at these points collapse towards zero. In the limit at e = 0 it is intuitive that Pl = Ph = P . Furthermore, at P = P the function G(P ) must exhibit an in ection point: GP P (P ) = 0. If we impose the three conditions that V a and V i match up to their second derivatives at P , we can determine the values of A1 , A2 and P . Some tedious algebra will reveal the intuitively reasonable (perhaps obvious) result that P = c, i.e., the optimal policy is to be active if the current price covers the variable costs (c) and to be inactive otherwise.

CHAPTER 11.

359


G(P) Functions for Entry/Exit Problem 8

6

e=1

4

e=3/4 2

G(P)

e=1/2 0 e=1/4 −2

−4

−6 e=0 −8 0

0.5

1

1.5

2

2.5

P

Figure 11.8 Although far from a proof, we hope the intuition behind the limiting process here is clear. The situation is much like passing from an impulse control problem to a barrier control problem as the xed costs of taking an action go to zero. In that case, as here, the value matching and smooth pasting conditions need to be supplemented with a zero condition on the second derivative. As we shall see in the next section, a similar situation holds in the case of bang-bang control problems. MATLAB code to solve the entry/exit problem is displayed in Code Box 3. The code is relatively simple, the rst section returning values of the four residual equations and the second initializing variables and calling the root nding algorithm. There is one point of interest in the way the code is written. We use the logs of Pl and Ph rather than their levels to prevent negative values from causing the program to act badly (in MATLAB taking

CHAPTER 11.


360

a non-integer power of a negative number results in a complex number which it is better to avoid; the program will continue to run even though it is producing garbage). Specifying the prices in logs does not guarantee that a solution will be found, but it is far more robust to poor starting values than if the levels are used.

Code Box 11.3: Entry/Exit Problem

Optimal Stopping Problems The optimal stopping problem is in many ways the simplest of the free boundary problems and arises in situations involving a once and for all decision. For example, suppose a rm is attempting to decide whether a certain project should be undertaken. The value of the project depends on a stochastic return that the project, once developed, will generate. The state variable can therefore be taken to be the present value of the developed project. Furthermore, the rm must invest a speci ed amount to develop the project. In this simple framework, the state space is partitioned into a region in which no investment takes place (when the present value of the developed project is low) and a region in which the project would be undertaken immediately. The boundary between these two areas represents the value of the state, that, if reached from below, would trigger the investment. It is important to emphasize that optimal stopping problems, although they have a binary control, dier from other binary control problems in that one value of the control pays out an immediate reward, after which no further decisions are made. The one time nature of the control makes the problem quite dierent from and, actually, easier to solve than problems with binary controls that can be turned on and o. Stopping problems in continuous time are characterized by a random state governed by dS = (S )dt + (S )dz; a reward stream f (S ) that is paid so long as the process is allowed to continue and a payout function R(S ) that is received when the process is stopped (for now we consider only in nite time discounted time autonomous problems; this will be relaxed presently).

CHAPTER 11.


361

Another way to view the stopping problem is as a problem of choosing an optimal time to stop a process. This leads to the following formal statement of the problem

V (S ) = max E

"Z t (S )

t (S )

0

e

f (S )d

+e

t (S )

#

R(S ) :

This value function is described by the dierential equation

V (S ) = f (S ) + (S )VS (S ) + 12 2 (S )VSS (S )

(17)

The optimal control problem consists of nding the boundary between the regions on which the process should be stopped and those on which it should be allowed to continue. For the present, assume that there is a single such switching point, S , with S < S indicating that the process should be allowed to continue. Thus the dierential equation is satis ed on [S; S ], where S is a (known) lower bound on the state. Any speci c choice of a control consists of a choice of the stopping point, say S . At this point the value function, to be continuous, must equal the reward

V (S ) = R(S ); known as the value-matching condition. The optimal choice of S is determined by the smooth pasting condition

VS (S ) = R0 (S ); the optimal choice of S makes the derivative of the value function equal the derivative of the reward function at the boundary between the continuation and stopping regions. Intuitively, the value matching and smooth pasting conditions are indierence relations; at S the decision maker is indierent between continuing and stopping. The value function must, therefore, equal the reward and the marginal value of an additional unit of the state variable must be equal regradless of whether the process is stopped or allowed to continue. This is the simplest of the optimal stopping problems. We can make them more complex by allowing time to enter the problem either through nonautonomous rewards, state dynamics or stopping payment or by imposing a nite time horizon. In the following example we examine a nite horizon problem.

CHAPTER 11.


362

Example: Exercising an American Put Option An American put option, if exercised, pays K P , where K is the exercise or strike price and P is the random price of the underlying asset, which evolves according to

dP = (P )dt + (P )dz: The option pays nothing when it is being held, so f (P ) = 0. Let T denote the option's expiration date, meaning that it must be exercised on or before t = T (if at all). In general, the option is written on a traded asset so we may use the form of the Bellman's Equation that is discounted at the risk-free rate and with mean function replaced by rP ÆP (see Section 10.1.3):

rV = Vt + (rP

ÆP ) VP + 21 2 (P )VP P

on the continuation region, where Æ represents the income ow (dividend, convenience yield, etc.) from the underlying asset. Notice that the constraint that t T means that the value function is a function of time and so Vt must be include in the Bellman's Equation. The solution involves determining the optimal exercise boundary, P (t). For puts P (t) is a lower bound so the continuation region on which the Bellman's Equation is de ned is [P ; 1). The boundary conditions for the put option are

V (P; T ) = max(K V (P ; t) = K P VP (P ; t) = 1

P; 0) (terminal condition) (value matching) (smooth-pasting)

and,

V (1; t) = 0:

Example: Machine Abandonment

Consider a situation in which a machine produces an output worth P per unit time, where

dP = P dt + P dz; i.e., that P is a geometric Brownian motion process. The machine has an operating cost of c per unit time. If the machine is shut down, it must

CHAPTER 11.


363

be totally abandoned and thus is lost. Furthermore, at time T , the machine must be abandoned. At issue is the optimal abandonment policy for an agent who maximizes the ow of net returns from the machine discounted at rate . For the nite time case de ne as equal to the time remaining until the machine must be abandoned, so = T t and d = dt. The optimal policy can be de ned in terms of a function, P ( ); for P > P ( ) it is optimal to keep the machine running, whereas for P < P ( ) it is optimal to abandon it. The current value of the operating machine satis es the Bellman's equation

V = P

c V + P VP + 21 2 P 2 VP P :

and boundary conditions

V (P; 0) = 0 terminal condition VP (1; ) = (1 e )=( ) natural boundary condition V (P ; ) = 0 value matching condition VP (P ; ) = 0 smooth pasting condition The rst boundary condition states that the machine is worthless when it must be abandoned. The second condition is derived by considering the expected value of a machine that is never abandoned: P c V (P; ) = 1 e (the derivation of this result is left as an exercise; p. 312). An alternative upper boundary condition is that VP P (1; ) = 0. The remaining two conditions are the value matching and smooth pasting conditions at P ( ). Consider rst the in nite horizon case, which corresponds to the situation that the machine never need be abandoned at any xed time. It still may be optimal to abandon it if the price is very low, because the odds that the price rises suÆciently fast would not justify taking current losses from operating the machine. Clearly, P (1) must be less than c and will equal c when there is no uncertainty ( = 0). To determine P (1) we solve the optimality conditions when V is not a function of . We have seen this problem before; its solution is V (P ) = A1 P + A2 P + P=( ) c=; 1

2

CHAPTER 11.


364

where solves 1 2 ( 2

1) +

= 0:

where A1 and A2 are constants to be determined by the boundary conditions. For economically meaningful parameter values, one of the is negative and the other greater than 1. To satisfy the boundary condition as P ! 1, we set A1 = 0, where 1 is the positive root. The value matching and smooth pasting conditions are

AP (1) + P (1)=( ) c= = 0 and

AP (1) 1 + 1=( ) = 0; which are solved by ( ) P (1) = c ( 1) and P (1)1 A= ( ) It should be noted that the ability to derive the in nite horizon cuto price depends on the assumption that P is a geometric Brownian motion process. Also, even in the geometric Brownian motion case, the nite horizon problem does not possess a closed-form solution and hence must be computed numerically. The nature of the problem is demonstrated in Figure 11.9. Notice that P (0) = c, i.e., that as the date at which the machine must be abandoned is reached, there is no point operating it unless it is currently pro table to do so.

11.2.4 Stochastic Bang-Bang Problems Bang-bang control problems arise when both the reward function and the state transition dynamics are linear in the control and the control is bounded. In such cases it is optimal to set the control at either its upper or lower bound. The control problem thus becomes one of dividing the state space into a set of points at which the control is at its upper bound and a set at which it is at

CHAPTER 11.

365


Optimal Timing of Machine Abandonment 1

0.95 µ=0 σ = 0.2 ρ = 0.1 c=1

0.9

P

0.85

0.8

0.75

0.7

0.65

0.6 0

2

4

6

8

10

τ

12

14

16

18

20

Figure 11.9 its lower bound. Equivalently, the problem is to nd the boundary between the two sets. If there is no cost to switching the control from the lower to upper bound, we are in precisely the same situation that we discussed in the last section when the switching costs go to zero. The optimal value function and control is found in a similar fashion: de ne a Feynman-Kac Equation on each side of the boundary and require that the value functions on either side of the boundary are equal up to their second derivative. The general bang-bang problem has reward function of the form

f0 (S ) + f1 (S )x and state dynamics of the form

dS = [g0 (S ) + g1 (S )x]dt + (S )dz:

CHAPTER 11.


366

Furthermore the control is constrained to lie on a given interval:

xa x xb : The Bellman's equation for this problem is 1 2 V = max f 0 (S ) + f1 (S )x + [g0 (S ) + g1 (S )x]VS + 2 (S )VSS x

subject to the control constraint. The Karush-Kuhn-Tucker conditions for this problem indicate that

x=

8 > < > :

xa if VS (S ) > fg1 ((SS )) 1 f xb if VS (S ) < g1 ((SS )) 1

This suggests that there is a point, S , at which

f1 (S ) + g1 (S )VS (S ) = 0:

(18)

Assuming that VS is decreasing in S , this suggests that we must solve for two functions, one for S < S that solves a V a = f0 (S ) + f1 (S )xa + [g0 (S ) + g1 (S )xa ]VSa + 12 2 (S )VSS

(19)

b : V b = f0 (S ) + f1 (S )xb + [g0 (S ) + g1 (S )xb ]VSb + 21 2 (S )VSS

(20)

and the other for S > S that solves

We will need three side conditions at S to completely specify the problem and to nd the optimal location of S , namely that

V a (S ) = V b (S ) VSa (S ) = VSb (S ) a (S ) = V b (S ): VSS SS Combining these conditions with (19) and (20) we see that [f1 (S ) + g1 (S )VS (S )] xa = [f1 (S ) + g1 (S )VS (S )] xb : Clearly this can only be true when the term in the [ ]s is zero, which gives us the optimality result (18).

CHAPTER 11.


367

Example: Harvesting a Renewable Resource

To illustrate the problem consider a manager of a biological (renewable) resource who must determine the optimal harvesting strategy. The state variable, the stock of the resource, is stochastic, uctuating according to

dS = [S (1 S ) hS ]dt + Sdz; where h, the control, is the proportional rate at which the resource is harvested. Assume that the per unit return is p and that 0 h C . The manager seeks to solve

V (S ) = max E h

Z 1 0

e

t phSdt :

In the notation of general problem, xa = 0, xb = C , f0 (S ) = 0, f1 (S ) = pS , g0 (S ) = S (1 S ) and g1 (S ) = S . The Bellman equation for this problem is

V = max phS + (S (1 S ) hS ) VS + 21 2 S 2 VSS : h The assumptions the stock dynamics imply that V (0) = 0 (once the stock reaches zero it never recovers and hence the resource is worthless). At high levels of the stock, the marginal value of an additional unit to the stock becomes constant and hence VSS (1) = 0. The rst order conditions for this problem suggest that it is optimal to set h = C if VS < p and set h = 0 if VS > p. The interpretation of these conditions is straightforward: only harvest when the value of a harvested unit of the resource is greater than an unharvested one and then harvest at maximum rate. Thus the problem becomes one of nding the sets

S 0 = fS : VS > pg and

S C = fS : VS < pg where V and V

S (1 S )VS

1 2 2 2 S VSS

(S (1 S ) CS ) VS

=0

1 2S 2V SS 2

on S 0

pCS = 0 on S C

CHAPTER 11.

368


The solution must also satisfy the boundary conditions at 0 and 1 and the continuity conditions at any points S such that VS (S ) = p. The fact that S (1 S ) hS is concave in S implies that S will be a single point, with S 0 = [0; S ) and S C = (S ; 1). Figure 11.10 illustrates a numerical approximation to the value function for the problem with p = C = 1, = 0:05, = 0:1, = 0:2. Figures 11.11 and 11.12 display the rst and second derivatives of the value function. Notice that the second derivative has a kink point at S . This illustrates the continuity of V and its rst two derivatives when S is choosen optimally, but also suggests that attempting to approximate the value function with single smooth function will prove problematic. We return to this issue in the next chapter (Section ??). Value Function 1.4

1.2

1

V

0.8

0.6

0.4

0.2

0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

S

Figure 11.10 The stochastic bang-bang problem generally requires numerical methods to nd the optimal switching point. Solutions to deterministic versions of the

CHAPTER 11.


Example Box 11.4: Harvesting a Renewable Resource Problem: max E

Z 1

h

0

e

t phSdt

s.t.

dS = [S (1 S ) hS ]dt + Sdz; and 0hC

Variables: h the proportional harvest rate (the control) S the stock of the resource (the state) Parameters: ; ;

Bellman equation V = max phS + (S (1 S ) hS ) VS + 21 2 S 2 VSS : h

Boundary Conditions: V (0) = 0 VSS (1) = 0

Optimality Conditions: h = C if VS < p h = 0 if VS > p or

VS (S ) = p

369

CHAPTER 11.

370


Marginal Value Function 5

4.5

4

3.5

V’

3

2.5

2

1.5

1

0.5

0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

S

Figure 11.11 problem, however, can often be found directly or by simple application of a root nding algorithm. In the deterministic version, it can be shown that the value matching condition implies the smooth-pasting condition. Also there is no need to specify anything about the second derivative because it drops out of the Bellman's Equation when = 0. The optimal switching point can be shown to satisfy the following condition (the proof is tedious so we've put it in an appendix)

d f0 (S ) g0 (S ) fg1((SS )) 1 f (S ) 1 + 1 = 0: dS g1 (S ) Notice also that the optimal trigger stock does not depend on the capacity constraint levels (xa and xb ). There is a condition that we have not mentioned, however, that is needed for a well de ned solution. We require

CHAPTER 11.

371


Curvature of Value Function 0

−0.1

−0.2

−0.3

V"

−0.4

−0.5

−0.6

−0.7

−0.8

−0.9

−1 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

S

Figure 11.12 that

xa < g0 (S )=g1 (S ) < xb : This condition ensures that setting x = xa when S < S will cause the stock level to increase and setting x = xb when S > S will cause the stock to decrease (technically, this says that S is a controllable process). As an example consider the deterministic version of the optimal harvesting problem. The reward function is

f (S; h) = phS and the state is governed by

dS = [S (1 S ) hS ]dt:

CHAPTER 11.


372

The optimality condition becomes dS (1 S ) p = 0; p dS which is solved at S = 21 (1 =). Note that the maximum sustainable yield occurs at S = 12 , so this rule suggests it is optimal to harvest to the point that the sustained yield is less than maximum (so long as > 0). Also notice that when < we are in a situation where the stock grows so slowly that it is optimal to drive it to extinction.19 Consider what happens as we let the capacity constraint, C get large. In the limit, as C ! 1, the problem is transformed onto a barrier control problem in which it is optimal to maintain the stock level at or below a free boundary point P . In this case it would only be necessary to determine the value function below the boundary. This would satisfy the Bellman's Equation

V = S (1 S )VS + 12 2 S 2 VSS along with the boundary conditions:

VS (S ) = p and

VSS (S ) = 0: An intuitive way to see that these conditions are correct is to consider the value of an addition unit of the resource when S S . The additional unit would be harvested immediately and generate a return of p. This means that the value function is linear for S S and hence the second derivative of the value function must be zero. Continuity of the rst and second derivatives at S then gives us the two boundary conditions.

Example: Production with a Learning Curve

More complicated bang-bang problems arise when there are two state variables. The free boundary is then a curve, which typically must be approximated. An example comes from Majd and Pindyck (1989), which develops

19 To avoid misunderstanding we stress that this simple model only values the resource

for the money its harvesting brings; no normative implications about the social value of this rule should be drawn.

CHAPTER 11.


373

a model of production with learning-by-doing. Up to some limit, a rm can reduce it marginal costs the more of a good it produces. It therefore has an incentive to produce more than it otherwise might due to the future cost reductions it thereby achieves. In their model, Majd and Pindyck assume that marginal and average costs are constant at any point in time but decline at an exponential rate in cumulative production until a minimum marginal cost level is achieved. They derive an optimal production rule for a rm maximizing the present value of returns (price less cost times output) over an in nite horizon. The model is summarized in Example Box 5. The Bellman equation (shown in the Example Box) uses the risk-free discount, r, and the \risk-free" mean, r Æ (see Section 10.1.3 for discussion). Notice that Bellman's Equation is linear in output (x) and hence the solution is of the bang-bang variety. The optimal control satis es the KKT conditions:

0, x 0; and C.S.

P

C (Q) + VQ

xc

x 0, 0; and C.S.

and These conditions are satis ed by choosing x to equal either 0 or xc according to:

x = 0 if P + VQ < C (Q) x = xc if P + VQ > C (Q): Substituting the optimal production rate into the Bellman Equation and rearranging yields

rV (P; Q) = (r

Æ )P VP (P; Q) + 21 2 P 2VP P + max(0; P

a partial dierential equation. The boundary conditions for this problem require that

V (0; Q) = 0 VP (1; Q) = xc =Æ V (P; Qm ) = V (P ) (de ned below)

C (Q) + VQ (P; Q))xc;

CHAPTER 11.


374

Example Box 11.5: Production with a Learning Curve Problem:

max x

Z 1 0

e

t (P

C (Q)) x dt

s.t.

dP = P dt + P dz dQ = x dt

Q if Q < Qm C (Q) = ce

Q m ce = c if Q Qm 0 x xc

price transition equation cumulative production identity marginal cost function control constraint

Variables: P output price (uncontrolled state) Q cumulative production (controlled state) x current production rate (control) Parameters:

r Æ c c Qm xc

rate of expected price appreciation discount rate risk free interest rate rate of return shortfall ( ) initial marginal cost minimum marginal cost minimum production associated with minimum cost maximum production rate (capacity)

Bellman's Equation:

rV = max (P x

C (Q)) x + xVQ + (r

s.t. 0 x xc Optimal Control (bang-bang):

P P

C (Q) + VQ < 0 C (Q) + VQ > 0

) x=0 ) x = xc:

Æ )P VP + 21 2 P 2VP P

CHAPTER 11.


375

and that V , VP , and VQ be continuous. The rst boundary condition re ects the fact that 0 is an absorbing state for P ; hence is P reaches 0, no revenue will ever be generated and hence the rm has no value. The second condition is derived from computing the expected revenue if the rm always produces at maximum capacity, as it would be if the price were to get arbitrarily large (i.e., if the probability that the price falls below marginal cost becomes arbitrarily small). The derivative of the expected revenue is xc =Æ . The third boundary condition is a \terminal" condition in Q. Once Qm units have been produced the rm has reached its minimum marginal cost. Further production decisions do not depend on Q nor does the value of the rm, V . An explicit solution can be derived for Q > Qm : (

A1 P if P c P c A2 P + Æ r if P c; where the solve the quadratic equation V (P ) = 1 2 (1 2

1 2

) + (r

Æ )

r=0

and the A1 and A2 are computed using the continuity of V and VP . The continuity requirements on the value function, even though the control is discontinuous, allow us to determine a free boundary between the regions of the state space in which production will and will not occur. Intuitively, there is a function P (Q) above which the price is high enough to justify current production and below which no production is justi ed. Notice that below the free boundary the Bellman's equation takes a particularly simple form

rV (P; Q) = (r

Æ )P VP (P; Q) + 12 2 P 2VP P ;

which together with the rst boundary condition (V (0; Q) = 0), is solved by

V (P; Q) = A1 (Q)P ; 1

where A1 (Q) is yet to be determined. Above the boundary, however, there is no closed form solution. A1 (Q); P (Q) and V (P; Q) for P P must be computed numerically. Figure 11.13 illustrates the problem using the base parameters in Majd and Pindyck. Solution methods for this problem are presented in the next chapter (Section ??). The solution methods for this problem depend on being able to determine the position of the free boundary. It is therefore worth exploring some of the

CHAPTER 11.

376


Sequential Learning Problem Q

m

50

Production Region: V(P,Q) must be

40

Production

computed numerically

Region: V known 30

P

P*

20

10

Non−Production Region: Non−Production

A(Q) computed from

Region:

value matching condition

V known 0 0

5

10

15

20

25

Q

Figure 11.13 consequences of the continuity conditions on V . First, consider the known form of the value function below the free boundary and its derivative:

V (P; Q) = A1 (Q)P

1

VP (P; Q) = 1 A1 (Q)P

1

1:

Eliminating A1 (Q) yields

P VP (P; Q) = 1 V (P; Q): This condition holds everywhere below the boundary and at it as well. By the continuity of the V and VS , it must also hold as the boundary is approached from above.

CHAPTER 11.


377

Another relationship that is useful to note concerns the continuity in the Q direction. Below the boundary,

VQ(P; Q) = A01 (Q)P : 1

The derivative of A1 is constant in P and may therefore be related to VQ as it approaches the boundary from above, which is known from the Bellman equation:

VQ(P; Q) = A01 (Q)P

1

= (LV (P ; Q)

(P

P C (Q))) P

1

where the dierential operator L is de ned as

LV (P; Q) = rV (P; Q)

(r

Æ )P VP (P; Q)

1 2 2 2 P VP P (P; Q)

But we have already seen that P C (Q) + VQ(P ; Q) = 0 and therefore LV (P ; Q) = 0. Summarizing these results, we see that

VQ(P; Q) =

(P C (Q)) LV (P; Q) (P

P 1 P

for P C (Q)) for P

P P

It is clear in this expression that VQ is continuous at P .20 20 We should note that our treatment diers somewhat from that of Majd and Pindyck. They discuss only two boundary conditions at P (Q), value matching and P C (Q) + VQ (P ; Q) = 0. To see that this is insuÆcient, consider the following form for the value function above the free boundary

V (P; Q) = A2 P 2 + P=Æ P (Q)=r where

P (Q) =

r e (Qm

Q) + e r(Qm Q)

r+

c

This function satis es the Bellman equation, and the condition that P C (Q) + VQ (P ; Q) = 0: Below the boundary the solution has the form A(Q)P 1 , so A(Q) is a free parameter that is determined by the value matching condition (it can be shown that this is the optimal boundary for the deterministic problem).

CHAPTER 11.


378

11.3 End Notes 11.3.1 Bibliographic Notes The renewable resource harvesting problem is from Pindyck, the optimal investement from Cox, Ingersoll and Ross, the portfolio choice example from Merton. Free boundary problems are increasingly common in economics. Dixit (1991), Dixit (1993) and Dixit and Pindyck contain useful discussions of these problems. Several of the examples are discussed in these sources. The original solution to the timber harvesting problem with replanting is attributed to Martin Faustmann, who discussed it in an article published in 1849. Irving Fisher discussed the related problem with abandonment in The Theory of Interest. For further discussion see Ganey (1960), Hershleifer (1970). To our knowledge, the problem has never been discussed in print as a stochastic continuous time problem. The entry/exit example originates with Brennan and Schwartz and McDonald and Seigel. Numerous authors have discussed renewable resource management problems. The bang-bang formulation is discussed most fully in a series of papers by Ludwig, where detailed proofs can be obtained. The proof in the appendix to this chapter is modeled after a similar proof in Ludwig (19??). The cow replacement examples originate with the authors.

CHAPTER 11.


379

11.3.2 References Faustmann, Martin, \On the Determination of the Value Which Forestland and Immature Stands Possess for Forestry." Trans. M. Gane. Oxford Institute Paper 42 (1968). Fisher, Irving. The Theory of Interest. Ganey, M. Concepts of Financial Maturity of Timber and Other Assets. A.E. Information Series #62, Department of Agricultural Economics, North Carolina State College, 1960. Hershleifer, J. Investment, Interest and Capital. Englewood Clis, NJ, 1970. Merton, Robert C. \An Asymptotic Theory of Growth Under Uncertainty." Review of Economic Studies. 42(1975):375-393. [INCOMPLETE]

Appendix A: Deriving the Boundary Conditions for Resetting Problems It is instructive to view the resetting problem from another perspective. In a simple resetting problem an asset is replaced at a discrete set of times when S = S , at which point a reward, f (S ) is obtained. Let us de ne (S; S ) to be the (random) time until the state rst hits S , given that it is now equal to S . The rst time the state hits S a reward worth f (S )e (S;S ) (in current units of account) will be generated and the state is reset to 0. The time elapsing after a resetting until the state next hits S depends on a random variable that has the same distributional properties as (0; S ) and is independent of previous hitting times (by the Markov property). The expected discounted rewards (i.e., the value function) can be therefore be written as 1 (S;S ) X i V (S ; S ) = f (S )E e E e (0;S )

i=0

f (S )E e (S;S ) = : 1 E [e (0;S ) ] To simplify the notation, let

(S; S ) = E e

(S;S ) ;

CHAPTER 11.


380

so the value function is f (S ) (S; S ) : V (S ; S ) = 1 (0; S ) From the de nition of it is clear that (S ; S ) = 0 so (S ; S ) = 1. Hence the boundary condition that f (S ) V (S ; S ) = 1 (0; S ) Combining this with the lower boundary condition f (S ) (0; S ) V (0; S ) = 1 (0; S ) leads to the value matching condition that

V (S ; S ) = V (0; S ) + f (S ): Notice that value matching does not indicate anything about the optimality of the choice of S . One way to obtain an optimality condition is to set the derivative of V (S; S ) with respect to S equal to zero. After suitable rearrangement the FOC is, for evert S , @ (S; S ) (S; S ) @ (0; S ) 0 f (S ) (S; S ) + f (S ) + = 0: (21) @S 1 (0; S ) @S In order to show that this is equivalent to the smooth pasting condition we will use two properties of . First, (S ; S ) is identically equal to 1, so dS = 1: dS S =S This implies that d (S ; S ) @ (S ; S ) @ (S ; S ) = + =0 dS @S @S and hence that @ (S ; S ) @ (S ; S ) = : @S @S The second fact, a result of the Markov assumption, is that

(S; S + dS ) = (S; S ) (S ; S + dS ):

CHAPTER 11.


381

taking limits as dS ! 0 we see that

@ (S; S ) ) @ (S ; S ) : = ( S; S @S @S If we evaluate (21) at S = S and rearrange, it is straightforward to see that (S ; S ) @ (0; S ) @ (S ; S ) 0 + f (S ) = f (S ) @S 1 (0; S ) @S (0; S ) @ (S ; S ) = f (S ) 1 + 1 (0; S ) @S f (S ) @ (S ; S ) = 1 (0; S ) @S f (S ) @ (S ; S ) = 1 (0; S ) @S @V (S ; S ) = @S

which is the desired result.

Appendix B: Deterministic Bang-Bang Problems The general form for a deterministic bang-bang type problem has a reward function

f0 (S ) + f1 (S )x state dynamics

dS = [g0 (S ) + g1 (S )x]dt and control constraint

xa x xb : Suppose we use a control, not necessarily optimal, with S as a switching point, e.g., set x = xa for S < S and x = xb for S > S .21 At S = S we 21 This assumes that the state is growing when xa is used and is shrinking when xb is

used. It is a simple matter to reverse these inequalities.

CHAPTER 11.


382

choose x in such a way that dS=dt = 0. Summarizing, de ne 8 xa if S < S > < g0 (S ) if S = S ; x(S; S ) = > : g1 (S ) xb if S > S with xa < g0 (S )=g1 (S ) < xb . The value function satis es the dierential equation h i 1 V (S; S ) = f0 (S )+f1 (S )x(S; S )+ g0 (S )+g1(S )x(S; S ) VS (S; S ) ;(22) which, evaluated at S = S , yields 1 g0 (S ) V (S ; S ) = f (S ) f1 (S ) : (23) 0 g1 (S ) In spite of the discontinuity of the control at S , the value function is continuous, as is readily apparent by writing it as

V (S; S )

=

Z 1 0

e

t (f

0 (S ) + f1 (S )x(S; S

)) dt;

and noting that as S approaches S from below (above), the amount of time during which the control is set at xa (xb ) goes to 0. The continuity of V can be used to demonstrate the continuity of VS (S; S ) at S = S , and to thereby determine its value:22 f (S ) VS (S ; S ) = 1 : (24) g1 (S ) 22 To determine the limit from below, note that continuity of V implies that

1 lim [f0 (S ) + f1(S )xa + (g0 (S ) + g1 (S )xa ) VS (S; S )] 1 = f (S ) + f1 (S )xa + (g0 (S ) + g1 (S )xa ) lim VS (S ; S ) S %S 0 f (S )g0 (S ) 1 = f (S ) 1 V (S ; S ): 0 g1 (S ) Rearranging, we see this expression implies that f (S )g0 (S ) (g0 (S ) + g1(S )xa ) lim V (S; S ) = f1 (S )xa + 1 S %S g1 (S ) f 1 (S ) = (g (S ) + g1 (S )xa ) g1(S ) 0 The same exercise can be applied to solving for the limit from below. lim V (S; S ) =

S %S

S %S

CHAPTER 11.

S .


383

So far, however, we have only considered the value function for the control To choose the control optimally, we must pick S to satisfy

VS (S; S ) = 0: For S 6= S we can dierentiate (22) to see that i 1h VS (S; S ) = f1 (S )+g1 (S )VS (S; S ) xS (S; S )+g1 (S )x(S; S )VSS (S; S ):(25) However, except at S = S , xS (S; S ) and VSS (S; S ) are zero and hence we only need to set this derivative to zero at S = S . (25) is not well de ned at S = S because the derivative xS (S; S ) is unde ned at this point. Instead we use the relationship

dV (S ; S ) = VS (S ; S ) + VS (S ; S ): dS Rearranging this and using (23) and (24) we get dV (S ; S ) VS (S ; S ) dS ) g (S ) f (S ) d f ( S 0 0 g (S ) 1 f1 (S ) + = dS g1 (S ) Thus the optimal switching points are found by solving for the roots of this expression. Ludwig (1979) discusses a case in which there are multiple roots, leading to a situation in which VS may be discontinuous at a root; this root represents an unstable equilibrium at which x is unde ned. VS (S ; S ) =

1

1

Exercises 1. Optimal Sales from an Inventory What follows is an example of a problem with a continuous control that is bounded below by zero. It is not explicitly bounded above but a constraint on the state leads to an optimal control of the bang-bang form (bang-bang problems are discussed further on page 364). Because of the nature of the state transition and the state constraint, the control problem is essentially identical to an optimal stopping problem. Speci cally, a rm with an irreplaceable inventory nds it optimal to

CHAPTER 11.


384

either hold the inventory or sell it completely all at once. The sell decision is triggered by the price of the good; if the price is high enough it is optimal to sell. The main problem arises in determining the trigger price. Consider a situation in which an agent has an inventory of S0 units of a good in inventory, all of which must be sold within T periods. It costs k dollars per unit in inventory per period to store the good. In this problem there is a single control, the sales rate q , and two state variables, the price P and the inventory level S . The price is an exogenously given Ito process:

dP = (P; t)dt + (P; t)dz: The amount in storage evolves according to

dS = qdt: Furthermore the control must be nonnegative (i.e., the agent cannot purchase additional amounts)

q0 and the inventory level must be non-negative:

S 0: The problem can be written as

V (S; P; t) = max Et q(S;P;t)

Z T t

e

rt (qP

kS ) dt

subject to the above constraints.23 Bellman's equation for this problem is

V = max (qP q

kS ) + Vt + VP + 21 VP P 2

VS q;

23 In addition to the usual regularity conditions this problem requires a constraint on the expected growth rate of price (if it grows too fast it always pays to hold). Also Et [Pt+t jPt ] must be an increasing function of Pt .

CHAPTER 11.


385

subject to the non-negativity constraints on S and q . The constraint on the state variable S is problematic; it clearly constrains the control but it does so in following way:

q 0 if S = 0: Given that q 0, this implies that

q = 0 if S = 0: To impose this constraint we introduce a multiplier, that exhibits complementary slackness with S . The optimality conditions can then be written

P

VS

0; q 0; C.S. S 0; 0; C.S:

There are three possible solutions to this problem:

P < VS P = VS P > VS

) q=0 ) indierence ) q = 1 if S > 0

Thus the optimal control is to either sell all of the inventory (if P > VS ) or to sell nothing (if P < VS ). Only when P = VS is the agent indierent between holding and selling (in which case it is harmless to assume that the inventory would be sold). There is, therefore, a region in which it is optimal to hold onto inventory bounded by a curve, P (t), at which it is optimal to sell the whole inventory. Within the holding region the value function satis es the PDE

rV = k + Vt + (P; t)VP + 21 2 (P; t)VP P : Furthermore, it is easy to see that the value function is proportional to the level of inventory; hence it is harmless to normalize by setting

CHAPTER 11.


386

S0 = 1. The value function is then a function of the price level and time alone. At the boundary the value matching and smooth-pasting conditions hold. Consider that, when the inventory is sold it worth P per unit. Hence the value-matching and smooth-pasting conditions are V (P (t); t) = P (t) and

VP (P (t); t) = 1: If the inventory must be sold on or before a xed date, T , an additional terminal boundary condition requires that

V (P; T ) = P: There may, in addition, be a lower boundary constraint, for example at P = 0. If zero is an absorbing barrier for P then V (0) = 0. 2. Show that a utility function of the form U (C ) = (C 1 1)=(1

) implies an optimal consumption rule of the form C (W ) = aW . Determine the constant a and, in the process, determine the value function and the optimal investment rule (W ). 3. Suppose that there are only two assets available to investors, which are governed by

dR = rRdt and

dS = Sdt + Sdz; i.e., R is a risk-free and S a risky asset. The controls for the investors problem are C , the consumption rate, and , the fraction of wealth held in the risky asset. Write the Bellman's Equation associated with this problem and derive expressions for the optimal controls.

CHAPTER 11.


387

4. Expand the analysis of the resetting problem that begins on page 379 to include the case in which a ow of payments f (S ) is received. In particular, show that value matching holds for any choice of the resetting state and that smooth pasting holds for the optimal choice. 5. In the general optimal stopping problem the value function can be written as

V (S ; S ) = E

"Z 0

(S;S )

#

e t f (St )dt + E e

(S;S ) R(S );

where (S; S ) is the rst time the state equals S given that it equals S at time 0. Show that value matching holds for arbitrary S and the smooth pasting holds for the optimal S . 6. Verify that the optimal harvest stand size in the timber management problem given on page 350 is correct. 7. Consider the manager of a cash account subject to random deposits and withdrawals. In the absense of active management the account is described by absolute Brownian motion

dS = dt + dz: The manager must maintain a positive cash balance. When the account hits 0, the manager must draw funds from an interest bearing account. To increase the cash account by z units, the manager bears a cost of f + cz , i.e., there are both xed and proportional variable costs of control. Similarly, the manager can place funds in the interest bearing account by withdrawing an amount z from the cash account, incurring costs of F + Cz . Suppose the manager uses a discount rate of and the interest bearing account generates interest at rate r. It is clear that the manager will want to adjust the account only at discrete times so as to minimize the adjustment costs. A control policy can therefore be described as a choice of three cash levels, S1 S2 S3 , where S1 is the amount of the addition to the fund when it hits 0, S3 is the trigger level for withdrawing funds (adding them to the interest bearing account) and

CHAPTER 11.


388

S2 is the target level (i.e., S3 S2 units are withdrawn when the fund hits S3 ). The value function associated with this problem solves the Bellman equation24 V (S ) = V 0 (S ) + 21 2 V 00 (S ); for S 2 [0; S3 ] with the side conditions that

V (0) = V (S1 ) f

(r= + c)S1

and

V (S3 ) = V (S2 ) F + (r= C )(S3

S2 ):

Furthermore, an optimal policy satis es

V 0 (S1 ) = (r= + c) and

V 0 (S3 ) = V 0 (S2 ) = (r= C ): The Bellman equation can be solved explicitly:

V (S ) = A exp(S ) + B exp( S ); where and are choosen to solve the dierential equation and A and B are chosen to satisfy the side conditions.

24 Although it is not necessary to solve the problem, it is useful to understand why

these conditions are appropriate. The value function here is interpreted as the present value of the current cash position, which does not depend on how much money is in the interest bearing account at the present moment. Cash pays no current ows and hence the Bellman equation is homogeneous (no reward term). The cost of withdrawing funds from the interest bearing account equals the control cost plus the opportunity cost of the lost interest, which is equal to r= times the amount withdrawn. The cost of adding funds to the interest bearing account equals the control cost less the present value of the interest earned on the funds put into the account (r= times the amount of these funds).

CHAPTER 11.


389

Write a MATLAB procedure that accepts the parameters , , , r, f , F , c, and C and returns the parameters A, B , , , S1 , S2 , and S3 . Also determine how the program needs to be modi ed if the proportional costs (c and C ) are zero. Check the answers you obtain using the following parameter values: = 0, = 0:5, = 0:4, r = 0:5, f = 1, F = 0:5, c = 0:1, and C = 0:1. You should obtain the result that S1 = 0:7408, S2 = 0:8442, and S3 = 2:2216. 8. Consider an extension to the renewable resource problem discussed on page 367. Suppose that the harvest rate is still constrained to lie on [0; C ] but that it cannot be adjusted instantaneously. Instead assume that the rate of adjustment in the harvest rate, x, must lie on [a; b], with a < 0 < b, with the proviso that x 0 is h = 0 and x 0 is h = C. This problem can be addressed by de ning h to be a second state variable with a deterministic state transition equation:

dh = xdt: The optimal control for this problem is de ned by two regions, one in which x = a and one in which x = b. The boundary between these regions is a curve in the space [0; 1) [0; C ]. Write the PDEs that must be satis ed by the value functions in each region and the value-matching and smooth pasting conditions that must hold at the boundaries. 9. Consider the optimal management of a renewable resource. Suppose that the stock of the resource evolves according to

dS = (m S )Sdt + Sdz: The (inverse) demand for the resource is given by

p = D 1 (q ) = a bq: and the cost of harvesting the resource is

c q2 : 2S

CHAPTER 11.


390

Assume the appropriate discount rate is and that the social preference function is consumer surplus less harvesting cost. (a) De ne the social planner's reward function. (b) Write the Bellman's equation for this problem. (c) Solve the rst order conditions and substitute out the optimal control from Bellman's equation to arrive at a concentrated Bellman's equation. (d) Discuss a computational strategy to solve for the optimal value function (implement it for extra credit).

Chapter 12 Continuous Time Dynamic Models: Methods In the previous two chapters we saw how continuous time economic models, whether deterministic or stochastic, result in either ordinary or partial dierential equations that must be evaluated subject to some boundary conditions. Ordinary dierential equations (ODEs) arise in in nite horizon single state models or in deterministic problems solved in terms of time paths. Partial dierential equations (PDEs) arise in models with multiple state variables or in nite horizon control problems. From a numerical point of view the distinction between ODEs and PDEs is less important than the distinction between problems which can be solved in a recursive or evolutionary fashion or those that require the entire solution be computed simultaneously because the solution at one point (in time and/or space) depends on the solution everywhere else. This is the distinction between initial value problems (IVPs) and boundary value problems (BVPs) that we discussed in Chapter ??. With an IVP, the solution is known at some point or points and the solution near these points can then be (approximately) determined. This, in turn, allows the solution at still other point to be approximated and so forth. When it is possible, it is usually faster to use recursive solution techniques, which include Euler and Runge-Kutta methods for ordinary dierential equations and recursive nite dierence methods or the method of lines for partial dierential equations. We begin this chapter with a discussion of various approaches to solving PDEs. First we discuss nite dierence approaches, which are very widely 391

CHAPTER 12.

CONTINUOUS TIME MODELS: METHODS

392

used and easy to program. We then discuss the method of lines, which uses nite dierences for the state variables and expresses the PDE as a system of ordinary dierential equations in time. The method of lines can be extended easily using function approximation methods to represent the solution as a function of the states and an ODE in time. Finally, we discuss using collocation as a general scheme to solve PDEs. The various methods are then applied to the solution of stochastic control problems, including problems involving free boundaries. There are a number of methods for solving PDEs and stochastic control problems that we do not discuss here. These include binary and trinomial tree methods and simulation methods for solving PDEs and discretizing the state and action space in control problems and solving the related discrete problem. Our main rationale for our choices of what to include is that the methods discussed build on general methods developed in previous chapters. Much of what is discussed here should look and feel familiar to readers that have persevered up to this point. We do, however, include some references to other approaches in the bibliographical notes at the end of the chapter.

12.1 Partial Dierential Equations In the previous two chapters we discussed a number of examples of partial dierential equations, like the Black-Scholes option pricing formula, for which there are relatively simple solutions. For most interesting problems, we are not so lucky, however. There are several numerical approaches that can be used, including Monte Carlo simulation, binomial trees, nite dierence methods and weighted residual methods. The only dierence between an ordinary dierential equation (ODE) and a partial dierential equation (PDE) is that the solution to the former is a function of a single variable, whereas the solution to the latter is a function of multiple variables. From a computational point of view, this generally means that PDEs are harder to solve than ODEs and, in fact, if there are too many arguments, it may be nearly impossible to solve without getting very clever. As with ODEs, the distinction between initial and boundary value problems is relevant for PDEs. For example, a function, V (S; t), that solves a PDE of the form Vt = V + 21 2 VSS ;

CHAPTER 12.

393


with V (S; 0) a known function of S , can be propagated with respect to t, starting at t = 0. This approach cannot be used to approximate a function V (S; R) satisfying

V = f (S; R) + 21 [SS VSS + 2SR VSR + RR VRR ] subject to boundary conditions at S = aS , S = bS , R = aR and R = bR . Instead, this type of problem must be solved simultaneously for all relevant values of (S; R).

12.1.1 Finite Dierence Methods for PDEs For evolutionary PDEs, the most common approach uses nite dierence methods, which are relatively easy to understand and implement from scratch, at least for low dimensional problems. Furthermore, they can have good numerical properties, especially if you are not looking for a high degree of accuracy. Essentially nite dierence methods amount to replacing terms involving derivatives with dierence approximations to those derivatives. Evolutionary PDEs (often called parabolic PDEs) are characterized by having no second derivatives for one of the variables entering the PDE. The parabolic case is particularly important in economic applications, where the form

(S )V (S; t) = f (S ) + Vt (S; t) + (S )VS (S; t) + 21 2 (S )VSS (S; t) is often encountered. We denote the rst order variable as t to suggest time because in many applications it will have that interpretation; the essential features, however, is that it is rst order in one variable and the function is known at some value of that variable. In simple applications, the PDE is de ned on [a; b] [0; T ]. Boundary conditions are speci ed at S = a, S = b and either at t = 0 or t = T . An initial value problem with the boundary condition speci ed at t = 0 can be propagated forward in the t direction. A terminal value problem with the boundary condition speci ed at t = T can be propagated backwards in time from T , or by rede ning the problem in terms of = T t and propagating forward in . The basic approach involves rst rewriting the PDE in terms of Vt

Vt (S; t) = (S )V (S; t) (S )VS (S; t)

1 2 2 (S )VSS (S; t)

f (S )

(1)

CHAPTER 12.


394

Given an initial condition V (S; 0) = B (S ) we compute VS (S; 0) = B 0 (S ) and VSS (S; 0) = B "(S ) and thereby compute Vt (S; 0). This allows us to take a step of size t in the t direction and compute an approximation to V (S; t). The algorithm then proceeds recursively. With a terminal condition we set V (S; T ) = B (S ) and use a time step of t; otherwise the problem is the same. In nite dierence implementations of this idea, a 2-dimensional grid is de ned in the S and t directions and all derivatives terms are replaced by nite dierence approximations (see Chapter ??). Using centered nite differences for the rst and second derivatives for O(h2) accuracy, (1) becomes:

V (Si+1 ; t) V (Si 1 ; t) Vt (Si ; t) = (Si )V (Si ; t) (Si) 2h V ( S ; t ) 2 V ( S ; t ) + V (Si 1 ; t) i +1 i 1 2 (S ) i 2 2 h 2 f (Si ) + O(h ) (Si) 2 (Si ) = V (Si 1 ; t) 2h 2h2 2 (Si ) + (Si ) + 2 V (Si ; t) h (Si ) 2 (Si ) V (Si+1 ; t) + 2h 2h2 f (Si ) + O(h2 ): Notice that the LHS of this expression depends on the value of the function at only three points, Si 1 , Si and Si+1 . We now must discretize the PDE in the t direction and it is here where we face several choices. The two obvious ones are to use either a forward dierence 1 Vt (Si ; t) = V (Si ; t + t) V (Si ; t) t or a backward dierence 1 Vt (Si ; t) = V (Si ; t) V (Si ; t t) t Although it may initially appear that either will work, it turns out that the choice makes an enormous dierence.

CHAPTER 12.


395

If we replace Vt (Si ; t) in (2) with the forward dierence operator and rearrange we get the system V (Si ; t + t) = afi V (Si 1 ; t) + bfi V (Si ; t) + cfi V (Si+1 ; t) f (Si )t + O(h2 ) + O(t); where t t 2 (Si ) 2 afi = (Si ) 2h 2h t bfi = 1 + (Si )t + 2 (Si ) 2 h t t cfi = (Si ) 2 (Si ) 2 2h 2h On the other hand, if we use the backward dierence operator we get V (Si ; t t) = abi V (Si 1 ; t) + bbi V (Si ; t) + cbi V (Si+1 ; t) + f (Si )t + O(h2 ) + O(t); where t t abi = (Si ) + 2 (Si ) 2 = afi 2h 2h t bbi = 1 (Si )t 2 (Si ) 2 = 2 bfi h t t cbi = (Si ) + 2 (Si ) 2 = cfi 2h 2h To implement the method we de ne a grid of m values of Si (S1 = a; S2 = a + h; : : : ; Sm = b) and n values of tj (t1 = 0; t2 = t; : : : ; tn = T ). For problems in which S is unbounded we must chose a and b such that the probability of attaining these values is very small. The grid can be represented as an m n matrix with rows representing values of the state and columns representing points in time. Letting Vij = V (Si ; tj ) and given values of V1j and Vmj , the forward dierence system can be written in matrix form as the m 2 equation system 2 3 2 3 1 0 0 0 V1j V1j +1 6 af2 bf2 cf2 7 6 7 f2 t 6 7 6 7 f f f 6 0 a b 7 6 7 f c 3 t 3 3 3 6 7 6 7 Vj +1 = 6 Vj 6 7 7 : : : : : : : : : : : : 6 7 6 7 f f f 4 4 fm 1 t 5 am 1 bm 1 cm 1 5 Vmj Vmj +1 0 0 0 1

CHAPTER 12.


whereas the backward dierence system can be written 2 6 6 6 6 6 6 4

1 0 0 ab2 bb2 cb2 0 ab3 bb3 cb3 ::: ::: 0

0

3

7 7 7 7 Vj +1 7 ::: 7 abm 1 bbm 1 cbm 1 5

0

0

1

2

6 6 6 = Vj + 6 6 6 4

396

V1j +1 V1j f2 t f3 t ::: fm 1 t Vmj +1 Vmj

3 7 7 7 7: 7 7 5

The rst of these de nes an explicit system of dierence equations in Vj +1 in terms of Vj . The second formulation de nes Vj +1 implicitly; obtaining an explicit solution requires solving the system of linear equations. Notice that the endpoints S1 and Sm are dealt with in a special way. The nite dierence method requires that the solution be known at S = a and S = b. We have written the rst and last rows of the two linear systems to re ect that these values are known. Although it may appear that the explicit approach is better, appearances can be deceiving. Although the explicit approach does not require a linear solve at each time step, it can be unstable when the time steps are not small enough. The instability arises because approximation errors are magni ed as they are propagated through time, thus producing a useless result. Implicit methods are stable, regardless of the size of the time step, because the approximation errors are damped as they are propagated. Hence larger time steps can be used, resulting in greater computational eÆciency. The trade-o is further tipped towards implicit methods by the fact that the linear system is sparse (indeed it is tridiagonal) and so special methods can be used to perform the linear solve. In practice, a hybrid approach is often used. The Crank-Nicholson approach evaluates V , VP , and VP P as a weighted average of the nite dierence approximations at time j and time j + 1. Often the weights used are 21 and 1. 2

Unconditionally Stable Explicit Finite Dierence Methods The main disadvantage in using implicit nite dierence methods is that the matrix inversion is relatively slow, even though the inverted matrix is tridiagonal. A stable explicit method could use considerably more evaluation points for the same computational time and thereby, hopefully, increase the accuracy of the approximation. There are a number of alternative explicit

CHAPTER 12.


397

methods that are unconditionally stable; we describe one here called the hopscotch method. Hopscotch Method: The hopscotch method alternately uses the explicit and implicit formulae for adjacent points. The idea is illustrated below: i 6 5 4 3 2 1 0

B I X I X I B 0

B X I X I X B 1

B I X I X I B 2

B B B B B B B 3 j

B: boundary values X: explicitly determined values I: implicitly determined values In the gure there are 5 interior space points and 3 interior time points. Starting at time 2, one rst calculates all of the points for which (i + j ) is even using the explicit scheme. Then one can calculate the points for which (i + j ) is odd using an implicit scheme that involves the previous computed spatial points on either side. This is continued for each successive time point. Thus for (i + j ) even, the value of V [i; j ] is computed using V [i 1; j 1], V [i; j 1] and V [i + 1; j 1]. For (i + j ) odd, on the other hand, V [i; j ] is computed using V[i,j-1], V[i-1,j] and V [i + 1; j ], both of the latter having already been computed explicitly.

CHAPTER 12.


398

Code Box 12.4: Hopscotch Method

Example: Financial Options

Finite dierence methods are now routinely used to value nancial options. Given a risk neutral process describing the value of an underlying asset:

dS = (r

Æ )Sdt + (S )dz;

an option value can be computed using the dierential equation

rV = Vt + (r

Æ )SVS + 12 2 (S )VSS :

For European call options (no early exercise) with strike price k, the terminal (time T ) boundary condition is

V (S; T ) = max(0; S

k);

the lower boundary condition is V (0; t) = 0 and the upper boundary condition is

V (S ) = exp( Æ (T

t))S

exp( r(T

t))k:

For put options the boundary conditions are

V (S; T ) = max(0; k

S );

V (0; t) = exp( r(T

t))k

and lim S ! 1 V (S; t) = 0. Code Box 5 displays code that sets up the problem and calls the Hopscotch method to produce a solution for the case in which (S ) = S (geometric Brownian motion).

Forward Dierence V Vi 1;j 1 2 Vi+1;j 2Vij + Vi 1;j Vi;j +1 Vij + ij i+1;j + 2 ij = ij Vij t 2h 2h2 Backward Dierence Vi;j +1 Vij V 2Vij +1 + Vi 1;j +1 V V + ij i+1;j +1 i 1;j +1 + 21 ij2 i+1;j +1 = ij Vij +1 t 2h 2h2 Note: if , and are time dependent then de ne ij = (xi ; tj + 12 t) and de ne ij and ij analogously.

Vt + (x; t)VS + 12 2 (S; t)VSS = (S; t)V

Table 12.1: Finite Dierence Approximations for Linear Parabolic PDEs

CHAPTER 12. CONTINUOUS TIME MODELS: METHODS

399

CHAPTER 12.


400

Code Box 12.5: Computing Premia for Financial Options

12.1.2 Method of Lines for PDEs Equation (2), which discretized a PDE in the state variable, de nes an ODE in time. The method of lines recognizes this and uses any dierential equation algorithm, such as the Runge-Kutta method, to solve the ODE. Suppose instead of using nite dierence approximations to the derivatives, we approximate the solution to the PDE as a weighted sum of a suitably chosen set of basis functions with time varying coeÆcients:

V (S; t) = (S )c(t): The PDE can then be written as

(S )c0(t) = (S )(S ) (S )0 (S )

1 2 (S )00 (S ) c(t) + f (S ): 2

If we select a set of n nodes, Si , and de ne the n n basis matrix , this has the form

c0 (t) = Bc(t) + f; where

B=

1 (S )(S )0 (S )

1 2 (S )00 (S ) 2

and

f = 1 f (S ): We thus have an ODE in the coeÆcients of the approximating function. Furthermore, for linear PDEs the associated ODE is linear and hence can be solved analytically in terms of the eigenvalues and eigenvectors of B . [INCOMPLETE]

CHAPTER 12.


401

12.1.3 Collocation Approaches to Solving PDEs The extended method of lines selects a set of basis functions and seeks to nd an approximation at each point in time by solving a system of dierential equations that the coeÆcients of the approximating function should satisfy. An alternative is de ne a set of basis functions for all of the variables and to determine the coeÆcients of an approximating function that satis es the PDE and boundary conditions at a selected set of points. Not surprisingly, we will discuss the use of polynomial approximations and polynomial spline approximations. For linear PDEs, we will see that this approach leads to a relationship in the form

Bc = f; where B is a basis matrix, c a vector of coeÆcients and f a vector. The coeÆcients are thus determined by solving a system of linear equations. For non-linear equations we have a more general

f (c; S; B ) = 0; where B here represents a set of basis matrices corresponding to the relevant partial derivatives. Although nite dierence methods are not associated with a speci c set of basis functions, the nite dierence operators can be viewed as de ning \basis" matrices for the function and its derivatives. Thus, nite dierence methods also lead to a relationship of the form Bc = f for linear problems and f (c; S; B ) for non-linear problems. [INCOMPLETE]

12.1.4 Variable Transformations It is often useful in numerical (and analytical) analysis to transform problem variables. For example, suppose that a dierential equation is de ned over the domain S 2 [0; 1). One way to handle this is to truncate the domain at some large value of S . An alternative is to transform the domain to a bounded range, e.g., to [0; 1]. Transformations will also prove useful in handling free (moving) boundary problems, which can be transformed to a domain with a constant boundary. Transformations of PDEs can be a bit tricky so it is worth spending a little time discussing them in general and to provide some simple transformations

CHAPTER 12.


402

that are useful for numerical work. The transform changes the PDE to be solved; in particular, the derivative with respect to S must be rewritten in terms of z . In general, we have dz VS = Vz dS and

dz d2 z VSS = Vz 2 + Vzz dS dS

2

:

A number of useful transformations for working with the various boundary conditions encountered in economics are summarized in Table 12.2. The rst of these, z = S=(c + S ), transforms the domain from [0; 1) to [0; 1]. The parameter c is a scaling factor; values of S below c will map into the [0; 21 ] interval and values of S above c will map into the [ 21 ; 1] interval. For this transform we have c VS = Vz (c + S )2 and

c2 2c Vzz : 4 (c + S ) (c + S )3 The transform is useful with problems in which the solution is known to be bounded as S ! 1, such as the value of a put option. Using the transformation results in greater accuracy and eliminates the need to de ne the upper price level.1 The second transformation in Table 12.2 is useful with problems in which there is a non-zero lower boundary, possibly a free boundary, and a domain that is unbounded above (e.g., for American put options). The third transformation is useful in cases in which one or both of the boundaries are free boundaries. The nal transformation listed is often used in models involving geometric Brownian motion (dx = xdt + xdz ). Although it makes a VSS = Vz

1 It can also work for valuing call option but both the call value and, possibly, the variance function (2 ) can become unbounded as z ! 1. This can be addressed by de ning the grid of z values to range from z0 = 0 to zn+1 = 1 z , where z = n=(n + 1)2 . This seems to work well in practice, although extremely deep-in-the money call option may not be accurately valued (the practical importance of such inaccuracies is small).

CHAPTER 12.


403

problem that is bounded below into a doubly unbounded problem, it can be quite useful in cases in which the behavior at S = 0 is hard to capture. To illustrate how these transformations are used consider the problem of numerically solving 1 2 S 2 V 00 (S ) + SV 0 (S ) 2

rV (S ) = 0

on [a; 1), with boundary conditions

V (a) = ga lim V (S ) = gb :

S !1

Using the second transformation on Table 12.2, z = (S becomes 1 2 (1 2

a)=S , the problem

z )2 v 00 (z ) + ( 2 )(1 z )v 0 (z ) rv (z ) = 0

on [0; 1], with boundary conditions

v (0) = ga v (1) = gb : To apply these notions to free boundary problems, consider a problem with a single state variable, de ned on [S ; 1), with two side conditions holding at S , say V (S ) = v0 and V 0 (S ) = v1 . Consider the transformation from the interval [S ; 1) to [1; 1) can be accomplished by using the transform

z = S=S : With this transformation we can solve for V conditional on S and S conditional on V . In a two dimensional problem the free boundary is generally expressed as a functional relationship between the two states, say as

S1 = g (S2 ); with the dierential equation de ned on [g (S2); 1) [a; b]. We can de ne a new state variable to replace the rst state

z = S1 =g (S2)

CHAPTER 12.


404

which will imply the following transformations

V (S1 ; S2 ) = v (z; S2 ) V1 (S1 ; S2 ) = v1 (z; S2 )=g (S2) V11 (S1 ; S2 ) = v11 (z; S2 )=g (S2)2 V2 (S1 ; S2 ) = v2 (z; S2 ) v1 S1 g 0(S2 )=g (S2)2 : We use this strategy below to solve stochastic control problems.

12.2 Solving Stochastic Control Problems In the previous chapter we saw that for problems of the form

V (S ) = max x(S )

Z 1 t

e

r f (S; x)d

s.t. dS = (S; x)dt + (S )dz;

Bellman's equation takes the form

rV (S ) = max f (S; x) + (S; x)V 0 (S ) + 12 2 (S )V 00 (S ); x(S )

subject to boundary conditions at S = a and S = b. Suppose we approximate the function value function using V (S ) (S )c. For a given policy function x(S ), the collocation equations are

r(S ) (s; x(S ))0 (S )

1 2 (S; x(S ))00 (s) c 2

= f (S; x(S )):

Any relevant boundary conditions can be appended to the matrix in [ ]. For example, the value function may be known at the boundaries, in which case we have (a)c = ga and (b)c = gb : The boundary conditions often are linear in the value function and its derivatives and hence are linear in the approximation coeÆcients. Given the linearity of the Bellman equation, the collocation equation is therefore linear in c and hence is easily solved. An iterative procedure analogous to policy function iteration uses the following steps:

[a; b] ! [0; 1]

[a; 1] ! [1; 1]

z = Sb aa S = a + (b a)z z = Sa S = az

1; 1)

[a; 1) ! [0; 1]

z = SS a S = 1az

[0; 1) ! (

(1 z )2 0 a f (z )

[0; 1) ! [0; 1]

z = c+SS S = 1czz

z = ln(S ) S = ez

(1 z )2 0 c f (z )

Domain/Codomain

Transform (1 z )3 c2

(1

F 00 (S ) S 2 F 00 (S )

zf 0 (z )

f 0 (z ) a

+ z f 0 (z )

f 0 (z ) b a

z )f 0 (z )

e z f 0 (z ) f 0 (z )

a b a

(1

e

f 00 (z ) a2 z 2 f 00 (z )

+ z f 00 (z )

f 00 (z ) f 0 (z ) f 00 (z ) f 0 (z ) 2z

a b a

f 00 (z ) (b a)2 2

z )f 00 (z ) 2f 0 (z ) (1 z ) (1 z )f 00 (z ) 2f 0 (z )

(1 z )3 (1 2 a

z )f 00 (z ) 2f 0 (z ) z (1 z )f 0 (z ) z 2 (1 z ) (1 z )f 00 (z ) 2f 0 (z )

F 0 (S ) SF 0(S )

Table 12.2: Variable Transformations For First and Second Order Dierential Equations CHAPTER 12. CONTINUOUS TIME MODELS: METHODS

405

CHAPTER 12.


406

A) guess an initial V (S ) and nd the corresponding approximation coeÆcient vector c. B) for each of the collocation nodes, Si , determine the optimal value of x given the current value of c C) solve the collocation equation for a new coeÆcient vector D) check for convergence; return to (2) if not converged An alternative when one can solve explicitly for the optimal control (in terms of the value function) is to substitute the control out of the Bellman Equation. This results in (generally) a nonlinear dierential equation in S , which can be solved directly using collocation. If the dierential equation is nonlinear, however, the collocation equations are also nonlinear and hence must be solved using a root nding algorithm. As we discussed above, it may be useful to employ a change-of-variables to make the problem more tractable or more amenable to numerical approximation. For example, the value function may get very steep at S = 0, making it diÆcult to approximate using polynomial or spline bases. The optimal renewable resource harvesting example (Section ??) displays this behavior, with the limits of V (0) = VSS (0) = 1 and VS (0) = 1. In such a case it is useful to employ the log transform (see Ludwig, 1979). The value function in the transformed variable is therefore approximately linear as the transformed state variable approaches 1 ; this is easily imposed on the transformed problem by forcing the second derivative to 0 at the lower boundary of the approximation. It is also reasonable to assume that the value becomes approximately linear as the state goes to 1; in fact VS should go to zero (the value of an additional unit of the resource is zero when the resource is in nitely abundant) implying that the value function approaches a constant. This can be insured by imposing that the transformed problem has a zero rst or second derivative at the upper limit of the approximation. In practice, imposing the zero on the second derivative is preferred, as it allows for more exibility in t (this point is illustrated below).

Example: Harvesting a Renewable Resource [INCOMPLETE]

CHAPTER 12.


407

12.2.1 Free Boundary Problems Many of the problems discussed in the previous chapter involved free boundaries which represent endogenously determined state values at which some action is taken. For example, consider a second order linear dierential equation with the general form

(S )V (S ) = f (S ) + (S )V 0 (S ) + 12 2 (S )VSS ;

(2)

where this equation holds on some interval [a; b]. The usual boundary value problem takes both a and b as known and requires boundary conditions such as V (a) = ga and V (b) = gb to be met, where ga and gb are known values. Numerically, one can approximate the solution using a function parameterized by an n-vector c: V (S ) (S )c. c is chosen so that (S )c satis es (2) at n 2 points and satis es the boundary conditions. This yields n equations in the n unknown parameters. In the free boundary problem one or both of the boundary locations a and b are unknown and must be determined by satisfying some additional conditions. Suppose, for example that the upper boundary, b, is unknown but V 0 (b) = hb , where hb is a known constant. Thus there are three boundary conditions and one additional parameter, b, implying that one must solve n+1 equation in n + 1 unknowns. If both boundaries are free, with V 0 (a) = ha , the problem becomes one with n + 2 equations and n + 2 parameters. The interval on which the approximating function is to be de ned, however, is unknown. Fortunately, this problem is easily addressed using a change in variable. Consider rst the case in which b is unknown and, for simplicity, a = 0. De ne

z = S=b; so the dierential equation is de ned on z 2 [0; 1]. De ne the function v (z ) such that

v (z ) = V (S ); using the chain rule it can be seen that

v 0 (z ) = V 0 (S )b and

v 00 (z ) = V 00 (S )b2 :

CHAPTER 12.


408

Inserting these de nitions into (2) demonstrates that the original problem is equivalent to

(bz )v (z ) = f (bz ) + for z 2 [0; 1], with

(bz ) 0 2 (bz ) 00 v (z ) + v (z ); b 2b

(3)

v (0) = ga ; v (1) = gb ; and

v 0 (1) = hb b:

Example: Timber Harvesting [INCOMPLETE]

Optimal Stopping Simple optimal stopping have the same form as the timber harvesting problem in that a PDE is solved over some unknown interval, where the endpoints of the interval must be determined along with the value function. We rst illustrate this with a simple investment example. More diÆcult stopping problems arise when the boundary changes over time, as is the case when solving American option pricing problems; American options can be exercised early and the free boundary represents the price, time values at which it is optimal to so. We illustrate one method for solving such problems; another method for solving such problem will be discussed in the context of stochastic bang-bang problems in the next section.

Example: Investment Under Uncertainty

Consider a simple irreversible investment problem in which an investment of I will generate a return stream with present value of S , where S is described by the Ito process

dS = (m S )Sdt + Sdz:

CHAPTER 12.


409

This process can be shown to have a mean reverting rate of return, with long-run mean m (see Section ??). When the investment is made it has value S I . Prior to making the investment, however, the value of the right to make such an investment is V (S ), which is the solution to the following dierential equation 1 2 2 00 2 S V (S ) + (m

S )SV 0 (S ) rV (S ) = 0;

where r is the risk-free interest rate. The lower boundary, S = 0, is associated with an investment value of 0, because once the process S goes to 0, it stays equal to 0 forever; hence V (0) = 0. The upper boundary is de ned as the value, S , at which investment actually occurs. At this value two conditions must be met. The value matching condition states that at S the value of investing and not investing are equal: V (S ) = S I . The smooth-pasting optimality condition requires that V 0 (S ) = 1. Applying the change of variables (z = S=S ) yields the equivalent problem 1 2 2 00 2 z v (z ) + (m

zS )zv 0 (z ) rv (z ) = 0;

(4)

on the interval [0; 1], with v (0) = 0, v (1) = S I , and v 0 (1) = S . To solve the problem we approximate the function v (z ) using

v (z; c) =

n X j =1

j (z )cj ;

where the j (z ) are convenient basis functions. Chebyshev polynomials are a natural choice for this problem because v (z ) should be relatively smooth. The parameter vector c and the optimal investment trigger S are selected to satisfy (4) at n 2 appropriately chosen nodes on the interior of [0; 1] (e.g., the roots of the order n 2 Chebyshev polynomial) and to satisfy the three boundary conditions. To make this a bit more explicit, given a guess of S , de ne the n 2 n matrix B

Bij = 21 2 zi2 00j (zi ) + (m zi S )zi 0j (zi ) rj (zi ) for i = 1; : : : ; n 2. Then concatenate the basis functions for the boundary conditions to the bottom of this matrix: Bn 1;j = j (0) and Bn;j = j (1).

CHAPTER 12.


410

This produces an n n matrix. The coeÆcients, conditional on the guess of S , are given by

c(S ) = B

1

0n 1 S I :

Given c we can de ne a residual function in one dimension to solve for S using the smooth-parting condition:

r (S ) = S

0 (1)c(S ):

This approach works well in some cases but this example has one additional problem that must be addressed. It will be observed that, for some parameter values, the approximate solution obtained becomes unstable, exhibiting wide oscillations at low values of z . The solution value for S , however, remains reasonable. The problem, therefore, seems due to the approximation having trouble satisfying the lower boundary. It can be shown that, for some parameter values, the derivative of v becomes unbounded as S approaches 0: lim V 0 (S ) = 1:

S &0

This type of behavior cannot be well approximated by polynomials, the derivatives of which (at every order) are bounded on a bounded domain. Fortunately this problem can be easily addressed by simply eliminating the lower boundary constraint and evaluating (4) at n 1 rather than n 2 nodes. This causes some error at very small values of z (or S ) but does not cause signi cant problems at higher values of z . The economic context of the problem places far more importance on the values of z near 1, which de nes the location of S and hence determines the optimal investment rule. This particular problem has a partially known solution. It can be shown that the solution can be written as

V (S ) = AS H (S ; ; ); where H (x; ; ) is the con uent hypergeometric function de ned by the series expansion

H (x; ; ) =

1 X i=0

( + i) ()xi : ( ) ( + i)i!

CHAPTER 12.

and

1 = 2


m + 2 r

=1+2

1 2

r

m 2 2r + 2 2 m 2 2r + 2 2

411

1 2

2 : 2 Thus, the problem can be seen to arise when < 1, which causes the term in the derivative involving S 1 to become unbounded as S ! 0. The solution is only partially known because the constants A and S must be determined numerically using the free boundary conditions:2 AS H (S ; ; ) (S I ) = 0 and A S 1 H (S ; ; ) + AS H 0(S ; ; ) 1 = 0: Eliminating A yields the relationship H (S ; + 1; + 1) S (S I ) 1 + S = 0; H (S ; ; ) a simple root nding problem in a single variable, which can be solved using the methods of chapter ??. MATLAB code solving the problem in both ways is shown below. This code produces Figure ??. The dashed line is the solution obtained using the hypergeometric function approach. The dashed line solves the problem with no lower end point condition imposed and the dotted line imposes the lower end point condition. The gure illustrates the diÆculties in tting the value function at the lower end but also illustrates that the computation of the location of the free boundary is not very sensitive to these problems.

=

Example: Pricing American Options [INCOMPLETE]

2 Notice from the series expansion that the derivative of H is given by 1 ( + i + 1) ()xi X 0 = H (x; + 1; + 1): H (x; ; ) = i=0 ( ) ( + i + 1)i!

CHAPTER 12.


412

Stochastic Bang-Bang Problems Problems with binary states that can be exited and reentered, as is the case with stochastic bang-bang problems, can lead to new challenges. These challenges arise because, in eect, two value functions, one for each of the binary states, must be simultaneously approximated. Furthermore, regions of the state space over which these value functions apply must be determined. Recall that the general framework giving rise to stochastic bang-bang problems occurs when the reward function is of the form

f (S; x) = f0 (S ) + f1 (S )x; the state variable is governed by

dS = [g0 (S ) + g1 (S )x]dt + (S )dz and the control is bounded:

xl x xu : Consider the discounted in nite time horizon problem

V (S ) = max E x

Z 1 t

e

t f (S; x)dt :

The optimal control is to set x = xl whenever f1 + g1 VS < 0 and to set x = xu whenever f1 + g1 VS > 0. Denoting these regions Sl and Su , the value function must satisfy

V

(g0 + g1 xl ) VS

1 2V SS 2

(f0 + f1 xl ) = 0 on Sl

V

(g0 + g1 xu ) VS

1 2V SS 2

(f0 + f1 xu ) = 0 on Su

and value-matching and smooth pasting at points were f1 = g1 VS (plus any additional boundary conditions at S = a and S = b). For concreteness suppose that there is a single point S such that f1 (S ) = g1 (S )VS (S ) and that Sl consists of points less than S and Su of points greater than S (generally the context of the problem will suÆce to determine the general nature of these sets). The numerical problem is to nd this S and the value function V (S ). The following strategy can be used. First, notice that the Bellman equation is linear given S and assume that the boundary conditions are also linear in V . Suppose we approximate two functions, one

CHAPTER 12.


413

on Sl , the other on Su that approximately satisfy the Bellman equations and the boundary conditions and also that, for any guess of S , satisfy value matching and smooth pasting at this guess. Let the approximations be de ned by (S )ci, for i = l; u and de ne the function B (S ) as

B (S ) = (S ) [g0 (S ) + g1 (S )xi ] 0 (S )

1 2 (S )00 (S ) 2

The ci can be determined by making

B (S )ci

[f0 (S ) + f1 (S )xi ] = 0

at a selected set of collocation nodes, together with the boundary conditions and (S )cl (S )cu = 0 (value matching) 0 (S )cl 0 (S )cu = 0 (smooth pasting). Determining the ci for some guess of S , therefore, amounts to solving a system of linear equations. Once the ci are determined, the residual

r(S ) = f1 (S ) + g1 (S )VS (S ) can be computed. The optimal value of S is then chosen to make r(S ) = 0.

Example: Optimal Fish Harvest

Recall the optimal sh harvesting problem from Section 11.1.4. The value function solves the coupled PDE

V =

S (1 S=k)VS + 12 2 S 2 VSS for S < S 1 2 2 ES + (S (1 S=k) ES ) VS + 2 S VSS for S > S

with S determined by = VS (S ) and continuity of V and VS at S . For present purposes, impose the scale normalization = k = E = 1 (by choosing scales for money, sh quantity and eort level). It is a good idea to transform this problem by setting

z = ln(S ) ln(S ): This transformation has two eects: rst, it simpli es the dierential equation by making the coeÆcients constant or linear in S , and, second, it places the boundary between the two solution functions at z = 0.

CHAPTER 12.


414

The transformation necessitates rewriting the value function in terms of z , say as v (z ). The transformation implies that

S = S ez ; vz (z ) = SVS (S ) and

vzz (z ) = S 2 VSS (S ): The transformed Bellman equation with the scale normalizations is

v =

for z < 0 (1 S ez ) 12 2 vz + 21 2 vzz 1 1 z z 2 2 S e + (1 S e ) 2 1) vz + 2 vzz for z > 0 :

It will be useful to rewrite this to isolate the S terms v 12 s 2 vz 12 2 vzz + S ez vz = 0 for z < 0 v 12 2 1 vz 12 2 vz z + S ez vz = S ez for z > 0 :

The two functions are coupled by imposing continuity of v and vz at z = 0. Technically there are also boundary conditions as z goes to 1 and 1 , but we will ignore these for the time being. Now let's approximate the two functions using 0 (z )c0 and 1 (z )c1 , where the i are ni -element basis vectors and the ci are the coeÆcients associated with these bases (not surprisingly, we will use Chebyshev polynomial bases). For a speci c guess of S , the Bellman equation can be written l (z ) u (z )

1 s 2 0 (z ) 1 s 2 00 (z ) c + S [ez 0 (z )] c = 0 l l l l l 2 2 1 s 2 00 (z ) c + S [ez 0 (z )] c 1 2 1 0 (z ) u u u u u 2 2

for z < 0 = S ez for z > 0 :

Evaluating this expression at a set of nodes, zl 2 [a; 0], and zu 2 [0; b], where a and b are arbitrary upper and lower bounds, with a < 0 and b > 0. The boundary conditions at z = 0 for a given S are

l (0)cl

u (0)cu = 0

0l (0)cl

0u (0)cu = 0:

and

CHAPTER 12.


415

If we choose zl and zu to have nl 1 and nu 1 elements, respectively, this yields the nl + nu system of linear equations: 02 3 2 31 0 1 Bl 0 Dl 0 0 zu C B6 0 6 7C Bu 7 cl = B B6 7 + S 6 0 Du 7C B S e C: @4 l (0) 4 0 @ 0 A u (0) 5 0 5A cu 0 0 l (0) u (0) 0 0 0 which has the form (B + S D)c = S f:

The unknowns here are S (a scalar) and c (an n0 + n1 vector). The matrices B , D and f do not depend on either S or c; hence they can be prede ned. Furthermore, this system of equations is linear in c and hence can be easily solved for a given S , thereby obtaining an approximation to the value function, v . We can therefore view c as a function of S :

c(S ) = (B + S D) 1 S f: The optimal S is then determined by solving the (non-linear) equation

S 0l (0)cl (S ) = 0: It should be noted that the linearity in the coeÆcient vector c is not a special property of this problem; it arises from the linearity of the Bellman equation for a speci ed control function. We can summarize the approach in following way: De ne the matrices B and D, both (nl + nu) (nl + nu), and the vector f , (nl + nu ) 1.

De ne a function that

{ accepts S as an argument, { computes c(S ), { returns S 0l (0)cl Pass this function to a root nding solver.

A MATLAB implementation is displayed in Code Box 6. A script which computes and plots results is given in in Code Box 7; this was used to produce Figures 11.10, 11.11 and 11.12.

CHAPTER 12.


Code Box 12.6: Collocation File for Fish Harvesting Problem

416

CHAPTER 12.


417

Code Box 12.7: Script File for Fish Harvesting Problem S is indicated in these plots with an `*'. Notice that the value function is continuous up to its second derivative, but that V 00 exhibits a kink at S = S . This indicates why it is a good idea to break the value function apart and approximate it on each region separately, and pasting the two approximations together at the cut-o stock level. It also allows us to use the high degree of accuracy that polynomial approximations provide. One could, of course, approximate the entire value function with, say, a cubic spline, so long as you ensured that z=0 was a node. This would avoid the need to de ne two functions and thus has something to recommend it. However, it would require more nodes to achieve the same level of accuracy.

Example: Sequential Learning

In the previous example, the free boundary consisted of a single point. A more challenging boundary is required in the of learning-by-doing discussed in the previous chapter. This problem has the same form as that of the American option pricing problem. Here we provide an alternative framework for solving such problems. Recall that the problem involved solving

rV = P

c(Q) + VQ + (r

Æ )P VP + 21 2 P 2 VP P

on [P (Q); 1) [0; Qm ], where P (Q) is a free boundary to be determined. The boundary conditions are

P (Q)VP (P (Q); Q) = V (P (Q); Q); P (Q)VP P (P (Q); Q) = (

1)VP (P (Q); Q)

VP P (1; Q) = 0; where is the positive solution to 1 2 2 (

1) + (r

Æ )

r = 0:

Also a terminal condition at Q = Qm is known and, for states below the free boundary, the value function is known up to a constant:

V (P; Q) = A(Q)P :

CHAPTER 12.


418

The diÆculty with free boundaries is the unknown shape of the space over which the dierential equation must hold. To get around this problem, we discuss a transformation method that regularizes the boundary. The PDE can then be solved over a rectangular region for a given guess of the location of the free boundary. An iterative root nding method is then applied to determine the position of the boundary. This method can be used with nite dierences but it is better to use smoother approximations such as cubic splines or polynomials. To illustrate the transformation method we de ne

z = ln(P ) ln(P (Q)) and will denote v (x; Q) = V (P; Q). We are interested in solving the PDE for values of P on [P (Q); 1), which translates into values on z on [0; 1) (in practice we will typically truncate P ). Given this transformation it is straightforward to verify the following relationships between the original and the transformed problem:

vz (z; Q) = P VP (P; Q) vzz

vz = P 2 VP P (P; Q)

and

P 0 (Q) VQ = vQ v: P (Q) z Substituting these expressions into the Bellman equation and the boundary conditions yields: rv = P ez

C (Q) + vQ + (r

Æ

1 2 2

vz (0; Q) v (0; Q) = 0 vzz (0; Q) vz (0; Q) = 0 and lim (vzz (z; Q) vz (z; Q)) exp( 2z ) = 0:

z !1

0

P =P )vz + 21 2 vzz ;

CHAPTER 12.


419

One approach that can be used is to begin with an initial approximation to P (Q). Use this approximation to obtain an approximate solution to

rv = P ez

C (Q) + vQ + (r

Æ

1 2 2

0

P =P )vz + 21 2 vzz ;

vz (0; Q) v (0; Q) = 0 and lim (vzz (P; Q) vz (P; Q))=P 2 = 0:

P !1

This is a linear problem and hence can be solved easily with the extended method of lines (treating Q as the \time" variable) or collocation in both P and Q.3 The remaining boundary condition is then used to de ne a residual function

R(Q) = vzz (0; Q) vz (0; Q) that is used to solve for the optimal P (Q). [INCOMPLETE]

3 The collocation method has two drawbacks. First, it is an equilibrium method that

does not utilize the propagation nature of the problem and therefore is slower than need be. Furthermore, the accuracy of the method is limited by the fact that, as the number of nodes is increased, the matrix operator used to de ne v(z; Q) (given P (Q)) becomes very ill-conditioned. The ill-conditioning problem arises because of increasing large terms in the basis matrices for the derivatives. The largest term in the derivative basis is approximately equal to d Y i=1

4(ni

1)=(bi

ai )

while that of the second derivative basis is approximately d Y i=1

4(ni

1)3 =(bi

ai )2

for a d-dimensional problem. Thus as the ni rise, the largest element in the matrix rises as well.

CHAPTER 12.


420

Bibliographic Notes The hopscotch method for solving PDEs is discussed by Ames, pp. 124-6. The investment under uncertainty with mean reversion in the risk neutral return process is due to Dixit and Pindyck (pp. 161-163). We have simpli ed the notation by taking as given the risk-neutral process for the value of the completed investment. Their treatment took the actual value process as given and assumed that the required discount rate on the completed investment, , is constant. This is equivalent to assuming that the market price of risk, , is a constant: (V ) = ( r)= . It is also equivalent to assuming that the return stream, Æ , generated by the completed investment is quadratic in V :

Æ (V ) = ( m)V + V 2 : The practical import of these assumption is to decrease the long-run mean of the value process by the amount = = ( r)= when converting from the actual to the risk-neutral process. The time-to-build exercise is from Madj and Pindyck.

Exercises 1. Modify the code in the sh harvesting example to compute the value function with a single cubic spline approximation. Plot the value function and its 1st and 2nd derivatives as functions of S (not z ) and the residual function for the dierential equation as a function of z . Be sure to include 0 as a node. 2. Consider the problem under the assumption that the eort (E ) is not bounded (the problem thus becomes a barrier control problem). Write a program to solve for the value function and the optimal stock level that triggers harvesting. Use the same parameter values as in the bounded eort model ( = 0:1, = 0:05, = 0:2). Also compute and plot the optimal trigger stock level as a function of eort (E ), using the above values for other parameters. 3. Cost Uncertainty Dixit and Pindyck (pp. 345-351) discuss the problem of determining an investment strategy when a project takes time to complete and completion costs are uncertain. The cost uncertainty takes two forms.

CHAPTER 12.


421

The rst, technical uncertainty, arises because of unforeseen technical problems that develop as the project progresses. Technical uncertainty is assumed to be diversi able and hence the market price of risk is zero. The second type of uncertainty is factor cost uncertainty, which is assumed to have market price of risk . De ne K to be the expected remaining cost to complete a project that is worth V upon completion. The dynamics of K are given by

p

dK = Idt + IKdz + Kdw; where I , the control, is the current investment rate and dz and dw are independent Weiner processes. The project cannot be completed immediately because I is constrained by 0 I k.Given the assumptions about the market price of risk, we convert the K process to its risk neutral form and use the risk free interest rate, r, to discount the future. Thus we act \as if"

p

dK = (I + K )dt + IKdz + Kdw and solve

F (K ) = max E e I (t)

rT V

Z T 0

e

rt I (t)dt ;

where T is the (uncertain) completion time given by K (T ) = 0. The Bellman equation for this problem is

rF = max I I

(I + K )F 0 (K ) + 21 ( 2 IK + 2 K 2 )F 00 (K );

with boundary conditions

F (0) = V F (1) = 0: The optimal control is of the bang-bang type:

I=

0 if K > K k if K < K

CHAPTER 12.


422

where K solves 1 2 F 00 (K ) 2

F 0 (K ) 1 = 0:

Notice that technical uncertainty increases with the level of investment. This is a case in which the variance of the process is in uenced by the control. Although we have not dealt with this explicitly, it raises no new problems. a) Solve F up to an unknown constant for K > K . b) Use the result in (a) to obtain a boundary condition at K = K by utilizing the continuity of F and F 0 . c) Solve the deterministic problem ( = = 0) and show that K = k ln(1 + rV=k)=r. d) Write the Bellman equation for K < K and transform it from the domain [0; K ] to [0; 1] using

z = K=K : Also transform the boundary conditions. e) Write a computer program using Chebyshev collocation to solve for F and K using the following parameters:

V r k

= = = = = =

10 0:05 0 2 0:5 0:25:

g) What alterations are needed to handle the case when = 0 and why are they needed. 4. Investment with Time-to-Build Constraints Consider a situation in which an investment project, which upon completion will have a random value V and can be built by making a maximum current investment of k. Suppose that the value of the completed project evolves according to

dV = ( Æ )V dt + V dz;

CHAPTER 12.


423

where is the return needed to compensate investors for the systematic risk associated with the project and Æ = r, where r is the risk free rate of return. The amount of investment needed to complete the project is K , which is a controlled process:

dK = Idt: In this situation it is optimal to either be investing at the maximum rate or not at all. Let the value of the investment opportunity in these two cases by denoted F (V; K ) and f (V; K ), respectively. These functions are governed by the following laws of motion: 1 2V 2F VV 2

+ (r

Æ )V FV

rF

1 2V 2f VV 2

+ (r

Æ )V fV

rf = 0;

kFK

k=0

and

subject to the boundary conditions

F (V; 0) = V ÆK=k

lim FV (V; K ) = e

V !1

f (0; K ) = 0 f (V ; K ) = F (V ; K ) fV (V ; K ) = FV (V ; K ): V is the value of the completed project needed to make a positive investment. It can be shown that f (V ) = A(K )V , where 1 = 2

r

2

Æ

s

+

1 2

r

2

Æ

2

+

2r : 2

(5)

CHAPTER 12.


424

and A(K ) is a function that must be determined by the boundary conditions. This may be eliminated by combining the free boundary conditions to yield

F (V ; K ) = V FV (V ; K ): Summarizing, the problem is to solve the following partial dierential equation for given values of , r, Æ and k: 1 2 2 2 V FV V

+ (r

Æ )V FV

rF

kFK

k = 0;

subject to

F (V; 0) = V lim FV (V; K ) = e

V !1

ÆK=k

F (V ; K ) = V FV (V ; K ); where is given by (5). This is a PDE in V and K , with an initial condition for K = 0, a limiting boundary condition for large V and a lower free boundary for V that is a function of K . Write MATLAB code to solve the time-to-build problem for the following parameter values:

Æ=0 r = 0:02 = 0:2 k=1

Appendix A Mathematical Background A.1 Normed Linear Spaces A linear space or vector space is a nonempty set X endowed with two operations, vector addition + and scalar multiplication , that satisfy

x + y = y + x for all x; y 2 X

(x + y ) + z = x + (y + z ) for all x; y; z 2 X

there is a 2 X such that x + = x for all x 2 X

for each x 2 X there is a y 2 X such that x + y = ( ) x = ( x) for all ; 2 < and x 2 X

(x + y ) = x + y for all 2 < and x; y 2 X

( + ) x = x + y for all ; 2 < and x 2 X 1 x = x for all x 2 X .

The elements of X are called vectors. A normed linear space is a linear space endowed with a real-valued function jj jj on X , called a norm, which measures the size of vectors. By de nition, a norm must satisfy

jjxjj 0 for all x 2 X ; jjxjj = 0 if and only if x = ; 425

APPENDIX A.

MATHEMATICAL BACKGROUND

426

jj xjj = jj jjxjj for all 2 < and x 2 X ; jjx + yjj jjxjj + jjyjj for all x; y 2 X . Every norm on a linear space induces a metric that measures the distance d(x; y ) between arbitrary vectors x and y . The induced metric is de ned via the relation d(x; y ) = jjx y jj. It meets all the conditions we normally expect a distance function to satisfy:

d(x; y) = d(y; x) 0 for all x; y 2 X ; d(x; y) = 0 if and only if x = y 2 X ; d(x; y) d(x; z) + d(z; y) for all x; y; z 2 X . Norms and metrics play a critical role in numerical analysis. In many numerical applications, we do not solve a model exactly, but rather compute an approximation via some iterative scheme. The iterative scheme is usually terminated when the change in successive iterates becomes acceptably small, as measured by the norm of the change. The accuracy of the approximation or approximation error is measured by the metric distance between the nal approximant and the true solution. Of course, in all meaningful applications, the distance between the approximant and true solution is unknown because the true solution is unknown. However, in many theoretical and practical applications, it is possible to compute upper bounds on the approximation error, thus giving a level of con dence in the approximation. In this book we will work almost exclusively with three classes of normed linear spaces. The rst normed linear space is the familiar