A Markov Decision Model to Evaluate Outsourcing in ...

International Congress on Logistics & Supply Chain (CiLOG2014)

A Markov Decision Model to Evaluate Outsourcing in Closed-Loop Supply Chains for Hazardous Materials

Abstract—In recent years, environmental issues have become a main topic worldwide. Governments around the world have established laws and policies to reduce the impact of industrial activity, forcing companies -especially those who produce or manage hazardous materials- to satisfy specific requirements on their supply chain systems. This is why many companies consider outsourcing as an option for these functions. Through this research, a set of Markov decision models are developed to support outsourcing decisions in a closed-loop supply chain system for hazardous materials. The models are based on the risk levels and sales behavior of the product considered. An optimal monotone nonincreasing policy is identified, which provides valuable insights for decision-makers involved in such systems. Keywords—Closed-Loop Supply Chain; Markov Decision Model; Hazardous Materials; Outsourcing

I.

INTRODUCTION

Due to laws, international agreements, pressure from society, among other factors, human safety and health, environmental protection and security concerning hazardous materials supply chain are main topics for many countries, industries and organizations around the world [1]. Hazardous materials are defined and regulated by a number of agencies around the world, whom establishes the regulations that concern the handling, storage and distribution of hazardous materials [2]. For this reason, the shipping of hazardous materials can only be performed by carriers that are registered and only when the material is properly classed, described, packaged, marked, labeled, and in condition for shipment. Integrating environmental concerns into supply chain management has become increasingly important for manufacturers to gain and maintain competitive advantage [3]. As more executives adopt environmental practices, supply chain strategies will only increase in importance. As companies focus more tightly on their core competencies, they will rely more heavily on their suppliers for non-core activities such as the transportation, recovery and disposal of their products [4]. The characteristics of each product and activity suggest specific strategies. Low-value activities require little attention and might even be completely outsourced [4]. With an outsource strategy, companies can improve benefits while they are focusing on their core activity [5].

II.

ENVIRONMENTAL RISK

The use of hazardous materials can cause unintentional accidents. Incidents have occurred in every system of the hazardous materials supply chain, including platforms, all modes of transport, chemical plants, terminals and storages [1]. Managers have come to realize that a large and increasing amount of environmental risk can be found in nearly every company’s supply chain, increasing the importance of the decisions in this area [4]. This risk implies that the companies must be more specialized on each one of these activities or outsource some of them in order to focus on their core activity [3]. A risk/cost framework for the hazardous materials management system must include an assessment of the risk due to storage, transportation, treatment and disposal. The risk cost calculation may vary by the type of activity involved, but it must be according to the accident rate and possible affected population [6]. Only for the case of petroleum products, from 1990 to 2000, 36 accidents were reported in the management of these materials, which resulted in more than 2200 deaths and about 3,000 injured people [7]. III.

CLOSED-LOOP SUPPLY CHAINS AND THEIR OUTSOURCE

The research of the supply chain management has passed through various stages, from the individual activities optimize to an entire analysis of the whole chain. This is the case of the closed-loop supply chain management, which is define as “the design, control, and operation of a system to maximize value creation over the entire life cycle of a product with dynamic recovery of value from different types and volumes of returns over time” [8]. In a Closed-loop supply chain (CLSC) network, which is the focus of this study, integrated management of bidirectional material movements that occur in the form of forward and reverse flows is of interest [9]. The major difference between CLSC and traditional supply chains is for a forward supply chain, the costumer is at the end of the process, and for a CLSC, there is value to be recovered from the costumer or enduser. The value to be recovered is significant; only in the United States is over $50 billion in annual sales of remanufactured products [10]. Reverse flows or product recovery activities include the used-product acquisition, reverse logistics, product disposition

Mexican Logistics & Supply Chain Association (AML)

International Congress on Logistics & Supply Chain (CiLOG2014)

(sort, test, and grade), remanufacturing, repair and remarketing. Product returns may occur for a variety of reasons over the product life cycle, as commercial returns, end-of-use returns, end-of-life returns, repair and warranty returns [8].

known probability distribution) and the company does not directly control. Fig. 1. Closed-Loop Supply Chain Model

The major issues in CLSC are about the product design: recoverable value, form factor, ease of recovery; consumer behavior: reuse options, reusability; supply chain actors: profitable for whom?, length of the chain [10]. Additional, there are some complicating characteristics for planning and controlling a supply chain with remanufacturing of external returns. Some of them are the requirement for a reverse logistics network, the uncertain timing and quality of cores, the uncertainly in material recovered from cores, the problem of stochastic routings for materials and highly variable processing times and the need to balance returns of cores with demands for remanufactured products [11]. Supply chain management is recognized as a strategy for improving competitiveness by improving customer value and reducing cost [12]. Given the logistics costs that are implied in this activities and the customers’ demands for shorter order cycles, some companies consider outsourcing these activities to third party logistics (3PL) providers. Warehouse, distribution and reverse logistics are the most common activities to outsource [13]. Also, it is common to outsource multiple logistics services, but just a few companies outsource the manufacture or production activity [14]. Reference [5] find that firms can improve customer service and reduce costs by outsourcing packages of functions and suggest as the main benefits of 3PL: cost savings, operational efficiency, flexibility and improved customer service. Generally, outsourcing logistics functions is a long term decision [15]. This is consistent with the survey by Boyson et al. (1999), where the respondents who have outsourced an activity in their company, only 4% reported that they stopped outsource this activity. IV.

PROBLEM DESCRIPTION

Because many of the activities of the closed-loop supply chain do not represent the core business of the company, one of the most important decisions for any company is which activities should be outsourced to a 3PL and when they should be outsourced, in order to minimize the total expected cost for the whole cycle. The research aims to develop a model for decision making for the problem described. The objective is minimize the expected total cost over the entire planning period, this by a Markov Decision Model (MDM), which is ideal for stochastic problems. For this model it is assumed that the CLSC consists of six main activities “Fig. 1”. Reference [14] indicate that in their surveys conducted in 2004, 67% of companies outsource distribution activities, warehousing activities 46% and 33% reverse logistics (RL) activities. According to this, it is assumed that these activities can be outsourced, while the production is a core activity and never will be outsourced and the market and re-use/disposal activity are probabilistic (with a

V.

MARKOV DECISION MODEL DEVELOPED

We define the follow notation for the MDMs developed: Sets j

Subactivities or expenses,

Parameters cj π β L W

Cost of subactivity or expense j Unit shortage cost Environmental risk cost, β ≥ 0 Length of the product life cycle Time length defined by the firm to continue managing the returns for the product analyzed T Length of the study horizon, T = L + W t Decision epoch, , t={1, …, T-1}, where decision epoch t represents the end of period t. λt Expected sales for period t r1 Rate for sales increase, 0 ≤ r1 ≤ 1 r2 Rate for devolutions, 0 ≤ r2 ≤ 1 Random variables xt Xt

Amount of units sold by the firm during period t Cumulative sales experienced by the firm from period 1 through the end of period t yt Number of units returned in period t Yt Cumulative number of units returned from period 1 to the end of period t State variables

Mexican Logistics & Supply Chain Association (AML)

International Congress on Logistics & Supply Chain (CiLOG2014)

xt nt

Amount of units sold by the firm during period t Number of units sold and not returned at the end of period t, nt = Xt - Yt Capacity at period t

kt

(4) Rewards. The following reward structure is defined for actions a = 0 or 1.

A. Model Assumptions • Returns are a function of the number of units previously sold but not yet returned. Each unit has a binomial distribution probability of being returned [15]. •

•

Sales are assumed to be distributed Poisson with mean λ. Average sales change at a known rate in each period [16]. There is a cost per unit for the collection and handling of a returned unit, which is considered less than the savings generated by remanufacturing one unit [17].

• Given that warehouse, distribution and RL does not represent a core activity, it is also assumed that once the outsourcing decision is taken, it remains in place for the rest of the problem horizon as an absorbing state [15]. •

If any activity is still done internally and it incurs in a shortage, then there would exist a cost associated to meet this demand. If the activity is outsourced, the 3PL would always have enough capacity to satisfy such a demand [15].

Because of the last assumption, each activity can be considered independent such that the decision will be made with a different and independent model, being analyzed together as a final stage in this research. B. Markov Decision Model for Warehouse States. The system state at each decision epoch t is defined as St = {kt, xt}, for t = 1, …, T. At decision epoch 0, the system state is S0 = {k0, 0}, where k0 > 0. Also, xT = 0. Actions. Given the purpose of the MDM, we assume that two actions are available: a = 0 Continue performing the activity internally, by updating the firm’s capacity to the expected amount of sales in the next period, i.e., k(t+1) = E[x(t+1)] = E[xt](r1+1) = λt(r1+1). a = 1 Adopt an outsourcing strategy for the activity by having a 3PL perform such activity and taking the firm’s activity capacity to zero; i.e., k(t+1) = 0. Transition probabilities. As the sales at each period follow a Poisson distribution, the transition probabilities among states are defined as p(t+1)[(k(t+1) , x(t+1) | (kt , xt) , a]; i.e.,

(5) (6) (7) +

Where (·) denotes max( · , 0). Environmental Risk Cost. This cost is defined as β=ρ*γ*α*δ [6]. Where: ρ Probability of an accident occurs γ Cost per person affected α Affected area in case of accident δ Population density of the area affected C. Markov Decision Model for Distribution States. The system state at each decision epoch t is defined as St = {kt, xt}, for t = 1, …, T. At decision epoch 0, the system state is S0 = {k0, 0}, where k0 > 0. Also, xT = 0. Actions. Given the purpose of the MDM, we assume that two actions are available: a = 0 Continue performing the activity internally, by updating the firm’s capacity to the expected amount of sales in the next period, i.e., k(t+1) = E[x(t+1)] = E[xt](r1+1) = λt(r1+1). a = 1 Adopt an outsourcing strategy for the activity by having a 3PL perform such activity and taking the firm’s activity capacity to zero; i.e., k(t+1) = 0. Transition probabilities. As the sales at each period follow a Poisson distribution, the transition probabilities among states are defined as p(t+1)[(k(t+1) , x(t+1) | (kt , xt) , a]; i.e.,

(8)

(9) (10) (11) Rewards. The following reward structure is defined for actions a = 0 or 1.

(1) (12) (2)

(13)

(3)

Mexican Logistics & Supply Chain Association (AML)

International Congress on Logistics & Supply Chain (CiLOG2014)

(14) Where (·)+ denotes max( · , 0). Environmental Risk Cost. This cost is defined as β=ρ*γ*χ*d/f [6]. Where: ρ Probability of an accident occurs γ Cost per person affected χ Population of the destiny d Travel distance f Distance between the route and the destiny

E. System Dynamic For all the models, the system follows the dynamic presented at “Fig. 2” for each period t. At the end of the last period of production and returns, all the capacity remaining in the system is sold. Fig. 2. System Dynamic

D. Markov Decision Model for RL States. The system state at each decision epoch t is defined as St = {kt , nt}, for t = 1, …, T. At decision epoch 0, the system state is S0 = {k0 , 0}, where k0 > 0. Actions. Given the purpose of the MDM, we assume that two actions are available: a = 0 Continue performing the activity internally, by updating the firm’s capacity to the expected amount of returns in the next period, i.e., k(t+1) = E[y(t+1)] = E[n(t+1)]r2 = (nt (1 - r2) + xt)r2.

The action decided for each period is defined by the maximum reward earned by continuing optimally from state (St) onwards for each independent activity at each period t. The optimal policy for each activity can be obtained by solving recursively:

a = 1 Adopt an outsourcing strategy for the activity by having a 3PL perform such activity and taking the firm’s activity capacity to zero; i.e., k(t+1) = 0. Transition probabilities. As the returns in each period follow a binomial distribution,, the transition probabilities among states are defined as p(t+1)[(k(t+1), nt + xtr1 - l) | (kt, nt) , a]; i.e.:

(15)

(16) Rewards. The following reward structure is defined for actions a=0 or 1.

(17) (18) (19) Where (·)+ denotes max( · , 0). Environmental Risk Cost. This cost is defined as β=ρ*γ*α*δ [6]. Where: ρ Probability of an accident occurs γ Cost per person affected α Affected area in case of accident δ Population density of the area affected

(20) The optimal policy will be the result of the decisions taken for each activity for each period. VI.

CONDITIONS FOR A MONOTONE OPTIMAL POLICY

In principle, this problem can be solved recursively backwards from period T to identify an optimal action for each possible state. However, depending on the conditions of the problem, the number of states to evaluate could grow very large [15]. For this reason, it is desirable to identify a simple form for an optimal policy. This policy corresponds to a threshold (in terms of the sales and cumulative returns given a particular capacity level), beyond which the outsourcing action a = 1 is optimal. Sets of conditions exist that ensure that optimal policies are monotone in the system state [18]. One set of conditions stated for the existence of a monotone optimal policy is: 1) R(t+1)[(kt, nt), (a)] is nonincreasing in (kt, nt) for a={0,1}. 2) q(t+1)[(k(t+1), n(t+1)) = kl | (k, n), (a)] is nondecreasing in (kt, nt) for all kl and a = {0,1}. 3) R(t+1)[(kt, nt), (a)] is a superadditive function on (kt, nt) × (a). 4) q(t+1)[(k(t+1), n(t+1)) = kl | (k, n), (a)] is a superadditive function on (kt, nt) × (a). 5) R(T+1)[(kT, nT)] is nonincreasing in (kT, nT).

Mexican Logistics & Supply Chain Association (AML)

International Congress on Logistics & Supply Chain (CiLOG2014)

Where (25) (21) When all of these conditions are satisfied, there exists a monotone non-increasing policy that is optimal. A. Condition 1 This condition holds when the reward of either action does not increase with the number of items sold for the MDM for warehouse and distribution, and with the number of items sold but not yet returned for the MDM for RL. 1) Condition 1 for MDM for warehouse and distribution For a = 0, the condition Rt[(kt, xt), 0] ≥ Rt[(kt, xt + i), 0] for i > 0, is equivalent to:

(22) for i > 0, where c1, c2, c3, c4, β, π > 0 and x is Poisson distributed with parameter λ. For a = 1, it requires that Rt [(kt, xt), 1] ≥ Rt[(kt, xt + i), 1] for i > 0. This is equivalent to -c5xt ≥ -c5(xt + i), which follows immediately from i > 0 and c5 > 0. 2) Condition 1 for MDM for RL For a = 0, the condition Rt[(kt, nt), 0] ≥ Rt[(kt, nt + i), 0] for 1 ≤ i ≤ Xt, is equivalent to:

for 1 ≤ i ≤ Xt. C. Condition 3 This inequality holds when for a fixed capacity, the incremental effect on the reward of switching to an outsourcing strategy increases with the number of sales or returns. 1) Condition 3 for MDM for warehouse and distribution This condition can be written as Rt[(kt, xt), 1] - Rt[(kt, xt), 0] ≤ Rt[(kt, xt + i), 1] - Rt[(kt, xt + i), 0] 2) Condition 3 for MDM for RL This condition can be written as Rt[(kt, nt), 1] - Rt[(kt, nt), 0] ≤ Rt[(kt, nt + i), 1] - Rt[(kt, nt + i), 0] D. Condition 4 This condition implies that the difference between the cumulative probability that sales or returns exceed a given number when taking the outsourcing option and when performing the activities internally, does not decrease with the sales or returns. 1) Condition 4 for MDM for warehouse and distribution This condition can be written as:

(26) where xt+ > xt-. It is satisfied as an equality because the action determines the next capacity level while random events affect only the sales. 2) Condition 4 for MDM for RL

(27) Where nt+ > nt-. It is satisfied as an equality because the action determines the next capacity level while random events affect only the sales and returns. (23) for i>0 For a = 1, it requires that Rt[(kt, nt), 1] ≥ Rt[(kt, nt + i), 1] for 1 ≤ i ≤ Xt. This is equivalent to -c5yt ≥ -c5(yt + i), which follows immediately from i > 0 and c5 > 0. B. Condition 2 This condition holds when it is more likely to meet or exceed a given number of sales or returns in the next period, if a higher number of sales or returns have been experienced up to the current period. 1) Condition 2 for MDM for warehouse and distribution This condition can be written as:

E. Condition 5 This condition implies that the terminal reward decreases with the number of sales for the MDM for warehouse and distribution and with the number of returns for the MDM for RL. 1) Condition 5 for MDM for warehouse and distribution The condition RT[(kT, xT), a] ≥ RT[(kT, xT + i), a] for i > 0 is satisfied because the terminal reward is defined by the capacity level only and not by the sales for that period. 2) Condition 5 for MDM for RL The inequality RT[(kT, nT), a] ≥ RT[(kT, nT + i), a] for 1 ≤ i ≤ Xt can be written as πnT ≤ π(nT + i), which follows from π > 0. VII. CONCLUSIONS AND FUTURE WORK

(24) for i > 0. 2) Condition 2 for MDM for RL

The importance of the CLSC for hazardous materials and the outsourcing as a strategy in order to minimize costs and focus on the core business has been stated in this study. Also,

Mexican Logistics & Supply Chain Association (AML)

International Congress on Logistics & Supply Chain (CiLOG2014)

a MDMs to support the outsource decision in a CLSC for hazardous materials was developed. The models considers several elements that are critical in defining the characteristics of an CLSC for hazardous materials, such as expected sales, uncertainty in the return volume, capacity, operating, shortage and environmental risk costs. Some sufficient conditions for the existence of an optimal monotone nonincreasing policy have been generally described. The existence of an optimal monotone nonincreasing policy implies the presence of a threshold above which it is optimal to follow an outsourcing strategy for the RL system; otherwise, to continue performing the RL activities internally. This threshold is defined in terms of a partial ordering for the system states, where given a fixed capacity at a decision epoch, the states are ordered according to the sales and cumulative returned units, such that if that volume goes above a particular level, then it is optimal to follow an outsourcing strategy and take advantage of the economies of scale implied by involving a 3PL. As a future research, the conditions 1, 2 and 3 for the existence of an optimal monotone nonincreasing policy must be fully verified and proven, as well as a numerical evaluation for a case of study must be done. Also, many other important characteristics of the system could be analyzed in order to take a better decision, as the life cycle length.

[3]

[4]

[5]

[6]

[7] [8] [9]

[10] [11]

[12] [13]

[14]

REFERENCES [1]

[2]

Mullai, A. and Larsson, E. Hazardous material incidents: Some key results of a risk analysis. WMU Journal of Maritime Affairs, 7(1): 65108, 2008. Murray, M. Federal Hazardous Materials Transportation Regulations. About.com. Retrieved February, 2013, from http://logistics.about.com/od/legalandgovernment/a/Federal-HazardousMaterials-Transportation-Regulations.htm

[15]

[16]

[17] [18]

Zhu, Q., Sarkis, J, and Lai, K. Confirmation of a measurement model for green supply chain management practices implementation. International Journal of Production Economics, 111: 261-273, 2008. Handfield, R., Sroufe, R. and Walton, S. Integrating environmental management and supply chain strategies. Business Strategy and the Environment. 14: 1-19, 2005. Boyson, S., Corse, T., Dresner, D. and Rabinovich, E. Managing effective third party logistics relationships: What does it take? Journal of Business Logistics, 20: 73–100, 1999. Killmer, K. A risk/cost framework for logistics policy evaluation: Hazardous waste management. The Journal of Business and Economics Studies. 8(1): 51-65, 2002. Alcantara, M., and Gonzalez, T. Modelacion de radios de afectacion por explosions en instalaciones de gas. CENAPRED. 2001. Mexico. Guide, V.D. and Van Wassenhove, L. The evolution of closed-loop supply chain research. Operations Research. 57(1): 10-18, 2009. Easwaran, G. and Üster, H. A closed-loop supply chain network design problem with integrated forward and reverse channel decisions, IIE Transactions, 42(11): 779 — 792, 2010. Guide, V. D. and Van Wassenhove, L. Special section on closed-loop supply chains. Interfaces 33(6): 1–2, 2003. Guide, Jr., V.D.R. Production planning and control for remanufacturing: Industry practice and research needs. Journal of Operations Management 18(4): 467-483. 2000. Mentzer, J.T. Fundamentals of Supply Chain Management, Sage, United States. 2004. Arroyo, P. Gaytan, J. and Boer, L. A survey of third party logistics in Mexico and a comparison with reports on Europe and USA. International Journal of Operations and Production Management. 26(6): 639-667, 2006. Lieb, R. and Bentz, B. The use of Third-Party Logistics Services by Large American Manufacturers: The 2004 Survey. Transportation Journal. 44(2): 5-15, 2005. Serrato, M. A., Ryan, S. M. and Gaytán, J. A Markov decision model to evaluate outsourcing in reverse logistics. International Journal of Production Research, 45(18): 4289 – 4315, 2007. Chen, Y. An Inventory Model with Poisson Demand and Linear Lost Sales, International Journal of Service Science, Management and Engineering. 1(1): 17-20, 2014. Savaskan, R., Bhattacharya, S. and Van Wassenhove, L. Closed-Loop Supply Chain Models. Management Science. 50(2): 239-252, 2004. Puterman, M.L., Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons. New York, NY. 1994.

Mexican Logistics & Supply Chain Association (AML)