A distributed approach solving partially flexible job

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect Wassim BOUAZZA¹², Yves SALLEZ², Bouziane BELDJILALI¹ ¹ LIO, Computer Sciences Department, University of Oran 1 Ahmed Ben Bella, ALGERIA ² LAMIH-CNRS, Department of Production Systems, University of Valenciennes & Hainaut-Cambrésis, FRANCE

2

Summary

1

Context & Motivation

2

Optimization Problem

3

Proposed approach

4

Experimentation

5

Conclusion & Perspectives

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Context & Motivation

Industry 4.0

3

CPPS Cyber-Physical Production System

More complexity

Partial flexibility of a cell makes the scheduling more difficult, complicates the search space, and increases the computation time (Kacem et al., 2002)

Deal with Partially Flexible Job-shop Scheduling Problem

Objectives

Consider realistic constraints: Interoperability, times variations …etc Heterarchical approach based on intelligent Cyber-Physical Product (CPP) Q-Learning effect to reduce weakness of distributed approaches

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Optimization Problem: Scheduling problem & heterogeneous machine The FJSP solving consists on select a sequence of services and an assignment of start/end times and resources for each service (Kacem et al., 2002) FJSP vs JSP

A service can be processed on several alternative resources

Total-FSP Partial-FSP

Job families as pre-grouped jobs with same process requirements (Chen et al., 2013) Processing & Setup time

• Family-dependent or Family-independent • Sequence-dependent or Sequence-independent

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

4

Optimization Problem: Scheduling in a Dynamic Environment

Centralized approaches Well adapted for small-sized problems

VS

Distributed approaches Produce a reactive response to face dynamic perturbation

Good Long-term optimization Inefficient and impractical for solving large-sized problems owing to the increased computation time requirement (Joo & Kim, 2015) Don’t deal well with perturbation

The decisions are then local and mainly do not go along with global performance of the system

This phenomenon, due to lack of visibility of the autonomous entities, is also called myopia (Zambrano Rey et al., 2014)

Use the past experience to reduce myopic phenomena by adding a Q-Learning technic

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

5

Proposed Approach: CPPS developed

6

Physical Level

Software Level

Learning cyber-physical products in manufacturing systems provide good opportunities for the future. The cyberphysical product coupled with machine learning method offers new chances to increase the product’s performance in term of flexibility and reactivity. (Bouazza et al., 2015)

D

D

D D D

Service Provider

D Cyberphysical Product

Traditionally, in the JSP, the assignment of operations to the SP is not a priori fixed. That is why many papers used a two-phase method to face the FJSP. (Trentesaux et al., 2013)

j

Manufacturing Cell

Decisional part

Physical Product

Resources

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Proposed Approach: Identifying the scheduling context

7

Flexibility (FCi)

Total SP1

Families

SP2

Partial

SP3

P

S

P

S

P

S

1

-

-

5

-

-

-

2

6

2

4

2

5

2

3

5

2

5

2

5

2

1Processing

Time 2Setup Time

Single machine Homogenous

Without

Resource-dependent

Homogenous

Family-dependent Processing Time (PTCi)

Heterogeneous Setup Time (STCi)

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Proposed Approach: Reinforcement Learning (QAlgo)

8

Cyber-Physical Product 1

Process Controller

Manufacturing Information System

Knowledge Database

Learning rates

Learning speed

a1∈ {SQ, LQE, SPT, SST}

3 Scheduler

2

Context Analysis & Identification

a2∈ {FIFO, SJF, HPF, LIFO}

Stochastic parameters

Assignment Module

A

Stochastic parameters

Sequencing Module

B

Q1 Table

Qt+1(St,A)=(α-1)Qt(St,A)+α(Rt+1+γQt(St,A))

Q2 Table

4 Reinforcing

5

Waiting for service completion

Post-Decisional Evaluation

Weighted Average Waiting Time=∑(wjWtj)/J

Internal model of CPP

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Experimentation: Simulation tool developed

CPP parameters

9

Manufacturing process

Decisional statistics

GUI of the MAS simulator developped

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Experimentation: Experimental data

10

Assumptions

1. 2. 3. 4. 5. 6. 7.

All SPs are assumed to be available at time 0. All CPPs arrive dynamically. Each CPP is assumed to have a priority (or criticality) that is a priori fixed. Each SP has an input queuing zone, which is assumed to be infinite. Each SP can process only one service at a time. Once a service begins on an SP, it cannot be interrupted. The availabilities and characteristics of SPs are supposed to remain unchanged.

SP1

Families

SP2

SP3

SP4

SP5

SP6

P1 S2

P

S

P

S

P

S

P

S

P

S

1

2

5

-

-

-

-

-

-

-

-

-

-

2

-

-

3

-

-

-

-

-

-

-

-

-

3

-

-

-

-

3

6

-

-

-

-

-

-

4

-

-

-

-

4

6

-

-

-

-

-

-

• Number of CPPs: J=500, j ∈ [1... 500]

5

3

-

3

-

3

-

3

-

3

-

3

-

• Number of families: F=9, f ∈ [1...9]

6

4

2

4

5

4

4

4

6

4

7

4

4

7

-

-

-

-

-

-

-

-

-

-

4

5

8

-

-

-

-

-

-

-

-

-

-

5

-

9

-

-

-

5

-

-

-

-

5

5

8

5

Input Data

• Priority range: wj ∈ [1...20] • CPP arrival times: Aij ∈ [1… 20999] • CPP arrival rate: 1 CPP per 2 time units

1Processing

Time 2Setup Time

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Experimentation: Results

11

Machine Selection Rules distribution

16 combinations of MSR x DR 10 Executions of QAlgo

Performance indicators

Dispatching Rules distribution

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Conclusion & Perspectives

• The scheduling of partially flexible job shop is a complex issue, especially in a dynamic environment. • A model of heterarchical Cyber-Physical Production System was presented. • Q-learning associated with an original contextualization make the problem "dynamically" redefined by CPP. • The use of learning techniques allows to enhance the global performance of the cyber-physical system. • Thus, the CPP can cope with these complicated scheduling problems in an efficient decentralized way.

• Those initial results encourage us to continue exploring this research way. • Work is already underway to extend the approach with multiple production stages. • It seems interesting to confront this method with even more realistic constraints: simultaneous production tasks and failures. • Comparative studies with metaheuristics as Genetic Algorithms or Particle Swarm Optimization.

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

12

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

Thanks for your attention

13

A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

14

Bouazza, W., Sallez, Y., Aissani, N. and Beldjilali, B. (2015) ‘A model for manufacturing scheduling optimization through learning intelligent products’, in Studies in Computational Intelligence. Springer International Publishing, pp. 233–241. doi: 10.1007/978-3-319-15159-5_22. Chen, G., Li, M. and Kotz, D. (2008) ‘Data-centric middleware for context-aware pervasive computing’, Pervasive and Mobile Computing, 4(2), pp. 216–253. doi: 10.1016/j.pmcj.2007.10.001. Joo, C. M. and Kim, B. S. (2015) ‘Hybrid genetic algorithms with dispatching rules for unrelated parallel machine scheduling with setup time and production availability’, Computers & Industrial Engineering, 85, pp. 102–109. doi: 10.1016/j.cie.2015.02.029. Kacem, I., Hammadi, S. and Borne, P. (2002) ‘Approach by localization and multiobjective evolutionary optimization for flexible job-shop scheduling problems’, IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 32(1), pp. 1–13. doi: 10.1109/TSMCC.2002.1009117. Trentesaux, D., Pach, C., Bekrar, A., Sallez, Y., Berger, T., Bonte, T., Leitão, P. and Barbosa, J. (2013) ‘Benchmarking flexible job-shop scheduling and control systems’, Control Engineering Practice, 21(9), pp. 1204–1225. doi: 10.1016/j.conengprac.2013.05.004. Zambrano Rey, G., Bonte, T., Prabhu, V. and Trentesaux, D. (2014) ‘Reducing myopic behavior in FMS control: A semi-heterarchical simulationoptimization approach’, Simulation Modelling Practice and Theory, 46(0), pp. 53–75. doi: 10.1016/j.simpat.2014.01.005.