Phase Transitions and Structure in Combinatorial ... - Semantic Scholar

6 downloads 63409 Views 1MB Size Report
IJCAI-01 Tutorial on Phase. Transitions and ... e.g.: scheduling, circuit design, cryptography, theorem proving, games .... phase transitions in AI. • pruning in ...
Phase Transitions and Structure in Combinatorial Problems Carla P. Gomes Tad Hogg Toby Walsh Weixiong Zhang

a statistical regularity in search • relates structure to behavior – e.g., problem features => search cost

• analogous to physical phase transitions • useful to improve search heuristics – i.e., new problem solving strategies

Gomes, Hogg, Walsh, Zhang

2

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

combinatorial search e.g.: scheduling, circuit design, cryptography, theorem proving, games, matching models to data, genetics, ...

• types – decision (NP): rapid test of candidate solutions – optimization

• behavior – worst case (e.g., is P=NP?) – typical performance of heuristics

Gomes, Hogg, Walsh, Zhang

3

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

10 variable 3-SAT example • each clause gives a “nogood” – inconsistent assignment for the 3 variables • e.g., v1=true, v4=false, v5=false

• search method: depth-1st backtrack – cost: steps to find a solution (if any)

• examine search tree as nogoods added – i.e., a particular sequence of clauses, randomly generated

Gomes, Hogg, Walsh, Zhang

4

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

search tree v1=true

v1=false

examined states

34 nogoods 1st solution Gomes, Hogg, Walsh, Zhang

another solution

5

other consistent states

many solutions little pruning IJCAI-01 lowTutorial coston Phase Transitions and Structures-SA4

search tree

55 nogoods Gomes, Hogg, Walsh, Zhang

6

one solution some pruning IJCAI-01 highTutorial coston Phase Transitions and Structures-SA4

search tree

110 nogoods Gomes, Hogg, Walsh, Zhang

7

no solutions much pruning IJCAI-01 lowTutorial coston Phase Transitions and Structures-SA4

search cost 140 120

cost

100

no solutions

80 60 40 20 0 0

25

underconstrained Gomes, Hogg, Walsh, Zhang

50

75 nogoods 8

100

125

150

overconstrained IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

observations • peak in cost vs. number of nogoods – competition between • decreasing number of solutions • increasing search path pruning

• peak corresponds to loss of last solution • peak location depends on choice of nogoods

Gomes, Hogg, Walsh, Zhang

9

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

generality • example uses – one search method – one sequence of problem instances

• Why is this interesting? – similar behavior for many • search problems • search techniques

– hence: simple parameter indicates likely cost

Gomes, Hogg, Walsh, Zhang

10

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

outline • statistical approach to search – local properties and global behavior

– problem ensembles • phase transitions • impact of structure • summary

Gomes, Hogg, Walsh, Zhang

11

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

statistical approach to search • How do many repeated local decisions combine to give global behavior? – e.g., heuristic choice accuracy => search cost

• key issue: – local properties => global behavior – addressed by statistical mechanics

Gomes, Hogg, Walsh, Zhang

12

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

statistical approach • identify problem ensemble • identify structure parameters – easy to measure (computationally efficient) – describe an average, local property of problem

• relate parameters to global behavior – e.g., search cost

Gomes, Hogg, Walsh, Zhang

13

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

problem ensembles • class of problem instances • probability for each instance • why? – focus on typical behaviors in class (averagecase) – not single instance, not worst-case

Gomes, Hogg, Walsh, Zhang

14

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

parameterized ensembles • problem size • properties of constraints – e.g., number, tightness

• can also specify global properties – e.g., number of solutions

Gomes, Hogg, Walsh, Zhang

15

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

simple ensemble examples • random k-SAT: n variables, m clauses – each clause picked independently • allowing duplicates

• random graphs: n nodes, e edges – each edge picked randomly • but no duplicates

Gomes, Hogg, Walsh, Zhang

16

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

“real” problems • constraints tend to be local and more clustered than random – e.g., small worlds graphs

• harder to work with than easily generated random ensembles – both analytically and empirically

Gomes, Hogg, Walsh, Zhang

17

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

nature of constraints • physical interactions – strength decreases with distance

• designed artifacts – often nearly decomposable, hierarchical

• evolved systems – e.g., ecology, economy, common law, social networks

Gomes, Hogg, Walsh, Zhang

18

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

local measures: global behavior • local measure of problem structure – number and tightness of constraints

• global behaviors – probability for a solution – search cost – solution quality

Gomes, Hogg, Walsh, Zhang

19

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

outline • statistical approach to search • phase transitions – decision problems – optimization – other complexity classes • impact of structure • summary

Gomes, Hogg, Walsh, Zhang

20

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

phase transitions • abrupt changes in global property of typical instance in a problem ensemble • characteristic of problem ensemble, not problem instance

Gomes, Hogg, Walsh, Zhang

21

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

phase transition phenomena • physics: – magnets, superconductors vs. temperature

• geology: – fluid percolation vs. porosity of rocks

• biology – epidemic spread vs. population density

Gomes, Hogg, Walsh, Zhang

22

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

phase transition: boiling • small change in average energy per molecule (temperature) • gives large change in overall state

Gomes, Hogg, Walsh, Zhang

23

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

commonality? • key idea: abrupt change in overall system property when size is large • why a useful general concept? – – – –

mathematical commonality in spite of different details suggests analyses, e.g., finite-size scaling universality: insight from simple models with “essential features”

Gomes, Hogg, Walsh, Zhang

24

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

phase transitions in AI • pruning in heuristic search • associative memory models • pattern matching

Gomes, Hogg, Walsh, Zhang

25

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

outline • statistical approach to search • phase transitions – decision problems – optimization – other complexity classes • impact of structure • summary

Gomes, Hogg, Walsh, Zhang

26

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

some decision problems • SAT – random 3-SAT: clause/variables near 4.2

• graph coloring – random graph, 3-COL: edges/nodes near 2.2

• number partitioning – L integers, random in [0,..,2L]

Gomes, Hogg, Walsh, Zhang

27

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

observe regularity • search cost varies greatly among problem instances of same size • hard instances rare and concentrated hard instance

Gomes, Hogg, Walsh, Zhang

28

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

intuition • few constraints: many solutions • many constraints: no solutions EASY

– bad choices pruned quickly

• intermediate number of constraints: – few solutions HARD – bad choices pruned only after many steps

Gomes, Hogg, Walsh, Zhang

29

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

typical cost of search heuristics hard cases: • near phase transitions • fastest exponential growth of cost

large size

behavior of many • NP problem ensembles • search methods (but not generate & test)

easy cases: • underconstrained: many solutions • overconstrained: rapid pruning typical cost

small size

underconstrained Gomes, Hogg, Walsh, Zhang

30

overconstrained

IJCAI-01 Tutorial on Phase number of constraints Transitions and Structures-SA4

typical behavior of solubility • for large problem ensembles – fraction of soluble instances drops suddenly from near 1 to near 0 – close to location of =1 =1

prob(soln)

1

few soluble problems but with very many solutions

0

exp. large

# constraints Gomes, Hogg, Walsh, Zhang

31

exp. small IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

solubility transition and cost • high costs associated with drop in solutions • not the whole story: cost peak for problems with fixed number of solutions !

Gomes, Hogg, Walsh, Zhang

32

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

theory • Why the association of high cost with transition, on average? • simplified models of the behavior: – exact: simple tree search • trivial problem • simple illustration of abrupt change

– approximate: graph coloring

Gomes, Hogg, Walsh, Zhang

33

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

tree search • branching ratio b, depth d • no solution – e.g., prior choices are already inconsistent

• heuristic prunes with prob. p – average branching ratio: z = b p

• expected cost for depth-1st backtrack

Gomes, Hogg, Walsh, Zhang

34

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

expected cost d=10

d=4

cost

d=20

constant

exponential

z

Gomes, Hogg, Walsh, Zhang

35

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

expected cost • average branching ratio: z = b p • avg. cost from depth j: Cj=1+z Cj+1 • avg. cost from root

 1 z 1  has abrupt changes when d infinite z − 1 36 IJCAI-01 Tutorial on Phase Gomes, Hogg, Walsh, Zhang Transitions and Structures-SA4

hard graph coloring • many consistent colorings for much of the graph • few for the whole graph • hence much backtracking

Gomes, Hogg, Walsh, Zhang

37

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

relate cost to structure • color k of n nodes, with 3 colors • 3k possible colorings – most with about equal use of each color

• random edge in subgraph with prob. 2 • edge in subgraph links same colors with ≈ (k/n) prob. 1/3

Gomes, Hogg, Walsh, Zhang

38

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

partial colorings • assuming independent constraints, expected number of partial colorings of k nodes is

 1 k k Nk = 3  1−    3  n Gomes, Hogg, Walsh, Zhang

39

2

  

e

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

behavior • few edges: monotonic increase, little backtrack • more edges: hard to continue beyond maximum, much backtrack • many edges: no solutions, pruning terminates search early

Gomes, Hogg, Walsh, Zhang

40

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

ln(# consistent colorings)

behavior: 100 nodes e=75

e=200

e=300

# colored nodes edges prune large subgraphs more than small ones Gomes, Hogg, Walsh, Zhang

41

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

independence assumption • • • •

simplifies analysis gives qualitative behavior reasonable quantitative accuracy cf., “mean-field” physics theories

Gomes, Hogg, Walsh, Zhang

42

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

which ensemble? • examples – random graphs – small-world graphs

• cf., physical systems – all states consistent with (few) parameters are equally likely – accuracy is an empirical question – e.g., classical => quantum statistical mechanics

Gomes, Hogg, Walsh, Zhang

43

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

open questions • math: prove thresholds in simple ensembles • empirical: which ensembles are realistic? – perhaps additional parameters required – e.g., random graph vs. small worlds graph

Gomes, Hogg, Walsh, Zhang

44

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Computational Complexity and Phase Transitions of Optimization Problems

Gomes, Hogg, Walsh, Zhang

45

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Combinatorial Optimization: Example 1 •

The Traveling Salesman Problem (TSP) – Given: a set of cities and inter-city distances – Goal: a shortest tour visit each city once – Asymmetric TSP (ATSP): distance(i,j) distance(j,i) – Many real-world applications • VLSI routing • Scheduling

– A well-studied NP-hard problem – TSP with distances 1 and 2 is still NP-hard! (Papadimitriou&Yannakakis 93) – NP-hardness is a worst-case measure – How does the average-case complexity behave?

Gomes, Hogg, Walsh, Zhang

46

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Combinatorial Optimization: Example 2 • Interleaving decision and execution: Decision making under limited information and restricted resources

Which path to take? decision

– Collecting information with restricted computational resources • Exploration by lookahead search

– Reasoning based on limited information • Calculation

Lookahead to collect information

– Executing the best decision • Action

• This idea is the foundation of almost all game playing programs (Shannon 1950) Gomes, Hogg, Walsh, Zhang

47

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Combinatorial Optimization: Example 2 •

Lookahead search in sliding-tile puzzle: • backup the minimal cost at a fixed depth



f=g+h: g=depth of a node, h= Manhattan distance to the goal. In one step, g increases by one; h increases or decreases by one; and so f increases by 2 or remains the same. 1 2 3 4 5 6 7 8 1 2 34 5 67 8 1 2 3 45 6 78

1 4 2 3 5 6 7 8

Gomes, Hogg, Walsh, Zhang

Anomaly of sliding-tile puzzles: For a given search depth, a large puzzle is easier to search; or for a given amount of computation, a large puzzle can be searched to a deeper level.

1 2 3 8 4 7 6 5 3 1 2 4 5 6 7 8

3 1 2 4 5 6 7 8

3 1 2 6 4 5 7 8

48

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Content • Expected complexity of optimization search – Abstract (statistical) tree model – Complexity results

• Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions Gomes, Hogg, Walsh, Zhang

49

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Abstract Model for Complexity Analysis • Incremental random tree – Node = states 

– Edges = state transitions 

– Tree depth d



7

– Edge costs (operator costs): i.id non-negative random variable (may take 0) – Node costs (state quality): sum of edge costs from the root to the nodes 50



2

– Branching factor b= # of children

Gomes, Hogg, Walsh, Zhang

0

1 

2

5





2

3







 





4

5

6

8

3

7

4

Goal state

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Complexity Analysis: A Quiz • • •

Given two random trees generated with the same edge cost distribution Finding the goal nodes (minimum-cost leaf nodes) using the same search algorithm (BFS, DFBnB, IDA*, …) Which tree is easier to search?

• Random tree 1: b=3, d=100

Gomes, Hogg, Walsh, Zhang

• Random tree 1: b=2, d=100

51

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Complexity Analysis: Anomaly • random tree with edge costs uniformly chosen from {0,1,2,3,4} • A tree with a larger branching factor is more difficult to search than one with a smaller branching factor !?

Gomes, Hogg, Walsh, Zhang

52

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Complexity Analysis: Control Parameter • Control parameter: – b = the mean branching factor – p = probability that an edge has cost 0 (edge costs can be discrete or continuous) – bp = expected number of children having the same cost as the parent

• A local property determines a global behavior! C e1

e2

C+ e1 C+ e2 Gomes, Hogg, Walsh, Zhang

53

eb C+ eb IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Complexity Analysis: The Main Results ’)

• Optimal solution cost C* (as d bp1 1

C* bounded by a constant

• Expected complexity (# of nodes generated as d algorithm

bp1

Best-first search

( d) optimal

( d2 ) optimal

(d) optimal

Depth-first branch and bound

( d) asymptotic optimal

O( d 3 )

O( d 2 )

’)

Node costs can be discrete or continuous Gomes, Hogg, Walsh, Zhang

54

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Complexity Analysis: History • J. Pearl (1980-1984) Analysis of A* (BFS) • R. Karp & J. Pearl (1983) Complexity of BFS on incremental random binary tree with {0,1} edge costs • C.J.H. McDiarmid (1990) Complexity of BFS on incremental random tree • W. Zhang & R.E. Korf (1992-1995) Complexity of linear-space search algorithms (DFBnB, IDA*, RBFS) on incremental random tree

Gomes, Hogg, Walsh, Zhang

55

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Content • Expected complexity of optimization search • Phase transitions of optimization search – Phase Transitions and Phase diagram – Answers to anomaly – Phase transitions in the TSP

• Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions Gomes, Hogg, Walsh, Zhang

56

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase Transition and Phase Diagram • Control parameter bp = expected number of children having the same cost as the parent

• Complexity: exponential vs. polynomial

(Zhang&Korf, 1995)

Gomes, Hogg, Walsh, Zhang

57

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Search Anomaly of Sliding-tile Puzzle Revised • Control parameter bp = expected # of same-cost children

(Zhang&Korf, 1995)

• f=g+h • g=depth of a node, which increases by one for each move • h=Manhattan distance, which increases or decreases by one for each move with probability ½ approximately (a tile moved toward or away from its goal position)

• Edge costs in the search tree take values 0 or 2; p=prob(cost zero)§1/2. • b increases with puzzle size • bp increases with puzzle size Gomes, Hogg, Walsh, Zhang

58

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase Transitions in the ATSP • Question: Does the ATSP with distances from {1,2,…,r} have the same average-case complexity as r increases? (Zhang&Korf, 1996) • Anwser: No, due to phase transitions



Assignment Problem (AP) is the cost function for the ATSP, which can be computed in O(n^3) time.

Gomes, Hogg, Walsh, Zhang

59

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Content • Expected complexity of optimization search • Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions

Gomes, Hogg, Walsh, Zhang

60

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase Transitions – Decision vs. Optimization • Decision problem

• Optimization problem

– Finding an YES/NO answer – Easy-hard-easy phase transitions

Gomes, Hogg, Walsh, Zhang

– Finding an optimal solution – Easy-hard phase transitions

61

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase Transitions –

Decision versus Optimization (A Closer Look) • Different phase transition patterns – Decision problem has easy-hard-easy transition pattern – Optimization problem has easy-hard transition pattern

• Complexity discrepancy - Tighter constraints have different impact – They make a decision problem easier – They make an optimization problem more difficult

• Optimization is hard! (Zhang, 2001, see also Slaney&Walsh, 2001, later slides) Gomes, Hogg, Walsh, Zhang

62

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Decision vs. Optimization Quality/Complexity Tradeoff (Experiments) • New, shifted phase transitions in MAX-3SAT (25 variables) (zhang, 2001)

Gomes, Hogg, Walsh, Zhang

63

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Content • Expected complexity of optimization search • Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions

Gomes, Hogg, Walsh, Zhang

64

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Backbone: Behavior of Finding All Solutions Why finding an optimal solution is hard when a problem is overconstrained?

(zhang, 2001)

Gomes, Hogg, Walsh, Zhang

65

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Backbone: Behavior of Finding All Solutions • Number of solutions of MAX-3SAT (25 variables): – The rate that the number of solutions changes is different in overconstrained and underconstrained regions!

Gomes, Hogg, Walsh, Zhang

66

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Backbone: Behavior of Finding All Solutions • Number of clauses unsatisfied in and complexity of MAX-3SAT (25 variables): – They all have different patterns in the overconstrained and underconstrained regions!

Gomes, Hogg, Walsh, Zhang

67

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Backbone: Behavior of Finding All Solutions • Backbone: The set of variables that have fixed values in all solutions • There is a phase transition in backbone in MAX 3-SAT (25 variables) • Backbone and satisfiability is roughly linearly correlated.

Gomes, Hogg, Walsh, Zhang

68

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Content • Expected complexity of optimization search • Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions – Parametric transformations – Structural transformations

Gomes, Hogg, Walsh, Zhang

69

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions • What do we do with a difficult problem in the exponential region?

Gomes, Hogg, Walsh, Zhang

70

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions: Parametric Transformation – Basic Idea • Treat a difficult problem in the exponential region as if it is an easy one in the polynomial region (Zhang&Pemberton, 94, Pemberton&Zhang, 96)

Gomes, Hogg, Walsh, Zhang

71

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions: Parametric Transformation – The Method • Increasing the number of “zero-cost edges” in search space

Gomes, Hogg, Walsh, Zhang

72

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions: Parametric Transformation – Improvement • For a real-world problem, edge costs are not known in advance, but can be learned on the fly. • If a solution found is not good enough, another transformation is applied with a smaller epsilon value – iterative parametric transformation

Gomes, Hogg, Walsh, Zhang

73

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions: Parametric Transformation – Application

Gomes, Hogg, Walsh, Zhang

74

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions: Structural Transformation – Idea and Method • Prune large cost edges in search space (Zhang, 1998, 2001)

Gomes, Hogg, Walsh, Zhang

75

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exploiting Phase Transitions: Structural Transformation – Applications

Gomes, Hogg, Walsh, Zhang

76

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Take Home Message (Summary) • Finding optimal solution is hard than deciding solubility • Phase transitions can be used to characterize complex problems and their behavior • Understanding phase transitions can help to design and develop more efficient search algorithms

Gomes, Hogg, Walsh, Zhang

77

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Beyond NP Other complexity classes Phase transitions in P, PSPACE, …

Structure Backbones, 2+p-SAT, small world topology, …

Heuristics Constrainedness knife-edge, minimize constrainedness, ... Gomes, Hogg, Walsh, Zhang

78

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Before we begin A little history ...

Where did this all start? • At least as far back as 60s with Erdos & Renyi – thresholds in random graphs

• Late 80s – pioneering work by Karp, Purdom, Kirkpatrick, Huberman, Hogg …

• Flood gates burst – Cheeseman, Kanefsky & Taylor’s IJCAI-91 paper In 91, I has just finished my PhD and was looking for some new research topics!

Gomes, Hogg, Walsh, Zhang

80

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Other complexity classes Enough of the history, are phase transitions just in NP? Conjecture in Cheeseman et al paper that phase transitions distinguish P from NP.

Random 2-SAT • 2-SAT is P – linear time algorithm x1 v x2, -x2 v x3, -x1 v x3, …

• Random 2-SAT displays “classic” phase transition – l/n < 1, almost surely SAT – l/n > 1, almost surely UNSAT – complexity peaks around l/n=1 Gomes, Hogg, Walsh, Zhang

82

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase transitions in P • 2-SAT – l/n=1

• Horn SAT – transition not “sharp”

• Arc-consistency – rapid transition in whether problem can be made AC – peak in (median) checks Gomes, Hogg, Walsh, Zhang

83

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase transitions above NP • PSpace – QSAT (SAT of QBF)

Gomes, Hogg, Walsh, Zhang

84

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Phase transitions above NP • PSpace-complete – QSAT (SAT of QBF) – stochastic SAT – modal SAT

• PP-complete – polynomial-time probabilistic Turing machines – counting problems – #SAT(>= 2^n/2) [Bailey, Dalmau, Kolaitis IJCAI-2001] Gomes, Hogg, Walsh, Zhang

85

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Exact phase boundaries in NP • Random 3-SAT is only known within bounds – 3.26 < l/n < 4.596

Are there any NP phase boundaries known exactly?

• Recent result gives an exact NP phase boundary – 1-in-k SAT at l/n = 2/k(k-1) – 2nd order transition (like 2SAT and unlike 3-SAT)

Gomes, Hogg, Walsh, Zhang

1st order transitions not a characteristic of NP as has been conjectured

86

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Structure Can we identify structure in (random) problems that makes problems hard? How do we model structural features found in real problems? How does such structure affect phase transition behaviour?

Backbone • Variables which take fixed values in all solutions – alias unit prime implicates

• Let fk be fraction of variables in backbone – in random 3-SAT l/n < 4.3, fk vanishing (otherwise adding clause could make problem unsat) l/n > 4.3, fk > 0

discontinuity at phase boundary!

Gomes, Hogg, Walsh, Zhang

88

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Backbone • Search cost correlated with backbone size – if fk non-zero, then can easily assign variable “wrong” value – such mistakes costly if at top of search tree

• Backbones seen in other problems – coloring, TSP, blocks world planning … see [Slaney, Walsh IJCAI-2001]

Can we make algorithms that identify and exploit the backbone structure of a problem?

Gomes, Hogg, Walsh, Zhang

89

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

2+p-SAT • Morph between 2-SAT and 3SAT – fraction p of 3-clauses – fraction (1-p) of 2-clauses

• 2-SAT is polynomial (linear) – phase boundary at l/n =1 – but no backbone discontinuity here!

• 2+p-SAT maps from P to NP – p>0, 2+p-SAT is NP-complete

Gomes, Hogg, Walsh, Zhang

90

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

2+p-SAT phase transition

Gomes, Hogg, Walsh, Zhang

91

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

2+p-SAT phase transition

l/n

p Gomes, Hogg, Walsh, Zhang

92

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

2+p-SAT phase transition • Lower bound – are the 2-clauses (on their own) UNSAT? – n.b. 2-clauses are much more constraining than 3clauses

• p 0.4

6HDUFKFRVWDJDLQVWQ

– but NP-complete for p>0 !

• search cost shifts from linear to exponential at p=0.4 • similar behavior seen with local search algorithms

Gomes, Hogg, Walsh, Zhang

94

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Structure How do we model structural features found in real problems? How does such structure affect phase transition behaviour?

The real world isn’t random? • Very true! Can we identify structural features common in real world problems?

• Consider graphs met in real world situations – – – –

social networks electricity grids neural networks ...

Gomes, Hogg, Walsh, Zhang

96

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Real versus Random • Real graphs tend to be sparse – dense random graphs contains lots of (rare?) structure

• Real graphs tend to have short path lengths

L, average path length C, clustering coefficient (fraction of neighbours connected to each other, cliqueness measure)

– as do random graphs

• Real graphs tend to be clustered

mu, proximity ratio is C/L normalized by that of random graph of same size and density

– unlike sparse random graphs Gomes, Hogg, Walsh, Zhang

97

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Small world graphs • Sparse, clustered, short path lengths • Six degrees of separation – Stanley Milgram’s famous 1967 postal experiment – recently revived by Watts & Strogatz – shown applies to: • • • • Gomes, Hogg, Walsh, Zhang

98

actors database US electricity grid neural net of a worm ... IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

An example • 1994 exam timetable at Edinburgh University – 59 nodes, 594 edges so relatively sparse – but contains 10-clique

• less than 10^-10 chance in a random graph – assuming same size and density

• clique totally dominated cost to solve problem

Gomes, Hogg, Walsh, Zhang

99

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Small world graphs • To construct an ensemble of small world graphs – morph between regular graph (like ring lattice) and random graph – prob p include edge from ring lattice, 1-p from random graph real problems often contain similar structure and stochastic components? Gomes, Hogg, Walsh, Zhang

100

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Small world graphs

• ring lattice is clustered but has long paths • random edges provide shortcuts without destroying clustering Gomes, Hogg, Walsh, Zhang

101

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Small world graphs

Gomes, Hogg, Walsh, Zhang

102

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Small world graphs

Gomes, Hogg, Walsh, Zhang

103

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Colouring small world graphs

Gomes, Hogg, Walsh, Zhang

104

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Small world graphs • Other bad news – disease spreads more rapidly in a small world

• Good news – cooperation breaks out quicker in iterated Prisoner’s dilemma

Gomes, Hogg, Walsh, Zhang

105

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Other structural features It’s not just small world graphs that have been studied

• Large degree graphs – Barbasi et al’s power-law model [Walsh, IJCAI 2001]

• Ultrametric graphs – Hogg’s tree based model

• Numbers following Benford’s Law – 1 is much more common than 9 as a leading digit! prob(leading digit=i) = log(1+1/i)

– such clustering, makes number partitioning much easier Gomes, Hogg, Walsh, Zhang

106

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Another structured problem • Quasigroup completion problem (QCP) Can we complete a partial Latin square?

"

• Regular structure found in real problems – sports tournaments – fibre optic routing See [Kautz et al, IJCAI 2001]

Gomes, Hogg, Walsh, Zhang

107

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

QCP phase transition

Gomes, Hogg, Walsh, Zhang

108

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

QCP phase transition

Gomes, Hogg, Walsh, Zhang

109

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Heuristics What do we understand about problem hardness at the phase boundary? How can this help build better heuristics?

Looking inside search • Constrainedness “knife-edge” – problems are critically constrained between SAT and UNSAT

• Suggests branching heuristics – also insight into branching mistakes

Gomes, Hogg, Walsh, Zhang

111

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Inside SAT phase transition • Random 3-SAT, l/n =4.3 • Davis Putnam algorithm – tree search through space of partial assignments – unit propagation

• Clause to variable ratio l/n drops as we search => problems become less constrained Aside: can anyone explain simple scaling? Gomes, Hogg, Walsh, Zhang

OQDJDLQVWGHSWKQ 112

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Inside SAT phase transition • But (average) clause length, k also drops => problems become more constrained

• Which factor, l/n or k wins? – Look at kappa which includes both! Aside: why is there again such simple scaling? &ODXVHOHQJWKNDJDLQVWGHSWKQ Gomes, Hogg, Walsh, Zhang

113

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Constrainedness knife-edge NDSSDDJDLQVWGHSWKQ

Gomes, Hogg, Walsh, Zhang

114

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Constrainedness knife-edge • Seen in other problem domains – number partitioning, …

• Seen on “real” problems – exam timetabling (alias graph colouring)

• Suggests branching heuristic – “get off the knife-edge as quickly as possible” – minimize or maximize-kappa heuristics must take into account branching rate, max-kappa often therefore not a good move! Gomes, Hogg, Walsh, Zhang

115

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Minimize constrainedness • Many existing heuristics minimize-kappa – or proxies for it

• For instance – – – –

Karmarkar-Karp heuristic for number partitioning Brelaz heuristic for graph colouring Fail-first heuristic for constraint satisfaction …

• Can be used to design new heuristics – removing some of the “black art”

Gomes, Hogg, Walsh, Zhang

116

IJCAI-01 Tutorial on Phase Transitions and Structures-SA4

Suggest Documents