IJCAI-01 Tutorial on Phase. Transitions and ... e.g.: scheduling, circuit design, cryptography, theorem proving, games .... phase transitions in AI. ⢠pruning in ...
Phase Transitions and Structure in Combinatorial Problems Carla P. Gomes Tad Hogg Toby Walsh Weixiong Zhang
a statistical regularity in search • relates structure to behavior – e.g., problem features => search cost
• analogous to physical phase transitions • useful to improve search heuristics – i.e., new problem solving strategies
Gomes, Hogg, Walsh, Zhang
2
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
combinatorial search e.g.: scheduling, circuit design, cryptography, theorem proving, games, matching models to data, genetics, ...
• types – decision (NP): rapid test of candidate solutions – optimization
• behavior – worst case (e.g., is P=NP?) – typical performance of heuristics
Gomes, Hogg, Walsh, Zhang
3
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
10 variable 3-SAT example • each clause gives a “nogood” – inconsistent assignment for the 3 variables • e.g., v1=true, v4=false, v5=false
• search method: depth-1st backtrack – cost: steps to find a solution (if any)
• examine search tree as nogoods added – i.e., a particular sequence of clauses, randomly generated
Gomes, Hogg, Walsh, Zhang
4
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
search tree v1=true
v1=false
examined states
34 nogoods 1st solution Gomes, Hogg, Walsh, Zhang
another solution
5
other consistent states
many solutions little pruning IJCAI-01 lowTutorial coston Phase Transitions and Structures-SA4
search tree
55 nogoods Gomes, Hogg, Walsh, Zhang
6
one solution some pruning IJCAI-01 highTutorial coston Phase Transitions and Structures-SA4
search tree
110 nogoods Gomes, Hogg, Walsh, Zhang
7
no solutions much pruning IJCAI-01 lowTutorial coston Phase Transitions and Structures-SA4
search cost 140 120
cost
100
no solutions
80 60 40 20 0 0
25
underconstrained Gomes, Hogg, Walsh, Zhang
50
75 nogoods 8
100
125
150
overconstrained IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
observations • peak in cost vs. number of nogoods – competition between • decreasing number of solutions • increasing search path pruning
• peak corresponds to loss of last solution • peak location depends on choice of nogoods
Gomes, Hogg, Walsh, Zhang
9
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
generality • example uses – one search method – one sequence of problem instances
• Why is this interesting? – similar behavior for many • search problems • search techniques
– hence: simple parameter indicates likely cost
Gomes, Hogg, Walsh, Zhang
10
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
outline • statistical approach to search – local properties and global behavior
– problem ensembles • phase transitions • impact of structure • summary
Gomes, Hogg, Walsh, Zhang
11
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
statistical approach to search • How do many repeated local decisions combine to give global behavior? – e.g., heuristic choice accuracy => search cost
• key issue: – local properties => global behavior – addressed by statistical mechanics
Gomes, Hogg, Walsh, Zhang
12
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
statistical approach • identify problem ensemble • identify structure parameters – easy to measure (computationally efficient) – describe an average, local property of problem
• relate parameters to global behavior – e.g., search cost
Gomes, Hogg, Walsh, Zhang
13
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
problem ensembles • class of problem instances • probability for each instance • why? – focus on typical behaviors in class (averagecase) – not single instance, not worst-case
Gomes, Hogg, Walsh, Zhang
14
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
parameterized ensembles • problem size • properties of constraints – e.g., number, tightness
• can also specify global properties – e.g., number of solutions
Gomes, Hogg, Walsh, Zhang
15
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
simple ensemble examples • random k-SAT: n variables, m clauses – each clause picked independently • allowing duplicates
• random graphs: n nodes, e edges – each edge picked randomly • but no duplicates
Gomes, Hogg, Walsh, Zhang
16
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
“real” problems • constraints tend to be local and more clustered than random – e.g., small worlds graphs
• harder to work with than easily generated random ensembles – both analytically and empirically
Gomes, Hogg, Walsh, Zhang
17
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
nature of constraints • physical interactions – strength decreases with distance
• designed artifacts – often nearly decomposable, hierarchical
• evolved systems – e.g., ecology, economy, common law, social networks
Gomes, Hogg, Walsh, Zhang
18
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
local measures: global behavior • local measure of problem structure – number and tightness of constraints
• global behaviors – probability for a solution – search cost – solution quality
Gomes, Hogg, Walsh, Zhang
19
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
outline • statistical approach to search • phase transitions – decision problems – optimization – other complexity classes • impact of structure • summary
Gomes, Hogg, Walsh, Zhang
20
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
phase transitions • abrupt changes in global property of typical instance in a problem ensemble • characteristic of problem ensemble, not problem instance
Gomes, Hogg, Walsh, Zhang
21
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
phase transition phenomena • physics: – magnets, superconductors vs. temperature
• geology: – fluid percolation vs. porosity of rocks
• biology – epidemic spread vs. population density
Gomes, Hogg, Walsh, Zhang
22
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
phase transition: boiling • small change in average energy per molecule (temperature) • gives large change in overall state
Gomes, Hogg, Walsh, Zhang
23
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
commonality? • key idea: abrupt change in overall system property when size is large • why a useful general concept? – – – –
mathematical commonality in spite of different details suggests analyses, e.g., finite-size scaling universality: insight from simple models with “essential features”
Gomes, Hogg, Walsh, Zhang
24
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
phase transitions in AI • pruning in heuristic search • associative memory models • pattern matching
Gomes, Hogg, Walsh, Zhang
25
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
outline • statistical approach to search • phase transitions – decision problems – optimization – other complexity classes • impact of structure • summary
Gomes, Hogg, Walsh, Zhang
26
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
some decision problems • SAT – random 3-SAT: clause/variables near 4.2
• graph coloring – random graph, 3-COL: edges/nodes near 2.2
• number partitioning – L integers, random in [0,..,2L]
Gomes, Hogg, Walsh, Zhang
27
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
observe regularity • search cost varies greatly among problem instances of same size • hard instances rare and concentrated hard instance
Gomes, Hogg, Walsh, Zhang
28
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
intuition • few constraints: many solutions • many constraints: no solutions EASY
– bad choices pruned quickly
• intermediate number of constraints: – few solutions HARD – bad choices pruned only after many steps
Gomes, Hogg, Walsh, Zhang
29
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
typical cost of search heuristics hard cases: • near phase transitions • fastest exponential growth of cost
large size
behavior of many • NP problem ensembles • search methods (but not generate & test)
easy cases: • underconstrained: many solutions • overconstrained: rapid pruning typical cost
small size
underconstrained Gomes, Hogg, Walsh, Zhang
30
overconstrained
IJCAI-01 Tutorial on Phase number of constraints Transitions and Structures-SA4
typical behavior of solubility • for large problem ensembles – fraction of soluble instances drops suddenly from near 1 to near 0 – close to location of =1 =1
prob(soln)
1
few soluble problems but with very many solutions
0
exp. large
# constraints Gomes, Hogg, Walsh, Zhang
31
exp. small IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
solubility transition and cost • high costs associated with drop in solutions • not the whole story: cost peak for problems with fixed number of solutions !
Gomes, Hogg, Walsh, Zhang
32
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
theory • Why the association of high cost with transition, on average? • simplified models of the behavior: – exact: simple tree search • trivial problem • simple illustration of abrupt change
– approximate: graph coloring
Gomes, Hogg, Walsh, Zhang
33
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
tree search • branching ratio b, depth d • no solution – e.g., prior choices are already inconsistent
• heuristic prunes with prob. p – average branching ratio: z = b p
• expected cost for depth-1st backtrack
Gomes, Hogg, Walsh, Zhang
34
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
expected cost d=10
d=4
cost
d=20
constant
exponential
z
Gomes, Hogg, Walsh, Zhang
35
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
expected cost • average branching ratio: z = b p • avg. cost from depth j: Cj=1+z Cj+1 • avg. cost from root
1 z 1 has abrupt changes when d infinite z − 1 36 IJCAI-01 Tutorial on Phase Gomes, Hogg, Walsh, Zhang Transitions and Structures-SA4
hard graph coloring • many consistent colorings for much of the graph • few for the whole graph • hence much backtracking
Gomes, Hogg, Walsh, Zhang
37
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
relate cost to structure • color k of n nodes, with 3 colors • 3k possible colorings – most with about equal use of each color
• random edge in subgraph with prob. 2 • edge in subgraph links same colors with ≈ (k/n) prob. 1/3
Gomes, Hogg, Walsh, Zhang
38
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
partial colorings • assuming independent constraints, expected number of partial colorings of k nodes is
1 k k Nk = 3 1− 3 n Gomes, Hogg, Walsh, Zhang
39
2
e
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
behavior • few edges: monotonic increase, little backtrack • more edges: hard to continue beyond maximum, much backtrack • many edges: no solutions, pruning terminates search early
Gomes, Hogg, Walsh, Zhang
40
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
ln(# consistent colorings)
behavior: 100 nodes e=75
e=200
e=300
# colored nodes edges prune large subgraphs more than small ones Gomes, Hogg, Walsh, Zhang
41
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
independence assumption • • • •
simplifies analysis gives qualitative behavior reasonable quantitative accuracy cf., “mean-field” physics theories
Gomes, Hogg, Walsh, Zhang
42
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
which ensemble? • examples – random graphs – small-world graphs
• cf., physical systems – all states consistent with (few) parameters are equally likely – accuracy is an empirical question – e.g., classical => quantum statistical mechanics
Gomes, Hogg, Walsh, Zhang
43
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
open questions • math: prove thresholds in simple ensembles • empirical: which ensembles are realistic? – perhaps additional parameters required – e.g., random graph vs. small worlds graph
Gomes, Hogg, Walsh, Zhang
44
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Computational Complexity and Phase Transitions of Optimization Problems
Gomes, Hogg, Walsh, Zhang
45
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Combinatorial Optimization: Example 1 •
The Traveling Salesman Problem (TSP) – Given: a set of cities and inter-city distances – Goal: a shortest tour visit each city once – Asymmetric TSP (ATSP): distance(i,j) distance(j,i) – Many real-world applications • VLSI routing • Scheduling
– A well-studied NP-hard problem – TSP with distances 1 and 2 is still NP-hard! (Papadimitriou&Yannakakis 93) – NP-hardness is a worst-case measure – How does the average-case complexity behave?
Gomes, Hogg, Walsh, Zhang
46
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Combinatorial Optimization: Example 2 • Interleaving decision and execution: Decision making under limited information and restricted resources
Which path to take? decision
– Collecting information with restricted computational resources • Exploration by lookahead search
– Reasoning based on limited information • Calculation
Lookahead to collect information
– Executing the best decision • Action
• This idea is the foundation of almost all game playing programs (Shannon 1950) Gomes, Hogg, Walsh, Zhang
47
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Combinatorial Optimization: Example 2 •
Lookahead search in sliding-tile puzzle: • backup the minimal cost at a fixed depth
•
f=g+h: g=depth of a node, h= Manhattan distance to the goal. In one step, g increases by one; h increases or decreases by one; and so f increases by 2 or remains the same. 1 2 3 4 5 6 7 8 1 2 34 5 67 8 1 2 3 45 6 78
1 4 2 3 5 6 7 8
Gomes, Hogg, Walsh, Zhang
Anomaly of sliding-tile puzzles: For a given search depth, a large puzzle is easier to search; or for a given amount of computation, a large puzzle can be searched to a deeper level.
1 2 3 8 4 7 6 5 3 1 2 4 5 6 7 8
3 1 2 4 5 6 7 8
3 1 2 6 4 5 7 8
48
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Content • Expected complexity of optimization search – Abstract (statistical) tree model – Complexity results
• Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions Gomes, Hogg, Walsh, Zhang
49
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Abstract Model for Complexity Analysis • Incremental random tree – Node = states
– Edges = state transitions
– Tree depth d
7
– Edge costs (operator costs): i.id non-negative random variable (may take 0) – Node costs (state quality): sum of edge costs from the root to the nodes 50
2
– Branching factor b= # of children
Gomes, Hogg, Walsh, Zhang
0
1
2
5
2
3
4
5
6
8
3
7
4
Goal state
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Complexity Analysis: A Quiz • • •
Given two random trees generated with the same edge cost distribution Finding the goal nodes (minimum-cost leaf nodes) using the same search algorithm (BFS, DFBnB, IDA*, …) Which tree is easier to search?
• Random tree 1: b=3, d=100
Gomes, Hogg, Walsh, Zhang
• Random tree 1: b=2, d=100
51
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Complexity Analysis: Anomaly • random tree with edge costs uniformly chosen from {0,1,2,3,4} • A tree with a larger branching factor is more difficult to search than one with a smaller branching factor !?
Gomes, Hogg, Walsh, Zhang
52
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Complexity Analysis: Control Parameter • Control parameter: – b = the mean branching factor – p = probability that an edge has cost 0 (edge costs can be discrete or continuous) – bp = expected number of children having the same cost as the parent
• A local property determines a global behavior! C e1
e2
C+ e1 C+ e2 Gomes, Hogg, Walsh, Zhang
53
eb C+ eb IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Complexity Analysis: The Main Results )
• Optimal solution cost C* (as d bp1 1
C* bounded by a constant
• Expected complexity (# of nodes generated as d algorithm
bp1
Best-first search
( d) optimal
( d2 ) optimal
(d) optimal
Depth-first branch and bound
( d) asymptotic optimal
O( d 3 )
O( d 2 )
)
Node costs can be discrete or continuous Gomes, Hogg, Walsh, Zhang
54
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Complexity Analysis: History • J. Pearl (1980-1984) Analysis of A* (BFS) • R. Karp & J. Pearl (1983) Complexity of BFS on incremental random binary tree with {0,1} edge costs • C.J.H. McDiarmid (1990) Complexity of BFS on incremental random tree • W. Zhang & R.E. Korf (1992-1995) Complexity of linear-space search algorithms (DFBnB, IDA*, RBFS) on incremental random tree
Gomes, Hogg, Walsh, Zhang
55
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Content • Expected complexity of optimization search • Phase transitions of optimization search – Phase Transitions and Phase diagram – Answers to anomaly – Phase transitions in the TSP
• Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions Gomes, Hogg, Walsh, Zhang
56
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase Transition and Phase Diagram • Control parameter bp = expected number of children having the same cost as the parent
• Complexity: exponential vs. polynomial
(Zhang&Korf, 1995)
Gomes, Hogg, Walsh, Zhang
57
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Search Anomaly of Sliding-tile Puzzle Revised • Control parameter bp = expected # of same-cost children
(Zhang&Korf, 1995)
• f=g+h • g=depth of a node, which increases by one for each move • h=Manhattan distance, which increases or decreases by one for each move with probability ½ approximately (a tile moved toward or away from its goal position)
• Edge costs in the search tree take values 0 or 2; p=prob(cost zero)§1/2. • b increases with puzzle size • bp increases with puzzle size Gomes, Hogg, Walsh, Zhang
58
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase Transitions in the ATSP • Question: Does the ATSP with distances from {1,2,…,r} have the same average-case complexity as r increases? (Zhang&Korf, 1996) • Anwser: No, due to phase transitions
•
Assignment Problem (AP) is the cost function for the ATSP, which can be computed in O(n^3) time.
Gomes, Hogg, Walsh, Zhang
59
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Content • Expected complexity of optimization search • Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions
Gomes, Hogg, Walsh, Zhang
60
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase Transitions – Decision vs. Optimization • Decision problem
• Optimization problem
– Finding an YES/NO answer – Easy-hard-easy phase transitions
Gomes, Hogg, Walsh, Zhang
– Finding an optimal solution – Easy-hard phase transitions
61
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase Transitions –
Decision versus Optimization (A Closer Look) • Different phase transition patterns – Decision problem has easy-hard-easy transition pattern – Optimization problem has easy-hard transition pattern
• Complexity discrepancy - Tighter constraints have different impact – They make a decision problem easier – They make an optimization problem more difficult
• Optimization is hard! (Zhang, 2001, see also Slaney&Walsh, 2001, later slides) Gomes, Hogg, Walsh, Zhang
62
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Decision vs. Optimization Quality/Complexity Tradeoff (Experiments) • New, shifted phase transitions in MAX-3SAT (25 variables) (zhang, 2001)
Gomes, Hogg, Walsh, Zhang
63
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Content • Expected complexity of optimization search • Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions
Gomes, Hogg, Walsh, Zhang
64
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Backbone: Behavior of Finding All Solutions Why finding an optimal solution is hard when a problem is overconstrained?
(zhang, 2001)
Gomes, Hogg, Walsh, Zhang
65
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Backbone: Behavior of Finding All Solutions • Number of solutions of MAX-3SAT (25 variables): – The rate that the number of solutions changes is different in overconstrained and underconstrained regions!
Gomes, Hogg, Walsh, Zhang
66
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Backbone: Behavior of Finding All Solutions • Number of clauses unsatisfied in and complexity of MAX-3SAT (25 variables): – They all have different patterns in the overconstrained and underconstrained regions!
Gomes, Hogg, Walsh, Zhang
67
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Backbone: Behavior of Finding All Solutions • Backbone: The set of variables that have fixed values in all solutions • There is a phase transition in backbone in MAX 3-SAT (25 variables) • Backbone and satisfiability is roughly linearly correlated.
Gomes, Hogg, Walsh, Zhang
68
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Content • Expected complexity of optimization search • Phase transitions of optimization search • Relationship between phase transitions in decision and optimization problems • Backbone phase transitions • New search strategies by exploiting phase transitions – Parametric transformations – Structural transformations
Gomes, Hogg, Walsh, Zhang
69
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions • What do we do with a difficult problem in the exponential region?
Gomes, Hogg, Walsh, Zhang
70
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions: Parametric Transformation – Basic Idea • Treat a difficult problem in the exponential region as if it is an easy one in the polynomial region (Zhang&Pemberton, 94, Pemberton&Zhang, 96)
Gomes, Hogg, Walsh, Zhang
71
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions: Parametric Transformation – The Method • Increasing the number of “zero-cost edges” in search space
Gomes, Hogg, Walsh, Zhang
72
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions: Parametric Transformation – Improvement • For a real-world problem, edge costs are not known in advance, but can be learned on the fly. • If a solution found is not good enough, another transformation is applied with a smaller epsilon value – iterative parametric transformation
Gomes, Hogg, Walsh, Zhang
73
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions: Parametric Transformation – Application
Gomes, Hogg, Walsh, Zhang
74
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions: Structural Transformation – Idea and Method • Prune large cost edges in search space (Zhang, 1998, 2001)
Gomes, Hogg, Walsh, Zhang
75
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exploiting Phase Transitions: Structural Transformation – Applications
Gomes, Hogg, Walsh, Zhang
76
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Take Home Message (Summary) • Finding optimal solution is hard than deciding solubility • Phase transitions can be used to characterize complex problems and their behavior • Understanding phase transitions can help to design and develop more efficient search algorithms
Gomes, Hogg, Walsh, Zhang
77
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Beyond NP Other complexity classes Phase transitions in P, PSPACE, …
Structure Backbones, 2+p-SAT, small world topology, …
Heuristics Constrainedness knife-edge, minimize constrainedness, ... Gomes, Hogg, Walsh, Zhang
78
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Before we begin A little history ...
Where did this all start? • At least as far back as 60s with Erdos & Renyi – thresholds in random graphs
• Late 80s – pioneering work by Karp, Purdom, Kirkpatrick, Huberman, Hogg …
• Flood gates burst – Cheeseman, Kanefsky & Taylor’s IJCAI-91 paper In 91, I has just finished my PhD and was looking for some new research topics!
Gomes, Hogg, Walsh, Zhang
80
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Other complexity classes Enough of the history, are phase transitions just in NP? Conjecture in Cheeseman et al paper that phase transitions distinguish P from NP.
Random 2-SAT • 2-SAT is P – linear time algorithm x1 v x2, -x2 v x3, -x1 v x3, …
• Random 2-SAT displays “classic” phase transition – l/n < 1, almost surely SAT – l/n > 1, almost surely UNSAT – complexity peaks around l/n=1 Gomes, Hogg, Walsh, Zhang
82
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase transitions in P • 2-SAT – l/n=1
• Horn SAT – transition not “sharp”
• Arc-consistency – rapid transition in whether problem can be made AC – peak in (median) checks Gomes, Hogg, Walsh, Zhang
83
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase transitions above NP • PSpace – QSAT (SAT of QBF)
Gomes, Hogg, Walsh, Zhang
84
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Phase transitions above NP • PSpace-complete – QSAT (SAT of QBF) – stochastic SAT – modal SAT
• PP-complete – polynomial-time probabilistic Turing machines – counting problems – #SAT(>= 2^n/2) [Bailey, Dalmau, Kolaitis IJCAI-2001] Gomes, Hogg, Walsh, Zhang
85
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Exact phase boundaries in NP • Random 3-SAT is only known within bounds – 3.26 < l/n < 4.596
Are there any NP phase boundaries known exactly?
• Recent result gives an exact NP phase boundary – 1-in-k SAT at l/n = 2/k(k-1) – 2nd order transition (like 2SAT and unlike 3-SAT)
Gomes, Hogg, Walsh, Zhang
1st order transitions not a characteristic of NP as has been conjectured
86
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Structure Can we identify structure in (random) problems that makes problems hard? How do we model structural features found in real problems? How does such structure affect phase transition behaviour?
Backbone • Variables which take fixed values in all solutions – alias unit prime implicates
• Let fk be fraction of variables in backbone – in random 3-SAT l/n < 4.3, fk vanishing (otherwise adding clause could make problem unsat) l/n > 4.3, fk > 0
discontinuity at phase boundary!
Gomes, Hogg, Walsh, Zhang
88
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Backbone • Search cost correlated with backbone size – if fk non-zero, then can easily assign variable “wrong” value – such mistakes costly if at top of search tree
• Backbones seen in other problems – coloring, TSP, blocks world planning … see [Slaney, Walsh IJCAI-2001]
Can we make algorithms that identify and exploit the backbone structure of a problem?
Gomes, Hogg, Walsh, Zhang
89
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
2+p-SAT • Morph between 2-SAT and 3SAT – fraction p of 3-clauses – fraction (1-p) of 2-clauses
• 2-SAT is polynomial (linear) – phase boundary at l/n =1 – but no backbone discontinuity here!
• 2+p-SAT maps from P to NP – p>0, 2+p-SAT is NP-complete
Gomes, Hogg, Walsh, Zhang
90
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
2+p-SAT phase transition
Gomes, Hogg, Walsh, Zhang
91
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
2+p-SAT phase transition
l/n
p Gomes, Hogg, Walsh, Zhang
92
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
2+p-SAT phase transition • Lower bound – are the 2-clauses (on their own) UNSAT? – n.b. 2-clauses are much more constraining than 3clauses
• p 0.4
6HDUFKFRVWDJDLQVWQ
– but NP-complete for p>0 !
• search cost shifts from linear to exponential at p=0.4 • similar behavior seen with local search algorithms
Gomes, Hogg, Walsh, Zhang
94
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Structure How do we model structural features found in real problems? How does such structure affect phase transition behaviour?
The real world isn’t random? • Very true! Can we identify structural features common in real world problems?
• Consider graphs met in real world situations – – – –
social networks electricity grids neural networks ...
Gomes, Hogg, Walsh, Zhang
96
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Real versus Random • Real graphs tend to be sparse – dense random graphs contains lots of (rare?) structure
• Real graphs tend to have short path lengths
L, average path length C, clustering coefficient (fraction of neighbours connected to each other, cliqueness measure)
– as do random graphs
• Real graphs tend to be clustered
mu, proximity ratio is C/L normalized by that of random graph of same size and density
– unlike sparse random graphs Gomes, Hogg, Walsh, Zhang
97
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Small world graphs • Sparse, clustered, short path lengths • Six degrees of separation – Stanley Milgram’s famous 1967 postal experiment – recently revived by Watts & Strogatz – shown applies to: • • • • Gomes, Hogg, Walsh, Zhang
98
actors database US electricity grid neural net of a worm ... IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
An example • 1994 exam timetable at Edinburgh University – 59 nodes, 594 edges so relatively sparse – but contains 10-clique
• less than 10^-10 chance in a random graph – assuming same size and density
• clique totally dominated cost to solve problem
Gomes, Hogg, Walsh, Zhang
99
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Small world graphs • To construct an ensemble of small world graphs – morph between regular graph (like ring lattice) and random graph – prob p include edge from ring lattice, 1-p from random graph real problems often contain similar structure and stochastic components? Gomes, Hogg, Walsh, Zhang
100
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Small world graphs
• ring lattice is clustered but has long paths • random edges provide shortcuts without destroying clustering Gomes, Hogg, Walsh, Zhang
101
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Small world graphs
Gomes, Hogg, Walsh, Zhang
102
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Small world graphs
Gomes, Hogg, Walsh, Zhang
103
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Colouring small world graphs
Gomes, Hogg, Walsh, Zhang
104
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Small world graphs • Other bad news – disease spreads more rapidly in a small world
• Good news – cooperation breaks out quicker in iterated Prisoner’s dilemma
Gomes, Hogg, Walsh, Zhang
105
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Other structural features It’s not just small world graphs that have been studied
• Large degree graphs – Barbasi et al’s power-law model [Walsh, IJCAI 2001]
• Ultrametric graphs – Hogg’s tree based model
• Numbers following Benford’s Law – 1 is much more common than 9 as a leading digit! prob(leading digit=i) = log(1+1/i)
– such clustering, makes number partitioning much easier Gomes, Hogg, Walsh, Zhang
106
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Another structured problem • Quasigroup completion problem (QCP) Can we complete a partial Latin square?
"
• Regular structure found in real problems – sports tournaments – fibre optic routing See [Kautz et al, IJCAI 2001]
Gomes, Hogg, Walsh, Zhang
107
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
QCP phase transition
Gomes, Hogg, Walsh, Zhang
108
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
QCP phase transition
Gomes, Hogg, Walsh, Zhang
109
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Heuristics What do we understand about problem hardness at the phase boundary? How can this help build better heuristics?
Looking inside search • Constrainedness “knife-edge” – problems are critically constrained between SAT and UNSAT
• Suggests branching heuristics – also insight into branching mistakes
Gomes, Hogg, Walsh, Zhang
111
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Inside SAT phase transition • Random 3-SAT, l/n =4.3 • Davis Putnam algorithm – tree search through space of partial assignments – unit propagation
• Clause to variable ratio l/n drops as we search => problems become less constrained Aside: can anyone explain simple scaling? Gomes, Hogg, Walsh, Zhang
OQDJDLQVWGHSWKQ 112
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Inside SAT phase transition • But (average) clause length, k also drops => problems become more constrained
• Which factor, l/n or k wins? – Look at kappa which includes both! Aside: why is there again such simple scaling? &ODXVHOHQJWKNDJDLQVWGHSWKQ Gomes, Hogg, Walsh, Zhang
113
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Constrainedness knife-edge NDSSDDJDLQVWGHSWKQ
Gomes, Hogg, Walsh, Zhang
114
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Constrainedness knife-edge • Seen in other problem domains – number partitioning, …
• Seen on “real” problems – exam timetabling (alias graph colouring)
• Suggests branching heuristic – “get off the knife-edge as quickly as possible” – minimize or maximize-kappa heuristics must take into account branching rate, max-kappa often therefore not a good move! Gomes, Hogg, Walsh, Zhang
115
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4
Minimize constrainedness • Many existing heuristics minimize-kappa – or proxies for it
• For instance – – – –
Karmarkar-Karp heuristic for number partitioning Brelaz heuristic for graph colouring Fail-first heuristic for constraint satisfaction …
• Can be used to design new heuristics – removing some of the “black art”
Gomes, Hogg, Walsh, Zhang
116
IJCAI-01 Tutorial on Phase Transitions and Structures-SA4