Algorithm - Computer Science - University College Cork

CS 4407 Algorithms Lecture 1: Introduction Prof. Gregory Provan Department of Computer Science University College Cork Comp 122

1

Course Information Tuesday 12-1, Thursday 11-12 http://www.cs.ucc.ie/~gprovan/CS4407/2013/CS4407_Algorithms-2013.html

Gregory Provan WGB 1-71 [email protected] , 420-5928 Office Hours: Wednesday, 11-1 (or by appointment)

CS 4407, Algorithms University College Cork, Gregory M. Provan

Course Objectives Flon's Law: There is not now, and never will be, a language in which it is the least bit difficult to write bad programs. Objective: Be able to write good code for solving real-world problems • Correct and efficient • Requires minimal debugging CS 4407, Algorithms University College Cork, Gregory M. Provan

Textbook & References 

Introduction to Algorithms, 3rd Ed. by Cormen, Leiserson, Rivest & Stein, MIT Press, 2001 (noted as CLRS)

ALTERNATIVE REFERENCES - 

    

The Design and Analysis of Algorithms, by Anany Levitin The Design and Analysis of Computer Algorithms, by Aho, Hopcroft and Ullman Algorithms, by Sedgewick Fundamentals of Algorithms, by Brassard & Bratley Writing Efficient Programs, by Bentley The Science of Programming, by Gries An Introduction to Bioinformatics Algorithms, by Jones and Pevzner

CS 4407, Algorithms University College Cork, Gregory M. Provan

Free Online References 

Wiki Book – http://en.wikibooks.org/wiki/Algorithms



Algorithms: S. Dasgupta, C. H. Papadimitriou, and U. V. Vazirani – http://beust.com/algorithms.pdf



Algorithm Design Manual: Skiena – http://sist.sysu.edu.cn/~isslxm/DSA/textbook/Ski ena.-.TheAlgorithmDesignManual.pdf

CS 4407, Algorithms University College Cork, Gregory M. Provan

Lecture Outline Introduction to Algorithms Objectives Logistics How do we approach the study of algorithms? Overview of course content Definition of algorithm Example

CS 4407, Algorithms University College Cork, Gregory M. Provan

Today’s Learning Objectives 

Overview: Topics covered in the course – Recurrence relations – Graph algorithms – MapReduce and graph algorithms – Mathematical principles of complexity – Approximation algorithms



Definition of algorithm

CS 4407, Algorithms University College Cork, Gregory M. Provan

The future belongs to the computer scientist who has

– Content: An up to date grasp of fundamental problems and solutions – Method: Principles and techniques to solve the vast array of unfamiliar problems that arise in a rapidly changing field

CS 4407, Algorithms University College Cork, Gregory M. Provan

Course Content A survey of algorithmic design techniques.  Abstract thinking.  How to develop new algorithms for any problem that may arise. 

CS 4407, Algorithms University College Cork, Gregory M. Provan

Solving a Computational Problem 

Problem definition & specification – specify input, output and constraints



Algorithm design & analysis – devise a correct & efficient algorithm

Implementation planning  Coding, testing and verification 

CS 4407, Algorithms University College Cork, Gregory M. Provan

Why Analyze Algorithms? An algorithm can be analyzed in terms of time efficiency or space utilization. We will consider only the former right now. The running time of an algorithm is influenced by several factors:  

  

Speed of the machine running the program Language in which the program was written. For example, programs written in assembly language generally run faster than those written in C or C++, which in turn tend to run faster than those written in Java. Efficiency of the compiler that created the program The size of the input: processing 1000 records will take more time than processing 10 records. Organization of the input: if the item we are searching for is at the top of the list, it will take less time to find it than if it is at the bottom.

CS 4407, Algorithms University College Cork, Gregory M. Provan

Generalizing Running Time Comparing the growth of the running time as the input grows to the growth of known functions.

Input Size: n

(1)

log n

n

n log n

n²

n³

2ⁿ

5

1

3

5

15

25

125

32

10

1

4

10

33

100

10³

10³

100

1

7

100

664

104

106

1030

1000

1

10

1000

104

106

109

10300

10000

1

13

10000

105

108

1012

103000

CS 4407, Algorithms University College Cork, Gregory M. Provan

General Approach    

  

Examine interesting problems Devise algorithms for solving them Prove their correctness Analyze their runtime performance Study data structures & core algorithms Learn problem-solving techniques Applications to real-world problems – Bioinformatics, computer networking, scheduling, etc.

CS 4407, Algorithms University College Cork, Gregory M. Provan

Goals  

   

Be very familiar with a collection of core algorithms Be fluent in algorithm design paradigms: divide & conquer, greedy algorithms, randomization, dynamic programming, approximation methods Be able to analyze the correctness and runtime performance of a given algorithm Be familiar with the inherent complexity (lower bounds & intractability) of some problems Be intimately familiar with basic data structures Be able to apply techniques in practical problems CS 4407, Algorithms University College Cork, Gregory M. Provan

Applications 

Traditional applications – Computer networking, scheduling, diagnostics, constraint-based inference



Recent “hot” application – Bioinformatics and Computational Biology, web algorithms (Google)

CS 4407, Algorithms University College Cork, Gregory M. Provan

Course Overview Introduction to algorithm design, analysis and their applications      

Algorithm Fundamentals (4 lectures) Graph Algorithms (4 lectures) Greedy Algorithms (3 lectures) MapReduce/Graph Algorithms (2 lectures) Complexity Measures (5 lectures) Approximation Algorithms (2 lectures)

CS 4407, Algorithms University College Cork, Gregory M. Provan

Sub-Topic Algorithm fundamentals

Details Introduction & motivation Growth functions

14 January 2014 Algorithm Analysis 16 January 2014 Fundamentals

Master Theorem

Iterative (non-recursive) algorithms Recursive algorithms

5

21 January 2014

3

6 7

23 January 2014 28 January 2014

Overview Basic graph searching algorithms Shortest Path

4

8

30 January 2014 Graph algorithms

Max-Flow

BFS & DFS Bellman-Ford, Dijkstra algorithms Ford-Fulkerson, Karp-Edmonds algorithms

Week

Lecture

Main Topic

Date

1

1 2

07 January 2014 09 January 2014

2

3 4

9

Review of graph theory and graph algorithms

04 February 2014 QUIZ 1

5

10

06 February 2014

Minimum Spanning Tree

Greedy Approach; Kruskal algorithms

6

11 12

11 February 2014 13 February 2014 Greedy Algorithms 18 February 2014 QUIZ 2

Other greedy applications Packing problems

Knapsack and Scheduling

7

13

20 February 2014

MapReduce

14

MapReduce and Graph 25 February 2014 Algorithms

Graph Algorithms

Introduction to MapReduce Using MapReduce for very large Graph Problems

Complexity Classes

Complexity measures; algorithm classes P and NP

8

9 10

11 12

15

27 February 2014

16 17 18 19 20 21

04 March 2014 06 March 2014 11 March 2014 13 March 2014 18 March 2014 20 March 2014 25 March 2014 27 March 2014

NP-complete Problems Approximation Algorithms QUIZ 3 Review

NP-complete reductions

CS 4407, Algorithms University College Cork, Gregory M. Provan

Prim and

Theory of NP-complete algorithms SAT, Vertex Cover, Clique Independent Set, 3SAT Problem reduction methodology Vertex Cover; TSP Other approximation methods

Algorithm Fundamentals 

Asymptotic Notation



Recurrence Relations



Theory of Growth Functions



Proof Techniques



Inherent Complexity

CS 4407, Algorithms University College Cork, Gregory M. Provan

Algorithmics Basics (4) Introduction to algorithms, complexity and proof of correctness (CLRS Chapters 1 & 2)  Asymptotic Notation (CLRS Chapter 3.1) 

GOALS: – Know how to write a problem specification, what a computational model is, and how to measure the efficiency of an algorithm. – Know the difference between upper and lower bounds for an algorithm and what they convey. – Be able to prove the correctness of an algorithm and establish computational complexity. CS 4407, Algorithms University College Cork, Gregory M. Provan

Graph Algorithms (4) Graph theory  Basic algorithms 

– Breadth-first, depth-first search

Minimum spanning tree  Greedy Algorithms  Shortest path 

CS 4407, Algorithms University College Cork, Gregory M. Provan

Graph Algorithms (4) 

Basic Graph Algorithms (Chapter 22)

GOALS: – Know how graphs arise, their definition and implications. – Be able to use the adjacency matrix representation of a graph and the edge list representation appropriately. – Understand and be able to use algorithms for simple graph problems.

CS 4407, Algorithms University College Cork, Gregory M. Provan

More Graph Algorithms (Greedy, Network Flows) (4) 

Greedy – Minimum Spanning Trees (Chapter 23)

Shortest Paths (Chapter 24)  Max-Flows (Chapter 26) 

GOALS: – Know when to use greedy algorithms and their essential characteristics. Be able to prove the correctness of a greedy algorithm in solving an optimization problem. – Understand where minimum spanning trees and shortest path computations arise in practice. Know the basic approaches for computing max-flow in a graph. CS 4407, Algorithms University College Cork, Gregory M. Provan

Complexity Measures 

Algorithm classes P and NP



Theory of NP-completeness



NP-complete reductions



Methods for circumventing intractability

CS 4407, Algorithms University College Cork, Gregory M. Provan

Complexity and NP-Completeness (5) Introduction to inherent complexity of particular computational tasks (CLRS Chapter 34.1)  Complexity Classes P and NP (CLRS Chapter 34.2)  NP-Complete Reductions (CLRS Chapter 34.3-5) 

GOALS: – Know how to specify the inherent complexity of a task, and the differences between the classes P and NP. Know the notion of NP-Complete and its significance. – Be able to reduce decision tasks to establish inclusion in the class NP-Complete of the computational complexity hierarchy.

CS 4407, Algorithms University College Cork, Gregory M. Provan

Prerequisites 

Basic discrete mathematics – Counting arguments – Set notation – Induction

Standard data structures  Graph theory -- will provide review 

CS 4407, Algorithms University College Cork, Gregory M. Provan

Special Topics (1-2) Case Studies of Real-World Problems  Selected from 

– MapReduce (Google Distribute Inference Algorithm) – Computer Networking – BioInformatics

GOAL: See how core algorithms can be put to use in real-world applications.

CS 4407, Algorithms University College Cork, Gregory M. Provan

Course Work & Assessment 

In-class tests (3): 30%



Final Exam: 70%



Homework will be assigned – Not assigned credit – Solutions posted – Solutions of key problems will be covered in class

CS 4407, Algorithms University College Cork, Gregory M. Provan

Examinations 

In-class tests: 4 February, 18 February, 25 March – 10% each



Final: TBA – 70%

CS 4407, Algorithms University College Cork, Gregory M. Provan

Communication 

All lecture notes and most handouts will be posted at the course website: http://www.cs.ucc.ie/~gprovan/CS4407/2014



Major messages will be sent by email

CS 4407, Algorithms University College Cork, Gregory M. Provan

Algorithm Definition 

A tool for solving a well-specified computational problem

Input 

Algorithm

Output

Example: sorting – input: A sequence of numbers – output: An ordered permutation of the input – issues: correctness, efficiency, storage, etc.

CS 4407, Algorithms University College Cork, Gregory M. Provan

Strengthening the Informal Definition An algorithm is a finite sequence of unambiguous instructions for solving a wellspecified computational problem.  Important Features: 

– – – – –

Finiteness. Definiteness. Input. Output. Effectiveness. CS 4407, Algorithms University College Cork, Gregory M. Provan

Algorithm – In Formal Terms… In terms of mathematical models of computational platforms (general-purpose computers).  One definition – Turing Machine that always halts.  Other definitions are possible. 

– Example: Based on Lambda Calculus. 

Mathematical basis is necessary to answer questions such as: – Is a problem solvable? (Does an algorithm exist?) – Complexity classes of problems. (Is an efficient algorithm possible?) CS 4407, Algorithms University College Cork, Gregory M. Provan

Analyzing Algorithms 

Assumptions – Generic-one processor, random access machine – running time (others: memory, communication, etc)



Worst Case Running Time: the longest time for any input of size n – upper bound on the running time for any input – in some cases like searching, this is close



Average Case Behavior: the expected performance averaged over all possible inputs – it is generally better than worst case behavior – sometimes it’s roughly as bad as worst case CS 4407, Algorithms University College Cork, Gregory M. Provan

Example: Complex Numbers Remember how to multiply 2 complex numbers?

(a+bi)(c+di) = [ac –bd] + [ad + bc] i Input: a,b,c,d

Output: ac-bd, ad+bc

If a real multiplication costs €1 and an addition costs

a penny, what is the cheapest way to obtain the output from the input? Can you do better than €4.02? CS 4407, Algorithms University College Cork, Gregory M. Provan

Gauss’ €3.05 Method: Input: a,b,c,d Output: ac-bd, ad+bc  m1 = ac  m2 = bd  A1 = m1 – m2 = ac-bd

 m3 = (a+b)(c+d) = ac + ad + bc + bd  A2 = m3 – m1 – m2 = ad+bc CS 4407, Algorithms University College Cork, Gregory M. Provan

Question: The Gauss “hack” saves one multiplication

out of four. It requires 25% less work. Could there be a context where

performing 3 multiplications for every 4 provides a more dramatic savings?

CS 4407, Algorithms University College Cork, Gregory M. Provan

Multiplication Algorithms “Kindergarten”

n2n

“Grade School”

n2

Karatsuba

n1.58…

Fastest Known

n logn loglogn

CS 4407, Algorithms University College Cork, Gregory M. Provan

Summary 

We approach the study of algorithms in terms of underlying principles – Provides greatest long-term benefits



Algorithm definition – An algorithm is a finite sequence of unambiguous instructions for solving a well-specified computational problem



Course focus – Graph algorithms, greedy algorithms – Mathematical principles of complexity CS 4407, Algorithms University College Cork, Gregory M. Provan

Appendix 

Details of Karutsuba multiplication – Why fast multiplication may matter

CS 4407, Algorithms University College Cork, Gregory M. Provan

Gaussified MULT (Karatsuba 1962) MULT(X,Y): If |X| = |Y| = 1 then RETURN XY

Break X into a;b and Y into c;d e = MULT(a,c) and f =MULT(b,d) RETURN e2n + (MULT(a+b, c+d) – e - f) 2n/2 + f T(n) = 3 T(n/2) + n Actually: T(n) = 2 T(n/2) + T(n/2 + 1) + kn CS 4407, Algorithms University College Cork, Gregory M. Provan

Dramatic improvement for large n

Not just a 25% savings! θ(

2 n)

vs θ(

CS 4407, Algorithms University College Cork, Gregory M. Provan

1.58.. n )