Restricted Orientation Visibility

Restricted Orientation Visibility Sven Schuierery Derick Wood z Dezember 1991 Abstract

Let O be some set of orientations, i.e., O  [0 ; 360 ). In this paper we look at the consequences of de ning visibility based on curves that are monotone w.r.t. to the orientations in O. We call such curves O-staircases. Two points p and q in a polygon P are said to Os-see each other if there exists an O-staircase from p to q that is completely contained in P. We investigate some of the structural properties of Os -visibility and then turn to the computation of the Os -kernel of a polygon. The Os -kernel of a polygon P is then the set of all points which Os -see all other points. We show that the Os-kernel of a simple polygon can be obtained as the intersection of all fgs-kernels, with  2 O. With the help of this observation we are able to develop an O(n log jOj) algorithm to compute the Os -kernel in a simple polygon, for nite O. We also show how to compute the external Os-kernel of a polygon in time O(n + jOj). Both algorithms can be combined to compute the Os kernel of a polygon with holes in time O(n2 + njOj).

 This work was supported by the Deutsche Forschungsgemeinschaft under Grant No. Ot 64/8-1. y Institut fur Informatik, Universitat Freiburg, Am Flughafen 17, D-79110 Freiburg, Fed. Rep. of Ger-

many; email: [email protected] z Department of Computer Science, University of Waterloo, Waterloo, Ontario, N2L3G1, Canada. email: [email protected]

1

Contents 1 Introduction 2 De nitions and Notation 2.1 2.2 2.3 2.4 2.5 2.6

Topology : : : : : : : : : : : : : : : : : Jordan Curves : : : : : : : : : : : : : : Polygons : : : : : : : : : : : : : : : : : Halfplanes, Lines, and Line Segments : Computational Foundations : : : : : : Restricted Orientation Convexity : : : 2.6.1 Two De nitions of O-Visibility 2.6.2 Previous Results : : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

3 The Topology of O-convex Sets 4 Computing the O-Kernel of a Polygon 4.1 The f0g-Kernel of a Simple Polygon : : : : : : : 4.2 The f0g-Kernel Algorithm : : : : : : : : : : : : 4.3 The O-kernel for polygons with holes : : : : : : : 5 Os-Convexity in Simple Polygons 5.1 O-Stairsegments : : : : : : : : : : : : : : : : : : : 5.2 The Os-Hull of Two Points : : : : : : : : : : : : : 5.3 Os-visibility with Dierent Sets of Orientations : 5.4 Os-Skulls : : : : : : : : : : : : : : : : : : : : : : 6 Computing the Os-Kernel of a Polygon 6.1 The Os-kernel for Simple Polygons : : : : : : : : 6.2 A Fast Algorithm for the Os-Kernel of a Polygon 6.3 The Os-Kernel in the General Setting : : : : : : : 6.4 6.5 6.6 6.7

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

1 2 2 3 4 4 5 6 8 9

11 14

: : : : : : : : : : : : : : 15 : : : : : : : : : : : : : : 19 : : : : : : : : : : : : : : 23 : : : :

: : : :

: : : :

: : : :

: : : :

: : : :

:::::: :::::: :::::: The External Kernel of a Polygon : : : : : : : : : : : : : : : Computing the External Kernel : : : : : : : : : : : : : : : : Computing the Os-Kernel of a Multiply Connected Polygon The Relationship between Os-kernel(P) and O-kernel(P) :

: : : : : : : : : : :

: : : : : : : : : : :

: : : : : : : : : : :

: : : : : : : : : : :

: : : : : : : : : : :

: : : : : : : : : : :

: : : : : : : : : : :

: : : : : : : : : : :

25 26 28 31 36

38 38 42 46 49 55 60 61

7 Intersecting Presorted Halfplanes

62

8 A Lower Bound for Re ex -Maxima

68

7.1 An Incremental Algorithm : : : : : : : : : : : : : : : : : : : : : : : : : : : 7.2 Halfplane Intersection and the Convex Hull : : : : : : : : : : : : : : : : : :

8.1 A Lower Bound for a Subset-Containment Problem : : : : : : : : : : : : : 8.2 A Reduction to the -Maxima Problem : : : : : : : : : : : : : : : : : : : : i

63 66

68 68

9 Conclusions

69

ii

1 Introduction Restricted orientation visibility is based on the concept of restricted orientation convexity. Restricted orientation convexity tries to bridge the gap between Euclidean convexity and f0; 90g-convexity. A set S is called f0; 90g-convex or orthogonally convex if the intersection of S with any axis-parallel line is connected, i.e., a point, line segment or the empty set. f0; 90g-convexity is a well-studied area; see [CR89, Kei86, MRS88b, NLLW83, RC87, WY88]. In [Raw87, RW88a, RW87, RW88b] Rawlins and Wood take the idea of f0; 90g-convexity one step further by developing the theory of restricted orientation convexity or O-convexity. Instead of considering only axis-parallel lines they allow lines with orientation in some xed set O. A set S is called O-convex if the intersection of S with any line whose orientation is in O is connected. Note that restricted orientation convexity encompasses both f0; 90g-convexity|if O = f0; 90g|and Euclidean convexity|if O is the set of all orientations|as special cases. The framework of f0; 90g-convexity spawns a de nition of visibility called staircase visibility which is based on f0 ; 90g-convex paths. Two points p and q in a set S are staircase visible from each other if there exists a f0 ; 90 g-convex path from p to q that is completely contained in S. Staircase visibility has been considered by Reckhow and Culberson [CR89] and Motwani et al. [MRS88a]. Both papers deal with covering polygons with the minimum number of f0; 90g-starshaped sets, i.e., sets that contain one point p such that all other points are staircase visible from p. In the same way O-convexity gives rise to a new de nition of visibility we call Os-visibility. Two points p and q in a set S are Os-visible from each other or Os-see each other if there exists an O-convex path from p to q that is completely contained in S. Note that Os-visibility again encompasses staircase visibility and Euclidean visibility as special cases. In this paper we present an O(n2 + njOj) algorithm to compute the Os-kernel of a multiply connected polygon. The problem of computing the Os-kernel of a simple polygon has already been treated in [Raw87, Chapter 6] where a linear time algorithm to compute the f0gs-kernel of a simple polygon is presented. In this paper we continue our investigations of generalized notions of visibility started in [SW91] in a much more specialized setting. While previously we were only concerned with the structural aspects of kernels under a very general notion of visibility, it is now our aim to give algorithms to compute kernels of polygons in the plane. Recall that the kernel of a set S is the set of points that see all other points in S. While for Euclidean visibility this is a well-studied problem where several optimal algorithms have been developed [CG88, LP79], the corresponding problem for f0; 90g-visibility has not been considered. We introduce a new de nition of sight called Os-visibility. It is based on the framework of restricted orientation convexity which was rst considered by G. Rawlins in [Raw87]. Os -visibility oers the opportunity to develop an algorithm that computes the Euclidean kernel as well as the f0; 90g-kernel depending on the input parameter O and that is competitive in both cases. The algorithm we present here is a rst step in this direction. An important requirement in the geometry of xed orientations is to try to nd algo-

1

rithms that do not solve the problem separately, for each given orientation, and then put the obtained solutions together; instead one is interested in algorithms that exploit the \coherence" of the orientations in the solution of the problem. Our algorithms show that this can be done very eciently in the computation of the Os-kernel. The paper is organized as follows. We start o with a precise de nition of O-convexity and Os-visibility in the next section. In Section 3 we take a short glimpse at the topology of O-convex sets in order to justify our assumption that the set of orientation O can be assumed to be closed. This is followed by the presentation of an algorithm for the computation of the O-kernel of a polygon which is based on a second de nition of Ovisibility which we did not mention above. In Section 5 we take a look at the structural properties of Os-visibility making use of the approach developed in [SW91]. Finally, in Section 6 we give an O(n2 + njOj) algorithm to compute the Os-kernel of a multiply connected polygon consisting of n vertices.

2 De nitions and Notation In this section we give short de nitions of the basic concepts we will make use of throughout the rest of this paper.

2.1 Topology

We start o with some notations of elementary topology. We are only concerned with the topology of the plane IE 2 which is based on the family of norms k
kp where p 2 IR, p  1. It is de ned as follows. If x = (x1 ; x2) is a point in the plane, then the k
kp-norm of x is

kxkp = (jx1jp + jx2jp)1=p: k
kp induces the Lp-metric (or dp-metric) which is de ned by Lp(x; y) = kx ? ykp where x

and y are two points in the plane. Note that all norms in IE 2 induce the same topology. We will only consider the L2-metric (d2-metric)|which we also call the Euclidean metric|and the L1 -metric. The interior of a set S is denoted by int(S) and the closureclosure of S by S. The boundary of S which is denoted by @ S is the intersection of the closure of S with the closure of IE 2 n S. Another important concept are curves or paths. A curve or path is an equivalence class C of continuous mappings ' : I ?! IE 2 where I is a real interval. ' 2 C is called a parametrization of C . We will mostly consider a curve as a point set but it should be kept in mind that a curve can always be endowed with a parametrization. Given two curves P and Q with two parametrizations 'P : ha; b] ?! IE 2 and 'Q : [c; di ?! IE 2 (the brackets h and i are used to indicate that it doesn't matter if the left or right end point belongs to the interval or not) such that 'P (b) = 'Q(c) we de ne the concatenation of P and Q, 2

denoted by P  Q, as the equivalence class of the mapping 'PQ : ha; d ? c + bi ?! IE 2 which is given by

(

'PQ(x) = ''P ((xx)+ c ? b) ifif xx 22 [hb;a;db]? c + bi: Q We say that a set S is path-connected if, for any two points x and y in S, there exists a path P connecting x and y that is completely contained in S. The sets we will consider in this paper are usually compact, path-connected sets. More precisely, they are the domain of a simple curve as de ned below.

2.2 Jordan Curves

A curve C is called simple if there exists a parametrization ' : ha; bi ?! IE 2 of C that is injective on [a; b). We say C is closed if there exists a parametrization ' of C over [0; 1] such that '(0) = '(1). We call a simple closed curve according to [Rin75] a Jordan curve. By the famous Jordan Curve Theorem [Rin75, 40.1] a simple closed curve J partitions the plane into two parts, one bounded called the interior of J and denoted by int(J ), and the other unbounded called the exterior of J and denoted by ext(J ). We de ne the domain of J to be the union of int(J ) and J and denote it by dom(J ). A simple, closed curve is a special case of a multiply connected curve1 that we de ne as follows. A closed curve C is multiply connected if it is a nite collection fC1 ; . . . ; Cm g of simple curves that satisfy the following properties. 1. All Ci are pairwise disjoint, 1  i  m. 2. There is one curve, say C1 , called the enclosing cycle such that Ci  int(C1 ), for 2  i  m. The remaining curves are called holes. 3. The interiors of the holes are pairwise disjoint. We de ne the domain of C to be

dom(C ) = (int(C1 ) n

[

2im

dom(Ci )) [ C :

Clearly, the domain D of a multiply connected curve C is path-connected and compact with @ D = C . The interior and exterior of C are de ned in the natural way, ext(C ) = IE 2 ndom(C ) and int(C ) = int(dom(C )). We are slightly misusing the de nition of curve here since a multiply connected curve is not the equivalence class of continuous mappings of an interval into the plane. Still this should not lead to misunderstandings. 1

3

2.3 Polygons

Polygons are a special case of multiply connected curves. Usually a polygon P is de ned to be a closed curve that consists of a nite number of line segments, i.e., there is a parametrization of P that is piecewise linear. The edges of a polygon P are the maximal line segments it consists of. Many of the results in this paper are concerned with simple polygons, i.e., piecewise linear Jordan curves. The vertices of a simple polygon are de ned to be the intersection points of the edges. We call a polygon P multiply connected if P is a multiply connected curve and all cycles of P are simple polygons. If not mentioned explicitly otherwise, we use the term polygon to denote a simple polygon. Since we are mostly interested in the properties of the domain of a polygon instead of the curve itself, we will also use the term polygon to denote the domain of a (multiply connected) polygon. In order to distinguish between curves and domains, we use capital calligraphic letters like P for the rst and capital bold-face letters like P for the latter.2 So whenever we want to refer to the curve that encloses a given domain D we will speak of the boundary of D. If we speak of the vertices and edges of P, we, of course, mean the vertices and edges of the boundary of P. If P is a multiply connected polygon, we call the domain of the enclosing cycle Q the enclosing polygon. The term holes is also used to denote the the domains of the polygons interior to the enclosing polygon. As for representing a polygon P we will assume that P is given as a list of its vertices sorted according to a counterclockwise traversal of @ P.

2.4 Halfplanes, Lines, and Line Segments

We now turn to the de nitions of the geometric objects we will be dealing with. As mentioned above we are only concerned with the plane IE 2. Sometimes it is helpful to consider IE 2 as a vector space. So, for a point p in IE 2, we denote the corresponding vector by p~. An oriented line l is a set of points in the plane of the form fp 2 IE 2 j ~p = p~1 + ~p2 ;  2 IRg, with p1 ; p2 2 IE 2, together with a unit vector ~u that is contained in the linear hull of p~2 . We call ~u the associated vector of l. We say two lines are parallel if their associated vectors are collinear. Let ~u? be the unit vector that is obtained if we turn ~u for 90 counterclockwise. We say a point p is (strictly) to the left of l if p has a non-negative (positive) second coordinate in the (~u; ~u?)-coordinate system with origin somewhere on l. Similarly, we say p is (strictly) to the right of l if p has a non-positive (negative) second coordinate. We denote the closed halfplane of all points that are to the left of l by h+(l) and the closed halfplane to the right of l by h?(l). Since ~u is completely speci ed if its angle with the x-axis is known, we will not distinguish between angles and unit vectors. We use the term orientation to denote either an unit vector or an angle. 2

Note that calligraphic letters may also be used to denote a family of sets.

4

We denote the line segment between two distinct points p and q by pq. In order to denote the oriented line from p to q we use the notation `(p; q) and, similarly, for the ray starting in p that goes through q the notation r(p; q). If we are given a point p and an orientation , `(p; ) denotes the oriented line through p in direction  and, in the same way, r(p; ) denotes the ray starting in p in the direction of .

2.5 Computational Foundations

The algorithms developed in this thesis are analysed according to their asymptotic worstcase behaviour. The underlying model of computation we make use of here is the real RAM (random-access machine) model. A real RAM consists of an arithmetic unit and an unbounded number of storage cells each of which is capable of storing one real number. The arithmetic unit consists of a constant number of registers each of which can also store a real number. The commands to control a real RAM resemble the assembly code of existing processors. So we assume that the following operations can be carried out in one step on a real RAM. (i) to store/load a real number in/from a storage location from/to a register; (ii) to execute a conditional or unconditional jump; (iii) to compare two reals; (iv) to add, subtract, multiply or divide two reals; (v) to compute the kth root or the kth power of a real or to perform a trigonometric operation. Since the algorithms in this thesis are not formulated as real RAM-programs, the analysis of the running time (depending on the input size) can only be accurate up to a constant factor, i.e., we are only interested in the growth rate of an algorithm. This can be conveniently expressed in the O- and -notation. We de ne for a function f : IN ?! IR+ (see [OW90]) O(f ) = fg j 9c1; c2 > 0 : for all n 2 IN : g(n)  c1f (n) + c2 g and

(f ) = fh j 9c > 0 : for in nitely many n 2 IN : h(n)  cf (n)g:

If a function g is in O(f ), we also write g(n) = O(f (n)) and in the same way if h 2 (f ), then we write h(n) = (f (n)). (f ) is de ned as the intersection of O(f ) and (f ). Again we write g(n) = (f (n)) if g 2 (f ). Finally, we de ne o(f ) = fg j n?!1 lim g(n)=f (n) = 0g: 5

We use the algebraic computation tree model to prove lower bounds for the problems we consider. It can be speci ed as follows. The statements si, 1  i  m, of an algebraic computation tree program are allowed to be only of the following four types (we assume that the statement we currently consider has index i) 1. z u  v with  2 f+; ?; ; =g; go to statement si+1; 2. If u : 0 (where : 2 f=; g), then go to statement sj else to statement sk with j , k > i; 3. Halt and output \yes"; 4. Halt and output \no". The variables u and v above have to be either results of previous computations or input parameters. Note that the algebraic computation tree is a weaker model of computation than the real RAM. The time needed by an algebraic computation tree program is de ned to be the longest possible executed sequence of statements starting with s1 . A problem is said to have complexity at least T (n) if the time needed by any algebraic computation tree program is  T (n). It has been shown that the following problem takes time (n log n) in the algebraic computation tree model [Yao89]. The Integer Element Uniqueness Problem: Given n integers decide whether two of them are equal.

This will be used in Section 4.3 to prove a lower bound on the problem of nding the

f0g-kernel for polygons with holes.

2.6 Restricted Orientation Convexity

Restricted orientation convexity can be understood as a generalization of the usual notion of convexity in the plane. There are many ways to de ne the family of convex sets. The de nition we give leads in a natural way to the family of restricted oriented convex sets. We say a set is convex if its intersection with any line in the plane is connected. Recall that we regard the empty set to be connected as well. If we consider only the intersection with a certain class of lines and still require the intersection of a set to have a connected intersection with each line in the class, the resulting family of sets shares some of the properties of convex sets while others are lost. In the following we will investigate the properties of the family of sets if we restrict the orientation of the lines we require a set to have a connected intersection with. If we are given an oriented line l in the plane by l = fp 2 IE 2 j p~ = p~1 + ~p2g with associated vector ~u, we de ne its orientation to be the angle that ~u forms with the x-axis and denote it by (l). Of course, we can speak in the same way of the orientation of a line 6

O

u2

C

u1

u

l

C0

(a)

(b)

Figure 1: C is O-convex while C0 is not. segment or a ray. The orientation of a point p is de ned to be the orientation of the line segment from the origin 0 to p. In order not to have to deal with angles that are greater than 360, we assume from now on that the addition and subtraction of two orientations is done modulo 360. The restricted oriented convex sets can now be de ned as follows. De nition 2.1 Let O be a subset of [0 ; 360). A set C  IE 2 is O-convex if l \ C is connected, for all lines l with (l) 2 O. An example is shown in Figure 1. The rst thing to note about this de nition is that we can assume that O is symmetric w.r.t. the origin, that is, if orientation  is in O, then a set is O-convex if and only if it is O [ f + 180g-convex. We denote the orientation  + 180 by ?1. So there is no loss in generality if we assume that, for all  2 O, we also have ?1 2 O. We also want to be able to compare two orientations and say which one comes (counterclockwise) before or after the other. In order to do so for two given orientations 1 and 2 , we consider an oriented line l through the origin such that the normal vectors u~1 and u~2 that correspond to 1 and 2 are to the left of l. We now say that 1 is before 2 if the angle enclosed by the associated vector ~u of l with u~1 is less than the angle between u~2 and ~u. More formally, we de ne

1  2 if 2 ? 1  180: For an illustration refer to Figure 1b. Since O always contains either both orientations  and ?1 or none of them, we use the notation jOj to denote half the cardinality of O. Furthermore, we will only specify the orientations in [0; 180) to de ne a particular O though it should be kept in mind that O also contains the opposite orientations in [180; 360). Another consequence of the fact that O is symmetric w.r.t. to the origin is that given some orientation  2 O we can speak of the next orientation in O w.r.t.  which is de ned 7

as the in mum of all orientations 0 in O with   0 .3 In the same way we can speak of the previous orientation in O w.r.t. . We say a range (1; 2 ) is O-free if (1 ; 2 ) \ O = ;. A range (1 ; 2) is called a maximal O-free range if (1 ; 2) is O-free and there is no other range (10 ; 20 ) that is also O-free and contains (1 ; 2 ). If  is some orientation in [0 ; 360), the maximal O-free range of  is the maximal O-free range that contains  or if such a range does not exist, the emptyset. In the same way we will speak of the maximal O-free range of a line, ray, line segment, or point which is, of course, meant to denote the maximal O-free range of the orientation of the geometric object. Two orientations 1 , 2 in O are called adjacent if (1 ; 2) is O-free. Since O-free ranges play an important role in the theory of O-convex sets, it is important to note that we can map any two linearly independent unit vectors u~1 and u~2 that correspond to two orientations 1 and 2 with a bijective linear mapping f to the unit vectors e~1 and e~2 that have orientation 0 and 90, respectively. Since f is linear and bijective, all incidences of lines are preserved and none are introduced. Hence, we may assume in the following that if we consider a particular non-empty O-free range (1 ; 2 ), then 1 = 0 and 2 = 90. Another property of O-free ranges is that they are only useful if jOj  2. Hence, it will be often necessary to treat the case jOj = 1 separately. It is easy to see that O-convex sets are closed under intersection and, thus, form a convexity space. We de ne the O-hull of a set S by

\

O-hull(S) = fC  IE 2 j C contains S and C is O-convexg: Obviously, we have that if O equals [0 ; 360), then the O-convex sets are exactly the convex sets in the usual sense.

2.6.1 Two De nitions of O-Visibility As we already mentioned before it is the aim of this paper to investigate some properties of visibility in the context of O-convexity. So the next thing we need is a \natural" generalization of the usual line segment visibility. Our rst approach is based on the O-hull of two points. De nition 2.2 Let P  IE 2 and p; q 2 P; we say p and q O-see each other or are O-visible from each other if O-hull(fp; qg)  P. Although this de nition is straightforward once O-convexity has been established, the resulting notion of visibility has some rather undesirable properties as the O-hull of two points can be disconnected (see Figure 2). In general we can say that, for any two points p; q 2 IE 2, we have that

(

if (pq) 2 O O-hull(fp; qg) = pq fp; qg if (pq) 62 O: 3 Note that since we have  and  ?1 in O , there always is one orientation in O that greater or equal to some other given orientation 0 and there is also one orientation that is less or equal to 0 .

8

q4

q4

q3 q2

p

q1

q3 q2

O

q1

p

O-visibility Os-visibility Figure 2: Two de nitions of visibility in the context of O-convexity. Taking the above equation as a de nition, the O-convex sets can also be characterized as those sets C that contain O-hull(fp; qg), for all p; q 2 C. On the other hand, connectedness is a vital part of \normal" visibility and, therefore, generalizations should try to preserve this strong property. This leads to a second de nition of visibility in the context of O-convexity (see Figure 2).

De nition 2.3 Let S be a subset of the plane and p, q be two points in S; we say p and q Os-see each other or are Os-visible from each other if there is an O-convex path from p to q that is completely contained in S. A subset S of the plane is called Os-convex if any two points of S Os-see each other. In Theorem 5.5.2 of [Raw87] it is shown that S is Os-convex if and only if S is O-convex and connected. Os-convexity is also a natural generalization of the notion of ortho-convexity (i.e., O = f0 ; 90g) as de ned in [RW88b]. From now on we will call an O-convex path an O-stairsegment. An O-stairsegment that consists of a nite number of edges is called an O-staircase. There is an interesting and rather obvious connection to monotone paths. Let O? denote the set of orientations which are orthogonal to those in O. Then, a path P is O-convex if and only if P is monotone with respect to all orientations in O?.

We are mostly interested in the two kernels of a planar set that arise from the above de nitions. Recall that the kernel of a set P is the set of points that see all of P. We will denote them by O-kernel and Os-kernel to distinguish between the two kinds of visibility we are concerned with.

2.6.2 Previous Results Since the work presented here continues the investigations of [Raw87], it will be sometimes necessary to refer to some of the results of [Raw87] as we did above with Theorem 5.5.2. We repeat all the results we reference for the convenience of the reader since we make use of them in several proofs. Since the terminology used in [Raw87] diers from the one developed here, we allow ourselves to adapt the formulation of the result where adequate. 9

De nition: A set P is said to have span [1 ; 2] (where 1  2 ) if, for all p, q 2 P, where p 6= q, (pq) 2 [1 ; 2 ] and this is true for no smaller range.

Lemma 4.2.1 A curve with span [1 ; 2] is an O-stairsegment if and only if the open range (1 ; 2) is O-free. Let h+(p; ) denote the halfplane to the left of the -oriented line through p. Lemma 4.3.1 Let p and q be two points in the plane with (pq) 62 O. If jOj  2, then

O-k[p; q] = h+(p; 1) \ h+(q; 2) \ h+(q; 1?1) \ h+(p; 2?1) if (1 ; 2 ) is the maximal O-free range of pq.

Lemma 5.2.2 If O is non-empty and P is connected, then O-hull(P) is simply connected. Theorem 5.3.1 Let P be connected and p 62 P. p 2 O-hull(P) if and only if there exists a  2 O such that the -line through p intersects P in at least two points on either side of p.

Theorem 5.5.1 If jOj  2 and P is a simply connected set, then P is O-convex if and only if the portions of its boundary between two consecutive O-extremal points are maximal O-stairsegments. Theorem 5.5.2 If jOj  2 and P is a simply connected set, then P is O-convex if and only if, for all p and q in P, there exists an O-stairsegment in P with endpoints p and q. Theorem 6.1.3 P is strong-O-convex if and only if P is convex and O-oriented. Lemma 6.4.1 Given a polygon P, f0gs-kernel(P) lies above any re ex maximum and below any re ex minimum.

Theorem 6.4.2 Let P be a simple polygon. Then \ (Si Oi )s-kernel(P) = Ois-kernel(P): i

Lemma 6.4.2 Let P be a polygon with at least one hole and let p be in P. If `(p; 0) intersects a hole, then p 62 f0 gs-kernel(P). Lemma 6.4.4 Finding the f0gs-kernel of an n-vertex polygon with holes requires

(n log n) time in the algebraic decision tree model of computation. 10

sa a q sn

p1

l

lna rnb

b

sbn

vq

a

rna

p2

lnb

sb

Figure 3: Illustration to the proof of Theorem 3.1.

3 The Topology of O-convex Sets Before we start to investigate the properties of O- and Os-visibility, we want to take a look at the topological properties of the family of O-convex sets. This aspect is not treated in [Raw87] though we feel that it also illustrates some of the more interesting properties of O-convex sets. In this section we abandon our convention that we only consider pathconnected sets. Hence, the term connected is used in its usual meaning. In [Raw87] it is assumed that O is always a closed set. This leads to the question if a (useful) theory of O-convexity can be developed if O is allowed to be any subset of [0 ; 360) (see Section 8.1.10 of [Raw87]). In the following theorem we can show that as long as we are concerned with closed connected sets the notion of O-convexity and O-convexity do not dier. Hence, there is no loss of generality in assuming O to be a closed set. Note, in particular, that our de nition of Os-visibility is based on O-stairsegments which are compact connected sets. Theorem 3.1 Let C be a closed connected set in the plane and O a set of orientations. C is O-convex if and only if C is O-convex. Proof: Since O  O, the if-part of the claim is immediate. So assume C is O-convex and let l be a line with (l) =  2 O n O. Let p1 and p2 be two points of l \ C. Since  2 O, there is a sequence (n)n2IN of orientations in O that converges to . W.l.o.g. assume that  = 0 and that all n are greater than 0 . Further label p1 and p2 such that p1 is to the left of p2. Let q be some point in p1 p2. We want to show that q 2 C (see Figure 3). Let vq be the vertical line through q; furthermore, let sa be the closest point to q in C \ vq above q and sb the closest point below q. Since C \ vq is closed and C is connected, one of sa and sb clearly exist. If one of them does not exist, we assume it to be an arbitrary point above or below q, respectively. If q equals either sa or sb , then we are done. So assume q 62 C. Choose n0 big enough such that, for all n  n0, all the lines lna through p1 with orientations n intersect vq below sa and all the lines lnb through p2 with orientations n 11

intersect vq above sb . We denote the intersection of vq and lna by san and the intersection of vq and lnb by sbn. Let rna be the ray starting in san with orientation n , i.e., rna does not contain p1 . rnb is de ned analogously, i.e., rnb has orientation n?1 . Let P be the biin nite curve rna  sansbn  rnb . It divides the plane into two disjoint open sets one of which contains p1 and the other contains p2. Since C is connected, it intersects P . C cannot intersect sansbn because of the de nition of sa and sb and the way we chose n. Therefore, C either intersects rna or rnb . Assume C intersects rna and let a be in the intersection of C and rna . Because a 2 lna , we have (p1a) = n 2 O and the O-convexity of C yields p1a  C. Furthermore, since a 2 rna , we have san 2 p1a and, hence, san 2 C. This contradicts our choice of sa as the closest point on vq to q. Therefore, q belongs to C. Since q is arbitrary in p1p2, we have p1p2  C. Hence, l \ C is path-connected and this immediately yields the claim. 2 Since the properties of O-convexity we are interested in are usually related to closed connected sets, we will assume from now on that O is closed. The necessity of connectedness in the above claim is established by two points which lie on a horizontal line and which are O-convex, for any set of orientations O that is contained in [0 ; 360) n f0; 180g. If we assume the sets we are dealing with to be compact, i.e., closed and bounded as well, we get some interesting properties from the Hausdor Metric. De nition 3.1 (R. Schneider: Convex Geometry Section 1.8 [Sch89]) Let C1 and C2 be two compact sets in the plane. We denote the Hausdor distance of C1 and C2 by dH (C1; C2) and de ne it to be the maximum of supp2C1 inf q2C2 d2(p; q) and supq2C2 inf p2C1 d2(p; q). It can be shown that the Hausdor distance de nes a metric in the space C 2 of compact sets in the plane and that the metric space (C 2 ; dH ) is complete. Our main intrest is in the properties of the subset of all connected O-convex sets of (C 2 ; dH ) which we denote by O-C 2.

Theorem 3.2 O-C 2 is a closed subset of C 2 . Proof: To show the claim let (Cn)n2IN be a sequence in O-C 2 that converges to C 2 C 2 . We have to show that C is O-convex and connected. First we will prove that C is O-convex. Let l be an O-line that has a non-empty intersection with C and let p and q in l \ C. W.l.o.g. we assume that (l) = 0 and that p is to the left of q. Let (pn)n2IN and (qn)n2IN be sequences with pn; qn 2 Cn and

limn!1 pn = p and limn!1 qn = q. Let r be a point in pq and let vr be the vertical line through r. We rst want to construct a sequence (rn)n2IN with rn 2 Cn that converges to r (for illustration see Figure 4). There is a n0 2 IN such that, for all n  n0, we have maxfd2(pn; p); d2(qn ; q)g < minfd2(p; r); d2(q; r)g and, hence, pn is on one side of vr and qn on the other. For n < n0 we choose rn to be pn. 12

rnq l hpn

p pn

sqn

r

spn vr

q qn hn q rp n s

Figure 4: The construction of a sequence the converges to r. Now let n  n0 and consider the points pn and qn. W.l.o.g. assume that pn is lower than qn. We consider the two horizontal lines hpn through pn and hqn through qn. The intersection point of vr and hpn is denoted by spn and the intersection point of vr and hqn is denoted by sqn, respectively. Let further rnp be the horizontal ray to the right that originates in spn, i.e., (rnp ) = 0. Analogously let rnq be the horizontal ray to the left that originates in sqn, i.e., (rnq ) = 180. Now let P be the biin nite curve rnp  spnsqn  rnq . It divides the plane into two disjoint open regions one of which contains pn and the other contains qn. Since Cn is connected, Cn intersects P . If Cn intersects spnsqn, then we can choose rn to be a point in Cn \ spnsqn. If C intersects rnp in, say, s, then spn is contained in Cn since Cn is f0g-convex. This yields spn 2 Cn and we can choose rn to be spn. If C intersects rnq , then we have sqn 2 Cn and we can choose rn to be sqn. In any case we have that d2(rn; r) < maxfd2(p; pn); d2(q; qn)g and, hence, rn ! r. With the help of (rn)n2IN we can now contruct a sequence (rk0 )k2IN in C that converges to r. For all k 2 IN , there is a nk 2 IN with dH (Cnk ; C) < 1=(2k) and d2(rnk ; r) < 1=(2k). Hence, there is a point rk0 in C with d2(rnk ; rk0 ) < 1=(2k) and, therefore, d2(r; rk0 )  d2(r; rnk ) + d2(rnk ; rk0 ) < 1=k. So, the sequence (rk0 )k2IN converges to r. Since C is closed, this implies r 2 C and, thus, pq  C. Now we proof that C is connected. For assume that this is not the case; then, there are two disjoint, non-empty sets C0 and C00 which are open and closed in C. Since they distance between C0 are closed in C, they are compact in IE 2 and, hence, the Hausdor S 00 and say " > 0. Therefore, the open sets D0 = fN"=2(p) j p 2 C0g and D00 = SfNC (isp) positiv, j p 2 C00g are disjoint. Now choose n big enough such that dH (Cn; C) < "=2. "=2 Cn is then contained in D0 [ D00 and D0 and D00 both have a non-empty intersection with Cn. This contradicts the connectedness of Cn. 2 It should be noted that the family of all path-connected, compact, and O-convex sets is not a closed subset of C 2 . To see this just consider the graph of the function sin(1=x). If we restrict the x-values to the interval [1=n; 1] the graph is a compact, path-connected, and f90g-convex set. But if n ! 1, the limit consists of the line segment (0; 0)(0; 1) together with the graph of sin(1=x) over the interval (0; 1] and it is well known that this is a connected but not path-connected set. On the other hand, if we give up connectivity the claim is false as well. As a counterexample consider the sequence Sn of sets which is de ned by Sn = f(0; 1=n); (1; 1=(n + 1))g. 13

Each Sn is f0g-convex but the sequences converges to f(0; 0); (1; 0)g which is not f0gconvex. Another interesting question is the topological nature of the O-hull. We have the following result on this. Theorem 3.3 If O is closed and A is a connected, compact set, then O-hull(A) is also connected and compact. Proof: The connectedness is just Lemma 5.2.2 of [Raw87]. It can be easily seen, that O-hull(A) is bounded. So we just focus on the closedness of O-hull(A). Let (pn)n2IN be a converging sequence in O-hull(A) with pn ! p. By Theorem 5.3.1 of [Raw87] there exist points qn and rn in A with pn 2 O-hull(fqn; rng) and (qnrn) 2 O. Since A is compact, there exists a converging subsequence (qnk1 )k12IN of qn . For the same reason there exists another converging subsequence (rnk2 )k22IN of rn with nk2 2 fnk1 j k1 2 IN g. Let q 2 A be the limit of qnk2 and r 2 A the limit of rnk2 . Since O is closed, the sequence (qnk2 rnk2 ), k2 2 IN , converges to a value in O. And it is easy to see that p 2 O-hull(fq; rg) = qr  O-hull(A). 2 Because of the above results, we will assume throughout the rest of this chapter that O is a closed subset of [0; 360).

4 Computing the O-Kernel of a Polygon In this section we present an algorithm to compute the O-kernel of a polygon if O is nite. The approach we take is the following. First we show that the O-kernel can be computed as the intersection of the fg-kernels, taken over all  2 O. Since these intersections can be computed eciently, we concentrate on the computation of the O-kernel if O consists of only one orientation. Clearly, we can assume w.l.o.g. that O = f0g. We give a linear time algorithm to compute the f0g-kernel of a simple polygon and show how to adapt it to handle multiply connected polygons. We rst turn to proving the above mentioned intersection property of the O-kernel. Lemma 4.1 (Intersection Lemma) Given a set P  IE 2 and sets of orientations Oi ; S i 2 I 4, with O = i2I Oi we have that

\

O-kernel(P) = Oi-kernel(P): i2I

Proof: Let p 2 O-kernel(P). We obtain O-hull(fp; qg)  P, for all q 2 P. Since O-hull(fp; qTg) is Oi -convex, for all i 2 I , we have that p 2 Oi-kernel(P) and this implies that p is in i2I TOi-kernel(P). Now let p 2 i2I Oi-kernel(P) and q 2 P arbitrary. Further, let # be the orientation of the line de ned by p and q. We distinguish two cases. 4I

denotes an arbitrary index set throughout this paper.

14

First we assume that # is in O. Since O = Si2I Oi , there is an i0 2 I with # 2 Oi0 . Hence, O-hull(fp; qg) = pq = Oi0 -hull(fp; qg)  P, because p Oi0 -sees q. But then p O-sees q. On the other hand, if # 62 O, then O-hull(fp; qg) = fp; qg  P, and p O-sees q.

2

As we already observed the above lemma enables us to compute the O-kernel, for nite O, once we are able to compute the fg-kernel of a polygon. In the following we present a linear time algorithm to compute f0g-kernel(P)5 , for a simple polygon P.

4.1 The f0g-Kernel of a Simple Polygon

In order to develop an algorithm that computes the f0g-kernel of a polygon, we need to gain some insight in its structure. As a rst step the next lemma describes the f0g-kernel of a polygon P as the area of those points that intersect the boundary of P only once if they are moved in nitely far to the left or right.

Lemma 4.2 Let P be a simple polygon. A point p in P belongs to f0g-kernel(P) if and only if the intersection of the horizontal line hp through p with P is connected. Furthermore, if p 2 f0 g-kernel(P), then any point on hp \ P belongs to f0 g-kernel(P). Proof: Let p be a point in P, hp the horizontal line through p, and q 2 P \ hp. If p belongs to f0g-kernel(P), then we have that f0g-hull(fp; qg) = pq  P. This implies that hp \ P is connected since q is chosen arbitrarily. Now let hp \ P be connected. Hence, p sees all the points in P that have the same y-coordinate as p. Since f0g-hull(fp; qg) = fp; qg  P, for q 62 hp, we have that p f0g-sees all points in P and, hence, p 2 f0g-kernel(P). To see the second claim, just note that we have hp \ P = hq \ P, for any point q in hp \ P. Therefore, we have that if p 2 f0g-kernel(P), then hq \ P is connected and by the rst claim q also belongs to f0g-kernel(P). 2 We give an example of the f0g-kernel of a polygon in Figure 5. Clearly, the above statement implies that the intersection of f0g-kernel(P) with any horizontal line is connected.

We state this as a corollary. Corollary 4.3 If P is a polygon, then f0g-kernel(P) is f0g-convex. The corollary yields that the f0g-kernel of a polygon P consists of f0g-convex components whose y-ranges do not overlap. There is an interesting relationship between the maximal O-convex subsets and the O-kernel of a polygon P if jOj = 1. The maximal Oconvex subsets of P are called O-skulls and are denoted by O-skulls(P). For the (rather simple) proof of the following lemma we need the notion of the O-join of the O-hull of twoSpoints with a third point. In general the O-join of a set C and a point p is de ned to be q2C O-hull(fp; qg) and denoted by O-join(p; C). In Section 6.1 we will make use of a dierent join concept to prove a similar result in the case of Os-visibility. 5 Remember that the f0 g-kernel (P) actually is the f0 ; 180 g-kernel of P. 15

P

Figure 5: An example of the f0 g-kernel of a polygon. Theorem 4.4 If P is a polygon, then we have \ f0g-kernel(P) = f0g-skulls(P): Proof: Since the semi-convex space of O-convex sets is simple and complete, we only have to prove that f0g-join(p; f0g-hull(fq; rg)) is f0g-convex, for all points p, q, r 2 IE 2 according to the Kernel Theorem of [SW91]. First we consider the case that f0g-hull(fq; rg) = qr. If p lies above or below qr, then the join just consists of p and qr which, obviously, is f0g-convex. If, on the other hand, p has the same height as q and r, then the join consists of one line segment, which, of course, is again f0g-convex. Now suppose that f0g-hull(fq; rg) = fq; rg. If p does not lie on a common horizontal

line with either q or r, then we have the three points (with dierent heights) as the join and the join is f0g-convex. Should p have the same height as one of the points q or r, then the join consists again of a line segment and a point and is, thus, f0g-convex. 2 It can easily be seen that this nice property does not carry over to the case jOj  2. An examples is shown in Figure 6. Since the connected components of the f0g-kernel of a polygon P are f0g-convex, they contain two monotone chains w.r.t. the y-axis and at most two extremal horizontal edges. Clearly, the monotone chains are part of the boundary of P. So the main task in computing f0g-kernel(P) consists of nding the maximal and minimal coordinates of the components. Before we continue, we need to introduce some more notation which we will also make use of later on in this paper. Let  be some orientation in O. We consider the coordinate system that has a -oriented x-axis and a ( + 90)-oriented y-axis. For a polygon P and a point p on the boundary of P, we say that p is a (local) -maximum of P if there exists a neighbourhood N of the connected component of @ P \ `(p; ) such that there is no point in @ P \ N that is higher than p in the -coordinate system. For illustration refer to Figure 7. 16

Figure 6: The f0; 90g-kernel of a polygon is not f0; 90g-convex.

int(P)

N p



@P

ext(P)

A re ex -maximum

N ext(P)

 p

@P

int(P)

A convex -maximum

Figure 7: The -maximum of a polygon P.

17

 + 90



The -coordinate system

P

vi?1

vi vi+1

vi?1

P

vi+1

ei

ei+1

vi

ei?1

ei?1 ei

ei+1

P

P

Figure 8: Possible maxima and minima of a polygon. p is called a (local) re ex -maximum if p is a -maximum and there exists a neighbourhood N of p such that ext(P) \ N is below the -oriented line through p in the -coordinate system, otherwise p is called a convex -maximum. A -maximum p is said to be global if it is the highest local -maximum of P. The de nition of -minima is analogous to the de nition of -maxima. If  = 0, we just speak of maxima and minima. Note that a maximum or minimum may consist of a vertex or an edge of P (see Figure 8). An (oriented ) tangent t of a set S is an oriented line such that t intersects the boundary of S and such that S is completely contained in h+(t). A horizontal tangent is a tangent t such that (t) is either equal to 0 or 180. With the above terminology, we have the following result about the f0g-kernel of a polygon P. Lemma 4.5 Let P be a polygon and let C be a component of f0g-kernel(P), then 1. Either C contains the highest vertex of P or the topmost horizontal tangent t to C contains a local minimum of P. t intersects C if and only if t does not touch a convex minimum. 2. Either C contains the lowest vertex of P or the bottommost horizontal tangent t to C contains a local maximum. t intersects C if and only if t does not touch a convex maximum. Proof: We only prove the rst claim since the proof of the second claim is completely symmetric to the rst one. Suppose that C does not contain one of the highest vertices of P.

Case 1 There is at least one point p 2 C with highest y-coordinate. Clearly, p is intersected by the topmost tangent to C. Let h be the horizontal line through p. By Lemma 4.2 we have h \ P  C. So there is a highest non-degenerate edge e in C.

Furthermore, there is some horizontal line in every neighbourhood above e that intersects P in at least two disconnected components. Hence, we obtain a sequence of horizontal lines hn that converge to h and that intersect P in at least two components. So, there are at least four non-horizontal edges ei in P, 1  i  4, that are intersected consecutively in that order from left to right by almost all hn (see Figure 9a). We can assume that e1 18

e1

e2

e3 (a)

e4

e1

hn h

e2 e3 (b)

e4

hn h

Figure 9: The two cases that can occur for the highest segment of a f0g-kernel component. and e3 are directed downward and e2 and e4 are directed upward if we orient the boundary of P counterclockwise. Since the edges ei are closed, we have that h also intersects them again in the same order from left to right. Therefore, e2 and e3 meet in one vertex or edge that is contained in h. Since e2 is a upward directed edge and e3 is a downward directed edge, they form a re ex minimum of P. Case 2 There is no point with highest y-coordinate in C. By the same reasoning as above we obtain a sequence of horizontal lines that converge and intersect P in two disconnected components (see Figure 9b); only now the limit h also intersects P in two disconnected components. Since every horizontal line in a suciently small neighbourhood below h has a connected intersection with P, there is at least one vertex or edge on h before or after the one connected component of h \ P that contains @ C; therefore, we get a local minimum with a convex vertex or edge that is touched by h.

2

4.2 The f0g-Kernel Algorithm

In this section we present a linear time algorithm to compute the f0g-kernel of a simple polygon P. Later we will give a modi cation of this algorithm that needs time O(n log n) to compute the f0g-kernel of a multiply connected polygon. The algorithm follows a very simple idea. Starting with the lowest vertex we scan the boundary of P counterclockwise. During this scan we maintain a list L of y-intervals that correspond either to possible components of f0g-kernel(P) or to ranges that have already been recognized as not being part of f0g-kernel(P), i.e., ranges in which we have already detected a horizontal line that intersects P in at least two line segments. With a y-interval that corresponds to a (possible) component C of f0g-kernel(P) we also store those edges of P that belong to the boundary of C and that we have discovered so far. For each edge e we encounter in the scan, we have to update the list L of y-intervals. Furthermore, we may have to add e to some part of the boundary of the currently considered component. In the scan we distinguish three kinds of y-intervals. 1. The intervals that do not intersect the y-range of any of the so far processed edges. We call these free intervals. In the beginning, the whole y-axis is one free interval. 19

L Ik

ek e e2

I2

e1

I1

component intervals

Figure 10: Splitting up edge e in subedges e1 ; . . . ; ek . After having looked at at least one edge there are exactly two free intervals, one above the highest and one below the lowest so far encountered edge. 2. The intervals which may contain the y-range of a component of f0g-kernel(P). We call these component intervals. A component interval I has neither been completely covered more than twice by the edges of P we have looked at so far nor have we encountered an upward edge to the left of a downward edge in the horizontal slab spanned by I . Both of these conditions would exclude I from the intervals that may contain a component of f0g-kernel(P). 3. Finally, there are the intervals that have been crossed at least three times or that contain an upward edge to the left of a downward edge. These are called deleted intervals since such an interval can never contain a component of f0 g-kernel(P). Note that the intervals in L are alternatingly component and deleted intervals except for the rst and last interval which are free intervals since any sequence of adjacent intervals of the same kind can, of course, be joined into one interval. So suppose that we have processed some part of the boundary of P and now encounter edge e. e can either be directed upward or downward. Since these cases are symmetric, we will assume that e is an upward edge, i.e., the interior of P is locally to the left of e. The y-range of e may span several intervals I1; . . . ; Ik in L. Therefore, we split e up into a number of subedges e1 ; . . . ; ek such that the y-range of ei is contained in Ii (see Figure 10). Now consider some subedge ei . Let vs be the start vertex of ei and vt the end vertex. We distinguish several cases depending on which kind of interval vs belongs to. If vs is on the boundary of two intervals, we just consider the upper one since e is directed upward. We denote the interval vs belongs to by Is and the interval below Is by Ib . 20

Case 1 Is is a free interval.

If Ib is a component interval, then we just extend Ib to the y-coordinate of vt and add ei to the right boundary of the component of f0g-kernel(P) that corresponds to Ib. If Ib is a deleted interval, then we create a new component interval Is0 between Ib and Is that spans the y-range of ei. Again the part of ei that is above Ib is added to the right boundary of the component that corresponds to Is0 . In both cases the lower end point of Is is moved up to the y-coordinate of vt . In the very beginning we have that vs belongs to the only interval of the list which is a free interval. In this case we create a new component interval Is0 that spans the y-range of ei and split the free interval Is up into two free intervals, I1 and I2, with I1 below and I2 above Is0 . Case 2 Is is a component interval. Let C be the (possible) component of f0g-kernel(P) that corresponds to Is. Case 2.1 There is a downward edge to the right of ei in C or there is another upward edge at the same height as ei in C. In both of theses cases a horizontal line that has the same height as ei intersects P in at least two disconnected components. We then delete the part Is0 of Is from the y-coordinate of vs to the y-coordinate of vt . This divides Is into at most three parts, the part I1 that is below Is0 , Is0 itself, and the part I2 that is above Is0 . If one of I1 or I2 is empty and there is a deleted interval below or above Is, we can combine it with Is0 . Note that both of I1 and I2 are non-empty if and only if the y-range of edge e is completely contained in Is. This will be important in the analysis of the algorithm. Case 2.2 There is no downward edge to the right of ei in C nor an upward edge at the same height. The right boundary of C has not been traversed yet and the insertion of ei does not lead to a disconnected intersection with a horizontal line. Hence, we add ei to the right boundary of C and leave L unchanged. Case 3 Is is a deleted interval. In this case there is nothing to do since Is is already deleted and ei cannot belong to any component of f0g-kernel(P). The proof of correctness of the above algorithm is given by the following two lemmas. Lemma 4.6 If P is a polygon in the plane and p 2 f0g-kernel(P), then there is a component interval I in the output of the f0 g-kernel algorithm for P that contains the y-coordinate of p. Furthermore, the component corresponding to I contains p. Proof: Let p 2 f0g-kernel(P). Consider the horizontal line h through p and assume it intersects the edge er 2 P on the right hand side. Clearly, this edge is directed upward. Since p belongs to f0g-kernel(P), h does not intersect any other edge of P to the right of it nor an upward edge to the left of it; hence, er was added either in Case 1 or 2.2. The same argument applies to the edge el that is intersected by h on the left hand side. 21

Therefore, the y-coordinate of p belongs to one of the component intervals that are created during the execution of the algorithm and the y-coordinate of p cannot be deleted from it since then there would be a third edge that intersects h. In the same way we obtain that the edges er and el form the right and left boundary of the component corresponding to the interval in L p belongs to. 2

Lemma 4.7 If P is a polygon in the plane and p 62 f0g-kernel(P), then there is a deleted or free interval in the output of the f0 g-kernel algorithm for P that contains the y-coordinate of p.

Proof: If p is above or below P, the claim is obvious. Hence, suppose that p belongs to the horizontal slab spanned by P. Again consider the horizontal line h through p. p is not contained in f0g-kernel(P), therefore, h intersects at least four edges e1; . . . ; em with m  4. Let the numbering denote the sequence in which the algorithm encounters the edges. Assume that e1 is an upward edge. By the topological properties of polygons e2 is then directed downward. If e2 is to the right of e1 , then we delete the y-coordinate of p from the component interval by Case 2.1. If e2 is to the left of e1 , then the y-coordinate of p is deleted from the component interval when we encounter the upward edge e3 . 2 Analysis So it just remains to be shown that the algorithm, indeed, needs linear time. Obviously, each of the cases we considered above needs only a constant amount of time. Hence, processing a subedge ei of e can be done in a constant number of steps. So, if e is not split up, then we are within our time bound. On the other hand, if e is split up into a number of subedges e1 ; . . . ; ek , then note that the y-range of e includes at least b(k ? 2)=2c component intervals since the intervals in L are alternatingly deleted and component intervals with the possible exception of the rst and last interval. If we denote the number of subedges edge e is split up into by ke and the number of component intervals the y-range of e contains by ce, we obtain the following estimate for the total number of subedges created during the algorithm X X ke  (2(ce + 1) + 2): e2P

e2P

Next we show that the sum on the right hand side is linear in the number of vertices of P. Our rst claim is that we create at most n component intervals during the scan if n is the size of P. To see this we note that any edge leads to the creation of at most one component interval. For, consider the Cases 1 and 2.1 which are the only ones that may lead to a new component interval. In both cases we have that the end point of the edge we consider is contained in the interval Is if the number of component intervals increases. Now by Case 2.1 a component interval I is contained in at most two y-ranges that belong to some edges of P before it is combined with a deleted interval. Hence, if we sum the number of edges of P whose y-range contain I over all component intervals I that are created in the scan, we cannot obtain more than twice the number of intervals. In 22

other words, if cI is the number ofPedges thatPcover component interval I we have that P I cI  2n. Clearly, we have that e2P ce = I cI since both sums count how often the y-range of an edge covers a component interval. Hence, we get the following estimate for the total number of subedges created during the execution of the algorithm

X

e2P

ke 

X

e2P

(2(ce + 1) + 2)

X

 2 cI + 4n I

= O(n): This completes the analysis of our algorithm. We have shown the following theorem.

Theorem 4.8 If P is a simple polygon in the plane, then f0g-kernel(P) can be computed in time linear in the number of vertices of P.

4.3 The O-kernel for polygons with holes

If we allow the polygons to be multiply connected (but they still have to be simple), we have to account for the regions of the O-kernel that are deleted by the presence of holes. Fortunately, the Intersection Lemma holds for any subset of the plane. So, again, we only have to show how to compute the f0g-kernel, for a given polygon P. The lemma stated below completely describes the relationship between the f0g-kernel of P and f0g-kernel of the enclosing polygon and the holes.

Lemma 4.9 If P is a simple multiply connected polygon with enclosing polygon Q and holes H1 ; . . . ; Hm, then we have that p 2 f0 g-kernel(P) if and only if 1. p 2 f0 g-kernel(Q) and 2. there is no hole that intersects the horizontal line hp through p.

Proof: Obviously, we have f0g-kernel(P)  f0g-kernel(Q). So let p 2 f0g-kernel(Q) and assume there is no Hi that intersects the horizontal line hp through p and let q 2 P. 6 0, then If (pq) = 0, then q 2 hp and, hence, f0g-hull(fp; qg) = pq  P. If (pq) = f0g-hull(fp; qg) = fp; qg  P. On the other hand, if p 2 f0g-kernel(P), then hp does not intersect any of the Hi since otherwise p cannot see all points in P \ hp. 2  The above lemma leads to the following algorithm to compute the f0 g-kernel of a multiply connected polygon consisting of the enclosing polygon Q and holes H1; . . . ; Hm (the algorithm is very similar to the algorithm described in [Raw87, p. 82]). We rst compute f0g-kernel(Q). The y-coordinates of the components of f0g-kernel(Q) are stored in a range tree T . We use T to answer stabbling queries with the end points of the y-intervals that are covered by the holes Hi. Any component interval that lies between 23

P

P

The f0 g-kernel of P without holes

The f0 g-kernel of P with holes

Figure 11: An example of the O-kernel of a polygon with holes.

two such end points, is deleted since its y-range intersects a hole. Since the y-ranges of the components are disjoint, we can answer a stabbling query in time O(log(# of components of f0g-kernel(Q))) = O(log n). The algorithm can now be described as follows (an example is shown in Figure 11).

Algorithm f0g-kernel for polygons with holes compute f0g-kernel(Q); for each Hi do

compute the maximal yimax and minimal yimin y-coordinate of Hi; nd the components of f0g-kernel(Q) that have a y-range in [yimin; yimax]. let Zmin be the component such that yimin is in the y-range of Zmin ; de ne Zmax analogously; delete the part of Zmin in the slab that is spanned by [yimin; yimax]; delete the part of Zmax in the slab that is spanned by [yimin; yimax]; delete all components between Zmin and Zmax ; end for;

The maximum number of components we can get this way is the sum of the number of components and the number of holes since every hole can split at most one component into two. Finding Zmin and Zmax can thus always be done in time O(log n) if we use the above mentioned range tree. Deletion of a part of a component can be accomplished in time O(log n) plus O(the number of edges deleted) and the deletion of all components between Zmin and Zmax can also be achieved in time proportional to the number of components deleted. Since we can get at most a linear number of edges and components and since each edge and component can be deleted only once, the overall deletion time is linear. Hence, 24

mi m1

mn 0

5

n

Figure 12: The construction of a polygon for n integers. the running time of the f0g-kernel algorithm for an n-vertex multiply connected simple polygon is O(n log n). This is indeed optimal, as the following lemma shows.6 Lemma 4.10 Finding the f0g-kernel of an n-vertex polygon with holes requires (n log n) time in the algebraic computation tree model. Proof: Suppose there is an algorithm that computes the f0g-kernel of a polygon with holes in time o(n log n). We give a linear reduction to the integer element uniqueness problem. Recall that we introduced this problem in Section 2.5 and that it requires time

(n log n) in the algebraic computation tree model. Given n integers m1 ; m2; . . . ; mn 2 IN we construct a polygon P with 4(n + 1) vertices containing n holes as follows (for illustration refer to Figure 12). The ith hole is the rectangle spanned by the points (i; mi + 1=2) and (i + 1=2; mi). We nd the minimum and maximum values of the mi and construct an appropriate surrounding box. All of these operations take linear time. After applying the f0g-kernel algorithm to P we can count, in linear time, the number of components. If it is equal to n + 1, all ni are distinct, otherwise there is some pair of equal integers. 2

5 Os-Convexity in Simple Polygons In the rest of this paper we will look at the second de nition of visibility we proposed in the context of O-convexity. Recall that in Section 2.6.1 we de ned that two points p and 6 The proof idea for this lemma is taken from the proof of Lemma 6.4.4 of [Raw87] which makes a similar statement about the time needed to compute the f0 gs -kernel of a polygon with holes. Though the construction used is exactly the same as presented here the proof in [Raw87] is not correct since a reduction to the element uniqueness problem over the reals is claimed which only works if point holes are considered. A lower bound for the Integer Element Uniqueness Problem was discovered only later.

25

r

r q

p1

p

p

(a)

q

(b)

p2

r

p2 q

l

p1 p

l (c)

Figure 13: How to join two staircases q in a polygon P Os-see each other if there exists an O-stairsegment in P that connects p and q. In this section we will investigate Os-visibility in simple polygons. Our main interest will be to obtain a result similar to the intersection lemma (Lemma 4.1) which drastically simpli es the problem of computing the Os-kernel. Since Os-visibility is based on O-stairsegments, the following section is concerned with some basic properties of O-stairsegments we will make use of throughout the rest of this section.

5.1

O-Stairsegments

We need some technical statements on how to construct new O-stairsegments from some given O-convex curves. The rst lemma shows how to glue two O-stairsegments together.

Lemma 5.1 Let p, q, and r be points in the plane and (; ) an O-free range with (pq) 2 [; ] and (qr) 2 [; ]. If Spq is an O-stairsegment from p to q and Sqr is an Ostairsegment from q to r, then Spq  Sqr is an O-stairsegment from p to r. Proof: We use the same notation as in the claim. We distinguish two cases according to the cardinality of O. Case 1 jOj = 1. W.l.o.g. suppose that O = f0g. Hence, q is above p and r is above q (see Figure 13a).7 So consider a horizontal line l that intersects Spq  Sqr . If l is strictly below q, then l only intersects Spq since Sqr is f0g-convex and, therefore, stays above q. Hence, l \ (Spq  Sqr ) is connected in this case. Similarly, if l is strictly above q, it only intersects Sqr and Note that we will use the terms \above" and \below" meaning at least as high as resp. at least as low as. If we want to indicate that one object is higher (lower) than some other object will use the terms strictly above (strictly below). The same remark applies to the terms \to the right of" and \to the left of". 7

26

l \ (Spq  Sqr ) is again connected. If l has the same height as q, then l \ Spq consists of a horizontal line segment s1 that contains q and l \ Sqr also consists of a horizontal line segment s2 that contains q. Thus, l \ (Spq  Sqr ) = s1 [ s2 is also a line segment. Case 2 jOj  2. We assume w.l.o.g. that the O-free range (; ) equals (0; 90). Consider any O-line l. Suppose that l intersects Spq  Sqr in at least two points p1 and p2 . We will show that p1 and p2 are in a connected component of l \ (Spq  Sqr ). If p1 and p2 are both contained in the same O-stairsegment, say Spq , then they belong to the same connected component of l \ (Spq  Sqr ) since Spq is O-convex. Hence, assume p1 and p2 do not both belong to the same staircase. Let p1 2 Spq and p2 2 Sqr . Since (l) 2 O and (0; 90) is O-free, we have that (l) 2 [90 ; 180] [ [270; 360]. W.l.o.g. we assume that (l) 2 [270; 360], i.e., l is either vertical or has a non-positive inclination. We distinguish two cases. Case 2.1 p1 is the rst intersection point on l. If l is not horizontal, then p2 is strictly below p1 and Sqr cannot contain p2 since q is above p1 and all points of Sqr are above q. Therefore, l is horizontal (see Figure 13b). Since q cannot be strictly above p2 or strictly below p1 , l contains q. Since Spq is O-convex and p1, q 2 Spq with (p1q) = 0, Spq contains p1q. The same holds for Sqr and qp2 . Hence, l \ (Spq  Sqr )  p1q [ qp2; thus, p1 and p2 are connected in l \ (Spq  Sqr ). Case 2.2 p2 is the rst intersection point on l. If l is not vertical, then p2 is strictly to the left of p1 and Sqr cannot contain p2 since q is to the right of p1 and Sqr is also to the right of q (see Figure 13c). Therefore, l is vertical and, as above, p1q and qp2 are contained in Spq  Sqr ; hence, p1 and p2 are connected in l \ (Spq  Sqr ).

2

Now consider two points p and q and an O-convex path S from p to q. If we choose any two points v, w on S with v before w, we claim that we can connect them by an O-stairsegment S 0 and the part Spv of S from p to v and the part Swq of S from w to q together with S 0 form an O-stairsegment, i.e., Spv  S 0  Swq is O-convex. To see this note that Spv and S 0 satisfy the conditions of Lemma 5.1. Hence, Spv  S 0 is O-convex. The same holds for Spv  S 0 and Swq . We state this as a corollary (see Figure 14).

Corollary 5.2 Let p, v, w, and q be four points in the plane that occur in this order on the O-stairsegment S . If we replace the part of S between v and w by some other O-stairsegment, the resulting path S 0 is O-convex. Another corollary tells us how to construct O-staircases from edges. Lemma 5.3 Let S be a polygonal curve consisting of the edges e1 , e2 ; . . . ; en in this sequence. S is an O-stairsegment if and only there exists an O-free range (; ) such that (ei) 2 [; ], for i 2 f1; . . . ; ng. 27

p Spv

v

S0 S

w Swq

s

O

Figure 14: The part of S from q to r can be replaced by any O-stairsegment S 0 . Proof: The \if"-part of the claim is an immediate consequence of Lemma 4.2.1 of [Raw87].

So we just have to show the \only if"-part. We prove it by induction on n. The induction hypothesis clearly holds for n = 1. Hence, assume it holds for all k < n and n is greater than 1. Let S be a polygonal curve of n line segments ei , i 2 f1; . . . ; ng such that (ei) 2 [; ]. Let S 0 be the staircase consisting of the edges e1; . . . ; en?1. S 0 is O-convex by induction hypothesis. If p is the start point of S 0 , q the start point of en, and r the end point of en, then we have the same situation as in Lemma 5.1. Hence, S = S 0  en is O-convex. 2

5.2 The Os-Hull of Two Points

In our further investigations of Os-visibility we want to make use of the results obtained in [SW91].8 In analogy to O-convexity one might want to introduce the following de nition. De nition 5.1 Let O be some set of orientations. We say a set C is Os-convex if any two points in C Os-see each other. The problem we encounter with this de nition is that the intersection of two Os-convex sets is not necessarily Os-convex again. Hence, the family of Os-convex sets does not form a semi-convex space9 . There are two ways to remedy this situation. The rst is to require that all O-stairsegments between two points are contained in the set instead of just one. This leads to the notion of strong-O-convexity which we will not pursue further. The second one makes use of a convexity concept based on \straight" lines w.r.t. some d. It is not very surprising that we actually can use a norm d such that O-stairsegments become d-straight lines. De nition 5.2 Let p be a point in IE 2. We denote the O-norm of p by kpkO and de ne it by ( pk2 if (p) 2 O kpkO = kmin fkp1k2 + kp2k2 j (pi) 2 O and p~1 + p~2 = ~pg if (p) 62 O: Figure 15 illustrates the de nition. Let pq be a line segment with (pq) 62 O and (; ) 8 This section is heavily based on the terminology used in [SW91]. We assume the reader to be familiar with Sections 3.3, 3.4, 5.3, and the Kernel Theorem of [SW91]. 9 A semi-convex space over a groundset X is a family C of subsets of X that is closed under intersection X (see [SW91]).

28

p

kpkO

0

kpk2 = kpkO

@P

p~2

p2

~p

0 p~1 p1

p~2 ~p = p~1 + p~2

kpkO = kp1k2 + kp2 k2 Figure 15: The de nition of k
kO .

q

@P

Os-hull(fp; qg)

O

O-k[p; q]

p

O

p

S0 p

(a)

Figure 16: The Os-hull of two points.

p0

q0

C0

q

dom(C 00 ) (b)

dom(C 00 ) can be added to C.

the maximal O-free range of pq. The distance from p to q is just half the perimeter of the parallelogram spanned by p and q with sides parallel to  and . As in the case of f0; 90gs-convexity (see [SW91]) it can be shown that k
kO is topological, extensible, and sti (for a formal de nition of these concepts we also refer to [SW91]). For further reference we state this as a lemma. Lemma 5.4 k
kO is a topological, extensible, and sti norm in the plane. Since the results obtained in [SW91] only apply to simple domains, we will restrict our discussion to simple polygons. Hence, from now on throughout the rest of this section we use the term polygon to denote a simple polygon.

De nition 5.3 If P is a polygon in the plane and p and q are two points in P, we de ne the Os -hull of p and q to be the union of all O-stairsegments in P that connect p and q and denote it by Os-hull(fp; qg). An example is shown in Figure 16a. Note that the de nition of the Os-hull of two points depends on the polygon we consider. If the Os-hull of two points p and q consists of all O-stairsegments from p to q, we also use the notation O-k[p; q] (read: the O-parallelogram 29

of p and q). In Theorem 4.3.1 of [Raw87] it is shown that if jOj  2, then O-k[p; q] is either the line segment between p and q if (pq) 2 O or the parallelogram spanned by p and q whose sides are parallel to the orientations of the maximal O-free range of (pq) if (pq) 62 O. Note that the de nition of O-k[p; q] is independent of the polygon we consider. Hence, if Os-hull(fp; qg) = O-k[p; q], then the polygon does not present an obstacle for the Os-visibility between p and q. The Os-hull of two points has the following nice property. Lemma 5.5 Let P be a polygon and p and q be two points in P. If C is a maximal, connected O-convex subset of O-k[p; q] \ P that contains p and q, then we have that C = Os-hull(fp; qg). Proof: \": Let r be a point of C. Since C contains p and q, we have by Theorem 5.5.2 of [Raw87] that p sees r via an O-stairsegment Sp in C. Of course, the same holds for q which sees r via an O-stairsegment Sq . Note that we have, for a point r in O-k[p; q] that (pr) 2 [; ] and (rq) 2 [; ] if (; ) is the maximal O-free range of pq. Therefore, we can directly apply Lemma 5.1 to Sp and Sq which yields that Sp  Sq is an O-stairsegment. Hence, r 2 Os-hull(fp; qg). \": Let S be an O-stairsegment from p to q. Assume S is not contained in C. Let p0 be the rst and q0 the second cross point of @ C and S (as seen from p). Let S 0 be the part of S and C 0 be the part of the boundary of C between p0 and q0 as shown in Figure 16b. Clearly, S 0  C 0 forms a simple closed curve C 00 . dom(C 00) belongs to P because P is simply connected; further, it is a part of O-k[p; q] since S 0 and C 0 are contained in O-k[p; q]. Any O-line l intersects S 0 at most once because S 0 is an O-stairsegment. The same holds for C 0 . Hence, l \ dom(C 00 ) is connected and since l \ C is also connected, C [ dom(C 00) is O-convex and clearly path-connected. This contradicts the maximality of C. 2 s The O -hull indeed satis es the conditions associated with a hull operator, i.e., we have that v; w 2 Os-hull(fp; qg) ) Os-hull(fv; wg)  Os-hull(fp; qg): as is shown in the next lemma. Lemma 5.6 Let P be a polygon and p and q be two points in P. If v; w 2 Os-hull(fp; qg), then Os -hull(fv; wg)  Os-hull(fp; qg). Proof: W.l.o.g. we assume that the maximal O-free range of pq is (0 ; 90). Let S be one arm of Os-hull(fp; qg) and assume that S crosses some path S 0 in Os-hull(fv; wg). We distinguish two cases. Case 1 (vw) 2 [0 ; 90]. Assume that S is the upper arm of Os-hull(fp; qg). Let p0 be the rst intersection point of S and S 0 and q0 the last. Consider the part S 00 of S 0 between p0 and q0. Since S crosses S 0 and v and w are in Os-hull(fp; qg), there is a point on S 00 not in Os-hull(fp; qg). But by Corollary 5.2 S 00 is part of an O-stairsegment from p to q. This contradicts Lemma 5.5 and, hence, S cannot cross S 0 . The same reasoning applies to the lower arm of Os-hull(fp; qg) and, therefore, S 0 is contained in Os-hull(fp; qg). Again by Lemma 5.5 we then get that Os-hull(fv; wg)  Os-hull(fp; qg). 30

Case 2 (vw) 62 [0 ; 90]. Assume that S is the upper arm of Os-hull(fp; qg) and w.l.o.g. assume that v is to the left and above w. And consider an O-stairsegment S 0 from p to q through v. Clearly, S has to stay above S 0 . Now S 0 cannot leave the rst and third quadrant around v. Since Os-hull(fv; wg) is contained in the forth quadrant around v, it is completely below S 0 and, hence, below S . For similar reasons, Os-hull(fv; wg) is above the lower arm of Os-hull(fp; qg). Therefore, Os-hull(fv; wg) is contained in Os-hull(fp; qg). 2

In the special case that O-k[v; w] is a subset of P we obtain the following result. Corollary 5.7 Let P be a polygon and p and q two points in P. For every pair of points v; w 2 Os-hull(fp; qg) with O-k[v; w]  P we have that O-k[v; w]  Os-hull(fp; qg). Proof: Just note that if O-k[v; w]  P, then Os-hull(fv; wg) = O-k[v; w]. 2

5.3

Os -visibility

with Dierent Sets of Orientations

The rst question we are interested in is how Os-visibility behaves if we change the set of orientations we consider. More precisely, given sets of orientations Oi, i 2 I , we want to make some statements about the semi-convex space induced by Si Oi . In order to do so we consider Os-hull(fp; qg), for two points p and q in a polygon P. The boundary of Os-hull(fp; qg) has a rather special structure. According to Theorem 5.5.1 of [Raw87] the part S1 of the boundary of Os-hull(fp; qg) from p to q is a maximal O-stairsegment as is the part S2 of @ Os -hull(fp; qg) from q to p if we assume that @ Os -hull(fp; qg) is oriented counterclockwise. If the maximal O-free range of pq is (; ), then S1 as well as S2 are f; g-stairsegments. Clearly, S1 and S2 are staircases and do not cross though they may touch each other. We call S1 the lower arm and S2 the upper arm of Os -hull(fp; qg). The arms of Os-hull(fp; qg) will be the main tool in investigating the structure of the s O -hull. We now can state some properties of the upper and lower arm of Os-hull(fp; qg).

Lemma 5.8 Let P be a polygon and p and q two points in P. If e is an edge of the boundary of Os-hull(fp; qg), then we have either that e  @ P or (e) 2 O. Proof: The proof is by contradiction. So suppose there is an edge e on the boundary of Os-hull(fp; qg) that is neither contained in @ P nor do we have (e) 2 O (see Figure 17). Since @ P is a polygonal path, there is a subedge e0 of e that does not intersect @ P. Since @ P is closed, we can choose two points r and s on e0 that are close enough to each other such that O-k[r; s] is contained in P. Furthermore, since r and s are on the boundary of Os-hull(fp; qg), there is one arm of O-k[r; s] that contains points outside Os-hull(fp; qg).10 Now r and s belong to one edge of @ Os -hull(fp; qg) and, therefore, there is one arm S of Os-hull(fp; qg) that contains both r and s. By Corollary 5.2 we can replace the part of 10

Note that since O is closed and (e) 62 O, O-k[r; s] does not equal pq.

31

p

S

r

0 S0 s e O-k[r; s]

S

q

Figure 17: An edge that is not contained in @ P. S from r to s with one arm of O-k[r; s] and the resulting new staircase S 0 will remain O-convex. But since one arm of O-k[r; s] is not contained in Os-hull(fr; sg) we obtain an O-staircase that is not contained in Os-hull(fp; qg) which contradicts the de nition of Os-hull(fp; qg). Note that this proof also carries through if jOj = 1 since then we just can take any O-convex path from r to s that is not contained in Os-hull(fp; qg) which exists by our assumption on e. 2

The main result of this section is stated in the following theorem which shows the strong analogy between O-visibility and Os-visibility. Theorem 5.9 Let P be a polygon, p and q two points in P, and O a set of orientations. We have that p Os-sees q if and only if p fgs-sees q, for all  2 O. Proof: The \only if" direction of the claim is obvious since any O-staircase from p to q is fg-convex, for all  2 O. So we turn immediately to the \if" direction. If (pq) 2 O, then there is a  2 O, with (pq) = . Since p fgs-sees q, we have that pq  P and, hence, p Os-sees q. So suppose that (pq) 62 O. W.l.o.g. we assume that the maximal O-free range of (pq) is (0 ; 90) and that p is to the left and below q. By assumption we have that f0gs-hull(fp; qg) and f90gs-hull(fp; qg) are both non-empty. Consider the leftmost arm S1 of f0gs-hull(fp; qg) and the topmost arm S2 of f90gs-hull(fp; qg). By Theorem 5.5.1 of [Raw87] and Lemma 5.5 S1 is a f0g-staircase and S2 is a f90g-staircase from p to q. We claim that the lower enveloppe S of S1 [ S2 is a f0; 90g-staircase from p to q. Note that S is the part of S1 [ S2 that is contained in f0gs-hull(fp; qg) \ f90gs-hull(fp; qg). For illustration see Figure 18a. Let Q1 ; . . . ; Qm be the connected components of S1 \ S2 numbered according to the sequence in which they appear on S1 . We denote the starting point of Qi by pi and the end point by qi . Since S1 is horizontally convex and S2 is vertically convex, Qi is f0; 90gconvex, for 1  i  m. More precisely, Qi consists of edges with orientation in [0; 90]. In the following we will show that we can connect qi to pi+1 with an edge of orientation 0 or 90. This immediately implies that the concatenation of the Qi with the connecting edges is a f0; 90g-staircase from p to q by Lemma 5.3. 32

q

S2 = S1 = S2 = lower enveloppe of

S1 [ S2

p

p

(a)

S10 S1

pi+1 S20 p00 p0

qi

(b)

Figure 18: The lower enveloppe of S1 [ S2 is f0; 90g-convex. Let S10 be the part of S1 from qi to pi+1 and S20 the part of S2 from qi to pi+1. Since qi occurs before pi+1 on S1, qi is below pi+1. If qi has the same height as pi+1, then the only f0g-staircase from qi to pi+1 is a horizontal line segment and we are done. So assume that pi+1 is strictly above qi . We claim that qi is to the left of pi+1 (refer to Figure 18b). To see this consider the vertical chord cpi+1 through pi+1 . cpi+1 splits P into two subpolygons one to the left of cpi+1 which we denote by Pl and the other to the right of cpi+1 which we denote by Pr . S2 is vertically convex and, hence, the part of S2 after pi+1 is contained in Pr . Since we assume qi to be to the right of pi+1, qi is also contained in Pr . On the other hand, p is in Pl since p occurs before pi+1 on S2 . This implies that S1 crosses cpi+1 in a point p0 below pi+1 that occurs before qi which contradicts the fact that S1 is the leftmost f0 g-staircase from p to q; for, we can substitute the part of cpi+1 from p0 to the next intersection point p00 of S1 above p0 for the part of S1 from p0 to p00 and the resulting path is still f0g-convex but contains points to the left of S1. Hence, qi is to the left of pi+1. In the same way as above we can prove that S10 does not cross the vertical chord cpi+1 through pi+1. Therefore, S10 and S20 are contained in Pl. Similarly, it can be shown that S20 does not cross the horizontal chord cqi through qi and which implies that S10 [ S20 is contained in the part Pa of P above cqi . Hence, Pi = Pl \ Pa contains both S10 and S20 . Let P be the part of the boundary of Pi from qi to pi+1 if we orient @ Pi counterclockwise. We claim that S10 does not intersect P . To see this assume the contrary. We consider the horizontal chord cx through a point x of S10 \P as illustrated in Figure 19a. . The part of S1 after x is contained in the part of Pi above cx, and, hence, the left intersection point 33

Pi

pi+1

S10 cx

Pl cpi+1

S10

pi+1

Pr

S20

P x

cqi

qi

P

Pa

(a)

(b)

Figure 19: S1 is to the left of P . of cx with @ Pi does not belong to P since the part of Pi above cx contains pi+1. Similarly, we have that S20 does not intersect P . Thus, C1 = S10  P , C2 = S20  P , and C3 = S10  S20 are simple closed curves. Since S10 , 0 S2 , and P start and end in the same points, either C1 contains S20 or C2 contains S10 or C3 contains P . Since P is a simple polygon and S10 and S20 are contained in P, P as part of the boundary of P cannot be contained in S10 S20 . So suppose w.l.o.g. that S2 is contained in C1 (see Figure 19b). Since an edge of S20 is non-vertical only if it is contained @ P, S20 contains no non-vertical edges. Therefore, S20 consists of a single vertical edge from qi to pi+1 . This implies that Q1  q1p2 Q2 


Qm?1  qm?1 pm Qm is a f0; 90g-staircase from p to q that is contained in P and, therefore, p Os-sees q. 2 Observe that the claim is false for polygons with holes. A counterexample is depicted in Figure 20. q can be reached from p with a f0g- and a f90g-staircase but there exists no f0 ; 90g-staircase from p to q that does not intersect the hole of P. The above theorem leads to following slightly more general corollary. Corollary 5.10 Let P be a polygon, p and q twoS points in P and Oi, i 2 I , a family of sets of orientations and O their union, i.e., O = i2I Oi. p Ois-sees q, for all i 2 I , if and only if p Os-sees q. Proof: S If p Ois-sees q, for all i 2 I , then we, in particular, have that p fgs-sees q, for all  2 Oi = O. Theorem 5.9 now implies that p Os-sees q. The reverse implication is obvious. 2 It is interesting to note that although the visibility structure of O-convexity and Osconvexity is essentially the same (an analog to Corollary 5.10 is contained in the proof of Lemma 4.1), the hull properties dier considerably. Recall that we have for the O-hull of 34

P q

p

Figure 20: p can not f0; 90gs-see q. two points p and q that

[

O-hull(fp; qg) = Oi -hull(fp; qg) i2I

and contrast this with the following corollary.

Corollary 5.11 Let P be a polygon and p and q two points in P. If Oi, i 2 I , is a family of sets of orientations and O is their union, then we have that \ Os-hull(fp; qg) = Ois-hull(fp; qg): i2I

Proof: Let I := Ti2I Ois-hull(fp; qg). Since every O-stairsegment is an Oi-stairsegment, for all i 2 I , we obviously have Os-hull(fp; qg)  I. Now let r be a point I. Since r 2 I, p Ois-sees r (in P), for all i 2 I . The prerequisites of Corollary 5.10 are satis ed, and therefore p Os-sees r. The same holds for q. Hence, there is an O-stairsegment Spr from p to r and an O-stairsegment Srq from r to T q. With Lemma 4.3.1 of [Raw87] it can be easily seen that O-k[p; q] = i Oi -k[p; q]. Since Ois-hull(fp; qg)  Oi -k[p; q], we have that r 2 O-k[p; q] and as in the proof of Lemma 5.5 Lemma 5.1 then implies that S = Spr  Srq is an O-stairsegment from p to q containing r. By De nition 5.3 we have that S  Os-hull(fp; qg) and, hence, r 2 Os-hull(fp; qg) which

proves the claim. 2 The main implication of Theorem 5.9 is the following corollary which is a restatement of Theorem 6.4.2 in [Raw87]. Unfortunately, the proof presented in [Raw87] is not correct. Corollary 5.12 If P is a simple polygon, Oi, i 2 I , a family of sets of orientations and O their union, then Os-kernel(P) = Ti Ois-kernel(P): 35

q0

S0 p0

@Q

q

Q

p

Figure 21: Any O-staircase from p to q belongs to Q. Proof: Let q be a point in P. If p is in Os-kernel(P), then p OTs-sees q; since every O-stairsegment is also anTOi-stairsegment, for all i 2 I , we have p 2 i Ois-kernel(P). On the other hand, if p is in i Ois-kernel(P), then p Ois-sees q, for all i 2 I and Corollary 5.10 implies that p Os-sees q. Hence, p 2 Os-kernel(P). 2

The above corollary will be very helpful in the next section where we are concerned with algorithms to compute Os-kernel(P). But before we turn to this problem we take a look at Os-skulls which correlate nicely to maximal, connected O-convex sets.

5.4

Os -Skulls

In order to use the notion of skull correctly we have to specify the semi-convex space we are looking at. So let P be a polygon. We de ne the family O-CP of Os-convex sets as the family of all sets C  P such that, for any two points p and q in C, we have that Os-hull(fp; qg)  C. An Os-skull Q of P is a maximal Os-convex subset of P. We denote the set of Os-skulls of P by Os-skulls(P). Recall that in Theorem 5.5.2 of [Raw87] it is shown that in a connected O-convex set any two points Os-see each other. Furthermore, note that an Os-convex set in a polygon P is necessarily O-convex and connected but the reverse is not true. But if we consider Os-skulls and the maximal O-convex connected subsets of a polygon P, we can show that the two notions coincide.

Lemma 5.13 If P is a polygon and Q a maximal, connected O-convex subset of P, then Q 2 Os-skulls(P). Proof: The proof is by contradiction. Let Q be a maximal, connected O-convex subset of P. Suppose there are two points p and q in Q such that Os-hull(fp; qg) is not contained in Q, i.e., there is an O-staircase S from p to q that is not contained in Q. Hence, S intersects @ Q at least twice. W.l.o.g. we assume that the maximal O-free range of pq is (0; 90) and that p is to the left and below q. Therefore, S is a f0; 90g-staircase. Let p0 be the rst cross point of S with @ Q and q0 be the second (see Figure 21). Since S is a f0; 90g-staircase, q0 is to the right and above p0. By Theorem 5.5.1 of [Raw87] 36

@ Q is also a f0; 90g-staircase from p0 to q0. If we replace the part of @ Q from p0 to q0 with the part S 0 of S from p0 to q0, then the new boundary still encloses an O-convex set Q0 by a similar argument as in the proof of Lemma 5.5. Since S 0 consists only of points outside Q, we have that Q0 properly contains Q which contradicts the maximality of Q. This concludes the proof. 2 If we denote the family of maximal connected O-convex sets of P by O-S (P), the above lemma states that O-S (P)  Os-skulls(P). We want to show that actually equality holds. So let Q be a set in Os-skulls(P). Clearly, Q is connected and O-convex. Our rst claim is that there exists a set Q0 2 O-S (P) containing Q. To seeSthis consider a nested chain N of connected O-convex sets containing Q and let C = N . C is clearly connected and O-convex. Hence, by Zorn's Lemma there exists a maximal connected O-convex set Q0 2 O-S (P) containing Q. By the above lemma we have Q0 2 Os-skulls(P). The de nition of a skull then implies that Q0 = Q and, hence, Q 2 O-S (P). Note that the semi-convex space (P; O-CP ) is induced by the norm k
kO . By Lemma 5.4 and Theorem 5.3 of [SW91] we have that (P; O-CP ) satis es the join condition which is stated in the following corollary for further reference. Corollary 5.14 If P is a simple polygon, then (P; O-CP ) satis es the join-condition. This, in particular, implies that we can apply the Kernel Theorem of [SW91], i.e., we have \ O-CP -kernel(P) = O-CP -skulls(P): Note that O-CP -skulls(P) is just another name for Os-skulls(P) by de nition. Lemma 5.13 now yields the connection to Os-visibility. Os-kernel(P) equals O-CP -kernel(P) since there is an O-stairsegment from p to q if and only if Os-hull(fp; qg) 6= ;. On the other hand, we have that the set of all maximal,Tconnected O-convex subsets of P equals Os-skulls(P). We conclude that Os-kernel(P) = O-S (P). Note that on both sides of the equation we have terms which are de ned just by making use of Os-visibility independently of the notion of Os-convexity. An immediate consequence of the above equation is the following corollary. Corollary 5.15 If P is a simple polygon, then the Os-kernel of P is Os-convex. In particular, Os-kernel(P) is a connected, O-convex set. The semi-convex space (P; O-CP ) obviously models the aspects of Os-visibility we are interested in quite adequately. Hence, we feel that other properties of this interesting example of a semi-convex space should be explored. As an example we present another corollary concerning Os-skulls which shows that Os-skulls can be obtained by intersection from Ois-skulls. Corollary 5.16 Let P be a polygon in the plane, Oi set of orientations, for i 2 I , and O their Tunion. If Q 2 Os -skulls(P) and Qi 2 Ois -skulls(P) with Q  Qi , for all i 2 I , then Q = i Qi . 37

Proof: We clearly have that Q  Ti Qi . So now let p be a point in Ti Qi . Since T p 2 i Qi  Q, p Ois-sees all points in Q, for all i 2 I . Corollary 5.10 implies that p Ossees all points in Q. Hence, the Os-join of p and Q is non-empty and contained in P. Since we have that the Os-join of a point and an Os-convex set is Os-convex by Corollary 5.14, the maximality of Q implies that p is contained in Q. 2

6 Computing the Os-Kernel of a Polygon In the following we turn to the computational aspects of Os-visibility. In particular, we are interested in nding the Os-kernel of a polygon. In the rst part of this section we are concerned with simple polygons. With Corollary 5.12 as the analogon to the Intersection Lemma of Section 4 the computation of the Os-kernel is reduced to the task of computing the fgs-kernel, for all  2 O, and of intersecting the resulting kernels. With the help of a simple observation we show how to obtain an algorithm that runs in time O(n log jOj), for nite O. The second part then deals with multiply connected polygons. Unfortunately, we lose many of the more pleasing properties that hold for the Os-kernel of simple polygons. This leads to the notion of the external Os-kernel of a polygon. The last part is then concerned with the relationship between the O- and Os-kernels where we show that the Os-kernel is a component of the O-kernel.

6.1 The Os-kernel for Simple Polygons

As we noted above Corollary 5.12 leaves us with the computation of the Os-kernel in the case jOj = 1. Of course, we can again assume that O = f0g. In the following we give a characterization the f0gs-kernel of a polygon. Although the characterization turns out to be quite a bit simpler than the one for the f0g-kernel of a polygon, the proofs get more involved. In particular, we need the following very basic lemma about Jordan curves which relates re ex and convex minima on the curve.

Lemma 6.1 Let J be a Jordan curve. If p and q are two (dierent) convex minima (maxima) of J , then there is at least one re ex maximum (minimum) of J above (below) them.

Proof: We only prove the lemma in the case that p and q are two convex minima of J . We assume that J is oriented in such a way that the interior of J is locally to the left of J . W.l.o.g. suppose that p is below q (see Figure 22a for illustration). We prove the claim by contradiction. So from now on suppose that all maxima of J above q are convex. We denote the part of J from q to p by Q. Let hq be the horizontal line through q and cq be the connected of component of J \ hq that contains q (see Figure 22b). Our rst claim is that the \ rst" intersection point of Q and hq is to the left of cq . By the rst intersection point we mean the rst point of Q that intersects hq if we follow the orientation of Q from q to p. To see this note that p is below q and, hence, Q intersects hq . 38

J J p (a)

Q

cq hq

q

q

rm

m

E1

rl

p (b)

C

Qm

q

E2

hq

q1

(c)

Figure 22: There is a re ex maximum between p and q. Suppose the rst intersection point q1 of Q with hq is to the right of cq . Since cq is a convex minimum, there are points of Q in any neighbourhood of cq that are above hq . Therefore, there is a maximum in the part Q1 of Q from q to q1 which is above q and q1. Let m be the leftmost of the highest maxima of Q1 . By our assumption on J , m is convex. Consider the biin nite curve C that consists of the ray rl towards the left starting in q, the part Qm of Q1 from q to m, and the ray rm towards the left in m as shown in Figure 22c. By our choice of m and the assumption that Q does not intersect rl , C is simple and divides the plane into two connected components E1 and E2. Let E1 be the component that is locally above rl and E2 the component that is locally below rl . By de nition q1 does not belong to Qm or rm since m is above q1 and Qm is only a part of Q1 . Furthermore, q1 is to the right of q and, thus, not part of rl . By our choice of q1, Qm does not intersect hq . This implies that Qm stays above hq and it can be easily seen that there are paths that connect q1 with points locally below rl without crossing C . On the other hand, the points locally above rl can be connected with a path that does not cross C to the points locally below rm and, thus, also to points locally below m. Since m is convex, there are points of Q1 in the part from m to q1 that are locally below rm , i.e., in E1. Hence, Q1 intersects points in E1 and E2 and, therefore, crosses C . By our choice of m, Q1 does not cross rm and, furthermore, since J is simple Q1 does not cross Qm . This implies that Q1 intersects rl in contradiction to our choice of q1 . So from now on we can assume that the rst intersection point q1 of Q with hq is to the left of cq . As above we denote the part of Q from q to q1 by Q1 . Note that Q1 stays above hq . We consider the Jordan curve J1 which consists of Q1 concatenated with the line segment q1q from q1 to q (see Figure 23a). Since Q1 stays above hq , the interior of J1 is locally above q1 q and, hence, also locally above cq . Let P be the part of J q1 to q. Since cq is a convex minimum, there are points of J before q, i.e., points of P , that are locally above cq . Therefore, P intersects int(J1). Let p1 be the last intersection point of P with q1q and P1 the part of P from p1 to q. Since the points of P1 that are locally before q belong to int(J1), P1 is contained in int(J1 ) and there is maximum of P1 contained in int(J1 ). We choose m1 to the rightmost of the highest maxima of P1 . By our assumption 39

m

J1

int(J1 )

P1

P1

q1

P

p1

q

J2

hq

m1

P1

q2

J2

int(J2 )

Q01

P10 p1

(a)

rm1

q (b)

Figure 23: m1 is not convex. on J , m1 is convex. We will show that this leads to a contradiction. We now consider the ray rm1 towards the right starting in m1 . Since m1 is contained in int(J1 ), rm1 intersects J1. Let q2 be the rst intersection point of rm1 with Q1 . We denote the part of P1 from m1 to q by P10 and the part of Q1 from q to q2 by Q01 . J2 = P10 Q01  q2m1 is a Jordan curve whose interior is locally below q2m1 (see Figure 23b). To see this just note the interior of J2 is locally above q since J2 is above hq and by following P10 we can connect the points locally below q2 q with the points locally above q and, hence, these points also belong to int(J2 ). Since m1 is convex, the part of P1 before m1 intersects int(J2). Furthermore we have that p1 does not belong to int(J2 ) since otherwise there are at least two intersection components of J2 with hq , one to the left and one to the right of p1 , which contradicts the fact that P10 and Q01 intersect hq only in cq . Hence, P1 intersects points in the interior and exterior of J2 and, thus, crosses J2. Furthermore, since J is simple, it intersects q2 m1 . But we chose m1 to be one of the highest maxima of P1 in the interior of J2 which contradicts the fact that P1 crosses q2m1 . This yields the desired contradiction. Hence, one of the maxima above p and q is re ex. 2 Of course, we get the same result if we exchange convex and re ex by reverting the orientation of the curve. Now we turn to the characterization of the f0 gs-kernel of a polygon. As we indicated above the conditions can be easily tested which results in a simple algorithm. This result was rst stated in [Raw87, Lemma 6.4.1]. Unfortunately, the proof contains only the easy \only if" part which we also give for completeness. Theorem 6.2 If P is a simple polygon, then p 2 f0gs-kernel(P) if and only if 1. p is above every re ex maximum of P and 2. p is below every re ex minimum of P. 40

@P

rq p3 p5 @P

N

q

p1 p2

P1

C

p4

p

hp

Figure 24: The construction of a f0g-convex path from p to q. Proof: As mentioned above, the only if part of the claim is very easy to see. W.l.o.g.

assume that p is below a re ex maximum and let q be some point of the same height as p on the other side of the maximum. Any curve that connects p and q cannot be f0g-convex. Now let p be above every re ex maximum and below every re ex minimum and q some point in P. W.l.o.g. assume that q is above p. We construct a f0g-convex curve C that connects p and q (for illustration refer to Figure 24). To do this let rq be the horizontal ray towards the left starting in q. We follow rq until we reach the exterior of P. Let the cross point with the boundary of P be p1. Consider a suciently small neighbourhood N of p1 (see Figure 24). Since we cross @ P, there is some part of @ P \ N below rq . We follow the part of @ P below rq until we encounter a re ex minimum or intersect the maximal horizontal line segment hp through p in p2 . In the latter is the case we just follow hp to p which is always possible since hp is a chord in P. Otherwise, we continue from p2 to the left until we again intersect @ P and continue the process. We claim that the resulting curve C nally reaches p and that C is f0g-convex. To see this note that hp splits the polygon P into a number of components. Let P1 be the component that contains q. By the construction of C we do not cross hp and, hence, C is contained in P1 [ hp . We rst show that C is f0g-convex. For assume the contrary. Then, there is a horizontal line h that intersects C in two disconnected components. Clearly, both components belong to P1 since the part of C that is contained in hp is connected by construction. There is a minimum m in the part of C between the two disconnected components. By the construction of C m is a convex minimum that belongs to @ P. If m is on hp, then C does not continue after p which contradicts the fact that h intersects C in two disconnected components. Hence, suppose that m is not on hp. Note that hp contains a convex minimum of P1. In the following we show that there is no other convex minimum in P1. This immediately yields the desired contradiction since we already noted above that C is contained in P1 [ hp. So assume there is a convex minimum m in P1 that is not on hp. Lemma 6.1 yields that there is a re ex maximum r 41

h1 h2

.

P

v1

v2

S (0 )

f0 g-kernel(P) Figure 25: Os-kernel(P) is the intersection of S (0) and P. between hp and m and above both. Clearly, r belongs to @ P and, therefore, this contradicts Conditions 1 of the claim. Hence, there can be no line that intersects C in two components and f0g-convex. Since C is f0g-convex, i.e., monotone and we follow each edge to its end point, we encounter each edge of P at most once. Hence, the procedure has to stop after a nite number of steps. Since we stop the construction only if we reach p, C is a f0g-convex path from q to p as claimed. 2 By the above theorem we are immediately led to an algorithm to compute  f0 gs-kernel(P). We just have to nd the lowest re ex minimum vl and highest re ex maximum vh and then output the left and right parts of the boundary of P with a y-range in the interval [vh ; vl]. All of this can be done in linear time. For more details see p. 75/76 of [Raw87]. We summarize our results in the following theorem. Theorem 6.3 Let P be a simple polygon, then f0gs-kernel(P) can be computed in time linear in the number of vertices.

6.2 A Fast Algorithm for the Os-Kernel of a Polygon

Above we presented an algorithm which computes the fgs-kernel of a polygon P, for each  2 O, and then intersects the kernels obtained to yield Os-kernel(P), for nite O. Since we have to apply the fgs-kernel subroutine jOj-times, the algorithm needs at least time (n
jOj) to compute Os-kernel(P) even if we do not take the time to intersect the fgs-kernels of P into account. In the following we will give an algorithm to compute the Os-kernel of a simple polygon P that only takes O(n log jOj) time if O is given as a sorted set. In order to develop an algorithm that obtains the above time bound, we have to view the f0gs-kernel a little dierently. Let v1 be the lowest re ex minimum and v2 the highest re ex maximum of P. If we denote the horizontal line through vi by hi, for i = 1; 2, then Theorem 6.2 implies that f0gs-kernel(P) is the intersection of the slab S (0) between h1 and h2 with P (see Figure 25). Hence, if we are given the slabs S () that lie between the -oriented lines through the lowest re ex -minimum and the highest re ex -maximum, for all  2 O, 42

then we have that

Os-kernel(P) = =

\ 02O

(S () \ P)

@\

2O

1 S ()A \ P:

In the following we denote T2O S () by SO (P). Since S () is the intersection of two -oriented halfplanes, SO (P) is strong-O-convex by Theorem 6.1.3 of [Raw87]. This, in particular, implies that it is convex and consists of at most 2jOj edges. Hence, SO (P) \ P can be computed in time O(n log jOj) by testing, for each edge e of P, if e intersects an edge of @SO (P). This yields the intersection points of @SO (P) with @ P and enables us to compute the vertices of SO (P) \ P in additional linear time. So we only have to show how to compute SO (P) in time O(n log jOj). In order to do so we introduce the following notation. Given a point p in the plane we denote the (closed) halfplane to the left of the -oriented line through p by h+(p; ). For a given orientation , we denote the highest re ex -maximum of P by v . The rst observation we make is that S () is the is the intersection of h+(v ; ) with h+(v?1 ; ?1) since the lowest re ex -minimum is, of course, the highest re ex ?1maximum. Since we always have that ?1 is in O if  2 O, we only have to nd the intersection of the -oriented halfplanes above the re ex -maxima, taken over all  2 O. Therefore, T h+(v if; v). is the highest re ex -maximum of P, for each  2 O, we want to compute  2O The idea of the algorithm is to scan the boundary of P counterclockwise and, thus, successively process the vertices of P. During the scan we keep a list L of the highest re ex -maxima we have encountered so far, for each  2 O. When we process the next re ex vertex v, we check, for which orientations  2 O, v is a re ex a -maximum. We replace those -maxima v for which v is higher than v (in the -coordinate system) by v. The problem is that there may be (jOj) of orientations  which have v replaced by v. Hence, the algorithm may still take time (njOj) if there are (n) re ex vertices, each being a re ex -maximum for (jOj) orientations . The important observation to reduce the running time of the algorithm is that we are only interested in the intersection of the halfplanes h+ (v ; ), with  2 O. If we have that the so far encountered highest re ex -maximum v is lower than v, for a subset O0 of orientations  in O, the following lemma shows that there exist two orientations 1 and 2 such that h+(v; 1) \ h+ (v; 2) already equals T2O0 h+ (v; ). If we can nd 1 and 2 in logarithmic time, we only have to replace v1 and v2 in L since only these may in uence the nal intersection. If L is stored as a dictionary, then replacing v1 and v2 can be achieved in time O(log jOj). Hence, it only has to be shown that 1 and 2 exist and can be found in time O(log jOj). Lemma 6.4 If P is a polygon, O a nite set of orientations, and v a re ex vertex of P, 43

l

'1 l

e1

v

e2 '2

(l) 2 ['2 ; '1 ]

e1

'?1 1

'?1 2 v

e2 '2

(l) 2 ['?1 1 ; '2 ]

'?1 1

e1

v

e2

l

?1 (l) 2 ['?1 2 ; '2 ]

Figure 26: v is a re ex -maximum only for the orientations  2 ['2 ; '1]. then there are two orientations 1 and 2 in O such that

\ h+(v; 1) \ h+(v; 2) = h+ (v; ) 

if the intersection on the right side is taken over all  2 O for which v is a re ex maximum. Furthermore, 1 and 2 can be found in time O(log jOj).

Proof: Let e1 be the edge that is (counterclockwise) before v and e2 the edge after v.

To see the claim we denote the orientation of ei by 'i. Note that v is a re ex -maximum only for the orientations  that are in the interval ['2 ; '1].11 This can be shown as follows ?1 (see Figure 26). If  2 ['1 ; '?1 2 ] or  2 ['1 ; '2 ], then the start point of e1 is on the other side of the line l(v; ) through v with orientation  than the end point of e2 . Hence, v is no ?1 -extremum. If  is in ['?1 2 ; '1 ], then e1 and e2 are above l(v; ) and v is no -maximum. Only if  2 ('2 ; '1), then both of e1 and e2 are completely below l(v; ). There is a little problem if '2 is in O, since then it depends on the edge after e2 if v is a '2 -maximum (refer to Figure 27). To see this suppose that '2 2 O and let e3 be the edge after e2 . If (e3) < '2, then e2 is a '2-maximum since '2 < '1 and if (e3) > '2, then e2 merely contains some '2-saddlepoints. Since we only have to spend a constant amount of time to nd the edge after e2, we will not consider this special case anymore. Clearly, it can be checked in logarithmic time if '2 is in O if O is sorted. So let 1 be the last orientation in O that is (counterclockwise) before '1 and 2 be the rst orientation in O after '2 if they exist (see Figure 27). We denote the interval ('2; '1) by I . If there is one  in O with  2 I , then, obviously, we have that 1 and 2 both exist. Hence, if they are not de ned, then there is no  in O for which v is a -maximum and we are done. So assume that 1 ; 2 2 I and that they are dierent (if 1 = 2 , then there is only one element in O \ I and the lemma is trivially true.) Consider the wedge w that is formed by h+(v; 1 ) \ h+(v; 2). It can easily be seen that w is contained in any halfplane h+(v; ) if  2 [2 ; 1 ]. Since we have that O\ I = O\ [2 ; 1 ] 11

Since v is re ex, we have that (e1 ) > (e2 ).

44

h+ (v; 1 ) \ h+ (v; 2 ) '1 1 (e3 ) > '2

v e2 e1

O

v

(e3 ) < '2

e1 e2

(a)

(b)

'2

2

Figure 27: How to nd 1 and 2. by our choice of 1 and 2 and the orientations in O \ I are the only ones for which v is a -maximum, we have that

h+(v; 1 ) \ h+(v; 2 ) =

\

2O\[1 ;2 ]

h+(v; ) =

\

2O\I

h+(v; ):

Hence, it suces to just look at v1 and v2 and possibly replace them with v. Note that if v1 is higher than v in the 1 -coordinate system, i.e., if we do not replace v1 with v, then h+(v1 ; 1 )  h+ (v; 1), and, hence, h+(v1 ; 1 ) \ h+ (v; 2) is still contained in each h+(v; ), with  2 [2 ; 1 ]. If O is given as a sorted array, it is easy to see that 1 and 2 can be found in time O(log jOj) given '1 and '2. 2 So the scanning algorithm computes a number of vertices v such that

\

2O

h+(v ; ) \ P

yields the Os-kernel of P. Above we have shown that we can compute the vertices v in time O(n log jOj). Observe that there are at most jOj such vertices, one for each orientation  2 O. Furthermore, note that the above computation yields the vertices v sorted according to . In Appendix 7 we show how to compute the intersection of halfplanes that are sorted according in time linear in the number of halfspaces. Hence, the intersection T h+(v to; )slope can be computed in time O(jOj). This proves the following theorem.  2O Theorem 6.5 The Os-kernel of a simple polygon with n vertices can be computed in time O(n log jOj + jOj), for nite O, given O(jOj log jOj) preprocessing time to sort O. 45

There are two questions in connection with the above result. (i) Is it possible to compute the Os-kernel of polygon within the same time bounds if O consists of a nite number of intervals, i.e., the algorithm should have a running time of O(n log r) if O consists of r intervals? (ii) Is there an algorithm that computes the Os-kernel faster or can a lower bound be proven? As to the rst question we conjecture that this is possible using an approach that is similar to the kernel algorithm of Lee and Preparata [LP79], i.e., processing the vertices in a scan as in our algorithm and keeping track of a suitable representation of the Os-kernel. For the second question the situation is less clear. On the one hand, a lower bound of (n log jOj) on nding all vertices which are a -maximum, for some  in O, is easy to show (see Appendix 8); on the other hand, there is a kernel algorithm by Cole and Goodrich [CG88] which could possibly be adapted to yield an O(n + jOj) time algorithm for the computation of Os-kernel(P).

6.3 The Os-Kernel in the General Setting

If we allow a polygon to have holes, the situation changes considerably. The Os-kernel(P) is not necessarily connected or O-convex anymore as shown in Figure 28. Furthermore, we lose the Intersection Lemma as Figure 29 illustrates. Since the algorithm to compute the Os-kernel of a simple polygon P and nite O is based on Corollary 5.12 for which there is no analog if P is multiply connected, we need a dierent approach to be able to compute the Os-kernel for multiply connected polygons. Holes can be viewed as polygons themselves. But instead of being able to see their inside we are now concerned with their external visibility. This gives rise to the de nition of an external kernel of a polygon. De nition 6.1 Let P be a polygon in the plane and O some set of orientations. We de ne the external kernel of P as the set of all points that belong to Os-kernel(IE 2 n int(P)) and denote it by Os-kernelext (P). It is not very surprising that the Os-kernel of a multiply connected polygon P is the intersection of the Os-kernel of the enclosing polygon of P intersected with the external kernels of the holes of P. This is shown in the following lemma. Lemma 6.6 If O is a set of orientations and P a multiply connected simple polygon with enclosing polygon Q and holes H1 ; . . . ; Hm, then m \

Os-kernel(P) = Os-kernel(Q) \ Os-kernelext(Hi): i=1

Proof: We denote the intersection of Os-kernel(Q) with the external kernels of the holes Hi, 1  i  k, by K. It can be easily seen that Os-kernel(P)  K so we concentrate on 46

P

= f0 ; 90 gs -kernel(P)

= Holes of P

Figure 28: The Os-kernel of P is disconnected and not O-convex.

f0 gs-kernel(P) \ f90 gs-kernel(P) 6 f0 ; 90 gs -kernel(P) = ; = Figure 29: The f0; 90gs-kernel is not the intersection of the f0gs-kernel and the f90gsf0 gs-kernel(P)

f90 gs -kernel(P)

kernel.

47

q

P p

p1 pa pb (a)

p2

P1

q2 bottom

q4 p

q5

P2

@P

@P

e

q1

e0

e1 e1 e0

q3 (b)

(c)

Figure 30: The construction of an O-convex path from q to p. proving the reverse inclusion. For this, let p be a point in K and q a point in the polygon P. We have to show that there is an O-convex path from p to q. If (pq) 2 O, then neither the holes nor Q intersect pq and, hence, p O-sees q in P. Therefore, suppose that (pq) 62 O. We rst treat the case that jOj  2. So, w.l.o.g. we assume that the maximal O-free range of p and q is (0 ; 90). Hence, we have to construct a f0; 90g-convex stairsegment in P that connects p and q. For brevity, we call the four line segments making up f0; 90g-k[p; q] top, bottom, left, and right with the obvious meaning. If some part of the boundary of P enters f0; 90g-k[p; q] through the lower arm of f0; 90g-k[p; q], it has to leave through it again since otherwise q would be cut o from p and p cannot belong to either the kernel of Q or the external kernel of some hole Hi. The rst observation we make is that @ P crosses bottom and left at most once. For suppose that there is a part of the boundary of P that crosses, say bottom, twice. Starting from p let p1 resp. p2 be two points that belong to the rst resp. second component of @ P \ bottom as illustrated in Figure 30a. Note that we can assume that p1 and p2 both belong to either the boundary of the same hole or to @ Q for the components of ext(P) just correspond to ext(Q) or int(Hi). W.l.o.g. suppose that p1 and p2 belong to @ Hi . Since @ Hi crosses bottom in p1, there are points pa and pb 2 @ Hi locally above and below p1. Consider a path P from p to p2 . P passes either above pa or below pb. Since its start and end point lie on bottom, there is a line parallel to bottom that intersects P twice; hence, P is not f0; 90g-convex which contradicts the fact that p belongs to Os-kernelext(Hi). We now construct a f0; 90g-convex path S from q to p. The intuitive idea is to follow a path like water would ow from q to p. Let q1 be the rst cross point of right and @ P (as seen from q). If q1 does not exist, we reach bottom and continue horizontally to p. Since there is at most one cross point of @ P with bottom, this is always possible. So suppose q1 exists and belongs to edge e (see Figure 30b). Since the interior of P is locally to the left of e, e is oriented from left to right. The slope of e is positive, i.e., (e) 2 (0; 90) since otherwise p does not f0; 90gs-see q1. Note that this is true for any edge of @ P in 48

f0; 90g-k[p; q] that is oriented from left to right, i.e., which has the interior of P locally above it. We continue downward, i.e., contrary to the orientation of e, along @ P until we reach a f90g-re ex, locally leftmost vertex q2. The rst observation we make is that the part P1 of @ P from q1 to q2 consists only of edges that oriented from left to right and that have an orientation in [0; 90]. Let e1 be the last edge of P1 with (e1 ) 2 [0; 90]. Note that there is at least one such edge since (e) 2 [0; 90]. Let e0 be the edge before e1 on @ P. If (e0) 2 (180 + (e1 ); 360), then e0 cannot be f0; 90g-seen by p and if (e0) 2 (90; 180 + (e1 )) then the common vertex of e0 and e1 is a f90g-re ex, locally leftmost vertex (see Figure 30c). Lemma 5.3 yields that P1 is f0; 90g-convex. Secondly, we note that P1 does not cross left. For assume the contrary. Since the interior of P is locally above P1 and p is below the cross point of P1 with left, there is a second intersection point of @ P with left which contradicts our above observation. From q2 we again drop vertically until we reach another part of @ P in q3 where q3 is the rst cross point of the vertical ray downward through q2 and @ P in f0; 90g-k[p; q]. We continue in this way until we nally reach bottom. Note that we nally have to arrive either at bottom or at left since we always move down or to the left and @ P consists only of a nite number of edges. From there on we complete the path horizontally or vertically to p. This is always possible since @ P crosses bottom and left only once. With the same arguments as above we can show that the parts Pi of @ P from q2i?1 to q2i , for i = 1; . . . ; m|if we have repeated the procedure m times|are f0 ; 90g-stairsegments. By Lemma 5.1 we have that the concatenation of the Pi with the vertical line segments connecting them is an O-stairsegment S . By construction S is contained in P. Hence, p Os-sees q. If jOj = 1, say O = f90g, a similar approach works. To see this let ei be again the last edge of Pi with orientation in [?90; +90] and e0 the edge before e1 (@ P). If (e0) 2 (180 +(e1); 270), then e0 is not f0g-seen by p and if (e0 ) 2 (90; 180 +(e1)), then e1 \ e0 is a leftmost vertex. 2 s The above lemma provides the tools to compute the O -kernel of a multiply connected polygon. Since we know how to compute the Os-kernel of the enclosing polygon, we only have to show how to compute the external Os-kernel of a polygon which is done in the following section.

6.4 The External Kernel of a Polygon

We will now look at the problem of computing the external Os-kernel of a simple polygon. Recall that Figure 29 implies that the external kernel cannot be reduced to the calculation of the kernel for one direction. So we have to develop an algorithm that directly computes Os-kernelext(P). This time we allow the set of orientations O to consist of a nite number r of closed intervals (or ranges) [1 ; 1]; . . . ; [r ; r ]. We call O interval nite in this case. Recall that an oriented tangent t of P is an oriented line such that t intersects @ P and that P is completely contained in h+ (t). Obviously, there is only one tangent to P, for any given  2 [0 ; 360). We denote it by t(). The halfplane to the left of t() is denoted by 49

u2

e1

t2 h2 = h? (2?1 )

P

u1 e2

1?1

P t(1 )

t1 h1 \ h2

t(1?1 )

t(2?1 )

h1 = h? (1 )

t(2)

O  [90 ; 180 ] 2 O-free 2?1

1

a 1 -maximum

a 2 -minimum

Figure 31: The external kernel of a polygon h+() and, similarly, the halfplane to the right by h?(). We start with a simple observation (see also Lemma 6.4.2 of [Raw87]). Observation 6.7 No point which lies between a pair of tangents to P that are parallel to some  2 O belongs to the external Os-kernel of P. This is an immediate consequence of Lemma 4.9 since Os-visibility implies O-visibility. We now turn to characterizing the components of the external kernel of a simple polygon P. Lemma 6.8 Let P be a polygon in the plane and O be a set of orientations with jOj  2. If 1 and 2 are two adjacent12 orientations in O, with 1  2 , such that 1. e1 = t(1?1 ) \ P and e2 = t(2 ) \ P are two (maybe degenerate) edges of P that meet in one vertex of P, and 2. P = @ P n (e1 [ e2 ) contains neither a local convex 1 -maximum nor a local convex 2 -minimum,13 then h? (1 ) \ h? (2?1 ) belongs to Os-kernelext (P). Proof: W.l.o.g. we assume that f1 ; 2g = f0; 90g. The situation is illustrated in Figure 31. Let h1 be the halfplane h?(1 ) = h?(0) and h2 the halfplane h? (1?1) = h?(270). For brevity, we denote t(0 ) by t1 , t(180) by t2 , t(90) by u1, and t(270) by u2. We have that t1 is below t2 and u1 is to the right of u2 (in the plane, not w.r.t. to their orientations). h1 \ h2 is below and to the left of P. First suppose there is some point p 2 h1 \ h2 that sees IE 2 n int(P). We have to show that Conditions 1 and 2 hold. 12 13

Note that adjacency, in particular, implies that (1 ; 2 ) is O-free. Refer to page 16 for the de nition of (local) -extremum.

50

m

p1

t0

S1 p

P

P0 p 2

p2

t2

qN C

p1 @P

S2

Figure 32: If p 2 f0gs-kernel(P), then P contains no f0g-maximum and t(1?1 ) \ @ P is connected.

1. Assume there is a convex f0g-maximum m in P . (The argument is analogous for a convex f90g-minimum.) Hence, we can choose some line t0 parallel to t1 such that t0 intersects P slightly below m in two points p1 and p2 , p1 to the left of p2, and such that the part P 0 of P from p1 to p2 contains m as its only f0g-maximum (see Figure 32). Let S1 and S2 be two O-stairsegments from p to p1 and p2, respectively which exist by our assumption. Since S1 and S2 are f0; 90g-convex, they do not cross t0 . By our choice of t0 the concatenation of P 0 with S1 and S2 forms a closed curve whose interior contains P. This holds since S1 and S2 are completely contained in the exterior of P and @ P is simple. Furthermore, the only part above t0 is P 0 . Hence, m is a global f0g-maximum of @ P and is intersected by t2 which contradicts the assumption that m belongs to P . 2. We now show that t2 \ @ P is connected and contains t2 \ u1 (The proof that u1 \ @ P is connected is, of course, analogous.) So let p1 and p2 be two points in t2 \ @ P such that p1 occurs before p2 on t2. We will show that p1 and p2 belong to the same connected component of t2 \ @ P. For assume the contrary. Then, there is a point q in the part of @ P from p1 to p2 that is below t2 . We consider the curve C consisting of the part of t2 to the right of p1 and the part of @ P from p1 to p2 and the part of t2 to the left of p2 . C clearly is a biin nite curve. W.l.o.g. we assume that C is simple and, hence, divides the plane into two components one of which contains p and the other of which contains the points exterior to P in a suciently small neighbourhood N of q. Therefore, any path P that connects p to q and that is contained in the exterior of P intersects t2 twice. Since p and q are below t2 , this implies that P is not f0g-convex and p is not in Os-kernelext(P). That u1 \ t2 belongs to @ P can be seen by employing an argument similar to the one above only that now p1 is chosen in u1 \ @ P and p2 in t2 \ @ P. We then consider the biin nite curve consisting of the part of u1 below p1, the part of @ P from p1 to p2 51

u2 t(2?1 ) = u2

u1

q t(1?1 )

t2

h2 = h? (2?1 ) p

h1 = h? (1 )

q

t(1 ) = t1

t(1 ) p

(a)

t(2?1 )

(b)

t(2 )

Figure 33: q is in h1 [ h2 or in h?(1?1) [ h?(2 ). and the part of t2 to the left of p1 . This concludes the only if direction of the claim. Now suppose that Conditions 1 and 2 hold. We will show that then every point in h1 \ h2 can see IE 2 n int(P). To this end let p 2 h1 \ h2 and q 2 IE 2 n int(P). We distinguish several cases. Case 1 q is in h1 [ h2 (see Figure 33a). Since p and q are on the same side of t1 or u1 and there are no obstacles in these half-planes, we have pq  IE 2 n int(P). We have treated the case that q is to the left of or below p. Hence, from now on we can assume that q is to the right and above p which implies that the maximal O-free range of (pq) is (0; 90). Therefore, it suces to construct a f0; 90g-convex stairsegment from p to q. Case 2 q is in h?(1?1 ) [ h?(2 ) and q is to the right and above p (see Figure 33b). We just consider the case that q is in h? (1?1). Starting from q vertically downward there is no obstruction until we reach t2 . We follow t2 to the left till we hit the x-coordinate of p. This point on t2 must be to the left of u2 (since p is to the left of u2) and, hence we can connect it to p by another vertical segment. Obviously, the vertical segments do not share common y-coordinates (apart from the one point on t2 ) and, hence, the path constructed is f0; 90g-convex. Note that this case also covers q 2 e1 [ e2 as a degeneracy since, then, the rst vertical segment is unnecessary. Case 3 q is in R = h+(1 ) \ h+ (2) \ h+(1?1 ) \ h+ (2?1) but not in the interior of P. Consider the vertical ray r downward from q. We assume that r intersects @ P and denote the rst edge that r crosses by e1 . We claim that e1 is tilted downward (if we follow the counterclockwise orientation of e1 ), i.e., (e1) 2 [180; 270), as shown in Figure 34. To 52

m1

q e1

et1

es1

P1

P

et1 es

2

r

q e1

es1

e2 r

et

2

P2 m2

Figure 34: q is in the bounding box of P. see this suppose that the contrary holds and e1 is tilted upward. Since q is on the outside of P, it is above the line through e1 , i.e., the source vertex es1 of e1 is to the right of q and the target vertex et1 is to the left of q. Furthermore, r crosses @ P in a second edge e2 with es2 to the left of r and et2 to the right of r. Let P1 be the part of @ P from from es1 to es2 and P2 the part of @ P from from es2 to es1. Since et1 is above es1, there is a convex f0g-maximum m1 on P1 . Also since et2 is to the right of es2 there is a rightmost point m2 on P2. m2 is a convex f90g-minimum. By Condition 2 we have that m1 belongs to t2 \ @ P and m2 belongs to u1 \ @ P. Condition 1 implies that the part of @ P from m2 to m1 consists only of two edges, one having orientation 90 and the other having orientation 180 which contradicts the fact that (e1) 2 (90; 180). We are now in a position to construct a f0; 90g-convex stairsegment from p to q in very much the same fashion as in Lemma 6.6. The idea is again to follow a path like water would ow from p to q. Any edge of @ P we encounter is tilted downward to the left (to see this just choose some point immediately above the edge and the previous argument applies). If we reach some locally leftmost edge of @ P, we continue vertically downward until we either intersect @ P or t1 . Proceeding in this way we will reach t1 in a point to the right of or on u2 since the number of edges of P is nite. Since p is to the left of u2 and below t1 , we clearly can continue the f0; 90g-convex path constructed so far to p.

2

With Observation 6.7 and the above lemma we are able to decide if a point p that is contained in a slab between two parallel O-oriented lines or in a wedge enclosed by two tangents with adjacent orientations belongs to Os-kernelext(P) or not. Are there any other points in IE 2 n int(P) that do not fall into one of these two categories? The answer is no. To see this note that there are 2r O-Oriented tangents to P (see Figure 35). The arrangement of the 2r tangents divides the plane into O(r2) components. It is easy to show that a point p in the plane either lies between two parallel tangents or p lies within a wedge formed by two tangents with adjacent orientations. More precisely, if P is a simple polygon and p 2 ext(P), then we have that either p lies between a pair of O-oriented parallel tangents to P or there are two tangents t1 and t2 to P whose orientations are adjacent in O such that 53

Figure 35: The arrangement of 2r parallel tangents to P. h?(t1 ) \ h?(t2) contains p. To see this we imagine the slabs h+() \ h+ (?1), with  2 O, as being painted black (see Figure 35). Since these slabs have a common intersection|all of them contain P by de nition|the arrangement of the slabs is equivalent to an arrangement of -oriented lines, with  2 O, that intersect in one point (shrinking a slab does not create any new regions as long as the common intersection of all slabs remains simply connected). For the latter arrangement it is obvious that a point p either lies on a -oriented line or in a wedge that is formed by two lines whose orientations are adjacent in O. The above approach obviously only works if jOj  2. So it remains to look at s O -kernelext(P) if O = fg. As expected the situation simpli es considerably and we get the following characterization. This result is contained on p. 79/80 of [Raw87] but not proven there. Lemma 6.9 If P is a simple polyon in the plane, O = f0g, and p is above the highest point of P, then

p belongs to Os-kernelext(P) if and only if P contains no re ex maxima.

Proof: First suppose that p belongs to Os-kernelext(P) and assume that P has a re ex

maximum m. Consider a horizontal line l slightly below m. Since m is a re ex maximum l intersects @ P in at least two points p1 and p2 that belong to dierent components of @ P \ l. W.l.o.g. we can assume that the part of @ P from p1 to p2 does not intersect l in a third component. Suppose that p1 is to the left of p2. We choose a point q on @ P above and to the right of p1 . The path which consists of l to the left of p1 and to the right of p2 and the part of the boundary of P between p1 and p2 divides the plane into two disconnected components one of which contains p and the other one contains all the points exterior to P in a suciently small neighbourhood N of q. Any path P in the exterior of P that connects p and q and that does not cross @ P intersects l either to the left of p1 or to the right of p2. Since q is above l, P intersects l a second time and, hence, P is not f0g-convex. 54

Now suppose we have no re ex maximum in @ P. Let q be some point in IE 2 n int(P). We have to show that p f0g-sees q. If q is above the highest point or below the lowest point of P, it is easy to construct a f0g-convex stairsegment from p to q. So let q be between the two horizontal tangents to P and assume the horizontal ray to the right of q intersects P. We follow the boundary of P upward until we reach a maximum m. Since this maximum is convex and q is to the left of @ P, the points immediately to the right of m belong to IE 2 n int(P) and, hence, we can continue our path along the horizontal ray to the right of m until we hit the boundary of P again or are to the right of the rightmost point of P. Carrying on in this fashion leads us to a point either completely to the right of or completely above P from where we can easily complete the f0g-convex path to p started in q. 2

6.5 Computing the External Kernel

In order to develop an ecient algorithm to compute Os-kernelext(P) we need to bound the number of components that it can maximally consist of. The following lemma shows that the external Os-kernel of a polygon has only a constant number of components independent of the cardinality of O or the size of P. In the subsequently presented algorithm we will make crucial use of this fact. Lemma 6.10 Let P be a simple polygon. The maximum number of components Os-kernelext(P) consists of is (i) 2 if jOj = 1, (ii) 4 if jOj = 2, (iii) 3 if jOj = 3, and (iv) 2 if jOj > 3, and these bounds are tight. Furthermore, if jOj = 3 and Os-kernelext(P) consists of three components, then P is a triangle. Proof: To see that the external kernel of a polygon can have as many components as claimed refer to Figure 36. In the rst case the claim is an immediate consequence of Lemma 6.9 since either all points in the halfplane above t(180) belong to Os-kernelext(P) or none of it. The same, of course, holds for the halfplane below t(0 ). The proof of the second claim is as easy to see since the two O-oriented slabs that contain P divide the plane into at most four quaterplanes. By Lemma 6.8 a quaterplane belongs either completely to the external Os-kernel of P or not at all. Hence, Os-kernelext(P) consists of at most four components. Note that Condition 1 of Lemma 6.8 implies that, for each component w of s O -kernelext(P), there is a vertex v of conv(P) whose adjacent edges have an interior 55

P

P

P

O

O

O

P O

Figure 36: Lower bounds on the number of components of the external kernel of a polygon.

v

2 ?1 w  1

w = h? (1 ) \ h? (2?1 )

w



conv(P)

O-free

  =  2 ? 1

Figure 37: There is a vertex in conv(P) that has an interior angle of at most .

56

2

Q 1?1

T1

w

1 1  p

w?1  q 2 2 T2

conv(P)

1

2?1

Figure 38: 1 + 2   and 1 + 2   angle which is less or equal to the O-free range of the two tangents that enclose w (see Figure 37). We now turn to proving the third claim. The proof is by contradiction. So suppose that jOj = 3 and there are at least four components w1, w2 , w3, and w4 of Os-kernelext(P). Suppose the enclosing tangents of w1 ; . . . ; w4 span the O-free ranges (1; 1 ), (2 ; 2), (3; 3 ), and (4; 4). As we noted above there are four vertices vi of conv(P), 1  i  4, with an interior angle i that is less or equal to i ? i. Since jOj = 3, the closure of the four ranges cannot cover all of [0; 360), i.e., S4i=1 [i; i] 6= [0 ; 360). Now conv(P)Pis a convex, say m-gon, whose sum of interior angles is (m ? 2)
180. P 4 Since i=1 i  4i=1P i ? i < 360 the remaining (m ? 4) interior angles have to sum up to (m ? 2)
180 ? 4i=1 i > (m ? 2)
180 ? 360 = (m ? 4)
180 which contradicts the fact that all the interior angles of conv(P) are less than 180. Hence, Os-kernelext(P) has at most three components if jOj = 3. Before we treat the case jOj  4, we need a technical statement. We say a wedge w is a (1 ; 2)-wedge if (1 ; 2) is O-free and w is the wedge between h? (1) and h? (2?1) (see again Figure 37). Let  = 2 ? 1 . We claim that if a (1 ; 2)-wedge w and its opposite, the (1?1 ; 2?1)-wedge w?1 belong to Os-kernelext(P), then all the remaining interior angles of conv(P) are greater than 180 ? . To see this note that conv(P) is contained in the parallelogram Q = h+(1 ) \ h+ (2) \ h+(1?1 ) \ h+(2?1) whose sides that are parallel to 1 and 2 and that has the (1; 2?1 )apex p and the (1?1; 2 )-apex q in common with conv(P) as shown in Figure 38. The line segment pq partitions Q into two triangles T1 and T2 . Let 1 , 2 be the angles of T1 at pq and 1, 2 the angles of T2 at pq. Elementary geometry yields that 1 + 2   and 1 + 2  . Hence, the remaining m1 interior angles i, i = 1; . . . ; m1 , of the part of 57

conv(P) in T1 satisfy the inequality m1 X i=1

i +   m1
180

and the analog holds for the m2 interior angles i, i = 1; . . . ; m2 , of the part of conv(P) in T2 . Since all i and i are less than 180, we, in particular, have i > 180 ?  and i > 180 ? . Now if there are three components of Os-kernelext(P), then we have three disjoint, O-free ranges (i; i) such that there are three vertices of conv(P) with interior angles i  i ? i, 1  i  3, by Lemma 6.8. If two of the ranges are opposites to each other, say 2 = 1?1 and 2 = 1?1, then 3  180 ? 1 by the above argument. This contradicts the fact that [1; 1] [ [3 ; 3] does not cover 180. Hence, (1; 1 ), (2; 2 ), and (3 ; 3) are not opposites of each other. Since jOj  4, we have that 1 + 2 + 3 < 180 and, hence, there are three vertices in conv(P) whose sum of interior angles is less than 180. If we denote the remaining interior angles of conv(P) by i, i = 1; . . . ; m ? 3, we have the following inqualities (m ? 2)
180 =

3 X i=1

i +

< 180 +

mX ?3

i=1 mX ?3 i=1

i

i

and, hence, Pim=1?3 i > (m ? 3)
180 which again contradicts the fact that conv(P) is a convex polygon, i.e., that i  180. If jOj = 3 and there are three components, then we claim that conv(P) is a triangle. To see this note that by the above argument all three ranges have to be dierent and, hence, add up to exactly 180. This implies that the interior angle of any other vertex equals 180, i.e., there are only trivial (collinear) other vertices. Clearly, P coincides with its convex hull in this case; otherwise it violates one of the conditions of Lemma 6.8. This proves the last claim of the lemma. 2 To compute the external kernel we now proceed as follows. We assume that P is given as a list of n edges and that O consists of r ranges [1 ; 1]; . . . ; [r ; r ] which are given as a sorted array. First we note that the vertices where the O-tangents touch the boundary of P belong to the convex hull of P. Hence, we compute the convex hull of P which can be done in linear time by [BG84, GY83, Lee83, MA79]. Suppose that conv(P) consists of the (counterclockwise oriented) edges e1 ; . . . ; em and vertices v1 ; . . . ; vm with vi 2 ei \ ei+1 (em+1 = e1 ). By Lemma 6.8 there is a vertex vj in conv(P), for each (1 ; 2 )-wedge, such that (ej ) and (ej+1)?1 are in [1 ; 2] (see Figure 39). Using the same argument as in the proof of Lemma 6.10 it can be seen that there are at most four vertices vj on conv(P) that satisfy [(ej+1)?1; (ej )]  [1 ; 2 ], for some O-free range (1 ; 2 ) and 1 ; 2 2 O. We say vj is a candidate vertex of (1 ; 2 ) in this case. 58

2 (ei ) (ei+1 )?1

1

vi ei+1 ei

Figure 39: [(ej+1)?1; (ei)] is contained in [1 ; 2 ]. The idea of the algorithm is to nd these candidate vertices and then to check if the conditions of Lemma 6.8 are satis ed. Since we have to test for 1 -maxima and 2 -maxima at most four times and this can be done in time linear in the number of edges of P, we need at most additional O(n) steps once we have found the candidate vertices. In the following we show how to compute the candidate vertices of conv(P) in time O(n + r). The idea is to step through the edges of conv(P) and the ranges of O simultaneously. If we have a pointer to the ranges in O that keeps track of the orientation of the currently processed edge ei, it can be checked in constant time whether [(ej+1)?1; (ei)] is contained in a maximal O-free range. We say a range r = [; ] corresponds to orientation  if (; ) is O-free and is the least upper end point of a maximal O-free range such that   . More precisely, the algorithm can now be described as follows.

Algorithm External Os-kernel Let r1 be the O-free range that corresponds to (e1); for each edge ei of P do if [(ei+1 )?1; (ei)]  ri then output vi as a candidate vertex for ri end if if (ei+1 )  i then let ri+1 be ri else step through O to the O-free range ri+1 that corresponds to (ei+1 ) () end if end for; Since we look at each range of O only once in Step (), the algorithm obviously needs time O(n + r). In this way we compute the at most four candidate vertices of conv(P) together with the O-free ranges they belong to. As we already mentioned above, we need only 59

additional linear time to check if the conditions of Lemma 6.8 are satis ed. This proofs the following theorem. Theorem 6.11 The external Os-kernel of a polygon with n vertices can be computed in time O(n + r), for a set of orientations O consisting of r ranges, given O(r log r) preprocessing time to sort the ranges of O.

6.6 Computing the Os-Kernel of a Multiply Connected Polygon

With the above result, Lemma 6.6, and Theorem 6.5 we can compute the Os-kernel of a multiply connected polygon P with enclosing polygon Q and holes H1; . . . Hm by computing Os-kernel(Q) and intersecting it with Os-kernelext(Hi), for 1  i  m. We now want to obtain an estimate on the time needed for this procedure. Note that we have to restrict ourselves to nite O. So suppose that Q consists of n0 edges and each Hi of ni edges. Let n = n0 + Pmi=1 ni be the number of edges P. The rst step is to compute Os-kernel(Q) and Os-kernelext(Hi), for 1  i  m, which requires time

O(n0 log jOj +

m X i=1

ni + mjOj):

With the scan-line algorithm by Chazelle and Edelsbrunner [CE88] we can intersect m polygons with all together n edges in time O(n log n + (the number of intersections of the edges)). Since the external kernel consists only of at most eight edges and Os-kernel(Q) consists of n0 edges, this takes time O((n0 + m) log(n0 + m) + (the number of intersections)). Let k be the number of intersections. We want to obtain an estimate on k. k consists of two parts, k1 and k2, where k1 counts the intersections of Os-kernel(Q) with the external kernels of the holes and k2 counts the intersections among the edges of the external kernels of the holes. Since an edge e of Os-kernelext (Hi) is O-oriented and Os-kernel(Q) is O-convex by Corollary 5.15, @ Os -kernel(Q) intersects e at most twice. Hence, k1 is at most 2
8m = O(m). The O(m) edges of the external kernels of the holes have Tat most O(m2) intersection points. Hence, k = O(m2). Note that Figure 28 shows that 1im Os-kernelext(Hi) may, indeed, consist of O(m2) components. Therefore, these bounds are optimal. We have proven the following theorem. Theorem 6.12 The Os-kernel of a multiply connected polygon with n vertices and m holes can be computed in time O(n(log jOj + log n) + m(jOj + m)) time, for nite O. Note that this result is optimal for small O, i.e., if jOj = O(m) by our above observation on the number of components and Lemma 6.4.4 of [Raw87]. Before we turn to the relationship between the O-kernel and the Os-kernel we want to take a look the components of Os-kernel(P), for a multiply connected polygon P. Let H denote the intersection of all external kernels of holes of P, i.e., \ s H= O -kernelext (Hi): 1im

60

In the following lemma we show that we do not introduce new components by intersecting

H with Os-kernel(Q). Lemma 6.13 Let P be a multiply connected polygon in the plane with enclosing polygon Q and holes HT1; . . . ; Hm. If p and q are in Os-kernel(P) and p and q belong to the same component of mi=1 Os-kernelext(Hi ), then there is an O-convex path connecting p and q in Os-kernel(P). Proof: Let C be the component of Tmi=1 Os-kernelext(Hi) that contains p and q. As we noted above C is strong-O-convex; hence, C contains O-k[p; q], i.e., C contains all Oconvex paths from p to q. By Lemma 5.15 Os-kernel(Q) is Os-convex. Therefore, there is an O-convex path P in Os-kernel(Q) that connects p and q. Since C contains all O-convex paths from p to q, we have P  C \ Os-kernel(Q)  Os-kernel(P). 2 2 s This again implies that there are at most O(m ) components in O -kernel(P). An immediate consequence of the above lemma is the following corollary. Corollary 6.14 The components of Os-kernel(P) are O-convex.

6.7 The Relationship between Os-kernel(P) and O-kernel(P)

In this subsection we restrict ourselves to simple polygons. We investigate the relationship between Os-kernel(P) and O-kernel(P) which is straightforward and not very surprising. In the following lemma we show that Os-kernel(P) is a component of O-kernel(P). Theorem 6.15 If P is a simple polygon, then we have that

Os-kernel(P) is a component of O-kernel(P). Proof: Quite obviously we have Os-kernel(P)  O-kernel(P). Hence, there is a component C  O-kernel(P) that contains Os-kernel(P). We have to show that C = Os-kernel(P). We rst prove the claim for jOj = 1 and, then, extend it with the intersection lemmas to a general O. So for now assume w.l.o.g. that O = f0g. Clearly, it suces to show that the edges of @ C that do not completely belong to @ P equal the edges in Os-kernel(P) n @ P. We know that every edge of C or Os-kernel(P) that contains inner points of P is horizontal. Let e be an upper edge of Os-kernel(P) that contains inner points. Since e exists, we know by Theorem 6.2 that e intersects a re ex f0g-minimum p. Clearly, no point q of C can be in a suciently small neighbourhood above e since otherwise the horizontal line through q crosses the edges incident on p. The same argument holds for the lower edge of Os-kernel(P). This proves the claim for jOj = 1. Now, let jOj > 1. For each  2 OT, we have fgs-kernel(P) is a component of fgT-kernel(P). Hence, Os-kernel(P) = 2O fgs-kernel(P) is one of the components 2 of 2O fg-kernel(P) = O-kernel(P). 61

l

HHim H HH jH A H HH Y HAH s mk ?1 H H AH AA AU  ij AAKA  A s k X XXX k+1  A ij XXX  A  XX yXX  XXX    X  XX  k+1  s XXXX    i1*  ij ?1    

v

l

C

v

l

A A

l v

l

u

v

A ij ?1 A A s A   :A    9
A  :  A 
   A K  A


k +1 A A

A A k+1

 A U A im  


A 

k A 

A   *

A s
 * 
 -. * A  ij 0  
A  i1 A

A

l

C

C

l

ui1

?

v

(a) The unbounded case. (b) The bounded case. Figure 40: Two cases can occur when halfplanes are intersected.

7 Intersecting Presorted Halfplanes In this section we present two linear time algorithms to compute the intersection of n halfplanes h1 ; . . . ; hn which are sorted according to slope. For each 1  i  n, we denote the unique oriented line l such that h+(l) = hi by li. We say that li is associated to hi in this case. The intersection of the halfplanes h1 ; . . . ; hi is denoted by Ci and, furthermore, the normal vector of line li that points away from hi by ui. The rst algorithm we present uses an incremental approach and computes the sequence C1; C2; . . . ; Cn. In order to avoid distinguishing between the case that Ci is unbounded and that Ci is bounded we will not represent Ci by its actual boundary, i.e., as a list of rays, vertices and edges as illustrated in Figure 40, but rather as the sequence li1 ; . . . ; lim of the \relevant" lines for Ci and their intersection points vij = lij \ lij+1 . We say a line lj is relevant for Ci if the intersection of the halfplanes hk , with 1  k  i and k 6= j does not equal Ci, i.e., li intersects @ Ci in more than one point. Of course, the edges and, if necessary rays, that constitute @ Ci can be easily recontructed from li1 ; . . . ; lim and vi1 ; . . . ; vim?1 . If Cn is unbounded, then there is a pair of halfplanes whose slope dier by more than 180. Obviously, there can be only one such pair since the slope of the halfplanes is contained in [0 ; 360). W.l.o.g. we assume in the following that such a gap is between hn and h1 if it occurs at all. Therefore, we will not consider the case that there are two consecutive halfplanes hi and hi+1 such that (hi+1) ? (hi) > 180 in the algorithm.

62

7.1 An Incremental Algorithm

Assume we are given a description of Ck as the list of relevant lines Lk = fli1 ; . . . ; lim g together with the list of vertices Vk = fvi1 ; . . . ; vim?1 g. We will show how to compute the new list of relevant lines and vertices of Ck+1 after the intersection of Ck with hk+1 in time proportional to the number of lines and vertices deleted in Lk [ Vk . This immediately yields that the algorithm runs in linear time since line lk is added at most once to the list of relevant lines Lk and maybe deleted once in a later step. First assume that (hk+1)  (h1 ) + 180. (See Figure 40 (a) for illustration). Since hk+1 diers by more than 180 from h1 if we measure the angle clockwise, Ck+1 is unbounded. Clearly, there is only one intersection point of @ Ck and lk+1. Note that (hi) < (hk+1), for i  k. To nd it we proceed as follows. Starting with vim?1 we walk clockwise along the boundary of Ck until we nd the rst vertex vij?1 that is contained in hk+1 and delete all the lines lim ; . . . ; lij+1 together with the vertices vim?1 ; . . . ; vij . The new list of lines Lk+1 is given by li1 ; li2 ; . . . ; lij ; lk+1. Vk is then given by vi1 ; . . . ; vij with vij = lij \ lk+1. The part of the boundary of Ck that has not been removed is obviously in hk+1 so that all the lines in Lk+1 are still relevant. To show the correctness of our algorithm we need the following lemma. Lemma 7.1 Let l1 , l2, and l3 be any three oriented lines with hi = h+ (li) such that (i) the lines are sorted according to slope, (ii) the angle between l1 and l2 and the angle between l2 and l3 is less than 180 , and (iii) the intersection of l1 with l2 is after the intersection of l2 with l3 on l2 . If p = l1 \ l3 is contained in h2 , then T hi is non-empty and h3 is irrelevant, that is h1 \ h2 \ h3 is the same as h1 \ h2 . On the other hand, if p is not contained in h2, then T hi is empty. Proof: W.l.o.g. assume that (l2) = 180, i.e., h2 is below l2, and that s = l1 \ l2 is the origin. Since t = l2 \ l3 is before s, t has a positive x-coordinate. Suppose the equations of the halfplanes are given by (cf. Figure 41) h1 : cos 1 x1 + sin 1 x2  0 and (1) h3 : cos 3 x1 + sin 3 x2 + c  0: (2) Let W be the wedge enclosed by h1 and h3. Before we start to prove the claim we derive an inequality that holds for all points in W. Note that 1 and 3 are the angles of the normal vectors of l1 and l3 . Hence, we have 1 2 (?90 ; 90) and 3 2 (90; 270). This implies that cos 1 > 0 and cos 3 < 0. Let q = (q1 ; q2 ) be a point in W. From Inequality 1 we immediately get q1  ?(sin 1 = cos 1 )q2 since cos 1 > 0. Further, Inequality 2 yields sin 3 q ? c  q ? cos 3 2 cos 3 1 63

l3

l1 l2

h1

s h2 p

t

W

h3

h1

l2

s

t h2

W

l1

h1 \ h2 \ h 3

Figure 41: (a) Either H1 \ H1 = H1 \ H1 \ H3

l3

(b) H1 \ H2 \ H3 is empty.

or

because cos 3 < 0. Combining these inequalities we get sin 1 q  sin 3 q + c : (3) cos 1 2 cos 3 2 cos 3 It is our aim to show that W is either completely above or completely below l2 . This immediately the claimed result since if p is above l2, then all of W is above l2 and T h is empty.implies On the other hand, if p is below l2, then W is completely contained in h2 i and h2 is irrelevant. The proof is by contradiction. So assume there are points q = (q1 ; q2) and r = (r1; r2) in W with q2 < 0 and r2 > 0. We then get sin 1 q  sin 3 q + c cos 1 2 cos 3 2 cos 3 sin 3 q : < cos  2 3

Note that since t = (t1 ; t2) is on l3 and t2 > 0, we have that c = ?t2 = cos 3 > 0 and, hence, ?c=(cos 3 ) > 0. Therefore, sin 3 = cos 3 < sin 1 = cos 1 . Substituting r in Inequality 3 we have that ! sin 1 ? sin 3 r  c cos  cos  2 cos  1

3

3

which, of course, contradicts the fact that sin 3 = cos 3 < sin 1 = cos 1 , r2 > 0 and c= cos 3 < 0. So there cannot be points above and below l2 in W. 2 If we set the rst line to be lis?1 , the second to be lis and the third to be lk+1, for j + 1  s  m, then we see that we can delete lis if vis?1 is not in hk+1 (see Figure 42). 64

lis

@ @

lis?1 v

@ s is?1 @ k +1 P PP I PP 6@ @ PP i PP @P @PP

l

Figure 42: lis can be deleted. On the other hand, since there is only one intersection point of lk+1 with @ Ck and this is on the edge vij?1 vij , the remaining part of @ Ck is entirely contained in hk+1. Now suppose that (hk+1) > (h1) + 180. We assume for the moment that Ck is bounded and de ne v as the intersection point of lim and li1 . If v is contained in hk+1, then hk+1 is irrelevant for Ck+1. To see this we again apply Lemma 7.1 with the rst line as lim , the second line as lk+1 and the third line as li1 . So now suppose that either Ck is unbounded or v is not contained in hk+1. Then, we proceed as outlined above to nd an index j such that vij?1 contained in hk+1. Furthermore, starting with vi1 we walk counterclockwise around the boundary until we reach the rst vertex vij0 that is contained in hk+1. Since lk+1 has at most two intersections with @ Ck , this suces. Lk+1 will be given as the list lij0 ; . . . ; lij ; lk+1 and Vk+1 = fvij0 ; . . . ; vij?1 ; vj g with vij = lij \ lk+1. We describe the algorithm in a PASCAL-like notation.

Algorithm Halfplane intersection Input: halfplanes hi : aix1 + bi x2 + ci  0, i 2 f1; . . . ; ng that are sorted according to slope and whose dierence between two consecutive slopes is never greater than 180. Output: Lists L and V of lines and vertices that give the boundary of Cn. Method: for k := 1 to n do if Ck?1 is bounded and lim \ li1 is in hk then continue with k + 1 else nd the rst vertex vij?1 in V starting at vim that is in hk ; delete the lines in L after lij and the vertices after vij in V ; add lk to L and vij = lk \ lij to V ; if ((lk ) > (li1 ) + 180) then nd the rst vertex vij0 in V starting at vi1 that is in hk ; delete the lines in L before lij0 and the vertices 65

end if end for;

end if

before vij0 in V ;

if V andTL are empty then hi is empty else the vertices of Cn are given by V ; if Cn is unbounded then the rays of Cn are given by li1 and lim else the remaining vertex of Cn is given by li1 \ lim ; end if end if

7.2 Halfplane Intersection and the Convex Hull

In this section we show how to reduce the intersection of halfplanes that are sorted according to slope to the computation of the convex hull of certain point sets that are sorted on one coordinate. To this end we apply the concept of geometric duality which maps points onto lines and vice versa while keeping incidence relations. More precisely we apply the dual map D0 of H. Edelsbrunner's book [Ede87] which is de ned by D0(p) = f(x; y) 2 IR2 j y = p1 x ? p2g with p = (p1; p2) and D0(l) = (a1 ; ?a2) with l = f(x; y) 2 IR2 j y = a1x + a2 g D0 has the special property that is preserves order, i.e., if p is above the line l, then the point D0(l) is above the line D0 (p). In order to compute the intersection of the halfplanes H := fh1; . . . ; hng we now proceed as follows. We split H into two sets H+ and H? such that all the halfplanes that have their interior above them are in H+ and all the halfplanes that have their interior below them are in H?. Further, let L+ be the set of lines that are toTthe halfplanes inTH+ and let L? be de ned analogously for H?. Hence, T Hassociated T = H+ \ H?. Now p 2 H+ if and only if p is above all lines l in L+; in the dual plane this corresponds to the fact that for all l 2 L+, D0(l) is above D0 (p), i.e., if the convex hull of D0(L+) is above D0 (p). If we consider line l0 through the vertices v1 and v2 on the lower convex hull of D0(L+), we have that all the points of D0(L+) are above l0 except v1 and v2 which lie on l0 . So l0 corresponds to a point p0 in the original plane that is above all linesTin L+ and is the intersection point of D0 (v1) and D0(v2 ). In other words, p0 is a vertex of H+ . So we have just shown the following observation. Observation 7.2 The lines through the edges of the lower convex hull of D0(L+) correspond to the vertices of T H+ . In the same way we get that p 2 T H? if and only if, for all l 2 L?, D0(l) is below D0(p), i.e., if the convex hull of D0(L?) is below D0(p). Since we obviously can nd the intersection 66

of T H+ and T H? in linear time, we only have to look at the problem of computing the convex hull of the points D0(L+) resp. D0(L?). Clearly, the slopes of the lines l 2 L+ are contained in the interval [0 ; 90) [ (270; 360). If the equation of the line bounding a halfplane with slope  is y = a1 x + a2, we have the relation a1 = tan . Since the halfplanes are sorted according to slope, we can sort the halfplanes in H+ according to a1 in linear time if we append the ones with slope in [0; 90) to the ones with slope in (270; 360). But then D0(L+) is sorted according to the rst coordinate and we can compute the convex hull in time O(n). This gives a second linear time algorithm to compute the intersection of presorted halfplanes. It is interesting to note that we are able to obtain an \ultimate" halfspace intersection algorithm, i.e., an algorithm that runs in time O(n log h) if h is the number of vertices in the intersection, if we use a slightly dierent technique. By applying Meggido's technique T for linear programming to the problem of nding a point in hi we can determine in T T linear time whether hi is empty or nd a point p in the interior of hi. Translating the halfplanes so that p becomes the origin we can apply the classical dual transformation D which maps a point (p1; p2) to the line given by the equation y = p1x + p2 and vice versa. If we apply D to the lines that correspond to the boundaries of the halfplanes, the vertices of the intersection of the halfplanes just correspond to the lines through the edges of the convex hull of the dual points. The convex hull of the points can be computed in time O(n log h) by the algorithm of Seidel and Kirkpatrick [KS86] if h is the number of edges on the boundary. By retransforming the convex hull in linear time we obtain the intersection of the halfplanes within the same time bound.

67

8 A Lower Bound for Re ex -Maxima In this section we give a lower bound of (n log r) for the following problem. Given a polygon P with n edges and a set of orientations O with jOj = r where r  n, compute all the vertices of P that are re ex -maxima, for some  2 O. We also say that a vertex or edge is an re ex O-maximum if it is a re ex -maximum, for some  2 O. In order to show this lower bound we rst have to look at a dierent problem.

8.1 A Lower Bound for a Subset-Containment Problem

So we consider the following situation. Given a multiset A = fa1; . . . ; ang and a set B = fb + (i ? 1)" j b; " 2 IR+ and 1  i  rg. How much time is needed to nd out if every ai 2 A is in B ? The solution space S to this problem is a subset of IRn+2 and can be speci ed as follows.

S = f(a1; . . . ; an; b; ") 2 IRn+2 j 81  i  n 90  k  r ? 1 : ai = b + k"g: If we are given the function b
c, then the membership problem for S of a point p 2 IRn+2 can be solved in time O(n); but we assume the algebraic computation tree model which was introduced in Section 2.5 and which does not contain the oor-function. By the theorem of Ben-Or [Ben83] a lower bound for the membership problem for S is (log #S ? n) if S consists of #S connected components. So we will compute the number of components S consists of. Fixing the last two coordinates of S corresponds to a projection S 0 of S onto IRn. Since a projection clearly does not introduce any new connected components, the number of components of S 0 is at least as large as the number of components of S . S 0 is given by

S 0 = f(a1; . . . ; an) 2 IRn j for all 1  i  n 9 0  k  r ? 1 : ai = b + k"g where b and " are arbitrary but xed values. If " 6= 0, then S 0 obviously consists of rn distinct points. Hence, by Ben-Or's theorem a lower bound of the above problem is given by (log rn ? n) = (n log r).

8.2 A Reduction to the -Maxima Problem

Above we have shown that the subset containment problem as we stated it has complexity

(n log r). We want to use this fact to prove that the problem of nding the re ex Omaxima of a n-vertex polygon has the same complexity. In order to do this we reduce the containment problem to the re ex O-maxima problem in linear time. So assume we are given a multiset A = (a1 ; . . . ; an) and two values b and ". W.l.o.g. we assume that b = 0, and 0 < " < =4, and that ai 2 (0; =4 ? "). We now construct a polygon P as follows. For each ai , we append two edges e2i?1 and e2i to the boundary of P, one with orientation 68

ai and the other with orientation ai + ". By Lemma 5.3 we obtain a f0; 90g-convex curve C in this way. If we translate C to the left and connect the start and end point of C by two horizontal edges to its translation we obtain a simple polygon P. Let the boundary of P be oriented counterclockwise and let the rst vertex v1 of P be at the beginning of edge e1. Assume we are given an algorithm Q that computes all -re ex maxima, with  2 O = f0; "; 2"; . . . ; (r ? 1)"g. Let L be the output of Q. If an edge ej is in L, i.e., ej is a re ex O-maximum, then adj=2e belongs to O since an edge e is a re ex O-maximum only if (ei) 2 O and by construction we have (e2i?1) 2 O if and only if (e2i ) 2 O. If vertex v2i is a re ex O-maximum but edge e2i is not, then there is an orientation  2 O that is between the orientation of e2i?1 and e2i which implies that neither of them can belong to O. Hence, we in particular, have that ai = (e2i?1 ) 62 O. So we can solve the above stated subset containment problem in the same time bound that is used by Q. This implies that the complexity of nding the re ex O-maxima is at least (n log jOj) time and proves the following theorem.

Theorem 8.1 The problem of computing all re ex O-maxima of a n-vertex polygon requires (n log r) time in the algebraic computation tree model.

9 Conclusions We have introduced a new de nition of sight that encompasses both the usual notion of visibility based on straight line segments as well as staircase visibility. It is based on the theory of restricted orientation convexity. We have been able to show that the Os-kernel of a simple polygon can be obtained as the intersection of the -kernels, taken over all  2 O. With some additional observations this leads to an O(n log jOj) algorithm to compute the Os-kernel of a simple polygon, for nite O. In order to compute the kernel of a multiply connected polygon we introduced the external Os- kernel of a polygon and gave an O(n + r) to compute it where r is the number of ranges O consists of. Combining the two algorithms the Os-kernel of a multiply connected polygon can be computed in time O(n2 + njOj). There are several open problems in connection with the above results. As we already mentioned in Section 6.2, it is not known if the algorithm to compute the Os-kernel of a simple polygon is optimal or if it can be adapted to handle sets of orientations O that consist of a nite number of ranges. The algorithm to compute the external kernel of a polygon clearly is optimal but we conjecture that the algorithm for the Os-kernel of multiply connected polygon can be improved upon. Above we have always assumed that we are given the set of orientations we use for our visibility considerations. In the light of Corollary 5.12 one might ask the following question. Given a polygon P, for which orientations  2 [0 ; 360) is P fgs-starshaped, i.e., fgs-kernel(P) non-empty? This question can be seen as a natural generalization of the problem if a polygon is starshaped. 69

References [Ben83] [BG84] [CE88]

[CG88]

[CR89]

[Ede87] [GY83] [Kei86] [KS86] [Lee83] [LP79] [MA79]

M. Ben-Or. Lower bounds for algebraic computation trees. In Proceedings of the 15th ACM Symposium on Theory of Computing, pages 80{86, 1983. B. K. Bhattachara and H. El Gindy. A new linear convex hull algorithm for simple polygons. IEEE Transactions on Information Theory, IT-30:85{88, 1984. Bernhard Chazelle and Herbert Edelsbrunner. An optimal algorithm to intersect line segments in the plane. In Proceedings of the 29th Annual Syposium on Foundations of Computer Science, pages 590{600, IEEE, Computer Society Press, 1988. Richard Cole and Michael T. Goodrich. Optimal parallel algorithms for polygon and point-set problems. In Proceedings of the Fourth Annual Symposium on Computational Geometry, pages 201{210, ACM, ACM Press, Champaign, Illinois, June 1988. J. Culberson and R. Reckhow. A Uni ed Approach to Orthogonal Polygon Covering Problems via Dent Diagrams. Technical Report TR 89-6, Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada, February 1989. Herbert Edelsbrunner. Algorithms in Combinatorial Geometry. Springer Verlag, Berlin Heidelberg New York, eatcs monographs on theoretical computer science edition, 1987. R. L. Graham and F. F. Yao. Finding the convex hull of a simple polygon. Journal of Algorithms, 4:324{331, 1983. J. M. Keil. Minimally covering a horizontally convex polygon. In Proceedings of the Second Annual Symposium on Computational Geometry, pages 43{51, ACM, Yorktown Heights, New York, June 1986. D. G. Kirkpatrick and R. Seidel. The ultimate planar convex hull algorithm. SIAM Journal of Computing, 12(1):287{299, 1986. D. T. Lee. On nding the convex hull of a simple polygon. International Journal of Computer and Information Sciences, 12:87{99, 1983. D. T. Lee and F. P. Preparata. An optimal algorithm for nding the kernel of a polygon. Journal of the ACM, 26(3):415{421, July 1979. D. McCallum and D. Avis. A linear algorithm for nding the convex hull of a simple polygon. Information Processing Letters, 9:201{206, 1979. 70

[MRS88a] Rajeev Motwani, Arvind Raghunathan, and Huzur Saran. Covering orthogonal polygons with star polygons: the perfect graph approach. In Proceedings of the Fourth Annual Symposium on Computational Geometry, pages 211{223, ACM, Urbana, Illinois, June 1988. [MRS88b] Rajeev Motwani, Arvind Raghunathan, and Huzur Saran. Perfect graphs and orthogonally convex covers. In Fourth Annual SIAM Conference on Discrete Mothematics, page , 1988. [NLLW83] T. M. Nicholl, D. T. Lee, Y. Z. Liao, and C. K. Wong. Constructing the x-y convex hull of a set of x-y convex polygons. BIT, 23:456{471, 1983. [OW90] Thomas Ottmann and Peter Widmayer. Algorithmen und Datenstrukturen. BI Wissenschaftsverlag, 1990. [Raw87] Gregory J. E. Rawlins. Explorations in Restricted-Orientation Geometry. PhD thesis, University of Waterloo, 1987. [RC87] Robert A. Reckhow and Joseph Culberson. Covering a simple orthogonal polygon with a minimum number of orthogonally convex polygons. In Proceedings of the Third Annual Symposium on Computational Geometry, pages 268{277, ACM, Waterloo, Ontario, Canada, June 1987. [Rin75] W. Rinow. Topologie. Volume 79 of Hochschulbucher fur Mathematik, VEB Deutscher Verlag der Wissenschaften, Berlin, 1975. [RW87] Gregory J. E. Rawlins and Derick Wood. On the optimal computation of nitely-oriented convex hulls. Information and Computation, 72:150{166, 1987. [RW88a] G. J. E. Rawlins and D. Wood. Computational geometry with restricted orientations. In Proceedings of the 13th IFIP Conference on System Modelling and Optimization, Springer Verlag, 1988. Lecture Notes in Computer Science. [RW88b] Gregory J. E. Rawlins and Derick Wood. Ortho-convexity and its generalizations. In Godfried T. Toussaint, editor, Computational Morphology, pages 137{ 152, Elsevier Science Publishers B. V., (North-Holland), 1988. [Sch89] R. Schneider. Convex geometry. SS 1989. Script to Lecture. [SW91] S. Schuierer and D. Wood. Visibility in Semi-convex Spaces. Technical Report 37, Institut fur Informatik, Universit"at Freiburg, 1991. [WY88] Derick Wood and Chee K. Yap. The orthogonal convex skull problem. Discrete and Computational Geometry, 3(4):349{365, 1988.

71

[Yao89]

Andrew Chi-Chih Yao. Lower bounds for algebraic computation trees with integer inputs. In IEEE, editor, Proceedings of the 30th Annual Symposium on Foundations of Computer Science, pages 308{313, Computer Society Press, October 1989.

72