2
This electroniconly manuscript is published on www.designofapproxalgs.com with the permission of Cambridge University Press. One copy per user may be taken for personal use only and any other use you wish to make of the work is subject to the permission of Cambridge University Press (
[email protected]). You may not post this ﬁle on any other website.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Preface
This book is designed to be a textbook for graduatelevel courses in approximation algorithms. After some experience teaching minicourses in the area in the mid1990s, we sat down and wrote out an outline of the book. Then one of us (DPW), who was at the time an IBM Research Staﬀ Member, taught several iterations of the course following the outline we had devised, in Columbia University’s Department of Industrial Engineering and Operations Research in Spring 1998, in Cornell University’s School of Operations Research and Industrial Engineering in Fall 1998, and at the Massachusetts Institute of Technology’s Laboratory for Computer Science in Spring 2000. The lecture notes from these courses were made available, and we got enough positive feedback on them from students and from professors teaching such courses elsewhere that we felt we were on the right track. Since then, there have been many exciting developments in the area, and we have added many of them to the book; we taught additional iterations of the course at Cornell in Fall 2006 and Fall 2009 in order to ﬁeld test some of the writing of the newer results. The courses were developed for students who have already had a class, undergraduate or graduate, in algorithms, and who were comfortable with the idea of mathematical proofs about the correctness of algorithms. The book assumes this level of preparation. The book also assumes some basic knowledge of probability theory (for instance, how to compute the expected value of a discrete random variable). Finally, we assume that the reader knows something about NPcompleteness, at least enough to know that there might be good reason for wanting fast, approximate solutions to NPhard discrete optimization problems. At one or two points in the book, we do an NPcompleteness reduction to show that it can be hard to ﬁnd approximate solutions to such problems; we include a short appendix on the problem class NP and the notion of NPcompleteness for those unfamiliar with the concepts. However, the reader unfamiliar with such reductions can also safely skip over such proofs. In addition to serving as a graduate textbook, this book is a way for students to get the background to read current research in the area of approximation algorithms. In particular, we wanted a book that we could hand our own Ph.D. students just starting in the ﬁeld and say, “Here, read this.” We further hope that the book will serve as a reference to the area of approximation algorithms for researchers who are generally interested in the heuristic solution of discrete optimization problems; such problems appear in areas as diverse as traditional operations research planning problems (such as facility location and network design) to computer science prob3
4
Preface
lems in database and programming language design to advertising issues in viral marketing. We hope that the book helps researchers understand the techniques available in the area of approximation algorithms for approaching such problems. We have taken several particular perspectives in writing the book. The ﬁrst is that we wanted to organize the material around certain principles of designing approximation algorithms, around algorithmic ideas that have been used in diﬀerent ways and applied to diﬀerent optimization problems. The title The Design of Approximation Algorithms was carefully chosen. The book is structured around these design techniques. The introduction applies several of them to a single problem, the set cover problem. The book then splits into two parts. In the ﬁrst part, each chapter is devoted to a single algorithmic idea (e.g., “greedy and local search algorithms,”“rounding data and dynamic programming”), and the idea is then applied to several diﬀerent problems. The second part revisits all of the same algorithmic ideas, but gives more sophisticated treatments of them; the results covered here are usually much more recent. The layout allows us to look at several central optimization problems repeatedly throughout the book, returning to them as a new algorithmic idea leads to a better result than the previous one. In particular, we revisit such problems as the uncapacitated facility location problem, the prizecollecting Steiner tree problem, the binpacking problem, and the maximum cut problem several times throughout the course of the book. The second perspective is that we treat linear and integer programming as a central aspect in the design of approximation algorithms. This perspective is from our background in the operations research and mathematical programming communities. It is a little unusual in the computer science community, and students coming from a computer science background may not be familiar with the basic terminology of linear programming. We introduce the terms we need in the ﬁrst chapter, and we include a brief introduction to the area in an appendix. The third perspective we took in writing the book is that we have limited ourselves to results that are simple enough for classroom presentation while remaining central to the topic at hand. Most of the results in the book are ones that we have taught ourselves in class at one point or another. We bent this rule somewhat in order to cover the recent, exciting work by Arora, Rao, and Vazirani [22] applying semideﬁnite programming to the uniform sparsest cut problem. The proof of this result is the most lengthy and complicated of the book. We are grateful to a number of people who have given us feedback about the book at various stages in its writing. We are particularly grateful to James Davis, Lisa Fleischer, Isaac Fung, Rajiv Gandhi, Igor Gorodezky, Nick Harvey, Anna Karlin, Vijay Kothari, Katherine Lai, Gwen Spencer, and Anke van Zuylen for very detailed comments on a number of sections of the book. Additionally, the following people spotted typos, gave us feedback, helped us understand particular papers, and made useful suggestions: Bruno Abrahao, HyungChan An, Matthew Andrews, Eliot Anshelevich, Sanjeev Arora, Ashwinkumar B.V., Moses Charikar, Chandra Chekuri, Joseph Cheriyan, Chao Ding, Dmitriy Drusvyatskiy, Michel Goemans, Sudipto Guha, Anupam Gupta, Sanjeev Khanna, Lap Chi Lau, Renato Paes Leme, Jan Karel Lenstra, Roman Rischke, Gennady Samorodnitsky, Daniel Schmand, Jiawei Qian, Yogeshwer Sharma, Viktor ´ Tardos, Mike Todd, Di Wang, and Ann Williamson. We also Simjanoski, Mohit Singh, Eva thank a number of anonymous reviewers who made useful comments. Eliot Anshelevich, Joseph Cheriyan, Lisa Fleischer, Michel Goemans, Nicole Immorlica, and Anna Karlin used various drafts of the book in their courses on approximation algorithms and gave us useful feedback about the experience of using the book. We received quite a number of useful comments from the students in Anna’s class: Benjamin Birnbaum, Punyashloka Biswal, Elisa Celis, Jessica Chang, Mathias Hallman, Alyssa Joy Harding, Trinh Huynh, Alex Jaﬀe, Karthik Mohan, Katherine Moore, Cam Thach Nguyen, Richard Pang, Adrian Sampson, William Austin Webb, and Kevin Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Preface
5
Zatloukal. Frans Schalekamp generated the image on the cover; it is an illustration of the tree metric algorithm of Fakcharoenphol, Rao, and Talwar [106] discussed in Section 8.5. Our editor at Cambridge, Lauren Cowles, impressed us with her patience in waiting for this book to be completed and gave us a good deal of useful advice. We would like to thank the institutions that supported us during the writing of this book, including our home institution, Cornell University, and the IBM T.J. Watson and Almaden Research Centers (DPW), as well as TU Berlin (DPW) and the Sloan School of Management at MIT and the Microsoft New England Research Center (DBS), where we were on sabbatical leave when the ﬁnal editing of the book occurred. We are grateful to the National Science Foundation for supporting our research in approximation algorithms. Additional materials related to the book (such as contact information and errata) can be found at the website www.designofapproxalgs.com. We are also grateful to our wives and children — to Ann, Abigail, Daniel, and Ruth, and ´ to Eva, Rebecca, and Amy — for their patience and support during the writing of this volume. Finally, we hope the book conveys some of our enthusiasm and enjoyment of the area of approximation algorithms. We hope that you, dear reader, will enjoy it too. David P. Williamson David B. Shmoys January 2011
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6
Preface
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Table of Contents
Preface
I
3
An introduction to the techniques
11
1 An introduction to approximation algorithms 1.1 The whats and whys of approximation algorithms . . . . . . . . . 1.2 An introduction to the techniques and to linear programming: problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 A deterministic rounding algorithm . . . . . . . . . . . . . . . . . 1.4 Rounding a dual solution . . . . . . . . . . . . . . . . . . . . . . 1.5 Constructing a dual solution: the primaldual method . . . . . . 1.6 A greedy algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 A randomized rounding algorithm . . . . . . . . . . . . . . . . . 2 Greedy algorithms and local search 2.1 Scheduling jobs with deadlines on a single machine 2.2 The kcenter problem . . . . . . . . . . . . . . . . 2.3 Scheduling jobs on identical parallel machines . . . 2.4 The traveling salesman problem . . . . . . . . . . . 2.5 Maximizing ﬂoat in bank accounts . . . . . . . . . 2.6 Finding minimumdegree spanning trees . . . . . . 2.7 Edge coloring . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . . . the set cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
13 . 13 . . . . . .
16 19 20 23 24 28
. . . . . . .
35 36 37 39 43 47 49 54
3 Rounding data and dynamic programming 65 3.1 The knapsack problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.2 Scheduling jobs on identical parallel machines . . . . . . . . . . . . . . . . . . . . 68 3.3 The binpacking problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4 Deterministic rounding of linear programs 81 4.1 Minimizing the sum of completion times on a single machine . . . . . . . . . . . 82 4.2 Minimizing the weighted sum of completion times on a single machine . . . . . . 84 7
8
Table of Contents
4.3 4.4 4.5 4.6
Solving large linear programs in polynomial time via the The prizecollecting Steiner tree problem . . . . . . . . . The uncapacitated facility location problem . . . . . . . The binpacking problem . . . . . . . . . . . . . . . . .
ellipsoid . . . . . . . . . . . . . . .
method . . . . . . . . . . . . . . .
5 Random sampling and randomized rounding of linear programs 5.1 Simple algorithms for MAX SAT and MAX CUT . . . . . . . . . . 5.2 Derandomization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Flipping biased coins . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Randomized rounding . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Choosing the better of two solutions . . . . . . . . . . . . . . . . . . 5.6 Nonlinear randomized rounding . . . . . . . . . . . . . . . . . . . . 5.7 The prizecollecting Steiner tree problem . . . . . . . . . . . . . . . . 5.8 The uncapacitated facility location problem . . . . . . . . . . . . . . 5.9 Scheduling a single machine with release dates . . . . . . . . . . . . 5.10 Chernoﬀ bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 Integer multicommodity ﬂows . . . . . . . . . . . . . . . . . . . . . . 5.12 Random sampling and coloring dense 3colorable graphs . . . . . . . 6 Randomized rounding of semideﬁnite programs 6.1 A brief introduction to semideﬁnite programming 6.2 Finding large cuts . . . . . . . . . . . . . . . . . 6.3 Approximating quadratic programs . . . . . . . . 6.4 Finding a correlation clustering . . . . . . . . . . 6.5 Coloring 3colorable graphs . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
7 The primaldual method 7.1 The set cover problem: a review . . . . . . . . . . . . . . . . . . 7.2 Choosing variables to increase: the feedback vertex set problem graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Cleaning up the primal solution: the shortest st path problem . 7.4 Increasing multiple variables at once: the generalized Steiner tree 7.5 Strengthening inequalities: the minimum knapsack problem . . . 7.6 The uncapacitated facility location problem . . . . . . . . . . . . 7.7 Lagrangean relaxation and the kmedian problem . . . . . . . . . 8 Cuts and metrics 8.1 The multiway cut problem and a minimumcutbased algorithm 8.2 The multiway cut problem and an LP rounding algorithm . . . 8.3 The multicut problem . . . . . . . . . . . . . . . . . . . . . . . 8.4 Balanced cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Probabilistic approximation of metrics by tree metrics . . . . . 8.6 An application of tree metrics: Buyatbulk network design . . 8.7 Spreading metrics, tree metrics, and linear arrangement . . . .
. . . . . . .
. . . . .
. . . . .
. . . . . . . . . . . .
. . . . .
. . . . . . . . . . . .
. . . . .
. . . . . . . . . . . .
. . . . .
. . . .
. . . . . . . . . . . .
. . . . .
. . . .
. . . . . . . . . . . .
. . . . .
. . . .
. . . .
86 88 91 95
. . . . . . . . . . . .
105 . 106 . 108 . 110 . 111 . 114 . 116 . 118 . 120 . 124 . 128 . 132 . 133
. . . . .
141 . 141 . 143 . 147 . 150 . 153
161 . . . . . . . . . 161 in undirected . . . . . . . . . 164 . . . . . . . . . 168 problem . . . . 170 . . . . . . . . . 178 . . . . . . . . . 180 . . . . . . . . . 184
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
195 . 196 . 197 . 203 . 208 . 211 . 216 . 220
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Table of Contents
II
Further uses of the techniques
9
231
9 Further uses of greedy and local search algorithms 9.1 A local search algorithm for the uncapacitated facility location problem 9.2 A local search algorithm for the kmedian problem . . . . . . . . . . . . 9.3 Minimumdegree spanning trees . . . . . . . . . . . . . . . . . . . . . . . 9.4 A greedy algorithm for the uncapacitated facility location problem . . .
. . . .
. . . .
. . . .
. . . .
. . . .
233 234 239 243 247
10 Further uses of rounding data and dynamic programming 257 10.1 The Euclidean traveling salesman problem . . . . . . . . . . . . . . . . . . . . . . 257 10.2 The maximum independent set problem in planar graphs . . . . . . . . . . . . . 269 11 Further uses of deterministic rounding of linear 11.1 The generalized assignment problem . . . . . . . 11.2 Minimumcost boundeddegree spanning trees . . 11.3 Survivable network design and iterated rounding
programs 281 . . . . . . . . . . . . . . . . . . 282 . . . . . . . . . . . . . . . . . . 286 . . . . . . . . . . . . . . . . . . 297
12 Further uses of random sampling and randomized rounding grams 12.1 The uncapacitated facility location problem . . . . . . . . . . . . 12.2 The singlesource rentorbuy problem . . . . . . . . . . . . . . . 12.3 The Steiner tree problem . . . . . . . . . . . . . . . . . . . . . . 12.4 Everything at once: ﬁnding a large cut in a dense graph . . . . . 13 Further uses of randomized rounding of 13.1 Approximating quadratic programs . . . 13.2 Coloring 3colorable graphs . . . . . . . 13.3 Unique games . . . . . . . . . . . . . . .
semideﬁnite . . . . . . . . . . . . . . . . . . . . . . . .
of linear pro309 . . . . . . . . . 310 . . . . . . . . . 313 . . . . . . . . . 316 . . . . . . . . . 322
programs 333 . . . . . . . . . . . . . . . 334 . . . . . . . . . . . . . . . 340 . . . . . . . . . . . . . . . 344
14 Further uses of the primaldual method 355 14.1 The prizecollecting Steiner tree problem . . . . . . . . . . . . . . . . . . . . . . . 355 14.2 The feedback vertex set problem in undirected graphs . . . . . . . . . . . . . . . 360 15 Further uses of cuts and metrics 15.1 Low distortion embeddings and the sparsest cut problem 15.2 Oblivious routing and cuttree packings . . . . . . . . . 15.3 Cuttree packings and the minimum bisection problem . 15.4 The uniform sparsest cut problem . . . . . . . . . . . . 16 Techniques in proving the hardness of approximation 16.1 Reductions from NPcomplete problems . . . . . . . . . 16.2 Reductions that preserve approximation . . . . . . . . . 16.3 Reductions from probabilistically checkable proofs . . . 16.4 Reductions from label cover . . . . . . . . . . . . . . . . 16.5 Reductions from unique games . . . . . . . . . . . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
. . . . .
. . . .
369 . 369 . 376 . 382 . 385
. . . . .
407 . 407 . 412 . 420 . 425 . 437
17 Open Problems
447
A Linear programming
453
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10
Table of Contents
B NPcompleteness
457
Bibliography
461
Author index
481
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Part I
An introduction to the techniques
11
Chapter 1
An introduction to approximation algorithms
1.1
The whats and whys of approximation algorithms
Decisions, decisions. The diﬃculty of sifting through large amounts of data in order to make an informed choice is ubiquitous in today’s society. One of the promises of the information technology era is that many decisions can now be made rapidly by computers, from deciding inventory levels, to routing vehicles, to organizing data for eﬃcient retrieval. The study of how to make decisions of these sorts in order to achieve some best possible goal, or objective, has created the ﬁeld of discrete optimization. Unfortunately, most interesting discrete optimization problems are NPhard. Thus, unless P = NP, there are no eﬃcient algorithms to ﬁnd optimal solutions to such problems, where we follow the convention that an eﬃcient algorithm is one that runs in time bounded by a polynomial in its input size. This book concerns itself with the answer to the question “What should we do in this case?” An old engineering slogan says, “Fast. Cheap. Reliable. Choose two.” Similarly, if P ̸= NP, we can’t simultaneously have algorithms that (1) ﬁnd optimal solutions (2) in polynomial time (3) for any instance. At least one of these requirements must be relaxed in any approach to dealing with an NPhard optimization problem. One approach relaxes the “for any instance” requirement, and ﬁnds polynomialtime algorithms for special cases of the problem at hand. This is useful if the instances one desires to solve fall into one of these special cases, but this is not frequently the case. A more common approach is to relax the requirement of polynomialtime solvability. The goal is then to ﬁnd optimal solutions to problems by clever exploration of the full set of possible solutions to a problem. This is often a successful approach if one is willing to take minutes, or even hours, to ﬁnd the best possible solution; perhaps even more importantly, one is never certain that for the next input encountered, the algorithm will terminate in any reasonable amount of time. This is the approach taken by those in the ﬁeld of operations research and mathematical programming who solve integer programming formulations of discrete optimization problems, or those in the area of artiﬁcial intelligence who consider techniques such as A∗ search or constraint programming. 13
14
An introduction to approximation algorithms
By far the most common approach, however, is to relax the requirement of ﬁnding an optimal solution, and instead settle for a solution that is “good enough,” especially if it can be found in seconds or less. There has been an enormous study of various types of heuristics and metaheuristics such as simulated annealing, genetic algorithms, and tabu search, to name but a few. These techniques often yield good results in practice. The approach of this book falls into this third class. We relax the requirement of ﬁnding an optimal solution, but our goal is to relax this as little as we possibly can. Throughout this book, we will consider approximation algorithms for discrete optimization problems. We try to ﬁnd a solution that closely approximates the optimal solution in terms of its value. We assume that there is some objective function mapping each possible solution of an optimization problem to some nonnegative value, and an optimal solution to the optimization problem is one that either minimizes or maximizes the value of this objective function. Then we deﬁne an approximation algorithm as follows. Deﬁnition 1.1: An αapproximation algorithm for an optimization problem is a polynomialtime algorithm that for all instances of the problem produces a solution whose value is within a factor of α of the value of an optimal solution. For an αapproximation algorithm, we will call α the performance guarantee of the algorithm. In the literature, it is also often called the approximation ratio or approximation factor of the algorithm. In this book we will follow the convention that α > 1 for minimization problems, while α < 1 for maximization problems. Thus, a 12 approximation algorithm for a maximization problem is a polynomialtime algorithm that always returns a solution whose value is at least half the optimal value. Why study approximation algorithms? We list several reasons. • Because we need algorithms to get solutions to discrete optimization problems. As we mentioned above, with our current information technology there are an increasing number of optimization problems that need to be solved, and most of these are NPhard. In some cases, an approximation algorithm is a useful heuristic for ﬁnding nearoptimal solutions when the optimal solution is not required. • Because algorithm design often focuses first on idealized models rather than the “realworld” application. In practice, many discrete optimization problems are quite messy, and have many complicating side constraints that make it hard to ﬁnd an approximation algorithm with a good performance guarantee. But often approximation algorithms for simpler versions of the problem give us some idea of how to devise a heuristic that will perform well in practice for the actual problem. Furthermore, the push to prove a theorem often results in a deeper mathematical understanding of the problem’s structure, which then leads to a new algorithmic approach. • Because it provides a mathematically rigorous basis on which to study heuristics. Typically, heuristics and metaheuristics are studied empirically; they might work well, but we might not understand why. The ﬁeld of approximation algorithms brings mathematical rigor to the study of heuristics, allowing us to prove how well the heuristic performs on all instances, or giving us some idea of the types of instances on which the heuristic will not perform well. Furthermore, the mathematical analyses of many of the approximation algorithms in this book have the property that not only is there an a priori guarantee for any input, but there is also an a fortiori guarantee that is provided on an inputbyinput Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.1
The whats and whys of approximation algorithms
15
basis, which allows us to conclude that speciﬁc solutions are in fact much more nearly optimal than promised by the performance guarantee. • Because it gives a metric for stating how hard various discrete optimization problems are. Over the course of the twentieth century, the study of the power of computation has steadily evolved. In the early part of the century, researchers were concerned with what kinds of problems could be solved at all by computers in ﬁnite time, with the halting problem as the canonical example of a problem that could not be solved. The latter part of the century concerned itself with the eﬃciency of solution, distinguishing between problems that could be solved in polynomial time, and those that are NPhard and (perhaps) cannot be solved eﬃciently. The ﬁeld of approximation algorithms gives us a means of distinguishing between various optimization problems in terms of how well they can be approximated. • Because it’s fun. The area has developed some very deep and beautiful mathematical results over the years, and it is inherently interesting to study these. It is sometimes objected that requiring an algorithm to have a nearoptimal solution for all instances of the problem — having an analysis for what happens to the algorithm in the worst possible instance — leads to results that are too loose to be practically interesting. After all, in practice, we would greatly prefer solutions within a few percent of optimal rather than, say, twice optimal. From a mathematical perspective, it is not clear that there are good alternatives to this worstcase analysis. It turns out to be quite diﬃcult to deﬁne a “typical” instance of any given problem, and often instances drawn randomly from given probability distributions have very special properties not present in realworld data. Since our aim is mathematical rigor in the analysis of our algorithms, we must content ourselves with this notion of worstcase analysis. We note that the worstcase bounds are often due to pathological cases that do not arise in practice, so that approximation algorithms often give rise to heuristics that return solutions much closer to optimal than indicated by their performance guarantees. Given that approximation algorithms are worth studying, the next natural question is whether there exist good approximation algorithms for problems of interest. In the case of some problems, we are able to obtain extremely good approximation algorithms; in fact, these problems have polynomialtime approximation schemes. Deﬁnition 1.2: A polynomialtime approximation scheme (PTAS) is a family of algorithms {Aϵ }, where there is an algorithm for each ϵ > 0, such that Aϵ is a (1 + ϵ)approximation algorithm (for minimization problems) or a (1 − ϵ)approximation algorithm (for maximization problems). Many problems have polynomialtime approximation schemes. In later chapters we will encounter the knapsack problem and the Euclidean traveling salesman problem, each of which has a PTAS. However, there exists a class of problems that is not so easy. This class is called MAX SNP; although we will not deﬁne it, it contains many interesting optimization problems, such as the maximum satisﬁability problem and the maximum cut problem, which we will discuss later in the book. The following has been shown. Theorem 1.3: For any MAX SNPhard problem, there does not exist a polynomialtime approximation scheme, unless P = NP. Finally, some problems are very hard. In the maximum clique problem, we are given as input an undirected graph G = (V, E). The goal is to ﬁnd a maximumsize clique; that is, Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
16
An introduction to approximation algorithms
we wish to ﬁnd S ⊆ V that maximizes S so that for each pair i, j ∈ S, it must be the case that (i, j) ∈ E. The following theorem demonstrates that almost any nontrivial approximation guarantee is most likely unattainable. Theorem 1.4: Let n denote the number of vertices in an input graph, and consider any constant ϵ > 0. Then there does not exist an O(nϵ−1 )approximation algorithm for the maximum clique problem, unless P = NP. To see how strong this theorem is, observe that it is very easy to get an n−1 approximation algorithm for the problem: just output a single vertex. This gives a clique of size 1, whereas the size of the largest clique can be at most n, the number of vertices in the input. The theorem states that ﬁnding something only slightly better than this completely trivial approximation algorithm implies that P = NP!
1.2
An introduction to the techniques and to linear programming: the set cover problem
One of the theses of this book is that there are several fundamental techniques used in the design and analysis of approximation algorithms. The goal of this book is to help the reader understand and master these techniques by applying each technique to many diﬀerent problems of interest. We will visit some problems several times; when we introduce a new technique, we may see how it applies to a problem we have seen before, and show how we can obtain a better result via this technique. The rest of this chapter will be an illustration of several of the central techniques of the book applied to a single problem, the set cover problem, which we deﬁne below. We will see how each of these techniques can be used to obtain an approximation algorithm, and how some techniques lead to improved approximation algorithms for the set cover problem. In the set cover problem, we are given a ground set of elements E = {e1 , . . . , en }, some subsets of those elements S1 , S2 , . . . , Sm where each Sj ⊆ E, and a nonnegative weight wj ≥ 0 for each subset Sj . The goal is to ﬁnd a minimumweight collection of subsets that∪covers all of ∑ E; that is, we wish to ﬁnd an I ⊆ {1, . . . , m} that minimizes j∈I wj subject to j∈I Sj = E. If wj = 1 for each subset j, the problem is called the unweighted set cover problem. The set cover problem is an abstraction of several types of problems; we give two examples here. The set cover problem was used in the development of an antivirus product, which detects computer viruses. In this case it was desired to ﬁnd salient features that occur in viruses designed for the boot sector of a computer, such that the features do not occur in typical computer applications. These features were then incorporated into another heuristic for detecting these boot sector viruses, a neural network. The elements of the set cover problem were the known boot sector viruses (about 150 at the time). Each set corresponded to some threebyte sequence occurring in these viruses but not in typical computer programs; there were about 21,000 such sequences. Each set contained all the boot sector viruses that had the corresponding threebyte sequence somewhere in it. The goal was to ﬁnd a small number of such sequences (much smaller than 150) that would be useful for the neural network. By using an approximation algorithm to solve the problem, a small set of sequences was found, and the neural network was able to detect many previously unanalyzed boot sector viruses. The set cover problem also generalizes the vertex cover problem. In the vertex cover problem, we are given an undirected graph G = (V, E) and a nonnegative weight wi ≥ 0 for each vertex i ∈ V . The goal is to ﬁnd a minimumweight subset of vertices C ⊆ V such that for each edge (i, j) ∈ E, either i ∈ C or j ∈ C. As in the set cover problem, if wi = 1 for each vertex i, Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.2 An introduction to the techniques and to linear programming: the set cover problem 17
the problem is an unweighted vertex cover problem. To see that the vertex cover problem is a special case of the set cover problem, for any instance of the vertex cover problem, create an instance of the set cover problem in which the ground set is the set of edges, and a subset Si of weight wi is created for each vertex i ∈ V containing the edges incident to i. It is not diﬃcult to see that for any vertex cover C, there is a set cover I = C of the same weight, and vice versa. A second thesis of this book is that linear programming plays a central role in the design and analysis of approximation algorithms. Many of the techniques introduced will use the theory of integer and linear programming in one way or another. Here we will give a very brief introduction to the area in the context of the set cover problem; we give a slightly less brief introduction in Appendix A, and the notes at the end of this chapter provide suggestions of other, more indepth, introductions to the topic. Each linear program or integer program is formulated in terms of some number of decision variables that represent some sort of decision that needs to be made. The variables are constrained by a number of linear inequalities and equalities called constraints. Any assignment of real numbers to the variables such that all of the constraints are satisﬁed is called a feasible solution. In the case of the set cover problem, we need to decide which subsets Sj to use in the solution. We create a decision variable xj to represent this choice. In this case we would like xj to be 1 if the set Sj is included in the solution, and 0 otherwise. Thus, we introduce constraints xj ≤ 1 for all subsets Sj , and xj ≥ 0 for all subsets Sj . This is not suﬃcient to guarantee that xj ∈ {0, 1}, so we will formulate the problem as an integer program to exclude fractional solutions (that is, nonintegral solutions); in this case, we are also allowed to constrain the decision variables to be integers. Requiring xj to be integer along with the constraints xj ≥ 0 and xj ≤ 1 is suﬃcient to guarantee that xj ∈ {0, 1}. We also want to make sure that any feasible solution corresponds to a set cover, so we introduce additional constraints. In order to ensure that every element ei is covered, it must be the case that at least one of the subsets Sj containing ei is selected. This will be the case if ∑
xj ≥ 1,
j:ei ∈Sj
for each ei , i = 1, . . . , n. In addition to the constraints, linear and integer programs are deﬁned by a linear function of the decision variables called the objective function. The linear or integer program seeks to ﬁnd a feasible solution that either maximizes or minimizes this objective function. Such a solution is called an optimal solution. The value of the objective function for a particular feasible solution is called the value of that solution. The value of the objective function for an optimal solution is called the value of the linear (or integer) program. We say we solve the linear program if we ﬁnd an optimal solution. In the case of the set cover problem, we want to ﬁnd a set cover of minimum weight. Given the decision variables xj and constraints described above, the weight ∑m of a set cover given the x variables is w j j=1 j xj . Thus, the objective function of the integer ∑m program is j=1 wj xj , and we wish to minimize this function. Integer and linear programs are usually written in a compact form stating ﬁrst the objective function and then the constraints. Given the discussion above, the problem of ﬁnding a Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
18
An introduction to approximation algorithms
minimumweight set cover is equivalent to the following integer program: minimize
m ∑
wj xj
j=1
∑
subject to
xj ≥ 1,
i = 1, . . . , n,
xj ∈ {0, 1} ,
j = 1, . . . , m.
(1.1)
j:ei ∈Sj
∗ denote the optimum value of this integer program for a given instance of the set cover Let ZIP ∗ = OPT, problem. Since the integer program exactly models the problem, we have that ZIP where OPT is the value of an optimum solution to the set cover problem. In general, integer programs cannot be solved in polynomial time. This is clear because the set cover problem is NPhard, so solving the integer program above for any set cover input in polynomial time would imply that P = NP. However, linear programs are polynomialtime solvable. In linear programs we are not allowed to require that decision variables are integers. Nevertheless, linear programs are still extremely useful: even in cases such as the set cover problem, we are still able to derive useful information from linear programs. For instance, if we replace the constraints xj ∈ {0, 1} with the constraints xj ≥ 0, we obtain the following linear program, which can be solved in polynomial time:
minimize
m ∑
wj xj
j=1
subject to
∑
xj ≥ 1,
i = 1, . . . , n,
xj ≥ 0,
j = 1, . . . , m.
(1.2)
j:ei ∈Sj
We could also add the constraints xj ≤ 1, for each j = 1, . . . , m, but they would be redundant: in any optimal solution to the problem, we can reduce any xj > 1 to xj = 1 without aﬀecting the feasibility of the solution and without increasing its cost. The linear program (1.2) is a relaxation of the original integer program. By this we mean two things: ﬁrst, every feasible solution for the original integer program (1.1) is feasible for this linear program; and second, the value of any feasible solution for the integer program has the same value in the linear program. To see that the linear program is a relaxation,∑note that any solution for the integer program such that xj ∈ {0, 1} for each j = 1, . . . , m and j:ei ∈Sj xj ≥ 1 for each i = 1, . . . , m will certainly satisfy all the constraints of the linear program. Furthermore, the objective functions of both the integer and linear programs are the same, so that any feasible ∗ denote the solution for the integer program has the same value for the linear program. Let ZLP optimum value of this linear program. Any optimal solution to the integer program is feasible ∗ . Thus, any optimal solution to the linear program will for the linear program and has value ZIP ∗ ∗ have value ZLP ≤ ZIP = OPT, since this minimization linear program ﬁnds a feasible solution of lowest possible value. Using a polynomialtime solvable relaxation of a problem in order to obtain a lower bound (in the case of minimization problems) or an upper bound (in the case of maximization problems) on the optimum value of the problem is a concept that will appear frequently in this book. In the following sections, we will give some examples of how the linear programming relaxation can be used to derive approximation algorithms for the set cover problem. In the next section, we will show that a fractional solution to the linear program can be rounded to Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.3
A deterministic rounding algorithm
19
a solution to the integer program of objective function value that is within a certain factor ∗ . Thus, the integer solution will cost no more than f of the value of the linear program ZLP f · OPT. In the following section, we will show how one can similarly round the solution to something called the dual of the linear programming relaxation. In Section 1.5, we will see that in fact one does not need to solve the dual of the linear programming relaxation, but in fact can quickly construct a dual feasible solution with the properties needed to allow a good rounding. In Section 1.6, a type of algorithm called a greedy algorithm will be given; in this case, linear programming need not be used at all, but one can use the dual to improve the analysis of the algorithm. Finally, in Section 1.7, we will see how randomized rounding of the solution to the linear programming relaxation can lead to an approximation algorithm for the set cover problem. Because we will frequently be referring to linear programs and linear programming, we will often abbreviate these terms by the acronym LP. Similarly, IP stands for either integer program or integer programming.
1.3
A deterministic rounding algorithm
Suppose that we solve the linear programming relaxation of the set cover problem. Let x∗ denote an optimal solution to the LP. How then can we recover a solution to the set cover problem? Here is a very easy way to obtain a solution: given the LP solution x∗ , we include subset Sj in our solution if and only if x∗j ≥ 1/f , where f is the maximum number of sets in which any element appears. More formally, let fi =  {j : ei ∈ Sj }  be the number of sets in which element ei appears, i = 1, . . . , n; then f = maxi=1,...,n fi . Let I denote the indices j of the subsets in this solution. In eﬀect, we round the fractional solution x∗ to an integer solution x ˆ by setting x ˆj = 1 if x∗j ≥ 1/f , and x ˆj = 0 otherwise. We shall see that it is straightforward to prove that x ˆ is a feasible solution to the integer program, and I indeed indexes a set cover. Lemma 1.5: The collection of subsets Sj , j ∈ I, is a set cover.
Proof. Consider the solution speciﬁed by the lemma, and call an element ei covered if this solution contains some subset containing ei . We show that each element ei is covered. Because ∑ ∗ the optimal solution x is a feasible solution to the linear program, we know that j:ei ∈Sj x∗j ≥ 1 for element ei . By the deﬁnition of fi and of f , there are fi ≤ f terms in the sum, so at least one term must be at least 1/f . Thus, for some j such that ei ∈ Sj , x∗j ≥ 1/f . Therefore, j ∈ I, and element ei is covered.
We can also show that this rounding procedure yields an approximation algorithm. Theorem 1.6: The rounding algorithm is an f approximation algorithm for the set cover problem.
Proof. It is clear that the algorithm runs in polynomial time. By our construction, 1 ≤ f · x∗j for each j ∈ I. From this, and the fact that each term f wj x∗j is nonnegative for j = 1, . . . , m, Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
20
An introduction to approximation algorithms
we see that ∑
≤
wj
m ∑
wj · (f · x∗j )
j=1 m ∑
j∈I
= f
wj x∗j
j=1 ∗ = f · ZLP
≤ f · OPT, ∗ ≤ OPT. where the ﬁnal inequality follows from the argument above that ZLP
In the special case of the vertex cover problem, fi = 2 for each vertex i ∈ V , since each edge is incident to exactly two vertices. Thus, the rounding algorithm gives a 2approximation algorithm for the vertex cover problem. This particular algorithm allows us to have an a fortiori guarantee for each input. While we know that for any input, the solution produced has cost at most a factor of f more than the cost of an optimal solution, we can for any input compare the value of the solution we ﬁnd with∑the value of the linear programming relaxation. If the algorithm ﬁnds a set cover I, ∗ . From the proof above, we know that α ≤ f . However, for any given let α = j∈I wj /ZLP input, it could be the case that α is signiﬁcantly smaller than f ; in this case we know that ∑ ∗ j∈I wj = αZLP ≤ α OPT, and the solution is within a factor of α of optimal. The algorithm can easily compute α, given that it computes I and solves the LP relaxation.
1.4
Rounding a dual solution
Often it will be useful to consider the dual of the linear programming relaxation of a given problem. Again, we will give a very brief introduction to the concept of the dual of a linear program in the context of the set cover problem, and more indepth introductions to the topic will be cited in the notes at the end of this chapter. To begin, we suppose that each element ei is charged some nonnegative price yi ≥ 0 for its coverage by a set cover. Intuitively, it might be the case that some elements can be covered with lowweight subsets, while other elements might require highweight subsets to cover them; we would like to be able to capture this distinction by charging low prices to the former and high prices to the latter. In order for the prices to be reasonable, it cannot be the case that the sum of the prices of elements in a subset Sj is more than the weight of the set, since we are able to cover all of those elements by paying weight wj . Thus, for each subset Sj we have the following limit on the prices: ∑ yi ≤ wj . i:ei ∈Sj
We can ﬁnd the highest total price that the elements can be charged by the following linear program: n ∑ maximize yi i=1
subject to
∑
yi ≤ wj ,
j = 1, . . . , m,
yi ≥ 0,
i = 1, . . . , n.
(1.3)
i:ei ∈Sj
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.4
Rounding a dual solution
21
This linear program is the dual linear program of the set cover linear programming relaxation (1.2). We can in general derive a dual linear program for any given linear program, but we will not go into the details of how to do so; see Appendix A or the references in the notes at the end of the chapter. If we derive a dual for a given linear program, the given program is sometimes called the primal linear program. For instance, the original linear programming relaxation (1.2) of the set cover problem is the primal linear program of the dual (1.3). Notice that this dual ∑ has a variable yi for each constraint of the primal linear program (that is, for the constraint j:ei ∈Sj xj ≥ 1), and has a constraint for each variable xj of the primal. This is true of dual linear programs in general. Dual linear programs have a number of very interesting and useful properties. For example, let x be any feasible solution to the set cover linear programming relaxation, and let y be any feasible set of prices (that is, any feasible solution to the dual linear program). Then consider the value of the dual solution y: n ∑
yi ≤
i=1
n ∑
∑
yi
xj ,
j:ei ∈Sj
i=1
∑
since for any ei , j:ei ∈Sj xj ≥ 1 by the feasibility of x. Then rewriting the righthand side of this inequality, we have m n ∑ ∑ ∑ ∑ yi . xj = xj yi i=1
j:ei ∈Sj
i:ei ∈Sj
j=1
Finally, noticing that since y is a feasible solution to the dual linear program, we know that ∑ i:ei ∈Sj yi ≤ wj for any j, so that m ∑ j=1
So we have shown that
∑
xj
yi ≤
i:ei ∈Sj n ∑ i=1
yi ≤
m ∑
xj wj .
j=1 m ∑
wj xj ;
j=1
that is, any feasible solution to the dual linear program has a value no greater than any feasible solution to the primal linear program. In particular, any feasible solution to the dual linear program has a value than the optimal solution to the primal linear program, so for ∑ no greater ∗ . This is called the weak duality property of linear programs. any feasible y, ni=1 yi ≤ ZLP ∑n ∗ ≤ OPT, we have that for any feasible y, Since we previously argued that ZLP i=1 yi ≤ OPT . This is a very useful property that will help us in designing approximation algorithms. Additionally, there is a quite amazing strong duality property of linear programs. Strong duality states that as long as there exist feasible solutions to both the primal and dual linear programs, their optimal values are equal. Thus, if x∗ is an optimal solution to the set cover linear programming relaxation, and y ∗ is an optimal solution to the dual linear program, then m ∑ j=1
wj x∗j =
n ∑
yi∗ .
i=1
Information from a dual linear program solution can sometimes be used to derive good approximation algorithms. Let y ∗ be an optimal solution to the dual LP (1.3), and consider Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
22
An introduction to approximation algorithms
the solution in which we choose all subsets for which the corresponding dual inequality is ∑ tight; that is, the inequality is met with equality for subset Sj , and i:ei ∈Sj yi∗ = wj . Let I ′ denote the indices of the subsets in this solution. We will prove that this algorithm also is an f approximation algorithm for the set cover problem. Lemma 1.7: The collection of subsets Sj , j ∈ I ′ , is a set cover. Proof. Suppose that there exists some uncovered element ek . Then for each subset Sj containing ek , it must be the case that ∑ yi∗ < wj . (1.4) i:ei ∈Sj
Let ϵ be the smallest diﬀerence between side ( the righthand ) and lefthand side of all constraints ∑ ∗ involving ek ; that is, ϵ = minj:ek ∈Sj wj − i:ei ∈Sj yi . By inequality (1.4), we know that ϵ > 0. Consider now a new dual solution y ′ in which yk′ = yk∗ + ϵ and every other component of y ′ is the same as in y ∗ . Then y ′ is a dual feasible solution since for each j such that ek ∈ Sj , ∑ ∑ yi′ = yi∗ + ϵ ≤ wj , i:ei ∈Sj
i:ei ∈Sj
by the deﬁnition of ϵ. For each j such that ek ∈ / Sj , ∑ ∑ yi′ = yi∗ ≤ wj , ∑n
i:ei ∈Sj
i:ei ∈Sj
∑n
as before. Furthermore, i=1 yi′ > i=1 yi∗ , which contradicts the optimality of y ∗ . Thus, it must be the case that all elements are covered and I ′ is a set cover. Theorem 1.8: The dual rounding algorithm described above is an f approximation algorithm for the set cover problem. Proof. The central idea is the following “charging” argument: when we choose a set Sj to be in the cover, we “pay” for it by charging yi∗ to each of its elements ei ; each element is charged at most once set that contains it (and hence at most f times), and so the total cost is at ∑ for each ∗ , or f times the dual objective function. y most f m i=1 i ∑ More formally, since j ∈ I ′ only if wj = i:ei ∈Sj yi∗ , we have that the cost of the set cover I ′ is ∑ ∑ ∑ wj = yi∗ j∈I ′
=
j∈I ′ i:ei ∈Sj n ∑
{ }  j ∈ I ′ : ei ∈ Sj  · yi∗
i=1
≤
n ∑
fi yi∗
i=1 n ∑
≤ f
yi∗
i=1
≤ f · OPT . The second equality follows from the fact that when we interchange the order of summation, the coeﬃcient of yi∗ is, of course, equal to the number of times that this term occurs overall. The ﬁnal inequality follows from the weak duality property discussed previously. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.5
Constructing a dual solution: the primaldual method
23
In fact, it is possible to show that this algorithm can do no better than the algorithm of the previous section; to be precise, we can show that if I indexes the solution returned by the primal rounding algorithm of the previous section, then I ⊆ I ′ . This follows from a property of optimal linear programming solutions called complementary slackness. We showed earlier the following string of inequalities for any feasible solution x to the set cover linear programming relaxation, and any feasible solution y to the dual linear program: n ∑ i=1
yi ≤
n ∑ i=1
yi
∑ j:ei ∈Sj
xj =
m ∑ j=1
xj
∑ i:ei ∈Sj
yi ≤
m ∑
xj wj .
j=1
Furthermore, we claimed that strong duality implies that for optimal solutions x∗ and y ∗ , ∑ ∑ m n ∗ ∗ ∗ ∗ j=1 wj xj . Thus, for any optimal solutions x and y the two inequalities in the i=1 yi = chain of inequalities above this can happen is that ∑ must in fact be equalities. The only way ∑ whenever yi∗ > 0 then j:ei ∈Sj x∗j = 1, and whenever x∗j > 0, then i:ei ∈Sj yi∗ = wj . That is, whenever a linear programming variable (primal or dual) is nonzero, the corresponding constraint in the dual or primal is tight. These conditions are known as the complementary slackness conditions. Thus, if x∗ and y ∗ are optimal solutions, the complementary slackness conditions must hold. The converse is also true: if x∗ and y ∗ are feasible primal and dual solutions, respectively, then if the complementary slackness conditions hold, the values of the two objective functions are equal and therefore the solutions must be optimal. In the case of the set cover program, if x∗j > 0 for any primal optimal solution x∗ , then the corresponding dual inequality for Sj must be tight for any dual optimal solution y ∗ . Recall that in the algorithm of the previous section, we put j ∈ I when x∗j ≥ 1/f . Thus, j ∈ I implies that j ∈ I ′ , so that I ′ ⊇ I.
1.5
Constructing a dual solution: the primaldual method
One of the disadvantages of the algorithms of the previous two sections is that they require solving a linear program. While linear programs are eﬃciently solvable, and algorithms for them are quick in practice, special purpose algorithms are often much faster. Although in this book we will not usually be concerned with the precise running times of the algorithms, we will try to indicate their relative practicality. The basic idea of the algorithm in this section is that the dual rounding algorithm of the previous section uses relatively few properties of an optimal dual solution. Instead of actually solving the dual LP, we can construct a feasible dual solution with the same properties. In this case, constructing the dual solution is much faster than solving the dual LP, and hence leads to a much faster algorithm. The ∑ algorithm of the previous section used the following properties. First, we used the fact′ that ni=1 yi ≤ ∑ OPT, which is true for any feasible dual solution y. Second, we include j ∈ I precisely when i:ei ∈Sj yi = wj , and I ′ is a set cover. These two facts together gave the proof that the cost of I ′ is no more than f times optimal. Importantly, it is the proof of Lemma 1.7 (that we have constructed a feasible cover) that shows how to obtain an algorithm that constructs a dual solution. Consider any feasible dual solution y, and let T be the set of the indices of all tight dual constraints; that is, T = {j : ∑ y i:ei ∈Sj i = wj }. If T is a set cover, then we are done. If T is not a set cover, then some item ei is uncovered, and as shown in the proof of Lemma 1.7 it is possible to improve the dual objective function by increasing yi by some ϵ > 0. More speciﬁcally, we can increase yi Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
24
An introduction to approximation algorithms
y←0 I←∅ ∪ while there exists ei ∈ / j∈I Sj do Increase ∑the dual variable yi until there is some ℓ with ei ∈ Sℓ such that j:ej ∈Sℓ yj = wℓ I ← I ∪ {ℓ} Algorithm 1.1: Primaldual algorithm for the set cover problem. ( ) ∑ by minj:ei ∈Sj wj − k:ek ∈Sj yk , so that the constraint becomes tight for the subset Sj that attains the minimum. Additionally, the modiﬁed dual solution remains feasible. Thus, we can add j to T , and element ei is now covered by the sets in T . We repeat this process until T is a set cover. Since an additional element ei is covered each time, the process is repeated at most n times. To complete the description of the algorithm, we need to give only an initial dual feasible solution. We can use the solution yi = 0 for each i = 1, . . . , n; this is feasible since each wj , j = 1, . . . , m, is nonnegative. A formal description is given in Algorithm 1.1. This yields the following theorem. Theorem 1.9: Algorithm 1.1 is an f approximation algorithm for the set cover problem. This type of algorithm is called a primaldual algorithm by analogy with the primaldual method used in other combinatorial algorithms. Linear programming problems, network ﬂow problems, and shortest path problems (among others) all have primaldual optimization algorithms; we will see an example of a primaldual algorithm for the shortest st path problem in Section 7.3. Primaldual algorithms start with a dual feasible solution, and use dual information to infer a primal, possibly infeasible, solution. If the primal solution is indeed infeasible, the dual solution is modiﬁed to increase the value of the dual objective function. The primaldual method has been very useful in designing approximation algorithms, and we will discuss it extensively in Chapter 7. We observe again that this particular algorithm allows us to have an a fortiori guarantee for each input, since we can compare the value of the solution obtained with the value of the dual solution generated by the algorithm. This ratio is guaranteed to be at most f by the proof above, but it might be signiﬁcantly better.
1.6
A greedy algorithm
At this point, the reader might be forgiven for feeling a slight sense of futility: we have examined several techniques for designing approximation algorithms for the set cover problem, and they have all led to the same result, an approximation algorithm with performance guarantee f . But, as in life, perseverance and some amount of cleverness often pay dividends in designing approximation algorithms. We show in this section that a type of algorithm called a greedy algorithm gives an approximation algorithm with a performance guarantee that is often signiﬁcantly better than f . Greedy algorithms work by making a sequence of decisions; each decision is made to optimize that particular decision, even though this sequence of locally optimal (or “greedy”) decisions might not lead to a globally optimal solution. The advantage of greedy algorithms is that they are typically very easy to implement, and hence greedy algorithms are a commonly used heuristic, even when they have no performance guarantee. We now present a very natural greedy algorithm for the set cover problem. Sets are chosen Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.6
A greedy algorithm
25
I←∅ Sˆj ← Sj ∀j while I is not a set cover do w ℓ ← arg minj:Sˆj ̸=∅ ˆj I ← I ∪ {ℓ} Sˆj ← Sˆj − Sℓ
Sj 
∀j
Algorithm 1.2: A greedy algorithm for the set cover problem.
in a sequence of rounds. In each round, we choose the set that gives us the most bang for the buck; that is, the set that minimizes the ratio of its weight to the number of currently uncovered elements it contains. In the event of a tie, we pick an arbitrary set that achieves the minimum ratio. We continue choosing sets until all elements are covered. Obviously, this will yield a polynomialtime algorithm, since there can be no more than m rounds, and in each we compute O(m) ratios, each in constant time. A formal description is given in Algorithm 1.2. Before we state the theorem, we need some notation and a useful mathematical fact. Let Hk denote the kth harmonic number: that is, Hk = 1 + 12 + 13 + · · · + k1 . Note that Hk ≈ ln k. The following fact is one that we will use many times in the course of this book. It can be proven with simple algebraic manipulations. Fact 1.10: Given positive numbers a1 , . . . , ak and b1 , . . . , bk , then ∑k ai ai ai ≤ max min ≤ ∑i=1 . k i=1,...,k bi i=1,...,k bi i=1 bi Theorem 1.11: Algorithm 1.2 is an Hn approximation algorithm for the set cover problem. Proof. The basic intuition for the analysis of the algorithm is as follows. Let OPT denote the value of an optimal solution to the set cover problem. We know that an optimal solution covers all n elements with a solution of weight OPT; therefore, there must be some subset that covers its elements with an average weight of at most OPT /n. Similarly, after k elements have been covered, the optimal solution can cover the remaining n − k elements with a solution of weight OPT, which implies that there is some subset that covers its remaining uncovered elements with an average weight of at most OPT /(n − k). So in general the greedy algorithm pays about ∑n OPT1/(n − k + 1) to cover the kth uncovered element, giving a performance guarantee of k=1 n−k+1 = Hn . We now formalize this intuition. Let nk denote the number of elements that remain uncovered at the start of the kth iteration. If the algorithm takes ℓ iterations, then n1 = n, and we set nℓ+1 = 0. Pick an arbitrary iteration k. Let Ik denote the indices of the sets chosen in iterations 1 through k − 1, and for each j = 1, . . . , m, let Sˆ∪ j denote the set of uncovered elements in Sj at the start of this iteration; that is, Sˆj = Sj − p∈Ik Sp . Then we claim that for the set j chosen in the kth iteration, wj ≤
nk − nk+1 OPT . nk
(1.5)
Given the claimed inequality (1.5), we can prove the theorem. Let I contain the indices of the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
26
An introduction to approximation algorithms
sets in our ﬁnal solution. Then ∑ j∈I
wj
≤
ℓ ∑ nk − nk+1
nk
k=1
OPT
) ℓ ( ∑ 1 1 1 ≤ OPT · + + ··· + nk nk − 1 nk+1 + 1 = OPT ·
k=1 n ∑ i=1
(1.6)
1 i
= Hn · OPT, where the inequality (1.6) follows from the fact that n1k ≤ nk1−i for each 0 ≤ i < nk . To prove the claimed inequality (1.5), we shall ﬁrst argue that in the kth iteration, wj OPT ≤ . ˆ nk j:Sˆj ̸=∅ Sj  min
(1.7)
If we let O contain the indices of the sets in an optimal solution, then inequality (1.7) follows from Fact 1.10, by observing that ∑ wj OPT OPT j∈O wj min ≤∑ , =∑ ≤ ˆj  nk Sˆj  Sˆj  j:Sˆj ̸=∅ S j∈O
j∈O
∪ where the last inequality follows from the fact that since O is a set cover, the set j∈O Sˆj must include all remaining nk uncovered elements. Let j index a subset that minimizes this ratio, so w ˆ that ˆj ≤ OPT nk . If we add the subset Sj to our solution, then there will be Sj  fewer uncovered Sj 
elements, so that nk+1 = nk − Sˆj . Thus, wj ≤
Sˆj  OPT nk − nk+1 = OPT . nk nk
We can improve the performance guarantee of the algorithm slightly by using the dual of the linear programming relaxation in the analysis. Let g be the maximum size of any subset ∗ Sj ; that is, g = maxj Sj . Recall that ZLP is the optimum value of the linear programming relaxation for the set cover problem. The following theorem immediately implies that the greedy ∗ ≤ OPT . algorithm is an Hg approximation algorithm, since ZLP ∑ ∗ . Theorem 1.12: Algorithm 1.2 returns a solution indexed by I such that j∈I wj ≤ Hg · ZLP ∑ Proof. To prove the theorem, we will construct an infeasible dual solution y such that j∈I wj = ∑n We will then show that y ′ = H1g y is a feasible dual solution. By the weak duality i=1 yi . ∑ ∑n ∑ ∑n ′ ∗ , so that theorem, ni=1 yi′ ≤ ZLP i=1 yi ≤ Hg · OPT. We will see at j∈I wj = i=1 yi = Hg the end of the proof the reason we choose to divide the infeasible dual solution y by Hg . The name dual ﬁtting has been given to this technique of constructing an infeasible dual solution whose value is equal to the value of the primal solution constructed, and such that scaling the dual solution by a single value makes it feasible. We will return to this technique in Section 9.4. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.6
A greedy algorithm
27
To construct the infeasible dual solution y, suppose we choose to add subset Sj to our solution in iteration k. Then for each ei ∈ Sˆj , we set yi = wj /Sˆj . Since each ei ∈ Sˆj is uncovered in iteration k, and is then covered for the remaining iterations of the algorithm (because we added subset Sj to the solution), the dual variable yi is set to a value exactly once; ∑ in particular, it is set in the iteration in which element ei is covered. Furthermore, wj = i:ei ∈Sˆj yi ; that is, the weight of the subset Sj chosen in the kth iteration is equal to the sum of the duals yi of∑the uncovered ∑n elements that are covered in the kth iteration. This immediately implies that j∈I wj = i=1 yi . It remains to prove that the dual solution y ′ = H1g y is feasible. We must show that for each ∑ subset Sj , i:ei ∈Sj yi′ ≤ wj . Pick an arbitrary subset Sj . Let ak be the number of elements in this subset that are still uncovered at the beginning of the kth iteration, so that a1 = Sj , and aℓ+1 = 0. Let Ak be the uncovered elements of Sj covered in the kth iteration, so that Ak  = ak − ak+1 . If subset Sp is chosen in the kth iteration, then for each element ei ∈ Ak covered in the kth iteration, wj wp ≤ , yi′ = ˆ H g ak Hg Sp  where Sˆp is the set of uncovered elements of Sp at the beginning of the kth iteration. The inequality follows because if Sp is chosen in the kth iteration, it must minimize the ratio of its weight to the number of uncovered elements it contains. Thus, ∑
yi′ =
ℓ ∑ ∑
yi′
k=1 i:ei ∈Ak
i:ei ∈Sj
≤
ℓ ∑
(ak − ak+1 )
k=1
≤ ≤
wj Hg ak
ℓ wj ∑ ak − ak+1 Hg ak k=1 ) ℓ ( wj ∑ 1 1 1 + + ··· + Hg ak ak − 1 ak+1 + 1 k=1
Sj  wj ∑ 1 ≤ Hg i i=1 wj = H Hg Sj  ≤ wj ,
where the ﬁnal inequality follows because Sj  ≤ g. Here we see the reason for scaling the dual solution by Hg , since we know that HSj  ≤ Hg for all sets j. It turns out that no approximation algorithm for the set cover problem with performance guarantee better than Hn is possible, under an assumption slightly stronger than P ̸= NP. Theorem 1.13: If there exists a c ln napproximation algorithm for the unweighted set cover problem for some constant c < 1, then there is an O(nO(log log n) )time deterministic algorithm for each NPcomplete problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
28
An introduction to approximation algorithms
Theorem 1.14: There exists some constant c > 0 such that if there exists a c ln napproximation algorithm for the unweighted set cover problem, then P = NP. We will discuss results of this sort at more length in Chapter 16; in Theorem 16.32 we show how a slightly weaker version of these results can be derived. Results of this type are sometimes called hardness theorems, as they show that it is NPhard to provide nearoptimal solutions for a certain problem with certain performance guarantees. The f approximation algorithms for the set cover problem imply a 2approximation algorithm for the special case of the vertex cover problem. No algorithm with a better constant performance guarantee is known at this point in time. Additionally, two hardness theorems, Theorems 1.15 and 1.16 below, have been shown. Theorem √ 1.15: If there exists an αapproximation algorithm for the vertex cover problem with α < 10 5 − 21 ≈ 1.36, then P = NP. The following theorem mentions a conjecture called the unique games conjecture that we will discuss more in Section 13.3 and Section 16.5. The conjecture is roughly that a particular problem (called unique games) is NPhard. Theorem 1.16: Assuming the unique games conjecture, if there exists an αapproximation algorithm for the vertex cover problem with constant α < 2, then P = NP. Thus, assuming P ̸= NP and the NPcompleteness of the unique games problem, we have found essentially the best possible approximation algorithm for the vertex cover problem.
1.7
A randomized rounding algorithm
In this section, we consider one ﬁnal technique for devising an approximation algorithm for the set cover problem. Although the algorithm is slower and has no better guarantee than the greedy algorithm of the previous section, we include it here because it introduces the notion of using randomization in approximation algorithms, an idea we will cover in depth in Chapter 5. As with the algorithm in Section 1.3, the algorithm will solve a linear programming relaxation for the set cover problem, and then round the fractional solution to an integral solution. Rather than doing so deterministically, however, the algorithm will do so randomly using a technique called randomized rounding. Let x∗ be an optimal solution to the LP relaxation. We would like to round fractional values of x∗ to either 0 or 1 in such a way that we obtain a solution x ˆ to the integer programming formulation of the set cover problem without increasing the cost too much. The central idea of randomized rounding is that we interpret the fractional value x∗j as the probability that x ˆj should be set to 1. Thus, each subset Sj is included in our solution ∗ with probability xj , where these m events (that Sj is included in our solution) are independent random events. We assume some basic knowledge of probability theory throughout this text; for those who need some additional background, see the notes at the end of the chapter for suggested references. Let Xj be a random variable that is 1 if subset Sj is included in the solution, and 0 otherwise. Then the expected value of the solution is m m m ∑ ∑ ∑ ∗ E wj Xj = wj Pr[Xj = 1] = wj x∗j = ZLP , j=1
j=1
j=1
or just the value of the linear programming relaxation, which is no more than OPT! As we will see, however, it is quite likely that the solution is not a set cover. Nevertheless, this illustrates Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.7
A randomized rounding algorithm
29
why randomized rounding can provide such good approximation algorithms in some cases, and we will see further examples of this in Chapter 5. Let us now calculate the probability that a given element ei is not covered by this procedure. This is the probability that none of the subsets containing ei are included in the solution, or ∏ (1 − x∗j ). j:ei ∈Sj
We can bound this probability by using the fact that 1 − x ≤ e−x for any x, where e is the base of the natural logarithm. Then ∏ Pr[ei not covered] = (1 − x∗j ) j:ei ∈Sj
∏
≤
∗
e−xj
j:ei ∈Sj ∑ − j:e
i ∈Sj
= e
x∗j
≤ e−1 ,
∑ where the ﬁnal inequality follows from the LP constraint that j:ei ∈Sj x∗j ≥ 1. Although e−1 is an upper bound on the probability that a given element is not covered, it is possible to approach this bound arbitrarily closely, so in the worst case it is quite likely that this randomized rounding procedure does not produce a set cover. How small would this probability have to be in order for it to be very likely that a set cover is produced? And perhaps even more fundamentally, what is the “right” notion of “very likely”? The latter question has a number of possible answers; one natural way to think of the situation is to impose a guarantee in keeping with our focus on polynomialtime algorithms. Suppose that, for any constant c, we could devise a polynomialtime algorithm whose chance of failure is at most an inverse polynomial n−c ; then we say that we have an algorithm that works with high probability. To be more precise, we would have a family of algorithms, since it might be necessary to give progressively slower algorithms, or ones with worse performance guarantees, to achieve analogously more failsafe results. If we could devise a randomized procedure such that Pr[ei not covered] ≤ n1c for some constant c ≥ 2, then Pr[there exists an uncovered element] ≤
n ∑
Pr[ei not covered] ≤
i=1
1 , nc−1
and we would have a set cover with high probability. In fact, we can achieve such a bound in the following way: for each subset Sj , we imagine a coin that comes up heads with probability x∗j , and we ﬂip the coin c ln n times. If it comes up heads in any of the c ln n trials, we include Sj in our solution, otherwise not. Thus, the probability that Sj is not included is (1 − x∗j )c ln n . Furthermore, ∏ Pr[ei not covered] = (1 − x∗j )c ln n j:ei ∈Sj
≤
∏
j:ei ∈Sj −(c ln n)
= e 1 ≤ , nc
∗
e−xj (c ln n) ∑ j:ei ∈Sj
x∗j
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
30
An introduction to approximation algorithms
as desired. We now need to prove only that the algorithm has a good expected value given that it produces a set cover. Theorem 1.17: The algorithm is a randomized O(ln n)approximation algorithm that produces a set cover with high probability. Proof. Let pj (x∗j ) be the probability that a given subset Sj is included in the solution as a function of x∗j . By construction of the algorithm, we know that pj (x∗j ) = 1 − (1 − x∗j )c ln n . Observe that if x∗j ∈ [0, 1] and c ln n ≥ 1, then we can bound the derivative p′j at x∗j by p′j (x∗j ) = (c ln n)(1 − x∗j )(c ln n)−1 ≤ (c ln n). Then since pj (0) = 0, and the slope of the function pj is bounded above by c ln n on the interval [0,1], pj (x∗j ) ≤ (c ln n)x∗j on the interval [0,1]. If Xj is a random variable that is 1 if the subset Sj is included in the solution, and 0 otherwise, then the expected value of the random procedure is m m ∑ ∑ E wj Xj = wj Pr[Xj = 1] j=1
j=1
≤
m ∑
wj (c ln n)x∗j
j=1
= (c ln n)
m ∑
∗ wj x∗j = (c ln n)ZLP .
j=1
However, we would like to bound the expected value of the solution given that a set cover is produced. Let F be the event that the solution obtained by the procedure is a feasible set cover, and let F¯ be the complement of this event. We know from the previous discussion that 1 Pr[F ] ≥ 1 − nc−1 , and we also know that m m m ∑ ∑ ∑ E wj Xj = E wj Xj F Pr[F ] + E wj Xj F¯ Pr[F¯ ]. j=1 j=1 j=1 Since wj ≥ 0 for all j,
Thus,
wj Xj F¯ ≥ 0. E j=1
E wj Xj F = j=1
m ∑
≤
m ∑
m m ∑ ∑ 1 E wj Xj − E wj Xj F¯ Pr[F¯ ] Pr[F ] j=1 j=1 m ∑ 1 ·E wj Xj Pr[F ] j=1
≤ ≤
∗ (c ln n)ZLP 1 1 − nc−1 ∗ 2c(ln n)ZLP
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.7
A randomized rounding algorithm
31
for n ≥ 2 and c ≥ 2. While in this case there is a simpler and faster approximation algorithm that achieves a better performance guarantee, we will see in Chapter 5 that sometimes randomized algorithms are simpler to describe and analyze than deterministic algorithms. In fact, most of the randomized algorithms we present in this book can be derandomized: that is, a deterministic variant of them can be created that achieves the expected performance guarantee of the randomized algorithm. However, these deterministic algorithms are sometimes more complicated to describe. In addition, there are some cases in which the deterministic variant is easy to state, but the only way in which we know how to analyze the algorithm is by analyzing a corresponding randomized algorithm. This brings us to the end of our introduction to approximation algorithms. In subsequent chapters, we will look at the techniques introduced here — as well as a few others — in greater depth, and see their application to many other problems.
Exercises 1.1 In the set ∑ cover problem, the goal is to ﬁnd a collection of subsets indexed by I that minimizes j∈I wj such that ∪ Sj = E. j∈I Consider the partial ∑ cover problem, in which one ﬁnds a collection of subsets indexed by I that minimizes j∈I wj such that ∪ Sj ≥ pE, j∈I where 0 < p < 1 is some constant. (a) Give a polynomialtime algorithm to ﬁnd a solution to the partial cover problem in which the value is no more than c(p) · OPT, where c(p) is a constant that depends on p, and OPT is the value of the optimal solution to the set cover problem. (b) Give an f (p)approximation algorithm for the partial cover problem, such that f is nondecreasing in p and f (1) ≤ HE . 1.2 In the directed Steiner tree problem, we are given as input a directed graph G = (V, A), nonnegative costs cij ≥ 0 for arcs (i, j) ∈ A, a root vertex r ∈ V , and a set of terminals T ⊆ V . The goal is to ﬁnd a minimumcost tree such that for each i ∈ T there exists a directed path from r to i. Prove that for some constant c there can be no c log T approximation algorithm for the directed Steiner tree problem, unless P = NP. 1.3 In the metric asymmetric traveling salesman problem, we are given as input a complete directed graph G = (V, A) with costs cij ≥ 0 for all arcs (i, j) ∈ A, such that the arc costs obey the triangle inequality: for all i, j, k ∈ V , we have that cij + cjk ≥ cik . The goal is to ﬁnd a tour of minimum cost, that is, a directed cycle that contains each vertex exactly once, such that the sum of the cost of the arcs in the cycle is minimized. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
32
An introduction to approximation algorithms
One approach to ﬁnding an approximation algorithm for this problem is to ﬁrst ﬁnd a minimumcost strongly connected Eulerian subgraph of the input graph. A directed graph is strongly connected if for any pair of vertices i, j ∈ V there is a path from i to j and a path from j to i. A directed graph is Eulerian if the indegree of each vertex equals its outdegree. Given a strongly connected Eulerian subgraph of the input to the problem, it is possible to use a technique called “shortcutting” (discussed in Section 2.4) to turn this into a tour of no greater cost by using the triangle inequality. One way to ﬁnd a strongly connected Eulerian subgraph is as follows: We ﬁrst ﬁnd a minimum meancost cycle in the graph. A minimum meancost cycle is a directed cycle that minimizes the ratio of the cost of the arcs in the cycle to the number of arcs in the cycle. Such a cycle can be found in polynomial time. We then choose one vertex of the cycle arbitrarily, remove all other vertices of the cycle from the graph, and repeat. We do this until only one vertex of the graph is left. Consider the subgraph consisting of all the arcs from all the cycles found. (a) Prove that the subgraph found by the algorithm is a strongly connected Eulerian subgraph of the input graph. (b) Prove that the cost of this subgraph is at most 2Hn · OPT, where n = V  and OPT is the cost of the optimal tour. Conclude that this algorithm is a 2Hn approximation algorithm for the metric asymmetric traveling salesman problem. 1.4 In the uncapacitated facility location problem, we have a set of clients D and a set of facilities F . For each client j ∈ D and facility i ∈ F , there is a cost cij of assigning client j to facility i. Furthermore, there is a cost fi associated with each facility i ∈ F . The goal of the problem is to choose a subset of facilities F ′ ⊆ F so as to minimize the total cost of the facilities in F ′ and the cost of assigning each client∑ j ∈ D to the ∑ nearest facility in F ′ . In other words, we wish to ﬁnd F ′ so as to minimize i∈F ′ fi + j∈D mini∈F ′ cij . (a) Show that there exists some c such that there is no (c ln D)approximation algorithm for the uncapacitated facility location problem unless P = NP. (b) Give an O(ln D)approximation algorithm for the uncapacitated facility location problem. 1.5 Consider the vertex cover problem. (a) Prove that any extreme point of the linear program ∑ minimize wi xi i∈V
subject to xi + xj ≥ 1, xi ≥ 0,
∀(i, j) ∈ E, i ∈ V,
has the property that xi ∈ {0, 12 , 1} for all i ∈ V . (Recall that an extreme point x is a feasible solution that cannot be expressed as λx1 + (1 − λ)x2 for 0 < λ < 1 and feasible solutions x1 and x2 distinct from x.) (b) Give a 32 approximation algorithm for the vertex cover problem when the input graph is planar. You may use the facts that polynomialtime LP solvers return extreme points, and that there is a polynomialtime algorithm to 4color any planar graph (i.e., the algorithm assigns each vertex one of four colors such that for any edge (i, j) ∈ E, vertices i and j have been assigned diﬀerent colors). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
1.7
A randomized rounding algorithm
33
1.6 In the nodeweighted Steiner tree problem, we are given as input an undirected graph G = (V, E), node weights wi ≥ 0 for all i ∈ V , edge costs ce ≥ 0 for all e ∈ E, and a set of terminals T ⊆ V . The cost of a tree is the sum of the weights of the nodes plus the sum of the costs of the edges in the tree. The goal of the problem is to ﬁnd a minimumweight tree that spans all the terminals in T . (a) Show that there exists some c such that there is no (c ln T )approximation algorithm for the nodeweighted Steiner tree problem unless P = NP. (b) Give a greedy O(ln T )approximation algorithm for the nodeweighted Steiner tree problem.
Chapter Notes The term “approximation algorithm” was coined by David S. Johnson [179] in an inﬂuential and prescient 1974 paper. However, earlier papers had proved the performance guarantees of heuristics, including a 1967 paper of Erd˝os [99] on the maximum cut problem (to be discussed in Section 6.2), a 1966 paper of Graham [142] on a scheduling problem (to be discussed in Section 2.3), and a 1964 paper of Vizing [284] on the edge coloring problem (to be discussed in Section 2.7). Johnson’s paper gave an O(log n)approximation algorithm for the unweighted set cover problem, as well as approximation algorithms for the maximum satisﬁability problem (to be discussed in Section 5.1), vertex coloring (Sections 5.12, 6.5, and 13.2), and the maximum clique problem. At the end of the paper, Johnson [179] speculates about the approximability of these various problems: The results described in this paper indicate a possible classiﬁcation of optimization problems as to the behavior of their approximation algorithms. Such a classiﬁcation must remain tentative, at least until the existence of polynomialtime algorithms for ﬁnding optimal solutions has been proved or disproved. In the meantime, many questions can be asked. Are there indeed O(log n) coloring algorithms? Are there any clique ﬁnding algorithms better than O(nϵ ) for all ϵ > 0? Where do other optimization problems ﬁt into the scheme of things? What is it that makes algorithms for diﬀerent problems behave in the same way? Is there some stronger kind of reducibility than the simple polynomial reducibility that will explain these results, or are they due to some structural similarity between the problems as we deﬁne them? And what other types of behavior and ways of analyzing and measuring it are possible? (p. 278) There has been substantial work done in attempting to answer these questions in the decades since Johnson’s paper appeared, with signiﬁcant progress; for instance, Theorem 1.4 shows that no clique algorithm of the kind Johnson mentions is possible unless P = NP. Other books on approximation algorithms are available, including the textbooks of Ausiello, Crescenzi, Gambosi, Kann, MarchettiSpaccamela, and Protasi [27] and of Vazirani [283], and the collection of surveys edited by Hochbaum [162]. Many books on algorithms and combinatorial optimization now contain sections on approximation algorithms, including the textbooks of Bertsimas and Tsitsiklis [47], Cook, Cunningham, Pulleyblank, and Schrijver [81], Cormen, Leiserson, Rivest, and Stein [82], Kleinberg and Tardos [198], and Korte and Vygen [203]. For solid introductions to linear programming, we suggest the books of Bertsimas and Tsitsiklis [47], Chv´atal [79] and Ferris, Mangasarian, and Wright [112]. Bertsekas and Tsitsiklis [45], Durrett [93, 94], and Ross [256] provide basic introductions to probability theory; the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
34
An introduction to approximation algorithms
ﬁrst few chapters of the book of Mitzenmacher and Upfal [226] provide a brief introduction to probability theory in the context of computer algorithms. The antivirus application of the set cover problem mentioned is due to Kephart, Sorkin, Arnold, Chess, Tesauro, and White [188]. Theorem 1.3 on the nonexistence of approximation schemes for problems in MAX SNP is due to Arora, Lund, Motwani, Sudan, and Szegedy [19], building on earlier work of Feige, Goldwasser, Lov´asz, Safra, and Szegedy [108] and Arora and Safra [23]. Theorem 1.4 on the hardness of approximating the maximum clique problem is due to H˚ astad [158], with a strengthening due to Zuckerman [296]. The LP rounding algorithm of Section 1.3 and the dual rounding algorithm of Section 1.4 are due to Hochbaum [160]. The primaldual algorithm of Section 1.5 is due to BarYehuda and Even [35]. The greedy algorithm and the LPbased analysis of Section 1.6 are due to Chv´atal [78]. The randomized rounding algorithm of Section 1.7 is apparently folklore. Johnson [179] and Lov´asz [218] give earlier greedy O(log n)approximation algorithms for the unweighted set cover problem. Theorem 1.13 on the hardness of approximating the set cover problem is due to Lund and Yannakakis [220], with a strengthening due to Bellare, Goldwasser, Lund, and Russell [43]. Theorem 1.14 on the hardness of the set cover problem is due to Feige [107]. Theorem 1.15 on the hardness of the vertex cover problem is due to Dinur and Safra [91], while Theorem 1.16, which uses the unique games conjecture, is due to Khot and Regev [194]. Exercise 1.3 is an unpublished result of Kleinberg and Williamson. The algorithm in Exercise 1.4 is due to Hochbaum [161]. Nemhauser and Trotter [231] show that all extreme points of the linear programming relaxation of vertex cover have value {0, 12 , 1} (used in Exercise 1.5). Exercise 1.6 is due to Klein and Ravi [196].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 2
Greedy algorithms and local search
In this chapter, we will consider two standard and related techniques for designing algorithms and heuristics, namely, greedy algorithms and local search algorithms. Both algorithms work by making a sequence of decisions that optimize some local choice, though these local choices might not lead to the best overall solution. In a greedy algorithm, a solution is constructed step by step, and at each step of the algorithm the next part of the solution is constructed by making some decision that is locally the best possible. In Section 1.6, we gave an example of a greedy algorithm for the set cover problem that constructs a set cover by repeatedly choosing the set that minimizes the ratio of its weight to the number of currently uncovered elements it contains. A local search algorithm starts with an arbitrary feasible solution to the problem, and then checks if some small, local change to the solution results in an improved objective function. If so, the change is made. When no further change can be made, we have a locally optimal solution, and it is sometimes possible to prove that such locally optimal solutions have value close to that of the optimal solution. Unlike other approximation algorithm design techniques, the most straightforward implementation of a local search algorithm typically does not run in polynomial time. The algorithm usually requires some restriction to the local changes allowed in order to ensure that enough progress is made during each improving step so that a locally optimal solution is found in polynomial time. Thus, while both types of algorithm optimize local choices, greedy algorithms are typically primal infeasible algorithms: they construct a solution to the problem during the course of the algorithm. Local search algorithms are primal feasible algorithms: they always maintain a feasible solution to the problem and modify it during the course of the algorithm. Both greedy algorithms and local search algorithms are extremely popular choices for heuristics for NPhard problems. They are typically easy to implement and have good running times in practice. In this chapter, we will consider greedy and local search algorithms for scheduling problems, clustering problems, and others, including the most famous problem in combinatorial optimization, the traveling salesman problem. Because greedy and local search algorithms are natural choices for heuristics, some of these algorithms were among the very ﬁrst approximation algorithms devised; in particular, the greedy algorithm for the parallel machine scheduling problem in Section 2.3 and the greedy algorithm for edge coloring in Section 2.7 were both given and analyzed in the 1960s, before the concept of NPcompleteness was invented. 35
36
Greedy algorithms and local search
Job 1 Job 2
0
2
3
Job 3
7
Time
Figure 2.1: An instance of a schedule for the onemachine scheduling problem in which p1 = 2, r1 = 0, p2 = 1, r2 = 2, p3 = 4, r3 = 1. In this schedule, C1 = 2, C2 = 3, and C3 = 7. If the deadlines for the jobs are such that d1 = −1, d2 = 1, and d3 = 10, then L1 = 2 − (−1) = 3, L2 = 3 − 1 = 2, and L3 = 7 − 10 = −3, so that Lmax = L1 = 3.
2.1
Scheduling jobs with deadlines on a single machine
One of the most common types of problems in combinatorial optimization is that of creating a schedule. We are given some type of work that must be done, and some resources to do the work, and from this we must create a schedule to complete the work that optimizes some objective; perhaps we want to ﬁnish all the work as soon as possible, or perhaps we want to make sure that the average time at which we complete the various pieces of work is as small as possible. We will often consider the problem of scheduling jobs (the work) on machines (the resources). We start this chapter by considering one of the simplest possible versions of this problem. Suppose that there are n jobs to be scheduled on a single machine, where the machine can process at most one job at a time, and must process a job until its completion once it has begun processing; suppose that each job j must be processed for a speciﬁed pj units of time, where the processing of job j may begin no earlier than a speciﬁed release date rj , j = 1, . . . , n. We assume that the schedule starts at time 0, and each release date is nonnegative. Furthermore, assume that each job j has a speciﬁed due date dj , and if we complete its processing at time Cj , then its lateness Lj is equal to Cj − dj ; we are interested in scheduling the jobs so as to minimize the maximum lateness, Lmax = maxj=1,...,n Lj . A sample instance of this problem is shown in Figure 2.1. Unfortunately, this problem is NPhard, and in fact, even deciding if there is a schedule for which Lmax ≤ 0 (i.e., deciding if all jobs can be completed by their due date) is strongly NPhard (the reader unfamiliar with strong NPhardness can consult Appendix B). Of course, this is a problem that we often encounter in everyday life, and many of us schedule our lives with the following simple greedy heuristic: focus on the task with the earliest due date. We will show that in certain circumstances this is a provably good thing to do. However, we ﬁrst argue that as stated, this optimization problem is not particularly amenable to obtaining nearoptimal solutions. If there were a ρapproximation algorithm, then for any input with optimal value 0, the algorithm must still ﬁnd a schedule of objective function value at most ρ · 0 = 0, and hence (given the NPhardness result stated above) this would imply that P = NP. (There is the further complication of what to do if the objective function of the optimal solution is negative!) One easy workaround to this is to assume that all due dates are negative, which implies that the optimal value is always positive. We shall give a 2approximation algorithm for this special case. We ﬁrst provide a good lower bound on the optimal value ∑ for this scheduling problem. Let S denote a subset of jobs, and let r(S) = minj∈S rj , p(S) = j∈S pj , and d(S) = maxj∈S dj . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.2
The kcenter problem
37
Let L∗max denote the optimal value. Lemma 2.1: For each subset S of jobs, L∗max ≥ r(S) + p(S) − d(S). Proof. Consider the optimal schedule, and view this simply as a schedule for the jobs in the subset S. Let job j be the last job in S to be processed. Since none of the jobs in S can be processed before r(S), and in total they require p(S) time units of processing, it follows that job j cannot complete any earlier than time r(S) + p(S). The due date of job j is d(S) or earlier, and so the lateness of job j in this schedule is at least r(S) + p(S) − d(S); hence, L∗max ≥ r(S) + p(S) − d(S). A job j is available at time t if its release date rj ≤ t. We consider the following natural algorithm: at each moment that the machine is idle, start processing next an available job with the earliest due date. This is known as the earliest due date (EDD) rule. Theorem 2.2: The EDD rule is a 2approximation algorithm for the problem of minimizing the maximum lateness on a single machine subject to release dates with negative due dates. Proof. Consider the schedule produced by the EDD rule, and let job j be a job of maximum lateness in this schedule; that is, Lmax = Cj − dj . Focus on the time Cj in this schedule; ﬁnd the earliest point in time t ≤ Cj such that the machine was processing without any idle time for the entire period [t, Cj ). Several jobs may be processed in this time interval; we require only that the machine not be idle for some interval of positive length within it. Let S be the set of jobs that are processed in the interval [t, Cj ). By our choice of t, we know that just prior to t, none of these jobs were available (and clearly at least one job in S is available at time t); hence, r(S) = t. Furthermore, since only jobs in S are processed throughout this time interval, p(S) = Cj − t = Cj − r(S). Thus, Cj ≤ r(S) + p(S); since d(S) < 0, we can apply Lemma 2.1 to get that L∗max ≥ r(S) + p(S) − d(S) ≥ r(S) + p(S) ≥ Cj . (2.1) On the other hand, by applying Lemma 2.1 with S = {j}, L∗max ≥ rj + pj − dj ≥ −dj .
(2.2)
Adding inequalities (2.1) and (2.2), we see that the maximum lateness of the schedule computed is Lmax = Cj − dj ≤ 2L∗max , which completes the proof of the theorem.
2.2
The kcenter problem
The problem of ﬁnding similarities and dissimilarities in large amounts of data is ubiquitous: companies wish to group customers with similar purchasing behavior, political consultants group precincts by their voting behavior, and search engines group webpages by their similarity of topic. Usually we speak of clustering data, and there has been extensive study of the problem of ﬁnding good clusterings. Here we consider a particular variant of clustering, the kcenter problem. In this problem, we are given as input an undirected, complete graph G = (V, E), with a distance dij ≥ 0 Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
38
Greedy algorithms and local search
Pick arbitrary i ∈ V S ← {i} while S < k do j ← arg maxj∈V d(j, S) S ← S ∪ {j} Algorithm 2.1: A greedy 2approximation algorithm for the kcenter problem.
2∗ 1∗
2
1
3 3∗
Figure 2.2: An instance of the kcenter problem where k = 3 and the distances are given by the Euclidean distances between points. The execution of the greedy algorithm is shown; the nodes 1, 2, 3 are the nodes selected by the greedy algorithm, whereas the nodes 1∗ , 2∗ , 3∗ are the three nodes in an optimal solution. between each pair of vertices i, j ∈ V . We assume dii = 0, dij = dji for each i, j ∈ V , and that the distances obey the triangle inequality: for each triple i, j, l ∈ V , it is the case that dij + djl ≥ dil . In this problem, distances model similarity: vertices that are closer to each other are more similar, whereas those farther apart are less similar. We are also given a positive integer k as input. The goal is to ﬁnd k clusters, grouping together the vertices that are most similar into clusters together. In this problem, we will choose a set S ⊆ V , S = k, of k cluster centers. Each vertex will assign itself to its closest cluster center, grouping the vertices into k diﬀerent clusters. For the kcenter problem, the objective is to minimize the maximum distance of a vertex to its cluster center. Geometrically speaking, the goal is to ﬁnd the centers of k diﬀerent balls of the same radius that cover all points so that the radius is as small as possible. More formally, we deﬁne the distance of a vertex i from a set S ⊆ V of vertices to be d(i, S) = minj∈S dij . Then the corresponding radius for S is equal to maxi∈V d(i, S), and the goal of the kcenter problem is to ﬁnd a set of size k of minimum radius. In later chapters we will consider other objective functions, such ∑ as minimizing the sum of distances of vertices to their cluster centers, that is, minimizing i∈V d(i, S). This is called the kmedian problem, and we will consider it in Sections 7.7 and 9.2. We shall also consider another variant on clustering called correlation clustering in Section 6.4. We give a greedy 2approximation algorithm for the kcenter problem that is simple and intuitive. Our algorithm ﬁrst picks a vertex i ∈ V arbitrarily, and puts it in our set S of cluster centers. Then it makes sense for the next cluster center to be as far away as possible from all the other cluster centers. Hence, while S < k, we repeatedly ﬁnd a vertex j ∈ V that determines the current radius (or in other words, for which the distance d(j, S) is maximized) and add it to S. Once S = k, we stop and return S. Our algorithm is given in Algorithm 2.1. An execution of the algorithm is shown in Figure 2.2. We will now prove that the algorithm Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.3
Scheduling jobs on identical parallel machines
39
is a good approximation algorithm. Theorem 2.3: Algorithm 2.1 is a 2approximation algorithm for the kcenter problem. Proof. Let S ∗ = {j1 , . . . , jk } denote the optimal solution, and let r∗ denote its radius. This solution partitions the nodes V into clusters V1 , . . . , Vk , where each point j ∈ V is placed in Vi if it is closest to ji among all of the points in S ∗ (and ties are broken arbitrarily). Each pair of points j and j ′ in the same cluster Vi are at most 2r∗ apart: by the triangle inequality, the distance djj ′ between them is at most the sum of djji , the distance from j to the center ji , plus dji j ′ , the distance from the center ji to j ′ (that is, djj ′ ≤ djji + dji j ′ ); since djji and dj ′ ji are each at most r∗ , we see that djj ′ is at most 2r∗ . Now consider the set S ⊆ V of points selected by the greedy algorithm. If one center in S is selected from each cluster of the optimal solution S ∗ , then every point in V is clearly within 2r∗ of some selected point in S. However, suppose that the algorithm selects two points within the same cluster. That is, in some iteration, the algorithm selects a point j ∈ Vi , even though the algorithm had already selected a point j ′ ∈ Vi in an earlier iteration. Again, the distance between these two points is at most 2r∗ . The algorithm selects j in this iteration because it is currently the furthest from the points already in S. Hence, all points are within a distance of at most 2r∗ of some center already selected for S. Clearly, this remains true as the algorithm adds more centers in subsequent iterations, and we have proved the theorem. The instance in Figure 2.2 shows that this analysis is tight. We shall argue next that this result is the best possible; if there exists a ρapproximation algorithm with ρ < 2, then P = NP. To see this, we consider the dominating set problem, which is NPcomplete. In the dominating set problem, we are given a graph G = (V, E) and an integer k, and we must decide if there exists a set S ⊆ V of size k such that each vertex is either in S, or adjacent to a vertex in S. Given an instance of the dominating set problem, we can deﬁne an instance of the kcenter problem by setting the distance between adjacent vertices to 1, and nonadjacent vertices to 2: there is a dominating set of size k if and only if the optimal radius for this kcenter instance is 1. Furthermore, any ρapproximation algorithm with ρ < 2 must always produce a solution of radius 1 if such a solution exists, since any solution of radius ρ < 2 must actually be of radius 1. This implies the following theorem. Theorem 2.4: There is no αapproximation algorithm for the kcenter problem for α < 2 unless P = NP.
2.3
Scheduling jobs on identical parallel machines
In Section 2.1, we considered the problem of scheduling jobs on a single machine to minimize lateness. Here we consider a variation on that problem, in which we now have multiple machines and no release dates, but our goal is to minimize the time at which all jobs are ﬁnished. Suppose that there are n jobs to be processed, and there are m identical machines (running in parallel) to which each job may be assigned. Each job j = 1, . . . , n, must be processed on one of these machines for pj time units without interruption, and each job is available for processing at time 0. Each machine can process at most one job at a time. The aim is to complete all jobs as soon as possible; that is, if job j completes at a time Cj (presuming that the schedule starts at time 0), then we wish to minimize Cmax = maxj=1,...,n Cj , which is often called the makespan or length of the schedule. An equivalent view of the same problem is as a loadbalancing problem: there are n items, each of a given weight pj , and they are to be distributed among m machines; Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
40
Greedy algorithms and local search
1 2 Machines 3 4 5 Time
1 2 Machines 3 4 5 Time
Figure 2.3: An example of a local move in the local search algorithm for scheduling jobs on parallel machines. The gray job on machine 2 ﬁnishes last in the schedule on the top, but the schedule can be improved by moving the gray job to machine 4. No further local moves are possible after this one since again the gray job ﬁnishes last. the aim is to assign each item to one machine so to minimize the maximum total weight assigned to one machine. This scheduling problem has the property that even the simplest algorithms compute reasonably good solutions. In particular, we will show that both a local search algorithm and a very simple greedy algorithm ﬁnd solutions that have makespan within a factor of 2 of the optimum. In fact, the analyses of these two algorithms are essentially identical. Local search algorithms are deﬁned by a set of local changes or local moves that change one feasible solution to another. The simplest local search procedure for this scheduling problem works as follows: Start with any schedule; consider the job ℓ that ﬁnishes last; check whether or not there exists a machine to which it can be reassigned that would cause this job to ﬁnish earlier. If so, transfer job ℓ to this other machine. We can determine whether to transfer job ℓ by checking if there exists a machine that ﬁnishes its currently assigned jobs earlier than Cℓ − pℓ . The local search algorithm repeats this procedure until the last job to complete cannot be transferred. An illustration of this local move is shown in Figure 2.3. In order to analyze the performance of this local search algorithm, we ﬁrst provide some ∗ . Since each job must be natural lower bounds on the length of an optimal schedule, Cmax processed, it follows that ∗ Cmax ≥ max pj . (2.3) ∑n
j=1,...,n
On the other hand, there is, in total, P = j=1 pj units of processing to accomplish, and only m machines to do this work. Hence, on average, a machine will be assigned P/m units of work, and consequently, there must exist one machine that is assigned at least that much work. Thus, ∗ Cmax ≥
n ∑
pj /m.
(2.4)
j=1
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.3
Scheduling jobs on identical parallel machines
41
Consider the solution produced by the local search algorithm. Let ℓ be a job that completes last in this ﬁnal schedule; the completion time of job ℓ, Cℓ , is equal to this solution’s objective function value. By the fact that the algorithm terminated with this schedule, every other machine must be busy from time 0 until the start of job ℓ at time Sℓ = Cℓ −pℓ . We can partition the schedule into two disjoint time intervals, from time 0 until Sℓ , and the time during which ∗ . Now consider job ℓ is being processed. By (2.3), the latter interval has length at most Cmax the former time interval; we know that each machine is busy processing jobs throughout this period. The total amount of work being ∑nprocessed in this interval is mSℓ , which is clearly no more than the total work to be done, j=1 pj . Hence, Sℓ ≤
n ∑
pj /m.
(2.5)
j=1 ∗ . But now, we see that the length of the By combining this with (2.4), we see that Sℓ ≤ Cmax ∗ schedule before the start of job ℓ is at most Cmax , as is the length of the schedule afterward; in ∗ . total, the makespan of the schedule computed is at most 2Cmax Now consider the running time of this algorithm. This local search procedure has the property that the value of Cmax for the sequence of schedules produced, iteration by iteration, never increases (it can remain the same, but then the number of machines that achieve the maximum value decreases). One natural assumption to make is that when we transfer a job to another machine, then we reassign that job to the machine that is currently ﬁnishing earliest. We will analyze the running time of this variant instead. Let Cmin be the completion time of a machine that completes all its processing the earliest. One consequence of focusing on the new variant is that Cmin never decreases (and if it remains the same, then the number of machines that achieve this minimum value decreases). We argue next that this implies that we never transfer a job twice. Suppose this claim is not true, and consider the ﬁrst time that a job j is transferred twice, say, from machine i to i′ , and later then to i∗ . When job j is reassigned from machine i to machine i′ , it then starts at Cmin for the current schedule. Similarly, when ′ . Furthermore, no job j is reassigned from machine i′ to i∗ , it then starts at the current Cmin ′ change occurred to the schedule on machine i in between these two moves for job j. Hence, ′ Cmin must be strictly smaller than Cmin (in order for the transfer to be an improving move), but this contradicts our claim that the Cmin value is nondecreasing over the iterations of the local search algorithm. Hence, no job is transferred twice, and after at most n iterations, the algorithm must terminate. We have thus shown the following theorem.
Theorem 2.5: The local search procedure for scheduling jobs on identical parallel machines is a 2approximation algorithm. In fact, it is not hard to see that the analysis of the approximation ratio can be reﬁned slightly. In deriving the inequality (2.5), we included job ℓ among the work to be done prior to the start of job ℓ. Hence, we actually derived that ∑ Sℓ ≤ pj /m, j̸=ℓ
and hence the total length of the schedule produced is at most pℓ +
∑ j̸=ℓ
( pj /m =
1 1− m
) pℓ +
n ∑
pj /m.
j=1
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
42
Greedy algorithms and local search
By applying the two lower bounds (2.3) and (2.4) to these two terms, we see that the schedule 1 ∗ . Of course, the diﬀerence between this bound and 2 is signiﬁcant has length at most (2− m )Cmax only if there are very few machines. Another natural algorithm to compute a schedule is a greedy algorithm that assigns the jobs as soon as there is machine availability to process them: whenever a machine becomes idle, then one of the remaining jobs is assigned to start processing on that machine. This algorithm is often called the list scheduling algorithm, since one can equivalently view the algorithm as ﬁrst ordering the jobs in a list (arbitrarily), and the next job to be processed is the one at the top of the list. Another viewpoint, from the loadbalancing perspective, is that the next job on the list is assigned to the machine that is currently the least heavily loaded. It is in this sense that one can view the algorithm as a greedy algorithm. The analysis of this algorithm is now quite trivial; if one uses this schedule as the starting point for the local search procedure, that algorithm would immediately declare that the solution cannot be improved! To see this, consider a job ℓ that is (one of the jobs) last to complete its processing. Each machine is busy until Cℓ − pℓ , since otherwise we would have assigned job ℓ to that other machine. Hence, no transfers are possible. Theorem 2.6: The list scheduling algorithm for the problem of minimizing the makespan on m identical parallel machines is a 2approximation algorithm. It is not hard to obtain a stronger result by improving this list scheduling algorithm. Not all lists yield the same schedule, and it is natural to use an additional greedy rule that ﬁrst sorts the jobs in nonincreasing order. One way to view the results of Theorems 2.5 and 2.6 is that the relative error in the length of the schedule produced is entirely due to the length of the last job to ﬁnish. If that job is short, then the error is not too big. This greedy algorithm is called the longest processing time rule, or LPT. Theorem 2.7: The longest processing time rule is a 4/3approximation algorithm for scheduling jobs to minimize the makespan on identical parallel machines. Proof. Suppose that the theorem is false, and consider an input that provides a counterexample to the theorem. For ease of notation, assume that p1 ≥ · · · ≥ pn . First, we can assume that the last job to complete is indeed the last (and smallest) job in the list. This follows without loss of generality: any counterexample for which the last job ℓ to complete is not the smallest can yield a smaller counterexample, simply by omitting all of the jobs ℓ + 1, . . . , n; the length of the schedule produced is the same, and the optimal value of the reduced input can be no larger. Hence the reduced input is also a counterexample. So we know that the last job to complete in the schedule is job n. If this is a counterexample, ∗ /3, then the analysis of Theorem 2.6 implies what do we know about pn (= pℓ )? If pℓ ≤ Cmax ∗ that the schedule length is at most (4/3)Cmax , and so this is not a counterexample. Hence, we know that in this purported counterexample, job n (and therefore all of the jobs) has a ∗ /3. This has the following simple corollary. processing requirement strictly greater than Cmax In the optimal schedule, each machine may process at most two jobs (since otherwise the total ∗ ). processing assigned to that machine is more than Cmax However, we have now reduced our assumed counterexample to the point where it simply cannot exist. For inputs of this structure, we have the following lemma. Lemma 2.8: For any input to the problem of minimizing the makespan on identical parallel machines for which the processing requirement of each job is more than onethird the optimal makespan, the longest processing time rule computes an optimal schedule. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.4
The traveling salesman problem
43
This lemma can be proved by some careful case checking, and we defer this to an exercise (Exercise 2.2). However, the consequence of the lemma is clear; no counterexample to the theorem can exist, and hence the theorem must be true. In Section 3.2, we will see that it is possible to give a polynomialtime approximation scheme for this problem.
2.4
The traveling salesman problem
In the traveling salesman problem, or TSP, there is a given set of cities {1, 2, . . . , n}, and the input consists of a symmetric n by n matrix C = (cij ) that speciﬁes the cost of traveling from city i to city j. By convention, we assume that the cost of traveling from any city to itself is equal to 0, and costs are nonnegative; the fact that the matrix is symmetric means that the cost of traveling from city i to city j is equal to the cost of traveling from j to i. (The asymmetric traveling salesman problem, where the restriction that the cost matrix be symmetric is relaxed, has already made an appearance in Exercise 1.3.) If we instead view the input as an undirected complete graph with a cost associated with each edge, then a feasible solution, or tour, consists of a Hamiltonian cycle in this graph; that is, we specify a cyclic permutation of the cities or, equivalently, a traversal of the cities in the order k(1), k(2), . . . , k(n), where each city j is listed as a unique image k(i). The cost of the tour is equal to ck(n)k(1) +
n−1 ∑
ck(i)k(i+1) .
i=1
Observe that each tour has n distinct representations, since it does not matter which city is selected as the one in which the tour starts. The traveling salesman problem is one of the most wellstudied combinatorial optimization problems, and this is certainly true from the point of view of approximation algorithms as well. There are severe limits on our ability to compute nearoptimal tours, and we start with a discussion of these results. It is NPcomplete to decide whether a given undirected graph G = (V, E) has a Hamiltonian cycle. An approximation algorithm for the TSP can be used to solve the Hamiltonian cycle problem in the following way: Given a graph G = (V, E), form an input to the TSP by setting, for each pair i, j, the cost cij equal to 1 if (i, j) ∈ E, and equal to n + 2 otherwise. If there is a Hamiltonian cycle in G, then there is a tour of cost n, and otherwise each tour costs at least 2n + 1. If there were to exist a 2approximation algorithm for the TSP, then we could use this algorithm to distinguish graphs with Hamiltonian cycles from those without any: run the approximation algorithm on the new TSP input, and if the tour computed has cost at most 2n, then there exists a Hamiltonian cycle in G, and otherwise there does not. Of course, there is nothing special about setting the cost for the “nonedges” to be n + 2; setting the cost to be αn + 2 has a similarly inﬂated consequence, and we obtain an input to the TSP of polynomial size provided that, for example, α = O(2n ). As a result, we obtain the following theorem. Theorem 2.9: For any α > 1, there does not exist an αapproximation algorithm for the traveling salesman problem on n cities, provided P ̸= NP. In fact, the existence of an O(2n )approximation algorithm for the TSP would similarly imply that P = NP. But is this the end of the story? Clearly not. A natural assumption to make about the input to the TSP is to restrict attention to those inputs that are metric; that is, for each triple i, j, k ∈ V , Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
44
Greedy algorithms and local search
k
k
j
j i
i
Figure 2.4: Illustration of a greedy step of the nearest addition algorithm. we have that the triangle inequality cik ≤ cij + cjk holds. This assumption rules out the construction used in the reduction for the Hamiltonian cycle problem above; the nonedges can be given cost at most 2 if we want the triangle inequality to hold, and this value is too small to yield a nontrivial nonapproximability result. We next give three approximation algorithms for this metric traveling salesman problem. Here is a natural greedy heuristic to consider for the traveling salesman problem; this is often referred to as the nearest addition algorithm. Find the two closest cities, say, i and j, and start by building a tour on that pair of cities; the tour consists of going from i to j and then back to i again. This is the ﬁrst iteration. In each subsequent iteration, we extend the tour on the current subset S by including one additional city, until we include the full set of cities. In each iteration, we ﬁnd a pair of cities i ∈ S and j ̸∈ S for which the cost cij is minimum; let k be the city that follows i in the current tour on S. We add j to S, and insert j into the current tour between i and k. An illustration of this algorithm is shown in Figure 2.4. The crux of the analysis of this algorithm is the relationship of this algorithm to Prim’s algorithm for the minimum spanning tree in an undirected graph. A spanning tree of a connected graph G = (V, E) is a minimal subset of edges F ⊆ E such that each pair of nodes in G is connected by a path using edges only in F . A minimum spanning tree is a spanning tree for which the total edge cost is minimized. Prim’s algorithm computes a minimum spanning tree by iteratively constructing a set S along with a tree T , starting with S = {v} for some (arbitrarily chosen) node v ∈ V and T = (S, F ) with F = ∅. In each iteration, it determines the edge (i, j) such that i ∈ S and j ̸∈ S is of minimum cost, and adds the edge (i, j) to F . Clearly, this is the same sequence of vertex pairs identiﬁed by the nearest addition algorithm. Furthermore, there is another important relationship between the minimum spanning tree problem and the traveling salesman problem. Lemma 2.10: For any input to the traveling salesman problem, the cost of the optimal tour is at least the cost of the minimum spanning tree on the same input. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.4
The traveling salesman problem
45
Proof. The proof is quite simple. For any input with n ≥ 2, start with the optimal tour. Delete any one edge from the tour. The result is a spanning tree (albeit a very special one), and this costs no more than the optimal tour. But the minimum spanning tree must cost no more than this special tree. Hence, the cost of the minimum spanning tree is at most the cost of the optimal tour. By combining these observations, we can obtain the following result with just a bit more work. Theorem 2.11: The nearest addition algorithm for the metric traveling salesman problem is a 2approximation algorithm. Proof. Let S2 , S3 , . . . , Sn = {1, . . . , n} be the subsets identiﬁed at the end of each iteration of the nearest addition algorithm (where Sℓ  = ℓ), and let F = {(i2 , j2 ), (i3 , j3 ), . . . , (in , jn )}, where (iℓ , jℓ ) is the edge identiﬁed in iteration ℓ − 1 (with iℓ ∈ Sℓ−1 , ℓ = 3, . . . , n). As indicated above, we also know that ({1, . . . , n}, F ) is a minimum spanning tree for the original input, when viewed as a complete undirected graph with edge costs. Thus, if OPT is the optimal value for the TSP input, then n ∑ OPT ≥ ciℓ jℓ . ℓ=2
The cost of the tour on the ﬁrst two nodes i2 and j2 is exactly 2ci2 j2 . Consider an iteration in which a city j is inserted between cities i and k in the current tour. How much does the length of the tour increase? An easy calculation gives cij + cjk − cik . By the triangle inequality, we have that cjk ≤ cji + cik or, equivalently, cjk − cik ≤ cji . Hence, the increase in cost in this iteration is at most cij + cji = 2cij . Thus, overall, we know that the ﬁnal tour has cost at most 2
n ∑
ciℓ jℓ ≤ 2 OPT,
ℓ=2
and the theorem is proved. In fact, this algorithm can be viewed from another perspective. Although this new perspective deviates from viewing the algorithm as a “greedy” procedure, this approach ultimately leads to a better algorithm. First we need some graphtheoretic preliminaries. A graph is said to be Eulerian if there exists a permutation of its edges of the form (i0 , i1 ), (i1 , i2 ), . . . , (ik−1 , ik ), (ik , i0 ); we will call this permutation a traversal of the edges, since it allows us to visit every edge exactly once. A graph is Eulerian if and only if it is connected and each node has even degree (where the degree of a node v is the number of edges with v as one of its endpoints). Furthermore, if a graph is Eulerian, one can easily construct the required traversal of the edges, given the graph. To ﬁnd a good tour for a TSP input, suppose that we ﬁrst compute a minimum spanning tree (for example, by Prim’s algorithm). Suppose that we then replace each edge by two copies of itself. The resulting (multi)graph has cost at most 2 OPT and is Eulerian. We can construct a tour of the cities from the Eulerian traversal of the edges, (i0 , i1 ), (i1 , i2 ), . . . , (ik−1 , ik ), (ik , i0 ). Consider the sequence of nodes, i0 , i1 , . . . , ik , and remove all but the ﬁrst occurrence of each city in this sequence. This yields a tour containing each city exactly once (assuming we then return to i0 at the end). To bound the length of this tour, consider two consecutive cities in this tour, iℓ and im . We have omitted iℓ+1 , . . . , im−1 because these cities have already been visited “earlier” in the tour. However, by the triangle inequality, the cost of the edge ciℓ ,im can be upper bounded by the total cost of the edges traversed in the Eulerian traversal between Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
46
Greedy algorithms and local search
iℓ and im , that is, the total cost of the edges (iℓ , iℓ+1 ), . . . , (im−1 , im ). In total, the cost of the tour is at most the total cost of all of the edges in the Eulerian graph, which is at most 2 OPT. Hence, we have also analyzed this doubletree algorithm. Theorem 2.12: The doubletree algorithm for the metric traveling salesman problem is a 2approximation algorithm. This technique of “skipping over” previously visited cities and bounding the cost of the resulting tour in terms of the total cost of all the edges is sometimes called shortcutting. The bigger message of the analysis of the doubletree algorithm is also quite useful; if we can eﬃciently construct an Eulerian subgraph of the complete input graph, for which the total edge cost is at most α times the optimal value of the TSP input, then we have derived an αapproximation algorithm as well. This strategy can be carried out to yield a 3/2approximation algorithm. Consider the output from the minimum spanning tree computation. This graph is certainly not Eulerian, since any tree must have nodes of degree one, but it is possible that not many nodes have odd degree. Let O be the set of odddegree nodes in the minimum spanning tree. For any graph, the sum of its node degrees must be even, since each edge in the graph contributes 2 to this total. The total degree of the evendegree nodes must also be even (since we are adding a collection of even numbers), but then the total degree of the odddegree nodes must also be even. In other words, we must have an even number of odddegree nodes; O = 2k for some positive integer k. Suppose that we pair up the nodes in O: (i1 , i2 ), (i3 , i4 ), . . . , (i2k−1 , i2k ). Such a collection of edges that contain each node in O exactly once is called a perfect matching of O. One of the classic results of combinatorial optimization is that given a complete graph (on an even number of nodes) with edge costs, it is possible to compute the perfect matching of minimum total cost in polynomial time. Given the minimum spanning tree, we identify the set O of odddegree nodes with even cardinality, and then compute a minimumcost perfect matching on O. If we add this set of edges to our minimum spanning tree, we have constructed an Eulerian graph on our original set of cities: it is connected (since the spanning tree is connected) and has even degree (since we added a new edge incident to each node of odd degree in the spanning tree). As in the doubletree algorithm, we can shortcut this graph to produce a tour of no greater cost. This algorithm is known as Christoﬁdes’ algorithm. Theorem 2.13: Christoﬁdes’ algorithm for the metric traveling salesman problem is a 3/2approximation algorithm. Proof. We want to show that the edges in the Eulerian graph produced by the algorithm have total cost at most 32 OPT. We know that the minimum spanning tree edges have total cost at most OPT. So we need only show that the perfect matching on O has cost at most OPT /2. This is surprisingly simple. First observe that there is a tour on just the nodes in O of total cost at most OPT. This again uses the shortcutting argument. Start with the optimal tour on the entire set of cities, and if for two cities i and j, the optimal tour between i and j contains only cities that are not in O, then include edge (i, j) in the tour on O. Each edge in the tour corresponds to disjoint paths in the original tour, and hence by the triangle inequality, the total length of the tour on O is no more than the length of the original tour. Now consider this “shortcut” tour on the node set O. Color these edges red and blue, alternating colors as the tour is traversed. This partitions the edges into two sets, the red set and the blue set; each of these is a perfect matching on the node set O. In total, these two Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.5
Maximizing ﬂoat in bank accounts
47
S←∅ while S < k do i ← arg maxi∈B v(S ∪ {i}) − v(S) S ← S ∪ {i} return S Algorithm 2.2: A greedy approximation algorithm for the ﬂoat maximization problem. edge sets have cost at most OPT. Thus, the cheaper of these two sets has cost at most OPT /2. Hence, there is a perfect matching on O of cost at most OPT /2. Therefore, the algorithm to ﬁnd the minimumcost perfect matching must ﬁnd a matching of cost at most OPT /2, and this completes the proof of the theorem. Remarkably, no better approximation algorithm for the metric traveling salesman problem is known. However, substantially better algorithms might yet be found, since the strongest negative result is as follows. Theorem 2.14: Unless P = NP, for any constant α < algorithm for the metric TSP exists.
220 219
≈ 1.0045, no αapproximation
We can give better approximation algorithms for the problem in special cases. In Section 10.1, we will see that it is possible to obtain a polynomialtime approximation scheme in the case that cities correspond to points in the Euclidean plane and the cost of traveling between two cities is equal to the Euclidean distance between the corresponding two points.
2.5
Maximizing ﬂoat in bank accounts
In the days before quick electronic check clearing, it was often advantageous for large corporations to maintain checking accounts in various locations in order to maximize ﬂoat. The ﬂoat is the time between making a payment by check and the time that the funds for that payment are deducted from the company’s banking account. During that time, the company can continue to accrue interest on the money. Float can also be used by scam artists for check kiting: covering a deﬁcit in the checking account in one bank by writing a check against another account in another bank that also has insuﬃcient funds — then a few days later covering this deﬁcit with a check written against the ﬁrst account. We can model the problem of maximizing ﬂoat as follows. Suppose we wish to open up to k bank accounts so as to maximize our ﬂoat. Let B be the set of banks where we can potentially open accounts, and let P be the set of payees to whom we regularly make payments. Let vij ≥ 0 be the value of the ﬂoat created by paying payee j ∈ P from bank account i ∈ B; this may take into account the amount of time it takes for a check written to j to clear at i, the interest rate at bank i, and other factors. Then we wish to ﬁnd a set S ⊆ B of banks at which to open accounts such that S ≤ k. Clearly we will pay payee j ∈ P ∑ from the account i ∈ S that maximizes vij . So we wish to ﬁnd S ⊆ B, S ≤ k, that maximizes j∈P maxi∈S vij . We deﬁne v(S) to be the value of this objective function for S ⊆ B. A natural greedy algorithm is as follows: we start with S = ∅, and while S < k, ﬁnd the bank i ∈ B that most increases the objective function, and add it to S. This algorithm is summarized in Algorithm 2.2. We will show that this algorithm has a performance guarantee of 1 − 1e . To do this, we require the following lemma. We let O denote an optimal solution, so that O ⊆ B and O ≤ k. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
48
Greedy algorithms and local search
Lemma 2.15: Let S be the set of banks at the start of some iteration of Algorithm 2.2, and let i ∈ B be the bank chosen in the iteration. Then v(S ∪ {i}) − v(S) ≥
1 (v(O) − v(S)). k
To get some intuition of why this is true, consider the optimal solution O. We can allocate shares of the value of the objective function v(O) to each bank i ∈ O: the value vij for each j ∈ P can be allocated to a bank i ∈ O that attains the maximum of maxi∈O vij . Since O ≤ k, some bank i ∈ O is allocated at least v(O)/k. So after choosing the ﬁrst bank i to add to S, we have v({i}) ≥ v(O)/k. Intuitively speaking, there is also another bank i′ ∈ O that is allocated at least a 1/k fraction of whatever wasn’t allocated to the ﬁrst bank, so that there is an i′ such that v(S ∪ {i′ }) − v(S) ≥ k1 (v(O) − v(S)), and so on. Given the lemma, we can prove the performance guarantee of the algorithm. Theorem 2.16: Algorithm 2.2 gives a (1 − 1e )approximation algorithm for the ﬂoat maximization problem. Proof. Let S t be our greedy solution after t iterations of the algorithm, so that S 0 = ∅ and S = S k . Let O be( an optimal solution. We set v(∅) = 0. Note that Lemma 2.15 implies that ) 1 1 t−1 t v(S ) ≥ k v(O) + 1 − k v(S ). By applying this inequality repeatedly, we have v(S) = v(S k ) ( ) 1 1 ≥ v(O) + 1 − v(S k−1 ) k k ( )( ( ) ) 1 1 1 1 k−2 ≥ v(O) + 1 − v(O) + 1 − v(S ) k k k k ( ) ( ) ( ) ) ( 1 2 1 k−1 v(O) 1 + 1− + ··· + 1 − ≥ 1+ 1− k k k k ) ( k v(O) 1 − 1 − k1 ) ( = · k 1 − 1 − k1 ( ) ) ( 1 k = v(O) 1 − 1 − k ( ) 1 ≥ v(O) 1 − , e where in the ﬁnal inequality we use the fact that 1 − x ≤ e−x , setting x = 1/k. To prove Lemma 2.15, we ﬁrst prove the following. Lemma 2.17: For the objective function v, for any X ⊆ Y and any ℓ ∈ / Y, v(Y ∪ {ℓ}) − v(Y ) ≤ v(X ∪ {ℓ}) − v(X). Proof. Consider any payee j ∈ P . Either j is paid from the same bank account in both X ∪ {ℓ} and X, or it is paid by ℓ from X ∪ {ℓ} and some other bank in X. Consequently, ) ∑ { ( )} ∑( v(X ∪ {ℓ}) − v(X) = max vij − max vij = max 0, vℓj − max vij . (2.6) j∈P
i∈X∪{ℓ}
i∈X
j∈P
i∈X
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.6
Finding minimumdegree spanning trees
Similarly, v(Y ∪ {ℓ}) − v(Y ) =
∑ j∈P
{ ( )} max 0, vℓj − max vij . i∈Y
49
(2.7)
Now since X ⊆ Y for a given j ∈ P , maxi∈Y vij ≥ maxi∈X vij , so that { ( )} { ( )} max 0, vℓj − max vij ≤ max 0, vℓj − max vij . i∈Y
i∈X
By summing this inequality over all j ∈ P and using the equalities (2.6) and (2.7), we obtain the desired result. The property of the value function v that we have just proved is one that plays a central role in a number of algorithmic settings, and is often called submodularity, though the usual deﬁnition of this property is somewhat diﬀerent (see Exercise 2.10). This deﬁnition captures the intuitive property of decreasing marginal beneﬁts: as the set includes more elements, the marginal value of adding a new element decreases. Finally, we prove Lemma 2.15. Proof of Lemma 2.15. Let O − S = {i1 , . . . , ip }. Note that since O − S ≤ O ≤ k, then p ≤ k. Since adding more bank accounts can only increase the overall value of the solution, we have that v(O) ≤ v(O ∪ S), and a simple rewriting gives v(O ∪ S) = v(S) +
p ∑
[v(S ∪ {i1 , . . . , ij }) − v(S ∪ {i1 , . . . , ij−1 })] .
j=1
By applying Lemma 2.17, we can upper bound the righthand side by v(S) +
p ∑
[v(S ∪ {ij }) − v(S)] .
j=1
Since the algorithm chooses i ∈ B to maximize v(S ∪ {i}) − v(S), we have that for any j, v(S ∪ {i}) − v(S) ≥ v(S ∪ {ij }) − v(S). We can use this bound to see that v(O) ≤ v(O ∪ S) ≤ v(S) + p[v(S ∪ {i}) − v(S)] ≤ v(S) + k[v(S ∪ {i}) − v(S)]. This inequality can be rewritten to yield the inequality of the lemma, and this completes the proof. This greedy approximation algorithm and its analysis can be extended to similar problems in which the objective function v(S) is given by a set of items S, and is monotone and submodular. We leave the deﬁnition of these terms and the proofs of the extensions to Exercise 2.10.
2.6
Finding minimumdegree spanning trees
We now turn to a local search algorithm for the problem of minimizing the maximum degree of a spanning tree. The problem we consider is the following: given a graph G = (V, E) we wish to ﬁnd a spanning tree T of G so as to minimize the maximum degree of nodes in T . We Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
50
Greedy algorithms and local search
u
u y
v
w
y
v
w
Figure 2.5: Illustration of a local move for minimizing the maximum degree of a spanning tree. The bold solid lines are in the tree, and the dashed lines are graph edges not in the tree. will call this the minimumdegree spanning tree problem. This problem is NPhard. A special type of spanning tree of a graph is a path that visits all nodes of the graph; this is called a Hamiltonian path. A spanning tree has maximum degree two if and only if it is a Hamiltonian path. Furthermore, deciding if a graph G has a Hamiltonian path is NPcomplete. Thus, we have the following theorem. Theorem 2.18: It is NPcomplete to decide whether or not a given graph has a minimumdegree spanning tree of maximum degree two. For a given graph G, let T ∗ be the spanning tree that minimizes the maximum degree, and let OPT be the maximum degree of T ∗ . We will give a polynomialtime local search algorithm that ﬁnds a tree T with maximum degree at most 2 OPT +⌈log2 n⌉, where n = V  is the number of vertices in the graph. To simplify notation, throughout this section we will let ℓ = ⌈log2 n⌉. The local search algorithm starts with an arbitrary spanning tree T . We will give a local move to change T into another spanning tree in which the degree of some vertex has been reduced. Let dT (u) be the degree of u in T . The local move picks a vertex u and tries to reduce its degree by looking at all edges (v, w) that are not in T but if added to T create a cycle C containing u. Suppose max(dT (v), dT (w)) ≤ dT (u) − 2. For example, consider the graph in Figure 2.5, in which the edges of the tree T are shown in bold. In this case, the degree of node u is 5, but those of v and w are 3. Let T ′ be the result of adding (v, w) and removing an edge from C incident to u. In the example, if we delete edge (u, y), then the degrees of u, v, and w will all be 4 after the move. The conditions ensure that this move provides improvement in general; the degree of u is reduced by one in T ′ (that is, dT ′ (u) = dT (u)−1) and the degrees of v and w in T ′ are not greater than the reduced degree of u; that is, max(dT ′ (v), dT ′ (w)) ≤ dT ′ (u). The local search algorithm carries out local moves on nodes that have high degree. It makes sense for us to carry out a local move to reduce the degree of any node, since it is possible that if we reduce the degree of a lowdegree node, it may make possible another local move that reduces the degree of a highdegree node. However, we do not know how to show that such an algorithm terminates in polynomial time. To get a polynomialtime algorithm, we apply the local moves only to nodes whose degree is relatively high. Let ∆(T ) be the maximum degree of T ; that is, ∆(T ) = maxu∈V dT (u). The algorithm picks a node in T that has degree at least Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.6
Finding minimumdegree spanning trees
51
∆(T ) − ℓ and attempts to reduce its degree using the local move. If there is no move that can reduce the degree of any node having degree between ∆(T ) − ℓ and ∆(T ), then the algorithm stops. We say that the algorithm has found a locally optimal tree. By applying local moves only to nodes whose degree is between ∆(T ) − ℓ and ∆(T ), we will be able to show that the algorithm runs in polynomial time. We now need to prove two things. First, we need to show that any locally optimal tree has maximum degree at most 2 OPT +ℓ. Second, we need to show that we can ﬁnd a locally optimal tree in polynomial time. For most approximation algorithms, the proof that the algorithm runs in polynomial time is relatively straightforward, but this is often not the case for local search algorithms. In fact, we usually need to restrict the set of local moves in order to prove that the algorithm converges to a locally optimal solution in polynomial time. Here we do this by restricting the local moves to apply only to nodes with high degree. Theorem 2.19: Let T be a locally optimal tree. Then ∆(T ) ≤ 2 OPT +ℓ, where ℓ = ⌈log2 n⌉. Proof. We ﬁrst explain how we will obtain a lower bound on OPT. Suppose that we remove k edges of the spanning tree. This breaks the tree into k + 1 diﬀerent connected components. Suppose we also ﬁnd a set of nodes S such that each edge in G connecting two of the k + 1 connected components is incident on a node in S. For example, consider the graph in Figure 2.6 that shows the connected components remaining after the bold edges are deleted, along with an appropriate choice of the set S. Observe that any spanning tree of the graph must have at least k edges with endpoints in diﬀerent components. Thus, the average degree of nodes in S is at least k/S for any spanning tree, and OPT ≥ k/S. Now we show how to ﬁnd the set of edges to remove and the set of nodes S so that we can apply this lower bound. Let Si be the nodes of degree at least i in the locally optimal tree T . We claim that for each Si , where i ≥ ∆(T ) − ℓ + 1, there are at least (i − 1)Si  + 1 distinct edges of T incident on the nodes of Si , and after removing these edges, each edge that connects distinct connected components is incident on a node of Si−1 . Furthermore, we claim there exists an i such that Si−1  ≤ 2Si , so that the value of OPT implied by removing these edges with S = Si−1 is OPT ≥
(i − 1)Si  + 1 (i − 1)Si  + 1 ≥ > (i − 1)/2 ≥ (∆(T ) − ℓ)/2. Si−1  2Si 
Rearranging terms proves the desired inequality. We turn to the proofs of the claims. We ﬁrst show that there must exist some i ≥ ∆(T )−ℓ+1 such that Si−1  ≤ 2Si . Suppose not. Then clearly S∆(T )−ℓ  > 2ℓ S∆(T )  or S∆(T )−ℓ  > nS∆(T )  ≥ n since S∆(T )  ≥ 1. This is a contradiction, since any Si can have at most n nodes. Now we show that there are at least (i − 1)Si  + 1 distinct edges of T incident on the nodes of Si , and after removing these edges, any edge connecting diﬀerent connected components is incident on a node of Si−1 . Figure 2.6 gives an example of this construction for i = 4. Each edge that connects distinct connected components after removing the edges of T incident on nodes of Si either must be one of the edges of T incident on Si , or must close a cycle C in T containing some node in Si . Because the tree is locally optimal, it must be the case that at least one of the endpoints has degree at least i − 1, and so is in Si−1 . In removing the edges in T incident on nodes in Si , there are at least iSi  edges incident on nodes in Si , since each node has degree at least i. At most Si  − 1 such edges can join two nodes in Si since T is a spanning tree. Thus, there are at least iSi  − (Si  − 1) distinct edges of T incident on the nodes Si , and this proves the claim. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
52
Greedy algorithms and local search
Figure 2.6: Illustration of lower bound on OPT. The vertices in S have white centers. If the bold edges are deleted, every potential edge for joining the resulting connected components has an endpoint in S.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.6
Finding minimumdegree spanning trees
53
Theorem 2.20: The algorithm ﬁnds a locally optimal tree in polynomial time. Proof. To prove that the algorithm runs in polynomial time, we will use a potential function argument. The idea of such an argument is that the function captures the current state of the algorithm, and that we can determine upper and lower bounds on this function for any feasible solution, as well as a lower bound on the amount that the function must decrease after each move. In this way, we can bound the number of moves possible before the algorithm must terminate, and of course, the resulting tree must therefore be locally optimal. ∑ For a tree T , we let the potential of T , Φ(T ), be Φ(T ) = v∈V 3dT (v) . Note that Φ(T ) ≤ n3∆(T ) , and so the initial potential is at most n3n . On the other hand, the lowest possible potential is for a Hamiltonian path, which has potential 2 · 3 + (n − 2)32 > n. We will show 2 that for each move, the potential function of the resulting tree is at most 1 − 27n 3 times the potential function previously. 4 After 27 2 n ln 3 local moves, the conditions above imply that the potential of the resulting tree is at most ( ) 27 n4 ln 3 2 2 1− · (n3n ) ≤ e−n ln 3 · (n3n ) = n, 27n3 using the fact that 1 − x ≤ e−x . Since the potential of a tree is greater than n, after O(n4 ) local moves there must be no further local moves possible, and the tree must be locally optimal. We must still prove the claimed potential reduction in each iteration. Suppose the algorithm reduces the degree of a vertex u from i to i − 1, where i ≥ ∆(T ) − ℓ, and adds an edge (v, w). Then the increase in the potential function due to increasing the degree of v and w is at most 2 · (3i−1 − 3i−2 ) = 4 · 3i−2 , since the degree of v and w can be increased to at most i − 1. The decrease in the potential function due to decreasing the degree of u is 3i − 3i−1 = 2 · 3i−1 . Observe that 3ℓ ≤ 3 · 3log2 n ≤ 3 · 22 log2 n = 3n2 . Therefore, the overall decrease in the potential function is at least 2 2 ∆(T ) 2 2 2 · 3i−1 − 4 · 3i−2 = 3i ≥ 3∆(T )−ℓ ≥ 3 ≥ Φ(T ). 9 9 27n2 27n3 Thus, for the resulting tree T ′ we have that Φ(T ′ ) ≤ (1 − proof.
2 )Φ(T ). 27n3
This completes the
By slightly adjusting the parameters within the same proof outline, we can actually prove a stronger result. Given some constant b > 1, suppose we perform local changes on nodes of degree at least ∆(T ) − ⌈logb n⌉. Then it is possible to show the following. Corollary 2.21: The local search algorithm runs in polynomial time and results in a spanning tree T such that ∆(T ) ≤ b OPT +⌈logb n⌉. In Section 9.3, we will prove a still stronger result: we can give a polynomialtime algorithm that ﬁnds a spanning tree T with ∆(T ) ≤ OPT +1. Given that it is NPhard to determine whether a spanning tree has degree exactly OPT, this is clearly the best possible result that can be obtained. In the next section, we give another result of this type for the edge coloring problem. In Section 11.2, we will show that there are interesting extensions of these results to the case of spanning trees with costs on the edges. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
54
Greedy algorithms and local search
Figure 2.7: A graph with a 3edgecoloring.
Figure 2.8: The Petersen graph. This graph is not 3edgecolorable.
2.7
Edge coloring
To conclude this chapter, we give an algorithm that has the elements of both a greedy algorithm and a local search algorithm: it attempts to make progress in a greedy way, but when blocked it makes local changes until progress can be made again. The algorithm is for the problem of ﬁnding an edge coloring of a graph. An undirected graph is kedgecolorable if each edge can be assigned exactly one of k colors in such a way that no two edges with the same color share an endpoint. We call the assignment of colors to edges a kedgecoloring. For example, Figure 2.7 shows a graph with a 3edgecoloring. An analogous notion of vertex coloring will be discussed in Sections 5.12, 6.5, and 13.2. For a given graph, we would like to obtain a kedgecoloring with k as small as possible. Let ∆ be the maximum degree of a vertex in the given graph. Clearly, we cannot hope to ﬁnd a kedgecoloring with k < ∆, since at least ∆ diﬀerent colors must be incident to any vertex of maximum degree. Note that this shows that the coloring given in Figure 2.7 is optimal. On the other hand, consider the example in Figure 2.8, which is called the Petersen graph; it is not too hard to show that this graph is not 3edgecolorable, and yet it is easy to color it with four colors. Furthermore, the following has been shown. Theorem 2.22: For graphs with ∆ = 3, it is NPcomplete to decide whether the graph is Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.7
Edge coloring
55
3edgecolorable or not. In this section, we give a polynomialtime algorithm that will ﬁnd a (∆ + 1)edgecoloring for any graph. Given the NPcompleteness result, it is clearly the best we can hope to do unless P = NP. We give the algorithm and its analysis in the proof of the theorem below. We repeatedly ﬁnd an uncolored edge (u, v) and attempt to color it with one of the ∆ + 1 colors. If no color is available such that coloring (u, v) would result in a (∆ + 1)edgecoloring, then we show that it is possible to locally change some of the edge colors in such a way that we can correctly color (u, v). Theorem 2.23: There is a polynomialtime algorithm to ﬁnd a (∆+1)edgecoloring of a graph. Proof. Our algorithm will start with a completely uncolored graph. In each iteration of the algorithm we will take some uncolored edge and color it. This will be done in such a way that the algorithm maintains a legal coloring of the graph at the beginning of each iteration of the main loop; that is, for any vertex v in the graph, no two colored edges incident on v have the same color, and at most ∆ + 1 distinct colors have been used. Clearly this is true initially, when the entire graph is uncolored. We show that this invariant is maintained throughout the course of each iteration. In the argument that follows, we say that a vertex v lacks color c if an edge of color c is not incident on v. We summarize the algorithm in Algorithm 2.3, and now explain it in detail. Let (u, v0 ) be the selected uncolored edge. We then construct a sequence of edges (u, v0 ), (u, v1 ), . . . and colors c0 , c1 , . . . . We will use this sequence to do some local recoloring of edges so that we can correctly color the uncolored edge. Note that since we are maintaining a legal coloring of ∆ + 1 colors, and the maximum degree is ∆, each vertex must lack at least one of the ∆ + 1 colors. To build this sequence of edges, consider the current vertex vi , starting initially with v0 ; if vi lacks a color that u also lacks, then we let ci be this color, and the sequence is complete. If not, then choose the color ci arbitrarily from among those that vi lacks; note that u will not lack this color. If this color has not appeared in the sequence c0 , c1 , . . . to this point, then we let vi+1 be the vertex such that (u, vi+1 ) is the edge incident to u of color ci , and this extends our sequence (sometimes called a fan sequence) one edge further. If we have that ci = cj , where j < i, then we also stop building our sequence. (See Figure 2.9 for an example of this construction.) We need to argue that this process terminates either in the case that u and vi lack the same color ci , or that we have ci = cj for some j < i. Let d be the number of edges incident to u that are currently colored. Suppose the sequence reaches vertex vd without terminating. If there is no color cd that both u and vd lack, then any color that vd lacks is one that u does not lack, which must be one of the colors on the edges (u, v1 ), . . . , (u, vd−1 ). Hence, the color we choose for cd must be the same as one of the previous colors c0 , . . . , cd−1 . Suppose we complete the sequence because we ﬁnd some ci that both u and vi lack. This case is easy: we recolor edges (u, vj ) with color cj for j = 0, . . . , i; call this shifting recoloring. This situation and the resulting recoloring are depicted in Figure 2.10. In eﬀect, we shift the uncolored edge to (u, vi ) and color it with ci since both u and vi lack ci . The recoloring is correct since we know that each vj lacks cj and for j < i, cj was incident on u previously via the edge (u, vj+1 ), which we now give another color. Now consider the remaining case, where we complete the sequence because ci = cj for some j < i. This situation and its recoloring are given in Figure 2.11. We shift the uncolored edge to (u, vj ) by recoloring edges (u, vk ) with color ck for 0 ≤ k < j and uncoloring edge (u, vj ); this is correct by the same argument as above. Now vi and vj lack the same color c = ci = cj . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
56
Greedy algorithms and local search
u
v0
v2
v1
v3
Figure 2.9: An example of building the fan sequence. Edge (u, v0 ) is uncolored. Vertex v0 lacks gray, but u does not lack gray due to (u, v1 ), so c0 is gray. Vertex v1 lacks black, but u does not lack black due to (u, v2 ), so c1 is black. Vertex v2 lacks the dashed color, but u does not lack dashed due to (u, v3 ), so c2 is dashed. Vertex v3 lacks black, so c3 is black, and the sequence repeats a color.
u
v0
v2
u
v3
v1
v0
v2
v3
v1
Figure 2.10: A slightly diﬀerent fan sequence and its recoloring. As before, edge (u, v0 ) is uncolored. Vertex v0 lacks black, but u does not lack black due to (u, v1 ), so c0 is black. Vertex v1 lacks gray, but u does not lack gray due to (u, v2 ), so c1 is gray. Vertex v2 lacks dashed, but u does not lack dashed due to (u, v3 ), so c2 is dashed. Vertex v3 lacks dotted, and u also lacks dotted, and thus c3 is dotted. Therefore, we shift colors as shown and color the edge (u, v3 ) with dotted.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.7
Edge coloring
u
v0
v2
57
u
v3
v1
v0
v2
v3
v1
Figure 2.11: The fan sequence from Figure 2.9. We start by shifting the uncolored edge to (u, v1 ). Now both v1 and v3 lack black. The dotted color can be the color cu that u lacks but that v1 and v3 do not lack. We let cu be a color that u lacks; by our selection (and the fact that we did not fall in the ﬁrst case), we know that both vi and vj do not lack cu . Consider the subgraph induced by taking all of the edges with colors c and cu ; since we have a legal coloring, this subgraph must be a collection of paths and simple cycles. Since each of u, vi , and vj has exactly one of these two colors incident to it, each is an endpoint of one of the path components, and since a path has only two endpoints, at least one of vi and vj must be in a diﬀerent component than u. Suppose that vj is in a diﬀerent component than u. Suppose we recolor every edge of color c with color cu and every edge of color cu with color c in the component containing u; call this path recoloring. Afterward, u now lacks c (and this does not aﬀect the colors incident to vj at all), and so we may color the uncolored edge (u, vj ) with c. See Figure 2.12 for an example. Finally, suppose that u and vj are endpoints of the same path, and so vi must be in a diﬀerent component. In this case, we can apply the previous shifting recoloring technique to ﬁrst uncolor the edge (u, vi ). We then apply the path recoloring technique on the uvj path to make u lack c; this does not aﬀect any of the colors incident on vi , and it allows us to color edge (u, vi ) with c. Clearly we color a previously uncolored edge in each iteration of the algorithm, and each iteration can be implemented in polynomial time.
Exercises 2.1 The ksuppliers problem is similar to the kcenter problem given in Section 2.2. The input to the problem is a positive integer k, and a set of vertices V , along with distances dij between any two vertices i, j that obey the same properties as in the kcenter problem. However, now the vertices are partitioned into suppliers F ⊆ V and customers D = V −F . The goal is to ﬁnd k suppliers such that the maximum distance from a supplier to a customer is minimized. In other words, we wish to ﬁnd S ⊆ F , S ≤ k, that minimizes maxj∈D d(j, S). (a) Give a 3approximation algorithm for the ksuppliers problem. (b) Prove that there is no αapproximation algorithm for α < 3 unless P = NP. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
58
Greedy algorithms and local search
u
v0
v2
u
v3
v1
v0
v2
v3
v1
Figure 2.12: The example of Figure 2.11 continued, now showing the components of black and dotted edges containing u, v1 , and v3 . Since u and v3 are at endpoints of the same black/dotted path, we switch the colors black and dotted on this path, then color (u, v1 ) black.
while G is not completely colored do Pick uncolored edge (u, v0 ) i ← −1 repeat // Build fan sequence i←i+1 if there is a color vi lacks and u lacks then Let ci be this color else Pick some color ci that vi lacks Let vi+1 be the edge (u, vi+1 ) of color ci until ci is a color u lacks or ci = cj for some j < i if u and vi lack color ci then Shift uncolored edge to (u, vi ) and color (u, vi ) with ci else Let j < i be such that ci = cj Shift uncolored edge to (u, vj ) Pick color cu that u lacks Let c = ci Let E ′ be edges colored c or cu if u and vj in diﬀerent connected components of (V, E ′ ) then Switch colors cu and c in component containing u of (V, E ′ ) Color (u, vj ) with color c else // u and vi in different components of (V, E ′ ) Shift uncolored edge to (u, vi ) Switch colors cu and c in component containing u of (V, E ′ ) Color (u, vi ) with color c Algorithm 2.3: A greedy algorithm to compute a (∆ + 1)edgecoloring of a graph.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.7
Edge coloring
59
2.2 Prove Lemma 2.8: show that for any input to the problem of minimizing the makespan on identical parallel machines for which the processing requirement of each job is more than onethird the optimal makespan, the longest processing time rule computes an optimal schedule. 2.3 We consider scheduling jobs on identical machines as in Section 2.3, but jobs are now subject to precedence constraints. We say i ≺ j if in any feasible schedule, job i must be completely processed before job j begins processing. A natural variant on the list scheduling algorithm is one in which whenever a machine becomes idle, then any remaining job that is available is assigned to start processing on that machine. A job j is available if all jobs i such that i ≺ j have already been completely processed. Show that this list scheduling algorithm is a 2approximation algorithm for the problem with precedence constraints. 2.4 In this problem, we consider a variant of the problem of scheduling on parallel machines so as to minimize the length of the schedule. Now each machine i has an associated speed si , and it takes pj /si units of time to process job j on machine i. Assume that machines are numbered from 1 to m and ordered such that s1 ≥ s2 ≥ · · · ≥ sm . We call these related machines. (a) A ρrelaxed decision procedure for a scheduling problem is an algorithm such that given an instance of the scheduling problem and a deadline D either produces a schedule of length at most ρ · D or correctly states that no schedule of length D is possible for the instance. Show that given a polynomialtime ρrelaxed decision procedure for the problem of scheduling related machines, one can produce a ρapproximation algorithm for the problem. (b) Consider the following variant of the list scheduling algorithm, now for related machines. Given a deadline D, we label every job j with the slowest machine i such that the job could complete on that machine in time D; that is, pj /si ≤ D. If there is no such machine for a job j, it is clear that no schedule of length D is possible. If machine i becomes idle at a time D or later, it stops processing. If machine i becomes idle at a time before D, it takes the next job of label i that has not been processed, and starts processing it. If no job of label i is available, it looks for jobs of label i + 1; if no jobs of label i + 1 are available, it looks for jobs of label i + 2, and so on. If no such jobs are available, it stops processing. If not all jobs are processed by this procedure, then the algorithm states that no schedule of length D is possible. Prove that this algorithm is a polynomialtime 2relaxed decision procedure. 2.5 In the minimumcost Steiner tree problem, we are given as input a complete, undirected graph G = (V, E) with nonnegative costs cij ≥ 0 for all edges (i, j) ∈ E. The set of vertices is partitioned into terminals R and nonterminals (or Steiner vertices) V − R. The goal is to ﬁnd a minimumcost tree containing all terminals. (a) Suppose initially that the edge costs obey the triangle inequality; that is, cij ≤ cik + ckj for all i, j, k ∈ V . Let G[R] be the graph induced on the set of terminals; that is, G[R] contains the vertices in R and all edges from G that have both endpoints in R. Consider computing a minimum spanning tree in G[R]. Show that this gives a 2approximation algorithm for the minimumcost Steiner tree problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
60
Greedy algorithms and local search
(b) Now we suppose that edge costs do not obey the triangle inequality, and that the input graph G is connected but not necessarily complete. Let c′ij be the cost of the shortest path from i to j in G using input edge costs c. Consider running the algorithm above in the complete graph G′ on V with edge costs c′ to obtain a tree T ′ . To compute a tree T in the original graph G, for each edge (i, j) ∈ T ′ , we add to T all edges in a shortest path from i to j in G using input edge costs c. Show that this is still a 2approximation algorithm for the minimumcost Steiner tree problem on the original (incomplete) input graph G. G′ is sometimes called the metric completion of G. 2.6 Prove that there can be no αapproximation algorithm for the minimumdegree spanning tree problem for α < 3/2 unless P = NP. 2.7 Suppose that an undirected graph G has a Hamiltonian path. Give a polynomialtime algorithm to ﬁnd a path of length at least Ω(log n/(log log n)). 2.8 Consider the local search algorithm of Section 2.6 for ﬁnding a minimumdegree spanning tree, and suppose we apply a local move to a node whenever it is possible to do so; that is, we don’t restrict local moves to nodes with degrees between ∆(T ) − ℓ and ∆(T ). What kind of performance guarantee can you obtain for a locally optimal tree in this case? 2.9 As given in Exercise 2.5, in the Steiner tree problem we are given an undirected graph G = (V, E) and a set of terminals R ⊆ V . A Steiner tree is a tree in G in which all the terminals are connected; a nonterminal need not be spanned. Show that the local search algorithm of Section 2.6 can be adapted to ﬁnd a Steiner tree whose maximum degree is at most 2 OPT +⌈log2 n⌉, where OPT is the maximum degree of a minimumdegree Steiner tree. 2.10 Let E be a set of items, and for S ⊆ E, let f (S) give the value of the subset S. Suppose we wish to ﬁnd a maximum value subset of E of at most k items. Furthermore, suppose that f (∅) = 0, and that f is monotone and submodular. We say that f is monotone if for any S and T with S ⊆ T ⊆ E, then f (S) ≤ f (T ). We say that f is submodular if for any S, T ⊆ E, then f (S) + f (T ) ≥ f (S ∪ T ) + f (S ∩ T ). Show that the greedy (1 − 1e )approximation algorithm of Section 2.5 extends to this problem. 2.11 In the maximum coverage problem, we have a set of elements E, and m subsets of elements S1 , . . . , Sm ⊆ E, each with a nonnegative weight wj ≥ 0. The goal is to choose k elements such that we maximize the weight of the subsets that are covered. We say that a subset is covered if we have chosen some element from it. Thus we want to ﬁnd S ⊆ E such that S = k and that we maximize the total weight of the subsets j such that S ∩ Sj ̸= ∅. (a) Give a (1 − 1e )approximation algorithm for this problem. (b) Show that if an approximation algorithm with performance guarantee better than 1 − 1e + ϵ exists for the maximum coverage problem for some constant ϵ > 0, then every NPcomplete problem has an O(nO(log log n) ) time algorithm. (Hint: Recall Theorem 1.13.) Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.7
Edge coloring
61
2.12 A matroid (E, I) is a set E of ground elements together with a collection I of subsets of E; that is, if S ∈ I, then S ⊆ E. A set S ∈ I is said to be independent. The independent sets of a matroid obey the following two axioms: • If S is independent, then any S ′ ⊆ S is also independent. • If S and T are independent, and S < T , then there is some e ∈ T − S such that S ∪ {e} is also independent. An independent set S is a base of the matroid if no set strictly containing it is also independent. (a) Given an undirected graph G = (V, E), show that the forests of G form a matroid; that is, show that if E is the ground set, and I the set of forests of G, then the matroid axioms are obeyed. (b) Show that for any matroid, every base of the matroid has the same number of ground elements. (c) For any given matroid, suppose that for each e ∈ E, we have a nonnegative weight we ≥ 0. Give a greedy algorithm for the problem of ﬁnding a maximumweight base of a matroid. 2.13 Let (E, I) be a matroid as deﬁned in Exercise 2.12, and let f be a monotone, submodular function as deﬁned in Exercise 2.10 such that f (∅) = 0. Consider the following local search algorithm for ﬁnding a maximumvalue base of the matroid: First, start with an arbitrary base S. Then consider all pairs e ∈ S and e′ ∈ / S. If S ∪ {e} − {e′ } is a base, and f (S ∪ {e′ } − {e}) > f (S), then set S ← S ∪ {e′ } − {e}. Repeat until a locally optimal solution is reached. The goal of this problem is to show that a locally optimal solution has value at least half the optimal value. (a) We begin with a simple case: suppose that the matroid is a uniform matroid; that is, S ⊆ E is independent if S ≤ k for some ﬁxed k. Prove that for a locally optimal solution S, f (S) ≥ 21 OPT. (b) To prove the general case, it is useful to know that for any two bases of a matroid, X and Y , there exists a bijection g : X → Y such that for any e ∈ X, S − {e} ∪ {g(e)} is independent. Use this to prove that for any locally optimal solution S, f (S) ≥ 1 2 OPT. (c) For any ϵ > 0, give a variant of this algorithm that is a ( 21 − ϵ)approximation algorithm. 2.14 In the edgedisjoint paths problem in directed graphs, we are given as input a directed graph G = (V, A) and k sourcesink pairs si , ti ∈ V . The goal of the problem is to ﬁnd edgedisjoint paths so that as many sourcesink pairs as possible have a path from si to ti . More formally, let S ⊆ {1, . . . , k}. We want to ﬁnd S and paths Pi for all i ∈ S such that S is as large as possible and for any i, j ∈ S, i ̸= j, Pi and Pj are edgedisjoint (Pi ∩ Pj = ∅). √ Consider the following greedy algorithm for the problem. Let ℓ be the maximum of m and the diameter of the graph (where m = A is the number of input arcs). For each i from 1 to k, we check to see if there exists an si ti path of length at most ℓ in the graph. If there is such a path Pi , we add i to S and remove the arcs of Pi from the graph. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
62
Greedy algorithms and local search
Show that this greedy algorithm is an Ω(1/ℓ)approximation algorithm for the edgedisjoint paths problem in directed graphs. 2.15 Prove that there is no αapproximation algorithm for the edge coloring problem for α < 4/3 unless P = NP. 2.16 Let G = (V, E) be a bipartite graph; that is, V can be partitioned into two sets A and B, such that each edge in E has one endpoint in A and the other in B. Let ∆ be the maximum degree of a node in G. Give a polynomialtime algorithm for ﬁnding a ∆edgecoloring of G.
Chapter Notes As discussed in the introduction to this chapter, greedy algorithms and local search algorithms are very popular choices for heuristics for discrete optimization problems. Thus it is not surprising that they are among the earliest algorithms analyzed for a performance guarantee. The greedy edge coloring algorithm in Section 2.7 is from a 1964 paper due to Vizing [284]. To the best of our knowledge, this is the earliest polynomialtime algorithm known for a combinatorial optimization problem that proves that its performance is close to optimal, with an additive performance guarantee. In 1966, Graham [142] gave the list scheduling algorithm for scheduling identical parallel machines found in Section 2.3. To our knowledge, this is the ﬁrst appearance of a polynomialtime algorithm with a relative performance guarantee. The longest processing time algorithm and its analysis is from a 1969 paper of Graham [143]. Other early examples of the analysis of greedy approximation algorithms include a 1977 paper of Cornuejols, Fisher, and Nemhauser [83], who introduce the ﬂoat maximization problem of Section 2.5, as well as the algorithm presented there. The analysis of the algorithm presented follows that given in Nemhauser and Wolsey [232]. The earliest due date rule given in Section 2.1 is from a 1955 paper of Jackson [174], and is sometimes called Jackson’s rule. The analysis of the algorithm in the case of negative due dates was given by Kise, Ibaraki, and Mine [195] in 1979. The nearest addition algorithm for the metric traveling salesman problem given in Section 2.4 and the analysis of the algorithm are from a 1977 paper of Rosenkrantz, Stearns, and Lewis [255]. The doubletree algorithm from that section is folklore, while Christoﬁdes’ algorithm is due, naturally enough, to Christoﬁdes [73]. The hardness result of Theorem 2.9 is due to Sahni and Gonzalez [257] while the result of Theorem 2.14 is due to Papadimitriou and Vempala [239]. There is an enormous literature on the traveling salesman problem. For booklength treatments of the problem, see the book edited by Lawler, Lenstra, Rinnooy Kan, and Shmoys [211] and the book of Applegate, Bixby, Chv´atal, and Cook [9]. Of course, greedy algorithms for polynomialtime solvable discrete optimization problems have also been studied for many years. The greedy algorithm for ﬁnding a maximumweight base of a matroid in Exercise 2.12 was given by Rado [246] in 1957, Gale [121] in 1968, and Edmonds [96] in 1971; matroids were ﬁrst deﬁned by Whitney [285]. Analysis of the performance guarantees of local search algorithms has been relatively rare, at least until some work on facility location problems from the late 1990s and early 2000s that will be discussed in Chapter 9. The local search algorithm for scheduling parallel machines given in Section 2.3 is a simpliﬁcation of a local search algorithm given in a 1979 paper of Finn and Horowitz [113]; Finn and Horowitz show that their algorithm has performance guarantee of at most 2. The local search algorithm for ﬁnding a maximumvalue base of Exercise 2.13 was given by Fisher, Nemhauser, and Wolsey [114] in 1978. The local search algorithm for ﬁnding Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
2.7
Edge coloring
63
a minimumdegree spanning tree in Section 2.6 is from 1992 and can be found in F¨ urer and Raghavachari [118]. Now to discuss the other results presented in the chapter: The algorithm and analysis of the kcenter problem in Section 2.2 is due to Gonzalez [141]. An alternative 2approximation algorithm for the problem is due to Hochbaum and Shmoys [163]. Theorem 2.4, which states that getting a performance guarantee better than 2 is NPhard, is due to Hsu and Nemhauser [172]. Theorem 2.22, which states that deciding if a graph is 3edgecolorable or not is NPcomplete, is due to Holyer [170]. The ksupplier problem of Exercise 2.1, as well as a 3approximation algorithm for it, were introduced in Hochbaum and Shmoys [164]. The list scheduling variant for problems with precedence constraints in Exercise 2.3 is due to Graham [142]. The idea of a ρrelaxed decision procedure in Exercise 2.4 is due to Hochbaum and Shmoys [165]; the 2relaxed decision procedure for related machines in that exercise is due to Shmoys, Wein, and Williamson [265]. Exercise 2.7 was suggested to us by Nick Harvey. Exercise 2.9 is due to F¨ urer and Raghavachari [118]. Exercises 2.10 and 2.11 are due to Nemhauser, Wolsey, and Fisher [233]. The hardness result of Exercise 2.11 is due to Feige [107]; Feige also shows that the same result can be obtained under the assumption that P ̸= NP. Kleinberg [199] gives the greedy algorithm for the edgedisjoint paths problem in directed graphs given in Exercise 2.14. K˝onig [201] shows that it is possible to ∆edgecolor a bipartite graph as in Exercise 2.16.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
64
Greedy algorithms and local search
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 3
Rounding data and dynamic programming
Dynamic programming is a standard technique in algorithm design in which an optimal solution for a problem is built up from optimal solutions for a number of subproblems, normally stored in a table or multidimensional array. Approximation algorithms can be designed using dynamic programming in a variety of ways, many of which involve rounding the input data in some way. For instance, sometimes weakly NPhard problems have dynamic programming algorithms that run in time polynomial in the input size if the input is represented in unary rather than in binary (so, for example, the number 7 would be encoded as 1111111). If so, we say that the algorithm is pseudopolynomial. Then by rounding the input values so that the number of distinct values is polynomial in the input size and an error parameter ϵ > 0, this pseudopolynomial algorithm can be made to run in time polynomial in the size of the original instance. We can often show that the rounding does not sacriﬁce too much in the quality of the solution produced. We will use this technique in discussing the knapsack problem in Section 3.1. For other problems, such as scheduling problems, we can often make distinctions between “large” and “small” parts of the input instance; for instance, in scheduling problems, we distinguish between jobs that have large and small processing times. We can then show that by rounding the sizes of the large inputs so that, again, the number of distinct, large input values is polynomial in the input size and an error parameter, we can use dynamic programming to ﬁnd an optimal solution on just the large inputs. Then this solution must be augmented to a solution for the whole input by dealing with the small inputs in some way. Using these ideas, we will devise polynomialtime approximation schemes for the problem of scheduling parallel machines introduced in the last chapter, and for a new problem of packing bins.
3.1
The knapsack problem
A traveler with a knapsack comes across a treasure hoard. Unfortunately, his knapsack can hold only so much. What items should he place in his knapsack in order to maximize the value of the items he takes away? This unrealistic scenario gives the name to the knapsack problem. In the knapsack problem, we are given a set of n items I = {1, . . . , n}, where each item i has a value vi and a size si . All sizes and values are positive integers. The knapsack has capacity B, where B 65
66
Rounding data and dynamic programming
A(1) ← {(0, 0), (s1 , w1 )} for j ← 2 to n do A(j) ← A(j − 1) for each (t, w) ∈ A(j − 1) do if t + sj ≤ B then Add (t + sj , w + vj ) to A(j) Remove dominated pairs from A(j) return max(t,w)∈A(n) w Algorithm 3.1: A dynamic programming algorithm for the knapsack problem. is ∑also a positive integer. The goal is to ﬁnd a subset of items S ⊆ I that maximizes the value to the constraint that the total size of these items is i∈S vi of items in the knapsack subject ∑ no more than the capacity; that is, i∈S si ≤ B. We assume that we consider only items that could actually ﬁt in the knapsack (by themselves), so that si ≤ B for each i ∈ I. Although the application stated above is unlikely to be useful in real life, the knapsack problem is well studied because it is a simpliﬁed model of a problem that arises in many realistic scenarios. We now argue that we can use dynamic programming to ﬁnd the optimal solution to the knapsack problem. We maintain an array entry A(j) for j = 1, . . . , n. Each entry A(j) is a list of pairs (t, w). A pair (t, w) in the list of entry A(j) indicates that there is a set S from the ﬁrst j items that uses space ∑ exactly t ≤ B and has ∑ value exactly w; that is, there exists a set S ⊆ {1, . . . , j}, such that i∈S si = t ≤ B and i∈S vi = w. Each list does not contain all possible such pairs, but instead keeps track of only the most eﬃcient ones. To do this, we introduce the notion of one pair dominating another one; a pair (t, w) dominates another pair (t′ , w′ ) if t ≤ t′ and w ≥ w′ ; that is, the solution indicated by the pair (t, w) uses no more space than (t′ , w′ ), but has at least as much value. Note that domination is a transitive property; that is, if (t, w) dominates (t′ , w′ ) and (t′ , w′ ) dominates (t′′ , w′′ ), then (t, w) also dominates (t′′ , w′′ ). We will ensure that in any list, no pair dominates another one; this means that we can assume each list A(j) is of the form (t1 , w1 ), . . . , (tk , wk ) with t1 < t2 < · · · < tk and w1 < w2 < · · · < wk . Since the sizes of the items are integers, ∑n this implies that there are at most B + 1 pairs in each list. Furthermore, if we let V = i=1 vi be the maximum possible value for the knapsack, then there can be at most ∑ V + 1 pairs in the list. Finally, we ensure that for each feasible set ∑ S ⊆ {1, ∑. . . , j} (with i∈S si ≤ B), the list A(j) contains some pair (t, w) that dominates ( i∈S si , i∈S vi ). In Algorithm 3.1, we give the dynamic program that constructs the lists A(j) and solves the knapsack problem. We start out with A(1) = {(0, 0), (s1 , w1 )}. For each j = 2, . . . , n, we do the following. We ﬁrst set A(j) ← A(j − 1), and for each (t, w) ∈ A(j − 1), we also add the pair (t + sj , w + vj ) to the list A(j) if t + sj ≤ B. We ﬁnally remove from A(j) all dominated pairs by sorting the list with respect to their space component, retaining the best value for each space total possible, and removing any larger space total that does not have a corresponding larger value. One way to view this process is to generate two lists, A(j − 1) and the one augmented by (sj , wj ), and then perform a type of merging of these two lists. We return the pair (t, w) from A(n) of maximum value as our solution. Next we argue that this algorithm is correct. Theorem 3.1: Algorithm 3.1 correctly computes the optimal value of the knapsack problem. Proof. By induction on j we prove that A(j) contains all nondominated pairs corresponding to feasible sets S ⊆ {1, . . . , j}. Certainly this is true in the base case by setting A(1) to Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
3.1
The knapsack problem
67
∑ {(0, 0), (s1 , w1∑ )}. Now suppose it is true for A(j − 1). Let S ⊆ {1, . . . , j}, and let t = i∈S si ≤ B and w = i∈S vi . We claim that there is some pair (t′ , w′ ) ∈ A(j) such that t′ ≤ t and w′ ≥ w. First, suppose that j ∈ / S. Then the claim follows by the induction hypothesis and by the fact that we initially set A(j) to A(j − 1) and removed dominated pairs. Now suppose ˆ ˆ ∈ A(j − 1) that j ∈ S. Then∑ for S ′ =∑ S − {j}, by the induction ∑ hypothesis, there ∑ is some (t, w) ˆ dominates ( i∈S ′ si , i∈S ′ vi ), so that t ≤ i∈S ′ si and w ˆ ≥ i∈S ′ vi . Then the algorithm will add the pair (tˆ + sj , w ˆ + vj ) to A(j), where tˆ + sj ≤ t ≤ B and w ˆ + vj ≥ w. Thus, there will be some pair (t′ , w′ ) ∈ A(j) that dominates (t, w). Algorithm 3.1 takes O(n min(B, V )) time. This is not a polynomialtime algorithm, since we assume that all input numbers are encoded in binary; thus, the size of the input number B is essentially log2 B, and so the running time O(nB) is exponential in the size of the input number B, not polynomial. If we were to assume that the input is given in unary, then O(nB) would be a polynomial in the size of the input. It is sometimes useful to make this distinction between problems. Deﬁnition 3.2: An algorithm for a problem Π is said to be pseudopolynomial if its running time is polynomial in the size of the input when the numeric part of the input is encoded in unary. If the maximum possible value V were some polynomial in n, then the running time would indeed be a polynomial in the input size. We now show how to get a polynomialtime approximation scheme for the knapsack problem by rounding the values of the items so that V is indeed a polynomial in n. The rounding induces some loss of precision in the value of a solution, but we will show that this does not aﬀect the ﬁnal value by too much. Recall the deﬁnition of an approximation scheme from Chapter 1. Deﬁnition 3.3: A polynomialtime approximation scheme (PTAS) is a family of algorithms {Aϵ }, where there is an algorithm for each ϵ > 0, such that Aϵ is a (1 + ϵ)approximation algorithm (for minimization problems) or a (1 − ϵ)approximation algorithm (for maximization problems). Note that the running time of the algorithm Aϵ is allowed to depend arbitrarily on 1/ϵ: this dependence could be exponential in 1/ϵ, or worse. We often focus attention on algorithms for which we can give a good bound of the dependence of the running time of Aϵ on 1/ϵ. This motivates the following deﬁnition. Deﬁnition 3.4: A fully polynomialtime approximation scheme (FPAS, FPTAS) is an approximation scheme such that the running time of Aϵ is bounded by a polynomial in 1/ϵ. We can now give a fully polynomialtime approximation scheme for the knapsack problem. Suppose that we measure value in (integer) multiples of µ (where we shall set µ below), and convert each value vi by rounding down to the nearest integer multiple of µ; more precisely, we set vi′ to be ⌊vi /µ⌋ for each item i. We can then run the dynamic programming algorithm of Figure 3.1 on the items with sizes si and values vi′ , and output the optimal solution for the rounded data as a nearoptimal solution for the true data. The main idea here is that we wish to show that the accuracy we lose in rounding is not so great, and yet the rounding enables us to have the algorithm run in polynomial time. Let us ﬁrst do a rough estimate; if we used values v˜i = vi′ µ instead of vi , then each value is inaccurate by at most µ, and so each feasible solution has its value changed by at most nµ. We want the error introduced to be at most ϵ times a lower bound on the optimal value (and so be sure that the true relative error is at most ϵ). Let M be the maximum value of an item; that is, M = maxi∈I vi . Then M is a lower bound Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
68
Rounding data and dynamic programming
M ← maxi∈I vi µ ← ϵM/n vi′ ← ⌊vi /µ⌋ for all i ∈ I Run Algorithm 3.1 for knapsack instance with values vi′ Algorithm 3.2: An approximation scheme for the knapsack problem. on OPT, since one possible solution is to pack the most valuable item in the knapsack by itself. Thus, it makes sense to set µ so that nµ = ϵM ∑ or, in other∑words, to set µ = ϵM/n. vi Note that with the modiﬁed values, V ′ = ni=1 vi′ = ni=1 ⌊ ϵM/n ⌋ = O(n2 /ϵ). Thus, the running time of the algorithm is O(n min(B, V ′ )) = O(n3 /ϵ) and is bounded by a polynomial in 1/ϵ. We can now prove that the algorithm returns a solution whose value is at least (1 − ϵ) times the value of an optimal solution. Theorem 3.5: Algorithm 3.2 is a fully polynomialtime approximation scheme for the knapsack problem. Proof. We need to show that the algorithm returns a solution whose value is at least (1 − ϵ) times the value of an optimal solution. Let S be the set of items returned by the algorithm. Let O be an optimal set of items. Certainly M ≤ OPT, since one possible solution is to put the most valuable item in a knapsack by itself. Furthermore, by the deﬁnition of vi′ , µvi′ ≤ vi ≤ µ(vi′ + 1), so that µvi′ ≥ vi − µ. Applying the deﬁnitions of the rounded data, along with the fact that S is an optimal solution for the values vi′ , we can derive the following chain of inequalities: ∑ ∑ vi ≥ µ vi′ i∈S
i∈S
≥ µ
∑
vi′
i∈O
≥
∑
vi − Oµ
i∈O
≥
∑
vi − nµ
i∈O
=
∑
vi − ϵM
i∈O
≥ OPT −ϵ OPT = (1 − ϵ) OPT .
3.2
Scheduling jobs on identical parallel machines
We return to the problem of scheduling a collection of n jobs on m identical parallel machines; in Section 2.3 we presented a result that by ﬁrst sorting the jobs in order of nonincreasing processing requirement, and then using a list scheduling rule, we ﬁnd a schedule of length guaranteed to be at most 4/3 times the optimum. In this section, we will show that this result contains the seeds of a polynomialtime approximation scheme: for any given value of ρ > 1, we give an algorithm that runs in polynomial time and ﬁnds a solution of objective function value at most ρ times the optimal value. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
3.2
Scheduling jobs on identical parallel machines
69
As in Section 2.3, we let the processing requirement of job j be pj , j = 1, . . . , n, and let Cmax denote the length (or makespan) of a given schedule with job completion times Cj , j = 1, . . . , n; ∗ . We shall assume that each processing requirement is a the optimal value is denoted Cmax positive integer. The key idea of the analysis of the list scheduling rule was that its error can be upper bounded by the processing requirement of the last job to complete. The 4/3approximation algorithm was based on this fact, combined with the observation that when ∗ /3, this natural greedytype algorithm each job’s processing requirement is more than Cmax actually ﬁnds the optimal solution. We present an approximation scheme for this problem based on a similar principle: we focus on a speciﬁed subset of the longest jobs, and compute the optimal schedule for that subset; then we extend that partial schedule by using list scheduling on the remaining jobs. We will show that there is a tradeoﬀ between the number of long jobs and the quality of the solution found. More precisely, let k be a ﬁxed positive integer; we will derive a family of algorithms, and focus on the algorithm Ak among them. Suppose that we partition the job set ∑ into two parts: n 1 the long jobs and the short jobs, where a job ℓ is considered short if pℓ ≤ km j=1 pj . Note that this implies that there are at most km long jobs. Enumerate all possible schedules for the long jobs, and choose one with the minimum makespan. Extend this schedule by using list scheduling for the short jobs; that is, given an arbitrary order of the short jobs, schedule these jobs in order, always assigning the next job to the machine currently least loaded. Consider the running time of algorithm Ak . To specify a schedule for the long jobs, we simply indicate to which of the m machines each long job is assigned; thus, there are at most mkm distinct assignments (since the order of processing on each machine is unimportant). If we focus on the special case of this problem in which the number of machines is a constant (say, 100, 1,000, or even 1,000,000), then this number is also a constant, not depending on the size of the input. Thus, we can check each schedule, and determine the optimal length schedule in polynomial time in this special case. As in the analysis of the local search algorithm in Section 2.3, we focus on the last job ℓ to ﬁnish. Recall that we derived the equality that ∑ pj /m; (3.1) Cmax ≤ pℓ + j̸=ℓ
the validity of this inequality relied only on the fact that each machine is busy up until the time that job ℓ starts. To analyze the algorithm that starts by ﬁnding the optimal schedule for the long jobs, we distinguish now between two cases. If the last job to ﬁnish (in the entire schedule), job ℓ, is a short job, then this job was scheduled by the list scheduling rule, and it ∑ follows that inequality (3.1) holds. Since job ℓ is short, and hence pℓ ≤ nj=1 pj /(mk), it also follows that ) n ( ) ( n ∑ ∑ 1 1 ∑ ∗ pj /m ≤ 1 + Cmax . Cmax ≤ pj /(mk) + pj /m ≤ 1 + k k j=1
j̸=ℓ
j=1
If job ℓ is a long job, then the schedule delivered by the algorithm is optimal, since its makespan is equal to the length of the optimal schedule for just the long jobs, which is clearly ∗ for the entire input. The algorithm Ak can easily be implemented to run in no more than Cmax polynomial time (treating m as a constant), and hence we have obtained the following theorem. Theorem 3.6: The family of algorithms {Ak } is a polynomialtime approximation scheme for the problem of minimizing the makespan on any constant number of identical parallel machines. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
70
Rounding data and dynamic programming
Of course, it is a signiﬁcant limitation of this theorem that the number of machines is restricted to be a constant. In fact, it is not too hard to extend these techniques to obtain a polynomialtime approximation scheme even if the number of machines is allowed to be an input parameter (and hence the algorithm must also have running time polynomially bounded in the number of machines m). The key idea is that we didn’t really need the schedule for the long jobs to be optimal. We used the optimality of the schedule for the long jobs only when the last job to ﬁnish was a long job. If we had found a schedule for the long jobs that had makespan at most 1 + k1 times the optimal value, then that clearly would have been suﬃcient. We will show how to obtain this nearoptimal schedule for long jobs by rounding input sizes and dynamic programming, as we saw in the previous section on the knapsack problem. It will be convenient to ﬁrst set a target length T for the schedule. As before, we also ﬁx a positive integer k; we will design a family of algorithms {Bk } where Bk either proves that no schedule of length T exists, or else ﬁnds a schedule of length (1 + k1 )T . Later we will show how such a family of algorithms also ∑ implies the existence of a polynomialtime approximation n 1 scheme. We can assume that T ≥ m j=1 pj , since otherwise no feasible schedule exists. The algorithm Bk is quite simple. We again partition the jobs into long and short jobs, but in this case, we require that pj > T /k for job j to be considered long. We round down the processing requirement of each long job to its nearest multiple of T /k 2 . We will determine in polynomial time whether or not there is a feasible schedule for these rounded long jobs that completes within time T . If there is such a schedule, we then interpret it as a schedule for the long jobs with their original processing requirements. If not, we conclude that no feasible schedule of length T exists for the original input. Finally, we extend this schedule to include the short jobs by using the list scheduling algorithm for the short jobs. We need to prove that the algorithm Bk always produces a schedule of length at most (1 + k1 )T whenever there exists a schedule of length at most T . When the original input has a schedule of length T , then so does the reduced input consisting only of the rounded long jobs (which is why we rounded down the processing requirements); in this case, the algorithm does compute a schedule for the original input. Suppose that a schedule is found. It starts with a schedule of length at most T for the rounded long jobs. Let S be the set of jobs assigned by this schedule to one machine. Since each job in S is long, and hence has rounded size at least T /k, it follows that S ≤ k. Furthermore, for each job j ∈ S, the diﬀerence between its true processing requirement and its rounded one is at most T /k 2 . Hence, ) ( ∑ 1 2 T. pj ≤ T + k(T /k ) = 1 + k j∈S
Now consider the eﬀect of assigning the short ∑ jobs: each job ℓ, in turn, is assigned ∑ to a machine for which the current load is smallest. Since nj=1 pj /m ≤ T , we also know that j̸=ℓ pj /m < T . Since the average load assigned to a machine is less than T , there must exist a machine that is currently assigned jobs of total processing requirement less than T . So, when we choose the machine that currently has the lightest load, and then add job ℓ, this machine’s new load is at most ( ) ∑ 1 pℓ + pj /m < T /k + T = 1 + T. k j̸=ℓ
Hence, the schedule produced by list scheduling will also be of length at most (1 + k1 )T . To complete the description of the algorithm Bk , we must still show that we can use dynamic programming to decide if there is a schedule of length T for the rounded long jobs. Clearly if Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
3.2
Scheduling jobs on identical parallel machines
71
there is a rounded long job of size greater than T , then there is no such schedule. Otherwise, we can describe an input by a k 2 dimensional vector, where the ith component speciﬁes the number of long jobs of rounded size equal to iT /k 2 , for each i = 1, . . . , k 2 . (In fact, we know that for i < k, there are no such jobs, since that would imply that their original processing 2 requirement was less than T /k, and hence not long.) So there are at most nk distinct inputs — a polynomial number! How many distinct ways are there to feasibly assign long jobs to one machine? Each rounded long job still has processing time at least T /k. Hence, at most k jobs are assigned to one machine. Again, an assignment to one machine can be described by a k 2 dimensional vector, where again the ith component speciﬁes the number of long jobs of rounded size equal to iT /k 2 that are assigned to that machine. Consider the vector (s1 , s2 , . . . , sk2 ); we shall call it a machine conﬁguration if k2 ∑ si · iT /k 2 ≤ T. i=1 2
Let C denote the set of all machine conﬁgurations. Note that there are at most (k +1)k distinct conﬁgurations, since each machine must process a number of rounded long jobs that is in the set {0, 1, . . . , k}. Since k is ﬁxed, this means that there are a constant number of conﬁgurations. Let OPT(n1 , . . . , nk2 ) denote the minimum number of (identical) machines suﬃcient to schedule this arbitrary input. This value is governed by the following recurrence relation, based on the idea that a schedule consists of assigning some jobs to one machine, and then using as few machines as possible for the rest: OPT(n1 , . . . , nk2 ) = 1 +
min (s1 ,...,sk2 )∈C
OPT(n1 − s1 , . . . , nk2 − sk2 ).
This can be viewed as a table with a polynomial number of entries (one for each possible input type), and to compute each entry, we need to ﬁnd the minimum over a constant number of previously computed values. The desired schedule exists exactly when the corresponding optimal value is at most m. Finally, we need to show that we can convert the family of algorithms {Bk } into a polynomialtime approximation scheme. Fix the relative error ϵ > 0. We use a bisection search procedure to determine a suitable choice of the target value T (which is required as part of the input for each algorithm Bk ). We know that the optimal makespan for our scheduling input is within the interval [L0 , U0 ], where n ∑ , max pj L0 = max p /m j j=1,...,n j=1 and
n ∑
U0 = pj /m max pj . + j=1,...,n j=1 (We strengthen the lower bound with the ⌈·⌉ by relying on the fact that the processing requirements are integral.) Throughout the bisection search, we maintain such an interval [L, U ], with ∗ , and (2) that we can compute a schedule with the algorithmic invariants (1) that L ≤ Cmax makespan at most (1 + ϵ)U . This is clearly true initially: by the arguments of Section 2.3, L0 is a lower bound on the length of an optimal schedule, and using a list scheduling algorithm we can compute a schedule of length at most U0 . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
72
Rounding data and dynamic programming
In each iteration where the current interval is [L, U ], we set T = ⌊(L + U )/2⌋, and run the algorithm Bk , where k = ⌈1/ϵ⌉. If the algorithm Bk produces a schedule, then update U ← T ; otherwise, update L ← T +1. The bisection continues until L = U , at which point the algorithm outputs the schedule associated with U (of length at most (1 + ϵ)U ). It is easy to see that the claimed invariant is maintained: (1) when we update the lower limit L, the algorithm Bk has just shown that no feasible schedule of length T exists, and by the integrality of the processing times, we know then that T + 1 is a valid lower bound; (2) when we update the upper limit U , the algorithm Bk has just produced the schedule of length at most (1 + ϵ)T , which is exactly what is required to justify this update. The diﬀerence between the upper and lower limits is initially at most maxj=1,...,n pj , and is halved in each iteration. Hence, after a polynomial number of iterations, the diﬀerence becomes less than 1 and, by integrality, is therefore 0. Since both invariants must still hold for this trivial ∗ interval [L, L], then we know that Cmax ≥ L, and that the ﬁnal schedule output by the algorithm ∗ is of length at most (1 + ϵ)L ≤ (1 + ϵ)Cmax . The algorithm is a (1 + ϵ)approximation algorithm. Theorem 3.7: There is a polynomialtime approximation scheme for the problem of minimizing the makespan on an input number of identical parallel machines. 2
Note that since we consider (k + 1)k conﬁgurations and k = ⌈1/ϵ⌉, the running time in the worst case is exponential in O(1/ϵ2 ). Thus, in this case, we did not obtain a fully polynomialtime approximation scheme (in contrast to the knapsack problem). This is for a fundamental reason. This scheduling problem is strongly NPcomplete; that is, even if we require that the processing times be restricted to values at most q(n), a polynomial function of the number of jobs, this special case is still NPcomplete. We claim that if a fully polynomialtime approximation scheme exists for this problem, it could be used to solve this special case in polynomial time, which would imply that P = NP. How can we use a fully polynomialtime approximation scheme to solve the special case with polynomially bounded processing times? Let P be the maximum processing time for an input satisfying this special structure. This implies that the optimal makespan is at most nP ≤ nq(n). If there were a fully polynomialtime approximation scheme {Ak }, suppose that we use the algorithm Ak that guarantees a solution with relative error at most 1/k where k = ⌈2nq(n)⌉. This implies that the algorithm ﬁnds a solution of makespan at most (
1 1+ k
)
1 ∗ ∗ Cmax ≤ Cmax + . 2
But since any feasible schedule is simply an assignment of jobs to machines, the makespan is clearly integer, and hence the algorithm must ﬁnd the optimal assignment. We know that q(n) is a polynomial; the requirements of a fully polynomialtime approximation scheme imply that the running time must be bounded by a polynomial in k, and hence we have computed this optimal schedule in polynomial time. Therefore, the existence of such a scheme implies that P = NP. This result is one special case of a much more general result; we know that (with very mild assumptions on the nature of the objective function) for any optimization problem, if the problem is strongly NPcomplete, then it does not have a fully polynomialtime approximation scheme. We give this as an exercise (Exercise 3.9). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
3.3
The binpacking problem
3.3
73
The binpacking problem
In the binpacking problem, we are given n pieces (or items) with speciﬁed sizes a1 , a2 , . . . , an , such that 1 > a1 ≥ a2 ≥ · · · ≥ an > 0; we wish to pack the pieces into bins, where each bin can hold any subset of pieces of total size at most 1, so as to minimize the number of bins used. The binpacking problem is related to a decision problem called the partition problem. In ∑n the partition problem, we are given n positive integers b1 , . . . , bn whose sum B = i=1 bi is even, and ∑ we wish to∑know if we can partition the set of indices {1, . . . , n} into sets S and T such that i∈S bi = i∈T bi . The partition problem is well known to be NPcomplete. Notice that we can reduce this problem to a binpacking problem by setting ai = 2bi /B and checking whether we can pack all the pieces into two bins or not. This gives the following theorem. Theorem 3.8: Unless P = NP, there cannot exist a ρapproximation algorithm for the binpacking problem for any ρ < 3/2. However, consider the FirstFitDecreasing algorithm, where the pieces are packed in order of nonincreasing size, and the next piece is always packed into the ﬁrst bin in which it ﬁts; that is, we ﬁrst open bin 1, and we start bin k + 1 only when the current piece does not ﬁt into any of the bins 1, . . . , k. If FFD(I) denotes the number of bins used by this algorithm on input I, and OPT(I) denotes the number of bins used in the optimal packing, then a celebrated classic result shows that that FFD(I) ≤ (11/9) OPT(I) + 4 for any input I. Thus, signiﬁcantly stronger results can be obtained by relaxing the notion of the performance guarantee to allow for small additive terms. In fact, it is completely consistent with our current understanding of complexity theory that there is an algorithm that always produces a packing with at most OPT(I) + 1 bins. Why is it that we have bothered to mention hardness results of the form “there does not exist a ρapproximation algorithm unless P = NP” if such a result can be so easily circumvented? The reason is that for all of the weighted problems that we have discussed, any distinction between the two types of guarantees disappears; any algorithm guaranteed to produce a solution of value at most ρ OPT +c can be converted to a ρapproximation algorithm. Each of these problems has a natural rescaling property: for any input I and any value κ, we can construct an essentially identical instance I ′ such that the objective function value of any feasible solution is rescaled by κ. For example, for the scheduling problem of Section 3.2, if one simply multiplies each processing time pj by κ, one can accomplish this rescaling, or for a combinatorial problem such as the unweighted vertex cover problem, one can consider an input with κ disjoint copies of the original input graph. Such rescaling makes it possible to blunt the eﬀect of any small additive term c < κ in the guarantee, and make it eﬀectively 0. Observe that the binpacking problem does not have this rescaling property; there is no obvious way to “multiply” the instance in a way that does not blur the combinatorial structure of the original input. (Think about what happens if you construct a new input that contains two copies of each piece of I!) Thus, whenever we consider designing approximation algorithms for a new combinatorial optimization problem, it is important to consider ﬁrst whether the problem does have the rescaling property, since that will indicate what sort of performance guarantee one might hope for. Although we will not prove the performance guarantee for the FirstFitDecreasing algorithm for the binpacking problem, we shall show that an exceedingly simple algorithm does perform reasonably well. Consider the FirstFit algorithm, which works exactly as FirstFitDecreasing, except that we don’t ﬁrst sort the pieces in nonincreasing size order. We can analyze its Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
74
Rounding data and dynamic programming
performance in the following way. If we pair up bins 1 and 2, then 3 and 4, and so forth, then any such pair must have the property that the total piece size in them is at least 1: in fact, the ﬁrst item placed in bin 2k is put there only because it did not ﬁt in∑bin 2k − 1. Thus, if we used ℓ bins, then the total size of the pieces in the input, SIZE(I) = ni=1 ai , is at least ⌊ℓ/2⌋. However, it is clear that OPT(I) ≥ SIZE(I), and hence, the number of bins used by FirstFit is FF(I) = ℓ ≤ 2 SIZE(I) + 1 ≤ 2 OPT(I) + 1. Of course, this analysis did not use practically any information about the algorithm; we used only that there should not be two bins whose contents can be feasibly combined into one bin. We shall present a family of polynomialtime approximation algorithms parameterized by ϵ > 0, where each algorithm has the performance guarantee of computing a packing with at most (1 + ϵ) OPT(I) + 1 bins for each input I. Throughout this discussion, we shall view ϵ as a positive constant. Note that this family of algorithms does not meet the deﬁnition of a polynomialtime approximation scheme because of the additive constant. This motivates the following deﬁnition. Deﬁnition 3.9: An asymptotic polynomialtime approximation scheme (APTAS) is a family of algorithms {Aϵ } along with a constant c where there is an algorithm Aϵ for each ϵ > 0 such that Aϵ returns a solution of value at most (1 + ϵ) OPT +c for minimization problems. One of the key ingredients of this asymptotic polynomialtime approximation scheme is the dynamic programming algorithm used in the approximation scheme for scheduling jobs on identical parallel machines, which was presented in Section 3.2. As stated earlier, that algorithm computed the minimum number of machines needed to assign jobs, so that each machine was assigned jobs of total processing requirement at most T . However, by rescaling each processing time by dividing by T , we have an input for the binpacking problem. The dynamic programming algorithm presented solves the binpacking problem in polynomial time in the special case in which there are only a constant number of distinct piece sizes, and only a constant number of pieces can ﬁt into one bin. Starting with an arbitrary input to the binpacking problem, we ﬁrst construct a simpliﬁed input of this special form, which we then solve by dynamic programming. The simpliﬁed input will also have the property that we can transform the resulting packing into a packing for the original input, without introducing too many additional bins. As in the scheduling result of Section 3.2, the ﬁrst key observation is that one may ignore small pieces of size less than any given threshold, and can analyze its eﬀect in a relatively simple way. Lemma 3.10: Any packing of all pieces of size greater than γ into ℓ bins can be extended to a 1 packing for the entire input with at most max{ℓ, 1−γ SIZE(I) + 1} bins. Proof. Suppose that one uses the FirstFit algorithm, starting with the given packing, to compute a packing that also includes these small pieces. If FirstFit never starts a new bin in packing the small pieces, then clearly the resulting packing has ℓ bins. If it does start a new bin, then each bin in the resulting packing (with the lone exception of the last bin started) must not have been able to ﬁt one additional small piece. Let k + 1 denote the number of bins used in this latter case. In other words, each of the ﬁrst k bins must have pieces totaling at least 1 − γ, and hence SIZE(I) ≥ (1 − γ)k. Equivalently, k ≤ SIZE(I)/(1 − γ), which completes the proof of the lemma. Suppose that we were aiming to design an algorithm with a performance guarantee that is better than the one proved for FirstFit (which is truly a modest goal); in particular, we are Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
3.3
The binpacking problem
75
trying to design an algorithm with performance guarantee 1 + ϵ with ϵ < 1. If we apply Lemma 3.10 by setting γ = ϵ/2, then since 1/(1 − ϵ/2) ≤ 1 + ϵ, we see that the composite algorithm produces a packing with at most max{ℓ, (1 + ϵ) OPT(I) + 1} bins. The elimination of the small pieces from the input also enforces one of our requirements for the special case solvable by dynamic programming: if each piece has size greater than ϵ/2, then each bin must contain fewer than 2/ϵ pieces in total. We shall assume for the rest of the discussion that our input I does not have such small pieces to begin with. The last element of the algorithm is a technique to reduce the number of distinct piece sizes, which is accomplished by a linear grouping scheme. This scheme works as follows, and is based on a parameter k, which will be set later. Group the pieces of the given input I as follows: the ﬁrst group consists of the k largest pieces, the next group consists of the next k largest pieces, and so on, until all pieces have been placed in a group. The last group contains h pieces, where h ≤ k. The rounded instance I ′ is constructed by discarding the ﬁrst group, and for each other group, rounding the size of its pieces up to the size of its largest piece. An input I and its transformed version I ′ are shown in Figure 3.1. We can prove the following lemma relating the optimal number of bins needed to pack I to the optimal number of bins needed to pack I ′ . Lemma 3.11: Let I ′ be the input obtained from an input I by applying linear grouping with group size k. Then OPT(I ′ ) ≤ OPT(I) ≤ OPT(I ′ ) + k, and furthermore, any packing of I ′ can be used to generate a packing of I with at most k additional bins. Proof. For the ﬁrst inequality, observe that any packing of the input I yields a packing of the input I ′ : for I ′ , pack its k largest pieces wherever the k largest pieces of I were packed. (There is a onetoone correspondence between these two sets of k pieces, and each piece in the group from I ′ is no larger than the piece to which it corresponds in I.) The second k largest pieces of I ′ are packed wherever the second group of I were packed, and so forth. It is clear that this yields a feasible packing of I ′ , and the ﬁrst inequality follows. To obtain the second inequality, we show how to take a packing of I ′ , and use it to obtain a packing of I. This is also quite simple. To pack I, we pack each of the k largest pieces in its own bin. Now, the next largest group of k pieces in I can be packed wherever the largest group of pieces in I ′ are packed. Again, to do this, we use the fact that one can construct a onetoone mapping between these two sets of k pieces, where each piece in I is no larger than its corresponding piece in I ′ . The fact that this last inequality is proved algorithmically is important, since this means that if we do obtain a good packing for I ′ , then we can use it to obtain an “almostasgood” packing for I. It is relatively straightforward to complete the derivation of an asymptotic polynomialtime approximation scheme for the binpacking problem. If we consider the input I ′ , then the number of distinct piece sizes is at most n/k, where n is the number of pieces in I. Since there are no small pieces in I, SIZE(I) ≥ ϵn/2. If we set k = ⌊ϵ SIZE(I)⌋, then we see that n/k ≤ 2n/(ϵ SIZE(I)) ≤ 4/ϵ2 , where we crudely bound ⌊α⌋ ≥ α/2 when α ≥ 1; we can assume ϵ SIZE(I) ≥ 1 since otherwise there are at most (1/ϵ)/(ϵ/2) = 2/ϵ2 (large) pieces and we could apply the dynamic programming algorithm to solve the input optimally without applying the linear grouping scheme. Consequently, after performing the linear grouping, we are left with a binpacking input in which there are a constant number of distinct piece sizes, and only a constant number of pieces can ﬁt in each bin; hence, we can obtain an optimal packing for the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
76
Rounding data and dynamic programming
1
0.8
0.6
0.4
0.2
0 
{z
G1
}

{z
G2
}

{z
G3
}

{z
G4
}

{z
G5
}

{z
G6
1
0.8
0.6
0.4
0.2
0
Figure 3.1: An input before and after linear grouping with group size k = 4.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
}
{z}
G7
3.3
The binpacking problem
77
input I ′ by the dynamic programming algorithm of Section 3.2. This packing for I ′ can then be used to get a packing for the ungrouped input, and then be extended to include all of the small pieces with at most (1 + ϵ) OPT(I) + 1 bins, as we show next. Theorem 3.12: For any ϵ > 0, there is a polynomialtime algorithm for the binpacking problem that computes a solution with at most (1 + ϵ) OPT(I) + 1 bins; that is, there is an APTAS for the binpacking problem. Proof. As we discussed earlier, the algorithm will open max{ℓ, (1+ϵ) OPT(I)+1} bins, where ℓ is the number of bins used to pack the large pieces. By Lemma 3.11, we use at most OPT(I ′ )+k ≤ OPT(I) + k bins to pack these pieces, where k = ⌊ϵ SIZE(I)⌋, so that ℓ ≤ OPT(I) + ϵ SIZE(I) ≤ (1 + ϵ) OPT(I), which completes the proof. It is possible to improve both the running time of this general approach, and its performance guarantee; we shall return to this problem in Section 4.6.
Exercises 3.1 Consider the following greedy algorithm for the knapsack problem. We initially sort all the items in order of nonincreasing ratio of value to size so that v1 /s1 ≥ v2 /s2 ≥ · · · ≥ vn /sn . Let i∗ be the index of an item of maximum value so that vi∗ = maxi∈I vi . The greedy algorithm puts items in the knapsack in index order ∑k+1until the next item no longer ﬁts; ∑k s ≤ B but that is, it ﬁnds k such that i=1 si > B. The algorithm returns i=1 i ∗ either {1, . . . , k} or {i }, whichever has greater value. Prove that this algorithm is a 1/2approximation algorithm for the knapsack problem. 3.2 One key element in the construction of the fully polynomialtime approximation scheme for the knapsack problem was the ability to compute lower and upper bounds for the optimal value that are within a factor of n of each other (using the maximum value piece that ﬁts in the knapsack to get the lower bound). Use the result of the previous exercise to derive a reﬁned approximation scheme that eliminates one factor of n in the running time of the algorithm. 3.3 Consider the following scheduling problem: there are n jobs to be scheduled on a single machine, where each job j has a processing time pj , a weight wj , and a due date dj , j = 1, . . . , n. The objective is to schedule the jobs so as to maximize the total weight of the jobs that complete by their due date. First prove that there always exists an optimal schedule in which all ontime jobs complete before all late jobs, and the ontime jobs complete in an earliest due date order; use this structural result to show ∑ how to solve this problem using dynamic programming in O(nW ) time, where W = j wj . Now use this result to derive a fully polynomialtime approximation scheme. 3.4 Instead of maximizing the total weight of the ontime set of jobs, as in the previous exercise, one could equivalently minimize the total weight of the set of jobs that complete late. This equivalence is only valid when one thinks of optimization, not approximation, since if only one job needs to be late then our approximation for the minimization problem can make only a small error, whereas for the maximization problem the situation is quite diﬀerent. Or even worse, suppose that all jobs can complete on time. The good news is that there is an O(n2 ) algorithm to solve the following problem: minimize the weight of the maximumweight job that completes late. First devise this algorithm, and then show Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
78
Rounding data and dynamic programming
how to use it to derive a fully polynomialtime approximation scheme for the problem of minimizing the total weight of the jobs that complete late. 3.5 Consider the following scheduling problem: there are n jobs to be scheduled on a constant number of machines m, where each job j has a processing time pj and a weight wj , j = 1, . . . , n; once started, each job must be processed to completion on that machine without interruption. For a given schedule, let∑Cj denote the completion time of job j, j = 1, . . . , n, and the objective is to minimize j wj Cj over all possible schedules. First show that there exists an optimal schedule where, for each machine, the jobs are scheduled in nondecreasing pj /wj order. Then use this property to derive a dynamic programming algorithm that can then be used to obtain a fully polynomialtime approximation scheme. 3.6 Suppose we are given a directed acyclic graph with speciﬁed source node s and sink node t, and each arc e has an associated cost ce and length ℓe . We are also given a length bound L. Give a fully polynomialtime approximation scheme for the problem of ﬁnding a minimumcost path from s to t of total length at most L. 3.7 In the proof of Theorem 3.7, we rounded down each processing time to the nearest multiple of T /k 2 . Suppose that instead of constructing identical length intervals that get rounded to the same value, we have intervals that are geometrically increasing in length. Design an alternative polynomialtime approximation scheme, where the term in the running that 2 is O(n(1/k) ) is reduced to O(n(1/k) log(1/k) ). 3.8 The makespan minimization problem discussed in Section 3.2 can be viewed as minimizing the L∞ norm of the machine loads. One can instead minimize the L2 norm of the machine loads or, equivalently, the sum of the squares of the machine loads. To extend the framework discussed for the makespan objective, ﬁrst give a dynamic programming algorithm that solves, in polynomial time, the special case in which there are only a constant number of job lengths, ∑ and each job length is at least a constant fraction of the average machine load (that is, j pj /m, where m is the number of machines). Then use this to derive a polynomialtime approximation scheme. (One additional idea might be useful: for some notion of a “small” job, clump the small jobs into relatively “small” clumps of jobs that are then assigned to machines together.) 3.9 Suppose we have a strongly NPhard minimization problem Π with the following two properties. First, any feasible solution has a nonnegative, integral objective function value. Second, there is a polynomial p, such that if it takes n bits to encode the input instance I in unary, OPT(I) ≤ p(n). Prove that if there is a fully polynomialtime approximation scheme for Π, then there is a pseudopolynomial algorithm for Π. Since there is no pseudopolynomial algorithm for a strongly NPhard problem unless P = NP, conclude that this would imply P = NP.
Chapter Notes The approach to describing the dynamic program for the knapsack problem in Section 3.1 is due to Lawler [210], as is Exercise 3.2. The approximation scheme is due to Ibarra and Kim [173]. Exercises 3.3 and 3.5 are due to Sahni [258]. Gens and Levner [129] proved the result stated in Exercise 3.4. Throughout the 1970s much of the work in this area was being done in parallel in both the United States and the Soviet Union (although not recognized at the time); Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
3.3
The binpacking problem
79
Gens and Levner [128] give a detailed comparison of the evolution of this area at that time. Exercise 3.6 is due to Hassin [157]. The approximation scheme for scheduling parallel machines with a constant number of machines given in Section 3.2 is due to Graham [143]. The approximation scheme for the case in which the number of machines is part of the input is due to Hochbaum and Shmoys [165]. Exercise 3.8 is due to Alon, Azar, Woeginger, and Yadid [6]. Fernandez de la Vega and Lueker [111] gave the ﬁrst approximation scheme for the binpacking problem. The approximation scheme given here is a variant of theirs. The analysis of the FirstFitDecreasing algorithm, showing it to be an 11/9approximation algorithm, is in the Ph.D. thesis of Johnson [178]. The analysis of FirstFit we give is also in this thesis, although we use the analysis there of another algorithm called NextFit. Exercise 3.9 is due to Garey and Johnson [122].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
80
Rounding data and dynamic programming
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 4
Deterministic rounding of linear programs
In the introduction, we said that one of the principal theses of this book is the central role played by linear programming in the design and analysis of approximation algorithms. In the previous two chapters, we have not used linear programming at all, but starting with this chapter we will be using it extensively. In this chapter, we will look at one of the most straightforward uses of linear programming. Given an integer programming formulation of a problem, we can relax it to a linear program. We solve the linear program to obtain a fractional solution, then round it to an integer solution via some process. The easiest way to round the fractional solution to an integer solution in which all values are 0 or 1 is to take variables with relatively large values and round them up to 1, while rounding all other variables down to 0. We saw this technique in Section 1.3 applied to the set cover problem, in which we chose sets whose corresponding linear programming variables were suﬃciently large. We will see another application of this technique when we introduce the prizecollecting Steiner tree problem in Section 4.4. We will revisit this problem several times in the course of the book. For this problem we give an integer programming relaxation in which there are 01 variables for both nodes and edges. We round up the node variables that are suﬃciently large in order to decide which nodes should be spanned in a solution; we then ﬁnd a tree spanning these nodes. In Sections 4.1 and 4.2, we consider a singlemachine scheduling problem, and see another way of rounding fractional solutions to integer solutions. We will see that by solving a relaxation, we are able to get information on how the jobs might be ordered. Then we construct a solution in which we schedule jobs in the same order as given by the relaxation, and we are able to show that this leads to a good solution. In the ﬁrst section, we use a relaxation of the problem to a preemptive scheduling problem, rather than a linear program. The analysis of this algorithm gives some ideas for an algorithm in the next section that uses a linear programming relaxation for a scheduling problem with a more general objective function. In Section 4.5, we introduce the uncapacitated facility location problem, another problem that we will revisit several times in the course of this book. In this problem we have clients and facilities; we must choose a subset of facilities to open, and every client must be assigned to some open facility. Here the rounding procedure is much more complex than simply choosing 81
82
Deterministic rounding of linear programs
Job 1
0
Job 2
2
4
5
Job 3
9
Time
Figure 4.1: An example of a nonpreemptive schedule, in which p1 = 2, r1 = 0, p2 = 1, r∑ 2 = 4, p3 = 4, r3 = 1. In this schedule, C1 = 2, C2 = 5, and C3 = 9, so that j Cj = 2 + 5 + 9 = 16. large values and rounding up. The fractional values indicate to us which facilities we might consider opening and which assignments we might think about making. Finally, in Section 4.6, we revisit the binpacking problem. We give an integer programming formulation in which we have an integer variable to indicate how many bins should be packed in a certain way. Then we round variables down, in the sense that we take the integer part of each variable and pack bins as indicated, then take all the pieces corresponding to the fractional part and iterate. In several cases in this chapter and later in the book, we will need to solve very large linear programs in which the number of constraints is exponential in the input size of the problem. These linear programs can be solved in polynomial time using an algorithm called the ellipsoid method. We introduce the ellipsoid method in Section 4.3, and discuss the cases in which it can be used to solve exponentially large linear programs in polynomial time.
4.1
Minimizing the sum of completion times on a single machine
In this section, we consider the problem of scheduling jobs on a single machine so as to minimize the sum of the job completion times. In particular, we are given as input n jobs, each of which has a processing time pj and release date rj . The values pj and rj are integers such that rj ≥ 0 and pj > 0. We must construct a schedule for these jobs on a single machine such that at most one job is processed at each point in time, no job is processed before its release date, and each job must be processed nonpreemptively; that is, once a job begins to be processed, it must be processed completely before any other job begins its processing. See Figure 4.1 for an example. If Cj denotes the time ∑n at which job j is ﬁnished processing, then the goal is to ﬁnd the schedule that minimizes j=1 Cj . Observe that this objective is equivalent to the problem of minimizing the average completion time, since the average completion time just rescales the objective function for each feasible solution by a factor of 1/n. Below we will show that we can convert any preemptive schedule into a nonpreemptive schedule in such a way that the completion time of each job at most doubles. In a preemptive schedule, we can still schedule only one job at a time on the machine, but we do not need to complete each job’s required processing consecutively; we can interrupt the processing of a job with the processing of other jobs. In the preemptive version of this problem, the goal is to ﬁnd a preemptive schedule that minimizes the sum of the completion times. An optimal solution to the preemptive version of the scheduling problem can be found in polynomial time via the shortest remaining processing time (SRPT) rule, which is as follows. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.1
Minimizing the sum of completion times on a single machine
Job 1
0
83
Job 3 Job 2
2
4
5
7
Time
Figure 4.2: An example of a preemptive schedule created using the SRPT rule, with the same ∑ instance as in Figure 4.1. In this schedule, C1 = 2, C2 = 5, and C3 = 7, so that j Cj = 2 + 5 + 7 = 14. Note that we do not interrupt the processing of job 1 when job 3 arrives at time 1, since job 1 has less remaining processing time, but we do interrupt the processing of job 3 when job 2 arrives. We start at time 0 and schedule the job with the smallest amount of remaining processing time as long as the job is past its release date and we have not already completed it. We schedule it until either it is completed or a new job is released. We then iterate. See Figure 4.2 for an example. Let CjP be the completion time of job j in an optimal preemptive schedule, and let OPT be the sum of completion times in an optimal nonpreemptive schedule. We have the following observation. Observation 4.1:
n ∑
CjP ≤ OPT .
j=1
Proof. This immediately follows from the fact that an optimal nonpreemptive schedule is feasible for the preemptive scheduling problem. Now consider the following scheduling algorithm. Find an optimal preemptive schedule using SRPT. We schedule the jobs nonpreemptively in the same order that they complete in this preemptive schedule. To be more precise, suppose that the jobs are indexed such that C1P ≤ C2P ≤ · · · ≤ CnP . Then we schedule job 1 from its release date r1 to time r1 + p1 . We schedule job 2 to start as soon as possible after job 1; that is, we schedule it from max(r1 +p1 , r2 ) to max(r1 + p1 , r2 ) + p2 . The remaining jobs are scheduled analogously. If we let CjN denote the completion time of job j in the nonpreemptive schedule that we construct, for j = 1, . . . , n, N , r } to max{C N , r } + p . then job j is processed from max{Cj−1 j j j−1 j We show below that scheduling nonpreemptively in this way does not delay the jobs by too much. Lemma 4.2: For each job j = 1, . . . , n, CjN ≤ 2CjP . Proof. Let us ﬁrst derive some easy lower bounds on CjP . Since we know that j is processed in the optimal preemptive schedule after jobs 1, . . . , j − 1, we have CjP
≥ max rk and k=1,...,j
CjP
≥
j ∑
pk .
k=1
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
84
Deterministic rounding of linear programs
By construction it is also the case that CjN ≥ max rk . k=1,...,j
Consider the nonpreemptive schedule constructed by the algorithm, and focus on any period of time that the machine is idle; idle time occurs only when the next job to be processed has not yet been released. Consequently, in the time interval from maxk=1,...,j rk to CjN , there cannot be any∑point in time at which the machine is idle. Therefore, this interval can be of length at most jk=1 pk since otherwise we would run out of jobs to process. This implies that CjN
≤ max rk + k=1,...,j
j ∑
pk ≤ 2CjP ,
k=1
where the last inequality follows from the two lower bounds on CjP derived above. This leads easily to the following theorem. Theorem 4.3: Scheduling in order of the completion times of an optimal preemptive schedule is a 2approximation algorithm for scheduling a single machine with release dates to minimize the sum of completion times. Proof. We have that
n ∑ j=1
CjN
≤2
n ∑
CjP ≤ 2 OPT,
j=1
where the ﬁrst inequality follows by Lemma 4.2 and the second by Observation 4.1.
4.2
Minimizing the weighted sum of completion times on a single machine
We now consider a generalization of the problem from the previous section. In this generalization, each job has a weight wj ≥ 0, and our goal is to minimize the weighted sum of completion times. If Cj denotes the time ∑ at which job j is ﬁnished processing, then the goal is to ﬁnd a schedule that minimizes nj=1 wj Cj . We will call the problem of the previous section the unweighted case, and the problem in this section the weighted case. Unlike the unweighted case, it is NPhard to ﬁnd an optimal schedule for the preemptive version of the weighted case. Although the algorithm and analysis of the previous section give us a way to round any preemptive schedule to one whose sum of weighted completion times is at most twice more, we cannot use the same technique of ﬁnding a lower bound on the cost of the optimal nonpreemptive schedule by ﬁnding an optimal preemptive schedule. Nevertheless, we can still get a constant approximation algorithm for this problem by using some of the ideas from the previous section. To obtain the 2approximation algorithm in the previous section, we used that CjN ≤ 2CjP ; if we look at the proof of Lemma 4.2, to prove this inequality we used only that the completion times CjP satisﬁed CjP ≥ maxk=1,...,j rk and ∑ CjP ≥ jk=1 pk (assuming that jobs are indexed such that C1P ≤ C2P ≤ · · · ≤ CnP ). Furthermore, ∑ in order to obtain an approximation algorithm, we needed that nj=1 CjP ≤ OPT. We will show below that we can give a linear programming relaxation of the problem with variables Cj such Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.2
Minimizing the weighted sum of completion times on a single machine
85
that these inequalities hold within a constant factor, which in turn will lead to a constant factor approximation algorithm for the problem. To construct our linear programming relaxation, we will let the variable Cj ∑ denote the completion time of job j. Then our objective function is clear: we want to minimize nj=1 wj Cj . We now need to give constraints. The ﬁrst set of constraints is easy: for each job j = 1, . . . , n, job j cannot complete before it is released and processed, so that Cj ≥ rj + pj . In order to introduce the second set of constraints, consider some set S ⊆ {1, . . . , n} of jobs. ∑ Consider the sum j∈S pj Cj . This sum is minimized when all the jobs in S have a release date of zero and all the jobs in S ﬁnish ﬁrst in the schedule. Assuming these two conditions hold, then any completion time Cj for j ∈ S is equal to pj plus the sum of all the processing times of the jobs in S that preceded j in the schedule. Then in the product pj Cj , pj∑ multiplies itself and the processing times of all jobs in S preceding j in the schedule. The sum j∈S pj Cj must contain the term pj pk for all pairs j, k ∈ S, since either k precedes j or j precedes k in the schedule. Thus, ∑ j∈S
pj Cj =
∑
pj pk =
j,k∈S:j≤k
1 2
∑
2 pj +
j∈S
1∑ 2
j∈S
p2j ≥
1 2
∑
2 pj .
j∈S
∑ p , so that the Simplifying notation somewhat, let N = {1, . . . , n}, and let p(S) = j∈S ∑ ∑ j inequality above becomes j∈S pj Cj ≥ 12 p(S)2 . As we said above, the sum j∈S pj Cj can be greater only if the jobs in S have a release date greater than zero or do not ﬁnish ﬁrst in the schedule, so the inequality must hold unconditionally for any S ⊆ N . Thus, our second set of constraints is ∑ 1 pj Cj ≥ p(S)2 2 j∈S
for each S ⊆ N. This gives our linear programming relaxation for the scheduling problem: minimize
n ∑
wj Cj
(4.1)
j=1
subject to
∑ j∈S
C j ≥ rj + p j , 1 pj Cj ≥ p(S)2 , 2
∀j ∈ N,
(4.2)
∀S ⊆ N.
(4.3)
By the arguments above, this LP is a relaxation of the problem, so that for an optimal LP ∑ solution C ∗ , nj=1 wj Cj∗ ≤ OPT, where OPT denotes the value of the optimal solution to the problem. There are an exponential number of the second type of constraint (4.3), but we will show in Section 4.3 that we can use an algorithm called the ellipsoid method to solve this linear program in polynomial time. Let C ∗ be an optimal solution for the relaxation that we obtain in polynomial time. Our algorithm is then almost the same as in the previous section: we schedule the jobs nonpreemptively in the same order as of the completion times of C ∗ . That is, suppose that the jobs are reindexed so that C1∗ ≤ C2∗ ≤ · · · ≤ Cn∗ . Then, as in the previous section, we schedule job 1 from its release date r1 to time r1 + p1 . We schedule job 2 to start as soon as possible after job 1; that is, we schedule it from max(r1 + p1 , r2 ) to max(r1 + p1 , r2 ) + p2 . The remaining jobs are scheduled similarly. We claim that this gives a 3approximation algorithm for the problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
86
Deterministic rounding of linear programs
Theorem 4.4: Scheduling in order of the completion time of C ∗ is a 3approximation algorithm for scheduling jobs on a single machine with release dates to minimize the sum of weighted completion times. Proof. Again, assume that the jobs are reindexed so that C1∗ ≤ C2∗ ≤ · · · ≤ Cn∗ . Let CjN be the completion time of job j in the schedule we construct. We will show that CjN ≤ 3Cj∗ for each ∑ ∑ j = 1, . . . , n. Then we have that nj=1 wj CjN ≤ 3 nj=1 wj Cj∗ ≤ 3 OPT, which gives the desired result. As in the proof of Lemma 4.2, there cannot be any idle time between maxk=1,...,j rk and CjN , ∑ and therefore it must be the case that CjN ≤ maxk=1,...,j rk + jk=1 pk . Let ℓ ∈ {1, . . . , j} be the index of the job that maximizes maxk=1,...,j rk so that rℓ = maxk=1,...,j rk . By the indexing of the jobs, Cj∗ ≥ Cℓ∗ , and Cℓ∗ ≥ rℓ by the LP constraint (4.2); thus, Cj∗ ≥ maxk=1,...,j rk . Let [j] denote the set {1, . . . , j}. We will argue that Cj∗ ≥ 21 p([j]), and from these simple facts, it follows that CjN ≤ p([j]) + max rk ≤ 2Cj∗ + Cj∗ = 3Cj∗ . k=1,...,j
Let S = [j]. From the fact that
C∗
is a feasible LP solution, we know that
∑ k∈S
1 pk Ck∗ ≥ p(S)2 . 2
However, by our relabeling, Cj∗ ≥ · · · ≥ C1∗ , and hence Cj∗
∑ k∈S
pk = Cj∗ · p(S) ≥
∑
pk Ck∗ .
k∈S
By combining these two inequalities and rewriting, we see that 1 Cj∗ · p(S) ≥ p(S)2 . 2 Dividing both sides by p(S) shows that Cj∗ ≥ 12 p(S) = 21 p([j]). In Section 5.9, we will show how randomization can be used to obtain a 2approximation algorithm for this problem.
4.3
Solving large linear programs in polynomial time via the ellipsoid method
We now turn to the question of how to solve the linear program (4.1) in polynomial time. The most popular and practical algorithm for solving linear programs is known as the simplex method. Although the simplex method is quite fast in practice, there is no known variant of the algorithm that runs in polynomial time. Interiorpoint methods are a class of algorithms for solving linear programs; while typically not as fast or as popular as the simplex method, interiorpoint methods do solve linear programs in polynomial time. However, this isn’t suﬃcient for solving the linear program above because the size of the linear program is exponential in the size of the input scheduling instance. Therefore, we will use a linear programming algorithm called the ellipsoid method. Because we will use this technique frequently, we will discuss the general technique before turning to how to solve our particular linear program. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.3
Solving large linear programs in polynomial time via the ellipsoid method
87
Suppose we have the following general linear program: minimize
n ∑
dj xj
(4.4)
j=1
subject to
n ∑
aij xj ≥ bi ,
i = 1, . . . , m,
j=1
xj ≥ 0,
∀j.
Suppose that we can give a bound ϕ on the number of bits needed to encode any inequality ∑n a x ij j ≥ bi . Then the ellipsoid method for linear programming allows us to ﬁnd an optimal j=1 solution to the linear program in time polynomial in n (the number of variables) and ϕ, given a polynomialtime separation oracle (which we will deﬁne momentarily). It is sometimes desirable for us to obtain an optimal solution that has an additional property of being a basic solution; we do not deﬁne basic solutions here, but the ellipsoid method will return such basic optimal solutions (see Chapter 11 or Appendix A for a deﬁnition). Note that this running time does not depend on m, the number of constraints of the linear program. Thus, as in the case of the previous linear program for the singlemachine scheduling problem, we can solve linear programs with exponentially many constraints in polynomial time given that we have a polynomialtime separation oracle. A separation oracle takes as input a supposedly feasible solution x to the linear program, and either veriﬁes that x is indeed a feasible solution to the linear program or, ∑nif it is infeasible, produces a constraint that is violated by x. That is, if it is not the case that j=1 ∑ aij xj ≥ bi for each i = 1, . . . , m, then the separation oracle returns some constraint i such that nj=1 aij xj < bi . In the notes at the end of the chapter, we sketch how a polynomialtime separation oracle leads to a polynomialtime algorithm for solving LPs with exponentially many constraints. It is truly remarkable that such LPs are eﬃciently solvable. Here, however, eﬃciency is a relative term: the ellipsoid method is not a practical algorithm. For exponentially large LPs that we solve via the ellipsoid method, it is sometimes the case that the LP can be written as a polynomially sized LP, but it is more convenient to discuss the larger LP. We will indicate when this is the case. Even if there is no known way of rewriting the exponentially sized LP, one can heuristically ﬁnd an optimal solution to the LP by repeatedly using any LP algorithm (typically the simplex method) on a small subset of the constraints and using the separation oracle to check if the current solution is feasible. If it is feasible, then we have solved the LP, but if not, we add the violated constraints to the LP and resolve. Practical experience shows that this approach is much more eﬃcient than can be theoretically justiﬁed, and should be in the algorithm designer’s toolkit. We now turn to the scheduling problem at hand, and give a polynomialtime separation oracle for the constraints (4.3). Given a solution C, let us reindex the variables so that C1 ≤ C2 ≤ · · · ≤ Cn . Let S1 = {1}, S2 = {1, 2}, . . . , Sn = {1, . . . , n}. We claim that it is suﬃcient to check whether the constraints are violated for the n sets S1 , . . . , Sn . If any of these n constraints are violated, then we return the set as a violated constraint. If not, we show below that all constraints are satisﬁed. Lemma 4.5: Given variables Cj , if constraints (4.3) are satisﬁed for the n sets S1 , . . . , Sn , then they are satisﬁed for all S ⊆ N . ∑ Proof. Let S be a constraint that is not satisﬁed; that is, j∈S pj Cj < 12 p(S)2 . We will show Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
88
Deterministic rounding of linear programs
that then there must be some set S∑ i that is also not satisﬁed. We do this by considering changes to S that decrease the diﬀerence j∈S pj Cj − 21 p(S)2 . Any such change will result in another set S ′ that also does not satisfy the constraint. Note that removing a job k from S decreases this diﬀerence if 1 −pk Ck + pk p(S − k) + p2k < 0, 2 or if Ck > p(S − k) + 12 pk . Adding a job k to S decreases this diﬀerence if 1 pk Ck − pk p(S) − p2k < 0, 2 or if Ck < p(S) + 12 pk . Now let ℓ be the highest indexed job in S. We remove ℓ from S if Cℓ > p(S − ℓ) + 21 pℓ ; by the reasoning above the resulting set S − ℓ also does not satisfy the corresponding constraint (4.3). We continue to remove the highest indexed job in the resulting set until ﬁnally we have a set S ′ such that its highest indexed job ℓ has Cℓ ≤ p(S ′ − ℓ) + 21 pℓ < p(S ′ ) (using pℓ > 0). Now suppose S ′ ̸= Sℓ = {1, . . . , ℓ}. Then there is some k < ℓ such that k ∈ / S ′ . Then since Ck ≤ Cℓ < p(S ′ ) < p(S ′ ) + 21 pk , adding k to S ′ can only decrease the diﬀerence. Thus, we can add all k < ℓ to S ′ , and the resulting set Sℓ will also not satisfy the constraint (4.3).
4.4
The prizecollecting Steiner tree problem
We now turn to a variation of the Steiner tree problem introduced in Exercise 2.5. This variation is called the prizecollecting Steiner tree problem. As input, we are given an undirected graph G = (V, E), an edge cost ce ≥ 0 for each e ∈ E, a selected root vertex r ∈ V , and a penalty πi ≥ 0 for each i ∈ V . The goal is to ﬁnd a tree T that contains the root vertex r so as to minimize the cost of the edges in the tree plus the penalties of all vertices not in∑ the tree. In ∑other words, if V (T ) is the set of vertices in the tree T , the goal is to minimize e∈T ce + i∈V −V (T ) πi . The Steiner tree problem of Exercise 2.5 is a special case of the problem in which for every i ∈ V either πi = ∞ (and thus i must be included in the tree) or πi = 0 (and thus i may be omitted from the tree with no penalty). The vertices that must be included in the solution in the Steiner tree problem are called terminals. One application of the prizecollecting Steiner tree problem is deciding how to extend cable access to new customers. In this case, each vertex represents a potential new customer, the cost of edge (i, j) represents the cost of connecting i to j by cable, the root vertex r represents a site that is already connected to the cable network, and the “penalties” πi represent the potential proﬁt to be gained by connecting customer i to the network. Thus, the goal is to minimize the total cost of connecting new customers to the network plus the proﬁts lost by leaving the unconnected customers out of the network. The prizecollecting Steiner tree problem can be modeled as an integer program. We use a 01 variable yi for each i ∈ V and xe for each e ∈ E. Variable yi is set to 1 if i is in the solution tree and 0 if it is not, while xe is set to 1 if e is in the solution tree and 0 if it is not. Obviously, then, the objective function is ∑ ∑ minimize ce xe + πi (1 − yi ). e∈E
i∈V
We now need constraints to enforce that the tree connects the root r to each vertex i with yi = 1. Given a nonempty set of vertices S ⊂ V , let δ(S) denote the set of edges in the cut Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.4
The prizecollecting Steiner tree problem
89
deﬁned by S; that is, δ(S) is the set of all edges with exactly one endpoint in S. We will introduce the constraints ∑ xe ≥ y i e∈δ(S)
for each i ∈ S, and each S ⊆ V − r. To see that these constraints enforce that the tree connects the root to each vertex i with yi = 1, take any feasible solution x and consider the graph G′ = (V, E ′ ) with E ′ = {e ∈ E : xe = 1}. Pick any i such that yi = 1. The constraints ensure that for any ri cut S, there must be at least one edge of E ′ in δ(S): that is, the size of the minimum ri cut in G′ must be at least one. Thus, by the maxﬂow/mincut theorem the maximum ri ﬂow in G′ is at least one, which implies that there is a path from r to i in G′ . Similarly, if x is not feasible, then there is some ri cut S for which there are no edges of E ′ in δ(S), which implies that the size of minimum ri cut is zero, and thus the maximum ri ﬂow is zero. Hence, there is a path from r to i with yi = 1 if and only if these constraints are satisﬁed, and if they are all satisﬁed, there will be a tree connecting r to all i such that yi = 1. Thus, the following integer program models the prizecollecting Steiner tree problem: ∑ ∑ minimize ce xe + πi (1 − yi ) e∈E
subject to
i∈V
∑
xe ≥ y i ,
∀S ⊆ V − r, S ̸= ∅, ∀i ∈ S,
(4.5)
e∈δ(S)
yr = 1, yi ∈ {0, 1} ,
∀i ∈ V,
xe ∈ {0, 1} ,
∀e ∈ E.
In order to apply deterministic rounding to the problem, we relax the integer programming formulation to a linear programming relaxation by replacing the constraints yi ∈ {0, 1} and xe ∈ {0, 1} with yi ≥ 0 and xe ≥ 0. We can now apply the ellipsoid method, introduced in Section 4.3, ∑ to solve the linear program in polynomial time. The separation oracle for the constraints e∈δ(S) xe ≥ yi is as follows: given a solution (x, y), we construct a network ﬂow problem on the graph G in which the capacity of each edge e is set to xe . For each vertex i, we check whether the maximum ﬂow from i to the root r is at least y∑ i . If not, then the minimum cut S separating i from r gives a violated constraint such that e∈δ(S) xe < yi for i ∈ S, S ⊆ ∑ V − r. If the ﬂow is at least yi , then for all cuts S separating i from r, i ∈ S, S ⊆ V − r, e∈δ(S) xe ≥ yi by the maxﬂow/mincut theorem. Hence, given a solution (x, y), we can ﬁnd a violated constraint, if any exists, in polynomial time. Given an optimal solution (x∗ , y ∗ ) to the linear programming relaxation, there is a very simple deterministic rounding algorithm for the problem, which we give in Algorithm 4.1. It is similar to the deterministic rounding algorithm for the set cover problem given in Section 1.3: for some value α ∈ [0, 1), let U be the set of all vertices such that yi ≥ α. Since yr = 1, the root r is in U . Now build a tree on the vertices in U in the graph G. Since we want to build a tree with cost as low as possible, this is just a Steiner tree problem on the graph G in which the terminals are the set U . We could apply the algorithm given in Exercise 2.5. Instead, we use another algorithm, given in Exercise 7.6 of Chapter 7, whose analysis will be very easy once we have studied the primaldual method of that chapter. Let T be the tree produced by that algorithm. We now begin the analysis of the algorithm in terms of the parameter α. Later we will ﬁx the value of α to give the best possible performance guarantee for the deterministic rounding Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
90
Deterministic rounding of linear programs
Solve LP, get optimal primal solution (x∗ , y ∗ ) U ← {i ∈ V : yi∗ ≥ α} Use algorithm of Exercise 7.6 to build tree T on U return T Algorithm 4.1: Deterministic rounding algorithm for the prizecollecting Steiner tree problem. algorithm. We will analyze the cost of the constructed tree T and the penalties on the vertices not in T separately, comparing them to their contribution to the objective function of the linear programming relaxation. We use the result of Exercise 7.6 to analyze the cost of the tree T . Lemma 4.6 (Exercise 7.6): The tree T returned by the algorithm of Exercise 7.6 has cost ∑
ce ≤
e∈T
2∑ ce x∗e . α e∈E
The analysis for the penalties is simple. Lemma 4.7:
∑
πi ≤
i∈V −V (T )
1 ∑ πi (1 − yi∗ ) 1−α i∈V
Proof. If i is not in the tree T , then clearly it was not in the set of vertices U , and so yi∗ < α. 1−y ∗ Thus, 1 − yi∗ > 1 − α, and 1−αi > 1. The lemma statement follows. Combining the two lemmas immediately gives the following theorem and corollary. Theorem 4.8: The cost of the solution produced by Algorithm 4.1 is ∑ e∈T
ce +
∑ i∈V −V (T )
πi ≤
1 ∑ 2∑ ce x∗e + πi (1 − yi∗ ). α 1−α e∈E
Corollary 4.9: Using Algorithm 4.1 with α = prizecollecting Steiner tree problem.
i∈V
2 3
gives a 3approximation algorithm for the
Proof. Clearly the algorithm runs in polynomial time. We can bound the performance guarantee 1 by the max{2/α, 1/(1 − α)}. We can minimize this maximum by setting α2 = 1−α ; thus, we 2 obtain α = 3 . Then by Theorem 4.8, the cost of the tree obtained is ( ) ∑ ∑ ∑ ∑ ce + πi ≤ 3 ce x∗e + πi (1 − yi∗ ) ≤ 3 OPT, e∈T
i∈V −V (T )
e∈E
i∈V
∑ ∑ since e∈E ce x∗e + i∈V πi (1−yi∗ ) is the objective function of the linear programming relaxation of the problem. In Algorithm 4.1, we choose a set of vertices U such that yi∗ ≥ α = 2/3 and construct a Steiner tree on that set of vertices. A very natural idea is to try all possible values α: since ∗ there are V  variables yi∗ , there { } are at most V  distinct values of yi . Thus, we can construct
V  sets Uj = i ∈ V : yi∗ ≥ yj∗ . For each j ∈ V , construct a Steiner tree Tj on the vertices Uj , and return the best overall solution out of the V  solutions constructed. Unfortunately, we do Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.5
The uncapacitated facility location problem
91
not know how to analyze such an algorithm directly. However, we will see in Chapter 5 that this algorithm can be analyzed as a deterministic variant of a randomized algorithm. We will return to the prizecollecting Steiner tree problem in Section 5.7.
4.5
The uncapacitated facility location problem
We now start our consideration of the uncapacitated facility location problem. We will return to this problem many times, since many of the techniques we discuss in this book are useful for devising approximation algorithms for this problem. In the uncapacitated facility location problem, we have a set of clients or demands D and a set of facilities F . For each client j ∈ D and facility i ∈ F , there is a cost cij of assigning client j to facility i. Furthermore, there is a cost fi associated with each facility i ∈ F . The goal of the problem is to choose a subset of facilities F ′ ⊆ F so as to minimize the total cost of the facilities in F ′ and the cost of assigning each client∑j ∈ D to∑the nearest facility in F ′ . In other words, we wish to ﬁnd F ′ so as to minimize i∈F ′ fi + j∈D mini∈F ′ cij . We call the ﬁrst part of the cost the facility cost and the second part of the cost the assignment cost or service cost. We say that we open the facilities in F ′ . The uncapacitated facility location problem is a simpliﬁed variant of a problem that arises in many diﬀerent contexts. In one large computer company, a version of this problem occurs in deciding where to build or lease facilities to warehouse expensive parts needed for computer repair. The clients represent customers with service agreements that might require the use of the part. The facility cost is the cost of building or leasing the warehouse, and the assignment cost is the distance from the warehouse to the customer. In this problem it is also important to ensure that almost all of the customers are within a fourhour trip of a facility containing the needed part. Other typical complications include limits on the number of clients that can be served by a given facility (the capacitated facility location problem), and multiple types of facilities (for example, both distribution centers and warehouses, with clients assigned to warehouses, and warehouses assigned to distribution centers). In its full generality, the uncapacitated facility location problem is as hard to approximate as the set cover problem (see Exercise 1.4). However, it is relatively common for facilities and clients to be points in a metric space, with assignment costs cij representing the distance from facility i to client j. For this reason, we will from here on consider the metric uncapacitated facility location problem, in which clients and facilities correspond to points in a metric space, and assignment costs obey the triangle inequality. More precisely, given clients j, l and facilities i, k, we have that cij ≤ cil + ckl + ckj (see Figure 4.3). Since the clients and facilities correspond to points in a metric space, we assume that we have distances cii′ between two facilities i and i′ , and cjj ′ between two clients j and j ′ ; we will not need this assumption in this section, but it will prove useful in later sections. We are able to give much better approximation algorithms for the metric version of the problem than we can for the more general version. Our ﬁrst approach to this problem will be to apply a deterministic rounding technique, from which we will get a 4approximation algorithm. In subsequent chapters we will get improved performance guarantees by applying randomized rounding, the primaldual method, local search, and greedy techniques. We begin by deﬁning an integer programming formulation for the problem. We will have decision variables yi ∈ {0, 1} for each facility i ∈ F ; if we decide to open facility i, then yi = 1 and yi = 0 otherwise. We also introduce decision variables xij ∈ {0, 1} for all i ∈ F and all j ∈ D; if we assign client j to facility i, then xij = 1 while xij = 0 otherwise. Then the objective function is relatively simple: we wish to minimize the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
92
Deterministic rounding of linear programs
l k
i j
Figure 4.3: An illustration of the inequality obeyed by the assignment costs. The circles represent clients, and the squares represent facilities. For clients j, l and facilities i, k, cij ≤ cil + ckl + ckj . total facility cost plus the total assignment cost. This can be expressed as ∑ ∑ minimize fi yi + cij xij . i∈F
i∈F,j∈D
We need to ensure that each client j ∈ D is assigned to exactly one facility. This can be expressed via the following constraint: ∑ xij = 1. i∈F
Finally, we need to make sure that clients are assigned to facilities that are open. We achieve this by introducing the constraint xij ≤ yi for all i ∈ F and j ∈ D; this ensures that whenever xij = 1 and client j is assigned to facility i, then yi = 1 and the facility is open. Thus, the integer programming formulation of the uncapacitated facility location problem can be summarized as follows: ∑ ∑ minimize fi yi + cij xij (4.6) i∈F
subject to
i∈F,j∈D
∑
xij = 1,
∀j ∈ D,
(4.7)
xij ≤ yi ,
∀i ∈ F, j ∈ D,
(4.8)
xij ∈ {0, 1},
∀i ∈ F, j ∈ D,
i∈F
yi ∈ {0, 1},
∀i ∈ F.
As usual, we obtain a linear programming relaxation from the integer program by replacing the constraints xij ∈ {0, 1} and yi ∈ {0, 1} with xij ≥ 0 and yi ≥ 0. It will be useful for us to consider the dual of the linear programming relaxation. Although one can derive the dual in a purely mechanical way, we will instead motivate it as a natural lower bound on the uncapacitated facility location problem. The most trivial lower bound for this problem is to ignore the facility costs entirely (that is, to pretend that the cost fi = 0, for each i ∈ F ). In that case, the optimal solution is to open all facilities and ∑ to assign each client to its nearest facility: if we set vj = mini∈F cij , then this lower bound is j∈D vj . How can this be improved? Suppose that ∑ each facility takes its cost fi , and divides it into shares apportioned among the clients: fi = j∈D wij , where each wij ≥ 0. The meaning of this share is that client Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.5
The uncapacitated facility location problem
93
j needs to pay this share only if it uses facility i. In this way, we no longer charge explicitly for opening a facility, but still recoup some of its cost. But in this way, we still have the situation that all of the (explicit) facility costs are 0, and so the optimal solution is to assign each client to the facility ∑ where the net cost is minimum; that is, we now can set vj = mini∈F (cij + wij ), and then j∈D vj is a lower bound on the true optimal value. Of course, we did not specify a way to determine each client’s share of facility cost fi ; we can make this an optimization problem by setting the shares so as to maximize the value of the ∑resulting lower bound. If we allow vj to be any value for which vj ≤ cij + wij and maximize j∈D vj , then at the optimum vj will be set to the smallest such righthand side. Thus, we have derived the following form for this lower bound, which is the dual linear program for the primal linear program (4.6): maximize
∑
vj
j∈D
subject to
∑
wij ≤ fi ,
∀i ∈ F,
j∈D
vj − wij ≤ cij , wij ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F, j ∈ D.
∗ If ZLP is the optimal value of the primal linear program (4.6) and OPT is the value of the optimal solution to the instance of the uncapacitated facility location problem, then ∑ for any feasible solution (v, w) to the dual linear program, by weak duality we have that j∈D vj ≤ ∗ ≤ OPT. ZLP Of course, as with any primaldual linear programming pair, we have a correspondence between primal constraints and dual variables, and vice versa. For example, the dual variable wij corresponds to the primal constraint xij ≤ yi , and the primal variable xij corresponds to the dual constraint vj − wij ≤ cij . We would like to use the information from an optimal solution (x∗ , y ∗ ) to the primal LP to determine a lowcost integral solution. In particular, if a client j is fractionally assigned to some facility i — that is, x∗ij > 0 — then perhaps we should also consider assigning j to i. We formalize this by saying that i is a neighbor of j (see Figure 4.4).
Deﬁnition 4.10: Given an LP solution x∗ , we say that facility i neighbors client j if x∗ij > 0. We let N (j) = {i ∈ F : x∗ij > 0}. The following lemma shows a connection between the cost of assigning j to a neighboring facility and the value of the dual variables. Lemma 4.11: If (x∗ , y ∗ ) is an optimal solution to the facility location LP and (v ∗ , w∗ ) is an optimal solution to its dual, then x∗ij > 0 implies cij ≤ vj∗ . ∗ = c . Furthermore, since w ∗ ≥ 0, Proof. By complementary slackness, x∗ij > 0 implies vj∗ − wij ij ij ∗ we have that cij ≤ vj .
The following is an intuitive argument about why neighboring facilities are useful. If we open a set of facilities S such that for all clients j ∈ D, there exists an open facility i ∈ N (j), and then the cost of assigning j∑to i is no more than vj∗ by Lemma 4.11. Thus, the total assignment cost is no more than j∈D vj∗ ≤ OPT, since this is the dual objective function. Unfortunately, such a set of facilities S might have a very high facility cost. What can we do to have a good facility cost? Here is an idea: suppose we can partition some subset F ′ ⊆ F Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
94
Deterministic rounding of linear programs
i j
h
l
Figure 4.4: A representation of the neighborhoods N (j) and N 2 (j). The circles represent clients, the squares represent facilities, and the edges are drawn between facilities i and clients k such that xik > 0. The central client is j. The surrounding facilities are the facilities N (j). The shaded clients are in N 2 (j). of the facilities into sets Fk such that each Fk = N (jk ) for some client jk . Then if we open the cheapest facility ik in N (jk ), we can bound the cost of ik by ∑ ∑ fi x∗ijk , x∗ijk ≤ fik = fik i∈N (jk )
i∈N (jk )
where the equality follows from constraint (4.7) of the linear program, and the inequality follows from the choice of ik as the cheapest facility in Fk . Using the LP constraints (4.8) that xij ≤ yi , we see that ∑ ∑ fi yi∗ . fi x∗ijk ≤ fik ≤ i∈N (jk )
i∈N (jk )
If we sum this inequality over all facilities that we open, then we have ∑ ∑ ∑ ∑ ∑ fi yi∗ = fi yi∗ ≤ fi yi∗ , fik ≤ k
k i∈N (jk )
i∈F ′
i∈F
where the equality follows since the N (jk ) partition F ′ . This scheme bounds the facility costs of open facilities by the facility cost of the linear programming solution. Opening facilities in this way does not guarantee us that every client will be a neighbor of an open facility. However, we can take advantage of the fact that assignment costs obey the triangle inequality and make sure that clients are not too far away from an open facility. We ﬁrst deﬁne an augmented version of neighborhood (see also Figure 4.4). Deﬁnition 4.12: Let N 2 (j) denote all neighboring clients of the neighboring facilities of client j; that is, N 2 (j) = {k ∈ D : client k neighbors some facility i ∈ N (j)}. Consider Algorithm 4.2. The algorithm loops until all clients are assigned to some facility. In each loop it picks the client jk that minimizes vj∗ ; we will see in a little bit why this is helpful. It then opens the cheapest facility ik in the neighborhood of N (jk ), and assigns jk and Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.6
The binpacking problem
95
Solve LP, get optimal primal solution (x∗ , y ∗ ) and dual solution (v ∗ , w∗ ) C←D k←0 while C ̸= ∅ do k ←k+1 Choose jk ∈ C that minimizes vj∗ over all j ∈ C Choose ik ∈ N (jk ) to be the cheapest facility in N (jk ) Assign jk and all unassigned clients in N 2 (jk ) to ik C ← C − {jk } − N 2 (jk ) Algorithm 4.2: Deterministic rounding algorithm for the uncapacitated facility location problem. all previously unassigned clients in N 2 (jk ) to ik . Note that by assigning the clients in N 2 (jk ) to ik , we ensure that the neighborhoods N (jk ) form a partition of a subset of the facilities: because no client in the neighborhood of any facility of N (jk ) is unassigned after iteration k, no facility of N (jk ) is a neighbor of some client jl chosen in later iterations (l > k). We can now analyze the performance of this algorithm. Theorem 4.13: Algorithm 4.2 is a 4approximation algorithm for the uncapacitated facility location problem. ∑ ∑ Proof. We have shown above that k fik ≤ i∈F fi yi∗ ≤ OPT . Fix an iteration k, and let j = jk and i = ik . By Lemma 4.11, the cost of assigning j to i is cij ≤ vj∗ . As depicted in Figure 4.4, consider the cost of assigning an unassigned client l ∈ N 2 (j) to facility i, where client l neighbors facility h that neighbors client j; then, applying the triangle inequality and Lemma 4.11, we see that cil ≤ cij + chj + chl ≤ vj∗ + vj∗ + vl∗ . Recall that we have selected client j in this iteration because, among all currently unassigned clients, it has the smallest dual variable vj∗ . However, l is also still unassigned, and so we know that vj∗ ≤ vl∗ . Thus, we can conclude that cil ≤ 3vl∗ . Combining all of these bounds, we see solution constructed has facility cost at most OPT and assignment cost at most ∑that the ∗ 3 j∈D vj ≤ 3 OPT (by weak duality), for an overall cost of at most 4 times optimal. In Section 5.8, we will see how randomization can help us improve this algorithm to a 3approximation algorithm. In subsequent chapters we will improve the performance guarantee still further. The following is known about the hardness of approximating the metric uncapacitated facility location problem via a reduction from the set cover problem. Theorem 4.14: There is no αapproximation algorithm for the metric uncapacitated facility location problem with constant α < 1.463 unless each problem in NP has an O(nO(log log n) ) time algorithm. We will prove this theorem in Section 16.2.
4.6
The binpacking problem
We return to the binpacking problem, for which we gave an asymptotic polynomialtime approximation scheme in Section 3.3. Recall that in the binpacking problem, we are given a collection of n items (or pieces), where item j is of size aj ∈ (0, 1], j = 1, . . . , n, and we wish Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
96
Deterministic rounding of linear programs
to assign each item to a bin, so that the total size assigned to each bin is at most 1; the objective is to use as few bins as possible. We showed previously that for each ϵ > 0, there is a polynomialtime algorithm that computes a solution with at most (1 + ϵ) OPTBP (I) + 1 bins, where OPTBP (I) is the optimal value for instance I of the problem. We shall improve on that result signiﬁcantly; we shall give a polynomialtime algorithm that ﬁnds a solution that uses at most OPTBP (I) + O(log2 (OPTBP (I))) bins. In obtaining this new bound, there will be three key components: an integer programming formulation to replace the dynamic programming formulation, and the integer program will be approximately solved by rounding its LP relaxation; an improved grouping scheme, which we call the harmonic grouping scheme; and an ingenious recursive application of the previous two components. We ﬁrst present the integer programming formulation on which the new algorithm will be based. Suppose that we group pieces of identical size, so that there are b1 pieces of the largest size s1 , b2 of the second largest size s2 , . . . , bm pieces of the smallest size sm . Consider the ways in which a single bin can be packed. The contents of each bin can be described by an mtuple (t1 , . . . , tm ), where ti indicates the ∑ number of pieces of size si that are included. We shall call such an mtuple a conﬁguration if i ti si ≤ 1; that is, the total size of the contents of the bin is at most 1, and so each conﬁguration corresponds to one feasible way to pack a bin. There might be an exponential number of conﬁgurations. Let N denote the number of conﬁgurations, and let T1 , . . . , TN be a complete enumeration of them, where tij denotes the ith component of Tj . We introduce a variable xj for each Tj that speciﬁes the number of bins packed according to conﬁguration Tj ; that is, xj is an integer variable. The total number of bins used can be computed by summing these variables. If we pack xj bins according to conﬁguration Tj , then this packs tij xj pieces of size si , i = 1, . . . , m. In this way, we can restrict attention to feasible solutions by requiring that we pack at least bi pieces of size si , in total. This leads to the following integer programming formulation of the binpacking problem; this is sometimes called the conﬁguration integer program:
minimize
N ∑
xj
(4.9)
j=1
subject to
N ∑
tij xj ≥ bi ,
i = 1, . . . , m,
xj ∈ N,
j = 1, . . . , N.
j=1
This formulation was introduced in the context of designing practical algorithms to ﬁnd optimal solutions to certain binpacking problems. Our algorithm is based on solving the linear programming relaxation of this integer program; we let ∑m OPTLP (I) denote the optimal value of this linear program. If we recall that SIZE(I) = i=1 si bi , then clearly we have that SIZE(I) ≤ OPTLP (I) ≤ OPTBP (I). Recall from Section 4.3 that the ellipsoid method is a polynomialtime algorithm for linear programs that have a polynomial number of variables, and for which there is a polynomialtime separation oracle to ﬁnd a violated constraint. The conﬁguration LP has few constraints, but an exponential number of variables. However, its dual linear program will then have m variables Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.6
The binpacking problem
97
and an exponential number of constraints, and is as follows: maximize subject to
m ∑
bi yi
i=1 m ∑
tij yi ≤ 1,
j = 1, . . . , N,
yi ≥ 0,
i = 1, . . . , m.
i=1
We observe that the problem of deciding whether or not there is a violated dual constraint given dual variables y is simply a knapsack problem. If we view yi as the value of piece i, then the dual constraints say that for all possible ways of packing pieces into a bin (or knapsack) of size 1, the total value is at most 1. Hence, to obtain a separation oracle, we can use an algorithm for the knapsack problem to decide, given values y, whether or not there is a way of packing pieces into a bin of size 1 that has value more than 1; such a packing will correspond to a violated dual constraint. Since the knapsack problem is NPhard, it would seem that it is not possible to obtain a polynomialtime separation oracle. However, by delving deeper into the analysis of the ellipsoid method, one can show that a fully polynomialtime approximation scheme for the knapsack problem (as given in Section 3.1) is suﬃcient to ensure the polynomialtime convergence of the ellipsoid method; we defer the details of how this can be done until Section 15.3, after which we give the problem as an exercise (Exercise 15.8). Hence, one can approximately solve the conﬁguration linear program, within an additive error of 1, in time bounded by a polynomial in m and log(n/sm ). In Section 3.3, one of the key ingredients for the asymptotic polynomialtime approximation scheme is a result that enables us to ignore pieces smaller than a given threshold γ. By applying this result, Lemma 3.10, we can assume, without loss of generality, that the smallest piece size sm ≥ 1/ SIZE(I), since smaller pieces can again be packed later without changing the order of magnitude of the additive error. The harmonic grouping scheme works as follows: process the pieces in order of decreasing size, close the current group whenever its total size is at least 2, and then start a new group with the next piece. Let r denote the number of groups, let Gi denote the ith group, and let ni denote the number of pieces in Gi . Observe that since we sort the pieces from largest to smallest, it follows that for i = 2, . . . , r − 1, we have that ni ≥ ni−1 . (For i = r, the group Gr need not have total size at least 2.) As in the proof of Lemma 3.11, from a given input I we form a new instance I ′ , where a number of pieces are discarded and packed separately. For each i = 2, 3, . . . , r − 1, we put ni−1 pieces in I ′ of size equal to the largest piece in Gi and discard the remaining pieces. In eﬀect, we discard G1 , Gr , and the ni − ni−1 smallest pieces in Gi , i = 2, . . . , r − 1, while increasing the size of the remaining pieces in a group to be equal to the largest one. Figure 4.5 shows the eﬀect of the harmonic grouping on one input. First of all, it should be clear that any packing of the rounded instance I ′ can be used to pack those pieces of the original instance that were not discarded; there is a natural mapping from any nondiscarded piece in the original instance to a piece at least as large in I ′ . The following lemma gives two key properties of the harmonic grouping scheme. Lemma 4.15: Let I ′ be the binpacking input obtained from an input I by applying the harmonic grouping scheme. The number of distinct piece sizes in I ′ is at most SIZE(I)/2; the total size of all discarded pieces is O(log SIZE(I)). Proof. The ﬁrst claim of the lemma is easy: each distinct piece size in I ′ corresponds to one Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
98
Deterministic rounding of linear programs
G1 n1 = 2 1
z }{
G2 n2 = 3 z
0.8
0.6
}
{
G3 n3 = 3 z
}
{
G4 n4 = 4 z
}
G5 n5 = 6
{ z
}
{
I 0.4
0.2
G6 n6 = 7 z
}
0
1
0.8
0.6
I0 0.4
0.2
0
Figure 4.5: An input before and after harmonic grouping.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
{
4.6
The binpacking problem
99
BinPack(I) if SIZE(I) < 10 then Pack remaining pieces using First Fit else Apply harmonic grouping scheme to create instance I ′ ; pack discards in O(log SIZE(I)) bins using First Fit Let x be optimal solution to conﬁguration LP for instance I ′ Pack ⌊xj ⌋ bins in conﬁguration Tj for j = 1, . . . , N ; call the packed pieces instance I1 Let I2 be remaining pieces from I ′ Pack I2 via BinPack(I2 ) Algorithm 4.3: Deterministic rounding algorithm for packing large pieces of the binpacking problem. of the groups G2 , . . . , Gr−1 ; each of these groups has size at least 2, and so there are at most SIZE(I)/2 of them. To prove the second property, suppose, for the moment, that each group Gi has at most one more piece than the previous group Gi−1 , i = 2, . . . , r − 1. Consequently, we discard at most one item from each of the groups G2 , . . . , Gr−1 . To bound the total size of the discarded pieces, the total size of each group is at most 3, and so the total size of G1 and Gr is at most 6. Furthermore, the size of the smallest piece in group Gi is at most 3/ni . Since we discard a piece from Gi , i = 2, . . . , r − 1, only when Gi is the ﬁrst group that has ∑ rni pieces and 3/j. Recall we discard its smallest piece, the total size of these discarded pieces is at most nj=1 1 1 from Section 1.6 that the kth harmonic number, Hk , is deﬁned to be Hk = 1+ 2 + 3 +· · ·+ k1 and that Hk = O(log k). Since each piece has size at least 1/ SIZE(I), we have that nr ≤ 3 SIZE(I), and so we see that the total size of the discarded pieces is O(log SIZE(I)). Now consider the general case in which each group need not be only one larger than the previous one. Consider any group Gi that contains more pieces than Gi−1 , i = 2, . . . , r − 1; more precisely, suppose that it contains k = ni − ni−1 more pieces. The k discarded pieces are of size at most 3(k/ni ) since they are the k smallest of ni pieces of total at most 3. We can upper bound this value by adding k terms, ∑ ieach of which is at least 3/ni ; that is, the total size of these discarded pieces is at most nj=n 3/j. Adding these terms together (for all i−1 +1 ∑ r groups), we get that the total size of the discarded pieces is at most nj=1 3/j, which we have already seen is O(log SIZE(I)). The harmonic grouping scheme can be used to design an approximation algorithm that always ﬁnds a packing with OPTLP (I) + O(log2 (SIZE(I))) bins. This algorithm uses the harmonic scheme recursively. The algorithm applies the grouping scheme, packs the discarded pieces using the FirstFit algorithm (or virtually any other simple algorithm), and solves the linear program for the rounded instance. The fractional variables of the LP are rounded down to the nearest integer to obtain a packing of a subset of the pieces. This leaves some pieces unpacked, and these are handled by a recursive call until the total size of the remaining pieces is less than a speciﬁed constant (say, for example, 10). The algorithm is summarized in Algorithm 4.3. We shall use I to denote the original instance, I ′ to denote the instance on which the ﬁrst linear program is solved, and I1 and I2 to denote the pieces packed based on the integer part of the fractional solution and those left over for the recursive call, respectively. The key to the analysis of this algorithm is the following lemma. Lemma 4.16: For any binpacking input I, from which the harmonic grouping scheme produces Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
100
Deterministic rounding of linear programs
an input I ′ , which is then partitioned into I1 and I2 , based on the integer and fractional parts of an optimal solution to the conﬁguration linear program, we have that OPTLP (I1 ) + OPTLP (I2 ) ≤ OPTLP (I ′ ) ≤ OPTLP (I).
(4.10)
Proof. The crucial property of the harmonic grouping scheme is that each piece in the input I ′ can be mapped to a distinct piece in the original input I of no lesser size. By inverting this mapping, any feasible solution for I can be interpreted as a feasible solution for I ′ (where each piece in I that is not the image of a piece in I ′ is simply deleted). Hence, the optimal value for the binpacking problem for I ′ is at most the optimal value for I. Similarly, each conﬁguration for I induces a corresponding conﬁguration for I ′ , and so we see that OPTLP (I ′ ) ≤ OPTLP (I). By deﬁnition, if x is an optimal solution to the conﬁguration LP for I ′ used to construct I1 and I2 , we have that ⌊xj ⌋, j = 1, . . . , N , is a feasible solution for the conﬁguration LP for I1 (which is even integer!), and xj − ⌊xj ⌋, j = 1, . . . , N , is a feasible solution for the conﬁguration LP for ∑ ′ I2 . Hence, the sum of the optimal values for these two LPs is at most N j=1 xj = OPTLP (I ). In fact, one can prove that OPTLP (I1 ) + OPTLP (I2 ) = OPTLP (I ′ ), but this is not needed. Each level of the recursion partitions the pieces into one of three sets: those pieces packed by the integer part of the LP solution, those pieces packed after having been discarded by the grouping scheme, and those pieces packed by a recursive call of the algorithm. (So, for the top level of the recursion, the ﬁrst corresponds to I1 and the last corresponds to I2 .) If one focuses on those pieces that fall in the ﬁrst category over all recursive calls of the algorithm, the inequality (4.10) implies that the sum of the LP values of these inputs is at most OPTLP (I). This implies that the only error introduced in each level of recursion is caused by the discarded pieces. We have bounded the total size of the pieces discarded in one level of recursion, and so we need to bound the number of levels of recursion. We will do this by showing that the total size of the input called in the recursive call is at most half the total size of the original input. The recursive input I2 is the leftover that corresponds to the fractional part of the optimal conﬁguration LP solution x; hence, we can bound its total size by the sum of the fractions ∑N x − ⌊x j ⌋. A very crude upper bound on this sum is to count the number of nonzeroes j=1 j in the optimal LP solution x. We claim that the number of nonzeroes in the optimal solution x can be bounded by the number of constraints in the conﬁguration LP. We leave this as an exercise to the reader (Exercise 4.5), though it follows directly from the properties of basic optimal solutions, which we discuss in Chapter 11 and Appendix A. The number of constraints is the number of distinct piece sizes in I ′ . By Lemma 4.15, this is at most SIZE(I)/2. By combining all of these arguments, we see that the total size of I2 , that is, SIZE(I2 ), is at most SIZE(I)/2. Hence, the size of the instance decreases by a factor of 2 in each level of recursion; the recursion terminates when the total size remaining is less than 10, and so there are O(log SIZE(I)) levels. In each of these levels, we use O(log SIZE(I)) bins to pack the discarded pieces; since SIZE(I) ≤ OPTLP (I) ≤ OPTBP (I), we obtain the following theorem. Theorem 4.17: The recursive application of the harmonic grouping scheme yields a polynomialtime approximation algorithm for the binpacking problem that uses OPTBP (I)+O(log2 (OPTBP (I))) bins. It is an important open question whether this result can be improved; it is possible that the binpacking problem can be approximated within an additive error of 1; however, showing that this is impossible would also be a striking advance. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.6
The binpacking problem
101
Exercises 4.1 The following problem arises in telecommunications networks, and is known as the SONET ring loading problem. The network consists of a cycle on n nodes, numbered 0 through n − 1 clockwise around the cycle. Some set C of calls is given; each call is a pair (i, j) originating at node i and destined to node j. The call can be routed either clockwise or counterclockwise around the ring. The objective is to route the calls so as to minimize the total load on the network. The load Li on link (i, i + 1(mod n)) is the number of calls routed through link (i, i + 1(mod n)), and the total load is max1≤i≤n Li . Give a 2approximation algorithm for the SONET ring loading problem. 4.2 Consider the scheduling problem of Section 4.2, but without release dates. That is, we must schedule jobs nonpreemptively on a single machine to minimize the weighted sum ∑n of completion times j=1 wj Cj . Suppose that jobs are indexed such that wp11 ≥ wp22 ≥ · · · ≥ wpnn . Then show it is optimal to schedule job 1 ﬁrst, job 2 second, and so on. This scheduling rule is called Smith’s rule. 4.3 Recall from Exercise 2.3 the concept of precedence constraints between jobs in a scheduling problem: We say i ≺ j if in any feasible schedule, job i must be completely processed before job j begins processing. Consider a variation of the singlemachine scheduling problem from Section 4.2 in which we have precedence constraints but no release dates. That is, we are given n jobs with processing times pj > 0 and weights wj ≥ 0, and the goal is to ﬁnd a nonpreemptive schedule on a single machine that is feasible with respect to the precedence constraints ≺ and that minimizes the weighted sum of completed times ∑ n j=1 wj Cj . Use the ideas of Section 4.2 to give a 2approximation algorithm for this problem. 4.4 In the algorithm for the binpacking problem, we used harmonic grouping in each iteration of the algorithm to create an instance I ′ from I. Consider the following grouping scheme: for i = 0, . . . , ⌈log2 SIZE(I)⌉, create a group Gi such that all pieces from I of size (2−(i+1) , 2−i ] are placed in group Gi . In each group Gi , create subgroups Gi,1 , Gi,2 , . . . , Gi,ki by arranging the pieces in Gi from largest to smallest and putting the 4 · 2i largest in the ﬁrst subgroup, the next 4 · 2i in the next subgroup, and so on. Now create instance I ′ from I in the following manner: for each i = 0, . . . , ⌈log2 SIZE(I)⌉, discard subgroups Gi,1 and Gi,ki (i.e., the ﬁrst and last subgroups of Gi ), and for j = 2, . . . , ki − 1 round each piece of subgroup Gi,j to the size of the largest piece in Gi,j . Prove that by using this grouping scheme within the binpacking algorithm, we obtain an algorithm that uses at most OPTBP (I) + O(log2 (OPTBP (I))) for all instances I. 4.5 Show that there exists an optimal solution x to the linear programming relaxation of the integer program (4.9) that has at most m nonzero entries, where m is the number of diﬀerent piece sizes. 4.6 Let G = (A, B, E) be a bipartite graph; that is, each edge (i, j) ∈ E has i ∈ A and j ∈ B. Assume that A ≤ B and that we are given nonnegative costs cij ≥ 0 for each edge (i, j) ∈ E. A complete matching of A is a subset of edges M ⊆ E such that each vertex in A has exactly one edge of M incident on it, and each vertex in B has at most one edge of M incident on it. We wish to ﬁnd a minimumcost complete matching. We can formulate an integer program for this problem in which we have an integer variable xij ∈ {0, 1} for Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
102
Deterministic rounding of linear programs
each edge (i, j) ∈ E, where xij = 1 if (i, j) is in the matching, and 0 otherwise. Then the integer program is as follows: ∑
minimize
cij xij
(i,j)∈E
∑
subject to
xij = 1,
∀i ∈ A,
xij ≤ 1,
∀j ∈ B,
xij ∈ {0, 1}
∀(i, j) ∈ E.
j∈B:(i,j)∈E
∑
i∈A:(i,j)∈E
Consider the linear programming relaxation of the integer program in which we replace the integer constraints xij ∈ {0, 1} with xij ≥ 0 for all (i, j) ∈ E. (a) Show that given any fractional solution to the linear programming relaxation, it is possible to ﬁnd in polynomial time an integer solution that costs no more than the fractional solution. (Hint: Given a set of fractional variables, ﬁnd a way to modify their values repeatedly such that the solution stays feasible, the overall cost does not increase, and at least one additional fractional variable becomes 0 or 1.) Conclude that there is a polynomialtime algorithm for ﬁnding a minimumcost complete matching. (b) Show that any extreme point of the linear programming relaxation has the property that xij ∈ {0, 1} for all (i, j) ∈ E. (Recall that an extreme point x is a feasible solution that cannot be expressed as λx1 + (1 − λ)x2 for 0 < λ < 1 and feasible solutions x1 and x2 distinct from x.) 4.7 This exercise introduces a deterministic rounding technique called pipage rounding, which builds on ideas similar to those used in Exercise 4.6. To illustrate this technique, we will consider the problem of ﬁnding a maximum cut in a graph with a constraint on the size of each part. In the maximum cut problem, we are given as input an undirected graph G = (V, E) with nonnegative weights wij ≥ 0 for all (i, j) ∈ E. We wish to partition the vertex set into two parts U and W = V − U so as to maximize the weight of the edges whose two endpoints are in diﬀerent parts. We will also assume that we are given an integer k ≤ V /2, and we must ﬁnd a partition such that U  = k (we will consider the maximum cut problem without this constraint in Sections 5.1 and 6.2). (a) Show that the following nonlinear integer program models the maximum cut problem with a constraint on the size of the parts: maximize
∑ (i,j)∈E
subject to
wij (xi + xj − 2xi xj ) ∑
xi = k,
i∈V
xi ∈ {0, 1} ,
∀i ∈ V.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
4.6
The binpacking problem
103
(b) Show that the following linear program is a relaxation of the problem: ∑ maximize wij zij (i,j)∈E
zij ≤ xi + xj ,
∀(i, j) ∈ E,
zij ≤ 2 − xi − xj , ∑ xi = k,
∀(i, j) ∈ E,
subject to
i∈V
0 ≤ zij ≤ 1,
∀(i, j) ∈ E,
0 ≤ xi ≤ 1,
∀i ∈ V.
∑ (c) Let F (x) = (i,j)∈E wij (xi +xj −2xi xj ) be the objective function from the nonlinear integer program. Show that for any (x, z) that is a feasible solution to the linear 1∑ programming relaxation, F (x) ≥ 2 (i,j)∈E wij zij . (d) Argue that given a fractional solution x, for two fractional variables xi and xj , it is possible to increase one by ϵ > 0 and decrease the other by ϵ such that F (x) does not decrease and one of the two variables becomes integer. (e) Use the arguments above to devise a 12 approximation algorithm for the maximum cut problem with a constraint on the size of the parts.
Chapter Notes Early deterministic LP rounding algorithms include the set cover algorithm given in Section 1.3 due to Hochbaum [160] and the binpacking algorithm of Section 4.6 due to Karmarkar and Karp [187]. Both of these papers appeared in 1982. Work on deterministic rounding approximation algorithms did not precede this date by much because the ﬁrst polynomialtime algorithms for solving linear programs did not appear until the late 1970s with the publication of the ellipsoid method. As discussed in the introduction to the chapter, the easiest way to round a fractional solution is to round some variables up to 1 and others down to 0. This is the case for the prizecollecting Steiner tree problem introduced in Section 4.4. The prizecollecting Steiner tree problem is a variant of a problem introduced by Balas [31]. This version of the problem and the algorithm of this section are due to Bienstock, Goemans, SimchiLevi, and Williamson [48]. The ellipsoid method discussed in Section 4.3 as a polynomialtime algorithm for solving linear programs was ﬁrst given by Khachiyan [189], building on earlier work for nonlinear programs by Shor [267]. The algorithm that uses a separation oracle for solving linear programs with an exponential number of constraints is due to Gr¨otschel, Lov´asz, and Schrijver [144]. An extended treatment of the topic can be found in the book of Gr¨otschel, Lov´asz, and Schrijver [145]; a surveylevel treatment can be found in Bland, Goldfarb, and Todd [50]. At a high level, the ellipsoid method works as follows. Suppose we are trying to solve the linear program (4.4) from Section 4.3. Initially, the algorithm ﬁnds an ellipsoid in ℜn containing all basic feasible solutions for the linear program (see Chapter 11 or Appendix A for discussion of basic feasible and basic optimal solutions). Let x ˜ be the center of the ellipsoid. The algorithm calls the separation oracle with x ˜ . If x ˜ is feasible, it creates a constraint ∑n ∑n ˜j , since a basic optimal solution must have objective function value no j=1 dj x j=1 dj xj ≤ greater than the feasible solution x ˜ (this constraint is sometimes called an objective function Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
104
Deterministic rounding of linear programs
∑ cut). If x ˜ is not feasible, the separation oracle returns a constraint nj=1 aij xj ≥ bi that is violated by x ˜. In either case, we have a hyperplane through x ˜ such that a basic optimal solution to the linear program (if one exists) must lie on one side of the hyperplane; in the case ∑n ∑n of a feasible x ˜ the∑hyperplane is ˜j , and in the case of an infeasible x ˜ ∑ j=1 dj xj ≤ j=1 dj x the hyperplane is nj=1 aij xj ≥ nj=1 aij x ˜j . The hyperplane containing x ˜ splits the ellipsoid in two. The algorithm then ﬁnds a new ellipsoid containing the appropriate half of the original ellipsoid, and then considers the center of the new ellipsoid. This process repeats until the ellipsoid is suﬃciently small that it can contain at most one basic feasible solution, if any; this then must be a basic optimal solution if one exists. The key to the proof of the running time of the algorithm is to show that after O(n) iterations, the volume of the ellipsoid has dropped by a constant factor; then by relating the size of the initial to the ﬁnal ellipsoid, the polynomial bound on the running time can be obtained. As we saw in the chapter, sometimes rounding algorithms are more sophisticated than simply choosing large fractional variables and rounding up. The algorithm for the unweighted singlemachine scheduling problem in Section 4.1 is due to Phillips, Stein, and Wein [241]. The algorithm for the weighted case in Section 4.2 is due to Hall, Schulz, Shmoys, and Wein [155]; the linear programming formulation used in the section was developed by Wolsey [292] and Queyranne [243], and the separation oracle for the linear program in Section 4.3 is due to Queyranne [243]. The algorithm for the uncapacitated facility location problem in Section 4.5 is due to Chudak and Shmoys [77], building on earlier work of Shmoys, Tardos, and Aardal [264]. The hardness result of Theorem 4.14 is due to Guha and Khuller [146]. The pipage rounding technique in Exercise 4.7 for the maximum cut problem with size constraints is due to Ageev and Sviridenko [3, 2]. Smith’s rule in Exercise 4.2 is due to Smith [270]. The result of Exercise 4.3 is due to Hall, Schulz, Shmoys, and Wein [155]. Exercise 4.4 is due to Karmarkar and Karp [187]. Exercise 4.6 is essentially due to Birkhoﬀ [49].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 5
Random sampling and randomized rounding of linear programs
Sometimes it turns out to be useful to allow our algorithms to make random choices; that is, the algorithm can ﬂip a coin, or ﬂip a biased coin, or draw a value uniformly from a given interval. The performance guarantee of an approximation algorithm that makes random choices is then the expected value of the solution produced relative to the value of an optimal solution, where the expectation is taken over the random choices of the algorithm. At ﬁrst this might seem like a weaker class of algorithm. In what sense is there a performance guarantee if it only holds in expectation? However, in most cases we will be able to show that randomized approximation algorithms can be derandomized: that is, we can use a certain algorithmic technique known as the method of conditional expectations to produce a deterministic version of the algorithm that has the same performance guarantee as the randomized version. Of what use then is randomization? It turns out that it is often much simpler to state and analyze the randomized version of the algorithm than to state and analyze the deterministic version that results from derandomization. Thus randomization gains us simplicity in our algorithm design and analysis, while derandomization ensures that the performance guarantee can be obtained deterministically. In a few cases, it is easy to state the deterministic, derandomized version of an algorithm, but we only know how to analyze the randomized version. Here the randomized algorithm allows us to analyze an algorithm that we are unable to analyze otherwise. We will see an example of this when we revisit the prizecollecting Steiner tree problem in Section 5.7. It is also sometimes the case that we can prove that the performance guarantee of a randomized approximation algorithm holds with high probability. By this we mean that the probability that the performance guarantee does not hold is one over some polynomial in the input size of the problem. Usually we can make this polynomial as large as we want (and thus the probability as small as we want) by weakening the performance guarantee by some constant factor. Here derandomization is less necessary, though sometimes still possible by using more sophisticated techniques. We begin the chapter by looking at very simple randomized algorithms for two problems, the maximum satisﬁability problem and the maximum cut problem. Here we show that sampling a solution uniformly at random from the set of all possible solutions gives a good randomized approximation algorithm. For the maximum satisﬁability problem, we are able to go still further 105
106
Random sampling and randomized rounding of linear programs
and show that biasing our choice yields a better performance guarantee. We then revisit the idea of using randomized rounding of linear programming relaxations introduced in Section 1.7, and show that it leads to still better approximation algorithms for the maximum satisﬁability problem, as well as better algorithms for other problems we have seen previously, such as the prizecollecting Steiner tree problem, the uncapacitated facility location problem, and a single machine scheduling problem. We then give Chernoﬀ bounds, which allow us to bound the probability that a sum of random variables is far away from its expected value. We show how these bounds can be applied to an integer multicommodity ﬂow problem, which is historically the ﬁrst use of randomized rounding. We end with a much more sophisticated use of drawing a random sample, and show that this technique can be used to 3color certain kinds of dense 3colorable graphs with high probability.
5.1
Simple algorithms for MAX SAT and MAX CUT
Two problems will play an especially prominent role in our discussion of randomization in the design and analysis of approximation algorithms: the maximum satisﬁability problem, and the maximum cut problem. The former will be highlighted in this chapter, whereas the central developments for the latter will be deferred to the next chapter. However, in this section we will give a simple randomized 12 approximation algorithm for each problem. In the maximum satisﬁability problem (often abbreviated as MAX SAT), the input consists of n Boolean variables x1 , . . . , xn (each of which may be set to either true or false), m clauses C1 , . . . , Cm (each of which consists of a disjunction (that is, an “or”) of some number of the variables and their negations – for example, x3 ∨ x ¯5 ∨ x11 , where x ¯i is the negation of xi ), and a nonnegative weight wj for each clause Cj . The objective of the problem is to ﬁnd an assignment of true/false to the xi that maximizes the weight of the satisﬁed clauses. A clause is said to be satisﬁed if one of the unnegated variables is set to true, or one of the negated variables is set to false. For example, in the clause x3 ∨ x ¯5 ∨ x11 , the clause is not satisﬁed only if x3 is set to false, x5 to true, and x11 to false. Some terminology will be useful in discussing the MAX SAT problem. We say that a variable xi or a negated variable x ¯i is a literal, so that each clause consists of some number of literals. A variable xi is called a positive literal and a negated variable x ¯i is called a negative literal. The number of literals in a clause is called its size or length. We will denote the length of a clause Cj by lj . Clauses of length one are sometimes called unit clauses. Without loss of generality, we assume that no literal is repeated in a clause (since this does not aﬀect the satisﬁability of the instance), and that at most one of xi and x ¯i appears in a clause (since if both xi and x ¯i are in a clause, it is trivially satisﬁable). Finally, it is natural to assume that the clauses are distinct, since we can simply sum the weights of two identical clauses. A very straightforward use of randomization for MAX SAT is to set each xi to true independently with probability 1/2. An alternate perspective on this algorithm is that we choose a setting of the variables uniformly at random from the space of all possible settings. It turns out that this gives a reasonable approximation algorithm for this problem. Theorem 5.1: Setting each xi to true with probability 1/2 independently gives a randomized 1 2 approximation algorithm for the maximum satisﬁability problem. Proof. Consider a random variable Yj such that Yj is 1 if clause j is satisﬁed and 0 otherwise. Let W∑be a random variable that is equal to the total weight of the satisﬁed clauses, so that W = m j=1 wj Yj . Let OPT denote the optimum value of the MAX SAT instance. Then, by Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.1
Simple algorithms for MAX SAT and MAX CUT
107
linearity of expectation, and the deﬁnition of the expectation of a 01 random variable, we know that m m ∑ ∑ E[W ] = wj E[Yj ] = wj Pr[clause Cj satisﬁed]. j=1
j=1
For each clause Cj , j = 1, . . . , n, the probability that it is not satisﬁed is the probability that each positive literal in Cj is set to false and each negative literal in Cj is set to true, each of which happens with probability 1/2 independently; hence ( Pr[clause Cj satisﬁed] =
( )l j ) 1 1 1− ≥ , 2 2
where the last inequality is a consequence of the fact that lj ≥ 1. Hence, 1∑ 1 E[W ] ≥ wj ≥ OPT, 2 2 m
j=1
where the last inequality follows from the fact that the total weight is an easy upper bound on the optimal value, since each weight is assumed to be nonnegative. that if lj ≥ k for each clause j, then the analysis above shows that the algorithm is a ( Observe ( 1 )k ) 1− 2 approximation algorithm for such instances. Thus the performance of the algorithm is better on MAX SAT instances consisting of long clauses. This observation will be useful to us later on. Although this seems like a pretty naive algorithm, a hardness theorem shows that this is the best that can be done in some cases. Consider the case in which lj = 3 for all clauses j; this restriction of the problem is sometimes called MAX E3SAT, since there are exactly 3 literals in each clause. The analysis above shows that randomized algorithm gives an approximation ( the ( 1 )3 ) = 78 . A truly remarkable result shows that algorithm with performance guarantee 1 − 2 nothing better is possible for these instances unless P = NP. Theorem 5.2: If there is an ( 78 +ϵ)approximation algorithm for MAX E3SAT for any constant ϵ > 0, then P = NP. We discuss this result further in Section 16.3. In the maximum cut problem (sometimes abbreviated MAX CUT), the input is an undirected graph G = (V, E), along with a nonnegative weight wij ≥ 0 for each edge (i, j) ∈ E. The goal is to partition the vertex set into two parts, U and W = V − U , so as to maximize the weight of the edges whose two endpoints are in diﬀerent parts, one in U and one in W . We say that an edge with endpoints in both U and W is in the cut. In the case wij = 1 for each edge (i, j) ∈ E, we have an unweighted MAX CUT problem. It is easy to give a 12 approximation algorithm for the MAX CUT problem along the same lines as the previous randomized algorithm for MAX SAT. Here we place each vertex v ∈ V into U independently with probability 1/2. As with the MAX SAT algorithm, this can be viewed as sampling a solution uniformly from the space of all possible solutions. Theorem 5.3: If we place each vertex v ∈ V into U independently with probability 1/2, then we obtain a randomized 12 approximation algorithm for the maximum cut problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
108
Random sampling and randomized rounding of linear programs
Proof. Consider a random variable Xij that is 1 if the edge (i, j) is in the cut, and 0 otherwise. Let ∑ Z be the random variable equal to the total weight of edges in the cut, so that Z = (i,j)∈E wij Xij . Let OPT denote the optimal value of the maximum cut instance. Then, as before, by linearity of expectation and the deﬁnition of expectation of a 01 random variable, we get that ∑ ∑ E[Z] = wij E[Xij ] = wij Pr[Edge (i, j) in cut]. (i,j)∈E
(i,j)∈E
In this case, the probability that a speciﬁc edge (i, j) is in the cut is easy to calculate: since the two endpoints are placed in the sets independently, they are in diﬀerent sets with probability equal to 21 . Hence, 1 ∑ 1 E[Z] = wij ≥ OPT, 2 2 (i,j)∈E
where the inequality follows directly from the fact that the sum of the (nonnegative) weights of all edges is obviously an upper bound on the weight of the edges in an optimal cut. We will show in Section 6.2 that by using more sophisticated techniques we can get a substantially better performance guarantee for the MAX CUT problem.
5.2
Derandomization
As we mentioned in the introduction to the chapter, it is often possible to derandomize a randomized algorithm; that is, to obtain a deterministic algorithm whose solution value is as good as the expected value of the randomized algorithm. To illustrate, we will show how the algorithm of the preceding section for the maximum satisﬁability problem can be derandomized by replacing the randomized decision of whether to set xi to true with a deterministic one that will preserve the expected value of the solution. These decisions will be made sequentially: the value of x1 is determined ﬁrst, then x2 , and so on. How should x1 be set so as to preserve the expected value of the algorithm? Assume for the moment that we will make the choice of x1 deterministically, and all other variables will be set true with probability 1/2 as before. Then the best way to set x1 is that which will maximize the expected value of the resulting solution; that is, we should determine the expected value of W , the weight of satisﬁed clauses, given that x1 is set to true, and the expected weight of W given that x1 is set to false, and set x1 to whichever value maximizes the expected value of W . It makes intuitive sense that this should work, since the maximum is always greater than an average, and the expected value of W is the average of its expected value given the two possible settings of x1 . In this way, we maintain an algorithmic invariant that the expected value is at least half the optimum, while having fewer random variables left. More formally, if E[W x1 ← true] ≥ E[W x1 ← false], then we set x1 true, otherwise we set it to false. Since by the deﬁnition of conditional expectations, E[W ] = E[W x1 ← true] Pr[x1 ← true] + E[W x1 ← false] Pr[x1 ← false] 1 = (E[W x1 ← true] + E[W x1 ← false]) , 2 if we set x1 to truth value b1 so as to maximize the conditional expectation, then E[W x1 ← b1 ] ≥ E[W ]; that is, the deterministic choice of how to set x1 guarantees an expected value no less than the expected value of the completely randomized algorithm. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.2
Derandomization
109
Assuming for the moment that we can compute these conditional expectations, the deterministic decision of how to set the remaining variables is similar. Assume that we have set variables x1 , . . . , xi to truth values b1 , . . . , bi respectively. How shall we set variable xi+1 ? Again, assume that the remaining variables are set randomly. Then the best way to set xi+1 is so as to maximize the expected value given the previous settings of x1 , . . . , xi . So if E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← true] ≥ E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← false] we set xi+1 to true (thus setting bi+1 to true), otherwise we set xi+1 to false (thus setting bi+1 to false). Then since E[W x1 ← b1 , . . . , xi ← bi ] = E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← true] Pr[xi+1 ← true] + E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← false] Pr[xi+1 ← false] 1 = (E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← true] + E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← false]) , 2 setting xi+1 to truth value bi+1 as described above ensures that E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← bi+1 ] ≥ E[W x1 ← b1 , . . . , xi ← bi ]. By induction, this implies that E[W x1 ← b1 , . . . , xi ← bi , xi+1 ← bi+1 ] ≥ E[W ]. We continue this process until all n variables have been set. Then since the conditional expectation given the setting of all n variables, E[W x1 ← b1 , . . . , xn ← bn ], is simply the value of the solution given by the deterministic algorithm, we know that the value of the solution returned is at least E[W ] ≥ 21 OPT . Therefore, the algorithm is a 12 approximation algorithm. These conditional expectations are not diﬃcult to compute. By deﬁnition, m ∑ E[W x1 ← b1 , . . . , xi ← bi ] = wj E[Yj x1 ← b1 , . . . , xi ← bi ] =
j=1 m ∑
wj Pr[clause Cj satisﬁedx1 ← b1 , . . . , xi ← bi ].
j=1
Furthermore, the probability that clause Cj is satisﬁed given that x1 ← b1 , . . . , xi ← bi is easily seen to be 1 if the settings of x1 , . . . , xi already satisfy the clause, and is 1 − (1/2)k otherwise, where k is the number of literals in the clause that remain unset by this procedure. For example, consider the clause x3 ∨ x ¯5 ∨ x ¯7 . It is the case that Pr[clause satisﬁedx1 ← true, x2 ← false, x3 ← true] = 1, since setting x3 to true satisﬁes the clause. On the other hand, Pr[clause satisﬁedx1 ← true, x2 ← false, x3 ← false] = 1 −
( )2 3 1 = , 2 4
since the clause will be unsatisﬁed only if x5 and x7 are set true, an event that occurs with probability 1/4. This technique for derandomizing algorithms works with a wide variety of randomized algorithms in which variables are set independently and the conditional expectations are polynomialtime computable. It is sometimes called the method of conditional expectations, due to its use of conditional expectations. In particular, an almost identical argument leads to a derandomized version of the randomized 12 approximation algorithm for the MAX CUT problem. Most of the randomized algorithms we discuss in this chapter can be derandomized via this method. The randomized versions of the algorithms are easier to present and analyze, and so we will frequently not discuss their deterministic variants. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
110
Random sampling and randomized rounding of linear programs
5.3
Flipping biased coins
How might we improve the randomized algorithm for MAX SAT? We will show here that biasing the probability with which we set xi is actually helpful; that is, we will set xi true with some probability not equal to 1/2. To do this, it is easiest to start by considering only MAX SAT instances with no unit clauses x ¯i , that is, no negated unit clauses. We will later show that we can remove this assumption. Suppose now we set each xi to be true independently with probability p > 1/2. As in the analysis of the previous randomized algorithm, we will need to analyze the probability that any given clause is satisﬁed. Lemma 5.4: If each xi is set to true with probability p > 1/2 independently, then the probability that any given clause is satisﬁed is at least min(p, 1−p2 ) for MAX SAT instances with no negated unit clauses. Proof. If the clause is a unit clause, then the probability the clause is satisﬁed is p, since it must be of the form xi , and the probability xi is set true is p. If the clause has length at least two, then the probability that the clause is satisﬁed is 1 − pa (1 − p)b , where a is the number of negated variables in the clause and b is the number of unnegated variables in the clause, so that a + b = lj ≥ 2. Since p > 21 > 1 − p, this probability is at least 1 − pa+b = 1 − plj ≥ 1 − p2 , and the lemma is proved. We can obtain the best performance guarantee by setting p = 1 − p2 . This yields p = √ 1 2 ( 5 − 1) ≈ .618. The lemma immediately implies the following theorem. Theorem 5.5: Setting each xi to true with probability p independently gives a randomized min(p, 1 − p2 )approximation algorithm for MAX SAT instances with no negated unit clauses. Proof. This follows since E[W ] =
m ∑
wj Pr[clause Cj satisﬁed] ≥ min(p, 1 − p2 )
j=1
m ∑
wj ≥ min(p, 1 − p2 ) OPT .
j=1
We would like to extend ∑ this result to all MAX SAT instances. To do this, we will use a better bound on OPT than m j=1 wj . Assume that for every i the weight of the unit clause xi appearing in the instance is at least the weight of the unit clause x ¯i ; this is without loss of generality since we could negate all occurrences of xi if the assumption is not true. Let vi be the weight of the unit clause x ¯i if it exists in the instance, and let vi be zero otherwise. ∑ ∑m Lemma 5.6: OPT ≤ j=1 wj − ni=1 vi . Proof. For each i, the optimal solution can satisfy exactly one of xi and x ¯i . Thus the weight of the optimal solution cannot include both the weight of the clause xi and the clause x ¯i . Since vi is the smaller of these two weights, the lemma follows. We can now extend the result.
√ Theorem 5.7: We can obtain a randomized 12 ( 5−1)approximation algorithm for MAX SAT. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.4
Randomized rounding
111
Proof. Let U be the set of indices of clauses of the instance excluding unit clauses of the form x ¯i . As above, we assume without loss of ∑ generality that ¯i is no ∑ the weight ∑n of each clause x greater than the weight of clause xi . Thus j∈U wj = m w − v . Then set each xi to j=1 j i=1 i √ 1 be true independently with probability p = 2 ( 5 − 1). Then E[W ] =
m ∑
wj Pr[clause Cj satisﬁed]
j=1
≥
∑
wj Pr[clause Cj satisﬁed]
j∈U
≥ p·
∑
wj
(5.1)
j∈U
m n ∑ ∑ wj − vi ≥ p · OPT, = p· j=1
i=1
where (5.1) follows by Theorem 5.5 and the fact that p = min(p, 1 − p2 ). This algorithm can be derandomized using the method of conditional expectations.
5.4
Randomized rounding
The algorithm of the previous section shows that biasing the probability with which we set xi true yields an improved approximation algorithm. However, we gave each variable the same bias. In this section, we show that we can do still better by giving each variable its own bias. We do this by returning to the idea of randomized rounding, which we examined brieﬂy in Section 1.7 in the context of the set cover problem. Recall that in randomized rounding, we ﬁrst set up an integer programming formulation of the problem at hand in which there are 01 integer variables. In this case we will create an integer program with a 01 variable yi for each Boolean variable xi such that yi = 1 corresponds to xi set true. The integer program is relaxed to a linear program by replacing the constraints yi ∈ {0, 1} with 0 ≤ yi ≤ 1, and the linear programming relaxation is solved in polynomial time. Recall that the central idea of randomized rounding is that the fractional value yi∗ is interpreted as the probability that yi should be set to 1. In this case, we set each xi to true with probability yi∗ independently. We now give an integer programming formulation of the MAX SAT problem. In addition to the variables yi , we introduce a variable zj for each clause Cj that will be 1 if the clause is satisﬁed and 0 otherwise. For each clause Cj let Pj be the indices of the variables xi that occur positively in the clause, and let Nj be the indices of the variables xi that are negated in the clause. We denote the clause Cj by ∨
xi ∨
i∈Pj
Then the inequality
∑ i∈Pj
yi +
∨
x ¯i .
i∈Nj
∑
(1 − yi ) ≥ zj
i∈Nj
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
112
Random sampling and randomized rounding of linear programs
must hold for clause Cj since if each variable that occurs positively in the clause is set to false (and its corresponding yi is set to 0) and each variable that occurs negatively is set to true (and its corresponding yi is set to 1), then the clause is not satisﬁed, and zj must be 0. This inequality yields the following integer programming formulation of the MAX SAT problem: m ∑
maximize subject to
∑
wj zj
j=1
yi +
i∈Pj
∑
(1 − yi ) ≥ zj ,
∨
∀Cj =
i∈Nj
xi ∨
i∈Pj
yi ∈ {0, 1} , 0 ≤ zj ≤ 1,
∨
x ¯i ,
i∈Nj
i = 1, . . . , n, j = 1, . . . , m.
∗ is the optimal value of this integer program, then it is not hard to see that Z ∗ = OPT. If ZIP IP
The corresponding linear programming relaxation of this integer program is m ∑
maximize subject to
∑ i∈Pj
wj zj
j=1
yi +
∑
(1 − yi ) ≥ zj ,
∀Cj =
i∈Nj
∨
xi ∨
i∈Pj
0 ≤ yi ≤ 1,
i = 1, . . . , n,
0 ≤ zj ≤ 1,
j = 1, . . . , m.
∨
x ¯i ,
i∈Nj
∗ is the optimal value of this linear program, then clearly Z ∗ ≥ Z ∗ = OPT . If ZLP LP IP Let (y ∗ , z ∗ ) be an optimal solution to the linear programming relaxation. We now consider the result of using randomized rounding, and setting xi to true with probability yi∗ independently. Before we can begin the analysis, we will need two facts. The ﬁrst is commonly called the arithmeticgeometric mean inequality, because it compares the arithmetic and geometric means of a set of numbers.
Fact 5.8 (Arithmeticgeometric mean inequality): For any nonnegative a1 , . . . , ak , (
k ∏ i=1
)1/k ai
k 1∑ ≤ ai . k i=1
Fact 5.9: If a function f (x) is concave on the interval [0, 1] (that is, f ′′ (x) ≤ 0 on [0, 1]), and f (0) = a and f (1) = b + a, then f (x) ≥ bx + a for x ∈ [0, 1] (see Figure 5.1). Theorem 5.10: Randomized rounding gives a randomized (1 − 1e )approximation algorithm for MAX SAT. Proof. As in the analyses of the algorithms in the previous sections, the main diﬃculty is analyzing the probability that a given clause Cj is satisﬁed. Pick an arbitrary clause Cj . Then, by applying the arithmeticgeometric mean inequality, we see that lj ∏ ∏ ∑ ∑ 1 Pr[clause Cj not satisﬁed] = (1 − yi∗ ) yi∗ ≤ (1 − yi∗ ) + yi∗ . lj i∈Pj
i∈Nj
i∈Pj
i∈Nj
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.4
Randomized rounding
a+b
a
0
1
Figure 5.1: An illustration of Fact 5.9.
By rearranging terms, we can derive that lj lj ∑ ∑ ∑ ∑ 1 1 yi∗ = 1 − (1 − yi∗ ) . (1 − yi∗ ) + yi∗ + lj lj
i∈Nj
i∈Pj
i∈Pj
i∈Nj
By invoking the corresponding inequality from the linear program, ∑ i∈Pj
yi∗ +
∑
(1 − yi∗ ) ≥ zj∗ ,
i∈Nj
we see that ( Pr[clause Cj not satisﬁed] ≤ ( The function f (zj∗ ) = 1 − 1 −
zj∗ lj
)lj
zj∗ 1− lj
)l j .
is concave for lj ≥ 1. Then by using Fact 5.9,
( ) zj∗ lj Pr[clause Cj satisﬁed] ≥ 1 − 1 − lj [ ( ) ] 1 lj ∗ ≥ 1− 1− zj . lj Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
113
114
Random sampling and randomized rounding of linear programs
Therefore the expected value of the randomized rounding algorithm is E[W ] =
m ∑
wj Pr[clause Cj satisﬁed]
j=1
)lj ] 1 ≥ wj zj∗ 1 − 1 − lj j=1 [ ] ) ( m 1 k ∑ ≥ min 1 − 1 − wj zj∗ . k≥1 k m ∑
[
(
j=1
[ ( )k ] ) ( is a nonincreasing function in k and that it approaches 1 − 1e from Note that 1 − 1 − k1 ∑ ∗ ∗ above as k tends to inﬁnity. Since m j=1 wj zj = ZLP ≥ OPT, we have that [
(
1 E[W ] ≥ min 1 − 1 − k≥1 k
)k ] ∑ m
wj zj∗
( ≥
j=1
1 1− e
) OPT .
This randomized rounding algorithm can be derandomized in the standard way using the method of conditional expectations.
5.5
Choosing the better of two solutions
In this section we observe that choosing the best solution from the two given by the randomized rounding algorithm of the previous section and the unbiased randomized algorithm of the ﬁrst section gives a better performance guarantee than that of either algorithm. This happens because, as we shall see, the algorithms have contrasting bad cases: when one algorithm is far from optimal, the other is close, and vice versa. This technique can be useful in other situations, and does not require using randomized algorithms. In this case, consider a given clause Cj of length lj . [ The randomized] rounding algorithm of ( )l 1 j Section 5.4 satisﬁes the clause with probability at least 1 − 1 − lj zj∗ , while the unbiased randomized algorithm of Section 5.1 satisﬁes the clause with probability 1 − 2−lj ≥ (1 − 2−lj )zj∗ . Thus when the clause is short, it is very likely to be satisﬁed by the randomized rounding algorithm, though not by the unbiased randomized algorithm, and when the clause is long the opposite is true. This observation is made precise and rigorous in the following theorem. Theorem 5.11: Choosing the better of the two solutions given by the randomized rounding algorithm and the unbiased randomized algorithm yields a randomized 34 approximation algorithm for MAX SAT. Proof. Let W1 be a random variable denoting the value of the solution returned by the randomized rounding algorithm, and let W2 be a random variable denoting the value of the solution returned by the unbiased randomized algorithm. Then we wish to show that E[max(W1 , W2 )] ≥
3 OPT . 4
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.5
Choosing the better of two solutions
115
1 randomized rounding flipping coins average
0.9
0.8
0.7
0.6
0.5 1
2
3
4
5
6
Figure 5.2: Illustration of the proof of Theorem 5.11. The “randomized rounding” line is the function 1 − (1 − k1 )k . The “ﬂipping coins” line is the function 1 − 2−k . The “average” line is the average of these two functions, which is at least 43 for all integers k ≥ 1. To obtain this inequality, observe that ] [ 1 1 E[max(W1 , W2 )] ≥ E W1 + W2 2 2 1 1 = E[W1 ] + E[W2 ] 2 2 [ ( ) ] m m ) ∑ 1 1 lj 1∑ ( ∗ ≥ wj zj 1 − 1 − wj 1 − 2−lj + 2 lj 2 j=1 j=1 [ ( ) ] ( ) m l ( ) j ∑ 1 1 1 ≥ wj zj∗ 1− 1− + 1 − 2−lj . 2 lj 2 j=1
We claim that
[ ( ] ( ) ) ) 1 1 lj 3 1( 1− 1− 1 − 2−lj ≥ + 2 lj 2 4
for all positive integers lj . We will prove this shortly, but this can be seen in Figure 5.2. Given the claim, we have that 3 ∗ 3 3∑ wj zj∗ = ZLP ≥ OPT . 4 4 4 m
E[max(W1 , W2 )] ≥
j=1
Now to prove the claim. Observe that the claim holds for lj = 1, since 1 1 3 1 ·1+ · = , 2 2 2 4 Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
116
Random sampling and randomized rounding of linear programs
and the claim holds for lj = 2, since ( ( )2 ) 1 1 3 1 · 1− + (1 − 2−2 ) = . 2 2 2 4 ( For all lj ≥ 3,
(
1− 1−
1 lj
) lj )
≥1− 1 2
(
1 e
1 1− e
( ) and 1 − 2−lj ≥ 78 , and ) +
1 7 3 · ≈ .753 ≥ , 2 8 4
so the claim is proven. Notice that taking the best solution of the two derandomized algorithms gives at least max(E[W1 ], E[W2 ]) ≥ 12 E[W1 ] + 21 E[W2 ]. The proof above shows that this quantity is at least 3 4 OPT. Thus taking the best solution of the two derandomized algorithms is a deterministic 3 4 approximation algorithm.
5.6
Nonlinear randomized rounding
Thus far in our applications of randomized rounding, we have used the variable yi∗ from the linear programming relaxation as a probability to decide whether to set yi to 1 in the integer programming formulation of the problem. In the case of the MAX SAT problem, we set xi to true with probability yi∗ . There is no reason, however, that we cannot use some function f : [0, 1] → [0, 1] to set xi to true with probability f (yi∗ ). Sometimes this yields approximation algorithms with better performance guarantees than using the identity function, as we will see in this section. In this section we will show that a 34 approximation algorithm for MAX SAT can be obtained directly by using randomized rounding with a nonlinear function f . In fact, there is considerable freedom in choosing such a function f : let f be any function such that f : [0, 1] → [0, 1] and 1 − 4−x ≤ f (x) ≤ 4x−1 .
(5.2)
See Figure 5.3 for a plot of the bounding functions. We will see that this ensures that the ∗ probability a clause Cj is satisﬁed is at least 1−4−zj ≥ 34 zj∗ , which will give the 34 approximation algorithm. Theorem 5.12: Randomized rounding with the function f is a randomized algorithm for MAX SAT.
3 4 approximation
Proof. Once again, we only need analyze the probability that a given clause Cj is satisﬁed. By the deﬁnition of f , ∏ ∏ ∏ ∏ ∗ ∗ Pr[clause Cj not satisﬁed] = (1 − f (yi∗ )) f (yi∗ ) ≤ 4−yi 4yi −1 . i∈Pj
i∈Nj
i∈Pj
i∈Nj
Rewriting the product and using the linear programming constraint for clause Cj gives us Pr[clause Cj not satisﬁed] ≤
∏ i∈Pj
−yi∗
4
∏
4
yi∗ −1
=4
(∑ ) ∑ ∗ ∗ − i∈P yi + i∈N (1−yi ) j
j
∗
≤ 4−zj .
i∈Nj
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.6
Nonlinear randomized rounding
117
1
0.5
0 0
0.5
1
Figure 5.3: A plot of the functions of (5.2). The upper curve is 4x−1 and the lower curve is 1 − 4−x . Then using Fact 5.9 and observing that the function g(z) = 1 − 4−z is concave on [0,1], we have ∗ 3 Pr[clause Cj satisﬁed] ≥ 1 − 4−zj ≥ (1 − 4−1 )zj∗ = zj∗ . 4 It follows that the expected performance of the algorithm is
E[W ] =
m ∑ j=1
3 3∑ wj zj∗ ≥ OPT . 4 4 m
wj Pr[clause Cj satisﬁed] ≥
j=1
Once again, the algorithm can be derandomized using the method of conditional expectations. There are other choices of the function f that also lead to a 34 approximation algorithm for MAX SAT. Some other possibilities are presented in the exercises at the end of the chapter. Is it possible to get an algorithm with a performance guarantee better than 34 by using some more complicated form of randomized rounding? It turns out that the answer is no, at least for any algorithm that derives its performance guarantee by comparing its value to that of the linear programming relaxation. To see this, consider the instance of the maximum satisﬁability problem with two variables x1 and x2 and four clauses of weight one each, x1 ∨ x2 , x1 ∨ x ¯2 , x ¯1 ∨ x2 , and x ¯1 ∨ x ¯2 . Any feasible solution, including the optimal solution, satisﬁes exactly three of the four clauses. However, if we set y1 = y2 = 21 and zj = 1 for all four clauses Cj , then this solution is feasible for the linear program and 4. Thus the value to any solution to ∑ has value ∗ . We say that this integer programming the MAX SAT instance can be at most 34 m w z j=1 j j formulation of the maximum satisﬁability problem has an integrality gap of 34 . Deﬁnition 5.13: The integrality gap of an integer program is the worstcase ratio over all instances of the problem of value of an optimal solution to the integer programming formulation to value of an optimal solution to its linear programming relaxation. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
118
Random sampling and randomized rounding of linear programs
The example above shows that the integrality gap is at most 34 , whereas the proof of Theorem 5.12 shows that it is at least 34 . If an integer programming formulation has integrality gap ρ, then any algorithm for a maximization problem whose performance guarantee α is proven by showing that the value of its solution is at least α times the value of the linear programming relaxation can have performance guarantee at most ρ. A similar statement holds for minimization problems.
5.7
The prizecollecting Steiner tree problem
We now consider other problems for which randomized techniques are useful. In particular, in the next few sections, we revisit some of the problems we studied earlier in the book and show that we can obtain improved performance guarantees by using the randomized methods that we have developed in this chapter. We start by returning to the prizecollecting Steiner tree problem we discussed in Section 4.4. Recall that in this problem we are given an undirected graph G = (V, E), an edge cost ce ≥ 0 for each e ∈ E, a selected root vertex r ∈ V , and a penalty πi ≥∑ 0 for each ∑ i ∈ V . The goal is to ﬁnd a tree T that contains the root vertex r so as to minimize e∈T ce + i∈V −V (T ) πi , where V (T ) is the set of vertices in the tree. We used the following linear programming relaxation of the problem: ∑ ∑ minimize ce xe + πi (1 − yi ) e∈E
subject to
i∈V
∑
xe ≥ yi ,
∀S ⊆ V − r, S ̸= ∅, ∀i ∈ S,
e∈δ(S)
yr = 1, yi ≥ 0,
∀i ∈ V,
xe ≥ 0,
∀e ∈ E.
In the 3approximation algorithm given in Section 4.4, we found an optimal LP solution (x∗ , y ∗ ), and for a speciﬁed value of α, built a Steiner tree on all nodes such that yi∗ ≥ α. We claimed that the cost of the edges in the tree is within a factor of 2/α of the fractional cost of the tree edges in the LP solution, while the cost of the penalties is within a factor of 1/(1 − α) of the cost of the penalties in the LP solution. Thus if α is close to 1, the cost of the tree edges is within a factor of two of the corresponding cost of the LP, while if α is close to 0, our penalty cost is close to the corresponding cost of the LP. In the previous section we set α = 2/3 to trade oﬀ the costs of the tree edges and the penalties, resulting in a performance guarantee of 3. Suppose instead that we choose the value of α randomly rather than considering just one value of α. We will see that this improves the performance guarantee from 3 to about 2.54. Recall Lemma 4.6 from Section 4.4, which allowed us to bound the cost of the spanning tree T constructed in terms of α and the LP solution (x∗ , y ∗ ). Lemma 5.14 (Lemma 4.6):
∑ e∈T
ce ≤
2∑ ce x∗e . α e∈E
Because the bound becomes inﬁnite as α tends to zero, we don’t want to choose α too close to zero. Instead, we will choose α uniformly from the range [γ, 1], and will later decide Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.7
The prizecollecting Steiner tree problem
119
how to set γ. Recall that in computing the expected value of a continuous random variable X, if that variable has a probability density function f (x) over the domain [a, b] (specifying the probability that X = x), then we compute the expectation of X by integrating f (x)xdx over that interval. The probability density function for a uniform random variable over [γ, 1] is the constant 1/(1 − γ). We can then analyze the expected cost of the tree for the randomized algorithm below. [
Lemma 5.15: E
∑
]
(
ce ≤
e∈T
2 1 ln 1−γ γ
)∑
ce x∗e
e∈E
Proof. Using simple algebra and calculus, we obtain ] [ ] [ ∑ 2∑ ce ≤ E ce x∗e E α e∈T e∈E [ ]∑ 2 = E ce x∗e α e∈E )∑ ( ∫ 1 2 1 dx ce x∗e = 1−γ γ x e∈E [ ]1 ∑ 2 = ln x · ce x∗e 1−γ γ e∈E )∑ ( 1 2 ln ce x∗e . = 1−γ γ e∈E
The expected penalty for the vertices not in the tree is also easy to analyze. Lemma 5.16:
∑
E
i∈V −V (T )
πi ≤
1 ∑ πi (1 − yi∗ ) 1−γ i∈V
Proof.∑Let U = {i ∈ V∑: yi∗ ≥ α}; any vertex not in the tree must not be in U , so we have ∗ / U is that i∈V −V (T ) πi ≤ i∈U / πi . Observe that if yi ≥ γ, then the probability that i ∈ ∗ ∗ (1 − yi )/(1 − γ). If yi < γ, then i ∈ / U with probability 1. But then 1 ≤ (1 − yi∗ )/(1 − γ), and so the lemma statement follows. Thus we have the following theorem and corollary. Theorem 5.17: The expected cost of the solution produced by the randomized algorithm is ( ) ∑ ∑ 2 1 ∑ 1 ∑ E ce + πi ≤ ln ce x∗e + πi (1 − yi∗ ). 1−γ γ 1−γ e∈T
i∈V −V (T )
e∈E
i∈V
Corollary 5.18: Using the randomized rounding algorithm with γ = e−1/2 gives a (1−e−1/2 )−1 approximation algorithm for the prizecollecting Steiner tree problem, where (1 − e−1/2 )−1 ≈ 2.54. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
120
Random sampling and randomized rounding of linear programs
Proof. We want to ensure that the maximum of the coeﬃcients of the two terms in the bound of Theorem 5.17 is as small as possible. The ﬁrst coeﬃcient is a decreasing function of γ, and the second is an increasing function. We can minimize the maximum by setting them equal. 2 1 ln γ1 = 1−γ , we obtain γ = e−1/2 . Then by Theorem 5.17, the expected cost of By setting 1−γ the tree obtained is no greater than 1 1 − e−1/2
(
∑
e∈E
ce x∗e +
∑ i∈V
) πi (1 − yi∗ )
≤
1 · OPT . 1 − e−1/2
∗ The derandomization of the algorithm is straightforward: since there are { V  variables y}i , there are at most V  distinct values of yi∗ . Thus, consider the V  sets Uj = i ∈ V : yi∗ ≥ yj∗ . Any possible value of α corresponds to one of these V  sets. Thus if a random choice of α has a certain expected performance guarantee, the algorithm which tries each set Uj and chooses the best solution generated will have a deterministic performance guarantee at least as good as that of the expectation of the randomized algorithm. Interestingly, the use of randomization allows us to analyze this natural deterministic algorithm, whereas we know of no means of analyzing the deterministic algorithm directly. Recall from the end of the previous section that we deﬁned the integrality gap of an integer programming formulation to be the worstcase ratio, over all instances of a problem, of the value of an optimal solution to the integer programming formulation to the value of an optimal solution to the linear programming relaxation. We also explained that the integrality gap bounds the performance guarantee that we can get via LP rounding arguments. Consider a graph G that is a cycle on n nodes, and let the penalty for each node be inﬁnite and the cost of each edge by 1. Then there is a feasible solution to the linear programming relaxation of cost n/2 by setting each edge variable to 1/2, while there is an optimal integral solution of cost n − 1 by taking every edge of the cycle except one (see Figure 5.4). Hence the integrality gap for this instance of the problem is at least (n − 1)/(n/2) = 2 − 2/n. Thus we cannot expect a performance guarantee for the prizecollecting Steiner tree problem better than 2 − n2 using LP rounding arguments with this formulation. In Chapter 14, we will return to the prizecollecting Steiner tree problem and show that the primaldual method can be used to obtain a 2approximation algorithm for the problem. The argument there will show that the integrality gap of the integer programming formulation is at most 2.
5.8
The uncapacitated facility location problem
In this section, we revisit the metric uncapacitated facility location problem introduced in Section 4.5. Recall that in this problem we are given a set of clients D and a set of facilities F , along with facility costs fi for all facilities i ∈ F , and assignment costs cij for all facilities i ∈ F and clients j ∈ D. All clients and facilities are points in a metric space, and given clients j, l and facilities i, k, we have that cij ≤ cil + ckl + ckj . The goal of the problem is to select a subset of facilities to open and an assignment of clients to open facilities so as to minimize the total cost of the open facilities plus the assignment costs. We used the following linear programming Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.8
The uncapacitated facility location problem
r
121
r
Figure 5.4: Example of integrality gap of the integer programming formulation for the prizecollecting Steiner tree problem. On the left is a feasible solution for the linear program in which each edge has value 1/2. On the right is an optimal solution for the integer programming formulation in which each edge shown has value 1. relaxation of the problem: minimize
∑
∑
fi yi +
i∈F
cij xij
i∈F,j∈D
∑
subject to
xij = 1,
∀j ∈ D,
xij ≤ yi ,
∀i ∈ F, j ∈ D,
xij ≥ 0,
∀i ∈ F, j ∈ D,
i∈F
yi ≥ 0,
∀i ∈ F,
where the variable xij indicates whether client j is assigned to facility i, and the variable yi indicates whether facility i is open or not. We also used the dual of the LP relaxation: ∑ maximize vj j∈D
subject to
∑
wij ≤ fi ,
∀i ∈ F,
j∈D
vj − wij ≤ cij , wij ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F, j ∈ D.
Finally, given an LP solution (x∗ , y ∗ ), we said that a client j neighbors a facility i if x∗ij > 0. We denote the neighbors of j as N (j) = {i ∈ F : x∗ij > 0}, and the neighbors of the neighbors of j as N 2 (j) = {k ∈ D : ∃i ∈ N (j), x∗ik > 0}. Recall that we showed in Lemma 4.11 that if (v ∗ , w∗ ) is an optimal dual solution and i ∈ N (j) then the cost of assigning j to i is bounded by vj∗ (that is, cij ≤ vj∗ ). We gave a 4approximation algorithm in Algorithm 4.2 of Section 4.5 that works by choosing an unassigned client j that minimizes the value of vj∗ among all remaining unassigned clients, opening the cheapest facility in the neighborhood of N (j), and then assigning j and all clients in N 2 (j) to this facility. We showed that for an ∑ optimal LP solution (x∗ , y ∗ ) and optimal dual ∑ ∗ ∗ solution v , this gave a solution of cost at most i∈F fi yi + 3 j∈D vj∗ ≤ 4 · OPT. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
122
Random sampling and randomized rounding of linear programs
Solve LP, get optimal primal solution (x∗ , y ∗ ) and dual solution (v ∗ , w∗ ) C←D k←0 while C ̸= ∅ do k ←k+1 Choose jk ∈ C that minimizes vj∗ + Cj∗ over all j ∈ C Choose ik ∈ N (jk ) according to the probability distribution x∗ijk Assign jk and all unassigned clients in N 2 (jk ) to ik C ← C − {jk } − N 2 (jk ) Algorithm 5.1: Randomized rounding algorithm for the uncapacitated facility location problem.
i j
h
l
Figure 5.5: Illustration of proof of Theorem 5.19. ∑ ∗ This analysis is a little unsatisfactory in∑ the sense that we bound i∈F fi yi by OPT, whereas ∑ we know that we have the stronger bound i∈F fi yi∗ + i∈F,j∈D cij x∗ij ≤ OPT. In this section, we show that by using randomized rounding we can modify the algorithm of Section 4.5 slightly and improve the analysis to a 3approximation algorithm. The basic idea is that once we have selected a client j in the algorithm, instead of opening the cheapest facility in N (j), we use randomized ∑ rounding to choose the facility, and open facility i ∈ N (j) with probability x∗ij (note that i∈N (j) x∗ij = 1). This improves the analysis since in the previous version of the algorithm we had to make worstcase assumptions about how far away the cheapest facility would be from the clients assigned to it. In this algorithm we can amortize the costs over all possible choices of facilities in N (j). In order to get our analysis ∑ to work, we modify the choice of client selected in each iteration as well. We deﬁne Cj∗ = i∈F cij x∗ij ; that is, the assignment cost incurred by client j in the LP solution (x∗ , y ∗ ). We now choose the unassigned client that minimizes vj∗ + Cj∗ over all unassigned clients in each iteration. Our new algorithm is given in Algorithm 5.1. Note that the only changes from the previous algorithm of Section 4.5 (Algorithm 4.2) are in the third to the last and second to the last lines. We can now analyze this new algorithm. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.8
The uncapacitated facility location problem
123
Theorem 5.19: Algorithm 5.1 is a randomized 3approximation algorithm for the uncapacitated facility location problem. Proof. In an iteration k, the expected cost of the facility opened is ∑ ∑ fi x∗ijk ≤ fi yi∗ , i∈N (jk )
i∈N (jk )
using the LP constraint x∗ijk ≤ yi∗ . As we argued in Section 4.5, the neighborhoods N (jk ) form a partition of a subset of the facilities so that the overall expected cost of facilities opened is at most ∑ ∑ ∑ fi yi∗ . fi yi∗ ≤ i∈F
k i∈N (jk )
We now ﬁx an iteration k and let j denote the client jk selected and let i denote the facility ik opened. The expected cost of assigning j to i is ∑ cij x∗ij = Cj∗ . i∈N (j)
As can be seen from Figure 5.5, the expected cost of assigning an unassigned client l ∈ N 2 (j) to i, where the client l neighbors facility h which neighbors client j is at most ∑ chl + chj + cij x∗ij = chl + chj + Cj∗ . i∈N (j)
By Lemma 4.11, chl ≤ vl∗ and chj ≤ vj∗ , so that this cost is at most vl∗ + vj∗ + Cj∗ . Then since we chose j to minimize vj∗ + Cj∗ among all unassigned clients, we know that vj∗ + Cj∗ ≤ vl∗ + Cl∗ . Hence the expected cost of assigning l to i is at most vl∗ + vj∗ + Cj∗ ≤ 2vl∗ + Cl∗ . Thus we have that our total expected cost is no more than ∑ ∑ ∑ ∑ ∑ fi yi∗ + (2vj∗ + Cj∗ ) = fi yi∗ + cij x∗ij + 2 vj∗ i∈F
j∈D
i∈F
i∈F,j∈D
j∈D
≤ 3 OPT .
Note that we were able to reduce the performance guarantee from 4 to 3 because the random choice of facility allows us to include the assignment cost Cj∗ in the analysis; instead of bounding only the facility cost by OPT, we can bound both the facility cost and part of the assignment cost by OPT. One can imagine a diﬀerent type of randomized rounding algorithm: suppose we obtain an optimal LP solution (x∗ , y ∗ ) and open each facility i ∈ F with probability yi∗ . Given the open facilities, we then assign each client to the open facility. This algorithm has the nice ∑ closest ∗ feature that the expected facility cost is i∈F fi yi . However, this simple algorithm clearly has the diﬃculty that with nonzero probability, the algorithm opens no facilities at all, and hence the expected assignment cost is unbounded. We consider a modiﬁed version of this algorithm later in the book, in Section 12.1. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
124
Random sampling and randomized rounding of linear programs
5.9
Scheduling a single machine with release dates
In this section, we return to the problem considered in Section 4.2 of scheduling a single machine with release dates so as to minimize the weighted sum of completion times. Recall that we are given as input n jobs, each of which has a processing time pj > 0, weight wj ≥ 0, and release date rj ≥ 0. The values pj , rj , and wj are all nonnegative integers. We must construct a schedule for these jobs on a single machine such that at most one job is processed at any point in time, no job is processed before its release date, and once a job begins to be processed, it must be processed nonpreemptively; that is, it must be processed completely before any other job can be scheduled. If Cj denotes the∑ time at which job j is ﬁnished processing, then the goal is to ﬁnd the schedule that minimizes nj=1 wj Cj . In Section 4.2, we gave a linear programming relaxation of the problem. In order to apply randomized rounding, we will use a diﬀerent integer programming formulation of this problem. In fact, we will not use an integer programming formulation of the problem, but an integer programming relaxation. Solutions in which jobs are scheduled preemptively are feasible; however, the contribution of job j to the objective function is less than wj Cj unless job j is scheduled nonpreemptively. Thus the integer program is a relaxation since for any solution corresponding to a nonpreemptive schedule, the objective function value is equal to the sum of weighted completion times of the schedule. Furthermore, although this relaxation is an integer program and has a number of constraints and variables exponential in the size of the problem instance, we will be able to ﬁnd a solution to it in polynomial time. ∑ We now give the integer programming relaxation. Let T equal maxj rj + nj=1 pj , which is the latest possible time any job can be processed in any schedule that processes a job nonpreemptively whenever it can. We introduce variables yjt for j = 1, . . . , n, t = 1, . . . , T , where { 1 if job j is processed in time [t − 1, t) yjt = 0 otherwise We derive a series of constraints for the integer program to capture the constraints of the scheduling problem. Since at most one job can be processed at any point in time, for each time t = 1, . . . , T we impose the constraint n ∑
yjt ≤ 1.
j=1
Since each job j must be processed for pj units of time, for each job j = 1, . . . , n we impose the constraint T ∑ yjt = pj . t=1
Since no job j can be processed before its release date, we set yjt = 0 for each job j = 1, . . . , n and each time t = 1, . . . , rj . For t = rj + 1, . . . , T , obviously we want yjt ∈ {0, 1} . Note that this integer program has size that is exponential in the size of the scheduling instance because T is exponential in the number of bits used to encode rj and pj . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.9
Scheduling a single machine with release dates
125
Finally, we need to express the completion time of job j in terms of the variables yjt , j = 1, . . . , n. Given a nonpreemptive schedule, suppose that job j completes at time D. If we set the variables yjt to indicate the times at which jobs are processed in the schedule, and so yjt = 1 for t = D −pj +1, . . . , D, whereas yjt = 0 otherwise. Observe that if we take the average of the midpoints of each unit of time at which job j is processed (t− 12 for t = D−pj +1, , . . . , D), p we get the midpoint of the processing time of the job, namely D − 2j . Thus 1 pj
(
D ∑ t=D−pj +1
1 t− 2
) =D−
pj . 2
Given the settings of the variables yjt , we can rewrite this as ( ) T pj 1 1 ∑ yjt t − =D− . pj 2 2 t=1
We wish to have variable Cj represent the completion time of job j. Rearranging terms, then, we set the variable Cj as follows: ( ) T pj 1 ∑ 1 Cj = yjt t − + . pj 2 2 t=1
This variable Cj underestimates the completion time of job j when all of the variables yjt that are set to 1 are not consecutive in time. To see this, ﬁrst start with the case above in which yjt = 1 for t = D − pj + 1, . . . , D for a completion time D. By the arguments above, Cj = D. If we then modify the variables yjt by successively setting yjt = 0 for some t ∈ [D − pj + 1, D − 1] and yjt = 1 for some t ≤ D − pj , it is clear that the variable Cj only decreases. The overall integer programming relaxation of the problem we will use is minimize
n ∑
wj Cj
(5.3)
j=1
subject to
n ∑
yjt ≤ 1,
t = 1, . . . , T,
(5.4)
yjt = pj ,
j = 1, . . . , n,
(5.5)
yjt = 0,
j = 1, . . . , n; t = 1, . . . , rj ,
j=1 T ∑ t=1
yjt ∈ {0, 1} , Cj =
1 pj
T ∑ t=1
j = 1, . . . , n; t = 1, . . . , T, ( yjt t −
1 2
) +
pj , 2
j = 1, . . . , n.
(5.6)
Even though we restrict the variables yjt to take on integer values, the corresponding linear programming relaxation is wellknown to have optimal solutions for which these variables have integer values (for a simpler case of this phenomenon, see Exercise 4.6). We can now consider a randomized rounding algorithm for this problem. Let (y ∗ , C ∗ ) be an optimal solution to the integer programming relaxation. For each job j, let Xj be a random ∗ ∑T yjt ∗ /p ; observe that by (5.5), variable which is t − 12 with probability yjt j t=1 pj = 1 so that Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
126
Random sampling and randomized rounding of linear programs
∗ /p give a probability distribution on time t for each job j. We defer for the moment a the yjt j discussion on how to make this run in randomized polynomial time, since T is exponential in the input size and we need to solve the integer program in order to perform the randomized rounding. As in the algorithms of Sections 4.1 and 4.2, we now schedule the jobs as early as possible in the same relative order as the value of the Xj . Without loss of generality, suppose that X1 ≤ X2 ≤ · · · ≤ Xn . Then we schedule job 1 as early as possible (that is, not before r1 ), then job 2, and in general schedule job j to start at the maximum of the completion time of job j − 1 and rj . Let Cˆj be a random variable denoting the completion time of job j in this schedule. We begin the analysis of this algorithm by considering the expected value of Cˆj given a ﬁxed value of Xj .
Lemma 5.20: E[Cˆj Xj = x] ≤ pj + 2x. Proof. As we argued in the proof of Lemma 4.2, there cannot be any idle time ∑ between ˆ ˆ maxk=1,...,j rk and Cj , and therefore it must be the case that Cj ≤ maxk=1,...,j rk + jk=1 pj . Because the ordering of the jobs results from randomized rounding, we let R be a random ∑j−1 variable such that R = maxk=1,...,j rk , and let random variable P = k=1 pj , so that the bound ˆ ˆ on Cj becomes Cj ≤ R + P + pj . ∗ = 0 for t ≤ r , it First, we bound the value of R given that Xj = x. Note that since ykt k must be the case that Xk ≥ rk + 21 for any job k. Thus, 1 1 1 ≤ Xj − = x − . 2 2 2
R ≤ max rk ≤ max Xk − k:Xk ≤Xj
k:Xk ≤Xj
Now we bound the expected value of P given that Xj = x. We can bound it as follows: ∑ pk Pr[job k is processed before jXj = x] E[P Xj = x] = k:k̸=j
=
∑
pk Pr[Xk ≤ Xj Xj = x]
k:k̸=j
=
∑ k:k̸=j
x+ 12
pk
∑ t=1
[ ] 1 Pr Xk = t − . 2
∗ /p , then with probability ykt k 1 1 x+ 2 x+ 12 ∑ x+ ∑ ∑ y∗ ∑2 ∑ ∑ ∗ ∗ kt E[P Xj = x] = pk = ykt . ykt = pk
Since we set Xk = t −
1 2
k:k̸=j
t=1
k:k̸=j t=1
t=1 k:k̸=j
Constraint (5.4) of the integer programming relaxation imposes that times t, so then x+ 12
E[P Xj = x] =
∑ ∑ t=1 k:k̸=j
∑
k:k̸=j
ykt ≤ 1 for all
1 ∗ ykt ≤x+ . 2
Therefore,
( ) ( ) 1 1 E[Cˆj Xj = x] ≤ pj + E[RXj = x] + E[P Xj = x] ≤ pj + x − + x+ = pj + 2x. 2 2
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.9
Scheduling a single machine with release dates
127
Given the lemma above, we can prove the following theorem. Theorem 5.21: The randomized rounding algorithm is a randomized 2approximation algorithm for the single machine scheduling problem with release dates minimizing the sum of weighted completion times. Proof. Using the lemma above, we have that ] [ ] 1 1 ˆ E Cj Xj = t − Pr Xj = t − 2 2 t=1 ) [ ] T ( ∑ 1 1 ≤ pj + 2 t− Pr Xj = t − 2 2 t=1 ) ( T ∗ ∑ 1 yjt = pj + 2 t− 2 pj t=1 [ ) ] T −1 ( pj 1 ∑ 1 ∗ = 2 + t− yjt 2 pj 2
E[Cˆj ] =
T ∑
[
t=0
= 2Cj∗ ,
(5.7)
where equation (5.7) follows from the deﬁnition of Cj∗ in the integer programming relaxation (equation (5.6)). Thus we have that n n n ∑ ∑ ∑ ˆ ˆ wj Cj = wj E[Cj ] ≤ 2 wj Cj∗ ≤ 2 OPT, E j=1
j=1
j=1
∑ since nj=1 wj Cj∗ is the objective function of the integer programming relaxation and thus a lower bound on OPT . Unlike the previous randomized algorithms of the section, we do not know directly how to derandomize this algorithm, although a deterministic 2approximation algorithm for this problem does exist. We now show that the integer programming relaxation can be solved in polynomial time, and that the randomized rounding algorithm can be made to run in polynomial time. First, sort the jobs in order of nonincreasing ratio of weight to processing time; we assume jobs are then indexed in this order so that wp11 ≥ wp22 ≥ · · · ≥ wpnn . Now we use the following rule to create a (possibly preemptive) schedule: we always schedule the job of minimum index which is available but not yet completely processed. ∑ More formally, as t varies from 1 to T , let j be ∗ the smallest index such that rj ≤ t − 1 and t−1 z=1 yjz < pj , if such a job j exists. Then set ∗ = 0 for all jobs j. ∗ ∗ yjt = 1 for job j and ykt = 0 for all jobs k ̸= j. If no such j exists, we set yjt Since in creating this schedule there are only n points in time corresponding to release dates rj and n points in time at which a job has ﬁnished being processed, there are at most 2n point in time at which the index attaining the minimum might change. Thus we can actually give the schedule as a sequence of at most 2n intervals of time, specifying which job (if any) is scheduled for each interval. It is not hard to see that we can compute this set of intervals in polynomial time without explicitly enumerating each time t. Furthermore, from the discussion above in which we explained how to express the variable Cj in terms of the yjt , we know that if yjt = 1 Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
128
Random sampling and randomized rounding of linear programs
( ) ∑ for t = a + 1 to b, then bt=a+1 yjt t − 12 = (b − a)(b − 12 (b − a)), so that we can compute the values of the variables Cj∗ from these intervals. The randomized rounding algorithm can be made to run in polynomial time because it is equivalent to the following algorithm: for each job j, choose a value αj ∈ [0, 1] independently and uniformly. Let Xj be the αj point of job j: that is, the time when αj pj units of job j have been processed in the preemptive schedule. Observe that it is easy to compute this point in time from the intervals describing the preemptive schedule. Then schedule the jobs according to the ordering of the Xj as in the randomized rounding algorithm. To see why this is equivalent to the original algorithm, consider the probability that Xj ∈ [t − 1, t). This is simply the probability that αj pj units of job j have ﬁnished processing in this interval, which is the probability that t−1 t ∑ ∑ ∗ ∗ yjs ≤ αj pj < yjs . s=1
s=1
∑t−1
∑t ∗ , 1 ∗ This, then, is the probability that αj ∈ [ p1j s=1 yjs s=1 yjs ); since αj is chosen uniformly, pj ∗ /p . So the probability that X ∈ [t − 1, t) in the α point algorithm is this probability is yjt j j j the same as in the original algorithm, and the proof of the performance guarantee goes through with some small modiﬁcations. Interestingly, one can also prove that if a single value of α is chosen uniformly from [0, 1], and Xj is the αpoint of job j, then scheduling jobs according to the ordering of the Xj is also a 2approximation algorithm. Proving this fact is beyond the scope of this section. However, the algorithm in which a single value of α is chosen is easy to derandomize, because it is possible to show that at most n diﬀerent schedules can result from all possible choices of α ∈ [0, 1]. Then to derive a deterministic 2approximation algorithm, we need only enumerate the n diﬀerent schedules, and choose the one that minimizes the weighted sum of the completion times. We need only show that the constructed solution (y ∗ , C ∗ ) is in fact optimal. This is left to the reader to complete by a straightforward interchange argument in Exercise 5.11. Lemma 5.22: The solution (y ∗ , C ∗ ) given above to the integer program is an optimal solution.
5.10
Chernoﬀ bounds
This section introduces some theorems that are extremely useful for analyzing randomized rounding algorithms. In essence, the theorems say that it is very likely that the sum of n independent 01 random variables is not far away from the expected value of the sum. In the subsequent two sections, we will illustrate the usefulness of these bounds. We begin by stating the main theorems. Theorem 5.23: Let X1 , . . . , Xn be ∑nn independent 01 random variables, not necessarily identically distributed. Then for X = i=1 Xi and µ = E[X], L ≤ µ ≤ U , and δ > 0, ( Pr[X ≥ (1 + δ)U ] < and
( Pr[X ≤ (1 − δ)L]
0, ( Pr[X ≥ (1 + δ)U ] < and
( Pr[X ≤ (1 − δ)L]
0. Proof. Since X takes on nonnegative values, E[X] ≥ a Pr[X ≥ a] and the inequality follows. Now we can prove Theorem 5.24. Proof of Theorem 5.24. We only prove the ﬁrst bound in the theorem; the proof of the other bound is analogous. Note that if E[X] = 0, then X = 0 and the bound holds trivially, so we can assume E[X] > 0 and E[Xi ] > 0 for some i. We ignore all i such that E[Xi ] =∑ 0 since Xi = 0 for such i. Let pi = Pr[Xi = ai ]. Since E[Xi ] > 0, pi > 0. Then µ = E[X] = ni=1 pi ai ≤ U . For any t > 0, Pr[X ≥ (1 + δ)U ] = Pr[etX ≥ et(1+δ)U ]. By Markov’s inequality, E[etX ] . et(1+δ)U [ n ] n [ ∑n ] ∏ ∏ tX t i=1 Xi tXi E[e ] = E e =E e = E[etXi ], Pr[etX ≥ et(1+δ)U ] ≤
Now
i=1
(5.8)
(5.9)
i=1
where the last equality follows by the independence of the Xi . Then for each i, E[etXi ] = (1 − pi ) + pi etai = 1 + pi (etai − 1). We will later show that etai − 1 ≤ ai (et − 1) for t > 0, so that E[etXi ] ≤ 1 + pi ai (et − 1). Using that 1 + x < ex for x > 0, and that t, ai , pi > 0, we obtain E[etXi ] < epi ai (e
t −1)
.
Plugging these back into equation (5.9), we have tX
E[e
]
0, we see that E[etX ] et(1+δ)U t eU (e −1) < t(1+δ)U e )U ( eδ , = (1 + δ)(1+δ)
Pr[X ≥ (1 + δ)U ] ≤
as desired. Finally, to see that eai t − 1 ≤ ai (et − 1) for t > 0, let f (t) = ai (et − 1) − eai t − 1. Then ′ f (t) = ai et − ai eai t ≥ 0 for any t ≥ 0 given that 0 < ai ≤ 1; thus f (t) is nondecreasing for t ≥ 0. Since f (0) = 0 and the function f is nondecreasing, the inequality holds. The righthand sides of the inequalities in Theorems 5.23 and 5.24 are a bit complicated, and so it will be useful to consider variants of the results in which the righthand side is simpler, at the cost of restricting the results somewhat. Lemma 5.26: For 0 ≤ δ ≤ 1, we have that (
eδ (1 + δ)(1+δ)
)U
≤ e−U δ
2 /3
and for 0 ≤ δ < 1, we have that (
e−δ (1 − δ)(1−δ)
)L
≤ e−Lδ
2 /2
.
Proof. For the ﬁrst inequality, we take the logarithm of both sides. We would like to show that U (δ − (1 + δ) ln(1 + δ)) ≤ −U δ 2 /3. We observe that the inequality holds for δ = 0; if we can show that the derivative of the lefthand side is no more than that of the righthand side for 0 ≤ δ ≤ 1, the inequality will hold for 0 ≤ δ ≤ 1. Taking derivatives of both sides, we need to show that −U ln(1 + δ) ≤ −2U δ/3. Letting f (δ) = −U ln(1 + δ) + 2U δ/3, we need to show that f (δ) ≤ 0 on [0, 1]. Note that f (0) = 0 and f (1) ≤ 0 since − ln 2 ≈ −0.693 < −2/3. As long as the function f (δ) is convex on [0, 1] (that is, f ′′ (δ) ≥ 0), we may conclude that f (δ) ≤ 0 on [0, 1], in the convex analog of Fact 5.9. We observe that f ′ (δ) = −U/(1 + δ) + 2U/3 and f ′′ (δ) = U/(1 + δ)2 , so that f ′′ (δ) ≥ 0 for δ ∈ [0, 1], and the inequality is shown. We turn to the second inequality for the case 0 ≤ δ < 1. Taking the logarithm of both sides, we would like to show that L(−δ − (1 − δ) ln(1 − δ)) ≤ −Lδ 2 /2. The inequality holds for δ = 0, and will hold for 0 < δ < 1 if the derivative of the lefthand side is no more than the derivative of the righthand side for 0 ≤ δ < 1. Taking derivatives of both sides, we would like to show that L ln(1 − δ) ≤ −Lδ Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.10
Chernoﬀ bounds
131
for 0 ≤ δ < 1. Again the inequality holds for δ = 0 and will hold for 0 ≤ δ < 1 if the derivative of the lefthand side is no more than the derivative of the righthand side for 0 ≤ δ < 1. Taking derivatives of both sides again, we obtain −L/(1 − δ) ≤ −L, which holds for 0 ≤ δ < 1. It will sometimes be useful to provide a bound on the probability that X ≤ (1 − δ)L in the case δ = 1. Notice that since the variables Xi are either 01 or 0ai this is asking for a bound on the probability that X = 0. We can give a bound as follows. Lemma 5.27: Let X1 , . . . , Xn be n independent random variables, not necessarily identically distributed, such ∑ that each Xi takes either the value 0 or the value ai for some 0 < ai ≤ 1. Then for X = ni=1 Xi and µ = E[X], L ≤ µ, Pr[X = 0] < e−L . X = 0 and L ≤ µ = 0 and the bound holds Proof. We assume µ = E[X] > 0 since otherwise ∑n trivially. Let pi = Pr[Xi = ai ]. Then µ = i=1 ai pi and Pr[X = 0] =
n ∏ (1 − pi ). i=1
Applying the arithmetic/geometric mean inequality from Fact 5.8, we get that ]n [ ]n [ n n n ∏ 1∑ 1∑ (1 − pi ) = 1 − pi . (1 − pi ) ≤ n n i=1
i=1
i=1
Since each ai ≤ 1, we then obtain ]n [ ]n [ [ ] n n 1∑ 1 n 1∑ pi ≤ 1 − ai pi = 1 − µ . 1− n n n i=1
i=1
Then using the fact that 1 − x < e−x for x > 0, we get [ ] 1 n 1 − µ < e−µ ≤ e−L . n
As a corollary, we can then extend the bound of Lemma 5.26 to the case δ = 1. In fact, since the probability that X < (1 − δ)L for δ > 1 and L ≥ 0 is 0, we can extend the bound to any positive δ. Corollary 5.28: Let X1 , . . . , Xn be n independent random variables, not necessarily identically distributed, ∑ such that each Xi takes either the value 0 or the value ai for some ai ≤ 1. Then for X = ni=1 Xi and µ = E[X], 0 ≤ L ≤ µ, and δ > 0, Pr[X ≤ (1 − δ)L] < e−Lδ
2 /2
.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
132
Random sampling and randomized rounding of linear programs
5.11
Integer multicommodity ﬂows
To see how Chernoﬀ bounds can be used in the context of randomized rounding, we will apply them to the minimumcapacity multicommodity ﬂow problem. In this problem, we are given as input an undirected graph G = (V, E) and k pairs of vertices si , ti ∈ V , i = 1, . . . , k. The goal is to ﬁnd, for each i = 1, . . . , k, a single simple path from si to ti so as to minimize the maximum number of paths containing the same edge. This problem arises in routing wires on chips. In this problem, k wires need to be routed, each wire from some point si on the chip to another point ti on the chip. Wires must be routed through channels on the chip, which correspond to edges in a graph. The goal is to route the wires so as minimize the channel capacity needed; that is, the number of wires routed through the same channel. The problem as stated here is a special case of a more interesting problem, since often the wires must join up three or more points on a chip. We give an integer programming formulation of the problem. Let Pi be the set of all possible simple paths P in G from si to ti , where P is the set of edges in the path. We create a 01 variable xP for each path P ∈ Pi to indicate when ∑ path P from si to ti is used. Then the total number of paths using an edge e ∈ E is simply P :e∈P xP . We create another decision variable W to denote the maximum number of paths using an edge, so that our objective function is to minimize W . We have the constraint that ∑ xP ≤ W P :e∈P
for each edge e ∈ E. Finally, we need to choose some path P ∈ Pi for every si ti pair, so that ∑ xP = 1 P ∈Pi
for each i = 1, . . . , k. This gives us the following integer programming formulation of the minimumcapacity multicommodity ﬂow problem: minimize subject to
∑
W
(5.10)
xP = 1,
i = 1, . . . , k,
xP ≤ W,
e ∈ E,
xP ∈ {0, 1} ,
∀P ∈ Pi , i = 1, . . . , k.
P ∈Pi
∑
(5.11)
P :e∈P
The integer program can be relaxed to a linear program by replacing the constraints xP ∈ {0, 1} with xP ≥ 0. We claim for now that this linear program can be solved in polynomial time and that most a polynomial number of variables xP of an optimal solution can be nonzero. We now apply randomized rounding to obtain a solution to the problem. For i = 1, . . . , k, we choose exactly one path P ∈ Pi according to the probability distribution x∗P on paths P ∈ Pi , where x∗ is an optimal solution of value W ∗ . Assuming W ∗ is large enough, we can show that the total number of paths going through any edge is close to W ∗ by using the Chernoﬀ bound from Theorem 5.23. Let n be the number of vertices in the graph. Recall that in Section 1.7 we said that a probabilistic event happens with high probability if the probability that it does not occur is at most n−c for some integer c ≥ 1. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.12
Random sampling and coloring dense 3colorable graphs
133
Theorem 5.29: If W ∗ ≥ c ln n for some constant c, then with high probability, the total number √ of paths using any edge is at most W ∗ + cW ∗ ln n. Proof. For each e ∈ E, deﬁne random variables Xei , where Xei = 1 if the chosen s∑ i ti path uses edge e, and Xei = 0 otherwise. Then the number of paths using edge e is Ye = ki=1 Xei . We want to bound maxe∈E Ye , and show that this is close to the LP value W ∗ . Certainly E[Ye ] =
k ∑
∑
x∗P =
i=1 P ∈Pi :e∈P
∑
x∗P ≤ W ∗ ,
P :e∈P
i by constraint (5.11) from the LP. For a ﬁxed edge e, the random √ variables Xe are independent, so we can apply the Chernoﬀ bound of Theorem 5.23. Set δ = (c ln n)/W ∗ . Since W ∗ ≥ c ln n by assumption, it follows that δ ≤ 1. Then by Theorem 5.23 and Lemma 5.26 with U = W ∗ ,
Pr[Ye ≥ (1 + δ)W ∗ ] < e−W
∗ δ 2 /3
= e−(c ln n)/3 =
1 . nc/3
√ Also (1 + δ)W ∗ = W ∗ + cW ∗ ln n. Since there can be at most n2 edges, [ ] ∑ ∗ Pr max Ye ≥ (1 + δ)W ≤ Pr[Ye ≥ (1 + δ)W ∗ ] e∈E
e∈E
≤ n2 ·
1 nc/3
= n2−c/3 .
For a constant c ≥ 12, this ensures that the theorem statement fails to hold with probability at most n12 , and by increasing c we can make the probability as small as we like. Observe that since W ∗ ≥ c ln n, the theorem above guarantees that the randomized algorithm produces a solution of no more than 2W ∗ ≤ 2 OPT. However, the algorithm might produce a solution considerably closer to optimal if W ∗ ≫ c ln n. We also observe the following corollary. Corollary 5.30: If W ∗ ≥ 1, then with high probability, the total number of paths using any edge is O(log n) · W ∗ . Proof. We repeat the proof above with U = (c ln n)W ∗ and δ = 1. In fact, the statement of the corollary can be sharpened by replacing the O(log n) with O(log n/ log log n) (see Exercise 5.13). To solve the linear program in polynomial time, we show that it is equivalent to a polynomiallysized linear program; we leave this as an exercise to the reader (Exercise 5.14, to be precise).
5.12
Random sampling and coloring dense 3colorable graphs
In this section we turn to another application of Chernoﬀ bounds. We consider coloring a δdense 3colorable graph. We say( )that a graph is dense if for some constant α the number of edges in the graph is at least α n2 ; in other words, some constant fraction of the edges that could exist in the graph do exist. A δdense graph is a special case of a dense graph. A graph is δdense if every node has at least δn neighbors for some constant δ; that is, every node has some constant fraction of neighbors it could have. Finally, a graph is kcolorable if each of the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
134
Random sampling and randomized rounding of linear programs
nodes can be assigned exactly one of k colors in such a way that no edge has its two endpoints assigned the same color. In general, it is NPcomplete to decide whether a graph is 3colorable; in fact, the following is known. Theorem 5.31: It is NPhard to decide if a graph can be colored with only 3 colors, or needs at least 5 colors. Theorem 5.32: Assuming a variant of the unique games conjecture, for any constant k > 3, it is NPhard to decide if a graph can be colored with only 3 colors, or needs at least k colors. In Sections 6.5 and 13.2 we will discuss approximation algorithms for coloring any 3colorable graph. Here we will show that with high probability we can properly color any δdense 3colorable graph in polynomial time. While this is not an approximation algorithm, it is a useful application of Chernoﬀ bounds, which we will use again in Section 12.4. In this case, we use the bounds to show that if we know the correct coloring for a small, randomly chosen sample of a δdense graph, we can give a polynomialtime algorithm that with high probability successfully colors the rest of the graph. This would seem to pose a problem, though, since we do not know the coloring for the sample. Nevertheless, if the sample is no larger than O(log n), we can enumerate all possible colorings of the sample in polynomial time, and run the algorithm above for each coloring. Since one of the colorings of the sample will be the correct one, for at least one of the possible colorings of the sample the algorithm will result in a correct coloring of the graph with high probability. More speciﬁcally, given a δdense graph, we will select a random subset S ⊆ V of O((ln n)/δ) vertices by including each vertex in the subset with probability (3c ln n)/δn for some constant c. We will show ﬁrst that the set size is no more than (6c ln n)/δ with high probability, and then that with high probability, every vertex has at least one neighbor in S. Thus given a correct coloring of S, we can use the information about the coloring of S to deduce the colors of the rest of the vertices. Since each vertex has a neighbor in S, its color is restricted to be one of the two remaining colors, and this turns out to be enough of a restriction that we can infer the remaining coloring. Finally, although we do not know the correct coloring of S we can run this algorithm for each of the 3(6c ln n)/δ = nO(c/δ) possible colorings of S. One of the colorings of S will be the correct coloring, and thus in at least one run of the algorithm we will be able to color the graph successfully. Lemma 5.33: With probability at most n−c/δ , the set S has size S ≥ (6c ln n)/δ. Proof. We use the Chernoﬀ bound (Theorem 5.23 and Lemma 5.26). Let Xi be a 01 random variable indicating whether vertex ∑ i is included in S. Then since each vertex is included with probability 3c ln n/δn, µ = E[ ni=1 Xi ] = (3c ln n)/δ. Applying the lemma with U = (3c ln n)/δ, the probability that S ≥ 2U is at most e−µ/3 = n−c/δ . Lemma 5.34: The probability that a given vertex v ∈ / S has no neighbor in S is at most n−3c . Proof. Let Xi∑ be a 01 random variable indicating whether the ith neighbor of v is in S or not. Lemma 5.27 Then µ = E[ i Xi ] ≥ 3c ln n, since v has at least δn neighbors. Then applying∑ with L = 3c ln n, the probability that v has no neighbors in S is no more than Pr[ i Xi = 0] ≤ e−L = n−3c . Corollary 5.35: With probability at least 1 − 2n−(c−1) , S ≤ (6c ln n)/δ and every v ̸∈ S has at least one neighbor in S. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.12
Random sampling and coloring dense 3colorable graphs
135
Proof. This follows from Lemmas 5.33 and 5.34. The probability that both statements of the lemma are true is at least one minus the sum of the probabilities that either statement is false. The probability that every v ∈ / S has no neighbor in S is at worst n times the probability that a given vertex v ∈ / S has no neighbor in S. Since δ ≤ 1, the overall probability that both statements are true is at least 1 − n−c/δ − n · n−3c ≥ 1 − 2n−(c−1) .
Now we assume we have some coloring of the vertices in S, not necessarily one that is consistent with the correct coloring of the graph. We also assume that every vertex not in S has at least one neighbor in S. We further assume that the coloring of S is such that every edge with both endpoints in S has diﬀerently colored endpoints, since otherwise this is clearly not a correct coloring of the graph. Assume we color the graph with colors {0, 1, 2}. Given a vertex v∈ / S, because it has some neighbor in S colored with some color n(v) ∈ {0, 1, 2}, we know that v cannot be colored with color n(v). Possibly v has other neighbors in S with colors other than n(v). Either this forces the color of v or there is no way we can successfully color v; in the latter case our current coloring of S must not have been correct, and we terminate. If the color of v is not determined, then we create a binary variable x(v), which if true indicates that we color v with color n(v) + 1 (mod 3), and if false indicates that we color v with color n(v) − 1 (mod 3). Now every edge (u, v) ∈ E for u, v ∈ / S imposes the constraint n(u) ̸= n(v). To capture this, we create an instance of the maximum satisﬁability problem such that all clauses are satisﬁable if and only if the vertices not in S can be correctly colored. For each possible setting of the Boolean variables x(u) and x(v) that would cause n(u) = n(v), we create a disjunction of x(u) and x(v) that is false if it implies n(u) = n(v); for example, if x(u) = true and x(v) = false implies that n(u) = n(v), then we create a clause (x(u) ∨ x(v)). Since G is 3colorable, given a correct coloring of S, there exists a setting of the variables x(v) which satisﬁes all the clauses. Since each clause has two variables, it is possible to determine in polynomial time whether the instance is satisﬁable or not; we leave it as an exercise to the reader to prove this (Exercise 6.3). Obviously if we ﬁnd a setting of the variables that satisﬁes all constraints, this implies a correct coloring of the entire graph, whereas if the constraints are not satisﬁable, our current coloring of S must not have been correct. In Section 12.4, we’ll revisit the idea from this section of drawing a small random sample of a graph and using it to determine the overall solution for the maximum cut problem in dense graphs.
Exercises 5.1 In the maximum kcut problem, we are given an undirected graph G = (V, E), and nonnegative weights wij ≥ 0 for all (i, j) ∈ E. The goal is to partition the vertex set V into k parts V1 , . . . , Vk so as to maximize the weight of all edges whose endpoints are in diﬀerent parts (i.e., max(i,j)∈E:i∈Va ,j∈Vb ,a̸=b wij ). Give a randomized
k−1 k approximation
algorithm for the MAX kCUT problem.
5.2 Consider the following greedy algorithm for the maximum cut problem. We suppose the vertices are numbered 1, . . . , n. In the ﬁrst iteration, the algorithm places vertex 1 in U . In the kth iteration of the algorithm, we will place vertex k in either U or in W . In order to decide which choice to make, we will look at all the edges F that have the vertex k as one Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
136
Random sampling and randomized rounding of linear programs
endpoint and whose other endpoint is 1, . . . , k−1, so that F = {(j, k) ∈ E : 1 ≤ j ≤ k − 1}. We choose to put vertex k in U or W depending on which of these two choices maximizes the number of edges of F being in the cut. (a) Prove that this algorithm is a 1/2approximation algorithm for the maximum cut problem. (b) Prove that this algorithm is equivalent to the derandomization of the maximum cut algorithm of Section 5.1 via the method of conditional expectations. 5.3 In the maximum directed cut problem (sometimes called MAX DICUT) we are given as input a directed graph G = (V, A). Each directed arc (i, j) ∈ A has nonnegative weight wij ≥ 0. The goal is to partition V into two sets U and W = V − U so as to maximize the total weight of the arcs going from U to W (that is, arcs (i, j) with i ∈ U and j ∈ W ). Give a randomized 41 approximation algorithm for this problem. 5.4 Consider the nonlinear randomized rounding algorithm for MAX SAT as given in Section 5.6. Prove that using randomized rounding with the linear function f (yi ) = 12 yi + 14 also gives a 43 approximation algorithm for MAX SAT. 5.5 Consider the nonlinear randomized rounding algorithm for MAX SAT as given in Section 5.6. Prove that using randomized rounding with the piecewise linear function 3 4 yi + 41 for 0 ≤ yi ≤ 31 1/2 for 13 ≤ yi ≤ 23 f (yi ) = 3 for 23 ≤ yi ≤ 1 4 yi also gives a 34 approximation algorithm for MAX SAT. 5.6 Consider again the maximum directed cut problem from Exercise 5.3. (a) Show that the following integer program models the maximum directed cut problem: ∑ maximize wij zij (i,j)∈A
subject to
zij ≤ xi ,
∀(i, j) ∈ A,
zij ≤ 1 − xj ,
∀(i, j) ∈ A,
xi ∈ {0, 1} ,
∀i ∈ V,
0 ≤ zij ≤ 1,
∀(i, j) ∈ A.
(b) Consider a randomized rounding algorithm for the maximum directed cut problem that solves a linear programming relaxation of the integer program and puts vertex i ∈ U with probability 1/4 + xi /2. Show that this gives a randomized 1/2approximation algorithm for the maximum directed cut problem. 5.7 In this exercise, we consider how to derandomize the randomized rounding algorithm for the set cover problem given in Section 1.7. We would like to apply the method of conditional expectations, but we need to ensure that at the end of the process we obtain a valid set cover. Let Xj be a random variable indicating whether set Sj is included in the solution. Then if wj is the weight of set Sj , let W be the weight of the set cover Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.12
Random sampling and coloring dense 3colorable graphs
137
∑ obtained by randomized rounding, so that W = m j=1 wj Xj . Let Z be a random variable such that Z = 1 if randomized rounding does not produce a valid set cover, and Z = 0 if it does. Then consider applying the method of conditional expectations to the objective function W + λZ for some choice of λ ≥ 0. Show that for the proper choice of λ, the method of conditional expectations applied to the randomized rounding algorithm yields an O(ln n)approximation algorithm for the set cover problem that always produces a set cover. 5.8 Consider a variation of the maximum satisﬁability problem in which all variables occur positively in each clause, and there is an additional nonnegative weight vi ≥ 0 for each Boolean variable xi . The goal is now to set the Boolean variables to maximize the total weight of the satisﬁed clauses plus the total weight of variables set to be false. Give an integer programming formulation for this problem, with 01 variables yi to indicate whether xi is set true. Show that a randomized rounding of the linear program in which √ 2 − 1)approximation variable xi is set true with probability 1 − λ + λyi∗ gives a 2( √ algorithm for some appropriate setting of λ; note that 2( 2 − 1) ≈ .828. 5.9 Recall the maximum coverage problem from Exercise 2.11; in it, we are given a set of elements E, and m subsets of elements S1 , . . . , Sm ⊆ E with a nonnegative weight wj ≥ 0 for each subset Sj . We would like to ﬁnd a subset S ⊆ E of size k that maximizes the total weight of the subsets covered by S, where S covers Sj if S ∩ Sj ̸= ∅. (a) Show that the following nonlinear integer program models the maximum coverage problem: ∑ ∏ (1 − xe ) maximize wj 1 − j∈[m]
e∈Sj
subject to
∑
xe = k,
e∈E
xe ∈ {0, 1} ,
∀e ∈ E.
(b) Show that the following linear program is a relaxation of the maximum coverage problem: ∑ maximize wj zj j∈[m]
subject to
∑
xe ≥ zj ,
∀j ∈ [m]
e∈Sj
∑
xe = k,
e∈E
0 ≤ zj ≤ 1,
∀j ∈ [m],
0 ≤ xe ≤ 1,
∀e ∈ E.
(c) Using the pipage rounding technique from Exercise 4.7, give an algorithm that deterministically rounds the optimal LP solution to an integer solution and has a performance guarantee of 1 − 1e . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
138
Random sampling and randomized rounding of linear programs
5.10 In the uniform labeling problem, we are given a graph G = (V, E), costs ce ≥ 0 for all e ∈ E, and a set of labels L that can be assigned to the vertices of V . There is a nonnegative cost civ ≥ 0 for assigning label i ∈ L to vertex v ∈ V , and an edge e = (u, v) incurs cost ce if u and v are assigned diﬀerent labels. The goal of the problem is to assign each vertex in V a label so as to minimize the total cost. We give an integer programming formulation of the problem. Let the variable xiv ∈ {0, 1} be 1 if vertex v is assigned label i ∈ L, and 0 otherwise. Let the variable zei be 1 if exactly one of the two endpoints of the edge e is assigned label i, and 0 otherwise. Then the integer programming formulation is as follows: ∑ 1∑ ∑ i ce ze + civ xiv 2 e∈E i∈L v∈V,i∈L ∑ subject to xiv = 1,
minimize
∀v ∈ V,
i∈L
zei ≥ xiu − xiv zei zei xiv
≥
xiv
−
xiu
∀(u, v) ∈ E, ∀i ∈ L, ∀(u, v) ∈ E, ∀i ∈ L,
∈ {0, 1} ,
∀e ∈ E, ∀i ∈ L,
∈ {0, 1} ,
∀v ∈ V, ∀i ∈ L.
(a) Prove that the integer programming formulation models the uniform labeling problem. Consider now the following algorithm. First, the algorithm solves the linear programming relaxation of the integer program above. The algorithm then proceeds in phases. In each phase, it picks a label i ∈ L uniformly at random, and a number α ∈ [0, 1] uniformly at random. For each vertex v ∈ V that has not yet been assigned a label, we assign it label i if α ≤ xiv . (b) Suppose that vertex v ∈ V has not yet been assigned a label. Prove that the probability that v is assigned label i ∈ L in the next phase is exactly xiv /L, and the probability that it is assigned a label in the next phase is exactly 1/L. Further prove that the probability that v is assigned label i by the algorithm is exactly xiv . (c) We say that an edge e is separated by a phase if both endpoints were not assigned labels prior to the phase, and exactly one of the endpoints is assigned a label in this∑phase. Prove that the probability that an edge e is separated by a phase is 1 i i∈L ze . L that the probability that the endpoints of edge e receive diﬀerent labels is at (d) Prove∑ most i∈L zei . (e) Prove that the algorithm is a 2approximation algorithm for the uniform labeling problem. 5.11 Prove Lemma 5.22, and show that the integer programming solution (y ∗ , C ∗ ) described at the end of Section 5.9 must be optimal for the integer program (5.3). 5.12 Using randomized rounding and ﬁrst ﬁt, give a randomized polynomialtime algorithm for the binpacking problem that uses ρ · OPT(I) + k bins for some ρ < 2 and some small constant k. One idea is to consider the linear program from Section 4.6. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
5.12
Random sampling and coloring dense 3colorable graphs
139
5.13 Show that the O(log n) factor in Corollary 5.30 can be replaced with O(log n/ log log n) by using Theorem 5.23. 5.14 Show that there is a linear programming relaxation for the integer multicommodity ﬂow problem of Section 5.10 that is equivalent to the linear program (5.10) but has a number of variables and constraints that are bounded by a polynomial in the input size of the ﬂow problem.
Chapter Notes The textbooks of Mitzenmacher and Upfal [226] and Motwani and Raghavan [228] give more extensive treatments of randomized algorithms. A 1967 paper of Erd˝os [99] on the maximum cut problem showed that sampling a solution uniformly at random as in Theorem 5.3 gives a solution whose expected value is at least half the sum of the edge weights. This is one of the ﬁrst randomized approximation algorithms of which we are aware. This algorithm can also be viewed as a randomized version of a deterministic algorithm given by Sahni and Gonzalez [257] (the deterministic algorithm of Sahni and Gonzalez is given in Exercise 5.2). Raghavan and Thompson [247] were the ﬁrst to introduce the idea of the randomized rounding of a linear programming relaxation. The result for integer multicommodity ﬂows in Section 5.11 is from their paper. Random sampling and randomized rounding are most easily applied to unconstrained problems, such as the maximum satisﬁability problem and the maximum cut problem, in which any solution is feasible. Even problems such as the prizecollecting Steiner tree problem and the uncapacitated facility location problem can be viewed as unconstrained problems: we merely need to select a set of vertices to span or facilities to open. Randomized approximation algorithms for constrained problems exist, but are much rarer. The results for the maximum satisﬁability problem in this chapter are due to a variety of authors. The simple randomized algorithm of Section 5.1 is given by Yannakakis [293] as a randomized variant of an earlier deterministic algorithm introduced by Johnson [179]. The “biased coins” algorithm of Section 5.3 is a similar randomization of an algorithm of Lieberherr and Specker [216]. The randomized rounding, “better of two”, and nonlinear randomized rounding algorithms in Sections 5.4, 5.5, and 5.6 respectively are due to Goemans and Williamson [137]. The derandomization of randomized algorithms is a major topic of study. The method of conditional expectations given in Section 5.2 is implicit in the work of Erd˝os and Selfridge [101], and has been developed by Spencer [271]. The randomized algorithm for the prizecollecting Steiner tree problem in Section 5.7 is an unpublished result of Goemans. The algorithm of Section 5.8 for uncapacitated facility location is due to Chudak and Shmoys [77]. The scheduling algorithm of Section 5.9 is due to Schulz and Skutella [261]. The algorithm for solving the integer programming relaxation of this problem is due to Goemans [132], and the αpoint algorithm that uses a single value of α is also due to Goemans [133]. Chernoﬀ [71] gives the general ideas used in the proof of Chernoﬀ bounds. Our proofs of these bounds follow those of Mitzenmacher and Upfal [226] and Motwani and Raghavan [228]. The randomized algorithm for 3coloring a dense 3colorable graph is due to Arora, Karger, and Karpinski [16]; a deterministic algorithm for the problem had earlier been given by Edwards Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
140
Random sampling and randomized rounding of linear programs
[97]. Theorems 5.31 and 5.32 are due to Khanna, Linial and Safra [190] (see also Guruswami and Khanna [152]) and Dinur, Mossel, and Regev [90] respectively. Exercises 5.4 and 5.5 are due to Goemans and Williamson [137]. Exercise 5.6 is due to Trevisan [279, 280]. Exercise 5.7 is due to Norton [237]. Ageev and Sviridenko [1] gave the algorithm and analysis in Exercise 5.8, while Exercise 5.9 is also due to Ageev and Sviridenko [2, 3]. The algorithm for the uniform labeling problem in Exercise 5.10 is due to Kleinberg and Tardos [197]; the uniform labeling problem models a problem arising in image processing. Exercise 5.12 is an unpublished result of Williamson.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 6
Randomized rounding of semideﬁnite programs
We now turn to a new tool which gives substantially improved performance guarantees for some problems. So far we have used linear programming relaxations to design and analyze various approximation algorithms. In this section, we show how nonlinear programming relaxations can give us better algorithms than we know how to obtain via linear programming; in particular we use a type of nonlinear program called a semideﬁnite program. Part of the power of semideﬁnite programming is that semideﬁnite programs can be solved in polynomial time. We begin with a brief overview of semideﬁnite programming. Throughout the chapter we assume some basic knowledge of vectors and linear algebra; see the notes at the end of the chapter for suggested references on these topics. We then give an application of semideﬁnite programming to approximating the maximum cut problem. The algorithm for this problem introduces a technique of rounding the semideﬁnite program by choosing a random hyperplane. We then explore other problems for which choosing a random hyperplane, or multiple random hyperplanes, is useful, including approximating quadratic programs, approximating clustering problems, and coloring 3colorable graphs.
6.1
A brief introduction to semideﬁnite programming
Semideﬁnite programming uses symmetric, positive semideﬁnite matrices, so we brieﬂy review a few properties of these matrices. In what follows, X T is the transpose of the matrix X, and vectors v ∈ ℜn are assumed to be column vectors, so that v T v is the inner product of v with itself, while vv T is an n by n matrix. Deﬁnition 6.1: A matrix X ∈ ℜn×n is positive semideﬁnite iﬀ for all y ∈ ℜn , y T Xy ≥ 0. Sometimes we abbreviate “positive semideﬁnite” as “psd.” Sometimes we will write X ≽ 0 to denote that a matrix X is positive semideﬁnite. Symmetric positive semideﬁnite matrices have some special properties which we list below. From here on, we will generally assume (unless otherwise stated) that any psd matrix X is also symmetric. Fact 6.2: If X ∈ ℜn×n is a symmetric matrix, then the following statements are equivalent: 1. X is psd; 141
142
Randomized rounding of semideﬁnite programs
2. X has nonnegative eigenvalues; 3. X = V T V for some V ∈ ℜm×n where m ≤ n; ∑n T n T 4. X = i=1 λi wi wi for some λi ≥ 0 and vectors wi ∈ ℜ such that wi wi = 1 and wiT wj = 0 for i ̸= j. A semideﬁnite program (SDP) is similar to a linear program in that there is a linear objective function and linear constraints. In addition, however, a square symmetric matrix of variables can be constrained to be positive semideﬁnite. Below is an example in which the variables are xij for 1 ≤ i, j ≤ n. maximize or minimize
∑
cij xij
(6.1)
i,j
∑
subject to
aijk xij = bk ,
∀k,
i,j
xij = xji ,
∀i, j,
X = (xij ) ≽ 0. Given some technical conditions, semideﬁnite programs can be solved to within an additive error of ϵ in time that is polynomial in the size of the input and log(1/ϵ). We explain the technical conditions in more detail in the notes at the end of the chapter. We will usually ignore the additive error when discussing semideﬁnite programs and assume that the SDPs can be solved exactly, since the algorithms we will use do not assume exact solutions, and one can usually analyze the algorithm that has additive error in the same way with only a small loss in performance guarantee. We will often use semideﬁnite programming in the form of vector programming. The variables of vector programs are vectors vi ∈ ℜn , where the dimension n of the space is the number of vectors in the vector program. The vector program has an objective function and constraints that are linear in the inner product of these vectors. We write the inner product of vi and vj as vi · vj , or sometimes as viT vj . Below we give an example of a vector program. maximize or minimize
∑
cij (vi · vj )
(6.2)
i,j
subject to
∑
aijk (vi · vj ) = bk ,
∀k,
i,j
vi ∈ ℜn ,
i = 1, . . . , n.
We claim that in fact the SDP (6.1) and the vector program (6.2) are equivalent. This follows from Fact 6.2; in particular, it follows since a symmetric X is psd if and only if X = V T V for some matrix V . Given a solution to the SDP (6.1), we can take the solution X, compute in polynomial time a matrix V for which X = V T V (to within small error, which we again will ignore), and set vi to be the ith column of V . Then xij = viT vj = vi · vj , and the vi are a feasible solution of the same value to the vector program (6.2). Similarly, given a solution vi to the vector program, we construct a matrix V whose ith column is vi , and let X = V T V . Then X is symmetric and psd, with xij = vi · vj , so that X is a feasible solution of the same value for the SDP (6.1). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.2
Finding large cuts
6.2
143
Finding large cuts
In this section, we show how to use semideﬁnite programming to ﬁnd an improved approximation algorithm for the maximum cut problem, or MAX CUT problem, which we introduced in Section 5.1. Recall that for this problem, the input is an undirected graph G = (V, E), and nonnegative weights wij ≥ 0 for each edge (i, j) ∈ E. The goal is to partition the vertex set into two parts, U and W = V − U , so as to maximize the weight of the edges whose two endpoints are in diﬀerent parts, one in U and one in W . In Section 5.1, we gave a 21 approximation algorithm for the maximum cut problem. We will now use semideﬁnite programming to give a .878approximation algorithm for the problem in general graphs. We start by considering the following formulation of the maximum cut problem: maximize
1 ∑ wij (1 − yi yj ) 2
(6.3)
(i,j)∈E
yi ∈ {−1, +1} ,
subject to
i = 1, . . . , n.
We claim that if we can solve this formulation, then we can solve the MAX CUT problem. Lemma 6.3: The program (6.3) models the maximum cut problem. Proof. Consider the cut U = {i : yi = −1} and W = {i : yi = +1}. Note that if an edge (i, j) is in this cut, then yi yj = −1, while if the edge is not in the cut, yi yj = 1. Thus 1 ∑ wij (1 − yi yj ) 2 (i,j)∈E
gives the weight of all the edges in the cut. Hence ﬁnding the setting of the yi to ±1 that maximizes this sum gives the maximumweight cut. We can now consider the following vector programming relaxation of the program (6.3): maximize
1 ∑ wij (1 − vi · vj ) 2
(6.4)
(i,j)∈E
subject to
vi · vi = 1, vi ∈ ℜ , n
i = 1, . . . , n, i = 1, . . . , n.
This program is a relaxation of (6.3) since we can take any feasible solution y and produce a feasible solution to this program of the same value by setting vi = (yi , 0, 0, . . . , 0): clearly vi · vi = 1 and vi · vj = yi yj for this solution. Thus if ZV P is the value of an optimal solution to the vector program, it must be the case that ZV P ≥ OPT. We can solve (6.4) in polynomial time. We would now like to round the solution to obtain a nearoptimal cut. To do this, we introduce a form of randomized rounding suitable for vector programming. In particular, we pick a random vector r = (r1 , . . . , rn ) by drawing each component from N (0, 1), the normal distribution with mean 0 and variance 1. The normal distribution can be simulated by an algorithm that draws repeatedly from the uniform distribution on [0,1]. Then given a solution to (6.4), we iterate through all the vertices and put i ∈ U if vi · r ≥ 0 and i ∈ W otherwise. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
144
Randomized rounding of semideﬁnite programs
Figure 6.1: An illustration of a random hyperplane. Another way of looking at this algorithm is that we consider the hyperplane with normal r containing the origin. All vectors vi lie on the unit sphere, since vi · vi = 1 and they are unit vectors. The hyperplane with normal r containing the origin splits the sphere in half; all vertices in one half (the half such that vi · r ≥ 0) are put into U , and all vertices in the other half are put into W (see Figure 6.1). As we will see below, the vector r/∥r∥ is uniform over the unit sphere, so this is equivalent to randomly splitting the unit sphere in half. For this reason, this technique is sometimes called choosing a random hyperplane. To prove that this is a good approximation algorithm, we need the following facts. Fact 6.4: The normalization of r, r/r, is uniformly distributed over the ndimensional unit sphere. Fact 6.5: The projections of r onto two unit vectors e1 and e2 are independent and are normally distributed with mean 0 and variance 1 iﬀ e1 and e2 are orthogonal. Corollary 6.6: Let r′ be the projection of r onto a twodimensional plane. Then the normalization of r′ , r′ /r′ , is uniformly distributed on a unit circle in the plane. We now begin the proof that choosing a random hyperplane gives a .878approximation algorithm for the problem. We will need the following two lemmas. Lemma 6.7: The probability that edge (i, j) is in the cut is
1 π
arccos(vi · vj ).
Proof. Let r′ be the projection of r onto the plane deﬁned by vi and vj . If r = r′ + r′′ , then r′′ is orthogonal to both vi and vj , and vi · r = vi · (r′ + r′′ ) = vi · r′ . Similarly vj · r = vj · r′ . Consider Figure 6.2, where line AC is perpendicular to the vector vi and line BD is perpendicular to the vector vj . By Corollary 6.6, the vector r′ with its tail at the origin O is oriented with respect to the vector vi by an angle α chosen uniformly from [0, 2π). If r′ is to the right of the line AC, vi will have a nonnegative inner product with r′ , otherwise not. If r′ is above the line BD, vj will have nonnegative inner product with r′ , otherwise not. Thus we have i ∈ W and j ∈ U if r′ is in the sector AB and i ∈ U and j ∈ W if r′ is in the sector CD. If the angle formed by vi and vj is θ radians, then the angles ∠AOB and ∠COD are also θ radians. Hence the fraction of values for which α, the angle of r′ , corresponds to the event Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.2
Finding large cuts
145
A
vj
B
θ
θ
O
vi
θ D C
Figure 6.2: Figure for proof of Lemma 6.7. in which (i, j) is in the cut is 2θ/2π. Thus the probability that (i, j) is in the cut is πθ . We know that vi · vj = ∥vi ∥∥vj ∥ cos θ. Since vi and vj are both unit length vectors, we have that θ = arccos(vi · vj ), which completes the proof of the lemma. Lemma 6.8: For x ∈ [−1, 1], 1 1 arccos(x) ≥ 0.878 · (1 − x). π 2 Proof. The proof follows from simple calculus. See Figure 6.3 for an illustration. Theorem 6.9: Rounding the vector program (6.4) by choosing a random hyperplane is a .878approximation algorithm for the maximum cut problem. Proof. Let Xij be a random variable for edge (i, j) such that Xij = 1 if (i, j) is in the cut given by the algorithm, ∑ and 0 otherwise. Let W be a random variable which gives the weight of the cut; that is, W = (i,j)∈E wij Xij . Then by Lemma 6.7, E[W ] =
∑
wij · Pr[edge (i, j) is in cut] =
(i,j)∈E
(i,j)∈E
By Lemma 6.8, we can bound each term E[W ] ≥ 0.878 ·
∑
1 π
wij ·
1 arccos(vi · vj ). π
arccos(vi · vj ) below by 0.878 · 12 (1 − vi · vj ), so that
1 ∑ wij (1 − vi · vj ) = 0.878 · ZV P ≥ 0.878 · OPT . 2 (i,j)∈E
We know that ZV P ≥ OPT. The proof of the theorem above shows that there is a cut of value 1 at least 0.878 · ZV P , so that OPT ≥ 0.878 · ZV P . Thus we have that OPT ≤ ZV P ≤ 0.878 OPT. It has been shown that there are graphs for which the upper inequality is met with equality. This implies that we can get no better performance guarantee for the maximum cut problem Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
146
Randomized rounding of semideﬁnite programs
1 (1/pi) arccos(x) 1/2 (1x)
0.8
0.6
0.4
0.2
0 1
0.5
0
0.5
1
2 ratio .878
1.5
1
0.5
0 1
0.5
0
0.5
1
Figure 6.3: Illustration of Lemma 6.8. The upper ﬁgure shows plots of the functions 1 1 π arccos(x) and 2 (1 − x). The lower ﬁgure shows a plot of the ratio of the two functions.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.3
Approximating quadratic programs
147
by using ZV P as an upper bound on OPT. Currently, .878 is the best performance guarantee known for the maximum cut problem problem. The following theorems show that this is either close to, or exactly, the best performance guarantee that is likely attainable. Theorem 6.10: If there is an αapproximation algorithm for the maximum cut problem with α > 16 17 ≈ 0.941, then P = NP. Theorem 6.11: Given the unique games conjecture there is no αapproximation algorithm for the maximum cut problem with constant α > min
1 π
−1≤x≤1
arccos(x) ≥ .878 − x)
1 2 (1
unless P = NP. We sketch the proof of the second theorem in Section 16.5. So far we have only discussed a randomized algorithm for the maximum cut problem. It is possible to derandomize the algorithm by using a sophisticated application of the method of conditional expectations that iteratively determines the various coordinates of the random vector. The derandomization incurs a loss in the performance guarantee that can be made as small as desired (by increasing the running time).
6.3
Approximating quadratic programs
We can extend the algorithm above for the maximum cut problem to the following more general problem. Suppose we wish to approximate the quadratic program below: ∑ maximize aij xi xj (6.5) 1≤i,j≤n
xi ∈ {−1, +1} ,
subject to
i = 1, . . . , n.
We need to be slightly careful in this case since as stated it is possible that the value of an optimal solution is negative (for instance, if the values of aii are negative and all other aij are zero). Thus far we have only been considering problems in which all feasible solutions have nonnegative value so that the deﬁnition of an αapproximation algorithm makes sense. To see that the deﬁnition might not make sense in the case of negative solution values, suppose we have an αapproximation algorithm for a maximization problem with α < 1 and suppose we have a problem instance in which OPT is negative. Then the approximation algorithm guarantees the value of our solution is at least α · OPT, which means that the value of our solution will be greater than that of OPT. In order to get around this diﬃculty, in this case we will restrict the objective function matrix A = (aij ) in (6.5) to itself be positive semideﬁnite. Observe then that for any feasible solution x, the value of the objective function will be xT Ax and will be nonnegative by the deﬁnition of positive semideﬁnite matrices. As in the case of the maximum cut problem, we can then have the following vector programming relaxation: ∑ maximize aij (vi · vj ) (6.6) 1≤i,j≤n
subject to
vi · vi = 1, vi ∈ ℜ , n
i = 1, . . . , n, i = 1, . . . , n.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
148
Randomized rounding of semideﬁnite programs
Let ZV P be the value of an optimal solution for this vector program. By the same argument as in the previous section, ZV P ≥ OPT. We can also use the same algorithm as we did for the maximum cut problem. We solve the vector program (6.6) in polynomial time and obtain vectors vi . We choose a random hyperplane with normal r, and generate a solution x ¯ for the quadratic program (6.5) by setting x ¯i = 1 if 2 r · vi ≥ 0 and x ¯i = −1 otherwise. We will show below that this gives a π approximation algorithm for the quadratic program (6.5). Lemma 6.12: E[¯ xi x ¯j ] =
2 arcsin(vi · vj ) π
Proof. Recall from Lemma 6.7 that the probability vi and vj will be on diﬀerent sides of the random hyperplane is π1 arccos(vi · vj ). Thus the probability that x ¯i and x ¯j have diﬀerent values 1 ¯i and x ¯j have diﬀerent values, then their product must be is π arccos(vi · vj ). Observe that if x −1, while if they have the same value, their product must be 1. Thus the probability that the product is 1 must be 1 − π1 arccos(vi · vj ). Hence E[¯ xi x ¯j ] = Pr[¯ xi x ¯j = 1] − Pr[¯ xi x ¯j = −1] ( ) ( ) 1 1 = 1 − arccos(vi · vj ) − arccos(vi · vj ) π π 2 = 1 − arccos(vi · vj ). π Using arcsin(x) + arccos(x) = π2 , we get ] 2 2 [π − arcsin(vi · vj ) = arcsin(vi · vj ). E[¯ xi x ¯j ] = 1 − π 2 π We would like to make the same argument as we did for the maximum cut problem in Theorem 6.9, but there is a diﬃculty with the analysis. Let α = min
−1≤x≤1
2 π
arcsin(x) . x
Then we would like to argue that the expected value of the solution is ∑ ∑ E aij x ¯i x ¯j = aij E[¯ xi x ¯j ] i,j
i,j
2∑ aij arcsin(vi · vj ) π i,j ∑ ≥ α aij (vi · vj ) =
i,j
≥ α · OPT by the same reasoning as in Theorem 6.9. However, the penultimate inequality is not necessarily correct since it may be the case that some of the aij are negative. We are assuming that the inequality holds on a termbyterm basis, and this is not true when some of the aij < 0. Thus in order to analyze this algorithm, we will have to use a global analysis, rather than a termbyterm analysis. To do so, we will need the following fact, called the Schur product theorem, and the subsequent two corollaries. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.3
Approximating quadratic programs
149
Fact 6.13 (Schur product theorem): For matrices A = (aij ) and B = (bij ), deﬁne A ◦ B = (aij bij ). Then if A ≽ 0 and B ≽ 0, then A ◦ B ≽ 0. ∑ Corollary 6.14: If A ≽ 0 and B ≽ 0, then i,j aij bij ≥ 0. Proof. By Fact 6.13, we know that A ◦ B ≽ 0, so that for the vector ⃗1 of all ones, ⃗1T (A ◦ B)⃗1 ≥ 0, by the deﬁnition of psd matrices.
∑ i,j
aij bij =
Corollary 6.15: If X ≽ 0, xij  ≤ 1 for all i, j, and Z = (zij ) such that zij = arcsin(xij ) − xij , then Z ≽ 0. Proof. We recall that the Taylor series expansion for arcsin x around zero is arcsin x = x +
1·3 5 1 · 3 · 5 · · · (2n + 1) 1 3 x + x + ··· + x2n+1 + · · · , 2·3 2·4·5 2 · 4 · 6 · · · 2n · (2n + 1)
and it converges for x ≤ 1. Since the matrix Z = (zij ) where zij = arcsin(xij ) − xij , we can express it as Z=
1·3 1 ((X ◦ X) ◦ X) + (((((X ◦ X) ◦ X) ◦ X) ◦ X) + · · · . 2·3 2·4·5
By Fact 6.13, because X ≽ 0, each term on the righthand side is a positive semideﬁnite matrix, and therefore their sum will be also. We can now show the following theorem. Theorem 6.16: Rounding the vector program (6.6) by choosing a random hyperplane is a π2 approximation algorithm for the quadratic program (6.5) when the objective function matrix A is positive semideﬁnite. Proof. We want to show that ∑ 2∑ 2 E aij x ¯i x ¯j ≥ aij (vi · vj ) ≥ · OPT . π π i,j
i,j
∑ 2∑ E aij x ¯i x ¯j = aij arcsin(vi · vj ). π
We know that
i,j
Thus we need to show that 2∑ π
i,j
aij arcsin(vi · vj ) −
i,j
2∑ aij (vi · vj ) ≥ 0. π i,j
By setting xij = vi · vj and letting θij denote the angle between the two vectors, we obtain that X = (xij ) ≽ 0 and xij  ≤ 1 since vi · vj  = ∥vi ∥∥vj ∥ cos θij  =  cos θij  ≤ 1. Thus we want to show that 2∑ aij (arcsin(xij ) − xij ) ≥ 0. π i,j
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
150
Randomized rounding of semideﬁnite programs
If we set zij = arcsin(xij ) − xij , then it is equivalent to show that 2∑ aij zij ≥ 0. π i,j
This follows since Z = (zij ) ≽ 0 by Corollary 6.15 and thus since A ≽ 0.
6.4
∑ i,j
aij zij ≥ 0 by Corollary 6.14
Finding a correlation clustering
In this section, we show that semideﬁnite programming can be used to obtain a good correlation clustering in an undirected graph. In this problem we are given an undirected graph in which + − each edge (i, j) ∈ E is given two nonnegative weights, wij ≥ 0 and wij ≥ 0. The goal is to cluster the vertices into sets of similar vertices; the degree to which i and j are similar is given + − by wij and the degree to which they are diﬀerent is given by wij . We represent a clustering of the vertices by a partition S of the vertex set into nonempty subsets. Let δ(S) be the set of edges that have endpoints in diﬀerent sets of the partition, and let E(S) be the set of edges that have both endpoints in the same part of the partition. Then the goal of the problem is to ﬁnd a partition S that maximizes the total w+ weight of edges inside the sets of the partition plus the total w− weight of edges between sets of the partition; in other words, we ﬁnd S to maximize ∑ ∑ + − wij + wij . (i,j)∈E(S)
(i,j)∈δ(S)
Observe that it is easy to get a 21 approximation algorithm for this problem. If∑we put all the + vertices into a single cluster (that is, S = {V }), then the value of this solution is (i,j)∈E wij . If we∑ make each vertex its own cluster (that is, S = {{i} : i ∈ V }), then the value of this solution ∑ − + − is (i,j)∈E wij . Since OPT ≤ (i,j)∈E (wij + wij ), at least one of these two solutions has a value 1 of at least 2 OPT. We now show that we can obtain a 34 approximation algorithm by using semideﬁnite programming. Here we model the problem as follows. Let ek be the kth unit vector; that is, it has a one in the kth coordinate and zeros elsewhere. We will have a vector xi for each vertex i ∈ V ; we set xi = ek if i is in the kth cluster. The model of the problem then becomes ) ∑ ( + − maximize wij (xi · xj ) + wij (1 − xi · xj ) (i,j)∈E
xi ∈ {e1 , . . . , en } ,
subject to
∀i,
since ek · el = 1 if k = l and 0 otherwise. We can then relax this model to the following vector program: ) ∑ ( + − maximize wij (vi · vj ) + wij (1 − vi · vj ) (6.7) (i,j)∈E
subject to
vi · vi = 1,
∀i,
vi · vj ≥ 0,
∀i, j,
vi ∈ ℜ , n
∀i.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.4
Finding a correlation clustering
151
Let ZCC be the value of an optimal solution for the vector program. Observe that the vector program is a relaxation, since any feasible solution to the model above is feasible for the vector program and has the same value. Thus ZCC ≥ OPT . In the previous two sections, we chose a random hyperplane to partition the vectors into two sets; in the case of the maximum cut problem, this gave the two sides of the cut, and in the case of quadratic programming this gave the variables of value +1 and −1. In this case, we will choose two random hyperplanes, with two independent random vectors r1 and r2 as their normals. This partitions the vertices into 22 = 4 sets: in particular, we partition the vertices into the sets R1 = {i ∈ V : r1 · vi ≥ 0, r2 · vi ≥ 0} R2 = {i ∈ V : r1 · vi ≥ 0, r2 · vi < 0} R3 = {i ∈ V : r1 · vi < 0, r2 · vi ≥ 0} R4 = {i ∈ V : r1 · vi < 0, r2 · vi < 0} . We will show that the solution consisting of these four clusters, with S = {R1 , R2 , R3 , R4 }, comes within a factor of 34 of the optimal value. We ﬁrst need the following lemma. Lemma 6.17: For x ∈ [0, 1],
and
(1 −
1 π
arccos(x))2 ≥ .75 x
1 − (1 − π1 arccos(x))2 ≥ .75. (1 − x)
Proof. These statements follow by simple calculus. See Figure 6.4 for an illustration. We can now give the main theorem. Theorem 6.18: Rounding the vector program (6.7) by using two random hyperplanes as above gives a 34 approximation algorithm for the correlation clustering problem. Proof. Let Xij be a random variable which is 1 if vertices i and j end up in the same cluster, and is 0 otherwise. Note that the probability that a single random hyperplane has the vectors vi and vj on diﬀerent sides of the hyperplane is π1 arccos(vi ·vj ) by Lemma 6.7. Then the probability that vi and vj are on the same side of a single random hyperplane is 1− π1 arccos(vi ·vj ). Furthermore, the probability that the vectors are both on the same sides of the two random hyperplanes deﬁned by r1 and r2 is (1 − π1 arccos(vi · vj ))2 , since r1 and r2 are chosen independently. Thus E[Xij ] = (1 − π1 arccos(vi · vj ))2 . Let W be a random variable denoting the weight of the partition. Observe that ) ∑ ( + − W = wij Xij + wij (1 − Xij ) . (i,j)∈E
Thus E[W ] =
∑ ( (i,j)∈E
=
∑ (i,j)∈E
) + − wij E[Xij ] + wij (1 − E[Xij ])
[
( + wij
( )2 ( )2 )] 1 1 − 1 − arccos(vi · vj ) + wij 1 − 1 − arccos(vi · vj ) . π π
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
152
Randomized rounding of semideﬁnite programs
2 (1(1/pi) arccos(x))^2/x .75
1.5
1
0.5
0 0
0.2
0.4
0.6
0.8
1
2 (1(1(1/pi)arccos(x))^2)/(1x) .75
1.5
1
0.5
0 0
0.2
0.4
0.6
0.8
1
Figure 6.4: Illustration of Lemma 6.17. The upper ﬁgure shows a plot of [ ] [ ] 1 1 2 2 (1 − π arccos(x)) /x and the lower ﬁgure shows a plot of 1 − (1 − π arccos(x)) /(1− x).
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.5
Coloring 3colorable graphs
153
We now want to use Lemma 6.17 to bound each term in the sum from below; we can do so because the constraints of the vector program imply that vi · vj ∈ [0, 1]. Thus ) ∑ ( + − E[W ] ≥ .75 wij (vi · vj ) + wij (1 − vi · vj ) = .75 · ZCC ≥ .75 · OPT . (i,j)∈E
6.5
Coloring 3colorable graphs
In Section 5.12, we saw that with high probability we can 3color a δdense 3colorable graph. The situation for arbitrary 3colorable graphs is much worse, however. We will give a quite √ simple algorithm that colors a 3colorable graph G = (V, E) with O( n) colors, where n = V . ˜ 0.387 ) colors Then by using semideﬁnite programming, we will obtain an algorithm that uses O(n ˜ is deﬁned below. where O ˜ (n)) if there exists some constant c ≥ 0 and some n0 Deﬁnition 6.19: A function g(n) = O(f such that for all n ≥ n0 , g(n) = O(f (n) logc n). ˜ 0.387 ) colors. Graph coloring The best known algorithm does not use much fewer than O(n is one of the most diﬃcult problems to approximate. Some coloring problems are quite simple: it is known that a 2colorable graph can be 2colored in polynomial time, and that an arbitrary graph with maximum degree ∆ can be (∆ + 1)colored in polynomial time. We leave these results as exercises (Exercise 6.4). Given these results, it is quite simple to give an algorithm that colors a 3colorable graph √ √ with O( n) colors. As long as there exists a vertex in the graph with degree at least n, we pick three new colors, color the vertex with one of the new colors, and use the 2coloring algorithm to 2color the neighbors of the vertex with the other two new colors; we know that we can do this because the graph is 3colorable. We remove all these vertices from the graph √ and repeat. When we have no vertices in the graph of degree at least n, we use the algorithm √ that colors a graph with ∆ + 1 colors to color the remaining graph with n new colors. We can prove the following. √ Theorem 6.20: The algorithm above colors any 3colorable graph with at most 4 n colors. √ Proof. Each time the algorithm ﬁnds a vertex of degree at least n, we use three new colors. √ √ This can happen at most n/ n times, since we remove at least n vertices from the graph each √ time we do this. Hence this loop uses at most 3 n colors. The ﬁnal step uses an additional √ n colors. We now turn to semideﬁnite programming to help improve our coloring algorithms. We will use the following vector program in which we have a vector vi for each i ∈ V : minimize
λ
(6.8)
subject to vi · vj ≤ λ, vi · vi = 1, vi ∈ ℜ , n
∀(i, j) ∈ E, ∀i ∈ V, ∀i ∈ V.
The lemma below suggests why the vector program will be useful in deriving an algorithm for coloring 3colorable graphs. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
154
Randomized rounding of semideﬁnite programs Blue
Red
Green
Figure 6.5: Proof of Lemma 6.21. Lemma 6.21: For any 3colorable graph, there is a feasible solution to (6.8) with λ ≤ − 12 . Proof. Consider an equilateral triangle, and associate the unit vectors for all the vertices with the three diﬀerent colors with the three diﬀerent vertices of the triangle (see Figure 6.5). Note that the angle between any two vectors of the same color is 0, while the angle between vectors of diﬀerent colors is 2π/3. Then for vi , vj such that (i, j) ∈ E, we have ( vi · vj = ∥vi ∥∥vj ∥ cos
2π 3
)
1 =− . 2
Since we have given a feasible solution to the vector program with λ = −1/2, it must be the case that for the optimal solution λ ≤ −1/2. Note that the proof of the lemma has actually shown the following corollary; this will be useful later on. Corollary 6.22: For any 3colorable graph, there is a feasible solution to (6.8) such that vi ·vj = −1/2 for all edges (i, j) ∈ E. To achieve our result, we will show how to obtain a randomized algorithm that produces a semicoloring. A semicoloring is a coloring of nodes such that at most n/4 edges have endpoints with the same color. This implies that at least n/2 vertices are colored such that any edge between them has endpoints that are colored diﬀerently. We claim that an algorithm for producing a semicoloring is suﬃcient, for if we can semicolor a graph with k colors, then we can color the entire graph with k log n colors in the following way. We ﬁrst semicolor the graph with k colors, and take the vertices that are colored correctly. We then semicolor the vertices left over (no more than n/2) with k new colors, take the vertices that are colored correctly, and repeat. This takes log n iterations, after which the entire graph is colored correctly with k log n colors. Now we give the randomized algorithm for producing a semicoloring. The basic idea is similar to that used in the correlation clustering algorithm in Section 6.4. We solve the vector program (6.8), and choose t = 2 + log3 ∆ random vectors r1 , . . . , rt , where ∆ is the maximum degree of any vertex in the graph. The t random vectors deﬁne 2t diﬀerent regions into which the vectors vi can fall: one region for each distinct possibility of whether rj · vi ≥ 0 or rj · vi < 0 for all j = 1, . . . , t. We then color the vectors in each region with a distinct color. Theorem 6.23: This coloring algorithm produces a semicoloring of 4∆log3 2 colors with probability at least 1/2. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.5
Coloring 3colorable graphs
155
Proof. Since we used 2t colors for t = 2 + log3 ∆, we use 4 · 2log3 ∆ = 4∆log3 2 colors. We now need to show that this produces a semicoloring with probability at least 1/2. First, we consider the probability that vertices i and j get the same color for a given edge (i, j). This probability is the probability that both i and j fall into the same region; that is, the probability that none of the t random hyperplanes separate i and j. Note that by Lemma 6.7, the probability that a single random hyperplane separates i and j is π1 arccos(vi · vj ). Therefore the probability that t independently chosen hyperplanes fail to separate i and j is (1 − π1 arccos(vi · vj ))t . Thus ( Pr[i and j get the same color for edge (i, j)] =
1−
)t ( )t 1 1 arccos(vi · vj ) ≤ 1 − arccos(λ) , π π
where the last inequality follows from the inequalities of the vector program (6.8) and since arccos is a nonincreasing function. Then by Lemma 6.21, (
)t ( )t 1 1 1 − arccos(λ) ≤ 1 − arccos(−1/2) . π π
Finally, using some simple algebra and the deﬁnition of t, (
1 1 − arccos(−1/2) π
)t
( =
1 2π 1− π 3
)t
( )t 1 1 . = ≤ 3 9∆
Therefore, 1 . 9∆ If m denotes the number of edges in the graph, then m ≤ n∆/2. Thus the expected number of edges which have both endpoints colored the same is no more than m/9∆, which is at most n/18. Let X be a random variable denoting the number of edges which have both endpoints colored the same. By Markov’s inequality (Lemma 5.25), the probability that there are more than n/4 edges which have both endpoints colored the same is at most Pr[i and j get the same color for edge (i, j)] ≤
Pr[X ≥ n/4] ≤
n/18 1 E[X] ≤ ≤ . n/4 n/4 2
If we use n as a bound on the maximum degree ∆, then we obtain an algorithm that ˜ log3 2 ) colors. Since produces a semicoloring with O(nlog3 2 ) colors and thus a coloring with O(n log3 2 ≈ .631, this is worse than the algorithm we presented at the beginning of the section that uses O(n1/2 ) colors. However, we can use some ideas from that algorithm to do better. Let σ be some parameter we will choose later. As long as there is a vertex in the graph of degree at least σ, we pick three new colors, color the vertex with one of the new colors, and use the 2coloring algorithm to 2color the neighbors of the vertex with the other two new colors; we know that we can do this because the graph is 3colorable. We remove all these vertices from the graph and repeat. When we have no vertices in the graph of degree at least σ, we use the algorithm above to semicolor the remaining graph with O(σ log3 2 ) new colors. We can now show the following. Theorem 6.24: The algorithm above semicolors a 3colorable graph with O(n0.387 ) colors with probability at least 1/2. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
156
Randomized rounding of semideﬁnite programs
Proof. The ﬁrst part of this algorithm uses 3n/σ colors in total, since we remove at least σ vertices from the graph each time. To balance the two parts of the algorithm, we set σ such that nσ = σ log3 2 , which gives σ = nlog6 3 , or σ ≈ n0.613 . Thus both parts use O(n0.387 ) colors, which gives the theorem. ˜ 0.387 ) colors. From the theorem we get an overall algorithm that colors a graph with O(n In Section 13.2, we will show how √ to use semideﬁnite programming to obtain an algorithm ˜ 1/3 ln ∆) colors. Using the same ideas as above, this can be that 3colors a graph with O(∆ ˜ 1/4 ) colors. converted into an algorithm that colors using O(n
Exercises 6.1 As with linear programs, semideﬁnite programs have duals. The dual of the MAX CUT SDP (6.4) is: minimize
1∑ 1∑ wij + γi 2 4 i<j
subject to
i
W + diag(γ) ≽ 0,
where the matrix W is the symmetric matrix of the edge weights wij and diag(γ) is the matrix of zeroes with γi as the ith entry on the diagonal. Show that the value of any feasible solution for this dual is an upper bound on the cost of any cut. 6.2 Semideﬁnite programming can also be used to give improved approximation algorithms for the maximum satisﬁability problem. First we start with the MAX 2SAT problem, in which every clause has at most two literals. (a) As in the case of the maximum cut problem, we’d like to express the MAX 2SAT problem as a “integer quadratic program” in which the only constraints are yi ∈ {−1, 1} and the objective function is quadratic in the yi . Show that the MAX 2SAT problem can be expressed this way. (Hint: it may help to introduce a variable y0 which indicates whether the value −1 or 1 is “TRUE”). (b) Derive a .878approximation algorithm for the MAX 2SAT problem. (c) Use this .878approximation algorithm for MAX 2SAT to derive a ( 34 +ϵ)approximation algorithm for the maximum satisﬁability problem, for some ϵ > 0. How large an ϵ can you get? 6.3 Given a MAX 2SAT instance as deﬁned in Exercise 6.2, prove that it possible to decide in polynomial time whether all the clauses can be satisﬁed or not. 6.4 Give polynomialtime algorithms for the following: (a) Coloring a 2colorable graph with 2 colors. (b) Coloring a graph of maximum degree ∆ with ∆ + 1 colors. 6.5 An important quantity in combinatorial optimization is called the Lov´ asz Theta Function. The theta function is deﬁned on undirected graphs G = (V, E). One of its many deﬁnitions Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.5
Coloring 3colorable graphs
157
is given below as a semideﬁnite program: ∑ ¯ = maximize ϑ(G) bij i,j
∑
subject to
bii = 1,
i
bij = 0, B = (bij ) ≽ 0,
∀i ̸= j, (i, j) ∈ / E, B symmetric.
¯ ≤ χ(G), where ω(G) is the size of the largest clique in Lov´asz showed that ω(G) ≤ ϑ(G) G and χ(G) is the minimum number of colors needed to color G. ¯ (a) Show that ω(G) ≤ ϑ(G). (b) The following is a small variation in the vector program we used for graph coloring: minimize
α
subject to vi · vj = α, vi · vi = 1,
∀(i, j) ∈ E, ∀i ∈ V,
vi ∈ ℜ , n
∀i ∈ V.
Its dual is maximize −
∑
u i · ui
i
subject to
∑
ui · uj ≥ 1,
i̸=j
ui · uj = 0, ui ∈ ℜ
n
∀(i, j) ∈ / E, i ̸= j, ∀i ∈ V.
¯ Show that the value of the dual is 1/(1 − ϑ(G)). By strong duality, this is also the value of the primal; however, see the chapter notes for a discussion of conditions under which strong duality holds. The value of this vector program is sometimes called the strict vector chromatic number of the graph, and the value of original vector programming relaxation (6.8) is the vector chromatic number of the graph. 6.6 Recall the maximum directed cut problem from Exercises 5.3 and 5.6: we are given as input a directed graph G = (V, A), with a nonnegative weight wij ≥ 0 for all arcs (i, j) ∈ A. The goal is to partition V into two sets U and W = V − U so as to maximize the total weight of the arcs going from U to W (that is, arcs (i, j) with i ∈ U and j ∈ W ). (a) As in the case of the maximum cut problem, we’d like to express the maximum directed cut problem as an integer quadratic program in which the only constraints are yi ∈ {−1, 1} and the objective function is quadratic in yi . Show that the maximum directed cut problem can be expressed in this way. (Hint: As is the case for MAX 2SAT in Exercise 6.2, it may help to introduce a variable y0 which indicates whether the value −1 or 1 means that yi is in the set U ). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
158
Randomized rounding of semideﬁnite programs
(b) Find an αapproximation algorithm for the maximum directed cut problem using a vector programming relaxation of the integer quadratic program above. Find the best value of α that you can. 6.7 Recall the MAX 2SAT problem from Exercise 6.2 above. We consider a variant, called MAX E2SAT, in which every clause has exactly two literals in it; that is, there are no unit clauses. We say that a MAX E2SAT instance is balanced if for each i, the weight of clauses in which xi appears is equal to the weight of the clauses in which x ¯i appears. Give a βapproximation algorithm for balanced MAX E2SAT instances, where β=
min
1 2
+
x:−1≤x≤1
1 2π arccos x 3 1 4 − 4x
≈ .94394.
6.8 Consider again the maximum directed cut problem given in Exercise 6.6. We say that an instance of the directed cut problem is balanced if for each vertex i ∈ V , the total weight of arcs entering i is equal to the total weight of arcs leaving i. Give an αapproximation algorithm for balanced maximum directed cut instances, where α is the same performance guarantee as for the maximum cut problem; that is, α=
min
x:−1≤x≤1
1 π
arccos(x) . − x)
1 2 (1
Chapter Notes Strang [273, 274] provides introductions to linear algebra that are useful in the context of our discussion of semideﬁnite programming and various operations on vectors and matrices. A 1979 paper of Lov´asz [219] gives an early application of SDP to combinatorial optimization with his ϑnumber for graphs (see Exercise 6.5). The algorithmic implications of the ϑnumber were highlighted in the work of Gr¨otschel, Lov´asz, and Schrijver [144]; they showed that the ellipsoid method could be used to solve the associated semideﬁnite program for the ϑnumber, and that in general the ellipsoid method could be used to solve convex programs in polynomial time given a polynomialtime separation oracle. Alizadeh [5] and Nesterov and Nemirovskii [236] showed that polynomialtime interiorpoint methods for linear programming could be extended to SDP. A good overview of SDP can be found in the edited volume of Wolkowicz, Saigal, and Vandenberghe [290]. Unlike linear programs, the most general case of semideﬁnite programs is not solvable in polynomial time without additional assumptions. There are examples of SDPs in which the coeﬃcients of all variables are either 1 or 2, but whose optimal value is doubly exponential in the number of variables (see [5, Section 3.3]), and so no polynomialtime algorithm is possible. Even when SDPs are solvable in polynomial time, they are only solvable to within an additive error of ϵ; this is in part due to the fact that exact solutions can be irrational, and thus not expressible in polynomial space. To show that an SDP is solvable in polynomial time to within an additive error of ϵ, it is suﬃcient for the feasible region to be nonempty and to be contained in a polynomiallysize ball centered on the origin. These conditions are met for the problems we discuss. Weak duality always holds for semideﬁnite programs. If there are points in the interior of the feasible region of the primal and dual (called the “Slater conditions”), strong duality also holds. As a quick way to see the solvability of semideﬁnite programs in polynomial time, we observe that the constraint X ≽ 0 is equivalent to the inﬁnite family of constraints y T Xy ≥ 0 for all y ∈ ℜn . Suppose we compute the minimum eigenvalue λ and corresponding Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
6.5
Coloring 3colorable graphs
159
eigenvector v of the matrix X. If λ ≥ 0, then X is positive semideﬁnite, whereas if λ < 0, then v T (Xv) = v T (λv) = λv T v < 0 gives a violated constraint. Thus we have a separation oracle and can apply the ellipsoid method to ﬁnd an approximate solution assuming the initial feasible region can be bounded by a polynomiallysized ellipsoid and assuming that we can carry out the eigenvalue and eigenvector computation to reasonable precision in polynomial time. Gr¨otschel, Lov´asz, and Schrijver [144] avoid the issue of computing an eigenvector in polynomial time by giving a polynomialtime separation oracle that computes a basis of the columns of X, and then computes determinants to check whether X is positive semideﬁnite and to return a constraint y T Xy < 0 if not. The SDPbased algorithm of Section 6.2 for the maximum cut problem is due to Goemans and Williamson [139]; they gave the ﬁrst use of semideﬁnite programming for approximation algorithms. Knuth [200, Section 3.4.1C] gives algorithms for sampling from the normal distribution via samples from the uniform [0,1] distribution. Fact 6.4 is from Knuth [200, p. 135136] and Fact 6.5 is a paraphrase of Theorem IV.16.3 of R´enyi [251]. Feige and Schechtman [109] give 1 OPT. Theorem 6.10 is due to H˚ astad [159]. Theorem 6.11 is due graphs for which ZV P = 0.878 to Khot, Kindler, Mossel, and O’Donnell [193] together with a result of Mossel, O’Donnell, and Oleszkiewicz [227]. The derandomization of the maximum cut algorithm is due to Mahajan and Ramesh [222]. This derandomization technique works for many of the randomized algorithms in this chapter that use random hyperplanes. Subsequent to the work of Goemans and Williamson, Nesterov [235] gave the algorithm for quadratic programs found in Section 6.3, Swamy [277] gave the algorithm of Section 6.4 on correlation clustering, and Karger, Motwani, and Sudan [182] gave the SDPbased algorithm √ for coloring 3colorable graphs in Section 6.5. The O( n)approximation algorithm for coloring 3colorable graphs given at the beginning of Section 6.5 is due to Wigderson [286]. Fact 6.13 is known as the Schur product theorem; see, for example, Theorem 7.5.3 of Horn and Johnson [171]. Exercises 6.1, 6.2, and Exercise 6.6 are from Goemans and Williamson [139]. Exercise 6.3 has been shown by Even, Itai, and Shamir [104], who also point to previous work on the 2SAT problem. Exercise 6.5 is due to Tardos and Williamson as cited in [182]. Exercises 6.7 and 6.8 on balanced MAX 2SAT and balanced maximum directed cut problem instances are due to Khot, Kindler, Mossel, and O’Donnell [193].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
160
Randomized rounding of semideﬁnite programs
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 7
The primaldual method
We introduced the primaldual method in Section 1.5, and showed how it gave an approximation algorithm for the set cover problem. Although there it did not give a better performance guarantee than various LP rounding algorithms, we observed that in practice the primaldual method gives much faster algorithms than those that require solving a linear program. In this chapter, we will cover primaldual algorithms in more depth. We begin by reviewing the primaldual algorithm for the set cover problem from Section 1.5. We then apply the primaldual method to a number of problems, gradually developing a number of principles in deciding how to apply the technique so as to get good performance guarantees. In discussing the feedback vertex set problem in Section 7.2, we see that it is sometimes useful to focus on increasing particular dual variables that correspond to small or minimal constraints not satisﬁed by the current primal solution. In Section 7.3, we discuss the shortest st path problem, and see that to obtain a good performance guarantee it is sometimes necessary to remove unneeded elements in the primal solution returned by the algorithm. In Section 7.4 we introduce the generalized Steiner tree problem (also known as the Steiner forest problem), and show that to obtain a good performance guarantee it can be helpful to increase multiple dual variables at the same time. In Section 7.5, we see that it can be useful to consider alternate integer programming formulations in order to obtain improved performance guarantees. We conclude the chapter with an application of the primaldual method to the uncapacitated facility location problem, and an extension of this algorithm to the related kmedian problem. For the latter problem, we use a technique called Lagrangean relaxation to obtain the appropriate relaxation for the approximation algorithm.
7.1
The set cover problem: a review
We begin by reviewing the primaldual algorithm and its analysis for the set cover problem from Section 1.5. Recall that in the set cover problem, we are given as input a ground set of elements E = {e1 , . . . , en }, some subsets of those elements S1 , S2 , . . . , Sm ⊆ E, and a nonnegative weight wj for each subset Sj . The goal is to ﬁnd a minimumweight collection of subsets that ∑ ∪ covers all of E; that is, we wish to ﬁnd an I ⊆ {1, . . . , m} that minimizes j∈I wj subject to j∈I Sj = E. We observed in Section 1.5 that the set cover problem can be modelled as the following 161
162
The primaldual method
y←0 I←∅ ∪ while there exists ei ∈ / j∈I Sj do ∑ Increase the dual variable yi until there is some ℓ such that j:ej ∈Sℓ yj = wℓ I ← I ∪ {ℓ} return I Algorithm 7.1: Primaldual algorithm for the set cover problem. integer program: minimize
m ∑
wj xj
(7.1)
j=1
∑
subject to
xj ≥ 1,
i = 1, . . . , n,
(7.2)
xj ∈ {0, 1}
j = 1, . . . , m.
(7.3)
j:ei ∈Sj
If we relax the integer program to a linear program by replacing the constraints xj ∈ {0, 1} with xj ≥ 0, and take the dual, we obtain maximize
n ∑
yi
i=1
subject to
∑
yi ≤ wj ,
j = 1, . . . , m,
yi ≥ 0,
i = 1, . . . , n.
i:ei ∈Sj
We then gave the following algorithm, which we repeat in Algorithm 7.1. We begin with the dual solution y = 0; this is a feasible solution since wj ≥ 0 for all j. We also have an infeasible primal solution I = ∅. As long as there is some element ei not covered by I, we look at all the sets Sj that contain ei , and consider the amount by which we can increase the dual variable associated )with ei and still maintain dual feasibility. This amount is ( yi ∑ ϵ = minj:ei ∈Sj wj − k:ek ∈Sj yk (note that possibly this is zero). We then increase yi by ϵ. This will cause some dual constraint associated with some set Sℓ to become tight; that is, after increasing yi we will have for this set Sℓ ∑ yk = wℓ . k:ek ∈Sℓ
We add the set Sℓ to our cover (by adding ℓ to I) and continue until all elements are covered. In Section 1.5, we argued that this algorithm is an f approximation algorithm for the set cover problem, where f = maxi  {j : ei ∈ Sj } . We repeat the analysis here, since there are several features of the analysis that are used frequently in analyzing primaldual approximation algorithms. Theorem 7.1: Algorithm 7.1 is an f approximation algorithm for the set cover problem. ∑ Proof. For the cover I constructed by the algorithm, we would like to show that j∈I wj ≤ ∗ f · OPT. Let ZLP be the optimal value of the linear programming relaxation of (7.1). It is Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.1
The set cover problem: a review
163
∑ ∑ suﬃcient to show that j∈I wj ≤ f · ni=1 yi for solution y, since by weak duality ∑the ﬁnal dual ∗ , and since the LP is a relaxation, we know that for any dual feasible solution y, ni=1 yi ≤ ZLP ∗ ≤ OPT. ZLP Because we only added a set Sj to our ∑ cover when its corresponding dual inequality was tight, we know that for any j ∈ I, wj = i:ei ∈Sj yi . Thus we have that ∑
wj
∑ ∑
=
yi
j∈I i:ei ∈Sj
j∈I
n ∑
=
yi ·  {j ∈ I : ei ∈ Sj } 
i=1
where the second equality comes from rewriting the double sum. We then observe that since  {j ∈ I : ei ∈ Sj }  ≤ f , we get that ∑
wj ≤ f ·
n ∑
yi ≤ f · OPT .
i=1
j∈I
We’ll be using several features of the algorithm and the analysis repeatedly in this chapter. In particular: we maintain a feasible dual solution, and increase dual variables until a dual constraint becomes tight. This indicates an object that we need to add to our primal solution (a set, in this case). When we analyze the cost of the primal solution, each object in the solution was given by a tight dual inequality. Thus we can rewrite the cost of the primal solution in terms of the dual variables. We then compare this cost with the dual objective function and show that the primal cost is within a certain factor of the dual objective, which shows that we are close to the value of an optimal solution. ∑ In this case, we increase dual variables until wj = i:ei ∈Sj yi for some set Sj , which we then add to our primal solution. When we have a feasible primal solution I, we can rewrite its cost in terms of the dual variables by using the tight dual inequalities, so that ∑ ∑ ∑ yi . wj = j∈I
j∈I i:ei ∈Sj
By exchanging the double summation, we have that ∑ j∈I
wj =
n ∑
yi ·  {j ∈ I : ei ∈ Sj } .
i=1
Then by bounding the value of  {j ∈ I : ei ∈ Sj }  by f , we get that the cost is at most f times the dual objective function, proving a performance guarantee on the algorithm. Because we will use this form of analysis frequently in this chapter, we will call it the standard primaldual analysis. This method of analysis is strongly related to the complementary slackness conditions discussed at the end of Section 1.4. Let I be the set cover returned by the primaldual algorithm, and consider an integer primal solution x∗ for the integer programming formulation (7.1) of the set cover problem in which we set x∗j = 1 for each set j ∈ I. Then we know that whenever x∗j > 0, the corresponding dual inequality is tight, so this part of the complementary slackness Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
164
The primaldual method
conditions is satisﬁed. If it were∑also the case that whenever yi∗ > 0, the corresponding primal inequality were tight (namely, j:ei ∈Sj x∗j = 1) then the complementary slackness conditions would imply that x∗ is an optimal solution. This is not the case, but we have an approximate form of the complementary slackness condition that holds; namely, whenever yi∗ > 0, ∑
x∗j =  {j ∈ I : ei ∈ Sj }  ≤ f.
j:ei ∈Sj
Whenever we can show that these complementary slackness conditions hold within a factor of α, we can then obtain an αapproximation algorithm. Recall from Section 5.6 that we deﬁned the integrality gap of an integer programming formulation to be the worstcase ratio over all instances of the problem of the optimal value of the integer program to the optimal value of the linear programming relaxation. Standard primaldual algorithms construct a primal integer solution and a solution to the dual of the linear programming relaxation, and the performance guarantee of the algorithm gives an upper bound on the ratio of these two values over all possible instances of the problem. Therefore, the performance guarantee of a primaldual algorithm provides an upper bound on the integrality gap of an integer programming formulation. However, the opposite is also true: the integrality gap gives a lower bound on the performance guarantee that can be achieved via a standard primaldual algorithm and analysis, or indeed any algorithm that compares the value of its solution with the value of the linear programming relaxation. In this chapter we will sometimes be able to show limits on primaldual algorithms that use particular integer programming formulations due to an instance of the problem with a bad integrality gap.
7.2
Choosing variables to increase: the feedback vertex set problem in undirected graphs
In the feedback vertex set problem in undirected graphs, we are given an undirected graph G = (V, E) and nonnegative weights wi ≥ 0 for vertices i ∈ V . The goal is to choose a minimumcost subset of vertices S ⊆ V such that every cycle C in the graph contains some vertex of S. We sometimes say that S hits every cycle of the graph. Another way to view the problem is that the goal is to ﬁnd a minimumcost subset of vertices S such that removing S from the graph leaves the graph acyclic. Let G[V − S] be the graph on the set of vertices V − S with the edges from G that have both endpoints in V − S; we say that G[V − S] is the graph induced by V − S. A third way to view the problem is to ﬁnd a minimumcost set of vertices S such that the induced graph G[V − S] is acyclic. We will give a primaldual algorithm for this problem. In the case of the set cover problem, it did not matter which dual variable was increased; we could increase any variable yi corresponding to an uncovered element ei and obtain the same performance guarantee. However, it is often helpful to carefully select the dual variable to increase, and we will see this principle in the course of devising an algorithm for the feedback vertex set problem. If we let C denote the set of all cycles C in the graph, we can formulate the feedback vertex Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.2 Choosing variables to increase: the feedback vertex set problem in undirected graphs 165
set problem in undirected graphs as the following integer program: ∑ minimize wi xi
(7.4)
i∈V
∑
subject to
xi ≥ 1,
∀C ∈ C,
xi ∈ {0, 1} ,
∀i ∈ V.
i∈C
Initially this might not seem like such a good choice of a formulation: the number of cycles in the graph can be exponential in the size of the graph. However, with the primaldual method we do not actually need to solve either the integer program or its linear programming relaxation; our use of the linear program and the dual only guides the algorithm and its analysis, so having an exponential number of constraints is not problematic. If we relax the integer program to a linear program by replacing the constraints xi ∈ {0, 1} with xi ≥ 0, and take its dual, we obtain ∑ maximize yC C∈C
subject to
∑
yC ≤ wi ,
∀i ∈ V,
yC ≥ 0,
∀C ∈ C.
C∈C:i∈C
Again, it might seem worrisome that we now have an exponential number of dual variables, since the primaldual method maintains a feasible dual solution. In the course of the algorithm, however, only a polynomial number of these will become nonzero, so we only need to keep track of these nonzero variables. By analogy with the primaldual algorithm for the setcover problem, we obtain Algorithm 7.2. We start with the dual feasible solution in which all yC are set to zero, and with the primal infeasible solution S = ∅. We see if there is any cycle C left in the induced graph G[V − S]. If there is, then we determine the amount by which we can increase the yC while ( ) ∑ dual variable still maintaining dual feasibility. This amount is ϵ = mini∈C wi − C ′ :i∈C ′ yC ′ . Increasing yC causes a dual inequality to become tight for some vertex ℓ ∈ C; in particular, it becomes tight for a vertex ℓ ∈ C that attains the minimum in the expression for ϵ. We add this vertex ℓ to our solution S and we remove ℓ from the graph. We also repeatedly remove any vertices of degree one from the graph (since they cannot be in any cycle) until we are left with a graph that contains only vertices of degree two or higher. Let n = V  be the number of vertices in the graph. Then note that we can add at most n vertices to our solution, so that we only go through the main loop at most n times, and at most n dual variables are nonzero. Suppose we now analyze the algorithm as we did for the set∑cover problem. Let S be the ﬁnal set of vertices chosen. We know that for any i ∈ S, wi = C:i∈C yC . Thus we can write the cost of our chosen solution as ∑ ∑ ∑ ∑ wi = yC = S ∩ CyC . i∈S
i∈S C:i∈C
C∈C
Note that S ∩ C is simply the number of vertices of the solution S in the ∑ cycle C. If we can ∑ show that S ∩ C ≤ α whenever yC > 0, then we will have i∈S wi ≤ α C∈C yC ≤ α · OPT. Unfortunately, if in the main loop of the algorithm we choose an arbitrary cycle C and increase its dual variable yC , it is possible that S ∩ C can be quite large. In order to do better, Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
166
The primaldual method
y←0 S←∅ while there exists a cycle C in G do ∑ Increase yC until there is some ℓ ∈ C such that C ′ ∈C:ℓ∈C ′ yC ′ = wℓ S ← S ∪ {ℓ} Remove ℓ from G Repeatedly remove vertices of degree one from G return S Algorithm 7.2: Primaldual algorithm for the feedback vertex set problem (ﬁrst attempt). we need to make a careful choice of cycle C. If we can always choose a short cycle, with C ≤ α, then certainly we will have S ∩ C ≤ C ≤ α. This isn’t always possible either: the graph itself can simply be one large cycle through all n vertices. In such a case, however, we only need to choose one vertex from the cycle in order to have a feasible solution. This leads to the following observation. Observation 7.2: For any path P of vertices of degree two in graph G, Algorithm 7.2 will choose at most one vertex from P ; that is, S ∩ P  ≤ 1 for the ﬁnal solution S given by the algorithm. Proof. Once S contains a vertex of P , we remove that vertex from the graph. Its neighbors in P will then have degree one and be removed. Iteratively, the entire path P will be removed from the graph, and so no further vertices of P will be added to S. Suppose that in the main loop of Algorithm 7.2 we choose the cycle C that minimizes the number of vertices that have degree three or higher. Note that in such a cycle, vertices of degree three or more alternate with paths of vertices of degree two (possibly paths of a single edge). Thus by Observation 7.2, the value of S ∩ C for the ﬁnal solution S will be at most twice the number of vertices of degree three or higher in C. The next lemma shows us that we can ﬁnd a cycle C with at most O(log n) vertices of degree three or higher. Lemma 7.3: In any graph G that has no vertices of degree one, there is a cycle with at most 2⌈log2 n⌉ vertices of degree three or more, and it can be found in linear time. Proof. If G has only vertices of degree two, then the statement is trivially true. Otherwise, pick an arbitrary vertex of degree three or higher. We start a variant of a breadthﬁrst search from this vertex, in which we treat every path of vertices of degree two as a single edge joining vertices of degree three or more; note that since there are no degree one vertices, every such path joins vertices of degree at least three. Thus each level of the breadthﬁrst search tree consists only of vertices of degree three or more. This implies that the number of vertices at each level is at least twice the number of vertices of the previous level. Observe then that the depth of the breadthﬁrst search can be at most ⌈log2 n⌉: once we reach level ⌈log2 n⌉, we have reached all n vertices of the graph. We continue the breadthﬁrst search until we close a cycle; that is, we ﬁnd a path of vertices of degree two from a node on the current level to a previously visited node. This cycle has at most 2⌈log2 n⌉ vertices of degree three or more, since at worst the cycle is closed by an edge joining two vertices at depth ⌈log2 n⌉. We give the revised algorithm in Algorithm 7.3. We can now show the following theorem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.2 Choosing variables to increase: the feedback vertex set problem in undirected graphs 167
y←0 S←∅ Repeatedly remove vertices of degree one from G while there exists a cycle in G do Find cycle C with at most 2⌈log2 n⌉ vertices of degree three or more ∑ Increase yC until there is some ℓ ∈ C such that C ′ ∈C:ℓ∈C ′ yC ′ = wℓ S ← S ∪ {ℓ} Remove ℓ from G Repeatedly remove vertices of degree one from G return S Algorithm 7.3: Primaldual algorithm for the feedback vertex set problem (second attempt).
Theorem 7.4: Algorithm 7.3 is a (4⌈log2 n⌉)approximation algorithm for the feedback vertex set problem in undirected graphs.
Proof. As we showed above, the cost of our ﬁnal solution S is ∑ i∈S
wi =
∑ ∑
yC =
i∈S C:i∈C
∑
S ∩ CyC .
C∈C
By construction, yC > 0 only when C contains at most 2⌈log2 n⌉ vertices of degree three or more in the graph at the time we increased yC . Note that at this time S ∩ C = ∅. By Observation 7.2, each path of vertices of degree two joining two vertices of degree three or more in C can contain at most one vertex of S. Thus if C has at most 2⌈log2 n⌉ vertices of degree three or more, it can have at most 4⌈log2 n⌉ vertices of S overall: possibly each vertex of degree 3 or more is in S, and then at most one of the vertices in the path joining adjacent vertices of degree three or more can be in S. Since as the algorithm proceeds, the degree of a vertex can only go down, we obtain that whenever yC > 0, S ∩ C ≤ 4⌈log2 n⌉. Thus we have that ∑ i∈S
wi =
∑ C∈C
S ∩ CyC ≤ (4⌈log2 n⌉)
∑
yC ≤ (4⌈log2 n⌉) OPT .
C∈C
The important observation of this section is that in order to get a good performance guarantee, one must choose carefully the dual variable to increase, and it is frequently useful to choose a dual variable that is small or minimal in some sense. The integrality gap of the integer programming formulation (7.4) is known to be Ω(log n), and so a performance guarantee of O(log n) is the best we can hope for from a primaldual algorithm using this formulation. However, this does not rule out obtaining a better performance guarantee by a primaldual algorithm using a diﬀerent integer programming formulation of the problem. In Section 14.2, we will show that we can obtain a primaldual 2approximation algorithm for the feedback vertex set problem in undirected graphs by considering a more sophisticated integer programming formulation of the problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
168
The primaldual method
7.3
Cleaning up the primal solution: the shortest st path problem
In the shortest st path problem, we are given an undirected graph G = (V, E), nonnegative costs ce ≥ 0 on all edges e ∈ E, and a pair of distinguished vertices s and t. The goal is to ﬁnd the minimumcost path from s to t. It is wellknown that the optimal solution can be found in polynomial time; for instance, Dijkstra’s algorithm ﬁnds an optimal solution in polynomial time. However, it is instructive to think about applying the primaldual method to this problem, in part because it gives us insight into using the primaldual method for related problems that are NPhard; we will see this in the next section. Additionally, the algorithm we ﬁnd using the primaldual method turns out to be equivalent to Dijkstra’s algorithm. Let S = {S ⊆ V : s ∈ S, t ∈ / S}; that is, S is the set of all st cuts in the graph. Then we can model the shortest st path problem with the following integer program: minimize
∑
ce xe
e∈E
subject to
∑
xe ≥ 1,
∀S ∈ S,
xe ∈ {0, 1} ,
∀e ∈ E,
e∈δ(S)
where δ(S) is the set of all edges that have one endpoint in S and the other endpoint not in S. To see that this integer program models the shortest st path problem, take any feasible solution x and consider the graph G′ = (V, E ′ ) with E ′ = {e ∈ E : xe = 1}. The constraints ensure that for any st cut S, there must be at least one edge of E ′ in δ(S): that is, the size of the minimum st cut in G′ must be at least one. Thus by the maxﬂow mincut theorem the maximum st ﬂow in G′ is at least one, which implies that there is a path from s to t in G′ . Similarly, if x is not feasible, then there is some st cut S for which there are no edges of E ′ in δ(S), which implies that the size of minimum st cut is zero, and thus the maximum st ﬂow is zero. Hence there is no path from s to t in G′ . Once again the number of constraints in the integer program is exponential in the size of the problem, and as with the feedback vertex set problem in Section 7.2, this is not problematic since we only use the formulation to guide the algorithm and its analysis. If we replace the constraints xe ∈ {0, 1} with xe ≥ 0 to obtain a linear programming relaxation and take the dual of this linear program, we obtain: maximize
∑
yS
S∈S
subject to
∑
yS ≤ ce ,
∀e ∈ E,
yS ≥ 0
∀S ∈ S.
S∈S:e∈δ(S)
The dual variables yS have a nice geometric interpretation; they can be interpreted as “moats” surrounding the set S of width yS ; see Figure 7.1 for an illustration. Any path from s to t must cross this moat, and hence must have cost at least yS . Moats must be nonoverlapping, and thus for any edge e, we cannot have edge e crossing moats of total width more than ce ; thus ∑ the dual constraint that S:e∈δ(S) yS ≤ ce . We ∑ can have many moats, and any st path must cross them all, and have total length at most S∈S yS . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.3
Cleaning up the primal solution: the shortest st path problem
169
s t yS S
Figure 7.1: An illustration of a moat separating s from t. The moat contains the nodes in S and its width is yS . y←0 F ←∅ while there is no st path in (V, F ) do Let C be the connected component of (V, F ) containing ∑s ′ Increase yC until there is an edge e ∈ δ(C) such that S∈S:e′ ∈δ(S) yS = ce′ F ← F ∪ {e′ } Let P be an st path in (V, F ) return P Algorithm 7.4: Primaldual algorithm for the shortest st path problem.
We now give a primaldual algorithm for the shortest st path problem in Algorithm 7.4 which follows the general lines of the primaldual algorithms we have given previously for the set cover and feedback vertex set problems. We start with a dual feasible solution y = 0 and the primal infeasible solution of F = ∅. While we don’t yet have a feasible solution, we increase the dual variable yC associated with an st cut C which the current solution does not satisfy; that is, for which F ∩ δ(C) = ∅. We call such constraints violated constraints. Following the lesson of the previous section, we carefully choose the “smallest” such constraint: we let C be the set of vertices of the connected component containing s in the set of edges F . Because F does not contain an st path, we know that t ∈ / C, and by the deﬁnition of a connected component we know that δ(C) ∩ F = ∅. We increase the dual variable yC until some constraint of the dual becomes tight for some edge e′ ∈ E, and we add e′ to F . Once our primal solution F is feasible and contains an st path, we end the main loop. Now we do something slightly diﬀerent than before: we don’t return the solution F , but rather a subset of F . Let P be any st path such that P ⊆ F . Our algorithm returns P . We do this since it may be too expensive to return all the edges in F , so we eﬀectively delete any edge we Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
170
The primaldual method
do not need and return only the edges in P . We begin the analysis by showing that the set of edges F found by the algorithm forms a tree; this implies that the st path P is unique. Lemma 7.5: At any point in Algorithm 7.4, the set of edges in F forms a tree containing the vertex s. Proof. We prove this by induction on the number of edges added to F . In each step of the main loop we consider the connected component C of (V, F ) containing s, and add an edge e′ from the set δ(C) to our solution F . Since exactly one endpoint of e′ is in C, e′ cannot close a cycle in the tree F , and causes F to span a new vertex not previously spanned. We can now show that this algorithm gives an optimal algorithm for the shortest st path problem. Theorem 7.6: Algorithm 7.4 ﬁnds a shortest path from s to t. Proof. We prove this using ∑the standard primaldual analysis . As usual, since for every edge e ∈ P , we have that ce = S:e∈δ(S) yS , and we know that ∑ e∈P
ce =
∑
∑
∑
yS =
P ∩ δ(S)yS .
S:s∈S,t∈S /
e∈P S:e∈δ(S)
If we can now show that whenever yS > 0, we have P ∩ δ(S) = 1, then we will show that ∑ e∈P
ce =
∑
yS ≤ OPT
S:s∈S,t∈S /
by weak duality. But of course since P is an st path of cost no less than OPT, it must have cost exactly OPT. We now show that if yS > 0, then P ∩ δ(S) = 1. Suppose otherwise, and P ∩ δ(S) > 1. Then there must be a subpath P ′ of P joining two vertices of S such that the only vertices of P ′ in S are its start and end vertices; see Figure 7.2. Since yS > 0, we know that at the time we increased yS , F was a tree spanning just the vertices in S. Thus F ∪ P ′ must contain a cycle. Since P is a subset of the ﬁnal set of edges F , this implies that the ﬁnal F contains a cycle, which contradicts Lemma 7.5. Thus it must be the case that P ∩ δ(S) = 1. As we stated previously, one can show that this algorithm behaves in exactly the same way as Dijkstra’s algorithm for solving the shortest st path problem; proving this equivalence is given as Exercise 7.1.
7.4
Increasing multiple variables at once: the generalized Steiner tree problem
We now turn to a problem known as the generalized Steiner tree problem or the Steiner forest problem. In this problem we are given an undirected graph G = (V, E), nonnegative costs ce ≥ 0 for all edges e ∈ E, and k pairs of vertices si , ti ∈ V . The goal is to ﬁnd a minimumcost subset of edges F ⊆ E such that every si ti pair is connected in the set of selected edges; that is, si and ti are connected in (V, F ) for all i. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.4
Increasing multiple variables at once: the generalized Steiner tree problem
171
S
P0
s t
Figure 7.2: Proof of Theorem 7.6. The heavy line is the path P ; the dotted heavy line is the subpath P ′ . Let Si be the subsets of vertices separating si and ti ; that is, Si = {S ⊆ V : S ∩ {si , ti }  = 1}. Then we can model this problem with the following integer program: ∑ minimize ce xe (7.5) e∈E
subject to
∑
xe ≥ 1,
∀S ⊆ V : S ∈ Si for some i,
xe ∈ {0, 1} ,
e ∈ E.
e∈δ(S)
The set of constraints enforce that for any si ti cut S with si ∈ S, ti ∈ / S or vice versa, we must select one edge from δ(S). The argument that this models the generalized Steiner tree problem is similar to that for the shortest st path problem in the previous section. Given the linear programming relaxation obtained by dropping the constraints xe ∈ {0, 1} and replacing them with xe ≥ 0, the dual of this linear program is ∑ maximize yS S⊆V :∃i,S∈Si
subject to
∑
yS ≤ ce ,
∀e ∈ E,
yS ≥ 0,
∃i : S ∈ Si .
S:e∈δ(S)
As in the case of the shortest st path problem, the dual has a natural geometric interpretation as moats. In this case, however, we can have moats around any set of vertices S ∈ Si for any i. See Figure 7.3 for an illustration. Our initial attempt at a primaldual algorithm is given in Algorithm 7.5, and is similar to the shortest st path algorithm given in the previous section. In every iteration, we choose some connected component C such that C ∩ {si , ti }  = 1 for some i. We increase the dual variable yC associated with C until the dual inequality associated with some edge e′ ∈ δ(C) becomes tight, and we add this edge to our primal solution F . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
172
The primaldual method
s1
δ
s2
t3
Figure 7.3: Illustration of moats for the generalized Steiner tree problem. Each of s1 , s2 , and t3 has a white moat surrounding just that node; additionally, there is a grey moat depicting y{s1 ,s2 ,t3 } = δ. y←0 F ←∅ while not all si ti pairs are connected in (V, F ) do Let C be a connected component of (V, F ) such that C ∑∩ {si , ti }  = 1 for some i Increase yC until there is an edge e′ ∈ δ(C) such that S∈Si :e′ ∈δ(S) yS = ce′ F ← F ∪ {e′ } return F Algorithm 7.5: Primaldual algorithm for the generalized Steiner tree problem (ﬁrst attempt). Using the standard primaldual analysis, we can analyze the cost of the ﬁnal solution F as follows: ∑ ∑ ∑ ∑ ce = yS = δ(S) ∩ F yS . e∈F
∑
e∈F S:e∈δ(S)
S
∑ In order to compare the term S δ(S) ∩ F yS with the dual objective function S yS , we will need to show that δ(S) ∩ F  ≤ α for some α whenever yS > 0. Unfortunately, it is possible to give an example showing that δ(S) ∩ F  = k (where k is the number of si ti pairs) for yS > 0 no matter what connected component is chosen in the main loop of Algorithm 7.5. To see this, consider the complete graph on k + 1 vertices. Let each edge have cost 1, let one vertex correspond to all si , and let the remaining k vertices be t1 , . . . , tk (see Figure 7.4). The algorithm must choose one of the k + 1 vertices initially for the set C. The dual yC gets value yC = 1, and this is the only nonzero dual at the end of the algorithm. The ﬁnal solution F has edges from the vertex chosen for C to all k other vertices, giving δ(C) ∩ F  = k. However, observe that in the ﬁnal solution if we take the average of δ(C) ∩ F  over ∑ all the k + 1 vertices we could have chosen for the initial set C, we get 2k/(k + 1) ≈ 2 since j∈V δ({j}) ∩ F  = 2k. This observation suggests that perhaps we should increase the dual variables for several sets C at once. We present this variation in Algorithm 7.6. Let C be the set of all the connected Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.4
Increasing multiple variables at once: the generalized Steiner tree problem
173
s1, s2, s3, s4
t1
t4
t2
t3
Figure 7.4: Bad example for Algorithm 7.5. If the algorithm chooses the vertex s1 , . . . , s4 as its initial connected component C, then it will eventually add all solid edges to F , and δ(C) ∩ F  = 4. components C such that C ∩ {si , ti }  = 1. We increase associated dual variables yC for all C ∈ C at the same rate until a dual inequality becomes tight for some edge e ∈ δ(C) for a set C whose dual we increased. We then add e to our solution F and continue. Additionally, we index the edges we add as we go along: e1 is added in the ﬁrst iteration, e2 in the second, and so on. Once we have a feasible solution F such that all si ti pairs are connected in F , we go through the edges in the reverse of the order in which they were added, from the edge added in the last iteration to the edge added in the ﬁrst iteration. If the edge can be removed without aﬀecting the feasibility of the solution, it is deleted. The ﬁnal set of edges returned after this “reverse deletion” step is F ′ . This reverse deletion step is solely to simplify our analysis; in Exercise 7.4 the reader is asked to show that removing all unnecessary edges in any order gives an equally good approximation algorithm. Given the geometric interpretation of the dual variables as moats, we can give an illustration of the algorithm in Figure 7.5. We can now show that our intuition gathered from the bad example in Figure 7.4 is essentially correct, and that the algorithm is a 2approximation algorithm for the generalized Steiner tree problem. In order to do this, we ﬁrst state a lemma, whose proof we defer for a moment. Lemma 7.7: For any C in any iteration of the algorithm, ∑ δ(C) ∩ F ′  ≤ 2C. C∈C
In terms of the geometric interpretation, we wish to show that the number of times that the edges in the solution cross a moat is at most twice the number of moats; see Figure 7.6. The main intuition of the proof is that the degree of nodes in a tree is at most twice the number of nodes, where we treat each connected component C as a node of the tree and the edges in δ(C) as the edges of the tree. The proof is slightly more complicated than this because only some components C are in C, but we show that every “leaf” of the tree is a component in C and this is suﬃcient to prove the result. We now show that the lemma implies the desired performance guarantee. Theorem 7.8: Algorithm 7.6 is a 2approximation algorithm for the generalized Steiner tree problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
174
The primaldual method
t3
s1
s2
t2
s1
t1
s1
s2
t2
t1
s2
s3
s3
t3
t3
t2
s1
t1
s1
t3
s2
t2
t1
s2
s3
s3
t3
t3
t2
s1
t1
s2
t2
t1 s3
s3
Figure 7.5: Illustration of the primaldual algorithm for the generalized Steiner tree problem. Two edges go tight simultaneously in the last iteration before the deletion step. The deletion step removes the edge (s1 , s2 ) added in the ﬁrst iteration. The ﬁnal set of edges is shown in the last ﬁgure. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.4
Increasing multiple variables at once: the generalized Steiner tree problem
175
y←0 F ←∅ ℓ←0 while not all si ti pairs are connected in (V, F ) do ℓ←ℓ+1 Let C be the set of all connected components C of (V, F ) such that C ∩ {si , ti }  = 1 for some i ′ ′ Increase ∑ yC for all C in C uniformly until for some eℓ ∈ δ(C ), C ∈ C, ceℓ = S:eℓ ∈δ(S) yS F ← F ∪ {eℓ } ′ F ←F for k ← ℓ downto 1 do if F ′ − ek is a feasible solution then Remove ek from F ′ return F ′ Algorithm 7.6: Primaldual algorithm for the generalized Steiner tree problem (second attempt). Proof. As usual, we begin by expressing the cost of our primal solution in terms of the dual variables: ∑ ∑ ∑ ∑ ce = yS = F ′ ∩ δ(S)yS . e∈F ′
We would like to show that
e∈F ′ S:e∈δ(S)
∑
S
F ′ ∩ δ(S)yS ≤ 2
S
∑
yS ,
(7.6)
S
since by weak duality, this will imply that the algorithm is a 2approximation algorithm. As is suggested by the bad example in Figure 7.4, we cannot simply prove this by showing that F ′ ∩ δ(S) ≤ 2 whenever yS > 0. Instead, we show inequality (7.6) by induction on the number of iterations of the algorithm. Initially all dual variables yS = 0, so inequality (7.6) holds. Suppose that the inequality holds at the beginning of some iteration of the main loop of the algorithm. Then in this iteration we increase each yC for C ∈ C by the same amount (call it ϵ). This increases the lefthand side of (7.6) by ∑ ϵ F ′ ∩ δ(C) C∈C
and the righthand side by 2ϵC. However, by the inequality of Lemma 7.7, this means that the increase in the lefthand side is no greater than the increase in the righthand side. Thus if the inequality held before the increase of the dual variables in this iteration, it will also hold afterwards. We now turn to the proof of Lemma 7.7. We ﬁrst need an observation. Observation 7.9: At any point in the algorithm, the set of edges F is a forest. Proof. We prove the statement by induction on the number of iterations of the algorithm. The set F = ∅ is initially a forest. Any time we add an edge to the solution, it has at most one Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
176
The primaldual method
t3
s1
s2
t2
t1 s3
Figure 7.6: Illustration of Lemma 7.7. Consider the 5 thin dark moats. The edges of F ′ cross these moats 8 times (twice each by the edges joining s1 t1 , s2 t2 , t2 t3 , and t2 s3 ). Each moat corresponds to a connected component C ∈ C in the iteration in which ∑ the corresponding duals yC were increased. Thus in this iteration we see that 8 = C∈C δ(C) ∩ F ′  ≤ 2C = 10. endpoint in any given connected component of C. Thus it joins two connected components of the forest F , and the forest remains a forest. Proof of Lemma 7.7. Consider the iteration in which edge ei is added to F . Let Fi be the edges already in F at this point; that is, Fi = {e1 , . . . , ei−1 }. Let H = F ′ − Fi . Observe that Fi ∪ H = Fi ∪ F ′ is a feasible solution to the problem, since F ′ by itself is a feasible solution to the problem. We claim also that if we remove any edge e ∈ H from Fi ∪ H, it will not be a feasible solution. This follows from the deletion procedure at the end of the algorithm: at the time we consider edge ei−1 for deletion, the edges in F ′ at that point in the procedure are exactly those in Fi ∪ H. Hence it must be the case that any edge already considered and remaining in F ′ is necessary for the solution to be feasible. These edges are exactly the edges in H. We form a new graph by contracting each connected component of (V, Fi ) to a single vertex. Let V ′ be this new set of vertices. Observe that since at any point F is a forest, once we have contracted all the edges in Fi into vertices V ′ , no edge in H can have both endpoints corresponding to the same vertex in V ′ . Thus we can consider the forest of edges H on the set of vertices V ′ , where each edge in H is connected to the two vertices in V ′ corresponding to the two connected components it joined in (V, Fi ). See Figure 7.7 for an illustration. We let deg(v) for v ∈ V ′ represent the degree of vertex v in this forest. We also color the vertices in V ′ with two diﬀerent colors, red and blue. The red vertices in V ′ are those connected components C of (V, Fi ) that are in the set C in this iteration (that is, for some j, C ∩ {sj , tj }  = 1). The blue vertices are all the other vertices of V ′ . Let R denote the set of red vertices in V ′ and B the set of blue vertices v that have deg(v) > 0. We now observe that we can rewrite the desired inequality ∑ δ(C) ∩ F ′  ≤ 2C C∈C
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.4
Increasing multiple variables at once: the generalized Steiner tree problem
177
Figure 7.7: Proof of Lemma 7.7. On the left is the current set of connected components, with the edges in H shown in dashed lines. The right side shows the contracted graph. in terms of this forest. The righthand side is 2R. Since F ′ ⊆ Fi ∪ H and no edge of Fi can appear in δ(C) of Fi ), the lefthand side is no ∑ for C ∈ C (since C is a connected component ∑ greater than v∈R deg(v). So we wish to prove that v∈R deg(v) ≤ 2R. To prove this, we claim that no blue vertex can have degree exactly one. If this is true, then we have that ∑ ∑ ∑ deg(v) = deg(v) − deg(v). v∈R
v∈R∪B
v∈B
Then since the total degree of a forest is no more than twice the number of vertices in it, and since all blue vertices of nonzero degree have degree at least two by the claim, we have that ∑ deg(v) ≤ 2(R + B) − 2B = 2R, v∈R
as desired. It remains to prove that no blue vertex can have degree one. Suppose otherwise, and let v ∈ V ′ be a blue vertex of degree one, let C be the connected component of the uncontracted graph corresponding to v, and let e ∈ H be the incident edge. By our initial discussion, e must be necessary for the feasibility of the solution. Thus it must be on some path between sj and tj for some j, and C ∩ {sj , tj }  = 1. But in this case, C must be in C, and v must be red, which is a contradiction. Since the proof shows that the algorithm ﬁnds a solution to the integer programming formulation (7.5) of cost at most twice the value of a dual solution to the dual of the linear programming relaxation, the proof implies that the integrality gap of the formulation is at most 2. We can use the integrality gap instance for the prizecollecting Steiner tree problem in Section 5.7 (see Figure 5.4) to show that the integrality gap for this formulation is essentially 2, so that no better performance guarantee can be obtained by using a primaldual algorithm with this formulation. In Section 16.4, we will consider a directed version of the generalized Steiner tree problem, and we will show that it is substantially more diﬃcult to approximate the directed version of the problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
178
The primaldual method
7.5
Strengthening inequalities: the minimum knapsack problem
In this section, we turn to a minimization version of the knapsack problem introduced in Section 3.1. As in the previous version of the knapsack problem, we are given a set I of n items, I = {1, 2, . . . , n}. Each item has a value vi and a size si . In the previous version of the problem, we additionally had a knapsack capacity B, and the goal was to ﬁnd the maximumvalue subset of items such that the total size of the items was at most the capacity of the knapsack. In the minimization version of the problem, we are given a demand D, and the goal of the problem is to ﬁnd a subset of items having minimum total size such that the total value ∑is at least the demand. That is, we try to ﬁnd a subset X ⊆ I of items minimizing s(X) = i∈X si subject ∑ to the constraint that v(X) = i∈X vi ≥ D. We can formulate the problem as the following integer program, in which the primal variable xi indicates whether item i is chosen to be in the solution or not: ∑ minimize si xi i∈I
subject to
∑
vi xi ≥ D,
i∈I
xi ∈ {0, 1} ,
∀i ∈ I.
We obtain a linear programming relaxation for the problem by replacing the constraints xi ∈ {0, 1} with linear constraints 0 ≤ xi ≤ 1 for all i ∈ I. However, this linear programming relaxation is not a particularly good one, in the sense that it has a bad integrality gap. Consider a twoitem set I = {1, 2}, where v1 = D − 1, v2 = D, s1 = 0 and s2 = 1. The only feasible integer solutions require us to take item 2 (x2 = 1), for a total size of 1. But the solution x1 = 1, x2 = 1/D is feasible for the linear programming relaxation and has total size 1/D. This example shows that the integrality gap of this formulation is at least 1/(1/D) = D. To get a primaldual algorithm with a reasonable performance guarantee, we must use a diﬀerent integer programming formulation of the same problem, one with a better integrality gap. We introduce a new set of constraints, one for every subset A ⊆ I of items such that v(A) < D. Let DA = D − v(A); we can think of DA as the demand left over if the items in A are added to the knapsack. Notice that even if we select every item in the set A, we must still choose additional items of total value at least DA . Given the set A, this is simply another minimum knapsack problem on the set of items I − A, where the desired demand is now DA . Given A, we can also reduce the value of each item in I − A to be the minimum of its value and DA ; since the desired demand is DA , we don’t need the value of the item to be any larger. We let viA = min(vi , DA∑ ). Then for any A ⊆ I and any set of items X ⊆ I such that v(X) ≥ D, it is the case that i∈X−A viA ≥ DA . We can then give the following integer programming formulation of the problem: ∑ minimize si xi i∈I
subject to
∑
viA xi ≥ DA ,
∀A ⊆ I,
i∈I−A
xi ∈ {0, 1} ,
∀i ∈ I.
As we argued above, any integer solution corresponding to a knapsack of value at least D is Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.5
Strengthening inequalities: the minimum knapsack problem
179
y←0 A←∅ while v(A) < D do ∑ B Increase yA until for some i ∈ I − A, B⊆I:i∈B / vi yB = si A ← A ∪ {i} return A. Algorithm 7.7: Primaldual algorithm for the minimum knapsack problem. a feasible solution to the integer program, and thus the integer program models the minimum knapsack problem. We now consider the linear programming relaxation of the integer program above, replacing the constraints xi ∈ {0, 1} with xi ≥ 0: ∑ si xi minimize i∈I
subject to
∑
viA xi ≥ DA ,
∀A ⊆ I,
xi ≥ 0,
∀i ∈ I.
i∈I−A
Observe that for the instance that gave a bad integrality gap for the previous integer programming formulation, the given LP solution is no longer feasible. In that example, we had I = {1, 2}, with v1 = D − 1, v2 = D, s1 = 0, and s2 = 1. Now consider the constraint corresponding∑ to A = {1}. We have DA = D − v1 = 1, and v2A = min(v2 , DA ) = 1, so that the constraint i∈I−A viA xi ≥ DA is x2 ≥ 1 for this choice of A. The constraint forces us to take item 2; we cannot take a 1/D fraction of item 2 to meet the missing single unit of demand. The dual of the linear programming relaxation is ∑ maximize DA yA A:A⊆I
∑
viA yA ≤ si ,
∀i ∈ I,
A⊆I:i∈A /
yA ≥ 0,
∀A ⊂ I.
We can now consider a primaldual algorithm for the problem using the primal and dual formulations as given above. We start with an empty set of selected items, and a dual solution yA = 0 for all A ⊆ I. Now we must select some dual variable to increase. Which one should it be? Following the idea introduced previously of choosing a variable corresponding to a minimal object of some sort, we increase the dual variable y∅ . A dual constraint will become tight for some item i ∈ I, and we will add this to our set of selected items. Which variable should we increase next? Notice that in order to maintain dual feasibility, it will have to be a variable yA for some set A such that i ∈ A; if i ∈ / A, then we cannot increase yA without violating the tight dual constraint for i. The most natural choice is to increase yA for A = {i}. We continue in this fashion, letting A be our set of selected items; whenever a dual constraint becomes tight for some new item j ∈ I, we add j to A, and in the next iteration increase the dual variable yA . The algorithm is given in Algorithm 7.7. The algorithm terminates when v(A) ≥ D. We can now show that the algorithm is a 2approximation algorithm for the minimum knapsack problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
180
The primaldual method
Theorem 7.10: Algorithm 7.7 is a 2approximation algorithm for the minimum knapsack problem. Proof. Let ℓ be the ﬁnal item selected by the algorithm, and let X be the set of items returned at the end of the algorithm. We know that v(X) ≥ D; since item ℓ was added to X, it must be the case that before ℓ was added, the total value of the set of items was less than D, so that v(X − ℓ) < D. Following the standard primaldual analysis, we know that ∑ ∑ ∑ si = viA yA . i∈X A⊆I:i∈A /
i∈X
Reversing the double sum, we have that ∑ ∑
viA yA =
i∈X A⊆I:i∈A /
∑
∑
yA
A⊆I
viA .
i∈X−A
Note that in any iteration of the algorithm except the last one, adding the next item i to the current set of items A did not cause the value of the knapsack to become at least D; that is, vi < D − v(A) = DA at that point in the algorithm. Thus for all items i ∈ X except ℓ, viA = min(vi , DA ) = vi for the point in the algorithm at which A was the current set of items. Thus we can rewrite ∑ ∑ viA = vℓA + viA = vℓA + v(X − ℓ) − v(A). i∈X−A
i∈X−A:i̸=ℓ
Note that vℓA ≤ DA by deﬁnition, and as argued at the beginning of the proof v(X − ℓ) < D so that v(X − ℓ) − v(A) < D − v(A) = DA ; thus we have that vℓA + v(X − ℓ) − v(A) < 2DA . Therefore
∑ i∈X
si =
∑ A⊆I
yA
∑ i∈X−A
viA < 2
∑
DA yA ≤ 2 OPT,
A:A⊆I
where the ﬁnal inequality follows by weak duality since function.
∑ A:A⊆I
DA yA is the dual objective
The proof of the performance guarantee of the algorithm shows that the integrality gap of the new integer programming formulation must be at most 2.
7.6
The uncapacitated facility location problem
We now return to the uncapacitated facility location problem introduced in Section 4.5 for which we gave a randomized approximation algorithm in Section 5.8. Recall that the input to the problem is a set of clients D and a set of facilities F , with facility costs fi for all facilities i ∈ F , and assignment costs cij for all facilities i ∈ F and clients j ∈ D. The goal is to select a subset of facilities to open and an assignment of clients to open facilities so as to minimize the total cost of the open facilities plus the assignment costs. As before, we consider the metric uncapacitated facility location and assume that the clients and facilities are points in a metric Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.6
The uncapacitated facility location problem
181
space, and the assignment cost cij is the distance between client j and facility i. In particular, given clients j, l and facilities i, k, we have that cij ≤ cil + ckl + ckj by the triangle inequality. We will now give a primaldual approximation algorithm for the problem. Recall the linear programming relaxation of the problem we used in previous sections: minimize
∑
∑
fi yi +
i∈F
cij xij
i∈F,j∈D
∑
subject to
xij = 1,
∀j ∈ D,
xij ≤ yi ,
∀i ∈ F, j ∈ D,
xij ≥ 0,
∀i ∈ F, j ∈ D,
i∈F
yi ≥ 0,
∀i ∈ F,
in which variable xij indicates whether client j is assigned to facility i, and variable yi indicates whether facility i is open or not. The dual of the LP relaxation is maximize
∑
vj
j∈D
subject to
∑
wij ≤ fi ,
∀i ∈ F,
j∈D
vj − wij ≤ cij , wij ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F, j ∈ D.
It is also useful to recall the intuition for the dual which we gave in Section 4.5. We can view the dual variables vj as the amount that each client j will pay towards its part of the cost of the solution. If facility costs are all zero, then vj = mini∈F cij . To handle nonzero facility costs,∑the cost fi is split into nonnegative cost shares wij apportioned among the clients, so that j∈D wij ≤ fi . A client j only needs to pay this share if it uses facility i. In this way, we no longer charge explicitly for opening a facility, but still recover some of its cost. Each client j is willing to pay the lowest cost over all facilities of its service cost and its share of the facility cost, so that vj = mini∈F (cij + wij∑ ). By allowing vj to be any value for which vj ≤ cij + wij , the objective function maximizing j∈D vj forces vj to be equal to the smallest righthand side over all facilities i ∈ F . Thus any feasible solution to the dual is a lower bound on the cost of an optimal solution to the facility location problem. To get some intuition for why a primaldual analysis will be useful for this problem, let us ﬁrst consider a dual feasible solution (v ∗ , w∗ ) that is maximal; that is, we cannot increase any ∗ that gives a dual feasible solution. Such vj∗ by any positive amount and then derive a set of wij a dual solution has some very nice structure. To discuss this further, we modify a deﬁnition used in previous sections, and introduce a new one. Deﬁnition 7.11: Given a dual solution (v ∗ , w∗ ),{we say that a client j neighbors a facility i } ∗ ∗ (or that i neighbors j) if vj ≥ cij . We let N (j) = i ∈ F : vj ≥ cij be the neighbors of a client { } ∗ j and N (i) = j ∈ D : vj ≥ cij be the neighbors of a facility i. Deﬁnition 7.12: Given a dual solution (v ∗ , w∗ ), we say that a client j contributes to a facility ∗ > 0. i if wij Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
182
The primaldual method
∗ for that In other words, a client j contributes to facility i if it has a nonzero cost share wij facility. We observe that given dual variables v ∗ , we can derive a feasible set of cost shares w∗ (if ∗ = max(0, v ∗ − c ). Observe that if we derive w ∗ in this way and they exist) by setting wij ij j ∗ > 0 implies v ∗ > c . client j contributes to facility i, then j neighbors i (j ∈ N (i)), since wij ij j ∗. Furthermore, if j ∈ N (i), then vj∗ = cij + wij Let T be the set of all facilities such that the sum of the cost shares is equal to the cost of the facility; in }other words, the corresponding dual inequality is tight. Then T = { ∑ ∗ = f . First we claim that in a maximal dual solution (v ∗ , w ∗ ), every client i ∈ F : j∈D wij i must neighbor some facility in T . To see this we claim that in a maximal dual solution, it ∗ ), and some facility i ∈ F attaining the minimum must be the case that vj∗ = mini∈F (cij + wij ∗ must be in T . Then for this facility i, vj ≥ cij , and j neighbors i ∈ T . To see the claim, ∗ ), we can feasibly increase v ∗ and the solution is not maximal. clearly if vj∗ < mini∈F (cij + wij j ∗ ∗ If v = min (c + w ) and all facilities i attaining the minimum are not in T , then since ij i∈F ij ∑ j ∗ < f for these facilities i we can feasibly increase v ∗ and w ∗ for these facilities i, and w i j ij k∈D ik once again the solution is not maximal. Given some facility i ∈ T , the cost of the facility plus the cost of assigning the neighbors N (i) to i is exactly equal to the sum of the dual variables of the neighboring clients; that is,
fi +
∑ j∈N (i)
cij =
∑ j∈N (i)
∗ (wij + cij ) =
∑
vj∗ ,
j∈N (i)
∗ > 0 implies that j ∈ N (i) and the second equality where the ﬁrst equality follows since wij ∗ +c = v ∗ . Since all clients have a neighbor in T , it would follows since j ∈ N (i) implies that wij ij j then seem that we could get an optimal algorithm by opening all facilities in T and assigning each client to its neighbor in T . The diﬃculty with this approach is that a given client j might neighbor several facilities in T and might contribute to many of them; we then use vj∗ multiple times to pay for the cost of these facilities. We can ﬁx this by only opening a subset T ′ of T such that each client contributes to the cost of at most one facility in T ′ . If we do this in such a way that clients not neighboring a facility in T ′ are nonetheless not too far away from a facility in T ′ , we can get a good performance guarantee for the algorithm. We now give the primaldual algorithm in Algorithm 7.8. We generate a maximal dual solution by increasing the dual variables vj . We let S be the set of clients whose duals we are increasing, and T be the set of facilities whose dual inequality is tight. Initially S = D and T = ∅. We increase vj uniformly for all j ∈ S. Once vj = cij for some i, we increase wij uniformly with vj . We increase vj until one of two things happens: either j becomes a neighbor of a facility in T , or a dual inequality becomes tight for some facility i. In the ﬁrst case, we remove j from S, and in the second case we add i to T , and remove all neighboring clients N (i) from S. Once S is empty and every client neighbors a facility in T , we select a subset T ′ of T by iteratively picking an arbitrary i ∈ T , then deleting all facilities i′ in T such that there exists some client j that contributes to both i and i′ . We then open all facilities in T ′ and assign every client to the closest open facility. We claim the following lemma about the set of facilities T ′ and the dual solution (v, w) produced by the algorithm. It shows that if a client does not have a neighbor in T ′ , then it is not far away from a facility in T ′ .
Lemma 7.13: If a client j does not have a neighbor in T ′ , then there exists a facility i ∈ T ′ such that cij ≤ 3vj . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.6
The uncapacitated facility location problem
183
v ← 0, w ← 0 S←D T ←∅ while S ̸= ∅ do // While not all clients neighbor a facility in T Increase vj for all j ∈ S and wij for all i ∈ N (j), j ∈ S uniformly until some j ∈ S neighbors some i ∈ T or some i ∈ / T has a tight dual inequality if some j ∈ S neighbors some i ∈ T then S ← S − {j} if i ∈ / T has a tight dual inequality then // facility i is added to T T ← T ∪ {i} S ← S − N (i) T′ ← ∅ while T ̸= ∅ do Pick i ∈ T ; T ′ ← T ′ ∪ {i} // remove all facilities h if some client j contributes to h and i T ← T − {h ∈ T : ∃j ∈ D, wij > 0 and whj > 0} Algorithm 7.8: Primaldual algorithm for the uncapacitated facility location problem. The intuition is that if a client j does not have a neighbor in T ′ , then it must have neighbored some tight facility h ∈ / T ′ such that some other client k contributed both to h and another facility i ∈ T ′ (see Figure 7.8). By applying triangle inequality we obtain the factor of 3. We defer the proof of the lemma for the moment and show that it implies a performance guarantee of 3 for the algorithm. Theorem 7.14: Algorithm 7.8 is a 3approximation algorithm for the uncapacitated facility location problem. Proof. For any client that contributes to a facility in T ′ , we assign it to this facility. Note that by construction of the algorithm, any client contributes to at most one facility in T ′ , so this assignment is unique. For clients that neighbor facilities in T ′ but do not contribute to any of them, assign each to some arbitrary neighboring facility in T ′ . Let A(i) ⊆ N (i) be the neighboring clients assigned to a facility i ∈ T ′ . Then as discussed above, the cost of opening the facilities in T ′ plus the cost of assigning the neighboring clients is ∑ i∈T ′
fi +
∑ j∈A(i)
cij =
∑ ∑ i∈T ′
(wij + cij ) =
j∈A(i)
∑ ∑ i∈T ′
vj ,
j∈A(i)
∑ where the ﬁrst equality holds because i ∈ T ′ implies j∈D wij = fi and wij > 0 implies j ∈ A(i). ∪ Let Z be the set of all clients not neighboring a facility in T ′ , so that Z = D − i∈T ′ A(i). We have by Lemma 7.13 that the cost of assigning any j ∈ Z to some facility in T ′ is at most 3vj . Thus the total assignment cost for these clients is at most 3
∑
vj .
j∈Z
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
184
The primaldual method h∈ / T0 whk > 0 → vk > chk vj ≥ chj k j
wik > 0 → vk > cik
i ∈ T0
Figure 7.8: Proof of Lemma 7.13. Putting everything together, we have that the cost of the solution is at most ∑ ∑ ∑ ∑ vj + 3 vj ≤ 3 vj ≤ 3 OPT, i∈T ′ j∈A(i)
j∈Z
j∈D
where the ﬁnal inequality follows by weak duality. Now we ﬁnish the proof of the lemma. Proof of Lemma 7.13. Let j be an arbitrary client that does not neighbor a facility in T ′ . During the course of the algorithm, we stopped increasing vj because j neighbored some h ∈ T . Obviously h ∈ / T ′ , since otherwise j would neighbor a facility in T ′ . The facility h must have been removed from T ′ during the ﬁnal phase of the algorithm because there exists another client k such that k contributes to both h and another facility i ∈ T ′ . See Figure 7.8. We would now like to show that the cost of assigning j to this facility i is at most 3vj . In particular, we will show that each of the three terms in chj + chk + cik is no more than vj , which will then prove by the triangle inequality that cij ≤ 3vj . We know that chj ≤ vj simply because j neighbors h. Now consider the point in the algorithm at which we stop increasing vj . By our choice of h, at this point in the algorithm either h is already in T or the algorithm adds h to T . Because client k contributes to facility h, it must be the case that either vk has already stopped increasing or we stop increasing it at the same point that we stop increasing vj . Because the dual variables are increased uniformly, we must have that vj ≥ vk . Since client k contributes to both facilities h and i, we know that vk ≥ chk and vk ≥ cik . Thus vj ≥ vk ≥ chk and vj ≥ vk ≥ cik , as claimed.
7.7
Lagrangean relaxation and the kmedian problem
In this section, we look at a variant of the uncapacitated facility location problem called the kmedian problem. As in the uncapacitated facility location problem, we are given as input a set of clients D and a set of facilities F , with assignment costs cij for all facilities i ∈ F and clients j ∈ D. However, there are no longer costs for opening facilities; instead, we are given as input a positive integer k that is an upper bound on the number of facilities that can be opened. The goal is to select a subset of facilities of at most k facilities to open and an assignment of clients to open facilities so as to minimize the total assignment costs. As before, we assume that the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.7
Lagrangean relaxation and the kmedian problem
185
clients and facilities are points in a metric space, and the assignment cost cij is the distance between client j and facility i. Since the facilities are points in a metric space, we also have distances between pairs of facilities, a fact we will use in our algorithm. For facilities h, i ∈ F , let chi denote the distance between h and i. An alternate perspective on the kmedian problem is that it is a type of clustering problem. In Section 2.2, we saw the kcenter problem, in which we wished to ﬁnd k clusters of a set of vertices. Each cluster was deﬁned by a cluster center; each vertex assigned itself to the closest cluster center, and the goal was to ﬁnd a set of k cluster centers such that the maximum distance of a vertex to its cluster center was minimized. In the kmedian problem, the facilities correspond to potential cluster centers, and the clients correspond to vertices. As in the kcenter problem, we choose k cluster centers, and each vertex assigns itself to the closest cluster center. However, rather than trying to minimize the maximum distance of a vertex to its cluster center, we minimize the sum of the distances of the vertices to their cluster centers. For the rest of this section, we will discuss the kmedian problem in terms of a facility location problem; since the clustering problem is completely equivalent, this is just a choice of terminology. We can formulate the kmedian problem as an integer program very similar to the one used for the uncapacitated facility location in Section 4.5. If we let yi ∈ {0, 1} indicate whether we open facility i, then in order to limit the number of open facilities to k, we introduce the ∑ constraint i∈F yi ≤ k. This gives the following integer programming formulation: ∑ minimize cij xij i∈F,j∈D
subject to
∑
∀j ∈ D,
xij = 1,
i∈F
xij ≤ yi , ∑ yi ≤
∀i ∈ F, j ∈ D, k,
i∈F
xij ∈ {0, 1} , yi ∈
∀i ∈ F, j ∈ D,
{0, 1} ,
∀i ∈ F.
The only diﬀerences from the uncapacitated facility location integer program of Section 4.5 is the extra constraint and the objective function, which has no facility costs. We use the idea of Lagrangean relaxation to reduce the kmedian problem to the uncapacitated facility location problem. In Lagrangean relaxation, we eliminate complicating constraints but add penalties for their violation to the objective function. For example, consider the linear programming relaxation of the integer program for the kmedian problem: ∑ minimize cij xij (7.7) i∈F,j∈D
subject to
∑
xij = 1,
∀j ∈ D,
i∈F
xij ≤ yi , ∑ yi ≤ k,
∀i ∈ F, j ∈ D,
i∈F
xij ≥ 0 yi ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
186
The primaldual method
To make the problem more closely resemble the uncapacitated facility location(∑ problem, we) ∑ would like to get rid of the constraint i∈F yi ≤ k. To do this, we add a penalty λ i∈F yi − k to the objective function for some constant λ ≥ 0. The penalty term favors solutions that obey the constraint. So our new linear program is then minimize
∑
cij xij +
∑
i∈F,j∈D
λyi − λk
i∈F
∑
subject to
(7.8)
xij = 1,
∀j ∈ D,
xij ≤ yi ,
∀i ∈ F, j ∈ D,
xij ≥ 0,
∀i ∈ F, j ∈ D,
i∈F
yi ≥ 0,
∀i ∈ F.
First, observe that any feasible solution for the linear programming relaxation (7.7) of the kmedian problem is also feasible for this linear program (7.8). Also, for any λ ≥ 0, any feasible solution for the linear programming relaxation of the kmedian problem (7.7) has an objective function value in this linear program (7.8) at most that of its value in (7.7). Therefore, this linear program (7.8) gives a lower bound on the cost of an optimal solution to the kmedian problem. We will denote the optimum cost of the kmedian problem as OPTk . Observe that except for the constant term of −λk, the linear program now looks exactly like the linear programming relaxation of the uncapacitated facility location problem in which each facility cost fi = λ. If we take the dual of this linear program, we obtain maximize
∑
vj − λk
j∈D
subject to
∑
(7.9)
wij ≤ λ,
∀i ∈ F,
j∈D
vj − wij ≤ cij , wij ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F, j ∈ D.
Again, this is the same as the dual of the linear programming relaxation of the uncapacitated facility location problem except that each facility cost is λ and there is an extra constant term of −λk in the objective function. We would like to use the primaldual algorithm for the uncapacitated facility location problem from the previous section in which all facility costs fi are set to λ for some choice of λ ≥ 0. While this will open some set of facilities, how can we then obtain a performance guarantee? In the previous section, we show that the algorithm opens a set S of facilities and constructs a feasible dual (v, w) so that ∑ ∑ ∑ min cij + fi ≤ 3 vj . j∈D
i∈S
For notational convenience, let c(S) = claim can be strengthened slightly to
i∈S
∑ j∈D
c(S) + 3
j∈D
mini∈S cij . In Exercise 7.8, we observe that this
∑ i∈S
fi ≤ 3
∑
vj .
j∈D
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.7
Lagrangean relaxation and the kmedian problem
187
Substituting fi = λ, and rearranging, this gives us that c(S) ≤ 3
∑
vj − λS .
(7.10)
j∈D
Note that if, by chance, the algorithm opens a set of facilities S such that S = k, we then have that ∑ c(S) ≤ 3 vj − λk ≤ 3 · OPTk . j∈D
This follows since (v, w) is a feasible solution to the dual program (7.9), ∑which is a lower bound on the cost of an optimal solution to the kmedian problem, and since j∈D vj − λk is the dual objective function. A natural idea is to try to ﬁnd some value of λ such that the facility location algorithm opens a set of facilities S with S = k. We will do this via a bisection search. To initialize the search, we need two initial values of λ, one for which the facility location algorithm opens at least k facilities, and one for which it opens at most k facilities. Consider what happens with the facility location algorithm when λ = 0. If the algorithm opens fewer than k facilities, then we can open an additional k − S facilities at no cost, and apply the previous reasoning to get a solution of cost at most 3 OPTk . So we assume that with 0, the algorithm opens more ∑λ =∑ than k facilities. It is also not hard to show that if λ = j∈D i∈F cij , then the algorithm opens a single facility. Thus ∑ we∑can run the bisection search on λ as follows. We initially set λ1 = 0 and λ2 = j∈D i∈F cij ; as discussed above, these two values of λ return solutions S1 and S2 (respectively) in which S1  > k and S2  < k. We run the algorithm on the value λ = 21 (λ1 + λ2 ). If the algorithm returns a solution S with exactly k facilities, then by the arguments above, we are done, and we have a solution of cost at most 3 OPTk . If S has more than k facilities, then we set λ1 = λ and S1 = S, otherwise S has fewer than k facilities and we set λ2 = λ and S2 = S. We then repeat until either the algorithm ﬁnds a solution with exactly k facilities or the interval λ2 − λ1 becomes suitably small, at which point we will be able to obtain a solution to the kmedian problem by appropriately combining the solutions from S1 and S2 . If we let cmin be the smallest nonzero assignment cost, we run the bisection search until either the algorithm ﬁnds a solution with exactly k facilities, or λ2 −λ1 ≤ ϵcmin /F . In the latter case, we will use S1 and S2 to obtain a solution S in polynomial time such that S = k and c(S) ≤ 2(3 + ϵ) OPTk . This will give us a 2(3 + ϵ)approximation algorithm for the kmedian problem. If we have not terminated with a solution with exactly k facilities, the algorithm terminates with solutions S1 and S2 and( corresponding )dual solutions (v 1 , w1 ) and (v 2 , w2 ) such that ∑ ℓ S1  > k > S2  and c(Sℓ ) ≤ 3 j∈D vj − λℓ k for ℓ = 1, 2 by inequality (7.10). Also, by the termination condition, λ2 − λ1 ≤ ϵcmin /F . Without loss of generality, we can assume that 0 < cmin ≤ OPTk , since otherwise OPTk = 0; we leave it as an exercise to show that we can ﬁnd an optimal solution to the kmedian problem in polynomial time if OPTk = 0 (Exercise 7.9). ∑ F  cij Note that the binary search on λ makes O(log ϵcmin ) calls to the facility location algorithm, and thus the overall algorithm runs in polynomial time. We will now show how we can use the two solutions S1 and S2 to obtain a solution S in polynomial time such that S = k and c(S) ≤ 2(3 + ϵ) OPTk . To do this, we ﬁrst need to relate the costs of the two solutions to OPTk . Let α1 and α2 be such that α1 S1  + α2 S2  = k, Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
188
The primaldual method
α1 + α2 = 1, with α1 , α2 ≥ 0. Note that this implies that α1 =
k − S2  S1  − k and α2 = . S1  − S2  S1  − S2 
We can then get a dual solution (˜ v , w) ˜ by letting v˜ = α1 v 1 + α2 v 2 and w ˜ = α1 w1 + α2 w2 . Note that (˜ v , w) ˜ is feasible for the dual linear program (7.9) with facility costs λ2 since it is a convex combination of two feasible dual solutions. We can now prove the following lemma, which states that the convex combination of the costs of S1 and S2 must be close to the cost of an optimal solution. Lemma 7.15: α1 c(S1 ) + α2 c(S2 ) ≤ (3 + ϵ) OPTk . Proof. We ﬁrst observe that c(S1 ) ≤ 3 = 3 = 3 ≤ 3
∑
vj1 − λ1 S1 
j∈D
∑
vj1 − (λ1 + λ2 − λ2 )S1 
j∈D
∑
vj1 − λ2 S1  + (λ2 − λ1 )S1 
j∈D
∑
vj1 − λ2 S1  + ϵ OPTk ,
j∈D
where the last inequality follows from our bound on the diﬀerence λ2 − λ1 . Now if we take the convex combination of the inequality above and our bound on c(S2 ), we obtain ∑ α1 c(S1 ) + α2 c(S2 ) ≤ 3α1 vj1 − λ2 S1  + α1 ϵ OPTk j∈D
+ 3α2
∑
vj2 − λ2 S2  .
j∈D
Recalling that v˜ = α1 v 1 + α2 v 2 is a dual feasible solution for facility costs λ2 , that α1 S1  + α2 S2  = k, and that α1 ≤ 1, we have that α1 c(S1 ) + α2 c(S2 ) ≤ 3
∑
v˜j − λ2 k + α1 ϵ · OPTk ≤ (3 + ϵ) OPTk .
j∈D
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.7
Lagrangean relaxation and the kmedian problem
189
Our algorithm then splits into two cases, a simple case when α2 ≥ 12 and a more complicated case when α2 < 21 . If α2 ≥ 12 , we return S2 as our solution. Note that S2  < k, and thus is a feasible solution. Using α2 ≥ 12 and Lemma 7.15, we obtain c(S2 ) ≤ 2α2 c(S2 ) ≤ 2 (α1 c(S1 ) + α2 c(S2 )) ≤ 2(3 + ϵ) OPTk , as desired. ∑ Before we give our algorithm for the remaining case, we let c(j, S) = mini∈S cij , so that j∈D c(j, S) = c(S). Now for each facility i ∈ S2 , we open the closest facility h ∈ S1 ; that is, the facility h ∈ S1 that minimizes cih . If this doesn’t open S2  facilities of S1 because some facilities in S2 are close to the same facility in S1 , we open some arbitrary facilities in S1 so that exactly S2  are opened. We then choose a random subset of k − S2  of the S1  − S2  facilities of S1 remaining, and open these. Let S be the resulting set of facilities opened. We now show the following lemma. Lemma 7.16: If α2 < 12 , then opening the facilities as above has cost E[c(S)] ≤ 2(3 + ϵ) OPTk . Proof. To prove the lemma, we consider the expected cost of assigning a given client j ∈ D to a facility opened by the randomized algorithm. Let us suppose that the facility 1 ∈ S1 is the open facility in S1 closest to j ; that is, c1j = c(j, S1 ); similarly, let 2 ∈ S2 be the open facility 2 in S2 closest to j. Note that with probability Sk−S = α1 , the facility 1 ∈ S1 is opened in 1 −S2  the randomized step of the algorithm if it has not already been opened by the ﬁrst step of the algorithm. Thus with probability at least α1 , the cost of assigning j to the closest open facility in S is at most c1j = c(j, S1 ). With probability at most 1− α1 = α2 , the facility 1 is not opened. In this case, we can at worst assign j to a facility opened in the ﬁrst step of the algorithm; in particular, we assign j to the facility in S1 closest to 2 ∈ S2 . Let i ∈ S1 be the closest facility to 2 ∈ S2 ; see Figure 7.9. Then cij ≤ ci2 + c2j by triangle inequality. We know that ci2 ≤ c12 since i is the closest facility in S1 to 2, so that cij ≤ c12 + c2j . Finally, by triangle inequality c12 ≤ c1j + c2j , so that we have cij ≤ c1j + c2j + c2j = c(j, S1 ) + 2c(j, S2 ). Thus the expected cost of assigning j to the closest facility in S is E[c(j, S)] ≤ α1 c(j, S1 ) + α2 (c(j, S1 ) + 2c(j, S2 )) = c(j, S1 ) + 2α2 c(j, S2 ). By the assumption α2 < 21 , so that α1 = 1 − α2 > 12 , we obtain E[c(j, S)] ≤ 2(α1 c(j, S1 ) + α2 c(j, S2 )). Then summing over all j ∈ D and using Lemma 7.15, we obtain E[c(S)] ≤ 2(α1 c(S1 ) + α2 c(S2 )) ≤ 2(3 + ϵ) OPTk .
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
190
The primaldual method
S2
S1 1 i 2
j Figure 7.9: The bad case of assigning clients. Client j is closest to 1 in S1 , closest to 2 in S2 , but gets assigned to i in S1 since this is the closest facility in S1 to 2 in S2 . This algorithm can be derandomized by the method of conditional expectations; we leave this as an exercise (Exercise 7.10). In Chapter 9, we will see improved algorithms for the kmedian problem using local search and greedy algorithms. In particular, in Section 9.4, we will see a dual ﬁtting greedy algorithm for the uncapacitated facility location problem that opens a set S of facilities such that c(S) + 2
∑
fi ≤ 2
i∈S
∑
vj ,
j∈D
where v is a feasible solution to the dual of the linear programming relaxation of the uncapacitated facility location problem. Then we will be able to follow the same logic as the algorithm above to get a 2(2 + ϵ)approximation algorithm for the kmedian problem. Note that it is crucial for the analysis that we have an uncapacitated facility location algorithm that returns a solution S such that ∑ ∑ c(S) + α fi ≤ α vj i∈S
j∈D
for some α. If this is the case, then when we set fi = λ, we are able to derive that c(S) ≤ α
∑
vj − λS ,
j∈D
which then allows us to use as a bound the objective function of the dual of the Lagrangean relaxation. Such algorithms are called Lagrangean multiplier preserving. In Chapter 14, we will see another example of a Lagrangean multiplier preserving algorithm: a primaldual 2approximation algorithm for the prizecollecting Steiner tree problem. The following hardness result is known for the kmedian problem via a reduction from the set cover problem. We discuss this result further in Section 16.2. Theorem 7.17: There is no αapproximation algorithm for the kmedian problem with constant α < 1 + 2e ≈ 1.736 unless each problem in NP has an O(nO(log log n) ) time algorithm. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.7
Lagrangean relaxation and the kmedian problem
191
Exercises 7.1 Prove that the shortest st path algorithm in Section 7.3 is equivalent to Dijkstra’s algorithm: that is, in each step, it adds the same edge that Dijkstra’s algorithm would add. 7.2 Consider the multicut problem in trees. In this problem, we are given a tree T = (V, E), k pairs of vertices si ti , and costs ce ≥ 0 for each edge e ∈ E. The goal is to ﬁnd a minimumcost set of edges F such that for all i, si and ti are in diﬀerent connected components of G′ = (V, E − F ). Let Pi be the set of edges in the unique path in T between si and ti . Then we can formulate the problem as the following integer program: ∑ minimize ce xe e∈E
subject to
∑
xe ≥ 1,
1 ≤ i ≤ k,
xe ∈ {0, 1} ,
e ∈ E.
e∈Pi
Suppose we root the tree at an arbitrary vertex r. Let depth(v) be the number of edges on the path from v to r. Let lca(si , ti ) be the vertex v on the path from si to ti whose depth is minimum. Suppose we use the primaldual method to solve this problem, where the dual variable we increase in each iteration corresponds to the violated constraint that maximizes depth(lca(si , ti )). Prove that this gives a 2approximation algorithm for the multicut problem in trees. 7.3 The local ratio technique is another technique that is highly related to the primaldual method; however, its use of duality is implicit. Consider the following local ratio algorithm for the set cover problem. As with the primaldual algorithm, we compute a collection I of indices of a set cover, where I is initially empty. In each iteration, we ﬁnd some element ei not covered by the current collection I. Let ϵ be the minimum weight of any set containing ei . We subtract ϵ from the weight of each set containing ei ; some such set now has weight zero, and we add the index of this set to I. We now analyze the performance of this algorithm. To do this, let ϵj be the value of ϵ from the jth iteration of the algorithm. ∑ (a) Show that the cost of the solution returned is at most f j ϵj , where f = maxi  {j : ei ∈ Sj } . ∑ (b) Show that the cost of the optimal solution must be at least j ϵj . (c) Conclude that the algorithm is an f approximation algorithm. In its most general application, the local ratio technique depends upon the local ratio theorem, stated below. For a minimization problem Π with weights w, we say that a feasible solution F is αapproximate with respect to w if the weight of F is at most α times the weight of an optimal solution given weights w. Then the local ratio theorem states that if we have nonnegative weights w such that w = w1 + w2 , where w1 and w2 are also nonnegative weights, and we have a feasible solution F such that F is αapproximate with respect to both w1 and w2 , then F is αapproximate with respect to w. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
192
The primaldual method
(d) Prove the local ratio theorem. (e) Explain how the set cover algorithm above can be analyzed in terms of the local ratio theorem to prove that it is an f approximation algorithm. Most of the algorithms of this chapter have local ratio variants. 7.4 In the 2approximation algorithm for the generalized Steiner tree problem that we gave in Section 7.4, we ﬁrst add certain edges, then remove unnecessary edges in the order opposite of the order in which they were added. Prove that one can in fact remove unnecessary edges in any order and still obtain a 2approximation algorithm for the problem. That is, we replace the edge removal steps in Algorithm 7.6 with a step that checks if there exists any edge e in F ′ such that F ′ − e is feasible. removed from F ′ , and if not, F ′ is returned as the ﬁnal solution. Prove ∑ If so, e is ∑ that e∈F ′ ce ≤ 2 S yS for the dual y generated by the algorithm. 7.5 In the minimumcost branching problem we are given a directed graph G = (V, A), a root vertex r ∈ V , and weights wij ≥ 0 for all (i, j) ∈ A. The goal of the problem is to ﬁnd a minimumcost set of arcs F ⊆ A such that for every v ∈ V , there is exactly one directed path in F from r to v. Use the primaldual method to give an optimal algorithm for this problem. 7.6 Recall that in our algorithms of Sections 4.4 and 5.7 for the prizecollecting Steiner tree problem, we used the following linear programming relaxation of the problem: ∑ ∑ minimize ce xe + πi (1 − yi ) e∈E
subject to
i∈V
∑
xe ≥ y i ,
∀i ∈ S, ∀S ⊆ V − r, S ̸= ∅,
e∈δ(S)
yr = 1, yi ≥ 0,
∀i ∈ V,
xe ≥ 0,
∀e ∈ E.
Given an optimal solution (x∗ , y ∗ ) to the linear program, we then selected a set of vertices U such that U = {i ∈ V : yi∗ ≥ α} for some value of α > 0. Give a primaldual algorithm that ﬁnds a Steiner tree T on the set of terminals U such that ∑ 2∑ ce ≤ ce x∗e . α e∈T
e∈E
(Hint: you should not need to design a new primaldual algorithm.) 7.7 In the kpath partition problem, we are given a complete, undirected graph G = (V, E) with edge costs ce ≥ 0 that obey the triangle inequality (that is, c(u,v) ≤ c(u,w) + c(w,v) for all u, v, w ∈ V ), and a parameter k such that V  is a multiple of k. The goal is to ﬁnd a minimumcost collection of paths of k vertices each such that each vertex is on exactly one path. A related problem is that of partitioning a graph into 0(mod k)trees. The input to this problem is the same as that above, except that the graph is not necessarily complete and Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
7.7
Lagrangean relaxation and the kmedian problem
193
edge costs do not necessarily obey the triangle inequality. The goal is to ﬁnd a minimumcost collection of trees such that each tree has 0(mod k) vertices, and each vertex is in exactly one tree. ( ) (a) Given an αapproximation algorithm for the second problem, produce a 2α 1 − k1 approximation algorithm for the ﬁrst. (b) Use the primaldual method to give a 2approximation algorithm for the second problem. ( ) (c) Give a 4 1 − k1 approximation algorithm for the problem of partitioning a graph into cycles of length exactly k. 7.8 Show that the performance guarantee of the primaldual algorithm for the uncapacitated facility location algorithm in Section 7.6 can be strengthened in the following way. Suppose that the algorithm opens the set T ′ of facilities and constructs the dual solution (v, w). Show that ∑ ∑ ∑ vj . min′ cij + 3 fi ≤ 3 j∈D
i∈T
i∈T ′
j∈D
7.9 Show that for the kmedian problem as deﬁned in Section 7.7, the optimal solution can be found in polynomial time if the optimum cost OP Tk = 0. 7.10 By using the method of conditional expectations, show that the randomized algorithm for choosing k facilities in the kmedian algorithm of Section 7.7 can be made deterministic.
Chapter Notes The primaldual method for approximation algorithms is a generalization of the primaldual method used for linear programming and combinatorial optimization problems such as the shortest st path problem, the maximum ﬂow problem, the assignment problem, the minimumcost branching problem, and others. For an overview of the primaldual method and its application to these problems, see Papadimitriou and Steiglitz [238]. Edmonds [95] gives the primaldual algorithm for the minimumcost branching problem in Exercise 7.5. The idea of Section 7.3 that the shortest st path problem can be solved by an algorithm that greedily increases dual variables is due to Hoﬀman [168]. Dijkstra’s algorithm for the same problem is due, of course, to Dijkstra [88]. The ﬁrst use of the primaldual method for approximation algorithms is due to BarYehuda and Even [35]; they gave the algorithm of Section 7.1 for the set cover problem. Work in primaldual approximation algorithms was revived by work on the generalized Steiner tree problem of Section 7.4. The ﬁrst 2approximation algorithm for the generalized Steiner tree problem is due to Agrawal, Klein, and Ravi [4], and the algorithm presented in Section 7.4 is essentially that of [4]. The use of linear programming and LP duality in the algorithm was made explicit Goemans and Williamson [138], who extended the technique to other problems (such as the kpath partition problem of Exercise 7.7). The idea of depicting dual variables as moats is due to J¨ unger and Pulleyblank [180]. Several uses of the primaldual method for approximation algorithms then followed. BarYehuda, Geiger, Naor, and Roth [37] gave the feedback vertex set algorithm of Section 7.2, using Lemma 7.3, which is due to Erd˝os and P´osa [100]. Jain and Vazirani [177] developed the primaldual algorithm for the uncapacitated facility location problem and the use of Lagrangean Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
194
The primaldual method
relaxation for the kmedian problem in Sections 7.6 and 7.7. Carnes and Shmoys [61] gave the primaldual algorithm for the minimum knapsack problem in Section 7.5; their algorithm uses an integer programming formulation of the problem due to Carr, Fleischer, Leung, and Phillips [62], who gave an LProunding 2approximation algorithm for the problem based on their formulation. Surveys of the primaldual method for approximation algorithms are given by Bertsimas and Teo [46], Goemans and Williamson [140], and Williamson [288]. The local ratio technique of Exercise 7.3 is due to BarYehuda and Even [36]. All of the algorithms in Sections 7.1 through 7.4 are known to have local ratio equivalents. A formal proof of the equivalence of the primaldual method and the local ratio technique for a deﬁned set of problems is given in BarYehuda and Rawitz [38]. Surveys of the local ratio technique have been given by BarYehuda [33], BarYehuda, Bendel, Freund, and Rawitz [34], and BarYehuda and Rawitz [38]. The hardness result for the kmedian problem in Theorem 7.17 is due to Jain, Mahdian, Markakis, Saberi, and Vazirani [176], following work of Guha and Khuller [146] for the hardness of the uncapacitated facility location problem. The result of Exercise 7.2 is due to Garg, Vazirani, and Yannakakis [127]. The results of Exercises 7.4 and 7.7 are from Goemans and Williamson [138]. The result of Exercise 7.8 is from Jain and Vazirani [177]; Exercise 7.10 is also from this paper.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
Chapter 8
Cuts and metrics
In this chapter, we think about problems involving metrics. A metric (V, d) on a set of vertices V gives a distance duv for each pair of vertices u, v ∈ V such that three properties are obeyed: (1) duv = 0 if and only if v = u; (2) duv = dvu for all u, v ∈ V and (3) duv ≤ duw + dwv for all u, v, w ∈ V . The ﬁnal property is sometimes called the triangle inequality. We will sometimes simply refer to the metric d instead of (V, d) if the set of vertices V is clear from the context. A concept related to a metric is a semimetric (V, d), in which properties (2) and (3) are obeyed, but not necessarily (1), so that if duv = 0, then possibly u ̸= v (a semimetric maintains that duu = 0). We may sometimes ignore this distinction between metrics and semimetrics, and call them both metrics. Metrics turn out to be a useful way of thinking about graph problems involving cuts. Many important problems in discrete optimization require ﬁndings cuts in graphs of various types. To see the connection between cuts and metrics, note that for any cut S ⊆ V , we can deﬁne d where duv = 1 if u ∈ S and v ∈ / S, and duv = 0 otherwise. Note that (V, d) is then a semimetric; it is sometimes called the cut semimetric associated with S. Then a problem in a graph G = (V, E) in which we are trying to ﬁnd a cut to minimize or maximize the sum of weights we of the ∑ edges e in the cut becomes one of ﬁnding a cut semimetric that minimizes or maximizes e=(u,v)∈E we duv . In several examples in this chapter, we set up a linear programming relaxation with variables duv in which we have constraints on d corresponding to the properties of a semimetric. Then we use the metric properties of duv to help ﬁnd the desired cut. In many cases, we consider for some vertex s ∈ V all the vertices within some small distance r of s (using the LP variables duv as distances) and put them on the same side of the cut and all other vertices on the other side; we view this as taking a ball of radius r around s. This is a technique we will use extensively. We will also consider the notion of approximating a given metric (V, d) by a metric of a simpler form. In particular, we will consider tree metrics; tree metrics are metrics that are deﬁned by shortest paths in a tree. Sometimes we wish to approximate problems involving distances in a graph, in which the problem becomes straightforward in a tree metric. We can sometimes get good approximation algorithms by ﬁrst approximating the graph distances by a tree metric, then solving the problem easily in the tree metric. We begin the chapter by considering a number of diﬀerent cut problems. We start in Section 8.1 with the multiway cut problem, and ﬁrst show that a simple algorithm involving ﬁnding 195
196
Cuts and metrics
a number of minimum st cuts gives a 2approximation algorithm. We then show that the randomized rounding of an LP solution along the lines described above improves this to a 32 approximation algorithm. We consider the multicut problem in Section 8.3, which also uses an LP relaxation and a rounding technique as described above. For this problem, we introduce an important technique called “region growing” that relates the edges in the cut formed by a ball to the value of the LP solution on edges inside the ball. In the following section, we apply the region growing technique to the problem of ﬁnding small balanced cuts; balanced cuts are ones in which the two parts of the cut have roughly equal numbers of vertices. The ﬁnal three sections of the chapter discuss the technique of approximating metrics by tree metrics, and present applications of this technique to the buyatbulk network design problem and the linear arrangement problem.
8.1
The multiway cut problem and a minimumcutbased algorithm
We begin by considering a simple variety of cut problem, and give an approximation algorithm that does not require using a linear programming relaxation. We then show that using a linear programming relaxation and treating its solution as a metric on the set of vertices gives a better performance guarantee. The problem is known as the multiway cut problem. We are given an undirected graph G = (V, E), costs ce ≥ 0 for all edges e ∈ E, and k distinguished vertices s1 , . . . , sk . The goal is to remove a minimumcost set of edges F such that no pair of distinguished vertices si and sj for i ̸= j are in the same connected component of (V, E − F ). One application of this problem arises in distributed computing. Each vertex represents an object, and an edge e between them of cost ce represents the amount of communication between the objects. The objects need to be partitioned to reside on k diﬀerent machines, with special object si needing to reside on the ith machine. The goal is to partition the objects onto the k machines in such a way that communication between the machines is minimized. Our algorithm for the multiway cut problem begins with some observations about the structure of any feasible solution F . Given a feasible F , let Ci be the set of vertices reachable in (V, E − F ) from each distinguished vertex si . Let Fi = δ(Ci ), where δ(S) is the set of all edges in E with exactly one endpoint in S. Observe that each Fi is a cut separating si from s1 , . . . , si−1 , si+1 , . . . , sk . We call Fi an isolating cut: it isolates si from the other distinguished vertices. Observe also that some edges might appear in two diﬀerent Fi : an edge e can have one endpoint in Ci and the other in Cj for some j ̸= i, so that e ∈ Fi and e ∈ Fj . Our algorithm will compute a minimum isolating cut Fi between si and s1 , . . . , si−1 , si+1 , . . . , sk for each i: we can do this by adding a sink vertex t to the graph with inﬁnite cost edges from the distinguished vertices (other than si ) to the sink, and then computing a minimum si t cut. ∪ We return as our solution ki=1 Fi . Theorem 8.1: The algorithm of computing a minimum cut between each si and the other distinguished vertices is a 2approximation algorithm for the multiway cut problem. Proof. As above, let Fi be the minimum isolating cut between si and the other distinguished ∗ vertices. Let F ∗ be an optimal solution, and let F∑ i be the isolating cut in the optimal solution for si . For a subset of edges A ⊆ E, let c(A) = e∈A ce . Because Fi is a minimum isolating cut for si , we know that c(Fi ) ≤ c(Fi∗ ). Hence the cost of the solution of the algorithm is at ∑ ∑ most ki=1 c(Fi ) ≤ ki=1 c(Fi∗ ). We observed above that each edge in the optimal solution F ∗ Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.2
The multiway cut problem and an LP rounding algorithm
197
can be in at most two Fi∗ , so that k ∑
c(Fi ) ≤
i=1
k ∑
c(Fi∗ ) ≤ 2c(F ∗ ) = 2 OPT .
i=1
By being only slightly cleverer, we can improve the performance guarantee a little bit. Without loss of generality, let∪Fk be the costliest cut of F1 , . . . , Fk . Note that the union of the ﬁrst k − 1 isolating cuts, F = k−1 i=1 Fi , is also a feasible solution for the problem: if sk can reach any other distinguished vertex si in (V, E − F ), then Fi was not an isolating cut for si . Then we have the following. Corollary 8.2: The algorithm of returning the cheapest k − 1 minimum isolating cuts is a (2 − k2 )approximation algorithm for the multiway cut problem. Proof. We use the same notation as in the ∑ proof of the theorem above. Observe that the cost of our new solution F is at most (1 − k1 ) ki=1 c(Fi ). Thus its cost is at most (
8.2
1 1− k
)∑ k i=1
( c(Fi ) ≤
1 1− k
)∑ k i=1
c(Fi∗ )
(
1 ≤2 1− k
) OPT .
The multiway cut problem and an LP rounding algorithm
We now show that one can obtain a better approximation algorithm for the multiway cut problem via LP rounding. First, we need to strengthen some of our observations above. We noted that for any feasible solution F to the problem, we can compute sets Ci of vertices reachable from each distinguished vertex si . We claim for any minimal solution F , the Ci must be a partition of the vertices V . To see this, suppose we are given some solution F such that the corresponding sets Ci do not partition V . Let S be all vertices not∪reachable from any si . Pick some j arbitrarily and add S to Cj . Let the new solution be F ′ = ki=1 δ(Ci ). Then we claim that F ′ ⊆ F . Observe that for any i ̸= j, δ(Ci ) is in F . Furthermore, any edge e ∈ δ(Cj ) must have some endpoint in some Ci with i ̸= j. Thus e ∈ δ(Ci ) and is in F also. Therefore, another way of looking at the multiway cut problem is ﬁnding an ∪ optimal partition of V into sets Ci such that si ∈ Ci for all i and such that the cost of F = ki=1 δ(Ci ) is minimized. Given this perspective, we formulate the problem as an integer program. For each vertex u ∈ V , we have k diﬀerent variables, xiu . The variable xiu = 1 if u is assigned to the set Ci and is 0 otherwise. We create a variable zei which will be 1 if e ∈ δ(Ci ) and 0 otherwise. Since if e ∈ δ(Ci ), it is also the case that e ∈ δ(Cj ) for some j ̸= i, the objective function of the integer program is then k 1∑ ∑ i ce ze ; 2 e∈E
i=1
∪ this will give exactly the cost of the edges in the solution F = ki=1 δ(Ci ) for the assignment of vertices to sets Ci given by the variables xiu . Now we consider constraints for the program. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
198
Cuts and metrics
Obviously si must be assigned to Ci , so we have xisi = 1 for all i. To enforce that zei = 1 when e ∈ δ(Ci ) for e = (u, v), we add constraints zei ≥ xiu − xiv and zei ≥ xiv − xiu ; this enforces that z i ≥ xiu − xiv . Since the integer program will minimize the objective function and zei appears with a nonnegative coeﬃcient in the objective function, at optimality we will have zei = xiu −xiv . Thus zei = 1 if one of the two endpoints of the edge e = (u, v) is assigned to the set Ci and the other is not. Then the overall integer program is as follows: 1∑ ∑ i minimize ze ce 2 k
e∈E
k ∑
subject to
(8.1)
i=1
xiu = 1,
∀u ∈ V,
zei ≥ xiu − xiv ,
∀e = (u, v) ∈ E,
zei ≥ xiv − xiu ,
∀e = (u, v) ∈ E,
(8.2)
i=1
xisi xiu
= 1,
i = 1, . . . , k,
∈ {0, 1},
∀u ∈ V, i = 1, . . . , k.
The linear programming relaxation of this integer program is closely connected with the ℓ1 metric for measuring distances in Euclidean space. Let x, y ∈ ℜn , and suppose that xi , y i are the ith coordinates of x and y. Then the ℓ1 metric is as follows. n Deﬁnition 8.3: Given ∑n x, y i∈ ℜ i , the ℓ1 metric is a metric such that the distance between x and y is ∥x − y∥1 = i=1 x − y .
We relax the integer program above to a linear program by replacing the integrality condition ∈ {0, 1} with xiu ≥ 0. Observe then that the linear program can be given a much more compact formulation. The variable xiu is the ith coordinate of a vector xu . Each xu ∈ ℜk ; k in ∑kfact,i because of constraint (8.2), each xu lies in the ksimplex ∆k , where ∆k = {x ∈ ℜ : ei is the vector with 1 in the i=1 x = 1}. Each distinguished vertex si has xsi = ei , where ∑ ∑k i i ith coordinate and zeroes elsewhere. Finally, we observe that ki=1 zei = i=1 xu − xv  = ∥xu − xv ∥1 , which is just the distance between the points xu and xv under the ℓ1 metric. Thus the linear programming relaxation becomes xiu
minimize subject to
1 2
∑
ce ∥xu − xv ∥1
(8.3)
e=(u,v)∈E
xsi = ei ,
i = 1, . . . , k,
xu ∈ ∆k ,
∀u ∈ V.
In Figure 8.1, we give a geometric representation of an instance with k = 3. We will give an approximation algorithm by the randomized rounding of the solution of the linear program. In particular, we will take all vertices that are close to the distinguished vertex si in the LP solution and put them in Ci . For any r ≥ 0 and 1 ≤ i ≤ k, let B(ei , r) be the set of vertices corresponding to the points }xu in a ball of radius r in the ℓ1 metric around ei ; that { is, B(ei , r) = u ∈ V : 12 ∥ei − xu ∥1 ≤ r . We will sometimes write B(si , r) instead of B(ei , r). We include the factor of 1/2 in the deﬁnition so that all vertices are within a ball of radius 1: B(ei , 1) = V for all i. See Figure 8.1 for an illustration. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.2
The multiway cut problem and an LP rounding algorithm
199
s3 (0,0,1)
s1 (1,0,0)
s2 (0,1,0)
Figure 8.1: A geometric representation of an LP solution for k = 3. The distinguished vertices s1 , s2 , and s3 are given by the coordinates (1, 0, 0), (0, 1, 0), and (0, 0, 1) respectively. Any other vertex lies in the triangle deﬁned by these three coordinates. The dotted line represents a ball around s1 . Let x be an LP solution to (8.3) for all 1 ≤ i ≤ k do Ci ← ∅ Pick r ∈ (0, 1) uniformly at random Pick a random permutation π of {1, . . . , k} X←∅ // X keeps track of all currently assigned vertices for i ← 1 to k − 1 do Cπ(i) ← B(sπ(i) , r) − X X ← X ∪ Cπ(i) Cπ(k) ← V − X ∪ return F = ki=1 δ(Ci ) Algorithm 8.1: Randomized rounding algorithm for the multiway cut problem. We now consider Algorithm 8.1. The algorithm selects r ∈ (0, 1) uniformly at random, and a random permutation π of the indices {1, . . . , k}. The algorithm proceeds through the indices in the order given by the permutation. For index π(i) in the ordering, the algorithm assigns all previously unassigned vertices in B(sπ(i) , r) to Cπ(i) . At the end of the order, any unassigned vertices are assigned to Cπ(k) . See Figure 8.2 for an example. Before we get into the details of the proof, let’s look at an example that will show why picking a random radius and a random permutation π are useful in giving a good performance guarantee. To simplify matters, suppose that k is large, and suppose we have an edge (u, v) where xv = (0, 1, 0, 0, . . .) and xu = ( 12 , 12 , 0, 0, . . .); see Figure 8.3 for an illustration. We suppose that xs1 = (1, 0, 0, . . .) and xs2 = (0, 1, 0, . . .). Note that in this case, u can only be in the balls B(s1 , r) or B(s2 , r) since r < 1 and 12 ∥ei − xu ∥1 = 1 for i ̸= 1, 2. Thus u can only be assigned to C1 , C2 , or Cπ(k) , with the last case occurring if xu ∈ / B(s1 , r) and xu ∈ / B(s2 , r). Somewhat similarly, v can only be assigned to C1 or C2 ; in this case, xv is always in B(s2 , r) since r > 0 and ∥e2 − xv ∥1 = 0. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
200
Cuts and metrics
s3 (0,0,1)
s2 (0,1,0)
s1 (1,0,0)
Figure 8.2: A geometric representation of the algorithm for k = 3. The random permutation is 1,3,2. The ball around s3 assigns to C3 all vertices in the ball not already assigned to C1 . Now, suppose we have a ﬁxed permutation π. If the permutation orders s1 before s2 , and s2 is not last in the permutation, then we claim (u, v) enters the cut with probability ∥xu − xv ∥1 = 1. To see this, note that if 1/2 ≤ r < 1, then u ∈ B(s1 , r) but v ∈ / B(s1 , r) so that u ∈ C1 and v ∈ C2 . If 0 < r < 1/2, then v ∈ B(s2 , r) but u ∈ / B(s2 , r), so that v ∈ C2 ; we observed above that u can only be assigned to C1 , C2 , and Cπ(k) , so that if u ∈ / B(s2 , r) and π(k) ̸= 2, it must be the case that u ∈ / C2 and (u, v) is in the cut. Note that if s2 is last in the permutation, this can only lower the probability that (u, v) is in the cut. In general, we can upper bound the probability that an edge (u, v) ends up in the cut by ∥xu − xv ∥1 , but analyzing the algorithm with this probability is only good enough to give a performance guarantee of 2, since the contribution of edge e = (u, v) to the objective function is 21 ce ∥xu − xv ∥1 . However, if the permutation π orders s2 before s1 , then the edge is in the cut if 0 < r < 1/2, since v ∈ C2 but u ∈ / C2 as before, while if r > 1/2, then the edge cannot end up in the cut at all because both u and v are in B(s2 , r) and hence both are assigned to C2 . Since the probability that s2 is ordered before s1 is 1/2 in a random permutation, the overall probability that (u, v) is in the cut is at most Pr[(u, v) in cut s1 before s2 ] Pr[s1 before s2 ] + Pr[(u, v) in cut s2 before s1 ] Pr[s2 before s1 ] 1 1 1 ≤ ∥xu − xv ∥1 · + ∥xu − xv ∥1 · 2 2 2 3 1 = · ∥xu − xv ∥1 , 2 2 which will give us a performance guarantee of 3/2. To start the analysis, we need some lemmas that will be helpful later on. The ﬁrst lemma observes that any coordinate accounts for at most half of the ℓ1 distance. The second lemma gives us a condition under which a vertex is in a ball of radius r. Lemma 8.4: For any index ℓ and any two vertices u, v ∈ V , 1 xℓu − xℓv  ≤ ∥xu − xv ∥1 . 2 Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.2
The multiway cut problem and an LP rounding algorithm
201
u = (1/2, 1/2, 0, . . .) s2 = v = (0, 1, 0, . . .)
s1 = (1, 0, 0, . . .)
Figure 8.3: An illustration of the ideas of the analysis. Proof. Without loss of generality, assume that xℓu ≥ xℓv . Then ∑ ∑ ∑ ∑ xju − 1 − xjv = (xjv − xju ) ≤ xju − xjv . xℓu − xℓv  = xℓu − xℓv = 1 − j̸=ℓ
j̸=ℓ
j̸=ℓ
j̸=ℓ
By adding xℓu − xℓv  to both sides, we have 2xℓu − xℓv  ≤ ∥xu − xv ∥1 , which gives the lemma. Lemma 8.5: u ∈ B(si , r) if and only if 1 − xiu ≤ r.
∑ Proof. We have u ∈ B(si , r) if and∑only if 12 ∥ei − xu ∥1 ≤ r. This is ∑ equivalent to 12 kℓ=1 eℓi − xℓu  ≤ r which is equivalent to 12 ℓ̸=i xℓu + 21 (1 − xiu ) ≤ r. Since ℓ̸=i xℓu = 1 − xiu , this is equivalent to 1 − xiu ≤ r. Theorem 8.6: Algorithm 8.1 is a randomized 32 approximation algorithm for the multiway cut problem. Proof. Pick an arbitrary edge (u, v) ∈ E. We claim that the probability that the endpoints lie in diﬀerent parts of the partition is at most 43 ∥xu − xv ∥1 . Let W be a random variable denoting the value of the cut, and let Zuv be ∑ a 01 random variable which is 1 if u and v are in diﬀerent parts of the partition, so that W = e=(u,v)∈E ce Zuv . Given the claim, we have ∑ E[W ] = E ce Zuv e=(u,v)∈E
=
∑
ce E[Zuv ]
e=(u,v)∈E
=
∑
ce · Pr[(u, v) in cut]
e=(u,v)∈E
≤
∑
e=(u,v)∈E
=
3 1 · 2 2
3 ce · ∥xu − xv ∥1 4
∑
ce ∥xu − xv ∥1
e=(u,v)∈E
3 OPT, 2 ∑ where the ﬁnal inequality follows since 12 e=(u,v)∈E ce ∥xu − xv ∥1 is the objective function of the LP relaxation. ≤
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
202
Cuts and metrics
Now to prove the claim. We say that an index i settles edge (u, v) if i is the ﬁrst index in the random permutation such that at least one of u, v ∈ B(si , r). We say that index i cuts edge (u, v) if exactly one of u, v ∈ B(si , r). Let Si be the event that i settles (u, v) and Xi be the event that i cuts (u, v). Note that Si depends on the random permutation, while Xi is independent of the random permutation. In order for (u, v) to be in the multiway cut, there must be some index i that both settles and cuts (u, v); if this happens,∑then (u, v) ∈ δ(Ci ). Thus the probability that edge (u, v) is in the multiway cut is at most ki=1 Pr[Si ∧ Xi ]. We ∑ will now show that ki=1 Pr[Si ∧ Xi ] ≤ 34 ∥xu − xv ∥1 . By Lemma 8.5, Pr[Xi ] = Pr[min(1 − xiu , 1 − xiv ) ≤ r < max(1 − xiu , 1 − xiv )] = xiu − xiv . Let ℓ be the index that minimizes mini (min(1 − xiu , 1 − xiv )); in other words, sℓ is the distinguished vertex that is closest to one of the two endpoints of (u, v). We claim that index i ̸= ℓ cannot settle edge (u, v) if ℓ is ordered before i in the random permutation π: by Lemma 8.5 and the deﬁnition of ℓ, if at least one of u, v ∈ B(ei , r), then at least one of u, v ∈ B(eℓ , r). Note that the probability that ℓ occurs after i in the random permutation π is 1/2. Hence for i ̸= ℓ, Pr[Si ∧ Xi ] = Pr[Si ∧ Xi ℓ occurs after i in π] · Pr[ℓ occurs after i in π] + Pr[Si ∧ Xi ℓ occurs before i in π] · Pr[ℓ occurs before i in π] 1 ≤ Pr[Xi ℓ occurs after i in π] · + 0. 2 Since the event Xi is independent of the choice of random permutation, Pr[Xi ℓ occurs after i in π] = Pr[Xi ], and therefore for i ̸= ℓ Pr[Si ∧ Xi ] ≤ Pr[Xi ] ·
1 1 = xiu − xiv . 2 2
We also have that Pr[Sℓ ∧ Xℓ ] ≤ Pr[Xℓ ] ≤ xℓu − xℓv . Therefore, we have that the probability that (u, v) is in the multiway cut is k ∑
Pr[Si ∧ Xi ] ≤ xℓu − xℓv  +
i=1
1∑ i xu − xiv  2 i̸=ℓ
=
1 ℓ 1 xu − xℓv  + ∥xu − xv ∥1 . 2 2
Using Lemma 8.4, 12 xℓu − xℓv  ≤ 14 ∥xu − xv ∥1 , so that k ∑ i=1
3 Pr[Si ∧ Xi ] ≤ ∥xu − xv ∥1 , 4
as desired. With only slightly more work, the performance guarantee of the algorithm can be improved to this is left to Exercise 8.1. One can also obtain a 23 approximation algorithm by choosing between two ﬁxed permutations; we explore this in Exercise 8.2. The idea of partitioning vertices by taking balls of ﬁxed radius in the order given by a random permutation is a useful one which we will apply again in Section 8.5. 3 1 2−k;
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.3
The multicut problem
8.3
203
The multicut problem
In this section, we consider a slightly diﬀerent cut problem, called the multicut problem. Rather than having a set of distinguished vertices s1 , . . . , sk , we now have a set of distinguished sourcesink pairs of vertices (s1 , t1 ), . . . , (sk , tk ). Given an undirected graph G = (V, E) with nonnegative costs ce ≥ 0 for all e ∈ E, our goal is to ﬁnd a minimumcost set of edges F whose removal disconnects all pairs; that is, for every i, 1 ≤ i ≤ k, there is no path connecting si and ti in (V, E − F ). Unlike the previous problem, there can be paths connecting si and sj or si and tj for i ̸= j. We previously considered special case of the multicut problem in trees in Exercise 7.2. Given a graph G, let Pi be the set of all paths P joining si and ti . Then an integer programming formulation of the problem is: minimize
∑
ce xe
e∈E
subject to
∑
xe ≥ 1,
∀P ∈ Pi , 1 ≤ i ≤ k,
xe ∈ {0, 1},
∀e ∈ E.
e∈P
The constraints ensure that for each i, for each path P ∈ Pi , some edge is selected from P . To obtain a linear programming relaxation, we replace the constraints xe ∈ {0, 1} with xe ≥ 0. Although the formulation is exponential in the size of the input (since there could be an exponential number of paths P ∈ Pi ), we can solve the linear program in polynomial time by using the ellipsoid method described in Section 4.3. The separation oracle works as follows: given a solution x, we consider the graph G in which the length of each edge e is xe . For each i, 1 ≤ i ≤ k, we compute the length of the shortest path between si and ti . If for some i, the length ∑ of the shortest path P is less than 1, we return it as a violated constraint, since we have e∈P xe < 1 for P ∈ Pi . If for each i, the length of the shortest path between si and ti is at least 1, then the length of every path P ∈ Pi is at least 1, and the solution is feasible. Thus we have a polynomialtime separation oracle for the linear program. Alternatively, it is possible to give an equivalent, polynomiallysized linear program that can be solved directly in polynomial time, and whose solution can be transformed into a solution of this linear program in polynomial time; we leave this as an exercise (Exercise 8.4). As in the LP rounding algorithm for the multiway cut problem, we will build our solution by taking balls around the vertex si for each i. In order to do this, we must deﬁne a notion of distance. Given an optimal solution x to the LP, we will let xe denote the length of edge e for the purposes of our algorithm. We then let dx (u, v) denote the length of the shortest path from vertex u to v using xe as edge lengths. Observe that dx will obey the triangle inequality by the deﬁnition of shortest paths. Also, in any feasible LP solution we have dx (si , ti ) ≥ 1 for all i. Let Bx (si , r) = {v ∈ V : dx (si , v) ≤ r} be the ball of radius r around vertex si using these distances. Additionally, it will be useful to think of each edge e ∈ E as being a pipe with length xe and crosssectional area ce . Then the product ce xe gives the volume of edge e. The LP produces the minimumvolume system of pipes such that dx (si , ti ) ≥ 1 for all i. See Figure 8.4 for an illustration of the pipe system, and Figure 8.5 for an ∑ illustration of a ball in the pipe system. ∗ Given an optimal solution x to the LP, we let V = e∈E ce xe be the total volume of the pipes. We know that V ∗ ≤ OPT . Let Vx (si , r) be the volume of pipes within distance r of si plus an Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
204
Cuts and metrics
s2 1/2 1/2
t1 1/2
s1 1/2 1/4
1/2
1/4 t2 1/4
Figure 8.4: An illustration of a pipe system. The width of the pipe corresponds to its cost ce , and the number next to it gives its length. The distance between each source/sink pair is at least 1 along any path. extra V ∗ /k term; that is, Vx (si , r) =
V∗ + k
∑ e=(u,v):u,v∈Bx (si ,r)
ce xe +
∑
ce (r − dx (si , u)).
e=(u,v):u∈Bx (si ,r),v ∈B / x (si ,r)
The ﬁrst term ensures that Vx (si , 0) is nonzero, and that the sum of V (si , 0) over all si is V ∗ ; it will later be clear why this is useful. The second term adds up the volume of all edges (u, v) such that both u and v are inside the ball of distance r around si , while the third term adds up the volume of pipes that fall partially within the ball. Let δ(S) be the set of all edges that have exactly one endpoint in the set of vertices S. For a given radius ∑ r, let c(δ(Bx (si , r))) be the cost of the edges in δ(Bx (si , r)); that is, c(δ(Bx (si , r))) = e∈δ(Bx (si ,r)) ce . We will ﬁrst claim that it is always possible to ﬁnd some radius r < 1/2 such that the cost c(δ(Bx (si , r))) of the cut induced by the ball of radius r around si is not much more than the volume of the ball; ﬁnding a ball of this sort is sometimes called region growing. Lemma 8.7: Given a feasible solution to the linear program x, for any si one can ﬁnd in polynomial time a radius r < 1/2 such that c(δ(Bx (si , r))) ≤ (2 ln(k + 1))Vx (si , r). This leads to the following algorithm, which is summarized in Algorithm 8.2. We start out with an empty set of edges F , and we sequence through each i from 1 to k. If si and ti are not already separated by the cut F , we invoke Lemma 8.7 to ﬁnd an appropriate radius r < 1/2 around si . We then add the edges of δ(Bx (si , r)) to F . We remove all the vertices of Bx (si , r) and all incident edges from the graph and continue. We note that in any iteration, the balls Bx (si , r) and volumes Vx (si , r) are taken with respect to the edges and vertices remaining in the current graph. We ﬁrst show that the algorithm produces a feasible solution. Lemma 8.8: Algorithm 8.2 produces a feasible solution for the multicut problem. Proof. The only possible way in which the solution might not be feasible is if we have some sj tj pair in the ball Bx (si , r) when the vertices in the ball get removed from the graph. However, if Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.3
The multicut problem
si
1/4 dr
1/4
1/4 u
v 1/2
Figure 8.5: An illustration of a ball in pipe system. The ball has radius 3/8 around si . Note that the volume of the ball will jump from radius r = 1/4 − ϵ to r = 1/4 due to the presence of edge (u, v). Observe also that if we consider a second ball of radius r + dr for suﬃciently small dr, then the volume of the diﬀerence of the two balls is just the crosssectional area of the pipes with one endpoint inside the ball and one outside; that is, it is c(δx (Bx (si , r)))dr.
Let x be an optimal solution to the LP F ←∅ for i ← 1 to k do if si and ti are connected in (V, E − F ) then Choose radius r < 1/2 around si as in Lemma 8.7 F ← F ∪ δ(Bx (si , r)) Remove Bx (si , r) and incident edges from graph return F Algorithm 8.2: Algorithm for the multicut problem.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
205
206
Cuts and metrics
sj , tj ∈ Bx (si , r) for r < 1/2, then dx (si , sj ) < 1/2 and dx (si , tj ) < 1/2 so that dx (sj , tj ) < 1. This contradicts the feasibility of the LP solution x. So it must be the case that whenever we remove the vertices in a ball Bx (si , r) from the graph, it can contain at most one of the pair of vertices sj , tj for all j. Assuming Lemma 8.7, we can now prove that the algorithm is a good approximation algorithm for the multicut problem. Theorem 8.9: Algorithm 8.2 is a (4 ln(k + 1))approximation algorithm for the multicut problem. Proof. Let Bi be the set of vertices in the ball Bx (si , r) chosen by the algorithm when the pair si ti are selected for separation; we set Bi = ∅ if no ball is chosen for i. Let Fi be the edges in δ(Bi ) when ∪ Bi and its incident edges are removed from the graph (where Fi = ∅ if Bi = ∅). Then F = ki=1 Fi . Let Vi be the total volume of edges removed when the vertices of Bi and ∗ its incident edges are removed from the graph. Note that Vi ≥ Vx (si , r) − Vk since Vi contains the full volume of edges in Fi , while Vx (si , r) only contains part of the volume of these edges, but has an extra term of V ∗ /k. Note that by of r in Lemma 8.7, we have that ( theV choice ∗) c(Fi ) ≤ (2 ln(k + 1))Vx (si , r) ≤ (2 ln(k + 1)) Vi + k . Further, observe that the volume of each edge belongs to at most one Vi ; once the edge is part of Vi it is removed from the graph and cannot be part of the volume of a ball Bj removed in a later iteration. This observation ∑ implies that ki=1 Vi ≤ V ∗ . Putting all of this together, we have that ∑ e∈F
ce =
k ∑ ∑ i=1 e∈Fi
ce ≤ (2 ln(k + 1))
k ( ∑ i=1
V∗ Vi + k
)
≤ (4 ln(k + 1))V ∗ ≤ (4 ln(k + 1)) OPT .
We ﬁnally turn to the proof of Lemma 8.7. Proof of Lemma 8.7. For simplicity we write V (r) = Vx (si , r) and c(r) = c(δ(Bx (si , r)). Our proof will show that for r chosen uniformly at random from [0, 1/2), the expected value of c(r)/V (r) is no more than 2 ln(k + 1). This implies that for some value of r, we have that c(r) ≤ (2 ln(k + 1))V (r). We will then show how we can quickly ﬁnd such a value of r deterministically. To set up the computation of the expectation, we need the following observations. For any value of r such that V (r) is diﬀerentiable, note that the derivative is exactly the cost of the edges in the cut given by the ball of radius r around si ; that is, dV dr = c(r) (see Figure 8.5). We observe that the points at which V (r) is not diﬀerentiable are values of r such that Bx (si , r) changes; that is, at values of r such that dx (si , v) = r for some vertex v. Furthermore, V (r) may not even be continuous for these values: if there are two vertices u and v such that there is an edge (u, v) of positive volume δ > 0 and dx (si , u) = dx (si , v) = r, then V (r) − V (r − ϵ) ≥ δ as ϵ ↓ 0 (see Figure 8.5). Nevertheless, note that V (r) is nondecreasing in r. Essentially we would now like to invoke the mean value theorem from calculus. Recall that the mean value theorem states that for a function f continuous on an interval [a, b] and (a) diﬀerentiable on (a, b), there exists some c ∈ (a, b) such that f ′ (c) = f (b)−f . If we set b−a f (r) = ln V (r), we note that d V (r) c(r) = . f ′ (r) = dr V (r) V (r) Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.3
The multicut problem
207
∗
Observe that V (1/2) ≤ V ∗ + Vk since V (1/2) cannot be more than the entire volume plus V ∗ /k, and V (0) is exactly V ∗ /k. Following the mean value theorem, we want to show that there is some r ∈ (0, 1/2) such that ( ) ( ) ∗ + V∗ V ln V (1/2) − ln V (0) V (1/2) k f ′ (r) ≤ = 2 ln ≤ 2 ln = 2 ln(k + 1). (8.4) 1/2 − 0 V (0) V ∗ /k However, the mean value theorem doesn’t apply since as we noticed earlier V (r) may not be continuous and diﬀerentiable in (0,1/2). Nevertheless, we show a form of the mean value theorem still holds in this case. We now sort and label the vertices in Bx (si , 1/2) according to their distance from si ; let rj = dx (si , vj ) be such that 0 = r0 ≤ r1 ≤ · · · ≤ rl−1 ≤ 1/2. Then the vertices are labelled si = v0 , v1 , v2 , . . . , vl−1 . For notational simplicity deﬁne rl = 1/2. Let rj− be a value inﬁnitesimally smaller than rj . The expected value of c(r)/V (r) for r chosen uniformly from [0, 1/2) is then 1 ∑ 1/2 l−1
j=0
∫
− rj+1
rj
∑ c(r) dr = 2 V (r) l−1
∫
rj
j=0
= 2
l−1 ∑
− rj+1
1 dV dr V (r) dr r−
[ln V (r)]rjj+1
j=0
= 2
l−1 [ ∑
] − ln V (rj+1 ) − ln V (rj )
j=0
Since V (r) is nondecreasing, this last sum is at most 2
l−1 ∑
[ln V (rj+1 ) − ln V (rj )] .
j=0
Then the sum telescopes so that 2
l−1 ∑
[ln V (rj+1 ) − ln V (rj )] = 2 (ln V (1/2) − ln V (0)) ,
j=0
and this can be bounded above by 2 ln(k + 1) as shown in (8.4). Thus it must be the case that there exists r ∈ [0, 1/2) such that c(r) ≤ (2 ln(k + 1))V (r). − How can we ﬁnd this value of r quickly? Observe that for r ∈ [rj , rj+1 ], c(r) stays constant (since Bx (si , r) is unchanged), while V (r) is nondecreasing. Thus if the inequality holds at any − point in this interval, it must also hold at rj+1 . Therefore, we need to check the inequality only − at rj+1 for j = 0, . . . , l − 1; by the argument above, the inequality must hold for some value of j. While no better approximation algorithm for the multicut problem is known, given the unique games conjecture, we cannot do signiﬁcantly better. Theorem 8.10: Assuming the unique games conjecture, for any constant α ≥ 1, there is no αapproximation algorithm for the multicut problem unless P = NP. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
208
Cuts and metrics
We will prove this theorem in Section 16.5. It will be useful in later sections to have a slight generalization of Lemma 8.7. We observe that by modifying the proof of Lemma 8.7 one can get the following corollary. Corollary 8.11: Given lengths xe on edges e ∈ E and a vertex u, one can ﬁnd in polynomial time a radius r ∈ [a, b) such that 1 c(δ(Bx (u, r))) ≤ ln b−a
8.4
(
Vx (u, b) Vx (u, a)
) Vx (u, r).
Balanced cuts
We now turn to the graph cut problem that most commonly arises in practice. We say that a set of vertices S is a bbalanced cut for b ∈ (0, 1/2] if ⌊bn⌋ ≤ S ≤ ⌈(1 − b)n⌉, where n = V . Given an undirected graph G = (V, E) with nonnegative edge costs ce ≥ 0 for all edges e ∈ E, and a b ∈ (0, 1/2], the goal is to ﬁnd a bbalanced cut S that minimizes the cost of the edges with exactly one endpoint in S. The case when b = 1/2 is sometimes called the minimum bisection problem. Finding bbalanced cuts arises in divideandconquer schemes used in a variety of applications: the cut S is found, some graph problem is solved on S and V − S, then the solution is the result of combining the two solutions in S and V − S via the edges between them. If the cut has low cost, then the combining step becomes easier. Furthermore, if S and V − S have approximately the same size, then the algorithm can be applied recursively to each side, and the total depth of the recursion will be O(log n). In this section, we will not give an approximation algorithm for the balanced cut problem, settling instead for a pseudoapproximation algorithm; by this we mean that our algorithm will ﬁnd a bbalanced cut whose cost is within a O(log n) factor of the cost of the optimal b′ balanced cut for b′ ̸= b. Let OPT(b) denote the value of the minimumcost bbalanced cut. In particular, we will show how to ﬁnd a 31 balanced cut of value at most O(log n) OPT(1/2). Note ﬁrst of all that OPT(1/3) ≤ OPT(1/2), since any 12 balanced cut is also a 13 balanced cut. However, OPT(1/2) could be substantially larger than OPT(1/3): in fact, if the graph consists of a clique on 32 n nodes connected by a single edge to a clique on 31 n nodes, and all edges have weight 1, OPT(1/3) = 1 while OPT(1/2) = Ω(n2 ). Thus the algorithm is not a true approximation algorithm since we compare the cost of the solution found by the algorithm with the optimum OPT(1/2) for a problem that could be much larger than the optimum OPT(1/3) of the problem for which the algorithm ﬁnds a solution. While we give an algorithm to ﬁnd a 13 balanced cut of cost at most O(log n) OPT(1/2), the discussion that follows can be generalized to give a bbalanced cut of cost at most O( b′ 1−b log n) OPT(b′ ) for b ≤ 1/3 and b < b′ . Our approach to the problem follows that used for the multicut problem in the previous section. We claim that the following is a linear programming relaxation of the minimum bisection problem; we will shortly prove that this is true. Given a graph G, let Puv be the set of all Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.4
Balanced cuts
209
paths between u and v in G. Consider the following linear program: minimize
∑
ce xe
e∈E
duv ≤
subject to ∑
∑ e∈P
(
duv ≥
v∈S
duv ≥ 0, xe ≥ 0,
∀u, v ∈ V, ∀P ∈ Puv ,
xe ,
2 1 − 3 2
)
⌈
n,
⌉ 2 ∀S ⊆ V : S ≥ n + 1, ∀u ∈ S, 3 ∀u, v ∈ V, ∀e ∈ E.
In this relaxation, we have a variable duv that is intended to denote the distance between u and v using edge lengths xe ; if we let dx (u, v) be the actual shortest path length between u and v using edge lengths xe , then duv ≤ dx (u, v). To solve the linear program in polynomial time, we once again apply the ellipsoid method as described in Section 4.3. Given a solution (d, x), we can ﬁrst check if the ﬁrst set of constraints is obeyed by computing the shortest path distance between u and v using edge lengths xe , and ensuring that duv ≤ dx (u, v). For each vertex u, we ﬁnd the closest ⌈ 32 n⌉ + 1 vertices using distances duv ; this will include u itself since duu = 0. Let u = v0 , v1 , v2 , . . . , v⌈ 2 n⌉ be the ⌈ 32 n⌉+1 3 ∑⌈ 32 n⌉ vertices that minimize duv . If j=0 duvj < ( 23 − 12 )n, then clearly the constraint is violated for ∑⌈ 23 n⌉ duvj ≥ ( 32 − 21 )n, then there is no S = {v0 , . . . , v⌈ 2 n⌉ } and this choice of u. Note that if j=0 3
other set S with S ≥ ⌈ 32 n⌉ + 1 and u ∈ S such that the constraint is violated, since any S containing v0 , . . . , v⌈ 2 n⌉ can only give rise to a larger sum, and since these vertices were chosen 3 to make the sum as small as possible. Thus in polynomial time we can check whether (d, x) is a feasible solution and ﬁnd a violated constraint if it is not feasible. We now argue that the linear program is a relaxation. Lemma 8.12: The linear program above is a relaxation of the minimum bisection problem. ¯x Proof. Given an optimal bisection S, we construct a solution (d, ¯) for the linear program by setting x ¯e = 1 if e ∈ δ(S) and x ¯e = 0 otherwise. We then set d¯uv = 1 for all u ∈ S, v ∈ / S, ¯ and duv = 0 otherwise. We argue that this solution is feasible, which is suﬃcient to prove the lemma. Clearly the ﬁrst set of constraints is obeyed, since any path P from u ∈ S to v ∈ / S must use an edge e ∈ δ(S). Now consider any set S ′ such that S ′  ≥ ⌈ 32 n⌉ + 1. Notice that S ′ − S ≥ ⌈ 23 n⌉ + 1 − ⌈ 12 n⌉ ≥ 2 ( 3 − 12 )n and S ′ ∩ S ≥ ⌈ 32 n⌉ + 1 − ⌈ 12 n⌉ ≥ ( 23 − 21 )n; call S ′ − S and S ′ ∩ S the two parts of S ′ . See Figure 8.6. One part (namely, S ′ ∩ S) is contained in S and the other (namely, S ′ − S) has no vertices of S, so for u and v in diﬀerent parts, d¯uv = 1. Pick any u ∈ S ′ . Since u is in one of the two parts of S ′ ,∑there are at least ( 32 − 12 )n vertices v in the other part of S ′ which does not contain u. Thus v∈S ′ d¯uv ≥ ( 23 − 12 )n.
Given an optimal solution (d, x) to the linear program, let Bx (u, r) be the set of vertices in the ball of radius r around ∑ u; that is, Bx (u, r) = {v ∈ V : dx (u, v) ≤ r}. As with the multicut problem, we let V ∗ = e∈E ce xe , so that we know V ∗ ≤ OPT(1/2) by Lemma 8.12. We deﬁne Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
210
Cuts and metrics
S0
S
u v
Figure 8.6: An illustration of the proof of Lemma 8.12. At least ( 23 − 21 )n vertices must be both in S ∩ S ′ and S ′ − S, so for any u ∈ S ′ , there must be at least ( 23 − 21 )n vertices v which lie on the other side of the bisection S. the volume of the ball of radius r around a vertex u as we did in the previous section: Vx (u, r) =
V∗ + n
∑ e=(v,w):v,w∈Bx (u,r)
ce xe +
∑
ce (r − dx (u, v)).
e=(v,w):v∈Bx (u,r),w∈B / x (u,r)
As before, deﬁne c(δ(Bx (u, r))) to be the cost of the edges with exactly one endpoint in Bx (u, r). We can then prove the following lemma, which is analogous to Lemma 8.7. Lemma 8.13: Given a feasible solution to the linear program (d, x) and two vertices u and v such that dx (u, v) ≥ 1/12, one can ﬁnd in polynomial time a radius r < 1/12 such that c(δ(Bx (u, r))) ≤ (12 ln(n + 1))Vx (u, r). Proof. We apply Corollary 8.11 to the interval [0, 1/12), and observe that ) ( ∗ ) ( V + V ∗ /n Vx (u, 1/12) ≤ ln = ln(n + 1). ln Vx (u, 0) V ∗ /n
Our algorithm for the 13 balanced cut problem is given in Algorithm 8.3. We let S be the set of vertices that will be in the cut; S is initially empty. We let F be a superset of the edges in the cut, and it is also initially empty. Our ﬁnal solution requires that ⌊ 31 n⌋ ≤ S ≤ ⌈ 23 n⌉. As long as S < ⌊ 31 n⌋, we will show that there must exist two vertices u, v ∈ / S such that dx (u, v) ≥ 16 . We apply Lemma 8.13 to ﬁnd balls of radius less than 1/12 around both u and v. We take the ball of smaller cardinality, add it to S, add the corresponding cut to F , and remove the ball and its incident edges from the graph. We now need to prove that the algorithm is correct, and returns a 13 balanced cut. Lemma 8.14: For any iteration of the algorithm in which S < ⌊ 13 n⌋, there exist u, v ∈ / S such that dx (u, v) ≥ 1/6. Proof. Consider∑ S ′ = V −S. Then S ′  ≥ ⌈ 23 n⌉+1. Then by the linear programming constraint, for any u ∈ S ′ , v∈S ′ duv ≥ ( 32 − 12 )n = 16 n. Since there are at most n vertices in S ′ , it must be the case that for some v ∈ S ′ , duv ≥ 1/6. Finally, we know that dx (u, v) ≥ duv . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.5
Probabilistic approximation of metrics by tree metrics
211
Let (d, x) be an optimal solution to the LP relaxation F ← ∅; S ← ∅ while S < ⌊ 13 n⌋ do Choose some u, v ∈ / S such that dx (u, v) ≥ 1/6 Choose radius r < 1/12 around u as in Lemma 8.13 Choose radius r′ < 1/12 around v as in Lemma 8.13 if Bx (u, r) ≤ Bx (v, r′ ) then S ← S ∪ Bx (u, r) F ← F ∪ δ(Bx (u, r)) Remove Bx (u, r) and incident edges from graph else S ← S ∪ Bx (v, r′ ) F ← F ∪ δ(Bx (v, r′ )) Remove Bx (v, r′ ) and incident edges from graph return S Algorithm 8.3: Algorithm for the 13 balanced cut problem. Lemma 8.15: The algorithm returns S such that ⌊ 13 n⌋ ≤ S ≤ ⌈ 32 n⌉. Proof. By construction of the algorithm, S ≥ ⌊ 13 n⌋. Thus we only need to show that S ≤ ⌈ 23 n⌉. Let Sˆ be the contents of S at the beginning of the iteration before the algorithm terˆ < ⌊ 1 n⌋ and we added the smaller of the two balls around u and v to minated. Then S 3 Sˆ to obtain the ﬁnal solution S. Since we considered Bx (u, r) and Bx (v, r′ ) for r < 1/12, r′ < 1/12, and dx (u, v) ≥ 1/6, it must be the case that these two balls are disjoint; that is, ˆ this implies that the size Bx (u, r) ∩ Bx (v, r′ ) = ∅. Since we chose the smaller ball to add to S, ˆ ˆ of the ball added to S had no more than half the remaining vertices, or no more than 12 (n − S) vertices. Thus ⌈ ⌉ ˆ = 1 n + 1 S ˆ ≤ 2n , ˆ + min(Bx (u, r), Bx (v, r′ )) ≤ S ˆ + 1 (n − S) S = S 2 2 2 3 ˆ < ⌊ 1 n⌋. since S 3 The proof of the performance guarantee is nearly the same as that for the multicut algorithm, and so we omit it. Theorem 8.16: Algorithm 8.3 returns a 13 balanced cut of cost no more than (24 ln(n+1))V ∗ ≤ (24 ln(n + 1)) OPT(1/2). As mentioned previously, the algorithm above can be generalized to give a bbalanced cut of cost at most O( b′1−b log n) OPT(b′ ) for b ≤ 1/3 and b < b′ . In Section 15.3, we will give an O(log n)approximation algorithm for the minimum bisection problem; this algorithm is not a pseudoapproximation algorithm, but is an algorithm that produces a bisection of cost at most O(log n) OPT(1/2).
8.5
Probabilistic approximation of metrics by tree metrics
In this section, we introduce the idea of tree metrics, which we will consider for the remainder of the chapter. Tree metrics have become an important tool for devising approximation algorithms Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
212
Cuts and metrics
for a wide variety of problems. We use tree metrics to approximate a given distance metric d on a set of vertices V . A tree metric (V ′ , T ) for a set of vertices V is a tree T deﬁned on some set of vertices V ′ ⊇ V , together with nonnegative lengths on each edge of T . The distance Tuv between vertices u, v ∈ V ′ is the length of the unique shortest path between u and v in T . We would like to have a tree metric (V ′ , T ) that approximates d on V in the sense that duv ≤ Tuv ≤ α · duv for all u, v ∈ V for some value of α. The parameter α is called the distortion of the embedding of d into the tree metric (V ′ , T ). Given a lowdistortion embedding, we can often reduce problems on general metric spaces to problems on tree metrics with a loss of a factor of α in the performance guarantee. It is often much easier to produce an algorithm for a tree metric than for a general metric. We will see examples of this in the following two sections, in which we discuss the buyatbulk network design problem and the linear arrangement problem. Further examples can be found in the exercises. Unfortunately, it can be shown that for a cycle on n vertices, no tree metric has distortion less than (n − 1)/8 (see Exercise 8.7 for a restricted proof of this). However, we can give a randomized algorithm for producing a tree T such that for each u, v ∈ V , duv ≤ Tuv and E[Tuv ] ≤ O(log n)duv ; that is, the expected distortion is O(log n). Another way to view this is that we can give a probability distribution on trees such that the expected distortion is O(log n). We refer to this as the probabilistic approximation of the metric d by a tree metric. Theorem 8.17: Given a distance metric (V, d), such that duv ≥ 1 for all u ̸= v, u, v ∈ V , there is a randomized, polynomialtime algorithm that produces a tree metric (V ′ , T ), V ⊆ V ′ , such that for all u, v ∈ V , duv ≤ Tuv and E[Tuv ] ≤ O(log n)duv . It is known that there exist metrics such that any probabilistically approximation of the metric by tree metrics must have distortion Ω(log n), so the result above is the best possible to within constant factors. We now begin to give the details of how the algorithm of Theorem 8.17 works. The tree is constructed via a hierarchical cut decomposition of the metric d. Let ∆ be the smallest power of 2 greater than 2 maxu,v duv . The hierarchical cut decomposition is a rooted tree with log2 ∆ + 1 levels. The nodes at each level of the tree correspond to some partitioning of the vertex set V : the root node (at level log2 ∆) corresponds to V itself, while each leaf of the tree (at level 0) corresponds to a single vertex of V . A given node at level i corresponds to some subset S of vertices V ; the children of the node corresponding to S correspond to a partitioning of S. For a given node at level i that corresponds to a set S, the vertices in S will be the vertices in a ball of radius less than 2i and at least 2i−1 centered on some vertex. Notice that each leaf of the tree at level 0 is in a ball of radius less than 20 = 1 centered on some vertex u, and so u is in the ball by itself since duv ≥ 1 for all v ̸= u. Observe also that by the deﬁnition of ∆, the set V is contained in a ball of radius ∆ centered on any vertex, since the radius of the ball at level log2 ∆ is at least 21 ∆ ≥ maxu,v duv . The length of the tree edge joining the children at level i − 1 to their parent at level i is 2i . See Figure 8.7 for an illustration of the hierarchical cut decomposition. Each node of the tree will be a vertex in V ′ , so that the tree is a tree on the vertices in V ′ . We let each leaf correspond to the unique v ∈ V that it contains, so that the leaves are the vertex set V , while the internal vertices are the remaining nodes in V ′ , and V ⊆ V ′ . Before we state precisely how to obtain the decomposition, we observe the following properties of the tree obtained in this way. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
8.5
Probabilistic approximation of metrics by tree metrics
Level log2 ∆
1 + 2, the rescaled local search algorithm using bigger steps yields a ρapproximation algorithm for the uncapacitated facility location problem.
9.2
A local search algorithm for the kmedian problem
In this section, we shall consider again the kmedian problem originally considered in Section 7.7; however, we shall consider a slightly simpler variant in which we have a set of locations Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
240
Further uses of greedy and local search algorithms
N , each of which is both a client and a potential facility location. For each pair of locations i and j, there is a cost cij of assigning location j to a facility at location i. We can select at most k locations at which to open facilities, where k is part of the input. The goal is to ﬁnd a set of locations ∑ S at which to open a facility, where S ≤ k, such that the assignment cost is minimized: j∈N mini∈S cij . Without loss of generality, we can assume that S = k. In other words, this problem can be viewed as the minsum analogue of the kcenter problem, previously considered in Section 2.2, which is a minmax problem. As in the kcenter problem, we shall assume that the distance matrix is symmetric, satisﬁes the triangle inequality, and has zeroes on the diagonal (i.e. cii = 0 for each i ∈ N ). We will give a local search algorithm for the kmedian problem. We will let S ⊆ N denote the set of open facilities for the current solution, and let S ∗ ⊆ N denote the set of open facilities in a ﬁxed optimal solution. For each of these two solutions, each client j is assigned to its nearest open facility (breaking ties arbitrarily); we let this mapping be denoted σ(j) and σ ∗ (j), respectively, for these two assignments. Analogously, we let C and C ∗ denote, respectively, the total cost of the current and optimal solutions. The local search algorithm that we shall consider is the most natural one. Each current solution is speciﬁed by a subset S ⊆ N of exactly k locations. To move from one feasible solution to a neighboring one, we swap two locations; that is, we select one location i ∈ S to delete from the current set, and choose one location i′ ∈ N − S to add to the current set of facilities. Afterward, we reassign each client to its nearest open facility. In our local search procedure, we repeatedly check to see if any swap move yields a solution of lower cost; if so, the resulting solution is our new current solution. We repeat this step until, from the current solution, no swap move decreases the cost. The current solution at this point is said to be locally optimal. We shall prove the following theorem. Theorem 9.6: For any input to the kmedian problem, any feasible solution S that is locally optimal with respect to the pairwise swap move has a cost that is at most ﬁve times the optimal value. Proof. The proof will focus on ﬁrst constructing a set of k special swaps, which we shall call the crucial swaps. Since the current solution S is locally optimal, we know that each of these swaps does not improve the objective function of the resulting solution. These swaps will all be constructed by swapping into the solution one location i∗ in S ∗ and swapping out of the solution one location i in S. Each i∗ ∈ S ∗ will participate in exactly one of these k swaps, and each i ∈ S will participate in at most 2 of these k swaps. (We will allow the possibility that i∗ = i, and hence the swap move is degenerate, but clearly such a “change” would also not improve the objective function of the current solution, even if we change the corresponding assignment.) Observe that the current solution provides a mapping from each facility i∗ ∈ S ∗ to a facility σ(i∗ ) ∈ S. As Figure 9.2 shows, this mapping allows us to categorize the facilities in S: • let O ⊆ S consist of those facilities i ∈ S that have exactly one facility i∗ ∈ S ∗ with σ(i∗ ) = i; • let Z ⊆ S consist of those facilities i ∈ S for which none of the facilities i∗ ∈ S ∗ have σ(i∗ ) = i; • and let T ⊆ S consist of those facilities i ∈ S such that i has at least two locations in S ∗ assigned to it in the current solution. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.2
A local search algorithm for the kmedian problem
241
O∗
R∗
S∗ σ
S T
Z
O
Figure 9.2: The mapping of locations in S ∗ to S used to construct the crucial swaps.
We make a few simple observations. The mapping σ provides a matching between a subset O∗ ⊆ S ∗ and the set O ⊆ S (by the deﬁnition of O). Hence, if ℓ denotes the number of locations in R∗ = S ∗ − O∗ , then ℓ must also equal Z ∪ T  (since S ∗  = S = k). Since each location in T is the image of at least two locations in R∗ , it follows that T  ≤ ℓ/2. Hence Z ≥ ℓ/2. We now construct the crucial swaps as follows: ﬁrst, for each i∗ ∈ O∗ , we swap i∗ with second, we build a collection of ℓ swaps, each of which swaps into the solution a distinct location in R∗ , and swaps out a location in Z, where each location in Z appears in at most two swaps. For the swaps involving locations in R∗ and Z, we are free to choose any mapping provided that each element of R∗ is swapped in exactly once, and each element of Z is swapped out once or twice. σ(i∗ );
Consider one crucial swap, where i∗ ∈ S ∗ and i ∈ S denote the swapped locations; we analyze the change of cost incurred by no longer opening a facility at location i ∈ S, and using i∗ instead. Let S ′ denote the set of selected locations after this swap; that is, S ′ = S −{i}∪{i∗ }. To complete the analysis, we also specify the assignment of each location in N to an open facility in S ′ . For each location j such that σ ∗ (j) = i∗ , we assign j to i∗ (since i∗ is in S ′ ). For each location j such that σ ∗ (j) ̸= i∗ , but σ(j) = i, then we need a new facility to serve j, and we now assign it to σ(σ ∗ (j)). All other locations j remain served by σ(j). One must argue that σ(σ ∗ (j)) ̸= i when σ ∗ (j) ̸= i∗ (since we need that this location must remain in S ′ ). Assume, for a contradiction, that σ(σ ∗ (j)) = i; then i ∈ O (since each location swapped out by a crucial swap is either in Z or O, and the former is clearly not possible by the deﬁnition of Z). Since i ∈ O, it is σ’s image of exactly one element in O∗ , and we build a crucial swap by swapping i with that one element. Hence, σ ∗ (j) = i∗ , but this is a contradiction. We have now constructed a new set of facilities S ′ and an assignment of each location in N to one of the facilities in S ′ . There may be some location j that is not served by its closest point in S ′ by this assignment. However, since there are no improving swaps from the current solution S, any swap combined with a suboptimal assignment must also not decrease the overall cost; hence, the change to S ′ along with the speciﬁed assignment also must not improve over using S with the assignment given by function σ. What is the change of cost for this swap? By focusing only on the clients j for which Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
242
Further uses of greedy and local search algorithms
S σ(j)
σ(σ ∗(j)) σ
S∗ σ ∗(j)
j
Figure 9.3: Bounding the length of the edge from j to σ(σ ∗ (j)). σ ∗ (j) = i∗ , or both σ ∗ (j) ̸= i∗ and σ(j) = i, we can compute the change in the cost to be ∑ ∑ (cσ∗ (j)j − cσ(j)j ) + (cσ(σ∗ (j))j − cσ(j)j ). ∗ ∗ j:σ ∗ (j)=i∗ j:σ (j)̸=i & σ(j)=i We can simplify the terms in the second summation by considering Figure 9.3. By the triangle inequality, we have that cσ(σ∗ (j))j ≤ cσ(σ∗ (j))σ∗ (j) + cσ∗ (j)j . In the current solution S, σ ∗ (j) is assigned to σ(σ ∗ (j)) instead of to σ(j), and so we know that cσ(σ∗ (j))σ∗ (j) ≤ cσ(j)σ∗ (j) , Once again, we can apply the triangle inequality to get that cσ(j)σ∗ (j) ≤ cσ∗ (j)j + cσ(j)j , and so, putting these pieces together, we have that cσ(σ∗ (j))j ≤ 2cσ∗ (j)j + cσ(j)j , or equivalently, cσ(σ∗ (j))j − cσ(j)j ≤ 2cσ∗ (j)j . (In fact, a little reﬂection indicates that we had already proved this, in Lemma 9.2, where now σ plays the same role as ϕ.) This yields a more compact upper bound on the change in the cost, which we know must be nonnegative; that is, for each crucial swap i∗ and i, we have that ∑ ∑ 0≤ (cσ∗ (j)j − cσ(j)j ) + 2cσ∗ (j)j . (9.11) ∗ ∗ j:σ ∗ (j)=i∗ j:σ (j)̸=i & σ(j)=i Now we add inequality (9.11) over all k crucial swaps. Consider the contribution for each of the two terms in the ﬁrst summation. Recall that each i∗ ∈ S ∗ participates in exactly one crucial swap. For the ﬁrst term, we add cσ∗ (j)j over all clients j for which σ ∗ (j) = i∗ , and this sum is then added for each choice of i∗ in S ∗ . Each client j has a unique facility σ ∗ (j) ∈ S ∗ to which it is assigned by the ﬁxed optimal solution, and so the net eﬀect is to compute the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.3
Minimumdegree spanning trees
243
∑ sum j∈N cσ∗ (j)j = C ∗ . However, the same is true for the second term, the double summation is ∑ merely the sum over all possible locations j, and so the second terms contribute a total of − j∈N cσ(j)j = −C. Now consider the second summation in (9.11). We can upper bound this expression by ∑ 2cσ∗ (j)j . j:σ(j)=i
What is the eﬀect of adding this summation over all crucial swaps? Each facility i ∈ S occurs in 0, 1, or 2 crucial swaps; let ni be the number of swaps in which each i ∈ S occurs. Thus, we can upper bound the double summation as follows: ∑ ∑ ∑ ∑ 2ni cσ∗ (j)j ≤ 4 cσ∗ (j)j . i∈S j:σ(j)=i
i∈S j:σ(j)=i
But now we can apply the same reasoning as above; each location j is served in the current solution by a unique facility location σ(j) ∈ S, and hence the eﬀect of the double summation is∑ merely to sum over each j in N . That is, we have now deduced that this term is at most 4 j∈N cσ∗ (j)j = 4C ∗ . Furthermore, we have concluded that 0 ≤ 5C ∗ − C, and hence C ≤ 5C ∗ . Finally, we observe that the same idea used in the previous section to obtain a polynomialtime algorithm can be applied here. The central ingredients to that proof are that if we restrict attention to moves (in this case swap moves) in which we improve the total cost by a factor of 1 − δ, then provided the analysis is based on a polynomial number of moves (each of which generates an inequality that the change in cost from this move is nonnegative), we can set δ so that we can obtain a polynomialtime bound, while degrading the performance guarantee by an arbitrarily small constant. Theorem 9.7: For any constant ρ > 5, the local search algorithm for the kmedian problem that uses bigger improving swaps yields a ρapproximation algorithm.
9.3
Minimumdegree spanning trees
In this section we return to the minimumdegree spanning tree problem introduced in Section 2.6. Recall that the problem is to ﬁnd a spanning tree T in a graph G = (V, E) that minimizes the maximum degree. If T ∗ is an optimal tree that minimizes the maximum degree, let OPT be the maximum degree of T ∗ . In Section 2.6, we showed that a particular local search algorithm ﬁnds a locally optimal tree of maximum degree at most 2 OPT +⌈log2 n⌉ in polynomial time. In this section, we will show that another variation on the local search algorithm ﬁnds a locally optimal tree of maximum degree at most OPT +1 in polynomial time. As we discussed in Section 2.6, since it is NPhard to minimize the maximum degree of a spanning tree, this is the best result possible unless P = NP. As in the algorithm of Section 2.6, we start with an arbitrary spanning tree T and we will make local changes to it in order to decrease the degree of nodes in the tree. Let dT (u) be the degree of u in T . We pick a node u and attempt to reduce its degree by adding an edge (v, w) to T that creates a cycle C containing u, then removing an edge of the cycle C incident on u. Let ∆(T ) = maxv∈V dT (v) be the maximum degree of the current tree T . We will make local changes in a way that is driven by the following lemma, which gives us a condition under which the current tree T has ∆(T ) ≤ OPT +1. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
244
Further uses of greedy and local search algorithms
Dk ∪ Dk−1
∈F
components C
Figure 9.4: Illustration of the terms used in the statement of Lemma 9.8. The edges in F are shown in bold. Some edges in G that are not in the tree T are shown as dotted lines; note that they are not all incident on nodes in Dk ∪ Dk−1 . The components C are the components of the tree T remaining after the edges in F are removed. Lemma 9.8: Let k = ∆(T ), let Dk be any nonempty subset of nodes of tree T with degree k and let Dk−1 be any subset of nodes of tree T with degree k − 1. Let F be the edges of T incident on nodes in Dk ∪ Dk−1 , and let C be the collection of F  + 1 connected components formed by removing the edges of F from T . If each edge of graph G that connects two diﬀerent components in C has at least one endpoint in Dk ∪ Dk−1 , then ∆(T ) ≤ OPT +1. Proof. See Figure 9.4 for an illustration of the terms. We use the same idea as in the proof of Theorem 2.19 to obtain a lower bound on OPT. Since any spanning tree in G will need F  edges of G to connect the components in C, the average degree of the nodes in Dk ∪ Dk−1 in any spanning tree is at least F /Dk ∪ Dk−1 . Thus OPT ≥ ⌈F /Dk ∪ Dk−1 ⌉. We now bound F  in order to prove the lemma. The sum of the degrees of the nodes in Dk and Dk−1 must be Dk k + Dk−1 (k − 1). However, this sum of degrees may double count some edges of F which have both endpoints in Dk ∪ Dk−1 . Because T is acyclic, there can be at most Dk  + Dk−1  − 1 such edges. Hence, F  ≥ Dk k + Dk−1 (k − 1) − (Dk  + Dk−1  − 1). Thus ⌉ ⌈ Dk k + Dk−1 (k − 1) − (Dk  + Dk−1  − 1) OPT ≥ Dk  + Dk−1  ⌈ ⌉ Dk−1  − 1 ≥ k−1− Dk  + Dk−1  ≥ k − 1, implying that k = ∆(T ) ≤ OPT +1. The goal of the local search algorithm is to continue to reduce the degree of the nodes of degree ∆(T ) while trying to attain the conditions of Lemma 9.8. The algorithm works in a sequence of phases, with each phase divided into subphases. At the beginning of the phase, for the current tree T , let k = ∆(T ). At the beginning of a subphase, we let Dk be all the degree k vertices in T , let Dk−1 be all the degree k − 1 vertices in T , let F be the edges of T incident to Dk ∪ Dk−1 , and let C be the components of T formed if F is removed from T . The goal of each phase is to make local moves to remove all nodes of degree k from the tree; Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.3
Minimumdegree spanning trees
245
the goal of each subphase is to make local moves to remove a single vertex of degree k. In the process of executing a subphase, we discover nodes of degree k − 1 in Dk−1 for which we can make a local move to reduce their degree. We do not yet execute these moves, but we mark the nodes as reducible via the particular local move, remove them from Dk−1 , and update F and C accordingly by removing edges from F and merging components in C. The algorithm is summarized in Algorithm 9.1, and we now discuss its details. In a subphase, we consider all edges of G that connect any two components of C. If all such edges have an endpoint in Dk ∪ Dk−1 , we meet the condition of Lemma 9.8, and the algorithm terminates with a tree T such that ∆(T ) ≤ OPT +1. If there is some such edge (v, w) without an endpoint in Dk ∪ Dk−1 , then consider the cycle formed by adding (v, w) to T . Because (v, w) connects two diﬀerent components of C, it must be the case that the cycle includes some node u ∈ Dk ∪ Dk−1 . Note that if we desire, we can make a local move to reduce the degree of u by adding (v, w) to the tree T and removing a tree edge incident to u. If the cycle contains nodes of Dk−1 only, then we do not yet make the local move, but we make a note that for any of the nodes in Dk−1 on the cycle, we could do so if necessary. We label these nodes in Dk−1 on the cycle as reducible via the edge (v, w), then remove them from Dk−1 , then update F to be the tree edges incident on the current set of Dk ∪ Dk−1 , and update C accordingly. Note that since we removed all nodes on the cycle from Dk−1 , this only removes edges from F and merges components in C; in particular, the two components connected by (v, w) will be merged in the updated C. If, on the other hand, the cycle includes a node of u ∈ Dk , we go ahead and reduce its degree by adding (v, w) to the tree and removing an edge incident on u. Decreasing the degree of u decreases the number of nodes of degree k in the tree, but we want to ensure that we do not increase the degree of nodes v and w to k. Note that this could only happen if the degree of v or w in the tree is k − 1 and at some previous point in the subphase the node was removed from Dk−1 and labelled reducible. In this case, we carry out the local move that allows us to reduce the degree of reducible node to k − 2, then add (v, w) to the tree and remove an edge incident to u from the tree. It is possible that carrying out the local move to reduce the degree of the reducible node to k − 2 might cause a cascade of local moves; for instance, if v has degree k − 1, and we can reduce its degree by adding edge (x, y), potentially x also has degree k − 1 and is reducible, and so on; we will show that it is possible to carry out all these moves, and reduce the degree of u to k − 1 without creating any new nodes of degree k. We say that we are able to propagate the local move for u. Once we reduce the degree of u from k to k − 1, we start a new subphase. If there are no further nodes of degree k, we start a new phase. We can now prove that the algorithm is correct and runs in polynomial time. Theorem 9.9: Algorithm 9.1 returns a spanning tree T with ∆(T ) ≤ OPT +1 in polynomial time. Proof. Because the algorithm terminates only when it meets the conditions of Lemma 9.8, it returns a tree T with ∆(T ) ≤ OPT +1 if it does indeed terminate. We claim that in each subphase, we can propagate local moves to reduce the degree of a reducible node in a component in C without creating any new nodes of degree k. Then either the algorithm terminates or in each subphase, we ﬁnd some node u of degree k whose degree can be reduced to k −1 by making a local move with an edge (v, w). Since v and w must be in separate components of C, either their degree is less than k − 1 or they have degree k − 1 and are reducible, and by the claim we can propagate local moves to reduce their degree. Thus we can reduce the degree of u from k to k − 1 without creating any new nodes of degree k, and so each phase eliminates all nodes of degree k. Since we cannot have a feasible spanning tree with ∆(T ) = 1, the algorithm must Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
246
Further uses of greedy and local search algorithms
Let T be an arbitrary spanning tree of G = (V, E) while true do k ← ∆(T ) // Start a new phase while there are nodes of degree k in T do // Start a new subphase Dk ← all nodes of degree k in T Dk−1 ← all nodes of degree k − 1 in T F ← all edges of T incident on nodes in Dk ∪ Dk−1 C ← all components formed by removing F from T All nodes u ∈ Dk−1 are unlabelled if for all (v, w) ∈ E connecting two components in C: either v or w in Dk ∪ Dk−1 then return T for all (v, w) ∈ E connecting two components in C: v, w ∈ / Dk ∪ Dk−1 do Let C be cycle created by adding (v, w) to T if C ∩ Dk = ∅ then Mark all u ∈ C ∩ Dk−1 reducible via (v, w) Remove C ∩ Dk−1 from Dk−1 Update F and C else if u ∈ C ∩ Dk then if v or w marked reducible then Reduce degree of v and/or w via local move and propagate local moves if necessary Reduce degree of u via local move with (v, w) Break for loop // Start new subphase Algorithm 9.1: Local search algorithm for the minimumdegree spanning tree problem.
eventually terminate. Clearly the algorithm runs in polynomial time. We now prove the claim by showing that at any iteration of the subphase, we can propagate local moves to reduce the degree of a reducible node in a component in C. We prove this by induction on the number of iterations in the subphase. In the ﬁrst iteration, no nodes are marked reducible and the claim is trivially true. Now suppose we are at some iteration i > 1 of the subphase, and let u be labelled reducible in this iteration. The node u is currently reducible because we have a local move with a nontree edge (v, w) that will reduce the degree of u from k − 1 to k − 2; furthermore, the components in C containing v and w are disjoint in the current iteration. If v is reducible, it was labelled such in an iteration j < i, and by induction we can carry out local moves to ensure that its degree is at most k − 2. The same is true for w, and by induction we can carry out the local moves for both v and w because they are in separate components in C. In the next iteration the components containing v and w are merged into a single component that also contains u. Since the only changes that can happen to components in C during a subphase is that components are merged, u, v, and w remain in the same component of C through the rest of the subphase, and the local moves of adding (v, w) and reducing the degree of u remain available. In Section 11.2, we will consider a version of the problem in which there are costs on the edges and speciﬁed bounds on the degrees of the nodes. If a tree with the given degree bounds Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.4
A greedy algorithm for the uncapacitated facility location problem
247
exists, we will show how to ﬁnd a minimumcost tree such that the degree bounds are only exceeded by one.
9.4
A greedy algorithm for the uncapacitated facility location problem
In this section, we give yet another approximation algorithm for the uncapacitated facility location problem. We will give a greedy algorithm for the problem, then use dual ﬁtting to analyze it; this is similar to what we did in Theorem 1.12 for the set cover problem. A very natural greedy algorithm is to repeatedly choose a facility and some clients to assign to that facility. We open the facility, assign the clients to the facility, remove the facility and clients from further consideration, and repeat. For a greedy algorithm, we would somehow like to ﬁnd a facility and set of clients that minimizes total cost for the amount of progress made. To do this, we use the same criterion we used for the greedy set cover algorithm in Section 1.6: we maximize the “bang for the buck” by minimizing the ratio of the total cost per client assigned. To be more precise, let X be the set of facilities opened so far, and let S be the set of clients that are not connected to facilities in X so far. We pick some i ∈ F − X and Y ⊆ S that minimizes the ratio ∑ fi + j∈Y cij . Y  We then add i to X, and remove Y from S, and repeat. Note that to ﬁnd the appropriate set Y ⊆ S, for any given facility i, we can sort the clients in S by their distance from i, from nearest to farthest, and the set Y minimizing the ratio for i will be some preﬁx of this ordering. We now add two simple improvements to this proposed algorithm. The ﬁrst is that once we select facility i, rather than removing it from the set of facilities that can be chosen in the future, we instead allow it to be chosen again and set its facility cost to zero. The intuition here is that in future iterations it may be more costeﬀective to assign some clients to i rather than opening another facility to serve them, and since i has already been opened, we should treat its facility cost as zero. The second idea is that rather than assigning clients to a facility and ﬁxing that assignment from then on, we consider switching assignments to other facilities we open later on. We include the savings gained by switching assignments when trying to choose the facility to open. Let c(j, X) = mini∈X cij , and let (a)+ = max(a, 0). Then if we have already assigned the clients in D − S to some facilities in X, and we are considering opening facility i, we can decrease assignment costs for all clients j∈ / S such that c(j, X) > cij by reassigning ∑ them from X to i. The savings achieved is j ∈S (c(j, X) − cij )+ . Thus in every step we pick / some i ∈ F and Y ⊆ S that minimizes the ratio ∑ ∑ fi − j ∈S / (c(j, X) − cij )+ + j∈Y cij . Y  Our revised greedy algorithm is now given in Algorithm 9.2. To analyze this algorithm, we will use a dual ﬁtting analysis: we will construct an infeasible solution to the dual of the linear programming relaxation such that the cost of the primal solution is equal to the value of the dual objective. Then we show that scaling the dual solution by a factor of 2 makes it feasible. This implies that the cost of the primal solution is at most twice the value of a solution to the dual of the linear programming relaxation, which implies that the algorithm is an 2approximation algorithm. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
248
Further uses of greedy and local search algorithms
S←D X←∅ while S ̸= ∅ do ∑ ∑ Choose i ∈ F and Y ⊆ D − S minimizing (fi − j ∈S / (c(j, X) − cij )+ + j∈Y cij )/Y  fi ← 0; S ← S − Y Open all facilities in X, assign client j to closest facility in X Algorithm 9.2: Greedy algorithm for the uncapacitated facility location problem. First, recall the dual of the linear programming relaxation of the uncapacitated facility location problem that we introduced in Section 4.5: ∑ maximize vj j∈D
subject to
∑
wij ≤ fi ,
∀i ∈ F,
j∈D
vj − wij ≤ cij , wij ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F, j ∈ D.
We claim that the greedy algorithm above can be restated in the following way. Each facility will make a bid αj towards its share of the service and facility costs. We increase the bids of clients uniformly until each client is connected to a facility whose cost is paid for by the bids. A client j that is not connected to a facility i bids the diﬀerence of αj and the service cost towards the cost of facility i; that is, it bids (αj − cij )+ towards the cost of facility i. When the total of the bids on a facility i equals its facility cost fi , we open the facility i. We also allow connected clients to bid the diﬀerence in service costs towards the facility cost of a closer facility; that is, if client j is currently connected to a facility in X, it bids (c(j, X) − cij )+ towards the facility cost of i. If facility i is opened, then client j connects itself to facility i instead, decreasing its service cost by exactly (c(j, X) − cij )+ . Once every client is connected to some open facility, the algorithm terminates. We summarize the algorithm in Algorithm 9.3. For ease of proofs, it turns out to be better to have the algorithm use facility costs fˆi = 2fi . As in the statement of the greedy algorithm, let the set S ⊆ D keep track of which clients have not yet been connected to an open facility, and let X ⊆ F keep track of the currently open facilities. We leave it as an exercise (Exercise 9.1) to prove that the two algorithms ∑ are equivalent. The basic idea is that the value of client j’s bid α is the value of the ratio (f − j i j ∈S / (c(j, X) − ∑ cij )+ + j∈Y cij )/Y  when j is ﬁrst connected to a facility. We observe in passing that there are some strong similarities between Algorithm 9.3 and the primaldual algorithm for the uncapacitated facility location problem in Section 7.6. Here we are increasing a bid αj uniformly for all unconnected clients, while in the primaldual algorithm, we increase a dual variable vj for each client uniformly until the dual inequality associated with a facility becomes tight, or until a client connects to a temporarily opened facility. However, in that algorithm, we only open a ∑ subset of the temporarily opened facilities, and in order to i, where the sum is over all remain dual feasible we need that j (v ∑ ∑j − cij )+ ≤ fi for facilities clients j. In this algorithm we allow j∈S (αj − cij )+ + j ∈S / (c(j, X) − cij )+ ≤ fi . In this algorithm, the clients j not in S only contribute (c(j, X) − cij )+ towards the sum, while in the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.4
A greedy algorithm for the uncapacitated facility location problem
249
α←0 S←D X←∅ fˆi = 2fi for all i ∈ F while S ̸= ∅ do // While not all clients neighbor a facility in X Increase αj for all j ∈ S uniformly until [∃j ∈ S, i ∈ X such that αj = cij ] or ∑ ∑ ˆ [∃i ∈ F − X : j∈S (αj − cij )+ + j ∈S / (c(j, X) − cij )+ = fi ] if ∃j ∈ S, i ∈ X such that αj = cij then // j becomes a neighbor of an existing facility i in X S ← S − {j} else // facility i is added to X X ← X ∪ {i} for all j ∈ S such that αj ≥ cij do S ← S − {j} Open all facilities in X, assign client j to closest facility in X Algorithm 9.3: Dual ﬁtting algorithm for the uncapacitated facility location problem. primaldual algorithm they contribute the potentially larger amount of (vj − cij )+ . For this reason, the bids α are not in general feasible for the dual linear program. We will shortly prove the following two lemmas. Let α be the ﬁnal set of bids from Algorithm 9.3, and let X be the set of facilities opened by the algorithm. The ﬁrst lemma says that the total bids of all clients equals the cost of the solution with facility costs fˆ. The second lemma says that α/2 is dual feasible. Lemma 9.10: For α and X given by the Algorithm 9.3, ∑ ∑ ∑ αj = c(j, X) + 2 fi . j∈D
j∈D
i∈X
Lemma 9.11: Let vj = αj /2, and let wij = (vj − cij )+ . Then (v, w) is a feasible solution to the dual. From these two lemmas, it is easy to show the following theorem. Theorem 9.12: Algorithm 9.3 is a 2approximation algorithm for the uncapacitated facility location problem. Proof. Combining Lemmas 9.10 and 9.11, we have that ∑ ∑ ∑ ∑ c(j, X) + fi ≤ c(j, X) + 2 fi j∈D
i∈X
j∈D
=
∑
i∈X
αj
j∈D
= 2
∑
vj
j∈D
∑
≤ 2 OPT,
where the ﬁnal inequality follows since j∈D vj is the dual objective function, and by weak duality is a lower bound on the cost of the optimal integer solution. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
250
Further uses of greedy and local search algorithms
Note that we actually prove that ∑ ∑ ∑ c(j, X) + 2 fi ≤ 2 vj j∈D
i∈X
j∈D
for the feasible dual solution (v, w). Thus the algorithm is Lagrangean multiplier preserving as we deﬁned it at the end of Section 7.7. As we argued there, plugging this algorithm into the algorithm for the kmedian problem given in that section results in a 2(2 + ϵ)approximation algorithm for the kmedian problem for any ϵ > 0. We now turn to the proofs of Lemmas 9.10 and 9.11. Proof of Lemma 9.10. We will prove by induction on the algorithm that at the beginning of each execution of the main while loop, ∑ ∑ ∑ αj = c(j, X) + 2 fi . j∈D−S
j∈D−S
i∈X
Since at the end of the algorithm S = ∅, this implies the lemma. The equality is initially true since initially S = D and X = ∅. In each execution of the while loop, either we connect some j ∈ S to a facility i already in X or we open a new facility in X. In the ﬁrst case, we have that αj = c(j, X), and we remove j from S. Thus the lefthand side of the equality increases by αj and the righthand side by c(j, X), so the equality continues to ∑ ∑ ˆ hold. In the second case, we have that j∈S (αj − cij )+ + j ∈S / (c(j, X) − cij )+ = fi , and i is ′ added to X. The algorithm removes from S all j ∈ S such that αj − ∑ cij ≥ 0. Let S represent this subset of S. Thus the lefthand side of the equality increases by j∈S ′ αj . Let S ′′ be the set of all j ∈ / S that make positive bids for facility i; that is, (c(j, X) − cij )+ > 0 for j ∈ S ′′ . Note that all of the clients in S ′′ are exactly those closer to i than any other facility in X, so when i is added to X, c(j, X ∪ {i}) = cij for all j ∈ S ′′ . Thus the change in the cost of the right hand side is ∑ ∑ ∑ ∑ 2fi + cij + (c(j, X ∪ {i}) − c(j, X)) = 2fi + cij − (c(j, X) − cij )+ . j∈S ′
j∈S ′′
j∈S:αj ≥cij
j ∈S /
∑ ∑ this for Using the fact that 2fi = fˆi = j∈S (αj − cij )+ + j ∈S / (c(j, X) − cij )+ , and substituting ∑ 2fi in the above, we obtain that the change in cost of the righthand side is j∈S:αj ≥cij αj = ∑ j∈S ′ αj , which is exactly the change in cost of the lefthand side. Thus the equality continues to hold. To prove Lemma 9.11, we ﬁrst prove a sequence of lemmas. In proving these lemmas, we use a notion of time in the algorithm. The algorithm starts at time 0, and uniformly increases all αj with j ∈ S. At time t, any client j not yet connected to a facility (and thus j ∈ S) has αj = t. Lemma 9.13: Consider the time αj at which j ﬁrst connects to some facility. Then the bid of client k on facility i at that time, for any client k such that αk ≤ αj , is at least αj − cij − 2cik . Proof. Either client k connects to a facility at the same time as j and αk = αj , or it connects to a facility at an earlier time that j, and αk < αj . If k connects to a facility at the same time as j, then αj = αk and at time αj its bid on facility i is (αk − cik )+ = (αj − cik )+ ≥ αj − cij − 2cik . Now suppose k connects to a facility at an earlier time than j. Let h be the facility that client k is connected to at time αj . Then at time αj , the bid that k oﬀers facility i is (chk −cik )+ . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.4
A greedy algorithm for the uncapacitated facility location problem
251
By the triangle inequality, we know that chj ≤ cij +cik +chk . Furthermore, since j ﬁrst connects to a facility at a time later than αk , it must be the case that j did not earlier connect to h, and so αj ≤ chj . Thus we have αj ≤ cij + cik + chk . So the bid of client k on facility i at time αj is (chk − cik )+ ≥ chk − cik ≥ αj − cij − 2cik , as claimed. Lemma 9.14: Let A ⊆ D be any subset of clients. Reindex the clients of A so that A = {1, . . . , p} and α1 ≤ · · · ≤ αp . Then for any j ∈ A, j−1 ∑
(αj − cij − 2cik ) +
k=1
p ∑ (αj − cik ) ≤ fˆi . k=j
Proof. We know that any time, the sum of the bids on facility i is at most the facility cost fˆi . By Lemma 9.13 at time αj , for all clients k with k < j, the bid of k for facility i is at least αj − cij − 2cik . For all clients k ≥ j, since αk ≥ αj , at any time just before αj , they have not connected to a facility, so their bid on facility i at time αj is (αj − cik )+ ≥ αj − cik . Putting these together gives the lemma statement. Lemma 9.15: Let A ⊆ D be any subset of clients. Reindex the clients of A so that A = {1, . . . , p} and α1 ≤ · · · ≤ αp . Then ∑ (αj − 2cij ) ≤ fˆi . j∈A
Proof. If we sum the inequality of Lemma 9.14 over all j ∈ A, we obtain p j−1 p ∑ ∑ ∑ (αj − cij − 2cik ) + (αj − cik ) ≤ pfˆi . j=1
k=1
k=j
This is equivalent to p
p ∑ j=1
which implies
p p p ∑ ∑ ∑ (k − 1)cik − p cik − (p − k)cik ≤ pfˆi , αj − k=1
k=1
p ∑
k=1
(αj − 2cij ) ≤ fˆi .
j=1
We can ﬁnally prove that vj = αj /2 gives a feasible dual solution. Proof of Lemma 9.11. Let vj = αj /2,∑ and let wij = (vj − cij )+ . Then certainly vj − wij ≤ cij . Now we must show that for all i ∈ F , j∈D wij ≤ fi . To do this, ∑pick an arbitrary i ∈ F , and let A = {j ∈ D : wij > 0}, so that it is suﬃcient to prove that j∈A wij ≤ fi . We have from Lemma 9.15 that ∑ (αj − 2cij ) ≤ fˆi . j∈A
Rewriting, we obtain
∑
(2vj − 2cij ) ≤ 2fi .
j∈A
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
252
Further uses of greedy and local search algorithms
1
n+1
2
n
3
n+2
n+3
2n
2n + 1
Figure 9.5: Instance for Exercise 9.2 showing a bad locality gap for the local search algorithm of Section 9.1. Dividing both sides by 2, we get
∑
(vj − cij ) ≤ fi .
j∈A
Finally, by the deﬁnition of A and w we have that wij = vj − cij for j ∈ A, and we are done. A signiﬁcantly more involved analysis of this algorithm shows that its performance guarantee is 1.61; see the notes at the end of the chapter for more details.
Exercises 9.1 Prove that the uncapacitated facility location algorithms in Algorithm 9.2 and Algorithm 9.3 are equivalent. 9.2 The locality gap of a local search algorithm for an optimization problem is the worstcase ratio of the cost of a locally optimal solution to the cost of an optimal solution, where the ratio is taken over all instances of the problem and over all locally optimal solutions to the instance. One can think of the locality gap as an analog of the integrality gap of a linear programming relaxation. We consider the locality gap of the local search algorithm for the uncapacitated facility location problem in Section 9.1. Consider the instance shown in Figure 9.5, where the facilities F = {1, . . . , n, 2n + 1}, and the clients D = {n + 1, . . . , 2n} . The cost of each facility 1, . . . , n is 1, while the cost of facility 2n + 1 is n − 1. The cost of each edge in the ﬁgure is 1, and the assignment cost cij is the shortest path distance in the graph between i ∈ F and j ∈ D. Use the instance to show that the locality gap is at least 3 − ϵ for any ϵ > 0. 9.3 Show that the local search algorithm of Section 9.3 can be adapted to ﬁnd a Steiner tree whose maximum degree is at most OPT +1, where OPT is the maximum degree of a minimumdegree Steiner tree. 9.4 Recall the uniform labeling problem from Exercise 5.10: we are given a graph G = (V, E), costs ce ≥ 0 for all e ∈ E, and a set of labels L that can be assigned to the vertices of V . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
9.4
A greedy algorithm for the uncapacitated facility location problem
253
There is a nonnegative cost civ ≥ 0 for assigning label i ∈ L to vertex v ∈ V , and an edge e = (u, v) incurs cost ce if u and v are assigned diﬀerent labels. The goal of the problem is to assign each vertex in V a label so as to minimize the total cost. In Exercise 5.10, we gave a randomized rounding 2approximation algorithm for the problem; here we give a local search algorithm with a performance guarantee of (2 + ϵ). Our local search algorithm will use the following local move. Given a current assignment of labels to vertices in V , it picks some label i ∈ L and considers the minimumcost iexpansion of the label i; that is, it considers the minimumcost assignment of labels to vertices in V in which each vertex either keeps its current label or is relabeled with label i (note that all vertices currently with label i do not change their label). If the cost of the labeling from the iexpansion is cheaper than the current labeling, then we switch to the labeling from the iexpansion. We continue until we ﬁnd a locally optimal solution; that is, an assignment of labels to vertices such that the minimumcost iexpansion for each i ∈ L costs no less than the current assignment. (a) Prove that for any given label i ∈ L, we can compute the minimumcost iexpansion in polynomial time (Hint: ﬁnd a minimum st cut in a graph where s corresponds to the label i and t corresponds to all other labels). (b) Prove that any locally optimal assignment has cost at most twice the optimal cost. (c) Show that for any constant ϵ > 0, we can obtain a (2 + ϵ)approximation algorithm.
9.5 The online facility location problem is a variant of the uncapacitated facility location problem in which clients arrive over time and we do not know in advance which clients will want service. As before, let F be the set of facilities that can be opened, and let D be a set of potential clients. Let fi be the cost of opening facility i ∈ F and cij the cost of assigning a client j ∈ D to facility i ∈ F . We assume that the assignment costs obey the triangle inequality. At each time step t, a new set Dt ⊆ D of clients arrive, and they must be connected to open facilities. We are allowed to open new facilities in each time step; once a client is assigned to a facility, it cannot be reassigned if a closer facility opens later on. For each time step t, we wish to minimize the total cost (facility plus assignment) incurred by all clients that have arrived up to and including time t. We compare this cost with the optimal cost of the uncapacitated facility location problem on the total set of clients that have arrived up to and including time t. The ratio of these two costs gives the competitive ratio of the algorithm for the online problem. Consider the following variation on Algorithm 9.3. As before, we let S be the set of clients that have not yet been connected to some facility, and let X be the set of currently open facilities. At each time step t, we sequence through the clients j in Dt . We increase the client’s bid αj from zero until either it connects to some previously open facility (αj = cij for some i ∈ X), or some facility receives enough bids to allow it to open. As in the greedy algorithm, we allow previously connected clients j to bid toward facility i the diﬀerence between the cost c(j, X) of connecting to the closest open facility and cost of connecting to facility i; that is,∑j bids (c(j, X) − cij )+ towards facility i. Thus facility i is opened when (αj − cij )+ + j ∈S / (c(j, X) − cij )+ = fi . Note that even if facility i is opened and is closer to some client j than previously opened facilities in X, we do not reassign j to i (per the requirements of the problem). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
254
Further uses of greedy and local search algorithms
(a) Prove that at the end of each time ∑ step t, the cost of the current solution is at most twice the sum of the client bids j∈D αj . (b) Consider two clients j and k such that we increase the bid for j before that of k. Let X be the set of facilities open when we increase αk . Prove that for any facility i, c(X, j) − cij ≥ αk − cik − 2cij . (c) For any time step t, pick any subset A of clients that have arrived so far and any facility i. Let A = {1, . . . , p}, where we increase the bids for the clients in A in order of the indices. ∑ (i) Prove that for any ℓ ∈ A, ℓ(αℓ − ciℓ ) − 2 j 0 if the minimum nonzero distance between points is at least 4, and all coordinates xi , yi are integers in [0, O(n)], where n is the number of points. Lemma 10.1: Given a polynomialtime approximation scheme for nice Euclidean TSP instances, we obtain a polynomialtime approximation scheme for all Euclidean TSP instances. Proof. We will show how to transform any instance into a nice instance such that the cost of any tour in the transformed instance doesn’t diﬀer by much from that of the original instance. This will allow us to prove the lemma. Let L be the length of a side of the smallest axisaligned square containing all of the points of the instance; thus L = max(maxi xi − mini xi , maxi yi − mini yi ). Note that since there are two points at least distance L apart it is the case that L ≤ OPT. Let ϵ > 0 be a constant parameter that we specify later. To get a nice instance, we create a grid of horizontal and vertical lines where the spacing between the lines is ϵL/2n; see Figure 10.1. We then move each point of the original instance to the nearest grid point (that is, the nearest intersection of a horizontal and a vertical grid line). Because we end up moving any point by a distance of at most ϵL/2n, the distance between any two points changes by at most ±2ϵL/2n. Thus the overall cost of any tour changes by at most ±ϵL. We now increase the grid spacing by a factor of 8n/ϵL, so that the spacing between any ϵL 8n two grid lines is 2n · ϵL = 4 and translate the bounding square so the lower lefthand corner is on the origin. This enforces that each point in the modiﬁed instance is now at nonnegative integer coordinates and that the minimum nonzero distance between any two points is at least 4. Furthermore, the maximum distance between any two points before increasing the grid spacing was at most 2L; after increasing grid spacing it is at most 2L · 8n ϵL = O(n), so that each x, y Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.1
The Euclidean traveling salesman problem
259
L 2n
L
L
Figure 10.1: An example of the smallest square containing all the points of the instance, and the grid of lines spaced ϵL/2n apart. coordinate is in the range [0, O(n)]. Thus for any tour in the original instance of cost C, its 8n cost in the nice instance is at least 8n ϵL (C − ϵL) and at most ϵL (C + ϵL). Let OPT be the cost of an optimal tour in the original instance, OPT′ the cost of an optimal tour in the corresponding nice instance, C ′ the cost of the tour returned by the PTAS on the nice instance, and C the cost of the corresponding tour in the original instance. We know that C ′ ≤ (1 + ϵ) OPT′ . Furthermore, it must be the case that OPT′ is no more than the cost of the optimal tour of the original instance in the corresponding nice instance, so that OPT′ ≤ 8n ϵL (OPT +ϵL). Putting everything together, we have that 8n 8n (C − ϵL) ≤ C ′ ≤ (1 + ϵ) OPT′ ≤ (1 + ϵ) (OPT +ϵL), ϵL ϵL or C − ϵL ≤ (1 + ϵ)(OPT +ϵL). Recalling that L ≤ OPT, we have that C ≤ (1 + 3ϵ + ϵ2 ) OPT . Choosing ϵ suitably small will allow us to obtain a tour of cost at most (1 + ϵ′ ) OPT for any choice of ϵ′ . From here on, we assume we have a nice Euclidean TSP instance, and we show how to obtain a PTAS for such instances. In order to perform dynamic programming on nice instances, we need to structure the problem so that we can build up a solution from smaller subproblems. To do this, we are going to recursively divide the plane into squares in a way that involves randomization. First, we will describe the recursive division without randomization, and then we will introduce the randomization. As in the proof of Lemma 10.1, we let L be the length of a side of the smallest square containing all the points of the instance. Let L′ be the smallest power of 2 that is at least 2L. We will take a square of side length L′ around the points of the instance and divide it into Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
260
Further uses of rounding data and dynamic programming
Figure 10.2: An example of a dissection. The level 0 square is outlined in a thick black line; the level 1 squares in thin black lines, and the level 2 squares in dashed lines. The thin black lines are level 1 lines, and the dashed lines are level 2 lines.
four equallysized squares, then recursively divide each of these squares into four equallysized squares, and so on; see Figure 10.2. We stop the process when each square has side length 1. We call this division of the instance into squares a dissection. It will be useful to refer to the level of a given square of the dissection; we say that the toplevel square of side length L′ is at level 0, the four squares dividing the toplevel square are at level 1, and so on. Since the maximum distance between points in a nice instance is O(n), we know that L′ = O(n). Therefore, the level of the smallest squares of the dissection (of side length 1) is O(log n). Since the minimum nonzero distance between any two points is 4, we know that in the smallest squares of side length 1, there can be at most one distinct point. So far we have not completely speciﬁed how to create the dissection because there are many squares of side length L′ that contain all the points of the instance. In fact, any translation of the square which has its lower lefthand coordinates at (a, b) for a, b ∈ (−L′ /2, 0] will work. It will be useful for us to choose this translation randomly by choosing integers a and b from (−L′ /2, 0] uniformly at random. We will call such a dissection after a and b are chosen an (a, b)dissection. We will give an intuitive explanation for the randomization in a moment. Now, to further help construct a dynamic program for the problem, we will consider tours that enter and exit the squares of the dissection only in prespeciﬁed points called portals. For each square of level i, we place portals at all four corners, then m − 1 additional portals equally spaced along each side, for m some power of two. We will call m the portal parameter. Note that since m is a power of two, each portal on the sides of a level i − 1 square are at the same location as a portal on the side of some level i square contained in the level i − 1 square. See Figure 10.3 for an illustration of portals. We now consider ptours, which are tours that optionally include portals; as in the case of the Steiner tree problem (from Exercise 2.5), the tour must include the points of the TSP instance, but may include portals. We say that a ptour is portalrespecting if for every square of the dissection, the tour only enters and exits the square via a portal for that square. See Figure 10.4 for an example of a portalrespecting ptour. We’ll say that a portalrespecting ptour is rlight if for every square of the dissection it crosses each side of the square at most r Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.1
The Euclidean traveling salesman problem
261
Figure 10.3: Portals added to the squares of the dissection with portal parameter m = 2. The black squares are portals for both the level 1 and level 2 squares; the white squares are portals for the level 2 squares. times. We can now state two central theorems in obtaining an approximation scheme for Euclidean TSP instances. The second builds on the ﬁrst and is somewhat more complicated to prove. Theorem 10.2: If we pick integers a and b from (−L′ /2, 0] uniformly at random, then with probability at least 1/2, the (a, b)dissection has an rlight portalrespecting ptour of cost at most (1 + ϵ) OPT for portal parameter m = O( 1ϵ log L′ ) and r = 2m + 4. Theorem 10.3: If we pick integer a and b from (−L′ /2, 0] uniformly at random, then with probability at least 1/2, the (a, b)dissection has an rlight portalrespecting ptour of cost at most (1 + ϵ) OPT for portal parameter m = O( 1ϵ log L′ ) and r = O( 1ϵ ). We will give the proofs of these theorems in a moment; the essential idea is that we will be able to modify an optimal tour to be a portalrespecting ptour without increasing the cost by too much. Any time the optimal tour crosses a square at a point that is not a portal for that square, we move it to the nearest portal; the distance we need to move the crossing depends on the level of the square, which given the randomized choice of the (a, b)dissection will trade oﬀ nicely with the probability that the square is of any given level. Similarly, if the tour crosses a side of a square too many times, we will be able to modify it so that it crosses fewer times at a cost proportional to the length of the side of the square; again, given the randomized choice of the (a, b)dissection, this cost trades oﬀ nicely with the probability that the square is of a given level. Before we show how this can be done, we ﬁrst show that for a nice Euclidean instance, we can ﬁnd a tour of cost at most (1 + ϵ) OPT in polynomial time, given the theorems, where the running time depends on the portal parameter and how many times the tour crosses each side of each square in the dissection. Theorem 10.4: With probability at least 1/2, we can ﬁnd a tour of cost at most (1 + ϵ) OPT in O(mO(r) n log n) time for nice Euclidean TSP instances. Recalling that L′ = O(n), that m = O( 1ϵ log L′ ) = O( 1ϵ log n) and that (log n)log n = nlog log n , Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
262
Further uses of rounding data and dynamic programming
Figure 10.4: A portalrespecting ptour. Note that only black portals are used to enter or exit the level 1 squares, while the portals on the level 2 squares are used to enter or exit the level 2 squares. we get the following corollaries. Corollary 10.5: For the choice of m and r as in Theorem 10.2, the running time is O(nO(log log n) ) for constant ϵ > 0. Corollary 10.6: For the choice of m and r as in Theorem 10.3, the running time is O(n logO(1/ϵ) n) for constant ϵ > 0. Proof of Theorem 10.4. From both Theorems 10.2 and 10.3, we know that with probability at least 1/2, with a, b chosen as in the theorem, there is an (a, b)dissection with an rlight portalrespecting ptour of cost at most (1 + ϵ) OPT for m = O( 1ϵ log L′ ) = O( 1ϵ log n); the only distinction between the two theorems is the value of r. For a given (a, b)dissection, we’ll show that we can ﬁnd the cheapest rlight portalrespecting ptour in O(mO(r) n log n) time, given the values of m and r. Note than any ptour can be converted to a standard tour of no greater cost by shortcutting the portals. As we mentioned before, we will ﬁnd the cheapest such ptour by dynamic programming. Consider any portalrespecting ptour of the given instance. Then for any given square in the (a, b)dissection, the ptour may enter the square, visit some of the points inside it, then exit, visit points outside the square, then reenter the square, and so on. We call the part of the ptour inside the square a partial ptour; note that a partial ptour visits all of the points of the TSP instance inside the square; see Figure 10.5. Because the ptour is rlight, we know that the ptour crosses each side of the square at most r times, and so it uses at most 4r portals on the sides of the square. Furthermore, we can pair up the portals used by the ptour into pairs, where each pair represents a portal through which the ptour entered the square, and a portal through which the ptour then left the square. The goal of our dynamic program will be to compute for every square in the dissection, for every choice of up to r portals per side, and for every pairing of the portals into entry/exit pairs, the cheapest partial ptour that visits all the points of the TSP instance inside the square. Clearly if we can do this, we can ﬁnd the cheapest overall rlight portal respecting ptour by considering the entry in the dynamic programming table for the level 0 square that does not use any of the portals on the boundary of the square. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.1
The Euclidean traveling salesman problem
263
Figure 10.5: Partial ptour inside a square. To begin discussion of the dynamic program, we start by computing the number of entries in the table that we will need. Note that for any given level of squares in the dissection, there are at most n that contain any points of the TSP instance. For any square that contains no points of the instance, we need only consider one square of any given size. Since there are O(log L′ ) = O(log n) levels of the dissection, there are at most O(n log n) squares that we must consider. For each square of level i, there are 4m portals on the square and thus at most (4m + 1)4r choices for the up to r portals on each side of the square (including not choosing any portals) through which an rlight portalrespecting ptour could cross. Finally, there are at most (4r)! possible pairings of the selected portals into entry/exit pairs. With r = O(m), the total number of entries in the table is O(n log n) × (4m + 1)4r × (4r)! = O(mO(r) n log n). Now we discuss how to build up the entries in the table. For each distinct point, we ﬁnd the largest square in the dissection that contains only that point. The case for each such square is straightforward; for each choice of portals and pairings of portals, we ﬁnd the shortest paths that enter/exit the square in the designated way and visit the one distinct point inside the square. We now build up the solutions to other entries in the table from previous entries, working our way up from smaller squares of the dissection to larger squares. We construct solutions for a square S from the solutions for the four smaller squares s1 , . . . , s4 that it contains. Note that any partial ptour for S might use portals on the four sides of the si that are not also on a side of S. Let’s call these sides the internal sides of the si . To combine solutions from the si to get a solution for the square S, we enumerate over all the portals on the internal sides that the partial ptour for S might have used, and the order in which the partial ptour visited these portals, and pick the best solution found. Notice that specifying a set of portals used on the internal sides and an order in which they are used implies for each si a set of portals which are used, as well as an entry/exit pairing on the portals, so we will be able to look up the best solution for each square si in our table. Therefore, given a square S, a set of portals of S that are used, and entry/exit pairings of the portals, we enumerate over all the portals on the internal sides that a partial ptour for this conﬁguration might have used, and an ordering on the portals. We can pick up to r portals from each of the four internal sides, each of which has m + 1 portals, so Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
264
Further uses of rounding data and dynamic programming
Figure 10.6: Illustration of shortcutting a portal. that there are no more than (m + 2)4r possibilities (including not choosing any portal). Since the partial ptour inside S uses at most 4r portals on the sides of S, there can be at most 2r paths entering and exiting S; thus we need to specify on which of these 2r paths the portals on the internal sides lie. This gives another (2r)4r possibilities. Finally, there are at most (4r)! ways to order the portals along the paths. Thus if we enumerate over the (m + 2)4r × (2r)4r × (4r)! = O(mO(r) ) possibilities, we will ﬁnd the best solution for the table entry consisting of the square S, the chosen portals from S, and the entry/exit pairing of the portals. This takes O(mO(r) ) time. Since we do the computation for O(mO(r) n log n) dynamic programming table entries, the total time taken is O(mO(r) n log n). We now turn to the proof of Theorem 10.2. As we mentioned previously, the proof works by showing that with reasonable probability, an optimal tour can be modiﬁed to a portalrespecting ptour without increasing the cost by too much. Notice that an rlight tour with r = 2m + 2 is implied by a tour that crosses the side of any square at most twice at any portal. If the tour crosses three or more times at a portal, it can be shortcut to cross at most twice; see Figure 10.6. Thus we need only to show that modifying the tour to be portalrespecting does not increase the cost by more than ϵ OPT. To prove this, we ﬁrst need some notation and a lemma. We will use ℓ to denote either a vertical line x = i or a horizontal line y = i for some integer i. Given an optimal tour, let t(ℓ) be the number of times that the optimal tour crosses the line ℓ. Let T be the sum of t(ℓ) over all such horizontal and vertical lines ℓ. Lemma 10.7: For nice Euclidean instances, T ≤ 2 OPT. point (x2 , y2 ). The edge Proof. Consider an edge in the optimal tour from point (x1 , y1 ) to √ contributes at most x1 − x2  + y1 − y2  + 2 to T , and has a length s = (x1 − x2 )2 + (y1 − y2 )2 . Recall that for a nice instance the minimum nonzero distance between points of the instance is at least 4, so that s ≥ 4. If we let x = x1 − x2  and y = y1 − y2 , then since (x − y)2 ≥ 0, we Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.1
The Euclidean traveling salesman problem
have that x2 + y 2 ≥ 2xy, or 2x2 + 2y 2 ≥ x2 + 2xy + y 2 = (x + y)2 , or Thus we can bound the contribution of the edge to T as follows:
√
265
2(x2 + y 2 ) ≥ x + y.
x + y + 2 = x1 − x2  + y1 − y2  + 2 √ ≤ 2[(x1 − x2 )2 + (y1 − y2 )2 ] + 2 √ ≤ 2s2 + 2 ≤ 2s, where the last inequality follows since s ≥ 4. Thus summing over all edges in the tour, we obtain that T ≤ 2 OPT . Now we can give the proof of Theorem 10.2, which we restate here for convenience. Theorem 10.2: If we pick integers a and b from (−L′ /2, 0] uniformly at random, then with probability at least 1/2, the (a, b)dissection has an rlight portalrespecting ptour of cost at most (1 + ϵ) OPT for portal parameter m = O( 1ϵ log L′ ) and r = 2m + 4. Proof. As described earlier, the general idea of the proof is that we modify the optimal tour to ensure that it crosses any square of the dissection at a portal for that square. When we move the crossing of the optimal tour, we increase it by an amount that depends on the distance between the portals for that square. As the level of a square gets smaller, this distance is greater, but the probability that any given line ℓ belongs to a square of that level also gets smaller, so that the overall expected distance we need to move the tour will simply depend on t(ℓ) and m, which we can relate to the cost of the optimal tour by Lemma 10.7. To show this formally, deﬁne the level of line ℓ to be the minimum level over all squares of the (a, b)dissection such that ℓ contains the side of the square. Observe that to split the level i squares into level i + 1 squares, we draw 2i horizontal and 2i vertical lines which are then level i + 1 lines; see Figure 10.2 for an illustration. Because we choose a and b uniformly at random, this implies that 2i 2i−1 Pr[level of ℓ is i] ≤ ′ = ′ . L /2 L Thus, for instance, the probability that a vertical line ℓ is the single level 1 vertical line splitting the bounding box in two is at most 1/(L′ /2). We modify the optimal tour so that the tour crosses any square of the dissection at a portal for that square. Consider any line ℓ of level i. Since it contains the boundary of a level i square, and the side length of a level i square is L′ /2i , the distance between portals is at most L′ /2i m. Observe also that by construction the portal for a level i square is also a portal for any smaller, level j square for j > i. Thus if we move any crossing of a level i line ℓ at most L′ /2i m, any square of the dissection whose side is contained in ℓ will have the tour crossing at some portal for that square. Recall that we deﬁned t(ℓ) to be the number of times that the optimal tour crosses line ℓ. Thus the expected increase in cost for moving every crossing of line ℓ to the nearest portal is at most log L′ ∑ i=1
L′ Pr[level of ℓ is i] · t(ℓ) · i 2m
≤
log L′ ∑ i=1
=
2i L′ · t(ℓ) · L′ 2i m
t(ℓ) log L′ . m
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
266
Further uses of rounding data and dynamic programming
We choose the portal parameter m to be the smallest power of two that is at least 4ϵ log L′ . Then the total expected increase is at most 4ϵ t(ℓ). Recall that we deﬁned T to be the sum over all lines ℓ of t(ℓ). Thus the total expected cost of what we have done in moving the crossings to portals as above is ∑ ϵ ϵ t(ℓ) = T. 4 4 lines ℓ Lemma 10.7 shows that T ≤ 2 OPT for nice Euclidean instances. Then we have that the expected increase in cost is at most 4ϵ T ≤ 2ϵ OPT. If the random variable X represents the expected increase in cost, then by Markov’s inequality (Lemma 5.25), Pr[X ≥ ϵ OPT] ≤ E[X]/(ϵ OPT) = ( 2ϵ OPT)/(ϵ OPT) = 1/2. Thus, we obtain that the increase in cost is at most ϵ OPT with probability at least 1/2. To prove Theorem 10.3, we additionally need to show that if a tour crosses a given line too many times, we can modify it at a small increase in cost to a tour that doesn’t cross the line too many times. We will call the following lemma the Patching lemma. Lemma 10.8 (Patching lemma): Given a line segment R of length l, if a tour crosses R three or more times, we can increase the length of the tour by no more than 6l to get a closed path that contains the previous tour and crosses R at most twice. Proof. Suppose the tour crosses the line segment k times. We take the tour, and break it at the k points at which it crosses line R. We add 2k new points by adding k new points on either side of R where the tour crossed R; see Figure 10.7. We now add a cycle and a matching to the new points on each side of the lines; if k is odd, the matching matches the ﬁrst k − 1 points on each side. If k is odd, we add an edge connecting the last pair of new points, while if k is even we add two edges connecting the last two pairs of new points. This results in an Eulerian graph which contains the points of the tour and crosses R at most twice. Each of the two cycles added has cost at most 2l, and each of the two matchings has cost l, for an overall cost of 6l. A traversal of the Eulerian graph contains all points of the tour. We can shortcut the traversal to obtain a closed path that contains all points of the tour. We can now prove Theorem 10.3, which we restate here for convenience Theorem 10.3: If we pick integer a and b from (−L′ /2, 0] uniformly at random, then with probability at least 1/2, the (a, b)dissection has an rlight portalrespecting ptour of cost at most (1 + ϵ) OPT for portal parameter m = O( 1ϵ log L′ ) and r = O( 1ϵ ). Proof. The basic idea is to repeatedly apply the Patching lemma so that the side of each square is not crossed more than r times. The cost of the patching depends on the sidelength of the square, which is larger for squares of smaller level; however, the probability that for any line ℓ that it contains a side of a square of that level is smaller for smaller levels. Once again, these quantities trade oﬀ so that the expected cost for patching a line ℓ is proportional to t(ℓ) and inversely proportional to r, so that for a good choice of r, we can relate the patching cost to the optimal cost via Lemma 10.7. Now we begin the formal proof. Consider a level i line ℓ. We invoke the Patching lemma (Lemma 10.8) to ensure that for every side of every square which ℓ contains, that side is not crossed by the tour more than r times. However, we must apply the Patching lemma carefully. We do the following: we start with the sides of the smallest squares whose sides are contained Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.1
The Euclidean traveling salesman problem
(a)
(b)
(c)
(d)
Figure 10.7: Illustration of the patching lemma: (a) The original tour; (b) Breaking the tour at the points at which it crosses the line; (c) Adding a tour and a matching to both sides of the line; (d) Shortcutting to a closed path.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
267
268
Further uses of rounding data and dynamic programming
in ℓ, and apply the lemma if the side of any square is crossed more than r times. Then we do the same for the next smallest squares, and so on, until we ﬁnish with the level i squares; since ℓ is a level i line, it contains no square side for any larger square. In other words, we perform a loop, setting j to log L′ and decrementing j down to i. In each iteration of the loop, we consider all level j squares that have a side contained in ℓ. If the side is crossed more than r times, we apply the Patching lemma to that side. To bound the cost of applying the Patching lemma, we let cj denote the ∑number of times that the Patching lemma is applied when considering squares of level j. Then j≥1 cj ≤ t(ℓ)/(r −1), since each time the Patching lemma is invoked, it replaces at least r + 1 crossings with at most 2. By the Patching lemma, the cost of replacing at least r crossings of the side of a level j square with at most two crossings is at most 6 times the side length of a level j square; thus the cost is at most 6L′ /2j . Thus if ℓ is level i, the total cost of applying the Patching lemma as ∑ L′ ′ j given above is log j=i cj (6L /2 ). Since the level of line ℓ depends on the random choices of a and b at the beginning of the algorithm, we have that the expected increase in cost of applying the Patching lemma as discussed above is at most log L′ ∑ i=1
Pr[level of ℓ is i]
log L′ ∑ j=i
6L′ cj j 2
≤
log L′ ∑ i=1
′
log L 2i ∑ 6L′ cj j L′ 2 j=i
log L′
= 6
j ∑ cj ∑ 2i 2j i=1
j=1
log L′
≤ 6
∑
2cj
j=1
≤ 12t(ℓ)/(r − 1). As in the proof of Theorem 10.2, we can modify the tour to cross the line ℓ only at portals ′ with an expected increase in cost of at most t(ℓ) m log L . Choosing m to be the smallest power of ′ two that is at least (r − 1) log L , this expected increase becomes at most t(ℓ)/(r − 1). Recall that we deﬁned T to be the sum over all lines ℓ of t(ℓ). Thus the total expected increase in cost of moving the crossings to portals as above is ∑ 13 13t(ℓ)/(r − 1) = T. r−1 lines ℓ Setting r = 52 ϵ + 1 and recalling from Lemma 10.7 that T ≤ 2 OPT for nice Euclidean instances, we have that the expected increase in cost is at most 4ϵ T ≤ 2ϵ OPT. As we argued at the end of the proof of Theorem 10.2, this implies that we obtain that the increase in cost is at most ϵ OPT with probability at least 1/2. To conclude, we need to argue that we have successfully made the tour into an rlight portalrespecting ptour. This becomes somewhat complicated, because it could be the case that in reducing the number of crossings for a vertical line ℓ we may increase the number of crossings for a horizontal line ℓ′ that intersects it. We note that the additional vertical crossings of ℓ′ introduced by the Patching lemma in reducing the crossings of ℓ are immediately on either side of ℓ, and thus go through the portal corresponding to the intersection of ℓ and ℓ′ . We can apply the Patching lemma again to ensure that the number of times ℓ′ is crossed at this point is at most twice at no increase in cost (because the crossings all occur at a single geometric point). Thus reducing the crossings of vertical lines ℓ may increase the number of crossings at Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.2
The maximum independent set problem in planar graphs
269
the corners of the squares of dissections by at most 4; we get 2 from each application of the Patching lemma on each side of ℓ; this increases the number of crossings on a side of a square by at most 8 (4 for each corner). Thus we get an (r + 8)light portalrespecting ptour, which is suﬃcient for the purposes of the theorem. The idea of this algorithm for the Euclidean TSP can be applied to many other standard problems in a Euclidean setting, such as the Steiner tree problem, the kmedian problem, and others.
10.2
The maximum independent set problem in planar graphs
Given an undirected graph G = (V, E), an independent set of vertices S ⊆ V is one such that no two vertices of S are joined by an edge; that is, for all i, j ∈ S, (i, j) ∈ / E. If we are given nonnegative weights wi ≥ 0 for all vertices i ∈ V , then the maximum ∑ independent set problem is that of ﬁnding an independent set S of maximum weight w(S) = i∈S wi . Sometimes we are interested in the unweighted version of the problem, in which wi = 1 for all vertices i ∈ V ; then we are interested in ﬁnding an independent set of maximum cardinality. The maximum independent set problem is essentially identical to another problem introduced at the very beginning of the book: the maximum clique problem. A clique of vertices S ⊆ V is one such that every pair i, j ∈ S has an edge connecting it; that is, for all i, j ∈ S, (i, j) ∈ E. Given nonnegative weights wi ≥ 0 for all vertices i ∈ V , the maximum clique problem is that of ﬁnding a clique of maximum weight. As in the case of the maximum independent set problem, we can consider an unweighted version in which we are interested in ﬁnding a clique of maximum cardinality. To see the connection between the maximum clique and maximum inde¯ = (V, E) ¯ be the complement of graph G = (V, E) where E ¯ is the set pendent set problems, let G ¯ of all pairs of vertices (i, j) that are not edges in E, so that E = {(i, j) : i, j ∈ V, i ̸= j, (i, j) ∈ / E}. ¯ Now notice that any clique S in G is an independent set in G of the same weight and vice versa. Thus given any approximation algorithm for the maximum independent set problem, we can convert it to an approximation algorithm of the same performance guarantee for the maximum clique problem simply by running it on the complement graph, and similarly if we have an approximation algorithm for the maximum clique problem. We recall Theorem 1.4 from Chapter 1 which says that the unweighted version of the maximum clique problem is very hard to approximate. Theorem 10.9 (Theorem 1.4): Let n denote the number of vertices in an input graph, and consider any constant ϵ > 0. Then there does not exist an O(nϵ−1 )approximation algorithm for the unweighted maximum clique problem, unless P = NP. Thus the maximum independent set problem is very hard to approximate as well. Nevertheless, for some special classes of graphs, it is possible to do better – in fact, much better. We observe that it is relatively easy to ﬁnd the maximum independent set in a tree in polynomial time via a dynamic programming algorithm. We suppose that the tree T is rooted at some node. The dynamic program will work in a bottomup fashion, starting with the leaves and working up to the root. Let Tu be the subtree of T rooted at a node u. The dynamic programming table will have two entries for each node u in the tree, I(Tu , u) and I(Tu , ∅): I(Tu , u) is the weight of a maximum independent set of Tu which includes u, and I(Tu , ∅) is weight of the maximum independent set of Tu which excludes u. If u is a leaf, we can easily compute the two entries for u. Now suppose that u is an internal node, with k children v1 , . . . , vk , and suppose we have already computed the entries for each child. We can Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
270
Further uses of rounding data and dynamic programming
Figure 10.8: Illustration of planar embeddings of planar graphs. Both graphs are planar, but only the bottom graph is outerplanar. then compute the two entries for u. Clearly if u is included in the independent set for Tu , then v1 , . . . , vk must be excluded, so the maximum independent set for Tu including u is u plus the union of the maximum independent sets for the Tvi excluding vi ; that is, I(Tu , u) = wu +
k ∑
I(Tvi , ∅).
i=1
If u is excluded from the independent set for Tu , then we have the choice for each child vi about whether we should take the maximum independent set of Tvi including vi or excluding vi . Since there are no edges from any Tvi to Tvj for i ̸= j, the decision for Tvi can be made independently from that for Tvj and we can choose either possibility; we simply pick the set of largest weight for ∑ each vi , and this gives the maximum independent set for Tu excluding u. Thus I(Tu , ∅) = ki=1 max(I(Tvi , vi ), I(Tvi , ∅)). Once we have computed both entries for the root vertex r, we take the entry that has largest weight, and this gives the maximum independent set for the entire tree. Trees are a very restricted class of graphs; we would like to be able to devise good algorithms for larger classes of graphs. In this section, we will show that for planar graphs, it is possible to get a polynomialtime approximation scheme for the maximum independent set problem. Planar graphs are graphs which can be drawn in the Euclidean plane without crossing edges. More precisely, we correspond each vertex of the graph to some Euclidean point in the plane, and each edge (i, j) of the graph to a curve in the plane joining the points corresponding to i and j. The graph is planar if it is possible to do this such that no two curves corresponding to edges intersect. The mapping of the graph G to points and curves is called a planar embedding for planar graphs G. A graph G is outerplanar if it has a planar embedding such that all the vertices lie on the exterior face of the embedding; roughly speaking they are all on the “outside” of the drawing of the graph. See Figure 10.8 for examples of planar and outerplanar graphs. To obtain the PTAS for planar graphs, we will need to deﬁne the concept of a kouterplanar Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.2
The maximum independent set problem in planar graphs
271
1 1 2 1 2 2 2
1 1
Figure 10.9: A 2outerplanar graph and a 3outerplanar graph. The levels of the vertices in the 2outerplanar graph are labelled. graph. Given a planar embedding of a graph G, we say that all the vertices on the exterior face are level 1 vertices of the graph. Then in general the level i vertices of the graph are all the vertices on the exterior face of the same planar embedding of G after removing all level 1, . . . , i − 1 vertices and incident edges. A graph G is kouterplanar if there exists a planar embedding of G such that all the vertices of the graph are level k or less; see Figure 10.9 for examples of kouterplanar graphs. It is possible to determine whether a graph G is kouterplanar in time polynomial in k and n, but the proof of this is outside the scope of this book. The central theorem we will need to prove the PTAS for planar graphs is the following. Theorem 10.10: There is a dynamic programming algorithm for ﬁnding the maximum independent set in kouterplanar graphs with running time O(2O(k) n2 ). The key to proving this theorem is to show that kouterplanar graphs can be decomposed into a treelike structure, so that we can run a dynamic programming algorithm similar to the one given above for trees. Given the theorem, it is relatively simple to obtain the approximation scheme. Theorem 10.11: There is a polynomialtime approximation scheme for the maximum independent set problem in planar graphs running in time O(2O(1/ϵ) n2 ). Proof. Given a planar graph G with a planar embedding of G, we let Li denote all the level i vertices of G. Observe that by the deﬁnition of level, it cannot be the case that an edge has endpoints that diﬀer in level by two or more, since then in the embedding the curve corresponding to the edge would intersect the curves corresponding to edges connecting intermediate level vertices. If we want a (1 − ϵ)approximation algorithm, let k be the smallest positive integer such that 1/k ≤ ϵ. Let Si be the set of all vertices whose level is i(mod k), for i = 0, . . . , k − 1. Now consider the graphs Gi induced on all vertices except those in Si ; that is, Gi = G[V − Si ]. For each i = 0, . . . , k − 1, by removing all the vertices in Si and edges incident on it, we obtain Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
272
Further uses of rounding data and dynamic programming
a graph Gi for which each connected component is a kouterplanar graph. For instance, in G0 , we remove vertices from levels k, 2k, 3k, and so on. Thus the vertices in L1 , . . . , Lk−1 are disconnected from all other vertices in Gi , and these form a kouterplanar graph. Similarly, the vertices in Lk+1 , . . . , L2k−1 are also disconnected from other vertices in Gi , and these also form a kouterplanar graph, and so on. We use the algorithm of Theorem 10.10 to ﬁnd a maximum independent set for each connected component of Gi separately, and the union of these independent sets gives a maximum independent set Xi for Gi . The running time is O(2O(1/ϵ) n2 ). Note that any independent set Xi for Gi is also independent for the original graph G. Then we return as our solution the independent set Xi whose weight is the largest. Let O be a maximum independent set for G, so that w(O) = OPT. We observe that since the Si partition all the vertices in V into k sets, there exists some j such that w(O ∩ Sj ) ≤ w(O)/k. Since O − Sj is an independent set for Gj , the weight of the independent set Xj found by the algorithm for Gj must be at least ( w(O − Sj ) = w(O) − w(O ∩ Sj ) ≥
1−
1 k
) w(O) ≥ (1 − ϵ) OPT .
Thus the algorithm will return a solution of weight at least (1−ϵ) OPT in O(2O(1/ϵ) n2 ) time. We now turn to the proof of Theorem 10.10. To do this, we shall introduce another concept, called the treewidth of a graph. The treewidth of a graph in some sense measures how close the graph is to being a tree. Our proof agenda then will be to show that any graph with treewidth t has a dynamic programming algorithm to solve the maximum independent set problem in time exponential in t but polynomial in n, and to show that kouterplanar graphs have treewidth at most 3k + 2. Given an undirected graph G = (V, E), a tree decomposition of G is a spanning tree T on a new set of nodes V ′ , where each i ∈ V ′ corresponds to a subset Xi of vertices of V . The tree decomposition has the following three properties: 1. For every vertex u ∈ V , there is some i ∈ V ′ such that u ∈ Xi ; 2. For every edge (u, v) ∈ E, there is some i ∈ V ′ such that both u, v ∈ Xi ; and 3. For i, j, k ∈ V ′ , if j lies on the path in T from i to k, then Xi ∩ Xk ⊆ Xj . As an example, suppose that the original graph G is itself a tree. Then we can make a tree decomposition of G by creating a node iu in V ′ for each node u ∈ V and a node ie for each edge in e ∈ E. The tree T will have edges (iu , ie ) for all u ∈ V and e ∈ E such that u is an endpoint of e; note that this gives a tree since G itself is a tree. The node iu ∈ V ′ will have corresponding subset Xiu = {u}, and the node ie ∈ V ′ for edge e = (u, v) ∈ E will have corresponding subset Xie = {u, v}. It can then be seen that this decomposition obeys all three properties of a tree decomposition. See Figure 10.10 for an example of a tree decomposition of a graph. The treewidth of a tree decomposition of G is the maximum over all i ∈ V ′ of Xi  − 1, and the treewidth of a graph G is the minimum treewidth over all tree decompositions of G. One might wonder about the −1 term in the deﬁnition, but note that given the tree decomposition of a tree given above, this deﬁnition yields that trees have a treewidth of 1; since the second property of tree decomposition requires that each edge of the graph have both endpoints in some subset of the decomposition, no smaller treewidth is possible unless the graph has no edges. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.2
The maximum independent set problem in planar graphs
273
3, 4, 5 1
2
3 2, 3, 5
8 1, 2, 5, 7 7
5 6
1, 7, 8
4 5, 6, 7
Figure 10.10: An example of a tree decomposition of a graph; the graph is on the left and the tree decomposition on the right. The treewidth of this tree decomposition is 3. We can now show that we can ﬁnd the maximum independent set in a graph with low treewidth in time exponential in the treewidth. The algorithm is simply a generalization of the dynamic program for ﬁnding a maximum independent set in a tree. Theorem 10.12: Given a tree decomposition of a graph G = (V, E) with nodes V ′ and treewidth t, we can ﬁnd a maximum independent set in G in time O(2O(t) V ′ ) via dynamic programming. Proof. Given the tree decomposition T of graph G = (V, E) with nodes V ′ and sets Xi ⊆ V for all i ∈ V ′ , we root the tree T . As in the dynamic program for trees, our algorithm will work bottomup on the tree T . Now for each i ∈ V ′ , let Ti be the subtree of T rooted at i. Let Vi be the set of vertices given by taking the union of all the Xj for all j in Ti , and let Gi be the subgraph of G induced by Vi . We will compute the maximum independent set of Gi for each i ∈ V ′ ; when we reach the root vertex r of the rooted tree T , the graph Gr = G, and we will have computed its maximum independent set. In our dynamic program, we have 2Xi  entries in the table for each i ∈ V ′ ; these correspond to the maximum independent set in Gi for all possible subsets U ⊆ Xi where we require the intersection of the independent set and Xi to be exactly U . Note that some of these possibilities might not correspond to a valid independent set (for instance, if we include both endpoints of an edge of Gi ), and these table entries will be marked ‘Not valid’. If i is a leaf vertex of T , then Vi = Xi . Then since each table entry for i dictates whether each vertex of Xi is included or excluded, we only need to check whether the given inclusions/exclusions yield a valid independent set to compute the entry. Now suppose that i is an interior node of the tree. To handle this case, we must ﬁrst establish two claims: (1) for any two distinct children j, k of i, Vj ∩ Vk ⊆ Xi ; (2) any edge of Gi must either have both endpoints in Xi or both endpoints in Vj for some child j of i. To see claim (1), suppose that there is some u ∈ Vj ∩ Vk . We have u ∈ Vj because u ∈ Xp for some node p in the subtree Tj and u ∈ Vk since u ∈ Xq for some q in the subtree Tk . Now since i is on the path from p to q in the tree T , by the third property of a tree decomposition, it must be the case that u ∈ Xp ∩ Xq ⊆ Xi . To see claim (2), pick any edge (u, v) in Gi . By the second property of a tree decomposition, there is some node ℓ in the tree such that both u, v ∈ Xℓ . If ℓ = i, then u, v ∈ Xi . If ℓ is in the subtree Ti , but ℓ ̸= i, then ℓ is in the subtree Tj for some child j of i, and both u, v ∈ Vj . Now suppose ℓ is not in the subtree Ti . Since u and v are in Gi , there must be nodes p and q in the subtree Ti such that u ∈ Xp and v ∈ Xq . The node i is Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
274
Further uses of rounding data and dynamic programming
on the path from p to ℓ in the decomposition tree T , and is also on the path from q to ℓ in T . Therefore, by the third property of the tree, u ∈ Xp ∩ Xℓ ⊆ Xi and v ∈ Xq ∩ Xℓ ⊆ Xi so both u, v ∈ Xi . We can now show how to compute the table entries for i when i is an interior node of the tree. For any given valid set U ⊆ Xi , and any child j of i, consider an entry W ⊆ Xj for j. We say that the entry W for the child j is compatible with the entry U for i if U and W agree on Xi ∩ Xj (that is, U ∩ Xi ∩ Xj = W ∩ Xi ∩ Xj ). Then to compute the entry for i for set U ⊆ Xi , for each child j of i, we ﬁnd the compatible entry of maximum weight; call this independent set Sj . We claim that U together with the union of the Sj over all children j is the maximum ∪ independent set for Gi whose intersection with Xi is exactly U ; call this set S = U ∪ j Sj . Note that by claim (1), Vj − Xi are disjoint over all children j of i, so S is well deﬁned. By induction, the Sj are maximum independent sets compatible with U , so if S is an independent set, there cannot be one of larger weight. Also, S is an independent set since by claim (2) any edge (u, v) in Gi must have either both endpoints in Xi or both endpoints in Vj for some child j of i. It cannot be the case that both u and v are in the independent set, since then either both of them were in U and U was not valid, or both of them were in Sj , and Sj is not a valid independent set. For each node in V ′ , there are 2t+1 entries we must compute. We compute these by checking for compatible entries at the child nodes; each of the 2t+1 entries of a child is checked once by the parent node for each entry of the parent node. Hence the overall time taken is O(2O(t) V ′ ). We now turn to showing that kouterplanar graphs have low treewidth. It will be useful for us to consider kouterplanar graphs with maximum degree at most three; we will show that these have treewidth at most 3k + 1. We show below that this is in fact general since any kouterplanar graph G can be transformed into a kouterplanar graph G′ of maximum degree three, such that we can compute a tree decomposition of G from that of G′ and the treewidth of G is bounded above by the treewidth of G′ . Lemma 10.13: For any kouterplanar graph G there is a kouterplanar graph G′ of maximum degree three such that given a tree decomposition of G′ we can compute a tree decomposition of G of no greater treewidth. Proof. Given the kouterplanar graph G with some vertex v of degree d greater than three, we can create a new graph G′ by splitting v into two vertices v1 and v2 joined by an edge, of degree three and degree d − 1, respectively. This can be done in such a way that the graph remains kouterplanar; see Figure 10.11. Given a tree decomposition T ′ of G′ , we can create a new tree decomposition T of G by taking any subset Xi′ containing either v1 or v2 and replacing them with v; that is, set Xi = (Xi′ − {v1 , v2 }) ∪ {v} if Xi′ contains v1 or v2 , otherwise Xi = Xi′ . Then T is a valid tree decomposition of G given that T ′ is a valid tree decomposition of G′ , and clearly the treewidth of the decomposition of G is no greater than that of G′ . Given any kouterplanar graph G of maximum degree greater than three, we can use the transformation above to create a sequence of new graphs G1 , G2 , . . . , Gz such that Gz has maximum degree at most three, and then from a tree decomposition of Gz , work backwards to create a sequence of decompositions of Gz−1 , . . . , G1 , G such that the treewidth of the decomposition of G is at most that of Gz . Given a maximum spanning forest (V, F ) of a graph G, consider any edge e ∈ E − F . The fundamental cycle of e is the cycle closed in the forest by adding e to F . We say that the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.2
The maximum independent set problem in planar graphs
v
v1
275
v2
Figure 10.11: Splitting vertex v into v1 and v2 , and maintaining kouterplanarity. load of a vertex v ∈ V is the number of fundamental cycles in which it participates over all e ∈ E − F . Similarly, the load of a tree edge e ∈ F is the number of fundamental cycles in which it participates over all e ∈ E − F . The maximum load of a maximum spanning forest (V, F ) for a graph G is the maximum over all v ∈ V and e ∈ F of the loads of v and e. Lemma 10.14: Every kouterplanar graph with maximum degree 3 has a maximum spanning forest (V, F ) of maximum load 3k. Proof. We show the result by induction on k. Suppose k = 1, and the graph G is outerplanar. Let R be the set of edges on the exterior face of the embedding of G. Note that after removing the edges of R the resulting graph must be acyclic since G is outerplanar; extend E − R to a maximum spanning forest F of G by adding as many edges of R as possible. Then any edge of E − F is on the exterior face; adding it to F creates a unique interior face of the embedding of G. Since any edge of G bounds at most two interior faces of the embedding, and any vertex bounds a number of interior faces at most its degree, each edge has load at most two and each vertex has load at most three. Now suppose we have a kouterplanar graph for k > 1. Again, let R be all the edges on the exterior face of the embedding of G. If we remove all the edges in R, then since the graph has degree three, the vertices on the exterior face will have degree at most one, and so the remaining graph will be at most (k − 1)outerplanar. By induction, we know that we can ﬁnd a maximum spanning forest F ′ of the graph (V, E − R) such that the maximum load is at most 3(k − 1). We extend F ′ to a maximum spanning forest F of G by adding as many edges of R as possible. As above, adding any edge of R − F closes a unique interior face of a planar graph (V, F ∪ R), and thus the additional load imposed on any edge of F by edges in R − F is at most two, and the additional load imposed on any vertex v by edges in R − F is at most three. See Figure 10.12 for an illustration. Hence the maximum load of the forest (V, F ) for G is at most 3k. Lemma 10.15: If a graph G of maximum degree 3 has a maximum spanning forest (V, F ) such that the maximum load of F is at most ℓ, then G has treewidth at most ℓ + 1. Proof. Given G = (V, E), we start by giving a tree decomposition of the forest F as shown Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
276
Further uses of rounding data and dynamic programming
Figure 10.12: Illustration of the proof of Lemma 10.14 for a 2outerplanar graph. The edges of the maximum spanning forest F , adding edges from R, are shown in thick lines. The remaining edges of R are dotted. previously in the discussion of tree decompositions: that is, we create a node iu for each u ∈ V , and a node ie for each edge e in the forest F . We start by setting Xiu = {u} and Xie = {u, v} for e = (u, v), and the decomposition tree T has edges (iu , ie ) whenever u is an endpoint of an edge e in the forest F . This decomposition obeys the ﬁrst and third properties of tree decompositions, but not the second because there may be edges in E − F that do not have both endpoints in the same subset of the decomposition. We now update the tree for each (u, v) ∈ E − F to ﬁx this problem. Given an edge e = (u, v) ∈ E − F we choose one of the endpoints arbitrarily, say u. Consider the fundamental cycle of e. For every vertex w ̸= v in this cycle, we add u to the set Xiw and for every edge e′ ̸= e in the cycle, we add u to the set Xie′ . We claim this gives a tree decomposition of G. The ﬁrst property is satisﬁed, as is the second for every edge e ∈ F . For any edge e = (u, v) ∈ E − F , consider the fundamental cycle of e; there must be an edge (w, v) ∈ F in this cycle incident on v. We added u to X(w,v) and this set already included v, so the second property is satisﬁed for all edges of E. The third property is also satisﬁed: it was satisﬁed in the initial decomposition of the spanning tree F , and then whenever we added a vertex u to subsets in the decomposition, we added it to all the subsets along a path in the decomposition tree T . Thus for nodes i, j, k in the tree decomposition, if j is on the path from i to k in the decomposition tree T , Xi ∩ Xk ⊆ Xj . Note that initially every set in the decomposition has size at most two, and we add to each subset Xiu a number of vertices that is at most the load of u ∈ V , and we add to each subset Xie a number of vertices that is at most the load of e ∈ E. Thus the treewidth of this decomposition is at most ℓ + 1. We can ﬁnally summarize the discussion above in the following theorem. Theorem 10.16: There is a dynamic programming algorithm to ﬁnd a maximum independent set in a kouterplanar graph in O(2O(k) n2 ) time. Proof. We will use the fact that a planar graph of n vertices has at most m ≤ 3n − 6 edges. We begin by noting that the arguments of Lemmas 10.13, 10.14, and 10.15 can be made algorithmic. We can apply Lemma 10.13 to create a kouterplanar graph of maximum degree at most 3; we create from the original graph of n vertices and m edges a graph with n′ vertices and m′ edges, Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.2
The maximum independent set problem in planar graphs
277
where n′ is at most the sum of the degrees of the original graph. Thus n′ ≤ 2m = O(n), and m′ ≤ 3n′ − 6 = O(n). We then use Lemma 10.14 to compute the appropriate maximum spanning forest of the degree 3 graph in O(m′ ) = O(n) time, and apply Lemma 10.15 to create a tree decomposition in O(m′ n′ ) = O(n2 ) time of size V ′  = O(m′ + n′ ) = O(n). Then we invoke Lemma 10.13 again in O(m′ V ′ ) = O(n2 ) time to get the tree decomposition of the original graph. Note that the tree decomposition of the original graph is still on the vertex set V ′ with V ′  = O(n). Using Theorem 10.12 then ﬁnds the maximum independent set. Many hard combinatorial optimization problems on graphs have algorithms whose running time is polynomial in the size of the graph and exponential in the treewidth of the graph. We give some examples in the exercises below.
Exercises 10.1 In the Euclidean Steiner tree problem, we are given as input a set T of points in the plane called terminals. We are allowed to choose any other set N of points in the plane; we call these points nonterminals. For any pair of points i, j ∈ T ∪ N , the cost of an edge connecting i and j is the Euclidean distance between the points. The goal is to ﬁnd a set N of nonterminals such that the cost of the minimum spanning tree on T ∪ N is minimized. Show that the polynomialtime approximation scheme for the Euclidean TSP can be adapted to give a polynomialtime approximation scheme for the Euclidean Steiner tree problem. 10.2 In this problem, we consider a Euclidean variant of the kmedian problem from Section 9.2. We are given a set N of points in the plane that are clients and potential locations for facilities. ∑ We must ﬁnd a subset S ⊆ N with S ≤ k of facilities to open so as to minimize j∈N mini∈S cij , where cij is the Euclidean distance between i, j ∈ N . Using the techniques of Section 10.1, give a polynomialtime approximation scheme for the Euclidean kmedian problem. (Extended hint: Several aspects of the proof must be adapted. For the Euclidean TSP, the side length L of the smallest square containing all points of the instance is a lower bound on the cost of an optimal solution, and then can be used in modifying the instance to be a nice instance. For the Euclidean kmedian problem, L isn’t necessarily a lower bound on the cost of an optimal solution. What can be used instead? In the dynamic program, for each square of side length s, it will be useful to keep track of the number of facilities opened in the square. Also for each portal of the square we may wish to keep track of two estimates: ﬁrst, the distance from the portal to the nearest open facility inside the square (if any) in increments of s/m (where m is the number of portals on a side); and second, the distance of the portal to the nearest open facility outside the square, also in increments of s/m. Note that for two adjacent portals, both estimates should diﬀer by at most one increment of s/m.) 10.3 Recall the vertex cover problem deﬁned in Section 1.2. In the problem we are given an undirected graph G = (V, E) and a nonnegative weight wi for each vertex i ∈ V . The goal is to ﬁnd a minimumweight subset of vertices C ⊆ V such that for each edge (i, j) ∈ E, either i ∈ C or j ∈ C. Show that the polynomialtime approximation scheme for the maximum independent set problem in planar graphs can be adapted to give a polynomialtime approximation scheme for the vertex cover problem in planar graphs. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
278
Further uses of rounding data and dynamic programming
10.4 Recall from Section 5.12 that we say a graph G = (V, E) is kcolorable if we can assign each vertex one of k colors such that for any edge (i, j) ∈ E, i and j are assigned diﬀerent colors. Suppose that G has a tree decomposition T of treewidth t, for constant t. Prove that for any k, one can decide in polynomial time whether the graph is kcolorable. 10.5 In the graphical traveling salesman problem, we are given a graph G = (V, E) with edge costs ce ≥ 0 for all e ∈ E. The goal is to ﬁnd a minimumcost multiset of edges F such that (V, F ) is Eulerian. One can view this as a variant of the traveling salesman problem in which cities can be visited multiple times, but it is not possible to travel directly from city i to city j when (i, j) ∈ / E. In this problem, we will show that we can compute the optimal solution to the graphical TSP when the graph G has low branchwidth, a concept related to treewidth. Given an undirected graph G = (V, E), a branch decomposition of G is a spanning tree T on a new set of nodes V ′ such that the degree of each internal node of T is exactly three, and T has exactly E leaves; each edge e of G maps to a unique leaf of T . Removing an edge of the tree T therefore partitions the edges of G into two parts A and B corresponding to the leaves in the two connected components of T after the edge has been removed. The width of the separation is the number of vertices of G that have edges from both A and B incident on them. The width of the branch decomposition is the maximum width of the separation over all edges in the tree T . The branchwidth of G is the smallest width of a branch decomposition over all branch decompositions of G. Suppose we are given an instance of the graphical traveling salesman problem, and a branch decomposition T of the input graph G = (V, E) of branchwidth t. We can root the tree T by removing an arbitrary edge (a, b) and replacing it with two edges (a, r) and (r, b). Show that by using dynamic programming, there is a polynomialtime algorithm for the graphical TSP when the branchwidth t is a constant.
Chapter Notes Approximation schemes for Euclidean instances of the traveling salesman problem and some other geometric problems were discovered independently by Arora [11] and Mitchell [225]. Arora’s scheme also generalizes to Euclidean problems in higher dimensions, as long as the dimension d is o(log log n). The algorithm and presentation given in Section 10.1 is due to Arora [11], following somewhat a survey of Arora [12]. As mentioned in the introduction, this technique turns out to be widely applicable to optimization problems in the Euclidean plane, including the Steiner tree problem (discussed in both [11] and [225]), given in Exercise 10.1, and the kmedian problem (due to Arora, Raghavan, and Rao [20]), given in Exercise 10.2. The approximation scheme of Section 10.2 for the maximum independent set in planar graphs is due to Baker [30], although we give a diﬀerent presentation here. The result for vertex cover in Exercise 10.3 is also from Baker [30]. The concept of the treewidth of a graph was introduced by Robertson and Seymour [252]; other equivalent concepts were introduced independently in the literature. For one such concept, partial ktrees, Arnborg and Proskurowski [10] give a “linear time” dynamic program for ﬁnding the maximum independent set. Bodlaender [52] shows that kouterplanar graphs have treewidth at most 3k − 1; we adapt (and weaken slightly) his presentation. The concept of branchwidth in Exercise 10.5 was introduced by Robertson and Seymour [253]; they showed that a graph of branchwidth t has treewidth at most 3t/2, and a graph with treewidth k has branchwidth at most k + 1. The graphical Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
10.2
The maximum independent set problem in planar graphs
279
traveling salesman problem was introduced by Cornu´ejols, Fonlupt, and Naddef [84], and the dynamic programming algorithm of Exercise 10.5 is due to Cook and Seymour [80].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
280
Further uses of rounding data and dynamic programming
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
C h a p t e r 11
Further uses of deterministic rounding of linear programs
In this chapter, we return to the technique of deterministically rounding linear programs. We begin with a generalization of the problem of scheduling jobs on identical machines which we introduced in Section 2.3. In this general case, not only does a job j take diﬀerent amounts of processing time depending on which machine i it is assigned to, but also a job j incurs a cost cij when assigned to machine i. If there is any feasible schedule of cost C in which the jobs are completed by time T , we show that by rounding a linear programming relaxation we can in polynomial time produce a schedule of cost at most C in which all jobs ﬁnish by time 2T . For the remaining applications, we use the fact that there is always a basic optimal solution to a linear program, and that algorithms for solving linear programs return such a solution. Suppose we have a linear program with n variables xj in which there is a constraint xj ≥ 0 for each variable xj . Then the number of constraints m ≥ n, since there is at least one constraint per variable. Let the ai be a vector and bi be a scalar giving the ith constraint, aTi x ≥ bi . A feasible solution x ¯ is a basic feasible solution if there are n linearly independent vectors ai such that the corresponding constraints are met with equality; that is, aTi x ¯ = bi . Observe that this linear system uniquely deﬁnes x ¯. Basic feasible solutions are equivalent to extreme points of the geometric region of feasible solutions deﬁned by the linear program; recall that an extreme point is a feasible solution that cannot be expressed as the convex combination of two other feasible solutions (see, for example, Exercise 1.5). There is always an optimal solution to the linear program that is a basic feasible solution, and we call such a solution a basic optimal solution. The structure of basic solutions is very useful in designing approximation algorithms. In Section 11.2, we consider a version of the minimumdegree spanning tree problem introduced in Section 2.6: here we have costs on the edges and bounds bv on the degree of each vertex v ∈ V . We show that if there exists a spanning tree of cost at most C such that the degree of each vertex is at most bv , we can in polynomial time ﬁnd a spanning tree of cost at most C in which each vertex v has degree at most bv + 1. The algorithm uses the properties of basic optimal solutions to generate a sequence of linear programming relaxations of the problem, where each LP in the sequence is deﬁned on a smaller graph and has fewer constraints than the previous LPs. The ﬁnal LP in the sequence returns a spanning tree with the desired properties. In Section 11.3, we consider a generalization of the generalized Steiner tree problem intro281
282
Further uses of deterministic rounding of linear programs
duced in Section 7.4 in which we must ﬁnd a minimumcost subgraph with several edgedisjoint paths between certain pairs of vertices. We show a remarkable theorem that any basic feasible solution to the linear programming relaxation of this problem must have some variable of value at least 1/2. We round this variable up to 1, then iterate on the remaining problem; this technique is called iterated rounding and it gives a 2approximation algorithm for the problem. This theorem can be viewed as a weakening of a statement we proved about a linear programming relaxation for the vertex cover problem in Exercise 1.5, in which all variables in a basic feasible solution have value either 0, 1, or 1/2. Here we only show that some variable will have value at least 1/2, but this is suﬃcient to give a good approximation algorithm.
11.1
The generalized assignment problem
In the generalized assignment problem, we are given a collection of n jobs to be assigned to m machines. Each job j = 1, . . . , n is to be assigned to exactly one machine; if it is assigned to machine i, then it requires pij time units of processing, and incurs a cost of cij . Furthermore, we are given a time bound T that limits the total processing of each machine i = 1, . . . , m. The aim is to ﬁnd a feasible assignment of minimum total cost. If we write this problem as an integer program, and introduce a 01 variable xij , i = 1, . . . , m, j = 1, . . . , n, to indicate whether job j is assigned to machine i, then we obtain minimize
m ∑ n ∑
cij xij
(11.1)
i=1 j=1
subject to
m ∑
xij = 1,
j = 1, . . . , n,
(11.2)
pij xij ≤ T,
i = 1, . . . , m,
(11.3)
i = 1, . . . , m, j = 1, . . . , n.
(11.4)
i=1 n ∑ j=1
xij ∈ {0, 1},
It is not hard to see that just deciding whether there is a feasible solution to the integer program is strongly NPhard; for instance, checking whether there is a feasible solution to the integer program captures the problem of minimizing the makespan on identical parallel machines (the case in which pij = pj , i = 1, . . . , m for each job j = 1, . . . , n). Nonetheless, we shall see that a rather strong approximation result is possible: for a given input, we either prove that no feasible solution exists, or else output a solution of total cost no greater than the optimum, but violate feasibility by allowing each machine to process jobs for a total time no greater than 2T . We shall give an LP rounding algorithm to ﬁnd the nearoptimal schedule. However, the linear programming relaxation of the integer program (11.1)–(11.4) in which the constraints (11.4) are replaced by xij ≥ 0, i = 1, . . . , m, j = 1, . . . , n, (11.5) does not provide a suﬃciently strong bound on which to base the algorithm. Instead, we will strengthen this linear program in a seemingly trivial way; we add the constraints that xij = 0 if pij > T,
(11.6)
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.1
The generalized assignment problem
283
which clearly hold for any feasible integer solution. In carrying out the rounding of the fractional solution x, we shall rely heavily on a beautiful wellknown result from matching theory (or more generally, from the theory of network ﬂows). Suppose we are given a bipartite graph with two disjoint sets of nodes V and W , and an edge set F where each edge has one endpoint in each of V and W ; let B = (V, W, F ) denote such a graph. We say that M ⊆ F is a complete matching for V in this graph if (a) for each node v ∈ V there is exactly one edge of M incident to v, and (b) for each node w ∈ W , there is at most one edge of M incident to w. (We shall assume that V  ≤ W , since otherwise, no such matching is possible.) Deciding whether a given bipartite graph B = (V, W, F ) has a complete matching is equivalent to deciding whether the following integer program has a feasible solution, where the 01 variable yvw indicates that edge (v, w) is in the matching: ∑ yvw ≤ 1, ∀w ∈ W, (11.7) v:(v,w)∈F
∑
yvw = 1,
∀v ∈ V,
(11.8)
yvw ∈ {0, 1},
∀(v, w) ∈ F.
(11.9)
w:(v,w)∈F
If we relax the binary constraints (11.9) to be nonnegativity constraints yvw ≥ 0,
∀(v, w) ∈ F,
(11.10)
then we get a linear program; a feasible solution is called a fractional complete matching. In Exercise 4.6 we showed that this linear program has the special property that each extreme point is integer. The exercise shows the following theorem. Theorem 11.1 (Exercise 4.6): For any bipartite graph B = (V, W, F ), each extreme point of the feasible region of (11.7), (11.8), and (11.10) has integer coordinates. Furthermore, given edge costs cvw , (v, w) ∈ F , and a feasible fractional solution yvw , (v, w) ∈ F , we can ﬁnd, in polynomial time, a feasible integer solution yˆvw such that ∑ ∑ cvw yˆvw ≤ cvw yvw . (v,w)∈F
(v,w)∈F
We next show how this result can be applied to obtain the main result of this section, which is as follows. Theorem 11.2: If the linear program (11.1)–(11.3), (11.5), and (11.6) has a feasible (fractional) solution x of total cost C, then we can round it to an (integer) assignment of total cost at most C in which each machine is assigned total processing at most 2T . solution Proof. We will prove this theorem by providing an algorithm that converts a feasible LP∑ x to the required (integer) assignment. This fractional solution x assigns, in total, nj=1 xij jobs to machine ∑i; the integer assignment that we construct will be similar to this, and so we allocate ki = ⌈ nj=1 xij ⌉ “slots” for machine i to be assigned jobs. Furthermore, the algorithm will assign job j to machine i only if xij > 0. We can model this restriction by constructing a bipartite graph B = (J, S, E), where one side of the bipartite graph consists of job nodes J = {1, . . . , n}, and the other side consists of machine slots S = {(i, s) : i = 1, . . . , m, s = 1, . . . , ki }. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
284
Further uses of deterministic rounding of linear programs
Slots Jobs y1,(i,1) = 3/4 xi1 = 3/4
1
(i, 1) y2,(i,1) = 1/4
xi2 = 1/2
2
xi3 = 1/2
3
y2,(i,2) = 1/4 (i, 2) y3,(i,2) = 1/2 y4,(i,2) = 1/4
xi4 = 1/3
4
(i, 3)
y4,(i,3) = 1/12
Figure 11.1: An example of how the fractional matching for B is created given xij . (Note that S ≥ J.) One natural idea is to simply include an edge ((i, s), j) in E whenever xij > 0; although this is a good start, we will need to reﬁne this later. We shall focus on assignments of the jobs that correspond to complete matchings in this graph B. Since an edge e that connects machine slot node (i, s) to job node j has the interpretation that job j is assigned to machine i, it is natural to set the cost of edge e equal to cij . Ideally, the graph B would have the following two properties: 1. B contains a fractional complete matching for J of cost C; 2. any (integer) complete matching for J in B corresponds to an assignment in which each machine is required to complete at most 2T time units of processing. If we succeed in constructing such a bipartite graph B, then we can compute the desired assignment by ﬁnding the minimumcost (integer) complete matching in B (for example, by the polynomialtime algorithm of Theorem 11.1). This matching must have cost at most C (by property (1) and Theorem 11.1), and assign each machine total processing at most 2T (by property (2)). Let us understand in detail why our graph B has the ﬁrst property. Focus on any machine i. We can convert our LP solution x into a fractional complete matching y in B. Consider the slot nodes (i, s), s = 1, . . . , ki as bins of capacity 1, and the values xij , j = 1, . . . , n as pieces of the n jobs to be packed in these bins. We can place the pieces in the bin corresponding to slot (i, 1) until the next piece j would cause the bin to be packed with pieces of total size greater than 1. Suppose this piece j is of size xij and there is only capacity z remaining in the bin (where z < xij ). Then we pack z of this job piece j (of size xij ) into slot (i, 1), and pack the remaining xij − z in the next bin (or equivalently, slot (i, 2).) Whenever we pack a positive fraction of job j in slot (i, s), we set yj,(i,s) equal to that fraction; all other components of y are set to 0. It is easy to see that ∑ repeating this for each machine i yields a fractional complete matching in B of total cost i,j cij xij . See Figure 11.1 for an illustration. However, as stated thus far, we do not have the second property. We will need to reﬁne the construction of B = (J, S, E). First, suppose that we include an edge (j, (i, s)) ∈ E only Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.1
The generalized assignment problem
285
if the corresponding value yj,(i,s) > 0. Clearly, this does not aﬀect the existence of a fractional complete matching y. Furthermore, let us reﬁne our “binpacking” procedure that constructs the solution y. Suppose that in packing the n pieces xij , j = 1, . . . , n, into the ki slots (or bins), we ﬁrst sort the jobs so that their corresponding processing requirements pij are in nonincreasing order. For ease of notation, suppose without loss of generality that pi1 ≥ pi2 ≥ · · · ≥ pin .
(11.11)
We shall consider the graph B formed by including an edge (j, (i, s)) in E exactly when this binpacking procedure places a positive fraction from xij into slot (i, s) (and hence makes the corresponding component of y positive). Consider the slot (i, s), and focus on those jobs j for which some positive fraction is packed in the bin corresponding to this slot; let max(i, s) denote the maximum processing requirement pij among these jobs. The total load assigned to machine i by any complete matching in B is at most ki ∑ max(i, s). s=1
Since (11.6) holds, we know that max(i, 1) ≤ T . We shall show that ki ∑
max(i, s) ≤ T,
s=2
and hence the total load on machine i for any ∑ complete matching in B is at most 2T . Observe that for each s = 1, . . . , ki −1, we have that j yj,(i,s) = 1, since we only ∑ start the next bin when we have completely ﬁlled the previous one. Thus, we can think of j yj,(i,s) pij as a weighted average of the relevant pij values; that is, those jobs for which a positive fraction is assigned to slot (i, s). By ∑our ordering assumption (11.11), for each s = 1, . . . , ki − 1, we have that max(i, s + 1) ≤ j yj,(i,s) pij , and hence, k∑ i −1
max(i, s + 1) ≤
k∑ i −1 ∑
s=1
s=1
yj,(i,s) pij ≤
ki ∑ ∑ s=1
j
yj,(i,s) pij .
j
∑ However, since xij = ∑s yj,(i,s) , by interchanging the order of the summations in the ﬁnal ∑ expression, we see that s yj,(i,s) pij = j pij xij , and hence, i −1 ∑ k∑
j
max(i, s + 1) ≤
s=1
∑
pij xij .
j
Since x is a feasible solution for our original LP, it satisﬁes (11.3), and hence, k∑ i −1
max(i, s + 1) ≤ T,
s=1
which completes the proof of the theorem.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
286
Further uses of deterministic rounding of linear programs
11.2
Minimumcost boundeddegree spanning trees
In this section, we consider the weighted case of a problem we considered in Sections 2.6 and 9.3. In those sections, we considered the problem of ﬁnding a spanning tree of a graph such that we minimized the maximum degree of the tree. Here, we consider the problem of ﬁnding a spanning tree of minimum cost such that the degree of node v is no more than some speciﬁed bound. We refer to this problem as the minimumcost boundeddegree spanning tree problem. More formally, we are given as input an undirected graph G = (V, E), costs ce ≥ 0 for all e ∈ E, a set W ⊆ V , and integer bounds bv ≥ 1 for all v ∈ W . Let OPT be the cost of the minimum spanning tree such that the degree of each node v ∈ W is no more than bv (if such a tree exists). In the ﬁrst part of this section, we give an algorithm that ﬁnds a spanning tree of cost at most OPT such that each node v ∈ W has degree at most bv + 2. In the latter part of this section, we show that we can ﬁnd a spanning tree of cost at most OPT such that each node v ∈ W has degree at most bv + 1. As we argued at the end of Section 2.6 for unweighted spanning trees, no better result is possible unless P = NP. We begin by giving an integer programming formulation of the problem for a graph G = (V, E). Given a vertex set S ⊆ V , let E(S) be the subset of edges in E that have both endpoints in S, and let δ(S) be the subset of edges that have exactly one endpoint in S. We will denote δ({v}) by δ(v). We let xe ∈ {0, 1} indicate whether an edge e is in the spanning tree or not. Every spanning tree has exactly V  − 1 edges, so we have ∑ xe = V  − 1. e∈E
Furthermore, since for any set S ⊆ V with S ≥ 2, a spanning tree does not have a cycle in E(S), we have ∑ xe ≤ S − 1. e∈E(S)
Finally, we want the spanning tree to respect the degree bounds for all v ∈ W , so that we have ∑ x e ≤ bv . e∈δ(v)
Relaxing the integrality constraints to xe ≥ 0 gives us the following linear programming relaxation of the problem: ∑ minimize ce xe (11.12) e∈E
subject to
∑
xe = V  − 1,
(11.13)
e∈E
∑
xe ≤ S − 1,
∀S ⊆ V, S ≥ 2,
(11.14)
xe ≤ bv ,
∀v ∈ W,
(11.15)
xe ≥ 0,
∀e ∈ E.
(11.16)
e∈E(S)
∑
e∈δ(v)
We can use the ellipsoid method introduced in Section 4.3 to solve the linear program by giving a polynomialtime separation oracle. It is easy to check that the constraints (11.13), Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.2
Minimumcost boundeddegree spanning trees
287
(11.15), and (11.16) are satisﬁed. To check that the constraints (11.14) are satisﬁed, we need to set up a sequence of maximum ﬂow problems. We ﬁrst explain the general maxﬂow construction, and then we modify it to check the constraints (11.14). We create a new graph G′ where we add source and sink vertices s and t, edges (s, v) from the source to every vertex v ∈ V , and edges (v, t) from every vertex v ∈ V . The capacity of every edge e from G is 12 xe , the capacity ∑ of every edge (s, v) is 21 e∈δ(v) xe , and the capacity of every edge (v, t) is 1. Then consider the capacity of any st cut S ∪ {s} for S ⊆ V . It contains edges (v, t) for all v ∈ S, all e ∈ δ(S) ∩ E and (s, v) for all v ∈ / S. So its total capacity is ∑ ∑ 1 ∑ 1∑ ∑ S + xe + xe = S + xe + xe , 2 2 e∈δ(S)
v ∈S / e∈δ(v)
e∈δ(S)
e∈E(V −S)
since adding up 21 xe for each e ∈ δ(v) where v ∈ / S gives xe for all edges e with both endpoints 1 not in S plus x for all edges e with exactly one endpoint not in S. Using the fact that e 2 ∑ x = V  − 1, this is equal to e∈E e ∑ ∑ S + V  − 1 − xe = V  + (S − 1) − xe . e∈E(S)
∑
e∈E(S)
Thus the capacity of this cut is at least V  if and only if e∈E(S) xe ≤ S − 1. We want to make sure S ≥ 2, so for each pair x, y ∈ V , we construct the maxﬂow instance as above, but alter the capacity of the edges (s, x) and (s, y) to be inﬁnite. This ensures that any minimum st cut S ∪ {s} has x, y ∈ S. Then by the reasoning above, the value of the maximum ﬂow for this instance will be at least V  if and only if constraints (11.14) are satisﬁed for all S ⊇ {x, y}. If the ﬂow is at least V  for all pairs x, y ∈ V , then all constraints (11.14) are satisﬁed. If the ﬂow value is less than V , then the corresponding minimum st cut S ∪ {s} will give a violated constraint. We assume from here on that there is a feasible solution to this linear program. If there is no feasible solution, then obviously there is no tree in the graph G that has the given degree bounds. As a warmup to the algorithm and its approach, we ﬁrst show that if we have no degree bounds (that is, W = ∅), then we can ﬁnd a spanning tree of cost no more than the value of the linear program (11.12); this gives us a minimum spanning tree. Given an LP solution x, deﬁne E(x) to be the support of x; that is, E(x) = {e ∈ E : xe > 0}. The algorithm for ﬁnding a minimum spanning tree depends on the following lemma. We defer the proof of the lemma for the moment. Lemma 11.3: For any basic feasible solution x to the linear program (11.12) with W = ∅, there is some v ∈ V such that there is at most one edge of E(x) incident on v. Our algorithm then works as follows. We maintain a set of edges F for our solution, which is initially empty. While there is more than one vertex in the current graph, we solve the linear program (11.12) for the current graph G = (V, E), and obtain a basic optimal solution x. We remove from the edge set E all edges e for which xe = 0. By Lemma 11.3, there exists some vertex v ∈ V such that there is at most one edge of E(x) incident on it; suppose the edge is (u, v). Then we add (u, v) to our solution set F , remove v and (u, v) from the graph, and repeat. Intuitively, each iteration ﬁnds a leaf v of the spanning tree, then recursively ﬁnds the rest of it. The algorithm is summarized in Algorithm 11.1. Theorem 11.4: Algorithm 11.1 yields a spanning tree of cost no more than the value of the linear program (11.12). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
288
Further uses of deterministic rounding of linear programs
F ←∅ while V  > 1 do Solve LP (11.12) on (V, E), get basic optimal solution x E ← E(x) Find v ∈ V such that there is one edge (u, v) in E(x) incident on v F ← F ∪ {(u, v)} V ← V − {v} E ← E − {(u, v)} return F Algorithm 11.1: Deterministic rounding algorithm for ﬁnding a minimumcost spanning tree. Proof. We begin by showing that if the edge e∗ is chosen to add to F in some ∑ iteration of the ∗ algorithm, then xe = 1 in that iteration. To prove this, we ﬁrst claim that e∈δ(w) xe ≥ 1 for any vertex w ∈ V . Then for the edge e∗ = (u, v) chosen by the algorithm, we know that there is ∑ one edge in E(x) incident on v, and therefore xe∗ ≥ 1. To see that e∈δ(v) xe ≥ 1, note that for ∑ S = V −{v}, ∑ = V − ∑ xe ≤ S−1∑= (V −1)−1 ∑ the LP constraint (11.14) enforces that e∈E(S) x − x = x = V  − 1 by constraint (11.13) and also 2. But e∈E e∈E(S) xe . e∈E e e∈δ(v) e ∑ e Therefore e∈δ(v) xe ≥ (V  − 1) − (V  − 2) = 1. Now consider the constraint (11.14) for S = {u, v}. This enforces xe∗ ≤ 1, so it must be the case that xe∗ = 1. We now prove by induction on the size of the graph that the algorithm produces a spanning tree of cost no more than the value of the linear program. In the base case, we have a graph with two vertices, and the algorithm returns a single edge e. By the above, we have xe = 1, so that the value of the LP is at least ce xe = ce , and the statement holds. Now suppose that the statement holds for every graph with at most k > 2 vertices, and our graph has k + 1 vertices. We solve the LP, get a solution x, and ﬁnd a vertex v such that there is only one edge e∗ = (u, v) in E(x) incident on v. Let V ′ = V − {v}, and E ′ = E(x) − {e∗ }. By the inductive hypothesis, we will ﬁnd a spanning tree F ′ on (V ′ , E ′ ) of cost at most the value of the LP on (V ′ , E ′ ); let x′ be an optimal solution to this LP. Obviously F ′ ∪ {e∗ } is a spanning ∑ tree of (V, E), so that the algorithm returns a spanning tree. To show that its cost is at most e∈E ce xe , we shall show that xe for e ∈ E ′ is a feasible solution to the LP on (V ′ , E ′ ). It then follows that the cost of the spanning tree returned is ∑ e∈F ′
ce + ce∗ ≤
∑ e∈E ′
ce x′e + ce∗ xe∗ ≤
∑ e∈E ′
ce xe + ce∗ xe∗ =
∑
c e xe ,
e∈E
as desired. We now must show that xe for e ∈ E ′ is a feasible solution to the LP on (V ′ , E ′ ). Since the LP constraints (11.14) for (V ′ , E ′ ) are a subset of the constraints for ∑ (V, E), x is feasible for them. Thus we only need to show that the ﬁrst constraint holds, namely e∈E ′ xe = V ′ −1 = V −2. ′ ∗ This follows since xe∗ = 1, and edges such that ∑ the only ∑edges in E − E are e , which has ′ ∗ xe = 0. Thus e∈E ′ xe = e∈E xe − xe = (V  − 1) − 1 = V  − 1, and xe for e ∈ E ′ is feasible for the LP on (V ′ , E ′ ). Now we return to the minimumcost boundeddegree spanning tree problem. Recall that we have a subset of vertices W such that we wish to ﬁnd a tree with the degree of v at most bv for all v ∈ W . We cannot hope for a result as strong as that of Lemma 11.3 for this problem, since we would be able to translate this into an algorithm to ﬁnd an optimal tree. Instead, we will Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.2
Minimumcost boundeddegree spanning trees
289
F ←∅ while V  > 1 do Solve LP (11.12) on (V, E) and W , get basic optimal solution x E ← E(x) if ∃ a v ∈ V such that there is one edge (u, v) in E(x) incident on v then F ← F ∪ {(u, v)} V ← V − {v} E ← E − {(u, v)} if u ∈ W then bu ← bu − 1 else ∃ a v ∈ W such that there are at most three edges in E(x) incident on v W ← W − {v} return F Algorithm 11.2: Deterministic rounding algorithm for ﬁnding a minimumcost boundeddegree spanning tree. be able to show the following. This will lead to a result in which degree bounds are exceeded by at most two. Lemma 11.5: For any basic feasible solution x to the linear program (11.12), either there is some v ∈ V such that there is at most one edge of E(x) incident on v, or there is some v ∈ W such that there are at most three edges of E(x) incident on v. Note that Lemma 11.3 is a special case of this lemma; the second possibility cannot occur when W = ∅. We again defer the proof of this lemma, and instead show how it leads to the desired algorithm. As before, we maintain a solution set F , which is initially empty. We solve the linear program for the current graph (V, E) and current bound set W , and obtain a basic optimal solution x. We remove all edges e with xe = 0 from the edge set. If, as before, there is some v ∈ V such that there is at most one edge (u, v) in E(x) incident on v, we add (u, v) to F , remove v from V , and remove (u, v) from E. If u ∈ W , we also decrease bu by one. We then iterate. If instead, there is some v ∈ W such that there are at most three edges of E(x) incident on v, we remove v from W and repeat. The algorithm is summarized in Algorithm 11.2. We can now prove the following. Theorem 11.6: Algorithm 11.2 produces a spanning tree F such that the degree of v in F is at most bv + 2 for v ∈ W , and such that the cost of F is at most the value of the linear program (11.12). Proof. In each step of the algorithm, we either add a spanning tree edge to F , or we remove a vertex from W . Thus the algorithm will terminate in at most (n − 1) + n = 2n − 1 iterations. Observe that the proof that the algorithm returns a spanning tree whose cost is at most the value of the linear program is almost identical to that of Theorem 11.4. In that proof we considered the graph (V ′ , E ′ ) resulting from adding an edge (u, v) to our solution F , then removing v and (u, v) from the graph. We showed that the LP solution x for the graph (V, E) is feasible for the new graph (V ′ , E ′ ) when restricted to the edges in E ′ . Here we also need to consider the set of degree bounds W , and show that x is feasible for the new graph (V ′ , E ′ ) and the new degree bounds ∑ after bu has been decreased by one. If x was feasible for the constraints (11.15) before, then e∈δ(u) xe ≤ bu . Thus after edge e = (u, v) with xe = 1 is removed from Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
290
Further uses of deterministic rounding of linear programs
Figure 11.2: Top: two intersecting sets. Bottom: A laminar collection of sets. the graph, the solution x restricted to the remaining edges in E ′ will be feasible for the same constraint on the new graph (V ′ , E ′ ) with the degree bound bu − 1. Now consider any vertex v initially in W . We show by induction on the algorithm that the degree of v in the solution F is at most bv + 2. In each iteration, one of three things can happen. First, we can choose an edge incident on v and decrease bv by 1, in which case the statement follows by induction. Second, we can choose an edge incident on v and remove v from the graph; in this case, we must have had bv ≥ 1 in order to have a feasible solution to the LP, so the statement holds. Third, we can remove v from W . In this case, we must have bv ≥ 1 in order for there to be any edges in E(x) incident on v; yet there are at most 3 edges in E(x) incident on v. Thus, in all future iterations we can add at most three edges incident on v, since all edges not in E(x) are removed from the graph for all future iterations. Thus, v will have degree at most bv + 2. We can now turn to the proof of Lemma 11.5. In order to do this, we will need to introduce some deﬁnitions and notation. From here on, for the given solution x, we assume that all edges e such that xe = 0 have been removed from the edge set; that is, E = E(x). ∑ Deﬁnition 11.7: For x ∈ ℜE and a subset of edges F , we deﬁne x(F ) = e∈F xe . Deﬁnition 11.8: For a solution x to LP (11.12), we say that a constraint (11.14) corresponding a set S ⊆ V , S ≥ 2, is tight if x(E(S)) = S − 1. A constraint (11.15) corresponding to a vertex v ∈ W is tight if x(δ(v)) = bv . We may also say that the set S ⊆ V is tight if x(E(S)) = S − 1 or that the vertex v is tight if x(δ(v)) = bv . Deﬁnition 11.9: We say two sets A and B are intersecting if A ∩ B, A − B, and B − A are all nonempty. Deﬁnition 11.10: We say a collection of sets S is laminar if no pair of sets A, B ∈ S are intersecting. See Figure 11.2 for an example of intersecting sets and laminar set collections. Deﬁnition 11.11: For a subset of edges F ⊆ E, the characteristic vector of F is χF ∈ {0, 1}E , where χF (e) = 1 if e ∈ F and 0 otherwise. We are now able to state the following theorem, which we will need to prove Lemma 11.5. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.2
Minimumcost boundeddegree spanning trees
291
Theorem 11.12: For any basic feasible solution x to the linear program (11.12), there is a set Z ⊆ W and a collection L of subsets of vertices with the following properties: 1. For all S ∈ L, S ≥ 2 and S is tight, and for all v ∈ Z, v is tight. 2. The vectors χE(S) for S ∈ L and χδ(v) for v ∈ Z are linearly independent. 3. L + Z = E. 4. The collection L is laminar. We will defer the proof of this theorem for a moment, and will next show how Lemma 11.5 can be derived from the theorem. First, however, we observe that for any basic feasible solution x to the LP, there is a collection of sets S and set Y ⊆ W such that the ﬁrst three properties hold for S and Y ; the key statement of the theorem is the last property which states that for any basic feasible solution, there is a laminar collection of such sets. A basic solution is formed by taking E linearly independent constraints from the linear program, setting them at equality, and solving the resulting linear system. This is precisely what the ﬁrst two properties state. The third property states that the number of constraints set to equality is equal to the number of nonzero variables (recall that we have assumed that E = E(x)). We ﬁrst need the following short lemma. Lemma 11.13: Let L be a laminar collection of subsets of V , where each S ∈ L has S ≥ 2. Then L ≤ V  − 1. Proof. We can prove this by induction on the size of V . For the base case V  = 2, obviously L can contain only one set of cardinality 2. For V  > 2, pick a minimum cardinality set R in L. Let V ′ be V with all but one vertex of R removed, and let L′ be the sets in L restricted to the elements of V ′ , with the set R removed. Note that L′ fulﬁlls the conditions of the lemma; it is still laminar, and any set must contain at least two elements. So L′  ≤ V ′  − 1 by induction, and since L′  = L − 1 and V ′  ≤ V  − 1, the lemma statement follows. Now we recall the statement of Lemma 11.5, and give its proof. Lemma 11.5: For any basic feasible solution x to the linear program (11.12), either there is some v ∈ V such that there is at most one edge of E(x) incident on v, or there is some v ∈ W such that there are at most three edges of E(x) incident on v. Proof. We prove the statement by contradiction. If the statement is not true, then for every vertex v ∈ V , there are at least two edges of E(x) incident on it, and for every v ∈ W , there are at least four edges of E(x) incident on it. Then it must be the case that E(x) ≥
1 (2(V  − W ) + 4W ) = V  + W . 2
However, by Theorem 11.12, we know that E(x) = L+Z ≤ L+W  for laminar L. Since each set S ∈ L has cardinality at least two, by Lemma 11.13, we get that E(x) ≤ V −1+W , which is a contradiction. Finally, we can turn to the proof of Theorem 11.12. The basic idea of the proof is simple, and has proven enormously useful in a number of diﬀerent contexts. We start out with a collection of sets S that may not be laminar, and as long as we have two intersecting sets S, T ∈ S, we show that we can “uncross” them and replace them with two nonintersecting sets. A ﬁrst step in this proof is to show the following. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
292
Further uses of deterministic rounding of linear programs
Lemma 11.14: If S and T are tight sets such that S and T are intersecting, then S ∪ T and S ∩ T are also tight sets. Furthermore, χE(S) + χE(T ) = χE(S∪T ) + χE(S∩T ) . Proof. We begin by showing that x(E(S)) is supermodular; namely, x(E(S)) + x(E(T )) ≤ x(E(S ∩ T )) + x(E(S ∪ T )). This follows by a simple counting argument: any edge in E(S) or in E(T ) is in E(S ∪ T ), while any edge in both E(S) and E(T ) appear both in E(S ∩ T ) and E(S ∪ T ). The righthand side may be greater than the lefthand side since edges with one endpoint in S − T and the other in T − S appear in E(S ∪ T ) but not in E(S) or E(T ). Since S and T are intersecting, S ∩ T ̸= ∅. Thus by the feasibility of x (S − 1) + (T  − 1) = (S ∩ T  − 1) + (S ∪ T  − 1) ≥ x(E(S ∩ T )) + x(E(S ∪ T )). By supermodularity, x(E(S ∩ T )) + x(E(S ∪ T )) ≥ x(E(S)) + x(E(T )). Finally, since S and T are tight sets, x(E(S)) + x(E(T )) = (S − 1) + (T  − 1). Thus all of these inequalities must be met with equality, and S ∩ T and S ∪ T are tight sets. Furthermore, as x(E(S ∩ T )) + x(E(S ∪ T )) = x(E(S)) + x(E(T )), it must be the case that χE(S) + χE(T ) = χE(S∪T ) + χE(S∩T ) , since we have assumed that the only edges in E have xe > 0. Let T be the collection of all tight sets for the LP solution x; that is, T = {S ⊆ V : x(E(S)) = S − 1, S ≥ 2}. Let span(T ) be the span of the set of vectors {χE(S) : S ∈ T }. We now show that we can ﬁnd a laminar collection of sets with span at least that of T . Lemma 11.15: There exists a laminar collection of sets L such that span(L) ⊇ span(T ), where each S ∈ L is tight and has S ≥ 2, and the vectors χE(S) for S ∈ L are linearly independent. Proof. Let L be a laminar collection of sets such that each S ∈ L is tight and has S ≥ 2, and the vectors χE(S) for S ∈ L are linearly independent. We assume that L is the maximal such collection; that is, we cannot add any additional sets to L such that all of these properties continue to hold. We will give a proof by contradiction that span(L) ⊇ span(T ); assume otherwise. Then there must be a tight set S with S ≥ 2 such that χE(S) ∈ span(T ) and χE(S) ∈ / span(L); we choose an S such that there is no other such set intersecting fewer sets in L. Note that such an S must be intersecting with at least one set in L, otherwise L is not maximal. Now pick a set T ∈ L such that S and T intersect. By Lemma 11.14, χE(S) + χE(T ) = χE(S∩T ) + χE(S∪T ) and both S ∩ T and S ∪ T are tight. We will argue that it cannot be the case that both χE(S∩T ) ∈ span(L) and χE(S∪T ) ∈ span(L). We know that T ∈ L so that χE(T ) ∈ span(L), and we know that χE(S) = χE(S∩T ) + χE(S∪T ) − χE(T ) . Thus if both χE(S∪T ) and χE(S∩T ) are in span(L), this implies χE(S) ∈ span(L), which contradicts our assumption that χE(S) ∈ / span(L). Hence at least one of χE(S∩T ) and χE(S∪T ) is not in span(L). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.2
Minimumcost boundeddegree spanning trees
S
293
T
Figure 11.3: Proof of Lemma 11.15. The only sets from the laminar collection L, with T ∈ L, that can intersect S ∩ T must also intersect S. Without loss of generality, suppose that χE(S∩T ) ∈ / span(L); note that this implies that χE(S∩T ) ̸= 0, so that S ∩ T  ≥ 2. We claim that S ∩ T must intersect fewer sets in L than S, and with this claim, we contradict the choice of S, since S ∩ T is tight, S ∩ T  ≥ 2, and χE(S∩T ) ∈ / span(L). To prove the claim, we observe that any set in the laminar collection L intersecting S ∩ T must also intersect S; see Figure 11.3. However, S intersects T , while S ∩ T does not intersect T , so S ∩ T must intersect fewer sets in L than S. We can now give the proof of Theorem 11.12, which we restate here. Theorem 11.12: For any basic feasible solution x to the linear program (11.12), there is a set Z ⊆ W and a collection L of subsets of vertices with the following properties: 1. For all S ∈ L, S ≥ 2 and S is tight, and for all v ∈ Z, v is tight. 2. The vectors χE(S) for S ∈ L and χδ(v) for v ∈ Z are linearly independent. 3. L + Z = E. 4. The collection L is laminar. Proof. As we said previously, we know that there exists a collection S and set Y ⊆ W that have the ﬁrst three properties of the theorem. Let span(S, Y ) be the span of the set of vectors {χE(S) : S ∈ S} ∪ {χδ(v) : v ∈ Y }. Since there are E linearly independent vectors in this set and the vectors have E coordinates, clearly span(S, Y ) = ℜE . By Lemma 11.15, if T is the set of all tight sets, then there exists a laminar collection of tight sets L such that span(L) ⊇ span(T ). Since S ⊆ T , then ℜE = span(S, Y ) ⊆ span(T , Y ) ⊆ span(L, Y ) ⊆ ℜE , so that span(S, Y ) = span(L, Y ) = ℜE . From the proof of Lemma 11.15 the vectors χE(S) for all S ∈ L are linearly independent. We now let Z = Y , and as long as there exists some v ∈ Z such that χδ(v) ∈ span(L, Z − v), we remove v from Z. Note that span(L, Z) never decreases, and so the span is always ℜE . When this process terminates, the vectors χE(S) for S ∈ L and χδ(v) for v ∈ Z must be linearly independent, and furthermore since their span is ℜE , we must have E = L + Z. To summarize, we have been able to show that we can ﬁnd a spanning tree of cost at most OPT such that each node v ∈ W has degree at most bv + 2. To do this, we used the properties Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
294
Further uses of deterministic rounding of linear programs
while W ̸= ∅ do Solve LP (11.12) on (V, E), W , get basic optimal solution x E ← E(x) Select v ∈ W such that there are at most bv + 1 edges in E incident on v W ← W − {v} Compute minimumcost spanning tree F on (V, E) return F Algorithm 11.3: Deterministic rounding algorithm for ﬁnding a minimumcost boundeddegree spanning tree.
of a basic feasible solution; we showed that the basic optimal solution has structure in terms of a laminar collection of tight sets. This structure allows us to prove Lemma 11.5, which, informally speaking, states that either we can ﬁnd a leaf of the tree or some node v ∈ W that will have degree at most bv + 2. The lemma leads naturally to an algorithm for ﬁnding the desired tree. We now give an algorithm that ﬁnds a spanning tree of cost at most OPT such that each node v ∈ W has degree at most bv + 1. As with the previous algorithm, we ﬁnd a basic optimal solution of a linear programming relaxation of the problem. The key to the algorithm is proving a stronger version of Lemma 11.5: we show that if W ̸= ∅, we can ﬁnd some v ∈ W such that there are at most bv + 1 edges of E(x) incident on v. Thus in our new algorithm, we will show that we will be able to remove degree bounds from the linear programming relaxation until none are left, in such a way that the cost of the solution to the corresponding relaxation does not increase. When all the degree bounds are removed, we will have a solution to the linear programming relaxation of the minimum spanning tree problem, and we previously saw in Theorem 11.4 that we can ﬁnd a minimum spanning tree of cost at most the value of the linear programming relaxation. As before, we deﬁne E(x) = {e ∈ E : xe > 0}. Our algorithm will depend on the following lemma, whose proof we defer for a moment. Lemma 11.16: For any basic feasible solution x to the linear program (11.12) in which W ̸= ∅, there is some v ∈ W such that there are at most bv + 1 edges incident on v in E(x). Given the lemma, the algorithm is relatively straightforward. We solve the LP relaxation to obtain a basic optimal solution. All edges e ∈ E such that xe = 0 are removed from E. If W ̸= ∅, the lemma states that there must exist some v ∈ W such that there are at most bv + 1 edges in E(x). We remove v from W , since we know that when we obtain the minimumcost spanning tree F , there can be at most bv + 1 edges of F incident on v. If W = ∅, then by Theorem 11.4, we can compute a minimum spanning tree of cost at most the value of the linear programming relaxation. The algorithm is summarized in Algorithm 11.3. The following proof of the correctness of the algorithm is then not diﬃcult. Theorem 11.17: Algorithm 11.3 produces a spanning tree F such that the degree of v in F is at most bv + 1 for v ∈ W , and such that the cost of F is at most the value of the linear program (11.12). Proof. As in the proofs of Theorems 11.4 and 11.6, we show by induction that in each iteration, the LP solution xe on (V, E) and W is feasible for the input (V, E ′ ) and W ′ for the next iteration. It is easy to see this since any time we remove an edge e from E, xe = 0, which does not aﬀect the feasibility of the solution. Also, we only remove vertices from W , imposing fewer Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.2
Minimumcost boundeddegree spanning trees
295
constraints. Thus the cost of the optimal LP solution on the input (V, E ′ ) and W ′ is no greater than that on the input (V, E) and W . Since we remove a vertex v from W in each iteration, eventually W = ∅, and the algorithm computes a minimum spanning tree F on the ﬁnal set of edges. By Theorem 11.4, this cost is at most the cost of the ﬁnal linear programming solution. Since the value of the linear program never increases as we modify E and W , the cost of the minimum spanning tree is at most the value of initial linear program (11.12). Furthermore, we removed v from W only if there were at most bv + 1 edges remaining in E incident on v. Thus the computed spanning tree can have at most degree bv + 1 for each vertex v in the initial set W . We now turn to the proof of Lemma 11.16. As before, for the basic feasible solution x, we use the existence of a laminar collection L and a set of vertices Z as given in Theorem 11.12. We also need the following lemma. Lemma 11.18: For any e ∈ E such that xe = 1, χe ∈ span(L). Proof. By the proof of Theorem 11.12, the laminar collection L is such that span(L) ⊇ span(T ), where T is the collection of all tight sets. Notice that if xe = 1 for e = (u, v), then the set S = {u, v} is tight, since xe = x(E(S)) = S − 1 = 1. Thus χe = χE(S) ∈ span(T ) ⊆ span(L). We now restate Lemma 11.16 and give its proof. Lemma 11.16: For any basic feasible solution x to the linear program (11.12) in which W ̸= ∅, there is some v ∈ W such that there are at most bv + 1 edges incident on v in E(x). Proof. Assume the statement of the lemma is false, so that there are at least bv + 2 edges in E incident on every v ∈ Z, with W ̸= ∅. We derive the contradiction via a charging scheme, in which we assign each v ∈ Z and each S ∈ L a certain nonnegative charge, possibly fractional. We will then show by the falsity of the lemma statement, each v ∈ Z and each S ∈ L receives a charge of at least one. This will imply that the total charge is at least Z + L = E. However, we will also show that the total charge assigned must have been strictly less than E, giving the contradiction. Thus there must exist some v ∈ W ̸= ∅ such that there are at most bv + 1 edges of E incident on v. To carry out our charging scheme, for each e ∈ E we assign a charge of 12 (1 − xe ) to each endpoint of e that is in Z, and we assign a charge of xe to the smallest S ∈ L that contains both endpoints of e, if such an S exists. Note then that have assigned a total charge of at most (1 − xe ) + xe = 1 per edge e ∈ E. Now we show that each v ∈ Z and each S ∈ L receives a charge of at least one. Each v ∈ Z receives a charge of 21 (1 − xe ) from each edge of E incident on it. By hypothesis, ∑ there are at least bv + 2 edges incident on v. Since v ∈ Z implies v is tight, we know that e∈δ(v) xe = bv . Thus the total charge received by v is at least one, since ∑ 1 ∑ 1 (1 − xe ) = δ(v) − xe 2 2 e∈δ(v)
e∈δ(v)
1 ≥ [(bv + 2) − bv ] 2 = 1. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
296
Further uses of deterministic rounding of linear programs
∑ Now consider a set S ∈ L. If S contains no other set C ∈ L, then since S is a tight set, e∈E(S) xe = S − 1, and the total charge assigned to S is S − 1. Since S ≥ 2, the total charge assigned to S is at least one. Now suppose that S contains some C ∈ L. We call a set C ∈ L a child of S if it is strictly contained in S, and no other set C ′ ∈ L that contains C is also strictly contained in S. Let C1 , . . . , Ck be the children of S. Recalling that the sets S and C1 , . . . , Ck are tight, we have that x(E(S)) = S − 1 and x(E(Ci )) = Ci  − 1 and thus are integral. Furthermore, the E(Ci ) are disjoint and are contained in E(S) by the laminarity of L. Hence the total charge assigned to S is x(E(S)) −
k ∑
x(E(Ci )) ≥ 0.
i=1
∪ ∑k However, it cannot be the case that E(S) = ki=1 E(Ci ) since then χE(S) = i=1 χE(Ci ) , which violates the second property of Theorem 11.12, which says that these vectors are linearly independent. Thus it must be the case that the total charge assigned to S is x(E(S)) −
k ∑
x(E(Ci )) > 0,
i=1
and since x(E(S)) −
k ∑ i=1
x(E(Ci )) = (S − 1) −
k ∑
(Ci  − 1)
i=1
is integral, the diﬀerence must be at least one. Therefore at least one unit of charge is assigned to S. Finally we show that the total charge is strictly less than E via a threecase argument. First, if V ∈ / L, then there must exist some edge e such that e ∈ / E(S) for all S ∈ L. The charge xe > 0 is not made to any set, so that the total charge is strictly less than E. Second, if there exists some e = (u, v) ∈ E with xe < 1 such that one of its two endpoints is not in Z (say u∈ / Z), then the charge of 12 (1 − xe ) > 0 is not made to u, and again the total charge is strictly less than E. Finally, suppose that V ∈ L and for any e ∈ E with xe < 1, both endpoints of e are in Z. We will show that this∑ case gives rise to a contradiction, so one of the ﬁrst two cases must occur. We observe that v∈Z χδ(v) = 2χE(Z) + χδ(Z) and by hypothesis, each edge e ∈ δ(Z) ∪ E(V − Z) has xe = 1. Now we show that we can express 2χE(Z) + χδ(Z) by the sum of vectors χE(S) for S ∈ L, which will contradict the linear independence of the vectors χE(S) for S ∈ L and χδ(v)∑ for v ∈ Z. By Lemma 11.18, for any e ∈ ∑ δ(Z) ∪ E(V − Z), ∑χe ∈ span(L) since xe = 1. Thus v∈Z χδ(v) = 2χE(Z) + χδ(Z) = 2χE(V ) − 2 e∈E(V −Z) χe − e∈δ(Z) χe , and every term on the righthand side is in span(L); this proves that χδ(v) for v ∈ Z and χE(S) for S ∈ L are linearly dependent, which is a contradiction. In summary, we have been able to show the we can ﬁnd a spanning tree of cost at most OPT with degree at most bv + 1 for v ∈ W . We did this by proving a lemma showing that a basic feasible solution x to the linear programming relaxation of the problem must have some v ∈ W such that at most bv + 1 edges of E(x) are incident on v. The proof of this lemma, given above, uses the structure of the basic feasible solution as a laminar collection of tight sets, as well a charging scheme that uses fractional charges. In the next section we will use many of these same ideas in giving an approximation algorithm for another network design problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.3
11.3
Survivable network design and iterated rounding
297
Survivable network design and iterated rounding
In this section, we turn to a generalization of the generalized Steiner tree problem which was introduced in Section 7.4. This problem is called the survivable network design problem. In this problem, we are given as input an undirected graph G = (V, E), costs ce ≥ 0 for all e ∈ E, and connectivity requirements rij for all pairs of vertices i, j ∈ V , where i ̸= j. The connectivity requirements are nonnegative integers. The goal is to ﬁnd a minimumcost set of edges F ⊆ E such that for all pairs of vertices i, j with i ̸= j, there are at least rij edgedisjoint paths connecting i and j in (V, F ). The generalized Steiner tree problem is a special case of the survivable network design problem in which all rij ∈ {0, 1}. The survivable network design problem is motivated by the telecommunications industry. We wish to design lowcost networks that can survive failures of the edges. In the case of rij − 1 edge failures, vertices i and j will still be connected in the resulting set of edges. We may wish to have certain pairs of vertices to be highly connected, with others with connectivity requirement 1 if it is not as crucial that they be connected in case of failures. The problem can be modelled by the following integer program: minimize
∑
ce xe
e∈E
subject to
∑
xe ≥ max rij ,
∀S ⊆ V,
xe ∈ {0, 1} ,
∀e ∈ E.
i∈S,j ∈S /
e∈δ(S)
Consider any pair of vertices i, j with i ̸= j, and a set of edges F . By the maxﬂow/mincut theorem, there are at least rij edgedisjoint paths connecting i and j in (V, F ) if and only if every cut S separating i and j contains at least rij edges of F ; that is, δ(S) ∩ F  ≥ rij . Hence, a set of edges F is feasible if and only if δ(S) ∩ F  ≥ maxi∈S,j ∈S / rij for all S ⊆ V , which is exactly the constraint imposed by the integer program. Note that this insight can be used to solve the following linear programming relaxation in polynomial time: minimize
∑
ce xe
e∈E
subject to
∑
e∈δ(S)
xe ≥ max rij , i∈S,j ∈S /
0 ≤ xe ≤ 1,
∀S ⊆ V, ∀e ∈ E.
We use the ellipsoid method introduced in Section 4.3 with the following separation oracle. Given a solution x, we ﬁrst check that 0 ≤ xe ≤ 1 for all edges e ∈ E. We then create a maxﬂow instance for each pair of vertices i, j, i ̸= j, with the capacity of each edge set to xe . If the maximum ﬂow that can be sent from i to j is at least rij for each i, j ∈ V , then by the argument above, we know that all the constraints are satisﬁed. If for some i, j, the ∑ maximum ﬂow is less than rij , then there must be some cut S with i ∈ S, j ∈ / S, such that e∈δ(S) xe < rij , giving a violated constraint. It will be useful to consider a more general form of the linear program above. We will consider the following linear program with functions f on the vertex set such that f (S) is an Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
298
Further uses of deterministic rounding of linear programs
integer for all S ⊆ V : minimize
∑
ce xe
(11.17)
e∈E
subject to
∑
xe ≥ f (S),
∀S ⊆ V,
e∈δ(S)
0 ≤ xe ≤ 1
∀e ∈ E.
Clearly the linear programming relaxation of the survivable network design problem corresponds to the case f (S) = maxi∈S,j ∈S / rij . We will consider functions f that are weakly supermodular. Deﬁnition 11.19: A function f : 2V → Z is weakly supermodular if f (∅) = f (V ) = 0, and for each two sets A, B ⊆ V , one of the following two statements holds: f (A) + f (B) ≤ f (A ∩ B) + f (A ∪ B),
(11.18)
f (A) + f (B) ≤ f (A − B) + f (B − A).
(11.19)
The function f that we use for the survivable network design problem falls into this class. Lemma 11.20: The function f (S) = maxi∈S,j ∈S / rij is weakly supermodular. Proof. Trivially f (∅) = f (V ) = 0. We observe that f (S) = f (V − S) for any S ⊆ V . Also, for any disjoint A, B, f (A ∪ B) ≤ max(f (A), f (B)): choose i ∈ A ∪ B and j ∈ / A ∪ B attaining the maximum maxi∈A∪B,j ∈A∪B rij that deﬁnes f (A ∪ B). Then either i ∈ A and j ∈ / A or i ∈ B and / j∈ / B, so that max(f (A), f (B)) ≥ rij = f (A ∪ B). We then observe that f obeys the following four inequalities: f (A) ≤ max(f (A − B), f (A ∩ B)); f (A) = f (V − A) ≤ max(f (B − A), f (V − (A ∪ B))) = max(f (B − A), f (A ∪ B)); f (B) ≤ max(f (B − A), f (A ∩ B)); f (B) = f (V − B) ≤ max(f (A − B), f (V − (A ∪ B)) = max(f (A − B), f (A ∪ B)). Then summing together the two inequalities involving the minimum of f (A − B), f (B − A), f (A ∪ B), and f (A ∩ B) gives the desired result; so, for instance, if f (A − B) achieves the minimum of the four values, then we sum the ﬁrst and last inequalities, which then implies f (A) + f (B) ≤ f (A ∪ B) + f (A ∩ B). We now state a remarkable theorem that will allow us to obtain a 2approximation algorithm for the survivable network design problem. We will ﬁrst show how the theorem gives the approximation algorithm, then turn to the proof of the theorem. Theorem 11.21: For any basic feasible solution x to the linear program (11.17) such that f is a weakly supermodular function, there exists some edge e ∈ E such that xe ≥ 1/2. Given the theorem, we have the following rather simple idea for an approximation algorithm for the survivable network design problem: we solve the linear programming relaxation, ﬁnd all edges whose LP value is at least 1/2, and include them in our solution. We ﬁx the value of these variables to 1, resolve the linear program, and repeat the process until we have a feasible solution to the problem. Intuitively, we are always rounding variables up by at most a factor Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.3
Survivable network design and iterated rounding
299
F ←∅ i←1 while F is not a feasible solution do Solve LP (11.17) on edge set E − F with function fi , where fi (S) = f (S) − δ(S) ∩ F , to obtain basic optimal solution x Fi ← {e ∈ E − F : xe ≥ 1/2} F ← F ∪ Fi i←i+1 return F Algorithm 11.4: Deterministic rounding algorithm for the survivable network design problem. of 2, which will lead to the performance guarantee of 2. There are a number of details to be taken care of, but this is the main idea. We state the algorithm more formally in Algorithm 11.4. We let F be the set of edges in the solution; F is initially empty. In the ith iteration of the algorithm, we solve the linear program (11.17) on edge set E − F with the function fi , where fi (S) = f (S) − δ(S) ∩ F . Given a basic optimal solution to this linear program, we set Fi = {e ∈ E − F : xe ≥ 1/2}, and add Fi to F . Because we iteratively round up the LP solution to create the ﬁnal feasible solution, this technique is called iterated rounding. To show that the algorithm works, we must show that each function fi is again weakly supermodular, so that Theorem 11.21 applies; this will imply that Fi ̸= ∅ in each iteration, and thus there are at most E iterations before the algorithm terminates. We will also need to show that we can solve the linear program in each iteration. We start with the following lemma, which will be useful in showing that the fi are weakly supermodular. ∑ Lemma 11.22: Pick any ze ≥ 0 for all e ∈ E, and let z(E ′ ) = e∈E ′ ze for any E ′ ⊆ E. Then for any subsets A, B ⊆ V , z(δ(A)) + z(δ(B)) ≥ z(δ(A ∪ B)) + z(δ(A ∩ B)) and z(δ(A)) + z(δ(B)) ≥ z(δ(A − B)) + z(δ(B − A)). Proof. The proof is a simple counting argument; to prove each inequality, we show that an edge included in the sums on the righthand side of the inequalities appears at least as many times on the lefthand side. See Figure 11.4 for an illustration. For instance, any edge whose two endpoints are in A − B and V − (A ∪ B) appears in δ(A ∪ B), δ(A − B), and δ(A), but not δ(A ∩ B), δ(B − A), or δ(B). Thus it is included once in the righthand side and lefthand side of the ﬁrst inequality, and once in the righthand side and lefthand side of the second inequality. We simply need to check all possible cases for the four diﬀerent places in which the two diﬀerent endpoints of an edge can be. We note that an edge whose two endpoints are in A − B and B − A appears twice on the lefthand side of the ﬁrst inequality, but not at all on the righthand side; similarly, an edge whose two endpoints are in A ∩ B and V − (A ∪ B) appears twice on the lefthand side of the second inequality, but not on the righthand side; thus the inequalities can be strict. Lemma 11.23: For any F ⊆ E, the function fi (S) = f (S) − δ(S) ∩ F  is weakly supermodular if f is weakly supermodular. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
300
Further uses of deterministic rounding of linear programs
A
B
Figure 11.4: Proof of Lemma 11.22. Proof. Set ze = 1 if e ∈ F and ze = 0 otherwise. Then it is equivalent to show that fi is weakly supermodular when fi (S) = f (S) − z(δ(S)). It is clear that fi (∅) = fi (V ) = 0. Now pick any two subsets A, B ⊆ V . We note that for the function f , it is the case that either f (A) + f (B) ≤ f (A ∪ B) + f (A ∩ B) or f (A) + f (B) ≤ f (A − B) + f (B − A). Suppose the former holds. Then fi (A) + fi (B) = f (A) + f (B) − z(δ(A)) − z(δ(B)) ≤ f (A ∪ B) + f (A ∩ B) − z(δ(A ∪ B)) − z(δ(A ∩ B)) = fi (A ∪ B) + fi (A ∩ B), where the inequality follows by hypothesis and by Lemma 11.22. The other case is identical. Lemma 11.24: For any F ⊆ E, we can solve the linear program (11.17) in polynomial time with edge set E − F and function g(S) = f (S) − δ(S) ∩ F  when f (S) = maxi∈S,j ∈S / rij . Proof. We use the ellipsoid method with a separation oracle similar to the one given to solve the LP with the original function f . For a solution x, we ﬁrst check that 0 ≤ xe ≤ 1 for all e ∈ E − F . Then for each pair of vertices i, j, with i ̸= j, we create a maximum ﬂow instance with the capacity of each edge e ∈ E − F set to xe and the capacity of each edge e ∈ F set to 1. We then check that the ﬂow between i and j is at least rij . If it is, then for any i, j ∈ V and any cut S with i ∈ S and j ∈ / S, it must be the case that the capacity is at least rij , which implies that x(δ(S)) + δ(S) ∩ F  ≥ rij , which implies x(δ(S)) ≥ rij − δ(S) ∩ F . Since this holds for all pairs i, j ∈ V , then x(δ(S)) ≥ g(S). Similarly, if for some pair i, j ∈ V , the maximum ﬂow is less than rij , then the capacity of the minimum cut S with i ∈ S and j ∈ / S is less than rij . Then x(δ(S)) + δ(S) ∩ F  < rij , or x(δ(S)) < rij − δ(S) ∩ F  ≤ g(S), and this is a violated constraint. We can now prove that Theorem 11.21 implies that the algorithm is a 2approximation algorithm. Theorem 11.25: Given Theorem 11.21, Algorithm 11.4 is a 2approximation algorithm for the survivable network design problem. Proof. We prove a slightly stronger statement by induction on the number of iterations of the algorithm; in particular, we will show that the algorithm has a performance guarantee of 2 for Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.3
Survivable network design and iterated rounding
301
any weakly supermodular function f . We only need that the function f is for the survivable network design problem in order to solve the linear programming relaxations in polynomial time. Let x be the solution to the original linear programming relaxation with function f∑ 1 = f for any weakly supermodular function f . For any subset E ′ ⊆ E of edges, let c(E ′ ) = e∈E ′ ce . The base case is straightforward: if after one iteration, ∑ F = F1 is a feasible solution, then since F1 = {e ∈ E : xe ≥ 1/2}, it is clear that c(F ) ≤ 2 e∈E ce xe ≤ 2 OPT . Now suppose that the statement holds if the algorithm takes k iterations; we show that it holds if the algorithm takes k + 1 iterations. By induction, the cost of all the edges added from the second iteration onwards is no more than twice the value of the LP solution for the weakly supermodular function f2 ; that is, if x′ ∑ is the solution found in the second iteration for the LP with function f2 , then c(F − F1 ) ≤ 2 e∈E−F1 ce x′e since the∑ algorithm ﬁnds a solution for function f2 in k iterations. For e ∈ F1 , we know that c(F1 ) ≤ 2 e∈F1 ce xe , since xe ≥ 1/2 for all e ∈ F1 . To complete the proof, we will show that x is a feasible solution on the edges E − F1 for the function f2 . Thus ∑ ∑ ce xe , ce x′e ≤ e∈E−F1
e∈E−F1
so that c(F ) = c(F − F1 ) + c(F1 ) ≤ 2
∑
ce x′e + 2
e∈E−F1
≤ 2
∑
= 2
ce xe
e∈F1
ce xe + 2
e∈E−F1
∑
∑ ∑
ce xe
e∈F1
ce xe ≤ 2 OPT .
e∈E
To see that x is feasible for the LP for the function f2 on edges E − F1 , we note that for any S ⊆ V , x(δ(S)) ≥ f1 (S) implies that x(δ(S)∩(E −F1 )) = x(δ(S))−x(δ(S)∩F1 ) ≥ f1 (S)−x(δ(S)∩F1 ) ≥ f1 (S)−δ(S)∩F1  = f2 (S), where for the second inequality we use that xe ≤ 1. We now turn to the proof of Theorem 11.21, and show that for any basic feasible solution x, xe ≥ 1/2 for some edge e ∈ E. We assume without loss of generality that 0 < xe < 1 for all e ∈ E; we can do this since if xe = 1, then we are done, while if for some edge e ∈ E, xe = 0, we can simply remove it from the graph. We will now need a deﬁnition. Deﬁnition 11.26: For the given solution x to LP (11.17), we say that a set S ⊆ V is tight if x(δ(S)) = f (S). We are now able to state the following theorem, which we will need to prove Theorem 11.21. Theorem 11.27: For any basic feasible solution x to the linear program (11.17) with f a weakly supermodular function, there is a collection L of subsets of vertices with the following properties: 1. For all S ∈ L, S is tight. 2. The vectors χδ(S) for S ∈ L are linearly independent. 3. L = E. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
302
Further uses of deterministic rounding of linear programs
4. The collection L is laminar. The proof of this theorem is almost identical to the proof of Theorem 11.12 in Section 11.2, and so we leave it as an exercise (Exercise 11.2). We now turn to our proof of Theorem 11.21. We restate the theorem here for convenience. The proof is similar to the fractional charging argument used for the minimumcost boundeddegree spanning tree problem in Lemma 11.16. Theorem 11.21: For any basic feasible solution x to the linear program (11.17) such that f is a weakly supermodular function, there exists some edge e ∈ E such that xe ≥ 1/2. Proof. We give a proof by contradiction. Suppose that for all e ∈ E, 0 < xe < 12 . Given this hypothesis, we will give a charging scheme in which we distribute charge to the sets S ∈ L whose total is strictly less than E. However, we will also show that each S ∈ L receives a charge of at least one, for a total charge of at least L = E. This will give our contradiction. For each edge e ∈ E we will distribute a charge of 1 − 2xe > 0 to the smallest set S ∈ L that contains both endpoints of e, if such a set exists, and for each endpoint v of e we distribute a charge of xe > 0 to the smallest set S ∈ L containing v, if such a set exists. Both 1 − 2xe and xe are positive since we assume that 0 < xe < 1/2. Thus the total charge distributed is at most 1 − 2xe + 2xe = 1 per edge. However, notice that for any set S ∈ L not contained in any other set of L, there must be edges e ∈ δ(S). These edges will not have both endpoints contained in some set S ∈ L and for these edges we distribute charge strictly less than 1 since the charge 1 − 2xe > 0 for the edge is not distributed. Thus the total charge distributed is strictly less than E. Now we show that each S ∈ L receives a charge of at least one. We say a set C ∈ L is a child of S if it is strictly contained in S, and no other set C ′ ∈ L that contains C is also strictly contained in S. Let C1 , . . . , Ck be the children of S (if any exist). Since S and the C∪ i are in L, they are all tight sets so that x(δ(S))∪ = f (S) and x(δ(Ci )) = f (Ci ) for all i. Let C = i Ci . We now divide the edges in ES = δ(S) ∪ i δ(Ci ) into four sets (see Figure 11.5 for an illustration): • Ecc is the set of edges e ∈ ES that have one endpoint in some Ci and the other in Cj for j ̸= i (“childchild edges”). By the charging scheme such edges contribute a charge of 1 − 2xe to S since S is the smallest set to contain both endpoints. • Ecp is the set of edges e ∈ ES that have one endpoint in Ci and the other in S − C (“childparent edges”). By the charging scheme such edges contribute a charge of xe to S for the one endpoint in S −C, and a charge of 1−2xe since S is the smallest set containing both endpoints of S. So the total charge given to S is 1 − xe for each edge e ∈ Ecp . • Epo is the set of edges e ∈ ES that have one endpoint in S − C and the other outside S (“parentout edges”). By the charging scheme such edges contribute a charge of xe to S for the one endpoint in S − C. • Eco is the set of edges of ES in both δ(S) and δ(Ci ) for some i (“childout edges”). Such edges contribute no charge to S. We claim that it cannot be the case that all edges of ES are in Eco . If this were ∑kthe case, then an edge is in δ(S) if and only if it is in δ(Ci ) for some i, which implies χδ(S) = i=1 χδ(Ci ) . This contradicts Theorem 11.27, which states that the χδ(T ) are linearly independent for T ∈ L. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.3
Survivable network design and iterated rounding
303
∈ Epo ∈ Eco S ∈ Ecp ∈ Ecc C1
Ck
C2
Figure 11.5: Illustration of the edge sets in the proof of Theorem 11.21. Thus at least one edge must be in Ecc , Ecp , or Epo ; S receives a positive charge for each such edge, and the total charge received is Ecc  − 2x(Ecc ) + Ecp  − x(Ecp ) + x(Epo ) > 0. By the deﬁnitions of the edge sets above, we note that x(δ(S)) −
k ∑
x(δ(Ci )) = x(Epo ) − x(Ecp ) − 2x(Ecc ).
i=1
Then we have that the total charge received by S is ( Ecc  − 2x(Ecc ) + Ecp  − x(Ecp ) + x(Epo ) = Ecc  + Ecp  +
x(δ(S)) −
k ∑
) x(δ(Ci )) .
i=1
Since all sets are tight, this total charge is equal to ( Ecc  + Ecp  +
f (S) −
k ∑
) f (Ci ) .
i=1
Since this last expression is a sum of integers, and we know the total charge is positive, the total charge must be at least one. Thus each S ∈ L gets a charge of at least one, for a total charge of at least L = E, but we distributed a total charge of strictly less than E, which gives the contradiction. As we observed at the end of Section 7.4, the linear programming relaxation for the generalized Steiner tree problem has an integrality gap that is essentially 2. Since the generalized Steiner tree problem is a special case of the survivable network design problem, and the linear program used for the generalized Steiner tree problem is a special case of the linear program (11.17) we use for the survivable network design problem, the integrality gap of (11.17) is also essentially 2. Thus we cannot obtain any better performance guarantee by comparing the value of the algorithm’s solution to the value of the LP relaxation as we do in the deterministic rounding argument above. As was mentioned at the beginning of the chapter, the motivation for studying the survivable network design problem comes from wanting to design lowcost networks that can survive failures in the edges of the network. Suppose we wish to design networks that can also survive failures in the vertices? In that case, we may wish for there to be rij vertexdisjoint paths Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
304
Further uses of deterministic rounding of linear programs
between i and j for all i, j ∈ V . In Section 16.4, we show that this variant of the survivable network design problem is substantially harder to approximate that the edgedisjoint version we have considered above.
Exercises 11.1 We consider a variant of the generalized assignment problem without costs. Suppose we are given a set of n jobs to be assigned to m machines. Each job j is to be scheduled on exactly one machine. If job j is scheduled on machine i, then it requires pij units of processing time. The goal is to ﬁnd a schedule of minimum length, which is equivalent to ﬁnding an assignment of jobs to machines that minimizes the maximum total processing time required by a machine. This problem is sometimes called scheduling unrelated parallel machines so as to minimize the makespan. We show that deterministic rounding of a linear program can be used to develop a polynomialtime 2relaxed decision procedure (recall the deﬁnition of a relaxed decision procedure from Exercise 2.4). Consider the following set of linear inequalities for a parameter T : m ∑
xij = 1,
j = 1, . . . , n,
pij xij ≤ T,
i = 1, . . . , m,
i=1 n ∑ j=1
xij ≥ 0,
i = 1, . . . , m, j = 1, . . . , n
xij = 0,
if pij > T.
If a feasible solution exists, let x be a basic feasible solution for this set of linear inequalities. Consider the bipartite graph G on machine nodes M1 , . . . , Mm and job nodes N1 , . . . , Nn with edges (Mi , Nj ) for each variable xij > 0. (a) Prove that the linear inequalities are a relaxation of the problem, in the sense that if the length of the optimal schedule is T , then there is a feasible solution to the linear inequalities. (b) Prove that each connected component of G of k nodes has exactly k edges, and so is a tree plus one additional edge. (c) If xij = 1, assign job j to machine i. Once all of these jobs are assigned, use the structure of the previous part to show that it is possible to assign at most one additional job to any machine. Argue that this results in a schedule of length at most 2T . (d) Use the previous parts to give a polynomialtime 2relaxed decision procedure, and conclude that there is a polynomialtime 2approximation algorithm for scheduling unrelated parallel machines to minimize the makespan. 11.2 In this exercise, we prove Theorem 11.27. (a) First, prove the following. Given two tight sets A and B, one of the following two statements must hold: • A ∪ B and A ∩ B are tight, and χδ(A) + χδ(B) = χδ(A∩B) + χδ(A∪B) ; or Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.3
Survivable network design and iterated rounding
305
• A − B and B − A are tight, and χδ(A) + χδ(B) = χδ(A−B) + χδ(B−A) . (b) Use the above to prove Theorem 11.27. 11.3 Consider the following LP relaxation for the traveling salesman problem: ∑ minimize ce xe e∈E
∑
subject to
xe ≥ 2,
∀S ⊂ V, S ̸= ∅,
e∈δ(S)
0 ≤ xe ≤ 1,
∀e ∈ E.
Show that for any basic feasible solution x to the linear program, there must exist some e ∈ E such that xe = 1. 11.4 Recall the deﬁnition of a branching from Exercise 7.5: We are given a directed graph G = (V, A), and a designated root vertex r ∈ V . A branching is a subset F ⊆ A of arcs such that for every v ∈ V , there is exactly one directed path from r to v. Note that this implies that in F the indegree of any node (except the root) is 1. In the boundeddegree branching problem, we are given degree bounds bv , and the goal is to ﬁnd a branching such that the outdegree of v is at most bv for all v ∈ V . In the following, we will give a polynomialtime algorithm to compute a branching in which the outdegree of v is at most bv + 2 for all v ∈ V , given that a branching exists with outdegree bv for all v ∈ V . Given the input graph G = (V, A) and any subset F ⊆ A, let δ + (S) be the set of all arcs in A with their tails in S and heads not in S, and δ − (S) be the set of all arcs in A with their heads in S and their tails not in S, δF+ (S) = δ + (S) ∩ F , and δF− (S) = δ − (S) ∩ F . Then consider the following linear programming relaxation, deﬁned for a given set of arcs A, F ⊆ A and W ⊆ V : ∑ minimize xa a∈A
subject to
∑
xa ≥ 1 − δF− (S),
∀S ⊆ V − r,
xa ≤ bv − δF+ (v),
∀v ∈ W,
a∈δ − (S)
∑
a∈δ + (v)
0 ≤ xa ≤ 1
∀a ∈ A − F.
Consider the following algorithm for the problem. Initially, F = ∅ and W = V . While A−F ̸= ∅, ﬁnd a solution to the linear programming relaxation for A, F , and W . Remove from A any arc a ∈ A − F such that xa = 0. Add to F any arc a ∈ A − F such that xa = 1. Then for any v ∈ W such that there are at most bv − δF+ (v) + 2 arcs coming out of v in A − F , remove v from W and add to F all outgoing arcs from v in A − F . When A − F = ∅, then output any branching rooted at r in F . (a) Prove that the linear programming relaxation is a relaxation to the problem in the sense that if there is a feasible solution to the problem given the degree bounds, then there is a feasible solution to the linear programming relaxation. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
306
Further uses of deterministic rounding of linear programs
(b) Prove that the following is true for any basic feasible solution x to the linear programming relaxation, for any A, F ⊆ A, and W ⊆ V : If 0 < xa < 1 for all a ∈ A−F , then there exists a set Z ⊆ W and a collection L of subsets of V such that: 1. x(δ − (S)) = 1 for all S ∈ L and x(δ + (v)) = bv − δF+ (v) for all v ∈ Z; 2. The characteristic vectors χδ− (S) over all S ∈ L and χδ+ (v) over all v ∈ Z are linearly independent; 3. A − F  = L + Z; 4. L is laminar. (c) Show that at the start of any iteration of the algorithm, every arc a ∈ A − F has its tail in W . (d) Prove that in each iteration, there must exist some v ∈ W such that there are at most ∑ bv − δF+ (v) + 2 arcs coming out of v in A − F . (Hint: ﬁrst show that if L < a∈A−F xa + 2W , then such a vertex in W must ∑ exist. Then design a charging argument that charges a total strictly less than a∈A−F xa + 2W  such that each S ∈ L gets at least one unit of charge.) (e) Prove that the algorithm runs in polynomial time and produces the desired output. 11.5 The minimum kedgeconnected subgraph problem takes as input an undirected graph G = (V, E) and a positive integer k. The goal is to ﬁnd the smallest set of edges F ⊆ E such that there are at least k edgedisjoint paths between each pair of vertices. Consider the following linear programming relaxation of the problem: minimize
∑
xe
e∈E
subject to
∑
xe ≥ k,
∀S ⊆ V,
e∈δ(S)
0 ≤ xe ≤ 1
e ∈ E.
(a) Prove that the linear program is indeed a relaxation of the problem. (b) Prove that the linear program can be solved in polynomial time. (c) Suppose we obtain a basic optimal solution to the LP relaxation and round up every fractional variable to 1. Prove that this gives a (1 + k4 )approximation algorithm for the problem. 11.6 In this problem, we revisit the generalized assignment problem from Section 11.1 and give an iterated rounding algorithm for it. We consider a slight generalization of the problem in which for each machine i there can be at most Ti units of processing assigned to i. We now modify the linear programming relaxation of the problem given in Section 11.1. We let E denote a set of possible (i, j) pairs for which we can assign job j to machine i. Let M = {1, . . . , m} be the set of machines and J = {1, . . . , n} be the set of jobs. Initially, E consists of all (i, j) such that i ∈ M , j ∈ J and pij ≤ Ti . We also have a subset M ′ ⊆ M , where initially M ′ = M , and a subset J ′ ⊆ J, where initially J ′ = J. We also have a total amount of processing Ti′ that can be assigned to machine i ∈ M ′ , Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
11.3
Survivable network design and iterated rounding
307
where Ti′ is initially Ti . Then the relaxation is as follows: ∑ minimize cij xij (i,j)∈E
subject to
∑
xij = 1,
∀j ∈ J ′ ,
i∈M :(i,j)∈E
∑
pij xij ≤ Ti′ ,
∀i ∈ M ′ ,
j∈J:(i,j)∈E
xij ≥ 0,
∀(i, j) ∈ E.
Consider the following algorithm: While J ′ ̸= ∅, we ﬁnd a basic optimal solution to the LP relaxation. We remove from E any pair (i, j) such that xij = 0. If there is a variable xij = 1, then we assign job j to machine i; remove j from J ′ and reduce Ti′ by pij . Let Ji be the jobs fractionally assigned to machine i ∈ M ′ , so that ∑Ji = {j ∈ J : xij > 0}. If there is a machine i such that either Ji  = 1, or Ji  = 2 and j∈Ji xij ≥ 1, then remove i from M ′ . (a) Show that for any basic feasible solution x to the linear program, the following is J ′′ ⊆ J ′ and M ′′ ⊆ M ′ such that the LP constraint ∑ true: there exist subsets ′ ′′ j∈J:(i,j)∈E pij xij = Ti for all i ∈ M , the vectors corresponding to the LP constraints for J ′′ and M ′′ are linearly independent, and J ′′  + M ′′  is equal to the number of variables xij > 0. (b) Prove that for any basic feasible solution x to the LP, either there is some (i, j) ∈ E ′ such that xij ∈ {0, 1}, or there ∑ exists some i ∈ M with Ji  = 1, or there exists some ′ i ∈ M with Ji  = 2 and j∈Ji xij ≥ 1. (c) Prove that the algorithm above returns a solution with total cost at most OPT, and such that machine i is assigned total processing time 2Ti for all i ∈ M .
Chapter Notes We have seen in earlier sections some examples of the use of the structure of basic feasible solutions in approximation algorithms. As mentioned in the introduction, in Exercise 1.5, we saw that for a linear programming relaxation of the vertex cover problem, for any basic feasible solution x, each variable xi is either 0, 1, or 1/2. Also, in the algorithm for the bin packing problem in Section 4.6, we used the fact that for any basic feasible solution to the linear programming relaxation, the number of nonzero variables is at most the number of distinct piece sizes. The ﬁrst more sophisticated use of the structure of a basic feasible solution in an approximation algorithm is due to Lenstra, Shmoys, and Tardos [215]. They give a 2approximation algorithm for scheduling unrelated parallel machines. Their algorithm is given as Exercise 11.1. This work led to the result of Section 11.1 on the generalized assignment problem; the result we give there is due to Shmoys and Tardos [263]. This result does not use the properties of a basic feasible solution; it only uses feasibility. The alternate result in Exercise 11.6 for the generalized assignment problem that does use the properties of a basic feasible solution is due to Singh [268]. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
308
Further uses of deterministic rounding of linear programs
The idea of iterated rounding for obtaining approximation algorithms is due to Jain [175]; he introduced the algorithm for the survivable network design problem in Section 11.3. He also introduced the use of a charging scheme to prove Theorem 11.21. The proof we give is a simpliﬁcation due to Nagarajan, Ravi, and Singh [230]. The ﬁrst algorithm to achieve an additive factor of 2 for the minimumcost bounded degree spanning tree problem is due to Goemans [134]. Iterated rounding was then applied by Singh and Lau [269] to the problem; they obtain the approximation algorithm of Section 11.2 with the additive factor of 2. Singh and Lau also have the ﬁrst algorithm bounding the degrees to within an additive factor of 1. We present a somewhat simpliﬁed version of this algorithm and analysis in Section 11.2 due to Lau and Singh [209] (see also Lau, Ravi, and Singh [208]) that draws upon work of Bansal, Khandekar, and Nagarajan [32] for a more general problem. Bansal et al. introduced the idea of a fractional charging scheme. The separation oracle we give for solving the linear program (11.12) is from Cunningham [85]. Exercise 11.2 is due to Jain [175]. The result of Exercise 11.3 was shown by Boyd and Pulleyblank [56] prior to all of the work on iterated rounding. Exercise 11.4 is due to Bansal, Khandekar, and Nagarajan [32]. Exercise 11.5 is due to Gabow, Goemans, Tardos, and Williamson [120].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
C h a p t e r 12
Further uses of random sampling and randomized rounding of linear programs
In this chapter, we revisit – for the ﬁnal time – the uncapacitated facility location problem, and consider a randomized rounding algorithm for it. As we observed in Section 5.8, a natural randomized rounding algorithm is to open each facility i independently with probability equal to its associated variable in an optimal solution to the linear programming relaxation. As we will show, the diﬃculty with this algorithm is that it is possible that a client will not be near an open facility. However, we overcome this diﬃculty by using a combination of this randomized rounding algorithm and our previous randomized algorithm that uses clustering to ensure that each client is not too far away from an open facility. This will result in an approximation algorithm with a performance guarantee of 1 + 2e ≈ 1.736. We also introduce a new problem, the singlesource rentorbuy problem, in which we must either rent or buy edges to connect a set of terminals to a root vertex. Here we make use once again of random sampling by using a sampleandaugment technique: we draw a random sample of the terminals, buy edges connecting them to the root, then augment the solution to a feasible solution by renting whatever edges are needed to make the solution feasible. We then turn to the Steiner tree problem introduced in Exercise 2.5; recall that the goal is to ﬁnd a minimumcost set of edges that connects a set of terminals. The Steiner tree problem is a special case of the prizecollecting Steiner tree problem, the generalized Steiner tree problem, and the survivable network design problem that we have considered in previous chapters; we introduced various LP rounding and primaldual algorithms for these problems. For the Steiner tree problem, we introduce a new linear programming relaxation, and then combine both iterated and randomized rounding to obtain a good approximation algorithm. Finally, we consider the maximum cut problem in dense graphs, and we obtain a polynomialtime approximation scheme for the problem by using a combination of most of our tools in randomization: random sampling, randomized rounding, and Chernoﬀ bounds. 309
310
Further uses of random sampling and randomized rounding of linear programs
12.1
The uncapacitated facility location problem
In this section, we turn to another randomized rounding algorithm for the uncapacitated facility location problem. This algorithm will give us the best overall performance guarantee for this problem of all the discussions in this book. Recall that the input to the problem is a set of clients D and a set of facilities F , with facility costs fi for all facilities i ∈ F , and assignment costs cij for all facilities i ∈ F and clients j ∈ D. We assume that the clients and facilities are in a metric space and that the assignment cost cij is the distance between client j and facility i. The goal is to select a subset of facilities to open and an assignment of clients to open facilities so as to minimize the total cost of the open facilities plus the assignment costs. Recall the following linear programming relaxation of the problem: ∑ ∑ minimize fi yi + cij xij i∈F
i∈F,j∈D
∑
subject to
xij = 1,
∀j ∈ D,
(12.1)
xij ≤ yi ,
∀i ∈ F, j ∈ D,
(12.2)
xij ≥ 0,
∀i ∈ F, j ∈ D,
i∈F
yi ≥ 0,
∀i ∈ F.
Recall also the dual of this LP relaxation: ∑ maximize vj j∈D
subject to
∑
wij ≤ fi ,
∀i ∈ F,
j∈D
vj − wij ≤ cij , wij ≥ 0,
∀i ∈ F, j ∈ D, ∀i ∈ F, j ∈ D.
In Sections 4.5 and 5.8, we said that given an optimal LP solution (x∗ , y ∗ ), a client j neighbors a facility i if x∗ij > 0, and we set N (j) = {i ∈ F : x∗ij > 0}. We set N 2 (j) = {k ∈ D : k neighbors some i ∈ N (j)}. We also showed the following lemma that relates the dual variable of a client to the cost of assigning the client to a neighboring facility. Lemma 12.1 (Lemma 4.11): If (x∗ , y ∗ ) is an optimal solution to the facility location LP and (v ∗ , w∗ ) is an optimal solution to its dual, then x∗ij > 0 implies cij ≤ vj∗ . In Section 5.8, we considered a randomized rounding algorithm for the problem that clustered the clients and facilities as in the algorithm of Section 4.5, but then used randomized rounding to decide which facility in the cluster to open. In this section, we consider applying randomized rounding directly to decide which facilities to open. In particular, suppose ∑ we de∗ cide to open each facility i with probability yi . Then the expected facility cost is i∈F fi yi∗ , which is at most OPT, and if a neighbor of a given client j is open, then the service cost for ∗ j∑is at most v∑ j by the lemma above. If all clients have a neighbor open, then the total cost is ∗ ∗ i∈F fi yi + j∈D vj ≤ 2 · OPT . However, it is possible that for a given client j, no neighbor of j is opened. Using 1 − x ≤ e−x , the probability that no neighbor of j is opened is ∏ ∏ ∗ Pr[no neighbor of j open] = (1 − yi∗ ) ≤ e−yi . i∈N (j)
i∈N (j)
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.1
The uncapacitated facility location problem
311
Solve LP, get optimal, complete primal solution (x∗ , y ∗ ) and dual solution (v ∗ , w∗ ) C←D T ←F k←0 while C ̸= ∅ do k ←k+1 Choose jk ∈ C that minimizes vj∗ over all j ∈ C Choose exactly one ik ∈ N (jk ) with probability x∗ik jk = yi∗k Open ik C ← C − {jk } − N 2 (jk ) T ← T − N (jk ) foreach i ∈ T do open i with probability yi∗ Assign each client j to nearest open facility Algorithm 12.1: Randomized rounding algorithm for the uncapacitated facility location problem. ∑ By LP constraint (12.2), x∗ij ≤ yi∗ , and by LP constraint (12.1), i∈N (j) x∗ij = 1, so that this probability is at most ∑ ∏ ∏ −x∗ ∗ ∗ e−yi ≤ e ij = e− i∈N (j) xij = e−1 . i∈N (j)
i∈N (j)
It is possible that this upper bound on the probability can be achieved, and in this case we do not have a good way of bounding the cost of assigning the client to the nearest open facility. Thus we need a more sophisticated approach. The main idea of the algorithm in this section is to combine this randomized rounding with the clustering ideas of earlier chapters. This way, if a client j has no neighboring facility open via randomized rounding, then we know that there is some facility open that is not too far away. Before we give the algorithm, we need a primal solution to the linear program that has a particular form. We say that a solution is complete if whenever x∗ij > 0, then x∗ij = yi∗ . It is possible to take any optimal solution to the linear program and create an equivalent complete solution with the same objective function value by making additional copies of each facility (where the facility costs and service costs of the copy are identical to the original). We leave this as an exercise (Exercise 12.1) and from here on assume that we have a complete solution. We give the algorithm in Algorithm 12.1. As in the algorithm in Section 5.8, we choose cluster centers jk via some criterion (here minimizing vj∗ among the remaining clients), then randomly choose a neighboring facility ik to open according to the probabilities x∗ik jk . Note that by completeness x∗ik jk = yi∗k . After all clients end up in some cluster, for all facilities i not in some cluster, we open them independently with probability yi∗ . Note that this algorithm is now not the same as opening each facility i with probability yi∗ independently: if a client j is chosen in some iteration, then exactly one of its neighboring facilities will be opened. Thus the probability of a facility being opened is now dependent on the probability of other facilities being opened. The following lemma shows that this does not aﬀect the bound on the probability that some neighbor of a given client is opened. Lemma 12.2: Given an arbitrary client j ∈ D, the probability that no neighbor of j is opened is at most 1e . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
312
Further uses of random sampling and randomized rounding of linear programs
Proof. We partition the facilities neighboring j into sets Xp as follows. Let Xk be the facilities in N (j) in the cluster formed in the kth iteration, and let each facility i remaining at the end of the algorithm be put in its own set Xp . Let Ok be the event that some facility in Xk is opened. Note that by the structure of the algorithm, the events Ok are now independent. ∑ Let Yk∗ be the probability that event Ok occurs. Then Yk∗ = Pr[Ok ] = i∈Xk yi∗ . Furthermore, by the completeness of the solution, ∑∑ ∑∑ ∑ yi∗ = x∗ij . Yk∗ = k i∈Xk
k
k i∈Xk
Then since the Xk partition N (j) and by LP constraint (12.1), ∑
Yk∗ =
∑∑
x∗ij =
k i∈Xk
k
∑
x∗ij = 1.
i∈F
By following the reasoning as given earlier, we have that ∑ ∏ ∏ ∗ ∗ (1 − Yk∗ ) ≤ e−Yk = e− k Yk = e−1 . Pr[no neighbor of j open] = k
k
We can now prove the following theorem. Theorem 12.3: Algorithm 12.1 is a (1 + 3e )approximation algorithm for the uncapacitated facility location problem, where 1 + 3e ≈ 2.104. Proof. We follow the proof of Theorem 5.19. In iteration k, the expected cost of the facility opened is ∑ ∑ fi yi∗ , fi x∗ijk ≤ i∈N (jk )
i∈N (jk )
using the LP constraint x∗ijk ≤ yi∗ , so that the expected cost of facilities opened in this way is ∑ ∑
fi yi∗ .
k i∈N (jk )
∪ ∗ We open each remaining facility ∑ i ∈ F∗ − k N (jk ) with probability yi , so that the total expected cost of opening facilities is i∈F fi yi . Given an arbitrary client j ∈ D, if some neighbor of j is not opened, then as argued in the proof of Theorem 4.13, we can assign j to some open facility in its cluster at cost at most 3vj∗ . Note that for any given facility i ∈ N (j), the probability that i is opened is yi∗ , so that although the probabilities that diﬀerent i ∈ N (j) are opened ∑ the expected assignment cost ∑ are dependent, for j given that some i ∈ N (j) is opened is i∈N (j) cij yi∗ = i∈N (j) cij x∗ij , with the equality following by the completeness of the solution. Thus the expected assignment cost for j is Pr[some neighbor of j is open] · E[assignment costsome neighbor of j is open] + Pr[no neighbor of j is open] · E[assignment costno neighbor of j is open] ∑ 1 ≤1· cij x∗ij + · (3vj∗ ). e i∈N (j)
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.2
The singlesource rentorbuy problem
313
Therefore, the overall expected assignment cost is at most ∑∑
cij x∗ij +
j∈D i∈F
3∑ ∗ vj , e j∈D
and the overall expected cost is ∑ i∈F
fi yi∗ +
∑∑
cij x∗ij +
j∈D i∈F
( ) 3∑ ∗ 3 3 vj ≤ OPT + · OPT = 1 + OPT . e e e j∈D
We can improve the algorithm to a (1 + 2e )approximation algorithm by making a small change to how client jk is selected in the kth iteration, and by tightening the analysis. Following ∗ our notation in ∑Section ∗5.8, we let Cj be the assignment cost incurred by j in the LP solution, ∗ so that Cj = i∈F cij xij . As in our algorithm in that section, in the kth iteration, we choose the client j that minimizes vj∗ + Cj∗ instead of choosing the client that minimizes vj∗ . Let pj denote the probability that no neighbor of client j is opened by the algorithm; we know pj ≤ 1e . The analysis in the theorem above is slightly loose in the sense that in analyzing the expected assignment cost, we bounded the probability that some neighbor of j is opened by 1. We can use the following lemma to make this part of the analysis slightly tighter. The proof of the lemma is quite technical, and so we omit it. Lemma 12.4: Let Aj be the expected assignment cost for j given that no neighbor of j is opened. Then the expected assignment cost of client j ∈ D is at most (1 − pj )Cj∗ + pj Aj . In the algorithm of Figure 12.1, we chose client jk in the kth iteration to minimize vj∗ , which gave us a bound Aj ≤ 3vj∗ . By choosing jk to minimize vj∗ + Cj∗ , we have that Aj ≤ 2vj∗ + Cj∗ (as in the analysis in Theorem 5.19). Thus we get that the expected assignment cost of the modiﬁed algorithm is 2 (1 − pj )Cj∗ + pj (2vj∗ + Cj∗ ) = Cj∗ + 2pj vj∗ ≤ Cj∗ + vj∗ . e ∑ Given that the expected facility cost is at most i∈F fi yi∗ , we get that the overall expected cost is at most ∑ ∑ 2 2∑ ∗ fi yi∗ + Cj∗ + vj ≤ OPT + OPT . e e i∈F
j∈D
j∈D
This yields the following theorem. Theorem 12.5: Algorithm 12.1, modiﬁed to choose jk to minimize vj∗ + Cj∗ , is a (1 + 2e )approximation algorithm for the uncapacitated facility location problem, where 1 + 2e ≈ 1.736.
12.2
The singlesource rentorbuy problem
In this section, we consider the singlesource rentorbuy problem. The input for the problem is an undirected graph G = (V, E) with edge costs ce ≥ 0 for all e ∈ E, a root vertex r ∈ V , a set of terminals X ⊆ V , and a parameter M > 1. We need to design a network connecting all terminals to the root; for each terminal we specify a path of edges from the terminal to the root. We say that a terminal uses an edge if the edge is on the terminal’s path to the root. To Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
314
Further uses of random sampling and randomized rounding of linear programs
build the paths, we can both buy and rent edges. We can buy edges at cost M ce , and once bought, any terminal can use the edge. We can also rent edges at cost ce , but then we need to pay the rental cost for each terminal using the edge. The goal is to ﬁnd a feasible network that minimizes the total cost. We can formalize this by letting B ⊆ E be the set of edges that are bought, and letting Rt be the set of edges that are rented by terminal t ∈ X. Then ∑ for each t ∈ X, the set of edges B ∪ Rt must contain a path from t to ∑ the root r. Let c(F ) = e∈F ce for any F ⊆ E. Then the total cost of the solution is M c(B) + t∈X c(Rt ). We must ﬁnd edges B to buy and Rt to rent that minimizes this overall cost. We will give a randomized approximation algorithm for the problem that cleverly trades oﬀ the cost of buying versus renting. The sampleandaugment algorithm draws a sample of terminals by marking each terminal t with probability 1/M independently. Let D be the random set of marked terminals. We then ﬁnd a Steiner tree T on the set of terminals D plus the root, and buy the edges of T . To ﬁnd a Steiner tree, we use the 2approximation algorithm of Exercise 2.5 that computes a minimum spanning tree on the metric completion of the graph. We then augment the solution by renting paths from the unmarked terminals to the tree T . To do this, we ﬁnd the shortest path from each unmarked t to the closest vertex in T , and rent these edges. The analysis of the sampleandaugment algorithm begins by observing that the expected cost of buying the edges in the tree T is at most twice the cost of an optimal solution to the rentorbuy problem. Lemma 12.6: E[M c(T )] ≤ 2 OPT . Proof. To prove the lemma, we demonstrate a Steiner tree T ∗ on the set of marked terminals such that the expected cost of buying the edges of T ∗ is at most OPT. Since we are using a 2approximation algorithm to ﬁnd T , the lemma statement then follows. We consider an optimal solution to the problem: let B ∗ be the set of bought edges, and let Rt∗ be the edges rented by terminal t. Consider the edges from B ∗ together with the union of edges of Rt∗ over the marked terminals t. Note that this set of edges certainly contains some Steiner tree T ∗ on the set of marked terminals plus the root. We now want to analyze the cost of buying this set of edges. The essential idea of the analysis is that although we now have to pay M times the cost of the rented edges in each Rt∗ for marked t, since we marked t with probability 1/M , the expected cost of these edges will be the same as the renting cost of the optimal solution. To see this formally, if D is the random set of marked terminals, then E[M c(T ∗ )] ≤ M c(B ∗ ) + E[M
∑
c(Rt∗ )]
t∈D
∗
= M c(B ) + M
∑
c(Rt∗ ) Pr[t ∈ D]
t∈X
∗
= M c(B ) +
∑
c(Rt∗ )
t∈X
= OPT .
To complete the analysis, we show that the expected renting cost is no more than the expected buying cost. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.2
The singlesource rentorbuy problem
[
Lemma 12.7: E
∑
315
] c(Rt ) ≤ E[M c(T )].
t∈X
Proof. To prove this, let us be a bit more precise about the algorithm; then we will alter the algorithm to give an equivalent algorithm, and prove the statement for the equivalent algorithm. Let D be the (random) set of marked terminals. We run Prim’s minimum spanning tree algorithm on the metric completion of the graph, starting with the root r (see Section 2.4 for a discussion of Prim’s algorithm). Prim’s algorithm maintains a set S ⊆ D ∪ {r} of vertices in the spanning tree, and chooses the cheapest edge e that has one endpoint in S and the other in D − S to add next to the spanning tree; the endpoint of e in D − S is then added to S. We now alter the algorithm by not choosing D in advance. Rather, we choose the cheapest edge e with one endpoint in S and the other from the set of all terminals whose marking status has not been determined. Let t be the endpoint of e whose marking status is not determined. At this point, we decide, with probability 1/M whether to mark t. If t is marked, then we add t to D and to S, and add the edge e to the tree. If t is not marked, then we do not add t to D or S, and edge e is not added to the tree. Note that we get the same tree T on D via this process as in the case that the random set D was drawn before running the algorithm. We let βt for a terminal t be a random variable associated with the cost of connecting t to the tree via bought edges; we call it the buying cost. We let βt of a marked terminal t be M times the cost of the edge that ﬁrst connects it to the tree, and we let βt be zero if t is not marked. In our modiﬁed algorithm above, βt is the cost of the edge that connects t to S when t is marked. The total cost of∑the tree is then the sum of the buying costs of all the marked terminals in the tree, so that t∈D βt = M c(T ). In a similar way, we let ρt for a terminal t be a random variable giving the cost of renting edges to connect t to the tree. Now consider a given terminal t at the time we decide whether to mark t or not. Let S be the set of vertices already selected by Prim’s algorithm at this point in time, and let e be the edge chosen by Prim’s algorithm with t as one endpoint and with the other endpoint in S. If we mark t, we buy edge e at cost M ce . If we do not mark t, then we could rent edge e at cost ce , and since all the vertices in S are marked, this will connect t to the root; thus ρt ≤ ce . Hence 1 · M ce = ce , whereas its expected cost of renting the expected buying cost of t is E[βt ] = M 1 edges to connect t to the root is E[ρt ] ≤ (1 − M ) · ce ≤ ce . Observe that this is true no matter what point in the algorithm t is considered. Thus [ ] [ ] [ ] [ ] ∑ ∑ ∑ ∑ E c(Rt ) = E ρt ≤ E βt = E βt = E[M c(T )]. t∈X
t∈X
t∈X
t∈D
The following theorem is then immediate. Theorem 12.8: The sampleandaugment algorithm described above is a randomized 4approximation algorithm for the singlesource rentorbuy problem. Proof. For the solution B = T and Rt computed by the randomized algorithm, we have that [ ] ∑ E M c(T ) + c(Rt ) ≤ 2 · E[M c(T )] ≤ 4 OPT . t∈X
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
316
Further uses of random sampling and randomized rounding of linear programs
In Exercise 12.2, we consider the multicommodity rentorbuy problem, which is an extension of the singlesource rentorbuy problem to multiple sourcesink pairs. We see that the sampleandaugment algorithm also leads to a good approximation algorithm for this problem.
12.3
The Steiner tree problem
The Steiner tree problem provides an excellent example of a problem for which our understanding of its combinatorial structure has worked handinhand with the design and analysis of a linear programmingbased approach to approximation algorithm design. Furthermore, we will combine two techniques already discussed for rounding LP solutions, by relying on an iterative use of randomized rounding. The Steiner tree problem, as discussed in Exercise 2.5, is as follows: given an undirected graph G = (V, E), and a subset of nodes R ⊆ V , along with a nonnegative edge cost ce ≥ 0 for each edge e ∈ E, ﬁnd a minimumcost subset of edges F ⊆ E, such that G = (V, F ) contains a path between each pair of nodes in R. As discussed in that exercise, by considering the metric completion of the graph, we may assume without loss of generality that the input graph G is complete, and that the costs satisfy the triangle inequality. In Section 7.4, we showed that the primaldual method, based on a relatively weak LP formulation, provides a 2approximation algorithm for a more general problem in which the connectivity requirement must be satisﬁed for only speciﬁed pairs of nodes. The nodes in R for which there is a connectivity requirement are traditionally called terminals, whereas the remaining nodes are called Steiner nodes. For the Steiner tree problem, a (minimal) feasible solution corresponds to a tree (a Steiner tree) in which all leaves are nodes in R; furthermore, there is a decomposition of each such Steiner tree into its full components, which will play a critical role in this discussion. A full component of a Steiner tree is a maximal subgraph in which no nonleaf node is a terminal. A Steiner tree and its decomposition into full components is shown in Figure 12.1. It is easy to see that if we start with the optimal Steiner tree, and this decomposition yields full components with node sets V1 , V2 , . . . , Vs , then the full component on Vi must be an optimal Steiner tree for the input on the subgraph induced by Vi , i = 1, . . . , s. Furthermore, it is also easy to see that if we contract the nodes in Vi in this optimal Steiner tree, then the tree resulting from this contraction must also be an optimal Steiner tree for the induced subproblem. Finally, an optimal Steiner tree for G that spans the set of nodes V ′ ⊆ V must be a minimum spanning tree for the input induced by the subset V ′ . This suggests a very natural approach to designing an approximation algorithm (or even an optimization algorithm!): identify a full component to contract, contract that component, and iterate. Indeed, all known approximation algorithms for the Steiner tree problem with constant performance guarantee better than 2 are based on variants of this idea. One natural approach to providing a stronger LP relaxation for the Steiner tree problem, as compared to the one discussed in Section 7.4, is to select one of the terminal nodes (arbitrarily) as a root node r, and to view each undirected edge connecting nodes u and v as two directed edges (u, v) and (v, u), both of cost equal to the cost of the original undirected edge. We then consider a directed network design problem in which we wish to select a subset F of these directed edges such that, for each nonroot node v ∈ R, there is a path from v to r using just edges in F . This gives rise to an integer programming formulation, known as the bidirected cut formulation, in which we require that, for each nonempty subset S ⊆ V − {r} with S ∩ R ̸= ∅, there exists an edge (u, v) ∈ F that crosses this cut, that is, u ∈ S and v ̸∈ S. Let δ + (S) be the set of arcs in which (u, v) ∈ S when u ∈ S and v ∈ / S. Let A be the set of arcs. Then the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.3
The Steiner tree problem
317
r
Figure 12.1: A Steiner tree. The terminals are squares, and the Steiner nodes are circles. The dashed ovals indicate the full components of the Steiner tree. linear programming relaxation of this integer program is ∑ minimize ce ye e∈A
subject to
∑
ye ≥ 1,
∀S ⊆ V − {r}, S ∩ R ̸= ∅,
ye ≥ 0,
∀e ∈ A.
e∈δ + (S)
Although not the most direct proof, one consequence of Exercise 7.5 is that if we consider an input for which V = R, i.e., there are no Steiner nodes, and hence this is really a minimum spanning tree instance, then this LP relaxation of this bidirected cut formulation has integer extreme points, and hence the LP relaxation is the initial integer program. We shall present algorithms for the Steiner tree problem that combine this idea with the notion of the full component decomposition. Again, we select one of the terminals as the root node r. Then, for any Steiner tree, we orient its edges towards the root, and decompose this tree into its full components. Each full component is now a directed graph (as shown in Figure 12.2). Furthermore, if we label each node in the graph with its distance (in terms of the number of edges in the (unique) path) to the root r, for each full component C, we shall call the node with minimum distance label the sink(C) – of course, this sink must be a terminal node, and the induced directed Steiner tree on C is directed toward sink(C), in much the same way that the entire Steiner tree is directed toward the root r. Such directed full components, with their speciﬁed sink nodes, will be the building blocks of our new integer programming formulation. Abusing notation slightly, let C now denote a directed full component with speciﬁed sink node; we let R(C) denote the terminals of C (which includes sink(C)). Let C denote the set of all such directed full components. We introduce a 01 decision variable xC for each such directed full component C ∈ C. Note that c(C) denotes the total cost of the edges in the directed full component C. To specify a Steiner tree, we merely list its directed full components. Of course, we need to formulate a set of constraints that enforces that a given set of directed full components is, in fact, a feasible Steiner tree. We can do this again by cut constraints: for each subset S ⊆ R − {r}, we require that there is a directed full component C included such that sink(C) ̸∈ S, but there exists some other terminal node v ∈ R(C) − {sink(C)} such that v ∈ S. Extending the usual notation in Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
318
Further uses of random sampling and randomized rounding of linear programs
r
Figure 12.2: The Steiner tree of Figure 12.1, in which each full component is directed. The sink of each full component is indicated by a black square. which δ(S) denotes the set of edges that cross the cut S, we introduce the notation that ∆(S) is the set of directed full components that satisfy this cutcrossing condition. Thus, we obtain the following linear programming relaxation of the Steiner tree problem: minimize
∑
c(C)xC
(12.3)
C∈C
subject to
∑
xC ≥ 1,
∀S ⊆ R − {r}, S ̸= ∅,
xC ≥ 0,
∀C ∈ C.
(12.4)
C∈C:C∈∆(S)
This LP reﬂects the beauty of mathematical notation: at a glance, it seems like a perfectly innocuous linear program. But encoded in this notation, we see that there are an exponential number of variables, and an exponential number of constraints. This means that we cannot make direct use of the ellipsoid method in order to solve this linear program in polynomial time. Fortunately, we can limit our attention to a weaker LP, without too great a loss in the quality of the bound that it provides: simply restrict attention to those full components with at most k terminals, where k is some ﬁxed constant. If we do this for integer solutions, we obtain a formulation of the socalled krestricted Steiner tree problem and it has been shown that the optimal solution to this problem has the following strong property. Theorem 12.9: For each input to the Steiner tree problem, and for each ﬁxed integer k, the optimal value of the krestricted Steiner tree problem is within a factor of 1 + ⌊log1 k⌋ of the 2 optimal value for the Steiner tree problem. We give a special case of this theorem as Exercise 12.6. This suggests that we consider a relaxed version of the linear program (12.3), in which we replace C by the set of directed full components with at most k terminals. In fact, without loss of generality, we can be even more restrictive in setting up this linear program. Recall that in an optimal (integer) solution, we know that each full component must be an optimal Steiner tree on the given set of terminals. Similarly, in any optimal fractional solution, we can restrict attention to those variables xC for which the directed full component C is an optimal Steiner tree on that set of terminals (with the speciﬁed sink node). Furthermore, since we have instances obeying the triangle inequality, we can assume without loss of generality that no Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.3
The Steiner tree problem
319
Steiner node has degree 2, since that node can be shortcut from the solution to yield a Steiner tree with one fewer node of no greater cost. Since the average degree of a node in a tree is less than 2 (in fact, exactly 2 − 2/n in an nnode graph), we see that in a full component with k terminals (each of degree 1), there can be at most k − 2 Steiner nodes (each of degree at least 3). Once we specify the terminals and the Steiner nodes for a full component, then an optimal Steiner tree is a minimum spanning tree on this combined set of nodes, and so, with a speciﬁed sink, we know that there are at most kn2k−2 directed full components that need to be considered for the krestricted component linear programming relaxation. Let Ck denote this restricted set of directed full components, and let M = Ck . By applying Theorem 12.9 to each full component in the support of the optimal fractional solution, it is straightforward to obtain the following corollary: if we consider the krestricted LP to be variant of the LP (12.3) in which each occurrence of the set C is replaced by the set Ck , then the two optimal LP values are within a factor of 1 + ⌊log1 k⌋ of each other. Thus, if we 2 can round the optimal solution of the krestricted component LP to yield a Steiner tree, while losing a factor of α, then the resulting approximation algorithm is “nearly” an αapproximation algorithm, in that, for any ﬁxed ϵ > 0, we can obtain an (α + ϵ)approximation algorithm by setting k to a suﬃciently large constant. Of course, we must still argue that the krestricted component linear program is polynomialtime solvable, since there are still an exponential number of constraints. There are a number of approaches that suﬃce; once again, simple minimumcut computations show that either the inequalities corresponding to (12.4) are all satisﬁed, or else identify a violated inequality. We leave the proof that the linear program is polynomialtime solvable to Exercise 12.5. We can solve this LP relaxation, but how do we make use of the optimal fractional solution? We shall combine randomized rounding with an iterative rounding approach. We start with the minimum spanning tree on the graph induced by the set of terminals (which we call the minimum terminal spanning tree), and in each iteration, we randomly select a directed full component in proportion to its fractional value in the optimal fractional solution, contract that component by identifying its terminals, and then iterate. This process continues for a ﬁxed number of iterations, at which point, we settle for the ﬁnal minimum terminal spanning tree to connect the remaining terminals. Since the set of terminals evolves over the execution of the algorithm, we shall let mst(R′ ) denote the cost of the minimum terminal spanning tree when R′ ⊆ R is the terminal set. In a given iteration, we start with a minimum terminal spanning tree T and perform a “contraction” deﬁned by a directed full component C to yield a remaining set of terminals R′ ; the contraction reﬂects the commitment to include the edges in C in the solution, incurring a cost equal to c(C), whereas it also “simpliﬁes” the residual problem providing a savings of dropT (C) = c(T ) − mst(R′ ). The analysis of the algorithm is based on the following lemma that links the value of the fractional solution to these incremental improvements. Lemma 12.10: Let T be a terminal spanning tree and let x be a feasible solution to the directed component cutcovering relaxation (12.3); then ∑ c(T ) ≤ dropT (C)xC . (12.5) C∈C
Before proving this lemma, we ﬁrst show how it provides the key to the analysis of our iterative approximation algorithm. Focus on one iteration of the algorithm; let T denote the Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
320
Further uses of random sampling and randomized rounding of linear programs
minimum terminal spanning tree at the start of the iteration,∑ let x denote the optimal fractional solution to the directed component relaxation, and let Σ = C xC . Let C denote the directed full component selected in this iteration (which occurred with probability xC /Σ), and let T ′ denote the resulting minimum terminal spanning tree. Then, by taking into account the random selection of the component C, we see that E[c(T ′ )] = c(T ) − E[dropT (C)] ∑ = c(T ) − (xC /Σ)dropT (C) C
(
)
1 c(T ) Σ ( ) 1 ≤ 1− · 2 OPT, Σ
≤
1−
where OPT denotes the optimal value for the given Steiner tree input. In fact, we can strengthen this ﬁnal inequality by the following lemma, which relates the cost of the minimum terminal spanning tree to the optimal value of the directed component linear programming relaxation. Lemma 12.11: For any input to the Steiner tree problem, the cost of the minimum terminal spanning tree T is at most twice the cost of the optimal fractional solution x for the directed component linear programming relaxation (12.3). Proof. We ﬁrst transform x into a feasible fractional solution y to the bidirected cut relaxation for the input induced on the terminals R. The cost of y will be at most twice the cost of x, and by the integrality of the bidirected cut relaxation in this case (since then all nodes are terminals), we may conclude that the minimum terminal spanning tree costs no more than y. This completes the proof of the lemma. To construct y, we initially set y = 0, then, in turn, consider each directed component C for which xC > 0. Consider the “doubling” of each edge in the Steiner tree for this component C (ignoring edge directions), to yield an Eulerian graph. This Eulerian tour can then be shortcut to yield a cycle on the terminals of C, R(C). We delete one edge of the cycle (chosen arbitrarily), and then orient the edges of this terminal spanning tree on R(C) towards the root of the component, sink(C). For each of the edges in this directed spanning tree, we increment its current value ye by xC . If we view each component C as providing capacity for xC to ﬂow from each terminal in R(C) to the node sink(C), then we see that we have provided exactly the same additional capacity from each terminal to sink(C) in the shortcut solution that uses only terminal nodes. Since the feasibility of x ensures that for each node there is at least total capacity 1 from that node to the root, so must the modiﬁed construction for y. But this means that y is a feasible fractional solution for the bidirected cut relaxation, and this completes the proof of the lemma. Intuitively, if in each iteration we decrease the cost of a minimum terminal spanning tree by a factor of (1 − 1/Σ), then if we apply the same technique for Σ iterations, we decrease the cost of the minimum terminal spanning tree by a factor that is at most 1/e. Therefore, if consider ℓΣ iterations, we decrease the resulting cost by a factor of (1/e)ℓ . By Lemma 12.11, we know in fact that, if we start with the minimum terminal spanning tree, the end result has expected cost at most 2(1/e)ℓ times the optimal value of the directed component relaxation. However, in each iteration, this decrease is paid for by the connection cost c(C) of the selected component C. ∑ Due to the random selection rule, the expected cost incurred is equal to C (xC /Σ)c(C), which Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.3
The Steiner tree problem
321
is 1/Σ times the cost of the optimal fractional solution x for the current directed component relaxation. One technical convenience of bounding this cost by the optimal fractional value is that this value is nonincreasing over the course of the algorithm (by again taking the capacity installation view of the directed component relaxation as in the proof of Lemma 12.11). Hence after ℓΣ iterations, we incur a total cost that is at most ℓ times the cost of the initial optimal solution to the directed component relaxation. In total, the result after ℓΣ iterations has cost at most (2e−ℓ + ℓ) times this LP value; if we set ℓ so as to minimize this quantity, then ℓ = ln), and we obtain a performance guarantee (and an integrality gap) of 1 + ln 2 ≤ 1.694. This intuition oversimpliﬁes one further technical issue – there is no reason that the variables of the relaxation need sum to Σ in each iteration. This can be avoided by the following simple workaround. We know that Σ ≤ M , the number of variables in the krestricted component LP in which we only have directed full components for the optimal Steiner tree on each set of at most k terminals. We add a dummy full component, corresponding to just the root node, and then can add the constraint that the variables must sum to M , and so we run the algorithm for (ln 2)M iterations, which is a polynomial bound. (It is signiﬁcant to note that by stating the algorithm in this way, we have made it much less eﬃcient, since an overwhelming fraction of the time, we will sample the dummy full component, which results in no contraction but incurs no additional cost.) We turn next to gaining a greater structural understanding of dropT (C), which we will use to prove Lemma 12.10. Let T be a minimum terminal spanning tree, and consider the contraction corresponding to some full component C, and hence we identify the nodes R(C); what happens to the minimum terminal spanning tree? Since R(C) nodes are replaced by 1 node, we need R(C) − 1 fewer edges to connect the resulting terminals. Suppose we ﬁrst just identify two terminal nodes u and v. This is equivalent to having an edge connecting them of cost 0, and so the eﬀect on the minimum terminal spanning tree is to delete from T the maximum cost edge on the (unique) path between u and v in T . More generally, in identifying all of the vertices in R(C), we can again consider the eﬀect of adding an edge of cost 0 between each pair of vertices in R(C). It is easy to see that a new minimum terminal spanning tree T ′ can be formed from a spanning tree on R(C) (of cost 0) plus a subset of the edges in T : if this is not the case and there is a new edge e used in T ′ but not in T (other than the dummy edges of cost 0), then we could do an interchange and replace e by a cheapest edge in T that crosses the cut deﬁned by deleting e from T ′ . Thus, we can deﬁne DropT (C) as the set of R(C) − 1 edges in T that are deleted from the minimum terminal spanning tree by contracting the nodes in R(C), and dropT (C) is the total cost of these edges. Notice also that we could ﬁnd the set DropT (C) by building an auxiliary complete graph with node set corresponding to R(C), where the weight for each edge (u, v) is the maximum cost edge in the path from u to v in T , and then ﬁnding a maximumweight spanning tree in this auxiliary graph. One interpretation is that there is a correspondence between the edges of T in DropT (C) and edges in selected maximumweight spanning tree in this auxiliary graph. We now turn to the proof of Lemma 12.10. Proof of 12.10. At the core of this proof is the integrality of the bidirected cut formulation for the minimum spanning tree problem. The basic steps of this proof are as follows: we construct an undirected multigraph H = (R, F ) with (new) edge costs, and devise a feasible fractional solution y to the bidirected cut formulation of cost equal to the righthand side of (12.5); on the other hand, we show that any spanning tree of H has cost at least c(T ). The integrality of the formulation implies the lemma. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
322
Further uses of random sampling and randomized rounding of linear programs
Consider in turn each directed full component C; each component C will cause us to include certain edges in H, and to adjust the fractional solution y, which is initially equal to 0. For the directed component C, consider the auxiliary (undirected) graph built on R(C) in the construction above. Take the maximumweight spanning tree on R(C) (whose edges correspond to DropT (C)), and include each of these edges in H with cost equal to its weight in the auxiliary graph. Now direct this tree towards the node sink(C) and for each edge e in this directed tree, increment ye by ∑ xC . It is clear that this process generates a fractional solution of total cost exactly equal to C xC dropT (C). We need to establish that y is a feasible fractional solution for the bidirected cut formulation for the multigraph H. The solution x is a feasible fractional solution to the directed component formulation and we can view this as installing, for each component C, capacity xC from each nonsink node in R(C) to the node sink(C); by the maxﬂow mincut theorem, the feasibility of x means that in total, these installations support a ﬂow of 1 from each node in R to the root node r. However, the solution y does this as well; for each component C, we increment y so as to install an additional capacity of xC (through a directed spanning tree) from each nonsink node in R(C) to the node sink(C). Hence, in total, we have installed suﬃcient capacity so that a ﬂow of 1 can be sent from each terminal to the root. Hence, y is a feasible solution to the bidirected cut formulation. Finally, we show that the cost of any spanning tree in H is at least c(T ). It suﬃces to show that if we consider G′ which is the union of H and T , then T is a minimum spanning tree in G′ . A suﬃcient condition for a spanning tree T to be minimumcost is that the cost of each edge (u, v) not in T is at least the maximum cost in the path in T connecting u and v. But note that by our construction, each edge inserted in H has cost exactly equal to that maximum cost, and so T is a minimum spanning tree in G′ . Putting the pieces together, we have now proved the following theorem. Theorem 12.12: The iterated randomized rounding algorithm yields a 1.694approximation algorithm for the Steiner tree problem, and furthermore, the integrality gap of the directed component relaxation is at most 1.694. In fact, a similar framework can be used to prove a signiﬁcantly stronger performance guarantee of 1.5. The key observation is that when one selects a directed full component C and identiﬁes its endpoints, not only does the minimum terminal spanning tree cost decrease, but the cost of the optimal Steiner tree decreases as well, albeit by a factor of (1 − 1/(2M )) instead. Nonetheless, this allows for a somewhat diﬀerent balancing of terms, and yields the stronger performance guarantee. Interestingly, this technique does not prove a stronger upper bound on the integrality gap of the directed component LP.
12.4
Everything at once: ﬁnding a large cut in a dense graph
We now turn to a result which will require most of the tools we developed in Chapter 5: randomized rounding, Chernoﬀ bounds, and random sampling. We will apply these to the maximum cut problem (MAX CUT) introduced in Section 5.1. Recall that in the maximum cut problem, the input is an undirected graph G = (V, E), and nonnegative weights wij ≥ 0 for each edge (i, j) ∈ E, and the goal is to partition the vertex set into two parts, U and W = V −U , so as to maximize the weight of the edges whose two endpoints are in diﬀerent parts, one in U and one in W . In the case wij = 1 for all edges (i, j) ∈ E, we have an unweighted maximum cut problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.4
Everything at once: ﬁnding a large cut in a dense graph
323
In this section, we will show that we can obtain a PTAS for the unweighted maximum cut problem in dense graphs by using a sophisticated combination of the randomization techniques ( ) introduced in the Chapter 5. Recall that a graph is dense if it has at least α n2 edges for some constant α > 0. In Theorem 5.3 of Section 5.1, we gave a simple 12 approximation algorithm for the maximum cut ∑ problem. The analysis shows that the algorithm ﬁnds∑a cut whose expected 1 value is at least 2 (i,j)∈E wij . Thus it must be the case that OPT ≥ 12 (i,j)∈E wij . It follows ( ) that in an unweighted dense graph, we know that OPT ≥ α2 n2 . Recall that in Section 5.12 we introduced a sampling technique for coloring dense 3colorable graphs. We would like to use the same sampling technique for the maximum cut problem on unweighted dense graphs. That is, suppose we can draw a sample of the vertices of the graph and assume that we know whether each vertex of the sample is in U or W for an optimal cut. If the sample size is O(log n) we can enumerate all the possible placements of these vertices in U and W , including the one corresponding to an optimal cut. In the case of trying to color a 3colorable graph, a knowledge of the correct coloring of the sample was enough to infer the coloring of the rest of the graph. What can we do in this case? In the case of coloring, we showed that with high probability, each vertex in the graph had some neighbor in the sample S. Here we will show that by using the sample we can get an estimate for each vertex of how many neighbors are in U in an optimal solution that is accurate to within ±ϵn. Once we have such estimates we can use randomized rounding of a linear program in order to determine which of the remaining vertices should be placed in U . Finally, we use Chernoﬀ bounds to show that the solution obtained by randomized rounding is close to the optimal solution. We draw our sample in a slightly diﬀerent fashion than we did for the 3coloring algorithm. Given a constant c > 0 and a constant ϵ, 0 < ϵ < 1, we draw a multiset S of exactly (c log n)/ϵ2 vertices by choosing vertices at random with replacement. As in the case of 3coloring a graph we can now in polynomial time enumerate all possible ways of splitting the sample set S into two parts. Let us say that xi = 0 if we assign vertex i to U and xi = 1 if we assign vertex i to W . Let x∗ be an optimal solution to the maximum cut problem. Let ui (x)∑ be the number of neighbors of vertex i in U given an assignment x of vertices. Observe that ni=1 ui (x)xi gives the value of cut for the assignment x: when xi = 1 and i ∈ W , there are ui (x) edges from i to vertices in U , so that this sum counts exactly the set of edges in the cut. We can give a reasonably good estimate of ui (x) for all vertices i by calculating the number of neighbors of i that are in S and assigned to U , then scaling by n/S. In other words, if u ˆi (x) is our estimate of the neighbors of i in U , then ∑ n (1 − xj ). u ˆi (x) = S j∈S:(i,j)∈E
Note that we can calculate this estimate given only the values of the xj for j ∈ S. To prove that these estimates are good, we will need the following inequality, known as Hoeﬀding’s inequality. Fact 12.13 (Hoeﬀding’s inequality): Let X1 , X2 , . . . , Xℓ∑ be ℓ independent 01 random variables, not necessarily identically distributed. Then for X = ℓi=1 Xi and µ = E[X], and b > 0, then 2 Pr[X − µ ≥ b] ≤ e−2b /ℓ . We can now prove bounds on the quality of the estimates. Lemma 12.14: With probability 1 − 2n−2c , ui (x) − ϵn ≤ u ˆi (x) ≤ ui (x) + ϵn Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
324
Further uses of random sampling and randomized rounding of linear programs
for any given i ∈ V . Proof. Let N (i) be the neighbors of i in G; that is, N (i) = {j ∈ V : (i, j) ∈ E} . Let Yj be a random variable for the jth vertex in S. If the jth vertex in S is some k ∈ N (i), we let Yj = 1 − xk and let Yj = 0 otherwise. Note that the probability that k ∈ N (i) is N (i)/n since we chose the jth vertex ∑of S randomly with replacement. Then the expected value of Yj given that j ∈ N (i) is N1(i) k∈N (i) (1 − xk ), so that the unconditional expected value of Yj is E[Yj ] =
∑ N (i) 1 1 · (1 − xk ) = ui (x). n N (i) n k∈N (i)
∑S n Thus if Y = j=1 Yj , we have µ = E[Y ] = S ˆi (x) = S Y . If we now apply n ui (x). Note that u the Hoeﬀding inequality with b = ϵS, we have that [ ] S 2 2 Pr Y − ui (x) ≥ ϵS ≤ 2e−2(ϵS) /S = 2e−2ϵ S = 2e−2c ln n = 2n−2c , n S −2c . Multiplying so that S n ui (x) − ϵS ≤ Y ≤ n ui (x) + ϵS with probability at least 1 − 2n the inequalities by n/S, we get the desired result.
Since S is suﬃciently small, we can enumerate all the possible settings of the xi for i ∈ S; one of these will correspond to an optimal solution x∗ , and thus we will have good estimates u ˆi (x∗ ) for this particular setting. We will now turn to showing how to use these estimates to obtain a good cut. Note that in enumerating all possible settings of xi for i ∈ S, we will not know which one corresponds to x∗ . However, we will show that for the setting that corresponds to x∗ , we will produce a cut of size at least (1 − ϵ′ ) OPT, for a speciﬁed ϵ′ > 0. Thus if we return the largest cut produced, we are guaranteed to produce a cut of size at least this large. From now on, we assume that we have estimates u ˆi (x∗ ) for all i such that ui (x∗ ) − ϵn ≤ ∗ ∗ u ˆi (x ) ≤ ui (x )+ϵn. To produce a good cut given these estimates, we use randomized rounding. Consider the following linear program: maximize
n ∑
u ˆi (x∗ )yi
i=1
subject to
∑
(1 − yj ) ≥ u ˆi (x∗ ) − ϵn,
i = 1, . . . , n,
(1 − yj ) ≤ u ˆi (x∗ ) + ϵn,
i = 1, . . . , n,
(12.6)
j:(i,j)∈E
∑
j:(i,j)∈E
0 ≤ yi ≤ 1
i = 1, . . . , n.
∑ Suppose that the variables yi are integer. Then since ui (y) = j:(i,j)∈E (1 − yj ), the constraints enforce that u ˆi (x∗ ) − ϵn ≤ ui (y) ≤ u ˆi (x∗ ) + ϵn. Note then that y ∑ = x∗ is a feasible solution for the linear program. Furthermore, if the objective function were ni=1 ui (x∗ )yi (rather than using the estimates u ˆi (x∗∑ )), then the value of the LP solution with y = x∗ would be OPT, since we earlier observed that ni=1 ui (x∗ )x∗i counts the number of edges in the cut of the assignment x∗ . We show below that by using the known estimates u ˆi (x∗ ) in the objective function instead of ∗ the unknown values ui (x ) the value of this linear program is still nearly OPT. Our algorithm Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.4
Everything at once: ﬁnding a large cut in a dense graph
325
then will be to apply randomized rounding to the solution to the linear program. Using Chernoﬀ bounds, we can then show that the solution obtained is only slightly smaller than that of the LP, and hence almost OPT. Lemma 12.15: The value of the linear programming relaxation (12.6) is at least (1 − 4ϵ α ) OPT. Proof. As observed above,∑the solution y = x∗ is a feasible solution to the linear program. Since the objective function is ni=1 u ˆi (x∗ )yi the solution y = x∗ has value n ∑
u ˆi (x∗ )x∗i ≥
i=1
n ∑
(ui (x∗ ) − ϵn)x∗i
i=1
≥ OPT −ϵn
n ∑
x∗i .
i=1
Since we assume there is at least one(node ) in U in an optimal cut, we know that α n Then since we know that OPT ≥ 2 2 , we have n ∑
∑n
∗ i=1 xi
≤ n−1.
u ˆi (x∗ )x∗i ≥ OPT −ϵn(n − 1)
i=1
( ≥
4ϵ 1− α
) OPT .
We now show that randomized rounding of the linear programming relaxation gives a good solution. Let y ∗ be an optimal solution to the linear program, and let y¯ be the integer solution obtained from y ∗ by randomized rounding. We prove the following theorem. Lemma 12.16: For n suﬃciently large, the randomized rounding of the linear program produces ) ( 13ϵ a solution of value at least 1 − α OPT with probability at least 1 − 2n−c+1 . ∑n Proof. From the discussion, we know y )¯ yi . that the value of the integral solution y ¯ is i=1 ui (¯ ∑ ∗ is close in value to OPT. We’ll ﬁrst show that We know from Lemma 12.15 that ni=1 u ˆi (x∗ )y∑ i ∑ ui (¯ y ) is close in value to u ˆi (x∗ ), and then that ni=1 u ˆi (x∗ )¯ yi is close in value to ni=1 u ˆi (x∗ )yi∗ , so that we prove that the solution y¯ has value that is close to OPT. ∑ First, we show that ui (¯ y ) = j:(i,j)∈E (1 − y¯j ) is close in value to u ˆi (x∗ ). To do this, observe that ∑ E[ui (¯ y )] = E (1 − y¯j ) j:(i,j)∈E
=
∑
(1 − E[¯ yj ])
j:(i,j)∈E
=
∑
(1 − yj∗ ) = ui (y ∗ )
j:(i,j)∈E
√ We now want to show that ui (¯ y ) ≥√(ui (y ∗ ) − (2c ln n)ui (y ∗ )) with high probability via a ∑ ln n Chernoﬀ bound. To do this, set δi = u2ci (y ¯j ), and let Y = j:(i,j)∈E Yj = ∗ ) > 0, let Yj = (1 − y Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
326
Further uses of random sampling and randomized rounding of linear programs
ui (¯ y ), so that µi = E[Y ] = ui (y ∗ ). By applying Corollary 5.28, we obtain that ui (¯ y ) ≥ (1 − δi )ui (y ∗ ) with probability at least n −ui (y ∗ ) uc ln (y ∗ )
1 − e−µi δi /2 ≥ 1 − e 2
i
= 1 − n−c .
Thus with probability at least 1 − n−c+1 , ui (¯ y ) is close to ui (y ∗ ) for all i ∈ V . Then we have that the value of the cut obtained by randomized rounding is n ∑
n ∑
ui (¯ y )¯ yi ≥
(1 − δi )ui (y ∗ )¯ yi
i=1 n ( ∑
i=1
≥
i=1 n ( ∑
≥
ui (y ∗ ) −
√
) (2c ln n)ui (y ∗ ) y¯i
u ˆi (x∗ ) − ϵn −
√
) 2cn ln n y¯i ,
i=1
where the last inequality follows by the linear ∑ programming constraints of (12.6), since ui (y ∗ ) = ∑ ∗ ˆi (x∗ ) − ϵn. Then since ni=1 y¯i ≤ n, we have that j:(i,j)∈E (1 − yj ) ≥ u n ∑ i=1
ui (¯ y )¯ yi ≥
n ∑
√ u ˆi (x∗ )¯ yi − ϵn2 − n 2cn ln n.
(12.7)
i=1
∑n ˆi (x∗ )¯ yi and show that it is close to We would now like to bound the value i=1 u ∑n of the ∗term ∗ ˆi (x )yi , which is just the objective function value OPT. Note that its expected value is i=1 u of the linear program, and hence close to OPT by Lemma 12.15. Let Z = maxi u ˆi (x∗ ). We will show via a Chernoﬀ bound that with high probability, v u n n n ∑ ∑ ∑ u ∗ ∗ ∗ u ˆi (x )¯ yi ≥ u ˆi (x )yi − t2cZ ln n u ˆi (x∗ )yi∗ . i=1
√
∑n2cZ ln n∗ ∗ i=1 ui (x )yi ∗ )y ∗ . Note u ˆ (x i i=1 i
Let δ = 1 ∑n
i=1
i=1
> 0, let Xi = u ˆi (x∗ )¯ yi /Z, and let X =
∑n
i=1 Xi ,
so that µ = E[X] =
that since we have scaled by Z, either Xi = 0 or some value no greater than 1. Then by Corollary 5.28, [ ] n n ∑n 1 ∑ 1 ∑ 2 ∗ ∗ ∗ ∗ ∗ Pr u ˆi (x )¯ yi ≥ (1 − δ) u ˆi (x )yi ≥ 1 − e−δ i=1 uˆi (x )yi /2Z Z Z Z
i=1
i=1
= 1 − n−c . Thus with high probability we have that n ∑ i=1
u ˆi (x∗ )¯ yi ≥ (1 − δ)
n ∑
u ˆi (x∗ )yi∗
i=1
√
) n ∑ 2cZ ln n u ˆi (x∗ )yi∗ = 1 − ∑n ∗ )y ∗ u ˆ (x i i=1 i i=1 v u n n ∑ ∑ u = u ˆi (x∗ )yi∗ − t2cZ ln n u ˆi (x∗ )yi∗ . (
i=1
i=1
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.4
Using Z ≤ n and
Everything at once: ﬁnding a large cut in a dense graph
∑n
ˆi (x∗ )yi∗ i=1 u n ∑
327
≤ n2 , we have that ∗
u ˆi (x )¯ yi ≥
i=1
n ∑
√ u ˆi (x∗ )yi∗ − n 2cn ln n.
i=1
Plugging this into inequality (12.7), we obtain n ∑
ui (¯ y )¯ yi ≥
n ∑
i=1
√ u ˆi (x∗ )yi∗ − 2n 2cn ln n − ϵn2 .
i=1
∑ Recall that ni=1 u ˆi (x∗ )yi∗ is the objective function of the linear program, and its value is at 4ϵ least ∑n (1 − α ) OPT by Lemma 12.15. Thus the value of the randomized rounding solution, y )¯ yi , is i=1 ui (¯ ( ) n ∑ √ 4ϵ ui (¯ y )¯ yi ≥ 1 − OPT −2n 2cn ln n − ϵn2 . α i=1
√ ( ) For n suﬃciently large 2n 2cn ln n ≤ ϵn(n − 1)/4 and ϵn2 ≤ 4ϵ n2 . Recall also that OPT ≥ ( ) α n 2 2 . Thus the value of the solution is at least (
or at least
4ϵ 1− α
) OPT − (
ϵ 8ϵ OPT − OPT, α α
13ϵ 1− α
) OPT .
To recap, our algorithm draws a multiset S of exactly (c log n)/ϵ2 vertices by choosing vertices randomly with replacement. We then enumerate all 2S possible placements of the vertices in S on each side of the cut (in U and W ) by setting xj to either 0 or 1 for each j ∈ S. For each setting x, we get estimates u ˆi (x), which we use in the linear program (12.6), and apply randomized rounding to the solution of the linear program to obtain the cut. Since one of the settings of the x variables corresponds to an optimal cut x∗ , during this iteration of the algorithm, the lemmas above will apply, and we will obtain a near optimal cut. This gives the following theorem. Theorem 12.17: For n suﬃciently large, the algorithm above obtains a cut of value at least −c+1 . (1 − 13ϵ α ) OPT with probability at least 1 − 4n Proof. From Lemma 12.14, we have that ui (x∗ ) − ϵn ≤ u ˆi (x∗ ) ≤ ui (x∗ ) + ϵn for all i ∈ V with −2c+1 −c+1 probability at least 1 − 2n ≥ 1 − 2n when we consider the solution x∗ for our sample S. Given that the estimates hold, we have from Lemma 12.16 that with probability at least 1 − 2n−c+1 that the randomized rounding of the linear program produces a solution of value 13ϵ at least (1 − 13ϵ α ) OPT. Thus the algorithm produces a solution of value at least (1 − α ) OPT with probability at least 1 − 4n−c+1 .
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
328
Further uses of random sampling and randomized rounding of linear programs
Exercises 12.1 Show how to transform any solution to the linear programming relaxation of the uncapacitated facility location problem into a complete solution as deﬁned on page 311, so that whenever x∗ij > 0 then x∗ij = yi∗ . 12.2 In the multicommodity rentorbuy problem, we are given an undirected graph G = (V, E) with edge costs ce ≥ 0 for all e ∈ E, a set of k sourcesink pairs si ti for i = 1, . . . , k, and a parameter M > 1. For each sourcesink pair, we need a path Pi in the solution connecting si to ti . As in the singlesource rentorbuy problem, we can either buy edges e at cost M ce , which then any pair can use, or we can rent edges e at cost ce , but every pair using edge e must pay the rental cost. If we let B be the set of bought edges, and Ri the set of rented edges for pair i, then there must be a path from ∑ si to ti in the set of edges B ∪ Ri for each i, and the cost of this solution is M c(B) + ki=1 c(Ri ). Consider a sampleandaugment algorithm that samples every sourcesink pair with probability 1/M . Let D be the set of sampled pairs. We run the generalized Steiner tree algorithm of Section 7.4 on the demand pairs in D and buy the edges given by the algorithm; let these edges be B. Then for any si ti pair not in D we rent edges on the shortest path from si to ti in which the edges in B are given cost 0. (a) Show that the expected cost of the bought edges B is at most 2 OPT. To analyze the cost of the rented edges, we use a notion of αstrict cost shares for the generalized Steiner tree problem. Suppose we have an instance of the generalized Steiner tree problem with si ti pairs for i ∈ R. We say we have an algorithm returning αstrict cost ∑ shares χi for all i ∈ R, if two conditions are met: ﬁrst, the sum of the cost shares, i∈R χi , is at most the optimum cost of the generalized Steiner tree on the pairs in R; and second, for any i ∈ R, the algorithm running on the instances with sourcesink pairs from R − {i} returns a solution F such that the cost of the shortest path from si to ti , treating edges in F as having cost 0, is at most αχi . (b) Use the idea of cost shares to show that the expected cost of the rented edges is at most α OPT. (Hint: deﬁne a random variable βi to be M χi if i ∈ D and 0 otherwise, and a random variable ρi to be the renting cost of the pair i if i ∈ / D and 0 otherwise. Show that conditioned on the set D − {i}, the expected value of ρi is at most αβi ). It is known that the primaldual generalized Steiner tree algorithm of Section 7.4 can produce 3strict cost shares. (c) Show that the sampleandaugment algorithm given above is a randomized 5approximation algorithm for the multicommodity rentorbuy problem. 12.3 In the unweighted maximum directed cut problem we are given as input a directed graph G = (V, A), and the goal is to partition V into two sets U and W = V − U so as to maximize the total weight of the arcs going from U to W (that is, arcs (i, j) with i ∈ U and j ∈ W ). Suppose that the graph (V, A) is dense; that is, for some constant α > 0, the total number of arcs is at least αn2 . Give a polynomialtime approximation scheme for the unweighted maximum directed cut problem in dense graphs. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.4
Everything at once: ﬁnding a large cut in a dense graph
329
12.4 In this exercise, we revisit the metric asymmetric traveling salesman problem introduced in Exercise 1.3. Recall that we are given as input a complete directed graph G = (V, A) with costs cij ≥ 0 for all arcs (i, j) ∈ A, such that the arc costs obey the triangle inequality: for all i, j, k ∈ V , we have that cij + cjk ≥ cik . The goal is to ﬁnd a tour of minimum cost; that is, a directed cycle that contains each vertex exactly once, such that the sum of the cost of the arcs in the cycle is minimized. As in Exercise 1.3, we will ﬁnd a lowcost, strongly connected Eulerian graph and shortcut this to a tour. Recall that a directed graph is strongly connected if for any pair of vertices i, j ∈ V there is a path from i to j and a path from j to i. A directed graph is Eulerian if it is strongly connected and the indegree of each vertex equals its outdegree. We will show that we can obtain an O(log n)approximation algorithm for the problem via randomized rounding. We start by giving a linear programming relaxation of the problem. For each arc (i, j) in the input graph, we introduce a variable xij . For a subset of vertices, S ⊆ V , let δ + (S) be all arcs that have their tail in S and their head not in S, and let δ − (S) be all arcs that have their head in S and their tail not in S. For simplicity, we let δ + (v) = δ + ({v}) and δ − (v) = δ − ({v}). Then consider the following linear program: minimize
∑
cij xij
(i,j)∈A
subject to
∑
xij = 1,
∀v ∈ V,
xij = 1,
∀v ∈ V,
xij ≥ 1,
∀S ⊂ V, S ̸= ∅,
xij ≥ 0,
∀(i, j) ∈ A.
(i,j)∈δ + (v)
∑
(i,j)∈δ − (v)
∑
(i,j)∈δ + (S)
For notational simplicity, ∑ given a solution x to the LP and a set F ⊆ A, we will sometimes write x(F ) to denote (i,j)∈F xij . Our algorithm is as follows. We obtain a solution x∗ to the linear program. For an appropriate choice of constant C, we make K = C ln n copies of each arc (i, j), and then we apply randomized rounding to the resulting graph, including arc (i, j) with probability x∗ij . Let zij be the number of copies of arc (i, j) in the resulting solution; note that z is a random variable. We note that z may not correspond to an Eulerian graph, so we must include additional arcs to make it Eulerian. Let bv = z(δ + (v)) − z(δ − (v)) be the number of additional arcs coming in to vertex v needed to make the graph Eulerian (note that if bv is negative we need to add bv  arcs coming out of vertex v); we call bv the demand of vertex v. We use a minimumcost ﬂow algorithm to ﬁnd ∑ an integral vector w ≥ 0 with − + w(δ (v)) − w(δ (v)) = bv for all v ∈ V that minimizes (i,j)∈V cij wij . (a) Show that the linear program above is a linear programming relaxation of the asymmetric traveling salesman problem. (b) Show that by choosing C and ϵ properly, with high probability (1 − ϵ)x∗ (δ + (S)) ≤ z(δ + (S)) ≤ (1 + ϵ)x∗ (δ + (S)), Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
330
Further uses of random sampling and randomized rounding of linear programs
for all S ⊂ V , S ̸= ∅, and use this to show that with high probability z(δ + (S)) ≤ 2z(δ − (S)) for all S ⊂ V , S ̸= ∅. (Hint: in an undirected graph with capacities on the edges, if λ is the capacity of the minimum cut, then there are at most n2α cuts of capacity at most αλ, where n = V . To apply this result, you will need to argue that in some sense the solution x∗ is like a capacitated undirected graph). (c) It is known that in a directed graph with capacities uij for all (i, ∑j) ∈ A and demands − (S)) ≥ bv , there is a feasible ﬂow satisfying all demands if u(δ v∈S bv for all S ⊂ V , ∑ S ̸= ∅. Prove that z(δ −∑ (S)) ≥ v∈S bv for all S ⊂ V , S = ̸ ∅, and thus that for the ∑ minimumcost ﬂow w, (i,j)∈A cij wij ≤ (i,j)∈A cij zij . (d) Show that the algorithm is a randomized O(log n)approximation algorithm for the metric asymmetric traveling salesman problem. 12.5 Give a polynomialtime separation oracle for the directed component linear programming relaxation of the krestricted Steiner tree problem (and hence a polynomialtime procedure to solve this linear program). 12.6 Consider the special case of Theorem 12.9 when the optimal Steiner tree is a complete binary tree, in which all terminals are leaves, and k = 2p for some integer p (in fact, k = 4 provides a good starting point); in this case, one can prove that the additional cost for small components is at most a factor of 1 + 1/p. One approach is to produce many krestricted Steiner trees from the unrestricted Steiner tree, and prove that the average cost among them has cost within the same factor 1 + 1/p of the original.
Chapter Notes The approximation algorithms for the uncapacitated facility location problem in Section 12.1 are due to Chudak and Shmoys [77]. As of this writing, the best known approximation algorithm for this problem is a 1.5approximation algorithm due to Byrka and Aardal [58]. The algorithm for the singlesource rentorbuy problem in Section 12.2 is due to Gupta, Kumar, and Roughgarden [149] (see also [148]). Williamson and van Zuylen [289] have given a derandomization of this algorithm. An algorithm that marks vertices with a slightly diﬀerent probability can be shown to given an improved performance guarantee; this result is due to Eisenbrand, Grandoni, Rothvoß, and Sch¨afer [98]. The sampleandaugment algorithm for the multicommodity rentorbuy problem in Exercise 12.2 is due to Gupta, Kumar, P´al, and Roughgarden [148]. The fact that the primaldual algorithm for the generalized Steiner tree problem gives 3strict cost shares is due to Fleischer, K¨onemann, Leonardi, and Sch¨afer [115]. The result of Section 12.3 that gives a 1.694approximation algorithm for the Steiner tree is due to Byrka, Grandoni, Rothvoß, and Sanit`a [59]. The ﬁrst approximation algorithm for the Steiner tree, giving a performance guarantee of 2, is attributed to Moore and described in the 1968 paper of Gilbert and Pollak [130]. Zelikovsky [294] gave the ﬁrst αapproximation algorithm for the Steiner tree problem with constant α < 2. In many ways, it is completely analogous to the LPbased result discussed in this section. It focuses on the 3restricted Steiner tree problem, and gives a greedybased improvement approximation algorithm for that problem by starting with the minimum terminal spanning tree; it attempts to ﬁnd the 3node subgraph for which adding a Steiner node causes the greatest drop in net cost, and consequently introduces Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
12.4
Everything at once: ﬁnding a large cut in a dense graph
331
that full component (if there is a net drop at all). This yields an 11/6approximation algorithm. Since that result, there were many improvements, ﬁrst by generalizing to krestricted Steiner tree, and using that as the basis for the approximation; Theorem 12.9 is due to Borchers and Du [53]. The best performance guarantee for a combinatorial algorithm is a 1.55approximation algorithm due to Robins and Zelikovsky [254], which is a greedy algorithm. The monograph by Pr¨omel and Steger [242] provides an excellent overview of the full range of work done on the Steiner tree as of its writing. Beyond the result presented here, Byrka et al. also prove a number of stronger results, most notably a ln 4approximation algorithm (where ln 4 is less than 1.39). Our presentation was also strongly inﬂuenced by a more recent 1.55approximation algorithm based on the same LP due to Chakrabarty, K¨onemann, and Pritchard [63], which matches the currently strongest integrality gap result known in this domain. The result of Section 12.4 giving a polynomialtime approximation scheme for the maximum cut problem in unweighted dense graphs is due to Arora, Karger, and Karpinski [16]; Fernandez de la Vega [110] independently developed a diﬀerent PTAS for the same case of the maximum cut problem. The Hoeﬀding inequality of Fact 12.13 is due to Hoeﬀding [166]. Arora, Karger, and Karpinski show that their techniques work for several other problems, such as the maximum directed cut problem in unweighted dense graphs, as given in Exercise 12.3. Since these results, several other approximation schemes have been developed for these problems and more general variants. Mathieu and Schudy [223] give a particularly simple algorithm for the maximum cut problem in unweighted dense graphs that draws a random sample, enumerates all possible cuts of the sample, and uses a greedy algorithm to augment it to a full solution. Exercise 12.4 is due to Goemans, Harvey, Jain, and Singh [136]; an improvement of this result due to Asadpour, Goemans, Madry, Oveis Gharan, and Saberi [25] gives a performance , guarantee of O(log n/ log log n). The bound on the number of cuts of capacity at most α times the minimum is due to Karger [183, 184]. The condition on the feasibility of a ﬂow is known as Hoﬀman’s circulation theorem and is due to Hoﬀman [167].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
332
Further uses of random sampling and randomized rounding of linear programs
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
C h a p t e r 13
Further uses of randomized rounding of semideﬁnite programs
We introduced the use of semideﬁnite programming for approximation algorithms in Chapter 6. The algorithms of that chapter solve a vector programming relaxation, then choose a random hyperplane (or possibly many hyperplanes) to partition the vectors in some way. The central component of the analysis of these algorithms is Lemma 6.7, which says that the probability of two vectors being separated by a random hyperplane is proportional to the angle between them. In this chapter, we look at ways in which we can broaden both the analysis of algorithms using semideﬁnite programming, and the algorithms themselves. To broaden our analytical techniques, we revisit two of the problems we discussed initially in Chapter 6. In particular, we consider the problem of approximating integer quadratic programs, which was introduced in Section 6.3, and the problem of coloring a 3colorable graph, which was introduced in Section 6.5. In our algorithms in this chapter, we again solve vector programming relaxations of the problems, and choose a random hyperplane by drawing its components from the normal distribution. Here, however, our analysis of the algorithms will rely on several more properties of the normal distribution than we used in the previous chapter; in particular, it will be helpful for us to use bounds on the tail of the normal distribution. We will also consider the application of semideﬁnite programming to the unique games problem. We have mentioned previously that several problems are hard to approximate assuming the unique games conjecture. In Section 13.3, we will deﬁne the unique games problem and give the unique games conjecture. The unique games problem is a type of constraint satisfaction problem, and the unique games conjecture states that it is NPhard to satisfy a small fraction of the constraints even when it is known that the optimal solution satisﬁes almost all constraints. The unique games conjecture underlies various results on the hardness of approximation. We will give an approximation algorithm for the problem, though its performance guarantee does not refute the unique games conjecture. Interestingly, the algorithm for unique games does not use a random hyperplane, but instead relies on the geometric properties of vector programs. 333
334
Further uses of randomized rounding of semideﬁnite programs
13.1
Approximating quadratic programs
We begin the chapter by returning to the quadratic programming problem introduced in Section 6.3; recall the quadratic program (6.5):
maximize
∑
aij xi xj
(13.1)
1≤i,j≤n
xi ∈ {−1, +1} ,
subject to
i = 1, . . . , n.
In Section 6.3, we restricted the objective function matrix A = (aij ) to be positive semideﬁnite in order to guarantee that the optimal value is nonnegative. We were then able to obtain a π2 approximation algorithm for that case. In this section, we instead restrict ourselves to any matrix A in which aii = 0 for all i, and obtain a much weaker approximation algorithm with performance ∑ ∑n guarantee Ω(1/ log n). In any solution to the quadratic program above, n 2 = a x i=1 ii i i=1 aii , so these terms are simply a constant that is added to any solution to the problem. We begin by showing that the optimal value for this case must be nonnegative, and thus it makes sense to talk about an approximation algorithm for this problem. Lemma 13.1: If aii = 0 for all i, then OPT ≥
1 n2
∑
aij + aji .
1≤i<j≤n
¯ via randomization such that Proof. We will construct a solution x E
∑ 1≤i,j≤n
aij x ¯i x ¯j ≥
1 n2
∑
aij + aji .
1≤i<j≤n
This will prove the lemma. Consider a complete undirected graph on n vertices, where each edge (i, j) for i < j has weight (aij + aji ). We construct a random matching M on the graph as follows: we pick an edge (i, j) of the graph at random, add it to the matching M , remove both of its endpoints i and j from the graph, and repeat until there is at most one vertex remaining. Since there are n(n − 1)/2 edges in the graph, the probability that we choose any given edge for the matching in the ﬁrst step is 2/n(n − 1). Thus the probability that any edge ends up in the matching is at least 1/n2 ; this is a very loose lower bound, but it is suﬃcient for our purposes. We now construct the solution x ¯. Given the matching M , for each (i, j) ∈ M with i < j, 1 we set x ¯i = 1 with probability 2 and x ¯i = −1 with probability 21 . We then set x ¯j = x ¯i if aij + aji ≥ 0 and x ¯j = −¯ xi otherwise. Finally, for vertex i not in the matching (if n is odd), then we set x ¯i = 1 with probability 21 and x ¯i = −1 with probability 12 . The crucial observation is that if (i, j) ∈ M , E[(aij + aji )¯ xi x ¯j ] = aij + aji , while if (i, j) ∈ / M , then E[(aij + aji )¯ xi x ¯j ] = 0. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.1
Approximating quadratic programs
335
Hence we obtain that ∑ ∑ E aij x ¯i x ¯j = E (aij + aji )¯ xi x ¯j 1≤i,j≤n
1≤i<j≤n
∑
=
Pr[(i, j) ∈ M ]E[(aij + aji )¯ xi x ¯j (i, j) ∈ M ]
1≤i<j≤n:(i,j)∈M
∑
+
Pr[(i, j) ∈ / M ]E[(aij + aji )¯ xi x ¯j (i, j) ∈ / M]
1≤i<j≤n:(i,j)∈M /
≥
1 n2
∑
aij + aji ,
1≤i<j≤n
and the proof is complete. As a ﬁrst step towards an algorithm, we show that the quadratic programming problem with integer constraints, in which we must have xi ∈ {−1, 1}, is actually equivalent, in terms of approximability, to a quadratic programming problem with linear constraints, in which we have −1 ≤ xi ≤ 1. We will then provide an approximation algorithm for the case of linear constraints. In particular, consider the following program: ∑ maximize aij yi yj (13.2) 1≤i,j≤n
−1 ≤ yi ≤ 1,
subject to
i = 1, . . . , n.
We can show the following. Lemma 13.2: Assume that aii = 0 for all i. Then given any αapproximation algorithm for the program (13.2) with linear constraints, we can obtain a randomized αapproximation algorithm for the program (13.1) with integer constraints. Proof. We prove the lemma by showing that any solution y¯ to the program (13.2) with linear constraints can be converted via randomized rounding to a solution x ¯ to the program (13.1) with integer constraints of the same expected value. To see this, set x ¯i = −1 with probability 1 ¯i ) and x ¯i = 1 with probability 12 (1 + y¯i ). Then for i ̸= j 2 (1 − y E[¯ xi x ¯j ] = Pr[¯ xi = x ¯j ] − Pr[¯ xi ̸= x ¯j ] 1 1 = ((1 − y¯i )(1 − y¯j ) + (1 + y¯i )(1 + y¯j )) − ((1 − y¯i )(1 + y¯j ) + (1 + y¯i )(1 − y¯j )) 4 4 1 1 (2 + 2¯ yi y¯j ) − (2 − 2¯ yi y¯j ) = 4 4 = y¯i y¯j , so that
E
∑
1≤i,j≤n
aij x ¯i x ¯j =
∑
aij y¯i y¯j ,
1≤i,j≤n
given that aii = 0 for all i. Let OPTlin be the value of the optimal solution to the program (13.2) with linear constraints, and let OPT be the value of the optimal solution to the original program (13.1) with integer Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
336
Further uses of randomized rounding of semideﬁnite programs
Solve vector programming (13.3), obtain vectors vi Draw random vector r Set zi = (vi · r)/T if zi  ≤ 1 then yi ← zi else if zi < −1 then yi ← −1 else yi ← 1 return y Algorithm 13.1: Approximation algorithm for the quadratic programming problem. constraints. Note that OPTlin ≥ OPT, since every integer solution is feasible for the program with linear constraints. However, it is also true that OPT ≥ OPTlin , since given an optimal solution y¯ to the program with linear constraints, the argument above implies there exists an integer solution of value at least as much. Hence OPT = OPTlin . Thus given any solution y¯ to the program (13.2) with linear constraints of value at least α OPTlin , we can convert it in randomized polynomial time to an integer solution x ¯ to the program (13.1) of the same expected value, and thus of value at least α OPTlin = α OPT. Therefore, given an αapproximation algorithm for the program (13.2), we can obtain a randomized αapproximation algorithm for the program (13.1). We now give an Ω(1/ log n)approximation algorithm for the program (13.2), which, by Lemma 13.2, will imply an Ω(1/ log n)approximation algorithm for the original problem. We now let OPT stand for the optimal value of the program (13.2) with linear constraints (the proof of the lemma shows that the two values are identical). We use the same vector programming relaxation as we used in Section 6.3: ∑ maximize aij (vi · vj ) (13.3) 1≤i,j≤n
subject to
vi · vi = 1, vi ∈ ℜ , n
i = 1, . . . , n, i = 1, . . . , n.
Let ZV P be the optimal value of the vector program. As we argued before, this program is a relaxation of the original program (13.1) with integer constraints, so that ZV P ≥ OPT. Our algorithm ﬁnds an optimal solution to the vector program (13.3) in polynomial time and obtains the vectors vi . We ﬁnd a random vector r, as usual, by drawing each component of r from N (0, 1), the normal distribution of mean 0 and variance 1. For a value of T ≥ 1 to be chosen later, we let zi = (vi · r)/T . Possibly z is infeasible if zi > 1 or zi < −1, so we create a solution y in which we clamp yi to 1 (or −1) if that happens; in other words, we set zi if zi  ≤ 1 −1 if zi < −1 yi = 1 if zi > 1. We return y as our solution. This algorithm is summarized in Algorithm 13.1. We will be able to show that the expected value of the product zi zj is the same as the inner product vi · vj scaled down by T 2 . If it were the case that zi  ≤ 1 for all i, then we could show Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.1
Approximating quadratic programs
337
that the expected value of the solution y would be ZV P /T 2 . However, possibly zi  > 1. By increasing T , we decrease the probability that this happens; we can show that the√expected 2 error incurred by the possibility that zi  > 1 is O(n2 e−T OPT). By setting T = Θ( ln n), we get a solution of expected value at least ZV P /Θ(ln n) ≥ OPT /Θ(ln n) minus an error term of O(OPT /n), which gives the desired performance guarantee for suﬃciently large n. We now prove the main lemmas we will need to obtain the result. Lemma 13.3: 1 (vi · vj ). T2
E[zi zj ] =
Proof. We want to calculate E[zi zj ] = T12 E[(vi · r)(vj · r)]. The value of the vector program is not changed under a rotation of the vectors and r is spherically symmetric (by Fact 6.4), so we can rotate the vectors so that vi = (1, 0, . . .) and vj = (a, b, 0, . . .). Then for r = (r1 , r2 , . . . , rn ), where each ri is drawn from N (0, 1), we have that vi · r = r1 and vj · r = ar1 + br2 . Therefore E[zi zj ] =
) 1 ( 1 2 E[r (ar + br )] = aE[r ] + bE[r r ] . 1 1 2 1 2 1 T2 T2
Because r1 is drawn from N (0, 1), E[r12 ] is the variance of r1 and is 1. Because r1 and r2 are drawn independently from N (0, 1), E[r1 r2 ] = E[r1 ]E[r2 ] = 0. Thus E[zi zj ] =
a 1 = 2 (vi · vj ). 2 T T
The expected value of the algorithm is E
∑
∑
aij yi yj =
1≤i,j≤n
aij E[yi yj ],
1≤i,j≤n
whereas the lemma above shows us that ∑ 1≤i,j≤n
aij E[zi zj ] =
1 T2
∑
aij (vi · vj ) =
1≤i,j≤n
1 ZV P . T2
In order to relate the expected value of the algorithm to the value of the semideﬁnite program, we consider the diﬀerence between zi zj and yi yj for each i, j. Let us denote this diﬀerence ∆ij = zi zj − yi yj . In the next lemma, we see that this diﬀerence is exponentially small in −T 2 . Lemma 13.4: E[∆ij ] ≤ 8e−T . 2
We defer the proof of this lemma for a moment. Given the lemma, we are able to obtain the proof of the performance guarantee of the algorithm. Theorem 13.5: For n suﬃciently large, Algorithm 13.1 is a randomized Ω(1/ log n)approximation algorithm for approximating the quadratic program (13.2) if aii = 0 for all i. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
338
Further uses of randomized rounding of semideﬁnite programs
Proof. Via Lemma 13.3, we have that E[zi zj ] = T12 (vi · vj ). Then the expected value of the algorithm is ∑ ∑ E aij yi yj = aij E[yi yj ] 1≤i,j≤n
1≤i,j≤n
=
∑
aij E[zi zj ] −
1≤i,j≤n
= =
≥ ≥ ≥
1 T2
∑
aij E[∆ij ]
1≤i,j≤n
∑
∑
aij (vi · vj ) −
1≤i,j≤n
aij E[∆ij ]
1≤i,j≤n
∑ 1 aij E[∆ij ] Z − V P T2 1≤i,j≤n ∑ 1 ZV P − aij E[∆ij ] 2 T 1≤i,j≤n ∑ 1 ZV P − aij + aji  · E[∆ij ] T2 1≤i<j≤n ∑ 1 −T 2 Z − 8e aij + aji , V P T2 1≤i<j≤n
where the last inequality follows from Lemma 13.4. Using Lemma 13.1, we know that aji  ≤ n2 · OPT, so that the expected value of the algorithm is at least 1 2 ZV P − 8n2 e−T OPT ≥ T2 Then if we set T =
√
(
∑
1≤i<j≤n aij +
) 1 2 −T 2 − 8n e OPT . T2
3 ln n, the expected value of the algorithm is at least ) ( 8 1 − OPT . 3 ln n n
For n larger than e8 ≥ 128, we have that 1/4 ln n ≥ 8/n, so that the expected value of the OPT algorithm is at least ( 31 − 14 ) OPT ln n = 12 ln n . This proves that the algorithm is an Ω(1/ log n)approximation algorithm for n suﬃciently large. Finally, we turn to the proof of Lemma 13.4. Proof of Lemma 13.4. Let Xi be the event that yi = zi and Xj be the event that yj = zj . For notational simplicity, we will write expectations conditioned on Xi as Ei , expectations ¯ i as E¬i , expectations conditioned on Xi ∧ Xj as Ei,j , and so on. Then conditioned on X ¯ i ∧ Xj ] E[∆ij ] ≤ Ei,j [∆ij ] Pr[Xi ∧ Xj ] + E¬i,j [∆ij ] Pr[X ¯ j ] + E¬i,¬j [∆ij ] Pr[X ¯i ∧ X ¯ j ]. + Ei,¬j [∆ij ] Pr[Xi ∧ X
(13.4)
Observe that ¯ i ] = E¬i,j [∆ij ] Pr[X ¯ i ∧ Xj ] + E¬i,¬j [∆ij ] Pr[X ¯i ∧ X ¯j ] E¬i [∆ij ] Pr[X Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.1
Approximating quadratic programs
339
and ¯ j ] = Ei,¬j [∆ij ] Pr[Xi ∧ X ¯ j ] + E¬i,¬j [∆ij ] Pr[X ¯i ∧ X ¯ j ]. E¬j [∆ij ] Pr[X ¯i ∧ X ¯ j ] is nonnegative, the righthand side of (13.4) is at most Thus since E¬i,¬j [∆ij ] Pr[X ¯ i ] + E¬j [∆ij ] Pr[X ¯ j ]. Ei,j [∆ij ] Pr[Xi ∧ Xj ] + E¬i [∆ij ] Pr[X We now bound these various terms. Given Xi ∧ Xj (that is, yi = zi and yj = zj ), ∆ij = 0, ¯ i ] = E¬j [∆ij ] Pr[X ¯ j ], so we only need to so Ei,j [∆ij ] = 0. By symmetry, E¬i [∆ij ] Pr[X ¯ bound E¬i [∆ij ] Pr[Xi ]. To do this, we recall that the density function p(x) of the normal distribution N (0, 1) is 1 2 p(x) = √ e−x /2 , 2π ∫x and its cumulative distribution function is Φ(x) = −∞ p(s)ds. We will let Φ(x) = 1 − Φ(x) = ∫∞ x p(s)ds. We also recall from the proof of Lemma 13.3 that we can assume that vi = (1, 0, . . .), vj = (a, b, 0, . . .), where a ≤ 1 and b ≤ 1 since vj is a unit vector. Further, r = (r1 , r2 , . . . , rn ) ¯ i occurs when zi  > 1, where each ri is drawn independently from N (0, 1). Since the event X or vi · r > T , we have ¯ i ] = 2Φ(T ). Pr[X Also, using yi yj  ≤ 1, ¯ i ] ≤ E¬i [yi yj  + zi zj ] Pr[X ¯ i ] ≤ 2Φ(T ) + E¬i [zi zj ]. E¬i [∆ij ] Pr[X ¯ i implies vi · r > T , we have that · r)(vj · r) = r1 (ar1 + br2 ), and X ∫ −T ∫ ∞ 1 r1 (ar1 + br2 )p(r1 )p(r2 )dr2 dr1 E¬i [zi zj ] = T 2 −∞ −∞ ∫ ∞∫ ∞ 1 + 2 r1 (ar1 + br2 )p(r1 )p(r2 )dr2 dr1 T T −∞ ∫ ∞∫ ∞ 2 = ar12 + br1 r2 p(r1 )p(r2 )dr2 dr1 T2 T −∞ (∫ ∞ ) (∫ ∞ ) ∫ ∞ 2 2 2 ≤ ar p(r )dr + br p(r )dr r p(r )dr 1 1 1 1 1 2 2 2 . 1 T2 T T2 T −∞
Since zi zj =
1 (v T2 i
We now bound each of these terms. Using integration by parts, and a ≤ 1 we obtain ∫ ∞ ∫ ∞ 1 2 ar2 p(r)dr ≤ √ r2 e−r /2 dr 2π T T ( ) ]∞ ∫ ∞ 1 −r 2 /2 −r2 /2 = √ −re + e dr T 2π T 1 2 = √ T e−T /2 + Φ(T ). 2π Also, since b ≤ 1, ] ∫ ∞ ∫ ∞ 1 1 1 −r2 /2 ∞ 2 −r2 /2 = √ e−T /2 , brp(r)dr ≤ √ re dr = − √ e 2π T 2π 2π T T and
∫
∞
−∞
∫ rp(r)dr = 2 0
∞
2 2 rp(r)dr = − √ e−r /2 2π
]∞ 0
2 =√ . 2π
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
340
Further uses of randomized rounding of semideﬁnite programs
Putting these pieces together, and using T ≥ 1, we get 2 2 2 2 2 √ e−T /2 + 2 Φ(T ) + 2 e−T /2 T T π T 2π 2 2 ≤ 4Φ(T ) + e−T /2 . T
¯ i ] ≤ 2Φ(T ) + E¬i [∆ij ] Pr[X
We can bound Φ(T ) for T ≥ 1 with ] ∫ ∞ ∫ ∞ ∫ ∞ 1 −x2 /2 ∞ 1 −T 2 /2 1 −x2 /2 xe = −√ e . Φ(T ) = p(x)dx ≤ xp(x)dx = √ ≤ e 2 2π T 2π T T T Now we can put everything together. We have that ¯ i ] ≤ 4e−T 2 /2 + 4 e−T 2 /2 ≤ 8e−T 2 /2 , E[∆ij ] ≤ 2E¬i [∆ij ] Pr[X T and we are done. Before we move on, it is worth pausing for a moment to reﬂect on why the analysis above is necessary. Consider an alternative: suppose we solve the vector program and pick a random hyperplane r; if zi = (vi · r)/T is such that zi  ≤ 1 for all i, return zi as a solution, otherwise return any nonnegative solution as given in Lemma 13.1. The probability that for a √ given i, 2 zi  > 1 is the probability that vi · r > T , which is 2Φ(T ) ≤ e−T /2 . If we set T = O( log n), then with high probability, for all i, zi  ≤ 1. We already know that the expected value E[zi zj ] = (vi · vj )/T 2 by Lemma 13.3, so the expected value is within Ω(1/ log n) of ZV P . Why isn’t this easier? The problem with this analysis is that it does not consider the expected value of E[zi zj ] conditioned on zi  ≤ 1 for all i; that is, the expected value of the solutions we actually return. Perhaps the value of solutions for which zi  > 1 is particularly high, and this is what makes the overall expectation large. It is this possibility that the analysis above of the error term ∆ij considers.
13.2
Coloring 3colorable graphs
In this section, we return to the problem discussed in Section 6.5 of coloring 3colorable graphs with as few colors as possible. Our coloring algorithm in this section will work by repeatedly ﬁnding a large independent set in the graph. Recall from Section 10.2 that an independent set of vertices S ⊆ V is one such that for all pairs i, j of vertices in S, there is no edge (i, j). Given an independent set in the graph, we can color all of its vertices the same color. We then remove the independent set from the graph, ﬁnd another large independent set in the remaining graph, color all its vertices with a new color, and repeat. Clearly this process results in a feasible coloring of the graph. Now we bound the number of colors used. If in each iteration we ﬁnd an independent set of size at least γ fraction of the remaining vertices, then after one iteration at most (1 − γ)n vertices remain. After k iterations, (1 − γ)k n vertices remain. Since (1 − γ)k ≤ e−γk , using 1 − x ≤ e−x , after k = γ1 ln n rounds, at most (1 − γ)k n ≤ e−γk n = e− ln n n = 1 vertex remains, which we can color with its own color. Thus if in each iteration we can ﬁnd an independent set of size at least a γ fraction of the remaining number of vertices, then we can color the graph with O( γ1 ln n) colors overall. Let ∆ be the maximum degree of the graph. We Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.2
Coloring 3colorable graphs
341
will modify the algorithm somewhat so that if ∆ ever becomes smaller than a constant in the remaining graph, we simply use the greedy ∆ + 1 coloring algorithm of Exercise 6.4 to color the remaining graph; we state the constant below. This modiﬁcation still yields a O( γ1 ln n) coloring algorithm. To make matters slightly more complicated, the algorithm we give below for ﬁnding an independent set is a randomized algorithm that returns an independent set whose expected size is γn. We will argue below that we can simply run this algorithm O( γ1 ln n) times to get an independent set of size at least γn/2 with high probability. If we let the random variable X denote the number of vertices not in the independent set, then if the expected size of the independent set is at least γn, it follows that E[X] ≤ n(1 − γ). Then by applying Markov’s inequality (Lemma 5.25), [ ( E[X] n(1 − γ) γ γ )] ( Pr X ≥ n 1 − ≤ ( γ) ≤ γ) ≤ 1 − . 2 2 n 1− 2 n 1− 2 Thus the probability that the independent set has size smaller than γn/2 is at most 1 − γ2 , and the probability that the independent set has size at least γn/2 is at least γ2 . Thus if we run the algorithm at least 2c γ ln n times for some constant c, the probability that the algorithm does not return an independent set of size at least γn/2 is at most ( ln n γ ) 2c 1 γ 1− ≤ e−c ln n ≤ c . 2 n To ﬁnd a large independent set, we will use a feasible solution to the following vector program: minimize
0
(13.5)
subject to vi · vj = −1/2, vi · vi = 1, vi ∈ ℜ , n
∀(i, j) ∈ E, ∀i ∈ V, ∀i ∈ V.
We use the objective function “minimize 0” since we are only interested in a feasible solution to the vector program. In Corollary 6.22, we showed that for any 3colorable graph, there exists a feasible solution to this vector program. Given a feasible solution to the vector program, we choose a random vector r = (r1 , r2 , . . . , rn ) by drawing each ri independently from the standard normal distribution N (0, 1). For a value of ϵ we specify momentarily, we ﬁnd S(ϵ) = {i ∈ V : vi · r ≥ ϵ}. The set S(ϵ) may not be independent, so we let S ′ (ϵ) ⊆ S(ϵ) be all the i ∈ S(ϵ) that have no neighbors in S(ϵ). The set S ′ (ϵ) is independent, and we return it as our solution. We summarize this algorithm in Algorithm 13.2. We will show below that this algorithm produces an independent set of expected size at least Ω(n∆−1/3 (ln ∆)−1/2 ). In particular, we √ will choose ϵ =
2 3
ln ∆ for reasons explained later, and we would like ϵ ≥ 1, so we use the
greedy coloring algorithm when ∆ ≤ e3/2 ≤ 5. Together with the previous discussion, this √ will yield an algorithm that colors any 3colorable graph with O(∆1/3 ln ∆ log n) colors. In ˜ 1/4 ) colors. Exercise 13.1, this is shown to lead to a coloring algorithm that uses O(n For our proofs we will need the following notation. As in the previous section, let the density function p(x) of the normal distribution N (0, 1) be 1 2 p(x) = √ e−x /2 , 2π Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
342
Further uses of randomized rounding of semideﬁnite programs
Solve vector program (13.5), get vectors vi Draw random vector r √ ϵ ← 23 ln ∆ S(ϵ) ← {i ∈ V : vi · r ≥ ϵ} S ′ (ϵ) ← {i ∈ S(ϵ) : ∀(i, j) ∈ E, j ∈ / S(ϵ)} return S ′ (ϵ) Algorithm 13.2: Algorithm for ﬁnding a large independent set in a 3colorable graph. ∫x its cumulative distribution function be Φ(x) = −∞ p(s)ds, and let Φ(x) = 1 − Φ(x) = ∫∞ x p(s)ds. We can now start with a simple lemma. Lemma 13.6: For any i ∈ V , the probability that i ∈ S(ϵ) is Φ(ϵ), so that E[S(ϵ)] = nΦ(ϵ). Proof. The probability that any i ∈ V is in S(ϵ) is equal to the probability that vi · r ≥ ϵ. By Fact 6.5, vi · r is normally distributed. By the deﬁnition of Φ, the probability that vi · r is at least ϵ is Φ(ϵ). Now we bound the probability that any vertex is not in S ′ (ϵ) given that it is in S(ϵ). √ Lemma 13.7: The probability that i ∈ / S ′ (ϵ) given that i ∈ S(ϵ) is at most ∆Φ( 3ϵ). Proof. If i ∈ S(ϵ), then the only reason that i would not be in S ′ (ϵ) is because some neighbor j of i is also in S(ϵ). Thus Pr[i ∈ / S ′ (ϵ)i ∈ S(ϵ)] = Pr[∃(i, j) ∈ E : vj · r ≥ ϵvi · r ≥ ϵ]. Because i and j are neighbors, there is an edge (i,√j) ∈ E. Thus by the vector program (13.5), vi · vj = −1/2. We can write that vj = − 21 vi + 23 u for a unit vector u orthogonal to vi in the vi vj plane (see Figure 13.1). Rewriting in terms of u, we have u = √23 ( 12 vi + vj ). Then √ if vi · r ≥ ϵ and vj · r ≥ ϵ, this implies that u · r ≥ √23 (ϵ + 12 ϵ) = 3ϵ. Since u is orthogonal to vi , by Fact 6.5, u · r is normally distributed and independent of vi · r. Thus given that vi · r ≥ ϵ, the probability that vj · r ≥ ϵ is at √ most the probability that the independent, √ normally distributed random variable u · r ≥ 3ϵ, which happens with probability Φ( 3ϵ). √ Thus Pr[vj · r ≥ ϵvi · r ≥ ϵ] ≤ Φ( 3ϵ). Since i has at most ∆ neighbors, we have that Pr[∃(i, j) ∈ E : vj · r ≥ ϵvi · r ≥ ϵ] ≤
∑
√ Pr[vj · r ≥ ϵvi · r ≥ ϵ] ≤ ∆Φ( 3ϵ),
j:(i,j)∈E
and we are done. √ 1 , then the probability that i ∈ / S ′ (ϵ) given Now if we can set ϵ so that Φ( 3ϵ) is at most 2∆ ′ that i ∈ S(ϵ) is at most 1/2, which implies that the expected size of S (ϵ) will be at least half the expected size of S(ϵ), or at least n2 Φ(ϵ). To determine an ϵ that gives this probability and a large independent set, we use the following bounds on Φ. Lemma 13.8: For x > 0, 1 x p(x) ≤ Φ(x) ≤ p(x). 2 1+x x Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.2
Coloring 3colorable graphs
343
u vj
vi
Figure 13.1: Figure for proof of Lemma 13.7. ( )′ ( ) Proof. Note that p′ (s) = −sp(s), so that − 1s p(s) = 1 + s12 p(s). To obtain the lower bound on Φ, observe that ( ) ) ∫ ∞( 1 1 1 + 2 Φ(x) = 1 + 2 p(s)ds x x x ) ∫ ∞( 1 1 + 2 p(s)ds ≥ s x ]∞ 1 1 = p(x). = − p(s) s x x Dividing both sides of the inequality by 1 + x12 gives the lower bound on Φ(x). To obtain the upper bound, observe ∫ ∞ Φ(x) = p(s)ds ) ∫x∞ ( 1 1 ≤ 1 + 2 p(s)ds = p(x) s x x as above. We can now give the following theorem. Theorem 13.9: Algorithm 13.2 produces an independent set of expected size at least Ω(n∆−1/3 (ln ∆)−1/2 ). √
2 Proof. We set ϵ = 3 ln ∆. Recall from the discussion that we only run the algorithm on graphs such that ϵ ≥ 1. By Lemma 13.8, we see that
√ Φ( 3ϵ) ≤ = ≤
1 1 2 √ √ e−3ϵ /2 3ϵ 2π 1 1 √ √ e− ln ∆ 2 ln ∆ 2π 1 . 2∆
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
344
Further uses of randomized rounding of semideﬁnite programs
As argued above, by Lemma 13.7, this implies that the probability that i ∈ / S ′ (ϵ) given that ′ i ∈ S(ϵ) is at most 1/2. Thus the expected size of S (ϵ) is E[S ′ (ϵ)] =
∑ i∈V
Pr[i ∈ S ′ (ϵ)i ∈ S(ϵ)] Pr[i ∈ S(ϵ)] ≥
n Φ(ϵ), 2
using Lemma 13.6. Using Lemma 13.8 and the fact that ϵ ≥ 1, we know that Φ(ϵ) ≥
ϵ 1 1 1 −(ln ∆)/3 2 √ e−ϵ /2 ≥ √ e = Ω((ln ∆)−1/2 ∆−1/3 ). 1 + ϵ2 2π 2ϵ 2π
Thus the expected size of the independent set returned is Ω(n∆−1/3 (ln ∆)−1/2 ). A somewhat improved algorithm is possible using the same semideﬁnite programming relaxation, and a slightly more improved algorithm is possible using a stronger semideﬁnite programming relaxation; see the notes at the end of the chapter for a discussion. However, the algorithms still use O(nc ) colors to color a 3colorable graph, for a constant c. Whether or not it is possible to obtain signiﬁcantly better results (such as O(logc n) colors) remains a large open question.
13.3
Unique games
In this section, we introduce the unique games problem. The unique games problem is a type of constraint satisfaction problem. In a constraint satisfaction problem, we are given n variables x1 , . . . , xn and a ﬁnite universe U of values for the variables. Furthermore, we are given m constraints that are given as functions mapping some subset of the variables to either 0 or 1. Typically, these constraints are drawn from a particular class of kary functions f : U k → {0, 1}. The goal is to ﬁnd a setting of the variables to values in U such that we maximize the number of constraints that are satisﬁed; we say that a constraint is satisﬁed if the corresponding function evaluates to 1 given the setting of the variables. We can also consider weighted versions of this problem in which a nonnegative weight wj ≥ 0 is given as input for the jth constraint, and the goal is to ﬁnd a setting of the variables that maximizes the total weight of the satisﬁed constraints. Several problems that we have seen thus far are kinds of constraint satisfaction problems. The maximum cut problem, studied in Sections 5.1, 6.2 and 12.4, is a weighted constraint satisfaction problem. Given the input graph G = (V, E) with weights wij ≥ 0 for all (i, j) ∈ E, the maximum cut problem corresponds to the weighted constraint satisfaction problem in which U = {0, 1}, there is a variable xi for each vertex i ∈ V , and there is a constraint f (xi , xj ) = xi ⊕ xj for each edge (i, j) with weight wij , where ⊕ is the exclusiveor function (which is 1 precisely when xi ̸= xj ). The maximum satisﬁability problem, discussed in several sections in Chapter 5 starting in Section 5.1, is also a constraint satisfaction problem in which U = {true, false} and each constraint corresponds to a clause of the input, and is a Boolean function evaluating to 1 exactly when the clause is satisﬁed. The unique games problem is a type of binary constraint satisfaction problem; that is, each constraint corresponds to a function on two variables. Furthermore, each function has the property that given the value of one of the two variables, there is exactly one value of the other variable for which the constraint is satisﬁed. For example, the constraint satisfaction problem given by the maximum cut problem above is a unique games problem on a universe of two elements because for any constraint on variables i and j (corresponding to an edge (i, j)), given Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.3
Unique games
345
the value of xi ∈ {0, 1}, there is only one value of xj for which the constraint is satisﬁed, and similarly given the value of xj . In general, we will be interested in the unique games problem for universes U of various sizes. In previous sections, we have mentioned that various theorems on the hardness of approximating particular problems depend on a conjecture involving the unique games problem. We are now in a position to state formally what this means. Informally, it means that it is NPhard to distinguish between instances of the unique games problem in which almost all of the constraints can be satisﬁed and almost none of the constraints can be satisﬁed. Conjecture 13.10 (Unique Games Conjecture): Given any ϵ, δ > 0, there exists some k > 0 depending on ϵ and δ, such that for the unique games problem with a universe of size k, it is NPhard to distinguish between instances in which at least a 1 − ϵ fraction of the constraints can be satisﬁed, and instances in which at most a δ fraction of the constraints can be satisﬁed. Because the unique games problem deals with binary constraints, it is typical to view it as a kind of undirected graph problem. In what follows, we will index the nodes of the graph by u, v, w, . . . and the values of the universe U by i, j, . . ., with U = [k] = {1, . . . , k}. We introduce a vertex u for each variable xu and an undirected edge (u, v) for each constraint f (xu , xv ). For each edge (u, v), we introduce a permutation πuv : U → U , such that πuv (i) = j exactly when f (i, j) = 1. Such a permutation exists by the deﬁnition of the unique games problem, since each label for u maps to a unique label for v that satisﬁes the constraint and vice versa. The problem becomes one of ﬁnding labels for the vertices from the universe U such that as many of the edges are satisﬁed as possible; that is, if vertex u is labelled with i ∈ U and vertex v is labelled with j ∈ U , edge (u, v) is satisﬁed if πuv (i) = j, since this corresponds to the original constraint being satisﬁed. Interestingly, it is easy to tell when all the edges (or constraints) are satisﬁable or not. Given the graph of constraints, consider a connected component of the graph and choose an arbitrary vertex v of the component. Given a label i for v, the labels for all the neighbors of v are uniquely determined if we satisfy all of the edges from v to its neighbors; in particular, we must label a neighbor w of v with πvw (i). We say that we propagate the label i for v to the neighbors of v. Similarly, we can propagate the labels of the neighbors of v to the labels of the neighbors of the neighbors of v, and so on, until we have labels for all vertices in the connected component. We then check whether all the edges of the component are satisﬁed. We do this for all possible labels of v; if there is some labelling that satisﬁes all the edges of the component, one of the choices of a label for v will be correct, and will lead to the correct labelling of all vertices in the component. Thus in polynomial time we can check whether all the edges of the graph are satisﬁable. This shows that a crucial part of the unique games conjecture is checking whether almost all of the constraints are satisﬁable, as opposed to checking whether all of the constraints are satisﬁable. At this point in time we do not know whether or not there exists an approximation algorithm for the unique games problem with a performance guarantee that would refute the unique games conjecture. In what follows, we give an algorithm which, given an instance of the unique games problem such that at least a√1 − ϵ fraction of the constraints are satisﬁable, outputs a solution satisfying at least a 1 − O( ϵ log n) fraction of the constraints. If ϵ = O(1/ log n), then the algorithm satisﬁes a constant fraction of the constraints. We begin by giving an integer quadratic program which models the unique games problem. For each node u ∈ V and each label i ∈ [k], we create a variable ui ∈ {0, 1}, where we set ui = 1 if the node u is assigned∑ the label i and 0 otherwise. Then since each node is assigned exactly one label, we have that ki=1 u2i = 1 and that ui uj = 0 for i ̸= j. Finally, since the edge Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
346
Further uses of randomized rounding of semideﬁnite programs
(u, v) is satisﬁed exactly when u is assigned some label i and v is assigned the label πuv (i), we ∑ have that ki=1 ui vπuv (i) = 1 exactly when edge (u, v) is satisﬁed, and is 0 otherwise. Hence the following program models the unique games problem:
k ∑ ∑
maximize
ui vπuv (i)
(13.6)
(u,v)∈E i=1 k ∑
subject to
u2i = 1,
∀u ∈ V,
i=1
ui uj = 0, ui ∈ {0, 1},
∀u ∈ V, i ∈ [k], j ∈ [k], i ̸= j, ∀u ∈ V, i ∈ [k].
We can add some additional constraints that are redundant for this integer quadratic program, but will be useful when we consider a vector programming relaxation of the problem. We can assume that (vj − ui )2 ≥ vj2 − u2i , since the inequality holds for all possible assignments of 0 and 1 to vj and ui . Finally, for any u, v, w ∈ V and h, i, j ∈ [k], it is the case that (wh − ui )2 ≤ (wh − vj )2 + (vj − ui )2 , again by checking all possible assignments of 0 and 1 to ui , vj , and wh . We can view this inequality as a type of triangle inequality. These additional inequalities yield the following integer quadratic program which also models the unique games problem:
maximize
k ∑ ∑
ui vπuv (i)
(13.7)
(u,v)∈E i=1
subject to
k ∑
∀u ∈ V,
u2i = 1,
i=1
∀u ∈ V, i ∈ [k], j ∈ [k], i ̸= j,
ui uj = 0, (vj − ui )2 ≥ vj2 − u2i ,
∀u, v ∈ V, i, j ∈ [k],
(wh − ui ) ≤ (wh − vj ) + (vj − ui ) , 2
ui ∈ {0, 1},
2
2
∀u, v, w ∈ V, h, i, j ∈ [k], ∀u ∈ V, i ∈ [k].
We relax the program (13.7) to a vector program by replacing all the scalar variables ui with vectors ui and multiplication with inner products. Since ui · ui = ∥ui ∥2 and (vj − ui ) · (vj − ui ) = Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.3
Unique games
347
∥vj − ui ∥2 , we obtain maximize
k ∑ ∑
ui · vπuv (i)
(13.8)
(u,v)∈E i=1
subject to
k ∑
∥ui ∥2 = 1,
∀u ∈ V,
ui · uj = 0,
∀u ∈ V, i ∈ [k], j ∈ [k], i ̸= j, (13.10)
(13.9)
i=1
∥vj − ui ∥2 ≥ ∥vj ∥2 − ∥ui ∥2 ,
∀u, v ∈ V, i, j ∈ [k],
∥wh − ui ∥ ≤ ∥wh − vj ∥ + ∥vj − ui ∥ , 2
2
(13.11)
∀u, v, w ∈ V, h, i, j ∈ [k], (13.12)
2
ui ∈ ℜkn ,
∀u ∈ V, i ∈ [k].
It is clear that this vector program is a relaxation of the integer quadratic program, and thus gives an upper bound on the number of constraints satisﬁable in the unique games instance. At a high level, our algorithm will work as follows. We solve the vector programming relaxation (13.8). We use this information to discard some number of constraints; for the remainder of the algorithm we will not care if we satisfy these constraints or not. The discarded edges will be of two types. First, we discard edges that contribute relatively little to the objective function. Second, we will remove edges so as to break the graph into lowradius balls via the regiongrowing technique introduced in Section 8.3. For the center w of ∑ each ball, we will randomly assign a label i to w with probability ∥wi ∥2 ; we can do this since ki=1 ∥wi ∥2 = 1 by constraint (13.9). Given that the center w is assigned the label i, then for every other vertex v in the ball, we assign v the label j that is closest to wi in the sense that ∥wi −vj ∥2 is minimized over all choices of label j. The fact that the ball is of low radius will imply that it is very likely, given an edge (u, v) with both endpoints in the ball, that the labels assigned to u and v satisfy the constraint (u, v). Before we dive into the details of the algorithm, we need some notation. Suppose we are given√ a unique games instance that satisﬁes at least a 1 − ϵ fraction of the constraints. Let δ = ϵ ln(n + 1). Let ui denote the optimal vectors from the vector program, and let ∑ ∑ Z ∗ = (u,v)∈E ki=1 ui · vπuv (i) denote the value of the optimal solution to the vector program. If m is the number of constraints, then Z ∗ ≥ OPT ≥ (1 − ϵ)m. For each edge (u, v) in the 1 ∑k unique games instance, we deﬁne a length ℓ(u, v) = 2 i=1 ∥ui − vπuv (i) ∥2 . The length of (u, v) captures how close the vectors for u are to the vectors for v that have labellings satisying edge (u, v); in particular the length ℓ(u, v) is zero when ui = vπuv (i) for all i ∈ [k]. Note that 1∑ ∥ui − vπuv (i) ∥2 2 k
ℓ(u, v) =
=
1 2
i=1 k ∑
(
) ∥ui ∥2 + ∥vπuv (i) ∥2 − 2ui · vπuv (i) .
i=1
Using constraint (13.9) and that πuv is a permutation, this implies ( ) k k ∑ ∑ 1 2−2 ui · vπuv (i) = 1 − ui · vπuv (i) , ℓ(u, v) = 2 i=1
i=1
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
348
Further uses of randomized rounding of semideﬁnite programs
so that the length ℓ(u, v) is exactly one minus the contribution of (u, v) to the vector program objective function. Let L∗ be the total length of all edges. Therefore, ∑ L∗ = ℓ(u, v) = m − Z ∗ ≤ m − (1 − ϵ)m = ϵm. (u,v)∈E
We can now ﬁll in some of the details of the algorithm. Again, the main intuition is that given a random assignment of a label to a node w, we will very likely be able to satisfy any edge whose endpoints are close to w in terms of the length ℓ. To that end, we want to break the graph up into components of low radius in√terms of length ℓ. We ﬁrst discard any edge (u, v) whose length ℓ(u, v) ≥ δ/4; recall δ = ϵ ln(n + 1). Note that since the total length of all edges is at most ϵm, we discard at most 4ϵm/δ = 4δm/ ln(n + 1) edges this way. We can then invoke the following lemma to remove at most another O(δm) edges and break the graph into balls of low radius according to the edge lengths ℓ. As mentioned previously, we will do this via the regiongrowing technique. Let dℓ (u, v) be the distance between u and v in the graph given edge lengths ℓ. Let Bℓ (u, r) be a ball of radius r around vertex u, so that Bℓ (u, r) = {v ∈ V : dℓ (u, v) ≤ r}. Lemma 13.11: There is a polynomialtime algorithm to remove at most 8δm edges from the graph so that the vertices of the graph is partitioned into t balls B1 , . . . , Bt , where each ball B is centered at some vertex w ∈ V and has radius at most δ/4. Proof. Let the volume Vℓ∗ (w, r) of a ball of radius r around w be the usual notion as in Sections 8.3 and 8.4; that is, for B = Bℓ (w, r) we set Vℓ∗ (w, r) =
L∗ + n
∑ e=(u,v):u,v∈B
ℓ(u, v) +
∑
(r − dℓ (w, u)).
e=(u,v):u∈B,v ∈B /
Note then that Vℓ∗ (w, r) ≤ L∗ + L∗ /n for any w ∈ V and any radius r. Then by Corollary 8.11, we can in polynomial time ﬁnd a radius r ≤ δ/4 around any vertex w ∈ V such that the number of edges with exactly one endpoint in Bℓ (w, r) is at most ( ∗ ) ( ∗ ) Vℓ (w, δ/4) 4 L + L∗ /n 4 4 ∗ ln Vℓ (w, r) ≤ ln Vℓ∗ (w, r) = ln(n + 1)Vℓ∗ (w, r). ∗ ∗ δ Vℓ (w, 0) δ L /n δ Following the proof of Theorem 8.9 for the multicut problem, we select a vertex w in the graph. If there is another vertex v at distance more than δ/4, we ﬁnd a ball around w of radius r ≤ δ/4 and remove at most 4δ ln(n + 1)Vℓ∗ (w, r) edges from the graph; we add this ball to our collection, and remove the vertices in the ball (and all adjacent edges) from the graph. We repeat until we select a vertex w in the remaining graph such that all vertices v in the remaining graph are no further than δ/4 away; the remaining graph then is the ﬁnal ball in the collection, with w as its center. The total volume Vℓ∗ (w, r) over all balls Bℓ (w, r) in the collection except the ﬁnal ball is at most 2L∗ , so the total number of edges removed is at most 8 8ϵm ln(n + 1)L∗ ≤ ln(n + 1) = 8δm, δ δ using L∗ ≤ ϵm and δ =
√
ϵ ln(n + 1).
We now consider each ball B in the collection separately. We show in the following lemma that if we label the center w with label i with probability ∥wi ∥2 , and label every other vertex Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.3
Unique games
349
v ∈ B with the label j minimizing ∥wi − vj ∥2 , then the probability that any edge (u, v) with both u, v ∈ B is satisﬁed is at least 1 − 3δ. Observe that the algorithm is easy to derandomize: we try all k labels for the center w. For each labelling of w, label the other v ∈ B as above and see which label for w satisﬁes the most edges that have both endpoints in the ball. By the lemma, some choice of label for w will satisfy at least a 1 − 3δ fraction of these edges. The central idea of the lemma is as follows. Consider a path from w to v, and suppose we assign label i to w with probability ∥wi ∥2 . Then consider two potential labels for v: the label for v that we get by propagating the label i from w to v along its path, and the label we get for v by assigning it the label j that minimizes ∥wi − vj ∥2 . We show that these labels disagree with probability at most 4 times the length of the path. Thus for any edge (u, v) with both endpoints in the ball, since the ball has low radius and the length of (u, v) is short, it is likely that u and v are both assigned labels that satisfy (u, v). Lemma 13.12: For any ball B with center w and any edge (u, v) such that u, v ∈ B, the probability that the algorithm assigns labels such that (u, v) is satisﬁed is at least 1 − 3δ.
Proof. Consider edge (u, v). Since u is in a ball of radius at most δ/4 around w, we know there is a path of length at most δ/4 from w to u. Let w = u0 , u1 , . . . , uq = u be this path. Since we discarded every edge of length greater than δ/4 we know that adding the edge (u, v) to this path makes a path from w of v of length at most δ/2. We want to be able to compute the label propagated to vertex ut along this path if w is labelled with label i. Let πt be the composition of the permutations πwu1 , . . . , πut−1 ut along the path from w to ut , so that πt (i) = πut−1 ut (πt−1 (i)). Let πu = πq (that is, the composition of all permutations along the path from w to u) and let πv be the composition of πu with πuv , so that πv (i) = πuv (πu (i)). Thus if we label w with i, then πt (i) is the label propagated to ut , πu (i) is the label propagated to u, and πv (i) is the label propagated to v. For any vertex z ∈ B, let A(z) be a random variable denoting the label assigned to vertex z by the algorithm. Let P be the event that the label A(u) given to u by the algorithm is equal to the label obtained by taking the label A(w) assigned to w by the algorithm and propagating it to u along the path; that is, P is the event that A(u) = πu (A(w)). We will show that the probability that this event does not occur is at most 4 times the length of the path from w to u, and so is at most δ. Thus the probability that these labels are the same is at least 1 − δ. Similarly, we can show that the probability that the label A(v) assigned to v by the algorithm is equal to the label propagated to v, πv (A(v)), is at least 1 − 4(δ/2) = 1 − 2δ. Thus with probability at least 1 − 3δ, A(v) = πv (A(w)) and A(u) = πu (A(w)) so that A(v) = πuv (A(u)), and the edge (u, v) is satisﬁed. Note that for any t,
ℓ(ut , ut+1 ) =
1∑ t 1∑ t 2 2 ∥ = ∥ui − ut+1 ∥uπt (i) − ut+1 πut ut+1 (i) πt+1 (i) ∥ 2 2 k
k
i=1
i=1
by our deﬁnitions and since πt is a permutation. Then by using the triangle inequality constraint Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
350
Further uses of randomized rounding of semideﬁnite programs
(13.12), we see that 1∑ ∥wi − uπu (i) ∥2 2 k
i=1
) 1 ∑( q 2 ∥wi − u1π1 (i) ∥2 + ∥u1π1 (i) − u2π2 (i) ∥2 + · · · + ∥uπq−1 − u ∥ πu (i) q−1 (i) 2 k
≤
i=1
=
k 1∑
2
1∑ 1 1 ∑ q−1 ∥uπ1 (i) − u2π2 (i) ∥2 + · · · + ∥uπq−1 (i) − uqπu (i) ∥2 2 2 k
∥wi − u1π1 (i) ∥2 +
i=1
k
i=1
= ℓ(w, u ) + ℓ(u , u ) + · · · + ℓ(u 1
1
2
i=1
q−1
q
,u )
≤ δ/4,
(13.13)
since the path length is at most δ/4. Let I be the set of labels i such that if we assign i to w then u is assigned a label diﬀerent than its propagated value πu (i); that is, some j ̸= πu (i) minimizes ∥wi − uj ∥2 . Then since we assign i to w with probability ∥wi ∥2 , we have that Pr[A(u) ̸= πu (A(w))] =
∑
∥wi ∥2 .
i∈I
We claim that for every i ∈ I, ∥wi − uπu (i) ∥2 ≥ 12 ∥wi ∥2 . Given the claim, we then have Pr[A(u) ̸= πu (A(w))] =
∑
∥wi ∥2
i∈I
≤ 2
∑
∥wi − uπu (i) ∥2
i∈I
≤ 2
k ∑
∥wi − uπu (i) ∥2
i=1
≤ 4 · (δ/4) = δ, where the last inequality follows from inequality (13.13). Since the length of the path from w to v is at most δ/2, by a similar argument, Pr[A(v) ̸= πv (A(w))] ≤ 4 · (δ/2) = 2δ. As we argued previously, this implies that edge (u, v) is satisﬁed with probability at least 1− 3δ. Now to prove the claim. We consider three cases. Assume that w is assigned the label i, and that j is the label for u that minimizes ∥wi − uj ∥2 . For simplicity of notation, let h be the propagated value πu (i); since i ∈ I, we know that πu (i) = h ̸= j. If ∥uh ∥2 ≤ 21 ∥wi ∥2 , then the inequality follows from the constraint (13.11), since 1 ∥wi − uh ∥2 ≥ ∥wi ∥2 − ∥uh ∥2 ≥ ∥wi ∥2 . 2 Now suppose ∥uj ∥2 ≤ 12 ∥wi ∥2 . Then since j is the label that minimizes ∥wi − uj ∥2 , we have that 1 ∥wi − uh ∥2 ≥ ∥wi − uj ∥2 ≥ ∥wi ∥2 − ∥uj ∥2 ≥ ∥wi ∥2 , 2 Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.3
Unique games
351
by constraint (13.11) and by hypothesis. Now suppose that neither condition holds, and both ∥uh ∥2 ≥ 12 ∥wi ∥2 and ∥uj ∥2 ≥ 12 ∥wi ∥2 . By the triangle inequality constraint (13.12) and the fact that j is the label minimizing ∥wi − uj ∥2 , we have that ∥uh − uj ∥2 ≤ ∥wi − uh ∥2 + ∥wi − uj ∥2 ≤ 2∥wi − uh ∥2 . By constraint (13.10) and the fact j ̸= h, it follows that uj · uh = 0 so that ∥uh − uj ∥2 = ∥uh ∥2 + ∥uj ∥2 − 2uh · uj = ∥uh ∥2 + ∥uj ∥2 . Thus
) 1 1 1( ∥wi − uh ∥2 ≥ ∥uh − uj ∥2 = ∥uh ∥2 + ∥uj ∥2 ≥ ∥wi ∥2 , 2 2 2
by hypothesis, and the claim holds. We can now prove the following theorem. Theorem 13.13: Given an instance of the unique games problem satisfying a 1 − ϵ fraction of √ the constraints, the algorithm above satisﬁes at least a 1 − 15δ = 1 − O( ϵ log n) fraction of the constraints. Proof. When we remove all long edges, we remove at most 4δm/ ln(n + 1) ≤ 4δm edges, and when we divide the graph up into balls, we remove at most 8δm edges. In each ball, we fail to satisfy at most a 3δ fraction of the edges with both endpoints in the balls, and so over all balls, we fail to satisfy at most 3δm edges. Thus we fail to satisfy at most (4 + 8 + 3)δm = 15δm edges, which gives the result. Somewhat better algorithms are known; these use the same vector programming relaxation of the unique games problem. One such algorithm shows the following. Theorem 13.14: Given a unique games instance in which at least a 1 − ϵ fraction of the constraints are satisﬁable, there is a polynomialtime algorithm that satisﬁes at least a 1 − √ O( ϵ log k) fraction of the constraints. The following theorem shows that this result is essentially the best possible assuming that the unique games conjecture is true. Theorem 13.15: Assuming that the unique games conjecture is true, for any ϵ > 0 there exists a k depending on ϵ such that for the unique games problem with universe size k, it is NPhard to distinguish between instances √ √ in which a 1 − ϵ fraction of constraints are satisﬁable, and instances in which a 1 − 2/π ϵ log k + o(1) fraction of constraints are satisﬁable.
Exercises 13.1 Show that the graph coloring algorithm of Section 13.2 can be used to color a 3colorable ˜ 1/4 ) colors. graph with O(n 13.2 In this problem, we consider an unweighted version of the minimum multicut problem from Section 8.3. We are given a graph G = (V, E) and k pairs of sourcesink vertices, si , ti ∈ V for i = 1, . . . , k. We wish to ﬁnd a subset of edges F ⊆ E that minimizes F  such that for each i = 1, . . . , k, there is no si ti path in (V, E − F ). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
352
Further uses of randomized rounding of semideﬁnite programs
Consider the following vector program: ∑ minimize (1 − vi · vj ) (i,j)∈E
subject to
vsi · vti = 0, vj · vj = 1, vj ∈ ℜ , n
i = 1, . . . , k, ∀j ∈ V, ∀j ∈ V.
Consider the demand graph H = (V, E ′ ), where E ′ = {(si , ti ) : i = 1, . . . , k}. Let ∆ be the maximum degree of a vertex in the demand graph. Suppose that that the optimal value of the vector programming relaxation is ϵE. Consider the following algorithm which is similar to the algorithm in Section 6.5 for coloring 3colorable graphs. We draw t = ⌈log2 (∆/ϵ)⌉ random vectors r1 , . . . , rt . The t random vectors deﬁne 2t diﬀerent regions into which the vectors vi can fall: one region for each distinct possibility of whether rj · vi ≥ 0 or rj · vi < 0 for all j = 1, . . . , t. Remove all edges (i, j) from the graph such that vi and vj are in diﬀerent regions. If for any si ti pair, there still exists an si ti path, remove all edges incident on si . We now analyze this algorithm. (a) Prove that the vector program is a relaxation of the unweighted minimum multicut problem. j) ∈ E, prove that the probability that i and j are in diﬀerent regions is (b) For any (i, √ at most t · 1 − vi · vj . (c) Prove that for any i = 1, . . . , k, the probability that we end up removing all the edges incident on i is at most ∆2−t . √ (d) Show that the expected number of edges removed is at most O( ϵ log(∆/ϵ))E. For the ﬁnal item, it may be useful to use Jensen’s inequality, which states that for any convex function f (that is, f ′′ (x) ≥ 0) and any positive pi , (∑ ) 1 ∑ i p i xi ∑ f ≤∑ pi f (xi ). i pi i pi i
The arithmeticgeometric mean inequality given in Fact 5.8 is a special case of Jensen’s inequality with f (x) = − log x and all the pi = 1. 13.3 In this problem, we consider another algorithm for the unweighted minimum multicut problem from the previous exercise, using the same vector programming relaxation as above. As before, assume that the optimal value of the vector programming relaxation is ϵE, and that √ ∆ is the maximum degree of a vertex in the demand graph. We set a threshold α = C ln(∆/ϵ) for some constant C. Suppose we draw a random vector r. Consider ∪ the set S(α) = {i ∈ V : vi · r ≥ α}, and S ′ (α) = S(α) − ki=1 {si , ti : si , ti ∈ S(α)}; that is, S ′ (α) is all elements in S(α) except for the si ,ti pairs that both end up in S(α). The following inequality will be useful for the analysis of the algorithm. Let vi and vj be unit vectors, and let r be a random vector. Let α be a quantity such that α > 1 and Φ(α) < 1/3. Then √ √ Pr[vi · r ≥ α and vj · r < α] = O( vi · vj Φ(α) log(1/Φ(α))). Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
13.3
Unique games
353
(a) Give a randomized algorithm that works by repeatedly drawing random vectors r and considering the set S ′ (α) as deﬁned above. Show that the expected size of the multicut returned by the algorithm is at most ∑ (i,j)∈E
Pr[(i, j) ∈ δ(S ′ (α))] . Pr[i ∈ S ′ (α) or j ∈ S ′ (α)]
(b) Prove that for any (i, j) ∈ E, Pr[(i, j) ∈ δ(S ′ (α))] √ ≤ O( vi · vj )α + ϵ. Pr[i ∈ S ′ (α) or j ∈ S ′ (α)] √ (c) Show that the randomized algorithm returns a multicut of size O( ϵ log(∆/ϵ))E edges in expectation.
Chapter Notes The approximation algorithm for quadratic programs given in Section 13.1 was rediscovered several times. To our knowledge, it ﬁrst appeared in a paper of Nemirovski, Roos, and Terlaky [234], then later in papers of Megretski [224] and Charikar and Wirth [67]. Our presentation here follows that of Charikar and Wirth. The approximation algorithm for coloring 3colorable graphs of Section 13.2 is due to Karger, Motwani, and Sudan [182], but the analysis we present follows that of Arora, Chlamtac, and Charikar [15]. Exercise 13.1 is due to Karger et al. Through an improved algorithm and analysis, Arora et al. are able to color 3colorable graphs with O(n0.211 ) colors. The unique games conjecture was ﬁrst formulated in a paper of Khot [192]. Since then it has prompted a signiﬁcant amount of research in algorithms and complexity theory, and has been used to show the conditional hardness of a quite signiﬁcant number of results; we will say more about the conjecture in Chapter 16. The approximation algorithm for the unique games problem given here is due to Trevisan [282] with a slightly improved analysis due to Gupta and Talwar [150]. Charikar, Makarychev, and Makarychev [65] give the improved result mentioned in Theorem 13.14. The hardness result of Theorem 13.15 is due to Khot, Kindler, Mossel, and O’Donnell [193] together with a result of Mossel, O’Donnell, and Oleszkiewicz [227]. Exercises 13.2 and 13.3 are due to Steurer and Vishnoi [272]. The bound given in Exercise 13.3 is from Chlamtac, Makarychev, and Makarychev [72, Lemma A.2].
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
354
Further uses of randomized rounding of semideﬁnite programs
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
C h a p t e r 14
Further uses of the primaldual method
In this chapter, we give two somewhat more sophisticated applications of the standard primaldual algorithm and analysis introduced in Chapter 7. We revisit the prizecollecting Steiner tree problem discussed in Sections 4.4 and 5.7, and give a primaldual 2approximation algorithm for it. We also revisit the feedback vertex set problem in undirected graphs introduced in Section 7.2, and give a primaldual 2approximation algorithm for this problem, as well. In this case, we must give an alternate integer programming formulation for the problem since the previous formulation has an integrality gap that implies that the O(log n)approximation algorithm of Section 7.2 is the best possible (within constant factors) using the formulation of that section.
14.1
The prizecollecting Steiner tree problem
In this section, we revisit the prizecollecting Steiner tree problem introduced in Section 4.4 and further discussed in Section 5.7. We show that a primaldual algorithm can be used to give a 2approximation algorithm for the problem; this improves on the performance guarantee of the previously given algorithms, and the primaldual algorithm does not require solving a linear programming relaxation. Recall that in this problem we are given an undirected graph G = (V, E), edge costs ce ≥ 0 for all e ∈ E, a selected root vertex r ∈ V , and penalties πi ≥ 0 for all i ∈ V . The goal is to ﬁnd a tree T that contains the root vertex r so the cost ∑as to minimize ∑ of the edges in T plus the penalties of the vertices not in T ; that is, e∈T ce + i∈V −V (T ) πi , where V (T ) is the set of vertices in the tree. We start by giving a diﬀerent integer programming formulation of the problem. We have decision variables xe ∈ {0, 1} indicating if an edge e ∈ E is part of the tree. We also have variables zX ∈ {0, 1} for all X ⊆ V − r, where zX = 1 if X is the set of vertices not spanned by the tree and zX = 0 otherwise. Then we need a constraint enforcing that for any subset S ⊆ V , either S is a subset of the set of vertices not spanned by the tree or some of S is being spanned by the tree. We do this with the constraint ∑ e∈δ(S)
xe +
∑ X:X⊇S
355
zX ≥ 1,
356
Further uses of the primaldual method
since if x∑ e = 1 for some e ∈ δ(S), then at least some vertex in S is spanned by the tree. Let π(X) = i∈X πi . Then we have the following integer programming formulation: minimize
∑ e∈E
subject to
∑
ce xe + ∑
π(X)zX
X⊆V −r
xe +
e∈δ(S)
∑
zX ≥ 1,
∀S ⊆ V − r,
xe ∈ {0, 1} ,
∀e ∈ E,
zX ∈ {0, 1} ,
∀X ⊆ V − r.
X:X⊇S
As in the case of the shortest st path problem and the generalized Steiner tree problem in Sections 7.3 and 7.4, it can be shown via the maxﬂow/mincut theorem that in any feasible solution there is a tree connecting the root r to every vertex i that is not in some set X for which zX = 1. If we replace the integrality conditions xe ∈ {0, 1} and zX ∈ {0, 1} with xe ≥ 0 and zX ≥ 0, and take the dual of this linear program, we obtain the following: ∑ maximize yS S⊆V −r
subject to
∑
yS ≤ ce ,
∀e ∈ E,
yS ≤ π(X),
∀X ⊆ V − r,
yS ≥ 0,
∀S ⊆ V − r.
S:e∈δ(S)
∑
S:S⊆X
Note that the dual LP has two types of constraints: one type associated with edges, and the other associated with sets of vertices. It will be convenient to introduce some notation regarding the constraint on sets. / X and a dual feasible solution y, Deﬁnition 14.1: For a given set∑of vertices X such that r ∈ we say that p(X, y, π) = π(X) − S:S⊆X yS is the remaining potential of the set X. We now give our primaldual algorithm in Algorithm 14.1. As in the case of the generalized Steiner tree algorithm, we maintain a feasible dual solution y (initially all zero) and an infeasible primal solution F (initially the empty set). We divide the connected components of F into two types: active and inactive. The connected component containing the root is always inactive. Any component C such that its potential p(C,∑ y, π) is zero is also inactive. Note that this corresponds to a tight dual constraint in which S:S⊆C yS = π(C), so we cannot increase the dual variable yC without making the dual solution infeasible. We increase the dual variables associated with all active components uniformly until one of the two types of dual constraints becomes tight. If a constraint associated with an edge e becomes tight for some e ∈ δ(C) for some active set C, we add e to F and continue. If a constraint associated with a set C becomes tight for some active set C (and its potential becomes zero), we make C inactive and continue. We end the main loop when all connected components of F are inactive. As with the generalized Steiner tree algorithm, we have a ﬁnal cleanup step in which we consider all the edges added to F in the reverse of the order in which they were added. We begin by letting F ′ be the connected component of F that contains the root vertex r. Then we iterate through the edges of F ′ in the reverse of the order in which they were added to F . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
14.1
The prizecollecting Steiner tree problem
357
y←0 F ←∅ ℓ←1 while not all connected components of (V, F ) are inactive do Let C be the set of all active connected components C of (V, F ) (p(C, y, π) > 0 and r∈ / C) ′ ′ Increase yC for ∑ all C in C uniformly until either for some eℓ ∈ δ(C ), C ∈ C, ceℓ = S:eℓ ∈δ(S) yS or for some C ∈ C, p(C, y, π) = 0 if for some C ∈ C, p(C, y, π) = 0 then Make C inactive and remove C from C else F ← F ∪ {eℓ } ℓ←ℓ+1 Let F ′ be the connected component of (V, F ) containing r for k ← ℓ − 1 downto 1 do if ek ∈ F ′ then Let C be vertices connected to r by ek if p(C, y, π) = 0 then Remove ek from F ′ and remove all edges of F ′ with both endpoints in C return F ′ Algorithm 14.1: Primaldual algorithm for the prizecollecting Steiner tree problem. Suppose we consider removing edge e. Removing e will disconnect some set of vertices C from the root. We check whether p(C, y, π) = 0. If it is, we remove from F ′ the edge e and all the edges of F ′ connecting vertices of C (so that F ′ remains a tree containing the root). Otherwise we keep edge e. We show that this algorithm gives us a performance guarantee of 2. As in the case of the generalized Steiner tree algorithm, this result follows from a lemma whose proof we will defer for a few moments. Lemma 14.2: Let F ′ be the tree returned by the algorithm, and let X be the vertices not spanned by F ′ . In any given iteration in which C is the set of active components, ∑ δ(C) ∩ F ′  +  {C ∈ C : C ⊆ X}  ≤ 2C. C∈C
From this lemma we can prove the performance guarantee. Theorem 14.3: Algorithm 14.1 is a 2approximation algorithm for the prizecollecting Steiner tree problem. Proof. Let F ′ be the ﬁnal tree containing the root, and let X be the vertices not spanned by F ′ . We ﬁrst claim that X can be partitioned into sets X1 , . . . , Xk such that the potential p(Xj , y, π) = 0 for each set Xj . To see this, note that for any vertex in X either it was connected to the root before the edge deletion process or it was not. If it was not connected to the root, then it was in some inactive component I not containing the root, and I was inactive since the potential p(I, y, π) = 0. If the vertex was connected to the root before the edge deletion process, then it must have been disconnected in some step in which an edge was removed and Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
358
Further uses of the primaldual method
a set of vertices I was disconnected from the root. But for this set, it must have been the case that the potential p(I, y, π) = 0. Hence the claim follows. ∑ Observe that the potential p(I, y, π) = 0 implies that π(I) = S⊆I yS .We now rewrite the cost of the primal solution in terms of dual variables: ∑
ce +
e∈F ′
∑
πi =
i∈X
=
∑
∑
e∈F ′
S:e∈δ(S)
∑
yS +
k ∑
π(Xj )
j=1 ′
δ(S) ∩ F yS +
k ∑ ∑
yS .
j=1 S⊆Xj
S
We would like to show that this cost is less than twice the dual objective function; that is, we would like to show that ∑
′
δ(S) ∩ F yS +
k ∑ ∑ j=1 S⊆Xj
S
yS ≤ 2
∑
yS .
(14.1)
S⊆V −r
As in the proof of Lemma 7.7, we do this by induction on the algorithm. Initially all dual variables are zero, and the inequality holds. Suppose that the inequality holds at the beginning of any given iteration. Let C be the active components in this iteration. Then in this iteration we increase each yS for S ∈ C by the same amount (call it ϵ). This increases the lefthand side of (14.1) by k ∑ ∑ ϵ F ′ ∩ δ(C) + ϵ  {C ∈ C : C ⊆ Xj }  j=1
C∈C
and the righthand side by 2ϵC. However, by the inequality of Lemma 14.2, this means that the increase in the lefthand side is no greater than the increase in the righthand side. Thus if the inequality held before the increase of the dual variables in this iteration, it will also hold afterwards. We now conclude with the proof of the lemma. Proof of Lemma 14.2. Given an iteration, let C be the set of active components, and let I be the set of connected components which are inactive. Note that the sets in C and I partition the set of vertices. We begin by simplifying the inequality we must prove. Observe that if for some C ∈ C, no vertex in C is spanned by the tree F ′ , then δ(C) ∩ F ′ = ∅. Hence if C ⊆ X, then δ(C) ∩ F ′ = ∅. Let C ′ = {C ∈ C : C ̸⊆ X}. Thus the desired inequality is implied by showing ∑ δ(C) ∩ F ′  ≤ 2C ′ . C∈C ′
As in the proof of Lemma 7.7 for the generalized Steiner tree, we consider the graph obtained by contracting each component in C ′ and I to a single node, ignoring components in C − C ′ ; let V ′ be the resulting vertex set. Let T be the set of edges on V ′ from F ′ once the components in C ′ and I are collapsed; the edges in T form a tree as shown in Observation 7.9 and Lemma 7.7. We let deg(v) for v ∈ V ′ represent the degree of vertex v in this tree. We also color the vertices in V ′ with two diﬀerent colors, red and blue: red vertices correspond to components from C ′ Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
14.1
The prizecollecting Steiner tree problem
359
and blue vertices correspond to components from I. Let R denote the set of red vertices in V ′ and B the set of blue vertices v that have deg(v) > 0. We can rewrite the desired inequality ∑
δ(C) ∩ F ′  ≤ 2C ′ 
C∈C ′
∑ in terms of this tree by showing v∈R deg(v) ≤ 2R. To prove this inequality, we claim that at most one blue vertex can have degree exactly one. If this is true, then we have that ∑
∑
deg(v) =
deg(v) −
deg(v).
v∈B
v∈R∪B
v∈R
∑
Since the total degree of a forest is no more than twice the number of vertices in it minus one, and since all but one of the blue vertices in B have degree at least two by the claim, we have that this quantity is ∑ v∈R∪B
deg(v) −
∑
deg(v) ≤ 2(R + B − 1) − 2(B − 1) − 1 ≤ 2R,
v∈B
as desired. Now to show the claim. At most one blue vertex corresponds to a component in the uncontracted graph containing the root. We show that all other blue vertices cannot have degree one. Suppose not, and there is a blue vertex v of degree one. Let e ∈ F ′ be the edge incident on this vertex, and let I ∈ I be the corresponding component in the uncontracted graph, with r ∈ / I. First observe that because e ∈ δ(I), it is the case that e was added after the current iteration, otherwise I would be part of a larger connected component containing e in the current iteration. Now when we considered e for removal, it joined some component C to the root. Since I remains connected to the root by e in the current iteration, it must be the case that I ⊆ C. Let Q1 , . . . , Qk be the parts of C that were removed by later steps of the edge deletion process, from the point that e is considered down to the current iteration, so that I and the Qj partition C (see Figure 14.1). Because the Qj were removed, we know that p(Qj , y, π) = 0. Because I is inactive and does not contain the root, it is also the case that p(I, y, π) = 0. By dual feasibility we know that π(C) ≥
∑
yS ≥
S⊆C
∑ S⊆I
yS +
k ∑ ∑
yS .
j=1 S⊆Qj
Because the potentials of I and the Qj are zero, the righthand side is
π(I) +
k ∑
π(Qj ) = π(C).
j=1
∑ Thus it must be the case that S⊆C yS = π(C), and that p(C, y, π) = 0. Hence we should have removed edge e, which contradicts e ∈ F ′ , and we are done. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
360
Further uses of the primaldual method
Q2 Q1
C
I
Q3
e Figure 14.1: Part of the proof of Lemma 14.2.
14.2
The feedback vertex set problem in undirected graphs
In this section, we return to the feedback vertex set problem in undirected graphs introduced in Section 7.2. Recall that in this problem we are given an undirected graph G = (V, E) and nonnegative weights wv ≥ 0 for vertices v ∈ V , and the goal is to choose a minimumcost subset of vertices S ⊆ V such that every cycle C in the graph contains some vertex of S. Alternatively, we must ﬁnd a minimumweight set of vertices S such that the induced graph G(V − S) is acyclic, where G(V − S) is the graph induced by the vertex set V − S. We will sometimes use the shorthand G − S for the graph induced on V − S. In Section 7.2, we gave a primaldual O(log n)approximation algorithm for the feedback vertex set problem. The algorithm used the following integer programming formulation for the problem, where C denotes the set of all cycles C in the graph: minimize
∑
wv xv
v∈V
subject to
∑
xv ≥ 1,
∀C ∈ C,
xv ∈ {0, 1} ,
v ∈ V.
v∈C
In that section we claimed that the integrality gap of this formulation is known to be Ω(log n), and so we cannot achieve a performance guarantee better than O(log n) using the integer programming formulation above. However, we can devise a primaldual 2approximation algorithm by using an alternate integer programming formulation for the problem. To do this, we start by considering the number of edges that will need to be removed from the graph in order for the graph to be acyclic. Let c(G) be the number of connected components in graph G. An acyclic subset of edges in a graph G = (V, E) can have at most V  − c(G) edges. Therefore, we must remove at least E − V  + c(G) edges for the graph to become acyclic. Note that if we remove a vertex v and its incident edges, and d(v) is the degree of vertex v, then it would seem that we remove d(v) edges towards the overall goal of removing E − V  + c(G) edges. However, note that by removing v, we also reduce the number of vertices that need to be spanned by the remaining edges. In particular, in the remaining graph (with v removed), we now need to remove at least (E−d(v))−(V −1)+c(G−v) edges for G−v to become acyclic. Thus removing v decreases Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
14.2
The feedback vertex set problem in undirected graphs
361
the total number of edges we need to remove by [E − V  + c(G)] − [(E − d(v)) − (V  − 1) + c(G − v)] = d(v) − 1 + (c(G) − c(G − v)). For simplicity of exposition, let us denote the quantity c(G − v) − c(G) + 1 by b(v), so that removing v decreases the total number of edges we need to remove by d(v) − b(v). We can formalize the reasoning above as follows. Lemma 14.4: For any feedback vertex set F of the graph G = (V, E), ∑ (d(v) − b(v)) ≥ E − V  + c(G). v∈F
Proof. We know that if we remove F and its adjacent edges from the graph, the remaining set ∑ of edges is acyclic. Let E(F ) be the set of edges that have both endpoints in F . Then v∈F d(v)−E(F ) is an upper bound on the number of edges removed. Since V −F −c(G−F ) is an upper bound on the number of edges left over, we have that ∑ d(v) − E(F ) + V  − F  − c(G − F ). E ≤ v∈F
We now claim that we can show that ∑ c(G) ≤ (1 − b(v)) + E(F ) + c(G − F )
(14.2)
v∈F
Given the claim, if we add the two inequalities, we obtain ∑ E + c(G) ≤ (d(v) − b(v)) + V . v∈F
Subtracting V  from both sides gives the desired inequality. We prove inequality (14.2) by induction on F ; we will not need the fact that F is a feedback vertex set to prove the claim. It is easy to see that the claim holds if F = ∅ or F = {v}; in the latter case, this follows by the deﬁnition of b(v) since c(G) ≤ (1 − b(v)) + c(G − v) = (1 − c(G − v) + c(G) − 1) + c(G − v) = c(G). Suppose the claim holds for F − v; we now show it holds for F . Consider the neighbors of v in G. If we remove v from G, these are partitioned into at least c(G − v) − c(G) = b(v) − 1 additional connected components. Suppose we choose one distinct neighbor of v in each of these b(v) − 1 connected components. For each such neighbor, either the neighbor is already in F , and hence E(F ) includes the edge from v to this neighbor, or the neighbor is not in F , and removing v from G − (F − v) will create a new connected component containing this neighbor. Thus we have that b(v) − 1 is at most E(F ) − E(F − v) plus c(G − F ) − c(G − (F − v)) or 0 ≤ 1 − b(v) + E(F ) − E(F − v) + c(G − F ) − c(G − (F − v)). By induction the inequality holds for F − v, so that ∑ c(G) ≤ (1 − b(u)) + E(F − v) + c(G − (F − v)). u∈F −v
Adding the previous inequality to this one gives inequality (14.2) for F . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
362
Further uses of the primaldual method
Consider any subset S ⊆ V of vertices. Let G(S) be the subgraph of G = (V, E) induced by S, let dS (v) for v ∈ S be the degree of v in G(S), and let bS (v) for v ∈ S be the value of b(v) in the graph G(S). Let f (S) = E(S) − S + c(G(S)) be the minimum number of edges we need to remove to have a feedback vertex set in G(S). Then by observing that F ∩ S is a feedback vertex set for G(S), we have the following corollary to Lemma 14.4. Corollary 14.5: For any subset of vertices S ⊆ V , ∑ (dS (v) − bS (v)) ≥ f (S). v∈F ∩S
We can now give the following integer program, which we claim models the feedback vertex set problem in undirected graphs: ∑ minimize wv xv subject to
∑
v∈V
(dS (v) − bS (v))xv ≥ f (S),
∀S ⊆ V,
v∈S
xv ∈ {0, 1} ,
v ∈ V.
As before, the variable xv denotes whether the vertex v is in the feedback vertex set or not. If a solution x corresponds to a feedback vertex set, then all the constraints are obeyed by Corollary 14.5. If a solution x does not correspond to a feedback vertex set, then there must exist a cycle C such that xv = 0 for all v ∈ C. Now consider the constraint corresponding to C: the lefthand side will be zero, but we claim that f (C) ≥ 1, and so the constraint will be violated. This will conclude the proof that the integer program indeed models the feedback vertex set problem. To see that f (C) ≥ 1, note that since C is a cycle, E(C) ≥ C, so that f (C) = E(C) − C + c(G(C)) ≥ c(G(C)) ≥ 1. The dual of the linear programming relaxation of the integer program is then ∑ maximize f (S)yS subject to
∑
S⊆V
(dS (v) − bS (v))yS ≤ wv ,
∀v ∈ S,
S:v∈S
yS ≥ 0,
∀S ⊆ V.
Following the standard primaldual algorithm as described in Chapter 7, we derive Algorithm 14.2. We start with a dual feasible solution of y = 0, and a primal infeasible solution of F = ∅. We maintain a set S of the vertices in the subgraph of G we are currently considering; initially, S = V . Then while F is not feasible (equivalently, G(S) is not acyclic), we increase the dual variable yS until some dual constraint becomes tight for some v ∈ S. We then add v to F , and remove v from S. We also remove from S all vertices that are no longer in a cycle in G(S − v). Once F is a feasible solution, we have a ﬁnal cleanup step similar to that used for the generalized Steiner tree problem in Section 7.4. In particular, we iterate through the vertices of F in the reverse of the order in which they were added to F . We check to see whether the current vertex v can be removed from F without aﬀecting its feasibility. If so, we remove it, otherwise we leave v in F . The cleanup step is designed to help prove the following lemma. We say that a feedback vertex set F is minimal for a graph G if for any v ∈ F , F − v is not a feedback vertex set for G. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
14.2
The feedback vertex set problem in undirected graphs
363
y←0 F ←∅ S←V ℓ←0 while F is not feasible (and G(S) contains a cycle) do ℓ←ℓ+1 ∑ Increase yS until for some vℓ ∈ S, C:vℓ ∈C (dC (vℓ ) − bC (vℓ ))yC = wvℓ . F ← F ∪ {vℓ } T ← {v ∈ S : v not on some cycle in G(S − vℓ )} S ← S − {vℓ } − T F′ ← F for k ← ℓ downto 1 do if F ′ − vk is a feasible solution then Remove vk from F ′ return F ′ Algorithm 14.2: Improved primaldual algorithm for the feedback vertex set problem in undirected graphs. Lemma 14.6: For the primal solution F ′ and dual solution y returned by Algorithm 14.2, for any S ⊆ V such that yS > 0, F ′ ∩ S is a minimal feedback vertex set for G(S). Proof. The proof is by contradiction. Suppose there is some vertex vj ∈ F ′ ∩ S such that F ′ ∩ S − vj is a feedback vertex set for G(S). By the construction of the cleanup step, we know that it must be the case that {v1 , . . . , vj−1 }∪F ′ −vj must not be a feedback vertex set for G, since otherwise we would have removed vj from F ′ in the cleanup step. Since {v1 , . . . , vj−1 } ∪ F ′ − vj is not a feedback vertex set for G, while F ′ is a feedback vertex set for G, it must be the case that vj is the unique vertex of {v1 , . . . , vj−1 } ∪ F ′ on some cycle C in G. Furthermore, C ⊆ S since vj ∈ S and {v1 , . . . , vj−1 } ∩ C = ∅, and thus by construction of the algorithm we cannot have removed any vertex of C from S in the iteration in which yS is increased. Thus since C is in G(S) and C ∩ F ′ = {vj }, it cannot be the case that F ′ ∩ S − vj is a feedback vertex set for G(S). As is typical for proofs of the analysis in primaldual algorithms, the performance guarantee of the algorithm can be reduced to a combinatorial lemma. In this case, the lemma is as follows. We defer the proof for a moment. Lemma 14.7: For any graph G such that every vertex v ∈ V is contained in some cycle, and for any minimal feedback feedback vertex set F for G, ∑
(d(v) − b(v)) ≤ 2f (V ) = 2(E − V  + c(G)).
v∈F
Given the lemma, we can now prove the performance guarantee of the algorithm. Theorem 14.8: Algorithm 14.2 is a 2approximation algorithm for the feedback vertex set problem in undirected graphs. Proof. Let F ′ be the ﬁnal feedback vertex set returned by the algorithm. Then by the standard Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
364
Further uses of the primaldual method
primaldual analysis, ∑
wv =
∑ ∑ v∈F ′
v∈F ′
=
∑
(dS (v) − bS (v))yS
S:v∈S
∑
yS
(dS (v) − bS (v)).
v∈F ′ ∩S
S⊆V
By Lemma 14.6, we know that if∑yS > 0, then F ′ ∩ S is a minimal feedback vertex set for G(S). By Lemma 14.7, we know that v∈F ′ ∩S (dS (v) − bS (v)) ≤ 2f (S) for the graph G(S). Thus we have that ∑ ∑ f (S)yS ≤ 2 OPT, wv ≤ 2 by weak duality, since
v∈F ′
∑ S⊆V
S⊆V
f (S)yS is the dual objective function.
We now turn to the proof of Lemma 14.7. ∑ Proof of Lemma 14.7. We know that v∈V d(v) = 2E, so subtracting this from both sides of the inequality and leaves us to prove that ∑ ∑ ∑ d(v) − d(v) − b(v) ≤ 2(c(G) − V ). v∈F
v∈V
Doing some rearranging, we get ∑ ∑
∑
v ∈F /
v∈F
d(v) ≥ 2V  −
∑
b(v) − 2c(G).
v∈F
∑ Note that v∈F / d(v) = v ∈F / dV −F (v)+δ(F ) and that since G(V −F ) is a forest, v ∈F / dV −F (v) = 2(V  − F  − c(G − F )). Thus the desired inequality is equivalent to proving that ∑ 2(V  − F  − c(G − F )) + δ(F ) ≥ 2V  − b(v) − 2c(G). v∈F
Rearranging terms again gives us 2F  + 2c(G − F ) ≤ δ(F ) +
∑
b(v) + 2c(G).
(14.3)
v∈F
Since F is a minimal feedback vertex set of G, for each v ∈ F there must be some cycle Cv in G such that v is the only vertex of F in the cycle, since otherwise we can remove v from F and still have a feedback vertex set. Therefore each v ∈ F must be adjacent to two edges from F to V − F from the cycle Cv ; we will call these edges cycle edges. Note then that we can charge the quantity 2F  in the desired inequality to the cycle edges in δ(F ). Observe also that the two cycle edges for any vertex v ∈ F must both be adjacent to the same connected component of G(V − F ). To complete the proof, we must show for each component of G(V − F ), we can charge 2 against the righthand side of the inequality (14.3) above. We show that we can appropriately charge the connected components of G, the noncycle edges of δ(F ), or b(v) for v ∈ F . Consider a given connected component S of G(V − F ). Note that by the hypothesis of the lemma, every vertex is in some cycle of the graph, so there must be at least two edges from F to S. Also note that if a cycle edge from some cycle Cv connects v to S, then the other cycle edge must also connect v to S; it cannot connect some other connected component of G(V − F ) to v. We now give the charging scheme in a case analysis. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
14.2
The feedback vertex set problem in undirected graphs
365
components of G − F
F
Figure 14.2: The ﬁnal case analysis of Lemma 14.7. Solid lines represent cycle edges, and dashed edges noncycle edges. The four cases are illustrated in order by the four components of G(V − F ). • If there are at least two noncycle edges from F to S, then we can charge 2 to δ(F ) for these noncycle edges. • If there are exactly two edges from F to S, and both are cycle edges corresponding to a cycle Cv , then either v ∪ S is a connected component of G and we can charge 2 to the corresponding connected component (that is, to the 2c(G) term), or removing v disconnects S from G, so that b(v) = c(G − v) − c(G) + 1 ≥ 2, and we can charge 2 to b(v). • If there are three edges from F to S, two of which are cycle edges corresponding to a cycle Cv , then we can charge 1 to the noncycle edge in δ(F ) and 1 to b(v) ≥ 1. • If there are four or more cycle edges from F to S, two of which correspond to a cycle Cv and two of which correspond to a cycle Cw , then we can charge 1 to b(v) ≥ 1 and 1 to b(w) ≥ 1. See Figure 14.2 for an illustration of the cases. In Exercise 14.4, we give another integer programming formulation for the feedback vertex set problem and show that it can also lead to a primaldual 2approximation algorithm for the problem. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
366
Further uses of the primaldual method
Exercises 14.1 Show that the performance guarantee of the primaldual algorithm for the prizecollecting Steiner tree algorithm in Section 14.1 can be strengthened in the following way. Let T be the tree returned by the algorithm and let y be the dual solution constructed by the algorithm. Show that ∑ ∑ ∑ ce + 2 πi ≤ 2 yS . i∈V −V (T )
e∈T
S⊆V −r
This proves that the algorithm is Lagrangean multiplier preserving (see Section 7.7). 14.2 In the kminimum spanning tree problem (kMST), we are given an undirected graph G = (V, E) with edge costs ce ≥ 0 for all e ∈ E, a root vertex r ∈ V , and a positive integer k. The goal is to ﬁnd a minimumcost tree spanning at least k vertices that includes the root. (a) Give an integer programming formulation for the kMST problem that has the same constraints as the integer program for the prizecollecting Steiner tree problem plus one additional constraint. Then apply Lagrangean relaxation to obtain an integer program in the same form as that for the prizecollecting Steiner tree problem modulo a constant term in the objective function. (b) Prove that an αapproximation algorithm which works in the case that the maximum distance from the root r is no more than OPTk (the cost of an optimal tree spanning k vertices) can be used to produce an αapproximation algorithm for the general case. (c) Using the prizecollecting Steiner tree algorithm as a subroutine and the analysis in Exercise 14.1, obtain a (5 + ϵ)approximation algorithm for the kMST problem. 14.3 Show that the linear programming relaxation for the prizecollecting Steiner tree problem used in Section 14.1 is equivalent to the linear programming relaxation for the problem used in Sections 4.4 and 5.7. 14.4 There are other possible primaldual 2approximation algorithms for the feedback vertex set problem in undirected graphs other than the one given in the chapter. In this exercise, we derive another such algorithm. (a) Argue that for any feedback vertex set F , ∑ (d(v) − 1) ≥ E − V  + 1. v∈F
(b) Show that the following is an integer programming formulation of the feedback vertex set problem for g(S) = E(S) − S + 1: ∑ minimize wv xv subject to
∑
v∈V
(dS (v) − 1)xv ≥ g(S),
∀S ⊆ V,
v∈S
xv ∈ {0, 1} ,
v ∈ V.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
14.2
The feedback vertex set problem in undirected graphs
367
(c) A cycle is semidisjoint if it at most one vertex of the cycle has degree greater than 2. Show that if every vertex of G has degree at least 2 and G contains no semidisjoint cycles, then for any minimal feedback vertex set F , ∑
(d(v) − 1) ≤ 2g(S).
v∈F
(d) Give a primaldual algorithm for the problem based on the formulation above and prove that it is a 2approximation algorithm (Hint: The algorithm will need to increase a diﬀerent dual variable if the graph contains a semidisjoint cycle). 14.5 Consider the prizecollecting generalized Steiner tree problem. In this problem, we are given an undirected graph G = (V, E) with nonnegative costs ce ≥ 0 on e ∈ E. We are also given k sourcesink pairs si ti , and a penalty πi ≥ 0 for each sourcesink pair. We want to ﬁnd a set of edges F ⊆ E that minimizes the total cost of the edges in F plus the sum of the penalties of the si ti pairs that are not connected in (V, F ). (a) Give an integer programming formulation of the problem with variables yi ∈ {0, 1} that indicate whether the pair si and ti are connected in the solution. Using a linear programming relaxation of this formulation, give a randomized rounding (1 − e−1/2 )−1 approximation algorithm for the problem, where (1 − e−1/2 )−1 ≈ 2.54. (b) Let X be a collection of subsets of vertices. Let Si be the collection of all sets S such that S ∩ {si , ti }  = 1. We give another integer programming formulation of the problem. In our formulation, we will have 01 variables zX , where zX = 1 for X containing all sets S such that we have selected no edge in δ(S). Let π(X ) be the sum of all penalties πi such that there exists some S ∈ Si that is also in X ; that is, ∑
π(X ) =
πi .
1≤i≤k:X ∩Si ̸=∅
Then our formulation is minimize
∑ e∈E
subject to
ce xe + ∑
e∈δ(S)
∑ X
xe +
π(X )zX ∑
zX ≥ 1,
∀S ⊆ V,
xe ∈ {0, 1} ,
∀e ∈ E,
zX ∈ {0, 1} ,
∀X .
X :S∈X
Prove that this formulation models the prizecollecting generalized Steiner tree problem. (c) Give a primaldual 3approximation algorithm for the prizecollecting generalized Steiner tree problem using the linear programming relaxation of the integer program above. For an easier variant of this problem, you do not need to explain how to detect in polynomial time when the dual constraint corresponding to the variables zX becomes tight. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
368
Further uses of the primaldual method
Chapter Notes The prizecollecting Steiner tree algorithm of Section 14.1 is due to Goemans and Williamson [138]. The ﬁrst 2approximation algorithms for the feedback vertex set problem in undirected graphs are due to Becker and Geiger [41] and Bafna, Berman, and Fujito [29], and were stated in terms of the local ratio technique. These algorithms were translated into primaldual approximation algorithms by Chudak, Goemans, Hochbaum, and Williamson [74]; Exercise 14.4 is from this paper. The algorithm given in Section 14.2 is based on another primaldual 2approximation algorithm for the problem due to Fujito [117]. The result of Exercise 14.1 is implicit in Goemans and Williamson [138], but made explicit independently by Blum, Ravi, and Vempala [51] and Goemans and Kleinberg [131]. The 5approximation algorithm for the kMST problem in Exercise 14.2 was ﬁrst given by Garg [125]; the use of Lagrangean relaxation for the problem was made explicit by Chudak, Roughgarden, and Williamson [75]. The equivalence of the two prizecollecting Steiner tree formulations given in Exercise 14.3 is found in Williamson [287]. The prizecollecting generalized Steiner tree problem in Exercise 14.5 is deﬁned by Hajiaghayi and Jain [154]; the two algorithms in the exercise are also from this paper.
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
C h a p t e r 15
Further uses of cuts and metrics
In Section 8.5, we introduced the idea of approximating one kind of metric with another one; namely, we looked at the idea of approximating a general metric with a tree metric. Here we will consider approximating a general metric with another metric more general than a tree metric, namely an ℓ1 embeddable metric. We show that we can approximate any metric (V, d) with an ℓ1 embeddable metric with distortion O(log n), where n = V . The ℓ1 embeddable metrics have a particularly close connection to cuts; we show that any such metric is a convex combination of the cut semimetrics we discussed at the beginning of Chapter 8. We show that the low distortion embeddings into ℓ1 embeddable metrics have applications to cut problems by giving an approximation algorithm for the sparsest cut problem. In Section 15.2, we give an algorithm that ﬁnds a packing of trees called cuttrees into a graph; this packing allows us to solve a particular routing problem. In the subsequent section, we show that the cuttree packing can be used in a way analogous to the probabilistic approximation of metrics by tree metrics in Section 8.5. In that section, we showed that given an algorithm to solve a problem on a tree metric, we could provide an approximate solution for the problem in a general metric with only an additional factor of O(log n) in the performance guarantee. Here we show that given an algorithm to solve a cut problem on trees, we can use a cuttree packing to ﬁnd a cut in general graphs with only an additional factor of O(log n) in the performance guarantee. We will see that the two results are intimately related. We use the algorithm to give an O(log n)approximation algorithm for the minimum bisection problem introduced in Section 8.4. Finally, in Section 15.4, we return to a special case of the sparsest cut problem called the uniform sparsest cut problem, and we show how using a vector programming relaxation leads √ to an O( log n)approximation algorithm for the problem.
15.1
Low distortion embeddings and the sparsest cut problem
In Section 8.5, we considered the idea of approximating a metric with a tree metric. Here we will consider approximating a general metric with another metric more general than a tree metric, namely an ℓ1 embeddable metric. Recall that if x ∈ ℜm and xi is the ith component of ∑m i the vector x then ∥x∥1 = i=1 x . 369
370
Further uses of cuts and metrics
Deﬁnition 15.1: A metric (V, d) is an ℓ1 embeddable metric (or embeds isometrically into ℓ1 ) if there exists a function f : V → ℜm for some m such that duv = ∥f (u) − f (v)∥1 for all u, v ∈ V . The function f is called the embedding of the metric into ℓ1 . It is not too hard to show that any tree metric is an ℓ1 embeddable metric, and that the converse is not true; we leave this as an exercise to the reader (Exercise 15.1). The ℓ1 embeddable metrics are very closely related to cuts. We will show that any ℓ1 metric (V, d) is a weighted sum of cuts of the vertex set V . To be clearer about we mean, let χδ(S) (u, v) be an indicator function for whether an edge (u, v) is in the cut S; that is, χδ(S) (u, v) = 1 if exactly one of u, v is in S and is 0 otherwise. Then we can show the following. Lemma 15.2: Let (V, d) be an ℓ1 embeddable metric, and let f : V → ℜm be the associated embedding. Then there exist λS ≥ 0 for all S ⊆ V such that for all u, v ∈ V , ∑ λS χδ(S) (u, v). ∥f (u) − f (v)∥1 = S⊆V
Furthermore, at most mn of the λS are nonzero, and the λS can be computed in polynomial time. Proof. We start by considering the simple case that f embeds V into one dimension and V = {1, 2, . . . , n}; i.e. f : {1, . . . , n} → ℜ. Let xi = f (i), and assume without loss of generality that x1 ≤ x2 ≤ · · · ≤ xn . Then we consider cuts {1},{1, 2},. . .,{1, 2, . . . , n − 1}, and let λ{1} = x2 − x1 , λ{1,2} = x3 − x2 ,. . .,λ{1,2,...,n−1} = xn − xn−1 . Then observe that for any i, j ∈ V , where i < j, j−1 ∑ ∑ xi − xj  = xj − xi = λ{1,...,k} = λS χδ(S) (i, j). k=i
S⊆V
For embeddings f into m > 1 dimensions, we generate the cuts and λS in each dimension using the process above as in the onedimensional case; i.e. we sort the vertices by the value of their component in that dimension, consider cuts S of the ﬁrst i points ordered in that dimension, and set λS to be the distance in that dimension between the (i + 1)st and ith points. Then certainly at most mn of the λS are nonzero. If xu = f (u), and xiu is the ith coordinate of xu , and λiS are the λ ∑ generated in the process for dimension i, we showed above that for any u, v ∈ V , xiu − xiv  = S⊆V λiS χS (u, v). Then we get that ∥f (u) − f (v)∥1 =
m ∑ i=1
xiu − xiv  =
m ∑ ∑
λiS χδ(S) (u, v) =
i=1 S⊆V
∑
λS χδ(S) (u, v),
S⊆V
as desired. Clearly the λS can be computed in polynomial time. Not every metric is ℓ1 embeddable, but we will show that every metric is in some sense not too far from an ℓ1 embeddable metric. Consider the following deﬁnition. Deﬁnition 15.3: A metric (V, d) embeds into ℓ1 with distortion α if there exists a function f : V → ℜm for some m and r such that r · duv ≤ ∥f (u) − f (v)∥1 ≤ rα · duv for all u, v ∈ V . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
15.1
Low distortion embeddings and the sparsest cut problem
371
The central result of this section is to show that there are relatively lowdistortion embeddings into ℓ1 for any metric. Theorem 15.4: Any metric (V, d) embeds into ℓ1 with distortion O(log n), where n = V . 2 Furthermore, the embedding f : V → ℜO(log n) is computable with high probability in randomized polynomial time. This result is in some sense the best possible. In Exercise 15.2, the reader is asked to show that there exist distance metrics that cannot be embedded into ℓ1 with distortion less than Ω(log n). To show an application of Theorem 15.4, we recall the sparsest cut problem, introduced earlier in Exercise 8.6. In the sparsest cut problem, we are given an undirected graph G = (V, E), costs ce ≥ 0 for all e ∈ E, and k pairs of vertices si , ti with associated positive integer demands di . We want to ﬁnd a set of vertices S that minimizes ∑ e∈δ(S) ce . ρ(S) ≡ ∑ i:S∩{si ,ti }=1 di That is, the sparsest cut problem ﬁnds a cut that minimizes the ratio of the cost of the edges in the cut to the sum of the demands separated by the cut. In the exercise, the reader is asked to show that the following is an LP relaxation of the problem and is solvable in polynomial time, where Pi is the set of all si ti paths: minimize
∑
ce xe
e∈E
subject to
k ∑
di yi = 1,
i=1
∑
xe ≥ y i ,
∀P ∈ Pi , 1 ≤ i ≤ k,
yi ≥ 0,
1 ≤ i ≤ k,
xe ≥ 0,
∀e ∈ E.
e∈P
Using this LP relaxation, we show below that we can obtain an O(log n)approximation algorithm for the sparsest cut problem via Theorem 15.4. Theorem 15.5: There is a randomized O(log n)approximation algorithm for the sparsest cut problem. Proof. To see this, suppose that we have an optimal solution (x, y) to the LP relaxation of the sparsest cut problem. Let dx (u, v) be the shortest path from u to v in the graph G using xe as the length of edge e. By Theorem 15.4, we can obtain in randomized polynomial time an 2 embedding f : V → ℜO(log n) such that for some r, ∥f (u) − f (v)∥1 ≤ r · O(log n) · dx (u, v) and ∥f (u) − f (v)∥1 ≥ r · dx (u, v) for all u, v ∈ V . By Lemma 15.2, given the embedding f , we can ﬁnd in polynomial time at most O(n log2 n) nonzero λS such that ∥f (u) − f (v)∥1 = ∑ ∗ S⊆V λS χδ(S) (u, v) for all u, v ∈ V . Let S be the minimum of ρ(S) over all S ⊆ V such that λS > 0; that is, ∑ e∈δ(S) ce ∗ ρ(S ) = min ρ(S) = min ∑ . S:λS >0 S:λS >0 i:S∩{si ,ti }=1 di Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
372
Further uses of cuts and metrics
∑ ∑ We will show∑that ρ(S ∗ ) ≤ O(log ∑n) OPT . Note that for any S ⊆ V , e∈δ(S) ce = e∈E ce · χδ(S) (e) and i:S∩{si ,ti }=1 di = i di · χδ(S) (si , ti ). Using this and Fact 1.10, we have that ∑ e∈δ(S) ce ∗ ρ(S ) = min ∑ S:λS >0 i:S∩{si ,ti }=1 di ∑ ce · χδ(S) (e) = min ∑ e∈E S:λS >0 i di · χδ(S) (si , ti ) ∑ ∑ S⊆V λS e∈E ce · χδ(S) (e) ≤ ∑ ∑k di · χδ(S) (si , ti ) S⊆V λS ∑ ∑i=1 e∈E ce S⊆V λS χδ(S) (e) = ∑k ∑ di S⊆V λS χδ(S) (si , ti ) ∑i=1 e=(u,v)∈E ce ∥f (u) − f (v)∥1 = ∑k i=1 di ∥f (si ) − f (ti )∥1 ∑ r · O(log n) e=(u,v)∈E ce · dx (u, v) ≤ , ∑ r · ki=1 di · dx (si , ti ) where the ﬁnal inequality follows since the function f embeds dx (u, v) into ℓ1 with distortion O(log n). Then noticing that ∑ xe ≥ dx (u, v) for all e = (u, v) ∈ E, and by the LP constraints yi ≤ dx (si , ti ) for all i and ki=1 di yi = 1, we have that ∑ e=(u,v)∈E ce · dx (u, v) ρ(S ∗ ) ≤ O(log n) ∑k di · dx (si , ti ) ∑ i=1 ce xe ≤ O(log n) ∑e∈E k d i yi ∑i=1 = O(log n) ce xe e∈E
≤ O(log n) · OPT .
If we look back at the proof above of Theorem 15.5, we can observe that we aren’t using all the properties required by a lowdistortion embedding. In the proof of the performance guarantee, we need that for every edge e = (u, v), ∥f (u) − f (v)∥1 ≤ r · O(log n)dx (u, v), but we only need that ∥f (si ) − f (ti )∥1 ≥ r · dx (si , ti ) for every i, 1 ≤ i ≤ k. With this weaker requirement, we can show that we can obtain a slightly stronger version of Theorem 15.4, and hence obtain a better approximation algorithm. In particular, we can obtain the following. Theorem 15.6: Given a metric (V, d) and k pairs of distinguished vertices, si , ti ∈ V , 1 ≤ i ≤ 2 k, we can in randomized polynomial time compute an embedding f : V → ℜO(log k) such that with high probability, ∥f (u) − f (v)∥1 ≤ r · O(log k) · duv for all u, v ∈ V and ∥f (si ) − f (ti )∥1 ≥ r · dsi ti for 1 ≤ i ≤ k. ( ) Note that it is a generalization of Theorem 15.4 since we can take all n2 pairs of vertices from V to be the si ti pairs, in which case the statement of Theorem 15.6 reduces to that of Theorem 15.4. Using the theorem above we can get the following improved approximation algorithm simply by repeating the analysis of Theorem 15.5. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
15.1
Low distortion embeddings and the sparsest cut problem
373
Corollary 15.7: There is a randomized O(log k)approximation algorithm for the sparsest cut problem. We now turn to the proof of Theorem 15.6. We start by explaining how to map a metric (V, d) to points in Euclidean space. To do this, we will use a Fr´echet embedding. Given (V, d) and a set of vertices A ⊆ V , deﬁne d(u, A) = minv∈A duv for every u ∈ V . Deﬁnition 15.8: Given a metric space (V, d) and p subsets of vertices A1 , . . . , Ap , a Fr´echet embedding f : V → ℜp is deﬁned by f (u) = (d(u, A1 ), d(u, A2 ), . . . , d(u, Ap )) ∈ ℜp for all u ∈ V. A Fr´echet embedding has the nice property that it is easy to compute given the sets Ai . It is also easy to show that the ℓ1 distance between points is at most the dimension p times the original distance in the metric. Lemma 15.9: Given a metric (V, d) and the Fr´echet embedding f : V → ℜp deﬁned above, for any u, v ∈ V , ∥f (u) − f (v)∥1 ≤ p · duv . Proof. For any A ⊆ V and any u, v ∈ V , let w be the closest point in A to v so that d(v, A) = dvw . Then d(u, A) ≤ duw ≤ duv + dvw = duv + d(v, A). Similarly, d(v, A) ≤ duv + d(u, A). Therefore d(u, A) − d(v, A) ≤ duv . Hence we have that ∥f (u) − f (v)∥1 =
p ∑
d(u, Aj ) − d(v, Aj ) ≤ p · duv .
j=1
Now the central problem of the proof is picking good sets Aj so that the distance between any si ti pair in the embedding is some reasonable fraction of the distance from si to ti . We will pick O(log2 k) such sets using randomization, then show that with high probability that the si ti distance in the embedding is at least Ω(log k)dsi ti . This will give the result of Theorem 15.6 by setting r = Θ(log k). The main lemma of the proof is as follows. Lemma 15.10: Given a metric space (V, d) with k distinguished pairs si , ti ∈ V , we can pick p = O(log2 k) sets Aj ⊆ V using randomization such that a Fr´echet embedding f : V → ℜp gives ∥f (si ) − f (ti )∥1 ≥ Ω(log k) · dsi ti for 1 ≤ i ≤ k with high probability. ∪ Proof. To deﬁne the sets Aj , let T = ki=1 {si , ti }. Assume that T  is a power of two; if it is not, we can pick some additional si ti pairs and add them to T until it is. Let τ = log2 (2k) so that T  = 2k = 2τ . Let L = q ln k for some constant q to be chosen later. Then we deﬁne the sets Atℓ by picking 2τ −t = 2k/2t vertices randomly from T with replacement, for ℓ = 1, . . . , L and t = 1, . . . , τ . We will show that the Fr´echet embedding f using these sets Atℓ has the desired properties. Note that we have chosen Lτ = O(log2 k) sets, as desired. Pick some i, 1 ≤ i ≤ k. We will show that ∥f (si ) − f (ti )∥1 ≥ Ω(Ldsi ti ) = Ω((log k)dsi ti ) with high probability. Then this inequality will hold for all i, 1 ≤ i ≤ k, with high probability. We deﬁne a closed ball B(u, r) of radius r around u ∈ V to be the vertices in T within distance r of u, so that B(u, r) = {v ∈ T : duv ≤ r}. The open ball B o (u, r) are all vertices in T that have distance strictly less than r from u, so that B o (u, r) = {v ∈ T : duv < r}. We deﬁne a particular set of distances rt with respect to the vertices si and ti (for the arbitrary Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
374
Further uses of cuts and metrics
r1
si
ti
r2 r3 r
Figure 15.1: An example of the choices of radii rt around si and ti . The radius r1 is the smallest such that B(si , r1 ) ≥ 2 and B(ti , r1 ) ≥ 2. The radius r2 is the smallest such that B(si , r2 ) ≥ 4 and B(ti , r2 ) ≥ 4. The radius r is the smallest such that B(si , r) ≥ 8 and B(ti , r) ≥ 8. Since r ≥ 14 dsi ti , but r2 is less than this, tˆ = 3, and rtˆ = r3 = 41 dsi ti . choice of i made above). We deﬁne r0 = 0, and let rt be the minimum distance r such that B(si , r) ≥ 2t and B(ti , r) ≥ 2t for t = 1, . . . , τ . Deﬁne tˆ to be the minimum index t so that rt ≥ 41 dsi ti ; we redeﬁne rtˆ = 14 dsi ti . Then note that the balls of radius rtˆ around si and ti don’t intersect, so that B(si , rtˆ) ∩ B(ti , rtˆ) = ∅. See Figure 15.1 for an illustration. Since B(si , rtˆ−1 ) ≥ 2tˆ−1 and B(ti , rtˆ−1 ) ≥ 2tˆ−1 and these balls also don’t intersect, if tˆ = τ , then B(si , rtˆ) = B(ti , rtˆ) = 2τ −1 as there are 2τ vertices total in T . The idea of the proof is to show that for any ℓ = 1, . . . , L and t = 1, . . . , tˆ, the randomly chosen set Atℓ has a constant probability of having an intersection with the ball of radius rt−1 around one of terminals si , ti and also having no intersection with the ball of radius rt around the other terminal. Then with constant probability, the distance from one terminal to Atℓ is at most rt−1 and the distance from the other terminal to Atℓ is at least rt , or d(si , Atℓ ) − d(ti , Atℓ ) ≥ rt − rt−1 . By applying a Chernoﬀ∑bound (from Section 5.10), we will be able to show that with high probability for t = 1, . . . , tˆ, L ℓ=1 d(si , Atℓ ) − d(ti , Atℓ ) ≥ Ω(L(rt − rt−1 )). This will allow us to conclude that with high probability, ∥f (si ) − f (ti )∥1 ≥
tˆ ∑ L ∑
d(si , Atℓ ) − d(ti , Atℓ )
t=1 ℓ=1 tˆ
≥
∑
Ω(L(rt − rt−1 ))
t=1
= Ω(Lrtˆ) = Ω(Ldsi ti ). Note that we only need to show the statement for t from 1 to tˆ, not from 1 to τ ; since the sum of rt − rt−1 telescopes to rtˆ and rtˆ = Ω(dsi ti ), this is suﬃcient. Assume that the ball around si deﬁnes the radius rt so that B(si , rt ) = 2t but B o (si , rt ) < t 2 ; otherwise, we can switch the roles of si and ti in the following discussion. Consider the event Etℓ , which occurs when the randomly chosen set Atℓ has Atℓ ∩ B o (si , rt ) = ∅ and Atℓ ∩ B(ti , rt−1 ) ̸= ∅. If Etℓ occurs, then it must be the case that d(si , Atℓ ) ≥ rt and d(ti , Atℓ ) ≤ rt−1 , which in turn implies that d(si , Atℓ ) − d(ti , Atℓ ) ≥ rt − rt−1 . We want to show that Pr[Etℓ ] is at least some constant. For notational simplicity, deﬁne G = B(ti , rt−1 ) (the “good set”), B = B o (si , rt ) (the “bad set”), and A = Atℓ . Then we know that G ≥ 2t−1 by the deﬁnition Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
15.1
Low distortion embeddings and the sparsest cut problem
375
of rt−1 , B < 2t by the assumption we made above on rt , and A = 2τ −t = T /2t ; these statements are true even in the case t = tˆ. Recall T  = 2τ . Then Pr[Etℓ ] = Pr[A ∩ B = ∅ ∧ A ∩ G ̸= ∅] = Pr[A ∩ G ̸= ∅A ∩ B = ∅] · Pr[A ∩ B = ∅] ≥ Pr[A ∩ G ̸= ∅] · Pr[A ∩ B = ∅]. Recall that the vertices in A are drawn randomly from T with replacement; if we want A∩B = ∅, then each vertex drawn for A must be from T − B. The probability that one vertex drawn from T is not in B is 1 − B T  . Thus ( Pr[A ∩ B = ∅] =
B 1− T 
)A . τ −t
Substituting bounds on the sizes of B and T , we obtain Pr[A ∩ B = ∅] ≥ (1 − 2t−τ )2 . Using (1 − x1 )x ≥ 14 for x ≥ 2, we get that Pr[A ∩ B = ∅] ≥ 14 as long as τ − t ≥ 1; this is certainly true if t ≤ tˆ < τ , while if t = tˆ = τ , we earlier observed that B = B o (si , rtˆ) ≤ 2τ −1 , so the statement still holds. Similarly, using 1 − x ≤ e−x , we obtain that ( ) G A t−1 t Pr[A ∩ G ̸= ∅] = 1 − 1 − ≥ 1 − e−GA/T  ≥ 1 − e−2 /2 = 1 − e−1/2 . T  Thus
1 Pr[Etℓ ] ≥ Pr[A ∩ G ̸= ∅] · Pr[A ∩ B = ∅] ≥ (1 − e−1/2 ) ≥ .098 4
for t ≤ tˆ. Now we apply a Chernoﬀ bound to show that the desired result occurs with high∑probability. Let Xtℓ be a 01 random variable which is 1 if event Etℓ occurs, and let Xt = L ℓ=1 Xtℓ for ∑ 1 ≤ t ≤ tˆ. Let µ = E[Xt ] ≥ .098L. Observe that L d(s , A ) − d(t , A ) ≥ X (r i i t t − rt−1 ). tℓ tℓ ℓ=1 Applying the Chernoﬀ bound of Lemma 5.26 and recalling that L = q ln k for some constant q, we get that the probability that Pr[Xt < 12 µ] ≤ e−L/8 = k −q/8 . Then with probability at least ∑ ˆ 1 − k −q/8 , L ℓ=1 d(si , Atℓ ) − d(ti , Atℓ ) ≥ .049L(rt − rt−1 ). Recalling that t ≤ τ = log2 (2k), we log2 (2k) have that with probability at least 1 − kq/8 , ∥f (si ) − f (ti )∥1 ≥
tˆ ∑ L ∑ t=1 ℓ=1
d(si , Atℓ ) − d(ti , Atℓ ) ≥
tˆ ∑
Ω(L(rt − rt−1 )) ≥ Ω(Ldsi ti ),
t=1
as explained earlier. Since this probability holds for one arbitrary si ti pair, the probability 2 (2k) that it holds for all si ti pairs is at least 1 − log . Choosing q suﬃciently large gives the result kq/8−1 with high probability. Given the lemmas above, the proof of Theorem 15.6 is straightforward. Proof of Theorem 15.6. We set r = Θ(L), where L = q ln k is the quantity used in the proof of Lemma 15.10. Then by that lemma ∥f (si ) − f (ti )∥1 ≥ r · dsi ti for all i with high probability. By Lemma 15.9, for all u, v ∈ V , ∥f (u) − f (v)∥1 ≤ O(log2 k)duv = r · O(log k)duv , and the theorem is shown. Note that we can consider embedding metrics (V, d) into metrics other tree metrics or √ ∑ than p i i p ℓ1 metrics. For instance, consider the ℓp metric, where x − yp = i (x − y ) . Then we have the following deﬁnition. Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
376
Further uses of cuts and metrics
Deﬁnition 15.11: A metric (V, d) embeds into ℓp with distortion α if there exists a function f : V → ℜm for some m and r such that r · duv ≤ ∥f (u) − f (v)∥p ≤ rα · duv for all u, v ∈ V . The exercises consider embedding into ℓp metrics with low distortion. Some recent work has shown that the sparsest cut problem may be quite diﬃcult to approximate. Theorem 15.12: Assuming the unique games conjecture, there is no αapproximation algorithm for the sparsest cut problem for constant α unless P = NP. However, some recent work has used semideﬁnite programming to achieve improved results for the sparsest cut problem. We discuss a special case of the sparsest cut problem called the uniform sparsest cut problem in Section 15.4, and show that in this case one can achieve an √ O( log n)approximation algorithm for the problem.
15.2
Oblivious routing and cuttree packings
In this section, we turn to routing problems. In routing problems, we are typically given an undirected graph G = (V, E) with capacities ce ≥ 0 for all edges e ∈ E. We are also given a set of demands duv for all u, v ∈ V ; each demand duv must be routed along some uv path in the graph G; we will consider the case in which each demand duv can be split and sent along several diﬀerent uv paths. The total amount of demand (or total ﬂow) using a given edge e ∈ E is the sum over all uv paths containing e of the amount of demand sent along that path. We would like to know if demands can be routed in such a way that the total ﬂow on each edge is at most its capacity; more generally, we would like to know by how much we would have to exceed the given capacities in order to route the ﬂows. The congestion ρ of a routing of demands is the minimum value such that the total ﬂow on each edge e is at most ρce . We would like to ﬁnd a routing that minimizes the overall congestion. The problem as stated thus far is solvable in polynomial time; one can write a linear program that will ﬁnd a routing with the minimum possible congestion. However, let us now consider a more diﬃcult variant of the problem in which we do not know the demands duv in advance. We must still ﬁnd a set of uv paths for each u, v ∈ V , and for each path we specify the fraction of the demand that will be sent on that path; for example, our output for u, v may be three uv paths P1 , P2 , P3 , and we specify that half the demand will be sent on P1 , a third on P2 , and the remaining sixth on P3 . We would like to ﬁnd paths such that the congestion of routing on these paths is close to the minimum possible congestion no matter what demands duv we are given. This is the oblivious routing problem. We will give an O(log n)approximation algorithm for the problem, using some of the ideas from the probabilistic approximation of metrics by tree metrics as discussed in Section 8.7. To begin, we will give some simple ideas about how to specify the paths, and then we will gradually reﬁne them into ideas that will allow for the approximation algorithm. To start, it will be easier to express the capacity of edges in the graph by letting cuv = ce for all e = (u, v) ∈ E, and cuv = 0 for all (u, v) ∈ / E. An initial idea on how to give the uv paths is via some spanning tree T in the graph G. Then for every u, v ∈ V , there is a unique uv path in T . Furthermore, we can give a simple way of deciding if routing demands on the tree will come within some factor α of the optimal congestion for any set of demands duv . Consider any tree edge (x, y) ∈ T . Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
15.2
Oblivious routing and cuttree packings
377
Removing edge (x, y) from T splits it into two connected pieces, and induces a cut in the graph, consisting of all edges that have exactly one endpoint in each piece. If S(x, y) is the set of vertices of one of the two connected pieces of ∑ T created by removing (x, y) from T , let C(x, y) be the capacity of this cut; that is, C(x, y) = u∈S(x,y),v∈S(x,y) / ∑ cuv . We observe that given any duv ; let us denote set of demands d, the amount of demand crossing this cut is u∈S(x,y),v∈S(x,y) / this amount (for a given set of demands d) by D(x, y). Thus the ﬂow per unit of capacity in the cut for any possible routing of this set of demands is at least D(x, y)/C(x, y). Since this is the ﬂow per unit capacity, some edge in this cut must have congestion at least D(x, y)/C(x, y) in any possible routing. Now we also observe that if we route paths in T along the paths uniquely deﬁned by the tree, then the total amount of demand using edge (x, y) is exactly D(x, y). Thus if the capacity cxy of edge (x, y) ∈ T is at least α1 C(x, y), then the congestion on edge (x, y) in our routing is D(x, y)/cxy ≤ αD(x, y)/C(x, y). Since the minimum congestion needed by any routing is at least D(x, y)/C(x, y), the congestion on edge (x, y) is then within a factor of α of the optimal congestion for routing the given set of demands d. Therefore, if for all (x, y) ∈ T , the capacity cxy of edge (x, y) is at least α1 C(x, y), then routing the demands on the tree has congestion at most α times the optimal congestion. Notice that this condition does not depend on the demand set d, and so if it holds, it will be true for any set of demands. To get the approximation factor α we desire, we modify our routing scheme in two ways. The ﬁrst ∑ is to give a collection of trees Ti and a weighting λi of them, such that λi ≥ 0 for all i and i λi = 1; see Figure 15.2. Now given any demand duv we send a λi fraction of it on the unique uv path in tree Ti for each i. For edge (x, y) ∈ Ti , let Ci (x, y) be the capacity of the cut induced by edge (x, y) in the tree Ti , and let Di (x, y) be the total demand crossing this cut for a given demand d; note that removing the same edge (x, y) from diﬀerent trees Ti may induce diﬀerent cuts. Then as argued above, for a given demand set d and for any i, the congestion of any routing of d is at least Di (x, y)/Ci (x, y), so any routing ∑ has congestion at least maxi Di (x, y)/Ci (x, y). The total ﬂow on edge (x, y) in our routing is i:(x,y)∈Ti λi Di (x, y). Thus if 1 ∑ λi Ci (x, y), (15.1) cxy ≥ α i:(x,y)∈Ti
then the congestion along edge (x, y) in our routing is ∑ ∑ Di (x, y) i:(x,y)∈Ti λi Di (x, y) i:(x,y)∈Ti λi Di (x, y) ≤ α∑ ≤ α max i cxy λ C (x, y) Ci (x, y) i:(x,y)∈Ti i i by Fact 1.10. If inequality (15.1) holds for all edges (x, y), then our routing will have congestion at most α times the optimal congestion. Although the condition given by inequality (15.1) is weaker than the condition on capacities for a single tree, in order to obtain the result we want we must modify our routing scheme in one additional way. For each tree Ti in our collection, and each edge (x, y) ∈ Ti , we specify a path in the graph G between the endpoints x and y. While this could be just the edge (x, y), it could also be some other path as well; for the tree Ti and edge (x, y) ∈ Ti , we denote the set of edges in this path by Pi (x, y). We now have i index both the tree i and the associated paths Pi (x, y) for all (x, y) ∈ Ti , so that a diﬀerent index i′ ̸= i may refer to the same tree (that is, Ti = Ti′ ) but with diﬀerent paths. For a given demand duv , to route the λi fraction of the demand in tree Ti , we consider the unique path between u and v in the tree, and then we concatenate the paths Pi (x, y) associated with each tree edge (x, y) along the path. This gives a path from u to v in the graph and we route the demand along this path; see Figure 15.2. Now Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
378
Further uses of cuts and metrics u u
v
v
v u weighting
1/2
1/3 u v
1/6
u
u
v
v
Figure 15.2: Two diﬀerent ways of specifying a routing. On top is a weighted collection of trees. To route demand from u to v, we send 1/2 the demand along the uv path in the ﬁrst tree, 1/3 along the uv path in the second tree, and a 1/6 in the last tree. On the bottom is a weighted collection of trees in which each tree edge has an associated path (shown as a dotted line only for edges on the uv path). To route demand from u to v, we send ﬂow along the concatenation of the paths associated with each in the uv path in the tree; we send 1/2 the demand along the concatenation of the paths in the ﬁrst tree, 1/3 along the concatenation of paths in the second tree, and 1/6 along the concatenation of paths in the last tree.
for any edge (u, v) in the graph, the total demand we route through it is the sum over all trees Ti of demand λi Di (x, y) for all edges (x, y) ∈ Ti that have edge (u, v) in the associated path Pi (x, y); that is, ∑ ∑ λi Di (x, y). i
(x,y)∈Ti :(u,v)∈Pi (x,y)
Now suppose that for any edge (u, v) ∈ E, we have that its capacity is at least an α fraction of the weighted cuts associated with the paths using (u, v); that is, cuv ≥
1∑ λi α i
∑
Ci (x, y).
(15.2)
(x,y)∈Ti :(u,v)∈Pi (x,y)
Then the congestion on (u, v) in our routing is ∑
i λi
∑
(x,y)∈Ti :(u,v)∈Pi (x,y) Di (x, y)
cuv
∑
∑
≤ α∑
∑
i λi i λi
(x,y)∈Ti :(u,v)∈Pi (x,y) Di (x, y) (x,y)∈Ti :(u,v)∈Pi (x,y) Ci (x, y)
≤ α max i
Di (x, y) . Ci (x, y)
As above, any routing will have congestion at least maxi Di (x, y)/Ci (x, y). Thus if inequality (15.2) holds for all (u, v) ∈ E, then our routing will have congestion within a factor α of the optimal congestion. To ﬁnd a collection of trees and associated paths, we will use a linear program. We have a decision variable α and decision variables λi . Each λi is the weight for a tree Ti together with paths Pi (x, y) for the edges (x, y) ∈ Ti . We will minimize α subject to the constraint that inequality (15.2) holds for all u, v ∈ V ; since cuv = 0 for (u, v) ∈ / E, we will not choose any Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
15.2
Oblivious routing and cuttree packings
379
paths using (u, v). The LP is then: minimize subject to
∑
λi
i
α
∑
Ci (x, y) ≤ αcuv ,
(x,y)∈Ti :(u,v)∈Pi (x,y)
∑
∀u, v ∈ V,
λi = 1,
i
λi ≥ 0,
∀i.
We call the collection of trees found by this linear program a cuttree packing. A cuttree is a weighted tree T such that the weight of the edge (x, y) is exactly C(x, y). One can view the LP above as trying to pack a convex combination of cuttrees into the graph G, stretching out each tree edge along a path, so as to minimize the overall factor by which each edge of G exceeds its capacity. We can now show nonconstructively that the LP has a solution of value O(log n). The nonconstructive proof will then give us ideas for an algorithm to ﬁnd the cuttree packing. This will prove that we can ﬁnd an O(log n)approximation algorithm for the oblivious routing problem. Lemma 15.13: The linear program has value O(log n). Proof. To prove this, we will take the dual of the linear program, then show that we can apply a result from the approximation of metrics by tree metrics in Section 8.7 to bound the value of the dual by O(log n). This will then bound the value of the primal by strong duality. The dual of the linear program above is: maximize z subject to
∑
cuv ℓuv = 1,
u,v∈V
z≤
∑
∑
Ci (x, y)
(x,y)∈Ti
ℓuv ,
∀i,
(u,v)∈Pi (x,y)
ℓuv ≥ 0,
∀u, v ∈ V.
We will now begin modifying the dual solution to the point at which we can apply our knowledge of tree metrics to the problem. We will treat the dual variables ℓuv as the length of an edge between u and v. It will be useful to think of the shortest path distance between two x, y ∈ V using edge lengths ℓuv ; we will denote this distance as dℓ (x, y). We ﬁrst observe that the dual variable z is smaller than the sum over (x, y) ∈ Ti of Ci (x, y) times the length of the associated path from x to y. Therefore, we only need to concentrate on trees whose associated xy paths are shortest paths using the edge lengths ℓuv . Thus for all trees Ti we have that ∑ z≤ Ci (x, y)dℓ (x, y). (x,y)∈Ti
We can rewrite the dual as ℓ≥0:
∑
max
u,v∈V
cuv ℓuv =1
min i
∑
Ci (x, y)dℓ (x, y).
(x,y)∈Ti
Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
380
Further uses of cuts and metrics
∑ We would now like to drop the restriction on the lengths ℓ that u,v∈V cuv ℓuv = 1. Observe ∑ that if we have lengths ℓ ∑ such that u,v∈V cuv ℓuv = β > 0, then if we multiply all the lengths ℓuv by 1/β, we have that u,v∈V cuv ℓuv = 1. The shortest path lengths dℓ (x, y) also change by the same factor 1/β, so that the dual objective function changes by a factor of 1/β. Thus we can rewrite the dual still further as ∑ (x,y)∈Ti Ci (x, y)dℓ (x, y) ∑ . max min i ℓ≥0 u,v∈V cuv ℓuv We recall from Theorem 8.25, combined with Theorem 8.19, that for any nonnegative set of costs cuv and any distance metric ∑ dℓ , we can ﬁnd a tree metric ∑ (V, T ) such that dℓ (u, v) ≤ Tuv for all u, v ∈ V and such that u,v∈V cuv Tuv ≤ O(log n) u,v∈V cuv dℓ (u, v). With two more observations, we will be done. First, we observe that dℓ (u, v) ≤ ℓuv for all u, v ∈ V since the shortest path length between u and∑v is no more than the length ∑ of the uv edge. Second, since dℓ (x, y) ∑ ≤ Txy for all x, y ∈ V , ∑(x,y)∈Ti Ci (x, y)dℓ (x, y) ≤ (x,y)∈Ti Ci (x, y)Txy . Further, observe that (x,y)∈Ti Ci (x, y)Txy = u,v∈V cuv Tuv , since any u, v ∈ V contributes cuv exactly to the cuts Ci (x, y) corresponding to the edges (x, y) on the unique path in Ti from u to v. Putting all of this together, we have that for any edge lengths ℓ ≥ 0, we have that ∑ ∑ ∑ ∑ Ci (x, y)dℓ (x, y) ≤ cuv Tuv ≤ O(log n) cuv dℓ (u, v) ≤ O(log n) cuv ℓuv . (x,y)∈Ti
u,v∈V
u,v∈V
u,v∈V
(15.3) Hence for any edge lengths ℓ ≥ 0, there exists some tree Ti such that ∑ (x,y)∈Ti Ci (x, y)dℓ (x, y) ∑ ≤ O(log n). u,v∈V cuv ℓuv This proves that the optimal value of the dual is O(log n), and since by strong duality, the value of the primal is equal to the value of the dual, the original primal LP has value O(log n). Finally, we can use our understanding from the lemma above to ﬁnd an appropriate collection of trees and corresponding paths such that we have an O(log n)approximation algorithm. One idea is to use the ellipsoid method of Section 4.3 to solve the dual linear program; the proof above shows that in a sense the tree metric algorithm can ﬁnd a violated constraint if z > Ω(log n). However, if the optimal z is much smaller than O(log n), we may not be able to ﬁnd violated constraints. The idea for our approximation algorithm is to set up a diﬀerent linear program so that we can indeed use the tree metric algorithm as a separation oracle for the ellipsoid method. We observe which tree metrics are used in deﬁning the violated constraints, and then we ﬁnd a cuttree packing using only these trees; we show that this is suﬃcient to obtain the approximation algorithm. Theorem 15.14: There is an O(log n)approximation algorithm for the oblivious routing problem. Proof. By the proof of the lemma above, we know that the value of the linear program is at most Z = 4 · 96 ln(n + 1), since Theorem 8.25 and Theorem 8.19 show that ∑ given a metric dℓ deﬁned by edge u,v∈V cuv Tuv ≤ ∑ lengths ℓ, we can ﬁnd a tree metric (V, T ) such that 4 · 96 ln(n + 1) u,v∈V cuv dℓ (u, v). We now apply the ellipsoid method to a slightly diﬀerent linear program. Consider the linear program as follows, where I is the set of indices of all possible trees and associated paths, and Z is treated as a constant. We are only looking for a Electronic web edition. Copyright 2011 by David P. Williamson and David B. Shmoys. To be published by Cambridge University Press
15.2
Oblivious routing and cuttree packings
381
feasible solution to the problem, so we use the objective function of minimizing the constant 0. Then the linear program is: minimize subject to
∑
<