Mar 23, 2009 - this problem as âadwordsâ (also done in other literature). This problem has attracted much interest in the research community in the last few ...
Sponsored Search with Reliable Query Estimates∗ Maria Montserrat Medina Martinez Borja M. Peleato-Inarrea Eldar Sadikov† March 23, 2009
1
Introduction
We study the problem of sponsored search, the online advertising mechanism employed by search engines such as Google, Yahoo, and MSN. It is essentially a large auction where individuals and businesses place bids on search keywords along with the limits on their maximum daily budget. The search engine companies earn money when they display an ad in response to a relevant search query and user clicks on the ad. More formally, the problem can be defined as follows. There is a set of advertisers A and a set of keywords W . Each advertiser i places bids bij on each keyword j ∈ W and specifies a maximum daily budget Bi . A sequence q1 q2 . . . qM of query words qj ∈ Q arrive in an online fashion during the day, and each query needs to be assigned to some advertiser i (such that the revenue is biqj ). The goal is to maximize the total revenue for the whole sequence Q of a given day while not exceeding the maximum daily budgets of the advertisers. From now on, we are going to refer to this problem as “adwords” (also done in other literature). This problem has attracted much interest in the research community in the last few years. A number of algorithms have been proposed [5, 4, 8] (we provide an overview of related work in the next section). However, as argued in reference [6], many papers have focused on the worst-case analysis of algorithms while ignoring that there is usually some empirical information about the input. In this work, we attempt to improve on the recently proposed algorithm by Mehta, Saberi, et. al. (from now on referred to as MSVV) [8] assuming that we have some known distribution of queries over keywords from the past experience. For this, we introduce the concept of advertiser flexibility. Intuitively, we define flexibility as a measure of how “desirable” the advertiser is to a search engine in terms of the keywords that she is bidding on. We model flexibility in terms of shortage of budget money for a set of keywords the advertiser is bidding on. Keywords for which there is a greater demand than there is supply are considered to be in “shortage.” Then if an advertiser is bidding on many keywords that have shortage, she is flexible. On the other hand, if the keywords she is bidding on have surplus (i.e. there is more supply for them than demand), then she is not flexible and, hence, is not of such importance to the search engine. Accordingly, the search engine will be interested to spend the money of inflexible bidders first. The reasoning above demonstrates the intuition behind the Flexus algorithm we propose in this paper that improves on the MSVV algorithm. In this paper, we provide a bound on the performance of Flexus in the worst case and describe the experiments we conducted to evaluate ∗ Prepared † Not
as part of CME305 class project taking the class
1
the improvement of Flexus over MSVV, given reliable estimates on the number of queries over keywords.
2
Related Work
The adwords problem can be seen as a generalization of online bipartite matching problem. The bipartite matching problem assumes unit bids and unit budget for each advertiser and tries to find maximum-cardinality matching for a given bipartite graph. Hopcropt and Karp presented a polynomial time algorithm for the offline version of this problem in reference [3]. The adwords problem, however, is more complicated because it is online. The greedy algorithm, in this case, achieves 1/2 competitive ratio (defined as the revenue earned by the online algorithm divided by the revenue earned by the offline algorithm). On the other hand, as shown by Karp [5], the competitive ratio of any online randomized algorithm is upper bounded by 1 − 1/e, and, as shown by Kalyanasundaram and Pruhs [4], the competitive ratio of any deterministic online algorithm is also upper bounded by 1 − 1/e. Both Karp and Kalyanasundaram have proposed algorithms with this competitive ratio, RANKING and BALANCE, respectively [5, 4]. The original BALANCE algorithm assumed unit bids, which is not realistic in practice. Hence, Mehta, Saberi, et. al. proposed MSVV, a more generalized version of BALANCE algorithm that maintains the same 1 − 1/e competitive ratio but works for arbitrary bids as long as the bids are much smaller than the budgets [8]. In our work, we adopt their algorithm and show how bidder flexibility can be incorporated to take advantage of the stochastic information without significantly penalizing the worst case performance. There have been attempts to include stochastic information into the online matching algorithms before. Buchbinder et. al. as one of the generalization of the MSVV algorithm proposed incorporating average fraction of the daily budget spent by an advertiser [1]. However, this approach falls short in cases when an advertiser changes the set of keywords she bids on and does not work in cases where there is no past information on a particular bidder. We take a more realistic approach and instead rely on the distribution of queries over keywords. Such information should always be available to a search engine and is generally not susceptible to decisions made by the bidders. This approach is similar to the approach taken by Mahdian et. al., where they assume to have an average number of queries for each keyword and propose a variation of MSVV algorithm that achieves a trade-off between the worst-case performance when the estimates are unreliable and the best case performance when the estimates are accurate [7]. Nonetheless, Mahdian et. al. do not focus on how past information can best be exploited but rather focus on how to maintain a good worst-case performance when the estimates are unreliable. The solution we propose, on the other hand, not only provides worst-case guarantees, but also specifies in detail how previous estimates can be best used beyond the simple offline linear programming solution proposed by Mahdian et. al.
3 3.1
The Flexus Algorithm Algorithm
We are going to use MSVV algorithm with a few changes [ref]. Recall that for a given query, MSVV allocates it to an advertiser i with the highest bij · ψ(T (i)) score among those bidding on the query keyword j. The function ψ is defined as ψ(T (i)) = 1 − eT (i)−1 where T (i) is the fraction of the advertiser i’s budget spent so far. The Flexus algorithm will use a different scoring function ˆbij · ψ(Tˆ(i)) that weighs advertisers’ budgets and bids based on their inflexibility If (i) ∈ (, 1] for some > 0. More specifically,
2
ˆbij = If (i) · bij and Tˆ(i) =
If (i)·mi If (i)·Bi ,
where mi is the amount of money spent so far by the advertiser i and Bi is the total budget of i. Note that in the formula for Tˆ(i), If (i) cancels out , so Tˆ(i) = T (i). Hence, Flexus picks the advertiser based on the following scoring function: bij · If (i) · ψ(T (i))
(1)
We provide details on how we compute If (i) in Section 3.3, but for now we treat If (i) as some function of the advertiser i, that remains constant during the execution of the algorithm. In the next section, we are going to show how we can bound the worst-case performance of Flexus with respect to mini∈A If (i).
3.2
Worst Case Competitive Ratio
Lemma 1. Let P be an instance of the adwords problem defined by a set of keywords W and a set of advertisers A where each advertiser i places bids bij on each keyword j ∈ W and has a maximum daily budget Bi . Let Pˆ be the same adwords problem as P but with the budget and bids of each advertiser i scaled by If (i), as defined by the Flexus algorithm. For any sequence of queries Q, Flexus applied onto P generates the same query-bidder assignments as MSVV applied onto Pˆ . Proof. The budget and bids of each advertiser i have been scaled by the same factor If (i) in Pˆ as Flexus scales them in P . Moreover, If (i) remains constant throughout both algorithms. Hence, both Flexus in P and MSVV in Pˆ applied to Q select the same advertiser and take the same proportion of budget from the selected advertiser for each query q. Thus, MSVV applied onto Pˆ produces exactly the same sequence of query-bidder assignments as Flexus applied onto P . Corollary 1. Flexus applied onto P generates at least as much revenue as MSVV applied onto Pˆ . Proof. From Lemma 1, we have that Flexus in P generates the same query-bidder assignments as MSVV in Pˆ . However, while MSVV charges each selected advertiser i for each query q If (i) · biq , Flexus charges her biq instead. Since If (i) ≤ 1, Flexus generates at least as much revenue as MSVV. Theorem 1. The competitive ratio of the Flexus algorithm is at least mini∈A If (i) · (1 − 1e ). Proof. Given a sequence of queries Q, let QB be the set assignments made by OPT applied onto P and assume we obtained bi(q)q dollars of revenue for each query P q. Then if ROP T (P ) denotes the optimal revenue that could be obtained on P , ROP T (P ) = q∈Q bi(q)q . Now consider the set of query-bidder assignments QB applied onto Pˆ . Observe that QB is still feasible with Pˆ because for each query q we subtract the same fraction of budget from each assigned advertiser as before, so we should never prematurely drain anybody’s budget and every assignment made by QB should still be feasible for all q in Q. Now using QB on Pˆ , we obtain at least min(If ) · bi(q)q dollars on P each query q. Hence, the total revenue RQB (Pˆ ) = q∈Q ˆbi(q)q of QB applied onto Pˆ is at least P equal to mini∈A If (i) q∈Q bi(q)q . Therefore, RQB (Pˆ ) ≥ mini∈A If (i)ROP T (P ). By Corollary 1, we know that the revenue of Flexus RF (P ) applied to the problem P is greater than the revenue RM SV V (Pˆ ) of MSVV applied to Pˆ . But we know from reference [8], that RM SV V (Pˆ ) ≥ (1 − 1/e) · ROP T (Pˆ ), where ROP T (Pˆ ) is the revenue earned by OPT applied to Pˆ . Furthermore, ROP T (Pˆ ) ≥ RQB (Pˆ ), so RM SV V (Pˆ ) ≥ (1 − 1/e) · RQB (Pˆ ). Combining the latter with RQB (Pˆ ) ≥ mini∈A If (i)ROP T (P ), gives us: 1 1 1 RF (P ) ≥ RM SV V (Pˆ ) ≥ (1 − )ROP T (Pˆ ) ≥ (1 − )RQB (Pˆ ) ≥ min If (i)(1 − )ROP T (P ) i∈A e e e From this, we conclude that the competitive ratio of our algorithm is at least mini∈A If (i)(1 − 1e ). 3
3.3
Inflexibility
Recall that A is the set of advertisers, W is the set of keywords, and Q be the sequence of queries which the algorithm receives in an online fashion. Each query in Q is for a keyword in W . As before, let the budget of each advertiser i be Bi , and the bid of advertiser i on keyword j be bij . The goal is to maximize revenue, i.e. the sum of bi(q)q ’s over all queries q ∈ Q, where i(q) is the advertiser allocated to query q. We are assuming we have past experience query distribution over keywords. Let d(j) be the average number of queries for word j on a given day. This is effectively an estimate of demand for keywords. To estimate supply for keywords, we need to estimate how much of each advertiser’s budget should be allocated towards each query. One possible way to make such estimate is try to maximize revenue assuming the mean case, i.e. assuming we have mean number of queries for each keyword in the query sequence Q. Then as shown in [7], to find the number of queries xij for each keyword j that should be allocated to each bidder i, we can solve the following maximization program with integrality constraints on xij ’s: P bij xij maximize i∈A,j∈WP subject to ∀j ∈ W :P i∈A xij ≤ d(j) ∀i ∈ A: j∈W bij xij ≤ Bj ∀i ∈ A, ∀j ∈ W : xij ≥ 0 Although linear programming gives us real-valued xij ’s, whereas what we want is integer-valued xij ’s, assuming bids are small compared to budgets, LP above gives us a relatively tight upperbound estimate on the optimal (in the mean case) allocation of queries across advertisers as to minimize the total shortage across keywords. However, the maximization problem as defined above does not fully distribute all the available budgets. For example, if an advertiser i with budget of $3 is bidding $1 on one word wj and we expect only 1 query for wj , then LP above will only allocate $1 of i’s budget and leave $2 untouched. To accurately estimate supply for all keywords though, we need to distribute all the remaining budget. Therefore, we are going to solve the second maximization problem identical to the first one but with P d(j) constraints removed: maximize i∈A,j∈W Pbij xij ∀i ∈ A: j∈W bij xij ≤ Bj ∀i ∈ A, ∀j ∈ W : xij ≥ 0 This second LP is going to allocate remaining budget Bi of each advertiser i towards the keyword she placed the highest bid on. Since we run this LP after the first LP, money will only be allocated to the keywords that have already been saturated (there is no shortage on them). Although this allocation scheme is not perfect since it does not respect other saturated keywords that an advertiser might be bidding on, it should in general provide us a good estimate of surplus across keywords. Using the xij ’s produced by the two LPs, let’s define supply s(j) of a keyword j as: X s(j) = xij (2) i∈A
Now using the supply and demand of a keyword, we can define shortage of a keyword as a ratio of the keyword demand to the keyword’s supply weighted by the keyword’s relative popularity: short(j) =
d(j) s(j)
max d(j) d(j) j∈W
where the relative popularity of keyword j is measured as
4
d(j) maxj∈W d(j) .
(3)
Now for each keyword, we have an estimate of its shortage in the average case. If Wi is the set of keywords an advertiser i is bidding on, using the keyword shortages, we define flexibility f (i) of an advertiser i as follows: P P P short(j) j∈Wi |Wj | j∈Wi short(j)|W | j∈Wi short(j) f (i) = |Wj | P = |Wj | P = |W | P (4) short(j) short(j)|W | j j∈W j∈W short(j) j∈W
|W |
Intuitively, the flexibility of an advertiser is the average shortage of keywords she is bidding on as a percentage of average shortage across all keywords scaled by the number of keywords an advertiser is bidding on. The reason why we use an average as opposed to just a sum of shortages on advertiser’s keywords is to avoid the flexibility becoming too small as a percentage of shortages accross all keywords. Now that we have defined an advertiser’s flexibility, we can define inflexibility If (i) of an advertiser i as follows: (5) If (i) = (1 − p) + pe−f (i) In this formula, inflexibility factor p is is a design parameter that bounds mini∈A If (i) within (1 − p, 1]. Depending on how much we trust past experience query distribution over keywords and how we want to bound our worst case performance (where worst case performance is lower bounded by mini∈A If (i)(1 − 1e ) as proved in Theorem 1), we can adjust p either up or down. In the next section, we are going to show that when the past query distribution over keywords is reliable and numbers of queries for keywords tend to have a manageable variance, by increasing p, we can improve the average case performance of the algorithm.
4
Experiments
We conducted a number of experiments to see if our model improves MSVV algorithm’s performance given reliable stochastic information about the number of queries for keywords. For our experiments, we use synthetic data where we use distributions initialized with random parameters and query sequences generated randomly from these distributions.
4.1
The Approach
In order to generate the data, we developed a system that relies on three different types of distributions: 1. Distributions of queries for each keyword 2. Distribution of bidders across keywords 3. Distribution of budgets across advertisers We assume that each of the components of the distributions above, i.e. number of queries for a keyword, number of bidders for a keyword, and budget of an advertiser, are all jointly independent random variables. Since the number of random variables in each of these distributions is large (which is true for popular search engines) and there is always an upper bound on all of the means (since it is real data) and, finally, since the sum of the variances does not converge to 0, by Central Limit Theorem for independent random variables, all of these distributions should tend to Normal [9]. The random variables N1 , N2 , . . . , N|W | corresponding to the number of bidders for each keyword are identically independently distributed as per the Normal distribution with some mean 2 . Likewise, the budgets B1 , B2 , . . . , B|A| are also identically independently µN and variance σN 5
2 distributed random variables as per the Normal distribution with the mean µB and variance σB . Finally, the number of queries for each keyword j in our design is a random variable drawn from its own Normal distribution (specific to j) with the mean µj and σj2 . Now let’s describe the experiment design for bids. We assume that the bids are uniformly ran(maxBid − BidV ar) and maxd(j) maxBid, domly distributed in the range between maxd(j) j∈W d(j) j∈W d(j) where d(j) is the demand for keyword j estimated by the mean number of queries for j. This setup ensures that the bids are proportional to the popularity of keywords but at the same time exhibit enough variance to differentiate bidders in terms of their bids. Finally, the number of keywords each bidder is bidding on is uniform randomly distributed between 1 and some maximum threshold defined as a percentage of |W |. Another important condition in our experiment design is that the distribution of the advertisers and their budgets should not perfectly line up with the expected distribution of keywords in Q. This implies that some keywords have excess of bidders and budget and other keywords have shortage. This condition is realistic (otherwise adwords problem would not be important) and guarantees that Flexus makes decisions different from the MSVV algorithm (since with this condition inflexibility is different among advertisers).
4.2
Setup
The experiment setup was as follows: • |W | = 150 • |Q| = 30, 000 • |A| = 75 • maxBid = $5 • bidV ar = $1 • A maximum number of keywords each advertiser can bid on was upper bounded by 0.1·|W | = 15 • p = 0.25 (i.e. mini∈A If (i) = 0.75 and worst case competitive ratio is lower bounded by 0.4741) • Number of random Q’s generated per test was 30 Query Distribution for Each Keyword We generated a Normal distribution with standard deviation 51 · |W | for µj ’s, i.e. is the mean number of queries over keywords. After this, for each keyword j, we randomly generated σj as a percentage of µj . In our setup, we upper bounded σj by 51 µj (from now on, we denote 51 as maxFactor ) and uniformly at random selected σj from (0, maxF actor · µj ]. Distribution of Advertisers among Keywords We choose the mean of the distribution of advertisers to be the same as that of the queries, i.e. the same set of keywords are deemed most popular by both the search engine users and search engine advertisers. However, we chose the standard deviation σN of the advertisers to be 16 · µN , i.e. slightly smaller than the standard deviation of the queries. The intuition behind this was as follows: any given advertiser bids on at least one keyword, but there may always be keywords with no advertisers for them, so if we assume that the overall budget available to a search engine is proportional to the overall demand for queries, the distribution of advertisers across keywords 6
should be more concentrated on the most popular words. Distribution of Budgets among Advertisers We chose the cumulative budget available to the search P engine P to be equal to E[b] · |Q|, where E[b] 1 is the expected (average) bid calculated as |A|·|W i∈A j∈W bij . To guarantee that no adver| tiser has a small budget relative to the bids (an important assumption for the adwords problem as specified in reference [8]), we first uniformly allocated 1/10 of the available total budget among all advertisers. Then we distributed the remaining 9/10 of the total budget according to the Normal distribution with a random mean µB and standard deviation σB equal to 71 µB . Our intuition for choosing a relatively small σB was because we expect a relatively low number of advertisers with high budget in relation to the total number of advertisers (we hypothesize that the budgets might even follow a power-law distribution). As specified in Section 3.3, to estimate the shortage of each keyword, we solved two Linear Programs using the generated Normal distribution of queries among keywords (from now on, referred to as offline query distribution). To solve LPs, we used CVX package for Matlab [2]. Figure 1 shows a sample offline query distribution among keywords together with keyword shortages and surpluses.
Figure 1: Example of an offline query distribution generated by solving the LP problems With the shortages as calculated above for each keyword, we are now able to calculate inflexibility of each bidder and thus, test Flexus versus MSVV. In what follows, we describe the results of experiments where we produce a series of online query sequences (30 total for each test as defined above). Each sequence of queries may have different length which accounts for that there may be different number of queries on different days. The number of times the keyword appears in a query sequence is drawn from its corresponding distribution. We then randomly permute the bag of keywords to produce an online order for each query sequence.
4.3 4.3.1
Results Performance of Flexus in Relation to MSVV
Figure 2 shows total revenue earned by both algorithms along with the optimal revenue estimated by LP. One can observe that Flexus consistently outperforms MSVV. In our experiments, we have run many other tests with other query sequences and Flexus was consistently better than
7
Figure 2: Total revenue achieved by Flexus and MSVV and LP approximation of optimal revenue for each of the 30 generated query sequences. MSVV whenever there was enough shortage and surplus among keywords such that differences in inflexibility among advertisers was significant enough to influence Flexus’ choice compared to the MSVV.
Figure 3: Competitive Ratio achieved by Flexus and MSVV over 30 generated query sequences. To illustrate the competitive ratio of Flexus, consider Figure 3. Figure 2 is the mirror of Figure 3 but demonstrates the improvement of Flexus over MSVV in terms of the competitive ratio percentage. Over 30 generated query sequences, the competitive ratio of Flexus is on average 0.37% better than that of MSVV. Although the improvement is very small, from the Figure 3 we can see that the competitive ratio of MSVV, in this case, is already high in the first place, thus there is not much room for improvement. More importantly, the true competitive ratio would be even higher considering that we are using an upper-bound approximation of the offline optimal solution obtained by LP. 4.3.2
Performance of Flexus as a Function of Inflexibility Factor p
If the stochastic data about query distribution among keywords is reliable (in this setting, we upper bound the variance of queries for each keyword j by 51 µj , i.e. maxF actor = 51 ), it is interesting 8
to look at how the average performance of Flexus changes as we vary p. The higher the value of p, the more trust we put into our previous experience estimates, thus, granted we have accurate estimates, the better performance we should expect from Flexus. On the other hand, the higher p, the “riskier” we are, since the worst case competitive ratio of Flexus is inversely related to p.
Figure 4: Improvement in competitive ratio achieved by Flexus over MSVV for different values of inflexibility factor p. Figure 4 shows the experimental results along with a cubic data fit, where p is plotted on x-axis and competitive ratio improvement of Flexus over MSVV (measured in %) is plotted on the y-axis. As expected, the improvement becomes more significant when p is large. 4.3.3
Performance of Flexus as We Vary Reliability of Estimates
As noted in the previous subsection, although Flexus can significantly benefit from the reliable query distribution estimates, a balance should be found as to retain a good competitive ratio in case of unreliable estimates. One possible way to make the estimates less reliable is to increase the variance of the keywords. Since the shortages of keywords are measured in terms of the mean number of queries, the higher the variance on each keyword, the less reliable the mean number of queries is, thus the less accurate our measure of the advertisers’ flexibility should be. Accordingly, as σj ’s increase, we expect our algorithm to perform worse. Recall that the maxF actor · µj defines an upper bound on the variance of queries for the keyword j. For this experiment, instead of using maxF actor · µj as an upper bound, we set the variance of queries on each keyword j strictly equal to maxF actor · µj and observe performance of Flexus as we increase maxF actor. Figure 5 shows the results along with a cubic data fit, just like in the previous experiment, where maxF actor is plotted on x-axis and competitive ratio improvement of Flexus over MSVV (measured in %) is plotted on the y-axis. To obtain these results, the inflexibility parameter was fixed at p = 0.5 (this provides a “riskier” strategy which should give better results for reliable estimates and possibly worse results for unreliable estimates). As expected, Figure 5 shows a deterioration in Flexus’ performance as we increase maxF actor, i.e. variance in the number of queries for keywords. However, to our own surprise, the performance of Flexus did not deteriorate as much as we anticipated.
9
Figure 5: Improvement in competitive ratio achieved by Flexus over MSVV for different values of maxF actor.
5
Conclusions
In this paper, we studied the problem of sponsored search. We first provided an overview of state of the art in the field and showed that although there exist algorithms that achieve the highest possible worst case competitive ratio, the problem of exploiting past information about the input has not been studied in depth yet. Thus, we proposed the new Flexus algorithm that exploits estimates of keyword frequencies to improve the average performance of MSVV. We have formally bounded the worst case competitive ratio of Flexus at mini∈A If (i) · (1 − 1e ), where mini∈A If (i) is a design parameter that can be easily adjusted. We conducted a number of experiments where we studied how well Flexus can benefit from the past experience estimates. Our experiments have demonstrated that Flexus consistently improves MSVV’s competitive ratio up to 1%, granted that there is accurate query distribution over keywords and keywords differ in terms of shortage/surplus among each other. In this work, we looked at the adwords problem from the search engine’s point of view. Flexibility, as defined, measures how much the search engine can benefit from retaining a particular advertiser. Hence, Flexus tends to give priority to the less flexible advertisers, which may send the wrong message to the advertisers. An advertiser bidding on a keyword will want to be displayed before its competition and may be incentivized to bid on surplus keywords. However, since the advertiser does not have the information on the keywords’ shortages and surpluses, she may be unable to game the system. Moreover, without the shortage/surplus information, Flexus mechanism from the outside may be simply seen as an incentive to bid on more keywords (since the more keywords an advertiser bids on, the more flexible she is). As a future direction, we could try to bound the average case (expected) performance of Flexus and compare it to the average case (expected) performance of MSVV (if such bound can be found for MSVV). Although this may prove to be challenging and require stochastic optimization modelling, this may provide a theoretical guarantee on the expected performance of Flexus and more formally demonstrate its merits. Furthermore, although our definition of advertiser’s flexibility has been shown to be effective, a more intelligent measure of flexibility could be considered. For example, when estimating inflexibility, we could weigh keyword shortages based on the keyword variance. We would give more weight (trust) to those keywords that have small variance. Additionally, when estimating surplus, we could use a more balanced approach to distribute remaining budget, since LP definition we are using may give too much importance to the keywords with highest bids.
10
References [1] N Buchbinder, K Jain, and J Naor. Online primal-dual algorithms for maximizing ad-auctions revenue. In Proceedings of the 15th Annual European Symposium on Algorithms (ESA, pages 253–264, 2007. [2] Cvx: Matlab software. http://www.stanford.edu/~ boyd/cvx. [3] J E Hopcroft and R M Karp. An n 5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on Computing, (2):225–231, 1973. [4] Bala Kalyanasundaram and Kirk R. Pruhs. An optimal deterministic algorithm for online b-matching. Theor. Comput. Sci., 233(1-2):319–325, 2000. [5] R. M. Karp, U. V. Vazirani, and V. V. Vazirani. An optimal algorithm for on-line bipartite matching. In STOC ’90: Proceedings of the twenty-second annual ACM symposium on Theory of computing, pages 352–358, New York, NY, USA, 1990. ACM. [6] Elias Koutsoupias and Christos H. Papadimitriou. Beyond competitive analysis. SIAM J. Comput., 30(1):300–317, 2000. [7] Mohammad Mahdian, Hamid Nazerzadeh, and Amin Saberi. Allocating online advertisement space with unreliable estimates. In EC ’07: Proceedings of the 8th ACM conference on Electronic commerce, pages 288–294, New York, NY, USA, 2007. ACM. [8] Aranyak Mehta, Amin Saberi, Umesh Vazirani, and Vijay Vazirani. Adwords and generalized online matching. J. ACM, 54(5):22, 2007. [9] Sheldon Ross. First Course in Probability, A (8th Edition). Prentice Hall, 8 edition, November 2008.
11
