Searchstrings revealing User Intent A Better Understanding of User Perception Carsten Stolz
Michael Barth
Katholische Universitat ¨ Eichstatt-Ingolstadt ¨ Germany
Ludwig-Maximilian-Universitat ¨ Munchen, ¨ Germany
[email protected] Maximilian Viermetz
[email protected] Klaus D. Wilde
Heinrich-Heine-Universitat ¨ Dusseldorf, ¨ Germany
Katholische Universitat ¨ Eichstatt-Ingolstadt ¨ Germany
[email protected]
[email protected]
ABSTRACT The evaluation of information driven websites by analysis of serverside available data is the objective of our approach. In our former work we developed techniques for evaluation of non-transactional websites by regarding the author’s intentions and using only based on implicit user feedback. In several case studies we got aware that in single cases unsatisfied users had been evaluated positively. This divergence could be explained by not having considered the user’s intentions. We propose in this approach to integrate search queries within referrer informaiton as freely available information about the user’s intentions. By integrating this new source of information into our meta model of website structure, content and author intention, we enhance the formerly developed web success metric GPI. We apply well understood techniques such as PLSA for text categorization. Based on the latent semantic we construct a new indicator evaluating the website with respect to the user intention. By ranking all webpages with respect to the user intention manifested in the search query, we acchieve an individualized measure to evaluate a session by the user’s initial intention. In contrast to manual assignments of weights by the website author, our proposed measure is purely calculated allowing a generic assessment of websites without manual intervention. In a case study we can show, that this indicator evaluates the quality and usability of a website more accurately by taking the user’s goals under consideration. We can also show, that the initially mentioned diverging user sessions, can now be assessed according to the user’s perception. Due to limited information on the host side, without direct access to the client side, still some assumptions remain to be made.
1.
INTRODUCTION
Observing user while visiting web sites is an area of great interest to anyone maintaining a web presence. The continuous optimization of a website to reflect the users needs is a key to success in the ever changing world of the interCopyright is held by the author/owner(s). ICWE’06, July 11-14, 2006, Palo Alto, California, USA. ACM 1-59593-352-2/06/0007.
net. The source of analyzable data ranges from usage data extracted from web logs or gathered by user tracking mechanisms to content or structural information from the website, allowing to analyze the user’s interaction in the semantical context of the website. The objective of a website owner is to optimally serve the user’s needs in order to attract more users, increasing visit frequency and achieve a higher user loyalty. To achieve this goal, the objectives and intentions of a user have to be analyzed. On transactional websites a user reveals his intentions by purchasing a product or service. By his willingness to pay for the website’s offer, he provides a direct feedback and makes the utility created by the website measureable. Websites not offering products or services for sale do not have this possibility to get hold of the users intentions directly. Owners of information driven websites have to further investigate the user’s intentions and objectives before measuring the success of their website. We propose means to allow the success oriented analysis of information driven website to improve design and content. In [11, 10] we have presented two approaches to this problem regarding information driven websites. In [11] we have introduced a check of consistency between the objectives of a website and the users’ perception of this site. Since this work analyzed a website as a whole, we extended it to allow clickwise analysis. By describing the user behavior within the website’s context by a formal model of the website in [1, 10] we created a scoring model to measure the success of a website and identify problematic webpages, that do not contribute to the website’s goals from the eyes of the website owner. The user intention has only be approximated by the behavior of the user and the semantic of the website. With this work we investigate how to approach the user intention directly from two directions. Since informatino about the user’s goals are limited on the server side, we first gathered a user feedback directly by conducting a traditional user study. In this paper we propose a second possibility to reveal the intentions of the users by analysis of search strings used with search engines directing users to our website. Search engines like Google or Yahoo provide the search string within the referrer information. Users from those search engines carry their search string within the referrer of the first click which is captured by our tracking system. Searchstrings have been used for searchengine optimization, their utility for
website engineering is hardly described.
1.1 Related Work In [2] Broder provides an overview about web search. He categorizes the goals of an user either as navigational, informational or transactional. The objective of an user with a navigational query, is to visit a particular web page he has in mind, whereas users with an informational query are interested to learn more about a certain topic independent of a particular web site. Since our websites do not offer any transactions, transactional search engine queries after Broder in [2] are not within the focus of this work. Neither are navigational queries, since all users analyzable to us have already reached our site. Therefore we consider informational queries as solely relevant for our search query analysis. Lee et al. focus their analysis of user goals in [6] on search engine optimization. They assume a user’s goal of a query to be inherently subjective. We concur with this assumption, whereas we analyze users that already have chosen a link and reached our analyzed websites. Thus the user’s focus is more or less narrowed to the area of interest of our websites. In contrast to Lee et al. we analyze search queries not from the point of view of the search engine, but from the position of the website author to which a search engine has linked. Teevan et al. [12] perform a study about web search personalization. They postulate, that it is unrealistic to assume that people specify their intent precisely when searching the web. Like Teevan, we analyze user interaction with the website to create an implicit user feedback. Since we do not optimize and personalize search engine usage, we deem it useful to assume a search string from a visitor to our website to be a strong indicator of his initial intentions. Spiliopoulou and Pohle described in [8] an approach to evaluate the success of non-sale-centric websites. They analyze the website’s goals by distinguishing betweens target pages and action pages. According to this concept, a user accessing an action page is stil pursuing the sites goal while an access to a target page shows that the user has achieved the sites goal. Similar to Spiliopoulou we assign web pages to categories of navigational and content pages. This work proposes new means to overcome the problem of the strict two category view on a website by integrating searchstrings into the usage analysis allowing a generic website assessment. This model can be used to detect flaws in the website structure where users tend to fail to arrive on content. This can be used to improve site design as described by Drott in [3]. It also reflects the user’s perception of the semantic structure of the website, which in turn can be used to reach a measure of web personalization as described by Eirinaki [4]. Thus the semantic structure can adapt to the expectations of the user.
1.2 Contribution Our overall objective is the optimization of our analyzed websites by gaining insight into the hidden user intentions through analysis of the users’ interaction with the semantics, content and structure of a website. In recent work [11, 1] we have introduced an indicator to measure information driven website performance. The Guidance Performance Indicator is derived from a host centric view as detailed information of user characteristics are not available on the server side. With the analysis of search strings we now have an unique window onto the user’s in-
tentions when entering a website. In this work we formulate and study means to incorporate the user’s intentions into our proposed formal website model in [1] and show, how this additional source of information can be applied to optimize a website. The so far unconsidered user intentions will increase the scope of the GPI to include user’s as well as the host’s view. Instead of approximating an implicit user feedback by analysis of his behavior, search queries within referrer information provide user intentions directly. In contrast to user studies, this additional source of information is free and can be used in large scale operations. Beside the integration into a scoring model for static optimization of a website, we propose means to use search string queries from referrer information in combination with user behavior as measure for search engine quality regarding intra-website search engine results. By this, we examine whether a search engine has guided the user to the optimal webpage within our website. Thus we discover potential for website personalization and search result optimization on the side of the search engine.
Overview In section 2 we describe the GPI measure and the formal model used to analyze websites. We evaluate the GPI results with a user study inquiring the success of a website from the user’s point of view. By highlighting the discrepancy between the host’s and user’s view on the success of an information driven website, we will suggest a solution to incorporate the user into the scoring model used to ascertain website success. In section 3 we will expand the scoring model by search queries revealing user intention. In section 4 we evaluate the proposed expanded scoring model, and discuss the direct implications on website maintenance and resulting consequences for search engines as well as motivating website personalization on this basis.
2. MOTIVATION We propose a generic success measure for information driven web sites. The idea of the measure is based on the observation of user behavior in the context of web site semantics. In [11] we have proposed a user perceived topic identification. Since this approach mixes usage and content of a web site, we now rely on a purely content based topic identification in order provide a more generic evaluation metric. We use Probabilistic Latent Semantic Analysis (PLSA) as described by Hofmann in [5], applied by us in [9], in conjunction with a crawler, which extracts text from webpages and calculates text length, removes stop words and applies stemming.
2.1 Topic Identification by PLSA As we are aiming to estimate the user’s session focus, we consider the content of the pages he visited containing all information as basis for further analysis. The understanding of textual or other pieces of information depends on the understanding and perception of the recipient, thus leaving the topic identification up to every user, author or analyst. Semantic information from taxonomies, ontologies or Semantic Web data have not been available for this analysis.
2.1.1 Topic Identification Algorithm Since we need to assign topic vectors to words and documents, we now give an overview of the applied PLSA. We
create a matrix of web pages and their keywords. Afterwards we consider word classes as topics {zj }L j=1 , and model the likelihood of a document d (web pages) as follows, Y`X ´n p(w|z, β)p(z|θd ) d,w (1) p(wd ) = w
z
where β are parameters specifying the probability of words given topics, θd are document-specific parameters that indicate the topics mixture for document d, wd is the word list of this document and nd,w is the number of occurrences of word w in wd .
Duration - Efficiency Factor. Let τ be a transition within the web site. We have the duration dτ which we compare with the average duration of all users avg(dAll τ ) having visited the specific webpage. Comparing dτ with avg(dAll τ ) we consider long durations on content pages as desired user behavior, whereas navigational webpages should guide the user quickly to the desired content. Therefore we assign greater values to short stays on navigation and long stays on content pages (as detailed in table 2). short d. + −
Navigation Content
long d. − +
• p(w|z) indicates the probability of occurrence of word w given topic z. The algorithm groups semantically related words into topics and thus explores the semantic relations between words. Intuitively, if several words often occur together, these words are likely to be associated with one topic.
Guidance Performance Indicator. We determine the ef-
• p(z|d) indicates the probability of topic z for document d.
fectiveness of a session σ with σ = (τ1 , τ2 , . . . τm ) by combining the transition type χτ , transition weight µτ and transition efficiency φτ for each transition τ ∈ σ:
Table 2: Web Page Duration Rating
2.2 Guidance Performance Indicator GPI GP Iσ =
|σ| X
χτi ∗ µτi ∗ φτi
(3)
By analysis of page transitions, duration and topic mixture the metric assigns positive or negative values to items of a clickstream. In particular we observe users on their way through the web site and assign positive and negative scores to their actions. The value of the score depends on the transitions between page types and their contribution to the web site’s objectives. Further details are described in [10, 1].
This constitutes our host centric view of session performance on information driven websites. As can be clearly seen, the measure utilizes only information freely available on the host side. Thus the user’s intention is implicitly captured by his browsing behavior.
Transition Type. We characterize the transition between
2.3 A User’s Perspective
certain categories of pages by the utility the author to the transition with respect to guiding the user to website content. As can be seen in table 1 the success of a transition to a content page is seen as positive step reflecting the focus of a website in delivering information.
Sitemap
Search
Content
Sess. End
Home Sitemap Search Content
Home Start
Destination
− − − −
− − − −
− − − −
+ + + +
− − − 0
Table 1: Page Category Transition Rating
Transition Weight. Let τ be a transition between content pages. The effectiveness of a transition is captured by an assigned weight factor µτ . We determine the overall topic mixture of a user session and contrast every click to the general nature of a session. Greater discrepancies indicate the user veering from his goal, which is now accounted by generating a lower score. negligible topic shif t ⇒ 1 < µτ µτ = (2) signif icant topic shif t ⇒ 0 < µτ < 1
i=1,τi ∈σ
In order to explicitly capture user intention, we have performed a user study consisting of users trying to accomplish standardized tasks and providing feedback on their experience. Consequently, we can contrast observed user experiences against a background of given user intentions with the analytical results provided by the GPI. We have performed a user study that provides user intentions. Such detailed user feedback analysis is not possible to perform on a large scale when analyzing production websites for longer periods. With this study we analyze if users and website owners deem the same sessions as successful.
User Study We have asked users to perform a given task on our websites and inquired their expectations and perceptions, while we observed their website traversal with our tracking mechanism. This gave us 40 sessions with given user intention and quality feedback from the user and an analytical measure provided by the GPI. As proposed by Pather in [7] we compare the expectations of the user with his perception about his session to generate a qualitative feedback - e.g. we compare the expected number of clicks to the target page with the realized number of clicks. Other questions concern the expected and perceived quality in navigability and intuitiveness of the website. We generate an aggregate vote by weighting the answers.
Results. We compare the GPI score of each session with the corresponding inquired score. The Pearson correlation coefficient between both values over all sessions results in 0.12.
Upon closer inspection we recognize that the GPI scores depend very much on the number of clicks per session, such that long sessions are more likely to collect high scores. By calculating the average GPI per click, the correlation rises to 0.37. After taking a second inspection of the data we find, that there is a general high correlation between the GPI and the user’s perception where both indicate a successful conclusion to the session. It is only in a few cases where correlation falls to -1, as the GPI and the user disagree on the success of the session. Whereas the GPI indicates, the user has meandered through the website without adequate guidance, the user nevertheless feels, he has accomplished his goal. In the eyes of the website owner, they were led to webpages identified to often cause session termination, namely lots of search page accesses not guiding to the desired content. These patterns receive negative scores by the GPI as they resemble undesired user guidance with high risk of session termination. By disregarding these outliers in the data, we find an increased correlation of 0.77 between the users evaluation and the GPI. The variety of users and user types with different browsing behavior makes it unlikely to generate a high correlation between the users judgment and the web owners represented by the GPI. Since the construction of the GPI purposely excluded the user intention, as it is not freely available on the host side, we propose to expand the GPI. By incorporating the user intention, we will be able to increase the accuracy of the GPI, as to when a session is to be considered successful with repsect to the host as well as the user. However, large scale observation and collection of user feedback is impossible to perform under operating conditions. Therefore we propose to gauge user intent by integrating another source of information provided on an automatic basis, namely the search strings carried within referrer information found in sessions, where users are directed to our website from an internet search engine.
3.
BUILDING A BETTER GPI
The analysis of search queries has been in the focus of recent work, such as [6] on the identification of user goals to improve search engines. These articles investigate, how far it can be assumed, that the search query carries the users goals to improve search engine results. In our work, the user has chosen a specific search result guiding him to our website. His active decision to follow a search engine result indicates his interest into our website. Consequently, we assume that the search strings carry the intentions of the users entering our website via search engines. In our previous work [1] we have introduced a formal model describing the usage of an information driven, non-salecentric website. This model of a meta environment, shown in Fig. 3.1 in an enhanced version, describes the relations between usage, content and structure within a web site focusing on evaluation. The web site author on the right side and the user on the left side of the model face each other. Only indirectly we can get hold of this by their artefacts, analyzing the web site and its usage. The packages SiteStructure and SessionStructure build the syntactic layer of the model. Assumptions and interpretation of the syntactic layer is positioned above in the semantic layer. Above all we positioned the evaluation
layer. We integrate SearchStrings in the SessionStructure as directly observable fact on the user side and connect it to UserIntention.
3.1 Data Preparation Not all user sessions are offspring of search engine inquiries, limiting the applicability to user sessions to those referred by search engines. The search strings we analyze have to be made fitting for our purposes by excluding navigational queries, as described in section 1.1. The relevance of referrer search strings depends on their significance to single web pages within the website and not to the whole website. In the later case, the user already achieved his intention in reaching our website. Thus we exclude navigational search strings dealing with words appearing on nearly all web pages such as company names. We focus on informational queries of users looking for certain information on our website.
3.2 Content Relevance of Search Strings The goals and intentions of a user on a website can either by inquired directly from the user or as we propose by search string analysis. The gained insight into the intention of a user enables us to view the users traversal through the website with regards whether the found information met his intentions or not. In order to do so, we analyze the content of the visited webpages using two approaches based on keywords and topics respectively.
Keyword based Search Query Analysis In the keyword based approach we analyze extracted textual information of each webpage and compare it with the words in the search queries. Depending on the number of occurrences of a search string sq out of a query q ∈ Q of a session σ in the bag of words sφ on webpage φ we calculate rank ρ for all webpages φ ∈ Φ with a function ρ : Q × Φ → N+
(4)
For each query q we now have a ranking of all webpages φ ∈ Φ with 1 ≤ ρ(q, φ) ≤ | Φ |. The ranking of the entry page φentryσ of session σ, when compared to the other pages φotherσ of the same session, allows the judgment whether a webpages exists, being a better match with the analyzed search query q, if the rank increases and ρ(q, φotherσ ) < ρ(q, φentryσ ).
Topic based Search Query Analysis Since keyword co-occurrence is only a basic technique for comparison of textual information, we use a more common approach to text analysis, Probabilistic Latent Semantic Analysis PLSA as presented in section 2.1. By applying PLSA we have calculated a matrix θ of topic vectors for each webpage and a matrix β with topic vectors for each word. Each topic vector represents the calculated probability for a given word of belonging to each of the topics (likewise for documents). In addition to the keyword based approach we insert an intermediate processing step. Regarding one session, we take the topic vector from θ describing the share a word has of each topic. By cumulating over all words within a search string of a user session, we find a topic mixture vector for each session. Then we calculate the Euclidean distance between this vector and the topic vectors for all webpages described by matrix θ. We perform a ranking, assigning the
MeassureDomain GPI
IntentionBased-GPI
SessionSucessIndicator
Measure
SiteSemantics
SessionSemantics
SiteCharacterisation
SessionCharcteristation 1..* Term UserIntention
PLSA Integration
...
Topic 1..* PageChar
PLSA
Description SingleInterest
Stem 1..* Keyword
SessionStructure 1..* SearchString
User 0..1 PageView
Duration
Session SessionSet
TopicMixture
SiteStructure TextString
SearchQuery Content
Author
...
Pic 1..*
Page 1..* 1
ReferrerInfo
SessionItem -prev 0..* -next PageChange
1..*
InLinkSet
OutLinkSet
Link
Site
Figure 1: Metamodel of a Website highest rank to the webpage with the shortest distance from our searchstring vector. We do this likewise for each session with search strings. The rank of the entry page of the session and the resulting improvement potential is calculated as described above in the keyword based analysis approach.
3.3 Enhanced Intention based GPI Building upon the above described topic based ranking of webpages, we evaluate each click of a session with respect to the search query. While a user can have multiple or changing interests and intentions, we concentrate on his initial intention as revealed by the search query. The fulfillment of this initial intention and the respective session evaluation are not bothered by the completion of parallel tasks of interest. With this assumption we tolerate the following errors: • a user has not completed his initial task, but other tasks → the whole session is evaluated as negative • a user has completed his initial task, but not another task → the whole session is evaluated as positive Leaving the duration factor untouched, we enhance the GPI calculation, described in sec. 2.2, by calculating the transition type and weight with respect to the topic mixture of the corresponding search string. We consider each transition τ ∈ T 1 with respect to the corresponding search query q ∈ Q. Let there be for each 1 Note a transition is an element of a session τ ∈ σ, while σ⊆T
transition an initial query of the session, it is located in. Let this query be accessible by a function: query : T → Q We use the search strings, derived from the query, to calcula− − te a query related topic vector → z ∈ Z. Here → z corresponds L − to the definition of {zj }j=1 in Section 2.1 and we claim → z to be calculated on base of the same classes, which is used to calculate the site’s topic vectors respectively. The derivation of a topic vector we denote with a function: ζ : Q→Z Now we consider each transition τ to be successful if the rank ρ, introduced in Section 3.2, of the target page φtarget is higher than the source page φsource . We calculate this by the difference (ρφsource − ρφtarget ). If a transition leads to a page with a lower rank, we will retrieve a negative value. This results in an implicit consideration of the transition type. Thus we no longer need to consider different page categories and their changes. We aim to replace the µτ of the classic GPI by a function δ : T → R, whereas µτ of the classic GPI was only dependent on site metrics. Now we receive different functions δ, each depending on topic vector → − z corresponding to the initial query, which belongs to the session the transition is located in. We introduce a higher order function γ to generate these functions δ: ` ´ γ : Z→ T →R Due to lack of space we abstract from technical details, like
e.g. scaling, and can define the intention based GPI like ibGP Iσ =
|σ| X
i=1,τi ∈σ
` ´` ´ γ ζ(query(τi )) τi ∗ φτi
(5)
Definition 5 shows a function δ, that retrieves the weight of a transition τ . The function itself is derived in dependency to τ via the query of the session in which τ is located and − the related topic vector → z which could be assigned to the set of search strings of the query. A set of differing user interests is regarded correctly, if corresponding search strings can be found in the initial que− ry. Thus they influence the calculation of → z . But we have to assume the user does not change his intention during the session as we are not able to identify these changes of interest. Also we should of course remark, that strings, when used in both domains, are assumed to have the same basic semantics.
3.4 Session Success Indicator The additional source of information from search strings allows us to judge a session to be successful with respect to a user’s intention. Since the clickbased score of the GPI can hide the fact, that a user reached his goal by aggregation of the clickwise scores. Therefore, we introduce as an additional measure a Session Success Indicator. The SSI uses the ranking function based on the relevance of webpages in comparison to search string topics as introduced in Sec. 3.2. The highest rank within a session indicates the proximity to his area of interest the user reached during his traversal of the website. If we regard two pages φsource , φtarget ∈ Φ, a transition τ = (φsource , φtarget ), we denote the source page of a transition with τ − = φsource and the rank of a page according to the function ρ, defined in section 3.2, we can define the SSI as follows: ̺SSI = 1 −
min{ρ(query(τ ), τ − ) . τ ∈ σ} |Φ|
Due to value of 1 as the best rank, we search for the minimum of the set. According to this definition, the range of ̺SSI may reach from 0 ≤ ̺SSI < 1. The distribution of sessions according to this indicator allows for further interesting insights. Further enhancements of the ibGP I may result from weighed interpretation regarding their overall success. This is part of our current work and will be presented in the near future.
4.
CASE STUDY
We will now demonstrate the potential of search strings carried within referrer information by analyzing two large websites. On both websites we observed users over a period of three months. In the same time period we performed our user study of Sec. 2.3. After a short overview concerning the underlying data, we take a look at the GPI and show the increase in accuracy achieved by the ibGPI. The application possibilities of search strings are not limited to our metric. Using the same techniques and data, we can not only evaluate our website, but combine search strings and user behavior providing a new approach for website personalization and search engine evaluation.
Seq.No. Webpage 1 Home 2 Search 3 Search 4 Search 5 1171 Total Seq.No. Webpage 1 2578 2 Home 3 2790 4 1067 5 1302 6 1166 7 1042 8 2579 Total
GPI -2.25 -2.25 -2.25 -2.25 0.4 -11 GPI 0.75 0.91 0.53 0.82 0.49 0.90 0.43 0.06 4.89
User Study good User Study not amused
ibGPI
SSI
0.94 -0.96 0 0.18 0.81 0.99 ibGPI
0.63 0.63 0.63 0.63 0.93 0.93 SSI
0.49 -0.13 0.008 0.12 -0.11 0.03 -0.02 -0.67 -0.282
0.96 0.96 0.96 0.99 0.99 0.99 0.99 0.99 0.99
Table 3: GPI and Intention based GPI
Data Preparation. We have performed our analysis on data from two large websites covering two separate time frames. The basic facts are depicted in table 4. The number of search strings parsed from gathered data categorized by referring entity yields that almost all referrals stem from Google, about 90 % of all strings, whereas 10 % originate from search engines like Yahoo or alltheweb.com. Website internal search engines were not analyzed in this work, since their search strings have not yet been recorded yet. From 10997 sessions in total we found 15 % with search strings. The usable ones are reduced to 11.4 % of all sessions. We exclude search strings dealing with the words appearing on nearly all web pages such as company names and other terms too general to be of use. The relevance of search queries depends on their significance to specific topics within our website, and not to the whole website.
4.1 Intention based GPI In Sec. 2.3 we have mentioned, that several sessions of the user study have opposite user and GPI evaluations, such as those two described in Tab. 3. In order to make the sessions of the user study compatible to the search-query-based new ibGPI, we convert the given tasks into search strings identifying the target page. Similarly as with search queries we have now sessions with known user intentions. The first example session is evaluated with a very negative score by the old GPI, whereas the user deems his session to be successful since he reached his target page. The ibGPI on the other hand indicates the user successfully headings towards his target page 1171. By reaching this page his session receives a high score from the ibGPI as well as by the SSI. For this type of session, the ibGPI performed much better in modeling the user session in the same way the user perceived his website visit. The second clickstream in table 3 does not diverge as much as the first session, but the GPI indicates a successfully session in contrast to the users judgment, since he was not sure to have reached all valuable webpages. He reached the relevant webpages, which is correctly reflected by a very high
B B
Histogram of improvement potential based on keywords
1000 800
topic
Nov.05- 410 Dec.05 Jan.06 390
750
564
72 %
549
319
Nov.05- 63 Dec.05 Jan.06 79
195
186
20.6% 55.4 % 14 % 19 %
191
180
68%
600
keyword 18 %
1200
optimization potential based on
Frequency
cleaned Search String Sessions
400
A
Sessions with Search Strings
200
A
No. of Webpages
94 %
0
Web- Time site Period
0
20
40
Table 4: Session Evaluation of Search Strings
4.2 Applications for Search Query Analysis Apart from the GPI we propose further possibilities for search query analysis. In sec. 3.2 to 3.2 we suggested two approaches to generate ranks in either a keyword-based or topic-based fashion. Now we show the differences of these two approaches.
Keyword based Search Query Analysis We deem a target webpage within the top 10% of the relevant web pages as optimal. In Tab. 4 we can see the results of both approaches. Whereas the keyword based approach distinguished with hard criteria between match and no-match, the topic-based-approach uses the latent semantic revealed by the PLSA to identify areas of interest to the respective search query. The potential of optimization varies depending on each website. For instance, the relatively high value of 68 % improvement potential from all analyzed search strings sessions in January 06 on website B results from the fact, that the pages found by the search engine are slightly lower ranked on level 11 of 99 pages than our top 10 pages.
Topic based Search Query Analysis We assume an entry page among the top 10 % of all webpages as good guidance performance of a search engine. Ho-
100
120
Figure 2: Keyword based Results for Webpage A Nov.-Dec.05
80
100
Histogram of improvement potential topic based
0
20
40
Frequency
Applications for ibGPI. Beside the evaluation of user sessions by the GPI or ibGPI, the revealed information can be used to evaluate each webpage or the website in total by aggregation of all clicks on a specific webpage or for the whole site. We also have analyzed which entry pages have resulted in a negative session score. This indicates that the navigation and website structure of this webpage could be improved, enabling the user to reach his desired content in a less circuitous manner. Similarly, the search strings can be evaluated by the corresponding session score. A search query resulting in a negative session score can help to identify content, that is difficult to find and is seldom being reached.
80
60
SSI. But he needed several additional clicks, which lead him around the website. This can be seen from the slightly negative ibGPI. The combination of both resembles the user’s feeling: having found all information, but under bad usability and navigational conditions.
60
improvement_potential
0
50
100
150
200
250
improvement_potential
Figure 3: Topic based Results for Webpage A Nov.Dec. 05
wever we receive a higher potential for improvement by the topic based search query analysis than from the keyword based approach in Fig. 2 even with a higher tolerance margin of 15 %. The whole improvement potential for website A can be seen in detail in Fig. 4.2. A summary of the other websites is depicted in Tab. 4. Fig. 4.2 shows clearly that the topic based approach deems more entry pages as suboptimal as the keyword based approach does in Fig. 2. One possible reason is that the latent semantic of a website created by the PLSI can identify semantically related webpages even if the keyword cooccurance is low.
Application: Search Engine Analysis. Beside the integration into a scoring model for static optimization of a website, we propose means to use search string queries from referrer information in combination with user behavior. The above described analysis and results can be applied to improve search results of search engines. This approach is particularly valuable for an assessment of the website’s internal search engine since one can influ-
ence its algorithms and results directly. Regarding external search engines and their potential for improvement, the ranking algorithms of search engines have to be analyzed and understood. Additionally, the recency of the last visit of a search engine’s indexation bot has to be taken into consideration. With the help of the ibGPI results, the ranking function and the search strings we have identified web pages that match better with a specific search query. Those pages should meet the users expectations in a better way. Nevertheless, there are reasons to guide most users to the homepage, the more impatient a user is, the more likely he will terminate his session. Since the average session length on our websites is only 2.7 clicks per session, the risk of session termination is relevant. The discovered knowledge is not only applicable for search result optimization, but offers so far unused potential to personalize a website directly and instantly to the users interest.
Application: Personalization. This approach is not only applicable for search engine analysis and optimization, but also for personalizing a website to the goals of users coming from external or internal search engines. Following the concept of collaborative filtering, one can derive recommendations out of the comparison between search queries and browsing behavior - ¨...users that have searched for...also visited page...¨. This approach adds a new perspective to traditional recommendation engines. It is not limited to the derivation of user intentions implicitly based on their behavior, but consideres the users interests directly, since said interest is provided freely within the referrer information. Even if the search result of an external or internal search engine is not optimal, the entry webpage can be personalized to the search string and offer links to websites more suitable to the user intention.
5.
CONCLUSION
It remains challenging for non-sale centric and information driven websites to optimize structure and content in accordance with the user’s needs. Direct feedback such as purchases on transactional websites are not available. But search queries within referrer information offer a useful insight into the intentions of the user. With this work we have investigated this source of information and have evaluated its applicability. By contrasting the host centric view of the GPI scoring model with a user enquiry into their perception of a website, we have found to require more information on the user’s intention when greater accuracy in analyzing website perception is desired. The user study indicated that the GPI scores sometimes differed radically in the same session evaluation, when compared to user responses. Different user types, individual browsing behavior and differing website perceptions do not always fit with a usage evaluation from the website author’s side. Since the GPI was designed as a generic evaluation measure for a website, we have to extend it and consider user intentions for a more precise modeling of user sessions. In our formal website model we have shown how to integrate search queries via PLSA into the existing meta model. By ranking webpages separately for each user session and differing search query respectively, the intention-based GPI
is calculated individually for each user and his unique intentions. As a result the ibGPI is a more robust and accurate than the normal GPI and is not dependant on manual webpage categorization nor scoring weight definitions. Having observed users with assigned intentions under laboratory conditions in our user study, we have shown the enhanced abilities of the ibGPI to cope with more sophisticated user behavior than the normal GPI. Future Work will deal with the enhancement of our formal website model and the identification and integration of user types to achieve a differentiated session evaluation. Another aspect arises from new browsing patterns resulting from parallel browsing behavior, like tabbed browsing or multi window browsing. Intentions when pursued in parallel have to be taken into consideration. In order to increase the scope and applicability of the ibGPI, we will explore the potential of search string analysis for website internal search engines. This will help to adapt search engine results as well as personalize websites instantly to users.
6. REFERENCES [1] M. Barth, M. Skubacz, and C. Stolz. Web performance indicator by implicit user feedback application and formal approach. In LNCS: Intl. Conf. WISE 2005 New York. Springer, 2005. [2] A. Broder. A taxonomy of web search. SIGIR Forum, 36(2), 2002. [3] M. Drott. Using web server logs to improve site design. In SIGDOC, 1998. [4] M. Eirinaki and M. Vazirgiannis. Web mining for web personalization. ACM Transactions on Internet Technology, 3:1–27, 2003. [5] T. Hofmann. Probabilistic Latent Semantic Indexing. In Proc. 22nd SIGIR, pages 50–57, Berkeley, California, August 1999. [6] U. Lee, Z. Liu, and J. Cho. Automatic identification of user goals in web search. In WWW Conference. IW3C2, ACM, May 2005. [7] S. Pather, G. Erwin, and D. Remenyi. Measuring e-commerce effectiveness: a conceptual model. SAICSIT ’03, pages 143–152, 2003. [8] M. Spiliopoulou and C. Pohle. Data mining for measuring and improving the success of web sites. Data Min. Knowl. Discov., 5(1-2):85–114, 2001. [9] C. Stolz, V. Gedov, K. Yu, R. Neuneier, and M. Skubacz. Measuring semantic relations of web sites by clustering of local context. In LNCS: ICWE 2004, Munich, pages 182–186. Springer, 2004. [10] C. Stolz, M. Viermetz, M. Skubacz, and R. Neuneier. Guidance Performance Indicator - Web Metrics for Information Driven Web Sites. In IEEE Intl. Conf. Web Intelligence 2005, Proc., 2005. [11] C. Stolz, M. Viermetz, M. Skubacz, and R. Neuneier. Improving semantic consistency of web sites by quantifying user intent. LNCS: ICWE, Sydney, 2005. [12] J. Teevan, S. Dumais, and E. Horvitz. Personalizing search via automated analysis of interests and activities. In SIGIR, 2005.
