Similarity-Based Visualization for Image Browsing Revisited Klaus Schoeffmann Institute of Information Technology Alpen-Adria-Universit¨at Klagenfurt Klagenfurt, Austria Email:
[email protected]
Abstract—We investigate whether users’ visual search performance in a commonly used grid-like arrangement of images (i.e., a storyboard) can be improved by using a similaritybased sorting of images. We propose a simple but efficient algorithm for sorting image based on their color similarity. The algorithm generates an intuitive arrangement of images and allows for general application with several different layouts (e.g., storyboard, simple row/column, 3D globe/cylinder). In difference to previous work, which rarely present results from user studies, we perform a fair user study and compare an interface with color sorted images to an interface with images positioned in a random order. Both interfaces use exactly the same screen estate and interaction means. Results show that users are 20% faster with the sorted interface. Keywords-Visual search, image browsing, image sorting, visual similarity, user study
I. I NTRODUCTION A storyboard is a two-dimensional grid-like arrangement of images, typically with a scrolling or paging function, that allows a user to interactively inspect and browse a set of images. While already introduced almost two decades ago [1], storyboards are still intensively used in the field of content-based image retrieval (CBIR) [2] and contentbased video retrieval (CBVR) as they provide an easy way to browse through the results of a query. For example, most systems in the interactive Known Item Search (KIS) task in TRECVID 2010 [3] used scrolling- or paging-based 2D storyboards for interactive browsing of pre-processed results. These results are usually presented as a list of thumbnails of images (or key-frames of video shots). Storyboards are also used by image and video browsing applications [4] that allow users to browse through a collection of images/videos in order to (i) get a rough overview of the content of the collection or to (ii) interactively find desired content in situations where no query can be specified. While in CBIR/CBVR applications the storyboard usually presents a ranked list of results, in image and video browsing the storyboard is typically sorted according to metadata such as creation date or filename. However, when users want to find a particular image in that list, they rather search by visual attributes of the target image instead of relying on metadata. This is especially true in situations where metadata is incomplete or invalid. Therefore, it makes sense to arrange
David Ahlstr¨om Dept. of Informatics Systems Alpen-Adria-Universit¨at Klagenfurt Klagenfurt, Austria Email:
[email protected]
images sorted by visual similarity, as already proposed in previous work [5]–[8] (see also Section II). Although several similarity-based visualizations for images can be found in the literature (reviewed below), to best of our knowledge no one has investigated similaritybased visualization to be used for simple scrolling-based storyboards and no one has evaluated it in a user study. Therefore, in this work we propose a simple and fast algorithm for sorting large image sets according to color similarity. This algorithm has the following characteristics: (i) it is fast in terms of run-time, and thus can be used also with mobile devices [9] as well as with adaptive interfaces, (ii) it results in an intuitively sorted layout, such that even novice users can easily estimate locations of specific images in the layout, and (iii) it preserves visibility of all images and can be used for a storyboard and many other layouts. We further test the efficiency of the resulting color sorted layout of images with a user study on interactive search tasks where users have to find target images in a long list of images as fast as possible – a use case that is also subject of the interactive KIS task of TRECVID [3]. We perform a direct comparison between a random and a colorsorted arrangement of images in a storyboard and show that users are significantly faster with the proposed color-sorted storyboard and also find it more preferable. II. R ELATED W ORK Rodden et al. [5] were among first to investigate search interfaces with visually sorted images. They evaluated visual search time for target images in a multidimensional scaling (MDS) [10] layout of 80 images against search time in a random layout where all images were shown in one screen. MDS produces a similarity-based visualization where similar images are placed close together, typically resulting in an overlapping of images. This overlapping is obviously greater for larger image collections than for small ones. Hence, if an MDS-based similarity arrangement is used to display items from a large image set, either some additional rearrangement algorithm is needed or the interface must support a hierarchical refinement through which similarity-based ‘clusters’ can be inspected in more detail. However, such a hierarchical
browsing process is typically inconvenient for users as it completely destroys the context of the inspected cluster. As pointed out by Nguyen and Worring [7], a similaritybased visualization of images in general should not only preserve the structure according to similarities between images (structure preservation requirement) but should also give a faithful overview of the distribution of all images (overview requirement) as well as preserve the visibility of all images to an extent that allows the user to understand the image content (visibility requirement). As previously mentioned, Rodden et al. [5] used a set of 80 images in their study. However, 10% of the images suffered from overlapping such that less than 70% of the content was visible. Therefore, Rodden et al. decided to only use the remaining 90% of images as potential targets for search trials. They report that users were significantly faster with the MDS arrangement than with the random one. It should be noted that although a rearrangement algorithm can be used to displace images in an MDS layout such that overlapping is reduced, it is difficult to generate a scrollable list of images for a large image set that is consistently sorted according to visual similarity. Moreover, MDS is a computationally intensive, and hence slow process, as it relies on an expensive interactive convergence procedure, as also stated in [8]. From the results of Rodden et al.‘s study we cannot directly derive that users would also be faster in visual image search with visually sorted grid-based and scrollable interfaces. Scrollable grid-like arrangements, i.e., storyboards, are however the most commonly used way to present images (or previews for videos), both on desktop computers and on portable devices. Therefore, we are interested in knowing how a similarity-based visualization for storyboards can be generated and if users are faster with this storyboard in interactive search tasks in comparison to a (visually) random arrangement (as used in most commercial interfaces as well). In a follow-up work, Rodden et al. [11] performed an evaluation of similarity- based visualization where an MDS arrangement of 100 images at one screen was evaluated against a random arrangement for the task of category-based image selection. Each user was asked to select all matching images for a given category (in the whole study 20 categories were used with 3-7 images per category). Interestingly, users were slightly faster with the random arrangement than with the MDS arrangement. Rodden et al. continued their evaluation in a later work [6], where a modified MDS algorithm was used to remove overlapping from the similarity-based visualization. In that work the authors evaluated whether a similarity-based visualization is preferred over a caption-based visualization for image selection tasks, where users have to select photographs for given locations. In every task the participant was requested to select three appropriate photos from a set of 100 entries (shown at one screen) for a given location
to be used in a “destination guide”. The study revealed that 38.8% of users found the caption-based arrangement more useful, 16.6% of users preferred the similarity-based arrangement, and the remaining 44.4% of users stated no obvious preference although most of them found it useful to have both arrangements available. Schaefer [8] proposes a 3D globe layout for similaritybased browsing of images. This layout arranges images according to color and brightness. Schaefer argues that arranging images according to Hue and Value attributes of the HSV color space is very intuitive for humans. We share this opinion and we also use the HSV color space as a basis for the color-based similarity sorting proposed in this paper (see Section III). Schaefer argues that when using Hue and Value to sort images it almost naturally leads to a 3D globe visualization, which furthermore would be immediately and easily understandable by average users as they are already familiar with the concept of a globe. Schaefer arranges images on the suggested globe according to their median HSV value, where the Hue attribute is mapped to the longitude and the Value attribute is mapped to the latitude of the globe. Similar to the MDS layout this arrangement also leads to overlapping of images in situations where several images are mapped to similar latitude/longitude coordinates. As a solution, the author proposes using a regular grid structure with the globe, i.e., using a grid that consists of predefined cells on fixed locations, in combination with hierarchical browsing. Of course, this regular grid structure limits the number of images that can be displayed in one view of the globe. In other words, for large image collections it is likely that more than one image is mapped to one cell in the grid. Consequently, Schaefer proposes a hierarchical browsing approach with the globe visualization: as the grid structure has a limited number of cells in one view, similar images are clustered and a representative image is shown for the cluster on the top-level view. A user may step deeper into the hierarchy by selecting this representative image, which causes the grid structure to be reloaded for the next view with images from the cluster only. To avoid the problem of mapping the similar images to the same cell again at the next level of the hierarchy, a different arrangement algorithm is used, which is based on distances between images. Hence, this approach of globe-based browsing conflicts not only with the requirement of visibility but also with structure preservation, as defined by Nguyen and Worring [7]. Moreover, no evaluation of the globe-based browsing approach has been performed (e.g., through a user study). Thus, it is questionable whether and how this interface could improve interactive browsing. Quadrianto et al. [12] propose kernelized sorting of images according to visual similarity to be used for several applications. Quadrianto and colleagues [13] have shown that kernelized sorting can also be applied to a 2D gridstructure as well as a 3D sphere layout to be used for
image browsing. However, in their work no evaluation with interactive image search has been performed that would assess the effectiveness of their sorting algorithm. Similar to the similarity-based 3D globe arrangement of images, Plant and Schaefer [14] proposed a Honeycomb visualization to be used for image browsing. Their Honeycomb Image Browser uses almost the same strategy for arranging images in “honeycomb cells”, as used for the globe arrangement in [8]. However, instead of mapping to longitude and latitude, Hue and Value are mapped to the x-axis and y-axis, respectively. Similarly, a visual clustering and hierarchical refinement approach is proposed to avoid overlapping of images. It should be noted that the structure of the visualization used in the Honeycomb browser and the globe visualization is highly dependent on the diversity of the underlying image collection and may produce an irregular structure with gaps (i.e., empty cells), as shown in the screenshots of [8], [14]. Moreover, as already mentioned above, a hierarchical browsing approach destroys the context of the current view at refinement steps, which makes it difficult for users to preserve an overview of the image collection. Our similarity visualization approach, and the underlying sorting algorithm, described in the following section is a general approach that can be used for any visualization layout as it simply sorts a list of images according to colorbased features. The result of this sorting is an intuitive arrangement of images that can also be applied to a commonly used storyboard layout. Furthermore, the user study presented in this paper performs a fair comparison with an unsorted storyboard: both tested interfaces use exactly the same screen and the same interaction means. III. S ORTING I MAGES BY C OLOR We propose to sort images by color according to an efficient and intuitive sorting algorithm, which is fast in terms of run-time and allows for general application with different layouts. This sorting algorithm consists of two steps. The basic idea is to sort images based on their dominant Hue attribute in the HSV color space. Therefore, we classify pixels of images into a 16-bin Hue histogram (each bin represents pixels belonging to a Hue range of 22.5 degrees) and use the index of the dominant bin as a basic sorting criteria in the first step. To give a more consistent view, images belonging to the same dominant bin are sorted again in the second step such that the Euclidian distance of a 24-bin HSV histogram (16 bins for Hue, 4 bins for Value and Saturation, respectively) between adjacent images is minimal. Moreover, we perform a special treatment for bright and dark images, such that these images are arranged at the beginning and the end of the list, respectively. To filter for bright and dark images we use two simple rules: (1) images with more than 50% of pixels having a Value attribute >= .75 and a Saturation attribute of 0 are detected
Figure 1. WANG 1000 dataset sorted with the described algorithm and aligned on a regular 25x40 grid with row-major order (top), and aligned on a 3D cylinder [9] with column-major order (bottom).
as bright images, (2) images with more than 50% of pixels having a Value attribute < .25 are detected as dark images. Our sorting algorithm produces an intuitive sequence of images that starts with bright images, followed by images according to the Hue progress in the HSV color space, finally followed by dark images. This list of images can be used for a linear layout algorithm with flexible arrangements (i.e., on a grid with row-major order, in a simple row/column, in more complex layouts such as 3D cylinders [9], [15], etc.). The top of Figure 1 shows how the WANG database, which is a well-known manual-selected subset of 1,000 images of the Corel stock photo database, is sorted and used for a grid-layout according to the described algorithm. The bottom of Figure 1 shows how the same sorted list can be arranged on a 3D cylindrical layout. On a common desktop computer – executed single-threaded on an Intel E5530 CPU with 2.4 GHz – it takes only a few seconds to perform such a sorting of 1,000 images. IV. U SER S TUDY To evaluate the effect of color sorting in a 2D scrolling storyboard, according to our algorithm, we conducted a
user experiment. Twelve volunteers (six female) aged 27 to 38 years (mean 30.5 years, SD 3.8) participated. All were right-handed and their self-estimated computer usage per week ranged from 10 to 60 hours (mean 42.8, SD 16.1). The experiment was conducted on a Dell Precision M4400 Laptop (running Windows 7) with its 15.4-inch display set at a resolution of 1440×900 pixels. A laser mouse (Dell LaserStream) was used as input device. The experiment software was coded in C# .NET 4.0 and used the Microsoft XNA Framework 4.0. We used twelve predefined storyboards for the experiment. These storyboards were constructed as follows. First we created six unique sets of 150 images by randomly drawing from a pool of 1,100 images taken from the IACC.1 TRECVID 2010 repository [3]. All images in a set were unique. After that we sorted the images in each of the six sets using our sorting algorithm. As a result, each image was assigned a number from 1 to 150 that indicated its position within a color-sorted storyboard containing 150 images arranged in 50 rows and 3 columns. We used a rowmajor order for position numbers and divided the storyboard in ten equally sized logical target groups (images at position 1 to 15 belonged to the first group, images at position 16 to 30 belonged to the second group, and so forth). Next, for each of the six sorted storyboards we randomly chose one position from each of the ten groups to serve as target positions for our experimental visual search trials. In this way we ended up with target positions for ten trials in each of the six storyboards that were roughly evenly distributed across all 150 positions. Finally, we constructed six unsorted storyboards using the same six image sets as exactly the same targets (both same image and same position) but randomized positions of non-target images. To summarize: six pairs of storyboards were created. Each pair consisted of one sorted and one unsorted version for a unique set of images. All storyboards contained 150 images arranged in 50 rows and 3 columns. The images were 160×100 pixels large and seven rows of images had place on the screen. The reason for using the same targets in both types of storyboards is that we wanted to perform a fair comparison, with exactly the same task difficulty. As already observed by Rodden et al. [5], image distinctiveness plays an important role in user studies with visual search tasks. There are always some images in a data set that are very easy to find in a list (almost regardless of the position) and some images which are very hard to find. We were not sure about the distribution of image distinctiveness in our data set and the number of required participants to balance the problems of image distinctiveness for completely randomized trials (in a previous study with 28 participants and randomized trials for three interfaces, post-hoc examination showed that image distinctiveness was very unbalanced among the three interfaces; in other words due to the random selection of images, coincidentally one interface was privileged and
another one was penalized, although 28 participants were used). Moreover, using the same target images also allows for comparison on an image basis, i.e., how many of the target images can be found faster with the unsorted/sorted version of the storyboard. We used a within-subject design for the experiment where all participants performed ten visual search tasks with each of the twelve storyboards. Six participants started with the sorted versions, the other started with the unsorted versions. We are aware of the fact that by using the same 60 target images in both storyboard versions, we introduce a learning effect. However, due to the balanced design of the experiment this learning effect should be the same for both storyboard types and therefore insignificant for the direct comparison. A trial began by displaying a target cue in the upper left part of an otherwise black screen, as shown in Figure 2a. Participants were instructed to carefully inspect the cue as long as necessary and that after the inspection they were required to search for the cued image within the storyboard without recourse to the cue. When the cue was clicked, it disappeared and the storyboard and a red scrollbar were displayed, as shown in Figure 2b. Participants were instructed to find and click on the previously cued image inside the storyboard as fast as possible.
a
b
Figure 2. Experimental task: the first step (a) showed the target cue, the second step (b) displayed the scrollable storyboard (here an unsorted one).
Participants were also told that they could use the scrollbar and the mouse wheel for navigation, the way they preferred. When an image in the storyboard was clicked, all other images were removed and the selected image was either colored green or red to indicate that a correct or a wrong image had been selected. After that, the next trial was loaded with a new target cue in the upper left corner of the screen. Trial time was measured from a click on the cue to the selection of an image inside the storyboard. If a wrong image was selected, an error was recorded and the trial was repositioned at the end of the queue of unfinished trials. A test session lasted on average 40 minutes, with one participant per session. First, demographic data was collected. Then the participant was asked to carefully study the written instructions. These included information about the experimental task and instructions on how to use the storyboard interface to be tested next. The written instructions were issued immediately before testing each interface
V. R ESULTS AND D ISCUSSION Error rate: With a total of 62 trials, where participants selected the wrong image, the overall error rate was 6.69%. The errors were evenly distributed between the two storyboard types, 29 (6.29%) with the sorted and 32 (7.09%) with unsorted ones. Errors were also evenly distributed between participants and between target groups for each storyboard type. We can conclude that the sorting of images did not influence the error rate. Trial time: Our time analyses included only successful trials from which 32 trials (3.7% of the total 864 successful trials) were removed as outliers (having a trial time more than 3 SD from the mean for the corresponding combination of storyboard type and target group). These outliers were roughly equally distributed between storyboard types and participants. With outliers removed, the overall trial time was 6186ms for the sorted storyboard type and 7807ms for the unsorted. A repeated measures ANOVA with factors storyboard type (sorted, unsorted) and target group (1 to 10) showed significant main effects for both factors (storyboard type: F1,11 = 14.53, p < .01, target group: F3.08,27.76 = 29.99, p < .0001, Greenhouse-Geisser adjusted). The interaction was not significant. Thus, we can conclude that the sorted storyboard type was faster to search than the unsorted one. Figure 3a visualizes the overall time difference of 20.76% and the significant effect for target group is plotted in Figure 3b. The effect for target group
15000 8000 12000
7000 6000
Trial time (ms)
5000 4000 3000 2000
9000 6000 3000
1000
or te d
So
U ns
rt ed
0
a
Unsorted y = 1431x + 1191 Sorted y = 1241x + 976
b
0
TG 1 TG 2 TG 3 TG 4 TG 5 TG 6 TG 7 TG 8 TG 9 TG 10
Trial time (ms)
and only the instructions for the sorted storyboard contained information about the color sorting algorithm (including an example screenshot). This means that the participants who started with the unsorted storyboard had no knowledge about the sorting algorithm, thus could not use this information to find target images in the unsorted list faster. Contrarily, participants starting with the sorted storyboard could possibly benefit from previously acquired image position knowledge as the same target images and positions were used. In order to reduce the benefits for the unsorted storyboard for this group of participants, we randomly pre-selected 60% of target images (six from each of the six image sets) and used the same target position only for this set of 36 targets. The positions of the remaining 24 targets were randomized. The dependent factors in our experiment were error rate and trial time, independent factors were storyboard type (sorted, unsorted) and target group (one through ten). The number of collected error free trials can be computed as follows: 12 participants × 2 storyboard types (sorted, unsorted) × 10 target groups × 6 trials (one from each of the 6 image sets) = 1440 trials. However, the following analyses only consider the 36 targets which had the same target position for both participant groups, i.e., 12 participants × 2 storyboard types × 36 images = 864 trials were used for the analyses. After the experiment participants were asked to state which storyboard they preferred.
Target Group
Figure 3. (a) Mean trial time across target groups for sorted and unsorted storyboards (Error bars represent ± 2 standard error). (b) Trial time vs. target group (note that target group 10 was excluded from the regressions).
is intuitive and agrees with both scrolling [16] and visual search [17] literature: scrolling time increases with distance (i.e., in our study scrolling to more remote target groups) and visual search time increases with the number of distractors (i.e., in our study scanning through an additional 15 images for each consecutive target group). However, as visible in Figure 3b there is a remarkable drop for the last target group. We suspect that the extraordinary distinctiveness of a few images in group 10 contributed to this drop. One such image is shown in Figure 4a. We suspect that in the sorted storyboards it was very easy for our participants to figure out that these distinctive images was positioned at the end of the storyboard, in the dark section. Thus, with high confidence participants could instantly move down to the end (either by rapid scrolling or a fast drag of the scrollbar) in order to click on the target image. Likewise, in the unsorted storyboards, participants could confidently scroll through the images at high speed being sure not to oversee such easily recognizable images. Anyway, if the last target group is excluded, regressing trial time against target group yields strong linear fits (as expected) with R2 values of .950 and .911 for sorted and unsorted respectively. The resulting equations are shown in Figure 3b. As expressed by the flatter slope for the sorted condition, with sorted images participants were able to scan through the storyboard more rapidly by first searching for the appropriate “colorsection” of the storyboard, being able to quickly scroll past uninteresting sections – sections containing images with colors dissimilar from that of the target image – until arriving at the suspected section and then perform a detailed scan.
a
b
Figure 4. (a): a highly distinctive image from the last target group, (b): the image at target position 59.
Trial time difference (%)
80 70 60 50 40 30 20 10 0 -10 -20 -30
Figure 5.
Unsorted Sorted
30
60
90
120
150
Target Position
Per centum difference for each image against target position.
However, it should be noted that the positive effect of the sorting is roughly uniformly distributed across the length of the storyboard and that the grand mean advantage (20.76%) is not caused by only a few images. Instead, as visible in Figure 5 which plots the per centum difference for each one of the 36 images against target position (1 through 150, with 15 positions in each target group), participants could capitalize on the sorting feature with almost all images. With 30 of the 36 images (83.3%) the sorting was effective. Only in a few cases the sorting did not yield any positive effect, most notably with the image at position 59. The majority of successful cases confirms the correctness of our sorting algorithm. The image causing the irregularity at position 59 is shown in Figure 4b. Whereas, our sorting algorithm placed this image among mainly brown images, the majority of our participants probably suspected it in the blue section of the storyboard. This mismatch presumably caused the considerable additional search time visible in Figure 5 at target position 59. Finally, when asked after the experiment, ten from twelve participants stated that they preferred the sorted storyboard over the unsorted. One participant was undecided and one participant stated she disliked the sorting since she did not ‘agree’ with how the images were sorted and thus often found images at positions where she did not expected them. VI. C ONCLUSION AND F UTURE W ORK We have proposed a new method to sort images by color, which can be used for similarity-based layout in a storyboard. Our user study has shown that this layout can improve interactive search performance by about 20% in a common storyboard. Furthermore, it has also shown that users are able to understand and efficiently use our sorting strategy for visual search tasks. Moreover, in our test about 83% of participants preferred the sorted storyboard over the random storyboard. In future work we will continue user studies with more advanced layouts, e.g., such as the 3D cylinder layout [9], [15], in order to find out if we can further improve the search performance of the sorted storyboard. R EFERENCES [1] F. Arman, R. Depommier, A. Hsu, and M.Y. Chiu, “Contentbased browsing of video sequences,” in Proc. of the 2nd ACM Int. Conf. on Multimedia, 1994, pp. 97–103.
[2] A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, “Content-based image retrieval at the end of the early years,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1349–1380, 2000. [3] A.F. Smeaton, P. Over, and W. Kraaij, “Evaluation campaigns and trecvid,” in Proc. of the 8th ACM International Workshop on Multimedia Information Retrieval, 2006, pp. 321–330. [4] K. Schoeffmann, F. Hopfgartner, O. Marques, L. B¨osz¨ormenyi, and J.M. Jose, “Video browsing interfaces and applications: a review,” SPIE Reviews, vol. 1, no. 1, 2010. [5] K. Rodden, W. Basalaj, D. Sinclair, and K. Wood, “Evaluating a visualization of image similarity as a tool for image browsing,” IEEE Symp. on Information Visual., p. 36, 1999. [6] K. Rodden, W. Basalaj, D. Sinclair, and K. Wood, “Does organisation by similarity assist image browsing?,” in Proc. of SIGCHI Conf. on Human factors in computing systems, 2001, pp. 190–197. [7] G.P. Nguyen and M. Worring, “Interactive access to large image collections using similarity-based visualization,” J. of Visual Languages & Comp., vol. 19, no. 2, pp. 203–224, 2008. [8] G. Schaefer, “A next generation browsing environment for large image repositories,” Multimedia Tools and Applications, vol. 47, pp. 105–120, 2010. [9] K. Schoeffmann, D. Ahlstr¨om, and C. Beecks, “3d image browsing on mobile devices,” in Proceedings of the IEEE Int. Symp. on Multimedia, 2011, (currently under review). [10] W. Basalaj, “Incremental multidimensional scaling method for database visualization,” in Proc. of Visual Data Expl. and Analysis VI, SPIE, 1999, vol. 3643, pp. 149–158. [11] K. Rodden, W. Basalaj, D. Sinclair, and K. Wood, “Evaluating a visualisation of image similarity,” in Proc. of SIGIR99, pp. 275–276. [12] N. Quadrianto, A.J. Smola, L. Song, and T. Tuytelaars, “Kernelized sorting,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 32, pp. 1809–1821, 2010. [13] N. Quadrianto, K. Kersting, T. Tuytelaars, and W. L. Buntine, “Beyond 2d-grids: a dependence maximization view on image browsing,” in Proc. of the international conference on Multimedia information retrieval, 2010, pp. 339–348. [14] W. Plant and G. Schaefer, “Image retrieval on the honeycomb image browser,” in Proc. 17th IEEE International Conference on Image Processing, 2010, pp. 3161–3164. [15] K. Schoeffmann, D. Ahlstr¨om, and L. B¨osz¨ormenyi, “A user study of visual search performance of interactive 2d and 3d storyboards,” in Proc. of the 9th Int. Workshop on Adaptive Multimedia Retrieval (AMR 2011), 2011. [16] K. Hinckley, E. Cutrell, S. Bathiche, and T. Muss, “Quantitative analysis of scrolling techniques,” in Proc. of SIGCHI Conf. on Human factors in comp. systems, 2002, pp. 65–72. [17] J.M. Wolfe, Integrated models of cognitive systems, chapter Guided search 4.0: current progress with a model of visual search, pp. 99–119, New York: Oxford, 2007.
