Developer Info : Collection Hosting
Introduction
Once a collection is built, it must be hosted on a server. This page contains information about the various ways to host collections in the following sections:
There are three primary types of collections:
- Simple collections containing up to 3,000 items. In these collections, the user's experience is defined by previously generated (or static) XML.
- Linked collections, generally containing several thousands of items. These collections consist of multiple inter-linked simple collections.
- Dynamic collections, unbounded in size. In these collections, the user's experience is defined by XML generated dynamically in response to user action.
From the hosting perspective, simple and linked collections are the most straightforward, as they require only a static file server.
Dynamic collections, on the other hand, require a more complex server structure for generating XML and image content on request. What follows is a brief description of how to host
simple and linked collections, followed by in-depth discussion on the more complex hosting of
dynamic collections.
Simple and Linked Collections
Serving Files
Because simple and linked collections are static files, they can be hosted on a simple file server. On each request, the server sends the Pivot client CXML or image files. In both
simple and linked scenarios, the client will only request one CXML at a time. The distinction between the two is only that in the
linked scenario, a user may navigate between different CXML files via links in the info panel (see Collection XML Schema). As such, there is no special handling of
linked collections on the server. The only potential difference is that the file server may need to host multiple CXMLs (one for each simple collection composing the
linked collection).
See the following illustration of a static server:
Tips:
- Due to the large amount of text common in some collections, server-side compression should be enabled and the CXML mime type should be set to "text/xml."
- Server-side caching and CDNs can be leveraged to further optimize performance.
- For tips on optimizing deployment of the large number of image files in a collection, see Collection Image Content.
When Is a Simple or Linked Collection Appropriate?
Due to their simplicity, simple and linked collections are significantly easier to create and host than
dynamic collections. If a data set is small (3000 items or less) and relatively static, it is probably most appropriate as a
simple collection. As simple collections get larger, however, client performance and bandwidth become limiting factors. These factors eventually provide an upper bound to the size of the collection.
One way to get around collection size limitation is to split a larger data set into a number of inter-linked simple collections, thus creating a linked collection. With
linked collections, a large data set can be presented to the user in up to 3000 item segments. Navigation between these segments (each a
simple collection) happens via the Link and Related Link facet types (see Collection XML Schema), or with item click-through links. If a data set numbers in the several to tens of thousands, is relatively static, and lends itself naturally to inter-linking between different parts, it is probably most appropriate as a
linked collection.
While linked collections enable a larger data set to be represented, when they grow very large, their complexity can become problematic. Key difficulties encountered when creating linked collections of very large data sets include:
- Storage - Large linked collections can result in very large numbers of image files. Due to likely overlaps among different segments of a
linked collection, the number of image files can quickly multiply into the many millions.
- Updates - If the collection content rarely changes, a large number of image files may not pose a major problem. The addition or removal of an item, however, often requires large portions of the Deep Zoom image tile pyramid to be regenerated (read more about Deep Zoom tile pyramids here: Collection Image Content). As a result, maintaining a large collection that needs to be updated frequently can be cumbersome.
- User Experience – Great linked collections can be difficult to create when the data set at hand does not lend itself naturally to division into smaller segments. They are also limited in their ability to respond to un-anticipated queries, and do not naturally support search across their segments.
When these difficulties are encountered, dynamic collections offer some solutions.
Back To Top
Dynamic Collections
Introduction
To support scenarios beyond those possible with linked collections, a dynamic collection can be used. For an example, see the Wikipedia Collection. Although any number of dynamic architectures is conceivable, the remainder of this document will focus on the approach taken by the Pivot team to create the Wikipedia collection.
Dynamic collections follow a similar sub-collection methodology to linked collections, but use a dynamic server structure to mitigate the issues raised in the previous section. In
dynamic collections, the CXML and corresponding Deep Zoom image collection is generated by a web server at request time. The following figure illustrates a sample architecture:
Although more complex to construct, dynamic collections have the following advantages:
- Storage & Updates - Because much of the CXML and image content is created at request time, file proliferation is less of an issue, and updates can be made by changing only a small number of files (or tables).
- User Experience – With a more flexible structure, dynamic collections enables scenarios such as search and serving of un-anticipated query patterns. Further, because the cost of duplicating items in various sub-collections is mitigated in this architecture, inter-linking between sub-collections can be much more complex.
- Enabling Large Existing Data Stores – Many existing large data sets already have dynamic data stores and pipelines, which can be leveraged to present the data in the Pivot collection format.
Back To Top
Pivot Team's Dynamic Collection Implementation
Overview
At first inspection, building sub-collections in real-time on a web server may seem straightforward. Achieving good performance and scalability properties, however, is not without its nuances. The figure below shows the request flow between the client and web server for the Wikipedia collection.
Request Handlers
The three types of request handlers required are:
- a CXML handler,
- a DZC handler, and
- a tile handler.
The CXML handler is responsible for building the primary CXML with facet definitions, item metadata and a link to the Deep Zoom image collection based on some query which defines the sub-collection. We used querystrings on the CXML files to represent client queries into the system. Note that when handling a tile request, Deep Zoom always asks for a tile in the form "<DZC
name>_files/<level>/<x>_<y>.<format>". A URL rewrite rule is used to map that URL format into a dynamic server request with appropriate query parameters.
One of the most important steps in the process is building collection tiles on demand. The tile building algorithm can be found in the appendix. The tile building process must determine, based on
level, x, and y, what items appear on the requested tile. It must then downsize the raw imagery for those items to the appropriate size for
level and place them in the tile according to appropriate Morton layout relative to the item’s position in the results set returned during the DZC step. Currently, JPEG’s are the preferred image to use for the raw image, since downsizing and decoding can happen in the same step given the nature of the Discrete Cosine Transforms by which the JPEG is compressed. For more information on image formats, see Collection Image Content.
The query often has to be enacted on a data storage system of some kind. In many cases, a relational database such as SQL server will suffice for this purpose. The only caveat is that raw imagery for each item will need to be stored for quick retrieval during the tile stitching phase.
Performance Optimizations - Request Caching
A careful examination of the system will reveal a nuance of great importance. Each request type, or even requests within the same type will often reuse the same piece of data. As an obvious example, consider the CXML and DZC building steps. To construct the CXML, items which match the query are found, their metadata serialized and then returned to the client. To construct the DZC, items which match the query are found, image information examined and then serialized. The step of finding items which match the query is duplicated. Likewise, for building tiles, at different levels, raw imagery is often reused from level to level, so fetching it repeatedly from disk would be wasteful. To prevent executing duplicate operations, request affinity and application server caching can be used, as seen in the figure above. Request hashing is a good choice for affinity, as it has the potential to group like queries together.
Since collection building is not computationally cheap, a content delivery network (CDN) can be used to group like requests and prevent re-computations of the same most visited sub-collections. If usage of a CDN is not available, web server response caching can be used.
Performance Optimizations – Image Tiles
The first few image pyramid tiles have the ironic property of being the smallest information gain for the client and the most difficult for the server to build. However, for client animation reasons, they are important. To circumvent the difficulty associated with building these first few tiles, the dominant color for each item can be pre-computed and then used. This allows the server to avoid loading all images into memory to render the first tile. However, the server should still proactively load the raw source images required to build the tile pyramid into memory. Starting a background operation as early as CXML request time is best.
Another implementation detail involves whether to build the entire image pyramid after the first request, or per tile request. It is advantageous from a server resource standpoint, and ultimately from an end-user standpoint, to build each tile per request. This is because the whole pyramid is rarely requested, so building the extra unused tiles may be a waste. In the event that the whole pyramid is requested, it is often requested in a timeframe slower than what the server is capable of generating the pyramid in, so resources are unnecessarily consumed.
Tile building can also be made much less computationally expensive by creating tiles from raw images, with a technique called macro-blocking. A JPEG image is primarily compressed by a Huffman encoding, followed by a discrete cosine transform of 8x8 portions of the image, referred to as sub-images. Since the images to be placed on the resulting pyramid tile above the 2nd level are 8x8 or some factor greater, it is possible to only undo the Huffman compression on the raw images, place the images accordingly on the resulting pyramid tile, and then re-apply the Huffman encoding.
Back To Top
Feedback
Join our technical discussion to interact directly with the Pivot team. We hope you’ll join this community and share your work.
Appendix: Tile Building Sample Code
For sample code illustrating how to create a tile builder, see "TileBuilder.cs" in the following zip file: PivotCollectionFiles.zip.