Understanding the distant past isn't easy; we weren't there. The discipline of Virtual Heritage emerged in the 1990s to remedy that situation. Archaeologists (those who study the material culture of the past) typically work with lots of jumbled and missing clues, and there is no instruction manual. The idea is to interpret the evidence as best we can to take into account both what was found and what might be missing. Archaeology arose during the age of fine draftsmanship, meticulously refined color renderings crafted slowly by hand, and equally elaborate prose. But that was centuries ago, and you don't see many people these days riding in horse-drawn carriages, writing their dissertations with quill pen, or finding their way by candlelight. So, why is the study of the past, for the most part, still typified by 18th-century conventions--hand-drawn plans, static images, and long descriptive text? [hover over all images to enlarge]

I'm not saying that those methods are bad; I'm just saying that we've moved on. Past lives, events, and settlement changes didn't happen in 2D, they didn't happen in black and white, and they were not static segmented slices of time. But how do we convince the broader historical community, and especially the archaeologists whose use of images to supplement their written descriptions is taken for granted, that interactive 3D environments are instructive, not eye candy?

How have we been visualizing the past? With apologies to Tommy Lee Jones and the Men in Black movie, 500 years ago, we could only muse about how surviving texts from the past might connect with the few known relics from the past.

150 years ago, we had photographs and hand-made sketches to record new finds and help make sense of our interpretations.

15 years ago, we had interactive computer graphics--for a million dollars. Today, archaeologists use micron-accuracy laser scanners, high-resolution satellite imagery, 3D earth viewers, location-aware smartphones, and pervasive wireless high-speed networks. Given this pace of change, just imagine what we'll have tomorrow (and to help you along with that, I'll provide you with some hints).

First, a little background, originally, as an architect and an archaeologist I worked amidst the dirt and the real, but now travel strictly amidst the pixels and the virtual.

For the last 15 years, my companies have been digitally re-creating ancient sites, buildings, and artifacts for individuals, excavations, museums, schools, antiquities services, foreign governments, publishers, and TV programs. We've worked on dozens of projects, with many archaeologists, covering all types of data and visualization goals, ranging from teaching packages, to research resources, to digital publications, to small hypothesis tests (see the sample renderings output from some of our virtual worlds at the left). We've found that each project routinely generates unexpected revelations about the excavated record, well beyond what the contributing scholars had believed possible. Most thought they knew their material thoroughly, but came to have a new appreciation of it when viewed in virtual reality.

Today, I'll be talking today about virtual heritage, discuss some virtual heritage projects that we've created over the past 20 years with a focus on the Northwest Palace of King Ashurnasirpal II at Nimrud (Assyria), and conclude with a glimpse at a paradigm shift about to hit archaeological fieldwork and visualization.

Throughout human development, people have attempted to capture the essence of an experience and make it enjoyable for others. Cave paintings (as at Lascaux, at the left), storytelling, sculpture, theater, music, and books all offer other views of the world, other experiences, other beliefs, and other times, to stimulate the imagination, to instill wonder about the fantastic, and to speculate about the spiritual. Radio, television, and moving pictures continued a quest for increased realism and of being there. Historically, these same presentation techniques (and often the same content, too) were used also to educate.

Visualization has become a recognized means of presenting data and concepts, increasing comprehension and assimilation (our brains take in much more information through our eyes than through our ears). Thus, in education, textbooks are illustrated and audiovisual materials are widely used. And now we have digital devices. Virtual reality is just another step in the age-old drive toward heightened visual experience based on the world around us, as it is today or as it might have been in the distant past. Although military, government, and scientific applications of immersive simulations and some form of virtual reality have been around for decades, the application of this technology remains under-explored for archaeology.

For example, I'm sure some of you have found, as I did, that when teaching archaeology using slides, students have a tough time deciphering ancient sites from just views and plans. Even archaeologists have difficulty interpreting a scatter of rocks, let alone envisioning how ancient people lived and used these sites.

Artists' impressions offer some additional information, but there has to be a better way to teach history.

It took a while for software to catch up, but now we can begin to appreciate ancient places in ways that approximate the viewpoints of the original inhabitants. Here is one future of archaeology (click here and scroll to the bottom to view the Learning Sites full virtual reality re-creation of the Vari House, Greece). Since the past happened in 3D, that's the way it should be visualized and studied--if there is anything that you take away from this talk, it is that point. Archaeologists cannot truly hope to understand the complexities of the distant past without testing their theories in an environment that as closely as possible mimics that distant past. It's the only way to ensure that our interpretations and hypotheses are accurate.

Already during the 1980s, computers were used to visualize archaeological finds, mostly low-resolution 3D models of hearths, pottery, and small finds. As software improved, entire buildings and sites were being digitally reconstructed with the output renderings enhanced using image editing tools (as in the example at the left). A leap in capability occurred in the early 1990s when virtual reality left the lab and hit mainstreet. Virtual reality is interactive, self-directed, real-time navigation through a computer-generated 3D space displaying a synthetic scene. When this technology was embraced by a few innovators, digital archaeology spawned virtual heritage. Virtual heritage then is the use of virtual reality technologies for the visualization and study of the past.

One of our models from that early era was based on the provincial Assyrian settlement at Til Barsib, Syria, which is comprised of a palace, lower town, and fortification walls mostly dating to the 9th century BCE. The site was excavated by the French between 1929 and 1931.

Today, the palace is completely gone with a new village built over its foundations.

The palace had the best preserved and most extensive set of Assyrian wall paintings that had survived from antiquity. The French team's artist, Lucien Cavro, made watercolor copies of them, and then workers detached the paintings from the wall surfaces and shipped them to the museum in Aleppo, where the ancient work was coated with shellac to protect them.

Old shellac, however, eventually turns opaque and chemically bonds itself to the surface beneath it. So, the unique originals can no longer be seen, and the shellac is now fused to the Assyrian paint, making removal impossible. All that leaves for anyone to study are the watercolor copies, but they are in the Louvre archives. As we built our 3D model, we applied new digital images of the archived drawings as texture maps in our reconstructions. Our visualizations became the first time anyone had seen the surviving decoration back on the walls in simulations their original architectural contexts. Now the Assyrian rooms and their amazing stories can be fully appreciated as never before in color and in 3D.

In another example, we were asked to focus on the major problem facing both statesmen and planners of the 5th century: what to do about the ruined temple of Athena Polias caused by the Persian sack.

The traditional theory assumed that the rubble was merely cleared away to make space for the Erechtheum, Parthenon, and Propylaea (note the empty space between the Parthenon and the Erechtheum). We were to test a bold new hypothesis: could a damaged Old Athena Temple have remained standing (as in our reconstruction at the upper right)? We sifted through 2 centuries of excavation data, ancient inscriptions, ancient writers descriptions; and 16th-19th centuries travelers' paintings, engravings, and sketches of the Acropolis. We learned that, no matter how many times such famous monuments had been measured, none of the sets of dimensions or elevations matched, leaving the buildings floating in relation to each other.

But with the help of old photos in Harvard's archives (as in the example at the left, showing definitively the relationship between the Erechtheum and the foundations of the Old Athena Temple), we were able to definitively fix that relationship, and to argue that the Old Athena Temple did remain standing in ruins as a war memorial that the Classical buildings were carefully arranged around the ruins, that the Erechtheum caryatids should be reinterpreted as mourners, and that, surprisingly, parts of the most important building in all of ancient Athens stood until the early 19th century, when they were unceremoniously swept away during the Acropolis clearing that followed the Greek war of independence. Thus, seeing the past in 3D forced a needed correction in the interpretation of basic bits of evidence.

Recent advances in digital technologies have allowed virtual heritage to inch toward another cusp of innovations. The latest buzz concerns projection holography, laser-plasma virtual displays, autostereo screens, and artificial intelligence-based interactive avatars. Today, databases linked to virtual worlds are common; more complex virtual environments are now easier to build and navigate, and an emphasis on lighting, vegetation, and character development make worlds more engaging. Viewing the past as 3D environments from a point of the original inhabitants allows researchers to ask new questions.

Such as, how can we study delicate objects, objects discovered in multiple fragments, or objects stored away at distant museums? In 1920, Sir Flinders Petrie's excavation at Gurob, Egypt, revealed a disassembled and broken wooden ship model in an unmarked New Kingdom tomb. The pieces are now housed in a drawer at the Petrie Museum in London. Recent research shows that the model represents a vessel type used by both the Mycenaeans and the Sea Peoples. This model is the most detailed representation presently known and is unique among all known examples in its polychrome painted decoration.

With such broken and fragile objects, museums are not about to allow anyone to handle and "experiment" with its many pieces and possible configurations while trying to figure out its original form. Our tasks were not only to use photographs and drawings to recreate various versions of the ship -- as-found and as-built -- but also to devise new tools for manipulating our reconstruction so researchers could compare various iterations with known representations of similar ships. Thus, interactive 3D models can provide global access to delicate and distant objects, enabling more efficient and effective analyses.

One of our projects has spanned the entire developmental history of VH, from the mid-90s until today. The ancient capital city of Nimrud (ancient Kalhu) sits east of the present course of the Tigris River. The citadel was founded on a ridge near the edge of the flood plain.

About 160 years ago, a young British adventurer named Austen Henry Layard began excavations there. He unearthed the prototypical and best preserved of the palaces of ancient Assyria, that of King Ashur-nasir-pal II (who ruled from 883-859 BCE). The building is called the Northwest Palace because it was found near the northwest corner of the citadel mound.

Layard worked from 1845-1851 excavating and documenting his finds. Here he is at Nimrud, supervising the removal of two human-headed, winged bull figures–called lamassu–from a gate in the central courtyard of Ashur-nasir-pal's palace. He also removed from the walls of the Palace the stone bas-reliefs that ornamented the main rooms.

Over the decades various visitors to the site received permission literally to mine the Palace of what were deemed duplicate sculptures, among them some of the finest artwork of the period. When American missionaries travelled to the site, which for them was both a curiosity and of religious interest, because Kalhu is mentioned in the Bible, they purchased whole or parts of relief slabs. The sculptures went home as gifts to their respective colleges and seminaries from Virginia to Maine. This letter is from the archives of Williams College in Williamstown, Massachusetts, the first American college to receive an example of a bas-relief from Nimrud. Records like this show that the bas-reliefs, if they were not already broken, were cut up into smaller pieces so that they could be transported more easily over land to where they were loaded on ships bound for Europe and then America.

That reminds me of an ironic story regarding the transport of other Assyrian reliefs out of Iraq during that period. Like those from Layard's excavation at Nimrud, others were cut up for easier loading onto camels for their trek across the desert to the Mediterranean Sea and then easily crated for passage to European and American museums. Those reliefs that were not damaged by cutting, but remained intact, were placed onto barges along the Tigris River to be floated downstream to the Persian Gulf and then loaded onto ships there. The ironic twist is, those reliefs that remained whole sank into the river when the barges were attacked soon after leaving the Assyrian palace sites. Unfortunately, since the reliefs are made of a soft calcareous limestone, not long after being plunged into water, the reliefs that were left in one piece melted and disappeared.

Proof of the practice of choosing specific artwork is everywhere evident at the site. For example, this slab, from a room in the palace's southern wing, must have proven too great a task for removal, even though its upper half was broken when found, or got broken while it was being considered as an example worth taking. In any case, the human head of the winged deity, one of the protective divine figures that feature prominently in the iconography of the audience halls of the Northwest Palace, was chopped away from the rest of the slab and found its way to Chicago's Art Institute as a single art object. In the old exhibit there, it was displayed in a frame like a picture, as a discrete piece of art, treated as if unrelated to the rest of the original figure. Today it's shown without the frame and with an accompanying drawing to show at least its immediate context.

Because there are several possible ways that sculpture was taken from Nimrud, with or without permission, we don't know how many reliefs remain to be discovered. Every few years, another fragment surfaces on the antiquities market. Tracing the buying and selling of these reliefs over the last 160 years has become a daunting task. We can now document the whereabouts of about 325 complete and fragmentary sculptures from the Northwest Palace out of context in 75 museums and private collections around the world (such as these Northwest Palace reliefs discovered in Newbattle Abbey, Scotland; image courtesy of Julian Reade; hover over to enlarge). Because of the sculpture's global distribution and their vast numbers, neither scholars nor the general public can fully comprehend the palace or its unique decorative stories and architectural splendor as King Ashur-nasir-pal intended, as the ancient Assyrians once used the complex, or even how it looked when first discovered.

Traditional methods of trying to understand this huge building (over 100 surviving rooms; covering five acres) focused on sorting and then assembling drawings or photographs of the reliefs into categories by subject or by room. This is how each room was typically studied (see the basic reference plan of Room I at the left; hover over to enlarge). First, researchers would mark up on a plan whether a relief was complete and restored in situ, partially restored, or bases only.

Second, an elevation drawing was then prepared to show what's in situ and what's missing.

Third, a scheme was devised that took into account the direction in which each figure stood relative to the sacred trees. As a result, a few new joins were found. Photos of the reliefs were then assembled to show all the pieces belonging to this room according to the schematic: matching inscription fragments, iconographic details, and the sizes and shapes of the cut and broken stone.

Fourth, the photographs were pasted up to scale which could be given to an artist who drew publication drawings.

Finally, the room could then be visualized as a whole composition; but the process took years. Needless to say, these attempts ultimately proved unsatisfactory, although they did allow Assyriologists to organize the reliefs' subject matter, to appreciate the scope of the global distribution of the material, and to make some broad assessments about the iconography.

Many key issues come together in this one project--the limitations of 2D media, site deterioration, loss of object context, and the advantages of 3D modeling. All these factors made the Palace and its history an ideal candidate for the application of virtual reality technology. This opportunity was recognized by the late Samuel Paley when he approached us in 1998 with the prospects of rethinking all of our knowledge and assumptions about the Palace.

We began by assessing the accuracy of previous visualizations, especially those drawings and renderings that have been our basis for understanding the site and Assyrian history since the mid-19th century. Here (at the left; hover over to enlarge) is one of Layard's drawings, supposedly of an Assyrian throne room and a standard rendering reproduced often, but note its questionable details--the open roof, the incorrect height of either the people or the reliefs, and the pastel Victorian color scheme.

Further, when creating texture maps for our 3D computer model, we discovered that the published drawings weren't very accurate, not in detail or the shape of figures, nor often in transcription of the inscriptions, which have been relied upon for decades as the basis for art historical comparisons and cultural extrapolations. This offers a cautionary tale for those relying on period source material--always go back to the originals.

To deal with the global dispersal issue, we began contacting each of the institutions and private individuals holding reliefs from the palace and negotiating with them for new digital photographs of the sculpture so that our model would be as precise and detailed as possible. This has been a slow process, and one that we've not yet completed. There are over 500 reliefs in the Palace. To fill in gaps in the evidence, we have colleagues combing the field notes and diaries of the many people who have excavated here over nearly two centuries. Nevertheless, things are progressing (see our renderings at the left; hover over to enlarge; click here to see our current progress on the Northwest Palace).

One important throne room relief (B13, seen at the left; hover over to enlarge) was vadalized either during the 1st Iraq war or soon thereafter.

A duplicate of that relief used to sit here, behind the throne.

Since the duplicate was in nearly perfect condition, Layard shipped it off to the British Museum. It's a gorgeous piece, but divorced from its original context, it's only an isolated work of art...

...not a useful piece of history that informs us about Assyrian ritual, the sculptural program of the throne room as a complete story, or about how carefully planned that story was for ancient visitors to the Palace.

Here's a view of that same relief in its reconstructed context.

And here we see how the king would have obscured and substituted for the central tree when the scene would have been seen by visitors. The focus of the scene on the back wall now becomes the king, rather than the tree, an iconographical nuance heretofore unrecognized until our models demonstrated this relationship.

The tree substitutes for the king in his absence or in special alignments, as with the view through the central (ceremonial) throne room doorway.

As you have seen, we are building a detailed computer model of the Northwest Palace based precise measurements of existing walls and reliefs, Layard's notebooks and drawings, archives in the British Museum, and the interpretive assistance of an international team of Assyriologists. We immediately gained new insight into Assyrian architecture, use of lighting, the carefully planned locational relationship between the wall reliefs and interior circulation and sightlines, and thus about the iconographic, educational, and propagandistic purposes of reliefs and the functions of spaces. Specifically, we were able to test new theories about how the rooms may have been lit, about a presumed arched decoration behind the throne, the use of sacred trees to represent the king in his absence, and how the throne room might look from a visitors' point of view (see a short snippet of our progress as of 2011 below).

Our results convinced Sam Paley, who told us: "I am particularly pleased with such virtual reconstructions, because I am able to visit the site and travel through it and see things that I could not see in one image before or even see easily and quickly if I were able to visit the actual ruined site; I could appreciate spatial relationships the way the ancient Assyrians intended." This project successfully assembled, for the first time, globally scattered sculpture so that the original decorative schemes and narrative programs could be fully appreciated in a simulation of their original scale, lighting, color, and 3D spatial complexity. What I've shown here is from our research resource that collocates and links all the information inside the virtual environment for in-depth analysis, teaching, hypothesis testing, and for the enjoyment of wandering around inside an ancient treasure. In the process, archaeologists came to realize how much data are missing despite 160 years of research; and how wrong the long-accepted illustrations of the building are. Testing data, and assumptions on that data, in three dimensions is crucial to verifying their validity. The Northwest Palace virtual world now acts also to document the on-going deterioration of this World Monuments Fund endangered site.

Nearly two decades of virtual heritage have proven that there is:

• no better way to test the validity of archaeological assumptions;
• no better way to test complex spatial, behavioral, and temporal hypotheses;
• no better way to test the accuracy of past interpretations and evidence;
• no better way to assemble globally dispersed artifacts back into a simulation of their original contexts;
• no better way to visualize intrasite change and development; and
• no better way to absorb complex datasets about the past than visually, interactively, and in 3D, just as we do in real life.

But what's really important is that we need virtual heritage in order to do our best work. And anything less than our best work would be unfair to colleagues and the general public.

A few years ago, historian David Staley [then Executive Director of the American Association for History and Computing] wrote that computer visualization when used "to represent simultaneity, multidimensionality, pattern and nonlinearity with … speed and efficiency" can do what "prose cannot." In his book Computers, Visualization, and History, Staley argues that the real impact of the computer has been as a graphics tool more than as a processor of words. The importance of 3D imagery lies in its ability to address longstanding philosophical observations that "Narrative is linear, Action is solid." Thus, the advantages of interactive computer graphics are that they can present a deeper and more richly rewarding history by giving a 3D solidity to past places and events, and at the same time act as a repository for the images, words, and objects that together define who we are and how we got here.

What if we now extrapolate from what I've shown by adding computer vision, machine learning, social media, and cloud-sourcing. I submit to you one not-so-distant future for archaeology.

We visit the fictional site of Trogitz, where the fieldteam arrives with their usual equipment, that is, only a few handheld devices. As the dig proceeds, dozens of progress photographs are taken, showing all changes in depth, all soil changes, all in situ finds, and some skirting shots of the surrounding landscape. While the fieldteam works, entering observations and object descriptions onto Web-based forms via the handheld devices, photographs are automatically linked to finds and geo-referenced from trench to trench.

Simultaneously, data extracted from the images are automatically converted into virtual reality models, each fully textured, dimensionable, and linked to the database notes and daily finds.

Every time an artifact is uncovered, it, too, is fully photographed during its excavation and after its removal and cleaning. These photos also are used to automatically create and geo-reference virtual reality object models.

This process continues as the season unfolds, producing files that can create 3D printed replicas like this. No laser scanners, no survey equipment, no architects, no waiting until the end of the season for the results.

Uploading new data, taking notes, doing hypothesis testing, preparing 3D models of architecture, querying the evidence so that you can readily adjust your dig strategies, and preparing publishable output ALL happen in real time, as the excavation unfolds. This is a new, more efficient and effective, way of working. This is REVEAL, a new, unique, and free software package, and the only one excavators may ever need for fieldwork.

But there's more, while all this is happening the continually updated database and the linked VR models are uploaded to the Trogitz online collaborative network, where 'friends' of Trogitz can explore the excavation and artifact models in realtime, search the database, and comment on the results by posting e-notes inside the virtual worlds of each trench. These digital comments can be instantly read by other 'friends' or by the fieldteam, allowing for global collaborations in real-time while the excavation progresses. Experts can advise the fieldteam in ways that would normally take years; publication-ready analyses, visualizations, and reports can be created and disseminated on demand rather than waiting years; and classroom-ready materials await the next semester's classes so that teachers and students always use the most recent, most accurate, and most comprehensive data possible. This wild-and-crazy Trogitz scenario is becoming a reality right now, as our team expands the functionality of REVEAL and the software is tested at excavations around the world.

Given the rapid pace of change in computer technologies, what may seem outrageously far-fetched today will likely become old hat in less time than it would take me to explain it. We already have location-aware smart sunglasses with embedded computers and high-resolution cameras; link them to a 3D-based semantic Internet with the capability of projecting holographic images of your REVEAL-created virtual reality models and you have a personal virtual timemachine. Please use it wisely. In conclusion, how much we can gain from archaeological investigations depends not only on the questions we ask about the past, but equally on the visualizations, virtual or otherwise, chosen to illuminate the answers.