Is displacement mapping the last secret?

Browsing for more clues concerning the Revolution´s last secret, I came across the issue of displacement mapping, an improved version of bump mapping, if you will. This cropped up briefly in relation to the Nintendo Revolution some months ago, but I believe it has not been given enough attention. But before we go into that, let´s understand what this technique is about.

---

1. What is displacement mapping?

Wikipedia has this to say about displacement mapping:

Hardware displacement mapping can be interpreted as a kind of vertex-texture mapping, where the values of the texture map do not alter the pixel color, but change the position of the vertex instead. Unlike bump mapping and normal mapping, displacement mapping can in this way produce a genuine rough surface.

Let´s look at the diagram accompanying that article and allow me to repeat the above in my own words.Rather than altering the colour of the pixels, the displacement map alters their height, actually raising or lowering the mesh. The obvious advantage over bumb mapping is that it actually creates a rough surface. Bump mapping only creates the illusion of one. Whereas the effect of bump mapping depends on how you look at the rendered object (the effect will be lost in a profile view, i.e. from the side), this technique works irrespective of viewing angles.

The following image is an example of what this technique can do, courtesy of ZBrush.
Displacement mapping allows you to can create elaborately detailed objects with a low polygon count. Here´s a comparison between the various mapping techniques, taken from Johannes Hirche´s 2004 Ph.D. thesis Adaptive sampling and tessellation for displacement mapping hardware, which focusses on ´efficient rendering of such displacement maps, mainly targeted at graphics hardware architectures´.I am sure you can tell that it is a significant step up from the previous techniques. However, it comes at a price. While the poly count is significantly lower, there is some strain on the CPU. Johannes Hirche writes:

Rendering displacement mapped surfaces is a process that involves a significant number of geometric and arithmetic operations. When applied to a triangle mesh, it involves prior retessellation of the base domain surface and transformation of the vertices and normals. Even on fast CPUs, it is a time consuming operation, wasting bandwidth and processing power.

This is why displacement mapping has not been widely used in real-time graphics. However, new and refined techniques allow for displacement mapping to be implemented in real-time. Again, Johannes Hirche writes:

The main focus was to explore new techniques suitable for hardware implementation in order to reduce the bandwidth strain on the system bus by moving the tessellation process onto the graphics subsystem. (...) A possibility to overcome these problems is to tessellate the individual triangles sequentially and to adaptively add triangles where necessary, until a desired level of accuracy is
reached. (...) With only minor user interaction or conservatively predefined input parameters the sampling schemes produce adaptive tessellations with very low error measures.

The above covered the immediate advantages of displacement mapping and some current problems with it, as well as how they may be overcome. Let´s see why this is relevant to Nintendo and, perhaps, the Revolution.

---

2. Why could displacement mapping be Nintendo related?

Firstly, there is a Nintendo patent that has caused this topic to crop up in this community before. It is entitled Method and apparatus for efficient generation of texture coordinate displacements for implementing emboss-style bump mapping in a graphics rendering system.Its abstract is a bit of a mouthful, unfortunately. Read my highlights, though:

A graphics system including a custom graphics and audio processor produces exciting 2D and 3D graphics and surround sound. The system includes a graphics and audio processor including a 3D graphics pipeline and an audio digital signal processor. Emboss style effects are created using fully pipelined hardware including two distinct dot-product computation units that perform a scaled model view matrix multiply without requiring the Normal input vector and which also compute dot-products between the Binormal and Tangent vectors and a light direction vector in parallel. The resulting texture coordinate displacements are provided to texture mapping hardware that performs a texture mapping operation providing texture combining in one pass. The disclosed pipelined arrangement efficiently provides interesting embossed style image effects such as raised and lowered patterns on surfaces.

This proves that Nintendo has not only been interested in this technique but is a patent holder. The section entitled ´cross-reference to related applications´ references 25 separate provisional patent applications that are thereby incorporated into the patent. Almost all of them date back to 2000. This would suggest that it is an important patent that has kept Nintendo busy but doesn´t date back too far to be cutting edge.

Secondly, relating back to making the process of displacement mapping more efficient and less of a strain on the CPU, one way of adaptive tessellation might actually be the last Nintendo patent I talked about in great detail, called Three-dimensional image generating apparatus, storage medium storing a three-dimensional image generating program, and three-dimensional image generating method. A number of readers pointed out that the patent had nothing to do with actually visualising graphics in 3D, but rather optimizing a 3D world to be viewed on a 2D display. Then, that patent made little sense to me. But in the context of trying to reduce the computational strain on the CPU involved in displacement mapping, this may make perfect sense.

Lastly, whether the Revolution´s graphics chip will turn out to be based on the R520 or R530, it will be Radeon technology. And its manufacturer ATI has the following advice for developers on their Designing for Radeon development support page:

Use multi-texturing effects for realistic low polygon primitives. For example, you can use emboss style bump mapping to achieve the illusion of a bumpy surface that would take a lot more polygons to approximate otherwise. Similarly, other intelligent use of texture maps can reduce the polygon count of your mesh designs.

This may not be unusual, since nVidia will undoubtedly have similar advice on their development support pages, but at least it shows that ATI is also very concerned with this technique. In fact, ATI supported this technology earlier than nVidia, it seems. While the Radeon 9500/9700 was capable of displacement mapping, the GeForce FX was only partly so. The Radeon 9700 Pro already supported adaptive tessellation. In fact, ATI has an exclusive technology called ´Truform 2.0´, which is a kind of tessellation.

---

Now, there have been numerous rumours about Nintendo having discovered some kind of secret development technique. This may not be secret per se, but it would make sense if Nintendo had discovered a way of implementing displacement mapping efficiently. They have a patent relating to this technology and they have a strong ally who has some expertise in this field.

It may explain why Nintendo have not yet talked about the graphics chip or shown any real game footage yet. It would also explain why the basic hardware features that have been suggested seem to be underpowered at face value, yet Nintendo maintains that their graphics will be on par. This may yet turn out to be the Revolution´s last secret.

Sources: Wikipedia, Adaptive sampling and tessellation for displacement mapping hardware by Johannes Hirche, Tweaktown, 3D Chips, HardOCP, 3D Test

Image sources: Wikipedia, ZBrush, Johannes Hirche

EDIT The patent was updated and changed patent numbers. I updated the link. The difference (as far as I can make out) is that claims 15 - 23 appear to have been cancelled. I don´t think they were before. The google cache for the patent does not exist. Does anyone have a complete copy of the original patent?