Thursday, December 15, 2005

Is parallax mapping the last secret?

Please welcome Nintendojo, Revo-Europe and 4 Color Rebellion to our little discussion. All of them have picked up on this debate (all without quoting me as the original source, I should mention), alongside uncountable forum posts. I would like to take this opportunity to thank every single person visiting my blog for helping to push the daily visits to a staggering count of 8.500.

Concerning the topic at hand, it has now been suggested by a number of people commenting on this story that Nintendo may in fact be planning to use yet another type of mapping called ´parallax mapping´. As ever, I have direct evidence in the form of new patents, but let´s look at the technique itself first.

Wikipedia is a good starting point again:
Parallax Mapping (also, Photonic Mapping, Offset Mapping or Virtual Displacement Mapping) is an alternative to the bump mapping and normal mapping techniques applied to textures in 3D rendering applications such as video games. To the end user, this means that textures (such as wooden floorboards) will have more apparent depth and realism with less of an influence on the speed of the game.

Parallax mapping is done by displacing the texture coordinates such that the texture occludes itself in accordance with a height map. Next-generation 3D applications may employ parallax mapping as new graphics algorithms are developed.

An easy way to understand this concept is to close one eye, take a pencil, point it at your eye, and move your head left and right. Parallax mapping takes that pixel on the far left of the pencil when it was facing you directly and stretches it accordingly to simulate your angle in comparison to the pencil.

Parallax mapping is also a way of faking displacement mapping.
To see exactly what parallax mapping does, let us look at some samples, courtesy of InfiScape.
The effects are obvious, particularly in the second example where it is combined with bump mapping. Basically, it works like this: Imagine looking at a raised point on a surface and then moving sideways. Due to perspective, the raised point will appear to move in the opposite direction. Parallax mapping can modify bump mapping to take account of such a move. This is referred to as offsetting. No extra polygons are needed. A height map is enough to calculate the offset value. It´s very simple to implement, is easy on the CPU and can thus be combined with other mapping techniques to achieve results, easily comparable to displacement mapping (though it does not actually alter the height of the pixels, I should point out).

Now, there are plenty of quotes in the patents mentioned in my earlier posts below that seem to relate to this technique. But I also came across this new one entitled ´ Method and apparatus for interleaved processing of direct and indirect texture coordinates in a graphics system´, which seemingly referrs (in some part, at least) to parallax mapping, as it often mentions an offset matrix that is applied to the textures:
In a graphics system having a memory containing texture data, a method of indirect texture referencing comprising the steps of: (a) using a set of indirect texture-coordinates to retrieve offset data from the memory; (b) multiplying the offset data by predetermined values forming elements of a texture offset matrix to produce a set of texture offset coordinates; and (c) using said set of offset texture coordinates for referencing texture data stored in the memory when mapping a predetermined texture to a rendered polygon.
Reading further, we learn about what effects this technique might have on a hardware arrangement (formatting is mine):
To provide an enhanced repertoire of texturing capabilities for a 3-D graphics system, the present invention provides a versatile texturing pipeline arrangement achieving a relatively low chip-footprint by utilizing a single texture address coordinate/data processing unit that interleaves the processing of logical direct and indirect texture coordinate data and provides a texture lookup data feedback path for "recirculating" retrieved indirect texture lookup data from a single texture retrieval unit back to the texture address coordinate/data processing unit. The interleaved coordinate processing and recirculated/feedback data arrangement of the present invention allow efficient processing of any number of logical direct and/or indirect texture mapping stages from a smaller number of hardware texture processing units while preserving a fine granularity in the overall data processing flow.
Concluding, it seems to me that this technique is the most likely to be used in the Revolution. It almost has the visual quality of displacement mapping while requiring next to no more calculations.

Sources: Wikipedia
Image source: InfiScape

25 comments:

Anonymous said...

First post! Woo!

I like the idea of displacement mapping better, but it's probable that they'll use all three techniques somehow next gen.

Anonymous said...

Please be patient, as there is a point after the long-winded stuff.

The "displacement mapping" patent was for using a hardware accelerated bump mapping. Contrary to what others wrote, there are other methods of displacement mapping that do not increase polygon count by several orders of magnitude.

For instance, the thesis paper linked from that article mentioned a ray-casting method in eye-space (i.e., what you see, not the model as it is defined), to build the image with about 12 extra polygons per viewable polygon. That could be tricky, but the Gamecube had a very fast frambuffer/z-buffer that would aid such a method very well.

However, I doubt that is the subject of the "emboss style bump mapping" patent application. I think it really is accelerated bump mapping, producing effects similar to the normal mapping that is very common on xbox360 and many PC titles (e.g., Half-life2, and the doom3-based games).

Now, *THIS* patent is a novel, and UNIQUE, means to reduce the amount of texture/pixel/texel processors necessary to produce the bump effects. It streamlines and optimizes the functionality of "pixel shaders" (the xbox360 has up to 48 and the PS3 24).

In fact, image number 5 shows a box labled "Texture Coordinate/Bump Processing". This patent is for the hardware used to accelerate the math/software/technique defined in the previously discussed "bump mapping" patent application.

Nothing more or less, but still a pretty nice look at what could be in Revolution: making bump mapping really cheap to use on all in-game objects.

Also, this hardware accelerates pretty much any effect/technique that uses texture lookups/offsets and such between multiple textures. The recent ATI work on "Parallax Offset Mapping" (not to be cofused with "parallax mapping" as they are different) provides a superior fake displacement mapping to bump/normal/parallax mapping for not-much-higher processing cost. This new hardware could make implimenting it in Nintendo games more of a reality, which would make for some great looking games.

--wellsed

Anonymous said...

woo hoo!!!!!!!!
U-S-A
U-S-A

Anonymous said...

Hi Falafelkid,
What I don't quite get is(bei all dem Fach-Chinesisch)..Why should Nintendo patent all this stuff when in fact Ati is in charge of the contract(developing Hollywood) that Nintendo hand over to them ?
Strange..

-Infamous-anonymous-light shedder.

Anonymous said...

Looking at some of the other patents Nintendo filed in the same period shows that they seem to be heavily reliant on texture processing to make up for less geometry.

There's a lot of patent applications for Nintendo in this domain and this new hardware patent helps *ALL* of them.

Anonymous said...

@ light shedder: I also am a bit confused as to why Nintendo rather than ATI would be patenting these graphic optimizations. Does Nintendo "own" the rights to the R&D that they have paid ATI to do? I don't know the answer to this, but I'm guessing that Nintendo has to patent the aspects of this technology that relate to gaming.

Although I do see this being used in the Revolution, along with many other graphical tricks, I don't really think this is the final secret. It might relate to the final secret in some way, but its not the secret itself. That said, it is these kinds of optimizations that will make Revolution's graphics comparable to the competition's.

Anonymous said...

Hi Falafelkid,
What I don't quite get is(bei all dem Fach-Chinesisch)..Why should Nintendo patent all this stuff when in fact Ati is in charge of the contract(developing Hollywood) that Nintendo handed over to them ?
Strange..

-Infamous-anonymous-light shedder.

leander said...

the last secret bla bla why should the last secret be some complicated rendering technique no-one would be excited about that.
My theory it is something mire tangible like a micro... and online chat or phoning (for free of course)
now that would be cool huh?

Anonymous said...

Exactly. The secret isn't some rendering technique. I mean come on. That's the one thing I do know for sure.

I like the guy in the last article comments section who said the 'A' button is deceptively simple and overly large. I think his idea that it has multiple functionality is as good a guess as any.

It seems to fit right into what Nintendo wants to do. NOt scare off new gamers, but provide functionality for the hardcore. Kind of like Apple with their new Mighty Mouse.

Again it's probably the best guess I've heard yet.

Anonymous said...

Hey! I've been following your blog for quite a while, it's really the best rev site I've found.
Anyway, I noticed how your last three posts have been about mapping techniques which make games look better. But I think we've established that Nintendo's not interested in making games look shinier but basicly to suit up to the on going generation.

I think lookin at their tech specs is good but knowing Nintendo their secret is bound to have something that goes a little more beyond the screen of the TV. Maybe you should try getting somethin a lil more juicy. Even if it's just speculation, exercising the imagination is one of Nintendo's principles so maybe that's what they're tryin to go for?

Still, it's pretty interesting how you brought up those three mapping styles.. they might even use all three at the same time and use that other digital space patent thingy that makes graphics stick out of the screen. That'd definitely make somethin great

Anonymous said...

no way. the last secret is never parallax mapping. the game "serious sam 2 already supports that technique and it really dosent look much better when you turn that on... in fact it just slows the game down a LOT on an x8000gto graphics card.

Anonymous said...

the last big secret to the rev will be somthing that will make sence to the average person that nintendo is targeting. If nintendo anounced a new way of mapping people all around the world will unite in thinking...

so?

Anonymous said...

The point that a majority of you doesn't seem to get, is that it's not equivocal that Nintendo won't simply announce that the secret is supposed to be Parallax- Cube- or Displacement-mapping verbatim


- these techniques merely represent different components, that to an extent compose what some developers reffer to -qouting the recent GamesIndustry.biz article- as a "a black box" and "nobody's seen the hardware yet" in other words the Hollywood GPU of course.


I keenly presume that Nintendo will debunk all the rumors by flaunty displaying the last secret that is Hollywood, with all patented inventions we all wittnessed lately included.

I guess he will make it clear to everyone by presnting the whole package respectively the games,a quick summary or contemplation of the philosophy,concept and intentions behind the GPU,the interaction of it with Broadway and probably what kind of benefits this approach means for developers.

"While Broadway is well-understood by developers, the ATI part remains "a bit of a black box", according to one senior developer we spoke to. "We have theoretical throughput figures and stats from Nintendo, but nobody's seen the hardware yet - we're just treating it like it's a faster version of the GameCube GPU, at the moment."

- The-Infamous-Anonymous-Light Shedder

Anonymous said...

Very intresting ideed.

My only qualm is using wikopedia as a source. Wikopedia can't be relied upon what-so-ever.

Infact I believe they are finally getting sued do to the fact that they dony have enough staff to research every entry.

Anonymous said...

Prallax mapping is not the last secret. The technique is nothing new and nothing special. Unreal Engine 3 uses Parallax mapping. It's been done.

Anonymous said...

Pekka:

Sorry I wasn't around to answer your question, but yes, "Parrallel Occlusion Mapping" was the correct term.

The Relief mapping seems very similar in effect from the videos, but I haven't the time to read the papers just yet. Maybe next week ;-)

However, both of those techniques subtract the bump and silouhette depth details from the geometry. Nintendo's patents specifically add the depth details to the geometry.

Now, with the 2003 patent entitled "Z Texturing" (these other things are patent applications, not real patents), Nintendo has a texture/depth buffer-based method to supply real per-pixel deformation of a model (including silouhette and occlusion by the deformations). I think the end results will be comparable to the relief texturing videos to which you've linked.

"From the patents shown here, Nintendo's putting many eggs into the texture basket. Could this be their trump card?"

YES!

I'm disregarding all notions of "final secret" and such, but YES!, this is a trump card Nintendo's waiting to reveal.

They have ~10 patents related to using the depth buffer (a.k.a., z-buffer) and textures to efficiently impliment bump mapping , per-pixel z texturing (i.e., a poor-man's displacemnt mapping, and self-shadowin provided by the bump-mapping), per light full-scene lighting, and some other effects.

The secret is much like the Gamecube: put low-latency, pretty-high bandwidth, cheap memory on the GPU die. Also, the patent linked by this article is for a Texture coordinate stage in the GPU that gets shared by all texturing and pixel processing that generates the appropriate coordinates in the color, bump, shadow, and "z-texturing" maps and hands them to the pixel engines.

So, Nintendo graphics WILL, beyond any doubt, look very similar to what we're seening of the xbox360 and PS3. They're just done using textures and it should be *very* easy for developers to generate the textures (which they already do with the other systems and PC today) and apply them (API calls insted of shader programs).

Anonymous said...

Crud, I keep forgetting to add my tag.

The last comment at 11:54 a.m. is mine.

--wellsed

Anonymous said...

light-shedder and jonny b:

"Why should Nintendo patent all this stuff when in fact Ati is in charge of the contract(developing Hollywood) that Nintendo hand over to them ?"

The patent applications are from 2000, when ArtX was finishing the development of the Gamecube Flipper GPU. I believe most of this technology is not in the Flipper, but the patent applications preceed ATI's purchase of ArtX and Nintendo's current contract with ATI.

The likelyhood of the deal is that ATI is providing Nintendo with the desired chips and gets some licensing rights to the patents.

There are around a dozen patents from this time period that all interlock. This patent, discussed by this article, is for a hardware feature that makes using texture-based effects more efficient on low-power, low-cost (read, Revolution) hardware.

--wellsed

Anonymous said...

?


Like great Post Production houses. Nintendo, desire to make and or bring most of its own console techology to the fullest. And like Industrial Light & Magic and the Soho based Glasswork who did not wait for Aliaswavefront's next Maya versions to knock on their door. No, they developed their own 3d sofwares. Hired great talented software programmers ect. This is what Nintendo is doing. Developing its own unique R&D, in this case, 3d graphic advancement.

Its just an evelution that some god company need to become great!

PS: guys, Falafelkid has explain over a couples of times that this texture mapping stuff was not "the" secret Nintendp has to offer, but that allong with it they might be a possible team of "other" tech to be reveald as "the" Secret-s

thx

/madox

Anonymous said...

arsenis:

Thanks, I didn't realize that.

Anywho, it appears a lot of these patents *ARE* actually Gamecube related.

The document:

http://cubemedia.ign.com/media/
news/documents/
embeddedforumkeynote.pdf

shows the TEV and texture coordinate parts of Flipper on page 30. These were the important parts related by the patent linked from the article.

Anonymous said...

Kojima Discusses Game Ideas, Revolution's Appeal
by Shawn White (12/15/05)


In a recent interview conducted by Japenese publication Famitsu, Metal Gear Creator Hideo Kojima discussed a few ideas concerning his in-development Revolution game as well as the console's appeal to a wider demographic.

During the interview, Kojima expressed his intent on producing games that can be considered stylish and artistic in nature; a goal he considers best accomplished on the Nintendo Revolution.

Kojima mentioned that he wants to make games outside of his multi-million dollar franchise Metal Gear and that with Revolution, he can create games "based on life, [or] something organic." He did not further elaborate on his comment.

On a final note, Kojima pointed out that some aspect of the Revolution will make it "popular with girls" without providing explicit details.

Revolution Report will have more enigmatic statements from Hideo Kojima as they become available.

Anonymous said...

Ha...people that revealing the revolution guy is back.
He has no new posts but his site isn't forzen anymore

Anonymous said...

yea we no hes back woo... er no

Anonymous said...

Yeah that might be special, if not the fact that several Xbox 360 games that are already out such as Kameo, Project Gotham Racing 3, and PDZ already used it and several PS3, and Xbox 360 games use the mapping technique.

Seriously did you really think it was some sort of secret or are you just that bored dude?

Anonymous said...

nintendo has been saying all along that it has new techniques to reduce development costs and production times. I dont think that that process will be the last secret. Remember the secret of the controller was a very big change. I think the secret is something else all together different. It probably has to do with the way we view the video game. Maybe 3d projection or somehting like that would be cool.