NVIDIA RTX Remix Open Beta Q&A with Nyle Usmani – Past, Present and Future of the Generalized Modding Platform

Today, almost a year and a half after the modding platform's announcement, NVIDIA releases the highly anticipated RTX Remix creator toolkit Open Beta. While the runtime has been available for some time, the public availability of the creator toolkit should really kickstart the modders' work to easily remaster classic PC games with cutting-edge rendering features like path tracing, DLSS 3, and AI-upscaled assets. NVIDIA is also partnering with ModDB to make it easier for users to check out which games are compatible with RTX Remix and to get the optimized configuration files to run the mods.

There's a series of improvements to the new version of the runtime as well, such as Parallax Occlusion Mapping (POM) being supported to make 2D textures look tridimensional and Subsurface Scattering (SSS) now working on foliage.

Last week, I had the pleasure of talking for almost an hour with NVIDIA's RTX Remix Product Manager Nyle Usmani about the past, present, and future of this ambitious generalized modding platform, including the thorny subject of potential paid mods made with the tool.

Nyle, can you talk about how the RTX Remix project got started to begin with?

Yeah, that's an interesting one. When Lightspeed Studios started getting into experimenting with classic games and seeing where we could bring path tracing into content ahead of the curve, ahead of when most gaming companies would have been able to implement it, it came from us looking backward and seeing how impressive we could revitalize classic content that had more simple geometry where you could drop in extremely futuristic cutting edge lighting systems and still have it run in real time.

At the start of the RTX generation, we were looking at classic content. You're probably familiar with Quake II RTX. We also did Minecraft RTX and then the person who led that project started experimenting with this idea of, well, we've hooked lighting systems into these classic games with some source code, but there's maybe an avenue here to do a generalizable solution. We started looking into what that could look like. We're masters of hardware; we're masters of GPUs. We understand rendering pipelines and structures. It was a natural marriage for us to start to understand how we could intercept draw calls, the rendering calls that were being sent from the GPU, and create a generalizable path tracing solution for old content. In doing that, we discovered all these other things.

Well, if we can intercept the draw calls and we need to identify objects temporally between every frame so that you can update lights realistically, then why can't we just swap out assets on the fly? If we have to identify every object frame by frame, create a lighting system that can update, then surely that also means we could substitute what you see on screen with something else, and so then that quickly changed from how do we get lighting into classic games into how do we remaster classic games. All of that became the basis and the foundation that would eventually become RTX Remix, which at the time we used to call T-Rex, which is a funny sort of code name thing.

We came up with that generalizable solution and then we wanted to prove that it worked. It started out as a Portal tech demo. Internally, the first proof point was Portal and then it naturally turned into a relationship with Valve in a collaboration for Portal with RTX.

That's also around when people started using AI to improve assets.

Yeah. The AI texture tool is interesting because NVIDIA has been in the AI game forever, right? We've been doing this since before I was here at NVIDIA. We knew our future would be tied to understanding how AI could elevate gaming as a whole, so we were interested in this idea of, well, there are a lot of modders out there. What can we do to accelerate them, and where is AI best plugged into the gaming pipeline? When you look at these classic games, you look at these 2D flat textures and you think about how 2D flat textures look in a path traced environment. They all kind of become the same material. They're either matte or they're glossy.

Why put a modern path tracer into a game if the materials are not going to react to the lighting? You kind of need to pair both, modern materials and assets with modern lighting, so I think the AI texture tool became a perfect fit since you have the combination of extremely muddy old textures that need updating and an incredible need to remaster all of them efficiently. To add to that, modders don't have a whole lot of time, right? A game studio, if they need more material artists or more texture artists, can create some headcount and staff up.

If a mod team wants more material artists and texture artists, they have to beg and borrow from all these other mod projects in the world. When we think about who would be helped the most, where are the biggest problems with textures? Both answers let us back to RTX Remix, back to classic games, and back to a modding platform. That's how it became part of our plans and I'm eager for you to try it.

The AI asset remastering tool is probably the biggest feature coming with the creator kit compared to the RTX Remix runtime, right?

Let's start with the runtime. Back in April, we open sourced the RTX Remix runtime. The runtime is what makes the game and the mod talk to each other. It basically is both the first thing that you do and the last thing you do with your mod. You start by dropping the runtime next to the game executable, and it essentially hooks into the game and updates the renderer from a legacy renderer to a modern Vulkan renderer. It path traces the game, adds DLSS, et cetera. But the most important thing the runtime is doing at the start of your modding process is it lets you hit a button to capture the entire game scene and capture the game assets and create a USD file that incorporates all that information, the lighting, the geometry, the assets, all of it.

We open sourced that in April for a lot of reasons, one of which is that we thought people would have fun starting to experience path traced mods even if they don't have a great way to remaster all the game assets. And then we also made it available early because the runtime is what affects compatibility, since the runtime is how the RTX Remix mod will talk to the game and hook into it. Improving the runtime means you can expand which games and which types of engines you're compatible with, and so by open sourcing it and making it widely available, we set out on this journey to start to get feedback, start to understand the broad scope of what compatibility looks like with the tool, and also encourage the community to collaborate with us to push the boundaries of what compatibility could be. That's what the runtime does.

What we never expected to happen was for people to actually start to add remastered assets into their mods with just the runtime. Our community is very resourceful. The modders found ways around that. RTX Remix is built on Omniverse, so a lot of the functionality of RTX Remix in the backend is similar to a lot of Omniverse applications. What we saw modders doing is they were using a combination of the Omniverse application USD composer alongside their understanding of the text files that the runtime would package and they were able to basically start clawing some of the feature set of Remix, kind of like with duct tape and gum do some of what Remix does, but with a bunch of workarounds, with a bunch of file manipulation, without any kind of easy or reasonable UX. They still managed to plug a bunch of remastered assets into their mods. Our community even started inventing their own AI texture tools. That's pretty cool, but the creator toolkit should make all that easy so you don't have to do a bunch of hacky workarounds to bring remastered assets into your game scene.

What we're releasing today is the toolkit. With that, you can bring in your capture from the game and it'll render beautifully in the viewport and you can very easily just click on any asset and upload your own replaced asset and swap them in.

Basically, the new thing is this whole UI, this whole workflow of being able to do everything seamlessly and easily. A Discord user would have to take weeks. This is what we've seen in our early access. The modders, the first thing they say when they get the toolkit is wow, it used to take me a week to do this and now I can do it in 10 minutes. That's what the toolkit is about.

It's about making that process of clicking on an asset and replacing the asset with a PBR beautiful high fidelity asset that has parallax occlusion mapping and height maps and emissive masks, normal roughness, metallic maps, whatever. You can now do that very easily. The other thing that lets you do is it also lets you create new lights. You can create cone lights, spotlights, point lights, and cylindrical lights. You have a great variety of ways to relight and reauthor the scene lighting, which is something that most of the modders are not able to do easily.

You can even parent a light onto an object. Do you remember the plasma balls that are flying around in Portal RTX? That's technically a mesh and you're parenting a light to it, so you can create these beautiful dynamic lights that make things look so cool with real time shadows and lighting. And then finally, like you noted, the AI texture tools. That's the last kind of new thing that the RTX Remix toolkit is gonna enable, which lets users increase their pixel counts by 4X. It's a super-resolution network and then it's also a generative AI network where it will create physically accurate materials on the fly.

It will analyze the texture and go, is that rock? Is that wood? Well, here's what the normal map should look like, and here's what the roughness map should look like, and now it'll reflect light properly like it was really rock or wood and all those cracks and crevices are gonna be more defined and will look 3D because it's using a normal map whereas the original was just flat.

I've got a couple of questions on that. Can you share if it's a proprietary algorithm from NVIDIA, or perhaps you are using some open source AI technique?

The AI model behind RTX Remix’s AI Texture Tools is our own proprietary model trained on our in-house dataset.

Got it. The other question came up a while ago in a conversation with the lead developer of Skyblivion, the upcoming total conversion mod. They need complex assets like clothing, architecture, and weapons. They were wondering whether RTX Remix would be able to handle that stuff.

So, you don't really build assets in RTX Remix outside of the AI texture tool. That gives you a way to qualitatively start out with asset A and then make it look better with asset B. But for most assets, RTX Remix is how you bring them into your game. It's not per se how you build the asset. When you think about the Half-Life 2 RTX HEV suit, the creators were using a workflow where you could produce a file format that could easily be brought into Remix. Then Remix was able to replace that old school asset with this beautiful high fidelity asset that had 34x more polygons compared to the original.

If you had tried to put that game into Source Engine, the game would have hit a 4GB VRAM limit and it would have just crashed. It wouldn't have been able to process these incredible high-fidelity PBR materials. It would have been too heavy, but RTX Remix is replacing that old renderer with a modern Vulkan renderer without those limitations, so you can put these modern-day assets into these old games.

So it's not an authoring platform, it's a way to plug assets easily into a variety of games and update the lighting and alter the lighting and create new lights and all that stuff. And you can replace characters as well. That can be a little more difficult. There's unskinned assets, which is like regular props, and then there's skinned assets, which would be things like the zombie in Half-Life 2 that have bones in a skeleton. You can even replace those assets that have skeletons and are skinned.

But currently, what we're releasing is not going to provide an easy way to do that. That takes a little bit more involvement and it's in our roadmap to try to find a way to generalize that and make it easy for folks to replace skinned assets. That is currently the only real massive asset limitation.  Of course, RTX Remix is gonna have a strange time with a variety of corner cases. You know, animated textures, textures that you shaders like, they're always gonna be some things that we have a hard time intercepting and replacing on the fly. But generally, I wouldn't say there's any kind of limitation.

He mentioned specifically architecture as something he would love to use if RTX Remix could do that. Could RTX Remix capture the texture of a building in Morrowind, for example?

Yeah. Let's take architecture. You have two approaches there. Actually, in our Morrowind showcase, you can see both. One is the AI texture tool, right? We have that beautiful pan of the ceiling. The ceiling of more wind and you've got the line in the middle and everything that's passing through it is getting AI upscaled and getting normal maps and roughness maps from AI.

So yeah, it can absolutely do a pass in architecture. Or you can do what we did for the actual showcase. All the walls, the stairs, the ground, all of those were high-quality textures that we did by hand and put into the scene, the tables, all that stuff we did by hand.

One of the official RTX Remix projects, Portal: Prelude RTX, also features RTX IO support. Is that going to be a part of the RTX Remix feature set?

Great question. RTX IO will not be there at the launch of the open beta, but it will absolutely be a part of the RTX Remix toolkit. It is on our product roadmap. It's just a matter of getting to it when we get to it and then every model will be able to experience the benefits. For Portal Prelude, it did something like reduce loading times by 5X and reduce the storage space of the mod itself by like I think it was 40 percent. You could essentially have 30 gigabytes of high-quality textures, but for the user, it only takes up 18 gigabytes. It's pretty valuable for modders who want to get lossless quality assets into their mods and still stay within the budget of these mod sites.

Speaking of possible future additions, are you going to add DLSS 3.5 (Ray Reconstruction)?

Ray Reconstruction is an interesting topic. We have nothing to announce with Ray Reconstruction and RTX Remix toolkit today, but I will say that Ray Reconstruction is in Half-Life 2 RTX.

RTX Remix modders are always asking about the potential for further compatibility, especially for later DirectX APIs or even OpenGL. Any news to share on this?

The community is always experimenting with compatibility. As I mentioned earlier, it's kind of in part why we open sourced the RTX Remix runtime, to give everyone a chance to try it on their favorite game, rather than us having to guess around and assume that we know everything.

Why assume you know everything when you can trust modders? They know better and are way more willing to try things than most people. We launched the runtime in April, and I think by now the Discord channel has found possibly hundreds of games that work with RTX Remix. Now, compatibility is sort of like a spectrum, right? This is a generalizable solution that interprets graphical draw calls from many different eras and tries to recompile a coherent image afterward and then identify every asset and then look at what the modder has assigned as a replacement for each asset and then replace it live.

It is bound to fail some of the time as it's not designed for a specific subset of games. As a result, some games you can mod end to end. Everything works amazing, every asset changes. Most of those are DirectX 8 and 9 games with fixed-function pipelines. Some games that were no longer using fixed function pipelines, that were using more shader based models are going to have more issues with properly hooking into RTX Remix. That's because with shaders we can't tell what the game is doing until it's drawn on screen. That means we can't intercept it and replace any of those assets or replace any of those lights when shaders are being used. At least complex shaders.

To bring it back to compatibility, we were impressed with what the community was doing. In some cases, they were finding out things that just work. In some cases, like in Dark Messiah of Might and Magic, they were literally reverse engineering the game. In that particular case, that modder reached out to Ubisoft and said, hey, we wanna reverse engineer your game. Would you be willing to let us do that? They are doing some impressive things to make the game fully compatible with RTX Remix. Some folks were taking traditionally shader based games and converting their graphical pipelines to fixed function pipelines.

So you're getting a range of solutions. Some of it's on the content, some of it is us working on the RTX Remix runtime and making it adapt to more and more game types and scenarios and some of it is people who are in the middle who are creating wrappers that help connect the content to the tool better. The results have been impressive. I mean, I didn't think I'd ever see Skyrim work with RTX Remix, and yet you can.

It's open source. On our GitHub, people can post any issues they're seeing with any particular game, or they can help contribute code or solutions. And then, starting today, we have a partnership with ModDB that we're launching that's going to vastly impact compatibility.ModDB is hosting the, I don't wanna say Bible, but like the source of truth for the whole Internet with RTX Remix. They're hosting a compatibility table that users can update or contribute to that will track which games are compatible and which runtimes work best for each particular game.

With RTX Remix, when you're trying to make it work, you have to tweak some settings and create a config file that's specific to the rendering techniques that are unique to that game. ModDB will be hosting those config files for every game, so theoretically, a user could take a game, go to ModDB, click on and go. Is this compatible? Read instructions on what unique things you have to do in this game to make it work, download the config file, put it all together in one folder and it should just work.

We're pretty excited about the future of compatibility, working with the community to expand it, to make it easier to discover and understand on ModDB and ourselves working on the runtime to make it compatible with more and more content. As you probably saw in our product roadmap, we're also looking at OpenGL.

I guess there is no ETA for OpenGL that you can share right now.

I have nothing to share today, unfortunately.

Okay. We actually did an email interview a few months ago where I asked a couple of questions about a potential scripting language being added or ways to improve particles. You said they could be interesting to add at some point. I was wondering if you had any kind of updates on this today.

I don't have too many updates on those topics. I will say that we're learning more and more. Particles are so tricky. They are very unique in every engine. I will say that the thing that helps us understand particles the most is some of the collaboration we're doing with Orbifold Studios on Half-Life 2 RTX because we're working with Source Engine masters there, including the guy who is probably the most well-known Source Engine particle artist in the entire world. He's done things like redoing how explosions work in Half-Life 2 RTX so that they create beautiful plumes of smoke where the light can penetrate through it and illuminate it with the color of the light. That's the combination of path tracing and particles. All I'll say is there's a lot of interesting things to do with path tracing and particles and we're always learning from these close collaborations on what things modders need most on this stuff. While I have nothing to announce, I will say I'm intrigued.

Could there be some form of module or plugin system for RTX Remix so modders can, for example, add post-process effects or stuff like that?

It's an interesting question, right? On the topic of plugins, every Omniverse application has a workflow for adding extensions to it, which extends the capabilities of each application. Because RTX Remix is an Omniverse application, you will be downloading the Omniverse launcher and you'll be launching it from the Omniverse launcher. You'll also be updating RTX Remix through the Omniverse launcher, but the Omniverse launcher is also a stage to discover these extensions that can instead extend the capabilities of any Omniverse app, and there was a rich ecosystem of open contributions of adding community extensions for all kinds of Omniverse apps. There is a way for people to extend the capability of the toolkit if they're resourceful, and we imagine people will.

Post-processing is an RTX Remix runtime feature. What I would suggest is that people who want to share new filters, for example, please get into the GitHub and create a contribution or a merge request. Send it our way. If we review it and it looks good, your addition to Remix can make it into the final product for everyone to try. Or alternatively, you're welcome to make a fork of the open source RTX Remix runtime and experiment and create your own version with all kinds of other features. But of course, we always love to see any contributions to Remix.

On this topic, I think maybe ReShade could help as well. The new 6.0 version added support for RTX Remix.

I saw that. It made me very happy. That was really cool.

One question I've got is on the topic of legality and potentially paid mods. The topic was recently resurrected by Bethesda since they reintroduced paid mods to Skyrim. It sparked a big debate in the modding community. Some modders decided to get Verified and enter the new Creations system, while others are against it. My question is, what would you do if someone used RTX Remix to make a paid mod? Would you be OK with it? What's your stance?

NVIDIA's stance is that we created these tools for modders to express themselves. We'll be updating our legal license so you can review the formal legal language, but it essentially translates to anything the modders create is up to the ownership of the modders themselves, not NVIDIA.

With that said, they do have to abide by any EULAs or licenses for any of the content they're using as part of their mods. Whether that's assets or content that's being used as a foundation that they're getting captures from or that they're modding, they have to abide by the legal licenses and terms, as those sort of trump ours. But when it comes to the question of can I make this type of mod, ultimately our position is that the mods the modders make, assuming they have approval from the license holders of the content, from our perspective, that's your mod. You can do whatever whatever you want. Just make sure you're abiding by all the rules of the content and all the rules of whatever site you're uploading it to.

Your position is that you are giving modders the tool, and they can do whatever they want with it as long as their work doesn't impact the copyright holder.

Right. You're operating on the risks associated with all the content that is being used in the process of making your mod, but the toolkit itself, we're not going to come at you for using the tool.

Something else that came up when I talked with some members of the Discord channel is that less experienced users could potentially cause legal trouble for themselves if they shipped copyrighted assets in their mod. I guess they have to be careful about that.

Anyone utilizing content from other IP holders or uploading mods on specific mod sites should definitely make sure that they understand the legal terms that they're agreeing to by using that content. I've done a lot of reading on modding EULAs and modding licenses, and I can tell you that companies very often publish very clear terms on this stuff. I encourage people to read them.

Some people are wondering if, at some point, RTX Remix will support non-NVIDIA upscalers.

We have out-of-the-box included upscalers for users who do not have NVIDIA GPUs to make it possible to run path traced experiences on other hardware. The only limitation is that upscaling is handled by the runtime, which is open source on GitHub. Essentially, any person can create a fork or contribute to the RTX Remix runtime code, building support for other upscalers. No one has done that up to now, which is why it's not there today. But we provided an avenue for people to extend the feature set of the Remix runtime, including with upscalers.

Do you think the release of this RTX Remix creator toolkit will attract a lot of brand new people to modding?

Oh boy, I hope so. I think so, and I think it has the capability of changing how modding is done everywhere. We've seen beautiful mods before. We've seen beautiful ray traced mods before without RTX Remix. There are ways of making high fidelity mods for a number of games. But I want you to think about what it's like to be in the shoes of a modder. Let's say you like a Bethesda game, for example, like Skyrim, and you want to mod that. Okay, you've modded Skyrim. What game do you want to mod next?

Well, you spent all your time figuring out all of the tools that are used to mod Skyrim. The only other games those tools can really work on might be another Bethesda game because the tools were made, you know, by Bethesda for their games. You're sort of locked into this small community based on what the tools can actually affect.

Even getting beyond the fact that there are a ton of classic games that were not easy to mod with, with amazing graphics. There's some classic games that have never been modded before because they weren't tools for it. And then there are a bunch of games that could be modded before, like Source Engine games, where if you knew the tool, you were locked into the ecosystem of content based on that engine.

With RTX Remix, you have a paradigm shift. If you have some know-how you can make a competent, beautiful, modern-day-looking remaster with one of the best lighting models in gaming history, with high-end PBR materials and assets where the only limitation is your artistic skill. And you can do it for a large, wide swath of games that spans a decade and a half or something. That's really powerful because suddenly, if I'm looking around and I wanna learn how to mod and I decide I want to be an RTX Remix modder, I can go from Portal to Half-Life to Morrowind, and that's impressive. And there's so many more.

I encourage people to look at ModDB on launch day and take a glance at how many games can be modded with RTX Remix because it's an incredible list and we've seen some incredible things in the Discord. Tons of games that I never knew about that can now be modded by this tool.

We've also got plenty of documentation and tutorial videos for those who want to get started.

Thank you for your time.