Author Topic: rerendering gameplay recording at higher resolution  (Read 2193 times)

0 Members and 1 Guest are viewing this topic.

Online NiHaoMikeTopic starter

  • Super Contributor
  • ***
  • Posts: 9330
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
rerendering gameplay recording at higher resolution
« on: December 21, 2024, 04:56:41 am »
How feasible would it be to record a gameplay (that is being rendered at a fairly low resolution like 1080p) in such a way that later on, a 4K or 8K render can be made from that recording and the game data? The purpose would be to create a gameplay video of very high resolution at maximum detail, that might not even be possible to do in real time with available technology.

I see that a few games could do that, does it require explicit support within the game engine or could there be a way to easily mod it into other games?
https://www.reddit.com/r/letsplay/comments/15gbxez/q_do_games_that_can_rerender_recorded_sequences/
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 

Offline Whales

  • Super Contributor
  • ***
  • Posts: 2143
  • Country: au
    • Halestrom
Re: rerendering gameplay recording at higher resolution
« Reply #1 on: December 21, 2024, 05:08:29 am »
Fundamentally this means you must NOT record 2D video.  You cannot upscale an image to a higher resolution without having to guess as its contents (like AI upscaling algorithms do).

Some games let you make "demo recordings" (in particular quake-derived game engines, including many Source-based games) or network data recordings, but this is far from universal and has many bugs.  For some older games (like Total Annihilation) the community has created their own game servers for the purpose of replaying matches (so you connect your client and just watch the game play out), but again far from universal nor bugless.

A "universal" solution is to use a tool that records the 3D geometry data and shaders sent to your graphics card.  You can replay these later, with hand-made changes (such as changing your resolution; which interestingly is not always trivial to do).  Unfortunately this is extremely disk space intensive and the tools are very immature.  I did a little bit of a writeup a while ago about these on my blog https://halestrom.net/darksleep/blog/044_3dto3d/

Online NiHaoMikeTopic starter

  • Super Contributor
  • ***
  • Posts: 9330
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
Re: rerendering gameplay recording at higher resolution
« Reply #2 on: December 21, 2024, 01:13:01 pm »
Fundamentally this means you must NOT record 2D video.  You cannot upscale an image to a higher resolution without having to guess as its contents (like AI upscaling algorithms do).

Some games let you make "demo recordings" (in particular quake-derived game engines, including many Source-based games) or network data recordings, but this is far from universal and has many bugs.  For some older games (like Total Annihilation) the community has created their own game servers for the purpose of replaying matches (so you connect your client and just watch the game play out), but again far from universal nor bugless.

A "universal" solution is to use a tool that records the 3D geometry data and shaders sent to your graphics card.  You can replay these later, with hand-made changes (such as changing your resolution; which interestingly is not always trivial to do).  Unfortunately this is extremely disk space intensive and the tools are very immature.  I did a little bit of a writeup a while ago about these on my blog https://halestrom.net/darksleep/blog/044_3dto3d/
I'm thinking the 2D capture can be a "ground truth", to easily allow checking that the 3D capture and replay didn't glitch out.

Even if the 3D captures end up taking a few GB per minute, that's not a big deal with modern hardware.
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 

Offline Whales

  • Super Contributor
  • ***
  • Posts: 2143
  • Country: au
    • Halestrom
Re: rerendering gameplay recording at higher resolution
« Reply #3 on: December 21, 2024, 10:27:10 pm »
Many modern games saturate the PCIE link going to the graphics card with all the rendering data they're sending, so we're talking gigabits per second of recording.  Possibly gigabytes per second.

Some NVME SSDs might be able to sustain this for a while, but if you drop any data at any point then the recordings will at a minimum show glitches (missing objects, lots of flashing/flickering, missing textures), and we're still talking minutes of recordings before you run out of space.

Older and less graphically intensive games are easier to record (which is why I chose half-life 1 for my tests). 

Online NiHaoMikeTopic starter

  • Super Contributor
  • ***
  • Posts: 9330
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
Re: rerendering gameplay recording at higher resolution
« Reply #4 on: December 22, 2024, 12:48:19 am »
Might it be possible to use a second PC (connected with 40G or 100G Ethernet) or a FPGA board to perform compression in real time?
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 

Online Marco

  • Super Contributor
  • ***
  • Posts: 7085
  • Country: nl
Re: rerendering gameplay recording at higher resolution
« Reply #5 on: December 22, 2024, 01:10:48 am »
But why do that instead of making a deterministic game engine which can just replay inputs?
 

Offline golden_labels

  • Super Contributor
  • ***
  • Posts: 1507
  • Country: pl
Re: rerendering gameplay recording at higher resolution
« Reply #6 on: December 22, 2024, 01:50:51 am »
Matching geometry isn’t a problem. Geometry is cheap. You can render geometry directly to 8K. Lights, normal mapping, parallax mapping, and materials is where the cost is. This can’t be validated by cheap comparison with raw data. It has to be calculated.

Currently the best approach for getting high-resolution image from a low-resolution render seems to be: rendering a collection (not grid!) of points to support correct geometry, visual features, and light information, followed by generating missing information with probabilistic means. Basically a kind of inpainting between very sparse points. The final step doesn’t even require game data or logic to be known, which is probably why Nvidia develops it: that ensures the game remains under absolute control of the publisher.

But I agree with Marco: a deterministic game engine should be able to provide a high quality render from inputs alone. It also stores data efficiently. Why do you insist on upscaling low-resolution renderings?
People imagine AI as T1000. What we got so far is glorified T9.
 

Offline garrettm

  • Frequent Contributor
  • **
  • Posts: 342
  • Country: us
Re: rerendering gameplay recording at higher resolution
« Reply #7 on: December 22, 2024, 01:51:29 am »
Well, if the game is 2D and fairly old (e.g., Diablo I (2D) or even Total Annihilation (2D+3D)), then Nearest Neighbor integer scaling is _lossless_ upscaling. A lot of older games are "pixely" and scale beautifully. But the problem is that all video formats I know of convert each frame into a _lossy_ format like jpeg which cannot be upscaled losslessly, even for 2D games.

So, ultimately, there is no good answer for you other than just re-record the game played at a higher resolution.

Many old games have mods and patches that allow for modern 16:9 display resolutions (as opposed to period accurate 4:3), such as Diablo II. And many do internal upscaling of 3D elements, so they actually render at that resolution and look surprisingly good. A good example is Unreal Tournament 1999, which as of patch 469d (from 2023) has native 4k support and looks pretty damn good for a 25 year-old game. There is also PCSX2, a PS2 emulator, that does similar internal upscaling of 3D elements and makes PS2 games look better than they ever could on the original console.
 

Offline garrettm

  • Frequent Contributor
  • **
  • Posts: 342
  • Country: us
Re: rerendering gameplay recording at higher resolution
« Reply #8 on: December 22, 2024, 02:02:31 am »
Rather than focus on how to make a video scale better (which is highly depended on the codec, game engine, and up scaling method), I'd focus on what "wrappers", mods and patches are available to _improve_ the game, if needed, and then re-record at each resolution using a macro or some other automation for the game play (so each video is as identical in execution as possible). Games are a moving target, and even very old games, if still popular/fun, get occasional love to keep them working. So showing new videos of what is currently available is more interesting to viewers than what was possible previously (except for comparisons, of course). (A good example is Might and Magic 6, which was pure software 3D, and only with the MM6/7/8 merge mod can play with hardware 3D at modern 16:9 4K resolutions--which made me replay it recently for nostalgia and to see what it could have been if not limited by technology of the time.)
« Last Edit: December 22, 2024, 02:05:44 am by garrettm »
 

Online NiHaoMikeTopic starter

  • Super Contributor
  • ***
  • Posts: 9330
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
Re: rerendering gameplay recording at higher resolution
« Reply #9 on: December 22, 2024, 01:03:04 pm »
The main interest is newer games since those are the ones that are difficult to render at high resolutions and high FPS. One of my friends is into posting gameplay videos and would like a way to set herself apart from all the others who do the same.
But why do that instead of making a deterministic game engine which can just replay inputs?
I read somewhere that a lot of newer game engines are not deterministic.
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 

Offline Whales

  • Super Contributor
  • ***
  • Posts: 2143
  • Country: au
    • Halestrom
Re: rerendering gameplay recording at higher resolution
« Reply #10 on: December 23, 2024, 12:19:48 am »
But why do that instead of making a deterministic game engine which can just replay inputs?

We don't make the million dollar games, we just play them :) 

Determinism conflicts heavily with the goal of performance.  If you design a game to be single threaded and block execution until resources are available then you can (emphasis on *can*) get perfect determinism, but the game will stutter and perform poorly.  Instead many modern engines load resources lazily in other threads and keep rendering the game in the interim, with objects either missing or temporarily stationary.

Quote
and then re-record at each resolution using a macro or some other automation for the game play (so each video is as identical in execution as possible).

If by "macro" you mean a recording of player inputs on their keyboard and mouse?

If you try and replay inputs on a game then you will get wildly divergent things happen each time.  In the TAS communities this is called "desync" and commonly looks like the player suddenly wanting to walk into walls and trees.  Even if you ignore the unintentional nondeterminism of games, your OS scheduling and your computer's performance: most games use random number generators for parts of their gameplay.

For this to work either the game engine needs to be specifically designed for it ("demo recording" often means "input recording") OR you need to run the game in an emulator (which can provide the deterministic game execution, RNG, timing and input synchronisation for you).  Half-life 2 is an example of the former, TAS speedruns of nintendo64 games are an example of the latter.
« Last Edit: December 23, 2024, 12:23:02 am by Whales »
 

Online Marco

  • Super Contributor
  • ***
  • Posts: 7085
  • Country: nl
Re: rerendering gameplay recording at higher resolution
« Reply #11 on: December 23, 2024, 02:01:42 am »
Determinism conflicts heavily with the goal of performance.

The major physics engines tend to be deterministic for client side prediction purposes ... that the game devs then don't manage to stretch that determinism past a few 100 ms when their code interferes is more sloppiness than necessity.
« Last Edit: December 23, 2024, 02:03:31 am by Marco »
 

Online NiHaoMikeTopic starter

  • Super Contributor
  • ***
  • Posts: 9330
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
Re: rerendering gameplay recording at higher resolution
« Reply #12 on: December 24, 2024, 04:58:50 am »
If capturing the GPU commands is done, does the playback GPU have to be the same architecture as the GPU the capture was done on? The Intel Arc GPUs are looking quite attractive for AV1 encoding.

Are there any open source projects about capturing the GPU commands? At the least, it would be good to get an idea of how much data is involved and how compressible it is.
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf