- Sony just patented a new reconstruction method using deep learning
-
From LinkedIn with somebody associated with the patent
Most recently I've spearheaded work in using Neural Rendering to enhance traditional rendering methods, focusing on using implicit neural representations and how to make them run efficiently. This includes creating custom high performance inference using compute shaders. The target for this work is PlayStation 5. Development done in Python, Unity, C#, compute shaders with offline training using Pytorch.
ID: hjevi85ID: hjh2fihYou need hardware acceleration for that type of upscaling though and it's typically too intensive just to do on regular shaders. That's why Nvidia has custom tensor cores on all their RTX 20 and 30 chips to implement DLSS.
If this was ever going to come to light my guess is it would be on a PS5 Pro or PS6.
ID: hjfnm1y[deleted]
ID: hjf19gjHopefully for PS5. This might be in the line for PS6 or the PlayStation PCs.
ID: hjevfw8From reading through the patent, it also has good implications on the next version of the PSVR, including having it capable of doing some computing in filling image gaps caused by bandwidth issues.
ID: hjhxrs9Sounds like an exciting prospect!
ID: hjew8bnCan someone Eli5?
ID: hjezzc0A version of DLSS (Deep Learning Super Sampling) for PS5. It’s an image reconstruction technique that can take lower resolution textures and blow them up to high resolutions without compromising fidelity. Basically, the PS5 could render an image at say 720p but use this technique to display it as 4K and have it look pin sharp. That’s a very high level explanation.
ID: hjfvhz4A technique to make the image look shaper with very little impact in performance.
ID: hjhjqmkIm a student in IT and every time i hear about projects like these, i get major imposter syndrome. knowing that someone out there is capable of working on a project as complex as this and im supposed to one day work along side people like this gives me the heebie jeebies.
Edit: deep learning is computer science on crack.
ID: hjipgm9deep learning is computer science on crack.
The people who write libraries like pytorch are impressive. Us commoners who train a premade model don't need much experience. It's mostly just fucking around and observing the results
ID: hjjj9orImposter syndrome is rampant in the industry and nothing fuels it quite like a white board interview or a hacker rank prescreen.
ID: hji0j4oThe patent talks about inpainting and your quote talks about neural rendering. This uses machine learning, but otherwise it doesn't seem all that similar to DLSS to me? If anything, this seems more similar to Nvidia Canvas / GauGAN to me, but with a more specific use case for filling "holes" in an image (rather than an entire image) along with a way to use nearby information for similar materials.
I'll be honest, I don't fully understand the use case for this based on the patent application. They talk about bandwidth-constrained (VR?) scenarios and when something goes from being occluded to being visible and you have a hole, I think. Would this be for scenarios where your head is moving around in VR at a rate where your traditionally-rendered framerate can't keep up? Rather than needing a completely new frame, you can basically apply the style/material information from other parts of the image to fill in a guess of what those areas probably look like?
EDIT: I forgot about the concept of "timewarp" / reprojection used in VR. If I'm understanding this patent correctly, I think this would add concepts of neural rendering / GAN's into the mix, which seems really interesting for creating high-quality, high-framerate, high-resolution VR experiences. (I think)
ID: hjfsr4uThat sounds exactly like the processor on their Bravia TVs that supposedly enhances the image in ways a human would emphasize.
ID: hjeumhrLook at the big brain on this guy
ID: hjew6no*Brad
ID: hjfgn53Development done in Python, Unity, C#, compute shaders with offline training using Pytorch
Sounds like some first proof-of-concept than some serious work ready to be deployed. C# is slow for these things, in general it is C/assembly instead.
ID: hjfgcf8Python and C#? Does DLSS use Python? I can see generating code with Python but it doesn’t perform very well where performance is key.
I’m familiar with embedded swe so I don’t know much about game dev.
ID: hjfuf8qPython is the most common language used for training deep learning models, which is part of the implementation of the algorithm. It's not being used at runtime.
-
I’m dumb. Does this mean more frames or more pixels?
ID: hjfm5mnYou use less pixels to achieve a sharper image, thus giving your GPU more time do compute more frames.
This is how DLSS works on PC, basically. Instead of rendering 3840x2160 (4K) you render (let's say) 2560x1440p but your magic image reconstruction program figures out the "missing pieces" by taking information from the previous frame, the current frame, grabbing some motion vectors (this pixel was going in this direction, so it's probably going to continue doing so in the next frame, probably?) applying some magic programming sauce to it and the algorithm spits out a 4K image from that information. And since your GPU is now only rendering 1440p (+ magic sauce upscaling), that means you have more GPU oomph to get higher frame rates.
So while outputting, say, 4K60 in ShootGuy IV at max graphics settings may be asking too much (maybe you only get 4K at ~40-45 fps), you can apply the magic sauce upscaler while only technically rendering a 1440p image and now you can output an image that looks very close to 4K and hit 60 fps no problem.
ID: hjfw3xaI’m glad they’re working on a technology such as this one. ShootGuy III had some terrible problems maintaining a stable framerate. Great game, though.
ID: hjftws9Dang that’s a great ELI5. Thanks for that info!
ID: hjf2p43First use less pixels to get more frames. Then magic adds more pixels.
ID: hjex06qLess missing pixels and frames.
ID: hjf2dse*fewer
ID: hjfv747Less pixel more frame then AI vomits in more pixel
ID: hjid3hfThink of this like painting. The first brush is the native render. The second pass of the brush that fills in the gaps is the machine learning algorithm being used here. So the PS5 does the first stroke at a lower pixel count very quickly then fills in the gaps using the algorithm at the same time. So in the end, it will not sacrifice pixel count or frame rate. The thing you will lose is image accuracy in fine details like hair and grass if it is done poorly. If done well, it just looks like they magically doubled the system performance.
-
/
would probably like this as well, if you want to post it there tooID: hjf2u1gIt isn’t much of a leak or rumor as it is a verified patent
ID: hjf31krPatents get posted there all the time; I just cba at the moment.
ID: hjfpr47That subreddit kinda lost its meaning as nearly all officially confirmed stuff is posted there. Doesn't matter if it's a leak or announced by a company.
-
Although this may be a cool look at where Sony may be headed, this is just a patent application. They haven't secured a patent yet
ID: hjjqpe8It generally takes 1-2 years before the patent is granted after the application is submitted
ID: hjk0rlcEh, since this is a PCT application, Sony has 30 months from the filing date to enter the national phase, which means another 2-3 years to grant in the US alone. If this application matures into a patent, it'll be closer to 4-5 years after filing
-
They have the tempest 3D Prozessor that can be used for ML.
-
People say things kind deep learning or machine learning but I never have found myself noticing something and thinking “wow that machine learning is working great!”
Maybe that’s the point; for me not to notice it.
-
Coming soon to PS8
引用元:https://www.reddit.com/r/PS5/comments/qnaz3j/sony_just_patented_a_new_reconstruction_method/
Well that is definite proof that they are working on a DLSS competitor that works on PS5.