Sony just patented a new reconstruction method using deep learning

1 : Anonymous2021/11/05 13:19 ID: qnaz3j
Sony just patented a new reconstruction method using deep learning
2 : Anonymous2021/11/05 13:20 ID: hjett5t

From LinkedIn with somebody associated with the patent

Most recently I've spearheaded work in using Neural Rendering to enhance traditional rendering methods, focusing on using implicit neural representations and how to make them run efficiently. This includes creating custom high performance inference using compute shaders. The target for this work is PlayStation 5. Development done in Python, Unity, C#, compute shaders with offline training using Pytorch.

ID: hjevi85

Well that is definite proof that they are working on a DLSS competitor that works on PS5.

ID: hjh2fih

You need hardware acceleration for that type of upscaling though and it's typically too intensive just to do on regular shaders. That's why Nvidia has custom tensor cores on all their RTX 20 and 30 chips to implement DLSS.

If this was ever going to come to light my guess is it would be on a PS5 Pro or PS6.

ID: hjfnm1y


ID: hjf19gj

Hopefully for PS5. This might be in the line for PS6 or the PlayStation PCs.

ID: hjevfw8

From reading through the patent, it also has good implications on the next version of the PSVR, including having it capable of doing some computing in filling image gaps caused by bandwidth issues.

ID: hjhxrs9

Sounds like an exciting prospect!

ID: hjew8bn

Can someone Eli5?

ID: hjezzc0

A version of DLSS (Deep Learning Super Sampling) for PS5. It’s an image reconstruction technique that can take lower resolution textures and blow them up to high resolutions without compromising fidelity. Basically, the PS5 could render an image at say 720p but use this technique to display it as 4K and have it look pin sharp. That’s a very high level explanation.

ID: hjfvhz4

A technique to make the image look shaper with very little impact in performance.

ID: hjhjqmk

Im a student in IT and every time i hear about projects like these, i get major imposter syndrome. knowing that someone out there is capable of working on a project as complex as this and im supposed to one day work along side people like this gives me the heebie jeebies.

Edit: deep learning is computer science on crack.

ID: hjipgm9

deep learning is computer science on crack.

The people who write libraries like pytorch are impressive. Us commoners who train a premade model don't need much experience. It's mostly just fucking around and observing the results

ID: hjjj9or

Imposter syndrome is rampant in the industry and nothing fuels it quite like a white board interview or a hacker rank prescreen.

ID: hji0j4o

The patent talks about inpainting and your quote talks about neural rendering. This uses machine learning, but otherwise it doesn't seem all that similar to DLSS to me? If anything, this seems more similar to Nvidia Canvas / GauGAN to me, but with a more specific use case for filling "holes" in an image (rather than an entire image) along with a way to use nearby information for similar materials.

I'll be honest, I don't fully understand the use case for this based on the patent application. They talk about bandwidth-constrained (VR?) scenarios and when something goes from being occluded to being visible and you have a hole, I think. Would this be for scenarios where your head is moving around in VR at a rate where your traditionally-rendered framerate can't keep up? Rather than needing a completely new frame, you can basically apply the style/material information from other parts of the image to fill in a guess of what those areas probably look like?

EDIT: I forgot about the concept of "timewarp" / reprojection used in VR. If I'm understanding this patent correctly, I think this would add concepts of neural rendering / GAN's into the mix, which seems really interesting for creating high-quality, high-framerate, high-resolution VR experiences. (I think)

ID: hjfsr4u

That sounds exactly like the processor on their Bravia TVs that supposedly enhances the image in ways a human would emphasize.

ID: hjeumhr

Look at the big brain on this guy

ID: hjew6no


ID: hjfgn53

Development done in Python, Unity, C#, compute shaders with offline training using Pytorch

Sounds like some first proof-of-concept than some serious work ready to be deployed. C# is slow for these things, in general it is C/assembly instead.

ID: hjfgcf8

Python and C#? Does DLSS use Python? I can see generating code with Python but it doesn’t perform very well where performance is key.

I’m familiar with embedded swe so I don’t know much about game dev.

ID: hjfuf8q

Python is the most common language used for training deep learning models, which is part of the implementation of the algorithm. It's not being used at runtime.

3 : Anonymous2021/11/05 13:39 ID: hjewbc0

I’m dumb. Does this mean more frames or more pixels?

ID: hjfm5mn

You use less pixels to achieve a sharper image, thus giving your GPU more time do compute more frames.

This is how DLSS works on PC, basically. Instead of rendering 3840x2160 (4K) you render (let's say) 2560x1440p but your magic image reconstruction program figures out the "missing pieces" by taking information from the previous frame, the current frame, grabbing some motion vectors (this pixel was going in this direction, so it's probably going to continue doing so in the next frame, probably?) applying some magic programming sauce to it and the algorithm spits out a 4K image from that information. And since your GPU is now only rendering 1440p (+ magic sauce upscaling), that means you have more GPU oomph to get higher frame rates.

So while outputting, say, 4K60 in ShootGuy IV at max graphics settings may be asking too much (maybe you only get 4K at ~40-45 fps), you can apply the magic sauce upscaler while only technically rendering a 1440p image and now you can output an image that looks very close to 4K and hit 60 fps no problem.

ID: hjfw3xa

I’m glad they’re working on a technology such as this one. ShootGuy III had some terrible problems maintaining a stable framerate. Great game, though.

ID: hjftws9

Dang that’s a great ELI5. Thanks for that info!

ID: hjf2p43

First use less pixels to get more frames. Then magic adds more pixels.

ID: hjex06q

Less missing pixels and frames.

ID: hjf2dse


ID: hjfv747

Less pixel more frame then AI vomits in more pixel

ID: hjid3hf

Think of this like painting. The first brush is the native render. The second pass of the brush that fills in the gaps is the machine learning algorithm being used here. So the PS5 does the first stroke at a lower pixel count very quickly then fills in the gaps using the algorithm at the same time. So in the end, it will not sacrifice pixel count or frame rate. The thing you will lose is image accuracy in fine details like hair and grass if it is done poorly. If done well, it just looks like they magically doubled the system performance.

4 : Anonymous2021/11/05 14:08 ID: hjf07jd


would probably like this as well, if you want to post it there too

ID: hjf2u1g

It isn’t much of a leak or rumor as it is a verified patent

ID: hjf31kr

Patents get posted there all the time; I just cba at the moment.

ID: hjfpr47

That subreddit kinda lost its meaning as nearly all officially confirmed stuff is posted there. Doesn't matter if it's a leak or announced by a company.

5 : Anonymous2021/11/05 23:04 ID: hjhbcyx

Although this may be a cool look at where Sony may be headed, this is just a patent application. They haven't secured a patent yet

ID: hjjqpe8

It generally takes 1-2 years before the patent is granted after the application is submitted

ID: hjk0rlc

Eh, since this is a PCT application, Sony has 30 months from the filing date to enter the national phase, which means another 2-3 years to grant in the US alone. If this application matures into a patent, it'll be closer to 4-5 years after filing

6 : Anonymous2021/11/05 19:06 ID: hjga60m

They have the tempest 3D Prozessor that can be used for ML.

7 : Anonymous2021/11/06 11:39 ID: hjjf3vl

People say things kind deep learning or machine learning but I never have found myself noticing something and thinking “wow that machine learning is working great!”

Maybe that’s the point; for me not to notice it.

8 : Anonymous2021/11/05 17:30 ID: hjfv7bf

Coming soon to PS8


Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x