Enable rendering with unbounded far distance

[Flarkk]

This PR allows rendering with unlimited camera zfar - or rather limited by the sole floating point range i.e. ~3.4e+38 in single precision and ~1.8e+308 with doubles.

This effectively untaps the potential of reverse-z depth buffer especially with single precision build.

Outline

Currently zfar cannot be pushed further than barely ~1e6 times znear (with single precision floats) because of numerical precision limitations with some methods of struct Projection.
With the default znear = 0.05 this sets the maximum view distance to ~50 km.
While it's enough in most cases, it can be a hard limitation for open worlds, space games, and any very large scene.
Beyond this limit, scene rendering currently breaks and Godot starts spaming errors.

This PR removes this limitation with 3 key changes :

• Stores the 6 frustum planes alongside the projection matrix in the rendering server's internals. Both are equivalent representations except that the former allows retrieving near/far distances and boundary points with much more accurate numerical precision
• Restricts the use of the matrix representation to the sole cases where projections / un-projections are performed. Use the 6 planes in any other situation (typically to retrieve znear / zfar, get boundary points, etc...)
• Bundles the 6 planes representation into a new struct Frustum with a few helper methods for convenience. This avoids relying of Vector<Plane> everywhere and improves readability

This PR focuses on in-game rendering and camera preview in Editor.
On top of it, PR #100896 allows editing very distant objects right in Godot's editor

TODOs (now complete)

• Z-fighting with very far away opaque objects (incl. Sky) in Compatibility and Mobile : Intended behavior (see caveats)
• Objects farther than 1e20 disappear when Occlusion culling is on : Solved by Switch occlusion culling to be based on depth instead of Euclidean distance #103798 (see Enable rendering with unbounded far distance #99986 (comment))

Demo : full-scale scene

large_zfar.zip

Vegetation is ~1m tall and ~10m from the observer
The terrain is 10 km large
The moon's radius is 1,700 km and is 400,000 km from the observer
The Andromeda galaxy is 152,000 light-years wide and is 2.5 million light years from the observer

Performance

This PR improves very slightly the frame process time of an empty scene by ~ 0.5%.
This is likely due to the significant number of planes retrieval operations avoided by the fact the 6 planes are already made available.

Before	After

Effects compatibility

I spent a considerable amount of time making sure all effects work with large zfar.
Some require additional PRs to be merged.
Test project for effects : effects.zip (make sure to preview the large znear camera in each scene)

Effects	Works with large zfar
Bokeh DOF	✔️ with #99755
SSR	✔️ with #99693
SSIL	✔️
SSAO	✔️
Sky	✔️
Shadows	✔️
SDFGI	✔️ with #104120
SSS	✔️ with #99755
Fog	✔️

Caveats

• XR is left unchanged for now because this would require the XR APIs to return frustum planes in addition to the projection matrix. This may be the subject of another PR
• Unlike Forward+, Mobile and Compatibility renderers use 24bit UNORM depth buffers which don't get any benefit from reverse-z, so z-fighting will happen near the far plane when zfar is very large. This can notably cause the sky to occlude very distant objects.

Optional dependencies

#99961 and #99962 may help ensuring non-regression on Viewport and Camera3D.

---

Closes godotengine/godot-proposals#10515
Closes #55070
Supersedes #95944

[Flarkk]

Rebased and fixed style issues to allow CI to run.

[Flarkk]

Rebased on top of #99974

[Calinou]

Objects farther than 1e20 disappear when Occlusion culling is on

This can probably be worked around by assuming the object is visible when it's further away from the camera than this distance (and skipping any occlusion culling checks).

Embree doesn't support double precision, so precision=double builds can't do any better here.

[Flarkk]

Haven't got back to this PR yet, but I think it's likely caused by an overflow with normalize() or length() somewhere.

1e19 ~ 1e20 is the threshold value that exceeds the maximum representable float value when squared (~1e38).

If this overflow happens in Embree then yes, we don't have any other choice but trying to work around it. Will take a look at your suggestion once I complete the work on another incoming related PR : uncapping zfar in the editor itself.

[Flarkk]

I got the culprit : indeed a call to length() that overflows on very far away geometry :

godot/servers/rendering/renderer_scene_occlusion_cull.h

Line 75 in efa1443

float min_depth = (closest_point - p_cam_position).length();

Just issued #100907 that solves the problem, and enables proper occlusion culling even on very far away objects.

---

Edit : turns out that this fix inadvertently reverts another fix. I'm figuring out alternatives
Edit 2 : this is solved by #103798

[Monniasza]

Is this issue being worked on? I'm trying to make a spaceflight simulation game that has objects both near and far.

[Flarkk]

Yes it is, although I couldn't spare time the past weeks.

Note that if the farthest objects in your scene are under ~1e19m close to the camera you will not encounter any occlusion culling issue with this PR. Also you can still exclude individual objects from occlusion culling processing in case they're farther than that.

[Flarkk]

Rebased on top of #103798

[JimmyCushnie]

I know a common objection against this PR is that it's too specific to space games, so I'd like to chime in and share a bit about my use case. My project, which is not a space game, would benefit a lot from this PR.

Along with a small team, I'm building a large-scale multiplayer sandbox game. Players can build structures in the world. We anticipate the biggest player-built structures to be a few kilometers on each axis, potentially even 10km+. We're aiming for pretty large multiplayer servers with a few hundred players building in a shared world.

To support large player-built structures and a high number of players building them, we'll have a large and expansive world. It is desirable for distant features of the world (i.e. mountain ranges), as well as distant player-built structures, to be visible. So we need a big zfar, but we also need a small znear as there are some quite small details on the player-built structures that players need to get very close to adjust.

Doing this is trivially easy with this PR. Without it, I expect we'd probably have to use camera stacking, which sucks for performance, complexity, and proneness to bugs.

I hope this context is useful in maintainer discussions. Cheers.

[thygrrr]

I am building a space game, but generally this is fantastic for anything with large view distances.

Since this is literally just some fancy matrix maths, has a net performance gain; doesn#t break compat - this can lift Godot out of the 30+-year-old adage of "don't put your far clip plane too far, or be prepared to compensate with an egregiously far near clip plane."

[Flarkk]

For the record, although this PR got 2 approving reviews already, it is still missing a proper code approval review.
As soon as a maintainer finds time and interest in reviewing the code, there shouldn't be any blocker merging this PR.

[octanejohn]

@roalyr is this useful to your project ? and is logarithmic depth buffer useful here ?

[roalyr]

Log depth as an option is good stuff for any 3D project, IMO, but re-implementing all the depth-related post-effects is bothersome.

For my personal project I decided to not to go for more than 1e6 z-far and since I am on GLES2 (Godot3) I can't use log depth.

In-editor z-far limit should definitely be more than 1e6. If it can be arbitrary - it would be for the best solely for the navigation convenience.

[Flarkk]

Log depth as an option is good stuff for any 3D project

Reverse z-buffer makes log depth unnecessary with Forward+. This PR fixes a few remaining numerical precision issues so that z-far can be effectively pushed to large values.

In-editor z-far limit should definitely be more than 1e6

This PR allows unlimited z-far in-game (and in camera preview mode in the editor).
For unlimited z-far in the editor, see #100896.

[octanejohn]

can Projective Geometric Algebra enhance parts of the renderer with this pr in the future, or are the two ideas very incompatible
releated
godotengine/godot-proposals#9841
#93418

[RobertBColton]

Very excited for this pr to be merged! Makes things a lot easier to draw infinite planes without messing up shadows.

[Calinou]

Very excited for this pr to be merged! Makes things a lot easier to draw infinite planes without messing up shadows.

This PR doesn't fix the limitations of shadow pancaking. You still need to subdivide your plane mesh sufficiently for it to cast shadows correctly, and/or adjust the Pancake Size property in DirectionalLight3D. Subdivision also helps with UV precision issues on very large meshes.

[RobertBColton]

You still need to subdivide your plane mesh sufficiently for it to cast shadows correctly

Interesting, would this apply to overhead projections too?

In my current project I have a world block that I sandwich the far planes around the bounding sphere of while I wait for this pr to make that solution obsolete. I use an infinite floor to separate the world block from the infinite void obviously.

However, I haven't subdivided the infinite floor at all and the PSSM shadowing works fine (right corner), hence my question.

[Flarkk]

I haven't subdivided the infinite floor at all and the PSSM shadowing works fine (right corner)

I think @Calinou is referring to subdividing meshes that cast shadows, not necessarily those who receive shadows (although the latter also helps avoiding UV precision limitations issues as he mentions too).

In your case the floor receives shadows, and your terrain mesh is likely subdivided enough, so everything works fine.

Makes things a lot easier to draw infinite planes without messing up shadows.

The projection matrix-related precision issues this PR fix also benefit to light culling and shadows rendering. You should have shadows on big far objects render just fine with long range lights with this PR. Is this what you meant ?

[RobertBColton]

Oh I see, yes! That's perfect then.

You should have shadows on big far objects render just fine with long range lights

Yes, that will be beneficial as well! Once this is merged I'll be able to get rid of the hacky scaling I am doing.

Just wanted to share my use case, even though I am not technically drawing universe scale things.
It still has benefits for me! 😄

[Flarkk]

can Projective Geometric Algebra enhance parts of the renderer with this pr in the future, or are the two ideas very incompatible

From what I can tell, there is no inherent incompatibility between them. It's still possible to rewrite the changes made using standard algebra in this PR with PGA formalism.

[aaronfranke]

@Flarkk Can you rebase this? I am highly interested in having this capability in Godot Engine.

[Flarkk]

@Flarkk Can you rebase this? I am highly interested in having this capability in Godot Engine.

@aaronfranke here it is. I hope you can spare some time to do a review ! Any question let me know.

[Calandiel]

Is there any chance for this to get merged in before the feature freeze for 4.6?

[Flarkk]

We're almost in feature freeze for 4.6 as we speak, and given it's a quite large and mathy PR, it's better if merged at the start of a cycle. I guess it's going to have to wait for a next release.

The key missing piece to move on with this PR is a proper code review (reviews left so far cover usability only). Feel free to contribute and leave one !