Project 5B: Ruoyu Fan

README.md

@@ -3,26 +3,185 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Ruoyu Fan
+* Tested on: **Google Chrome 54.0.2840.87** on
+  * Windows 10 x64, i7-4720HQ @ 2.60GHz, 16GB Memory, GTX 970M 3072MB (personal laptop)
+
+__NOTE:__ my submission requires an additional WebGL extension - `EXT_frag_depth`, used in `defered/ambient.frag.glsl` to properly write depth data into lighting passes' frame buffer to do __inverted depth test__ with __front-face culling__
 
 ### Live Online
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+[![](img/thumb.png)](https://windydarian.github.io/Project5-WebGL-Deferred-Shading-with-glTF/)
+
+[Click Me!](https://windydarian.github.io/Project5-WebGL-Deferred-Shading-with-glTF/)
+
+#### Demo GIF
+
+![](screenshots/preview.gif)
+
+### Work Done
+
+* Basic __Deferred Shading__ pipeline
+* __Blinn-Phong shading__ with normal mapping
+  * Using `clamp(1.0 - light_distance * light_distance / (u_lightRad * u_lightRad), 0.0, 1.0) ` as attenuation model for point lights
+* __Bloom__ post-processing effect with __two-pass Gaussian blur__ using three steps:
+  * First extract bright areas with a threshold
+  * Then do a __two-pass Gaussian blur__ using separable convolution (vertical then horizontal)
+    * Use menu option to control blur size (which changes uniform variable `u_scale` passing to `bloom.frag.glsl`)
+  * Finally combine the blurred image to the original output
+* __Scissor test for lighting__: when accumulating shading from each point light source, only render in a rectangle around the light.
+  * Use `debugScissor` option to toggle scissor visual, or select `6 Light scissors` to show scissor only.
+  * This is used to compare with my __light proxy__ implementation
+* __Light proxy__: instead of rendering a scissored full-screen quad for every light, I render __proxy geometry__ which covers the part of the screen affected by a light (using spheres for point lights), thus __reducing wasted fragments in lighting pass__.
+  * Using __inverted depth test__ with __front-face culling__ to avoid lighting geometries that are far behind the light, thus further reducing wasted fragments
+  * This feature requires WebGL's `EXT_frag_depth` extension to write depth data into frame buffer at defered shading stage in order to do the depth test
+  * Use `useLightProxy` option to toggle on/off and use `useInvertedDepthTestForLightProxy` option to toggle depth test and front face culling for lighting pass
+* __Optimized g-buffer__ from 4*vec4 to 2*vec4 by compressing normal to two floats, increasing framerate to __167%__, see below for details
+
+#### Light Proxy:
+
+| Render           | With quad scissor test|
+|------------------|------------------|
+| ![](screenshots/proxy1.png) | ![](screenshots/proxy2.png) |
+
+| With light proxy (no depth test) | light proxy + depth test + front-face culling|
+|------------------|------------------|
+| ![](screenshots/proxy3.png) | ![](screenshots/proxy4.png) |
+
+#### Bloom:
+
+| No bloom          | Bloom (size 0.003)|
+|------------------|------------------|
+| ![](screenshots/nobloom.png) | ![](screenshots/bloom1.png) |
+
+| Bloom (size 0.01) | Bloom (size 0.05)|
+|------------------|------------------|
+| ![](screenshots/bloom2.png) | ![](screenshots/bloom3.png) |
+
+### Optimizing G-Buffer
+
+The initial g-buffer implementation was 4*vec4 (4 textures), using each one for world position, geometry normal, color map and normal map. But in fact position and color can be stored in vec3, and we can use the information that __the length of a normal is one__ to __compress normal to 2 floats__.
+
+So I optimized my g-buffer by using 2*vec4 (2 textures) and stored world position, color map, and world normal (applying normal map to geometry normal) into it.
+
+The g-buffer before optimization was like:
+
+|          | x         | y         | z         | w       |
+|----------|-----------|-----------|-----------|---------|
+| gbuffer0 | pos.x     | pox.y     | pox.z     | nothing |
+| gbuffer1 | geomnor.x | geomnor.y | geomnor.z | nothing |
+| gbuffer2 | color.r   | color.g   | color.b   | nothing |
+| gbuffer3 | normap.x  | normap.y  | normap.z  | nothing |
+
+
+__After my optimization__, the g-buffer is like:
+
+| 0        | x                     | y                     | z                     | w               |
+|----------|-----------------------|-----------------------|-----------------------|-----------------|
+| gbuffer0 | pos.x                 | pox.y                 | pox.z                 | 2_comp_normal.x |
+| gbuffer1 | color.r & normal sign | color.g & normal axis | color.b & normal axis | 2_comp_normal.y |
+
+A typical way to do a 2-component normal is `normal.xy/normal.z` (and restore by `normalize(normal.x, normal.y, 1.0)`), but if `normal.z` is too small, there will be drastic precision loss for normals in that direction; and we cannot tell if the normal is inverted. So, I used some information from "color map" to "help" the normal.
+
+What is `normal sign` and `normal axis`? Well, they are just a sign (+/-). Since colors in glsl are unsigned information stored in signed float, I made use of the sign of the floats to store some information about the normal by using a black magic.
+
+Here is how I compress the normal
+
+```glsl
+// BLACK MAGIC: use color map signs to represent which axis is seen as 1 in normal map
+vec2 two_comp_normal;
+if (abs(normal.z) > 0.33)
+{
+    two_comp_normal = normal.xy/normal.z;
+    colmap.z *= -1.0;
+    colmap.x *= sign(normal.z); // and use x to store if normal is inverted
+}
+else if (abs(normal.y) > 0.33)
+{
+    two_comp_normal = normal.xz/normal.y;
+    colmap.y *= -1.0;
+    colmap.x *= sign(normal.y);
+}
+else
+{
+    two_comp_normal = normal.yz/normal.x;
+    colmap.x *= sign(normal.x);
+}
+```
+
+And here is how I restore the normal:
+
+```glsl
+vec3 extractNormal(float nor_x, float nor_y, vec3 colmap)
+{
+    // Black magic: I colmap sign to prevent normal losing too much precision on a particular axis
+    if (colmap.z < 0.0)
+    {
+        return normalize(vec3(nor_x, nor_y, 1.0)) * sign(colmap.x);
+    }
+    else if(colmap.y < 0.0)
+    {
+        return normalize(vec3(nor_x, 1.0, nor_y)) * sign(colmap.x);
+    }
+    else
+    {
+        return normalize(vec3(1.0, nor_x, nor_y)) * sign(colmap.x);
+    }
+}
+```
+
+Using the sign information as aid, I can store and restore World-space normal in two floats without much precision loss.
+
+And here is the comparison for them
+
+| Number of Lights | Gbuffer Size - 4 | Gbuffer Size - 2 |
+|------------------|------------------|------------------|
+| 50               | 22.2             | 20.4             |
+| 100              | 45.5             | 30.3             |
+| 200              | 83.3             | 50               |
+
+![](img/chart_gbuffer.png)
+
+We can see drastic performance boost here. But this is not a completely correct comparison - I migrated the combination of geometry normal and normal map from lighting stage to copy-to-g-buffer stage. So part of the performance bonus may come from not combining normals for every light.
+
+### Light proxy
+
+instead of rendering a scissored full-screen quad for every light, I render __proxy geometry__ which covers the part of the screen affected by a light (using spheres for point lights), thus __reducing wasted fragments in lighting pass__.
+* Using __inverted depth test__ with __front-face culling__ to avoid lighting geometries that are far behind the light, thus further reducing wasted fragments
+* This feature requires WebGL's `EXT_frag_depth` extension to write depth data into frame buffer at defered shading stage in order to do the depth test
+* Use `useLightProxy` option to toggle on/off and use `useInvertedDepthTestForLightProxy` option to toggle depth test and front face culling for lighting pass
+
+Here are images for comparison (bloom off, 20 lights):
+
+| Render           | With quad scissor test|
+|------------------|------------------|
+| ![](screenshots/proxy1.png) | ![](screenshots/proxy2.png) |
+
+| With light proxy (no depth test) | light proxy + depth test + front-face culling|
+|------------------|------------------|
+| ![](screenshots/proxy3.png) | ![](screenshots/proxy4.png) |
+
+Given 200 lights and bloom off, these are performance comparison:
+
+| Quad scissor | Light proxy | Light proxy with depth test |
+|--------------|-------------|-----------------------------|
+| 76.9ms       | 66.7ms      | 43.5ms                      |
 
-### Demo Video/GIF
+![](img/chart_proxy.png)
 
-[![](img/video.png)](TODO)
+### Bloom
 
-### (TODO: Your README)
+I implemented 2-pass Gaussian Blur with adjustable size.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+Please refer to `glsl/post/bloom.frag.glsl`!
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+| No bloom          | Bloom (size 0.003)|
+|------------------|------------------|
+| ![](screenshots/nobloom.png) | ![](screenshots/bloom1.png) |
 
+| Bloom (size 0.01) | Bloom (size 0.05)|
+|------------------|------------------|
+| ![](screenshots/bloom2.png) | ![](screenshots/bloom3.png) |
 
 ### Credits
 

glsl/clear.frag.glsl

@@ -3,7 +3,7 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 2
 
 void main() {
     for (int i = 0; i < NUM_GBUFFERS; i++) {

glsl/copy.frag.glsl

@@ -10,11 +10,43 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
-void main() {
-    // TODO: copy values into gl_FragData[0], [1], etc.
-    // You can use the GLSL texture2D function to access the textures using
-    // the UV in v_uv.
+vec3 applyNormalMap(vec3 geomnor, vec3 normap)
+{
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+void main()
+{
+    vec3 geomnor = v_normal;
+    vec3 normap = texture2D(u_normap, v_uv).xyz;
+    vec3 normal = applyNormalMap(geomnor, normap);
+    vec2 two_comp_normal;
+
+    vec3 colmap = texture2D(u_colmap, v_uv).xyz;
+
+    // BLACK MAGIC: use color map signs to represent which axis is seen as 1 in normal map
+    if (abs(normal.z) > 0.33)
+    {
+        two_comp_normal = normal.xy/normal.z;
+        colmap.z *= -1.0;
+        colmap.x *= sign(normal.z); // and use x to store if normal is inverted
+    }
+    else if (abs(normal.y) > 0.33)
+    {
+        two_comp_normal = normal.xz/normal.y;
+        colmap.y *= -1.0;
+        colmap.x *= sign(normal.y);
+    }
+    else
+    {
+        two_comp_normal = normal.yz/normal.x;
+        colmap.x *= sign(normal.x);
+    }
 
-    // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[0] = vec4(v_position,  two_comp_normal.x); // world-space position
+    gl_FragData[1] = vec4(colmap, two_comp_normal.y);  // Normals of the geometry as defined, without normal mapping
 }

glsl/deferred/ambient.frag.glsl

@@ -1,27 +1,34 @@
-
 #version 100
+#extension GL_EXT_frag_depth : enable
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 2
 
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
 varying vec2 v_uv;
+const vec3 ambient_color = vec3(0.15,0.15,0.15);
+
+const vec4 SKY_COLOR = vec4(0.98, 0.98, 0.98, 1.0);
+//const vec4 SKY_COLOR = vec4(0.01, 0.14, 0.42, 1.0);
 
 void main() {
-    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    //vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
-    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
 
+    gl_FragDepthEXT = depth; // for use in light proxy
+
     if (depth == 1.0) {
-        gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
+        //gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
+        gl_FragColor = SKY_COLOR;
         return;
     }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    vec3 colmap = abs(gb1.rgb);
+    //gl_FragColor = vec4(vec3(depth),1.0);
+    gl_FragColor = vec4(colmap * ambient_color, 1);  // DONE: replace this
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -2,38 +2,70 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 2
 
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
 uniform float u_lightRad;
+uniform vec3 u_camPos;
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
-varying vec2 v_uv;
+//varying vec2 v_uv;
 
-vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
-    normap = normap * 2.0 - 1.0;
-    vec3 up = normalize(vec3(0.001, 1, 0.001));
-    vec3 surftan = normalize(cross(geomnor, up));
-    vec3 surfbinor = cross(geomnor, surftan);
-    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+vec3 extractNormal(float nor_x, float nor_y, vec3 colmap)
+{
+    // Black magic: I colmap sign to prevent normal losing too much precision on a particular axis
+    if (colmap.z < 0.0)
+    {
+        return normalize(vec3(nor_x, nor_y, 1.0)) * sign(colmap.x);
+    }
+    else if(colmap.y < 0.0)
+    {
+        return normalize(vec3(nor_x, 1.0, nor_y)) * sign(colmap.x);
+    }
+    else
+    {
+        return normalize(vec3(1.0, nor_x, nor_y)) * sign(colmap.x);
+    }
 }
 
 void main() {
-    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
-    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
-    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
-    float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
-
-    // If nothing was rendered to this pixel, set alpha to 0 so that the
-    // postprocessing step can render the sky color.
-    if (depth == 1.0) {
-        gl_FragColor = vec4(0, 0, 0, 0);
-        return;
+
+    vec2 uv = vec2(gl_FragCoord.x / 800.0 , gl_FragCoord.y / 600.0);
+
+    vec4 gb0 = texture2D(u_gbufs[0], uv);
+    vec4 gb1 = texture2D(u_gbufs[1], uv);
+    float depth = texture2D(u_depth, uv).x;
+    // DONE: Extract needed properties from the g-buffers into local variables
+    vec3 pos = gb0.xyz;     // World-space position
+    vec3 colmap = gb1.rgb;  // The color map - unlit "albedo" (surface color)
+    vec3 nor = extractNormal (gb0.w, gb1.w, colmap);   // gb1: geometry normal; gb3: raw normal map
+    colmap = abs(colmap);
+
+    // // If nothing was rendered to this pixel, set alpha to 0 so that the
+    // // postprocessing step can render the sky color.
+    // if (depth == 1.0) {
+    //     gl_FragColor = vec4(0, 0, 0, 0);
+    //     return;
+    // }
+
+    vec3 lightDir = normalize(u_lightPos - pos);
+    float lambertian = max(dot(lightDir, nor), 0.0);
+    float specular = 0.0;
+    if(lambertian > 0.0)
+    {
+        vec3 viewDir = normalize(u_camPos - pos);
+        vec3 halfDir = normalize(lightDir + viewDir);
+        float specAngle = max(dot(halfDir, nor), 0.0);
+        specular = pow(specAngle, 32.0); // TODO?: spec color & power in g-buffer?
     }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    // square falloff
+    float light_distance = distance(u_lightPos, pos);
+    float att = clamp(1.0 - light_distance * light_distance / (u_lightRad * u_lightRad), 0.0, 1.0);
+
+    vec3 color = (lambertian * colmap + specular) * u_lightCol * att;
+    gl_FragColor = vec4(color, 1.0);  // DONE: perform lighting calculations
+    //gl_FragColor = vec4(v_uv, 0.0, 1.0);
 }