Blog – Page 28 – The Renegade Coder

As I near the end of my first semester in grad school, I have a confession to make: one of my classes ruined computer graphics for me, and it's had some pretty depressing consequences. Coursework As someone studying in the graphics department, I should probably have some experience with computer graphics. Luckily, I'm taking a real-time rendering course now, but this isn't my first time playing with computer graphics. CSE5542: Real-Time Rendering Currently, I am taking CSE5542, a real-time rendering course at The Ohio State University. Coincidentally, it's also taught by my advisor, Professor Han-Wei Shen. If you're wondering what the course covers, here's a brief syllabus in no particular order: Spaces (local, eye, world, clip)Shaders (vertex & fragment)WebGL & OpenGLMatrix Transformations (including hierarchical)Lighting (local & global)Texture Mapping (regular, environment, & bump)Procedural TexturesVisibilityGraphics HardwareCoordinate Systems To supplement these topics, there are five projects and two exams which cover all the material above. Previous Courses Now, I should mention that this isn't my first exposure to computer graphics. In fact, I completed a computer gaming minor at Case Western Reserve University which included the following courses: Modeling and Simulation (MATLAB)Game Design 1 & 2 (Unity)Computer Graphics (OpenGL) In addition, I spent a semester at the University of Manchester where I took a sound design course which leveraged Blender for game development. Over the course of that semester, I learned about object modeling and basic game design. As an aside, one of my best friends does professional 3D modeling in SOLIDWORKS, so I've even gotten some exposure to the mechanical side of computer graphics: https://youtu.be/1zxHfX_qTpU Overall, I've been exposed to computer graphics at various levels from shader programming in OpenGL to game development in Unity, so you'd probably think I know my stuff. Turns out, this semester proved me wrong. Learning the Hard Way Despite having some previous experience with graphics, my current real-time rendering class has been a major struggle for me. Between fighting JavaScript and stumbling through WebGL, I can't catch a break. In other words, the easiest of concepts can take several hours to implement, and that's just the beginning. As a result, a lot of the magic of modern computer graphics has been ruined for me. Now, I can't even watch Coco or play Destiny 2 without thinking about how their lighting, texturing, and physics algorithms work. Since I have to deal with this pain on a daily basis, so do you! Here are a few topics that I've struggled with this semester: Modeling In computer graphics, real world objects have to be modeled as a set of vertices and triangular faces. For some objects, like triangles, this is relatively easy. Select your 3 vertices and connect them in any order. Unfortunately, for anything more complicated, choosing your vertices and connecting them becomes significantly harder. Just to illustrate this absurdity, try to model a square as four vertices in clockwise order (a, b, c, d) and two triangles (X, Y). You'll find out very quickly that the order in which you specify the vertices per face matters. # Incorrect X = [a, b, c] Y = [b, c, d] # Correct X = [a, b, c] Y = [c, d, a] In the first case, you'll end up with a concave pentagon as the segment between d and a is never drawn. Now, imagine literally any other object, and you'll find that connecting vertices can be very tricky business. In fact, there's an entire field of Computer Science dedicated to this class of problems known as computational geometry. Fortunately, most modeling software can export objects with the proper vertex to face mapping, so you'll never have to worry about doing this sort of thing by hand like I just did above. Transformation Now that we have a model, we should be able to transform it any way we like with relative ease. After all, what's so hard about moving, scaling, and rotating an object? Of course, I ask that question because transforming an object is not easy. As it turns out, moving an object requires you to move each pixel individually. For translation, this is again relatively easy: X = [a, b, c] # Moves X face to the "right" by 1 unit a.x += 1 b.x += 1 c.x += 1 Here, we have a face (X) which is composed of three vertices (a, b, c). To move the face to the right, we modify the x component of each vertex by 1. Of course, this is extremely tedious and impractical. Instead, we typically generate a translation matrix and apply it to every vertex directly. However, that's still pretty tedious, so we usually leave our original vertices alone and pass the translation matrix to the GPU. That way, we can take advantage of all the GPU cores for matrix multiplication. Instead of performing three sequential additions, the GPU will perform three parallel matrix multiplications. If we'd like to perform rotation or scaling, there are matrices for that as well. Unfortunately, however, even the matrix method has its drawbacks. For example, let's say we want to use our square from earlier as a rectangle. To do that, we would just need to scale one axis of the square. No harm done, right? Wrong! Now, all future rotations will cause some bizarre skewing to occur. To be safe, you have to make sure you always perform the rotation first. Sadly, I learned this the hard way, but I couldn't find a clean fix. So, I cheated by creating a rectangle model. Projection Up until this point, the only transformation we've done is with respect to the object or its world. In other words, how do we get that object from its 3D world to our 2D screen. That's where projection comes in! To flatten our object to the screen, we're going to need a camera. After all, our scene can look drastically different depending on where we're looking. To add a camera, we need to know a few things: its location, what it's looking at, and which way is up. These three vectors come together to form the camera. Aren't things just getting delightfully complex now? Having a camera is not enough unfortunately. First, we have to transform our vertices of our object from local space to world space. This gives our object context within its world. From there, we have to convert the vertices from world space to eye space which puts the world in the context of the camera. Finally, we have to project what we can see from the camera down to the screen. There are generally two ways to do this: orthogonal projection and perspective projection. Perspective projection retains depth information while orthogonal projection does not. In other words, perspective projection shrinks objects that are further away. All of these transformations require matrix multiplication which can be done most efficiently by the GPU. As a result, we usually try to multiply all these matrices into a single matrix before passing that information to the GPU. Shading If those three topics weren't complicated enough, I have some bad news for you: we haven't even drawn anything yet. That's where shading comes in! With your transformation matrix ready to go, it's time to pass it to the vertex shader. The vertex shader is responsible for transforming all vertices based on your transformation matrix. In addition, you're free to specify other information here like vertex colors, normals, and texture coordinates. Anything specified at the vertex level will be interpolated across the surface of the face. For example, if you specify red for each vertex in a triangle, then the triangle will be red. However, if you change one of those vertices to blue, then you'll get a nice red to blue gradient over the surface of the triangle. After all the vertices are processed, the GPU moves to the fragment shader which handles pixel shading. This is where your scene is drawn. Between the two shaders, it's up to you to figure out what information to manipulate and pass at each stage. For example, if you want local illumination, you'll need to decided whether you want per-vertex lighting or per-fragment lighting. Be aware that per-vertex lighting tends to create a disco ball look. As a bit of a curve ball, shader code has its own language, so I had fun trying to play with that. Unfortunately, there aren't many tools to help debug a shader, so it's pretty normal to see a black screen for several hours at a time. Animation & Physics At this point, we've covered the basics. In other words, everything we've talked about has formed the basis of computer graphics for about as long as the field has existed. Everything else is just extra. For instance, animation and physics are just an extension of graphics pipeline. Do you want a fancy lighting effect that changes over time? Consult the graphics pipeline. Do you want to simulate gravity? Consult the graphics pipeline. As I said, everything else is just extra. The Final Project For the final project, we've been asked to add game-like mechanics to a typical graphics scene. Beyond that, the description is pretty lax: 1. Enhance your graphics with game-like actions: this action is to be controlled by the user of the program through mouse or keyboard. This game-like action can be simply a tour through some pre-set path or be some simple actions that turn your program into a playable game.2. Enhance your scene and complete the environment as the result of this semester3. Show a draft of your final product in front of the class Naturally, I decided to expand on my previous project, so I could reuse some code. In that project, we were asked to load external models and apply textures to them. In particular, I chose to load a plane with a camo texture as well as an island environment texture. To expand on this environment, I decided to add the ability to shoot projectiles. In addition, I've added some flying animation which allows you to control the strafe of the plane left and right. On top of that, I've attached the camera to the plane, so it follows the plane as it strafes. Pitfalls I should mention that this project has been excruciatingly difficult. As I've already mentioned, anytime you want to do anything, you have to work through the graphics pipeline. As a result, even the most basic of changes can be a significant challenge. Just take a look at my commit history! Camera Tracking One of the first pitfalls I ran into was binding the camera to the plane. At first, I wanted to be able to free roam the environment with the plane, but I quickly realized that moving the camera in 3D space is not a trivial matter. As it turns out, keeping the camera locked on an object is easy. Manipulating which way is up while changing the camera location is not easy. As a workaround, I decided to restrict the motion of the plane to strafing left and right. That way, I could steal the plane's location and use it as the "look at" vector. As for the camera's position, I slightly modified the plane's location, so the camera would strafe with the plane. Loading Models In the previous project, we were asked to load external models. In other words, we would need to store a model locally and load it into the scene rather than generating an object using math. Sadly, this is really, really hard. Models come in an infinite supply of formats including the JavaScript favorite JSON. Unfortunately, not even JSON models are standardized, so you have to parse every new model by hand. Naturally, this leads to all kinds of nasty problems like missing vertices. improperly mapped textures, etc. Ultimately, I found that I can download objects from 3D Paint as an .FBX file and convert them to JSON using assimp2json. Unfortunately, even this gave me problems, but it was significantly more reliable than a model site like clara.io. Optimization As it turns out, loading objects and textures is a major bottleneck in any graphics application as the resources need to be loaded in from disk or over the network. For me, that ended up being 8 texture images and 3 sizable models. After getting everything working, I found that my game was taking awhile to load even on my desktop which houses a GTX 1070. As a result, I spent a ton of time optimizing my application by preloading all the assets and caching them in main memory. Unfortunately, none of this seemed to be helping, so I added a loading bar to track the loading process. Turns out the projectiles were taking forever to load even though the textures and models were all already cached. Eventually, I discovered the the true bottleneck was in applying textures that weren't even being used. For instance, I was applying a cube map to every single object in the scene despite only using it on the plane. After cutting that code, my scene began to load almost instantly. Needless to say, I was ecstatic albeit a little frustrated. A Functioning Game Despite all my complaints, I now have a fully functioning game. If you'd like to check it out, head over to my GitHub repository and select Lab 05. Feel free to play with the other projects as well. Unfortunately, Raw Git is shutting down, so these links will only work for a few more months. After that, the repo will probably need to be converted to GitHub pages, but I'm too lazy to do that at the moment. If you'd be interested in doing that, head over and make a pull request. I'd appreciate it! Back to Reality While I'm almost done with this class, I feel like it's going to have a lasting impact on my ability to enjoy even the most basic digital entertainment. Even now, I play Destiny 2 and wonder how they manage to render some of their effects. How is a person supposed to enjoy life when they can't even play video games the same way? This class may have just ruined computer graphics for me, and I can't wait to see what else will be ruined for me next semester!