Shadows - Causing JVM crash.

[Virtudude]

Hi,

When I enable shadows, using the shadowhelper, around 60 percent of the time I get a JVM fatal error that causes the application to crash. The other 40 percent the shadows are displayed as expected (my avatar casts a shadow on the floor). Below is the error, the stack trace, and snippets of my code. The error is related to the buffer when openGl is drawing the elements. The crash happened outside the Java Virtual Machine in native code.

Code Select Expand


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000fc55ea9, pid=2340, tid=8428
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode windows-amd64 )
# Problematic frame:
# C  [atio6axx.dll+0x6a5ea9]
#


And the trace:

Code Select Expand


Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.lwjgl.opengl.GL11.nglDrawElements(IIILjava/nio/Buffer;IJ)V+0
j  org.lwjgl.opengl.GL11.glDrawElements(ILjava/nio/IntBuffer;)V+37
j  com.threed.jpct.CompiledInstance.compileToDL()V+139
j  com.threed.jpct.GLBase.compileDLs()V+57
j  com.threed.jpct.AWTGLRenderer.drawVertexArray(Lcom/threed/jpct/AWTDisplayList;I)V+9
j  com.threed.jpct.AWTJPCTCanvas.paintGL()V+149
j  org.lwjgl.opengl.AWTGLCanvas.paint(Ljava/awt/Graphics;)V+165
j  org.lwjgl.opengl.AWTGLCanvas.update(Ljava/awt/Graphics;)V+2


As seen in the above stack, the error occurs when jPCT calls glDrawElements on GL11. I see a few forums that discuss this issue, such as: http://www.gamedev.net/community/forums/topic.asp?topic_id=463211 and http://jmonkeyengine.org/groups/graphics/forum/topic/trouble-with-dynamically-built-geometry/

However, since no one else has posted this error, I'm assuming that I must be doing something wrong. However, the steps are straight forward so I am hoping that this requires a simple change.


Framebuffer mode: FrameBuffer.SAMPLINGMODE_GL_AA_4X
and the width and height is set at runtime based on the browser size calculations.

Also, since this is running in HW mode, I compile the objects as follows:

Code Select Expand


objs.compile(true,  true, true, true, 4000);


My code snippets:

Initialization code for shadow helper (runs only once):

Code Select Expand


projector = new Projector();
projector.setFOV(0.5f);
projector.setYFOV(0.5f);

shadowHelper = new ShadowHelper(jpctWorld, frameBuffer, projector, 2048);
shadowHelper.setCullingMode(false);
shadowHelper.setAmbientLight(new Color(80, 80, 80));
shadowHelper.setLightMode(true);      
shadowHelper.setBorder(1);      
shadowHelper.addCaster(avatar);      
shadowHelper.addReceiver(floor);


Then, in the game loop, after the initialization has completed:

Code Select Expand


frameBuffer.clear();

projector.lookAt(avatar.getTransformedCenter());
projector.setPosition( new SimpleVector(10, -20, 0) );
shadowHelper.updateShadowMap();
shadowHelper.drawScene();

//Code to render scene to buffer, draw to the buffer, etc...
jpctWorld.renderScene( frameBuffer );
jpctWorld.draw( frameBuffer );
frameBuffer.update();
frameBuffer.displayGLOnly();
hardwareOpenGLWorldCanvas.repaint();


Note: when I set the indexed mode to false when compiling objects that cast shadows, that is:

Code Select Expand


objs.compile(true,  true, true, false, 4000);

then I get a similar error, but the  glDrawArrays is called instead of glDrawElements.

Code Select Expand


j  org.lwjgl.opengl.GL11.nglDrawArrays(IIIJ)V+0
j  org.lwjgl.opengl.GL11.glDrawArrays(III)V+20
j  com.threed.jpct.CompiledInstance.compileToDL()V+151


I've tried decreasing the shadow buffer to 256 and still the same error. I've looked at other code samples and I don't see any major differences. I don't see a similar issue on the forum, except one where updating the graphics driver solved the problem. Any suggestions of what I can try would be greatly appreciated (except for update your graphics drivers, because I have done that).  Perhaps the way I am using jPCT to specify the Open GL buffer is incorrect? BTW, my card is an ATI Mobility Radeon HD 4650.

Thanks!

[EgonOlsen]

I don't think that it actually causes the problem, but these calls

Code Select Expand


jpctWorld.renderScene( frameBuffer );
jpctWorld.draw( frameBuffer );


are actually not needed. drawScene() in the ShadowHelper already does this. If that doesn't help (which is most likely...), there are a few options to try:

• Try to lower batch size from 4000 to 2000 to 1000 to...and see if that helps.
• compile(...) with preferDisplayLists set to false. This will be slower but it would be interesting to see if it actually works.
• Set Config.glUseFBO to false at application startup to see if that helps. Will be slower and limited in shadow map resolution, but again it would be interesting to see if it works that way.
• Try to synchronize your render loop to the object returned by FrameBuffer.getLock();
• Try to render the first frame without using shadow mapping, i.e. by using the normal renderScene...draw...display-sequence instead of the ShadowHelper.

[EgonOlsen]

If nothing else helps, you might want to give this jar a try: http://www.jpct.net/download/beta/jpct.jar

It ports jPCT-AE's VBO support back to jPCT. You can try to use them instead of display lists by setting Config.glUseVBO=true; before creating the frame buffer.

[Virtudude]

Thank you for the quick response and all the suggestions. My results are below. I noted your suggestions, and then the results.

I commented out:
<code>
jpctWorld.renderScene( frameBuffer );
jpctWorld.draw( frameBuffer );
</code>
and still have the same issue. But I'll keep this commented out when the shadow helper is enabled since you mentioned this is done in the shadow helper (thanks for the suggestion).

ATTEMPT 1) Suggestion - Try to lower batch size from 4000 to 2000 to 1000 to...and see if that helps.
2000 gives the same issue.

Result:
I tried 2000, 1000, 500, 100. The lower the value the less the crash happens, especially when the number is set to 100. At 100 I have not been able to cause the crash (yet).

Other observations (when batch size was set to 100).
 I noticed that when I set the max shadow map to 8192 the memory of my java process is 1.6GB of RAM at startup, and then eventually after a minute falls down to 910MB, a huge number. When I disable the shadows completely this drops to 380MB. That is a lot of additional memory to display shadows. Is this typical?

 I noticed that the memory of my java process is 760MB of RAM when I set the max shadow map to 1024 instead of 8192 at startup, and drops to 680MB after a minute.



ATTEMPT 2) Suggestion -  compile(...) with preferDisplayLists set to false. This will be slower but it would be interesting to see if it actually works.
Results: I set the batch size to 10000 to force a consistent JVM crash before applying this change (so I can see if this change alone fixes the issue).

After making this change I did not see the crash (even when I set the batch size to 100). However, like you said, there was a hit in performance. My FPS dropped significantly, to the point my application is too slow.

ATTEMPT 3) Suggestion - Set Config.glUseFBO to false at application startup to see if that helps. Will be slower and limited in shadow map resolution, but again it would be interesting to see if it works that way.
Results: I set the batch size to 10000 to force a consistent JVM crash before applying this change (so I can see if this change alone fixes the issue).

When I made the change I got the same JVM crash consistently.


ATTEMPT 4) Suggestion - Try to synchronize your render loop to the object returned by FrameBuffer.getLock();

I grabbed the lock in the render thread, and ran all the framebuffer calls in a synchronized block as suggested. I really try to avoid doing this, since often it kills performance.

Result: Unfortunately, synchronizing all the frame buffer calls does not work and the exception is the same. The crash still happens after waiting for all the framebuffer code is finished. Here is the output:

<code>
AWT-EventQueue-1 - EVWorld - FPS: 0
AWT-EventQueue-1 - EVWorld - paint() - Entering synchronized block.
AWT-EventQueue-1 - EVWorld - paint() - In synchronized block.
AWT-EventQueue-1 - EVWorld - paint() - Left synchronized block ...
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000010815ea9, pid=10328, tid=9472
# ...
</code>


<code>
j  org.lwjgl.opengl.GL11.nglDrawElements(IIILjava/nio/Buffer;IJ)V+0
j  org.lwjgl.opengl.GL11.glDrawElements(ILjava/nio/IntBuffer;)V+37
j  com.threed.jpct.CompiledInstance.compileToDL()V+139
j  com.threed.jpct.GLBase.compileDLs()V+57
j  com.threed.jpct.AWTGLRenderer.drawVertexArray(Lcom/threed/jpct/AWTDisplayList;I)V+9
j  com.threed.jpct.AWTJPCTCanvas.paintGL()V+149
j  org.lwjgl.opengl.AWTGLCanvas.paint(Ljava/awt/Graphics;)V+165
</code>

ATTEMPT 5) Suggestion - Try to render the first frame without using shadow mapping, i.e. by using the normal renderScene...draw...display-sequence instead of the ShadowHelper.

There are two ways I tried this. First, I still create a new shadowHelper, but I delay calling updateShadowMap() and drawScene() on the shadow helper until after 5 canvas paints. This fails, because the JVM crash is not caused by updateShadowMap() and drawScene() – in fact if I comment them out the crash still happens. So long as a create a new shadow mapper the crash will happen.

The second way is to wait 5 paint cycles before creating a new shadowHelper. The problem with this approach is once the regular rendering sequence is done, and then you later try to create a new shadowHelper, the program freezes indefinitely when trying to create a new Shadow Helper.

ATTEMPT 6) If nothing else helps, you might want to give this jar a try: http://www.jpct.net/download/beta/jpct.jar It ports jPCT-AE's VBO support back to jPCT. You can try to use them instead of display lists by setting Config.glUseVBO=true; before creating the frame buffer.

Result:
The JVM crash did not happen when I set Config.glUseVBO=true.


Conclusion:
Attempt 2 slows things down too much. Attempt 1 forces me to use a small batch size.  Attempt 6 works, does not seem to restrict the batch size, and I don't see a major impact on performance. Attempt 6 adds a little time to startup (to compile VBOs), but this is minor.

Now I have two more questions
- Are there any things I should be aware of when using VBOs?
- Would you kindly suggest ways of reducing memory when using shadows (other than reducing the max shadow map size)? Even for small shadow size values, my memory jumps an additional 100 percent when shadows are enabled.

Thanks again.

[EgonOlsen]

Ok...display lists on ATI/AMD cards have been a bit wiggly in the past for me too in some scenarios. I consider this to be a driver problem. VBO are almost as fast on ATI and maybe even faster on NVidia, so there's no problem with using them. Actually, there's nothing to be aware of except that it's a feature that i hacked in yesterday within a few hours, so i might have overlooked something. However, all of my test cases worked fine (at least on ATI), so i think that they are pretty much stable. I case of problems related to VBO usage, please report back.

About the shadows: I'm not sure right now, i'll have a look later...

[EgonOlsen]

The high memory usage when using shadow maps is a flaw in the setup code for the render targets. I'll correct that and post an updated version.

[EgonOlsen]

I've updated the beta jar with a version that should fix the memory problem.

[Virtudude]

Thanks for the update, this will really benefit everyone who is using shadows with large shadow maps.

I found that your changes made a big difference with large shadow maps – RAM was cut from 1.177GB to 404MB for a shadow map setting of 8192. For small maps, i.e. 1024, I did not see a major difference.

Scenario:
-   Using Config.glUseVBO=true;
-   Removed some of the higher poly models for this test to speed up profiling.
-   Use HW mode with AA.

Test results:

Case 1: Set the max shadow map to 1024 the memory
- Before (Jpct jar prior to memory utilization enhancement):

- Before (Jpct jar prior to memory utilization enhancement):
Memory: 370MB with a spike to 490MB during the first 20 paint frames.
CPU: Spike to 83 percent at startup, stabilizes at 16 percent once running.

- After (Jpct jar after to memory utilization enhancement):
memory: 370MB with a spike to 480MB during the first 20 paint frames.
Cpu: Spike to 82 percent at startup, stabilizes at 16 percent once running.


Case 2: Set the max shadow map to 8192 the memory.

- Before (Jpct jar prior to memory utilization enhancement):
Memory: 1.177GB with a spike to 1.53GB during the first 20 paint frames.
CPU: Spike to 83 percent at startup, stabilizes at 45 percent once running.

- After (Jpct jar after to memory utilization enhancement):
Memory: 404MB with a spike to 570GB during the first 20 paint frames.
CPU: Spike to 83 percent at startup, stabilizes at 45 percent once running.

Case 3: No shadows
Memory: 320MB with a spike to 470MB during the first 5 paint frames.
CPU: Spike to 80 percent at startup, stabilizes at 15 percent once running for several paint() frames.

Again, thanks for the update