JDK1.6 Hotspot VM leak - Out of swap space

There's a leak in JDK1.6 HotSpot VM and it doesn't appear to be fixed in JDK1.6. I was surprised to see this behavior because things are fine from java heap point of view. After spending quite amount of time debugging the application, figured out the leak is in VM.

In JDK1.6, if there's any "ClassCastException" happens, those are kept in the JVM native layer and those are never get cleared. This would result in "OutOfMemoryError: ... Out of swap space?". There's a quite lot of such exception happens when the application is executing XA transactions, added to my surprise this happens on the underlying infrastructure (j2ee server) so the application is not the one to be blamed though ;).

For the above reason, jvm crashes in less than two hours (depends on the load). Tracked the code share/vm/runtime/sharedRuntime.cpp from the crash log. Things appears to go wrong in the function generate_class_cast_message.

Problem: When there's a "ClassCastException", the exception trace/message has been added to "NEW_C_HEAP_ARRAY" and those messages never gets cleared.

Fix: Just use "NEW_RESOURCE_ARRAY" to hold the exception message, things should be fine.

Build:

One could pick up the source from java.net (please refer the licensing agreement of OpenJDK before making any change, there's JRL one for research and study purpose you may try ...). Follow the instructions provided in the source bundle to build your fixes.

I just set the following in my linux terminal (pre-req. in my linux box) and i am all set to go.

GCC - pre-req: In the prerequisite section at the top, it's mentioned as gcc 4 but i was on gcc version 3.4.6 20060404 and could build VM successfully.

1. ALT_BOOTDIR
2. ALT_BINARY_PLUGS_PATH
3. cd [src]/hotspot/make
4. make

Note: I was interested in building only the VM and not the whole JDK. If you want to build the JDK, just do it from the top level.

Debug:
gdb: is fairly fine for native code debugging.