Jinwoo Hwang

Subscribe to Jinwoo Hwang: eMailAlertsEmail Alerts
Get Jinwoo Hwang via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Java EE Journal, IBM Journal, Java Developer Magazine


Unveiling the java.lang.Out OfMemoryError

And dissecting Java heap dumps

Dissecting IBM Text(Classic) Heap Dump File Structure
The following is a sample Java heap dump taken from an IBM Java Virtual Machine on Windows:

// Version: J2RE 6.0 IBM J9 2.4 Windows x86-32 build 20080209
0x10000000 [1000] OBJ ObjectA
0x10000100 [2000] OBJ ObjectB
0x10000200 0x10000300
0x10000200 [3000] OBJ ObjectC
0x10000300 [4000] CLS ClassD
0x10000400 [5000] OBJ byte[]
0x10000500 [6000] OBJ char[]
// Breakdown - Classes: 1, Objects: 3, ObjectArrays: 0, PrimitiveArrays: 2
// EOF:

Each entry has information about a live object. For instance, the first entry:

0x10000000 [1000] OBJ ObjectA

shows that there's an object named ObjectA at the address of 0x10000000. Its size is 1,000 bytes. In an older IBM JVM, you might not be able to see the "OBJ" tag. The object, ObjectA, has a reference to another object that is located at the address, 0x10000100, which is the addresss of the object, ObjectB, as you can see in the next entry:

0x10000100 [2000] OBJ ObjectB
0x10000200 0x10000300

This entry shows that the object, ObjectB, is located at the address of 0x10000100. The size of the object ObjectB is 2,000 bytes. The object, ObjectB, has reference to an object located at the address, 0x10000200, which is the address of the object, ObjectC, and to another object located at the address 0x10000300, which is the address of the class, ClassD, as you can see in the next entries:

0x10000200 [3000] OBJ ObjectC
0x10000300 [4000] CLS ClassD

The object, ObjectC, is located at the address of 0x10000200. The size of ObjectC is 3,000 bytes. It has references to the address, 0x10000500, which is the address of the array of character(char[]).

The class, ClassD, is located at the address of 0x10000300. The size of classC is 4,000 bytes. In an older JVM, you might not be able to see the "CLS" tag. It has references to the address, 0x10000400, which is the address of an array of bytes(byte[]). The primitive arrays, byte[] and char[], do not have any reference to other objects.

Now we can draw the directional graph shown in Figure 3 to visualize the references from each object. The char[] and byte[] do not have any references, so they are shown as leaf nodes in the graph.

We need to assess the weight of each object, class or primitive type. It's called TotalSize. The TotalSize of an object is the sum of an object's size and sum of its children's TotalSizes. If an object does not have any children, TotalSize is the same as object size of the object.

If you put the graph shown in Figure 3 upside down, it may look like an apple tree with fruit. The TotalSize is like the sum of all the weights of all the apples hanging above a specific branch. If you cut off a branch, pick all the apples from the branch, and put them on a scale, you will get the TotalSize of that branch. By looking at the TotalSize of each object, we can tell which object has references to objects using larger amounts of the Java heap, which is most likely where a memory leak might occur.

The graph shown in Figure 4 has total sizes calculated from each branch:

The TotalSize of byte[] is just the size of itself, 5,000 bytes, because it does not have any descendents. The TotalSize of classD (4,000 + 5,000) is the sum of the size of itself (4,000 bytes) and the sum of all its descendents, byte[] (5,000 bytes).

We could easily spend a night or two interpreting millions of objects in heap dumps and drawing directional graphs. That's why we need the IBM HeapAnalyzer as it will do all the work for you.

Figure 5 is a screenshot of the IBM HeapAnalyzer with the sample Java heap dump loaded. The directional graph is represented as a tree in the IBM HeapAnalyzer. There are an analysis view and a tree view. We can select any object in the tree view and get detailed information about the object such as address, name, and number of parent objects.

Dissecting the HPROF ASCII Java Heap Dump File Structure
Let's take a look at the hprof ASCII output.

The Java heap dump shown in Listing 1 was taken from Sun's JVM. The first line indicates a version of the Java heap dump format, JAVA PROFILE 1.0.1 followed by a timestamp of the file creation. (The listings for this article can be downloaded here.)

JAVA PROFILE 1.0.1, created Sat Feb 14 07:07:45 2009

The next record is thread information. Each thread record has a reference to an object, an identifier, a thread name and a thread group.

THREAD START (obj=5ecd1b8, id = 1, name="Signal dispatcher", group="system")

We don't see this information in IBM Java heap dumps. Next to Java thread records, we can see Java stack trace records. Each Java stack trace has a series of Java stack frames. Java stack traces are referenced from Java heap dump entries for example object record.  We do not see Java stack trace records in IBM Java heap dumps.

Finally we can find Java heap dump records. They are located between the following the two lines:

HEAP DUMP BEGIN (333 objects, 97716 bytes) Sat Feb 14 07:08:15 2009



This section has list of all live instances in the JVM. There are four types of instances:

  • ROOT : Root instance
  • CLS : Class
  • OBJ : Object
  • ARR : Array

The first entry in the heap dump record has the following:

ROOT 5ef2ee0 (kind=<thread>, id=3, trace=6)

This is a ROOT. The address of this instance is 5ef2ee0. We can find what's at the address 5ef2ee0 by looking further into the heap dump entries. It's an instance of java/lang/ref/Finalizer$FinalizerThread:

OBJ 5ef2ee0 (sz=76, trace=0, class=java/lang/ref/Finalizer$FinalizerThread@5ef2ea0)

This instance is referenced from <thread>. The thread ID is 3. We can find the thread from the THREAD section:

THREAD START (obj=5ef2ee0, id = 3, name="Finalizer", group="system")

Java stack trace ID is 6.We saw the  Java stack trace 6 in the TRACE section:

java/lang/Object.wait(Object.java:Compiled method)

This is very useful information. It says java/lang/ref/Finalizer$FinalizerThread is a root object referenced from a thread. It provides even the Java stack trace of the thread, thread name, and thread group name. Unfortunately, we don't see this kind of information from an IBM Java heap dump.

Let's move on to a class entry:

CLS 607fd58 (name=HeapExhaustionSimulator, trace=0)
super          5e2aba8
loader          60a2490
domain        60a9590
constant[3]   5f5f2f8

This is a class(CLS). It's located at 607fd58. The name of this class is HeapExhaustionSimulator, and then we have a list of references. The super class is located at 5e2aba8.

Have you noticed anything that's not found in the IBM Java heap dump? We have the names of variables: super, loader and domain. In the IBM Java heap dump, we have only addresses. The names of variables provide critical information in lots of cases where you want to track down the reference chains. The following is an array entry.

ARR 60a9a30 (sz=44, trace=0, nelems=20, elem type=char)

This array is located at the address 60a9a30. The size of the array is 44 bytes. There are 20 elements in this array. This is a character array. The last entry is an object:

OBJ 5ef2ee0 (sz=76, trace=0, class=java/lang/ref/Finalizer$FinalizerThread@5ef2ea0)

This object is located at 5ef2ee0. The size of this object is 76 bytes. The Java stack trace identifier is 0. The class of this object is located at 5ef2ea0. The name of the class is java/lang/ref/Finalizer$FinalizerThread.

Generally hprof heap dumps are larger than IBM Java heap dumps. The following is the list of Java heap dump files:

02/14/2009  07:07 AM          9,087,076 heapdump.20090214.070746.6092.phd
02/14/2009  07:08 AM        49,468,469 heapdump.20090214.070747.6092.txt
02/14/2009  07:10 AM        94,548,647 java.hprof.bin
02/14/2009  07:08 AM      199,752,135 java.hprof.txt

IBM binary (PHD) format takes up the least space. The IBM Text (Classic) file takes about five times more space than the PHD format does. The Hprof binary takes almost 10 times the space as the IBM PHD file does. The Hprof ASCII heap dump is 20 times larger than the IBM PHD file. Generally hprof heap dumps are larger than IBM heap dumps because the hprof format carries much more information that the IBM heap dumps do not have. Text/ASCII heap dumps are larger than binary heap dumps because the binary format carries record tables to avoid duplicate records that are often written multiple times (or millions of times in worst case) in Text/ASCII heap dumps. Text/ASCII Java heap dumps are rarely used unless there's a problem generating binary Java heap dumps.

If you are running a JVM based on Sun's implementation, you don't have many choices: hprof binary or hprof ascii format. Since the two formats contain same information, you would probably pick the binary format that is chosen by default.

If you're running the IBM JVM, you have two more choices in addition to the hprof formats. PHD (binary) is the preferable format and the default format. PHD is the smallest in size in a majority of cases. If you need to read the contents of the objects, the Java thread name, the Java stack traces associated with the objects, or names of the variables, the hprof binary format is your choice.

Table 1 provides a list of the available hprof options from the IBM Java 2 Runtime Environment, Standard Edition Service Release 11 (build 1.4.2, J2RE 1.4.2 IBM Windows 32 build cn142-20080515) :

Hprof usage: -Xrunhprof[:help]|[:<option>=<value>, ...]

Table 2 shows a list of the available hprof options from the IBM Java SE Runtime Environment Service Release 2 (build pwa6460sr2-20080818_01)

hprof usage: java -agentlib:hprof=[help]|[<option>=<value>, ...]

Please be aware that IBM's hprof is demonstration code for the Java Virtual Machine Tool Interface (JVMTI); it's not an official product or a formal part of the Java Virtual Machine.

Actually for more than a decade, Sun has stated that the hprof format is still considered highly experimental and is subject to change without notice. If you are tired of reading now, let's have some fun with experimenting and simulation.

Experiment and Simulation of java.lang.OutOfMemoryError
First, let's simulate the native memory exhaustion and analyze the different artifacts generated by the JVM (see Listing 2a). In this experiment, we are creating a list of direct byte buffers. Because we're using native memory excessively, we expect to see native memory exhaustion. Please don't run this application on your production system since it could cause system hang, abnormal termination, or unexpected failures.

IBM's Java Virtual Machine 1.4.2 throws the message shown in Listing 2. It says the JVM was unable to allocate 1,000,000 bytes of direct memory after five unsuccessful retries. IBM's Java Virtual Machine 1.6 throws the following message:

Exception in thread "main" java.lang.OutOfMemoryError: native memory exhausted
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:300)
at NativeMemorySimulator.main(NativeMemorySimulator.java:11)

It definitely says native memory is exhausted.

Sun's JVM 1.4.2 and 1.6 throws the following message:

Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at NativeMemorySimulator.main(NativeMemorySimulator.java:11)

It indicates it's out of memory while the Java Virtual Machine is reserving the direct buffer memory. We can take a look at line 11 in the NativeMemorySimulator.java to see if we are using too much direct buffer memory.

Even though the Java heap is not exhausted, let's make sure that we're not missing anything. We can bring up the IBM Pattern Modeling and Analysis Tool for the Java Garbage Collector (http://www.alphaworks.ibm.com/tech/pmat) and analyze the garbage collection trace as shown in Figure 6.

Used heap size does not even reach the total heap size but we encounter java.lang.OutOfMemoryError, which is indicated by red vertical line. Let's experiment with the Java heap exhaustion (see Listing 2b). In this simulation, we keep creating a list of strings, an array of characters, and an array of integers. The following is the command line I used to generate the hprof ASCII Java heap dump on the IBM 1.4.2 JVM service release 11:

E:\IBMSDK14211\bin\java -Xrunhprof:heap=dump,format=a,dooom=y -Xmx50m -classpath e:\heapdump -verbosegc HeapExhaustionSimulator 2> verbosegc.txt

The following is the command line I used to generate THE hprof binary Java heap dump on THE IBM 1.4.2 Java Virtual Machine service release 11:

E:\IBMSDK14211\bin\java -Xrunhprof:heap=dump,dooom=y -Xmx50m -classpath e:\heapdump -verbosegc HeapExhaustionSimulator 2> verbosegc2.txt

If you want to generate the IBM PHD binary Java heap dump and IBM Text/ASCII Java heap dumps at the same time, you can set the environment variable, IBM_JAVA_HEAPDUMP_TEST, to true, for example, on Windows.


The following is the command line to generate the hprof binary Java heap dump on Sun Microsystem's Java Virtual Machine 1.6.0 update 12:

E:\sun1.6.0_12\bin\java -Xmx50m -classpath e:\heapdump -verbosegc -XX:+HeapDumpOnOutOfMemoryError HeapExhaustionSimulator > verbosegc3.log

Do we expect to see the Java heap exhaustion? Yes, the Java heap will be exhausted.

IBM's Java Virtual Machine throws the following message:

JVMDG274: Dump Handler has Processed OutOfMemory.
Exception in thread "main" java.lang.OutOfMemoryError
at HeapExhaustionSimulator.main(HeapExhaustionSimulator.java:20)

It doesn't indicate whether it's caused by native memory or Java heap.

Sun's Java Virtual Machine throws the following message:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6280.hprof ...
Heap dump file created [50549348 bytes in 1.444 secs]

It says the JVM ran out of Java heap space. Let's take a look at the Java heap usage from the garbage collection trace.

Analysis of Garbage Collection Trace from Java Heap Exhaustion
We can bring up the IBM Pattern Modeling and Analysis Tool for the Java Garbage Collector and analyze garbage collection trace as shown in Figure 7. The analysis indicates that the Java heap was completely exhausted. The tool also provides recommendations.

Assess required Java heap and increase maximum Java heap size using -Xmx option.

If it does not help, analyze Java heap dump with IBM HeapAnalyzer.

If we are still in doubt, we can take a look at the chart view in Figure 8. The red solid line indicates the used Java heap size. The yellow line indicates the total Java heap size. The blue line indicates the amount of the requested Java heap. The vertical red-dotted line indicates java.lang.OutOfMemoryError. When java.lang.OutOfMemoryError occurred, even though the Java heap usage was smaller than the total Java heap size, more Java heap than was available was requested. If increasing the maximum Java heap size does not alleviate the situation, we definitely need to take a look at theJava heap dumps.

What if you don't have a garbage collection trace? If you have an IBM Java Virtual Machine, it probably would generate the Java thread dump or javacore.

Analysis of Java Thread Dump from Java Heap Exhaustion
Let's bring up the IBM Thread and Monitor Dump Analyzer for Java and open a javacore generated during java.lang.OutOfMemoryError (see Figure 9). The tool alerts you with a warning: Java heap is almost exhausted (6% free space) as well as a recommendation. Okay, all the tools recommend the Java heap dump analysis.

Analysis of Java Heap Dump from Java Heap Exhaustion
Let's bring up the IBM HeapAnalyzer and open the Java heap dump. If you've already downloaded a copy of the IBM HeapAnalyzer several months ago, it's a good time to get the latest version. IBM HeapAnalyzer is updated frequently in order to accommodate recently discovered memory leak patterns. Figure 10 is the first screen after the Java heap dump is loaded.

Java heap usage size is 52,404,992 bytes (approximately 49.77 Mbytes), which is pretty close to the maximum Java heap size of 50 Mbytes. This tells us that the Java heap is almost exhausted. Let's find out which objects are using most of the Java heap. We can take a look at the Java heap from a reference tree perspective in the lower part of the screen. If there are no memory leak suspects, we need to open the Tree View manually by selecting Tree View from the Analysis Menu.

Here's list of columns in the Reference Tree Window:

TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(501) ObjectName Address

  • [ObjectSize] is the amount of space used by a specific object, class, or array.
  • TotalSize is the sum of all [ObjectSize] of objects referenced from an object and its decendants.
  • (TotalSize/HeapSize%) is the ratio of TotalSize to total Java heap usage.
  • NumberOfChildObject is the number of children under this level or branch.
  • ObjectName is the name of an object, a class, or an array.
  • Address is the address of an object, a class, or an array.
  • Here are the first couple of entries:

    TotalSize (TotalSize/HeapSize%) [ObjectSize] NumberOfChildObject(501) ObjectName Address
    50,001,176 (95%) [32] 1 java/util/ArrayList 0x1abb6d8
    50,001,144 (95%) [152] 26 array of [Ljava/lang/Object; 0x29fdd70
    2,000,016 (3%) [2,000,016] 0 char[] 0x1502760

    The object, java/util/ArrayList, has references to other objects and the sum of the sizes of all the objects referenced from the object and its descendants is 50,001,176 bytes.

    The next largest TotalSize is 2,000,016 bytes from char[]. If there's any memory issue, the root cause should be under the object java/util/ArrayList not from long[] because the object java/util/ArrayList has a much larger TotalSize, which is as much as 95% of the total Java heap usage. We could say that most of apples are under this branch.

    How can we find a memory leak? It's already been highlighted. There's also a menu called "Subpoena Leak Suspect(s)" in the Reference Tree. Leak suspects are displayed in blue. By default the most suspected area is displayed (see Figure 11).

    It's very straightforward. We have a Java heap leak suspect, an array of java/lang/Object that is responsible for 50,001,144 bytes (95.41293 %) of the Java heap. The parent of the array of java/lang/Object is java/util/ArrayList, which means java/util/ArrayList contains the array of java/lang/Object. What objects do we have in the array? We can take a look at the children of the array of java/lang/Object. There are lots of arrays of integer (int[]) and arrays of character(char[]). Interestingly we cannot find who created java/util/ArrayList from the Java heap dump since it's marked as a root, which is not referenced from any object. We'll revisit this later in this article.

    In summary, we have java/util/ArrayList that contains an array of int[] and char[]. It's responsible for 95.4 % of the Java heap based on the IBM HeapAnalyzer's analysis. We need to find out why java/util/ArrayList consumes so much Java heap. If there's any memory leak, we should address it. If there's no memory leak and we just need more memory, we could increase maximum Java heap size by adjusting -Xmx command-line option.

    We've use very simple application. What if there's any memory leak in static variable? Can we find which class is responsible for memory leak from a static variable?

    More Stories By Jinwoo Hwang

    Jinwoo Hwang is a software engineer, inventor, author, and technical leader at IBM WebSphere Application Server Technical Support in Research Triangle Park, North Carolina. He joined IBM in 1995 and worked with IBM Global Learning Services, IBM Consulting Services, and software development teams prior to his current position at IBM. He is an IBM Certified Solution Developer and IBM Certified WebSphere Application Server System Administrator as well as a SUN Certified Programmer for the Java platform. He is the architect and creator of the following technologies:

    Mr. Hwang is the author of the book C Programming for Novices (ISBN:9788985553643, Yonam Press, 1995) as well as the following webcasts and articles:

    Mr. Hwang is the author of the following IBM technical articles:

    • VisualAge Performance Guide,1999
    • CORBA distributed object applet/servlet programming for IBM WebSphere Application Server and VisualAge for Java v2.0E ,1999
    • Java CORBA programming for VisualAge for Java ,1998
    • MVS/CICS application programming for VisualAge Generator ,1998
    • Oracle Native/ODBC application programming for VisualAge Generator ,1998
    • MVS/CICS application Web connection programming for VisualAge Generator ,1998
    • Java applet programming for VisualAge WebRunner ,1998
    • VisualAge for Java/WebRunner Server Works Java Servlet Programming Guide ,1998
    • RMI Java Applet programming for VisualAge for Java ,1998
    • Multimedia Database Java Applet Programming Guide ,1997
    • CICS ECI Java Applet programming guide for VisualAge Generator 3.0 ,1997
    • CICS ECI DB2 Application programming guide for VigualGen, 1997
    • VisualGen CICS ECI programming guide, 1997
    • VisualGen CICS DPL programming guide, 1997

    Mr. Hwang holds the following patents in the U.S. / other countries:

    Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.