Memory management help (Java)

[[+Duracell-]]

Hopefully someone can help me here. I have this block of code (description will follow below it):

if (log.toString().contains("IFX"))
{
	Date startTime, stopTime;
	long timer = 0;
	int msgCount = 0;
	String date, time, msg, svc;
	String[] timeTemp;
	
	// Regex to see if the line matches what we need.
	Pattern pattern = Pattern.compile("(\\d{4}-\\d{2}-\\d{2})[ ](\\d{2}:\\d{2}:\\d{2}\\.\\d{3}).*context=MQListener-[0-9]*:(.*?),.*clientName=(.*?),.*");
	Matcher matcher;
	String line = "";
	
	startTime = new Date();
	BufferedReader reader = new BufferedReader( new FileReader(log));
	while ( (line = reader.readLine()) != null )
	{
		matcher = pattern.matcher(line);
	    	
		if ( matcher.find())
		{		
			// Extract information from line
			date = matcher.group(1);
			timeTemp = matcher.group(2).split(":");
			time = timeTemp[0]+timeTemp[1];
			msg = matcher.group(3);
			svc = matcher.group(4);
			
			// If the channel does not exist in our hashmap yet, add
			// the channel to the hashmap.
			if (!(channels.containsKey(svc)))
			{
				System.out.printf("New Channel found: %s\n", svc);
				channels.put(svc, new Channel());
			}
			
			// Add the message to the channel
			channels.get(svc).addMessage( new Message( date, time, false), msg );
		}
		
		matcher.reset();
	}

	stopTime = new Date();
	timer = stopTime.getTime() - startTime.getTime();
	System.out.printf("Ran in %d milliseconds\nMessages counted: %d\n", timer, msgCount);
	reader.close();
}

The Message object contains 3 member variables: String date, String time, boolean tc.
date should be no larger than 10 bytes (yyyy-mm-dd)
time should be no larger than 4 bytes (hhmm)

Channel contains only one member variable:
HashMap<String, ArrayList<Message>>

Simple walkthrough: First, this block checks to see if there is a log file with IFX in the name. If there is, open it up and start reading it. It'll attempt to match lines with the regex pattern above. If it does find a match, it'll extract the information, create a Message object, and store that Message in a Channel container.

The files are 260-300MB in size, and the program at its current state reads one file in about 2 minutes. The problem is when I read even two 260MB files, I get over 500MB of memory usage. It should never be this much, I think. I read in about 550,000 Message objects, so 550,000 x 16 bytes = ~8.5MB in memory. Even if we multiply that by 10 to account for overhead, it should still be only <100MB of usage.

Am I missing anything here? Is Java overhead really that horrible? This is Java 6, btw.

Is there any place where I can forcefully deallocate objects in the code? And if not, what are ways I can optimize my memory usage?

 

[generelz]

I don't see anything glaring with the code you have posted. It would also be nice to see a sample line that this code is supposed to parse. However one thing that does look suspicious is addMessage( new Message( date, time, false), msg );

Since I don't know about your data set I can't say what the possible size of the msg String could be however its something you didn't provide any information about.

There are plenty of free Java profilers out there, also, which might give you a good idea of where you might be going wrong.

 

[nameless_centurian]

i recall that in java, strings are immutable and are never destroyed. additionally, the string "hi" is different than the string "hiya"..... your memory usage seems to possibly be indicative of this java feature.

 

[[+Duracell-]]

Yea, I noticed that. I got an evaluation of YourKit Java Profiler, and it looks like the BufferedReader.readLine() statement is to blame. It's allocating almost 340MB for its char[] buffer, I'm guessing. And since the method returns a String, I'm guessing that String object for each line isn't destroyed.

My Message objects only take up ~8MB, with 17MB of overhead for the Strings. So now I have to figure out how to clear out the char[] and prevent BufferedReader from allocating more than x amount of space for it.

EDIT: Never mind...somehow, deleting the date string from the message object lowered my memory usage tremendously. So confused.