Memory leak

Started by Oleander, April 27, 2011, 04:01:09 AM

Previous topic - Next topic
Print
Go Down Pages1
User actions

Oleander

April 27, 2011, 04:01:09 AM
When starting ExifTool with the arguments '-stay_open 1 -@ -' and subsequently writing (XMP) tags to images memory is leaked.
The amount of memory leaked seems to correlate to the number of tags written.

OS: Win XP SP3
ExifTool version: Windows executable 8.50/8.56

Phil Harvey

#1
April 27, 2011, 08:04:11 PM
I can pretty well guarantee that there is no memory leak, but it is normal for memory to grow to a point.  The memory handling is all done automatically by Perl.  At some point all unused memory is recovered by the regular garbage collection.  Most Perl implementations are very good, but the Active Perl (used by the Windows executable) seems to be more of a memory hog.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Oleander

#2
May 02, 2011, 06:54:23 AM
The memory usage of ExifTool grows until there is no memory left (2GB RAM + 2GB swapfile in this case) and then terminates.

Attached a crude sample batch file. Adjust paths as needed. Use a smaller test image / add more tags to speed up memory usage.

Phil Harvey

#3
May 02, 2011, 07:42:55 PM
I can't test this until next week when I will have access once again to a Windows machine.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Oleander

#4
June 01, 2011, 03:26:59 AM
Any progress?

Phil Harvey

#5
June 01, 2011, 07:55:55 AM
Thanks for reminding me about this.  I just ran a test extracting information from an image (832 KB jpeg image) repeatedly using the -stay_open option.  The system was Windows XP SP3, with the Windows executable version of ExifTool 8.57.  After each -execute, I checked the memory usage.  Here are the results (in KB):

16452
16432
16436
16440
16384
16396
16392
16372
16388
16372
16388
16388
16372

So it looks to me as if the ActivePerl memory reclamation is working properly, at least for this test.

I'll try your .bat file.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#6
June 01, 2011, 08:48:34 AM
OK, I can reproduce the leak using your .bat file.  For now, I suggest restarting exiftool every few hundred files or so, but I will continue to look into this to see if I can find the source of the problem.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#7
June 01, 2011, 10:32:48 AM Last Edit: June 01, 2011, 10:44:58 AM by Phil Harvey
FOUND IT!!

It turns out that the Perl garbage collection algorithm has a known bug: It can't handle circular references when deallocating objects.

I was using a circular reference when storing information for tags to be written.  I now remove these references before the object goes out of scope and the problem is SOLVED!  (This also explains why I didn't see the problem when just reading information.)

This fix will appear in version 8.58 when it is released.

Thanks for pointing out this problem.  I learned something.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Oleander

#8
June 01, 2011, 11:16:08 AM
I'm glad I could help.
Print
Go Up Pages1
User actions