Low speed when writing to a large TIF file

[Archive]

[Originally posted by magr on 2005-11-21 11:52:49-08]

When I write EXIF information from an Olympus ORF file (10MB) to a TIF (20MB) using TagsFromFile option, it takes about 3 minutes. I tried to restrict the number of tags written, but it didn't make a significant difference.

Is there a way to speed this up significantly?

Thanks

magr

[Archive]

[Originally posted by exiftool on 2005-11-21 12:57:17-08]

Wow, that's extreme!

Writing to large TIFF images is very memory intensive, and will be extremely slow if there isn't enough memory on your system to contain the entire TIFF.  I would guess that this is why it is so slow for you.

On my system here (450 MHz CPU, 128 MB RAM), copying all of the tags from a 14MB ORF image to a 22MB TIFF takes 13 seconds of clock time (only 5 sec of which is CPU time, and the rest is spent waiting for disk I/O).

I suggest buying more memory or closing some applications to free more memory to try to speed things up.

Or maybe there could be another reason for the lack of memory... What system are  you using?  It is possible that the memory Perl uses is limited on some systems (I would only think this could be possible for ActivePerl on Windows for instance) -- in this case it may be possible to configure Perl to use more memory.

[Archive]

[Originally posted by magr on 2005-11-21 17:47:20-08]

My system has 1G of RAM on a 1300 MHz Duron, so Memory size should not be a problem. I do however run on  Windows, so a memory limitation to perl might be the problem.

I use ActivePerl, so how to configure this for more memory ?

[Archive]

[Originally posted by exiftool on 2005-11-22 00:51:09-08]

RAM certainly isn't a problem on your system!  I was only guessing about a possible ActivePerl memory limit.  I don't use ActivePerl myself so unfortunately I can't help with configuration.  But given the amount of memory you have, I would be surprised if there was such a small limit imposed by ActivePerl.

Maybe we can get some help from other ActivePerl users who read this forum?

Do you get the same slow performance when you set individual tags from the command line?  I'm just trying to rule out the possibility of ExifTool getting into a semi-endless loop somehow.

[Archive]

[Originally posted by magr on 2005-11-22 09:04:09-08]

Thanks for your help, I now found a workaround for my problem. Somehow, it appears that the TIF file that I was using was the problem. This file was created using IrfanView to convert from PPM to TIF. Now I use pnmtotiff, and the problem is gone, only 5 seconds for conversion!

It must have something to do with file compression or such. I don't know exactly the TIF format IrfanView writes.

magr

[Archive]

[Originally posted by exiftool on 2005-11-22 12:35:02-08]

That's good it isn't a general problem on your system.  5 seconds is much more reasonable.

But I would like to know why it was so slow with the IrfanView image.  Maybe I could fix this problem if I could reproduce it here.  If possible, could you make the image available for download, or send it to me by email?  My email should be able to handle a 20MB image.  My address is philharvey66 at gmail.com

It could be that ExifTool is spending time unnecessarily decoding a very large data structure as you suspected.  If this is the problem, I should be able to fix it.

Thanks

[Archive]

[Originally posted by exiftool on 2005-11-25 12:46:50-08]

I found a PC to test this out, downloaded IrfanView and the formats plugins, and installed ActivePerl 5.6.1 (I know you're probably running 5.8, but I don't think this should make a difference and I wanted to run some tests with an older version of Perl at the same time), and I used IrfanView to convert a E-300 ORF  file to TIFF.  The ORF was 14MB and the converted TIFF was 25MB.

Then I ran exiftool to copy all tags from the ORF to the TIFF.

Frankly, I'm amazed by the result.  It took 4 seconds.  What is amazing to me is that it takes 3 seconds just to copy the file using "copy" in the command shell.  So exiftool somehow manages to extract all that information from the ORF image and re-structure the TIFF in 1 second.  Pretty amazing I think, considering this is only 25% of the total time.  The test was run on a 2.4GHz laptop.  ...but I digress.

Now I'm stuck because I can't reproduce your problem.  But I'm more convinced now than ever that it is something I could probably fix if I could reproduce it.

So please, if you're still reading this, it would be very useful if I could have a copy of the file you were having problems with.

[Archive]

[Originally posted by exiftool on 2005-11-28 13:55:37-08]

Thank you for the sample image.

I have downloaded ActivePerl 5.8.7 (the version you are using) and have reproduced your problem.

The slow-down occurs when reading a large number of big blocks from file.  In your sample file, the image is split into 2308 strips which are read individually from the file.

It turns out that the problem isn't with ExifTool.  I have run tests with different versions of ActivePerl to measure exactly how long it takes to read these strips from file and concatenate them into a single data buffer in memory:

With ActivePerl 5.8.7 (build 815), it takes 82.709 seconds to read and join these 2308 strips.

With ActivePerl 5.6.1 (build 638), it takes 0.150 seconds on the same system. (Yes, 550 times faster!!)

There are a number of bug reports involving similar slow performance with 5.8.7.  (bug report ID numbers 26762, 28167, 28194, 28377).  It looks like there is a memory handling problem specific to this version of ActivePerl that is causing the poor performance.

I suggest using ActivePerl 5.6.1 until this bug is resolved.

[Archive]

[Originally posted by exiftool on 2005-12-13 20:31:37-08]

This problem has been patched with ExifTool version 5.83 (released yesterday).  With this version, I check beforehand to see if the TIFF image data is contiguous in the file, and if so it is loaded as a single block.  Since most TIFF images are stored this way, you shouldn't have a problem with slow performance if you decide to go back to ActivePerl 5.8.x in the future.