"Out of memory" after applying Canon Lens data

[flederwiesel]

Hi all,
using Image-ExifTool-9.63 on CYGWIN_NT-6.1-WOW64, I get the following error when processing a *.CR2 file after applying Canon Lens data:

$ exiftool -Copyright=flederwiesel '106_1008_LensDataON.CR2'
Out of memory during "large" request for 33558528 bytes, total sbrk() is 379604992 bytes at /usr/lib/perl5/site_perl/5.10/Image/ExifTool/Writer.pl line 4373.

Before applying Lens Data everything works OK:
$ exiftool -Copyright=flederwiesel '106_1008_LensDataOFF.CR2'
    1 image files updated


One can see that Canon Digital Photo Professional seems to append the processed image data, rather than merely appending lens specification, since the file size is twice as much afterwards.

Is this a problem caused by Exiftool or is this a perl/cygwin limitation? Anyone encountered this issue already? Is there a workaround (besides the obvious one - removing the lens data)?

See
http://flederwiesel.com/download/106_1008_LensDataOFF.CR2 [21MB] /
http://flederwiesel.com/download/106_1008_LensDataON.CR2 [42MB] for testing...

Regards,
Tobias

[Phil Harvey]

Hi Tobias,

Thanks for this report.

Unfortunately, ExifTool loads the entire Canon VRD trailer into memory for processing.  This really shouldn't be too much of a problem because it is only 21 MB, but it seems to somehow push the process over its Cygwin memory limit.  I don't understand a few things about what is happening (why the request is for 33 MB, when the VRD block is only 21;  why ExifTool is already using 379 MB, when I would expect less than 200;  why there seems to be a limit of around 400 MB in Cygwin).

I'll post back here after I get a chance to play with this under Cygwin.  I tested your file on my Mac, and it doesn't have a problem like this.

- Phil

[flederwiesel]

Thanks Phil for the quick response. If you need more information, just let me know.

Tobias.

[Phil Harvey]

I have looked into this and didn't find a way I could easily reduce ExifTool's memory usage.  However, I did find a way to increase the Cygwin memory limit (which is apparently 384 MB by default):  see this post for more information.

- Phil

[flederwiesel]

Hello Phil,
I tried the suggestion from the post, unfortunately this did not work. I also tried the peflags tool, unfortunately with the same result.

I found (in a simple C program) out that cygwin itself lets me malloc() as much as 1GiB, however, the following perl script gives a similar result:

$echo '#!/usr/bin/perl

$size = 1024;

my @array;

do {
   $array[$size - 1] = 0;

   printf("%4u %3s\n",
      $size >= 1073741824 ?
         $size / 1073741824 : $size >= 1048576 ?
            $size / 1048576 : $size >= 1024 ?
               $size / 1024 : $size,
      $size >= 1073741824 ?
         "GiB" : $size >= 1048576 ?
         "MiB" : $size >= 1024 ?
         "kiB" : "B");

   $size *= 2;
} while (true);

exit 0;
' > ./malloc.pl

$ chmod +x ./malloc.pl

$ ./malloc.pl
   1 kiB
   2 kiB
   4 kiB
   8 kiB
  16 kiB
  32 kiB
  64 kiB
128 kiB
256 kiB
512 kiB
   1 MiB
   2 MiB
   4 MiB
   8 MiB
  16 MiB
  32 MiB
Out of memory during "large" request for 536875008 bytes, total sbrk() is 134455296 bytes at ./malloc.pl line 8.


So far,
Tobias

[orax]

I noticed with Cygwin64 (native 64 bits version), I have a better result than the 32 bits version (with malloc.pl).
Maybe you can try the 64 bits version ?

Cygwin64
$ ./malloc.pl
[...]
512 MiB
Out of memory!

Cygwin (32 bits - WOW64)
128 MiB

DOS/Powershell
C:\Perl64\bin\perl.exe C:\cygwin64\home\***\malloc.pl
256 MiB

[orax]

On a virtual machine Win7 x64 with 776 MiB RAM.

With only 16 MiB pagefile.sys (swap) :
$ ./malloc.pl
   1 kiB
   2 kiB
   4 kiB
   8 kiB
  16 kiB
  32 kiB
  64 kiB
128 kiB
256 kiB
512 kiB
   1 MiB
Out of memory!

Now with 8192 MiB pagefile :
It's very very long because ram is saturated and there are many page faults (write on disk) but it can do better than before with fewer pagefile.
$ ./malloc.pl
   1 kiB
   2 kiB
   4 kiB
   8 kiB
  16 kiB
  32 kiB
  64 kiB
128 kiB
256 kiB
512 kiB
   1 MiB
   2 MiB
   4 MiB
   8 MiB
  16 MiB
  32 MiB
  64 MiB
128 MiB
256 MiB
Out of memory!

Therefore I think you should increase your pagefile.sys if its size isn't automatically managed by the OS. Or... add RAM.

[Phil Harvey]

I think it is still worthwhile to figure out a solution to the Cygwin limitation, but after some work on the ExifTool side I have been able to reduce the memory usage somewhat.  It turns out that a majority of the memory usage is due to the large lens data block since the CanonVRD data containing this block is processed in memory.  Here are some stats when run on my Mac (which for some reason seems to have better memory handling than the Cygwin Perl):

ExifTool 9.64	ExifTool 9.65	Operation
26 MB	26 MB	writing to CR2 without lens data
269 MB	208 MB	writing to CR2 with lens data
183 MB	183 MB	reading from CR2 with lens data
183 MB	82 MB	reading from CR2 with lens data, specifically excluding CanonVRD and DLOData tags
298 MB	278 MB	copying tags from one location to another inside CR2 with lens data
298 MB	217 MB	copying tags from one location to another inside CR2 with lens data, specifically excluding CanonVRD and DLOData tags

So you can expect a bit of relief in terms of memory usage from ExifTool 9.65 when it is released.

- Phil

[flederwiesel]

Thanks orax, thanks Phil,
with 16G physical RAM and 8G pagefile, still no luck; malloc script still fails @32MiB.
I will try the 9.65 as soon as available, but meanwhile look for a cygwin solution anyway...

Thanks for your efforts so far,
Tobias

[orax]

16G physical RAM and 8G pagefile

Oh ok, sorry, it's enough ! I was totally wrong...

I just tested your *.CR2 in Cygwin (last version) and Image-ExifTool-9.64. And it works. I haven't any error. Strange. It works on mine, it should work on your. I guess the Force is with me.

Out of memory during "large" request for 33558528 bytes, total sbrk() is 379604992 bytes at /usr/lib/perl5/site_perl/5.10/Image/ExifTool/Writer.pl line 4373.

Have you the last Perl installed ? I don't know if there is a relation but I read 5.10.

perl --version
This is perl 5, version 14, subversion 4 (v5.14.4) built for cygwin-thread-multi
uname -a
CYGWIN_NT-6.1 xxxx 1.7.30(0.272/5/3) 2014-05-23 10:36 x86_64 Cygwin

[Phil Harvey]

I've been spending some more time optimizing the memory usage.  Unfortunately the original CanonVRD writer wasn't very efficient because it didn't need to be (before Canon added the large DLOData record), so it has been a fair bit of work to re-design how it handles memory.  Below are the updated stats (v9.65b1 is my previous attempt, and v9.65b2 v9.65b3 is my current working version):

	ExifTool 9.64	ExifTool 9.65b1	ExifTool 9.65b2	ExifTool 9.65b3	Operation
a)	26 MB	26 MB	26 MB	26 MB	writing to CR2 without lens data
b)	269 MB	208 MB	46 MB	46 MB	writing to CR2 with lens data (no CanonVRD tags written)
c)	368 MB	307 MB	287 MB	249 MB	writing to CR2 with lens data (CanonVRD tags written)
e)	183 MB	183 MB	183 MB	143 MB	reading from CR2 with lens data
f)	183 MB	82 MB	82 MB	41 MB	reading from CR2 with lens data, specifically excluding CanonVRD and DLOData tags
g)	298 MB	278 MB	217 MB	177 MB	copying tags from one location to another inside CR2 with lens data (no CanonVRD written)
h)	400 MB	379 MB	339 MB	298 MB	copying tags from one location to another inside CR2 with lens data (CanonVRD written)
i)	298 MB	217 MB	95 MB	55 MB	copying tags from one location to another inside CR2 with lens data, specifically excluding CanonVRD and DLOData tags (no CanonVRD written)

So for the specific command you were using (b), v9.65b2 reduces the memory usage by more than a factor of 5 over v9.64 (on my OS X dev system).

- Phil

Edit:  Added v9.65b3 with further improvements

[flederwiesel]

Thanks orax for the hint!
Indeed I was using perl 5.10. Don't know why updating perl didn't come to my mind  Today I checked installing a fresh cygwin x64 /w perl 5.14 and it worked! So it turns out that perl 5.10 (cygwin wow64 build) for some reason cannot handle the "large" request - whatever you try to patch your environment...

Phil, I will leave the older perl and test exiftool 9.65 on one of my machines anyway...

Thanks guys and sorry for the noise!

Tobias

[Phil Harvey]

Hi Tobias,

sorry for the noise!

Not at all.  It has been a very useful exercise.  I hadn't realized how much memory the CanonVRD processing was using, and I've learned some things about Cygwin that I didn't know before.

- Phil

[flederwiesel]

Hi Phil,
I tested 9.65 on my old configuration, and it works with the Lens Data! Well done!

Thank you,
Tobias

[Phil Harvey]

Hi Tobias,

Great!  Thanks for testing this.

- Phil