'Out of memory!' with large files since 11.x releases

[Phil Harvey]

Thanks.  This confirms that the problem is due to reading ImageSourceData into memory.  It looks like I have to try to reproduce this problem myself in the GUI.  I'll post back here with my results.  It could be a while.

- Phil

[Phil Harvey]

Is there any way you could upload your large test file to WeTransfer or somewhere so I can try to reproduce this?

- Phil

Edit:  I created a test file myself with an ImageSourceData size of 1155658112 bytes.  Reading this in ExifToolGUI with ExifTool 10.99 hangs, but it is fine with 11.04.  I do get an "Out of memory!" error with exiftool -imagesourcedata FILE, but that is expected, and there is no way around this.  I also get an error when trying to write the file, but I should be able to fix this.  It will still be useful to have your file because I haven't yet been able to reproduce your observations.

Edit2: (Note that 10.98 will also run out of memory if you request -imageSourceData.)

[ChasaImago]

Is there any way you could upload your large test file to WeTransfer or somewhere so I can try to reproduce this?

- Phil

Yes, sure.
Just give me an email address to which I can send the WeTransfer link. (DM?)

[Phil Harvey]

Thanks.

My email is philharvey66 at gmail.com

[ChasaImago]

1.6Gb of test.tif has been transferred with WeTransfer.
Happy testing! 

[Phil Harvey]

I got the file, thanks.

I was afraid of this.  I can't reproduce your problem when using ExifToolGUI 5.16 and ExifTool 10.04.  I can extract the metadata from this file without problems in the GUI. 

You are just reading the file, right?  If writing, then you would get this problem.  Also, could you re-verify that it is 11.04 (by looking at the "ExifToolVersion" tag when reading another file)?  I don't know where to go from here because I can't reproduce this problem.  My steps were:

1. copy "exiftool.exe" version 11.04 to c:\windows

2. run ExifToolGUI 5.16

3. click on another file and display "All" metadata.

4. verify that ExifToolVersion is 11.04

5. click on your "test.tif" to extract the metadata for this file (ExifTool 11.03 hangs for me at this point, but 11.04 does not).

Just FYI, here is extra metadata extracted by ExifTool 10.99 and later for this file:

Code Select Expand

[Photoshop]     Layer Count                     : 2
[Photoshop]     Layer Rectangles                : 0 0 6302 30000, 0 0 6302 30000
[Photoshop]     Layer Blend Modes               : Normal, Normal
[Photoshop]     Layer Opacities                 : 100%, 100%
[Photoshop]     Layer Names                     : Background, Pro Contrast (CEP 4)
[Photoshop]     Layer Unicode Names             : Background, Pro Contrast (CEP 4)
[Photoshop]     Layer IDs                       : 1, 11
[Photoshop]     Layer Modify Dates              : 2018:07:05 10:10:59-04:00, 2018:07:05 10:10:59-04:00

- Phil

[ChasaImago]

Reading is not the issue (like I said in my first post)

...When trying to modify large files
...
Trying to run ExifTool directly in a command window gives the error message:
Out of memory! when using ExifTool 11.x
and the .TIF file is not modified.

Editing/modifying the metadata IS (with large files) an issue, since ExifTool 10.99. 

I use ExifToolGUI to modify my large panorama files in order to get the proper metadata in, that is omitted by other software like PTGui or RawTherapee, etc.
ExifToolGUI crashes with the previously mentioned error messages, as soon as I try to save the file again, with some modified metatags.

[Phil Harvey]

Ah, OK.  Great!

That will be fixed in 11.05.  I missed the fact that you were modifying files.

I'll release 11.05 ASAP.

- Phil

[Phil Harvey]

11.05 is now available.  (It was actually a LOT simpler to fix the writing problem.)

Thanks for hanging in there and being patient with me.

- Phil

[ChasaImago]

11.05 is now available.

Thanks Phil!

Tested & Approved. 

I've tried a single 3Gb TIFF file and this works also fine (combination of ExifToolGUI + ExifTool 11.05)
I selected several large TIFF files at once with ExifToolGUI, modify metadata and hit 'Save'. Even with 15Gb of image files it works without any error. Great work Phil!

slight remark: I'm under the impression that writing back the large image file takes somewhat more time. Can this be the effect of not loading everything into memory e.g?

Although the -imagesourcedata parameter still produces a Out of memory! error. 

Code Select Expand


H:\Install\ExifTool\exiftool-11.05>exiftool -imagesourcedata test.tif
Out of memory!


Code Select Expand


H:\Install\ExifTool\exiftool-11.05>exiftool test.tif
ExifTool Version Number         : 11.05
File Name                       : test.tif
Directory                       : .
File Size                       : 1601 MB
File Modification Date/Time     : 2018:07:05 16:11:04+02:00
File Access Date/Time           : 2018:07:05 19:43:41+02:00
File Creation Date/Time         : 2018:07:05 19:43:41+02:00
File Permissions                : rw-rw-rw-
File Type                       : TIFF
File Type Extension             : tif
MIME Type                       : image/tiff
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 30000
Image Height                    : 6302
Bits Per Sample                 : 8 8 8
Compression                     : Uncompressed
Photometric Interpretation      : RGB
Image Description               : TEST IMAGE EXIFTOOL for Phil Harvey
Make                            : NIKON CORPORATION
Camera Model Name               : NIKON D800
Strip Offsets                   : 1111949988
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 3
Rows Per Strip                  : 6302
Strip Byte Counts               : 567180000
X Resolution                    : 300
Y Resolution                    : 300
Planar Configuration            : Chunky
Resolution Unit                 : inches
Software                        : Adobe Photoshop CC 2018 (Windows)
Modify Date                     : 2018:07:05 16:10:59
Artist                          : Erik Haak
XMP Toolkit                     : Adobe XMP Core 5.6-c142 79.160924, 2017/07/13-01:06:39
Creator Country                 : The Netherlands
Creator Work URL                : www.panomeister.com
Description                     : TEST IMAGE EXIFTOOL for Phil Harvey
Format                          : image/tiff
Creator                         : Erik Haak
Rights                          : Erik Haak/Pano:Meister/Chasa:Imago
Subject                         : TEST IMAGE EXIFTOOL for Phil Harvey
Authors Position                : Pano:Meister
Color Mode                      : RGB
ICC Profile Name                : Nikon sRGB 4.0.0.3002
Creator Tool                    : Adobe Photoshop CC 2018 (Windows)
Metadata Date                   : 2018:07:05 16:10:59+02:00
Document ID                     : xmp.did:64d1e86e-16a2-6e4c-931e-857d8ce21c70
History Action                  : saved, saved, derived, saved, derived, saved, saved
History Changed                 : /, /, /, /, /
History Instance ID             : xmp.iid:64d1e86e-16a2-6e4c-931e-857d8ce21c70, xmp.iid:377a6b6c-80f2-5748-9560-8281e03a81d8, xmp.iid:de065c5e-cddf-904c-b6c7-9511860f21b9, xmp.iid:e21d3e65-cbab-5a48-95cf-abfb317e0154, xmp.iid:6ed0f02c-fbd8-9b43-819b-78c2093af05d
History Software Agent          : Adobe Photoshop CC 2018 (Windows), Adobe Photoshop CC 2018 (Windows), Adobe Photoshop CC 2018 (Windows), Adobe Photoshop CC 2018 (Windows), Adobe Photoshop CC 2018 (Windows)
History When                    : 2018:06:11 23:09:11+02:00, 2018:06:11 23:28:01+02:00, 2018:06:11 23:33:36+02:00, 2018:06:11 23:47:30+02:00, 2018:07:05 16:10:59+02:00
History Parameters              : converted from image/tiff to application/vnd.adobe.photoshop, converted from application/vnd.adobe.photoshop to image/tiff
Instance ID                     : xmp.iid:6ed0f02c-fbd8-9b43-819b-78c2093af05d
Original Document ID            : xmp.did:64d1e86e-16a2-6e4c-931e-857d8ce21c70
Derived From Instance ID        : xmp.iid:de065c5e-cddf-904c-b6c7-9511860f21b9
Derived From Document ID        : xmp.did:64d1e86e-16a2-6e4c-931e-857d8ce21c70
Derived From Original Document ID: xmp.did:64d1e86e-16a2-6e4c-931e-857d8ce21c70
Marked                          : True
Copyright                       : Erik Haak/Pano:Meister/Chasa:Imago
Current IPTC Digest             : e643fd342e8d7886b16f4b842dbf8ae5
Envelope Record Version         : 4
Coded Character Set             : UTF8
Application Record Version      : 0
Caption-Abstract                : TEST IMAGE EXIFTOOL for Phil Harvey
By-line                         : Erik Haak
By-line Title                   : Pano:Meister
Credit                          : Erik Haak
Source                          : Erik Haak
Date Created                    : 2018:06:10
Time Created                    : 19:50:41+00:00
Keywords                        : TEST IMAGE EXIFTOOL for Phil Harvey
Copyright Notice                : Erik Haak/Pano:Meister/Chasa:Imago
IPTC Digest                     : e643fd342e8d7886b16f4b842dbf8ae5
Displayed Units X               : inches
Displayed Units Y               : inches
Print Style                     : Centered
Print Position                  : 0 0
Print Scale                     : 1
Global Angle                    : 30
Global Altitude                 : 30
Copyright Flag                  : True
URL List                        :
Slices Group Name               :
Num Slices                      : 1
Pixel Aspect Ratio              : 1
Photoshop Thumbnail             : (Binary data 3095 bytes, use -b option to extract)
Has Real Merged Data            : Yes
Writer Name                     : Adobe Photoshop
Reader Name                     : Adobe Photoshop CC 2018
F Number                        : 8.0
ISO                             : 400
Date/Time Original              : 2018:06:10 19:50:41
Create Date                     : 2018:06:10 19:50:41
Max Aperture Value              : 2.0
Flash                           : No Flash
Focal Length                    : 105.0 mm
User Comment                    : Copyright Erik Haak
Color Space                     : sRGB
Exif Image Width                : 30000
Exif Image Height               : 6302
Lens Make                       : Nikon
Lens Model                      : NIKKOR 105mm 1:2.5 Ai
Profile CMM Type                : Nikon Corporation
Profile Version                 : 2.2.0
Profile Class                   : Display Device Profile
Color Space Data                : RGB
Profile Connection Space        : XYZ
Profile Date Time               : 2009:02:20 17:07:10
Profile File Signature          : acsp
Primary Platform                : Apple Computer Inc.
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : none
Device Model                    :
Device Attributes               : Reflective, Glossy, Positive, Color
Rendering Intent                : Perceptual
Connection Space Illuminant     : 0.9642 1 0.82491
Profile Creator                 :
Profile ID                      : 0
Profile Description             : Nikon sRGB 4.0.0.3002
Red Matrix Column               : 0.43607 0.2225 0.01392
Green Matrix Column             : 0.38507 0.71687 0.09706
Blue Matrix Column              : 0.14305 0.06061 0.71399
Media White Point               : 0.9505 1 1.0891
Profile Copyright               : Nikon Inc. & Nikon Corporation 2009
Red Tone Reproduction Curve     : (Binary data 8204 bytes, use -b option to extract)
Green Tone Reproduction Curve   : (Binary data 8204 bytes, use -b option to extract)
Blue Tone Reproduction Curve    : (Binary data 8204 bytes, use -b option to extract)
GPS Latitude Ref                : North
GPS Longitude Ref               : East
GPS Altitude Ref                : Above Sea Level
Layer Count                     : 2
Layer Rectangles                : 0 0 6302 30000, 0 0 6302 30000
Layer Blend Modes               : Normal, Normal
Layer Opacities                 : 100%, 100%
Layer Names                     : Background, Pro Contrast (CEP 4)
Layer Unicode Names             : Background, Pro Contrast (CEP 4)
Layer Modify Dates              : 2018:07:05 16:10:59+02:00, 2018:07:05 16:10:59+02:00
Aperture                        : 8.0
Date/Time Created               : 2018:06:10 19:50:41+00:00
GPS Altitude                    : 50 m Above Sea Level
GPS Latitude                    : 52 deg 30' 7.92" N
GPS Longitude                   : 5 deg 24' 59.82" E
GPS Position                    : 52 deg 30' 7.92" N, 5 deg 24' 59.82" E
Image Size                      : 30000x6302
Megapixels                      : 189.1
Focal Length                    : 105.0 mm


Nevertheless; thanks for tackling this issue so fast.

[Phil Harvey]

Tested & Approved. 

Great!

slight remark: I'm under the impression that writing back the large image file takes somewhat more time. Can this be the effect of not loading everything into memory e.g?

It could slow things down a bit for a traditional hard disk because the head has to seek back and forth more.  But if the hard disk has a good cache size this may not be noticeable.

Although the -imagesourcedata parameter still produces a Out of memory! error. 

There is no way around this.  If you request that ExifTool extracts a tag, then it has to load it into memory.  The only thing I can think of is to maybe catch this error before it crashes the program.  I don't know how to do this, but I'll look into it.  Perl's memory handling is all automatic, so you can't just check for a null pointer like some other languages.

- Phil