I'm getting an "Out of memory!" error when trying to extract GPS data from large MP4 video files.
Monitoring exiftool while its running shows that the failure occurs when process memory gets to approx. 1.9 GB.
For example, one video file is 60 GB, 27 min 27 sec duration, captured at 5 fps.
The command I am using is:
exiftool -p csv4.fmt -ee -api LargeFileSupport video.mp4 > video.csv
The csv4.fmt file contains:
#[IF] ${gpslatitude#;$_ = undef if $_ == 0} ${gpslongitude#;$_ = undef if $_ == 0 ${gpsaltitude#;$_ = undef if $_ == 0}
#[BODY]$sampletime#,$gpslatitude#,$gpslongitude#,$gpsaltitude#
#[TAIL]
I'm currently using exiftool.exe for Windows version 11.9.4.0. System memory is 32 GB.
Please advise how I can resolve this.
Thank you
This might be a variation on Known Problem 2013-04-21 (https://exiftool.org/#problems). The Windows EXE version has a limited amount of memory available.
You might try installing Perl and the Perl version of exiftool and see if that helps.
Quote from: StarGeek on June 05, 2020, 11:24:52 AM
This might be a variation on Known Problem 2013-04-21 (https://exiftool.org/#problems). The Windows EXE version has a limited amount of memory available.
You might try installing Perl and the Perl version of exiftool and see if that helps.
Thank you for the suggestion. Where would I find information on the version of Perl to use and the instructions on how to install and use the Perl version of exiftool?
I use Strawberry Perl for Windows (http://strawberryperl.com/). Grab and install that. Then download the first link on the main page (http://exiftool.org) (the one that says Download Version 11.99 (4.6 MB) - May 11, 2020). Open the .tar.gz file up (use 7zip (https://www.7-zip.org/download.html) if you don't already have something that opens it), extract the lib directory from it and copy the two directories in that (File and Image) into the C:\Strawberry\perl\lib file. Then extract the exiftool file, place it somewhere in your path and rename it to exiftool.pl.
Quote from: StarGeek on June 05, 2020, 11:40:34 AM
I use Strawberry Perl for Windows (http://strawberryperl.com/). Grab and install that. Then download the first link on the main page (http://exiftool.org) (the one that says Download Version 11.99 (4.6 MB) - May 11, 2020). Open the .tar.gz file up (use 7zip (https://www.7-zip.org/download.html) if you don't already have something that opens it), extract the lib directory from it and copy the two directories in that (File and Image) into the C:\Strawberry\perl\lib file. Then extract the exiftool file, place it somewhere in your path and rename it to exiftool.pl.
Thanks. I need a portable way to run exiftool since I use it as a tool that I run via SHELL commands from a master program. Is that possible? I see there is a portable version of Strawberry Perl. Could that be used? I would need to install both without making changes to the user's PATH.
Unfortunately, I don't know. But if there's a portable version of Perl, it seems likely that it could be possible.
It would be interesting to know if the alternate Windows exe version of ExifTool (https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows) has the same problem.
- Phil
Quote from: Phil Harvey on June 05, 2020, 04:17:30 PM
It would be interesting to know if the alternate Windows exe version of ExifTool (https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows) has the same problem.
- Phil
Thank you. It looks interesting. I'll try it.
Quote from: Phil Harvey on June 05, 2020, 04:17:30 PM
It would be interesting to know if the alternate Windows exe version of ExifTool (https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows) has the same problem.
- Phil
I just tried it. Unfortunately, the same thing happens.
I would appreciate any other suggestions. Could I ask why does ExifTool run out of memory and what would be involved in enabling it to use more?
One thought. If I split the large video files into segments and processed them separately with ExifTool and then merge the separate GPS outputs should that work? I would suspect maybe not since the splitting operation might interfere with how the embedded files are stored and processed.
I have doubts you could find a splitting program that would leave the GPS data intact. Exiftool knows over 40 ways GPS data can be embedded in the file and I have doubts that most splitting programs would be able know about all that.
Yes. I have my doubts too but I'm going to try it unless Phil says it won't work.
No harm in trying. If it works I'd be interested in knowing the program you used.
I've experimented a bit with creating segments with ffmpeg but it doesn't seem possible or I can't find a way to do it. I may be because the GPS data has a different time frequency than the video stream. For example, 1 sec GPS samples vs 5, 25, 30, 60 fps for video frames. The stream containing the GPS samples also has other types of camera metadata samples such as acceleration which may have other sample rates. Splitting the file into segments would have to account for the different sample frequencies and split on the appropriate time boundaries so that the video is split on a frame that does not occur in the middle of a metadata sample.
I'll try the original Perl version suggested and see if that avoids the Out of memory condition.
It would still be interesting to hear why the ExifTool exe cannot use more memory.
I've experimented a bit with creating segments with ffmpeg but it doesn't seem possible or I can't find a way to do it. It may be because the GPS data has a different time frequency than the video stream. For example, 1 sec GPS samples vs 5, 25, 30, 60 fps for video frames. The stream containing the GPS samples also has other types of camera metadata samples such as acceleration which may have other sample rates. Splitting the file into segments would have to account for the different sample frequencies and split on the appropriate time boundaries so that the video is split on a frame that does not occur in the middle of a metadata sample.
I'll try the original Perl version suggested and see if that avoids the Out of memory condition.
It would still be interesting to hear why the ExifTool exe cannot use more memory and avoid this error. The problem seems to have existed for a long time.
I was able to avoid the "Out of memory!" condition by running the Perl version of exiftool using the Portable version of Strawberry Perl for Windows. I had to modify the bat file used to start it so that I could run it with the arguments I need to pass to exiftool at runtime. Thank you StarGeek for your help!
It would still be interesting to hear why the ExifTool exe cannot use more memory and avoid this error. The problem seems to have existed for a long time.
That I cannot answer. I don't know if it's a problem with the Windows command line or the Perl PAR packer that the Windows .exe version uses.
Quote from: dzeek on June 07, 2020, 06:30:10 AM
I was able to avoid the "Out of memory!" condition by running the Perl version of exiftool using the Portable version of Strawberry Perl for Windows.
did you use the 64 bit version of portable Strawberry Perl or the 32 bit version?
Then I could provide a 64bit version of my installer mentioned above.
Oliver
I don't know why there is a memory limit like this. It seems to be a limitation only for command-line apps.
- Phil
it might be also a 64bit <-> 32bit issue since the amount of memory was huge.
That's why I would appreciate feedback from dzeek.
Yes, it would be good to know if the 64-bit app impacts this, but I don't think ExifTool should be anywhere near the 32-bit limit here. Most of the memory usage should be in storing the large number of extracted tags. The video data itself isn't loaded into memory, so the huge size of the file shouldn't be a problem (other than the fact that it contains so many GPS tags). If I am correct here, there should be no memory problem without the -ee option.
- Phil
Quote from: Phil Harvey on June 09, 2020, 09:09:18 AM
I don't think ExifTool should be anywhere near the 32-bit limit here. Most of the memory usage should be in storing the large number of extracted tags. The video data itself isn't loaded into memory, so the huge size of the file shouldn't be a problem
Maybe I misinterpreted the OP reporting 1.9GB for ExifTool:
Quote from: dzeek on June 05, 2020, 11:13:27 AM
Monitoring exiftool while its running shows that the failure occurs when process memory gets to approx. 1.9 GB.
Oliver
Quote from: obetz on June 07, 2020, 03:53:40 PM
Quote from: dzeek on June 07, 2020, 06:30:10 AM
I was able to avoid the "Out of memory!" condition by running the Perl version of exiftool using the Portable version of Strawberry Perl for Windows.
did you use the 64 bit version of portable Strawberry Perl or the 32 bit version?
Then I could provide a 64bit version of my installer mentioned above.
Oliver
I am using the 64-bit version of the portable Strawberry Perl for Windows.
This enabled the perl version of exiftool to use approx. 13 GB of process memory vs the 1.9 GB when running the exe versions.
Sorry for the delay responding.
Quote from: obetz on June 09, 2020, 09:29:06 AM
Maybe I misinterpreted the OP reporting 1.9GB for ExifTool:
Right, sorry. You are correct. It really was using that much memory!
So a 64-bit version should fix it. :)
- Phil
Quote from: Phil Harvey on June 09, 2020, 09:44:03 AM
Quote from: obetz on June 09, 2020, 09:29:06 AM
Maybe I misinterpreted the OP reporting 1.9GB for ExifTool:
Right, sorry. You are correct. It really was using that much memory!
So a 64-bit version should fix it. :)
- Phil
Could I ask why a 64-bit version of the exe versions of ExifTool for Windows does not already exist? Are there disadvantages to using a 64-bit version?
Quote from: dzeek on June 09, 2020, 09:47:20 AM
Could I ask why a 64-bit version of the exe versions of ExifTool for Windows does not already exist? Are there disadvantages to using a 64-bit version?
Until now, there was no clear reason to build this version. And it is more work to build two versions. Also, the binary is likely larger, and I don't think that there are any speed improvements. But Oliver will know more about this.
- Phil
Hi dzeek,
no problem, thanks for the reply.
Your case is one of the rare opportunities to improve my package.
Would you test a 64 bit version of my "alternate ExifTool" package? I'm not sure when I can prepare it, maybe this evening, maybe Friday.
Oliver
Quote from: obetz on June 09, 2020, 09:55:31 AM
Hi dzeek,
no problem, thanks for the reply.
Your case is one of the rare opportunities to improve my package.
Would you test a 64 bit version of my "alternate ExifTool" package? I'm not sure when I can prepare it, maybe this evening, maybe Friday.
Oliver
Yes. I would be happy to test a 64-bit version. No rush. Thank you.
Quote from: Phil Harvey on June 09, 2020, 09:54:40 AM
Quote from: dzeek on June 09, 2020, 09:47:20 AM
Could I ask why a 64-bit version of the exe versions of ExifTool for Windows does not already exist? Are there disadvantages to using a 64-bit version?
Until now, there was no clear reason to build this version. And it is more work to build two versions. Also, the binary is likely larger, and I don't think that there are any speed improvements. But Oliver will know more about this.
- Phil
Oh. Sorry. I saw some posts reporting the same problem in the past so I thought there might an issue creating or using a 64-bit version.
Quote from: dzeek on June 09, 2020, 11:47:50 AM
Yes. I would be happy to test a 64-bit version. No rush. Thank you.
Here we go: https://oliverbetz.de/cms/files/Artikel/ExifTool-for-Windows/exiftool-11.99_64.zip
Please check that you don't have any other Perl installation in the Path environment variable and tell me whether my 64 bit version solves your issue.
Oliver
Quote from: obetz on June 11, 2020, 06:37:12 AM
Quote from: dzeek on June 09, 2020, 11:47:50 AM
Yes. I would be happy to test a 64-bit version. No rush. Thank you.
Here we go: https://oliverbetz.de/cms/files/Artikel/ExifTool-for-Windows/exiftool-11.99_64.zip
Please check that you don't have any other Perl installation in the Path environment variable and tell me whether my 64 bit version solves your issue.
Oliver
This solves the issue. Thank you.
thanks for the feedback.
I will update my build batch and provide 32 and 64 bit versions as installer and zip package for new versions of ExifTool starting with version 12.00 (likely tomorrow).
Oliver