"Out of memory!" error when extracting GPS data from Large MP4 Video files

[dzeek]

I'm getting an "Out of memory!" error when trying to extract GPS data from large MP4 video files.
Monitoring exiftool while its running shows that the failure occurs when process memory gets to approx. 1.9 GB.

For example, one video file is 60 GB, 27 min 27 sec duration, captured at 5 fps.

The command I am using is:

Code Select Expand

exiftool -p csv4.fmt -ee -api LargeFileSupport video.mp4 > video.csv

The csv4.fmt file contains:

Code Select Expand

#[IF] ${gpslatitude#;$_ = undef if $_ == 0} ${gpslongitude#;$_ = undef if $_ == 0 ${gpsaltitude#;$_ = undef if $_ == 0}
#[BODY]$sampletime#,$gpslatitude#,$gpslongitude#,$gpsaltitude#
#[TAIL]


I'm currently using exiftool.exe for Windows version 11.9.4.0. System memory is 32 GB.

Please advise how I can resolve this.

Thank you

[StarGeek]

This might be a variation on  Known Problem 2013-04-21.  The Windows EXE version has a limited amount of memory available.

You might try installing Perl and the Perl version of exiftool and see if that helps.

[dzeek]

This might be a variation on  Known Problem 2013-04-21.  The Windows EXE version has a limited amount of memory available.

You might try installing Perl and the Perl version of exiftool and see if that helps.

Thank you for the suggestion. Where would I find information on the version of Perl to use and the instructions on how to install and use the Perl version of exiftool?

[StarGeek]

I use Strawberry Perl for Windows.  Grab and install that. Then download the first link on the main page (the one that says Download Version 11.99 (4.6 MB) - May 11, 2020).  Open the .tar.gz file up (use 7zip if you don't already have something that opens it), extract the lib directory from it and copy the two directories in that (File and Image) into the C:\Strawberry\perl\lib file.  Then extract the exiftool file, place it somewhere in your path and rename it to exiftool.pl.

[dzeek]

I use Strawberry Perl for Windows.  Grab and install that. Then download the first link on the main page (the one that says Download Version 11.99 (4.6 MB) - May 11, 2020).  Open the .tar.gz file up (use 7zip if you don't already have something that opens it), extract the lib directory from it and copy the two directories in that (File and Image) into the C:\Strawberry\perl\lib file.  Then extract the exiftool file, place it somewhere in your path and rename it to exiftool.pl.

Thanks. I need a portable way to run exiftool since I use it as a tool that I run via SHELL commands from a master program. Is that possible? I see there is a portable version of Strawberry Perl. Could that be used? I would need to install both without making changes to the user's PATH.

[StarGeek]

Unfortunately, I don't know.  But if there's a portable version of Perl, it seems likely that it could be possible.

[Phil Harvey]

It would be interesting to know if the alternate Windows exe version of ExifTool has the same problem.

- Phil

[dzeek]

It would be interesting to know if the alternate Windows exe version of ExifTool has the same problem.

- Phil

Thank you. It looks interesting. I'll try it.

[dzeek]

It would be interesting to know if the alternate Windows exe version of ExifTool has the same problem.

- Phil

I just tried it. Unfortunately, the same thing happens.

I would appreciate any other suggestions. Could I ask why does ExifTool run out of memory and what would be involved in enabling it to use more?

[dzeek]

One thought. If I split the large video files into segments and processed them separately with ExifTool and then merge the separate GPS outputs should that work? I would suspect maybe not since the splitting operation might interfere with how the embedded files are stored and processed.

[StarGeek]

I have doubts you could find a splitting program that would leave the GPS data intact. Exiftool knows over 40 ways GPS data can be embedded in the file and I have doubts that most splitting programs would be able know about all that.

[dzeek]

Yes. I have my doubts too but I'm going to try it unless Phil says it won't work.

[StarGeek]

No harm in trying.  If it works I'd be interested in knowing the program you used.

[dzeek]

I've experimented a bit with creating segments with ffmpeg but it doesn't seem possible or I can't find a way to do it. I may be because the GPS data has a different time frequency than the video stream. For example, 1 sec GPS samples vs 5, 25, 30, 60 fps for video frames. The stream containing the GPS samples also has other types of camera metadata samples such as acceleration which may have other sample rates. Splitting the file into segments would have to account for the different sample frequencies and split on the appropriate time boundaries so that the video is split on a frame that does not occur in the middle of a metadata sample.

I'll try the original Perl version suggested and see if that avoids the Out of memory condition.

It would still be interesting to hear why the ExifTool exe cannot use more memory.

[dzeek]

I've experimented a bit with creating segments with ffmpeg but it doesn't seem possible or I can't find a way to do it. It may be because the GPS data has a different time frequency than the video stream. For example, 1 sec GPS samples vs 5, 25, 30, 60 fps for video frames. The stream containing the GPS samples also has other types of camera metadata samples such as acceleration which may have other sample rates. Splitting the file into segments would have to account for the different sample frequencies and split on the appropriate time boundaries so that the video is split on a frame that does not occur in the middle of a metadata sample.

I'll try the original Perl version suggested and see if that avoids the Out of memory condition.

It would still be interesting to hear why the ExifTool exe cannot use more memory and avoid this error. The problem seems to have existed for a long time.