Extracting FLIR thermal image and metadata in two output files in one command?

[nametyr]

Good day gentlemen,

As part of a research project involving infrared image analysis for building integrated photovoltaics, I am building a Matlab program that extracts thermal image (raw uint16 luminosity pixels) from standard FLIR thermal JPEGs, as well as the embedded Planck values (based on this). The program analyses 192 images (1 picture every 15 minutes * 2 cameras) per day, which currently represents about a 20 minutes time-lapse in Matlab. A quick look at Matlab profiler reveals that ExifTool clearly is the bottleneck of the process, because I need to launch it twice for each image. The commands I am sending from Matlab are the following:

%Extracting raw thermal image from JPEG image_file_name to temporary PNG raw_thermal_image
[~,~] = system(['exiftool -b -rawthermalimage ',image_directory,image_file_name,' > raw_thermal_image.png']);

%Extracting various metadata tags into temporary CSV file
[~,~] = system(['exiftool -csv -planck* -csv -emissivity -csv -objectdistance -csv -*reflect* -csv -*humidity -csv -atmospherict* -csv -rawthermalimagetype -csv -date* ',image_directory,image_file_name,' > data_out.csv']);

I am trying to decrease the processing time to between 10 and 15 minutes for 192 images; to achieve this, I want to launch ExifTool only once for each image instead of twice. Therfore, I was wondering if it was possible to merge these two commands into a single ExifTool command. I am relatively new to the ExifTool universe, so I read several threads as well as the documentation, and tried several combinations. For example, for a thermal JPEG image named test.jpg:

Code Select Expand

exiftool -csv -planck* test.jpg > data_out.csv -b -rawthermalimage test.jpg > raw_thermal_image.png

...gives the error Sorry, -b may not be combined with -csv. I tried the -sep and -@ argument_file options, but the same error appears.

I also tried extracting metadata in XML since it can be combined with -b, but I cannot achieve two output files:

Code Select Expand

exiftool -xml -planck* test.jpg > data_out.xml -b -rawthermalimage test.jpg > raw_thermal_image.png

...seems to ignore data_out.xml and creates a single, seemingly corrupted raw_thermal_image.png file.

I don't know if the -stay_open command can be useful here; I did stumble accross different topics that stated ExifTool can indeed stay open between 2 different commands, but I did not manage to make it work so far. Can anyone help with this?

Louis

[Phil Harvey]

Hi Louis,

The limitation you are running into is in the shell, which can only redirect the console output to a single file (the ">" redirection).

There are -w and -W options which allow you to write console output to file, but these are not compatible with -csv.  But we can work around this by using -w for the RawThermalImage, and shell redirection for the CSV output:

exiftool -rawthermalimage -b -w %0frawthermalimage.png -q -execute -csv "-planck*" -emissivity -objectdistance "-*reflect*" "-*humidity" "-atmospherict*" -rawthermalimagetype "-date*" -common_args test.jpg > data_out.csv

Alternately, you could use JSON format instead:

exiftool -rawthermalimage -b -j "-planck*" -emissivity -objectdistance "-*reflect*" "-*humidity" "-atmospherict*" -rawthermalimagetype "-date*" test.jpg > data_out.json

And get RawThermalImage from the JSON file.

- Phil

[nametyr]

Thank you Phil, this is exactly the kind of workaround I was hoping for. For those interested, this sole optimization made the program processing time for 181 infrared images go down from 21 minutes to just over 13 minutes:

181 files successfully processed in automatic mode
   Total elapsed time: 1297.18 seconds

181 files successfully processed in automatic mode
   Total elapsed time: 791.05 seconds

[Phil Harvey]

You could reduce this by another 8 minutes (ie. 5 minutes total processing time) using the -stay_open feature with a fair bit more work, but it sounds like you're reached your goal already.

- Phil

[nametyr]

I sure did reach my goal but I've been looking forward to decreasing the processing time even more today. I have been playing with the -stay_open option along with -@ argfile and managed to cut another half out of it (now around 600 seconds of processing, of which around a third is because of image data extraction while the remaining time is advanced processing such as corrective projections and temperature calculations). The main problem now is that the Matlab program sometimes works faster than ExifTool. For instance, I call it once with this command:

Code Select Expand

    [~,~] = system('exiftool -stay_open True -@ exif_args.txt &');

The trailing '&' meaning now Matlab has no way to know when the {ready} prompt comes up (ie when the new commands in the argfile are done executing). This means the code often tries to reach files that are either empty or don't even exist yet if I don't force pauses between the commands, or verify the existence of each file through a 'while' loop before accessing their content.

I believe the most solid solution would be to create the two output files for all images (for instance rawthermalimage1.jpg; data_out1.txt; rawthermalimage2.jpg; data_out2.txt, ... etc) in one shot and then proceeding to analysis instead of going file by file, but this is not ideal considering how the program is built. I might look into it if it is worth the re-writing time. Anyhow thank you Phil, I am quite satisfied by the trimming I have achieved so far.

[Phil Harvey]

Yes, it could be a problem restraining matlab until the ExifTool output is complete.  To work around this, you could send the ExifTool output to a temporary directory then use ExifTool to move it to the matlab directory with a simple -directory=MATLABDIR FILE -execute.

- Phil

[nametyr]

Alright, so I decided to go ahead and rebuild most of the program knowing that the fastest and most reliable solution would be to dump the list of files beforehand and send two huge commands (one for raw thermal image and one for metadata). I also used a temporary folder inside the Matlab working repository as you suggested. All in all I managed to get the processing time well under 10 minutes, including lightning-quick 30 seconds ExifTool extraction (most of which is Matlab sleep time to improve reliability). My guess is that there probably are even faster and more logical solutions, but since the remaining post-processing time is still a whopping 400+ seconds I don't believe working around ExifTool is worth it anymore by now. Thank you for your tips Phil, they definitely helped me quite a lot on this one.

Edit: with further changes not linked to ExifTool, total processing time is now down to 5 minutes for 181 images. Mission successful!

[Phil Harvey]

Excellent!