pipe to exiftool

[Archive]

[Originally posted by bozi on 2006-06-09 16:33:17-07]

hi,
I used exiftool and wget together with a pipe and it works well with an older version (6.10 ?).
Exiftool has decoded the data asap (after about 10-20kbyte fetched). I remember that in the this old version the -htmlDump parameter didnt worked.
With newer versions exiftool decodes the data only after the whole file is fetched.

wget -qO - http://....bigfile.jpg | exiftool - -p ~/tpl/e.tpl

when i fetch the file separately and abort the transfer in the beginning then exiftool decodes the saved part.

Whats the matter with the pipe handling in exiftool?

bozi

[Archive]

[Originally posted by exiftool on 2006-06-09 17:12:16-07]

Hi Bozi,

This is because ExifTool now checks for an AFCP trailer containing IPTC information.  ExifTool would also scan to the end of a JPEG if the meta information indicates there might be a PreviewImage tacked on the end, so you might notice this behaviour even with the older version for some images.

The only way around this would be for me to add an option to ignore JPEG trailers.  I will consider adding this if it is important enough to you.

[Archive]

[Originally posted by exiftool on 2006-06-09 18:06:13-07]

Well, I decided that other people may benefit from this too, so I've added a "-fast" option and released ExifTool 6.24 with this new feature.

With the -fast option, ExifTool won't check for an AFCP or PreviewImage trailer on JPEG images.

And what's this about the -htmlDump option not working?  I thought you meant that it didn't exist in your version, but it should work fine from version 6.00 onward.  Let me know if you have any problems with it.

[Archive]

[Originally posted by bozi on 2006-06-09 18:14:15-07]

Hi,

Thnx for the quick reply. An option to ignore JPEG trailers would be great when thats the only reason for the slow decoding / waiting for the EOF respectively .

bozi

[Archive]

[Originally posted by bozi on 2006-06-09 19:17:23-07]

The -htmlDump option works fine in the last versions.

Only in the old version (i tried months ago) it doesnt seems to work. But this (old) version didnt scan the trailer and has the fast pipe handling (i think).

thnx, bozi

[Archive]

[Originally posted by exiftool on 2006-06-10 12:17:33-07]

I had another idea.  Currently, ExifTool reads the image in blocks which are a multiple of 64kB and buffers them in memory when reading from a pipe.  This buffering is done in File::RandomAccess.pm, and the minimum block size is set by the variable $CHUNK_SIZE in that file.  If you want to tweak the performance you could play with this block size.  I suspect 4k, 8k or 16k might be optimal if piping across slow network connections.  The best value would depend on the average size of the EXIF data in your images -- a block size just larger than this may be best.

Let me know if this improves performance for you.  The 64k block size was rather arbitrary, and a smaller value might be best.  I will do some testing myself.  Since the buffering is not done when reading directly from a file (only pipes are buffered), we are free to play with the buffer size without affecting the processing speed for most users.

[Archive]

[Originally posted by bozi on 2006-06-13 16:24:50-07]

Phil, I played with 4k $CHUNK_SIZE ... here the results:

64k ..........: 5.544s

-fast and 64k : 1.221s

4k............: 5.339s

-fast and 4k .: 0.958s

thnx for the tip with chunk_size that speeds up the whole process additionally with 22%.

bozi

[Archive]

[Originally posted by exiftool on 2006-06-13 16:38:52-07]

Thanks for doing these tests.  I have also done some testing myself with piped files from disk using a buffer size of 8k.  For my tests there was no noticable difference in speed, so I will lower the $CHUNK_SIZE to 8k for the next release.  This should provide you with most of the speed benefits you get using 4k blocks.  I felt a little uneasy going less than 8k due to the potential of increasing i/o overhead on systems with slow CPUs, so 8k sounded like a good compromise.

- Phil

[Archive]

[Originally posted by bozi on 2006-06-13 17:59:36-07]

what about linking the $CHUNK_SIZE with the -fast parameter?

-fast decrease the size to 4k an without -fast let it 64k?

bozi

[Archive]

[Originally posted by exiftool on 2006-06-13 18:09:10-07]

Interesting idea, but I think that just setting the buffer size to 8k is sufficient.  This size was never really optimized when I decided on 64k originally, and now that we've done this testing I'm happy with 8k.  However, your idea remains a possibility for the future if required.