Unable to flush stdout:

[jrjdavidson]

Hi,

I've written a python script to scan a drive for jpg images. It worked fine until recently, when it started throwing a flush stdout error. Possibly due to the number of photos? the script work fine on a smaller subset. Any help appreciated.

1) Windows 10

2)11.89

3) here is the relevant excerpt of the script:
ExifOutputFile = "out.csv"
ExifScanFolder = "V:"
FileType = "JPG"
os.system("exiftool -ext "+FileType+" -GPSLongitude -GPSLatitude -GPSAltitude -SourceFile -CreateDate -ImageSize -FileSize -Make -Model -csv -c '%.6f' -n "+ExifScanFolder+" > "+ExifOutputFile+" -r -progress")

4)
[...]
83594 directories scanned
307163 image files read
  176 files could not be read
Unable to flush stdout: Invalid argumentExiftool created file: out.csv
[Finished in 24548.2s]

[Phil Harvey]

Wow.  It is definitely not recommended to run the -csv option on such a large number of files.  (Read the -csv documenation.)

I'm guessing that the massive memory requirements of this technique may be contributing to the problem.

Could you use JSON or XML output format instead?

- Phil

[obetz]

307163 image files read
  176 files could not be read
Unable to flush stdout: Invalid argumentExiftool created file: out.csv
[Finished in 24548.2s]

uh, 80ms per image results in nearly seven hours to complete...

You might try my Strawberry Perl based ExifTool https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows although I think a memory problem in Perl would result in a different error message.

I tried a csv output on 12000 images and the ExifTool process used less than 30MB. Since it started at 24MB, I expect to stay way below the Perl memory limit (IIRC 256MB).

You could also try a 64 Bit Perl. This is somewhat slower but it might have higher memory limits. I can provide assistance if you want to do it.

By the way: My test processed the images much faster (30ms per image in the first run, 13ms per image in the second run) although they are on a 5400rpm HDD.

@Phil: Does the image size matter or does ExifTool read only the metadata portion?

Oliver

[obetz]

more thoughts and observations:

There is a slight possibility that the "PAR magic" has an influence on the redirected output handling. My Perl launcher runs Perl as simple and clean as possible.

The original ExifTool.exe needs noticeable more memory (>40MB vs. <30MB), but it runs 7% faster.

The CPU speed seems to be more important than HDD speed, but I'm not completely sure about this since the first run took much longer and I don't know whether this comes from Windows Defender or cached HDD data.

BTW: I have set no exclusions for ExifTool in Windows Defender (never needed it) and I work as a restricted user.

Oliver

[Phil Harvey]

@Phil: Does the image size matter or does ExifTool read only the metadata portion?

This depends on the specific file.  But for JPEG images, adding -fast means that only the metadata header is read.

- Phil

[jrjdavidson]

Thanks for the replies guys.

Apologies for this- clearly I hadn't read the documentation properly. Using JSON seems like an easy fix, I'll give t a try.

Thank you for the suggestions on speeding up processing, I'll look into this at a later stage but not too worried for now as it is a script that will only run once every few months..

Cheers,
Jonathan

[obetz]

...Perl memory limit (IIRC 256MB)...

I was wrong, according to https://www.perlmonks.org/?node_id=845951 there seems to be no intentional memory limit in Perl.

A set of 347362 images in 28703 directories (using junctions) was processed on my system in approximately 45 Minutes by original ExifTool as well as my ExifTool package, both using 400..480MB of memory. The resulting output file size is 47MB.

So I can't replicate the problem and I don't see a fundamental difference between both versions of ExifTool/Win.

Oliver