Is there an undocumented limit on the number of exifTool calls?

[MikeFromMesa]

I keep a large (currently > 2600) library of photos from trips and use them as wallpaper on my Mac. I have tried to add the month and year to the photo using Photoshop or other photo editors, but noticed that some of those dates are too small or the wrong color to be easily seen.

Rather than go through all of those photos trying to find the correct one I wrote a Cocoa application, using Swift 5 and exifTool, to search through the library and return all photos matching the year and month that I am looking for so I can correct the display date. When I first ran the app it marked almost 100 photos as not having a valid CreateDate entry, so I copied them out to check. When I did that I found that they all did have a valid CreateDate, but that the calls in my code starting failing with the 2544th call. If I copied enough images out of my wallpaper folder I no longer got the failures. If I copied enough back in to reach 2544, then that image, and every subsequent image, failed to return a valid CreateDate data line.

So, is there some undocumented exifTool usage limit that I am unaware of? Is there some way to release the tool during processing so I don't run into this issue? Or is there some other issue involved here? As far as I can tell I am not running out of memory and the code works perfectly as long as I keep under 2544 images.

As I mentioned, I am not running exifTool from a script, but from a full blown MacOS Cocoa application using Swift5 and created using Apple's Xcode. Thanks for any help.

[StarGeek]

So, is there some undocumented exifTool usage limit that I am unaware of?

No, I have run exiftool directly on over 100,000 files in one command several times.  But since you are looking a specific date, you might try running exiftool directly to list all the files.  Something like this, which will list all files with a CreateDate between June 2022 and January 2023
exiftool -if '$CreateDate ge "2022:06" and $CreateDate le "2023:01"' -Filepath /path/to/files/

Is there some way to release the tool during processing so I don't run into this issue? Or is there some other issue involved here? As far as I can tell I am not running out of memory and the code works perfectly as long as I keep under 2544 images.

As I mentioned, I am not running exifTool from a script, but from a full blown MacOS Cocoa application using Swift5 and created using Apple's Xcode. Thanks for any help.

I'm am not a Mac user so I don't have any knowledge of what might be happening with those apps. But the problem wouldn't be with exiftool.  Hopefully someone with more Mac knowledge can help.

[greybeard]

As StarGreek says it is rare that you can't come up with a command line combination that will solve this kind of search problem (and will likely be much faster than a custom application).

If there is something specific that requires an application - is it a specific file that is causing the problem (are you sure it is the number of files)? Does the application actually fail?

Maybe you can share the code you are using? Its tough to debug without knowing exactly what you are doing.

[MikeFromMesa]

As StarGreek says it is rare that you can't come up with a command line combination that will solve this kind of search problem (and will likely be much faster than a custom application).

I do all of my photo work on my Mac, including photo editing, and when I have issues like trying to find specific photos I generally write an app to solve the problem. Not because I can't write a script, but more to keep myself busy now that I am retired. I actually enjoy writing software and trying to keep current.

If there is something specific that requires an application - is it a specific file that is causing the problem (are you sure it is the number of files)? Does the application actually fail?

No. There is no specific file that is causing the issue. As part of my code I get all of the file listings in an array, sort it and then process through them in sorted order. I always run into issues at file number 2544. If I remove some earlier images from the folder and then run the app I still run into the issue at image 2544 even though it is now a completely different image.

As part of this I copied all of the supposed bad images to a separate folder (there were 97 of them) and then ran the same app on them, but without any problems. I am going to try something different this morning. I will create and release the swift library object that I use and see if that makes any difference. And I will probably sort the array in backwards order to make sure I see the same thing at item 2544.

Maybe you can share the code you are using? Its tough to debug without knowing exactly what you are doing.

Sure. Here is the library routine that calls exifTool. I am not sure how much help it is since you can not see the params that are used, but they conform to the exifTool specs and work properly for the first 2543 calls.  And, of course, it requires that you know Swift.

I will post again after I do some additional testing and let you know what, if anything, happened differently.

  //  ************************************  //
  //  **  make the call of exifTool to  **  //
  //  **  get the date information      **  //
  //  ************************************  //
  func makeExifCall(params: Array<String>) -> (Bool, String)  {
    let task = Process();
    let pipe = Pipe();
    var data = Data();
    let dataErr = Data();
    var rtn = (false, "");
   
    //  set up call parameters
    task.standardOutput = pipe;
    task.arguments = params;
    task.executableURL = URL(fileURLWithPath: exifPath);
   
    //  run the task
    do  {
      try task.run();
      task.waitUntilExit();
      data = try pipe.fileHandleForReading.readToEnd() ?? dataErr;
    }
    catch  {
      rtn.1 = "\(error)";
      return rtn;
    }

    //  return the raw data as string
    rtn.0 = true;
    rtn.1 = String(data: data, encoding: .utf8)!;
    return rtn;
  }

Just to be clear. The error occurs in the do-catch block. The call is made to exifTool and the block catches an exception when it (exifTool) returns a null value. There are three parameters for the call and they are as follows:

-CreateDate
-s
/Volumes/Photos/Wallpaper/P5130440.jpg

One thing I learned early on when writing software is that anything is possible, so perhaps there is some error in the code here, but I don't think so.

First, I can not find anything that seems like a coding error.

Second, it all works perfectly for 2543 calls, and then fails with every call after that, and always with the same exception raised when trying to read the response from exifTool.

There might be some memory issue (although Xcode tells me I am using very little memory) or perhaps some pointer is stepping on another pointer and, once messed up, continues to error. But unless that comes from a coding error I can see and fix there is not much I can do to fix it.

I may rewrite this code using objective-c as that might tell me if there is some coding issue somewhere, but it won't help if there is some issue in the MacOS version of exifTool that others, using scripting on non-Mac machines, don't see.

[MikeFromMesa]

Well, that testing has been informative if not helpful.

1. Reverse the sorted order - get the same error after the same number of files (2544), just a different file name this time.

2. One added wrinkle is that the app is running multi-threaded so I made it single threaded and reran it. Same results. Error at file 2544.

3. Revised the app so that the library object, which contains the code that calls exifTool, was created every 100 calls rather than just using the same app. The idea was to see if there was something going on in the object that might be causing this problem, but the results were the same. Failure at item 2544.

I am not going to give up here. I am retired and have plenty of time  so I will continue experimenting, but at least now I know that there is no issue with the number of calls if I am using a script. I might do that as well, just to see, but it would not solve the problem of what is going on here.

Thanks for the answers to my questions.

[StarGeek]

As part of my code I get all of the file listings in an array, sort it and then process through them in sorted order.

I can't tell from the code as I'm not familiar with it, but are you running exiftool once per file?  If so, this is Common Mistake #3.  You say you create a list of files.  Instead of going through the list one by one, pass the entire list to exiftool and let exiftool process all the files in one go.

This post shows how to use -@ - to read from a pipe on the command line. I'm not sure how your program's pipe works.

Edit:  A long shot, but try adding -v0 to your command.  This tells exiftool to "flush the output buffer after each line" (see the -v (-verbose) option)

[greybeard]

So you are running that function once for each file?

My guess is that you have hit the macOS process limit

[MikeFromMesa]

So you are running that function once for each file?

Yes.

My guess is that you have hit the macOS process limit

Ah. Now that makes sense.

I was so concerned with something in the code that might have been the problem that I did not really address other limitations other than memory. It would explain everything, so I guess I need to do a bit more research and see what limits MacOS imposes.

Thanks for the suggestion.

[MikeFromMesa]

As part of my code I get all of the file listings in an array, sort it and then process through them in sorted order.

I can't tell from the code as I'm not familiar with it, but are you running exiftool once per file?  If so, this is Common Mistake #3.  You say you create a list of files.  Instead of going through the list one by one, pass the entire list to exiftool and let exiftool process all the files in one go.

This post shows how to use -@ - to read from a pipe on the command line. I'm not sure how your program's pipe works.

Edit:  A long shot, but try adding -v0 to your command.  This tells exiftool to "flush the output buffer after each line" (see the -v (-verbose) option)

Thanks. I will give these suggestions a try and see if they solve the problem. Right after I stop banging my head against the wall ...

[MikeFromMesa]

For what it is worth:

One of the reasons I used a one file call to exifTool was that some of the images I have are missing the internal date and time information and others have invalid values. That seems to have come from some of the photo editing tools that I used for panoramas and HDR shots that did not bother to include the date and time information in the outputs. That means that some of the returned data from exifTool was invalid and it was much, much easier to address those issues when there was only one image to fix at a time.

I have since converted to doubling up. That means that I now handle two images at a time to reduce the process count and have finally fixed the parsing code to fix the date and time issue with multiple images. Now that that is working I may next try adding all images at one time to the exifTool call and see how that works, although even doubling up has solved my process problem, so thanks for that.

[philbond87]

@MikeFromMesa,

Another thing you could do is spawn multiple instances of exiftool - each running in a different Process() – and run them concurrently on different threads.

(I do this in a few of my Swift apps to greatly speed up the time it takes to handle very large numbers of image files.)