I am calling the following in an NSOperation:
ExifTool* tool = new ExifTool([toolPath cStringUsingEncoding:NSUTF8StringEncoding]);
// read metadata from the file
TagInfo* info = tool->ImageInfo([[[self element] path] cStringUsingEncoding:NSUTF8StringEncoding]);
if (info)
{
// do stuff
delete info;
}
delete tool;
It works well with a queue running 4 operations simultaneously and 50 or so in the queue.
The problem arises when I try to cancel and then requeue the same file for gathering metadata. The NSOperation does not stop immediately because the ExifTool is running. If I create a new NSOperationQueue (such as in a new document), then enqueue the same file to be processed by ExifTool, it sometimes hangs.
Could there be an odd interaction with the monitoring process that ExifTool uses? Should two ExifTool objects be able to retrieve meta data from the same file at the same time?
Absolutely, two ExifTool processes should be able to read from the same file.
However, you should note that deleting the ExifTool object doesn't happen instantaneously. It waits until the exiftool process terminates. I don't know if this could cause problems inside your Cocoa NSOperation.
- Phil
Thanks for this.
ExifTool::sNoSigPipe = 1; // this seems to help a lot
ExifTool::sNoWatchdog = 1; // this seems to help slightly
There is no good error when it hangs and I have not been able to track down where the hang is, but it feels like a race condition. I am only using on ExifTool per thread (per NSOperation really)
Does "delete tool" (on the ExifTool object) block until it quits or does it happen asynchronously? I am assuming it blocks.
If setting sNoSigPipe helps, then you probably aren't letting the exiftool process terminate properly. (ie. pulling the plug before it is done)
Quote from: DesertNomad on October 04, 2018, 02:39:43 PM
Does "delete tool" (on the ExifTool object) block until it quits or does it happen asynchronously? I am assuming it blocks.
Yes, it blocks.
- Phil
I'm pretty sure it is done correctly. This is the main function of an NSOperation.
-(void)main
{
if (![self isCancelled])
{
NSMutableArray* metadata = [[[NSMutableArray alloc] init] autorelease];
ExifTool* tool = new ExifTool([[self toolPath] cStringUsingEncoding:NSUTF8StringEncoding]);
TagInfo* info = tool->ImageInfo([[[self mediaObject] path] cStringUsingEncoding:NSUTF8StringEncoding]);
if (info)
{
for (TagInfo* i = info; i; i = i->next)
{
NSMutableDictionary* dict = [[[NSMutableDictionary alloc] init] autorelease];
[dict setValue:@(i->name) forKey:@"Name"];
[dict setValue:@(i->value) forKey:@"Value"];
[metadata addObject:dict];
}
// we are responsible for deleting the information when done
delete info;
}
// we are responsible for deleting the information when done
delete tool;
// deliver our completed data on the main queue
if (![self isCancelled])
{
dispatch_async(dispatch_get_main_queue(), ^{
[[self delegate] didFinishOperationForMediaObject:[self mediaObject] metadata:metadata];
});
}
}
}
I'm not much of a Cocoa programmer, but shouldn't you be creating an autorelease pool for this thread?
- Phil
NSOperation provides its own autorelease pool in this case.
OK. So that isn't the problem then.
I see nothing obviously wrong with your code. And while it should be good for stress testing, and I would like to figure out why this is hanging, it isn't the way that the C++ ExifTool object is meant to be used. The idea is to create the ExifTool object in advance and keep it running to avoid the lag due to the startup time.
- Phil
What would be he best way to do that?
The docs say that I should only have one ExifTool object per thread and I have one thread per file. To process 100 files, I create an NSOperationQueue which I limit to 4 operations at the same time and then queue up all 100 operations. They get fed through the queue and each one performs the above code. The threads are created by the NSOperationQueue as needed (but only 4 at a time).
I use this technique to extract images and file properties as well so I'm just trying to add ExifTool into the mix.
I'm not sure if the hang is related to tearing down the tool or trying to have too many going as I have tried dropping the same set of 100 files into two different documents (each of which has their own operation queue).
It's certainly a good stress-case.
Some more data below. This will sometimes hang because it can't create the tool object. Any idea if trying to create tool objects too quickly might cause this? I don't really see where or how the ExifTool constructor is returning a value to the caller, so not sure where this is failing.
Application Specific Information:
*** multi-threaded process forked ***
crashed on child side of fork pre-exec
*** error for object 0x7fa98d045800: pointer being freed was not allocated
Thread 0 Crashed:: Dispatch queue: NSOperationQueue 0x7fa98a544fa0 :: NSOperation 0x7fa98a6540d0 (QOS: USER_INITIATED)
0 libsystem_kernel.dylib 0x00007fff8958af06 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff9009c4ec pthread_kill + 90
2 libsystem_c.dylib 0x00007fff8cb586df abort + 129
3 libsystem_malloc.dylib 0x00007fff9a5ed041 free + 425
4 libsystem_c.dylib 0x00007fff8cb59463 __cxa_finalize_ranges + 345
5 libsystem_c.dylib 0x00007fff8cb59767 exit + 55
6 com.myapp 0x0000000108163909 ExifTool::ExifTool(char const*, char const*) + 1369 (ExifTool.cpp:206)
7 com.myapp 0x00000001081639b5 ExifTool::ExifTool(char const*, char const*) + 37 (ExifTool.cpp:211)
8 com.myapp 0x00000001081d526c -[MetadataOperation main] + 300 (MDPropertiesOperation.mm:59)
9 com.apple.Foundation 0x00007fff8a18ea5a -[__NSOperationInternal _start:] + 654
10 com.apple.Foundation 0x00007fff8a18aa44 __NSOQSchedule_f + 194
11 libdispatch.dylib 0x00007fff9567440b _dispatch_client_callout + 8
12 libdispatch.dylib 0x00007fff9567903b _dispatch_queue_drain + 754
13 libdispatch.dylib 0x00007fff9567f707 _dispatch_queue_invoke + 549
14 libdispatch.dylib 0x00007fff95677d53 _dispatch_root_queue_drain + 538
15 libdispatch.dylib 0x00007fff95677b00 _dispatch_worker_thread3 + 91
16 libsystem_pthread.dylib 0x00007fff900994de _pthread_wqthread + 1129
17 libsystem_pthread.dylib 0x00007fff90097341 start_wqthread + 13
-(void)main
{
if (![self isCancelled])
{
NSMutableArray* metadata = [[[NSMutableArray alloc] init] autorelease];
ExifTool* tool = new ExifTool([[self toolPath] cStringUsingEncoding:NSUTF8StringEncoding]);
delete tool;
// deliver our completed properties data on the main queue
if (![self isCancelled])
{
dispatch_async(dispatch_get_main_queue(), ^{
[[self delegate] didFinishOperationForMediaObject:[self mediaObject] options:[self options] metadata:metadata];
});
}
}
}
Responding to your previous post...
First I would create 4 NSOperation tasks that each create an ExifTool object then go into a sleep (wait) state.
Then I would have a shared queue to which you push the file names and delegates to be messaged after the file has been processed. The NSOperation task should wake up when a new filename is pushed into the queue. Each NSOperation then tries to grab the next filename/delegate from the queue. If successful, it runs ImageInfo on the file and builds the metadata object, then sends a message to the delegate. The NSOperation task should then loop back to grab another file from the queue, and go to sleep if there isn't one available.
This technique should be up to 60x faster than the way you are doing it now.
- Phil
Edit: You may see some additional performance benefits by letting the ExifTool object queue the commands, but doing this may be a bit more difficult. If you don't do this, you may want to decrease the wait time in ExifTool::Complete from 1000 microseconds (maybe I should make this configurable).
Edit2: I have just released a new version of cpp_exiftool that allows you to configure this wait time.
I wonder if the delete process is what is causing this?
When I delete the tool object, the NSOperation ends immediately thereafter and the thread dies. is there a good way to tell when the tool has actually been deleted so that I could keep my NSOperation thread around until then? You said the delete tool would block until it is really gone but I'm wondering if killing the thread too soon after is causing some issue.
This would affect things even if I were to somehow sleep the NSOperation threads (they aren't really designed to do that) so that the tool didn't have to relaunch for each file.
Could the watchdog process somehow get messed up by my thread (and not the whole app) ending? I've never really run into this before with NSOperation.
Looking closely at the Exiftool destructor, I found a possible problem because I wasn't setting mCmdQueue to NULL after deleting it. I doubt this is related to the problem you see, but you might want to grab the new version of 1.06 just in case.
- Phil
Looking back at your debug stack trace, the constructor failed when the watchdog process exits. Odd. Cocoa is catching the exit of the forked process, which is bad. So disabling the watchdog will solve this one. What happens when you set sNoWatchdog in this test?
- Phil
BTW,
Quote from: DesertNomad on October 05, 2018, 08:19:13 AM
The docs say that I should only have one ExifTool object per thread
Where did you see this? You should be able to have any number of ExifTool objects per thread. The limitation is that you may not access a single ExifTool object asynchronously from multiple threads.
- Phil
Yes, I was trying to say that I can't access the ExifTool object from multiple threads. I'll try 1.06 and disable the watchdog.
I've been playing around with this myself using a basic Cocoa project with the attached main.mm
I don't get any crashes, but always it seems that 1 operation never completes.
- Phil
Edit: Debugging this, I don't think that exiftool is actually running at all, so I have done something wrong. The path to ExifTool was wrong, but if I set it correctly it does the same thing.
Some odd is happening in Cocoa. If I compile with gcc and run this code as a C++ main
int main(int argc, char **argv)
{
printf("Created!\n");
ExifTool::sNoWatchdog = 1;
ExifTool* tool = new ExifTool("/usr/bin/perl","/Users/phil/bin/exiftool");
printf("Is Running before: %d\n",tool->IsRunning());
TagInfo *info = tool->ImageInfo("/Users/phil/source/exiftool_cvs/t/images/ExifTool.jpg");
printf(" %s\n", info ? "good" : "BAD!");
if (not info) {
printf(" LastComplete = %d\n",tool->LastComplete());
char *err = tool->GetError();
if (err) printf(" ERROR %s\n",err);
}
printf("Is Running after: %d\n",tool->IsRunning());
delete tool;
}
it gives this output:
Created!
Is Running before: 1
good
Is Running after: 1
but if I compile in XCode as a .mm main function (no NSOperations at all), it gives this:
Created!
Is Running before: 1
BAD!
LastComplete = -1
Is Running after: 0
That's about as far as I can go now because I can't even get this working in Cocoa without the NSOperation complication. And I've run out of time today to look at this.
- Phil
It might not like how you are creating the path in the NSOperation. I was using:
NSString* toolPath = [[[NSBundle mainBundle] resourcePath] stringByAppendingPathComponent:@"/exiftool/exiftool"];
ExifTool* tool = new ExifTool([toolPath cStringUsingEncoding:NSUTF8StringEncoding]);
in dealloc, you should be calling [super dealloc];
Maybe this would be easier from an NSTask running the command line tool. Is there a way to get the same data as exists in a TagInfo struct in a clean (json for example) way from the command line?
I can't seem to get group 0, 1 and 2 at the same time using the terminal in the same way that I can using the C++ TagInfo struct. When I use the NSTask based command line, I don't get a hang, but the data returned has less detail than I can get with the C++ method.
With a TagInfo struct I can get:
Type(g0) = XMP;
Location(g1) = "XMP-dc";
Category(g2) = Image;
Description = Subject;
Name = Subject;
NumberValue = "KeywordA, KeywordB";
StringValue = "KeywordA, KeywordB";
With the terminal I can only see one of the group values.
UPDATE: It is easy to get the hang when using the C++ method while running from Xcode (and thus with a debugger active). In the release build, I have not yet seen the hang. Could this be some interaction with the debugger? When the hang happens, it is my app that has the spinning beech ball, not Xcode. Such a mystery as during the hang I can't break into the debugger and see anything useful.
Ah. It could be that the debugger is trapping signals that are necessary for the ExifTool processes (like the SIGINT that is used to kill the watchdog process).
From the command line, use -php -l -G:0:1:2:4 -D -sep ", " to get the output that the ExifTool object uses. This will give you all three groups plus the copy number.
- Phil
That works well enough to parse... just looks likes I have to break out the Groups by the colon separator.
Is there a way to get a json output with those already split?
The debug version hangs even if I launch it without the debugger running, but the release build seems to work fine (so far). I wonder if there are hooks in the debug build that interfere with it?
Quote from: DesertNomad on October 06, 2018, 02:58:11 PM
Is there a way to get a json output with those already split?
No. You've got to split them yourself.
QuoteThe debug version hangs even if I launch it without the debugger running, but the release build seems to work fine (so far). I wonder if there are hooks in the debug build that interfere with it?
Sounds like there is, but maybe someone on a Cocoa forum knows about this. It would be good to know so I could add some tips to the documentation for Cocoa programmers.
- Phil
I'll let you know if I find out anything more about the hang.
As to splitting the groups, I assume that a colon can't appear in the group names so is safe to use to split things.
Yes. Group and tag names may contain only A-Z, a-z, 0-9, - and _.
- Phil