"No writable tags set" results in those files being copied to current dir?

Started by jasauders, January 08, 2018, 12:07:01 AM

Previous topic - Next topic

jasauders

Hi friends. I noticed some behavior that threw me off a bit this evening. Two of my dash cam videos are citing that they have no writable tags set. I'm using CreateDate to sort, however CreateDate doesn't exist with these two files in particular. That's not the concern, as I know why these two files have no writable tags. The thing is, I've ran into this before with totally different files. At the time I choked it up to assuming I messed something up when running exiftool and that it was user error. Now that I was able to replicate this I wanted to ask here for some insight.

The two files in question that cite no writable tags end up getting copied to my current directory. In the below text, you can see I run an ls and the files are not present. Yet once I run the script and see the two errors for no writable tags being present, then after run ls again, you can see the two files in my current directory. Likewise, at the end, you can see that the files still reside in their original directory too (thus suggesting they were copied and not moved).

Side note: I seem to think this may be partly why I added "exiftool -overwrite_original_in_place -P -if 'not $CreateDate' '-CreateDate<FileModifyDate' -r" to all of my exiftool commands (excluding dash cam since this is pretty much a non-issue with this particular device). With the above command, if CreateDate does not exist, it generates CreateDate based on FileModifyDate, thereby ensuring that CreateDate exists for the very next command which uses CreateDate for sorting.

But again, I have no idea why it is copying files to my currently working directory. When I launched the script when cd'd into /usr/local/bin, it placed two copies of these files in /usr/local/bin. Below I execute it from my administrator's home directory and likewise, it created two copies of the video files in my home directory too.

The sort-pictures-all.sh script, specifically for the Dash Cam section, is simply executing this command:
exiftool -o . '-Directory<CreateDate' -d /mnt/vault/pictures/%Y/%m -r /mnt/vault/unsorted_pictures/staging/dash_cam

The full output:

administrator@vault:~$ ls
dead.letter  speedtest

administrator@vault:~$ bash /usr/local/bin/sort-pictures-all.sh
Sorting Files...
Warning: No writable tags set from /mnt/vault/unsorted_pictures/staging/dash_cam/2018_0107_205530_246.MOV
Warning: No writable tags set from /mnt/vault/unsorted_pictures/staging/dash_cam/2018_0107_210832_246.MOV
    1 directories scanned
    0 image files updated
    2 image files copied
    2 files weren't updated due to errors
Sorting Complete

administrator@vault:~$ ls
2018_0107_205530_246.MOV  2018_0107_210832_246.MOV  dead.letter  speedtest


Can anybody give me some insight as to why this might be happening?

Thanks all!

EDIT - Looking at things more closely, it almost looks like it's just a side effect of me using -o to copy the files instead of move. The question is, I really don't understand why when trying to "exiftool copy" from /mnt/vault/unsorted_pictures to /mnt/vault/pictures it ends up placing the files in /home/administrator upon hitting that error. When I remove -o and run the script, it simply fails and does not exhibit the behavior above. But like I said, how it ends up placing a copy of the files within my currently-working-directory is the part I'm not understanding. Anyway, if anyone has any insight, I'm all ears. :)

StarGeek

You can find part of the explanation of what is happening in FileName and Directory tags Example 5.

Normally, the -o . would copy the file to the current directory, as the dot represents the current directory.  This would be overridden by the '-Directory<CreateDate' argument.  Except in the case of when there's not CreateDate, it doesn't get overridden.

You might want to make sure that a CreateDate is generated before this command is run or go back to the -if 'not $CreateDate' argument.  There might be a better solution in the forum history.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

jasauders

Hey StarGeek. This particular scenario came up totally on accident. My dash cam always writes CreateDate so this is the *one* device in my script I skipped using the -if not CreateDate line, as it's simply not needed. What got me is I forgot I was testing two dash cam files where I nuked the exif data and forgot to nuke the files when I was done. So when this came up, I said "ah ha!" since I was able to see this situation happen again and took the opportunity to post and ask about it.

I can kind of follow what's happening here. I read someone's exiftool cheatsheet on github which in particular noted:

"-o ~/dummy -- This flag is to copy, not move. The directory is a fallback if the flag isn't available on the given photo. Good if using something like DateTimeOriginal"

I guess my question would be... why? Is there a technical reason why exiftool needs a directory to be set when using -o? I guess I'm not seeing the connection as to why I can't do exiftool -o, and if the parameter isn't set, it just simply fail instead of having to copy the file to the listed directory (which in my case being . is whichever directory I'm cd'd into when I ran the script/command)

Thanks again for the insight. :)

EDIT - side note... I wonder where these files would go if they are executed by root via crontab...
EDIT II - Looks like it goes to the /root/ directory. Heh. Imagine that. :P

StarGeek

Quote from: jasauders on January 08, 2018, 01:37:50 AM
I guess my question would be... why? Is there a technical reason why exiftool needs a directory to be set when using -o? I guess I'm not seeing the connection as to why I can't do exiftool -o, and if the parameter isn't set, it just simply fail instead of having to copy the file to the listed directory

How does exiftool decide if a parameter is not set?  It's a fairly common mistake in these forums for someone who wants to just copy files to forget the target of the copy.  The -o grabs the next argument and would use that.  Who's to say that I don't want a new directory named "-Keywords"?  You'd have to do something to change the way exiftool reads in the next argument to that option.  Do you change that operation for just this option, making it work differently than the rest of the options in exiftool?  Or change all the options, thus breaking most every script that calls exiftool?

The other choice would be making another option to cover just this situation and Phil has shown in the the past that he's reluctant to add new options when current ones work.  The docs are so extensive as it is.

Of course, that's just my thoughts on the subject.  Phil might weigh in with his own opinion.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Phil Harvey

Stargeek is right.  The -o option requires a parameter, and unless it is the last option on the command line, the option after this would be interpreted as a directory name if the parameter is omitted.

It would be easier to understand if I had added a new option like -copy or something when you wanted to copy a file, but since the -o option already results in copying the file, a new option wasn't required.  And as StarGeek mentions, I try to avoid adding new options unless absolutely necessary.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

jasauders

Thanks for the info! I guess in reality, issuing exiftool -o . doesn't result in any activity happening to the . directory. What I mean is, because of the way the flags are set, I must provide a path for -o to work, but if I'm copying files from sourcedir to destinationdir and all of those files have the parameter set, what is happening to . ? If I could watch the directory in real time, is it writing tmp files to . ? Or is it simply seeing . then instantly skipping it once it successfully detects the parameter and executes the copy action?

I wonder if it might be better to direct failed files to a specific directory, just so I have an easier way to monitor it should it ever surface again. Vault itself is a network share, so it'd be easy to run across if I leveraged a sub directory inside vault. Thinking something like:

exiftool -o /mnt/vault/unsorted_pictures/failed '-Directory<CreateDate' -d /mnt/vault/pictures/%Y/%m -r /mnt/vault/unsorted_pictures/staging/dash_cam

That should work, no?

Phil Harvey

Quote from: jasauders on January 08, 2018, 09:23:28 AM
if I'm copying files from sourcedir to destinationdir and all of those files have the parameter set, what is happening to . ?

Nothing will be written to '.' in this command.

QuoteI wonder if it might be better to direct failed files to a specific directory

Yes.  That is another very useful feature of the -o option.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

jasauders

As always, thanks Phil. Exiftool can be quite the rabbit hole to travel down (in a fun but surprising way). I started with one command and figured I'd be done once I got that ironed out, but after learning how all of my devices record date/time differently it's taken me up to about 14 commands now in my script. There's 7 different sections, with each section targeting a different directory (where each directory is for a specific device), allowing for customizations for each device/staging directory that are necessary. Some require overriding CreateDate with FileModifyDate because they recorded it incorrectly, some require -api QuickTimeUTC so the date/time is read accurately for sorting purposes, etc.

In short, I'm finding it very convenient to just split off the pictures/videos from different devices into their own directories to initialize the sort. That makes it super easy to let exiftool give the device-directories individualized attention, accurate to whatever their standard or non-standard date/time recording practices may be. It's all going to end up in the same directory in the end -- it's just the start of the process that is handy to have split up for control.

I'm actually finding the use of copying problematic files to a sort_failed directory to be quite useful already. Gives me an easy-to-find visual of the problematic files so I can make a call on what to do next. I have another directory where the script will take everything inside that directory and will override CreateDate with FileModifyDate, regardless of what the original CreateDate is. I noticed after I made a home movie in kdenlive and saw the CreateDate was just.... absolutely wrong. Easier to just drop the rendered file in that override_createdate directory and let the exiftool script sort it in seconds.

Anyway, felt compelled to share a bit. As always, thanks for the continued insight. :)

Phil Harvey

I can empathize.  Doing something seemingly simple can quickly get very complicated because there are so many variations in the way that metadata is written by different devices or to different file formats.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

jasauders

I hear you. When you really dive into it, it's shocking how different date/time stamps are. I mean, I've seen anything from 1903 to 2018, so... kind of a big gap until you figure out how to interface with exiftool to rectify those things.

Things got a little crazy pretty quick when I started tinkering around, but in reality, if you look at exiftool objectively, it's not that difficult once you see the completed syntax of the commands. I'm literally just issuing 1, at most 2 commands per device directory. It's keeping separate directories for each device that was the part that saved my sanity and avoided unnecessary complexity. And all I did to figure out what options I needed was to take a picture + video of each device, then read that exif data, then look at the current time of my computer to compare. That's how I identified some devices I have needed -api QuickTimeUTC while some others did not (granted I didn't know -api QuickTimeUTC existed until I asked here, but once known it was easy mode). But hey, it all works harmonically in the end. One script to punch through all the steps on a server with no GUI. Huzzah.

The only thing I'm somewhat confused over is how some GUI applications handle sorting. I assume they must look at CreateDate as primary, and if that's not available, then run with FileModifyDate as the identifier. I know I've seen some GUI's bring some photos over with ridiculous date/times from decades ago where I had really off-the-wall CreateDates set, so that's not even to suggest that GUI apps have any sort of advantage, but given their inner workings are a bit "masked" by the graphical interface (whereas I can see via the command exactly what exiftool is doing), it makes me wonder. Anyway, just a random day dream thought.

StarGeek

Quote from: jasauders on January 08, 2018, 01:05:45 PM
The only thing I'm somewhat confused over is how some GUI applications handle sorting. I assume they must look at CreateDate as primary, and if that's not available, then run with FileModifyDate as the identifier.

Actually, in my experience, DateTimeOriginal is usually the primary date/time stamp, at least in the case of photos (since videos don't usually have such a tag).  An extreme example is Google Photos, which looks at 12 different tags (see this post for list) to decide the date for an image.  But just about every program does something different.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

jasauders

Quote from: StarGeek on January 08, 2018, 04:26:17 PM
Quote from: jasauders on January 08, 2018, 01:05:45 PM
The only thing I'm somewhat confused over is how some GUI applications handle sorting. I assume they must look at CreateDate as primary, and if that's not available, then run with FileModifyDate as the identifier.

Actually, in my experience, DateTimeOriginal is usually the primary date/time stamp, at least in the case of photos (since videos don't usually have such a tag).  An extreme example is Google Photos, which looks at 12 different tags (see this post for list) to decide the date for an image.  But just about every program does something different.

Interesting. I wonder what they do for videos then, as I've had some photo managers that populate videos as well that just launch your default video player when you open them. I suppose they have a list of priority tags to tackle and probably go down the list until they snag a hit. In my far-from-expert-opinion on this subject, I think it would make sense to target the approach like that. Maybe someday I'll look into photo managers again. I've had some issues with different file managers with having all of my data stored on my file server. Even tried different protocols, i.e. samba/cifs/nfs/sshfs/etc. Some try to refresh the database on every launch, others don't do that but have other weird hurdles, etc. Trusty file manager over the wire never let me down though! Plus exiftool's sorting management makes the need for a local GUI photo manager kind of null. :D

I'm sure Google Photos and Google Hangouts may very well be unbolted from one another in this regard, but the whole issue of me wanting to figure out the "if CreateDate is not set, generate a CreateDate parameter based on FileModifyDate" issue originated from photos saved in Hangouts conversations. When you save photos from Hangouts, all you get is FileModifyDate, FileAccessDate, FileInodeChangeDate from -time:all. It's not a constant thing that I save photos from Hangouts, but more than once my parents or in-laws have caught good shots of my kids that I wanted to keep (which lead to me discovering the lack of metadata when I saw they weren't being sorted). I'm sure Hangouts isn't alone in this regard, which makes me feel better about passing the "-if not CreateDate create from FileModifyDate etc" line. That little bit of logic may remove some yet-to-be-seen headaches. :)