Creation date on macOS (delay in mdls causing problems for MDItemFSCreationDate)

[drobin]

Hi Phil, I have an interesting problem for you, and hopefully a solution to offer.

I was getting a mysterious error complaining about date formats and I tracked it down to a delay in mdls after a file is created.  Apparently mdls is reading some information that is lazily updated by a fsevent listener or something like that. 

The bottom line is that after you create a file, reading -MDItemFSCreationDate with exiftool is not reliable for a few seconds afterwards, which was causing the mysterious complaints spewing to the console.

This is the "real world" function that was causing the problem.  If the first call to exiftool results in making a new file, then the second call fails with a complaint about date format because mdls is returning the literal string "(null)" to you and then you are trying to parse that as a date string.

-------------------- you don't really need to read this, a test function follows ----------------
exifEnsureDateTimeOriginal() {  # give list of files or directories
    # first, for movies especially, get DateTimeOriginal from CreationDate
    /usr/local/bin/exiftool -P -if 'not $DateTimeOriginal && $CreationDate' '-DateTimeOriginal<CreationDate' '-CreateDate<CreationDate' "$@"
    # then try from file creation date (if it's a sane value!)
    /usr/local/bin/exiftool -P -if 'not $DateTimeOriginal && ($MDItemFSCreationDate gt "1980:01:01 00:00:00")' '-DateTimeOriginal<MDItemFSCreationDate' '-CreateDate<MDItemFSCreationDate' "$@"
    # if that didn't do it, then set it from the file modification DATE
    /usr/local/bin/exiftool -P -if 'not $DateTimeOriginal' '-DateTimeOriginal<FileModifyDate' '-CreateDate<FileModifyDate' "$@"
}
------------------------------------------------------------------------

But being the programmer that I am, I thought I would track down the root cause before posting and whining here.  Your documentation very helpfully and thoroughly (thank you!) says that you use mdls to get the mac specific metadata, which pointed me in the right direction.  So here is a function to see mdls as the root of the problem:

------------------------------------------------------------------------
testMDItemFSCreationDate() { # give only one file
    while true; do
        rm "$1"_new
        cp "$1" "$1"_new
        sleep 1 # <--- make this 0 for mdls failure every time, 2 for never fail, and 1 for intermittent (on my machine, anyway)
        echo -n "stat:     "; stat -f '%SB' -t '%Y:%m:%d %H:%M:%S%z' "$1"_new  # stat always succeeds regardless of sleep
        echo -n "mdls:     "; mdls -name kMDItemFSCreationDate "$1"_new
        echo -n "exiftool: "; exiftool '-MDItemFSCreationDate' "$1"_new
        echo '======'
    done
}
------------------------------------------------------------------------

From this test, you can see that my suggested solution is to use 'stat' for MDItemFSCreationDate.  For the rest of the md stuff, you'll have to rely on mdls, but for the special case of MDItemFSCreationDate, a quick call to stat will get you a reliable answer every time.

Thanks for a great and powerful tool!
Dave

macOS 10.12.6
exiftool 10.90

=========================
below is the output of the test script running with various values for sleep.

------------ With No Sleep --------------

drobin-m9:~ drobin$ y testMDItemFSCreationDate /Users/drobin/Temp/temp/IMG_8918.MOV
stat:     2018:04:05 16:39:38-0400
mdls:     kMDItemFSCreationDate = (null)
exiftool: MD Item FS Creation Date        :
======
stat:     2018:04:05 16:39:38-0400
mdls:     kMDItemFSCreationDate = (null)
exiftool: MD Item FS Creation Date        :
======
stat:     2018:04:05 16:39:38-0400
mdls:     kMDItemFSCreationDate = (null)
exiftool: MD Item FS Creation Date        :
======
stat:     2018:04:05 16:39:39-0400
mdls:     kMDItemFSCreationDate = (null)
exiftool: MD Item FS Creation Date        :
======
^C

------------ With Sleep set to 1 --------------

drobin-m9:~ drobin$ y testMDItemFSCreationDate /Users/drobin/Temp/temp/IMG_8918.MOV
stat:     2018:04:05 16:38:33-0400
mdls:     kMDItemFSCreationDate = (null)
exiftool: MD Item FS Creation Date        :
======
stat:     2018:04:05 16:38:35-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:38:35 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:38:35-04:00
======
stat:     2018:04:05 16:38:36-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:38:36 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:38:36-04:00
======
stat:     2018:04:05 16:38:37-0400
mdls:     kMDItemFSCreationDate = (null)
exiftool: MD Item FS Creation Date        :
======
stat:     2018:04:05 16:38:38-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:38:38 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:38:38-04:00
======
stat:     2018:04:05 16:38:40-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:38:40 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:38:40-04:00
======
^C

------------ With Sleep set to 2 --------------

drobin-m9:~ drobin$ y testMDItemFSCreationDate /Users/drobin/Temp/temp/IMG_8918.MOV
stat:     2018:04:05 16:41:29-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:41:29 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:41:29-04:00
======
stat:     2018:04:05 16:41:31-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:41:31 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:41:31-04:00
======
stat:     2018:04:05 16:41:33-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:41:33 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:41:33-04:00
======
stat:     2018:04:05 16:41:36-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:41:36 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:41:36-04:00
======
stat:     2018:04:05 16:41:38-0400
mdls:     kMDItemFSCreationDate = 2018-04-05 20:41:38 +0000
exiftool: MD Item FS Creation Date        : 2018:04:05 16:41:38-04:00
======
^C

[Phil Harvey]

Hi Dave,

Thanks for this report and solution!! Excellent.  You're two steps ahead of me because I didn't even know about the delay in updating the mdls values, and I didn't know that stat would return the creation time.

I'll see about implementing this, and post back here if I have any questions.

- Phil

[Phil Harvey]

OK,

I think I'll leave the existing MDItemFSCreationDate alone, and add MacOS read/write support for the FileCreateDate tag.  Reading will use "stat", and writing will be the same as for MDItemFSCreationDate.

I'll also add a note in the tag name documentation that the readback value may be delayed by a few seconds for MDItemFSCreationDate.

- Phil

[drobin]

Direct support for FileCreateDate will be great.  I hated having the ugly "MDItemFSCreationDate" in my scripts because everything else I did was standard portable tags and not tied to macs, and I always worried that if I shared that part of my script library with a windows user, it would break.  Thanks again for all you do and your incredible responsiveness to the huge community you've created.

[Phil Harvey]

This was implemented in version 10.91 (Apr 9).

- Phil