FileModifyDate reported differently on Linux and Windows

[Xlnt]

When looking at the FileModifyDate reported by ExifTool, I notice an offset of 1 hour from what I expect. At least for some files, not all. On Windows, not on Linux. Let me explain in detail.

On Windows, this is the output of the following commands:
> exiftool -s -FileModifyDate FILENAME
FileModifyDate                  : 2012:09:01 11:45:18+02:00
> stat -c %y FILENAME
2012-09-01 12:45:18.000000000 +0200

On Linux (same folder, mounted using cifs), the output is:
$ exiftool -s -FileModifyDate FILENAME
FileModifyDate                  : 2012:09:01 12:45:18+02:00
$ stat -c %y FILENAME
2012-09-01 12:45:18.000000000 +0200

As you can see, the FileModifyDate reported by ExifTool on Windows is different compared to ExifTool on Linux, or the output of stat. When digging a little further, I noticed this is only the case for files with a modification time that falls under daylight saving time. My 'normal' time zone offset is +01:00, the +02:00 results from an extra hour due to daylight saving time.

I believe the time reported by stat is correct, and the time reported by ExifTool on Windows is not. Especially, since ExifTool on Linux gives different output. However, I have no idea about the root cause, neither any means to debug it further. Can someone else comment on this?

The problem can be easily reproduced using (assuming local time zone has daylight saving time during September):
> touch -t 201209011245.18 test.xxx
> stat -c %y test.xxx
> exiftool -s -FileModifyDate test.xxx

Versions used:
Windows: 6.1.7601
Linux: debian 3.2.0-4-amd64
ExifTool (Windows): 9.82 (9.74 shows same behavior)
ExifTool (Linux): 9.74

[Phil Harvey]

I remember seeing this problem before.  There are definitely various time zone problems in Windows.  Especially if you are using a FAT32 filesystem, which I believe stores local time so the absolute file times change with the system settings.  (At least they do on my thumb drives.)

I would like some more details.  Does ExifTool agree with the date/time returned by the "dir" command?  If so, there is really nothing that I can do.  I seem to remember something about the current Windows time zone being applied to all times, regardless of whether or not DST is different, but I can't find a reference for that now.

- Phil

[Xlnt]

Indeed, dir shows the same output as ExifTool on Windows. It turns out to be a design choice of Microsoft for the implementation of the C library function stat.

However, I found a Perl module that seems to solve the problem:
Win32::UTCFileTime

The following 'quick and dirty' fix in ExifTool.pm (around line 1816) works for me (at least for the display of the FileModifyDate):

Code Select Expand


            my @stat;
            if ($^O eq 'MSWin32' and eval 'require Win32::UTCFileTime') {
              import Win32::UTCFileTime;
              @stat = Win32::UTCFileTime::stat $filename;
            }
            else {
              @stat = stat $filename;
            }
            $self->FoundTag('FileModifyDate',  $stat[9]) if defined $stat[9];


By the way, the other file properties (e.g. FileSize, FileAccesDate) can also be retrieved using stat. A patch file for ExifTool.pm is attached. Is there a specific reason stat is already used to retrieve FilePermissions, but not the other file properties?

I only tested reporting of the FileModifyDate, not updating it. According to the documentation of the module above, it should be possible to globally override stat, utime, etc. without having to modify code that uses stat etc. But I could not get it to work. Probably because I am not an experienced Perl programmer. Perhaps you can?

[Phil Harvey]

Thanks for the fix, but I'm loath to add another patch to work around a Windows bug.

And I think that your patch will break the recent Windows Unicode file name patch for the tags.

Also, this is only half of the story as you mentioned.  The writing side would also have to be patched.

- Phil

[Xlnt]

I spent some more time in working out a solution. First I looked at how file times are read/written. I believe only the following functions are used:

• Win32API::File::Time::GetFileTime: For getting times. Used on Windows, only if the package is available
• Win32API::File::Time::SetFileTime: For setting times. Used on Windows, only if the package is available
• file operators (-A -M -C): For getting times. Used on Linux, and as fallback on Windows when Win32API::File::Time is not available
• utime: For setting times. Used on Linux, and as fallback on Windows when Win32API::File::Time is not available

Note: stat is not used for getting times, only for permissions, uid, gid.

I used the following regex to find the places where they are used:

Code Select Expand

(::[SG]etFileTime|(?<![\$@])\b(stat|utime)\b|^(?!#).*\W-[MCA] )

Overriding utime globally was easy:

Code Select Expand

if ($^O eq 'MSWin32' and eval 'require Win32::UTCFileTime') {
    eval('import Win32::UTCFileTime');
    eval('*CORE::GLOBAL::stat  = \&Win32::UTCFileTime::stat');
    eval('*CORE::GLOBAL::lstat = \&Win32::UTCFileTime::lstat');
    eval('*CORE::GLOBAL::utime = \&Win32::UTCFileTime::utime');
}
Note: Eval is needed to prevent messages 'Name "CORE::GLOBAL::utime" used only once: possible typo at ...' (etc.) on Linux.

File operators are not (cannot be?) overridden by Win32::UTCFileTime package. However, the overridden stat can be used instead. By using stat, code gets even more readible because calculations like int($^T - (-A _) * (24 * 3600) + 0.5) are no longer needed. Two places in code are affected:

• In GetFileTime in Writer.pl, file operators are used in combination with a file name. Here the overridden stat can be used.
• In ExtractInfo in ExifTool.pm, file operators are used in combination with a file handle. Here the overridden stat cannot be used, because it doesn't support file handles. I replaced it by a call to $self->GetFileTime($filename), which already supports unicode file names and uses the overridden stat as fallback.

I moved the SetFileTime function from exiftool.pl to Writer.pl (there was already a GetFileTime function), to prevent code duplication and have only two places where Win32API::File::Time::GetFileTime/SetFileTime is used.

A patch file that accomplishes the above is attached. The behavior is then as follows:

• When Win32::UTCFileTime is not available, behavior is unchanged.
• When Win32::UTCFileTime is available:
  
  
  
  • When Win32API::File::Time is not available, file times are read/written taking into account daylight saving time (i.e. by the overridden stat/utime).
  • When Win32API::File::Time is available, behavior is unchanged (i.e. still not taking into account daylight saving time). This is because, because the package doesn't use stat/utime. See below.

After some debugging and consulting the source of the Win32API::File::Time package, I found out it is actually mimicking the behavior of the Windows stat function (i.e. not taking into account daylight saving):

Code Select Expand


# _filetime_to_perltime
#
# This subroutine takes as input a number of Windows file times
# and converts them to Perl times.
#
# The algorithm is due to the unsung heros at Hip Communications
# Inc (currently known as ActiveState), who found a way around
# the fact that Perl and Windows have a fundamentally different
# idea of what local time corresponds to a given GMT when summer
# time was in effect at the given GMT, but not at the time the
# conversion is made. The given algorithm is consistent with the
# results of the stat () function.

This is unfortunate, and actually very strange, because the Microsoft GetFileTime/SetFileTime functions work with file times which are UTC by default and therefore not affected by daylight saving time. If they were called directly (i.e. without conversion with FileTimeToLocalFileTime etc.), there would be no problem.

I tried to work around it by calling Win32API::File::Time::SetFileTime followed by a call to the (overridden) stat on a temporary file, and subtracting the difference before calling Win32API::File::Time::SetFileTime on the target file. However, this doesn't work correctly when the file time is within one hour of a daylight saving transition. Therefore, the only (and probably best solution) is to call the Microsoft GetFileTime/SetFileTime functions directly without first doing a conversion. I haven't tried this yet. Do you want me to look into that? Extra benefit is that it would get rid of a dependency to another module.

[Xlnt]

By the way, during testing I noticed a difference between:

> exiftool.pl -FileModifyDate="2015:03:29 03:30:00+02:00" test.txt
1 image files updated

and

> exiftool.pl -File*Date="2015:03:29 03:30:00+02:00" test.txt
Warning: Sorry, FileModifyDate is unsafe for writing
Nothing to do.

Any idea where this difference comes from?

[Phil Harvey]

> exiftool.pl -File*Date="2015:03:29 03:30:00+02:00" test.txt
Warning: Sorry, FileModifyDate is unsafe for writing
Nothing to do.

Any idea where this difference comes from?

Yes.  FileModifyDate is marked as "Unsafe" to write, so you can't write it with wildcards.  (See the Extra tags in the tag name documentation.)

I'll spend some time thinking about your previous post as soon as I get a chance.

- Phil

[Phil Harvey]

I had a quick look at your code.  It looks very good so far.  I think that your idea of calling the Windows GetFileTime/SetFileTime directly would be great if possible.

- Phil

[Xlnt]

Hi, I think I have a solution which doesn't require Win32API::File::Time nor Win32::UTCFileTime.

Instead, Win32::API and Win32API::File are needed. I don't think this is a issue, because:

• These modules are by default available (at least in my ActivePerl installation).
• There is a fallback to stat/utime (in combination with a warning) when any of the modules is not available.
• Win32API::File is already required for unicode support.
• Utime itself doesn't even work without these modules (at least in my ActivePerl installation). I tried removing them from the lib directory, for both resulting in the message The futimes function is unimplemented at ... pointing to a location where utime is called.

In addition Time::Local and Scalar::Util are used:

• Time::Local::gmtime/timegm are used during the conversion between FILETIME and perl time. These functions are already used elsewhere, but could be circumvented by doing the conversion between FILETIME (100e-9 seconds since 01-01-1601, uint64) to perl time (seconds since 01-01-1970) 'manualy' using the bigint module instead of using e.g. FileTimeToSystemTime and Time::Local::timegm. Perhaps that is even nicer, at least more efficient. But I couldn't figure out how to convert the Math::BigInt value to a 'normal' scalar (perhaps not even needed).
• Scalar::Util::openhandle is used to determine whether the first argument suplied to GetFileTime/SetFileTime is an open handle or a file name. Perhaps a simple ref check is sufficient.

Both Time::Local and Scalar::Util are directly used without fallback mechanism, because they are pretty standard (at least in my Debian and ActivePerl installation), CMIIW.

Attached is a patch containing changes compared to ExifTool version 9.82. I tested reading/updating the various file times, both on Linux and Windows, both when daylight saving time is in effect and when not. All existing tests run successfully. I didn't test for unicode compatibility, but it should work since either a file handle or the existing Open sub (although slightly modified to support read/write) is used to access files.

Some notes regarding the changes:

• I used four 'static' variables to store the kernel32 entry points (e.g. FileTimeToSystemTime). I made them static and local to GetFileTime/SetFileTime by wrapping the variables and the subs in a block. If you are willing to require Perl 5.10, the variables can be declared directly in the subs with the state keyword.
• There are two remaining locations where stat is used. However, only file permssions, gid, uid are used at those locations. I wonder, however, if CopyFileAttrs (which calls stat with a file name) is unicode safe.
• I also wonder what the timezone bug is, to which you are referring in code (sub TimeLocal). Does this perhaps have to do with the issue I mentioned in my first post? I.e. the date time part is not accounted for daylight saving offset, but the timezone part is.

[Phil Harvey]

Thanks!  Your Perl and Windows API skills are very impressive.

I have applied the patch and am about to start testing.

Just one thing makes me a bit nervous.  You localize *FH and say the file will be closed at the end of the sub.  This is a new one for me.  I am willing to accept that the file will be closed when the localized file handle goes out of scope, but I guess I don't understand the scoping rules of Perl because I thought the scope would be the "if" block in which *FH is defined.  However, your comment states it will be closed at the end of the subroutine.  Why is that?

- Phil

Edit:  I answered my own question:  *FH stays open until the end of the subroutine because it is assigned to $handle, which has subroutine scope.

Also, you're right that CopyFileAttrs() is not Unicode safe.  There are still some things that don't work with Windows Unicode file names.  But I'm glad that you're thinking about this.  As far as I can tell your patch should handle Unicode names properly.

BTW, eventually I will probably move this thread over to Bug Reports / Feature Requests.

[Xlnt]

Thanks!  Your Perl and Windows API skills are very impressive.

Thanks . My programming background is actually C/C++/C#, with reasonable knowledge of the Windows API. I used Win32API::File::Time and Win32::UTCFileTime a lot for inspiration.

Edit:  I answered my own question:  *FH stays open until the end of the subroutine because it is assigned to $handle, which has subroutine scope.

Indeed. I wasn't to sure about this either, until I tested with some printf's, sleep's and Process Monitor to see that the file is actually closed directly at the end of the sub.

BTW, eventually I will probably move this thread over to Bug Reports / Feature Requests.

Sounds good to me. When starting this thread, I wasn't sure it was a bug. Actually it isn't a bug in ExifTool, but for a user it will appear to be.

I am looking forward for the release that contains this patch, because then I can remove my local repository . There isn't a public (svn/git/etc) repository, or am I mistaken?

Thanks for your support and prompt replies.

[Phil Harvey]

Testing is going well so far.  Unless I run into a problem, the public release should be within a few days.

There is a public git repository at sourceforge (https://sourceforge.net/p/exiftool/code/ci/master/tree/), but currently it is just a mirror of the released code.

- Phil

[Phil Harvey]

I'm impressed.  It seems to work really well.  The only minor bugs I found were a couple of placees where you used single quotes in the argument to WarnOnce where a variable ($tag) required interpolation.

I have changed the code a bit to conform with the existing coding style, and changed the time conversion to do it manually as you suggested (but not using Math::BigInt).  Here are the two functions as they now stand:

Code Select Expand

#------------------------------------------------------------------------------
# Get file times (Unix seconds since the epoch)
# Inputs: 0) ExifTool ref, 1) file name or ref
# Returns: 0) access time, 1) modification time, 2) creation time (or undefs on error)
my $Kernel32GetFileTime;
sub GetFileTime($$)
{
    my ($self, $file) = @_;

    # open file by name if necessary
    unless (ref $file) {
        local *FH;
        $self->Open(\*FH, $file) or $self->Warn("Open '$file' failed"), return ();
        $file = *FH;  # (not \*FH, so *FH will be kept open until $file goes out of scope)
    }
    # on Windows, try to work around incorrect file times when daylight saving time is in effect
    if ($^O eq 'MSWin32') {
        if (not eval { require Win32::API }) {
            $self->WarnOnce('Install Win32::API for proper handling of Windows file times');
        } elsif (not eval { require Win32API::File }) {
            $self->WarnOnce('Install Win32API::File for proper handling of Windows file times');
        } else {
            # get Win32 handle, needed for GetFileTime
            my $win32Handle = Win32API::File::GetOsFHandle($file);
            unless ($win32Handle) {
                $self->Warn("Win32API::File::GetOsFHandle returned invalid handle");
                return ();
            }
            # get FILETIME structs
            my ($atime, $mtime, $ctime, $time);
            $atime = $mtime = $ctime = pack 'VV', 0, 0;
            unless ($Kernel32GetFileTime) {
                $Kernel32GetFileTime = new Win32::API('KERNEL32', 'GetFileTime', 'NPPP', 'I');
            }
            unless ($Kernel32GetFileTime->Call($win32Handle, $ctime, $atime, $mtime)) {
                $self->Warn("Kernel32::GetFileTime returned " . Win32::GetLastError());
                return ();
            }
            # convert FILETIME structs to Unix seconds
            foreach $time ($atime, $mtime, $ctime) {
                my ($lo, $hi) = unpack 'VV', $time; # unpack FILETIME struct
                # FILETIME is in 100 ns intervals since 0:00 UTC Jan 1, 1601
                # (89 leap years between 1601 and 1970)
                $time = ($hi * 4294967296 + $lo) * 1e-7 - (((1970-1601)*365+89)*24*3600);
            }
            return ($atime, $mtime, $ctime);
        }
    }
    # other os (or Windows fallback)
    return (stat($file))[8, 9, 10];
}

Code Select Expand

#------------------------------------------------------------------------------
# Set file times (Unix seconds since the epoch)
# Inputs: 0) ExifTool ref, 1) file name or ref, 2) access time, 3) modification time,
#         4) inode change or creation time (or undef for any time to avoid setting)
# Returns: 1 on success, 0 on error
my $Kernel32SetFileTime;
sub SetFileTime($$;$$$)
{
    my ($self, $file, $atime, $mtime, $ctime) = @_;

    # open file by name if necessary
    unless (ref $file) {
        local *FH;
        $self->Open(\*FH, $file, '+<') or $self->Warn("Open '$file' failed"), return 0;
        $file = *FH;  # (not \*FH, so *FH will be kept open until $file goes out of scope)
    }
    # on Windows, try to work around incorrect file times when daylight saving time is in effect
    if ($^O eq 'MSWin32') {
        if (not eval { require Win32::API }) {
            $self->WarnOnce('Install Win32::API for proper handling of Windows file times');
        } elsif (not eval { require Win32API::File }) {
            $self->WarnOnce('Install Win32API::File for proper handling of Windows file times');
        } else {
            # get Win32 handle, needed for SetFileTime
            my $win32Handle = Win32API::File::GetOsFHandle($file);
            unless ($win32Handle) {
                $self->Warn("Win32API::File::GetOsFHandle returned invalid handle");
                return 0;
            }
            # convert Unix seconds to FILETIME structs
            my $time;
            foreach $time ($atime, $mtime, $ctime) {
                # set to NULL if not defined (i.e. do not change)
                defined $time or $time = 0, next;
                # convert to 100 ns intervals since 0:00 UTC Jan 1, 1601
                # (89 leap years between 1601 and 1970)
                my $wt = ($time + (((1970-1601)*365+89)*24*3600)) * 1e7;
                my $hi = int($wt / 4294967296);
                $time = pack 'VV', int($wt - $hi * 4294967296), $hi; # pack FILETIME struct
            }
            unless ($Kernel32SetFileTime) {
                $Kernel32SetFileTime = new Win32::API('KERNEL32', 'SetFileTime', 'NPPP', 'I');
            }
            unless ($Kernel32SetFileTime->Call($win32Handle, $ctime, $atime, $mtime)) {
                $self->Warn("Kernel32::SetFileTime returned " . Win32::GetLastError());
                return 0;
            }
            return 1;
        }
    }
    # other os (or Windows fallback)
    return utime($atime, $mtime, $file);
}

I'm more impressed than ever with your solution, and really appreciate all of the work you put into this!

- Phil

[Xlnt]

Few notes regarding your changes:

• VV instead of LL with pack/unpack: With this the fields of the FILETIME struct are explicitly interpreted as little-endian instead of 'native byteorder'. Although correct for most Windows systems (running x86, little-endian), I don't think this is correct if the code would ever run on Windows with a big-endian processor architecture (e.g. ARM).
• 64-bit calculation, note1: I think you should do the calculations within a block with use integer to prevent Perl from using a floating-point to prevent resolution problems. I am not sure how Perl decides here, but in C/C++ multiplying with 1e7 would result in a floating-point number.
• 64-bit calculation, note2: I noticed that 64-bit integer calculation worked out of the box on my Windows system (x86, 64-bits) with 32-bit Perl, and also on my rasperry pi (ARM, 32-bit). But I concluded that is because both 32-bit Perl distributions were compiled with 64-bit int support (checked with use Config; print $Config{use64bitint}). I didn't try compiling Perl from source without 64-bit int support to see if if fails then, but I guess it will. So this code will probably fail on 32-bit Windows systems using Perl without 64-bit int support.

Some minor remarks:

• Instead of ($hi * 4294967296 + $lo) I would use ($hi << 32 + $lo), because it more clearly shows the goal, and there is less chance for a type. It should also be more efficient, although you shouldn't be using Perl if that really matters .
• Instead of int($wt - $hi * 4294967296), $hi I would use $wt & 0xFFFFFFFF, $wt >> 32 (or even $wt % 4294967296, $wt >> 32), for same reasons.

Edit:

I just found out that Perl by default uses double-precision floating-point for calculations:
Because Perl uses doubles (or long doubles, if configured) internally for all numeric calculation, ...

Because of this and the fact that only whole second resolution is needed, it should be fine. Resolution problems will only arise after 2^52/365.25/24/3600 ~= 142710460 years since 1970 (mantissa of IEEE-745 double comprises 52 bits). Although we are never going to experience that, it is also more than the range of a FILETIME struct, which is 2^64/365.25/24/3600*100e-9 ~= 58454 years since 1601.

Just to be sure, I ran the following script to verify that dates uptill now are converted correctly:

Code Select Expand


my $n = time;
for (my $t = 0; $t < $n; $t++)
{
  my $wt = ($t + (((1970-1601)*365+89)*24*3600)) * 1e7;
  my $hi = int($wt / 4294967296);
  my $lo = int($wt - $hi * 4294967296);
  my $p = pack 'VV', $lo, $hi;

  my ($lo_ref, $hi_ref) = unpack 'VV', $p;
  my $t_ref = ($hi_ref * 4294967296 + $lo_ref) * 1e-7 - (((1970-1601)*365+89)*24*3600);

  if ($t_ref != $t)
  {
    printf("%15d; %24.2f; %15d; %15d; %15d; %15d; %15d\n", $t, $wt, $hi, $lo, $hi_ref, $lo_ref, $t_ref);
  }
}


It took about 50 minutes, and didn't show any misses.

Only the first bullet about endianness still holds I think.

[Phil Harvey]

VV instead of LL with pack/unpack: With this the fields of the FILETIME struct are explicitly interpreted as little-endian instead of 'native byteorder'. Although correct for most Windows systems (running x86, little-endian), I don't think this is correct if the code would ever run on Windows with a big-endian processor architecture (e.g. ARM).

Excellent.  I wasn't sure about this and was hoping you would comment (which is part of the reason I made this change).  I will change back to 'LL', but then there is a question about the word order.  If it is stored as a 64-bit integer, then I also have to swap words on big-endian systems.  Worse, some ARM processors store the msw first, even on little-endian systems.  Your thoughts about this would be appreciated.

64-bit calculation, note1: I think you should do the calculations within a block with use integer to prevent Perl from using a floating-point to prevent resolution problems. I am not sure how Perl decides here, but in C/C++ multiplying with 1e7 would result in a floating-point number.

Yes.  This is on my list of things to do today.  But I will solve it by properly rounding the floating point because I don't want to have to worry about integer overflow on some systems.

64-bit calculation, note2: I noticed that 64-bit integer calculation worked out of the box on my Windows system (x86, 64-bits) with 32-bit Perl, and also on my rasperry pi (ARM, 32-bit). But I concluded that is because both 32-bit Perl distributions were compiled with 64-bit int support (checked with use Config; print $Config{use64bitint}). I didn't try compiling Perl from source without 64-bit int support to see if if fails then, but I guess it will. So this code will probably fail on 32-bit Windows systems using Perl without 64-bit int support.

My fear exactly.

Instead of ($hi * 4294967296 + $lo) I would use ($hi << 32 + $lo), because it more clearly shows the goal, and there is less chance for a type. It should also be more efficient, although you shouldn't be using Perl if that really matters .

But I worry about integer overflow.  << is an integer operator.  This is the same reason I don't use 0x100000000, although it is more readable (but still prone to typos).

Instead of int($wt - $hi * 4294967296), $hi I would use $wt & 0xFFFFFFFF, $wt >> 32 (or even $wt % 4294967296, $wt >> 32), for same reasons.

Here too.

I just found out that Perl by default uses double-precision floating-point for calculations:
Because Perl uses doubles (or long doubles, if configured) internally for all numeric calculation, ...

Because of this and the fact that only whole second resolution is needed, it should be fine. Resolution problems will only arise after 2^52/365.25/24/3600 ~= 142710460 years since 1970 (mantissa of IEEE-745 double comprises 52 bits). Although we are never going to experience that, it is also more than the range of a FILETIME struct, which is 2^64/365.25/24/3600*100e-9 ~= 58454 years since 1601.

Yes.  Perl will automatically promote an integer calculation to double if necessary.  And doubles have more than sufficient precision to represent seconds over the range of interest.

Just to be sure, I ran the following script to verify that dates uptill now are converted correctly:
[...]

Nice.  Thanks for testing this!  It looks like I don't have to worry about rounding after all.  This is a good thing because the simple solution of adding 0.5 then truncating to integer won't work for times before 1970 (negative time_t's are allowed on some systems).

- Phil

Edit:  I was reading about the FILETIME structure, and it seems that the lsw always comes first.  So it looks as if word order may not be a problem.