ExifTool Forum

I've released ExifTool 13.01 with a new API WindowsLongPath option.

The hope is to be able to set this by default unless it causes other problems.

Setting this option (with -api windowslongpath) enables support for long path names in Windows as well as wide characters in file names.

This option should be considered experimental, and we need to do lots of testing first if we are to eventually make this the default.

- Phil

Quote from: Phil Harvey on November 01, 2024, 04:19:32 PMThis option should be considered experimental, and we need to do lots of testing first if we are to eventually make this the default.

Great news Phil.

Initial test looks promising, I will keep on using it, and report any disturbances.

I found a problem, but the same problem exists with 13.00:

If I am in a directory which requires a long path to specify (thanks Frank for the "Long Paths" test file), and I try to write the FileName tag, ExifTool hangs when trying to load the Encode module from a "require Encode" statement in the Sanitize routine.  Unfortunately this problem in Encode is somewhat out of my control unless we can find a patch to avoid using the Encode module.  I can reproduce the hang with this command in the same directory:

perl -e "require Encode"

Require-ing other modules seems to work.  It is just Encode that causes problems.  Same thing happens with both Strawberry Perl and ActiveState Perl.

- Phil

Too bad that doesn't work.
But the new option doesn't make it worse!

Let me know if I can help.

Meanwhile I have checked Windows versions 11, 10, 8, 7 and even XP. They all work, except for renaming of course.

Frank

Did some tests on Windows 10 Pro 64 bits, and it all works!

Renamed some (long path) files using ExifToolGui. OK

Then I replayed it using the CMD prompt. OK

- perl -e "require Encode" OK. No Hang.
- exiftool -createdate *.jpg.
  Without the API 2 files could not be read, with the API all 6.

- exiftool -filename<${Exif:CreateDate} %f.%e" -d %Y%m%d (Some expression to prefix the createdate)
  Without the API 3 files could not be renamed, with the API all 6.

Are you using Windows 7 64bits?

Attached the output of the CMD prompt as txt files.
require.txt
createdate.txt
rename No WindowsLongPath.txt
rename WindowsLongPath.txt   

For anyone that wants to test this feature.

- If you are using ExifToolGui. You can add the API option via 'Options/Custom options' type in '-API WindowsLongPath=1' and press OK. To get rid of it, just empty the edit box.

custom options.jpg

- If your drive supports 8dot3 names, ExifToolGui/Windows can fool you by substituting the 8dot3 name if the 'real' name gets to long. The will typically show as something like: 122CF5~1.JPG.
The utility fsutil can be used (When you're Admin, and brave enough) to query, and change that setting.
Note that this setting only takes effect for newly created files. Existing files will not be affected.

Microsoft Windows [Version 10.0.22631.4317]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\System32>cd \

C:\>fsutil 8dot3name query c:
The volume state is: 0 (8dot3 name creation is ENABLED)
The registry state is: 2 (Per volume setting - the default)

Based on the above settings, 8dot3 name creation is ENABLED on "c:"

C:\>fsutil 8dot3name set c: 1
Successfully DISABLED 8dot3name generation on "c:"

C:\>fsutil 8dot3name query c:
The volume state is: 1 (8dot3 name creation is DISABLED)
The registry state is: 2 (Per volume setting - the default)

Based on the above settings, 8dot3 name creation is DISABLED on "c:"

C:\>fsutil 8dot3name set c: 0
Successfully ENABLED 8dot3name generation on "c:"

C:\>fsutil 8dot3name query c:
The volume state is: 0 (8dot3 name creation is ENABLED)
The registry state is: 2 (Per volume setting - the default)

Based on the above settings, 8dot3 name creation is ENABLED on "c:"

C:\>

Hi Frank,

Thanks for running these tests!

Quote from: FrankB on November 02, 2024, 04:47:35 AM- perl -e "require Encode" OK. No Hang.

Interesting.

QuoteAre you using Windows 7 64bits?

I'm running a Windows 10 virtual machine on my Mac.  I need to do more playing around with this.

- Phil

Quote from: Phil Harvey on November 02, 2024, 08:43:54 AMI'm running a Windows 10 virtual machine on my Mac.  I need to do more playing around with this.

OK. Surely there's difference somewhere, but what? Let me know if I can do anything to help.

Meanwhile I did some more testing. On an old Windows 8 image. Works like a charm, also renaming.

But... There is something I found using network shares. UNC paths, not using drive letters. What's bothering me is that I think I've put you on the wrong foot with my examples. Sorry for that.

The problem is that UNC paths need this form of prepending: \\?\UNC\Server\share.

https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry (https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry)
unc_share.jpg

I had an idea about the hang.  I had put the LONG DIR inside a deeper directory so the full path itself then exceeded the long threshold.

I'll consider your UNC point when I have a bit of time

I read what you said about UNC paths, and don't understand how you think you put me on the wrong foot.

Do you have a UNC drive you can test?  Does ExifTool fail for these?  If so, what is the output of this command after you cd to one of the directories on this drive?:

perl -e "require Cwd; print Cwd::getcwd()"

or if it is easier to run exiftool:

exiftool -p "${filename;require Cwd;$_=Cwd::getcwd()}" FILE

- Phil

Quote from: Phil Harvey on November 02, 2024, 12:54:07 PMI read what you said about UNC paths, and don't understand how you think you put me on the wrong foot.

In the mail I sent you, there was also a modified version of 'encodeFileName' that I used for testing. That does not prepend UNC properly. I thought you had used that.

Quote from: Phil Harvey on November 02, 2024, 12:54:07 PMDo you have a UNC drive you can test?  Does ExifTool fail for these?  If so, what is the output of this command after you cd to one of the directories on this drive?:

Will prepare that and attach it as a txt file.

Quote from: Phil Harvey on November 02, 2024, 12:54:07 PMperl -e "require Cwd; print Cwd::getcwd()"

or if it is easier to run exiftool:

exiftool -p "${filename;require Cwd;$_=Cwd::getcwd()}" FILE

That doesn't work in Windows. You cant open a CMD, or PS prompt and change dir to an UNC path. Or more correct, I wouldn't know how to do that. (Also a reason why I didn't notice it earlier)

Frank

Here is my UNC test.

The directory structure is on a hard drive (NTFS) drive K:
Copied to share \\10.10.10.12\foto that is also mapped as drive Z:

See directory structure.txt for details
Directory structure.txt

I executed 3 commands, working directory c:\users\xxxx\Desktop. A simple command -Filename and supplying the complete path. As mentioned in my previous post, for UNC paths that is the only option if you execute if from a CMD window.
(For ExifToolGui it is possible, to have a UNC path as working directory)

The results show that the command fails for the UNC path.

Drive K:
Harddrive.cmd

Drive Z:
Netuse.cmd

UNC:
UNC.cmd

Results:
Results.txt

Of course I had a look at the ExifTool code, and I was able to fix it by modifying WindowsLongPath.

When called from the desktop (CWD is c:/users/xxxx/Desktop) it returns '\\?\C:\Users\xxxx\Desktop\10.10.10.12\foto\LONG....'.

If called from GUI (CWD is //10.10.10.12/foto...), it returns '\\?\\\10.10.10.12\foto\LONG....

In both cases it should return '\\?\UNC\10.10.10.12\foto\LONG ......'

I would be glad to share my fix, but am afraid that that would take away the fun for you.

Frank

Edit: Fixed typos and corrected slashes in CWD (Windows!)

Hi Frank,

Thanks for running these tests.

I would be happy to see your fix.  I don't understand under what cases the "\\?\UNC\" should be added.

The code I have implemented is based on the Github PR (https://github.com/exiftool/exiftool/pull/283), but implemented in a way similar to the code you sent me.  However it has the advantage that "." and ".." are (hopefully) handled properly.

Reading this page (https://learn.microsoft.com/en-us/dotnet/standard/io/file-path-formats#unc-paths), it seems that UNC paths may also have relative path components ("." and ".."), so we'll have to consider these here too.

- Phil

Edit: Added link to Github PR

Quote from: Phil Harvey on November 02, 2024, 07:35:43 PMI would be happy to see your fix.

Coming up, see the attachments.

Quote from: Phil Harvey on November 02, 2024, 07:35:43 PMI don't understand under what cases the "\\?\UNC\" should be added.

If you mean 'requirements', it's on the page 'Learn Microsoft' page you referred to.
If you mean 'in code', I have identified 2 cases in my fix. For the test cases I could think of that's enough. I'm not quite sure if I have covered all.

For these tests I concentrated on the UNC case from the previous tests that failed. I have put 'print' statements (not indented on purpose to make them stand out) in the code showing the input, and what is returned.

Test 1: From Cmd prompt using a Full path.
Test 2: From ExifToolGui using a relative path.
Test 3: From ExifToolGui using a relative path starting with '.'.
WindowsLongPath.txt
Test 1.txt
Test 2.txt
Test 3.txt

Tests 2 and 3 were done from ExifToolGui, because that was the easiest for me. Output from these tests is taken from the Log Window. You have to believe that GUI doesn't mess up the output.

Frank

Hi Frank,

Thanks.

QuoteIf you mean 'requirements', it's on the page 'Learn Microsoft' page you referred to.
If you mean 'in code',

I meant in the code.

If I understand, it looks like the first "\" must be removed and "\\?\UNC" must be added and the start if the path begins with "\\" but not "\\?\".

I'm wondering why you tested for $path =~ /\\\\/ rather than $path =~ /^\\\\/.  Can the "\\" appear in the middle of the path too?

- Phil

Hi Phil,

Quote from: Phil Harvey on November 03, 2024, 09:32:17 AMI'm wondering why you tested for $path =~ /\\\\/ rather than $path =~ /^\\\\/.  Can the "\\" appear in the middle of the path too?

That is because I'm not familiar with regex'es. I try to stay away from them. 
The test should be as you suggest '$path =~ /^\\\\/.'.
'\\' could be in the middle, but that should not be considered.

Quote from: Phil Harvey on November 03, 2024, 09:32:17 AMbut not "\\?\"

True. I assume that prepending has already been done then.

Quote from: Phil Harvey on November 03, 2024, 09:32:17 AMIf I understand, it looks like the first "\" must be removed and "\\?\UNC" must be added.

That was easiest for me to do given the existing code. To put it in 'human' words:

Only if it is a UNC path (no matter if it is relative or absolute) the result should be: \\?\UNC\server\share\rest of the path.

Examples:

1. Absolute path (Path starts with \\, CWD doesn't matter)
Path: \\10.10.10.12\foto\longdir1\abcd.jpg
result: \\?\UNC\10.10.10.12\foto\longdir1\abcd.jpg

2. Relative path. (Path does not start with \\, CWD starts with \\)
CWD: \\10.10.10.12\foto\longdir1
Path: abcd.jpg, or .\abcd.jpg (..\ not easy to test for me)
result: \\?\UNC\10.10.10.12\foto\longdir1\abcd.jpg

If you have thoughts why all this is needed, dont blame me. I'm only the messenger. I'm just repeating what's on the Microsoft learn page (https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry)

Really appreciate your Open-Mindedness

Frank
 

One more remark, about mitigation.

With -Api WindowsLongPath=1 you always prepend '\\?\', or '\\?\UNC\'. You could have opted to only do that if the resulting complete path exceeds 260 chars. Both methods have their advantage:

Only when length>260: Less chance that this change affects existing use(users)
Always: Any errors will show up right away. (I'm not native English, but I think the phrase 'No guts, no glory' applies here)

To be honoust, I wouldn't know what to choose.

Frank

Thanks Frank.  Your input has been very helpful.  I'll continue testing at my end because I still think I could be missing something.

- Phil

I found 2 cases that need patching:

1. If an absolute path on the current drive is specified.  (ie. path starts with a single "\")

2. If a relative path on a different drive is specified.  (ie. path starts with "d:relpath" without a "/" before relpath)

I'll fix these.

- Phil

I agree.

I'll be glad to test it, if you have something. You dont need to release a complete version, I can merge the differences.

Frank

Thanks Frank.

I've patched these 2 cases and also handled the case where relative directories ("." or "..") appear in fully qualified directory names.  Of course, this necessitated a major rewrite.  Attached is my current version of ExifTool.pm

I've also added an API Debug option to give you some debugging output.

I'm sorry to say that your UNC tests will need repeating because it would have been easy to mess that up with the changes I made.  One tricky thing is the case where the current drive is UNC and an absolute path on the current drive is specified (ie. starting with a single "/") -- I think I got this right, that I need to keep both the server and the share name from the UNC drive, then add the absolute path after those.  Also, I couldn't test the case where a relative path on a different drive was specified because I only have one drive.

- Phil

Great Phil,

I'll test it later on today.

Frank

Hi Phil,

compiler error:

Undefined subroutine Image::ExifTool::ContainsWildcards called at C:\Program Files\ExifTool\exiftool_files/lib/Image/ExifTool.pm line 5485

ContainsWildcards is removed from .pm, but still referenced from .pl
lines
1055
1441
4071
4197

Do I also need an updated exiftool.pl?

Oh, sorry.  I renamed ContainsWildcards to HasWildcards.  That should be the only change.

- Phil

Ok.

I'll try that later.

Hold off testing until I come up with a fix for a bug I just found.

- Phil

Here's the fix ("test6" release).  You'll still have to rename IncludesWildcards to HasWildcards in exiftool.pl

Wow, what a pain in the ass this is.  I can see why the Perl porters gave up on trying to support the Windows filename quirks.  :(

I really appreciate your help here.

- Phil

Will do. Not at home right now. Could take a few hours.

Glad to help

Hi Phil,

I would say leave it like it is. All tests I could think of are OK.

Directory on Harddrive
< 260 Chars relative and absolute path
> 260 Chars relative and absolute path

Directory on Server (UNC)
< 260 Chars relative and absolute path
> 260 Chars relative and absolute path

Played with . and .. OK

This scenario also:
Working directory in CMD prompt c:\users....
Last directory on Z: is \Long dirxxxx (>240 chars)
exiftool -filename z:*.jpg will not read 2 files
exiftool -api windowslongpath=1 -filename z:*.jpg will read all files

For UNC paths this does not work, because CMD has no notion of a working directory for a UNC path. (It can be done maybe with PowerShell, dont know)

Executed commands in GUI, CMD prompt and PowerShell, with and without Api Debug

Only thing that I noticed, but not an error. Prepending on a harddrive is only done when the path name > 250 chars(approx). For UNC paths, and absolute paths it is always done. (To prove that I really tested!)

If there is some specific scenario that you want tested, let me know.

Thanks so far Phil,
Frank

Hi Frank,

Excellent, thanks!

Yes, I'm not adding the "\\?\" prefix unless the length gets close the the limit, or if it is needed for a UNC drive.  I liked the suggestion of yours to add it only if necessary.  I set a slightly lower threshold because after doing some research it seems that there are times the limit is less than 260, so I set it a 247 which is the lowest value I saw.

- Phil

Quote from: Phil Harvey on November 04, 2024, 04:45:40 PMYes, I'm not adding the "\\?\" prefix unless the length gets close the the limit, or if it is needed for a UNC drive.

In my view the only difference between UNC and Non-UNC is the prefix itself \\?\ <=> \\?UNC\. UNC paths, just like Non-UNC paths, work fine without prefixing if the length is < 247.

But that's only for the record. You can leave it like it is.
I will continue to use this fix while testing GUI, and if anything comes up, I will notify you.

Frank

I see. OK.  No worries though.  I'll leave it as is.

- Phil

Problem found with -Api WindowsLongPath and proposed solution

Hi Phil,

The problem I found with -Api WindowsLongPath is that it doesn't accept International (Wide) characters in the directory name. (EG: Greek = Ελληνικα) The undocumented option -Api debug=1 shows that getcwd is the culprit.
K:\TestPath\SHORT DIR\SUB DIR1\SUB DIR2\SUB DIR3\Ελληνικα>exiftool -filename -api windowslongpath=1 -api debug=1 Greek.jpg
WindowsLongPath input : Greek.jpg
WindowsLongPath getcwd: K:\TestPath\SHORT DIR\SUB DIR1\SUB DIR2\SUB DIR3\???????a
Side Note: Later on when I scanned where getcwd is used in ExifTool, I noticed that the -filepath is not supported for these directory names, because getcwd is unable to cope with wide characters.

Thinking of a solution I came up with the Win32API method GetFullPathName in kernel32.dll. This function can handle wide characters, and as bonus it handles all the special cases. Like relative paths on another drive, dos device filenames, UNC paths etc.

I ran tests on Windows XP, 7, 8, 10 and 11, on filenames containing combinations of short, long, Ansi and Wide characters, on local hard drives, mapped network drives and unc paths. In a CMD prompt and from ExifToolGui. They all behaved well.

See the attached zip for the full exiftool.pm and the directory with filenames I used for testing.

Curious to hear what you think of this.

Frank
#------------------------------------------------------------------------------
# Rebuild a path as an absolute long path to be usable in Windows system calls
# Inputs: 0) ExifTool ref, 1) path string
# Returns: normalized long path
# Note: this should only be called for Windows systems
# References:
# - https://learn.microsoft.com/en-us/dotnet/standard/io/file-path-formats
# - https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
#
# GetFullPathName supported by Windows XP and later. It handles:
# full path names                EG: c:\foto\sub\abc.jpg
# relative                       EG: .\abc.jpg,  ..\abc.jpg
# full UNC paths                 EG: \\server\share\abc.jpg
# relative UNC paths             EG: .\abc.jpg,  ..\abc.jpg
# Dos device paths               EG: \\.\c:\fotoabc.jpg
# relative path on other drives  EG: z:abc.jpg (working dir on z: z:\foto called from c:\foto)
# Wide chars                     EG: Chars that need UTF8.
#
# Dont know exactly how Win32::API::More->new works, but I would imagine it does a LoadLibrary and a GetProcAddress.
# So I only want to do that once. The variable $GetFullPathName should by defined globally to achieve this.
#
sub WindowsLongPath($$)
{
    my ($self, $path) = @_;
    my $debug = $$self{OPTIONS}{Debug};
    my $out = $$self{OPTIONS}{TextOut};

    $debug and print $out "WindowsLongPath input : $path\n";
    $path =~ tr(/)(\\); # convert slashes to backslashes

    if ($path =~ /^\\\\\?\\/) {                                                               # already a device path in the format we want
      $debug and print $out "WindowsLongPath (Already prefixed) return: $path\n";
      return $path;
    }

    if (!defined($GetFullPathName)) {                                                         # Need to import (once) GetFullPathName?
      $GetFullPathName = Win32::API::More->new('kernel32.dll',                                # Note: Last param lpFilePart not used!
      'DWORD GetFullPathNameW(LPCWSTR lpFileName,' .                                          # We need the W(ide) version.
      ' DWORD nBufferLength,' .                                                               # Length buffer provided. If you pass 0, the return value is the length required.
      ' LPWSTR lpBuffer,' .                                                                   # Receives output from GetFullPathName
      ' LPWSTR *lpFilePart);');                                                               # Pointer within the buffer, where the filename starts.

      $debug and print $out "GetFullPathName loaded : defined($GetFullPathName) \n";
    }

    my $enc = $$self{OPTIONS}{CharsetFileName};
    my $encPath = $self->Encode($path, 'UTF16', 'II', $enc);                                  # Need to encode to UTF16
    my $LenReq = $GetFullPathName->Call($encPath, 0, 0, 0) + 1;                               # first pass gets length required. Add +1 for safety, Needs Null terminator?
    my $FullPath = \0 x $LenReq x 2;                                                          # create buffer to hold Full Path
    $GetFullPathName->Call($encPath, $LenReq, $FullPath, 0);                                  # FullPath is UTF16 now
    $path = $self->Decode($FullPath, 'UTF16', 'II', $enc);                                    # Decode

    if ($path =~ /^\\\\/) {
      $path = '\\\\?\UNC' . substr($path, 1) unless length($path) <= 247;
    } else {
      $path = '\\\\?\\' . $path unless length($path) <= 247;
    }

    $debug and print $out "WindowsLongPath return: $path\n";
    return $path;
}

TestLongPath.zip

Hi Frank,

Excellent!  Thanks!!  It will probably be Monday before I can take a close look at this.

- Phil

Hi Frank,

I found a bit of time today.

Wow, this is excellent!  Much cleaner that what I was doing before.  There were a couple of minor things that I fixed.  I also changed Win32::API::More->new to Win32::API->new to match the way it is used in the rest of the code.  It seems to work for me, but I couldn't figure out how to cd into your test directory with Unicode characters so my testing wasn't complete.  Attached is my new version of the code.

- Phil

Thanks for your high opinion Phil.
I felt I had to come up with something that works, after I bet on the wrong horse by proposing Cwd!

I may be able to learn you something. How to CD into a Unicode directory.
- Select the text from the address bar in Windows Explorer, and copy that CTRL/C.
- Type CD " in the CMD prompt, and use Right click, or from the File menu Edit/Paste

copy_and_paste.jpg

Will look into your changes, but no doubt I'll be fine with them. And after all, I'm not a Perl programmer.

Frank

Thanks.  I was able to "cd" into the Unicode directories and all my tests ran without a problem.

- Phil

Frank,

 I've just released 13.03 with a few minor changes, but it should behave the same as the last version I sent you unless I broke something.  I'm much happier with the WindowsLongPath code in this release vs. 13.02.  Thanks for figuring this out.

- Phil

Thanks a million Phil.

FYI I will release ExiftoolGui V636 soon, within a couple of days.
That will have this as a default option!

Did some more testing meanwhile, also performance comparisons. All OK.

Frank

Hello Phil, hello Frank,

Great, absolutly great!
Thanks for this feature.

But please allow a question.
Searching for "support of long path in windows" I always read
- a special registry key has to be set and
- the application needs a "long path aware" setting in its manifest

How does ExifTool fulfill these requirements?

Thanks in advance
Best regards
herb

@Phil: Allow me to answer that one, correct me if I'm wrong.

@Herb: Short answer ExifTool (And Gui) dont do it that way. There is also the option to prefix Paths with \\?\ (or \\?\UNC\ for UNC paths) thereby 'opting in' for Long paths, for selected API calls. That is what ExifTool (And Gui) do.

Read the last part of this page:
https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#functions-without-max_path-restrictions (https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#functions-without-max_path-restrictions)

Frank

Hello,

thanks for your quick reply, thanks for the private lesson
and thanks again for this great feature.

Best regards
herb

Just to manage expectations...

The total length of the path can exceed 260 chars, but you may still experience limitations. To name a few:

- The individual parts of the Path (sub directories, or filename) can not exceed 260 chars.
(c:\dir1\sub dir1\sub dir2\sub di4\file.ext; individual part = dir1, or sub dir1 etc.)
- Currently the CreateProcess does not support a CurrentDirectory (AKA Working directory) longer than 260 chars. This prevents GUI from starting ExifTool in a directory longer that 260 chars.
- Depending on the device/filesystem used (Hard drive, Network share, fat32, ntfs etc) you may get different results, or even errors.
- You may even notice that Windows Explorer will not always accept a long file name.
- etc.

But I think it's an important step forward, and we will just have to find out what the limits are. That's why it's useful to test this feature.

Frank