New API WindowsLongPath option

[Phil Harvey]

I've released ExifTool 13.01 with a new API WindowsLongPath option.

The hope is to be able to set this by default unless it causes other problems.

Setting this option (with -api windowslongpath) enables support for long path names in Windows as well as wide characters in file names.

This option should be considered experimental, and we need to do lots of testing first if we are to eventually make this the default.

- Phil

[FrankB]

This option should be considered experimental, and we need to do lots of testing first if we are to eventually make this the default.

Great news Phil.

Initial test looks promising, I will keep on using it, and report any disturbances.

[Phil Harvey]

I found a problem, but the same problem exists with 13.00:

If I am in a directory which requires a long path to specify (thanks Frank for the "Long Paths" test file), and I try to write the FileName tag, ExifTool hangs when trying to load the Encode module from a "require Encode" statement in the Sanitize routine.  Unfortunately this problem in Encode is somewhat out of my control unless we can find a patch to avoid using the Encode module.  I can reproduce the hang with this command in the same directory:

perl -e "require Encode"

Require-ing other modules seems to work.  It is just Encode that causes problems.  Same thing happens with both Strawberry Perl and ActiveState Perl.

- Phil

[FrankB]

Too bad that doesn't work.
But the new option doesn't make it worse!

Let me know if I can help.

Meanwhile I have checked Windows versions 11, 10, 8, 7 and even XP. They all work, except for renaming of course.

Frank

[FrankB]

Did some tests on Windows 10 Pro 64 bits, and it all works!

Renamed some (long path) files using ExifToolGui. OK

Then I replayed it using the CMD prompt. OK

- perl -e "require Encode" OK. No Hang.
- exiftool -createdate *.jpg.
  Without the API 2 files could not be read, with the API all 6.

- exiftool -filename<${Exif:CreateDate} %f.%e" -d %Y%m%d (Some expression to prefix the createdate)
  Without the API 3 files could not be renamed, with the API all 6.

Are you using Windows 7 64bits?

Attached the output of the CMD prompt as txt files.
require.txt
createdate.txt
rename No WindowsLongPath.txt
rename WindowsLongPath.txt   

[FrankB]

For anyone that wants to test this feature.

- If you are using ExifToolGui. You can add the API option via 'Options/Custom options' type in '-API WindowsLongPath=1' and press OK. To get rid of it, just empty the edit box.



- If your drive supports 8dot3 names, ExifToolGui/Windows can fool you by substituting the 8dot3 name if the 'real' name gets to long. The will typically show as something like: 122CF5~1.JPG.
The utility fsutil can be used (When you're Admin, and brave enough) to query, and change that setting.
Note that this setting only takes effect for newly created files. Existing files will not be affected.

Code Select Expand

Microsoft Windows [Version 10.0.22631.4317]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\System32>cd \

C:\>fsutil 8dot3name query c:
The volume state is: 0 (8dot3 name creation is ENABLED)
The registry state is: 2 (Per volume setting - the default)

Based on the above settings, 8dot3 name creation is ENABLED on "c:"

C:\>fsutil 8dot3name set c: 1
Successfully DISABLED 8dot3name generation on "c:"

C:\>fsutil 8dot3name query c:
The volume state is: 1 (8dot3 name creation is DISABLED)
The registry state is: 2 (Per volume setting - the default)

Based on the above settings, 8dot3 name creation is DISABLED on "c:"

C:\>fsutil 8dot3name set c: 0
Successfully ENABLED 8dot3name generation on "c:"

C:\>fsutil 8dot3name query c:
The volume state is: 0 (8dot3 name creation is ENABLED)
The registry state is: 2 (Per volume setting - the default)

Based on the above settings, 8dot3 name creation is ENABLED on "c:"

C:\>

[Phil Harvey]

Hi Frank,

Thanks for running these tests!

- perl -e "require Encode" OK. No Hang.

Interesting.

Are you using Windows 7 64bits?

I'm running a Windows 10 virtual machine on my Mac.  I need to do more playing around with this.

- Phil

[FrankB]

I'm running a Windows 10 virtual machine on my Mac.  I need to do more playing around with this.

OK. Surely there's difference somewhere, but what? Let me know if I can do anything to help.

Meanwhile I did some more testing. On an old Windows 8 image. Works like a charm, also renaming.

But... There is something I found using network shares. UNC paths, not using drive letters. What's bothering me is that I think I've put you on the wrong foot with my examples. Sorry for that.

The problem is that UNC paths need this form of prepending: \\?\UNC\Server\share.

https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

[Phil Harvey]

I had an idea about the hang.  I had put the LONG DIR inside a deeper directory so the full path itself then exceeded the long threshold.

I'll consider your UNC point when I have a bit of time

[Phil Harvey]

I read what you said about UNC paths, and don't understand how you think you put me on the wrong foot.

Do you have a UNC drive you can test?  Does ExifTool fail for these?  If so, what is the output of this command after you cd to one of the directories on this drive?:

perl -e "require Cwd; print Cwd::getcwd()"

or if it is easier to run exiftool:

exiftool -p "${filename;require Cwd;$_=Cwd::getcwd()}" FILE

- Phil

[FrankB]

I read what you said about UNC paths, and don't understand how you think you put me on the wrong foot.

In the mail I sent you, there was also a modified version of 'encodeFileName' that I used for testing. That does not prepend UNC properly. I thought you had used that.

Do you have a UNC drive you can test?  Does ExifTool fail for these?  If so, what is the output of this command after you cd to one of the directories on this drive?:

Will prepare that and attach it as a txt file.

perl -e "require Cwd; print Cwd::getcwd()"

or if it is easier to run exiftool:

exiftool -p "${filename;require Cwd;$_=Cwd::getcwd()}" FILE

That doesn't work in Windows. You cant open a CMD, or PS prompt and change dir to an UNC path. Or more correct, I wouldn't know how to do that. (Also a reason why I didn't notice it earlier)

Frank

[FrankB]

Here is my UNC test.

The directory structure is on a hard drive (NTFS) drive K:
Copied to share \\10.10.10.12\foto that is also mapped as drive Z:

See directory structure.txt for details
Directory structure.txt

I executed 3 commands, working directory c:\users\xxxx\Desktop. A simple command -Filename and supplying the complete path. As mentioned in my previous post, for UNC paths that is the only option if you execute if from a CMD window.
(For ExifToolGui it is possible, to have a UNC path as working directory)

The results show that the command fails for the UNC path.

Drive K:
Harddrive.cmd

Drive Z:
Netuse.cmd

UNC:
UNC.cmd

Results:
Results.txt

Of course I had a look at the ExifTool code, and I was able to fix it by modifying WindowsLongPath.

When called from the desktop (CWD is c:/users/xxxx/Desktop) it returns '\\?\C:\Users\xxxx\Desktop\10.10.10.12\foto\LONG....'.

If called from GUI (CWD is //10.10.10.12/foto...), it returns '\\?\\\10.10.10.12\foto\LONG....

In both cases it should return '\\?\UNC\10.10.10.12\foto\LONG ......'

I would be glad to share my fix, but am afraid that that would take away the fun for you.

Frank

Edit: Fixed typos and corrected slashes in CWD (Windows!)

[Phil Harvey]

Hi Frank,

Thanks for running these tests.

I would be happy to see your fix.  I don't understand under what cases the "\\?\UNC\" should be added.

The code I have implemented is based on the Github PR, but implemented in a way similar to the code you sent me.  However it has the advantage that "." and ".." are (hopefully) handled properly.

Reading this page, it seems that UNC paths may also have relative path components ("." and ".."), so we'll have to consider these here too.

- Phil

Edit: Added link to Github PR

[FrankB]

I would be happy to see your fix.

Coming up, see the attachments.

I don't understand under what cases the "\\?\UNC\" should be added.

If you mean 'requirements', it's on the page 'Learn Microsoft' page you referred to.
If you mean 'in code', I have identified 2 cases in my fix. For the test cases I could think of that's enough. I'm not quite sure if I have covered all.

For these tests I concentrated on the UNC case from the previous tests that failed. I have put 'print' statements (not indented on purpose to make them stand out) in the code showing the input, and what is returned.

Test 1: From Cmd prompt using a Full path.
Test 2: From ExifToolGui using a relative path.
Test 3: From ExifToolGui using a relative path starting with '.'.
WindowsLongPath.txt
Test 1.txt
Test 2.txt
Test 3.txt

Tests 2 and 3 were done from ExifToolGui, because that was the easiest for me. Output from these tests is taken from the Log Window. You have to believe that GUI doesn't mess up the output.

Frank

[Phil Harvey]

Hi Frank,

Thanks.

If you mean 'requirements', it's on the page 'Learn Microsoft' page you referred to.
If you mean 'in code',

I meant in the code.

If I understand, it looks like the first "\" must be removed and "\\?\UNC" must be added and the start if the path begins with "\\" but not "\\?\".

I'm wondering why you tested for $path =~ /\\\\/ rather than $path =~ /^\\\\/.  Can the "\\" appear in the middle of the path too?

- Phil