ExifTool Forum

Hi Phil,

after spending a lot of time with the pp PAR Packer sources (not for the faint-hearted!) and asking for advice in the par@perl.org list (in vain), I decided not to investigate further how to make pp better but to provide an alternative.

IMO, the main problem with pp is that it unpacks executables and scripts to the Windows temp directory which causes several problems:

• Unwanted change of files by maintenance tools (e.g. cleaning the temp tree) or even malware.
• Running executables in temp is considered suspicious by AV products and might be even blocked by SRPs
• Cluttering temp with old installations

I found other programs being affected, and there are patches tinkering with the symptoms, e.g. "biber" files vanished from the cache directory, the workaround was to put a back-dated "canary file" in the cache directory: https://groups.google.com/forum/#!topic/perl.par/_obYvAgHeEc (hmm: "the code base is very brittle and prone to breaking stuff at a distance").

To get the "right" solutions, we need to define requirements. Hard to estimate the average Windows user, but I have an idea what the application shall do:

• The executable files shall be in a location being controlled by the user. Rationale: The user can decide whether to write-protect them or not, simple maintenance, no obfuscation an hidden leftovers.
• exiftool shall be optionally reachable via the search path
• Calling exiftool shall be consistent with the existing version, IOW no "perl.exe exiftool parameters"
• Several versions of exiftool must be able to co-exist. Rationale: Tools like IMatch want a specific version.

I don't think that it is required to hide the quite normal fact that a program consists of a lot of files, IOW we don't need to make the user believe to run just one monolithic .exe. Better think about a simple "installer".

So the programming question is: How can I run exiftool in a non-intrusive manner?

As far as I see, it doesn't require more than the files already present (e.g. in the packed exiftool.exe) plus a small exe calling the Perl interpreter in the perl5xx.dll.

See the small C program attached. It takes it's execution path and mangles it to point to a script name in the same directory. Then it invokes a Perl interpreter the same way as perl.exe does, passing the script name and all user provided parameters.

How to use?

• Choose a naming convention and change pathreplace[] if desired
• Compile it with libperl5xx.a and the header files from your Perl distribution
• Put the resulting exe and perl5xx.dll in the same directory as your Perl script
• Put the required Perl libs in the "lib" subdirectory (automatically added to @INC by Perl)
• Run the exe.

To use it with exiftool, I used the files unpacked from the exiftool.exe PAR archive. This seems to work but I think there is some redundancy in it (duplicate files), so there is room for improvement. But at the moment, it's good as a proof of concept.

The automated tests don't work with an exe version, therefore I did no thorough test.

What do you think?

Oliver

Small improvements added to the C sources: By renaming the exe to "perl.exe", it behaves like the standard perl.exe (e.g. to run the automated tests).

With a little helper script, it runs also the test suite.

I added a simple installer using NSIS. This way one can easily add the directory to the path. The forum doesn't accept the compiled installer, though.

If a user dislikes the installer, he can unpack the exe with 7-Zip.

BTW: By using a self-compiled exe, it's very easy to add version info and an icon. to exiftool.exe

the latest version can automatically find a perl5*.dll and load it, IOW use "explicit linking" instead of "implicit linking".

This way, we don't need the include files and libperl5xx4.a from the Perl distribution, and the resulting .exe can be used universally.

PAR/pp generated Windows exectuables can easily be converted to straightforward packages.

Future version will be published here: https://oliverbetz.de/pages/Artikel/Portable-Perl-Applications

Oliver

added a batch file to automatically convert a pp packed "exiftool.exe" to a "unpar-ed" installation.

Extract the ppl.zip archive contents to a new directory and drag/drop exiftool.exe onto unpar.cmd

This will run exiftool once, you need to press Q to stop the help output.

As soon as exiftool is finished, unpar.cmd builds the new package and deletes the temp files.

the last version was designed to hold perl5xx.dll and the Perl script in the same directory as the exe, which would cause collisions if more than one application packed this way is in the same directory. I think that that was not a good design decision.

The new version can deal with replacement strings containing a directory name to get a structure like this:

exiftool.exe
exiftool_files/exiftool.pl
exiftool_files/perl524.dll
exiftool_files/lib

The "installation directory" now contains only exiftool.exe and the directory "exiftool_files".

I deliberately didn't implement a dynamically constructed directory name for this configuration, so you explicitly have to set the directory names in the c sources.

As a result, the name of the exe has no influence and can be chosen / changed freely.

Use unparx.cmd to convert exiftool.exe to such a package.

Things have finally settled down a bit for me and I've got a chance to read through all this.  Thank you very much for thinking about this problem and coming up with a solution.  This is something that I may think about implementing in a future version of ExifTool.  Honestly though, my real focus is metadata, which is why I chose Perl for ExifTool in the first place (ie. almost no compiler/platform issues -- compare this to other metadata projects like Exiv2 which spend most of their time dealing with compilation issues).  In this respect, Windows has been a thorn in my side, but it seems like a majority of ExifTool users run this platform so I must deal with it, and I appreciate the work you have done on this because it isn't likely something I would ever get around to doing otherwise.

- Phil

Quote from: Phil Harvey on June 07, 2019, 07:33:00 AM
Honestly though, my real focus is metadata, which is why I chose Perl for ExifTool in the first place (ie. almost no compiler/platform issues -- compare this to other metadata projects like Exiv2 which spend most of their time dealing with compilation issues).

Don't know much about Exiv2, but Perl itself has issues with Windows and pp adds more. My motivation are the resulting Windows issues of ExifTool. It can't be that users try to run ExifTool with elevated privileges or disable the virus scanner (especially because ExifTool resides in world writable %temp%).

Quote from: Phil Harvey on June 07, 2019, 07:33:00 AM
In this respect, Windows has been a thorn in my side, but it seems like a majority of ExifTool users run this platform so I must deal with it

That's it. And ExifTool is integrated in many great applications running (also) under Windows like IMatch, Geosetter, Affinity, FastPhotoTagger. BTW: Especially for this "integrated" use of ExifTool, the "pp-less" operation would be much cleaner!

My current status: The "unpacked" exiftool.exe currently works pretty well for me, and I'm also experimenting with Strawberry Perl.

Can you estimate the coverage of the automated tests? Or tell me what should be tested in addition to them?

From the past, I know there can be subtle differences between Perl distributions, therefore it would be good to know what the automated tests do not cover.

For example, I generally avoid "unusual" path/file names with spaces and non-ASCII characters so I did no thorough tests regarding this "minefield". But I know that other people don't care about this.

The parameter mangling in the pp bootstrapping (boot.c (https://github.com/rschupp/PAR-Packer/blob/master/myldr/boot.c) -> shell_quote) might influence quoted parameters, so this should be also tested.

There are known problems with daylight saving time zones, and Cygwin handles them differently. Likely ExifTool doesn't rely on the built-in file time functions, correct?

I'm not sure whether Test::Harness::runtests() has all necessary capabilities, maybe Windows specific tests are needed (e.g. with a cmd file or even PowerShell.

Besides, runtests() seems to rely on access to "perl.exe", therefore it doesn't handle the specifics of a Windows exe with a different name.

Oliver

Quote from: obetz on June 07, 2019, 12:45:21 PM
Can you estimate the coverage of the automated tests? Or tell me what should be tested in addition to them?

They test the API fairly well, but don't test the "exiftool" application at all.

QuoteFor example, I generally avoid "unusual" path/file names with spaces and non-ASCII characters so I did no thorough tests regarding this "minefield". But I know that other people don't care about this.

The tests don't cover this at all, but spaces shouldn't be a problem.  However, special characters in Windows filenames continue to be a problem.

QuoteThere are known problems with daylight saving time zones, and Cygwin handles them differently. Likely ExifTool doesn't rely on the built-in file time functions, correct?

It does use built-in time functions, but doesn't rely on them, and in general the tests are designed to pass even with time zone differences, etc.

QuoteI'm not sure whether Test::Harness::runtests() has all necessary capabilities, maybe Windows specific tests are needed (e.g. with a cmd file or even PowerShell.

Yes.

- Phil

Updated version with a minor cleanup allowing Strawberry Perl (needing several DLLs) to be used in a subdirectory.

For a brief period, I make an alternative ExifTool version available here: https://oliverbetz.de/exiftool-11.52_with_tests.zip (Edit: removed the installer due to virus scanner false positives)

I used the NSIS installer, so it can be extracted with 7-Zip or similar.

It's based on current Strawberry Perl 32 bit because this seems to be better maintained than AS and has unambiguous licensing.

Although it passes the selftests, I would consider it "beta". There is so much "magic" in pp that it's hard to predict whether something behaves different.

I had to make a minor patch to windows_exiftool to enable again to embed parameters in the filename (although I'd prefer links).

Oliver

Hi Oliver.

After all the hard work you put into a project like this, in the end you may have a problem distributing an exe file like this, and getting people to actually feel comfortable testing it.

I naturally always run "any" exe file that I download through VirusTotal (https://www.virustotal.com/gui/file/3dc0b5eea05591066a2d57757ce0125fb93ee26c645415c387e53c7008a5543b/detection) and if I get more than about 1x possible warning of Malware/viruses, even "if" I think it may just be false positives, standard operating procedures say: "Don't run it!".

So I am too hesitant to run it. Sorry. Currently has 4x warnings (Yes I know, most probably just installer false positives... but...?? :-\ )

Even Phil's Windows unzipped exiftool(k).exe (v11.52) has one engine detect "something" (albeit benign): VirusTotal output (https://www.virustotal.com/gui/file/c98ca5837dba5e0c78ff64ac6b6f76b293c62a7811aff614e603425ca8a38eb5/detection) (currently 1x warning)


EDIT:
I only just read this now: Running exiftool from Windows GUI (https://exiftool.org/forum/index.php/topic,10206.msg53385.html#msg53385).

So unfortunately I doubt you'll get many folks willing to test it, until you get it signature verified, which is probably both a hassle and time-consuming.

well, this seems to come from NSIS (Nullsoft Scriptable Install System) https://nsis.sourceforge.io/ I removed the installer.

https://oliverbetz.de/exiftool-11.52_with_tests.zip has the same contents and no virustotal positives (check it by unpacking the exe with an archiver). Unpack it where you like and set the path variable if you want.

The archive contents are just a mix of ExifTool (tarball) and parts of Strawberry Perl plus my tiny exe (source code also posted).

Oliver

Ok Oliver. I'll do that.

Yeah please don't be offended, as I really wasn't expecting that your code was doing anything deliberately nasty, but with those VirusTotal warnings, I'd be stupid to totally ignore them of course, and it will concern other people also.

The end of signature-based virus scanners has long been predicted - they simply can't deal well with the current situation. It could also be that NSIS has been used by evil people in the past (just a wild guess).

Bkav (https://blog.nirsoft.net/2015/10/18/antivirus-statistics-and-scores-according-to-false-positives-of-nirsoft-tools/) and Panda (https://www.av-comparatives.org/tests/false-alarm-test-march-2018/) have been reported to throw false positives (depending on the tester). Only Jangmin (??) didn't like the included uninstaller exe. McAfee Gateway Edition complained but not McAfee - don't they use the same signature database?

But what I've learned is that selecting an installer solution has do be done carefully to avoid problems. NSIS was easy to use, maybe I should check Inno Setup.

Oliver

Oliver. I'm confused.

Is this what would get installed with Phil's regular exiftool(-k).exe file anyway after it was run the first time, but together with a helpful installer to help the n00bies get the first-time install done correctly?

Or, is this the full ExifTool Perl package, for those users who already have Perl installed on their Windows pc. I don't have Perl, nor can I program it.

Quote from: mpegleg on June 20, 2019, 05:36:47 AM
Is this what would get installed with Phil's regular exiftool(-k).exe file anyway after it was run the first time, but together with a helpful installer to help the n00bies get the first-time install done correctly?

Or, is this the full ExifTool Perl package, for those users who already have Perl installed on their Windows pc. I don't have Perl, nor can I program it.

Neither of them. It is Perl (portable) bundled with ExifTool (source distribution).

The main difference between the Windows Executable you get from https://exiftool.org/ and my package is that I don't use the PAR pp packer, therefore avoid Potential PAR pp problems (https://oliverbetz.de/pages/Artikel/Portable-Perl-Applications#Potential_problems_using_the_PAR_Packer_pp) already mentioned.

The exe generated by PAR/pp is a self-extracting archive, unpacking ExifTool + Perl (hundreds of files) silently to %temp% during the first call.

In contrast, my package allows the user to unpack all files to a directory of his choice (before the first run, of course).

Anther difference is the way Perl is called:

pp has a two-stage calling process, this means exiftool(...).exe spawns a second process (exe in temp dir). For this spawn, it needs to mangle the command line arguments, set environment variables, execute a 20kb perl one-liner (too complicated for me to understand completely) etc.

In contrast, my Perl launcher (52 kb small ExifTool.exe in the archive) uses exactly the same code as the original Perl.exe from the Perl distribution. I just set the DLL search path and insert exiftool.pl as first argument. Only a few dozen lines of C code including error checking.

I think that's the simplest and cleanest way possible to run ExifTool in a windows environment.

Oliver

O.k. You kind of lost me after the first PAR/pp bit. But that's cool. I don't think I'm your intended target for this discussion anyway, so I'll melt away into the background and lurk as I do, and watch the discussion unfold with all you very clever people.

But do let me know when you have it all working, and I'll give it a rip.  :)

the only challenge is to extract the zip archive to a place of your choice

Now I tried Inno Setup. Didn't help much against false positives, but the installer is interesting since it allows to chose "install for me" or "install for all" (requesting administrative rights) out of the box.

My demo automatically sets a suitable directory (program files for "all" and localappdata for "me") then.

In addition, the user can select whether to add ExifTool to the path and whether to have an uninstaller and an entry in the add/remove panel.

Anybody willing to try it?

https://oliverbetz.de/exiftool_install_11.52_Inno.exe the new Inno Setup based installer
https://oliverbetz.de/exiftool_install_11.52_NSIS.exe the NSIS based installer with less options

The installer sources are attached to this posting.

I tested them thoroughly, but since they modify the registry (add/remove path and uninstaller), use at your own risk.

About false positives: This is a well known problem for both Inno Setup and NSIS, caused by crappy antivirus products and malware being packed with NSIS or Inno. The not so experienced user might be scared if he uses virustotal and gets false positives from dumb scanners.

Windows SmartScreen might be another annoyance. Signing with a free certificate (e.g. https://www.globalsign.com/en/ssl/ssl-open-source/ ) could help. But a good reputation can also be gained by a large number of downlads as exiftool.exe demonstrates.

Oliver

Very nice!  I ran both installers and uninstallers.  They worked very well, but I liked the Inno interface better.

The Inno install put ExifTool in \Users\MyName\AppData\Local\Programs, while the NSIS install put it one level up (in Local).  Any reason for this difference?

I ran some basic tests, but it is not possible for me to test for compatibility with all of the many ways that people are using ExifTool out there, so introducing this type of installer means that I would have to prepare two Windows versions for each release, at least for a while, so people can fall back to the old version if this doesn't work for them.

How difficult is it to prepare the install package?  Currently I have the whole release process scripted to make it very easy for me, and in Windows I just type two commands, hit the return key and click the mouse about 3 times and that's it.

- Phil

From experience with NSIS, the installer creation is fully automatic. Heck, you can even create a Windows installer under Linux (iirc including Mac). Very handy.

Also, I would have to figure out how to change the install to allow the drag-and-drop features of exiftool(-k).exe, since many people use ExifTool without the command line.

- Phil

Quote from: Phil Harvey on June 21, 2019, 09:30:28 AM
Very nice!  I ran both installers and uninstallers.  They worked very well, but I liked the Inno interface better.

Yes, also my opinion.

NSIS has also a modern layout but I was too lazy to investigate it.

Quote from: Phil Harvey on June 21, 2019, 09:30:28 AM
The Inno install put ExifTool in \Users\MyName\AppData\Local\Programs, while the NSIS install put it one level up (in Local).  Any reason for this difference?

When I made the NSIS script, I wasn't aware of the Programs directory. I just knew that Google Chrome installed to %localappdata%\Google\Chrome\Application\ so I did the same.

Inno Setup has an automatic constant/variable pointing to %localappdata%\Programs for non-admin installs, and the docs state that this is the way to go.

Quote from: Phil Harvey on June 21, 2019, 09:30:28 AM
How difficult is it to prepare the install package?  Currently I have the whole release process scripted to make it very easy for me, and in Windows I just type two commands, hit the return key and click the mouse about 3 times and that's it.

There are 2..3 steps depending on your expectations:

0. Make a exiftool.exe launcher with version information. This is only needed if you want exiftool.exe to reflect the version. Update the resource file and run the compiler / make.

BTW: It's pretty easy to incorporate an icon in the exiftool.exe. May I use your favicon for my demonstration?

1. Prepare the files to be packed. You can use pp and unpack it with the cmd file I provided, but I preferred to copy the unmangled files from Strawberry Perl and Exiftool.

The Perl part is rather static (unless you need new modules), so it needs to prepared very seldom.

The ExifTool files are simple merged / copied to the Perl files.

I will open a new thread about the Windows specific version.

2. Run Inno Setup or NSIS. I didn't yet look at the mechanism to get the version information fed to the installer. It might be possible to use command line arguments, else a config file has to be created and included be the installer script.

Quote from: Phil Harvey on June 21, 2019, 10:23:08 AM
Also, I would have to figure out how to change the install to allow the drag-and-drop features of exiftool(-k).exe, since many people use ExifTool without the command line.

This is still working with my package. Just rename the exe as ever and the parameters should be used. The quick hack I made ($^X instead of $0) can be made cleaner.

Since the exe is so small, making copies with different parameters is cheaper than ever.

But Inno Setup allows even more sophisticated things. I didn't check this in detail, but as far as I see, you can create Windows links wherever you want.

For example, it should be possible to but a link to ExifTool in the Sendto directory, adding some parameters (-k).

I consider links the better way to pass parameters to ExifTool. If people have a working example to be modified, they should understand the method.

Oliver

I tried the Inno install.  Straightforward.  No surprises.

First test:

Windows 7 (64-bit), only one account on this computer, an Admin account:  Installed for only the user into a folder I created in my Portables folder structure, installed everything, took all the defaults.  I noted the path was entered under User variables for ....

I opened a command prompt in a folder on another drive and entered exiftool -all= *.jpg.  Didn't work.  I thought it should, but I might be assuming things.

Uninstall.

Second test:

Windows 7 (64-bit), only one account on this computer, an Admin account:  Installed for all users, installed into "C:\Program Files (x86)", installed everything, took all the defaults.

I opened a command prompt in a folder on another drive and entered exiftool -all= *.jpg.  Worked.

ExifToolGUI works.

Next tests will be on a Windows 10 computer with Admin and Restricted accounts.

thanks for the report!

Quote from: allsan8 on June 22, 2019, 01:43:21 PM
Installed for only the user [...]  I noted the path was entered under User variables for ....

do you consider this unexpected behavior?

IMO "only for me" should write exactly my environment.

Installing "for all" writes the path to HKLM (requires admin privileges).

Of course you can select "for all" and change the installation directory to whatever you consider well suited. The predefined path is just an educated guess what people might want (if they care at all)

I consider improving the texts describing both options.

Quote from: allsan8 on June 22, 2019, 01:43:21 PM
I opened a command prompt in a folder on another drive and entered exiftool -all= *.jpg.  Didn't work.  I thought it should, but I might be assuming things.

Can you elaborate "Didn't work"?

Didn't you get any message?

exiftool -all= *.jpg is somewhat hefty, what's the intention?

Quotedo you consider this unexpected behavior?

No.  Just reporting it was there.  Thus, I figured I could perform a command prompt from any folder and it would work and it did not.  I apologize I didn't copy the message when it didn't work, but I am almost sure it is the message I have seen when I haven't had ExifTool in the folder.

QuoteOf course you can select "for all" and change the installation directory to whatever you consider well suited.

Yes.  Saw that, but I took the default, because I thought yeah that is a good place.

Quoteexiftool -all= *.jpg is somewhat hefty, what's the intention?

Strip the metadata.  I use it all the time.

Quote from: allsan8 on June 22, 2019, 02:53:38 PM
I apologize I didn't copy the message when it didn't work, but I am almost sure it is the message I have seen when I haven't had ExifTool in the folder.

Could it be that you had the command prompt open before the ExifTool was finished and closed?

Win7 doesn't propagate environment changes to running command windows.

Type "set" or "path" in the command window to check whether the path variable is updated.

Windows 7 ... I uninstalled the all users install.  Installed this user only.  Installed everything.  Installed in a folder in my portable folder.  Took defaults.

Under Environmental Variables, the location of ExifTool (Value) is included as Path (Variable) in the User variables for "user account" section.  The location of ExifTool (Value) is not included as Path (Variable) in the System variables section.

When I run from a command prompt this is what I get:

'exiftool' is not recognized as an internal or external command,
operable program or batch file.

ExifToolGUI does not work.

Again:

1. After running the installer, you need to open a new command window. Already running command windows are not updated!

2. Type "path" in the command window and check whether the exiftool path is listed correctly.

If the exiftool path is not propagated to the new command window (strange), you could try to log off and on the user. You don't need to reboot, just log off and on.

But on my W7 systems, it is sufficient to open a new command window.

just confirmed on "yet another W7" system:

1. Open a command window #1
2. Install ExifTool
3. Command window #1 doesn't have the path added
4. Open a new command window #2. It has the new path

That's what I tried to express two postings earlier, forgive me if I was not clear in my wording.

Quoteforgive me if I was not clear in my wording

You were clear.  (I avoid the command prompt.   :)  It isn't open unless it has to be open.)  I just uninstalled and installed to see what I was doing wrong.  I think I figured out why it wasn't working here.  After both installs, I immediately checked the path information.  I left those windows open while I tried the command prompt.  This time, I checked the path information, closed out their related windows, opened the command prompt.  Worked.  ExifToolGUI worked.  Sorry about that.  Thank you for your patience and making this available.

Installed on the Windows 10 computer.  Installed in the Admin account.  Chose all users.  Installed everything.  Used all defaults.

Went into the Restricted account ... everything works (that I could quickly test and use most often) without Admin privileges.  Command prompt from folder.  ExifToolGUI.  Programs that open other programs such as ExifToolGUI.

Thank you!

thanks for the feedback.

A benefit of the "for all" installation is that ExifTool is by default installed to %Programfiles(x86)%, there being protected against accidental or malicious corruption.

Oliver

Hi Oliver,

I just learned about your Portable Perl Applications on Windows -- great stuff.

It just occurred to me that, with only a small amount of effort, it could greatly simplify ExifTool's long-standing issues with Unicode filenames on Windows.  To get a feel for those complexities, see my notes attempting to fully document the current ExifTool behavior: https://exiftool.org/forum/index.php/topic,8382.msg43116.html#msg43116.

Summary of the proposal: Change "ppl.c" to use wmain() to accept 16-bit Unicode characters in the argument and environment vectors. "ppl.c" would then convert that 16-bit Unicode to UTF-8 before invoking RunPerl().

Details

I believe the root of the problem is that Windows Perl uses a C-standard main() defined to accept 8-bit char strings:

int main(int argc, char **argv, char **env)

Those 8-bit character strings are interpreted in the current Windows System Code Page.

Internally, Windows uses 16-bit Unicode throughout. When a process is started, it receives its arguments and environment in 16-bit Unicode. The Windows Standard C library converts the 16-bit Unicode argument and environment vector (wchar_t) to 8-bit characters (char) before invoking main(). It  uses the current Windows System Code Page to do the conversion, which will truncate any Unicode characters not in the current system code page.

For example, by default, Windows computers in English (United States) are in code page 1251 Windows Latin 1 (ANSI). If you pass a 16-bit argument vector to Windows Perl containing Unicode characters that aren't in Windows Latin 1, those characters get truncated to 8 bits. This behavior is different from Unix systems, in which UTF-8 is the usual encoding of characters passed in "argv".

The solution is straightforward -- "ppl.c" should use wmain() instead:

int wmain(int argc, wchar_t **wargv, wchar_t **wenv)

It then converts "wargv" and "wenv" from 16-bit Unicode to UTF-8 and passes those converted argument vectors to RunPerl().  I found sample code here: https://github.com/circulosmeos/Perl-with-Unicode-for-Windows/blob/master/runperl.c

This would then allow ExifTool to receive full Unicode command-line arguments from any Windows program, including the Windows Console (cmd.exe), without any changes to ExifTool itself.

Warning: I am not an expert in Windows, Perl, ANSI C, or ExifTool. But I've spent too many hours trying to write Unicode-capable plugins that invoke ExifTool and I've learned way more than I wanted to.

"ExifTool's long-standing issues with Unicode filenames" => "ExifTool's long-standing issues with Unicode command-line arguments"

Hi johnrellis,

it would be great to have this additional benefit.

I will look into this in the next days.

Can you provide a simple test case, IOW the ExifTool command, the expected result and the current result?

Oliver

I packaged up a simple ExifTool test case in this folder: https://www.dropbox.com/s/jqg6szsyhntc3vr/exiftool-unicode-test-2019-06-27.zip?dl=0 . See "test.bat" for a detailed explanation.

"test.bat" also runs two programs, "wecho.exe" and "echo.exe", which use wmain() and main() respectively and which hex-dump their arguments to stdout. This lets you see easily what the console and cmd.exe are passing to wmain() and main().  (The whole business with the console's current code page and the Windows System Code Page can be quite confusing.)

Also, "ppl.c" should use the wide-char versions of the file functions (e.g. FindFirstFile).  See https://docs.microsoft.com/en-us/windows/desktop/learnwin32/working-with-strings. I think this allow "exiftool" to be installed in a directory path containing arbitrary Unicode characters, something that doesn't currently work.

this could be a minefield, full of legacies, strange development decisions, potential side-effects. It will take a  while to sort this out.

Quote from: johnrellis on June 27, 2019, 08:30:00 PM
Also, "ppl.c" should use the wide-char versions of the file functions (e.g. FindFirstFile).  See https://docs.microsoft.com/en-us/windows/desktop/learnwin32/working-with-strings. I think this allow "exiftool" to be installed in a directory path containing arbitrary Unicode characters, something that doesn't currently work.

Do you seriously think somebody should want to install ExifTool (or any other program) in a folder with non-ASCII characterse.g. c:\програ́мма\ExifTool? I don't think that I will actively support this unless somebody points out a very good reason.

Oliver

Hello to all,

please allow to give my 2 cents.
The answer to your question is simple: YES

My friend in Singapore always uses paths with chinese characters.

Best regards
Herb

Quote from: herb on June 29, 2019, 07:01:41 AM
My friend in Singapore always uses paths with chinese characters.

for installing programs?

Or just for his data?

Oliver

Hello,

Afaik, he uses it for both.
Files that are installed for testing (e.g. also Exiftool) have path with unicode characters.

Best regards
Herb

Quote from: obetz on June 29, 2019, 03:12:39 AM
Do you seriously think somebody should want to install ExifTool (or any other program) in a folder with non-ASCII characterse.g. c:\програ́мма\ExifTool? I don't think that I will actively support this unless somebody points out a very good reason.

There are a fair number of Adobe Lightroom plugins from at least several different authors (me and others) that include ExifTool (and other utilities like ImageMagick) bundled in their implementation.  These plugins can (and do) get installed anywhere. Typically they get installed either in C:\Users\user or on an external drive, both of which can have non-ANSI Latin 1 characters.

Sometimes installing ExifTool on such a path works, sometimes not -- it depends on whether the path character is in the current Windows System Code Page.  My experience is that lots of non-English-speaking users end up using path characters that aren't in the current Windows System Code Page, probably because they're using computers that have been configured for English.

Note that if "ppl.c" uses wmain(), it will receive its command-line arguments as 16-bit characters, so using the wide string and file routines is the natural thing to do (and a straightforward textual change using "#define Unicode").

Quote from: obetz on June 29, 2019, 03:12:39 AM
this could be a minefield, full of legacies, strange development decisions, potential side-effects. It will take a  while to sort this out.

Agreed that there's a lot of legacy here. But with some surgical care, your "ppl.c" could significantly improve a very sore pain point with ExifTool.  The typical user would get full Unicode compatibility from the command line merely by doing "chcp 65001" in their batch files.  That's much simpler than the current situation (see my notes I linked above).

(The original sin is the Windows port of Perl using main() instead of wmain().)

Luckily, ExifTool has a clean interface at the command line -- it will accept UTF-8 in all of its arguments, including paths. So that provides a clean interface for "ppl.c".

Quote from: johnrellis on June 29, 2019, 11:36:15 AM
The typical user would get full Unicode compatibility from the command line merely by doing "chcp 65001" in their batch files.

I'm not yet sure about this.

Quote from: johnrellis on June 29, 2019, 11:36:15 AM
The original sin is the Windows port of Perl using main() instead of wmain().

GetCommandLineW() is an alternative to wmain. Many aspects have to be considered...

In the mean, I found that Unicode support of several tools is still, hmm, "not mature".

Even the BMP (Basic Multilingual Plane) isn't handled correctly by several widespread text editors, e.g. Latin-D ꜦꜧꜨꜩꜲꜳꝎ is rendered wrong in PSPad and NPP (Notepad++) as well as in comparison windows of Beyond Compare. Don't ask what my older programming editor does...

Even worse is the handling of characters from higher planes (beyond UCS-2).  This forum software (SMF) told me "The following error or errors occurred while posting this message: The message body was left empty" when I composed this reply initially containing a "Five spoked asterisk" U+1F7AF. Windows Explorer doesn't display the clef U+1D11E.

Complex diacritics like Ỗ (O + Circumflex + Tilde) make funny results.

In the attached ZIP, you find some files with "interesting" names.

Now I know even better why use only 7 bit ASCII and no spaces in file names, and I'm glad that my native language has only few non-ASCII characters easily replaced by ASCII tupels.

Oliver

QuoteI found that Unicode support of several tools is still, hmm, "not mature".

Windows 10 made some significant changes in which fonts get installed by default, and these tools may not have been updated accordingly: https://support.microsoft.com/en-us/help/3083806/why-does-some-text-display-with-square-boxes-in-some-apps-on-windows-1. Or they could be just not setting suitable fallback fonts to display all Unicode characters.

QuoteIn the attached ZIP, you find some files with "interesting" names.

I compared the display of those in Windows File Explorer with Mac Finder, and with the exception of the MUSICAL SYMBOL G CLEF in File Explorer, they all displayed the same. File Explorer's fall back fonts must not include that character.

Quote from: johnrellis on June 30, 2019, 08:29:14 PM

QuoteI found that Unicode support of several tools is still, hmm, "not mature".

Windows 10 made some significant changes in which fonts get installed by default, and these tools may not have been updated accordingly:

the problems of the tools mentioned in my previous post are far beyond missing glyphs.

Even notepad.exe counts two columns for each character in plane 1 (SMP) but is not a powerful editor. At least, it showed no severe display or editing errors.

Sublime worked, but is not (yet?) my preferred editor.

Quote from: johnrellis on June 30, 2019, 08:29:14 PM

QuoteIn the attached ZIP, you find some files with "interesting" names.

I compared the display of those in Windows File Explorer with Mac Finder, and with the exception of the MUSICAL SYMBOL G CLEF in File Explorer, they all displayed the same.

Try "dir" in a cmd window, since we are talking about command line handling...

The command window doesn't seem to have fallback to other (proportional) fonts, and even "DejaVu Sans Mono" (reported to have the best coverage) lacks many glyphs.

Ceterum censeo: I still consider it a risk to install software to directories with Unicode path names. I'm so old that I prefer to stick with the "POSIX portable filename character set".

I'm sorting out now some subtle issues and likely will publish then a Unicode-enabled launcher.

Re editors, I primarily use Sublime on Mac, and I've had no issues with its Unicode support, but I haven't used it on Windows. On Windows, I do a limited amount of editing with Unicode on Visual Studio 2017 and Word, and no issues there. I used TextPad on Windows for code editing (including the occasional non-ANSI Latin characters) up until 2014, and it was OK.

Re the command console, when I need to work with a larger range of languages, I use NSimSun, which of the seven fonts pre-installed in the console for English (United States), seems to have the best language coverage.  It doesn't cover many of the more exotic symbols.

(My Lightroom plugins that include ExifTool are used in 112 countries, so I've had a fair amount of experience troubleshooting problems in many other languages, including Cyrillic and Asian languages. This motivated me, with Phil's help, to figure out exactly how the Windows console, main(), Perl, and ExifTool interacted.)

I experimented with modifications to "ppl.c" to use wmain() and encode the argv in UTF-8 before passing it to ExifTool, and in my testing so far it works fine (as long as ExifTool's -charset is set to the default "utf8").   

To support ExifTool's current semantics, a small change would need to be made to ExifTool. So to decouple the Unicode issue from your proposed "ppl.c" launcher (which is very worthwhile independent of Unicode), I posted a separate proposal for Unicode: https://exiftool.org/forum/index.php/topic,10246.0.html

You can see my modified version here: https://www.dropbox.com/s/prdk5yyqsb2j40t/ppl-2019-07-01.c?dl=0. I was focusing on the ExifTool use case, not the more general use cases you're supporting.  A diff should cleanly highlight the changes:

- Using "wchar_t" instead of "char" for command-line arguments and file paths.
- Providing encode(str, codepage), which encodes a wide string into any code page (e.g. UTF-8 or the current system code page).
- Using ARRAYSIZE instead of "sizeof" to compute the number of elements in a wchar_t array.
- Using the wide-char versions of the string library rountes (e.g. "wcsrchr" instead of "strrchr").
- Using the printf format code %S instead of %s for wide strings.

- Converting "env" from UTF-16 to the current system code page, so that RunPerl receives it just as it would from main().

- Converting argv [2 - argc-1] from UTF-16 to UTF-8, and converting argv [0 - 1] to the current system code page. Windows Perl always opens scripts with the 8-bit file i/o routines, regardless of whether -Ci is specified, so RunPerl needs to receive the script argument encoded in the current system code page, just as it is passed by main().

There wasn't any need to change the names of the file routines, since I compiled with UNICODE defined in the preprocessor. (Visual Studio did that for me -- I don't know how that works precisely with gcc.)

Quote from: johnrellis on July 01, 2019, 05:35:23 PM
I experimented with modifications to "ppl.c" to use wmain() and encode the argv in UTF-8 before passing it to ExifTool, and in my testing so far it works fine (as long as ExifTool's -charset is set to the default "utf8").   

You can see my modified version here: https://www.dropbox.com/s/prdk5yyqsb2j40t/ppl-2019-07-01.c?dl=0. I was focusing on the ExifTool use case, not the more general use cases you're supporting.

Starting with simple tests, with plain Perl and a diagnostic script, shows the details and saves a lot of time.

I converted your source file from (unusual?) UTF-16 LE BOM to ASCII and replaced the wide function calls by universal calls so the same sources can be used for Unicode and 8 Bit.

I enabled the perl.exe mode and added some debug statements.

Find sources and diagnostic script attached.

Quote from: johnrellis on July 01, 2019, 05:35:23 PM
- Converting "env" from UTF-16 to the current system code page, so that RunPerl receives it just as it would from main().

"just as it would from main()"? Or just as 8 Bit?

I tried to feed Perl with UTF-8 encoded environment with funny results: The names are taken in UTF-8, but not the values.

Try env vars with Unicode names, Unicode values and both, e.g.
Set Φοο=Bar
Set Foo=Βαρ
Set Βαρ=Φοο

I didn't manage to get perl info.pl e€e to output all parts correctly.

Quote from: johnrellis on July 01, 2019, 05:35:23 PM
- Converting argv [2 - argc-1] from UTF-16 to UTF-8, and converting argv [0 - 1] to the current system code page. Windows Perl always opens scripts with the 8-bit file i/o routines, regardless of whether -Ci is specified, so RunPerl needs to receive the script argument encoded in the current system code page, just as it is passed by main().

In my understanding, -Ci is not intended to resolve this. Besides, you can have valid Unicode file paths not accessible via any 8 bit code page. Since I don't see much value in running software in "c:\users\Влади́мир Παπαδόπουλος" I will not investigate more work here.

Quote from: johnrellis on July 01, 2019, 05:35:23 PM
There wasn't any need to change the names of the file routines, since I compiled with UNICODE defined in the preprocessor. (Visual Studio did that for me -- I don't know how that works precisely with gcc.)

There are universal macros resolving to the short or wide function depending on UNICODE or _UNICODE.

TEXT() or _T() makes wide or short literals.

An uppercase %S format specifier is a non-standard alias to %ls but gcc assumes wide arguments even with lowercase %s.

Oliver

I forgot to mention an important observation: The unicode exe hangs just before the "return" for several tenths of a second.

The delay doesn't happen if the RunPerl() line is commented. So calling RunPerl() in the Unicode environment has a delayed side-effect, really weird! I will postpone this Unicode experiment until new information is available.

Oliver

P.S.: I updated the (traditional non-Unicode) Strawberry Perl  based ExifTool Windows Installer to version 11.54:
https://oliverbetz.de/ExifTool_install_11.54.exe

"I converted your source file from (unusual?) UTF-16 LE BOM"

That's the default encoding used by Visual Studio.

Re the delay on exit, do you observe that with gcc?  I now see a subsecond delay with Visual Studio.

I installed mingw-w64 and compiled and linked with this:

gcc -DUNICODE -D_UNICODE -DEXPLICITLINKING -municode -m32 -o exiftool.exe ppl.c

and I don't observe any delay. Interesting.

MinGW also here: "gcc (i686-posix-dwarf-rev0, Built by MinGW-W64 project) 8.1.0"

The delay is caused by the "-O2" optimization switch. The options I used initially:


CFLAGS    := -Wall -Wstrict-prototypes -O2 -s -mms-bitfields -fwrapv -municode -D_UNICODE
# -O2 Optimize even more, performs nearly all supported optimizations that do not involve a space-speed tradeoff
# -Wstrict-prototypes is not needed if a reasonable static code checker is used. Wall might (!) catch problems not detected by PC-Lint
# -s Remove all symbol table and relocation information from the executable, reduces size by more than 50%!
# -mms-bitfields Microsoft compatible bitfields
# -fwrapv instruct the compiler to assume that signed arithmetic overflow wraps, else "i+1 > i" is always true


Without any -O statement, it exits fast. Really weird.

We don't really need the optimization, but as long as I don't know how the delay is possible at all, I don't trust the code.

The delay is caused by incorrectly declaring RunPerl_t with the CALLBACK calling convention (a macro for __stdcall):

typedef int (CALLBACK* RunPerl_t)(int argc, char **argv, char **env);

RunPerl in "perl524.dll" was compiled with the _cdecl calling convention. Deleting CALLBACK (causing __cdecl to assumed) eliminates the delay, with and without O2, in both Visual Studio and mingw-w64 "gcc".

Details

I verified that -O[123] with "gcc" also caused the delay.  I looked at unoptimized and optimized assembly and noticed a difference in how they implemented "return". The unoptimized code pops the wmain() stack frame by subtracting a constant from the frame pointer saved away on entry to  wmain():

leal   -12(%ebp), %esp   

Whereas the optimized code pops the stack from by adding a constant to the stack pointer:

addl   $636, %esp   

This suggested that perhaps some function called by wmain() wasn't properly restoring the stack pointer on return. In unoptimized code, this wouldn't matter, since wmain() would properly restore the stack point from the saved-away frame pointer regardless.  But in optimized code, wmain() would return with an improperly restored stack pointer.

I read up on the calling conventions: In __stdcall, the called function pops the arguments from the stack before returning, while in __cdecl, the caller pops the arguments after the function returns.  So the obvious suspect was RunPerl.

The perl-5.30.0 sources show this declaration for RunPerl in "perllib.c":

EXTERN_C DllExport int
RunPerl(int argc, char **argv, char **env)

The DllExport macro is defined in "win32/win32.h" as "__declspec(dllexport)", which doesn't specify a calling convention.

I didn't track down the compile flags that the distribution would set by default for compiling "perllib.c", but I assumed they wouldn't override the default calling convention (__cdecl).  But using the Visual Studio Disassembly window, I did verify that RunPerl in "perl524.dll" uses the __cdecl convention, not popping its arguments from the stack.

I don't know why this bug caused wmain() to delay on return but not main(). But strange behavior is always to be expected when called functions mangle the stack.

Quote from: johnrellis on July 05, 2019, 04:40:18 PM
The delay is caused by incorrectly declaring RunPerl_t with the CALLBACK calling convention (a macro for __stdcall):

typedef int (CALLBACK* RunPerl_t)(int argc, char **argv, char **env);

thanks for spotting this stupid copy/paste error. The installer has been updated.

Caution: The "unicode" version is to be considered "experimental".

Oliver

this is an answer to a post in another thread, since I think that it belongs to this thread:

Quote from: herb on August 10, 2019, 06:31:12 AM
(Windows)Exiftool.exe is also used by other applications (e.g. XnViewMP or IMatch) and the *.exe file is stored in a subdirectory of the application installation directory.
For me an advantage is that all these Exiftool.exe files use the "unpacked Perl modules" in ONE common directory - in %appdata%\... or defined by global environment variable PAR_GLOBAL_TEMP.

At the moment I do not see how this can be achieved using your Ppl(exiftool).exe.
All necessary files stored in directory "exiftool_files" should be stored only once on a Windows system.

First of all: PAR_GLOBAL_TEMP is deprecated and dangerous especially for the scenario you describe. Don't use it.

Without PAR_GLOBAL_TEMP, having the executables in %temp% has also several drawbacks. See https://oliverbetz.de/pages/Artikel/Portable-Perl-Applications In the end, you waste even more space because there is no automatic cleanup of old versions.

IMatch prefers to have it's specific version of ExifTool because there are certain version dependencies and there is a risk breaking IMatch by using a different ExifTool version than shipped with IMatch, so don't fiddle with the embedded ExifTool unless there is a good reason.

Other applications like ExifTool Gui can use an "installed" ExifTool added to the "Path" environment variable.

I didn't yet investigate the ExifTool integration of XnViewMP.

Let me know whether you use both IMatch and XnViewMP or any other set of applications bringing their own ExifTool, then I will check for impact and options.

Any other concern you think I should investigate?

Oliver

Hello Oliver,

thanks for your quick reply and sorry that I asked within another post.
Asking my question I thougth on a small enhancement of (Ppl)Exiftool.exe.

Today it starts Perl.exe that is stored in the subdirectory "exiftool_files" of its installation directory.
I thought about an global environment variable or an *.cfg file (parallel to *.exe) or ... that points to Perl.exe (to be used) - with an absolute path.

In my case it would allow to have only "1 installation" of (Ppl)Exiftool.exe for XnViewMP and my application.
I agree that some more investigations have to be done for IMatch.

Only a thougth
Best regards
Herb

(Again) Hello Oliver,

I have an additional question to default config file .exiftool_config.
Why must this file be stored inside subdirectory exiftool_files using your (Ppl)Exiftool.exe?

It is not used if it is stored inside the current directory parallel to (Ppl)Exiftool.exe.
This makes my goal "to have only 1 installation of exiftool_files" a little bit more complicated.

Best regards
Herb

Quote from: herb on August 10, 2019, 09:06:35 AM
Asking my question I thougth on a small enhancement of (Ppl)Exiftool.exe.

Today it starts Perl.exe that is stored in the subdirectory "exiftool_files" of its installation directory.
I thought about an global environment variable or an *.cfg file (parallel to *.exe) or ... that points to Perl.exe (to be used) - with an absolute path.

My launcher doesn't call Perl.exe but perl5xx.dll

The larger part of ExifTool are not the dll files but the ExifTool and Perl components being executed.

There are well established mechanisms to run an "installed" version of a program from "anywhere", for example the "Path" environment variable.

If a software author embedding ExifTool decides not to use "Path", it's not exactly my job to undermine his decision.

I could imagine ways to achieve what you want, for example:

• Check whether we were called via a symlink
• Use a stub

But is it worth the effort?

Maybe you can symlink the whole exiftool_files folder and duplicate only the small exiftool.exe. That's not beautiful but might do the job.

Quote from: herb on August 10, 2019, 09:47:12 AM
I have an additional question to default config file .exiftool_config.
Why must this file be stored inside subdirectory exiftool_files using your (Ppl)Exiftool.exe?

It is not used if it is stored inside the current directory parallel to (Ppl)Exiftool.exe.

Thanks for pointing this out.

Checking the directory of the Perl launcher exe requires a patch in ExifTool.pm

ExifTool.pm parses $0 pointing to the "script name" in a standard Perl environment. The par packer fills $0 with the exe path.

Since my package is "a standard Perl environment", $^X needs to be used instead of $0.

Phil, do you have any objections to duplicating this line as below (around line 8430)?

    -r $file or $config =~ /^\// or $file = ($0 =~ /(.*[\\\/])/ ? $1 : './') . $config;
    -r $file or $config =~ /^\// or $file = ($^X =~ /(.*[\\\/])/ ? $1 : './') . $config;


Oliver

Hello Oliver,

thanks for your investigations and your detailed explanations.
Some points are now much clearer to me.

Thanks again and
best regards
Herb

Quote from: obetz on August 10, 2019, 10:38:22 AM
Phil, do you have any objections to duplicating this line as below (around line 8430)?

Would it be possible to instead set the EXIFTOOL_HOME environment?

- Phil

Quote from: Phil Harvey on August 11, 2019, 08:27:26 AM

Quote from: obetz on August 10, 2019, 10:38:22 AM
Phil, do you have any objections to duplicating this line as below (around line 8430)?

Would it be possible to instead set the EXIFTOOL_HOME environment?

it wouldn't make "my" ExifTool package compatible with the pp (PAR Packer) packed one.

The ExifTool documentation describes that the "exiftool application directory" is searched for ".ExifTool_config".

In case of "plain Perl" ExifTool, the "exiftool application" is the exiftool Perl script, it's path is in $0. No ambiguity here.

But in case of a "native" Windows ExifTool, users likely expect exiftool.exe being the "exiftool application". Therefore such a Windows variant of ExifTool should look in the directory containing exiftool.exe for ".ExifTool_config".

Perl distinguishes $0 (the script path) and $^X (the exe path), therefore a "native Windows" ExifTool should check $^X instead of $0.

The pp packed ExifTool.exe doesn't notice this difference, because pp sets $0 to the exe path and $^X to a non-existing "perl.exe". Kind of a hack, in my opinion.

I could try to duplicate this pp hack in my Perl environment and make $0 = $^X but I don't feel good about fiddling with well-defined Perl variables.

What do you think?

Oliver

If I add $^X to the mix, then the directory containing Perl will be checked for the ExifTool config file on Mac/Linux/Cygwin systems.  This is a behaviour change that would at least have to be documented, but it probably wouldn't cause any problems.

- Phil

Quote from: Phil Harvey on August 12, 2019, 07:28:15 AM
If I add $^X to the mix, then the directory containing Perl will be checked for the ExifTool config file on Mac/Linux/Cygwin systems.

That's true and would happen not only for Cygwin but all "installed Perl on Windows" users.

After spending a night on it, I don't like the idea of always evaluating both variables.

Maybe it's cleaner to let ExifTool check the environment and then use only one variable.

By the way, there is another "$0 vs. $^X" place: The code extracting parameters embedded in the ExifTool file name.

So I need a way to determine whether ExifTool is running as the "standalone Windows version".

The script filename $0 itself could hold this information. Maybe "ExifTool_winpack.pl" or an abbreviation?

Speaking of choosing names: Do you like the current directory name "exiftool_files" or do you have a better suggestion?

Oliver

"exiftool_files" is fine by me

- Phil

Are there any plans of releasing the alternate Windows version on the official ExifTool website?

This should happen eventually, but I need to find some time to be able to test this and build this version myself.

- Phil

Thanks,

I am getting the following error at the moment when running ExifTools from another program on a Windows Server 2019 at the moment and I believe that it is related.

IO error: Can't open file C:\Users\john.doe\AppData\Local\Temp\2\par-626f622e6275696c646572\cache-exiftool-11.11\inc\l
ib\unicore\lib\WB\EX.pl for write : Permission denied

/Andreas

Quote from: Eriksson on October 02, 2019, 11:40:11 AM
IO error: Can't open file C:\Users\john.doe\AppData\Local\Temp\2\par-626f622e6275696c646572\cache-exiftool-11.11\inc\l
ib\unicore\lib\WB\EX.pl for write : Permission denied

this seems to be PAR related.

Do you want to try my package?

I had no time to keep my version current due to holidays and workload but can prepare it this weekend.

Oliver

I'm happy to test it, thanks.

/Andreas

I try to update to the current version tomorrow.

Quote from: Eriksson on October 03, 2019, 03:33:44 AM
I'm happy to test it, thanks.

The current version 11.69 is now available at https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows as installer or plain ZIP package.

Andreas, please tell me whether it solved your problem (any other suggestions also welcome).

Oliver

I have added another link to this version from the top of the ExifTool home page.

- Phil

Quote from: Eriksson on October 03, 2019, 03:33:44 AM
I'm happy to test it, thanks.

Any results you might report, Andreas?

I've been able to run all the unit tests that do reading and writing for exiftool-vendored against both your 11.69 build, and, using your unparx.cmd, a new 11.71 build: https://ci.appveyor.com/project/mceachen/exiftool-vendored/builds/28279077 (https://ci.appveyor.com/project/mceachen/exiftool-vendored/builds/28279077).

I also ran an extensive read test (reading some 4.5k example images) to reproduce the Tags typings (see https://exiftool-vendored.js.org/interfaces/tags.html (https://exiftool-vendored.js.org/interfaces/tags.html), but caution, it's a huge webpage!), and the differences were expected (due to timestamp differences and other expected system noise).

It looks stable to me! The added security of not running perl scripts out of a temporary directory is also quite appealing.

I had some issues with trying to reproduce your 11.69 zipfile by using your build instructions, and sent you an email directly, but the diff and steps I took are here, if anyone else cares to look: https://github.com/mceachen/exiftool-vendored.exe/commit/5294df9fd23b8149b8dc2d3f96ccb85f09232466 (https://github.com/mceachen/exiftool-vendored.exe/commit/5294df9fd23b8149b8dc2d3f96ccb85f09232466). Given how frequently Phil pushes out versions and bugfixes, it'd be nice if other people could reliably reproduce the installer (or, best-case scenario: have Phil switch to this installer!)

Thanks again for your work here, Oliver!

Quote from: mceachen on October 22, 2019, 11:05:48 PM
I had some issues with trying to reproduce your 11.69 zipfile by using your build instructions, and sent you an email directly, but the diff and steps I took are here, if anyone else cares to look: https://github.com/mceachen/exiftool-vendored.exe/commit/5294df9fd23b8149b8dc2d3f96ccb85f09232466

Either use unparx on Phil's exiftool.exe or use my Strawberry Perl based package, don't mix them.

unparx.cmd from https://oliverbetz.de/pages/Artikel/Portable-Perl-Applications will unpack Phil's exiftool.exe to use it's contents without the pp magic. Phil's exiftool.exe is based on a different Perl distribution than "my" package.

The ExifTool package available from https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows is based on Strawberry Perl.

The "several hundred different files" are simply the different Perl distributions.

It makes no sense to replace all the Perl related files manually just to get a new ExifTool version.

You need to follow the matching "build instructions". If you want to upgrade the Strawberry Perl based package, follow the instructions at the end of my related page: "Sources How to build the installer yourself".

Again: Do not use files unpacked from Phil's exiftool.exe (using unparx) together with my Strawberry Perl based package.

After all: Even though I can't keep up with Phil's pace, the versions I release should be sufficient for the needs of most users.

Oliver

Sorry, I haven't been able to test it yet.

Quote from: obetz on October 22, 2019, 07:44:36 AM

Quote from: Eriksson on October 03, 2019, 03:33:44 AM
I'm happy to test it, thanks.

Any results you might report, Andreas?

Quote from: obetz on October 23, 2019, 07:10:21 AM

Quote from: mceachen on October 22, 2019, 11:05:48 PM
I had some issues with trying to reproduce your 11.69 zipfile by using your build instructions, and sent you an email directly, but the diff and steps I took are here, if anyone else cares to look: https://github.com/mceachen/exiftool-vendored.exe/commit/5294df9fd23b8149b8dc2d3f96ccb85f09232466

Either use unparx on Phil's exiftool.exe or use my Strawberry Perl based package, don't mix them.

Ah! Sorry, I misread your instructions!

I converted your instructions into a short shell script: https://github.com/mceachen/exiftool-vendored.exe/blob/master/update.sh (https://github.com/mceachen/exiftool-vendored.exe/blob/master/update.sh).

The build from that script passes all exiftool-vendored tests: https://ci.appveyor.com/project/mceachen/exiftool-vendored/builds/28336491 (https://ci.appveyor.com/project/mceachen/exiftool-vendored/builds/28336491).

This is great: it's wonderful to have ExifTool on Windows launch in fractions of a second, rather than tens of seconds.

Thanks again, Oliver!

thanks for providing this shell script.

Regarding the speed: Your system must be somewhat different from most other user's setup.

Some time ago I did a simple bechmark test: https://exiftool.org/forum/index.php/topic,10230.msg53763.html#msg53763

On my PC, Phil's exiftool.exe started in less than 250ms. My "par-less" exiftool starts in 150ms. This is faster, but I can't see how the original exiftool.exe might take "tens of seconds".

Also on https://exiftool-vendored.js.org/ you write "On Windows, for every invocation, exiftool installs a distribution of Perl and extracts the ~1000 files that make up ExifTool, and then runs the perl script". This should not be the case, the extracted files stay in the temp directory and are re-used at the next time.

Even a fresh install should not take "tens of seconds". Which virus scanner do you use?

Quote from: obetz on October 24, 2019, 01:52:32 AM

Even a fresh install should not take "tens of seconds". Which virus scanner do you use?

I'm running tests on the latest Windows 10 (version 1903) on an older XPS 15 (i7 6700) on an SSD using Windows Defender.

I've just "refreshed" windows a month or two ago, but it does have a full development environment on it, and something there may be mucking with I/O performance, but I have other windows boxes that are just vanilla, nothing-else-installed test machines, and the prior version of exiftool would sometimes take 2 to sometimes >10 seconds (!!) to return the result of `exiftool -ver`.

Windows Defender spikes the CPU while I'm waiting for the version number, fwiw.

In any event, I was able to delete that comment, as I'm pushing out your new approach today: https://github.com/photostructure/exiftool-vendored.js/commit/6a3ade83d9ec3374af09c64d1845ede3d104c52c#commitcomment-35661276 (https://github.com/photostructure/exiftool-vendored.js/commit/6a3ade83d9ec3374af09c64d1845ede3d104c52c#commitcomment-35661276)

Quote from: mceachen on October 25, 2019, 12:57:02 PM
I have other windows boxes that are just vanilla, nothing-else-installed test machines, and the prior version of exiftool would sometimes take 2 to sometimes >10 seconds (!!) to return the result of `exiftool -ver`.

that's strange, I never observed such a long delay. And I'm using ExifTool for many years now.

Due to other reasons, I just had to do a "Reset this PC" on this machine, so it's Windows 10 with nothing other than what comes with a "clean" install, plus git for windows and vs2015:

Current official release, first run: `time exiftool -j image.jpg`

real    0m11.715s
user    0m0.000s
sys     0m0.015s

Immediate subsequent runs are much faster, though:

real    0m0.435s
user    0m0.015s
sys     0m0.031s


However your release, first time and every time: `time exiftool -j image.jpg`

real    0m0.249s
user    0m0.000s
sys     0m0.015s

Oliver:
I just today found and tried out your portable ExifTool. Super great idea, and it works flawlessly! I never did care for the self extracting (to the %temp% directory) workings, but understood that Phil knew this worked fine for most all users. As a software application developer I prefer your concept, as it installs cleanly in the program directory, and there is no clean up in the temp folder when an application is uninstalled.
Thank you for your hard work.
Michael Lee
https://gpstamper.com

thanks for your feedback.

I suggest to keep the original file times from my archive instead of the time when you extracted it.

Maybe the "zone identifier" issue? https://devblogs.microsoft.com/oldnewthing/20110504-00/?p=10743

Oliver

Hi Oli!

Ich wollte einfach mal Danke sagen, dass du dir Mühe machst, den Installer auf dem neusten Stand zu halten.  🙏
Ohne deinen Installer müsste ich ein Loch in die Sicherheit meiner Systeme bohren.

Malus

Hallo Malus,

danke für das Lob und die Erinnerung an 12.29 ;-)

Es freut mich, einen Beitrag leisten zu können. Das Erstellen ist halbwegs automatisiert. Ein paar ausführlichere Selbsttests (zusätzlich zu denen von Phil) für die Windows-Spezialitäten mache ich noch "zu Fuß", aber nicht bei jeder Version.

Thanks for the kudos,

I'm glad to be able to contribute. The build is rather automated. The more detailed self-tests (in addition to Phil's) for the Windows specialties I still do manually, but not for each version.

Oliver

This version of ExifTool reduced memory usage from >12GB to about 3.5GB for one user who was having out-of-memory problems.  See the github issue here. (https://github.com/exiftool/exiftool/issues/113)

- Phil

Quote from: Phil Harvey on December 30, 2021, 07:55:28 PM
This version of ExifTool reduced memory usage from >12GB to about 3.5GB for one user who was having out-of-memory problems

Interesting, I have no explanation... Maybe Strawberry Perl or the Perl version is the reason (we don't even know whether the 32 or 64 bit version was used).

Oliver