Proposal for a robust and simple Windows version

[obetz]

Hi Phil,

after spending a lot of time with the pp PAR Packer sources (not for the faint-hearted!) and asking for advice in the par@perl.org list (in vain), I decided not to investigate further how to make pp better but to provide an alternative.

IMO, the main problem with pp is that it unpacks executables and scripts to the Windows temp directory which causes several problems:

• Unwanted change of files by maintenance tools (e.g. cleaning the temp tree) or even malware.
• Running executables in temp is considered suspicious by AV products and might be even blocked by SRPs
• Cluttering temp with old installations

I found other programs being affected, and there are patches tinkering with the symptoms, e.g. "biber" files vanished from the cache directory, the workaround was to put a back-dated "canary file" in the cache directory: https://groups.google.com/forum/#!topic/perl.par/_obYvAgHeEc (hmm: "the code base is very brittle and prone to breaking stuff at a distance").

To get the "right" solutions, we need to define requirements. Hard to estimate the average Windows user, but I have an idea what the application shall do:

• The executable files shall be in a location being controlled by the user. Rationale: The user can decide whether to write-protect them or not, simple maintenance, no obfuscation an hidden leftovers.
• exiftool shall be optionally reachable via the search path
• Calling exiftool shall be consistent with the existing version, IOW no "perl.exe exiftool parameters"
• Several versions of exiftool must be able to co-exist. Rationale: Tools like IMatch want a specific version.

I don't think that it is required to hide the quite normal fact that a program consists of a lot of files, IOW we don't need to make the user believe to run just one monolithic .exe. Better think about a simple "installer".

So the programming question is: How can I run exiftool in a non-intrusive manner?

As far as I see, it doesn't require more than the files already present (e.g. in the packed exiftool.exe) plus a small exe calling the Perl interpreter in the perl5xx.dll.

See the small C program attached. It takes it's execution path and mangles it to point to a script name in the same directory. Then it invokes a Perl interpreter the same way as perl.exe does, passing the script name and all user provided parameters.

How to use?

• Choose a naming convention and change pathreplace[] if desired
• Compile it with libperl5xx.a and the header files from your Perl distribution
• Put the resulting exe and perl5xx.dll in the same directory as your Perl script
• Put the required Perl libs in the "lib" subdirectory (automatically added to @INC by Perl)
• Run the exe.

To use it with exiftool, I used the files unpacked from the exiftool.exe PAR archive. This seems to work but I think there is some redundancy in it (duplicate files), so there is room for improvement. But at the moment, it's good as a proof of concept.

The automated tests don't work with an exe version, therefore I did no thorough test.

What do you think?

Oliver

[obetz]

Small improvements added to the C sources: By renaming the exe to "perl.exe", it behaves like the standard perl.exe (e.g. to run the automated tests).

With a little helper script, it runs also the test suite.

I added a simple installer using NSIS. This way one can easily add the directory to the path. The forum doesn't accept the compiled installer, though.

If a user dislikes the installer, he can unpack the exe with 7-Zip.

BTW: By using a self-compiled exe, it's very easy to add version info and an icon. to exiftool.exe

[obetz]

the latest version can automatically find a perl5*.dll and load it, IOW use "explicit linking" instead of "implicit linking".

This way, we don't need the include files and libperl5xx4.a from the Perl distribution, and the resulting .exe can be used universally.

PAR/pp generated Windows exectuables can easily be converted to straightforward packages.

Future version will be published here: https://oliverbetz.de/pages/Artikel/Portable-Perl-Applications

Oliver

[obetz]

added a batch file to automatically convert a pp packed "exiftool.exe" to a "unpar-ed" installation.

Extract the ppl.zip archive contents to a new directory and drag/drop exiftool.exe onto unpar.cmd

This will run exiftool once, you need to press Q to stop the help output.

As soon as exiftool is finished, unpar.cmd builds the new package and deletes the temp files.

[obetz]

the last version was designed to hold perl5xx.dll and the Perl script in the same directory as the exe, which would cause collisions if more than one application packed this way is in the same directory. I think that that was not a good design decision.

The new version can deal with replacement strings containing a directory name to get a structure like this:

exiftool.exe
exiftool_files/exiftool.pl
exiftool_files/perl524.dll
exiftool_files/lib

The "installation directory" now contains only exiftool.exe and the directory "exiftool_files".

I deliberately didn't implement a dynamically constructed directory name for this configuration, so you explicitly have to set the directory names in the c sources.

As a result, the name of the exe has no influence and can be chosen / changed freely.

Use unparx.cmd to convert exiftool.exe to such a package.

[Phil Harvey]

Things have finally settled down a bit for me and I've got a chance to read through all this.  Thank you very much for thinking about this problem and coming up with a solution.  This is something that I may think about implementing in a future version of ExifTool.  Honestly though, my real focus is metadata, which is why I chose Perl for ExifTool in the first place (ie. almost no compiler/platform issues -- compare this to other metadata projects like Exiv2 which spend most of their time dealing with compilation issues).  In this respect, Windows has been a thorn in my side, but it seems like a majority of ExifTool users run this platform so I must deal with it, and I appreciate the work you have done on this because it isn't likely something I would ever get around to doing otherwise.

- Phil

[obetz]

Honestly though, my real focus is metadata, which is why I chose Perl for ExifTool in the first place (ie. almost no compiler/platform issues -- compare this to other metadata projects like Exiv2 which spend most of their time dealing with compilation issues).

Don't know much about Exiv2, but Perl itself has issues with Windows and pp adds more. My motivation are the resulting Windows issues of ExifTool. It can't be that users try to run ExifTool with elevated privileges or disable the virus scanner (especially because ExifTool resides in world writable %temp%).

In this respect, Windows has been a thorn in my side, but it seems like a majority of ExifTool users run this platform so I must deal with it

That's it. And ExifTool is integrated in many great applications running (also) under Windows like IMatch, Geosetter, Affinity, FastPhotoTagger. BTW: Especially for this "integrated" use of ExifTool, the "pp-less" operation would be much cleaner!

My current status: The "unpacked" exiftool.exe currently works pretty well for me, and I'm also experimenting with Strawberry Perl.

Can you estimate the coverage of the automated tests? Or tell me what should be tested in addition to them?

From the past, I know there can be subtle differences between Perl distributions, therefore it would be good to know what the automated tests do not cover.

For example, I generally avoid "unusual" path/file names with spaces and non-ASCII characters so I did no thorough tests regarding this "minefield". But I know that other people don't care about this.

The parameter mangling in the pp bootstrapping (boot.c -> shell_quote) might influence quoted parameters, so this should be also tested.

There are known problems with daylight saving time zones, and Cygwin handles them differently. Likely ExifTool doesn't rely on the built-in file time functions, correct?

I'm not sure whether Test::Harness::runtests() has all necessary capabilities, maybe Windows specific tests are needed (e.g. with a cmd file or even PowerShell.

Besides, runtests() seems to rely on access to "perl.exe", therefore it doesn't handle the specifics of a Windows exe with a different name.

Oliver

[Phil Harvey]

Can you estimate the coverage of the automated tests? Or tell me what should be tested in addition to them?

They test the API fairly well, but don't test the "exiftool" application at all.

For example, I generally avoid "unusual" path/file names with spaces and non-ASCII characters so I did no thorough tests regarding this "minefield". But I know that other people don't care about this.

The tests don't cover this at all, but spaces shouldn't be a problem.  However, special characters in Windows filenames continue to be a problem.

There are known problems with daylight saving time zones, and Cygwin handles them differently. Likely ExifTool doesn't rely on the built-in file time functions, correct?

It does use built-in time functions, but doesn't rely on them, and in general the tests are designed to pass even with time zone differences, etc.

I'm not sure whether Test::Harness::runtests() has all necessary capabilities, maybe Windows specific tests are needed (e.g. with a cmd file or even PowerShell.

Yes.

- Phil

[obetz]

Updated version with a minor cleanup allowing Strawberry Perl (needing several DLLs) to be used in a subdirectory.

[obetz]

For a brief period, I make an alternative ExifTool version available here: https://oliverbetz.de/exiftool-11.52_with_tests.zip (Edit: removed the installer due to virus scanner false positives)

I used the NSIS installer, so it can be extracted with 7-Zip or similar.

It's based on current Strawberry Perl 32 bit because this seems to be better maintained than AS and has unambiguous licensing.

Although it passes the selftests, I would consider it "beta". There is so much "magic" in pp that it's hard to predict whether something behaves different.

I had to make a minor patch to windows_exiftool to enable again to embed parameters in the filename (although I'd prefer links).

Oliver

[mpegleg]

Hi Oliver.

After all the hard work you put into a project like this, in the end you may have a problem distributing an exe file like this, and getting people to actually feel comfortable testing it.

I naturally always run "any" exe file that I download through VirusTotal and if I get more than about 1x possible warning of Malware/viruses, even "if" I think it may just be false positives, standard operating procedures say: "Don't run it!".

So I am too hesitant to run it. Sorry. Currently has 4x warnings (Yes I know, most probably just installer false positives... but...?? )

Even Phil's Windows unzipped exiftool(k).exe (v11.52) has one engine detect "something" (albeit benign): VirusTotal output (currently 1x warning)


EDIT:
I only just read this now: Running exiftool from Windows GUI.

So unfortunately I doubt you'll get many folks willing to test it, until you get it signature verified, which is probably both a hassle and time-consuming.

[obetz]

well, this seems to come from NSIS (Nullsoft Scriptable Install System) https://nsis.sourceforge.io/ I removed the installer.

https://oliverbetz.de/exiftool-11.52_with_tests.zip has the same contents and no virustotal positives (check it by unpacking the exe with an archiver). Unpack it where you like and set the path variable if you want.

The archive contents are just a mix of ExifTool (tarball) and parts of Strawberry Perl plus my tiny exe (source code also posted).

Oliver

[mpegleg]

Ok Oliver. I'll do that.

Yeah please don't be offended, as I really wasn't expecting that your code was doing anything deliberately nasty, but with those VirusTotal warnings, I'd be stupid to totally ignore them of course, and it will concern other people also.

[obetz]

The end of signature-based virus scanners has long been predicted - they simply can't deal well with the current situation. It could also be that NSIS has been used by evil people in the past (just a wild guess).

Bkav and Panda have been reported to throw false positives (depending on the tester). Only Jangmin (??) didn't like the included uninstaller exe. McAfee Gateway Edition complained but not McAfee - don't they use the same signature database?

But what I've learned is that selecting an installer solution has do be done carefully to avoid problems. NSIS was easy to use, maybe I should check Inno Setup.

Oliver

[mpegleg]

Oliver. I'm confused.

Is this what would get installed with Phil's regular exiftool(-k).exe file anyway after it was run the first time, but together with a helpful installer to help the n00bies get the first-time install done correctly?

Or, is this the full ExifTool Perl package, for those users who already have Perl installed on their Windows pc. I don't have Perl, nor can I program it.