ExifTool for Dummies

[Phil Harvey]

The best thing to stop a user from using an application is always telling him "read the fucking manual" only once. You can bet he dumps the thing immediately if he's not desperate.

I think I agree (for the average user).  Hence, ExifTool is software for desperate users.

- I personally always look for examples which come close to my situation and then from the example look backwards to the abstract command help to try to figure out what happens.

I tried approaching it from this angle, but the problem is that reading through examples to list all possible common uses of exiftool would take more time than reading the full documentation.  There are just far too many variations.

- Any commands discussed here which do multiple things in one line I experience as counterproductive as they confuse me too much.

I agree, and this is why most of my examples are simple and do just one thing.  But I have found that MANY people run exiftool multiple times because they do not realize that you can do more than one thing at once.  So in the end, the simple examples may be counterproductive.

- I'd favor lots of generously explained examples which are structured in a helpful way so I am not confused by the mass of examples.

I started working on this, but didn't get too far before I was overcome with the vast number of possibilities.

As a sidenote: Even minor details can be non-intuitive. For example for me as non English native speaker reading "DSTTAG<SRCTAG" at first was anything but intuitive because my eyes didnt really automatically turn the "DSTTAG" into "destination tag".

Good point.  I should clarify this.

- Phil

[Beholder3]

I guess this thing could end up in a whole philosophical discussion on approaches to online education methodology for grown-ups.

Basically I think to get anywhere you fare best if you define the target audience precisely first. In communications there is no "one size fits all" IMHO.

I see three levels of exiftool skills/experience as one segmentation dimension (newbie/used coupleof times/power user) and then the other as technology-prone (e.g. developer like) or enduser type ("I dont need to understand it , I want it to work"; "Press A for typical install"). I assume you can only satisfy 1 or 2 of the resulting six matrix quadrants with whatever approach. The people in the rest quadrants will likely not be happy.

Not only the tool itself is more than good in my opinion, the only documentation also is way beyond expectation complete and good. Few tools have this. I think it's also part of the incredible posibilities of that piece of software that make it difficult to explain. It easy to document a wooden wheel, it's more complex to document the exact functioning of all clockwork in a swissmechanical watch.

[dhackney]

Phil,
I've written documentation and supported products I've developed in the past, so I feel your pain on this.

IMO, you are facing one of life's most daunting challenges: describing the fishbowl while residing inside it.

While in the fishbowl, and engineering and building the fishbowl, you are intimately familiar with what it is and how it functions.

However, while in the fishbowl, it is extremely challenging, if not impossible, to describe what the fishbowl looks like, how it fits into the context of other things around it, etc.

I don't think you can change these fundamental challenges. In my experience, it's better to leverage the perspective of those on the outside of the fishbowl than to tilt at the windmills of trying to alter the basic challenge of describing it while being inside it.

In my view, having also been involved with allocating and managing support resources, it is best to apply your time and energy related to community support where everyone can get the most bang for your time and energy buck.

In reading through the documentation and lots and lots and lots of forum posts searching for answers, I see a recurring theme that goes something like this:

Phil: "I've written this a thousand times..."

This is another fundamental challenge that no one, to date, has solved: Forums suck as a resource for knowledge retrieval. They are tremendous at providing a medium of and place for exchange around a topic of interest. They are fantastic as a black-hole sucking domain-specific knowledge in. In short, they are great as a place to put knowledge into, but they are severely challenged when it comes to getting the knowledge back out, especially about a nuanced technical topic.

In that regard, for those on the outside of the fishbowl, it doesn't get much more nuanced than the command line of ExifTool.

But, all is not lost.

I think there are a few strategic and tactical things that you can do to make the energy and time you put into supporting ExifTool more efficient for all involved.

Segment and target.
As Beholder3 noted, it is important to understand who you are trying to communicate to and then form the communication appropriately to that audience. I agree that there are a variety of types of people who seek to utilize the tool and consequently often need help and support in using ExifTool.

ToolRefugees
People who can be served by front-end tools such as ExifToolGUI are the majority of the ExifTool user population. The vast majority of these people never know that ExifTool is the engine under the hood that enables all the metadata goodness in tools such as GeoSetter. As these tools continue to evolve and grow in capability, they can meet more and more of the needs of the general photo / document / etc. metadata community. For support, those users will look first to the tool provider for help and support.

At some point, some of those tool users hit a wall of capability with those tools and need more than the tool can provide. They then do a search on "metadata" and wash up on the shores of ExifTool Island, seeking metadata salvation. We'll call them ToolRefugees.

ToolRefugees' characteristics include:
•   Timeline: immediate
•   Urgency: medium to high
•   Scope: specific to a particular task/tool/workflow
•   Technical ability: low
•   Command line familiarity/comfort: non-existent to very low

Most ToolRefugees who wash up on ExifTool Island are scared away by the forbidding walls of the Command Line, the width and depth of the Moat of Documentation and the arcane language of the local tribe. Consequently, there are almost certainly tens of thousands of people out there who need what ExifTool can give, but are too intimidated by ExifTool Island to even ask for it. Most of them swim back out to sea and are eaten by sharks. Those who don't, die of exhaustion from fruitlessly searching the island for a simple, accomplishable, understandable way to solve their problem.

Journeymen
A second group of ExifTool users are people who use the tool occasionally.  They live the majority of their lives in other lands, but venture through ExifTooland while accomplishing specific tasks or working on specific projects. The skills of the Journeymen are many and varied, but none are experts on ExifTool. When they need a metadata problem solved, they stop by, dive into the Moat of Documentation and wander through the shadows amongst the posts in the Forum Forest, seeking the ray of sunlight that shines upon the Holy Grail: the specific answer to their specific challenge. Most Journeymen will eventually find the answer they seek and then go on their way to other lands. A few months, years or professional eras later, they may return, the next time they have a metadata challenge.

Journeymen characteristics include:
•   Timeline: near to mid term
•   Urgency: medium to high
•   Scope: ranges from specific to system/process
•   Technical ability: medium to very high
•   Command line familiarity/comfort: very high to somewhat, mostly dependent on age and areas of experience

Ninjas
A third group of ExifTool users consists of people who live, breathe and sleep metadata. They are metadata professionals who rely on ExifTool to do their day-to-day work. From the systems perspective, ExifTool is an essential component of their mission critical systems. These people are ExifTool Ninjas. When they interact with the ExifToolCore, it is to confirm, discuss or diagnose arcane aspects of the tool, deep secrets of metadata known only to the High Masters of Metadata or complex implementations of metadata processes.

Ninja characteristics include:
•   Timeline: mid to long term, except in emergencies, bugs and lightning strikes
•   Urgency: medium to high
•   Scope: ranges from deeply arcane specific to enterprise system / business process
•   Technical ability: very high to not-of-this-world savant
•   Command line familiarity/comfort: they order coffee at Starbucks via the command line and know all the switches and flags required to create all 87,000 possible drinks

There are, undoubtedly, scores of other personas among the ExifTool user community, but I think these three capture the majority of the bell curve.

What is the most efficient way to support these three personas?

ToolRefugees need tutorials, how-to guides, 10 easy steps to "hello world" and lots and lots of handholding.

Journeymen need some good documentation with an excellent index, and, most critically, fully verbose examples. Well written and indexed documentation and a set of 10 to 20 examples of solutions, fully documented, with explanations and examples of the primary OS variations, would be 90% to 95% of what the Journeymen would need. Remember, by their very nature, Journeymen generally prefer to be self-sufficient.

Ninjas don't need no stinkin' badges. They don't need no stinkin' much of anything, except, most importantly, access to The One, The Oracle, The Knower Of All Things ExifTool. Ninjas understand ExifTool shorthand. They understand it all. They just need a little insight once in a while.

Think about how the time and energy to support the community is currently being invested and in what form. IMO there is a mismatch between what is most efficient and effective for the personas and what is currently provided and how it is provided.

The core question is, "How can you more efficiently serve the most people the fastest?"

Take the easiest ones first: the Ninjas. The Ninjas are best and most efficiently served by the methods you currently employ:  high frequency, off-the-charts deep interaction via the forum.

The toughest case is the ToolRefugees. They require the most content development and will still require the most handholding because they will never, and I mean never—ever--RTFM.

That leaves the Journeymen. I think they can be very efficiently served with just a little bit of additional content and some additional channels that they themselves will help build.

continued...

[edit: wordsmithing]

[dhackney]

continued...

Solutions
For instance, create a new, top-level, category on the forum titled: Solutions. Put a sticky post on it that details the format and structure required for the OP for each topic in that category.
•   Solution name:
•   Metadata type/form, e.g. photo, video, PDF, etc.:
•   Solution type, e.g. date/time, filename, etc.:
•   Problem solved:
•   Brief description:
•   Full description:
•   Code / command line:
•   Caveats/limitations:
•   NewbieNotes (specific notes for newbies explaining basic concepts, caveats, etc.):
•   Tags (in the blog post sense, e.g. date/time shift, Nikon, thumbnail rebuild, scans, etc.):

What the Solutions forum category enables is the creation of documented solutions. The solutions are important in and of themselves, but what is most important is that they contain actual, verbose examples of exactly what was created, including full directory paths, actual file names, etc.

It is impossible to express how valuable actual code examples are to the Journeymen persona.

I believe that in short order, the ExifTool community would populate the solutions forum category with documented examples that, along with the documentation, would serve most of the needs of the Journeymen persona. I'm confident a few Ninjas would also throw some solutions in there, if for no other reason than to give the unenlightened something to scratch their heads over. 

Verbose Documentation
The next step is to start building a verbose version of the documentation. Before you go into cardiac arrest, this does not need to be complete. What it does need to be is written in a way that pre-supposes nothing in the way of existing knowledge.

There would incalculable value in something as simple as a documented copy operation command line that explains what each element of the command line does, how it does it, in what order it does it and what dependencies and groupings exist. It would be extraordinarily valuable if you then took each element of that sample command line and broke out its primary OS variations.
•   Command line Unix/Mac: '-operator option'
•   Command line Windows: "-operator option"
•   Command line Unix/Mac/Windows: %X
•   Batch file Windows: %%X

How many times have you written those things in forum post responses?

The things you keep repeating are the things that need to be featured at the major topic level. That means a "Unix vs. Mac vs. Windows" major topic level in the FAQ, the documentation, etc.

Currently, the " versus ' issue is mentioned in the FAQ only within (parenthesis). Is it documented? Technically, yes. Is it out there where anybody who doesn't read every. single. word. of the documentation, FAQ, etc. is going to see it? No.

And, yes, I know there are 8 or 9 different shells, and you've probably got somebody running ExifTool on an HP calculator, but concentrate on the 80/20 of your users. The 80 of the 80/20 is what you want to address, first and foremost.

Remember: You can expend every single bit of your energy and time trying to support every single possible exception and you'll end up supporting everyone poorly. Invest your time and energy where it will yield the greatest return for the majority involved.

Something as basic as this would be very helpful:
•   (everything on an ExifTool command line is processed together (except for this) (and that))

(Humans are hard-wired to organize and categorize things they see, hear and experience. The open, free form ExifTool command line is tough for basic human comprehension. Your mind probably subconsciously groups all of those operators and options and parameters. The non-Ninja mind does not do that. Please help the non-Ninjas.)

•   Anything on the command line can be in any order except A must proceed B except when it is paired with D, as in DA

•   There are X basic types of command lines. They take the form of:

• (source operator)
• (source operator target)
• (Source (operator (optionA paramter1) (optionB parameter2)) ((format) target))

In general, from the outside looking in, one thing you struggle with is being overly concerned with communicating all possible uses/exceptions/possibilities of every single command/option/capability of the tool. IMO, that's why you abandoned the ExifTool Command-Line Examples page and why the ExifTools for Dummies project looks so insurmountable.

Try picking up an Excel for Dummies book and leafing through it. If you really wanted to, you could probably launch the space shuttle with Excel and land it with ExifTool. However, one thing you'll notice in the Excel for Dummies book is that there is no discussion of orbital mechanics or how to manage memory allocation for dynamically constructed DLLs written in Sanskrit.

The book will talk a lot about how to add 2 plus 2 to get 4. That's all the information a heck of a lot of people need to know about ExifTool as well. They just need to do the 2+2=4 stuff.

Meanwhile, the ExifTool documentation and a lot of the forum responses include tremendous amounts of orbital mechanics. Are you indeed documenting the ability of ExifTool to land the space shuttle? Technically, yes. Is anybody really able to make use of the bits and pieces of the orbital mechanics story that is spread out over dozens of pages and posts? Not so much.

The thing to keep in mind is that the majority of support needs come from people who are struggling to *initially* understand and implement the product, usually for a limited scope, specific problem. They will solve that problem and then disappear until the next time they have a problem to solve.

Most of those people currently don't understand the overall concepts and context of the tool and the command line syntax. Most of those people copy a command line out of a post and hack around on it in between spending *lots* of time searching for answers until they get it to work or give up.

Give them some conceptual understanding and context first. It will help them swim the Moat of Documentation.

Give them some fully documented, extremely verbosely explained sample code. That will help them translate the local tribe's arcane language.

In addition, many people are visual learners, meaning that text is a very tough way for them to learn. A visual image such as the following can be very helpful in communicating overall concepts and context.





A second major struggle that stands out from the outside looking in perspective is elegance.

Earlier in this thread you stated, "But I have found that MANY people run ExifTool multiple times because they do not realize that you can do more than one thing at once.  So in the end, the simple examples may be counterproductive."

Think about that statement for a few moments. Compare it to the example of the Excel spreadsheet that uses a half-dozen or a dozen cells to build toward an answer instead of one, long, deeply-nested formula statement in a single cell. Both accomplish the same goal. One, the single-cell formula, is very elegant and makes efficient use of system resources. The other, the multiple cell solution, was quick and easy to build, easy to de-bug since every step is visible, and, while not system-resource efficient, provided the answer the user needed in exchange for a reasonable amount of user time and energy investment.

You may not mean it or think about it this way, but your distress regarding people running ExifTool many times - versus the elegance and system-resource efficiency of a single ExifTool command or code line that would accomplish the same goal - values system time and energy more than human time and energy. Again, you may not mean it that way or think about it that way, but try viewing the documentation, etc. through that lens for a moment. There's a whole lot of elegance going on there.

You may not mean it or think about it this way, but your statement places priority on processor cycles versus human cycles. I contend that for your users, there is no scarcity of processor cycles versus human time and energy cycles. Especially time.

The farther across the spectrum of users you are from the Ninjas, the more those users just want to Get Stuff Done. If running ExifTool 18 times in a batch file, step-by-easily understood-step, accomplishes the Get Stuff Done goal, then that is better use of a user's time and energy that spending hours to days trying to find the elegant, single-command-line answer in the existing support resources.

Try going anti-elegance on a few of your hair-ball command line examples and break them out into the maximum possible number of batch command lines required to accomplish the same goal. Take a look at the result, especially with a rem comment line for each batch command line, and try to tell me that the multi-line approach would not be more comprehensible, understandable and achievable for a non-Ninja user who just wants to Get Stuff Done.

Ironically, and counter-intuitively, elegance is often the enemy of Get Stuff Done. I guarantee you that the majority of people who are using your wonderful tool are just desperately attempting to Get Stuff Done.

Consequently, I think you'll find that the more you explain things like an Excel sheet that does each step of the solution in discrete cells, the more people will understand, the less they will need further interaction and the more people who can be supported and make use of the tool with your available resources.

And that last bit, the number of people who can make use of the tool with your available resources, is the core of the challenge.

There's no doubt you can individually support a solid core of Ninjas.

A goodly number of Journeymen will put in the effort required to make use of the tool, if for no other reason than professional pride. There is probably a 60-70% yield of all Journeymen who enter ExifTooland who leave with something to show for it. The rest of them are lost, their problems unsolved, solutions un-provided.

As things currently stand, the ToolRefugees have very little hope of leaving ExifTool Island alive. Things there are just too arcane, the challenges too daunting and the promise of a reward too distant to justify the learning curve.

I believe you can greatly aid the Journeymen by helping them to help themselves by building a Solutions category on the forum. In addition, some conceptual, contextual content would be of tremendous  help to Journeymen who are freshly arriving or returning after a long absence. A limited set, perhaps no more than 5 to 10 examples, of common command lines with verbose documentation with fully populated examples, including specific variations for the major OS platforms, would answer the vast majority of questions for new arrivals.

And, I believe the ToolRefugees are not without hope. However, instead of trying to build the entire range of content they need, I propose you leverage the tools you and the community both use now: the forum. I suggest that you encourage experienced users to add NewbieNotes to posts whenever possible.

If regular posters got into the habit of including a sentence or two that explained the basics and some of the nuances of the tool in the everyday flow of questions, posts and responses, and included a NewbieNote: tag in each one, you could dramatically increase the number of people who were successfully using the tool.
For example:
NewbieNote: When you see the word DIR in the samples in the documentation and on the forum, you need to replace DIR with your absolute, specific directory path, such as c:\photos\work or a relative directory path such as ..\ or . as used in the command line in this post.

If you use a consistent tag, such as NewbieNote, a simple forum search for that tag will return an entry-level tutorial to ExifTool.

In this way, the ToolRefugees solution builds itself. It also provides a fantastic resource for new or returning Journeymen.

The reward for the community is that you have more time to concentrate on providing the one-on-one support for people who are past the initial learning curve who can make the most use of it. Everyone gets more bang for your time and energy buck

Regarding the OP challenge, ExifTool for Dummies, I think a few months of normal use and traffic in the forum around a Solutions category and a NewbieNote tag, in addition to a little bit of conceptual documentation and a few very verbosely annotated code examples would yield you something you could easily combine into a viable "ExifTool for Dummies." I don't think it would be a big jump from there to a real ebook or self-published or commercial print title, if you chose to go that route.

And, just think, a commercial blockbuster "ExifTool for Dummies" book could fund the purchase and operation of the ExifTool Island Ninja Resort.   

Doug

[edit: wordsmithing]

[Phil Harvey]

Hi Doug,

Thank you very much for this analysis and your suggestions.  Your comments are very insightful, and your classification of ExifTool users is pretty-well right on the money.  The distribution of these users is pyramidal, with maybe a 100:10:1 distribution for ToolRefugees:Journeymen:Ninjas.  As you also point out, most (90%?) of what the ToolRefugees need is help with all things non-exiftool (command-line syntax, metadata concepts, etc).

Invest your time and energy where it will yield the greatest return for the majority involved.

If I spend time/effort in proportion to what is needed, simple math shows that I should spend no more than 19% of my time documenting ExifTool.  The bulk of my effort should be spent documenting the other concepts necessary to make use of ExifTool.  Fair enough.  I had already come to this (rather depressing) realization.

Your suggestion of enhancing the forum to create a channel for ExifTool users to generate the documentation themselves is a wonderful concept.  While I'm sure it would provide a better solution than currently exists, it still wouldn't satisfy the majority of ExifTool wannabe's.  But you also point out that perfection in documentation is not achievable, and since something is better than nothing I will seriously consider your suggestions about the forum changes.  However, people are individuals, and my feeling is that even with a template this idea will fail without serious comprehensive editing/indexing/additions by an all-seeing overlord (yay, more work for me!).

Re your comments about "I have found that MANY people run ExifTool multiple times":  When I said this I had in mind people who create batch files and/or scripts to perform certain tasks.  We are already talking a class above ToolRefugees here (but not necessarily to the point of Journeymen).  From this class of users, I often get the comment that "ExifTool is too slow, how do I speed it up?".  Hence my dilemma.  But I don't think this is a serious problem in the grand scheme of things.

Well those are my initial impressions.  I'll digest this for a while then read it again and think more about it later.  You make a number of good points and I appreciate the time and effort you put into thinking about this.

Thanks!!

- Phil

[dhackney]

Phil,

You are so very welcome. It was the absolute least I could do to thank you for all that you have done and continue to do.

RE: 100:10:1 and your investment of time
I contend that your time and energies are best invested in:

For the ToolRefugees:

• Create a list of recommended reading, search terms, links and introductory level books. When a ToolRefugee washes up on shore, that list is their Welcome Wagon basket. "Welcome to ExifTool Island. To Get Stuff Done here, use these resources to get up to speed quickly and become productive."
• Create a specific forum category for the ToolRefugees and ExifTool wannabes. Have lots of sticky posts at the top that are very, very simple and strictly single subject covering introductory topics.
• Enable early success. Give them some incredibly verbosely documented examples of simple things they can do that will yield success. Something that is the ExifTool equivelent of "Hello World."
• Assume *nothing* in the way of prior knowledge as these days, most of these people have *NEVER* done anything except point and click.
• Get out of the business of handholding this group of people so that your time and energy can yield more greater good for the majority of people who can leverage it. I think that line, of yielding the most output from your time and energy input, begins with the ToolRefugees who read through the tutorials in step one but really takes hold with self-sufficient Journeymen.

For the Journeymen:
If you want to remove 90% of your support workload, go heads-down for a while and build a debug/trace mode into the command line product. Based on the forum, most of the support interaction you do is command line related. As you've alluded to in a few forum replies, the command line is getting to the point of incomprehensibility, even for you and the Ninjas.

I believe that, based on their self-sufficient nature, the journeymen would much prefer to debug/trace things on their own to figure out what the heck is going on. The only thing anybody can do right now is hack around until it works or they give up or they tap into your time and energy.

I suggest a trace/debug mode that:
First, lists how the tool is interpreting the command line structure, e.g. "source = X, target = Y, valid tags = a, b, c, d, valid options = 1, 2, 3, option and parameter pairs = AB, CD, EF, execution will run against these files/file types: abcef.123, etc.; "Press any key to continue"

Second, shows how each command, option and parameter is being applied against the objects involved, e.g. source, tags, target, etc.

I strongly believe that 5 to 10 fully verbosely documented examples of fully complete code samples including verbose directory paths, file names, and OS variations, plus this trace/debug mode would make the Journeymen persona cohort 99% self-supporting.

The goal is to code yourself out of a command line tech support job. 

Wouldn't life be more fun if you could spend more time with the Ninjas and less time repeating, for kazillionth time, that you need " in Windows and ' in Unix/Mac?

Along those lines, think about leveraging available resources to bring some of these needed deliverables to the table.

You need fully indexed documentation. Why not have a summer intern or a grad student do that?

From the career management perspective, any project associated with ExifTool would look great on a CV.

There is lots of time and energy available among undergrads for the indexing level work and excellent capability available at the grad student level for putting together Welcome Wagon basket kits on "10 Easy Steps to ExifTool Hello World."

If your institution's policies prevent you from utilizing enrolled resources, there is no shortage of undergrads and grad students out here who would love to have ExifTool on their CVs.

I think the user community would be willing to take some of this on as well.

What you have built has saved many, many people's bacon. A good portion of those people would love a way to express their gratitude. Writing a forum post that explains the basics of a solution they provided or explains how to create a simple batch file, etc., is a great way to give back.

Overall, I think it is critical that you find some way to build yourself out of the support role that you are currently in.

Over time, you've built an incredible resource that is solving the problems of, literally, millions of users.

The "rate limiting factor," as they say in biology, of your ecosystem is currently: you.

You've also built a system that puts you on the critical path to success for the majority of people who attempt to *initially* use the tool.

The good news of the current model is that you'll always have something to do every morning when you wake up: support the newbies and the returning Journeymen.

The bad news of the current model is that there are 6.3 billion people on the planet. The majority of those people have phones that include cameras. Soon, the majority of those people will have smartphones.

Immutable law of the universe: People plus digital photos = metadata problems.

Soon there will be, literally, billions of humans with metadata problems.

Currently, there is exactly one (1) solution on the planet for anything but the very simple first layer of those problems: ExifTool.

Your challenge: you can't clone yourself.

Soon, very soon, millions of people are going to be howling for a solution to "my iPhone ate my photos" or "my Android phone lost all of my people tags" or some variation thereof.

A lot of them will end up banging on your door. And they won't--ever--RTFM.

You can't hand-hold that many people.

So, I advocate that you invest the majority of your time now into building yourself out of the ToolRefugee, ExifTool Wannabe and new or returning Journeymen direct interaction support business.

I think you stand a chance of surviving the approaching Tsunami at the Ninja interaction level, but I cannot envision any possible way that you can scale the support model you currently have in place for the command line tool.

And, full disclosure (should have included this in my first post): Queen's was a client of mine and I have friends whose children currently attend. As such, I have a vested interest in seeing ExifTool continue as a world-class example of what Queen's University's people and programs can produce. However, I do not know Phil or anyone else who is directly involved with the ExifTool project personally, or have any professional interest or involvement in the project. I'm just another person that ExifTool has saved from MetaData Hades.

Doug


[edit: wordsmithing]

[Phil Harvey]

Hi Doug,

Your idea of a debug/trace mode already exists at various levels, but probably not in the form you wanted.  The -v options provide additional details about metadata when reading, and additional details about what ExifTool is thinking when writing, and the -htmlDump option is an invaluable tool for debugging EXIF structure.  But exiftool is totally missing is a command-line debug mode as you suggested.  Simply spitting back the options as parsed by exiftool sounds interesting, but I'm not sure it would be as useful as you imagine.

I certainly agree that fully-indexed documentation would be wonderful, and is something I plan to work towards.  You also have other very useful suggestions about documentation that I will be acting on.

In the interests of full-disclosure (a trend you have so kindly initiated), I think I should say a few words myself:

ExifTool is only very loosely related to my work here at Queen's, and as such I can't expect help from any of the students here.

In reality, I think that ExifTool is and always will be a niche tool, used by relatively few people.  In fact, I have positioned it this way purposely by sticking to a command-line interface and using a relatively obscure language for the code.  I did this specifically so I wouldn't get overwhelmed with demands on my time.  If I were looking to make a profit, things would be different.  I enjoy helping users on a personal level and helping a few people per day is well within my capabilities, but I don't want much more than that.

In my view, a major benefit of my work with ExifTool is as a developer's resource.  I have gone out of my way to provide other developers with the information necessary to create their own tools.  Although this effort is largely unrewarded, I feel it is necessary that someone assumes this role (as a central repository of knowledge about metadata structures, especially when it comes to proprietary maker note information).  Due to this, if metadata ever becomes a mainstream necessity as you suggest, other tools will appear to fill the void and ExifTool will continue to cater to its niche cliental.

- Phil

[dhackney]

Phil,

RE: debug mode
Understood. Agreed on the value of some type of command line debug/trace mode. I believe it would greatly reduce the amount of Journeymen support and free you up to concentrate on supporting the Ninjas (the development community).

RE: positioning
Understood and appreciated. Again, kudos for your work and efforts that you put into this.

RE: Queen's and students
There are plenty of undergrads and grad students out here that would *love* to have ExifTool on their CVs. I encourage you to leverage those resources.

Doug

[dhackney]

Phil,

I didn't want to start another thread on this so put it here. Thought you could move it to a new thread if you wanted/needed to solicit further input.

Goal: manage the expectations of new arrivals and position them for success with ExifTool

MS Word file attached.

I lost the embedded links when I pasted into the editor, but they are in the Word file.

Doug

---------------------------------------------------------------------------------------------------

Welcome to ExifTool!

ExifTool is the world's leading utility to manage, fix and manipulate photo, video and PDF document metadata. ExifTool is used by millions of people every day to solve metadata problems.

ExifTool can help you:
•   Save broken digital photos
•   Correct dates and times on digital photos
•   Fix missing information on digital photos
•   Copy, insert and manage keywords/tags
•   Rename digital photo files based on any metadata attribute, e.g. date, photographer, location, etc.
•   Create advanced metadata workflows
•   Build metadata tools and applications

What ExifTool Costs
ExifTool is an open source software project, so the cost is zero.

How You Get Help
ExifTool is supported by its development team and by the worldwide ExifTool community.


Solving Problems with ExifTool
If you found ExifTool, you probably got here because you've got a problem with your digital photos.

At this point, you may be frustrated and even desperate for a solution that will fix or save your photos.

The first step is to immediately back up your photo files onto another disk drive or into another directory/folder so that anything you attempt to do can be reversed by restoring the original photos. You can use Windows Explorer or Finder to back up your photo files.

Since ExifTool can require intermediate to advanced computer skills, if you are a regular computer user, the best next step is to try and solve your problem with one of the many point-and-click digital photo tools that are available prior to going up the learning curve for ExifTool.

If you need to shift the date/time of photos because you forgot to set or adjust your camera's clock, add a photo caption, geolocate your photos, view and/or edit the metadata embedded in your photos or other basic tasks then one of the point-and-click tools is a great choice.

Some good point-and-click photo metadata tools include:

Windows
•   ExifTool GUI for Windows: GUI for viewing meta information with some editing features
•   Exif Stats Utility: Analyzes images to tabulate apertures, exposure times, etc
•   Auto ISO Tool: GUI front-end for ExifTool to patch Canon ISO information
•   ExifAuto: GUI front-end for ExifTool to perform simple operations
•   GeoSetter: Utility for showing and changing geo data of image files
•   ExifMixer: GUI extension for the exiftool command-line interface
•   ExifDropper: Graphical front-end for ExifTool with drag-and-drop
•   GPStamper Geotagger: A simple Windows GUI front end for geotagging images
•   AvPicFaceXmpTagger: Write Picasa 3.5 face recognition information as XMP to your images
•   FileTagSleuth: MP3, ID3 and EXIF viewer
•   Exif Copier: Copy Exif data between sets of multiple images
•   ExifyMe: Restore Exif metadata from an original image
•   PhotoMove: Automatically move photos to directories or folders based on Exif date taken
•   TinyExif: Simple Windows GUI for ExifTool written in Python using the Qt4 toolkit

Mac OS X
•   Download a stand-alone droplet to extract preview images from RAW files (thanks to Brett Gross)
•   Download three droplets to extract information [exiftool must be installed] (thanks to Rob Lewis)
•   Tierprogramm: Droplet to perform some basic operations
•   MacMetaMod: Droplet for adding Keywords to images
•   GPSPhotoLinker: Geotagging on the Mac
•   PhotoGPSEditor and PhotoInfoEditor: Geocoding utilities
•   MetaDataMover: GUI-based automator utility for moving/renaming images
•   CS1ToCR2: Utility that uses Sony GPS-CS1 log files to add GPS information to CR2 images
•   SetEXIFData: Utility to set EXIF data
•   Geotagger: Droplet for inserting GPS coordinates in your photos
•   Raw Photo Processor: Raw converter for Mac OS X
•   GraphicConverter: Full-featured image editor
•   GeoTag: Geotagging application for iPhone and OS X
•   ImageFuser: Fuses multiple exposures of a scene into one image with improved detail/exposure
•   GeoNamesTagger: Docklet to easily update image metadata with location specific information

Multi-Platform
•   ImageIngester: Windows and Mac OS X image workflow automator
•   gpsPhoto: Geotag your images from a GPS (GPX) track log
•   renrot: Perl utility to perform various processing tasks on images
•   Proxel EXIF Tool: Photoshop plugin based on ExifTool
•   GPicSync: Windows/Linux utility to geocode photos from a GPX track log and create KML files
•   Jeffrey's Exif Viewer: Web utility to view meta information in online images
•   Jeffrey's Geoencoding Plugin for Lightroom: Geoencode your photos from within Lightroom
•   FlickFleck: Tool to transfer images from memory card, rotate, rename, and organize by date
•   gpx2exif: Python script to geocode photos and produce KML files from a GPS track log
•   Geotag: Open source Java-based geotagging application
•   PhotoGrok: Java-based GUI front-end for ExifTool to display images organized by any EXIF tag
•   XnView: View and convert graphic files
•   Mapivi: Open-source and cross-platform picture manager
•   ResourceSpace: Open source digital asset management system

Linux
•   rawimage: A kfile plugin and thumbnail image handler for RAW formats
•   Sagittarius: Linux XMP metadata editor
•   Photo related scripts: Includes EXIFInfo script to provide statistics from EXIF data
•   Fotoxx: Open source Linux program for photo editing and collection management
•   Hugin: Panorama photo stitcher



What You Need to Know to Use ExifTool

If you've tried those tools and can't solve your problem, then it is very likely that ExifTool can.

However, as mentioned above, ExifTool can require intermediate to advanced computer skills. The level of skill required depends on the problem you are trying to solve.

Things you will need to know about to utilize ExifTool include:
•   Command line terms, commands, actions, keystrokes, etc.
•   Navigating around your disk drive(s) via the command line
•   ExifTool command line syntax

Additional advanced capabilities are available with ExifTool if you know:
•   How to create and edit an ExifTool config file
•   Batch or scripting commands and syntax for your computer operating system
•   Batch or scripting variable management for your computer operating system
•   The Perl programming language


Existing Solutions Built with ExifTool

The ExifTool community has developed numerous solutions to common digital photo problems. Many of those solutions can be found on the ExifTool forum here: https://exiftool.org/forum/index.php

Please note, however, that solutions mentioned in the forum may not be well documented and may assume an intermediate to advanced level of computer operating system and programming language capability.


Known Limitations
In order save you from spending a lot of time here and eventually hitting a wall with ExifTool's capabilities, here's what ExifTool cannot do:
-   Change the digital photo file create date
-   Make logic (if/then) decisions based on operating system (OS) level file attributes such as file extension case (upper or lower case)
-   * list more known limitations *

Other known limitations of ExifTool include:
-   * list other known limitations *

Some capabilities of ExifTool require the ability to create and manage scripts.

Things that ExifTool can do that require scripting include:
-   * list known capabilities or categories of capabilities that require scripting *

Advanced capabilities of ExifTool require the ability to develop applications and leverage the ExifTool API.

Things that ExifTool can do that require the API include:
-   * list known capabilities or categories of capabilities that require the API *


If your problem is not excluded by a known limitation of ExifTool then the solution to your metadata problem probably lies here.


How To Succeed With ExifTool

ExifTool is a very powerful utility. With that power comes complexity.

If you are like most people, you want to solve your problem as quickly and easily as possible.

In order to reach that goal you need to have the minimum skills and information required to use ExifTool to solve your problem.

Although there is extensive documentation and a rich library of forum posts regarding ExifTool and its capabilities, there is currently not a lot of step-by-step tutorial information available. This is a gap that the ExifTool community is working to fill, and hopefully there will be a library of tutorials available soon.

In the meantime, there are steps you can take to shorten the time it will take to solve your problem.

Because most people interact with ExifTool via the command line interface, the first thing you need to know about is the command line on your computer operating system. Do some searches and get yourself up to speed on how to initiate a command line session, the basic commands and, especially, how to navigate around your disk drives by the command line.

In particular, you'll need to know how to:
•   Change directories
•   Execute a command
•   Recall past commands
•   Edit the command line
•   Copy and paste on the command line

You may be able to solve your problem with a single ExifTool command line statement containing all required ExifTool commands, options, parameters, etc.

That single command line can get very long, very confusing and very difficult to manage.

One way to make that command line statement less obtuse is to place some or all of the elements into a separate file that contains ExifTool commands and parameters. That file is called an ExifTool arguments file, and that is something you will want to know about if you have a complex problem. You can learn about the arguments file by searching for -@ ARGFILE on the documentation page here: https://exiftool.org/exiftool_pod.html

The next level of capability with ExifTool involves script or batch on your computer operating system. A script or batch allows you to perform multiple actions with ExifTool, as well as a wide variety of operating system level tasks. Again, do some searches and read some tutorials on the basics of batch/scripting on your computer. At a minimum, you will need to know how to create, edit and execute a batch or script.

In addition, you will want to know how to:
•   Use special script or batch syntax required by your operating system
•   Display messages on the computer monitor
•   Pause the batch or script and wait for user input

Additional script/batch capabilities that will be very useful include:
•   Creating and managing variables
•   Creating and populating text files with the log of your job
•   Creating and managing menu prompts and inputs

Lastly, it is very important to know the unique ExifTool requirements for your computer operating system. Almost all of these unique requirements are related to your operating system's shell and piping requirements. Almost all of these unique requirements involve particular formatting requirements for commands and parameters.

Some widely encountered unique requirements include:

Windows:
-   Batch files require %% instead of a single % on the command line
-   Double quotes " are required versus single quote ' used on Unix/Mac
-   * add additional common known Windows requirements *

Unix/Mac:
-   Single quote ' is required versus double quotes " used on Windows
-   * add additional common known Unix/Mac requirements *


In summary, success with ExifTool requires the following prerequisite knowledge:
•   Command line commands, syntax, navigation and editing
•   ExifTool arguments file structure and use
•   Script / batch syntax, structure, variables and input / output commands
•   Unique ExifTool requirements for your operating system


Once you equip yourself with the prerequisite skills and knowledge, you can solve almost any metadata problem with ExifTool.

Please ensure you know the prerequisites before attempting to solve your problem with ExifTool.

Diving right into the documentation is an easy way to be overwhelmed and intimidated by the deep complexity and capability of the tool.

It really is possible to easily solve your problem with ExifTool, but if you attempt to solve your problem by skimming the documentation and/or by hacking around with a sample command line you copied from the forum without the prerequisite knowledge, you are setting yourself up for failure and frustration.

Instead, set yourself up for success by obtaining the prerequisite knowledge first. Success can and will often quickly follow.


Additional Steps to Success with ExifTool:

Search the ExifTool Forum
Many problems have already been solved with ExifTool, and you can find those existing solutions by searching for terms related to your problem on the forum here:  https://exiftool.org/forum/index.php

Note that the forum search syntax uses the dash "-" character as an "exclude from this search" parameter. If you want to search for an ExifTool command or parameter that leads with the dash "-" character, you'll need to enclose the term in double quotes, such as "-tags".

Search the Web
Many people have documented their solutions to metadata problems using ExifTool in blogs, web pages and other forums.

Again, those posts and pages may assume intermediate to advanced knowledge of computer operating systems and programming languages.

Post a question on the ExifTool Forum
The ExifTool development team and the user community provide free support on the ExifTool forum here: https://exiftool.org/forum/index.php

Before posting a question, please ensure that you have equipped yourself with the prerequisites for success listed above and that you have made an honest, earnest effort to discover the answer in the available documentation and forum posts.

When you post a question, please ensure the post title is specific and descriptive of your issue. A good forum post title is "Unable to copy xmp CreateDate tag to tiff file on Mac." A poor forum post title is "can't get ExifTool to work."

When you post a question, please include:
•   Problem: the metadata problem you are trying to solve
•   Operating System (OS): the computer operating system you are using
•   ExifTool version: the version of ExifTool  you are using (obtained by the command line: exiftool –ver)
•   Issue: The specific issue you are having with ExifTool. Please be very verbose and include copies of all error messages, directory listings, file types, etc. that are involved with your challenge.

You can generate a listing of what ExifTool is doing when it executes by including the –v flag in the command line. The levels of detail vary from –v0 to –v5.

You can pipe that detailed output into a text file by including a –w flag or using the operating system's shell, such as a > parameter in windows.

An example command line to capture the detail in a file on the Windows operating system is:
-exiftool –listg1 sample.jpg –a –v2>exiftool-output.txt

That command will create and populate a text file named exiftool-output.txt that can be opened with Notepad.

You can attach files to forum posts to help document your challenge.

Please be patient with forum posts and remember that everyone who is helping you is doing so for free, on their own time, for no reward other than the satisfaction of helping a fellow ExifTool user. Respect and appreciation are in order.

Please help keep the ExifTool forum a respectful and positive community.


Support, Documentation and FAQ

ExifTool support is provided via the ExifTool forum here: https://exiftool.org/forum/index.php

ExifTool is extensively documented and includes an FAQ.

The basic documentation includes:
•   ExifTool FAQ
•   exiftool Application Documentation
•   Supported File Types
•   System Requirements
•   Example Output
•   Tag Names Explained

Advanced documentation includes:
•   ExifTool Tag Names
•   Working with Metadata Sidecar Files
•   Sample ExifTool Configuration File (user-defined tags)
•   Image::ExifTool API Documentation
•   "Under the Hood" of ExifTool
•   JPEG Image Samples (and Table of Makernote Types)
•   Canon RAW (CRW) File Format Specification
•   MIE 1.1 File Format Specification (pdf)

General metadata information includes:
•   Problems with current Metadata Standards
•   Comments on the Subject of Writing Meta Information

[Phil Harvey]

Hi Doug,

Re your reply post, one minor point:  I make a distinction between the Ninjas (power users who actually run the ExifTool code or incorporate ExifTool into their projects or applications) and the developers I mentioned (people who aren't interested in running ExifTool, but use the resources I supply to build their own metadata tools -- ie. Exiv2).

Re your "Welcome to ExifTool" post:

Wow, thanks for the effort.  I think this is a great idea, but my impression is that your document wades into the deep end far too quickly for ToolRefugees.  If I were a ToolRefugee this document would likely scare me away before I went any further.  I wouldn't mention scripting and/or config file type stuff at all, unless you put it into an advanced chapter.  Also, I would try to demonstrate as early as possible just how easy it is to use ExifTool, and get more specific with command-line examples early in the text, with hyperlinks and balloons for people to get more help with things that may confuse them.   These changes would boil down making this more of an "ExifTool Quickstart" document, more than the introduction you had in mind.

And one more minor point:  ExifTool can make decisions based on the case of the source file extension.  What you wanted to do was to make decisions based on the case of the extension of some other file.  (But I just thought of a trick that would allow this also -- I'll post in the other thread.)

- Phil

[dhackney]

Phil,

RE: devs vs. Ninjas
Understood. My comment should have read (and the development community).

RE: extension case
You've exposed the soft underbelly of my ExifTool status. I only know as much about what ExifTool can do as I need to get my problem(s) solved. 

RE: welcome to ExifTool
From the communications standpoint, a welcome/expectation-management document is very different from a quick start guide.

I perceive your greatest challenge to be high levels of support interaction for people who are new to the tool. The welcome document can help mitigate some of that by informing people up front what it takes to make use of the tool. As you've said yourself, you designed it to be an advanced tool for a niche audience. The ToolRefugees, by definition, are a mass audience. You really can't have it both ways, which is why I agree with your assessment that if metadata turns into a market of billions scale problem, then a more accessable solution will arise. In the meantime, however, ExifTool isn't going to get any more approachable or accessible than it is, at least in the near term.

ExifTool is like a rebreather for diving. It's highly advanced, has a learning curve, and can only be safely and successfully used by those who are properly equipped, trained and have the prerequisite knowledge and experience.

Right now, you've got lots of people showing up to the party without so much as a bathing suit, much less the skills and experience to stay alive on an extended dive.

Just like scuba diving with a rebreather, you need to prepare yourself to be successful with ExifTool.

You, kind sir, are the designer, engineer and builder of the rebreather. Yes, to you it is easy to use ExifTool.

For the rest of us out here, outside the fishbowl, it is, ... I'm trying to find the right phrase here ..., not so easy. 


Based on many years of consulting and management experience with a wide variety of software companies, from the communications and customer (user) management perspective, IMO, you are a lot better off to manage people's expectations about what they need to be successful seperately from a "Quick Start" document. They are two different pieces of communications designed and produced to accomplish two different, but complimentary, goals.

People, especially ToolRefugees who have never done anything in their lives but point and click, are not going to go from ground zero to achieving success with ExifTool in a couple of paragraphs in a welcome document, no matter how many code examples you provide.

Instead, prepare a seperate "Quick Start Guide" document that walks people through a few simple command lines.

That's the document for the conceptual / contextual information such as:
•   (everything on an ExifTool command line is processed together (except for this) (and that))

•   Anything on the command line can be in any order except A must proceed B except when it is paired with D, as in DA

•   There are X basic types of command lines. They take the form of:
     1.(source operator)
     2.(source operator target)
     3.(Source (operator (optionA paramter1) (optionB parameter2)) ((format) target))
     4. etc.


That's the document for all those exceedinly verbosely documented and attributed sections that I suggested earlier. That's the document where, via a few step-by-easily understood-step, heavily documented code examples, you can launch a fully prepared ToolRefugee toward success and give the Journeymen what they need most: fully documented and referenced code samples.

I am not qualified to write the "quick start" guide or I'd take a pass at it.

I'm like everybody else. I just find a command line that I can hack around on and spend a lot of time searching for answers until I either make it work or give up.

The "Welcome to ExifTool" document can help ToolRefugees and people like me confirm or deny we're ready for the ExifTool journey.

The "Quick Start Guide" would be a godsend to people like me who wander back in here every time we've got a problem to solve and need to re-learn all of the ExifTool stuff we forgot in the intervening months/years/professional eras.

Doug

[dhackney]

Also, I keep forgetting to include this:

It would be a great thing for people who read left-to-right languages if you could always have the data flow left to right.

Meaning, when you give examples of source to target, always have the data flow left to right.

It makes it more challenging for left-to-right language reading people to follow examples when some flow -targettag<sourcetag and others flow -sourcetag>targettag.

For left-to-right language reading audiences, such as English, German, etc., sourcetag>targettag is much easier to follow.

[Phil Harvey]

Hi Doug,

I hear what you are saying about a Welcome document, but I also heard what you said about ToolRefugees not reading documentation, so requiring them to wade through an extra level of documentation is worrisome to me.

People [...] are not going to go from ground zero to achieving success with ExifTool in a couple of paragraphs in a welcome document [...]

Ground zero --> download "exiftool-##.zip" --> unpack "exiftool(-k).exe" --> drag-and-drop image --> success!

Easy.  This is what most people want to do -- simply extract metadata from an image.  Many other simple tasks can also be accomplished with drag-and-drop by adding options to the "exiftool(-k).exe" file name.

For you, I agree that ExifTool is more complicated.  But that is because the task you wanted to achieve was much more complex.

It makes it more challenging for left-to-right language reading people to follow examples when some flow -targettag<sourcetag and others flow -sourcetag>targettag.

I agree, but if I would standardize on a single direction it would have to be right-to-left because things like "-dsttag<$srctag1 $srctag2" can not be done left-to-right.  But I think that right-to-left actually makes more sense from a programming point of view, when you consider expressions like -tag=value.

- Phil

[dhackney]

Phil,

You make excellent points here.

RE: easy
I think we both share the view that for the ToolRefugee type of user the "easy" tasks, such as extract/read/view metadata from an image, is best done in a tool such as ExifTool GUI.

As the metadata point-and-click tools continue to grow in capability, they can take on more and more tasks of higher levels of complexity all within a point-and-click environment. 

My take on the ToolRefugee persona is that they end up staring down ExifTool only when they hit the limits of the capabilities of the point-and-click tools.

BTW, that was the reason that I put the list of point-and-click tools at the head of the "Welcome to ExifTool" draft. I wanted the ToolRefugee persona to first make sure they had fully exhausted all possible solutions within the environment they know best: point-and-click.

When a true ToolRefugee gets to ExifTool, they are, by definition, a refugee from a class of tool that cannot meet their needs.

In my view, that's where they need to know how to be a success with ExifTool, what the prerequisites for success are, etc. Once they meet those prerequisites, then they can leverage a Quick Start Guide to rapidly get up to speed and accomplish goals.

Like the ToolRefugees, I use the metadata capbility in my workflow and in the point-and-click metadata tools for whatever I possibly can before I use ExifTool. They are just quicker and easier to use on an ad hoc basis. If you don't use ExifTool all the time, there's no possible way to remember all the flags / options / parameters / etc. required to do what you want.

I think that type of utilization is typical of all personas except the Ninjas and the application developers.


RE: read
Regarding what they will read or not read, in my experience, people will read things that they believe will help them quickly reach their goals and achieve success.

I think the Welcome document and the Quick Start document both fall into that category. Typically, the full suite of documentation does not.


Doug

[Phil Harvey]

Fair enough.  It may be that there is a simple Welcome document that would make us both happy.  My main concern is that we don't scare people away if they just want to do something simple.

I agree that ExifToolGUI should be used for simple tasks in Windows, but ground zero in Mac/Linux land already belongs to the ToolRefugee because there are no similar general tools for these systems (and simple drag-and-drop doesn't work either).

- Phil