robcole.com Forum
November 17, 2017, 02:48:03 pm
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Welcome to SMF For Free
 
  Home Help Search Gallery Staff List Login Register  

It ain't workin' for raw files.

Pages: [1]
  Print  
Author Topic: It ain't workin' for raw files.  (Read 1011 times)
alanterra
Newbie
*
Posts: 5


View Profile
« on: August 24, 2011, 08:00:00 pm »

Hi

I just installed RC ExifMeta, and maybe I don't understand something. (Well, probably I don't understand something). But...

I followed the directions, updated a bunch of files (all raw) and a bunch of metadata fields show up. But when I go to the filter panel, there are no distinct values.

I then updated some jpgs, and for the jpgs I can see, e.g., distinct values for XMPdc:Title, but for the raw files there are no values for this field in the RC Custom panel. The lightroom standard panel EXIF and IPTC does show a value for Title for both raw and jpg files. This value is stored in the xmp sidecar for raw files, but in the file itself for jpg files.

Am I doing something wrong? Or is this just a limitation of the plugin?

TIA
Report Spam   Logged

Social Buttons

areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #1 on: August 24, 2011, 08:25:14 pm »

The 'RC Custom' "tagset" is a mostly empty shell, intended to be customized by you, if you want (and if you can deal with the lua syntax of the text configuration file, aka "Advanced Settings").

The 'RC Standard' tagset is more complete, so step 1 would be to try that one.

Another option is to use the Lightroom tagset for "All Plugin Metadata".

Anyway, library filters and smart collections should have an entry for each field enabled (leftmost checkboxes in plugin manager), *except* the "big block: - are you not seeing that? (Requires reloading after change/commit in Windows, and restarting Lightroom if Mac).

PS - It might help to post a screen-cap or two to make sure we're talking about the same things...

PPS - ExifMeta just blindly translates exiftool output - no interpretation / no formatting... So, whatever is returned by exiftool should be available as an option (to include in lib-filters/smart-collections iff checked in plugin manager, committed, and reloaded/restarted).

R
Report Spam   Logged
alanterra
Newbie
*
Posts: 5


View Profile
« Reply #2 on: August 24, 2011, 08:34:44 pm »

I think you didn't follow what I was trying to say. The first two are the EXIF-IPTC data for a jpg, the next two are the same for a raw file.

I note that if you are just using exiftool to extract data from a raw file, it does not check the xmp side car. You need to write your program so that you check both the raw file and the sidecar, and then add smarts to combine any inconsistent fields.

Report Spam   Logged
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #3 on: August 24, 2011, 09:01:56 pm »

Sorry. I see what you mean now:

exiftool includes xmp info as "exif-metadata" when its in the jpeg (or DNG?), but not when its a proprietary raw.

Begging the question: Would it be better to strip it out in the case of jpeg/DNG, or add it in the case of proprietary raw, or just leave it the way it is.

I mean, I personally don't need xmp info via exif-meta: *almost* everything that's in it is already available through a combination of Lightroom native (e.g. title, rating) and DevMeta (e.g. develop settings).

So, at this point, I'm not sure whether to consider this a bug or a feature. If you think its the former (more of a bug), please educate me: are there any specific fields you could mention that are otherwise unavailable?

Thanks,
Rob
Report Spam   Logged
alanterra
Newbie
*
Posts: 5


View Profile
« Reply #4 on: August 24, 2011, 11:03:45 pm »

Hi Rob

So here is the thing. xmp is incredibly complicated, and the rules are not at all clear.

I had to figure out how to write xmp data to files, so that it could be read by Lightroom, so I had to understand some of this.

The first thing to realize is that the rules for which kinds of files require "xmp sidecar" files is defined in an ad hoc fashion, and there don't appear to be any publicly defined rules about them. Adobe has adopted the strategy of always using sidecar files for proprietary raw formats, but never using them (and never consulting them) for formats like jpg, psd, dmg, tif, etc. This is not a requirement of the spec (which Adobe wrote), but only Adobe's particular decision. Adobe says (in "XMP Specification"):


It is recommended that XMP metadata be embedded in the file that the metadata describes. There are cases where this is not appropriate or possible, such as database storage models, extremes of file size, or due to format and access issues. Small content intended to be frequently transmitted over the Internet might not tolerate the overhead of embedded metadata. Archival systems for video and audio might not have any means to represent the metadata. Some high-end digital cameras have a proprietary, non-extensible file format for “raw” image data and typically store EXIF metadata as a separate file.
If metadata is stored separately from content, there is a risk that the metadata can be lost. The question arises of how to associate the metadata with the file containing the content. Applications should:
●   Write the external file as a complete well-formed XML document, including the leading XML declaration.
●   The file extension should be .xmp. For Mac OS, optionally set the file’s type to 'TEXT'.
●   If a MIME type is needed, use application/rdf+xml.
●   Write external metadata as though it were embedded and then had the XMP Packets extracted and catenated by a postprocessor.
●   If possible, place the values of the xmpMM:DocumentID, xmpMM:InstanceID, or other appropriate properties within the file the XMP describes, so that format-aware applications can make sure they have the right metadata.
For applications that need to find external XMP files, look in the same directory for a file with the same name as the main document but with an .xmp extension. (This is called a sidecar XMP file.)


If you read this carefully, they are only giving guidelines about whether to use sidecar files, and suggesting that in the absence of a good reason, don't use them. Yet, in practice, Adobe seems always use xmp sidecars for every file format other than the "open" formats like jpg, tif, dng, psd, gif, png, etc.

Once you have a sidecar file, this means that the same field, like xmp:dc-title, can exist in both the raw file and the sidecar, and the field can have different values in the two files. In fact, one of my cameras (I can't remember which right now) put the camera model into the title field, which was stupid, but there it is. So when I give a "title" to a photo, and Lightroom writes it to the xmp sidecar, there is a guaranteed conflict between the two values.

Now we get the other complication that various xmp fields are supposed to have the same meaning. As far as I can tell, XMPxmpRights_Marked and Photoshop_CopyrightFlag are the same field, under different names, and, potentially, different values. Adobe applications have a lot of smarts to try to keep these various fields which mean "the same thing" in sync. See the Adobe document "IPTC Standard Photo Metadata (July 2010)" which says

The same information can appear multiple times within Adobe Photoshop's Custom panels/tabs. The data is not duplicated. It is stored only once, and all the panels, tabs or schemas that read or write to that field use it as a “shared property.” Some IPTC Core properties already appear as part of Adobe’s Description, Origin and Categories in the File Info panels as well as the Adobe Photoshop File Browser’s Metadata panel. As an example, enter the name “John Doe” in the “Creator” field of the "IPTC Contact" section of the IPTC tab, then switch to the Adobe Photoshop “Description” tab — notice that the name “John Doe” automatically appears in the "Author" field in that panel. Change that “Author” entry to “Jane Doe” and it will appear in the “Creator” field in the IPTC Contact section using the new name. Both tabs simply provide two different views of the same metadata.

When image files are opened and saved by Adobe Photoshop (version 7.01 or higher), the “IPTC Fields” stored within those files are synchronized with the stored XMP metadata. To maintain compatibility with older versions of Adobe Photoshop, no pre-existing mappings have changed.

Workflow interoperability is another reason why some metadata appears shared. For example, many types of documents need a “Title” (technical specifications, tax forms, blueprints), but only news items need an IPTC Subject Code. If another metadata standard has already defined a useful property, such as the PLUS schema (http://www.useplus.org/), it is adopted in one of the IPTC schemas. In many cases (title and keywords are two examples) this mapping was already established by Adobe Photoshop’s mapping of binary IPTC IIM metadata to XMP. Likewise, metadata entered using the IPTC Core panels will continue to appear in other locations. The IPTC Core, IPTC Extension, and PLUS fields that are shared are noted in the descriptions of each field that follow.


What this means (as far as I can tell), is the the "same" information can be (a) in one of two different files, and (b) in any of many different xmp fields. Adobe attempts to keep all this in sync, or if it can't, it figures out which is the "correct" one using various heuristics.

Now, exiftool is not an Adobe product, and it has different rules. As I found out through long experimentation (this doesn't seem to be written down anywhere, but maybe I just didn't find it), exiftool tries hard to unify the various "names" for the "same field". You can ask for the -title field, or you can ask for the -xmp-dc:title field. If the former, exiftool tries to report the value after consulting various xmp fields all of which might be the "title" field. If the latter, it just looks at the one variant of the title field. And if you set the value (using "-title=NewValue"), then it tries to keep the various "title" fields in sync, while if you set the title using "-xmp-dc:title=NewValue", it just sets that Dublin Core xmp field.

However, the one thing exiftool does not do (which Lightroom does) is to consult both the raw file and the xmp sidecar file, and try to combine the data from both files into one unambiguous set of metadata. Similarly, when you set the metadata for a file, Lightroom will decide whether to write to the file or a sidecar file depending on the file type. Exiftool, on the other hand, will just write to the file you tell it to, if it can.

What this means is that any program, like your plugin, needs to decide whether to consult an xmp sidecar or whether to consult the file itself (for instance, the rule might be, to read from raw files, if there is an xmp sidecar, use it, if not use the raw file directly, while for writing always use the xmp sidecar, but for jpg/tif/psd/dng/gif/png always ignore the sidecar if it exists -- these are, I believe, Adobe's rules).

But none of this seems to be written down, and you need to figure out what to do by following what others do.

I suspect that much of what I am saying here is already well-known to you. And. i'm sure, some of what I am saying might be wrong.. But this is my understanding of how metadata currently works.

I find this all very annoying, and wish that someone could come in and set up some simple rules, write them down, and we could all follow them

In short, if your metadata plugin is not consulting the xmp sidecars of raw files, in a way consistent with how Lightroom uses sidecar files, I don't think I can use your plugin. It might be perfect for everyone who converts their raw files to dng's. I have a legacy of tens of thousands of raw files, and don't want to convert them right now.

Thanks for all your good work.

Alan
Report Spam   Logged
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #5 on: August 24, 2011, 11:50:21 pm »

Thanks for the info Alan.

I get the general idea, but I'm still not sure whether to take remedial action or not.

You say: "if your metadata plugin is not consulting the xmp sidecars of raw files, in a way consistent with how Lightroom uses sidecar files, I don't think I can use your plugin".

I say: "Why not?".

I mean, so far, I have only used exif-meta for things like Nikon CLS flash settings, and such stuff. In fact, I generally filter all stuff with "XMP" in it, since it doesn't interest me. So the critical question in my mind is still unanswered:

What info is missing or wrong due to present handling?

I dont necessarily need a complete list, just some idea - i.e. is there some interesting stuff added to xmp by photoshop, that isn't otherwise available in Lightroom?, or is it stuff added by other apps, or is it stuff added by Lightroom itself? Any examples??

Summary:
-----------
I get that there is some stuff in xmp that's not in original raw, and that there is some potential for redundency/overlap/conflict. But I don't understand the value of getting new stuff from xmp, or trying to reconcile differences in common fields between the two. I need to really understand the problem, before I decide whether to embark upon a solution.

Rob
Report Spam   Logged
alanterra
Newbie
*
Posts: 5


View Profile
« Reply #6 on: August 25, 2011, 12:51:02 am »

As just one example, I use the title field as a unique identifier for my photos (so photos with various names are all united as "the same" photo). This is the suggested use for the title field in Adobe's xmp specification. (see the screen grabs above for the title values that I use).

However, Adobe does not allow a column in the metadata selection panel for title, nor does it allow sorting by title, both of which would be very useful to me.

When I set the title field, it is written into the xmp sidecar for all my raw files. So, to your plug-in, my raw files appear to not have a title field. This means that I cannot search, nor sort, the title field in my raw files using your plugin. If I was using dng files (and so not using sidecars) it would work fine. Using LR's built-in tools it works fine, because LR consults the xmp sidecar files for raw files. However, even though your plugin offers features I would love to have, I can't use it because it doesn't see the title field for most of my files.

Does this make it clear?

Best

A
« Last Edit: August 25, 2011, 12:52:53 am by alanterra » Report Spam   Logged
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #7 on: August 25, 2011, 03:52:29 am »

Hi Alan,

I'm still struggling to understand.

Perhaps the example of Title was not a good one, since Lightroom itself natively supports Title in library filters as "Text", likewise for smart collections, and of course its present in the right-hand lib panel.

You can't sort by Title, but then you can't sort on *any* plugin metadata, either.

Is there another example for something that is *not* already supported via some other means?

Thank you for being patient with me,
Rob





« Last Edit: August 25, 2011, 04:30:16 am by areohbee » Report Spam   Logged
alanterra
Newbie
*
Posts: 5


View Profile
« Reply #8 on: August 25, 2011, 08:27:50 am »

Rob

I'm out of time today. Why don't you shoot some raw images and play around with them. I think you'll see what I am talking about. Check out the screenshots above too.
Cheers

A
Report Spam   Logged
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #9 on: August 25, 2011, 08:39:54 am »

I mostly shoot raw-only (proprietary, not DNG), and play lots... - when you get some time please fill in the blanks...
Rob
Report Spam   Logged
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #10 on: December 09, 2011, 10:20:45 pm »

I consider this issue closed unless/until somebody can come up with an example of xmp (sidecar) metadata that is desired, and isn't already supported better in Lightroom proper (or my DevMeta plugin...).
Report Spam   Logged
jwoude
Newbie
*
Posts: 1


View Profile
« Reply #11 on: May 13, 2012, 01:29:06 pm »

Hello Rob,

Today I think I came across just such a tag:

GPSAltitude.

The RC exifmeta plugin works fine for jpg's:
The altitude - as recorded in the jpg file - is shown in the library search panels of Lightroom

The altitude - as recorded in xmp sidecars of Canon CR2 raws -  is not shown in the library search panels.

The altitude *is* shown in the right side Metadata panel of Lightroom for all types of files.

This issue has surfaced, because the latest version of Jeffrey's geotagging plugin allows me to populate the GPSAltitude fields for all my 55.000 geotagged photos using Google's Altitude lookup servers.

I would want to use the filter so answer questions like:
What is the highest/lowest altitude  I have ever taken a picture?
On my last hiking trip what was the highest altitude I have taken a picture at?
etc

Regards

Jos van der Woude
« Last Edit: May 13, 2012, 02:58:49 pm by jwoude » Report Spam   Logged
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #12 on: May 15, 2012, 05:18:19 pm »

Thanks Jos.

After Adobe releases the SDK for Lr4, I will reconsider this.

Rob
Report Spam   Logged
PSimon23
Newbie
*
Posts: 1


View Profile
« Reply #13 on: June 11, 2012, 08:11:26 am »

Similar matter has already been discussed at yahoo answers. I can post the link if needed
Report Spam   Logged

I'm on Twitter and my essay
areohbee
Administrator
Jr. Member
*****
Posts: 61


View Profile
« Reply #14 on: September 12, 2012, 06:04:11 am »

ExifMeta 5.1 supports option to include metadata from xmp sidecars along with raw metadata.
Report Spam   Logged
Pages: [1]
  Print  
 
Jump to:  

Bookmark this site! | Upgrade This Forum
SMF For Free - Create your own Forum | Buy traffic for your forum/website

Powered by SMF | SMF © 2016, Simple Machines