23. Advanced Metadata Techniques

Sometimes it is impractical, inefficient or undesirable to expect receivers of Items to retrieve rich structured information about concepts from remote resources. In these cases a sub-set of metadata relevant to the Item can be stored directly in the Item using Assert. For example:

For example, a Subject property can reference a local Assert, allowing the use of properties that are not directly supported by the <subject> element:

This example shows how a concept identified by "Acme Widget" which is a Subject of the Item, can be expanded locally using an Assert.

It illustrates the use of @literal as an identifier whose scope is purely local to the Item. Both @qcode and/or @uri may be used as globally-unique identifiers but note that the concept information contained in an Assert in an Item is ALWAYS local in scope, whatever the type of identifier. Receivers must NOT use the information contained in the <assert> outside the scope of the Item providing the information, for example by extracting it and storing it in a cache of concepts, or in a knowledge base.

This is because the <assert> is intended to be a single-use "snapshot" of information about a concept, if the information needs to used permanently and periodically updated, the full Concept Item or Knowledge Item, which also convey management metadata, should be used instead.

Many media organisations use the IPTC’s IIM standard (Information Interchange Model), particularly for pictures. IIM has been embedded into professional picture workflows because it was adopted by Adobe for the "IPTC Header" fields in Photoshop. This has been succeeded by Adobe XMP and IPTC Core for XMP, (See Mapping Embedded Photo Metadata to NewsML-G2)

When migrating IIM or IPTC Core (XMP) metadata to NewsML-G2, <assert> offers a convenient method for conveying the contents of some fields which express the location shown by a picture using machine-readable or human-readable values.

For example, the location of the subject of a picture is conveyed using the NewsML-G2 <subject> property. However, <subject> cannot directly include detailed geographic information that may be contained in the embedded metadata. An Assert can be used to convey this information:

A simple example illustrates the use of <inlineRef> to provide supplementary information about a concept found in an XHTML press release. In this case, the article refers to "the Met". With 100% confidence, the provider declares that the name of this concept is "The Metropolitan Opera" (linking @id and @idrefs highlighted in bold):

Inline Ref can provide basic information about a concept, but more detailed information can be expressed by linking the concept in the content, via <inlineRef>, to an <assert> element, using @literal or @qcode.

A news provider uses a customer’s metadata rules to signal to the customer that the content is part of a collection of news entitled "Greek Debt". In the code snippet below, @creator expresses that the client rule was used to add this metadata, and @modified contains the date that the property was added. The @literal contains the customer’s code for "Greek Debt" (also given in the name child element)

When @custom is set to "1" or "true" as in this example, this is a signal to receivers that the metadata may NOT exist in other instances of a NewsML-G2 Item of the same guid and version number.

The @why attribute uses the "inferred" value from the IPTC Why Present NewsCodes, and this defines that the property was derived by lookup in a taxonomy or database. The @how attribute is not used as it is implied that the property was extracted using a tool.

The @custom attribute is one of several grouped under the Common Power Attributes, available to ALL elements in NewsML-G2 except the root elements of Items, elements of Ruby mark-up and a few elements of the News Message that are not root elements. The following table details the Group:

Since the <keyword> property was added to NewsML-G2 in v 2.4, we learned that its use case aligns closely with the pre-existing <subject> element. In v 2.10, the attributes @confidence and @relevance were added to <keyword>, and @why and @how were added to both elements.

From version 2.27 of the Standard onwards, @confidence and @relevance have also been added to <slugline>, <headline> and <description>, thus aligning these properties with <subject>, <keyword> and <genre> in the Descriptive Metadata Group. With the increased use of automated metadata generation by software tools, it helps both providers and end-users to be able to express these values as an aid to processing of Items.

Relevance is a measure of the usefulness of the concept to the user who wants to learn more about the principal subject matter of the content. A high relevance indicates that the metadata truly expresses the essence of the content, while a low relevance indicates a low correlation between the metadata and the content. For example, a concept that references Barack Obama would be 100 per-cent relevant to an article that about the 2012 U.S. presidential election. This would be represented as:

A concept referencing The White House would be less relevant, and might be represented as:

Confidence asserts the reliability of the association between the concept and the content and is usually obtained by software engines that automatically categorise news, where "100" is the highest value and "0" the lowest:

In NewsML-G2 v 2.10, an attribute @derivedfrom was added to elements that supported the attribute @why. However, this restricted the expression of the derivation to property values expressed as QCodes: in fact the use case for derivation is often to give this attribution to metadata encoded outside the NewsML-G2 framework.

In NewsML-G2 v 2.12, with an extension of @why to almost all NewsML-G2 elements, it was decided to add a specific <derivedFrom> element to NewsML-G2 as a child of the root element and using IDREFS to express the relationship between the derivedFrom element and the referring property. NOTE: The attribute @derivedfrom is deprecated from NewsML-G2 v 2.11 on.

Note the direction of the relationship, as illustrated by the following snippet: an article about an event in Barcelona uses the event’s Twitter hashtag "#sgbcn" as a keyword. The editor of the article subsequently creates a NewsML-G2 Event with a Concept ID of "ex-myevents:SGBCN" and indicates by the derivedFrom element that the <keyword> with the hashtag was derived from this concept. This is done by applying the keyword’s id "s1" to the idrefs of the <derivedFrom> element which represents the event in Barcelona:

Based on this it can be asserted that the value of the keyword property with the id "s1" has been derivedFrom the event concept identified by the QCode "ex-myevents:SGBCN".

See also Hashtags

The <derivedFromValue> element, introduced in NewsML-G2 v2.28, represents the non-Concept value used to derive the value of one or more properties in a NewsML-G2 Item. This complements the <derivedFrom> element (above), which represents the concept value used to derive the value of an element.

The derivedFromValue element is a child of any NewsML-G2 Item except <catalogItem> and may be inserted after <inlineRef> and <derivedFrom>. It is optional and repeatable. The mandatory attribute @sourceidref refers to the id of the element that provides the value used for the derivation; the mandatory attribute @idrefs refers to the ids of elements whose values have been derived from the value represented by this property.

For example, the following <slugline> has an @id="slug1"

The Item contains the following <subject> properties:

The derivedFromValue shows that the Subjects were derived from the value "MIDEAST-CRISIS" of the Slugline:

Automated processing using rules or AI tools has also led to a need to attribute the creation of a set of metadata properties. The new element <metadataCreator>, an optional child element of <itemMeta> complements the existing {element}/@creator attribute of an element, as follows:

Thus, for example:

Hint Points allow NewsML-G2 properties to be extracted from a target resource and added to the local document as an aid to processing. For example the caption of a remote picture may be displayed without loading the picture itself.

Hint properties must be consistent with the target resource but do not have to be extracted directly from it. For example a <headline> may not be available for a referenced image, but it is permissible to add <headline> at the Hint Point.

When using a property as a Hint it MUST be wrapped by its parent element(s) (excluding the root element), EXCEPT for the immediate child elements of <itemMeta>, <contentMeta> or <concept>. For example, to add a copyright notice Hint, the parent element <rightsInfo> must be supplied, as shown in the following examples of <itemRef> in a Package Item:

A <headline> is added directly because it is an immediate child element of <contentMeta>:

The following elements have a Hint Point. Follow the links to see more information about each of them. Note that, apart from Assert, they are Link 1 Type properties.