13. Controlled Vocabularies and QCodes

The key to resolving scheme aliases is the Catalog information, a child of the root element of every NewsML-G2 Item. A scheme alias may be resolved directly using the <catalog> element:

Using the information in <catalog>, a processor now has a Scheme URI that can be used as the next step in resolving the QCode.

Appending the QCode Code Value to the Scheme URI produces the Concept URI.

http://cv.iptc.org/newscodes/pubstatusg2/usable

The IPTC recommends that Scheme/Concept URIs can be resolved to a Web resource that contains information in both machine-readable and human-readable form (This is also a recommendation for the Semantic Web), i.e. they are URLs. The concept resolution mechanism used by the IPTC is http-based, and the NewsML-G2 Specification describes how an http-URL should be resolved.

Entering the above Concept URI in a web browser results in the following page being displayed:

A question often asked by implementers is: "What happens if I receive files from two providers who inadvertently have a clash of scheme aliases?"

The scenarios they envisage are either:

The answer is "everything works fine!"; QCode to Concept URI mappings must be unique only within the scope of each document in which they appear.

A processor should correctly process two files with different aliases to the same Concept URI:

This is because the concept resolution process is local to each document. The processor can unambiguously resolve the QCode to a Concept URI via the <catalog> in each case.

The following example does NOT work because the same alias is mapped to two different URIs within the same document and the processor is unable to resolve the QCode to a single Concept URI:

But the following is CORRECT because it is possible to have different aliases within the same document pointing to the same URI and the processor can resolve both QCodes:

In this document, there are many references to IPTC NewsCodes and their scheme aliases. From the above, it will be obvious that these specific alias values are not mandatory, although the IPTC recommends the consistent use of scheme alias values by implementers.

In a workflow where partners are exchanging news information using NewsML-G2, Knowledge Items are the most compliant method of distributing a new CV: first by creating a Scheme (see Creating a New Scheme) and next creating a Knowledge Item from a set of existing Concept Items (see Creating a new Knowledge Item for distributing as a CV) for distribution to customers and partners.

Knowledge Items do not necessarily contain all of the information that a provider possesses about any given set of concepts. This, after all, may be commercially valuable information that the provider makes available on a per-subscriber basis. For example, a lower fee might entitle the subscriber to basic information about a concept, say a person, while a higher fee might give access to full biographical details and pictures.

It is not mandatory that information about CVs be stored or distributed in the technical format of a Knowledge Item. It is sufficient, for the correct processing of a NewsML-G2 Item, only that a Scheme Alias/Code Value pair (defining the Concept URI) is unambiguous. The IPTC makes the following recommendations about CVs:

A NewsML-G2 controlled vocabulary is a set of concepts. To create a CV as a NewsML-G2 Scheme:

Add this Scheme Alias and URI to the catalog:

If using a remote catalog, change the catalog URI to reflect a new version of the catalog (so that recipients know that they should add this to their cache of catalogs) and ensure that all NewsML-G2 Items using the new Scheme refer to the new version of the remote catalog.

The above Concept Identifier resolves to the Concept URI http://cv.example.org/schemeA/concept-x

The Chapter on Knowledge Items shows an example Controlled Vocabulary expressed in XML. A Knowledge Item contains concepts from one or more Schemes. The steps to begin creating a KI are:

Each QCode Scheme Alias that is used in a NewsML-G2 Item must have a reference in the Item’s <catalog> to the Controlled Vocabulary in which it is included. A <catalog> contains the Scheme Alias and the Scheme URI:

As the CVs used by a provider are usually quite consistent across the NewsML-G2 Items they publish, the IPTC recommends that the <catalog> references are aggregated into a stand-alone file which is made available as a web resource: known as a Remote Catalog. This file is referenced by a <catalogRef> in the Item:

Note: the <catalog> element in such a stand-alone file needs an XML namespace definition:

The use of stand-alone web resources is preferable because all of the QCode mappings are shared across many NewsML-G2 Items; the local <catalog> can only be used by the single Item.

Simple management of Remote Catalogs over time is relatively straightforward: whenever a new scheme is added or the alias or the uri of any of the existing schemes are changed, a new Catalog must be published with a new URL. This URL may reflect a version number of the catalog. This is the method that the IPTC uses to maintain the NewsCodes, simply by increasing the file suffix digit as required:

Note that ALL versions of a Remote Catalog must continue to be available as a web resource, otherwise existing NewsML-G2 Items and QCodes that reference it will not be able to resolve related Scheme Aliases to Scheme URIs.

Receiving applications MUST use the catalog information contained in the NewsML-G2 document being processed. If a provider updates a catalog, this is likely to be because new schemes have been added. Using a catalog other than that indicated in the document could cause errors or unintended results.

For providers who wish to use the same basic means for managing a Catalog as are available for news content, the Catalog Item inherits the additions and changes to <catalog> and <scheme> described above. The example below shows how a Scheme Authority (in this example, the IPTC) might distribute its catalog to subscribers.

The Catalog Item shares the generic identification and processing instruction attributes associated with all NewsML-G2 items, for example:

The child properties of Catalog Item are restricted to a basic set of essentials required for Catalog management:

The catalogRef, rights information and itemMeta elements follow normal practice. Note that the Item Class is set to "catinat:catalog"; this uses the IPTC Catalog Item Nature NewsCodes, recommended Scheme Alias "catinat":

The Catalog information conveyed by the Item is wrapped in the <catalogContainer> element, which must contain one and only one <catalog>. The Catalog contains one or more <scheme> elements, as previously described:

All Scheme Aliases used in the listing below indicate IPTC NewsCodes vocabularies.

Some NewsML-G2 properties are important for the correct processing of an Item, for example the Item Class property tells a receiving application the type of content being conveyed by a NewsML-G2 Item: a processor may expect to apply some rule according to the value present in the <itemClass>, for example to route all pictures to the Picture Desk.

Other CVs may be important for correctly processing an Item, for example the presence of specific subject codes could cause an Item’s content to be routed to certain staff or departments in a workflow.

The schemes used by <itemClass> property are mandatory, and the IPTC recommends that implementers use the scheme aliases appropriate to the Item Type, for example "ninat" for News item Nature or "cinat" for Concept Item Nature. Note that the use of these specific alias values is NOT mandatory; they could already being used by a provider as aliases for other CVs.

This illustrates the flexibility of the NewsML-G2 model: consistency of scheme aliases between different providers – or even by the same provider – cannot be guaranteed, and in NewsML-G2 they do not have to be guaranteed. For this reason it would be unwise for NewsML-G2 processor implementers to assume that a given scheme alias can be "hard coded" into their applications.

However, this flexibility does not mean that these "needed for processing" CVs must be accessed every time an Item is processed. This could be an unnecessary overhead and performance burden.

Processing rules such as those described above would be based on acting in response to expected values. In the case of the News Item Nature Scheme, these values include "text, "picture", "audio" etc. The problem is not in obtaining the contents of the CV in real time, but in verifying that it is the correct CV.

For example:

How does the processor "know" that a <subject> property with a QCode containing 04000000 is an IPTC Media Topic NewsCode? The processor should not rely on the scheme alias "medtop"; it could be an alias to another CV, or the provider may use another alias:

By following the IPTC advice to retrieve all catalog information used by Items, and cache the information indefinitely, CVs can be processed without reference to external resources.

In the example, the catalog used by the Item resolves the scheme alias "ex-sc" contained in the QCode. The Item contains the line pointing to the catalog file:

The processor should have retrieved and cached the contents of the file at this URL, and would have in memory the mapping of this alias to the Scheme URI:

…​this verifies that the QCode value is from the Media Topic NewsCodes scheme. A rule "all items with a Media Topic NewsCode of 04000000 to be routed to the Business News department" is satisfied and the NewsML-G2 Item processed appropriately.

The IPTC recommends that Schemes SHOULD resolve to a Web resource, and that Scheme Authorities who disseminate news using NewsML-G2 should make their Schemes available as Knowledge Items.

Some useful and well-known CVs do not have the values of properties such as name and definition in the local language of a news provider. For example, although some IPTC NewsCodes contain concept information in several languages, the IPTC does not have the resources to provide concept details in every language being used by news providers.

Yet some NewsCodes schemes are recommended or mandatory. How can a provider use a local language version of these NewsCodes AND conform to the NewsML-G2 Specification?

Using the <sameAsScheme> property, a news provider can create its own CV, containing all of the NewsCodes it wishes to use with local language names and definitions, while making it clear to receivers that these codes identify the same concepts as the original IPTC Scheme.

For example, the IPTC Item Relation NewsCodes irel:seeAlso resolves via the Catalog to a Concept URI http://cv.iptc.org/newscodes/itemrelation/seeAlso which hosts the following information about the seeAlso concept:

A provider wishes to provide this same information in the Czech language. As a first step, it creates a new Controlled Vocabulary containing the required concepts from the original scheme with translated name and definition, for example:

The provider then creates a Catalog file, or a new version of an existing Catalog file, containing a Scheme Alias and Scheme URI for the new CV, thus:

This asserts that the ALL of the codes in the private scheme identified by the Scheme URI are semantically the "same as" the corresponding codes in the original scheme indicated by the <sameAsScheme> child element of <scheme>.

Finally, the provider adds a reference to the new Catalog file to NewsML-G2 Items that it publishes.

By using the <sameAsScheme> element to the Catalog, the provider is able apply a Same As relationship at the level of a set of Concepts. So for example, this code:

can be efficiently resolved: a processor does not have to search for a Same As relationship at the individual concept level but can map this relationship directly from the @rel value’s scheme (alias "irel") to the scheme identified by the <sameAsScheme> property.

A news provider needs to add concepts to a workflow role CV which is shared with information partners, but is maintained by some other organisation (the Scheme Authority).

The news provider would have two courses of action:

The encoding described above assumes that the character(s) to be per-cent encoded are from the US-ASCII character set (consisting of 94 printable characters plus the space).

If a code contains non-ASCII characters, for example accented characters, the Unicode encoding UTF-8 must be used, in line with normal practice.

For example, the UTF-8 encoding of the å character is a two-byte value of C3hex A5hex which would be percent-coded as %C3%A5.

Codes in controlled vocabularies which have been created with no regards to the NewsML-G2 specifications may contain spaces. As the NewsML-G2 Specification does not allow white space characters in Codes, this section recommends a workaround.

Whitespace characters in Codes - in practice, only spaces (20hex) - are replaced by a sequence of one or more unreserved characters that is reused for this purpose according to the practices of the provider; it is recommended that such a sequence is not part of the any of the codes used by the provider.

For example, if a code contains a space, the space character might be replaced by ~~. Receivers would be informed to translate this string back to a space character in order to match the QCode against a list of codes that contain spaces.

To resolve a QCode received in an Item to a Concept URI, use the following steps:

After the introduction of @uri and later the "URI Sibling" attributes, a @qcode and other QCodeType attributes (collectively termed "QCode") remained mandatory for some properties. From v2.20 this constraint is lifted and the cardinality of QCode attributes in these properties is changed from (1) to (0..1) in order to allow @uri and "URI sibling" attributes to be used on their own. The table Properties affected by changed cardinality of QCode attributes below lists the properties affected by lifting the mandatory use of QCodes. The rules are now as follows, according to the version of NewsML-G2 being used:

Literals may be used as in the following cases:

1. As an identifier for linking with an assert element inside a NewsML-G2 document. In this case the literal value could be a random one. If a literal value is used with an assert element then all instances of that literal value in that item must identify the same concept.

2. When a code from a vocabulary which is known to the provider and the recipient is used without a reference to the vocabulary. The details of the vocabulary are, in this case, communicated outside NewsML-G2. Such a contract could express that a specific vocabulary of literals is used with a specific property.

3. When importing metadata which may contain codes which have not yet been checked to be from an identified vocabulary: the code values are represented as literals until the vocabulary is identified; thereafter, a controlled identifier can be used.

The following rules govern the use of @literal:

It is permissible for a NewsML-G2 element with a Flexible Property type to have no concept identifier:

As a special case, when a <bag> child element is used with a property to create a composite concept, a concept identifier MUST NOT be used with the parent property (neither @qcode, nor @uri nor @literal). The new composite concept is created by the multiple existing concepts identified by a @qcode in each <bit> element of the <bag>: