Surface Weather Observation XML (SW OB ML) Internal SWOB ML Product User Guide V8.2 E

Surface Weather Observation XML

(SW-OB-XML)

-Client User Guide-

February 8, 2019

Version 8.2

Data

Management

Framework

Cadre de

Gestion des

Données

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 2 of 111

DMF Team    SW-OB-XML User Guide  
February 8, 2019 SW-OB-XML User Guide Page 3 of 111 
Table of Contents 
 
 
DOCUMENT INFORMATION ................................................................................................................................. 5 
1 HISTORY .................................................................................................................................................................... 5 
2 FILENAME & LOCATION ........................................................................................................................................... 11 
3 REFERENCED DOCUMENTS ....................................................................................................................................... 11 
INTRODUCTION ...................................................................................................................................................... 12 
1 OVERVIEW ............................................................................................................................................................... 12 
2 PURPOSE AND SCOPE ............................................................................................................................................... 13 
3 INTENDED AUDIENCE ............................................................................................................................................... 14 
DATA STANDARDIZATION .................................................................................................................................. 15 
1 STANDARDIZATION OF INCOMING DATA BY THE DMS ............................................................................................. 15 
2 STANDARDIZATION FOR THE SWOB ........................................................................................................................ 15 
2.1 Element Short Labels: .................................................................................................................................... 15 
2.2 Units: ............................................................................................................................................................. 16 
2.3 Codes: ............................................................................................................................................................ 17 
SW-OB-XML FORMAT AND STRUCTURE ........................................................................................................ 19 
1 OVERVIEW ............................................................................................................................................................... 19 
2 STRUCTURAL ORGANIZATION OF SWOB ................................................................................................................. 20 
3 SAMPLE SWOB ....................................................................................................................................................... 23 
4 TAXONOMY AND URI .............................................................................................................................................. 27 
5 QUALITY ASSESSMENT FLAG ................................................................................................................................... 30 
6 DATA FLAG .............................................................................................................................................................. 33 
7 MULTIPLICITY .......................................................................................................................................................... 34 
DATASETS ................................................................................................................................................................ 35 
1 OVERVIEW ............................................................................................................................................................... 35 
2 DATA ELEMENT TERMINOLOGY ............................................................................................................................... 35 
2.1 Wind ............................................................................................................................................................... 35 
2.2 Cloud Amount/Cover ..................................................................................................................................... 36 
2.3 Precipitation and Rainfall ............................................................................................................................. 37 
2.4 Station elevation ............................................................................................................................................ 37 
3 LEGACY MSC & PARTNER MANNED (WINIDE/MIDS) ........................................................................................... 38 
4 DND HWOS ............................................................................................................................................................ 40 
5 MSC & PARTNER AWOS (RA MESSAGES) .............................................................................................................. 41 
6 DND AWOS ............................................................................................................................................................ 43 
7 NAV CANADA HWOS .............................................................................................................................................. 45 
8 NAV CANADA AWOS .............................................................................................................................................. 47 
9 MSC & PARTNER SURFACE WEATHER NETWORK (CAMPBELL SCI. CA MESSAGES) ................................................. 49 
10 PANAM – MINUTELY MSC & PARTNER SURFACE WEATHER NETWORK - COMPACT ........................................ 53 
11 OPP MOORED BUOY ........................................................................................................................................... 55 
12 BC MINISTRY OF TRANSPORTATION & INFRASTRUCTURE (BC-TRAN) ............................................................. 56 
13 BC WILDFIRE MANAGEMENT BRANCH (BC-FLNR-WMB) ............................................................................... 58 
14 BC MINISTRY OF ENVIRONMENT AND CLIMATE CHANGE STRATEGY-AIR QUALITY (BC-ENV-AQMET) ......... 58 
15 BC MINISTRY OF ENVIRONMENT AND CLIMATE CHANGE STRATEGY-SNOW WEATHER (BC-ENV-SNOWWX) . 60 
APPENDICES ............................................................................................................................................................ 62 
1 GLOSSARY ............................................................................................................................................................... 62 
2 SHORT LABEL ABBREVIATIONS ............................................................................................................................... 65 
3 UNITS OF MEASURE ................................................................................................................................................. 74 

DMF Team    SW-OB-XML User Guide  
February 8, 2019 SW-OB-XML User Guide Page 4 of 111 
4 UNIT CONVERSIONS ................................................................................................................................................. 77 
5 STANDARD CODE TABLES ........................................................................................................................................ 79 
5.1 buoy_type ....................................................................................................................................................... 79 
5.2 ceiling_type .................................................................................................................................................... 81 
5.3 data_flags ...................................................................................................................................................... 81 
5.4 obscuring_phenomena ................................................................................................................................... 82 
5.5 operating_agency .......................................................................................................................................... 84 
5.6 precipitation_measurement_method .............................................................................................................. 85 
5.7 present_weather ............................................................................................................................................. 86 
5.8 rapid_pressure_change ............................................................................................................................... 103 
5.9 report_type ................................................................................................................................................... 104 
5.10 sky_condition .......................................................................................................................................... 106 
5.11 station_type ............................................................................................................................................. 108 
5.12 tendency_characteristic .......................................................................................................................... 109 
5.13 total_cloud_amount ................................................................................................................................ 109 
5.14 wind_gust_squall_indicator .................................................................................................................... 111 

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 5 of 111

1. Document Information

1

1.1 History

Author

Date

Ver.

Remarks

Thinesh Sornalingam

June 7, 2012

1.0d

Initial draft

AbdulAziz Raouf

June 8, 2012

1.1d

Added in Section 4 Datasets

Thinesh Sornalingam

June 11, 2012

1.2d

Added units of measures and conversions

Dale Boudreau

June 13, 2012

1.3d

1st overall edit of whole document

Dale Boudreau

June 19, 2012

1.4d

2nd overall edit of whole document

Dale Boudreau

June 20, 2012

1.5d

Changed max_vis to vis for RA XML sample and

mapping table

Dale Boudreau

June 21, 2012

1.6d

Minor tweaks to data set mapping tables. Dropped

vis_code and horizontal_visibility code table

Dale Boudreau

June 25, 2012

1.7d

NC-AWOS: cld_cvr_# becomes sum_cld_cvr_#,

added ‘sum’ to Appendix 6.2

Dale Boudreau

June 26, 2012

1.8d

Updated RA XML example (Sect. 4.3)

Dale Boudreau

June 28, 2012

1.9d

Changed references to External XML and E-ML to

“SW-OB-XML” or “SWOB”. Added details (table)

on incoming code source meanings in section 3.2.3.

Abdulaziz Raouf

June 29,2012

1.9d

Updated NCHWOS dataset table to include full list

of elements present section 5.5

Dale Boudreau

July 9, 2012

2.0d

Changed rpt_typ to stn_typ for NC-AWOS (code

table 002196).

Dale Boudreau

July 10, 2012

2.1d

Added clg_hgt to WinIDE and RA mapping tables.

Dale Boudreau

July 24, 2012

2.2d

Edited XML structure example. New RA sample.

Section 4.2 part D added info in “MSNG”.

Changed avg_vis_mt50-60 to just vis for NC-

AWOS.

Dale Boudreau

July 27, 2012

2.3d

Removed ceiling height (clg_hgt) from RA dataset

Dale Boudreau

July 30, 2012

2.4d

For NC-HWOS dataset, added prsnt_wx_# and

changed cld_amt_code to cld_amt_code_#. Also

did global search from std_code_source and

replaced with std_code_src.

Tahreem Ali

July 30, 2012

2.5d

Removed indexed temperatures for CA

(air_temp_#, min_air_temp_pst1hr_#,

max_air_temp_pst1hr_#) and duplicate row for

pcpn_amt_pst1hr

Abdulaziz Raouf

July 30, 2012

2.6d

Changed label name and description for NCAWOS

element wnd_dir_10m_mt50-60_max_spd to

wnd_dir_10m_pst1hr_max_spd

Dale Boudreau

August 9, 2012

2.7d

Added std code 88 to report_type code table (from

1

For more details on additions and changes to the actual SWOB-ML product, please refer to the Release Notes cited in

Section 1.3

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 6 of 111

incoming of 126 from BUFR58 descriptor 001196).

Added icao_stn_id to WinIDE and RA.

Dale Boudreau

August 29, 2012

2.8d

Added word “snow” to descriptions of codes 83, 84

for present_weather code table.

Dale Boudreau

August 31, 2012

3.0

Final Version

Dale Boudreau

October 25, 2012

3.2

Updated description for codes 46, 47 in table

total_cloud_amount

Dale Boudreau

December 21, 2012

4.0

Multiple changes made for the DMS 2.3.12.1

release deployed on Data Depot January 15, 2013.

Changes are as follows (see Release Notes for more

detail):

WinIDE

tot_cld_amt – changed units from 1/10 to %

tot_cld_opcty – changed units from 1/10 to %

CA Added the following elements:

 avg_cum_pcpn_gag_wt_fltrd_55-60

 snow_dpth_#

 logr_panl_temp

 max_batry_volt_pst1hr

 min_batry_volt_pst1hr

 hdr_fwd_pwr

 hdr_refltd_pwr

 hdr_suply_volt

 hdr_oscil_drft

Dale Boudreau

February 14, 2013

5.0

Multiple changes made for the DMS 2.4.0 release

deployed on Data Depot early March, 2013.

Changes are as follows (see Release Notes for more

detail):

RA

Element cld_amt_code_# wasn’t being reported

for clear sky because of a change to the

decoded code value from ‘CLR BLO 100’ to

‘CLR BLO’. The code substitution XML was

updated to accommodate this code change and

resolve this issue.

Added the following elements:

 max_air_temp_pst6hrs

 min_air_temp_pst6hrs

 max_air_temp_pst24hrs

 min_air_temp_pst24hrs

CA Added the following elements:

 pcpn_amt_pst3hrs

 pcpn_amt_pst6hrs

 pcpn_amt_pst24hrs

 max_air_temp_pst6hrs

 min_air_temp_pst6hrs

 max_air_temp_pst24hrs

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 7 of 111

 min_air_temp_pst24hrs

‘air_temp_#’ added User Guide table 5.7 as

well as an explanation of how it may appear

‘snw_dpth_#’ added User Guide table 5.7 as

well as an explanation of how it may appear

NC-AWOS

Added element cld_amt_code_#

Fixed label name for the following elements by

appending an ‘s’ at the end (i.e. hr to hrs):

 pcpn_amt_pst3hr

 pcpn_amt_pst6hr

 pcpn_amt_pst24hr

 max_air_temp_pst6hr

 min_air_temp_pst6hr

 max_air_temp_pst24hr

 min_air_temp_pst24hr

Changed ‘wmo_id’ to ‘wmo_synop_id’, and

changed ‘max_10m_wnd_gst_ spd_mt50-60’

to ‘max_wnd_gst_spd_10m_mt50-60

NC-HWOS

Added elements:

 cor

 pcpn_amt_pst6hrs

WinIDE

Changed ‘wmo_id’ to ‘wmo_synop_id’

Dale Boudreau

February 28, 2013

5.1

Modified code descriptions for

obscuring_phenomena (table 6.5.2) codes 28 and 34

to deal with specific meanings for NC-HWOS.

Dale Boudreau

March 18, 2013

5.2

DMS Release 2.4.1: Added the new element

max_vis_pst1hr to CA dataset. More modifications

to code descriptions for obscuring_phenomena

(table 6.5.2) codes 5, 15, 29, and 46 to deal with

specific meanings for NC-HWOS.

Dale Boudreau

May 7, 2013

5.3

DMS Release 2.4.2: Added cld_amt_code_# to the

NC-AWOS table (Sect. 5.6)

Dale Boudreau

July 15, 2013

6.0

DMS Release 2.4.2: Added NC-AWOS to

“station_type” code table (6.5.8) as code 12, and

added codes 17, 18. Removed “product_status”

code table from the appendix as it was not

referenced by any SWOB elements in any of the

networks. New codes (86-98) added to bottom of

“report_type” code table (6.5.6). Updated code

descriptions in tables 6.5.7, 6.5.8, and 6.5.10.

element additions/modifications to the various

networks:

WinIDE

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 8 of 111

Added the following element:

 clg_hgt

NC-HWOS

Added 3 additional identification elements:

 clim_id

 msc_id

 rtp_typ

Learned that the following elements have an

incorrect label since they should be for a 24

hour period. Therefore changed the labels of

the following elements to reflect that this is a

24 hour peak wind speed, not a 1-hour peak as

the SWOB label originally indicated:

Changed

 max_pk_wnd_spd_10m_pst1hr to

max_pk_wnd_spd_10m_pst24hrs

 wnd_dir_10m_pst1hr_pk_spd to

wnd_dir_10m_pst24hrs_pk_spd

NC-AWOS

Added the following identification element:

 rtp_typ

CA

Removed duplicated element rows from Table 5.7.

Added the following element:

 avg_wnd_spd_pcpn_gag_mt58-60

Dale Boudreau

August 13, 2014

6.1

CA

Added additional elements:

 avg_globl_solr_radn_pst1hr

 tot_globl_solr_radn_pst1hr

 avg_wnd_spd_pcpn_gag_mt50-60

 data_avail (future release)

Removed the following element:

 stn_id (last 4 digits of wmo_synop_id)

 avg_wnd_spd_pcpn_gag_mt58-60

NC-AWOS

Added element stn_elev

Removed the following elements (always missing):

 pcpn_amt_pst3hrs

 pcpn_amt_pst24hrs

GENERAL

 Removed duplicate element rows from

Table 5.7

 Added code table for

wind_gust_squall_indicator

 Changed descriptions for codes 13, 10 in

station_type code table

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 9 of 111

Tahreem Ali /

Dale Boudreau

March 24, 2014

7.0

Added the following new datasets:

 PanAM – MSC & Partner (CA messages)

 PanAM – MSC & Partner (Compact

messages)

 PanAM – MSC & Partner (ATMOS

messages)

Updated the following code tables:

 station_type

 report_type

 present_weather

Added the following code table:

 buoy_type

Reflected data changes:

 removed T-12 from SWOBs

 removed pcpn_amt_pst3hrs and

pcpn_amt_pst24hrs from NC-AWOS

SWOBs (not observed)

 Changed precision of Pressure elements to

1 decimal place

 Changed precision of snow depth

elements to 0 decimal places

 Changed precision of all temperature

elements to 1 decimal place

 Changed precision of relative humidity to

0 decimal places

 Changed precision of all

precipitation/rainfall elements to 1 decimal

place

Many editorial changes:

 Many minor editorial changes

 Added text to sections 2.1, 2.3, 4.4, 5.1,

5.2.1,

 Added new sections: 5.2.3, 5.2.4,

 Modified Qa flag descriptions in section

4.5

 Added new data set descriptions and

element tables (sections 5.9 – 5.22

 Added additional glossary items

 Added additional short label descriptions

 Added additional units and conversions

Dale Boudreau

September 15, 2015

7.1

Added new units to section 6.3

Tahreem Ali

August 11, 2016

7.1

Added the following new datasets:

 DND AWOS

 DND HWOS

Updated short labels for the following datasets:

 CA

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 10 of 111

Removed PanAm datasets except for CA-Compact

Updated taxonomy sections to account for new

datasets (DND)

Updated Qa section to remove the qa summary = 20

rule

Updated CA table to account for new labels

(pstXmts)

Added section on multiple sensors (Multiplicities)

Tahreem Ali

July 27, 2018

8.0

Updated all section with Generic SWOB

taxonomies, output, rules

Updated Sample SWOB section with new example

Added generic swob labels to remaining datasets

(NavCan, RA, WinIDE, CA Minutely)

Tahreem Ali

November 6, 2018

8.1

Update document to include new dataset: OPP

Marine Buoy

Added section on data_flag

Updated code tables: total_cloud_amount,

report_type, buoy_type

Updated units: 0.01in

Updated WinIDE dataset (Section 5.3) with new

elements: max_pk_wnd_spd_10m_pst24hrs,

wnd_dir_10m_pst24hrs_pk_spd,

max_pk_wnd_tm_pst24hrs, snw_dpth,

pcpn_amt_pst6hrs, pcpn_amt_pst24hrs,

avg_wnd_dir_10m_pst10mts,

avg_wnd_spd_10m_pst10mts

Tahreem Ali

February 8, 2019

8.2

Added BC datasets:

 BC Forestry

 BC Tran

 BC SnowWx

 BC AQMet

Added code table:

precipitation_measurement_method

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 11 of 111

1.2 Filename & Location

Approved (public) Version:

http://dd.weatheroffice.gc.ca/observations/doc/SWOB-ML_Product_User_Guide_v8.2_e.pdf

Current Draft (internal version):

http://ecollab.ncr.int.ec.gc.ca/org/1275692/wem/MS_lib/DMS_SW-OB-

ML_Product_User_Guide_v8.2_Public_e_Final.doc

1.3 Referenced Documents

Document

Author

Version

DMF External Met-ML Specification

http://ecollab.ncr.int.ec.gc.ca/org/1275692/wem/MS_lib/DMFExterna

lMet-MLspecification.doc

Dale Boudreau,

Thinesh Sornalingam,

Abdulaziz Raouf

1.1

Taxonomy Documentation

Ioanne Carlo Bugash

0.4.1d

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 12 of 111

2. Introduction

2.1 Overview

The creation of a product for surface weather observations has been undertaken by the Data

Management Initiative (DMI) project to provide the Metrological Service of Canada (MSC) and

external clients with a concise, user-friendly, easy to read product containing typical hourly surface

weather data from MSC and partner atmospheric monitoring networks.

These products will be generated by applications within the Data Management System (DMS). The

DMS is collection of a real-time data acquisition, decoding, standardization, quality assessment (Qa)

and product generation components for observation, forecast, and warning data. It is to become

MSC's primary disseminator of meteorological data to internal and external clients.

There are multiple phases or processing stages within DMS that incoming raw data passes through.

Each phase has an associated XML output. In the raw phase, the incoming messages are in their

original form (e.g. ASCII, BUFR, etc.). Then they are converted into a “parsed” product (typically an

XML). In the parsed phase, the data has not yet been standardized or fully "decoded". The parsed

phase prepares the data to be decoded. The parsed phase is optional in the sense that not all data must

pass through it. In some cases, the data proceeds straight to the decode phase from the raw phase. In

the decode phase, the data is standardized (i.e. mapped to a standard classification, element definition,

and name). The data emerges as a decoded XML product following the completion of its passage

through the decode phase. The decoded XML is a standards-based format, which is then put through

any number of enhanced, quality assessment and/or product generation (PG) phases to add further

value and meet client requirements. The Surface Weather Observation XML product (SW-OB-XML)

is the result of the PG phase. The SW-OB-XML is in a basic XML format patterned after the DMS

internal Met-ML format. The SW-OB-XML will be referred to by the condensed acronym ‘SWOB’

hereafter in this document. The diagram at the end of section 2.1 captures this flow for selected

networks.

As opposed to the more comprehensive DMS decoded Met-ML products, the SWOB is a simplified

XML product that focuses on core elemental data without the clutter and complexity of auxiliary

content such as non-critical metadata, and detailed Qa results. To the extent possible, this information

has been captured in a short element label and an optional qualifier to indicate a summary of any Qa.

Although the SWOB XML is intended for machine-to machine data transmission, the format and

clarity of this self-descriptive format is easily human-readable without any specific training or

knowledge of markup languages. However, for clients needing to interact with and display data over

long intervals or comparing observations from multiple stations geospatially, the use tools such as

XML parsers, graphical displays and GIS is advisable.

There are many different streams of the SWOB product, each containing data from the following

networks (a short reference name is to the right of the arrow):

1. Legacy MSC & Partner Manned aviation weather stations using the WinIDE or MIDS interface  WinIDE

2. DND HWOS  DND HWOS

3. Legacy MSC & Partner AWOS aviation weather stations  RA

4. DND AWOS  DND AWOS

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 13 of 111

5. Nav Canada HWOS aviation weather stations  NC-HWOS

6. Nav Canada AWOS aviation weather stations  NC-AWOS

7. MSC & Partner public surface weather stations using Campbell Scientific data loggers  CA

7.1. Minutely data sets supporting the PanAm Games:

- MSC Compact stations  COMPACT-minutely

8. OPP Moored buoy weather stations  OPP

9. BC Ministry of Transportation weather stations  BC-TRAN

10. British Columbia Wildfire Management Branch (i.e. BC Forestry) weather stations  BC-FLNR-WMB

11. BC Ministry of Envrionment Air Quality Meteorological weather stations  BC-ENV-AQMet

12. BC Ministry of Envrionment Snow weather stations  BC-ENV-SnowWx

Each dataset product can be uniquely identified via its URI (will be discussed in more detail in Section

4.4).

During the production of a SWOB, the following tasks are carried out:

 Incoming DMS element packages are assigned an abbreviated label

 Unit conversion from incoming units to standard units, and if necessary, rounded to a given

precision to trim insignificant digits resulting from some unit conversions.

 If the incoming element is a numeric code or a text value from a list of controlled vocabulary

(so in effect a code), then code substitution is performed to map to a DMS standard code.

 An optional quality assessment (Qa) summary flag (using incoming ‘native’ and DMS quality

assessments whenever available) may be attached to applicable elements.

2.2 Purpose and Scope

One of the main purposes of the SW-OB-XML is to offer a replacement for much of the content

traditionally found in the legacy SA (surface analysis) product still being used within MSC, although

officially it was to have been retired years ago. Although the SWOB will not capture all of the SA’s

content, it will include the most sought-after hourly surface weather observation elements and any

elements pertaining to longer time frames. Some of the more subtle and obscure elements contained in

the SA will be excluded from the initial offering of the SWOB. Clients requiring specific aviation,

synoptic, or marine data may be better served acquiring the METAR or SYNOP products.

Meanwhile, additional elements not present in SA will also be available in the SWOB.

As mentioned above, there are many streams of the SWOB product, each pertaining to a specific

network. Each product will be accessible via its own URI within the DMS, or possibly a file system

containing the XML files referenced using a filename (e.g. CMC’s DataDepot).

The observation elements included in the majority of SWOBs are from these basic observation groups,

although some data sets have other groups such as radiation, wave, etc:

 present weather

 sky condition

 visibility

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 14 of 111

 pressure

 wind

 temperature

 humidity

 precipitation

The format of this product is XML. It will be generated operationally by MSC's Data Management

System (DMS) at CMC in Montreal. It will be produced in real-time from the incoming raw data of

each dataset. It will be encoded in a standards based XML, which conforms to global meteorological

observation conventions (discussed in detail in Section 4).

In addition to the elemental data from the groups indicated above, the SWOB may also contain

optional quality assessment (Qa) information in the form of a summary quality flag attached to each

element, whenever available. This flag's value is computed by considering any incoming Qa

performed on the element at the source (so-called “Native QC”), any Qa conducted in-house by the

DMS, or a combination of the two. The absence of a Qa summary flag on an element in the SWOB

indicates the quality is unknown.

The SWOB is a very condensed and convenient product of hourly observational data, as opposed to

the fuller and more comprehensive decoded XMLs being generated by DMS for the listed datasets.

Those clients requiring full element definitions, full quality assessments, etc., should gain access to the

DMS decoded, or decoded_enchanced, products of each dataset.

2.3 Intended Audience

Any clients interested in hourly surface weather data will find the SWOB product attractive due to its

content, simplicity and compactness. Most clients currently using the legacy SA format should also

find this product a suitable replacement given it has the majority of the elements in a clear format and

may have additional content which could never be encoded using the SA format. Clients who would

like quick and easy access to MSC data from the DMS, will also generally benefit from the SW-OB-

XML product, especially given that data viewing and access tools are under development in the DMS.

This version of the User Guide is intended for internal users within Environment Canada, as some

data sets described herein are not available on the publicly accessible CMC DD (DataDepot) site, but

rather only on the DDi (internal). The non-public SWOBs are either third party data sets that have as

part of their data sharing agreements restrictions on data usage or distribution, or are Beta data sets

that have not completed user acceptance testing.

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 15 of 111

3. Data Standardization

3.1 Standardization of incoming data by the DMS

The DMS decodes and processes data from many networks. The same element may have a variety of

different names across the input data sets. For example, air temperature may be called dry-bulb

temperature, temperature, temp, ambient temperature, etc. The DMS standardizes elements names so

they may be more easily inter-compared, quality assessed, and extracted. Furthermore, the elements

may have optional qualifiers assigned to them to covey important metadata. Below is an example of

how a particular wind speed from MSC networks is elementalized by the DMS and stored in XML

format:

- <element group="wind" name="wind_speed" orig-name="011012" uom="m/s" value="0.0">

</element>

The above “standard element package” would be applied consistently across the networks and look as

the example above for cases where the incoming element is an average wind speed over the last 2

minutes of the hour and a height of 10 metres.

To avoid conversion and rounding errors, all incoming codes and units are left as-is. Such operations

are typically reserved for product generators, datamarts or display tools, just as the data leaves the

DMS for client use.

3.2 Standardization for the SWOB

The SWOB product generator needs to ingest data from many networks to make one product where

the element names, units and code tables are consistent. To make the SWOB as concise as possible

and remove any remaining element description variations in the DMS decoded output, a “short label”

was devised to encapsulate all the critical element-defining metadata into one phrase. Furthermore, all

the various unit and code variations for identical elements across the networks were handled by

converting to DMS “standard units” and “standard codes”.

3.2.1 Element Short Labels:

Critical element-defining metadata such as data type, element name, statistical significance, time

period displacement, time period duration, height/depth, index, etc., have been used to devise a “short

label” for each DMS element package. For example, the element package shown in Section 3.1 would

have the short label of “avg_wnd_spd_10m_pst2mts”. For the element name portion of the short label,

the name as it appears in the original DMS decoded element has been abbreviated using the

abbreviation glossary in Appendix 6.2. The length of these labels has been kept to a minimum and

special characters and spaces have been avoided so that the labels may also be used as column names

in database tables and meet the most stringent of requirements.

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 16 of 111

3.2.2 Units:

All element packages have a standard element assigned by the DMS based on the data class (e.g.

velocity, temperature, azimuth, pressure, etc.), although some variations exist for certain elements

within a class. For example, most pressure-related elements will have a DMS standard unit of hPa, but

in the case of altimeter, the standard unit is inHg since it is used by a specific client community and in

practically all cases measured and used in that unit. Examples of typical standard units for some

classes are:

 precipitation amount = mm

 wind speed = km/h

 pressure = hPa

 visibility = km

 height = m

 temperature = °C

The conversion to DMS standard units is only done at the last possible moment, typically when data

leaves the DMS via product generators, like the one that creates the SWOB, or during the population

of custom datamarts and display tools where client requirements need to be satisfied. The DMS

standard units will meet the majority of client needs, but inevitably some clients will have different

preferences and will need to do some conversions. To assist in this, Appendix 6.4 has a list of unit

conversions so clients can see what was used to convert incoming units to DMS standard units for a

given element, or to apply client-side conversions. The data set tables in Section 5 show the incoming

uom (unit of measure) and the standard units they were converted to. As well, the rounding precision

is given where the value represents the number of digits after the decimal (e.g. 2 would represent

0.01). A value of 0 represents integer values. The intent is to remove insignificant digits as a result of

a unit conversion. To avoid giving the appearance of extra precision that was not intended, the

following rules were applied:

 In the element mappings the following convention is used to represent precision:

0 = whole number

1 = one decimal digit = 0.1

2 = two decimal digits = 0.01

3 = three decimal digit = 0.001

…

 Mathematical rounding precision only applies to numerical values and for elements a precision is

indicated in the Precision column (see Section 5). For instance if the decoded value = 5.67 and the

Rounding Precision is 1, then the SWOB value = 5.7. If on the other hand the decoded value has a

lower precision than what is specified for that element in the mappings, then preserve the decoded

value as-is. For example, if a decoded value or unit conversion = 5, Precision = 1 (i.e. 0.1), then

SWOB value = 5.

The following page has some examples of rounding to a specified precision:

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 17 of 111

Decoded Value

Rounding Precision

External Element Value

12.3

0 (to the nearest whole number)

12

23.3

2 (two decimal digits)

23.3 (decoded value precision is less than

requested precision, so preserve decoded

value as-is)

45.12346666666

6 (six decimal digits)

45.123467

23.549

1 (one decimal digit)

23.5 (given the requested precision is 1

decimal digit, one needs to look at the digit

immediately following it for rounding (4). As

a general rule: if precision = x, then always

look at x * 10-1 to carry out rounding)

17.6

0

18

3.2.3 Codes:

As with units, the incoming code values are preserved as-is after DMS decoding, with the code table

source and type (i.e. name) cited. Prior to the creation of the SWOB, incoming code tables are left in

their original form. The source of these tables can be any of the following:

Incoming

code-src

Description

Documentation Source

Sample code-source and

code-type

wmo_bufr

A WMO code table for data encoded

in BUFR format

WMO Pub. No. 306 =- Manual on Codes Part B –

Binary Codes:

http://www.wmo.int/pages/prog/www/WMOCodes/WM

O306_vI2/VolumeI.2.html

wmo_bufr 020003

local_bufr

A local Canadian code table for

MSC data encoded in BUFR format.

Defined by CMC

CMC:

ftp://depot.cmc.ec.gc.ca/ftp/cmoi/bufr/english/tabloc_b

ufr_e

local_bufr 020197

wmo_tac

A WMO code table for data encoded

in Traditional Alphanumeric Code

forms (TAC) such as SYNOP

WMO Pub. No. 306 =- Manual on Codes Part A –

Alphanumeric Codes

http://www.wmo.int/pages/prog/www/WMOCodes/WM

O306_vI1/VolumeI.1.html

wmo_tac 000500

local_tac

A local code table defind by the

DMS for incoming data encoded in

Traditional Alphanumeric Code

forms (TAC)

DMS code tables and encode/decode specification

documents

local_tac 008197

Various

sources

A local code table defind by the

DMS for incoming data encoded in

ASCII formats. The code source

may be the name of the network,

product, message, etc.

DMS code tables and encode/decode specification

documents

ra present_weather

rwin essPrecipSituation

metar visibility

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 18 of 111

However, a “standard” code table is also associated with each of these incoming code tables for a

given element. The master list of code tables that the DMS maintains has cross referenced similar

code tables for a given entity to a DMS standard table, which is in effect a superset of all the similar

code tables for that entity. This allows for products or clients to use one standard code value for an

element to map to their preferred codes, expressions or interpretation rather than having to map too

many different tables for a given element across multiple networks. For example, present weather is

reported by many networks, but most use different code tables or even text strings (note, in the DMS

text strings that are controlled vocabulary are also treated as if they were codes).

Example 1. Present weather arriving to the DMS in different code tables for each network is mapped

to a code value in a single DMS standard code table. Below are some examples:

Network

Observation

Incoming

code-src

Incoming

code-type

Incoming

value

SWOB code-

src

SWOB code-

type

Std code

value

WinIDE

Manned Observation: Light

rain (not freezing,

continuous)

local_bufr

020210

11

std_code_src

present_weather

65

NC-

HWOS

Manned Observation: Light

rain (not freezing,

continuous)

wmo_bufr

020019

-RA

std_code_src

present_weather

65

RA

Automated Station

Observation: Light rain

ra

present_weather

R-

std_code_src

present_weather

364

RWIN

Automated Station

Observation: Light rain

rwin

WMO4680

61

std_code_src

present_weather

364

Example 2. Cloud type and obscuring phenomena arriving to the DMS in a different code tables for

each network is mapped to a code value in a single DMS standard code table. Below are some

examples:

Network

Observation

Incoming

code-src

Incoming

code-type

Incoming

value

SWOB code-src

SWOB code-type

Std

code

value

WinIDE

Altocumulus

local_bufr

020197

0

std_code_src

obscuring_phenomena

0

NC-HWOS

Altocumulus

wmo_bufr

020012

3

std_code_src

obscuring_phenomena

0

ASCII

SYNOP

FM-12

Altocumulus

wmo_tac

000500

3

std_code_src

obscuring_phenomena

0

To see the meaning of the standard code table vales for coded SWOB elements, please refer to

Appendix 6.5. With this information clients can map the SWOB standard code values to their

preferred expression or code using a single mapping table, rather than one for each network.

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 19 of 111

4. SW-OB-XML Format and Structure

4.1 Overview

As with all DMS decoded XML products, the SWOB conforms to global standards such as OGC's

Observation and Measurement schema and GML. Being compliant with such standards enhances the

interoperability of the format and also offers a common look and feel among similar products.

The two standards employed in the SWOB are the following:

Open Geospatial Consortium's Observations and Measurements Encoding Standard

(O&M) defines an abstract model and an XML schema encoding for observations and it

provides support for common sampling strategies. O&M also provides a general framework

for systems that deal in technical measurements in science and engineering. This is one of the

OGC Sensor Web Enablement (SWE) suite of standards.

Additional information of O&M can be obtained from here:

http://www.opengeospatial.org/standards/om

Open Geospatial Consortium's Geography Markup Language Encoding Standard

(GML) The Geography Markup Language (GML) is an XML grammar for expressing

geographical features. GML serves as a modeling language for geographic systems as well as

an open interchange format for geographic transactions on the Internet. As with most XML

based grammars, there are two parts to the grammar – the schema that describes the document

and the instance document that contains the actual data.

A GML document is described using a GML Schema. This allows users and developers to

describe generic geographic data sets that contain points, lines and polygons.

Additional information of GML can be obtained from here:

http://www.opengeospatial.org/standards/gml

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 20 of 111

4.2 Structural Organization of SWOB

<om:ObservationCollection>

<om:member>

<om:Observation>

<om:metadata>

<set>

<id/>

</general>

<identification-elements>

<element name=" " uom="" value=""/>*

<element name=" " uom="code" code-src="" code-type="" value=""/>*

</identification-elements>

</set>

</om:metadata>

<om:samplingTime>

</gml:TimeInstant>

</om:samplingTime>

<om:resultTime>

</gml:TimeInstant>

</om:resultTime>

<om:procedure>

<om:observedProperty>

<om:featureOfInterest>

</gml:FeatureCollection>

</om:featureOfInterest>

<om:result>

<orig-header/>

<orig-msg/>

<element name=" " uom="" value=""/>*

</element>*

<element name=" " uom="code" code-src="" code-type="" value=""/>*

</element>*

</elements>

</om:result>

</om:Observation>

</om:member>

</om:ObservationCollection>

*Note: zero to many lines in this format can be present

A

B

C

D

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 21 of 111

Section A:

This section provides metadata around the DMS component that produces this XML.

 <author> is the component's name

 <dataset> contains the full taxonomy of this dataset (discussed below in detail)

 <phase> the DMS phase at which point this XML was generated

 <id> is the full URI (uniform resource indicator) of this instance of the XML

 <parent> is the full URI of the input file that led to the generation of this XML instance. In

the case of a product generator, which produces the SWOB, the input is either the decoded

or decoded_enchanced XML.

Section B:

This section of the SWOB contains metadata elements about the observation. For instance, one

could find the time of observation, the reporting station identifier (e.g. MSC ID, ICAO ID,

WMO Synoptic ID, etc), the station's latitude, longitude and elevation, correction level of the

observation, etc. The elements in this section are of the form <element name=" " uom=""

value=""/>, where

 name is an abbreviated label (less than 30 characters) assigned to each element

definition

 uom is the unit of measure

 value is the value of the element.

In the case when uom="code", then two additional attributes will be included, these are:

 code-src is the authoritative source of which this code table originates

 code-type is the type or name of a given code table available for a given source

The resulting element would look like this:

<element name="" uom="code" code-src="" code-type="" value=""/>. Such elements contain

coded values.

There can be any number of identification elements for a given observation, depending on the

dataset.

Section C:

This section contains additional metadata about the observation.

 <om:samplingTime> is the full date time of this observation encoded within a GML

element

 <om:resultTime> is the full date-time when the DMS product generator produced this

instance of the SWOB encoded within a GML element

 <om:featureOfInterest> is the latitude and longitude of the station that reported this

observation encoded within a GML element.

Section D:

This section of the SWOB is the body of the observation, where one would expect to find the

observational elements. Fundamentally an element is a single unit of observation. For

instance, air temperature, relative humidity, wind speed, wind direction, visibility, etc. are all

examples of a singular observed phenomenon, which is encoded as an element in the SWOB.

Elements in this section are of the form:

<element name=" " uom="" value=""/> where

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 22 of 111

 name is an abbreviated label (less than 30 characters) assigned to each element

definition

 uom is the unit of measure

 value is the value of the element

In the case when uom="code", then two additional attributes will be included, these are:

 code-src is the authoritative source of which this code table originates

 code-type is the type or name of a given code table available for a given source

The resulting element would look like this:

<element name="" uom="code" code-src="" code-type="" value=""/>. Such elements contain

coded integer or values or a text value from a list of controlled vocabulary (so in effect a code).

The incoming element that is encoded here, comes in with a native code source and type (an

example for a present weather code source and type may be: wmo_bufr, 020003, respectively).

During the production of the SWOB, a “standard code value” is substituted for the incoming

codes. The exact code substitutions used for each dataset of the SWOB product will be

discussed in Section 5 of this document.

For elements which are reported in the raw observation that have an empty or illegal value, the

value/code will be designated as MSNG to denote “missing”.

Finally, if Qa information is available for the element, then a qualifier will be tagged onto the

element to provide an over-all summary of the quality assessments. In this case, the element

will look like this:

</element>

or

</element>

See Section 4.5 for details on the creation of the Qa qualifier and the meaning of the code value

(i.e. Qa flags).

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 23 of 111

4.3 Sample SWOB

Here is a full sample output of a SWOB instance for the RA (MSC AWOS) dataset:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<om:ObservationCollection xmlns:om="http://www.opengis.net/om/1.0"

xmlns="http://dms.ec.gc.ca/schema/point-observation/2.0"

xmlns:gml="http://www.opengis.net/gml" xmlns:xlink="http://www.w3.org/1999/xlink"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<om:member>

<om:Observation>

<om:metadata>

<set>

<dataset name="msc/observation/atmospheric/surface_weather/ra-1.1-ascii"

/>

<id xlink:href="/data/msc/observation/atmospheric/surface_weather/ra-1.1-

ascii/product_generic_swob-xml-2.0/201207181400/7018573/yoy/orig/data_60" />

<parent xlink:href="/data/msc/observation/atmospheric/surface_weather/ra-

1.1-ascii/decoded_enhanced-xml-2.0/201207181400/7018573/yoy/orig/data_60" />

</general>

<identification-elements>

<element code-src="std_code_src" code-type="report_type" name="rpt_typ"

uom="code" value="0" />

<element code-src="std_code_src" code-type="station_type" name="stn_typ"

uom="code" value="4" />

</identification-elements>

</set>

</om:metadata>

<om:samplingTime>

<gml:TimeInstant>

<gml:timePosition>2012-07-18T14:00:00.000Z</gml:timePosition>

</gml:TimeInstant>

<om:samplingTime>

<om:resultTime>

<gml:TimeInstant>

<gml:timePosition>2012-07-18T14:01:04.657Z</gml:timePosition>

</gml:TimeInstant>

</om:resultTime>

<om:procedure xlink:href="/data/msc/metadata/station/surface_weather/metadata_instance-2.0-

xml/product-jicc_xml-2.0/201207181143/yoy" />

<om:observedProperty gml:remoteSchema="/schema/point-observation/2.0.xsd" />

<om:featureOfInterest>

<gml:FeatureCollection>

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 24 of 111

<gml:location>

<gml:Point>

<gml:pos>46.9 71.5</gml:pos>

</gml:Point>

</gml:location>

</gml:FeatureCollection>

<om:featureOfInterest>

<om:result>

<element code-src="std_code_src" code-type="total_cloud_amount"

name="cld_amt_code_1" uom="code" value="2">

</element>

</element>

<element code-src="std_code_src" code-type="total_cloud_amount"

name="cld_amt_code_2" uom="code" value="2">

</element>

</element>

</element>

<element code-src="std_code_src" code-type="present_weather" name="prsnt_wx"

uom="code" value="409">

</element>

</element>

</element>

</element>

</element>

</element>

</element>

</element>

</element>

</element>

DMF Team SW-OB-XML User Guide

February 8, 2019 SW-OB-XML User Guide Page 25 of 111

</element>

</element>

</element>

</element>

</element>

</element>

</element>

</element>

</element>

</element>