AQUARIUS Time Series Developer Guide: Field Data Plug In Framework AQUARIUSDeveloper Guide Plugin

User Manual: Pdf

Open the PDF directly: View PDF .
Page Count: 19

Download
Open PDF In Browser	View PDF

AQUARIUS Time-Series Developer Guide: Field Data
Plug-In Framework
2017.4
AQUARIUS
Time-Series Developer
Guide: Field Data Plug-In Framework |0
December
2017

Table of Contents
Table of Contents ................................................................................................................................................1
About this Guide .................................................................................................................................................2
Overview .............................................................................................................................................................3
Control Flow .............................................................................................................................................................. 4
Data Model .........................................................................................................................................................6
Setting up a Development Environment .............................................................................................................. 9
Writing a Field Data Plug-In ............................................................................................................................... 10
Tips .......................................................................................................................................................................... 10
Parsing the Field Data File ................................................................................................................................. 12
Logging .............................................................................................................................................................. 13
Installing and Registering your Plug-In ............................................................................................................... 14
Tips .......................................................................................................................................................................... 15
Unregistering your Plug-In ................................................................................................................................. 16
Debugging your Plug-In ..................................................................................................................................... 17
Tips .......................................................................................................................................................................... 18

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |1

About this Guide
A Field Data Plug-In Framework has been added to AQUARIUS Time-Series.
A Field Data Plug-In Framework SDK is now included with the AQUARIUS Time-Series Server installation
package. Integrators can use this SDK to extend AQUARIUS Time-Series by creating custom field data
plug-ins to parse custom field data files.
After installing the Plug-In on the AQUARIUS Time-Series Server and registering it with the Field Data
Plug-In Framework, users can import their proprietary/custom field data files to the AQUARIUS TimeSeries system using the available drag-and-drop upload method from Springboard, Location Manager,
and the Field Data Editor, or by using the AQUARIUS Acquisition API.
This guide provides guidance for developing custom field data plug-ins for AQUARIUS Time-Series.
If you would like to see more examples of field data plug-ins, please visit our Examples Repository on
GitHub:
•

https://github.com/AquaticInformatics/Examples/tree/master/TimeSeries/PublicApis/FieldDataP
lugins

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |2

Overview
Figure 1 illustrates how the Field Data Plug-In Framework fits into the overall architecture of AQUARIUS
Time-Series.
The Framework is called when:
•

A user utilizes the available drag-and-drop upload method from Springboard, Location Manager,
or the Field Data Editor;
OR

•

A program calls the PostFieldDataAttachment operation from the AQUARIUS Acquisition API
(http:///AQUARIUS/Acquisition/v2/location/{LocationUniqueId}/visits/uploads/plug
-ins).

Figure 1: Architectural Overview

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |3

Field data plug-ins are dynamically loaded at runtime and run within its own AppDomain. Each plug-in is
sand-boxed, running locally and protected from other areas of the system.

Control Flow
Figure 2 illustrates the control flow after a user uploads a field data file to the Framework.

Figure 2: Control Flow Diagram
Plug-ins are registered with the Framework and given a priority. The priority determines the order that
the Framework runs the plug-ins to parse the field data file. When a field data file is submitted to the
Framework, the file is passed to each parser, in ascending priority order. Each parser returns one of the
following statuses:
•

CannotParse. The plug-in could not parse the file. The Framework will pass the file to the next
highest priority plug-in. If the file is submitted through the drag-and-drop interface in
Springboard, Location Manager, or the Field Data Editor, and there are no more registered plugins, then the Framework passes the built-in plug-ins (e.g., SWAMI, FlowTracker *.DIS, AquaCalc)
to parse. If the file is submitted through the Acquisition API, an HTTP error is returned.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |4

•

•

SuccessfullyParsedButDataInvalid. The plug-in parsed the file but failed to process the data.
The framework will exit with an error. The error will be logged to a log file
(FieldDataPluginFramework.log) and a “Failed” status will be reported in Springboard or an
HTTP error is returned in the Acquisition API.
SuccessfullyParsedAndDataValid. The plug-in parsed the file and processed the data without
error. The framework will try to save the field data to the database and the save status will be
reported below the drag-and-drop area in Springboard, Location Manager or Field Data Editor
(FDE).

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |5

Data Model
The Framework SDK provides a number of data objects. Your plug-in is responsible for mapping the
contents of a field data file to one or more of these data objects. The Framework uses these data objects
to save field data to the AQUARIUS Time-Series Server.
Figure 3 illustrates the data objects and their relationships, as currently supported by the Framework.

Figure 3: Data Object Diagram

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |6

LocationInfo is an immutable object. When a plug-in is called without location context, as is the case
when data files are submitted through the Springboard drag-and-drop, the plug-in can use the
IFieldDataResultsAppender to look-up a location by its identifier or unique ID. We recommend that
plug-ins and their field data file reference locations by its unique ID for the following reasons:
•
•

•

The unique ID is a string identifier assigned by AQUARIUS Time-Series. It is guaranteed to be
unique.
The location identifier can be changed from Springboard or by using the Provisioning API. When
this happens, any field data that references the location identifier needs to be updated or
AQUARIUS Time-Series will throw a LocationNotFound exception.
If the AQUARIUS Time-Series Server has an Oracle database, the location identifier is case
sensitive. In this instance, locations with different capitalization can co-exist as separate locations
(e.g., “My Location” vs. “my location”). This introduces fragility in the field data because it must
be precise when identifying its location context.

All Framework SDK data objects require timestamps specified as DateTimeOffset. However, some plugins may process field data containing timestamps that do not include an UTC-offset. Plug-ins can use
these timestamps to construct a DateTimeOffset object by combining it with the UtcOffset property in
LocationInfo.
Similarly, FieldVisitInfo is an immutable object. It is created when the plug-in uses the
IFieldDataResultsAppender to add an instance of a FieldVisitDetails object to a location.
There are factory methods to create an instance of CrossSectionSurvey, DischargeActivity, and
ManualGaugingDischargeSection object. Plug-ins should also use IFieldDataResultsAppender to add
Reading, CrossSectionSurvey and DischargeActivity objects to the IFieldVisitInfo.
Some enumerated types in the Framework SDK default to “Unknown” value. When creating data objects
that have these enumerated types as properties, the properties should be set to “Unspecified” value if
there is no other appropriate value for it so that the data displays correctly in the Field Data Editor. For
example, when creating an instance of ManualGaugingDischargeSection,
//These enums fail if you use “Unknown”. “Unspecified” is what should be used
//when nothing else fits;
var factory = new ManualGaugingDischargeSectionFactory(UnitSystem);
var manualGauging = factory.CreateManualGaugingDischargeSection(measurementPeriod,
results.TotalDischarge.Value);
manualGauging.MeterSuspension = MeterSuspensionType.Unspecified;
manualGauging.DeploymentMethod = DeploymentMethodType.Unspecified;

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |7

If these properties are left as the default “Unknown” value, the Framework will save the field data, but
when the data is viewed in the Field Data Editor, it will show an invalid icon (orange exclamation mark)
by the discharge value. Unfortunately, this value cannot be corrected in the Field Data Editor.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |8

Setting up a Development Environment
Plug-ins are 64-bit libraries written using .NET Framework 4.7. If the .NET Framework 4.7 developer pack
is not already installed, it can be downloaded here:
•

https://www.microsoft.com/en-us/download/details.aspx?id=55170

The Field Data Plug-In Framework SDK is found on the AQUARIUS Time-Series Server at:
•

{AppInstallRoot}\FieldDataPlugins\Library

… where {AppInstallRoot} is the installation root of the AQUARIUS Time-Series Server. The default
installation root is %ProgramFiles%\Aquatic Informatics\AQUARIUS Server.
The Framework SDK (plug-in interface and data objects) is defined in the following library:
•

FieldDataPluginFramework.dll.

This library must be added as a reference in your plug-in project.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |9

Writing a Field Data Plug-In
A plug-in implements the interface, IFieldDataPlugin:
namespaceFieldDataPluginFramework
{
publicinterfaceIFieldDataPlugin
{
ParseFileResultParseFile(StreamfileStream,
IFieldDataResultsAppenderfieldDataResultsAppender, ILog logger);
ParseFileResultParseFile(StreamfileStream, LocationInfotargetLocation,
IFieldDataResultsAppenderfieldDataResultsAppender, ILog logger);
}
}

In IFieldDataPlugin, there are two ParseFile method signatures:
1. Global (i.e. no location context)
a. Called by the Framework when Springboard is used to import a data file;
b. Data file must contain a reference to the location that owns the field data;
2. With location context (i.e. targetLocation)
a. Called by the Framework when LocationManager, Field Data Editor or Acquisition API is
used to import a data file;
b. Field data will be created in the targetLocation. If the data file references a location, it
must match the targetLocation or the field data will not be imported.
Each ParseFile method returns a ParseFileResult object. ParseFileResult has static constructors to
create an instance to represent the plugin status (SuccessullyParsedAndDataValid,
SuccessfullyParsedButDataInvalid, CannotParse).

Tips
1. Plug-ins should not access the network. The framework runs plug-ins locally in a sandboxed
environment.
2. Plug-ins should be thread safe. In the future, the Framework will call plug-ins concurrently from
multiple threads or processes.
3. Plug-ins should only contain managed code. Aquatic Informatics will not provide developer
support for plug-ins containing unmanaged code.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |10

4. Plug-ins should be able to process field data files in less than one second. In the future, the
Framework will terminate long-running plug-ins.
5. Plug-ins process any file that can be uploaded to the AQUARIUS Time-Series Server (through
drag-and-drop in Springboard, Location Manager and Field Data Editor), including images and
video files. Plug-ins should try to quickly inspect the start of the file stream and return the
CannotParse status if the content does not make sense.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |11

Parsing the Field Data File
A plug-in is responsible for reading the field data file and mapping its contents to one or more of the
data objects provided by the Framework SDK (FieldDataPluginFramework.dll). The ParseFile method
provides a Stream as one of its inputs. The Stream is opened by the framework and contains the
memory stream byte representation of the field data file. Below is an example of how the contents of
the stream are read (in C#):
public void ReadFile(Stream stream)
{
using (var reader = CreateStreamReader(stream, Encoding.UTF8))
{
//Read data from file.
}
}
private static StreamReader CreateStreamReader(Stream stream,
Encoding fileEncoding)
{
const int defaultByteBufferSize = 1024;
//NOTE: Make sure to set leaveOpen property on StreamReader so the Stream’s
Dispose() method is not called when StreamReader.Dispose() is called..
//Framework will take care of closing the Stream.
return new StreamReader(fileStream, fileEncoding,
detectEncodingFromByteOrderMarks: true,
bufferSize: defaultByteBufferSize, leaveOpen: true);
}

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |12

Logging
The Field Data Plug-In Framework log messages
FieldDataPluginFramework.log, found on the server at:
•

are

written

to

the

log

file,

%ProgramData%\Aquatic Informatics\AQUARIUS\Logs.

Each plug-in is provided a reference to an ILog object when its ParseFile method is called. The ILog
object writes log messages from the plug-in to the FieldDataPluginFramework.log. It is important to note
that since both the Framework and plug-in log to FieldDataPluginFramework.log, error messages from a
plug-in are logged as warnings because plug-in errors are not the same severity as errors from the
Framework (framework errors indicate problems saving field data to the AQUARIUS Time-Series Server).
The log messages written to FieldDataPluginFramework.log are designed to be easily parsed by log
analytic tools, such as LogStash.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |13

Installing and Registering your Plug-In
To install and register your plug-in:
1. On the AQUARIUS Time-Series Server, navigate to the folder:
{AppInstallRoot}\Aquatic Informatics\AQUARIUS Server\FieldDataPlugins
2. In the FieldDataPlugins folder, create a sub-folder with a name that matches your plug-in’s name.
3. Copy your plug-in and its dependencies into the sub-folder.
4. Register your plug-in with the AQUARIUS Time-Series Server using the POST /fielddataplug-ins
endpoint in the Provisioning API (http:///AQUARIUS/Provisioning/v1). The
AQUARIUS Time-Series Server is not aware that your plug-in is installed until it is registered. The
POST endpoint requires the following parameters:
a. PluginFolderName –the name of the sub-folder;
b. AssemblyQualifiedTypeName –the type name and display name of your plug-in assembly,
which allows the Framework to load your plug-in using reflection. The minimum
specification of an AssemblyQualifiedTypeName is {classTypeName}, {assemblyName}.
c. PluginPriority –the unique number representing the order that the Framework will run
your plug-in will be run relative to other registered plug-ins (the framework runs plugins
in ascending priority order). The highest priority plug-in has PluginPriority = 1;
5. When a plug-in is registered, it is assigned a unique ID. The plug-in’s unique ID will be used to
unregister the plug-in from the Framework.
For example, you have installed your plug-in assembly on the AQUARIUS Time-Series Server at
%ProgramFiles%\Aquatic Informatics\AQUARIUS
Server\FieldDataPlugins\MyPlugin\MyFirstPlugin.dll. The assembly contains a class,
Foo.FieldDataPlugin that implements the IFieldDataPluginInterface. To register your plug-in, you
would call POST /fielddata plugins in Provisioning API with the following input parameters:
- PluginFolderName = MyPlugin ;
- AssemblyQualifiedTypeName = Foo.FieldDataPlugin, MyFirstPlugin
- PluginPriority = 2500

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |14

Tips
1. To update the registration of your plug-in (e.g., change the plug-in priority), you need to unregister and re-register your plug-in.
2. We recommend selecting plug-in priorities so that there is a gap between each priority. This
makes it easier to insert new plug-ins without having to re-order the priorities of every registered
plug-in.
For example, your AQUARIUS Time-Series Server has two plug-ins registered so that PluginA is
assigned priority = 1 and PluginC is assigned priority = 3.
Now, you would like to install a new plugin, PluginB, so that it is the second plugin that will be run
by the framework. To do so, you need to un-register and re-register PluginC with priority = 3 so
that you can register PluginB with priority = 2.
However, if PluginA and PluginB were registered so that PluginA is assigned priority = 100 and
PluginC is assigned priority = 300, you do not need to change the priority of PluginC; you only
need to assign PluginC a priority between 100 and 300.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |15

Unregistering your Plug-In
A plug-in can be unregistered from the Framework using the DELETE /fielddataplugin endpoint in
Provisioning API. The delete endpoint only removes the Framework’s awareness of the plug-in (i.e.
when a field data file is submitted, the framework will not run the plug-in), it does not delete the plugin’s libraries from the AQUARIUS Time-Series Server.
To unregister a plug-in, you need to refer to the plug-in by its unique ID. A plug-in’s unique id can be
found by using the GET /fielddataplugin endpoint in Provisioning API to retrieve a list of all plug-ins
registered with the Framework.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |16

Debugging your Plug-In
After installing and registering your plug-in on the AQUARIUS Time-Series Server, you can test and debug
your plug-in by using the drag-and-drop upload method, (available from Springboard, Location Manager
or the Field Data Editor), to upload your test field data files. If there is a problem with your plug-in, the
feedback from these drag-and-drop interfaces is not very informative:

To help debug problems with your plug-in, look in FieldDataPluginFramework.log to find more details
about why the data failed to be saved to the AQUARIUS Time-Series Server.
Another approach to testing and debugging your plug-in is to use the Acquisition API, which provides
more details in its response to help you quickly diagnose problems:

However, we still recommend that you refer to FieldDataPluginFramework.log to find more details about
your plug-in failure.
Additionally, the Aquatic Informatics Example repo on GitHub provides the PluginTester.exe as a tool to
help debug your plug-in outside of the AQUARIUS Time-Series Server environment. The Plug-in Tester
can be found at:

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |17

•

https://github.com/AquaticInformatics/Examples/tree/master/TimeSeries/PublicApis/FieldDataP
lugins/PluginTester

In particular, when the –Json=AppendedResults.json command line option is used, the JSON written to
disk will include both the framework data models written by your plug-in and the correct
AssemblyQualifiedTypeName to use during plug-in registration.
If you make changes to your plug-in, it is not necessary to unregister and re-register your plug-in with
AQUARIUS Time-Series Server. Simply replace the existing plug-in library on the server with its newer
version.

Tips
1. Debugging your plug-in can be difficult if it throws any unhandled exceptions because these
exception messages are logged to the FieldDataPluginFramework.log at the DEBUG log level. By
default, all AQUARIUS Time-Series Server logs are set to the INFO log level. To avoid this problem,
we recommend that you write your own top-level exception handler that will log all exceptions to
FieldDataPluginFramework.log.
2. The DEBUG log level is the most verbose log level supported. You can increase the logging
verbosity for all AQUARIUS Time-Series Server logs by changing the log level from
 to  in %ProgramData%\Aquatic
Informatics]AQUARIUS\Logs\RemotingAppender.config.
This change will take effect immediately, without needing to re-start the server. While changing
the log level to DEBUG can help with debugging, it will also fill up the log file with a large number
of logging messages.
This will make reading the log files much more difficult, so you may want to resort to searching
for specific terms instead of scanning the logs for errors.
When you have finished debugging your plugin, we recommend that you restore the log level in
RemotingAppender.config to the INFO log level.

AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework |18



---

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.5
Linearized                      : No
Author                          : 2017.4
Category                        : December 2017
Company                         : 
Content Type Id                 : 0x010100616DDB13151CD54AB114B1FCEE456274
Create Date                     : 2017:12:08 16:08:43-08:00
Modify Date                     : 2017:12:08 16:09:10-08:00
Order                           : 800.000000
Source Modified                 : D:20171209000706
Language                        : EN-US
Tagged PDF                      : Yes
XMP Toolkit                     : Adobe XMP Core 5.6-c015 84.159810, 2016/09/10-02:41:30
Metadata Date                   : 2017:12:08 16:09:10-08:00
Creator Tool                    : Acrobat PDFMaker 18 for Word
Document ID                     : uuid:275b1c2b-575c-4d9b-a585-d2bfec3247e7
Instance ID                     : uuid:c63ff6c4-e716-4ac7-b732-c2a6b74f23e8
Subject                         : 44
Format                          : application/pdf
Title                           : AQUARIUS Time-Series Developer Guide: Field Data Plug-In Framework
Creator                         : 2017.4
Producer                        : Adobe PDF Library 15.0
Page Layout                     : SinglePage
Page Mode                       : UseOutlines
Page Count                      : 19

EXIF Metadata provided by EXIF.tools