Install Guide

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 13

VIAN
Datamodel, JSON Serialization and XML Export Format
Author:
Gaudenz Halter, 27.06.2018
ERC Advanced Grant FilmColors
Department of Film Studies
University of Z¨
urich
Visualization and MultiMedia Lab
Department of Informatics
University of Z¨
urich
VIANProject
+screenshotGroups: List<ScreenshotGroup>
+segmentations: List<Segmentation>
+annotationLayers: List<AnnotationLayer>
+experiments: List<Experiment>
+mediaDescriptor: MediaDescriptor
+analyzes: List<IAnalysis>
+mediaObjects: List<ExternalMediaObjects>
ITimeRange
+t_start
+t_end
AnnotationLayer
+annotations: List<Annotation>
Segmentation
+segments: List<Segments>
Experiment
+ClassificationObjects: Classification Object[0..*]
+MendatoryAnalyzes
+ClassificationResults
MediaDescriptor
+name: String
+file_path: String
Classification Object
+Vocabularies: List<Vocabulary>
+TargetContainers: IContainerGroup[0..1]
Vocabulary
+words: List<VocabularyWord>
+category: String
VocabularyWord
UniqueKeyword
+classificationObject
+vocabularyWord
Segment
+body: String
VisualAnnotationBaseClass
+position: Vector2
+size: Vector2
+type: EnumAnnotationType
+0..*
+0...* +0...*
+0...*
ScreenshotGroup
+screenshots: List<Screenshot>
Screenshot
+0...*
+1..*
IContainerGroup
+children
+0..*
+0...n
+0..n
ClassificationEntry
+Classifyable: IClassifyable
+Keyword: UniqueKeyword
IClassifyable
1
1
IAnalysis
+Parameters: Dict
+AnalysisData: Dict
ExternalMediaObject
+name: String
+file_path: String
+0..n
+0..*
IMediaObjectAttachable
+ExternalMediaObjects: List<ExternalMediaObject>
ii
1 Introduction
Essentially, VIAN projects are represented as a directory system. VIAN serializes its projects into a json file and
a collection of mostly numeric analysis result which are hold in a sqlite database. (Currently there exist additional
binary files that stored in a projects filesystem, but these will be deprecated shortly.)
1VIAN FileSystem:
2Root - |
3| - myproject.json
4| - data ------- |
5| - database.sqlite
6| - results // contains visualizations
7| - export // default exported data
8| - shots // default exported shots
Since a lot of the data stored is application and python specific, VIAN supports an XML export of its projects,
its elements are described in this document. Links between different entities are serialized using the ID attribute.
IDs are guaranteed to be unique and declared before referenced in the document.
Additionally the VIAN project json file is described at the end of this document by an example.
2 XML Schema
2.1 Element: ANNOTATION DOCUMENT
1<ANNOTATION_DOCUMENT AUTHOR="author" FORMAT="2.8" VERSION="2.8">
2
3@AUTHOR: The Person that created the File
4@DATE: The creation date
5@VERSION: The VIAN version this file was produced with
2.2 Element: HEADER
1<HEADER PROJECT_NAME="107_1_1_Leave Her to Heaven_1945">
2
3## HEADER
4@ PROJECT_NAME: The name of the project
2.3 Element: MEDIA DESCRIPTOR
1<MEDIA_DESCRIPTOR DURATION="9807872" FPS="30" IS_RELATIVE="False" MEDIA_URL="//abs_path
/3774_1_1_Blade_Runner_2049_DVD.m4v" TIME_UNITS="milliseconds" YEAR="1982"/>
2
3@FPS: a float indicating the fps of this media
4@IS_RELATIVE: if MEDIA_URL is relative or not {True, False}
5@MEDIA_URL: Absolute or relative path to the media
6@YEAR: The creation year of the media
iii
2.4 Element: TIME ORDER and TIME SLOT
1<TIMER_ORDER>
2<TIME_SLOT TIME_SLOT_ID="ts0" TIME_VALUE="0"/>
3<TIME_SLOT TIME_SLOT_ID="ts1" TIME_VALUE="19300"/>
4<TIME_SLOT TIME_SLOT_ID="ts2" TIME_VALUE="56300"/>
5<TIME_SLOT TIME_SLOT_ID="ts3" TIME_VALUE="91700"/>
6...
7</TIMER_ORDER>
8
9@TIME_SLOT_ID: A string that can be referenced by the serialization of any ITimeRange
implementing object
10 @TIME_VALUE: The time of this slot in MS
VIAN uses an implicit timeline, that is, each Segment, Visual Annotation or Screenshot object stores it’s media
time directly. In the XML export format, a list of TIME SLOTS is created, and each serialization of an object
media-time information references one or two TIME SLOTS.
Note: Currently TIME SLOTS are unique, thus for each time there can not be more than one TIME SLOT.
2.5 Element: SEGMENTATION and SEGMENT
1<SEGMENTATION ID="7660852745" NAME="Main Segmentation">
2<SEGMENT BODY="Some Annotation Text" ID="1697026929" T_END="ts4" T_START="ts3"/>
3</SEGMENTATION>
4
5##SEGMENTATION
6@ID: Unique ID of this segmentation
7@NAME: The name of this segmentation
8
9#SEGMENT
10 @BODY: The annotation text
11 @ID: The unique ID of this annotation
12 @T_END: A reference to a TIME_SLOT where this annotation ends
13 @T_START: A reference to a TIME_SLOT where this annotation starts
The first kind of annotation supported by VIAN are called ”Segment” and are aggregated in a Tier called ”Seg-
mentation”. A segment has a start and end-point and may have a text body.
2.6 Element: SCREENSHOTS and SCREENSHOT
1<SCREENSHOTS>
2<SCREENSHOT ID="2373585929" T_START="ts69"/>
3<SCREENSHOT ID="7301671299" T_START="ts70"/>
4<SCREENSHOT ID="7129636405" T_START="ts71"/>
5...
6</SCREENSHOTS>
7
8@ID: The unique id of this screenshot
9@T_START: A reference to a TIME_SLOT indicating the start of this annotation
The second type of annotations are ”Screenshots”, which are aggregated in ”ScreenshotGroups”. A screenshot
only has a start-time in the XML export.
iv
2.7 Element: ANNOTATION LAYER and VISUAL ANNOTATION
Finally there are ”Visual Annotations”, aggregated in ”Annotation Layers”, which represent vector graphics, that
are placed on the screen.
1
2<ANNOTATION_LAYER ID="5865990447" NAME="New Layer">
3<VISUAL_ANNOTATION A_TYPE="AnnotationType.Rectangle" COLOR="[231, 20, 221]" ID="
3736115394" POS="495, 81" RESSOURCE_PATH="" SIZE="986.1626617375231,
248.42144177449177" TEXT="" T_END="ts68" T_START="ts67"/>
4</ANNOTATION_LAYER>
5
6## ANNOTATION_LAYER
7@ID: The unique id of this annotation layer
8@NAME: The name of this annotation layer
9
10 ## VISUAL_ANNOTATION
11 @A_TYPE: The type of this annotation {AnnotationType.Rectangle, AnnotationType.Ellipse,
AnnotationType.Line, AnnotationType.Text, AnnotationType.Image, AnnotationType.
FreeHand}
12 @COLOR: The Color of this annotation
13 @ID: The unique id of this annotation
14 @POS: A vector2 indicating the screen position of this annotation
15 @RESSOURCE_PATH: If A_TYPE == AnnotationType.Image, this specifies the relative path to
the Image-Source {.png, .jpg format}
16 @SIZE: A vector2 indicating the screen size of this annotation
17 @TEXT: If A_TYPE == AnnotationType.Text, this specifies the text of the annotation
18 @T_END: A reference to a TIME_SLOT, when the annotation ends
19 @T_START: A reference to a TIME_SLOT, when the annotation starts
Note that the export of visual annotations does currently not support:
•Export of keys on annotations
•Export of FreeHand annotations.
v
2.8 Element: EXPERIMENT
The process of data acquisition (i.e. creating segments, annotations and screenshots) and classification of these
entities using vocabularies is splitted, such that different classifications can be performed on the same segments
and annotations. The second process is contained in the Experiment entity.
Essentially a subject of interest (Classification Object), can be present in several tiers (the targets), and can be
classified by multiple vocabularies. For each classification object a list of unique keywords is generated which
can then be attached to the classification targets as tags. All this information is contained in an experiment.
Example: The classification object ”Background” can be present in the scene segmentation
as well as the screenshots, and would probably be classified with the vocabularies ”Color”,
”Shadows”.
1<EXPERIMENT ID="8835132857" NAME="ERC Advanced Grant FilmColors">
2<CLASSIFICATION_OBJECTS>
3<CLASSIFICATION_OBJECT ID="3719655258" NAME="Global" PARENT="-1">
4<TARGET ID="1057466290"/>
5...
6</CLASSIFICATION_OBJECT>
7...
8</CLASSIFICATION_OBJECTS>
9<VOCABULARIES>
10 <VOCABULARY CATEGORY="Key Words" ID="6613879615" INFO_URL="" NAME="
Intertextuality Intermediality">
11 <VOCABULARY_WORD ID="5429449722" INFO_URL="" NAME="allusion"/>
12 <VOCABULARY_WORD ID="2997485372" INFO_URL="" NAME="citation"/>
13 <VOCABULARY_WORD ID="3149334985" INFO_URL="" NAME="irony"/>
14 ...
15 </VOCABULARY>
16
17 </VOCABULARIES>
18 <KEYWORDS>
19 <KEYWORD CLASSIFICATION_OBJECT_ID="3719655258" ID="8808338181" WORD_ID="5131968152
"/>
20 <KEYWORD CLASSIFICATION_OBJECT_ID="3719655258" ID="7010099893" WORD_ID="6882728027
"/>
21 <KEYWORD CLASSIFICATION_OBJECT_ID="3719655258" ID="6579133855" WORD_ID="6898493964
"/>
22 ...
23 </KEYWORDS>
24 <CLASSIFICATION>
25 <KEYWORD KEYWORD_ID="2040265900" TARGET_ID="7776732753"/>
26 <KEYWORD KEYWORD_ID="3250706923" TARGET_ID="7776732753"/>
27 <KEYWORD KEYWORD_ID="7875625863" TARGET_ID="7776732753"/>
28 ...
29 </CLASSIFICATION>
30 </EXPERIMENT>
vi
2.9 Element: CLASSIFICATION OBJECT
1<CLASSIFICATION_OBJECT ID="6353598060" NAME="Global">
2<TARGET ID="1049798528"/>
3</CLASSIFICATION_OBJECT>
4
5@ID: The unique id of this classification object
6@NAME: The name of this classification object
7@TARGET_ID: A reference to IContainerGroup (Segmentation, AnnotationLayer,
ScreenshotGroup) this classification object can be found in
2.10 Element:VOCABULARY and VOCABULARY WORD
1<VOCABULARY CATEGORY="Lighting" ID="9296687589" INFO_URL="" NAME="Shadows">
2<VOCABULARY_WORD ID="1154796472" INFO_URL="" NAME="body_shadow"/>
3<VOCABULARY_WORD ID="8894958800" INFO_URL="" NAME="cast_shadows"/>
4<VOCABULARY_WORD ID="9365131971" INFO_URL="" NAME="colored"/>
5<VOCABULARY_WORD ID="1864858843" INFO_URL="" NAME="cookie"/>
6</VOCABULARY>
7
8## VOCABULARY
9@CATEGORY: The topic this vocabulary should belong to
10 @ID: The unique id of this Vocabulary
11 @INFO_URL: A optional url to more information about this vocabulary
12
13 #VOCABULARY_WORD
14 @ID: The unique id of this VocabularyWord
15 @INFO_URL: A optional url to more information about this vocabulary
16 @NAME: The name of this word
2.11 Element: KEYWORDS
1<KEYWORDS>
2<KEYWORD CLASSIFICATION_OBJECT_ID="6353598060" ID="4400252826" WORD_ID="5741784264"/>
3<KEYWORD CLASSIFICATION_OBJECT_ID="6353598060" ID="8650499482" WORD_ID="2534552219"/>
4<KEYWORD CLASSIFICATION_OBJECT_ID="6353598060" ID="1149034722" WORD_ID="9103031091"/>
5<KEYWORD CLASSIFICATION_OBJECT_ID="6353598060" ID="9690251691" WORD_ID="3412950183"/>
6<KEYWORD CLASSIFICATION_OBJECT_ID="6353598060" ID="2806440146" WORD_ID="8178481154"/>
7</KEYWORDS>
8
9@CLASSIFICATION_OBJECT_ID: The unique ID of the classification object this keyword
target.
10 @ID: The unique ID of this keyword
11 @WORD_ID: The unique ID of the VocabularyWord this keyword targets.
2.12 Element: CLASSIFICATION
The classification represents the mapping of tags their targets. (screenshots, segments, visual annotations)
1<CLASSIFICATION>
2<KEYWORD KEYWORD_ID="2016330087" TARGET_ID="1166372514"/>
3<KEYWORD KEYWORD_ID="3935737480" TARGET_ID="1166372514"/>
4...
5</CLASSIFICATION>
6
7@KEYWORD_ID: The unique ID of the keyword
8@TARGET_ID: The unique ID of the IClassifyable
vii
2.13 Element: EXTERNAL MEDIA OBJECT
1<EXTERNAL_MEDIA_OBJECTS>
2<EXTERNAL_MEDIA ID="1741505363" PATH="01_DOF_Explanation.png" TARGET_ID="1166372514"/>
3</EXTERNAL_MEDIA_OBJECTS>
4
5@ID: The unique id of this external media object
6@PATH: The relative path to the external media object in the export directory
7@TARGET_ID: The unique ID of the container this media objects is attached to
2.14 Element: ANALYSES
1<ANALYSES>
2<COLORIMETRY_ANALSIS ID="8908799312" NAME="Colorimetry" PATH="8908799312.npz"/>
3<JOB_ANALYSIS CLASSIFICATION_OBJECT_REF="-1" ID="1522242163" NAME="
SemanticSegmentationAnalysis" PATH="No Export" TYPE="MASKS"/>
4</ANALYSES>
5
6## COLORIMETRY_ANALSIS
7@ID: The unique id of this analysis object
8@NAME: Constant = "Colorimetry"
9@PATH: The path to the numpy numeric serialized data
10
11 ## JOB_ANALYSIS
12 @ID: The unique id of this analysis object
13 @NAME: The name of the JobAnalysis Class (can be used in an python::eval() statement)
14 @PATH: The path to the numpy numeric serialized data or "No Export" if data export has
been turned off
15 @CLASSIFICATION_OBJECT_REF: The unique id of the classification object this analysis has
been performed of or -1 if None
viii
3 Json Schema
Since the VIAN json file has not been intended to be read by other applications, the XML export may be the
better way to enforce interoperability with other applications. But for the sake of completeness, a documented
JSON is shown in the rest of this document.
1{
2
3"name":"3774_1_1_Blade_Runner_1900_DVD",
4"path":"C://****/****/documents/VIAN/3774_1_1_Blade_Runner_1900_DVD/3774
_1_1_Blade_Runner_1900_DVD.eext",// The abs path to the project
5"version":"0.6.4",
6"corpus_id": -1,
7"main_segmentation_index":0,// THe Index of the Main Segmentation (Which is used for
sorting screenshots)
8"notes":"",// Additional information that can be set in the Inspector
9
10 /*Project Directories */
11 "data_dir":"C://****/****/documents/VIAN/3774_1_1_Blade_Runner_1900_DVD/data",
12 "shots_dir":"C://****/****/documents/VIAN/3774_1_1_Blade_Runner_1900_DVD/shots",
13 "export_dir":"C://****/****/documents/VIAN/3774_1_1_Blade_Runner_1900_DVD/export",
14 "results_dir":"C://****/****/documents/VIAN/3774_1_1_Blade_Runner_1900_DVD/results",
15
16 /*SEGMENTATIONS*/
17 "segmentation":[/*A List of serialized Segmentations */
18 {
19 "segments":[/*A List of serialized Segments */
20 {
21 "name":"1",// The Name of the Segment
22 "end": 107607, // MS of the End
23 "notes":"",// Additional Notes
24 "start":0,// MS of the Start
25 "media_objects":[// A List of serialized Media Objects
26 {
27 "dtype":2,// What type the media object is (External, Data,)
28 "file_path":"C:\\*******/documents/VIAN//3774_1_1_Blade_Runner_1900_DVD//
data/01_DOF_Explanation.png",// An absolute path to the File, this
is regenerated when loading the project
29 "name":"DOF_Explanation.png",// The name of the file in the data_dir
directory
30 "unique_id": 1741505363
31 }
32 ],
33 "scene_id":1,// The index of this segment in the Segmentation (starting with
1)
34 "unique_id": 1166372514,
35 "annotation_body":"Intertitle explains the already passed plot",// The text
annotation
36 "locked":false,// If the segment is locked in the timelnie
37 "duration": null // @obsolete
38 }
39 /*... more Segments ...*/
40 ],
41 "notes":"",// Additional information that can be set in the Inspector
42 "unique_id": 1049798528,
43 "locked":false,// If the segmentation is locked in the Timeline
44 "name":"Auto Segmentation" // The name of the segmentation
45 }
46 /*... more Segmentation ...*/
47 ],
48
49 /*ANNOTATION LAYERS */
ix
50 "annotation_layers": [
51 {
52 "name":"New Layer",
53 "locked":false,// If it is locked in the Timeline
54 "is_visible":true,// If it should be displayed
55 "unique_id": 2927829419,
56 "notes":"",// Additional information that can be set in the Inspector
57 "annotations":[// A list of Annotation serializations
58 {
59 "free_hand_paths": [], // A list of FreeHand Paths if this is a FreeHand
annotation
60 "name":"New Rectangle",
61 "notes":"",// Additional information that can be set in the Inspector
62 "t_end": 82300, // When the visibility should be turned off
63 "is_automated":false,// if this annotation is driven by another containers
value
64 "line_w":5,// the line width of the frame if it has one
65 "orig_position":[178,6],// The position in MovieSpace
66 "color": [ 230, 83, 162 ], // The color of the frame if it has one
67 "automate_property": null, // The value this is driven by if it is automated
68 "automated_source": -1, // The unique id of the object that contains the value
above if it has any
69 "curr_size": [ 1512.5951940850277, 421.8022181146026 ], // The size of this
annotation in MovieSpace
70 "t_start": 53900, // The MS start when the annotation should start beeing
visible
71 "unique_id": 1432985272,
72 "font_size":6,// The Font-Size if it has text
73 "resource_path":"",// The Image path it is an Image Annotation
74 "text":"",// The Text if it is a Text Annotation
75 "size": [ 1512.5951940850277, 421.8022181146026 ], // The current size (no
need to be serialized) in PlayerSpace
76 "media_objects": [], // Attached media objects if there are any
77 "a_type":0,// The AnnotationType Enum Value
78 "font":"Agency FB",// The Font Family
79 "tracking":"Static",// If this annotation should track something
80 "keys": [], // A list of Position Tuples if this annotation is animated
81 "widget": null // The displaying Widget
82 }
83 /*... more Annotations .. */
84 ],
85 "is_current_layer":false // If this is the current top layer
86 }
87 ],
88
89 /*NODE SCRIPTS*/
90 "scripts": [
91 {
92 "notes":"",// Additional information that can be set in the Inspector
93 "connections": [
94 {
95 "output_node": 6264180387,
96 "input_pin_id": 2,
97 "output_pin_id": 0,
98 "input_node": 8803892119
99 }
100 /*... more Connections ...*/
101 ],
102 "name":"New Script",// The Name of the Script
103 "nodes": [
104 {
105 "name":"Read Frame Movie",
x
106 "default_values":["test.mp4", 3000 ], // The Default Values of the input
pins
107 "node_pos": [ 107, 94 ], // The position of the node in the Editor
108 "unique_id": 8803892119,
109 "notes":"",// Additional information that can be set in the Inspector
110 "operation":"OperationFrameReader",// The associated OperationClass Name.
111 "node_size": [ 200, 200 ] // The size of the Node in the editor
112 }
113 /*... more Nodes ...*/
114 ],
115 "unique_id": 1284084303
116 }
117 /*... more Scripts ...*/
118 ],
119
120 /*VOCABULARIES */
121 "vocabularies": [
122 {
123 "name":"Scene Locations",
124 "category":"default",// The topic category it belongs to
125 "unique_id": 1095049086,
126 "words": [
127 {
128 "parent": 1095049086,
129 "name":"House",
130 "unique_id": 3390127240,
131 "children":[]// Child words if any
132 }
133 //....
134 ]
135 }
136 // ....
137 ],
138 "screenshot_groups": [
139 {
140 "unique_id": 1404536708,
141 "name":"All Shots",
142 "shots": [ 1651945337, 7522186061, 8671808241 /*...*/]// A list of associated
shots as unique id
143 }
144 ],
145
146 /*EXPERIMENTS */
147 "experiments": [
148 {
149 "name":"ERC Advanced Grant FilmColors",
150 "analyses": [], // A list Analysis classes to perform
151 "unique_id": 4846731777,
152 "classification_objects": [
153 {
154 "parent": 4846731777,
155 "target_container": [ 1049798528 ], // A list of IClassifyable that contain
this classification object
156 "name":"Global",
157 "children": [], // Child Classification objects if any
158 "classification_vocabularies": [ 7835076715, 3427161829 /*...*/], // The
Vocabularies associated with this classification object
159 "unique_id": 6353598060,
160 "unique_keywords":[// These are essentially the tags consisting of VocWord,
ClassificationObject and Vocabulary
161 {
162 "voc_obj": 7835076715,
163 "external_id": 8,
xi
164 "word_obj": 5741784264,
165 "unique_id": 4400252826,
166 "class_obj": 6353598060
167 }
168 /*...*/
169 ]
170 }
171 ],
172 "classification_results": [ // A List of Mappings: (Target.unique_id, UKeyqord.
unique_id)
173 [ 1166372514, 2016330087 ],
174 [ 1166372514, 3935737480 ],
175 [ 1166372514, 3145357142 ],
176 [ 1166372514, 3663305810 ]
177 /*...*/
178 ]
179 }
180 ],
181
182 /*SCREENSHOTS */
183 "screenshots": [
184 {
185 "shot_id_global":1,// The index of the screenshot in all screenshots sorted by
time
186 "notes":"",// Additional notes from the inspector
187 "title":"Auto_Scr_0",
188 "creation_timestamp":"2018-06-27 10:28:14.640474",
189 "movie_timestamp": 42, // Time in MS
190 "frame_pos":1,// Frame IDX of this shot
191 "shot_id_segm":1,// The Index of this Shot within the segment of the main
segmentation (Starting with one)
192 "scene_id":1,// The idx of the segment in the main segmentation it is associated
with (Starting with one)
193 "unique_id": 1651945337,
194 "annotation_item_ids": null // Annotations that should be rendered ontop of this
Screenshot
195 }
196 /*...*/
197 ],
198
199 /*ANALYSES */
200 "analyzes": [
201 {
202 "has_finished":true,// If the Colorimetry is completely finished
203 "unique_id": 8908799312,
204 "notes":"",
205 "name":"Colormetry",
206 "analysis_container_class":"ColormetryAnalysis"
207 },
208 {
209 "parameters":{"resolution": 100 },
210 "name":"Color-Palette",
211 "analysis_job_class":"ColorPaletteAnalysis",
212 "notes":"",
213 "unique_id": 1105697457,
214 "container": 3778593545,
215 "analysis_container_class":"IAnalysisJobAnalysis"
216 }
217 ],
218
219 /*MOVIE DESCRIPTOR */
220 "movie_descriptor": {
221 "is_relative":false,// If the movie is located in the project directory
xii
222 "movie_name":"Blade Runner",
223 "movie_id":"3774_1_1",// A ID related to the ERC FilmColors Project
224 "notes":"",
225 "year": 2017, // The Production Year
226 "source":"DVD",// The Source of the Movie
227 "movie_path":"//****/Filme/spaete_Filme/3774_Blade_Runner_2049/3774
_1_1_Blade_Runner_2049_DVD.m4v",// Abs or rel path depending on the location of
the movie
228 "unique_id": 5420540984,
229 "duration": 9807872 // Duration in MS
230 }
231
232 }
xiii

Navigation menu