HDF5_Users_Guide HDF5 Users Guide

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 393 [warning: Documents this large are best viewed by clicking the View PDF Link!]

HDF5User’sGuide
HDF5Release1.10.0
March2016
HDF5User’sGuide
ii TheHDFGroup
HDF5User’sGuide CopyrightNoticeandLicenseTerms
TheHDFGroup iii
CopyrightNoticeandLicenseTerms
ThispagehascopyrightnoticeandlicensetermsfortheHDF5(HierarchicalDataFormat5)Software
LibraryandUtilities.
HDF5(HierarchicalDataFormat5)SoftwareLibraryandUtilities
Copyright2006‐2016byTheHDFGroup.
NCSAHDF5(HierarchicalDataFormat5)SoftwareLibraryandUtilities
Copyright1998‐2006bytheBoardofTrusteesoftheUniversityofIllinois.
Allrightsreserved.
Redistributionanduseinsourceandbinaryforms,withorwithoutmodification,arepermittedforany
purpose(includingcommercialpurposes)providedthatthefollowingconditionsaremet:
• Redistributionsofsourcecodemustretaintheabovecopyrightnotice,thislistofconditions,and
thefollowingdisclaimer.
• Redistributionsinbinaryformmustreproducetheabovecopyrightnotice,thislistofconditions,
andthefollowingdisclaimerinthedocumentationand/ormaterialsprovidedwiththedistribu‐
tion.
•Inaddition,redistributionsofmodifiedformsofthesourceorbinarycodemustcarryprominent
noticesstatingthattheoriginalcodewaschangedandthedateofthechange.
•Allpublicationsoradvertisingmaterialsmentioningfeaturesoruseofthissoftwareareasked,but
notrequired,toacknowledgethatitwasdevelopedbyTheHDFGroupandbytheNationalCenter
forSupercomputingApplicationsattheUniversityofIllinoisatUrbana‐Champaignandcreditthe
contributors.
• NeitherthenameofTheHDFGroup,thenameoftheUniversity,northenameofanyContributor
maybeusedtoendorseorpromoteproductsderivedfromthissoftwarewithoutspecificprior
writtenpermissionfromTheHDFGroup,theUniversity,ortheContributor,respectively.
DISCLAIMER:THISSOFTWAREISPROVIDEDBYTHEHDFGROUPANDTHECONTRIBUTORS"ASIS"WITH
NOWARRANTYOFANYKIND,EITHEREXPRESSEDORIMPLIED.InnoeventshallTheHDFGrouporthe
Contributorsbeliableforanydamagessufferedbytheusersarisingoutoftheuseofthissoftware,evenif
advisedofthepossibilityofsuchdamage.
CopyrightNoticeandLicenseTerms HDF5User’sGuide
iv TheHDFGroup
Contributors:NationalCenterforSupercomputingApplications(NCSA)attheUniversityofIllinois,Fort‐
nerSoftware,UnidataProgramCenter(netCDF),TheIndependentJPEGGroup(JPEG),Jean‐loupGailly
andMarkAdler(gzip),andDigitalEquipmentCorporation(DEC).
PortionsofHDF5weredevelopedwithsupportfromtheLawrenceBerkeleyNationalLaboratory(LBNL)
andtheUnitedStatesDepartmentofEnergyunderPrimeContractNo.DE‐AC02‐05CH11231.
PortionsofHDF5weredevelopedwithsupportfromtheUniversityofCalifornia,LawrenceLivermore
NationalLaboratory(UCLLNL).Thefollowingstatementappliestothoseportionsoftheproductandmust
beretainedinanyredistributionofsourcecode,binaries,documentation,and/oraccompanyingmateri‐
als:
ThisworkwaspartiallyproducedattheUniversityofCalifornia,LawrenceLivermoreNationalLab‐
oratory(UCLLNL)undercontractno.W‐7405‐ENG‐48(Contract48)betweentheU.S.Department
ofEnergy(DOE)andTheRegentsoftheUniversityofCalifornia(University)fortheoperationof
UCLLNL.
DISCLAIMER:ThisworkwaspreparedasanaccountofworksponsoredbyanagencyoftheUnited
StatesGovernment.NeithertheUnitedStatesGovernmentnortheUniversityofCalifornianor
anyoftheiremployees,makesanywarranty,expressorimplied,orassumesanyliabilityor
responsibilityfortheaccuracy,completeness,orusefulnessofanyinformation,apparatus,prod‐
uct,orprocessdisclosed,orrepresentsthatitsusewouldnotinfringeprivately‐ownedrights.Ref‐
erencehereintoanyspecificcommercialproducts,process,orservicebytradename,trademark,
manufacturer,orotherwise,doesnotnecessarilyconstituteorimplyitsendorsement,recom‐
mendation,orfavoringbytheUnitedStatesGovernmentortheUniversityofCalifornia.Theviews
andopinionsofauthorsexpressedhereindonotnecessarilystateorreflectthoseoftheUnited
StatesGovernmentortheUniversityofCalifornia,andshallnotbeusedforadvertisingorproduct
endorsementpurposes.
HDF5isavailablewiththeSZIPcompressionlibrarybutSZIPisnotpartofHDF5andhasseparatecopyright
andlicenseterms.See“SzipCompressioninHDFProducts”forfurtherdetails.
HDF5User’sGuide TheHDFGroupHelpDesk
TheHDFGroup v
TheHDFGroupHelpDesk
TheHDFGroupHelpDesk:help@hdfgroup.org
Seethe“SupportServices”pageonTheHDFGroupwebsiteforinformationonthefollowing:
• Frequentlyaskedquestions
• Tutorials
•Howtosubscribetothehdf‐forum
Seethe“HDF5Examples”pageonTheHDFGroupwebsiteatforasetofcodeexamples.
TheHDFGroupHelpDesk HDF5User’sGuide
vi TheHDFGroup
HDF5User’sGuide UpdateStatus
TheHDFGroup vii
UpdateStatus
NomajorchangeshavebeenmadetotheHDF5User’sGuideforHDF5Release1.10.0yet.Thechanges
havebeenpostedonTheHDFGroupwebsite.Seethe“NewFeaturesinHDF5Release1.10.0”pagefor
moreinformation.
Wewelcomefeedbackonthedocumentation.Pleasesendyourcommentstodocs@hdfgroup.org.
UpdateStatus HDF5User’sGuide
viii TheHDFGroup
HDF5User’sGuide TableofContents
TheHDFGroup ix
TableofContents
CopyrightNoticeandLicenseTerms ........................................................iii
TheHDFGroupHelpDesk ..................................................................v
UpdateStatus ...........................................................................vii
ListofFigures...........................................................................xv
ListofTables ............................................................................xix
ListofCodeExamples ....................................................................xxi
ListofFunctionListings..................................................................xxv
1.TheHDF5DataModelandFileStructure....................................................1
1.1.Introduction .......................................................................1
1.2.TheAbstractDataModel............................................................5
1.2.1.File..........................................................................5
1.2.2.Group........................................................................7
1.2.3.Dataset.......................................................................8
1.2.4.Dataspace ....................................................................9
1.2.5.Datatype ....................................................................10
1.2.6.Attribute ....................................................................11
1.2.7.PropertyList .................................................................13
1.2.8.Link.........................................................................14
1.3.TheHDF5StorageModel...........................................................14
1.3.1.TheAbstractStorageModel:theHDF5FormatSpecification .........................14
1.3.2.ConcreteStorageModel .......................................................15
1.4.TheStructureofanHDF5File........................................................16
1.4.1.OverallFileStructure..........................................................16
1.4.2.HDF5PathNamesandNavigation ...............................................18
1.4.3.ExamplesofHDF5FileStructures................................................18
2.TheHDF5LibraryandProgrammingModel ................................................21
2.1.Introduction ......................................................................21
2.2.TheHDF5ProgrammingModel......................................................22
2.2.1.CreatinganHDF5File ..........................................................22
2.2.2.CreatingandInitializingaDataset................................................23
2.2.3.ClosinganObject .............................................................24
2.2.4.WritingorReadingaDatasettoorfromaFile .....................................25
2.2.5.ReadingandWritingaPortionofaDataset ........................................26
2.2.6.GettingInformationaboutaDataset.............................................32
2.2.7.CreatingandDefiningCompoundDatatypes .......................................32
2.2.8.CreatingandWritingExtendableDatasets.........................................34
2.2.9.CreatingandWorkingwithGroups...............................................37
2.2.10.WorkingwithAttributes ......................................................40
TableofContents HDF5User’sGuide
xTheHDFGroup
2.3.TheDataTransferPipeline ..........................................................43
3.TheHDF5File.........................................................................45
3.1.Introduction ......................................................................45
3.2.FileAccessModes.................................................................45
3.3.FileCreationandFileAccessProperties...............................................46
3.4.Low‐levelFileDrivers ..............................................................47
3.5.ProgrammingModelforFiles........................................................48
3.5.1.CreatingaNewFile............................................................48
3.5.2.OpeninganExistingFile ........................................................49
3.5.3.ClosingaFile.................................................................49
3.6.Usingh5dumptoViewaFile ........................................................49
3.7.FileFunctionSummaries............................................................50
3.8.CreatingorOpeninganHDF5File....................................................56
3.9.ClosinganHDF5File ...............................................................57
3.10.FilePropertyLists ................................................................58
3.10.1.CreatingaPropertyList.......................................................58
3.10.2.FileCreationProperties.......................................................58
3.10.3.FileAccessProperties.........................................................60
3.11.AlternateFileStorageLayoutsandLow‐levelFileDrivers................................61
3.11.1.IdentifyingthePreviously‐usedFileDriver........................................65
3.11.2.ThePOSIX(akaSEC2)Driver ...................................................66
3.11.3.TheDirectDriver.............................................................66
3.11.4.TheLogDriver...............................................................67
3.11.5.TheWindowsDriver..........................................................68
3.11.6.TheSTDIODriver.............................................................68
3.11.7.TheMemory(akaCore)Driver .................................................68
3.11.8.TheFamilyDriver............................................................70
3.11.9.TheMultiDriver.............................................................71
3.11.10.TheSplitDriver .............................................................72
3.11.11.TheParallelDriver ..........................................................73
3.12.CodeExamplesforOpeningandClosingFiles .........................................74
3.12.1.ExampleUsingtheH5F_ACC_TRUNCFlag........................................74
3.12.2.ExamplewiththeFileCreationPropertyList ......................................74
3.12.3.ExamplewiththeFileAccessPropertyList .......................................75
3.13.WorkingwithMultipleHDF5Files ...................................................76
4.HDF5Groups .........................................................................79
4.1.Introduction ......................................................................79
4.2.DescriptionoftheGroupObject .....................................................81
4.2.1.TheGroupObject.............................................................81
4.2.2.TheHierarchyofDataObjects...................................................83
4.2.3.HDF5PathNames .............................................................84
4.2.4.GroupImplementationsinHDF5.................................................85
4.3.Usingh5dump....................................................................86
4.4.GroupFunctionSummaries .........................................................87
4.5.ProgrammingModelforGroups .....................................................91
4.5.1.CreatingaGroup..............................................................92
4.5.2.OpeningaGroupandAccessinganObjectinthatGroup.............................93
4.5.3.CreatingaDatasetinaSpecificGroup............................................93
HDF5User’sGuide TableofContents
TheHDFGroup xi
4.5.4.ClosingaGroup...............................................................94
4.5.5.CreatingLinks ................................................................94
4.5.6.DiscoveringInformationaboutObjects...........................................97
4.5.7.DiscoveringObjectsinaGroup ..................................................97
4.5.8.DiscoveringAlloftheObjectsintheFile ..........................................97
4.6.ExamplesofFileStructures .........................................................98
5.HDF5Datasets .......................................................................103
5.1.Introduction .....................................................................103
5.2.DatasetFunctionSummaries.......................................................104
5.3.ProgrammingModelforDatasets...................................................110
5.3.1.GeneralModel ..............................................................110
5.3.2.CreateDataset ..............................................................112
5.3.3.DataTransferOperationsonaDataset ..........................................115
5.3.4.RetrievethePropertiesofaDataset.............................................121
5.4.DataTransfer ....................................................................122
5.4.1.TheDataPipeline............................................................124
5.4.2.DataPipelineFilters..........................................................126
5.4.3.FileDrivers..................................................................127
5.4.4.DataTransferPropertiestoManagethePipeline ..................................127
5.4.5.StorageStrategies............................................................129
5.4.6.PartialI/OSub‐settingandHyperslabs ...........................................131
5.5.AllocationofSpaceintheFile ......................................................131
5.5.1.StorageAllocationintheFile:Early,Incremental,Late..............................135
5.5.2.DeletingaDatasetfromaFileandReclaimingSpace...............................139
5.5.3.ReleasingMemoryResources..................................................139
5.5.4.ExternalStorageProperties ....................................................140
5.6.UsingHDF5Filters ................................................................143
5.6.1.UsingtheN‐bitFilter .........................................................143
5.6.2.UsingtheScale‐offsetFilter ....................................................160
5.6.3.UsingtheSzipFilter ..........................................................172
6.HDF5Datatypes......................................................................173
6.1.IntroductionandDefinitions .......................................................173
6.2.HDF5DatatypeModel.............................................................175
6.2.1.DatatypeClassesandProperties ................................................177
6.2.2.PredefinedDatatypes.........................................................179
6.3.HowDatatypesareUsed ..........................................................182
6.3.1.TheDatatypeObjectandtheHDF5DatatypeAPI ..................................182
6.3.2.DatasetCreation .............................................................183
6.3.3.DataTransfer(ReadandWrite).................................................183
6.3.4.DiscoveryofDataFormat......................................................184
6.3.5.CreatingandUsingUser‐definedDatatypes ......................................185
6.4.Datatype(H5T)FunctionSummaries.................................................186
6.5.ProgrammingModelforDatatypes ..................................................192
6.5.1.DiscoveryofDatatypeProperties...............................................194
6.5.2.DefinitionofDatatypes .......................................................199
6.6.OtherNon‐numericDatatypes......................................................232
6.6.1.Strings .....................................................................232
6.6.2.Reference ..................................................................234
TableofContents HDF5User’sGuide
xii TheHDFGroup
6.6.3.ENUM......................................................................236
6.6.4.Opaque ....................................................................238
6.6.5.Bitfield.....................................................................238
6.7.FillValues .......................................................................238
6.8.ComplexCombinationsofDatatypes ................................................241
6.8.1.CreatingaComplicatedCompoundDatatype .....................................242
6.8.2.AnalyzingandNavigatingaCompoundDatatype..................................248
6.9.LifeCycleoftheDatatypeObject....................................................249
6.10.DataTransfer:DatatypeConversionandSelection....................................253
6.11.TextDescriptionsofDatatypes:Conversiontoandfrom ...............................260
7.HDF5DataspacesandPartialI/O........................................................265
7.1.Introduction .....................................................................265
7.2.Dataspace(H5S)FunctionSummaries................................................265
7.3.DefinitionofDataspaceObjectsandtheDataspaceProgrammingModel ..................268
7.3.1.DataspaceObjects ...........................................................268
7.3.2.DataspaceProgrammingModel ................................................269
7.4.DataspacesandDataTransfer......................................................276
7.4.1.DataSelection ...............................................................278
7.4.2.ProgrammingModel..........................................................282
7.5.DataspaceSelectionOperationsandDataTransfer.....................................292
7.6.ReferencestoDatasetRegions .....................................................292
7.6.1.ExampleUsesforRegionReferences............................................292
7.6.2.CreatingReferencestoRegions.................................................294
7.6.3.ReadingReferencestoRegions .................................................298
7.7.SamplePrograms.................................................................300
7.7.1.h5_write.c..................................................................300
7.7.2.h5_write.f90................................................................302
7.7.3.h5_write_tr.f90..............................................................304
8.HDF5Attributes......................................................................307
8.1.Introduction .....................................................................307
8.2.ProgrammingModelforAttributes ..................................................308
8.2.1.ToOpenandReadorWriteanExistingAttribute..................................309
8.3.Attribute(H5A)FunctionSummaries ................................................309
8.4.WorkingwithAttributes ...........................................................312
8.4.1.TheStructureofanAttribute ..................................................312
8.4.2.Creating,Writing,andReadingAttributes........................................312
8.4.3.AccessingAttributesbyNameorIndex ..........................................313
8.4.4.ObtainingInformationRegardinganObject’sAttributes............................313
8.4.5.IteratingacrossanObject’sAttributes ...........................................314
8.4.6.DeletinganAttribute.........................................................314
8.4.7.ClosinganAttribute..........................................................315
8.5.SpecialIssues ....................................................................315
9.HDF5ErrorHandling ..................................................................321
9.1.Introduction .....................................................................321
9.2.ProgrammingModelforErrorHandling..............................................321
9.3.ErrorHandling(H5E)FunctionSummaries............................................321
9.4.BasicErrorHandlingOperations ....................................................323
HDF5User’sGuide TableofContents
TheHDFGroup xiii
9.4.1.ErrorStackandErrorMessage .................................................323
9.4.2.PrintandClearanErrorStack..................................................324
9.4.3.MuteErrorStack.............................................................325
9.4.4.CustomizedPrintingofanErrorStack ...........................................326
9.4.5.WalkthroughtheErrorStack ..................................................327
9.4.6.TraverseanErrorStackwithaCallbackFunction ..................................327
9.5.AdvancedErrorHandlingOperations ................................................329
9.5.1.MoreErrorAPIFunctions......................................................331
9.5.2.PushinganApplicationErrorMessageontoErrorStack.............................333
10.PropertiesandPropertyListsinHDF5 ...................................................337
10.1.Introduction....................................................................337
10.2.PropertyListClasses,PropertyLists,andProperties ...................................338
10.2.1.PropertyListClasses.........................................................339
10.2.2.PropertyLists ..............................................................340
10.2.3.Properties.................................................................341
10.3.ProgrammingModelforPropertiesandPropertyLists .................................343
10.3.1.UsingDefaultPropertyLists ..................................................343
10.3.2.BasicStepsoftheProgrammingModel.........................................344
10.3.3.AdditionalPropertyListOperations............................................346
10.4.GenericPropertiesInterfaceandUser‐definedProperties ..............................347
10.5.PropertyListFunctionSummaries ..................................................347
10.6.AdditionalPropertyListResources.................................................350
10.7.Notes .........................................................................351
11.AdditionalResources .................................................................353
Index.................................................................................355
TableofContents HDF5User’sGuide
xiv TheHDFGroup
HDF5User’sGuide ListofFigures
TheHDFGroup xv
ListofFigures
Figure1‐1.HDF5modelsandimplementations................................................1
Figure1‐2.Thelibrary,theapplicationprogram,andothermodules..............................2
Figure1‐3.Datastructuresindifferentlayers.................................................4
Figure1‐4.TheHDF5file..................................................................6
Figure1‐5.Groupmembershipvialinkobjects................................................7
Figure1‐6.Classesofnamedobjects........................................................8
Figure1‐7.Thedataset....................................................................9
Figure1‐8.Thedataspace.................................................................10
Figure1‐9.Datatypeclassifications.........................................................11
Figure1‐10.Attributedataelements.......................................................12
Figure1‐11.Thepropertylist..............................................................13
Figure1‐12.AnHDF5filewithonedataset..................................................17
Figure1‐13.ABNFgrammarforpathnames.................................................18
Figure1‐14.AnHDF5filestructurewithgroups..............................................19
Figure1‐15.AnHDF5filestructurewithgroupsandadataset..................................19
Figure1‐16.AnHDF5filestructurewithgroupsanddatasets...................................20
Figure1‐17.AnotHDF5filestructurewithgroupsanddatasets.................................20
Figure2‐1.Datasetselections.............................................................27
Figure2‐2.Aone‐dimensionalarray........................................................30
Figure2‐3.Extendingadataset............................................................35
Figure2‐4.Adatatransferfromstoragetomemory...........................................43
Figure3‐1.UMLmodelforanHDF5fileanditspropertylists...................................47
Figure3‐2.I/OpathfromapplicationtoVFLandlow‐leveldriverstostorage......................62
Figure3‐3.Twoseparatefiles.............................................................76
Figure3‐4.File2mountedonFile1.........................................................77
Figure4‐1.Afilewithastrictlyhierarchicalgroupstructure....................................79
Figure4‐2.Afilewithacircularreference...................................................80
Figure4‐3.Afilewithonegroupasamemberofitself.........................................80
Figure4‐4.AbstractmodeloftheHDF5groupobject..........................................81
Figure4‐5.Classesofnamedobjects.......................................................82
Figure4‐6.Thegroupobject..............................................................82
Figure4‐7.ABNFgrammarforHDF5pathnames.............................................84
Figure4‐8.Afilewithacircularreference...................................................85
Figure4‐9.Somefilestructures............................................................98
Figure4‐10.Moresamplefilestructures....................................................99
Figure4‐11.Hardandsoftlinks...........................................................100
Figure5‐1.Applicationviewofadataset...................................................103
Figure5‐2.Datasetprogrammingsequence.................................................111
Figure5‐3.Awriteoperation.............................................................116
Figure5‐4.Datalayoutsinanapplication...................................................123
Figure5‐5.Theprocessingorderinthedatapipeline.........................................125
Figure5‐6.Contiguousdatastorage.......................................................129
ListofFigures HDF5User’sGuide
xvi TheHDFGroup
Figure5‐7.Chunkeddatastorage.........................................................130
Figure5‐8.Compactdatastorage.........................................................131
Figure5‐9.Atwodimensionalarraystoredasacontiguousdataset.............................133
Figure5‐10.Atwodimensionalarraystoredinchunks........................................134
Figure5‐11.Externalfilestorage..........................................................141
Figure5‐12.Partitioninga2‐Ddatasetforexternalstorage....................................142
Figure5‐13.H5T_NATIVE_INTinmemory..................................................145
Figure5‐14.Passedtothen‐bitfilter......................................................145
Figure5‐15.H5T_NATIVE_FLOATinmemory................................................146
Figure5‐16.Passedtothen‐bitfilter......................................................146
Figure6‐1.Datatypes,dataspaces,anddatasets.............................................173
Figure6‐2.Thedatatypemodel...........................................................176
Figure6‐3.Compositedatatypes..........................................................176
Figure6‐4.Datatypeclasses..............................................................177
Figure6‐5.Thedatatypeobject...........................................................193
Figure6‐6.Thestoragelayoutforanew128‐bitlittle‐endiansignedintegerdatatype.............205
Figure6‐7.MemoryLayoutfora32‐bitunsignedinteger......................................206
Figure6‐8.Auser‐definedintegerdatatypewitharangeof‐1,048,583to1,048,584...............207
Figure6‐9.Auser‐definedfloatingpointdatatype...........................................208
Figure6‐10.Layoutofacompounddatatype................................................210
Figure6‐11.Layoutofacompounddatatypenestedinacompounddatatype....................211
Figure6‐12.Memorylayoutofacompounddatatypethatrequirespadding.....................213
Figure6‐13.Representingdatawithmultiplemeasurements..................................226
Figure6‐14.Memorylayoutofatwo‐dimensionalarraydatatype..............................228
Figure6‐15.MemorylayoutofaVLdatatype...............................................231
Figure6‐16.Astringstoredasone‐characterelementsinaone‐dimensionalarray................233
Figure6‐17.Storinganenumarray........................................................237
Figure6‐18.Acompounddatatypebuiltwithdifferentdatatypes..............................242
Figure6‐19.Logicaltreeforthecompounddatatypewithfourmembers........................245
Figure6‐20.Thestoragelayoutforthefourmemberdatatypes................................246
Figure6‐21.Thestoragelayoutofthecombinedfourmembers................................247
Figure6‐22.Thelayoutofthedataset.....................................................248
Figure6‐23.Lifecycleofadatatype.......................................................251
Figure6‐24.Transientdatatypestates:modifiable,read‐only,andimmutable....................252
Figure6‐25.Layoutofadatatypeconversion...............................................255
Figure6‐26.Anenumdatatypeconversion.................................................256
Figure6‐27.Alignmentofacompounddatatype.............................................258
Figure6‐28.Layoutwhenanelementisskipped.............................................260
Figure7‐1.Asimpledataspace...........................................................269
Figure7‐2.ComparingCandFortrandataspaces.............................................275
Figure7‐3.Datalayoutbeforeandafterareadoperation.....................................277
Figure7‐4.Movingdatafromdisktomemory...............................................277
Figure7‐5.Accessasub‐setofdatawithahyperslab.........................................279
Figure7‐6.Buildcomplexregionswithhyperslabunions......................................280
Figure7‐7.Usehyperslabstocombineordispersedata.......................................280
Figure7‐8.Pointselection...............................................................281
Figure7‐9.Selectingahyperslab..........................................................282
Figure7‐10.Writefromaonedimensionalarraytoatwodimensionalarray.....................285
HDF5User’sGuide ListofFigures
TheHDFGroup xvii
Figure7‐11.Transferringhyperslabunions.................................................287
Figure7‐12.Writedatatoseparatepoints..................................................290
Figure7‐13.Featuresindexedbyatable...................................................293
Figure7‐14.Storingthetablewithacompounddatatype.....................................294
Figure7‐15.Afilewiththreedatasets.....................................................295
Figure8‐1.TheUMLmodelforanHDF5attribute............................................308
Figure8‐2.AlargeorsharedHDF5attributeanditsassociateddataset(s)........................318
Figure10‐1.TheHDF5propertyenvironment...............................................337
Figure10‐2.HDF5propertylistclassinheritancehierarchy....................................340
ListofFigures HDF5User’sGuide
xviii TheHDFGroup
HDF5User’sGuide ListofTables
TheHDFGroup xix
ListofTables
Table1‐1.Propertylistclassesandtheirusage................................................13
Table2‐1.TheHDF5APInamingscheme.....................................................21
Table2‐2.Hyperslabparameters...........................................................28
Table2‐3.Compounddatatypememberproperties ...........................................33
Table3‐1.Accessflagsandmodes..........................................................46
Table3‐2.Supportedfiledrivers............................................................63
Table3‐3.Logginglevels..................................................................67
Table5‐1.Requiredinputs ...............................................................113
Table5‐2.Optionalinputs ................................................................113
Table5‐3.Categoriesoftransferproperties.................................................117
Table5‐4.Stagesofthedatapipeline ......................................................124
Table5‐5.Datapipelinefilters ............................................................126
Table5‐6.I/Ofiledrivers .................................................................127
Table5‐7.Datasetstoragestrategies .......................................................129
Table5‐8.Initialdatasetsize..............................................................132
Table5‐9.Metadatastoragesizes .........................................................132
Table5‐10.Filestorageallocationoptions ..................................................135
Table5‐11.Defaultstorageoptions ........................................................136
Table5‐12.Whentowritefillvalues.......................................................136
Table5‐13.Fillvaluestowrite ............................................................137
Table5‐14.Storageallocationandfillsummary ..............................................137
Table5‐15.H5Dreadsummary ............................................................138
Table5‐16.ExternalstorageAPI...........................................................140
Table6‐1.Datatypeclassesandtheirproperties .............................................178
Table6‐2.Architecturesusedinpredefineddatatypes........................................179
Table6‐3.Basetypes ....................................................................180
Table6‐4.Byteorder ....................................................................180
Table6‐5.Somepredefineddatatypes .....................................................181
Table6‐6.Nativeand32‐bitCdatatypes....................................................181
Table6‐7.Datatypeuses .................................................................183
Table6‐8.Generaloperationsondatatypeobjects...........................................193
Table6‐9.Functionstodiscoverpropertiesofatomicdatatypes................................195
Table6‐10.Functionstodiscoverpropertiesofatomicnumericdatatypes........................196
Table6‐11.Functionstodiscoverpropertiesofatomicstringdatatypes ..........................198
Table6‐12.Functionstodiscoverpropertiesofatomicopaquedatatypes........................198
Table6‐13.Functionstodiscoverpropertiesofcompositedatatypes ............................198
Table6‐14.Functionstocreateeachdatatypeclass ..........................................200
Table6‐15.APImethodsthatsetpropertiesofatomicdatatypes ...............................201
Table6‐16.APImethodsthatsetpropertiesofnumericdatatypes..............................202
Table6‐17.APImethodsthatsetpropertiesofstringdatatypes ................................203
Table6‐18.APImethodsthatsetpropertiesofopaquedatatypes...............................204
Table6‐19.MemoryLayoutfora32‐bitunsignedinteger......................................205
ListofTables HDF5User’sGuide
xx TheHDFGroup
Table6‐20.Representingdatawithmultiplemeasurements ...................................225
Table6‐21.Storagemethodadvantagesanddisadvantages ....................................227
Table6‐22.Anenumerationwithfiveelements ..............................................236
Table6‐23.DatatypeAPIs ................................................................253
Table6‐24.Defaultactionsfordatatypeconversionexceptions.................................254
Table7‐1.Hyperslabelements............................................................279
Table7‐2.Selectionoperations ...........................................................291
Table7‐3.Theinquiryfunctions ...........................................................299
Table10‐1.PropertylistclassesinHDF5....................................................339
Table11‐1.Additionalresources ..........................................................353
HDF5User’sGuide ListofCodeExamples
TheHDFGroup xxi
ListofCodeExamples
CodeExample2‐1.CreatingandclosinganHDF5file...........................................23
CodeExample2‐2.Createadataset.........................................................24
CodeExample2‐3.Closeanobject..........................................................24
CodeExample2‐4.Writingadataset ........................................................25
CodeExample2‐5.Definetheselectiontobereadfromstorage .................................29
CodeExample2‐6.Definethememorydataspaceandselection.................................30
CodeExample2‐7.Thedestinationselection.................................................31
CodeExample2‐8.Routinestogetdatasetparameters ........................................32
CodeExample2‐9.Acompounddatatypeforcomplexnumbers .................................34
CodeExample2‐10.Declaringadataspacewithunlimiteddimensions ............................36
CodeExample2‐11.Enablechunking........................................................36
CodeExample2‐12.Createadataset .......................................................36
CodeExample2‐13.Extendthedatasetbysevenrows .........................................37
CodeExample2‐14.Createagroup .........................................................37
CodeExample2‐15.Createagroupwithinagroup............................................38
CodeExample2‐16.Createadatasetwithinagroupusinganabsolutename......................39
CodeExample2‐17.Createadatasetwithinagroupusingarelativename........................39
CodeExample2‐18.Accessingagroupusingitsabsolutename..................................40
CodeExample2‐19.Accessingagroupusingitsrelativename...................................40
CodeExample2‐20.Createanattribute .....................................................41
CodeExample2‐21.Readaknownattribute .................................................42
CodeExample2‐22.Readanunknownattribute..............................................42
CodeExample3‐1.CreatinganHDF5fileusingpropertylistdefaults .............................48
CodeExample3‐2.CreatinganHDF5fileusingpropertylists ....................................48
CodeExample3‐3.OpeninganHDF5file.....................................................49
CodeExample3‐4.ClosinganHDF5file......................................................49
CodeExample3‐5.Identifyingadriver ......................................................65
CodeExample3‐6.UsingthePOSIX,akaSEC2,driver ..........................................66
CodeExample3‐7.UsingtheDirectdriver ...................................................66
CodeExample3‐8.Loggingfileaccess .......................................................67
CodeExample3‐9.UsingtheWindowsdriver.................................................68
CodeExample3‐10.UsingtheSTDIOdriver ..................................................68
CodeExample3‐11.Managingfileaccessforin‐memoryfiles ...................................69
CodeExample3‐12.Managingfilefamilyproperties ...........................................70
CodeExample3‐13.Managingaccesspropertiesformultiplefiles ...............................72
CodeExample3‐14.Managingaccesspropertiesforsplitfiles ...................................72
CodeExample3‐15.Managingparallelfileaccessproperties ....................................73
CodeExample3‐16.Creatingafilewithdefaultcreationandaccessproperties ....................74
CodeExample3‐17.Creatingafilewith64‐bitoffsets ..........................................75
CodeExample3‐18.OpeninganexistingfileforparallelI/O.....................................75
CodeExample3‐19.UsingH5Fmount .......................................................77
CodeExample4‐1.Creatingthreenewgroups ................................................92
CodeExample4‐2.Openadatasetwithrelativeandabsolutepaths ..............................93
CodeExample4‐3.Createadatasetwithabsoluteandrelativepaths .............................93
ListofCodeExamples HDF5User’sGuide
xxii TheHDFGroup
CodeExample4‐4.Closeagroup...........................................................94
CodeExample4‐5.Createahardlink........................................................95
CodeExample4‐6.Deletealink ............................................................95
CodeExample4‐7.Findingthenumberoflinkstoanobject.....................................95
CodeExample4‐8.Createasoftlink........................................................96
CodeExample5‐1.Createanemptydataset.................................................114
CodeExample5‐2.Createadatasetwithfillvalueset.........................................115
CodeExample5‐3.Writeanarrayofintegers................................................118
CodeExample5‐4.Writeanarrayusingapropertylist ........................................119
CodeExample5‐5.Readanarrayfromadataset .............................................121
CodeExample5‐6.Retrievedataset........................................................122
CodeExample5‐7.UsingH5Dset_extenttoincreasethesizeofadataset ........................134
CodeExample5‐8.Externalstorage........................................................141
CodeExample5‐9.Partitioninga2‐Ddatasetforexternalstorage...............................142
CodeExample5‐10.N‐bitcompressionforintegerdata.......................................152
CodeExample5‐11.N‐bitcompressionforfloating‐pointdata..................................156
CodeExample5‐12.Scale‐offsetcompressionintegerdata ....................................165
CodeExample5‐13.Scale‐offsetcompressionfloating‐pointdata ...............................169
CodeExample6‐1.Usingadatatypetocreateadataset .......................................183
CodeExample6‐2.Writingtoadataset.....................................................184
CodeExample6‐3.Readingfromadataset ..................................................184
CodeExample6‐4.Discoveringdatatypeproperties ..........................................185
CodeExample6‐5.Createanewdatatype ..................................................200
CodeExample6‐6.Createanew128‐bitlittle‐endiansignedintegerdatatype ....................204
CodeExample6‐7.Auser‐defineddatatypewitha24‐bitsignedinteger.........................207
CodeExample6‐8.Auser‐defined24‐bitfloatingpointdatatype ...............................207
CodeExample6‐9.AcompounddatatypeforcomplexnumbersinC............................209
CodeExample6‐10.AcompounddatatypeforcomplexnumbersinFortran......................210
CodeExample6‐11.Codeforacompounddatatypenestedinacompounddatatype ..............211
CodeExample6‐12.Anothercompounddatatypenestedinacompounddatatype ................212
CodeExample6‐13.Acompounddatatypethatrequirespadding...............................213
CodeExample6‐14.CreateapackedcompounddatatypeinC.................................214
CodeExample6‐15.CreateapackedcompounddatatypeinFortran ............................214
CodeExample6‐16.CreateandwriteadatasetwithacompounddatatypeinC...................215
CodeExample6‐17.Createandwritealittle‐endiandatasetwithacompounddatatypeinC........216
CodeExample6‐18.Writingfloatsanddoublestoadataset ...................................217
CodeExample6‐19.Writingfloatsanddoublestoadatasetonalittle‐endiansystem..............218
CodeExample6‐20.CreateandwriteadatasetwithacompounddatatypeinFortran..............219
CodeExample6‐21.Readadatasetusingamemorydatatype..................................222
CodeExample6‐22.ReadadatasetusingH5Tget_native_type.................................223
CodeExample6‐23.Readonefloatingpointmemberofacompounddatatype ...................224
CodeExample6‐24.Readfloatanddoublemembersofacompounddatatype ....................225
CodeExample6‐25.Createatwo‐dimensionalarraydatatype..................................227
CodeExample6‐26.Createavariable‐lengthdatatypeofunsignedintegers......................229
CodeExample6‐27.DataelementstorageformembersoftheVLdatatype ......................229
CodeExample6‐28.WriteVLdata .........................................................230
CodeExample6‐29.ReadVLdata .........................................................230
CodeExample6‐30.SetthestringdatatypesizetoH5T_VARIABLE ..............................234
HDF5User’sGuide ListofCodeExamples
TheHDFGroup xxiii
CodeExample6‐31.Readvariable‐lengthstringsintoCstrings .................................234
CodeExample6‐32.Createobjectreferencesandwritetoadataset ............................235
CodeExample6‐33.Readadatasetwithareferencedatatype .................................235
CodeExample6‐34.Createanenumerationwithfiveelements.................................236
CodeExample6‐35.Createadatasetwithafillvalueof‐1.....................................239
CodeExample6‐36.Createafillvalueforacompounddatatype................................239
CodeExample6‐37.Retrieveafillvalue ....................................................240
CodeExample6‐38.Readthefillvalueforacompounddatatype...............................240
CodeExample6‐39.Createacompounddatatypewithfourmembers ...........................243
CodeExample6‐40.Outputfromh5dumpforthecompounddatatype..........................245
CodeExample6‐41.Analyzingacompounddatatypeanditsmembers ..........................249
CodeExample6‐42.Createashareabledatatype ............................................252
CodeExample6‐43.SpecifythedestinationdatatypewithH5Dread.............................254
CodeExample6‐44.Createanalignedandpackedcompounddatatype..........................257
CodeExample6‐45.Transfersomefieldsofacompounddatatype ..............................259
CodeExample6‐46.ThedefinitionofHDF5datatypesfromtheHDF5DDL .......................261
CodeExample6‐47.Olddefinitionsoftheopaqueandcompounddatatypes .....................263
CodeExample6‐48.Creatingavariable‐lengthstringdatatypefromatextdescription .............263
CodeExample6‐49.Creatingacomplexarraydatatypefromatextdescription ...................264
CodeExample7‐1.Selectingahyperslab....................................................283
CodeExample7‐2.Definingthedestinationmemory .........................................284
CodeExample7‐3.Asamplereadspecifyingsourceanddestinationdataspaces...................284
CodeExample7‐4.Writefromaonedimensionalarraytoatwodimensionalarray ................285
CodeExample7‐5.Selectsourcehyperslabs.................................................288
CodeExample7‐6.Selectdestinationhyperslabs.............................................289
CodeExample7‐7.Writedatatoseparatepoints ............................................290
CodeExample7‐8.Createanarrayofregionreferences .......................................296
CodeExample7‐9.Writethearrayofreferencestoadataset..................................297
CodeExample7‐10.Readanarrayofregionreferences;readfromthefirstselection ..............298
CodeExample8‐1.Createalargeattributeindensestorage...................................316
CodeExample9‐1.Anerrorreport.........................................................324
CodeExample9‐2.Turnofferrormessageswhileprobingafunction............................325
CodeExample9‐3.Disableautomaticprintingandexplicitlyprinterrormessages .................326
CodeExample9‐4.Definingafunctiontoprintasimpleerrormessage ..........................326
CodeExample9‐5.Theuser‐definederrorhandler ...........................................326
CodeExample9‐6.Auser‐definedcallbackfunction..........................................328
CodeExample9‐7.Anerrorreport.........................................................329
CodeExample9‐8.Defininganerrorclass...................................................330
CodeExample9‐9.Createanerrorclassanderrormessages ...................................332
CodeExample9‐10.Closingerrormessagesandunregisteringtheerrorclass .....................332
CodeExample9‐11.Pushinganerrormessagetoanerrorstack................................334
CodeExample9‐12.Registeringtheerrorstack..............................................334
ListofCodeExamples HDF5User’sGuide
xxiv TheHDFGroup
HDF5User’sGuide ListofFunctionListings
TheHDFGroup xxv
ListofFunctionListings
FunctionListing3‐1.Generallibraryfunctionsandmacros(H5)..................................50
FunctionListing3‐2.Filefunctions(H5F) .....................................................51
FunctionListing3‐3.Filecreationpropertylistfunctions(H5P) ..................................53
FunctionListing3‐4.Fileaccesspropertylistfunctions(H5P) ....................................54
FunctionListing3‐5.Filedriverfunctions(H5P)...............................................55
FunctionListing4‐1.Groupfunctions(H5G)..................................................87
FunctionListing4‐2.Link(H5L)andobject(H5O)functions .....................................88
FunctionListing4‐3.Groupcreationpropertylistfunctions(H5P) ................................90
FunctionListing4‐4.Otherexternallinkfunctions .............................................91
FunctionListing5‐1.Datasetfunctions(H5D) ................................................105
FunctionListing5‐2.Datasetcreationpropertylistfunctions(H5P)..............................106
FunctionListing5‐3.Datasetaccesspropertylistfunctions(H5P) ...............................108
FunctionListing5‐4.Retrievedatasetinformation............................................121
FunctionListing5‐5.Datatransferpropertylistfunctions ......................................128
FunctionListing5‐6.Filedriverpropertylistfunctions ........................................128
FunctionListing6‐1.Generaldatatypeoperations ............................................186
FunctionListing6‐2.Conversionfunctions ..................................................187
FunctionListing6‐3.Atomicdatatypeproperties .............................................188
FunctionListing6‐4.Enumerationdatatypes ................................................189
FunctionListing6‐5.Compounddatatypeproperties .........................................190
FunctionListing6‐6.Arraydatatypes.......................................................191
FunctionListing6‐7.Variable‐lengthdatatypes ..............................................191
FunctionListing6‐8.Opaquedatatypes.....................................................191
FunctionListing6‐9.Conversionsbetweendatatypeandtext ..................................192
FunctionListing6‐10.Datatypecreationpropertylistfunctions(H5P) ...........................192
FunctionListing6‐11.Datatypeaccesspropertylistfunctions(H5P).............................192
FunctionListing7‐1.Dataspacemanagementfunctions .......................................265
FunctionListing7‐2.Dataspacequeryfunctions..............................................266
FunctionListing7‐3.Dataspaceselectionfunctions:hyperslabs.................................267
FunctionListing7‐4.Dataspaceselectionfunctions:points.....................................267
FunctionListing8‐1.Attributefunctions(H5A) ...............................................309
FunctionListing8‐2.Attributecreationpropertylistfunctions(H5P) ............................311
FunctionListing9‐1.Errorhandlingfunctions(H5E)...........................................322
FunctionListing10‐1.Generalpropertylistfunctions(H5P)....................................348
FunctionListing10‐2.Objectpropertyfunctions(H5P)........................................348
FunctionListing10‐3.Linkcreationpropertyfunctions(H5P) ...................................350
ListofFunctionListings HDF5User’sGuide
xxvi TheHDFGroup
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 1
1.TheHDF5DataModelandFile
Structure
1.1.Introduction
TheHierarchicalDataFormat(HDF)implementsamodelformanagingandstoringdata.Themodel
includesanabstractdatamodelandanabstractstoragemodel(thedataformat),andlibrariestoimple‐
menttheabstractmodelandtomapthestoragemodeltodifferentstoragemechanisms.TheHDF5
Libraryprovidesaprogramminginterfacetoaconcreteimplementationoftheabstractmodels.The
libraryalsoimplementsamodelofdatatransfer,anefficientmovementofdatafromonestoredrepresen‐
tationtoanotherstoredrepresentation.Thefigurebelowillustratestherelationshipsbetweenthemod‐
elsandimplementations.Thischapterexplainsthesemodelsindetail.
TheAbstractDataModelisaconceptualmodelofdata,datatypes,anddataorganization.Theabstract
datamodelisindependentofstoragemediumorprogrammingenvironment.TheStorageModelisastan‐
dardrepresentationfortheobjectsoftheabstractdatamodel.TheHDF5FileFormatSpecificationdefines
thestoragemodel.
Figure1‐1.HDF5modelsandimplementations
TheHDF5DataModelandFileStructure HDF5User’sGuide
2TheHDFGroup
TheProgrammingModelisamodelofthecomputingenvironmentandincludesplatformsfromsmallsin‐
glesystemstolargemultiprocessorsandclusters.Theprogrammingmodelmanipulates(instantiates,pop‐
ulates,andretrieves)objectsfromtheabstractdatamodel.
TheLibraryistheconcreteimplementationoftheprogrammingmodel.TheLibraryexportstheHDF5APIs
asitsinterface.Inadditiontoimplementingtheobjectsoftheabstractdatamodel,theLibrarymanages
datatransfersfromonestoredformtoanother.Datatransferexamplesincludereadingfromdisktomem‐
oryandwritingfrommemorytodisk.
StoredDataistheconcreteimplementationofthestoragemodel.Thestoragemodelismappedtoseveral
storagemechanismsincludingsinglediskfiles,multiplefiles(familyoffiles),andmemoryrepresentations.
TheHDF5LibraryisaCmodulethatimplementstheprogrammingmodelandabstractdatamodel.The
HDF5Librarycallstheoperatingsystemorotherstoragemanagementsoftware(forexample,theMPI/IO
Library)tostoreandretrievepersistentdata.TheHDF5Librarymayalsolinktoothersoftwaresuchasfil‐
tersforcompression.TheHDF5LibraryislinkedtoanapplicationprogramwhichmaybewritteninC,C++,
Fortran,orJava.Theapplicationprogramimplementsproblemspecificalgorithmsanddatastructuresand
callstheHDF5Librarytostoreandretrievedata.Thefigurebelowshowsthedependenciesofthesemod‐
ules.
Itisimportanttorealizethateachofthesoftwarecomponentsmanagesdatausingmodelsanddata
structuresthatareappropriatetothecomponent.Whendataispassedbetweenlayers(duringstorageor
Figure1‐2.Thelibrary,theapplicationprogram,andothermodules
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 3
retrieval),itistransformedfromonerepresentationtoanother.Thefigurebelowsuggestssomeofthe
kindsofdatastructuresusedinthedifferentlayers.
TheApplicationProgramusesdatastructuresthatrepresenttheproblemandalgorithmsincludingvari‐
ables,tables,arrays,andmeshesamongotherdatastructures.Dependingonitsdesignandfunction,an
applicationmayhavequiteafewdifferentkindsofdatastructuresanddifferentnumbersandsizesof
objects.
TheHDF5LibraryimplementstheobjectsoftheHDF5abstractdatamodel.Someoftheseobjectsinclude
groups,datasets,andattributes.Theapplicationprogrammapstheapplicationdatastructurestoahier‐
archyofHDF5objects.Eachapplicationwillcreateamappingbestsuitedtoitspurposes.
TheobjectsoftheHDF5abstractdatamodelaremappedtotheobjectsoftheHDF5storagemodel,and
storedinastoragemedium.Thestoredobjectsincludeheaderblocks,freelists,datablocks,B‐trees,and
otherobjects.Eachgroupordatasetisstoredasoneormoreheaderanddatablocks.SeetheHDF5File
FormatSpecificationformoreinformationonhowtheseobjectsareorganized.TheHDF5Librarycanalso
useotherlibrariesandmodulessuchascompression.
TheHDF5DataModelandFileStructure HDF5User’sGuide
4TheHDFGroup
Theimportantpointtonoteisthatthereisnotnecessarilyanysimplecorrespondencebetweenthe
objectsoftheapplicationprogram,theabstractdatamodel,andthoseoftheFormatSpecification.The
organizationofthedataofapplicationprogram,andhowitismappedtotheHDF5abstractdatamodelis
uptotheapplicationdeveloper.Theapplicationprogramonlyneedstodealwiththelibraryandthe
Figure1‐3.Datastructuresindifferentlayers
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 5
abstractdatamodel.MostapplicationsneednotconsideranydetailsoftheHDF5FileFormatSpecifica‐
tionorthedetailsofhowobjectsofabstractdatamodelaretranslatedtoandfromstorage.
1.2.TheAbstractDataModel
Theabstractdatamodel(ADM)definesconceptsfordefininganddescribingcomplexdatastoredinfiles.
TheADMisaverygeneralmodelwhichisdesignedtoconceptuallycovermanyspecificmodels.Manydif‐
ferentkindsofdatacanbemappedtoobjectsoftheADM,andthereforestoredandretrievedusingHDF5.
TheADMisnot,however,amodelofanyparticularproblemorapplicationdomain.Usersneedtomap
theirdatatotheconceptsoftheADM.
Thekeyconceptsinclude:
•File‐acontiguousstringofbytesinacomputerstore(memory,disk,etc.),andthebytesrepre‐
sentzeroormoreobjectsofthemodel
•Group‐acollectionofobjects(includinggroups)
•Dataset‐amultidimensionalarrayofdataelementswithattributesandothermetadata
•Dataspace‐adescriptionofthedimensionsofamultidimensionalarray
•Datatype‐adescriptionofaspecificclassofdataelementincludingitsstoragelayoutasapattern
ofbits
•Attribute‐anameddatavalueassociatedwithagroup,dataset,ornameddatatype
•PropertyList‐acollectionofparameters(somepermanentandsometransient)controlling
optionsinthelibrary
•Link‐thewayobjectsareconnected
Thesekeyconceptsaredescribedinmoredetailbelow.
1.2.1.File
Abstractly,anHDF5fileisacontainerforanorganizedcollectionofobjects.Theobjectsaregroups,data‐
sets,andotherobjectsasdefinedbelow.Theobjectsareorganizedasarooted,directedgraph.Every
HDF5filehasatleastoneobject,therootgroup.Seethefigurebelow.Allobjectsaremembersoftheroot
groupordescendantsoftherootgroup.
TheHDF5DataModelandFileStructure HDF5User’sGuide
6TheHDFGroup
HDF5objectshaveauniqueidentitywithinasingleHDF5fileandcanbeaccessedonlybytheirnames
withinthehierarchyofthefile.HDF5objectsindifferentfilesdonotnecessarilyhaveuniqueidentities,
anditisnotpossibletoaccessapermanentHDF5objectexceptthroughafile.Formoreinformation,see
"TheStructureofanHDF5File"onpage 16.
Whenthefileiscreated,thefilecreationpropertiesspecifysettingsforthefile.Thefilecreationproper‐
tiesincludeversioninformationandparametersofglobaldatastructures.Whenthefileisopened,thefile
accesspropertiesspecifysettingsforthecurrentaccesstothefile.Fileaccesspropertiesincludeparame‐
tersforstoragedriversandparametersforcachingandgarbagecollection.Thefilecreationpropertiesare
setpermanentlyforthelifeofthefile,andthefileaccesspropertiescanbechangedbyclosingand
reopeningthefile.
AnHDF5filecanbe“mounted”aspartofanotherHDF5file.ThisisanalogoustoUnixfilesystemmounts.
Therootofthemountedfileisattachedtoagroupinthemountingfile,andallthecontentscanbe
accessedasifthemountedfilewerepartofthemountingfile.
Figure1‐4.TheHDF5file
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 7
1.2.2.Group
AnHDF5groupisanalogoustoafilesystemdirectory.Abstractly,agroupcontainszeroormoreobjects,
andeveryobjectmustbeamemberofatleastonegroup.Therootgroupisaspecialcase;itmaynotbea
memberofanygroup.
Groupmembershipisactuallyimplementedvialinkobjects.Seethefigurebelow.Alinkobjectisowned
byagroupandpointstoanamedobject.Eachlinkhasaname,andeachlinkpointstoexactlyoneobject.
Eachnamedobjecthasatleastoneandpossiblymanylinkstoit.
Therearethreeclassesofnamedobjects:group,dataset,andcommitted(named)datatype.Seethefig‐
urebelow.Eachoftheseobjectsisthememberofatleastonegroup,andthismeansthereisatleastone
linktoit.
Figure1‐5.Groupmembershipvialinkobjects
TheHDF5DataModelandFileStructure HDF5User’sGuide
8TheHDFGroup
1.2.3.Dataset
AnHDF5datasetisamultidimensional(rectangular)arrayofdataelements.Seethefigurebelow.The
shapeofthearray(numberofdimensions,sizeofeachdimension)isdescribedbythedataspaceobject
(describedinthenextsectionbelow).
Adataelementisasingleunitofdatawhichmaybeanumber,acharacter,anarrayofnumbersorcharac‐
ters,orarecordofheterogeneousdataelements.Adataelementisasetofbits.Thelayoutofthebitsis
describedbythedatatype(seebelow).
Thedataspaceanddatatypearesetwhenthedatasetiscreated,andtheycannotbechangedforthelife
ofthedataset.Thedatasetcreationpropertiesaresetwhenthedatasetiscreated.Thedatasetcreation
propertiesincludethefillvalueandstoragepropertiessuchaschunkingandcompression.Theseproper‐
tiescannotbechangedafterthedatasetiscreated.
Thedatasetobjectmanagesthestorageandaccesstothedata.Whilethedataisconceptuallyacontigu‐
ousrectangulararray,itisphysicallystoredandtransferredindifferentwaysdependingonthestorage
propertiesandthestoragemechanismused.Theactualstoragemaybeasetofcompressedchunks,and
theaccessmaybethroughdifferentstoragemechanismsandcaches.Thedatasetmapsbetweenthecon‐
ceptualarrayofelementsandtheactualstoreddata.
Figure1‐6.Classesofnamedobjects
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 9
1.2.4.Dataspace
TheHDF5dataspacedescribesthelayoutoftheelementsofamultidimensionalarray.Conceptually,the
arrayisahyper‐rectanglewithoneto32dimensions.HDF5dataspacescanbeextendable.Therefore,
eachdimensionhasacurrentsizeandamaximumsize,andthemaximummaybeunlimited.The
dataspacedescribesthishyper‐rectangle:itisalistofdimensionswiththecurrentandmaximum(or
unlimited)sizes.Seethefigurebelow.
Figure1‐7.Thedataset
TheHDF5DataModelandFileStructure HDF5User’sGuide
10 TheHDFGroup
Dataspaceobjectsarealsousedtodescribehyperslabselectionsfromadataset.Anysubsetoftheele‐
mentsofadatasetcanbeselectedforreadorwritebyspecifyingasetofhyperslabs.Anon‐rectangular
regioncanbeselectedbytheunionofseveral(rectangular)dataspaces.
1.2.5.Datatype
TheHDF5datatypeobjectdescribesthelayoutofasingledataelement.Adataelementisasingleele‐
mentofthearray;itmaybeasinglenumber,acharacter,anarrayofnumbersorcarriers,orotherdata.
Thedatatypeobjectdescribesthestoragelayoutofthisdata.
Datatypesarecategorizedinto11classesofdatatype.Eachclassisinterpretedaccordingtoasetofrules
andhasaspecificsetofpropertiestodescribeitsstorage.Forinstance,floatingpointnumbershaveexpo‐
nentpositionandsizeswhichareinterpretedaccordingtoappropriatestandardsfornumberrepresenta‐
tion.Thus,thedatatypeclasstellswhattheelementmeans,andthedatatypedescribeshowitisstored.
Thefigurebelowshowstheclassificationofdatatypes.Atomicdatatypesareindivisible.Eachmaybea
singleobjectsuchasanumberorastring.Compositedatatypesarecomposedofmultipleelementsof
atomicdatatypes.Inadditiontothestandardtypes,userscandefineadditionaldatatypessuchasa24‐bit
integerora16‐bitfloat.
Adatasetorattributehasasingledatatypeobjectassociatedwithit.SeeFigure7above.Thedatatype
objectmaybeusedinthedefinitionofseveralobjects,butbydefault,acopyofthedatatypeobjectwill
beprivatetothedataset.
Optionally,adatatypeobjectcanbestoredintheHDF5file.Thedatatypeislinkedintoagroup,andthere‐
foregivenaname.Acommitteddatatype(formerlycalledanameddatatype)canbeopenedandusedin
anywaythatadatatypeobjectcanbeused.
Formoreinformation,see"HDF5Datatypes"onpage 173.
Figure1‐8.Thedataspace
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 11
1.2.6.Attribute
AnyHDF5nameddataobject(group,dataset,ornameddatatype)mayhavezeroormoreuserdefined
attributes.Attributesareusedtodocumenttheobject.Theattributesofanobjectarestoredwiththe
object.
AnHDF5attributehasanameanddata.Thedataportionissimilarinstructuretoadataset:adataspace
definesthelayoutofanarrayofdataelements,andadatatypedefinesthestoragelayoutandinterpreta‐
tionoftheelementsSeethefigurebelow.
Figure1‐9.Datatypeclassifications
TheHDF5DataModelandFileStructure HDF5User’sGuide
12 TheHDFGroup
Infact,anattributeisverysimilartoadatasetwiththefollowinglimitations:
•Anattributecanonlybeaccessedviatheobject
• Attributenamesaresignificantonlywithintheobject
•Anattributeshouldbeasmallobject
•Thedataofanattributemustbereadorwritteninasingleaccess(partialreadingorwritingisnot
allowed)
• Attributesdonothaveattributes
Notethatthevalueofanattributecanbeanobjectreference.Asharedattributeoranattributethatisa
largearraycanbeimplementedasareferencetoadataset.
Thename,dataspace,anddatatypeofanattributearespecifiedwhenitiscreatedandcannotbechanged
overthelifeoftheattribute.Anattributecanbeopenedbyname,byindex,orbyiteratingthroughallthe
attributesoftheobject.
Figure1‐10.Attributedataelements
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 13
1.2.7.PropertyList
HDF5hasagenericpropertylistobject.Eachlistisacollectionofname‐valuepairs.Eachclassofproperty
listhasaspecificsetofproperties.Eachpropertyhasanimplicitname,adatatype,andavalue.Seethe
figurebelow.ApropertylistobjectiscreatedandusedinwayssimilartotheotherobjectsoftheHDF5
Library.
PropertyListsareattachedtotheobjectinthelibrary,theycanbeusedbyanypartofthelibrary.Some
propertiesarepermanent(forexample,thechunkingstrategyforadataset),othersaretransient(for
example,buffersizesfordatatransfer).AcommonuseofaPropertyLististopassparametersfromthe
callingprogramtoaVFLdriveroramoduleofthepipeline.
Propertylistsareconceptuallysimilartoattributes.Propertylistsareinformationrelevanttothebehavior
ofthelibrarywhileattributesarerelevanttotheuser’sdataandapplication.
Propertylistsareusedtocontroloptionalbehaviorforfilecreation,fileaccess,datasetcreation,dataset
transfer(read,write),andfilemounting.Somepropertylistclassesareshowninthetablebelow.Details
ofthedifferentpropertylistsareexplainedintherelevantsectionsofthisdocument.
Figure1‐11.Thepropertylist
Table1‐1.Propertylistclassesandtheirusage
PropertyListClass Used Examples
H5P_FILE_CREATE Propertiesforfilecreation. Setsizeofuserblock.
H5P_FILE_ACCESS Propertiesforfileaccess. SetparametersforVFLdriver.
AnexampleisMPII/O.
TheHDF5DataModelandFileStructure HDF5User’sGuide
14 TheHDFGroup
1.2.8.Link
Thissectionisunderconstruction.
1.3.TheHDF5StorageModel
1.3.1.TheAbstractStorageModel:theHDF5FormatSpecification
TheHDF5FileFormatSpecificationdefineshowHDF5objectsanddataaremappedtoalinearaddress
space.Theaddressspaceisassumedtobeacontiguousarrayofbytesstoredonsomerandomaccess
medium.1Theformatdefinesthestandardforhowtheobjectsoftheabstractdatamodelaremappedto
linearaddresses.Thestoredrepresentationisself‐describinginthesensethattheformatdefinesallthe
informationnecessarytoreadandreconstructtheoriginalobjectsoftheabstractdatamodel.
TheHDF5FileFormatSpecificationisorganizedinthreeparts:
1. Level0:Filesignatureandsuperblock
2. Level1:Fileinfrastructure
a. Level1A:B‐linktreesandB‐treenodes
b. Level1B:Group
c. Level1C:Groupentry
d. Level1D:Localheaps
e. Level1E:Globalheap
f. Level1F:Free‐spaceindex
H5P_DATASET_CREATE Propertiesfordatasetcre‐
ation.
Setchunking,compression,
orfillvalue.
H5P_DATASET_XFER Propertiesforrawdatatrans‐
fer(readandwrite).
Tunebuffersizesormemory
management.
H5P_FILE_MOUNT Propertiesforfilemounting.
1. HDF5requiresrandomaccesstothelinearaddressspace.Forthisreasonitisnotwellsuitedforsome
datamediasuchasstreams.
Table1‐1.Propertylistclassesandtheirusage
PropertyListClass Used Examples
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 15
3. Level2:Dataobject
a. Level2A:Dataobjectheaders
b. Level2B:Shareddataobjectheaders
c. Level2C:Dataobjectdatastorage
TheLevel0specificationdefinestheheaderblockforthefile.Headerblockelementsincludeasignature,
versioninformation,keyparametersofthefilelayout(suchaswhichVFLfiledriversareneeded),and
pointerstotherestofthefile.Level1definesthedatastructuresusedthroughoutthefile:theB‐trees,
heaps,andgroups.Level2definesthedatastructureforstoringthedataobjectsanddata.Inallcases,the
datastructuresarecompletelyspecifiedsothateverybitinthefilecanbefaithfullyinterpreted.
ItisimportanttorealizethatthestructuresdefinedintheHDF5fileformatarenotthesameasthe
abstractdatamodel:theobjectheaders,heaps,andB‐treesofthefilespecificationarenotrepresentedin
theabstractdatamodel.Theformatdefinesanumberofobjectsformanagingthestorageincluding
headerblocks,B‐trees,andheaps.TheHDF5FileFormatSpecificationdefineshowtheabstractobjects
(forexample,groupsanddatasets)arerepresentedasheaders,B‐treeblocks,andotherelements.
TheHDF5LibraryimplementsoperationstowriteHDF5objectstothelinearformatandtoreadfromthe
linearformattocreateHDF5objects.ItisimportanttorealizethatasingleHDF5abstractobjectisusually
storedasseveralobjects.Adataset,forexample,mightbestoredinaheaderandinoneormoredata
blocks,andtheseobjectsmightnotbecontiguousontheharddisk.
1.3.2.ConcreteStorageModel
TheHDF5fileformatdefinesanabstractlinearaddressspace.Thiscanbeimplementedindifferentstor‐
agemediasuchasasinglefileormultiplefilesondiskorinmemory.TheHDF5Librarydefinesanopen
interfacecalledtheVirtualFileLayer(VFL).TheVFLallowsdifferentconcretestoragemodelstobe
selected.
TheVFLdefinesanabstractmodel,anAPIforrandomaccessstorage,andanAPItopluginalternativeVFL
drivermodules.ThemodeldefinestheoperationsthattheVFLdrivermustandmaysupport,andthe
plug‐inAPIenablestheHDF5Librarytorecognizethedriverandpassitcontrolanddata.
AnumberofVFLdrivershavebeendefinedintheHDF5Library.Someworkwithasinglefile,andsome
workwithmultiplefilessplitinvariousways.Someworkinserialcomputingenvironments,andsome
workinparallelcomputingenvironments.MostworkwithdiskcopiesofHDF5files,butoneworkswitha
memorycopy.Thesedriversarelistedinthe“Supportedfiledrivers”table.Formoreinformation,see
"AlternateFileStorageLayoutsandLow‐levelFileDrivers"onpage 61.
EachdriverisolatesthedetailsofreadingandwritingstoragesothattherestoftheHDF5Libraryanduser
programcanbealmostthesamefordifferentstoragemethods.TheexceptiontothisruleisthatsomeVFL
driversneedinformationfromthecallingapplication.Thisinformationispassedusingpropertylists.For
example,theParalleldriverrequirescertaincontrolinformationthatmustbeprovidedbytheapplication.
TheHDF5DataModelandFileStructure HDF5User’sGuide
16 TheHDFGroup
1.4.TheStructureofanHDF5File
1.4.1.OverallFileStructure
AnHDF5fileisorganizedasarooted,directedgraph.Nameddataobjectsarethenodesofthegraph,and
linksarethedirectedarcs.Eacharcofthegraphhasaname,andtherootgrouphasthename“/”.Objects
arecreatedandtheninsertedintothegraphwiththelinkoperationwhichcreatesanamedlinkfroma
grouptotheobject.Forexample,thefigurebelowillustratesthestructureofanHDF5filewhenonedata‐
setiscreated.Anobjectcanbethetargetofmorethanonelink.Thenamesonthelinksmustbeunique
withineachgroup,buttheremaybemanylinkswiththesamenameindifferentgroups.Linknamesare
unambiguous:someancestorwillhaveadifferentname,ortheyarethesameobject.Thegraphisnavi‐
gatedwithpathnamessimilartoUnixfilesystems.Anobjectcanbeopenedwithafullpathstartingatthe
rootgrouporwitharelativepathandastartingnode(group).Notethatallpathsarerelativetoasingle
HDF5file.Inthissense,anHDF5fileisanalogoustoasingleUnixfilesystem.2
2. ItcouldbesaidthatHDF5extendstheorganizingconceptsofafilesystemtotheinternalstructureofa
singlefile.
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 17
Note:Inthefigureabovearetwofigures.Thetopfigurerepresentsanewlycreatedfilewithonegroup,/.Inthebot‐
tomfigure,adatasetcalled/dset1hasbeencreated.
Itisimportanttonotethat,justliketheUnixfilesystem,HDF5objectsdonothavenames.Thenamesare
associatedwithpaths.Anobjecthasaunique(withinthefile)objectidentifier,butasingleobjectmay
havemanynamesbecausetheremaybemanypathstothesameobject.Anobjectcanberenamed
(movedtoanothergroup)byaddinganddeletinglinks.Inthiscase,theobjectitselfnevermoves.Forthat
matter,membershipinagrouphasnoimplicationforthephysicallocationofthestoredobject.
Deletingalinktoanobjectdoesnotnecessarilydeletetheobject.Theobjectremainsavailableaslongas
thereisatleastonelinktoit.Afterallthelinkstoanobjectaredeleted,itcannolongerbeopened
althoughthestoragemayormaynotbereclaimed.3
Itisimportanttorealizethatthelinkingmechanismcanbeusedtoconstructverycomplexgraphsof
objects.Forexample,itispossibleforanobjecttobesharedbetweenseveralgroupsandeventohave
morethanonenameinthesamegroup.Itisalsopossibleforagrouptobeamemberofitselfortobein
a“cycle”inthegraph.Anexampleofacycleiswhereachildistheparentofoneofitsownancestors.
Figure1‐12.AnHDF5filewithonedataset
3. AsofHDF5‐1.4,thestorageusedforanobjectisreclaimed,evenifalllinksaredeleted.
TheHDF5DataModelandFileStructure HDF5User’sGuide
18 TheHDFGroup
1.4.2.HDF5PathNamesandNavigation
Thestructureofthefileconstitutesthenamespacefortheobjectsinthefile.Apathnameisastringof
componentsseparatedby‘/’.Eachcomponentisthenameofalinkorthespecialcharacter“.”forthecur‐
rentgroup.Linknames(components)canbeanystringofASCIIcharactersnotcontaining‘/’(exceptthe
string“.” whichisreserved).However,usersareadvisedtoavoidtheuseofpunctuationandnon‐printing
charactersbecausetheymaycreateproblemsforothersoftware.ThefigurebelowgivesaBNFgrammar
forHDF5pathnames.
Anobjectcanalwaysbeaddressedbyafullorabsolutepathwhichwouldstartattherootgroup.As
alreadynoted,agivenobjectcanhavemorethanonefullpathname.Anobjectcanalsobeaddressedby
arelativepathwhichwouldstartatagroupandincludethepathtotheobject.
ThestructureofanHDF5fileis“self‐describing.”Thismeansthatitispossibletonavigatethefiletodis‐
coveralltheobjectsinthefile.Basically,thestructureistraversedasagraphstartingatonenodeand
recursivelyvisitingthenodesofthegraph.
1.4.3.ExamplesofHDF5FileStructures
ThefiguresbelowshowsomepossibleHDF5filestructureswithgroupsanddatasets.Thefirstfigure
showsthestructureofafilewiththreegroups.Thesecondshowsadatasetcreatedin“/group1”.Thethird
figureshowsthestructureafteradatasetcalleddset2hasbeenaddedtotherootgroup.Thefourthfigure
showsthestructureafteranothergroupanddatasethavebeenadded.
PathName ::= AbsolutePathName | RelativePathName
Separator ::= "/" ["/"]*
AbsolutePathName ::= Separator [ RelativePathName ]
RelativePathName ::= Component [ Separator RelativePathName ]*
Component ::= "." | Name
Name ::= Character+ - {"."}
Character ::= {c: c in {{ legal ASCII characters } - {'/'}}
Figure1‐13.ABNFgrammarforpathnames
HDF5User’sGuide TheHDF5DataModelandFileStructure
TheHDFGroup 19
Note:Thefigureaboveshowsthreegroups;/group1and/group2aremembersoftherootgroup.
Note:Thefigureaboveshowsthatadatasethasbeencreatedin/group1:/group1/dset1.
Figure1‐14.AnHDF5filestructurewithgroups
Figure1‐15.AnHDF5filestructurewithgroupsandadataset
TheHDF5DataModelandFileStructure HDF5User’sGuide
20 TheHDFGroup
Note:Inthefigureabove,anotherdatasethasbeenaddedasamemberoftherootgroup:/dset2.
Note:Inthefigureabove,anothergroupanddatasethavebeenaddedreusingobjectnames:/group2/group2/dset2.
Figure1‐16.AnHDF5filestructurewithgroupsanddatasets
Figure1‐17.AnotHDF5filestructurewithgroupsanddatasets
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 21
2.TheHDF5LibraryandProgramming
Model
2.1.Introduction
TheHDF5LibraryimplementstheHDF5abstractdatamodelandstoragemodel.Thesemodelswere
describedintheprecedingchapter.
TwomajorobjectivesoftheHDF5productsaretoprovidetoolsthatcanbeusedonasmanycomputa‐
tionalplatformsaspossible(portability),andtoprovideareasonablyobject‐orienteddatamodelandpro‐
gramminginterface.
Tobeasportableaspossible,theHDF5LibraryisimplementedinportableC.Cisnotanobject‐oriented
language,butthelibraryusesseveralmechanismsandconventionstoimplementanobjectmodel.
OnemechanismtheHDF5Libraryusesistoimplementtheobjectsasdatastructures.Torefertoan
object,theHDF5Libraryimplementsitsownpointers.Thesepointersarecalledidentifiers.Anidentifieris
thenusedtoinvokeoperationsonaspecificinstanceofanobject.Forexample,whenagroupisopened,
theAPIreturnsagroupidentifier.Thisidentifierisareferencetothatspecificgroupandwillbeusedto
invokefutureoperationsonthatgroup.Theidentifierisvalidonlywithinthecontextitiscreatedand
remainsvaliduntilitisclosedorthefileisclosed.Thismechanismisessentiallythesameasthemecha‐
nismthatC++orotherobject‐orientedlanguagesusetorefertoobjectsexceptthatthesyntaxisC.
Similarly,object‐orientedlanguagescollectallthemethodsforanobjectinasinglenamespace.Anexam‐
pleisthemethodsofaC++class.TheClanguagedoesnothaveanysuchmechanism,buttheHDF5Library
simulatesthisthroughitsAPInamingconvention.APIfunctionnamesbeginwithacommonprefixthatis
relatedtotheclassofobjectsthatthefunctionoperateson.ThetablebelowliststheHDF5objectsand
thestandardprefixesusedbythecorrespondingHDF5APIs.Forexample,functionsthatoperateondata‐
typeobjectsallhavenamesbeginningwithH5T.
Table2‐1.TheHDF5APInamingscheme
Prefix Operateson
H5A Attributes
H5D Datasets
H5E Errorreports
H5F Files
H5G Groups
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
22 TheHDFGroup
2.2.TheHDF5ProgrammingModel
InthissectionweintroducetheHDF5programmingmodelbymeansofaseriesofshortcodesamples.
ThesesamplesillustrateabroadselectionofcommonHDF5tasks.Moredetailsareprovidedinthefollow‐
ingchaptersandintheHDF5ReferenceManual.
2.2.1.CreatinganHDF5File
BeforeanHDF5filecanbeusedorreferredtoinanymanner,itmustbeexplicitlycreatedoropened.
Whentheneedforaccesstoafileends,thefilemustbeclosed.TheexamplebelowprovidesaCcode
fragmentillustratingthesesteps.Inthisexample,thevaluesforthefilecreationpropertylistandthefile
accesspropertylistaresettothedefaultsH5P_DEFAULT.
H5I Identifiers
H5L Links
H5O Objects
H5P Propertylists
H5R References
H5S Dataspaces
H5T Datatypes
H5Z Filters
Table2‐1.TheHDF5APInamingscheme
Prefix Operateson
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 23
Note:Ifthereisapossibilitythatafileofthedeclarednamealreadyexistsandyouwishtoopenanewfile
regardlessofthatpossibility,theflagH5F_ACC_TRUNCwillcausetheoperationtooverwritetheprevious
file.Iftheoperationshouldfailinsuchacircumstance,usetheflagH5F_ACC_EXCLinstead.
2.2.2.CreatingandInitializingaDataset
Theessentialobjectswithinadatasetaredatatypeanddataspace.Theseareindependentobjectsandare
createdseparatelyfromanydatasettowhichtheymaybeattached.Hence,creatingadatasetrequires,at
aminimum,thefollowingsteps:
1. Createandinitializeadataspaceforthedataset
2. Defineadatatypeforthedataset
3. Createandinitializethedataset
Thecodeintheexamplebelowillustratestheexecutionofthesesteps.
hid_t file; /* declare file identifier */
/*
* Create a new file using H5F_ACC_TRUNC
* to truncate and overwrite any file of the same name,
* default file creation properties, and
* default file access properties.
* Then close the file.
*/
file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
status = H5Fclose(file);
CodeExample2‐1.CreatingandclosinganHDF5file
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
24 TheHDFGroup
2.2.3.ClosinganObject
Anapplicationshouldcloseanobjectsuchasadatatype,dataspace,ordatasetoncetheobjectisnolon‐
gerneeded.Sinceeachisanindependentobject,eachmustbereleased(orclosed)separately.Thisaction
isfrequentlyreferredtoasreleasingtheobject’sidentifier.Thecodeintheexamplebelowclosesthe
datatype,dataspace,anddatasetthatwerecreatedintheprecedingsection.
hid_t dataset, datatype, dataspace; /* declare identifiers */
/*
* Create a dataspace: Describe the size of the array and
* create the dataspace for a fixed-size dataset.
*/
dimsf[0] = NX;
dimsf[1] = NY;
dataspace = H5Screate_simple(RANK, dimsf, NULL);
/*
* Define a datatype for the data in the dataset.
* We will store little endian integers.
*/
datatype = H5Tcopy(H5T_NATIVE_INT);
status = H5Tset_order(datatype, H5T_ORDER_LE);
/*
* Create a new dataset within the file using the defined
* dataspace and datatype and default dataset creation
* properties.
* NOTE: H5T_NATIVE_INT can be used as the datatype if
* conversion to little endian is not needed.
*/
dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
CodeExample2‐2.Createadataset
H5Tclose(datatype);
H5Dclose(dataset);
H5Sclose(dataspace);
CodeExample2‐3.Closeanobject
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 25
ThereisalonglistofHDF5Libraryitemsthatreturnauniqueidentifierwhentheitemiscreatedor
opened.Eachtimethatoneoftheseitemsisopened,auniqueidentifierisreturned.Closingafiledoes
notmeanthatthegroups,datasets,orotheropenitemsarealsoclosed.Eachopeneditemmustbeclosed
separately.
Formoreinformation,see“UsingIdentifiers”inthe“AdvancedTopics”page.
HowClosingaFileEffectsOtherOpenStructuralElements
EverystructuralelementinanHDF5filecanbeopened,andtheseelementscanbeopenedmorethan
once.Elementsrangeinsizefromtheentirefiledowntoattributes.Whenanelementisopened,the
HDF5Libraryreturnsauniqueidentifiertotheapplication.Everyelementthatisopenedmustbeclosed.
Ifanelementwasopenedmorethanonce,eachidentifierthatwasreturnedtotheapplicationmustbe
closed.Forexample,ifadatasetwasopenedtwice,bothdatasetidentifiersmustbereleased(closed)
beforethedatasetcanbeconsideredclosed.Supposeanapplicationhasopenedafile,agroupinthefile,
andtwodatasetsinthegroup.Inorderforthefiletobetotallyclosed,thefile,group,anddatasetsmust
eachbeclosed.Closingthefilebeforethegrouporthedatasetswillnoteffectthestateofthegroupor
datasets:thegroupanddatasetswillstillbeopen.
Thereareseveralexceptionstotheabovegeneralrule.OneiswhentheH5closefunctionisused.
H5closecausesageneralshutdownofthelibrary:alldataiswrittentodisk,allidentifiersareclosed,and
allmemoryusedbythelibraryiscleanedup.Anotherexceptionoccursonparallelprocessingsystems.
Supposeonaparallelsystemanapplicationhasopenedafile,agroupinthefile,andtwodatasetsinthe
group.IftheapplicationusestheH5Fclosefunctiontoclosethefile,thecallwillfailwithanerror.The
opengroupanddatasetsmustbeclosedbeforethefilecanbeclosed.Athirdexceptioniswhenthefile
accesspropertylistincludesthepropertyH5F_CLOSE_STRONG.Thispropertyclosesanyopenelements
whenthefileisclosedwithH5Fclose.Formoreinformation,seetheH5Pset_fclose_degreefunc‐
tionintheHDF5ReferenceManual.
2.2.4.WritingorReadingaDatasettoorfromaFile
Havingcreatedthedataset,theactualdatacanbewrittenwithacalltoH5Dwrite.Seetheexample
below.
/*
* Write the data to the dataset using default transfer
* properties.
*/
status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
CodeExample2‐4.Writingadataset
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
26 TheHDFGroup
NotethatthethirdandfourthH5Dwriteparametersintheaboveexampledescribethedataspacesin
memoryandinthefile,respectively.Fornow,thesearebothsettoH5S_ALLwhichindicatesthatthe
entiredatasetistobewritten.Theselectionofpartialdatasetsandtheuseofdifferingdataspacesin
memoryandinstoragewillbediscussedlaterinthischapterandinmoredetailelsewhereinthisguide.
Readingthedatasetfromstorageissimilartowritingthedatasettostorage.Toreadanentiredataset,
substituteH5DreadforH5Dwriteintheaboveexample.
2.2.5.ReadingandWritingaPortionofaDataset
Theprevioussectiondescribedwritingorreadinganentiredataset.HDF5alsosupportsaccesstoportions
ofadataset.Thesepartsofdatasetsareknownasselections.
Thesimplesttypeofselectionisasimplehyperslab.Thisisann‐dimensionalrectangularsub‐setofa
datasetwherenisequaltothedataset’srank.Otheravailableselectionsincludeamorecomplexhyper‐
slabwithuser‐definedstrideandblocksize,alistofindependentpoints,ortheunionofanyofthese.
Thefigurebelowshowsseveralsampleselections.
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 27
Note:Inthefigureabove,selectionscantaketheformofasimplehyperslab,ahyperslabwithuser‐definedstrideand
block,aselectionofpoints,oraunionofanyoftheseforms.
Figure2‐1.Datasetselections
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
28 TheHDFGroup
Selectionsandhyperslabsareportionsofadataset.Asdescribedabove,asimplehyperslabisarectangu‐
lararrayofdataelementswiththesamerankasthedataset’sdataspace.Thus,asimplehyperslabisalog‐
icallycontiguouscollectionofpointswithinthedataset.
Themoregeneralcaseofahyperslabcanalsobearegularpatternofpointsorblockswithinthe
dataspace.Fourparametersarerequiredtodescribeageneralhyperslab:thestartingcoordinates,the
blocksize,thestrideorspacebetweenblocks,andthenumberofblocks.Theseparametersareeach
expressedasaone‐dimensionalarraywithlengthequaltotherankofthedataspaceandaredescribedin
thetablebelow.
ReadingDataintoaDifferentlyShapedMemoryBlock
Formaximumflexibilityinuserapplications,aselectioninstoragecanbemappedintoadifferently‐
shapedselectioninmemory.Allthatisrequiredisthatthetwoselectionscontainthesamenumberof
dataelements.Inthisexample,wewillfirstdefinetheselectiontobereadfromthedatasetinstorage,
andthenwewilldefinetheselectionasitwillappearinapplicationmemory.
Supposewewanttoreada3x4hyperslabfromatwo‐dimensionaldatasetinafilebeginningatthedata‐
setelement<1,2>.Thefirsttaskistocreatethedataspacethatdescribestheoverallrankanddimensions
ofthedatasetinthefileandtospecifythepositionandsizeofthein‐filehyperslabthatweareextracting
fromthatdataset.Seethecodebelow.
Table2‐2.Hyperslabparameters
Parameter Definition
start Thecoordinatesofthestartinglocationofthehyperslabinthedataset’s
dataspace.
block Thesizeofeachblocktobeselectedfromthedataspace.Iftheblockparam‐
eterissettoNULL,theblocksizedefaultstoasingleelementineachdimen‐
sion,asiftheblockarraywassettoall1s(allones).Thiswillresultinthe
selectionofauniformlyspacedsetofcountpointsstartingatstartandon
theintervaldefinedbystride.
stride Thenumberofelementsseparatingthestartingpointofeachelementor
blocktobeselected.IfthestrideparameterissettoNULL,thestridesize
defaultsto1(one)ineachdimensionandnoelementsareskipped.
count Thenumberofelementsorblockstoselectalongeachdimension.
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 29
Thenexttaskistodefineadataspaceinmemory.Supposethatwehaveinmemoryathree‐dimensional7
x7x3arrayintowhichwewishtoreadthetwo‐dimensional3x4hyperslabdescribedaboveandthatwe
wantthememoryselectiontobeginattheelement<3,0,0>andresideintheplaneofthefirsttwodimen‐
sionsofthearray.Sincethein‐memorydataspaceisthree‐dimensional,wehavetodescribethein‐mem‐
oryselectionasthree‐dimensional.Sincewearekeepingtheselectionintheplaneofthefirsttwo
dimensionsofthein‐memorydataset,thein‐memoryselectionwillbea3x4x1arraydefinedas<3,4,1>.
Noticethatwemustdescribetwothings:thedimensionsofthein‐memoryarray,andthesizeandposi‐
tionofthehyperslabthatwewishtoreadin.Thecodebelowillustrateshowthiswouldbedone.
/*
* Define dataset dataspace in file.
*/
dataspace = H5Dget_space(dataset); /* dataspace identifier */
rank = H5Sget_simple_extent_ndims(dataspace);
status_n = H5Sget_simple_extent_dims(dataspace, dims_out, NULL);
/*
* Define hyperslab in the dataset.
*/
offset[0] = 1;
offset[1] = 2;
count[0] = 3;
count[1] = 4;
status = H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, offset,
NULL, count, NULL);
CodeExample2‐5.Definetheselectiontobereadfromstorage
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
30 TheHDFGroup
Thehyperslabdefinedinthecodeabovehasthefollowingparameters:start=(3,0,0),
count=(3,4,1),strideandblocksizeareNULL.
WritingDataintoaDifferentlyShapedDiskStorageBlock
Nowlet’sconsidertheoppositeprocessofwritingaselectionfrommemorytoaselectioninadatasetin
afile.Supposethatthesourcedataspaceinmemoryisa50‐element,one‐dimensionalarraycalledvec-
torandthatthesourceselectionisa48‐elementsimplehyperslabthatstartsatthesecondelementof
vector.Seethefigurebelow.
Furthersupposethatwewishtowritethisdatatothefileasaseriesof3x2‐elementblocksinatwo‐
dimensionaldataset,skippingonerowandonecolumnbetweenblocks.Sincethesourceselectioncon‐
tains48dataelementsandeachblockinthedestinationselectioncontains6dataelements,wemust
/*
* Define memory dataspace.
*/
dimsm[0] = 7;
dimsm[1] = 7;
dimsm[2] = 3;
memspace = H5Screate_simple(RANK_OUT,dimsm,NULL);
/*
* Define memory hyperslab.
*/
offset_out[0] = 3;
offset_out[1] = 0;
offset_out[2] = 0;
count_out[0] = 3;
count_out[1] = 4;
count_out[2] = 1;
status = H5Sselect_hyperslab(memspace, H5S_SELECT_SET,
offset_out, NULL, count_out, NULL);
CodeExample2‐6.Definethememorydataspaceandselection
Figure2‐2.Aone‐dimensionalarray
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 31
definethedestinationselectionwith8blocks.Wewillwrite2blocksinthefirstdimensionand4inthe
second.Thecodebelowshowshowtoachievethisobjective.
/* Select the hyperslab for the dataset in the file, using
* 3 x 2 blocks, a (4,3) stride, a (2,4) count, and starting
* at the position (0,1).
*/
start[0] = 0; start[1] = 1;
stride[0] = 4; stride[1] = 3;
count[0] = 2; count[1] = 4;
block[0] = 3; block[1] = 2;
ret = H5Sselect_hyperslab(fid, H5S_SELECT_SET, start, stride,
count, block);
/*
* Create dataspace for the first dataset.
*/
mid1 = H5Screate_simple(MSPACE1_RANK, dim1, NULL);
/*
* Select hyperslab.
* We will use 48 elements of the vector buffer starting at the
* second element. Selected elements are 1 2 3 . . . 48
*/
start[0] = 1;
stride[0] = 1;
count[0] = 48;
block[0] = 1;
ret = H5Sselect_hyperslab(mid1, H5S_SELECT_SET, start, stride,
count, block);
/*
* Write selection from the vector buffer to the dataset in the
* file.
*
ret = H5Dwrite(dataset, H5T_NATIVE_INT, mid1, fid, H5P_DEFAULT,
vector)
CodeExample2‐7.Thedestinationselection
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
32 TheHDFGroup
2.2.6.GettingInformationaboutaDataset
Althoughreadingisanalogoustowriting,itisoftenfirstnecessarytoqueryafiletoobtaininformation
aboutthedatasettoberead.Forinstance,weoftenneedtodeterminethedatatypeassociatedwitha
dataset,oritsdataspace(inotherwords,rankanddimensions).Asillustratedinthecodeexamplebelow,
thereareseveralgetroutinesforobtainingthisinformation.
2.2.7.CreatingandDefiningCompoundDatatypes
Acompounddatatypeisacollectionofoneormoredataelements.Eachelementmightbeanatomic
type,asmallarray,oranothercompounddatatype.
Theprovisionfornestedcompounddatatypesallowsthesestructurestobecomequitecomplex.AnHDF5
compounddatatypehassomesimilaritiestoaCstructoraFortrancommonblock.Thoughnotoriginally
designedwithdatabasesinmind,HDF5compounddatatypesaresometimesusedinawaythatissimilar
toadatabaserecord.Compounddatatypescanbecomeeitherapowerfultooloracomplexanddifficult‐
to‐debugconstruct.Reasonablecautionisadvised.
/*
* Get datatype and dataspace identifiers,
* then query datatype class, order, and size, and
* then query dataspace rank and dimensions.
*/
datatype = H5Dget_type (dataset); /* datatype identifier */
class = H5Tget_class (datatype);
if (class == H5T_INTEGER) printf("Dataset has INTEGER type \n");
order = H5Tget_order (datatype);
if (order == H5T_ORDER_LE) printf("Little endian order \n");
size = H5Tget_size (datatype);
printf ("Size is %d \n", size);
dataspace = H5Dget_space (dataset); /* dataspace identifier */
/* Find rank and retrieve current and maximum dimension
* sizes.
*/
rank = H5Sget_simple_extent_dims (dataspace, dims, max_dims);
CodeExample2‐8.Routinestogetdatasetparameters
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 33
Tocreateanduseacompounddatatype,youneedtocreateadatatypewithclasscompound(H5T_COM-
POUND)andspecifythetotalsizeofthedataelementinbytes.Acompounddatatypeconsistsofzeroor
moreuniquelynamedmembers.Memberscanbedefinedinanyorderbutmustoccupynon‐overlapping
regionswithinthedatum.Thetablebelowliststhepropertiesofcompounddatatypemembers.
Propertiesofthemembersofacompounddatatypearedefinedwhenthememberisaddedtothecom‐
poundtype.Thesepropertiescannotbemodifiedlater.
DefiningCompoundDatatypes
Compounddatatypesmustbebuiltoutofotherdatatypes.Todothis,youfirstcreateanemptycom‐
pounddatatypeandspecifyitstotalsize.Membersarethenaddedtothecompounddatatypeinany
order.
Eachmembermusthaveadescriptivename.Thisisthekeyusedtouniquelyidentifythememberwithin
thecompounddatatype.AmembernameinanHDF5datatypedoesnotnecessarilyhavetobethesame
asthenameofthecorrespondingmemberintheCstructinmemoryalthoughthisisoftenthecase.You
alsodonotneedtodefineallthemembersoftheCstructintheHDF5compounddatatype(orviceversa).
UsuallyaCstructwillbedefinedtoholdadatapointinmemory,andtheoffsetsofthemembersinmem‐
orywillbetheoffsetsofthestructmembersfromthebeginningofaninstanceofthestruct.Thelibrary
definesthemacrothatcomputestheoffsetofmembermwithinastructvariables:
HOFFSET(s,m)
Thecodebelowshowsanexampleinwhichacompounddatatypeiscreatedtodescribecomplexnum‐
berswhosetypeisdefinedbythecomplex_tstruct.
Table2‐3.Compounddatatypememberproperties
Parameter Definition
Index AnindexnumberbetweenzeroandN‐1,whereNisthenumberof
membersinthecompound.Theelementsareindexedintheorderof
theirlocationinthearrayofbytes.
Name Astringthatmustbeuniquewithinthemembersofthesamedatatype.
Datatype AnHDF5datatype.
Offset Afixedbyteoffsetwhichdefinesthelocationofthefirstbyteofthat
memberinthecompounddatatype.
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
34 TheHDFGroup
2.2.8.CreatingandWritingExtendableDatasets
Anextendabledatasetisonewhosedimensionscangrow.OnecandefineanHDF5datasettohavecertain
initialdimensionswiththecapacitytolaterincreasethesizeofanyoftheinitialdimensions.Forexample,
thefigurebelowshowsa3x3dataset(a)whichislaterextendedtobea10x3datasetbyadding7rows
(b),andfurtherextendedtobea10x5datasetbyaddingtwocolumns(c).
Typedef struct {
double re; /*real part */
double im; /*imaginary part */
} complex_t;
complex_t tmp; /*used only to compute offsets */
hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);
H5Tinsert (complex_id, "real", HOFFSET(tmp,re),
H5T_NATIVE_DOUBLE);
H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im),
H5T_NATIVE_DOUBLE);
CodeExample2‐9.Acompounddatatypeforcomplexnumbers
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 35
HDF5requirestheuseofchunkingwhendefiningextendabledatasets.Chunkingmakesitpossibleto
extenddatasetsefficientlywithouthavingtoreorganizecontiguousstorageexcessively.
Tosummarize,anextendabledatasetrequirestwoconditions:
1. Definethedataspaceofthedatasetasunlimitedinalldimensionsthatmighteventuallybe
extended
2. Enablechunkinginthedatasetcreationproperties
Forexample,supposewewishtocreateadatasetsimilartotheoneshowninthefigureabove.Wewant
tostartwitha3x3dataset,andthenlaterwewillextendit.Todothis,gothroughthestepsbelow.
First,declarethedataspacetohaveunlimiteddimensions.Seethecodeshownbelow.Notetheuseofthe
predefinedconstantH5S_UNLIMITEDtospecifythatadimensionisunlimited.
Figure2‐3.Extendingadataset
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
36 TheHDFGroup
Next,setthedatasetcreationpropertylisttoenablechunking.Seethecodebelow.
Thenextstepistocreatethedataset.Seethecodebelow.
Finally,whenthetimecomestoextendthesizeofthedataset,invokeH5Dextend.Extendingthedataset
alongthefirstdimensionbysevenrowsleavesthedatasetwithnewdimensionsof<10,3>.Seethecode
below.
/* dataset dimensions at creation time */
Hsize_t dims[2] = {3, 3};
hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};
/*
* Create the data space with unlimited dimensions.
*/
dataspace = H5Screate_simple(RANK, dims, maxdims);
CodeExample2‐10.Declaringadataspacewithunlimiteddimensions
hid_t cparms;
hsize_t chunk_dims[2] ={2, 5};
/*
* Modify dataset creation properties to enable chunking.
*/
cparms = H5Pcreate (H5P_DATASET_CREATE);
status = H5Pset_chunk(cparms, RANK, chunk_dims);
CodeExample2‐11.Enablechunking
/*
* Create a new dataset within the file using cparms
* creation properties.
*/
dataset = H5Dcreate(file, DATASETNAME, H5T_NATIVE_INT, dataspace,
H5P_DEFAULT, cparms, H5P_DEFAULT);
CodeExample2‐12.Createadataset
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 37
2.2.9.CreatingandWorkingwithGroups
GroupsprovideamechanismfororganizingmeaningfulandextendablesetsofdatasetswithinanHDF5
file.TheH5GAPIprovidesseveralroutinesforworkingwithgroups.
CreatingaGroup
Withnodatatype,dataspace,orstoragelayouttodefine,creatingagroupisconsiderablysimplerthan
creatingadataset.Forexample,thefollowingcodecreatesagroupcalledDataintherootgroupoffile.
Agroupmaybecreatedwithinanothergroupbyprovidingtheabsolutenameofthegrouptothe
H5Gcreatefunctionorbyspecifyingitslocation.Forexample,tocreatethegroupData_newinthe
groupData,youmightusethesequenceofcallsshownbelow.
/*
* Extend the dataset. Dataset becomes 10 x 3.
*/
dims[0] = dims[0] + 7;
size[0] = dims[0];
size[1] = dims[1];
status = H5Dextend (dataset, size);
CodeExample2‐13.Extendthedatasetbysevenrows
/*
* Create a group in the file.
*/
grp = H5Gcreate(file, "/Data", H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
CodeExample2‐14.Createagroup
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
38 TheHDFGroup
ThisfirstparameterofH5Gcreateisalocationidentifier.fileinthefirstexamplespecifiesonlythefile.
grpinthesecondexamplespecifiesaparticulargroupinaparticularfile.Notethatinthisinstance,the
groupidentifiergrpisusedasthefirstparameterintheH5Gcreatecallsothattherelativenameof
Data_newcanbeused.
ThethirdparameterofH5Gcreateoptionallyspecifieshowmuchfilespacetoreservetostorethenames
ofobjectsthatwillbecreatedinthisgroup.Ifanon‐positivevalueissupplied,thelibraryprovidesa
defaultsize.
UseH5Gclosetoclosethegroupandreleasethegroupidentifier.
CreatingaDatasetwithinaGroup
Aswithgroups,adatasetcanbecreatedinaparticulargroupbyspecifyingeitheritsabsolutenameinthe
fileoritsrelativenamewithrespecttothatgroup.Thenextcodeexcerptusestheabsolutename.
/*
* Create group "Data_new" in the group "Data" by specifying
* absolute name of the group.
*/
grp_new = H5Gcreate(file, "/Data/Data_new", H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
or
/*
* Create group "Data_new" in the "Data" group.
*/
grp_new = H5Gcreate(grp, "Data_new", H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
CodeExample2‐15.Createagroupwithinagroup
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 39
Alternatively,youcanfirstobtainanidentifierforthegroupinwhichthedatasetistobecreated,andthen
createthedatasetwitharelativename.
/*
* Create the dataset "Compressed_Data" in the group Data using
* the absolute name. The dataset creation property list is
* modified to use GZIP compression with the compression
* effort set to 6. Note that compression can be used only when
* the dataset is chunked.
*/
dims[0] = 1000;
dims[1] = 20;
cdims[0] = 20;
cdims[1] = 20;
dataspace = H5Screate_simple(RANK, dims, NULL);
plist = H5Pcreate(H5P_DATASET_CREATE);
H5Pset_chunk(plist, 2, cdims);
H5Pset_deflate(plist, 6);
dataset = H5Dcreate(file, "/Data/Compressed_Data",
H5T_NATIVE_INT, dataspace, H5P_DEFAULT, plist, H5P_DEFAULT);
CodeExample2‐16.Createadatasetwithinagroupusinganabsolutename
/*
* Open the group.
*/
grp = H5Gopen(file, "Data", H5P_DEFAULT);
/*
* Create the dataset "Compressed_Data" in the "Data" group
* by providing a group identifier and a relative dataset
* name as parameters to the H5Dcreate function.
*/
dataset = H5Dcreate(grp, "Compressed_Data", H5T_NATIVE_INT,
dataspace, H5P_DEFAULT, plist, H5P_DEFAULT);
CodeExample2‐17.Createadatasetwithinagroupusingarelativename
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
40 TheHDFGroup
AccessinganObjectinaGroup
Anyobjectinagroupcanbeaccessedbyitsabsoluteorrelativename.Thefirstcodesnippetbelowillus‐
tratestheuseoftheabsolutenametoaccessthedatasetCompressed_DatainthegroupDatacreated
intheexamplesabove.Thesecondcodesnippetillustratestheuseoftherelativename.
2.2.10.WorkingwithAttributes
Anattributeisasmalldatasetthatisattachedtoanormaldatasetorgroup.Attributessharemanyofthe
characteristicsofdatasets,sotheprogrammingmodelforworkingwithattributesissimilarinmanyways
tothemodelforworkingwithdatasets.Theprimarydifferencesarethatanattributemustbeattachedto
adatasetoragroupandsub‐settingoperationscannotbeperformedonattributes.
Tocreateanattributebelongingtoaparticulardatasetorgroup,firstcreateadataspacefortheattribute
withthecalltoH5Screate,andthencreatetheattributeusingH5Acreate.Forexample,thecode
shownbelowcreatesanattributecalledInteger_attributethatisamemberofadatasetwhoseiden‐
tifierisdataset.Theattributeidentifierisattr2.H5Awritethensetsthevalueoftheattributeofthat
oftheintegervariablepoint.H5Aclosethenreleasestheattributeidentifier.
/*
* Open the dataset "Compressed_Data" in the "Data" group.
*/
dataset = H5Dopen(file, "/Data/Compressed_Data", H5P_DEFAULT);
CodeExample2‐18.Accessingagroupusingitsabsolutename
/*
* Open the group "data" in the file.
*/
grp = H5Gopen(file, "Data", H5P_DEFAULT);
/*
* Access the "Compressed_Data" dataset in the group.
*/
dataset = H5Dopen(grp, "Compressed_Data", H5P_DEFAULT);
CodeExample2‐19.Accessingagroupusingitsrelativename
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 41
Toreadascalarattributewhosenameanddatatypeareknown,firstopentheattributeusingH5Aop-
en_by_name,andthenuseH5Areadtogetitsvalue.Forexample,thecodeshownbelowreadsascalar
attributecalledInteger_attributewhosedatatypeisanativeintegerandwhoseparentdatasethas
theidentifierdataset.
Int point = 1; /* Value of the scalar attribute */
/*
* Create scalar attribute.
*/
aid2 = H5Screate(H5S_SCALAR);
attr2 = H5Acreate(dataset, "Integer attribute", H5T_NATIVE_INT,
aid2, H5P_DEFAULT, H5P_DEFAULT);
/*
* Write scalar attribute.
*/
ret = H5Awrite(attr2, H5T_NATIVE_INT, &point);
/*
* Close attribute dataspace.
*/
ret = H5Sclose(aid2);
/*
* Close attribute.
*/
ret = H5Aclose(attr2);
CodeExample2‐20.Createanattribute
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
42 TheHDFGroup
Toreadanattributewhosecharacteristicsarenotknown,gothroughthesesteps.First,querythefileto
obtaininformationabouttheattributesuchasitsname,datatype,rank,anddimensions,andthenread
theattribute.ThefollowingcodeopensanattributebyitsindexvalueusingH5Aopen_by_idx,andthen
itreadsininformationaboutthedatatypewithH5Aread.
Inpractice,ifthecharacteristicsofattributesarenotknown,thecodeinvolvedinaccessingandprocess‐
ingtheattributecanbequitecomplex.Forthisreason,HDF5includesafunctioncalledH5Aiterate.This
functionappliesauser‐suppliedfunctiontoeachofasetofattributes.Theuser‐suppliedfunctioncan
containthecodethatinterprets,accesses,andprocesseseachattribute.
/*
* Attach to the scalar attribute using attribute name, then
* read and display its value.
*/
attr = H5Aopen_by_name(file_id, dataset_name,
"Integer attribute", H5P_DEFAULT, H5P_DEFAULT);
ret = H5Aread(attr, H5T_NATIVE_INT, &point_out);
printf("The value of the attribute \"Integer attribute\"
is %d \n", point_out);
ret = H5Aclose(attr);
CodeExample2‐21.Readaknownattribute
/*
* Attach to the string attribute using its index, then read and
* display the value.
*/
attr = H5Aopen_by_idx(file_id, dataset_name, index_type,
iter_order, 2, H5P_DEFAULT, H5P_DEFAULT);
atype = H5Tcopy(H5T_C_S1);
H5Tset_size(atype, 4);
ret = H5Aread(attr, atype, string_out);
printf("The value of the attribute with the index 2 is %s \n",
string_out);
CodeExample2‐22.Readanunknownattribute
HDF5User’sGuide TheHDF5LibraryandProgrammingModel
TheHDFGroup 43
2.3.TheDataTransferPipeline
TheHDF5Libraryimplementsdatatransfersbetweendifferentstoragelocations.Atthelowestlevels,the
HDF5Libraryreadsandwritesblocksofbytestoandfromstorageusingcallstothevirtualfilelayer(VFL)
drivers.Inadditiontothis,theHDF5LibrarymanagescachesofmetadataandadataI/Opipeline.Thedata
I/Opipelineappliescompressiontodatablocks,transformsdataelements,andimplementsselections.
AsubstantialportionoftheHDF5Library’sworkisintransferringdatafromoneenvironmentormediato
another.Thismostofteninvolvesatransferbetweensystemmemoryandastoragemedium.Datatrans‐
fersareaffectedbycompression,encryption,machine‐dependentdifferencesinnumericalrepresenta‐
tion,andotherfeatures.So,thebit‐by‐bitarrangementofagivendatasetisoftensubstantiallydifferentin
thetwoenvironments.
Considertherepresentationondiskofacompressedandencryptedlittle‐endianarrayascomparedtothe
samearrayafterithasbeenreadfromdisk,decrypted,decompressed,andloadedintomemoryonabig‐
endiansystem.HDF5performsalloftheoperationsnecessarytomakethattransitionduringtheI/Opro‐
cesswithmanyoftheoperationsbeinghandledbytheVFLandthedatatransferpipeline.
Thefigurebelowprovidesasimplifiedviewofasampledatatransferwithfourstages.Notethatthemod‐
ulesareusedonlywhenneeded.Forexample,ifthedataisnotcompressed,thecompressionstageis
omitted.
ForagivenI/Orequest,differentcombinationsofactionsmaybeperformedbythepipeline.Thelibrary
automaticallysetsupthepipelineandpassesdatathroughtheprocessingsteps.Forexample,foraread
request(fromdisktomemory),thelibrarymustdeterminewhichlogicalblockscontaintherequested
Figure2‐4.Adatatransferfromstoragetomemory
TheHDF5LibraryandProgrammingModel HDF5User’sGuide
44 TheHDFGroup
dataelementsandfetcheachblockintothelibrary’scache.Ifthedataneedstobedecompressed,then
thecompressionalgorithmisappliedtotheblockafteritisreadfromdisk.Ifthedataisaselection,the
selectedelementsareextractedfromthedatablockafteritisdecompressed.Ifthedataneedstobe
transformed(forexample,byteswapped),thenthedataelementsaretransformedafterdecompression
andselection.
Whileanapplicationmustsometimessetupsomeelementsofthepipeline,useofthepipelineisnor‐
mallytransparenttotheuserprogram.Thelibrarydetermineswhatmustbedonebasedonthemetadata
forthefile,theobject,andthespecificrequest.Anexampleofwhenanapplicationmightberequiredto
setupsomeelementsinthepipelineisiftheapplicationusedacustomerror‐checkingalgorithm.
Insomecases,itisnecessarytopassparameterstoandfrommodulesinthepipelineoramongother
partsofthelibrarythatarenotdirectlycalledthroughtheprogrammingAPI.Thisisaccomplishedthrough
theuseofdatasettransferanddataaccesspropertylists.
TheVFLprovidesaninterfacewherebyuserapplicationscanaddcustommodulestothedatatransfer
pipeline.Forexample,acustomcompressionalgorithmcanbeusedwiththeHDF5Librarybylinkingan
appropriatemoduleintothepipelinethroughtheVFL.Thisrequirescreatinganappropriatewrapperfor
thecompressionmoduleandregisteringitwiththelibrarywithH5Zregister.Thealgorithmcanthenbe
appliedtoadatasetwithanH5Pset_filtercallwhichwilladdthealgorithmtotheselecteddataset’s
transferpropertylist.
HDF5User’sGuide TheHDF5File
TheHDFGroup 45
3.TheHDF5File
3.1.Introduction
ThepurposeofthischapteristodescribehowtoworkwithHDF5datafiles.
IfHDF5dataistobewrittentoorreadfromafile,thefilemustfirstbeexplicitlycreatedoropenedwith
theappropriatefiledriverandaccessprivileges.Onceallworkwiththefileiscomplete,thefilemustbe
explicitlyclosed.
Thischapterdiscussesthefollowing:
•Fileaccessmodes
•Creating,opening,andclosingfiles
•Theuseoffilecreationpropertylists
•Theuseoffileaccesspropertylists
•Theuseoflow‐levelfiledrivers
Thischapterassumesanunderstandingofthematerialpresentedinthedatamodelchapter.Formore
information,see"TheHDF5DataModelandFileStructure"onpage 1.
3.2.FileAccessModes
Therearetwoissuesregardingfileaccess:
•Whatshouldhappenwhenanewfileiscreatedbutafileofthesamenamealreadyexists?
Shouldthecreateactionfail,orshouldtheexistingfilebeoverwritten?
•Isafiletobeopenedwithread‐onlyorread‐writeaccess?
Fouraccessmodesaddresstheseconcerns.TwoofthesemodescanbeusedwithH5Fcreate,andtwo
modescanbeusedwithH5Fopen.
•H5FcreateacceptsH5F_ACC_EXCLorH5F_ACC_TRUNC
•H5FopenacceptsH5F_ACC_RDONLYorH5F_ACC_RDWR
Theaccessmodesaredescribedinthetablebelow.
HDF5User’sGuide TheHDF5File
TheHDFGroup 46
Bydefault,H5Fopenopensafileforread‐onlyaccess;passingH5F_ACC_RDWRallowsread‐writeaccessto
thefile.
Bydefault,H5Fcreatefailsifthefilealreadyexists;onlypassingH5F_ACC_TRUNCallowsthetruncating
ofanexistingfile.
3.3.FileCreationandFileAccessProperties
Filecreationandfileaccesspropertylistscontrolthemorecomplexaspectsofcreatingandaccessingfiles.
Filecreationpropertylistscontrolthecharacteristicsofafilesuchasthesizeoftheuserblock,auser‐
definabledatablock;thesizeofdataaddressparameters;propertiesoftheB‐treesthatareusedtoman‐
agethedatainthefile;andcertainHDF5Libraryversioninginformation.
Formoreinformation,see"FileCreationProperties"onpage 58.Thissectionhasamoredetaileddiscus‐
sionoffilecreationproperties.Ifyouhavenospecialrequirementsforthesefilecharacteristics,youcan
simplyspecifyH5P_DEFAULTforthedefaultfilecreationpropertylistwhenafilecreationpropertylistis
calledfor.
Fileaccesspropertylistscontrolpropertiesandmeansofaccessingafilesuchasdataalignmentcharac‐
teristics,metadatablockandcachesizes,datasievebuffersize,garbagecollectionsettings,andparallelI/
O.Dataalignment,metadatablockandcachesizes,anddatasievebuffersizearefactorsinimprovingI/O
performance.
Formoreinformation,see"FileAccessProperties"onpage 60.Thissectionhasamoredetaileddiscussion
offileaccessproperties.Ifyouhavenospecialrequirementsforthesefileaccesscharacteristics,youcan
simplyspecifyH5P_DEFAULTforthedefaultfileaccesspropertylistwhenafileaccesspropertylistis
calledfor.
Table3‐1.Accessflagsandmodes
AccessFlag ResultingAccessMode
H5F_ACC_EXCL Ifthefilealreadyexists,H5Fcreatefails.Ifthefiledoesnot
exist,itiscreatedandopenedwithread‐writeaccess.(Default)
H5F_ACC_TRUNC Ifthefilealreadyexists,thefileisopenedwithread‐writeaccess,
andnewdatawilloverwriteanyexistingdata.Ifthefiledoesnot
exist,itiscreatedandopenedwithread‐writeaccess.
H5F_ACC_RDONLY Anexistingfileisopenedwithread‐onlyaccess.Ifthefiledoes
notexist,H5Fopenfails.(Default)
H5F_ACC_RDWR Anexistingfileisopenedwithread‐writeaccess.Ifthefiledoes
notexist,H5Fopenfails.
HDF5User’sGuide TheHDF5File
TheHDFGroup 47
3.4.Low‐levelFileDrivers
TheconceptofanHDF5fileisactuallyratherabstract:theaddressspaceforwhatisnormallythoughtof
asanHDF5filemightcorrespondtoanyofthefollowingatthestoragelevel:
• Singlefileonastandardfilesystem
•Multiplefilesonastandardfilesystem
•Multiplefilesonaparallelfilesystem
•Blockofmemorywithinanapplication’smemoryspace
•Moreabstractsituationssuchasvirtualfiles
ThisHDF5addressspaceisgenerallyreferredtoasanHDF5fileregardlessofitsorganizationatthestor‐
agelevel.
HDF5accessesafile(theaddressspace)throughvarioustypesoflow‐levelfiledrivers.ThedefaultHDF5
filestoragelayoutisasanunbufferedpermanentfilewhichisasingle,contiguousfileonlocaldisk.Alter‐
nativelayoutsaredesignedtosuittheneedsofavarietyofsystems,environments,andapplications.
Figure3‐1.UMLmodelforanHDF5fileanditspropertylists
HDF5User’sGuide TheHDF5File
TheHDFGroup 48
3.5.ProgrammingModelforFiles
Programmingmodelsforcreating,opening,andclosingHDF5filesaredescribedinthesub‐sections
below.
3.5.1.CreatingaNewFile
TheprogrammingmodelforcreatinganewHDF5filecanbesummarizedasfollows:
• Definethefilecreationpropertylist
• Definethefileaccesspropertylist
•Createthefile
First,considerthesimplecasewhereweusethedefaultvaluesforthepropertylists.Seetheexample
below.
Note:TheexampleabovespecifiesthatH5FcreateshouldfailifSampleFile.h5alreadyexists.
Amorecomplexcaseisshownintheexamplebelow.Inthisexample,wedefinefilecreationandaccess
propertylists(thoughwedonotassignanyproperties),specifythatH5FcreateshouldfailifSample-
File.h5alreadyexists,andcreateanewfilenamedSampleFile.h5.Theexampledoesnotspecifya
driver,sothedefaultdriver,H5FD_SEC2,willbeused.
Notes:
Arootgroupisautomaticallycreatedinafilewhenthefileisfirstcreated.
Filepropertylists,oncedefined,canbereusedwhenanotherfileiscreatedwithinthesameapplication.
file_id = H5Fcreate ("SampleFile.h5", H5F_ACC_EXCL,
H5P_DEFAULT, H5P_DEFAULT)
CodeExample3‐1.CreatinganHDF5fileusingpropertylistdefaults
fcplist_id = H5Pcreate (H5P_FILE_CREATE)
<...set desired file creation properties...>
faplist_id = H5Pcreate (H5P_FILE_ACCESS)
<...set desired file access properties...>
file_id = H5Fcreate ("SampleFile.h5", H5F_ACC_EXCL,
fcplist_id, faplist_id)
CodeExample3‐2.CreatinganHDF5fileusingpropertylists
HDF5User’sGuide TheHDF5File
TheHDFGroup 49
3.5.2.OpeninganExistingFile
TheprogrammingmodelforopeninganexistingHDF5filecanbesummarizedasfollows:
• Defineormodifythefileaccesspropertylistincludingalow‐levelfiledriver(optional)
•Openthefile
Thecodeintheexamplebelowshowshowtoopenanexistingfilewithread‐onlyaccess.
3.5.3.ClosingaFile
TheprogrammingmodelforclosinganHDF5fileisverysimple:
•Closefile
WecloseSampleFile.h5withthecodeintheexamplebelow.
NotethatH5Fcloseflushesallunwrittendatatostorageandthatfile_idistheidentifierreturnedfor
SampleFile.h5byH5Fopen.
Morecomprehensivediscussionsregardingallofthesestepsareprovidedbelow.
3.6.Usingh5dumptoViewaFile
h5dumpisacommand‐lineutilitythatisincludedintheHDF5distribution.Thisprogramprovidesa
straight‐forwardmeansofinspectingthecontentsofanHDF5file.Youcanuseh5dumptoverifythata
programisgeneratingtheintendedHDF5file.h5dumpdisplaysASCIIoutputformattedaccordingtothe
HDF5DDLgrammar.
faplist_id = H5Pcreate (H5P_FILE_ACCESS)
status = H5Pset_fapl_stdio (faplist_id)
file_id = H5Fopen ("SampleFile.h5", H5F_ACC_RDONLY,
faplist_id)
CodeExample3‐3.OpeninganHDF5file
status = H5Fclose (file_id)
CodeExample3‐4.ClosinganHDF5file
HDF5User’sGuide TheHDF5File
TheHDFGroup 50
Thefollowingh5dumpcommandwilldisplaythecontentsofSampleFile.h5:
h5dump SampleFile.h5
Ifnodatasetsorgroupshavebeencreatedinandnodatahasbeenwrittentothefile,theoutputwilllook
somethinglikethefollowing:
HDF5 "SampleFile.h5" {
GROUP "/" {
}
}
Notethattherootgroup,indicatedaboveby/,wasautomaticallycreatedwhenthefilewascreated.
h5dumpisdescribedontheToolspageoftheHDF5ReferenceManual.TheHDF5DDLgrammaris
describedinthedocumentDDLinBNFforHDF5.
3.7.FileFunctionSummaries
Generallibraryfunctionsandmacros(H5),filefunctions(H5F),filerelatedpropertylistfunctions(H5P),
andfiledriverfunctions(H5P)arelistedbelow.
FunctionListing3‐1.Generallibraryfunctionsandmacros(H5)
CFunction
FortranFunction
Purpose
H5check_version
h5check_version_f
VerifiesthatHDF5Libraryversionsareconsis‐
tent.
H5close
h5close_f
Flushesalldatatodisk,closesallopenidenti‐
fiers,andcleansupmemory.
H5dont_atexit
h5dont_atexit_f
Instructsthelibrarynottoinstalltheatexit
cleanuproutine.
H5garbage_collect
h5garbage_collect_f
Garbagecollectsonallfree‐listsofalltypes.
H5get_libversion
h5get_libversion_f
ReturnstheHDFlibraryreleasenumber.
H5open
h5open_f
InitializestheHDF5Library.
H5set_free_list_limits
h5set_free_list_limits_f
Setsfree‐listsizelimits.
HDF5User’sGuide TheHDF5File
TheHDFGroup 51
H5_VERSION_GE
(noFortransubroutine)
Determineswhethertheversionofthelibrary
beingusedisgreaterthanorequaltothe
specifiedversion.
H5_VERSION_LE
(noFortransubroutine)
Determineswhethertheversionofthelibrary
beingusedislessthanorequaltothespeci‐
fiedversion.
FunctionListing3‐2.Filefunctions(H5F)
CFunction
FortranFunction
Purpose
H5Fclear_elink_file_cache
(noFortransubroutine)
Clearstheexternallinkopenfilecachefora
file.
H5Fclose
h5fclose_f
ClosesHDF5file.
H5Fcreate
h5fcreate_f
CreatesnewHDF5file.
H5Fflush
h5fflush_f
FlushesdatatoHDF5fileonstoragemedium.
H5Fget_access_plist
h5fget_access_plist_f
Returnsafileaccesspropertylistidentifier.
H5Fget_create_plist
h5fget_create_plist_f
Returnsafilecreationpropertylistidentifier.
H5Fget_file_image
h5fget_file_image_f
Retrievesacopyoftheimageofanexisting,
openfile.
H5Fget_filesize
h5fget_filesize_f
ReturnsthesizeofanHDF5file.
H5Fget_freespace
h5fget_freespace_f
Returnstheamountoffreespaceinafile.
H5Fget_info
(noFortransubroutine)
Returnsglobalinformationforafile.
H5Fget_intent
(noFortransubroutine)
Determinestheread/writeorread‐onlystatus
ofafile.
FunctionListing3‐1.Generallibraryfunctionsandmacros(H5)
CFunction
FortranFunction
Purpose
HDF5User’sGuide TheHDF5File
TheHDFGroup 52
H5Fget_mdc_config
(noFortransubroutine)
Obtaincurrentmetadatacacheconfiguration
fortargetfile.
H5Fget_mdc_hit_rate
(noFortransubroutine)
Obtaintargetfile’smetadatacachehitrate.
H5Fget_mdc_size
(noFortransubroutine)
Obtaincurrentmetadatacachesizedatafor
specifiedfile.
H5Fget_mpi_atomicity
h5fget_mpi_atomicity_f
Retrievestheatomicitymodeinuse.
H5Fget_name
h5fget_name_f
Retrievesthenameofthefiletowhichthe
objectbelongs.
H5Fget_obj_count
h5fget_obj_count_f
Returnsthenumberofopenobjectidentifiers
foranopenfile.
H5Fget_obj_ids
h5fget_obj_ids_f
Returnsalistofopenobjectidentifiers.
H5Fget_vfd_handle
(noFortransubroutine)
Returnspointertothefilehandlefromthe
virtualfiledriver.
H5Fis_hdf5
h5fis_hdf5_f
DetermineswhetherafileisintheHDF5for‐
mat.
H5Fmount
h5fmount_f
Mountsafile.
H5Fopen
h5fopen_f
OpensexistingHDF5file.
H5Freopen
h5freopen_f
Returnsanewidentifierforapreviously‐
openedHDF5file.
H5Freset_mdc_hit_rate_stats
(noFortransubroutine)
Resethitratestatisticscountersforthetarget
file.
H5Fset_mdc_config
(noFortransubroutine)
Usetoconfiguremetadatacacheoftarget
file.
H5Fset_mpi_atomicity
h5fset_mpi_atomicity_f
UsetosettheMPIatomicitymode.
H5Funmount
h5funmount_f
Unmountsafile.
FunctionListing3‐2.Filefunctions(H5F)
CFunction
FortranFunction
Purpose
HDF5User’sGuide TheHDF5File
TheHDFGroup 53
FunctionListing3‐3.Filecreationpropertylistfunctions(H5P)
CFunction
FortranFunction
Purpose
H5Pset/get_userblock
h5pset/get_userblock_f
Sets/retrievessizeofuserblock.
H5Pset/get_sizes
h5pset/get_sizes_f
Sets/retrievesbytesizeofoffsetsandlengths
usedtoaddressobjectsinHDF5file.
H5Pset/get_sym_k
h5pset/get_sym_k_f
Sets/retrievessizeofparametersusedtocon‐
trolsymboltablenodes.
H5Pset/get_istore_k
h5pset/get_istore_k_f
Sets/retrievessizeofparameterusedtocon‐
trolB‐treesforindexingchunkeddatasets.
H5Pget_file_image
h5pget_file_image_f
Retrievesacopyofthefileimagedesignated
astheinitialcontentandstructureofafile.
H5Pset_file_image
h5pset_file_image_f
Setsaninitialfileimageinamemorybuffer.
H5Pset_shared_mesg_nindexes
h5pset_shared_mesg_nindexes_f
Setsnumberofsharedobjectheadermes‐
sageindexes.
H5Pget_shared_mesg_nindexes
(noFortransubroutine)
Retrievesnumberofsharedobjectheader
messageindexesinfilecreationpropertylist.
H5Pset_shared_mesg_index
h5pset_shared_mesg_index_f
Configuresthespecifiedsharedobjectheader
messageindex.
H5Pget_shared_mesg_index
(noFortransubroutine)
Retrievestheconfigurationsettingsfora
sharedmessageindex.
H5Pset_shared_mesg_phase_change
(noFortransubroutine)
Setssharedobjectheadermessagestorage
phasechangethresholds.
H5Pget_shared_mesg_phase_change
(noFortransubroutine)
Retrievessharedobjectheadermessage
phasechangeinformation.
H5Pget_version
h5pget_version_f
Retrievesversioninformationforvarious
objectsforfilecreationpropertylist.
HDF5User’sGuide TheHDF5File
TheHDFGroup 54
FunctionListing3‐4.Fileaccesspropertylistfunctions(H5P)
CFunction
FortranFunction
Purpose
H5Pset/get_alignment
h5pset/get_alignment_f
Sets/retrievesalignmentproperties.
H5Pset/get_cache
h5pset/get_cache_f
Sets/retrievesmetadatacacheandrawdata
chunkcacheparameters.
H5Pset/get_elink_file_cache_size
(noFortransubroutine)
Sets/retrievesthesizeoftheexternallink
openfilecachefromthespecifiedfileaccess
propertylist.
H5Pset/get_fclose_degree
h5pset/get_fclose_degree_f
Sets/retrievesfileclosedegreeproperty.
H5Pset/get_gc_references
h5pset/get_gc_references_f
Sets/retrievesgarbagecollectingreferences
flag.
H5Pset_family_offset
h5pset_family_offset_f
Setsoffsetpropertyforlow‐levelaccesstoa
fileinafamilyoffiles.
H5Pget_family_offset
(noFortransubroutine)
Retrievesadataoffsetfromthefileaccess
propertylist.
H5Pset/get_meta_block_size
h5pset/get_meta_block_size_f
Setstheminimummetadatablocksizeor
retrievesthecurrentmetadatablocksizeset‐
ting.
H5Pset_mdc_config
(noFortransubroutine)
Settheinitialmetadatacacheconfigurationin
theindicatedFileAccessPropertyListtothe
suppliedvalue.
H5Pget_mdc_config
(noFortransubroutine)
Getthecurrentinitialmetadatacacheconfig‐
urationfromtheindicatedFileAccessProp‐
ertyList.
H5Pset/get_sieve_buf_size
h5pset/get_sieve_buf_size_f
Sets/retrievesmaximumsizeofdatasieve
buffer.
H5Pset_libver_bounds
h5pset_libver_bounds_f
Setsboundsonlibraryversions,andindirectly
formatversions,tobeusedwhencreating
objects.
H5Pget_libver_bounds
(noFortransubroutine)
Retrieveslibraryversionboundssettingsthat
indirectlycontroltheformatversionsused
whencreatingobjects.
HDF5User’sGuide TheHDF5File
TheHDFGroup 55
H5Pset_small_data_block_size
h5pset_small_data_block_size_f
Setsthesizeofacontiguousblockreserved
forsmalldata.
H5Pget_small_data_block_size
h5pget_small_data_block_size_f
Retrievesthecurrentsmalldatablocksize
setting.
FunctionListing3‐5.Filedriverfunctions(H5P)
CFunction
FortranFunction
Purpose
H5Pset_driver
(noFortransubroutine)
Setsafiledriver.
H5Pget_driver
h5pget_driver_f
Returnstheidentifierforthedriverusedto
createafile.
H5Pget_driver_info
(noFortransubroutine)
Returnsapointertofiledriverinformation.
H5Pset/get_fapl_core
h5pset/get_fapl_core_f
Setsthedriverforbufferedmemoryfiles(in
RAM)orretrievesinformationregardingthe
driver.
H5Pset_fapl_direct
h5pset_fapl_direct_f
SetsupuseofthedirectI/Odriver.
H5Pget_fapl_direct
h5pget_fapl_direct_f
RetrievesthedirectI/Odriversettings.
H5Pset/get_fapl_family
h5pset/get_fapl_family_f
Setsdriverforfilefamilies,designedforsys‐
temsthatdonotsupportfileslargerthan2
gigabytes,orretrievesinformationregarding
driver.
H5Pset_fapl_log
(noFortransubroutine)
Setsloggingdriver.
H5Pset/get_fapl_mpio
h5pset/get_fapl_mpio_f
Setsdriverforfilesonparallelfilesystems
(MPII/O)orretrievesinformationregarding
thedriver.
H5Pset_fapl_mpiposix
h5pset_fapl_mpiposix_f
Nolongeravailable.
FunctionListing3‐4.Fileaccesspropertylistfunctions(H5P)
CFunction
FortranFunction
Purpose
HDF5User’sGuide TheHDF5File
TheHDFGroup 56
3.8.CreatingorOpeninganHDF5File
Thissectiondescribesinmoredetailhowtocreateandhowtoopenfiles.
NewHDF5filesarecreatedandopenedwithH5Fcreate;existingfilesareopenedwithH5Fopen.Both
functionsreturnanobjectidentifierwhichmusteventuallybereleasedbycallingH5Fclose.
Tocreateanewfile,callH5Fcreate:
hid_t H5Fcreate (const char *name, unsigned flags, hid_t fcpl_id,
hid_t fapl_id)
H5Fcreatecreatesanewfilenamednameinthecurrentdirectory.Thefileisopenedwithreadandwrite
access;iftheH5F_ACC_TRUNCflagisset,anypre‐existingfileofthesamenameinthesamedirectoryis
truncated.IfH5F_ACC_TRUNCisnotsetorH5F_ACC_EXCLissetandifafileofthesamenameexists,
H5Fcreatewillfail.
H5Pget_fapl_mpiposix
h5pget_fapl_mpiposix_f
Nolongeravailable.
H5Pset/get_fapl_multi
h5pset/get_fapl_multi_f
Setsdriverformultiplefiles,separatingcate‐
goriesofmetadataandrawdata,orretrieves
informationregardingdriver.
H5Pset_fapl_sec2
h5pset_fapl_sec2_f
Setsdriverforunbufferedpermanentfilesor
retrievesinformationregardingdriver.
H5Pset_fapl_split
h5pset_fapl_split_f
Setsdriverforsplitfiles,alimitedcaseofmul‐
tiplefileswithonemetadatafileandoneraw
datafile.
H5Pset_fapl_stdio
H5Pset_fapl_stdio_f
Setsdriverforbufferedpermanentfiles.
H5Pset_fapl_windows
(noFortransubroutine)
SetstheWindowsI/Odriver.
H5Pset_multi_type
(noFortransubroutine)
Specifiestypeofdatatobeaccessedviathe
MULTIdriverenablingmoredirectaccess.
H5Pget_multi_type
(noFortransubroutine)
RetrievestypeofdatapropertyforMULTI
driver.
FunctionListing3‐5.Filedriverfunctions(H5P)
CFunction
FortranFunction
Purpose
HDF5User’sGuide TheHDF5File
TheHDFGroup 57
Thenewfileiscreatedwiththepropertiesspecifiedinthepropertylistsfcpl_idandfapl_id.fcplis
shortforfilecreationpropertylist.faplisshortforfileaccesspropertylist.SpecifyingH5P_DEFAULTfor
eitherthecreationoraccesspropertylistcallsforthelibrary’sdefaultcreationoraccessproperties.
IfH5Fcreatesuccessfullycreatesthefile,itreturnsafileidentifierforthenewfile.Thisidentifierwillbe
usedbytheapplicationanytimeanobjectidentifier,anOID,forthefileisrequired.Oncetheapplication
hasfinishedworkingwithafile,theidentifiershouldbereleasedandthefileclosedwithH5Fclose.
Toopenanexistingfile,callH5Fopen:
hid_t H5Fopen (const char *name, unsigned flags, hid_t fapl_id)
H5Fopenopensanexistingfilewithread‐writeaccessifH5F_ACC_RDWRissetandread‐onlyaccessif
H5F_ACC_RDONLYisset.
fapl_idisthefileaccesspropertylistidentifier.Alternatively,H5P_DEFAULTindicatesthattheapplica‐
tionreliesonthedefaultI/Oaccessparameters.Creatingandchangingaccesspropertylistsisdocu‐
mentedfurtherbelow.
AfilecanbeopenedmorethanonceviamultipleH5Fopencalls.Eachsuchcallreturnsauniquefileiden‐
tifierandthefilecanbeaccessedthroughanyofthesefileidentifiersaslongastheyremainvalid.Eachof
thesefileidentifiersmustbereleasedbycallingH5Fclosewhenitisnolongerneeded.
Formoreinformation,see"FileAccessModes"onpage 45.
Formoreinformation,see"FilePropertyLists"onpage 58.
3.9.ClosinganHDF5File
H5FclosebothclosesafileandreleasesthefileidentifierreturnedbyH5FopenorH5Fcreate.H5F-
closemustbecalledwhenanapplicationisdoneworkingwithafile;whiletheHDF5Librarymakesevery
efforttomaintainfileintegrity,failuretocallH5Fclosemayresultinthefilebeingabandonedinan
incompleteorcorruptedstate.
Tocloseafile,callH5Fclose:
herr_t H5Fclose (hid_t file_id)
Thisfunctionreleasesresourcesassociatedwithanopenfile.Afterclosingafile,thefileidentifier,
file_id,cannotbeusedagainasitwillbeundefined.
H5Fclosefulfillsthreepurposes:toensurethatthefileisleftinanuncorruptedstate,toensurethatall
datahasbeenwrittentothefile,andtoreleaseresources.UseH5Fflushifyouwishtoensurethatall
datahasbeenwrittentothefilebutitisprematuretocloseit.
Noteregardingserialmodebehavior:WhenH5Fcloseiscalledinserialmode,itclosesthefileandtermi‐
natesnewaccesstoit,butitdoesnotterminateaccesstoobjectsthatremainindividuallyopenwithinthe
file.Thatis,ifH5Fcloseiscalledforafilebutoneormoreobjectswithinthefileremainopen,those
objectswillremainaccessibleuntiltheyareindividuallyclosed.Toillustrate,assumethatafile,fileA,
containsadataset,data_setA,andthatbothareopenwhenH5FcloseiscalledforfileA.data_setA
HDF5User’sGuide TheHDF5File
TheHDFGroup 58
willremainopenandaccessible,includingwritable,untilitisexplicitlyclosed.Thefilewillbeautomati‐
callyandfinallyclosedonceallobjectswithinithavebeenclosed.
Noteregardingparallelmodebehavior:OnceH5Fclosehasbeencalledinparallelmode,accessisno
longeravailabletoanyobjectwithinthefile.
3.10.FilePropertyLists
AdditionalinformationregardingfilestructureandaccessarepassedtoH5FcreateandH5Fopen
throughpropertylistobjects.Propertylistsprovideaportableandextensiblemethodofmodifyingfile
propertiesviasimpleAPIfunctions.Therearetwokindsoffile‐relatedpropertylists:
•Filecreationpropertylists
•Fileaccesspropertylists
Inthefollowingsub‐sections,wediscussonlyonefilecreationproperty,userblocksize,indetailasa
modelfortheuser.Otherfilecreationandfileaccesspropertiesarementionedanddefinedbriefly,but
themodelisnotexpandedforeach;completesyntax,parameter,andusageinformationforeveryprop‐
ertylistfunctionisprovidedinthe"H5P:PropertyListInterface"sectionoftheHDF5ReferenceManual.
Formoreinformation,see"PropertiesandPropertyListsinHDF5"onpage 337.
3.10.1.CreatingaPropertyList
Ifyoudonotwishtorelyonthedefaultfilecreationandaccessproperties,youmustfirstcreateaprop‐
ertylistwithH5Pcreate.
hid_t H5Pcreate (hid_t cls_id)
typeisthetypeofpropertylistbeingcreated.Inthiscase,theappropriatevaluesareH5P_FILE_CRE-
ATEforafilecreationpropertylistandH5P_FILE_ACCESSforafileaccesspropertylist.
Thus,thefollowingcallscreateafilecreationpropertylistandafileaccesspropertylistwithidentifiers
fcpl_idandfapl_id,respectively:
fcpl_id = H5Pcreate (H5P_FILE_CREATE)
fapl_id = H5Pcreate (H5P_FILE_ACCESS)
Oncethepropertylistshavebeencreated,thepropertiesthemselvescanbemodifiedviathefunctions
describedinthefollowingsub‐sections.
3.10.2.FileCreationProperties
Filecreationpropertylistscontrolthefilemetadata,whichismaintainedinthesuperblockofthefile.
Thesepropertiesareusedonlywhenafileisfirstcreated.
HDF5User’sGuide TheHDF5File
TheHDFGroup 59
UserblockSize
herr_t H5Pset_userblock (hid_t plist, hsize_t size)
herr_t H5Pget_userblock (hid_t plist, hsize_t *size)
Theuserblockisafixed‐lengthblockofdatalocatedatthebeginningofthefileandisignoredbytheHDF5
Library.Thisblockisspecificallysetasideforanydataorinformationthatdevelopersdeterminetobeuse‐
fultotheirapplicationsbutthatwillnotbeusedbytheHDF5Library.Thesizeoftheuserblockisdefined
inbytesandmaybesettoanypoweroftwowithaminimumsizeof512bytes.Inotherwords,userblocks
mightbe512,1024,or2048bytesinsize.
ThispropertyissetwithH5Pset_userblockandqueriedviaH5Pget_userblock.Forexample,ifan
applicationneededa4Kuserblock,thenthefollowingfunctioncallcouldbeused:
status = H5Pset_userblock(fcpl_id, 4096)
Thepropertylistcouldlaterbequeriedwith
status = H5Pget_userblock(fcpl_id, size)
andthevalue4096wouldbereturnedintheparametersize.
Otherproperties,describedbelow,aresetandqueriedinexactlythesamemanner.Syntaxandusageare
detailedinthe"H5P:PropertyListInterface"sectionoftheHDF5ReferenceManual.
OffsetandLengthSizes
ThispropertyspecifiesthenumberofbytesusedtostoretheoffsetandlengthofobjectsintheHDF5file.
Valuesof2,4,and8bytesarecurrentlysupportedtoaccommodate16‐bit,32‐bit,and64‐bitfileaddress
spaces.
ThesepropertiesaresetandqueriedviaH5Pset_sizesandH5Pget_sizes.
SymbolTableParameters
ThesizeofsymboltableB‐treescanbecontrolledbysettingthe1/2‐rankand1/2‐nodesizeparametersof
theB‐tree.
ThesepropertiesaresetandqueriedviaH5Pset_sym_kandH5Pget_sym_k.
IndexedStorageParameters
ThesizeofindexedstorageB‐treescanbecontrolledbysettingthe1/2‐rankand1/2‐nodesizeparameters
oftheB‐tree.
ThesepropertiesaresetandqueriedviaH5Pset_istore_kandH5Pget_istore_k.
Vers ionInformation
VariousobjectsinanHDF5filemayovertimeappearindifferentversions.TheHDF5Librarykeepstrackof
theversionofeachobjectinthefile.
VersioninformationisretrievedviaH5Pget_version.
HDF5User’sGuide TheHDF5File
TheHDFGroup 60
3.10.3.FileAccessProperties
Thissectiondiscussesfileaccesspropertiesthatarenotrelatedtothelow‐levelfiledrivers.Filedriversare
discussedseparatelylaterinthischapter.Formoreinformation,see"AlternateFileStorageLayoutsand
Low‐levelFileDrivers"onpage 61.
FileaccesspropertylistscontrolvariousaspectsoffileI/Oandstructure.
DataAlignment
Sometimesfileaccessisfasterifcertaindataelementsarealignedinaspecificmanner.Thiscanbecon‐
trolledbysettingalignmentpropertiesviatheH5Pset_alignmentfunction.Therearetwovalues
involved:
•Athresholdvalue
•Analignmentinterval
Anyallocationrequestatleastaslargeasthethresholdwillbealignedonanaddressthatisamultipleof
thealignmentinterval.
MetadataBlockAllocationSize
Metadatatypicallyexistsasverysmallchunksofdata;storingmetadataelementsinafilewithoutblock‐
ingthemcanresultinhundredsorthousandsofverysmalldataelementsinthefile.Thiscanresultina
highlyfragmentedfileandseriouslyimpedeI/O.Byblockingmetadataelements,thesesmallelements
canbegroupedinlargersets,thusalleviatingbothproblems.
H5Pset_meta_block_sizesetstheminimumsizeinbytesofmetadatablockallocations.
H5Pget_meta_block_sizeretrievesthecurrentminimummetadatablockallocationsize.
MetadataCache
MetadataandrawdataI/Ospeedareoftengovernedbythesizeandfrequencyofdiskreadsandwrites.
Inmanycases,thespeedcanbesubstantiallyimprovedbytheuseofanappropriatecache.
H5Pset_cachesetstheminimumcachesizeforbothmetadataandrawdataandapreemptionvaluefor
rawdatachunks.H5Pget_cacheretrievesthecurrentvalues.
DataSieveBufferSize
DatasievebufferingisusedbycertainfiledriverstospeeddataI/Oandismostcommonlywhenworking
withdatasethyperslabs.Forexample,usingabufferlargeenoughtoholdseveralpiecesofadatasetasit
isreadinforhyperslabselectionswillboostperformancenoticeably.
H5Pset_sieve_buf_sizesetsthemaximumsizeinbytesofthedatasievebuffer.
H5Pget_sieve_buf_sizeretrievesthecurrentmaximumsizeofthedatasievebuffer.
GarbageCollectionReferences
DatasetregionreferencesandotherreferencetypesusespaceinanHDF5file’sglobalheap.Ifgarbage
collectionison(1)andtheuserpassesinanuninitializedvalueinareferencestructure,theheapmight
HDF5User’sGuide TheHDF5File
TheHDFGroup 61
becomecorrupted.Whengarbagecollectionisoff(0),however,andtheuserre‐usesareference,thepre‐
viousheapblockwillbeorphanedandnotreturnedtothefreeheapspace.Whengarbagecollectionis
on,theusermustinitializethereferencestructuresto0orriskheapcorruption.
H5Pset_gc_referencessetsthegarbagecollectingreferencesflag.
3.11.AlternateFileStorageLayoutsandLow‐levelFile
Drivers
TheconceptofanHDF5fileisactuallyratherabstract:theaddressspaceforwhatisnormallythoughtof
asanHDF5filemightcorrespondtoanyofthefollowing:
• Singlefileonstandardfilesystem
•Multiplefilesonstandardfilesystem
•Multiplefilesonparallelfilesystem
•Blockofmemorywithinapplication’smemoryspace
•Moreabstractsituationssuchasvirtualfiles
ThisHDF5addressspaceisgenerallyreferredtoasanHDF5fileregardlessofitsorganizationatthestor‐
agelevel.
HDF5employsanextremelyflexiblemechanismcalledthevirtualfilelayer,orVFL,forfileI/O.Afull
understandingoftheVFLisonlynecessaryifyouplantowriteyourowndrivers(see"VirtualFileLayer"
and"ListofVFLFunctions"intheHDF5TechnicalNotes).Forourpurposeshere,itissufficienttoknow
thatthelow‐leveldriversusedforfileI/OresideintheVFL,asillustratedinthefollowingfigure.Notethat
H5FD_STREAMisnotavailablewith1.8.xandlaterversionsofthelibrary.
HDF5User’sGuide TheHDF5File
TheHDFGroup 62
Asmentionedabove,HDF5applicationsaccessHDF5filesthroughvariouslow‐levelfiledrivers.The
defaultdriverforthatlayoutisthePOSIXdriver(alsoknownastheSEC2driver),H5FD_SEC2.Alternative
layoutsanddriversaredesignedtosuittheneedsofavarietyofsystems,environments,andapplications.
Thedriversarelistedinthetablebelow.
Figure3‐2.I/OpathfromapplicationtoVFLandlow‐leveldriverstostorage
HDF5User’sGuide TheHDF5File
TheHDFGroup 63
Table3‐2.Supportedfiledrivers
Driver
Name
Driver
Identifier
Description RelatedAPI
POSIXH5FD_SEC2 ThisdriverusesPOSIX
file‐systemfunctions
likereadandwriteto
performI/Otoasingle,
permanentfileonlocal
diskwithnosystem
buffering.Thisdriveris
POSIX‐compliantandis
thedefaultfiledriver
forallsystems.
H5Pset_fapl_sec2
Direct H5FD_DIRECT ThisistheH5FD_SEC2
driverexceptdatais
writtentoorreadfrom
thefilesynchronously
withoutbeingcachedby
thesystem.
H5Pset_fapl_direct
Log H5FD_LOG ThisistheH5FD_SEC2
driverwithlogging
capabilities.
H5Pset_fapl_log
Windows H5FD_WINDOWS Thisdriverwasmodified
inHDF5‐1.8.8tobea
wrapperofthePOSIX
driver,H5FD_SEC2.This
changeshouldnot
affectuserapplications.
H5Pset_fapl_windows
STDIO H5FD_STDIO Thisdriverusesfunc‐
tionsfromthestandard
Cstdio.htoperform
I/Otoasingle,perma‐
nentfileonlocaldisk
withadditionalsystem
buffering.
H5Pset_fapl_stdio
HDF5User’sGuide TheHDF5File
TheHDFGroup 64
Memory H5FD_CORE Withthisdriver,an
applicationcanwork
withafileinmemory
forfasterreadsand
writes.Filecontentsare
keptinmemoryuntil
thefileisclosed.Atclos‐
ing,thememoryversion
ofthefilecanbewrit‐
tenbacktodiskor
abandoned.
H5Pset_fapl_core
Family H5FD_FAMILY Withthisdriver,the
HDF5file’saddress
spaceispartitionedinto
piecesandsenttosepa‐
ratestoragefilesusing
anunderlyingdriverof
theuser’schoice.This
driverisforsystemsthat
donotsupportfiles
largerthan2gigabytes.
H5Pset_fapl_family
Multi H5FD_MULTI Withthisdriver,data
canbestoredinmulti‐
plefilesaccordingtothe
typeofthedata.I/O
mightworkbetterif
dataisstoredinsepa‐
ratefilesbasedonthe
typeofdata.TheSplit
driverisaspecialcase
ofthisdriver.
H5Pset_fapl_multi
Split H5FD_SPLIT Thisfiledriversplitsa
fileintotwoparts.One
partstoresmetadata,
andtheotherpart
storesrawdata.This
splittingafileintotwo
partsisalimitedcaseof
theMultidriver.
H5Pset_fapl_split
Table3‐2.Supportedfiledrivers
Driver
Name
Driver
Identifier
Description RelatedAPI
HDF5User’sGuide TheHDF5File
TheHDFGroup 65
Formoreinformation,seetheHDF5ReferenceManualentriesforthefunctioncallsshowninthecolumn
ontherightinthetableabove.
Notethatthelow‐levelfiledriversmanagealternativefilestoragelayouts.Datasetstoragelayouts(chunk‐
ing,compression,andexternaldatasetstorage)aremanagedindependentlyoffilestoragelayouts.
Ifanapplicationrequiresaspecial‐purposelow‐leveldriver,theVFLprovidesapublicAPIforcreatingone.
Formoreinformationonhowtocreateadriver,see“VirtualFileLayer”and“ListofVFLFunctions”inthe
HDF5TechnicalNotes.
3.11.1.IdentifyingthePreviously‐usedFileDriver
WhencreatinganewHDF5file,nohistoryexists,sothefiledrivermustbespecifiedifitistobeotherthan
thedefault.
Whenopeningexistingfiles,however,theapplicationmayneedtodeterminewhichlow‐leveldriverwas
usedtocreatethefile.ThefunctionH5Pget_driverisusedforthispurpose.Seetheexamplebelow.
H5Pget_driverreturnsaconstantidentifyingthelow‐leveldriverfortheaccesspropertylistfapl_id.
Forexample,ifthefilewascreatedwiththePOSIX(akaSEC2)driver,H5Pget_driverreturnsH5F-
D_SEC2.
Parallel H5FD_MPIO Thisisthestandard
HDF5filedriverforpar‐
allelfilesystems.This
driverusestheMPI
standardforbothcom‐
municationandfileI/O.
H5Pset_fapl_mpio
Parallel
POSIX
H5FD_MPIPOSIX Thisdriverisnolonger
available.
Stream H5FD_STREAM Thisdriverisnolonger
available.
hid_t H5Pget_driver (hid_t fapl_id)
CodeExample3‐5.Identifyingadriver
Table3‐2.Supportedfiledrivers
Driver
Name
Driver
Identifier
Description RelatedAPI
HDF5User’sGuide TheHDF5File
TheHDFGroup 66
IftheapplicationopensanHDF5filewithoutbothdeterminingthedriverusedtocreatethefileandset‐
tinguptheuseofthatdriver,theHDF5Librarywillexaminethesuperblockandthedriverdefinitionblock
toidentifythedriver.SeetheHDF5FileFormatSpecificationfordetaileddescriptionsofthesuperblock
andthedriverdefinitionblock.
3.11.2.ThePOSIX(akaSEC2)Driver
ThePOSIXdriver,H5FD_SEC2,usesfunctionsfromsection2ofthePOSIXmanualtoaccessunbuffered
filesstoredonalocalfilesystem.ThisdriverisalsoknownastheSEC2driver.TheHDF5Librarybuffers
metadataregardlessofthelow‐leveldriver,butusingthisdriverpreventsdatafrombeingbufferedagain
bythelowestlayersofthelibrary.
ThefunctionH5Pset_fapl_sec2setsthefileaccesspropertiestousethePOSIXdriver.Seetheexample
below.
Anypreviously‐defineddriverpropertiesareerasedfromthepropertylist.
Additionalparametersmaybeaddedtothisfunctioninthefuture.Sincetherearenoadditionalvariable
settingsassociatedwiththePOSIXdriver,thereisnoH5Pget_fapl_sec2function.
3.11.3.TheDirectDriver
TheDirectdriver,H5FD_DIRECT,functionslikethePOSIXdriverexceptthatdataiswrittentoorreadfrom
thefilesynchronouslywithoutbeingcachedbythesystem.
ThefunctionsH5Pset_fapl_directandH5Pget_fapl_directareusedtomanagefileaccessprop‐
erties.Seetheexamplebelow.
herr_t H5Pset_fapl_sec2 (hid_t fapl_id)
CodeExample3‐6.UsingthePOSIX,akaSEC2,driver
herr_t H5Pset_fapl_direct( hid_t fapl_id, size_t alignment,
size_t block_size, size_t cbuf_size )
herr_t H5Pget_fapl_direct( hid_t fapl_id, size_t *alignment,
size_t *block_size, size_t *cbuf_size )
CodeExample3‐7.UsingtheDirectdriver
HDF5User’sGuide TheHDF5File
TheHDFGroup 67
H5Pset_fapl_directsetsthefileaccesspropertiestousetheDirectdriver;anypreviouslydefined
driverpropertiesareerasedfromthepropertylist.H5Pget_fapl_directretrievesthefileaccessprop‐
ertiesusedwiththeDirectdriver.fapl_idisthefileaccesspropertylistidentifier.alignmentisthe
memoryalignmentboundary.block_sizeisthefilesystemblocksize.cbuf_sizeisthecopybuffer
size.
Additionalparametersmaybeaddedtothisfunctioninthefuture.
3.11.4.TheLogDriver
TheLogdriver,H5FD_LOG,isdesignedforsituationswhereitisnecessarytologfileaccessactivity.
ThefunctionH5Pset_fapl_logisusedtomanageloggingproperties.Seetheexamplebelow.
H5Pset_fapl_logsetsthefileaccesspropertylisttousetheLogdriver.Fileaccesscharacteristicsare
identicaltoaccessviathePOSIXdriver.Anypreviouslydefineddriverpropertiesareerasedfromtheprop‐
ertylist.
Logrecordsarewrittentothefilelogfile.
Thelogginglevelssetwiththeverbosityparameterareshowninthetablebelow.
ThereisnoH5Pget_fapl_logfunction.
Additionalparametersmaybeaddedtothisfunctioninthefuture.
herr_t H5Pset_fapl_log (hid_t fapl_id, const char *logfile,
unsigned int flags, size_t buf_size)
CodeExample3‐8.Loggingfileaccess
Table3‐3.Logginglevels
Level Comments
0Performsnologging.
1Recordswherewritesandreadsoccurinthefile.
2Recordswherewritesandreadsoccurinthefileandwhatkindofdataiswrit‐
tenateachlocation.Thisincludesrawdataoranyofseveraltypesofmetadata
(objectheaders,superblock,B‐treedata,localheaders,orglobalheaders).
HDF5User’sGuide TheHDF5File
TheHDFGroup 68
3.11.5.TheWindowsDriver
TheWindowsdriver,H5FD_WINDOWS,wasmodifiedinHDF5‐1.8.8tobeawrapperofthePOSIXdriver,
H5FD_SEC2.Inotherwords,iftheWindowsdriversisused,anyfileI/Owillinsteadusethefunctionality
ofthePOSIXdriver.Thischangeshouldbetransparenttoalluserapplications.TheWindowsdriverused
tobethedefaultdriverforWindowssystems.ThePOSIXdriverisnowthedefault.
ThefunctionH5Pset_fapl_windowssetsthefileaccesspropertiestousetheWindowsdriver.Seethe
examplebelow.
Anypreviously‐defineddriverpropertiesareerasedfromthepropertylist.
Additionalparametersmaybeaddedtothisfunctioninthefuture.Sincetherearenoadditionalvariable
settingsassociatedwiththePOSIXdriver,thereisnoH5Pget_fapl_windowsfunction.
3.11.6.TheSTDIODriver
TheSTDIOdriver,H5FD_STDIO,accessespermanentfilesinalocalfilesystemlikethePOSIXdriverdoes.
TheSTDIOdriveralsohasanadditionallayerofbufferingbeneaththeHDF5Library.
ThefunctionH5Pset_fapl_stdiosetsthefileaccesspropertiestousetheSTDIOdriver.Seetheexam‐
plebelow.
Anypreviouslydefineddriverpropertiesareerasedfromthepropertylist.
Additionalparametersmaybeaddedtothisfunctioninthefuture.Sincetherearenoadditionalvariable
settingsassociatedwiththeSTDIOdriver,thereisnoH5Pget_fapl_stdiofunction.
3.11.7.TheMemory(akaCore)Driver
Thereareseveralsituationsinwhichitisreasonable,sometimesevenrequired,tomaintainafileentirely
insystemmemory.Youmightwanttodosoif,forexample,eitherofthefollowingconditionsapply:
herr_t H5Pset_fapl_windows (hid_t fapl_id)
CodeExample3‐9.UsingtheWindowsdriver
herr_t H5Pset_fapl_stdio (hid_t fapl_id)
CodeExample3‐10.UsingtheSTDIOdriver
HDF5User’sGuide TheHDF5File
TheHDFGroup 69
•Performancerequirementsaresostringentthatdisklatencyisalimitingfactor
•Youareworkingwithsmall,temporaryfilesthatwillnotberetainedand,thus,neednotbewrit‐
tentostoragemedia
TheMemorydriver,H5FD_CORE,providesamechanismforcreatingandmanagingsuchin‐memoryfiles.
ThefunctionsH5Pset_fapl_coreandH5Pget_fapl_coremanagefileaccessproperties.Seethe
examplebelow.
H5Pset_fapl_coresetsthefileaccesspropertylisttousetheMemorydriver;anypreviouslydefined
driverpropertiesareerasedfromthepropertylist.
Memoryforthefilewillalwaysbeallocatedinunitsofthespecifiedblock_size.
Thebacking_storeBooleanflagissetwhenthein‐memoryfileiscreated.backing_storeindicates
whethertowritethefilecontentstodiskwhenthefileisclosed.Ifbacking_storeissetto1(TRUE),the
filecontentsareflushedtoafilewiththesamenameasthein‐memoryfilewhenthefileisclosedor
accesstothefileisterminatedinmemory.Ifbacking_storeissetto0(FALSE),thefileisnotsaved.
TheapplicationisallowedtoopenanexistingfilewiththeH5FD_COREdriver.WhileusingH5Fopento
openanexistingfile,ifbacking_storeissetto1andtheflagforH5FopenissettoH5F_ACC_RDWR,
changestothefilecontentswillbesavedtothefilewhenthefileisclosed.Ifbacking_storeissetto0
andtheflagforH5FopenissettoH5F_ACC_RDWR,changestothefilecontentswillbelostwhenthefile
isclosed.IftheflagforH5FopenissettoH5F_ACC_RDONLY,nochangetothefilewillbeallowedeither
inmemoryoronfile.
IfthefileaccesspropertylistissettousetheMemorydriver,H5Pget_fapl_corewillreturnblock_-
sizeandbacking_storewiththerelevantfileaccesspropertysettings.
Notethefollowingimportantpointsregardingin‐memoryfiles:
•Localtemporaryfilesarecreatedandaccesseddirectlyfrommemorywithouteverbeingwritten
todisk
•Totalfilesizemustnotexceedtheavailablevirtualmemory
•OnlyoneHDF5fileidentifiercanbeopenedforthefile,theidentifierreturnedbyH5Fcreateor
H5Fopen
•Thechangestothefilewillbediscardedwhenaccessisterminatedunlessbacking_storeisset
to1
Additionalparametersmaybeaddedtothesefunctionsinthefuture.
herr_t H5Pset_fapl_core (hid_t access_properties,
size_t block_size, hbool_t backing_store)
herr_t H5Pget_fapl_core (hid_t access_properties,
size_t *block_size), hbool_t *backing_store)
CodeExample3‐11.Managingfileaccessforin‐memoryfiles
TheHDF5File HDF5User’sGuide
70 TheHDFGroup
Seethe"HDF5FileImageOperations"sectionforinformationonmoreadvancedusageoftheMemoryfile
driver,andseethe"ModifiedRegionWrites"sectionforinformationonhowtosetwriteoperationsso
thatonlymodifiedregionsarewrittentostorage.
3.11.8.TheFamilyDriver
HDF5filescanbecomequitelarge,andthiscancreateproblemsonsystemsthatdonotsupportfiles
largerthan2gigabytes.TheHDF5filefamilymechanismisdesignedtosolvetheproblemsthiscreatesby
splittingtheHDF5fileaddressspaceacrossseveralsmallerfiles.Thisstructuredoesnotaffecthowmeta‐
dataandrawdataarestored:theyaremixedintheaddressspacejustastheywouldbeinasingle,contig‐
uousfile.
HDF5applicationsaccessafamilyoffilesviatheFamilydriver,H5FD_FAMILY.ThefunctionsH5Pset_-
fapl_familyandH5Pget_fapl_familyareusedtomanagefilefamilyproperties.Seetheexample
below.
Eachmemberofthefamilyisthesamelogicalsizethoughthesizeanddiskstoragereportedbyfilesystem
listingtoolsmaybesubstantiallysmaller.Examplesoffilesystemlistingtoolsare’ls -l’onaUnixsys‐
temorthedetailedfolderlistingonanAppleMacintoshorMicrosoftWindowssystem.Thenamepassed
toH5FcreateorH5Fopenshouldincludeaprintf(3c)‐styleintegerformatspecifierwhichwillbe
replacedwiththefamilymembernumber.Thefirstfamilymemberisnumberedzero(0).
H5Pset_fapl_familysetstheaccesspropertiestousetheFamilydriver;anypreviouslydefineddriver
propertiesareerasedfromthepropertylist.member_propertieswillserveasthefileaccessproperty
listforeachmemberofthefilefamily.memb_sizespecifiesthelogicalsize,inbytes,ofeachfamilymem‐
ber.memb_sizeisusedonlywhencreatinganewfileortruncatinganexistingfile;otherwisethemem‐
bersizeisdeterminedbythesizeofthefirstmemberofthefamilybeingopened.Note:Ifthesizeofthe
off_ttypeisfourbytes,themaximumfamilymembersizeisusually2^31‐1becausethebyteatoffset
2,147,483,647isgenerallyinaccessible.
H5Pget_fapl_familyisusedtoretrievefilefamilyproperties.Ifthefileaccesspropertylistissettouse
theFamilydriver,member_propertieswillbereturnedwithapointertoacopyoftheappropriate
memberaccesspropertylist.Ifmemb_sizeisnon‐null,itwillcontainthelogicalsize,inbytes,offamily
members.
Additionalparametersmaybeaddedtothesefunctionsinthefuture.
herr_t H5Pset_fapl_family (hid_t fapl_id,
hsize_t memb_size, hid_t member_properties)
herr_t H5Pget_fapl_family (hid_t fapl_id,
hsize_t *memb_size, hid_t *member_properties)
CodeExample3‐12.Managingfilefamilyproperties
HDF5User’sGuide TheHDF5File
TheHDFGroup 71
3.11.8.1.UnixToolsandanHDF5Utility
Itoccasionallybecomesnecessarytorepartitionafilefamily.Acommand‐lineutilityforthispurpose,
h5repart,isdistributedwiththeHDF5Library.
h5repart [-v] [-b block_size[suffix]] [-m member_size[suffix]] source
destination
h5repartrepartitionsanHDF5filebycopyingthesourcefileorfilefamilytothedestinationfileorfile
family,preservingholesintheunderlyingUNIXfiles.Familiesareusedforthesourceand/ordestinationif
thenameincludesaprintf‐styleintegerformatsuchas%d.The-vswitchprintsinputandoutputfile
namesonthestandarderrorstreamforprogressmonitoring,-bsetstheI/Oblocksize(thedefaultis
1KB),and-msetstheoutputmembersizeifthedestinationisafamilyname(thedefaultis1GB).
block_sizeandmember_sizemaybesuffixedwiththelettersg,m,orkforGB,MB,orKBrespectively.
Theh5repartutilityisdescribedontheToolspageoftheHDF5ReferenceManual.
AnexistingHDF5filecanbesplitintoafamilyoffilesbyrunningthefilethroughsplit(1)onaUNIXsys‐
temandnumberingtheoutputfiles.However,theHDF5Libraryislazyaboutextendingthesizeoffamily
members,soavalidfilecannotgenerallybecreatedbyconcatenationofthefamilymembers.
Splittingthefileandrejoiningthesegmentsbyconcatenation(split(1)andcat(1)onUNIXsystems)
doesnotgeneratefileswithholes;holesarepreservedonlythroughtheuseofh5repart.
3.11.9.TheMultiDriver
Insomecircumstances,itisusefultoseparatemetadatafromrawdataandsometypesofmetadatafrom
othertypesofmetadata.SituationsthatwouldbenefitfromuseoftheMultidriverincludethefollowing:
•Innetworkedsituationswherethesmallmetadatafilescanbekeptonlocaldisksbutlargerraw
datafilesmustbestoredonremotemedia
•Incaseswheretherawdataisextremelylarge
•InsituationsrequiringfrequentaccesstometadataheldinRAMwhiletherawdatacanbeeffi‐
cientlyheldondisk
Ineithercase,accesstothemetadataissubstantiallyeasierwiththesmaller,andpossiblymorelocalized,
metadatafiles.Thisoftenresultsinimprovedapplicationperformance.
TheMultidriver,H5FD_MULTI,providesamechanismforsegregatingrawdataanddifferenttypesof
metadataintomultiplefiles.ThefunctionsH5Pset_fapl_multiandH5Pget_fapl_multiareusedto
manageaccesspropertiesforthesemultiplefiles.Seetheexamplebelow.
TheHDF5File HDF5User’sGuide
72 TheHDFGroup
H5Pset_fapl_multisetsthefileaccesspropertiestousetheMultidriver;anypreviouslydefineddriver
propertiesareerasedfromthepropertylist.WiththeMultidriverinvoked,theapplicationwillprovidea
basenametoH5FopenorH5Fcreate.Thefileswillbenamedbythatbasenameasmodifiedbytherule
indicatedinmemb_name.Fileaccesswillbegovernedbythefileaccesspropertylistmemb_properties.
SeeH5Pset_fapl_multiandH5Pget_fapl_multiintheHDF5ReferenceManualfordescriptionsof
thesefunctionsandtheirusage.
Additionalparametersmaybeaddedtothesefunctionsinthefuture.
3.11.10.TheSplitDriver
TheSplitdriver,H5FD_SPLIT,isalimitedcaseoftheMultidriverwhereonlytwofilesarecreated.One
fileholdsmetadata,andtheotherfileholdsrawdata.
ThefunctionH5Pset_fapl_splitisusedtomanageSplitfileaccessproperties.Seetheexamplebelow.
H5Pset_fapl_splitsetsthefileaccesspropertiestousetheSplitdriver;anypreviouslydefineddriver
propertiesareerasedfromthepropertylist.
herr_t H5Pset_fapl_multi (hid_t fapl_id,
const H5FD_mem_t *memb_map,
const hid_t *memb_fapl,
const char * const *memb_name,
const haddr_t *memb_addr,
hbool_t relax)
herr_t H5Pget_fapl_multi (hid_t fapl_id,
const H5FD_mem_t *memb_map,
const hid_t *memb_fapl,
const char **memb_name,
const haddr_t *memb_addr,
hbool_t *relax)
CodeExample3‐13.Managingaccesspropertiesformultiplefiles
herr_t H5Pset_fapl_split (hid_t access_properties,
const char *meta_extension, hid_t meta_properties,
const char *raw_extension, hid_t raw_properties
CodeExample3‐14.Managingaccesspropertiesforsplitfiles
HDF5User’sGuide TheHDF5File
TheHDFGroup 73
WiththeSplitdriverinvoked,theapplicationwillprovideabasefilenamesuchasfile_nametoH5F-
createorH5Fopen.Themetadataandrawdatafilesinstoragewillthenbenamed
file_name.meta_extensionandfile_name.raw_extension,respectively.Forexample,if
meta_extensionisdefinedas.metaandraw_extensionisdefinedas.raw,thefinalfilenameswill
befile_name.metaandfile_name.raw.
Eachfilecanhaveitsownfileaccesspropertylist.Thisallowsthecreativeuseofotherlow‐levelfiledriv‐
ers.Forinstance,themetadatafilecanbeheldinRAMandaccessedviatheMemorydriverwhiletheraw
datafileisstoredondiskandaccessedviathePOSIXdriver.Metadatafileaccesswillbegovernedbythe
fileaccesspropertylistinmeta_properties.Rawdatafileaccesswillbegovernedbythefileaccess
propertylistinraw_properties.
Additionalparametersmaybeaddedtothesefunctionsinthefuture.Sincetherearenoadditionalvari‐
ablesettingsassociatedwiththeSplitdriver,thereisnoH5Pget_fapl_splitfunction.
3.11.11.TheParallelDriver
Parallelenvironmentsrequireaparallellow‐leveldriver.HDF5’sdefaultdriverforparallelsystemsiscalled
theParalleldriver,H5FD_MPIO.ThisdriverusestheMPIstandardforbothcommunicationandfileI/O.
ThefunctionsH5Pset_fapl_mpioandH5Pget_fapl_mpioareusedtomanagefileaccessproperties
fortheH5FD_MPIOdriver.Seetheexamplebelow.
ThefileaccesspropertiesmanagedbyH5Pset_fapl_mpioandretrievedbyH5Pget_fapl_mpioare
theMPIcommunicator,comm,andtheMPIinfoobject,info.commandinfoareusedforfileopen.info
isaninformationobjectmuchlikeanHDF5propertylist.BotharedefinedinMPI_FILE_OPENofMPI‐2.
Thecommunicatorandtheinfoobjectaresavedinthefileaccesspropertylistfapl_id.fapl_idcan
thenbepassedtoMPI_FILE_OPENtocreateand/oropenthefile.
H5Pset_fapl_mpioandH5Pget_fapl_mpioareavailableonlyintheparallelHDF5Libraryandarenot
collectivefunctions.TheParalleldriverisavailableonlyintheparallelHDF5Library.
Additionalparametersmaybeaddedtothesefunctionsinthefuture.
herr_t H5Pset_fapl_mpio (hid_t fapl_id, MPI_Comm comm,
MPI_info info)
herr_t H5Pget_fapl_mpio (hid_t fapl_id, MPI_Comm *comm,
MPI_info *info)
CodeExample3‐15.Managingparallelfileaccessproperties
TheHDF5File HDF5User’sGuide
74 TheHDFGroup
3.12.CodeExamplesforOpeningandClosingFiles
3.12.1.ExampleUsingtheH5F_ACC_TRUNCFlag
ThefollowingexampleusestheH5F_ACC_TRUNCflagwhenitcreatesanewfile.Thedefaultfilecreation
andfileaccesspropertiesarealsoused.UsingH5F_ACC_TRUNCmeansthefunctionwilllookforanexist‐
ingfilewiththenamespecifiedbythefunction.Inthiscase,thatnameisFILE.Ifthefunctiondoesnot
findanexistingfile,itwillcreateone.Ifitdoesfindanexistingfile,itwillemptythefileinpreparationfor
anewsetofdata.Theidentifierforthe"new"filewillbepassedbacktotheapplicationprogram.For
moreinformation,see"FileAccessModes"onpage 45.
3.12.2.ExamplewiththeFileCreationPropertyList
Theexamplebelowshowshowtocreateafilewith64‐bitobjectoffsetsandlengths.
hid_t file; /* identifier */
/* Create a new file using H5F_ACC_TRUNC access, default
* file creation properties, and default file access
*/ properties.
file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);
/* Close the file. */
status = H5Fclose(file);
CodeExample3‐16.Creatingafilewithdefaultcreationandaccessproperties
HDF5User’sGuide TheHDF5File
TheHDFGroup 75
3.12.3.ExamplewiththeFileAccessPropertyList
ThisexampleshowshowtoopenanexistingfileforindependentdatasetsaccessbyMPIparallelI/O:
hid_t create_plist;
hid_t file_id;
create_plist = H5Pcreate(H5P_FILE_CREATE);
H5Pset_sizes(create_plist, 8, 8);
file_id = H5Fcreate(“test.h5”, H5F_ACC_TRUNC,
create_plist, H5P_DEFAULT);
.
.
.
H5Fclose(file_id);
CodeExample3‐17.Creatingafilewith64‐bitoffsets
hid_t access_plist;
hid_t file_id;
access_plist = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_fapl_mpi(access_plist, MPI_COMM_WORLD,
MPI_INFO_NULL);
/* H5Fopen must be called collectively */
file_id = H5Fopen(“test.h5”, H5F_ACC_RDWR, access_plist);
.
.
.
/* H5Fclose must be called collectively */
H5Fclose(file_id);
CodeExample3‐18.OpeninganexistingfileforparallelI/O
TheHDF5File HDF5User’sGuide
76 TheHDFGroup
3.13.WorkingwithMultipleHDF5Files
MultipleHDF5filescanbeassociatedsothatthefilescanbeworkedwithasthoughalltheinformationis
inasingleHDF5file.AtemporaryassociationcanbesetupbymeansoftheH5Fmountfunction.Aperma‐
nentassociationcanbesetupbymeansoftheexternallinkfunctionH5Lcreate_external.
ThepurposeofthissectionistodescribewhathappenswhentheH5Fmountfunctionisusedtomount
onefileonanother.
Whenafileismountedonanother,themountedfileismountedatagroup,andtherootgroupofthe
mountedfiletakestheplaceofthatgroupuntilthemountedfileisunmountedoruntilthefilesareclosed.
Thefigurebelowshowstwofilesbeforeoneismountedontheother.File1hastwogroupsandthree
datasets.ThegroupthatisthetargetoftheAlinkhaslinks,ZandY, totwoofthedatasets.Thegroupthat
isthetargetoftheBlinkhasalink,W,totheotherdataset.File2hasthreegroupsandthreedatasets.The
groupsinFile2arethetargetsoftheAA,BB,andCClinks.ThedatasetsinFile2arethetargetsoftheZZ,
YY,andWWlinks.
ThefigurebelowshowsthetwofilesafterFile2hasbeenmountedFile1atthegroupthatisthetargetof
theBlink.
Figure3‐3.Twoseparatefiles
HDF5User’sGuide TheHDF5File
TheHDFGroup 77
Note:Inthefigureabove,thedatasetthatisthetargetoftheWlinkisnotshown.Thatdatasetismaskedbythe
mountedfile.
Ifafileismountedonagroupthathasmembers,thosemembersarehiddenuntilthemountedfileis
unmounted.Therearetwowaysaroundthisifyouneedtoworkwithagroupmember.Oneistomount
thefileonanemptygroup.Anotheristoopenthegroupmemberbeforeyoumountthefile.Openingthe
groupmemberwillreturnanidentifierthatyoucanusetolocatethegroupmember.
TheexamplebelowshowshowH5FmountmightbeusedtomountFile2ontoFile1.
Note:Inthecodeexampleabove,loc_idisthefileidentifierforFile1,/BisthelinkpathtothegroupwhereFile2is
mounted,child_idisthefileidentifierforFile2,andplist_idisapropertylistidentifier.
Formoreinformation,see"HDF5Groups"onpage 79.SeetheentriesforH5Fmount,H5Funmount,and
H5Lcreate_externalintheHDF5ReferenceManual.
Figure3‐4.File2mountedonFile1
status = H5Fmount(loc_id, "/B", child_id, plist_id)
CodeExample3‐19.UsingH5Fmount
TheHDF5File HDF5User’sGuide
78 TheHDFGroup
HDF5User’sGuide HDF5Groups
TheHDFGroup 79
4.HDF5Groups
4.1.Introduction
AssuggestedbythenameHierarchicalDataFormat,anHDF5fileishierarchicallystructured.TheHDF5
groupandlinkobjectsimplementthishierarchy.
Inthesimpleandmostcommoncase,thefilestructureisatreestructure;inthegeneralcase,thefile
structuremaybeadirectedgraphwithadesignatedentrypoint.Thetreestructureisverysimilartothe
filesystemstructuresemployedonUNIXsystems,directoriesandfiles,andonAppleMacintoshandMic‐
rosoftWindowssystems,foldersandfiles.HDF5groupsareanalogoustothedirectoriesandfolders;HDF5
datasetsareanalogoustothefiles.
TheoneveryimportantdifferencebetweentheHDF5filestructureandtheabove‐mentionedfilesystem
analogsisthatHDF5groupsarelinkedasadirectedgraph,allowingcircularreferences;thefilesystems
arestrictlyhierarchical,allowingnocircularreferences.Thefiguresbelowillustratetherangeofpossibili‐
ties.
Inthefirstfigurebelow,thegroupstructureisstrictlyhierarchical,identicaltothefilesystemanalogs.
Inthenexttwofiguresbelow,thestructuretakesadvantageofthedirectedgraph’sallowanceofcircular
references.Inthesecondfigure,GroupAisnotonlyamemberoftherootgroup,/,butamemberof
GroupC.SinceGroupCisamemberofGroupBandGroupBisamemberofGroupA,Dataset1canbe
accessedbymeansofthecircularreference/Group A/Group B/Group C/Group A/Dataset1.The
thirdfigurebelowillustratesanextremecaseinwhichGroupBisamemberofitself,enablingareference
toamemberdatasetsuchas/Group A/Group B/Group B/Group B/Dataset2.
Figure4‐1.Afilewithastrictlyhierarchicalgroupstructure
HDF5User’sGuide HDF5Groups
TheHDFGroup 80
Asbecomesapparentuponreflection,directedgraphstructurescanbecomequitecomplex;cautionis
advised!
Thebalanceofthischapterdiscussesthefollowingtopics:
•TheHDF5groupobject(oragroup)anditsstructureinmoredetail
•HDF5linkobjects(orlinks)
•Theprogrammingmodelforworkingwithgroupsandlinks
Figure4‐2.Afilewithacircularreference
Figure4‐3.Afilewithonegroupasamemberofitself
HDF5User’sGuide HDF5Groups
TheHDFGroup 81
•HDF5functionsprovidedforworkingwithgroups,groupmembers,andlinks
• Retrievinginformationaboutobjectsinagroup
•DiscoveryofthestructureofanHDF5fileandthecontainedobjects
• Examplesoffilestructures
4.2.DescriptionoftheGroupObject
4.2.1.TheGroupObject
Abstractly,anHDF5groupcontainszeroormoreobjectsandeveryobjectmustbeamemberofatleast
onegroup.Therootgroup,thesoleexception,maynotbelongtoanygroup.
Groupmembershipisactuallyimplementedvialinkobjects.Seethefigureabove.Alinkobjectisowned
byagroupandpointstoanamedobject.Eachlinkhasaname,andeachlinkpointstoexactlyoneobject.
Eachnamedobjecthasatleastoneandpossiblymanylinkstoit.
Therearethreeclassesofnamedobjects:group,dataset,andcommitteddatatype(formerlycalled
nameddatatype).Seethefigurebelow.Eachoftheseobjectsisthememberofatleastonegroup,which
meansthereisatleastonelinktoit.
Figure4‐4.AbstractmodeloftheHDF5groupobject
HDF5User’sGuide HDF5Groups
TheHDFGroup 82
Theprimaryoperationsonagrouparetoaddandremovemembersandtodiscovermemberobjects.
Theseabstractoperations,aslistedinthefigurebelow,areimplementedintheH5GAPIs.Formoreinfor‐
mation,see"GroupFunctionSummaries"onpage 87.
Toaddanddeletemembersofagroup,linksfromthegrouptoexistingobjectsinthefilearecreatedand
deletedwiththelinkandunlinkoperations.Whenanewnamedobjectiscreated,theHDF5Library
executesthelinkoperationinthebackgroundimmediatelyaftercreatingtheobject(inotherwords,a
newobjectisaddedasamemberofthegroupinwhichitiscreatedwithoutfurtheruserintervention).
Giventhenameofanobject,theget_object_infomethodretrievesadescriptionoftheobject,including
thenumberofreferencestoit.Theiteratemethoditeratesthroughthemembersofthegroup,returning
thenameandtypeofeachobject.
Figure4‐5.Classesofnamedobjects
Figure4‐6.Thegroupobject
HDF5User’sGuide HDF5Groups
TheHDFGroup 83
EveryHDF5filehasasinglerootgroup,withthename/.TherootgroupisidenticaltoanyotherHDF5
group,except:
•TherootgroupisautomaticallycreatedwhentheHDF5fileiscreated(H5Fcreate).
•Therootgrouphasnoparent,butbyconventionhasareferencecountof1.
•Therootgroupcannotbedeleted(inotherwords,unlinked)!
4.2.2.TheHierarchyofDataObjects
AnHDF5fileisorganizedasarooted,directedgraphusingHDF5groupobjects.Thenameddataobjects
arethenodesofthegraph,andthelinksarethedirectedarcs.Eacharcofthegraphhasaname,withthe
specialname/reservedfortherootgroup.Newobjectsarecreatedandtheninsertedintothegraphwith
alinkoperationthatisautomaticallyexecutedbythelibrary;existingobjectsareinsertedintothegraph
withalinkoperationexplicitlycalledbytheuser,whichcreatesanamedlinkfromagrouptotheobject.
Anobjectcanbethetargetofmorethanonelink.
Thenamesonthelinksmustbeuniquewithineachgroup,buttheremaybemanylinkswiththesame
nameindifferentgroups.Theseareunambiguous,becausesomeancestormusthaveadifferentname,or
elsetheyarethesameobject.Thegraphisnavigatedwithpathnames,analogoustoUnixfilesystems.For
moreinformation,see"HDF5PathNames"onpage 84.Anobjectcanbeopenedwithafullpathstarting
attherootgroup,orwitharelativepathandastartingpoint.Thatstartingpointisalwaysagroup,though
itmaybethecurrentworkinggroup,anotherspecifiedgroup,ortherootgroupofthefile.Notethatall
pathsarerelativetoasingleHDF5file.Inthissense,anHDF5fileisanalogoustoasingleUNIXfilesystem.4
Itisimportanttonotethat,justliketheUNIXfilesystem,HDF5objectsdonothavenames,thenamesare
associatedwithpaths.Anobjecthasanobjectidentifierthatisuniquewithinthefile,butasingleobject
mayhavemanynamesbecausetheremaybemanypathstothesameobject.Anobjectcanberenamed,
ormovedtoanothergroup,byaddinganddeletinglinks.Inthiscase,theobjectitselfnevermoves.For
thatmatter,membershipinagrouphasnoimplicationforthephysicallocationofthestoredobject.
Deletingalinktoanobjectdoesnotnecessarilydeletetheobject.Theobjectremainsavailableaslongas
thereisatleastonelinktoit.Afteralllinkstoanobjectaredeleted,itcannolongerbeopened,andthe
storagemaybereclaimed.
Itisalsoimportanttorealizethatthelinkingmechanismcanbeusedtoconstructverycomplexgraphsof
objects.Forexample,itispossibleforanobjecttobesharedbetweenseveralgroupsandeventohave
morethanonenameinthesamegroup.Itisalsopossibleforagrouptobeamemberofitself,ortocreate
othercyclesinthegraph,suchasinthecasewhereachildgroupislinkedtooneofitsancestors.
HDF5alsohassoftlinkssimilartoUNIXsoftlinks.Asoftlinkisanobjectthathasanameandapathname
forthetargetobject.Thesoftlinkcanbefollowedtoopenthetargetofthelinkjustlikearegularorhard
link.Thedifferencesarethatthehardlinkcannotbecreatedifthetargetobjectdoesnotexistandit
alwayspointstothesameobject.Asoftlinkcanbecreatedwithanypathname,whetherornottheobject
4. ItcouldbesaidthatHDF5extendstheorganizingconceptsofafilesystemtotheinternalstructureofa
singlefile.
HDF5User’sGuide HDF5Groups
TheHDFGroup 84
exists;itmayormaynot,therefore,bepossibletofollowasoftlink.Furthermore,asoftlink’starget
objectmaybechanged.
4.2.3.HDF5PathNames
ThestructureoftheHDF5fileconstitutesthenamespacefortheobjectsinthefile.Apathnameisastring
ofcomponentsseparatedbyslashes(/).Eachcomponentisthenameofahardorsoftlinkwhichpointsto
anobjectinthefile.Theslashnotonlyseparatesthecomponents,butindicatestheirhierarchicalrelation‐
ship;thecomponentindicatedbythelinknamefollowingaslashisaalwaysamemberofthecomponent
indicatedbythelinknameprecedingthatslash.
Thefirstcomponentinthepathnamemaybeanyofthefollowing:
•Thespecialcharacterdot(.,aperiod),indicatingthecurrentgroup
•Thespecialcharacterslash(/),indicatingtherootgroup
•Anymemberofthecurrentgroup
ComponentlinknamesmaybeanystringofASCIIcharactersnotcontainingaslashoradot(/and.,
whicharereservedasnotedabove).However,usersareadvisedtoavoidtheuseofpunctuationandnon‐
printingcharacters,astheymaycreateproblemsforothersoftware.ThefigurebelowprovidesaBNF
grammarforHDF5pathnames.
Anobjectcanalwaysbeaddressedbyaeitherafullorabsolutepathname,startingattherootgroup,or
byarelativepathname,startinginaknownlocationsuchasthecurrentworkinggroup.Asnotedelse‐
where,agivenobjectmayhavemultiplefullandrelativepathnames.
Consider,forexample,thefileillustratedinthefigurebelow.Dataset1canbeidentifiedbyeitherof
theseabsolutepathnames:
/GroupA/Dataset1
/GroupA/GroupB/GroupC/Dataset1
SinceanHDF5fileisadirectedgraphstructure,andisthereforenotlimitedtoastricttreestructure,and
sincethisillustratedfileincludesthesortofcircularreferencethatadirectedgraphenables,Dataset1
canalsobeidentifiedbythisabsolutepathname:
PathName ::= AbsolutePathName | RelativePathName
Separator ::= "/" ["/"]*
AbsolutePathName ::= Separator [ RelativePathName ]
RelativePathName ::= Component [ Separator RelativePathName ]*
Component ::= "." | Characters
Characters ::= Character+ - { "." }
Character ::= {c: c Î { { legal ASCII characters } - {'/'} }
Figure4‐7.ABNFgrammarforHDF5pathnames
HDF5User’sGuide HDF5Groups
TheHDFGroup 85
/GroupA/GroupB/GroupC/GroupA/Dataset1
Alternatively,ifthecurrentworkinglocationisGroupB,Dataset1canbeidentifiedbyeitheroftheserel‐
ativepathnames:
GroupC/Dataset1
GroupC/GroupA/Dataset1
NotethatrelativepathnamesinHDF5donotemploythe../notation,theUNIXnotationindicatinga
parentdirectory,toindicateaparentgroup.
4.2.4.GroupImplementationsinHDF5
TheoriginalHDF5groupimplementationprovidedasingleindexedstructureforlinkstorage.Anewgroup
implementation,inHDF5Release1.8.0,enablesmoreefficientcompactstorageforverysmallgroups,
improvedlinkindexingforlargegroups,andotheradvancedfeatures.
•Theoriginalindexedformatremainsthedefault.LinksarestoredinaB‐treeinthegroup’slocal
heap.
•Groupscreatedinthenewcompact‐or‐indexedformat,theimplementationintroducedwith
Release1.8.0,canbetunedforperformance,switchingbetweenthecompactandindexedfor‐
matsatthresholdssetintheuserapplication.
•Thecompactformatwillconservefilespaceandprocessingoverheadwhenworkingwith
smallgroupsandisparticularlyvaluablewhenagroupcontainsnolinks.Linksarestoredasa
listofmessagesinthegroup’sheader.
Figure4‐8.Afilewithacircularreference
HDF5User’sGuide HDF5Groups
TheHDFGroup 86
•Theindexedformatwillyieldimprovedperformancewhenworkingwithlargegroups.Alarge
groupmaycontainthousandstomillionsofmembers.Linksarestoredinafractalheapand
indexedwithanimprovedB‐tree.
•Thenewimplementationalsoenablestheuseoflinknamesconsistingofnon‐ASCIIcharacter
sets(seeH5Pset_char_encoding)andisrequiredforalllinktypesotherthanhardorsoft
links;thelinktypesotherthanhardorsoftlinksareexternallinksanduser‐definedlinks(seethe
H5LAPIs).
Theoriginalgroupstructureandthenewerstructuresarenotdirectlyinteroperable.Bydefault,agroup
willbecreatedintheoriginalindexedformat.Anexistinggroupcanbechangedtoacompact‐or‐indexed
formatiftheneedarises;thereisnocapabilitytochangeback.Asstatedabove,onceinthecompact‐or‐
indexedformat,agroupcanswitchbetweencompactandindexedasneeded.
Groupswillbeinitiallycreatedinthecompact‐or‐indexedformatonlywhenoneormoreofthefollowing
conditionsismet:
•ThelowversionboundvalueofthelibraryversionboundspropertyhasbeensettoRelease1.8.0
orlaterinthefileaccesspropertylist(seeH5Pset_libver_bounds).Currently,thatwould
requireanH5Pset_libver_boundscallwiththelowparametersettoH5F_LIBVER_LATEST.
WhenthispropertyissetforanHDF5file,allobjectsinthefilewillbecreatedusingthelatest
availableformat;noeffortwillbemadetocreateafilethatcanbereadbyolderlibraries.
•Thecreationordertrackingproperty,H5P_CRT_ORDER_TRACKED,hasbeensetinthegroupcre‐
ationpropertylist(seeH5Pset_link_creation_order).
Anexistinggroup,currentlyintheoriginalindexedformat,willbeconvertedtothecompact‐or‐indexed
formatupontheoccurrenceofanyofthefollowingevents:
•Anexternaloruser‐definedlinkisinsertedintothegroup.
•Alinknamedwithastringcomposedofnon‐ASCIIcharactersisinsertedintothegroup.
Thecompact‐or‐indexedformatoffersperformanceimprovementsthatwillbemostnotableatthe
extremes(forexample,ingroupswithzeromembersandingroupswithtensofthousandsofmembers).
Butmeasurabledifferencesmaysometimesappearatathresholdaslowaseightgroupmembers.Since
theseperformancethresholdsandcriteriadifferfromapplicationtoapplication,tunablesettingsarepro‐
videdtogoverntheswitchbetweenthecompactandindexedformats(see
H5Pset_link_phase_change).Optimalthresholdswilldependontheapplicationandtheoperating
environment.
FutureversionsofHDF5willretaintheabilitytocreate,read,write,andmanipulateallgroupsstoredin
eithertheoriginalindexedformatorthecompact‐or‐indexedformat.
4.3.Usingh5dump
Youcanuseh5dump,thecommand‐lineutilitydistributedwithHDF5,toexamineafileforpurposeseither
ofdeterminingwheretocreateanobjectwithinanHDF5fileortoverifythatyouhavecreatedanobject
intheintendedplace.
HDF5User’sGuide HDF5Groups
TheHDFGroup 87
Inthecaseofthenewgroupcreatedlaterinthischapter,thefollowingh5dumpcommandwilldisplaythe
contentsofFileA.h5:
h5dump FileA.h5
Formoreinformation,see"CreatingaGroup"onpage 92.
Assumingthatthediscussedobjects,GroupAandGroupBaretheonlyobjectsthatexistinFileA.h5,
theoutputwilllooksomethinglikethefollowing:
HDF5 "FileA.h5" {
GROUP "/" {
GROUP GroupA {
GROUP GroupB {
}
}
}
}
h5dumpisdescribedonthe“HDF5Tools”pageoftheHDF5ReferenceManual.
TheHDF5DDLgrammarisdescribedinthedocumentDDLinBNFforHDF5.
4.4.GroupFunctionSummaries
Functionsthatcanbeusedwithgroups(H5Gfunctions)andpropertylistfunctionsthatcanusedwith
groups(H5Pfunctions)arelistedbelow.Anumberofgroupfunctionshavebeendeprecated.Mostof
thesehavebecomelink(H5L)orobject(H5O)functions.Thesereplacementfunctionsarealsolisted
below.
FunctionListing4‐1.Groupfunctions(H5G)
CFunction
FortranSubroutine
Purpose
H5Gcreate
h5gcreate_f
Createsanewemptygroupandgivesita
name.TheCfunctionisamacro:see“API
CompatibilityMacrosinHDF5.”
H5Gcreate_anon
h5gcreate_anon_f
Createsanewemptygroupwithoutlinkingit
intothefilestructure.
H5Gopen
h5gopen_f
Opensanexistinggroupformodificationand
returnsagroupidentifierforthatgroup.The
Cfunctionisamacro:see“APICompatibility
MacrosinHDF5.”
HDF5User’sGuide HDF5Groups
TheHDFGroup 88
H5Gclose
h5gclose_f
Closesthespecifiedgroup.
H5Gget_create_plist
h5gget_create_plist_f
Getsagroupcreationpropertylistidentifier.
H5Gget_info
h5gget_info_f
Retrievesinformationaboutagroup.Use
insteadofH5Gget_num_objs.
H5Gget_info_by_idx
h5gget_info_by_idx_f
Retrievesinformationaboutagroupaccord‐
ingtothegroup’spositionwithinanindex.
H5Gget_info_by_name
h5gget_info_by_name_f
Retrievesinformationaboutagroup.
(noCfunction)
h5gget_obj_info_idx_f
Returnsnameandtypeofthegroupmember
identifiedbyitsindex.Usewiththe
h5gn_members_ffunction.h5gget_ob-
j_info_idx_fandh5gn_members_fare
theFortranequivalentoftheCfunction
H5Literate.
(noCfunction)
h5gn_members_f
Returnsthenumberofgroupmembers.Use
withtheh5gget_obj_info_idx_ffunc‐
tion.
FunctionListing4‐2.Link(H5L)andobject(H5O)functions
CFunction
FortranSubroutine
Purpose
H5Lcreate_hard
h5lcreate_hard_f
Createsahardlinktoanobject.Replaces
H5GlinkandH5Glink2.
H5Lcreate_soft
h5lcreate_soft_f
Createsasoftlinktoanobject.Replaces
H5GlinkandH5Glink2.
H5Lcreate_external
h5lcreate_external_f
Createsasoftlinktoanobjectinadifferent
file.ReplacesH5GlinkandH5Glink2.
H5Lcreate_ud
(noFortransubroutine)
Createsalinkofauser‐definedtype.
H5Lget_val
(noFortransubroutine)
Returnsthevalueofasymboliclink.Replaces
H5Gget_linkval.
FunctionListing4‐1.Groupfunctions(H5G)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Groups
TheHDFGroup 89
H5Literate
h5literate_f
Iteratesthroughlinksinagroup.Replaces
H5Giterate.SeealsoH5Ovisitand
H5Lvisit.
H5Literate_by_name
h5literate_by_name_f
Iteratesthroughlinksinagroup.
H5Lvisit
(noFortransubroutine)
Recursivelyvisitsalllinksstartingfromaspec‐
ifiedgroup.
H5Ovisit
h5ovisit_f
Recursivelyvisitsallobjectsaccessiblefroma
specifiedobject.
H5Lget_info
h5lget_info_f
Returnsinformationaboutalink.Replaces
H5Gget_objinfo.
H5Oget_info
(noFortransubroutine)
Retrievesthemetadataforanobjectspecified
byanidentifier.ReplacesH5Gget_objinfo.
H5Lget_name_by_idx
h5lget_name_by_idx_f
Retrievesnameofthenthlinkinagroup,
accordingtotheorderwithinaspecifiedfield
orindex.ReplacesH5Gget_ob-
jname_by_idx.
H5Oget_info_by_idx
(noFortransubroutine)
Retrievesthemetadataforanobject,identi‐
fyingtheobjectbyanindexposition.
ReplacesH5Gget_objtype_by_idx.
H5Oget_info_by_name
h5oget_info_by_name_f
Retrievesthemetadataforanobject,identi‐
fyingtheobjectbylocationandrelative
name.
H5Oset_comment
(noFortransubroutine)
Setsthecommentforspecifiedobject.
ReplacesH5Gset_comment.
H5Oget_comment
(noFortransubroutine)
Getsthecommentforspecifiedobject.
ReplacesH5Gget_comment.
H5Ldelete
h5ldelete_f
Removesalinkfromagroup.Replaces
H5Gunlink.
H5Lmove
h5lmove_f
RenamesalinkwithinanHDF5file.Replaces
H5GmoveandH5Gmove2.
FunctionListing4‐2.Link(H5L)andobject(H5O)functions
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Groups
TheHDFGroup 90
FunctionListing4‐3.Groupcreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pall_filters_avail
(noFortransubroutine)
Verifiesthatallrequiredfiltersareavailable.
H5Pget_filter
h5pget_filter_f
Returnsinformationaboutafilterinapipe‐
line.TheCfunctionisamacro:see“APICom‐
patibilityMacrosinHDF5.”
H5Pget_filter_by_id
h5pget_filter_by_id_f
Returnsinformationaboutthespecifiedfilter.
TheCfunctionisamacro:see“APICompati‐
bilityMacrosinHDF5.”
H5Pget_nfilters
h5pget_nfilters_f
Returnsthenumberoffiltersinthepipeline.
H5Pmodify_filter
h5pmodify_filter_f
Modifiesafilterinthefilterpipeline.
H5Premove_filter
h5premove_filter_f
Deletesoneormorefiltersinthefilterpipe‐
line.
H5Pset_deflate
h5pset_deflate_f
Setsthedeflate(GNUgzip)compression
methodandcompressionlevel.
H5Pset_filter
h5pset_filter_f
Addsafiltertothefilterpipeline.
H5Pset_fletcher32
h5pset_fletcher32_f
SetsupuseoftheFletcher32checksumfilter.
H5Pset_fletcher32
h5pset_fletcher32_f
SetsupuseoftheFletcher32checksumfilter.
H5Pset_link_phase_change
h5pset_link_phase_change_f
Setstheparametersforconversionbetween
compactanddensegroups.
H5Pget_link_phase_change
h5pget_link_phase_change_f
Queriesthesettingsforconversionbetween
compactanddensegroups.
H5Pset_est_link_info
h5pset_est_link_info_f
Setsestimatednumberoflinksandlengthof
linknamesinagroup.
H5Pget_est_link_info
h5pget_est_link_info_f
Queriesdatarequiredtoestimaterequired
localheaporobjectheadersize.
H5Pset_nlinks
h5pset_nlinks_f
Setsmaximumnumberofsoftoruser‐defined
linktraversals.
HDF5User’sGuide HDF5Groups
TheHDFGroup 91
4.5.ProgrammingModelforGroups
Theprogrammingmodelforworkingwithgroupsisasfollows:
1. Createanewgrouporopenanexistingone.
H5Pget_nlinks
h5pget_nlinks_f
Retrievesthemaximumnumberoflinktra‐
versals.
H5Pset_link_creation_order
h5pset_link_creation_order_f
Setscreationordertrackingandindexingfor
linksinagroup.
H5Pget_link_creation_order
h5pget_link_creation_order_f
Querieswhetherlinkcreationorderistracked
and/orindexedinagroup.
H5Pset_create_intermediate_group
h5pset_create_inter_group_f
Specifiesinthepropertylistwhethertocre‐
atemissingintermediategroups.
H5Pget_create_intermediate_group
(noFortransubroutine)
Determineswhetherthepropertyissetto
enablecreatingmissingintermediategroups.
H5Pset_char_encoding
h5pset_char_encoding_f
Setsthecharacterencodingusedtoencodea
string.UsetosetASCIIorUTF‐8character
encodingforobjectnames.
H5Pget_char_encoding
h5pget_char_encoding_f
Retrievesthecharacterencodingusedtocre‐
ateastring.
FunctionListing4‐4.Otherexternallinkfunctions
CFunction
FortranSubroutine
Purpose
H5Pset/get_elink_file_cache_size
(noFortransubroutine)
Sets/retrievesthesizeoftheexternallink
openfilecachefromthespecifiedfileaccess
propertylist.
H5Fclear_elink_file_cache
(noFortransubroutine)
Clearstheexternallinkopenfilecachefora
file.
FunctionListing4‐3.Groupcreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Groups
TheHDFGroup 92
2. Performthedesiredoperationsonthegroup.
•Createnewobjectsinthegroup.
•Insertexistingobjectsasgroupmembers.
•Deleteexistingmembers.
•Openandclosememberobjects.
•Accessinformationregardingmemberobjects.
• Iterateacrossgroupmembers.
• Manipulatelinks.
3. Terminateaccesstothegroup(Closethegroup).
4.5.1.CreatingaGroup
Tocreateagroup,useH5Gcreate,specifyingthelocationandthepathofthenewgroup.Thelocationis
theidentifierofthefileorthegroupinafilewithrespecttowhichthenewgroupistobeidentified.The
pathisastringthatprovideswitheranabsolutepathorarelativepathtothenewgroup.Formoreinfor‐
mation,see"HDF5PathNames"onpage 84.Apaththatbeginswithaslash(/)isanabsolutepathindi‐
catingthatitlocatesthenewgroupfromtherootgroupoftheHDF5file.Apaththatbeginswithanyother
characterisarelativepath.Whenthelocationisafile,arelativepathisapathfromthatfile’srootgroup;
whenthelocationisagroup,arelativepathisapathfromthatgroup.
Thesamplecodeintheexamplebelowcreatesthreegroups.ThegroupDataiscreatedintherootdirec‐
tory;twogroupsarethencreatedin/Data,onewithabsolutepath,theotherwitharelativepath.
ThethirdH5Gcreateparameteroptionallyspecifieshowmuchfilespacetoreservetostorethenames
thatwillappearinthisgroup.Ifanon‐positivevalueissupplied,adefaultsizeischosen.
hid_t file;
file = H5Fopen(....);
group = H5Gcreate(file, "/Data", H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
group_new1 = H5Gcreate(file, "/Data/Data_new1", H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
group_new2 = H5Gcreate(group, "Data_new2", H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
CodeExample4‐1.Creatingthreenewgroups
HDF5User’sGuide HDF5Groups
TheHDFGroup 93
4.5.2.OpeningaGroupandAccessinganObjectinthatGroup
Thoughitisnotalwaysnecessary,itisoftenusefultoexplicitlyopenagroupwhenworkingwithobjectsin
thatgroup.Usingthefilecreatedintheexampleabove,theexamplebelowillustratestheuseofaprevi‐
ously‐acquiredfileidentifierandapathrelativetothatfiletoopenthegroupData.
Anyobjectinagroupcanbealsoaccessedbyitsabsoluteorrelativepath.Toopenanobjectusingarela‐
tivepath,anapplicationmustfirstopenthegrouporfileonwhichthatrelativepathisbased.Toopenan
objectusinganabsolutepath,theapplicationcanuseanylocationidentifierinthesamefileasthetarget
object;thefileidentifieriscommonlyused,butobjectidentifierforanyobjectinthatfilewillwork.Both
oftheseapproachesareillustratedintheexamplebelow.
Usingthefilecreatedintheexamplesabove,theexamplebelowprovidessamplecodeillustratingtheuse
ofbothrelativeandabsolutepathstoaccessanHDF5dataobject.Thefirstsequence(twofunctioncalls)
usesapreviously‐acquiredfileidentifiertoopenthegroupData,andthenusesthereturnedgroupiden‐
tifierandarelativepathtoopenthedatasetCData.Thesecondapproach(onefunctioncall)usesthe
samepreviously‐acquiredfileidentifierandanabsolutepathtoopenthesamedataset.
4.5.3.CreatingaDatasetinaSpecificGroup
Anydatasetmustbecreatedinaparticulargroup.Aswithgroups,adatasetmaybecreatedinaparticular
groupbyspecifyingitsabsolutepathorarelativepath.Theexamplebelowillustratesbothapproachesto
creatingadatasetinthegroup/Data.
group = H5Gopen(file, "Data", H5P_DEFAULT);
dataset1 = H5Dopen(group, "CData", H5P_DEFAULT);
dataset2 = H5Dopen(file, "/Data/CData", H5P_DEFAULT);
CodeExample4‐2.Openadatasetwithrelativeandabsolutepaths
dataspace = H5Screate_simple(RANK, dims, NULL);
dataset1 = H5Dcreate(file, "/Data/CData", H5T_NATIVE_INT,
dataspace, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
group = H5Gopen(file, "Data", H5P_DEFAULT);
dataset2 = H5Dcreate(group, "Cdata2", H5T_NATIVE_INT,
dataspace, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
CodeExample4‐3.Createadatasetwithabsoluteandrelativepaths
HDF5User’sGuide HDF5Groups
TheHDFGroup 94
4.5.4.ClosingaGroup
ToensuretheintegrityofHDF5objectsandtoreleasesystemresources,anapplicationshouldalwayscall
theappropriateclosefunctionwhenitisthroughworkingwithanHDF5object.Inthecaseofgroups,
H5GcloseendsaccesstothegroupandreleasesanyresourcestheHDF5Libraryhasmaintainedinsup‐
portofthataccess,includingthegroupidentifier.
Asillustratedintheexamplebelow,allthatisrequiredforanH5Gclosecallisthegroupidentifier
acquiredwhenthegroupwasopened;therearenorelativeversusabsolutepathconsiderations.
Anon‐negativereturnvalueindicatesthatthegroupwassuccessfullyclosedandtheresourcesreleased;a
negativereturnvalueindicatesthattheattempttoclosethegrouporreleaseresourcesfailed.
4.5.5.CreatingLinks
Aspreviouslymentioned,everyobjectiscreatedinaspecificgroup.Oncecreated,anobjectcanbemade
amemberofadditionalgroupsbymeansoflinkscreatedwithoneoftheH5Lcreate_*functions.
Alinkis,ineffect,apathbywhichthetargetobjectcanbeaccessed;itthereforehasanamewhichfunc‐
tionsasasinglepathcomponent.AlinkcanberemovedwithanH5Ldeletecall,effectivelyremovingthe
targetobjectfromthegroupthatcontainedthelink(assuming,ofcourse,thattheremovedlinkwasthe
onlylinktothetargetobjectinthegroup).
HardLinks
Therearetwokindsoflinks,hardlinksandsymboliclinks.Hardlinksarereferencecounted;symbolic
linksarenot.Whenanobjectiscreated,ahardlinkisautomaticallycreated.Anobjectcanbedeleted
fromthefilebyremovingallthehardlinkstoit.
Workingwiththefilefromthepreviousexamples,thecodeintheexamplebelowillustratesthecreation
ofahardlink,namedData_link,intherootgroup,/,tothegroupData.Oncethatlinkiscreated,the
datasetCdatacanbeaccessedviaeitheroftwoabsolutepaths,/Data/Cdataor/Data_Link/Cdata.
herr_t status;
status = H5Gclose(group);
CodeExample4‐4.Closeagroup
HDF5User’sGuide HDF5Groups
TheHDFGroup 95
Theexamplebelowshowsexamplecodetodeletealink,deletingthehardlinkDatafromtherootgroup.
Thegroup/Dataanditsmembersarestillinthefile,buttheycannolongerbeaccessedviaapathusing
thecomponent/Data.
Whenthelasthardlinktoanobjectisdeleted,theobjectisnolongeraccessible.H5Ldeletewillnotpre‐
ventyoufromdeletingthelastlinktoanobject.Toseeifanobjecthasonlyonelink,usethe
H5Oget_infofunction.Ifthevalueoftherc(referencecount)fieldintheisgreaterthan1,thenthelink
canbedeletedwithoutmakingtheobjectinaccessible.
TheexamplebelowshowsH5Oget_infotothegrouporiginallycalledData.
Itispossibletodeletethelasthardlinktoanobjectandnotmaketheobjectinaccessible.Supposeyour
applicationopensadataset,andthendeletesthelasthardlinktothedataset.Whilethedatasetisopen,
status = H5Lcreate_hard(Data_loc_id, "Data", DataLink_loc_id,
"Data_link", H5P_DEFAULT, H5P_DEFAULT)
dataset1 = H5Dopen(file, "/Data_link/CData", H5P_DEFAULT);
dataset2 = H5Dopen(file, "/Data/CData", H5P_DEFAULT);
CodeExample4‐5.Createahardlink
status = H5Ldelete(Data_loc_id, "Data", H5P_DEFAULT);
dataset1 = H5Dopen(file, "/Data_link/CData", H5P_DEFAULT);
/* This call should succeed; all path components
* still exist
*/
dataset2 = H5Dopen(file, "/Data/CData", H5P_DEFAULT);
/* This call will fail; the path component '/Data'
* has been deleted.
*/
CodeExample4‐6.Deletealink
status = H5Oget_info(Data_loc_id, object_info);
CodeExample4‐7.Findingthenumberoflinkstoanobject
HDF5User’sGuide HDF5Groups
TheHDFGroup 96
yourapplicationstillhasaconnectiontothedataset.Ifyourapplicationcreatesahardlinktothedataset
beforeitclosesthedataset,thenthedatasetwillstillbeaccessible.
SymbolicLinks
Symboliclinksareobjectsthatassignanameinagrouptoapath.Notably,thetargetobjectisdetermined
onlywhenthesymboliclinkisaccessed,andmay,infact,notexist.Symboliclinksarenotreference
counted,sotheremaybezero,one,ormoresymboliclinkstoanobject.
Themajortypesofsymboliclinksaresoftlinksandexternallinks.Softlinksaresymboliclinkswithinan
HDF5fileandarecreatedwiththeH5Lcreate_softfunction.Symboliclinkstoobjectslocatedinexter‐
nalfiles,inotherwordsexternallinks,canbecreatedwiththeH5Lcreate_externalfunction.Symbolic
linksareremovedwiththeH5Ldeletefunction.
Theexamplebelowshowsthecreatingtwosoftlinkstothegroup/Data.
Withthesoftlinksdefinedintheexampleabove,thedatasetCDatainthegroup/Datacannowbe
openedwithanyofthenames/Data/CData,/Soft2/CData,or/Soft3/CData.
Inrelease1.8.7,acachewasaddedtoholdthenamesoffilesaccessedviaexternallinks.Thesizeofthis
cachecanbechangedtohelpimproveperformance.Formoreinformation,seetheentryintheHDF5Ref‐
erenceManualfortheH5Pset_elink_file_cache_sizefunctioncall.
NoteRegardingHardLinksandSoftLinks
Notethatanobject’sexistenceinafileisgovernedbythepresenceofatleastonehardlinktothatobject.
Ifthelasthardlinktoanobjectisremoved,theobjectisremovedfromthefileandanyremainingsoftlink
becomesadanglinglink,alinkwhosetargetobjectdoesnotexist.
MovingorRenamingObjects,andaWarning
AnobjectcanberenamedbychangingthenameofalinktoitwithH5Lmove.Thishasthesameeffectas
creatinganewlinkwiththenewnameanddeletingthelinkwiththeoldname.
ExercisecautionintheuseofH5LmoveandH5Ldeleteasthesefunctionseachincludeastepthat
unlinksapointertoanHDF5object.IfthelinkthatisremovedisontheonlypathleadingtoanHDF5
object,thatobjectwillbecomepermanentlyinaccessibleinthefile.
status = H5Lcreate_soft(path_to_target, link_loc_id, "Soft2",
H5P_DEFAULT, H5P_DEFAULT);
status = H5Lcreate_soft(path_to_target, link_loc_id, "Soft3",
H5P_DEFAULT, H5P_DEFAULT);
dataset = H5Dopen(file, "/Soft2/CData", H5P_DEFAULT);
CodeExample4‐8.Createasoftlink
HDF5User’sGuide HDF5Groups
TheHDFGroup 97
Scenario1:RemovingtheLastLink
Toavoidremovingthelastlinktoanobjectorotherwisemakinganobjectinaccessible,usethe
H5Oget_infofunction.Makesurethatthevalueofthereferencecountfield(rc)isgreaterthan1.
Scenario2:MovingaLinkthatIsolatesanObject
Considerthefollowingexample:assumethatthegroupgroup2canonlybeaccessedviathefollowing
path,wheretop_groupisamemberofthefile’srootgroup:
/top_group/group1/group2/
UsingH5Lmove,top_groupisrenamedtobeamemberofgroup2.Atthispoint,sincetop_groupwas
theonlyroutefromtherootgrouptogroup1,thereisnolongerapathbywhichonecanaccessgroup1,
group2,oranymemberdatasets.Andsincetop_groupisnowamemberofgroup2,top_groupitself
andanymemberdatasetshavetherebyalsobecomeinaccessible.
MountingaFile
Anexternallinkisapermanentconnectionbetweentwofiles.Atemporaryconnectioncanbesetupwith
theH5Fmountfunction.Formoreinformation,see"TheHDF5File"onpage 45.Formoreinformation,see
the H5FmountfunctionintheHDF5ReferenceManual.
4.5.6.DiscoveringInformationaboutObjects
Thereisoftenaneedtoretrieveinformationaboutaparticularobject.TheH5Lget_infoand
H5Oget_infofunctionsfillthisnichebyreturningadescriptionoftheobjectorlinkinanH5L_info_t
orH5O_info_tstructure.
4.5.7.DiscoveringObjectsinaGroup
Toexaminealltheobjectsorlinksinagroup,usetheH5LiterateorH5Ovisitfunctionstoexaminethe
objects,andusetheH5Lvisitfunctiontoexaminethelinks.H5Literateisusefulbothwithasingle
groupandinaniterativeprocessthatexaminesanentirefileorsectionofafile(suchasthecontentsofa
grouporthecontentsofallthegroupsthataremembersofthatgroup)andactsonobjectsastheyare
encountered.H5Ovisitrecursivelyvisitsallobjectsaccessiblefromaspecifiedobject.H5Lvisitrecur‐
sivelyvisitsallthelinksstartingfromaspecifiedgroup.
4.5.8.DiscoveringAlloftheObjectsintheFile
ThestructureofanHDF5fileisself‐describing,meaningthatanapplicationcannavigateanHDF5fileto
discoverandunderstandalltheobjectsitcontains.Thisisaniterativeprocesswhereinthestructureistra‐
versedasagraph,startingatonenodeandrecursivelyvisitinglinkednodes.Toexploretheentirefile,the
traversalshouldstartattherootgroup.
HDF5User’sGuide HDF5Groups
TheHDFGroup 98
4.6.ExamplesofFileStructures
ThissectionpresentsseveralsamplesofHDF5filestructures.
Thefigureaboveshowsexamplesofthestructureofafilewiththreegroupsandonedataset.Thefilein
partacontainsthreegroups:therootgroupandtwomembergroups.Inpartb,thedatasetdset1has
beencreatedin/group1.Inpartc,alinknameddset2from/group2tothedatasethasbeenadded.
Notethatthereisonlyonecopyofthedataset;therearetwolinkstoitanditcanbeaccessedeitheras/
group1/dset1oras/group2/dset2.
a)Thefilecontainsthreegroups:theroot
group,/group1,and/group2.
b)Thedatasetdset1(or/group1/dset1)is
createdin/group1.
c)Alinknameddset2tothesamedatasetis
createdin/group2.
d)Thelinkfrom/group1todset1isremoved.
Thedatasetisstillinthefile,butcanbe
accessedonlyas/group2/dset2.
Figure4‐9.Somefilestructures
HDF5User’sGuide HDF5Groups
TheHDFGroup 99
Thefigureinpartdaboveillustratesthatoneofthetwolinkstothedatasetcanbedeleted.Inthiscase,
thelinkfrom/group1hasbeenremoved.Thedatasetitselfhasnotbeendeleted;itisstillinthefilebut
canonlybeaccessedas/group1/dset2.
ThefigureaboveillustratesloopsinanHDF5filestructure.Thefileinpartacontainsthreegroupsanda
dataset;group2isamemberoftherootgroupandoftherootgroup’sothermembergroup,group1.
group2thuscanbeaccessedbyeitheroftwopaths:/group2or/group1/GXX.Similarly,thedataset
canbeaccessedeitheras/group2/dset1oras/group1/GXX/dset1.
Partbillustratesadifferentcase:thedatasetisamemberofasinglegroupbutwithtwolinks,ornames,
inthatgroup.Inthiscase,thedatasetagainhastwonames,/group1/dset1and/group1/dset2.
a)dset1hastwonames:/group2/dset1and
/group1/GXX/dset1.
b)dset1againhastwonames:/group1/
dset1and/group1/dset2.
c)dset1hasthreenames:/group1/dset1,/
group2/dset2,and/group1/GXX/dset2.
d)dset1hasaninfinitenumberofavailable
pathnames.
Figure4‐10.Moresamplefilestructures
HDF5User’sGuide HDF5Groups
TheHDFGroup 100
Inpartc,thedatasetdset1isamemberoftwogroups,oneofwhichcanbeaccessedbyeitheroftwo
names.Thedatasetthushasthreepathnames:/group1/dset1,/group2/dset2,and/group1/GXX/
dset2.
Andinpartd,twoofthegroupsaremembersofeachotherandthedatasetisamemberofbothgroups.
Inthiscase,thereareaninfinitenumberofpathstothedatasetbecauseGXXandGYYcanbetraversed
anynumberoftimesonthewayfromtherootgroup,/,tothedataset.Thiscanyieldapathnamesuchas
/group1/GXX/GYY/GXX/GYY/GXX/dset2.
Thefigureabovetakesusintotherealmofsoftlinks.Theoriginalfile,inparta,containsonlythreehard
links.Inpartb,asoftlinknameddset2fromgroup2to/group1/dset1hasbeencreated,makingthis
datasetaccessibleas/group2/dset2.
a)Thefilecontainsonlyhardlinks. b)Asoftlinkisaddedfromgroup2to
/group1/dset1.
c)Asoftlinknameddset3isaddedwithatar‐
getthatdoesnotyetexist.
d)Thetargetofthesoftlinkiscreatedorlinked.
Figure4‐11.Hardandsoftlinks
HDF5User’sGuide HDF5Groups
TheHDFGroup 101
Inpartc,anothersoftlinkhasbeencreatedingroup2.Butthistimethesoftlink,dset3,pointstoatar‐
getobjectthatdoesnotyetexist.Thattargetobject,dset,hasbeenaddedinpartdandisnowaccessible
aseither/group2/dsetor/group2/dset3.
HDF5User’sGuide HDF5Groups
TheHDFGroup 102
HDF5User’sGuide HDF5Datasets
TheHDFGroup 103
5.HDF5Datasets
5.1.Introduction
AnHDF5datasetisanobjectcomposedofacollectionofdataelements,orrawdata,andmetadatathat
storesadescriptionofthedataelements,datalayout,andallotherinformationnecessarytowrite,read,
andinterpretthestoreddata.Fromtheviewpointoftheapplicationtherawdataisstoredasaone‐
dimensionalormulti‐dimensionalarrayofelements(therawdata),thoseelementscanbeanyofseveral
numericalorcharactertypes,smallarrays,orevencompoundtypessimilartoCstructs.Thedataset
objectmayhaveattributeobjects.Seethefigurebelow.
Adatasetobjectisstoredinafileintwoparts:aheaderandadataarray.Theheadercontainsinformation
thatisneededtointerpretthearrayportionofthedataset,aswellasmetadata(orpointerstometadata)
thatdescribesorannotatesthedataset.Headerinformationincludesthenameoftheobject,itsdimen‐
sionality,itsnumber‐type,informationabouthowthedataitselfisstoredondisk(thestoragelayout),and
otherinformationusedbythelibrarytospeedupaccesstothedatasetormaintainthefile’sintegrity.
TheHDF5datasetinterface,comprisingtheH5Dfunctions,providesamechanismformanagingHDF5
datasetsincludingthetransferofdatabetweenmemoryanddiskandthedescriptionofdatasetproper‐
ties.
Figure5‐1.Applicationviewofadataset
HDF5User’sGuide HDF5Datasets
TheHDFGroup 104
AdatasetisusedbyotherHDF5APIs,eitherbynameorbyanidentifier.Formoreinformation,see“Using
Identifiers.”
Link/Unlink
AdatasetcanbeaddedtoagroupwithoneoftheH5Lcreatecalls,anddeletedfromagroupwith
H5Ldelete.Thelinkandunlinkoperationsusethenameofanobject,whichmaybeadataset.Thedata‐
setdoesnothavetoopentobelinkedorunlinked.
ObjectReference
Adatasetmaybethetargetofanobjectreference.TheobjectreferenceiscreatedbyH5Rcreatewith
thenameofanobjectwhichmaybeadatasetandthereferencetypeH5R_OBJECT.Thedatasetdoesnot
havetobeopentocreateareferencetoit.
Anobjectreferencemayalsorefertoaregion(selection)ofadataset.Thereferenceiscreatedwith
H5RcreateandareferencetypeofH5R_DATASET_REGION.
AnobjectreferencecanbeaccessedbyacalltoH5Rdereference.Whenthereferenceistoadatasetor
datasetregion,theH5RdeferencecallreturnsanidentifiertothedatasetjustasifH5Dopenhasbeen
called.
AddingAttributes
Adatasetmayhaveuser‐definedattributeswhicharecreatedwithH5Acreateandaccessedthroughthe
H5AAPI.Tocreateanattributeforadataset,thedatasetmustbeopen,andtheidentifierispassedto
H5Acreate.TheattributesofadatasetarediscoveredandopenedusingH5Aopen_name,H5Aop-
en_idx,orH5Aiterate;thesefunctionsusetheidentifierofthedataset.Anattributecanbedeleted
withH5Adeletewhichalsousestheidentifierofthedataset.
5.2.DatasetFunctionSummaries
Functionsthatcanbeusedwithdatasets(H5Dfunctions)andpropertylistfunctionsthatcanusedwith
datasets(H5Pfunctions)arelistedbelow.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 105
FunctionListing5‐1.Datasetfunctions(H5D)
CFunction
FortranSubroutine
Purpose
H5Dcreate
h5dcreate_f
Createsadatasetatthespecifiedlocation.
TheCfunctionisamacro:see“APICompati‐
bilityMacrosinHDF5.”
H5Dcreate_anon
h5dcreate_anon_f
Createsadatasetinafilewithoutlinkingit
intothefilestructure.
H5Dopen
h5dopen_f
Opensanexistingdataset.TheCfunctionisa
macro:see“APICompatibilityMacrosin
HDF5.”
H5Dclose
h5dclose_f
Closesthespecifieddataset.
H5Dget_space
h5dget_space_f
Returnsanidentifierforacopyofthe
dataspaceforadataset.
H5Dget_space_status
h5dget_space_status_f
Determineswhetherspacehasbeenallo‐
catedforadataset.
H5Dget_type
h5dget_type_f
Returnsanidentifierforacopyofthedata‐
typeforadataset.
H5Dget_create_plist
h5dget_create_plist_f
Returnsanidentifierforacopyofthedataset
creationpropertylistforadataset.
H5Dget_access_plist
(noFortransubroutine)
Returnsthedatasetaccesspropertylistasso‐
ciatedwithadataset.
H5Dget_offset
h5dget_offset_f
Returnsthedatasetaddressinafile.
H5Dget_storage_size
h5dget_storage_size_f
Returnstheamountofstoragerequiredfora
dataset.
H5Dvlen_get_buf_size
h5dvlen_get_max_len_f
Determinesthenumberofbytesrequiredto
storevariable‐length(VL)data.
H5Dvlen_reclaim
h5dvlen_reclaim_f
ReclaimsVLdatatypememorybuffers.
H5Dread
h5dread_f
Readsrawdatafromadatasetintoabuffer.
H5Dwrite
h5dwrite_f
Writesrawdatafromabuffertoadataset.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 106
H5Diterate
(noFortransubroutine)
Iteratesoverallselectedelementsina
dataspace.
H5Dgather
(noFortransubroutine)
Gathersdatafromaselectionwithinamem‐
orybuffer.
H5Dscatter
(noFortransubroutine)
Scattersdataintoaselectionwithinamem‐
orybuffer.
H5Dfill
h5dfill_f
Fillsdataspaceelementswithafillvalueina
memorybuffer.
H5Dset_extent
h5dset_extent_f
Changesthesizesofadataset’sdimensions.
FunctionListing5‐2.Datasetcreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pset_layout
h5pset_layout_f
Setsthetypeofstorageusedtostoretheraw
dataforadataset.
H5Pget_layout
h5pget_layout_f
Returnsthelayoutoftherawdataforadata‐
set.
H5Pset_chunk
h5pset_chunk_f
Setsthesizeofthechunksusedtostorea
chunkedlayoutdataset.
H5Pget_chunk
h5pget_chunk_f
Retrievesthesizeofchunksfortherawdata
ofachunkedlayoutdataset.
H5Pset_deflate
h5pset_deflate_f
Setscompressionmethodandcompression
level.
H5Pset_fill_value
h5pset_fill_value_f
Setsthefillvalueforadataset.
H5Pget_fill_value
h5pget_fill_value_f
Retrievesadatasetfillvalue.
H5Pfill_value_defined
(noFortransubroutine)
Determineswhetherthefillvalueisdefined.
H5Pset_fill_time
h5pset_fill_time_f
Setsthetimewhenfillvaluesarewrittentoa
dataset.
FunctionListing5‐1.Datasetfunctions(H5D)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datasets
TheHDFGroup 107
H5Pget_fill_time
h5pget_fill_time_f
Retrievesthetimewhenfillvaluearewritten
toadataset.
H5Pset_alloc_time
h5pset_alloc_time_f
Setsthetimingforstoragespaceallocation.
H5Pget_alloc_time
h5pget_alloc_time_f
Retrievesthetimingforstoragespacealloca‐
tion.
H5Pset_filter
h5pset_filter_f
Addsafiltertothefilterpipeline.
H5Pall_filters_avail
(noFortransubroutine)
Verifiesthatallrequiredfiltersareavailable.
H5Pget_nfilters
h5pget_nfilters_f
Returnsthenumberoffiltersinthepipeline.
H5Pget_filter
h5pget_filter_f
Returnsinformationaboutafilterinapipe‐
line.TheCfunctionisamacro:see“APICom‐
patibilityMacrosinHDF5.”
H5Pget_filter_by_id
h5pget_filter_by_id_f
Returnsinformationaboutthespecifiedfilter.
TheCfunctionisamacro:see“APICompati‐
bilityMacrosinHDF5.”
H5Pmodify_filter
h5pmodify_filter_f
Modifiesafilterinthefilterpipeline.
H5Premove_filter
h5premove_filter_f
Deletesoneormorefiltersinthefilterpipe‐
line.
H5Pset_fletcher32
h5pset_fletcher32_f
SetsupuseoftheFletcher32checksumfilter.
H5Pset_nbit
h5pset_nbit_f
Setsupuseofthen‐bitfilter.
H5Pset_scaleoffset
h5pset_scaleoffset_f
Setsupuseofthescale‐offsetfilter.
H5Pset_shuffle
h5pset_shuffle_f
Setsupuseoftheshufflefilter.
H5Pset_szip
h5pset_szip_f
SetsupuseoftheSzipcompressionfilter.
H5Pset_external
h5pset_external_f
Addsanexternalfiletothelistofexternal
files.
FunctionListing5‐2.Datasetcreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datasets
TheHDFGroup 108
H5Pget_external_count
h5pget_external_count_f
Returnsthenumberofexternalfilesfora
dataset.
H5Pget_external
h5pget_external_f
Returnsinformationaboutanexternalfile.
H5Pset_char_encoding
h5pset_char_encoding_f
Setsthecharacterencodingusedtoencodea
string.UsetosetASCIIorUTF‐8character
encodingforobjectnames.
H5Pget_char_encoding
h5pget_char_encoding_f
Retrievesthecharacterencodingusedtocre‐
ateastring.
FunctionListing5‐3.Datasetaccesspropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pset_buffer
h5pset_buffer_f
Setstypeconversionandbackgroundbuffers.
H5Pget_buffer
h5pget_buffer_f
Readsbuffersettings.
H5Pset_chunk_cache
h5pset_chunk_cache_f
Setstherawdatachunkcacheparameters.
H5Pget_chunk_cache
h5pget_chunk_cache_f
Retrievestherawdatachunkcacheparame‐
ters.
H5Pset_edc_check
h5pset_edc_check_f
Setswhethertoenableerror‐detectionwhen
readingadataset.
H5Pget_edc_check
h5pget_edc_check_f
Determineswhethererror‐detectionis
enabledfordatasetreads.
H5Pset_filter_callback
(noFortransubroutine)
Setsuser‐definedfiltercallbackfunction.
H5Pset_data_transform
h5pset_data_transform_f
Setsadatatransformexpression.
H5Pget_data_transform
h5pget_data_transform_f
Retrievesadatatransformexpression.
FunctionListing5‐2.Datasetcreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datasets
TheHDFGroup 109
H5Pset_type_conv_cb
(noFortransubroutine)
Setsuser‐defineddatatypeconversioncall‐
backfunction.
H5Pget_type_conv_cb
(noFortransubroutine)
Getsuser‐defineddatatypeconversioncall‐
backfunction.
H5Pset_hyper_vector_size
h5pset_hyper_vector_size_f
SetsnumberofI/Ovectorstoberead/written
inhyperslabI/O.
H5Pget_hyper_vector_size
h5pget_hyper_vector_size_f
RetrievesnumberofI/Ovectorstoberead/
writteninhyperslabI/O.
H5Pset_btree_ratios
h5pset_btree_ratios_f
SetsB‐treesplitratiosforadatasettransfer
propertylist.
H5Pget_btree_ratios
h5pget_btree_ratios_f
GetsB‐treesplitratiosforadatasettransfer
propertylist.
H5Pset_vlen_mem_manager
(noFortransubroutine)
Setsthememorymanagerforvariable‐length
datatypeallocationinH5DreadandH5Dv-
len_reclaim.
H5Pget_vlen_mem_manager
(noFortransubroutine)
Getsthememorymanagerforvariable‐length
datatypeallocationinH5DreadandH5Dv-
len_reclaim.
H5Pset_dxpl_mpio
h5pset_dxpl_mpio_f
Setsdatatransfermode.
H5Pget_dxpl_mpio
h5pget_dxpl_mpio_f
Returnsthedatatransfermode.
H5Pset_dxpl_mpio_chunk_opt
(noFortransubroutine)
Setsaflagspecifyinglinked‐chunkI/Oor
multi‐chunkI/O.
H5Pset_dxpl_mpio_chunk_opt_num
(noFortransubroutine)
Setsanumericthresholdforlinked‐chunkI/O.
H5Pset_dxpl_mpio_chunk_opt_ratio
(noFortransubroutine)
SetsaratiothresholdforcollectiveI/O.
H5Pset_dxpl_mpio_collective_opt
(noFortransubroutine)
Setsaflaggoverningtheuseofindependent
versuscollectiveI/O.
H5Pset_multi_type
(noFortransubroutine)
SetsthetypeofdatapropertyfortheMULTI
driver.
FunctionListing5‐3.Datasetaccesspropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datasets
TheHDFGroup 110
5.3.ProgrammingModelforDatasets
Thissectionexplainstheprogrammingmodelfordatasets.
5.3.1.GeneralModel
Theprogrammingmodelforusingadatasethasthreemainphases:
•Obtainaccesstothedataset
•Operateonthedatasetusingthedatasetidentifierreturnedataccess
•Releasethedataset
Thesethreephasesorstepsaredescribedinmoredetailbelowthefigure.
Adatasetmaybeopenedseveraltimesandoperationsperformedwithseveraldifferentidentifierstothe
samedataset.Alltheoperationsaffectthedatasetalthoughthecallingprogrammustsynchronizeifnec‐
essarytoserializeaccesses.
Notethatthedatasetremainsopenuntileveryidentifierisclosed.Thefigurebelowshowsthebasic
sequenceofoperations.
H5Pget_multi_type
(noFortransubroutine)
Retrievesthetypeofdatapropertyforthe
MULTIdriver.
H5Pset_small_data_block_size
h5pset_small_data_block_size_f
Setsthesizeofacontiguousblockreserved
forsmalldata.
H5Pget_small_data_block_size
h5pget_small_data_block_size_f
Retrievesthecurrentsmalldatablocksize
setting.
FunctionListing5‐3.Datasetaccesspropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datasets
TheHDFGroup 111
Creationanddataaccessoperationsmayhaveoptionalparameterswhicharesetwithpropertylists.The
generalprogrammingmodelis:
•Createpropertylistofappropriateclass(datasetcreate,datasettransfer)
•Setpropertiesasneeded;eachtypeofpropertyhasitsownformatanddatatype
•PassthepropertylistasaparameteroftheAPIcall
Thestepsbelowdescribetheprogrammingphasesorstepsforusingadataset.
Step1.ObtainAccess
AnewdatasetiscreatedbyacalltoH5Dcreate.Ifsuccessful,thecallreturnsanidentifierforthenewly
createddataset.
AccesstoanexistingdatasetisobtainedbyacalltoH5Dopen.Thiscallreturnsanidentifierfortheexisting
dataset.
Anobjectreferencemaybedereferencedtoobtainanidentifiertothedatasetitpointsto.
Ineachofthesecases,thesuccessfulcallreturnsanidentifiertothedataset.Theidentifierisusedinsub‐
sequentoperationsuntilthedatasetisclosed.
Step2.OperateontheDataset
Thedatasetidentifiercanbeusedtowriteandreaddatatothedataset,toqueryandsetproperties,and
toperformotheroperationssuchasaddingattributes,linkingingroups,andcreatingreferences.
Figure5‐2.Datasetprogrammingsequence
HDF5User’sGuide HDF5Datasets
TheHDFGroup 112
Thedatasetidentifiercanbeusedforanynumberofoperationsuntilthedatasetisclosed.
Step3.ClosetheDataset
Whenalloperationsarecompleted,thedatasetidentifiershouldbeclosed.Thisreleasesthedataset.
Aftertheidentifierisclosed,itcannotbeusedforfurtheroperations.
5.3.2.CreateDataset
AdatasetiscreatedandinitializedwithacalltoH5Dcreate.Thedatasetcreateoperationsetspermanent
propertiesofthedataset:
•Name
•Dataspace
•Datatype
•Storageproperties
Thesepropertiescannotbechangedforthelifeofthedataset,althoughthedataspacemaybeexpanded
uptoitsmaximumdimensions.
Name
AdatasetnameisasequenceofalphanumericASCIIcharacters.Thefullnamewouldincludeatracingof
thegrouphierarchyfromtherootgroupofthefile.Anexampleis/rootGroup/groupA/subgroup23/
dataset1.Thelocalnameorrelativenamewithinthelowest‐levelgroupcontainingthedatasetwould
includenoneofthegrouphierarchy.AnexampleisDataset1.
Dataspace
Thedataspaceofadatasetdefinesthenumberofdimensionsandthesizeofeachdimension.The
dataspacedefinesthenumberofdimensions,andthemaximumdimensionsizesandcurrentsizeofeach
dimension.ThemaximumdimensionsizecanbeafixedvalueortheconstantH5D_UNLIMITED,inwhich
casetheactualdimensionsizecanbechangedwithcallstoH5Dset_extent,uptothemaximumsetwith
themaxdimsparameterintheH5Screate_simplecallthatestablishedthedataset’soriginaldimen‐
sions.Themaximumdimensionsizeissetwhenthedatasetiscreatedandcannotbechanged.
Datatype
Rawdatahasadatatypewhichdescribesthelayoutoftherawdatastoredinthefile.Thedatatypeisset
whenthedatasetiscreatedandcanneverbechanged.Whendataistransferredtoandfromthedataset,
theHDF5Librarywillassurethatthedataistransformedtoandfromthestoredformat.
StorageProperties
Storagepropertiesofthedatasetaresetwhenitiscreated.Therequiredinputstablebelowshowsthe
categoriesofstorageproperties.Thestoragepropertiescannotbechangedafterthedatasetiscreated.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 113
Filters
Whenadatasetiscreated,optionalfiltersarespecified.Thefiltersareaddedtothedatatransferpipeline
whendataisreadorwritten.Thestandardlibraryincludesfilterstoimplementcompression,datashuf‐
fling,anderrordetectioncode.Additionaluser‐definedfiltersmayalsobeused.
Therequiredfiltersarestoredaspartofthedataset,andthelistmaynotbechangedafterthedatasetis
created.TheHDF5Libraryautomaticallyappliesthefilterswheneverdataistransferred.
Summary
Anewlycreateddatasethasnoattributesandnodatavalues.Thedimensions,datatype,storageproper‐
ties,andselectedfiltersareset.Thetablebelowliststherequiredinputs,andthesecondtablebelowlists
theoptionalinputs.
Example
Tocreateanewdataset,gothroughthefollowinggeneralsteps:
•Setdatasetcharacteristics(optionalwheredefaultsettingsareacceptable)
•Datatype
•Dataspace
•Datasetcreationpropertylist
•Createthedataset
Table5‐1.Requiredinputs
RequiredInputs Description
Dataspace Theshapeofthearray.
Datatype Thelayoutofthestoredelements.
Name Thenameofthedatasetinthegroup.
Table5‐2.Optionalinputs
OptionalInputs Description
StorageLayout Howthedataisorganizedinthefileincludingchunking.
FillValue Thebehaviorandvalueforuninitializeddata.
ExternalStorage Optiontostoretherawdatainanexternalfile.
Filters Selectoptionalfilterstobeapplied.Oneofthefiltersthatmightbe
appliediscompression.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 114
•Closethedatatype,dataspace,andpropertylist(asnecessary)
•Closethedataset
Example1belowshowsexamplecodetocreateanemptydataset.Thedataspaceis7x8,andthedata‐
typeisabig‐endianinteger.Thedatasetiscreatedwiththename“dset1”andisamemberoftheroot
group,“/”.
Example2belowshowsexamplecodetocreateasimilardatasetwithafillvalueof‘‐1’.Thiscodehasthe
samestepsasintheexampleabove,butusesanon‐defaultpropertylist.Afilecreationpropertylistiscre‐
ated,andthenthefillvalueissettothedesiredvalue.ThenthepropertylistispassedtotheH5Dcreate
call.
hid_t dataset, datatype, dataspace;
/*
* Create dataspace: Describe the size of the array and
* create the dataspace for fixed-size dataset.
*/
dimsf[0] = 7;
dimsf[1] = 8;
dataspace = H5Screate_simple(2, dimsf, NULL);
/*
* Define datatype for the data in the file.
* For this example, store little-endian integer numbers.
*/
datatype = H5Tcopy(H5T_NATIVE_INT);
status = H5Tset_order(datatype, H5T_ORDER_LE);
/*
* Create a new dataset within the file using defined
* dataspace and datatype. No properties are set.
*/
dataset = H5Dcreate(file, "/dset", datatype, dataspace,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
H5Dclose(dataset);
H5Sclose(dataspace);
H5Tclose(datatype);
CodeExample5‐1.Createanemptydataset
HDF5User’sGuide HDF5Datasets
TheHDFGroup 115
Afterthiscodeisexecuted,thedatasethasbeencreatedandwrittentothefile.Thedataarrayisuninitial‐
ized.Dependingonthestoragestrategyandfillvalueoptionsthathavebeenselected,someorallofthe
spacemaybeallocatedinthefile,andfillvaluesmaybewritteninthefile.
5.3.3.DataTransferOperationsonaDataset
DataistransferredbetweenmemoryandtherawdataarrayofthedatasetthroughH5Dwriteand
H5Dreadoperations.Adatatransferhasthefollowingbasicsteps:
1. Allocateandinitializememoryspaceasneeded
2. Definethedatatypeofthememoryelements
3. Definetheelementstobetransferred(aselection,oralltheelements)
4. Setdatatransferproperties(includingparametersforfiltersorfiledrivers)asneeded
hid_t dataset, datatype, dataspace;
hid_t plist; /* property list */
int fillval = -1;
dimsf[0] = 7;
dimsf[1] = 8;
dataspace = H5Screate_simple(2, dimsf, NULL);
datatype = H5Tcopy(H5T_NATIVE_INT);
status = H5Tset_order(datatype, H5T_ORDER_LE);
/*
* Example of Dataset Creation property list: set fill value
* to '-1'
*/
plist = H5Pcreate(H5P_DATASET_CREATE);
status = H5Pset_fill_value(plist, datatype, &fillval);
/* Same as above, but use the property list */
dataset = H5Dcreate(file, "/dset", datatype, dataspace,
H5P_DEFAULT, plist, H5P_DEFAULT);
H5Dclose(dataset);
H5Sclose(dataspace);
H5Tclose(datatype);
H5Pclose(plist);
CodeExample5‐2.Createadatasetwithfillvalueset
HDF5User’sGuide HDF5Datasets
TheHDFGroup 116
5. CalltheH5DAPI
Notethatthelocationofthedatainthefile,thedatatypeofthedatainthefile,thestorageproperties,
andthefiltersdonotneedtobespecifiedbecausethesearestoredasapermanentpartofthedataset.A
selectionofelementsfromthedataspaceisspecified;theselectedelementsmaybethewholedataspace.
Thefigurebelowshowsadiagramofawriteoperationwhichtransfersadataarrayfrommemorytoa
datasetinthefile(usuallyondisk).Areadoperationhassimilarparameterswiththedataflowingthe
otherdirection.
MemorySpace
Thecallingprogrammustallocatesufficientmemorytostorethedataelementstobetransferred.Fora
write(frommemorytothefile),thememorymustbeinitializedwiththedatatobewrittentothefile.For
aread,thememorymustbelargeenoughtostoretheelementsthatwillberead.Theamountofstorage
neededcanbecomputedfromthememorydatatype(whichdefinesthesizeofeachdataelement)and
thenumberofelementsintheselection.
MemoryDatatype
Thememorylayoutofasingledataelementisspecifiedbythememorydatatype.Thisspecifiesthesize,
alignment,andbyteorderoftheelementaswellasthedatatypeclass.Notethatthememorydatatype
Figure5‐3.Awriteoperation
HDF5User’sGuide HDF5Datasets
TheHDFGroup 117
mustbethesamedatatypeclassasthefile,butmayhavedifferentbyteorderandotherproperties.The
HDF5Libraryautomaticallytransformsdataelementsbetweenthesourceanddestinationlayouts.For
moreinformation,see"HDF5Datatypes"onpage 173.
Forawrite,thememorydatatypedefinesthelayoutofthedatatobewritten;anexampleisIEEEfloating‐
pointnumbersinnativebyteorder.Ifthefiledatatype(definedwhenthedatasetiscreated)isdifferent
butcompatible,theHDF5Librarywilltransformeachdataelementwhenitiswritten.Forexample,ifthe
filebyteorderisdifferentthanthenativebyteorder,theHDF5Librarywillswapthebytes.
Foraread,thememorydatatypedefinesthedesiredlayoutofthedatatoberead.Thismustbecompati‐
blewiththefiledatatype,butshouldgenerallyusenativeformatssuchasbyteorders.TheHDF5Library
willtransformeachdataelementasitisread.
Selection
Thedatatransferwilltransfersomeoralloftheelementsofthedatasetdependingonthedataspace
selection.Theselectionhastwodataspaceobjects:oneforthesource,andoneforthedestination.These
objectsdescribewhichelementsofthedataspacetobetransferred.Some(partialI/O)orallofthedata
maybetransferred.PartialI/Oisdefinedbydefininghyperslabsorlistsofelementsinadataspaceobject.
Thedataspaceselectionforthesourcedefinestheindicesoftheelementstobereadorwritten.Thetwo
selectionsmustdefinethesamenumberofpoints,buttheorderandlayoutmaybedifferent.TheHDF5
Libraryautomaticallyselectsanddistributestheelementsaccordingtotheselections.Itmight,forexam‐
ple,performascatter‐gatherorsub‐setofthedata.
DataTransferProperties
Forsomedatatransfers,additionalparametersshouldbesetusingthetransferpropertylist.Thetable
belowliststhecategoriesoftransferproperties.ThesepropertiessetparametersfortheHDF5Libraryand
maybeusedtopassparametersforoptionalfiltersandfiledrivers.Forexample,transferpropertiesare
usedtoselectindependentorcollectiveoperationwhenusingMPI‐I/O.
Table5‐3.Categoriesoftransferproperties
Properties Description
Libraryparameters Internalcaches,buffers,B‐Trees,etc.
Memorymanagement Variable‐lengthmemorymanagement,dataoverwrite
Filedrivermanagement Parametersforfiledrivers
Filtermanagement Parametersforfilters
HDF5User’sGuide HDF5Datasets
TheHDFGroup 118
DataTransferOperation(ReadorWrite)
ThedatatransferisdonebycallingH5DreadorH5Dwritewiththeparametersdescribedabove.The
HDF5Libraryconstructstherequiredpipeline,whichwillscatter‐gather,transformdatatypes,applythe
requestedfilters,andusethecorrectfiledriver.
Duringthedatatransfer,thetransformationsandfiltersareappliedtoeachelementofthedatainthe
requiredorderuntilallthedataistransferred.
Summary
Toperformadatatransfer,itisnecessarytoallocateandinitializememory,describethesourceanddesti‐
nation,setrequiredandoptionaltransferproperties,andcalltheH5DAPI.
Examples
Thebasicproceduretowritetoadatasetisthefollowing:
•Openthedataset.
•Setthedatasetdataspaceforthewrite(optionalifdataspaceisH5S_SELECT_ALL).
•Writedata.
•Closethedatatype,dataspace,andpropertylist(asnecessary).
•Closethedataset.
Example3belowshowsexamplecodetowritea4x6arrayofintegers.Intheexample,thedataisinitial‐
izedinthememoryarraydset_data.Thedatasethasalreadybeencreatedinthefile,soitisopenedwith
H5Dopen.
ThedataiswrittenwithH5Dwrite.Theargumentsarethedatasetidentifier,thememorydatatype
(H5T_NATIVE_INT),thememoryandfileselections(H5S_ALLinthiscase:thewholearray),andthe
default(empty)propertylist.Thelastargumentisthedatatobetransferred.
hid_t file_id, dataset_id; /* identifiers */
herr_t status;
int i, j, dset_data[4][6];
/* Initialize the dataset. */
for (i = 0; i < 4; i++)
for (j = 0; j < 6; j++)
dset_data[i][j] = i * 6 + j + 1;
CodeExample5‐3.Writeanarrayofintegers
HDF5User’sGuide HDF5Datasets
TheHDFGroup 119
Example4belowshowsasimilarwriteexceptforsettinganon‐defaultvalueforthetransferbuffer.The
codeisthesameasExample3,butatransferpropertylistiscreated,andthedesiredbuffersizeisset.The
H5Dwritefunctionhasthesamearguments,butusesthepropertylisttosetthebuffer.
/* Open an existing file. */
file_id = H5Fopen("dset.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Open an existing dataset. */
dataset_id = H5Dopen(file_id, "/dset", H5P_DEFAULT);
/* Write the entire dataset, using 'dset_data':
memory type is 'native int'
write the entire dataspace to the entire dataspace,
no transfer properties,
*/
status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL,
H5S_ALL, H5P_DEFAULT, dset_data);
status = H5Dclose(dataset_id);
hid_t file_id, dataset_id;
hid_t xferplist;
herr_t status;
int i, j, dset_data[4][6];
file_id = H5Fopen("dset.h5", H5F_ACC_RDWR, H5P_DEFAULT);
dataset_id = H5Dopen(file_id, "/dset", H5P_DEFAULT);
/*
* Example: set type conversion buffer to 64MB
*/
CodeExample5‐4.Writeanarrayusingapropertylist
CodeExample5‐3.Writeanarrayofintegers
HDF5User’sGuide HDF5Datasets
TheHDFGroup 120
Thebasicproceduretoreadfromadatasetisthefollowing:
• Definethememorydataspaceoftheread(optionalifdataspaceisH5S_SELECT_ALL).
•Openthedataset.
•Getthedatasetdataspace(ifusingH5S_SELECT_ALLabove).
Elsedefinedatasetdataspaceofread.
• Definethememorydatatype(optional).
• Definethememorybuffer.
•Openthedataset.
•Readdata.
•Closethedatatype,dataspace,andpropertylist(asnecessary).
•Closethedataset.
Theexamplebelowshowscodethatreadsa4x6arrayofintegersfromadatasetcalled“dset1”.First,the
datasetisopened.TheH5Dreadcallhasparameters:
•Thedatasetidentifier(fromH5Dopen)
•Thememorydatatype(H5T_NATVE_INT)
•Thememoryandfiledataspace(H5S_ALL,thewholearray)
•Adefault(empty)propertylist
•Thememorytobefilled
xferplist = H5Pcreate(H5P_DATASET_XFER);
status = H5Pset_buffer( xferplist, 64 * 1024 *1024, NULL, NULL);
/* Write the entire dataset, using 'dset_data':
memory type is 'native int'
write the entire dataspace to the entire dataspace,
set the buffer size with the property list,
*/
status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL,
H5S_ALL, xferplist, dset_data);
status = H5Dclose(dataset_id);
CodeExample5‐4.Writeanarrayusingapropertylist
HDF5User’sGuide HDF5Datasets
TheHDFGroup 121
5.3.4.RetrievethePropertiesofaDataset
Thefunctionslistedbelowallowtheusertoretrieveinformationregardingadatasetincludingthedata‐
type,thedataspace,thedatasetcreationpropertylist,andthetotalstoredsizeofthedata.
hid_t file_id, dataset_id;
herr_t status;
int i, j, dset_data[4][6];
/* Open an existing file. */
file_id = H5Fopen("dset.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Open an existing dataset. */
dataset_id = H5Dopen(file_id, "/dset", H5P_DEFAULT);
/* read the entire dataset, into 'dset_data':
memory type is 'native int'
read the entire dataspace to the entire dataspace,
no transfer properties,
*/
status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL,
H5S_ALL, H5P_DEFAULT, dset_data);
status = H5Dclose(dataset_id);
CodeExample5‐5.Readanarrayfromadataset
FunctionListing5‐4.Retrievedatasetinformation
QueryFunction Description
H5Dget_space Retrievethedataspaceofthedatasetasstoredin
thefile.
H5Dget_type Retrievethedatatypeofthedatasetasstoredin
thefile.
H5Dget_create_plist Retrievethedatasetcreationproperties.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 122
Theexamplebelowillustrateshowtoretrievedatasetinformation.
5.4.DataTransfer
TheHDF5Libraryimplementsdatatransfersthroughapipelinewhichimplementsdatatransformations
(accordingtothedatatypeandselections),chunking(asrequested),andI/Ooperationsusingdifferent
mechanisms(filedrivers).ThepipelineisautomaticallyconfiguredbytheHDF5Library.Metadataisstored
inthefilesothatthecorrectpipelinecanbeconstructedtoretrievethedata.Inaddition,optionalfilters
suchascompressionmaybeaddedtothestandardpipeline.
H5Dget_storage_size Retrievethetotalbytesforallthedataofthe
dataset.
H5Dvlen_get_buf_size Retrievethetotalbytesforallthevariable‐length
dataofthedataset.
hid_t file_id, dataset_id;
hid_t dspace_id, dtype_id, plist_id;
herr_t status;
/* Open an existing file. */
file_id = H5Fopen("dset.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Open an existing dataset. */
dataset_id = H5Dopen(file_id, "/dset", H5P_DEFAULT);
dspace_id = H5Dget_space(dataset_id);
dtype_id = H5Dget_type(dataset_id);
plist_id = H5Dget_create_plist(dataset_id);
/* use the objects to discover the properties of the dataset */
status = H5Dclose(dataset_id);
CodeExample5‐6.Retrievedataset
FunctionListing5‐4.Retrievedatasetinformation
QueryFunction Description
HDF5User’sGuide HDF5Datasets
TheHDFGroup 123
ThefigurebelowillustratesdatalayoutsfordifferentlayersofanapplicationusingHDF5.Theapplication
dataisorganizedasamultidimensionalarrayofelements.TheHDF5formatspecificationdefinesthe
storedlayoutofthedataandmetadata.Thestoragelayoutpropertiesdefinetheorganizationofthe
abstractdata.Thisdataiswrittenandreadtoandfromsomestoragemedium.
Thelaststageofawrite(andfirststageofaread)ismanagedbyanHDF5filedrivermodule.Thevirtual
filelayeroftheHDF5LibraryimplementsastandardinterfacetoalternativeI/Omethods,includingmem‐
ory(AKA“core”)files,singleserialfileI/O,multiplefileI/O,andparallelI/O.Thefiledrivermapsasimple
abstractHDF5filetothespecificaccessmethods.
TherawdataofanHDF5datasetisconceivedtobeamultidimensionalarrayofdataelements.Thisarray
maybestoredinthefileaccordingtoseveralstoragestrategies:
• Contiguous
• Chunked
•Compact
Thestoragestrategydoesnotaffectdataaccessmethodsexceptthatcertainoperationsmaybemoreor
lessefficientdependingonthestoragestrategyandtheaccesspatterns.
Overall,thedatatransferoperations(H5DreadandH5Dwrite)workidenticallyforanystoragemethod,
foranyfiledriver,andforanyfiltersandtransformations.TheHDF5Libraryautomaticallymanagesthe
Figure5‐4.Datalayoutsinanapplication
HDF5User’sGuide HDF5Datasets
TheHDFGroup 124
datatransferprocess.Insomecases,transferpropertiesshouldormustbeusedtopassadditionalparam‐
eterssuchasMPI/IOdirectiveswhenusingtheparallelfiledriver.
5.4.1.TheDataPipeline
WhendataiswrittenorreadtoorfromanHDF5file,theHDF5Librarypassesthedatathroughasequence
ofprocessingstepswhichareknownastheHDF5datapipeline.Thisdatapipelineperformsoperationson
thedatainmemorysuchasbyteswapping,alignment,scatter‐gather,andhyperslabselections.TheHDF5
Libraryautomaticallydetermineswhichoperationsareneededandmanagestheorganizationofmemory
operationssuchasextractingselectedelementsfromadatablock.Thedatapipelinemodulesoperateon
databuffers:eachmoduleprocessesabufferandpassesthetransformedbuffertothenextstage.
Thetablebelowliststhestagesofthedatapipeline.Thefigurebelowthetableshowstheorderofpro‐
cessingduringareadorwrite.
Table5‐4.Stagesofthedatapipeline
Layers Description
I/Oinitiation InitiationofHDF5I/Oactivities(H5DwriteandH5Dread)ina
user’sapplicationprogram.
Memoryhyperslabopera‐
tion
Dataisscatteredto(forread),orgatheredfrom(forwrite)the
application’smemorybuffer(bypassedifnodatatypeconversion
isneeded).
Datatypeconversion Datatypeisconvertedifitisdifferentbetweenmemoryandstor‐
age(bypassedifnodatatypeconversionisneeded).
Filehyperslaboperation Dataisgatheredfrom(forread),orscatteredto(forwrite)tofile
spaceinmemory(bypassedifnodatatypeconversionis
needed).
Filterpipeline Dataisprocessedbyfilterswhenitpasses.Datacanbemodified
andrestoredhere(bypassedifnodatatypeconversionis
needed,nofilterisenabled,ordatasetisnotchunked).
VirtualFileLayer Facilitateeasyplug‐infiledriverssuchasMPIOorPOSIXI/O.
ActualI/O ActualfiledriverusedbythelibrarysuchasMPIOorSTDIO.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 125
TheHDF5Libraryautomaticallyappliesthestagesasneeded.
Whenthememorydataspaceselectionisotherthanthewholedataspace,thememoryhyperslabstage
scatters/gathersthedataelementsbetweentheapplicationmemory(describedbytheselection)anda
contiguousmemorybufferforthepipeline.Onawrite,thisisagatheroperation;onaread,thisisascat‐
teroperation.
Whenthememorydatatypeisdifferentfromthefiledatatype,thedatatypeconversionstagetransforms
eachdataelement.Forexample,ifdataiswrittenfrom32‐bitbig‐endianmemory,andthefiledatatypeis
32‐bitlittle‐endian,thedatatypeconversionstagewillswapthebytesofeveryelements.Similarly,when
dataisreadfromthefiletonativememory,byteswappingwillbeappliedautomaticallywhenneeded.
Figure5‐5.Theprocessingorderinthedatapipeline
HDF5User’sGuide HDF5Datasets
TheHDFGroup 126
Thefilehyperslabstageissimilartothememoryhyperslabstage,butismanagingthearrangementofthe
elementsaccordingtothedataspaceselection.Whendataisread,dataelementsaregatheredfromthe
datablocksfromthefiletofillthecontiguousbufferswhicharethenprocessedbythepipeline.When
dataisread,theelementsfromabufferarescatteredtothedatablocksofthefile.
5.4.2.DataPipelineFilters
Inadditiontothestandardpipeline,optionalstages,calledfilters,canbeinsertedinthepipeline.The
standarddistributionincludesoptionalfilterstoimplementcompressionanderrorchecking.Userapplica‐
tionsmayaddcustomfiltersaswell.
TheHDF5Librarydistributionincludesoremploysseveraloptionalfilters.Thesearelistedinthetable
below.Thefiltersareappliedinthepipelinebetweenthevirtualfilelayerandthefilehyperslaboperation.
Seethefigureabove.Theapplicationcanuseanynumberoffiltersinanyorder.
Filtersmaybeusedonlyforchunkeddataandareappliedtochunksofdatabetweenthefilehyperslab
stageandthevirtualfilelayer.Atthisstageinthepipeline,thedataisorganizedasfixed‐sizeblocksofele‐
ments,andthefilterstageprocesseseachchunkseparately.
Filtersareselectedbydatasetcreationproperties,andsomebehaviormaybecontrolledbydatatransfer
properties.Thelibrarydetermineswhatfiltersmustbeappliedandappliesthemintheorderinwhich
theyweresetbytheapplication.Thatis,ifanapplicationcallsH5Pset_shuffleandthenH5Pset_de-
flatewhencreatingadataset’screationpropertylist,thelibrarywillapplytheshufflefilterfirstand
thenthedeflatefilter.
Formoreinformation,see"UsingtheN‐bitFilter"onpage 143.Formoreinformation,see"Usingthe
Scale‐offsetFilter"onpage 160.
Table5‐5.Datapipelinefilters
Filter Description
gzipcompression Datacompressionusingzlib.
Szipcompression DatacompressionusingtheSziplibrary.SeeTheHDFGroup
websiteformoreinformationregardingtheSzipfilter.
N‐bitcompression Datacompressionusinganalgorithmspecializedforn‐bit
datatypes.
Scale‐offsetcompression Datacompressionusinga“scaleandoffset”algorithm.
Shuffling Toimprovecompressionperformance,dataisregroupedby
itsbytepositioninthedataunit.Inotherwords,the1st,2nd,
3rd,and4thbytesofintegersarestoredtogetherrespectively.
Fletcher32 Fletcher32checksumforerror‐detection.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 127
5.4.3.FileDrivers
I/OisperformedbytheHDF5virtualfilelayer.Thefiledriverinterfacewritesandreadsblocksofdata;
eachdrivermoduleimplementstheinterfaceusingdifferentI/Omechanisms.Thetablebelowliststhefile
driverscurrentlysupported.NotethattheI/Omechanismsareseparatedfromthepipelineprocessing:
thepipelineandfilteroperationsareidenticalnomatterwhatdataaccessmechanismisused.
Eachfiledriverwrites/readscontiguousblocksofbytesfromalogicallycontiguousaddressspace.Thefile
driverisresponsibleformanagingthedetailsofthedifferentphysicalstoragemethods.
Inserialenvironments,everythingabovethevirtualfilelayertendstoworkidenticallynomatterwhat
storagemethodisused.
Someoptionsmayhavesubstantiallydifferentperformancedependingonthefiledriverthatisused.In
particular,multi‐fileandparallelI/Omayperformconsiderablydifferentlyfromserialdriversdepending
onchunkingandothersettings.
5.4.4.DataTransferPropertiestoManagethePipeline
Datatransferpropertiessetoptionalparametersthatcontrolpartsofthedatapipeline.Thefunctionlist‐
ingbelowshowstransferpropertiesthatcontrolthebehaviorofthelibrary.
Table5‐6.I/Ofiledrivers
FileDriver Description
H5FD_CORE Storeinmemory(optionalbackingstoretodiskfile).
H5FD_FAMILY Storeinasetoffiles.
H5FD_LOG Storeinloggingfile.
H5FD_MPIO StoreusingMPI/IO.
H5FD_MULTI Storeinmultiplefiles.Thereareseveraloptionstocontrollayout.
H5FD_SEC2 SerialI/OtofileusingUnix“section2”functions.
H5FD_STDIO SerialI/OtofileusingUnix“stdio”functions.
HDF5Datasets HDF5User’sGuide
128 TheHDFGroup
Somefiltersandfiledriversrequireoruseadditionalparametersfromtheapplicationprogram.Thesecan
bepassedinthedatatransferpropertylist.Thetablebelowshowsfiledriverpropertylistfunctions.
ThetransferpropertiesaresetinapropertylistwhichispassedasaparameteroftheH5Dreador
H5Dwritecall.Thetransferpropertiesarepassedtoeachpipelinestage.Eachstagemayuseorignore
anypropertyinthelist.Inshort,thereisonepropertylistthatcontainsalltheproperties.
FunctionListing5‐5.Datatransferpropertylistfunctions
CFunction Purpose
H5Pset_buffer Maximumsizeforthetypeconversionbufferandtheback‐
groundbuffer.Mayalsosupplypointerstoapplication‐allo‐
catedbuffers.
H5Pset_hyper_cache WhethertocachehyperslabblocksduringI/O.
H5Pset_btree_ratios SettheB‐treesplitratiosforadatasettransferpropertylist.
Thesplitratiosdeterminewhatpercentofchildrengointhe
firstnodewhenanodesplits.
FunctionListing5‐6.Filedriverpropertylistfunctions
CFunction Purpose
H5Pset_dxpl_mpio ControltheMPII/Otransfermode(independent
orcollective)duringdataI/Ooperations.
H5Pset_small_data_block_size Reservesblocksofsizebytesforthecontiguous
storageoftherawdataportionofsmalldatasets.
TheHDF5Librarythenwritestherawdatafrom
smalldatasetstothisreservedspacewhich
reducesunnecessarydiscontinuitieswithinblocks
ofmetadataandimprovesI/Operformance.
H5Pset_edc_check Disable/enableEDCcheckingforread.When
selected,EDCisalwayswritten.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 129
5.4.5.StorageStrategies
Therawdataisconceptuallyamulti‐dimensionalarrayofelementsthatisstoredasacontiguousarrayof
bytes.Thedatamaybephysicallystoredinthefileinseveralways.Thetablebelowliststhestoragestrat‐
egiesforadataset.
Thedifferentstoragestrategiesdonotaffectthedatatransferoperationsofthedataset:readsandwrites
workthesameforanystoragestrategy.
Thesestrategiesaredescribedinthefollowingsections.
Contiguous
Acontiguousdatasetisstoredinthefileasaheaderandasinglecontinuousarrayofbytes.Seethefigure
below.Inthecaseofamulti‐dimensionalarray,thedataisserializedinrowmajororder.Bydefault,data
isstoredcontiguously.
Contiguousstorageisthesimplestmodel.Ithasseverallimitations.First,thedatasetmustbeafixed‐size:
itisnotpossibletoextendthelimitofthedatasetortohaveunlimiteddimensions.Inotherwords,ifthe
numberofdimensionsofthearraymightchangeovertime,thenchunkingstoragemustbeusedinstead
ofcontiguous.Second,becausedataispassedthroughthepipelineasfixed‐sizeblocks,compressionand
otherfilterscannotbeusedwithcontiguousdata.
Table5‐7.Datasetstoragestrategies
StorageStrategy Description
Contiguous Thedatasetisstoredasonecontinuousarrayofbytes.
Chunked Thedatasetisstoredasfixed‐sizechunks.
Compact Asmalldatasetisstoredinthemetadataheader.
Figure5‐6.Contiguousdatastorage
HDF5Datasets HDF5User’sGuide
130 TheHDFGroup
Chunked
Thedataofadatasetmaybestoredasfixed‐sizechunks.Seethefigurebelow.Achunkisahyper‐rectan‐
gleofanyshape.Whenadatasetischunked,eachchunkisreadorwrittenasasingleI/Ooperation,and
individuallypassedfromstagetostageofthedatapipeline.
Chunksmaybeanysizeandshapethatfitsinthedataspaceofthedataset.Forexample,athreedimen‐
sionaldataspacecanbechunkedas3‐Dcubes,2‐Dplanes,or1‐Dlines.Thechunksmayextendbeyond
thesizeofthedataspace.Forexample,a3x3datasetmightbychunkedin2x2chunks.Sufficientchunks
willbeallocatedtostorethearray,andanyextraspacewillnotbeaccessible.So,tostorethe3x3array,
four2x2chunkswouldbeallocatedwith5unusedelementsstored.
Chunkeddatasetscanbeunlimitedinanydirectionandcanbecompressedorfiltered.
Sincethedataisreadorwrittenbychunks,chunkingcanhaveadramaticeffectonperformancebyopti‐
mizingwhatisreadandwritten.Note,too,thatforspecificaccesspatternssuchasparallelI/O,decompo‐
sitionintochunkscanhavealargeimpactonperformance.
Tworestrictionshavebeenplacedonchunkshapeandsize:
•Therankofachunkmustbelessthanorequaltotherankofthedataset
• Chunksizecannotexceedthesizeofafixed‐sizedataset;forexample,adatasetconsistingofa5x
4fixed‐sizearraycannotbedefinedwith10x10chunks
Compact
Forcontiguousandchunkedstorage,thedatasetheaderinformationanddataarestoredintwo(ormore)
blocks.Therefore,atleasttwoI/Ooperationsarerequiredtoaccessthedata:onetoaccesstheheader,
andone(ormore)toaccessdata.Forasmalldataset,thisisconsiderableoverhead.
Asmalldatasetmaybestoredinacontinuousarrayofbytesintheheaderblockusingthecompactstor‐
ageoption.Thisdatasetcanbereadentirelyinoneoperationwhichretrievestheheaderanddata.The
datasetmustfitintheheader.Thismayvarydependingonthemetadatathatisstored.Ingeneral,acom‐
pactdatasetshouldbeapproximately30KBorlesstotalsize.Seethefigurebelow.
Figure5‐7.Chunkeddatastorage
HDF5User’sGuide HDF5Datasets
TheHDFGroup 131
5.4.6.PartialI/OSub‐settingandHyperslabs
Datatransferscanwriteorreadsomeofthedataelementsofthedataset.Thisiscontrolledbyspecifying
twoselections:oneforthesourceandoneforthedestination.Selectionsarespecifiedbycreatinga
dataspacewithselections.
Selectionsmaybeaunionofhyperslabsoralistofpoints.Ahyperslabisacontiguoushyper‐rectangle
fromthedataspace.Selectedfieldsofacompounddatatypemaybereadorwritten.Inthiscase,the
selectioniscontrolledbythememoryandfiledatatypes.
Summaryofprocedure:
1. Openthedataset
2. Definethememorydatatype
3. Definethememorydataspaceselectionandfiledataspaceselection
4. Transferdata(H5DreadorH5Dwrite)
Formoreinformation,see"HDF5DataspacesandPartialI/O"onpage 265.
5.5.AllocationofSpaceintheFile
Whenadatasetiscreated,spaceisallocatedinthefileforitsheaderandinitialdata.Theamountofspace
allocatedwhenthedatasetiscreateddependsonthestorageproperties.Whenthedatasetismodified
(dataiswritten,attributesadded,orotherchanges),additionalstoragemaybeallocatedifnecessary.
Figure5‐8.Compactdatastorage
HDF5Datasets HDF5User’sGuide
132 TheHDFGroup
Header
Adatasetheaderconsistsofoneormoreheadermessagescontainingpersistentmetadatadescribingvar‐
iousaspectsofthedataset.TheserecordsaredefinedintheHDF5FileFormatSpecification.Theamount
ofstoragerequiredforthemetadatadependsonthemetadatatobestored.Thetablebelowsummarizes
themetadata.
Theheaderblocksalsostorethenameandvaluesofattributes,sothetotalstoragedependsonthenum‐
berandsizeoftheattributes.
Inaddition,thedatasetmusthaveatleastonelink,includinganame,whichisstoredinthefileandinthe
groupitislinkedfrom.
Thedifferentstoragestrategiesdeterminewhenandhowmuchspaceisallocatedforthedataarray.See
thediscussionoffillvaluesbelowforadetailedexplanationofthestorageallocation.
ContiguousStorage
Foracontinuousstorageoption,thedataisstoredinasingle,contiguousblockinthefile.Thedatais
nominallyafixed‐size,(numberofelementsxsizeofelement).Thefigurebelowshowsanexampleofa
twodimensionalarraystoredasacontiguousdataset.
Table5‐8.Initialdatasetsize
Object Size
Header Variable,buttypicallyaround256bytesatthecreationofasimpledataset
withasimpledatatype.
Data Sizeofthedataarray(numberofelementsxsizeofelement).Spaceallocated
inthefiledependsonthestoragestrategyandtheallocationstrategy.
Table5‐9.Metadatastoragesizes
HeaderInformation ApproximateStorageSize
Datatype(required) Bytesormore.Dependsontype.
Dataspace(required) Bytesormore.Dependsonnumberofdimensionsandhsize_t.
Layout(required) Pointstothestoreddata.Bytesormore.Dependsonhsize_tand
numberofdimensions.
Filters Dependsonthenumberoffilters.Thesizeofthefiltermessage
dependsonthenameanddatathatwillbepassed.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 133
Dependingonthefillvalueproperties,thespacemaybeallocatedwhenthedatasetiscreatedorwhen
firstwritten(default),andfilledwithfillvaluesifspecified.ForparallelI/O,bydefaultthespaceisallo‐
catedwhenthedatasetiscreated.
ChunkedStorage
Forchunkedstorage,thedataisstoredinoneormorechunks.Eachchunkisacontinuousblockinthefile,
butchunksarenotnecessarilystoredcontiguously.Eachchunkhasthesamesize.Thedataarrayhasthe
samenominalsizeasacontiguousarray(numberofelementsxsizeofelement),butthestorageisallo‐
catedinchunks,sothetotalsizeinthefilecanbelargerthatthenominalsizeofthearray.Seethefigure
below.
Ifafillvalueisdefined,eachchunkwillbefilledwiththefillvalue.Chunksmustbeallocatedwhendatais
written,buttheymaybeallocatedwhenthefileiscreated,asthefileexpands,orwhendataiswritten.
ForserialI/O,bydefaultchunksareallocatedincrementally,asdataiswrittentothechunk.Forasparse
dataset,chunksareallocatedonlyforthepartsofthedatasetthatarewritten.Inthiscase,ifthedataset
isextended,nostorageisallocated.
ForparallelI/O,bydefaultchunksareallocatedwhenthedatasetiscreatedorextendedwithfillvalues
writtentothechunk.
Ineithercase,thedefaultcanbechangedusingfillvalueproperties.Forexample,usingserialI/O,the
propertiescanselecttoallocatechunkswhenthedatasetiscreated.
Figure5‐9.Atwodimensionalarraystoredasacontiguousdataset
HDF5Datasets HDF5User’sGuide
134 TheHDFGroup
ChangingDatasetDimensions
H5Dset_extentisusedtochangethecurrentdimensionsofthedatasetwithinthelimitsofthe
dataspace.Eachdimensioncanbeextendeduptoitsmaximumorunlimited.Extendingthedataspace
mayormaynotallocatespaceinthefileandmayormaynotwritefillvalues,iftheyaredefined.Seethe
examplecodebelow.
Thedimensionsofthedatasetcanalsoreduced.Ifthesizesspecifiedaresmallerthanthedataset’scur‐
rentdimensionsizes,H5Dset_extentwillreducethedataset’sdimensionsizestothespecifiedvalues.It
istheuser’sresponsibilitytoensurethatvaluabledataisnotlost;H5Dset_extentdoesnotcheck.
Figure5‐10.Atwodimensionalarraystoredinchunks
hid_t file_id, dataset_id;
Herr_t status;
size_t newdims[2];
/* Open an existing file. */
file_id = H5Fopen("dset.h5", H5F_ACC_RDWR, H5P_DEFAULT);
CodeExample5‐7.UsingH5Dset_extenttoincreasethesizeofadataset
HDF5User’sGuide HDF5Datasets
TheHDFGroup 135
5.5.1.StorageAllocationintheFile:Early,Incremental,Late
TheHDF5Libraryimplementsseveralstrategiesforwhenstorageisallocatedifandwhenitisfilledwith
fillvaluesforelementsnotyetwrittenbytheuser.Differentstrategiesarerecommendedfordifferent
storagelayoutsandfiledrivers.Inparticular,aparallelprogramneedsstorageallocatedduringacollective
call(forexample,createorextend)whileserialprogramsmaybenefitfromdelayingtheallocationuntil
thedataiswritten.
Twofilecreationpropertiescontrolwhentoallocatespace,whentowritethefillvalue,andtheactualfill
valuetowrite.
WhentoAllocateSpace
Thetablebelowshowstheoptionsforwhendataisallocatedinthefile.Earlyallocationisdoneduringthe
datasetcreatecall.Certainfiledrivers(especiallyMPI‐I/OandMPI‐POSIX)requirespacetobeallocated
whenadatasetiscreated,soallprocessorswillhavethecorrectviewofthedata.
/* Open an existing dataset. */
dataset_id = H5Dopen(file_id, "/dset", H5P_DEFAULT);
/* Example: dataset is 2 x 3, each dimension is UNLIMITED */
/* extend to 2 x 7 */
newdims[0] = 2;
newdims[1] = 7;
status = H5Dset_extent(dataset_id, newdims);
/* dataset is now 2 x 7 */
status = H5Dclose(dataset_id);
Table5‐10.Filestorageallocationoptions
Strategy Description
Early Allocatestorageforthedatasetimmediatelywhenthedatasetiscre‐
ated.
Late Deferallocatingspaceforstoringthedatasetuntilthedatasetiswritten.
CodeExample5‐7.UsingH5Dset_extenttoincreasethesizeofadataset
HDF5Datasets HDF5User’sGuide
136 TheHDFGroup
Lateallocationisdoneatthetimeofthefirstwritetodataset.Spaceforthewholedatasetisallocatedat
thefirstwrite.
Incrementalallocation(chunksonly)isdoneatthetimeofthefirstwritetothechunk.Chunksthathave
neverbeenwrittenarenotallocatedinthefile.Inasparselypopulateddataset,thisoptionallocates
chunksonlywheredataisactuallywritten.
The“Default”propertyselectstheoptionrecommendedasappropriateforthestoragemethodand
accessmethod.Thedefaultsareshowninthetablebelow.NotethatEarlyallocationisrecommendedfor
allParallelI/O,whileotheroptionsarerecommendedasthedefaultforserialI/Ocases.
WhentoWritetheFillValue
Thesecondpropertyiswhentowritethefillvalue.Thepossiblevaluesare“Never”and“Allocation”.The
tablebelowshowstheseoptions.
Incremental Deferallocatingspaceforstoringeachchunkuntilthechunkiswritten.
Default Usethestrategy(Early,Late,orIncremental)forthestoragemethodand
accessmethod.Thisistherecommendedstrategy.
Table5‐11.Defaultstorageoptions
StorageType SerialI/O ParallelI/O
Contiguous Late Early
Chunked Incremental Early
Compact Early Early
Table5‐12.Whentowritefillvalues
When Description
Never Fillvaluewillneverbewritten.
Allocation Fillvalueiswrittenwhenspaceisallocated.(Defaultforchunkedandcontigu‐
ousdatastorage.)
Table5‐10.Filestorageallocationoptions
Strategy Description
HDF5User’sGuide HDF5Datasets
TheHDFGroup 137
WhatFillValuetoWrite
Thethirdpropertyisthefillvaluetowrite.Thetablebelowshowsthevalues.Bydefault,thedataisfilled
withzeros.Theapplicationmaychoosenofillvalue(Undefined).Inthiscase,uninitializeddatamayhave
randomvalues.Theapplicationmaydefineafillvalueofanappropriatetype.Formoreinformation,see
"FillValues"onpage 238.
Togetherthesethreepropertiescontrolthelibrary’sbehavior.Thetablebelowsummarizesthepossibili‐
tiesduringthedatasetcreate‐write‐closecycle.
Table5‐13.Fillvaluestowrite
WhattoWrite Description
Default Bydefault,thelibraryfillsallocatedspacewithzeros.
Undefined Allocatedspaceisfilledwithrandomvalues.
User‐defined Theapplicationspecifiesthefillvalue.
Table5‐14.Storageallocationandfillsummary
Whento
allocate
space
Whento
writefill
value
Whatfill
valueto
write
Librarycreate‐write‐closebehavior
Early Never ‐Libraryallocatesspacewhendatasetiscre‐
ated,butneverwritesafillvaluetodataset.A
readofunwrittendatareturnsundefinedval‐
ues.
Late Never ‐Libraryallocatesspacewhendatasetiswrit‐
tento,butneverwritesafillvaluetothe
dataset.Areadofunwrittendatareturns
undefinedvalues.
Incremental Never ‐Libraryallocatesspacewhenadatasetor
chunk(whicheveristhesmallestunitof
space)iswrittento,butitneverwritesafill
valuetoadatasetorachunk.Areadof
unwrittendatareturnsundefinedvalues.
‐Allocation Undefined Erroroncreatingthedataset.Thedatasetis
notcreated.
HDF5Datasets HDF5User’sGuide
138 TheHDFGroup
DuringtheH5Dreadfunctioncall,thelibrarybehaviordependsonwhetherspacehasbeenallocated,
whetherthefillvaluehasbeenwrittentostorage,howthefillvalueisdefined,andwhentowritethefill
value.Thetablebelowsummarizesthedifferentbehaviors.
Early Allocation Defaultor
User‐defined
Allocatespaceforthedatasetwhenthedata‐
setiscreated.Writethefillvalue(defaultor
user‐defined)totheentiredatasetwhenthe
datasetiscreated.
Late Allocation Defaultor
User‐defined
Allocatespaceforthedatasetwhentheappli‐
cationfirstwritesdatavaluestothedataset.
Writethefillvaluetotheentiredataset
beforewritingapplicationdatavalues.
Incremental Allocation Defaultor
User‐defined
Allocatespaceforthedatasetwhentheappli‐
cationfirstwritesdatavaluestothedataset
orchunk(whicheveristhesmallestunitof
space).Writethefillvaluetotheentiredata‐
setorchunkbeforewritingapplicationdata
values.
Table5‐15.H5Dreadsummary
Isspace
allocatedin
thefile?
Whatisthe
fillvalue?
Whento
writethe
fillvalue?
Libraryreadbehavior
No Undefined <<any>> Error.Cannotcreatethisdataset.
No Defaultor
User‐defined
<<any>> Fillthememorybufferwiththefillvalue.
Yes Undefined <<any>> Returndatafromstorage(dataset).Trashis
possibleiftheapplicationhasnotwritten
datatotheportionofthedatasetbeingread.
Yes Defaultor
User‐defined
Never Returndatafromstorage(dataset).Trashis
possibleiftheapplicationhasnotwritten
datatotheportionofthedatasetbeingread.
Yes Defaultor
User‐defined
Allocation Returndatafromstorage(dataset).
Table5‐14.Storageallocationandfillsummary
Whento
allocate
space
Whento
writefill
value
Whatfill
valueto
write
Librarycreate‐write‐closebehavior
HDF5User’sGuide HDF5Datasets
TheHDFGroup 139
Therearetwocasestoconsiderdependingonwhetherthespaceinthefilehasbeenallocatedbeforethe
readornot.Whenspacehasnotyetbeenallocatedandifafillvalueisdefined,thememorybufferwillbe
filledwiththefillvaluesandreturned.Inotherwords,nodatahasbeenreadfromthedisk.Ifspacehas
beenallocated,thevaluesarereturnedfromthestoreddata.Theunwrittenelementswillbefilledaccord‐
ingtothefillvalue.
5.5.2.DeletingaDatasetfromaFileandReclaimingSpace
HDF5doesnotatthistimeprovideaneasymechanismtoremoveadatasetfromafileortoreclaimthe
storagespaceoccupiedbyadeletedobject.
RemovingadatasetandreclaimingthespaceitusedcanbedonewiththeH5Ldeletefunctionandthe
h5repackutilityprogram.WiththeH5Ldeletefunction,linkstoadatasetcanberemovedfromthefile
structure.Afterallthelinkshavebeenremoved,thedatasetbecomesinaccessibletoanyapplicationand
iseffectivelyremovedfromthefile.Thewaytorecoverthespaceoccupiedbyanunlinkeddatasetisto
writealloftheobjectsofthefileintoanewfile.Anyunlinkedobjectisinaccessibletotheapplicationand
willnotbeincludedinthenewfile.Writingobjectstoanewfilecanbedonewithacustomprogramor
withtheh5repackutilityprogram.
Formoreinformation,see"HDF5Groups"onpage 79.
5.5.3.ReleasingMemoryResources
ThesystemresourcesrequiredforHDF5objectssuchasdatasets,datatypes,anddataspacesshouldbe
releasedonceaccesstotheobjectisnolongerneeded.Thisisaccomplishedviatheappropriateclose
function.ThisisnotuniquetodatasetsbutageneralrequirementwhenworkingwiththeHDF5Library;
failuretocloseobjectswillresultinresourceleaks.
Inthecasewhereadatasetiscreatedordatahasbeentransferred,thereareseveralobjectsthatmustbe
closed.Theseobjectsincludedatasets,datatypes,dataspaces,andpropertylists.
Theapplicationprogrammustfreeanymemoryvariablesandbuffersitallocates.Whenaccessingdata
fromthefile,theamountofmemoryrequiredcanbedeterminedbycalculatingthesizeofthememory
datatypeandthenumberofelementsinthememoryselection.
Variable‐lengthdataareorganizedintwoormoreareasofmemory.Formoreinformation,see"Variable‐
lengthDatatypes"onpage 228.Whenwritingdata,theapplicationcreatesanarrayofvl_info_twhich
containspointerstotheelements.Theelementsmightbe,forexample,strings.Inthefile,thevariable‐
lengthdataisstoredintwoparts:aheapwiththevariable‐lengthvaluesofthedataelementsandanarray
ofvl_info_telements.Whenthedataisread,theamountofmemoryrequiredfortheheapcanbe
determinedwiththeH5Dget_vlen_buf_sizecall.
Thedatatransferpropertymaybeusedtosetacustommemorymanagerforallocatingvariable‐length
dataforaH5Dread.ThisissetwiththeH5Pset_vlen_mem_managercall.
HDF5Datasets HDF5User’sGuide
140 TheHDFGroup
Tofreethememoryforvariable‐lengthdata,itisnecessarytovisiteachelement,freethevariable‐length
data,andresettheelement.Theapplicationmustfreethememoryithasallocated.Formemoryallocated
bytheHDF5Libraryduringaread,theH5Dvlen_reclaimfunctioncanbeusedtoperformthisopera‐
tion.
5.5.4.ExternalStorageProperties
Theexternalstorageformatallowsdatatobestoredacrossasetofnon‐HDF5files.Asetofsegments(off‐
setsandsizes)inoneormorefilesisdefinedasanexternalfilelist,orEFL,andthecontiguouslogical
addressesofthedatastoragearemappedontothesesegments.Currently,onlytheH5D_CONTIGUOUS
storageformatallowsexternalstorage.Externalstorageisenabledbyadatasetcreationproperty.The
tablebelowshowstheAPI.
Table5‐16.ExternalstorageAPI
Function Description
herr_t H5Pset_external (hid_t
plist, const char *name, off_t
offset, hsize_t size)
Thisfunctionaddsanewsegmenttotheend
oftheexternalfilelistofthespecifieddataset
creationpropertylist.Thesegmentbeginsa
byteoffsetoffilenameandcontinuesforsize
bytes.Thespacerepresentedbythissegment
isadjacenttothespacealreadyrepresented
bytheexternalfilelist.Thelastsegmentina
filelistmayhavethesizeH5F_UNLIMITED,in
whichcasetheexternalfilemaybeofunlim‐
itedsizeandnomorefilescanbeaddedto
theexternalfileslist.
int H5Pget_external_count (hid_t
plist)
Callingthisfunctionreturnsthenumberof
segmentsinanexternalfilelist.Ifthedataset
creationpropertylisthasnoexternaldata,
thenzeroisreturned.
herr_t H5Pget_external (hid_t
plist, int idx, size_t name_size,
char *name, off_t *offset, hsize_t
*size)
ThisisthecounterpartfortheH5Pset_ex-
ternal()function.Givenadatasetcreation
propertylistandazero‐basedindexintothat
list,thefilename,byteoffset,andsegment
sizearereturnedthroughnon‐nullargu‐
ments.Atmostname_sizecharactersare
copiedintothenameargumentwhichisnot
nullterminatedifthefilenameislongerthan
thesuppliednamebuffer(thisissimilarto
strncpy()).
HDF5User’sGuide HDF5Datasets
TheHDFGroup 141
Thefigurebelowshowsanexampleofhowacontiguous,one‐dimensionaldatasetispartitionedinto
threepartsandeachofthosepartsisstoredinasegmentofanexternalfile.Thetoprectanglerepresents
thelogicaladdressspaceofthedatasetwhilethebottomrectanglerepresentsanexternalfile.
Theexamplebelowshowscodethatdefinestheexternalstoragefortheexample.Notethatthesegments
aredefinedinorderofthelogicaladdressestheyrepresent,nottheirorderwithintheexternalfile.It
wouldalsohavebeenpossibletoputthesegmentsinseparatefiles.Careshouldbetakenwhensettingup
segmentsinasinglefilesincethelibrarydoesnotautomaticallycheckforsegmentsthatoverlap.
Thefigurebelowshowsanexampleofhowacontiguous,two‐dimensionaldatasetispartitionedinto
threepartsandeachofthosepartsisstoredinaseparateexternalfile.Thetoprectanglerepresentsthe
logicaladdressspaceofthedatasetwhilethebottomrectanglesrepresentexternalfiles.
Figure5‐11.Externalfilestorage
Plist = H5Pcreate (H5P_DATASET_CREATE);
H5Pset_external (plist, "velocity.data", 3000, 1000);
H5Pset_external (plist, "velocity.data", 0, 2500);
H5Pset_external (plist, "velocity.data", 4500, 1500);
CodeExample5‐8.Externalstorage
HDF5Datasets HDF5User’sGuide
142 TheHDFGroup
Theexamplebelowshowscodeforthepartitioningdescribedabove.Inthisexample,thelibrarymapsthe
multi‐dimensionalarrayontoalinearaddressspaceasdefinedbytheHDF5formatspecification,andthen
mapsthataddressspaceintothesegmentsdefinedintheexternalfilelist.
Thesegmentsofanexternalfilecanexistbeyondtheendofthe(external)file.Thelibraryreadsthatpart
ofasegmentaszeros.Whenwritingtoasegmentthatexistsbeyondtheendofafile,theexternalfileis
automaticallyextended.Usingthisfeature,onecancreateasegment(orsetofsegments)whichislarger
thanthecurrentsizeofthedataset.Thisallowsthedatasettobeextendedatafuturetime(providedthe
dataspacealsoallowstheextension).
Figure5‐12.Partitioninga2‐Ddatasetforexternalstorage
Plist = H5Pcreate (H5P_DATASET_CREATE);
H5Pset_external (plist, "scan1.data", 0, 24);
H5Pset_external (plist, "scan2.data", 0, 24);
H5Pset_external (plist, "scan3.data", 0, 16);
CodeExample5‐9.Partitioninga2‐Ddatasetforexternalstorage
HDF5User’sGuide HDF5Datasets
TheHDFGroup 143
AllreferencedexternaldatafilesmustexistbeforeperformingrawdataI/Oonthedataset.Thisisnor‐
mallynotaproblemsincethosefilesarebeingmanageddirectlybytheapplicationorindirectlythrough
someotherlibrary.However,ifthefileistransferredfromitsoriginalcontext,caremustbetakentoassure
thatalltheexternalfilesareaccessibleinthenewlocation.
5.6.UsingHDF5Filters
Thissectiondescribesindetailhowtousethen‐bitandscale‐offsetfilters.
5.6.1.UsingtheN‐bitFilter
N‐bitdatahasnsignificantbits,wherenmaynotcorrespondtoaprecisenumberofbytes.Ontheother
hand,computingsystemsandapplicationsuniversally,ornearlyso,runmostefficientlywhenmanipulat‐
ingdataaswholebytesormultiplebytes.
Considerthecaseof12‐bitintegerdata.Inmemory,thatdatawillbehandledinatleast2bytes,or16bits,
andonsomeplatformsin4oreven8bytes.Thesizeofsuchadatasetcanbesignificantlyreducedwhen
writtentodiskiftheunusedbitsarestrippedout.
Then‐bitfilterisprovidedforthispurpose,packingn‐bitdataonoutputbystrippingoffallunusedbits
andunpackingoninput,restoringtheextrabitsrequiredbythecomputationalprocessor.
N‐bitDatatype
Ann‐bitdatatypeisadatatypeofnsignificantbits.Unlessitispacked,ann‐bitdatatypeispresentedasan
n‐bitbitfieldwithinalarger‐sizedvalue.Forexample,a12‐bitdatatypemightbepresentedasa12‐bit
fieldina16‐bit,or2‐byte,value.
Currently,thedatatypeclassesofn‐bitdatatypeorn‐bitfieldofacompounddatatypeoranarraydata‐
typearelimitedtointegerorfloating‐point.
TheHDF5usercancreateann‐bitdatatypethroughaseriesofoffunctioncalls.Forexample,thefollow‐
ingcallscreatea16‐bitdatatypethatisstoredina32‐bitvaluewitha4‐bitoffset:
hid_t nbit_datatype = H5Tcopy(H5T_STD_I32LE);
H5Tset_precision(nbit_datatype, 16);
H5Tset_offset(nbit_datatype, 4);
Inmemory,onevalueoftheaboveexamplen‐bitdatatypewouldbestoredonalittle‐endianmachineas
follows:
HDF5Datasets HDF5User’sGuide
144 TheHDFGroup
Note:Key:S‐signbit,P‐significantbit,?‐paddingbit.Signbitisincludedinsignedintegerdatatypeprecision.
N‐bitFilter
Whendataofann‐bitdatatypeisstoredondiskusingthen‐bitfilter,thefilterpacksthedatabystripping
offthepaddingbits;onlythesignificantbitsareretainedandstored.Thevaluesondiskwillappearasfol‐
lows:
Note:Key:S‐signbit,P‐significantbit,?‐paddingbit.Signbitisincludedinsignedintegerdatatypeprecision.
Then‐bitfiltercanbeusedeffectivelyforcompressingdataofann‐bitdatatype,includingarraysandthe
n‐bitfieldsofcompounddatatypes.Thefiltersupportscomplexsituationswhereacompounddatatype
containsmember(s)ofacompounddatatypeoranarraydatatypehasacompounddatatypeasthebase
type.
Atpresent,then‐bitfiltersupportsalldatatypes.Fordatatypesofclasstime,string,opaque,reference,
ENUM,andvariable‐length,then‐bitfilteractsasano‐opwhichisshortfornooperation.Forconve‐
nience,therestofthissectionreferstosuchdatatypesasno‐opdatatypes.
AsisthecasewithallHDF5filters,anapplicationusingthen‐bitfiltermuststoredatawithchunkedstor‐
age.
HowDoestheN‐bitFilterWork?
Then‐bitfilteralwayscompressesanddecompressesaccordingtodatasetpropertiessuppliedbythe
HDF5Libraryinthedatatype,dataspace,ordatasetcreationpropertylist.
ThedatasetdatatypereferstohowdataisstoredinanHDF5filewhilethememorydatatypereferstohow
dataisstoredinmemory.TheHDF5Librarywilldodatatypeconversionwhenwritingdatainmemoryto
thedatasetorreadingdatafromthedatasettomemoryifthememorydatatypediffersfromthedataset
datatype.DatatypeconversionisperformedbyHDF5Librarybeforen‐bitcompressionandaftern‐bit
decompression.
Thefollowingsub‐sectionsexaminethecommoncases:
byte3byte2byte1byte0
???????? ????SPPP PPPPPPPP PPPP????
1stvalue 2ndvalue
SPPPPPPP PPPPPPPP SPPPPPPP PPPPPPPP ...
HDF5User’sGuide HDF5Datasets
TheHDFGroup 145
•N‐bitintegerconversions
•N‐bitfloating‐pointconversions
N‐bitIntegerConversions
Integerdatawithadatasetofintegerdatatypeoflessthanfullprecisionandamemorydatatypeof
H5T_NATIVE_INT,providesthesimplestapplicationofthen‐bitfilter.
TheprecisionofH5T_NATIVE_INTis8multipliedbysizeof(int).Thisvalue,thesizeofanintin
bytes,differsfromplatformtoplatform;weassumeavalueof4forthefollowingillustration.Wefurther
assumethememorybyteordertobelittle‐endian.
Inmemory,therefore,theprecisionofH5T_NATIVE_INTis32andtheoffsetis0.OnevalueofH5T_NA-
TIVE_INTislaidoutinmemoryasfollows:
Note:Key:S‐signbit,P‐significantbit,?‐paddingbit.Signbitisincludedinsignedintegerdatatypeprecision.
Supposethedatasetdatatypehasaprecisionof16andanoffsetof4.AfterHDF5convertsvaluesfromthe
memorydatatypetothedatasetdatatype,itpassessomethinglikethefollowingtothen‐bitfilterfor
compression:
Note:Key:S‐signbit,P‐significantbit,?‐paddingbit.Signbitisincludedinsignedintegerdatatypeprecision.
Noticethatonlythespecified16bits(15significantbitsandthesignbit)areretainedintheconversion.All
othersignificantbitsofthememorydatatypearediscardedbecausethedatasetdatatypecallsforonly16
bitsofprecision.Aftern‐bitcompression,noneofthesediscardedbits,knownaspaddingbitswillbe
storedondisk.
Figure5‐13.H5T_NATIVE_INTinmemory
Figure5‐14.Passedtothen‐bitfilter
HDF5Datasets HDF5User’sGuide
146 TheHDFGroup
N‐bitFloating‐pointConversions
Thingsgetmorecomplicatedinthecaseofafloating‐pointdatasetdatatypeclass.Thissub‐sectionpro‐
videsanexamplethatillustratestheconversionfromamemorydatatypeofH5T_NATIVE_FLOATtoa
datasetdatatypeofclassfloating‐point.
Asbefore,lettheH5T_NATIVE_FLOATbe4byteslong,andletthememorybyteorderbelittle‐endian.
PertheIEEEstandard,onevalueofH5T_NATIVE_FLOATislaidoutinmemoryasfollows:
Note:Key:S‐signbit,E‐exponentbit,M‐mantissabit,?‐paddingbit.Signbitisincludedinfloating‐pointdatatype
precision.
Supposethedatasetdatatypehasaprecisionof20,offsetof7,mantissasizeof13,mantissapositionof7,
exponentsizeof6,exponentpositionof20,andsignpositionof26.Formoreinformation,see"Definition
ofDatatypes"onpage 199.
AfterHDF5convertsvaluesfromthememorydatatypetothedatasetdatatype,itpassessomethinglike
thefollowingtothen‐bitfilterforcompression:
Note:Key:S‐signbit,E‐exponentbit,M‐mantissabit,?‐paddingbit.Signbitisincludedinfloating‐pointdatatype
precision.
ThesignbitandtruncatedmantissabitsarenotchangedduringdatatypeconversionbytheHDF5Library.
Ontheotherhand,theconversionofthe8‐bitexponenttoa6‐bitexponentisalittletricky:
Thebiasforthenewexponentinthen‐bitdatatypeis:
2(n-1)-1
Thefollowingformulaisusedforthisexponentconversion:
exp8 - (2(8-1)-1) = exp6 - (2(6-1)-1) = actual exponent value
Figure5‐15.H5T_NATIVE_FLOATinmemory
Figure5‐16.Passedtothen‐bitfilter
HDF5User’sGuide HDF5Datasets
TheHDFGroup 147
whereexp8isthestoreddecimalvalueasrepresentedbythe8‐bitexponent,andexp6isthe
storeddecimalvalueasrepresentedbythe6‐bitexponent.
Inthisexample,cautionmustbetakentoensurethat,afterconversion,theactualexponentvalueis
withintherangethatcanberepresentedbya6‐bitexponent.Forexample,an8‐bitexponentcanrepre‐
sentvaluesfrom‐127to128whilea6‐bitexponentcanrepresentvaluesonlyfrom‐31to32.
N‐bitFilterBehavior
Then‐bitfilterwasdesignedtotreattheincomingdatabytebybyteatthelowestlevel.Thepurposewas
tomakethen‐bitfilterasgenericaspossiblesothatnopointercastrelatedtothedatatypeisneeded.
Bitwiseoperationsareemployedforpackingandunpackingatthebytelevel.
Recursivefunctioncallsareusedtotreatcompoundandarraydatatypes.
N‐bitCompression
Themainideaofn‐bitcompressionistousealooptocompresseachdataelementinachunk.Depending
onthedatatypeofeachelement,then‐bitfilterwillcalloneoffourfunctions.Eachofthesefunctionsper‐
formsoneofthefollowingtasks:
•Compressadataelementofano‐opdatatype
•Compressadataelementofanatomicdatatype
•Compressadataelementofacompounddatatype
•Compressadataelementofanarraydatatype
No‐opdatatypes:Then‐bitfilterdoesnotactuallycompressno‐opdatatypes.Rather,itcopiesthedata
bufferoftheno‐opdatatypefromthenon‐compressedbuffertotheproperlocationinthecompressed
buffer;thecompressedbufferhasnoholes.Theterm“compress”isusedheresimplytodistinguishthis
functionfromthefunctionthatperformsthereverseoperationduringdecompression.
Atomicdatatypes:Then‐bitfilterwillfindthebyteswheresignificantbitsarelocatedandtrytocompress
thesebytes,onebyteatatime,usingaloop.Atthislevel,thefilterneedsthefollowinginformation:
•Thebyteoffsetofthebeginningofthecurrentdataelementwithrespecttothebeginningofthe
inputdatabuffer
•Datatype
size,precision,offset,andbyteorder
Then‐bitfiltercompressesfromthemostsignificantbytecontainingsignificantbitstotheleastsignificant
byte.Forbig‐endiandata,therefore,theloopindexprogressesfromsmallertolargerwhileforlittle‐
endian,theloopindexprogressesfromlargertosmaller.
Intheextremecaseofwhenthen‐bitdatatypehasfullprecision,thisfunctioncopiesthecontentofthe
entirenon‐compresseddatatypetothecompressedoutputbuffer.
Compounddatatypes:Then‐bitfilterwillcompresseachdatamemberofthecompounddatatype.Ifthe
memberdatatypeisofanintegerorfloating‐pointdatatype,then‐bitfilterwillcallthefunctiondescribed
above.Ifthememberdatatypeisofano‐opdatatype,thefilterwillcallthefunctiondescribedabove.If
thememberdatatypeisofacompounddatatype,thefilterwillmakearecursivecalltoitself.Ifthemem‐
berdatatypeisofanarraydatatype,thefilterwillcallthefunctiondescribedbelow.
HDF5Datasets HDF5User’sGuide
148 TheHDFGroup
Arraydatatypes:Then‐bitfilterwillusealooptocompresseacharrayelementinthearray.Ifthebase
datatypeofarrayelementisofanintegerorfloating‐pointdatatype,then‐bitfilterwillcallthefunction
describedabove.Ifthebasedatatypeisofano‐opdatatype,thefilterwillcallthefunctiondescribed
above.Ifthebasedatatypeisofacompounddatatype,thefilterwillcallthefunctiondescribedabove.If
thememberdatatypeisofanarraydatatype,thefilterwillmakearecursivecallofitself.
N‐bitDecompression
Then‐bitdecompressionalgorithmisverysimilarton‐bitcompression.Theonlydifferenceisthatatthe
bytelevel,compressionpacksoutallpaddingbitsandstoresonlysignificantbitsintoacontinuousbuffer
(unsignedchar)whiledecompressionunpackssignificantbitsandinsertspaddingbits(zeros)atthe
properpositionstorecoverthedatabytesastheyexistedbeforecompression.
StoringN‐bitParameterstoArraycd_value[]
Alloftheinformation,orparameters,requiredbythen‐bitfilteraregatheredandstoredinthearray
cd_values[]bytheprivatefunctionH5Z_set_local_nbitandarepassedtoanotherprivatefunc‐
tion,H5Z_filter_nbit,bytheHDF5Library.
Theseparametersareasfollows:
• Parametersrelatedtothedatatype
•Thenumberofelementswithinthechunk
•Aflagindicatingwhethercompressionisneeded
ThefirstandsecondparameterscanbeobtainedusingtheHDF5dataspaceanddatatypeinterfacecalls.
Acompounddatatypecanhavemembersofarrayorcompounddatatype.Anarraydatatype’sbasedata‐
typecanbeacomplexcompounddatatype.Recursivecallsarerequiredtosetparametersforthesecom‐
plexsituations.
Beforesettingtheparameters,thenumberofparametersshouldbecalculatedtodynamicallyallocatethe
arraycd_values[],whichwillbepassedtotheHDF5Library.Thisalsorequiresrecursivecalls.
Foranatomicdatatype(integerorfloating‐point),parametersthatwillbestoredincludethedatatype’s
size,endianness,precision,andoffset.
Forano‐opdatatype,onlythesizeisrequired.
Foracompounddatatype,parametersthatwillbestoredincludethedatatype’stotalsizeandnumberof
members.Foreachmember,itsmemberoffsetneedstobestored.Otherparametersformemberswill
dependsontherespectivedatatypeclass.
Foranarraydatatype,thetotalsizeparametershouldbestored.Otherparametersforthearray’sbase
typedependonthebasetype’sdatatypeclass.
Further,tocorrectlyretrievetheparameterforuseofn‐bitcompressionordecompressionlater,parame‐
tersfordistinguishingbetweendatatypeclassesshouldbestored.
Implementation
Threefiltercallbackfunctionswerewrittenforthen‐bitfilter:
HDF5User’sGuide HDF5Datasets
TheHDFGroup 149
• H5Z_can_apply_nbit
• H5Z_set_local_nbit
• H5Z_filter_nbit
ThesefunctionsarecalledinternallybytheHDF5Library.Anumberofutilityfunctionswerewrittenfor
thefunctionH5Z_set_local_nbit.Compressionanddecompressionfunctionswerewrittenandare
calledbyfunctionH5Z_filter_nbit.AllthesefunctionsareincludedinthefileH5Znbit.c.
ThepublicfunctionH5Pset_nbitiscalledbytheapplicationtosetuptheuseofthen‐bitfilter.This
functionisincludedinthefileH5Pdcpl.c.Theapplicationdoesnotneedtosupplyanyparameters.
HowN‐bitParametersareStored
Aschemeofstoringparametersrequiredbythen‐bitfilterinthearraycd_values[]wasdevelopeduti‐
lizingrecursivefunctioncalls.
Fourprivateutilityfunctionswerewrittenforstoringtheparametersassociatedwithatomic(integeror
floating‐point),no‐op,array,andcompounddatatypes:
• H5Z_set_parms_atomic
• H5Z_set_parms_array
• H5Z_set_parms_nooptype
•H5Z_set_parms_compound
Theschemeisbrieflydescribedbelow.
First,assignanumericcodefordatatypeclassatomic(integerorfloat),no‐op,array,andcompounddata‐
type.Thecodeisstoredbeforeotherdatatyperelatedparametersarestored.
Thefirstthreeparametersofcd_values[]arereservedfor:
1. Thenumberofvalidentriesinthearraycd_values[]
2. Aflagindicatingwhethercompressionisneeded
3. Thenumberofelementsinthechunk
Throughoutthebalanceofthisexplanation,irepresentstheindexofcd_values[].
InthefunctionH5Z_set_local_nbit:
1. i=2
2. Getthenumberofelementsinthechunkandstoreincd_value[i];incrementi
3. Gettheclassofthedatatype:
•Foranintegerorfloating‐pointdatatype,callH5Z_set_parms_atomic
•Foranarraydatatype,callH5Z_set_parms_array
•Foracompounddatatype,callH5Z_set_parms_compound
•Fornoneoftheabove,callH5Z_set_parms_noopdatatype
4. Storeiincd_value[0]andflagincd_values[1]
InthefunctionH5Z_set_parms_atomic:
1. Storetheassignednumericcodefortheatomicdatatypeincd_value[i];incrementi
HDF5Datasets HDF5User’sGuide
150 TheHDFGroup
2. Getthesizeoftheatomicdatatypeandstoreincd_value[i];incrementi
3. Gettheorderoftheatomicdatatypeandstoreincd_value[i];incrementi
4. Gettheprecisionoftheatomicdatatypeandstoreincd_value[i];incrementi
5. Gettheoffsetoftheatomicdatatypeandstoreincd_value[i];incrementi
6. Determinetheneedtodocompressionatthispoint
InthefunctionH5Z_set_parms_nooptype:
1. Storetheassignednumericcodefortheno‐opdatatypeincd_value[i];incrementi
2. Getthesizeoftheno‐opdatatypeandstoreincd_value[i];incrementi
InthefunctionH5Z_set_parms_array:
1. Storetheassignednumericcodeforthearraydatatypeincd_value[i];incrementi
2. Getthesizeofthearraydatatypeandstoreincd_value[i];incrementi
3. Gettheclassofthearray’sbasedatatype.
•Foranintegerorfloating‐pointdatatype,callH5Z_set_parms_atomic
•Foranarraydatatype,callH5Z_set_parms_array
•Foracompounddatatype,callH5Z_set_parms_compound
•Ifnoneoftheabove,callH5Z_set_parms_noopdatatype
InthefunctionH5Z_set_parms_compound:
1. Storetheassignednumericcodeforthecompounddatatypeincd_value[i];incrementi
2. Getthesizeofthecompounddatatypeandstoreincd_value[i];incrementi
3. Getthenumberofmembersandstoreincd_values[i];incrementi
4. Foreachmember
•Getthememberoffsetandstoreincd_values[i];incrementi
•Gettheclassofthememberdatatype
•Foranintegerorfloating‐pointdatatype,callH5Z_set_parms_atomic
•Foranarraydatatype,callH5Z_set_parms_array
•Foracompounddatatype,callH5Z_set_parms_compound
•Ifnoneoftheabove,callH5Z_set_parms_noopdatatype
N‐bitCompressionandDecompressionFunctions
Then‐bitcompressionanddecompressionfunctionsabovearecalledbytheprivateHDF5functionH5Z_-
filter_nbit.Thecompressanddecompressfunctionsretrievethen‐bitparametersfromcd_val-
ues[]asitwaspassedbyH5Z_filter_nbit.Parametersareretrievedinexactlythesameorderin
whichtheyarestoredandlower‐levelcompressionanddecompressionfunctionsfordifferentdatatype
classesarecalled.
N‐bitcompressionisnotimplementedinplace.Duetothedifficultyofcalculatingactualoutputbuffer
sizeaftercompression,thesamespaceasthatoftheinputbufferisallocatedfortheoutputbufferas
passedtothecompressionfunction.However,thesizeoftheoutputbufferpassedbyreferencetothe
compressionfunctionwillbechanged(smaller)afterthecompressioniscomplete.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 151
UsageExamples
Thefollowingcodeexampleillustratestheuseofthen‐bitfilterforwritingandreadingn‐bitintegerdata.
HDF5Datasets HDF5User’sGuide
152 TheHDFGroup
#include "hdf5.h"
#include "stdlib.h"
#include "math.h"
#define H5FILE_NAME "nbit_test_int.h5"
#define DATASET_NAME "nbit_int"
#define NX 200
#define NY 300
#define CH_NX 10
#define CH_NY 15
int main(void)
{
hid_t file, dataspace, dataset, datatype, mem_datatype,
dset_create_props;
hsize_t dims[2], chunk_size[2];
int orig_data[NX][NY];
int new_data[NX][NY];
int i, j;
size_t precision, offset;
/* Define dataset datatype (integer), and set precision,
* offset
*/
datatype = H5Tcopy(H5T_NATIVE_INT);
precision = 17; /* precision includes sign bit */
if(H5Tset_precision(datatype,precision)<0) {
printf("Error: fail to set precision\n");
return -1;
}
offset = 4;
if(H5Tset_offset(datatype,offset)<0) {
printf("Error: fail to set offset\n");
return -1;
}
/* Copy to memory datatype */
mem_datatype = H5Tcopy(datatype);
CodeExample5‐10.N‐bitcompressionforintegerdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 153
/* Set order of dataset datatype */
if(H5Tset_order(datatype, H5T_ORDER_BE)<0) {
printf("Error: fail to set endianness\n");
return -1;
}
/* Initialize data buffer with random data within correct
* range corresponding to the memory datatype's precision
* and offset.
*/
for (i=0; i < NX; i++)
for (j=0; j < NY; j++)
orig_data[i][j] = rand() % (int)pow(2, precision-1)
<<offset;
/* Describe the size of the array. */
dims[0] = NX;
dims[1] = NY;
if((dataspace = H5Screate_simple (2, dims, NULL))<0) {
printf("Error: fail to create dataspace\n");
return -1;
}
/*
* Create a new file using read/write access, default file
* creation properties, and default file access properties.
*/
if((file = H5Fcreate (H5FILE_NAME, H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT))<0) {
printf("Error: fail to create file\n");
return -1;
}
/*
* Set the dataset creation property list to specify that
* the raw data is to be partitioned into 10 x 15 element
* chunks and that each chunk is to be compressed.
*/
chunk_size[0] = CH_NX;
chunk_size[1] = CH_NY;
CodeExample5‐10.N‐bitcompressionforintegerdata
HDF5Datasets HDF5User’sGuide
154 TheHDFGroup
if((dset_create_props = H5Pcreate (H5P_DATASET_CREATE))<0) {
printf("Error: fail to create dataset property\n");
return -1;
}
if(H5Pset_chunk (dset_create_props, 2, chunk_size)<0) {
printf("Error: fail to set chunk\n");
return -1;
}
/*
* Set parameters for n-bit compression; check the description
* of the H5Pset_nbit function in the HDF5 Reference Manual
* for more information.
*/
if(H5Pset_nbit (dset_create_props)<0) {
printf("Error: fail to set nbit filter\n");
return -1;
}
/*
* Create a new dataset within the file. The datatype
* and dataspace describe the data on disk, which may
* be different from the format used in the application's
* memory.
*/
if((dataset = H5Dcreate(file, DATASET_NAME, datatype,
dataspace, H5P_DEFAULT,
dset_create_props, H5P_DEFAULT))<0) {
printf("Error: fail to create dataset\n");
return -1;
}
/*
* Write the array to the file. The datatype and dataspace
* describe the format of the data in the 'orig_data' buffer.
* The raw data is translated to the format required on disk,
* as defined above. We use default raw data transfer
* properties.
*/
CodeExample5‐10.N‐bitcompressionforintegerdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 155
Note:Thecodeexampleaboveillustratestheuseofthen‐bitfilterforwritingandreadingn‐bitintegerdata.
Thefollowingcodeexampleillustratestheuseofthen‐bitfilterforwritingandreadingn‐bitfloating‐point
data.
if(H5Dwrite (dataset, mem_datatype, H5S_ALL, H5S_ALL,
H5P_DEFAULT, orig_data)<0) {
printf("Error: fail to write to dataset\n");
return -1;
}
H5Dclose (dataset);
if((dataset = H5Dopen(file, DATASET_NAME, H5P_DEFAULT))<0) {
printf("Error: fail to open dataset\n");
return -1;
}
/*
* Read the array. This is similar to writing data,
* except the data flows in the opposite direction.
* Note: Decompression is automatic.
*/
if(H5Dread (dataset, mem_datatype, H5S_ALL, H5S_ALL,
H5P_DEFAULT, new_data)<0) {
printf("Error: fail to read from dataset\n");
return -1;
}
H5Tclose (datatype);
H5Tclose (mem_datatype);
H5Dclose (dataset);
H5Sclose (dataspace);
H5Pclose (dset_create_props);
H5Fclose (file);
return 0;
}
CodeExample5‐10.N‐bitcompressionforintegerdata
HDF5Datasets HDF5User’sGuide
156 TheHDFGroup
#include "hdf5.h"
#define H5FILE_NAME "nbit_test_float.h5"
#define DATASET_NAME "nbit_float"
#define NX 2
#define NY 5
#define CH_NX 2
#define CH_NY 5
int main(void)
{
hid_t file, dataspace, dataset, datatype, dset_create_props;
hsize_t dims[2], chunk_size[2];
/* orig_data[] are initialized to be within the range that
* can be represented by dataset datatype (no precision
* loss during datatype conversion)
*/
float orig_data[NX][NY] = {{188384.00, 19.103516,
-1.0831790e9, -84.242188, 5.2045898}, {-49140.000,
2350.2500, -3.2110596e-1, 6.4998865e-5, -0.0000000}};
float new_data[NX][NY];
size_t precision, offset;
/* Define single-precision floating-point type for dataset
*---------------------------------------------------------------
* size=4 byte, precision=20 bits, offset=7 bits,
* mantissa size=13 bits, mantissa position=7,
* exponent size=6 bits, exponent position=20,
* exponent bias=31.
* It can be illustrated in little-endian order as:
* (S - sign bit, E - exponent bit, M - mantissa bit,
* ? - padding bit)
*
*3210
* ?????SEE EEEEMMMM MMMMMMMM M???????
*
* To create a new floating-point type, the following
* properties must be set in the order of
* set fields -> set offset -> set precision -> set size.
CodeExample5‐11.N‐bitcompressionforfloating‐pointdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 157
* All these properties must be set before the type can
* function. Other properties can be set anytime. Derived
* type size cannot be expanded bigger than original size
* but can be decreased. There should be no holes
* among the significant bits. Exponent bias usually
* is set 2^(n-1)-1, where n is the exponent size.
*---------------------------------------------------------------*/
datatype = H5Tcopy(H5T_IEEE_F32BE);
if(H5Tset_fields(datatype, 26, 20, 6, 7, 13)<0) {
printf("Error: fail to set fields\n");
return -1;
}
offset = 7;
if(H5Tset_offset(datatype,offset)<0) {
printf("Error: fail to set offset\n");
return -1;
}
precision = 20;
if(H5Tset_precision(datatype,precision)<0) {
printf("Error: fail to set precision\n");
return -1;
}
if(H5Tset_size(datatype, 4)<0) {
printf("Error: fail to set size\n");
return -1;
}
if(H5Tset_ebias(datatype, 31)<0) {
printf("Error: fail to set exponent bias\n");
return -1;
}
/* Describe the size of the array. */
dims[0] = NX;
dims[1] = NY;
if((dataspace = H5Screate_simple (2, dims, NULL))<0) {
printf("Error: fail to create dataspace\n");
return -1;
}
/*
* Create a new file using read/write access, default file
* creation properties, and default file access properties.
*/
CodeExample5‐11.N‐bitcompressionforfloating‐pointdata
HDF5Datasets HDF5User’sGuide
158 TheHDFGroup
if((file = H5Fcreate (H5FILE_NAME, H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT))<0) {
printf("Error: fail to create file\n");
return -1;
}
/*
* Set the dataset creation property list to specify that
* the raw data is to be partitioned into 2 x 5 element
* chunks and that each chunk is to be compressed.
*/
chunk_size[0] = CH_NX;
chunk_size[1] = CH_NY;
if((dset_create_props = H5Pcreate (H5P_DATASET_CREATE))<0) {
printf("Error: fail to create dataset property\n");
return -1;
}
if(H5Pset_chunk (dset_create_props, 2, chunk_size)<0) {
printf("Error: fail to set chunk\n");
return -1;
}
/*
* Set parameters for n-bit compression; check the description
* of the H5Pset_nbit function in the HDF5 Reference Manual
* for more information.
*/
if(H5Pset_nbit (dset_create_props)<0) {
printf("Error: fail to set nbit filter\n");
return -1;
}
/*
* Create a new dataset within the file. The datatype
* and dataspace describe the data on disk, which may
* be different from the format used in the application's
* memory.
*/
CodeExample5‐11.N‐bitcompressionforfloating‐pointdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 159
if((dataset = H5Dcreate(file, DATASET_NAME, datatype,
dataspace, H5P_DEFAULT,
dset_create_plists, H5P_DEFAULT))<0) {
printf("Error: fail to create dataset\n");
return -1;
}
/*
* Write the array to the file. The datatype and dataspace
* describe the format of the data in the 'orig_data' buffer.
* The raw data is translated to the format required on disk,
* as defined above. We use default raw data transfer
* properties.
*/
if(H5Dwrite (dataset, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, orig_data)<0) {
printf("Error: fail to write to dataset\n");
return -1;
}
H5Dclose (dataset);
if((dataset = H5Dopen(file, DATASET_NAME, H5P_DEFAULT))<0) {
printf("Error: fail to open dataset\n");
return -1;
}
/*
* Read the array. This is similar to writing data,
* except the data flows in the opposite direction.
* Note: Decompression is automatic.
*/
CodeExample5‐11.N‐bitcompressionforfloating‐pointdata
HDF5Datasets HDF5User’sGuide
160 TheHDFGroup
Note:Thecodeexampleaboveillustratestheuseofthen‐bitfilterforwritingandreadingn‐bitfloating‐pointdata.
Limitations
Becausethearraycd_values[]hastofitintoanobjectheadermessageof64K,then‐bitfilterhasan
upperlimitonthenumberofn‐bitparametersthatcanbestoredinit.Tobeconservative,amaximumof
4Kisallowedforthenumberofparameters.
Then‐bitfiltercurrentlyonlycompressesn‐bitdatatypesorfieldsderivedfromintegerorfloating‐point
datatypes.Then‐bitfilterassumespaddingbitsofzero.ThismaynotbetruesincetheHDF5usercanset
paddingbittobezero,one,orleavethebackgroundalone.However,itisexpectedthen‐bitfilterwillbe
modifiedtoadjusttosuchsituations.
Then‐bitfilterdoesnothaveawaytohandlethesituationwherethefillvalueofadatasetisdefinedand
thefillvalueisnotofann‐bitdatatypealthoughthedatasetdatatypeis.
5.6.2.UsingtheScale‐offsetFilter
Generallyspeaking,scale‐offsetcompressionperformsascaleand/oroffsetoperationoneachdatavalue
andtruncatestheresultingvaluetoaminimumnumberofbits(minimum‐bits)beforestoringit.
Thecurrentscale‐offsetfiltersupportsintegerandfloating‐pointdatatypesonly.Forthefloating‐point
datatype,floatanddoublearesupported,butlongdoubleisnotsupported.
Integerdatacompressionusesastraight‐forwardalgorithm.Floating‐pointdatacompressionadoptsthe
GRiBdatapackingmechanismwhichofferstwoalternatemethods:afixedminimum‐bitsmethod,anda
variableminimum‐bitsmethod.Currently,onlythevariableminimum‐bitsmethodisimplemented.
LikeotherI/OfilterssupportedbytheHDF5Library,applicationsusingthescale‐offsetfiltermuststore
datawithchunkedstorage.
if(H5Dread (dataset, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, new_data)<0) {
printf("Error: fail to read from dataset\n");
return -1;
}
H5Tclose (datatype);
H5Dclose (dataset);
H5Sclose (dataspace);
H5Pclose (dset_create_props);
H5Fclose (file);
return 0;
}
CodeExample5‐11.N‐bitcompressionforfloating‐pointdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 161
Integertype:Theminimum‐bitsofintegerdatacanbedeterminedbythefilter.Forexample,ifthemaxi‐
mumvalueofdatatobecompressedis7065andtheminimumvalueis2970.Thenthe“span”ofdataset
valuesisequalto(max‐min+1),whichis4676.Ifnofillvalueisdefinedforthedataset,theminimum‐bits
is:ceiling(log2(span)) = 12.Withfillvalueset,theminimum‐bitsis:ceiling(log2(span+1))
= 13.
HDF5userscanalsosettheminimum‐bits.However,iftheusergivesaminimum‐bitsthatislessthanthat
calculatedbythefilter,thecompressionwillbelossy.
Floating‐pointtype:Thebasicideaofthescale‐offsetfilterforthefloating‐pointtypeistotransformthe
databysomekindofscalingtointegerdata,andthentofollowtheprocedureofthescale‐offsetfilterfor
theintegertypetodothedatacompression.Duetothedatatransformationfromfloating‐pointtointe‐
ger,thescale‐offsetfilterislossyinnature.
Twomethodsofscalingthefloating‐pointdataareused:theso‐calledD‐scalingandE‐scaling.D‐scalingis
morestraightforwardandeasytounderstand.ForHDF51.8release,onlytheD‐scalingmethodhasbeen
implemented.
Design
Beforethefilterdoesanyrealwork,itneedstogathersomeinformationfromtheHDF5Librarythrough
APIcalls.Theparametersthefilterneedsare:
•Theminimum‐bitsofthedatavalue
•Thenumberofdataelementsinthechunk
•Thedatatypeclass,size,sign(onlyforintegertype),byteorder,andfillvalueifdefined
Sizeandsignareneededtodeterminewhatkindofpointercasttousewhenretrievingvaluesfromthe
databuffer.
Thepipelineofthefiltercanbedividedintofourparts:(1)pre‐compression;(2)compression;(3)decom‐
pression;(4)post‐decompression.
Dependingonwhetherafillvalueisdefinedornot,thefilterwillhandlepre‐compressionandpost‐
decompressiondifferently.
Thescale‐offsetfilteronlyneedsthememorybyteorder,sizeofdatatype,andminimum‐bitsforcompres‐
sionanddecompression.
Sincedecompressionhasnoaccesstotheoriginaldata,theminimum‐bitsandtheminimumvalueneed
tobestoredwiththecompresseddatafordecompressionandpost‐decompression.
IntegerType
Pre‐compression:Duringpre‐compressionminimum‐bitsiscalculatedifitisnotsetbytheuser.Formore
informationonhowminimum‐bitsarecalculated,seesection6.1.“TheN‐bitFilter.”
Ifthefillvalueisdefined,findingthemaximumandminimumvaluesshouldignorethedataelement
whosevalueisequaltothefillvalue.
Ifnofillvalueisdefined,thevalueofeachdataelementissubtractedbytheminimumvalueduringthis
stage.
HDF5Datasets HDF5User’sGuide
162 TheHDFGroup
Ifthefillvalueisdefined,thefillvalueisassignedtothemaximumvalue.Inthiswayminimum‐bitscan
representadataelementwhosevalueisequaltothefillvalueandsubtractstheminimumvaluefroma
dataelementwhosevalueisnotequaltothefillvalue.
Thefillvalue(ifdefined),thenumberofelementsinachunk,theclassofthedatatype,thesizeofthe
datatype,thememoryorderofthedatatype,andothersimilarelementswillbestoredintheHDF5object
headerforthepost‐decompressionusage.
Afterpre‐compression,allvaluesarenon‐negativeandarewithintherangethatcanbestoredbymini‐
mum‐bits.
Compression:Allmodifieddatavaluesafterpre‐compressionarepackedtogetherintothecompressed
databuffer.Thenumberofbitsforeachdatavaluedecreasesfromthenumberofbitsofinteger(32for
mostplatforms)tominimum‐bits.Thevalueofminimum‐bitsandtheminimumvalueareaddedtothe
databufferandthewholebufferissentbacktothelibrary.Inthisway,thenumberofbitsforeachmodi‐
fiedvalueisnomorethanthesizeofminimum‐bits.
Decompression:Inthisstage,thenumberofbitsforeachdatavalueisresumedfromminimum‐bitstothe
numberofbitsofinteger.
Post‐decompression:Forthepost‐decompressionstage,thefilterdoestheoppositeofwhatitdoesduring
pre‐compressionexceptthatitdoesnotcalculatetheminimum‐bitsortheminimumvalue.Thesevalues
weresavedduringcompressionandcanberetrievedthroughtheresumeddatabuffer.Ifnofillvalueis
defined,thefilteraddstheminimumvaluebacktoeachdataelement.
Ifthefillvalueisdefined,thefilterassignsthefillvaluetothedataelementwhosevalueisequaltothe
maximumvaluethatminimum‐bitscanrepresentandaddstheminimumvaluebacktoeachdataelement
whosevalueisnotequaltothemaximumvaluethatminimum‐bitscanrepresent.
Floating‐pointType
Thefilterwilldodatatransformationfromfloating‐pointtypetointegertypeandthenhandlethedataby
usingtheprocedureforhandlingtheintegerdatainsidethefilter.Insignificantbitsoffloating‐pointdata
willbecutoffduringdatatransformation,sothisfilterisalossycompressionmethod.
Therearetwoscalingmethods:D‐scalingandE‐scaling.TheHDF51.8releaseonlysupportsD‐scaling.D‐
scalingisshortfordecimalscaling.E‐scalingshouldbesimilarconceptually.Inordertotransformdata
fromfloating‐pointtointeger,ascalefactorisintroduced.Theminimumvaluewillbecalculated.Each
dataelementvaluewillsubtracttheminimumvalue.Themodifieddatawillbemultipliedby10(Decimal)
tothepowerofscale_factor,andonlytheintegerpartwillbekeptandmanipulatedthroughtherou‐
tinesfortheintegertypeofthefilterduringpre‐compressionandcompression.Integerdatawillbe
dividedby10tothepowerofscale_factortotransformbacktofloating‐pointdataduringdecompres‐
sionandpost‐decompression.Eachdataelementvaluewillthenaddtheminimumvalue,andthefloating‐
pointdataareresumed.However,theresumeddatawilllosesomeinsignificantbitscomparedwiththe
originalvalue.
Forexample,thefollowingfloating‐pointdataaremanipulatedbythefilter,andtheD‐scalingfactoris2.
{104.561, 99.459, 100.545, 105.644}
Theminimumvalueis99.459,eachdataelementsubtracts99.459,themodifieddatais
HDF5User’sGuide HDF5Datasets
TheHDFGroup 163
{5.102, 0, 1.086, 6.185}
SincetheD‐scalingfactoris2,allfloating‐pointdatawillbemultipliedby10^2withthisresult:
{510.2, 0, 108.6, 618.5}
Thedigitafterdecimalpointwillberoundedoff,andthenthesetlookslike:
{510, 0, 109, 619}
Afterdecompression,eachvaluewillbedividedby10^2andwillbeaddedtotheoffset99.459.
Thefloating‐pointdatabecomes
{104.559, 99.459, 100.549, 105.649}.
Therelativeerrorforeachvalueshouldbenomorethan5*(10^(D‐scalingfactor+1)).D‐scalingsome‐
timesisalsoreferredasavariableminimum‐bitsmethodsincefordifferentdatasetstheminimum‐bitsto
representthesamedecimalprecisionwillvary.Thedatavalueismodifiedto2topowerofscale_fac-
torforE‐scaling.E‐scalingisalsocalledfixed‐bitsmethodsincefordifferentdatasetstheminimum‐bits
willalwaysbefixedtothescalefactorofE‐scaling.Currently,HDF5ONLYsupportstheD‐scaling(variable
minimum‐bits)method.
Implementation
Thescale‐offsetfilterimplementationwaswrittenandincludedinthefileH5Zscaleoffset.c.Function
H5Pset_scaleoffsetwaswrittenandincludedinthefile“H5Pdcpl.c”.TheHDF5usercansupply
minimum‐bitsbycallingfunctionH5Pset_scaleoffset.
Thescale‐offsetfilterwasimplementedbasedonthedesignoutlinedinthissection.However,thefollow‐
ingfactorsneedtobeconsidered:
1. Thefilterneedstheappropriatecastpointerwheneveritneedstoretrievedatavalues.
2. TheHDF5Librarypassestothefiltertheto‐be‐compresseddataintheformatofthedatasetdata‐
type,andthefilterpassesbackthedecompresseddatainthesameformat.Ifafillvalueis
defined,itisalsoindatasetdatatypeformat.Forexample,ifthebyteorderofthedatasetdata‐
typeisdifferentfromthatofthememorydatatypeoftheplatform,compressionordecompres‐
sionperformsanendiannessconversionofdatabuffer.Moreover,itshouldbeawarethat
memorybyteordercanbedifferentduringcompressionanddecompression.
3. Thedifferenceofendiannessanddatatypebetweenfileandmemoryshouldbeconsideredwhen
savingandretrievalofminimum‐bits,minimumvalue,andfillvalue.
4. Iftheusersetstheminimum‐bitstofullprecisionofthedatatype,nooperationisneededatthe
filterside.Ifthefullprecisionisaresultofcalculationbythefilter,thentheminimum‐bitsneeds
tobesavedfordecompressionbutnocompressionordecompressionisneeded(onlyacopyof
theinputbufferisneeded).
5. Ifbycalculationofthefilter,theminimum‐bitsisequaltozero,specialhandlingisneeded.Since
itmeansallvaluesarethesame,nocompressionordecompressionisneeded.Buttheminimum‐
bitsandminimumvaluestillneedtobesavedduringcompression.
6. Forfloating‐pointdata,theminimumvalueofthedatasetshouldbecalculatedatfirst.Eachdata
elementvaluewillthensubtracttheminimumvaluetoobtainthe“offset”data.Theoffsetdata
HDF5Datasets HDF5User’sGuide
164 TheHDFGroup
willthenfollowthestepsoutlinedaboveinthediscussionoffloating‐pointtypestododatatrans‐
formationtointegerandrounding.Formoreinformation,see"Floating‐pointType"onpage 162.
UsageExamples
Thefollowingcodeexampleillustratestheuseofthescale‐offsetfilterforwritingandreadinginteger
data.
HDF5User’sGuide HDF5Datasets
TheHDFGroup 165
#include "hdf5.h"
#include "stdlib.h"
#define H5FILE_NAME "scaleoffset_test_int.h5"
#define DATASET_NAME "scaleoffset_int"
#define NX 200
#define NY 300
#define CH_NX 10
#define CH_NY 15
int main(void)
{
hid_t file, dataspace, dataset, datatype, dset_create_props;
hsize_t dims[2], chunk_size[2];
int orig_data[NX][NY];
int new_data[NX][NY];
int i, j, fill_val;
/* Define dataset datatype */
datatype = H5Tcopy(H5T_NATIVE_INT);
/* Initiliaze data buffer */
for (i=0; i < NX; i++)
for (j=0; j < NY; j++)
orig_data[i][j] = rand() % 10000;
/* Describe the size of the array. */
dims[0] = NX;
dims[1] = NY;
if((dataspace = H5Screate_simple (2, dims, NULL))<0) {
printf("Error: fail to create dataspace\n");
return -1;
}
/*
* Create a new file using read/write access, default file
* creation properties, and default file access properties.
*/
if((file = H5Fcreate (H5FILE_NAME, H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT))<0) {
printf("Error: fail to create file\n");
return -1;
}
CodeExample5‐12.Scale‐offsetcompressionintegerdata
HDF5Datasets HDF5User’sGuide
166 TheHDFGroup
/*
* Set the dataset creation property list to specify that
* the raw data is to be partitioned into 10 x 15 element
* chunks and that each chunk is to be compressed.
*/
chunk_size[0] = CH_NX;
chunk_size[1] = CH_NY;
if((dset_create_props = H5Pcreate (H5P_DATASET_CREATE))<0) {
printf("Error: fail to create dataset property\n");
return -1;
}
if(H5Pset_chunk (dset_create_props, 2, chunk_size)<0) {
printf("Error: fail to set chunk\n");
return -1;
}
/* Set the fill value of dataset */
fill_val = 10000;
if (H5Pset_fill_value(dset_create_props, H5T_NATIVE_INT,
&fill_val)<0) {
printf("Error: can not set fill value for dataset\n");
return -1;
}
/*
* Set parameters for scale-offset compression. Check the
* description of the H5Pset_scaleoffset function in the
* HDF5 Reference Manual for more information [3].
*/
if(H5Pset_scaleoffset (dset_create_props, H5Z_SO_INT,
H5Z_SO_INT_MINIMUMBITS_DEFAULT)<0) {
printf("Error: fail to set scaleoffset filter\n");
return -1;
}
/*
* Create a new dataset within the file. The datatype
* and dataspace describe the data on disk, which may
* or may not be different from the format used in the
* application's memory. The link creation and
* dataset access property list parameters are passed
* with default values.
*/
CodeExample5‐12.Scale‐offsetcompressionintegerdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 167
if((dataset = H5Dcreate (file, DATASET_NAME, datatype,
dataspace, H5P_DEFAULT,
dset_create_props, H5P_DEFAULT))<0) {
printf("Error: fail to create dataset\n");
return -1;
}
/*
* Write the array to the file. The datatype and dataspace
* describe the format of the data in the 'orig_data' buffer.
* We use default raw data transfer properties.
*/
if(H5Dwrite (dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, orig_data)<0) {
printf("Error: fail to write to dataset\n");
return -1;
}
H5Dclose (dataset);
if((dataset = H5Dopen(file, DATASET_NAME, H5P_DEFAULT))<0) {
printf("Error: fail to open dataset\n");
return -1;
}
/*
* Read the array. This is similar to writing data,
* except the data flows in the opposite direction.
* Note: Decompression is automatic.
*/
CodeExample5‐12.Scale‐offsetcompressionintegerdata
HDF5Datasets HDF5User’sGuide
168 TheHDFGroup
Note:Thecodeexampleaboveillustratestheuseofthescale‐offsetfilterforwritingandreadingintegerdata.
Thefollowingcodeexampleillustratestheuseofthescale‐offsetfilter(setforvariableminimum‐bits
method)forwritingandreadingfloating‐pointdata.
if(H5Dread (dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, new_data)<0) {
printf("Error: fail to read from dataset\n");
return -1;
}
H5Tclose (datatype);
H5Dclose (dataset);
H5Sclose (dataspace);
H5Pclose (dset_create_props);
H5Fclose (file);
return 0;
}
CodeExample5‐12.Scale‐offsetcompressionintegerdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 169
#include "hdf5.h"
#include "stdlib.h"
#define H5FILE_NAME "scaleoffset_test_float_Dscale.h5"
#define DATASET_NAME "scaleoffset_float_Dscale"
#define NX 200
#define NY 300
#define CH_NX 10
#define CH_NY 15
int main(void)
{
hid_t file, dataspace, dataset, datatype, dset_create_props;
hsize_t dims[2], chunk_size[2];
float orig_data[NX][NY];
float new_data[NX][NY];
float fill_val;
int i, j;
/* Define dataset datatype */
datatype = H5Tcopy(H5T_NATIVE_FLOAT);
/* Initiliaze data buffer */
for (i=0; i < NX; i++)
for (j=0; j < NY; j++)
orig_data[i][j] = (rand() % 10000) / 1000.0;
/* Describe the size of the array. */
dims[0] = NX;
dims[1] = NY;
if((dataspace = H5Screate_simple (2, dims, NULL))<0) {
printf("Error: fail to create dataspace\n");
return -1;
}
/*
* Create a new file using read/write access, default file
* creation properties, and default file access properties.
*/
CodeExample5‐13.Scale‐offsetcompressionfloating‐pointdata
HDF5Datasets HDF5User’sGuide
170 TheHDFGroup
if((file = H5Fcreate (H5FILE_NAME, H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT))<0) {
printf("Error: fail to create file\n");
return -1;
}
/*
* Set the dataset creation property list to specify that
* the raw data is to be partitioned into 10 x 15 element
* chunks and that each chunk is to be compressed.
*/
chunk_size[0] = CH_NX;
chunk_size[1] = CH_NY;
if((dset_create_props = H5Pcreate (H5P_DATASET_CREATE))<0) {
printf("Error: fail to create dataset property\n");
return -1;
}
if(H5Pset_chunk (dset_create_props, 2, chunk_size)<0) {
printf("Error: fail to set chunk\n");
return -1;
}
/* Set the fill value of dataset */
fill_val = 10000.0;
if (H5Pset_fill_value(dset_create_props, H5T_NATIVE_FLOAT,
&fill_val)<0) {
printf("Error: can not set fill value for dataset\n");
return -1;
}
/*
* Set parameters for scale-offset compression; use variable
* minimum-bits method, set decimal scale factor to 3. Check
* the description of the H5Pset_scaleoffset function in the
* HDF5 Reference Manual for more information [3].
*/
if(H5Pset_scaleoffset (dset_create_props, H5Z_SO_FLOAT_DSCALE,
3)<0) {
printf("Error: fail to set scaleoffset filter\n");
return -1;
}
CodeExample5‐13.Scale‐offsetcompressionfloating‐pointdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 171
/*
* Create a new dataset within the file. The datatype
* and dataspace describe the data on disk, which may
* or may not be different from the format used in the
* application's memory.
*/
if((dataset = H5Dcreate (file, DATASET_NAME, datatype,
dataspace, H5P_DEFAULT,
dset_create_props, H5P_DEFAULT))<0) {
printf("Error: fail to create dataset\n");
return -1;
}
/*
* Write the array to the file. The datatype and dataspace
* describe the format of the data in the 'orig_data' buffer.
* We use default raw data transfer properties.
*/
if(H5Dwrite (dataset, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, orig_data)<0) {
printf("Error: fail to write to dataset\n");
return -1;
}
H5Dclose (dataset);
if((dataset = H5Dopen(file, DATASET_NAME, H5P_DEFAULT))<0) {
printf("Error: fail to open dataset\n");
return -1;
}
/*
* Read the array. This is similar to writing data,
* except the data flows in the opposite direction.
* Note: Decompression is automatic.
*/
CodeExample5‐13.Scale‐offsetcompressionfloating‐pointdata
HDF5Datasets HDF5User’sGuide
172 TheHDFGroup
Note:Thecodeexampleaboveillustratestheuseofthescale‐offsetfilterforwritingandreadingfloating‐pointdata.
Limitations
Forfloating‐pointdatahandling,therearesomealgorithmiclimitationstotheGRiBdatapackingmecha‐
nism:
1. BoththeE‐scalingandD‐scalingmethodsarelossycompression
2. FortheD‐scalingmethod,sincedatavalueshavebeenroundedtointegervalues(positive)before
truncatingtotheminimum‐bits,theirrangeislimitedbythemaximumvaluethatcanberepre‐
sentedbythecorrespondingunsignedintegertype(thesamesizeasthatofthefloating‐point
type)
Suggestions
Thefollowingaresomesuggestionsforusingthefilterforfloating‐pointdata:
1. Itisbettertoconverttheunitsofdatasothattheunitsarewithincertaincommonrange(for
example,1200mto1.2km)
2. Ifdatavaluestobecompressedareveryneartozero,itisstronglyrecommendedthattheuser
setsthefillvalueawayfromzero(forexample,alargepositivenumber);iftheuserdoesnothing,
theHDF5Librarywillsetthefillvaluetozero,andthismaycauseundesirablecompressionresults
3. Usersarenotencouragedtouseaverylargedecimalscalefactor(forexample,100)fortheD‐
scalingmethod;thiscancausethefilternottoignorethefillvaluewhenfindingmaximumand
minimumvalues,andtheywillgetamuchlargerminimum‐bits(poorcompression)
5.6.3.UsingtheSzipFilter
SeeTheHDFGroupwebsiteforfurtherinformationregardingtheSzipfilter.
if(H5Dread (dataset, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, new_data)<0) {
printf("Error: fail to read from dataset\n");
return -1;
}
H5Tclose (datatype);
H5Dclose (dataset);
H5Sclose (dataspace);
H5Pclose (dset_create_props);
H5Fclose (file);
return 0;
}
CodeExample5‐13.Scale‐offsetcompressionfloating‐pointdata
HDF5User’sGuide HDF5Datasets
TheHDFGroup 173
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 173
6.HDF5Datatypes
6.1.IntroductionandDefinitions
AnHDF5datasetisanarrayofdataelements,arrangedaccordingtothespecificationsofthedataspace.In
general,adataelementisthesmallestaddressableunitofstorageintheHDF5file.(Compounddatatypes
aretheexceptiontothisrule.)TheHDF5datatypedefinesthestorageformatforasingledataelement.
Seethefigurebelow.
ThemodelforHDF5attributesisextremelysimilartodatasets:anattributehasadataspaceandadata‐
type,asshowninthefigurebelow.Theinformationinthischapterappliestobothdatasetsandattributes.
Abstractly,eachdataelementwithinthedatasetisasequenceofbits,interpretedasasinglevaluefroma
setofvalues(forexample,anumberoracharacter).Foragivendatatype,thereisastandardorconven‐
tionforrepresentingthevaluesasbits,andwhenthebitsarerepresentedinaparticularstoragethebits
arelaidoutinaspecificstorageschemesuchas8‐bitbyteswithaspecificorderingandalignmentofbytes
withinthestoragearray.
HDF5datatypesimplementaflexible,extensible,andportablemechanismforspecifyinganddiscovering
thestoragelayoutofthedataelements,determininghowtointerprettheelements(forexample,asfloat‐
ingpointnumbers),andfortransferringdatafromdifferentcompatiblelayouts.
Figure6‐1.Datatypes,dataspaces,anddatasets
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 174
AnHDF5datatypedescribesonespecificlayoutofbits.Adatasethasasingledatatypewhichappliesto
everydataelement.Whenadatasetiscreated,thestoragedatatypeisdefined.Afterthedatasetorattri‐
buteiscreated,thedatatypecannotbechanged.
•Thedatatypedescribesthestoragelayoutofasingledataelement
•Allelementsofthedatasetmusthavethesametype
•Thedatatypeofadatasetisimmutable
Whendataistransferred(forexample,areadorwrite),eachendpointofthetransferhasadatatype,
whichdescribesthecorrectstoragefortheelements.Thesourceanddestinationmayhavedifferent(but
compatible)layouts,inwhichcasethedataelementsareautomaticallytransformedduringthetransfer.
HDF5datatypesdescribecommonlyusedbinaryformatsfornumbers(integersandfloatingpoint)and
characters(ASCII).Agivencomputingarchitectureandprogramminglanguagesupportscertainnumber
andcharacterrepresentations.Forexample,acomputermaysupport8‐,16‐,32‐,and64‐bitsignedinte‐
gers,storedinmemoryinlittle‐endianbyteorder.ThesewouldpresumablycorrespondtotheCprogram‐
minglanguagetypes‘char’,‘short’,‘int’,and‘long’.
Whenreadingandwritingfrommemory,theHDF5Librarymustknowtheappropriatedatatypethat
describesthearchitecturespecificlayout.TheHDF5Libraryprovidestheplatformindependent‘NATIVE’
types,whicharemappedtoanappropriatedatatypeforeachplatform.Sothetype‘H5T_NATIVE_INT’is
analiasfortheappropriatedescriptorforeachplatform.
Datainmemoryhasadatatype:
•Thestoragelayoutinmemoryisarchitecture‐specific
•TheHDF5‘NATIVE’typesarepredefinedaliasesforthearchitecture‐specificmemorylayout
•Thememorydatatypeneednotbethesameasthestoreddatatypeofthedataset
Inadditiontonumbersandcharacters,anHDF5datatypecandescribemoreabstractclassesoftypes
includingenumerations,strings,bitstrings,andreferences(pointerstoobjectsintheHDF5file).HDF5
supportsseveralclassesofcompositedatatypeswhicharecombinationsofoneormoreotherdatatypes.
Inadditiontothestandardpredefineddatatypes,userscandefinenewdatatypeswithinthedatatype
classes.
TheHDF5datatypemodelisverygeneralandflexible:
•Forcommonsimplepurposes,onlypredefinedtypeswillbeneeded
•Datatypescanbecombinedtocreatecomplexstructureddatatypes
•Ifneeded,userscandefinecustomatomicdatatypes
•Committeddatatypescanbesharedbydatasetsorattributes
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 175
6.2.HDF5DatatypeModel
TheHDF5Libraryimplementsanobject‐orientedmodelofdatatypes.HDF5datatypesareorganizedasa
logicalsetofbasetypes,ordatatypeclasses.Eachdatatypeclassdefinesaformatforrepresentinglogical
valuesasasequenceofbits.ForexampletheH5T_INTEGERclassisaformatforrepresentingtwoscom‐
plementintegersofvarioussizes.
Adatatypeclassisdefinedasasetofoneormoredatatypeproperties.Adatatypepropertyisaproperty
ofthebitstring.Thedatatypepropertiesaredefinedbythelogicalmodelofthedatatypeclass.Forexam‐
ple,theintegerclass(twoscomplementintegers)haspropertiessuchas“signedorunsigned”,“length”,
and“byte‐order”.Thefloatclass(IEEEfloatingpointnumbers)hastheseproperties,plus“exponentbits”,
“exponentsign”,etc.
Adatatypeisderivedfromonedatatypeclass:agivendatatypehasaspecificvalueforthedatatypeprop‐
ertiesdefinedbytheclass.Forexample,for32‐bitsignedintegers,storedbig‐endian,theHDF5datatype
isasub‐typeofintegerwiththepropertiessettosigned=1,size=4(bytes),andbyte-order=BE.
TheHDF5datatypeAPI(H5Tfunctions)providesmethodstocreatedatatypesofdifferentdatatype
classes,tosetthedatatypepropertiesofanewdatatype,andtodiscoverthedatatypepropertiesofan
existingdatatype.
ThedatatypeforadatasetisstoredintheHDF5fileaspartofthemetadataforthedataset.
Adatatypecanbesharedbymorethanonedatasetinthefileifthedatatypeissavedtothefilewitha
name.Thisshareabledatatypeisknownasacommitteddatatype.Inthepast,thiskindofdatatypewas
calledanameddatatype.
Whentransferringdata(forexample,areadorwrite),thedataelementsofthesourceanddestination
storagemusthavecompatibletypes.Asageneralrule,dataelementswiththesamedatatypeclassare
compatiblewhileelementsfromdifferentdatatypeclassesarenotcompatible.Whentransferringdataof
onedatatypetoanothercompatibledatatype,theHDF5Libraryusesthedatatypepropertiesofthe
sourceanddestinationtoautomaticallytransformeachdataelement.Forexample,whenreadingfrom
datastoredas32‐bitsignedintegers,big‐endianinto32‐bitsignedintegers,little‐endian,theHDF5Library
willautomaticallyswapthebytes.
Thus,datatransferoperations(H5Dread,H5Dwrite,H5Aread,H5Awrite)requireadatatypeforboth
thesourceandthedestination.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 176
TheHDF5Librarydefinesasetofpredefineddatatypes,correspondingtocommonlyusedstoragefor‐
mats,suchastwoscomplementintegers,IEEEFloatingpointnumbers,etc.,4‐and8‐bytesizes,big‐endian
andlittle‐endianbyteorders.Inaddition,ausercanderivetypeswithcustomvaluesfortheproperties.
Forexample,auserprogrammaycreateadatatypetodescribea6‐bitinteger,ora600‐bitfloatingpoint
number.
Inadditiontoatomicdatatypes,theHDF5Librarysupportscompositedatatypes.Acompositedatatypeis
anaggregationofoneormoredatatypes.Eachclassofcompositedatatypeshaspropertiesthatdescribe
theorganizationofthecompositedatatype.Seethefigurebelow.Compositedatatypesinclude:
• Compounddatatypes:structuredrecords
•Array:amultidimensionalarrayofadatatype
• Variable‐length:aone‐dimensionalarrayofadatatype
Figure6‐2.Thedatatypemodel
Figure6‐3.Compositedatatypes
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 177
6.2.1.DatatypeClassesandProperties
ThefigurebelowshowstheHDF5datatypeclasses.Eachclassisdefinedtohaveasetofpropertieswhich
describethelayoutofthedataelementandtheinterpretationofthebits.Thetablebelowliststheprop‐
ertiesforthedatatypeclasses.
Figure6‐4.Datatypeclasses
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 178
Table6‐1.Datatypeclassesandtheirproperties
Class Description Properties Notes
Integer Twoscomplement
integers
Size(bytes),precision
(bits),offset(bits),pad,
byteorder,signed/
unsigned
Float FloatingPoint
numbers
Size(bytes),precision
(bits),offset(bits),pad,
byteorder,signposition,
exponentposition,expo‐
nentsize(bits),exponent
sign,exponentbias,man‐
tissaposition,mantissa
(size)bits,mantissasign,
mantissanormalization,
internalpadding
SeeIEEE754foradefini‐
tionoftheseproperties.
Thesepropertiesdescribe
non‐IEEE754floating
pointformatsaswell.
Character Arrayof1‐byte
characterencoding
Size(characters),Charac‐
terset,byteorder,pad/no
pad,padcharacter
Currently,ASCIIandUTF‐8
aresupported.
Bitfield Stringofbits Size(bytes),precision
(bits),offset(bits),pad,
byteorder
Asequenceofbitvalues
packedintooneormore
bytes.
Opaque Uninterpreteddata Size(bytes),precision
(bits),offset(bits),pad,
byteorder,tag
Asequenceofbytes,
storedandretrievedasa
block.The‘tag’isastring
thatcanbeusedtolabel
thevalue.
Enumeration Alistofdiscrete
values,withsym‐
bolicnamesinthe
formofstrings.
Numberofelements,ele‐
mentnames,elementval‐
ues
Enumerationisalistof
pairs(name,value).The
nameisastring;thevalue
isanunsignedinteger.
Reference Referenceto
objectorregion
withintheHDF5
file
SeetheReferenceAPI,
H5R
Array Array(1‐4dimen‐
sions)ofdataele‐
ments
Numberofdimensions,
dimensionsizes,base
datatype
Thearrayisaccessed
atomically:noselectionor
sub‐setting.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 179
6.2.2.PredefinedDatatypes
TheHDF5Librarypredefinesamodestnumberofcommonlyuseddatatypes.Thesetypeshavestandard
symbolicnamesoftheformH5T_arch_basewherearchisanarchitecturenameandbaseisapro‐
grammingtypename(Table2).Newtypescanbederivedfromthepredefinedtypesbycopyingthepre‐
definedtype(seeH5Tcopy())andthenmodifyingtheresult.
Thebasenameofmosttypesconsistsofalettertoindicatetheclass(Table3),aprecisioninbits,andan
indicationofthebyteorder(Table4).
Table5showsexamplesofpredefineddatatypes.Thefulllistcanbefoundinthe“HDF5PredefinedData‐
types”sectionoftheHDF5ReferenceManual.
Variable‐
length
Avariable‐length
1‐dimensional
arrayofdataele‐
ments
Currentsize,basetype
Compound ADatatypeofa
sequenceofData‐
types
Numberofmembers,
membernames,member
types,memberoffset,
memberclass,member
size,byteorder
Table6‐2.Architecturesusedinpredefineddatatypes
ArchitectureName Description
IEEE IEEE‐754standardfloatingpointtypesinvariousbyteorders.
STD Thisisanarchitecturethatcontainssemi‐standarddatatypeslike
signedtwo’scomplementintegers,unsignedintegers,andbitfieldsin
variousbyteorders.
C
FORTRAN
TypeswhicharespecifictotheCorFortranprogramminglanguages
aredefinedinthesearchitectures.Forinstance,H5T_C_S1definesa
basestringtypewithnullterminationwhichcanbeusedtoderive
stringtypesofotherlengths.
Table6‐1.Datatypeclassesandtheirproperties
Class Description Properties Notes
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 180
NATIVE ThisarchitecturecontainsC‐likedatatypesforthemachineonwhich
thelibrarywascompiled.Thetypeswereactuallydefinedbyrunning
theH5detectprogramwhenthelibrarywascompiled.Inordertobe
portable,applicationsshouldalmostalwaysusethisarchitectureto
describethingsinmemory.
CRAY Crayarchitectures.Theseareword‐addressable,big‐endiansystems
withnon‐IEEEfloatingpoint.
INTEL AllIntelandcompatibleCPU’sincluding80286,80386,80486,Pen‐
tium,Pentium‐Pro,andPentium‐II.Thesearelittle‐endiansystems
withIEEEfloating‐point.
MIPS AllMIPSCPU’scommonlyusedinSGIsystems.Thesearebig‐endian
systemswithIEEEfloating‐point.
ALPHA AllDECAlphaCPU’s,little‐endiansystemswithIEEEfloating‐point.
Table6‐3.Basetypes
B Bitfield
F Floatingpoint
I Signedinteger
R References
SCharacterstring
U Unsignedinteger
Table6‐4.Byteorder
BE Big‐endian
LE Little‐endian
Table6‐2.Architecturesusedinpredefineddatatypes
ArchitectureName Description
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 181
TheHDF5LibrarypredefinesasetofNATIVEdatatypeswhicharesimilartoCtypenames.Thenative
typesaresettobeanaliasfortheappropriateHDF5datatypeforeachplatform.Forexample,H5T_NA-
TIVE_INTcorrespondstoaCinttype.OnanIntelbasedPC,thistypeisthesameasH5T_STD_I32LE,
whileonaMIPSsystemthiswouldbeequivalenttoH5T_STD_I32BE.Table6showsexamplesofNATIVE
typesandcorrespondingCtypesforacommon32‐bitworkstation.
Table6‐5.Somepredefineddatatypes
Example Description
H5T_IEEE_F64LE Eight‐byte,little‐endian,IEEEfloating‐point
H5T_IEEE_F32BE Four‐byte,big‐endian,IEEEfloatingpoint
H5T_STD_I32LE Four‐byte,little‐endian,signedtwo’scomplementinteger
H5T_STD_U16BE Two‐byte,big‐endian,unsignedinteger
H5T_C_S1 One‐byte,null‐terminatedstringofeight‐bitcharacters
H5T_INTEL_B64 Eight‐bytebitfieldonanIntelCPU
H5T_CRAY_F64 Eight‐byteCrayfloatingpoint
H5T_STD_ROBJ Referencetoanentireobjectinafile
Table6‐6.Nativeand32‐bitCdatatypes
Example CorrespondingCType
H5T_NATIVE_CHAR char
H5T_NATIVE_SCHAR signedchar
H5T_NATIVE_UCHAR unsignedchar
H5T_NATIVE_SHORT short
H5T_NATIVE_USHORT unsignedshort
H5T_NATIVE_INT int
H5T_NATIVE_UINT unsigned
H5T_NATIVE_LONG long
H5T_NATIVE_ULONG unsignedlong
H5T_NATIVE_LLONG longlong
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 182
6.3.HowDatatypesareUsed
6.3.1.TheDatatypeObjectandtheHDF5DatatypeAPI
TheHDF5Librarymanagesdatatypesasobjects.TheHDF5datatypeAPImanipulatesthedatatypeobjects
throughCfunctioncalls.Newdatatypescanbecreatedfromscratchorcopiedfromexistingdatatypes.
WhenadatatypeisnolongerneededitsresourcesshouldbereleasedbycallingH5Tclose().
ThedatatypeobjectisusedinseveralrolesintheHDF5datamodelandlibrary.Essentially,adatatypeis
usedwhenevertheformatofdataelementsisneeded.TherearefourmajorusesofdatatypesintheHDF5
Library:atdatasetcreation,duringdatatransfers,whendiscoveringthecontentsofafile,andforspecify‐
inguser‐defineddatatypes.Seethetablebelow.
H5T_NATIVE_ULLONG unsignedlonglong
H5T_NATIVE_FLOAT float
H5T_NATIVE_DOUBLE double
H5T_NATIVE_LDOUBLE longdouble
H5T_NATIVE_HSIZE hsize_t
H5T_NATIVE_HSSIZE hssize_t
H5T_NATIVE_HERR herr_t
H5T_NATIVE_HBOOL hbool_t
H5T_NATIVE_B8 8‐bitunsignedintegeror8‐bitbufferinmemory
H5T_NATIVE_B16 16‐bitunsignedintegeror16‐bitbufferinmemory
H5T_NATIVE_B32 32‐bitunsignedintegeror32‐bitbufferinmemory
H5T_NATIVE_B64 64‐bitunsignedintegeror64‐bitbufferinmemory
Table6‐6.Nativeand32‐bitCdatatypes
Example CorrespondingCType
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 183
6.3.2.DatasetCreation
Allthedataelementsofadatasethavethesamedatatype.Whenadatasetiscreated,thedatatypeforthe
dataelementsmustbespecified.Thedatatypeofadatasetcanneverbechanged.Theexamplebelow
showstheuseofadatatypetocreateadatasetcalled“/dset”.Inthisexample,thedatasetwillbestored
as32‐bitsignedintegersinbig‐endianorder.
6.3.3.DataTransfer(ReadandWrite)
Probablythemostcommonuseofdatatypesistowriteorreaddatafromadatasetorattribute.Inthese
operations,eachdataelementistransferredfromthesourcetothedestination(possiblyrearrangingthe
orderoftheelements).Sincethesourceanddestinationdonotneedtobeidentical(inotherwords,one
isdiskandtheotherismemory),thetransferrequiresboththeformatofthesourceelementandthedes‐
tinationelement.Therefore,datatransfersusetwodatatypeobjects,forthesourceanddestination.
Table6‐7.Datatypeuses
Use Description
Datasetcreation Thedatatypeofthedataelementsmustbedeclared
whenthedatasetiscreated.
Datatransfer Thedatatype(format)ofthedataelementsmustbe
definedforboththesourceanddestination.
Discovery Thedatatypeofadatasetcanbeinterrogatedto
retrieveacompletedescriptionofthestoragelayout.
Creatinguser‐defineddatatypes Userscandefinetheirowndatatypesbycreating
datatypeobjectsandsettingtheirproperties.
hid_t dt;
dt = H5Tcopy(H5T_STD_I32BE);
dataset_id = H5Dcreate(file_id, “/dset”, dt, dataspace_id,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
CodeExample6‐1.Usingadatatypetocreateadataset
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 184
Whendataiswritten,thesourceismemoryandthedestinationisdisk(file).Thememorydatatype
describestheformatofthedataelementinthemachinememory,andthefiledatatypedescribesthe
desiredformatofthedataelementondisk.Similarly,whenreading,thesourcedatatypedescribesthe
formatofthedataelementondisk,andthedestinationdatatypedescribestheformatinmemory.
Inthemostcommoncases,thefiledatatypeisthedatatypespecifiedwhenthedatasetwascreated,and
thememorydatatypeshouldbetheappropriateNATIVEtype.
Theexamplesbelowshowsamplesofwritingdatatoandreadingdatafromadataset.Thedatainmem‐
oryisdeclaredCtype‘int’,andthedatatypeH5T_NATIVE_INTcorrespondstothistype.Thedatatypeof
thedatasetshouldbeofdatatypeclassH5T_INTEGER.
6.3.4.DiscoveryofDataFormat
TheHDF5Libraryenablesaprogramtodeterminethedatatypeclassandpropertiesforanydatatype.In
ordertodiscoverthestorageformatofdatainadataset,thedatatypeisobtained,andthepropertiesare
determinedbyqueriestothedatatypeobject.Theexamplebelowshowscodethatanalyzesthedatatype
foranintegerandprintsoutadescriptionofitsstorageproperties(byteorder,signed,size).
int dset_data[DATA_SIZE];
status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, dset_data);
CodeExample6‐2.Writingtoadataset
int dset_data[DATA_SIZE];
status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, dset_data);
CodeExample6‐3.Readingfromadataset
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 185
6.3.5.CreatingandUsingUser‐definedDatatypes
Mostprogramswillprimarilyusethepredefineddatatypesdescribedabove,possiblyincompositedata‐
typessuchascompoundorarraydatatypes.However,theHDF5datatypemodelisextremelygeneral;a
userprogramcandefineagreatvarietyofatomicdatatypes(storagelayouts).Inparticular,thedatatype
propertiescandefinesignedandunsignedintegersofanysizeandbyteorder,andfloatingpointnumbers
withdifferentformats,size,andbyteorder.TheHDF5datatypeAPIprovidesmethodstosettheseproper‐
ties.
switch (H5Tget_class(type)) {
case H5T_INTEGER:
ord = H5Tget_order(type);
sgn = H5Tget_sign(type);
printf(“Integer ByteOrder= ”);
switch (ord) {
case H5T_ORDER_LE:
printf(“LE”);
break;
case H5T_ORDER_BE:
printf(“BE”);
break;
}
printf(“ Sign= ”);
switch (sgn) {
case H5T_SGN_NONE:
printf(“false”);
break;
case H5T_SGN_2:
printf(“true”);
break;
}
printf(“ Size= ”);
sz = H5Tget_size(type);
printf(“%d”, sz);
printf(“\n”);
break;
CodeExample6‐4.Discoveringdatatypeproperties
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 186
User‐definedtypescanbeusedtodefinethelayoutofdatainmemory;examplesmightincludetomatch
someplatformspecificnumberformatorapplicationdefinedbit‐field.Theuser‐definedtypecanalso
describedatainthefilesuchasanapplication‐definedformat.Theuser‐definedtypescanbetranslatedto
andfromstandardtypesofthesameclass,asdescribedabove.
6.4.Datatype(H5T)FunctionSummaries
Functionsthatcanbeusedwithdatatypes(H5Tfunctions)andpropertylistfunctionsthatcanbeused
withdatatypes(H5Pfunctions)arelistedbelow.
FunctionListing6‐1.Generaldatatypeoperations
CFunction
FortranSubroutine
Purpose
H5Tcreate
h5tcreate_f
Createsanewdatatype.
H5Topen
h5topen_f
Opensacommitteddatatype.TheCfunction
isamacro:see“APICompatibilityMacrosin
HDF5.”
H5Tcommit
h5tcommit_f
Commitsatransientdatatypetoafile.The
datatypeisnowacommitteddatatype.TheC
functionisamacro:see“APICompatibility
MacrosinHDF5.”
H5Tcommit_anon
h5tcommit_anon_f
Commitsatransientdatatypetoafile.The
datatypeisnowacommitteddatatype,butit
isnotlinkedintothefilestructure.
H5Tcommitted
h5tcommitted_f
Determineswhetheradatatypeisacommit‐
tedoratransienttype.
H5Tcopy
h5tcopy_f
Copiesanexistingdatatype.
H5Tequal
h5tequal_f
Determineswhethertwodatatypeidentifiers
refertothesamedatatype.
H5Tlock
(noFortransubroutine)
Locksadatatype.
H5Tget_class
h5tget_class_f
Returnsthedatatypeclassidentifier.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 187
H5Tget_create_plist
h5tget_create_plist_f
Returnsacopyofadatatypecreationprop‐
ertylist.
H5Tget_size
h5tget_size_f
Returnsthesizeofadatatype.
H5Tget_super
h5tget_super_f
Returnsthebasedatatypefromwhichadata‐
typeisderived.
H5Tget_native_type
h5tget_native_type_f
Returnsthenativedatatypeofaspecified
datatype.
H5Tdetect_class
(noFortransubroutine)
Determineswhetheradatatypeisofthe
givendatatypeclass.
H5Tget_order
h5tget_order_f
Returnsthebyteorderofadatatype.
H5Tset_order
h5tset_order_f
Setsthebyteorderingofadatatype.
H5Tdecode
h5tdecode_f
Decodeabinaryobjectdescriptionofdata‐
typeandreturnanewobjectidentifier.
H5Tencode
h5tencode
Encodeadatatypeobjectdescriptionintoa
binarybuffer.
H5Tclose
h5tclose_f
Releasesadatatype.
FunctionListing6‐2.Conversionfunctions
CFunction
FortranSubroutine
Purpose
H5Tconvert
h5tconvert_f
Convertsdatabetweenspecifieddatatypes.
H5Tcompiler_conv
h5tcompiler_conv_f
Checkwhetherthelibrary’sdefaultconver‐
sionishardconversion.
H5Tfind
(noFortransubroutine)
Findsaconversionfunction.
FunctionListing6‐1.Generaldatatypeoperations
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 188
H5Tregister
(noFortransubroutine)
Registersaconversionfunction.
H5Tunregister
(noFortransubroutine)
Removesaconversionfunctionfromallcon‐
versionpaths.
FunctionListing6‐3.Atomicdatatypeproperties
CFunction
FortranSubroutine
Purpose
H5Tset_size
h5tset_size_f
Setsthetotalsizeforanatomicdatatype.
H5Tget_precision
h5tget_precision_f
Returnstheprecisionofanatomicdatatype.
H5Tset_precision
h5tset_precision_f
Setstheprecisionofanatomicdatatype.
H5Tget_offset
h5tget_offset_f
Retrievesthebitoffsetofthefirstsignificant
bit.
H5Tset_offset
h5tset_offset_f
Setsthebitoffsetofthefirstsignificantbit.
H5Tget_pad
h5tget_pad_f
Retrievesthepaddingtypeoftheleastand
most‐significantbitpadding.
H5Tset_pad
h5tset_pad_f
Setstheleastandmost‐significantbitspad‐
dingtypes.
H5Tget_sign
h5tget_sign_f
Retrievesthesigntypeforanintegertype.
H5Tset_sign
h5tset_sign_f
Setsthesignpropertyforanintegertype.
H5Tget_fields
h5tget_fields_f
Retrievesfloatingpointdatatypebitfield
information.
H5Tset_fields
h5tset_fields_f
Setslocationsandsizesoffloatingpointbit
fields.
H5Tget_ebias
h5tget_ebias_f
Retrievestheexponentbiasofafloating‐
pointtype.
FunctionListing6‐2.Conversionfunctions
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 189
H5Tset_ebias
h5tset_ebias_f
Setstheexponentbiasofafloating‐point
type.
H5Tget_norm
h5tget_norm_f
Retrievesmantissanormalizationofafloat‐
ing‐pointdatatype.
H5Tset_norm
h5tset_norm_f
Setsthemantissanormalizationofafloating‐
pointdatatype.
H5Tget_inpad
h5tget_inpad_f
Retrievestheinternalpaddingtypefor
unusedbitsinfloating‐pointdatatypes.
H5Tset_inpad
h5tset_inpad_f
Fillsunusedinternalfloatingpointbits.
H5Tget_cset
h5tget_cset_f
Retrievesthecharactersettypeofastring
datatype.
H5Tset_cset
h5tset_cset_f
Setscharactersettobeused.
H5Tget_strpad
h5tget_strpad_f
Retrievesthestoragemechanismforastring
datatype.
H5Tset_strpad
h5tset_strpad_f
Definesthestoragemechanismforcharacter
strings.
FunctionListing6‐4.Enumerationdatatypes
CFunction
FortranSubroutine
Purpose
H5Tenum_create
h5tenum_create_f
Createsanewenumerationdatatype.
H5Tenum_insert
h5tenum_insert_f
Insertsanewenumerationdatatypemember.
H5Tenum_nameof
h5tenum_nameof_f
Returnsthesymbolnamecorrespondingtoa
specifiedmemberofanenumerationdata‐
type.
H5Tenum_valueof
h5tenum_valueof_f
Returnsthevaluecorrespondingtoaspeci‐
fiedmemberofanenumerationdatatype.
FunctionListing6‐3.Atomicdatatypeproperties
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 190
H5Tget_member_value
h5tget_member_value_f
Returnsthevalueofanenumerationdata‐
typemember.
H5Tget_nmembers
h5tget_nmembers_f
Retrievesthenumberofelementsinacom‐
poundorenumerationdatatype.
H5Tget_member_name
h5tget_member_name_f
Retrievesthenameofacompoundorenu‐
merationdatatypemember.
H5Tget_member_index
(noFortransubroutine)
Retrievestheindexofacompoundorenu‐
merationdatatypemember.
FunctionListing6‐5.Compounddatatypeproperties
CFunction
FortranSubroutine
Purpose
H5Tget_nmembers
h5tget_nmembers_f
Retrievesthenumberofelementsinacom‐
poundorenumerationdatatype.
H5Tget_member_class
h5tget_member_class_f
Returnsdatatypeclassofcompounddatatype
member.
H5Tget_member_name
h5tget_member_name_f
Retrievesthenameofacompoundorenu‐
merationdatatypemember.
H5Tget_member_index
h5tget_member_index_f
Retrievestheindexofacompoundorenu‐
merationdatatypemember.
H5Tget_member_offset
h5tget_member_offset_f
Retrievestheoffsetofafieldofacompound
datatype.
H5Tget_member_type
h5tget_member_type_f
Returnsthedatatypeofthespecifiedmem‐
ber.
H5Tinsert
h5tinsert_f
Addsanewmembertoacompounddata‐
type.
H5Tpack
h5tpack_f
Recursivelyremovespaddingfromwithina
compounddatatype.
FunctionListing6‐4.Enumerationdatatypes
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 191
FunctionListing6‐6.Arraydatatypes
CFunction
FortranSubroutine
Purpose
H5Tarray_create
h5tarray_create_f
Createsanarraydatatypeobject.TheCfunc‐
tionisamacro:see“APICompatibilityMacros
inHDF5.”
H5Tget_array_ndims
h5tget_array_ndims_f
Returnstherankofanarraydatatype.
H5Tget_array_dims
h5tget_array_dims_f
Returnssizesofarraydimensionsanddimen‐
sionpermutations.TheCfunctionisamacro:
see“APICompatibilityMacrosinHDF5.”
FunctionListing6‐7.Vari ab le ‐lengthdatatypes
CFunction
FortranSubroutine
Purpose
H5Tvlen_create
h5tvlen_create_f
Createsanewvariable‐lengthdatatype.
H5Tis_variable_str
h5tis_variable_str_f
Determineswhetherdatatypeisavariable‐
lengthstring.
FunctionListing6‐8.Opaquedatatypes
CFunction
FortranSubroutine
Purpose
H5Tset_tag
h5tset_tag_f
Tagsanopaquedatatype.
H5Tget_tag
h5tget_tag_f
Getsthetagassociatedwithanopaquedata‐
type.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 192
6.5.ProgrammingModelforDatatypes
TheHDF5Libraryimplementsanobject‐orientedmodelofdatatypes.HDF5datatypesareorganizedasa
logicalsetofbasetypes,ordatatypeclasses.TheHDF5Librarymanagesdatatypesasobjects.TheHDF5
datatypeAPImanipulatesthedatatypeobjectsthroughCfunctioncalls.Thefigurebelowshowsthe
FunctionListing6‐9.Conversionsbetweendatatypeandtext
CFunction
FortranSubroutine
Purpose
H5LTtext_to_dtype
(noFortransubroutine)
Createsadatatypefromatextdescription.
H5LTdtype_to_text
(noFortransubroutine)
Generatesatextdescriptionofadatatype.
FunctionListing6‐10.Datatypecreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pset_char_encoding
h5pset_char_encoding_f
Setsthecharacterencodingusedtoencodea
string.UsetosetASCIIorUTF‐8character
encodingforobjectnames.
H5Pget_char_encoding
h5pget_char_encoding_f
Retrievesthecharacterencodingusedtocre‐
ateastring.
FunctionListing6‐11.Datatypeaccesspropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pset_type_conv_cb
(noFortransubroutine)
Setsuser‐defineddatatypeconversioncall‐
backfunction.
H5Pget_type_conv_cb
(noFortransubroutine)
Getsuser‐defineddatatypeconversioncall‐
backfunction.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 193
abstractviewofthedatatypeobject.Thetablebelowshowsthemethods(Cfunctions)thatoperateon
datatypeobjects.Newdatatypescanbecreatedfromscratchorcopiedfromexistingdatatypes.
Figure6‐5.Thedatatypeobject
Table6‐8.Generaloperationsondatatypeobjects
APIFunction Description
hid_t H5Tcreate (H5T_class_t class,
size_t size)
Createanewdatatypeobjectofdatatype
classclass.Thefollowingdatatype
classesaresupportedwiththisfunction:
• H5T_COMPOUND
• H5T_OPAQUE
• H5T_ENUM
Otherdatatypesarecreatedwith
H5Tcopy().
hid_t H5Tcopy (hid_t type)Obtainamodifiabletransientdatatype
whichisacopyoftype.Iftypeisadata‐
setidentifierthenthetypereturnedisa
modifiabletransientcopyofthedatatype
ofthespecifieddataset.
hid_t H5Topen (hid_t location, const
char *name, H5P_DEFAULT)
Openacommitteddatatype.Thecommit‐
teddatatypereturnedbythisfunctionis
read‐only.
htri_t H5Tequal (hid_t type1, hid_t
type2)
Determinesiftwotypesareequal.
herr_t H5Tclose (hid_t type)Releasesresourcesassociatedwithadata‐
typeobtainedfromH5Tcopy,H5Topen,
orH5Tcreate.Itisillegaltoclosean
immutabletransientdatatype(forexam‐
ple,predefinedtypes).
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 194
Inordertouseadatatype,theobjectmustbecreated(H5Tcreate),orareferenceobtainedbycloning
fromanexistingtype(H5Tcopy),oropened(H5Topen).Inaddition,areferencetothedatatypeofadata‐
setorattributecanbeobtainedwithH5Dget_typeorH5Aget_type.Forcompositedatatypesarefer‐
encetothedatatypeformembersorbasetypescanbeobtained(H5Tget_member_type,
H5Tget_super).Whenthedatatypeobjectisnolongerneeded,thereferenceisdiscardedwith
H5Tclose.
TwodatatypeobjectscanbetestedtoseeiftheyarethesamewithH5Tequal.Thisfunctionreturnstrue
ifthetwodatatypereferencesrefertothesamedatatypeobject.However,iftwodatatypeobjectsdefine
equivalentdatatypes(thesamedatatypeclassanddatatypeproperties),theywillnotbeconsidered
‘equal’.
Adatatypecanbewrittentothefileasafirstclassobject(H5Tcommit).Thisisacommitteddatatypeand
canbeusedinthesamewayasanyotherdatatype.
6.5.1.DiscoveryofDatatypeProperties
AnyHDF5datatypeobjectcanbequeriedtodiscoverallofitsdatatypeproperties.Foreachdatatype
class,thereareasetofAPIfunctionstoretrievethedatatypepropertiesforthisclass.
6.5.1.1.PropertiesofAtomicDatatypes
Table9liststhefunctionstodiscoverthepropertiesofatomicdatatypes.Table10liststhequeriesrele‐
vanttospecificnumerictypes.Table11givesthepropertiesforatomicstringdatatype,andTable12gives
thepropertyoftheopaquedatatype.
herr_t H5Tcommit (hid_t location,
const char *name, hid_t type, H5P_DE-
FAULT, H5P_DEFAULT, H5P_DEFAULT)
Commitatransientdatatype(not
immutable)toafiletobecomeacommit‐
teddatatype.Committeddatatypescan
beshared.
htri_t H5Tcommitted (hid_t type)Testwhetherthedatatypeistransientor
committed(named).
herr_t H5Tlock (hid_t type)Makeatransientdatatypeimmutable
(read‐onlyandnotclosable).Predefined
typesarelocked.
Table6‐8.Generaloperationsondatatypeobjects
APIFunction Description
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 195
Table6‐9.Functionstodiscoverpropertiesofatomicdatatypes
Functions Description
H5T_class_t H5Tget_class (hid_t type)Thedatatypeclass:H5T_INTEGER,
H5T_FLOAT,H5T_STRING,H5T_BIT-
FIELD,H5T_OPAQUE,H5T_COMPOUND,
H5T_REFERENCE,H5T_ENUM,H5T_VLEN,
H5T_ARRAY
size_t H5Tget_size (hid_t type)Thetotalsizeoftheelementinbytes,
includingpaddingwhichmayappearon
eithersideoftheactualvalue.
H5T_order_t H5Tget_order (hid_t type)Thebyteorderdescribeshowthebytesof
thedatatypearelaidoutinmemory.Ifthe
lowestmemoryaddresscontainstheleast
significantbyteofthedatumthenitis
saidtobelittle‐endianor
H5T_ORDER_LE.Ifthebytesareinthe
oppositeorderthentheyaresaidtobe
big‐endianorH5T_ORDER_BE.
size_t H5Tget_precision (hid_t type)Theprecisionpropertyidentifiesthe
numberofsignificantbitsofadatatype
andtheoffsetproperty(definedbelow)
identifiesitslocation.Somedatatypes
occupymorebytesthanwhatisneededto
storethevalue.Forinstance,ashorton
aCrayis32significantbitsinaneight‐byte
field.
int H5Tget_offset (hid_t type)Theoffsetpropertydefinesthebitloca‐
tionoftheleastsignificantbitofabitfield
whoselengthisprecision.
herr_t H5Tget_pad (hid_t type,
H5T_pad_t *lsb, H5T_pad_t *msb)
Paddingisthebitsofadataelement
whicharenotsignificantasdefinedbythe
precisionandoffsetproperties.Pad‐
dinginthelow‐numberedbitsislsbpad‐
dingandpaddinginthehigh‐numbered
bitsismsbpadding.Paddingbitscanbe
settozero(H5T_PAD_ZERO)orone
(H5T_PAD_ONE).
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 196
Table6‐10.Functionstodiscoverpropertiesofatomicnumericdatatypes
Functions Description
H5T_sign_t H5Tget_sign (hid_t type)(INTEGER)Integerdatacanbesigned
two’scomplement(H5T_SGN_2)or
unsigned(H5T_SGN_NONE).
herr_t H5Tget_fields (hid_t type,
size_t *spos, size_t *epos, size_t
*esize, size_t *mpos, size_t *msize)
(FLOAT)Afloating‐pointdataelementhas
bitfieldswhicharetheexponentand
mantissaaswellasamantissasignbit.
Thesepropertiesdefinethelocation(bit
positionofleastsignificantbitofthefield)
andsize(inbits)ofeachfield.Thesignbit
isalwaysoflengthoneandnoneofthe
fieldsareallowedtooverlap.
size_t H5Tget_ebias (hid_t type)(FLOAT)Theexponentisstoredasanon‐
negativevaluewhichisebiaslargerthan
thetrueexponent.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 197
H5T_norm_t H5Tget_norm (hid_t type)(FLOAT)Thispropertydescribesthenor‐
malizationmethodofthemantissa.
•H5T_NORM_MSBSET:themantissa
isshiftedleft(ifnon‐zero)until
thefirstbitaftertheradixpointis
setandtheexponentisadjusted
accordingly.Allbitsoftheman‐
tissaaftertheradixpointare
stored.
•H5T_NORM_IMPLIED:theman‐
tissaisshiftedleft\(ifnon‐zero)
untilthefirstbitaftertheradix
pointissetandtheexponentis
adjustedaccordingly.Thefirstbit
aftertheradixpointisnotstored
sinceit’salwaysset.
•H5T_NORM_NONE:thefractional
partofthemantissaisstored
withoutnormalizingit.
H5T_pad_t H5Tget_inpad (hid_t type)(FLOAT)Ifanyinternalbits(thatis,bits
betweenthesignbit,themantissafield,
andtheexponentfieldbutwithinthepre‐
cisionfield)areunused,thentheywillbe
filledaccordingtothevalueofthisprop‐
erty.Thepaddingcanbe:H5T_PAD_-
NONE,H5T_PAD_ZERO,or
H5T_PAD_ONE.
Table6‐10.Functionstodiscoverpropertiesofatomicnumericdatatypes
Functions Description
HDF5Datatypes HDF5User’sGuide
198 TheHDFGroup
6.5.1.2.PropertiesofCompositeDatatypes
Thecompositedatatypeclassescanalsobeanalyzedtodiscovertheirdatatypepropertiesandthedata‐
typesthataremembersorbasetypesofthecompositedatatype.Thememberorbasetypecan,inturn,
beanalyzed.Thetablebelowliststhefunctionsthatcanaccessthedatatypepropertiesofthedifferent
compositedatatypes.
Table6‐11.Functionstodiscoverpropertiesofatomicstringdatatypes
Functions Description
H5T_cset_t H5Tget_cset (hid_t type)Twocharactersetsarecurrentlysup‐
ported:ASCII(H5T_CSET_ASCII)and
UTF‐8(H5T_CSET_UTF8).
H5T_str_t H5Tget_strpad (hid_t type)Thestringdatatypehasafixedlength,but
thestringmaybeshorterthanthelength.
Thispropertydefinesthestoragemecha‐
nismfortheleftoverbytes.Theoptions
are:H5T_STR_NULLTERM,
H5T_STR_NULLPAD,or
H5T_STR_SPACEPAD.
Table6‐12.Functionstodiscoverpropertiesofatomicopaquedatatypes
Functions Description
char *H5Tget_tag(hid_t type_id) Auser‐definedstring.
Table6‐13.Functionstodiscoverpropertiesofcompositedatatypes
Functions Description
int H5Tget_nmembers(hid_t type_id) (COMPOUND)Thenumberoffieldsinthe
compounddatatype.
H5T_class_t H5Tget_member_class
(hid_t cdtype_id, unsigned member_no)
(COMPOUND)Thedatatypeclassofcom‐
pounddatatypemembermember_no.
char * H5Tget_member_name (hid_t
type_id, unsigned field_idx)
(COMPOUND)Thenameoffield
field_idxofacompounddatatype.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 199
6.5.2.DefinitionofDatatypes
TheHDF5Libraryenablesuserprogramstocreateandmodifydatatypes.Theessentialstepsare:
1. Createanewdatatypeobjectofaspecificcompositedatatypeclass,orcopyanexistingatomic
datatypeobject
2. Setpropertiesofthedatatypeobject
3. Usethedatatypeobject
4. Closethedatatypeobject
Tocreateauser‐definedatomicdatatype,theprocedureistocloneapredefineddatatypeoftheappropri‐
atedatatypeclass(H5Tcopy),andthensetthedatatypepropertiesappropriatetothedatatypeclass.The
tablebelowshowshowtocreateadatatypetodescribea1024‐bitunsignedinteger.
size_t H5Tget_member_offset (hid_t
type_id, unsigned memb_no)
(COMPOUND)Thebyteoffsetofthe
beginningofafieldwithinacompound
datatype.
hid_t H5Tget_member_type (hid_t
type_id, unsigned field_idx)
(COMPOUND)Thedatatypeofthespeci‐
fiedmember.
int H5Tget_array_ndims (hid_t
adtype_id)
(ARRAY)Thenumberofdimensions(rank)
ofthearraydatatypeobject.
int H5Tget_array_dims (hid_t
adtype_id, hsize_t *dims[])
(ARRAY)Thesizesofthedimensionsand
thedimensionpermutationsofthearray
datatypeobject.
hid_t H5Tget_super(hid_t type) (ARRAY,VL,ENUM)Thebasedatatype
fromwhichthedatatypetypeisderived.
herr_t H5Tenum_nameof(hid_t type void
*value, char *name, size_t size)
(ENUM)Thesymbolnamethatcorre‐
spondstothespecifiedvalueoftheenu‐
merationdatatype.
herr_t H5Tenum_valueof(hid_t type
char *name, void *value)
(ENUM)Thevaluethatcorrespondsto
thespecifiednameoftheenumeration
datatype.
herr_t H5Tget_member_value (hid_t
type unsigned memb_no, void *value)
(ENUM)Thevalueoftheenumeration
datatypemembermemb_no.
Table6‐13.Functionstodiscoverpropertiesofcompositedatatypes
Functions Description
HDF5Datatypes HDF5User’sGuide
200 TheHDFGroup
CompositedatatypesarecreatedwithaspecificAPIcallforeachdatatypeclass.Thetablebelowshows
thecreationmethodforeachdatatypeclass.Anewlycreateddatatypecannotbeuseduntilthedatatype
propertiesareset.Forexample,anewlycreatedcompounddatatypehasnomembersandcannotbe
used.
Oncethedatatypeiscreatedandthedatatypepropertiesset,thedatatypeobjectcanbeused.
Predefineddatatypesaredefinedbythelibraryduringinitializationusingthesamemechanismsas
describedhere.Eachpredefineddatatypeislocked(H5Tlock),sothatitcannotbechangedordestroyed.
User‐defineddatatypesmayalsobelockedusingH5Tlock.
6.5.2.1.User‐definedAtomicDatatypes
Table15summarizestheAPImethodsthatsetpropertiesofatomictypes.Table16showspropertiesspe‐
cifictonumerictypes,Table17showspropertiesspecifictothestringdatatypeclass.Notethatoffset,
pad,etc.donotapplytostrings.Table18showsthespecificpropertyoftheOPAQUEdatatypeclass.
hid_t new_type = H5Tcopy (H5T_NATIVE_INT);
H5Tset_precision(new_type, 1024);
H5Tset_sign(new_type, H5T_SGN_NONE);
CodeExample6‐5.Createanewdatatype
Table6‐14.Functionstocreateeachdatatypeclass
DatatypeClass FunctiontoCreate
COMPOUND H5Tcreate
OPAQUE H5Tcreate
ENUM H5Tenum_create
ARRAY H5Tarray_create
VL H5Tvlen_create
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 201
Table6‐15.APImethodsthatsetpropertiesofatomicdatatypes
Functions Description
herr_t H5Tset_size (hid_t type,
size_t size)
Setthetotalsizeoftheelementinbytes.
Thisincludespaddingwhichmayappear
oneithersideoftheactualvalue.Ifthis
propertyisresettoasmallervaluewhich
wouldcausethesignificantpartofthe
datatoextendbeyondtheedgeofthe
datatype,thentheoffsetpropertyisdec‐
rementedabitatatime.Iftheoffset
reacheszeroandthesignificantpartof
thedatastillextendsbeyondtheedgeof
thedatatypethentheprecisionproperty
isdecrementedabitatatime.Decreasing
thesizeofadatatypemayfailifthe
H5T_FLOATbitfieldswouldextend
beyondthesignificantpartofthetype.
herr_t H5Tset_order (hid_t type,
H5T_order_t order)
Setthebyteordertolittle‐endian
(H5T_ORDER_LE)orbig‐endian
(H5T_ORDER_BE).
herr_t H5Tset_precision (hid_t type,
size_t precision)
Setthenumberofsignificantbitsofa
datatype.Theoffsetproperty(defined
below)identifiesitslocation.Thesize
propertydefinedaboverepresentsthe
entiresize(inbytes)ofthedatatype.Ifthe
precisionisdecreasedthenpaddingbits
areinsertedontheMSBsideofthesignif‐
icantbits(thiswillfailforH5T_FLOAT
typesifitresultsinthesign,mantissa,or
exponentbitfieldextendingbeyondthe
edgeofthesignificantbitfield).Onthe
otherhand,iftheprecisionisincreasedso
thatit“hangsover”theedgeofthetotal
sizethentheoffsetpropertyisdecre‐
mentedabitatatime.Iftheoffset
reacheszeroandthesignificantbitsstill
hangovertheedge,thenthetotalsizeis
increasedabyteatatime.
HDF5Datatypes HDF5User’sGuide
202 TheHDFGroup
herr_t H5Tset_offset (hid_t type,
size_t offset)
Setthebitlocationoftheleastsignificant
bitofabitfieldwhoselengthispreci-
sion.Thebitsoftheentiredataarenum‐
beredbeginningatzeroattheleast
significantbitoftheleastsignificantbyte
(thebyteatthelowestmemoryaddress
foralittle‐endiantypeorthebyteatthe
highestaddressforabig‐endiantype).
Theoffsetpropertydefinesthebitloca‐
tionoftheleastsignificantbitofabitfield
whoselengthisprecision.Iftheoffsetis
increasedsothesignificantbits“hang
over”theedgeofthedatum,thenthe
sizepropertyisautomaticallyincre‐
mented.
herr_t H5Tset_pad (hid_t type,
H5T_pad_t lsb, H5T_pad_t msb)
Setthepaddingtozeros(H5T_PAD_ZERO)
orones(H5T_PAD_ONE).Paddingisthe
bitsofadataelementwhicharenotsig‐
nificantasdefinedbytheprecisionand
offsetproperties.Paddinginthelow‐
numberedbitsislsbpaddingandpad‐
dinginthehigh‐numberedbitsismsb
padding.
Table6‐16.APImethodsthatsetpropertiesofnumericdatatypes
Functions Description
herr_t H5Tset_sign (hid_t type,
H5T_sign_t sign)
(INTEGER)Integerdatacanbesigned
two’scomplement(H5T_SGN_2)or
unsigned(H5T_SGN_NONE).
herr_t H5Tset_fields (hid_t type,
size_t spos, size_t epos, size_t
esize, size_t mpos, size_t msize)
(FLOAT)Setthepropertiesdefinethe
location(bitpositionofleastsignificant
bitofthefield)andsize(inbits)ofeach
field.Thesignbitisalwaysoflengthone
andnoneofthefieldsareallowedto
overlap.
herr_t H5Tset_ebias (hid_t type,
size_t ebias)
(FLOAT)Theexponentisstoredasanon‐
negativevaluewhichisebiaslargerthan
thetrueexponent.
Table6‐15.APImethodsthatsetpropertiesofatomicdatatypes
Functions Description
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 203
herr_t H5Tset_norm (hid_t type,
H5T_norm_t norm)
(FLOAT)Thispropertydescribesthenor‐
malizationmethodofthemantissa.
•H5T_NORM_MSBSET:themantissa
isshiftedleft(ifnon‐zero)until
thefirstbitaftertheradixpointis
setandtheexponentisadjusted
accordingly.Allbitsoftheman‐
tissaaftertheradixpointare
stored.
•H5T_NORM_IMPLIED:theman‐
tissaisshiftedleft(ifnon‐zero)
untilthefirstbitaftertheradix
pointissetandtheexponentis
adjustedaccordingly.Thefirstbit
aftertheradixpointisnotstored
sinceitisalwaysset.
•H5T_NORM_NONE:thefractional
partofthemantissaisstored
withoutnormalizingit.
herr_t H5Tset_inpad (hid_t type,
H5T_pad_t inpad)
(FLOAT)Ifanyinternalbits(thatis,bits
betweenthesignbit,themantissafield,
andtheexponentfieldbutwithinthepre‐
cisionfield)areunused,thentheywillbe
filledaccordingtothevalueofthisprop‐
erty.Thepaddingcanbe:H5T_PAD_-
NONE,H5T_PAD_ZEROorH5T_PAD_ONE.
Table6‐17.APImethodsthatsetpropertiesofstringdatatypes
Functions Description
herr_t H5Tset_size (hid_t type,
size_t size)
Setthelengthofthestring,inbytes.The
precisionisautomaticallysetto8*size.
herr_t H5Tset_precision (hid_t type,
size_t precision)
Theprecisionmustbeamultipleof8.
Table6‐16.APImethodsthatsetpropertiesofnumericdatatypes
Functions Description
HDF5Datatypes HDF5User’sGuide
204 TheHDFGroup
Examples
Theexamplebelowshowshowtocreatea128‐bitlittle‐endiansignedintegertype.Increasingthepreci‐
sionofatypeautomaticallyincreasesthetotalsize.Notethattheproperprocedureistobeginfroma
typeoftheintendeddatatypeclasswhichinthiscaseisaNATIVE INT.
herr_t H5Tset_cset (hid_t type_id,
H5T_cset_t cset)
Twocharactersetsarecurrentlysup‐
ported:ASCII(H5T_CSET_ASCII)and
UTF‐8(H5T_CSET_UTF8).
herr_t H5Tset_strpad (hid_t type_id,
H5T_str_t strpad)
Thestringdatatypehasafixedlength,but
thestringmaybeshorterthanthelength.
Thispropertydefinesthestoragemecha‐
nismfortheleftoverbytes.Themethod
usedtostorecharacterstringsdifferswith
theprogramminglanguage:
•Cusuallynullterminatesstrings
•Fortranleft‐justifiesandspace‐
padsstrings
Validstringpaddingvalues,aspassedin
theparameterstrpad,areasfollows:
•H5T_STR_NULLTERM:Nulltermi‐
nate(asCdoes)
•H5T_STR_NULLPAD:Padwith
zeros
•H5T_STR_SPACEPAD:Padwith
spaces(asFORTRANdoes)
Table6‐18.APImethodsthatsetpropertiesofopaquedatatypes
Functions Description
herr_t H5Tset_tag (hid_t type_id
const char *tag)
Tagstheopaquedatatypetype_idwithan
ASCIIidentifiertag.
hid_t new_type = H5Tcopy (H5T_NATIVE_INT);
H5Tset_precision (new_type, 128);
H5Tset_order (new_type, H5T_ORDER_LE);
CodeExample6‐6.Createanew128‐bitlittle‐endiansignedintegerdatatype
Table6‐17.APImethodsthatsetpropertiesofstringdatatypes
Functions Description
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 205
Thefigurebelowshowsthestoragelayoutasthetypeisdefined.TheH5Tcopycreatesadatatypethatis
thesameasH5T_NATIVE_INT.Inthisexample,supposethisisa32‐bitbig‐endiannumber(Figurea).The
precisionissetto128bits,whichautomaticallyextendsthesizeto8bytes(Figureb).Finally,thebyte
orderissettolittle‐endian(Figurec).
Thesignificantbitsofadataelementcanbeoffsetfromthebeginningofthememoryforthatelementby
anamountofpadding.Theoffsetpropertyspecifiesthenumberofbitsofpaddingthatappeartothe
“rightof”thevalue.Thetableandfigurebelowshowhowa32‐bitunsignedintegerwith16‐bitsofpreci‐
sionhavingthevalue0x1122willbelaidoutinmemory.
Figure6‐6.Thestoragelayoutforanew128‐bitlittle‐endiansignedintegerdatatype
Table6‐19.MemoryLayoutfora32‐bitunsignedinteger
BytePosition Big‐Endian
Offset=0
Big‐Endian
Offset=16
Little‐Endian
Offset=0
Little‐Endian
Offset=16
0: [pad] [0x11] [0x22] [pad]
1: [pad] [0x22] [0x11] [pad]
2: [0x11] [pad] [pad] [0x22]
3: [0x22] [pad] [pad] [0x11]
HDF5Datatypes HDF5User’sGuide
206 TheHDFGroup
Iftheoffsetisincrementedthenthetotalsizeisincrementedalsoifnecessarytopreventsignificantbitsof
thevaluefromhangingovertheedgeofthedatatype.
Thebitsoftheentiredataarenumberedbeginningatzeroattheleastsignificantbitoftheleastsignifi‐
cantbyte(thebyteatthelowestmemoryaddressforalittle‐endiantypeorthebyteatthehighest
addressforabig‐endiantype).Theoffsetpropertydefinesthebitlocationoftheleastsignificantbitof
abitfieldwhoselengthisprecision.Iftheoffsetisincreasedsothesignificantbits“hangover”the
edgeofthedatum,thenthesizepropertyisautomaticallyincremented.
Toillustratethepropertiesoftheintegerdatatypeclass,theexamplebelowshowshowtocreateauser‐
defineddatatypethatdescribesa24‐bitsignedintegerthatstartsonthethirdbitofa32‐bitword.The
datatypeisspecializedfroma32‐bitinteger,theprecisionissetto24bits,andtheoffsetissetto3.
Figure6‐7.MemoryLayoutfora32‐bitunsignedinteger
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 207
Thefigurebelowshowsthestoragelayoutforadataelement.Notethattheunusedbitsintheoffsetwill
besettozeroandtheunusedbitsattheendwillbesettoone,asspecifiedintheH5Tset_padcall.
Toillustrateauser‐definedfloatingpointnumber,theexamplebelowshowshowtocreatea24‐bitfloat‐
ingpointnumberthatstarts5bitsintoa4byteword.Thefloatingpointnumberisdefinedtohaveaman‐
tissaof19bits(bits5‐23),anexponentof3bits(25‐27),andthesignbitisbit28.(Notethatthisisan
illustrationofwhatcanbedoneandisnotnecessarilyafloatingpointformatthatauserwouldrequire.)
hid_t dt;
dt = H5Tcopy(H5T_SDT_I32LE);
H5Tset_precision(dt, 24);
H5Tset_offset(dt,3);
H5Tset_pad(dt, H5T_PAD_ZERO, H5T_PAD_ONE);
CodeExample6‐7.Auser‐defineddatatypewitha24‐bitsignedinteger
Figure6‐8.Auser‐definedintegerdatatypewitharangeof‐1,048,583to1,048,584
hid_t dt;
dt = H5Tcopy(H5T_IEEE_F32LE);
H5Tset_precision(dt, 24);
H5Tset_fields (dt, 28, 25, 3, 5, 19);
H5Tset_pad(dt, H5T_PAD_ZERO, H5T_PAD_ONE);
H5Tset_inpad(dt, H5T_PAD_ZERO);
CodeExample6‐8.Auser‐defined24‐bitfloatingpointdatatype
HDF5Datatypes HDF5User’sGuide
208 TheHDFGroup
Thefigureaboveshowsthestoragelayoutofadataelementforthisdatatype.Notethatthereisan
unusedbit(24)betweenthemantissaandtheexponent.Thisbitisfilledwiththeinpadvaluewhichinthis
caseis0.
Thesignbitisalwaysoflengthoneandnoneofthefieldsareallowedtooverlap.Whenexpandingafloat‐
ing‐pointtypeoneshouldsettheprecisionfirst;whendecreasingthesizeoneshouldsetthefieldposi‐
tionsandsizesfirst.
6.5.2.2.CompositeDatatypes
Allcompositedatatypesmustbeuser‐defined;therearenopredefinedcompositedatatypes.
6.5.2.2.1.CompoundDatatypes
Thesubsectionsbelowdescribehowtocreateacompounddatatypeandhowtowriteandreaddataofa
compounddatatype.
DefiningCompoundDatatypes
CompounddatatypesareconceptuallysimilartoaCstructorFortranderivedtypes.Thecompounddata‐
typedefinesacontiguoussequenceofbytes,whichareformattedusingoneupto2^16datatypes(mem‐
bers).Acompounddatatypemayhaveanynumberofmembers,inanyorder,andthemembersmayhave
anydatatype,includingcompound.Thus,complexnestedcompounddatatypescanbecreated.Thetotal
sizeofthecompounddatatypeisgreaterthanorequaltothesumofthesizeofitsmembers,uptoamax‐
imumof2^32bytes.HDF5doesnotsupportdatatypeswithdistinguishedrecordsortheequivalentofC
unionsorFortranEQUIVALENCEstatements.
UsuallyaCstructorFortranderivedtypewillbedefinedtoholdadatapointinmemory,andtheoffsets
ofthemembersinmemorywillbetheoffsetsofthestructmembersfromthebeginningofaninstanceof
Figure6‐9.Auser‐definedfloatingpointdatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 209
thestruct.TheHDF5ClibraryprovidesamacroHOFFSET (s,m)tocalculatethemember’soffset.The
HDF5Fortranapplicationshavetocalculateoffsetsbyusingsizesofmembersdatatypesandbytakingin
considerationtheorderofmembersintheFortranderivedtype.
HOFFSET(s,m)
Thismacrocomputestheoffsetofmembermwithinastructs
offsetof(s,m)
Thismacrodefinedinstddef.hdoesexactlythesamethingastheHOFFSET()macro.
NoteforFortranusers:OffsetsofFortranstructurememberscorrespondtotheoffsetswithinapacked
datatype(seeexplanationbelow)storedinanHDF5file.
Eachmemberofacompounddatatypemusthaveadescriptivenamewhichisthekeyusedtouniquely
identifythememberwithinthecompounddatatype.AmembernameinanHDF5datatypedoesnotnec‐
essarilyhavetobethesameasthenameofthememberintheCstructorFortranderivedtype,although
thisisoftenthecase.NordoesoneneedtodefineallmembersoftheCstructorFortranderivedtypein
theHDF5compounddatatype(orviceversa).
Unlikeatomicdatatypeswhicharederivedfromotheratomicdatatypes,compounddatatypesarecreated
fromscratch.First,onecreatesanemptycompounddatatypeandspecifiesitstotalsize.Thenmembers
areaddedtothecompounddatatypeinanyorder.Eachmembertypeisinsertedatadesignatedoffset.
Eachmemberhasanamewhichisthekeyusedtouniquelyidentifythememberwithinthecompound
datatype.
TheexamplebelowshowsawayofcreatinganHDF5Ccompounddatatypetodescribeacomplexnum‐
ber.Thisisastructurewithtwocomponents,“real”and“imaginary”,andeachcomponentisadouble.An
equivalentCstructwhosetypeisdefinedbythecomplex_tstructisshown.
TheexamplebelowshowsawayofcreatinganHDF5Fortrancompounddatatypetodescribeacomplex
number.ThisisaFortranderivedtypewithtwocomponents,“real”and“imaginary”,andeachcomponent
isDOUBLEPRECISION.AnequivalentFortranTYPEwhosetypeisdefinedbytheTYPEcomplex_tis
shown.
typedef struct {
double re; /*real part*/
double im; /*imaginary part*/
} complex_t;
hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof (complex_t));
H5Tinsert (complex_id, “real”, HOFFSET(complex_t,re),
H5T_NATIVE_DOUBLE);
H5Tinsert (complex_id, “imaginary”, HOFFSET(complex_t,im),
H5T_NATIVE_DOUBLE);
CodeExample6‐9.AcompounddatatypeforcomplexnumbersinC
HDF5Datatypes HDF5User’sGuide
210 TheHDFGroup
ImportantNote:Thecompounddatatypeiscreatedwithasizesufficienttoholdallitsmembers.IntheC
exampleabove,thesizeoftheCstructandtheHOFFSETmacroareusedasaconvenientmechanismto
determinetheappropriatesizeandoffset.Alternatively,thesizeandoffsetcouldbemanuallydeter‐
mined:thesizecanbesetto16with“real”atoffset0and“imaginary”atoffset8.However,differentplat‐
formsandcompilershavedifferentsizesfor“double”andmayhavealignmentrestrictionswhichrequire
additionalpaddingwithinthestructure.ItismuchmoreportabletousetheHOFFSETmacrowhich
assuresthatthevalueswillbecorrectforanyplatform.
ThefigurebelowshowshowthecompounddatatypewouldbelaidoutassumingthatNATIVE_DOUBLE
are64‐bitnumbersandthattherearenoalignmentrequirements.Thetotalsizeofthecompounddata‐
typewillbe16bytes,the“real”componentwillstartatbyte0,and“imaginary”willstartatbyte8.
TYPE complex_t
DOUBLE PRECISION re ! real part
DOUBLE PRECISION im; ! imaginary part
END TYPE complex_t
CALL h5tget_size_f(H5T_NATIVE_DOUBLE, re_size, error)
CALL h5tget_size_f(H5T_NATIVE_DOUBLE, im_size, error)
complex_t_size = re_size + im_size
CALL h5tcreate_f(H5T_COMPOUND_F, complex_t_size, type_id)
offset = 0
CALL h5tinsert_f(type_id, “real”, offset, H5T_NATIVE_DOUBLE,
error)
offset = offset + re_size
CALL h5tinsert_f(type_id, “imaginary”, offset, H5T_NATIVE_DOUBLE,
error)
CodeExample6‐10.AcompounddatatypeforcomplexnumbersinFortran
Figure6‐10.Layoutofacompounddatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 211
ThemembersofacompounddatatypemaybeanyHDF5datatypeincludingthecompound,array,and
variable‐length(VL)types.Thefigureandexamplebelowshowthememorylayoutandcodewhichcre‐
atesacompounddatatypecomposedoftwocomplexvalues,andeachcomplexvalueisalsoacompound
datatypeasinthefigureabove.
Figure6‐11.Layoutofacompounddatatypenestedinacompounddatatype
typedef struct {
complex_t x;
complex_t y;
} surf_t;
hid_t complex_id, surf_id; /*hdf5 datatypes*/
CodeExample6‐11.Codeforacompounddatatypenestedinacompounddatatype
HDF5Datatypes HDF5User’sGuide
212 TheHDFGroup
Notethatasimilarresultcouldbeaccomplishedbycreatingacompounddatatypeandinsertingfour
fields.Seethefigurebelow.Thisresultsinthesamelayoutasthefigureabove.Thedifferencewouldbe
howthefieldsareaddressed.Inthefirstcase,therealpartof‘y’iscalled‘y.re’;inthesecondcaseitis‘y‐
re’.
Themembersofacompounddatatypedonotalwaysfillallthebytes.TheHOFFSETmacroassuresthat
thememberswillbelaidoutaccordingtotherequirementsoftheplatformandlanguage.Theexample
belowshowsanexampleofaCstructwhichrequiresextrabytesofpaddingonmanyplatforms.Thesec‐
ondelement,‘b’,isa1‐bytecharacterfollowedbyan8bytedouble,‘c’.Onmanysystems,the8‐bytevalue
mustbestoredona4‐or8‐byteboundary.Thisrequiresthestructtobelargerthanthesumofthesizeof
itselements.
complex_id = H5Tcreate (H5T_COMPOUND, sizeof(complex_t));
H5Tinsert (complex_id, “re”, HOFFSET(complex_t,re),
H5T_NATIVE_DOUBLE);
H5Tinsert (complex_id, “im”, HOFFSET(complex_t,im),
H5T_NATIVE_DOUBLE);
surf_id = H5Tcreate (H5T_COMPOUND, sizeof(surf_t));
H5Tinsert (surf_id, “x”, HOFFSET(surf_t,x), complex_id);
H5Tinsert (surf_id, “y”, HOFFSET(surf_t,y), complex_id);
typedef struct {
complex_t x;
complex_t y;
} surf_t;
hid_t surf_id = H5Tcreate (H5T_COMPOUND, sizeof(surf_t));
H5Tinsert (surf_id, “x-re”, HOFFSET(surf_t,x.re),
H5T_NATIVE_DOUBLE);
H5Tinsert (surf_id, “x-im”, HOFFSET(surf_t,x.im),
H5T_NATIVE_DOUBLE);
H5Tinsert (surf_id, “y-re”, HOFFSET(surf_t,y.re),
H5T_NATIVE_DOUBLE);
H5Tinsert (surf_id, “y-im”, HOFFSET(surf_t,y.im),
H5T_NATIVE_DOUBLE);
CodeExample6‐12.Anothercompounddatatypenestedinacompounddatatype
CodeExample6‐11.Codeforacompounddatatypenestedinacompounddatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 213
Intheexamplebelow,sizeofandHOFFSETareusedtoassurethatthemembersareinsertedatthecor‐
rectoffsettomatchthememoryconventionsoftheplatform.Thefigurebelowshowshowthisdataele‐
mentwouldbestoredinmemory,assumingthedoublemuststartona4‐byteboundary.Noticetheextra
bytesbetween‘b’and‘c’.
However,datastoredondiskdoesnotrequirealignment,sounalignedversionsofcompounddatastruc‐
turescanbecreatedtoimprovespaceefficiencyondisk.Theseunalignedcompounddatatypescanbe
createdbycomputingoffsetsbyhandtoeliminateinter‐memberpadding,orthememberscanbepacked
bycallingH5Tpack(whichmodifiesadatatypedirectly,soitisusuallyprecededbyacalltoH5Tcopy).
Theexamplebelowshowshowtocreateadiskversionofthecompounddatatypefromthefigureabove
inordertostoredataondiskinascompactaformaspossible.Packedcompounddatatypesshouldgener‐
allynotbeusedtodescribememoryastheymayviolatealignmentconstraintsforthearchitecturebeing
used.Notealsothatusingapackeddatatypefordiskstoragemayinvolveahigherdataconversioncost.
typedef struct s1_t {
int a;
char b;
double c;
} s1_t;
s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a), H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b), H5T_NATIVE_CHAR);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c), H5T_NATIVE_DOUBLE);
CodeExample6‐13.Acompounddatatypethatrequirespadding
Figure6‐12.Memorylayoutofacompounddatatypethatrequirespadding
HDF5Datatypes HDF5User’sGuide
214 TheHDFGroup
TheexamplebelowshowsthesequenceofFortrancallstocreateapackedcompounddatatype.AnHDF5
Fortrancompounddatatypeneverdescribesacompounddatatypeinmemoryandcompounddatais
ALWAYSwrittenbyfieldsasdescribedinthenextsection.Thereforepackingisnotneededunlesstheoff‐
setofeachconsecutivememberisnotequaltothesumofthesizesofthepreviousmembers.
CreatingandWritingDatasetswithCompoundDatatypes
CreatingdatasetswithcompounddatatypesissimilartocreatingdatasetswithanyotherHDF5datatypes.
Butwritingandreadingmaybedifferentsincedatasetsthathavecompounddatatypescanbewrittenor
readbyafield(member)orsubsetsoffields(members).Thecompounddatatypeistheonlycomposite
datatypethatsupports“sub‐setting”bytheelementsthedatatypeisbuiltfrom.
TheexamplebelowshowsaCexampleofcreatingandwritingadatasetwithacompounddatatype.
hid_t s2_tid = H5Tcopy (s1_tid);
H5Tpack (s2_tid);
CodeExample6‐14.CreateapackedcompounddatatypeinC
CALL h5tcopy_f(s1_id, s2_id, error)
CALL h5tpack_f(s2_id, error)
CodeExample6‐15.CreateapackedcompounddatatypeinFortran
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 215
Theexamplebelowshowsthecontentofthefilewrittenonalittle‐endianmachine.
typedef struct s1_t {
int a;
float b;
double c;
} s1_t;
s1_t data[LENGTH];
/* Initialize data */
for (i = 0; i < LENGTH; i++) {
data[i].a = i;
data[i].b = i*i;
data[i].c = 1./(i+1);
...
s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b),
H5T_NATIVE_FLOAT);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c),
H5T_NATIVE_DOUBLE);
...
dataset_id = H5Dcreate(file_id, “SDScompound.h5”, s1_t,
space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
H5Dwrite (dataset_id, s1_tid, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
CodeExample6‐16.CreateandwriteadatasetwithacompounddatatypeinC
HDF5Datatypes HDF5User’sGuide
216 TheHDFGroup
Itisnotnecessarytowritethewholedataatonce.Datasetswithcompounddatatypescanbewrittenby
fieldorbysubsetsoffields.Inordertodothisonehastoremembertosetthetransferpropertyofthe
datasetusingtheH5Pset_preservecallandtodefinethememorydatatypethatcorrespondstoafield.
Theexamplebelowshowshowfloatanddoublefieldsarewrittentothedataset.
HDF5 “SDScompound.h5” {
GROUP “/” {
DATASET “ArrayOfStructures” {
DATATYPE H5T_COMPOUND {
H5T_STD_I32LE “a_name”;
H5T_IEEE_F32LE “b_name”;
H5T_IEEE_F64LE “c_name”;
}
DATASPACE SIMPLE { ( 3 ) / ( 3 ) }
DATA {
(0): {
0,
0,
1
},
(1): {
1,
1,
0.5
},
(2): {
2,
4,
0.333333
}
}
}
}
}
CodeExample6‐17.Createandwritealittle‐endiandatasetwithacompounddatatypeinC
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 217
Thefigurebelowshowsthecontentofthefilewrittenonalittle‐endianmachine.Onlyfloatanddouble
fieldsarewritten.Thedefaultfillvalueisusedtoinitializetheunwrittenintegerfield.
typedef struct sb_t {
float b;
double c;
} sb_t;
typedef struct sc_t {
float b;
double c;
} sc_t;
sb_t data1[LENGTH];
sc_t data2[LENGTH];
/* Initialize data */
for (i = 0; i < LENGTH; i++) {
data1.b = i*i;
data2.c = 1./(i+1);
}
...
/* Create dataset as in example 15 */
...
/* Create memory datatypes corresponding to float */
/* and double datatype fields */
sb_tid = H5Tcreate (H5T_COMPOUND, sizeof(sb_t));
H5Tinsert(sb_tid, “b_name”, HOFFSET(sb_t, b),
H5T_NATIVE_FLOAT);
sc_tid = H5Tcreate (H5T_COMPOUND, sizeof(sc_t));
H5Tinsert(sc_tid, “c_name”, HOFFSET(sc_t, c),
H5T_NATIVE_DOUBLE);
...
/* Set transfer property */
xfer_id = H5Pcreate(H5P_DATASET_XFER);
H5Pset_preserve(xfer_id, 1);
H5Dwrite (dataset_id, sb_tid, H5S_ALL, H5S_ALL,
xfer_id, data1);
H5Dwrite (dataset_id, sc_tid, H5S_ALL, H5S_ALL,
xfer_id, data2);
CodeExample6‐18.Writingfloatsanddoublestoadataset
HDF5Datatypes HDF5User’sGuide
218 TheHDFGroup
TheexamplebelowcontainsaFortranexamplethatcreatesandwritesadatasetwithacompounddata‐
type.Asthisexampleillustrates,writingandreadingcompounddatatypesinFortranisalwaysdoneby
fields.Thecontentofthewrittenfileisthesameasshownintheexampleabove.
HDF5 “SDScompound.h5” {
GROUP “/” {
DATASET “ArrayOfStructures” {
DATATYPE H5T_COMPOUND {
H5T_STD_I32LE “a_name”;
H5T_IEEE_F32LE “b_name”;
H5T_IEEE_F64LE “c_name”;
}
DATASPACE SIMPLE { ( 3 ) / ( 3 ) }
DATA {
(0): {
0,
0,
1
},
(1): {
0,
1,
0.5
},
(2): {
0,
4,
0.333333
}
}
}
}
}
CodeExample6‐19.Writingfloatsanddoublestoadatasetonalittle‐endiansystem
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 219
! One cannot write an array of a derived datatype in
! Fortran.
TYPE s1_t
INTEGER a
REAL b
DOUBLE PRECISION c
END TYPE s1_t
TYPE(s1_t) d(LENGTH)
! Therefore, the following code initializes an array
! corresponding to each field in the derived datatype
! and writes those arrays to the dataset
INTEGER, DIMENSION(LENGTH) :: a
REAL, DIMENSION(LENGTH) :: b
DOUBLE PRECISION, DIMENSION(LENGTH) :: c
! Initialize data
do i = 1, LENGTH
a(i) = i-1
b(i) = (i-1) * (i-1)
c(i) = 1./i
enddo
...
! Set dataset transfer property to preserve partially
! initialized fields during write/read to/from dataset
! with compound datatype.
!
CALL h5pcreate_f(H5P_DATASET_XFER_F, plist_id, error)
CALL h5pset_preserve_f(plist_id, .TRUE., error)
...
!
! Create compound datatype.
!
! First calculate total size by calculating sizes of
! each member
!
CALL h5tget_size_f(H5T_NATIVE_INTEGER, type_sizei, error)
CALL h5tget_size_f(H5T_NATIVE_REAL, type_sizer, error)
CALL h5tget_size_f(H5T_NATIVE_DOUBLE, type_sized, error)
type_size = type_sizei + type_sizer + type_sized
CALL h5tcreate_f(H5T_COMPOUND_F, type_size, dtype_id, error)
CodeExample6‐20.CreateandwriteadatasetwithacompounddatatypeinFortran
HDF5Datatypes HDF5User’sGuide
220 TheHDFGroup
!
! Insert members
!
!
! INTEGER member
!
offset = 0
CALL h5tinsert_f(dtype_id, “a_name”, offset,
H5T_NATIVE_INTEGER, error)
!
! REAL member
!
offset = offset + type_sizei
CALL h5tinsert_f(dtype_id, “b_name”, offset, H5T_NATIVE_REAL,
error)
!
! DOUBLE PRECISION member
!
offset = offset + type_sizer
CALL h5tinsert_f(dtype_id, “c_name”, offset,
H5T_NATIVE_DOUBLE, error)
!
! Create the dataset with compound datatype.
!
CALL h5dcreate_f(file_id, dsetname, dtype_id, dspace_id, &
dset_id, error, H5P_DEFAULT_F, H5P_DEFAULT_F,
H5P_DEFAULT_F)
!
...
! Create memory types. We have to create a compound
! datatype for each member we want to write.
!
CALL h5tcreate_f(H5T_COMPOUND_F, type_sizei, dt1_id, error)
offset = 0
CALL h5tinsert_f(dt1_id, “a_name”, offset,
H5T_NATIVE_INTEGER, error)
!
CALL h5tcreate_f(H5T_COMPOUND_F, type_sizer, dt2_id, error)
offset = 0
CodeExample6‐20.CreateandwriteadatasetwithacompounddatatypeinFortran
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 221
ReadingDatasetswithCompoundDatatypes
Readingdatasetswithcompounddatatypesmaybeachallenge.Forgeneralapplicationsthereisnoway
toknowapriorithecorrespondingCstructure.Also,Cstructurescannotbeallocatedontheflyduringdis‐
coveryofthedataset’sdatatype.ForgeneralC,C++,FortranandJavaapplicationthefollowingstepswill
berequiredtoreadandtointerpretdatafromthedatasetwithcompounddatatype:
1. GettheidentifierofthecompounddatatypeinthefilewiththeH5Dget_typecall
2. FindthenumberofthecompounddatatypememberswiththeH5Tget_nmemberscall
3. Iteratethroughcompounddatatypemembers
•GetmemberclasswiththeH5Tget_member_classcall
•GetmembernamewiththeH5Tget_member_namecall
•Checkclasstypeagainstpredefinedclasses
• H5T_INTEGER
• H5T_FLOAT
• H5T_STRING
• H5T_BITFIELD
• H5T_OPAQUE
• H5T_COMPOUND
• H5T_REFERENCE
• H5T_ENUM
• H5T_VLEN
• H5T_ARRAY
CALL h5tinsert_f(dt2_id, “b_name”, offset, H5T_NATIVE_REAL,
error)
!
CALL h5tcreate_f(H5T_COMPOUND_F, type_sized, dt3_id, error)
offset = 0
CALL h5tinsert_f(dt3_id, “c_name”, offset, H5T_NATIVE_DOUBLE,
error)
!
! Write data by fields in the datatype. Fields order
! is not important.
!
CALL h5dwrite_f(dset_id, dt3_id, c, data_dims, error,
xfer_prp = plist_id)
CALL h5dwrite_f(dset_id, dt2_id, b, data_dims, error,
xfer_prp = plist_id)
CALL h5dwrite_f(dset_id, dt1_id, a, data_dims, error,
xfer_prp = plist_id)
CodeExample6‐20.CreateandwriteadatasetwithacompounddatatypeinFortran
HDF5Datatypes HDF5User’sGuide
222 TheHDFGroup
•IfclassisH5T_COMPOUND,thengotostep2andrepeatallstepsunderstep3.Ifclassisnot
H5T_COMPOUND,thenamemberisofanatomicclassandcanbereadtoacorrespondingbuf‐
ferafterdiscoveringallnecessaryinformationspecifictoeachatomictype(forexample,size
oftheintegerorfloats,superclassforenumeratedandarraydatatype,anditssizes)
Theexamplesbelowshowhowtoreadadatasetwithaknowncompounddatatype.
Thefirstexamplebelowshowsthestepsneededtoreaddataofaknownstructure.First,buildamemory
datatypethesamewayitwasbuiltwhenthedatasetwascreated,andthensecondusethedatatypeina
H5Dreadcall.
Insteadofbuildingamemorydatatype,theapplicationcouldusetheH5Tget_native_typefunction.
Seetheexamplebelow.
typedef struct s1_t {
int a;
float b;
double c;
} s1_t;
s1_t *data;
...
s1_tid = H5Tcreate(H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b),
H5T_NATIVE_FLOAT);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c),
H5T_NATIVE_DOUBLE);
...
dataset_id = H5Dopen(file_id, “SDScompound.h5”,
H5P_DEFAULT);
...
data = (s1_t *) malloc (sizeof(s1_t)*LENGTH);
H5Dread(dataset_id, s1_tid, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
CodeExample6‐21.Readadatasetusingamemorydatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 223
Theexamplebelowshowshowtoreadjustonefloatmemberofacompounddatatype.
typedef struct s1_t {
int a;
float b;
double c;
} s1_t;
s1_t *data;
hid_t file_s1_t, mem_s1_t;
...
dataset_id = H5Dopen(file_id, “SDScompound.h5”,
H5P_DEFAULT);
/* Discover datatype in the file */
file_s1_t = H5Dget_type(dataset_id);
/* Find corresponding memory datatype */
mem_s1_t = H5Tget_native_type(file_s1_t,
H5T_DIR_DEFAULT);
...
data = (s1_t *) malloc (sizeof(s1_t)*LENGTH);
H5Dread (dataset_id, mem_s1_tid, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
CodeExample6‐22.ReadadatasetusingH5Tget_native_type
HDF5Datatypes HDF5User’sGuide
224 TheHDFGroup
Theexamplebelowshowshowtoreadfloatanddoublemembersofacompounddatatypeintoastruc‐
turethathasthosefieldsinadifferentorder.PleasenoticethatH5Tinsertcallscanbeusedinanorder
differentfromtheorderofthestructure’smembers.
typedef struct s1_t {
float b;
} sf_t;
sf_t *data;
...
sf_tid = H5Tcreate(H5T_COMPOUND, sizeof(sf_t));
H5Tinsert(s1_tid, “b_name”, HOFFSET(sf_t, b),
H5T_NATIVE_FLOAT);
...
dataset_id = H5Dopen(file_id, “SDScompound.h5”,
H5P_DEFAULT);
...
data = (sf_t *) malloc (sizeof(sf_t)*LENGTH);
H5Dread(dataset_id, sf_tid, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
CodeExample6‐23.Readonefloatingpointmemberofacompounddatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 225
6.5.2.2.2.Array
Manyscientificdatasetshavemultiplemeasurementsforeachpointinaspace.Thereareseveralnatural
waystorepresentthisdata,dependingonthevariablesandhowtheyareusedincomputation.Seethe
tableandthefigurebelow.
typedef struct s1_t {
double c;
float b;
} sdf_t;
sdf_t *data;
...
sdf_tid = H5Tcreate(H5T_COMPOUND, sizeof(sdf_t));
H5Tinsert(sdf_tid, “b_name”, HOFFSET(sdf_t, b),
H5T_NATIVE_FLOAT);
H5Tinsert(sdf_tid, “c_name”, HOFFSET(sdf_t, c),
H5T_NATIVE_DOUBLE);
...
dataset_id = H5Dopen(file_id, “SDScompound.h5”,
H5P_DEFAULT);
...
data = (sdf_t *) malloc (sizeof(sdf_t)*LENGTH);
H5Dread(dataset_id, sdf_tid, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
CodeExample6‐24.Readfloatanddoublemembersofacompounddatatype
Table6‐20.Representingdatawithmultiplemeasurements
StorageStrategy Storedas Remarks
Multipleplanes Severaldatasetswith
identicaldataspaces
Thisisoptimalwhenvariablesareaccessed
individually,orwhenoftenusesonlyselected
variables.
Additionaldimen‐
sion
Onedataset,thelast
“dimension”isavec‐
torofvariables
Thiscangivegoodperformance,although
selectingonlyafewvariablesmaybeslow.
Thismaynotreflectthescience.
HDF5Datatypes HDF5User’sGuide
226 TheHDFGroup
TheHDF5H5T_ARRAYdatatypedefinesthedataelementtobeahomogeneous,multi‐dimensionalarray.
SeeFigure13dabove.TheelementsofthearraycanbeanyHDF5datatype(includingcompoundand
array),andthesizeofthedatatypeisthetotalsizeofthearray.Adatasetofarraydatatypecannotbesub‐
dividedforI/Owithinthedataelement:theentirearrayofthedataelementmustbetransferred.Ifthe
dataelementsneedtobeaccessedseparately,forexample,byplane,thenthearraydatatypeshouldnot
beused.Thetablebelowshowsadvantagesanddisadvantagesofvariousstoragemethods.
Recordwithmulti‐
plevalues
Onedatasetwith
compounddatatype
Thisenablesthevariablestobereadall
togetherorselected.Alsohandles“vectors”of
heterogeneousdata.
VectororTensor
value
Onedataset,each
dataelementisa
smallarrayofvalues.
Thisusesthesameamountofspaceasthe
previoustwo,andmayrepresentthescience
modelbetter.
Figure6‐13.Representingdatawithmultiplemeasurements
Table6‐20.Representingdatawithmultiplemeasurements
StorageStrategy Storedas Remarks
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 227
Anarraydatatypemaybemulti‐dimensionalwith1toH5S_MAX_RANK(themaximumrankofadatasetis
currently32)dimensions.Thedimensionscanbeanysizegreaterthan0,butunlimiteddimensionsare
notsupported(althoughthedatatypecanbeavariable‐lengthdatatype).
AnarraydatatypeiscreatedwiththeH5Tarray_createcall,whichspecifiesthenumberofdimensions,
thesizeofeachdimension,andthebasetypeofthearray.Thearraydatatypecanthenbeusedinanyway
thatanydatatypeobjectisused.Theexamplebelowshowsthecreationofadatatypethatisatwo‐
dimensionalarrayofnativeintegers,andthisisthenusedtocreateadataset.Notethatthedatasetcanbe
adataspacethatisanynumberandsizeofdimensions.Thefigurebelowshowsthelayoutinmemory
assumingthatthenativeintegersare4bytes.Eachdataelementhas6elements,foratotalof24bytes.
Table6‐21.Storagemethodadvantagesanddisadvantages
Method Advantages Disadvantages
a)MultipleDatasets Easytoaccesseachplane,can
selectanyplane(s)
Lessefficienttoaccessa‘col‐
umn’throughtheplanes
b)N+1Dimension Allaccesspatternssupported Mustbehomogeneousdata‐
type
Theaddeddimensionmaynot
makesenseinthescientific
model
c)CompoundDatatype Canbeheterogeneousdatatype Planesmustbenamed,selec‐
tionisbyplane
Notanaturalrepresentationfor
amatrix
d)Array Anaturalrepresentationfor
vectorortensordata
Cannotaccesselementssepa‐
rately(noaccessbyplane)
hid_t file, dataset;
hid_t datatype, dataspace;
hsize_t adims[] = {3, 2};
datatype = H5Tarray_create(H5T_NATIVE_INT, 2, adims,
NULL);
dataset = H5Dcreate(file, datasetname, datatype,
dataspace, H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
CodeExample6‐25.Createatwo‐dimensionalarraydatatype
HDF5Datatypes HDF5User’sGuide
228 TheHDFGroup
6.5.2.2.3.Variable‐lengthDatatypes
Avariable‐length(VL)datatypeisaone‐dimensionalsequenceofadatatypewhicharenotfixedinlength
fromonedatasetlocationtoanother.Inotherwords,eachdataelementmayhaveadifferentnumberof
members.Variable‐lengthdatatypescannotbedivided,theentiredataelementmustbetransferred.
VLdatatypesareusefultothescientificcommunityinmanydifferentways,possiblyincluding:
•Raggedarrays:Multi‐dimensionalraggedarrayscanbeimplementedwiththelast(fastestchang‐
ing)dimensionbeingraggedbyusingaVLdatatypeasthetypeoftheelementstored.
•Fractalarrays:AnestedVLdatatypecanbeusedtoimplementraggedarraysofraggedarrays,to
whatevernestingdepthisrequiredfortheuser.
•Polygonlists:Acommonstoragerequirementistoefficientlystorearraysofpolygonswithdiffer‐
entnumbersofvertices.AVLdatatypecanbeusedtoefficientlyandsuccinctlydescribeanarray
ofpolygonswithdifferentnumbersofvertices.
•Characterstrings:PerhapsthemostcommonuseofVLdatatypeswillbetostoreC‐likeVLcharac‐
terstringsindatasetelementsorasattributesofobjects.
Figure6‐14.Memorylayoutofatwo‐dimensionalarraydatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 229
•Indices(forexample,ofobjectswithinafile):AnarrayofVLobjectreferencescouldbeusedasan
indextoalltheobjectsinafilewhichcontainaparticularsequenceofdatasetvalues.
•ObjectTracking:AnarrayofVLdatasetregionreferencescanbeusedasamethodoftracking
objectsorfeaturesappearinginasequenceofdatasets.
AVLdatatypeiscreatedbycallingH5Tvlen_createwhichspecifiesthebasedatatype.Thefirstexample
belowshowsanexampleofcodethatcreatesaVLdatatypeofunsignedintegers.Eachdataelementisa
one‐dimensionalarrayofzeroormoremembersandisstoredinthehvl_tstructure.Seethesecond
examplebelow.
ThefirstexamplebelowshowshowtheVLdataiswritten.Foreachofthe10dataelements,alengthand
databuffermustbeallocated.Belowthetwoexamplesisafigurethatshowshowthedataislaidoutin
memory.
Ananalogousproceduremustbeusedtoreadthedata.Seethesecondexamplebelow.Anappropriate
arrayofvl_tmustbeallocated,andthedataread.Itisthentraversedonedataelementatatime.The
H5Dvlen_reclaimcallfreesthedatabufferforthebuffer.Witheachelementpossiblybeingofdifferent
sequencelengthsforadatasetwithaVLdatatype,thememoryfortheVLdatatypemustbedynamically
allocated.CurrentlytherearetwomethodsofmanagingthememoryforVLdatatypes:thestandardC
malloc/freememoryallocationroutinesoramethodofcallinguser‐definedmemorymanagementrou‐
tinestoallocateorfreememory(setwithH5Pset_vlen_mem_manager).Sincethememoryallocated
whenreading(orwriting)maybecomplicatedtorelease,theH5Dvlen_reclaimfunctionisprovidedto
traverseamemorybufferandfreetheVLdatatypeinformationwithoutleakingmemory.
tid1 = H5Tvlen_create (H5T_NATIVE_UINT);
dataset=H5Dcreate(fid1, “Dataset1”, tid1, sid1,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
CodeExample6‐26.Createavariable‐lengthdatatypeofunsignedintegers
typedef struct {
size_t len; /* Length of VL data */
/*(in base type units) */
void *p; /* Pointer to VL data */
} hvl_t;
CodeExample6‐27.DataelementstorageformembersoftheVLdatatype
HDF5Datatypes HDF5User’sGuide
230 TheHDFGroup
hvl_t wdata[10]; /* Information to write */
/* Allocate and initialize VL data to write */
for(i=0; i < 10; i++) {
wdata[i].p = malloc((i+1)*sizeof(unsigned int));
wdata[i].len = i+1;
for(j=0; j<(i+1); j++)
((unsigned int *)wdata[i].p)[j]=i*10+j;
}
ret=H5Dwrite(dataset, tid1, H5S_ALL, H5S_ALL, H5P_DEFAULT,
wdata);
CodeExample6‐28.WriteVLdata
hvl_t rdata[SPACE1_DIM1];
ret=H5Dread(dataset, tid1, H5S_ALL, H5S_ALL, xfer_pid, rdata);
for(i=0; i<SPACE1_DIM1; i++) {
printf(“%d: len %d ”,rdata[i].len);
for(j=0; j<rdata[i].len; j++) {
printf(“ value: %u\n”,((unsigned int *)rdata[i].p)[j]);
}
}
ret=H5Dvlen_reclaim(tid1, sid1, xfer_pid, rdata);
CodeExample6‐29.ReadVLdata
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 231
Theuserprogrammustcarefullymanagetheserelativelycomplexdatastructures.TheH5Dvlen_re-
claimfunctionperformsastandardtraversal,freeingallthedata.Thisfunctionanalyzesthedatatype
anddataspaceobjects,andvisitseachVLdataelement,recursingthroughnestedtypes.Bydefault,the
systemfreeiscalledforthepointerineachvl_t.Obviously,thiscallassumesthatallofthismemorywas
allocatedwiththesystemmalloc.
Theuserprogrammayspecifycustommemorymanagerroutines,oneforallocatingandoneforfreeing.
ThesemaybesetwiththeH5Pvlen_mem_manager,andmusthavethefollowingprototypes:
• typedef void *(*H5MM_allocate_t)(size_t size, void *info);
• typedef void (*H5MM_free_t)(void *mem, void *free_info);
Figure6‐15.MemorylayoutofaVLdatatype
HDF5Datatypes HDF5User’sGuide
232 TheHDFGroup
TheutilityfunctionH5Dget_vlen_buf_sizechecksthenumberofbytesrequiredtostoretheVLdata
fromthedataset.ThisfunctionanalyzesthedatatypeanddataspaceobjecttovisitalltheVLdataele‐
ments,todeterminethenumberofbytesrequiredtostorethedatafortheinthedestinationstorage
(memory).Thesizevalueisadjustedfordataconversionandalignmentinthedestination.
6.6.OtherNon‐numericDatatypes
Severaldatatypeclassesdefinespecialtypesofobjects.
6.6.1.Strings
Textdataisrepresentedbyarraysofcharacters,calledstrings.Manyprogramminglanguagessupportdif‐
ferentconventionsforstoringstrings,whichmaybefixedorvariable‐length,andmayhavedifferentrules
forpaddingunusedstorage.HDF5canrepresentstringsinseveralways.Seethefigurebelow.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 233
First,adatasetmayhaveadatasetwithdatatypeH5T_NATIVE_CHARwitheachcharacterofthestringas
anelementofthedataset.Thiswillstoreanunstructuredblockoftextdata,butgiveslittleindicationof
anystructureinthetext.Seeitemainthefigureabove.
AsecondalternativeistostorethedatausingthedatatypeclassH5T_STRINGwitheachelementafixed
length.Seeitembinthefigureabove.Inthisapproach,eachelementmightbeawordorasentence,
addressedbythedataspace.Thedatasetreservesspaceforthespecifiednumberofcharacters,although
somestringsmaybeshorter.Thisapproachissimpleandusuallyisfasttoaccess,butcanwastestorage
spaceifthelengthoftheStringsvaries.
Athirdalternativeistouseavariable‐lengthdatatype.Seeitemcinthefigureabove.Thiscanbedone
usingthestandardmechanismsdescribedabove.Theprogramwouldusevl_tstructurestowriteand
readthedata.
Thestringstostoreare“Fourscore”and“lazyprogrammers.”
a)H5T_NATIVE_CHAR:Thedatasetisaone‐dimensionalarraywith29elements,andeachelement
isasinglecharacter.
b)Fixed‐lengthstring:Thedatasetisaone‐dimensionalarraywithtwoelements,andeachelement
is20characters.
c)Variable‐lengthstring:Thedatasetisaone‐dimensionalarraywithtwoelements,andeachele‐
mentisavariable‐lengthstring.Thisisthesameresultwhenstoredasafixed‐lengthstringexcept
thatthefirstelementofthearraywillneedonly11bytesforstorageinsteadof20.
Figure6‐16.Astringstoredasone‐characterelementsinaone‐dimensionalarray
HDF5Datatypes HDF5User’sGuide
234 TheHDFGroup
Afourthalternativeistouseaspecialfeatureofthestringdatatypeclasstosetthesizeofthedatatypeto
H5T_VARIABLE.Seeitemcinthefigureabove.Theexamplebelowshowsadeclarationofadatatypeof
typeH5T_C_S1whichissettoH5T_VARIABLE.TheHDF5Libraryautomaticallytranslatesbetweenthis
andthevl_tstructure.Note:theH5T_VARIABLEsizecanonlybeusedwithstringdatatypes.
Variable‐lengthstringscanbereadintoCstrings(inotherwords,pointerstozeroterminatedarraysof
char).Seetheexamplebelow.
6.6.2.Reference
InHDF5,objects(groups,datasets,andcommitteddatatypes)areusuallyaccessedbyname.Thereis
anotherwaytoaccessstoredobjects‐byreference.Therearetworeferencedatatypes:objectreference
andregionreference.ObjectreferenceobjectsarecreatedwithH5Rcreateandothercalls(crossrefer‐
ence).Theseobjectscanbestoredandretrievedinadatasetaselementswithreferencedatatype.The
firstexamplebelowshowsanexampleofcodethatcreatesreferencestofourobjects,andthenwritesthe
arrayofobjectreferencestoadataset.Thesecondexamplebelowshowsadatasetofdatatypereference
beingreadandoneofthereferenceobjectsbeingdereferencedtoobtainanobjectpointer.
Inordertostorereferencestoregionsofadataset,thedatatypeshouldbeH5T_REGION_OBJ.Notethat
adataelementmustbeeitheranobjectreferenceoraregionreference:thesearedifferenttypesand
cannotbemixedwithinasinglearray.
tid1 = H5Tcopy (H5T_C_S1);
ret = H5Tset_size (tid1, H5T_VARIABLE);
CodeExample6‐30.SetthestringdatatypesizetoH5T_VARIABLE
char *rdata[SPACE1_DIM1];
ret=H5Dread(dataset, tid1, H5S_ALL, H5S_ALL, xfer_pid, rdata);
for(i=0; i<SPACE1_DIM1; i++) {
printf(“%d: len: %d, str is: %s\n”, i, strlen(rdata[i]),
rdata[i]);
}
ret=H5Dvlen_reclaim(tid1, sid1, xfer_pid, rdata);
CodeExample6‐31.Readvariable‐lengthstringsintoCstrings
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 235
AreferencedatatypecannotbedividedforI/O:anelementisreadorwrittencompletely.
dataset=H5Dcreate(fid1, “Dataset3”, H5T_STD_REF_OBJ, sid1,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
/* Create reference to dataset */
ret = H5Rcreate(&wbuf[0], fid1,“/Group1/Dataset1”, H5R_OBJECT,
-1);
/* Create reference to dataset */
ret = H5Rcreate(&wbuf[1], fid1, “/Group1/Dataset2”, H5R_OBJECT,
-1);
/* Create reference to group */
ret = H5Rcreate(&wbuf[2], fid1, “/Group1”, H5R_OBJECT, -1);
/* Create reference to committed datatype */
ret = H5Rcreate(&wbuf[3], fid1, “/Group1/Datatype1”,
H5R_OBJECT, -1);
/* Write selection to disk */
ret=H5Dwrite(dataset, H5T_STD_REF_OBJ, H5S_ALL, H5S_ALL,
H5P_DEFAULT, wbuf);
CodeExample6‐32.Createobjectreferencesandwritetoadataset
rbuf = malloc(sizeof(hobj_ref_t)*SPACE1_DIM1);
/* Read selection from disk */
ret=H5Dread(dataset, H5T_STD_REF_OBJ, H5S_ALL, H5S_ALL,
H5P_DEFAULT, rbuf);
/* Open dataset object */
dset2 = H5Rdereference(dataset, H5R_OBJECT, &rbuf[0]);
CodeExample6‐33.Readadatasetwithareferencedatatype
HDF5Datatypes HDF5User’sGuide
236 TheHDFGroup
6.6.3.ENUM
Theenumdatatypeimplementsasetof(name,value)pairs,similartoC/C++enum.Thevaluesarecur‐
rentlylimitedtonativeintegerdatatypes.Eachnamecanbethenameofonlyonevalue,andeachvalue
canhaveonlyonename.
ThedataelementsoftheENUMERATIONarestoredaccordingtothedatatype.Anexamplewouldbeasan
arrayofintegers.Theexamplebelowshowsanexampleofhowtocreateanenumerationwithfiveele‐
ments.Theelementsmapsymbolicnamesto2‐byteintegers.Seethetablebelow.
Thefigurebelowshowshowanarrayofeightvaluesmightbestored.Conceptually,thearrayisanarrayof
symbolicnames[BLACK,RED,WHITE,BLUE,...].Seeitemainthefigurebelow.Thesearestoredastheval‐
uesandareshortintegers.So,thefirst2bytesarethevalueassociatedwith“BLACK”,whichisthenumber
4,andsoon.Seeitembinthefigurebelow.
hid_t hdf_en_colors = H5Tcreate(H5T_ENUM, sizeof(short));
short val;
H5Tenum_insert(hdf_en_colors, “RED”, (val=0,&val));
H5Tenum_insert(hdf_en_colors, “GREEN”, (val=1,&val));
H5Tenum_insert(hdf_en_colors, “BLUE”, (val=2,&val));
H5Tenum_insert(hdf_en_colors, “WHITE”, (val=3,&val));
H5Tenum_insert(hdf_en_colors, “BLACK”, (val=4,&val));
H5Dcreate(fileid, datasetname, hdf_en_colors, spaceid,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
CodeExample6‐34.Createanenumerationwithfiveelements
Table6‐22.Anenumerationwithfiveelements
Name Value
RED 0
GREEN 1
BLUE 2
WHITE 3
BLACK 4
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 237
Theorderthatmembersareinsertedintoanenumerationtypeisunimportant;theimportantpartisthe
associationsbetweenthesymbolnamesandthevalues.Thus,twoenumerationdatatypeswillbeconsid‐
eredequalifandonlyifbothtypeshavethesamesymbol/valueassociationsandbothhaveequalunder‐
lyingintegerdatatypes.TypeequalityistestedwiththeH5Tequalfunction.
Ifaparticulararchitecturetypeisrequired,alittle‐endianorbig‐endiandatatypeforexample,useanative
integerdatatypeastheENUMbasedatatypeanduseH5Tconvertonvaluesastheyarereadfromor
writtentoadataset.
a)Logicaldatatobewritten‐eightelements
b)Thestoragelayout.Totalsizeofthearrayis16bytes,2bytesperelement.
Figure6‐17.Storinganenumarray
HDF5Datatypes HDF5User’sGuide
238 TheHDFGroup
6.6.4.Opaque
Insomecases,ausermayhavedataobjectsthatshouldbestoredandretrievedasblobswithnoattempt
tointerpretthem.Forexample,anapplicationmightwishtostoreanarrayofencryptedcertificateswhich
are100byteslong.
Whileanarbitraryblockofdatamayalwaysbestoredasbytes,characters,integers,orwhatever,this
mightmisleadprogramsaboutthemeaningofthedata.Theopaquedatatypedefinesdataelements
whichareuninterpretedbyHDF5.TheopaquedatamaybelabeledwithH5Tset_tagwithastringthat
mightbeusedbyanapplication.Forexample,theencryptedcertificatesmighthaveatagtoindicatethe
encryptionandthecertificatestandard.
6.6.5.Bitfield
Somedataisrepresentedasbits,wherethenumberofbitsisnotanintegralbyteandthebitsarenotnec‐
essarilyinterpretedasastandardtype.Someexamplesmightincludereadingsfrommachineregisters(for
example,switchpositions),acloudmask,ordatastructureswithseveralsmallintegersthatshouldbe
storeinasinglebyte.
Thisdatacouldbestoredasintegers,strings,orenumerations.However,thesestoragemethodswould
likelyresultinconsiderablewastedspace.Forexample,storingacloudmaskwithonebytepervalue
woulduseuptoeighttimesthespaceofapackedarrayofbits.
TheHDF5bitfielddatatypeclassdefinesadataelementthatisacontiguoussequenceofbits,whichare
storedondiskinapackedarray.Theprogrammingmodelisthesameasforunsignedintegers:thedata‐
typeobjectiscreatedbycopyingapredefineddatatype,andthentheprecision,offset,andpaddingare
set.
Whiletheuseofthebitfielddatatypewillreducestoragespacesubstantially,therewillstillbewasted
spaceifthebitfieldasawholedoesnotmatchthe1‐,2‐,4‐,or8‐byteunitinwhichitiswritten.The
remainingunusedspacecanberemovedbyapplyingtheN‐bitfiltertothedatasetcontainingthebitfield
data.Formoreinformation,see"UsingtheN‐bitFilter"onpage 143.
6.7.FillValues
The“fillvalue”foradatasetisthespecificationofthedefaultvalueassignedtodataelementsthathave
notyetbeenwritten.Inthecaseofadatasetwithanatomicdatatype,thefillvalueisasinglevalueofthe
appropriatedatatype,suchas‘0’or‘‐1.0’.Inthecaseofadatasetwithacompositedatatype,thefillvalue
isasingledataelementoftheappropriatetype.Forexample,foranarrayorcompounddatatype,thefill
valueisasingledataelementwithvaluesforallthecomponentelementsofthearrayorcompounddata‐
type.
Thefillvalueisset(permanently)whenthedatasetiscreated.Thefillvalueissetinthedatasetcreation
propertiesintheH5Dcreatecall.NotethattheH5Dcreatecallmustalsoincludethedatatypeofthe
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 239
dataset,andthevalueprovidedforthefillvaluewillbeinterpretedasasingleelementofthisdatatype.
Theexamplebelowshowscodewhichcreatesadatasetofintegerswithfillvalue‐1.Anyunwrittendata
elementswillbesetto‐1.
hid_t plist_id;
int filler;
filler = -1;
plist_id = H5Pcreate(H5P_DATASET_CREATE);
H5Pset_fill_value(plist_id, H5T_NATIVE_INT, &filler);
/* Create the dataset with fill value ‘-1’. */
dataset_id = H5Dcreate(file_id, “/dset”, H5T_STD_I32BE,
dataspace_id, H5P_DEFAULT, plist_id, H5P_DEFAULT);
CodeExample6‐35.Createadatasetwithafillvalueof‐1
typedef struct s1_t {
int a;
char b;
double c;
} s1_t;
s1_t filler;
s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b),
H5T_NATIVE_CHAR);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c),
H5T_NATIVE_DOUBLE);
filler.a = -1;
filler.b = ‘*’;
filler.c = -2.0;
plist_id = H5Pcreate(H5P_DATASET_CREATE);
H5Pset_fill_value(plist_id, s1_tid, &filler);
/* Create the dataset with fill value */
/* (-1, ‘*’, -2.0). */
dataset = H5Dcreate(file, datasetname, s1_tid, space,
H5P_DEFAULT, plist_id, H5P_DEFAULT);
CodeExample6‐36.Createafillvalueforacompounddatatype
HDF5Datatypes HDF5User’sGuide
240 TheHDFGroup
Thefigureaboveshowshowtocreateafillvalueforacompounddatatype.Theprocedureisthesameas
thepreviousexampleexceptthefillermustbeastructurewiththecorrectfields.Eachfieldisinitializedto
thedesiredfillvalue.
Thefillvalueforadatasetcanberetrievedbyreadingthedatasetcreationpropertiesofthedatasetand
thenbyreadingthefillvaluewithH5Pget_fill_value.Thedatawillbereadintomemoryusingthe
storagelayoutspecifiedbythedatatype.ThistransferwillconvertdatainthesamewayasH5Dread.The
examplebelowshowshowtogetthefillvaluefromthedatasetcreatedintheexample"Createadataset
withafillvalueof‐1"onpage 239.
Asimilarprocedureisfollowedforanydatatype.Theexamplebelowshowshowtoreadthefillvaluefor
thecompounddatatypecreatedinanexampleabove.Notethattheprogrammustpassanelementlarge
enoughtoholdafillvalueofthedatatypeindicatedbytheargumenttoH5Pget_fill_value.Also,the
programmustunderstandthedatatypeinordertointerpretitscomponents.Thismaybedifficultto
determinewithoutknowledgeoftheapplicationthatcreatedthedataset.
hid_t plist2;
int filler;
dataset_id = H5Dopen(file_id, “/dset”, H5P_DEFAULT);
plist2 = H5Dget_create_plist(dataset_id);
H5Pget_fill_value(plist2, H5T_NATIVE_INT, &filler);
/* filler has the fill value, ‘-1’ */
CodeExample6‐37.Retrieveafillvalue
char * fillbuf;
int sz;
dataset = H5Dopen( file, DATASETNAME, H5P_DEFAULT);
s1_tid = H5Dget_type(dataset);
CodeExample6‐38.Readthefillvalueforacompounddatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 241
6.8.ComplexCombinationsofDatatypes
Severalcompositedatatypeclassesdefinecollectionsofotherdatatypes,includingothercompositedata‐
types.Ingeneral,adatatypecanbenestedtoanydepth,withanycombinationofdatatypes.
Forexample,acompounddatatypecanhavemembersthatareothercompounddatatypes,arrays,VL
datatypes.Anarraycanbeanarrayofarray,anarrayofcompound,oranarrayofVL.AndaVLdatatype
canbeavariable‐lengtharrayofcompound,array,orVLdatatypes.
Thesecomplicatedcombinationsofdatatypesformalogicaltree,withasinglerootdatatype,andleaves
whichmustbeatomicdatatypes(predefinedoruser‐defined).Thefigurebelowshowsanexampleofa
logicaltreedescribingacompounddatatypeconstructedfromdifferentdatatypes.
Recallthatthedatatypeisadescriptionofthelayoutofstorage.Thecomplicatedcompounddatatypeis
constructedfromcomponentdatatypes,eachofwhichdescribethelayoutofpartofthestorage.Any
datatypecanbeusedasacomponentofacompounddatatype,withthefollowingrestrictions:
1. Nobytecanbepartofmorethanonecomponentdatatype(inotherwords,thefieldscannot
overlapwithinthecompounddatatype)
2. Thetotalsizeofthecomponentsmustbelessthanorequaltothetotalsizeofthecompound
datatype
TheserestrictionsareessentiallytherulesforCstructuresandsimilarrecordtypesfamiliarfromprogram‐
minglanguages.Multipletyping,suchasaCunion,isnotallowedinHDF5datatypes.
sz = H5Tget_size(s1_tid);
fillbuf = (char *)malloc(sz);
plist_id = H5Dget_create_plist(dataset);
H5Pget_fill_value(plist_id, s1_tid, fillbuf);
printf(“filler.a: %d\n”,((s1_t *) fillbuf)->a);
printf(“filler.b: %c\n”,((s1_t *) fillbuf)->b);
printf(“filler.c: %f\n”,((s1_t *) fillbuf)->c);
CodeExample6‐38.Readthefillvalueforacompounddatatype
HDF5Datatypes HDF5User’sGuide
242 TheHDFGroup
6.8.1.CreatingaComplicatedCompoundDatatype
Toconstructacomplicatedcompounddatatype,eachcomponentisconstructed,andthenaddedtothe
enclosingdatatypedescription.Theexamplebelowshowshowtocreateacompounddatatypewithfour
members:
•“T1”,acompounddatatypewiththreemembers
•“T2”,acompounddatatypewithtwomembers
•“T3”,aone‐dimensionalarrayofintegers
•“T4”,
astring
Belowtheexamplecodeisafigurethatshowsthisdatatypeasalogicaltree.Theoutputoftheh5dump
utilityisshownintheexamplebelowthefigure.
Eachdatatypeiscreatedasaseparatedatatypeobject.Figure20belowshowsthestoragelayoutforthe
fourindividualdatatypes.Thenthedatatypesareinsertedintotheouterdatatypeatanappropriateoff‐
set.Figure21belowshowstheresultingstoragelayout.Thecombinedrecordis89byteslong.
TheDatasetiscreatedusingthecombinedcompounddatatype.Thedatasetisdeclaredtobea4by3
arrayofcompounddata.Eachdataelementisaninstanceofthe89‐bytecompounddatatype.Figure22
belowshowsthelayoutofthedataset,andexpandsoneoftheelementstoshowtherelativepositionof
thecomponentdataelements.
Figure6‐18.Acompounddatatypebuiltwithdifferentdatatypes
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 243
Eachdataelementisacompounddatatype,whichcanbewrittenorreadasarecord,oreachfieldmaybe
readorwrittenindividually.Thefirstfield(“T1”)isitselfacompounddatatypewiththreefields(“T1.a”,
“T1.b”,and“T1.c”).“T1”canbereadorwrittenasarecord,orindividualfieldscanbeaccessed.Similarly,
thesecondfiledisacompounddatatypewithtwofields(“T2.f1”,“T2.f2”).
Thethirdfield(“T3”)isanarraydatatype.Thus,“T3”shouldbeaccessedasanarrayof40integers.Array
datacanonlybereadorwrittenasasingleelement,soall40integersmustbereadorwrittentothethird
field.Thefourthfield(“T4”)isasinglestringoflength25.
typedef struct s1_t {
int a;
char b;
double c;
} s1_t;
typedef struct s2_t {
float f1;
float f2;
} s2_t;
hid_t s1_tid, s2_tid, s3_tid, s4_tid, s5_tid;
/* Create a datatype for s1 */
s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b),
H5T_NATIVE_CHAR);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c),
H5T_NATIVE_DOUBLE);
/* Create a datatype for s2. *.
s2_tid = H5Tcreate (H5T_COMPOUND, sizeof(s2_t));
H5Tinsert(s2_tid, “f1”, HOFFSET(s2_t, f1),
H5T_NATIVE_FLOAT);
H5Tinsert(s2_tid, “f2”, HOFFSET(s2_t, f2),
H5T_NATIVE_FLOAT);
/* Create a datatype for an Array of integers */
s3_tid = H5Tarray_create(H5T_NATIVE_INT, RANK, dim);
/* Create a datatype for a String of 25 characters */
s4_tid = H5Tcopy(H5T_C_S1);
H5Tset_size(s4_tid, 25);
CodeExample6‐39.Createacompounddatatypewithfourmembers
HDF5Datatypes HDF5User’sGuide
244 TheHDFGroup
/*
* Create a compound datatype composed of one of each of
* these types. The total size is the sum of the size of
* each.
*/
sz = H5Tget_size(s1_tid) + H5Tget_size(s2_tid) +
H5Tget_size(s3_tid) + H5Tget_size(s4_tid);
s5_tid = H5Tcreate (H5T_COMPOUND, sz);
/* Insert the component types at the appropriate */
* offsets.
*/
H5Tinsert(s5_tid, “T1”, 0, s1_tid);
H5Tinsert(s5_tid, “T2”, sizeof(s1_t), s2_tid);
H5Tinsert(s5_tid, “T3”, sizeof(s1_t)+sizeof(s2_t), s3_tid);
H5Tinsert(s5_tid, “T4”, (sizeof(s1_t) +sizeof(s2_t)+
H5Tget_size(s3_tid)), s4_tid);
/*
* Create the dataset with this datatype.
*/
dataset = H5Dcreate(file, DATASETNAME, s5_tid, space,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
CodeExample6‐39.Createacompounddatatypewithfourmembers
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 245
Figure6‐19.Logicaltreeforthecompounddatatypewithfourmembers
DATATYPE H5T_COMPOUND {
H5T_COMPOUND {
H5T_STD_I32LE “a_name”;
H5T_STD_I8LE “b_name”;
H5T_IEEE_F64LE “c_name”;
} “T1”;
H5T_COMPOUND {
H5T_IEEE_F32LE “f1”;
H5T_IEEE_F32LE “f2”;
} “T2”;
H5T_ARRAY { [10] H5T_STD_I32LE } “T3”;
H5T_STRING {
STRSIZE 25;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
} “T4”;
}
CodeExample6‐40.Outputfromh5dumpforthecompounddatatype
HDF5Datatypes HDF5User’sGuide
246 TheHDFGroup
a)Compoundtype‘s1_t’,size16bytes.
b)Compoundtype‘s2_t’,size8bytes.
c)Arraytype‘s3_tid’,40integers,totalsize40bytes.
d)Stringtype‘s4_tid’,size25bytes.
Figure6‐20.Thestoragelayoutforthefourmemberdatatypes
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 247
Figure6‐21.Thestoragelayoutofthecombinedfourmembers
HDF5Datatypes HDF5User’sGuide
248 TheHDFGroup
6.8.2.AnalyzingandNavigatingaCompoundDatatype
Acomplicatedcompounddatatypecanbeanalyzedpiecebypiecetodiscovertheexactstoragelayout.In
theexampleabove,theouterdatatypeisanalyzedtodiscoverthatitisacompounddatatypewithfour
members.Eachmemberisanalyzedinturntoconstructacompletemapofthestoragelayout.
Theexamplebelowshowsanexampleofcodethatpartiallyanalyzesanestedcompounddatatype.The
nameandoveralloffsetandsizeofthecomponentdatatypeisdiscovered,andthenitstypeisanalyzed
dependingonthedatatypeclass.Throughthismethod,thecompletestoragelayoutcanbediscovered.
a)A4x3arrayofCompoundDatatype
b)Element[1,1]expanded
Figure6‐22.Thelayoutofthedataset
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 249
6.9.LifeCycleoftheDatatypeObject
ApplicationprogramsaccessHDF5datatypesthroughidentifiers.Identifiersareobtainedbycreatinga
newdatatypeorbycopyingoropeninganexistingdatatype.Theidentifiercanbeuseduntilitisclosedor
s1_tid = H5Dget_type(dataset);
if (H5Tget_class(s1_tid) == H5T_COMPOUND) {
printf(“COMPOUND DATATYPE {\n”);
sz = H5Tget_size(s1_tid);
nmemb = H5Tget_nmembers(s1_tid);
printf(“ %d bytes\n”,sz);
printf(“ %d members\n”,nmemb);
for (i =0; i < nmemb; i++) {
s2_tid = H5Tget_member_type(s1_tid, i);
if (H5Tget_class(s2_tid) == H5T_COMPOUND) {
/* recursively analyze the nested type. */
} else if (H5Tget_class(s2_tid) == H5T_ARRAY) {
sz2 = H5Tget_size(s2_tid);
printf(“ %s: NESTED ARRAY DATATYPE offset %d size %d
{\n”,
H5Tget_member_name(s1_tid, i),
H5Tget_member_offset(s1_tid, i),
sz2);
H5Tget_array_dims(s2_tid, dim);
s3_tid = H5Tget_super(s2_tid);
/* Etc., analyze the base type of the array */
} else {
/* analyze a simple type */
printf(“ %s: type code %d offset %d size %d\n”,
H5Tget_member_name(s1_tid, i),
H5Tget_class(s2_tid),
H5Tget_member_offset(s1_tid, i),
H5Tget_size(s2_tid));
}
/* and so on…. */
CodeExample6‐41.Analyzingacompounddatatypeanditsmembers
HDF5Datatypes HDF5User’sGuide
250 TheHDFGroup
untilthelibraryshutsdown.Seeitemsaandbinthefigurebelow.Bydefault,adatatypeistransient,and
itdisappearswhenitisclosed.
Whenadatasetorattributeiscreated(H5DcreateorH5Acreate),itsdatatypeisstoredintheHDF5file
aspartofthedatasetorattributeobject.Seeitemcinthefigurebelow.Onceanobjectcreated,itsdata‐
typecannotbechangedordeleted.ThedatatypecanbeaccessedbycallingH5Dget_type,H5Aget_-
type,H5Tget_super,orH5Tget_member_type.Seeitemdinthefigurebelow.Thesecallsreturnan
identifiertoatransientcopyofthedatatypeofthedatasetorattributeunlessthedatatypeisacommitted
datatype.
Notethatwhenanobjectiscreated,thestoreddatatypeisacopyofthetransientdatatype.Iftwoobjects
arecreatedwiththesamedatatype,theinformationisstoredineachobjectwiththesameeffectasiftwo
differentdatatypeswerecreatedandused.
AtransientdatatypecanbestoredusingH5TcommitintheHDF5fileasanindependent,namedobject,
calledacommitteddatatype.Committeddatatypeswereformerlyknownasnameddatatypes.Seeiteme
inthefigurebelow.Subsequently,whenacommitteddatatypeisopenedwithH5Topen(itemf),oris
obtainedwithH5Tget_typeorsimilarcall(itemk),thereturnisanidentifiertoatransientcopyofthe
storeddatatype.Theidentifiercanbeusedinthesamewayasotherdatatypeidentifiersexceptthatthe
committeddatatypecannotbemodified.WhenacommitteddatatypeiscopiedwithH5Tcopy,thereturn
isanew,modifiable,transientdatatypeobject(itemf).
Whenanobjectiscreatedusingacommitteddatatype(H5Dcreate,H5Acreate),thestoreddatatypeis
usedwithoutcopyingittotheobject.Seeitemjinthefigurebelow.Inthiscase,ifmultipleobjectsarecre‐
atedusingthesamecommitteddatatype,theyallsharetheexactsamedatatypeobject.Thissavesspace
andmakesclearthatthedatatypeisshared.Notethatacommitteddatatypecanbesharedbyobjects
withinthesameHDF5file,butnotbyobjectsinotherfiles.Formoreinformationoncopyingcommitted
datatypestootherHDF5files,seethe“CopyingCommittedDatatypeswithH5Ocopy”topicinthe“Addi‐
tionalResources”chapter.
AcommitteddatatypecanbedeletedfromthefilebycallingH5LdeletewhichreplacesH5Gunlink.See
itemiinthefigurebelow.Ifoneormoreobjectsarestillusingthedatatype,thecommitteddatatypecan‐
notbeaccessedwithH5Topen,butwillnotberemovedfromthefileuntilitisnolongerused.H5Tget_-
typeandsimilarcallswillreturnatransientcopyofthedatatype.
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 251
Transientdatatypesareinitiallymodifiable.Notethatwhenadatatypeiscopiedorwhenitiswrittento
thefile(whenanobjectiscreated)orthedatatypeisusedtocreateacompositedatatype,acopyofthe
currentstateofthedatatypeisused.Ifthedatatypeisthenmodified,thechangeshavenoeffectondata‐
sets,attributes,ordatatypesthathavealreadybeencreated.Seethefigurebelow.
Atransientdatatypecanbemaderead‐only(H5Tlock).Notethatthedatatypeisstilltransient,andoth‐
erwisedoesnotchange.Adatatypethatisimmutableisread‐onlybutcannotbeclosedexceptwhenthe
entirelibraryisclosed.ThepredefinedtypessuchasH5T_NATIVE_INTareimmutabletransienttypes.
Figure6‐23.Lifecycleofadatatype
HDF5Datatypes HDF5User’sGuide
252 TheHDFGroup
Tocreatetwoormoredatasetsthatshareacommondatatype,firstcommitthedatatype,andthenuse
thatdatatypetocreatethedatasets.Seetheexamplebelow.
Figure6‐24.Transientdatatypestates:modifiable,read‐only,andimmutable
hid_t t1 = ...some transient type...;
H5Tcommit (file, “shared_type”, t1, H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
hid_t dset1 = H5Dcreate (file, “dset1”, t1, space, H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
hid_t dset2 = H5Dcreate (file, “dset2”, t1, space, H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
hid_t dset1 = H5Dopen (file, “dset1”, H5P_DEFAULT);
hid_t t2 = H5Dget_type (dset1);
hid_t dset3 = H5Dcreate (file, “dset3”, t2, space, H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
hid_t dset4 = H5Dcreate (file, “dset4”, t2, space, H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);
CodeExample6‐42.Createashareabledatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 253
6.10.DataTransfer:DatatypeConversionandSelection
Whendataistransferred(writeorread),thestoragelayoutofthedataelementsmaybedifferent.For
example,anintegermightbestoredondiskinbig‐endianbyteorderandreadintomemorywithlittle‐
endianbyteorder.Inthiscase,eachdataelementwillbetransformedbytheHDF5Libraryduringthedata
transfer.
Theconversionofdataelementsiscontrolledbyspecifyingthedatatypeofthesourceandspecifyingthe
intendeddatatypeofthedestination.Thestorageformatondiskisthedatatypespecifiedwhenthedata‐
setiscreated.Thedatatypeofmemorymustbespecifiedinthelibrarycall.
Inordertobeconvertible,thedatatypeofthesourceanddestinationmusthavethesamedatatypeclass
(withtheexceptionofenumerationtype).Thus,integerscanbeconvertedtootherintegers,andfloatsto
otherfloats,butintegerscannot(yet)beconvertedtofloats.Foreachatomicdatatypeclass,thepossible
Table6‐23.DatatypeAPIs
Function Description
hid_t H5Topen (hid_t location, const
char *name)
Acommitteddatatypecanbeopenedby
callingthisfunction,whichreturnsadata‐
typeidentifier.Theidentifiershouldeven‐
tuallybereleasedbycallingH5Tclose()
toreleaseresources.Thecommitteddata‐
typereturnedbythisfunctionisread‐only
oranegativevalueisreturnedforfailure.
Thelocationiseitherafileorgroupiden‐
tifier.
herr_t H5Tcommit (hid_t location,
const char *name, hid_t type, H5P_DE-
FAULT, H5P_DEFAULT, H5P_DEFAULT)
Atransientdatatype(notimmutable)can
bewrittentoafileandturnedintoacom‐
mitteddatatypebycallingthisfunction.
Thelocationiseitherafileorgroupiden‐
tifierandwhencombinedwithname
referstoanewcommitteddatatype.
htri_t H5Tcommitted (hid_t type) Atypecanbequeriedtodetermineifitis
acommittedtypeoratransienttype.If
thisfunctionreturnsapositivevaluethen
thetypeiscommitted.Datasetswhich
returncommitteddatatypeswith
H5Dget_type()areabletosharethe
datatypewithotherdatasetsinthesame
file.
HDF5Datatypes HDF5User’sGuide
254 TheHDFGroup
conversionsaredefined.Anenumerationdatatypecanbeconvertedtoanintegerorafloating‐pointnum‐
berdatatype.
Basically,anydatatypecanbeconvertedtoanotherdatatypeofthesamedatatypeclass.TheHDF5Library
automaticallyconvertsallproperties.Ifthedestinationistoosmalltoholdthesourcevaluethenanover‐
floworunderflowexceptionoccurs.IfahandlerisdefinedwiththeH5Pset_type_conv_cbfunction,it
willbecalled.Otherwise,adefaultactionwillbeperformed.Thetablebelowsummarizesthedefault
actions.
Forexample,whenreadingdatafromadataset,thesourcedatatypeisthedatatypesetwhenthedataset
wascreated,andthedestinationdatatypeisthedescriptionofthestoragelayoutinmemory.Thedestina‐
tiondatatypemustbespecifiedintheH5Dreadcall.Theexamplebelowshowsanexampleofreadinga
datasetof32‐bitintegers.Thefigurebelowtheexampleshowsthedatatransformationthatisperformed.
Table6‐24.Defaultactionsfordatatypeconversionexceptions
DatatypeClass PossibleExceptions DefaultAction
Integer Size,offset,pad
Float Size,offset,pad,ebits
String Size Truncates,zeroterminateifrequired.
Enumeration Nofield Allbitsset
/* Stored as H5T_STD_BE32 */
/* Use the native memory order in the destination */
mem_type_id = H5Tcopy(H5T_NATIVE_INT);
status = H5Dread(dataset_id, mem_type_id, mem_space_id,
file_space_id, xfer_plist_id, buf );
CodeExample6‐43.SpecifythedestinationdatatypewithH5Dread
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 255
OnethingtonoteintheexampleaboveistheuseofthepredefinednativedatatypeH5T_NATIVE_INT.
Recallthatinthisexample,thedatawasstoredasa4‐bytesinbig‐endianorder.Theapplicationwantsto
readthisdataintoanarrayofintegersinmemory.Dependingonthesystem,thestoragelayoutofmem‐
orymightbeeitherbigorlittle‐endian,sothedatamayneedtobetransformedonsomeplatformsand
notonothers.TheH5T_NATIVE_INTtypeissetbytheHDF5Librarytobethecorrecttypetodescribe
thestoragelayoutofthememoryonthesystem.Thus,thecodeintheexampleabovewillworkcorrectly
onanyplatform,performingatransformationwhenneeded.
Therearepredefinednativetypesformostatomicdatatypes,andthesecanbecombinedincomposite
datatypes.Ingeneral,thepredefinednativedatatypesshouldalwaysbeusedfordatastoredinmemory.
SourceDatatype:H5T_STD_BE32
....
DestinationDatatype:H5T_STD_LE32
....
Figure6‐25.Layoutofadatatypeconversion
StorageProperties
Predefinednativedatatypesdescribethestorage
propertiesofmemory.
HDF5Datatypes HDF5User’sGuide
256 TheHDFGroup
Forcompositedatatypes,thecomponentatomicdatatypeswillbeconverted.Foravariable‐lengthdata‐
type,thesourceanddestinationmusthavecompatiblebasedatatypes.Forafixed‐sizestringdatatype,
thelengthandpaddingofthestringswillbeconverted.Variable‐lengthstringsareconvertedasvariable‐
lengthdatatypes.
Foranarraydatatype,thesourceanddestinationmusthavethesamerankanddimensions,andthebase
datatypemustbecompatible.Forexampleanarraydatatypeof4x332‐bitbig‐endianintegerscanbe
transferredtoanarraydatatypeof4x3little‐endianintegers,butnottoa3x4array.
Foranenumerationdatatype,dataelementsareconvertedbymatchingthesymbolnamesofthesource
anddestinationdatatype.Thefigurebelowshowsanexampleofhowtwoenumerationswiththesame
namesanddifferentvalueswouldbeconverted.Thevalue‘2’inthesourcedatasetwouldbeconvertedto
‘0x0004’inthedestination.
Ifthesourcedatastreamcontainsvalueswhicharenotinthedomainoftheconversionmapthenanover‐
flowexceptionisraisedwithinthelibrary.
Thelibraryalsoallowsconversionfromenumerationtoanumericdatatype.Anumericdatatypeiseither
anintegerorafloating‐pointnumber.Thisconversioncansimplifytheapplicationprogrambecausethe
basetypeforanenumerationdatatypeisanintegerdatatype.Theapplicationprogramcanreadthedata
fromadatasetofenumerationdatatypeinfileintoamemorybufferofnumericdatatype.Anditcanwrite
enumerationdatafrommemoryintoadatasetofnumericdatatypeinfile,too.
Forcompounddatatypes,eachfieldofthesourceanddestinationdatatypeisconvertedaccordingtoits
type.Thenameofthefieldsmustbethesameinthesourceandthedestinationinorderforthedatato
beconverted.
Theexamplebelowshowsthecompounddatatypesshowssamplecodetocreateacompounddatatype
withthefieldsalignedonwordboundaries(s1_tid)andwiththefieldspacked(s2_tid).Theformerissuit‐
ableasadescriptionofthestoragelayoutinmemory,thelatterwouldgiveamorecompactstoreondisk.
Thesetypescanbeusedfortransferringdata,withs2_tidusedtocreatethedataset,ands1_tidused
asthememorydatatype.
Figure6‐26.Anenumdatatypeconversion
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 257
Whenthedataistransferred,thefieldswithineachdataelementwillbealignedaccordingtothedatatype
specification.Thefigurebelowshowshowonedataelementwouldbealignedinmemoryandondisk.
Notethatthesizeandbyteorderoftheelementsmightalsobeconvertedduringthetransfer.
Itisalsopossibletotransfersomeofthefieldsofcompounddatatypes.Basedontheexampleabove,the
examplebelowshowsacompounddatatypethatselectsthefirstandthirdfieldsofthes1_tid.Thesec‐
onddatatypecanbeusedasthememorydatatype,inwhichcasedataisreadfromorwrittentothesetwo
fields,whileskippingthemiddlefield.Thesecondfigurebelowshowsthelayoutfortwodataelements.
typedef struct s1_t {
int a;
char b;
double c;
} s1_t;
s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b),
H5T_NATIVE_CHAR);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c),
H5T_NATIVE_DOUBLE);
s2_tid = H5Tcopy(s1_tid);
H5Tpack(s2_tid);
CodeExample6‐44.Createanalignedandpackedcompounddatatype
HDF5Datatypes HDF5User’sGuide
258 TheHDFGroup
Figure6‐27.Alignmentofacompounddatatype
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 259
typedef struct s1_t {
int a;
char b;
double c;
} s1_t;
typedef struct s2_t { /* two fields from s1_t */
int a;
double c;
} s2_t;
s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s1_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “b_name”, HOFFSET(s1_t, b),
H5T_NATIVE_CHAR);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s1_t, c),
H5T_NATIVE_DOUBLE);
s2_tid = H5Tcreate (H5T_COMPOUND, sizeof(s2_t));
H5Tinsert(s1_tid, “a_name”, HOFFSET(s2_t, a),
H5T_NATIVE_INT);
H5Tinsert(s1_tid, “c_name”, HOFFSET(s2_t, c),
H5T_NATIVE_DOUBLE);
CodeExample6‐45.Transfersomefieldsofacompounddatatype
HDF5Datatypes HDF5User’sGuide
260 TheHDFGroup
6.11.TextDescriptionsofDatatypes:Conversiontoand
from
HDF5providesameansforgeneratingaportableandhuman‐readabletextdescriptionofadatatypeand
forgeneratingadatatypefromsuchatextdescription.Thiscapabilityisparticularlyusefulforcreating
complexdatatypesinasinglestep,forcreatingatextdescriptionofadatatypefordebuggingpurposes,
andforcreatingaportabledatatypedefinitionthatcanthenbeusedtorecreatethedatatypeonmany
platformsorinotherapplications.
ThesetasksarehandledbytwofunctionsprovidedintheHDF5Litehigh‐levellibrary:
Figure6‐28.Layoutwhenanelementisskipped
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 261
•H5LTtext_to_dtype CreatesanHDF5datatypeinasinglestep.
•H5LTdtype_to_text TranslatesanHDF5datatypeintoatextdescription.
NotethatthisfunctionalityrequiresthattheHDF5High‐LevelLibrary(H5LT)beinstalled.
WhileH5LTtext_to_dtypecanbeusedtogenerateanysortofdatatype,itisparticularlyusefulfor
complexdatatypes.
H5LTdtype_to_textismostlikelytobeusedintwosortsofsituations:whenadatatypemustbe
closelyexaminedfordebuggingpurposeortocreateaportabletextdescriptionofthedatatypethatcan
thenbeusedtorecreatethedatatypeonotherplatformsorinotherapplications.
ThesetwofunctionsworkforallvalidHDF5datatypesexcepttime,bitfield,andreferencedatatypes.
ThecurrentlysupportedtextformatusedbyH5LTtext_to_dtypeandH5LTdtype_to_textisthe
datadescriptionlanguage(DDL)andconformstotheHDF5DDL.TheportionoftheHDF5DDLthatdefines
HDF5datatypesappearsbelow.
<datatype> ::= <atomic_type> | <compound_type> |
<array_type> | <variable_length_type>
<atomic_type> ::= <integer> | <float> | <time> |
<string> | <bitfield> | <opaque> |
<reference> | <enum>
<integer> ::= H5T_STD_I8BE | H5T_STD_I8LE |
H5T_STD_I16BE | H5T_STD_I16LE |
H5T_STD_I32BE | H5T_STD_I32LE |
H5T_STD_I64BE | H5T_STD_I64LE |
H5T_STD_U8BE | H5T_STD_U8LE |
H5T_STD_U16BE | H5T_STD_U16LE |
H5T_STD_U32BE | H5T_STD_U32LE |
H5T_STD_U64BE | H5T_STD_U64LE |
H5T_NATIVE_CHAR | H5T_NATIVE_UCHAR |
H5T_NATIVE_SHORT | H5T_NATIVE_USHORT |
H5T_NATIVE_INT | H5T_NATIVE_UINT |
H5T_NATIVE_LONG | H5T_NATIVE_ULONG |
H5T_NATIVE_LLONG | H5T_NATIVE_ULLONG
<float> ::= H5T_IEEE_F32BE | H5T_IEEE_F32LE |
H5T_IEEE_F64BE | H5T_IEEE_F64LE |
H5T_NATIVE_FLOAT | H5T_NATIVE_DOUBLE |
H5T_NATIVE_LDOUBLE
<time> ::= TBD
CodeExample6‐46.ThedefinitionofHDF5datatypesfromtheHDF5DDL
HDF5Datatypes HDF5User’sGuide
262 TheHDFGroup
ThedefinitionsofopaqueandcompounddatatypeabovearerevisedforHDF5Release1.8.InRelease
1.6.5.andearlier,theyweredefinedasfollows:
<string> ::= H5T_STRING { STRSIZE <strsize> ;
STRPAD <strpad> ;
CSET <cset> ;
CTYPE <ctype> ;}
<strsize> ::= <int_value> | H5T_VARIABLE
<strpad> ::= H5T_STR_NULLTERM | H5T_STR_NULLPAD |
H5T_STR_SPACEPAD
<cset> ::= H5T_CSET_ASCII | H5T_CSET_UTF8
<ctype> ::= H5T_C_S1 | H5T_FORTRAN_S1
<bitfield> ::= TBD
<opaque> ::= H5T_OPAQUE { OPQ_SIZE <opq_size>;
OPQ_TAG <opq_tag>; }
opq_size ::= <int_value>
opq_tag ::= "<string>"
<reference> ::= Not supported
<compound_type> ::= H5T_COMPOUND { <member_type_def>+ }
<member_type_def> ::= <datatype> <field_name> <offset>opt;
<field_name> ::= "<identifier>"
<offset> ::= : <int_value>
<variable_length_type> ::= H5T_VLEN { <datatype> }
<array_type> ::= H5T_ARRAY { <dim_sizes> <datatype> }
<dim_sizes> ::= [<dimsize>] | [<dimsize>] <dim_sizes>
<dimsize> ::= <int_value>
<enum> ::= H5T_ENUM { <enum_base_type>; <enum_def>+ }
<enum_base_type> ::= <integer>
// Currently enums can only hold integer type data, but
// they may be expanded in the future to hold any
// datatype
<enum_def> ::= <enum_symbol> <enum_val>;
<enum_symbol> ::= "<identifier>"
<enum_val> ::= <int_value>
CodeExample6‐46.ThedefinitionofHDF5datatypesfromtheHDF5DDL
HDF5User’sGuide HDF5Datatypes
TheHDFGroup 263
Examples
ThecodesamplebelowillustratestheuseofH5LTtext_to_dtypetogenerateavariable‐lengthstring
datatype.
ThecodesamplebelowillustratestheuseofH5LTtext_to_dtypetogenerateacomplexarraydata‐
type.
<opaque> ::= H5T_OPAQUE { <identifier> }
<compound_type> ::= H5T_COMPOUND { <member_type_def>+ }
<member_type_def> ::= <datatype> <field_name> ;
<field_name> ::= <identifier>
CodeExample6‐47.Olddefinitionsoftheopaqueandcompounddatatypes
hid_t dtype;
if((dtype = H5LTtext_to_dtype(“H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}”, H5LT_DDL))<0)
goto out;
CodeExample6‐48.Creatingavariable‐lengthstringdatatypefromatextdescription
HDF5Datatypes HDF5User’sGuide
264 TheHDFGroup
hid_t dtype;
if((dtype = H5LTtext_to_dtype(“H5T_ARRAY { [5][7][13] H5T_ARRAY
{ [17][19] H5T_COMPOUND
{
H5T_STD_I8BE
\“arr_compound_1\”;
H5T_STD_I32BE
\“arr_compound_2\”;
}
}
}”, H5LT_DDL))<0)
goto out;
CodeExample6‐49.Creatingacomplexarraydatatypefromatextdescription
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 265
7.HDF5DataspacesandPartialI/O
7.1.Introduction
TheHDF5dataspaceisarequiredcomponentofanHDF5datasetorattributedefinition.Thedataspace
definesthesizeandshapeofthedatasetorattributerawdata.Inotherwords,adataspacedefinesthe
numberofdimensionsandthesizeofeachdimensionofthemultidimensionalarrayinwhichtherawdata
isrepresented.Thedataspacemustbedefinedwhenthedatasetorattributeiscreated.
ThedataspaceisalsousedduringdatasetI/Ooperations,definingtheelementsofthedatasetthatpartic‐
ipateintheI/Ooperation.
Thischapterexplainsthedataspaceobjectanditsuseindatasetandattributecreationanddatatransfer.
Italsodescribesselectionoperationsonadataspaceusedtoimplementsub‐setting,sub‐sampling,and
scatter‐gatheraccesstodatasets.
7.2.Dataspace(H5S)FunctionSummaries
Thissectionprovidesareferencelistofdataspacefunctions,theH5SAPIs,withbriefdescriptions.The
functionsarepresentedinthefollowingcategories:
•Dataspacemanagementfunctions
•Dataspacequeryfunctions
•Dataspaceselectionfunctions:hyperslabs
•Dataspaceselectionfunctions:points
Therestofthechapterwillprovideexamplesandexplanationsofhowtousethesefunctions.
FunctionListing7‐1.Dataspacemanagementfunctions
CFunction
FortranSubroutine
Purpose
H5Screate
h5screate_f
Createsanewdataspaceofaspecifiedtype.
H5Scopy
h5scopy_f
Createsanexactcopyofadataspace.
H5Sclose
h5sclose_f
Releasesandterminatesaccesstoa
dataspace.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 266
H5Sdecode
h5sdecode_f
Decodeabinaryobjectdescriptionofa
dataspaceandreturnanewobjectidentifier.
H5Sencode
h5sencode
Encodeadataspaceobjectdescriptionintoa
binarybuffer.
H5Screate_simple
h5screate_simple_f
Createsanewsimpledataspaceandopensit
foraccess.
H5Sis_simple
h5sis_simple_f
Determineswhetheradataspaceisasimple
dataspace.
H5Sextent_copy
h5sextent_copy_f
Copiestheextentofadataspace.
H5Sextent_equal
h5sextent_equal_f
Determineswhethertwodataspaceextents
areequal.
H5Sset_extent_simple
h5sset_extent_simple_f
Setsorresetsthesizeofanexisting
dataspace.
H5Sset_extent_none
h5sset_extent_none_f
Removestheextentfromadataspace.
FunctionListing7‐2.Dataspacequeryfunctions
CFunction
FortranSubroutine
Purpose
H5Sget_simple_extent_dims
h5sget_simple_extent_dims_f
Retrievesdataspacedimensionsizeandmaxi‐
mumsize.
H5Sget_simple_extent_ndims
h5sget_simple_extent_ndims_f
Determinesthedimensionalityofa
dataspace.
H5Sget_simple_extent_npoints
h5sget_simple_extent_npoints_f
Determinesthenumberofelementsina
dataspace.
H5Sget_simple_extent_type
h5sget_simple_extent_type_f
Determinethecurrentclassofadataspace.
FunctionListing7‐1.Dataspacemanagementfunctions
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 267
FunctionListing7‐3.Dataspaceselectionfunctions:hyperslabs
CFunction
FortranSubroutine
Purpose
H5Soffset_simple
h5soffset_simple_f
Setstheoffsetofasimpledataspace.
H5Sget_select_type
h5sget_select_type_f
Determinesthetypeofthedataspaceselec‐
tion.
H5Sget_select_hyper_nblocks
h5sget_select_hyper_nblocks_f
Getnumberofhyperslabblocks.
H5Sget_select_hyper_blocklist
h5sget_select_hyper_blocklist_f
Getsthelistofhyperslabblockscurrently
selected.
H5Sget_select_bounds
h5sget_select_bounds_f
Getstheboundingboxcontainingthecurrent
selection.
H5Sselect_all
h5sselect_all_f
Selectstheentiredataspace.
H5Sselect_none
h5sselect_none_f
Resetstheselectionregiontoincludenoele‐
ments.
H5Sselect_valid
h5sselect_valid_f
Verifiesthattheselectioniswithintheextent
ofthedataspace.
H5Sselect_hyperslab
h5sselect_hyperslab_f
Selectsahyperslabregiontoaddtothecur‐
rentselectedregion.
FunctionListing7‐4.Dataspaceselectionfunctions:points
CFunction
FortranSubroutine
Purpose
H5Sget_select_npoints
h5sget_select_npoints_f
Determinesthenumberofelementsina
dataspaceselection.
H5Sget_select_elem_npoints
h5sget_select_elem_npoints_f
Getsthenumberofelementpointsinthecur‐
rentselection.
H5Sget_select_elem_pointlist
h5sget_select_elem_pointlist_f
Getsthelistofelementpointscurrently
selected.
H5Sselect_elements
h5sselect_elements_f
Selectsarrayelementstobeincludedinthe
selectionforadataspace.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 268
7.3.DefinitionofDataspaceObjectsandtheDataspace
ProgrammingModel
ThissectionintroducesthenotionoftheHDF5dataspaceobjectandaprogrammingmodelforcreating
andworkingwithdataspaces.
7.3.1.DataspaceObjects
AnHDF5dataspaceisarequiredcomponentofanHDF5datasetorattribute.Adataspacedefinesthesize
andtheshapeofadataset’soranattribute’srawdata.Currently,HDF5supportsthefollowingtypesofthe
dataspaces:
• Scalardataspaces
• Simpledataspaces
•Nulldataspaces
Ascalardataspace,H5S_SCALAR,representsjustoneelement,ascalar.Notethatthedatatypeofthisone
elementmaybeverycomplex;examplewouldbeacompoundstructurewithmembersbeingofany
allowedHDF5datatype,includingmultidimensionalarrays,strings,andnestedcompoundstructures.By
convention,therankofascalardataspaceisalways0(zero);thinkofitgeometricallyasasingle,dimen‐
sionlesspoint,thoughthatpointmaybecomplex.
Asimpledataspace,H5S_SIMPLE,isamultidimensionalarrayofelements.Thedimensionalityofthe
dataspace(ortherankofthearray)isfixedandisdefinedatcreationtime.Thesizeofeachdimensioncan
growduringthelifetimeofthedataspacefromthecurrentsizeuptothemaximumsize.Boththecurrent
sizeandthemaximumsizearespecifiedatcreationtime.Thesizesofdimensionsatanyparticulartimein
thelifeofadataspacearecalledthecurrentdimensions,orthedataspaceextent.Theycanbequeried
alongwiththemaximumsizes.
Anulldataspace,H5S_NULL,containsnodataelements.Notethatnoselectionscanbeappliedtoanull
datasetasthereisnothingtoselect.
AsshownintheUMLdiagraminthefigurebelow,anHDF5simpledataspaceobjecthasthreeattributes:
therankornumberofdimensions;thecurrentsizes,expressedasanarrayoflengthrankwitheachele‐
mentofthearraydenotingthecurrentsizeofthecorrespondingdimension;andthemaximumsizes,
expressedasanarrayoflengthrankwitheachelementofthearraydenotingthemaximumsizeofthe
correspondingdimension.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 269
Note:Asimpledataspaceisdefinedbyitsrank,thecurrentsizeofeachdimension,andthemaximumsizeofeach
dimension.
Thesizeofacurrentdimensioncannotbegreaterthanthemaximumsize,whichcanbeunlimited,speci‐
fiedasH5S_UNLIMITED.NotethatwhiletheHDF5fileformatandlibraryimposenomaximumsizeonan
unlimiteddimension,practicallyspeakingitssizewillalwaysbelimitedtothebiggestintegeravailableon
theparticularsystembeingused.
Dataspacerankisrestrictedto32,thestandardlimitinContherankofanarray,inthecurrentimplemen‐
tationoftheHDF5Library.TheHDF5fileformat,ontheotherhand,allowsanyrankuptothemaximum
integervalueonthesystem,sothelibraryrestrictioncanberaisedinthefutureifhigherdimensionalityis
required.
NotethatmostofthetimeFortranapplicationscallingHDF5willworkwithdataspacesofranklessthanor
equaltoseven,sincesevenisthemaximumnumberofdimensionsinaFortranarray.Butdataspacerank
isnotlimitedtosevenforFortranapplications.
Thecurrentdimensionsofadataspace,alsoreferredtoasthedataspaceextent,definetheboundingbox
fordatasetelementsthatcanparticipateinI/Ooperations.
7.3.2.DataspaceProgrammingModel
TheprogrammingmodelforcreatingandworkingwithHDF5dataspacescanbesummarizedasfollows:
1. Createadataspace
2. Usethedataspacetocreateadatasetinthefileortodescribeadataarrayinmemory
3. ModifythedataspacetodefinedatasetelementsthatwillparticipateinI/Ooperations
4. Usethemodifieddataspacewhilereading/writingdatasetrawdataortocreatearegionrefer‐
ence
5. Closethedataspacewhennolongerneeded
Therestofthissectionwilladdresssteps1,2,and5oftheprogrammingmodel;steps3and4willbedis‐
cussedinlatersectionsofthischapter.
Figure7‐1.Asimpledataspace
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 270
7.3.2.1.CreatingaDataspace
AdataspacecanbecreatedbycallingtheH5Screatefunction(h5screate_finFortran).Sincethedefi‐
nitionofasimpledataspacerequiresthespecificationofdimensionality(orrank)andinitialandmaximum
dimensionsizes,theHDF5LibraryprovidesaconvenienceAPI,H5Screate_simple(h5screate_sim-
ple_f)tocreateasimpledataspaceinonestep.
ThefollowingexamplesillustratetheusageoftheseAPIs.
7.3.2.2.CreatingaScalarDataspace
AscalardataspaceiscreatedwiththeH5Screateortheh5screate_ffunction.
InC:
hid_t space_id;
. . .
space_id = H5Screate(H5S_SCALAR);
InFortran:
INTEGER(HID_T) :: space_id
. . .
CALL h5screate_f(H5S_SCALAR_F, space_id, error)
Asmentionedabove,thedataspacewillcontainonlyoneelement.Scalardataspacesareusedmoreoften
fordescribingattributesthathavejustonevalue.Forexample,theattributetemperaturewiththevalue
CelsiusisusedtoindicatethatthedatasetwiththisattributestorestemperaturevaluesusingtheCel‐
siusscale.
7.3.2.3.CreatingaNullDataspace
AnulldataspaceiscreatedwiththeH5Screateortheh5screate_ffunction.
InC:
hid_t space_id;
. . .
space_id = H5Screate(H5S_NULL);
InFortran:
(H5S_NULLnotyetimplementedinFortran.)
INTEGER(HID_T) :: space_id
. . .
CALL h5screate_f(H5S_NULL_F, space_id, error)
Asmentionedabove,thedataspacewillcontainnoelements.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 271
7.3.2.4.CreatingaSimpleDataspace
Let’sassumethatanapplicationwantstostoreatwo‐dimensionalarrayofdata,A(20,100).Duringthelife
oftheapplication,thefirstdimensionofthearraycangrowupto30;thereisnorestrictiononthesizeof
theseconddimension.Thefollowingstepsareusedtodeclareadataspaceforthedatasetinwhichthe
arraydatawillbestored.
InC:
hid_t space_id;
int rank = 2;
hsize_t current_dims[2] = {20, 100};
hsize_t max_dims[2] = {30, H5S_UNLIMITED};
. . .
space_id = H5Screate(H5S_SIMPLE);
H5Sset_extent_simple(space_id,rank,current_dims,max_dims);
InFortran:
INTEGER(HID_T) :: space_id
INTEGER :: rank = 2
INTEGER(HSIZE_T) :: current dims = /( 20, 100)/
INTEGER(HSIZE_T) :: max_dims = /(30, H5S_UNLIMITED_F)/
INTEGER error
. . .
CALL h5screate_f(H5S_SIMPLE_F, space_id, error)
CALL h5sset_extent_simple_f(space_id, rank, current_dims, max_dims,
error)
Alternatively,theconvenienceAPIsH5Screate_simple/h5screate_simple_fcanreplacethe
H5Screate/h5screate_fandH5Sset_extent_simple/h5sset_extent_simple_fcalls.
InC:
space_id = H5Screate_simple(rank, current_dims, max_dims);
InFortran:
CALL h5screate_simple_f(rank, current_dims, space_id, error, max_dims)
Inthisexample,adataspacewithcurrentdimensionsof20by100iscreated.Thefirstdimensioncanbe
extendedonlyupto30.Theseconddimension,however,isdeclaredunlimited;itcanbeextendedupto
thelargestavailableintegervalueonthesystem.
Notethatwhenthereisadifferencebetweenthecurrentdimensionsandthemaximumdimensionsofan
array,thenchunkingstoragemustbeused.Inotherwords,ifthenumberofdimensionsmaychangeover
thelifeofthedataset,thenchunkingmustbeused.Ifthearraydimensionsarefixed(ifthenumberofcur‐
rentdimensionsisequaltothemaximumnumberofdimensionswhenthedatasetiscreated),thencon‐
tiguousstoragecanbeused.Formoreinformation,see"DataTransfer"onpage 122.
Maximumdimensionscanbethesameascurrentdimensions.Insuchacase,thesizesofdimensionscan‐
notbechangedduringthelifeofthedataspaceobject.InC,NULLcanbeusedtoindicatetotheH5Scre-
ate_simpleandH5Sset_extent_simplefunctionsthatthemaximumsizesofalldimensionsarethe
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 272
sameasthecurrentsizes.InFortran,themaximumsizeparameterisoptionalforh5screate_simple_f
andcanbeomittedwhenthesizesarethesame.
InC:
space_id = H5Screate_simple(rank, current_dims, NULL);
InFortran:
CALL h5screate_f(rank, current_dims, space_id, error)
Thecreateddataspacewillhavecurrentandmaximumdimensionsof20and100correspondingly,andthe
sizesofthosedimensionscannotbechanged.
7.3.2.5.CversusFortranDataspaces
Dataspacedimensionsarenumberedfrom1torank.HDF5usesCstorageconventions,assumingthatthe
lastlisteddimensionisthefastest‐changingdimensionandthefirst‐listeddimensionistheslowestchang‐
ing.TheHDF5fileformatstoragelayoutspecificationadherestotheCconventionandtheHDF5Library
adherestothesameconventionwhenstoringdataspacedimensionsinthefile.ThisaffectshowCpro‐
gramsandtoolsinterpretdatawrittenfromFortranprogramsandviceversa.Theexamplebelowillus‐
tratestheissue.
WhenaFortranapplicationdescribesadataspacetostoreanarrayasA(20,100),itspecifiesthevalueof
thefirstdimensiontobe20andthesecondtobe100.SinceFortranstoresdatabycolumns,thefirst‐listed
dimensionwiththevalue20isthefastest‐changingdimensionandthelast‐listeddimensionwiththe
value100istheslowest‐changing.InordertoadheretotheHDF5storageconvention,theHDF5Fortran
wrappertransposesdimensions,sothefirstdimensionbecomesthelast.Thedataspacedimensions
storedinthefilewillbe100,20insteadof20,100inordertocorrectlydescribetheFortrandatathatis
storedin100columns,eachcontaining20elements.
WhenaFortranapplicationreadsthedataback,theHDF5Fortranwrappertransposesthedimensions
oncemore,returningthefirstdimensiontobe20andthesecondtobe100,describingcorrectlythesizes
ofthearraythatshouldbeusedtoreaddataintheFortranarrayA(20,100).
WhenaCapplicationreadsdataback,thedimensionswillcomeoutas100and20,correctlydescribing
thesizeofthearraytoreaddatainto,sincethedatawaswrittenas100recordsof20elementseach.
ThereforeCtoolssuchash5dumpandh5lsalwaysdisplaytransposeddimensionsandvaluesforthedata
writtenbyaFortranapplication.
ConsiderthefollowingsimpleexampleofequivalentC3x5andFortran5x3arrays.Asillustratedinthe
figurebelow,aCapplicationwillstorea3x52‐dimensionalarrayasthree5‐elementrows.Inorderto
storethesamedatainthesameorder,aFortranapplicationmustviewthearrayasa5x3arraywiththree
5‐elementcolumns.Thedataspaceofthisdataset,aswrittenfromFortran,willthereforebedescribedas
5x3intheapplicationbutstoredanddescribedinthefileaccordingtotheCconventionasa3x5array.
ThisensuresthatCandFortranapplicationswillalwaysreadthedataintheorderinwhichitwaswritten.
TheHDF5Fortraninterfacehandlesthistranspositionautomatically.
InC(fromh5_write.c):
#define NX 3 /* dataset dimensions */
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 273
#define NY 5
. . .
int data[NX][NY]; /* data to write */
. . .
/*
* Data and output buffer initialization.
*/
for (j = 0; j < NX; j++) {
for (i = 0; i < NY; i++)
data[j][i] = i + 1 + j*NY;
}
/*
* 1 2 3 4 5
* 6 7 8 9 10
* 11 12 13 14 15
*/
. . .
dims[0] = NX;
dims[1] = NY;
dataspace = H5Screate_simple(RANK, dims, NULL);
Formoreinformation,see"h5_write.c"onpage 300.
InFortran(fromh5_write.f90):
INTEGER, PARAMETER :: NX = 3
INTEGER, PARAMETER :: NY = 5
. . .
INTEGER(HSIZE_T), DIMENSION(2) :: dims = (/3,5/) ! Dataset dimensions
---
INTEGER :: data(NX,NY)
. . .
!
! Initialize data
!
do i = 1, NX
do j = 1, NY
data(i,j) = j + (i-1)*NY
enddo
enddo
!
! Data
!
! 1 2 3 4 5
! 6 7 8 9 10
! 11 12 13 14 15
. . .
CALL h5screate_simple_f(rank, dims, dspace_id, error)
Formoreinformation,see"h5_write.f90"onpage 302.
InFortran(fromh5_write_tr.f90):
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 274
INTEGER, PARAMETER :: NX = 3
INTEGER, PARAMETER :: NY = 5
. . .
INTEGER(HSIZE_T), DIMENSION(2) :: dims = (/NY, NX/) ! Dataset dimensions
. . .
!
! Initialize data
!
do i = 1, NY
do j = 1, NX
data(i,j) = i + (j-1)*NY
enddo
enddo
!
! Data
!
! 1 6 11
! 2 7 12
! 3 8 13
! 4 9 14
! 5 10 15
. . .
CALL h5screate_simple_f(rank, dims, dspace_id, error)
Formoreinformation,see"h5_write_tr.f90"onpage 304.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 275
Note:TheHDF5Librarystoresarraysalongthefastest‐changingdimension.Thisapproachisoftenreferredtoas
being“inCorder.”C,C++,andJavaworkwitharraysinrow‐majororder.Inotherwords,therow,orthelastdimen‐
sion,isthefastest‐changingdimension.Fortran,ontheotherhand,handlesarraysincolumn‐majorordermakingthe
column,orthefirstdimension,thefastest‐changingdimension.Therefore,theCandFortranarraysillustratedinthe
topportionofthisfigurearestoredidenticallyinanHDF5file.Thisensuresthatdatawrittenbyanylanguagecanbe
meaningfullyread,interpreted,andmanipulatedbyanyother.
7.3.2.6.FindingDataspaceCharacteristics
TheHDF5LibraryprovidesseveralAPIsdesignedtoquerythecharacteristicsofadataspace.
ThefunctionH5Sis_simple(h5sis_simple_f)returnsinformationaboutthetypeofadataspace.
Thisfunctionisrarelyusedandcurrentlysupportsonlysimpleandscalardataspaces.
Tofindoutthedimensionality,orrank,ofadataspace,useH5Sget_simple_extent_ndims(h5sget_-
simple_extent_ndims_f).H5Sget_simple_extent_dimscanalsobeusedtofindouttherank.See
theexamplebelow.Ifbothfunctionsreturn0forthevalueofrank,thenthedataspaceisscalar.
AdatasetstoredbyaCprogramina3x5array:
ThesamedatasetstoredbyaFortranprogramina5x3array:
ThefirstdatasetaboveaswrittentoanHDF5filefromCortheseconddatasetaboveaswritten
fromFortran:
ThefirstdatasetaboveaswrittentoanHDF5filefromFortran:
Figure7‐2.ComparingCandFortrandataspaces
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 276
Toquerythesizesofthecurrentandmaximumdimensions,useH5Sget_simple_extent_dims
(h5sget_simple_extent_dims_f).
Thefollowingexampleillustratesqueryingtherankanddimensionsofadataspaceusingthesefunctions.
InC:
hid_t space_id;
int rank;
hsize_t *current_dims;
hsize_t *max_dims;
---------
rank=H5Sget_simple_extent_ndims(space_id);
(or rank=H5Sget_simple_extent_dims(space_id, NULL, NULL);)
current_dims= (hsize_t)malloc(rank*sizeof(hsize_t));
max_dims=(hsize_t)malloc(rank*sizeof(hsize_t));
H5Sget_simple_extent_dims(space_id, current_dims, max_dims);
Print values here for the previous example
7.4.DataspacesandDataTransfer
ReadandwriteoperationstransferdatabetweenanHDF5fileondiskandinmemory.Theshapethatthe
arraydatatakesinthefileandinmemorymaybethesame,butHDF5alsoallowsuserstheabilitytorep‐
resentdatainmemoryinadifferentshapethaninthefile.Iftheshapeofanarrayinthefileandinmem‐
orywillbethesame,thenthesamedataspacedefinitioncanbeusedforboth.Iftheshapeofanarrayin
memoryneedstobedifferentthantheshapeinthefile,thenthedataspacedefinitionfortheshapeofthe
arrayinmemorycanbechanged.Duringareadoperation,thearraywillbereadintothedifferentshape
inmemory,andduringawriteoperation,thearraywillbewrittentothefileintheshapespecifiedbythe
dataspaceinthefile.Theonlyqualificationisthatthenumberofelementsreadorwrittenmustbethe
sameinboththesourceandthedestinationdataspaces.
Itemainthefigurebelowshowsasimpleexampleofareadoperationinwhichthedataisstoredasa3by
4arrayinthefile(itemb)ondisk,buttheprogramwantsittobea4by3arrayinmemory.Thisisaccom‐
plishedbysettingthememorydataspacetodescribethedesiredmemorylayout,asinitemc.Theread
operationreadsthedatainthefilearrayintothememoryarray.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 277
Boththesourceanddestinationarestoredascontiguousblocksofstoragewiththeelementsintheorder
specifiedbythedataspace.Thefigureaboveshowsonewaytheelementsmightbeorganized.Initema,
theelementsarestoredas3blocksof4elements.Thedestinationisanarrayof12elementsinmemory
(seeitemc).Asthefiguresuggests,thetransferreadsthediskblocksintoamemorybuffer(seeitemb),
Figure7‐3.Datalayoutbeforeandafterareadoperation
Figure7‐4.Movingdatafromdisktomemory
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 278
andthenwritestheelementstothecorrectlocationsinmemory.Asimilarprocessoccursinreversewhen
dataiswrittentodisk.
7.4.1.DataSelection
Inadditiontorearrangingdata,thetransfermayselectthedataelementsfromthesourceanddestina‐
tion.
Dataselectionisimplementedbycreatingadataspaceobjectthatdescribestheselectedelements(within
thehyperrectangle)ratherthanthewholearray.Twodataspaceobjectswithselectionscanbeusedin
datatransferstoreadselectedelementsfromthesourceandwriteselectedelementstothedestination.
Whendataistransferredusingthedataspaceobject,onlytheselectedelementswillbetransferred.
ThiscanbeusedtoimplementpartialI/O,including:
•Sub‐setting‐readingpartofalargedataset
• Sampling‐readingselectedelements(forexample,everysecondelement)ofadataset
• Scatter‐gather‐readnon‐contiguouselementsintocontiguouslocations(gather)orreadcontigu‐
ouselementsintonon‐contiguouslocations(scatter)orboth
Touseselections,thefollowingstepsarefollowed:
1. Getordefinethedataspaceforthesourceanddestination
2. Specifyoneormoreselectionsforsourceanddestinationdataspaces
3. Transferdatausingthedataspaceswithselections
Aselectioniscreatedbyapplyingoneormoreselectionstoadataspace.Aselectionmayoverrideany
otherselections(H5T_SELECT_SET)ormaybe“Ored”withpreviousselectionsonthesamedataspace
(H5T_SELECT_OR).Inthelattercase,theresultingselectionistheunionoftheselectionandallprevi‐
ouslyselectedselections.Arbitrarysetsofpointsfromadataspacecanbeselectedbyspecifyinganappro‐
priatesetofselections.
Twoselectionsareusedindatatransfer,sothesourceanddestinationmustbecompatible,asdescribed
below.
Therearetwoformsofselection,hyperslabandpoint.Aselectionmustbeeitherapointselectionoraset
ofhyperslabselections.Selectionscannotbemixed.
Thedefinitionofaselectionwithinadataspace,notthedataintheselection,cannotbesavedtothefile
unlesstheselectiondefinitionissavedasaregionreference.Formoreinformation,see"Referencesto
DatasetRegions"onpage 292.
7.4.1.1.HyperslabSelection
Ahyperslabisaselectionofelementsfromahyperrectangle.AnHDF5hyperslabisarectangularpattern
definedbyfourarrays.Thefourarraysaresummarizedinthetablebelow.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 279
Theoffsetdefinestheoriginofthehyperslabintheoriginaldataspace.
Thestrideisthenumberofelementstoincrementbetweenselectedelements.Astrideof‘1’iseveryele‐
ment,astrideof‘2’iseverysecondelement,etc.Notethattheremaybeadifferentstrideforeachdimen‐
sionofthedataspace.Thedefaultstrideis1.
Thecountisthenumberofelementsinthehyperslabselection.Whenthestrideis1,theselectionisa
hyperrectanglewithacornerattheoffsetandsizecount[0]bycount[1]by....Whenstrideisgreaterthan
one,thehyperslabboundedbytheoffsetandthecornersdefinedbystride[n]*count[n].
Theblockisacountonthenumberofrepetitionsofthehyperslab.Thedefaultblocksizeis‘1’,whichis
onehyperslab.Ablockof2wouldbetwohyperslabsinthatdimension,withthesecondstartingatoff‐
set[n]+(count[n]*stride[n])+1.
Ahyperslabcanbeusedtoaccessasub‐setofalargedataset.Thefigurebelowshowsanexampleofa
hyperslabthatreadsarectanglefromthemiddleofalargertwodimensionalarray.Thedestinationisthe
sameshapeasthesource.
Hyperslabscanbecombinedtoselectcomplexregionsofthesourceanddestination.Thefigurebelow
showsanexampleofatransferfromonenon‐rectangularregionintoanothernon‐rectangularregion.The
sourceisdefinedastheunionoftwohyperslabs,andthedestinationistheunionofthreehyperslabs.
Table7‐1.Hyperslabelements
Parameter Description
Offset Thestartinglocationforthehyperslab.
Stride Thenumberofelementstoseparateeachelementorblocktobeselected.
Count Thenumberofelementsorblockstoselectalongeachdimension.
Block Thesizeoftheblockselectedfromthedataspace.
Figure7‐5.Accessasub‐setofdatawithahyperslab
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 280
Hyperslabsmayalsobeusedtocollectorscatterdatafromregularpatterns.Thefigurebelowshowsan
examplewherethesourceisarepeatingpatternofblocks,andthedestinationisasingle,onedimensional
array.
7.4.1.2.SelectPoints
Thesecondtypeofselectionisanarrayofpointssuchascoordinates.Essentially,thisselectionisalistof
allthepointstoinclude.Thefigurebelowshowsanexampleofatransferofsevenelementsfromatwo
dimensionaldataspacetoathreedimensionaldataspaceusingapointselectiontospecifythepoints.
Figure7‐6.Buildcomplexregionswithhyperslabunions
Figure7‐7.Usehyperslabstocombineordispersedata
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 281
7.4.1.3.RulesforDefiningSelections
Aselectionmusthavethesamenumberofdimensions(rank)asthedataspaceitisappliedto,althoughit
mayselectfromonlyasmallregionsuchasaplanefroma3Ddataspace.Selectionsdonotaffectthe
extentofthedataspace,theselectionmaybelargerthanthedataspace.Theboundariesofselectionsare
reconciledwiththeextentatthetimeofthedatatransfer.
7.4.1.4.DataTransferwithSelections
Adatatransfer(readorwrite)withselectionsisthesameasanyreadorwrite,exceptthesourceanddes‐
tinationdataspacehavecompatibleselections.
Duringthedatatransfer,thefollowingstepsareexecutedbythelibrary:
•Thesourceanddestinationdataspacesarecheckedtoassurethattheselectionsarecompatible.
•Eachselectionmustbewithinthecurrentextentofthedataspace.Aselectionmaybe
definedtoextendoutsidethecurrentextentofthedataspace,butthedataspacecannotbe
accessediftheselectionisnotvalidatthetimeoftheaccess.
•Thetotalnumberofpointsselectedinthesourceanddestinationmustbethesame.Note
thatthedimensionalityofthesourceanddestinationcanbedifferent(forexample,the
sourcecouldbe2D,thedestination1Dor3D),andtheshapecanbedifferent,butthenum‐
berofelementsselectedmustbethesame.
•Thedataistransferred,elementbyelement.
Selectionshaveaniterationorderforthepointsselected,whichcanbeanypermutationofthedimen‐
sionsinvolved(defaultingto‘C’arrayorder)oraspecificorderfortheselectedpoints,forselectionscom‐
posedofsinglearrayelementswithH5Sselect_elements.
Figure7‐8.Pointselection
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 282
Theelementsoftheselectionsaretransferredinrow‐major,orCorder.Thatis,itisassumedthatthefirst
dimensionvariesslowest,thesecondnextslowest,andsoforth.Forhyperslabselections,theordercan
beanypermutationofthedimensionsinvolved(defaultingto‘C’arrayorder).Whenmultiplehyperslabs
arecombined,thehyperslabsarecoalescedintocontiguousreadsandwrites.
Inthecaseofpointselections,thepointsarereadandwrittenintheorderspecified.
7.4.2.ProgrammingModel
7.4.2.1.SelectingHyperslabs
Supposewewanttoreada3x4hyperslabfromadatasetinafilebeginningattheelement<1,2>inthe
dataset,andreaditintoa7x7x3arrayinmemory.Seethefigurebelow.Inordertodothis,wemustcre‐
ateadataspacethatdescribestheoverallrankanddimensionsofthedatasetinthefileaswellastheposi‐
tionandsizeofthehyperslabthatweareextractingfromthatdataset.
Figure7‐9.Selectingahyperslab
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 283
Thecodeinthefirstexamplebelowillustratestheselectionofthehyperslabinthefiledataspace.Thesec‐
ondexamplebelowshowsthedefinitionofthedestinationdataspaceinmemory.Sincethein‐memory
dataspacehasthreedimensions,thehyperslabisanarraywiththreedimensionswiththelastdimension
being1:<3,4,1>.Thethirdexamplebelowshowsthereadusingthesourceanddestinationdataspaces
withselections.
/*
* get the file dataspace.
*/
dataspace = H5Dget_space(dataset); /* dataspace */
/* identifier */
/*
* Define hyperslab in the dataset.
*/
offset[0] = 1;
offset[1] = 2;
count[0] = 3;
count[1] = 4;
status = H5Sselect_hyperslab(dataspace, H5S_SELECT_SET,
offset, NULL, count, NULL);
CodeExample7‐1.Selectingahyperslab
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 284
7.4.2.2.ExamplewithStridesandBlocks
Consideran8x12dataspaceintowhichwewanttowriteeight3x2blocksinatwodimensionalarray
fromasourcedataspaceinmemorythatisa50‐elementonedimensionalarray.Seethefigurebelow.
/*
* Define memory dataspace.
*/
dimsm[0] = 7;
dimsm[1] = 7;
dimsm[2] = 3;
memspace = H5Screate_simple(3,dimsm,NULL);
/*
* Define memory hyperslab.
*/
offset_out[0] = 3;
offset_out[1] = 0;
offset_out[2] = 0;
count_out[0] = 3;
count_out[1] = 4;
count_out[2] = 1;
status = H5Sselect_hyperslab(memspace, H5S_SELECT_SET,
offset_out, NULL, count_out, NULL);
CodeExample7‐2.Definingthedestinationmemory
ret = H5Dread(dataset, H5T_NATIVE_INT, memspace,
dataspace, H5P_DEFAULT, data);
CodeExample7‐3.Asamplereadspecifyingsourceanddestinationdataspaces
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 285
Theexamplebelowshowscodetowrite48elementsfromtheonedimensionalarraytothefiledataset
startingwiththesecondelementinvector.Thedestinationhyperslabhasthefollowingparameters:off‐
set=(0,1),stride=(4,3),count=(2,4),block=(3,2).Thesourcehastheparameters:offset=(1),stride=(1),
count=(48),block=(1).Aftertheseoperations,thefiledataspacewillhavethevaluesshowninitembin
thefigureabove.Noticethatthevaluesareinsertedinthefiledatasetinrow‐majororder.
Figure7‐10.Writefromaonedimensionalarraytoatwodimensionalarray
/* Select hyperslab for the dataset in the file, using
* 3 x 2 blocks, (4,3) stride (2,4) count starting at
* the position (0,1).
*/
offset[0] = 0; offset[1] = 1;
stride[0] = 4; stride[1] = 3;
count[0] = 2; count[1] = 4;
block[0] = 3; block[1] = 2;
ret = H5Sselect_hyperslab(fid, H5S_SELECT_SET, offset,
stride, count, block);
CodeExample7‐4.Writefromaonedimensionalarraytoatwodimensionalarray
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 286
7.4.2.3.SelectingaUnionofHyperslabs
TheHDF5Libraryallowstheusertoselectaunionofhyperslabsandwriteorreadtheselectioninto
anotherselection.Theshapesofthetwoselectionsmaydiffer,butthenumberofelementsmustbe
equal.
/*
* Create dataspace for the first dataset.
*/
mid1 = H5Screate_simple(MSPACE1_RANK, dim1, NULL);
/*
* Select hyperslab.
* We will use 48 elements of the vector buffer starting
* at the second element. Selected elements are
* 1 2 3 . . . 48
*/
offset[0] = 1;
stride[0] = 1;
count[0] = 48;
block[0] = 1;
ret = H5Sselect_hyperslab(mid1, H5S_SELECT_SET, offset,
stride, count, block);
/*
* Write selection from the vector buffer to the dataset
* in the file.
*/
ret = H5Dwrite(dataset, H5T_NATIVE_INT, midd1, fid,
H5P_DEFAULT, vector)
CodeExample7‐4.Writefromaonedimensionalarraytoatwodimensionalarray
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 287
Thefigureaboveshowsthetransferofaselectionthatistwooverlappinghyperslabsfromthedatasetinto
aunionofhyperslabsinthememorydataset.Notethatthedestinationdatasethasadifferentshapefrom
thesourcedataset.Similarly,theselectioninthememorydatasetcouldhaveadifferentshapethanthe
selectedunionofhyperslabsintheoriginalfile.Forsimplicity,theselectionisthatsameshapeatthedes‐
tination.
Toimplementthistransfer,itisnecessaryto:
1. Getthesourcedataspace
2. Defineonehyperslabselectionforthesource
3. Defineasecondhyperslabselection,unionedwiththefirst
4. Getthedestinationdataspace
5. Defineonehyperslabselectionforthedestination
6. Defineasecondhyperslabseletion,unionedwiththefirst
7. Executethedatatransfer(H5DreadorH5Dwrite)usingthesourceanddestinationdataspaces
Theexamplebelowshowsexamplecodetocreatetheselectionsforthesourcedataspace(thefile).The
firsthyperslabissize3x4andtheleftuppercornerattheposition(1,2).Thehyperslabisasimplerectan‐
Figure7‐11.Transferringhyperslabunions
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 288
gle,sothestrideandblockare1.Thesecondhyperslabis6x5attheposition(2,4).Thesecondselection
isaunionwiththefirsthyperslab(H5S_SELECT_OR).
Theexamplebelowshowsexamplecodetocreatetheselectionforthedestinationinmemory.Thesteps
aresimilar.Inthisexample,thehyperslabsarethesameshape,butlocatedindifferentpositionsinthe
dataspace.Thefirsthyperslabis3x4andstartsat(0,0),andthesecondis6x5andstartsat(1,2).
Finally,theH5Dreadcalltransferstheselecteddatafromthefiledataspacetotheselectioninmemory.
Inthisexample,thesourceanddestinationselectionsaretwooverlappingrectangles.Ingeneral,any
numberofrectanglescanbeOR’ed,andtheydonothavetobecontiguous.Theorderoftheselections
doesnotmatter,butthefirstshoulduseH5S_SELECT_SET;subsequentselectionsareunionedusing
H5S_SELECT_OR.
Itisimportanttoemphasizethatthesourceanddestinationdonothavetobethesameshape(ornumber
ofrectangles).Aslongasthetwoselectionshavethesamenumberofelements,thedatacanbetrans‐
ferred.
fid = H5Dget_space(dataset);
/*
* Select first hyperslab for the dataset in the file.
*/
offset[0] = 1; offset[1] = 2;
block[0] = 1; block[1] = 1;
stride[0] = 1; stride[1] = 1;
count[0] = 3; count[1] = 4;
ret = H5Sselect_hyperslab(fid, H5S_SELECT_SET, offset,
stride, count, block);
/*
* Add second selected hyperslab to the selection.
*/
offset[0] = 2; offset[1] = 4;
block[0] = 1; block[1] = 1;
stride[0] = 1; stride[1] = 1;
count[0] = 6; count[1] = 5;
ret = H5Sselect_hyperslab(fid, H5S_SELECT_OR, offset,
stride, count, block);
CodeExample7‐5.Selectsourcehyperslabs
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 289
7.4.2.4.SelectingaListofIndependentPoints
ItisalsopossibletospecifyalistofelementstoreadorwriteusingthefunctionH5Sselect_elements.
Theprocedureissimilartohyperslabselections.
1. Getthesourcedataspace
2. Settheselectedpoints
3. Getthedestinationdataspace
4. Settheselectedpoints
5. Transferthedatausingthesourceanddestinationdataspaces
Thefigurebelowshowsanexamplewherefourvaluesaretobewrittentofourseparatepointsinatwo
dimensionaldataspace.Thesourcedataspaceisaonedimensionalarraywiththevalues53,59,61,67.
Thedestinationdataspaceisan8x12array.Theelementsaretobewrittentothepoints(0,0),(3,3),(3,5),
/*
* Create memory dataspace.
*/
mid = H5Screate_simple(MSPACE_RANK, mdim, NULL);
/*
* Select two hyperslabs in memory. Hyperslabs has the
* same size and shape as the selected hyperslabs for
* the file dataspace.
*/
offset[0] = 0; offset[1] = 0;
block[0] = 1; block[1] = 1;
stride[0] = 1; stride[1] = 1;
count[0] = 3; count[1] = 4;
ret = H5Sselect_hyperslab(mid, H5S_SELECT_SET, offset,
stride, count, block);
offset[0] = 1; offset[1] = 2;
block[0] = 1; block[1] = 1;
stride[0] = 1; stride[1] = 1;
count[0] = 6; count[1] = 5;
ret = H5Sselect_hyperslab(mid, H5S_SELECT_OR, offset,
stride, count, block);
ret = H5Dread(dataset, H5T_NATIVE_INT, mid, fid,
H5P_DEFAULT, matrix_out);
CodeExample7‐6.Selectdestinationhyperslabs
HDF5DataspacesandPartialI/O HDF5User’sGuide
290 TheHDFGroup
and(5,6).Inthisexample,thesourcedoesnotrequireaselection.Theexamplebelowthefigureshows
examplecodetoimplementthistransfer.
Apointselectionliststheexactpointstobetransferredandtheordertheywillbetransferred.Thesource
anddestinationarerequiredtohavethesamenumberofelements.Apointselectioncanbeusedwitha
hyperslab(forexample,thesourcecouldbeapointselectionandthedestinationahyperslab,orvice
versa),solongasthenumberofelementsselectedarethesame.
Figure7‐12.Writedatatoseparatepoints
hsize_t dim2[] = {4};
int values[] = {53, 59, 61, 67};
/* Array to store selected points from the
* file dataspace
*/
hssize_t coord[4][2];
/*
* Create dataspace for the second dataset.
*/
CodeExample7‐7.Writedatatoseparatepoints
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 291
7.4.2.5.CombinationsofSelections
Selectionsareaveryflexiblemechanismforreorganizingdataduringadatatransfer.Withdifferentcom‐
binationsofdataspacesandselections,itispossibletoimplementmanykindsofdatatransfersincluding
sub‐setting,sampling,andreorganizingthedata.Thetablebelowgivessomeexamplecombinationsof
sourceanddestination,andtheoperationstheyimplement.
mid2 = H5Screate_simple(1, dim2, NULL);
/*
* Select sequence of NPOINTS points in the file
* dataspace.
*/
coord[0][0] = 0; coord[0][1] = 0;
coord[1][0] = 3; coord[1][1] = 3;
coord[2][0] = 3; coord[2][1] = 5;
coord[3][0] = 5; coord[3][1] = 6;
ret = H5Sselect_elements(fid, H5S_SELECT_SET, NPOINTS,
(const hssize_t **)coord);
ret = H5Dwrite(dataset, H5T_NATIVE_INT, mid2, fid,
H5P_DEFAULT, values);
Table7‐2.Selectionoperations
Source Destination Operation
All All Copywholearray
All All(differentshape) Copyandreorganizearray
Hyperslab All Sub‐set
Hyperslab Hyperslab(sameshape) Selection
Hyperslab Hyperslab(differentshape) Selectandrearrange
Hyperslabwithstrideorblock Allorhyperslabwithstride
1
Sub‐sample,scatter
Hyperslab Points Scatter
Points Hyperslaborall Gather
CodeExample7‐7.Writedatatoseparatepoints
HDF5DataspacesandPartialI/O HDF5User’sGuide
292 TheHDFGroup
7.5.DataspaceSelectionOperationsandDataTransfer
Thissectionisunderconstruction.
7.6.ReferencestoDatasetRegions
Anotheruseofselectionsistostoreareferencetoaregionofadataset.AnHDF5objectreferenceobject
isapointertoanobject(dataset,group,orcommitteddatatype)inthefile.Aselectioncanbeusedtocre‐
ateapointertoasetofselectedelementsofadataset,calledaregionreference.Theselectioncanbe
eitherapointselectionorahyperslabselection.
AregionreferenceisanobjectmaintainedbytheHDF5Library.Theregionreferencecanbestoredina
datasetorattribute,andthenread.Thedatasetorattributeisdefinedtohavethespecialdatatype,
H5T_STD_REF_DSETREG.
Todiscovertheelementsand/orreadthedata,theregionreferencecanbedereferenced.TheH5Rde-
frerencecallreturnsanidentifierforthedataset,andthentheselecteddataspacecanberetrievedwith
H5Rget_selectcall.Theselecteddataspacecanbeusedtoreadtheselecteddataelements.
Formoreinformation,see"Reference"onpage 234.
7.6.1.ExampleUsesforRegionReferences
Regionreferencesareusedtoimplementstoredpointerstodatawithinadataset.Forexample,features
inalargedatasetmightbeindexedbyatable.Seethefigurebelow.ThistablecouldbestoredasanHDF5
datasetwithacompounddatatype,forexample,withafieldforthenameofthefeatureandaregionref‐
erencetopointtothefeatureinthedataset.Seethesecondfigurebelow.
Points Points(same) Selection
Points Points(different) Reorderpoints
Table7‐2.Selectionoperations
Source Destination Operation
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 293
Figure7‐13.Featuresindexedbyatable
HDF5DataspacesandPartialI/O HDF5User’sGuide
294 TheHDFGroup
7.6.2.CreatingReferencestoRegions
Tocreatearegionreference:
1. Createoropenthedatasetthatcontainstheregion
2. Getthedataspaceforthedataset
3. Defineaselectionthatspecifiestheregion
4. Createaregionreferenceusingthedatasetanddataspacewithselection
5. Writetheregionreference(s)tothedesireddatasetorattribute
Figure7‐14.Storingthetablewithacompounddatatype
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 295
Thefigurebelowshowsadiagramofafilewiththreedatasets.DatasetD1andD2aretwodimensional
arraysofintegers.DatasetR1isaonedimensionalarrayofreferencestoregionsinD1andD2.Theregions
canbeanyvalidselectionofthedataspaceofthetargetdataset.
Note:Inthefigureabove,R1isa1Darrayofregionpointers;eachpointerreferstoaselectioninonedataset.
Theexamplebelowshowscodetocreatethearrayofregionreferences.Thereferencesarecreatedinan
arrayoftypehdset_reg_ref_t.Eachregionisdefinedasaselectiononthedataspaceofthedataset,
andareferenceiscreatedusingH5Rcreate().ThecalltoH5Rcreate()specifiesthefile,dataset,and
thedataspacewithselection.
Figure7‐15.Afilewiththreedatasets
HDF5DataspacesandPartialI/O HDF5User’sGuide
296 TheHDFGroup
/* create an array of 4 region references */
hdset_reg_ref_t ref[4];
/*
* Create a reference to the first hyperslab in the first
* Dataset.
*/
offset[0] = 1; offset[1] = 1;
count[0] = 3; count[1] = 2;
status = H5Sselect_hyperslab(space_id, H5S_SELECT_SET,
offset, NULL, count, NULL);
status = H5Rcreate(&ref[0], file_id, "D1",
H5R_DATASET_REGION, space_id);
/*
* The second reference is to a union of hyperslabs in
* the first Dataset
*/
offset[0] = 5; offset[1] = 3;
count[0] = 1; count[1] = 4;
status = H5Sselect_none(space_id);
status = H5Sselect_hyperslab(space_id, H5S_SELECT_SET,
offset, NULL, count, NULL);
offset[0] = 6; offset[1] = 5;
count[0] = 1; count[1] = 2;
status = H5Sselect_hyperslab(space_id, H5S_SELECT_OR,
offset, NULL, count, NULL);
status = H5Rcreate(&ref[1], file_id, "D1",
H5R_DATASET_REGION, space_id);
/*
* the fourth reference is to a selection of points in
* the first Dataset
*/
CodeExample7‐8.Createanarrayofregionreferences
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 297
Whenallthereferencesarecreated,thearrayofreferencesiswrittentothedatasetR1.Thedatasetis
declaredtohavedatatypeH5T_STD_REF_DSETREG.Seetheexamplebelow.
status = H5Sselect_none(space_id);
coord[0][0] = 4; coord[0][1] = 4;
coord[1][0] = 2; coord[1][1] = 6;
coord[2][0] = 3; coord[2][1] = 7;
coord[3][0] = 1; coord[3][1] = 5;
coord[4][0] = 5; coord[4][1] = 8;
status = H5Sselect_elements(space_id, H5S_SELECT_SET,
num_points, (const hssize_t **)coord);
status = H5Rcreate(&ref[3], file_id, "D1",
H5R_DATASET_REGION, space_id);
/*
* the third reference is to a hyperslab in the second
* Dataset
*/
offset[0] = 0; offset[1] = 0;
count[0] = 4; count[1] = 6;
status = H5Sselect_hyperslab(space_id2, H5S_SELECT_SET,
offset, NULL, count, NULL);
status = H5Rcreate(&ref[2], file_id, "D2",
H5R_DATASET_REGION, space_id2);
Hsize_t dimsr[1];
dimsr[0] = 4;
/*
* Dataset with references.
*/
spacer_id = H5Screate_simple(1, dimsr, NULL);
dsetr_id = H5Dcreate(file_id, "R1", H5T_STD_REF_DSETREG,
spacer_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
/*
* Write dataset with the references.
*/
status = H5Dwrite(dsetr_id, H5T_STD_REF_DSETREG, H5S_ALL,
H5S_ALL, H5P_DEFAULT, ref);
CodeExample7‐9.Writethearrayofreferencestoadataset
CodeExample7‐8.Createanarrayofregionreferences
HDF5DataspacesandPartialI/O HDF5User’sGuide
298 TheHDFGroup
Whencreatingregionreferences,thefollowingrulesareenforced.
•Theselectionmustbeavalidselectionforthetargetdataset,justaswhentransferringdata
•Thedatasetmustexistinthefilewhenthereferenceiscreated(H5Rcreate)
•Thetargetdatasetmustbeinthesamefileasthestoredreference
7.6.3.ReadingReferencestoRegions
Toretrievedatafromaregionreference,thereferencemustbereadfromthefile,andthenthedatacan
beretrieved.Thestepsare:
1. Openthedatasetorattributecontainingthereferenceobjects
2. Readthereferenceobject(s)
3. Foreachregionreference,getthedataset(H5R_dereference)anddataspace(H5Rget_space)
4. Usethedataspaceanddatatypetodiscoverwhatspaceisneededtostorethedata,allocatethe
correctstorageandcreateadataspaceanddatatypetodefinethememorydatalayout
Theexamplebelowshowscodetoreadanarrayofregionreferencesfromadataset,andthenreadthe
datafromthefirstselectedregion.Notethattheregionreferencehasinformationthatrecordsthedata‐
set(withinthefile)andtheselectiononthedataspaceofthedataset.Afterdereferencingtheregionsref‐
erence,thedatatype,numberofpoints,andsomeaspectsoftheselectioncanbediscovered.(Foraunion
ofhyperslabs,itmaynotbepossibletodeterminetheexactsetofhyperslabsthathasbeencombined.)
Thetablebelowthecodeexampleshowstheinquiryfunctions.
Whenreadingdatafromaregionreference,thefollowingrulesareenforced:
•Thetargetdatasetmustbepresentandaccessibleinthefile
•Theselectionmustbeavalidselectionforthedataset
dsetr_id = H5Dopen (file_id, "R1", H5P_DEFAULT);
status = H5Dread(dsetr_id, H5T_STD_REF_DSETREG, H5S_ALL,
H5S_ALL, H5P_DEFAULT, ref_out);
/*
* Dereference the first reference.
* 1) get the dataset (H5Rdereference)
* 2) get the selected dataspace (H5Rget_region)
*/
CodeExample7‐10.Readanarrayofregionreferences;readfromthefirstselection
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 299
dsetv_id = H5Rdereference(dsetr_id, H5R_DATASET_REGION,
&ref_out[0]);
space_id = H5Rget_region(dsetr_id, H5R_DATASET_REGION,
&ref_out[0]);
/*
* Discover how many points and shape of the data
*/
ndims = H5Sget_simple_extent_ndims(space_id);
H5Sget_simple_extent_dims(space_id,dimsx,NULL);
/*
* Read and display hyperslab selection from the dataset.
*/
dimsy[0] = H5Sget_select_npoints(space_id);
spacex_id = H5Screate_simple(1, dimsy, NULL);
status = H5Dread(dsetv_id, H5T_NATIVE_INT, H5S_ALL,
space_id, H5P_DEFAULT, data_out);
printf("Selected hyperslab: ");
for (i = 0; i < 8; i++)
{
printf("\n");
for (j = 0; j < 10; j++)
printf("%d ", data_out[i][j]);
}
printf("\n");
Table7‐3.Theinquiryfunctions
Function Information
H5Sget_select_npoints Thenumberofelementsintheselection
(hyperslaborpointselection).
H5Sget_select_bounds Theboundingboxthatenclosestheselected
points(hyperslaborpointselection).
H5Sget_select_hyper_nblocks Thenumberofblocksintheselection.
H5Sget_select_hyper_blocklist Alistoftheblocksintheselection.
H5Sget_select_elem_npoints Thenumberofpointsintheselection.
H5Sget_select_elem_pointlist Thepoints.
CodeExample7‐10.Readanarrayofregionreferences;readfromthefirstselection
HDF5DataspacesandPartialI/O HDF5User’sGuide
300 TheHDFGroup
7.7.SamplePrograms
Thissectioncontainsthefullprogramsfromwhichseveralofthecodeexamplesinthischapterwere
derived.Theh5dumpoutputfromtheprogram’soutputfileimmediatelyfollowseachprogram.
7.7.1.h5_write.c
----------
#include "hdf5.h"
#define H5FILE_NAME "SDS.h5"
#define DATASETNAME "C Matrix"
#define NX 3 /* dataset dimensions */
#define NY 5
#define RANK 2
int
main (void)
{
hid_t file, dataset; /* file and dataset identifiers */
hid_t datatype, dataspace; /* identifiers */
hsize_t dims[2]; /* dataset dimensions */
herr_t status;
int data[NX][NY]; /* data to write */
int i, j;
/*
* Data and output buffer initialization.
*/
for (j = 0; j < NX; j++) {
for (i = 0; i < NY; i++)
data[j][i] = i + 1 + j*NY;
}
/*
* 1 2 3 4 5
* 6 7 8 9 10
* 11 12 13 14 15
*/
/*
* Create a new file using H5F_ACC_TRUNC access,
* default file creation properties, and default file
* access properties.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 301
*/
file = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/*
* Describe the size of the array and create the data space for fixed
* size dataset.
*/
dims[0] = NX;
dims[1] = NY;
dataspace = H5Screate_simple(RANK, dims, NULL);
/*
* Create a new dataset within the file using defined dataspace and
* datatype and default dataset creation properties.
*/
dataset = H5Dcreate(file, DATASETNAME, H5T_NATIVE_INT, dataspace,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
/*
* Write the data to the dataset using default transfer properties.
*/
status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data);
/*
* Close/release resources.
*/
H5Sclose(dataspace);
H5Dclose(dataset);
H5Fclose(file);
return 0;
}
SDS.out
-------
HDF5 "SDS.h5" {
GROUP "/" {
DATASET "C Matrix" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 3, 5 ) / ( 3, 5 ) }
DATA {
1, 2, 3, 4, 5,
6, 7, 8, 9, 10,
11, 12, 13, 14, 15
}
}
HDF5DataspacesandPartialI/O HDF5User’sGuide
302 TheHDFGroup
}
}
7.7.2.h5_write.f90
------------
PROGRAM DSETEXAMPLE
USE HDF5 ! This module contains all necessary modules
IMPLICIT NONE
CHARACTER(LEN=7), PARAMETER :: filename = "SDSf.h5" ! File name
CHARACTER(LEN=14), PARAMETER :: dsetname = "Fortran Matrix" ! Dataset name
INTEGER, PARAMETER :: NX = 3
INTEGER, PARAMETER :: NY = 5
INTEGER(HID_T) :: file_id ! File identifier
INTEGER(HID_T) :: dset_id ! Dataset identifier
INTEGER(HID_T) :: dspace_id ! Dataspace identifier
INTEGER(HSIZE_T), DIMENSION(2) :: dims = (/3,5/) ! Dataset dimensions
INTEGER :: rank = 2 ! Dataset rank
INTEGER :: data(NX,NY)
INTEGER :: error ! Error flag
INTEGER :: i, j
!
! Initialize data
!
do i = 1, NX
do j = 1, NY
data(i,j) = j + (i-1)*NY
enddo
enddo
!
! Data
!
! 1 2 3 4 5
! 6 7 8 9 10
! 11 12 13 14 15
!
! Initialize FORTRAN interface.
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 303
!
CALL h5open_f(error)
!
! Create a new file using default properties.
!
CALL h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
!
! Create the dataspace.
!
CALL h5screate_simple_f(rank, dims, dspace_id, error)
!
! Create and write dataset using default properties.
!
CALL h5dcreate_f(file_id, dsetname, H5T_NATIVE_INTEGER, dspace_id, &
dset_id, error, H5P_DEFAULT_F, H5P_DEFAULT_F, &
H5P_DEFAULT_F)
CALL h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, dims, error)
!
! End access to the dataset and release resources used by it.
!
CALL h5dclose_f(dset_id, error)
!
! Terminate access to the data space.
!
CALL h5sclose_f(dspace_id, error)
!
! Close the file.
!
CALL h5fclose_f(file_id, error)
!
! Close FORTRAN interface.
!
CALL h5close_f(error)
END PROGRAM DSETEXAMPLE
SDSf.out
--------
HDF5 "SDSf.h5" {
GROUP "/" {
DATASET "Fortran Matrix" {
DATATYPE H5T_STD_I32BE
HDF5DataspacesandPartialI/O HDF5User’sGuide
304 TheHDFGroup
DATASPACE SIMPLE { ( 5, 3 ) / ( 5, 3 ) }
DATA {
1, 6, 11,
2, 7, 12,
3, 8, 13,
4, 9, 14,
5, 10, 15
}
}
}
}
7.7.3.h5_write_tr.f90
---------------
PROGRAM DSETEXAMPLE
USE HDF5 ! This module contains all necessary modules
IMPLICIT NONE
CHARACTER(LEN=10), PARAMETER :: filename = "SDSf_tr.h5" ! File name
CHARACTER(LEN=24), PARAMETER :: dsetname = "Fortran Transpose Matrix"
! Dataset name
INTEGER, PARAMETER :: NX = 3
INTEGER, PARAMETER :: NY = 5
INTEGER(HID_T) :: file_id ! File identifier
INTEGER(HID_T) :: dset_id ! Dataset identifier
INTEGER(HID_T) :: dspace_id ! Dataspace identifier
INTEGER(HSIZE_T), DIMENSION(2) :: dims = (/NY, NX/) ! Dataset dimensions
INTEGER :: rank = 2 ! Dataset rank
INTEGER :: data(NY,NX)
INTEGER :: error ! Error flag
INTEGER :: i, j
!
! Initialize data
!
do i = 1, NY
do j = 1, NX
data(i,j) = i + (j-1)*NY
enddo
enddo
HDF5User’sGuide HDF5DataspacesandPartialI/O
TheHDFGroup 305
!
! Data
!
! 1 6 11
! 2 7 12
! 3 8 13
! 4 9 14
! 5 10 15
!
! Initialize FORTRAN interface.
!
CALL h5open_f(error)
!
! Create a new file using default properties.
!
CALL h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
!
! Create the dataspace.
!
CALL h5screate_simple_f(rank, dims, dspace_id, error)
!
! Create and write dataset using default properties.
!
CALL h5dcreate_f(file_id, dsetname, H5T_NATIVE_INTEGER, dspace_id, &
dset_id, error, H5P_DEFAULT_F, H5P_DEFAULT_F, &
H5P_DEFAULT_F)
CALL h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, dims, error)
!
! End access to the dataset and release resources used by it.
!
CALL h5dclose_f(dset_id, error)
!
! Terminate access to the data space.
!
CALL h5sclose_f(dspace_id, error)
!
! Close the file.
!
CALL h5fclose_f(file_id, error)
!
! Close FORTRAN interface.
HDF5DataspacesandPartialI/O HDF5User’sGuide
306 TheHDFGroup
!
CALL h5close_f(error)
END PROGRAM DSETEXAMPLE
SDSf_tr.out
-----------
HDF5 "SDSf_tr.h5" {
GROUP "/" {
DATASET "Fortran Transpose Matrix" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 3, 5 ) / ( 3, 5 ) }
DATA {
1, 2, 3, 4, 5,
6, 7, 8, 9, 10,
11, 12, 13, 14, 15
}
}
}
}
HDF5User’sGuide HDF5Attributes
TheHDFGroup 307
8.HDF5Attributes
8.1.Introduction
AnHDF5attributeisasmallmetadataobjectdescribingthenatureand/orintendedusageofaprimary
dataobject.Aprimarydataobjectmaybeadataset,group,orcommitteddatatype.
Attributesareassumedtobeverysmallasdataobjectsgo,sostoringthemasstandardHDF5datasets
wouldbequiteinefficient.HDF5attributesarethereforemanagedthroughaspecialattributesinterface,
H5A,whichisdesignedtoeasilyattachattributestoprimarydataobjectsassmalldatasetscontaining
metadatainformationandtominimizestoragerequirements.
Consider,asexamplesofthesimplestcase,asetoflaboratoryreadingstakenunderknowntemperature
andpressureconditionsof18.0degreesCelsiusand0.5atmospheres,respectively.Thetemperatureand
pressurestoredasattributesofthedatasetcouldbedescribedasthefollowingname/valuepairs:
temp=18.0
pressure=0.5
WhileHDF5attributesarenotstandardHDF5datasets,theyhavemuchincommon:
•Anattributehasauser‐defineddataspaceandtheincludedmetadatahasauser‐assigneddata‐
type
•MetadatacanbeofanyvalidHDF5datatype
• Attributesareaddressedbyname
Buttherearesomeveryimportantdifferences:
•Thereisnoprovisionforspecialstoragesuchascompressionorchunking
•ThereisnopartialI/Oorsub‐settingcapabilityforattributedata
• Attributescannotbeshared
• Attributescannothaveattributes
•Beingsmall,anattributeisstoredintheobjectheaderoftheobjectitdescribesandisthus
attacheddirectlytothatobject
The“SpecialIssues”sectiondescribeshowtohandleattributesthatarelargeinsizeandhowtohandle
largenumbersofattributes.Formoreinformation,see"SpecialIssues"onpage 315.
Thischapterdiscussesorliststhefollowing:
•TheHDF5attributesprogrammingmodel
•H5Afunctionsummaries
•WorkingwithHDF5attributes
•Thestructureofanattribute
•Creating,writing,andreadingattributes
HDF5Attributes HDF5User’sGuide
308 TheHDFGroup
• Accessingattributesbynameorindex
• Obtaininginformationregardinganobject’sattributes
•Iteratingacrossanobject’sattributes
• Deletinganattribute
•Closingattributes
•Specialissuesregardingattributes
Inthefollowingdiscussions,attributesaregenerallyattachedtodatasets.Attributesattachedtootherpri‐
marydataobjectssuchasgroupsorcommitteddatatypesarehandledinexactlythesamemanner.
8.2.ProgrammingModelforAttributes
ThefigurebelowshowstheUMLmodelforanHDF5attributeanditsassociateddataspaceanddatatype.
Creatinganattributeissimilartocreatingadataset.Tocreateanattribute,theapplicationmustspecify
theobjecttowhichtheattributeisattached,thedatatypeanddataspaceoftheattributedata,andthe
attributecreationpropertylist.
ThefollowingstepsarerequiredtocreateandwriteanHDF5attribute:
1. Obtaintheobjectidentifierfortheattribute’sprimarydataobject
2. Definethecharacteristicsoftheattributeandspecifytheattributecreationpropertylist
• Definethedatatype
• Definethedataspace
•Specifytheattributecreationpropertylist
3. Createtheattribute
Figure8‐1.TheUMLmodelforanHDF5attribute
HDF5User’sGuide HDF5Attributes
TheHDFGroup 309
4. Writetheattributedata(optional)
5. Closetheattribute(anddatatype,dataspace,andattributecreationpropertylist,ifnecessary)
6. Closetheprimarydataobject(ifappropriate)
8.2.1.ToOpenandReadorWriteanExistingAttribute
Thefollowingstepsarerequiredtoopenandread/writeanexistingattribute.SinceHDF5attributesallow
nopartialI/O,youneedspecifyonlytheattributeandtheattribute’smemorydatatypetoreadit:
1. Obtaintheobjectidentifierfortheattribute’sprimarydataobject
2. Obtaintheattribute’snameorindex
3. Opentheattribute
•Get
attributedataspaceanddatatype(optional)
4. Specifytheattribute’smemorytype
5. Readand/orwritetheattributedata
6. Closetheattribute
7. Closetheprimarydataobject(ifappropriate)
8.3.Attribute(H5A)FunctionSummaries
Functionsthatcanbeusedwithattributes(H5Afunctions)andfunctionsthatcanbeusedwithproperty
lists(H5Pfunctions)arelistedbelow.
FunctionListing8‐1.Attributefunctions(H5A)
CFunction
FortranSubroutine
Purpose
H5Acreate
h5acreate_f
Createsadatasetasanattributeofanother
group,dataset,orcommitteddatatype.TheC
functionisamacro:see“APICompatibility
MacrosinHDF5.”
H5Acreate_by_name
h5acreate_by_name_f
Createsanattributeattachedtoaspecified
object.
H5Aexists
h5aexists_f
Determineswhetheranattributewithagiven
nameexistsonanobject.
H5Aexists_by_name
h5aexists_by_name_f
Determineswhetheranattributewithagiven
nameexistsonanobject.
HDF5Attributes HDF5User’sGuide
310 TheHDFGroup
H5Aclose
h5aclose_f
Closesthespecifiedattribute.
H5Adelete
h5adelete_f
Deletesanattribute.
H5Adelete_by_idx
h5adelete_by_idx_f
Deletesanattributefromanobjectaccording
toindexorder.
H5Adelete_by_name
h5adelete_by_name_f
Removesanattributefromaspecifiedloca‐
tion.
H5Aget_create_plist
h5aget_create_plist_f
Getsanattributecreationpropertylistidenti‐
fier.
H5Aget_info
h5aget_info_f
Retrievesattributeinformationbyattribute
identifier.
H5Aget_info_by_idx
h5aget_info_by_idx_f
Retrievesattributeinformationbyattribute
indexposition.
H5Aget_info_by_name
h5aget_info_by_name_f
Retrievesattributeinformationbyattribute
name.
H5Aget_name
h5aget_name_f
Getsanattributename.
H5Aget_name_by_idx
h5aget_name_by_idx_f
Getsanattributenamebyattributeindex
position.
H5Aget_space
h5aget_space_f
Getsacopyofthedataspaceforanattribute.
H5Aget_storage_size
h5aget_storage_size_f
Returnstheamountofstoragerequiredforan
attribute.
H5Aget_type
h5aget_type_f
Getsanattributedatatype.
H5Aiterate
(noFortransubroutine)
Callsauser’sfunctionforeachattribute
attachedtoadataobject.TheCfunctionisa
macro:see“APICompatibilityMacrosin
HDF5.”
H5Aiterate_by_name
(noFortransubroutine)
Callsuser‐definedfunctionforeachattribute
onanobject.
FunctionListing8‐1.Attributefunctions(H5A)
CFunction
FortranSubroutine
Purpose
HDF5User’sGuide HDF5Attributes
TheHDFGroup 311
H5Aopen
h5aopen_f
Opensanattributeforanobjectspecifiedby
objectidentifierandattributename.
H5Aopen_by_idx
h5aopen_by_idx_f
Opensanexistingattributethatisattachedto
anobjectspecifiedbylocationandname.
H5Aopen_by_name
h5aopen_by_name_f
Opensanattributeforanobjectbyobject
nameandattributename.
H5Aread
h5aread_f
Readsanattribute.
H5Arename
h5arename_f
Renamesanattribute.
H5Arename_by_name
h5arename_by_name_f
Renamesanattribute.
H5Awrite
H5awrite_f
Writesanattribute.
FunctionListing8‐2.Attributecreationpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pset_char_encoding
h5pset_char_encoding_f
Setsthecharacterencodingusedtoencodea
string.UsetosetASCIIorUTF‐8character
encodingforobjectnames.
H5Pget_char_encoding
h5pget_char_encoding_f
Retrievesthecharacterencodingusedtocre‐
ateastring.
H5Pget_attr_creation_order
h5pget_attr_creation_order_f
Retrievestrackingandindexingsettingsfor
attributecreationorder.
H5Pget_attr_phase_change
h5pget_attr_phase_change_f
Retrievesattributestoragephasechange
thresholds.
H5Pset_attr_creation_order
h5pget_attr_creation_order_f
Setstrackingandindexingofattributecre‐
ationorder.
H5Pset_attr_phase_change
h5pset_attr_phase_change_f
Setsattributestoragephasechangethresh‐
olds.
FunctionListing8‐1.Attributefunctions(H5A)
CFunction
FortranSubroutine
Purpose
HDF5Attributes HDF5User’sGuide
312 TheHDFGroup
8.4.WorkingwithAttributes
8.4.1.TheStructureofanAttribute
Anattributehastwoparts:nameandvalue(s).
HDF5attributesaresometimesdiscussedasname/valuepairsintheformname=value.
Anattribute’snameisanull‐terminatedASCIIorUTF‐8characterstring.Eachattributeattachedtoan
objecthasauniquename.
Thevalueportionoftheattributecontainsoneormoredataelementsofthesamedatatype.
HDF5attributeshaveallthecharacteristicsofHDF5datasetsexceptthatthereisnopartialI/Ocapability.
Inotherwords,attributescanbewrittenandreadonlyinfullwithnosub‐setting.
8.4.2.Creating,Writing,andReadingAttributes
IfattributesareusedinanHDF5file,thesefunctionswillbeemployed:H5Acreate,H5Awrite,and
H5Aread.H5AcreateandH5Awriteareusedtogethertoplacetheattributeinthefile.Ifanattributeis
tobeusedandisnotcurrentlyinmemory,H5Areadgenerallycomesintoplayusuallyinconcertwithone
eachoftheH5Aget_*andH5Aopen_*functions.
Tocreateanattribute,callH5Acreate:
hid_t H5Acreate (hid_t loc_id, const char *name,
hid_t type_id, hid_t space_id, hid_t create_plist,
hid_t access_plist)
loc_ididentifiestheobject(dataset,group,orcommitteddatatype)towhichtheattributeistobe
attached.name,type_id,space_id,andcreate_plistconvey,respectively,theattribute’sname,
datatype,dataspace,andattributecreationpropertylist.Theattribute’snamemustbelocallyunique:it
mustbeuniquewithinthecontextoftheobjecttowhichitisattached.
H5Acreatecreatestheattributeinmemory.TheattributedoesnotexistinthefileuntilH5Awritewrites
itthere.
Towriteorreadanattribute,callH5AwriteorH5Aread,respectively:
herr_t H5Awrite (hid_t attr_id, hid_t mem_type_id,
const void *buf)
herr_t H5Aread (hid_t attr_id, hid_t mem_type_id,
void *buf)
attr_ididentifiestheattributewhilemem_type_ididentifiesthein‐memorydatatypeoftheattribute
data.
HDF5User’sGuide HDF5Attributes
TheHDFGroup 313
H5Awritewritestheattributedatafromthebufferbuftothefile.H5Areadreadsattributedatafrom
thefileintobuf.
TheHDF5Libraryconvertsthemetadatabetweenthein‐memorydatatype,mem_type_id,andthein‐file
datatype,definedwhentheattributewascreated,withoutuserintervention.
8.4.3.AccessingAttributesbyNameorIndex
Attributescanbeaccessedbynameorindexvalue.Theuseofanindexvaluemakesitpossibletoiterate
throughalloftheattributesassociatedwithagivenobject.
Toaccessanattributebyitsname,usetheH5Aopen_by_namefunction.H5Aopen_by_namereturnsan
attributeidentifierthatcanthenbeusedbyanyfunctionthatmustaccessanattributesuchas
H5Aread.UsethefunctionH5Aget_nametodetermineanattribute’sname.
Toaccessanattributebyitsindexvalue,usetheH5Aopen_by_idxfunction.Todetermineanattribute
indexvaluewhenitisnotalreadyknown,usetheH5Oget_infofunction.H5Aopen_by_idxisgenerally
usedinthecourseofopeningseveralattributesforlateraccess.UseH5Aiterateiftheintentistoper‐
formthesameoperationoneveryattributeattachedtoanobject.
8.4.4.ObtainingInformationRegardinganObject’sAttributes
InthecourseofworkingwithHDF5attributes,onemayneedtoobtainanyofseveralpiecesofinforma‐
tion:
•Anattributename
•Thedataspaceofanattribute
•Thedatatypeofanattribute
•Thenumberofattributesattachedtoanobject
Toobtainanattribute’sname,callH5Aget_namewithanattributeidentifier,attr_id:
ssize_t H5Aget_name (hid_t attr_id, size_t buf_size,
char *buf)
Aswithotherattributefunctions,attr_ididentifiestheattribute;buf_sizedefinesthesizeofthebuf‐
fer;andbufisthebuffertowhichtheattribute’snamewillberead.
Ifthelengthoftheattributename,andhencethevaluerequiredforbuf_size,isunknown,afirstcallto
H5Aget_namewillreturnthatsize.Ifthevalueofbuf_sizeusedinthatfirstcallistoosmall,thename
willsimplybetruncatedinbuf.AsecondH5Aget_namecallcanthenbeusedtoretrievethenameinan
appropriately‐sizedbuffer.
Todeterminethedataspaceordatatypeofanattribute,callH5Aget_spaceorH5Aget_type,respec‐
tively:
hid_t H5Aget_space (hid_t attr_id)
HDF5Attributes HDF5User’sGuide
314 TheHDFGroup
hid_t H5Aget_type (hid_t attr_id)
H5Aget_spacereturnsthedataspaceidentifierfortheattributeattr_id.
H5Aget_typereturnsthedatatypeidentifierfortheattributeattr_id.
Todeterminethenumberofattributesattachedtoanobject,usetheH5Oget_infofunction.Thefunc‐
tionsignatureisbelow.
herr_t H5Oget_info( hid_t object_id, H5O_info_t *object_info )
Thenumberofattributeswillbereturnedintheobject_infobuffer.Thisisgenerallythepreferredfirst
stepindeterminingattributeindexvalues.IfthecallreturnsN,theattributesattachedtotheobject
object_idhaveindexvaluesof0throughN-1.
8.4.5.IteratingacrossanObject’sAttributes
Itissometimesusefultobeabletoperformtheidenticaloperationacrossalloftheattributesattachedto
anobject.Atthesimplestlevel,youmightjustwanttoopeneachattribute.Atahigherlevel,youmight
wishtoperformarathercomplexoperationoneachattributeasyouiterateacrosstheset.
Toiterateanoperationacrosstheattributesattachedtoanobject,onemustmakeaseriesofcallsto
H5Aiterate:
herr_t H5Aiterate (hid_t obj_id, H5_index_t index_type,
H5_iter_order_t order, hsize_t *n, H5A_operator2_t op,
void *op_data)
H5Aiteratesuccessivelymarchesacrossalloftheattributesattachedtotheobjectspecifiedinloc_id,
performingtheoperation(s)specifiedinop_funcwiththedataspecifiedinop_dataoneachattribute.
WhenH5Aiterateiscalled,indexcontainstheindexoftheattributetobeaccessedinthiscall.When
H5Aiteratereturns,indexwillcontaintheindexofthenextattribute.Ifthereturnedindexisthenull
pointer,thenallattributeshavebeenprocessed,andtheiterativeprocessiscomplete.
op_funcisauser‐definedoperationthatadherestotheH5A_operator_tprototype.Thisprototype
andcertainrequirementsimposedontheoperator’sbehavioraredescribedintheH5Aiterateentryin
theHDF5ReferenceManual.
op_dataisalsouser‐definedtomeettherequirementsofop_func.Beyondprovidingaparameterwith
whichtopassthisdata,HDF5providesnotoolsforitsmanagementandimposesnorestrictions.
8.4.6.DeletinganAttribute
Onceanattributehasoutliveditsusefulnessorisnolongerappropriate,itmaybecomenecessaryto
deleteit.
Todeleteanattribute,callH5Adelete:
herr_t H5Adelete (hid_t loc_id, const char *name)
HDF5User’sGuide HDF5Attributes
TheHDFGroup 315
H5Adeleteremovestheattributenamefromthegroup,dataset,orcommitteddatatypespecifiedin
loc_id.
H5Adeletemustnotbecalledifthereareanyopenattributeidentifiersontheobjectloc_id.Sucha
callcancausetheinternalattributeindexestochange;futurewritestoanopenattributewouldthenpro‐
duceunintendedresults.
8.4.7.ClosinganAttribute
AsisthecasewithallHDF5objects,onceaccesstoanattributeitisnolongerneeded,thatattributemust
beclosed.Itisbestpracticetocloseitassoonaspracticable;itismandatorythatitbeclosedpriortothe
H5closecallclosingtheHDF5Library.
Tocloseanattribute,callH5Aclose:
herr_t H5Aclose (hid_t attr_id)
H5Acloseclosesthespecifiedattributebyterminatingaccesstoitsidentifier,attr_id.
8.5.SpecialIssues
Somespecialissuesforattributesarediscussedbelow.
LargeNumbersofAttributesStoredinDenseAttributeStorage
Thedenseattributestorageschemewasaddedinversion1.8sothatdatasets,groups,andcommitted
datatypesthathavelargenumbersofattributescouldbeprocessedmorequickly.
Attributesstartoutbeingstoredinanobject'sheader.Thisisknownascompactstorage.Formoreinfor‐
mation,see"StorageStrategies"onpage 129.
Asthenumberofattributesgrows,attribute‐relatedperformanceslows.Toimproveperformance,dense
attributestoragecanbeinitiatedwiththeH5Pset_attr_phase_changefunction.SeetheHDF5Refer‐
enceManualformoreinformation.
Whendenseattributestorageisenabled,athresholdisdefinedforthenumberofattributeskeptincom‐
pactstorage.Whenthenumberisexceeded,thelibrarymovesalloftheattributesintodensestorageat
anotherlocation.Thelibraryhandlesthemovementofattributesandthepointersbetweenthelocations
automatically.Ifsomeoftheattributesaredeletedsothatthenumberfallsbelowthethreshold,thenthe
attributesaremovedbacktocompactstoragebythelibrary.
Theimprovementsinperformancefromusingdenseattributestoragearetheresultofholdingattributes
inaheapandindexingtheheapwithaB‐tree.
Notethattherearesomedisadvantagestousingdenseattributestorage.Oneisthatthisisanewfeature.
Datasets,groups,andcommitteddatatypesthatusedensestoragecannotbereadbyapplicationsbuilt
HDF5Attributes HDF5User’sGuide
316 TheHDFGroup
withearlierversionsofthelibrary.Anotherdisadvantageisthatattributesindensestoragecannotbe
compressed.
LargeAttributesStoredinDenseAttributeStorage
Wegenerallyconsiderthemaximumsizeofanattributetobe64Kbytes.Thelibraryhastwowaysofstor‐
ingattributeslargerthan64Kbytes:indenseattributestorageorinaseparatedataset.Usingdenseattri‐
butestorageisdescribedinthissection,andstoringinaseparatedatasetisdescribedinthenextsection.
Tousedenseattributestoragetostorelargeattributes,setthenumberofattributesthatwillbestoredin
compactstorageto0withtheH5Pset_attr_phase_changefunction.Thiswillforceallattributestobe
putintodenseattributestorageandwillavoidthe64KBsizelimitationforasingleattributeincompact
attributestorage.
Theexamplecodebelowillustrateshowtocreatealargeattributethatwillbekeptindensestorage.
/*
* Test use of dense attribute
*/
#define N 82000000
#include "hdf5.h"
#include <stdio.h>
#include <stdlib.h>
int main(){
hid_t fid, gid, sid, aid, gpid, fpid;
hsize_t dims[] = {N};
double *buf;
int i;
herr_t status;
buf = (double *) malloc(sizeof(double) * N);
for (i=0; i <N; i++) { buf[i] = -100.0; }
fpid = H5Pcreate (H5P_FILE_ACCESS);
status = H5Pset_libver_bounds (fpid, H5F_LIBVER_LATEST,
H5F_LIBVER_LATEST);
fid = H5Fcreate("adense.h5", H5F_ACC_TRUNC, H5P_DEFAULT,
fpid);
gpid = H5Pcreate (H5P_GROUP_CREATE);
status = H5Pset_attr_phase_change (gpid, 0, 0);
CodeExample8‐1.Createalargeattributeindensestorage
HDF5User’sGuide HDF5Attributes
TheHDFGroup 317
LargeAttributesStoredinaSeparateDataset
Inadditiontodenseattributestorage(seeabove),alargeattributecanbestoredinaseparatedataset.In
thefigurebelow,DatasetAholdsanattributethatistoolargefortheobjectheaderinDataset1.By
puttingapointertoDatasetAasanattributeinDataset1,theattributebecomesavailabletothose
workingwithDataset1.
Thiswayofhandlinglargeattributescanbeusedinsituationswherebackwardcompatibilityisimportant
andwherecompressionisimportant.Applicationsbuiltwithversionsbefore1.8.xcanreadlargeattri‐
butesstoredinseparatedatasets.Datasetscanbecompressedwhileattributescannot.
gid = H5Gcreate(fid, "testgrp", H5P_DEFAULT, gpid,
H5P_DEFAULT);
sid = H5Screate_simple(1, dims, NULL);
aid = H5Acreate(gid, "bar", H5T_NATIVE_DOUBLE, sid,
H5P_DEFAULT, H5P_DEFAULT);
status = H5Awrite(aid, H5T_NATIVE_DOUBLE, buf);
/* If you remove these two lines, it doesn't crash */
status = H5Aclose(aid);
status = H5Pclose (gpid);
status = H5Pclose (fpid);
status = H5Gclose(gid);
status = H5Fclose (fid);
return 0;
}
CodeExample8‐1.Createalargeattributeindensestorage
HDF5Attributes HDF5User’sGuide
318 TheHDFGroup
Note:Inthefigureabove,DatasetAisanattributeofDataset1thatistoolargetostoreinDataset1'sheader.Data‐
setAisassociatedwithDataset1bymeansofanobjectreferencepointerattachedasanattributetoDataset1.The
attributeinDatasetAcanbesharedamongmultipledatasetsbymeansofadditionalobjectreferencepointers
attachedtoadditionaldatasets.
SharedAttributes
AttributeswrittenandmanagedthroughtheH5Ainterfacecannotbeshared.Ifsharedattributesare
required,theymustbehandledinthemannerdescribedaboveforlargeattributesandillustratedinthe
figureabove.
AttributeNames
WhileanyASCIIorUTF‐8charactermaybeusedinthenamegiventoanattribute,itisusuallywiseto
avoidthefollowingkindsofcharacters:
•Commonlyusedseparatorsordelimiterssuchasslash,backslash,colon,andsemi‐colon(\,/,:,;)
•Escapecharacters
•Wildcardssuchasasteriskandquestionmark(*,?)
NULLcanbeusedwithinaname,butHDF5namesareterminatedwithaNULL:whatevercomesafterthe
NULLwillbeignoredbyHDF5.
TheuseofASCIIorUTF‐8charactersisdeterminedbythecharacterencodingproperty.See
H5Pset_char_encodingintheHDF5ReferenceManual.
Figure8‐2.AlargeorsharedHDF5attributeanditsassociateddataset(s)
HDF5User’sGuide HDF5Attributes
TheHDFGroup 319
NoSpecialI/OorStorage
HDF5attributeshaveallthecharacteristicsofHDF5datasetsexceptthefollowing:
• Attributesarewrittenandreadonlyinfull:thereisnoprovisionforpartialI/Oorsub‐setting
•Nospecialstoragecapabilityisprovidedforattributes:thereisnocompressionorchunking,and
attributesarenotextendable
HDF5Attributes HDF5User’sGuide
320 TheHDFGroup
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 321
9.HDF5ErrorHandling
9.1.Introduction
TheHDF5Libraryprovidesanerrorreportingmechanismforboththelibraryitselfandforuserapplication
programs.Itcantraceerrorsthroughfunctionstackanderrorinformationlikefilename,functionname,
linenumber,anderrordescription.
"BasicErrorHandlingOperations"beginningonpage 323discussesthebasicerrorconceptssuchaserror
stack,errorrecord,anderrormessageanddescribestherelatedAPIfunctions.Theseconceptsandfunc‐
tionsaresufficientforapplicationprogramstotraceerrorsinsidetheHDF5Library.
"AdvancedErrorHandlingOperations"beginningonpage 329talksabouttheadvancedconceptsoferror
classanderrorstackhandleandtalksabouttherelatedfunctions.Withtheseconceptsandfunctions,an
applicationlibraryorprogramusingtheHDF5LibrarycanhaveitsownerrorreportblendedwithHDF5’s
errorreport.
StartingwithRelease1.8,wehaveanewsetofErrorHandlingAPIfunctions.Forthepurposeofbackward
compatibilitywithversion1.6andbefore,westillkeeptheoldAPIfunctions,H5Epush,H5Eprint,
H5Ewalk,H5Eclear,H5Eget_auto,H5Eset_auto.Thesefunctionsdonothavetheerrorstackas
parameter.Thelibraryallowsthemtooperateonthedefaulterrorstack.Usersdonothavetochange
theircodetocatchupwiththenewErrorAPIbutareencouragedtodoso.
TheoldAPIissimilartofunctionalitydiscussedin"BasicErrorHandlingOperations"beginningon
page 323.Thefunctionalitydiscussedin"AdvancedErrorHandlingOperations"beginningonpage 329,
theabilityofallowingapplicationstoaddtheirownerrorrecords,isthenewdesignfortheErrorHandling
API.
9.2.ProgrammingModelforErrorHandling
Thissectionisunderconstruction.
9.3.ErrorHandling(H5E)FunctionSummaries
Functionsthatcanbeusedtohandleerrors(H5Efunctions)arelistedbelow.
HDF5ErrorHandling HDF5User’sGuide
322 TheHDFGroup
FunctionListing9‐1.Errorhandlingfunctions(H5E)
CFunction
FortranSubroutine
Purpose
H5Eauto_is_v2
(noFortransubroutine)
Determinesthetypeoferrorstack.
H5Eclear
h5eclear_f
Clearstheerrorstackforthecurrentthread.
TheCfunctionisamacro:see“APICompati‐
bilityMacrosinHDF5.”
H5Eclear_stack
(noFortransubroutine)
Clearstheerrorstackforthecurrentthread.
H5Eclose_msg
(noFortransubroutine)
Closesanerrormessageidentifier.
H5Eclose_stack
(noFortransubroutine)
Closesobjecthandleforerrorstack.
H5Ecreate_msg
(noFortransubroutine)
Addmajorerrormessagetoanerrorclass.
H5Eget_auto
h5eget_auto_f
Returnsthecurrentsettingsfortheautomatic
errorstacktraversalfunctionanditsdata.The
Cfunctionisamacro:see“APICompatibility
MacrosinHDF5.”
H5Eget_class_name
(noFortransubroutine)
Retrieveserrorclassname.
H5Eget_current_stack
(noFortransubroutine)
Registersthecurrenterrorstack.
H5Eget_msg
(noFortransubroutine)
Retrievesanerrormessage.
H5Eget_num
(noFortransubroutine)
Retrievesthenumberoferrormessagesinan
errorstack.
H5Epop
(noFortransubroutine)
Deletesspecifiednumberoferrormessages
fromtheerrorstack.
H5Eprint
h5eprint_f
Printstheerrorstackinadefaultmanner.The
Cfunctionisamacro:see“APICompatibility
MacrosinHDF5.”
H5Epush
(noFortransubroutine)
Pushesnewerrorrecordontoerrorstack.The
Cfunctionisamacro:see“APICompatibility
MacrosinHDF5.”
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 323
9.4.BasicErrorHandlingOperations
Letusfirsttrytounderstandtheerrorstack.Anerrorstackisacollectionoferrorrecords.Errorrecords
canbepushedontoorpoppedofftheerrorstack.Bydefault,whenanerroroccursdeepwithintheHDF5
Library,anerrorrecordispushedontoanerrorstackandthatfunctionreturnsafailureindication.Its
callerdetectsthefailure,pushesanotherrecordontothestack,andreturnsafailureindication.Thiscon‐
tinuesuntiltheAPIfunctioncalledbytheapplicationreturnsafailureindication.ThenextAPIfunction
beingcalledwillresettheerrorstack.AllHDF5Libraryerrorrecordsbelongtothesameerrorclass.For
moreinformation,see"AdvancedErrorHandlingOperations"onpage 329.
9.4.1.ErrorStackandErrorMessage
Innormalcircumstances,anerrorcausesthestacktobeprintedonthestandarderrorstreamautomati‐
cally.Thisautomaticerrorstackisthelibrary’sdefaultstack.Forallthefunctionsinthissection,whenever
anerrorstackIDisneededasaparameter,H5E_DEFAULTcanbeusedtoindicatethelibrary’sdefault
stack.Thefirsterrorrecordoftheerrorstack,number#000,isproducedbytheAPIfunctionitselfandis
usuallysufficienttoindicatetotheapplicationwhatwentwrong.
H5Eregister_class
(noFortransubroutine)
Registersaclientlibraryorapplicationpro‐
gramtotheHDF5errorAPI.
H5Eset_auto
h5eset_auto_f
Turnsautomaticerrorprintingonoroff.TheC
functionisamacro:see“APICompatibility
MacrosinHDF5.”
H5Eset_current_stack
(noFortransubroutine)
Replacesthecurrenterrorstack.
H5Eunregister_class
(noFortransubroutine)
Removesanerrorclass.
H5Ewalk
(noFortransubroutine)
Walkstheerrorstackforthecurrentthread,
callingaspecifiedfunction.TheCfunctionisa
macro:see“APICompatibilityMacrosin
HDF5.”
FunctionListing9‐1.Errorhandlingfunctions(H5E)
CFunction
FortranSubroutine
Purpose
HDF5ErrorHandling HDF5User’sGuide
324 TheHDFGroup
Example:AnErrorReport
IfanapplicationcallsH5Tcloseonapredefineddatatype,thenthemessageintheexamplebelowis
printedonthestandarderrorstream.Thisisasimpleerrorthathasonlyonecomponent,theAPIfunc‐
tion;othererrorsmayhavemanycomponents.
Intheexampleabove,wecanseethatanerrorrecordhasamajormessageandaminormessage.Amajor
messagegenerallyindicateswheretheerrorhappens.Thelocationcanbeadatasetoradataspace,for
example.Aminormessageexplainsfurtherdetailsoftheerror.Anexampleis“unabletoopenfile”.
Anotherspecificdetailabouttheerrorcanbefoundattheendofthefirstlineofeacherrorrecord.This
errordescriptionisusuallyaddedbythelibrarydesignertotellwhatexactlygoeswrong.Intheexample
above,the“predefineddatatype”isanerrordescription.
9.4.2.PrintandClearanErrorStack
Besidestheautomaticerrorreport,theerrorstackcanalsobeprintedandclearedbythefunctions
H5Eprint()andH5Eclear_stack().IfanapplicationwishestomakeexplicitcallstoH5Eprint()to
printtheerrorstack,theautomaticprintingshouldbeturnedofftopreventerrormessagesfrombeing
displayedtwice(seeH5Eset_auto()below).
Toprintanerrorstack:
herr_t H5Eprint(hid_t error_stack, FILE * stream)
Thisfunctionprintstheerrorstackspecifiedbyerror_stackonthespecifiedstream,stream.Ifthe
errorstackisempty,aone‐linemessagewillbeprinted.Thefollowingisanexampleofsuchamessage.
ThismessagewouldbegeneratediftheerrorwasintheHDF5Library.
HDF5-DIAG: Error detected in HDF5 Library version: 1.5.62 thread 0.
Toclearanerrorstack:
herr_t H5Eclear_stack(hid_t error_stack)
TheH5Eclear_stackfunctionshownaboveclearstheerrorstackspecifiedbyerror_stack.H5E_DE-
FAULTcanbepassedintoclearthecurrenterrorstack.ThecurrentstackisalsoclearedwheneveranAPI
functioniscalled;therearecertainexceptionstothisrulesuchasH5Eprint().
HDF5-DIAG: Error detected in HDF5 (1.6.4) thread 0.
#000: H5T.c line 462 in H5Tclose(): predefined datatype
major: Function argument
minor: Bad value
CodeExample9‐1.Anerrorreport
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 325
9.4.3.MuteErrorStack
Sometimesanapplicationcallsafunctionforthesakeofitsreturnvalue,fullyexpectingthefunctionto
fail;sometimestheapplicationwantstocallH5Eprint()explicitly.Inthesesituations,itwouldbemis‐
leadingifanerrormessagewerestillautomaticallyprinted.UsingtheH5Eset_auto()functioncancon‐
troltheautomaticprintingoferrormessages.
Toenableordisableautomaticprintingoferrors:
herr_t H5Eset_auto(hid_t error_stack, H5E_auto_t func,
void *client_data)
TheH5Eset_auto functioncanbeusedtoturnsonorofftheautomaticprintingoferrorsfortheerror
stackspecifiedbyerror_stack.Whenturnedon(non‐nullfuncpointer),anyAPIfunctionwhich
returnsanerrorindicationwillfirstcallfunc,passingitclient_dataasanargument.Whenthelibrary
isfirstinitializedtheautoprintingfunctionissettoH5Eprint()(castappropriately)andclient_data
isthestandarderrorstreampointer,stderr.
Toseethecurrentsettings:
herr_t H5Eget_auto(hid_t error_stack, H5E_auto_t * func,
void **client_data)
Thefunctionabovereturnsthecurrentsettingsfortheautomaticerrorstacktraversalfunction,func,and
itsdata,client_data.Ifeitherorbothoftheargumentsarenull,thenthevalueisnotreturned.
Example:ErrorControl
Anapplicationcantemporarilyturnofferrormessageswhile“probing”afunction.Seetheexample
below.
/* Save old error handler */
H5E_auto2_t oldfunc;
void *old_client_data;
H5Eget_auto(error_stack, &old_func, &old_client_data);
/* Turn off error handling */
H5Eset_auto(error_stack, NULL, NULL);
/* Probe. Likely to fail, but that’s okay */
status = H5Fopen (......);
/* Restore previous error handler */
H5Eset_auto(error_stack, old_func, old_client_data);
CodeExample9‐2.Turnofferrormessageswhileprobingafunction
HDF5ErrorHandling HDF5User’sGuide
326 TheHDFGroup
Orautomaticprintingcanbedisabledaltogetheranderrormessagescanbeexplicitlyprinted.
9.4.4.CustomizedPrintingofanErrorStack
Applicationsareallowedtodefineanautomaticerrortraversalfunctionotherthanthedefault
H5Eprint().Forinstance,onecandefineafunctionthatprintsasimple,one‐lineerrormessagetothe
standarderrorstreamandthenexits.Thefirstexamplebelowdefinesasuchafunction.Thesecondexam‐
plebelowinstallsthefunctionastheerrorhandler.
/* Turn off error handling permanently */
H5Eset_auto(error_stack, NULL, NULL);
/* If failure, print error message */
if (H5Fopen (....)<0) {
H5Eprint(H5E_DEFAULT, stderr);
exit (1);
}
CodeExample9‐3.Disableautomaticprintingandexplicitlyprinterrormessages
herr_t
my_hdf5_error_handler(hid_t estack, void *unused)
{
fprintf (stderr, “An HDF5 error was detected. Bye.\n”);
exit (1);
}
CodeExample9‐4.Definingafunctiontoprintasimpleerrormessage
H5Eset_auto(H5E_DEFAULT, my_hdf5_error_handler, NULL);
CodeExample9‐5.Theuser‐definederrorhandler
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 327
9.4.5.WalkthroughtheErrorStack
TheH5Eprint()functionisactuallyjustawrapperaroundthemorecomplexH5Ewalk()function
whichtraversesanerrorstackandcallsauser‐definedfunctionforeachmemberofthestack.Theexam‐
plebelowshowshowH5Ewalkisused.
herr_t H5Ewalk(hid_t err_stack, H5E_direction_t direction,
H5E_walk_t func, void *client_data)
Theerrorstackerr_stackistraversedandfunciscalledforeachmemberofthestack.Itsarguments
areanintegersequencenumberbeginningatzero(regardlessofdirection)andtheclient_data
pointer.IfdirectionisH5E_WALK_UPWARD,thentraversalbeginsattheinner‐mostfunctionthat
detectedtheerrorandconcludeswiththeAPIfunction.UseH5E_WALK_DOWNWARDfortheopposite
order.
9.4.6.TraverseanErrorStackwithaCallbackFunction
Anerrorstacktraversalcallbackfunctiontakesthreearguments:nisasequencenumberbeginningat
zeroforeachtraversal,eptrisapointertoanerrorstackmember,andclient_dataisthesamepointer
usedintheexampleabovepassedtoH5Ewalk().Seetheexamplebelow.
typedef herr_t (*H5E_walk_t)(unsigned n, H5E_error2_t *eptr,
void *client_data)
TheH5E_error2_tstructureisshownbelow.
typedef struct {
hid_t cls_id;
hid_t maj_num;
hid_t min_num;
unsigned line;
const char *func_name;
const char *file_name;
const char *desc;
} H5E_error2_t;
Themaj_numandmin_numaremajorandminorerrorIDs,func_nameisthenameofthefunctionwhere
theerrorwasdetected,file_nameandlinelocatetheerrorwithintheHDF5Librarysourcecode,and
descpointstoadescriptionoftheerror.
Example:CallbackFunction
Thefollowingexampleshowsauser‐definedcallbackfunction.
HDF5ErrorHandling HDF5User’sGuide
328 TheHDFGroup
#define MSG_SIZE 64
herr_t
custom_print_cb(unsigned n, const H5E_error2_t *err_desc,
void* client_data)
{
FILE *stream = (FILE *)client_data;
char maj[MSG_SIZE];
char min[MSG_SIZE];
char cls[MSG_SIZE];
const int indent = 4;
/* Get descriptions for the major and minor error
* numbers
*/
if(H5Eget_class_name(err_desc->cls_id, cls, MSG_SIZE)<0)
TEST_ERROR;
if(H5Eget_msg(err_desc->maj_num, NULL, maj, MSG_SIZE)<0)
TEST_ERROR;
if(H5Eget_msg(err_desc->min_num, NULL, min, MSG_SIZE)<0)
TEST_ERROR;
fprintf (stream, “%*serror #%03d: %s in %s():
line %u\n”,
indent, “”, n, err_desc->file_name,
err_desc->func_name, err_desc->line);
fprintf (stream, “%*sclass: %s\n”, indent*2, “”, cls);
fprintf (stream, “%*smajor: %s\n”, indent*2, “”, maj);
fprintf (stream, “%*sminor: %s\n”, indent*2, “”, min);
return 0;
error:
return -1;
}
CodeExample9‐6.Auser‐definedcallbackfunction
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 329
ProgrammingNoteforC++DevelopersUsingCFunctions
IfaCroutinethattakesafunctionpointerasanargumentiscalledfromwithinC++code,theCroutine
shouldbereturnedfromnormally.
ExamplesofthiskindofroutineincludecallbackssuchasH5Pset_elink_cbandH5Pset_type_con-
v_cbandfunctionssuchasH5TconvertandH5Ewalk2.
ExitingtheroutineinitsnormalfashionallowstheHDF5CLibrarytocleanupitsworkproperly.Inother
words,iftheC++applicationjumpsoutoftheroutinebacktotheC++“catch”statement,thelibraryisnot
giventheopportunitytocloseanytemporarydatastructuresthatweresetupwhentheroutinewas
called.TheC++applicationshouldsavesomestateastheroutineisstartedsothatanyproblemthat
occursmightbediagnosed.
9.5.AdvancedErrorHandlingOperations
Thesectionabove,see"BasicErrorHandlingOperations"beginningonpage 323,discussesthebasicerror
handlingoperationsofthelibrary.Inthatsection,alltheerrorrecordsontheerrorstackarefromthe
libraryitself.Inthissection,wearegoingtointroducetheoperationsthatallowanapplicationprogramto
pushitsownerrorrecordsontotheerrorstackonceitdeclaresanerrorclassofitsownthroughtheHDF5
ErrorAPI.
Example:AnErrorReport
Anerrorreportshowsboththelibrary’serrorrecordandtheapplication’serrorrecords.Seetheexample
below.
Error Test-DIAG: Error detected in Error Program (1.0)
thread 8192:
#000: ../../hdf5/test/error_test.c line 468 in main():
Error test failed
major: Error in test
minor: Error in subroutine
#001: ../../hdf5/test/error_test.c line 150 in
test_error(): H5Dwrite failed as supposed to
major: Error in IO
minor: Error in H5Dwrite
HDF5-DIAG: Error detected in HDF5 (1.7.5) thread 8192:
#002: ../../hdf5/src/H5Dio.c line 420 in H5Dwrite():
not a dataset
major: Invalid arguments to routine
minor: Inappropriate type
CodeExample9‐7.Anerrorreport
HDF5ErrorHandling HDF5User’sGuide
330 TheHDFGroup
Inthelineaboveerrorrecord#002intheexampleabove,thestartingphraseisHDF5.Thisistheerror
classnameoftheHDF5Library.Allofthelibrary’serrormessages(majorandminor)areinthisdefault
errorclass.TheError Testinthebeginningofthelineaboveerrorrecord#000isthenameofthe
application’serrorclass.Thefirsttwoerrorrecords,#000and#001,arefromapplication’serrorclass.
Bydefinition,anerrorclassisagroupofmajorandminorerrormessagesforalibrary(theHDF5Libraryor
anapplicationlibrarybuiltontopoftheHDF5Library)oranapplicationprogram.Theerrorclasscanbe
registeredforalibraryorprogramthroughtheHDF5ErrorAPI.Majorandminormessagescanbedefined
inanerrorclass.Anapplicationwillhaveobjecthandlesfortheerrorclassandformajorandminormes‐
sagesforfurtheroperation.Seetheexamplebelow.
#define MSG_SIZE 64
herr_t
custom_print_cb(unsigned n, const H5E_error2_t *err_desc,
void* client_data)
{
FILE *stream = (FILE *)client_data;
char maj[MSG_SIZE];
char min[MSG_SIZE];
char cls[MSG_SIZE];
const int indent = 4;
/* Get descriptions for the major and minor error
* numbers
*/
if(H5Eget_class_name(err_desc->cls_id, cls, MSG_SIZE)<0)
TEST_ERROR;
if(H5Eget_msg(err_desc->maj_num, NULL, maj, MSG_SIZE)<0)
TEST_ERROR;
CodeExample9‐8.Defininganerrorclass
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 331
9.5.1.MoreErrorAPIFunctions
TheErrorAPIhasfunctionsthatcanbeusedtoregisterorunregisteranerrorclass,tocreateorcloseerror
messages,andtoqueryanerrorclassorerrormessage.Thesefunctionsareillustratedbelow.
Toregisteranerrorclass:
hid_t H5Eregister_class(const char* cls_name, const char* lib_name,
const char* version)
ThisfunctionregistersanerrorclasswiththeHDF5Librarysothattheapplicationlibraryorprogramcan
reporterrorstogetherwiththeHDF5Library.
Toaddanerrormessagetoanerrorclass:
hid_t H5Ecreate_msg(hid_t class, H5E_type_t msg_type, const char* mesg)
Thisfunctionaddsanerrormessagetoanerrorclassdefinedbyanapplicationlibraryorprogram.The
errormessagecanbeeithermajororminorwhichisindicatedbyparametermsg_type.
Togetthenameofanerrorclass:
ssize_t H5Eget_class_name(hid_t class_id, char* name, size_t size)
ThisfunctionretrievesthenameoftheerrorclassspecifiedbytheclassID.
Toretrieveanerrormessage:
ssize_t H5Eget_msg(hid_t mesg_id, H5E_type_t* mesg_type, char* mesg,
if(H5Eget_msg(err_desc->min_num, NULL, min, MSG_SIZE)<0)
TEST_ERROR;
fprintf (stream, “%*serror #%03d: %s in %s():
line %u\n”,
indent, “”, n, err_desc->file_name,
err_desc->func_name, err_desc->line);
fprintf (stream, “%*sclass: %s\n”, indent*2, “”, cls);
fprintf (stream, “%*smajor: %s\n”, indent*2, “”, maj);
fprintf (stream, “%*sminor: %s\n”, indent*2, “”, min);
return 0;
error:
return -1;
}
CodeExample9‐8.Defininganerrorclass
HDF5ErrorHandling HDF5User’sGuide
332 TheHDFGroup
size_t size)
Thisfunctionretrievestheerrormessageincludingitslengthandtype.
Tocloseanerrormessage:
herr_t H5Eclose_msg(hid_t mesg_id)
Thisfunctionclosesanerrormessage.
Toremoveanerrorclass:
herr_t H5Eunregister_class(hid_t class_id)
ThisfunctionremovesanerrorclassfromtheErrorAPI.
Example:ErrorClassanditsMessage
Theexamplebelowshowshowanapplicationcreatesanerrorclassanderrormessages.
Theexamplebelowshowshowanapplicationcloseserrormessagesandunregisterstheerrorclass.
/* Create an error class */
class_id = H5Eregister_class(ERR_CLS_NAME, PROG_NAME,
PROG_VERS);
/* Retrieve class name */
H5Eget_class_name(class_id, cls_name, cls_size);
/* Create a major error message in the class */
maj_id = H5Ecreate_msg(class_id, H5E_MAJOR, “... ...”);
/* Create a minor error message in the class */
min_id = H5Ecreate_msg(class_id, H5E_MINOR, “... ...”);
CodeExample9‐9.Createanerrorclassanderrormessages
H5Eclose_msg(maj_id);
H5Eclose_msg(min_id);
H5Eunregister_class(class_id);
CodeExample9‐10.Closingerrormessagesandunregisteringtheerrorclass
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 333
9.5.2.PushinganApplicationErrorMessageontoErrorStack
Anapplicationcanpusherrorrecordsontoorpoperrorrecordsoffoftheerrorstackjustasthelibrary
doesinternally.Anerrorstackcanberegistered,andanobjecthandlecanbereturnedtotheapplication
sothattheapplicationcanmanipulatearegisterederrorstack.
Toregisterthecurrentstack:
hid_t H5Eget_current_stack(void)
Thisfunctionregistersthecurrenterrorstack,returnsanobjecthandle,andclearsthecurrenterrorstack.
AnemptyerrorstackwillalsobeassignedanID.
Toreplacethecurrenterrorstackwithanother:
herr_t H5Eset_current_stack(hid_t error_stack)
Thisfunctionreplacesthecurrenterrorstackwithanothererrorstackspecifiedbyerror_stackand
clearsthecurrenterrorstack.Theobjecthandleerror_stackisclosedafterthisfunctioncall.
Topushanewerrorrecordtotheerrorstack:
herr_t H5Epush(hid_t error_stack, const char* file, const char* func,
unsigned line, hid_t cls_id, hid_t major_id, hid_t minor_id,
const char* desc, ... )
Thisfunctionpushesanewerrorrecordontotheerrorstackforthecurrentthread.
Todeletesomeerrormessages:
herr_t H5Epop(hid_t error_stack, size_t count)
Thisfunctiondeletessomeerrormessagesfromtheerrorstack.
Toretrievethenumberoferrorrecords:
int H5Eget_num(hid_t error_stack)
Thisfunctionretrievesthenumberoferrorrecordsfromanerrorstack.
Tocleartheerrorstack:
herr_t H5Eclear_stack(hid_t error_stack)
Thisfunctionclearstheerrorstack.
Toclosetheobjecthandleforanerrorstack:
herr_t H5Eclose_stack(hid_t error_stack)
Thisfunctionclosestheobjecthandleforanerrorstackandreleasesitsresources.
Example:WorkingwithanErrorStack
Theexamplebelowshowshowanapplicationpushesanerrorrecordontothedefaulterrorstack.
HDF5ErrorHandling HDF5User’sGuide
334 TheHDFGroup
Theexamplebelowshowshowanapplicationregistersthecurrenterrorstackandcreatesanobjecthan‐
dletoavoidanotherHDF5functionfromclearingtheerrorstack.
/* Make call to HDF5 I/O routine */
if((dset_id=H5Dopen(file_id, dset_name, access_plist))<0)
{
/* Push client error onto error stack */
H5Epush(H5E_DEFAULT,__FILE__,FUNC,__LINE__,cls_id,
CLIENT_ERR_MAJ_IO,CLIENT_ERR_MINOR_OPEN,
“H5Dopen failed”);
/* Indicate error occurred in function */
return(0);
}
CodeExample9‐11.Pushinganerrormessagetoanerrorstack
if(H5Dwrite(dset_id, mem_type_id, mem_space_id,
file_space_id, dset_xfer_plist_id, buf)<0)
{
/* Push client error onto error stack */
H5Epush(H5E_DEFAULT,__FILE__,FUNC,__LINE__,cls_id,
CLIENT_ERR_MAJ_IO,CLIENT_ERR_MINOR_HDF5,
“H5Dwrite failed”);
/* Preserve the error stack by assigning an object
* handle to it
*/
error_stack = H5Eget_current_stack();
/* Close dataset */
H5Dclose(dset_id);
/* Replace the current error stack with the
* preserved one
*/
H5Eset_current_stack(error_stack);
Return(0);
}
CodeExample9‐12.Registeringtheerrorstack
HDF5User’sGuide HDF5ErrorHandling
TheHDFGroup 335
HDF5ErrorHandling HDF5User’sGuide
336 TheHDFGroup
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 337
10.PropertiesandPropertyListsinHDF5
10.1.Introduction
HDF5propertiesandpropertylistsmakeitpossibletoshapeormodifyanHDF5file,group,dataset,attri‐
bute,committeddatatype,orevenanI/Ostream,inanumberofways.Forexample,youcandoanyofthe
following:
•Customizethestoragelayoutofafiletosuitaprojectortask.
•Createachunkeddataset.
• Applycompressionorfilterstorawdata.
•UseeitherASCIIorUTF‐8characterencodings.
•Createmissinggroupsonthefly.
•SwitchbetweenserialandparallelI/O.
•Createconsistencywithinasinglefileoracrossaninternationalproject.
SomepropertiesenableanHDF5applicationtotakeadvantageofthecapabilitiesofaspecificcomputing
environmentwhileothersmakeafilemorecompact;somespeedthereadingorwritingofdatawhileoth‐
ersenablemorerecord‐keepingataper‐objectlevel.HDF5offersnearlyonehundredspecificproperties
thatcanbeusedinliterallythousandsofcombinationstomaximizetheusabilityofHDF5‐storeddata.
Atthemostbasiclevel,apropertylistisacollectionofproperties,representedbyname/valuepairsthat
canbepassedtovariousHDF5functions,usuallymodifyingdefaultsettings.Apropertylistinheritsaset
ofpropertiesandvaluesfromapropertylistclass.Butthatstatementhardlyprovidesacompletepicture;
intherestofthissectionandinthenextsection,“PropertyListClasses,PropertyLists,andProperties”,we
willdiscussthesethingsinmuchmoredetail.Afterreadingthatmaterial,thereadershouldhaveareason‐
ablycompleteunderstandingofhowpropertiesandpropertylistscanbeusedinHDF5applications.
Figure10‐1.TheHDF5propertyenvironment
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
338 TheHDFGroup
Theremainingsectionsinthischapterdiscussthefollowingtopics:
•Whatareproperties,propertylists,andpropertylistclasses?
•Propertylistprogrammingmodel
•Genericpropertyfunctions
•Summarylistingsofpropertylistfunctions
• Additionalresources
Thediscussionsandfunctionlistingsinthischapterfocusongeneralpropertyoperations,objectandlink
properties,andrelatedfunctions.
File,group,dataset,datatype,andattributepropertiesarediscussedinthechaptersdevotedtothosefea‐
tures,wherethatinformationwillbemostconvenienttousers.Forexample,"HDF5Datasets"on
page 103discussesdatasetcreationpropertylistsandfunctions,datasetaccesspropertylistsandfunc‐
tions,anddatasettransferpropertylistsandfunctions.Thischapterdoesnotduplicatethosediscussions.
Genericpropertyoperationsareanadvancedfeatureandarebeyondthescopeofthisguide.
ThischapterassumesanunderstandingofthefollowingchaptersofthisHDF5User’sGuide:
•"TheHDF5DataModelandFileStructure"onpage 1
•"TheHDF5LibraryandProgrammingModel"onpage 21
10.2.PropertyListClasses,PropertyLists,andProperties
HDF5propertylistsandthepropertylistinterfaceH5Pprovideamechanismforstoringcharacteristicsof
objectsinanHDF5fileandeconomicallypassingthemaroundinanHDF5application.Inthiscapacity,
propertylistssignificantlyreducetheburdenofadditionalfunctionparametersthroughouttheHDF5API.
AnotheradvantageofpropertylistsisthatfeaturescanoftenbeaddedtoHDF5byaddingonlyproperty
listfunctionstotheAPI;thisisparticularlytruewhenallotherrequirementsofthefeaturecanbeaccom‐
plishedinternallytothelibrary.
Forinstance,afilecreationoperationneedstoknowseveralthingsaboutafile,suchasthesizeofthe
userblockorthesizesofvariousfiledatastructures.Bundlingthisinformationasapropertylistsimplifies
theinterfacebyreducingthenumberofparameterstothefunctionH5Fcreate.
Asillustratedinthefigureabove("TheHDF5propertyenvironment"onpage 337),theHDF5property
environmentisathree‐levelhierarchy:
•Propertylistclasses
•Propertylists
• Properties
Thefollowingsubsectionsdiscusspropertylistclasses,propertylists,andpropertiesinmoredetail.
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 339
10.2.1.PropertyListClasses
Apropertylistclassdefinestherolesthatpropertylistsofthatclasscanplay.Eachclassincludesallprop‐
ertiesthatarevalidforthatclasswitheachpropertysettoitsdefaultvalue.HDF5offersapropertylists
classforeachofthefollowingsituations.
Note:Inthetableabove,theabbreviationstotherightofeachpropertylistclassnameinthistablearewidelyused
inbothHDF5programmerdocumentationandHDF5sourcecode.Forexample,FCPLisfilecreationpropertylist,
Table10‐1.PropertylistclassesinHDF5
PropertyListClass Forfurtherdiscussion
Filecreation(FCPL)
Fileaccess(FAPL)
Filemount(FMPL)
H5P_FILE_CREATE
H5P_FILE_ACCESS
H5P_FILE_MOUNT
Seevarioussectionsof"The
HDF5File"beginningonpage 45.
UsedonlyasH5P_DEFAULT.For
moreinformation,see"File
MountProperties"onpage 351.
Objectcreation(OCPL)
Objectcopy(OCPYPL)
H5P_OBJECT_CREATE
H5P_OBJECT_COPY
Seethetableof"Objectproperty
functions(H5P)"onpage 348.
Groupcreation(GCPL)
Groupaccess(GAPL)
H5P_GROUP_CREATE
H5P_GROUP_ACCESS
See"ProgrammingModelfor
Groups"onpage 91.
Linkcreation(LCPL)
Linkaccess(LAPL)
H5P_LINK_CREATE
H5P_LINK_ACCESS
Seeexamplesin"Programming
ModelforPropertiesandProp‐
ertyLists"onpage 343andthe
tableof"Linkcreationproperty
functions(H5P)"onpage 350.
Datasetcreation(DCPL)
Datasetaccess(DAPL)
Datasettransfer(DXPL)
H5P_DATASET_CREATE
H5P_DATASET_ACCESS
H5P_DATASET_XFER
See"ProgrammingModelfor
Datasets"onpage 110.
Datatypecreation(TCPL)
Datatypeaccess(TAPL)
H5P_DATATYPE_CREATE
H5P_DATATYPE_ACCESS
Seevarioussectionsof"HDF5
Datatypes"beginningon
page 173.
Stringcreation(STRCPL) H5P_STRING_CREATE See"ProgrammingModelfor
Datasets"onpage 110and"Pro‐
grammingModelforDatatypes"
onpage 192.
Attributecreation(ACPL) H5P_ATTRIBUTE_CREATE See"WorkingwithAttributes"on
page 312.
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
340 TheHDFGroup
OCPLisobjectcreationpropertylist,OCPYPLisobjectcopypropertylist,andSTRCPLisstringcreationpropertylist.
Theseabbreviationsmayappearineitheruppercaseorlowercase.
The“HDF5propertylistclassinheritancehierarchy”figure,immediatelyfollowing,illustratestheinheri‐
tancehierarchyofHDF5’spropertylistclasses.PropertiesaredefinedattherootoftheHDF5property
environment(“PropertyListClassRoot”inthefigurebelow).Propertylistclassestheninheritproperties
fromthatroot,eitherdirectlyorindirectlythroughaparentclass.Ineverycase,apropertylistclassinher‐
itsonlythepropertiesrelevanttoitsrole.Forexample,theobjectcreationpropertylistclass(OCPL)inher‐
itsallpropertiesthatarerelevanttothecreationofanyobjectwhilethegroupcreationpropertylistclass
(GCPL)inheritsonlythosepropertiesthatarerelevanttogroupcreation.
Note:Inthefigureabove,propertylistclassesdisplayedinblackaredirectlyaccessiblethroughtheprogramming
interface;therootofthepropertyenvironmentandtheSTRCPLandOCPLpropertylistclasses,ingrayabove,arenot
user‐accessible.Theredemptysetsymbolindicatesthatthefilemountpropertylistclass(FMPL)isanemptyclass;
thatis,ithasnosettableproperties.Formoreinformation,see"FileMountProperties"onpage 351.Abbreviations
usedinthisfigurearedefinedintheprecedingtable,“PropertylistclassesinHDF5”.
10.2.2.PropertyLists
Apropertylistisacollectionofrelatedpropertiesthatareusedtogetherinspecificcircumstances.Anew
propertylistcreatedfromapropertylistclassinheritsthepropertiesofthepropertylistclassandeach
property’sdefaultvalue.Afreshdatasetcreationpropertylist,forexample,includesalloftheHDF5prop‐
ertiesrelevanttothecreationofanewdataset.
Propertylistsareimplementedascontainersholdingacollectionofname/valuepairs.Eachpairspecifies
apropertynameandavaluefortheproperty.Apropertylistusuallycontainsinformationforonetomany
properties.
HDF5’sdefaultpropertyvaluesaredesignedtobereasonableforgeneralusecases.Therefore,anapplica‐
tioncanoftenuseapropertylistwithoutmodification.Ontheotherhand,adjustingpropertylistsettings
isaroutineactionandtherearemanyreasonsforanapplicationtodoso.
Figure10‐2.HDF5propertylistclassinheritancehierarchy
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 341
Anewpropertylistmayeitherbederivedfromapropertylistclassorcopiedfromanexistingpropertylist.
Whenapropertylistiscreatedfromapropertylistclass,itcontainsallthepropertiesthatarerelevantto
theclass,witheachpropertysettoitsdefaultvalue.Anewpropertylistcreatedbycopyinganexisting
propertylistwillcontainthesamepropertiesandpropertyvaluesastheoriginalpropertylist.Ineither
case,thepropertyvaluescanbechangedasneededthroughtheHDF5API.
Propertylistscanbefreelyreusedtocreateconsistency.Forexample,asinglesetoffile,group,anddata‐
setcreationpropertylistsmightbecreatedatthebeginningofaprojectandusedtocreatehundreds,
thousands,evenmillions,ofconsistentfiles,filestructures,anddatasetsovertheproject’slife.When
suchconsistencyisimportanttoaproject,thisisaneconomicalmeansofprovidingit.
10.2.3.Properties
Apropertyisthebasicelementofthepropertylisthierarchy.HDF5offersnearlyonehundredproperties
controllingthingsrangingfromfileaccessrights,tothestoragelayoutofadataset,throughoptimizingthe
useofaparallelcomputingenvironment.
Furtherexamplesincludethefollowing:
Eachpropertyisinitializedwithadefaultvalue.Foreachproperty,thereareoneormorededicated
H5Pset_*callsthatcanbeusedtochangethatvalue.
Creation,access,andtransferproperties:
Propertiesfallintooneofseveralmajorcategories:creationproperties,accessproperties,andtransfer
properties.
Creationpropertiescontrolpermanentobjectcharacteristics.Thesecharacteristicsmustbeestablished
whenanobjectiscreated,cannotchangethroughthelifeoftheobject(theyareimmutable),andthe
propertysettingusuallyhasapermanentpresenceinthefile.
Examplesofcreationpropertiesinclude:
Purpose Examples PropertyList
Specifythedrivertobeusedtoopen
afile
APOSIXdriveroranMPIIOdriver FAPL
Specifyfilterstobeappliedtoadata‐
set
Gzipcompressionorchecksumevalu‐
ation
DCPL
Specifywhethertorecordkeytimes
associatedwithanobject
Creationtimeand/orlast‐modified
time
OCPL
Specifytheaccessmodeforafile
openedviaanexternallink
Read‐onlyorread‐write LAPL
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
342 TheHDFGroup
•Whetheradatasetisstoredinacompact,contiguous,orchunkedlayout
Thedefaultforthisdatasetcreationproperty(H5Pset_layout)isthatadatasetisstoredina
contiguousblock.Thisworkswellfordatasetswithaknownsizelimitthatwillfiteasilyinsystem
memory.
Achunkedlayoutisimportantifadatasetistobecompressed,toenableextendingthedataset’s
size,ortoenablecachingduringI/O.
Acompactlayoutissuitableonlyforverysmalldatasetsbecausetherawdataisstoredinthe
objectheader.
• CreationofintermediategroupswhenaddinganobjecttoanHDF5file
Thislinkcreationproperty(H5Pset_create_intermediate_group)enablesanapplicationto
addanobjectinafilewithouthavingtoknowthatthegrouporgrouphierarchycontainingthat
objectalreadyexists.Withthispropertyset,HDF5automaticallycreatesmissinggroups.Ifthis
propertyisnotset,anapplicationmustverifythateachgroupinthepathexists,andcreatethose
thatdonot,beforecreatingthenewobject;ifanygroupismissing,thecreateoperationwillfail.
•WhetheranHDF5fileisasinglefileorasetoftightlyrelatedfilesthatformavirtualHDF5file
Certainfilecreationpropertiesenabletheapplicationtoselectoneofseveralfilelayouts.Exam‐
plesoftheavailablelayoutsincludeastandardPOSIX‐compliantlayout(H5Pset_fapl_sec2),a
familyoffiles(H5Pset_fapl_family),andasplitfilelayoutthatseparatesrawdataandmeta‐
dataintoseparatefiles(H5Pset_fapl_split).Theseandotherfilelayoutoptionsarediscussed
in"AlternateFileStorageLayoutsandLow‐levelFileDrivers"onpage 61.
•Toenableerrordetectionwhencreatingadataset
Insettingswheredataintegrityisvulnerable,itmaybedesirabletosetchecksummingwhendata‐
setsarecreated(H5Pset_fletcher32).Asubsequentapplicationwillthenhaveameanstover‐
ifydataintegritywhenreadingthedataset.
Accesspropertiescontroltransientobjectcharacteristics.Thesecharacteristicsmaychangewiththecir‐
cumstancesunderwhichanobjectisaccessed.
Examplesofaccesspropertiesinclude:
•Thedriverusedtoopenafile
Forexample,afilemightbecreatedwiththeMPII/Odriver(H5Pset_fapl_mpio)duringhigh‐
speeddataacquisitioninaparallelcomputingenvironment.Thesamefilemightlaterbeanalyzed
inaserialcomputingenvironmentwithI/OaccesshandledthroughtheserialPOSIXdriver
(H5Pset_fapl_sec2).
• Optimizationsettingsinspecializedenvironments
Optimizationsdifferacrosscomputingenvironmentsandaccordingtotheneedsofthetaskbeing
performed,soaretransientbynature.
TransferpropertiesapplyonlytodatasetsandcontroltransientaspectsofdataI/O.Thesecharacteristics
maychangewiththecircumstancesunderwhichdataisaccessed.
Examplesofdatasettransferpropertiesinclude:
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 343
•Toenableerrordetectionwhenreadingadataset
Ifchecksumminghasbeensetonadataset(withH5Pset_fletcher32,inthedatasetcreation
propertylist),anapplicationreadingthatdatasetcanchoosewhethercheckfordataintegrity
(H5Pset_edc_check).
•VariouspropertiestooptimizechunkeddataI/Oonparallelcomputingsystems
HDF5providesseveralpropertiesfortuningI/Oofchunkeddatasetsinaparallelcomputingenvi‐
ronment(H5Pset_dxpl_mpio_chunk_opt,H5Pset_dxpl_mpio_chunk_opt_num,
H5Pset_dxpl_mpio_chunk_opt_ratio,andH5Pget_mpio_actual_chunk_opt_mode).
Optimalsettingsdifferduetothecharacteristicsofacomputingenvironmentandduetoanappli‐
cation’sdataaccesspatterns;evenwhenworkingwiththesamefile,thesesettingsmightchange
foreveryapplicationandeveryplatform.
10.3.ProgrammingModelforPropertiesandProperty
Lists
TheprogrammingmodelforHDF5propertylistsisactuallyquitesimple:
1. Createapropertylist.
2. Modifythepropertylist,ifrequired.
3. Usethepropertylist.
4. Closethepropertylist.
Therearenuances,ofcourse,butthatisthebasicprocess.
Insomecases,youwillnothavetodefinepropertylistsatall.Ifthedefaultpropertysettingsaresufficient
foryourapplication,youcantellHDF5tousethedefaultpropertylist.
Thefollowingsectionsfirstdiscusstheuseofdefaultpropertylists,theneachstepoftheprogramming
model,andfinallyafewlessfrequentlyusedpropertylistoperations.
10.3.1.UsingDefaultPropertyLists
DefaultpropertylistscansimplifymanyroutineHDF5tasksbecauseyoudonotalwayshavetocreate
everypropertylistyouuse.
Anapplicationthatwouldbewell‐servedbyHDF5’sdefaultpropertysettingscanusethedefaultproperty
listssimplybysubstitutingthevalueH5P_DEFAULTforapropertylistidentifier.HDF5willthenapplythe
defaultpropertylistfortheappropriatepropertylistclass.
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
344 TheHDFGroup
Forexample,thefunctionH5Dcreate2callsforalinkcreationpropertylist,adatasetcreationproperty
list,andadatasetaccesspropertylist.Ifthedefaultpropertiesaresuitableforadataset,thiscallcanbe
madeas
dset_id = H5Dcreate2( loc_id, name, dtype_id, space_id;
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT );
HDF5willthenapplythedefaultlinkcreation,datasetcreation,anddatasetaccesspropertylistscorrectly.
Ofcourse,youwouldnotwanttodothiswithoutconsideringwhereitisappropriate,astheremaybe
unforeseenconsequences.Consider,forexample,theuseofchunkeddatasets.Optimalchunkingisquite
dependentonthemakeupofthedatasetandthemostcommonaccesspatterns,bothofwhichmustbe
takenintoaccountinsettingupthesizeandshapeofchunks.
10.3.2.BasicStepsoftheProgrammingModel
Thestepsofthepropertylistprogrammingmodelaredescribedinthesub‐sectionsbelow.
10.3.2.1.CreateaPropertyList
Anewpropertylistcanbecreatedeitherasaninstanceofapropertylistclassorbycopyinganexisting
propertylist.Considerthefollowingexamples.Anewdatasetcreationpropertylistisfirstcreated“from
scratch”withH5Pcreate.Aseconddatasetcreationpropertylististhencreatedbycopyingthefirstone
withH5Pcopy.
dcplA_id = H5Pcreate (H5P_DATASET_CREATE);
ThenewdatasetcreationpropertylistiscreatedasaninstanceofthepropertylistclassH5P_-
DATASET_CREATE.
Thenewdatasetcreationpropertylist’sidentifierisreturnedindcplA_idandthepropertylist
isinitializedwithdefaultdatasetcreationpropertyvalues.
Alistofvalidclassesappearsinthetable"PropertylistclassesinHDF5"onpage 339.
dcplB_id = H5Pcopy (dcplA_id);
Anewdatasetcreationpropertylist,dcplB_id,iscreatedasacopyofdcplA_idandisinitial‐
izedwithdatasetcreationpropertyvaluescurrentlyindcplA_id.
Atthispoint,dcplA_idanddcplB_idareidentical;theywillbothcontainanymodifiedpropertyvalues
thatwerechangedindcplA_idbeforedcplB_idwascreated.Theymay,however,divergeasadditional
propertyvaluesareresetineach.
Whilewearecreatingpropertylists,let’screatealinkcreationpropertylist;wewillneedthispropertylist
whenthenewdatasetislinkedintothefilebelow:
lcplAB_id = H5Pcreate (H5P_LINK_CREATE);
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 345
10.3.2.2.ChangePropertyValues
Thissectiondescribeshowtosetpropertyvalues.
Laterinthissection,thedatasetcreationpropertylistsdcplA_idanddcplB_idcreatedinthesection
abovewillbeusedrespectivelytocreatechunkedandcontiguousdatasets.Tosetthisup,wemustsetthe
layoutpropertyineachpropertylist.ThefollowingexamplesetsdcplA_idforchunkeddatasetsand
dcplB_idforcontiguousdatasets:
error = H5Pset_layout (dcplA_id, H5D_CHUNKED);
error = H5Pset_layout (dcplB_id, H5D_CONTIGUOUS);
SincedcplA_idspecifiesachunkedlayout,wemustalsosetthenumberofdimensionsandthesizeof
thechunks.TheexamplebelowspecifiesthatdatasetscreatedwithdcplA_idwillbe3‐dimensionaland
thatthechunksizewillbe100ineachdimension:
error = H5Pset_chunk (dcplA_id, 3, [100,100,100]);
ThesedatasetswillbecreatedwithUTF‐8encodednames.Toaccomplishthat,thefollowingexamplesets
thecharacterencodingpropertyinthelinkcreationpropertylisttocreatelinknameswithUTF‐8encod‐
ing:
error = H5Pset_char_encoding (lcplAB_id, H5T_CSET_UTF8);
dcplA_idcannowbeusedtocreatechunkeddatasetsanddcplB_idtocreatecontiguousdatasets.
AndwiththeuseoflcplAB_id,theywillbecreatedwithUTF‐8encodednames.
10.3.2.3.UsethePropertyList
Oncetherequiredpropertylistshavebeencreated,theycanbeusedtocontrolvariousHDF5processes.
Forillustration,considerdatasetcreation.
AssumethatthedatatypedtypeABandthedataspacesdspaceAanddspaceBhavebeendefinedand
thatthelocationidentifierlocAB_idspecifiesthegroupABinthecurrentHDF5file.Wehavealreadycre‐
atedtherequiredlinkcreationanddatasetcreationpropertylists.Forthesakeofillustration,weassume
thatthedefaultdatasetaccesspropertylistmeetsourapplicationrequirements.Thefollowingcallswould
createthedatasetsdsetAanddsetBinthegroupAB.TherawdataindsetAwillbecontiguouswhile
dsetBrawdatawillbechunked;bothdatasetswillhaveUTF‐8encodedlinknames:
dsetA_id = H5Dcreate2( locAB_id, dsetA, dtypeAB, dspaceA_id,
lcplAB_id, dcplA_id, H5P_DEFAULT );
dsetB_id = H5Dcreate2( locAB_id, dsetB, dtypeAB, dspaceB_id,
lcplAB_id, dcplB_id, H5P_DEFAULT );
10.3.2.4.ClosethePropertyList
Generally,creatingoropeninganythinginanHDF5fileresultsinanHDF5identifier.Theseidentifiersare
ofHDF5typehid_tandincludethingslikefileidentifiers,oftenexpressedasfile_id;datasetidentifi‐
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
346 TheHDFGroup
ers,dset_id;andpropertylistidentifiers,plist_id.Toreducetheriskofmemoryleaks,allofthese
identifiersmustbeclosedoncetheyarenolongerneeded.
Propertylistidentifiersarenoexceptiontothisrule,andH5Pcloseisusedforthispurpose.Thecalls
immediatelyfollowingwouldclosethepropertylistscreatedandusedintheexamplesabove.
error = H5Pclose (dcplA_id);
error = H5Pclose (dcplB_id);
error = H5Pclose (lcplAB_id);
10.3.3.AdditionalPropertyListOperations
Afewpropertylistoperationsfalloutsideoftheprogrammingmodeldescribedabove.Thissection
describesthoseoperations.
10.3.3.1.QuerytheClassofanExistingPropertyList
Occasionallyanapplicationwillhaveapropertylistbutnotknowthecorrespondingpropertylistclass.A
callsuchasinthefollowingexamplewillretrievetheunknownclassofaknownpropertylist:
PList_Class = H5Pget_class (dcplA_id);
Uponthisfunction’sreturn,PList_ClasswillcontainthevalueH5P_DATASET_CREATEindicatingthat
dcplA_idisadatasetcreationpropertylist.
10.3.3.2.DetermineCurrentCreationPropertyListSettingsinanExistingObject
Afterafilehasbeencreated,anotherapplicationmayworkonthefilewithoutknowinghowthecreation
propertiesforthefileweresetup.Retrievingthesepropertyvaluesisoftenunnecessary;HDF5canread
thedataandknowshowtodealwithanypropertiesitencounters.
Butsometimesanapplicationmustdosomethingthatrequiresknowingthecreationpropertysettings.
HDF5makestheacquisitionofthisinformationfairlystraight‐forward;foreachpropertysettingcall,
H5Pset_*,thereisacorrespondingH5Pget_*calltoretrievetheproperty’scurrentsetting.
Considerthefollowingexampleswhichillustratethedeterminationofdatasetlayoutandchunkingset‐
tings:
Theapplicationmustfirstidentifythecreationpropertylistwiththeappropriategetcreation
propertylistcall.Thereisonesuchcallforeachkindofobject.
H5Dget_create_plistwillreturnapropertylistidentifierforthecreationpropertylist
thatwasusedtocreatethedataset.CallitDCPL1_id.
H5Pset_layoutsetsadataset’slayouttobecompact,contiguous,orchunked.
H5Pget_layoutcalledwithDCPL1_idwillreturnthedataset’slayout,eitherH5D_-
COMPACT,H5D_CONTIGUOUS,orH5D_CHUNKED.
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 347
H5Pset_chunksetstherankofadataset,thatisthenumberofdimensionsitwillhave,andthe
maximumsizeofeachdimension.
H5Pget_chunk,alsocalledwithDCPL1_id,willreturntherankofthedatasetandthe
maximumsizeofeachdimension.
Ifacreationpropertyvaluehasnotbeenexplicitlyset,theseH5Pget_callswillreturntheproperty’s
defaultvalue.
10.3.3.3.DetermineAccessPropertySettings
Accesspropertysettingsarequitedifferentfromcreationproperties.Sinceaccesspropertysettingsare
notretainedinanHDF5fileorobject,thereisnormallynoknowledgeofthesettingsthatwereusedinthe
past.Ontheotherhand,sinceaccesspropertiesdonotaffectcharacteristicsofthefileorobject,thisis
notnormallyanissue.Formoreinformation,see"AccessandCreationPropertyExceptions"onpage 352.
Onecircumstanceunderwhichanapplicationmightneedtodetermineaccesspropertysettingsmightbe
whenafileorobjectisalreadyopenbuttheapplicationdoesnotknowthepropertylistsettings.Inthat
case,theapplicationcanusetheappropriategetaccesspropertylistcalltoretrieveapropertylistidenti‐
fier.Forexample,ifthedatasetdsetAfromtheearlierexamplesisstillopen,thefollowingcallwould
returnanidentifierforthedatasetaccesspropertylistinuse:
dsetA_dacpl_id = H5Dget_access_plist( dsetA_id );
Theapplicationcouldthenusethereturnedpropertylistidentifiertoanalyzethepropertysettings.
10.4.GenericPropertiesInterfaceandUser‐defined
Properties
HDF5’sgenericpropertyinterfaceprovidestoolsformanagingtheentirepropertyhierarchyandforthe
creationandmanagementofuser‐definedpropertylistsandproperties.Thisinterfacealsomakesitpossi‐
bleforanapplicationoradrivertocreate,modify,andmanagecustomproperties,propertylists,and
propertylistclasses.Acomprehensivelistoffunctionsforthisinterfaceappearsunder“GenericProperty
Operations(Advanced)”inthe“H5P:PropertyListInterface”sectionoftheHDF5ReferenceManual.
FurtherdiscussionofHDF5’sgenericpropertyinterfaceanduser‐definedpropertiesandpropertylistsis
beyondthescopeofthisdocument.
10.5.PropertyListFunctionSummaries
Generalpropertyfunctions,genericpropertyfunctionsandmacros,propertyfunctionsthatareusedwith
multipletypesofobjects,andobjectandlinkpropertyfunctionsarelistedbelow.
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
348 TheHDFGroup
Propertylistfunctionsthatapplytoaspecifictypeofobjectarelistedinthechapterthatdiscussesthat
object.Forexample,theDatasetschapterhastwopropertylistfunctionlistings:onefordatasetcreation
propertylistfunctionsandonefordatasetaccesspropertylistfunctions.Ashasbeenstated,thischapter
isnotintendedtodescribeeverypropertylistfunction.
Objectpropertyfunctionscanbeusedwithseveralkindsofobjects.
FunctionListing10‐1.Generalpropertylistfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pcreate
h5pcreate_f
Createsanewpropertylistasaninstanceofa
specifiedparentpropertylistclass.
H5Pcopy
h5pcopy_f
Createsanewpropertylistbycopyingthe
specifiedexistingpropertylist.
H5Pget_class
h5pget_class_f
Retrievestheparentpropertylistclassofthe
specifiedpropertylist.
H5Pclose
h5pclose_f
Closesthespecifiedpropertylist.
FunctionListing10‐2.Objectpropertyfunctions(H5P)
CFunction
FortranSubroutine
Purpose
ObjectCreationProperties
H5Pget_attr_creation_order
h5pget_attr_creation_order_f
Retrievestrackingandindexingsettings
forattributecreationorder.
H5Pget_attr_phase_change
h5pget_attr_phase_change_f
Retrievesattributestoragephasechange
thresholds.
H5Pget_obj_track_times
h5pget_obj_track_times_f
Determineswhethertimesassociated
withanobjectarebeingrecorded.
H5Pset_attr_creation_order
h5pset_attr_creation_order_f
Setstrackingandindexingofattribute
creationorder.
H5Pset_attr_phase_change
h5pset_attr_phase_change_f
Setsattributestoragephasechange
thresholds.
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 349
Thefollowingtablelistslinkcreationproperties.Sincethecreationofalinkisalmostalwaysastepinthe
creationofanobject,thesepropertiesmayalsobesetingroupcreationpropertylists,datasetcreation
propertylists,datatypecreationpropertylists,andthemoregenericobjectcreationpropertylists.Some
arealsoapplicabletotheattributecreationpropertylists.
H5Pset_obj_track_times
h5pset_obj_track_times_f
Setstherecordingoftimesassociated
withanobject.
ObjectCopyProperties
H5Padd_merge_committed_dtype_path
(noFortransubroutine)
Addsapathtothelistofpathsthatwill
besearchedinthedestinationfilefora
matchingcommitteddatatype.
H5Pfree_merge_committed_dtype_paths
(noFortransubroutine)
Clearsthelistofpathsstoredinanobject
copypropertylist.
H5Pget_copy_object
h5pget_copy_object_f
Retrievesthepropertiestobeusedwhen
anobjectiscopied.
H5Pget_mcdt_search_cb
(noFortransubroutine)
Retrievesthecallbackfunctionfromthe
specifiedobjectcopypropertylist.
H5Pset_copy_object
h5pset_copy_object_f
Setsthepropertiestobeusedwhenan
objectiscopied.
H5Pset_mcdt_search_cb
(noFortransubroutine)
SetsthecallbackfunctionthatH5Ocopy
willinvokebeforesearchingtheentire
destinationfileforamatchingcommitted
datatype.
FunctionListing10‐2.Objectpropertyfunctions(H5P)
CFunction
FortranSubroutine
Purpose
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
350 TheHDFGroup
Note:Inthefunctionlistingabove,thepropertiescanbeusedwithanyoftheindicatedpropertylists.
10.6.AdditionalPropertyListResources
PropertylistsareubiquitousinanHDF5environmentandarethereforediscussedinmanyplacesinHDF5
documentation.ThefollowingsectionsandlistingsintheHDF5User’sGuideareofparticularinterest:
•Inthe“HDF5DataModelandFileStructure”chapter,see"PropertyList"onpage 13.
•Inthe“HDF5File”chapter,seethefollowingsectionsandlistings:
•"FileCreationandFileAccessProperties"onpage 46
•"FilePropertyLists"onpage 58
•"ExamplewiththeFileCreationPropertyList"onpage 74
•"ExamplewiththeFileAccessPropertyList"onpage 75
•"Filecreationpropertylistfunctions(H5P)"onpage 53
•"Fileaccesspropertylistfunctions(H5P)"onpage 54
FunctionListing10‐3.Linkcreationpropertyfunctions(H5P)
CFunction
FortranSubroutine
Purpose
H5Pget_char_encoding
h5pget_char_encoding_f
Queriesthecharacterencodingusedto
encodelinkorattributenames.
Note:Usewithlink,object,dataset,datatype,
group,orattributecreationpropertylists.
H5Pset_char_encoding
h5pset_char_encoding_f
Setsthecharacterencodingusedto
encodelinkandattributenames.
Note:Usewithlink,object,dataset,datatype,
group,orattributecreationpropertylists.
H5Pget_create_intermediate_group
h5pget_create_intermediate_group_f
Queriessettingforcreationofintermedi‐
ategroups.
Note:Usewithlinkcreationpropertylists,
whichinturncanbeusedinthecreatecallfor
anydataset,datatype,orgroup.
H5Pset_create_intermediate_group
h5pset_create_intermediate_group_f
Specifieswhethertocreateintermediate
groupswhentheydonotalreadyexist.
Note:Usewithlinkcreationpropertylists,
whichinturncanbeusedinthecreatecallfor
anydataset,datatype,orgroup.
HDF5User’sGuide PropertiesandPropertyListsinHDF5
TheHDFGroup 351
•"Filedriverfunctions(H5P)"onpage 55
•Inthe“HDF5Attributes”chapter,see"Attributecreationpropertylistfunctions(H5P)"on
page 311.
•Inthe“HDF5Groups”chapter,see"Groupcreationpropertylistfunctions(H5P)"onpage 90.
•Propertylistsarediscussedthroughout"HDF5Datasets"beginningonpage 103.
Allpropertylistfunctionsaredescribedinthe“H5P:PropertyListInterface”sectionoftheHDF5Reference
Manual.Thefunctionindexatthetopofthepageprovidesacategorizedlistinggroupedbypropertylist
class.Thoseclassesarelistedbelow:
•Filecreationproperties
•Fileaccessproperties
•Groupcreationproperties
•Datasetcreationproperties
•Datasetaccessproperties
•Dataset
transferproperties
• Linkcreationproperties
• Linkaccessproperties
•Objectcreationproperties
•Objectcopyproperties
Additionalcategoriesnotrelatedtotheclassstructureareasfollows:
• Generalpropertylistoperations
• Genericpropertylistfunctions
Thegeneralpropertyfunctionscanbeusedwithanypropertylist;thegenericpropertyfunctions
constituteanadvancedfeature.
Thein‐memoryfileimagefeatureofHDF5usespropertylistsinamannerthatdifferssubstantiallyfrom
theiruseelsewhereinHDF5.Thosewhoplantousein‐memoryfileimagesmuststudy“FileImageOpera‐
tions”(PDF)intheAdvancedTopicsinHDF5collection.
10.7.Notes
FileMountProperties
WhilethefilemountpropertylistclassH5P_FILE_MOUNTisavalidHDF5propertylistclass,nofilemount
propertiesaredefinedbytheHDF5Library.Referencestoafilemountpropertylistshouldalwaysbe
expressedasH5P_DEFAULT,meaningthedefaultfilemountpropertylist.
PropertiesandPropertyListsinHDF5 HDF5User’sGuide
352 TheHDFGroup
AccessandCreationPropertyExceptions
Thereareasmallnumberofexceptionstotherulethatcreationpropertiesarealwaysretainedinafileor
objectandaccesspropertiesareneverretained.
Thefollowingpropertiesarefileaccesspropertiesbuttheyarenottransient;theyhavepermanentand
differenteffectsonafile.Theycouldbevalidlyclassifiedasfilecreationpropertiesastheymustbesetat
creationtimetoproperlycreatethefile.Buttheyareaccesspropertiesbecausetheymustalsobeset
whenafileisreopenedtoproperlyaccessthefile.
Thefollowingisalinkcreationproperty,butitisnotrelevantafteranobjecthasbeencreatedandisnot
retainedinthefileorobject.
Property Relatedfunction
Familyfiledriver H5Pset_fapl_family
Splitfiledriver H5Pset_fapl_split
Corefiledriver H5Pset_fapl_core
Property Relatedfunction
Createmissingintermediategroups H5Pset_create_intermediate_groups
HDF5User’sGuide AdditionalResources
TheHDFGroup 353
11.AdditionalResources
ThesedocumentsprovideadditionalinformationfortheuseandtuningofspecificHDF5features.
Table11‐1.Additionalresources
Document Comments
HDF5Examples CodeexamplesbyAPI.
ChunkinginHDF5 Structuringtheuseofchunkingandtun‐
ingitforperformance.
UsingtheDirectChunkWriteFunction Describesanotherwaythatchunkscanbe
writtentodatasets.
CopyingCommittedDatatypeswithH5Ocopy Describeshowtocopytoanotherfilea
datasetthatusesacommitteddatatype
oranobjectwithanattributethatusesa
committeddatatypesothatthecommit‐
teddatatypeinthedestinationfilecanbe
usedbymultipleobjects.
MetadataCachinginHDF5 ManagingtheHDF5metadatacacheand
tuningitforperformance.
HDF5DynamicallyLoadedFilters DescribeshowanHDF5applicationcan
applyafilterthatisnotregisteredwith
theHDF5Library.
HDF5FileImageOperations DescribeshowtoworkwithHDF5filesin
memory.DiskI/Oisnotrequiredwhen
fileimagesareopened,created,read
from,orwrittento.
ModifiedRegionWrites Describeshowtosetwriteoperationsfor
in‐memoryfilessothatonlymodified
regionsarewrittentostorage.Available
whentheCore(Memory)VFDisused.
UsingIdentifiers Describeshowidentifiersbehaveand
howtheyshouldbetreated.
UsingUTF‐8EncodinginHDF5Applications DescribestheuseofUTF‐8Unicodechar‐
acterencodingsinHDF5applications.
AdditionalResources HDF5User’sGuide
354 TheHDFGroup
FreeingMemoryAllocatedbytheHDF5Library Describeshowinconsistentmemoryman‐
agementcancauseheapcorruptionor
resourceleaksandpossiblesolutions.
HDF5Glossary Aglossaryofterms.
Table11‐1.Additionalresources
Document Comments
HDF5User’sGuide Index
TheHDFGroup 355
Index
A
abstractdatamodel, 1
accessproperties, 342
array, 176
arraydatatype, 148
atomicdatatype, 147
attr_id, 312,313,314
B
backing_store, 69
big‐endian, 195
block, 279
C
chunked, 123
committeddatatype, 10,81
compact, 85,123
complex_t, 33
compounddatatype, 32,147,176
compression, 162
contiguous, 123
count, 279
creationproperties, 341
D
danglinglink, 96
datapipeline, 124
datapipelinefilters, 126
datatransferpipeline, 113
datasetcreationproperties, 8
datasetstoragelayouts, 65
dataspaceextent, 268
decompression, 162
Direct, 63
E
errorrecord, 324
errorstack, 323
externalstorage, 140
F
fileaccessproperties, 6
fileaccesspropertylists, 46
filecreationproperties, 6
filecreationpropertylists, 46
filestoragelayouts, 65
filters, 126
fractalarrays, 228
free, 231
G
get_fapl_family, 55
H
H5A, 21
H5Aclose, 40,41,42,310,315,317
h5aclose_f, 310
H5Acreate, 40,41,104,250,309,312,317
H5Acreate_by_name, 309
h5acreate_by_name_f, 309
h5acreate_f, 309
H5Adelete, 104,310,314,315
H5Adelete_by_idx, 310
h5adelete_by_idx_f, 310
H5Adelete_by_name, 310
h5adelete_by_name_f, 310
h5adelete_f, 310
H5Aexists, 309
H5Aexists_by_name, 309
h5aexists_by_name_f, 309
h5aexists_f, 309
H5Aget_create_plist, 310
h5aget_create_plist_f, 310
H5Aget_info, 310
H5Aget_info_by_idx, 310
h5aget_info_by_idx_f, 310
H5Aget_info_by_name, 310
h5aget_info_by_name_f, 310
h5aget_info_f, 310
H5Aget_name, 310,313
H5Aget_name_by_idx, 310
h5aget_name_by_idx_f, 310
h5aget_name_f, 310
H5Aget_space, 310,313,314
h5aget_space_f, 310
H5Aget_storage_size, 310
h5aget_storage_size_f, 310
H5Aget_type, 194,250,310,313,314
Index HDF5User’sGuide
356 TheHDFGroup
h5aget_type_f, 310
H5Aiterate, 42,104,310,313,314
H5Aiterate_by_name, 310
H5Aopen, 311
H5Aopen_by_idx, 42,311,313
h5aopen_by_idx_f, 311
H5Aopen_by_name, 41,42,311,313
h5aopen_by_name_f, 311
h5aopen_f, 311
H5Aopen_idx, 104
H5Aopen_name, 104
H5A_operator_t, 314
H5Aread, 41,42,175,311,312,313
h5aread_f, 311
H5Arename, 311
H5Arename_by_name, 311
h5arename_by_name_f, 311
h5arename_f, 311
H5Awrite, 40,41,175,311,312,313,317
H5awrite_f, 311
H5check_version, 50
h5check_version_f, 50
H5close, 25,50,315
h5close_f, 50
H5D, 21
H5D_CHUNKED, 346
H5Dclose, 24,105,114,115,119,120,121,122,
135,155,159,160,167,168,172,334
h5dclose_f, 105
H5D_COMPACT, 346
H5D_CONTIGUOUS, 140,346
H5Dcreate, 24,36,39,93,105,111,112,114,115,
154,159,167,171,183,215,227,229,235,236,
238,239,244,250,252,297
H5Dcreate2, 344
H5Dcreate_anon, 105
h5dcreate_anon_f, 105
h5dcreate_f, 105
H5detect, 180
H5Dextend, 36,37
H5Dfill, 106
h5dfill_f, 106
H5Dgather, 106
H5Dget_access_plist, 105
H5Dget_create_plist, 105,122,240,241,346
h5dget_create_plist_f, 105
H5Dget_offset, 105
h5dget_offset_f, 105
H5Dget_space, 29,32,105,122,283,288
h5dget_space_f, 105
H5Dget_space_status, 105
h5dget_space_status_f, 105
H5Dget_storage_size, 105
h5dget_storage_size_f, 105
H5Dget_type, 32,105,122,194,221,223,240,
249,250,252,253
h5dget_type_f, 105
H5Dget_vlen_buf_size, 139,232
H5Diterate, 106
H5dont_atexit, 50
h5dont_atexit_f, 50
H5Dopen, 40,93,95,96,104,105,111,118,119,
120,121,122,135,155,159,167,223,240,252,
298,334
h5dopen_f, 105
H5Dread, 26,105,109,115,118,120,121,123,
124,128,131,138,139,155,160,168,172,175,
184,222,223,224,225,230,234,235,240,254,
284,287,288,289,298,299
h5dread_f, 105
H5Dscatter, 106
H5Dset_extent, 106,112,134
h5dset_extent_f, 106
h5dump, 86,87,272,300
H5D_UNLIMITED, 112
H5Dvlen_get_buf_size, 105
h5dvlen_get_max_len_f, 105
H5Dvlen_reclaim, 105,109,140,229,230,231,
234
h5dvlen_reclaim_f, 105
H5Dwrite, 25,26,31,105,115,118,119,120,123,
124,128,131,155,159,167,171,175,184,215,
217,230,235,286,287,291,297,334
h5dwrite_f, 105,221
H5E, 21
H5Eauto_is_v2, 322
H5Eclear, 321,322
h5eclear_f, 322
H5Eclear_stack, 322,324,333
H5Eclose_msg, 322,332
H5Eclose_stack, 322,333
H5Ecreate_msg, 322,331,332
H5E_DEFAULT, 323,334
H5E_error2_t, 327
H5Eget_auto, 321,322,325
h5eget_auto_f, 322
H5Eget_class_name, 322,328,330,331,332
H5Eget_current_stack, 322,333
H5Eget_msg, 322,328,330,331
H5Eget_num, 322,333
HDF5User’sGuide Index
TheHDFGroup 357
H5E_MAJOR, 332
H5E_MINOR, 332
H5Epop, 322,333
H5Eprint, 321,322,324,325,326,327
h5eprint_f, 322
H5Epush, 321,322,333,334
H5Eregister_class, 323,331,332
H5Eset_auto, 321,323,324,325,326
h5eset_auto_f, 323
H5Eset_current_stack, 323,333
H5Eunregister_class, 323,332
H5Ewalk, 321,323,327
H5Ewalk2, 329
H5E_WALK_DOWNWARD, 327
H5E_walk_t, 327
H5E_WALK_UPWARD, 327
H5F, 21
H5F_ACC_EXCL, 23,45,46,48
H5F_ACC_RDONLY, 45,46,49,57,69
H5F_ACC_RDWR, 45,46,57,69,75,121
H5F_ACC_TRUNC, 23,45,46,74,75,153,158,165,
170,316
H5Fclear_elink_file_cache, 51,91
H5Fclose, 23,25,49,51,57,74,75,155,160,168,
172,317
h5fclose_f, 51
H5F_CLOSE_STRONG, 25
H5Fcreate, 23,45,46,48,51,56,74,75,83,153,
158,165,170,316,338
h5fcreate_f, 51
H5FD_CORE, 64,127
H5FD_DIRECT, 63
H5FD_FAMILY, 64,127
H5FD_LOG, 63,127
H5FD_MPIO, 65,73,127
H5FD_MPIPOSIX, 65
H5FD_MULTI, 64,71,127
H5FD_SEC2, 63,127
H5FD_SPLIT, 64,72
H5FD_STDIO, 63,68,127
H5FD_STREAM, 65
H5FD_WINDOWS, 63
H5Fflush, 51
h5fflush_f, 51
H5Fget_access_plist, 51
h5fget_access_plist_f, 51
H5Fget_create_plist, 51
h5fget_create_plist_f, 51
H5Fget_file_image, 51
h5fget_file_image_f, 51
H5Fget_filesize, 51
h5fget_filesize_f, 51
H5Fget_freespace, 51
h5fget_freespace_f, 51
H5Fget_info, 51
H5Fget_intent, 51
H5Fget_mdc_config, 52
H5Fget_mdc_hit_rate, 52
H5Fget_mdc_size, 52
H5Fget_mpi_atomicity, 52
h5fget_mpi_atomicity_f, 52
H5Fget_name, 52
h5fget_name_f, 52
H5Fget_obj_count, 52
h5fget_obj_count_f, 52
H5Fget_obj_ids, 52
h5fget_obj_ids_f, 52
H5Fget_vfd_handle, 52
H5FILE_NAME, 158,165
H5Fis_hdf5, 52
h5fis_hdf5_f, 52
H5F_LIBVER_LATEST, 86,316
H5Fmount, 52,77,97
h5fmount_f, 52
H5Fopen, 45,46,49,52,57,69,75,92,119,121,
122,134,325
h5fopen_f, 52
H5Freopen, 52
h5freopen_f, 52
H5Freset_mdc_hit_rate_stats, 52
H5Fset_mdc_config, 52
H5Fset_mpi_atomicity, 52
h5fset_mpi_atomicity_f, 52
H5F_UNLIMITED, 140
H5Funmount, 52
h5funmount_f, 52
H5G, 21
H5garbage_collect, 50
h5garbage_collect_f, 50
H5Gclose, 38,88,94,317
h5gclose_f, 88
H5Gcreate, 37,38,87,92,317
H5Gcreate_anon, 87
h5gcreate_anon_f, 87
h5gcreate_f, 87
H5get_libversion, 50
h5get_libversion_f, 50
H5Gget_comment, 89
H5Gget_create_plist, 88
h5gget_create_plist_f, 88
Index HDF5User’sGuide
358 TheHDFGroup
H5Gget_info, 88
H5Gget_info_by_idx, 88
h5gget_info_by_idx_f, 88
H5Gget_info_by_name, 88
h5gget_info_by_name_f, 88
h5gget_info_f, 88
H5Gget_linkval, 88
H5Gget_num_objs, 88
H5Gget_objinfo, 89
h5gget_obj_info_idx_f, 88
H5Gget_objname_by_idx, 89
H5Gget_objtype_by_idx, 89
H5Giterate, 89
H5Glink, 88
H5Glink2, 88
H5Gmove, 89
H5Gmove2, 89
h5gn_members_f, 88
H5Gopen, 39,40,87,93
h5gopen_f, 87
H5Gset_comment, 89
H5Gunlink, 89,250
H5I, 22
H5L, 22
H5Lcreate, 104
H5Lcreate_external, 88,96
h5lcreate_external_f, 88
H5Lcreate_hard, 88,95
h5lcreate_hard_f, 88
H5Lcreate_soft, 88,96
h5lcreate_soft_f, 88
H5Lcreate_ud, 88
H5Ldelete, 89,94,95,96,104,139,250
h5ldelete_f, 89
H5Lget_info, 89,97
h5lget_info_f, 89
H5Lget_name_by_idx, 89
h5lget_name_by_idx_f, 89
H5Lget_val, 88
H5L_info_t, 97
H5Literate, 88,89,97
H5Literate_by_name, 89
h5literate_by_name_f, 89
h5literate_f, 89
H5Lmove, 89,96,97
h5lmove_f, 89
h5ls, 272
H5LTdtype_to_text, 192,261
H5LTtext_to_dtype, 192,261,263,264
H5Lvisit, 89,97
H5O, 22
H5Oget_comment, 89
H5Oget_info, 89,95,97,313,314
H5Oget_info_by_idx, 89
H5Oget_info_by_name, 89
h5oget_info_by_name_f, 89
H5O_info_t, 97
H5open, 50
h5open_f, 50
H5Oset_comment, 89
H5Ovisit, 89,97
h5ovisit_f, 89
H5P, 22
H5Padd_merge_committed_dtype_path, 349
H5Pall_filters_avail, 90,107
H5P_ATTRIBUTE_CREATE, 339
H5Pclose, 115,155,160,168,172,317,348
h5pclose_f, 348
H5Pcopy, 344,348
h5pcopy_f, 348
H5Pcreate, 36,39,48,49,58,75,115,120,141,
142,154,158,166,170,217,239,316,344,348
h5pcreate_f, 219,348
H5P_CRT_ORDER_TRACKED, 86
H5P_DATASET_ACCESS, 339
H5P_DATASET_CREATE, 14,36,39,115,141,142,
154,158,166,170,239,339,344,346
H5P_DATASET_XFER, 14,120,217,339
H5P_DATASET_XFER_F, 219
H5P_DATATYPE_ACCESS, 339
H5P_DATATYPE_CREATE, 339
H5Pdcpl.c, 163
H5P_DEFAULT, 22,31,343,351
H5P_FILE_ACCESS, 13,48,49,58,75,316,339
H5P_FILE_CREATE, 13,48,58,75,339
H5P_FILE_MOUNT, 14,339,351
H5Pfill_value_defined, 106
H5Pfree_merge_committed_dtype_paths, 349
H5Pget_alignment, 54
h5pget_alignment_f, 54
H5Pget_alloc_time, 107
h5pget_alloc_time_f, 107
H5Pget_attr_creation_order, 311,348
h5pget_attr_creation_order_f, 311,348
H5Pget_attr_phase_change, 311,348
h5pget_attr_phase_change_f, 311,348
H5Pget_btree_ratios, 109
h5pget_btree_ratios_f, 109
H5Pget_buffer, 108
h5pget_buffer_f, 108
HDF5User’sGuide Index
TheHDFGroup 359
H5Pget_cache, 54,60
h5pget_cache_f, 54
H5Pget_char_encoding, 91,108,192,311,350
h5pget_char_encoding_f, 91,108,192,311,350
H5Pget_chunk, 106,347
H5Pget_chunk_cache, 108
h5pget_chunk_cache_f, 108
h5pget_chunk_f, 106
H5Pget_class, 348
h5pget_class_f, 348
H5Pget_copy_object, 349
h5pget_copy_object_f, 349
H5Pget_create_intermediate_group, 91,350
h5pget_create_intermediate_group_f, 350
H5Pget_data_transform, 108
h5pget_data_transform_f, 108
H5Pget_driver, 55,65
h5pget_driver_f, 55
H5Pget_driver_info, 55
H5Pget_dxpl_mpio, 109
h5pget_dxpl_mpio_f, 109
H5Pget_edc_check, 108
h5pget_edc_check_f, 108
H5Pget_elink_file_cache_size, 54,91
H5Pget_est_link_info, 90
h5pget_est_link_info_f, 90
H5Pget_external, 108
H5Pget_external_count, 108
h5pget_external_count_f, 108
h5pget_external_f, 108
H5Pget_family_offset, 54
H5Pget_fapl_core, 55,69
h5pget_fapl_core_f, 55
H5Pget_fapl_direct, 55,66
h5pget_fapl_direct_f, 55
H5Pget_fapl_family, 55,70
h5pget_fapl_family_f, 55
H5Pget_fapl_log, 67
H5Pget_fapl_mpio, 55,73
h5pget_fapl_mpio_f, 55
H5Pget_fapl_mpiposix, 56
h5pget_fapl_mpiposix_f, 56
H5Pget_fapl_multi, 56,72
h5pget_fapl_multi_f, 56
H5Pget_fclose_degree, 54
h5pget_fclose_degree_f, 54
H5Pget_file_image, 53
h5pget_file_image_f, 53
H5Pget_fill_time, 107
h5pget_fill_time_f, 107
H5Pget_fill_value, 106,240
h5pget_fill_value_f, 106
H5Pget_filter, 90,107
H5Pget_filter_by_id, 90,107
h5pget_filter_by_id_f, 90,107
h5pget_filter_f, 90,107
H5Pget_gc_references, 54
h5pget_gc_references_f, 54
H5Pget_hyper_vector_size, 109
h5pget_hyper_vector_size_f, 109
H5Pget_istore_k, 53,59
h5pget_istore_k_f, 53
H5Pget_layout, 106,346
h5pget_layout_f, 106
H5Pget_libver_bounds, 54
H5Pget_link_creation_order, 91
h5pget_link_creation_order_f, 91
H5Pget_link_phase_change, 90
h5pget_link_phase_change_f, 90
H5Pget_mcdt_search_cb, 349
H5Pget_mdc_config, 54
H5Pget_meta_block_size, 54,60
h5pget_meta_block_size_f, 54
H5Pget_mpio_actual_chunk_opt_mode, 343
H5Pget_multi_type, 56,110
H5Pget_nfilters, 90,107
h5pget_nfilters_f, 90,107
H5Pget_nlinks, 91
h5pget_nlinks_f, 91
H5Pget_obj_track_times, 348
h5pget_obj_track_times_f, 348
H5Pget_shared_mesg_index, 53
H5Pget_shared_mesg_nindexes, 53
H5Pget_shared_mesg_phase_change, 53
H5Pget_sieve_buf_size, 54,60
h5pget_sieve_buf_size_f, 54
H5Pget_sizes, 53,59
h5pget_sizes_f, 53
H5Pget_small_data_block_size, 55,110
h5pget_small_data_block_size_f, 55,110
H5Pget_sym_k, 53,59
h5pget_sym_k_f, 53
H5Pget_type_conv_cb, 109,192
H5Pget_userblock, 53,59
h5pget_userblock_f, 53
H5Pget_version, 53,59
h5pget_version_f, 53
H5Pget_vlen_mem_manager, 109
H5P_GROUP_ACCESS, 339
H5P_GROUP_CREATE, 316,339
Index HDF5User’sGuide
360 TheHDFGroup
H5P_LINK_ACCESS, 339
H5P_LINK_CREATE, 339
H5Pmodify_filter, 90,107
h5pmodify_filter_f, 90,107
H5P_OBJECT_COPY, 339
H5P_OBJECT_CREATE, 339
H5Premove_filter, 90,107
h5premove_filter_f, 90,107
H5Pset_alignment, 54,60
h5pset_alignment_f, 54
H5Pset_alloc_time, 107
h5pset_alloc_time_f, 107
H5Pset_attr_creation_order, 311,348
h5pset_attr_creation_order_f, 348
H5Pset_attr_phase_change, 311,315,316,348
h5pset_attr_phase_change_f, 311,348
H5Pset_btree_ratios, 109
h5pset_btree_ratios_f, 109
H5Pset_buffer, 108,120
h5pset_buffer_f, 108
H5Pset_cache, 54,60
h5pset_cache, 54
H5Pset_char_encoding, 86,91,108,192,311,318,
350
h5pset_char_encoding_f, 91,108,192,311,350
H5Pset_chunk, 36,39,106,154,158,166,347
H5Pset_chunk_cache, 108
h5pset_chunk_cache_f, 108
h5pset_chunk_f, 106
H5Pset_copy_object, 349
h5pset_copy_object_f, 349
h5pset_create_inter_group_f, 91
H5Pset_create_intermediate_group, 91,342,350
h5pset_create_intermediate_group_f, 350
H5Pset_create_intermediate_groups, 352
H5Pset_data_transform, 108
h5pset_data_transform_f, 108
H5Pset_deflate, 39,90,106,126
h5pset_deflate_f, 90,106
H5Pset_driver, 55
H5Pset_dxpl_mpio, 109
H5Pset_dxpl_mpio_chunk_opt, 109,343
H5Pset_dxpl_mpio_chunk_opt_num, 109,343
H5Pset_dxpl_mpio_chunk_opt_ratio, 109,343
H5Pset_dxpl_mpio_collective_opt, 109
h5pset_dxpl_mpio_f, 109
H5Pset_edc_check, 108,343
h5pset_edc_check_f, 108
H5Pset_elink_cb, 329
H5Pset_elink_file_cache_size, 54,91,96
H5Pset_est_link_info, 90
h5pset_est_link_info_f, 90
H5Pset_external, 107,140,141,142
h5pset_external_f, 107
H5Pset_family_offset, 54
h5pset_family_offset_f, 54
H5Pset_fapl_core, 55,64,69,352
h5pset_fapl_core, 55
H5Pset_fapl_direct, 55,63,66
h5pset_fapl_direct_f, 55
H5Pset_fapl_family, 55,64,70,342,352
h5pset_fapl_family, 55
H5Pset_fapl_log, 55,63,67
H5Pset_fapl_mpi, 75
H5Pset_fapl_mpio, 55,65,73,342
h5pset_fapl_mpio_f, 55
H5Pset_fapl_mpiposix, 55
h5pset_fapl_mpiposix_f, 55
H5Pset_fapl_multi, 56,64,72
h5pset_fapl_multi_f, 56
H5Pset_fapl_sec2, 56,63,66,342
h5pset_fapl_sec2_f, 56
H5Pset_fapl_split, 56,64,72,342,352
h5pset_fapl_split_f, 56
H5Pset_fapl_stdio, 49,56,63,68
H5Pset_fapl_stdio_f, 56
H5Pset_fapl_windows, 56,63,68
H5Pset_fclose_degree, 25,54
h5pset_fclose_degree_f, 54
H5Pset_file_image, 53
h5pset_file_image_f, 53
H5Pset_fill_time, 106
h5pset_fill_time_f, 106
H5Pset_fill_value, 106,115,166,239
h5pset_fill_value_f, 106
H5Pset_filter, 44,90,107
H5Pset_filter_callback, 108
h5pset_filter_f, 90,107
H5Pset_fletcher32, 90,107,342,343
h5pset_fletcher32_f, 90,107
H5Pset_gc_references, 54,61
h5pset_gc_references_f, 54
H5Pset_hyper_vector_size, 109
h5pset_hyper_vector_size_f, 109
H5Pset_istore_k, 53,59
h5pset_istore_k_f, 53
H5Pset_layout, 106,342,346
h5pset_layout_f, 106
H5Pset_libver_bounds, 54,86,316
h5pset_libver_bounds_f, 54
HDF5User’sGuide Index
TheHDFGroup 361
H5Pset_link_creation_order, 86,91
h5pset_link_creation_order_f, 91
H5Pset_link_phase_change, 86,90
h5pset_link_phase_change_f, 90
H5Pset_mcdt_search_cb, 349
H5Pset_mdc_config, 54
H5Pset_meta_block_size, 54,60
h5pset_meta_block_size_f, 54
H5Pset_multi_type, 56,109
H5Pset_nbit, 107,149,158
h5pset_nbit_f, 107
H5Pset_nlinks, 90
h5pset_nlinks_f, 90
H5Pset_obj_track_times, 349
h5pset_obj_track_times_f, 349
H5Pset_preserve, 216,217
h5pset_preserve_f, 219
H5Pset_scaleoffset, 107,163,166,170
h5pset_scaleoffset_f, 107
H5Pset_shared_mesg_index, 53
h5pset_shared_mesg_index_f, 53
H5Pset_shared_mesg_nindexes, 53
h5pset_shared_mesg_nindexes_f, 53
H5Pset_shared_mesg_phase_change, 53
H5Pset_shuffle, 107,126
h5pset_shuffle_f, 107
H5Pset_sieve_buf_size, 54,60
h5pset_sieve_buf_size_f, 54
H5Pset_sizes, 53,59,75
h5pset_sizes_f, 53
H5Pset_small_data_block_size, 55,110
h5pset_small_data_block_size_f, 55,110
H5Pset_sym_k, 53,59
h5pset_sym_k_f, 53
H5Pset_szip, 107
h5pset_szip_f, 107
H5Pset_type_conv_cb, 109,192,254,329
H5Pset_userblock, 53,59
h5pset_userblock_f, 53
H5Pset_vlen_mem_manager, 109,139,229
H5P_STRING_CREATE, 339
H5Pvlen_mem_manager, 231
H5R, 22
H5Rcreate, 234,235,295,296,298
H5R_DATASET_REGION, 104,296,299
H5Rdefrerence, 292
H5R_dereference, 298
H5Rdereference, 104,235,299
h5repack, 139
h5repart, 71
H5Rget_region, 299
H5Rget_select, 292
H5Rget_space, 298
H5R_OBJECT, 104,235
H5S, 22
H5S_ALL, 25,26,118,119,120,121,155,159,160,
167,184
H5Sclose, 24,41,114,115,155,160,168,172,265
h5sclose_f, 265
H5Scopy, 265
h5scopy_f, 265
H5Screate, 40,41,265,270,271
h5screate_f, 265,270
H5Screate_simple, 24,30,31,36,39,93,112,114,
115,153,157,165,169,266,270,271,284,286,
289,291,297,299,317
h5screate_simple_f, 266,270,272
H5Sdecode, 266
h5sdecode_f, 266
H5Sencode, 266
h5sencode, 266
H5set_free_list_limits, 50
h5set_free_list_limits_f, 50
H5Sextent_copy, 266
h5sextent_copy_f, 266
H5Sextent_equal, 266
h5sextent_equal_f, 266
H5Sget_select_bounds, 267,299
h5sget_select_bounds_f, 267
H5Sget_select_elem_npoints, 267,299
h5sget_select_elem_npoints_f, 267
H5Sget_select_elem_pointlist, 267,299
h5sget_select_elem_pointlist_f, 267
H5Sget_select_hyper_blocklist, 267,299
h5sget_select_hyper_blocklist_f, 267
H5Sget_select_hyper_nblocks, 267,299
h5sget_select_hyper_nblocks_f, 267
H5Sget_select_npoints, 267,299
h5sget_select_npoints_f, 267
H5Sget_select_type, 267
h5sget_select_type_f, 267
H5Sget_simple_extent_dims, 29,32,266,276,299
h5sget_simple_extent_dims_f, 266
H5Sget_simple_extent_ndims, 29,266,275,299
h5sget_simple_extent_ndims_f, 266
H5Sget_simple_extent_npoints, 266
h5sget_simple_extent_npoints_f, 266
H5Sget_simple_extent_type, 266
h5sget_simple_extent_type_f, 266
H5Sis_simple, 266,275
Index HDF5User’sGuide
362 TheHDFGroup
h5sis_simple_f, 266
H5S_MAX_RANK, 227
H5S_NULL, 268,270
H5Soffset_simple, 267
h5soffset_simple_f, 267
H5S_SCALAR, 41,268
H5S_SELECT_ALL, 118,120
H5Sselect_all, 267
h5sselect_all_f, 267
H5Sselect_elements, 267,281,289,291,297
h5sselect_elements_f, 267
H5Sselect_hyperslab, 29,30,31,267,283,284,
285,288,289,296
h5sselect_hyperslab_f, 267
H5Sselect_none, 267,296,297
h5sselect_none_f, 267
H5S_SELECT_OR, 288,289,296
H5S_SELECT_SET, 29,30,31,283,284,285,288,
289,291,296
H5Sselect_valid, 267
h5sselect_valid_f, 267
H5Sset_extent_none, 266
h5sset_extent_none_f, 266
H5Sset_extent_simple, 266,271
h5sset_extent_simple_f, 266
H5S_SIMPLE, 268
H5S_UNLIMITED, 35,36,269
H5T, 22
H5T_ARRAY, 195,221,226,245,249,264
H5Tarray_create, 191,200,227,243
h5tarray_create_f, 191
H5T_BITFIELD, 195,221
H5Tclose, 24,114,115,155,160,168,172,182,
187,193,194,253,324
h5tclose_f, 187
H5Tcommit, 186,194,250,252,253
H5Tcommit_anon, 186
h5tcommit_anon_f, 186
h5tcommit_f, 186
H5Tcommitted, 186,194,253
h5tcommitted_f, 186
H5Tcompiler_conv, 187
h5tcompiler_conv_f, 187
H5T_COMPOUND, 33,34,193,195,209,212,215,
218,221,222,239,243,244,245,249,257,259
H5T_COMPOUND_F, 210
H5Tconvert, 187,237,329
h5tconvert_f, 187
H5Tcopy, 24,42,114,115,152,157,165,169,179,
183,186,193,194,199,200,204,205,207,213,
214,234,243,250,254,257
h5tcopy_f, 186,214
H5T_CRAY_F64, 181
H5Tcreate, 34,186,193,194,200,209,212,213,
215,217,222,224,225,236,239,243,244,257,
259
h5tcreate_f, 186,210,219,220
H5T_C_S1, 179,181,234,243,245
H5T_CSET_ASCII, 198,204,245
H5T_CSET_UTF8, 198,204
H5Tdecode, 187
h5tdecode_f, 187
H5Tdetect_class, 187
H5T_DIR_DEFAULT, 223
H5Tencode, 187
h5tencode, 187
H5T_ENUM, 193,195,221,236
H5Tenum_create, 189,200
h5tenum_create_f, 189
H5Tenum_insert, 189,236
h5tenum_insert_f, 189
H5Tenum_nameof, 189,199
h5tenum_nameof_f, 189
H5Tenum_valueof, 189,199
h5tenum_valueof_f, 189
H5Tequal, 186,193,194,237
h5tequal_f, 186
H5Tfind, 187
H5T_FLOAT, 195,201,221
H5Tget_array_dims, 191,199,249
h5tget_array_dims_f, 191
H5Tget_array_ndims, 191,199
h5tget_array_ndims_f, 191
H5Tget_class, 32,186,249
h5tget_class_f, 186
H5Tget_create_plist, 187
h5tget_create_plist_f, 187
H5Tget_cset, 189,198
h5tget_cset_f, 189
H5Tget_ebias, 188,196
h5tget_ebias_f, 188
H5Tget_fields, 188,196
h5tget_fields_f, 188
H5Tget_inpad, 189,197
h5tget_inpad_f, 189
H5Tget_member_class, 190,198,221
h5tget_member_class_f, 190
H5Tget_member_index, 190
HDF5User’sGuide Index
TheHDFGroup 363
h5tget_member_index_f, 190
H5Tget_member_name, 190,198,221,249
h5tget_member_name_f, 190
H5Tget_member_offset, 190,199,249
h5tget_member_offset_f, 190
H5Tget_member_type, 190,194,199,249,250
h5tget_member_type_f, 190
H5Tget_member_value, 190,199
h5tget_member_value_f, 190
H5Tget_native_type, 187,222,223
h5tget_native_type_f, 187
H5Tget_nmembers, 190,198,221,249
h5tget_nmembers_f, 190
H5Tget_norm, 189,197
h5tget_norm_f, 189
H5Tget_offset, 188
h5tget_offset_f, 188
H5Tget_order, 185,187
h5tget_order_f, 187
H5Tget_pad, 188
h5tget_pad_f, 188
H5Tget_precision, 188
h5tget_precision_f, 188
H5Tget_sign, 185,188,196
h5tget_sign_f, 188
H5Tget_size, 32,185,187,249
h5tget_size_f, 187,210,219
H5Tget_strpad, 189,198
h5tget_strpad_f, 189
H5Tget_super, 187,194,199,249,250
h5tget_super_f, 187
H5Tget_tag, 191,198
h5tget_tag_f, 191
H5Tget_type, 250
H5T_IEEE_F32BE, 157,181
H5T_IEEE_F32LE, 207,245
H5T_IEEE_F64LE, 181,245
H5Tinsert, 34,190,209,212,213,217,222,224,
225,239,243,244,257,259
h5tinsert_f, 190,210,220
H5T_INTEGER, 32,175,184,185,195,221
H5T_INTEL_B64, 181
H5Tis_variable_str, 191
h5tis_variable_str_f, 191
H5Tlock, 186,194,200,251
H5T_NATIVE_CHAR, 213,233
H5T_NATIVE_DOUBLE, 34,209,210,213
H5T_NATIVE_FLOAT, 146,159,160,169,170,171
H5T_NATIVE_INT, 24,25,31,36,39,41,42,93,
114,115,118,120,121,145,152,165,166,167,
168,174,181,184,200,204,205,213,251,255
H5T_NATVE_INT, 120
H5T_NORM_IMPLIED, 197,203
H5T_NORM_MSBSET, 197,203
H5T_NORM_NONE, 197,203
H5T_OPAQUE, 193,195,221,263
H5Topen, 186,193,194,250,253
h5topen_f, 186
H5T_ORDER_BE, 153,185,195,201
H5T_ORDER_LE, 24,32,114,115,185,195,201,
204
H5Tpack, 190,213,214,257
h5tpack_f, 190,214
H5T_PAD_NONE, 197,203
H5T_PAD_ONE, 195,197,202,203,207
H5T_PAD_ZERO, 195,197,202,203,207
H5T_REFERENCE, 195,221
H5T_REGION_OBJ, 234
H5Tregister, 188
H5T_SDT_I32LE, 207
H5T_SELECT_OR, 278
H5T_SELECT_SET, 278
H5Tset_cset, 189
h5tset_cset_f, 189
H5Tset_ebias, 157,189,202
h5tset_ebias_f, 189
H5Tset_fields, 157,188,202,207
h5tset_fields_f, 188
H5Tset_inpad, 189,203,207
h5tset_inpad_f, 189
H5Tset_norm, 189,203
h5tset_norm_f, 189
H5Tset_offset, 152,157,188,202,207
h5tset_offset_f, 188
H5Tset_order, 24,114,115,187,201,204
h5tset_order_f, 187
H5Tset_pad, 188,202,207
h5tset_pad_f, 188
H5Tset_precision, 152,157,188,200,201,204,
207
h5tset_precision_f, 188
H5Tset_sign, 188,200,202
h5tset_sign_f, 188
H5Tset_size, 42,157,188,201,234,243
h5tset_size_f, 188
H5Tset_strpad, 189
h5tset_strpad_f, 189
H5Tset_tag, 191,238
Index HDF5User’sGuide
364 TheHDFGroup
h5tset_tag_f, 191
H5T_SGN_2, 185,196,202
H5T_SGN_NONE, 185,196,200,202
H5T_STD_BE32, 254
H5T_STD_I32BE, 181,183,239
H5T_STD_I32LE, 181,245
H5T_STD_I8LE, 245
H5T_STD_REF_DSETREG, 292,297,298
H5T_STD_REF_OBJ, 235
H5T_STD_ROBJ, 181
H5T_STD_U16BE, 181
H5T_STRING, 195,221,233,245,263
H5T_STR_NULLPAD, 198,204
H5T_STR_NULLTERM, 198,204
H5T_STR_SPACEPAD, 198,204
H5Tunregister, 188
H5T_VARIABLE, 234
H5T_VLEN, 195,221
H5Tvlen_create, 191,200,229
h5tvlen_create_f, 191
H5_VERSION_GE, 51
H5_VERSION_LE, 51
H5Z, 22
H5Z_can_apply_nbit, 149
H5Z_filter_nbit, 148,149,150
H5Zregister, 44
H5Zscaleoffset.c, 163
H5Z_set_local_nbit, 148,149
H5Z_set_parms_array, 149,150
H5Z_set_parms_atomic, 149,150
H5Z_set_parms_compound, 149,150
H5Z_set_parms_noopdatatype, 149,150
H5Z_set_parms_nooptype, 149,150
hardlink, 83
hdset_reg_ref_t, 295
HOFFSET, 34,209
hvl_t, 229
hyperslab, 26,28,278
I
immutable, 251,341
immutabletransient, 251
indexed, 86
L
library, 2
link, 94
linkobjects, 81
little‐endian, 195
loc_id, 312,314,315
lowversionbound, 86
low‐levelfiledrivers, 62
M
majormessage, 324
maxdims, 112
mem_type_id, 312
minormessage, 324
MPI_COMM_WORLD, 75
N
nameddatatype, 10,81
namedobject, 7,81
NATIVE, 181
n‐bitcompression, 147
n‐bitdatatype, 143
n‐bitdecompression, 148
n‐bitfilter, 143,147
no‐opdatatypes, 147
nulldataspace, 268
O
objectidentifier, 17,83
object_id, 314
object_info, 314
offset, 279
P
paddingbits, 145
permanent, 341
pipeline, 122
polygonlists, 228
POSIX, 63
post‐compression, 162
pre‐compression, 161
primarydataobject, 307
programmingmodel, 2
property, 341
propertylist, 340
propertylistclass, 339
R
raggedarrays, 228
rawdata, 103
regionreference, 292
HDF5User’sGuide Index
TheHDFGroup 365
repartition, 71
S
scalardataspace, 268
scale_factor, 162
scale‐offsetcompression, 160
scale‐offsetfilter, 160
simpledataspace, 268
softlinks, 83
storagelayout, 103
storagemodel, 1
storeddata, 2
stride, 279
superblock, 58
T
transferproperties, 342
transient, 250,342
U
userblock, 59
V
variable‐length, 176
vector, 30
VFL, 61
virtualfilelayer, 15,61,123
vl_info_t, 139
vl_t, 229,231,233,234
Z
zlib, 126
Index HDF5User’sGuide
366 TheHDFGroup

Navigation menu