DDL and ADTF 3

Table of Contents

• Usage of Stream Meta Type "adtf/default"
• Methods to create valid DDL and "adtf/default"
  • Creating a DDL file and use it
    • MD Generator Tool
    • Detailed description of MD Generator Tool
      • Options
    • MD Generator Tool from Header
    • Detailed description of MD Generator Tool from Header
    • Read the DDL file and setup Stream Type
  • Define DDL Description in code and use it
    • Use structure class
    • Use structure<> template
    • Define a huge string for DDL Description
• Serialization of "adtf/default" Samples
• DDL Access to the Sample Content

The DDL (Data Definition Language) is to describe memory content in a formal way.

So, it is possible to provide functionality, executables and utilities those are able to access and display data in a generic way:

• Qt5 Media Description Display
• Demo Time Trigger Plugin
• Media Description Dumper Plugin
• Media Description Service
• Qt5 Scripting Filter Plugins
• DDL Editor

The DDL will have some advantages over other ways to describe data like IDL (Interface Description Language), Protocol Buffers or FlatBuffers:

• It describes only the layout of data in a common formal way! Each other description language format for describing data can be created.
• This kind of data description makes it usable on and between cross-platforms, compiler-, architecure- and memory-layout independent machines.
• The DDL does not force any specific method of serialization and deserialisation.
• Describing data in DDL does not force you to use any code generation or toolings if you do not want that.
• Reading data memory and the values via DDL Access methods does not mean, that the creator of that data must have created it with DDL writing methods.
• You can dynamically describe your data in runtime.
• All kinds of basic datatypes existing in C/C++ are supported. No restriction on that.
• Direct access to structured content via C++-Code, the DDL will only describe that content.

There are also some disadvantages on DDL:

• DDL is old fashioned XML, but it makes no sense to change to famous JSON, break the users code and confuse them.
• No optimization while compile time, access methods will dynamically address values only in runtime.

Usage of Stream Meta Type "adtf/default"

The most important Stream Type instance within ADTF uses the default Stream Meta Type "adtf/default". (See also Stream Type and Stream Meta Type for a common definition)
This Meta Type is define in stream_meta_type_default. It will use 3 important properties:

md_struct

• strMDStructProperty
• This property contains a complex type name / struct name out of the complete self-contained DDL definition in strMDDefinitionsProperty. See also Demo Media Description Code Generation Filters Plugin.

md_definitions

• strMDDefinitionsProperty
• This property contains a complete self-contained DDL definition which should contain the description of the complex type set in strMDStructProperty. See stream_type_default for creating the type with the help of the template adtf::mediadescription::description. See also Demo Media Description Code Generation Filters Plugin.

md_data_serialized

• strMDDataSerialized
• This property defines wether the sample data are serialized (true) or not serialized (false). By default the value is set to false! This determines that the DDL definitons for deserialized interpretation is used. For more information on serialized/deserialized see page_ddl_specification and Serialization of "adtf/default" Samples.

Note

It is very important to know, that many dynamic DDL functionality in ADTF does not require the default Stream Type itself but uses the properties "md_struct" and "md_definitions". See i.e. the compatibiliy functionality: Stream Meta Type "adtf/default".

In ADTF 2 the DDL File was an optional side-by-side file description next to the ADTF DAT File. It described the content of the DAT File.
Since the ADTF DAT File in ADTF 3 contains the complete DDL description if "adtf/default" Meta Type is used, only the units, datatypes, enums and structs tags are relevant. Within ADTF 3 we do not use the streammetatypes or streams tags anymore which are mentioned within page_ddl_specification.

Methods to create valid DDL and "adtf/default"

Creating a DDL file and use it

It is possible to create a DDL file before you define any structured data within your code. To make sure the resulting DDL file is valid, we recommend to use the DDL Editor.

MD Generator Tool

ADTF provides a tool that creates C++ headers containing structure definitions and access classes from a given ADTF Media Description file (*.description).

If you are using the adtf_add_filter, adtf_add_streaming_service, adtf_add_system_service CMake macros, you only need to specify your description file within the sources of your target:

cmake_minimum_required(VERSION 3.18 FATAL_ERROR)
project(demo_code_generation)
find_package(ADTF COMPONENTS filtersdk)
 
adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description)

This will generate a header named after your description file, that you can include in your source files. In the above case:

#include <mytypes.h>

You may use following code within your filter if the DDL file contains any valid complex type (struct type) named "tMyType":

using namespace adtf::streaming;
using namespace adtf::mediadescription;
adtf::ucom::object_ptr<IStreamType> pMyDDLType = adtf::ucom::make_object_ptr<stream_type_default<tMyType>>();

If you want the structure definitions to be generated in a namespace use the "MD_NAMESPACE" argument:

adtf_add_filter(myfilter
    myfilter.cpp
    mytypes.description
    MD_NAMESPACE a::b::c)

If for some reasons you cannot use these macros, there is the adtf_md_generate_cpp CMake macro provided that will add header generation to existing targets.

The executable of the tool is accessible via the imported adtf::mdgen target. A non target based CMake macro is provided via adtf_md_generate_from_description.

An example of the usage of the generation facilities can be found at Demo Media Description Code Generation Filters Plugin.

Note

The generated header file will be directly accessible after the Filter, Streaming Service or System Service has been built for the first time. E.g Qt moc-files

Detailed description of MD Generator Tool

To describe the MD Generator Tool of the above chapter in more detail we need to tell, that it will create exactly two header files and one cpp-file.

mytypes_description.h //-> containing the generated structs, enums out of the description file
mytypes.h             //-> containing the md_sample_data template specializations to access
                      //   the generated string description and the data of one sample via comfortable access functions
mytypes.cpp           //-> implementation of the access functions

Options

It is possible to use additional MD_ARGUMENTS within the CMake macro as follows:

Instead of using the stdtypes ( bool, int8_t, uint8_t, ... ) in generated header files this will lead to a usage of adtftypes ( tBool, tInt8, tUInt8, ...):

adtf_add_filter(myfilter myfilter.cpp mytypes.description MD_ARGUMENTS --use-adtftypes)

Instead of using enum class for enumerations in generated header files this will lead to a usage of enum only:

adtf_add_filter(myfilter myfilter.cpp mytypes.description MD_ARGUMENTS --no-scoped-enum)

Instead of using the global namepace qualifier for each type usage this will drop the leading '::':

adtf_add_filter(myfilter myfilter.cpp mytypes.description MD_ARGUMENTS --no-global-namespace)

Instead of using alignment for positioning the elements, this will use padding bytes only:

adtf_add_filter(myfilter myfilter.cpp mytypes.description MD_ARGUMENTS --padded-only)

MD Generator Tool from Header

ADTF provides a additional toolset that creates a description out of struct definitions within a simple C++ header and the necessary access classes.

To do so there are 3 tools provided:

• a castxml tool that is able to parse a C++ header file and write them into a xml-based temporary castxml file
• a header2ddl tool using this castxml file and generates the description file
• the MD Generator Tool that generates the access classes for the given description file

If you are using the adtf_add_filter, adtf_add_streaming_service, adtf_add_system_service CMake macros, you only need to specify your header files to generate descriptions for within the MD_HEADERS of your target:

cmake_minimum_required(VERSION 3.18 FATAL_ERROR)
project(demo_description_generation)
find_package(ADTF COMPONENTS filtersdk)
 
adtf_add_filter(myfilter
    myfilter.cpp
    MD_HEADERS mytypes.h)

This will generate the temporary castxml and description file within your build folder. The MD Generator Tool will generate the necessary access template classes within a header, that you can include in your source files. In the above case:

#include <mytypes_mdgen.h>  //<== generated header with access classes will also include <mytypes.h>

Now you can use following code within your filter like the the example under MD Generator Tool with a complex type named "tMyType":

using namespace adtf::streaming;
using namespace adtf::mediadescription;
adtf::ucom::object_ptr<IStreamType> pMyDDLType = adtf::ucom::make_object_ptr<stream_type_default<tMyType>>();

If for some reasons you cannot use these macros, there is the adtf_md_generate_cpp CMake macro provided that will add header generation to existing targets.

A non target based CMake macro is provided via adtf_md_generate_from_header.

An example of the usage of the generation facilities can be found at Demo Media Description Generation Filters Plugin.

Note

The generated header file will be directly accessible after the Filter, Streaming Service or System Service has been built for the first time. E.g Qt moc-files

Detailed description of MD Generator Tool from Header

To describe the MD Generator Tool of the above chapter in more detail we need to tell, that it will create exactly two header files and one cpp-file.

mytypes.h             //-> MD_HEADERS source file containing the "hand written" structure definitions to generate the description for
mytypes.castxml       //-> generated, temporary castxml type information based on clang parser 
mytypes.description   //-> generated, temporary DDL media description file with the struct definitions out of 'mytypes.h'
mytypes_mdgen.h       //-> generated file containing the md sample access functions for the structs, enums out of 'mytypes.h'
mytypes_mdgen.cpp     //-> implementation of the access functions

Options

It is possible to use MD_HEADERS_ARGUMENTS also combined with the MD_ARGUMENTS like described above within the CMake macro as follows:

Instead generating all struct types only a description for the type 'tMyType' is generated:

adtf_add_filter(myfilter myfilter.cpp MD_HEADERS mytypes.h MD_HEADERS_ARGUMENTS --types tMyType) adtf_add_filter(myfilter myfilter.cpp MD_HEADERS mytypes.h MD_HEADERS_TYPES tMyType)

Instead of using the adtftypes ( tBool, tInt8, tUInt8, ...) in generated description files this will lead to a forced usage of stdtypes ( bool, int8_t, uint8_t, ... ):

adtf_add_filter(myfilter myfilter.cpp MD_HEADERS mytypes.h MD_HEADERS_ARGUMENTS --force-datatypes stdtypes)

Instead of using the current platform dependent byte order with the generated description, use the given one: LE or BE

adtf_add_filter(myfilter myfilter.cpp MD_HEADERS mytypes.h MD_HEADERS_ARGUMENTS --byteorder BE)

Read the DDL file and setup Stream Type

It is also possible to read a file in code and create the description while runtime dynamically. To do so, please use the ddl::DDFile functionality.

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    std::string oDDLFileFromConfig = "./my_type.description"; 
 
    //********************************
    //new code to import a DDL file
    //********************************
    auto oDDLFile = ddl::DDFile::fromXMLFile(oDDLFileFromConfig);
    
    //resolve the struct to a small description with the DDString functionality 
    //and create the type from imported file 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>("tMyTypeFromFile", ddl::DDString::toXMLString("tMyTypeFromFile", oDDLFile).c_str());
 

Define DDL Description in code and use it

Use structure class

It is possible to use the structure API provided via this ddl package. It will create the necessary DDL description while runtime and compile-time:

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    //we create a structure description and its default DDL type
    struct tMyType
    {
        int32_t m_nValue1;
        double m_fValue2;
        bool m_bArray[2];
    };
    auto oDescriptionForMyType = structure<>("tMyType")
                                 .Add<int32_t>("m_nValue1")
                                 .Add<double>("m_fValue2")
                                 .Add<tBool>("m_bArray", 2);
 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>(oDescriptionForMyType);
 

Use structure<> template

It is possible to use the type-reflection API provided via DDL package. It will create the necessary DDL description while runtime and compile-time:

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    //we create a structure description and its default DDL type
    struct tMyType
    {
        int32_t   m_nValue1;
        double m_fValue2;
        bool m_bArray[2];
    };
    auto oDescriptionForMyType = structure<tMyType>("tMyType")
                                .Add("m_nValue1", &tMyType::m_nValue1)
                                .Add("m_fValue2", &tMyType::m_fValue2)
                                .Add("m_bArray", &tMyType::m_bArray);
 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>(oDescriptionForMyType);
 

Define a huge string for DDL Description

This is a very old fashioned way to create your own DDL Decription. We do not recommend to use that, but for completeness it is specified here:

#include <adtffiltersdk/adtf_filtersdk.h>
//...

    using namespace adtf::ucom;
    using namespace adtf::streaming;
    using namespace adtf::mediadescription;
 
    //we create a structure description and its default DDL type
    std::string strMyTypeString;
    strMyTypeString += "<struct name=\"tMyType\" version=\"1\" arraysize=\"1\" alignment=\"4\">";
    strMyTypeString += "   <element type=\"tInt32\" name=\"m_nValue1\">";
    strMyTypeString += "       <deserialized alignment=\"4\"/>"; 
    strMyTypeString += "       <serialized bytepos=\"0\" byteorder=\"LE\"/>";
    strMyTypeString += "   </element>";
    strMyTypeString += "   <element type=\"tFloat64\" name=\"m_fValue2\">";
    strMyTypeString += "       <deserialized alignment=\"4\"/>";
    strMyTypeString += "       <serialized bytepos=\"4\" byteorder=\"LE\"/>";
    strMyTypeString += "   </element>";
    strMyTypeString += "   <element type=\"tBool\" name=\"m_bArray\" arraysize=\"2\">";
    strMyTypeString += "       <deserialized alignment=\"1\"/>";
    strMyTypeString += "       <serialized bytepos=\"8\" byteorder=\"LE\"/>";
    strMyTypeString += "   </element>";
    strMyTypeString += "</struct>";
 
    object_ptr<IStreamType> pDDLType = make_object_ptr<stream_type_default<>>("tMyType", strMyTypeString.c_str());
 

Serialization of "adtf/default" Samples

The DDL describes the data layout in 2 separate representations:

Deserialized Memory Respresentation

The memory representation of data is the definition of value positions within RAM. To calculate the positions of the values within the memory only alignment, type information and arraysize are relevant attributes. Keep in mind: The byteorder attribute is absolutely not relevant here, because the values exists always in the same byteorder as the platform and the used compiler for that binary.

Serialized Respresentation

The serialized representation of data is the definition of values in a file or network. Within ADTF context this is the case storaging data in a DAT File or while sharing data at the IPC Sinks and Sources. To access the positions of that values and transform from one to the other one the bytepos, byteorder, type and arraysize are relevant attributes only.

We separate these representations to access the values in a fast way in memory, when deserialized and the deserialized representation will assure that data are described in such a formal way to use it on every single platform. It enables exchanging data between different architectures (big endian, little endian) and processors (Intel, ARM) vie network and file.

DDL Access to the Sample Content

Reading access via md_sample_data

Please note, that you are encouraged to use the md_sample_data template to access sample data via Media Descriptions (DDL), instead of using a decoder directly.

Reading access via DDLDecoder

To read data from a Sample you only need to use the osborn::cStaticSampleDecoder or osborn::cSampleDecoder. See also The ADTF Codecs and SampleCodecs.

Writing access via output_sample_data

If you're generating output samples, please use the adtf::streaming::flash::output_sample_data template with your structure definition. There is no need to fill samples via a codec.

Writing access via DDLCodec

To write data to a Sample you can use the osborn::cStaticSampleCodec or osborn::cSampleCodec. See also The ADTF Codecs and SampleCodecs.