ADTF3 Guides - ADTF DAT Tool

ADTF DAT Tool

Since ADTF version 3.3.0 there is a extendable command line tool (based on ADTF File Library) to handle and access ADTFDAT Files for read and write using Processors and Readers, which can be implemented by the use of adtfdat_processing library.

Since ADTF version 3.7.0 the ADTF File Library package (as well as DDL and A_Util Package) has been removed from the 3rdparty folder (<ADTF_DIR>/3rdparty) and will be delivered within each other ADTF SDK Packages (<ADTF_DIR>/pkg). So you can use them the same way as common ADTF SDK Packages.

ADTFDAT files contain one or more streams representing recorded data. The ADTF DAT Tool operates on these streams in one of two ways:

Export Mode

In this mode the DAT Tool makes use of Processors to operate on streams inside the DAT file. Processors are written by software developers to do versatile data manipulation and extraction. The file ending of a processor is *.adtffileplugin.

Currently the ADTF delivery provides one such Processor to export the samples of a stream into a CSV file (see ADTF File Library). Another processor can be found in the programming examples of this guide which is capable of extracting images from a video stream.

To extract data from streams of an ADTF DAT file use the --export argument. Select the streams you want to extract by specifying the --stream argument. You can specify a processor for each stream with the --processorid argument. If you do not specify one explicitly, the first one that supports the stream is used. Properties of processors can be specified with the --property argument. The destination filename is specified with the --output argument. Here is an example that exports two streams:

          
adtf_dattool --plugin csv_processor.adtffileplugin --export test.dat --stream in1 --processorid csv --output test_in1.csv --stream in2 --output test_in2.csv

Apply the parameter --liststreams to query all information about a given input (DAT file, or any other supported input). To load additional plugins use the --plugin argument as often as you like.

Create Mode

In this mode you can select arbitrary streams (even from multiple DAT files) to create a new DAT file. To do so use the --create argument. The --input argument is used to specify an input file. The --readerid argument can be used to pass in the reader that should be used to read the file. If none is given, the first one that supports the file is used. Set the --start and --end arguments to select the range of the input that should be imported into the new DAT file. To shift the timstamps of all imported stream items apply the --offset parameter.
To select streams from an input, use the --stream argument. If you do not select one or more streams explicitly, all streams will be added. Streams can be renamed with the --name argument.
Use the --serializerid argument to choose the serializer of your liking. If not specified, sample_copy_serialization.serialization.adtf.cid will be used.
Here is an example that creates a new ADTF DAT file from two inputs:

        
adtf_dattool --create new.dat --input input1.dat --readerid adtfdat --stream in1 --serializerid special.serialization.adtf.cid --input input2.dat

Full Command Line Help

The following lists all options the tool supports. Just issue the following command: adtf_dattool.exe --help

        
usage:
  adtf_dattool  options

where options are:
  -?, -h, --help                         display usage information
  --progress                             Show progress.
  --skipstreamtypesandtriggers           Do not process stream types and
                                         triggers.
  --plugin <plugin>                      Load an additional plugin.
  --liststreams <file name>              List all available information about
                                         the given file.
  --export <file name>                   Export streams from the given file.
  --create <file name>                   Create a new dat file.
  --modify <file name>                   Modify an existing dat file.
  --fileversion <adtf2|adtf3|adtf3ns>    File Version of the created file.
                                         Creating ADTF 2 files is completely
                                         experimental!
  --input <source>                       Specifies an input for the new dat
                                         file or extension. In case of
                                         extensions data will be read from
                                         stdin if this is not specified.
  --readerid <reader id>                 The id of the reader implementation
                                         that should be used to open the last
                                         input source.
  --start <timestamp>                    Process only chunks of the last
                                         input source with timestamps larger
                                         than this (microseconds).
  --end <timestamp>                      Process only chunks of the last
                                         input source with timestamps smaller
                                         than this (microseconds).
  --offset <timestamp>                   This offset is added to all chunk
                                         timestamps of the last input source
                                         (microseconds).
  --start-ns <timestamp>                 Process only chunks of the last
                                         input source with timestamps larger
                                         than this (nanoseconds).
  --end-ns <timestamp>                   Process only chunks of the last
                                         input source with timestamps smaller
                                         than this (nanoseconds).
  --offset-ns <timestamp>                This offset is added to all chunk
                                         timestamps of the last input source
                                         (nanoseconds).
  --stream <stream name>                 Select a stream for export or when
                                         creating a new dat file.
  --name <stream name>                   Sets the name for the last specified
                                         stream that is used in the newly
                                         created dat file.
  --processorid <processor id>           The id of the processor
                                         implementation used for the last
                                         specified stream.
  --property <name=value>                Sets a property of the last
                                         specified stream or input.
  --extension <extension name>           Select an extension for export, when
                                         creating a new dat file or updating
                                         one.
  --output <file name>                   Sets the output file name for the
                                         last specified stream or extension.
                                         If not specified, the processor is
                                         free to choose one and in case of
                                         extensions data will be written to
                                         stdout
  --serializerid <serializer id>         The id of the serializer
                                         implementation used for the last
                                         specified stream.
  --userid <user id>                     Sets the user id of the last
                                         extension to be updated.
  --typeid <type id>                     Sets the type id of the last
                                         extension to be updated.
  --versionid <version id>               Sets the version id of the last
                                         extension to be updated.


-----------------
  ADTF DAT Tool
-----------------

With the help of this tool you can extract data from ADTF DAT files, create new ADTF DAT files
from various inputs or add file extensions in combination with the adtf_datdump tool
to an existing ADTF DAT file.

Use --liststreams to query all information about a given input (ADTF DAT file, or any other
supported input).

To load additional plugins use the --plugin argument as often as you like.

-----------
  EXPORT:  
-----------
To extract data from streams or file extensions of an ADTF DAT file use the --export argument.
Select the streams you want to extract by using the --stream argument. You can specify a processor
for each stream with the --processorid argument. If you do not specify one explicitly, the first one
that supports the stream is used. Properties of processors can be specified with the --property
argument. 

Use the --extension argument to specify the required file extension. The destination
filename is specified with the --output argument. Without --output argument the file extension data
will be written to stdout.

Examples:
---------
Here is an example that exports two streams:
adtf_dattool --plugin csv_processor.adtffileplugin --export test.dat --stream in1
--processorid csv --output test_in1.csv --stream in2 --output test_in2.csv

And one that exports extension data:
adtf_dattool --export test.dat --extension adtf_version --output adtf_version.txt

-------------
  CREATION:  
-------------
To create a new ADTF DAT file use the --create argument. The --input argument is used to specify an
input file. The --readerid argument can be used to specify the reader that should be used to read
the file. If none is specified, the first one that supports the file is used.

Use the --start and --end arguments to select the range of the input that should be
imported into the ADTF DAT file. Use the --offset parameter to shift the timstamps of all imported
stream items.

To select streams from an input, use the --stream argument. If you do not select one or more
streams neither a extension, all streams will be added. Streams can be renamed with the --name argument.

Use the --serializerid argument to choose the serializer of your liking. If not specified,
sample_copy_serialization_ns.serialization.adtf.cid will be used.

To select file extensions from an input, use the --extension argument. If you do not select one or
more file extensions explicitly, no extension will be added or updated from the source file.

Examples:
---------
Here is an example that creates a new ADTF DAT file from two inputs:
adtf_dattool --create new.adtfdat --input input1.dat --readerid adtfdat --stream in1 --serializerid
special.serialization.adtf.cid --input input2.dat

And one that copies a stream and two extensions from a source file
adtf_dattool --create new.adtfdat --input input1.dat --stream in1 --extension attached_files
--extension attached_files_configuration

------------------------
  MODIFYING:
------------------------
To modify an existing ADTF DAT File use the --modify argument
Currently there is only support for adding and updating extensions.

Examples:
---------
An example to put a files content into an extension:
adtf_dattool --modify existing.adtfdat --extension my_extension --input input_file.txt

Mind that if you do not specify an --input for an extension its data will be read from stdin:
tar cz /myfolder | adtf_dattool --modify existing.adtfdat --extension attached_files

or use:
adtf_dattool --export source.adtfdat --extension attached_files | adtf_dattool --modify destination.adtfdat --extension attached_files

to copy a explicit file extension of an existing ADTF DAT file to another.

Where to go next?

Enough about tooling and basics, it's time to have a look at API and start with CMake.