Description
The ADTF DAT Tool is the perfect assist for
- gathering information from an
.adtfdat file
- cutting existing recordings
- create new adtfdat files from existing ones
- load and inspect adtffileplugins to extend functionality (import/export streams)
- import data from recordings different from
.adtfdat files
- export streams from adtfdat to any available file format
as comfortable and extendable command line tooling.
Getting started
For any purpose, please execute the adtdfat_tool with the option --help, which will give you an overview and example usage of available options and commands.
usage:
adtf_dattool.exe options
where options are:
-?, -h, --help display usage information
--progress Show progress.
--skipstreamtypesandtriggers Do not process stream types and
triggers.
--plugin <adtffileplugin> Load an additional plugin.
--inspect <adtffileplugin> Inspect a plugin and print its
contained items in a JSON string.
--inspect-builtins Inspect all built-in classes.
--json Output information in json format.
--liststreams <file name> List all available information about
the given file.
--listsubstreams <stream name> List all substreams available in the
given stream.
--export <file name> Export streams from the given file.
--create <file name> Create a new dat file.
--modify <file name> Modify an existing dat file.
--fileversion <adtf2|adtf3|adtf3ns> File Version of the created file.
Creating ADTF 2 files is completely
experimental!
--input <source> Specifies an input for the new dat
file or extension. In case of
extensions data will be read from
stdin if this is not specified.
--readerid <reader id> The id of the reader implementation
that should be used to open the last
list/input/export source.
--referencedfiles Process referenced files (i.e.
splits) from the last input/export
source as well.
--ignoreopenerrors Ignore errors when referenced files
cannot be opened.
--start <timestamp> Process only chunks of the last
input source with timestamps larger
than this (microseconds).
--end <timestamp> Process only chunks of the last
input source with timestamps smaller
than this (microseconds).
--offset <timestamp> This offset is added to all chunk
timestamps of the last input source
(microseconds).
--start-ns <timestamp> Process only chunks of the last
input source with timestamps larger
than this (nanoseconds).
--end-ns <timestamp> Process only chunks of the last
input source with timestamps smaller
than this (nanoseconds).
--offset-ns <timestamp> This offset is added to all chunk
timestamps of the last input source
(nanoseconds).
--stream <stream name> Select a stream for export,
modification or when creating a new
dat file.
--substream <substream name> Select a substream for export. Keep
in mind that if you want to export
multiple substreams from the same
stream, you need to specify --stream
for each substream. If the substream
is not available within the initial
stream type you always need to
specify a processor id with --
processorid.
--name <stream name> Sets the name for the last specified
stream that is used in the newly
created dat file.
--processorid <processor id> The id of the processor
implementation used for the last
specified stream or substream.
--property <name=value> Sets a property of the last
specified export job, stream or
input.
--extension <extension name> Select an extension for export, when
creating a new dat file or updating
one.
--output <file name> Sets the output file name for the
last specified stream or extension.
If not specified, the processor is
free to choose one and in case of
extensions data will be written to
stdout
--serializerid <serializer id> The id of the serializer
implementation used for the last
specified stream.
--userid <user id> Sets the user id of the last
extension to be updated.
--typeid <type id> Sets the type id of the last
extension to be updated.
--versionid <version id> Sets the version id of the last
extension to be updated.
--rename <new name> Renames the given stream.
--repair <file name> Repair the given corrupted file.
Requires --reference and --output as
well.
--reference <file name> The reference file for the repair
operation.
-----------------
ADTF DAT Tool
-----------------
With the help of this tool you can extract data from ADTF DAT files, create new ADTF DAT files
from various inputs or add file extensions in combination with the adtf_datdump tool
to an existing ADTF DAT file.
Use --liststreams to query all information about a given input (ADTF DAT file, or any other
supported input).
To load additional adtffileplugins use the --plugin argument as often as you like.
-----------
EXPORT:
-----------
To extract data from streams or file extensions of an ADTF DAT file use the --export argument.
Select the streams you want to extract by using the --stream argument. You can specify a processor
for each stream with the --processorid argument. If you do not specify one explicitly, the first one
that supports the stream is used. Properties of processors can be specified with the --property
argument.
Use the --extension argument to specify the required file extension. The destination
filename is specified with the --output argument. Without --output argument the file extension data
will be written to stdout.
To process a set of files in one go specify '--readerid multiple_files' right after your
--export or --input argument and specify the set of files as a ';' seperated list.
To process referenced files transparently specify --referencedfiles right after your
source dat file.
Examples:
---------
Here is an example that exports two streams:
adtf_dattool --plugin csv_exporter.adtffileplugin --export test.dat --stream in1
--processorid csv --output test_in1.csv --stream in2 --output test_in2.csv
And one that exports extension data:
adtf_dattool --export test.dat --extension adtf_version --output adtf_version.txt
Process referenced files as well:
adtf_dattool --plugin csv_exporter.adtffileplugin --export first_split_001.dat
--referencedfiles --stream in1 --processorid csv --output in1.csv
-------------
CREATION:
-------------
To create a new ADTF DAT file use the --create argument. The --input argument is used to specify an
input file. The --readerid argument can be used to specify the reader that should be used to read
the file. If none is specified, the first one that supports the file is used.
Use the --start and --end arguments to select the range of the input that should be
imported into the ADTF DAT file. Use the --offset parameter to shift the timstamps of all imported
stream items.
To select streams from an input, use the --stream argument. If you do not select one or more
streams neither a extension, all streams will be added. Streams can be renamed with the --name argument.
Use the --serializerid argument to choose the serializer of your liking. If not specified,
sample_copy_serialization_ns.serialization.adtf.cid will be used.
To select file extensions from an input, use the --extension argument. If you do not select one or
more file extensions explicitly, no extension will be added or updated from the source file.
Examples:
---------
Here is an example that creates a new ADTF DAT file from two inputs:
adtf_dattool --create new.adtfdat --input input1.dat --readerid adtfdat --stream in1 --serializerid
special.serialization.adtf.cid --input input2.dat
And one that copies a stream and two extensions from a source file
adtf_dattool --create new.adtfdat --input input1.dat --stream in1 --extension attached_files
--extension attached_files_configuration
------------------------
MODIFYING:
------------------------
To modify an existing ADTF DAT File use the --modify argument.
Currently there is only support for adding and updating extensions.
Examples:
---------
An example to put a files content into an extension:
adtf_dattool --modify existing.adtfdat --extension my_extension --input input_file.txt
Mind that if you do not specify an --input for an extension its data will be read from stdin:
tar cz /myfolder | adtf_dattool --modify existing.adtfdat --extension attached_files
or use:
adtf_dattool --export source.adtfdat --extension attached_files | adtf_dattool --modify destination.adtfdat --extension attached_files
to copy a explicit file extension of an existing ADTF DAT file to another.
To rename a stream use:
adtf_dattool --modify existing.adtfdat --stream old_name --rename new_name
------------------------
INSPECTION:
------------------------
To inspect an adtffileplugin use the --inspect argument. This will list all classes and their configuration
properties contained within the given plugin file. To list all built-in classes use '--inspect-builtins'.
Usage
Basic
The ADTF DAT Tool can be extended using .adtffileplugins. These binaries can contain reader (implemented using adtf_file::Reader from adtf_file) for importing data as well as processor (implemented using adtfdat_processing::Processor from adtfdat_processing) for exporting among others.
Export Mode
In this mode the ADTF DAT Tool makes use of processors to operate on streams inside the .adtfdat file.
Currently the ADTF File Library delivers two such processors to export the samples of a stream into a .csv file (see csv_exporter) and another to process the data as a wrapper using python (see python_processor). There are serval further exporter available, such as .mdf, .parquet and .pcap/pcapng among others at for example digitalwerk.
To extract data from streams of an ADTF DAT file use the --export argument. Select the streams you want to extract by specifying the --stream argument. You can specify a processor for each stream with the --processorid argument.
Note: If you do not specify one explicitly, the first one that supports the stream is used.
Properties of processors can be specified with the --property argument.
The destination filename is specified with the --output argument. Here is an example that exports two streams:
adtf_dattool --plugin csv_processor.adtffileplugin --export test.dat --stream in1 --processorid csv --output test_in1.csv --stream in2 --output test_in2.csv
Apply the parameter --liststreams to query all information about a given input (.adtfdat file or any other supported input). To load additional plugins use the --plugin argument as often as you like.
Create Mode
In this mode you can select arbitrary streams (even from multiple .adtfdat files or any other supported input) to create a new .adtfdat file. To do so use the --create argument. The --input argument is used to specify an input file. The --readerid argument can be used to pass in the reader that should be used to read the file.
Note: If none is given, the first one that supports the file is used.
Set the --start and --end arguments to select the range of the input that should be imported into the new .adtfdat file. To shift the timstamps of all imported stream items apply the --offset parameter.
To select streams from an input, use the --stream argument. If you do not select one or more streams explicitly, all streams will be added. Streams can be renamed with the --name argument.
Use the --serializerid argument to choose the serializer of your liking.
Note: If not specified, sample_copy_serialization.serialization.adtf.cid will be used.
Here is an example that creates a new ADTF DAT file from two inputs:
adtf_dattool --create new.dat --input input1.dat --readerid adtfdat --stream in1 --serializerid special.serialization.adtf.cid --input input2.dat
Best Practice
This chapter should show you a workflow how to start from scratch to use the ADTF DAT Tool, extend the functionality using adtffileplugins and how to export containing streams from a recording. The separated steps will teach you about architecture and how to components and functionality are connected. For a better overview, we strip the paths from the call and arguments.
Get information about the tooling
First of all, the command line help should always be the first address for information, no matter as getting started or for a deeper dive later:
Get information from the adtfdat file
As next step you should have a look at the recorded file. The --liststreams option will dump the content, in this case, one video and one ddl described stream:
adtf_dattool --liststreams example_file.adtfdat
adtfdat:
VIDEO:
type: adtf/image
processors:
time range (ns): [405356000, 14805306000]
items: 874
NESTED_STRUCT:
type: adtf2/legacy
processors:
time range (ns): [0, 14805306000]
items: 595
Get information from the adtffileplugin
With the knowledge about the containing streams, it is time to find a suitable processor to export some data. The --inspect option can be used to look inside an .adtffileplugin, in this case the csv_exporter with the ADTF File Library besides the ADTF DAT Tool itself.
adtf_dattool --inspect csv_processor.adtffileplugin
{
"processors": [
{
"id": "csv",
"properties": [
{
"name": "decimal_places",
"type": "uint32",
"value": "4"
},
{
"name": "separator",
"type": "string",
"value": ";"
},
{
"name": "timestamp_base_ns",
"type": "uint64",
"value": "1"
},
{
"name": "timestamp_decimals",
"type": "uint32",
"value": "0"
}
]
}
],
"readers": [
],
"sample_serializers": [
],
"sample_deserializers": [
],
"type_serializers": [
],
"type_deserializers": [
]
}
Extend the functionality
With this information, the calls can be extend by using --plugin which provides the csv processor to export the stream NESTED_STRUCT compared to the --liststreams call without that .adtffileplugin.
adtf_dattool --liststreams example_file.adtfdat --plugin csv_processor.adtffileplugin
adtfdat:
VIDEO:
type: adtf/image
processors:
time range (ns): [405356000, 14805306000]
items: 874
NESTED_STRUCT:
type: adtf2/legacy
processors: csv
time range (ns): [0, 14805306000]
items: 595
Export the data
Now all required information for exporting are available:
- the tool call from the help
- a valid adtfdat file
- the name of streams
- an adtffileplugin containing logic to export data
- the suitable processorid for csv export
- information about optional properties
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output exported_data_as.csv
Use cases for using properties for this processor could be adapting the decimal places:
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output exported_data_as.csv --property decimal_places=1
or exporting timestamps in microseconds:
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output exported_data_as.csv --property timestamp_base_ns=1000
Handle substreams
Processing substreams is almost the same workflow, with tiny extensions. The --liststreams command will show you the content as already learned:
adtf_dattool --liststreams example_file_substreams.adtfdat
adtfdat:
substreams:
type: adtf/substreams
processors:
time range (ns): [66746000, 14805306000]
items: 1467
The important identifier in this case is the type adtf/substreams for the stream substreams. So you have to dive deeper using --listsubstreams:
adtf_dattool.exe --liststreams example_file_substreams.adtfdat --listsubstreams substreams
adtfdat:
substreams:
NESTED_STRUCT:
type: adtf2/legacy
processors:
VIDEO:
type: adtf/image
processors:
You will receive the same required information compared to the common sample streams use case. Now you can also extend the functionality using a processor and export csv in the same manner
Note: We skip the step with a relook and inspect because we already know the information from our previous steps
The only difference during the --export call is to add the (containing) --substream after the specified --stream:
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file_substreams.adtfdat --stream substreams --substream NESTED_STRUCT --processorid csv --output exported_data_as.csv
Side note
Now you have a good start and also common workflow for the export use case. Of course you can use this procedure with less adaption for create and modify option as well.
Note: Please keep also in mind that you can load several adtffileplugins at once, each with an own --plugin call to load readers and processors as a kind of file conversion or several readers and processors for exporting more than one stream with processors from different adtffileplugins (that's why it is important to address them by using --processorid).
For more help, please always refer to --help which provides you the information and basic examples.