3. Configuration model¶
Table of Contents
This section describes configuration files, how they are structured, and how they are represented in C
structures. The bitd-agent
uses a common object model to describe all types of configuration. It is easier to understand the object model starting with its C
language representation, then understand how it is translated into and from yaml
and xml
.
The bitd-agent
configuration file is passed in using the -c
argument. The configuration file can be formatted as either yaml
or xml
. The bitd-agent
uses utilities in the libbitd
library to parse the configuration into C
data types, then runs the configured task instances, and converts back the output of task instances from C
data type format to yaml
or xml
.
Understanding the conversion mechanism between yaml
, xml
and C
data types, thus, facilitates working with the bitd-agent
.
3.1. The C object model¶
Look at the include/bitd/platform-types.h
header for the data types defined below. The primary C
data types are:
3.1.1. bitd_void¶
bitd_void
represents the empty type, and is defined as the C
type void
.
3.1.2. bitd_boolean¶
bitd_boolean
is defined as signed char
, and holds FALSE
as 0
and TRUE
as any non-zero value.
3.1.3. bitd_int64, bitd_uint64¶
bitd_int64
, bitd_uint64
are unsigned and signed 64 bit data types, defined as long long
, resp. unsigned long long
on all platforms where long long
is 64 bits.
The header file defines, in similar vein, data types for smaller width integers: bitd_int8
, bitd_uint8
, bitd_int16
, bitd_uint16
, bitd_int32
, bitd_uint32
. While these types are used frequently in the source code, the Bitdribble object model represents all integers parsed from yaml
or xml
as either bitd_int64
or bitd_uint64
, for simplicity.
3.1.4. bitd_double¶
bitd_double
is mapped to the C
language double
float type. No bitd_float
type is defined - we use bitd_double
instead.
The composite data types for C
objects are:
3.1.5. bitd_string¶
bitd_string
is used for NULL-terminated char *
strings.
3.1.6. bitd_blob¶
bitd_blob
holds arbitray buffers that may contain the character 0
. The bitd_blob
contains a 4 byte length field followed by the actual blob payload.
/* The blob type */
typedef struct {
bitd_uint32 nbytes;
} bitd_blob;
#define bitd_blob_size(b) ((b)->nbytes)
#define bitd_blob_payload(b) ((char *)(((bitd_blob *)b)+1))
3.1.7. Nvp arrays¶
bitd_nvp_t
holds an array of name-value pairs, each element of which has its own type. In short, this type is called an nvp
array. Elements in an nvp
array can have any simple or composite type, including the nvp
array type itself.
/* Forward declaration */
struct bitd_nvp_s;
/* Enumeration of types */
typedef enum {
bitd_type_void,
bitd_type_boolean,
bitd_type_int64,
bitd_type_uint64,
bitd_type_double,
bitd_type_string,
bitd_type_blob,
bitd_type_nvp,
bitd_type_max
} bitd_type_t;
/* Untyped value */
typedef union {
bitd_boolean value_boolean;
bitd_int64 value_int64;
bitd_uint64 value_uint64;
bitd_double value_double;
bitd_string value_string;
bitd_blob *value_blob;
struct bitd_nvp_s *value_nvp;
} bitd_value_t;
/* A name-value-pair element - or 'nvp element' */
typedef struct {
char *name;
bitd_value_t v;
bitd_type_t type;
} bitd_nvp_element_t;
/* A name-value-pair array - or 'nvp' */
typedef struct bitd_nvp_s {
int n_elts;
int n_elts_allocated;
bitd_nvp_element_t e[1]; /* Array of named objects */
} *bitd_nvp_t;
3.1.8. Objects¶
The bitd_object_t
type holds any arbitrary typed value:
/* An object is a typed value */
typedef struct {
bitd_value_t v;
bitd_type_t type;
} bitd_object_t;
Any object, thus, can be represented as bitd_object_t
. This means objects can be bitd_boolean
, or bitd_int64
, or of nvp
type. And, since nvp`
is an array type, the objects of type nvp
can be thought of as arrays of other objects.
3.2. The Json object model¶
For the definition of json
, see https://www.json.org. Simple bitdribble types are represented in json
as follows:
3.2.1. bitd_void¶
bitd_void
is formatted as the null
json value, and the null
json value is represented as a bitd_void
type.
3.2.2. bitd_boolean¶
bitd_boolean
TRUE
is represented in json
as true
, and bitd_boolean
FALSE
as false
. Conversely, json true
, false
are respectively represented as bitd_boolean
values TRUE
and FALSE
.
3.2.3. bitd_int64 and bitd_uint64¶
bitd_int64
and bitd_uint64
are represented in json
as numeric integers, except when the bitd_uint64
value is larger than LONG_MAX
, in which case it is represented as a string. When bitd_uint64
is represented as string, the key name is appended the suffix _!!uint64
.
Conversely, numeric json
integers are represented as bitd_int64
, or if the key name has a _!!uint64
suffix, as a bitd_uint64
.
3.2.4. bitd_double¶
bitd_double
is represented in json
as a numeric string formatted as a floating point number, in decimal format. Numeric strings in json
that have a floating point, or are outside of the int64
and uint64
range are represented in C
as bitd_double
.
Composite bitdribble types are represented in json
as follows:
3.2.5. bitd_string¶
bitd_string
is represented in json
as a string. Json strings that are non-void, non-numeric, and do not have a key suffix of _!!uint64
or _!!blob
format are represented as bitd_string
.
3.2.6. bitd_blob¶
bitd_blob
types are represented in json
as base64
encoded strings, with a key name that gets appended a _!!blob
suffix. Conversely, json values of string type with a key name suffix of _!!blob
are base64
decoded and represented as bitd_blob
types.
3.2.7. Nvp arrays¶
bitd_nvp_t
types are represented as json object
, if at least one of the element names are non-NULL and a non-zero-length string - and as a json array
otherwise. A json object
is represented as nvp
array, and a json
array is represented as nvp array with NULL-named elements.
3.2.8. Objects¶
The bitd_object_t
type is represented in json
simply by representing the underlying type and value of the object in json
. Conversely, a json
object is represented by a bitd_object_t
type.
This sets a correspondence between bitd_object_t
objects and json
objects that is onto, in a mathematical sense: any json
object corresponds to one or more bitd_object_t
objects.
3.2.9. Using Json attributes¶
The bitd_object_t
type can be implied from the json
value type, but can also be set explicitly in the json
object by appending _!!<type>
to the key name corresponding to the json
value. Here the <type>
is any of void
, boolean
, int64
, uint64
, double
, string
, blob
or nvp
.
3.2.10. The source code¶
The implementation of the json
object model is in src/libs/bitd/types-json.c.
3.3. The Xml object model¶
For an introduction to xml
, see https://en.wikipedia.org/wiki/XML. For a quick introduction, xml
documents emply angle brackets to delineate element names and element content. Elements can have zero or more attributes:
<?xml version='1.0'?>
<root-element-name>
<element-name1/>
<element-name2>127</element-name2>
<element-name3 attribute1='value1' attribute2='value2'>abc</element-name3>
<element-name4>
<embedded-element-name5 attribute1='value1'>def</embedded-element-name5>
</element-name4>
</root-element-name>
The order of attributes is not important in an element, but the order of subelements matters - in the sense that changing the attribute order does not change the xml
document, but changing the element order does change the xml
document.
We will describe a partial correspondence between xml
documents and named bitdribble objects. The root-element-name
corresponds to the name of the object
. Each element-name
corresponds to the name
of a value in an nvp
name-value pair array. If no attribute is specified, the type of the content is inferred:
- If the element is empty, the type is
bitd_void
. - If the element is the string
TRUE
,True
,true
,FALSE
,False
orfalse
, the type isbitd_boolean
. - If the element is numeric string, the type is
bitd_int64
if an integer betweenLLONG_MIN
andLLONG_MAX
, otherwisebitd_uint64
if an integer betweenLLONG_MAX+1
andULLONG_MAX
, and otherwise abitd_double
. - If the element is any other string, the type is
bitd_string
. - If the element has sub-elements, the type is
bitd_nvp_t
.
Using specific xml
attributes changes the type of the element:
- If the element has an attribute named
type
with valuevoid
, respectivelyboolean
,int64
,uint64
,double
,string
, the type isbitd_void
, respectivelybitd_boolean
,bitd_int64
,bitd_uint64
,bitd_double
,bitd_string
. - If the element has the attribute
type='blob'
, the value is interpreted to be a base64 encodedbitd_blob
. - If the element has the attribute
type='nvp'
, the value is interpreted to be of typebitd_nvp_t
.
The converse correspondence is described below:
3.3.1. bitd_void¶
bitd_void
types are represented as empty xml
elements. Optionally, these elements can be assigned a type='void'
attribute.
<element-name/>
<!-- or -->
<element-name type='void'/>
3.3.2. bitd_boolean¶
bitd_boolean
types are represented as xml
elements having the TRUE
or FALSE
boolean value as contents. Optionally, these elements can be assigned a type='boolean'
attribute.
<element-name>FALSE</element-name>
<!-- or -->
<element-name type='boolean'>FALSE</element-name>
3.3.3. bitd_int64¶
bitd_int64
types are represented as xml
elements having as contents the integer value. Optionally, these elements can be assigned a type='int64'
attribute. If the integer is between LLONG_MIN
and LLONG_MAX
, the attribute can be omitted.
<element-name>123</element-name>
<!-- or -->
<element-name type='int64'>123</element-name>
3.3.4. bitd_uint64¶
bitd_uint64
types are also represented as xml
elements having as contents the integer value. Optionally, these elements can be assigned a type='uint64'
attribute. If the integer is between LLONG_MAX+1
and ULLONG_MAX
, the attribute can be omitted.
<element-name>18446744073709551615</element-name>
<!-- or -->
<element-name type='uint64'>123</element-name>
3.3.5. bitd_double¶
bitd_double
types are represented as xml
elements having as contents the numeric value. Optionally, these elements can be assigned a type='double'
attribute. If the number has a decimal point or is not a bitd_int64
or bitd_uint64
, the attribute can be omitted.
<element-name>123.1</element-name>
<!-- or -->
<element-name>123.0</element-name>
<!-- or -->
<element-name type='double'>123</element-name>
<!-- but not -->
<element-name>123</element-name><!-- ...which would be interpreted as int64 -->
3.3.6. bitd_string¶
bitd_string
types are represented as xml
elements having as contents the string value. Optionally, these elements can be assigned a type='string'
attribute. If the value cannot be interpreted as a bitd_void
, bitd_boolean
, bitd_int64
, bitd_uint64
, bitd_double
, then the attribute can be omitted.
<element-name>abc</element-name>
<!-- or -->
<element-name type='string'>abc</element-name>
<!-- but not -->
<element-name></element-name><!-- ...which would be interpreted as void -->
<!-- and not -->
<element-name>TRUE</element-name><!-- ...which would be interpreted as boolean -->
<!-- and not -->
<element-name>123</element-name><!-- ...which would be interpreted as int64 -->
<!-- and not -->
<element-name>123.0</element-name><!-- ...which would be interpreted as double -->
3.3.7. bitd_blob¶
bitd_blob
types are represented as xml
elements having as contents the base64 encoded blob. These elements must be assigned a type='blob'
attribute, to be distinguished from other strings. The attribute can never be omitted.
<element-name type='blob'>MDEyMzQ1Njc4OQo=</element-name>
To find out to which blob contents this corresponds, you can uudecode it as follows:
$ echo MDEyMzQ1Njc4OQo= | base64 -d
0123456789
3.3.8. bitd_nvp¶
bitd_nvp
types are name-value pair arrays and are represented as xml
elements with subelements. Here is, for example, an nvp
with elements of all possible types:
<?xml version='1.0'?>
<nvp type='nvp'>
<name-void type='void'/>
<name-boolean type='boolean'>FALSE</name-boolean>
<name-int8 type='int64'>-128</name-int8>
<name-int8 type='int64'>127</name-int8>
<name-uint8 type='int64'>255</name-uint8>
<name-int16 type='int64'>-32768</name-int16>
<name-int16 type='int64'>32767</name-int16>
<name-uint16 type='int64'>65535</name-uint16>
<name-int32 type='int64'>-2147483648</name-int32>
<name-int32 type='int64'>2147483647</name-int32>
<name-uint32 type='int64'>4294967295</name-uint32>
<name-int64 type='int64'>-9223372036854775808</name-int64>
<name-int64 type='int64'>9223372036854775807</name-int64>
<name-uint64 type='uint64'>18446744073709551615</name-uint64>
<name-double type='double'>100000.0</name-double>
<name-string type='string'/>
<name-string type='string'>True</name-string>
<name-string type='string'>123</name-string>
<name-string type='string'>123.0</name-string>
<name-string type='string'>string-value</name-string>
<name-blob type='blob'>MDEyMzQ1Njc4OQo=</name-blob>
<empty-nvp-value type='nvp'/>
<full-nvp-value type='nvp'>
<name-void type='void'/>
<name-boolean type='boolean'>FALSE</name-boolean>
<name-int8 type='int64'>-127</name-int8>
<name-uint8 type='int64'>255</name-uint8>
<name-int16 type='int64'>-32767</name-int16>
<name-uint16 type='int64'>65535</name-uint16>
<name-int32 type='int64'>-2147483647</name-int32>
<name-uint32 type='int64'>4294967295</name-uint32>
<name-int64 type='int64'>-9223372036854775807</name-int64>
<name-uint64 type='uint64'>18446744073709551615</name-uint64>
<name-double type='double'>1.99</name-double>
<name-string type='string'/>
<name-string type='string'>True</name-string>
<name-string type='string'>123</name-string>
<name-string type='string'>123.0</name-string>
<name-string type='string'>string-value</name-string>
<name-blob type='blob'>MDEyMzQ1Njc4OQo=</name-blob>
<empty-nvp-value type='nvp'/>
</full-nvp-value>
</nvp>
If the nvp is named, the name will be stored as the xml
root element name. If the nvp is unnamed, or has an empty name, by convention the root element name is set to nvp
- as was the case in the example above. Here is the same xml
document leaving out all type
attributes that are optional (meaning that the type of the contents can be inferred from the value of the contents):
<?xml version='1.0'?>
<nvp>
<name-void/>
<name-boolean>FALSE</name-boolean>
<name-int8>-128</name-int8>
<name-int8>127</name-int8>
<name-uint8>255</name-uint8>
<name-int16>-32768</name-int16>
<name-int16>32767</name-int16>
<name-uint16>65535</name-uint16>
<name-int32>-2147483648</name-int32>
<name-int32>2147483647</name-int32>
<name-uint32>4294967295</name-uint32>
<name-int64>-9223372036854775808</name-int64>
<name-int64>9223372036854775807</name-int64>
<name-uint64>18446744073709551615</name-uint64>
<name-double>100000.0</name-double>
<name-string type='string'/>
<name-string type='string'>True</name-string>
<name-string type='string'>123</name-string>
<name-string type='string'>123.0</name-string>
<name-string>string-value</name-string>
<name-blob type='blob'>MDEyMzQ1Njc4OQo=</name-blob>
<empty-nvp-value type='nvp'/>
<full-nvp-value>
<name-void/>
<name-boolean>FALSE</name-boolean>
<name-int8>-127</name-int8>
<name-uint8>255</name-uint8>
<name-int16>-32767</name-int16>
<name-uint16>65535</name-uint16>
<name-int32>-2147483647</name-int32>
<name-uint32>4294967295</name-uint32>
<name-int64>-9223372036854775807</name-int64>
<name-uint64>18446744073709551615</name-uint64>
<name-double>1.99</name-double>
<name-string type='string'/>
<name-string type='string'>True</name-string>
<name-string type='string'>123</name-string>
<name-string type='string'>123.0</name-string>
<name-string>string-value</name-string>
<name-blob type='blob'>MDEyMzQ1Njc4OQo=</name-blob>
<empty-nvp-value type='nvp'/>
</full-nvp-value>
</nvp>
3.4. The Yaml object model¶
For a quick introduction to yaml
, see https://en.wikipedia.org/wiki/YAML. Simple bitdribble types are represented in yaml
as follows:
3.4.1. bitd_void¶
bitd_void
is represented as the empty yaml
string. An empty yaml
string is represented as a bitd_void
type.
3.4.2. bitd_boolean¶
bitd_boolean
is represented as the yaml
string TRUE
or FALSE
. The yaml
strings TRUE
and FALSE
are represented as bitd_boolean
.
3.4.3. bitd_int64 and bitd_uint64¶
bitd_int64
and bitd_uint64
are represented in yaml
as numeric strings. Integer in yaml
are represented as bitd_int64
, if between LLONG_MIN
and LLONG_MAX
, and bitd_uint64
if between LLONG_MAX+1
and ULLONG_MAX
.
3.4.4. bitd_double¶
bitd_double
is represented in yaml
as a numeric string formatted as a floating point number, in decimal format. Numeric strings in yaml
that are not integers, or are outside of the int64
and uint64
range are represented in C
as bitd_double
.
Composite bitdribble types are represented in yaml
as follows:
3.4.5. bitd_string¶
bitd_string
is represented in yaml
as a string. Yaml strings that are non-void, non-numeric, and not TRUE
, True
, true
, FALSE
, False
or false
are represented in C
as bitd_string
types.
3.4.6. bitd_blob¶
bitd_blob
types are represented in yaml
as base64
encoded !!binary
types. Conversely, !!binary
yaml types are base64
decoded and represented in C
as bitd_blob
types.
3.4.7. Nvp arrays¶
bitd_nvp_t
types are represented in yaml
as non-scalar name-value pairs. Nvp
arrays with all elements having NULL names are represented as yaml
sequences. Conversely, yaml
composite types are represented as nvp
arrays, and yaml
sequences are represented as nvp arrays with NULL-named elements.
3.4.8. Objects¶
The bitd_object_t
type is represented in yaml
simply by representing the underlying type and value of the object in yaml
. Conversely, a yaml
document is represented by a bitd_object_t
type.
This sets a correspondence between objects and yaml
documents that is onto, in a mathematical sense: any yaml
document corresponds to one or more objects. To see why this correspondence is not also one to one, observe that objects containing a string that is an integer corresponds to a yaml
document containing that number’s value, which in turn corresponds to an object of integer type.
Yaml
files can also contain a stream of documents. For example, the task instance results output of the bitd-agent
is a yaml
stream, with each task instance result being its own document. A yaml
stream corresponds to an ordered set of C
objects.
3.4.9. Using Yaml attributes¶
As seen above, yaml
strings are parsed into bitd_void
if empty, or into bitd_boolean
if equal to TRUE
, True
, true
, FALSE
, False
or false
, or into bitd_int64
if integers within the LLONG_MIN
and LLONG_MAX
, or otherwise into bitd_uint64
if between LLONG_MAX+1
and ULLONG_MAX
, or otherwise into bitd_double
if numeric - or, if none of the above, they are parsed as bitd_string
.
This represents the default conversion of yaml
string scalars. The conversion can also be controlled by use of the following yaml
attributes:
tag:yaml.org,2002:null
is converted tobitd_void
type.tag:yaml.org,2002:bool
is converted tobitd_boolean
type.tag:yaml.org,2002:int
is converted tobitd_int64
. The value is truncated if too large.tag:yaml.org,2002:str
is converted tobitd_string
.tag:yaml.org,2002:binary
is converted tobitd_blob
.
3.4.10. The source code¶
The implementation of the yaml
object model is in src/libs/bitd/types-yaml.c.