2. The bitd-agent¶
Table of Contents
2.1. An example run of the bitd-agent¶
The main Bitribble application is the bitd-agent
. The bitd-agent yaml
or xml
configuration file lists the task instances that the bitd-agent will run. For example:
bitd-agent -c echo-config.yml
The configuration file describes the task instance parameters, the modules that contain the task implementation, as well as the task instance schedule. Here is the echo-config.yml:
1 2 3 4 5 6 7 8 9 10 11 | modules:
module-name: bitd-echo
task-inst:
task-name: echo
task-inst-name: Echo task inst 1
schedule:
type: periodic
interval: 1s
args:
name-int64: -128
name-int64: 127
|
By default, bitd-agent
will print the task instace results to stdout
in yaml format:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | $ bitd-agent -c modules/echo-config.yml
---
tags:
task: echo
task-instance: Echo task inst 1
run-id: 0
run-timestamp: 1539370698248977666
exit-code: 0
output:
name-int64: -128
name-int64: 127
---
tags:
task: echo
task-instance: Echo task inst 1
run-id: 1
run-timestamp: 1539370699250138126
exit-code: 0
output:
name-int64: -128
name-int64: 127
---
^C...
|
The task instance will run every second until the bitd-agent
is interrupted with Ctrl-C
.
The echo
task just echoes back the input as output. Other task modules are available:
- The
exec
task will execute any shell command - The
sink-graphite
task in thebitd-sink-graphite
module will send results directed to it to a Graphite back-end database - The
sink-influxdb
task in thebitd-sink-influxdb
module will send results directed to it to an Influxdb back-end database
2.1.1. Annotated configuration and output¶
This section can be skipped on first reading. Here is an annotate version of the configuration file:
1 2 3 4 5 6 7 8 9 10 11 | modules: # The list of modules
module-name: bitd-echo # The module implementing the echo task
task-inst: # The task instance block
task-name: echo # The echo task
task-inst-name: Echo task inst 1 # Multiple task instances can be defined for the same task
schedule:
type: periodic # Running periodically
interval: 1s # ... every second
args: # The equivalent of Linux process command line arguments
name-int64: -128
name-int64: 127
|
And here is an annotated version of the output:
1 2 3 4 5 6 7 8 9 10 11 12 13 | $ bitd-agent -c modules/echo-config.yml
---
tags: # The task and task instance name are reported back as tags
task: echo
task-instance: Echo task inst 1
run-id: 0 # The run-id is incremented with each task instance run
run-timestamp: 1539370698248977666 # The Unix time, in nanosecs, when results were reported
exit-code: 0 # The exit code of the task instance
output: # The task instance output
name-int64: -128
name-int64: 127
---
^C...
|
2.2. The bitd-agent command line parameters¶
The full list of bitd-agent command line parameters are displayed by running
bitd-agent -h
2.2.1. Configuration file¶
The config file is passed in using a -c
parameter. The type of config file can be yaml or xml. The filename suffix is used to detect the syntax used in the file: .yml or .yaml files will be parsed as yaml files, and .xml files will be parsed as xml files.
-c config_file.{xml|yml|yaml}
The bitd-agent configuration file. Format is determined from
file suffix. Default format is yaml.
If you don’t wish to rely on the filename suffix to determine the syntax of the file, pass the configuration file used -cx
for xml, and -cy
for yaml:
-cx config_file.xml
The bitd-agent configuration file, in xml format.
-cy config_file.yaml
The bitd-agent configuration file, in yaml format.
2.2.2. Results display¶
The task instance results can be displayed in yaml or xml format. These parameters control the display format, and where the results are saved:
-r result_file
The bitd-agent result file. Default: stdout.
-rx
Format the result output as xml.
-ry
Format the result output as yaml (default).
To discard the results, pass -r /dev/null
. If the result file is stdout
, results will be sent to stdout
. On Unix type systems, including Cygwin, the same effect can be achieved passing /dev/stdout
as a result file - but Windows does not have a notion of dev/stdout
. On Windows, it is therefore handy to be about to specify stdout
directly as a parameter to -r
.
2.2.3. Module load path¶
The modules containing task implementations are DLLs (i.e., Linux shared libraries and Windows dynamically loaded libaries). By default, the bitd-agent
will look for modules in the path specified by the
LD_LIBRARY_PATH
environment variable on LinuxPATH
environment variable in Windows Win32 and CygwinDYLD_FALLBACK_LIBRARY_PATH
environment variable on OSX
Additional path lookup folders can be specified with the following parameter:
-lp load_path
DLL load library path.
2.2.4. Worker thread pool size¶
Task instances are executed in the context of threads in a worker thread pool. If insufficient threads are available, task instances are delayed until a thread becomes available. The default number of threads in the pool rarely needs to be adjusted, but it can be with the following parameter:
--n-worker-threads thread_count
Set the max number of worker theads.
2.2.5. Log level¶
The log level can be controlled using the following options:
-l|--log-level none|crit|error|warn|info|debug|trace
Set the log level (default: none).
-lk|--log-key-level key_name none|crit|error|warn|info|debug|trace
Set log key level.
-lf|--log-file file_name
Save log messages to a file. Default: stdout.
-ls|--log-file-size size
Set the log file size. Default: 16777216.
-lc|--log-file-count count
Set the log file count. Default: 3.
The -l level
option controls the global log level. Sometimes, when many task instances are running, increasing the global log level results in too many messages. In that case, it is better to keep the global log level set to none
and instead enable the log for specific keys, using -lk key_name level
. Each task instance will send its log messages to a specific key. The task instance log key name will be specific to the task instance.
Log levels can also be controlled using the config-log
task located in the bitd-config-log
module. Here is an example configuration enabling logs only for the module-mgr
key (in effect, enabling log messages only from the module-mgr subsystem):
1 2 3 4 5 6 7 8 9 10 11 12 | modules:
module-name: bitd-config-log
task-inst:
task-name: config-log
task-inst-name: config-log
schedule:
type: config
input:
log-level: none
log-key:
key-name: module-mgr
log-level: info
|
This configuration file can be merged into any other configuration file - for example, here it is merged into the echo task instance configuration file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | modules:
module-name: bitd-echo
module-name: bitd-config-log
task-inst:
task-name: echo
task-inst-name: Echo task inst 1
schedule:
type: periodic
interval: 1s
args:
name-int64: -128
name-int64: 127
task-inst:
task-name: config-log
task-inst-name: config-log
schedule:
type: config
input:
log-level: none
log-key:
key-name: module-mgr
log-level: info
|
2.3. The bitd-agent configuration file¶
The bitd-agent
configuration can be specified either in yaml
or xml
format. In this section, we will describe the yaml
version of the configuration file. All the examples in this section can be converted to xml
using the test-object
unit tester tool. (Run test-object -h
for options.)
It is easiest to work through the structure of the configuration file through examples. Here is the basic example we will start with:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | modules:
module-name: bitd-echo
module-name: bitd-sink-graphite
module-name: bitd-config-log
task-inst:
task-name: config-log
task-inst-name: config-log
schedule:
type: config
input:
log-level: warn
log-key:
key-name: bit-sink-graphite
log-level: debug
task-inst:
task-name: echo
task-inst-name: Echo task inst
schedule:
type: random
interval: 1s
args:
name-int64: -128
name-int64-2: 127
task-inst:
task-name: sink-graphite
task-inst-name: Graphite sink inst
schedule:
type: triggered-raw
task-name: echo
args:
server: localhost:2013
queue-size: 1000
|
The config file has two types of top-level elemets: modules
and task-inst
. The task-inst
element may be repeated multiple times, and each task-inst
contains configuration for a single task instance. The modules
element appears only once, and contains a single module-name
element per module.
Here is the modules
section highlighted:
1 2 3 4 5 6 7 8 | modules:
module-name: bitd-echo
module-name: bitd-sink-graphite
module-name: bitd-config-log
task-inst:
task-name: config-log
task-inst-name: config-log
#...
|
The module-names
get translated into DLL names, and the DLLs are loaded into the running bitd-agent
. Different platforms have different naming conventions for the module DLLs. The DLL name corresponding to module-name abc
is:
libabc.so
on Unixabc.dll
on Win32 Windowscygabc.dll
on Cygwin Windowslibabc.dylib
on OSX
These DLLs must be found in the platform-specific PATH
, or must be located in a folder specified using the -lp
command line parameter to bitd-agent
(see Module load path). Each DLL will implement one or more tasks in the configuration.
Each task-inst
block has one task-name
element (a task implemented in one of the modules), and a task-inst-name
that should be unique per task. Here are the task names and task instance names highlighted:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | modules:
module-name: bitd-echo
module-name: bitd-sink-graphite
module-name: bitd-config-log
task-inst:
task-name: config-log
task-inst-name: config-log
schedule:
type: config
input:
log-level: warn
log-key:
key-name: bitd-sink-graphite
log-level: debug
task-inst:
task-name: echo
task-inst-name: Echo task inst
schedule:
type: random
interval: 1s
args:
name-int64: -128
name-int64-2: 127
task-inst:
task-name: sink-graphite
task-inst-name: Graphite sink inst
schedule:
type: triggered-raw
task-name: echo
args:
server: localhost:2013
queue-size: 1000
|
The task instances each have a schedule element. The schedule determines when the task instance will run.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | modules:
module-name: bitd-echo
module-name: bitd-sink-graphite
module-name: bitd-config-log
task-inst:
task-name: config-log
task-inst-name: config-log
schedule:
type: config
input:
log-level: warn
log-key:
key-name: bitd-sink-graphite
log-level: debug
task-inst:
task-name: echo
task-inst-name: Echo task inst
schedule:
type: random
interval: 1s
args:
name-int64: -128
name-int64-2: 127
task-inst:
task-name: sink-graphite
task-inst-name: Graphite sink inst
schedule:
type: triggered-raw
task-name: echo
args:
server: localhost:2013
queue-size: 1000
|
2.3.1. Task instance schedule types¶
The task instance schedule can be of one of these types:
periodic:
most tasks are periodic. You can specify the task instance run interval at which the task is running. The period is specified by theinterval
element, which is of the form[number]us|ms|s|m|h|d
, whereus
is microseconds,ms
is milliseconds,s
is seconds,m
is minutes,h
is hours,d
is days.random:
this is similar toperiodic
, but the start time is random within the period. Theinterval
element specifies the period duration from which the random start is picked.once:
these tasks run only onceconfig:
these tasks run once when thebitd-agent
is initialized, and also each time the config is reloaded. On Unix and OSX, the config is reloaded when thebitd-agent
receives aSIGHUP
signal. On Windows it’s not possible to reload the config.triggered
andtriggered-raw:
these tasks are triggered by other tasks in the config file, and take as input the output of these other tasks. When a task instance istriggered
, it gets only the output of the task instance run that triggered it. When a task instance has atriggered-raw
schedule, it gets thetags
,run-id
,run-timestamp
,exit-code
,output
anderror
of the task instance run that triggered it.
2.3.2. Modules and tasks¶
For a listing of the available modules, and for the tasks supported in each module, see Modules.