Configuration

All DataForge services (df-server, df-collector, df-preprocessor, df-renderer, df-deliverer, df-ai-trainer, df-ai-tester, df-ai-server) can be configured using command-line parameters, a main and override configuration file, or environment variables.

For service deployment in Docker, it is recommended to use environment variables for configuration.

Order of precedence

Configuration options are applied in a defined order. Every subsequent method can therefore override previously applied configurations.

1. Main configuration file (for example, df-server.yaml)
2. Override configuration file (for example, df-server-user.yaml)
3. Environment variables (for example, DF_SERVER_CRYPTO_MASTER_KEY)
4. Command-line options (for example, --crypto.master-key)

Configuration file names

Configuration files are recognized by name on a per-service basis. A main configuration file is named as the respective service with the .yaml file extension.
The override configuration file has the suffix -user appended to the service name.
For the DataForge Server these are respectively:

• df-server.yaml - for the main configuration file
• df-server-user.yaml - for the override configuration file

Configuration key names

All configuration keys exist in all formats, only the names are transformed in specific ways, depending on the configuration method. For example, in configuration files the encryption key is set as crypto.master_key: "example". A service prefix is prepended when using environment variables, and the key is written as DF_SERVER_CRYPTO_MASTER_KEY=example. On the command-line, the same option is set as --crypto.master-key=example. The cheat sheet below explains the differences:

• Configuration files: crypto.master_key: "example"
  • Not case-sensitive
  • Words are divided by underscores
  • crypto.Master_Key: "example" - This would be just as valid. Only applies to configuration files.
• Environment variables: DF_SERVER_CRYPTO_MASTER_KEY=example
  • Case-sensitive
  • Words are divided by underscores
  • Requires prefix of the service name
• Command-line flags: --crypto.master-key=example
  • Case-sensitive
  • Words are divided by hyphens
  • Prepended with double hyphen --

Listing configuration keys

All DataForge binaries can be executed with the following flags to list the appropriate set of configuration keys:

• --list-environment-keys
• --list-config-keys
• --list-cli-keys

The --help option shows usage and default values for each option. Additionally, the --create-empty-config option prints an empty sample configuration to standard output.

For reference configurations, see the included Docker Compose file.

The following sections provide an overview of all configuration options and their usage. For configuration options of third-party services, refer to their respective documentation.