Installation

A step by step guide for setting up a self-hosted DataForge instance.

This chapter will provide a reference docker compose file to use for deployment. All services of a Daforge deployment may run in Docker, except the Dataforge server, which is installed natively as a systemd service.

Deploying DataForge using docker-compose

1. Obtaining necessary credentials

Once you have purchased a DataForge license, we will send you various credentials for your DataForge instance:

Your license key (a file ending in .key) will authenticate your DataForge server to our license server and enable all features included in your subscription
The docker registry credentials are used to pull the Docker images from our Docker registry

You will need these credentials during the installation process. Please do not share the credentials with anybody.

2. Setting up the environment

Download the included DataForge deployment package and extract it to a directory on your server (We recommend the name “dataforge”).

Rename the included template.env file to .env, which contains the most important configuration parameters.

Edit this file with an editor of your choice (for example nano or vim) and replace YOUR_DATAFORGE_VERSION at the top of the file with the version of DataForge that you want to install.

The .env file also contains placeholders ([REPLACE_ME]) that should be replaced by random secrets. For your convenience, we provide a gen_passwords.sh file which will automatically generate new secrets for your DataForge installation. Simply run the command bash ./gen_passwords.sh to generate random secrets.

Take note of the newly generated DF_SERVER_DEFAULTACCOUNTPASSWORD in the .env file, as you will need it for your first login to your DataForge server.

If you have to manually set the secrets, make sure to choose a different secret for each NSQ secret option (For example DF_SERVER_NSQSECRET must not be the same as DF_COLLECTOR_NSQSECRET).

Secrets and some key configuration options are managed using a .env file.

# Set your DataForge version in the line below (For example 7.6.13)
DF_VERSION=YOUR_DATAFORGE_VERSION
# The docker registry the DataForge images are pulled from
DF_DOCKER_REGISTRY_PATH=images.intellitrend.de/dataforge/dataforge-core

# The timezone of your DataForge server
TIMEZONE=Europe/Berlin

# The database (root) credentials
DB_DATABASE=dataforge
DB_PASSWORD=[REPLACE_ME]

# The master key used to encrypt database values and sign sessions
MASTER_KEY=[REPLACE_ME]

# The credentials of the DataForge default account (admin@dataforge.loc)
DF_SERVER_DEFAULTACCOUNTENABLED=true
DF_SERVER_DEFAULTACCOUNTPASSWORD=[REPLACE_ME]

# Secrets for authentication on the NSQ message queue
DF_SERVER_NSQSECRET=[REPLACE_ME]
DF_COLLECTOR_NSQSECRET=[REPLACE_ME]
DF_PREPROCESSOR_NSQSECRET=[REPLACE_ME]
DF_RENDERER_NSQSECRET=[REPLACE_ME]
DF_DELIVERER_NSQSECRET=[REPLACE_ME]

3. Installing your license key file

Copy your DataForge license key (from step 1) to a file with the name dataforge.key in the same directory as your docker-compose.yaml.

Your DataForge server will use this file to communicate with one of our redundant license servers and activate the modules in your subscription. For this reason, the DataForge server will need to be able to reach our license servers.

4. Review the docker-compose.yaml file

The included docker-compose.yaml file defines how the services will start.

If you have any problems setting up the DataForge server, you may need to modify this file to match your system configuration.

You can consult the official docker-compose file reference or contact our support for further assistance.

The following is the content of the included docker compose file (docker-compose.yaml). This docker compose file is used to deploy all services including the DataForge server.

services:
    # ------------ DataForge Server ------------
    df-server:
        image: ${DF_DOCKER_REGISTRY_PATH}/df-server:${DF_VERSION}
        container_name: df-server
        command: ./df-server
        depends_on:
            - mariadb
        environment:
            # API Server Configuration
            DF_SERVER_LOGLEVEL: "info"
            DF_SERVER_GRPCPORT: "8090"
            DF_SERVER_WEBPORT: "8091"
            DF_SERVER_RESTPORT: "8092"
            DF_SERVER_SLOWQUERYTHRESHOLD: "1500"
            DF_SERVER_ENCRYPTIONKEY: $MASTER_KEY
            DF_SERVER_DEFAULTACCOUNTENABLED: $DF_SERVER_DEFAULTACCOUNTENABLED
            DF_SERVER_DEFAULTACCOUNTPASSWORD: $DF_SERVER_DEFAULTACCOUNTPASSWORD
            DF_SERVER_THREADGUARDENABLED: "true"
            DF_SERVER_SESSIONEXPIRY: 3600
            # User Sync
            DF_SERVER_USERSYNCONSTART: true
            DF_SERVER_USERSYNCSCHEDULE: "0 3 * * *" # we will sync with all zabbix servers, every day at 03:00
            # User Cache
            DF_SERVER_USERCACHEEVICTIONTIME: 3600 # users get evicted after one hour of inactivity
            DF_SERVER_USERCACHEHOUSEKEEPERSCHEDULE: "*/15 * * * *" # run housekeeper every 15min
            # HA
            DF_SERVER_ENABLECLUSTERING: "false"
            DF_SERVER_HALEADERLEASELENGTH: 60
            DF_SERVER_HALEADERINITIALLEASELENGTH: 150
            DF_SERVER_HANODELONGPOLL: 30
            DF_SERVER_HANODESHORTPOLL: 15
            # NSQ
            DF_SERVER_NSQDISABLE: "false"
            DF_SERVER_NSQTRANSACTIONTIMEOUT: 600 # 10 minutes maximum transactions between services
            DF_SERVER_SERVICEIDENTITY: "dataforge_server"
            # NSQ connection
            DF_SERVER_NSQCONSUMEPORT: "4161"
            DF_SERVER_NSQCONSUMEADDRESS: "nsqlookupd"
            DF_SERVER_NSQPRODUCEPORT: "4150"
            DF_SERVER_NSQPRODUCEADDRESS: "nsqd"
            # NSQ TLS
            DF_SERVER_NSQTLSENABLE: "false"
            DF_SERVER_NSQTLSCACERTIFICATE: "/usr/local/intellitrend/df/ssl/ca.cert"
            DF_SERVER_NSQTLSSKIPCERTIFICATEVALIDATION: "true"
            # NSQ Authentication
            DF_SERVER_NSQAUTHENABLE: "true"
            DF_SERVER_NSQAUTHPORT: "4180"
            DF_SERVER_NSQSECRET: $DF_SERVER_NSQSECRET
            DF_SERVER_NSQAUTHSERVERSECRET: $DF_SERVER_NSQSECRET
            DF_SERVER_NSQAUTHCOLLECTORSECRET: $DF_COLLECTOR_NSQSECRET
            DF_SERVER_NSQAUTHPREPROCESSORSECRET: $DF_PREPROCESSOR_NSQSECRET
            DF_SERVER_NSQAUTHRENDERERSECRET: $DF_RENDERER_NSQSECRET
            DF_SERVER_NSQAUTHDELIVERERSECRET: $DF_DELIVERER_NSQSECRET
            DF_SERVER_LICENSE: "/usr/local/intellitrend/df/etc/df.key"
            # Zabbix
            DF_SERVER_ZABBIXAPITRACEENABLE: "false"
            # TLS
            DF_SERVER_TLSENABLE: "false"
            DF_SERVER_TLSCERTIFICATEPATH: "/usr/local/intellitrend/df/ssl/df-server.cert"
            DF_SERVER_TLSPRIVATEKEYPATH: "/usr/local/intellitrend/df/ssl/df-server.key"
            # Database
            DF_SERVER_DBPORT: 3306
            DF_SERVER_DBNAME: $DB_DATABASE
            DF_SERVER_DBADDRESS: mariadb
            DF_SERVER_DBUSER: root
            DF_SERVER_DBPASSWORD: $DB_PASSWORD
            DF_SERVER_DBDRIVER: "mysql"
            DF_SERVER_DBTRACEENABLE: "false"
            DF_SERVER_DBRECONNECTDELAY: 15
            # LDAP
            DF_SERVER_LDAPADDRESS: ""
            DF_SERVER_LDAPPORT: ""
            DF_SERVER_LDAPDOMAIN: ""
            DF_SERVER_LDAPTLSENABLE: "false"
            DF_SERVER_LDAPTLSVERIFYPEER: "false"
            DF_SERVER_LDAPTLSCACERTIFICATE: "false"
            # Metrics
            DF_SERVER_METRICSPORT: "8094"
            DF_SERVER_METRICSHISTORYSIZE: 1000
            DF_SERVER_METRICSSTATEFILE: "/usr/local/intellitrend/df/var/metrics/state.json"
        #ports:
        #  - 4180:4180 # NSQ-Auth
        #  - 8090:8090 # gRPC
        #  - 8091:8091 # gRPC-Web (exposed through df-frontend reverse proxy)
        #  - 8092:8092 # REST
        #  - 8094:8094 # Metrics
        volumes:
            # Place your license key here
            - ./dataforge.key:/usr/local/intellitrend/df/etc/df.key:ro
            # A directory used to persist metrics between restarts
            - ./df-server/data/metrics/:/usr/local/intellitrend/df/var/metrics:rw

    # ------------   DF Frontend   ------------
    df-frontend:
        container_name: df-frontend
        image: ${DF_DOCKER_REGISTRY_PATH}/df-frontend:${DF_VERSION}
        ports:
            - 80:80
            #- 443:443
        volumes:
            - ./df-frontend/data/cfg/df.conf:/etc/nginx/conf.d/default.conf
            # Uncomment the line below to mount certificates and use port 443, if you do not whish to use an external ssl offloading proxy
            #- ./df-frontend/data/ssl/:/etc/nginx/ssl/
            - ./df-frontend/data/params/proxy_params:/etc/nginx/proxy_params

    # ------------  DF Collector   ------------
    df-collector:
        container_name: df-collector
        image: ${DF_DOCKER_REGISTRY_PATH}/df-collector:${DF_VERSION}
        environment:
            DF_COLLECTOR_LOGLEVEL: "info"
            DF_COLLECTOR_NSQSERVICEIDENTITY: "dataforge_collector"
            DF_COLLECTOR_NSQSECRET: $DF_COLLECTOR_NSQSECRET
            DF_COLLECTOR_NSQCONSUMEPORT: "4161"
            DF_COLLECTOR_NSQCONSUMEADDRESS: "nsqlookupd"
            DF_COLLECTOR_NSQPRODUCEPORT: "4150"
            DF_COLLECTOR_NSQPRODUCEADDRESS: "nsqd"
        depends_on:
            - nsqd

    # ------------  DF Preprocessor  ------------
    df-preprocessor:
        container_name: df-preprocessor
        image: ${DF_DOCKER_REGISTRY_PATH}/df-preprocessor:${DF_VERSION}
        environment:
            DF_PREPROCESSOR_LOGLEVEL: "info"
            DF_PREPROCESSOR_NSQSERVICEIDENTITY: "dataforge_preprocessor"
            DF_PREPROCESSOR_NSQSECRET: $DF_PREPROCESSOR_NSQSECRET
            DF_PREPROCESSOR_NSQCONSUMEPORT: "4161"
            DF_PREPROCESSOR_NSQCONSUMEADDRESS: "nsqlookupd"
            DF_PREPROCESSOR_NSQPRODUCEPORT: "4150"
            DF_PREPROCESSOR_NSQPRODUCEADDRESS: "nsqd"
        depends_on:
            - nsqd

    # ----------- DF Deliverer   ------------
    df-deliverer:
        container_name: df-deliverer
        image: ${DF_DOCKER_REGISTRY_PATH}/df-deliverer:${DF_VERSION}
        environment:
            DF_DELIVERER_LOGLEVEL: "trace"
            DF_DELIVERER_NSQSERVICEIDENTITY: "dataforge_deliverer"
            DF_DELIVERER_NSQCONSUMEPORT: "4161"
            DF_DELIVERER_NSQCONSUMEADDRESS: "nsqlookupd"
            DF_DELIVERER_NSQPRODUCEPORT: "4150"
            DF_DELIVERER_NSQPRODUCEADDRESS: "nsqd"
            DF_DELIVERER_NSQSECRET: $DF_DELIVERER_NSQSECRET
        depends_on:
            - nsqd

    # ------------   DF Renderer   ------------
    df-renderer:
        container_name: df-renderer
        image: ${DF_DOCKER_REGISTRY_PATH}/df-renderer:${DF_VERSION}
        environment:
            DF_RENDERER_LOGLEVEL: "info"
            DF_RENDERER_NSQSERVICEIDENTITY: "dataforge_renderer"
            DF_RENDERER_NSQCONSUMEPORT: "4161"
            DF_RENDERER_NSQCONSUMEADDRESS: "nsqlookupd"
            DF_RENDERER_NSQPRODUCEPORT: "4150"
            DF_RENDERER_NSQPRODUCEADDRESS: "nsqd"
            DF_RENDERER_NSQSECRET: $DF_RENDERER_NSQSECRET
            DF_RENDERER_CHROMEADDRESS: "pdf-worker"
            DF_RENDERER_CHROMEPORT: "9222"
            DF_RENDERER_WEBSERVERADDRESS: "df-renderer"
        depends_on:
            - nsqd

    # ------------ Chrome Headless ------------
    pdf-worker:
        image: chromedp/headless-shell:114.0.5735.199
        container_name: pdf-worker

    # ------------ Mariadb Database ------------
    mariadb:
        image: mariadb
        restart: always
        container_name: mariadb
        command: mariadbd --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
        environment:
            MYSQL_DATABASE: $DB_DATABASE
            MYSQL_ROOT_PASSWORD: $DB_PASSWORD
        volumes:
            - ./mariadb/data/var/lib/mysql:/var/lib/mysql:rw
        #ports:
        #    - "3306:3306"

    # ------------ NSQLookupd ------------
    nsqlookupd:
        image: nsqio/nsq
        container_name: nsqlookupd
        command: /nsqlookupd
        #ports:
        #    - 4161:4161

    # ------------ NSQD ------------
    nsqd:
        image: nsqio/nsq
        container_name: nsqd
        command: /nsqd --lookupd-tcp-address=nsqlookupd:4160 --auth-http-address=df-server:4180 --mem-queue-size=1000000 --max-msg-size=419430400 --data-path /nsqd-data --broadcast-address=nsqd
        depends_on:
            - nsqlookupd
        volumes:
            - ./nsqd/data/nsqd-data:/nsqd-data
        #ports:
        #    - 4150:4150

    # ------------ SeaweedFS / S3 ------------
    s3:
        image: chrislusf/seaweedfs
        container_name: s3
        #ports:
        #    - 8333:8333 # S3 API HTTP
        #    - 9327:9327 # Metrics port (must be enabled in command)
        command: 'server -s3 -s3.config=/seaweed-config.json -master.volumePreallocate=false -ip.bind=0.0.0.0' # -metricsPort=9327
        volumes:
            - ./seaweed/data:/data:rw
            - ./seaweed-config.json:/seaweed-config.json:ro

5. Run services

First, make sure to sign in to the docker image registry (docker login images.intellitrend.de) using the credentials that were provided to you when purchasing DataForge.

Start the all services by running docker compose up -d.

You may see the server logs to see if any errors occur using docker compose logs.

You can now access the DataForge frontend using HTTP on port 80.

To enable HTTPS on the server (which is also required for logging in from most web browsers), you need to configure a reverse proxy to provide DataForge over HTTPS.

6. Configure HTTPS / reverse proxy

By default, the df-frontend container listens on port 80 (unprotected HTTP) and serves the DataForge frontend to browsers. It also forwards requests matching requests to the gRPC-Web API of your DataForge server.

The DataForge frontend (and the gRPC-Web API of the DataForge server) needs to be reachable from your browser using HTTPS.

You can either configure an external reverse proxy or you can set up the nginx server in the frontend container to serve HTTPS.

6.1. Using an external reverse proxy

If you have an existing reverse proxy handling HTTPS termination, you can simply configure the reverse proxy to pass any requests to a subdomain of your choice to port 80 of the df-frontend container.

Most common reverse proxies (such as Traefik, nginx or Apache) should be usable for this purpose.

For configuration instructions, please consult the documentation of your specific reverse proxy software. The following endpoints need to be configured:

Any URLs at https://yoursubdomain/ need to point to the HTTP or HTTPS endpoint of the df-frontend container.
The reverse proxy must allow websocket upgrades for any URLs starting with https://yoursubdomain/service.intellimon/Stream*, where Stream* means any method that starts with Stream.

If you use an external reverse proxy, you need to make sure that the connection from the reverse proxy to the df-frontend container is trusted (i.e. the external reverse proxy is installed on the same server), because otherwise the traffic might be intercepted.

This could allow attackers to leak secrets or modify requests sent to the DataForge server.

If the reverse proxy is running on a different host or the connection is not trusted for some other reason, you can follow the steps described in Section 6.2. using an internal (potentially self-signed) certificate to additionally protect this communication.

6.2. Serve HTTPS using included nginx

The df-frontend container includes an nginx server which serves the DataForge frontend to browsers and also forwards any requests at /service.intellimon to the gRPC-Web API of your DataForge server.

By default, this container only serves unprotected HTTP, but you can configure this container to serve HTTPS.

To configure this nginx server to serve HTTPS, simply follow these steps:

Create a new directory df-frontend/data/ssl.
Rename your HTTPS/TLS certificate and private key to df-frontend.cert and df-frontend.key respectively and place them both inside the newly created df-frontend/data/ssl/ directory

Edit the file df-frontend/data/cfg/df.conf in a text editor of your choice.

Uncomment the block of example configuration for HTTPS at the bottom of the file.
Optionally, you can comment out the first server {...} block in the file to disable HTTP over port 80 entirely.

Your df.conf (nginx configuration file) should now look like this (with additional lines omitted at [...]):

upstream df-server {
        server df-server:8091;
}

# [...]

server {
        listen 443 ssl;
        listen [::]:443 ssl;
        server_name df-frontend-secure;
        root /usr/share/nginx/html/;
        index index.html index.htm;
        ssl_certificate /etc/nginx/ssl/df-frontend.cert;
        ssl_certificate_key /etc/nginx/ssl/df-frontend.key;
        ssl_protocols       TLSv1.2 TLSv1.3;
        ssl_ciphers         HIGH:!aNULL:!MD5;

        # [...]

        # Websockets for GRPC Streams - always starting with Stream like StreamNSQ, StreamRTCNS
        location ~ ^/service.intellimon/Stream.*$ {
                proxy_pass http://df-server;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";
        }

        # Regular calls
        location /service.intellimon {
                proxy_pass http://df-server;
        }

        location / {
                try_files $uri $uri/ =404;
        }
}

Edit the docker-compose.yaml file:
- In the df-frontend service definition, uncomment the port bind mount of ./df-frontend/data/ssl/, so the certificates are accessible by the frontend container.
- In the df-frontend service definition, uncomment the port mapping 443:443, so the HTTPS service is exposed outside the container.
- Optionally, you can comment out the port mapping for port 80 to force clients to use HTTPS over port 443.
- The df-frontend service definition in your docker-compose.yaml file should now look like this (with additional lines omitted at every [...]):
```
# [...]

df-frontend:
    # [...]
    ports:
        - 80:80
        - 443:443
    volumes:
        # [...]
        - ./df-frontend/data/ssl/:/etc/nginx/ssl/
        # [...]

# [...]
```
Restart your DataForge installation using docker compose down && docker compose up -d

7. Configure S3 bucket

If you would like to use the reporting and DataForge AI functionality, you need to configure an S3 bucket in DataForge.

While you can use any S3-compatible object storage, the default docker-compose file includes a SeaweedFS container. The credentials for the S3-API are configured in the seaweed-config.json file. If you ran the ./gen_passwords.sh script, a secure secret key will have already been created, otherwise, you’ll need to manually configure a secure password here.

Use s3:8333 (the docker hostname and port) as the object storage URL, the credentials in the seaweed-config.json file, as well as a bucket name of your choice when configuring the S3 storage for your Zabbix server in DataForge. Keep in mind that you need to use different bucket names for different Zabbix servers.

If you need to access the S3 bucket from outside of the DataForge instance, we recommend setting up a reverse proxy to forward https://s3.example.org/ to port 8333 on the S3 / SeaweedFS docker container.

Last modified January 28, 2025: (3a62159cf)