Add Subpage

Suggest an Edit

Share Feedback

Contributors

See Revisions

Connector Catalog

Data Providers

OEM

Provider A

Fleet & Telematics

Provider B

Data Aggregator

Provider C

Connectivity Provider

Provider D

Other Provider

Provider E

DIANA

Data Integration

Source Connectors

Push Pull

Data Ingest API Managed Kafka

AMQP Azure Blob Store Fleet Telemetry FTP Geotab Kafka Kinesis MQTT Pulsar Webfleet

Processing

+more Filtering Data Enrichment VSS Standardization

Sink Connectors

Push Pull

AMQP AWS S3 Azure Blob Store HTTP IoT-Insights Kafka MQTT

Managed Kafka

No Data Loss

Retry Logic

Retention Time

Fully Managed

Data Consumers

Bosch Project

Consumer A

Fleet Manager

Consumer B

Data Customer

Consumer C

Bosch BRIM

Consumer D

Bosch SSH

Analytics Hub

Managed

Data Sources

Connectors that bring data into DIANA. The customer either sends data directly to DIANA (Push), or DIANA actively fetches it from the customer’s infrastructure (Pull).

Push Connectors

Data Ingest API — Any HTTP Client Bosch CIAM

Capabilities

• Generic data ingest without schemas

• Metadata attachable as HTTP headers (prefix X-METADATA-)

• Records the OAuth client ID of the technical user as metadata

• Max upload limit: 15 MB per request

Configuration

• API Documentation

• OAuth Authentication (Bosch CIAM):

  • Tenant

  • Client ID

  • Client Secret

Metadata

Field	Name	Type	Description
datacontenttype	Content Type	String	Content type from HTTP request header or application/octet-stream as default
Custom Extensions	HTTP Headers	String	All HTTP headers prefixed with X-METADATA-, sanitized and added as CloudEvent extensions
ingestsubpath	Request URI	String	Complete request URI including path and query parameters

Managed Kafka — DIANA-hosted Kafka Topic

Capabilities

• Secure, isolated Kafka topics — fully operated by the DIANA team

• QoS levels: poc, low, high

• Customer produces to DIANA-managed topics; DIANA auto-consumes

• Compression strongly recommended — see Confluent Cloud supported codecs

Configuration

Credentials and bootstrap server details are provided by the DIANA team.

• Bootstrap Server URL (provided by DIANA)

• Topic name (provided by DIANA)

• SASL/SSL Authentication:

  • Username (provided by DIANA)

  • Password (provided by DIANA)

Metadata

Field	Name	Type	Description
topic	Topic	String	Name of the Kafka topic from which the record was consumed
kafkakey	Kafka Key	String	Kafka message key (only when present)
Custom Extensions	Kafka Headers	Various	All Kafka message headers with sanitized values
content-type	Content Type	String	application/octet-stream

Pull Connectors

AMQP — Active MQ IBM MQ Rabbit MQ Azure Event Hub Solace

Capabilities

• AMQP 1.0 protocol

• Read from multiple queues of the same broker

• AMQP headers forwarded as event metadata

Configuration

• Broker URL (e.g. amqp://my-amqp-broker.amqp:5672)

• Topic(s) for data retrieval

• Password Authentication:

  • Username

  • Password

• Certificate Authentication:

  • PKCS#12 or PEM certificate file

  • Certificate password

Metadata

Field	Name	Type	Description
queue	Queue	String	Name of the AMQP queue from which the message was consumed
Custom Extensions	JMS Properties	Various	Application-defined message properties (excluding JMSX prefix)
content-type	Content Type	String	application/octet-stream

Azure Blob Store — Azure Blob Storage Azure Data Lake Gen2

Capabilities

• Flat file ingestion from Azure Blob Storage

• Azure Data Lake Gen2 with directory structures

• Deletes files from storage account after successful ingest

• Limited to one application instance per source

Configuration

• Container name (e.g. data)

• Connection URL (e.g. storageaccountfoo.microsoft.com)

• ADLS Gen2 enabled (whether to use the Azure Data Lake Gen2 API)

• Directory name (e.g. dir/subdir)

• Shared Key Authentication:

  • Account Name

  • Account Key

• Azure AD / Service Principal Authentication:

  • Client ID

  • Client Secret

  • Tenant ID

Metadata

Field	Name	Type	Description
blobname	Blob Name	String	Name / path of the Azure blob that was processed
Custom Extensions	Blob Metadata	Various	User-defined metadata key-value pairs attached to the blob
content-type	Content Type	String	application/octet-stream

Fleet Telemetry — Tesla Fleet Telemetry Server

Capabilities

• Receives vehicle telemetry from Tesla vehicles via mTLS on port 8443

• Record types: vehicle data (V), alerts, errors, connectivity

• Individual record types can be routed to logs instead of Kafka

• Configurable rate limiting

Configuration

Requires a self-hosted Tesla Vehicle Command server configured to forward telemetry to the Fleet Telemetry receiver. Data enters DIANA via a Kafka Source consuming from the importer topics.

Metadata

Field	Name	Type	Description
topic	Topic	String	Kafka topic from which the record was consumed (e.g. importer.TESLA_TELEMETRY._V)
content-type	Content Type	String	application/octet-stream

FTP — SFTP Server

Capabilities

• New file discovery based on file modification time (mtime)

• Files must have strictly increasing mtime values — no two files may share the same mtime

• Retry exports must produce a more recent mtime than any previously ingested file

Configuration

• Server URL (e.g. my-ftp-server.ftp)

• Server Port (e.g. 22)

• Directory to read from (e.g. upload)

• Key fingerprint of the FTP server

• Auth Type: password, publickey, or publickey-password

• Password Authentication:

  • Username

  • Password

• Public Key Authentication:

  • Public Key

• Public Key + Password Authentication:

  • Public Key

  • Password

Metadata

Field	Name	Type	Description
ftpfilename	FTP File Name	String	Name of the file that was processed
ftpfilepath	FTP File Path	String	Full path of the file on the server
content-type	Content Type	String	application/octet-stream

Geotab — Geotab API

Capabilities

• Feeds: Device, Diagnostic, FaultData, LogRecorder, StatusData

• All-data endpoints: FailureMode

• Poll state persisted to the file system via Azure CSI driver / hermes-file-storage StorageClass

Configuration

• Result count per poll (e.g. 1000)

• Initial poll version (e.g. 0)

• Scheduling rates:

  • Feed rate (e.g. 60000 ms)

  • All-data rate (e.g. 50000000 ms)

• Database(s):

  • Name (e.g. rb_internal)

  • Server (e.g. my.geotab.com)

• Basic Authentication:

  • Username

  • Password

Metadata

Field	Name	Type	Description
database	Database	String	Name of the Geotab database being polled
user	User	String	Username used for authentication
deviceid	Device ID	String	Device identifier (only when device-specific data)
version	Version	String	Current poll version from the API response
fromversion	From Version	String	Starting version for the poll request
prefix	Prefix	String	Data type class name (e.g. LogRecord, StatusData)
content-type	Content Type	String	application/json

Kafka — Azure Event Hub AWS MSK Native Kafka Broker

Capabilities

• OAuth authentication (generic, Azure EventHub, AWS MSK)

• mTLS certificate support

• Kafka headers forwarded as event metadata

• Partition key forwarding to preserve message order

Configuration

• Bootstrap Server URL (e.g. my-kafka-cluster.kafka:9092)

• Topic(s) for data retrieval

• Auth mechanism (SASL/PLAINTEXT)

• Profiles: sasl, mtls, oauth, aws, eventhub

• SASL Authentication:

  • Username

  • Password

• mTLS Authentication:

  • PKCS#12 or PEM certificate file

  • Certificate password

• OAuth Authentication (Generic, EventHub, AWS):

  • Client ID

  • Client Secret

  • Provider-specific details

Metadata

Field	Name	Type	Description
topic	Topic	String	Name of the Kafka topic from which the message was consumed
kafkakey	Kafka Key	String	Kafka message key (only when present)
partitionkey	Partition Key	String	Kafka partition key (optional; can be disabled)
Custom Extensions	Kafka Headers	Various	All Kafka message headers with sanitized values
content-type	Content Type	String	application/octet-stream

Kinesis — AWS Kinesis Data Streams

Capabilities

• Polls AWS Kinesis Data Streams for new records

• Each record forwarded as an individual CloudEvent

• Limited to one application instance per source

Configuration

• Stream name (e.g. my-stream)

• AWS region (e.g. eu-central-1)

• Scheduling rate (e.g. 20000 ms)

• Record limit per poll (e.g. 100)

• Start position: TRIM_HORIZON, LATEST, or a specific timestamp

• AWS IAM Authentication:

  • Access Key

  • Secret Key

Metadata

Field	Name	Type	Description
partitionkey	Partition Key	String	Partition key from the record (when present)
streamname	Stream Name	String	Configured Kinesis stream name
content-type	Content Type	String	application/octet-stream

MQTT — MQTT Broker v3.1.1 MQTT Broker v5.0

Capabilities

• Persistent sessions

• QoS level 0, 1, 2

• MQTT v3.1.1 and v5.0

• Limited to one application instance per source

Configuration

• Broker URL (e.g. mqtt://my-mqtt-broker:1883)

• Topics and QoS

• Password Authentication:

  • Username

  • Password

• Certificate Authentication:

  • PKCS#12 client certificate (base64 encoded)

  • Certificate password

  • JKS truststore (base64 encoded)

  • Truststore password

• Retry configuration

Metadata

Field	Name	Type	Description
topic	Topic	String	Name of the MQTT topic from which the message was consumed
Custom Extensions	MQTT Headers	Various	Application-defined header properties passed to the message processor
content-type	Content Type	String	application/octet-stream

Pulsar — Apache Pulsar

Capabilities

• mTLS certificate support

• Pulsar message properties forwarded as event metadata

• Durable subscriptions with offset management

Configuration

• Broker URL (e.g. pulsar://my-pulsar-broker:6650)

• Topic(s) for data retrieval

• Basic Authentication:

  • Username

  • Password

• OAuth Authentication:

  • Client ID

  • Client Secret

  • Token URL

• Certificate Authentication:

  • PKCS#12 or PEM certificate file

  • Certificate password

Metadata

Field	Name	Type	Description
topic	Topic	String	Name of the Pulsar topic from which the message was consumed
Custom Extensions	Message Properties	Various	Application-defined message properties attached to the Pulsar message
content-type	Content Type	String	application/octet-stream

Webfleet — Webfleet Connect API

Capabilities

• Creates and manages a message queue subscription for configured message classes

• Pulls and acknowledges messages after successful ingestion

• Monitors outstanding message count

• Limited to one application instance per source

Configuration

• Webfleet API base URL (e.g. https://api.webfleet.com)

• Webfleet Account ID

• Message classes for queue subscription

• Scheduling rate (e.g. 20000 ms)

• Max memory buffer for message retrieval (e.g. 16 MB)

• Basic Authentication:

  • Username

  • Password

• API Key Authentication:

  • API Key

Metadata

Field	Name	Type	Description
account	Account ID	String	The Webfleet account ID for the origin of the data
content-type	Content Type	String	application/json

Data Sinks

Connectors that distribute processed data out of DIANA. DIANA either delivers data to the customer’s infrastructure (Push), or the customer consumes it from DIANA’s managed topics at their own pace (Pull).

Push Connectors

AMQP — Active MQ IBM MQ Rabbit MQ Azure Event Hub Solace

Capabilities

• AMQP 1.0 protocol

• Payload compression before forwarding

Configuration

• Broker URL (e.g. amqp://my-amqp-broker.amqp:5672)

• Topic to send data to

• Password Authentication:

  • Username

  • Password

• Certificate Authentication:

  • Certificate file (PKCS#12 or PEM)

  • Certificate password

• Compression on/off

Metadata

Field	Name	Type	Description
uid	UID	String	Randomly generated UUID for message tracking
messageId	Message ID	String	Diana ingestion ID from the CloudEvent extension metadata

AWS S3 — Amazon S3 Minio

Capabilities

• Blob name dynamically built from CloudEvent header fields

• Payload compression before upload (gzip / zip / none)

• Duplicate prevention

Configuration

• Endpoint (e.g. s3.eu-central-1.amazonaws.com)

• Region (e.g. eu-central-1)

• Bucket name (e.g. data-bucket)

• URL path-style on/off

• File naming pattern

• AWS IAM Authentication:

  • Access Key

  • Access Secret

• Compression: gzip / zip / none

• Duplicate prevention on/off

Metadata

Field	Name	Type	Description
fileName	File Name	String	Generated file name based on the configured pattern
content-type	Content Type	String	Data content type from the CloudEvent (e.g. application/json)
contentMD5	Content MD5	String	Base64 encoded MD5 hash of the compressed payload for integrity verification
S3 object metadata	Custom Extensions	Various	All CloudEvent extension properties as S3 object metadata with normalized names

Azure Blob Store — Azure Blob Store Azurite Dev Container

Capabilities

• Blob name dynamically built from CloudEvent header fields

• Payload compression before upload

• Duplicate prevention

Configuration

• Container name (e.g. data)

• Connection URL (e.g. storageaccountfoo.microsoft.com)

• Blob naming pattern

• Duplicate prevention on/off

• Shared Key Authentication:

  • Account Name

  • Account Key

• Azure AD / Service Principal Authentication:

  • Client ID

  • Client Secret

  • Tenant ID

• Compression on/off

Metadata

Field	Name	Type	Description
Content-Type	Content Type	String	Data content type from the CloudEvent (used as blob HTTP header)
Content-MD5	Content MD5	String	MD5 hash of the compressed payload for integrity verification
blob metadata	Custom Extensions	String	All CloudEvent extension properties normalized and included as blob metadata

HTTP — Any HTTP API / Backend

Capabilities

• Sends data to exactly one endpoint via POST requests

• Content type taken from the originally ingested data

• Responses evaluated for errors only (4XX / 5XX); body discarded

• Custom request headers via Spring Expression Language (SpEL)

Configuration

• Target backend URL (e.g. http://foo.bar/data)

• Basic Authentication:

  • Username

  • Password

• OAuth:

  • IDP Token URL

  • Client ID

  • Client Secret

  • Scope

  • Grant Type (client_credentials)

• API Key Authentication:

  • API Key

• C-Technology Update API Authentication:

  • Auth URL

  • Token URL

  • Client Secret

  • Token Prefetch Seconds

• Geotab DIG Authentication:

  • Additional properties per Geotab DIG specification

Metadata

Field	Name	Type	Description
Content-Type	Content Type	String	Content type from the CloudEvent or configured value (defaults to application/octet-stream)
Custom Headers	Custom Extensions	Various	HTTP headers built from SpEL expressions with access to all CloudEvent properties

IoT-Insights — IoT Insights Data Recorder API

Capabilities

• Uploads data to exactly one IoT Insights Project

• Basic authentication as provided by IoT Insights

Configuration

• IoT Insights data recorder project URL (e.g. https://bosch-iot-insights.com/data-recorder-service/v2/pqc8349)

• Basic Authentication:

  • Username

  • Password

Metadata

Field	Name	Type	Description
X-Metadata	X-Metadata	String	Comma-separated list of typed CloudEvent extensions and content type with data type annotations
Content-MD5	Content MD5	String	Base64 encoded MD5 hash of the compressed payload for integrity verification

Kafka — Azure Event Hub AWS MSK Native Kafka Broker

Capabilities

• OAuth authentication (generic, Azure EventHub, AWS MSK)

• mTLS certificate support

• CloudEvent metadata forwarded as Kafka headers

Configuration

• Bootstrap Server URL (e.g. my-kafka-cluster.kafka:9092)

• Topic to send data to

• Auth mechanism (SASL/PLAINTEXT)

• Profiles: sasl, mtls, oauth, aws, eventhub

• SASL Authentication:

  • Username

  • Password

• mTLS Authentication:

  • Certificate file

  • Certificate password

• OAuth Authentication (Generic, EventHub, AWS):

  • Client ID

  • Client Secret

  • Provider-specific details

Metadata

Field	Name	Type	Description
content-type	Content Type	String	Data content type from the CloudEvent (if present)
Kafka Headers	Custom Extensions	String	All CloudEvent extension properties as individual Kafka headers

MQTT — MQTT Broker v3.1.1 MQTT Broker v5.0

Capabilities

• Persistent sessions

• QoS level 0, 1, 2

• MQTT v3.1.1 and v5.0

Configuration

• Broker URL (e.g. mqtt://my-mqtt-broker:1883)

• Topics, QoS, and retained message flag

• Retry configuration

• SASL Authentication:

  • Username

  • Password

• mTLS Authentication:

  • Certificate file

  • Certificate password

• Event Hub Mode Authentication:

  • Azure Tenant ID

  • Azure Client ID

  • Azure Client Secret

Metadata

Field	Name	Type	Description
content-type	Content Type	String	Data content type from the CloudEvent (if present)
MQTT User Properties	Custom Extensions	String	All CloudEvent extension properties as individual MQTT user properties

Pull Connectors

Managed Kafka — DIANA-hosted Kafka Topic

Capabilities

• Secure, isolated Kafka topics — fully operated by the DIANA team

• QoS levels: poc, low, high

• DIANA publishes processed events; customer consumes with any standard Kafka client

• No infrastructure setup required on the customer side

Configuration

Credentials and bootstrap server details are provided by the DIANA team.

• Bootstrap Server URL (provided by DIANA)

• Topic name (provided by DIANA)

• SASL/SSL Authentication:

  • Username (provided by DIANA)

  • Password (provided by DIANA)

Metadata

Field	Name	Type	Description
content-type	Content Type	String	Data content type from the CloudEvent (if present)
Kafka Headers	Custom Extensions	String	All CloudEvent extension properties as individual Kafka headers

---

Platform Metadata

The DIANA platform attaches comprehensive metadata to events at every stage of the integration pipeline, creating a rich context for data lineage, routing decisions, filtering, and processing logic. Metadata accumulates as an event moves through the system:

• Source Stage — initial headers attached when data is ingested from a source connector

• Transformation Stage — additional headers may be added or modified during processing

• Sink Stage — final metadata applied before data reaches its destination

Default values are only set when fields are absent, preserving any metadata already present through multiple processing stages.

Source Event Metadata

Field	Display Name	Type	Description
dianaingestid	Ingest ID	String	Unique identifier for the ingestion process.
dianaingesttimestamp	Ingest Timestamp	String/OffsetDateTime	Timestamp when the event entered DIANA. Auto-generated if not present.
dianadatasize	Data Size	String/Integer	Payload size in bytes. Calculated from actual data if not present.
dianaproducertenant	Producer Tenant	String	Tenant identifier of the event producer. Set from configuration.
dianadatastreamid	Data Stream ID	String	Identifier for the data stream. Set from sidecar configuration.
dianamd5hash	MD5 Hash	String	MD5 hash of the original payload before compression or transformation.

Sink Event Metadata

Field	Display Name	Type	Description
dianaconsumertenant	Consumer Tenant	String	Tenant identifier of the event consumer.
dianaconsumerpipelineid	Consumer Pipeline ID	String	Identifier for the consumer pipeline.