An event-sourced system that stores domain events in EventStoreDB could benefit a lot from replicating these events to a search engine, such as Elasticsearch. When all the events are available in Elasticsearch, the system can be queried for events, and when events properly convey information about business operations, you can discover quite a lot using tools like Kibana.
This connector allows you to replicate events to Elasticsearch without having any knowledge about your system insights like event contracts, or event types. The natural limitation of this approach is that events must be stored in EventStoreDB in JSON format.
The target configuration is used to connect to Elasticsearch, as well as create the necessary elements in Elasticsearch (index template, index rollover policy, and data stream).
The first step is to specify the target assembly in the
connector section of the configuration file.
connector: connectorId: "esdb-elastic-connector" connectorAssembly: "Eventuous.Connector.EsdbElastic"
The target configuration has the following parameters:
connectionString- Elasticsearch connection string, should not be used when the
cloudId- Elasticsearch cloud id, should be used when the
connectionStringis not specified
apiKey- Elasticsearch API key
dataStream- the index configuration section
indexName- the index name (data stream name)
template- the template section
templateName- the template name
numberOfShards- the number of shards for the data stream, default is
numberOfReplicas- the number of replicas for the data stream, default is
lifecycle- the lifecycle section
policyName- the rollover policy name
tiers- the rollover policy tiers, see the structure of a
tier section is used to configure the rollover policy tiers. The tier name must match the available tier in your Elasticsearch cluster.
tier- the tier name (
minAge- the minimum age of the data stream (for example
10dfor 10 days)
priority- the priority of the tier (
0is the lowest priority)
rollover- the rollover policy section
maxAge- the maximum index age
maxSize- the maximum index size
maxDocs- the maximum index documents
forceMerge- the force merge policy section
maxNumSegments- the maximum number of segments
readOnly- if the tier will be read only
delete- if the tier will be deleted
When running in Producer mode, the Connector will replicate events from EventStoreDB to Elasticsearch using data streams. Documents in the data stream are immutable, which is a good choice for storing events.
Based on the configuration, the connector will create the following elements in Elasticsearch:
- Index template
- Data stream
- Index rollover policy
You can optimize the rollover policy to keep the index size optimal, as well as move older events to a cheaper storage tier.
Events are replicated to Elasticsearch in the following format:
messageId- the unique identifier of the event
messageType- the type of the event
streamPosition- event position in the original stream
stream- original stream name
globalPosition- position of the event in the global stream (
message- the event payload
metadata- flattened event metadata
@timestamp- the timestamp of the event
There’s no need for a sidecar to run the Connector in Producer mode.