Building an ETL Application for LDAP Migration Using Kafka, KsqlDB and Spring Boot
Recently I was working on a project that required me to migrate data from one LDAP server to a different one. This migration included some data cleaning and transformation so I looked for off-the-shelf ETL (Extract, Transform, Load) solutions but couldn’t find any that fit this specific use case.
The source LDAP system is a proprietary system with limited export options, so I had to work with an LDIF file as the source.
Part of the requirement was to have thorough monitoring of the migration process, visibility into the migration status, and the ability to retry failed migrations.
Instead of building a custom ETL solution from scratch to fulfill these requirements, I decided to use Apache Kafka as a base for the ETL flow and Stream processing as a base for monitoring. On top of that I used Spring Boot with Kotlin to implement the ETL components.
An overview of the solution architecture is shown below:
cn=1,dc=example,dc=com
cn=2,dc=example,dc=com
cn=3,dc=example,dc=com
....."]] Extractor["Extractor
- read entry from ldif
- produce message to
kafka topic 'users'"] Kafka(("Kafka Broker
topic: users")) Transformer["Transformer
- transform fields
(e.g. first_name+last_name -> name)"] Writer["Writer
- write user to destination LDAP"] LDAP(("LDAP")) KsqlDB(("ksqlDB Monitoring
# users / # migrated / # failed")) Extractor -- read --> Source Extractor -- "produce
migration_status: read" --> Kafka Kafka -- poll --> Transformer Transformer -- "produce
migration_status: transformed|transformation_failed" --> Kafka Kafka -- poll --> Loader Loader -- "produce
migration_status: loaded|loading_failed" --> Kafka Loader -- write --> LDAP Kafka -- monitor --> KsqlDB
Kafka topics, message format and partitions
The idea is to have one topic for the domain entity (e.g. users), and each processor (Extractor, Transformer, Writer) would produce messages to this topic with a migration_status field to indicate the current status of the migration. The key of the message should be the unique identifier of the entity (e.g. cn for users) to ensure that messages related to the same entity are processed in order.
Having a single topic with different migration statuses makes monitoring easier and allows us to trace the migration status of each entity.
Infrastructre for the tool
Since it based on Kafka, we need to run a Kafka broker and a ksqlDB server to monitor the migration status.
A docker compose file with the following services should achieve this:
- Kafka
broker - ksqlDB server
ksqldb-server - An extra
ksqldb-cliservice to interact with the ksqlDB server - An optional Kafka UI service to visualize the messages Kafbat UI
kafbat-ui, which is an open source lightweight web UI for managing Kafka clusters.
The tool is then organized into three main components: extract, transform, and load. Each component is a separate Spring Boot application that can be run independently, with a main Spring Boot application that launches each of them based on the --mode argument.
There’s one image for all three components, it can be optimized later to have separate images for each component.
Running the tool
Starting the infrastructure services can be done using a simple:
docker compose up -d
1. Running the extractor:
docker run \
-v /path/to/your/ldif/file.ldif:/app/input.ldif \
--network etl-template_default \
-e KAFKA_BROKER_URL=broker:29092 \
-e LDIF_FILE_PATH=file:/app/input.ldif \
-e JAVA_TOOL_OPTIONS="-Xms2g -Xmx4g"
--rm registry.gitlab.com/bigsolom/etl-template:latest --mode=extract
2. Running the transformer:
docker run \
--network etl-template_default \
-e KAFKA_BROKER_URL=broker:29092 \
--rm registry.gitlab.com/bigsolom/etl-template:latest --mode=transform
3. Running the loader:
One caveat is that the loader needs to run on the host network to connect to the LDAP server which is supposed to be accessible from the host machine where the container is running. If we skipped this the loader will try to connect to an LDAP server in etl-template_default network which doesn’t exist.
docker run \
--network host \
-e LDAP_HOST=localhost \
-e LDAP_PORT=389 \
-e LDAP_ADMIN_DN="cn=admin,dc=example,dc=com" \
-e LDAP_ADMIN_PASSWORD=passwd \
-rm registry.gitlab.com/bigsolom/etl-template:latest --mode=load
Alternatives
One of the strongest alternatives would be Apache airflow which is an overkill for my use case for the following reasons
- Heavyweight setup they have a docker compose version for running it: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
- Need to clean up sample data otherwise the interface will be confusing for the client
- Doing a clean run is not trivial, since every time we will need to run the init commands again, compared to the kafka approach wher there’s no init step and starting from a clean state just requires
docker compose down --volumes
Thanks for reading! Let me know if you have had a similar use case and what approach you took, or if you have any feedback, suggestions or corrections.