Streaming XML messages from IBM MQ into Kafka into MongoDB
A technical guide on streaming XML data from IBM MQ to Kafka, transforming it, and then loading it into MongoDB using Kafka Connect.
Robin Moffatt is a Principal DevEx Engineer and seasoned conference speaker with 15+ years of experience presenting at top events like QCon, Devoxx, Kafka Summit, and Strata. He shares insights on developer experience, distributed systems, and cloud technologies through his blog, YouTube, and public talks.
625 articles from this blog
A technical guide on streaming XML data from IBM MQ to Kafka, transforming it, and then loading it into MongoDB using Kafka Connect.
Explores using the Kafka Connect FilePulse connector to ingest and process XML data into Apache Kafka, including configuration and troubleshooting.
A technical guide on using Kafka Connect with Single Message Transform to ingest and process XML data into Apache Kafka.
A technical guide on using Unix pipelines with curl, xq, and kafkacat to ingest and transform XML data into JSON for Kafka.
Introduction to ingesting XML data into Kafka, covering challenges and approaches for handling XML schemas in Kafka topics.
Fixing abcde CD ripping error due to permissions by adding user to cdrom group or using chmod.
Troubleshooting guide for resolving 'Channel was blocked' errors when connecting a client to IBM MQ running in a Docker container.
A technical guide on setting Kafka message keys when piping JSON data from jq to kafkacat, using a separator character.
A developer shares a list of UK-centric public and open data sources for building realistic applications and demos.
A curated collection of Apache Kafka and ksqlDB talks, including recordings, slides, and code examples for building streaming data pipelines.
A technical guide on setting up ksqlDB with embedded Kafka Connect to ingest CDC events from MS SQL Server using the Debezium connector via Docker Compose.
A technical guide on using Asciidoc's include directive within Hugo to embed external code snippets from sources like GitHub.
An overview of Kafka Connect, Apache Kafka's integration API, covering its features, resources, and use cases for data pipelines.
Explores methods to count messages in a Kafka topic, comparing kafkacat commands with wc to ksqlDB for more detailed, grouped aggregations.
A technical guide on managing and cleaning up custom search engines in Google Chrome by directly manipulating its SQLite database.
Explains the BOT_COMMAND_INVALID error in Telegram bots, showing that commands must be lowercase when using the setMyCommands API.
A technical guide on processing chunked HTTP responses in Go to handle streaming JSON data from a database without waiting for EOF.
A developer shares their journey learning Go by building a utility to check Apache Kafka's advertised.listeners configuration for common issues.
A guide to organizing Go code by splitting functions into separate source files and building a standalone binary executable.
A developer's notes on learning Go by building a Kafka utility, focusing on error handling, command-line arguments, and goroutines.