PinnedPublished inData Engineer ThingsChange Data Capture (CDC): Comprehensive Guide- PostgreSQL To S3(MinIO) Using NiFiThis article delves into leveraging CDC for PostgreSQL using Debezium, Kafka and NiFi and subsequently storing the processed data in MinIONov 25, 20241Nov 25, 20241
PinnedPublished inTowards DevKafka + NiFi + Clickhouse in Docker PART-IIn this article, we will process real-time car entry/exit data from 5 different parking lots using big data tools and Apache NiFi, and we…Mar 20, 20241Mar 20, 20241
Published inTowards Data EngineeringHow To Convert Traditional Database To Lakehouse?You can easily migrate to Lakehouse from a classical database with Kafka Connect and transfer the relational databases you want to…Feb 24Feb 24
Published inData Engineer ThingsThe Simplest Way to Implement CDC from PostgreSQL to S3 (MinIO)In this article, CDC for PostgreSQL database using Kafa Connect. Subsequently storing the processed data in MinIOFeb 7Feb 7
Published inData Engineer ThingsE2E Build Data Lakehouse with Reddit API & Iceberg: Part 2In the previous section, we discussed cleaning the data and loading it into the OpenSearch & MinIO.Nov 6, 20241Nov 6, 20241
Published inData Engineer ThingsE2E Data Lakehouse Using Reddit API & Airflow & Kafka & NiFi & OpenSearch & Minio & Iceberg &…This article will use the Reddit API to examine posts made to desired topics in real time using NIFI, AIRFLOW, MINIO, OpenSearch and SPARK.Nov 6, 2024Nov 6, 2024
Published inTowards DevWhat’s Branch & Tag in Apache IcebergHello, in this article I will touch upon the most important features of apache iceberg, namely branch, snapshot and tag.Oct 23, 2024Oct 23, 2024
Published inTowards Data EngineeringCreate Data Lakehouse Using Spark+Iceberg+Nessie+DremioHi everyone, in this article I will talk about the concept of data lakehouse and develop example using Spark+Iceberg+Nessie+Dremio+MinioSep 19, 20241Sep 19, 20241
PX4 Drone Flight Log Analysis & Data Pipeline PART-IIIn this second part, let’s continue with MINIO + NIFI => OPENSEARCHAug 23, 2024Aug 23, 2024
Published inTowards DevPX4 Drone Flight Log Analysis & Data Pipeline PART-IDrone flight log data engineering pipeline using Apache Nifi+MinIO+Opensearch+Apache AirflowAug 22, 20241Aug 22, 20241
