PinnedPublished inData Engineer ThingsChange Data Capture (CDC): Comprehensive Guide- PostgreSQL To S3(MinIO) Using NiFiThis article delves into leveraging CDC for PostgreSQL using Debezium, Kafka and NiFi and subsequently storing the processed data in MinIONov 25, 2024Nov 25, 2024
PinnedPublished inTowards DevKafka + NiFi + Clickhouse in Docker PART-IIn this article, we will process real-time car entry/exit data from 5 different parking lots using big data tools and Apache NiFi, and we…Mar 20, 20241Mar 20, 20241
Published inData Engineer ThingsE2E Build Data Lakehouse with Reddit API & Iceberg: Part 2In the previous section, we discussed cleaning the data and loading it into the OpenSearch & MinIO.Nov 6, 20241Nov 6, 20241
Published inData Engineer ThingsE2E Data Lakehouse Using Reddit API & Airflow & Kafka & NiFi & OpenSearch & Minio & Iceberg &…This article will use the Reddit API to examine posts made to desired topics in real time using NIFI, AIRFLOW, MINIO, OpenSearch and SPARK.Nov 6, 2024Nov 6, 2024
Published inTowards DevWhat’s Branch & Tag in Apache IcebergHello, in this article I will touch upon the most important features of apache iceberg, namely branch, snapshot and tag.Oct 23, 2024Oct 23, 2024
Published inTowards Data EngineeringCreate Data Lakehouse Using Spark+Iceberg+Nessie+DremioHi everyone, in this article I will talk about the concept of data lakehouse and develop example using Spark+Iceberg+Nessie+Dremio+MinioSep 19, 2024Sep 19, 2024
PX4 Drone Flight Log Analysis & Data Pipeline PART-IIIn this second part, let’s continue with MINIO + NIFI => OPENSEARCHAug 23, 2024Aug 23, 2024
Published inTowards DevPX4 Drone Flight Log Analysis & Data Pipeline PART-IDrone flight log data engineering pipeline using Apache Nifi+MinIO+Opensearch+Apache AirflowAug 22, 20241Aug 22, 20241
Published inTowards DevHow to Monitoring Apache Nifi ?Performance monitoring is very important for IT. You need to get information about the status of the services or servers you have. If you…Aug 3, 2024Aug 3, 2024
Published inTowards DevKafka + NiFi + Clickhouse in Docker PART-IIIn the first part of this article, I touched on the following topics:Mar 31, 20242Mar 31, 20242
