PX4 Drone Flight Log Analysis & Data Pipeline PART-II
In this second part, let’s continue with:
- Creating flowfiles in NiFi + Opensearch
- Monitoring System using Grafana + Prometheus
⚡Create Flowfile in MINIO + NIFI => OPENSEARCH
At this stage, we will do a few simple operations. In this process;
- Pull the csv names in the minio buckets
- Pull the data with csv names from minio buckets
- Convert the CSV to JSON
- Add timestamp to JSON
- Write json to Opensearch index
1. Pull the CSV Names
We need access key and secret key to retrieve the data in the bucket. You must get these from the minio interface.
Place the access key and secret keys in the relevant fields and add the Endpoint Override URL minio server ip and port
We received the document names in the S3 bucket, but we also received useless information. We need to get the “key”,”bucket” fields.
Then we convert the document names to json format. We parse the bucket key, index etc. names separately and add them to the flowfile as attribute.
2. Pull the Data From Minio Bucket
This stage is very simple: FetchS3Object processor fetches the data for us. All we need to do is bring JSON format.
3. Add Timestamp
This stage includes adding and editing the current timestamp json file one by one. I will share the flowfile file at the end of the article for you to review.
4. Write JSON to Opensearch Index
At this stage we will only need to set the PutElasticsearchHttpProcessor.
The important point is to set the Index file as the index we added at the beginning of the flowfile. If you do not set this, the data in the drone will be added in a complex manner, but by adding this we are grouping it according to the UROB message type.
Everything is ok, now let’s look at opensearch.
⚡Configure Opensearch
For those who haven’t used it before, the home screen will look like this.
On the Nifi side, indexes will be created automatically, so there is no need to create an index and data types here, all we need to do is to create an index pattern to see the data.
Crete Index pattern and select index name.
For example i added sensor baro urob message index.
Go Discover in left side bar.
Select any index name.
Create Dashboard
Go Dashboard in left side bar.
Add chart in dashboard. I leave this to your imagination.
⚡Configure Monitoring
I have written an article on this subject before, so I will not repeat it, those who want can review it from the link below.
Source Code
If you encounter any problems, you can contact me to ask questions.
Feel free to share your comments and ideas! Are you using MINIO + NIFI+OPENSEARCH? What are your thoughts?
