Data Ingestion Using NiFi: Failure And Recovery

Apache NiFi needs no introduction. It is a powerful, easy to use and reliable system to distribute data between systems is named after Niagara files technology developed by NASA and later offered to Apache software solutions. NiFi is a data ingestion platform that can transfer data between different systems. However, data ingestion using NiFi comes with different complexities and errors. 

In this article we will discuss the failures that can occur during data ingestion and how to recover them. 

NiFi data ingestion and clustering

Data ingestion is the process of collecting data from any data source and can be performed in two ways- real time streaming and Batch streaming. Since NiFi supports both single and multi-node deployment and has better error handling data ingestion using NiFi is the ideal choice. 

Although NiFi can be deployed in a single mode deployment it is always advised to prefer NiFi multi-node deployment and that is why NiFi cluster administration is crucial as a single instance of NiFi is not enough to handle large  volumes of data. NiFi works on Zero-master clustering so that even if the master nodes gets lost, we can always make changes in any node. We can also deploy metrics for NiFi cluster monitoring for better performance.

NiFi failure and recovery during the data ingestion

When we talk about NiFi, we must talk about the coolest feature it has, the ability for error handling. Here we will discuss how NiFi failure and recovery can make your flow development easier.

NiFi has different relations for each failed processor. While publishing any event to kafka, we auto-terminate the success relation since the event is in the destination data store. Failure relations require more work. There can be several errors like- destination not reachable, event data not compliant with scheme, etc. while some of these errore resolve by themselves, however many of them require proper handling. 

• NiFi supports reflexive loops to connect failure relation of the processor to incoming queue, but if the issue is permanent the event will have to go through infinite loops.
• A better idea is to try to publish an event several times and send it to DLQ if failure persists. 
• It uses UpdateAttribute and RouteOnAttribute to recover the failure.

This seems simple right, but doing this at scale can increase the complexity of the process.

Here comes at your rescue- NiFi 1.10

NiFi 1.10 has a new processor called RetryFlowFile to ease NiFi failure and recovery. The strategy is the same, the catch is, it has replaced UpdateAttribute and RouteOnAttribute and made the configuration more easier. This processor can penalize flowfile to make it wait before trying again. This will result in seconds to resolve issues. The penalty duration can be set to 30 seconds by default. 

How can Ksolves help you in NiFi failure and recovery during data ingestion?

NiFi comes with many features that makes error handling easier and cleaner. Hope we have covered how NiFi failure and recovery features can help you with smooth data ingestion without any errors and complexities. Ksolves is your only stop for all your Big data problems. We have a team of 350+ developers specialized in delivering tailor made solutions. And that is why, we have a plethora of happy clients all around the world. If you wish to learn more about Apache NiFi or need a development partner, write to us in the comments below or give us a call. 

Follow us for more informative articles on big data.