Project Name

Data Collection Using SNMP Protocol with Apache NiFi

Industry

Telecommunication

Technology

Apache NiFi, Java, SNMP, Cassandra, ZooKeeper

Overview

Our client was a leading service provider in the telecommunications industry who works on managing the data and entire data collection. They were facing issues in managing a massive number of modems on the production system and needed a supportive system to rescan the single and multiple modems.

Challenges

The main challenge is that there were millions of modems on the production system for which the client needs data to be collected at least 2 times a day.
The client was searching for a support system that helps them with on-demand rescans on single and multiple modems.
Issue in collecting the data for the optimal usage of the primary memory on the servers.

Our Solution

We have provided a robust solution to our client that includes:

By leveraging Apache NiFi, our team created multiple custom processors for handling data collection from several types of devices including CMTS or modems and for moving the data directly through the pipeline.
Through Apache NiFi, it becomes possible for us to keep the usage of primary memory below the critical threshold as the NiFi stores the data on the secondary storage and loads it onto the memory once it is needed by the pipeline.
Utilizing the Apache Cassandra, our team gained the ability to save the data for millions of modems with quick read access, fault tolerance, and high data availability.
Even, we had used the snmp4j library with the custom processors for direct communication with the modems and CMTS devices for data polling.
Our team used Zookeeper with custom changes for handling the Apache NiFi cluster that contains multiple NiFi nodes and works on Job orchestration, inventory management, and load balancing.

Data Flow Diagram

Conclusion

Hence, we had successfully implemented Apache NiFi with SNMP usage. It becomes possible for us to collect the data for millions of modem devices multiple times during the day while also supporting on-demand rescans and simultaneously doing transformation and processing on polled data by implementing custom processors in NiFi while offloading the data into Cassandra tables for quick read access and high data availability for the users.