Stream processing with Apache Flink and Minio

What is Stream processing?

Stream processing enables analyzing continuous data streams. In this approach, data is seen as a continuous stream that processing engines ingest, analyze and return the response within a small time frame — few milliseconds to minutes.

The response time is generally based on the use-case and criticality of response time. For example, you’d expect IoT sensor data from a nuclear reactor to be processed in a much smaller time frame as compared to data from a user’s website visit.

There are several situations where the streaming approach to data analysis is better suited when compared to the batch analysis:

• With modern technologies (IoT, transaction logs, application logs, activity logs, visit logs) generating continuous data streams, processing that data in a similar, continuous manner is the natural approach.
• Batch processing takes a bigger chunk of data and processes them at once while stream processing takes data as they come in, hence spreading the processing over time. This allows stream processing work with fewercompute resources compared to batch processing.
• Sometimes data is too huge and it is not economically sensible to store it all. Stream processing let you handle large fire-hose style data and retainonly useful bits.
• Streaming allows detecting patterns, inspect results, and also easily look at data from multiple streams simultaneously. This means you get approximate results in a shorter time frame. In contrast, with batch processing, you need to process multiple batches and aggregate results across these batches to get better results, but it takes longer.

Stream processing use-cases

Apache Flink

Apache Flink is a distributed processing engine for stateful computations over data streams. Flink excels at processing unbounded and bounded data sets.

Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.

While Apache Spark is well know to provide Stream processing support as one of its features, stream processing is an after thought in Spark and under the hoods Spark is known to use mini-batches to emulate stream processing.

Apache Flink on the other hand has been designed ground up as a stream processing engine. This means Flink

• Does better memory management and avoids occasional spikes in memory usage.
• Manages faster speeds by allowing iterative processing to take place on the same node rather than having the cluster run them independently.

Minio with Apache Flink

Apache Flink supports three different data targets in its typical processing flow — data source, sink and checkpoint target. While data source and sink are fairly obvious, checkpoint target is used to persist states at certain intervals, during processing, to guard against data loss and recover consistently from a failure of nodes.

With AWS S3 API support a first class citizen in Apache Flink, all the three data targets can be configured to work with any AWS S3 API compatible object store, including ofcourse, Minio.

Minio can be configured with Flink in four broad ways, let’s take a look at all four below:

1. Minio event notifications: Minio event logs can be sent to Flink via Kafka as event streams. Such event data is beneficial in cases where object access logs are important for business to understand certain user behaviour trends or data access trends.

• Remote object storage target like Minio de-couples state from Flink’s compute nodes. This means Flink becomes stateless i.e. free to grow and shrink as and when needed (saving cost) while state is safely stored on Minio.
• Minio performance (upto 10 GBps per node) ensures that even if the state is de-coupled, it is readily available and adds no latency to Flink processing.
• With configurable erasure coding, scaleable design, server side encryption, Minio ensures safe, scaleable and reliable storage of data in cost efficient manner.
• Native AWS S3 API support in Flink means out of the box integration and support for Minio, reducing configuration and maintenance costs.

Configure Minio with Flink

Let us now take a look at how to configure Apache Flink with Minio as the remote storage backend. In this example, we’ll use Minio as both the source and sink.

To start with, you’ll need Minio server deployed, refer this document for details. Next, download Flink binary as explained in the quick start document.

Then update $FLINK_DIR/conf/flink-conf.yaml and add the below sections:

state.backend: filesystem
s3.endpoint: http://127.0.0.1:9000
s3.path-style: true
s3.access-key: minio
s3.secret-key: minio123

$FLINK_DIR here is the directory where you untarred Flink tar file. Also, don’t forget to update the s3. fields based on actuals from your Minio server deployment.

Now, start Flink. The setup is now ready to use Minio as the default storage system. To test this I used the WordCount example from Flink documentation

./bin/flink run examples/batch/WordCount.jar — input s3://input/test.txt — output s3://testbucket/output

Here test.txt is a sample text file (use any file with lots of text data). Once the job finishes, you can see the word count in the testbucket/output file.

Conclusion