I'm new to big data processing and I'm reading about tools for stream processing and building data pipelines. I found Apache Spark and Spring Cloud Data Flow. I want to know the main differences and the pros and cons of them. Could anybody help me?
I'm new to big data processing and I'm reading about tools for stream processing and building data pipelines. I found Apache Spark and Spring Cloud Data Flow. I want to know the main differences and the pros and cons of them. Could anybody help me?
They are 2 completely different tools.
Spring Data Flow is a toolkit for building data integration and real-time data processing pipelines. This tool will help you to orchestrate data pipelines using Spring Boot Apps (Stream or Task). Under the hood, SCDF might use Spring Batch. Note this Spring Boot Apps can call Spark or Kafka applications to support Stream processing.
Apache Spark is an engine for data processing, it is being highly used for data intensive processing and data science. It has libraries such as ML (Machine Learning), Graph (graph processing), integration with Apache Kafka (Spark Streaming), among others.
For streaming, I highly recommend you to study Apache Kafka.
As mentioned on the https://dataflow.spring.io/docs/concepts/architecture/#comparison-to-other-architectures
Comparison to Other Architectures
Spring Cloud Data Flow’s architectural style is different than other Stream and Batch processing platforms. For example in Apache Spark, Apache Flink, and Google Cloud Dataflow, applications run on a dedicated compute engine cluster. The nature of the compute engine gives these platforms a richer environment for performing complex calculations on the data as compared to Spring Cloud Data Flow, but it introduces the complexity of another execution environment that is often not needed when creating data-centric applications. That does not mean that you cannot do real-time data computations when you use Spring Cloud Data Flow. For example, you can develop applications that use the Kafka Streams API that time-sliding-window and moving-average functionality as well as joins of the incoming messages against sets of reference data.
"The solution is better than average and some of the valuable features include efficiency and stability."
"The solution is very stable and reliable."
"The most valuable feature is real-time streaming."
"There are a lot of options in Spring Cloud. It's flexible in terms of how we can use it. It's a full infrastructure."
"There could be an improvement in the area of the user configuration section, it should be less developer-focused and more business user-focused."
"The solution itself could be easier to use."
"Some of the features, like the monitoring tools, are not very mature and are still evolving."
"The configurations could be better. Some configurations are a little bit time-consuming in terms of trying to understand using the Spring Cloud documentation."
