Version
You can check the version of Spark you are using by running the following command in the terminal:
spark-shell --version
About Apache Spark
Apache Spark is a framework that, like with all of Apache endorsed projects, is open-source, which means that it has
an extremely active
community of users and developers that need not be direct employees of Spark. Nevertheless, Spark was originally
developed by the prestigious
University UCBerkeley and was then donated to the Apache foundation. Spark features multiple modules that can be
useful to a lot of developers
from vastly different backgrounds, mainly in the areas of data science and analysis.
One of the main functionalities of Spark is the ease of development that it gives to Java and Kotlin developers.
The extremely expressive and simple structure and syntax of the framework give it the power to be very helpful for
programmers to develop
complex applications and keep the backend as simple as it can be. Spark has immense power when developing machine
learning applications
because of the native module Spark MLlib, which allows complex processes to go smoothly and be at the disposition of
the programmer
without having to go through the trouble of coding it from scratch or installing and integrating an external module.
Spark also has applications in data analysis and visualizations with the native modules that are installed in the
framework.
Some of the main features of Spark are crucial to understanding. For example, Spark is entirely modular,
meaning that you only need to use the modules and the cores that you require to develop your ideas. It is developed
to solve many
of the issues with the Java Enterprise framework, which many people come from. This can help new developers come up
with more efficient solutions,
and also for senior developers that come from Java EE to migrate and optimize their existing applications.
Releases
Spark was launched as we know it on the 30th of May of 2014, but there are some early development releases like
version 0.5,
which was made available to the public as early as 2012. In fact, the origins of Spark trace back to development in
UCBerkeley as early as 2009,
where the idea began forming for this tool. After quite some time and a lot of developer efforts, Spark is currently
on the 3.1.1 stable release,
made available to everyone in March of 2021. Spark has seen so much popularity that an online teaching and course
community is forming around the tool,
pointing to future widespread usage and more updates to be expected from the developer team.