#Spark for mac troubleshoot series

The methods people usually use to try to solve them often derive from that person’s role on the data team.

#Spark for mac troubleshoot series

The problems we mentioned in Part 1 of this series have many potential solutions. In Part 3, the final piece, we’ll introduce Unravel Data, which makes solving many of these problems easier. We’ll show what they do well, and where they fall short. In this guide, Part 2 in a series, we’ll show ten major tools that people use for Spark troubleshooting. (You also need the holistic view when you’re creating the Spark job, and as a check before you start running it, to help you avoid having problems in the first place. But that’s just what you need to truly solve problems. These tools don’t present a holistic view. Widely used tools generally focus on part of the environment – the Spark job, infrastructure, the network layer, etc. But Spark jobs are very important to the success of the business when a job crashes, or runs slowly, or contributes to a big increase in the bill from your cloud provider, you have no choice but to fix the problem. Problems in running a Spark job can be the result of problems with the infrastructure Spark is running on, an inappropriate configuration of Spark, Spark issues, the currently running Spark job, other Spark jobs running at the same time – or interactions among these layers. Spark is known for being extremely difficult to debug. Note: This guide applies to running Spark jobs on any platform, including Cloudera platforms cloud vendor-specific platforms – Amazon EMR, Microsoft HDInsight, Microsoft Synapse, Google DataProc Databricks, which is on all three major public cloud providers and Apache Spark on Kubernetes, which runs on nearly all platforms, including on-premises and cloud.