Robin Moffatt • 4/5/2023

Using Delta from pySpark - `java.lang.ClassNotFoundException: delta.DefaultSource`

This technical blog post details a solution for a common PySpark and Delta Lake integration error. The author explains that creating both a SparkSession and a SparkContext incorrectly can cause the 'java.lang.ClassNotFoundException: Failed to find data source: delta' error. The article provides working and non-working code examples for initializing Spark with Delta Lake configurations in a Jupyter Notebook.

0 comments

#Pyspark #Jupyter #Delta Lake