Thursday, November 3, 2016

Initiating a SparkContext throws javax.servlet.FilterRegistration SecurityException


If you are trying to load hive-jdbc, hadoop-client and jetty all in the same Scala project along with your Spark dependencies, you might not be able to load a standalone Spark application. 
While trying to initiate the SparkContext, it will throw a javax.servlet.FilterRegistration SecurityException because of mixed javax.servlet dependencies imported with different versions from several sources. 

How to avoid this conflict?
You will need to add several ExclusionRules to some of your dependencies located at the build.sbt file:

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.6.4" 
excludeAll(ExclusionRule(organization = "javax.servlet"), 
ExclusionRule(organization = "org.mortbay.jetty"))

libraryDependencies += "org.apache.hive" % "hive-jdbc" % "1.2.1" 
excludeAll ExclusionRule(organization = "javax.servlet")

Good luck

No comments: