问题描述
您好,我正在尝试下载以下Build.sbt文件中的spark-core
、spark-streaming
、twitter4j
和spark-streaming-twitter
:
name := "hello"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"
libraryDependencies ++= Seq(
"org.twitter4j" % "twitter4j-core" % "3.0.3",
"org.twitter4j" % "twitter4j-stream" % "3.0.3"
)
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"
我只是把这个libraryDependencies
放在网上,所以我不确定要使用哪个版本等。
有人能向我解释一下我应该如何修复这个.sbt文件吗?我花了几个小时试图弄清楚,但所有的建议都没有奏效。我通过自制软件安装了scala
,我的版本是2.11.8
我所有的错误都是关于:
Modules were resolved with conflicting cross-version suffixes.
推荐答案
问题是您混合了Scala2.11和2.10构件。您拥有:
scalaVersion := "2.11.8"
然后:
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"
其中需要2.10
构件。您还混合了Spark版本,而不是使用一致的版本:
// spark 1.6.1
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
// spark 1.4.1
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"
// spark 0.9.0-incubating
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"
以下是解决这两个问题的build.sbt
:
name := "hello"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)
您也不需要手动添加twitter4j
依赖项,因为它们是由spark-streaming-twitter
传递添加的。