-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot perform join between points and polygon using Scala 2.11 and Spark 2.3.1 #230
Comments
Similar error with same versions in test code:
This is the trace:
|
@djpirra and @lmerchante |
It does not work with full compatibility. I have compiled from master and
was able to read the points and polygons, but it didn’t worked when I try
to intersect them...
Something is still not working right.
Em qua, 19 de set de 2018 às 17:29, Diego Mora Cespedes <
[email protected]> escreveu:
@djpirra <https://github.com/djpirra> and @lmerchante
<https://github.com/lmerchante> Magellan 1.0.5 is not compatible with Spark
2.3.1. You have to wait until the next release or compile from source,
since the master branch is already compatible with Spark 2.3.1. I tested it
last week and it works just fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#230 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/APPYdgpqxlv5NItwjAlGo_OwHA873MSVks5uckbBgaJpZM4WXHgj>
.
--
Melhores Cumprimentos,
Luis Simões
|
I have the same problem as @djpirra. |
@djpirra I see a Also, don't forget to run |
The same for me compiling the master branch and using it on Azure Databricks. |
Is this because of the way we compile the master or is there something in our notebooks that prevents the success? (Zeppelin 0.8.0) Update: I just tested the with master branch and Spark 2.3.2 shell and the moment I get to
I can see this |
I was able to get this working in a Databricks notebook environment using the 5.0 Runtime (Spark 2.4.0, Scala 2.11). You will need to change the sparkVersion in build.sbt to 2.4.0 from 2.3.1 |
I get the similar error. any resolutions please? Stoping Spark Session Explicitly |
Currently trying to join to dataframes with the following command:
val df_green_pickup = green_data.join(neighborhoods).where($"pickup_point" within $"polygon") display(df_green_pickup)
Having the following exception:
SparkException: Job aborted due to stage failure: Task 0 in stage 44.0 failed 4 times, most recent failure: Lost task 0.3 in stage 44.0 (TID 875, 10.139.64.11, executor 10): java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.codegen.ExprCode.value()Ljava/lang/String; at org.apache.spark.sql.catalyst.expressions.Within$$anonfun$doGenCode$2.apply(predicates.scala:202) at org.apache.spark.sql.catalyst.expressions.Within$$anonfun$doGenCode$2.apply(predicates.scala:180) at org.apache.spark.sql.catalyst.expressions.BinaryExpression.nullSafeCodeGen(Expression.scala:553) at org.apache.spark.sql.catalyst.expressions.Within.doGenCode(predicates.scala:180) at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:111) at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:108) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:108) at org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate$.create(GeneratePredicate.scala:60) at org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate$.generate(GeneratePredicate.scala:46) at org.apache.spark.sql.execution.SparkPlan.newPredicate(SparkPlan.scala:382) at org.apache.spark.sql.execution.joins.CartesianProductExec$$anonfun$doExecute$1.apply(CartesianProductExec.scala:84) at org.apache.spark.sql.execution.joins.CartesianProductExec$$anonfun$doExecute$1.apply(CartesianProductExec.scala:81) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndexInternal$1$$anonfun$apply$24.apply(RDD.scala:830) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndexInternal$1$$anonfun$apply$24.apply(RDD.scala:830) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:42) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:336) at org.apache.spark.rdd.RDD.iterator(RDD.scala:300) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:112) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:384) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Did anyone tried this on the same versions?
Thank you
The text was updated successfully, but these errors were encountered: