4

I am new in Scala. So I want to process the DataSet of Apache Flink in Java. But I have a DataSet in Scala code that I want to pass in a methods of Java Class. For example: In Scala:

val dS: DataSet[Vector] = ...
SampleJavaClass.sendDS(ds)

In SampleJavaClass.java there is a method like sendDs(DataSet<Vector> javaDs)

I am trying to do this but show error message: type mismatch; found :

org.apache.flink.api.scala.DataSet[org.apache.flink.ml.math.Vector] required: org.apache.flink.api.java.ExecutionEnvironment Error occurred in an application involving default arguments.

How can I solve this problems. Please help me on this.

Improve this question

2 Answers 2

Reset to default
3

The Scala DataSet is indeed a wrapper around the Java DataSet. However, since the method javaSet is package private, you cannot access it. Thus, there is currently only a very hacky way to convert a Scala DataSet to a Java DataSet.

In order to access the javaSet method, you have to define your package to be a sub-package of org.apache.flink. This is, admittedly, not best practice, though.

If this is a feature which is strictly required, you should file a JIRA ticket for it.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I have one package of scala files: (package org.apache.flink.quickstart). Do I need to create (package org.apache.flink.quickstart.java) for the java files? Then I can send scala DS to Java DS?
The file in which you call javaSet has to be in a org.apache.flink package. Thus, you could write yourself a helper function which lives in a org.apache.flink package and simply calls the javaSet method for a given DataSet.
But without function, it is not working. Why? what is the hack here. Could you please explain if possible?
What do you mean with without function it is not working? The problem is that the DataSet.javaSet method is declared as private[flink] which means that it can only be accessed by code which resides in a org.apache.flink or sub package. Usually you should define your own package names to avoid name conflicts, but here it's the only way to access the javaSet function.
1

There are two DataSet classes in Flink a java one https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/api/scala/DataSet.html and a scala one. You can access the java one from the underlying scala DataSet via javaSet().

I would try to 

SampleJavaClass.sendDS(ds.javaSet())

that might appease your code.

Improve this answer

1 Comment

I have tried this approach. But It is showing the above error I mentioned. (Error occurred in an application involving default arguments.) What does it mean?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.