No suitable FileSourceFactory found (jet)

[agathis the great]

Hi all!

I'm trying to read parquet files with jet on hazelcast 5.1.2

But whatever I do, it ends up the same: Exception in thread "main" com.hazelcast.jet.JetException: No suitable FileSourceFactory found. Do you have Jet's Hadoop module on classpath?

What am I doing wrong? I've tried adding hazelcast-jet-hadoop-all to classpath in addition to having it in my build dependencies but it doesn't seem to affect the result in any way.

My code (kotlin):

import com.hazelcast.core.Hazelcast
import com.hazelcast.jet.pipeline.*
import com.hazelcast.jet.pipeline.file.*

private val hcClient = Hazelcast.bootstrappedInstance()

fun main() {
val pipeline: Pipeline = Pipeline.create()

val source = FileSources.files("/path/to/parquets")
.useHadoopForLocalFiles(true)
.format(FileFormat.parquet<Map<String, Any>>())
.build()

pipeline.readFrom(source).writeTo(Sinks.logger())

hcClient.jet.newJob(pipeline);
}

gradle dependencies:

implementation("com.hazelcast:hazelcast:5.1.2")
implementation("com.hazelcast.jet:hazelcast-jet-files-azure:5.1.2")
implementation("com.hazelcast.jet:hazelcast-jet-hadoop-all:5.1.2")

Thanks!

[Josef Cacek]

Hi Agathis,
you don't need to specify both - jet:hazelcast-jet-files-azure and
jet:hazelcast-jet-hadoop-all. One of them is enough to make it work.

How do you run your application? Do you run it using a Gradle task
(e.g. the application plugin)? Or do you create a fat JAR and execute
it?

-- Josef

> --
> You received this message because you are subscribed to the Google Groups "Hazelcast" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/b7d9a8d8-2531-4170-811a-67f04b998735n%40googlegroups.com.

--
This message contains confidential information and is intended only for the
individuals named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. E-mail transmission cannot be
guaranteed to be secure or error-free as information could be intercepted,
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message, which arise as a result of e-mail
transmission. If verification is required, please request a hard-copy
version. -Hazelcast