Using Spark related libraries within Beam pipeline
8 views
Skip to first unread message
jona...@simility.com
Nov 27, 2018, 9:13:15 AM11/27/18
to BEAM Community
Hey folks,
Is it possible to use Spark libraries such as Spark.ml within a Beam pipeline?
From my understanding, you will write your pipeline in "beam syntax" and let Beam
execute it on spark using spark as a runner.
Hence, I don't see how you could use spark.ml within beam.
But maybe I get something wrong here?
Did someone already try to use it, if not, do other ML libraries exist for native usage in Beam (except from Tensorflow Transform)?
Many Thanks,
Jonathan
Is it possible to use Spark libraries such as Spark.ml within a Beam pipeline?
From my understanding, you will write your pipeline in "beam syntax" and let Beam
execute it on spark using spark as a runner.
Hence, I don't see how you could use spark.ml within beam.
But maybe I get something wrong here?
Did someone already try to use it, if not, do other ML libraries exist for native usage in Beam (except from Tensorflow Transform)?
Many Thanks,
Jonathan
José Valim
Nov 27, 2018, 9:26:02 AM11/27/18
to beam-co...@googlegroups.com
Hi, this is a community related to the Erlang VM (also known as BEAM) and not the Apache Beam project.
--
You received this message because you are subscribed to the Google Groups "BEAM Community" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/beam-community/0b19e220-b131-42d6-969b-a609dd4409f3%40googlegroups.com.
jona...@simility.com
Nov 27, 2018, 9:27:32 AM11/27/18
to BEAM Community
Ups, my bad. Sorry for that. I will close that thread.
Reply all
Reply to author
Forward
0 new messages