About how to parallelize multiple Machine Learning Algorithm using a pipeline with spark.

You basically need to make a Pipeline and build a ParamGrid with different algorithms as stages.  Here is an simple example:


val dt = new DecisionTreeClassifier()
.setLabelCol("label")
.setFeaturesCol("features")


val lr = new LogisticRegression()
.setLabelCol("label")
.setFeaturesCol("features")



val pipeline = new Pipeline()


val paramGrid = new ParamGridBuilder()
.addGrid(pipeline.stages, Array(Array[PipelineStage](dt), Array[PipelineStage](lr)))


val cv = new CrossValidator()
.setEstimator(pipeline)
.setEstimatorParamMaps(paramGrid)


More info in https://issues.apache.org/jira/browse/SPARK-19357

Thank you Brian Cutler.

https://bryancutler.github.io/cv-pipelines/

Advertisement