AN EVENT-DRIVEN, SCALABLE, NEAR-REAL-TIME, HIGH-AVAILABILITY, PREDICTIVE IMAGE ARCHITECTURE.

Lately I've been looking at little things to see which is the best way for an event-oriented architecture (with Confluent platform, for example) to get predictions on images, besides fighting with Kubernetes and the confluence platform, you know, a way to have packaged and super easy a Kafka/Zookeeper Cluster, while encouraging you to use that…

Advertisement

Working with spark, again.

It has been a while since I last published something, so I am going to refresh how to work with spark. These are some notes that I think I will need some day. I start with the basic operation with the Dataframes. I am going to run a spark-shell, actual version is 2.4.4. // It…

Playing with Word2Vec, my cv, spark and scala

Hi,i just have to play and learn how to use this algorithm provided by spark-ml to do some feature extractions from some text using Google`s Word2Vec algorithm, i mean, why not to use my actual cv? Before that, probably you will have to convert the pdf file to text file. Actually i am working with…

About how to parallelize multiple Machine Learning Algorithm using a pipeline with spark.

You basically need to make a Pipeline and build a ParamGrid with different algorithms as stages.  Here is an simple example: val dt = new DecisionTreeClassifier() .setLabelCol("label") .setFeaturesCol("features") val lr = new LogisticRegression() .setLabelCol("label") .setFeaturesCol("features") val pipeline = new Pipeline() val paramGrid = new ParamGridBuilder() .addGrid(pipeline.stages, Array(Array[PipelineStage](dt), Array[PipelineStage](lr))) val cv = new CrossValidator() .setEstimator(pipeline) .setEstimatorParamMaps(paramGrid)…

About how to work with RDDs using scala

About how to work with RDDs using scala I am doing a big data course with formacionhadoop.com, concretly, online master big data expert, 150 hours and i am going to write this post in order to remember in the future how to work with RDDs using scala code, a pure functional language for the JVM.…