For some time I did not update the blog, so I want to add a small repository in github on how to create a parquet file using the spark-csv library of databricks. Super easy to use, so simple that the code is self-explanatory. https://github.com/alonsoir/parquet-utils
Hi everyone, as usual, I put here a recipe to create fast a table in Hive using the data from a data frame. I am using a spark-shell connected to a development cluster, cloudera version is cdh5.5.2, so, according to this official cloudera site, hive version is 1.1.0. It should be important when I have […]
Twitter Sentiment Analysis ========================== * retrieve tweets using Spark Streaming * language detection * sentiment analysis (StanfordNLP) * index tweets in Elasticsearch * live dashboard using Kibana First at all, thank you to Vincent Spiewak, original author of this project. I have modified a bit the code in order to run it in local mode […]
Hi, recently i was receiving classes from formacionhadoop.com, Master online big data expert, 150 hours, which means that i received notions about hadoop, spark and nosql databases. A good course, i recommend it to everyone to learn the basis of big data technology. I already have taken classes from Andrew Ng with its Machine Learning coursera course. […]
About how to work with RDDs using scala I am doing a big data course with formacionhadoop.com, concretly, online master big data expert, 150 hours and i am going to write this post in order to remember in the future how to work with RDDs using scala code, a pure functional language for the JVM. […]
This is the first post about how to create a recommendation engine, the preliminar know how to build a good recommendation engine. What is it a recommendation engine? well, the users from Spotify, Amazon and Netflix know that the recommendations shown to us are really accurate, it looks like there are somebody who knows us […]
Hi everyone, in my process of mastering scala and big data technologies, i am learning how to integrate apache kafka, spark-streaming, mongodb and twitter. The idea is simple, a producer process trying to push json tweets from twitter within a kafka topic and another spark-streaming process trying to pull data from that topic and storing […]
