Straight to the point, the code is so intuitive that I think it doesn't need much more. To say that I personally am very excited about this possibility, because the ability on the one hand to tell the spark cluster what to do, plus the possibility of having an agent with all the knowledge and…
A proposal for a lambda architecture for modern real-time telecommunications
To obtain real consistency with Delta Lake, Hudi and Iceberg leaving behind Apache Impala and classic Spark.
Una propuesta de una arquitectura lambda para telecomunicaciones modernas en tiempo real
Una propuesta para tratar de dejar atrás Impala cuando necesitas consistencia de datos y la tratas de conseguir mediante software
𝗦𝗽𝗮𝗿𝗸 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹𝗶𝘀𝗺
more from spark and parallelism tips.
Parsing an exception in a long running Spark Streaming application.
We start from the fact that the exception appears at some point in the execution: java.io.IOException: Filesystem closed. and GC HEAD OVER LIMIT. It does not indicate how this last message appears, but I think it is not very important. Now, I'm going to try to figure out what's going on by asking questions and…
Analizando una excepción en una aplicación Spark streaming de larga duración.
Analizando una excepción Filesystem closed en una app spark streaming de larga duración.
Preguntas y respuestas sobre Spark en ChatGPT
Consejos sobre Apache Spark.
About capturing log data in a distributed system
Recently I did a test for a company in which they asked for two apparently very simple exercises but which contain a great deal of complexity as soon as you realise, so I'm going to do an exercise in analysis of how I saw that problem and how I would deal with it. I give…
About a few simple examples with Kafka, Spark, Flink and Kafka Streams
Recently I have been very busy for health reasons in myself, taking care of myself, going to the hospital to receive treatment, taking care of my parents too, who are already an age, so I have not been able to devote as much time as one would like to the blog, but some time I…
Sentiment analysis with spark, twitter, elastic and kibana
Twitter Sentiment Analysis ========================== * retrieve tweets using Spark Streaming * language detection * sentiment analysis (StanfordNLP) * index tweets in Elasticsearch * live dashboard using Kibana First at all, thank you to Vincent Spiewak, original author of this project. I have modified a bit the code in order to run it in local mode…
Aironman techblog 