About how to avoid snappy exception in a test scope

Have you suffered this exception while you are developing with the apache spark framework?:

org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null

Congratulations, you are not the only one that suffered such that way, you are not alone 🙂

This library is provided by spark-core itself, so, it is expected that this library will be there with your cloudera or whatever spark provider you use, but, in development time, you do not have any cluster, only your local machine, that it can work in pseudo distributed mode but it is not the usual because you will need a decent machine, so you will probably have a spark-shell in your local machine, which is great but it can be difficult to use if you want to test your code. For that reason, you will have to write unit test, so these instructions are focused on that task.

I am actually developing with IntelliJ IDEA 2016.3.2, so I will provide instructions about how to avoid this exception when you are developing spark code and unit test:
Go to

File –>Other Settings –> Default settings –>Build Execution Deployment –> Build Tools –> Maven –> Runner 

These are my VM options:

-Xmx2048m -XX:MaxPermSize=1024m -Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib -Dorg.xerial.snappy.tempdir=/tmp

and then, remember to add these VM options to your unit test.

Run->Edit configurations –> Default –> JUnit 

These are my  VM options:

-ea -Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib -Dorg.xerial.snappy.tempdir=/tmp

Thats all.
Have fun in the process.
Alonso

Responder

Introduce tus datos o haz clic en un icono para iniciar sesión:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión / Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión / Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión / Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión / Cambiar )

Conectando a %s