Spark on the go lab ?
BMPCreated with Sketch.BMPZIPCreated with Sketch.ZIPXLSCreated with Sketch.XLSTXTCreated with Sketch.TXTPPTCreated with Sketch.PPTPNGCreated with Sketch.PNGPDFCreated with Sketch.PDFJPGCreated with Sketch.JPGGIFCreated with Sketch.GIFDOCCreated with Sketch.DOC Error Created with Sketch.
question

Spark on the go lab ?

Par
DataConvergenceTeam
Créé le 2018-10-04 10:21:13 (edited on 2024-09-04 12:34:28) dans Public Cloud OVHcloud

Hello fellow spark users !

We are wondering, for a couple of month yet, if you would be interested in a spark tool ?
We think that managing spark cluster can be painful, especially for dev jobs. That is why we created a tool which spawn a spark cluster on the go when launching a `spark-submit` like command.

To sum up, to make it work we need :

* an OVH PCI project
* an OpenStack token
* install a binary called : `ovh-spark-submit`

Then to run a job (and spawn a cluster to run it) just type for example:
`ovh-spark-submit --OS_TOKEN {your token} --class com.ovh.example.SparkScalaApp --name SparkJob1 --executor-memory 2G --total-executor-cores 4 spark-word-count-assembly-0.1.jar`

No need to have spark client / libs installed, cluster size is computed according to `--executor-memory` and `--total-executor-cores` values.

Do you want us to add it in labs.ovh.com ?
We really would like to get your insight :)


Les réponses sont actuellement désactivées pour cette question.