Hello fellow spark users !
We are wondering, for a couple of month yet, if you would be interested in a spark tool ?
We think that managing spark cluster can be painful, especially for dev jobs. That is why we created a tool which spawn a spark cluster on the go when launching a `spark-submit` like command.
To sum up, to make it work we need :
* an OVH PCI project
* an OpenStack token
* install a binary called : `ovh-spark-submit`
Then to run a job (and spawn a cluster to run it) just type for example:
`ovh-spark-submit --OS_TOKEN {your token} --class com.ovh.example.SparkScalaApp --name SparkJob1 --executor-memory 2G --total-executor-cores 4 spark-word-count-assembly-0.1.jar`
No need to have spark client / libs installed, cluster size is computed according to `--executor-memory` and `--total-executor-cores` values.
Do you want us to add it in labs.ovh.com ?
We really would like to get your insight :)
Spark on the go lab ?
Sujets apparentés
- [RESOLU] Connexion impossible en SSH
13994
05.06.2019 20:05
- Bonjour, Je n'est reçus aucun mot de passe root lors de mon achat!
10186
05.02.2018 20:47
- Configuration IP failover avec netplan (Ubuntu 17.10)
8381
12.01.2018 23:23
- IP Failover sur Debian 9
6637
18.11.2016 20:40
- Ssh connection timed out port 22
5639
11.12.2019 08:21
- Problème connexion ssh
5362
04.02.2018 09:46
- Connexion OpenStack Swift Object Storage
5073
11.04.2019 10:09
- Désactivation de mon site pour Phishing
4819
12.05.2021 08:36
- [RESOLU] VNC Console - Coller un texte
4049
14.01.2018 18:48
- [Officiel] Roadmap Public Cloud
3985
02.06.2017 08:53