I have the need to set up and run "embarrassingly" parallel jobs on Google Compute Engine. I am looking for tools to facilitate this.
On EC2, I was using MIT's Starcluster to set up the cluster, and then just submitting the job to SGE.
Are there similar tools available yet for Google Compute Engine?
If not, I will probably manually set up a Condor Cluster to do the jobs. Buzz kill.
On
Starcluster is really cool. Unfortunately, there is no support for GCE and I'm familiar with any similar systems at this point.
I'd love to see that happen though so that if you get something working, please share!
On
Elasticluster has Google Compute Engine support. It is very similar to Starcluster and works on AWS using boto, GCE, Openstack. https://github.com/gc3-uzh-ch/elasticluster
I didn't find anything out there. Starcluster's current path to GCE support is to wait for official boto support for GCE. That all may take a while, so I just went ahead and wrote some python code to start up and terminate a basic cluster.
I shared it here on github.
I hope someone else finds it useful, at the very least as sample code.