We're using Amazon EMR for our oozie workflows which contains Spark jobs. In our cluster, we have 1 Master, 2 core nodes and using third party tool for Task nodes as spot instances.
Autoscaling is setup for task nodes based on Yarn memory usage. We have configured to launch Application Master only in core nodes as task nodes are spot instances which would go down any time.
Now the problem is sometimes running jobs occupy the Core nodes memory fully (AM + Task executors), which leaves other jobs in accepted state waiting for core node to free up to launch AM.
I'd like to know is it possible to limit only AM to launch in Core nodes and task executors in task nodes. This way we'll be able to run multiple jobs in parallel.