Hi Jesse, thank you very much for the detailed description and the suggestions. I am working on a different project right now, but will try out your points as soon as I get to work on this some more.

I think setting the thread numbers might do the trick, as my calculation is not using any subkernels, but clearly the Mathematica functions I use are using multiple threads (e.g. 4 threads on my local machine with 4 physical cores). I will test this as well as comparing the speed with Intel based instances and limiting the threads to half the hardware threads to achieve one thread per physical core.

A lot of what you wrote here would be good to have in the documentation. Even though I read early on that vCPUs do not correspond to physical cores, it still took a while to understand how all the numbers are connected. For example, it was not clar to me in the beginning if the vCPU count given for RemoteBatchMapSubmit is per one job or per the whole array.

Thank you so much for your help!

Regarding the slowdowns you saw with multiple concurrent jobs, it occurred to me that although $ProcessorCount is overridden to reflect the "VCPUCount" of each job, the processor counts used for internal multithreaded computation will still reflect the host instance's total physical core count. This would cause excessive contention with multiple concurrent jobs, where e.g. an N-physical-core instance has X jobs each running N software threads, resulting in N * X total threads contending for only N physical cores. You can change this internal processor count by evaluating at the start of your job code (or for array batch jobs, in the Initialization option):

SetSystemOptions["ParallelOptions" -> {
    "MKLThreadNumber" -> $ProcessorCount,
    "ParallelThreadNumber" -> $ProcessorCount
}]

Let me know if this has any effect for your application. This is probably something RemoteBatchSubmit should be doing automatically; I will look into having it do so.