Community RSS Feed
https://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag High-Performance Computing sorted by activeRemoteBatchSubmit : how to access large file system (EFS)
https://community.wolfram.com/groups/-/m/t/2200785
Submitting jobs to EC2 through RemoteBatchSubmit works very well. However, I need to access a lot of data for training deep nets. How Can I access this data efficiently?
- I can' t send the data through the option InputFile, because it is too big.
- Access must be fast from the EC2 instance
- Data must be shared through instances, and is read-only
I think EFS would be great, but I don't see a way to mount it automatically to instances, or to mount the file system from inside the Notebook.
Any help or pointer will be appreciated!Sebastien Roy2021-02-21T20:55:55ZSlow ParallelDo
https://community.wolfram.com/groups/-/m/t/2199270
Hello,
I am trying to speed up my code with ParallelDo, but I ended up with a super slow case. Here is an artificial example of the case:
inits = RandomReal[{0, 5}, 10000];
sols = Table[
NDSolve[{f'[t] == Sin[t], f[0] == inits[[j]]}, f[t], {t, 0, 10}],
{j, 1, Length[inits]}];
Now, the parallel part
ParallelDo[
qsol = sols[[1]];
f[t] /. qsol /. t -> 4
, {i, 1, 50}
] // AbsoluteTiming
this took {15.5051, Null}
qsol = sols[[1]];
ParallelDo[
f[t] /. qsol /. t -> 4
, {i, 1, 50}
] // AbsoluteTiming
this took {0.01021, Null}
The situation gets worse as the number of sols increases. What is the problem here?
I am guessing that inside the parallel block, sols is copied to each kernel and this copying slows down as the number of elements increases in sols. So, I tried
SetSharedVariable[sols[[1]]];
ParallelDo[
qsol = sols[[1]];
f[t] /. qsol /. t -> 4
, {i, 1, 50}
] // AbsoluteTiming
This took {2.07197, Null} which is an improvement.
But when I tried
SetSharedVariable[sols];
ParallelDo[
qsol = sols[[1]];
f[t] /. qsol /. t -> 4
, {i, 1, 50}
] // AbsoluteTiming
This took {125.364, Null}. This is ridiculusly long.Yeso Alde2021-02-19T15:48:13Z