Speed up the evaluating of a function?

Questions
,
1

Dear community,
I need to evaluate a function of this type

f[List_]:=Product[1-i,{i,List}]

and I need to do this for 10 millions of times and then I need to calculate the average, I am only interested in the numerical value of the function. Each of this times the list obviously change but not only in the content but also in length. Using the standard Mathematica tools I solved the problem defining a not constant list of two levels, e.g.:

List2= {{1,2,3},{2,3,4,5},{1,2}}

in my case I have 10 millions of sublists of different length, these length are randomly assigned through a Poisson distribution, the content of the lists is as well randomly assigned.
I evaluate the mean in this way:

Mean[Table[f[List2[[i]],{i,1,n}]]

if I use n=1 million my laptop needs 207s, with n=10 millions it needs more than 1 hour ( I never ended the simulation).
Since I need to repeat this process a lot of times, is there a method to speed up the code without using a more powerful pc?
In a similar case I really improved my code using the built-in function Compile, but I do not think it is possible use an irregular list like mine with this function, isn’t it?
Thanks for the attention

2

Alessio,

You have multiple syntax errors here.

List_ can’t be a pattern (must be lower case or something else), Generally its a bad idea to capitalize your variables because you will interfere with Mathematica’s symbols (as you do here).

Next time, please upload code that demonstrates your issue. Since you did not, I created some random data with 1 Million lists:

list2 = Table[RandomInteger[5, RandomInteger[{1, 10}]], 1000000];

Define the function:

f[lt_] := Product[1 - i, {i, lt}]

Your approach:

AbsoluteTiming[Mean[Table[f[list2[[i]]], {i, 1, Length[list2]}]]]

7.2 seconds

Change to Map:

AbsoluteTiming[Mean[Map[f, list2]]]

4.25 seconds

Compile the function and Map it:

foo = Compile[{{lst, _Integer, 1}}, Product[1 - i, {i, lst}]]
AbsoluteTiming[Mean[Map[foo, list2]]]

0.45 seconds

Simple steps got a factor of 16 improvement. You likely could do even better by reworking the algorithm but this is a start…

Regards

3

Thank you for the answer. I did not post my code because it is pretty long and I tried to concentrate to the delicate point. It was my fault because your answer does not work in my case.
Assuming that n=10^6. I have a list of this kind

a=RandomReal[{0,1},1000000];

then using this numbers I calculate these lists:

d = 4;
df = RandomVariate[PoissonDistribution[d], n];
p = RandomSample[a, n];
p1 = RandomSample[a, n];
p2 = RandomSample[a, n];
pp = Table[RandomSample[a, df[[i]]], {i, 1, n}];

as you can see pp is the irregular one the others are one level lists.

The function, that I need to evaluate, is (y is just a real number)

S2[y_, d_, p_, p1_, p2_, pp_] := 
 y (1 - p - d/2*(Exp[-y] p1 p2)/(1 - (1 - Exp[-y]) p1 p2)) + 
  Log[Exp[-y] + (1 - Exp[-y]) Product[1 - j, {j, pp}]] - 
  d/2 Log[1 - (1 - Exp[-y]) p1 p2]

At this point I calculated the million of contributions

 Table[S2[y, d, p[[i]], p1[[i]], p2[[i]], pp[[i]]], {i, 1, n}];

and then the mean (of course I can do two passage in one but the mean is not important here, I think).
In this case I do not understand how can I use Map instead of Table.
Then can you explain to me why Map is much more efficient than Table in this case?

4

Alessio,

The code you posted is more helpful. In this case MapThread sped things up by about 10%. Compile was 14 times faster than Map uncompiled and that is without reworking the algorithm to be faster. I really suggest you stop using Capital letters for your variables. One day they will get you in trouble.

s2c=Compile[{{y, _Integer},{d, _Integer},{p, _Real},{p1, _Real},{p2, _Real},{pp,_Real,1}},
       y (1-p-d/2*(Exp[-y] p1 p2)/(1-(1-Exp[-y]) p1 p2))+Log[Exp[-y]+(1-Exp[-y]) Product[1-j,{j,pp}]]-d/2 Log[1-(1-Exp[-y]) p1 p2]]

I created a test function since I do not know y, I just fixed y=1 and d.

smc[p_, p1_, p2_, pp_] := s2c[1, d, p, p1, p2, pp]

AbsoluteTiming[MapThread[smc, {p, p1, p2, pp}]]

Hope this helps

5

Dear Neil
Thank you very much. You really helped me a lot. I will try to remember your hint: I will not use anymore the capital letters.