Hi David Reiss, thanks for your comment. I think the code is intentionally made for general case, for any size of vector A and B.
I found his other paper which also has the similar mathematica code as well as the explanation of the code. I attach the paper below (on appendix B). It helps me to understand the structure of the input of the function.
Anyway, I apply YFoxH for a special case, as given on equation (17) of the previous paper (outage capacity multicarrier.pdf), with L = 1 ; m1,m2,...ml = 1; Cth = 1. Hence, I modify the code as attach below. Now the error is not because of the Do loop, but still I do not get the final result :(
Attachments:
