Hi David,
With the Nest command one applies a function several times to an expression. The evaluation of, for instance,
Nest[U,x,2]
gives the same output as
U[U[x]]
So in your example
Nest[UConv, {U, y}, 2]
is equivalent to
UConv[UConv[{U, y}]]
With that in mind it should be easier to understand what is going on. The first element of Convolve in the output you obtain
{Convolve[Piecewise[{{1, 0 <= x <= 1}}, 0], Piecewise[{{2 - y, 1 < y < 2}, {y, 0 < y <= 1}}, 0][x], x, y], y}
is U[x] (confer definition of UConv), the second element is the first element of the output list given by
UConv[{U, y}]
to which an [x] is added on the right (confer again definition of UConv). The last element of Convolve and the last element of the output you obtain are both the second element of the output list given by
UConv[{U, y}]
so in this case y. Now, Mathematica does not evaluate
Convolve[Piecewise[{{1, 0 <= x <= 1}}, 0], Piecewise[{{2 - y, 1 < y < 2}, {y, 0 < y <= 1}}, 0][x], x, y]
because there is an extra [x] at the second Piecewise. You can check that the first piecewise function does not have any [x], because none is needed in effect. So the definition for UConv should be instead
UConv[w_] := {Convolve[U[x], w[[1]], x, w[[2]]], w[[2]]}
and the [x] should be written when calling it, namely
UConv[{U[x], y}]
However we run into a new problem when calling twice UConv, with
Nest[UConv,{U[x], y},2]
since the ouput is the same as
UConv[{U[x], y}]
(The second evaluation of UConv did not process.) The thing is that UConv[{U[x], y}] returns a Piecewise whose variable is y, but we want piecewise functions of variable x since this is what we ask for in the definition of UConv. This is the third argument of
UConv[w_] := {Convolve[U[x], w[[1]], x, w[[2]]], w[[2]]}
We simply need to change y (which is actually w[[2]]) to x by redefining UConv as
UConv[w_] := {Convolve[U[x], w[[1]] /. w[[2]] -> x, x, w[[2]]], w[[2]]}
Using this last definition, you should be able to Nest it without any problem (I have tried it with some examples and it seems to work OK) with
Nest[UConv, {U[x], y}, 2]
Note at last that, contrary to your remark, this is
UConv[{U[x], y}]
which is equivalent to
Convolve[U[x], U[x], x, y]
The evaluation of
Nest[UConv, {U[x], y}, 2]
is equivalent to
Convolve[U[x], U[x], x, y];
Convolve[U[y], %, y, t]
Best,
Xavier