Okay, I'll try to explain. Yes, it has to do with the evaluation sequence. Let's consider a simpler example. We want to construct the body of a Function ahead of defining the actual function, so something like this:
expression = Slot[1] + Slot[2]
At some later point, we want to construct an actual Function, in your case with parameters but that doesn't really matter, even this simple case will demonstrate the problem.
function = Function[Null, expression]
(* This is a verbose form, but it is equivalent to Function[expression] and expression & *)
And now we want to apply our function:
function[a, b]
but the result is
#1 + #2
You can see what happened with Trace:
function[a, b] // Trace
(*
{{function,expression&},
(expression&)[a,b],
expression,#1+#2}
*)
The Function symbol has the HoldAll attribute. So, expression doesn't get evaluated until the Function starts evaluating. Specifically, Function will hold its body unevaluated until the arguments have been bound.
Now, in this simple case, we could actually address this with Evaluate:
function2 = Function[Null, Evaluate[expression]];
function2[a, b]
(* a + b *)
But Evaluate is not a solution to all such problems, because it only works at the first level. If it's buried deeper, then it doesn't force evaluation until outer levels have been evaluated. Specifically, you wouldn't be able to add Evaluate directly to this
ds[All,Association[#,expression]&]
like this
ds[All, Association[#, Evaluate[expression]] &]
because it's buried too deep. There is probably some way to unravel all of your computations to get this to work, but I just don't think it's worth trying. My suggestion decomposes the problem more clearly and would be easier to update for new scenarios (in my opinion).
