I recommend posting exact code in the from:
In[1]:= $Version
Out[1]= "10.4.1 for Mac OS X x86 (64-bit) (April 11, 2016)"
In[2]:= AbsoluteTiming[JoinAcross[{<|a -> 1, b -> X|>}, {<|a -> 1, c -> Y|>}, Key[a]];]
Out[2]= {0.000039, Null}
otherwise it is hard to track. You of course can generate larger random Associations.