When applying the chain rule, it still needs to apply the derivative to the av operator, and then my rule would apply, wouldn't it?

When applying the chain rule, Mathematica would only need to evaluate D[av(u), u], which is 1 (df/dx = df/du du/dx).

Now that I look at it again, I think both answers (yours and Mathematica's) are correct. In fact, can't we use both results to conclude av(x) is the identity function?

Your way (special D rule): D[Log[av[f(x)]], x] == D[av[f(x)], x] / av[f(x)] == av[D[f(x), x]] / av[f(x)] == av[f'(x)] / av[f(x)].

Mathematica (chain rule): D[Log[av[f(x)]], x] == D[av[f(x)], x] / av[f(x)] == (D[av(u), u] /. u -> f(x)) D[f(x), x] / av[f(x)] == 1 * f'(x) / av[f(x)].

This means av[f'(x)] == f'(x), and letting f(x) == x^2/2 gives us av(x) == x?