I finally had the time to do some additional checking, and found that even this seemingly harmless minimal example eats ~43 KB of memory every time you run it, with seemingly no way of clearing up that memory.
Remove["Global`*"];
ClearSystemCache[];
$HistoryLength = 0;
Share[];
MemoryInUse[]
Needs["NDSolve`FEM`"];
region = Rectangle[{0, 0}, {1, 1}];
op = Laplacian[u[x, y], {x, y}];
dirichlet = {DirichletCondition[u[x, y] == 0, x == 1],
DirichletCondition[u[x, y] == 1, x == 0]};
uif = NDSolveValue[{op == 0, dirichlet}, u, {x, y} \[Element] region];
Could someone with Mathematica 10 (required for NDSolve`FEM) try this out and see if the same happens for them? Just paste it into a notebook, run it a few times until the memory usage "stabilizes" (first three runs or so the memory usage is all over the place), and then see if the memory consumption consistently goes up.
In fact, just calling the first six lines (i.e. up to and including the Needs-statement) seems to suck up 1 KB of memory every time it's run.
P.S. Just to clarify, I know 43 KB seems like a small amount, but keep in mind that this is a minimal example. In my actual work, I'm seeing memory leakage of up to 2 GB in as little as an hour.
P.P.S. Before you ask: yes, I'll make a bug report about this soon unless someone finds a miracle cure for it. I just want to check if the issue is replicateable before I send the report.