Not sure if I understand entirely, but the following might be what you want. I'll build up to it gradually.
You said,
I tried to get the middle points of each side and then rotate by 45 degree
If you can get the middle point, then you've already got the vertices for the next square. No need for a separate rotation. A simple way to get midpoints is with Mean:
Mean[{{a, b}, {c, d}}]
output: {(a + c)/2, (b + d)/2}.
Now what we'd like to do is to apply Mean to adjacent pairs in your squarePts. You could use Partition to make pairs and then map Mean over the result. Or you could use BlockMap. There are probably many other equivalent approaches, but I'll use BlockMap.
BlockMap[Mean, squarePts, 2, 1]
output: {{1/2, 0}, {1, 1/2}, {1/2, 1}}.
Oops. We need to "wrap around". BlockMap doesn't have an option to do that, but we can alter the data BlockMap is applied to.
BlockMap[Mean, PadRight[squarePts, 5, squarePts], 2, 1]
output: {{1/2, 0}, {1, 1/2}, {1/2, 1}, {0, 1/2}}.
Your NestList idea looks promising. It would be a bit clearer if we bundled what we've done into a function first.
InnerPolyPts[pts_] := BlockMap[Mean, PadRight[pts, 5, pts], 2, 1]
Now use NestList:
NestList[InnerPolyPts, squarePts, 3]
output: {{{0, 0}, {1, 0}, {1, 1}, {0, 1}}, {{1/2, 0}, {1, 1/2}, {1/2, 1}, {0, 1/2}}, {{3/4, 1/4}, {3/4, 3/4}, {1/4, 3/4}, {1/4, 1/4}}, {{3/4, 1/2}, {1/2, 3/4}, {1/4, 1/2}, {1/2, 1/4}}}.
Let's visualize our nested squares:
Graphics[{FaceForm[None], EdgeForm[Black], Polygon[NestList[InnerPolyPts, squarePts, 3]]}]
