I have posted this problem a week ago on Stack Exchange, and it seems like a system dependent bug or something. I have reported this to Wolfram Support but I get no response, so I think maybe I should post the problem here. Below is the description:

I am using Mathematica 11.3 with Windows 10.

When calling Export to export a List of Images as a QuickTime movie, the memory used by Mathematica 11.3 increases but it is not released after the evaluation. The code below is a simple example:

$HistoryLength = 0;
frame = Rasterize[Plot[x, {x, 1, 2}], RasterSize -> 1000, ImageSize -> {960, 720}];
frame = Table[frame, {i, 100}];
Export["D:\\example.mov", frame, "FrameRate" -> 100];

After runing this Cell, open Task Manager to check how much memory Mathematica is using. On my PC, it shows that Mathematica is using 769.0MB memory. Then run the Cell below:

Export["D:\\example.mov", frame, "FrameRate" -> 100];
MemoryInUse[]/1024.^2

Now Task Manager shows that Mathematica is using 1055.4MB memory, while Mathematica 11.3 shows that the memory in use in MB is:

91.6286

If you run the second Cell for more several times, the number in Task Manager keeps increaing but the number in Mathematica does not change obviously.

This problem is forcing me to restart the kernal everytime i export a lots of images as a QuickTime video.