Message Boards Message Boards

Speed problem: Import and post-process 1000 images

GROUPS:

Hi community :-)

I have 1024 csv-files in my folder. Each csv-file represents an 16-bit image with 1024 x 1024 (rows and columns).

My goal: Import the images, apply ImageTrim[] to every image to get rid of the unnecessary parts (the same for every image). Afterwards calculate the total power of each image. Finally, save this result. Repeat this for every image.

Problem: Import takes very long (20-30 minutes). Post-Processing takes another 20 minutes.

Question: How can I optimize my code?

Read file names and create temporary file to save results:

SetDirectory["C:\\Users\\images"];  (*path to the folder where all the images are stored*)
NameofFiles = FileNames[]; (*lists all files in the current working directory.*)
NumberofFiles = Length[FileNames["*"]]; (*number of files in the current working directory.*)
file = OpenWrite["file.dat", FormatType -> OutputForm](*Opens a file called file.dat which is used later to temporary save data*)

Loop in which I import every single csv-file and calculate and save intensity:

Do[
 ImagesCSV = Import[NameofFiles[[t]]] ;    (*import file*)
 Img = Image[ImagesCSV, "Bit16"];  (*transforms image to csv*)
 ImgTrim = ImageTrim[Img, {{150, 130}, {1000, 890}}];  (*trimms the image*)
ImageTrimmed = ImageMeasurements[ImgTrim, "Total"]; (*calculate total intensity of trimmed image*)
Write[file, ImageTrimmed];  (*writes result into temporary file*)
, {t, 1, NumberofFiles}]

Now it try to close my temporary file: file.dat (I think it doesn't really close because when I try to delete it with DeleteFile[] it doesn't really work)

The code works fine, but is very slow. Any ideas how to speed it up?

Peter

EDIT: For some reason the first step (read file names...) seems to be significantly faster today. Maybe I have to quit the kernel before I run the code. To loop takes now 15-20 minutes which I think is reasonable. So, seems that today is much faster then yesterday. Anyway, do you have ideas to improve the code in terms of speed? If not, I think I can live with 20 minutes processing time.

Attachments:
POSTED BY: Peter Parker
Answer
23 days ago

Please provide a sample file to benchmark on.

POSTED BY: Szabolcs Horvát
Answer
23 days ago

Hi

The complete code is already included in my first post. Additionally I have attached 2 (out of 1000) csv-files to it.

POSTED BY: Peter Parker
Answer
23 days ago

I was unable to get ReadList to handle commas as separators, but the following at least provides a ~2.3x speed boost to importing on my computer.

csvRead[file_] :=
 Module[{stream, data},
  stream = StringToStream@StringReplace[ReadString[file], "," -> "\t"];
  data = ReadList[stream, Real, RecordLists -> True];
  Close[stream];
  data
  ]

In[59]:= csvRead["~/Downloads/11.4449.csv"]; // AbsoluteTiming

Out[59]= {0.664697, Null}

In[60]:= Import["~/Downloads/11.4449.csv", "CSV"]; // AbsoluteTiming

Out[60]= {1.57064, Null}
POSTED BY: Szabolcs Horvát
Answer
23 days ago

Hi Szabolcs

Indeed, this technique is definitely faster. Thanks a lot. I think with this improvement I can now say, the speed is reasonably okay. Yesterday it was quite slow but with your code and "Quit Kernel" the speed is much higher.

Thanks a lot :-)

Peter

POSTED BY: Peter Parker
Answer
23 days ago

Actually, a speedup factor of 2 isn't very great. I would love to see a faster solution.

POSTED BY: Szabolcs Horvát
Answer
23 days ago

Hahaha, true :-) It is not very fast. I am sure it can be done much faster. But how much effort you have to put into this just to be a little bit faster....? Is it worth it? But I agree, would be interesting to see and I am very curious about it.

POSTED BY: Peter Parker
Answer
23 days ago

Group Abstract Group Abstract