Message Boards Message Boards

One pixel thermal imaging camera with Mathematica and Arduino

Triggered by a leak in my hot water boiler at home I built a thermal imaging camera using an Arduino and interfacing it with Mathematica. I tried to make up for the "one-pixel-resolution" by using Mathematica's powerful image analysis abilities. This is a work in progress and I would be delighted to get some comments/suggestions from the Community. In this project, I had a lot of help from Bjoern Schelter, who has recently joined this Community. If you have the components and use the programs below, you should have a "working" one-pixel thermal camera after 30 minutes or so of DIY. Here's a sneak peek of what we want to get out (this one is a "selfie"):



I use the following components:
  1. Arduino Uno R3
  2. MELEXIS / MLX90614ESF-DCI / DS Digital non-contact Infrared Temperature Sensor  (~ £35 and more or less the same in USD)
  3. MG995 Servo Sensor Mount Kit 2 DOF Pan and Tilt Black (~ £24, similar in USD)
  4. Two 4.7 kOhm resistors.
  5. One 0.1 uF capacitor.
  6. One small breadboard.
  7. 5V power source.
  8. Wires. 
The idea is illustrated in this Youtube video. To my best knowledge, the original idea comes from a project of Steffen Strobel in the German science competition "Jugend Forscht". The main idea is to mount a non-contact temperature sensor on a pan and tilt mechanism (i.e. two servos) on a tripod. An Arduino microcontroller is then used to communicate via the serial port with Mathematica, which is used to control the servos and triggers the measurements. After the data acquisition Mathematica cleans the data and produces some thermal images (see below).

We use the following wiring diagram to connect the servos and the temperature sensor to the Arduino.


The resistors are 4.7kOhm and the capacitor is 0.1uF. The sensor part is taken from the bildr.blog, which also shows how to make Arduino talk to the sensor. The Melexis sensor that we chose has a temperature resolution of 0.02 degrees Celsius and a rather narrow field of view, which is important for our application. 

For the servo part, we use the standard servo.h library; an example of its application can be found here.

Here is a photo of the sensor/head of the device.


The entire device looks like this.


The idea is to use Mathematica to send instructions to the servos and to initiate the measurements. To interface Mathematica with Arduino we use the SerialIO package. I found this website by William Turkel very useful to make SerialIO work on my Mac; following the steps and adapting some directories makes the package work without any problems.

At that point, we have everything in place, and only need to put the bits together. We first need to upload this piece of code (also attached at the bottom) to the Arduino.
  #include <i2cmaster.h>
  #include <Servo.h>
  
  //Servo setup
  int servoPin1 = 9;
  int servoPin2 = 10;
  Servo servo1; 
  Servo servo2;
  int angle1 = 40;   // servo start positions in degrees
 int angle2 = 50;
 
 
 //Melexis setup
 int sensor = 0;
 int inByte = 0;
 
 
 void setup()
 {
     Serial.begin(9600);
    
        // attach pan-tilt servos
        servo1.attach(servoPin1);
        servo2.attach(servoPin2);
 
 
        servo1.write(angle1);
        servo2.write(angle2);
 
 
     //Initialise the i2c bus
     i2c_init();
     PORTC = (1 << PORTC4) | (1 << PORTC5);//enable pullups
        establishContact();
 }
 
 
 void loop()
 {
  if (Serial.available() > 0)
   {
    inByte = Serial.read();
    
    int dev = 0x5A<<1;
    int data_low = 0;
    int data_high = 0;
    int pec = 0;
 
 
    i2c_start_wait(dev+I2C_WRITE);
    i2c_write(0x07);
 
 
    // read
    i2c_rep_start(dev+I2C_READ);
    data_low = i2c_readAck(); //Read 1 byte and then send ack
    data_high = i2c_readAck(); //Read 1 byte and then send ack
    pec = i2c_readNak();
    i2c_stop();
 
 
    //This converts high and low bytes together and processes temperature, MSB is a error bit and is ignored for temps
    double tempFactor = 0.02; // 0.02 degrees per LSB (measurement resolution of the MLX90614)
    double tempData = 0x0000; // zero out the data
    int frac; // data past the decimal point
 
 
  // Serial.print(tempData);
  // Serial.write(inByte);
    // This masks off the error bit of the high byte, then moves it left 8 bits and adds the low byte.
    tempData = (double)(((data_high & 0x007F) << 8) + data_low);
    tempData = (tempData * tempFactor)-0.01;
 
 
    //inByte = (float)(((data_high & 0x007F) << 8) + data_low);
 
 
   float celsius = tempData - 273.15;
    sensor=(int)(celsius*100);
    //float fahrenheit = (celsius*1.8) + 32;
 
   Serial.print(sensor);
  
    
    // horizontal "H"-> 72; reverse "R"-> 82; vertical "V"-> 86; end "E"-> 69
    
   if(inByte==72)
   {
    angle1=angle1+1;
    servo1.write(angle1);
   }
    if(inByte==82)
   {
    angle1=40;
    servo1.write(angle1);
   }
   if(inByte==86)
   {
    angle2=angle2+1;
   servo2.write(angle2);
  }
    if(inByte==69)
  {
   angle1 = 40;   // servo back to start
   angle2 = 50;
   servo1.write(angle1);
   servo2.write(angle2);
  }
 
   delay(15); // 15 works; wait 15 milliseconds before printing again
}


}



void establishContact()
{
while (Serial.available() <= 0)
{
   Serial.print('A');
   delay(100);
}
}

The idea is to make Mathematica communicate with the Arduino via the serial connection. The Arduino sketch shows that Ardunio is waiting for instructions, e.g. "H" to move horizontally, "V" to move vertically and "E" to go to the end position. 
 (*First we load the SerialIO package. See instructions above.*)
 
 << SerialIO`
 
 (*We test whether Mathematica's applications folder is in the Path. On some Macs Mathematica will be in the /Library directory - used in this example- and in others in the /Users/username/Library directory, where "username" needs to be replaced by the correct user name.*)
 
 MemberQ[$Path, "/Library/Mathematica/Applications"]
 
 (*If this gives True all is fine. If it evaluates to False execute
AppendTo[$Path, "/Library/Mathematica/Applications"]
*)

(*Connect to the Arduino*)

myArduino =
  SerialOpen[Quiet[FileNames["tty.usb*", {"/dev"}, Infinity]][[1]]];
SerialSetOptions[myArduino, "BaudRate" -> 9600];
While[SerialReadyQ[myArduino] == False, Pause[0.1]];

(*Data collection, in this case 40 vertical and 70 horizontal pixels; runtime 2-3 minutes; pauses cannot be reduced much further.*)

pixels = {}; SerialRead[myArduino]; For[j = 1, j < 41, j++,
For[i = 1, i < 71, i++, SerialWrite[myArduino, "H"];
  AppendTo[pixels, (SerialRead[myArduino] // ToExpression)/100.];
  Pause[0.1]]; SerialWrite[myArduino, "R"];
SerialWrite[myArduino, "V"]; SerialRead[myArduino];
Pause[0.1];]; SerialWrite[myArduino, "E"];

(*After the data aquisition close the connection to Arduino*)
SerialClose[myArduino]

(*Now we can use several different ways to represent the data, note that some point at the beginning/end of the scanned lines are removed; there were too many measurements errors just after the "carriage return"*)

ArrayPlot[Partition[Reverse[pixels], 70][[All, 2 ;; -10]], 
ColorFunction -> "Rainbow"]

(*here's another colour scheme.*)
ArrayPlot[Partition[Reverse[pixels], 70][[All, 2 ;; -10]],

(*Occasionally there are some outliers in the measurements; here we clean them out.*)
ArrayPlot[
Partition[Reverse[pixels /. x_ /; x > 35. -> 35.], 70][[All, 
   2 ;; -10]], ColorFunction -> "Temperature"]
ColorFunction -> "Temperature"]

(*This last one uses interpolation to make the image smoother.*)

ListContourPlot[
Partition[Reverse[Log /@ pixels /. x_ /; x > 35. -> 35.],
   70][[-1 ;; 1 ;; -1, 1 ;; -10]], AspectRatio -> 0.9,
ColorFunction -> "Rainbow", PlotRange -> All,
InterpolationOrder -> 2, Contours -> 60, ContourStyle -> None]

So here's a photo of my broken boiler and its scan:


Because of the scanning procedure (which just looks at the angle and does not use any projection), the scan is slightly distorted, but it is possible to recognize the main features and even the sticker on the front!

It appears that this rather primitive device can also be used to analyse electrical components. Here is an image of my MacBook Pro. 


 The position of the CPU becomes quite obvious.

There are many things that need to be improved:

(i) First, there is the projection issue. The scanner does scan angles. It needs to be projected to a 2D plane. One might use an ultrasonic distance sensor to get better results.
(ii) The device needs to be calibrated.
(iii) A user interface is needed. It would be useful to click on the image and get the temperature reading.
(iv) The communication between Mathematica and the Arduino need to be improved. The starting position of 40/50 degrees is hard-coded into the Arduino sketch. It should be done by the Mathematica code.
(v) We have not even started to use Mathematica's features on this. Much image processing could be done. The image should be overlayed to a normal photo of the object that is scanned. Manipulate could be used to change thresholds, i.e. the threshold to cut-off outliers, which is currently set to 35 degrees. 
(vi) The speed might be improved. I suppose that the scanning time of 3 minutes or so is typical for these devices, but one might improve that a bit. Also, Mathematica could use edge-detection to determine regions where a higher scan density would be helpful to get a better resolution. This only makes sense if the servos could be directed to a certain position much more precisely; alternatively, we could use random positions, which then are precisely determined using an accelerometer or so.

There is of course much more to do. In spite of this being work in progress, I wanted to share this project, and hope for helpful comments.

I attach the Mathematica notebook. I have a movie of the scanning process and the actual arduino sketch which I cannot upload directly. Here are links to the arduino sketch and the scanning movie.

M.
POSTED BY: Marco Thiel
9 Replies

enter image description here -- you have earned Featured Contributor Badge enter image description here Your exceptional post has been selected for our editorial column Staff Picks http://wolfr.am/StaffPicks and Your Profile is now distinguished by a Featured Contributor Badge and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: Moderation Team

Dear Mike,

thank you very much for your comment. Yes, I am aware that the FOV is rather bad. I was quite surprised that the images were as good given the large FOV. I think that the movement of the servos is about 1 degree. The sensor averages somehow over its FOV so what I actually observe should be some sort of convolution of the underlying image and some kernel.

As I said in the post, the power of Mathematica can help to improve the images. The idea is to use cheap sensors and then quite a bit of maths to improve the image. I have not had much time to look into this but one of the simple things that come to mind is using some sort of deconvolution to improve the image. Something like this:

pixels = Import["~/Desktop/Selfie.dat"];

(*Original*)

img = ListContourPlot[Flatten[Partition[ (pixels /. x_ /; x > 37. -> 37.) /. x /; x < 12. -> 12., 70][[All, 9 ;; -10]], {3}][[1]], AspectRatio -> 0.9, ColorFunction -> "Rainbow", PlotRange -> All, InterpolationOrder -> 0, Contours -> 60, ContourStyle -> None]

(*Deconvolution*)

img2 = ListContourPlot[ListDeconvolve[BoxMatrix[1.75], Flatten[Partition[ (pixels /. x_ /; x > 37. -> 37.) /. x /; x < 12. -> 12., 70][[All, 9 ;; -10]], {3}][[1]]], AspectRatio -> 0.9, ColorFunction -> "Rainbow", PlotRange -> All, InterpolationOrder -> 0, Contours -> 60, ContourStyle -> None]

enter image description here

The original is on the left and the de-convoluted image is on the right. There is clearly more structure on the image on the right. Of course, it is doubtful whether the box-type kernel is ideal. It is of course easy to choose a different, e.g. Gaussian kernel:

Manipulate[ListContourPlot[ListDeconvolve[GaussianMatrix[k], Flatten[Partition[ (pixels /. x_ /; x > 37. -> 37.) /. x /; x < 12. -> 12., 70][[All, 9 ;; -10]], {3}][[1]]], AspectRatio -> 0.9, ColorFunction -> "Rainbow", PlotRange -> All, InterpolationOrder -> 1, Contours -> 60, ContourStyle -> None], {{k, 2.83}, 1, 5}]

When I animate this it gives this movie:

enter image description here

We can also look at a couple of frames (from k=1 to 4 in steps of 0.2):

enter image description here

None of this is really satisfactory, but I suppose that with some thought and Mathematica's vast functionality one can improve the image quite a bit and compensate for the low quality of the hardware. I would be very happy to see what people here in the community can do with this image to improve the quality.

Cheers,

Marco

PS: here's a link to the data file: Selfie.dat

POSTED BY: Marco Thiel
Posted 9 years ago

Hello

I would like to ask regarding the thermal sensor. It’s FOV is 5 degree so if you take your measurement at distance of 1 meter you will measure a 10 cm. heat source resolution. Are you aware to this when you are controlling the Servo motors.

Regards, Mike M.

POSTED BY: mike mendel
This really awesome! So did the selfie took to scan also about 3 minutes and do you think body motion during scan affects the data or it is below thermal errors?
POSTED BY: Sam Carrettie
Dear Sam, 

sorry for the late reply. Yes, the selfie did take about three minutes; I had to sit still. If the person moves normally it will distroy the image. Also, the distance from the sensor influcence the reading greatly. I can later post an image of a TV screen taken at an angle and you will see that the nearer parts of the TV appear hotter than the parts that are further away. If someone moves in front of the camera, the closest position of his/her body will lead to a very high reading at that point - a kind of over exposure. 

Because of the scanning technique we need slow moving/still objects. Having said that, we have many pictures that are relatively clear; but the sensor is quite primitive. This is exactly where I think Mathematica's power comes in handy: the sensor does not measure temperature at  "points" but rather for small angles. Therefore, I will need to use some de-convolution technique for post processing. Also we can correct for the distance problem. I want to add a simple ultrasonic sensor, which might help a bit to determine the distance from the object. Also there is the projection issue that I already mentioned in the first example. 

Also, I hope that in Mathematica 10 I will be able to use inbuilt functions to read the serial port. That might speed up the data aquisition. Also Melexis (who produce the sensor) offer another sensor that measures a grid of 4 times 16 points. The problem is, of course, how to combine the pixels from different measurements, given that the servos are not very precise. On the other side using one of these sensors would potentially decrease the scannig time by a factor of 64.

Also there are some stepper motors which are much more precise. I would actually try and sample bits which higher temperature gradients better, after some initial scan. This might increase the quality of the image. 

I think that the main thing is that we can use Mathematica to mitigate some of the disadvantages of a simple, cheap sensor. Unfortunately, I do not have access to Mathematica 10 yet, but from what I heard so far I believe that there will be great features there that will help to speed this up.

M.
POSTED BY: Marco Thiel
I thought I would add some data files. If anyone has an idea of how to "un-distort" the image, so that it is correcly projected to the 2D plane or any other ideas on how to improve the image quality that would be greatly appreciated.

The following command can be used to read the data:
pixels = Flatten[Import["~/Desktop/Selfie.txt", "Data"]]

The data can be plotted just as in the notebook above:
ListContourPlot[
Partition[Reverse[Log /@ pixels /. x_ /; x > 35. -> 35.],
   70][[-1 ;; 1 ;; -1, 1 ;; -10]], AspectRatio -> 0.7,
ColorFunction -> "Rainbow", PlotRange -> All,
InterpolationOrder -> 2, Contours -> 60, ContourStyle -> None]

On the website of Szabolcs Horvat I found a  very nice function "zoom" which allows interactive zooming into the image - on this particular image it is a bit slow, but it works:

 zoom[graph_Graphics] :=
  With[{gr = First[graph],
    opt = DeleteCases[Options[graph], PlotRange -> _],
    plr = PlotRange /. Options[graph, PlotRange],
    rectangle = {Dashing[Small],
       Line[{#1, {First[#2], Last[#1]}, #2, {First[#1],
          Last[#2]}, #1}]} &},
   DynamicModule[{dragging = False, first, second, range = plr},
    Panel@EventHandler[
     Dynamic@Graphics[
       If[dragging, {gr, rectangle[first, second]}, gr],
       PlotRange -> range,
       Sequence @@ opt], {{"MouseDown",
        1} :> (first = MousePosition["Graphics"]), {"MouseDragged",
        1} :> (dragging = True;
        second = MousePosition["Graphics"]), {"MouseUp", 1} :>
       If[dragging, dragging = False;
        range = Transpose@{first, second}, range = plr]}]]]


pixels = Flatten[Import["~/Desktop/Selfie.txt", "Data"]];


ListContourPlot[
  Partition[Reverse[Log /@ pixels /. x_ /; x > 35. -> 35.],
    70][[-1 ;; 1 ;; -1, 1 ;; -10]], AspectRatio -> 0.7,
  ColorFunction -> "Rainbow", PlotRange -> All,
  InterpolationOrder -> 2, Contours -> 60,
  ContourStyle -> None] // zoom

Cheers,
M.
Attachments:
POSTED BY: Marco Thiel
Thanks Marco.  I see your point.  It looks like it would not see much until the temperature was up in the range of 500 DegC.  The flames would be visible long before that :-).
POSTED BY: Carl Lemp
It might take all the fun out of it but there is also the off-the-shelf 5 megapixel version
that connects directly to the PI.
http://www.adafruit.com/products/1567?gclid=CODj-8-Zkb4CFU4aOgodDCsA_g
  It has no IR filter so it can take infra-red images directly.
POSTED BY: Carl Lemp
Dear Carl,

that is a camera that is sensitive to a part of the infrared spectrum (near infrared,750-900 nm?) that is not (!) showing heat (thermal infrared, 10400-14000 nm?). I have one of those cameras and they are nice for monitoring, e.g. the health of plants. They do not work for heat signals, but things like checking whether your IR remote works. 

Proper thermal imaging cameras, with a high resolution and higher frame rate are usually a couple of thousands dollars. I believe that Flir offers a camera which can be attached to an iPhone and is relatively inexpensive. 

M.


PS: Here is a photo of the laptop I showed in my original post, made with the Raspberry Noir cam that you suggest.



It looks quite like an ordinary photo and no heat signatures can be seen. The advantage of this camera is that in the dark I could use IR light and take photos then. It would appear dark to the human eye but the photo would have a reasonable exposure; but no trace of a heat signal.
POSTED BY: Marco Thiel
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract