# Best way to process many timestamps?

Posted 6 months ago
727 Views
|
5 Replies
|
5 Total Likes
|
 Hi everyone, I need to do calculations on millions of timestamps, such as t0 = "2011-01-11 11:30:01.321" If I want to subtract 12 hours from the timestamp and convert the result to a list of date items, I could use DatePlus and DateList like this DateList[DatePlus[t0, {-12, "Hour"}]] and it takes {0.003359,{2011,1,10,23,30,1.}} on my MacBook Pro. But it is apparently a lot faster subtracting the 12 hours this way DateList[DateList[t0] + {0, 0, 0, -12, 0, 0}] taking only {0.000801,{2011,1,10,23,30,1.321}}, where the outer DateList conveniently adjusts the day so that the hours position shows a positive integer. The second way also keeps the fractions of a second (1.321) where the first way drops them.I'm leaning towards doing it the second way, and am curious to know whether anyone see a downside.
Answer
5 Replies
Sort By:
Posted 6 months ago
 Hello Greg:If you do not wish to roll you own then go with the second faster method. If you have time then try a simple test between the second method and rolling your own function. Such as below: minus12[x_] := Module[{}, ToExpression[x] - 12 ]; AbsoluteTiming[ DateList[MapAt[minus12, Map[Function[ToExpression[#]], StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]], 4]]] {0.00025374, {2011, 1, 10, 23, 30, 1.321}} 
Answer
Posted 6 months ago
 Hey Hans,Clever! Will do.Thanks,Greg
Answer
Posted 6 months ago
 Greg: You may also try this without Module[{},functionhere] for each function minus12[x_] := x - 12 ; toN[y_] := ToExpression[y]; fd[d_] := DateList[ MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]]; AbsoluteTiming[fd["2011-01-11 11:30:01.321"]] {0.000249526, {2011, 1, 10, 23, 30, 1.321}} with Module minus12[x_] := Module[{}, x - 12 ]; toN[x_] := Module[{}, ToExpression[x]]; fd[d_] := Module[{}, DateList[ MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]]]; AbsoluteTiming[fd["2011-01-11 11:30:01.321"]] {0.000253439, {2011, 1, 10, 23, 30, 1.321}} With and Block were a bit slower.
Answer
Posted 6 months ago
 Greg:There is a side effect to all this date time stuff and it is time zone and daylight saving time. If your data is not in a UTC, then may need to take daylight saving time when it is at play. If all the date time is from one time zone, then you may need to change to UTC then subtract the 12 hours. I found the following to be consistently the fastest: AbsoluteTiming[ DateList[AbsoluteTime[ Map[Function[ToExpression[#]], StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]]] - 43200]] {0.000216116, {2011, 1, 10, 23, 30, 1.321}} I tried with MapAll[ToExpression, ...] It was a bit slower. If time zone and DST are an issue, let me know I think I have dealt with this issue before. Also I did not previously present the above trial due to small dislike of two conversion an AbsoluteTime and a DateList.
Answer
Posted 6 months ago
 Hi Hans,Thanks for the heads up. I had thought about time zones because I did a lot of coding in LiverCode some years ago on a Canada Newswire project where timestamps were an important consideration. Fortunately, time zones are not an issue on this project because the timestamps are GMT.Greg
Answer
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments