Best way to process many timestamps?

Posted 8 months ago
968 Views
|
5 Replies
|
5 Total Likes
|
 Hi everyone, I need to do calculations on millions of timestamps, such as t0 = "2011-01-11 11:30:01.321" If I want to subtract 12 hours from the timestamp and convert the result to a list of date items, I could use DatePlus and DateList like this DateList[DatePlus[t0, {-12, "Hour"}]] and it takes {0.003359,{2011,1,10,23,30,1.}} on my MacBook Pro. But it is apparently a lot faster subtracting the 12 hours this way DateList[DateList[t0] + {0, 0, 0, -12, 0, 0}] taking only {0.000801,{2011,1,10,23,30,1.321}}, where the outer DateList conveniently adjusts the day so that the hours position shows a positive integer. The second way also keeps the fractions of a second (1.321) where the first way drops them.I'm leaning towards doing it the second way, and am curious to know whether anyone see a downside.
5 Replies
Sort By:
Posted 8 months ago
 Hello Greg:If you do not wish to roll you own then go with the second faster method. If you have time then try a simple test between the second method and rolling your own function. Such as below: minus12[x_] := Module[{}, ToExpression[x] - 12 ]; AbsoluteTiming[ DateList[MapAt[minus12, Map[Function[ToExpression[#]], StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]], 4]]] {0.00025374, {2011, 1, 10, 23, 30, 1.321}} 
Posted 8 months ago
 Hey Hans,Clever! Will do.Thanks,Greg
Posted 8 months ago
 Greg: You may also try this without Module[{},functionhere] for each function minus12[x_] := x - 12 ; toN[y_] := ToExpression[y]; fd[d_] := DateList[ MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]]; AbsoluteTiming[fd["2011-01-11 11:30:01.321"]] {0.000249526, {2011, 1, 10, 23, 30, 1.321}} with Module minus12[x_] := Module[{}, x - 12 ]; toN[x_] := Module[{}, ToExpression[x]]; fd[d_] := Module[{}, DateList[ MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]]]; AbsoluteTiming[fd["2011-01-11 11:30:01.321"]] {0.000253439, {2011, 1, 10, 23, 30, 1.321}} With and Block were a bit slower.
 Greg:There is a side effect to all this date time stuff and it is time zone and daylight saving time. If your data is not in a UTC, then may need to take daylight saving time when it is at play. If all the date time is from one time zone, then you may need to change to UTC then subtract the 12 hours. I found the following to be consistently the fastest: AbsoluteTiming[ DateList[AbsoluteTime[ Map[Function[ToExpression[#]], StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]]] - 43200]] {0.000216116, {2011, 1, 10, 23, 30, 1.321}} I tried with MapAll[ToExpression, ...] It was a bit slower. If time zone and DST are an issue, let me know I think I have dealt with this issue before. Also I did not previously present the above trial due to small dislike of two conversion an AbsoluteTime and a DateList.