Message Boards Message Boards

0
|
8502 Views
|
5 Replies
|
5 Total Likes
View groups...
Share
Share this post:

Best way to process many timestamps?

Posted 6 years ago

Hi everyone, I need to do calculations on millions of timestamps, such as

t0 = "2011-01-11 11:30:01.321"

If I want to subtract 12 hours from the timestamp and convert the result to a list of date items, I could use DatePlus and DateList like this

DateList[DatePlus[t0, {-12, "Hour"}]]

and it takes {0.003359,{2011,1,10,23,30,1.}} on my MacBook Pro. But it is apparently a lot faster subtracting the 12 hours this way

DateList[DateList[t0] + {0, 0, 0, -12, 0, 0}]

taking only {0.000801,{2011,1,10,23,30,1.321}}, where the outer DateList conveniently adjusts the day so that the hours position shows a positive integer. The second way also keeps the fractions of a second (1.321) where the first way drops them.

I'm leaning towards doing it the second way, and am curious to know whether anyone see a downside.

POSTED BY: Gregory Lypny
5 Replies
Posted 6 years ago

Hi Hans,

Thanks for the heads up. I had thought about time zones because I did a lot of coding in LiverCode some years ago on a Canada Newswire project where timestamps were an important consideration. Fortunately, time zones are not an issue on this project because the timestamps are GMT.

Greg

POSTED BY: Gregory Lypny

Greg:

There is a side effect to all this date time stuff and it is time zone and daylight saving time. If your data is not in a UTC, then may need to take daylight saving time when it is at play. If all the date time is from one time zone, then you may need to change to UTC then subtract the 12 hours. I found the following to be consistently the fastest:

AbsoluteTiming[
 DateList[AbsoluteTime[
    Map[Function[ToExpression[#]], 
     StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]]] - 
   43200]]
{0.000216116, {2011, 1, 10, 23, 30, 1.321}}

I tried with MapAll[ToExpression, ...] It was a bit slower. If time zone and DST are an issue, let me know I think I have dealt with this issue before. Also I did not previously present the above trial due to small dislike of two conversion an AbsoluteTime and a DateList.

POSTED BY: Hans Michel

Greg: You may also try this without Module[{},functionhere] for each function

minus12[x_] := x - 12 ;
toN[y_] := ToExpression[y];
fd[d_] := DateList[
   MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]];
AbsoluteTiming[fd["2011-01-11 11:30:01.321"]]
{0.000249526, {2011, 1, 10, 23, 30, 1.321}}

with Module

minus12[x_] := Module[{}, x - 12 ];
toN[x_] := Module[{}, ToExpression[x]];
fd[d_] := Module[{}, 
   DateList[
    MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]]];
AbsoluteTiming[fd["2011-01-11 11:30:01.321"]]
{0.000253439, {2011, 1, 10, 23, 30, 1.321}}

With and Block were a bit slower.

POSTED BY: Hans Michel
Posted 6 years ago

Hey Hans,

Clever! Will do.

Thanks,

Greg

POSTED BY: Gregory Lypny

Hello Greg:

If you do not wish to roll you own then go with the second faster method. If you have time then try a simple test between the second method and rolling your own function. Such as below:

minus12[x_] := Module[{}, ToExpression[x] - 12 ];
AbsoluteTiming[
 DateList[MapAt[minus12, 
   Map[Function[ToExpression[#]], 
    StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]], 4]]]
{0.00025374, {2011, 1, 10, 23, 30, 1.321}}
POSTED BY: Hans Michel
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract