Group Abstract Group Abstract

Message Boards Message Boards

0
|
10K Views
|
5 Replies
|
5 Total Likes
View groups...
Share
Share this post:

Best way to process many timestamps?

Gregory Lypny
Posted 8 years ago

Hi everyone, I need to do calculations on millions of timestamps, such as

t0 = "2011-01-11 11:30:01.321"

If I want to subtract 12 hours from the timestamp and convert the result to a list of date items, I could use DatePlus and DateList like this

DateList[DatePlus[t0, {-12, "Hour"}]]

and it takes {0.003359,{2011,1,10,23,30,1.}} on my MacBook Pro. But it is apparently a lot faster subtracting the 12 hours this way

DateList[DateList[t0] + {0, 0, 0, -12, 0, 0}]

taking only {0.000801,{2011,1,10,23,30,1.321}}, where the outer DateList conveniently adjusts the day so that the hours position shows a positive integer. The second way also keeps the fractions of a second (1.321) where the first way drops them.

I'm leaning towards doing it the second way, and am curious to know whether anyone see a downside.
POSTED BY: Gregory Lypny
5 Replies
Sort By:
Gregory Lypny
Gregory Lypny
Posted 8 years ago

Hi Hans,

Thanks for the heads up. I had thought about time zones because I did a lot of coding in LiverCode some years ago on a Canada Newswire project where timestamps were an important consideration. Fortunately, time zones are not an issue on this project because the timestamps are GMT.

Greg
POSTED BY: Gregory Lypny
POSTED BY: Hans Michel

Greg: You may also try this without Module[{},functionhere] for each function

minus12[x_] := x - 12 ;
toN[y_] := ToExpression[y];
fd[d_] := DateList[
   MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]];
AbsoluteTiming[fd["2011-01-11 11:30:01.321"]]
{0.000249526, {2011, 1, 10, 23, 30, 1.321}}

with Module

minus12[x_] := Module[{}, x - 12 ];
toN[x_] := Module[{}, ToExpression[x]];
fd[d_] := Module[{}, 
   DateList[
    MapAt[minus12, Map[toN, StringSplit[d, {"-", " ", ":"}]], 4]]];
AbsoluteTiming[fd["2011-01-11 11:30:01.321"]]
{0.000253439, {2011, 1, 10, 23, 30, 1.321}}

With and Block were a bit slower.
POSTED BY: Hans Michel
Gregory Lypny
Gregory Lypny
Posted 8 years ago

Hey Hans,

Clever! Will do.

Thanks,

Greg
POSTED BY: Gregory Lypny

Hello Greg:

If you do not wish to roll you own then go with the second faster method. If you have time then try a simple test between the second method and rolling your own function. Such as below:

minus12[x_] := Module[{}, ToExpression[x] - 12 ];
AbsoluteTiming[
 DateList[MapAt[minus12, 
   Map[Function[ToExpression[#]], 
    StringSplit["2011-01-11 11:30:01.321", {"-", " ", ":"}]], 4]]]
{0.00025374, {2011, 1, 10, 23, 30, 1.321}}
POSTED BY: Hans Michel
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard