Message Boards Message Boards

Avoid truncated tweets texts when using ServiceExecute?

Posted 6 years ago

I use the Twitter command to extract tweets and do some analysis

twitter = ServiceConnect["Twitter", "New"]
listTws = twitter["TweetSearch", "Query" -> "Trump", MaxItems -> 10];

For example, the text of the first tweet can be extracted with

listTws[[1]]["Text"]

whose output is

"RT @BreitbartNews: \"Let\[CloseCurlyQuote]s turn a negative into a     
positive... I want them to say, \[OpenCurlyQuote]Look, we are not
profiting off the deaths of children. We..."

The tweet is truncated after "We..." . If I look at the original tweet (ID 966454631361589249) , the original phrase is completed.

Who can I extract the whole sentence??

Is there a way to avoid to import retweets?? They are the majority and look like "RT @name blabla"

Here I read that I need to set "tweet_mode=extended". How can I do that with Mathematica?

UPDATEI have noticed that the truncated tweets' length is 140, whereas the total length is 280. This may be related to the new feature of twitter. Is there a way to fix it?

6 Replies

@Francesco Sgarlata

The twitter update is available using

PacletUpdate["ServiceConnection_Twitter","UpdateSites"->True]
Quit

It intentionally does not do everything that the workaround I gave before did. Most times when a user retweets a long tweet the original tweet is truncated in the retweet. The workaround I gave before was grabbing the full original tweet from the metadata. In this update we decided to preserve the retweet as twitter provides it. If you want to exclude retweets entirely you can use Twitter's query language like this

twitter["TweetSearch", "Query" -> "Baseball -filter:retweets", MaxItems -> 100]
POSTED BY: Bob Sandheinrich

Good catch. Try this instead

In[108]:= 
fulltweets = 
  Lookup[Lookup[#, "retweeted_status", #] & /@ 
    ImportString[response[[2]], "RawJSON"]["statuses"], "full_text"];

In[109]:= StringLength /@ fulltweets

Out[109]= {45, 19, 206, 223, 172, 263, 240, 131, 126, 60}
POSTED BY: Bob Sandheinrich

Now it works! thanks!

The only thing is that in this way I cannot import more than 100 tweets, whereas with the Mathematica function this was possible.

You're right. One of the benefits of ServiceExecute is that we've automated things like pagination. I'm afraid there won't be a simple way to get full length tweets out of ServiceExecute until the update is released.

POSTED BY: Bob Sandheinrich

There will be a Twitter service connection update soon that retrieves longer tweets without truncating. Until then you can use this code to directly specify the query. This code is not guaranteed to be supported forever, so when the documented method you have above starts retrieving full texts - when we release our update - you should go back to using that approach.

twitter = ServiceConnect["Twitter", "New"]
response = 
  OAuthClient`rawoauthdata[twitter["ID"], 
   "https://api.twitter.com/1.1/search/tweets.json?count=10&q=Olympics&tweet_mode=extended"];
Lookup[ImportString[response[[2]], "RawJSON"]["statuses"], "full_text"]
POSTED BY: Bob Sandheinrich

Thank you Bob, but this doesn't seem to resolve the problem. The texts are still truncated.

For example, enter image description here

this may be related to "displaytextrange" which is set to be [0,140]? (see next figure)

enter image description here

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract