Message Boards Message Boards

Playing with YouTube from Mathematica

Posted 6 years ago

Originally here (the embedded iframes actually work there)


Playing with YouTube from Mathematica

Today I'm gonna quickly show off something I made for a friend (and about which I'd been thinking for some time). We're gonna use the YouTube API in Mathematica to do a have some fun (but not too much fun, mind you!).

Installing the YouTube Connection

My connection to the YouTube API is built on Mathematica's ServiceConnect framework. I've talked about these before in many different contexts but today we'll just use it. To install we pull the paclet off my server :

PacletInstall["ServiceConnection_YouTube",
  "Site"->
    "http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer/"
  ]

(*Out:*)

post22-8529256040837869960

Authenticating with YouTube

One installed, we connect to the service as usual:

$so = ServiceConnect["YouTube"]

This will open a dialog:

post22-2534758878895670558

We'll click the sign in button, pick and account (I'll use my new personal account steve.wolfraum@gmail.com ):

post22-6891487155877384696

When we click "Allow" either we'll be taken to a nicely formatted local page with an access code running via HTTPHandling`StartWebServer or if that's being glitchy as it was right now, we'll copy the part of the URL starting with "code=4/..." :

post22-5755985516881850876

Finally, we're connected:

(*Out:*)

post22-6049558537553480280

Getting Data

The majority of the currently supported methods have to do with getting data about YouTube:

$so["Requests"]

(*Out:*)

{"Authentication","DeleteVideo","ID","Information","LastRequest","ListCaptions","ListChannelActivity","ListChannels","ListChannelSections","ListComments","ListCommentThreads","ListPlaylistItems","ListPlaylists","ListSubscriptions","ListVideos","Name","RawRequests","RequestData","RequestParameters","Search","UpdateVideo","UploadVideo","VideoURL"}

We'll start with a search, maybe. First let's see what parameters we have to work with:

$so["RequestParameters", "Request"->"Search"]

(*Out:*)

<|"Parameters"->{"q","part","forContentOwner","forDeveloper","forMine","relatedToVideoId","channelId","channelType","eventType","location","locationRadius","maxResults","onBehalfOfContentOwner","order","pageToken","publishedAfter","publishedBefore","q","regionCode","relevanceLanguage","safeSearch","topicId","type","videoCaption","videoCategoryId","videoDefinition","videoDimension","videoDuration","videoEmbeddable","videoLicense","videoSyndicated","videoType"},"Required"->{"part"}|>

Then we'll search for something bland, like "Steve Wolfraum". The YouTube API requires this "part" parameter to know what to return to use. I like to get the "id" and "snippet" parts, but you can pick for yourself which parts you want. Just comma-separate your values and you can get multiple different ones:

mySearch = $so["Search", "q"->"Steve Wolfraum", "part"->"id,snippet"]

(*Out:*)

post22-4316213154658629458

We see there are tons of results for "my" name but we only get them five at a time. We can use the "nextPageToken" to get the next page of results, though. We can bump up the number we get with the "maxResults" parameter, but I see no need for that right now. Let's look at the first result here. I'll drill straight into that snippet parameter, as that's the most interesting part:

mySearch["items", 1, "snippet"]

(*Out:*)

post22-498969337448306280

And we see, sadly, that instead of getting results for my name, Steve Wolfraum, we get them for some random dude name Stephen Wolfram. Not to be deterred, let's at least get something out of this and see a thumbnail for all our results. Tragically, we can't download videos with the YouTube API, but at least we get thumbnails...hooray? To make things more fun, we'll also add a filter to all these results, as the raw thumbnails really aren't all that interesting.

ImageAdjust@LaplacianGaussianFilter[Import[#], 3]&/@
  Normal@mySearch["items", All, "snippet", "thumbnails", "high", "url"]

(*Out:*)

post22-8416895123856568403

Cool. There's some data. We can also pull out statistics and things, say for different videos broadcast on the Wolfram Research channel. First we'll get their channel ID:

wriChannel=
  $so["Search", "q"->"Wolfram Research", "type"->"channel",
    "part"->"id", "maxResults"->"1"]["items", 1, "id", "channelId"]

(*Out:*)

"UCJekgf6k62CQHdENWf2NgAQ"

Now we'll use this to extract the 50 most popular videos:

wriVids=$so["Search", "channelId"->wriChannel, "type"->"video", 
  "order"->"viewCount", "part"->"id,snippet", "maxResults"->"50"]

(*Out:*)

post22-5403531027782801587

And now we'll get view counts:

wriStats=$so["ListVideos", 
  "id"->StringRiffle[Normal@wriVids["items", All, "id", "videoId"], ","], 
  "part"->"statistics"
  ]

(*Out:*)

post22-4181911101890639189

And then we'll make a WordCloud of titles vs viewcounts:

WordCloud@
  AssociationThread[
    Normal@wriVids["items", All, "snippet", "title"],
    ToExpression@Normal@wriStats["items", All, "statistics", "viewCount"]
    ]

(*Out:*)

post22-948349109694102953

And we see as far as YouTube cares, there's really only one video WRI has made. But dropping that one we might see some more interesting structure:

WordCloud@
  AssociationThread[
    Rest@Normal@wriVids["items", All, "snippet", "title"],
    Rest@ToExpression@Normal@wriStats["items", All, "statistics", "viewCount"]
    ]

(*Out:*)

post22-978266547142309807

And it seems that people really like the basics and...really like seismic waves?

I'll embed the first of those so we can see how good the really are:

$so["EmbedIFrame", "id"->wriVids["items", 3, "id", "videoId"]]

(* Out *)
<iframe id="ytplayer" type="text/html" width="640" height="360"
src="https://www.youtube.com/embed/2rYjlVPU9U4?autoplay=0"
  frameborder="0"></iframe>

Uploading Videos

A YouTube API wouldn't be much of a YouTube API if it couldn't upload videos, of course, so I cooked that in as well.

Here's a sample of that. We'll first pull an animation off the Manipulate docs:

anim =
  Export[
    FileNameJoin@{$TemporaryDirectory, "plot3d.flv"},
    Manipulate[
      Plot3D[Sin[n x y], 
        {x, 0, 3}, {y, 0, 3}, 
        ViewPoint -> Dynamic[{2, v, 2}], SphericalRegion -> True, Ticks -> None,
        PerformanceGoal->"Quality"
        ], 
      {n, 1, 4}, 
      {v, -2, 2}
      ]
    ];

Then we'll upload the video we made:

upload = $so["UploadVideo", "part"->"id", "BodyContent"->anim];

Then we attach a title and things in a second request:

$so["UpdateVideo", "id" -> upload["id"],
 "part" -> "snippet",
  "Title" -> "Mathematica Examples: Plot 3D",
 "CategoryID" -> "22"
 ]

(*Out:*)

post22-4324612576805415951

And then now we'll embed that video we just made:

$so["EmbedIFrame", "id"->upload["id"]]

(* Out *)
<iframe id="ytplayer" type="text/html" width="640" height="360"
src="https://www.youtube.com/embed/Quzcr4Vuq_Q?autoplay=0"
  frameborder="0"></iframe>
POSTED BY: b3m2a1 ​ 
19 Replies
Posted 2 years ago

The Wolfram cloud links in this post give 403 (forbidden).

POSTED BY: julian

After playing with this paclet for a while I got a failure from YouTube indicating that the API had exceeded its allowed "quota".

Read more about it here: https://developers.google.com/youtube/v3/getting-started

It sounds like any YouTube app / API comes with a default amount of "quota" -- it indicates "10,000 units per day".

  • Each video upload is 1,600 units of quota.
  • Each write is 50 units.
  • Each simple read is 1 unit.

Presumably anyone that uses your paclet chews through the quota for your "app", "MathematicaLink", until the max daily quota is hit.

I suppose ideally the paclet would allow a user to substitute their own Google "app" / API in place of your "MathematicaLink", so that they could manage their own quota, etc.

As for me, I think I'll be OK -- I'll just wait until tomorrow to keep playing with the API, but figured I'd let you know about this incase it was of interest.

POSTED BY: Daniel Bigham
Posted 5 years ago

I'm well aware of that, but mostly I don't care. This was a fun side project for me, but I didn't sink enough time into it to make it worth figuring out some easy way for others to supply their own credentials inside the link. I've done that for my GoogleAPI extension stack in BTools, but the ServiceConnect framework is clunky.

In any case, if you want to use your own API credentials it's actually pretty easy. They're cooked directly into the paclet. Go to YouTube.m in Kernel and swap them out in youtubeclientid and youtubeclientsecret. I'm violating the TOS by directly exposing them in the first place so you'd be doing me a favor.

POSTED BY: b3m2a1 ​ 

Thanks for that info

Thanks, that solved the error.

Now I'm getting a response that contains an error:

{<|"domain" -> "global", "reason" -> "authError", "message" -> "Invalid Credentials", "locationType" -> "header", "location" -> "Authorization"|>}

When I ran:

$so = ServiceConnect["YouTube"]

... a couple of days ago, I followed your above instructions and checked the box for it to remember. When I evaluate $so = ServiceConnect["YouTube"] now, it doesn't launch the web browser but instead immediately returns a ServiceObject[...], that when used, results in the above error.

POSTED BY: Daniel Bigham
Posted 5 years ago

That looks like a ServiceConnect issue, not one on my end. Try using ServiceDisconnect on the object.

POSTED BY: b3m2a1 ​ 

Did ServiceDisconnect[$so].

Then re-ran ServiceConnect["YouTube"] and got a ServiceObject showing "Connected". (didn't redirect me to the web browser, just evaluated immediately)

Did $so["UploadVideo", ...] and got same failure.

POSTED BY: Daniel Bigham
Posted 5 years ago

If it didn’t redirect it’s not connected. Maybe try ServiceConnect[..., "New"]? Past that I don’t really know what’s wrong. If it continues I’d ask the team in charge of service connect because it is something on their end.

POSTED BY: b3m2a1 ​ 

That did the trick. Thanks, and well done!

POSTED BY: Daniel Bigham

Thank for your reply. Wrapping it in File[...] led to the same result. Is there a particular source file and line that I should go to to inspect how you tried to work around this?

POSTED BY: Daniel Bigham
Posted 5 years ago

Turns out the issue is here: https://github.com/b3m2a1/mathematica-BTools/blob/eef23617d0fc565745cabf79cfb7b38000a5cb8f/Resources/Templates/Frameworks/%24ServiceConnection/Kernel/%24ServiceConnectionFunctions.m#L882

Mathematica doesn't know how to handle MP4 as a "MIMEType", but it looks like I set it up so you can specify your own as an argument, try adding "MIMEType"->the/right/mimetype" as an argument to the function. Alternately overload ImportExport`GetMIMEType to understand MP4

POSTED BY: b3m2a1 ​ 

I was delighted to see that code already exists to upload videos to YouTube from WL, so I gave it a try with an MP4 file, and got this: enter image description here

POSTED BY: Daniel Bigham
Posted 5 years ago

Ah looks like it's trying to treat the "BodyContent" as a file to be exported first... let me think... I'm reasonably sure you can use something like "file" as the upload keyword but I can't remember. Want to check what the "RequestParameters" for this are? The right keyword will be somewhere in there.

Edit:

Nevermind, I have to use the appropriate multipart stuff to upload it but it looks like my check to see if "BodyContent" is a file depends on Mathematica knowing it's a file but Mathematica is still stupid about MP4. Try wrapping it in File and if that fails take a look inside the paclet to see exactly how I tried to work around this.

POSTED BY: b3m2a1 ​ 

"but you can get a link to the actual URL on YouTube where the video will display or use some YouTube downloader with that URL to get the video." Exactly that is what I want. How to do it?

Posted 6 years ago

With a response dataset you can get the ID for the video or list IDs for a list of videos by inspecting the response dataset structure and using the appropriate extractors.

Then you use the request $so["VideoURL", "id"->id] to get the display URL. You can map this over the IDs if you want to get a bunch of URLs.

POSTED BY: b3m2a1 ​ 

Thanks for the instruction. I verified that it works on V11.3 win 10.

enter image description here

POSTED BY: Shenghui Yang

Hats off!! really impressive. Question: isn't there a way simply to get all the urls of your query? This would be quite useful. Even if they can't be played from Mathematica...

Posted 6 years ago

It depends on what you mean by the URLs. You can't download YouTube videos with the API, but you can get a link to the actual URL on YouTube where the video will display or use some YouTube downloader with that URL to get the video.

POSTED BY: b3m2a1 ​ 

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

POSTED BY: EDITORIAL BOARD
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract