Message Boards Message Boards

Generate graph from structure of web sites

Posted 4 years ago

Hello

I'm searching for the way to generate graph from structure of web sites. Does anyone here knows how I can do this with Wolfram or any other language?

Any suggestion would be appreciated.

Thank you.

POSTED BY: po m
4 Replies
Posted 4 years ago

Hi po,

Sure, that is exactly what the webcrawler function from the link I posted does.

webcrawler[rooturl_, depth_] := Flatten[Rest[
    NestList[Union[Flatten[Thread[# -> Import[#, "Hyperlinks"]] & /@ 
         Last /@ #]] &, {"" -> rooturl}, depth]]];

Crawl www.wolfram.com to depth 1.

g = webcrawler["http://www.wolfram.com", 1];

Take the first 3 edges. -> is short for Rule which is one way to specify edges in a graph.

Take[g, 3]
 (* 
   {
     "http://www.wolfram.com" -> "http://blog.wolfram.com/?source=footer",
     "http://www.wolfram.com" -> "http://blog.wolfram.com/?source=nav", 
     "http://www.wolfram.com" -> "http://community.wolfram.com/content?curTag=jobs&source=nav"
   }
*)
POSTED BY: Rohit Namjoshi
Posted 4 years ago

Thank you so much.

It's so useful.

POSTED BY: po m
Posted 4 years ago

Take a look at this.

POSTED BY: Rohit Namjoshi
Posted 4 years ago

Thank you for your answer

I'm noob in Wolfram and i want to get output in form of nodes and edges of graph. Is it possible?

POSTED BY: po m
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract