Generate graph from structure of web sites

Questions
,
1

Hello

I’m searching for the way to generate graph from structure of web sites. Does anyone here knows how I can do this with Wolfram or any other language?

Any suggestion would be appreciated.

Thank you.

2

Take a look at this.

3

Thank you for your answer

I’m noob in Wolfram and i want to get output in form of nodes and edges of graph. Is it possible?

4

Hi po,

Sure, that is exactly what the webcrawler function from the link I posted does.

webcrawler[rooturl_, depth_] := Flatten[Rest[
    NestList[Union[Flatten[Thread[# -> Import[#, "Hyperlinks"]] & /@ 
         Last /@ #]] &, {"" -> rooturl}, depth]]];

Crawl www.wolfram.com to depth 1.

g = webcrawler["http://www.wolfram.com", 1];

Take the first 3 edges. -> is short for Rule which is one way to specify edges in a graph.

Take[g, 3]
 (* 
   {
     "http://www.wolfram.com" -> "http://blog.wolfram.com/?source=footer",
     "http://www.wolfram.com" -> "http://blog.wolfram.com/?source=nav", 
     "http://www.wolfram.com" -> "http://community.wolfram.com/content?curTag=jobs&source=nav"
   }
*)
5

Thank you so much.

It’s so useful.