Skip to main content
Engineering LibreTexts

16.2: Visualizing Networks and Interconnections

  • Page ID
  • \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

    \( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

    ( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

    \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

    \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

    \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

    \( \newcommand{\Span}{\mathrm{span}}\)

    \( \newcommand{\id}{\mathrm{id}}\)

    \( \newcommand{\Span}{\mathrm{span}}\)

    \( \newcommand{\kernel}{\mathrm{null}\,}\)

    \( \newcommand{\range}{\mathrm{range}\,}\)

    \( \newcommand{\RealPart}{\mathrm{Re}}\)

    \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

    \( \newcommand{\Argument}{\mathrm{Arg}}\)

    \( \newcommand{\norm}[1]{\| #1 \|}\)

    \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

    \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

    \( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

    \( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

    \( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vectorC}[1]{\textbf{#1}} \)

    \( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

    \( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

    \( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

    \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

    In this application, we will perform some of the functions of a search engine. We will first spider a small subset of the web and run a simplified version of the Google page rank algorithm to determine which pages are most highly connected, and then visualize the page rank and connectivity of our small corner of the web. We will use the D3 JavaScript visualization library to produce the visualization output.

    You can download and extract this application from:

    A Page Ranking

    A Page Ranking

    The first program ( program crawls a web site and pulls a series of pages into the database (spider.sqlite), recording the links between pages. You can restart the process at any time by removing the spider.sqlite file and rerunning

    Enter web url or enter:
    How many pages:2
    1 12
    2 57
    How many pages:

    In this sample run, we told it to crawl a website and retrieve two pages. If you restart the program and tell it to crawl more pages, it will not re-crawl any pages already in the database. Upon restart it goes to a random non-crawled page and starts there. So each successive run of is additive.

    Enter web url or enter:
    How many pages:3
    3 57
    4 1
    5 13
    How many pages:

    You can have multiple starting points in the same database—within the program, these are called "webs". The spider chooses randomly amongst all non-visited links across all the webs as the next page to spider.

    If you want to dump the contents of the spider.sqlite file, you can run as follows:

    (5, None, 1.0, 3, '')
    (3, None, 1.0, 4, '')
    (1, None, 1.0, 2, '')
    (1, None, 1.0, 5, '')
    4 rows.

    This shows the number of incoming links, the old page rank, the new page rank, the id of the page, and the url of the page. The program only shows pages that have at least one incoming link to them.

    Once you have a few pages in the database, you can run page rank on the pages using the program. You simply tell it how many page rank iterations to run.

    How many iterations:2
    1 0.546848992536
    2 0.226714939664
    [(1, 0.559), (2, 0.659), (3, 0.985), (4, 2.135), (5, 0.659)]

    You can dump the database again to see that page rank has been updated:

    (5, 1.0, 0.985, 3, '')
    (3, 1.0, 2.135, 4, '')
    (1, 1.0, 0.659, 2, '')
    (1, 1.0, 0.659, 5, '')
    4 rows.

    You can run as many times as you like and it will simply refine the page rank each time you run it. You can even run a few times and then go spider a few more pages sith and then run to reconverge the page rank values. A search engine usually runs both the crawling and ranking programs all the time.

    If you want to restart the page rank calculations without respidering the web pages, you can use and then restart

    How many iterations:50
    1 0.546848992536
    2 0.226714939664
    3 0.0659516187242
    4 0.0244199333
    5 0.0102096489546
    6 0.00610244329379
    42 0.000109076928206
    43 9.91987599002e-05
    44 9.02151706798e-05
    45 8.20451504471e-05
    46 7.46150183837e-05
    47 6.7857770908e-05
    48 6.17124694224e-05
    49 5.61236959327e-05
    50 5.10410499467e-05
    [(512, 0.0296), (1, 12.79), (2, 28.93), (3, 6.808), (4, 13.46)]

    For each iteration of the page rank algorithm it prints the average change in page rank per page. The network initially is quite unbalanced and so the individual page rank values change wildly between iterations. But in a few short iterations, the page rank converges. You should run long enough that the page rank values converge.

    If you want to visualize the current top pages in terms of page rank, run to read the database and write the data for the most highly linked pages in JSON format to be viewed in a web browser.

    Creating JSON output on spider.json...
    How many nodes? 30
    Open force.html in a browser to view the visualization

    You can view this data by opening the file force.html in your web browser. This shows an automatic layout of the nodes and links. You can click and drag any node and you can also double-click on a node to find the URL that is represented by the node.

    If you rerun the other utilities, rerun and press refresh in the browser to get the new data from spider.json.

    This page titled 16.2: Visualizing Networks and Interconnections is shared under a CC BY-NC-SA license and was authored, remixed, and/or curated by Chuck Severance.

    • Was this article helpful?