Generate_SEO_reports.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Generate Multiple Client Reports",
      "provenance": [],
      "authorship_tag": "ABX9TyO0nu50enePzBJIh3SG9I+N",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/akash-yede/Twittorials/blob/master/Generate_SEO_reports.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VA4OcKLGFrlg"
      },
      "source": [
        "<center><b><h1>Generate SEO reports</h1></b>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ikX_YYqfGQ20"
      },
      "source": [
        "Marketers can generate reports to find page speed insights of all their sites at once. This script allows you to get insights for all your sites in a single document. We will be using Python's pypeteer library to snapshot the reports for each of the sites. These images will then be consolidated to form a report using Python's docx library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fl2ZnxgmxE5l"
      },
      "source": [
        "# Installing libraries\n",
        "!pip install -U git+https://github.com/pyppeteer/pyppeteer@dev\n",
        "!pip install nest_asyncio\n",
        "!pip install python-docx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0JodcvSqxi0y"
      },
      "source": [
        "#Install the chromium driver for pypeteer\n",
        "!apt-get update\n",
        "!apt install chromium-chromedriver"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YGcxZuojIbdi"
      },
      "source": [
        "Once you have installed the necessary libraries, you should now include all the site URLs you wish to add to the report.\n",
        "\n",
        "To add additional URLs, click on the three dots on the code block, select Form, and click Add Form Field. Replace the variable_name with the new URL. To remove them, delete the variable name directly from the code block."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nYvvg3sL96IR"
      },
      "source": [
        "#@title Add additional URLs.\n",
        "url1 = \"www.example1.com\" #@param {type:\"string\"}\n",
        "url2 = \"www.example2.com\" #@param {type:\"string\"}\n",
        "url3 = \"www.example3.com\" #@param {type:\"string\"}\n",
        "url4 = \"www.example4.com\" #@param {type:\"string\"}\n",
        "url5 = \"www.example5.com\" #@param {type:\"string\"}\n"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMN4OWqEs1oe"
      },
      "source": [
        "Adding the URLs to form a list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "13WCR9PS_Fhk",
        "outputId": "124a3465-e089-4243-f3fc-49b18c4966a0"
      },
      "source": [
        "list_url = [url1, url2, url3, url4, url5]\n",
        "print(list_url)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['www.example1.com', 'www.example2.com', 'www.example3.com', 'www.example4.com', 'www.example5.com']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S13E_4Jhs6s2"
      },
      "source": [
        "Now, we will be using Python's pypeteer library to crawl through the page speed insights site. The below code block will use the above-listed sites to create individual reports. It will take a snapshot of all the reports in png format.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GT-6n16yzvMb"
      },
      "source": [
        "from pyppeteer import launch\n",
        "import asyncio\n",
        "import time\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()\n",
        "\n",
        "async def main():\n",
        "    #Launch the chromium browser. This code will run in the background.\n",
        "    browser = await launch(executablePath=\"/usr/lib/chromium-browser/chromium-browser\",args=['--no-sandbox'])\n",
        "\n",
        "    #Creating a new page on the browser\n",
        "    page = await browser.newPage()\n",
        "    await page.setViewport({ 'width': 800, 'height': 1100})\n",
        "    \n",
        "    for url in list_url:\n",
        "\n",
        "      #This code will reach the web page on the browser.\n",
        "      await page.goto('https://developers.google.com/speed/pagespeed/insights/?url='+url+'&tab=desktop')\n",
        "    \n",
        "      #Add a timeout for 120 seconds. This is the approximate time that the website will take to generate the report.\n",
        "      time.sleep(60)\n",
        "      item = list_url.index(url)\n",
        "\n",
        "      #Take a screenshot of the report and save it locally on colab.\n",
        "      await page.screenshot({'path': 'screenshot'+str(item)+'.png'})\n",
        "    \n",
        "    await browser.close()\n",
        "\n",
        "asyncio.get_event_loop().run_until_complete(main())"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5zKApcxydSf"
      },
      "source": [
        "Once we have all the snapshots saved, we'll further consolidate them to create a report. The new file will be generated and ready to download."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6NfVgyoez4Wi"
      },
      "source": [
        "import docx\n",
        "# Creating a blank word document\n",
        "mydoc = docx.Document()\n",
        "\n",
        "for i in list_url:\n",
        "  #Iterating through each URL and adding screenshots to the word file.\n",
        "  item = list_url.index(i)\n",
        "  mydoc.add_picture(\"screenshot\"+str(item)+\".png\", width=docx.shared.Inches(6), height=docx.shared.Inches(8))\n",
        "\n",
        "#Saving the word file with a new name.\n",
        "mydoc.save(\"generatedfile.docx\")"
      ],
      "execution_count": 6,
      "outputs": []
    }
  ]
}