{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "9GIt_orUtNvA"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h7AT6h2ItNvD"
      },
      "source": [
        "## Use BigQuery DataFrames to visualize COVID-19 data\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/googleapis/python-bigquery-dataframes/blob/main/notebooks/visualization/bq_dataframes_covid_line_graphs.ipynb\">\n",
        "      <img src=\"https://raw.githubusercontent.com/googleapis/python-bigquery-dataframes/refs/heads/main/third_party/logo/colab-logo.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/googleapis/python-bigquery-dataframes/tree/main/notebooks/visualization/bq_dataframes_covid_line_graphs.ipynb\">\n",
        "      <img src=\"https://raw.githubusercontent.com/googleapis/python-bigquery-dataframes/refs/heads/main/third_party/logo/github-logo.png\" width=\"32\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/googleapis/python-bigquery-dataframes/tree/main/notebooks/visualization/bq_dataframes_covid_line_graphs.ipynb\">\n",
        "      <img src=\"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTW1gvOovVlbZAIZylUtf5Iu8-693qS1w5NJw&s\" alt=\"BQ logo\" width=\"35\">\n",
        "      Open in BQ Studio\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "n-MFJQxLtNvE"
      },
      "source": [
        "## Overview\n",
        "\n",
        "The goal of this notebook is to demonstrate creating line graphs from a ~20 million-row BigQuery dataset using BigQuery DataFrames. We will first create a plain line graph using matplotlip, then we will downsample and download our data to create a graph with a line of best fit using seaborn.\n",
        "\n",
        "If you're like me, during 2020 (and/or later years) you often found yourself looking at charts like [these](https://health.google.com/covid-19/open-data/explorer/statistics) visualizing COVID-19 cases over time. For our first graph, we're going to recreate one of those charts by filtering, summing, and then graphing COVID-19 data from the United States. BigQuery DataFrame's default integration with matplotlib will get us a satisfying result for this first graph.\n",
        "\n",
        "For our second graph, though, we want to use a scatterplot with a line of best fit, something that matplotlib will not do for us automatically. So, we'll demonstrate how to downsample our data and use seaborn to make our plot. Our second graph will be of symptom-related search trends against new cases of COVID-19, so we'll see if searches for things like \"cough\" and \"fever\" are more common in the places and times where more new cases of COVID-19 occur."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ffqBzbNztNvF"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "This notebook uses the [BigQuery COVID-19 Open Data](https://pantheon.corp.google.com/marketplace/product/bigquery-public-datasets/covid19-open-data). In this dataset, each row represents a new observation of the COVID-19 situation in a particular time and place. We will use the \"new_confirmed\" column, which contains the number of new COVID-19 cases at each observation, along with the \"search_trends_cough\", \"search_trends_fever\", and \"search_trends_bruise\" columns, which are [Google Trends](https://trends.google.com/trends/) data for searches related to cough, fever, and bruises. In the first section of the notebook, we will also use the \"country_code\" and \"date\" columns to compile one data point per day for a particular country."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nf__tMR-tNvF"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* BigQuery (compute)\n",
        "\n",
        "Learn about [BigQuery compute pricing](https://cloud.google.com/bigquery/pricing#analysis_pricing_models),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_rsbkCktNvG"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "3. [Enable the BigQuery API](https://console.cloud.google.com/flows/enableapi?apiid=bigquery.googleapis.com).\n",
        "\n",
        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XZKC6iMFxmMG"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4aooKMmnxrWF"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pv5A8Tm-yC1U"
      },
      "source": [
        "#### Set the region\n",
        "\n",
        "You can also change the `REGION` variable used by BigQuery. Learn more about [BigQuery regions](https://cloud.google.com/bigquery/docs/locations#supported_locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "bk03Rt_HyGx-"
      },
      "outputs": [],
      "source": [
        "REGION = \"US\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9RWxD1btNvK"
      },
      "source": [
        "Now we are ready to use BigQuery DataFrames!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJ0gXezj2w1t"
      },
      "source": [
        "## Visualization #1: Cases over time in the US"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "xckgWno6ouHY"
      },
      "source": [
        "### Set up project and filter data"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "-uiY0hh4tNvK"
      },
      "source": [
        "First, let's do project setup. We use options to tell BigQuery DataFrames what project and what region to use for our cloud computing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "R7STCS8xB5d2"
      },
      "outputs": [],
      "source": [
        "import bigframes.pandas as bpd\n",
        "\n",
        "# Note: The project option is not required in all environments.\n",
        "# On BigQuery Studio, the project ID is automatically detected.\n",
        "bpd.options.bigquery.project = PROJECT_ID\n",
        "\n",
        "# Note: The location option is not required.\n",
        "# It defaults to the location of the first table or query\n",
        "# passed to read_gbq(). For APIs where a location can't be\n",
        "# auto-detected, the location defaults to the \"US\" location.\n",
        "bpd.options.bigquery.location = REGION\n",
        "# Improves performance by avoiding generating total row ordering\n",
        "bpd.options.bigquery.ordering_mode = \"partial\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v6FGschEowht"
      },
      "source": [
        "Next, we read the data from a publicly available BigQuery dataset. This will take ~1 minute."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "zDSwoBo1CU3G"
      },
      "outputs": [],
      "source": [
        "all_data = bpd.read_gbq(\"bigquery-public-data.covid19_open_data.covid19_open_data\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9qV2y3iHp13y"
      },
      "source": [
        "Using pandas syntax, we will select from our all_data input dataframe only those rows where the country_code is US. This is called row filtering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "UjMT_qhjf8Fu"
      },
      "outputs": [],
      "source": [
        "usa_data = all_data[all_data[\"country_code\"] == \"US\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IYCUayWkwq8c"
      },
      "source": [
        "We're only concerned with the date and the total number of confirmed cases for now, so select just those two columns as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "IaoUf57ZwrJ8"
      },
      "outputs": [],
      "source": [
        "usa_data = usa_data[[\"date\", \"new_confirmed\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94oqNRnDvGkr"
      },
      "source": [
        "### Sum data"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "TNCQWZW83U0b"
      },
      "source": [
        "`usa_data.groupby(\"date\")` will give us a groupby object that lets us perform operations on groups of rows with the same date. We call sum on that object to get the sum for each day. This process might be familiar to pandas users."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "tYDoaKgJChiq"
      },
      "outputs": [],
      "source": [
        "# numeric_only = True because we don't want to sum dates\n",
        "new_cases_usa = usa_data.groupby(\"date\").sum(numeric_only = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3jcwFPgK5BLh"
      },
      "source": [
        "### Line graph"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8GvJAgnH5Nzi"
      },
      "source": [
        "BigQuery DataFrames implements some plotting methods with the matplotlib backend. Use `DataFrame.plot.line()` to draw a simple line graph."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "gFbCgfFC2gHw"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Axes: xlabel='date', ylabel='New Cases'>"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 43540 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "new_cases_usa.plot.line(\n",
        "    rot=45,\n",
        "    ylabel=\"New Cases\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sM5-HFDx70RG"
      },
      "source": [
        "## Visualization #2: Symptom-related searches compared to new cases"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "se1b6Vf4XB9_"
      },
      "source": [
        "### Filter data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wl2o-NYMoygb"
      },
      "source": [
        "We're curious if searches for symptoms like \"cough\" and \"fever\" went up in the same times and places that new COVID-19 cases occured, compared to non-symptoms like \"bruise.\" Let's plot searches vs. new cases to see if it looks like there's a correlation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "olfnCzyg8jYi"
      },
      "source": [
        "First, we select the new cases column and the search trends we're interested in."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {
        "id": "LqqHzjty8jk0"
      },
      "outputs": [],
      "source": [
        "regional_data = all_data[all_data[\"aggregation_level\"] == 1]    # get only region level data,\n",
        "symptom_data = regional_data[[\"location_key\", \"new_confirmed\", \"search_trends_cough\", \"search_trends_fever\", \"search_trends_bruise\", \"population\", \"date\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3DlJX-k9SPk"
      },
      "source": [
        "Not all rows have data for all of these columns, so let's select only the rows that do. Finally, lets add a new column capturing new confirmed cases as a percentage of area population."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g4MeM8Oe9Q6X"
      },
      "outputs": [],
      "source": [
        "symptom_data = symptom_data.dropna()\n",
        "symptom_data = symptom_data[symptom_data[\"new_confirmed\"] > 0]\n",
        "symptom_data[\"new_cases_percent_of_pop\"] = (symptom_data[\"new_confirmed\"]  / symptom_data[\"population\"]) * 100\n",
        "\n",
        "\n",
        "# remove impossible data points\n",
        "symptom_data = symptom_data[(symptom_data[\"new_cases_percent_of_pop\"] >= 0)]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# group data up by week\n",
        "weekly_data = symptom_data.groupby([symptom_data.location_key, symptom_data.date.dt.isocalendar().week]).agg({\"new_cases_percent_of_pop\": \"sum\", \"search_trends_cough\": \"mean\", \"search_trends_fever\": \"mean\", \"search_trends_bruise\": \"mean\"})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IlXt__om9QYI"
      },
      "source": [
        "We want to use a line of best fit to make the correlation stand out. Matplotlib does not include a feature for lines of best fit, but seaborn, which is built on matplotlib, does.\n",
        "\n",
        "BigQuery DataFrames does not currently integrate with seaborn by default. So we will demonstrate how to downsample and download a DataFrame, and use seaborn on the downloaded data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T9Hub_EAXWvY"
      },
      "source": [
        "### Graph with lines of best fit"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "hoQ9TPgUPJnN"
      },
      "source": [
        "We will now use seaborn to make the plots with the lines of best fit for cough, fever, and bruise. Note that since we're working with a local pandas dataframe, you could use any other Python library or technique you're familiar with, but we'll stick to seaborn for this notebook.\n",
        "\n",
        "Seaborn will take a few minutes to calculate the lines. Since cough and fever are symptoms of COVID-19, but bruising isn't, we expect the slope of the line of best fit to be positive in the first two graphs, but not the third, indicating that there is a correlation between new COVID-19 cases and cough- and fever-related searches."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "id": "EG7qM3R18bOb"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Axes: >"
            ]
          },
          "execution_count": 59,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 61380 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import seaborn as sns\n",
        "\n",
        "# draw the graph. This might take ~30 seconds.\n",
        "sns.regplot(x=\"new_cases_percent_of_pop\", y=\"search_trends_cough\", data=weekly_data, scatter_kws={'alpha': 0.2, \"s\" :5})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "id": "5nVy61rEGaM4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Axes: >"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 60004 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# similarly, for fever\n",
        "sns.regplot(x=\"new_cases_percent_of_pop\", y=\"search_trends_fever\", data=weekly_data, scatter_kws={'alpha': 0.2, \"s\" :5})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "id": "-S1A9E3WGaYH"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Axes: >"
            ]
          },
          "execution_count": 63,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 58408 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# similarly, for bruise\n",
        "sns.regplot(\n",
        "    x=\"new_cases_percent_of_pop\",\n",
        "    y=\"search_trends_bruise\",\n",
        "    data=weekly_data,\n",
        "    scatter_kws={'alpha': 0.2, \"s\" :5}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hd2A8707Uhz2"
      },
      "source": [
        "We see that the slope of the line is positive in the graphs for cough and fever, but flat for bruise. That means that in places with increasing new cases of COVID-19, we saw increasing searches for cough and fever, but we didn't see increasing searches for unrelated symptoms like bruises. Interesting!"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Recap"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We used matplotlib to draw a line graph of COVID-19 cases over time in the USA. Then, we used downsampling to download only a portion of the available data, used seaborn to plot lines of best fit to observe corellation between COVID-19 cases and searches for related versus unrelated symptoms.\n",
        "\n",
        "Thank you for using BigQuery DataFrames!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "venv",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
