{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2024 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing package downloads from PyPI with BigQuery DataFrames\n",
    "\n",
    "In this notebook, you'll use the [PyPI public dataset](https://console.cloud.google.com/marketplace/product/gcp-public-data-pypi/pypi) and the [deps.dev public dataset](https://deps.dev/) to visualize Python package downloads for a package and its dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a package which you want to visualize.\n",
    "package_name = \"pandas\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bigframes.pandas as bpd\n",
    "\n",
    "# Use `ordering_mode=\"partial\"` for more efficient query generation, but\n",
    "# some pandas-compatible methods may not be possible without a total ordering.\n",
    "bpd.options.bigquery.ordering_mode = \"partial\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting downloads and tracking dependencies\n",
    "\n",
    "The [PyPI `file_downloads`](https://console.cloud.google.com/bigquery?ws=!1m5!1m4!4m3!1sbigquery-public-data!2spypi!3sfile_downloads) table contains a row for each time there is a download request for a package. The [deps.dev Dependencies](https://console.cloud.google.com/bigquery?ws=!1m5!1m4!4m3!1sbigquery-public-data!2sdeps_dev_v1!3sDependencies) table contains a row for each dependency of each package.\n",
    "\n",
    "When `ordering_mode = \"partial\"`, `read_gbq_table` creates a DataFrame representing the table, but the DataFrame has no native ordering or index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bigframes.enums\n",
    "\n",
    "# Without ordering_mode = \"partial\" it is recommended that you set\n",
    "# the \"filters\" parameter to limit the number of rows subsequent queries\n",
    "# have to read.\n",
    "pypi = bpd.read_gbq_table(\n",
    "    \"bigquery-public-data.pypi.file_downloads\",\n",
    "\n",
    "    # Using ordering_mode = \"partial\" changes the default index to a \"NULL\"\n",
    "    # index, meaning no index is available for implicit joins.\n",
    "    #\n",
    "    # Setting this explicitly avoids a DefaultIndexWarning.\n",
    "    index_col=bigframes.enums.DefaultIndexKind.NULL,\n",
    ")\n",
    "deps = bpd.read_gbq_table(\n",
    "    \"bigquery-public-data.deps_dev_v1.Dependencies\",\n",
    "    index_col=bigframes.enums.DefaultIndexKind.NULL,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limit to the most recent 7 days of data\n",
    "\n",
    "The PyPI and deps.dev tables are partitioned by date. Query only the most recent 7 days of data to reduce the number of bytes scanned.\n",
    "\n",
    "Just as with the default ordering mode, filters can be describe in a pandas-compatible way by passing a Boolean Series to the DataFrame's `__getitem__` accessor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "now = datetime.datetime.now(datetime.timezone.utc)\n",
    "last_7_days = now - datetime.timedelta(days=7)\n",
    "last_30_days = now - datetime.timedelta(days=30)\n",
    "pypi = pypi[pypi[\"timestamp\"] > last_7_days]\n",
    "deps = deps[deps[\"SnapshotAt\"] > last_30_days]  # deps are refreshed less frequently\n",
    "deps = deps[deps[\"System\"] == \"PYPI\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**⚠ Warning**\n",
    "\n",
    "Without `ordering_mode = \"partial\"`, these filters do not change the number of bytes scanned. Instead, add column and row filters at \"read\" time. For example,\n",
    "\n",
    "```\n",
    "import datetime\n",
    "\n",
    "last_7_days = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=7)\n",
    "\n",
    "# Without ordering_mode = \"partial\", one must limit the data at \"read\" time to reduce bytes scanned.\n",
    "pypi = bpd.read_gbq_table(\n",
    "    \"bigquery-public-data.pypi.file_downloads\",\n",
    "    columns=[\"timestamp\", \"project\"],\n",
    "    filters=[(\"timestamp\", \">\", last_7_days)],\n",
    ")\n",
    "```\n",
    "\n",
    "`head()` is not available when no ordering has been established. It fails with `OrderRequiredError`. Use `peek()` instead to download a sample of the data. This will be much more efficient, as the query doesn't need to order all rows to determine which are first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Query job 423d9d93-1495-4c76-b8c2-e830a6e19ff4 is DONE. 110.3 MB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:423d9d93-1495-4c76-b8c2-e830a6e19ff4&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>country_code</th>\n",
       "      <th>url</th>\n",
       "      <th>project</th>\n",
       "      <th>file</th>\n",
       "      <th>details</th>\n",
       "      <th>tls_protocol</th>\n",
       "      <th>tls_cipher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2024-09-18 18:15:04+00:00</td>\n",
       "      <td>US</td>\n",
       "      <td>/packages/ff/c8/4cd4b2834012ffc71ae3fd69187f08...</td>\n",
       "      <td>aiobreaker</td>\n",
       "      <td>{'filename': 'aiobreaker-1.2.0-py3-none-any.wh...</td>\n",
       "      <td>{'installer': {'name': 'pip', 'version': '21.1...</td>\n",
       "      <td>TLSv1.3</td>\n",
       "      <td>TLS_AES_128_GCM_SHA256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2024-09-18 18:29:50+00:00</td>\n",
       "      <td>US</td>\n",
       "      <td>/packages/21/8e/4562029e179226051cd4aa3135444d...</td>\n",
       "      <td>aiobotocore</td>\n",
       "      <td>{'filename': 'aiobotocore-1.3.0.tar.gz', 'proj...</td>\n",
       "      <td>{'installer': {'name': 'pip', 'version': '24.1...</td>\n",
       "      <td>TLSv1.2</td>\n",
       "      <td>ECDHE-RSA-AES128-GCM-SHA256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2024-09-18 18:22:14+00:00</td>\n",
       "      <td>US</td>\n",
       "      <td>/packages/11/16/4226e59bb72e096d9809ccedf349a1...</td>\n",
       "      <td>aiobotocore</td>\n",
       "      <td>{'filename': 'aiobotocore-2.0.1.tar.gz', 'proj...</td>\n",
       "      <td>{'installer': {'name': 'pip', 'version': '24.2...</td>\n",
       "      <td>TLSv1.2</td>\n",
       "      <td>ECDHE-RSA-AES128-GCM-SHA256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2024-09-18 18:22:08+00:00</td>\n",
       "      <td>US</td>\n",
       "      <td>/packages/11/16/4226e59bb72e096d9809ccedf349a1...</td>\n",
       "      <td>aiobotocore</td>\n",
       "      <td>{'filename': 'aiobotocore-2.0.1.tar.gz', 'proj...</td>\n",
       "      <td>{'installer': {'name': 'pip', 'version': '24.2...</td>\n",
       "      <td>TLSv1.2</td>\n",
       "      <td>ECDHE-RSA-AES128-GCM-SHA256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2024-09-18 18:29:22+00:00</td>\n",
       "      <td>US</td>\n",
       "      <td>/packages/54/b7/453119271cc4c36b07fdeab9b0ff25...</td>\n",
       "      <td>aiobotocore</td>\n",
       "      <td>{'filename': 'aiobotocore-2.3.3.tar.gz', 'proj...</td>\n",
       "      <td>{'installer': {'name': 'pip', 'version': '24.1...</td>\n",
       "      <td>TLSv1.2</td>\n",
       "      <td>ECDHE-RSA-AES128-GCM-SHA256</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   timestamp country_code  \\\n",
       "0  2024-09-18 18:15:04+00:00           US   \n",
       "1  2024-09-18 18:29:50+00:00           US   \n",
       "2  2024-09-18 18:22:14+00:00           US   \n",
       "3  2024-09-18 18:22:08+00:00           US   \n",
       "4  2024-09-18 18:29:22+00:00           US   \n",
       "\n",
       "                                                 url      project  \\\n",
       "0  /packages/ff/c8/4cd4b2834012ffc71ae3fd69187f08...   aiobreaker   \n",
       "1  /packages/21/8e/4562029e179226051cd4aa3135444d...  aiobotocore   \n",
       "2  /packages/11/16/4226e59bb72e096d9809ccedf349a1...  aiobotocore   \n",
       "3  /packages/11/16/4226e59bb72e096d9809ccedf349a1...  aiobotocore   \n",
       "4  /packages/54/b7/453119271cc4c36b07fdeab9b0ff25...  aiobotocore   \n",
       "\n",
       "                                                file  \\\n",
       "0  {'filename': 'aiobreaker-1.2.0-py3-none-any.wh...   \n",
       "1  {'filename': 'aiobotocore-1.3.0.tar.gz', 'proj...   \n",
       "2  {'filename': 'aiobotocore-2.0.1.tar.gz', 'proj...   \n",
       "3  {'filename': 'aiobotocore-2.0.1.tar.gz', 'proj...   \n",
       "4  {'filename': 'aiobotocore-2.3.3.tar.gz', 'proj...   \n",
       "\n",
       "                                             details tls_protocol  \\\n",
       "0  {'installer': {'name': 'pip', 'version': '21.1...      TLSv1.3   \n",
       "1  {'installer': {'name': 'pip', 'version': '24.1...      TLSv1.2   \n",
       "2  {'installer': {'name': 'pip', 'version': '24.2...      TLSv1.2   \n",
       "3  {'installer': {'name': 'pip', 'version': '24.2...      TLSv1.2   \n",
       "4  {'installer': {'name': 'pip', 'version': '24.1...      TLSv1.2   \n",
       "\n",
       "                    tls_cipher  \n",
       "0       TLS_AES_128_GCM_SHA256  \n",
       "1  ECDHE-RSA-AES128-GCM-SHA256  \n",
       "2  ECDHE-RSA-AES128-GCM-SHA256  \n",
       "3  ECDHE-RSA-AES128-GCM-SHA256  \n",
       "4  ECDHE-RSA-AES128-GCM-SHA256  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Warning: Ensure bpd.options.bigquery.ordering_mode = \"partial\" or else\n",
    "# this query() will cause a full table scan because of the sequential index.\n",
    "assert bpd.options.bigquery.ordering_mode == \"partial\"\n",
    "pypi.peek()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Query job 3a421217-59e2-4722-8382-0930f0a3b9ee is DONE. 1.5 MB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:3a421217-59e2-4722-8382-0930f0a3b9ee&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SnapshotAt</th>\n",
       "      <th>System</th>\n",
       "      <th>Name</th>\n",
       "      <th>Version</th>\n",
       "      <th>Dependency</th>\n",
       "      <th>MinimumDepth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2024-08-29 04:39:16.121656+00:00</td>\n",
       "      <td>PYPI</td>\n",
       "      <td>zxkane-cdk-construct-simple-nat</td>\n",
       "      <td>0.2.89</td>\n",
       "      <td>{'System': 'PYPI', 'Name': 'attrs', 'Version':...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2024-08-29 04:39:16.121656+00:00</td>\n",
       "      <td>PYPI</td>\n",
       "      <td>zxkane-cdk-construct-simple-nat</td>\n",
       "      <td>0.2.82</td>\n",
       "      <td>{'System': 'PYPI', 'Name': 'attrs', 'Version':...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2024-08-29 04:39:16.121656+00:00</td>\n",
       "      <td>PYPI</td>\n",
       "      <td>zxkane-cdk-construct-simple-nat</td>\n",
       "      <td>0.2.88</td>\n",
       "      <td>{'System': 'PYPI', 'Name': 'attrs', 'Version':...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2024-08-29 04:39:16.121656+00:00</td>\n",
       "      <td>PYPI</td>\n",
       "      <td>zxkane-cdk-construct-simple-nat</td>\n",
       "      <td>0.2.91</td>\n",
       "      <td>{'System': 'PYPI', 'Name': 'attrs', 'Version':...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2024-08-29 04:39:16.121656+00:00</td>\n",
       "      <td>PYPI</td>\n",
       "      <td>zxkane-cdk-construct-simple-nat</td>\n",
       "      <td>0.2.77</td>\n",
       "      <td>{'System': 'PYPI', 'Name': 'attrs', 'Version':...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         SnapshotAt System                             Name  \\\n",
       "0  2024-08-29 04:39:16.121656+00:00   PYPI  zxkane-cdk-construct-simple-nat   \n",
       "1  2024-08-29 04:39:16.121656+00:00   PYPI  zxkane-cdk-construct-simple-nat   \n",
       "2  2024-08-29 04:39:16.121656+00:00   PYPI  zxkane-cdk-construct-simple-nat   \n",
       "3  2024-08-29 04:39:16.121656+00:00   PYPI  zxkane-cdk-construct-simple-nat   \n",
       "4  2024-08-29 04:39:16.121656+00:00   PYPI  zxkane-cdk-construct-simple-nat   \n",
       "\n",
       "  Version                                         Dependency  MinimumDepth  \n",
       "0  0.2.89  {'System': 'PYPI', 'Name': 'attrs', 'Version':...             2  \n",
       "1  0.2.82  {'System': 'PYPI', 'Name': 'attrs', 'Version':...             2  \n",
       "2  0.2.88  {'System': 'PYPI', 'Name': 'attrs', 'Version':...             2  \n",
       "3  0.2.91  {'System': 'PYPI', 'Name': 'attrs', 'Version':...             2  \n",
       "4  0.2.77  {'System': 'PYPI', 'Name': 'attrs', 'Version':...             2  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deps.peek()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find dependencies for pandas\n",
    "\n",
    "Use assign to add columns to the DataFrame after a scalar operations, such as extracting a sub-field from a `STRUCT` column.\n",
    "\n",
    "Because the DataFrame has no index, this does not work if the new column belongs to a different table expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "deps = deps.assign(DependencyName=deps[\"Dependency\"].struct.field(\"Name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use an aggregation to identify the unique `DependencyName`s for the `pandas` package. Note: `drop_duplicates()` is not supported, as the order-based behavior such as `keep=\"first\"` is not applicable when using `ordering_mode = \"partial\"`.\n",
    "\n",
    "A DataFrame with no index still supports aggregation operations. Set `as_index=False` to keep the GROUP BY keys as regular columns, instead of turning them into an index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Query job 5b69917f-9ed7-483a-9241-0083acea9990 is DONE. 1.1 GB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:5b69917f-9ed7-483a-9241-0083acea9990&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Query job ac94c55d-ce8e-4694-ad97-55c933cf3053 is DONE. 123 Bytes processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:ac94c55d-ce8e-4694-ad97-55c933cf3053&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>DependencyName</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pandas</td>\n",
       "      <td>pytz</td>\n",
       "      <td>168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pandas</td>\n",
       "      <td>numpy</td>\n",
       "      <td>168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pandas</td>\n",
       "      <td>python-dateutil</td>\n",
       "      <td>168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pandas</td>\n",
       "      <td>six</td>\n",
       "      <td>168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pandas</td>\n",
       "      <td>tzdata</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Name   DependencyName  size\n",
       "0  pandas             pytz   168\n",
       "1  pandas            numpy   168\n",
       "2  pandas  python-dateutil   168\n",
       "3  pandas              six   168\n",
       "4  pandas           tzdata    56"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package_deps = deps[deps[\"Name\"] == package_name].groupby([\"Name\", \"DependencyName\"], as_index=False).size()\n",
    "package_deps.peek()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Count downloads for pandas and its dependencies\n",
    "\n",
    "The previous step created `pandas_deps` with all the dependencies of `pandas` but not pandas itself.\n",
    "\n",
    "Combine two DataFrames with the same column names with the `bigframes.pandas.concat` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_and_deps = bpd.concat(\n",
    "    [\n",
    "        package_deps.drop(columns=[\"Name\", \"size\"]).rename(columns={\"DependencyName\": \"Name\"}),\n",
    "        bpd.DataFrame({\"Name\": [package_name]}),\n",
    "    ],\n",
    "\n",
    "    # To join DataFrames that have a NULL index, set ignore_index = True.\n",
    "    ignore_index=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there is no index to implicitly join on, use the `merge` method to join two DataFrames by column name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_pypi = pandas_and_deps.merge(pypi, how=\"inner\", left_on=\"Name\", right_on=\"project\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a time series to visualize by grouping by the date, extracted from the `timestamp` column.\n",
    "\n",
    "**Note:** If you don't `peek()` at your data and only do grouped aggregations, BigQuery DataFrames can eliminate unnecessary ordering from the compilation even without `ordering_mode = \"partial\"`.\n",
    "\n",
    "When BigQuery DataFrames aggregates over columns, those columns provide a\n",
    "unique key post-aggregation that is used for ordering. Any ordering applied before is overridden. By aggregating over\n",
    "a time series, the line plots will render in the expected order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Query job 57037a4c-5b8b-4f30-a5c6-bfeb9731a38f is DONE. 270.4 GB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:57037a4c-5b8b-4f30-a5c6-bfeb9731a38f&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Query job a8ea0b8e-2260-4175-b80d-668a2411c6ad is DONE. 2.0 kB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:a8ea0b8e-2260-4175-b80d-668a2411c6ad&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "date        project        \n",
       "2024-09-17  numpy               2572982\n",
       "            pandas              2195692\n",
       "            python-dateutil     3608119\n",
       "            pytz                1863133\n",
       "            six                 3064640\n",
       "            tzdata               991989\n",
       "2024-09-18  numpy              13282573\n",
       "            pandas             10856758\n",
       "            python-dateutil    17877058\n",
       "            pytz                9450103\n",
       "            six                15225000\n",
       "            tzdata              5230039\n",
       "2024-09-19  numpy              13637868\n",
       "            pandas             11077817\n",
       "            python-dateutil    18449777\n",
       "            pytz                9690329\n",
       "            six                15706263\n",
       "            tzdata              5473910\n",
       "2024-09-20  numpy              12609524\n",
       "            pandas             10758593\n",
       "            python-dateutil    17257536\n",
       "            pytz                9082050\n",
       "            six                14489456\n",
       "            tzdata              5206738\n",
       "2024-09-21  numpy               8316481\n",
       "            pandas              7483241\n",
       "            python-dateutil    11604691\n",
       "            pytz                5494178\n",
       "            six                 8814983\n",
       "            tzdata              3141578\n",
       "2024-09-22  numpy               7768078\n",
       "            pandas              6566272\n",
       "            python-dateutil    10835755\n",
       "            pytz                5130018\n",
       "            six                 8297507\n",
       "            tzdata              2811247\n",
       "2024-09-23  numpy              12389164\n",
       "            pandas             10758931\n",
       "            python-dateutil    17153013\n",
       "            pytz                9045824\n",
       "            six                14512209\n",
       "            tzdata              5214048\n",
       "2024-09-24  numpy              10385658\n",
       "            pandas              8830996\n",
       "            python-dateutil    14066307\n",
       "            pytz                7425446\n",
       "            six                11917222\n",
       "            tzdata              4550626\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas_pypi = pandas_pypi.assign(date=pandas_pypi[\"timestamp\"].dt.date)\n",
    "downloads_per_day = pandas_pypi.groupby([\"date\", \"project\"]).size()\n",
    "\n",
    "# Cache after the aggregation so that the aggregation only runs once.\n",
    "downloads_per_day.cache()\n",
    "downloads_per_day.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BigQuery DataFrames has several built-in visualization methods. Alternatively, download the time series with the `to_pandas()` method for further analysis and visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Query job 319558aa-e092-4fd0-a8aa-447fca216a57 is DONE. 1.6 kB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:319558aa-e092-4fd0-a8aa-447fca216a57&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Query job e64eb861-bd12-4748-9c29-d97990aa1241 is DONE. 1.2 kB processed. <a target=\"_blank\" href=\"https://console.cloud.google.com/bigquery?project=swast-scratch&j=bq:US:e64eb861-bd12-4748-9c29-d97990aa1241&page=queryresults\">Open Job</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='date', ylabel='daily downloads'>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 97820 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "downloads_per_day.unstack().plot.line(rot=45, ylabel=\"daily downloads\", ylim=(0, 2e7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
