{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8ebb6e6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2025 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f3bbfa",
   "metadata": {},
   "source": [
    "# BigFrames Timedelta\n",
    "\n",
    "<table align=\"left\">\n",
    "\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/googleapis/python-bigquery-dataframes/blob/main/notebooks/data_types/timedelta.ipynb\">\n",
    "      <img src=\"https://raw.githubusercontent.com/googleapis/python-bigquery-dataframes/refs/heads/main/third_party/logo/colab-logo.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/googleapis/python-bigquery-dataframes/blob/main/notebooks/data_types/timedelta.ipynb\">\n",
    "      <img src=\"https://raw.githubusercontent.com/googleapis/python-bigquery-dataframes/refs/heads/main/third_party/logo/github-logo.png\" width=\"32\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/googleapis/python-bigquery-dataframes/blob/main/notebooks/data_types/timedelta.ipynb\">\n",
    "      <img src=\"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTW1gvOovVlbZAIZylUtf5Iu8-693qS1w5NJw&s\" alt=\"BQ logo\" width=\"35\">\n",
    "      Open in BQ Studio\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f74e2573",
   "metadata": {},
   "source": [
    "In this notebook, you will use timedeltas to analyze the taxi trips in NYC. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f74dec4",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "51173665",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bigframes.pandas as bpd\n",
    "\n",
    "PROJECT = \"bigframes-dev\" # replace this with your project\n",
    "LOCATION = \"us\" # replace this with your location\n",
    "\n",
    "bpd.options.bigquery.project = PROJECT\n",
    "bpd.options.bigquery.location = LOCATION\n",
    "bpd.options.display.progress_bar = None\n",
    "\n",
    "bpd.options.bigquery.ordering_mode = \"partial\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d64fd3e3",
   "metadata": {},
   "source": [
    "# Timedelta arithmetics and comparisons"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e10bd798",
   "metadata": {},
   "source": [
    "First, you load the taxi data from the BigQuery public dataset `bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2021`. The size of this table is about 6.3 GB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f1b11138",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vendor_id</th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>rate_code</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>imp_surcharge</th>\n",
       "      <th>airport_fee</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>pickup_location_id</th>\n",
       "      <th>dropoff_location_id</th>\n",
       "      <th>data_file_year</th>\n",
       "      <th>data_file_month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2021-06-09 07:44:46+00:00</td>\n",
       "      <td>2021-06-09 07:45:24+00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>2.200000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>4</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>263</td>\n",
       "      <td>263</td>\n",
       "      <td>2021</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2021-06-07 11:59:46+00:00</td>\n",
       "      <td>2021-06-07 12:00:00+00:00</td>\n",
       "      <td>2</td>\n",
       "      <td>0.010000000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>N</td>\n",
       "      <td>2</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>263</td>\n",
       "      <td>263</td>\n",
       "      <td>2021</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2021-06-23 15:03:58+00:00</td>\n",
       "      <td>2021-06-23 15:04:34+00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>193</td>\n",
       "      <td>193</td>\n",
       "      <td>2021</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2021-06-12 14:26:55+00:00</td>\n",
       "      <td>2021-06-12 14:27:08+00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>3</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>143</td>\n",
       "      <td>143</td>\n",
       "      <td>2021</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>2021-06-15 08:39:01+00:00</td>\n",
       "      <td>2021-06-15 08:40:36+00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>0E-9</td>\n",
       "      <td>193</td>\n",
       "      <td>193</td>\n",
       "      <td>2021</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  vendor_id            pickup_datetime           dropoff_datetime  \\\n",
       "0         1  2021-06-09 07:44:46+00:00  2021-06-09 07:45:24+00:00   \n",
       "1         2  2021-06-07 11:59:46+00:00  2021-06-07 12:00:00+00:00   \n",
       "2         2  2021-06-23 15:03:58+00:00  2021-06-23 15:04:34+00:00   \n",
       "3         1  2021-06-12 14:26:55+00:00  2021-06-12 14:27:08+00:00   \n",
       "4         2  2021-06-15 08:39:01+00:00  2021-06-15 08:40:36+00:00   \n",
       "\n",
       "   passenger_count  trip_distance rate_code store_and_fwd_flag payment_type  \\\n",
       "0                1    2.200000000       1.0                  N            4   \n",
       "1                2    0.010000000       3.0                  N            2   \n",
       "2                1           0E-9       1.0                  N            1   \n",
       "3                0    1.000000000       1.0                  N            3   \n",
       "4                1           0E-9       1.0                  N            1   \n",
       "\n",
       "   fare_amount  extra  mta_tax  tip_amount  tolls_amount  imp_surcharge  \\\n",
       "0         0E-9   0E-9     0E-9        0E-9          0E-9           0E-9   \n",
       "1         0E-9   0E-9     0E-9        0E-9          0E-9           0E-9   \n",
       "2         0E-9   0E-9     0E-9        0E-9          0E-9           0E-9   \n",
       "3         0E-9   0E-9     0E-9        0E-9          0E-9           0E-9   \n",
       "4         0E-9   0E-9     0E-9        0E-9          0E-9           0E-9   \n",
       "\n",
       "   airport_fee  total_amount pickup_location_id dropoff_location_id  \\\n",
       "0         0E-9          0E-9                263                 263   \n",
       "1         0E-9          0E-9                263                 263   \n",
       "2         0E-9          0E-9                193                 193   \n",
       "3         0E-9          0E-9                143                 143   \n",
       "4         0E-9          0E-9                193                 193   \n",
       "\n",
       "   data_file_year  data_file_month  \n",
       "0            2021                6  \n",
       "1            2021                6  \n",
       "2            2021                6  \n",
       "3            2021                6  \n",
       "4            2021                6  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi_trips = bpd.read_gbq(\"bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2021\").dropna()\n",
    "taxi_trips = taxi_trips[taxi_trips['pickup_datetime'].dt.year == 2021]\n",
    "taxi_trips.peek(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5b13623",
   "metadata": {},
   "source": [
    "Based on the dataframe content, you calculate the trip durations and store them under the column “trip_duration”. You can see that the values under \"trip_duartion\" are timedeltas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "12fc1a5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "duration[us][pyarrow]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi_trips['trip_duration'] = taxi_trips['dropoff_datetime'] - taxi_trips['pickup_datetime']\n",
    "taxi_trips['trip_duration'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b18b8d9",
   "metadata": {},
   "source": [
    "To remove data outliers, you filter the taxi_trips to keep only the trips that were less than 2 hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "62b2d42e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "taxi_trips = taxi_trips[taxi_trips['trip_duration'] <= pd.Timedelta(\"2h\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "665fb8a7",
   "metadata": {},
   "source": [
    "Finally, you calculate the average speed of each trip, and find the median speed of all trips."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e79e23c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The median speed of an average taxi trip is: 10.58 mph.\n"
     ]
    }
   ],
   "source": [
    "average_speed = taxi_trips[\"trip_distance\"] / (taxi_trips['trip_duration'] / pd.Timedelta(\"1h\"))\n",
    "print(f\"The median speed of an average taxi trip is: {average_speed.median():.2f} mph.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "261dbdf1",
   "metadata": {},
   "source": [
    "Given how packed NYC is, a median taxi speed of 10.58 mph totally makes sense."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6122c32e",
   "metadata": {},
   "source": [
    "# Use timedelta for rolling aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05ae6fbb",
   "metadata": {},
   "source": [
    "Using your existing dataset, you can now calculate the taxi trip count over a period of two days, and find out when NYC is at its busiest and when it is fast asleep.\n",
    "\n",
    "First, you pick two workdays (a Thursday and a Friday) as your target dates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7dc50b1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records: 255434\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "\n",
    "target_dates = [\n",
    "    datetime.date(2021, 12, 2), \n",
    "    datetime.date(2021, 12, 3)\n",
    "]\n",
    "\n",
    "two_day_taxi_trips = taxi_trips[taxi_trips['pickup_datetime'].dt.date.isin(target_dates)]\n",
    "print(f\"Number of records: {len(two_day_taxi_trips)}\")\n",
    "# Number of records: 255434\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5f39bd8",
   "metadata": {},
   "source": [
    "Your next step involves aggregating the number of records associated with each unique \"pickup_datetime\" value. Additionally, the data undergo upsampling to account for any absent timestamps, which are populated with a count of 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f25b34f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "two_day_trip_count = two_day_taxi_trips['pickup_datetime'].value_counts()\n",
    "\n",
    "full_index = pd.date_range(\n",
    "    start='2021-12-02 00:00:00',\n",
    "    end='2021-12-04 00:00:00',\n",
    "    freq='s',\n",
    "    tz='UTC'\n",
    ")\n",
    "two_day_trip_count = two_day_trip_count.reindex(full_index).fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3394b2ba",
   "metadata": {},
   "source": [
    "You'll then calculate the sum of trip counts within a 5-minute rolling window. This involves using the `rolling()` method, which can accept a time window in the form of either a string or a timedelta object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4d5987d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "two_day_trip_rolling_count = two_day_trip_count.sort_index().rolling(window=\"5m\").sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3811b1fb",
   "metadata": {},
   "source": [
    "Finally, you visualize the trip counts throughout the target dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "871c32c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 102228 characters]",
      "text/plain": [
       "<Figure size 1600x700 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "plt.figure(figsize=(16,7))\n",
    "ax = plt.gca()\n",
    "formatter = mdates.DateFormatter(\"%D %H:%M:%S\")\n",
    "ax.xaxis.set_major_formatter(formatter)\n",
    "\n",
    "ax.tick_params(axis='both', labelsize=10)\n",
    "\n",
    "two_day_trip_rolling_count.plot(ax=ax, legend=False)\n",
    "plt.xlabel(\"Pickup Time\", fontsize=12)\n",
    "plt.ylabel(\"Trip count in last 5 minutes\", fontsize=12)\n",
    "plt.grid()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "336b78ce",
   "metadata": {},
   "source": [
    "The taxi ride count reached its lowest point around 5:00 a.m., and peaked around 7:00 p.m. on a workday. Such is the rhythm of NYC."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
