This week

So, we have locations expressed in terms of Latitude and Longitude, but this representation is to fine grained for a Neural Network.

If we remember the dataset of 2019, locations where expressed in terms of Zone IDs.

In fact, from the website of NYC Open Data we can find a map showing the taxi zones and many options to download the relative files:

NYC Taxi Zones This map shows the NYC Taxi Zones, which correspond to the pickup and drop-off zones, or LocationIDs, included in the Yellow, Green, and FHV Trip Records published to Open Data. The taxi zones are roughly based on NYC Department of City Planning’s Neighborhood Tabulation Areas (NTAs) and are meant to approximate neighbourhoods, so you can see which neighbourhood a passenger was picked up in, and which neighbourhood they were dropped off in.

NYC Taxi Zones | NYC Open Data

But how to use these files?

I found this nice article on Medium in which the writer used Google BigQuery to analyze the data and also to map coordinates to zones.

<aside> 💡 BigQuery also provides a public dataset that contains NYC boroughs, zone names, and their geography spatial data. It can be found under this dataset name: new_york_taxi_trips.taxi_zone_geom. Using this dataset, I will be able to identify which coordinates are really under NYC by doing INNER JOIN.

</aside>

She used 2 functions:

  1. ST_GEOGPOINT: function can only work when the latitudes and longitudes are within the correct range.
  2. ST_DWithin: function which takes a point and a polygon and return true if the point is within the polygon

Pretty straightforward, but:

  1. Google BigQuery, as any GCP services, is not free.
  2. We have a lot of data, even using just the coordinates columns to perform the join will require uploading and downloading GBs of CSV files.

Another problem is that SQL do not provide those kind of spacial functions, so we need to use another method:

Python GIS

What is GIS?