This week

• [x] Compute new features
• [x] Compute some statistics on data
• [x] Data cleaning pt. 2

Last week we cleaned the dataset using the attributes already provided, can we do better?

From the attributes we have we can derive some derived features like:

• Trip distance using the locations coordinates
• Trip duration in time using the pickup and dropoff datetimes
• Trip average speed from the previous two features

Haversine trip distance

How to compute the distance between 2 pairs of coordinates?

Reading this interesting analysis on the taxi dataset I found out that there is a way to measure the distance of 2 points on a sphere, the Haversine distance.

The Haversine (or great circle) distance is the angular distance between two points on the surface of a sphere. The first distance of each point is assumed to be the latitude, the second is the longitude, given in radians.

To compute the Haversine distance of each trip we can simply use the related library and apply it to each row of our valid records:

N.B. The Haversine distance is an approximation of the real trip distance, in the real world, if not flying, we cannot move in a straight line. This mean that Real_distance >> Haversine_distance

And so: Estimated_Speed < Real_Speed

from haversine import haversine

# Compute the haversine distance of a single trip (var_row) in Km
def haversine_distance(var_row):
  h_distance =  haversine((var_row["pickup_latitude"], var_row["pickup_longitude"]), 
                     (var_row["dropoff_latitude"], var_row["dropoff_longitude"]))
	# Round the computed distance to the third decimal
  return round(h_distance, 3)

Trip time