Last week we left with an open question, how to compute the N last places for the whole dataframe?
We can simply iterate across the indexes of the dataframe and call the function for each index, doing this we will create a list of dictionaries, where each dict will represent a record of the new dataframe which we are going to create and then join with the original:
# Get the N last places for each record
# The new columns names will be letters (e.g. A, B, C, .. and so on)
def get_dataframe_last_N_places(indexes, dataframe, N, placeholder=-1, columns_names = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'):
# One dict for each record
records_list = []
# For each record index
for i in indexes:
# Get the N places for that record
places = get_record_last_N_places(i, dataframe, N, placeholder)
# N last places dict
record = {}
# The key will be a letter, the place will be the place ID
for i, place in enumerate(places):
# Assign the value in the dict
record[ascii_uppercase[i]] = place
# append the record to the record list
records_list.append(record)
# Return a dataframe
return pd.DataFrame(records_list)
Let's test it out:
# The column names parameter would be a string
ascii_uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
N = 3
records_places = get_dataframe_last_N_places(df_test_2.index.values, df_test_2, N)
# Join the original dataframe with the dataframe with the N last places on the index
df_with_places = pd.merge(df_test_2, records_places, how='left', left_index=True, right_index=True)
df_with_places.head()
Let's say that -1 as placeholder would complicate the encoding section and other stuffs on the model declaration, so it's better to change this value to the maximum place id + 1:
# The column names parameter would be a string
ascii_uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
placeholder = max(df_test_2.start_place.max(), df_test_2.end_place.max()) + 1
N = 3
records_places = get_dataframe_last_N_places(df_test_2.index.values, df_test_2, N, placeholder=placeholder )
# Join the original dataframe with the dataframe with the N last places on the index
df_with_places = pd.merge(df_test_2, records_places, how='left', left_index=True, right_index=True)
df_with_places.head()
Nice! now we can start thinking about the model
To begin, we will consider just the new sequence of places and try to get an RNN working, but then our goal is to build something like this:
Because we still need to use a target to be able to test the model fitting, we will use the start_place as a target given the sequence of the last 3 places.