Here is a solution based on @NoahWeber and @etiennedm answers. It is based on a juxtaposition of splittings, a 1) repeated k fold splitting (to get training customers and testing customers), and 2) a time series splits on each k fold.
This strategy is based on a time series' splitting using a custom CV split iterator on dates (whereas usual CV split iterators are based on sample size / folds number).
An implementation within sklearn ecosystem is provided.
Let's restate the problem.
Say you have 10 periods and 3 customers indexed as follows :
example_data = pd.DataFrame({
'index': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
'cutomer': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
'date': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
})
We do a repeated k fold with 2 folds and 2 iterations (4 folds in total) and within each k fold split we split again with time series split such that each time series split has 2 folds
kfold split 1 : training customers are [0, 1] and testing customers are [2]
kfold split 1 time series split 1 : train indices are [0, 1, 2, 3, 10, 11, 12, 13] and test indices are [24, 25, 26]
kfold split 1 time series split 2 : train indices are [0, 1, 2, 3, 4, 5, 6, 10, 11, 12, 13, 14, 15, 16] and test indices are [27, 28, 29]
kfold split 2 : training customers are [2] and testing customers are [0, 1]
kfold split 2 time series split 1 : train indices are [20, 21, 22, 23] and test indices are [4, 5, 6, 7, 15, 16, 17]
kfold split 2 time series split 2 : train indices are [20, 21, 22, 23, 24, 25, 26] and test indices are [7, 8, 9, 17, 18, 19]
kfold split 3 : training customers are [0, 2] and testing customers are [1]
kfold split 3 time series split 1 : train indices are [0, 1, 2, 3, 20, 21, 22, 23] and test indices are [14, 15, 16]
kfold split 3 time series split 2 : train indices are [0, 1, 2, 3, 4, 5, 6, 20, 21, 22, 23, 24, 25, 26] and test indices are [17, 18, 19]
kfold split 4 : training customers are [1] and testing customers are [0, 2]
kfold split 4 time series split 1 : train indices are [10, 11, 12, 13,] and test indices are [4, 5, 6, 24, 25, 26]
kfold split 4 time series split 2 : train indices are [10, 11, 12, 13, 14, 15, 16] and test indices are [7, 8, 9, 27, 28, 29]
Usually, cross-validation iterators, such as those in sklearn, which are based on the number of folds, i.e., the sample size in each fold. These are unfortunately not suited in our kfold / time series split with real data. In fact, nothing guarantees that data is perfectly distributed over time and over groups. (as we assumed in the previous example).
For instance, we can have the 4th observation in the consumer training sample (say customer 0 and 1 in kfold split 1 in the example) that comes after the 4th observation in the test sample (say customer 2). This violates condition 1.
Here is one CV splits strategy based on dates by fold (not by sample size or the number of folds).
Say you have previous data but with random dates. Define an initial_training_rolling_months, rolling_window_months. say for example 6 and 1 months.
kfold split 1 : training customers are [0, 1] and testing customers are [2]
kfold split 1 time series split 1 : train sample is the 6 first months of customers [0, 1] and test sample is the month starting after train sample for customers [2]
kfold split 1 time series split 2 : train sample is the 7 first months of customers [0, 1] and test sample is the month starting after train sample for customers [2]
Below a suggestion of implementation to build such a time series split iterator.
The returned iterator is a list of tuples that you can use as another cross-validation iterator.
With a simple generated data like in our previous example to debug the folds generation, noting that customers 1 (resp. 2) data begins at index 366 and (resp. 732).
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
df = generate_happy_case_dataframe()
grouped_ts_validation_iterator = build_grouped_ts_validation_iterator(df)
gridsearch = GridSearchCV(estimator=RandomForestClassifier(), cv=grouped_ts_validation_iterator, param_grid={})
gridsearch.fit(df[['feat0', 'feat1', 'feat2', 'feat3', 'feat4']].values, df['label'].values)
gridsearch.predict([[0.1, 0.2, 0.1, 0.4, 0.1]])
With randomly generated data like in @etiennedm's example (to debug split, I covered simple cases such as when the test sample begins before the training samples or just after).
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
df = generate_fake_random_dataframe()
grouped_ts_validation_iterator = build_grouped_ts_validation_iterator(df)
gridsearch = GridSearchCV(estimator=RandomForestClassifier(), cv=grouped_ts_validation_iterator, param_grid={})
gridsearch.fit(df[['feat0', 'feat1', 'feat2', 'feat3', 'feat4']].values, df['label'].values)
gridsearch.predict([[0.1, 0.2, 0.1, 0.4, 0.1]])
The implementation :
import pandas as pd
import numpy as np
from sklearn.model_selection import RepeatedKFold
def generate_fake_random_dataframe(start=pd.to_datetime('2015-01-01'), end=pd.to_datetime('2018-01-01')):
fake_date = generate_fake_dates(start, end, 500)
df = pd.DataFrame(data=np.random.random((500,5)), columns=['feat'+str(i) for i in range(5)])
df['customer_id'] = np.random.randint(0, 5, 500)
df['label'] = np.random.randint(0, 3, 500)
df['dates'] = fake_date
df = df.reset_index() # important since df.index will be used as split index
return df
def generate_fake_dates(start, end, n):
start_u = start.value//109
end_u = end.value//109
return pd.DatetimeIndex((10*9np.random.randint(start_u, end_u, n, dtype=np.int64)).view('M8[ns]'))
def generate_happy_case_dataframe(start=pd.to_datetime('2019-01-01'), end=pd.to_datetime('2020-01-01')):
dates = pd.date_range(start, end)
length_year = len(dates)
lenght_df = length_year * 3
df = pd.DataFrame(data=np.random.random((lenght_df, 5)), columns=['feat'+str(i) for i in range(5)])
df['label'] = np.random.randint(0, 3, lenght_df)
df['dates'] = list(dates) * 3
df['customer_id'] = [0] * length_year + [1] * length_year + [2] * length_year
return df
def build_grouped_ts_validation_iterator(df, kfold_n_split=2, kfold_n_repeats=5, initial_training_rolling_months=6, rolling_window_months=1):
rkf = RepeatedKFold(n_splits=kfold_n_split, n_repeats=kfold_n_repeats, random_state=42)
CV_iterator = list()
for train_customers_ids, test_customers_ids in rkf.split(df['customer_id'].unique()):
print("rkf training/testing with customers : " + str(train_customers_ids)+"/"+str(test_customers_ids))
this_k_fold_ts_split = split_with_dates_for_validation(df=df,
train_customers_ids=train_customers_ids,
test_customers_ids=test_customers_ids,
initial_training_rolling_months=initial_training_rolling_months,
rolling_window_months=rolling_window_months)
print("In this k fold, there is", len(this_k_fold_ts_split), 'time series splits')
for split_i, split in enumerate(this_k_fold_ts_split) :
print("for this ts split number", str(split_i))
print("train ids is len", len(split[0]), 'and are:', split[0])
print("test ids is len", len(split[1]), 'and are:', split[1])
CV_iterator.extend(this_k_fold_ts_split)
print('***')
return tuple(CV_iterator)
def split_with_dates_for_validation(df, train_customers_ids, test_customers_ids, initial_training_rolling_months=6, rolling_window_months=1):
start_train_df_date, end_train_df_date, start_test_df_date, end_test_df_date =
fetch_extremas_train_test_df_dates(df, train_customers_ids, test_customers_ids)
start_training_date, end_training_date, start_testing_date, end_testing_date = \
initialize_training_dates(start_train_df_date, start_test_df_date, initial_training_rolling_months, rolling_window_months)
ts_splits = list()
while not stop_time_series_split_decision(end_train_df_date, end_test_df_date, start_training_date, end_testing_date, rolling_window_months):
# The while implies that if testing sample is les than one month, then the process stops
this_ts_split_training_indices = fetch_this_split_training_indices(df, train_customers_ids, start_training_date, end_training_date)
this_ts_split_testing_indices = fetch_this_split_testing_indices(df, test_customers_ids, start_testing_date, end_testing_date)
if this_ts_split_testing_indices:
# If testing data is not empty, i.e. something to learn
ts_splits.append((this_ts_split_training_indices, this_ts_split_testing_indices))
start_training_date, end_training_date, start_testing_date, end_testing_date =\
update_testing_training_dates(start_training_date, end_training_date, start_testing_date, end_testing_date, rolling_window_months)
return ts_splits
def fetch_extremas_train_test_df_dates(df, train_customers_ids, test_customers_ids):
train_df, test_df = df.loc[df['customer_id'].isin(train_customers_ids)], df.loc[df['customer_id'].isin(test_customers_ids)]
start_train_df_date, end_train_df_date = min(train_df['dates']), max(train_df['dates'])
start_test_df_date, end_test_df_date = min(test_df['dates']), max(test_df['dates'])
return start_train_df_date, end_train_df_date, start_test_df_date, end_test_df_date
def initialize_training_dates(start_train_df_date, start_test_df_date, initial_training_rolling_months, rolling_window_months):
start_training_date = start_train_df_date
# cover the case where test consumers begins long after (initial_training_rolling_months after) train consumers
if start_training_date + pd.DateOffset(months=initial_training_rolling_months) < start_test_df_date:
start_training_date = start_test_df_date - pd.DateOffset(months=initial_training_rolling_months)
end_training_date = start_train_df_date + pd.DateOffset(months=initial_training_rolling_months)
start_testing_date = end_training_date
end_testing_date = start_testing_date + pd.DateOffset(months=rolling_window_months)
return start_training_date, end_training_date, start_testing_date, end_testing_date
def stop_time_series_split_decision(end_train_df_date, end_test_df_date, end_training_date, end_testing_date, rolling_window_months):
no_more_training_data_stoping_condition = end_training_date + pd.DateOffset(months=rolling_window_months) > end_train_df_date
no_more_testing_data_stoping_condition = end_testing_date + pd.DateOffset(months=rolling_window_months) > end_test_df_date
stoping_condition = no_more_training_data_stoping_condition or no_more_testing_data_stoping_condition
return stoping_condition
def update_testing_training_dates(start_training_date, end_training_date, start_testing_date, end_testing_date, rolling_window_months):
start_training_date = start_training_date
end_training_date += pd.DateOffset(months=rolling_window_months)
start_testing_date += pd.DateOffset(months=rolling_window_months)
end_testing_date += pd.DateOffset(months=rolling_window_months)
return start_training_date, end_training_date, start_testing_date, end_testing_date
def fetch_this_split_training_indices(df, train_customers_ids, start_training_date, end_training_date):
train_df = df.loc[df['customer_id'].isin(train_customers_ids)]
in_training_period_df = train_df.loc[(train_df['dates'] >= start_training_date) & (train_df['dates'] < end_training_date)]
this_ts_split_training_indices = in_training_period_df.index.to_list()
return this_ts_split_training_indices
def fetch_this_split_testing_indices(df, test_customers_ids, start_testing_date, end_testing_date):
test_df = df.loc[df['customer_id'].isin(test_customers_ids)]
in_testing_period_df = test_df.loc[(test_df['dates'] >= start_testing_date) & (test_df['dates'] < end_testing_date)]
this_ts_split_testing_indices = in_testing_period_df.index.to_list()
return this_ts_split_testing_indices
