Is NeuralForecast able to handle "duplicates" of dates?

[meteoDaniel]

Dear nixtla community,

I am actually screening your documentation and there is a very specific requirement from our use case so I hope you are able to help me.

In this section about the exog. variables you say that future variable can be weather forecast. And this is the case I am dealing with.

But normally weather forecast having a lead time of more than 24 hours which yields to the fact that there are overlapping future time stamps.

At this time we build up numpy arrays with duplicates which works very well in tensorflow lstm compared to just using the most recent 24 hours of the forecast data.

So is it possible to feed in a dataframe with duplicates in the ds and y columns?

Thanks a lot for helping my on that point.

[meteoDaniel]

Here is a visualisation of available forecasts that can be used! Ignore the colors.

I think expanding the timeseries to use all forecasts is the reason why it works better than sliciing the forecasts.

[kdgutier]

Hi @meteoDaniel,

We currently make a distinction between ds date stamp and fcd forecast creation date.
As you mention when feeding data into a NeuralForecast model through futr_exog_list we run into overlapping date stamps.

For the moment, I have two suggestions/work-arounds:

1. Easy: Filter the predictions to make the ds not-overlapping, i.e. just feed the predictions every horizon steps.
2. Medium Hack: Feed the predictions treating the fcd as the ds and the date stamps as columns. Example:

ds	horizon1	horizon2	...	horizonH
0	y_hat_ds0_horizon1	y_hat_ds0_horizon2	...	y_hat_ds0_horizonH
1	y_hat_ds1_horizon1	y_hat_ds1_horizon2	...	y_hat_ds1_horizonH
2	y_hat_ds2_horizon1	y_hat_ds2_horizon2	...	y_hat_ds2_horizonH

Let me know your thoughts :)

[meteoDaniel]

First of all, thanks for your reply. That sounds really good so far.

1. Easy: Filter the predictions to make the ds not-overlapping, i.e. just feed the predictions every horizon steps.
-> I understand this solution in that way, that all overlapping ds will be dropped. You are totally right that this is a suitable way to get the model running, but not the solution for the problem to keep all of the data. Going this way means you assume that the dropped data has the same correlation and uncertainty as the data filtered for the use in the model when it comes to operational real-time use.

2. Medium Hack: Feed the predictions treating the fcd as the ds and the date stamps as columns. Example:
I understand this suggestion in that way:

ds	temperature_horizon_1	temperature_horizon_2	temperature_horizon_3
ds_horizon_1	1	1,5	1,2
ds_horizon_2	nan	1,5	1,2
ds_horizon_3	nan	nan	1,2

And what I have figured out here is the operational use case in real time where a new weather forecast is available. For the first timestamp/day (let's talk about daily intervals for better understanding) there is the recent forecast and the ones from yesterday and the day before yesterday (horizon 1-3). But for tomorrow there is no forecast for horizon 1 because it will be generated tomorrow (in the real world future). When it comes to an operational use it will run into the case of missing data.

One way that is possible but actually not applicable is to fake the timestamps to write every forecast in sequence one after the other one. But e.g. in pytorch-forecasting this will not work because of data leakage, it is not possible to tell the pre Processing to not use e.g. the last 2 datapoints from the forecast before and the others. Imagine you have 3 forecasts:

forecast_run	temperature
forecast_1	1
forecast_1	2
forecast_1	3
forecast_2	4
forecast_2	5
forecast_2	6
forecast_3	7
forecast_3	8
forecast_3	9

pytorch-forecasting will generate sequences e.g. [3, 4, 5] which is not valid , too.

In my tensorflow world I have created a dataframe like that (sequence length = 2):

ds	temperature	humidity	day_of_year	horizon
1	[0, 1]	[80, 83]	[355, 356]	[0, 1]
1	[1, 2]	[83, 82]	[355, 356]	[1, 2]
1	[2,2.5]	[82, 84]	[355, 356]	[2, 3]

Afterwards I derive a numpy array from it. So the model does not care anymore about the fact of duplicated timestamps.

So I am pretty sure to get input data with overlapping timeseries in a suitable and capable array that can be feed into a tft or other kinds of timeseries architecture in encoder/decoder structure.

Note: Today I have installed nixtla and will start with some tests asap.

[kdgutier]

Sounds good, let us know your findings.