How to increase the tolerance for comparing time points when synchronizing timetables?

Hi,

TL;DR: I am suffering from floating-point errors in time comparisons when synchronizing timetables and I am hoping that there exist a simple way to increase the tolerance of the comparison to mitigate the problem. Thanks in advance!

---

The following code generates two timetables with timelines that are supposed to be identical besides a shift in their initial values:

N = 9;
dt = .3;

mkTime = @(t0) seconds(t0+dt : dt : t0+N*dt);
t1 = mkTime(0);
t2 = mkTime(4*dt);

v1 = (1:N)';
v2 = (1:N)';

tt1 = timetable(v1, 'RowTimes', t1)

tt1 = 9×1 timetable
     Time      v1
    _______    __

    0.3 sec    1 
    0.6 sec    2 
    0.9 sec    3 
    1.2 sec    4 
    1.5 sec    5 
    1.8 sec    6 
    2.1 sec    7 
    2.4 sec    8 
    2.7 sec    9 

tt2 = timetable(v2, 'RowTimes', t2)

tt2 = 9×1 timetable
     Time      v2
    _______    __

    1.5 sec    1 
    1.8 sec    2 
    2.1 sec    3 
    2.4 sec    4 
    2.7 sec    5 
    3 sec      6 
    3.3 sec    7 
    3.6 sec    8 
    3.9 sec    9 

However, when tt1 and tt2 are synchronized, the resulting timetable contains multiple 'duplicates' of time points that are on the order of 1e-16 apart from each other (in this case at times 1.8 and 2.1):

tt = synchronize(tt1, tt2)

tt = 15×2 timetable
     Time      v1     v2 
    _______    ___    ___

    0.3 sec      1    NaN
    0.6 sec      2    NaN
    0.9 sec      3    NaN
    1.2 sec      4    NaN
    1.5 sec      5      1
    1.8 sec      6    NaN
    1.8 sec    NaN      2
    2.1 sec      7    NaN
    2.1 sec    NaN      3
    2.4 sec      8      4
    2.7 sec      9      5
    3 sec      NaN      6
    3.3 sec    NaN      7
    3.6 sec    NaN      8
    3.9 sec    NaN      9

tt.Time(7) - tt.Time(6)

ans = duration
   2.2737e-16 sec

I would very much like for these duplicates to be recognized as having identical times instead, hence my question: How can the tolerance for comparing time points be increased, so that it does not suffer from these precision errors?

---

Some remarks:

The older version of synchronize for timeseries offered this possibility via the 'tolerance' keyword, which leads me to assume that something similar should be available for timetable.

The documentation for synchronize offers multiple interpolation and resampling strategies for dealing with overlapping but misaligned time data:

tt_wrong = synchronize(tt1, tt2, 'regular', 'linear', TimeStep = seconds(dt))

tt_wrong = 13×2 timetable
     Time      v1    v2
    _______    __    __

    0.3 sec     1    -3
    0.6 sec     2    -2
    0.9 sec     3    -1
    1.2 sec     4     0
    1.5 sec     5     1
    1.8 sec     6     2
    2.1 sec     7     3
    2.4 sec     8     4
    2.7 sec     9     5
    3 sec      10     6
    3.3 sec    11     7
    3.6 sec    12     8
    3.9 sec    13     9

However, it seems like overkill to employ such methods for mere precision errors and, moreover, they do unfortunately not apply in the current scenario, since the result pads the nonoverlapping pieces of each timetable with nonsense values.

---

Any help will be much appreciated, thanks in advance!

Julius

[Julius]

Note to self: it appears that these errors are preventable by swapping the timelines to single-precision floats, but I am not knowledgeable enough to predict whether this will prevent these issues in all cases. In case anyone can comment on this I would be happy to hear it.

[Star Strider]

I am not certain what the exact problem is (since this seems to be a proxy problem). Consider using the ismembertol function.

Answer 1

[Les Beckham]

Based on this section of the documentation: Synchronize Timetables to Arbitrary Time Vector

N = 9;
dt = .3;

mkTime = @(t0) seconds(t0+dt : dt : t0+N*dt);
t1 = mkTime(0);
t2 = mkTime(4*dt);

v1 = (1:N)';
v2 = (1:N)';

tt1 = timetable(v1, 'RowTimes', t1);
tt2 = timetable(v2, 'RowTimes', t2);

% create the time vector for synchronizing with a new time vector
t = seconds(uniquetol(seconds(union(t1, t2)), 1e-4)) % this is the tricky/messy bit

t = 1×13 duration array
   0.3 sec   0.6 sec   0.9 sec   1.2 sec   1.5 sec   1.8 sec   2.1 sec   2.4 sec   2.7 sec     3 sec   3.3 sec   3.6 sec   3.9 sec

TT = synchronize(tt1, tt2, t, 'linear')

TT = 13×2 timetable
     Time      v1    v2
    _______    __    __

    0.3 sec     1    -3
    0.6 sec     2    -2
    0.9 sec     3    -1
    1.2 sec     4     0
    1.5 sec     5     1
    1.8 sec     6     2
    2.1 sec     7     3
    2.4 sec     8     4
    2.7 sec     9     5
    3 sec      10     6
    3.3 sec    11     7
    3.6 sec    12     8
    3.9 sec    13     9

[Julius]

Unfortunately this snippet suffers from the same problem as many examples on the synchronize documentation page: since my time vectors are nonoverlapping I am unable to use any other data adjustment method than fillwithmissing, otherwise the resulting timetable is padded with nonsense values at the nonoverlapping parts.

[Star Strider]

The Time Series verision of synchronize has a tolerance option.

[Les Beckham]

If your tables are "nonoverlapping" then you should not need to worry about tolerance between nearly equal timestamps. You should be able to just concatenate the tables and then (if you wish) sort the result.

When asking for help with a specific issue you should provide an example of that specific issue, rather than one that doesn't represent that issue.

Please provide data that represents the issue that you are actually having, rather than one that doesn't.

I'm pretty sure that the answer I provided solved the problem as you described it in your question.

If you actually have a different problem, you need to provide an example of that problem, not the one you described in your original question.

[Julius]

@Les Beckham My problem is as I described in my original post: I am dealing with "timetables with timelines that are supposed to be identical besides a shift in their initial values", i.e. partially overlapping and partially nonoverlapping. Further, I wish to obtain a synchronized timetable containing both original timetables as separate columns. This combined scenario disallows both vertical and horizontal concatenation of timetables, the former because it would attempt to merge the tables into a single column and the latter because that operation requires equal timelines for the operands.

Don't mistake my appreciation for your efforts, but your answer does not solve the problem stated in my question. Please note that I even added a remark at the bottom to explain why interpolation is not a possible solution for my problem.

If the problem scenario is not clear enough I will happily supply further data/explanation, but it seems to me that all the relevant parts are present.

[Julius]

@Star Strider Thank you, I am aware of this feature for timeseries, which was part of the motivation for this inquiry about potential equivalent functionality for timetables, since they hold my preference over timeseries. More importantly, however, as far as I can tell, timeseries synchronization offers no possibility of including nonoverlapping parts of the input series. In case my perception is wrong and you know a way to enable this I would love to hear it!

Answer 2

[Seth Furman]

If possible, please provide more detail about the actual data so that I can try to provide better guidance. For example,

• Where are the row-times coming from?
• Are the row-times regular?
• Do the data only need to be precise to second?

In this particular case, simply creating the time vectors using duration arithmetic instead of double arithmetic avoids the precision issue.

N = 9;
dt = seconds(.3);

mkTime = @(t0) t0+dt : dt : t0+N*dt;
t1 = mkTime(seconds(0));
t2 = mkTime(4*dt);

v1 = (1:N)';
v2 = (1:N)';

tt1 = timetable(v1, 'RowTimes', t1)

tt1 = 9×1 timetable
     Time      v1
    _______    __

    0.3 sec    1 
    0.6 sec    2 
    0.9 sec    3 
    1.2 sec    4 
    1.5 sec    5 
    1.8 sec    6 
    2.1 sec    7 
    2.4 sec    8 
    2.7 sec    9 

tt2 = timetable(v2, 'RowTimes', t2)

tt2 = 9×1 timetable
     Time      v2
    _______    __

    1.5 sec    1 
    1.8 sec    2 
    2.1 sec    3 
    2.4 sec    4 
    2.7 sec    5 
    3 sec      6 
    3.3 sec    7 
    3.6 sec    8 
    3.9 sec    9 

tt = synchronize(tt1, tt2)

tt = 13×2 timetable
     Time      v1     v2 
    _______    ___    ___

    0.3 sec      1    NaN
    0.6 sec      2    NaN
    0.9 sec      3    NaN
    1.2 sec      4    NaN
    1.5 sec      5      1
    1.8 sec      6      2
    2.1 sec      7      3
    2.4 sec      8      4
    2.7 sec      9      5
    3 sec      NaN      6
    3.3 sec    NaN      7
    3.6 sec    NaN      8
    3.9 sec    NaN      9

tt.Time(7) - tt.Time(6)

ans = duration
   0.3 sec