DDSIRTP23 — GAP lack information on related (instance/key) needed for correct DDS behavior

Title: Must send GAP when filtering on writer-side regardless of reliability QoS setting
Source:
Real-Time Innovations, Inc. (Ken Brophy, ken@rti.com)
Summary:
This issue is not currently in a state to be resolved. What follows are various thoughts on the issue and possible solutions to be discussed. This issue will not be resolved in time for the finalization task force and is included here to be documented for the revision task force.
Discussion topic: what are DDS semantics for combining filtering with the deadline QoS?
Should the deadline be triggered when all samples during the deadline period were filtered out?
That is, does deadline require at least one sample to arrive every deadline period seconds that passed the filter?
Or is the deadline satisfied when any sample arrives within the period, whether filtered out or not?

If deadline only applies to samples that pass the filter, RTPS needs no changes, simply use the GAP subMessage to avoid incorrect onDataLost callbacks.
If not, we run into a problem when using keyed DataWriters and finite deadlines. As the deadline applies on a per instance basis, the Reader expects at least one update for every instance, even when none of the updates for a particular instance pass the filter. A GAP message does not indicate for which instance an update was filtered out, so it cannot be used by the Reader to verify the deadline constraint. Instead, we should consider using an empty DATA message instead, possibly with a flag that states the update did not pass any filter. This would also be useful to add a new instance state NOT_ALIVE_FILTERED or so later on to the DDS spec.
Another possibility would be to add a list of KeyHashes to the GAP message. SO that a GAP that is caused by a CFT actually encodes the instances that are being gapped. This would not cause incorrect firings of the DEADLINE and as a result would maintain ownership of instances even if they are filtered out…
There are two ways to do this.
Either we separate GAPs that correspond to filtered samples from those that correspond to irrelevant samples. So in effect we have two kinds of GAP messages
Or we list explicitly the sequence number of each filtered message along with its KeyHash.
Not clear what would be easier implementationwise. The samples that have been filtered are still on the writer so it appears that either implementation would work.
Option (1) would save putting the sequence number with each KeyHash. This can be 4 or 8 bytes per instance, depending on whether we put the sequence number as is, or we encode it as an increment
Option (2) would cause additional GAP submessages to be sent which is an overhead of 28 bytes. Not clear what is less costly…
Also, if we use Option(1), then the messages that represent real GAPs can be sent via multicast; but this is only likely to occur when late joiners appear as normally there would be no "irrelevant" gaps if data is published immediately. Moreover we can in practice still do this and separate the GAP messages that represent real GAPs from the ones that don't. Option 1 does not force us to combine, just provides the means to do so…
Option (2) has the problem that in certain edge cases the overhead is significant. For example if each we have a irrelevant-sample GAP followed by a filtered sample, followed by an irrelevant sample gap, etc. then we end up sending one GAP message per filtered sample with is 28 bytes of overhead per filtered sample versus a single GAP with 4 bytes of overhead per filtered sample. Also the processing is much more efficient as each GAP message is dispatched separately up the receiver's stack.
For this reason it appears that Option 1 is more flexible, and the overhead is more stable. Opt
Proposal(s):
Always send GAPs for filtered-out messages (both in the BestEffort and in the RELIABLE) cases
If the type is Keyed, then the GAP also includes at the end a sequence of :
struct FilteredSampleDesc

;
The GAP message gets two additional flags:
KeyHashFlag
indicates the presence of the KeyHashPrefix
FilteredSamplesFlag
Indicates the presence of the sequence< FilteredSampleDesc>
An issue needs to be filed against the DDS spec to clarify:
(a) Whether the deadline as specified by a DataReader should apply to the samples that pass the DataReader filter or to the samples sent by any writer?
(b) Whether a new instance_state ALIVE_FILTERED should be added such that the DataReader can determine that a sample was filtered and potentially take action on that.
(c) Whether an API or a QoS should be added to the DataReader to allow the DataReader to remove the instance information for instances with state ALIVE_FILTERED after all samples are taken. This allows resources to be conserved in the case where once filtered we expect the instance to remain filtered and also allows a reader to be notified if the instance becomes unfiltered.
(d) Weather to add a filtered_generation_count that the instance has becomed ALIVE after being in the ALIVE_FILTERED

Resolution:
T4 should include code and description that states that when the sample is not relevant, send a GAP…same for the stateful best effort writer.
Revised Text:

Introduce a mechanism to inform a Reader that an instance is filtered

Extend the StatusInfo_t that is sent via inline Qos (PID_STATUS_INFO) such that it can be used to indicate that an instance has been filtered by a reader. This extends the current usage to send DISPOSE and UNREGISTER messages.

Add an issue to the DDS RTF 1.5 to introduce the concept of ALIVE_FILTERED Instance state to complement the existing ones (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS).

Define more precisely when a writer that does writer-side content filter should send the DISPOSE, UNREGISTER, and FILTERED messages.