DDSXTY14 — Impossible to handle @must_understand AND @optional fields in an @appendable struct correctly

The XTypes spec seems to suggest that @must_understand fields are applicable to any non-final datatype. Section 7.3.1.2.1.9 mentions this:

"By default, the assignment from an object of type T2 into an object of type T1 where T1 and T2 are non-final types will ignore any members in T2 that are not present in T1. This behavior may be changed by applying the @must_understand annotation to a member within a type definition."

That would mean that @must_understand fields are also applicable to @appendable structs, but in this case there is no way for a field to be identified as must_understand field in its Delimited-CDR representation.

Of course you could state that in case of @appendable structs, you should determine assignability at discovery time and not at run time, in which case you are not dependent on the availability of a must_understand bit in the serialized blob. However, in case a Writer publishes a sample with a field that is both @must_understand AND @optional, according Table 19 (section 7.2.4.4.8) there is no mismatch between this Writer and a Reader that is missing the field (that is only the case if the field has @optional set to FALSE), which means the sample needs to be matched at runtime.

The spec is a little unclear of the semantics in this case: I guess that if the optional value is empty but must_understand you can slice it out of the projected type, but when it is not empty you are not allowed to slice the value out since it is a must_understand value and you will have to discard the whole sample instead. It would be nice if the spec could explicitly confirm or deny this.

But working from the presumption that you are indeed allowed to slice out values that are both empty and must_understand, the next problem arises immediately: how does a Reader know that a sample containing an unfamiliar field actually represents an optional field that is must_understand?

Let's consider a scenario where I have a Reader that is reading a type A with the following defintion:

@appendable struct A {
    @key long id;
     long x;
};

Now this Reader can be matched against all sorts of Writers, including Writers that have appended all kinds of fields to the end of this datatype: its deserializer simply slices everything out that comes after field x.

This works well for most appended versions of this datatype, but now suddenly an additional Writer joins the system with the following definition:

@appendable struct A {
    @key long id;
     long x;
    @optional @must_understand long y;
};

This Writer will match the Reader according to Table 19 since although the @must_understand annotation is TRUE, the corresponding @optional field is NOT false. So now the deserializer for our Reader has to deserialize a sample for which it doesn't know what the payload after the x actually represents. It might represent a value that it can safely slice out, or it might represent a value that may not be sliced out.

The spec states in rule number 20 of section 7.4.3.5.3 that in case of XCDRV2, an optional member is preceeded by a boolean called is_present. The problem here is that our deserializer doesn't know where the sample originates from therefore has no knowledge whether the byte following field x represent an is_present boolean or just some other sliceable field.

So the basic question boils down to this:

• Do we allow fields that are both must_understand and optional in case of Appendable extensibility?
• If so, how do we indicate in this case that a field is both optional and must_understand in a preferably backward compatible way?