Subject: [Asdf] PR#22: Actually define scaleMinimum and scaleMaximum

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Mar 2021 17:59:26 +0100

scaleMinimum and scaleMaximum have been part of the grammar from the start, but never were actually defined as to their semantics.

I have put in a PR with a proposed meaning:

https://github.com/ietf-wg-asdf/SDF/pull/22

➔ Is this the right semantics?

We currently don’t have models in the playground that use scalemin/max, but there are proposals in https://github.com/one-data-model/playground/pull/48

➔ Is this feature sufficiently stable to be in SDF 1.1?

Comments welcome.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Mar 2021 18:38:36 +0100

> ➔ Is this the right semantics?

The weirdness that may make this text hard to understand is that the number given by minimum and maximum really is a data model level number (often a scale integer).  scaleMinimum and scaleMaximum tell you how to extract the information model level value (the actual value in the unit given) from that.

That is a bit backwards for a model that tries to be an information model.

But, more importantly, an ecosystem specific mapping may want to add information of this kind, and for this, it would be better if the converged SDF model were about the information model level.

So here is my plan for (at least some of the information in) mapping files:
(1) The mapping file can augment the converged SDF model with ecosystem specific information, such as the mapping from the abstract numbers describing a property or other affordance to platform specific data types.
(2) Some of these mappings are so universal (everybody agrees on them), we’d actually want to write them into the converged model.

I’d like to be able to use the same SDF qualities for (1) and for (2).

But scaleMinimum/Maximum are not good qualities to add after the fact in a mapping file (for (1)):  Only they tell you what range needs to be represented in a mapping.
So, really, scaleMinimum, scaleMaximum, and unit are the information model level qualities, and minimum and maximum as used by the definition in PR#22 degrade to mapping file (ecosystem-specific data model) level qualities.

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 13:55:39 -0800

The canonical example of the pattern being modeled here, is where the device represents a decimal scale using an integer with a fixed point multiplier e.g. 0.001 . In some cases, there is also an offset (added/subtracted).

This pattern is very common and really needs to be modeled in order to represent Zigbee and Modbus devices without making some hidden assumptions about every value that isn't a directly mapped integer. Without some way to describe the semantics of the information model, we can only describe the raw data model.

There are some other examples where this pattern can also be used to model conversion from one quantity to another e,g, Fahrenheit to Celsius.

I have looked at multiplier+offset schemes but these don't seem to be as clear as this pattern. This works well for the common use cases mentioned above.

To clarify how it works: There is a raw value or input value that has minimum, maximum, and multpleOf specified. There is a scaled or output value that has a scaleMmimum, scaleMaximum, and (engineering) unit. Sometimes this is called display value, or engineering value, etc. 

The engineering unit is always associated with the scaled, or output, value when scaleMinimum and scaleMaximum are used. scaleMaximum might be used by itself if both scales are zero-based.

Other names for scaleMinimum and scaleMaximum might be unitMinimum and unitMaximum, engineeringValueMinimum, etc. 

A couple more points of reference:

I have a Modbus PID controller that takes standard raw analog input scale values (4-20mA, 1-5V, etc) and converts them to "Engineering Data" using "Engineering Range Low Value" and "Engineering Range High Value". For example, if I want to map a 4-20 input to 700-1100 degrees celsius, I define Engineering Range Low Value = 700 and Engineering Range High Value = 1100, and the instrument then outputs "Engineering Data" to be displayed in Celsius, 4mA = 700C and 20mA = 1100C with linear interpolation in between.

Node-red has a standard "range" node that does the came calculation to map one numeric range to another. They call it input range and target range which is more generalized.

Map the input range:
 from:    to: 
 to the target range:
 from:    to: 

  Round result to the nearest integer?

So maybe we need to change the name of the qualities, or do a better job of explaining what it is for and how to use it. 

I can write up the text we need for the md file.

PS
I also would like to be able to add scale information to SenML units, bc there is an information model loss when you have to say 120000 volts when you really want kV to be displayed, likewise .002 amperes when you want uA or mA to be displayed.

See my example modbus application for use case examples:
http://71.202.145.92:1880/#flow/590c69.67079398 <http://71.202.145.92:1880/#flow/590c69.67079398>
And an example node-red dashboard:
http://71.202.145.92:1880/ui <http://71.202.145.92:1880/ui>

> On Mar 7, 2021, at 9:38 AM, Carsten Bormann <[email-masked]> wrote:
> 
>> 
> -- 
> ASDF mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/asdf

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 14:14:28 -0800

Another point to be made here, is that all of these constraints are usually applied in the binding, when the SDF model is mapped to an implementation. In the base model we might only indicate number and that it is a Voltage unit, the assignment of raw values *and* scaled values are done in the binding.

@carsten, I think there is some question about using "multipleOf=1" instead of "integer". One generalization of the formula uses the multipleOf value, so if you say "integer" it would need to map that to multipleOf=1 anyway. I think we need to handle both forms in the tools. Is there a best practice, independent of case by case sensbility?

Best regards,

Michael

> On Mar 7, 2021, at 1:55 PM, Michael Koster <[email-masked]> wrote:
> 
>> On Mar 7, 2021, at 9:38 AM, Carsten Bormann <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>>> 
>>> https://github.com/ietf-wg-asdf/SDF/pull/22 <https://github.com/ietf-wg-asdf/SDF/pull/22>
>>> 
>> [email-masked] <mailto:[email-masked]>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Mar 2021 23:21:44 +0100

> @carsten, I think there is some question about using "multipleOf=1" instead of "integer". One generalization of the formula uses the multipleOf value, so if you say "integer" it would need to map t…

Let’s do this one first, because it is easy :-)

In many cases, the whole point of the scaling is to go to integer data items.
In that case, simply saying that makes more sense to me.

If there is some application semantics to the multipleOf, of course go for it.

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 14:26:26 -0800

Yes, in my examples, the "type" of the raw data is integer but the "type" of the scaled data is decimal. 

In this case "number" covers both raw and scaled data types.

Best regards,

Michael

> On Mar 7, 2021, at 2:21 PM, Carsten Bormann <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Mar 2021 23:42:29 +0100

> Another point to be made here, is that all of these constraints are usually applied in the binding, when the SDF model is mapped to an implementation. In the base model we might only indicate numbe…

One of the reasons for doing a data model is that we want to be able to interpret data when it has become detached from the physical device.  In which case you are better off when the value given is already referenced to a unit.  This was the idea behind SenML.

Now if I have a 4–20 mA ADC, that may be all the data I get, so I need some context information to know the range that this actually stands for.
This is one of the points of adding a mapping (binding): Being able to provide this context.

So we have:

— data item transferred: 0..65535
- measured value: 4..20 mA
— indicated sensor reading: 700..1100 K

(This ignores off-scale detection etc.)

I do like being able to put down the ranges to be mapped and essentially use the formula in https://raw.githack.com/ietf-wg-asdf/SDF/scaleminmax/sdf.html#section-4.7-10 for computing the linear transform.

The data model (really: information model) of a temperature sensor should talk about 700..1100 K.  The data model of an instrumentation ADC should talk about 4..20 mA and rely on additional information given by the usage context mapping (binding) to derive the 700..1100 K.  The mapping (binding) information to a bluetooth characteristic may tell you this range is represented by 0..65535.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Mar 2021 23:52:12 +0100

> Yes, in my examples, the "type" of the raw data is integer but the "type" of the scaled data is decimal. 

(I hope it is not “decimal” but a fraction, really a rational value, which we tend to represent with binary floating point values or sometimes fixed point values.)

> In this case "number" covers both raw and scaled data types.

Ouch.

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 15:06:10 -0800

Yes, decimal fraction is correct.

"number" works for all cases but prevents one from correctly using other types when it is semantically useful. There isn't a type for decimal fraction/ fixed point but "number" with multipleOf seems to handle it well enough so far. 

"scaleType" ?

> On Mar 7, 2021, at 2:52 PM, Carsten Bormann <[email-masked]> wrote:
>

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 15:15:01 -0800

I think we're in agreement. My 4-20 mA example is the ADC rationalization that happens in the PID controller, and the Modbus decimal value scaling is what happens in the client. The model works for both cases, and I do have some SDF models for building C++ code that scale raw a/d counts to values using the same vocabulary. There is a case where one SDF file describes the a/d scaling to a modbus (decimal) integer, then another to scale the modbus value to the correct decimal scale.

"ihv_counts" : {
  "type": "number",
  "minimum": 0,
  "maximum": 1024,
  "multipleOf" :1,
  "scaleMinimum": 0,
  "scaleMaximum": 2000
}

        "IHVData": {
          "type": "number",
          "units": "A",
          "minimum": 0,
          "maximum": 2000,
          "multipleOf": 1,
          "scaleMinimum": 0,
          "scaleMaximum": 0.002
        },

> On Mar 7, 2021, at 2:42 PM, Carsten Bormann <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 8 Mar 2021 00:16:10 +0100

> Yes, decimal fraction is correct.

I still hope there is nothing fundamentally “decimal” about the fraction.

A fixed point number is essentially represented by the numerator while you know the denominator.  (The latter may or may not be a power of 10, which would make it “decimal”.)
This is not actually different from a scaled integer.

> "number" works for all cases but prevents one from correctly using other types when it is semantically useful. There isn't a type for decimal fraction/ fixed point but "number" with multipleOf seem…

JSON numbers are binary floating point, so I don’t think they can represent fractions with a denominator that is not a power of 2.

If we can liberate ourselves from the idea that “number” is a JSON number, this becomes easier.

> "scaleType" ?

Not sure which part would be the “scaleType" here — the unscaled or the scaled one?

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 15:32:36 -0800

Not fundamentally; the Modbus example happens to be an integer scaled by powers of 10.

> On Mar 7, 2021, at 3:16 PM, Carsten Bormann <[email-masked]> wrote:
>

---

From: Michael Koster <[email-masked]>
Date: Sun, 7 Mar 2021 15:33:10 -0800

scaledValueType

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 8 Mar 2021 08:36:50 +0100

So why do we need two SDF property definitions to represent this single property?

We probably want to have all three qualities:

dataMinimum..dataMaximum (here 0..1023), as an integer (“multipleOf = 1”)
measurementMinimum..measurementMaximum (here 0..2000), as an abstract value only (compute the multipleOf from these four corners)
quantityMinimum..quantityMaximum (here 0..2e-3), again as an abstract value only

There may be no need to actually express measurementMinimum..measurementMaximum unless the whole thing is composed from a data model and a mapping/binding.

How would that composition split?

If the data model knows this is about a specific quantity (say temperature), it would give the unit for that quantity.
The actual quantityMinimum..quantityMaximum may be a specific property of the sensor, or it may be hardcoded in the model.

If the SDF model only has the measurement level (measurementMinimum..measurementMaximum), then it becomes (contextual) binding information how to obtain the unit, quantityMinimum..quantityMaximum.

(The below example says 0..1024, which is also interesting, because in practice you wouldn’t get 1024, only 1023, out of a 10-bit ADC.  
In the linear transform, 1024 might still map to 0.002, so that quantity value would actually be off-scale!)

Grüße, Carsten

> On 8. Mar 2021, at 00:15, Michael Koster <[email-masked]> wrote:
> 
>