DMF (Data Mapping File) spec

Tracker: Add support for external Data Mapping File #182

Version: v0.9

See also: DMF Implementation Status and DMF Implementation notes

Revision history

v0.9: Initial proposal

Introduction

NUT generic drivers (SNMP, XML/PDC, USB/HID, and upcoming Modbus) hold hard-coded textual data that map NUT variables to the native protocols ones. As an example, each interesting SNMP OID (data) has an entry (line) in the snmp-ups driver, or more precisely in a subdriver implementation file (such as eaton-mib for Eaton ePDUs G1 and G2 declarations) that instructs the driver how to retrieve and process this data.

{ "device.model", ST_FLAG_STRING, SU_INFOSIZE, ".1.3.6.1.4.1.534.6.6.7.1.2.1.2.0", "Eaton Powerware ePDU", SU_FLAG_STATIC | SU_FLAG_OK, NULL, NULL },

Reference: https://github.com/networkupstools/nut/blob/master/drivers/eaton-mib.c#L240

The above example maps the SNMP OID (numeric format) ".1.3.6.1.4.1.534.6.6.7.1.2.1.2.0" to the NUT variable "device.model". It also instruct the snmp-ups driver that this is a static string (i.e. retrieved only once at startup time). Also, in case of failure to retrieve the SNMP OID value, the default value "Eaton Powerware ePDU" will be published by NUT.

Hard-coded means that these declarations are located in implementation files (.c). So each data addition requires recompilation of the driver, which is painful for a user and developer point of view.

As a corollary, holding these declarations in the driver binary implies that these drivers are growing (in size) very fast. So moving these information out of the driver binary will additionally improve the driver footprint and efficiency, which is desirable.

These data should be moved out of the drivers source code, into some text files, so that these data can be created and edited without the need of recompiling NUT drivers.

Data Mapping Files specification proposal

This is the consolidated proposal of the Data Mapping Files (DMF) specification. It takes into account the various comments, along with REQs and EXs requirements, both from the NUT Community and Eaton, in a consolidated form.

Notes:

• Provided examples are based on eaton-mib (Eaton ePDUs G1 and G2 declarations for snmp-ups),
• The selected implementation format is JSON, for its flexibility and extensibility. Moreover, it's aligned with the state-of-the-art, and permits a wide consolidation of requirements. Other considered formats were TOML and NUT historic INI style.

Notes on the scope of DMF

Data mapping files will in the end have to cover requirements for all the generic NUT drivers, either currently existing or upcoming (REQ requirements). That is to say:

• snmp-ups (existing)
  *   Note: this driver is scheduled for rewrite ([NUT Github #20: SNMP driver Gen3](https://github.com/networkupstools/nut/issues/20))
  

• netxml-ups (existing)
  *   Notes: this driver is scheduled for evolution ([NUT Github #188: Add support for Eaton XML/PDC v4](https://github.com/networkupstools/nut/issues/188))
  

• nutdrv_modbus (not yet existing)
  *   Note: this driver is scheduled for implementation ([NUT Github #50 Modbus driver](https://github.com/networkupstools/nut/issues/50))
  

• usbhid-ups and mge-shut (existing)
  

This will probably imply some drivers evolution, which is for example scheduled for the SNMP driver. Otherwise, the loader function of the driver (or common function) will take care of the suitable transformation to match the currently available data structures.

Moreover, some additional inputs and requirements were considered (Eaton EX (extended / optional) requirements).

Global section

The global section provides generic information on the Data Mapping File, and its applicability. It is partially equivalent to "mib2nut_info_t" for "snmp-ups", and similar structures in other drivers.

Generic entries (applicable to all drivers)

Note: all these fields are mandatory!

Field	Details	Example(s)
Name	Name of the Data Mapping File	Eaton (Aphel) Genesis II MIB (monitored ePDU)
Identifier	Single word identifier of the DMF	eaton_aphel_genesisII
Version	Version of the Data Mapping File for the specific device	0.46
DMFVersion	Version of the Data Mapping File specification used	1.0
Protocol or DriverName	Type of protocol, used by the drivers to match if appropriate	SNMP, USB/HID, XML/PDC FIXME: Decide on wether Protocol or DriverName (or both?!). DriverName is probably more suitable, since being more focused, while providing indirectly the supported Protocol
Description	Description of the Data Mapping File	Definitions to monitor Eaton and Aphel ePDUs using Genesis II SNMP MIB
Author	Author(s) and Copyright information	Arnaud Quette arnaud.quette@gmail.com Arnaud Quette ArnaudQuette@Eaton.com
License	License information	GNU GPL v2+

Possible additions to generic entries

Note: these field(s) are optional!

Field	Details	Example
ExtensionLibrary	Name or path to a library (.so, .dll) that provides additional callback functions to process data. For more details, refer to On the conversion functions	nutdrv_snmp_eaton_genesis.so
Include	Directive to include an existing DMF into another one. The use case is that a DMF may already exist in NUT, but with limited scope (entries). And Eaton can extend this DMF with another one, possibly under a different license.	in Eaton_genesis_advanced (Eaton version) => Include Eaton_genesis_(NUT version)

Other directives are under consideration, such as:

• Replace: instead of extending an existing DMF, we may want to replace it. Rationale: instead of extending an existing DMF shipped with NUT, we may want to fully override it. i.e. fully replace the provided DMF by another implementation. In this case, the Include directive would not be sufficient.

Specific entries (applicable to specific drivers)

Specific global entries are mainly used for the initial matching of the right DMF by the drivers, or by the nut-scanner. Depending on the protocol, different strategies are used.

SNMP driver

For SNMP, the discovery / DMF matching strategy is that:

• We try first to match the "sysObjectID", when provided.
  

• If "sysObjectID" is missing, we use the provided "device.model" entry to match the DMF.
  

• This requires a general manifest file that provides these information, as it is currently with "mib2nut_info_t".<br>The creation of this manifest file requires a script that iterates over all DMFs, and automatically create this file using the DMF information.
  

Note: all these fields are mandatory!

Field	Details	Example
sysObjectID	systemObjectID, allowing to detect if the DMF is applicable to a target (or discovered device) Note: This can also be a list of sysOIDs.	.1.3.6.1.4.1.17373

XML/HTTP (XML/PDC) driver

...To be completed

USB/HID driver

...To be completed

Modbus driver

...To be completed

Data mapping entries

General considerations

Data Mapping entries are the actual mapping between protocol specific data and NUT namespace.

These entries MUST list the following information to be useful:

• protocol-specific-variable: for example, an SNMP OID, XML/PDC data, ...
• NUT variable name: the NUT data that will receive and store the protocol-specific-variable value
  • Reference: NUT command and variable naming scheme

    •    [Formatted version](http://www.networkupstools.org/docs/developer-guide.chunked/apa.html)
      

    •    [Plain text version](https://github.com/networkupstools/nut/blob/master/docs/nut-names.txt)
      

  • Note: in case there is no suitable variable in NUT namespace, an RFC must be issued to the NUT project to propose a new variable and get an approval.
• flags: information related to the type of data, so that the NUT driver knows how to process it. It can be:
  • a variable (value only),
  • a command (stop an outlet, launch battery test, ...)
  • or a setting (change a parameter in the device)
• Data value lookup information: a reference to the below presented value lookup structure, to transform the protocol-specific-variable value into NUT standard compliant ones.

General format specification

Field	Details	Comments	Allowed values	Example value
NUTVariable	NUT variable name		String value.	device.model
ProtocolVariable	Protocol variable name		String value.	Protocol dependent: * SNMP: .1.3.6.1.4.1.17373.3.1.1.0" * XML: "System.Description" * HID: N/A in its current form
NUTFlags	NUT general data flags (used by dstate_setflags())	Relates to nut/include/extstate.h #define ST_FLAG_RW 0x0001 #define ST_FLAG_STRING 0x0002 #define ST_FLAG_IMMUTABLE 0x0004 Extended (FFE: For Future Extension)	String value. Data type related: * _Integer _(default if nothing specified), * String, * Float Data access related: * _ReadOnly _(default if nothing specified), * ReadWrite * Immutable * _Sensitive _(value published as ****, apart if user logged with right privileges ; FFE)	"String, ReadWrite"
NUTVariableLength	NUT general data length (used by dstate_setaux()) i.e. maximum length of the string	NUT checks for max. 256 chars for ST_FLAG_STRING	Integer value	32
DriverFlags	Drivers specific flags	Usefulness needs to be asserted WRT_NUT_variable_flags_ FIXME: Consider merging both to simplify{color}	String value.	Static
DefaultValue	Default value if the data can't be retrieved	If DefaultValue is present, and the data can't be retrieved, the data will still be published by NUT. Otherwise, if omitted, the data will not be published.	String value.	"Eaton Powerware ePDU Monitored"
ValueLookup	A named reference to the value lookup structure		String value. Can be: * _NULL (empty), i.e. not specified * an existing callback function, either: ** in the driver (built-in), ** in the extension library, ** as inlined-code (embedded script)

JSON format specification

"<NUT variable>" : {
  "ProtocolVariable" : "...",
  "NUTFlags" : "...",
  "NUTVariableLength" : "...",
  "DriverFlags" : "...",
  "DefaultValue" : "...",
  "ValueLookup" : "..."
}

Example (SNMP)

"device.model" : {
  "ProtocolVariable" : ".1.3.6.1.4.1.17373.3.1.1.0",
  "NUTFlags" : "ST_FLAG_STRING",
  "NUTVariableLength" : "32",
  "DriverFlags" :  [ "SU_FLAGS_STATIC", "SU_FLAGS_ABSENT", "SU_FLAGS_OK"],
  "DefaultValue" : "Eaton Powerware ePDU Monitored",
  "ValueLookup" : { "..." }
}

Value lookup structure

General considerations

These data structures are used to process values from the protocol to NUT, and vice-versa.

Beside from simple 1:1 value mapping, we can have to:

• post-process data, to adapt the published value,
• condition data publication depending on other variables, such as:
  • the "device.model", or any other NUT data obtained using "dstate_getinfo()"
  • multiple mapping entries. If already published, don't override for example.
• ...

In details:

• Data values lookup structure in "snmp-ups" are basically simple key-value pairs,

• JSON format would be suitable,

• However, "snmp-ups" is still a G1 generic driver. Value lookup structure is still a simple keypair.

• The data values lookup structure in "usbhid-ups" (generic driver G2), called "info_lkp_t" is more complete, with back-and-forth conversion callbacks:

  const long hid_value; /* HID value */
  const char nut_value; / NUT value */
  const char *(fun)(double hid_value); / optional HID to NUT mapping */
  double (*nuf)(const char nut_value); / optional NUT to HID mapping */
  

• The lookup structure MUST have the same kind of mapping

  • protocol-value
  • nut-value
  • conversion function from protocol-value to nut-value: to transform the value from the protocol to NUT (i.e. when reading / acquiring data)
  • conversion function from nut-value to protocol-value: to transform the value from NUT to the protocol (i.e. when sending setting or command to the device)

On the conversion functions, extension libraries and Inlined-code

Conversion functions (or callbacks) are functions to process data, or condition it (i.e. conditional processing / publication of data) or whatever, that are stored in the DMF (either directly or indirectly, see below for details). There are several ways to address this point:

• Use built-in functions, that are available in the driver binary code
• Provide ways to add additional code, part of the Data Mapping Files, through:
  • Inlined-scripting.
    • Mention
      • the function (callback) name
      • the language interpreter (bash, perl, python)
      • the code snippet
    • pros is that it's simple for users
    • cons is that it's not as fast nor efficient as native code (either from driver or extension libraries)
  • Extension libraries (.so...)
    • mention a lib name / path that implements the function
    • pros: fasts / efficient, cons: requires coding and compilation
  • there are then lua and alike
  • and we can still not limit and provide choices...

General format specification

Field	Details
ProtocolValue	Protocol specific (raw) value
NUTValue	NUT specific (transformed) value
ProtocolConversionFunction	aka fun, a callback function to provide advanced value processing from the protocol to NUT
NUTConversionFunction	aka nuf, a callback function to provide advanced value processing from NUT to the protocol

Examples don't exist currently for "snmp-ups" (due to generic drivers G1 limitation). For an advanced example using the 4 fields, check Eaton ECO mode feature in "usbhid-ups":

• context
  • this feature allows to enable master / slave ECO mode on specific models.
  • It limits standard "yes/no" info publication to "device.model" = Protection Station, Ellipse Eco and 3S (US 750 and AUS 700 only!), and enable special RW flag used in "hid_info_t" entries
• pegasus_yes_no_info
• fun / nuf declarations
• Data mapping entry

JSON format specification

"<value lookup structure name>" : {
  "ProtocolValue" : "...",
  "NUTValue" : "...",
  "ProtocolConversionFunction" : "...",
  "NUTConversionFunction" : "..."
}

Example (SNMP)

---

Do we actually need named fields, or positional tuple is enough?
I.e:
{ "ProtocolValue" : 0,
"NUTValue" : "off" },
vs.
[ 0, "off" ],
or [ 0, "off", , ],
I know that, with named fields, we can omit any specific (like the first two), and still have the last two.
However, the general case is the 2 firsts...

---

Complete example

Note: The markup below is at the moment not syntactically correct JSON, since comments are used and objects are not comma-separated, and tuples may seem like objects; this is currently rather a bunch of snippets with different items that need to be in a DMF file to help illustrate the subject.

{
"Name" : "Eaton (Aphel) Genesis II MIB (monitored ePDU)",
"Version" : "0.46",
"Description" : "Definitions to monitor Eaton and Aphel ePDUs using Genesis II SNMP MIB",
"Author" : ["Arnaud Quette <arnaud.quette@gmail.com>", "Arnaud Quette <ArnaudQuette@Eaton.com>"],
"License" : "GNU GPL v2+",
 
"sysObjectID" : ".1.3.6.1.4.1.17373"
}
{
 // Includes RO / RW variables
 "Variables" : [
	"device.mfr" : {
	  // "Protocol_variable" : "...", // Omitted, hard coded data
	  "NUT_variable_flags" : "ST_FLAG_STRING",
	  "NUT_variable_length" : "...",
	  "Driver_flags" : [ "DRV_FLAGS_STATIC", "DRV_FLAGS_ABSENT" ],
	  "Default_value" : "EATON"
	},
	"device.type" : {
	  // "Protocol_variable" : "...", // Omitted, hard coded data
	  "NUT_variable_flags" : "ST_FLAG_STRING",
	  "NUT_variable_length" : "3",
	  "Driver_flags" : [ "DRV_FLAGS_STATIC", "DRV_FLAGS_ABSENT" ],
	  "Default_value" : "pdu"
	},
   "outlet.%i.status" { // "%i" may better be explicit, i.e. $OutletNumber, may need scoping like ${OutletNumber}
	  "Protocol_variable" : ".1.3.6.1.4.1.534.6.6.7.6.6.1.2.0.%i",
	  "NUT_variable_flags" : [ "ST_FLAG_STRING" ],
	  "NUT_variable_length" : "5",
	  "Driver_flags" : [ "DRV_FLAGS_OUTLET" ], // For the driver to address "%i" or similar
	  "ValueLookup" : "marlin_outlet_status_info"
   }
   ...
 ]
}
{
 // Commands
 "Commands" : [
   ...
 ]
}
{
 "ValueLookup" : [
	"marlin_outlet_status_info" : {
	  { 0, "off" },
	  { 1, "on" },
	  { 2, "pendingOff" }, // transitional status
	  { 3, "pendingOn" },  // transitional status
	  { 0, NULL }          // FIXME: is the termination element needed here? Don't think so!!!
   }
 ]
}

Implementation notes

Refer to Implementation notes