MQTT and Modbus does not meet the architecture eligible criteria for a protocol binding

[relu91]

As @benfrancis rightly pointed out here:

-1 for domain-specific profiles or those using non-web protocols like Modbus and MQTT (on the basis that they don't meet the eligibility criteria for a WoT transfer protocol)

Both Modbus and MQTT sadly do not meet the eligibility criteria that we have defined in architecture:

Eligible protocols for W3C WoT MUST have an associated URI scheme [RFC3986] that is registered with IANA (see [IANA-URI-SCHEMES]). Hypermedia controls rely on URIs [RFC3986] to identify link and submission targets. Thereby, the URI scheme (the first component up to ":") identifies the communication protocol to be used for Interaction Affordances with the Thing. W3C WoT refers to these protocols as transfer protocols.

The problem is that both mqtt: and modbus+tpc: are sadly not registered in IANA-Registry. My humble opinion is to understand the process for the registration and propose them ourselves. Since these two protocols gained interest from the industry I would not recommend removing them completely just for this missing registry entry.

[egekorkan]

Some points:

• I want to remove that assertion from the architecture spec. Assertion review wot-architecture#625
• MQTT cannot have a URI scheme since their standardization group does not want to define one. I need find the source of this but basically, MQTT does not define a transfer protocol. You can use it over TCP (most usual one), UDP, Websockets and the URI scheme can just change. See https://github.com/mqttjs/MQTT.js#connect . I find this very annoying but it is what it is.
• I think the comment from @benfrancis is not the fact that they are not registered but that these are from the very beginning not compatible with WoT. I disagree with this. Of course, they are not usual Web protocols but the design of WoT shows that they can be used in a very similar way to Web protocols from the Consumer application logic point of view. I think this is a powerful statement that no other standard is able to achieve afaik.

[relu91]

Thank you Ege for the fast response, I wasn't aware of the fact that you're already tackling this issue in w3c/wot-architecture#625. I found that assertion reasonable, but I totally agree with the fact that one selling point of WoT is the ability to plug any IP-based IoT protocol. I would not lose that for sure.

As far as understood that assertion is from our work for 1.1 so it is not yet normative. I would be ok removing it or at least re-phrasing it. I think this is possibly related to our discussion about having a registry of allowed protocols 🤔

[benfrancis]

@egekorkan wrote:

• I think the comment from @benfrancis is not the fact that they are not registered but that these are from the very beginning not compatible with WoT. I disagree with this. Of course, they are not usual Web protocols but the design of WoT shows that they can be used in a very similar way to Web protocols from the Consumer application logic point of view. I think this is a powerful statement that no other standard is able to achieve afaik.

That's correct, I don't think they should be considered in scope for the Web of Things.

Firstly, if a protocol doesn't have a URI scheme (whether registered with IANA or not), then how can it be described by a form? You can make up your own unofficial URI scheme, but it may not work for existing implementations of the protocol.

Secondly (and more controversially), in my opinion the Web of Things should only use web protocols. Currently the only web protocols I am aware of are HTTP (including HTTP/2 and extensions like SEE and WebSockets, although WebSockets is arguably a bit of a stretch) and CoAP (including extensions like CoAP Observe).

MQTT and Modbus are not just "not usual web protocols", they are completely unrelated to the World Wide Web. They are IoT protocols, not WoT protocols and therefore, in my opinion, should be out of scope for the Web of Things and the work of the World Wide Web Consortium. Please see my explanation below if you'd like to know more about why I take this position (I'm not just being dogmatic, it's about the way we achieve interoperability).

@relu91 wrote

As far as understood that assertion is from our work for 1.1 so it is not yet normative. I would be ok removing it or at least re-phrasing it. I think this is possibly related to our discussion about having a registry of allowed protocols

I'm afraid this assertion also appears in WoT Architecture 1.0 and is therefore already a W3C recommendation. This was the result of a long debate, and as I remember it was a kind of compromise between two extreme positions:

1. A WoT Thing Description should be able to describe any protocol, including non-web protocols (like MQTT and AMQP) and non-internet protocols (like Zigbee and Z-Wave)
2. A WoT Thing Description should only use HTTP

I would also argue that it's an inevitable outcome of the choice to use hypermedia forms as the mechanism by which a Thing Description describes a protocol binding. You can't describe a protocol using a form if it doesn't have a URI scheme.

Whilst not everyone agrees with my position on non-web protocols, I hope we can agree that a protocol has to have a URI scheme in order to be described by a form. This is really just a statement of fact since href is a mandatory member of a Form which has to be a URI, and forms are currently a mandatory member of an InteractionAffordance.

---

Some further background below for anyone who is interested.

From the very beginning of the Web of Things activity at the W3C I think there have been two schools of thought about how to achieve interoperability on the Internet of Things using the Web of Things.

1. By using forms in Thing Descriptions to describe to a Consumer how to communicate with an IoT device using any existing and future IoT protocol, like OpenAPI but for any protocol.
2. By providing a common data model and protocol (ideally only one) for Consumers to communicate with devices, with the Web of Things as an application layer protocol for the Internet of Things like the World Wide Web is for the Internet.

The way I see it is that currently the Working Group is pursuing both approaches:

1. The former approach through Protocol Binding Templates, which aim to provide a vocabulary for describing each individual protocol. This approach uses declarative protocol bindings which are provided by forms in a Thing Description and dynamically interpreted by a Consumer at runtime. This approach can be applied directly to brownfield devices but requires that Consumers implement each individual protocol.
2. The latter approach through Profiles, by defining a fixed protocol binding for communicating with a device over the web using the Core Profile (using HTTP, but with the potential for a constrained profile using CoAP in the future). This approach uses a concrete protocol binding which is provided by a specification which is then implemented by a Consumer. This approach can be applied directly for greenfield devices and Consumers only need to implement one or two protocols, but it requires a gateway or cloud service to bridge brownfield devices).

My personal opinion continues to be that the former approach is not a viable long term option for enabling interoperability, for two reasons:

1. I don't believe it's feasible for a single JSON file format to be able to describe any possible device interaction using any possible existing or future protocol. This is demonstrated by all the existing APIs (even simple things like action queues) which still can't adequately be described by a Thing Description. Continuing down this road will result in an ever growing vocabulary which will eventually become unmanageable.
2. I believe it actually works against interoperability because in order for ad-hoc interoperability to be possible, every Consumer would have to implement every possible IoT protocol, which just isn't feasible. The long term result of this will be that rather than one Web of Things, we'll have lots of separate Webs of Things, each using a different protocol, which can't communicate with each other. This will just add to the problem we're trying to solve by cementing the existing IoT silos.

I know not everyone agrees with this position, and I accept that the Working Group will continue to pursue both approaches (declarative protocol bindings through Protocol Binding Templates, and concrete protocol bindings through Profiles). But for me, trying to shoehorn in protocols like MQTT and Modbus which don't even have a URI scheme is a step too far.

[relu91]

I'm afraid this assertion also appears in WoT Architecture 1.0 and is therefore already a W3C recommendation. This was the result of a long debate, and as I remember it was a kind of compromise between two extreme positions:

That's was my fear and that's why I open this issue, however, @egekorkan answer got me thinking that maybe I misunderstood the situation.

Whilst not everyone agrees with my position on non-web protocols, I hope we can agree that a protocol has to have a URI scheme in order to be described by a form. This is really just a statement of fact since href is a mandatory member of a Form which has to be a URI, and forms are currently a mandatory member of an InteractionAffordance.

Same as above, the constraint described in the architecture seems reasonable. What I sadly don't want to do is to drop MQTT and other relevant IoT protocols. We have a good amount of use cases that employed these protocols plus we already received industrial feedback about using them as a WoT protocol. I'd say we all have our points and I see pros and cons with both approaches actually but IMHO the group has reached a good balance between the two positions that @benfrancis recalled in the answer above. We should use the spec that we wrote as a guide and play alongside.

Concerning MQTT issue, can't we have a dedicated scheme for each transport? like in the Modbus community has modbus+tcp or modbus+rtu? If it does not really work I would say it is a major blocker for having it as an official protocol binding.

---

I don't want to go too much out of topic here but since we are sharing ower own perspective about how we should deal with IoT interoperability here's my answer to these points:

1. I don't believe it's feasible for a single JSON file format to be able to describe any possible device interaction using any possible existing or future protocol. This is demonstrated by all the existing APIs (even simple things like action queues) which still can't adequately be described by a Thing Description. Continuing down this road will result in an ever growing vocabulary which will eventually become unmanageable.
2. I believe it actually works against interoperability because in order for ad-hoc interoperability to be possible, every Consumer would have to implement every possible IoT protocol, which just isn't feasible. The long term result of this will be that rather than one Web of Things, we'll have lots of separate Webs of Things, each using a different protocol, which can't communicate with each other. This will just add to the problem we're trying to solve by cementing the existing IoT silos.

I believe that being open to different protocols does not really mean that all Consumers out there will implement all IoT protocols. Companies, users, and even programmers have the interest that their devices/services are easily used, but on the header hand, they might have different target users/use cases that can't be covered by one/or two "web" protocols (I am using " just because here I'm treating every protocol as equal). In the end, I foresee a market that finds the best set of protocols suited within the frame of the Thing Description and WoT Architecture model.

Anyhow, I think that Profile work is also important but merely as a means to speed up acceptance and implementation.

[jyhi]

MQTT and Modbus are not just "not usual web protocols", they are completely unrelated to the World Wide Web. They are IoT protocols, not WoT protocols and therefore, in my opinion, should be out of scope for the Web of Things and the work of the World Wide Web Consortium.

IMHO it's like WebAssembly -- it's designed for the Web, but is also considered useful in many non-Web situations. Saying "Web of Things" perhaps doesn't always mean that the Things are on the Web (of course this is the best); they could simply be only described in Web standards.

From my understanding, a WoT TD is a standalone descriptor of an IoT device; think of a USB descriptor. By saying descriptor one doesn't assume its form of existence -- it might be a JSON-TD document on the Web, or it might be a bunch of data stored in a database. However, someone needs to know exactly how to read it, so what we're doing is defining a basic way to present the descriptor, so that a Consumer (client) can parse it and understand the capabilities of an IoT device. The Consumer doesn't need to actually operate on the described Thing, so sometimes interoperability may not be that important. For example, Digital Twins. When it comes to interoperability, since TD has an abstract model, we need to specify the mapping from the abstract verbs (e.g. readproperty) to the protocol-specific ones (e.g. GET); they are Protocol Bindings.

One neither assumes how a descriptor is distributed. The best way of course would be an automatic discovery, but what if we prefer an alternative way to distribute it. For example, a TD directory, a QR code, or a completely different URL pointing to a TD in which the endpoints are at another different place. As a TD is simply a descriptor, it doesn't always need to stay on the thing, as long as it represents that thing. However, we know that a Web-enabled IoT device would be very great, so we have WoT Profiles putting constraints on the concrete protocols to use to enable "ad-hoc interoperability".

If we say TD can describe IoT devices, then we have to make it expressive enough to describe any IoT devices -- including both the "brown-field" and the "green-field" ones. Certainly, there are still details that a WoT TD cannot describe, like the undefined URL schema. I don't think excluding these undefined things a good idea; in contrast, we should try pursuing their standardization, either through talking to the people in charge of them or by doing them ourselves.

My two cents worth.

[sebastiankb]

In my opinion, this assertion is not perfectly formulated and leads to a contradiction of the document itself. E.g., MQTT is mentioned in many places as an example to be supported by WoT.

Instead to reopen the never ending protocol discussion again, I would suggest how we can overcome the situation. The work for 1.1 is also intended to improve the documents. I see here the need to reformulate the assertion to be consistent by simple adding:

Eligible protocols for W3C WoT MUST have an associated URI scheme [RFC3986] that is registered with IANA (see [IANA-URI-SCHEMES]) or is directly given by the specification of the origin protocol (e.g., MQTT, Modbus, OPC-UA, etc.).

A simpler solution would be to remove the assertion. Doing this, we need the result from this discussion first w3c/wot-architecture#625

[benfrancis]

@relu91 wrote:

I believe that being open to different protocols does not really mean that all Consumers out there will implement all IoT protocols.

If a different Consumer is needed for each IoT protocol, then how does that "counter the fragmentation of the IoT"?

@lmy441900 wrote:

Saying "Web of Things" perhaps doesn't always mean that the Things are on the Web
...
One neither assumes how a descriptor is distributed.

If a Web Thing isn't described on the web and isn't communicated with via the web, then what makes it the "Web of Things", as distinct from the "Internet of Things"?

From my understanding, a WoT TD is a standalone descriptor of an IoT device; think of a USB descriptor.

If Consumers have to be protocol-specific anyway, then what incentive does a developer have to provide a WoT Thing Description rather than the protocol-native mechanism for describing device capabilities (like a USB descriptor)?

The Consumer doesn't need to actually operate on the described Thing, so sometimes interoperability may not be that important.

I agree that if you don't care about interoperability or about communicating with a device then the protocol is not important, but then why choose WoT at all?

@sebastiankb wrote:

I see here the need to reformulate the assertion to be consistent by simple adding:

Eligible protocols for W3C WoT MUST have an associated URI scheme [RFC3986] that is registered with IANA (see [IANA-URI-SCHEMES]) or is directly given by the specification of the origin protocol

As I understand it the problem is not just that the URI schemes are not registered with IANA, but that the "specification of the origin protocol" doesn't define a URI scheme at all.

[egekorkan]

We can further go into this discussion which is a bit philosophical in my point of view but there are already valid use cases on using other protocols and treating them from the Web point of view. Web does not necessarily mean that we are using protocols that are commonly used on the Web (from my point of view). Web in its essence is about linking resources that are represented via URIs where URI does not mandate a certain URI scheme. Even browsers used to support more protocols and they died out, which can also happen in the IoT but it does not seem to be the case at all. In the end, if we can use WoT for these protocols, why not do it? The protocols in focus of WoT do not need any specific hardware so if a browser would support MQTT, does MQTT suddenly become a WoT protocol? One can say that the WG should not focus on these protocols and redirect its workforce to more web-browser compatible protocols and building specs on top of that but I don't see this happening with the general composition of the organizations that participate in this activity.

Now to some specific answer:

As I understand it the problem is not just that the URI schemes are not registered with IANA, but that the "specification of the origin protocol" doesn't define a URI scheme at all.

Would it be fine if MQTT defined a URI scheme itself and registered it? I think that is just one part of the problem. Apple Facetime has a URI scheme and it is registered but good luck using it (https://www.iana.org/assignments/uri-schemes/prov/facetime). More than half of the registered URI Schemes are submitted by Dave Thaler or are even Microsoft specific protocols (or identifiers to be more precise). We can say that in a TD, MQTT should be identified with abc123://mybroker and it would be fine. TDs need to be written anyways so we can prescribe how they should be written (which we do).

If a different Consumer is needed for each IoT protocol, then how does that "counter the fragmentation of the IoT"?

This is something I see often and it is definitely possible to have consumers that speak multiple protocols, it is not even that difficult to implement (see node-wot) since we have defined WoT operations via the op keyword. Ideally, the market should converge but I don't see this happening, or at least not fast enough.

And if we do go with usual Web protocols, we would not be able to address a bigger community, allow for other standardizations to compete with us in their non-Web world

[jyhi]

As I'm typing @egekorkan has given a good reply, so I'll rephrase a little bit.

Probably not the correct place to discuss, but Web of Things should be created to avoid creating the 15th standard. Instead of creating a competing standard, what I see is that WoT is trying to be inclusive enough by describing them instead of be like them. Setting apart IoT and WoT doesn't really make sense, as WoT is only a subset of IoT -- think of the Internet compared with the Web.

The fragmentation of the current IoT market won't end in the foreseeable future anyways -- the current technologies are good enough. On the other hand, a very large segment of IoT application cannot even afford Web-based protocols running on wire (not mentioning passive IoT like RFID tags don't even have a choice). They preserve every J energy; even an ASCII character is too big to accommodate. It's impossible for them to directly join the Web, unless we copy them to the Web -- hence TD.

Of course, it's great to have a fully Web-enabled IoT system ("green-field"); people are not preventing this from happening. @benfrancis has done a great job pushing this forward. I'm looking forward to it. It's just the reality and what people have done in the past years have prompted us to be careful. If we only draft a Web-based protocol then it may end up being only an advanced toy among tech-savvies.

I'd also like to see what has been achieved by the companies trying to do something real using WoT, as they look like already having some sort of real-world use cases, which is the best support for the decision of future routes.

a bit philosophical in my point of view

It actually is -- Epistemology, but that's another story.

[benfrancis]

@egekorkan wrote:

We can further go into this discussion which is a bit philosophical in my point of view

I'm sorry about that, but lots of key decisions (and disagreements) flow from these basic assumptions.

Web in its essence is about linking resources that are represented via URIs where URI does not mandate a certain URI scheme.

I do agree with this. See this definition from the Architecture of the World Wide Web:

The World Wide Web (WWW, or simply Web) is an information space in which the items of interest, referred to as resources, are identified by global identifiers called Uniform Resource Identifiers (URI).

if a browser would support MQTT, does MQTT suddenly become a WoT protocol?

It has nothing to do with web browsers, a WoT Consumer doesn't need to be a web browser or be implemented as a web application. But IMO in order to be part of the Web of Things a Thing does need to be:

1. Linkable
2. Client agnostic

Would it be fine if MQTT defined a URI scheme itself and registered it?

Yes, that would at least meet the current very broad criteria, and make it possible to provide protocol bindings using hypermedia forms.

We can say that in a TD, MQTT should be identified with abc123://mybroker and it would be fine.

That is one solution, but the IANA registry exists for a reason. If someone else then implements another protocol using the abc123:// URI scheme then that could create problems.

And if we do go with usual Web protocols, we would not be able to address a bigger community, allow for other standardizations to compete with us in their non-Web world

That would be fine in my view, I would actually prefer the Web of Things to have a more focused goal of building an ecosystem of services using a common web abstraction layer on top of the IoT (the World Wide Web of the Internet of Things), rather than the huge and impossible task of trying to describe every possible IoT protocol in a single JSON file format. Devices themselves could continue to use more specialised protocols better suited to constrained devices and specific application domains, with gateways and cloud services used to bridge them to WoT.

However, while that is not the case we at least have to agree that a protocol has to have a URI scheme in order to be linked from a hypermedia form.

[benfrancis]

@lmy441900 Yawn, that infamous XKDC comic could basically apply to everything the W3C does, it isn't really very helpful IMO.

WoT is only a subset of IoT -- think of the Internet compared with the Web.

I completely agree. The Web of Things should be the World Wide Web of the Internet of Things.

a very large segment of IoT application cannot even afford Web-based protocols running on wire

And neither should they. HTTP is a terrible choice of protocol for constrained IoT devices.

I'd also like to see what has been achieved by the companies trying to do something real using WoT, as they look like already having some sort of real-world use cases, which is the best support for the decision of future routes.

I agree with this, including my own company.

In fact I think we all broadly agree on the high level goals, just maybe not how to get there.

[benfrancis]

Coming back to the topic...

I'm pleased attention has been drawn to this contradiction in the WoT specifications.

@sebastiankb wrote:

A simpler solution would be to remove the assertion.

I would strongly oppose that given the years-long debate that it took to get it there in the first place. But besides that, simply removing the assertion does not change the fact that a protocol has to have a URI scheme in order for a protocol binding to be defined using a hypermedia form.

In order to define protocol bindings for MQTT and Modbus in a Thing Description, they need to have URI schemes one way or another.

(e.g., MQTT, Modbus, OPC-UA, etc.).

It may not be such a problem for OPC-UA since (at least according to Wikipedia) that protocol can be used with thehttp:// URI scheme.

[relu91]

I'm a little bit regretting opening this issue 😄 . To be honest I strongly agree with @sebastiankb:

Instead to reopen the never-ending protocol discussion again, I would suggest how we can overcome the situation.

I'm going to address only the topic I've been inquired about and then I would suggest keeping looking for solutions about the original issue.

@relu91 wrote:

I believe that being open to different protocols does not really mean that all Consumers out there will implement all IoT protocols.
If a different Consumer is needed for each IoT protocol, then how does that "counter the fragmentation of the IoT"?

I didn't mean that every Consumer will implement only one IoT protocol, on the contrary, I was thinking about a 1-p mapping with p as the number of useful protocols for one particular Consumer domain. In the end, there might be good overlapping like in the diagram below:

For what it is worth I agree with most of the other points that have been raised in the other comments.

[relu91]

Back to the original issue:

A simpler solution would be to remove the assertion.

I agree with @benfrancis I would keep it, but I like this new form:

Eligible protocols for W3C WoT MUST have an associated URI scheme [RFC3986] that is registered with IANA (see [IANA-URI-SCHEMES]) or is directly given by the specification of the origin protocol (e.g., MQTT, Modbus, OPC-UA, etc.).

As a W3C WG, we have also the ability to define registries, it might be a good place for having a curated list of URI schemes. DID spec already use this document type for their goals. What do you think?

[sebastiankb]

@benfrancis

As I understand it the problem is not just that the URI schemes are not registered with IANA, but that the "specification of the origin protocol" doesn't define a URI scheme at all.

Ok, I see. I think there are many de facto standard protocol scheme assumptions that can be also found in many prominent implementations such as for MQTT and OPC-UA.
For WoT we can make them more official by providing the scheme value which is used to identify the protocol. Sure, IANA registration would be more desirable, but it should be initiated by the relevant SDOs. Maybe we should ping them.

I would strongly oppose that given the years-long debate that it took to get it there in the first place. But besides that, simply removing the assertion does not change the fact that a protocol has to have a URI scheme in order for a protocol binding to be defined using a hypermedia form.

Well, why you want this assertion in the architecture spec? This should be more defined in the TD specification or in the binding document, right?

[benfrancis]

@relu91 wrote:

I'm a little bit regretting opening this issue smile

Sorry for causing trouble! 😄

IMHO the group has reached a good balance between the two positions that @benfrancis recalled in the answer above

FWIW I agree that pursuing both protocol binding templates and profiles is a good balance, considering the range of views, and I'm really pleased to see profiles slowly coming together.

I agree with @benfrancis I would keep it, but I like this new form:

Eligible protocols for W3C WoT MUST have an associated URI scheme [RFC3986] that is registered with IANA (see [IANA-URI-SCHEMES]) or is directly given by the specification of the origin protocol (e.g., MQTT, Modbus, OPC-UA, etc.).

I wouldn't object to that wording, but the examples may be misleading if those specifications don't actually define a URI scheme. Do they? It does also obviously create the risk that multiple specifications could make a claim to the same URI scheme which could lead to conflicts, so they should ideally be registered with IANA eventually.

As a W3C WG, we have also the ability to define registries, it might be a good place for having a curated list of URI schemes. DID spec already use this document type for their goals. What do you think?

Creating a W3C registry of URI schemes which diverges from the existing IANA registry doesn't seem like a great idea to me I'm afraid.

As @egekorkan suggested, you could just define your own unofficial URI scheme in one of the protocol binding template documents (which are non-normative anyway), but it seems like a risky strategy.

@sebastiankb wrote:

it should be initiated by the relevant SDOs. Maybe we should ping them.

I agree, but they as @egekorkan has pointed out, they may not want to.

Well, why you want this assertion in the architecture spec? This should be more defined in the TD specification or in the binding document, right?

Yes, it would be much better in general if the normative assertions were moved to the Thing Description specification.

[sebastiankb]

For MQTT, I found this discussion here AMWA-TV/is-07#54 (comment)

There seems also a proposal for the IANA registration here: https://github.com/mqtt/mqtt.org/wiki/URI-Scheme

[benfrancis]

@sebastiankb wrote:

here seems also a proposal for the IANA registration here: https://github.com/mqtt/mqtt.org/wiki/URI-Scheme

Yes I've seen this before, this wiki page has been there since 2014 but the proposal was never implemented. As linked in the comment you found there's also an OASIS issue regarding an MQTT URI scheme which was closed with a resolution of "no action" in 2017.

[relu91]

Creating a W3C registry of URI schemes which diverges from the existing IANA registry doesn't seem like a great idea to me I'm afraid.

As @egekorkan suggested, you could just define your own unofficial URI scheme in one of the protocol binding template documents (which are non-normative anyway), but it seems like a risky strategy.

I was thinking more about extending IANA registry. Since we'll moderate the registry we can control new registrations and block the ones that conflict with the IANA registry. About binding template documents, FYI the new document structure has a section dedicated to the definition/explanation of the URI Scheme. See modbus for example (still a draft).

[sebastiankb]

PR is available, please review w3c/wot-thing-description#1301

[egekorkan]

In the call of 15.12 we have decided to close this issue since the PR linked in the above comment is merged and loosens the requirement. Further comments should be made at w3c/wot-thing-description#1326