Message Transfers in MQTT
In this section we cover some details of how messages are transferred in MQTT. There are three different ways which differ in the level of reliability between the communicating parties. Note that these mechanisms are independent of the underlying network technology and therefore reliability can be enforced also if using a potentially unreliable network technology (like many wireless network links). The "Quality Of Service" (QoS) parameter in the message header defines which method is chosen for the transfer. In the case of the publish/subscribe mechanism the QoS is chosen independently by the publishing client and the subscribing client and hence they might be different.
Quality of Service (QoS)
There are three levels of QoS in MQTT:
- QoS 0: At most once
- QoS 1: At least once
- QoS 2: Exactly once
The short descriptions of the levels above give an indication on the intention of the reliability involved. The higher the QoS level the more network traffic will be generated for sending a message. The three levels are explained in the following. The figures illustrate the use of the QoS levels when publishing a message, however, the same mechanisms hold when the broker sends a message to a subscribed client.
QoS 0
QoS 0 is also called "fire and forget":
The sender sends out a message and no acknowledgment is sent back from the receiving side. Hence the sender does not have a guarantee that the message really arrived at the destination. Once the message has been sent out the client forgets about the message.
QoS 1
This QOS level message ensures that the message is at least delivered once to the receiver.
Upon reception the receiver sends back a PUBACK packet to the sender confirming the successful reception of the original message. If the sender does not receive this PUBACK packet within a given time it re-sends the original message. When re-sending the message it sets a flag named "DUP" in the message header indicating that the message has been sent more than once. The sender continues to send the same message over and over again until it receives a PUBACK packet from the receiver.
In order for this method to work, there must be a way to identify the message unambiguously. For this the header of the original message contains a "Packet Identifier" field (16 bit). It has to be filled by the sender with a number which is unique for all messages currently circulating between the client and the receiver. This identifier is copied into the PUBACK message by the receiver so that the sender knows to which message the PUBACK packet refers. Once the PUBACK is received by by the sender the Packet Identifier may be reused for a new message.
Also note that the sender is allowed to send another message (with a different Packet Identifier) to the broker while it is waiting for a PUBACK message.
This mechanism guarantees that the message is delivered at least once to the receiver, however it might happen that the message is received more than once. E.g. when a client publishes a topic with QoS 1 it might happen that subscribers for this topic receive the message more than once since the Broker processes every incoming message immediately on reception. Note that the DUP flag from the publishing client to the broker will not be sent to the subscribers, since the connection between broker and subscribing has it's own QoS defined. The QoS for THAT connection can be anything and if it is QoS 1, the DUP flag is used for the message transfer between Broker and subscriber.
QoS 2
QoS 2 ensures that a message is exactly delivered once at the final destination. For this a handshake with 4 messages is used.
As in the case of QoS 1 the sender includes a Packet Identifier in the original packet which is currently available (i.e. not in use for another message sent by this sender). Once the message is received the receiver sends back a PUBREC message (including the same Packet Identifier) to tell the sender it received the message. Once the sender receives the PUBREC message it discards the original message which it does not need anymore but it still memorizes the Packet Identifier. It sends a PUBREL message to the receiver indicating that it "forgets" the message from now on. From this point on the sender is not allowed anymore to re-send the message, HOWEVER, the PUBREL packet itself might be re-sent until it is acknowledged (see below)!. Once the receiver receives the PUBREL message, it sends a PUBCOMP packet back to the sender indicating that the message transfer to the receiver now is completed. At this point the sender may re-use the Packet Identifier for another message.
The receiver has the possibility to start the processing of the incoming message (e.g. the broker will forward a publish message to the subscribers) when the message is received for the first time. However when it receives the message a second time (because the sender re-sent it) it must not process the message again (it knows that a message is re-sent since it can be unambiguously identified with the Client Id and the Packet Identifier in the message.) Alternatively the receiver may choose to start the processing of the message once the PUBREL message is received. Then it knows it will not receive any further copies of the same message (re-sent by the sender).
Note that this is different from QoS 1 where the receiver starts processing any incoming message, also those which have been re-sent. This is why for example subscribers might receive published packets twice when using QoS 1.
Pathological (and theoretical) problems
To deepen the understanding of the QoS 1 and QoS 2 transfers let's look at two pathological examples:
Imagine a message transfer with QoS 1 but for some reason the PUBACK packets do not arrive at the sender. In this case the sender keeps re-sending the original messages to the receiver (which processes them all, which could mean in the case of publish messages that all subscribed clients over and over again receive the same message). In addition the Packet Identifier will never be released by the client. If the client starts sending other messages at the same time, it starts to use up more and more Packet Identifiers.
This problem is a theoretical one since it would require only the PUBACK packets to get lost. In addition many MQTT client implementations (especially those for microcontrollers which must not use a lot of resources) do not send a new message until the previous messsage was acknowledged.
In practice the sender (or also the receiver) might choose to close the connection in this pathological scenario and try to re-open it again.
An equally pathological case would be, if in a QoS 2 message transfer only the PUBCOM messages get lost. However, in this case the message would be processed only once by the receiver and in case of a publish message the subscribers would get the message only once.
QoS levels for publishers and subscribers
Earlier we said the QoS for the connections publisher-broker and broker-subscriber can be independently chosen. While this is correct you should note that the broker might not always respect the request for a specific QoS level in a subscription.
If a subscriber asks for a higher QoS level in the subscription than the publisher uses during the sending of the topic to the broker, then the broker will continue the message transfer to the subscriber with the lower QoS which was chosen by the publisher.
The following table summarize all 9 possible combinations of QoS levels between publisher and subscriber:
| QoS publishing | QoS subscribing | QoS Publisher >> Broker | QoS Broker >> Subscriber |
|---|---|---|---|
| 0 | any | 0 | 0 |
| 1 or 2 | 0 | 1 or 2 | 0 |
| 1 | 2 | 1 | 1 |
| 2 | 1 | 2 | 1 |
| 1 | 1 | 1 | 1 |
| 2 | 2 | 2 | 2 |