Here's the major usability pain points I've run in to with various OTR clients over the years. Feel free to chime in with any additional issues you've noticed.
1. The multiple client problem: If you have two devices online simultaneously under the same account, they will fight and endlessly renegotiate.
This issue may be specific to OTRv2 only. Arlo pointed out that this change in OTRv3 may solve a lot of these problems:
"Both fragmented and unfragmented messages contain sender and recipient instance tags. This avoids an issue on IM networks that always relay all messages to all sessions of a client who is logged in multiple times. In this situation, OTR clients can attempt to establish an OTR session indefinitely if there are interleaving messages from each of the sessions." -- https://otr.cypherpunks.ca/Protocol-v3-4.0.0.html
We should be sure we use these features properly, and make sure the js-ctypes wrappers are updated to support the new libotr v3 APIs with these instance tags. The current js-ctypes wrapper may still be v2 only.
If for some reason these tags in v3 can only be used if both parties support v3, perhaps we can add some additional code to track OTR state using XMPP resource/device info and tell the OTR lib underneath that different XMPP resources are actually different users, so that at least XMPP accounts won't have this problem.
2. Better TOFU support: We should track only one OTR key for a given XMPP resource/device ID.
Some people I communicate with are constantly rotating OTR keys for security and other reasons. However, this doesn't help me any: I have accumulated about 50 keys of unknown age and usage pattern in my OTR fingerprint list for them, and I have no idea which ones have been changed due to compromise, device loss, or simply due to reinstall. Worse, I have no idea which keys should correspond to which XMPP resources/devices.
If a user changes the key for a given XMPP resource ID, we should expire the previous key and loudly alert the user of the fact that the resource's key has changed, and ask them if they want to replace the old key with the new key. We should not simply accumulate more keys by default, especially for the same XMPP resource/device, and especially not for unverified keys.
At the same time, we should allow a different OTR key for each unique XMPP resource ID, and perhaps allow non-XMPP keys to accumulate at user option.
3. "Unreadable encrypted message" errors should be visible only to advanced users/in the error console, and the client should transparently renegotiate and resend.
It looks like OTR has an ERROR_START_AKE flag for this, but for some reason, some states of OTR in Pidgin still cause me to get and send these errors and miss messages though, usually if one client has crashed or restarted without properly ending the conversation. We should try extra hard to force key re-negotiation in these cases.
We probably also want an auto-responder after the renegotiation that informs the other party that N messages were missed (for clients that don't support OTR's built-in notifications).
4. OTR shouldn't get stuck in the "Finished" state when you end a conversation.
I've noticed in the past that OTR seemed to refuse to automatically negotiate/re-negotiate if it is still "Finished" regardless of your "Require/Force OTR" setting. This is just bad, and often causes plaintext to leak.
5. It should be impossible for our client to send a message in plaintext by default, and the client should discourage counterparties from sending more than one plaintext message.
Perhaps related to issue 3, I've not had very good results with the "Require OTR"/"Force OTR" options on many clients.
It also seems like many people's clients will still send the first message unencrypted (to get the handshake hint), and on all clients OTR doesn't seem to try very hard to negotiate if an unencrypted message is received.
On our side, we should hold those messages and send only the handshake initiation (ie our default OTR mode should be REQUIRE_ENCRYPTION).
We may also want to consider a plaintext auto-reply for when we receive unencrypted messages. Something like "Please install and enable OTR before sending further messages" as a separate message in *addition* to attempting an OTR handshake.
6. The UI should have per-message encryption indicators (send and receive)
For many clients, the UI is not clear which messages were sent encrypted, especially if the OTR state has changed due to autonegotiation or "Finishing". We should have per-message icons for both sent and received messages that indicate encryption status for that message in the chat scrollback window.
7. Logging should be off by default when OTR is on.