how to solve duplicate event processing

Feature%3A-Room-State-Management.md

@@ -3,32 +3,49 @@
 Planning and design of the room state management "feature."
 
 * The bot must keep track of rooms it is in, and what users are in each room.
+* The bot should keep track of information about a room: the room ID, the room name, etc.
 
-## Bot Changes
-
-Listen to Matrix event for when users enter and leave rooms. This is easily accomplished with the `on_room_member` event. 
-
-* When the bot joins a room for the first time, member events are fired for every member in the room. 
-* It seems to sync everything, so you will get a join and leave notification many times, potentially. This is problematic for a few reasons.
-* **Solution:** The simple solution in the beginning is to ignore any event more than a few seconds old, and then do a full sync of the users in the room when processing the join event. This works until there's a big network slowdown, or something else happens. A more reliable solution is to simply track every single event ID in the database, and keep track of when it was processed. If it's in the database, do not process it again.
-
-If the bot is kicked or banned, it does not get events for users leaving a room. Instead, it will get a member event of leaving the room for itself. In that circumstance, it should delete all user information in the database for the room.
-
-RoomInfo struct is created/updated when the bot joins a room.
 
 ## Database Changes
 
-New DB object `Rooms`. Keep track of users and the rooms they are in, both by username to a list of room IDs, and a room ID to list of usernames. It will also keep track of actual room information.
+New DB object `Rooms`. Keep track of users and the rooms they are in, both by username to a list of room IDs, and a room ID to list of usernames. It will also keep track of actual room information and other useful stuff.
 
 Trees:
 * username -> list of room ids, for easy implementation of private conversation command execution.
 * room id -> list of users
 * room id -> RoomInfo
+* processed events: room id `0xff` event id -> timestamp (basically hash set of processed event IDs)
 
-## Architcture Changes
+## Matrix Event Processing
+
+This is a core problem to solve before the rest, because all information needed to implement this feature comes from room member events.
+
+One of the core aspects of Matrix is that everything is a series of events. A room with messages is basically a decentralized database of event information. When logging in, or joining a room, it is possible for the bot to receive events it's already gotten in the past (notably this happens when re-joining a room that the bot was kicked from). Since our handling of Matrix events is not idempotent, and we cannot guarantee we will get all events from the server to properly replay everything, we must not process events we've already processed in the past.
+
+The simplest way to do this is to keep track of processed events in the database.
+
+* A `Tree` that uses room ID `0xff` event ID as the key and the local processed timestamp as the value gives us an efficient way to stop event duplication.
+* This applies to every single event handler. Check if the event is in the database, and if not, record it and allow processing. This should be able to be accomplished atomically with Sled.
+
+
+## Keeping Track of Users and the Rooms They Are In
+
+### Bot Changes
+
+Listen to Matrix event for when users enter and leave rooms. This is easily accomplished with the `on_room_member` event. When the bot joins a room for the first time, member events are fired for every member in the room. 
+
+* We may not get information for all participants, if their join events are old. So instead, sync all user info when bot joins a room for the first time.
+* If the bot is kicked or banned, it does not get events for users leaving a room. Instead, it will get a member event of leaving the room for itself. In that circumstance, it should delete all user information in the database for the room.
+* Otherwise, we can simply processed user join/leave events as they come in, and update the data for that specific user.
+
+RoomInfo struct is created/updated when the bot joins a room.
+
+### Architcture Changes
 
 A new `RoomInfo` struct must be created, and possibly a whole new layer of the application. This layer doesn't exist yet, because we've been dealing only with strings and integers so far.
 
+Create `src/models.rs`.
+
 RoomInfo must store at least:
 * Room ID
 * Room name