Hi all, it’s Jure Purgar here again. I was going over ideas on our feedback forum (by the way Thanks, we really appreciate it) and there are some ideas / questions about sync process, so I decide to write a blog post about it, to explain how we are doing it and why we are doing it this way. On the other hand, I’m not saying the sync process is perfect so we are always open for ideas that are technically possible and make sense. This might be a bit technical, so if you are happy with how sync works and you think that this is something you really don’t need to know, than you can stop reading.
Sync process consist from two main parts, your states (history, collection, watchlist, etc.) and show / movie info (description, images, episodes, etc.). We need both parts to show data within the app. The first part will tell us know which items you watched, collected, added to watchlist, etc. The second part is needed to show info about the item, in case of shows, we are also using this data to get air dates, etc. The sync process ensures that required data from both parts is downloaded / updated.
The first thing we do is check when states (history, collection, etc.) were last modified on trak.tv. We than we check if we have up to date states for each state type (history, collection, etc.) locally, so we do not need to download it again. That means we only download states when you change something: watch an episode, add movie to watchlist, etc. Whenever we have to refresh, we have to download all states for particular state type. For instance if you watch an episode we have to download your entire watch history. This is a limitation of Trakt API so we cannot do anything about it. However the size of states data for user that watched 1000 shows and 1000 movies is only about 400KB.
Beside downloading states we also have to upload it. Whenever you change a state in the app we save it locally, so you can do that also when you are not connected to internet. The sync process than picks up that local states and uploads them to Trakt. We only upload the states that were changed, we are not uploading your entire history again.
Syncing item info
Besides states we also need info about shows and movies like: description, episodes, images. We are downloading this data for each item we discover during states sync. Similar to states, we check for each item when it was last modified on trakt.tv. This is done via something called HTTP HEAD request which only returns headers (info about when item was last modified). Not a lot of data is transmitted during this checking and we are doing this for multiple items simultaneously so it takes less time. Surprisingly data is modified on trakt.tv quite often, or at least the date that we get back as last modified changes very often, even if nothing was really changed. Because of this and because of the fact that also checking if item was modified, for a lot of items, would take a lot of time we first check if an item is “eligible” for updating. This is done by a “smart” algorithm that considers the possibility the item is really updated and that you would care to see most up to date info:
- For shows
- If show is ended and you watched all episodes we only update it every 20 to 30 days
- If show is ended we only update it every 10 to 20 days
- In other cases we only update it every 12 to 36 hours
- For movies
- If movie was release we only update it every 100 to 180 days
- In other cases we only update it every 20 to 30 days
Before we even check on Trakt if item was updated we first run it thru this algorithm. You may ask what X to Y days is all about. When we check the item we are using a random number between X and Y, so it does not happen that sometimes in the future you would start the app and a lot of items would be updated because condition would be met for a lot of items.
The algorithm and check when the item was last updated provides us with a list of items that we actually have to update. The info we fetch includes: item details (description, status, genre, etc.), people, episodes info and image. We only download image if it was actually changed and we don’t have the new image locally. We are downloading all of this data so you have it locally and you can check your shows and movies details even when you are not connected to internet. We are also using this info to calculate stuff (unwatched episodes), provide notifications and live tiles, etc. In the latest version (3.1511.1209) we are doing the updating in parallel so it takes less time as before that we were doing item by item.
I hope you were able to understand most of the stuff, I know it can be complicated, because it is complicated also for me sometimes. Anyway, if you have additional questions let us know. Do you think we are doing this the right way or not? Let us know in the comments or on Twitter.