How is that Push-based synchronization implemented in NIA? provide a better guide for beginners

[Premjit-Developer]

As per comments provided by devs it really help us to understand the code implementations but when it comes to Push-based sync implementation it is hard to understand how it is working and how can i implement this in my project without knowing network data source side implementation. Because all over the internet there is only one article which talk about Push-based sync implementation in android by @tunjid but there is also not clearly mentioned how they implemented it in NIA how's that NetworkChangeList class is working under the hood?

[tunjid]

Now In Android has 2 data types:

Data types in Now in Android

News Resource

Name	Type	Description
id	alphanumeric	Id of the news resource
title	alphanumeric	Title of the news resource
content	alphanumeric	Short description of the content
headerImageUrl	url	Url of header image
publishDate	date-time	Date time of the news resource publishment in RFC 3339 format
topics	array (alphanumeric)	Array of topics’s ids
type	enum [article, video, api, docs]	Identifies the type of the resource.
url	url	URL of the resource

Topic

Name	Type	Description
id	alphanumeric
name	alphanumeric	Name of the topic
shortDescription	alphanumeric	Brief description of the topic
longDescription	alphanumeric	Long description of the topic
imageUrl	alphanumeric	URL of the topic’s SVG icon

Both of these are persisted in the backend in a database that allows for a simple RESTful API to be built on top of it. Since Now In Android is a read only app at the moment, we have 2 REST endpoints:

• /newsresources: returns a list of news resources
• /topics: returns a list of topics

Modern database solutions including FireStore, PostgreSQL, MySQL, MongoDB and so on provide write triggers. That is, anytime a record is written, a callback can be invoked to perform some action. This is the abstraction the offline first approach in NiA is built on. This abstraction is called change list invalidation and is a publish-subscribe pattern implementation.

Change list invalidation

Change list invalidation is an incremental cache invalidation strategy most similar to the git version control system. It allows apps to be kept up to date in near real time. It doesn't focus on how often an app hit the server's source of truth (it should hit it as often as it needs to), but how efficiently the server responds to requests.

It has the following premise:

• Each model table will have a sibling table that has a row that tracks a unique field that identifies a CRUD update (change_list_version).
• This unique field must have natural ordering.
• CRUD updates to any model will cause an update to its corresponding change_list_version (akin to a new commit in git).
• Clients will then hit an endpoint asking for changes since the last change_list_version it has, or its local HEAD. A changelist of model ids that have changed will then be sent (akin to a git fetch)
• The clients will then process this change list, updating its local HEAD as the update is consumed.

The pros and cons of this approach are as follows:

Pros:

• Minimum data usage: Only data that has changed will be fetched.
• Efficient: Requesting change lists is very cheap, as the query and associated json payload is very cheap.
• Eager, proactive sync: Sync can be run very often and cheaply. We can run eager sync on each new page we visit in the app with our local HEAD. The endpoint will simply return no changelist if nothing has changed.
• Real time updates: Sync may also be triggered by server side data push notifications using FCM essentially providing real time app updates

Cons:

• Backend infrastructure needed to support change lists (Implementation discussion to follow)
• Slightly more complicated client processing logic

Firebase FireStore as the change list server for Now In Android

The following processes are used to implement a change list server with Firebase Cloud Functions:

• Write Triggers
• Change list transactions
• Client notification and sync

Write triggers

Firebase FireStore has triggers that may be called upon any write to the database. These writes are only triggered if the underlying dataset is actually changed. That is, if the same data was written, the trigger will not be pulled.

Change List Transactions

When the change list cloud function is triggered, a transaction is run where:

• The most recent change_list_version for a collection is read
• The model that was updated has a change list item upserted for it where its change_list_version is incremented by one
• Care is taken to ensure that the cloud function is idempotent as specified in the Firestore docs.

Client Notification and Sync

Upon completion of the transaction above, Firebase Cloud Messaging can be used to ping the client and instruct it to sync with the server. The client then:

• Pulls the latest change list after its current change_list_version
• Requests the models that have changed by passing in the ids from the change list pulled
• Saving its current version

This process ensures that even if notifications are missed, the client can sync at its earliest opportune time, and still be up to date.

Changelist invalidation as a push based offline first approach

The above works well for Now In Android because users are essentially subscribed to all news resources so all updates to news can be pushed to the app. If an app is not interested in all updates that occur server-side the app should only subscribe to the set of topics its interested in, and should probably use a pull based approach for others.

Miscellaneous notes

The change list server could have been built on any kind of database. In fact if your app uses FireStore you should just use the native FireStore SDK. Now In Android uses regular RESTful APIs as that is what most apps consume.