Improve performance/usability when navigating between folders

[starypatyk]

I have noticed a few issues related to performance/usability when navigating between folders - esp. when a folder contains larger number of files. In my testing I use a folder with about 300 photos.

I plan to create separate issues/PRs for each of the items below and use this issue to track the overall progress.

• Full refresh of a folder is performed during onBrowseUp (back button) Avoid full folder refresh during onBrowseUp #10688
• WebDAV call sent by the Android app to read folder contents executes a few database queries for each file.
  • Reading information about share notes requires 4 DB queries per file. Reduce number of database queries during WebDAV propfind request server#34471
  • Reading DAV:displayname property requires 1 query per file. WebDAV - use file/folder name for dav:displayname server#34508
• API call to retrieve information about files shared from a folder requires a few database queries for each file. Reduce number of queries to read shares in a folder server#34918
• File list "jumps" after scrolling when an asynchronous operation finishes #11033
• When changing folder in the app the file list gets refreshed three times, i.e. OCFileListAdapter::swapDirectory is invoked three times. Two of the calls seem redundant when all the information cached on the device is still valid.
  • OCFileListAdapter::swapDirectory is invoked in response to EVENT_SINGLE_FOLDER_CONTENTS_SYNCED broadcast - even when the ETag check in RefreshFolderOperation determines that the folder has not changed. Refresh UI only when changes are detected on the server #11100
  • OCFileListAdapter::swapDirectory is invoked again in response to EVENT_SINGLE_FOLDER_SHARES_SYNCED broadcast. It seems that the code that retrieves the shares does not try to check, if the data about shares retrieved from the server is the same as cached data. Trying to optimize the process of saving shares in the local DB in Use Room for saving shares in FileDataStorageManager #11255.
• Reading file information from the local SQLite database (FileDataStorageManager::getFolderContent) takes much longer than expected. Profiling suggests that most of the time is spent inside createFileInstance method in these operations:
  • deserializing JSON data (~50% of time) Initial attempt to improve this (Defer expensive operations until results are read from OCFile #11181) abandoned. Simpler approach proposed: Avoid JSON deserialization in common trivial cases #11251.
  • obtaining column indices in cursor.getColumnIndexOrThrow (~30% of time) Use Room for filelist table reads #11098
  • Skip check for existing files when reading file data from local DB #11227

[starypatyk]

Baseline results from my dev environment (331 files in a folder). When entering a folder the app executes the following API calls:

Request	Execution time [ms]	# of DB queries
WebDAV PROPFIND - check ETags (always)	45	33
WebDAV PROPFIND - full (only when folder changed)	1300	1699
Share information (always)	918	2006
Total	2263	3738

[tobiasKaminsky]

Thanks for your awesome help and it was very nice to meet you at our conf!

[starypatyk]

Updated results from my dev environment (331 files in a folder) - with changes from nextcloud/server#34471, nextcloud/server#34508 and nextcloud/server#34918.

Request	Execution Time [ms] (before)	Execution Time [ms] (after)	# of DB queries (before)	# of DB queries (after)
WebDAV PROPFIND - check ETags	45	50	33	34
WebDAV PROPFIND - full	1300	250	1699	41
Share information	918	50	2006	25
Total	2263	350	3738	100

[AndyScherzinger]

Awesome work and super nice improvements with the update(s) @starypatyk 🚀

[starypatyk]

After upgrading my production server to 25.0.1 and applying changes from nextcloud/server#34918 manually (as these are not yet merged), I have found that there is still room for improvements on the client side. 😉

I have updated the list in the first comment above and plan to submit additional issues/PRs soon.

[AlvaroBrey]

Thanks @starypatyk , this is very useful information.

Reading file information from the local SQLite database (FileDataStorageManager::getFolderContent) takes much longer than expected. Profiling suggests that most of the time is spent inside createFileInstance method in two operations:

DB code is quite inefficcient in the app in general; and it's also not helping that all DB calls go through a ContentResolver and get executed in the main thread. I introduced Room recently (#10977) to incrementally deal with some of that; it should at least get rid of the getColumnIndex load. But it needs to actually be applied to the files table access code first.

The ugly truth is that in order to have a properly working DB (see #10213 ) at some point it will be unavoidable to change DB structure to remove serialized fields, and similar stuff, and replace them with proper SQL relations.