In low memory setups, the updater fails at step 5

[tomtom]

In low memory setups, the upgrader fails at step 5. The other steps work without problems.

Solution: The updater should either provide

1. a low memory version to verify the integrity of the downloaded zip file at step 5 or
2. provide a way for users to continue at step 5 without successful verification of the zip file (there should be a not about where to find the downloaded zip file an how to verify the integrity of the ZIP file yourself before letting the updater continue)

[joshtrichards]

Hi @tomtom - Can you clarify what you define as "low"? Also, how, specifically did the integrity step failure manifest itself in your situation? Was there an error reported somewhere?

[tomtom]

AFAIK the php memory limit is 192 MB for my setup. The step failed with a red cross as error marker and an empty error message. The .step file shows step 5 has started. There was no other message. When I reload the page, I get the Message (something like) "Step 5 has started; please reload the page later". Maybe, the script running step 5 got killed (due to a out of mem of a timeout maybe?) and the updater goes on waiting. Users can make the updater script continue by editing the .step file. The update (usually) succeeds without problems. Does this suffice?

[joshtrichards]

Okay. Could you post the last few sections of your updater.log (it'll be inside the data directory)?

That will contain more context.

[tomtom]

I suppose you mean the lines where step 5 is invoked? ``` 2023-08-22T14:55:47+0200 tl1Xo9fHVJ [info] Step 4 is in state "end". 2023-08-22T14:55:47+0200 tl1Xo9fHVJ [info] POST request for step "5" 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] startStep("5") 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] verifyIntegrity() 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] getUpdateServerResponse() 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] updaterServer: https://updates.nextcloud.com/updater_server/ 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] releaseChannel: stable 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] internal version: 27.0.2.1 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] updateURL: https://updates.nextcloud.com/updater_server/?version=27x0x2x1xxxstablexx2023-08-08T08%3A23%3A54%2B00%3A00+9234ae4a0a98ee924cd63974e88c22c3bb3f6b21x8x2x11 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] getUpdateServerResponse response: Array ( [version] => 27.1.1.0 [versionstring] => Nextcloud 27.1.1 [url] => https://download.nextcloud.com/server/releases/nextcloud-27.1.1.zip [web] => https://docs.nextcloud.com/server/27/admin_manual/maintenance/upgrade.html [changes] => https://updates.nextcloud.com/changelog_server/?version=27.1.1 [autoupdater] => 1 [eol] => 0 [signature] => Ryk8LIlqnZQztW6Y3OkHF5IBwrMUD+F8dE3Ym6HNHhN2leTEjo1hRfriuadCsIDZ G+qcCR9aQA0kl1aMZg2hX15ssC47fvC4dGbbN1sb24V37X4+QNBpc/Y6tFqXbUjh jZxIesNVZPrTQT3CEnuqOC7S8UzGkVAEusHXqyZKwi/i88/FNCNrEqP/0X+AGnc5 imRsJr/RJ5DCdNd7DTNi9qECCPSsSv9kPpIbOjpfV2g6BHhPL24c10ykvBcigMSm DvSLUmGMpfcIhnnWMqfsqYxHKsluyMk4MCoVOZAu3q8bjgT/l+mvVFd5rlXTNN+O l+YD1ENLZZwGJCZj2AanQw== ) 2023-10-05T10:32:10+0200 Z258tnSBAQ [info] storage location: [...]/data/updater-[...]/downloads/ 2023-10-05T10:32:20+0200 ccjOVMX8qc [info] request to updater 2023-10-05T10:32:20+0200 ccjOVMX8qc [info] currentStep() 2023-10-05T10:32:20+0200 ccjOVMX8qc [info] Step 5 is in state "start". 2023-10-05T10:32:23+0200 bpvYXs6Jd8 [info] request to updater 2023-10-05T10:32:23+0200 bpvYXs6Jd8 [info] currentStep() 2023-10-05T10:32:23+0200 bpvYXs6Jd8 [info] Step 5 is in state "start". 2023-10-05T10:32:37+0200 tZAfXqsTMY [info] request to updater 2023-10-05T10:32:37+0200 tZAfXqsTMY [info] currentStep() ``` I assume the last few messages were triggered by me reloading the page.

[liqueurdetoile]

I've also add the same issue on a 2GB RAM VPS. It sheds nextcloud + coolwsd + open project instances. All is usually working fine as I'm the sole user.

After web updater failed with no reason, I switched to command line installer which obviously triggered a memory allocation error (requested something like 225Mb of memory) . There's no memory limit in CLI php.ini but, after check, I had only 65Mb free at this time. Stopping web server and openproject allowed to complete update by freeing most of the RAM.

Not practical but it works. Anyway, if there's a way to reduce memory footprint to check download integrity, it would be great !

[joshtrichards]

I looked into this a bit and did a little testing.

99.9% of the memory usage in verifyIntegrity() is arising from needing to read the entire archive into a string (using file_get_contents()) to pass it off to openssl_verify(). This is necessary to verify authenticity of the downloaded archive file.

If we were only interested in verifying the integrity (i.e. we don't care about confirming the signature), we could use hash_file() to only check hash of downloaded archive. It does not require reading the entire archive into memory.

This would not authenticate the download against the certificate built into the Updater. It would merely confirm the file likely hasn't been corrupted during transit. It would remain vulnerable to replacement however.

a low memory version to verify the integrity of the downloaded zip file at step 5

The failure here is because we're running openssl_verify() against the entire downloaded file. I suppose we could start signing the digest only and having the update server provide that (signed) which we'd run openssl_verify() against as usual. Since we wouldn't be verifying against the entire archive in real-time, that would have a tiny memory footprint (~1M).

To accomplish this we'd have to make more drastic changes to our updater infrastructure. Namely, replacing (or augmenting) the existing signing.

provide a way for users to continue at step 5 without successful verification of the zip file (there should be a not about where to find the downloaded zip file an how to verify the integrity of the ZIP file yourself before letting the updater continue)

I suppose we could add another flag akin to --no-backup - e.g. --no-verify to skip that check entirely. It would only be an option at the command line. The responsibility would be 100% on the admin to verify - if so desired - the archive (in whatever way they feel comfortable doing so and they can get to work on their infrastructure).