Wrong alghoritm work with coding in external storage, for example webdav

[kuimovvg]

Nextcloud send propfind request for folder, for get list of files and receive response with RFC 3986 strings names. The file name can be a sequence of codes %d0%b8%cc%86. It is cyrilic letter "й". In method OC\Files\Storage\DAV -> opendir
this string stransform by function urldecode to "canonical view" "й". Next request to webdav server try to get this file size. Algorytm is next: nextcloud transform "canonical view" with character "й" and transform it by function rawurlencode to RFC 3986 string. In this point, coding of this character will be %d0%b9. Letter "й" can be encoding by two ways %d0%b9 and %d0%b8%cc%86. But when nextcloud send propfind request for this character sequence, webdav server response 404.

NextCloud server version 11.0.2

1. when open folder, with file like this, we can see less files than real count.

error

{"reqId":"65oVVtGvgcJeBz\/BJvhB","remoteAddr":"**.**.**.**","app":"webdav","message":"Exception: {\"Message\":\"Not Found\",\"Exception\":\"Sabre\\\\HTTP\\\\ClientHttpException\",\"Code\":404,\"Trace\":\"#0 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Client.php(231): Sabre\\\\HTTP\\\\Client->send(Object(Sabre\\\\HTTP\\\\Request))\\n#1 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/DAV.php(268): Sabre\\\\DAV\\\\Client->propFind('http:\\\/\\\/10.0.3.9...', Array)\\n#2 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/DAV.php(708): OC\\\\Files\\\\Storage\\\\DAV->propfind('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#3 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Common.php(649): OC\\\\Files\\\\Storage\\\\DAV->getPermissions('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#4 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(577): OC\\\\Files\\\\Storage\\\\Common->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#5 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(577): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#6 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Availability.php(460): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#7 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Encryption.php(173): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Availability->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#8 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(577): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Encryption->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#9 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(577): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#10 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(577): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#11 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(114): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->getMetaData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#12 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(153): OC\\\\Files\\\\Cache\\\\Scanner->getData('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...')\\n#13 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(420): OC\\\\Files\\\\Cache\\\\Scanner->scanFile('test\\\/\\\\xD0\\\\x9D\\\\xD0\\\\xBE\\\\xD0\\\\xB2\\\\xD1\\\\x8B\\\\xD0\\\\xB9...', 3, '455639', NULL, true)\\n#14 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(388): OC\\\\Files\\\\Cache\\\\Scanner->handleChildren('test', false, 3, '455639', true, -1)\\n#15 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(322): OC\\\\Files\\\\Cache\\\\Scanner->scanChildren('test', false, 3, '455639', true)\\n#16 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1301): OC\\\\Files\\\\Cache\\\\Scanner->scan('test', false)\\n#17 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1342): OC\\\\Files\\\\View->getCacheEntry(Object(OCA\\\\FilesAccessControl\\\\StorageWrapper), 'test', 'test_mail_ru\\\/te...')\\n#18 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/ObjectTree.php(160): OC\\\\Files\\\\View->getFileInfo('\\\/mnt\\\/files\\\/t...')\\n#19 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(939): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\ObjectTree->getNodeForPath('test_mail_ru\\\/te...')\\n#20 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(336): Sabre\\\\DAV\\\\Server->getPropertiesForPath('test_mail_ru\\\/te...', Array, 1)\\n#21 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPropFind(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#22 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#23 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PROPFIND', Array)\\n#24 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#25 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(60): Sabre\\\\DAV\\\\Server->exec()\\n#26 \\\/var\\\/www\\\/nextcloud\\\/remote.php(165): require_once('\\\/var\\\/www\\\/nextcl...')\\n#27 {main}\",\"File\":\"\\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/http\\\/lib\\\/Client.php\",\"Line\":160,\"User\":\"*****\"}","level":4,"time":"2017-03-25T14:54:21+00:00","method":"PROPFIND","url":"\/nextcloud\/remote.php\/webdav\/test_mail_ru\/test","user":"****","version":"11.0.2.7"}

2. occ files:scan stoped on files like this with error

error

Exception while scanning: Not Found #0 /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Client.php(231): Sabre\HTTP\Client->send(Object(Sabre\HTTP\Request)) #1 /var/www/nextcloud/lib/private/Files/Storage/DAV.php(268): Sabre\DAV\Client->propFind('https://webdav....', Array) #2 /var/www/nextcloud/lib/private/Files/Storage/DAV.php(708): OC\Files\Storage\DAV->propfind('glazyrinanv@gma...') #3 /var/www/nextcloud/lib/private/Files/Storage/Common.php(649): OC\Files\Storage\DAV->getPermissions('glazyrinanv@gma...') #4 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(577): OC\Files\Storage\Common->getMetaData('glazyrinanv@gma...') #5 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(577): OC\Files\Storage\Wrapper\Wrapper->getMetaData('glazyrinanv@gma...') #6 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php(460): OC\Files\Storage\Wrapper\Wrapper->getMetaData('glazyrinanv@gma...') #7 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php(173): OC\Files\Storage\Wrapper\Availability->getMetaData('glazyrinanv@gma...') #8 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(577): OC\Files\Storage\Wrapper\Encryption->getMetaData('glazyrinanv@gma...') #9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(114): OC\Files\Storage\Wrapper\Wrapper->getMetaData('glazyrinanv@gma...') #10 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(153): OC\Files\Cache\Scanner->getData('glazyrinanv@gma...') #11 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(420): OC\Files\Cache\Scanner->scanFile('glazyrinanv@gma...', 3, '452537', Object(OC\Files\Cache\CacheEntry), true) #12 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(388): OC\Files\Cache\Scanner->handleChildren('glazyrinanv@gma...', true, 3, '452537', true, 1296698) #13 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(322): OC\Files\Cache\Scanner->scanChildren('glazyrinanv@gma...', true, 3, '452537', true) #14 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(196): OC\Files\Cache\Scanner->scan('glazyrinanv@gma...', true, 3) #15 /var/www/nextcloud/apps/files/lib/Command/Scan.php(159): OC\Files\Utils\Scanner->scan('/mnt/files/\xD0...') #16 /var/www/nextcloud/apps/files/lib/Command/Scan.php(227): OCA\Files\Command\Scan->scanFiles('***', '/mnt/files/\xD0...', true, Object(Symfony\Component\Console\Output\ConsoleOutput), false) #17 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(256): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #18 /var/www/nextcloud/core/Command/Base.php(161): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #19 /var/www/nextcloud/3rdparty/symfony/console/Application.php(818): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #20 /var/www/nextcloud/3rdparty/symfony/console/Application.php(186): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #21 /var/www/nextcloud/3rdparty/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #22 /var/www/nextcloud/lib/private/Console/Application.php(169): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #23 /var/www/nextcloud/console.php(90): OC\Console\Application->run() #24 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...') #25 {main}

[kuimovvg]

one more thing. if file name on webdav server contain symbol "+" and webdav server response this name like %D0%B2%D0%B0%D1%80_DBC+%D0%A1%D0%A5.docx, then this name in class OC\Files\Storage\DAV function opendir transform this plus to space. After that, when nextcloud send propfind request, name with space transform to wrong url, and server respons 404 error

[LukasReschke]

cc @icewind1991

[icewind1991]

Letter "й" can be encoding by two ways %d0%b9 and %d0%b8%cc%86.

Not 100% and that's probably the problem, both sequences encode different characters that happen to look the same (NFC vs NFD encoding if you feel like googling deeper) since mixing these 2 forms is extremely confusing for the user (having multiple files in a folder with the "same" name) we normalize all file names to NFC (which rfc5198 recommends for names being send over the network) thus the %d0%b9 form is the one we use when requesting files over DAV.

[WaitF0r1t]

I can confirm that "+" (plus) symbols in filenames do not work as expected as they are decoded to " " (space) in \OC\Files\Storage\DAV::opendir. Thus reading e.g. subdirectories containing a "+" will result in a 404 status code when trying to retrieve their properties.

How to reproduce:
Set up a simple WebDAV server that serves a directory containing files and/or subdirectories containing "+" characters. In my case, I did this for an Apache 2 webserver with the following configuration:

  Alias /webdav "/path/to/directory/"
  <Directory "/path/to/directory/">
    Dav on
    Options Indexes
    IndexOptions +Charset=UTF-8

    AuthType Basic
    AuthName "WEBDAV"
    AuthUserFile /path/to/webdav.passwd
    Require valid-user
  </Directory>

After adding this resource to external storage, none of the directories/files containing "+" symbols appear in the directory listing and the Apache access log shows lines similar to the following

"PROPFIND /webdav/path/to/directory/with/plus/symbol HTTP/1.1" 404 928 "-" "sabre-dav/3.2.2 (http://sabre.io/)"

The accessed URLs have both spaces and plus symbols replaced with %20.

Possible solution:

Instead of urldecode you might want to use rawurldecode in \OC\Files\Storage\DAV::opendir, which keeps "+" symbols intact. I patched my installation and fixed it this way.

[carowsolutions]

I can confirm the same issue on a client's nextcloud instance. He synced files with a "+" in the filename. Which couldn't be found by nextcloud later. The files weren't deleted but not found on the file system because of the above mentioned issue.
Of course one shouldn't use "+" in file names but the underlying issue needs to be adressed.
In what jar should I put money to get this fixed?
Thanks for the amazing work by the way.

[nextcloud-bot]

Hey, this issue has been closed because the label stale is set and there were no updates for 14 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[WaitF0r1t]

Has this been fixed? If not, I think this issue should be reopened.

[GrayZeee]

Nextcloud 15.0.5. Issue is still reproducing

[bendem]

This is the same as #15849