Hello everyone, I have observed an interesting behavior since I am running on v1.10 on my two biggest shared folders. The scanning times increased a lot as it is going very slowly through the file system and take a couple of days to complete (initial scan and continuing scans). Log sample for shared folder tmc-vc:
Oct 20 06:20:22 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:22.451065 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698748137/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:20:24 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:24.450609 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698748137/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:20:26 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:26.450950 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698748137/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:20:28 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:28.451001 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698748137/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:20:30 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:30.451323 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698748137/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:20:32 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:32.451287 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698750223/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:20:34 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:20:34.451049 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698750223/12424138566 at 0.0 MiB/s (78%)
# from 06:20:32 till 06:21:06 it seems to stuck, than
Oct 20 06:21:06 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:21:06.450766 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698751266/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:21:08 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:21:08.451085 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698751266/12424138566 at 0.0 MiB/s (78%)
Oct 20 06:21:10 syncthingsrv1 syncthing_tmc[6770]: [WWQT1F] 2020/10/20 06:21:10.450562 walk.go:177: DEBUG: Walk tmc-vc [] current progress 9698751266/12424138566 at 0.0 MiB/s (78%)
# than it stuck again for nearly 30s and so on
From the webinterface where the “Scan Time Remaining” is jumping between 12 hours and > 1 month:
| Folder ID | tmc-vc |
|---|---|
| Folder Path | /data/shared/var/ |
| Global State | 754,936 4,783 ~131 GiB |
| Local State | 755,683 4,782 ~131 GiB |
| Reduced by ignore patterns | |
| Out of Sync Items | 1,231 items, ~65.3 MiB |
| Scan Time Remaining | ~ 3d 20h |
| Failed Items | 119 items |
On the another server which is on software level the exactly same, it is working with out any issues. The only difference is, that the working one is on running syncthing version 1.5 with 20G of RAM. In this version the servers were able to scan the full shared folder within 2 - 4 hours. In one forum post there were some talking about to less ram, so I increased it on the affected hosts to 32G.
Server specs
- OS: Debian 10
- CPU Cores: 8
- Mem: 32G
- FS: xfs inside of drbd, read: IOPS=55.0k, BW=215MiB/s (225MB/s)(3118MiB/14509msec)
Resources used
Syncthing uses 25.3% of the RAM and has a CPU usage (overall) between 480% and 620% and hase a nice level of 11 (which is default) and prio 31
Downgrade test
I have performed a downgrade for testing if it recovers and it did. So when it was back to version 1.5 it performs the scans way faster, after the upgrade to 1.10 again, it slows down.
Question
What can I do there to improve the scanning time again?
7 posts - 2 participants