Hi All
have been using Syncthing for Offsite and local backups of a Ubuntu webserver dealing with a lot of files (attachments)
All the nodes are running Ubuntu 18.04 VM’s on a Proxmox hypervisor. The folder I am syncing (send only on the host and receive only on the destinations)
It’s about 400GB of data, about 1.1mil files in one folder. All 3 hosts have been running fine over the last few years. A full scan on the slowest of the host takes about 11 hours running at 5MB/s in spinning disks (files and DB)
I recently setup a new VM for the webserver with a new syncthing install. I removed the folders from all the nodes (leaving data in place) and setup a new folder as before.
The destination nodes rescanned the data overnight, but the host as not even completed 1%. Its scanning at between 50kbps and 100kbps. The files are on a 4 Drive RAID 10 mechanical drive (sdb) and the db folder is on a 4 disk RAID 10 SSD (sda).
Syncthing seems to be mostly stuck on one thread running at 100% (VM has 10 threads)
top - 13:40:38 up 16:37, 1 user, load average: 1.26, 1.40, 1.45*
Tasks: 211 total, 1 running, 117 sleeping, 0 stopped, 0 zombie*
%Cpu(s): 2.7 us, 3.1 sy, 11.4 ni, 82.1 id, 0.3 wa, 0.0 hi, 0.1 si, 0.3 st*
KiB Mem : 25661780 total, 5813364 free, 10338184 used, 9510232 buff/cache*
KiB Swap: 8388604 total, 8388604 free, 0 used. 14903748 avail Mem*
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND*
20233 workpool 31 11 4209228 2.563g 16852 S 142.9 10.5 2:08.32 syncthing*
1503 workpool 20 0 5515768 2.361g 36372 S 31.9 9.6 140:47.11 java*
1427 www-data 20 0 245420 18692 9396 S 2.7 0.1 3:48.84 nginx*
1303 mysql 20 0 23.422g 4.539g 35544 S 2.3 18.5 18:12.81 mysqld*
Disk idle times are very high
avg-cpu: %user %nice %system %iowait %steal %idle
1.03 9.29 3.87 0.31 0.27 85.23
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
loop0 0.21 0.00 0.00 12 0
loop1 0.00 0.00 0.00 0 0
sdb 2.23 0.07 0.01 4287 418
sda 16.48 0.06 0.68 3443 40541
I played around with various tuning options to no effect. Eventually, I nuked the config and reinstalled syncthing with defaults. Now it’s not even connected to the other nodes, I only added the folder for scanning, still not getting anywhere.
Log file is not showing any errors.
Syncthing version is v1.18.3, Linux (64-bit Intel/AMD), previously I was using 1.7.1
Am I possibly looking at a bug, or is there something else I should look at?
5 posts - 3 participants