Cluster of three servers, A, B, & C.
- A is master/send only, Centos 7, syncthing 1.5.0 Linux x64
- B is receive/send, FreeBSD 11.2 release p15, syncthing 1.5.0 FreeBSD x64, UFS
- C is receive/send, FreeBSD 11.2 release p15, syncthing 1.5.0 FreeBSD x64, ZFS
These have been running beautifully for close to 14 months.
-
B was suddenly throwing out “too many open files” errors, increasing files limits and restart/reboot wasn’t making any difference.
-
Upgraded B to Syncthing 1.6.0-rc5 to see if that would help.
-
Suddenly receiving “connect: permission denied” on B for connections to A & C.
-
Upgrading A & C to 1.6.0-rc5 did not change the error message (restarted all three Syncthing servers). All are running as an unprivileged user, not root.
I’ve checked obvious things like DNS resolution consistency from all three viewpoints, firewall rules (but nothing had changed in the firewall rules when this started).
I’ve since seen it happening briefly between A & B when resuming one folder pair after hitting “Pause All” when seeing “too many open files” errors on B (with ZFS).
Possibly related or not, I’m seeing a new-to-me message under discovery errors:
global@https://discovery-v4.syncthing.net/v2/: Post "https://discovery-v4.syncthing.net/v2/": dial tcp: lookup discovery-v4.syncthing.net on 208.67.220.220:53: dial udp 208.67.220.220:53: socket: too many open files
(This does appear to disappear after a few minutes.)
There are quite a few “panic-date-time.reported.log” files in the syncthing config directory, but they are all within about a 2 hour time frame from several weeks ago.
Looking for suggestions and/or troubleshooting ideas.
5 posts - 3 participants