How to sync files across all your devices with Syncthing...

How to do things, solutions, recipes, tutorials
Post Reply
Message
Author
User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

How to sync files across all your devices with Syncthing...

#1 Post by Mike Walsh »

Evening, kiddiwinks.

Upholding the Puppy maxim of choice, I give you the following.....

I was browsing this evening (as you do!), and came across a link from the tech site MakeUseOf to a new type of free, open-source file-synchronisation app, called Syncthing:-

https://syncthing.net/

It works via peer-to-peer, in very much the same way that torrent clients (such as Transmission) do. In other words, all the work is performed by your own machine, and those you're connected to over the network; there is no 'middleman', or remote, cloud-based server, as with, for instance, DropBox. This ensures somewhat greater privacy.

(So it's an 'old', proven formula re-deployed in a different way for an alternative desired outcome. Nowt wrong wi' that....) :)

The first time you fire it up, it creates a folder alongside the 'Syncthing' directory, in which ever parent directory you've chosen to use, called, appropriately, 'Sync'. This can be renamed if you wish. You can sym-link other folders into this, as Syncthing respects hard links (except in Windoze). There will be a fair bit of CPU usage the first time around, as Syncthing gets everything 'matched up'; after this, it will be fairly modest.

Syncthing doesn't use a separate desktop GUI; just like CUPS in Puppy, you access it via the browser, on 'http://127.0.0.1:8384/'. There is apparently a GTK-based GUI available from the website; I haven't, however, tried this, since the browser window is quite good enough for my needs.


Image


It takes more setting-up than, say, 'ResilioSync' (formerly BitTorrentSync, which MakeUseOf compares it with), which is a paid app; this has a GUI, and a lot of the settings are performed 'behind the scenes'. However, I thought this one might appeal to some of you, as it's highly configurable, and it's possible to manually set-up just about every aspect of its operation.....which gives you very much greater control.

Personally, I'm running this from my remote data partition, sym-linked into /root (/mnt/home would work just as well, to keep things outside of PuppySpace), with the executable binary within the Syncthing directory sym-linked into /root/Startup, for auto-starting at every boot. Even though I'm effectively running just a single instance of Syncthing, you still need to set things up for each individual Pup, as it recognises each one through a unique 56-digit encrypted 'identifier' ID.

-------------------------------------------------------------

Upon extracting the tarball, I've renamed the folder inside the extracted one to 'Syncthing'; this is the one I've placed on sda1 (my remote data partition), and which has been sym-linked into each Pup. It contains everything needed for the app to run. The 'Sync' directory will be created alongside it in the same parent directory, wherever you decide to place it. If you do decide to place it in /root (and have it set to auto-start), when the default browser opens up with the Syncthing page showing, you will get a discreet suggestion not to 'run as root'. This can be safely ignored; all will still function correctly.

As with all sync apps, the hard work gets done the first time it runs. After this, it's just differentially scanning & updating recent changes only.

It's available in both 32- and 64-bit versions, and, so far, has fired straight up in every Pup I've tried it in. I'm assuming the binary must be statically compiled, with all required dependencies built-in.

The website has a comprehensive FAQ and extensive documentation to help with any queries. It's all pretty straight-forward; just takes a wee while to work your way through it.

I've tried many file-syncing applications in the past, and never got on too well with them; this, even a dunderhead like me can figure out. As the years roll by, and you amass more and more in the way of files'n'stuff, so it becomes more imperative that you find ways of managing and organising it all.

Hope some of you may find this useful. Enjoy.


Mike. :wink:

User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

#2 Post by Mike Walsh »

Upon further investigation, I've discovered I can sym-link the remote app into any directory I like to actually run it. It could go in /usr/local; /usr/lib; /opt.....anywhere at all, since it's totally self-contained; very much along the lines of the Electron-based apps, which are starting to become more common.

Just remember to sym-link the 'syncthing' binary inside the folder to /root/Startup, to auto-start at boot.

You can also set the actual 'Sync' directory to any location you like, too; simply specify the path when selecting 'Edit' on the local device in the left-hand pane.

----------------------------------------------------------------

As previously stated, when running multiple Puppies on the same machine, you can sym-link a single, remote instance of Syncthing into each Pup you wish to use it in, since, like the Electron-based WIRE VoiP app I discussed a few months ago, it's totally self-contained.....along with sym-linking the binary into /root/Startup.

However, because Syncthing creates a totally unique 56-digit 'identifier' for each hostname it encounters, and sets up a new profile accordingly, you will need to re-scan the 'Sync' directory's contents again.....which, when scanning a 40 GB directory with an elderly P4, does take a while..!

You'll also need to set things up to your liking in the 'Settings' window again.

-----------------------------------------------------------------

I previously said that Syncthing respects sym-links, and will follow them to update those directories. It doesn't.

I've chosen to sync three directories, which I run from a remote partition, sym-linked into /root; 'my-documents', 'Downloads', and 'Mike's Stuff'. I've created the 'Sync' directory in this partition, moved these three directories into it, then re-symlinked them back into /root from within this one. I then specified this as the path for the 'Sync' directory in the 'Local device' in the left-hand pane. Which works nicely!

-----------------------------------------------------------------

I'm very impressed with this app. In the words of a well-known British TV commercial, it 'does exactly what it says on the tin'; it keeps files in sync between selected machines on a continuous, ongoing basis. The 'scan' interval can be set to whatever you wish; the app comes set to once every 60 seconds by default. I find once every 5 mins (300 seconds) is just about right.

Recommended! :D


Mike. :wink:

User avatar
davids45
Posts: 1326
Joined: Sun 26 Nov 2006, 23:33
Location: Chatswood, NSW

Sync-ing either way?

#3 Post by davids45 »

G'day Mike,

Sounds like one of those complicated-reading/looking packages that are a doddle once you know what to do. Like rocket science?

My need (want) is for a package that will work both ways - primarily from my main desktop which stores everything but can it take changes from another device that has updated some stuff from the desktop?

When travelling interstate to visit grandchildren, I usually take a tablet and laptop. While browsing is OK, answering emails is a problem since a browser-based reply is lost from my main Mail archive on my desktop back home.

I'm thinking if I used this package to synchronise my Email on the laptop from the desktop, then when travelling and I did some emailing, could I then update the Mail archive on the desktop from the laptop when back home?

I read it is usable with Windows and Android, which could also be useful - if it means with old versions of Windows and Android OS.

For simple and quick file transfers between all my vices, oops devices :oops: , I use Dukto, but the possibility of synchronising between these is interesting.

Thanks for the posts.

David S.

User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

#4 Post by Mike Walsh »

Morning, David.

Yes, it's quite simple, once you get the hang of it. To answer your query about your eMail folders, it is b-directional; it uploads/downloads in both directions, synchronising both folders at the same time. It would have to, really.....'cos how's it going to 'sync' otherwise..?

There's only a handful of things you need to get right, TBH.

Click on the folder (top left) to expand it:-


Image


...then click on 'Edit'. This will bring up the Edit window. Initially, scroll down to the bottom & click on 'Remove'. This will remove the default folder, and allow you to set your Sync folder up wherever you want it, OK?

While things are non-operational (because it doesn't have a folder to aim at currently), you want to create the 'Sync' folder where you want it. If you want to sync multiple folders, it works best to move them all into the 'Sync' folder, them sym-link them back again to where you normally find them. But then, I don't need to explain that to you of all people!

Now; click on 'Folders' (top-left) again, followed by 'Edit' again. Now, you can set the details of your folder location, etc.


Image


1. Folder label:- I just use 'Sync'.

2. Folder ID:- This must be the same for all machines in the 'cluster'; it's what makes sure they're all singing from the same hymn-sheet!

3. Folder path:- Set this to the location of the 'Sync' folder.

Then, in the 'Advanced Settings':-


Image


1. Rescan interval. Set this to whatever you like.

2. Folder type. Leave this as is.

3. Minimum free disk space. I have bags of storage, so have this set to 1 GB. Again, set this to whatever you're happy with.

4. File pull order. Self-explanatory; I go with 'Newest first'.

5. File versioning. Not certain what this is, so I've left it alone. I daresay you can find the answer in the FAQ, by clicking on 'Documentation' at the bottom of the web-page.

6. Finally, hit 'Save', and it will begin to scan the 'Sync' folder's contents, to build the initial database.

--------------------------------------------------------------------------

Once it's scanned, and is showing as 'Unshared' in yellow lettering, you can then repeat the procedure with the next Puppy (or machine.) Repeat until as many as you wish are set-up.

--------------------------------------------------------------------------

When you have all your Pups set up with Syncthing, and the 'local' databases created, it's then time to set up syncing.

Each instance of Syncthing will create a unique 56-digit 'identifier'. You can find this top-right corner->Actions->Show ID. It's best, I find, to do this initially with your main machine. Make a note of this somewhere, 'cos you'll need it for the next step.

On your 'travel'(?) machine, open your browser, if it isn't already, to '127.0.0.1:8384', to bring the GUI up. This time, you want 'Remote Devices' (bottom right corner)->Add Remote Device. And this bit is super easy.

In the window that comes up, assuming both machines you want to sync are connected to the network, you'll see the unique ID of the other machine already displayed. Just like a smartphone's Bluetooth, it'll discover any other device nearby that has Syncthing installed. Unlike a smartphone, you don't need to turn it to 'Discover'. It's already on.

Just tick the boxes bottom left corner for the folder(s) you want to share, and hit 'Save'.

That's all there is to it. If you open the browser GUI in both machines, you can watch the progress. Once both browser GUIs are showing 'Up to date' in green, for both local and remote devices, you're 'in sync'.

It's cross-platform.....and available in both 32- and 64-bit.

Let me know how you get on, if you decide to try it.


Mike. :wink:

User avatar
Puppus Dogfellow
Posts: 1667
Joined: Tue 08 Jan 2013, 01:39
Location: nyc

this is excellent

#5 Post by Puppus Dogfellow »

thanks for this, Mike!

i like it so much i made a couple of pets from the linked packages. the icon is more "thing" than "sync" (looks like a turquoise gecko hand or neuron cluster more than rox folders being synced over distance, but you could grab a better icon from the gtk package (which i couldn't get to work)), but maybe this makes it a bit more convenient for folks:

syncthing-014-32.pet
syncthing-014-64.pet

click the desktop file and Syncthing opens in your default browser. it runs in the background (very lightly from what i've seen) so you can bookmark the tab to return to it rather than reclicking the icon. i think overall it's easier than dropbox to use--sync any folder or file on your network with any other folder or file on your network in as long as takes you to walk to the second computer plus maybe a minute or so. you can edit the cryptic names it gives you for the computers and folders after the fact---i feel like it unites my machines in a way dropbox didn't (it's nice to have backup without the third party, and i feel less tied into the previous configuration. if not a replacement, it's a great convenience). working great so far and thanks again, Mike!

____
updated (though automatic updates have been working well):

syncthing-014.38-64.pet and syncthing-014.38-32.pet
Last edited by Puppus Dogfellow on Fri 06 Oct 2017, 16:18, edited 1 time in total.

User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

#6 Post by Mike Walsh »

Hiya, Puppus.

Ah, good, good. Glad you like it. I'm really pleased to have found this one; I had an idea there might be others, besides myself, who'd find it handy.

Never got on with DropBox. Tried it for a couple of weeks, didn't like it.....gave it up as a bad job in the end. I guess all those years of running Windoze must have affected me more than I realised; I've always preferred utilities (like this kinda thing) that just quietly get on with their jobs in the background.....without making a fuss, or constantly trying to grab your attention. This fits in really well with the Linux philosophy of everything just being one big, common file-system.

Yes, it's not bad at all. I've previously always used Bill's Samba-TNG package to move stuff around from one machine to another, but with this, of course, it's all done automatically in the background. And it is limited to the selected folders, etc. If you were continuously syncing several hundred gigabytes of stuff between multiple machines, I reckon you'd need plenty of high-speed RAM, and high-capacity, top-notch SSDs. For starters..... :lol:

I shall still keep using Samba-TNG for stuff outside of those folders.

It is light on resources, isn't it? Like I said, on initial 'syncing' (when it's first building its database), it does use plenty of CPU cycles. But it only does this once. On my old P4, it uses around 2% in the background, once everything's synced together; on the big Compaq, with its dual-core Athlon64 X2, it's around the 0.6 % mark. Which is pretty good!

It's all about community 'choice', as far as I'm concerned. I know you can't please all of the folk, all of the time.....but the bigger the selection of stuff to choose from, most people should be able to find something to 'fit the bill'.

Nice one for making the pets, BTW. Cheers. Enjoy!


Mike. :wink:

slavvo67
Posts: 1610
Joined: Sat 13 Oct 2012, 02:07
Location: The other Mr. 305

#7 Post by slavvo67 »

Hi Mike,

This worked well in RU Xerus64. A couple of quick comments:

The generated ID is enormously long AND the very bad part is that there are O's and zero's? I can't stand when it's not easy to distinguish between the two. Like the small "L" and capital "i". Anyway, I got around this by copying the ID to a text file, saving to another usb and mounting the usb on the 2nd networked computer to do a copy and paste back into Syncthing. For the record, I think they are capital letter "O" and not zero;s but there was no way of telling for sure.

Also, and very important, I recommend Peasy Firewall Monitor to Build Firewall on Trusted Lan. Otherwise, the default Firewall settings won't allow the computers to connect to each other (unless you manually configure).

Worked pretty well so a nice addition, I think.

Thanks,

Slavvo67

User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

#8 Post by Mike Walsh »

Morning, slavvo.

Mm. It is a leetle bit on the long side, isn't it?

(I'll let you into a little secret.....I can't tell the difference either! *Shh...*) That's where keeping Samba-TNG alongside it comes in handy. I can use it for things like this; moving a Leafpad file with the 'identifier' from one machine to another, then pasting it in, as you've mentioned.

And what I really like about all this is that when you have two or more machines 'live' on the local network, as soon as the first 'scan' is completed (by whichever box, depending on how your scan intervals are set up), all Syncthing-equipped boxes recognise each other, auto-connect, and just get on with syncing in the background. I know Linux doesn't have anything like the ridiculous number of background services that Windoze does.....but this is a very useful one that I don't mind adding (especially with its low use of resources).

If I had more than two machines, I'd probably keep all the 'identifiers' in a folder on my NAS drive (which consists of a SanDisk 128 GB 'nano' flash drive, permanently plugged into the back of the router!! :lol: Works, though...) But then of course, you really only need them once.

I know it takes a wee while to set up, but it really is 'set it & forget it'; once it's done, you don't have to mess with it again.....except for maybe opening the browser GUI every so often to keep an eye on progress, and make modifications if, as & when needed.

If you have it set as a permanent bookmark on the bookmarks bar (I do in Chrome; left-hand end, next to my G-Drive), the icon will show you what's happening; when it's syncing, when it's disconnected, when it's 'unshared' (but waiting for a connection), etc. Neat.

Does the job nicely. Glad you like it, too.


Mike. :wink:

slavvo67
Posts: 1610
Joined: Sat 13 Oct 2012, 02:07
Location: The other Mr. 305

#9 Post by slavvo67 »

In theory, I can configure say 4 sticks to sync and send them 4 offices across the country and as long as they are on the same network, they will sync?

User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

#10 Post by Mike Walsh »

Hi, slavvo.

Mm. Well, I guess so. Not 100% sure, TBH. I mean, the whole idea of Syncthing is that it's designed to work over a network (usually LAN), but I don't suppose it matters if the network's 'spread-out'.....so long as all boxes are on the same one.

Give it a try. Can't hurt. 'No harm, no foul...'


Mike. :wink:

User avatar
spiritwild
Posts: 181
Joined: Mon 03 Oct 2016, 10:06

#11 Post by spiritwild »

Interesting... and quite useful.

I actually got this running. Took a bit of work and a lot of patience but it's now doing what it's designed to do. I also manage to not break everything else on my system as well.

(insert confetti and balloon celebration visual here)

On the down side, I'm now capable of beyondo amazing and brutally mind numbing quantitative, sub atomic, technical adventures.

Welcome to my Isotropic hell.

fernan
Posts: 449
Joined: Tue 23 Jan 2007, 13:56
Location: Buenos Aires

#12 Post by fernan »

Hello Guys. I'm trying to get rid of Dropbox, since I don't need a "cloud" storage, or access files from my phone, I just need to sync files in 4 computers I use usually to download docs, to be able to read or print them from the other computers.

Have you tried bittorentsync too? Do you recommend this Syncthing app instead of bittorrentsync?

I have 3 tahrpup computers and one (wife's) running Windows Vista.

Sharing links to be downloaded by external users would be an useful feature, I don't know if it can be done with this kind of setup

Thanks for your help

EDITED: answering my own question, I see there isn't any "bittorrentsync" anymore. So there is no other choice, I'm downloading the syncthing PET right now.

User avatar
Mike Walsh
Posts: 6351
Joined: Sat 28 Jun 2014, 12:42
Location: King's Lynn, UK.

#13 Post by Mike Walsh »

Hi, fernan.

TBH, I came across this completely by accident.

I was trawling through MakeUseOf one day, as I often do, and this was a 'link' in another article. The descriptor looked interesting, so I followed the link.....and the more I read about this, the neater it seemed.

So I downloaded it, tried it out (it does take a bit of doing to figure out all the necessary initial configuration, it's true).....and was very pleasantly surprised by the results.

I keep a lot of stuff on my above-mentioned small 'NAS', but it does mean manually opening it with YASSM and adding the things to it all the time. I'd been looking for something like this for a long time, which would automatically 'sync' & keep everything up-to-date, completely automatically, in the background, across a LAN. And that's exactly what this does, once you've set it all up (which does only need doing once for each Pup you add.)

Works brilliantly, too. The two important things to remember are to make sure everything you want to keep in sync goes into the 'Sync' directory.....and to make sure that every other Pup knows where to find it. (And it helps to have the partition where you keep the 'Sync' directory auto-mount at boot.....if you're running multiple Pups on one machine, as I do.) Really, if you want further advice, drop Puppus a line; he's more 'au fait' with anything to do with data manipulation than I am. I get the impression he's made it a bit of a specialty over the years..... :)


Mike. :wink:

fernan
Posts: 449
Joined: Tue 23 Jan 2007, 13:56
Location: Buenos Aires

#14 Post by fernan »

Well, downloaded and installed in 2 puppy machines on the same LAN, followed the user guide, but it's not syncing yet...

Should I shut down the puppy firewall , or open some ports to make it work?

Syncthing installed, added both devices, I can see them from the web interface, they show the folders location (added a new folder in both computers, the existent Dropbox folder) but the message "not sync" is always there...

I just added a new folder, added a new device with device ID at both computers, and that's all, started to scan the folders at both computers, restarted the daemon , but still "not sync". A message telling me that the other computer wants to share XX folder was shown, so they can see the other device.

User avatar
Puppus Dogfellow
Posts: 1667
Joined: Tue 08 Jan 2013, 01:39
Location: nyc

#15 Post by Puppus Dogfellow »

fernan wrote:Well, downloaded and installed in 2 puppy machines on the same LAN, followed the user guide, but it's not syncing yet...

Should I shut down the puppy firewall , or open some ports to make it work?

Syncthing installed, added both devices, I can see them from the web interface, they show the folders location (added a new folder in both computers, the existent Dropbox folder) but the message "not sync" is always there...

I just added a new folder, added a new device with device ID at both computers, and that's all, started to scan the folders at both computers, restarted the daemon , but still "not sync". A message telling me that the other computer wants to share XX folder was shown, so they can see the other device.
should work so long as you accept the request to share. only other thing i can see as the problem is the firewall--i generally run without one so i'm not sure how much of a factor it is, but if syncthing actually is running, and all the permissions are in order and paths correct, and it's still not syncing, it's probably that.

just noticed this earlier bit by slavvo on the firewall issue:
slavvo67 wrote:[...]
Also, and very important, I recommend Peasy Firewall Monitor to Build Firewall on Trusted Lan. Otherwise, the default Firewall settings won't allow the computers to connect to each other (unless you manually configure).

Worked pretty well so a nice addition, I think.

Thanks,

Slavvo67
so it seems it can be made to work with the firewall, just not with its default settings.

hth/enjoy/report problems

:D


here's a link to the Peasy Firewall Monitor thread:
http://www.murga-linux.com/puppy/viewto ... 273#742273

increa
Posts: 29
Joined: Sat 21 May 2011, 17:24

#16 Post by increa »

reply to 14 jul 17 post..

Mike Walsh,

When you posted on 14 July 2017, it appears maybe you haven't realized, you don't need a Sync folder, per se. You can have many separate and distinct sync folders or "sync pools" as I like to call them (because 2 or 3 or 5 or 50 computers can be in each pool). The only rule is that the folders cannot be embedded in each other. So no need to use symlinks at all.

This works as easily as well on smartphones, so I have a "photo" sync pool. For any picture I take on my smartphone, the pictures automatically show up on several other computers where I do mail or photo editing.

I have also used to it make a "backup aggregator" where one computer is in all the pools, and stores *its* copy of each file onto an encfs virtual directory. Then I have a separate sync pool that connect the actual encfs encrypted files to a friend's computer half way across the nation. It's my own encrypted cloud solution. Linux Format published a 2-part article from me about how to do this last year.

Also, the default web GUI is at 127.0.0.1:8384, but you can move it to the actual IP of your computer and any other port, say 192.168.1.45:45356. With a suitable username/password, this lets you log onto any Syncthing node from any other node on your local network using your browser. Routed through your modem/gateway/router, you can even give Syncthing a public internet facing connection and access the control GUI from across the nation.

The only problem I've seen is when mixing Linux/Mac/Windows computers and then doing an UC/LC name change, say from ProgramData.dat to Programdata.dat. Syncthing can't overcome the disagreement between the OS's if that's the same file or a different file. Linux/Mac say yes. Windows is tricky. It will report the different names accurately, but both point to the same actual file. Sort of (I guess) like two Linux hard links to the same disk file. Anyhow, it hoses Syncthing. One OS sees a change, and the other tries to make a change, but nothing changes, and Syncthing gets confused, issuing a caution error message "data changed but not scanned" or something like that. I get out of this trap by temporarily using a far-off name like Programdata-syncthing-temp.dat, letting that propagate through the system and then renaming it to what I want.

Also, syncthing now offers (summer 2019) send-only and receive-only access to a data share pool. So you can have 4 computers sharing the same data directory, but some can only read from it, some can only write to it and some can do both.

geo_c
Posts: 115
Joined: Mon 15 Nov 2010, 04:38

#17 Post by geo_c »

Two questions about sync thing.

I have it running on two machines and it's great!

Question one has to do with save files and device identifiers. I often just copy my pupsave directory from a frugal hard drive install into another separate USB install, so I can take my system in my pocket. If I were to do that with syncthing, would it get confused about which device it's running? Or would it assume it's on the same machine?

Question two, can I copy the files into a sync folder manually, off line, to avoid having to do it over the network, presumably after I accept the share from the first machine? Hoping that I could save time, and have syncthing essentially compare the two directories rather than copy all the files itself.

geo_c
Posts: 115
Joined: Mon 15 Nov 2010, 04:38

#18 Post by geo_c »

Here's a trick to get the best of both worlds.

Sync a dropbox directory, in other words a directory within the dropbox directory. As long as the machine with dropbox installed is on the network, dropbox files are uploaded to their server. So you can still have web based access on random computers.

So in other words, when I drop a file in that sync directory on a puppy install with no dropbox, the dropbox machine uploads it. So it's even easier to have a set of "web" files, and personal files all sync'd from the same device.

Post Reply