Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 17 Aug 2017, 02:17
All times are UTC - 4
 Forum index » House Training » HOWTO ( Solutions )
How to sync files across all your devices with Syncthing...
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [11 Posts]  
Author Message
Mike Walsh


Joined: 28 Jun 2014
Posts: 2828
Location: King's Lynn, UK.

PostPosted: Wed 12 Jul 2017, 19:57    Post subject:  How to sync files across all your devices with Syncthing...
Subject description: Easy-to-use file syncing software using only YOUR devices - NO 'middleman'
 

Evening, kiddiwinks.

Upholding the Puppy maxim of choice, I give you the following.....

I was browsing this evening (as you do!), and came across a link from the tech site MakeUseOf to a new type of free, open-source file-synchronisation app, called Syncthing:-

https://syncthing.net/

It works via peer-to-peer, in very much the same way that torrent clients (such as Transmission) do. In other words, all the work is performed by your own machine, and those you're connected to over the network; there is no 'middleman', or remote, cloud-based server, as with, for instance, DropBox. This ensures somewhat greater privacy.

(So it's an 'old', proven formula re-deployed in a different way for an alternative desired outcome. Nowt wrong wi' that....) Smile

The first time you fire it up, it creates a folder alongside the 'Syncthing' directory, in which ever parent directory you've chosen to use, called, appropriately, 'Sync'. This can be renamed if you wish. You can sym-link other folders into this, as Syncthing respects hard links (except in Windoze). There will be a fair bit of CPU usage the first time around, as Syncthing gets everything 'matched up'; after this, it will be fairly modest.

Syncthing doesn't use a separate desktop GUI; just like CUPS in Puppy, you access it via the browser, on 'http://127.0.0.1:8384/'. There is apparently a GTK-based GUI available from the website; I haven't, however, tried this, since the browser window is quite good enough for my needs.





It takes more setting-up than, say, 'ResilioSync' (formerly BitTorrentSync, which MakeUseOf compares it with), which is a paid app; this has a GUI, and a lot of the settings are performed 'behind the scenes'. However, I thought this one might appeal to some of you, as it's highly configurable, and it's possible to manually set-up just about every aspect of its operation.....which gives you very much greater control.

Personally, I'm running this from my remote data partition, sym-linked into /root (/mnt/home would work just as well, to keep things outside of PuppySpace), with the executable binary within the Syncthing directory sym-linked into /root/Startup, for auto-starting at every boot. Even though I'm effectively running just a single instance of Syncthing, you still need to set things up for each individual Pup, as it recognises each one through a unique 56-digit encrypted 'identifier' ID.

-------------------------------------------------------------

Upon extracting the tarball, I've renamed the folder inside the extracted one to 'Syncthing'; this is the one I've placed on sda1 (my remote data partition), and which has been sym-linked into each Pup. It contains everything needed for the app to run. The 'Sync' directory will be created alongside it in the same parent directory, wherever you decide to place it. If you do decide to place it in /root (and have it set to auto-start), when the default browser opens up with the Syncthing page showing, you will get a discreet suggestion not to 'run as root'. This can be safely ignored; all will still function correctly.

As with all sync apps, the hard work gets done the first time it runs. After this, it's just differentially scanning & updating recent changes only.

It's available in both 32- and 64-bit versions, and, so far, has fired straight up in every Pup I've tried it in. I'm assuming the binary must be statically compiled, with all required dependencies built-in.

The website has a comprehensive FAQ and extensive documentation to help with any queries. It's all pretty straight-forward; just takes a wee while to work your way through it.

I've tried many file-syncing applications in the past, and never got on too well with them; this, even a dunderhead like me can figure out. As the years roll by, and you amass more and more in the way of files'n'stuff, so it becomes more imperative that you find ways of managing and organising it all.

Hope some of you may find this useful. Enjoy.


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
Mike Walsh


Joined: 28 Jun 2014
Posts: 2828
Location: King's Lynn, UK.

PostPosted: Thu 13 Jul 2017, 18:30    Post subject:  

Upon further investigation, I've discovered I can sym-link the remote app into any directory I like to actually run it. It could go in /usr/local; /usr/lib; /opt.....anywhere at all, since it's totally self-contained; very much along the lines of the Electron-based apps, which are starting to become more common.

Just remember to sym-link the 'syncthing' binary inside the folder to /root/Startup, to auto-start at boot.

You can also set the actual 'Sync' directory to any location you like, too; simply specify the path when selecting 'Edit' on the local device in the left-hand pane.

----------------------------------------------------------------

As previously stated, when running multiple Puppies on the same machine, you can sym-link a single, remote instance of Syncthing into each Pup you wish to use it in, since, like the Electron-based WIRE VoiP app I discussed a few months ago, it's totally self-contained.....along with sym-linking the binary into /root/Startup.

However, because Syncthing creates a totally unique 56-digit 'identifier' for each hostname it encounters, and sets up a new profile accordingly, you will need to re-scan the 'Sync' directory's contents again.....which, when scanning a 40 GB directory with an elderly P4, does take a while..!

You'll also need to set things up to your liking in the 'Settings' window again.

-----------------------------------------------------------------

I previously said that Syncthing respects sym-links, and will follow them to update those directories. It doesn't.

I've chosen to sync three directories, which I run from a remote partition, sym-linked into /root; 'my-documents', 'Downloads', and 'Mike's Stuff'. I've created the 'Sync' directory in this partition, moved these three directories into it, then re-symlinked them back into /root from within this one. I then specified this as the path for the 'Sync' directory in the 'Local device' in the left-hand pane. Which works nicely!

-----------------------------------------------------------------

I'm very impressed with this app. In the words of a well-known British TV commercial, it 'does exactly what it says on the tin'; it keeps files in sync between selected machines on a continuous, ongoing basis. The 'scan' interval can be set to whatever you wish; the app comes set to once every 60 seconds by default. I find once every 5 mins (300 seconds) is just about right.

Recommended! Very Happy


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
davids45


Joined: 26 Nov 2006
Posts: 958
Location: Chatswood, NSW

PostPosted: Thu 13 Jul 2017, 19:53    Post subject: Sync-ing either way?
Subject description: From or to main desktop from other device?
 

G'day Mike,

Sounds like one of those complicated-reading/looking packages that are a doddle once you know what to do. Like rocket science?

My need (want) is for a package that will work both ways - primarily from my main desktop which stores everything but can it take changes from another device that has updated some stuff from the desktop?

When travelling interstate to visit grandchildren, I usually take a tablet and laptop. While browsing is OK, answering emails is a problem since a browser-based reply is lost from my main Mail archive on my desktop back home.

I'm thinking if I used this package to synchronise my Email on the laptop from the desktop, then when travelling and I did some emailing, could I then update the Mail archive on the desktop from the laptop when back home?

I read it is usable with Windows and Android, which could also be useful - if it means with old versions of Windows and Android OS.

For simple and quick file transfers between all my vices, oops devices Embarassed , I use Dukto, but the possibility of synchronising between these is interesting.

Thanks for the posts.

David S.
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 2828
Location: King's Lynn, UK.

PostPosted: Fri 14 Jul 2017, 06:11    Post subject:  

Morning, David.

Yes, it's quite simple, once you get the hang of it. To answer your query about your eMail folders, it is b-directional; it uploads/downloads in both directions, synchronising both folders at the same time. It would have to, really.....'cos how's it going to 'sync' otherwise..?

There's only a handful of things you need to get right, TBH.

Click on the folder (top left) to expand it:-





...then click on 'Edit'. This will bring up the Edit window. Initially, scroll down to the bottom & click on 'Remove'. This will remove the default folder, and allow you to set your Sync folder up wherever you want it, OK?

While things are non-operational (because it doesn't have a folder to aim at currently), you want to create the 'Sync' folder where you want it. If you want to sync multiple folders, it works best to move them all into the 'Sync' folder, them sym-link them back again to where you normally find them. But then, I don't need to explain that to you of all people!

Now; click on 'Folders' (top-left) again, followed by 'Edit' again. Now, you can set the details of your folder location, etc.





1. Folder label:- I just use 'Sync'.

2. Folder ID:- This must be the same for all machines in the 'cluster'; it's what makes sure they're all singing from the same hymn-sheet!

3. Folder path:- Set this to the location of the 'Sync' folder.

Then, in the 'Advanced Settings':-





1. Rescan interval. Set this to whatever you like.

2. Folder type. Leave this as is.

3. Minimum free disk space. I have bags of storage, so have this set to 1 GB. Again, set this to whatever you're happy with.

4. File pull order. Self-explanatory; I go with 'Newest first'.

5. File versioning. Not certain what this is, so I've left it alone. I daresay you can find the answer in the FAQ, by clicking on 'Documentation' at the bottom of the web-page.

6. Finally, hit 'Save', and it will begin to scan the 'Sync' folder's contents, to build the initial database.

--------------------------------------------------------------------------

Once it's scanned, and is showing as 'Unshared' in yellow lettering, you can then repeat the procedure with the next Puppy (or machine.) Repeat until as many as you wish are set-up.

--------------------------------------------------------------------------

When you have all your Pups set up with Syncthing, and the 'local' databases created, it's then time to set up syncing.

Each instance of Syncthing will create a unique 56-digit 'identifier'. You can find this top-right corner->Actions->Show ID. It's best, I find, to do this initially with your main machine. Make a note of this somewhere, 'cos you'll need it for the next step.

On your 'travel'(?) machine, open your browser, if it isn't already, to '127.0.0.1:8384', to bring the GUI up. This time, you want 'Remote Devices' (bottom right corner)->Add Remote Device. And this bit is super easy.

In the window that comes up, assuming both machines you want to sync are connected to the network, you'll see the unique ID of the other machine already displayed. Just like a smartphone's Bluetooth, it'll discover any other device nearby that has Syncthing installed. Unlike a smartphone, you don't need to turn it to 'Discover'. It's already on.

Just tick the boxes bottom left corner for the folder(s) you want to share, and hit 'Save'.

That's all there is to it. If you open the browser GUI in both machines, you can watch the progress. Once both browser GUIs are showing 'Up to date' in green, for both local and remote devices, you're 'in sync'.

It's cross-platform.....and available in both 32- and 64-bit.

Let me know how you get on, if you decide to try it.


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
Puppus Dogfellow


Joined: 07 Jan 2013
Posts: 1483
Location: nyc

PostPosted: Mon 17 Jul 2017, 10:26    Post subject: this is excellent
Subject description: working well in tahr605-64 and xenial704-32
 

thanks for this, Mike!

i like it so much i made a couple of pets from the linked packages. the icon is more "thing" than "sync" (looks like a turquoise gecko hand or neuron cluster more than rox folders being synced over distance, but you could grab a better icon from the gtk package (which i couldn't get to work)), but maybe this makes it a bit more convenient for folks:

syncthing-014-32.pet
syncthing-014-64.pet

click the desktop file and Syncthing opens in your default browser. it runs in the background (very lightly from what i've seen) so you can bookmark the tab to return to it rather than reclicking the icon. i think overall it's easier than dropbox to use--sync any folder or file on your network with any other folder or file on your network in as long as takes you to walk to the second computer plus maybe a minute or so. you can edit the cryptic names it gives you for the computers and folders after the fact---i feel like it unites my machines in a way dropbox didn't (it's nice to have backup without the third party, and i feel less tied into the previous configuration. if not a replacement, it's a great convenience). working great so far and thanks again, Mike!
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 2828
Location: King's Lynn, UK.

PostPosted: Mon 17 Jul 2017, 13:30    Post subject:  

Hiya, Puppus.

Ah, good, good. Glad you like it. I'm really pleased to have found this one; I had an idea there might be others, besides myself, who'd find it handy.

Never got on with DropBox. Tried it for a couple of weeks, didn't like it.....gave it up as a bad job in the end. I guess all those years of running Windoze must have affected me more than I realised; I've always preferred utilities (like this kinda thing) that just quietly get on with their jobs in the background.....without making a fuss, or constantly trying to grab your attention. This fits in really well with the Linux philosophy of everything just being one big, common file-system.

Yes, it's not bad at all. I've previously always used Bill's Samba-TNG package to move stuff around from one machine to another, but with this, of course, it's all done automatically in the background. And it is limited to the selected folders, etc. If you were continuously syncing several hundred gigabytes of stuff between multiple machines, I reckon you'd need plenty of high-speed RAM, and high-capacity, top-notch SSDs. For starters..... Laughing

I shall still keep using Samba-TNG for stuff outside of those folders.

It is light on resources, isn't it? Like I said, on initial 'syncing' (when it's first building its database), it does use plenty of CPU cycles. But it only does this once. On my old P4, it uses around 2% in the background, once everything's synced together; on the big Compaq, with its dual-core Athlon64 X2, it's around the 0.6 % mark. Which is pretty good!

It's all about community 'choice', as far as I'm concerned. I know you can't please all of the folk, all of the time.....but the bigger the selection of stuff to choose from, most people should be able to find something to 'fit the bill'.

Nice one for making the pets, BTW. Cheers. Enjoy!


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
slavvo67

Joined: 12 Oct 2012
Posts: 1421
Location: The other Mr. 305

PostPosted: Wed 19 Jul 2017, 19:50    Post subject:  

Hi Mike,

This worked well in RU Xerus64. A couple of quick comments:

The generated ID is enormously long AND the very bad part is that there are O's and zero's? I can't stand when it's not easy to distinguish between the two. Like the small "L" and capital "i". Anyway, I got around this by copying the ID to a text file, saving to another usb and mounting the usb on the 2nd networked computer to do a copy and paste back into Syncthing. For the record, I think they are capital letter "O" and not zero;s but there was no way of telling for sure.

Also, and very important, I recommend Peasy Firewall Monitor to Build Firewall on Trusted Lan. Otherwise, the default Firewall settings won't allow the computers to connect to each other (unless you manually configure).

Worked pretty well so a nice addition, I think.

Thanks,

Slavvo67
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 2828
Location: King's Lynn, UK.

PostPosted: Thu 20 Jul 2017, 04:55    Post subject:  

Morning, slavvo.

Mm. It is a leetle bit on the long side, isn't it?

(I'll let you into a little secret.....I can't tell the difference either! *Shh...*) That's where keeping Samba-TNG alongside it comes in handy. I can use it for things like this; moving a Leafpad file with the 'identifier' from one machine to another, then pasting it in, as you've mentioned.

And what I really like about all this is that when you have two or more machines 'live' on the local network, as soon as the first 'scan' is completed (by whichever box, depending on how your scan intervals are set up), all Syncthing-equipped boxes recognise each other, auto-connect, and just get on with syncing in the background. I know Linux doesn't have anything like the ridiculous number of background services that Windoze does.....but this is a very useful one that I don't mind adding (especially with its low use of resources).

If I had more than two machines, I'd probably keep all the 'identifiers' in a folder on my NAS drive (which consists of a SanDisk 128 GB 'nano' flash drive, permanently plugged into the back of the router!! Laughing Works, though...) But then of course, you really only need them once.

I know it takes a wee while to set up, but it really is 'set it & forget it'; once it's done, you don't have to mess with it again.....except for maybe opening the browser GUI every so often to keep an eye on progress, and make modifications if, as & when needed.

If you have it set as a permanent bookmark on the bookmarks bar (I do in Chrome; left-hand end, next to my G-Drive), the icon will show you what's happening; when it's syncing, when it's disconnected, when it's 'unshared' (but waiting for a connection), etc. Neat.

Does the job nicely. Glad you like it, too.


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
slavvo67

Joined: 12 Oct 2012
Posts: 1421
Location: The other Mr. 305

PostPosted: Fri 21 Jul 2017, 09:48    Post subject:  

In theory, I can configure say 4 sticks to sync and send them 4 offices across the country and as long as they are on the same network, they will sync?
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 2828
Location: King's Lynn, UK.

PostPosted: Fri 21 Jul 2017, 17:46    Post subject:  

Hi, slavvo.

Mm. Well, I guess so. Not 100% sure, TBH. I mean, the whole idea of Syncthing is that it's designed to work over a network (usually LAN), but I don't suppose it matters if the network's 'spread-out'.....so long as all boxes are on the same one.

Give it a try. Can't hurt. 'No harm, no foul...'


Mike. Wink

_________________
If I've helped you.....please say 'Thanks'!
MY PUPPY PACKAGES
--------------------------------------

Back to top
View user's profile Send private message Visit poster's website 
spiritwild


Joined: 03 Oct 2016
Posts: 78

PostPosted: Fri 21 Jul 2017, 20:32    Post subject:  

Interesting... and quite useful.

I actually got this running. Took a bit of work and a lot of patience but it's now doing what it's designed to do. I also manage to not break everything else on my system as well.

(insert confetti and balloon celebration visual here)

On the down side, I'm now capable of beyondo amazing and brutally mind numbing quantitative, sub atomic, technical adventures.

Welcome to my Isotropic hell.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [11 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » HOWTO ( Solutions )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1174s ][ Queries: 14 (0.0121s) ][ GZIP on ]