That Syncing Feeling

The main reason I ended up not cancelling my Dropbox subscription last year was awkward, and is simultaneously also the reason why this post has been an entire year in the making.

Update: Two months later, I canceled my Dropbox subscription and moved to a OneDrive/SyncThing combo, almost exactly as described here. Notes on that are now online, but the short of it is that it works well enough for now.

But I hope it’s worth it to pick up again and expand upon, because this time maybe (just maybe) my notes on this topic can be useful.

The reason I am doing so is that after reading Nikita Prokopov’s post I decided to revisit the issues I had with getting Dropbox and OneDrive to work for me, since despite having used Dropbox since the very beginning I started using OneDrive for a lot of my personal stuff even before I joined Microsoft (obligatory Disclaimer).

And over the years, I have grown more and more disillusioned with Dropbox and its ever-increasing bloat, and yet couldn’t find the right fit even as I got more and more into using OneDrive.

SyncThing, which I was only tangentially considering last year, now looks like it actually make a lot of things much easier for me–but, alas, not everything, since it has a few fatal flaws that Nikita didn’t cover in his piece.

But let’s start with the compelling event for all of this–Dropbox’s increasing bloat and lack of focus on why people started using it in the first place.

Last year I summarized things like this (but never got around to post it):

Dropbox is a resource hog that has completely lost the focus on what is its most critical function–simple, transparent, zero-hassle file sync. Over the past couple of years, they went on a “value added” rampage that hiked prices (doubling storage even if you had no use for it), emphasised dubiously useful features, put the Linux client on life support and even shipped an entirely new, unremovable file manager across the board, without fixing any of the gripes people have had with it for many years.

If I could, I would ditch it and move on.

And a year later, all of it is still true:

• There is still no personal plan below 2TB (I am using 150GB, give or take, and paying for a massive amount of storage I don’t use).
• It still captures file change events outside its own folder, hogging the CPU whenever I do any significant amount of I/O outside the Dropbox folder.
• The menu/taskbar app has taken up ever-increasing amounts of RAM because it started shipping with its own Chromium runtime to render a UI I never use.
• The added bloat provides zero usable functionality inside its massive pop-up, which now also includes an ad to get me to use it on mobile even though Dropbox has to know I have mobile devices linked to my account.
• Adding insult to injury, Dropbox has enforced a three-device limit for free accounts (so they have to know how many and what devices people use.).
• Besides its intrusive Office plugin (which I took pains to disable) it now requires installing a system extension to do Smart Sync (i.e., on-demand access to offline files), which is a bit too much system access than I’m willing to grant.

And now they’re “adding value” with calendar syncing, a password manager, and goodness knows what else, when I need exactly none of those things (not even Smart Sync).

I don’t even need the Dropbox client on iOS–all I need is the file provider, which works very, very well (except just after an update, where I need to re-open the Dropbox app to get it working again for some reason).

Again, I had a great experience with the “core” Dropbox functionality for many years. I even used it to directly sync this website’s content from my machines (which was awesome for posting instantly), but it became so much of a hassle to keep the Linux daemon running that I now post things via git instead.

That was a challenge at first since the current repository has 2GB of content alone, but thanks to Anders Borum’s amazing Working Copy I can actually post link blog entries via Siri Shortcuts and have a pretty decent authoring experience.

For work and personal projects, I’ve been working with full-blown git repositories inside Dropbox for years now, and Dropbox has been rock solid in terms of speed, flexibility, reliability, and platform support.

I only had a couple of issues where conflicts or sync lag caused problems–the risks of having those were far outweighed by the ability to switch machines and have everything right there without having to git pull from slow (and sometimes hard to reach) internal repositories.

I could hack on something from my laptop, move a couple of floors, and pick things up again from my desktop without any friction–even if my laptop only picked up Wi-Fi for a portion of that time.

And then they decided to hobble the Linux client, and I decided to start looking for alternatives in earnest.

The reason I didn’t write more about this last year was that I tried migrating everything to OneDrive–and failed.

And I failed because my primary (personal) use case for file syncing is across two Macs and a few Linux machines, and it’s just not suitable for that.

It bears mentioning at this point that (besides using it every day for work, on Windows) I’ve been using Microsoft 365 Family (which used to be called “Office 365 Home”) for almost seven years (one more and change than I’ve been at Microsoft).

And it provides 1TB of OneDrive storage for each of its up to 6 users–which is still much more than I need to sync across machines, but effectively free when compared to Dropbox since in my mind I’m paying for the Office app licenses (across macOS and iOS) and not so much the associated services¹.

OneDrive works great for my family (the kids use it to store and share homework, etc.), but even after five years at Microsoft, I have exactly zero personal Windows machines except for a VM on my KVM box for running very specific software.

This may be atypical, but I’m not made of money, and I do (for the moment) prefer using a Mac for my personal projects (although to be honest that is looking like a dicier proposition with every macOS release).

As it turns out, OneDrive on the Mac is a far cry from the Windows version. It works mostly OK with Office files of any size and with relatively small numbers of files, but is completely unusable with many hundreds of thousands of small files.

And, guess what, what I do on my personal machines (coding, mostly) involves the constant manipulation of… hundreds of thousands of small files.

Which is why last year I wrote:

Whereas Dropbox can effortlessly (well, with some CPU usage, but quickly and transparently) sync all of my git repositories and project trees (even the ones with many thousands of junk duplicates inside multiple node_modules folders), OneDrive chokes on a working set that is less than a tenth of my current project tree (around 60000 files, including the ones inside .git folders). It simply does not sync anything and spends hours “processing changes”.

Or, worse, it will occasionally give up halfway, complain that there are already multi-gigabyte top-level folders with the same names and ask me to rename them to continue syncing.

Well, it’s been a year, and it still has the same issues, and that means I still can’t use OneDrive on my machines for what I need.

Besides there being no official Linux client (more on that later), the Mac client lags behind the Windows one in many respects–sometimes annoyingly so. Even though it seems to share some of the Qt code base (I am also not a fan of its huge menu/tray pop-up on any platform, but it is vastly better than Dropbox’s), it is a fundamentally different beast, and traditionally lags behind in features.

For instance (and this is a very dear example to me), the Personal Vault feature only works on Windows. I would switch to it in a heartbeat if it worked on macOS as well, for the very simple reason that I have kept an encrypted disk image on Dropbox to store critical documents since the very first day I started using Dropbox and would love to have a fully cross-platform solution for that.

But no, I can only use OneDrive’s Personal Vault from iOS or Windows…

Another dimension of feature parity is that it doesn’t handle native macOS files very well. For example, it won’t sync (sometimes even partially):

• Extended attributes
• Perfectly legal macOS filenames
• Very long pathnames
• Very large files (I have two VM images backed up on Dropbox that exceed the 15GB file limit)

All of these are known issues listed in this support page (which also includes variations for OneDrive for Business and SharePoint–but I’m only talking about OneDrive Personal here).

They might not be relevant for most users, but they are all blockers for me in one form or another, and during my failed migration attempt last year, I had a lot of issues with some older projects, to the extent where I just gave up and filed them away into disk images.

But to this day I still came up against filename and pathname issues whenever I am dealing with, say, a node_modules folder.

As to the Linux client aspect, abraunegg/onedrive is excellent for what it is (I had it installed on both my Linux laptops and a couple of Raspberry Pis to sync source trees), but it comes with a lot of caveats:

• It is written in D (for some reason), so contributing to it (let alone compiling it) is much harder than you’d expect (but yes, I got it to work on a Raspberry Pi).
• Lacks a lot of the niceties you get from Dropbox on Linux (it has no UI whatsoever).
• Has trouble tracking local changes effectively, since it uses a mix of inotify and polling that ended up generating quite a few conflicts while I was trying it out.
• It is not multi-threaded (still, in 2020), which effectively means it is several orders of magnitude slower than anything else (and that slowness also increases the chance of having conflicts).

It is pretty great if you have a few thousand files (or if you want to make periodic read-only snapshots from OneDrive), but on my huge working tree it would simply cause too many sync issues and be unusable on my slowest Linux laptop.

So much so that I had this (or a variant of it) right at the top of my .history on all of my machines for the next couple of months:

find . -name "*origin-Rui*" -type d -print0 | xargs -0 rm -rf

The conclusion I came to last year when I tried to move all my significant files from Dropbox to OneDrive before my Dropbox subscription renewed was that there was just no way I could get it to work for my personal projects–in particular, my Development folder and the hundreds of git repositories inside it.

Recapping what I already wrote above, I had never-ending issues with the amount, name, pathname depth, and sometimes even the kind of the files in question, to the point of OneDrive locking up the machine, failing to sync, or just crashing.

I tried everything from using gitbundle files to archiving stuff away, but gave up after a week or two of trying different ways to migrate things across (including diving into OCaml a bit and fixing a long-standing issue in the Unison file synchronizer, which I used to push some things across in lots).

That is not to say that there haven’t been changes and improvements in OneDrive.

For instance, about this time last year (mid-June, if I recall correctly), and after my first attempt at migrating, OneDrive shipped differential sync on the Mac, which sped up syncing large files considerably, as well as Files On-Demand (the equivalent to Dropbox Smart Sync), which I also tried to leverage.

But Files On-Demand did not improve the situation–in fact, it would frequently get stuck (and, worse, get other applications stuck) to an extent where I was unable to force quit an application using a remote file.

It also didn’t work with CLI tools (which is sad, since OneDrive in Windows even works–albeit slowly–with the Windows Subsystem for Linux).

Worse, there were instances when I could not even restart my Mac without resorting to sudo reboot, since stuck applications waiting for Files On-Demand would also cause the Dock and the Finder to become stuck themselves.

So I stopped using it altogether, and haven’t turned it back on since.

OneNote Weirdness

Another unexpected thing that I came across was “losing” all my OneNote notebooks during another migration attempt.

This is due to a strange interdependency between OneNote and OneDrive–despite OneNote having effectively gone web-only and stopped storing local document databases in the filesystem (which I understand from a sync perspective), it kept some filesystem shims in place that are a bit of an annoyance. OneNote notebooks are represented in OneDrive as .url files that look like this:

[InternetShortcut]
URL=https://onedrive.live.com/redir.aspx?cid=XXXXXXXXXXXXXX&resid=YYYYYYYYYYYY&type=3

When you create or open a new notebook, you are placed into the OneDrive virtual filesystem to do so, even though none of your data is actually accessible via the filesystem itself. You can’t for instance, see individual notes nor figure out how much storage a notebook is taking.

This is broken in many ways, and I’ve come to live with it everywhere, on and off work. But the biggest annoying side-effect I’ve come across from this approach is that conflicts on these files cause your notebooks to be renamed, which means that notebook Foo every now and then becomes Foo-Conflict-PCBARBAZ because (you guessed it) that .url file got renamed automatically.

Also, the second most critical feature I needed (seamless access from Files on iOS) worked unreliably at best–OneDrive occasionally takes hours to update the file provider view for some reason, and storing files from non-Office apps into OneDrive silently fails, with the new file not being uploaded at all for days².

Moving wholesale to OneDrive was (and still is) a no-go for my particular use case.

But for “civilians”, it’s plenty good enough. You have to keep in mind that despite the failure at moving my projects across, I keep all my Office documents and PDFs in it, as well as massive chunks of data like the entire working set of family photos I’m (patiently) curating and a few video projects.

So if you’re reading this because of Dropbox feature bloat and are on a Mac, I suggest you go and get it from the App Store right now and give it a go.

Because, unlike Dropbox, it can run sandboxed and comply with all of Apple’s security requirements, and I think that for most security-conscious people, that should be more than enough reason to use OneDrive instead of Dropbox (I could go on about data sovereignty and other things, but I do enough of that at work…)

So What Is This SyncThing… Thing Like?

Enter Nikita Prokopov’s post, and his unabashed praise of SyncThing’s simplicity.

Which is well-warranted, because it works brilliantly for syncing the kind of data I need, although to be fair I’ve only thrown a few tens of gigabytes at it yet (one of which is the 2GB repository this site lives in).

It just ate it up and synced it across three machines, in mere seconds. Impressive stuff. I’ve had no issues for the past three days or so, so I’m now moving to pile up on more data on a daily basis.

I’m going to do that piecemeal and systematically test for the same kind of trouble I came across last year, because reverting this kind of thing is a major pain, and I’ve been there before.

But even without being able to claim any sort of success yet, here are a few highlights from my experience so far:

• Decent (if limited) desktop clients for both macOS and Linux that do the basics (show it’s running, animate a tray icon to show activity and let you easily get at the config).
• “Magic” peer discovery/direct LAN sync (which is one thing at which Dropbox used to excel, and that OneDrive never did for me on the Mac).
• Ability to specify in a sane way what should be synced, with a simple .stignore text file (which is much nicer than Dropbox’s per-folder xattr -w com.dropbox.ignored 1, which is hidden away in a dark corner of the docs).
• Trivial setup on my Synology NAS using linuxserver/docker-syncthing (there is a “proper” Synology package, but it runs under it’s own UID and I wanted more control).
• Shared folders can be “send only” or “receive only”, which means my Synology can act as a mirror between machines and not have to bother with keeping track of local changes.
• Versioning is configurable on a per-shared-folder basis (I have it switched off for most folders, since it really piles up when you’re doing development, but on for personal documents).
• None of your data is stored anyplace you don’t control.

• Lack of cohesion across desktop clients.
• Having a web server (even if only bound to localhost). At least there’s an API key being exchanged with the desktop clients.
• Having to do most things through the Web UI.
• Playing around with keys and IDs (even if it does do local discovery, it’s not user-friendly enough for my tastes).
• File versions are stored locally on the client.
• None of your data is stored anyplace with automatic backups.

• Folder sync indicators (of debatable use, but I’m used to them and it’s hard to know what’s been synced or not before closing your laptop).
• Can’t easily toggle subfolder sync on/off (Nikita hates the Dropbox UX for that and I’m not a fan either, but it is useful and I used it a lot).
• Work on iOS (at all). It will never happen with the current codebase and/or iOS restrictions.
• Simple per-file or per-folder sharing with other people (which is a key feature of OneDrive, especially where it regards security and rich documents).
• Sync macOS Extended Attributes.

And this last one is the current showstopper for me (well, at least for general use).

What it means is that your Finder tags and pretty important metadata get stripped when you drop “old school” macOS bundles into SyncThing (or OneDrive, but that’s besides the point now).

So (for instance) application bundles (which you shouldn’t be trying to sync anyway, but which are a great example), some project bundles and other innocuous-looking document formats like Mindnode mind maps (or anything from any native macOS application that stores its data into a folder sewn together with extended attributes) is going to break.

Permanently, because essential (meta)data is not going to be synced at all. Which is kind of understandable, but apparently not even on SyncThing’s roadmap.

And out of all the “cross-platform” sync services, Dropbox is the only one that preserves (or at least doesn’t mangle) macOS extended attributes the way I need them to be handled. iCloud seems to do the right thing (it better, right?), but to be honest I don’t trust it enough yet to put really important files in.

This effectively means that your Documents folder has to be split up depending on what kind of data you have in it, which is impossible to manage from a project perspective. And even if you’re just talking about single files, it’s a pain to keep track of which apps’ data files can survive unscathed in which cloud service.

The way forward (for me at least) seems to be splitting things up: get my development projects and massive archives off Dropbox and onto SyncThing, keep all my “Office” stuff, photos and shared folders on OneDrive, and have a small subset of them in iCloud (and pray the latter works at all).

My Synology NAS may come in handy here since besides acting as a private SyncThing hub it can sync to/from both OneDrive and Dropbox (admittedly slowly, as I’ve seen over the past 48h)³.

And it is quite appealing to do that even if it means I’ll have a single point of failure, because it might well be that I can set up just SyncThing on my personal Mac and Linux machines (and have only one piece of software hammering my hard disks).

Then I’d point my machines at each other, add the NAS to the group and have that act as a back-to-back relay between select SyncThing folders and both Dropbox and OneDrive, as well as backing up the whole thing to Azure storage.

That would allow me to keep using the iOS clients I need (and probably stick to the Dropbox free tier on one or two apps that only work with Dropbox) as well as letting me achieve my primary goal, which is to migrate the bulk of my data off Dropbox and onto something that does solely what I need instead of wasting CPU cycles with random frippery…

Hopefully I won’t take another year to go about it, although to be fair I might end up spending more time (and money) fighting Dropbox than paying for another year.

Ironically, I’d probably pay the same for a plain client that worked just like the original and ten times less disk space, with zero added features. Zero!