-1

It's common knowledge that it's a bad idea to symlink (or straight up store) your code (with .git repos) and various config files (plist, ~/.config, zsh_history) in iCloud. It's gonna be a mess when you inevitably start seeing myfile (1), myfile (2) piling up in various subfolders. I tried, and indeed that's what happened.

I tried to go a different route: made a launchd-based daemon that runs my script hourly, in which I use rsync to copy files from where they are to iCloud-synced directory. This is better, and worked for a bit, but it's always breaking for some reason. One day rsync stops having permissions at random, another day my daemon just stops running. And somehow, the myfile (1), myfile (2) problem crops up anyway (albeit rarely).

So my question is — is there a way (a tool) that reliably and efficiently copies a few specific files and folders into an iCloud-synced dir? Without causing a mess, and without randomly seizing operation. Any help appreciated.

Max Chernyak
  • 109
  • iCloud is not a backup. If you lose it on your machine, you lose it in iCloud. It is only designed to mirror what you have locally. If you are looking to backup Git then push it up to a private repo on Github. As for a local backup use Time Machine. – Andy Griffiths Oct 14 '22 at 18:31
  • @AndyGriffiths Thank you Andy. I edited the word "backup" to the word "sync". I hope the question makes a bit more sense now. I use Backblaze backups, B2, Github, and various other methods of keeping actual backups. – Max Chernyak Oct 14 '22 at 19:11
  • Understood. In that case... maybe zip it up first? Pay attention to softlinks and how they are zipped. – Andy Griffiths Oct 14 '22 at 19:14
  • Another thought, if you're using iCloud to xfer your repo to another machine, why not just setup git to do the sync instead? You've not really said why you want to put a repo in iCloud in the first place. – Andy Griffiths Oct 14 '22 at 19:18
  • @AndyGriffiths For code: I like being able to access it on other devices. For configs (and code to some extent): I use iCloud as my starting point when changing machines. Login to iCloud, and run a few scripts/actions that restore most of my setup. – Max Chernyak Oct 14 '22 at 19:21
  • In that case I think making Github the authoritative copy, the origin, would be safest. Just get into the habit of pushing changes before changing machines. – Andy Griffiths Oct 14 '22 at 19:32
  • You're trying to avoid getting my file (1) and myfile (2), but there's no way to do that and also support two-directional syncing since the sync software can't know if my file (1) should be preserved. And if you're just trying to backup the files - then yeah use a dedicated solution for that. – Ezekiel Oct 14 '22 at 20:33
  • @AndyGriffiths Having a uncommitted working tree synced across devices is very important for me. I write a lot of demo and WIP code, and 20y of experience showed that no amount of discipline will make me commit every time. – Max Chernyak Oct 14 '22 at 20:52
  • @Ezekiel The thing is, I only have one source (my Mac). I never sync bi-directionally. These conflicts occur rarely due to various small glitches and race conditions. Ideally there shouldn't be a reason to ever get conflicts in uni-directional sync. And tools that work via iCloud API to store data do avoid conflicts. I would like to find something just as reliable for files. I don't mind if it periodically, carefully dumb-copies files only one way — from a folder, to an iCloud dir. And prevents any more activity until it's sure iCloud handled the writes completely. – Max Chernyak Oct 14 '22 at 20:52
  • Do you have the files which get duplicated by iCloud open in editors on several devices at once? – nohillside Oct 15 '22 at 05:45
  • @nohillside highly unlikely. There are legit reasons for conflicts, but these are definitely not them. I have not found any correlation between conflicts appearing and anything relevant being open. For example, I never open zsh_history on other devices, but it keeps multiplying anyway, even with “periodic copy” approach, without symlink or direct writes. – Max Chernyak Oct 15 '22 at 14:47
  • Your shell opens zsh_history automatically, I‘m not surprised that you get conflicts there. Actually, no syncing solution will solve conflicts like those. – nohillside Oct 15 '22 at 15:47
  • @nohillside The shell doesn't open it because I do cp -f "$HOME/.zsh_history" "$CLOUD_CONFIGS/Zsh/zsh_history" hourly, instead of pointing the shell directly at the file in iCloud. I'm just copying the file every hour. I don't open it on other devices. Still conflicts occur. – Max Chernyak Oct 15 '22 at 15:52
  • I'd still argue you are not using Git to its full potential. Branches are exactly the sort of mechanism one would use for your demo & WIP code. Commits are no big deal, I do them all the time when working on any of my coding projects. There's no real excuse for leaving things uncommitted for a long time. The work does not have to reach a major milestone to be worthy of a commit. More on why Git & iCloud aren't a good mix, see https://stackoverflow.com/questions/35853139/can-git-and-icloud-drive-be-effectively-used-together though I'm sure you already have researched that 'common knowledge'. – Andy Griffiths Oct 15 '22 at 15:59
  • @AndyGriffiths I hear you, but that's not quite it. I don't worry about making dirty commits. It's working from home, with kids/family who can urgently require you to leave your desk, while you have many work repos, oss contribs, and admin duties happening at once. At this point in life, you seek out ways to reduce demands placed on you by tech, instead paying good money to place your own demands on it. Dropbox doesn't seem to have this issue (based on my usage). However, iCloud is otherwise the most convenient drive for me. I'm willing to pay to solve this in iCloud. – Max Chernyak Oct 15 '22 at 16:28
  • If you get conflicts/duplicates from simply writing a file to iCloud on one computer only I doubt that any syncing method will work. OTOH I wonder what‘s going on because this use case shouldn’t create conflicts/duplicates at all (and I haven’t seen any for years, with files edited on several devices within a short time). – nohillside Oct 15 '22 at 22:58
  • @nohillside If you get conflicts[…] from […] writing a file […] on one computer […] I doubt that any syncing method will work. — Isn't it the opposite? It's more likely to be iCloud/FileProvider-specific because it's so odd. I have some evidence that this doesn't happen in Dropbox. I'm sure if we dig, we'll uncover some bug. For example due to a disconnect (I unplug cable and switch to wifi), while iCloud upload is in progress, and another file change occurs, it becomes a conflict. But I'd rather just find a tool that auto-handles this via queue, checks, delays, etc, until Apple fixes it. – Max Chernyak Oct 16 '22 at 01:02
  • What I meant was that if a simply periodic copy into iCloud leads to conflicts, any other copying method into iCloud will lead to the same issues. – nohillside Oct 16 '22 at 06:38
  • @nohillside ah, that’s a good point. Probably no naive copying method is going to work. However, I could imagine a piece of software doing some clever tricks to prevent this. – Max Chernyak Oct 16 '22 at 14:16

1 Answers1

0

For those who will find this by googling, here's an answer that gets you as close to working reliably as I could.

TLDR: Use cron with a script that runs cp/rsync commands. Give /usr/sbin/cron Full Disk Access in your Security & Privacy settings. This is more stable than launchd.

The long story

Technically, copying files into iCloud periodically should work without issues, especially if there's only one source. It didn't for me.

There are 2 "native" ways to implement periodic tasks in MacOS.

  1. Cron (supposed to be deprecated)
  2. Launchd (build your own daemon)

I went with #2, because of cron deprecation. It ended up being quite a hassle to wrap my script into a .app package to give it "Full Disk Access", but it sort of worked (thanks to the linked info). I also setup 2 log files (stdout/stderr) to make sure it's working.

I started having problems:

  1. The log files started being filled with permission issues. I solved some of them by disabling "checksum" checks in rsync. Still, issues continued, even in simple single file cp calls. I checked everything and couldn't figure out why issues persisted.
  2. Occasionally I started getting myfile (1) myfile (2) files show up due to conflicts, which shouldn't really be there. I haven't opened or changed these files on other devices.
  3. The job would just stop working at random. I couldn't figure out why. I would see an old last entry in logs.

Finally, after posting here, I gave up and tried to switch to cron (without much hope, because I didn't see how changing to a different runner would fundamentally address any of these issues). I added /usr/sbin/cron to Full Disk Access, and setup my hourly script like this:

0 * * * * /Users/max/Documents/Configurations/Executables/hourly-job >/Users/max/Library/Logs/HourlyJob/HourlyJob-stdout.log 2>/Users/max/Library/Logs/HourlyJob/HourlyJob-stderr.log

(These log locations allow me to see them in Console.app.)

I removed all the launchd stuff.

Surprise, now it just works. No more permission issues. So far no more conflicts, but that's harder to check so soon. They might recur later.

Also interestingly, according to this, cron is unlikely to ever be removed. It's been 9 years since that answer and cron is still here.

For those interested, here's how I copy files in my script:

#!/usr/bin/env bash
set -e

echo "job @ date +"%F %T %Z%z""


HOME="/Users/max"
LIBRARY=$HOME/Library
APP_SUPPORT="$LIBRARY/Application Support"
ICLOUD="$LIBRARY/Mobile Documents/com~apple~CloudDocs"
LOGS="$LIBRARY/Logs"
CLOUD_CONFIGS="$HOME/Documents/Configurations"
PREFERENCES="$LIBRARY/Preferences"


function rsync_file() {
  rsync -pgohuWE --inplace --stats "$1" "$2"
}


function rsync_dir() {
  rsync -auhOE --delete --stats "$1" "$2"
}


echo "syncing ~/Engineering"
rsync_dir "$HOME/Engineering" "$ICLOUD/"


echo "copying ScanSnap profiles"
cp -fp "$PREFERENCES/jp.co.pfu.ScanSnap.V10L10.plist" 

       "$CLOUD_CONFIGS/ScanSnap/jp.co.pfu.ScanSnap.V10L10.plist"


echo "syncing Bartender preferences"
rsync_file "$PREFERENCES/com.surteesstudios.Bartender.plist" 

           "$CLOUD_CONFIGS/Bartender/"


echo "copying zsh_history"
cp -fp "$HOME/.zsh_history" "$CLOUD_CONFIGS/Zsh/zsh_history"


… etc …
Max Chernyak
  • 109