Technical Deep Dives

• Verify remote
• Number of files
• Swift concurrency
• Tagging of data

Verify remote

Overview

Synchronization of Multiple Macs to a Remote Server

I have over 3,000 bird photos (130 GB) from the past four years that are synchronized using RsyncUI to a local remote server at home. New photos are added, old photos are deleted, and updates are made to photo sidecars. As long as I was using only one Mac, all updates were made on that Mac. However, with two Macs, I now use both to work on my photos. When I synchronize my changes, I need to transfer those changes to my second Mac.

My Setup for securing bird photos

I back up my bird photography to multiple locations:

• a Raspberry Pi 5 server configured with two WD Red SA500 2.5" SSD 1TB drives set up as a mirrored ZFS pool
• a remote cloud service (JottaCloud)
• two 1TB NVMe external SSD drives attached to my computers

Typically, a synchronization action operates in a one-way direction. Local data is synchronized to backup media, such as an attached disk or a remote server. Restoring data, for instance, involves retrieving data from a backup when local data has become corrupted or inaccessible.

If you are using multiple Macs, as I do, and all Macs synchronize data to the same remote storage, there may be challenges maintaining synchronization and preventing data loss, particularly if the remote storage is not a Git server, such as GitHub and Gitea. If the remote destinations are stored on a Git server, regular git push and git pull commands will suffice.

Git is a superior tool for version control. However, in certain situations, creating a Git repository may not be feasible, and this function may prove useful. As a reminder, the Verify function is designed for multiple Macs synchronizing data to a single remote server as a backup. It also assists in deciding whether to push or pull changes to keep the local repository updated.

Arguments for rsync

The following arguments are used in both push and pull.

• --itemize-changes - output change-summary for all updates
• --dry-run - rsync execute an estimate run
• --update - evaluates the timestamp

Itemized output - push or pull

The parameter -i or --itemize-changes produces details about each file. The format of the output is:

YXcstpoguax
|||||||||||
`-------------------------- the TYPE OF UPDATE:
 ||||||||||   <: file is being transferred to the remote host (pushed).
 ||||||||||   >: file is being transferred to the local host (pulled).
 ||||||||||   c: local change/creation for the item, such as:
 ||||||||||      - the creation of a directory
 ||||||||||      - the changing of a symlink,
 ||||||||||      - etc.
 ||||||||||   h: the item is a hard link to another item (requires --hard-links).
 ||||||||||   "+" - the file is newly created
 ||||||||||   .: the item is not being updated (though it might have attributes that are being modified).
 ||||||||||   *: means that the rest of the itemized-output area contains a message (e.g. "deleting").
 ||||||||||
 `----------------------------- the FILE TYPE:
  |||||||||   f for a file,
  |||||||||   d for a directory,
  |||||||||   L for a symlink,
  |||||||||   D for a device,
  |||||||||   S for a special file (e.g. named sockets and fifos).
  |||||||||
  `--------- c: different checksum (for regular files)
   ||||||||     CHANGED VALUE (for symlink, device, and special file)
   `-------- s: Size is different
    `------- t: Modification time is different
     `------ p: Permission are different
      `----- o: Owner is different
       `---- g: Group is different
        `--- u: The u slot is reserved for future use.
         `-- a: The ACL information changed

Position  Letter  Meaning
--------  ------  ----------------------------------
    1       Y     Update type (<, >, c, h, ., *)
    2       X     File type (f, d, L, D, S)
    3       c     Checksum/content
    4       s     Size
    5       t     Time (modification)
    6       p     Permissions
    7       o     Owner
    8       g     Group
    9       u     Reserved/user time
   10       a     ACL
   11       x     Extended attributes

Special:  '.' = unchanged, '+' = new item

The Stand alone application

This application is currently in development. The majority of the code is derived from the code base of RsyncUI, and there are some new code modules associated with the Verify Remote function. Additionally, some new views have been added for evaluation purposes.

Overview

The --itemize-changes (or -i) option in rsync provides a detailed, itemized list of changes being made to each file during synchronization. The output format is an 11-character string followed by the file path.

Position 1: Update Type (Y)

The first character indicates the type of update operation.

Position 2: File Type (X)

The second character indicates what type of filesystem object is being updated.

Position 3: Checksum/Content (c)

Position 4: Size (s)

Position 5: Modification Time (t)

Position 6: Permissions (p)

Position 7: Owner (o)

Position 8: Group (g)

Position 9: User/Reserved (u)

Note: This position is reserved and its behavior may vary between rsync versions.

Position 10: ACL (a)

Note: Requires ACL support to be compiled into rsync.

Position 11: Extended Attributes (x)

Note: Requires extended attribute support to be compiled into rsync.

Common Examples

New Files

>f+++++++++ documents/report.pdf

• > = Receiving from remote
• f = Regular file
• +++++++++ = All attributes are new (new file)

Updated File

>f. st.. .... images/photo.jpg

• > = Receiving from remote
• f = Regular file
• s = Size changed
• t = Modification time changed
• Other attributes unchanged

Directory Timestamp Change

.d..t...... src/components/

• . = Not being transferred (no update)
• d = Directory
• t = Modification time changed
• Other attributes unchanged

Permission Change

.f... p. .... scripts/deploy.sh

• . = Not being transferred
• f = Regular file
• p = Permissions changed (e.g., made executable)
• Other attributes unchanged

New Directory

cd+++++++++ backup/2024/

• c = Local creation
• d = Directory
• +++++++++ = All attributes are new

Symlink Changes

cLc.t...... config/current -> v2.0

• c = Local change
• L = Symlink
• c = Target/value changed (position 3)
• t = Modification time changed
• Other attributes unchanged

Hard Link

hf...... .... docs/readme.txt => docs/README.txt

• h = Hard link
• f = File
• Other positions indicate which attributes differ between the link and its target

Deletion

*deleting   old/obsolete.txt

• * = Special message
• Message indicates file is being deleted

File Being Sent

<f. st...... data/export.csv

• < = Sending to remote
• f = Regular file
• s = Size differs
• t = Time differs

Complex Example: Multiple Changes

>f.stpog...  /var/www/index.html

• > = Receiving file
• f = Regular file
• s = Size changed
• t = Modification time changed
• p = Permissions changed
• o = Owner changed
• g = Group changed

Version Differences

rsync 3.0.9+ (Modern)

• Uses 11 characters: YXcstpoguax
• Includes ACL (a) and extended attributes (x)

rsync 2.6.8 and earlier (Legacy)

• Uses 9 characters: YXcstpogz
• No ACL or extended attribute indicators
• Position 9 was z (related to compression)

Number of files

Numbers updated: December 21, 2025 (version 2.8.4).

RsyncUI depends only on the standard Swift and SwiftUI toolchain—no external libraries.

cloc DecodeEncodeGeneric/Sources ParseRsyncOutput/Sources RsyncArguments/Sources RsyncUI/RsyncUI RsyncUIDeepLinks/Sources SSHCreateKey/
Sources RsyncProcessStreaming/Sources ProcessCommand/Sources
     207 text files.
     207 unique files.                                          
      15 files ignored.

github.com/AlDanial/cloc v 2.06  T=0.06 s (3336.2 files/s, 362295.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Swift                          201           2551           2447          17056
C                                2             36             72            254
XML                              2              0              0             53
JSON                             1              0              0              6
C/C++ Header                     1              1              3              0
-------------------------------------------------------------------------------
SUM:                           207           2588           2522          17369
-------------------------------------------------------------------------------

Main Repository

• RsyncUI (https://github.com/rsyncOSX/RsyncUI) - the main repository for RsyncUI

Swift Packages used by RsyncUI

All SPM packages are refactored, updated, and tracked in the main branch. RsyncUI depends on all of them except the last, which is optional because SSH keys can be generated from the command line.

• RsyncProcessStreaming (https://github.com/rsyncOSX/RsyncProcessStreaming) - A core function of RsyncUI
  • listens for output from the rsync process as well as termination signal
• ProcessCommand (https://github.com/rsyncOSX/ProcessCommand) - As above, but for commands other than rsync
• RsyncArguments (https://github.com/rsyncOSX/RsyncArguments) - Generate parameters for rsync based on configurations
• DecodeEncodeGeneric (https://github.com/rsyncOSX/DecodeEncodeGeneric) - Generic code for decoding and encoding JSON data
• ParseRsyncOutput (https://github.com/rsyncOSX/ParseRsyncOutput) - Parse and extract numerical values from the output of rsync
  • this data is used to display details and log results for synchronized tasks
• RsyncUIDeepLinks (https://github.com/rsyncOSX/RsyncUIDeepLinks) - Parse and return valid URL deeplinks to execute tasks directly within RsyncUI
• sshCreateKey (https://github.com/rsyncOSX/sshCreateKey) - Assist in creating an SSH identity file and key using RsyncUI
  • generate an RSA-based SSH key for default and user-defined keys, including the SSH port number

Swift concurrency

RsyncUI is a GUI app; most work happens on the main thread. Heavier tasks run on threads from the cooperative thread pool (CTP) without blocking the UI. The Swift runtime manages the executors and CTP. There are three kinds of executors:

• the main executor manages jobs on the main thread
• the global concurrent executor and the serial executor execute jobs on threads from the CTP

Most work in RsyncUI runs on the main thread. SwiftUI keeps UI updates there by default. Examples include:

• preparing and executing rsync synchronization tasks (including building arguments)
• monitoring progress and termination of rsync tasks
  • the collection of output from rsync is performed by an actor, while the actual reporting of the number of files transferred is executed on a main thread for UI updates
• some write and read operations

Swift Version 6 and the New Concurrency Model

Swift version 6 introduced strict concurrency checking. By enabling Swift 6 language mode and strict concurrency checking, Xcode helps identify and resolve possible data races at compile time.

Quote from swift.org: “More formally, a data race occurs when one thread accesses memory while the same memory is being modified by another thread. The Swift 6 language mode eliminates these issues by preventing data races at compile time.”

RsyncUI adheres to the Swift 6 concurrency model.

Swift Concurrency and Asynchronous Execution

Swift makes asynchronous code more approachable through async, await, and actor. RsyncUI adopts these features even though most work can remain on the main thread—as long as it does not block the UI.

Asynchronous work can happen on the main thread or on background threads from the CTP. On the main thread, structured concurrency with async/await is key; every await yields control so other tasks can proceed.

Cooperative Thread Pool (CTP)

These RsyncUI tasks run asynchronously on CTP threads under the actor protocol:

• read synchronization tasks from file
  • JSON data decoding: asynchronous decoding that inherits the actor’s thread
  • JSON data encoding: synchronous encoding on the main thread
• read and sort log records
• delete log records
• prepare output from rsync for display
• prepare data from the log file for display
• check for updates to RsyncUI

Adhering to the actor protocol, all access to actor properties must be asynchronous. RsyncUI has five actors, plus additional async functions; some also run on the main thread.

Structured Concurrency

Some functions use async let for structured concurrency. Multiple async let bindings run concurrently, and execution resumes once they all complete.

Structured concurrency also shapes ordering. Each await suspends until that async work finishes; sequential awaits run one after another.

Example of structured concurrency

About 800 of 1500 log records are selected for deletion. Two operations run concurrently on a background thread: deleting selected logs and updating the remaining records for display.

Deletion starts on the main thread. The user selects logs and taps Delete to begin or cancel. The asynchronous deletelogs() function is launched on the main thread.

Button("Delete", role: .destructive) {
      	Task {
              await deletelogs(selectedloguuids)
            }
  }

deletelogs starts on the main thread and continues on a background thread inside Task {}.

func deletelogs(_ uuids: Set<UUID>) async {
        Task {
        
            print("(1) start async let updatedRecords deletelogs")
            
            async let updatedRecords: [LogRecords]? = ActorReadLogRecordsJSON().deletelogs(
                uuids,
                logrecords: logrecords,
                profile: rsyncUIdata.profile,
                validhiddenIDs: validhiddenIDs
            )
            let records = await updatedRecords
            
            print("(2) awaited updatedRecords deletelogs from (1))")
            print("(3) start async let updatedRecords updatelogsbyhiddenID)")
            
            async let updatedLogs: [Log]? = ActorReadLogRecordsJSON().updatelogsbyhiddenID(records, hiddenID)
            logrecords = records
            logs = await (updatedLogs ?? [])
            
            print("(4) awaited updatedLogs from (3)")

            WriteLogRecordsJSON(rsyncUIdata.profile, records)
            selectedloguuids.removeAll()
        }
    }

The debug windows in Xcode display the following:

The actors also print whether they execute on the main thread. The first async let statement initiates execution, and the subsequent await statement for the result above (2) suspends the function’s execution until the asynchronous result is computed. The await statement is crucial for suspending the execution of the function until the asynchronous result is available. And then the next (3) and (4).

(1) start async let updatedRecords deletelogs
ActorReadLogRecordsJSON: deletelogs() NOT on main thread, currently on <NSThread: 0xa49e3c200>{number = 18}
ActorReadLogRecordsJSON: DEINIT
(2) awaited updatedRecords deletelogs from (1))
(3) start async let updatedRecords updatelogsbyhiddenID)
ActorReadLogRecordsJSON: updatelogsbyhiddenID() NOT on main thread, currently on <NSThread: 0xa49e3c280>{number = 17}
ActorReadLogRecordsJSON: DEINIT
(4) awaited updatedLogs from (3)
WriteLogRecordsJSON: writeJSONToPersistentStore file:///Users/thomas/.rsyncosx/VPxxxxxxxx/WDBackup/logrecords.json
WriteLogRecordsJSON DEINIT
ActorReadLogRecordsJSON: updatelogsbyhiddenID() NOT on main thread, currently on <NSThread: 0xa4bb72800>{number = 19}
ActorReadLogRecordsJSON: DEINIT

Tagging of data

Overview

Tagging is computed in the ParseRsyncOutput Swift package bundled with RsyncUI.

Example:

• Input string: Number of created files: 7,191 (reg: 6,846, dir: 345)
• Converted to: [7191, 6846, 345] (the thousands separator is also removed from the string before parsing)

The function below extracts only numbers from the input:

public func returnIntNumber(_ input: String) -> [Int] {
    var numbers: [Int] = []
    let str = input.replacingOccurrences(of: ",", with: "")
    let stringArray = str.components(separatedBy: CharacterSet.decimalDigits.inverted)
                         .compactMap { $0.isEmpty == true ? nil : $0 }
    for item in stringArray where item.isEmpty == false {
        if let number = Int(item) {
            numbers.append(number)
        }
    }
    if numbers.count == 0 {
        return [0]
    } else {
        return numbers
    }
}

Parsing the rsync output is straightforward, but the formats differ between the latest rsync release and the default macOS version.

Version 3.4.x

The trailing output from the latest version of rsync (version 3.4.1) looks like:

....
Number of files: 7,192 (reg: 6,846, dir: 346)
Number of created files: 7,191 (reg: 6,846, dir: 345)
Number of deleted files: 0
Number of regular files transferred: 6,846
Total file size: 24,788,299 bytes
Total transferred file size: 24,788,299 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.003 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 394,304
Total bytes received: 22,226
sent 394,304 bytes  received 22,226 bytes  833,060.00 bytes/sec
total size is 24,788,299  speedup is 59.51 (DRY RUN)

Openrsync

The trailing output from the default version of rsync looks like:

....
Number of files: 7192
Number of files transferred: 6846
Total file size: 24788299 bytes
Total transferred file size: 24788299 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 336861
File list generation time: 0.052 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 380178
Total bytes received: 43172
sent 380178 bytes  received 43172 bytes  169340.00 bytes/sec
total size is 24788299  speedup is 58.55

How the Tagging Works

The output from rsync is parsed and numbers are extracted. After parsing, these numbers determine whether data should be tagged for synchronization.

Latest Version of rsync

Three numbers decide whether data needs synchronization: updates (regular files transferred), new files, and deleted files. Each can be zero or positive, and all three are checked.

Default Versions

Only one number decides: the updates (files transferred).