Knowledgebase: How Stuff Works
PresSTORE Synchronize - Principles of Operation
Posted by Ibrahim Tannir on 17 April 2012 11:02

For Synchronize, PresSTORE does not keep a permanent index. An index of the folders and files of the source is built on the fly. This index is sent over to the client where the target for the replication resides. The client checks the target folders and files against the index and returns a list of the differences back to the server.  The change is determined based upon the file's attributes, but not upon its contents. The attributes include: modification time - mtime, file inode change - ctime,  size, Posix permissions, owner, group,  extended attributes, ACLs and windows backup flag. Since the contents is not considered, there is consequently no checksum done on the file.

All operations on the target are done directory-wise, including deleting the files that no longer exist  on the source, as well as removing old cycles and old versions of files. Consequently, it can happen that if the file system on the source changes in such a way that large amounts of data are removed in one folder and added in another, this can lead to a shortage of space on the target, although theoretically the target is large enough to fit all the data of the source. However, this only happens in rare occasions when the target file system is close to full and it also depends on which of the two folders is visited first.

PresSTORE can also be configured to receive notifications from the file system about any changes that occur on the source. This eliminates the need to run a scan and build an index of the source. For this purpose, PresSTORE collects and keeps a temporary list of changes that occur in between two runs of a synchronize plan. However, if PresSTORE determines that it has missed receiving some notifications or that it is uncertain about the state of the file system, it will fall back to a complete scan of the file system until it re-establishes a deterministic state. This is the case also every time PresSTORE is restarted.

All the Synchronization plans are kept in the configuration database which is found in ~/conf/customerconfig.
The temporary indexes and files containing notifications are kept in ~/temp.

When copying the files, PresSTORE will copy all it's parts including Windows streams, Helios, Xinet and NetAtalk resource forks and the extended attributes and ACLs when possible. If the file systems are not compatible, the parts that cannot be copied are left out or converted into a compatible format, as in the case of Apple Double files. ACLs are not copied accross different file and operating systems, ownership and Posix permissions are kept as far as possible.

One can choose to encrypt the data during transfer.

(4 vote(s))
Helpful
Not helpful

Comments (1)
Marcel Hochuli
07 June 2012 16:46
Good description, thanks. I have a question and a suggestion:
Q: Which filesystems have Ā«notifications about any changesĀ» that PresSTORE supports and are they usable when source and target are different file systems?
S: For the case that the volumes are almost full and syncing fails because of target full: Could PresSTORE switch to a special state that it syncs in two passes. Fist pass just deletes what has to be deleted, second pass tries to copy what has to be copied.
Post a new comment
 
 
Full Name:
Email:
Comments:
CAPTCHA Verification 
 
Please enter the text you see in the image into the textbox below (we use this to prevent automated submissions).