Queue processing

As already said ftpcluster makes extensive use of a queue to increase the cluster's fault tolerance. The queue is processed by the programs

store

moves uploaded files from the cluster server to one of your nodes.

replicate

replicates files between cluster nodes increasing their availability in case one of your nodes isn't operative.

remove

removes files from your nodes that are already deleted on the cluster server.

queue

lists, en- or dequeues records from the queue.

1. Queue processing is not done automatically.

You must start the cluster processor if you want to process the queue or parts of it. The server submits jobs to the queue (for file storage and deletition) but it does not trigger it's processing.

This may sound like an item for the todo list but it's not. Let's look at the upload queue. server submits files to this queue as they are uploaded and later moved to a cluster node by store. But store doesn't have any idea about the storage strategy you want. Do you want some kind of load balancing (read: balance used space) perhaps with some omission? I don't know and I don't care. Instead I leave this up to you.

Well, then how is the upload queue intended to work? Before you start your store process you can inspect the contents of the current upload queue, calculate prefered cluster nodes on whatever strategy you want and resubmit the files to the queue. You can use the store-queue for the resubmitted files to mark these jobs as already processed from the upload queue.

The same idea is true for replicate and it's queue. You create queue jobs according to your needs which are then processed by replicate.

2. There can be only one queue processor at any given time.

The queue processors maintain the queue lock file .queue.lock. If a particular queue processor can set this lock file it terminates immediatly without doing anything.

This means simply that what ever you do the cluster's queue is safe from processed by two (or more) concurrent queue processors which would introduce various race conditions.

The lock file is usually automatically removed when the program that created it terminates. If however the program terminates unexpected (the queue processors are gawk scripts and terminate unexpected on a very low level) you might come into a situation where you have the .queue.lock but no running queue processor. In this case remove the lock file and continue.

3. Entries that can't be processed are resubmitted.

If one of the queue processors can't execute a job (because the addressed node isn't operative or no node server is available) the entry is resubmitted to the queue.

Additionally queue file (created by one of the cluster programs) show date and time of their creation in their names. Furthermore the exist as long as there is at least one pending queue job in it. This way you can easyly monitor who old a particular queue job is.