The cluster queues

The reason for queuing is simple. Consider you want to delete a file on you server. The file is stored on two nodes, one of them is up and running but the second is down. If the cluster server would try to delete the files on you nodes in real time it would come back with either leaving the second copy on the yet unavailable node with submitting the deletition to a queue for later processing when the node is up again.

Obviously the second variant is preferable since it can deal with broken nodes and in an environment where we have a lot of node server we have to accept that a node might be out of service. So we end up with a queue anyway.

The cluster extends the idea of the queue in so far that the server doesn't even attempt to delete files on the nodes on real time. It submits the whole job immediatly to the queue.

Queue formats

First of all the queue is made up from the files in the .queue directory. Well, not all files. Queuefiles must match a naming convention to be recognized, see below.

Second, the queue stores entries for file uploads, file storage, replication and deletition. Each job description has a slightly different format. The common thing among these different sub-queues is that (1) each queue job is one line in a queue file and (2) queue job's type is determined by a keyword.

• The upload queue
  Upload jobs are created by server whenever a file is uploaded via FTP. The queue format is

  upload filename uid size mtime [nodelist]
  

  where

  • upload is the magic keyword which marks this job as a file upload,

  • filename is the uploaded file's name relative to the cluster base directory -- notice that it's not the info/link file but the real data file,

  • uid is the user id of the uploading user and

  • size and mtime are the file's size in bytes and modification time in seconds since 1970,

  • nodelist is an optional white-space separated list of nodes where the file should be stored.

  While the purpose for the upload keyword and the filename parameter should be fully clear you might ask why the upload queue holds the other parameters. uid and size might be used to direct uploads to a particular cluster node based on the owning user and/or it's file size.

  Furthermore the size and mtime parameters are used to verify if this queue job is still valid. Consider the case where the same file is uploaded twice during one FTP session. This gives you two upload queue entries with only the second (or more general the last) being valid. When store reads the upload queue later it uses the entry's size and mtime to verify if the entry is still valid or not. If not it's ignored and dropped and processed otherwise. This scheme works as long as a file isn't uploaded twice with the same timestamp, that is within the same second.

  Files are usually stored on a simple round-robin scheme. store chooses a different node server for each upload starting again with the first node server if it has reached the end of the list. With the optional nodelist argument store will put the file on one of the given node servers and consider the job as done. You may also prefix one or more node servers with a minus to tell store to not consider these nodes for the upload and you may also mix a node list with an exlusion list although this doesn't make really sense.

• The store queue
  The second queue that is processed by the store program is the store queue. The format of this queue is almost identical to the upload queue.

  store filename size mtime [nodelist]
  

  As you can see it's only the uid field that's missing. The other parameters are the same as for upload with the same meaning.

• Replication queue entries
  relicate queue entries look like storage entries:

  replicate info/link-file size mtime [nodelist]
  

  but this time the second parameter does not refer to the real file but the file's info/link file. This is because replicate can't work with files that are stored on the cluster server itself, it expects the files to be already on one of your nodes.

• File deletition
  The delete queue's is different compared to the queues above:

  remove node filename size
  

  The entry's explanation is:

  • Again, remove is the queue's keyword, delete can be also used.

  • node is the cluster's node where to delete filename. Notice that filename is the file's name on node, it's in no way related to any file on the cluster server.

  • size is simple filename's size in bytes. This value isn't used yet.

  Notice that the remove program (which processes this queue) simply delete files on nodes. It does not modify the cluster info/link files after a succesful deletition. It expects this to be already done.

The DELE queue

In addition to the queues above there is also another "helper" queue. If a file is deleted via FTP it's info/link file is only moved into the .deleted directory. A later run of the remove program deletes the files from the cluster node or, if this is not possible, resubmits the files back into the queue using the .queue directory this time.

Notice that the DELE queue might be discontinued in a future release by submitting deleted files directly to the remove queue.

Queue filenames

Ok, now we know about the different queue types and their formats. But what are queue files and where are they stored? The answer is simple: all files in the .queue directory with the .q extension are considered as queue files containing entries for one or more queue types of the above.

But furthermore (due to the way queueing was implemented) each of the queue processors store, replicate and remove accepts files that meet another naming convention.

store

reads files beginning with store- or uploads-.

replicate

reads files beginning with either replicate- or replicator-.

remove

reads file starting with remove-.