Although listserver is easy to use and it's output is simple to read it's also worth reading this text just to know why it's not working for your particular setup.

Listing your node servers

ftpcluster comes already with the list command. This command lists the files in your cluster. But list does it from the cluster's point of view, based on the infolink files on the cluster server. In other words: list shows you the cluster content as it should be which is not neccessarily the way it is. The only one that can tell you which files you have really on a node server is the node server itself. This is done by the listserver program.

listserver is a little bit different than the other cluster programs. They work with almost no assumption about a node server, they should work with any FTP server software on the nodes. listserver assumes that

  1. the files are stored in two-level directory structure,
  2. directory names are made of exactly two digits, and
  3. the node's FTP server supports the MDTM and SIZE commands.

The first two prerequisites are defined and already in use by the other cluster programs so it's nothing really new.

Compatibility problems

The third one needs some discussion. MDTM returns a file's modification time and SIZE returns it's size. Both commands are obviously neccessary to list a node's content but they are also not defined in RFC 959. As far as I know they are only part of a current FTP RFC draft but on the other hand these command are fairly common for today's FTP servers. However notice that listserver may not work with every FTP server around.

You might ask now why listserver doesn not use the FTP LIST command to retrieve the file information? Although the LIST command if defined in RFC 959 it's output format is not. A server may return directory listings in any form it likes and this is difficult to parse. I know that there are clever FTP clients that know almost all server formats and how to parse them. I could have implemented listserver the same way (and perhaps I will have to) but I don't like this approach. I see the real solution in the FTP RFC draft I mentioned already above. Part of this draft are the MLSD and MLST commands that return directory and/or file information in a clear and well defined machine readable format.

More listserver problems

So far we have a listserver program that may not work with every FTP server software. Perhaps especially with that FTP server you have already in use. But things can still get more worse.

As I said above the FTP's LIST command output is not defined in RFC 959, a server might choose a format of his choice. But there is also NLST. This command returns only filenames, one per line, and nothing else. Not difficult to parse. Having the filenames listserver can send SIZE and MDTM for each file and collect the information is wants.

But how can listserver find out which directories have already been created on a node server (they are created when needed). listserver is straight forward here: it uses NLST also for this. And here is the problem. Unfortunatly RFC 959 says that on NLST "the server will return a stream of names of files ..." and "names of files" is important here. Obviously some server implementors concentrated more on this than on the preceeding "This command [NLST] causes a directory listing to be sent from server to user site.". Again: directory listing. As I understand it this includes names of sub directories. But all this doesn't help, there are FTP servers that really only return filenames omiting directory names, wu-ftpd is one of them.

The workaround is as follows. If a node server does return no matching directory names (two digit assumption from above) listserver queries the server's system identification. If it finds the magic string "unix" in the system type listserver uses the FTP LIST command to retrieve directory names assuming that the file or directory names appear in the last column of LIST's output. There is yet no workaround for windows server, just because I don't know how the average windows server returns it's directory listings, any help is appreciated.

Running listserver

listserver is simple to use: 

# su - cluster
$ listserver
...output appears here...

lists the node server, file's path size and modification time for each file listserver finds on all cluster nodes. You can add one or mode node names on listserver's command line if your are only interested in particular nodes. That's basically all.

If you want to debug listserver you can specify the -d option to see listserver's FTP conversations. Specifying -d a second time prints also the mpec dialog. Use this only if you suspect the mpec coding to be wrong. Usually you don't have to turn mpec debugging on.

Due to the way how listserver works (sending MDTM and SIZE for each file) running listserver can take some minutes or even longer. This depends on the number of files you have alreay stored and the type of FTP server software on your nodes. It could then happen that while listserver is running a queue runner (e.g. store or replicate) modifies parts of the cluster after listserver inspected them giving you then a wrong picture of your nodes. For this reason listsserver supports the -l option for queue locking. If -l is specified listsserver tries first to set the cluster's queue lock (and removes it later automatically) to prevent the queue workers from running in parallel.