One of the requirements on a current project is to handle an FTP feed of XML data. Yes, for a protocol invented in the 70s, FTP is still a popular way of transporting data!
In processing an FTP feed, there’s always the question of how to check for files (usually with cron) and when to know that the FTP transfer is complete (using FTP rename is one approach to this).
Today, I learned about a Linux feature that makes it easier to process FTP feeds. In later versions of the Linux kernel (2.6.13 or later), a file change notification system named inotify is available. People have written a number of utilities to take advantage of inotify. One of these is the inotify-tools package.
The inotify-tools package includes a library and two command line programs, inotifywait and inotifywatch.
The inotifywait utility looked like it would work for me, so I downloaded the inotify-tools source, compiled it, and installed in on my Linux system.
I then wrote the following shell script to monitor for new files within the feed directory, and execute a program to process the file.
#!/bin/sh
FEED_DIR="/var/feeds/news/incoming"
while file=$(inotifywait -q -e close_write "$FEED_DIR" --format "%f"); do
exec /path/to/myapp "$FEED_DIR/$file"
done
This script quietly (-q) waits for the “close_write” event (-e) from the inotifywait call and outputs the name of the file (–format “%f”). In my testing, the “close_write” event is triggered when the FTP transfer is completed. For feeds that use FTP rename, the event would be ‘create’ instead of ‘close_write’. See the man page for inotifywait for more information about these and other events.
This approach to processing incoming FTP files has the benefit of not requiring cron to periodically check for new files. It also solves the problem of knowing when the FTP transfer is complete.