A major new feature of FileZoomer Beta 0.9 is “Batch Processing,” designed to facilitate and automate repetitive S3 transfer tasks.

We needed to upload web site backups and logs to S3 from Linux systems, and the only access from Linux to S3, at the time, was with a Perl module which did not handle large files well. S3 access code from FileZoomer was reused to create a command line java jar file that could run using the scheduling system provided by the operating system. Being java-based, it could be used on Windows and OS X systems as well.

Manually creating the configuration file was soon replaced by an interactive GUI component in FileZoomer itself. After that it was a small step to also add a “Run Batch” command to FileZoomer so that common, repetitive tasks could be easily initiated from the application itself, when that is easier or more useful than using a scheduler to run an actual batch job. The ability to do interactive configuration and easily initiate regular “housekeeping” tasks (like backing up new and changed files in a Documents folder) soon made it popular around the office for Windows and Mac use.

Now it’s available to FileZoomer users. It’s a powerful, but somewhat complicated tool, so if you use it take some time to understand it.

We’ll describe here how to get started using Batch Processing, using as our example the most obvious and common reason to use it — uploading and updating a folder to S3. And remember that it works with all the other new S3 features, most notably Object Life Cycle migration to Glacier for low-cost file archiving.

To start, choose “Batch Configuration” from the File menu:

Then Highlight “Add New Configuration File” and click “OK”. Later after you’ve created one or more configurations, you will see them listed for selection.

Next, specify all the important details about  your configuration:

Notice the current bucket and S3 path is filled in for you. Browse to select where you want log files to be saved. Then “Add New Action” to specify what your batch configuration will accomplish:

This example shows an Upload to S3 of all files modified (or created) since the last time the job ran. This means the first time it runs all the files in the selected path will be uploaded. We’ve chosen “Server Side Encryption” so the files are “encrypted at rest” at AWS. We want to process subfolders. We’re not going to compress on upload, which would cause files to be stored in zip format (if you are concerned about saving space and related charges, consider setting up a migration to Glacier). Uncompress on Download is checked so that IF you used FileZoomer compression the application would uncompress files on a later download.

Browse for the local path of the folder and files to be processed, and click “OK.”

Be sure to give the configuration a descriptive name, notice your new configuration is now listed, and BE SURE TO “SAVE CONFIG.”

Now you are ready to “Run Batch” from the File menu:

Select the desired configuration (there could be more than one) and click “OK”:

FileZoomer will tell you it’s “Running Batch Configuration” but if you need details on what’s transpiring you need to look at the log file.

There are more options available for Batch Processing.

In addition to the “Run Batch” command you can also run scheduled, or from a command line..

An earlier post introduced the new FileZoomer Batch Processing feature and walked through the configuration steps needed to prepare to do a “Run Batch” using a single, but very common, batch processing option. Here we go into more detail on all the available options and what they offer:

When you add a new configuration file (File…Batch Configuration…Add New Configuration File) and then “Add a New Action” you see this dialog that contains all the options available:

Let’s go through all the options starting with Select Action:

• Upload will upload files from the Local Path on the PC to the current S3 bucket and path.
• Download will download files from the current S3 bucket and path to the Local Path on the PC.
• Sync will synchronize the current S3 bucket and path and the Local Path on your PC.  Sync will not do subfolders. It will only do files in the specified folder itself. Files that exist only on S3 will be copied to the PC. Files that exist only on the PC will be copied to S3. Files that exist in both places will be compared, with the newer version replacing the older. That means if you delete a file on your PC that is also on S3 it will be copied back from S3. Likewise if you delete a file on S3 that is also on your Pc it will be copied back to S3. Sync will NOT resolve any update conflict issues. For instance, if an S3 file is updated from a source other than where FileZoomer Batch Processing is being used, and is the same file is also updated on the local PC then a FileZoomer Batch Sync will overwrite the older file. It will NOT merge changes to the file from multiple sources. Sync does “All Files” regardless of other settings.
• Prune is a specialized action that offers  a way to control the number of files in an S3 bucket and path. Unlike Object Life Cycle, which removes (via Delete or Glacier migration) files based on a number of days elapsed, Prune deletes files based on the “Prune File Count”, retaining  the most recent Prune-File-Count number of files in the current S3 bucket and path and deleting the rest. Prune should usually not be used on a bucket with Versioning enabled as the two features work at cross-purposes. Note that when doing a “Prune” action, other parameters are ignored as not relevant (Local Path, Process Subfolders, Encryption, Compression, All Files, Files Modified Today, Files Modified since last run). If you don’t have a clear idea of why you need Prune, it’s best not to use it.

 .

FileZoomer now supports Amazon S3 Versioning, which keeps track of all versions of files in a bucket, and allows you to display and retrieve older versions of updated files. There are two important things to know about this S3 feature:

• Versioning is set for the whole bucket.
• Once set you can’t turn Versioning back off for that bucket and its files. Versioning can, however, be suspended.
• Versioning is incompatible with the Object Life Cycle feature (automatic deletion or migration to Glacier).

So plan ahead before enabling Versioning for a bucket.

To turn on Versioning navigate to the bucket and use “File…Bucket Versioning”

Check “Enable Versioning” and click OK.

To access earlier versions of a particular file, right-click the file and “Show Versions”

The version list will initially display the current version at the top, along with all previous versions. Right-click to Download or Delete a version of the file.

 .

The newest version of FileZoomer includes an option to upload only those files within a directory that have been updated after a specific date and time.  This allows you to keep a complete version of a local folder in your S3 account without the need to constantly re-upload everything.

As an example – let’s say I want to upload the “Documents” folder from my Mac.  I’d open and log into File Zoomer, click the upload button, navigate to the “Documents” folder and select it to upload the entire folder (with sub-folders) into FileZoomer.  Because my documents folder is pretty large it’s going to take awhile to complete.

However, the next time I want to update the contents of my Documents folder I’ll just log into FZ, and “right click” the folder containing my documents folder.  That brings up the following menu:

Right clicking the Folder Name Brings up this menu

If I select the “Upload by Date” option I’ll see the following:

Upload by date dialog box

The date of the last upload to this folder is displayed so you can either use it or enter your own date.  Clicking OK will then launch an upload of every file that’s been updated after the date displayed in the dialog.

Check it out and let us know how you like it in the comments..

The user community has spoken and file compression, the feature you’ve requested most has been added to FileZoomer.

The benefits of compressing files are probably obvious but we’ll list them anyway.  Compressing files reduces bandwidth costs, reduces transfer times and reduces storage costs.  Now  FileZoomer is not only considerably faster but using it to manage your S3 accounts will actually save you money on your monthly Amazon S3 bills.

File compression is an account option that can be turned on and off as you wish.  The default setting for file compression is off so you’ll need to turn it on to see it work.  To turn on file compression just launch File Zoomer and log in to your S3 account, then click the “preferences” menu item.   Click “Zip Compressible Files” and once the checkmark is there all compressible files will be compressed during uploads, downloads and while stored.  You’ll see a new .zip extension added to each compressed file so that you can easily identify compressed files.

Downloading files that were compressed during upload with FileZoomer will automatically uncompress them.  If you choose to use another utility to download your files you’ll be able to unzip them using standard zip tools.

One of the things we use S3 for is to store log files, which are highly compressible.  In our testing of daily log file uploads we’re seeing transfer speed improvements of nearly 80% with a corresponding reduction in space used to store the files.

File types that are inherently already compressed, for instance jpeg, mp3, zip, and most video files, will not get compressed as they are not “compressible”.

Please let us know what you think of this feature and what other features you’d like to see added.

 .

To Search your the active bucket within your Amazon S3 account using FileZoomer just launch the product and log into your account.

Once inside the account you’ll see a see a search section at the top right of the navigation screen.  Simply enter your search argument and click “Go”.

You’ll see the results of your search in a window below.  From there you can act on the file, or if you want to go to the directory holding the file you can right click the file name and then click “Open Containing Folder..

Uploading files to your Amazon S3 account is both intuitive and easy using FileZoomer.  Just log into your account, navigate into the folder you want the files uploaded into and then click the “Upload” button.  Then navigate through your local file system, select the files and they’ll begin uploading.

Drag and drop is also supported, just drag the files into file structure of the folder you want the files uploaded into.

 .