The newest version of FileZoomer Adds support for several new Amazon Web Services S3 capabilities, including:

Object Life Cycle: specify that files be deleted or moved to low-cost AWS Glacier storage after a set number of days or after a certain date.

Versioning: Turned on at the bucket level, versioning means that even if you upload multiple updates to a file all previous versions are saved. The newest version shows up as usual, but if you right-click the file and “show versions” all the prior versions will be displayed, and they can then be downloaded.

It’s important to know that with the current version of S3 these two features — Object Life Cycle and Versioning — are mutually exclusive. If you turn on Versioning you can’t also use Life Cycle rules, and if you are using Life Cycle you can’t turn on Versioning.

The new version of FileZoomer also includes a Batch Processing option. After interactively defining a batch process using “File…Batch Configuration”, you can later initiate that process in FileZoomer with “File…Run Batch”. This makes it easy, for instance, to update a folder and its contents with all new and updated files since the last time the batch upload was run.

Using a pure batch processing version of the filezoomer java “jar” file, along with a configuration file you have created interactively, you can also do things like schedule an unattended run of an upload job.

For more details on these new features see the individual posts for Object Life Cycle, Versioning, and Batch Processing..

AWS recently announced Object life Cycle support for S3, which enables automated actions to be performed on objects based on a date or based on a time interval having elapsed. There are currently two actions available:

• Delete the objects (files)
• Move the object to low-cost Amazon Glacier storage

To automatically delete files specify a number of days or a specific date and the objects will be deleted after the specified time.

The migrate files to Glacier option moves the object to the AWS Glacier system for archiving using cheaper storage. Files are moved to Glacier after the specified number of days have elapsed or a specific date is reached.

Object life cycle support is based on the object prefix, better known as the folder path.

A bucket may contain folder paths with different rules. For example your BACKUP bucket might contain a sqldb/backup folder path and a sqldb/logs path. The sqldb/backup path could have a rule to move the objects to Glacier after 10 days and delete the objects after 365 days. The sqldb/logs folder may have a rule to simply delete the logs after 60 days.

There are three key things to know about the Object Life Cycle Migrate to Glacier option:

• It’s cheaper. As of this writing, standard S3 pricing starts at $0.095 per GB. Glacier costs $0.010 per GB, nearly an order of magnitude cheaper.
• It takes time to get the files back to S3, about three to five hours after you “initiate recovery”.
• Object Life Cycle (both Glacier migration and automatic delete) is currently incompatible with S3 Versioning. You can’t do both.

Other things to be aware of:

When you tell S3 to migrate files to Glacier the only way to access them is through S3. There is no vault support and you cannot use other Glacier tools to access your files. You don’t sign up for Glacier and you don’t get a separate bill for Glacier. S3 just takes care of it and your S3 bill goes down.

When displaying files in S3 that have been moved to Glacier there will be an indication that the file is in Glacier. To access a file in Glacier you must use an S3 request to recover the file(s) . In FileZoomer you right-click and “Initiate Recovery”. File recovery takes 3-5 hours. Once a file is recovered it is available for a limited time. In FileZoomer you specify the number of days to keep it available. To make the recovery permanent you must copy the recovered file to another file in the S3 bucket. If you don’t the recovered file will be moved back to Glacier.

Configuring Object Life Cycle

• Navigate to the bucket and folder path for which you want to create a life cycle rule.
• Select the file drop down menu and then select the Object Life Cycle menu option.

The display will show the current bucket and the current folder path in the bucket. It will also show the life cycle rules for the current bucket

Save Rules saves the displayed rules for the current bucket. If any rules are added or deleted you must use the Save Rules button to finalize the changes.

To delete a rule select the rule and Delete Selected Rule. When you confirm with Ok the rule will be removed from the rule display but it has not been removed from the bucket. Use Save Rules to apply the displayed rules to the bucket.

Use Add Rule to create a new rule.

The Add Rule dialog will appear. Verify the bucket and path displayed are the bucket and path to which you want to apply the rule.

Name the rule using Rule ID. Use a meaningful description.

Check “Use Transition Rule” to migrate files to Glacier. Check “Use Expiration Rule” to automatically delete files. You can do both. For instance you could migrate files to Glacier after 90 days and then have them deleted automatically after 365 days.

Set the number of days after file upload to apply the transition and/or expiration rules.

You can also specify a date in the future when the migration or deletion will occur. The date must be in the format yyyy-mm-dd.

Add Rule To List saves the new rule..

A major new feature of FileZoomer Beta 0.9 is “Batch Processing,” designed to facilitate and automate repetitive S3 transfer tasks.

We needed to upload web site backups and logs to S3 from Linux systems, and the only access from Linux to S3, at the time, was with a Perl module which did not handle large files well. S3 access code from FileZoomer was reused to create a command line java jar file that could run using the scheduling system provided by the operating system. Being java-based, it could be used on Windows and OS X systems as well.

Manually creating the configuration file was soon replaced by an interactive GUI component in FileZoomer itself. After that it was a small step to also add a “Run Batch” command to FileZoomer so that common, repetitive tasks could be easily initiated from the application itself, when that is easier or more useful than using a scheduler to run an actual batch job. The ability to do interactive configuration and easily initiate regular “housekeeping” tasks (like backing up new and changed files in a Documents folder) soon made it popular around the office for Windows and Mac use.

Now it’s available to FileZoomer users. It’s a powerful, but somewhat complicated tool, so if you use it take some time to understand it.

We’ll describe here how to get started using Batch Processing, using as our example the most obvious and common reason to use it — uploading and updating a folder to S3. And remember that it works with all the other new S3 features, most notably Object Life Cycle migration to Glacier for low-cost file archiving.

To start, choose “Batch Configuration” from the File menu:

Then Highlight “Add New Configuration File” and click “OK”. Later after you’ve created one or more configurations, you will see them listed for selection.

Next, specify all the important details about  your configuration:

Notice the current bucket and S3 path is filled in for you. Browse to select where you want log files to be saved. Then “Add New Action” to specify what your batch configuration will accomplish:

This example shows an Upload to S3 of all files modified (or created) since the last time the job ran. This means the first time it runs all the files in the selected path will be uploaded. We’ve chosen “Server Side Encryption” so the files are “encrypted at rest” at AWS. We want to process subfolders. We’re not going to compress on upload, which would cause files to be stored in zip format (if you are concerned about saving space and related charges, consider setting up a migration to Glacier). Uncompress on Download is checked so that IF you used FileZoomer compression the application would uncompress files on a later download.

Browse for the local path of the folder and files to be processed, and click “OK.”

Be sure to give the configuration a descriptive name, notice your new configuration is now listed, and BE SURE TO “SAVE CONFIG.”

Now you are ready to “Run Batch” from the File menu:

Select the desired configuration (there could be more than one) and click “OK”:

FileZoomer will tell you it’s “Running Batch Configuration” but if you need details on what’s transpiring you need to look at the log file.

There are more options available for Batch Processing.

In addition to the “Run Batch” command you can also run scheduled, or from a command line..

Previous posts on the new FileZoomer Batch Option have shown how to interactively create a batch configuration file, and start a batch process from within FileZoomer.

You can also use the true batch version of the java jar file and run it from a command line, batch file, or terminal session on any operating system that supports command line or shell commands or terminal commands (and of course they all do). This makes it easy to create a configuration file (do it interactively) but to run the process unattended, including on a server or other machine different from the one used to create the configuration file.

When you save a configuration file you will notice that it tells you where that config file  was saved, and the name:

You can also check the box to put the path in the clipboard. You can also get this info anytime by clicking “Show Config File” on the Batch Processing Configuration page:

Next, get a copy of FileZoomerBatch.jar and its associated lib folder (the link is to a zip file, unzip and and put it where you want to execute it from) Here’s the zip file’s MD5 if you want to confirm it..

Then use the appropriate technique for your OS to execute FileZoomerBatch.jar, remembering to provide the location of the config file. For instance, a Windows .BAT file located in the same directory as FileZoomerBatch.jar and the config file (mcfg) would have a command that looked like this:

java -jar FileZoomerBatch.jar “xxxxxx.mcfg”

(where xxxxxx was the name you gave your config file).

If you are executing on a linux or Mac OS X system use appropriate method. If you aren’t comfortable with that consider executing the batch process from within the FileZoomer App with “Run Batch”.

 .

FileZoomer now supports Amazon S3 Versioning, which keeps track of all versions of files in a bucket, and allows you to display and retrieve older versions of updated files. There are two important things to know about this S3 feature:

• Versioning is set for the whole bucket.
• Once set you can’t turn Versioning back off for that bucket and its files. Versioning can, however, be suspended.
• Versioning is incompatible with the Object Life Cycle feature (automatic deletion or migration to Glacier).

So plan ahead before enabling Versioning for a bucket.

To turn on Versioning navigate to the bucket and use “File…Bucket Versioning”

Check “Enable Versioning” and click OK.

To access earlier versions of a particular file, right-click the file and “Show Versions”

The version list will initially display the current version at the top, along with all previous versions. Right-click to Download or Delete a version of the file.

 .

The newest version of FileZoomer includes an option to upload only those files within a directory that have been updated after a specific date and time.  This allows you to keep a complete version of a local folder in your S3 account without the need to constantly re-upload everything.

As an example – let’s say I want to upload the “Documents” folder from my Mac.  I’d open and log into File Zoomer, click the upload button, navigate to the “Documents” folder and select it to upload the entire folder (with sub-folders) into FileZoomer.  Because my documents folder is pretty large it’s going to take awhile to complete.

However, the next time I want to update the contents of my Documents folder I’ll just log into FZ, and “right click” the folder containing my documents folder.  That brings up the following menu:

Right clicking the Folder Name Brings up this menu

If I select the “Upload by Date” option I’ll see the following:

Upload by date dialog box

The date of the last upload to this folder is displayed so you can either use it or enter your own date.  Clicking OK will then launch an upload of every file that’s been updated after the date displayed in the dialog.

Check it out and let us know how you like it in the comments..

The latest release of FileZoomer ads support for the most recently added Amazon S3 Regions: US West (Oregon) and South America (Sao Paulo).

Because FileZoomer is a Java Webstart Application there is nothing you need to do to update your software. The next time you run FileZoomer it will automatically update to the most current version.

.

FileZoomer beta .6 has been released and it can now generate a report that shows the total amount of storage you’re using in an Amazon S3 account across all of your S3 Buckets.  We’re also providing a high level estimate of the monthly cost of that storage from Amazon.  The estimate uses US standard pricing and is based on standard storage costs so it may not be accurate for your particular location but it should provide a useful estimate.

To generate the report from within FileZoomer just click the menu item for view and then the usage report.

Click the menu item for view, then click "usage report"

The resulting report will look a bit like this, but hopefully you are making better use of your S3 account than we are with this test account.

S3 Storage Report

As you can see this test account has only a few files and folders and contains only 141,692,710 Kb of storage, or less than 140 MBs.

Let us know in the comments if there are any questions on how to use the report, or if you have any suggestions on improving the it..

The user community has spoken and file compression, the feature you’ve requested most has been added to FileZoomer.

The benefits of compressing files are probably obvious but we’ll list them anyway.  Compressing files reduces bandwidth costs, reduces transfer times and reduces storage costs.  Now  FileZoomer is not only considerably faster but using it to manage your S3 accounts will actually save you money on your monthly Amazon S3 bills.

File compression is an account option that can be turned on and off as you wish.  The default setting for file compression is off so you’ll need to turn it on to see it work.  To turn on file compression just launch File Zoomer and log in to your S3 account, then click the “preferences” menu item.   Click “Zip Compressible Files” and once the checkmark is there all compressible files will be compressed during uploads, downloads and while stored.  You’ll see a new .zip extension added to each compressed file so that you can easily identify compressed files.

Downloading files that were compressed during upload with FileZoomer will automatically uncompress them.  If you choose to use another utility to download your files you’ll be able to unzip them using standard zip tools.

One of the things we use S3 for is to store log files, which are highly compressible.  In our testing of daily log file uploads we’re seeing transfer speed improvements of nearly 80% with a corresponding reduction in space used to store the files.

File types that are inherently already compressed, for instance jpeg, mp3, zip, and most video files, will not get compressed as they are not “compressible”.

Please let us know what you think of this feature and what other features you’d like to see added.

 .

This is our second post in a series on how to decipher the byzantine Amazon S3 pricing Schedule.  In our last post we covered S3 pricing for storage, and how it was calculated.  This post will delve into the bandwidth calculations for Amazon S3 storage in hopes that you can avoid that surprise bill that we read about from time to time.

The bandwidth charges for S3 are fairly straightforward but we need to get our terms straight before getting into details about the pricing.  First there is bandwidth in (into Amazon S3) and bandwidth out (data transferred out of Amazon S3).

Bandwidth in represents the bandwidth you use when transferring files into an S3 account.  I would call this uploading files into your account.  This is extremely straightforward.

Bandwidth out represents the bandwidth used whenever anyone transfers a file from, or out of, your S3 account.

So – In simple terms if you are just using S3 to hold your own files your bandwidth in and bandwidth out will be fairly easy to monitor.  But, and this is what gets people, if you’re using Amazon to share files with others, each time a file is downloaded you’re going to be charged for the bandwidth.

Here is the actual pricing and then we’ll work through an example:

Data Transfer In

• All Data transfer in is now free priced at .10 (ten cents) per GB

Data  Transfer Out

• First 1 GB / Monthly is free
• Up to to 10 TB /Monthly  $.12 15 (fifteen twelve cents) per GB
• Next 40 TB / Monthly $.09.11 (eleven nine cents) per GB
• Next 100 TB/ Monthly $.07 09 (nine seven cents) per GB
• Over 350 150 TB/ Monthly $..0508 (eight five cents) per GB

Now let’s move onto an example.  Let’s say you’re a creator of animated videos and you decide to upload a 700 MB movie into your account to share.  You’ve done a great job and you want as many people to see it as possible.

The cost to upload that particular file would be only around free seven cents.  So far, no problem.  Let’s say you email 10 of your friends about the video and they all download and watch it.  The cost to you is again no problem, only around a dollar.  But let’s say that one of your friends tweets the link and it goes viral.  Hey, that’s great right?  You want as many people as possible to see your work.  But, keep in mind that each person that sees it is costing you around  11 8.4 cents.  So, if 1,000 people watch your video your bill will be $84.   Now let’s say that 10,000 people see your video suddenly your cost is over $1,000 $800 for the month.

The moral of the story is that if you’re going to share large files on Amazon S3 you need to keep track of your bandwidth used and be prepared to remove the file sharing on the file if you exceed a certain threshold.

You can check your bandwidth used by logging into your S3 account, and clicking “usage reports.”  I download my data into a CSV file, then sort by usage type and summarize the bandwidth out data to get this information.

 .