An earlier post introduced the new FileZoomer Batch Processing feature and walked through the configuration steps needed to prepare to do a “Run Batch” using a single, but very common, batch processing option. Here we go into more detail on all the available options and what they offer:
When you add a new configuration file (File…Batch Configuration…Add New Configuration File) and then “Add a New Action” you see this dialog that contains all the options available:
Let’s go through all the options starting with Select Action:
- Upload will upload files from the Local Path on the PC to the current S3 bucket and path.
- Download will download files from the current S3 bucket and path to the Local Path on the PC.
- Sync will synchronize the current S3 bucket and path and the Local Path on your PC. Sync will not do subfolders. It will only do files in the specified folder itself. Files that exist only on S3 will be copied to the PC. Files that exist only on the PC will be copied to S3. Files that exist in both places will be compared, with the newer version replacing the older. That means if you delete a file on your PC that is also on S3 it will be copied back from S3. Likewise if you delete a file on S3 that is also on your Pc it will be copied back to S3. Sync will NOT resolve any update conflict issues. For instance, if an S3 file is updated from a source other than where FileZoomer Batch Processing is being used, and is the same file is also updated on the local PC then a FileZoomer Batch Sync will overwrite the older file. It will NOT merge changes to the file from multiple sources. Sync does “All Files” regardless of other settings.
- Prune is a specialized action that offers a way to control the number of files in an S3 bucket and path. Unlike Object Life Cycle, which removes (via Delete or Glacier migration) files based on a number of days elapsed, Prune deletes files based on the “Prune File Count”, retaining the most recent Prune-File-Count number of files in the current S3 bucket and path and deleting the rest. Prune should usually not be used on a bucket with Versioning enabled as the two features work at cross-purposes. Note that when doing a “Prune” action, other parameters are ignored as not relevant (Local Path, Process Subfolders, Encryption, Compression, All Files, Files Modified Today, Files Modified since last run). If you don’t have a clear idea of why you need Prune, it’s best not to use it.
Effect of All Files, Files Modified Today, and Files Modified Since Last Run:
- The “Default” is All Files, which means all files will be processed regardless date and time stamps. On a Download this could mean, for example, that older files on S3 replace updated files on the local PC. This is always used for “Sync“, and ignored for “Prune“.
- Files Modified Today uses the date on the PC for Upload. Not recommended for use with Download, and ignored for “Sync” and “Prune“.
- Files Modified Since Last Run works the same as All Files for the first run. After that it looks at the date and time stamp on the files and only does those files created or modified since the last run. IMPORTANT: on Windows and OS X if you move an older file to the local path the date is not changed, and if it is a file created or modified before the Last Run date it will NOT be uploaded using this option.
Effect of Server Side Encryption, Compress on Upload, Process Sub Folders and Uncompress on Download:
- Server Side Encryption, when checked, tells S3 to store the file encrypted (“encryption at rest”). Technically all this does is protect files in the event AWS discards or otherwise loses control of a drive containing your files without first destroying the files. Note that when files are being uploaded or downloaded using FileZoomer, SSL (encrypted transmission) is also used. Neither requires any key management on the user’s part.
- Process Sub Folders, when checked, includes files in subfolders (and will create a subfolder as needed) when doing an Upload or a Download. This option is ignored for Sync and Prune.
- Compress on Upload, if checked, zips files before uploading and marks the files as being “FileZoomer Zipped“. When paired with “Uncompress on Download” any file zipped by FileZoomer will be unzipped. Other S3 clients will simply see the files as regular zip files. Files already in a compressed format (e.g. jpeg, png, mp3, and actual zip files) will not be compressed. This option makes uploads complete more quickly and saves a bit on S3 cost.
- Uncompress on Download, which checked, will unzip any files marked on S3 as having been ”FileZoomer Zipped“. Otherwise it has no effect. It’s a good idea to leave this checked.
Local Path is required (except for “Prune”, and identifies the folder on the PC to include in processing.
Once your configuration is created, you can use the “Run Batch” option, or set it up to run from a command line, or on a schedule.
