Hammerspace and Data Lifecycle Management
There are many definitions of data lifecycle management and what stages are included. Hammerspace provides advanced capabilities for most or all stages including:
-
Ingest - Assimilation or directly creating data on Hammerspace.
-
Store and Manage - Different use cases will have different performance, protection including WORM functionality, and access requirements. Objectives and storage tiers in Hammerspace facilitate this stage.
-
Use and Share - Hammerspace supports multiprotocol including NFS v3, v4.1 and v4.2, all versions of SMB, and S3 so different applications and use cases can access data the way the application was written without having to refactor. You can ingest via one protocol and analyze via another. Global File System is a key feature used in combination with objectives to put the right data close to users at different locations, often before they need it.
-
Archive - Objectives combined with the use of less expensive tiers such as object or tape make it easy to move unused data to less expensive tiers including cloud and object.
-
Delete - While Hammerspace does not delete files as a function of the product, queries based on metadata, even going as far as creating a collection, can be used to determine data that has expired and can be deleted.
Note that Hammerspace Assimilation and Data Orchestration [Support article may require login to view.] and How to Configure Hammerspace Objectives [Support article may require login to view.] are covered in other support articles for reference. At the end of the data lifecycle, queries and collections can be used to determine sets of data that may be appropriate for deletion.
This section is broken down into the following categories:
-
Introduction to Hammerspace Collections
-
Using Collections for Data Lifecycle Management
Introduction to Hammerspace Collections
As discussed previously, collections are subsets of files determined by a metadata query that go in attributes evaluated when the file is created or written to and/or on each pass of the sweeper. The hidden collections directory is created in the root of every Hammerspace share and can be accessed by changing into the /<share>/.collections directory.
The following default Hammerspace collections are available:
# List all the default Hammerspace collections
$ ls .collections
all live open silent-access
assimilation-failed local-objectives permanent-data-loss snapshot
backup misaligned promoted threat
do-not-move not-selected replication-collision undelete
durable not-selected2 scan volatile
errored offline selected
links online selected2We won’t go into all of these in this document. Most of these are determined by predefined queries, but there are two that can be controlled by a Hammerscript expression determined by a user or administrator, selected and selected2, and their inverses, not-selected and not-selected2.
An expression that evaluates as true applied to the selected or selected2 attribute for a file will cause that file to appear in the selected or selected2 directory. Something as simple as this will work:
# Set selected to TRUE for files that match telemetry_apollo.aag*
$ hs attribute set selected -e 'TRUE' telemetry_apollo.aag*
##### telemetry_apollo.aaga
##### telemetry_apollo.aagb
##### telemetry_apollo.aagc
##### telemetry_apollo.aagd
. . .| Double quotes may be required if using the Windows command line. |
Now, files that match telemetry_apollo.aag* will appear in .collections/selected, within the same directory structure as they did within the share root:
# List contents of selected collection for the telemetry_dump3 directory
$ /.collections/selected/telemetry_dump3 # ls
telemetry_apollo.aaga telemetry_apollo.aagg telemetry_apollo.aagm telemetry_apollo.aags telemetry_apollo.aagy telemetry_apollo.aagb telemetry_apollo.aagh telemetry_apollo.aagn telemetry_apollo.aagt telemetry_apollo.aagz telemetry_apollo.aagc telemetry_apollo.aagiSetting an individual file with something as simple as "TRUE" works, but isn’t terribly useful. More likely, you would use an expression based on some metadata property that can be evaluated and changed over time, and set it recursively on a directory or share.
Expression - Tag with Value
The following example embeds an expression that evaluates as true if the file has a tag named Apollo with the value mission_log (EXPRESSION(get_tag("Apollo")=="mission_log")). The -r means the command will execute recursively, so with the current directory (".") as the target, this expression would be set on all files from the current directory down.
# Place files that have the Apollo tag with value mission_log in the selected collection
$ hs attribute set selected -r -e 'EXPRESSION(GET_TAG("Apollo")=="mission_log")' telemetry_dump2 .| If you set the expression on a directory, its contents will inherit it. While there may be scenarios where you want the attribute applied to each file individually, in most cases, it is enough to set it on the parent directories. |
To test this workflow, the user (or a script or tool) can set the Apollo tag to the value mission_log using the hs tag set command as shown in the following example:
# Apply the Apollo tag with value mission_log to the file telemetry.txt
$ hs tag set Apollo -s 'mission_log' telemetry.txtThe file telemetry.txt would now appear in the SELECTED collection directory.
To remove a tag from a file or directory, use the hs tag delete command as shown in the following example. It is not necessary to supply the value.
# Delete the tag Apollo from file telemetry.txt
$ hs tag delete Apollo telemetry.txtExpression - Tag with No Specified Value
Alternatively, you could have the expression match files that have just the tag Apollo, ignoring the value:
# Place files that have the Apollo tag with any value in the selected collection
$ hs attribute set selected -r -e 'EXPRESSION(has_tag("Apollo"))'Based on the previous expression, applying either of the following tags would cause the files to match and be placed in the SELECTED collection.
# Apply the Apollo tag with value mission_log to the file telemetry.txt
$ hs tag set Apollo -s 'mission_log' telemetry.txt
# Apply the Apollo tag with the default value to the file telemetry.txt
$ hs tag set Apollo telemetry.txtTo remove a tag from a file or directory, use the hs tag delete command as shown in the following example. Note that it is not necessary to supply the value.
# Delete the tag Apollo from file telemetry.txt
$ hs tag delete Apollo telemetry.txtDetermining Where Attributes Were Applied
To check if a file or directory has the attribute set with an expression that determines if a file will be added to the SELECTED collection, use the hs eval command. Note that the following examples use two similar commands, but one uses GET_ATTRIBUTE_LOCAL_UNBOUND and the other GET_ATTRIBUTE_INHERITED_UNBOUND.
If the LOCAL version returns #EMPTY, but the INHERITED returns the expression, you know that the file inherited the attribute from a directory above it.
# Show the applied expression that determines if a file is added to the SELECTED collection, if the attribute was inherited then #EMPTY will be returned
$ hs eval -e 'GET_ATTRIBUTE_LOCAL_UNBOUND("SELECTED")' telemetry.txt
#EMPTY
# Show the applied expression that determines if a file is added to the SELECTED collection, if the attribute was inherited then the expression will be returned
$ hs eval -e 'GET_ATTRIBUTE_INHERITED_UNBOUND("SELECTED")' telemetry.txt
EXPRESSION(HAS_TAG("Apollo"))To determine where the inherited attribute is set, run the LOCAL version of the command against the parent directories until you find where it is set. In the following example, we see that the file inherited the attribute from its parent telemetry_dump directory:
# Same command as the previous example, but run on the directory that contains the telemetry.txt file
$ hs eval -e 'GET_ATTRIBUTE_LOCAL_UNBOUND("SELECTED")' telemetry_dump2
EXPRESSION(HAS_TAG("Apollo"))Clearing File and Directory Attributes
To clear the selected and selected2 expressions, use the hs attribute set command:
# Set all attributes for the current directory and all contents to empty (removes them)
$ hs attribute set selected -r -e '#empty' .| Consult the Applying Queries using the Hammerspace Metadata Plugin for Windows section in Appendix A - General of this document for information about using that tool to apply query expressions. |
Using Collections for Data Lifecycle Management
In this section, we will look at how to use collections to perform a simple data lifecycle management task. We will identify unused files, place them in a collection, and delete them.
The following hs eval command would return a list of files with full path names (PATH) within the telemetry_dump directory that have not been accessed or modified in 7 years (LAST_USE_AGE>7YEAR):
# Return names and path for all files last used more than 7 years ago within the telemetry_dump directory
$ hs eval -r -e 'LAST_USE_AGE>7YEAR?PATH' telemetry_dump
"./telemetry_dump/file.aaaa"
"./telemetry_dump/file.aaab"
"./telemetry_dump/file.aaac"
"./telemetry_dump/file.aaad"
"./telemetry_dump/file.aaae"
"./telemetry_dump/file.aaaf"
"./telemetry_dump/file.aaag"
. . .You could add additional elements to the query to only include certain file types or other metadata. The directory location is already specified as the path in the command itself, but you could go further and specify a file name:
# Return names and path for all files matching the name *.ada* last used more than 7 years ago within the telemetry_dump directory
$ hs eval -r -e 'LAST_USE_AGE>7YEAR&&FNMATCH("*.ada*",NAME)?PATH' telemetry_dump
"./telemetry_dump/file.adaa"
"./telemetry_dump/file.adac"
"./telemetry_dump/file.adab"
"./telemetry_dump/file.adad"
"./telemetry_dump/file.adaf"
"./telemetry_dump/file.adae"
"./telemetry_dump/file.adag"
"./telemetry_dump/file.adai"
. . .The output could be piped to other OS-level commands to take some action, such as deleting the files as shown in the following example:
# Return names and path for all files matching the name *.ada* last used more than 7 YEARs ago within the telemetry_dump directory, then pipe the list to the rm command to delete the files
$ hs eval -r -e 'LAST_USE_AGE>7YEAR&&FNMATCH("*.ada*",NAME)?PATH' telemetry_dump | xargs rm
# Return names and PATH for all files matching the name *.ada* last used more than 7 years ago within the telemetry_dump directory, and verify that the files are no longer present
$ hs eval -r -e 'LAST_USE_AGE>7YEAR&&FNMATCH("*.ada*",NAME)?PATH' telemetry_dump| You should always test your HSTK commands before piping their output into any command that will alter or delete files, directories, or their metadata. |
Another safer approach is to add the files to an existing collection (selected), which (as long as the attribute remains set) dynamically updates the list of files or directories within the specified target (telemetry_dump) that match the expression (EXPRESSION(LAST_USE_AGE>7YEAR&&FNMATCH(".afa*",NAME))*). No action is taken on the files. They are merely added to a collection that will only include the files that match the expression you provided.
# Place files in the telemetry_dump directory matching *.afa* last used more than 7 years ago in the selected collection
$ hs attribute set selected -e 'EXPRESSION(LAST_USE_AGE>7YEAR&&FNMATCH("*.afa*",NAME))' telemetry_dumpThis sets an expression on the telemetry_dump directory that its contents will inherit. This expression will be reevaluated as metadata changes and time passes, and makes files appear in /<share>/.collections/selected/<path to file> as their last usage goes over 7 years. In other words, the selected collection would contain a dynamically generated tree of files more than 7 years old.
With this attribute in place, you could then simply delete all files in /<share>/.collections/selected as the last stage in the data lifecycle, though again, you should verify the collection includes the data that you intend.
The following example builds upon everything we reviewed in this section, starting from setting the attribute through the deletion of the files added to the collection:
# Place files located in the telemetry_dump2 directory that both match the name *.acw* and were last used less than 1 hour ago in the selected collection
$ hs attribute set selected -e 'EXPRESSION(LAST_USE_AGE<1hour&&FNMATCH("*.acw*",NAME))' telemetry_dump2
# List files in the telemetry_dump2 directory that match *.acw*
$ ls telemetry_dump2/*.acw*
telemetry_dump2/file.acwa
telemetry_dump2/file.acwh
telemetry_dump2/file.acwo
telemetry_dump2/file.acwu
# List current members of the selected collection for directory telemetry_dump2 (currently there are none)
$ ls .collections/selected/telemetry_dump2
# Add data to file.acwu to change the LAST_USE_AGE metadata value, meaning it will now match the LAST_USE_AGE<1hour condition
$ echo 'Add New Data' >> telemetry_dump2/file.acwu
# List current members of the selected collection for directory telemetry_dump2 (file.acwu that was edited is now added)
$ ls .collections/selected/telemetry_dump2
file.acwu
# Perform a recursive delete of the .collections/selected/telemetry_dump2 directory, note the Directory not empty message is expected
$ rm -rf .collections/selected/telemetry_dump2
rm: cannot remove 'telemetry_dump2': Directory not empty
# List current members of the selected collection for directory telemetry_dump2 (file.acwu has been deleted)
$ ls .collections/selected/telemetry_dump2
# As expected, file.acwu was deleted from the live file tree
$ ls telemetry_dump2/file.acwu
ls: cannot access 'telemetry_dump2/file.acwu': No such file or directory
|