Using the Hammerspace Toolkit

The hs tag command enables you to view, set and remove Hammerspace custom metadata tags. Tags are a key-value pair, where the value can be specified or excluded. If no value is supplied, the tag will be set to TRUE. Tags can have a value added as a string (-s ‘some value') or a Hammescript expression (-e ‘some expression'). In this document, all examples used are strings.

Tags are set using the hs tag set <tag name> <file or directory name> command, checked using the hs tag list <file or directory name>, and removed with hs tag delete <tag name> <file or directory name> as shown in the following examples:

As stated in the example command, you do not need to specify the tag value when removing tags.

Best Practice: Where possible, implement controls or standards regarding the usage of tags. Given that tags are often targeted by objectives or even HSTK queries, it’s important that their spelling, case, and exact terms used are consistent.

You can also set tags using the Hammerspace Metadata Plugin for Microsoft Windows. Consult the Hammerspace Metadata Plugin for Windows File Explorer section of this document for information about using that tool to manage file or directory tags.

The hs keyword command can be used to apply keywords to files and directories, which allows them to be targeted by objectives or even HSTK queries. Keywords are set using the hs keyword add <keyword name> <file or directory name> command, checked using the hs keyword list, and removed removed with hs keyword delete <keyword name> <file or directory name> as shown in the following examples:

Best Practice: Note that keywords do not use a schema, meaning they are free-form. Where possible, implement controls or standards regarding the usage of keywords. Given that keywords are often targeted by objectives or even HSTK queries, it’s important that their spelling, case, and exact terms used are consistent.

You can also set keywords using the Hammerspace Metadata Plugin for Microsoft Windows. Consult the Hammerspace Metadata Plugin for Windows File Explorer section of this document for information about using that tool to manage file or directory keywords.

The HSTK can be used to assign existing labels to files and directories as metadata. Labels have an administrator-defined schema that is set up using the Hammerspace CLI or API. Users can only apply labels that are already defined.

This type of metadata only lives within the file system itself, and does not change the contents of the file. The metadata is also replicated when used together with the Global File System feature.

Attributes have a system-defined hierarchy. In other words, the set of attributes is predefined as part of the Hammerspace product. In this section, we will review some of the most basic uses of attributes. For a more advanced example that better illustrates how they might be used, refer to the Hammerspace and Data Lifecycle Management section of this document.

You can list the some of the defined attributes and their values for a file using the hs attribute list command:

While we won’t go into all the attributes and their uses in this document, we will use "COLORS" as an example. The COLORS attribute contains a set of colors applied to a file. The allowed list of colors is predefined in the Hammerspace product, similar to how colors are available in Apple MacOS. The hs attribute set colors command is used to set file colors:

You can set multiple colors by separating the lookups with a semicolon:

Once set, you can use the hs eval command to query for files with a particular color selected:

One of the default objectives applied to a Hammerspace share keeps files online that have the keep-on-site value set for that site. Assuming that the default objective is in place (or one similar to it), setting this value will keep the file (or contents of the directory) online using nothing more than the default share objectives.

When preparing to set this value, first list the sites using the hs keep-on-site available command:

In this example, we see that the share is available on two sites - NYC and LA.

The value can then be set with the Hammerspace toolkit using the hs keep-on-site command specifying add or delete as needed:

In the previous example, we targeted the current directory using a period (.) - you can also specify a directory or file name.

The hs rekognition-tag command is also used to set file and directory tags, and follows the same format as the hs tag command. Consult the tag section of this document for the command syntax and examples.

In this section, we will outline some of the most commonly used metadata values within Hammerscript expressions, be it for objectives or while using the HSTK.

Many of the Hammerscript examples in this document leverage values such as SPACE_USED and LAST_USE_AGE. The former is self explanatory, while the latter represents the last time a file is closed and is the preferred value for measuring when a file was last used.

Additional metadata values are detailed in the Appendix A section Additional Metadata Objects And Functions. For a full list of metadata objects Hammerspace shares, refer to the File Metadata Example (Full) section of this document.

The hs eval command allows us to evaluate Hammerscript expressions on a file, or simply return a specified metadata value. This can be used for a wide range of purposes such as reporting, testing objective conditions, or advanced searches.

Eval commands are not recursive by default. They will be performed on the current directory OR the path if one is provided. To make the command recursive, add the -r switch.

Best Practice: If you are looking for output suitable for later analysis in spreadsheet tools, hs eval is likely to require the least amount of effort to parse. That said, hs eval does not summarize output, but rather displays results for each file individually, so you may find that hs sum (described in the next section) is a more suitable command for some use cases. The same queries will often work for both commands, though it’s important to note that hs sum is recursive, so it is easy enough to try both and compare the output.

The hs sum command is used to perform fast calculations on a set of files. Typically, a Hammerscript expression used with hs eval can be used with hs sum, but note that hs sum is recursive while hs eval is not (unless specified).

The hs eval command differs from hs sum in that you can sum and sort data using the HSTK itself, rather than needing to import it into spreadsheet tools to perform that task. Plus, hs sum also grants you access to built in reports for the top 10, 100, or 1000 examples for supported metadata objects.

The hs collsum command provides usage details about one or more of the default share collation summaries. While it is possible to create an unlimited number of custom summaries using hs eval or hs sum, the default collsums are automatically generated and cover many of the most common storage reports.

The following command leverages grep to obtain the standard list of collsums. A description of each is provided to the right of the name:

Collsums are continually updated, and are the fastest method to obtain filesystem statistics. They will always return data for the entire share, regardless of where the command was actually executed from within the share.

Many collsums provide data similar to what you find in the Hammerspace GUI per-share dashboards, though if you are looking to prepare your own custom reports, the HSTK is the preferred method for obtaining this information.

Best Practice: Exploring the output of the various collsums is an easy way to learn about the different metadata values that you might want to use to create your own custom reports.

The hs usage command shows information about file instance count, capacity consumption, volume usage, and more about files in a given path.

The hs perf command provides performance and operation stats on a per-share basis, and is out of scope for this document. Hammerspace recommends using the integrated Prometheus exporters to collect cluster performance statistics and display them using the custom Hammerspace dashboards for Grafana.

For information regarding the use of Grafana and Prometheus to gather Hammerspace cluster performance data, see Monitoring Cluster Health in the Hammerspace Administration Guide.

The following three examples show the same Hammerscript expression being used with hs eval and then hs sum. When choosing which command, know that it will ultimately depend on the question you are trying to answer. That being said, it is usually quite easy to insert your expression into either one of those commands. While only hs sum can build the TOP tables, hs eval can do thorough reporting, and then analyze the data how you see fit using any number of tools.

The following examples show the difference in output between hs eval and hs sum for three different expressions.

Commands similar to this would make it easy to find those files which are consuming the most space.

Hammerspace does not implement user or group quotas, though you can set a size limit on each share which functions as a share quota. However, Hammerspace does provide the ability to generate sophisticated usage reports as you might expect from a quota tool or system.

For these examples, we are mounted to a Hammerspace share, and the Python virtual environment is in the command search path. The commands work identically on Windows and Linux.

In this section, we will review a variety of reports used to report on file data for the entire cluster, find specific file types, or files that meet certain characteristics.

The hs cp command performs a fast offloaded recursive copy using a clone operation. The following example shows how it is used:

The hs rm command is a multithreaded version of the Linux rm -rf command. This command performs the operation directly on the Hammerspace cluster, which provides the most efficient way to delete large directories from Hammerspace shares.

The command should be executed using the same syntax as the Linux equivalent (rm -rf) as shown in the following example.

The hs rsync command is an offloaded version of the Linux recursive directory equalizer. Note that like rsync, when updating a directory, files are both added and removed based on the state of the source directory when the command was run. By offloading the operation, this command ensures the best performance when syncing data from within locations on a Hammerspace cluster.

The following is an example of the command syntax, note that the --delete and --archive options are required:

The hs rm-st is identical to the hs rm command, though it is not multi-threaded. The command syntax and options are identical. Refer to the RM section of this document for an example of how the command is used.

The hs status command provides views into the status of various processes, tasks, and objects in the Hammerspace environment.

These commands generally require no options. Simply execute them from within a Hammerspace share similar to the following example (partial command output shown):

In this section, we will review the HSTK commands used to list and manage objectives.

Objectives are defined by the administrator using the Hammerspace CLI, GUI, or API. However, they can be applied to files and directories (with inheritance) by users with the HTSK. The HSTK also provides a way to list the inherited and directly applied objectives on files and directories.

Best Practice:

The hs dump command provides a number of subcommands used to obtain information about the cluster or the files and directories that it contains. The information obtained from subcommands such as misaligned, threat, files_on_volume can also be obtained using hs sum or hs eval commands, while others such as inode are typically only used by Hammerspace support.

A common use case for the hs dump command is to list the objectives whose names you may need to know. The hs dump objectives command lists all objectives present on the cluster. The following example is a shortened version of the command output.

Remember that objectives are created per-cluster. While some are created by default, all others are created when you add volumes, create volume groups, or create your own custom objectives.