Spike: Investigate a data storage interface that simplifies table config.

[ddanielr]

Is your feature request related to a problem? Please describe.
The current Erasure Coding (EC) policy settings are set via table properties and performed via the ErasureCodeConfigurer class

There was a recent desire to exclude a particular volume from the EC policy due to storage layer configuration.
While it is possible to add volume knowledge into the ErasureCodeConfigurer properties, the configuration model gets complicated.

Erasure Coding is a data storage format setting that seems highly related to volumes whereas table settings generally manage data structure.

Describe the solution you'd like
I think reworking the EC policy settings to be attached to/underneath a volume definition makes the interface cleaner.
EC policy relates to how data is stored. The current settings are based on object sizes, but if volume selection can be done based on size then I'm not sure the table has any need to know its storage format.

Unless there is a limitation on the number of volumes that can be defined in accumulo.

There are also other settings that may have been applied at the table level that should also be consolidated behind the volume definition.

The various data storage format configuration options in accumulo should be investigated and see if the complexity is simplified if put behind a common interface (volumes, storage formats, etc).

[ctubbsii]

I'm not too familiar with erasure coding policy settings. Can you briefly explain what kinds of settings are available and how they are configured (at what granularity, like per-file, per-directory, etc.) in HDFS?

[dlmarion]

I'm not too familiar with erasure coding policy settings. Can you briefly explain what kinds of settings are available and how they are configured (at what granularity, like per-file, per-directory, etc.) in HDFS?

The setPolicy command allows the administrator to set an EC policy on a path. EC can also be set system-wide via Hadoop configuration. New files created under the path use the EC scheme set. However, options exist on the FSDataOutputStream to override this behavior.

Something to note as part of this conversation - users already have the ability to override Hadoop properties on a per-volume basis via #3180.

Correct me if I'm wrong, but I think what is being described here is giving users more control over the options set on an RFile on a per-file basis, not volume. There are ec and setDropBehind options set on the OutputStream in RFileOperations.openWriter that are based on table properties.

I think it makes sense to move the code that modifies the RFile OutputStream to a more volume-centric location as there is a 1:1 relationship between the Volume and the File.