How to delete 1 or more records from a dataset

[Mats Naslund]

Just wondering what would be the best approach to delete 1 or more records from a partitioned dataset. When I attempt to use deleteAll I get "Cannot cleanly delete view" and I've read "Because records are grouped into files by partition values, we either have to delete an entire file because we know all records match the deletion criteria, or rewrite a file to remove the matching records. We decided to delete entire files and reject deletes that require rewriting." Does this still apply? How difficult would it be to change this?

Thank you

[Micah Whitacre]

The rewriting the entire file or deleting the entire file is still the approach.

Manipulating a single record is not trivial because typically the records are stored in side of a larger file and that larger file is also typically compressed.  In the case of data stored as Avro, the data is block compressed so you'd be manipulating a specific block in a file.  

This restriction is also built on top of HDFS which doesn't support modification but instead append only.[1]

If you need to modify individual records you might look at something like HBase.

[1] - https://www.quora.com/Is-HDFS-an-append-only-file-system-Then-how-do-people-modify-the-files-stored-on-HDFS

[Mats Naslund]

Thank you