set TTL from spark

[Antonio Si]

Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

[Chaomin Yu]

Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:

•  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  
•  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]

2. For now, Alluxio does not support setting TTL as path suffix. You can use either

• "CreateFileOptions" to set TTL upon file creation, or
• "SetAttributeOption" after the file has been created.
  

Hope this helps,

Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Chaomin

Alluxio Inc

[Antonio Si]

Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException

at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)

at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)

at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)

at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)

at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)

at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)

at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)

at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)

at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)

at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)

at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)

at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)

at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)

at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

[Haoyuan Li]

Which version of Alluxio are you running?

Alluxio, Inc. 

[Antonio Si]

Hi Haoyuan,

I am using Alluxio 1.0.1.

Thanks.

Antonio.

[Antonio Si]

Hi,

A followup question. Does the setAttribute only work on a file or I can set it to a directory also?

Thanks.

Antonio.

[Chaomin Yu]

Hi Antonio,

setAttribute in general can be applied to a directory. 

But setAttribute for the purpose of TTL only works on a file, because TTL is not yet supported on directories.

For the TTransportException you saw, did you set TTL on a file or a directory?

[Antonio Si]

Thanks Chaomin.

I set the attribute on a directory. I changed it to the files under the directory and it is working fine.

A related question. Let say I set the TTL on a file for 2 hrs. Before it expires, I set it

again for 2 hrs. Does it reset the timer and measure 2 hrs from that point on?

Thanks.

Antonio.

[Chaomin Yu]

Glad to hear that it works for you!

Yes, if you set TTL again, mtime (last modified time) is updated and the new TTL will apply.

You can refer to alluxio.master.file.FileSystemMaster#setAttributeInternal for the code.

Cheers,

Chaomin