how use StructuredRecord get function?

[todd.l...@gmail.com]

I am confused about how to use the get() method of the StructuredRecord class to get a field value from a structured record.

The Javadoc for StructuredRecord at http://docs.cdap.io/cdap/current/en/reference-manual/javadocs/index.html shows the signature as:

public <T> T get(String fieldName)

That agrees with the code for the StructuredRecord class at: https://github.com/caskdata/cdap/blob/develop/cdap-api-common/src/main/java/co/cask/cdap/api/data/format/StructuredRecord.java

public <T> T get(String fieldName) {

return (T) fields.get(fieldName);

}

I see an example of the useage of this function in the StructuredToTextTransformer class at https://github.com/caskdata/hydrator-plugins/blob/develop/core-plugins/src/main/java/co/cask/hydrator/plugin/common/StructuredToTextTransformer.java.

It is very simply invoked there as:
builder.append(getFieldAsString(structuredRecord.get(fieldIterator.next().getName())));

However, when I try to invoke the get() method that same way on the StructuredRecord that is passed into my transform plugin I get the error: Cannot resolve method 'get(java.lang.string)'

Here is my code

  public void transform(StructuredRecord input, Emitter<StructuredRecord> emitter) throws Exception {
    String fieldname = "body";
    String s = input.get(fieldname);
    
I think I am invoking get the same as is done in the StructuredToTextTransformer class example, but I must be doing something wrong.

How to I use the get() method of StructuredRecord to get the value of a field?

Thanks,

Todd

[todd.l...@gmail.com]

Strangely, if I cast the variable input, already a StructuredRecord,  to (co.cask.cdap.api.data.format.StructuredRecord) input then it works.

I don't import co.cask.cdap.api.data.format.StructuredRecord because then IntelliJ says the import is unused. I tried importing it and it didn't make any difference.

Here is the complete code to my transform plugin. Without the cast the compiler will complain about not finding the get() method:


import co.cask.cdap.api.annotation.Description;
import co.cask.cdap.api.annotation.Name;
import co.cask.cdap.api.annotation.Plugin;
import co.cask.cdap.api.plugin.PluginConfig;
import co.cask.cdap.etl.api.Emitter;
import co.cask.cdap.etl.api.Transform;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * try3
 */
@Plugin(type = "transform")
@Name("try3")
@Description("This is a demonstration TCA transform")
public class try3Transform<StructuredRecord> extends Transform<StructuredRecord, StructuredRecord> {

  private static final Logger logger = LoggerFactory.getLogger(try3Transform.class);

  private final Config config;

  public try3Transform(Config config) {
        this.config = null;
    }


  @Override

  public void transform(StructuredRecord input, Emitter<StructuredRecord> emitter) throws Exception {

    // Perform transformation
    //emitter.emit(value)
    String body = ((co.cask.cdap.api.data.format.StructuredRecord) input).get("body");
    logger.debug("input= {}", body );
    emitter.emit(input);
  }
}

[Terence Yim]

Hi Todd,

I believe it’s because you have try3Transform<StructuredRecord>. Try to remove the parameterized type then it should work. The reason for the error is because, if you have <StructuredRecord> as the parameterized type of your class, it becomes a Type Variable instead of the actual StructuredRecord class in the CDAP api.

public class try3Transform extends Transform<StructuredRecord, StructuredRecord> {

…

}

Terence

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/2196ac72-d4e3-4aff-8eb2-6efda95322e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[todd.l...@gmail.com]

Terence.

I don't completely understand how a parameterized type of a class becomes a Type Variable and hides the StructuredRecord class in the CDAP api, maybe I need to hit the books more about generics, but I removed it as you suggested and it worked.

I also went back and looked more closely at the Hydrator plugin code I used as examples, and now I see that they don't have the class type parameterized.

I was afraid to remove the parameterized type because it came from the sample code in the maven archetype, but it is actually not used at all in the method. Maybe it should be removed from the cdap-etl-transform-archetype maven archetype.

[Terence Yim]

Hi Todd,

Glad that it is working for you now.

When you have something like

class MyClass<StructuredRecord>

the “StructureRecord” is not referring to a any concrete class, but instead it is a label (called type variable in Java generic term) to denote a generic type.

I’ve looked at the maven archetype and I understand why it gave you confusion. We’ll improve the archetype in future release to make it less confusing.

Thanks,

Terence

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/54a9b49f-a21e-440a-b256-233c127aa364%40googlegroups.com.

[todd.l...@gmail.com]

Terence,

I think the issue is that that the TransformPlugin archetype is set up to be an abstract base class. The archetype is:

@Plugin(type = "transform")
@Name("MyTransform")
@Description("This is my transform")
public class TransformPlugin<T> extends Transform<T, T> {

  @Override
  public void transform(T input, Emitter<T> emitter) throws Exception {
    // Perform transformation
    //emitter.emit(value);
  }
}

The proper way to use it I think is to derive my own class from it like so:
public class MyTransformPlugin extends TransformPlugin<StructuredRecord>

Instead of that I short-cutted one class derivation, which made it hard to figure out what to do with the generic type declaration.

Thanks again,

Todd