Dataprovider with Custom return type

[Aswathy]

Hi,

I am using testng v7.6.1. Whether dataprovider can be configured to return anything Custom Type that Iterator<Object[]> or Object[][].

public class DummyTest {

@DataProvider
public Iterator<MyCustomClass> test() {


List<MyCustomClass> list = Arrays.asList(new MyCustomClass());
return list.iterator();
}
}

public class MyCustomClass {


}

I am getting error saying Dataprovider return type should Object[] []/ Iterator<Object[]> 

Regards,

Aswathy

[⇜Krishnan Mahadevan⇝]

Aswathy,

Just change 

public Iterator<MyCustomClass> test()

to 

public Iterator<MyCustomClass[]> test()

and try again.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/9089c4e4-f119-4522-9133-9b4383d7ce5bn%40googlegroups.com.

[Aswathy]

Hi Krishnan,

I have tried above approach. But getting error to change the return type to Object[][] / Iterator<Object[]>

public class DummyTest {

@DataProvider
public Iterator<String[]> test() {

List<String[]> arrayList = new LinkedList<>();

String[] array = new String[] {"abc", "efg"};

arrayList.add(array);

return arrayList.iterator();
}

Also I have testdata set of 5000 size. When I tried to use dataprovider , getting OutOfMemoryException. On analyzing heap dump, the culprit is "new Object[] {"data"} initialization. Any suggestion on how to handle these large data set cases.

Thank you

Aswathy

[⇜Krishnan Mahadevan⇝]

You can do something like this

public class LazyLoadingDataProviders {

  @Test(dataProvider = "dp")
  public void greetings(String person) {
    System.err.println("Hello there " + person);
  }

  @DataProvider(name = "dp")
  Iterator<Object[]> testData() {
    return Arrays.asList(
        new Object[]{"He-man"},
        new Object[]{"Spider-man"},
        new Object[]{"Super-man"}
    ).iterator();
  }
}

For the OutOfMemoryError, can you please share more context ? What version are you using ? What is your data source ?

You should ideally speaking build your own implementation of java.util.Iterator that you would return.

If you use ArraysList().iterator() then your data has already been loaded up in the memory.

You need to ensure that only when the call goes to hasNext() is when the actual read happens and the call to next() returns the read value from hasNext().

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/9589d146-400c-4329-8d10-82144a75363cn%40googlegroups.com.

[Aswathy]

Hi Krishnan,

I have test data present in cassandra. I am reading the data to a list of java objects. Then using collections parallel stream I am creating Object[] of desired variables. I am using a similar approach you shared above. I think I am loading whole data causing outofmemory. Need to check lazy loading part u mentioned. Can you share if any reference examples are there.

version : 7.6.1

Aswathy

[⇜Krishnan Mahadevan⇝]

I dont have any samples that i can share. But you basically need to have the hasNext() and next() in an iterator work such that it queries the next record of cassandra table.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/bf4f732c-6f54-46c6-89a8-506b90aa7430n%40googlegroups.com.

[Aswathy]

Hi krishnan,

I have implemented custom iterator as mentioned.

However I have a question. If I want to use the same data provider for all the tests, lets say - I have 6 @Test in class, and next() I have overridden in such a way that, I load do some tranformation of the data and then pass to test.

In above case, I could see, for all the 6 tests, when next() is called, it creates transformation objects that many times for same data.

Is there way to restrict this or may be I am doing this wrongly. 

Sample below:

public class DummyTest {

List<String> data = new ArrayList<>();

@BeforeClass
public void before() {
data.add("abc");
data.add("efg");
data.add("hij");
data.add("lmn");
}

@DataProvider(parallel = true)
public Iterator<Object[]> data() {

TestIterator iterator = new TestIterator(data);

return iterator;
}


@Test(dataProvider = "data")
public void test1(String d) {

System.out.println("Test1 :" + d);
}

@Test(dataProvider = "data")
public void test2(String d) throws Exception {

System.out.println("Test2 :" + d);
}

@Test(dataProvider = "data")
public void test3(String d) throws Exception {
System.out.println("Test3 :" + d);
}


private class TestIterator implements Iterator<Object[]> {

private List<String> list;
private int size = 0;

public TestIterator(List<String> list) {
this.list = list;

}

@Override
public void remove() {
throw new UnsupportedOperationException("Removal of items is not supported");
}

@Override
public boolean hasNext() {
return size < list.size();
}

@Override
public Object[] next() {

String item = list.get(size ++);

Object[] next = new Object[] {getUpper(item)};
return next;

}
private String getUpper(String str) {
return str.toUpperCase();
}
}

}

In the above getUpper() method is called for same data 3 times. I want to avoid that processing. I am running with parallel="methods". Tried with caching. It's not working. 

Regards

Aswathy

[Aswathy]

Hi @krishnan,

I have solved the above issue using @Factory. But even if dataprovider (parallel= true/ false), I could see all the data is getting loaded first below calling the test which is opposite to what lazy iterator implementation says.

This is similar to loading the entire data upfront.

Can you pls suggest if I am doing something wrong in the initialisation.

public class DummyTest {

String d;

@Factory(dataProvider = "data")
public DummyTest(String data) {
this.d = data;
}

@DataProvider(parallel = true)
public static Iterator<Object[]> data() {

TestIterator iterator = new TestIterator();

return iterator;
}

@Test
public void test1() {

System.out.println("Test1 :" + d + ":" + new Timestamp(System.currentTimeMillis()));
}

@Test
public void test2() throws Exception {

System.out.println("Test2 :" + d + ":" + new Timestamp(System.currentTimeMillis()));
}

@Test
public void test3() throws Exception {
System.out.println("Test3 :" + d + ":" + new Timestamp(System.currentTimeMillis()));
}

private static class TestIterator implements Iterator<Object[]> {

private List<String> list;

private int size = 0;

public TestIterator() {

list = new ArrayList<>() {
{
add("abc");
add("efg");
add("hij");
add("lmn");
}
};

System.out.println("Constructor called :"+ list.hashCode());

}

@Override
public void remove() {
throw new UnsupportedOperationException("Removal of items is not supported");
}

@Override
public boolean hasNext() {
return size < list.size();
}

@Override
public Object[] next() {

String item = list.get(size ++);

System.out.println("next() :" + item);

Object[] next = new Object[] {getUpper(item)};
return next;
}

private String getUpper(String str) {
return str.toUpperCase();
}
}

}

Regards

Aswathy

[⇜Krishnan Mahadevan⇝]

Aswathy,

Basically I was suggesting that you work with a sample that looks like below:

Here's how a data population script would look like ( This basically inserts records into a H2DB)

Java

import com.arakelian.faker.service.RandomPerson;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class PopulateDB {

  private Connection conn;

  @BeforeClass
  public void setup() throws SQLException {
    String location = "./src/test/resources/my_db";
    conn = DriverManager.getConnection("jdbc:h2:file:" + location, "sa", "");
    conn.setAutoCommit(true);
    createTable();
  }

  @Test(invocationCount = 100)
  public void insertData() throws SQLException {
    String name = RandomPerson.get().next().getFirstName();
    String query = "insert into my_user(name) values (?)";
    PreparedStatement statement = conn.prepareStatement(query);
    statement.setString(1, name);
    if (statement.execute()) {
      System.err.println("Inserted " + statement.getUpdateCount() + " record");
    }
  }

  @AfterClass
  public void cleanup() throws SQLException {
    if (conn != null) {
      conn.close();
    }
  }

  private void createTable() throws SQLException {
    String query = "CREATE TABLE my_user(ID INT AUTO_INCREMENT, NAME VARCHAR(50))";
    conn.prepareStatement(query).execute();
  }

}

Here's how your test code would look like :

Java

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Iterator;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Function;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class ReadFromDB {

  private Connection conn;

  @BeforeClass
  public void setup() throws SQLException {
    String location = "./src/test/resources/my_db";
    conn = DriverManager.getConnection("jdbc:h2:file:" + location, "sa", "");
    conn.setAutoCommit(true);
  }

  @DataProvider(name = "dp")
  public Iterator<Student[]> getTableData() throws SQLException {
    String query = "select * from my_user";
    ResultSet rs = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
            ResultSet.CONCUR_READ_ONLY)
        .executeQuery(query);
    return new TableData<>(rs, Student::newInstance);
  }

  @Test(dataProvider = "dp")
  public void testMethod(Student s) {
    System.err.println("Student ==> " + s);
  }

  @AfterClass
  public void cleanup() throws SQLException {
    if (conn != null) {
      conn.close();
    }
  }

  public static class Student {

    private final int id;
    private final String name;

    private Student(int id, String name) {
      this.id = id;
      this.name = name;
    }

    public static Student[] newInstance(ResultSet rs) {
      try {
        String name = rs.getString("name");
        int id = rs.getInt("id");
        return new Student[]{new Student(id, name)};
      } catch (SQLException e) {
        throw new RuntimeException(e);
      }
    }

    @Override
    public String toString() {
      return "Student{id = " + id + ", name='" + name + "'}";
    }
  }

  public static class TableData<T> implements Iterator<T[]> {

    private final ResultSet resultSet;
    private final Function<ResultSet, T[]> supplier;
    private final AtomicBoolean readFlag = new AtomicBoolean(false);

    public TableData(ResultSet resultSet, Function<ResultSet, T[]> supplier) {
      this.resultSet = resultSet;
      this.supplier = supplier;
    }

    @Override
    public boolean hasNext() {
      try {
        if (!readFlag.get() && !resultSet.isBeforeFirst()) {
          // Data was not read. So reset the cursor by one row.
          resultSet.previous();
        }
        return resultSet.next();
      } catch (SQLException e) {
        throw new RuntimeException(e);
      } finally {
        readFlag.set(false);
      }
    }

    @Override
    public T[] next() {
      try {
        return supplier.apply(resultSet);
      } finally {
        readFlag.set(true);
      }
    }
  }
}

When you couple a test factory with a test data provider, the test data provider would be called up front so that the factory can create all the instances. So there's really no lazy data provider thingy happening there.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/bbf9acee-e68d-412f-9940-31fe8f8f5c7an%40googlegroups.com.