Re: How to read/save huge files

280 views

Skip to first unread message

Message has been deleted

Tom Fennelly

unread,

Feb 20, 2017, 11:21:24 AM2/20/17

to smook...@googlegroups.com

Seems like you're trying to get a lot of things working at one time and any one of them could be causing a failure.

Suggestion would be to simplify your use case considerably and then start adding to it bit by bit. At least then, if it fails, you'll have some clue as to which piece of the puzzle is causing the problems.

Simplify the CSV format... batch size of one (or better again - remove the persistence step completely for now) ... remove all the com.ibm.jbatch stuff out of the picture.

On 20 February 2017 at 15:02, Everson <ever...@gmail.com> wrote:

Hello I'm facing two problems

Problem 1)

I have csv files between 300k and 500k lines, even more sometimes. I'm getting memory problems. What kind approach should I use ?

I was thinking something by batch, read each 5k lines and insert in database(mongo) or even by each line

Piece of my code:

Smooks smooks = new Smooks(getParameters().getProperty("smooks-config")); ExecutionContext executionContext = smooks.createExecutionContext(); Map<String, String> params = new HashMap<>(); params.put("domainId", getParameters().getProperty("domain")); executionContext.getBeanContext().addBean("params", params); String messageIn = StreamUtils.readStreamAsString(new FileInputStream(getParameters().getProperty("csvFilePath"))); JavaResult result = new JavaResult(); smooks.filterSource(executionContext, new StringSource(messageIn), result); List<Customer> customers = (List<Customer>) result.getBean("customerList"); //the problem is here, smooks read all 500k lines of csv and then save //save on mongodb customerService.insert(customers);

<params> <param name="stream.filter.type">SAX </param> </params> <csv:reader fields="email1,email2,email3,dddcel1,dddcel2,dddcel3,dddcel4,dddtel1,dddtel2,dddtel3,firstName,document,value,cc,$ignore$,bank,agency,$ignore$,titulo,tel_retorn,street,neighborhood,zip,city,state,$ignore$" separator=";" indent="true" > </csv:reader> <jb:bean beanId="customerList" class="java.util.ArrayList" createOnElement="csv-set"> <jb:wiring beanIdRef="customer" /> </jb:bean> <jb:bean beanId="customer" class="package.Customer" createOnElement="csv-set/csv-record/"> ...

Exceptions:

------------------------------------------------------------------------------1

Severe: Exception in thread "Thread-8"
Severe: java.lang.OutOfMemoryError: GC overhead limit exceeded
at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:359)
at java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:349)
at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580)
at com.sun.enterprise.server.logging.UniformLogFormatter.uniformLogFormat(UniformLogFormatter.java:275)
at com.sun.enterprise.server.logging.UniformLogFormatter.format(UniformLogFormatter.java:178)
at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at com.sun.common.util.logging.LoggingOutputStream.log(LoggingOutputStream.java:163)
at com.sun.common.util.logging.LoggingOutputStream$1.run(LoggingOutputStream.java:123)
Warning: Caught exception executing step: com.ibm.jbatch.container.exception.BatchContainerRuntimeException: org.milyn.SmooksException: Failed to filter source.
at com.ibm.jbatch.container.artifact.proxy.BatchletProxy.process(BatchletProxy.java:36)
at com.ibm.jbatch.container.impl.BatchletStepControllerImpl.invokeBatchlet(BatchletStepControllerImpl.java:80)
at com.ibm.jbatch.container.impl.BatchletStepControllerImpl.invokeCoreStep(BatchletStepControllerImpl.java:108)
at com.ibm.jbatch.container.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:144)
at com.ibm.jbatch.container.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:112)
at com.ibm.jbatch.container.impl.JobThreadRootControllerImpl.originateExecutionOnThread(JobThreadRootControllerImpl.java:110)
at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:80)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
Caused by: org.milyn.SmooksException: Failed to filter source.
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:97)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:64)
at org.milyn.Smooks._filter(Smooks.java:526)
at org.milyn.Smooks.filterSource(Smooks.java:482)
at core.jobs.CustomerSmooksBatchlet.process(CustomerSmooksBatchlet.java:70)
at com.ibm.jbatch.container.artifact.proxy.BatchletProxy.process(BatchletProxy.java:33)
... 13 more
Caused by: org.milyn.SmooksException: Error in 'org.milyn.javabean.BeanInstancePopulator' while processing the visitAfter event.
at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:578)
at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:564)
at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:401)
at org.milyn.delivery.sax.SAXHandler.endElement(SAXHandler.java:204)
at org.milyn.delivery.SmooksContentHandler.endElement(SmooksContentHandler.java:96)
at org.milyn.flatfile.FlatFileReader.parse(FlatFileReader.java:173)
at org.milyn.delivery.sax.SAXParser.parse(SAXParser.java:76)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:86)
... 18 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.milyn.javabean.BeanInstancePopulator.bindSaxDataValue(BeanInstancePopulator.java:446)
at org.milyn.javabean.BeanInstancePopulator.visitAfter(BeanInstancePopulator.java:378)
at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:394)
... 23 more

------------------------------------------------------------------------------2

Warning: Caught exception executing step: com.ibm.jbatch.container.exception.BatchContainerRuntimeException: org.milyn.SmooksException: Failed to filter source.
at com.ibm.jbatch.container.artifact.proxy.BatchletProxy.process(BatchletProxy.java:36)
at com.ibm.jbatch.container.impl.BatchletStepControllerImpl.invokeBatchlet(BatchletStepControllerImpl.java:80)
at com.ibm.jbatch.container.impl.BatchletStepControllerImpl.invokeCoreStep(BatchletStepControllerImpl.java:108)
at com.ibm.jbatch.container.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:144)
at com.ibm.jbatch.container.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:112)
at com.ibm.jbatch.container.impl.JobThreadRootControllerImpl.originateExecutionOnThread(JobThreadRootControllerImpl.java:110)
at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:80)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
Caused by: org.milyn.SmooksException: Failed to filter source.
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:97)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:64)
at org.milyn.Smooks._filter(Smooks.java:526)
at org.milyn.Smooks.filterSource(Smooks.java:482)
at core.jobs.CustomerSmooksBatchlet.process(CustomerSmooksBatchlet.java:59)
at com.ibm.jbatch.container.artifact.proxy.BatchletProxy.process(BatchletProxy.java:33)
... 13 more
Caused by: org.milyn.SmooksException: Error in 'org.milyn.javabean.BeanInstancePopulator' while processing the visitAfter event.
at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:578)
at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:564)
at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:401)
at org.milyn.delivery.sax.SAXHandler.endElement(SAXHandler.java:204)
at org.milyn.delivery.SmooksContentHandler.endElement(SmooksContentHandler.java:96)
at org.milyn.flatfile.FlatFileReader.parse(FlatFileReader.java:173)
at org.milyn.delivery.sax.SAXParser.parse(SAXParser.java:76)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:86)
... 18 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Problem 2)

Trying to solve problem 1 I got a new exception

i changed smooks config to csv:singleBinding instead a List

<csv:reader fields="email1,email2,email3,dddcel1,dddcel2,dddcel3,dddcel4,dddtel1,dddtel2,dddtel3,firstName,document,value,cc,$ignore$,bank,agency,$ignore$,titulo,tel_retorn,street,neighborhood,zip,city,state,$ignore$" separator=";" indent="true" > <csv:singleBinding beanId="customer" class="core.entity.Customer" /> </csv:reader> <jb:bean beanId="customer" class="package.Customer" createOnElement="csv-set/csv-record/"> ... <jb:wiring beanIdRef="emails" property="emails" /> <jb:wiring beanIdRef="mobiles" property="mobiles" /> <jb:wiring beanIdRef="phones" property="phones" /> <jb:wiring property="invoiceDetail" beanIdRef="invoiceDetail" /> <jb:wiring beanIdRef="address" property="address" /> </jb:bean> <jb:bean beanId="csvMap" class="java.util.HashMap" createOnElement="csv-record"> <jb:value property="@email1" data="csv-set/csv-record/email1" /> <jb:value property="@email2" data="csv-set/csv-record/email2" /> <jb:value property="@email3" data="csv-set/csv-record/email3" /> <jb:value property="@dddcel1" data="csv-set/csv-record/dddcel1" /> <jb:value property="@dddcel2" data="csv-set/csv-record/dddcel2" /> <jb:value property="@dddcel3" data="csv-set/csv-record/dddcel3" /> <jb:value property="@dddcel4" data="csv-set/csv-record/dddcel4" /> <jb:value property="@dddtel1" data="csv-set/csv-record/dddtel1" /> <jb:value property="@dddtel2" data="csv-set/csv-record/dddtel2" /> <jb:value property="@dddtel3" data="csv-set/csv-record/dddtel3" /> </jb:bean>  <jb:bean beanId="emails" class="java.util.ArrayList" createOnElement="csv-set/csv-record"> <jb:expression execOnElement="csv-set/csv-record/email1"> (csvMap["email1"] == null || csvMap["email1"].isEmpty() ) ? null : csvMap["email1"]; </jb:expression> <jb:expression execOnElement="csv-set/csv-record/email2"> (csvMap["email2"] == null || csvMap["email2"].isEmpty() ) ? null : csvMap["email2"]; </jb:expression> <jb:expression execOnElement="csv-set/csv-record/email3"> (csvMap["email3"] == null || csvMap["email3"].isEmpty() ) ? null : csvMap["email3"]; </jb:expression> </jb:bean>

... and changed my code to get each line of csv and then save...

executionContext.getBeanContext().addObserver(new BeanContextLifecycleObserver() { @Override public void onBeanLifecycleEvent(BeanContextLifecycleEvent event) { Customer customer = null; if (event.getLifecycle().equals(BeanLifecycle.END_FRAGMENT)) { if (event.getBeanId().toString().equals("customer")) { customer = (Customer) event.getBean(); Map<String, Object> map = event.getExecutionContext().getBeanContext().getBeanMap(); customer.setEmails((ArrayList) map.get("emails")); customer.setPhones((ArrayList) map.get("phones")); customer.setMobiles((ArrayList) map.get("mobiles")); customer.setAddress((Address) map.get("address")); customerService.insert(customer); } } } });

But I'm getting a new exception when de code is running on glassfish.

Warning: Caught exception executing step: com.ibm.jbatch.container.exception.BatchContainerRuntimeException: org.milyn.SmooksException: Failed to filter source.
at com.ibm.jbatch.container.artifact.proxy.BatchletProxy.process(BatchletProxy.java:36)
at com.ibm.jbatch.container.impl.BatchletStepControllerImpl.invokeBatchlet(BatchletStepControllerImpl.java:80)
at com.ibm.jbatch.container.impl.BatchletStepControllerImpl.invokeCoreStep(BatchletStepControllerImpl.java:108)
at com.ibm.jbatch.container.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:144)
at com.ibm.jbatch.container.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:112)
at com.ibm.jbatch.container.impl.JobThreadRootControllerImpl.originateExecutionOnThread(JobThreadRootControllerImpl.java:110)
at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:80)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
Caused by: org.milyn.SmooksException: Failed to filter source.
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:97)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:64)
at org.milyn.Smooks._filter(Smooks.java:526)
at org.milyn.Smooks.filterSource(Smooks.java:482)
at core.jobs.CustomerSmooksBatchlet.process(CustomerSmooksBatchlet.java:118)
at com.ibm.jbatch.container.artifact.proxy.BatchletProxy.process(BatchletProxy.java:33)
... 13 more
Caused by: org.milyn.SmooksException: Error in 'org.milyn.javabean.BeanInstancePopulator' while processing the visitAfter event.
at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:578)
at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:564)
at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:401)
at org.milyn.delivery.sax.SAXHandler.endElement(SAXHandler.java:204)
at org.milyn.delivery.SmooksContentHandler.endElement(SmooksContentHandler.java:96)
at org.milyn.flatfile.FlatFileReader.parse(FlatFileReader.java:165)
at org.milyn.delivery.sax.SAXParser.parse(SAXParser.java:76)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:86)
... 18 more
Caused by: org.milyn.cdr.SmooksConfigurationException: Bean [customer] configuration invalid. Bean setter method [setEmail1(java.lang.String)] not found on type [core.entity.Customer]. You may need to set a 'decoder' on the binding config.
at org.milyn.javabean.BeanInstancePopulator.setPropertyValue(BeanInstancePopulator.java:560)
at org.milyn.javabean.BeanInstancePopulator.decodeAndSetPropertyValue(BeanInstancePopulator.java:512)
at org.milyn.javabean.BeanInstancePopulator.bindSaxDataValue(BeanInstancePopulator.java:448)
at org.milyn.javabean.BeanInstancePopulator.visitAfter(BeanInstancePopulator.java:378)
at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:394)
... 23 more

--
You received this message because you are subscribed to the Google Groups "Smooks Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smooks-user+unsubscribe@googlegroups.com.
To post to this group, send email to smook...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/smooks-user/efa61e32-d843-4bcb-9f47-77c18a286d65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Message has been deleted

Tom Fennelly

unread,

Feb 22, 2017, 2:37:55 PM2/22/17

to smook...@googlegroups.com

Sorry dude ... seems like you're just doing something daft in there that required plain good old debugging. There's one obvious error there in your stack trace (I assume you've seen it) that you should obviously resolve before doing more (wrt the Customer type not having a setEmail method).

On 22 February 2017 at 13:25, Everson <ever...@gmail.com> wrote:

public class Customer implements Serializable { @Embedded private Address address; private List<String> emails; private List<String> mobiles; private List<String> phones;

Em quarta-feira, 22 de fevereiro de 2017 10:17:53 UTC-3, Everson escreveu:
I have a junit test in the project which works fine. When i'm testing, I'm using a csv file with just 4 lines. I tested also calling the code below inside an ejb and call it from a serlvet. The same error happened likew when I call from a job/Batchlet. The thing is, when I was using bindingList it worked fine(read the csv file). just when I changed to singleBinding this error(1) starts (start when I was trying to solve another one... save huge file )

public void testCustomer1() throws IOException, SAXException { Smooks smooks = new Smooks(props.getProperty("smooks-config")); smooks.addVisitor(new SAXVisitAfter() { public void visitAfter(SAXElement element, ExecutionContext executionContext) throws SmooksException, IOException { Customer c = (Customer) executionContext.getBeanContext().getBean("customer"); List<String> emails = (ArrayList) executionContext.getBeanContext().getBean("emails"); List<String> phones = (ArrayList) executionContext.getBeanContext().getBean("phones"); List<String> mobiles = (ArrayList) executionContext.getBeanContext().getBean("mobiles"); Address address = (Address) executionContext.getBeanContext().getBean("address"); c.setEmails(emails); c.setPhones(phones); c.setMobiles(mobiles); c.setAddress(address); datastore.save(c, new InsertOptions().continueOnError(true)); } }, "csv-set/csv-record");

ExecutionContext executionContext = smooks.createExecutionContext(); Map<String, String> params = new HashMap<>();

params.put("domainId", props.getProperty("domain")); executionContext.getBeanContext().addBean("params", params); String messageIn = StreamUtils.readStreamAsString(new FileInputStream(props.getProperty("csvFilePath")));

JavaResult result = new JavaResult(); smooks.filterSource(executionContext, new StringSource(messageIn), result);

Customer c = (Customer) result.getBean("customer"); Assert.assertNotNull(c); }

1)

Caused by: org.milyn.SmooksException: Error in 'org.milyn.javabean.BeanInstancePopulator' while processing the visitAfter event. at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:578) at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:564) at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:401) at org.milyn.delivery.sax.SAXHandler.endElement(SAXHandler.java:204) at org.milyn.delivery.SmooksContentHandler.endElement(SmooksContentHandler.java:96) at org.milyn.flatfile.FlatFileReader.parse(FlatFileReader.java:165) at org.milyn.delivery.sax.SAXParser.parse(SAXParser.java:76) at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:86)

... 36 more

Caused by: org.milyn.cdr.SmooksConfigurationException: Bean [customer] configuration invalid. Bean setter method [setEmail1(java.lang.String)] not found on type [core.entity.Customer]. You may need to set a 'decoder' on the binding config. at org.milyn.javabean.BeanInstancePopulator.setPropertyValue(BeanInstancePopulator.java:560) at org.milyn.javabean.BeanInstancePopulator.decodeAndSetPropertyValue(BeanInstancePopulator.java:512) at org.milyn.javabean.BeanInstancePopulator.bindSaxDataValue(BeanInstancePopulator.java:448) at org.milyn.javabean.BeanInstancePopulator.visitAfter(BeanInstancePopulator.java:378) at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:394)

... 41 moreHandler.java:578) at org.milyn.delivery.sax.SAXHandler.processVisitorException(SAXHandler.java:564) at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:401) at org.milyn.delivery.sax.SAXHandler.endElement(SAXHandler.java:204) at org.milyn.delivery.SmooksContentHandler.endElement(SmooksContentHandler.java:96) at org.milyn.flatfile.FlatFileReader.parse(FlatFileReader.java:165) at org.milyn.delivery.sax.SAXParser.parse(SAXParser.java:76) at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:86) ... 36 more

Caused by: org.milyn.cdr.SmooksConfigurationException: Bean [customer] configuration invalid. Bean setter method [setEmail1(java.lang.String)] not found on type [core.entity.Customer]. You may need to set a 'decoder' on the binding config. at org.milyn.javabean.BeanInstancePopulator.setPropertyValue(BeanInstancePopulator.java:560) at org.milyn.javabean.BeanInstancePopulator.decodeAndSetPropertyValue(BeanInstancePopulator.java:512) at org.milyn.javabean.BeanInstancePopulator.bindSaxDataValue(BeanInstancePopulator.java:448) at org.milyn.javabean.BeanInstancePopulator.visitAfter(BeanInstancePopulator.java:378) at org.milyn.delivery.sax.SAXHandler.visitAfter(SAXHandler.java:394)

... 41 more

1)

To unsubscribe from this group and stop receiving emails from it, send an email to smooks-user...@googlegroups.com.

To post to this group, send email to smook...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/smooks-user/efa61e32-d843-4bcb-9f47-77c18a286d65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Smooks Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smooks-user+unsubscribe@googlegroups.com.
To post to this group, send email to smook...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/smooks-user/1f3a497f-8eb7-453d-8512-ca15ebaf95ab%40googlegroups.com.