ZoieSegmentReader.getUID maybe throws exceptioin

[genuine]

2011-12-21 16:48:21 org.apache.solr.common.SolrException log
严重: java.lang.ArrayIndexOutOfBoundsException: 952
at
proj.zoie.api.ZoieSegmentReader.getUID(ZoieSegmentReader.java:251)
at proj.zoie.solr.ZoieUpdateHandler
$2.collect(ZoieUpdateHandler.java:287)
at org.apache.lucene.search.TermScorer.score(TermScorer.java:
75)
at org.apache.lucene.search.TermScorer.score(TermScorer.java:
67)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:526)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:339)
at
proj.zoie.solr.ZoieUpdateHandler.deleteByQuery(ZoieUpdateHandler.java:
276)



where I call deleteByQuery() ,It will throw the Exception...


what 's the problem?

[John Wang]

Are you using a standard Query implementation or your own?

-John

--
You received this message because you are subscribed to the Google Groups "zoie" group.
To post to this group, send email to zo...@googlegroups.com.
To unsubscribe from this group, send email to zoie+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/zoie?hl=en.

[genuine]

standard

[genuine]

I change the source like that:

@Override public void collect(int doc) throws IOException {
try { // long uid = zoieReader.getUID(doc + base); long uid
= zoieReader.getUID(doc); if (uid != DocIDMapper.NOT_FOUND)
{ delList.add(uid); } } catch (Exception e) {
errorCount++; delCountList.set(0, errorCount); } }

And I delete by Query do success.

[康大强]

I have met the exception, too. I hava created a issue.

2011/12/21 genuine <yzyge...@gmail.com>

[John Wang]

Awesome!

Can you create a jira ticket at: linkedin.jira.com

and also send a patch on github?

Thanks!

-John

[John Wang]

Which class did you make the change?

-John

On Mon, Dec 26, 2011 at 12:05 AM, genuine <yzyge...@gmail.com> wrote:

[Gino Yu]

package proj.zoie.solr;

import it.unimi.dsi.fastutil.longs.LongArrayList;
import it.unimi.dsi.fastutil.longs.LongList;

import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.Future;

import org.apache.log4j.Logger;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.MultiReader;
import org.apache.lucene.search.Collector;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.Scorer;
import org.apache.lucene.store.Directory;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.SimpleOrderedMap;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.core.SolrCore;
import org.apache.solr.search.QueryParsing;
import org.apache.solr.update.AddUpdateCommand;
import org.apache.solr.update.CommitUpdateCommand;
import org.apache.solr.update.DeleteUpdateCommand;
import org.apache.solr.update.MergeIndexesCommand;
import org.apache.solr.update.RollbackUpdateCommand;
import org.apache.solr.update.UpdateHandler;

import proj.zoie.api.DataConsumer.DataEvent;
import proj.zoie.api.DocIDMapper;
import proj.zoie.api.Zoie;
import proj.zoie.api.ZoieException;
import proj.zoie.api.ZoieIndexReader;
import proj.zoie.api.indexing.IndexingEventListener;
import proj.zoie.impl.indexing.ZoieSystem;

public class ZoieUpdateHandler extends UpdateHandler {
    private static Logger log = Logger.getLogger(ZoieUpdateHandler.class);

    private SolrCore _core;

    /**
     * 间隔多长时间重新打开searcher,默认值是一分钟
     */
    private static long intervalTime = 1000 * 60;// 一分钟

    private Directory indexDir;

    private Zoie<IndexReader, DocumentWithID> zoie;
    /**
     * 记录最后一个版本
     */
    private String lastVersion;

    public ZoieUpdateHandler(SolrCore core) {
        super(core);
        _core = core;
        SolrConfig solrConfig = _core.getSolrConfig();
        try {
            indexDir = _core.getDirectoryFactory().open(solrConfig.getDataDir());
        } catch (Exception e) {
            log.error("", e);
        }
        long interval = solrConfig.getInt("solrIndexSearcher.reopen.batchDelay", (int) intervalTime);
        intervalTime = interval;
        log.info("intervalTime：" + intervalTime + "    ,间隔时间重新打开SolrIndexSearcher");
        ZoieSystemHome zoieHome = ZoieSystemHome.getInstance(_core);
        if (zoieHome == null) {
            log.error("zoie home is not setup");
            throw new RuntimeException("zoie is not setup");
        }
        zoie = zoieHome.getZoieSystem();
        if (zoie == null) {
            log.error("zoie is not setup");
            throw new RuntimeException("zoie is not setup");
        }
        lastVersion = zoie.getCurrentReaderVersion();
        log.info("###########################################################################" + lastVersion);
        runOpenSolrIndexSearcher();
        log.info("初始化结束");

        ZoieSystem _zoie = (ZoieSystem) zoie;
        _zoie.addIndexingEventListener(new IndexingEventListener() {
            @Override
            public void handleIndexingEvent(IndexingEvent evt) {
                log.info(" ###########刷数据到硬盘的事件 ###########");
            }

            @Override
            public void handleUpdatedDiskVersion(String version) {
                log.info("indexingEventListener update version###########");
            }
        });
    }

    /**
     * 监控是否重新打开searcher
     */
    private void runOpenSolrIndexSearcher() {
        log.info("准备启动监控重新打开solrIndexSearcher的线程。。");
        Thread thread = new Thread() {
            public void run() {
                try {
                    log.info("休息10秒。。");
                    Thread.sleep(10000);
                } catch (Exception e) {
                    log.error("", e);
                }
                try {
                    while (true) {
                        log.info("休息" + intervalTime + "  毫秒。。");
                        Thread.sleep(intervalTime);
                        boolean hasChange = readerHasChange();
                        log.info("reader是否有修改过############################:" + hasChange);
                        if (hasChange) {
                            log.info("重新打开solrIndexSearcher");
                            try {
                                updateReader(false);
                            } catch (Exception e) {
                                log.error("", e);
                            }
                        }
                        log.info("休息");
                    }
                } catch (Exception e) {
                    log.error("", e);
                }
            }
        };
        thread.start();
        log.info("启动监听线程");
    }

    /**
     * 检测reader是否有修改，如果有修改重新new searcher
     *
     * @return
     * @throws Exception
     */
    private boolean readerHasChange() throws Exception {
        String version = zoie.getCurrentReaderVersion();
        log.info("###########################################version:" + version);
        if (version == null) {
            return false;
        }
        if ((version != null && lastVersion == null) || version.compareTo(lastVersion) > 0) {
            lastVersion = version;
            return true;
        } else {
            return false;
        }
    }

    @Override
    public int addDoc(AddUpdateCommand cmd) throws IOException {
        if (zoie == null) {
            throw new IOException("zoie is not setup");
        }
        // String id = cmd.getIndexedId(_core.getSchema());
        String uid2 = (String) cmd.solrDoc.getFieldValue(idField.getName());
        long zoieUid;
        try {
            zoieUid = Long.parseLong(uid2);
        } catch (Exception e) {
            throw new IOException("index uid must exist and of type long: " + uid2);
        }
        Document doc = cmd.doc;
        long time = System.currentTimeMillis();
        String version = String.valueOf(time);
        DataEvent<DocumentWithID> event = new DataEvent<DocumentWithID>(new DocumentWithID(zoieUid, doc), version);
        try {
            zoie.consume(Arrays.asList(event));
            return 1;
        } catch (ZoieException e) {
            log.error("", e);
            throw new IOException(e.toString());
        }
    }

    @Override
    public void close() throws IOException {
        ZoieSystemHome zoieHome = ZoieSystemHome.getInstance(_core);
        if (zoieHome != null) {
            zoieHome.shutdown();
        }
    }

    private void updateReader(boolean waitForSearcher) throws IOException {
        callPostCommitCallbacks();
        Future[] waitSearcher = null;
        if (waitForSearcher) {
            waitSearcher = new Future[1];
        }
        core.getSearcher(true, false, waitSearcher);
    }

    @Override
    public void commit(CommitUpdateCommand cmd) throws IOException {
        if (zoie != null) {
            try {
                zoie.flushEvents(10000);
            } catch (ZoieException e) {
                log.error(e.getMessage(), e);
            }
        }
        updateReader(cmd.waitSearcher);
    }

    @Override
    public void delete(DeleteUpdateCommand cmd) throws IOException {
        String id = cmd.id;
        long zoieUid;
        try {
            zoieUid = Long.parseLong(id);
        } catch (Exception e) {
            throw new IOException("index uid must exist and of type long: " + id);
        }
        if (zoie == null) {
            throw new IOException("zoie is not setup");
        }
        // String version = zoie.getVersion();
        long time = System.currentTimeMillis();
        String version = String.valueOf(time);
        DataEvent<DocumentWithID> event = new DataEvent<DocumentWithID>(new DocumentWithID(zoieUid, true), version);
        try {
            zoie.consume(Arrays.asList(event));
        } catch (ZoieException e) {
            log.error(e.getMessage(), e);
            throw new IOException(e.toString());
        }
    }

    @Override
    public void deleteByQuery(DeleteUpdateCommand cmd) throws IOException {
        log.info("删除操作");
        Query q = QueryParsing.parseQuery(cmd.query, schema);
        if (zoie == null) {
            throw new IOException("zoie is not setup");
        }
        final LongList delList = new LongArrayList();
        final int[] count = new int[1];
        count[0] = 0;
        List<ZoieIndexReader<IndexReader>> readerList = null;
        IndexSearcher searcher = null;
        try {
            readerList = zoie.getIndexReaders();
            MultiReader reader = new MultiReader(readerList.toArray(new IndexReader[readerList.size()]), false);
            searcher = new IndexSearcher(reader);
            searcher.search(q, new Collector() {
                ZoieIndexReader<IndexReader> zoieReader = null;
                int base = 0;

                @Override
                public boolean acceptsDocsOutOfOrder() {
                    return true;

                }

                @Override
                public void collect(int doc) throws IOException {
                    try {
                        // long uid = zoieReader.getUID(doc + base);
                        long uid = zoieReader.getUID(doc);
                        if (uid != DocIDMapper.NOT_FOUND) {
                            delList.add(uid);
                        }
                    } catch (Exception e) {

                        count[0] += 1;
                    }
                }

                @Override
                public void setNextReader(IndexReader reader, int base) throws IOException {
                    zoieReader = (ZoieIndexReader<IndexReader>) reader;
                    this.base = base;
                }

                @Override
                public void setScorer(Scorer scorer) throws IOException {
                }
            });

        } catch (Exception e) {
            log.error("", e);
        } finally {
            try {
                if (searcher != null) {
                    searcher.close();
                }
            } finally {
                if (readerList != null) {
                    zoie.returnIndexReaders(readerList);
                }
            }
        }

        log.info("删除条目：" + delList.size() + ",获取失败:" + count[0]);

        if (delList.size() > 0) {
            long time = System.currentTimeMillis();
            String version = String.valueOf(time);
            ArrayList<DataEvent<DocumentWithID>> eventList = new ArrayList<DataEvent<DocumentWithID>>(delList.size());
            for (long val : delList) {
                eventList.add(new DataEvent<DocumentWithID>(new DocumentWithID(val, true), version));
            }
            try {
                zoie.consume(eventList);
            } catch (ZoieException e) {
                log.error(e.getMessage(), e);
                throw new IOException(e.toString());
            }

        }
    }

    @Override
    public int mergeIndexes(MergeIndexesCommand cmd) throws IOException {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ZoieUpdateHandler.class + " doesn't support mergeIndexes.");
    }

    @Override
    public void rollback(RollbackUpdateCommand cmd) throws IOException {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ZoieUpdateHandler.class + " doesn't support rollback.");
    }

    // ///////////////////////////////////////////////////////////////////
    // SolrInfoMBean stuff: Statistics and Module Info
    // ///////////////////////////////////////////////////////////////////

    public String getName() {
        return ZoieUpdateHandler.class.getName();
    }

    public String getVersion() {
        return SolrCore.version;
    }

    public String getDescription() {
        return "Update handler builds on Zoie system";
    }

    public Category getCategory() {
        return Category.CORE;
    }

    public String getSourceId() {
        return "$Id$";
    }

    public String getSource() {
        return "$URL$";
    }

    public URL[] getDocs() {
        return null;
    }

    public NamedList getStatistics() {
        NamedList lst = new SimpleOrderedMap();
        return lst;
    }

}

2011/12/27 John Wang <john...@gmail.com>

[John Wang]

Is there a question or comment related to this post?

-John

2011/12/27 Gino Yu <yzyge...@gmail.com>

[Gino Yu]

That's I change the source..

  // long uid = zoieReader.getUID(doc + base);
   long uid = zoieReader.getUID(doc);

I change it at deleteByQuery.And it work,but I can't sure it ..

[John Wang]

Oh. I think yo uare using a very old jar.

See source at:

https://github.com/javasoze/zoie/blob/master/zoie-solr/src/main/java/proj/zoie/solr/ZoieUpdateHandler.java

So your change is definitely correct :)

We will do a release shortly, stay tuned :)

[genuine]

I use the zoie-core-3.1.0-SNAPSHOT,and I change some source,I have
change it base this version ,just like you send that source.

On Dec 30, 12:30 am, John Wang <john.w...@gmail.com> wrote:
> Oh. I think yo uare using a very old jar.
>
> See source at:
>

> https://github.com/javasoze/zoie/blob/master/zoie-solr/src/main/java/...

>
> So your change is definitely correct :)
>
> We will do a release shortly, stay tuned :)
>
> -John
>

> 2011/12/27 Gino Yu <yzygenu...@gmail.com>

>
>
>
>
>
>
>
> > That's I change the source..
>
> > // long uid = zoieReader.getUID(doc + base);
> > long uid = zoieReader.getUID(doc);
> > I change it at deleteByQuery.And it work,but I can't sure it ..
>

> > 在 2011年12月27日 下午11:34，John Wang <john.w...@gmail.com>写道：
>
> > Is there a question or comment related to this post?
>
> >> -John
>

> >> 2011/12/27 Gino Yu <yzygenu...@gmail.com>

> ...
>
> read more >>

[genuine]

I saw the soure,but it has not change :

// long uid = zoieReader.getUID(doc + base);
long uid = zoieReader.getUID(doc);

The zoieReader is refer the ZoieSegmentReader, not ZoieMultiReader,so
doc doesn't need to add base.

On Dec 30 2011, 12:30 am, John Wang <john.w...@gmail.com> wrote:
> Oh. I think yo uare using a very old jar.
>
> See source at:
>

> https://github.com/javasoze/zoie/blob/master/zoie-solr/src/main/java/...

>
> So your change is definitely correct :)
>
> We will do a release shortly, stay tuned :)
>
> -John
>

> 2011/12/27 Gino Yu <yzygenu...@gmail.com>

>
>
>
>
>
>
>
> > That's I change the source..
>
> > // long uid = zoieReader.getUID(doc + base);
> > long uid = zoieReader.getUID(doc);
> > I change it at deleteByQuery.And it work,but I can't sure it ..
>

> > 在 2011年12月27日 下午11:34，John Wang <john.w...@gmail.com>写道：
>
> > Is there a question or comment related to this post?
>
> >> -John
>

> >> 2011/12/27 Gino Yu <yzygenu...@gmail.com>

> ...
>
> read more >>

[John Wang]

You are right! Thank you.

https://github.com/javasoze/zoie/blob/master/zoie-solr/src/main/java/proj/zoie/solr/ZoieUpdateHandler.java

Will do a snapshot deploy today.

Master is upgraded with lucene 3.5.0 btw.

-John

2012/1/4 genuine <yzyge...@gmail.com>

[genuine]

I see you fix it . And when you public a release version.

On Jan 5, 11:54 pm, John Wang <john.w...@gmail.com> wrote:
> You are right! Thank you.
>

> https://github.com/javasoze/zoie/blob/master/zoie-solr/src/main/java/...

>
> Will do a snapshot deploy today.
>
> Master is upgraded with lucene 3.5.0 btw.
>
> -John
>

> 2012/1/4 genuine <yzygenu...@gmail.com>

> ...
>
> read more >>