new docs require re-ingesting for langchain4j?

[William Burke]

I have a set of new RAG docs.  It seems if I only ingest these new docs, the old vectors are overridden and embedding searches only go off of the new RAG docs.

Do you have to re-ingest older docs too when you bring in a new additional set of docs?  Or is there some special API/config I'm missing here?

Thanks,

Bill

--

Bill Burke

Red Hat

[Georgios Andrianakis]

cc @Dmytro Liubarskyi 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/quarkus-dev/CAL%3DE%3DjSibbKM1py8PrTB7%2Bq0pyZiDBF8u8EPnY1Rm76HB-OBQQ%40mail.gmail.com.

--

Georgios Andrianakis

Independent Contractor

[Dmytro Liubarskyi]

Hi,

In general, there should be no override, but I need a bit more info to answer for sure.

Which embedding store are you using? What is your config? A code snippet would be great.

[Dmytro Liubarskyi]

Hi,

In general, there should be no override, but I need a bit more info to answer for sure.

Which embedding store are you using? What is your config? A code snippet would be great.

[William Burke]

Code is here.  If I call EquipmentDB.ingest more than once the old ingested things are gone.  Is it because I'm creating an ingester every time maybe?

https://github.com/patriot1burke/baldurs-forge/blob/main/armory/src/main/java/org/baldurs/forge/toolbox/EquipmentDB.java#L181

config isn't anything special

https://github.com/patriot1burke/baldurs-forge/blob/main/armory/src/main/resources/application.properties

To view this discussion visit https://groups.google.com/d/msgid/quarkus-dev/CAAM%3DMjX5gq%2BgoTGfkrF7HrmeB3tnO2OPwqx%3D8PFOSvtPvUmDAQ%40mail.gmail.com.

[Dmytro Liubarskyi]

Hi, I just did a simple test of ingesting into PgVector multiple times, and it works fine.

Could it be because you unintentionally call embeddingStore.removeAll() here in-between ingestions?

[Emmanuel Bernard]

You call load() when the app starts https://github.com/patriot1burke/baldurs-forge/blob/main/armory/src/main/java/org/baldurs/forge/toolbox/EquipmentDB.java#L70C4-L73C6

And load removes all the vector data via removeAll()

https://github.com/patriot1burke/baldurs-forge/blob/main/armory/src/main/java/org/baldurs/forge/toolbox/EquipmentDB.java#L175

To view this discussion visit https://groups.google.com/d/msgid/quarkus-dev/CAL%3DE%3DjQ4Dme3PXpBC4-%2BxU7tCBpwsT%2Bk4VtsWS%3DRu-B0ARS1Nw%40mail.gmail.com.

[William Burke]

yeah, that was only on startup.  I had another method that would upload and add items to the db and doclist during runtime.  I swear I 100% thought that I wasn't calling load(), but I can't replicate the issue. Probably a combination of bad eyes and human hallucinations.

To view this discussion visit https://groups.google.com/d/msgid/quarkus-dev/CANYWk7P2s6v24pmvMuDd%3Dyn55JVdTJCFcp3fsP8qx6doDQbGUQ%40mail.gmail.com.

[William Burke]

I wasted your time.  Apologies.  That must have been it.

To view this discussion visit https://groups.google.com/d/msgid/quarkus-dev/8278de95-b889-41a0-84b1-f1b439c30808n%40googlegroups.com.