Basic training (Red Hat Demo Platform | InstructLab RHEL VM)
26 views
Skip to first unread message
Nir Rashti
Jun 10, 2024, 9:10:29 AM6/10/24
to users, Odai Odeh, Omri Sabach
Hey,
I was using the basic Workshop VM (created by using Red Hat Demo Platform | InstructLab RHEL VM) in order to get started with InstructLab.
First of all just as a starting point I serve the basic model and chat with it (all works well btw).
Second, I tried 2 things, first was to add Q&A file that covers what is instructLab (same as was demonstrated in the instruction lab section), I ran the ilab diff command of course and saw that the new file is there as expected.
Than, ran: 'ilab generate' and right after 'ilab train --device=cuda', it was clear via the train logs file (generate by the ilab train command) that the model actually being improved, seeing in the logs the previous answer before the train and the new answer which indeed based on the generate files as expected.
The problem was that when the ilab train command finished and a new model was established, I ran the ilab serve command with the newly create model, than I was chatting with this model (I make sure few times that the chat session in indeed with the new model) but when asking the new model what is InstructLab I start to get very weird answers (not what I accept to get, as appear in the logs files generate by ilab train e.g).
Same thing happened when I try to teach ilab with some Oscar questions (as was shown in the github repo example), when I trained the model with the Oscar Q&A, the logs of the train command looks good. But after serving the newly created model and asking him basic questions like When did the 2024 Oscar took place or something similar it start to hallucinate completely (the answers appear to be worse, compare to the answer I got for the same question with the basic model, not the trained one).
Can you pinpoint what can be wrong? Should I expect those basic example to work well with instructlab newly created models (the trained models)?
Thanks you very much, Nir.
I was using the basic Workshop VM (created by using Red Hat Demo Platform | InstructLab RHEL VM) in order to get started with InstructLab.
First of all just as a starting point I serve the basic model and chat with it (all works well btw).
Second, I tried 2 things, first was to add Q&A file that covers what is instructLab (same as was demonstrated in the instruction lab section), I ran the ilab diff command of course and saw that the new file is there as expected.
Than, ran: 'ilab generate' and right after 'ilab train --device=cuda', it was clear via the train logs file (generate by the ilab train command) that the model actually being improved, seeing in the logs the previous answer before the train and the new answer which indeed based on the generate files as expected.
The problem was that when the ilab train command finished and a new model was established, I ran the ilab serve command with the newly create model, than I was chatting with this model (I make sure few times that the chat session in indeed with the new model) but when asking the new model what is InstructLab I start to get very weird answers (not what I accept to get, as appear in the logs files generate by ilab train e.g).
Same thing happened when I try to teach ilab with some Oscar questions (as was shown in the github repo example), when I trained the model with the Oscar Q&A, the logs of the train command looks good. But after serving the newly created model and asking him basic questions like When did the 2024 Oscar took place or something similar it start to hallucinate completely (the answers appear to be worse, compare to the answer I got for the same question with the basic model, not the trained one).
Can you pinpoint what can be wrong? Should I expect those basic example to work well with instructlab newly created models (the trained models)?
Thanks you very much, Nir.
Nir Rashti
Jun 16, 2024, 3:13:00 AM6/16/24
to users, Nir Rashti, Odai Odeh, Omri Sabach
Hey, any update on that?
Leslie Hawthorn
Jun 18, 2024, 10:57:01 AM6/18/24
to Nir Rashti, users, Odai Odeh, Omri Sabach
Hi Nir,
Sorry it took some time to get back in touch with you. If you're using the Red Hat Demo Platform, the best place to get help is actually from help-rhelai...@redhat.com.
If that doesn't get you the help you need, you can ping this list again or me personally and I will make sure you get the assistance you require.
Cheers,
LH
--
You received this message because you are subscribed to the Google Groups "users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@instructlab.ai.
To view this discussion on the web visit https://groups.google.com/a/instructlab.ai/d/msgid/users/c5c23735-16b6-4492-be24-b8766c63a3c9n%40instructlab.ai.
Leslie Hawthorn
Senior Manager, Vertical Community Strategy
Senior Manager, Vertical Community Strategy
Open Source Program Office - Office of the CTO
She / Her / Hers
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
Nir Rashti
Jun 20, 2024, 7:56:36 AM6/20/24
to Leslie Hawthorn, users, Odai Odeh, Omri Sabach
Hey @Leslie Hawthorn, thank you for your email.
Not sure you completely understand the issue we are experiencing, the VM we are using is great (so I don't see why we need to pinpoint the issue to rhelai).
The problem seems to be with the InstructLab infrastructure so after you train the model with some basic Q&A files (one covers what is InstructLab, one covers the oscar ceremony) so after having a newly created model. The new model completely hallucinates with the same questions we train him with.
So just wonder and want to understand if it is some known issue or something else.
Thanks,
Nir.
Not sure you completely understand the issue we are experiencing, the VM we are using is great (so I don't see why we need to pinpoint the issue to rhelai).
The problem seems to be with the InstructLab infrastructure so after you train the model with some basic Q&A files (one covers what is InstructLab, one covers the oscar ceremony) so after having a newly created model. The new model completely hallucinates with the same questions we train him with.
So just wonder and want to understand if it is some known issue or something else.
Thanks,
Nir.
Leslie Hawthorn
Jun 20, 2024, 8:08:21 AM6/20/24
to Nir Rashti, users, Odai Odeh, Omri Sabach
Hi Nir,
On Thu, Jun 20, 2024 at 1:56 PM Nir Rashti <nra...@redhat.com> wrote:
Hey @Leslie Hawthorn, thank you for your email.
Not sure you completely understand the issue we are experiencing,
You're probably right. :D
the VM we are using is great (so I don't see why we need to pinpoint the issue to rhelai).
The problem seems to be with the InstructLab infrastructure so after you train the model with some basic Q&A files (one covers what is InstructLab, one covers the oscar ceremony) so after having a newly created model. The new model completely hallucinates with the same questions we train him with.
I think your issue may be related to https://github.com/instructlab/instructlab/issues/1305. Please take a look - if it does map to your problems, please update issue with any useful info for the Maintainers team. If it is not related, please open another issue in the InstructLab/InstructLab issues queue.
If you do find your problem is related to Issue #1305, you will be happy to know fixes are planned in the next release, which should be out in the next few days.
HTH, if not, we figure it out,
LH
Nir Rashti
Jun 20, 2024, 8:46:27 AM6/20/24
to Leslie Hawthorn, users, Odai Odeh, Omri Sabach, Eric Zhou
Alright, thanks for your reply.
Indeed 'https://github.com/instructlab/instructlab/issues/1305.' cover the same major problem as we did, my only concern is that even when we created a new model on a VM with GPU (we invoke the ilab generate as well) even when we trained it with 5 basic questions such as what is instructLab it wasn't able to respond well. So I feel like our case is the most basic and it still didn't work for us. (not sure what the user in case 1305 exactly tried to teach the model).
Anyway I will wait for the next release and will test it again with some basic samples.
Appreciate,
Nir.
Indeed 'https://github.com/instructlab/instructlab/issues/1305.' cover the same major problem as we did, my only concern is that even when we created a new model on a VM with GPU (we invoke the ilab generate as well) even when we trained it with 5 basic questions such as what is instructLab it wasn't able to respond well. So I feel like our case is the most basic and it still didn't work for us. (not sure what the user in case 1305 exactly tried to teach the model).
Anyway I will wait for the next release and will test it again with some basic samples.
Appreciate,
Nir.
Leslie Hawthorn
Jun 20, 2024, 8:51:37 AM6/20/24
to Nir Rashti, users, Odai Odeh, Omri Sabach, Eric Zhou
Sounds good to me. We'll announce latest release in the project Slack and the announcement mailing lists. We released v 0.17.1 on Tuesday but I still see a user reporting issues with that version in issue 1305, so we clearly still need to get a fix in.
For Slack workspace info in case it is helpful:
Cheers,
LH
To view this discussion on the web visit https://groups.google.com/a/instructlab.ai/d/msgid/users/CAPaJaAiMXsxbewMSydBaLzhS_M2LabHOcXArr8t6yLgOz25R0g%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages