Already exist in catalog

[Joshua Maryniewski]

Hello,

I had a package in the testing and unstable manifest and catalog.  I unlocked the package so I could switch it to include stable.  I can get the package to be in the stable catalog and manifest now but it still has it listed in the testing and unstable catalog and manifest even though those options aren't checked in Package Admin.  Nor will it let me to try and re-add them to unstable and testing in package admin.

I hope that makes sense.

[Justin McWilliams]

Have you checked again?  I'm curious if this is merely a caching / race condition, or if they're still wrongfully part of unstable and testing.  

Are there any associated error logs on App Engine?  Catalogs and manifests are created in a background job, so if they fail they should repeatedly retry indefinitely, and logs should make this very clear.

What happens when you try to re-add them to unstable/testing?  You probably can't because they're locked?  So, have you tried this:  unlock -> add to unstable/testing -> unlock -> add to stable?  This is essentially what you did initially, but I'm curious if you ever tried again (if you can reproduce this, please look at logs immediately following the unexpected case).

--
You received this message because you are subscribed to the Google Groups "Simian Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simian-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joshua Maryniewski]

I check and it appears all the cron jobs worked.  I looked at the error logs.  This is the closest error I can find.

I did try unlocking it and adding it to the catalogs again this morning.

Is it possible to just delete catalogs and manifests and recreate them?

-Josh

[Justin McWilliams]

The logs you pasted are quite unrelated; that's a cron job to generate install duration estimates, not a background job to regenerate catalogs or manifests.

It's possible to delete the catalogs (if you're comfortable mutating Datastore directly), and they should be recreated when you make any subsequent modification (add or remove pkg) to them.  However, that's quite a hammer.  I'd suggest first deleting and reuploading the package (even that is a bit of a hammer, but easier and safer).

How/where are you confirming the removed packages are still in the catalog and manifest?  Are you confirming on a client where there may be caching issues, or on the server?  Have you manually visited /catalogs/unstable (and /manifests/unstable) and /catalogs/testing (and /manifests/testing) on your server?  

- Justin

[Joshua Maryniewski]

I have tried deleting the package completely and it stayed in my testing and unstable manifest/catalog.  To verify its still there, I am going to my appspot page and I can see it under manifests on the left side.

[Justin McWilliams]

And there are still no failures in the logs?

Can you try clearing memcache to see if this is still an issue (cloud.google.com -> My console -> your project -> Compute -> App Engine -> Memcache -> Flush Cache)?

If you *add* a new package, does it show up in the manifest and catalog, while the old one remains?  Does it not get added?  Does it get added and only then does the deleted one get removed?

On Wed, Oct 7, 2015 at 10:03 AM, Joshua Maryniewski <joshua.ma...@gmail.com> wrote:

I have tried deleting the package completely and it stayed in my testing and unstable manifest/catalog.  To verify its still there, I am going to my appspot page and I can see it under manifests on the left side.

[Joshua Maryniewski]

I tried the memfllush no change.  I have a few errors in the appengine page for going over the 128mb limit.  Thats all I see.  

To make matters worse, I unlocked the package.  Unchecked both stable options, save it.  Now I can't add it back to stable.  

Is there a place specifically I should be looking for an error?

[Justin McWilliams]

Please answer my other questions (re adding a new package).  It's really hard to debug this without that information.

"Now I can't add it back to stable." --- any error in the UI or logs?  What happens when you try?  What if you give the package a new name first?

"Is there a place specifically I should be looking for an error?" -- either the UI, or request logs in the Cloud Console.

[Joshua Maryniewski]

Sorry, I thought I had the error message in a previous post...

This is from the UI:

[Joshua Maryniewski]

I think it would work w/ another name, but the old one is still listed in the the manifests according to the UI.

[Justin McWilliams]

This is helpful.  Answers to my original questions about adding new packages will be even more helpful:

- If you *add* a new (aka different) package, does it show up in the manifest and catalog (manually hitting the /manifests/<track> and /catalogs/<track> URLs), while the old one still remains?  Does it not get added?  Does it get added and only then does the deleted one get removed?

Succintly:  does adding a new package work, and if so, does it have any affect on the deleted package?

[Joshua Maryniewski]

I don't completely understand what you want me to do.  Do you mean for me to add a package that has the same name or a completely unrelated package?

[Justin McWilliams]

On Mon, Oct 12, 2015 at 11:22 AM, Joshua Maryniewski <joshua.ma...@gmail.com> wrote:

I don't completely understand what you want me to do.  Do you mean for me to add a package that has the same name or a completely unrelated package?

Add a completely unrelated package.  I want to see if 1) is added to the catalog/manifests as expected and if 2) that has any effect on the previously removed package being present in the catalog/manifest.

[Joshua Maryniewski]

I uploaded Java as a new package.  I saved it to the unstable and testing catalogs and manifests.  After waiting a little bit, simian was gone from the unstable and testing catalog/manifests.  I then removed java from all catalogs/manifests.  Java is now still listed in Testing and Unstable catalog/manifests.   

Basically I have the same issue, just now its Java instead of simian.   Let me know if thats not enough info for you.

-Josh

[Justin McWilliams]

Yeah, it sounds like removing packages doesn't take affect until a new package is added.  With that, I know what to look for, so I'll dig in a bit more.

[Justin McWilliams]

I've identified one case which the code does not currently cover.  If there are NO package in a given catalog, it will not be rebuilt: https://github.com/google/simian/blob/master/src/simian/mac/models/munki.py#L128

That is, if you remove the last remaining package from a catalog, the catalog will still contain that package until you add something else to it.

This seems very similar to what you've described.  Can you confirm this is only reproducible when you're removing the last remaining package?

[Justin McWilliams]

Please push this patch and try again: https://github.com/google/simian/commit/909bd22a254a62ebe7aea9cbc121203ede957151

(note, you may have a lingering lock in Admin Tools -> Lock Admin to clear)

[Joshua Maryniewski]

It was not the last package, there was another still left.  I had Flash in there at the time of the issue w/ the simian.pkg.

[Joshua Maryniewski]

I did the "git pull" and re-uploaded it.  

[Justin McWilliams]

On Fri, Oct 16, 2015 at 10:41 AM, Joshua Maryniewski <joshua.ma...@gmail.com> wrote:

I did the "git pull" and re-uploaded it.  

And the problem persists?

[Michael Holt]

I don't have the latest version yet but I ran into this issue just now when removing tje old version of Excel2016 and adding the newer version to the catalog.  Here were my steps, which i've done on about 10 other updates this morning without issues:

1) Open Old Package

2) Unselect Unstable, Testing, Stable Catalogs and Manifests

3) Save Package

4) Open New Package

5) Select Unstable, Testing, Stable Catalogs and Manifests

6) Save Package

When I did this, I encountered the "unable to add to unstable catalog" error, but it was sucessfully added to Testing and Stable.

How I fixed it was to go into datastore viewer and delete the unstable catalog, then add Excel2016 to the Unstable Catalog.  This worked and regenerated the unstable catalog fully

[Joshua Maryniewski]

I am not having that issue currently, so I think we're good.