Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
updatemeta: correctly handle empty metadata
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  9 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Jun Fang  
View profile  
 More options Jul 23 2012, 5:20 pm
From: Jun Fang <junf...@fb.com>
Date: Mon, 23 Jul 2012 14:20:20 -0700
Local: Mon, Jul 23 2012 5:20 pm
Subject: [PATCH] updatemeta: correctly handle empty metadata
# HG changeset patch
# User Jun Fang <junf...@fb.com>
# Date 1343077239 25200
# Node ID 3a31f765153dbbb599c042feab5b644935703378
# Parent  761a8713450164a5754adff55223d6f08e37cfab
updatemeta: correctly handle empty metadata

When the repo metadata is empty (just created/cloned but not populated yet),
there is no lastpulled file or the revmap doesn't have a last entry with hash.
Currently "hg svn updatemeta" would crash due to unexpected exception.

See issue reported at https://bitbucket.org/durin42/hgsubversion/issue/356/updatemeta-crash...

The fix is to check the existence of lastpulled file and the hash entry in
revmap, as part of "hg svn updatemeta" command. When they are not present,
do a rebuildmetadata.

diff --git a/hgsubversion/svncommands.py b/hgsubversion/svncommands.py
--- a/hgsubversion/svncommands.py
+++ b/hgsubversion/svncommands.py
@@ -64,12 +64,19 @@
     if partial:
         try:
             youngestpath = os.path.join(svnmetadir, 'lastpulled')
-            youngest = int(util.load_string(youngestpath).strip())
-            sofar = list(maps.RevMap.readmapfile(repo))
-            lasthash = sofar[-1].split(' ', 2)[1]
-            startrev = repo[lasthash].rev() + 1
-            branchinfo = pickle.load(open(os.path.join(svnmetadir,
-                                                       'branch_info')))
+            foundpartialinfo = False
+            if os.path.exists(youngestpath):
+                youngest = int(util.load_string(youngestpath).strip())
+                sofar = list(maps.RevMap.readmapfile(repo))
+                if sofar and len(sofar[-1].split(' ', 2)) > 1:
+                    lasthash = sofar[-1].split(' ', 2)[1]
+                    startrev = repo[lasthash].rev() + 1
+                    branchinfo = pickle.load(open(os.path.join(svnmetadir,
+                                                           'branch_info')))
+                    foundpartialinfo = True
+            if not foundpartialinfo:
+                ui.status('missing some metadata -- doing a full rebuild\n')
+                partial = False
         except IOError, err:
             if err.errno != errno.ENOENT:
                 raise
diff --git a/tests/test_rebuildmeta.py b/tests/test_rebuildmeta.py
--- a/tests/test_rebuildmeta.py
+++ b/tests/test_rebuildmeta.py
@@ -45,6 +45,20 @@
     origchildren = getattr(context.changectx, 'children')
     extensions.wrapfunction(context.changectx, 'children', failfn)

+    # test updatemeta on an empty repo
+    try:
+        svncommands.updatemeta(u, dest,
+                                args=[test_util.fileurl(repo_path +
+                                                        subdir), ])
+    finally:
+        # remove the wrapper
+        context.changectx.children = origchildren
+
+    self._run_assertions(name, stupid, single, src, dest, u)
+
+    # insert a wrapper that prevents calling changectx.children()
+    extensions.wrapfunction(context.changectx, 'children', failfn)
+
     try:
         svncommands.rebuildmeta(u, dest,
                                 args=[test_util.fileurl(repo_path +


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Augie Fackler  
View profile  
 More options Jul 26 2012, 7:06 pm
From: Augie Fackler <r...@durin42.com>
Date: Thu, 26 Jul 2012 18:06:47 -0500
Local: Thurs, Jul 26 2012 7:06 pm
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata

On Jul 23, 2012, at 4:20 PM, Jun Fang wrote:

> # HG changeset patch
> # User Jun Fang <junf...@fb.com>
> # Date 1343077239 25200
> # Node ID 3a31f765153dbbb599c042feab5b644935703378
> # Parent  761a8713450164a5754adff55223d6f08e37cfab
> updatemeta: correctly handle empty metadata

Implementation looks reasonable, tests are still concerning me.

> +    # test updatemeta on an empty repo
> +    try:
> +        svncommands.updatemeta(u, dest,
> +                                args=[test_util.fileurl(repo_path +
> +                                                        subdir), ])
> +    finally:
> +        # remove the wrapper
> +        context.changectx.children = origchildren
> +
> +    self._run_assertions(name, stupid, single, src, dest, u)
> +
> +    # insert a wrapper that prevents calling changectx.children()
> +    extensions.wrapfunction(context.changectx, 'children', failfn)
> +

Sorry I missed this before: doesn't this rebuild dest, so that the following test assertions are vacuously succeeding? I think it might be best to have some duplicated test method code here so these two cases are obviously tested and working independently.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Augie Fackler  
View profile  
 More options Aug 6 2012, 1:48 pm
From: Augie Fackler <r...@durin42.com>
Date: Mon, 6 Aug 2012 12:48:26 -0500
Local: Mon, Aug 6 2012 1:48 pm
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata
Can you start a new thread using 'hg email' so the patch is inline?

Thanks,
Augie


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jun Fang  
View profile  
 More options Aug 6 2012, 1:46 pm
From: Jun Fang <junf...@fb.com>
Date: Mon, 6 Aug 2012 17:46:31 +0000
Local: Mon, Aug 6 2012 1:46 pm
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata

Hi Augie,

I created a new unit test, test_updatemeta.py, to test the updatemeta
command specifically.

Here is the new change.

Please take a look.

Thanks,
Jun

On 7/26/12 4:06 PM, "Augie Fackler" <r...@durin42.com> wrote:

  Fix_issue_356.txt
7K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jun Fang  
View profile  
 More options Aug 6 2012, 1:50 pm
From: Jun Fang <junf...@fb.com>
Date: Mon, 06 Aug 2012 10:50:59 -0700
Local: Mon, Aug 6 2012 1:50 pm
Subject: [PATCH] updatemeta: correctly handle empty metadata
# HG changeset patch
# User Jun Fang <junf...@fb.com>
# Date 1344274551 25200
# Node ID 6b7ac659c85574bb819650848af501c95b825baa
# Parent  8faa91951bb10b580cbfc2ad065cd26aae47c618
updatemeta: correctly handle empty metadata

When the repo metadata is empty (just created/cloned but not populated yet),
there is no lastpulled file or the revmap doesn't have a last entry with hash.
Currently "hg svn updatemeta" would crash due to unexpected exception.

See issue reported at https://bitbucket.org/durin42/hgsubversion/issue/356/updatemeta-crash...

The fix is to check the existence of lastpulled file and the hash entry in
revmap, as part of "hg svn updatemeta" command. When they are not present,
do a rebuildmetadata.

Also added a new unit test, test_updatemeta.py.

diff --git a/hgsubversion/svncommands.py b/hgsubversion/svncommands.py
--- a/hgsubversion/svncommands.py
+++ b/hgsubversion/svncommands.py
@@ -64,12 +64,19 @@
     if partial:
         try:
             youngestpath = os.path.join(svnmetadir, 'lastpulled')
-            youngest = int(util.load_string(youngestpath).strip())
-            sofar = list(maps.RevMap.readmapfile(repo))
-            lasthash = sofar[-1].split(' ', 2)[1]
-            startrev = repo[lasthash].rev() + 1
-            branchinfo = pickle.load(open(os.path.join(svnmetadir,
-                                                       'branch_info')))
+            foundpartialinfo = False
+            if os.path.exists(youngestpath):
+                youngest = int(util.load_string(youngestpath).strip())
+                sofar = list(maps.RevMap.readmapfile(repo))
+                if sofar and len(sofar[-1].split(' ', 2)) > 1:
+                    lasthash = sofar[-1].split(' ', 2)[1]
+                    startrev = repo[lasthash].rev() + 1
+                    branchinfo = pickle.load(open(os.path.join(svnmetadir,
+                                                           'branch_info')))
+                    foundpartialinfo = True
+            if not foundpartialinfo:
+                ui.status('missing some metadata -- doing a full rebuild\n')
+                partial = False
         except IOError, err:
             if err.errno != errno.ENOENT:
                 raise
diff --git a/tests/run.py b/tests/run.py
--- a/tests/run.py
+++ b/tests/run.py
@@ -32,6 +32,7 @@
     import test_template_keywords
     import test_utility_commands
     import test_unaffected_core
+    import test_updatemeta
     import test_urls

     sys.path.append(os.path.dirname(__file__))
diff --git a/tests/test_updatemeta.py b/tests/test_updatemeta.py
new file mode 100644
--- /dev/null
+++ b/tests/test_updatemeta.py
@@ -0,0 +1,81 @@
+import test_util
+
+import os
+import pickle
+import unittest
+import test_rebuildmeta
+
+from mercurial import context
+from mercurial import extensions
+from mercurial import hg
+from mercurial import ui
+
+from hgsubversion import svncommands
+from hgsubversion import svnmeta
+
+
+
+def _do_case(self, name, stupid, single):
+    subdir = test_util.subdir.get(name, '')
+    layout = 'auto'
+    if single:
+        layout = 'single'
+    repo, repo_path = self.load_and_fetch(name, subdir=subdir, stupid=stupid,
+                                          layout=layout)
+    assert len(self.repo) > 0
+    wc2_path = self.wc_path + '_clone'
+    u = ui.ui()
+    src, dest = test_util.hgclone(u, self.wc_path, wc2_path, update=False)
+    src = getattr(src, 'local', lambda: src)()
+    dest = getattr(dest, 'local', lambda: dest)()
+
+    # insert a wrapper that prevents calling changectx.children()
+    def failfn(orig, ctx):
+        self.fail('calling %s is forbidden; it can cause massive slowdowns '
+                  'when rebuilding large repositories' % orig)
+
+    origchildren = getattr(context.changectx, 'children')
+    extensions.wrapfunction(context.changectx, 'children', failfn)
+
+    # test updatemeta on an empty repo
+    try:
+        svncommands.updatemeta(u, dest,
+                                args=[test_util.fileurl(repo_path +
+                                                        subdir), ])
+    finally:
+        # remove the wrapper
+        context.changectx.children = origchildren
+
+    self._run_assertions(name, stupid, single, src, dest, u)
+
+
+def _run_assertions(self, name, stupid, single, src, dest, u):
+    test_rebuildmeta._run_assertions(self, name, stupid, single, src, dest, u)
+
+
+skip = set([
+    'project_root_not_repo_root.svndump',
+    'corrupt.svndump',
+])
+
+attrs = {'_do_case': _do_case,
+         '_run_assertions': _run_assertions,
+         }
+for case in [f for f in os.listdir(test_util.FIXTURES) if f.endswith('.svndump')]:
+    # this fixture results in an empty repository, don't use it
+    if case in skip:
+        continue
+    bname = 'test_' + case[:-len('.svndump')]
+    attrs[bname] = test_rebuildmeta.buildmethod(case, bname, False, False)
+    name = bname + '_stupid'
+    attrs[name] = test_rebuildmeta.buildmethod(case, name, True, False)
+    name = bname + '_single'
+    attrs[name] = test_rebuildmeta.buildmethod(case, name, False, True)
+
+UpdateMetaTests = type('UpdateMetaTests', (test_util.TestBase,), attrs)
+
+
+def suite():
+    all_tests = [unittest.TestLoader().loadTestsFromTestCase(UpdateMetaTests),
+          ]
+    return unittest.TestSuite(all_tests)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jun Fang  
View profile  
 More options Aug 14 2012, 4:56 pm
From: Jun Fang <junf...@fb.com>
Date: Tue, 14 Aug 2012 20:56:05 +0000
Local: Tues, Aug 14 2012 4:56 pm
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata
Hi Augie,

Did you get a chance to take a look? Please let me know if there is still
issue with the unit test.

Thanks,
Jun

On 8/6/12 10:50 AM, "Jun Fang" <junf...@fb.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Augie Fackler  
View profile  
 More options Aug 14 2012, 5:15 pm
From: Augie Fackler <r...@durin42.com>
Date: Tue, 14 Aug 2012 16:15:29 -0500
Local: Tues, Aug 14 2012 5:15 pm
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata
backlogged, should have time friday. sorry about the wait.

On Aug 14, 2012, at 3:56 PM, Jun Fang wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Augie Fackler  
View profile  
 More options Aug 28 2012, 9:09 am
From: Augie Fackler <r...@durin42.com>
Date: Tue, 28 Aug 2012 08:09:26 -0500
Local: Tues, Aug 28 2012 9:09 am
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata
queued, thanks

On Aug 6, 2012, at 12:50 PM, Jun Fang wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jun Fang  
View profile  
 More options Aug 28 2012, 12:08 pm
From: Jun Fang <junf...@fb.com>
Date: Tue, 28 Aug 2012 16:08:36 +0000
Local: Tues, Aug 28 2012 12:08 pm
Subject: Re: [PATCH] updatemeta: correctly handle empty metadata
Thanks!

On 8/28/12 6:09 AM, "Augie Fackler" <r...@durin42.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »