Repositories not accessible

[Andy Forceno]

I recently reinstalled Rdiffweb from the git repo, following a reinstall of my OS. I found that the Rdiffweb interface has been tweaked, and now it's not displaying my repositories correctly. I can only access 2 of 9 repos, and they are not organized correctly on the page. See the attached file for an example of the issue.

I'm seeing a number of errors in the log, but I'm not sure if they are related:

RepoObject(db, u'admin', u'/Poincare/root')]
Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 117, in job_run
    self._remove_older(user, repo, keepdays)
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 133, in _remove_older
    d = d.days + keepdays
TypeError: unsupported operand type(s) for +: 'int' and 'unicode'

And also this: 

[2017-10-04 20:07:29,294][WARNING][192.168.1.5][none][CP Server Thread-6][rdiffweb.librdiff] fail to parse date ['2017-02-15T00\xef\x80\xa200\xef\x80\xa202-05\xef\x80\xa200']
Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/rdiffweb/librdiff.py", line 681, in _extract_date
    return rdw_helpers.rdwTime(date_string.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 13: ordinal not in range(128)
[2017-10-04 20:07:29,295][WARNING][192.168.1.5][none][CP Server Thread-6][rdiffweb.librdiff] fail to parse date ['2016-12-12T06\xef\x80\xa234\xef\x80\xa207-05\xef\x80\xa200']

Should I have installed from the zip on the website, rather than Github? 

[Andy Forceno]

I should have mentioned that I should be seeing two folders under each of the repositories, for the home and root directories of each machine. Instead, I have just the two folders, neither of which is displayed under the correct repo.

[Patrik Dufresne]

Hi Andy,

I will have a look to your issue. Installing from github or from pypi is the same as long as you install from master branch.

For the display issue, I need some info to reproduce. May you provide something similar where each leaf is a rdiff-backup repo ?
- /var/backups/rdiffweb
-- my-laptop
-- servers
--- Malta
--- Ranculos

Thanks

--
You received this message because you are subscribed to the Google Groups "rdiffweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[t4exanadu]

H Patrik,

Let me know if I misunderstand your request. The folder structure of
the repo looks like this:
- /media/backups
-- Aurorabox
--- root
--- home
-- Poincare
--- root
--- home
-- Ariadne
--- root
-- Nyx
--- root

Individual machines are each backed up to /media/backups/<hostname>/

One thing that is different about my setup now: /media/backup/ is a
symlink to /media/external-hdd/backups/. Not sure if that is
significant. I would think that shouldn't matter.

[Patrik Dufresne]

I've test a similar configuration here and it is shown as expected.

Symlink to /media/external-hdd/backups/ may cause few problems. Rdiffweb is not following symlink for security reason.

May you try to refresh your repositories. This options is available in user preferences.

Than look in the logs (/var/log/rdiffweb.log) for lines similar to the following:

[2017-10-08 12:25:44,803][DEBUG  ][127.0.0.1][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] find repos for UserObject[admin]

[2017-10-08 12:25:44,806][DEBUG  ][127.0.0.1][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] set UserObject[admin] repos: [u'/MainFolder/SubFolder/home', u'/MainFolder/SubFolder/root', u'/data', u'/MyLaptop/D', u'/MyLaptop/C', u'/broker-repo', u'/Servers/Kase', u'/Servers/Malta', u'/testcases'] 

Most likely yours will look like this:

[2017-10-08 12:27:09,797][DEBUG  ][127.0.0.1][none][CP Server Thread-10][rdiffweb.rdw_spider_repos] set UserObject[admin] repos: [] 

To resolve the issue, you will need to change the path from /media/backups to /media/external-hdd/backups/

--

Patrik Dufresne

[Andy]

I changed it to the full path, and refreshed the repositories. It still looks the same, though. I tried it in another browser, to rule that out, but it looks the same. 

In rdiffweb.log, I see that same error repeated for all repositories. I do not see any lines indicating it is finding the repos, not within the last few days. Nothing seems to be logged when I refresh the repositories. 

[2017-10-06 23:04:33,820][ERROR  ][127.0.0.1][none][RemoveOlderPlugin][rdiffweb.plugins.remove_older] fail to remove older for user [<rdiffweb.user.UserObject object at 0xb5e44fec>] repo [RepoObject(db, u'admin', u'/Ariadne/root')]

Traceback (most recent call last):

  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 117, in job_run

    self._remove_older(user, repo, keepdays)

  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 133, in _remove_older

    d = d.days + keepdays

TypeError: unsupported operand type(s) for +: 'int' and 'unicode'

--
You received this message because you are subscribed to a topic in the Google Groups "rdiffweb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdiffweb/NgVBjP4gcGU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

[Patrik Dufresne]

Hello Andy,

This line: 

fail to remove older for user [<rdiffweb.user.UserObject object at 0xb5e44fec>] repo [RepoObject(db, u'admin', u'/Ariadne/root')]

Is related to the exception you first mention:

RepoObject(db, u'admin', u'/Poincare/root')]

Traceback (most recent call last):

  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 117, in job_run

    self._remove_older(user, repo, keepdays)

  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 133, in _remove_older

    d = d.days + keepdays

TypeError: unsupported operand type(s) for +: 'int' and 'unicode'

I manage to reproduce the problem somehow and made a fix in develop branch. But it's not related to your repositories not showing off.

To have better logging, I recommend you to change the loglevel to debug in the configuration file /etc/rdiffweb/rdw.conf and restart.

Then you should have more information regarding the refresh. If not, something is wrong with your installation or configuration.

--

Patrik Dufresne

To unsubscribe from this group and all its topics, send an email to rdiffweb+unsubscribe@googlegroups.com.

[Andy]

With log level set to DEBUG, I see the following when I access rdiffweb in my browser or refresh the repositories:

[2017-10-08 16:25:38,703][DEBUG  ][192.168.1.5][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] find repos for UserObject[admin]

[2017-10-08 16:25:39,000][DEBUG  ][192.168.1.5][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] set UserObject[admin] repos: [u'/Aurorabox/root', u'/Aurorabox/home', u'/Poincare/root', u'/Poincare/home', u'/Ariadne/root', u'/Fractal/root', u'/Fractal/home'] 

[2017-10-08 16:25:41,057][INFO   ][192.168.1.5][none][CP Server Thread-13][rdiffweb.filter_authentication] no special path, running do_check

[2017-10-08 16:25:41,058][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.filter_authentication] setting request.login to UserObject[admin]

[2017-10-08 16:25:41,059][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.page_locations] browsing locations

[2017-10-08 16:25:41,064][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] pid [15978] does not exists

[2017-10-08 16:25:41,065][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] get backup dates for ['/media/big-linux/backups/Poincare/root']

[2017-10-08 16:25:41,134][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] pid [6464] does not exists

[2017-10-08 16:25:41,134][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] get backup dates for ['/media/big-linux/backups/Poincare/home']

[2017-10-08 16:25:41,137][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] pid [20955] does not exists

[2017-10-08 16:25:41,137][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] get backup dates for ['/media/big-linux/backups/Ariadne/root']

That repeats for each of my 9 repos. 

To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

[Patrik Dufresne]

Hello Andy,

I have a hard time to understand what is wrong. According to the log everything seams to be working as expected. For the sake of testing, I will try to mimic as close as possible your current setup. I will keep you posted if I found anything.

Question:

May you browse to ...?

http://localhost:8080/browse/Aurorabox/root/

http://localhost:8080/browse/Aurorabox/home/

http://localhost:8080/browse/Poincare/root/

http://localhost:8080/browse/Poincare/home/

http://localhost:8080/browse/Ariadne/root/

http://localhost:8080/browse/Fractal/root/

http://localhost:8080/browse/Fractal/home/

--

Patrik Dufresne

To unsubscribe from this group and all its topics, send an email to rdiffweb+unsubscribe@googlegroups.com.

[Andy]

Using a tunnel to localhost, I can browse to all the repositories.

To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

[Kristian Pedersen]

Hi Patrick,

Im wondering if this issue was ever identified/resolved? We just reinstalled our backup server (for a clean upgrade from Debian 6 to Debian 9.3), installed rdiffweb using pypi as mentioned in the documentation, and we are facing what appears to be this same issue.

Our repositories are organized as:
/data/dumps (Root directory)
/data/dumps/<server>/<partition>/ (rdiff-repository)

We have 62 servers with a total of 81 partitions/repositories. The repositories are typically called "boot", "root", "data" and a few "mnt1" / "mnt2", no special characters. Rdiffweb will pick up all 62 servers but only 5 of the 81 repositories, and the repositories are not displayed under the server where they belong (attaching screenshot):



I have enabled debug, and when browsing the list of repositories i see consistent "pid [xxxx] does not exist" and "get backup dates for [yyyy]" for all 81 repositories, and when updating the list of repositories, all 81 repositories are mentioned in the "repos: [aaaa, bbbb, cccc ...]" log lines. If i construct the url, i can access all repositories just fine, so it appears its only the listing of repositories that has an issue.

Looking at it again, I can actually see only one boot, one root, one data and one mnt1 /mnt2 repository is displayed, so maybe the issues is related to having repositories with same name in sub folders, ie:

/data/dumps/srv1/root
/data/dumps/srv2/root

Kind regards,
Kristian

--

Patrik Dufresne

To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

[Patrik Dufresne]

Hello Kristian,

We didn't find a solution to Andy's issue. Was not able to reproduce it on my side. There is something very odd with the display obviously.

Are you using a nginx reverse proxy to access rdiffweb ? If so, may you try to query the page directly without nginx reverse proxy ?

In any case, would it be possible to query the page using curl and dump it into a file ? Would love to see the html generated.

Thanks

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

130 rue Doris

St-Colomban, QC J5K 1T9

You received this message because you are subscribed to the Google Groups "rdiffweb" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.

[Kristian Pedersen]

Hi Patrick,

Thanks for getting back to me so fast :)

I am accessing rdiffweb through the build-in http, so no nginx/apache reverse proxy going on.

I will reply to you in private with a copy of the generated html obtained with curl, I hope this is okay:

curl -c cookie.txt -d "login=admin" -d "password=somepassword" http://backup.xxx:8080/login/
curl -b cookie.txt http://backup.xxx:8080/ -o out.html

Kind regards,
Kristian

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+u...@googlegroups.com.

[Kristian Pedersen]

Update:

I am able to reproduce this issue using two very simple rdiff-backup repositories using same directory structure.

Using commands:
rdiff-backup --preserve-numerical-ids --exclude-other-filesystems --exclude-sockets --tempdir /tmp /root/kptmp/src/srv-a.example.net/ /root/kptmp/dumps/srv-a.example.net/root/
rdiff-backup --preserve-numerical-ids --exclude-other-filesystems --exclude-sockets --tempdir /tmp /root/kptmp/src/srv-b.example.net/ /root/kptmp/dumps/srv-b.example.net/root/

curl -c /tmp/cookie.txt -d "login=kp" -d "password=test123" http://127.0.0.1:8080/login/ 1>/dev/null
curl -b /tmp/cookie.txt http://127.0.0.1:8080/ -o out.html

tar czf example.tar.gz *

Attaching example.tar.gz with directory structure and out.html for debugging

Regards,
Kristian

[Patrik Dufresne]

Ho thank. I will give it a try here.

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

130 rue Doris

St-Colomban, QC J5K 1T9

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.

[Patrik Dufresne]

Hello Kristian,

Sorry if I took too long to reply back. I didn't have the chance to work on rdiffweb recently. I've give a shoot to your example.tar.gz archive. I don't have any issue to display the content of /dumps/. I suspect the problem is related to your environment. Probably a different jinja2 version. Here we are using Jinja2 version 2.8.1.

If you want, we may try to debug this issue live via IRC or similar. My nickname is ikus060.

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

130 rue Doris

St-Colomban, QC J5K 1T9

[Andy]

I don't know if it helps, but I'm also having this issue, and I've got Jinja 2.9.6. You can execute `pip freeze | grep -i jinja2` to get the version number.

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Patrik Dufresne]

I've try to narrow down the issue. It's clearly related to a Jinja2 bug. I can't figure out the exact version causing the issue, but everything greater then 2.8.1 is causing trouble. Will add this as a constraint to the project.

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

130 rue Doris

St-Colomban, QC J5K 1T9

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rdiffweb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdiffweb/NgVBjP4gcGU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rdiffweb+unsubscribe@googlegroups.com.

[S Kulkarni]

Has anyone tested rdiffweb 1.0.0 with of Jinja2 > 2.8.1?  I installed it on opensuse 15.0 with python 2.7.14 and python-Jinja2 2.10.1. It seems to be working fine, without the problems described above, but I just have one fairly simple simple repository.

Shanti

To unsubscribe from this group and stop receiving emails from it, send an email to rdif...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rdiffweb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdiffweb/NgVBjP4gcGU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rdif...@googlegroups.com.

[Patrik Dufresne]

Not sure how to manage to install rdiffweb with Jinja2 2 v2.10.1. Rdiffweb is configure with this requirement: Jinja2>=2.6,<=2.8.1. That should exclude v2.10

I could probably determine which version of Jinja2 is properly working with rdiffweb.

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

130 rue Doris

St-Colomban, QC J5K 1T9

To view this discussion on the web visit https://groups.google.com/d/msgid/rdiffweb/da10b4a9-425d-40bc-b98c-08e8c0b525a4%40googlegroups.com.

[S Kulkarni]

I'd made an rdiffweb RPM without an upper limit on the python-Jinja2 version, so I ended up with 2.10.1, which is packaged with opensuse 15.  Rdiffweb refused to run because of the Jinja2 requirement. Rather than be responsible and install 2.8.1, I found and deleted the version requirement in requires.txt.  It seems to be working, so I thought I'd post to see if anyone else has tried this.

Shanti

To unsubscribe from this group and stop receiving emails from it, send an email to rdif...@googlegroups.com.

[Patrik Dufresne]

Are you redistributing your RPM ? Maybe we could work together to make it available as part of rdiffweb cicd pipeline

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

130 rue Doris

St-Colomban, QC J5K 1T9

To view this discussion on the web visit https://groups.google.com/d/msgid/rdiffweb/4b9d4c9a-31cc-44b2-a7dc-8d1317538ff9%40googlegroups.com.

[S Kulkarni]

I'm building it as a project on the opensuse build service (https://build.opensuse.org/project/show/home:shantikulkarni:rdiffweb), so the RPMs are publicly available.  But  I'm running it for personal use only, so I update it infrequently (generally, when something breaks, and I after find time to fix it).  I don't want to agree to maintain a current RPM that people depend on, only to fall behind due to work/family obligations.

To view this discussion on the web visit https://groups.google.com/d/msgid/rdiffweb/4b9d4c9a-31cc-44b2-a7dc-8d1317538ff9%40googlegroups.com.