Repositories not accessible

119 views
Skip to first unread message

Andy Forceno

unread,
Oct 6, 2017, 11:07:47 PM10/6/17
to rdiffweb
I recently reinstalled Rdiffweb from the git repo, following a reinstall of my OS. I found that the Rdiffweb interface has been tweaked, and now it's not displaying my repositories correctly. I can only access 2 of 9 repos, and they are not organized correctly on the page. See the attached file for an example of the issue.

I'm seeing a number of errors in the log, but I'm not sure if they are related:

RepoObject(db, u'admin', u'/Poincare/root')]
Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 117, in job_run
    self._remove_older(user, repo, keepdays)
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 133, in _remove_older
    d = d.days + keepdays
TypeError: unsupported operand type(s) for +: 'int' and 'unicode'

And also this: 

[2017-10-04 20:07:29,294][WARNING][192.168.1.5][none][CP Server Thread-6][rdiffweb.librdiff] fail to parse date ['2017-02-15T00\xef\x80\xa200\xef\x80\xa202-05\xef\x80\xa200']
Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/rdiffweb/librdiff.py", line 681, in _extract_date
    return rdw_helpers.rdwTime(date_string.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 13: ordinal not in range(128)
[2017-10-04 20:07:29,295][WARNING][192.168.1.5][none][CP Server Thread-6][rdiffweb.librdiff] fail to parse date ['2016-12-12T06\xef\x80\xa234\xef\x80\xa207-05\xef\x80\xa200']

Should I have installed from the zip on the website, rather than Github? 
rdiffweb.png

Andy Forceno

unread,
Oct 6, 2017, 11:10:26 PM10/6/17
to rdiffweb
I should have mentioned that I should be seeing two folders under each of the repositories, for the home and root directories of each machine. Instead, I have just the two folders, neither of which is displayed under the correct repo.

Patrik Dufresne

unread,
Oct 8, 2017, 8:36:58 AM10/8/17
to Andy Forceno, rdiffweb

Hi Andy,

I will have a look to your issue. Installing from github or from pypi is the same as long as you install from master branch.

For the display issue, I need some info to reproduce. May you provide something similar where each leaf is a rdiff-backup repo ?
- /var/backups/rdiffweb
-- my-laptop
-- servers
--- Malta
--- Ranculos

Thanks

--
You received this message because you are subscribed to the Google Groups "rdiffweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

t4exanadu

unread,
Oct 8, 2017, 11:40:31 AM10/8/17
to Patrik Dufresne, rdiffweb
H Patrik,

Let me know if I misunderstand your request. The folder structure of
the repo looks like this:
- /media/backups
-- Aurorabox
--- root
--- home
-- Poincare
--- root
--- home
-- Ariadne
--- root
-- Nyx
--- root

Individual machines are each backed up to /media/backups/<hostname>/

One thing that is different about my setup now: /media/backup/ is a
symlink to /media/external-hdd/backups/. Not sure if that is
significant. I would think that shouldn't matter.

Patrik Dufresne

unread,
Oct 8, 2017, 12:31:11 PM10/8/17
to t4exanadu, rdiffweb
I've test a similar configuration here and it is shown as expected.

Symlink to /media/external-hdd/backups/ may cause few problems. Rdiffweb is not following symlink for security reason.

May you try to refresh your repositories. This options is available in user preferences.

Than look in the logs (/var/log/rdiffweb.log) for lines similar to the following:
[2017-10-08 12:25:44,803][DEBUG  ][127.0.0.1][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] find repos for UserObject[admin]
[2017-10-08 12:25:44,806][DEBUG  ][127.0.0.1][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] set UserObject[admin] repos: [u'/MainFolder/SubFolder/home', u'/MainFolder/SubFolder/root', u'/data', u'/MyLaptop/D', u'/MyLaptop/C', u'/broker-repo', u'/Servers/Kase', u'/Servers/Malta', u'/testcases'] 

Most likely yours will look like this:

[2017-10-08 12:27:09,797][DEBUG  ][127.0.0.1][none][CP Server Thread-10][rdiffweb.rdw_spider_repos] set UserObject[admin] repos: [] 

To resolve the issue, you will need to change the path from /media/backups to /media/external-hdd/backups/



--
Patrik Dufresne

Andy

unread,
Oct 8, 2017, 1:09:13 PM10/8/17
to Patrik Dufresne, rdiffweb
I changed it to the full path, and refreshed the repositories. It still looks the same, though. I tried it in another browser, to rule that out, but it looks the same. 

In rdiffweb.log, I see that same error repeated for all repositories. I do not see any lines indicating it is finding the repos, not within the last few days. Nothing seems to be logged when I refresh the repositories. 

[2017-10-06 23:04:33,820][ERROR  ][127.0.0.1][none][RemoveOlderPlugin][rdiffweb.plugins.remove_older] fail to remove older for user [<rdiffweb.user.UserObject object at 0xb5e44fec>] repo [RepoObject(db, u'admin', u'/Ariadne/root')]
Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 117, in job_run
    self._remove_older(user, repo, keepdays)
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 133, in _remove_older
    d = d.days + keepdays
TypeError: unsupported operand type(s) for +: 'int' and 'unicode'
--
You received this message because you are subscribed to a topic in the Google Groups "rdiffweb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdiffweb/NgVBjP4gcGU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

Patrik Dufresne

unread,
Oct 8, 2017, 2:58:07 PM10/8/17
to Andy, rdiffweb
Hello Andy,

This line: 
fail to remove older for user [<rdiffweb.user.UserObject object at 0xb5e44fec>] repo [RepoObject(db, u'admin', u'/Ariadne/root')]

Is related to the exception you first mention:

RepoObject(db, u'admin', u'/Poincare/root')]
Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 117, in job_run
    self._remove_older(user, repo, keepdays)
  File "build/bdist.linux-i686/egg/rdiffweb/plugins/remove_older/__init__.py", line 133, in _remove_older
    d = d.days + keepdays
TypeError: unsupported operand type(s) for +: 'int' and 'unicode'

I manage to reproduce the problem somehow and made a fix in develop branch. But it's not related to your repositories not showing off.

To have better logging, I recommend you to change the loglevel to debug in the configuration file /etc/rdiffweb/rdw.conf and restart.
Then you should have more information regarding the refresh. If not, something is wrong with your installation or configuration.



--
Patrik Dufresne


To unsubscribe from this group and all its topics, send an email to rdiffweb+unsubscribe@googlegroups.com.

Andy

unread,
Oct 8, 2017, 4:34:36 PM10/8/17
to Patrik Dufresne, rdiffweb
With log level set to DEBUG, I see the following when I access rdiffweb in my browser or refresh the repositories:

[2017-10-08 16:25:38,703][DEBUG  ][192.168.1.5][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] find repos for UserObject[admin]
[2017-10-08 16:25:39,000][DEBUG  ][192.168.1.5][none][CP Server Thread-9][rdiffweb.rdw_spider_repos] set UserObject[admin] repos: [u'/Aurorabox/root', u'/Aurorabox/home', u'/Poincare/root', u'/Poincare/home', u'/Ariadne/root', u'/Fractal/root', u'/Fractal/home'] 
[2017-10-08 16:25:41,057][INFO   ][192.168.1.5][none][CP Server Thread-13][rdiffweb.filter_authentication] no special path, running do_check
[2017-10-08 16:25:41,058][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.filter_authentication] setting request.login to UserObject[admin]
[2017-10-08 16:25:41,059][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.page_locations] browsing locations
[2017-10-08 16:25:41,064][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] pid [15978] does not exists
[2017-10-08 16:25:41,065][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] get backup dates for ['/media/big-linux/backups/Poincare/root']
[2017-10-08 16:25:41,134][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] pid [6464] does not exists
[2017-10-08 16:25:41,134][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] get backup dates for ['/media/big-linux/backups/Poincare/home']
[2017-10-08 16:25:41,137][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] pid [20955] does not exists
[2017-10-08 16:25:41,137][DEBUG  ][192.168.1.5][none][CP Server Thread-13][rdiffweb.librdiff] get backup dates for ['/media/big-linux/backups/Ariadne/root']

That repeats for each of my 9 repos. 


To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

Patrik Dufresne

unread,
Oct 11, 2017, 11:14:54 AM10/11/17
to Andy, rdiffweb
Hello Andy,

I have a hard time to understand what is wrong. According to the log everything seams to be working as expected. For the sake of testing, I will try to mimic as close as possible your current setup. I will keep you posted if I found anything.

Question:

May you browse to ...?





--
Patrik Dufresne


To unsubscribe from this group and all its topics, send an email to rdiffweb+unsubscribe@googlegroups.com.

Andy

unread,
Oct 11, 2017, 7:17:00 PM10/11/17
to Patrik Dufresne, rdiffweb
Using a tunnel to localhost, I can browse to all the repositories.

To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.

Kristian Pedersen

unread,
Dec 21, 2017, 11:39:06 AM12/21/17
to rdiffweb
Hi Patrick,

Im wondering if this issue was ever identified/resolved? We just reinstalled our backup server (for a clean upgrade from Debian 6 to Debian 9.3), installed rdiffweb using pypi as mentioned in the documentation, and we are facing what appears to be this same issue.

Our repositories are organized as:
/data/dumps (Root directory)
/data/dumps/<server>/<partition>/ (rdiff-repository)

We have 62 servers with a total of 81 partitions/repositories. The repositories are typically called "boot", "root", "data" and a few "mnt1" / "mnt2", no special characters. Rdiffweb will pick up all 62 servers but only 5 of the 81 repositories, and the repositories are not displayed under the server where they belong (attaching screenshot):



I have enabled debug, and when browsing the list of repositories i see consistent "pid [xxxx] does not exist" and "get backup dates for [yyyy]" for all 81 repositories, and when updating the list of repositories, all 81 repositories are mentioned in the "repos: [aaaa, bbbb, cccc ...]" log lines. If i construct the url, i can access all repositories just fine, so it appears its only the listing of repositories that has an issue.

Looking at it again, I can actually see only one boot, one root, one data and one mnt1 /mnt2 repository is displayed, so maybe the issues is related to having repositories with same name in sub folders, ie:

/data/dumps/srv1/root
/data/dumps/srv2/root

Kind regards,
Kristian

--
Patrik Dufresne


To unsubscribe from this group and all its topics, send an email to rdiffweb+u...@googlegroups.com.
Auto Generated Inline Image 1

Patrik Dufresne

unread,
Dec 21, 2017, 11:45:11 AM12/21/17
to Kristian Pedersen, rdiffweb
Hello Kristian,

We didn't find a solution to Andy's issue. Was not able to reproduce it on my side. There is something very odd with the display obviously.

Are you using a nginx reverse proxy to access rdiffweb ? If so, may you try to query the page directly without nginx reverse proxy ?
In any case, would it be possible to query the page using curl and dump it into a file ? Would love to see the html generated.

Thanks


--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
130 rue Doris
St-Colomban, QC J5K 1T9

You received this message because you are subscribed to the Google Groups "rdiffweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.

Kristian Pedersen

unread,
Dec 21, 2017, 3:17:49 PM12/21/17
to rdiffweb
Hi Patrick,

Thanks for getting back to me so fast :)

I am accessing rdiffweb through the build-in http, so no nginx/apache reverse proxy going on.

I will reply to you in private with a copy of the generated html obtained with curl, I hope this is okay:

curl -c cookie.txt -d "login=admin" -d "password=somepassword" http://backup.xxx:8080/login/
curl -b cookie.txt http://backup.xxx:8080/ -o out.html

Kind regards,
Kristian
To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+u...@googlegroups.com.

Kristian Pedersen

unread,
Dec 28, 2017, 1:06:44 PM12/28/17
to rdiffweb
Update:

I am able to reproduce this issue using two very simple rdiff-backup repositories using same directory structure.

Using commands:
rdiff-backup --preserve-numerical-ids --exclude-other-filesystems --exclude-sockets --tempdir /tmp /root/kptmp/src/srv-a.example.net/ /root/kptmp/dumps/srv-a.example.net/root/
rdiff-backup --preserve-numerical-ids --exclude-other-filesystems --exclude-sockets --tempdir /tmp /root/kptmp/src/srv-b.example.net/ /root/kptmp/dumps/srv-b.example.net/root/

curl -c /tmp/cookie.txt -d "login=kp" -d "password=test123" http://127.0.0.1:8080/login/ 1>/dev/null
curl -b /tmp/cookie.txt http://127.0.0.1:8080/ -o out.html

tar czf example.tar.gz *

Attaching example.tar.gz with directory structure and out.html for debugging

Regards,
Kristian
example.tar.gz

Patrik Dufresne

unread,
Jan 4, 2018, 8:48:02 AM1/4/18
to Kristian Pedersen, rdiffweb
Ho thank. I will give it a try here.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
130 rue Doris
St-Colomban, QC J5K 1T9

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.

Patrik Dufresne

unread,
Jan 30, 2018, 2:51:35 PM1/30/18
to Kristian Pedersen, rdiffweb
Hello Kristian,

Sorry if I took too long to reply back. I didn't have the chance to work on rdiffweb recently. I've give a shoot to your example.tar.gz archive. I don't have any issue to display the content of /dumps/. I suspect the problem is related to your environment. Probably a different jinja2 version. Here we are using Jinja2 version 2.8.1.

If you want, we may try to debug this issue live via IRC or similar. My nickname is ikus060.


--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
130 rue Doris
St-Colomban, QC J5K 1T9

Andy

unread,
Jan 30, 2018, 4:26:32 PM1/30/18
to Patrik Dufresne, Kristian Pedersen, rdiffweb
I don't know if it helps, but I'm also having this issue, and I've got Jinja 2.9.6. You can execute `pip freeze | grep -i jinja2` to get the version number.

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Patrik Dufresne

unread,
Feb 15, 2018, 6:19:41 PM2/15/18
to Andy, Kristian Pedersen, rdiffweb
I've try to narrow down the issue. It's clearly related to a Jinja2 bug. I can't figure out the exact version causing the issue, but everything greater then 2.8.1 is causing trouble. Will add this as a constraint to the project.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
130 rue Doris
St-Colomban, QC J5K 1T9

To unsubscribe from this group and stop receiving emails from it, send an email to rdiffweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rdiffweb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdiffweb/NgVBjP4gcGU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rdiffweb+unsubscribe@googlegroups.com.

S Kulkarni

unread,
Sep 19, 2019, 1:15:52 PM9/19/19
to rdiffweb
Has anyone tested rdiffweb 1.0.0 with of Jinja2 > 2.8.1?  I installed it on opensuse 15.0 with python 2.7.14 and python-Jinja2 2.10.1. It seems to be working fine, without the problems described above, but I just have one fairly simple simple repository.

Shanti
To unsubscribe from this group and stop receiving emails from it, send an email to rdif...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rdiffweb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdiffweb/NgVBjP4gcGU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rdif...@googlegroups.com.

Patrik Dufresne

unread,
Sep 19, 2019, 1:27:08 PM9/19/19
to S Kulkarni, rdiffweb
Not sure how to manage to install rdiffweb with Jinja2 2 v2.10.1. Rdiffweb is configure with this requirement: Jinja2>=2.6,<=2.8.1. That should exclude v2.10
I could probably determine which version of Jinja2 is properly working with rdiffweb.

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
130 rue Doris
St-Colomban, QC J5K 1T9

S Kulkarni

unread,
Sep 19, 2019, 1:40:20 PM9/19/19
to rdiffweb
I'd made an rdiffweb RPM without an upper limit on the python-Jinja2 version, so I ended up with 2.10.1, which is packaged with opensuse 15.  Rdiffweb refused to run because of the Jinja2 requirement. Rather than be responsible and install 2.8.1, I found and deleted the version requirement in requires.txt.  It seems to be working, so I thought I'd post to see if anyone else has tried this.

Shanti
To unsubscribe from this group and stop receiving emails from it, send an email to rdif...@googlegroups.com.

Patrik Dufresne

unread,
Sep 19, 2019, 1:42:37 PM9/19/19
to S Kulkarni, rdiffweb
Are you redistributing your RPM ? Maybe we could work together to make it available as part of rdiffweb cicd pipeline

--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com/
130 rue Doris
St-Colomban, QC J5K 1T9

S Kulkarni

unread,
Sep 19, 2019, 2:08:38 PM9/19/19
to rdiffweb
I'm building it as a project on the opensuse build service (https://build.opensuse.org/project/show/home:shantikulkarni:rdiffweb), so the RPMs are publicly available.  But  I'm running it for personal use only, so I update it infrequently (generally, when something breaks, and I after find time to fix it).  I don't want to agree to maintain a current RPM that people depend on, only to fall behind due to work/family obligations.
Reply all
Reply to author
Forward
0 new messages