Issue restoring files

[Toma Morris]

Hi,

I am trying to restore files from one of my backups. In the Restore tab of the webui, I select the client and then after a delay I get the following error message:

{"id":"#","xhr":{"readyState":4,"responseText":"Decoding failed: Syntax errorError: API 2 not available on director.\n                      Please upgrade to verison 15.2.2 or greater and/or compile with jansson support.","status":200,"statusText":"OK"}

Bareos is running on an Ubuntu 20.04 server with hostname cscisbu01, and I am trying to access a backup from a client with hostname csspfile01. I initially got this error while on 23.0.4~pre169.6644119b5-114. I upgraded bareos on both the server and client to the most recent version via apt, which brought me up to 23.0.5~pre146.7e91df1c0-130 with no apparent change in behavior.

I have successfully restored files within the last couple of months on this server from the same client, and have not changed any configurations since then.

I've attached my apache2 error logs, which contain some possibly-relevant php errors. I didn't see anything relevant in /var/log/bareos/bareos.log or /var/log/bareos/bareos-audit.log, but can provide those or anything else if needed.

Thanks,

-Toma

[Sebastian Sura]

Hi Toma,

could you try doing a debug trace on the director when you are trying the restore ?
You can enable the debug log by writing

setdebug level=200 trace=1 director

in the bconsole.   Once you have reproduced the issue, you can turn it off with e.g. setdebug level=10 trace=0 director again.
Having this debug trace would be immensely helpful.

Kind Regards,

Sebastian Sura

Am 23.10.24 um 19:47 schrieb Toma Morris:

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/4ea3f42e-9b0e-4c3f-b28d-a2f382b3d81an%40googlegroups.com.

-- 
 Sebastian Sura                  sebasti...@bareos.com
 Bareos GmbH & Co. KG            Phone: +49 221 630693-0
 https://www.bareos.com
 Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
 Komplementär: Bareos Verwaltungs-GmbH
 Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz

[Toma Morris]

Here is the debug trace. Let me know if there's anything else I can provide. Thanks,

-Toma

You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/t7mBaL-hNKI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/44571472-7df2-4035-835f-1f9cbf4cab39%40bareos.com.

[Toma Morris]

Here is the debug trace during attempted restore, resulting in the same error condition. Thank you!

-Toma

[Bruno Friedmann (bruno-at-bareos)]

Hi Toma,

It look like the query building the tree didn't finish or vanish. 

This may happen when your PostgreSQL cluster doesn't have enough free space to store tmp queries/tables.... During a very limited time, the disk space become full and the connection is then cleared, PG dropping the running worker, free space reappear just after.

You may want to try the following query (issue from your trace) and watch carefully the free space available where you PG cluster reside and build its temp data

SELECT DISTINCT ON (PathId) 'D', SpecialDir.PathId, SpecialDir.Path, JobId, LStat, FileId FROM ( SELECT 2711925 AS PathId, '.' AS Path UNION SELECT PPathId AS PathId, '..' AS Path FROM   PathHierarchy WHERE  PathId = 2711925 ) AS SpecialDir LEFT JOIN ( SELECT PathId, JobId, LStat, FileId FROM File WHERE File.Name = '' AND File.JobId IN (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) ORDER BY PathId ASC, JobId DESC ) AS DirAttribute ON (SpecialDir.PathId = DirAttribute.PathId)  UNION ( SELECT DISTINCT ON (Path) 'D', PathId, Path, JobId, LStat, FileId FROM ( SELECT Path1.PathId AS PathId, Path1.Path AS Path, lower(Path1.Path) AS lpath, listfile1.JobId AS JobId, listfile1.LStat AS LStat, listfile1.FileId AS FileId FROM ( SELECT listpath1.PathId AS PathId FROM ( SELECT DISTINCT PathHierarchy1.PathId AS PathId FROM PathHierarchy AS PathHierarchy1 INNER JOIN Path AS Path2 ON (PathHierarchy1.PathId = Path2.PathId) INNER JOIN PathVisibility AS PathVisibility1 ON (PathHierarchy1.PathId = PathVisibility1.PathId) WHERE PathHierarchy1.PPathId = 2711925 AND PathVisibility1.JobId IN (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) ) AS listpath1 LEFT JOIN ( SELECT PVD1.PathId AS PathId FROM ( SELECT PV1.PathId AS PathId, MAX(JobId) AS MaxJobId FROM PathVisibility AS PV1 WHERE JobId IN (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) GROUP BY PathId ) AS PVD1 INNER JOIN File AS F2 ON (F2.PathId = PVD1.PathId AND F2.JobId = PVD1.MaxJobId AND F2.FileIndex = 0 AND F2.Name = '') ) AS listpath2 ON (listpath1.PathId = listpath2.PathId) WHERE listpath2.PathId IS NULL  ) AS listpath3 INNER JOIN Path AS Path1 ON (listpath3.PathId = Path1.PathId) LEFT JOIN ( SELECT File1.PathId AS PathId, File1.JobId AS JobId, File1.LStat AS LStat, File1.FileId AS FileId FROM File AS File1 WHERE File1.Name = '' AND File1.JobId IN (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) ) AS listfile1 ON (listpath3.PathId = listfile1.PathId) ) AS A ORDER BY Path ASC, JobId DESC )  ORDER BY Path ASC,JobId DESC LIMIT 1000 OFFSET 0

[Toma Morris]

Thank you, that pointed me in the right direction. That query succeeded, but took a very long time. I did "explain analyze" on it, and saw that it was using much more than than the allotted working memory (using 334Mb, where work_mem in postgres was set to 4Mb), which caused postgres to use an external merge rather than quicksort. I set work_mem to 512MB, which brought the execution time from 224 s to 38 s and allowed the Bareos restore interface to function.

You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/t7mBaL-hNKI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/cb1770ec-d5b0-4744-9d56-46eb8a30ccfdn%40googlegroups.com.

[Bruno Friedmann (bruno-at-bareos)]

Thanks for the feedback and the report.

[Toma Morris]

Ok, now that the memory issues have been resolved, it has uncovered another issue. It appears that the job that I am attempting to restore from failed to include the target files, because it did not descend into the /export directory with the following error message:

CSSPFILE01-fd JobId 51714: /export is a different filesystem. Will not descend from / into it.

The fileset used for this job specifies "One FS = No", and at some point in the past this job was descending into /export. / and /export are separate ext4 partitions. Here are the most relevant configs for the job and fileset:

FileSet {
  Name = agent-fs
  Description = "Backup all regular filesystems, including postgres."
  Include {
    Options {
      #compression=GZIP
      Signature = MD5
      One FS = No     # change into other filessytems
      FS Type = btrfs
      FS Type = ext2
      FS Type = ext3
      FS Type = ext4
      FS Type = jfs
      FS Type = xfs
      FS Type = zfs
    }
    Options {
    # Wild Dir directives are not allowed in Exclude blocks, so placing here
      Regex Dir = ".*/\.opam/*"
      Regex Dir = ".*/\.gradle/*"
      Regex Dir = ".*/\.vscode/*"
      Wild Dir = "/usr/src"
      Wild Dir = "/lib/modules"
      Exclude = yes
    }
    File = /
    Exclude Dir Containing = .nobackup
  }
  Exclude {
    File = /var/lib/bareos
    File = /var/lib/bareos/storage
    File = /proc
    File = /tmp
    File = /dev
    File = /run
    File = /sys
    File = /var/tmp
    File = /.journal
    File = /.fscka
    File = /var/lib/lxcfs
    File = */.local
    File = /swapfile
    File = /export/archive
    File = /snap
  }
}

Job {
  Name = file01-bu
  Client = csspfile01-fd
  Type = Backup
  FileSet = agent-fs
  Schedule = MonthlyCycle
  Messages = StandardNew
  Pool = IncrPool1
  Level = Incremental
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Full Backup Pool = FullPool0
  Differential Backup Pool = DiffPool0
  Incremental Backup Pool = IncrPool1
  Priority = 10
  RunScript {
    RunsWhen = Before
    RunsOnClient = yes
    Fail Job On Error = yes
    Command = "sudo  rear mkrescue"
  }
  Run Script {
    Console = ".bvfs_update"
    RunsWhen = After
    RunsOnClient = No
  }
}

I have been able to force the backup of /export going forward by adding a "File = /export" directive, but my understanding of how this should work (based on https://docs.bareos.org/Configuration/Director.html#config-Dir_Fileset_Include_Options_OneFs) is that this should not be necessary. Happy to provide any other configs, traces, etc as needed.

-Toma

[Sebastian Sura]

Hi Toma,

it looks like somehow One FS = NO is not honered here.  According to our docs: https://docs.bareos.org/Configuration/Director.html#config-Dir_Fileset_Include_Options_OneFs the warning you receive is expected even if you explicitly add "File = /export", but it should still get backed up in that case.
Could you verify once more that "/export" is not backed up even if you list it explicitly ?  Is it possible that you forgot to reload your config ?

There is still the issue of why it ignores your OneFS option however. Could you run a backup with trace level 500 and post the output here ?  This should print the config that the file daemon received.
Its very possible that the cause of the problem is using multiple options blocks.  They are a bit buggy and we recommend against using them.  Could you try switching up the order of your option blocks ? For some options only the last option block is considered.

Kind Regards
Sebastian Sura

Am 30.10.24 um 16:10 schrieb Toma Morris:

To view this discussion visit https://groups.google.com/d/msgid/bareos-users/0d04ff3a-043d-4f41-8cee-ae5386e11580n%40googlegroups.com.

[Toma Morris]

Hi Sebastian,

I ran the job as normal, and am attaching the trace for that run. I then reversed the order of the Option blocks, and it appears that it did honor the OneFS option.

As for a single Option block, I currently have this:

    Options {

    # Wild Dir directives are not allowed in Exclude blocks, so placing here

      Wild Dir = "/usr/src"
      Wild Dir = "/lib/modules"
      Exclude = yes
    }

    Options {

      Signature = MD5
      One FS = No     # change into other filessytems
      FS Type = btrfs
      FS Type = ext2
      FS Type = ext3
      FS Type = ext4

      FS Type = reiserfs

      FS Type = jfs
      FS Type = xfs
      FS Type = zfs
    }

I presume it would work correctly if I combined these into a single Options, like this:

    Options {

    # Wild Dir directives are not allowed in Exclude blocks, so placing here

      Wild Dir = "/usr/src"
      Wild Dir = "/lib/modules"
      Exclude = yes

      Signature = MD5
      One FS = No     # change into other filessytems
      FS Type = btrfs
      FS Type = ext2
      FS Type = ext3
      FS Type = ext4

      FS Type = reiserfs

      FS Type = jfs
      FS Type = xfs
      FS Type = zfs
    }

Does this seem right? I inherited this configuration, and don't have a strong intuition yet about whether or not any given options directives will interact with each other. Thanks,

Toma

 bareos-dir.trace01.txt

To view this discussion visit https://groups.google.com/d/msgid/bareos-users/ba245412-8ecf-4a28-b9b8-91a366b4277f%40bareos.com.

[Sebastian Sura]

Hi Toma

from what i can tell you are just trying to exclude some directories.  This should be possible with a simple exclude block, i.e.

Fileset {
    Include { ... }
    Exclude {
        File = /usr/src
        File = /lib/modules  
    }
}

This way you only need to use one Options block in your include block itself.  Combining them should also work from what I remember.

Kind Regards
Sebastian Sura

Am 06.11.24 um 20:16 schrieb Toma Morris:

To view this discussion visit https://groups.google.com/d/msgid/bareos-users/CAKfWHbq%3DdgYJo_LpGSaVe9U%2BF8jDHS9U5jFbdPR0gtQeKDWC3g%40mail.gmail.com.