How to merage the corpus.db

[ljia...@gmail.com]

Hi:

    I want to merge the corpus.db after fuzzing test in different machine.How could I do? The syz-db don't have the parameter for merge.

   If I use the syz-db unpack and syz-db pack, there is another problem as below:

   when I use the syz-db pack, if the corpus number is very much, It seems need  a lot of memory. How to solve it ?

[Dmitry Vyukov]

Hi,

What is size of the resulting corpus? of the parts that your merge?
syz-manager will need to load whole corpus into memory during start,
so 2 possibilities:
1. The resulting corpus is too large for syz-manager is well (it will
also crash with out of memory).
2. It's a deficiency in syz-db tool that makes it use more memory than
necessary.


If it's 2, try the following patch:

--- a/tools/syz-db/syz-db.go
+++ b/tools/syz-db/syz-db.go
@@ -46,7 +46,7 @@ func pack(dir, file string) {
if err != nil {
failf("failed to open database file: %v", err)
}
- for _, file := range files {
+ for i, file := range files {
data, err := ioutil.ReadFile(filepath.Join(dir, file.Name()))
if err != nil {
failf("failed to read file %v: %v", file.Name(), err)
@@ -65,6 +65,11 @@ func pack(dir, file string) {
key = sig
}
db.Save(key, data, seq)
+ if i%1000 == 0 {
+ if err := db.Flush(); err != nil {
+ failf("failed to save database file: %v", err)
+ }
+ }
}
if err := db.Flush(); err != nil {
failf("failed to save database file: %v", err)



If it does not help, then maybe you need to increase amount of RAM on
the machine. How much do you have now on the machine?

The last resort option is to merge corpus in small parts, but it will
be tedious.
You can merge a small batch of programs from another corpus, then
start syz-manager with this new corpus, wait till syz-manager triages
all programs and minimizes corpus on disk. Then stop syz-manager and
merge in another batch and repeat the process.

Also you may consider using syz-hub:
https://github.com/google/syzkaller/wiki/Connecting-Several-Managers-(Hub)
Then you don't need to merge corpuses manually.