How can I add a new linux rekall profile to a local repository?

[yevgen...@gmail.com]

Hi,

I'm interested to understand the subject. The main goal is to be able to analyze memory images of linux machines without the need to supply a profile every time.

Thanks.

[wlamb...@gmail.com]

Have you seen the following?

http://www.rekall-forensic.com/faq.html

Thanks,

Wes

[Michael Cohen]

Hi,
On linux Rekall uses a profile index to automatically detect the
right profile from /proc/kallsyms when used in live mode. This is how
it works:

0) On your unsupported system install rekall (pip install rekall) and
go to VirtualEnvRoot/tools/linux and type make there.

1) You check out the profile repository locally (use git clone
https://github.com/google/rekall-profiles.git --depth 1)
2) You put your generated zip file (with the kernel modules in them)
in the linux directory somewhere that makes sense (e.g.
src/Linux/Debian/4.2.0.zip)
3) You then run the repository management tool which will sync the
repository from the src directory - it will automatically convert the
json file and add it to the index:

$ rekall manage_repo --path_to_repository ~/projects/rekall-profiles/

Generally the manage_repo plugin will maintain the repository
consistency and generate indexes, inventories etc. Once your new
profile is in the index, rekall should just recognize it immediately.

Currently updating the index takes a long time so if you can batch
your updates it would be faster (i.e. updating 10 new profiles takes
as long as 1).

We would also appreciate if you can send us the new profiles so we can
add them to the public repo (e.g. a PR would be nice).

Note too that the profile repository also has the original zip file
checked in (the src/Linux) directory and that zip file has the pmem
module in it! So if you ever find yourself on a system with
/proc/kcore disabled for some reason you can use the pmem module from
there. I know many of the Centos systems these days disable
/proc/kcore so it would be nice to have a pipeline supporting these
systems.

Hope this helps,
Michael.

[yevgen...@gmail.com]

On Friday, August 12, 2016 at 1:50:41 AM UTC+3, Michael Cohen wrote:

Thanks, Michael.

I've done as you've instructed but the following continues to return :

2016-08-14 02:43:44,369:CRITICAL:rekall.1:Traceback (most recent call last):
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/session.py", line 840, in RunPlugin
result = plugin_obj.render(ui_renderer) or plugin_obj
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/tools/repository_manager.py", line 533, in render
handler.Build(renderer, *self.builder_args)
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/tools/repository_manager.py", line 406, in Build
self.BuildIndex()
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/tools/repository_manager.py", line 110, in BuildIndex
manager=repository).build_index(spec)
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/tools/profile_tool.py", line 1128, in build_index
return self.BuildSymbolsIndex(spec)
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/tools/profile_tool.py", line 995, in BuildSymbolsIndex
profile["$CONSTANTS"])
File "/home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/common/profile_index.py", line 353, in RelativizeSymbols
raise ValueError("Symbol %s not found in profile", base_symbol)
ValueError: ('Symbol %s not found in profile', u'linux_proc_banner')

> /home/[user]/linmem/local/lib/python2.7/site-packages/rekall/plugins/common/profile_index.py(353)RelativizeSymbols()
-> raise ValueError("Symbol %s not found in profile", base_symbol)

I've checked in System.map and the linux_proc_banner is there. I think, that the side effect of the above is that the new profile will not be added to the inventory as rekall still fails to recognize the profile of my linux dump.

I'll appreciate your help.

P.S. I've tried the same process on CentOS 7 and Debian 8 machines. I'm getting the same mistake.

[Jordi Sanchez]

Can you run

$ unzip -l ${YOURPROFILE}.zip

And paste the output here? It's likely the System.map was never included in your profile for some reason.

FYI manage_repo does 3 things for Linux profiles:

 1. Converts a source profile into a profile (no more rekal convert_profile myprofile.zip > myprofile).

 2. Adds the profile to the inventory. This is just a list of all the profiles that exist in the repository.

 3. Automatically indexes the profile. That is: finds what's unique in it inside the repository so that live analysis or aff4dumps (and only aff4dumps) can be identified accurately. Then stores it in an index file.

The old linux autodetection method uses the kernel version string and, thus, only needs the inventory. But it can easily misidentify kernel versions.

The new autodetection is used on live and aff4dumps and uses the index.

The error that you're getting above is for step [3], when it's trying to index the profile but it appears it can't find the right base symbol.

Unless we broke something (which isn't unlikely), I believe there's an issue with how your profile was generated.

--
You received this message because you are subscribed to the Google Groups "rekall-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rekall-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to rekall-discuss@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[yevgen...@gmail.com]

> To unsubscribe from this group and stop receiving emails from it, send an email to rekall-discus...@googlegroups.com.
>
> To post to this group, send email to rekall-...@googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

Archive: v1.0/src/Linux/Debian/Debian_3.16.0-4-amd64.zip
Length Date Time Name
--------- ---------- ----- ----
370128 2016-08-08 23:34 module_dwarf.ko
2672227 2015-11-09 15:35 boot/System.map-3.16.0-4-amd64
157726 2015-11-09 15:35 boot/config-3.16.0-4-amd64
185888 2016-08-08 23:34 pmem-3.16.0-4-amd64.ko
--------- -------
3385969 4 files

regarding the third bullet, I'm analyzing ELF file created with lmap.
"(and only aff4dumps)" - so, rekall can't auto detect kernel version in raw (dd) memory dump?

[Michael Cohen]

Rekall can not use the Linux index to detect profiles in a raw memory image. If you have a raw image you will need to supply the exact profile each time. The index works by looking at /proc/kallsyms to identify the right kernel. When using an aff4 image this should be also acquired so it should work.

To unsubscribe from this group and stop receiving emails from it, send an email to rekall-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to rekall-discuss@googlegroups.com.

[yevgen...@gmail.com]

ELF file format concept looks very close to raw memory image and LinuxBannerDetector does a good job right now and detects the profile automatically.

So from your experience, how LinuxBannerDetector is reliable ? I've tested it on Debian.

[Michael Cohen]

No it is not reliable. It basically just searches for a string that looks like the banner and chooses the profile with that name. There are two issues with this approach

1. The banner could be confused for example you are running a guest Linux which will also add it's own banner.

2. The banner is not unique. There are many kernels which are different but they have the same banner. It's not possible to tell them apart with just the banner string.

The profile index does not use the name of the profile as a unique identifier so it's fine to have many profiles with the same version string but they are different.

Using the profile index in live mode is extremely reliable (so long as we have the profile in the index of course). W just need to make sure the proc/kallsyms file ends up in the image somewhere.