Audit Pipe Drops

[b via Darwin-dev]

Hello everyone,

I am crossposting this message to the darwin dev list and the trustedbsd-audit list since relevant knowledge might be shared among members of both lists and I was not really sure how to properly present the issue to both lists separately. Also, my post is quite long as it bundles the outcome of several days of debugging. Please excuse.

I am currently reading a cloned bsm audit pipe from a user space client on macOS to retrieve information about process creation and operations on the system (pc|ex). In the future I want to add other audit classes as well. I am currently looking at bsmtrace as a reference implementation of the read loop.

There is one major issue, though, that took me a while to become aware of. The audit pipe is allowed to drop audit records in the kernel, which eventually is technically inevitable of course. The issue I am facing is finding a reliable solution to avoid this.

Drops can occur only in audit_pipe_append (audit_pipe.c) under two conditions, (1) the queue is full or (2) the kernel was not able to allocate memory without blocking.

(1) can generally be managed in user space client code. There is only one scenario I found (up to now) where even the maximum audit pipe length is insufficient and that is system wake up procedure. Even before the system emits kIOMessageSystemWillPowerOn there are lots and lots of audit events that eventually max out the audit pipe.

(2) can’t be influenced from a user space client, at least not that I am aware of. This happens sporadically but reproducibly when the system is under a lot of stress, e.g. when a lot of processes are spawned at a time and start executing some audited code.

All this leads to several questions of which I am not sure if they qualify as potential bug reports or enhancement requests. I would be more than happy if someone with a better understanding of things could comment on them or even suggest possible solution approaches.

- (1) could clearly be solved by increasing the allowed maximum audit pipe length which currently is 1024. This maximum value is probably several years old by now and I could imagine that in the meantime the xnu kernel has become a lot more „talkative“. Is there some technical limitation that would prevent an increase?

- I am wondering how the global audit trail that is written to disk is able to perform better, i.e. does not drop (seemingly), than the cloned audit pipe purely in memory? Is there some penalty associated with passing memory from the kernel to user space? How does the global audit trail not struggle with the M_NOWAIT policy?

- Since the audit pipe tries to acquire memory by calling malloc with the M_NOWAIT flag, which I assume happens for a reason, is there some strategy or configuration available from user space to ease the restraint on kernel memory allocation – or am I simply out of luck here?

- I am surely not an expert on memory allocation and even less so in kernel space. With that said, I imagine that allocating only one block of memory for both, pointer (ape) and memory blob (ape->ape_record), instead of two separate memory allocations would half the potential for M_NOWAIT failure.

- Potential for both (1) and (2) could at least be reduced by further filtering events in the kernel. I am not interested in each and every audit event type. The FreeBSD Handbook has the notion that „audit event classes may be customized by modifying the audit_class and audit_event configuration files“. I assume this could also mean adding custom audit classes and associating them with the event types of interest. How can this be done programmatically for local audit trails? Are there any code examples available?

I hope this does not come across (too) snarky but my expectation towards a security mechanism is first and foremost reliability. Information loss is definitely not supportive in that. Therefore I either hope for a viable existing solution for my problem or am very eager for a fix to the issues at hand.

Thanks,

Benjamin