Issue 26 in fabricate: OSX support with DYLD_INSERT_LIBRARIES/interposing

[fabr...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 26 by comexk: OSX support with DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

While OSX has dtruss which has similar output to strace, it (and anything
else using DTrace) requires root access making it impractical. Here is an
alternative implementation using DYLD_INSERT_LIBRARIES, similar to
LD_PRELOAD on Linux, to detect what files are being accessed. This
requires bundling C code into fabricate.py, but seems to work correctly and
quickly (probably faster than strace as it doesn't need to use regular
expressions).

Attachments:
interposing.diff 5.7 KB

[Berwyn Hoyt]

Hi comexk,

Thanks for submitting this patch.  We have no way of testing it on OSX, so if we do include this, we'll need you to take over maintenance of the OSX functionality, and document it on the fabricate website.  Would you be willing to do this?  If so, I'll add you as a user on the fabricate website.

Thanks,
Berwyn

--
Check out our winning design in this Bronchoscope Simulator for AirwaySkills (TV coverage).

Berwyn Hoyt, Electronic Solutions & Business  --  Brush Technology
Ph: +64 3 359 2101     Mobile: +64 21 045 7830
Web: brush.co.nz

[fabr...@googlecode.com]

Comment #1 on issue 26 by comexk: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

Improved version that handles more cases.

Attachments:
interposing-001.diff 5.8 KB

[fabr...@googlecode.com]

Comment #2 on issue 26 by berh...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

Hi comexk. Nice work. I have been trying to contact you about these patches
before accepting them. Please email me. My address is at:
http://brush.co.nz/contact -- Berwyn

[fabr...@googlecode.com]

Updates:
Labels: -Type-Defect Type-Enhancement

Comment #3 on issue 26 by simon.al...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

I would like to apply this patch to fabricate to support strace like
functionality on OSX. The problem I have is that I don't have access to an
OSX system to test on.

Are there any volunteers to test the patch on OSX. Failing that, are there
any OSX machines that I could remote access, or be donated to developing
this patch? Something like http://virtualmacosx.com/ that is free would be
a solution if any one knows of one. Maybe an unused licence for OSX could
be donated so I could install it as a virtual machine on my windows box.

Any other suggestions are welcome.

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[fabr...@googlecode.com]

Comment #4 on issue 26 by jaros...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

This patch works just fine on Mac OS X 10.8.4 with a trivial example.

[fabr...@googlecode.com]

Comment #5 on issue 26 by jaros...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

As is, the patch doesn't seem to work. Because of the condition:

self.parallel_ok = parallel_ok and is_strace and _pool is not None

The build always runs serially. When the 'is_strace' predicate is loosened
to allow the interposing runner, the script hangs on my input. I'll try to
put together a minimal reproduction case.

[fabr...@googlecode.com]

Comment #6 on issue 26 by jaros...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

Inside of '_results_handler' in the second for-loop, there is a test of the
form:

if still_to_do == 0:
if isinstance(a.do, _todo):

If the predicate is updated to:

if still_to_do == 0 or not no_error:

This will prevent hangs on OS X.

[fabr...@googlecode.com]

Comment #7 on issue 26 by jaros...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

On further debug I've ascertained that the dependency file is not being
properly generated. I don't know if this is an issue with the Interposer,
or what. I will continue to dig in.

[fabr...@googlecode.com]

Comment #8 on issue 26 by jaros...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
http://code.google.com/p/fabricate/issues/detail?id=26

The 'dirs' option was not being set. Whole thing works with the following
two changes. The first change allows the interposer to run in parallel
mode. The second change allows scripts to be built using fabricate.py as a
pseudo-library. (Obviously that is a hack, but it gets the job done, for
now.)

@@ -1153,7 +1152,8 @@ class Builder(object):
self.runner = SmartRunner(self)

is_strace = isinstance(self.runner.actual_runner(), StraceRunner)
- self.parallel_ok = parallel_ok and is_strace and _pool is not None
+ is_interpose = isinstance(self.runner.actual_runner(),
InterposingRunner)
+ self.parallel_ok = parallel_ok and (is_strace or is_interpose) and
_pool is not None
if self.parallel_ok:
global _results
_results = threading.Thread(target=_results_handler,
@@ -1696,7 +1696,7 @@ def main(globals_dict=None, build_dir=None,
extra_options=None, builder=None,
if not options.quiet and os.path.abspath(build_dir) !=
original_path:
print "Leaving directory '%s' back to '%s'" % (build_dir,
original_path)
os.chdir(original_path)
- sys.exit(status)
+ return status

[Simon Alford]

Thanks for the info on this. I will look at applying the fixes you suggest.

I currently have no way of testing OSX features or fixes. So will let you know when they are available to test.

You received this message because you are subscribed to the Google Groups "fabricate users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabricate-users+unsubscribe@googlegroups.com.
To post to this group, send email to fabricate-users@googlegroups.com.
Visit this group at http://groups.google.com/group/fabricate-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Rosen Matev]

Hi Simon,

Do you have any progress on OSX support? I am very interested in using fabricate on my main workstation which runs OSX. I am willing to help with testing.

Best regards,

Rosen

To unsubscribe from this group and stop receiving emails from it, send an email to fabricate-use...@googlegroups.com.
To post to this group, send email to fabrica...@googlegroups.com.

[Simon Alford]

Hi Rosen,

Not done much on OSX support recently. There are some patches in the git repository on an OSX specific branch. They are not merged because I have no ability to test them. I don't have access to any OSX machines.

If you test the branch at all let me know how it goes. If I remember correctly one of the patches I added should give you dependency calculation via system call interception much like the strace version does on Linux.

Simon

[fabr...@googlecode.com]

Comment #9 on issue 26 by daniel.l...@gmail.com: OSX support with
DYLD_INSERT_LIBRARIES/interposing
https://code.google.com/p/fabricate/issues/detail?id=26

Great job with this patch, thanks folks.

Working for me without too much effort in OSX 10.9, running in VirtualBox,
on a non-parallel build (haven't tried parallel building).

I had one problem. During a link command, with one of the last object files
that had a fairly long path name, the line for it in the Interposer's
output got split up like this (being caught by an assertion when Fabricate
tried to read it in at the end):

[lots of !read.open... lines]
!read.open /Volumes/[bunch more
subpaths]/debug/text_old_fontRenderers_F!read.open /Volumes/[bunch more
subpaths]/debug/libDanDebug.a
!write.open /Volumes/[bunch more subpaths]/debug/libDanDebug.a
!read.stat /Volumes/[bunch more subpaths]/debug/libDanDebug.a
!write.open /Volumes/[bunch more subpaths]/debug/libDanDebug.a
romGlTextureFontToGlSurface.o
[a handful more !read.open... lines]

So, the info about the linker's file writing sort of jumping in there
before all the info about its file reading had finished printing out.
(Again, I should note that this wasn't a parallel build). I
added "fflush(outfp);" after each fprintf to outfp in the Interposer code
and that fixed it, moving all the libDanDebug.a stuff in that example down
to the bottom under all of the !read.open lines.