Why I can't split a static library?

[Frederick Gotham]

I added a static library to my project (Linux ".a" file), but then I went to commit the changes to a repository, but the commit failed because the ".a" file was too big.

So first I split the static library file into individual object files:

ar -x libwhatever.a

And then I moved the files into four different folders, and made four library files:

cd 1
ar -r lib1.a *
cd ../2
ar -r lib2.a *
cd ../3
ar -r lib3.a *
cd ../4
ar -r lib4.a *

And so finally I linked my project with these 4 files.

The linking failed, there were undefined references.

Is there something I'm missing here?

[Ian Collins]

On 25/10/2019 22:21, Frederick Gotham wrote:
>
> I added a static library to my project (Linux ".a" file), but then I
> went to commit the changes to a repository, but the commit failed
> because the ".a" file was too big.

So don't do that. There's seldom a need to commit generated files to SCM.

--
Ian.

[Öö Tiib]

On Friday, 25 October 2019 12:21:47 UTC+3, Frederick Gotham wrote:
> I added a static library to my project (Linux ".a" file), but then I went to commit the changes to a repository, but the commit failed because the ".a" file was too big.

Git isn't designed as storage of frequently changing large binary
files. Common solutions are:
1) avoid doing that, store the textual source from what such
files are compiled or generated.
2) use git LFS (literally "Large File Storage") extension for large
binary files.

Why you had undefined references to symbols is impossible
to tell without actual reproducible example. Linkers are rather
dumb so perhaps you do not put the libraries in correct order
or there is cyclic dependency between those.

[Paavo Helde]

First, it appears that you did not read the error messages and did not
find out which piece is missing which symbols and where they should come
from. So we must use our crystal balls here and figure this out in blind.

Anyway, the most obvious error would be to link the pieces in the wrong
order. Beware that in case of circular dependencies there might not be a
right order, maybe one could solve it by listing the pieces twice.

[Frederick Gotham]

On Friday, October 25, 2019 at 1:21:19 PM UTC+1, Paavo Helde wrote:

> Anyway, the most obvious error would be to link the pieces in the wrong
> order.

Thank you, this hadn't crossed my mind. A function in "1.a" must be looking for a function that resides in "2.a".

[Frederick Gotham]

On Friday, October 25, 2019 at 12:21:25 PM UTC+1, Ian Collins wrote:

> So don't do that. There's seldom a need to commit generated files to SCM.

I am deliberately obfuscating my code. I'm working on a security product, and so after I produce my binary executables and libraries, I run them through a decompiler and try to see how difficult it would be to reverse-engineer.

My program would be too easy to reverse-engineer if I linked the library dynamically, so I'm linking it statically and obfuscating it as best I can.

[Jorgen Grahn]

I'll not comment on obscurity as a technique for security, but what
you write is orthogonal to what Ian Collins wrote. You don't have to
commit libfoo.a binaries to SCM in order to link statically.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[David Brown]

On 25/10/2019 15:31, Frederick Gotham wrote:
> On Friday, October 25, 2019 at 12:21:25 PM UTC+1, Ian Collins wrote:
>
>> So don't do that. There's seldom a need to commit generated files
>> to SCM.
>
>
> I am deliberately obfuscating my code. I'm working on a security
> product, and so after I produce my binary executables and libraries,
> I run them through a decompiler and try to see how difficult it would
> be to reverse-engineer.
>

My initial thoughts are that obfuscating your code is a bad idea, a
misunderstanding of what is important for security, and even if it was a
good idea, then it does not sound like a good way to achieve this. My
initial thoughts may be wrong, as I don't know anything more than the
short paragraph above, but "security by obscurity" is rarely a good idea.

> My program would be too easy to reverse-engineer if I linked the
> library dynamically, so I'm linking it statically and obfuscating it
> as best I can.
>

My recommendation would be to avoid the library altogether - certainly
do not consider it a separate item to be produced and checked into an
SCM (the clue is in the name - "Source Control Manager").

Rather, you should consider using link-time optimisation. With
link-time optimisation, high compiler optimisation levels, no debugging
information, careful control of elf symbol visibility, and stripped
executables, your generated code will be incomprehensible. I doubt if
any other kind of obfuscation would make a measurable difference - and
yet you still have clear and maintainable source code. (You might also
want to disable RTTI.)

[Vir Campestris]

Put the sources in the SCM. You'll need them next time you do a rebuild.

And your security will last until two things happen:
- You protect something of sufficient importance
- About 3 more weeks pass.

Been there, done that, got the tee shirt. We were making several
releases a year. Luckily security of a few weeks was saleable.

Andy

[Juha Nieminen]

It might not be generated code. Sometimes third-party libraries (especially
closed-source, or semi-closed-source ones) may be distributed in binary
form. Or sometimes the static library may indeed be built from the project
files themselves, but this might take a very long time (like hours), and
it might be convenient for other members of a project not to have to do that.

Of course there are alternatives even in the first case (other than just
putting instructions that tell other team members where to download the
library and how to install it in the project directories), for example
by using some kind of build script that automatically downloads the
library and puts it in the proper place. But it can become complicated.

[Frederick Gotham]

On Friday, October 25, 2019 at 4:20:56 PM UTC+1, David Brown wrote:

> Rather, you should consider using link-time optimisation. With
> link-time optimisation, high compiler optimisation levels, no debugging
> information, careful control of elf symbol visibility, and stripped
> executables, your generated code will be incomprehensible. I doubt if
> any other kind of obfuscation would make a measurable difference - and
> yet you still have clear and maintainable source code. (You might also
> want to disable RTTI.)

I'm taking apart the library and using

__attribute__((always_inline))

pretty much everywhere. Then I intend to use every form of optimisation there is, both at compile time and at link time.

The level of security is 'airport'. I can't tell you exactly what I'm doing, but suffice to say I'm trying to secure something that could fall into the wrong hands.

I'm compiling my program and running it through two different decompilers, and then also I will do stuff like monitor the processes and file system while my program's running. And of course a simple hexdump for the static duration data.

A team of hackers paid by a government can try to reverse-engineer my project, but I will have changed everything by the time they figure it out (if they ever do).

[Jorgen Grahn]

On Mon, 2019-10-28, Frederick Gotham wrote:
> On Friday, October 25, 2019 at 4:20:56 PM UTC+1, David Brown wrote:
>
>> Rather, you should consider using link-time optimisation. With
>> link-time optimisation, high compiler optimisation levels, no debugging
>> information, careful control of elf symbol visibility, and stripped
>> executables, your generated code will be incomprehensible. I doubt if
>> any other kind of obfuscation would make a measurable difference - and
>> yet you still have clear and maintainable source code. (You might also
>> want to disable RTTI.)
>
>
> I'm taking apart the library and using
>
> __attribute__((always_inline))
>
> pretty much everywhere. Then I intend to use every form of
> optimisation there is, both at compile time and at link time.

I think his point is, if you enable link-time-optimization, you don't
have to do all those other complicated things.

[Scott Lurndal]

Frederick Gotham <cauldwel...@gmail.com> writes:
>On Friday, October 25, 2019 at 4:20:56 PM UTC+1, David Brown wrote:
>
>> Rather, you should consider using link-time optimisation. With
>> link-time optimisation, high compiler optimisation levels, no debugging
>> information, careful control of elf symbol visibility, and stripped
>> executables, your generated code will be incomprehensible. I doubt if
>> any other kind of obfuscation would make a measurable difference - and
>> yet you still have clear and maintainable source code. (You might also
>> want to disable RTTI.)
>
>
>I'm taking apart the library and using
>
> __attribute__((always_inline))
>
>pretty much everywhere. Then I intend to use every form of optimisation there is, both at compile time and at link time.
>
>The level of security is 'airport'. I can't tell you exactly what I'm doing, but suffice to say I'm trying to secure something that could fall into the wrong hands.

If you're (1) talking about it here, and (2) taking advice from the internet
you are in way over your head.

[Frederick Gotham]

On Tuesday, October 29, 2019 at 12:12:41 AM UTC, Scott Lurndal wrote:

> If you're (1) talking about it here, and (2) taking advice from the internet
> you are in way over your head.

I'd agree with you if I didn't know already how a good skilled hacker will go about trying to figure out what my program does.

[David Brown]

If you are working on code which may be targeted by groups with
significant financial or political motivation, significant resources,
and insignificant moral scruples, then Scott is right - you show that
you are /way/ over your head by posting about it. Such groups will have
no problem tracing you, whether your name is Frederick Gotham or Thomas
Cauldwell, and will have no problem persuading you to reveal the secrets
of your code. They don't need to de-compile it - there are faster,
cheaper and more reliable methods when the know who wrote it.

And if your code is not going to be targeted by such resourceful groups,
then your obfuscation is a waste of time, and more likely to introduce
errors in your code than help protect it.


For security, you keep the /key/ secret - not the code or the design of
the lock. You make it secure enough that the cheapest way to break the
coding is by burglary - then you stop well before the cheapest method
becomes rubber hose cryptoanalysis. And you certainly don't advertise
what you are doing - that significantly reduces the cost of the rubber
hose cryptoanalysis.

[Öö Tiib]

On Tuesday, 29 October 2019 15:00:07 UTC+2, David Brown wrote:
> On 29/10/2019 11:26, Frederick Gotham wrote:
> > On Tuesday, October 29, 2019 at 12:12:41 AM UTC, Scott Lurndal
> > wrote:
> >
> >> If you're (1) talking about it here, and (2) taking advice from the
> >> internet you are in way over your head.
> >
> >
> > I'd agree with you if I didn't know already how a good skilled hacker
> > will go about trying to figure out what my program does.
> >
>
> If you are working on code which may be targeted by groups with
> significant financial or political motivation, significant resources,
> and insignificant moral scruples, then Scott is right - you show that
> you are /way/ over your head by posting about it. Such groups will have
> no problem tracing you, whether your name is Frederick Gotham or Thomas
> Cauldwell, and will have no problem persuading you to reveal the secrets
> of your code. They don't need to de-compile it - there are faster,
> cheaper and more reliable methods when the know who wrote it.

It seems that you guys take yet another attempt of this author of
doing something that he claims isn't trolling too seriously. He did
likely read from somewhere that the security experts have rejected
this approach as far back as 1851 as illusionary. For example there:
<https://en.wikipedia.org/wiki/Security_through_obscurity>
So he pretends implementing it and then read replies for "lulz" or
"keks" (or what those being deliberately grotesque say about it).

[Frederick Gotham]

On Tuesday, October 29, 2019 at 2:38:36 PM UTC, Öö Tiib wrote:

> It seems that you guys take yet another attempt of this author of
> doing something that he claims isn't trolling too seriously. He did
> likely read from somewhere that the security experts have rejected
> this approach as far back as 1851 as illusionary. For example there:
> <https://en.wikipedia.org/wiki/Security_through_obscurity>
> So he pretends implementing it and then read replies for "lulz" or
> "keks" (or what those being deliberately grotesque say about it).

You still don't know what I'm doing. Because I haven't told you.

[Vir Campestris]

On 29/10/2019 16:25, Frederick Gotham wrote:
> You still don't know what I'm doing. Because I haven't told you.

My earlier comment stands: Protect something important enough, and your
code will be cracked in weeks. Ours always was.

Assuming the cracker has access to the code of course.

Standard cryptography relies on protecting the keys from malicious agents.

Your code protection obviously expects the program binary to be
available to malicious agents; you are giving them a complex key to analyse.

Andy

[Mr Flibble]

Obfuscation is a totally useless form of security, dear. Basically YOU ARE
DOING IT WRONG (tm). I do hope lives will not be endangered or lost when
your software is released/used due to your ignorant fucktardary.

/Flibble

--
"Snakes didn't evolve, instead talking snakes with legs changed into
snakes." - Rick C. Hodgin

“You won’t burn in hell. But be nice anyway.” – Ricky Gervais

“I see Atheists are fighting and killing each other again, over who
doesn’t believe in any God the most. Oh, no..wait.. that never happens.” –
Ricky Gervais

"Suppose it's all true, and you walk up to the pearly gates, and are
confronted by God," Bryne asked on his show The Meaning of Life. "What
will Stephen Fry say to him, her, or it?"
"I'd say, bone cancer in children? What's that about?" Fry replied.
"How dare you? How dare you create a world to which there is such misery
that is not our fault. It's not right, it's utterly, utterly evil."
"Why should I respect a capricious, mean-minded, stupid God who creates a
world that is so full of injustice and pain. That's what I would say."

[Chris M. Thomasson]

On 11/2/2019 2:32 PM, Vir Campestris wrote:
> On 29/10/2019 16:25, Frederick Gotham wrote:
>> You still don't know what I'm doing. Because I haven't told you.
>
> My earlier comment stands: Protect something important enough, and your
> code will be cracked in weeks. Ours always was.

Fwiw, the games Dungeon Master and Crystal Dragons had some interesting
copy-protection:

http://dmweb.free.fr/?q=node/210
(interesting read...? ;^)

DM had a fuzzy bit that made it hard for the crackers. It was able to
sell a lot of copies before it was busted.

https://www.myabandonware.com/game/crystal-dragon-77e

Most of the game code was encrypted. DM had hidden data inside of some
of the graphics.

Iirc, Starcraft had a hidden track on its CD that would not copy.

[Chris M. Thomasson]

On 11/2/2019 10:59 PM, Chris M. Thomasson wrote:
> On 11/2/2019 2:32 PM, Vir Campestris wrote:
>> On 29/10/2019 16:25, Frederick Gotham wrote:
>>> You still don't know what I'm doing. Because I haven't told you.
>>
>> My earlier comment stands: Protect something important enough, and
>> your code will be cracked in weeks. Ours always was.
>
> Fwiw, the games Dungeon Master and Crystal Dragons had some interesting
> copy-protection:
>
> http://dmweb.free.fr/?q=node/210
> (interesting read...? ;^)
>
> DM had a fuzzy bit that made it hard for the crackers. It was able to
> sell a lot of copies before it was busted.

[...]

Here is the patent:

https://patents.google.com/patent/US4849836

[David Brown]

On 03/11/2019 06:59, Chris M. Thomasson wrote:
> On 11/2/2019 2:32 PM, Vir Campestris wrote:
>> On 29/10/2019 16:25, Frederick Gotham wrote:
>>> You still don't know what I'm doing. Because I haven't told you.
>>
>> My earlier comment stands: Protect something important enough, and
>> your code will be cracked in weeks. Ours always was.
>
> Fwiw, the games Dungeon Master and Crystal Dragons had some interesting
> copy-protection:
>
> http://dmweb.free.fr/?q=node/210
> (interesting read...? ;^)
>
> DM had a fuzzy bit that made it hard for the crackers. It was able to
> sell a lot of copies before it was busted.
>
> https://www.myabandonware.com/game/crystal-dragon-77e
>
> Most of the game code was encrypted. DM had hidden data inside of some
> of the graphics.
>
> Iirc, Starcraft had a hidden track on its CD that would not copy.
>

Successful copy protection always relied on something physical, such as
intentionally bad CD tracks, or asking the user to look up a page, line,
word reference in the manual.

Andy is right - you protect the keys, not the code. The bad CD track is
the key here.