Help with organization of CLOS code: classes, defmethods

[jeosol]

Hi Guys,

I am currently reading the OOP in CL book by Sonja E Keene and practicing with a small CLOS based application. It is anticipated that the application will grow so I wanted to explore what are the best practices for code organization. I will explain what I mean by this shortly using an example.

For now, I am using a single package.lisp for say a module which contain several classes. I read somewhere that it is recommended or better (?) to use single defpackage per file. So I wanted to understand the merit of a single defpackage (for the module) defined in a package.lisp file vs. multiple defpackages per file, before this grows and (potentially) becomes difficult to manage.

The application uses inheritance and I want to be able to call functions on object instances and have them dispatch correctly. For example, take the following simple example (under a single package).

--------------------------------------------------------------
;; Base class for all animals
(defclass animal ()
((name :reader name :initform nil :initarg :name)))

;; Class of herbivores
defclass herbivore (animal) ())

(defun make-herbivore (name)
(make-instance 'herbivore :name name))

(defmethod eats-what ((a herbivore))
(format nil "~&Herbivore animal ~a eats grass" (name a)))

;; Class of carnivores
defclass carnivore (animal) ())

(defun make-carnivore (name)
(make-instance 'carnivore :name name))

(defmethod eats-what ((a carnivore))
(format nil "~&Carnivore animal ~a eats grass" (name a)))

Having defined those classes, I want to be able to write code like this:

(defun run-test ()
(let* ((animal-a (make-herbivore "cow"))
(animal-b (make-carnivore "tiger")))

(print (eats-what animal-a))
(print (eats-what animal-b)))
'done)
This works okay fine for now.
--------------------------------------------------------------

As I continue to add functions to each derived class, I will eventually move them to their individual files, e.g., animal.lisp, carnivore.lisp, and herbivore.lisp but with the same package across the files. This should still work okay.

However, when I use different package names for each of the classes and have to import the eats-what functions, I get name conflicts as expected.

What is the best way to deal with this with the per-source defpackage option and still be able to write code like run-test above (which could be in a different package). Is this is an overkill? Also, are there other factors I need to be aware or consider.

Thanks.

[Jinwoo Lee]

In general, you should define a DEFGENERIC for eats-what in the same package as the animal class.

Each sub-class must prefix eats-what with the package name of the animal class:

;; In herbivores.lisp
(defmethod animal:eats-what ((a herbivore))
...)

[jeosol]

Jinwoo Lee,

Thanks for the tip.

I have split the classes to their separate files, kept the defgeneric in the base case, and prefixed the eats-what defmethod with the base class' package name. It works correctly with my test.

[Mart van de Wege]

jeosol <jeron...@gmail.com> writes:

> As I continue to add functions to each derived class, I will
> eventually move them to their individual files, e.g., animal.lisp,
> carnivore.lisp, and herbivore.lisp but with the same package across
> the files. This should still work okay.
>
> However, when I use different package names for each of the classes
> and have to import the eats-what functions, I get name conflicts as
> expected.

Why would you want to use different package names for the classes? The
package is just that: a package. You use it to package up related
classes, regardless of the fact if these classes are in single files or
one big file.

You create new packages if there is a clean break in functionality, say
for example in an MVC situation, where I'd break up the packages in a
Model, View and Controller package (and probably a single meta-package
holding an application).

Mart

--
"We will need a longer wall when the revolution comes."
--- AJS, quoting an uncertain source.

[Pascal J. Bourguignon]

Instead of spliting files by class, it would be better to split them by
feature (use case, user story, whatever).

Notice how in Common Lisp we have multimethods, so you cannot assign a
given method to a single class, so you couldn't find a file where to
store it if you use one file per class!



(defgeneric eat (animal food))

(defclass food () (…))
(defclass cheese (food) (…))
(defclass animal (food) (…))
(defclass cat (animal) (…))
(defclass mouse (animal) (…))

(defmethod eat ((a cat) (f mouse)) …)
(defmethod eat ((a mouse) (f cheese)) …)


It would be better to store the eat method in the feeding.lisp file, and
the move method in the moving.lisp file.

(But really, the best would be to let emacs store things wherever it
wants, so you don't have to deal with files anymore).

--
__Pascal J. Bourguignon
http://www.informatimago.com

[Marco Antoniotti]

On Saturday, May 20, 2017 at 9:10:06 AM UTC+2, Mart van de Wege wrote:
> jeosol <jeron...@gmail.com> writes:
>
> > As I continue to add functions to each derived class, I will
> > eventually move them to their individual files, e.g., animal.lisp,
> > carnivore.lisp, and herbivore.lisp but with the same package across
> > the files. This should still work okay.
> >
> > However, when I use different package names for each of the classes
> > and have to import the eats-what functions, I get name conflicts as
> > expected.
>
> Why would you want to use different package names for the classes?

Java Newspeak is the best answer to this question :)
--
MA

[jeosol]

Thanks to @Mart, @informatigo for the contributions.

My original point of putting each class in a file is just my first thought/pass but you guys provide other important considerations. I see the limitation of one class per file that method when I have defmethods with multiple classes as arguments. At this point, I am more concerned about best organization option.

@Mart, the reason I considered using different packages for each file is following common lisp style guide (see http://labs.ariel-networks.com/cl-style-guide.html). In that case, I can only import relevant functions into each file. This may be an overkill though. In an old code, I used a single package method and it was sure a lot simpler to deal with. I am trying to find out merits, e.g., faster compilation of one option vs. the other.

[Mart van de Wege]

jeosol <jeron...@gmail.com> writes:

> Thanks to @Mart, @informatigo for the contributions.
>
> My original point of putting each class in a file is just my first
> thought/pass but you guys provide other important considerations. I
> see the limitation of one class per file that method when I have
> defmethods with multiple classes as arguments. At this point, I am
> more concerned about best organization option.
>
> @Mart, the reason I considered using different packages for each file
> is following common lisp style guide (see
> http://labs.ariel-networks.com/cl-style-guide.html).

That is not *the* CL style guide. It reads more like a Java programmer
trying to ram the round CL peg into a square Java hole.

> In that case, I can only import relevant functions into each
> file. This may be an overkill though. In an old code, I used a single
> package method and it was sure a lot simpler to deal with. I am trying
> to find out merits, e.g., faster compilation of one option vs. the
> other.

If it helps the organisation of the code, by all means define as many
packages as you like. But don't constrain yourself artifically. When
dealing with package dependencies becomes more of a hassle than just
putting classes in different files and just compile them all into one
package, just do that.

[Pascal J. Bourguignon]

jeosol <jeron...@gmail.com> writes:

> Thanks to @Mart, @informatigo for the contributions.
>
> My original point of putting each class in a file is just my first
> thought/pass but you guys provide other important considerations. I
> see the limitation of one class per file that method when I have
> defmethods with multiple classes as arguments. At this point, I am
> more concerned about best organization option.

It really depends on your problem.

Take for example this lambda-list parser:

https://framagit.org/com-informatimago/com-informatimago/blob/master/common-lisp/lisp-sexp/source-form.lisp#L576

There are multiple variants of lambda-lists, that are defined as
subclasses of lambda-list, but also of a mixin defining the various
groups of attributes a lambda-list can have.

It would be quite inconvenient to split those classes over multiple
files, when the block of defclass forms from 585 to 594 defines all the
variants in a clear and declarative way.

Notice also that quite a number of generic functions are defined with
all their methods at once, and again, it would be much harder to
maintain consistency if they were split out into defmethod in separate
files.

> @Mart, the reason I considered using different packages for each file
> is following common lisp style guide (see
> http://labs.ariel-networks.com/cl-style-guide.html). In that case, I
> can only import relevant functions into each file. This may be an
> overkill though. In an old code, I used a single package method and it
> was sure a lot simpler to deal with. I am trying to find out merits,
> e.g., faster compilation of one option vs. the other.

In general I don't like style guides because they take into account only
one or a few use cases, and they may not match your current situation.

When you're a newbie and you're developing code that falls in the coding
style use case, or you're paid to follow it, you should certainly follow
it, but if this condition is false, you should rather use your brains.

Notice also how coding style guides are given in the form of rules
generaly more formal than less. If you can formalize rules, then you
can implement software to enact them. See for example slime indent and
paredit. If you have software implementing some rules, don't try to
fight it (and do use those software, because most other programmers will
use them, and unfortunately, currently editors and IDE don't reformat
code upon loading to follow the taste of the reader).

But it should tells you something: that coding rules are not to be
followed by human beings! Style is a question of taste, and cannot be
formalized, and foremost, cannot be generalized.


For example, this program has perfectly good style:

http://www.99-bottles-of-beer.net/language-common-lisp-114.html

(And notice that slime indents it incorrectly, since it insists on
aligning STREAM and the first line of the string literal with FORMAT).

[Kaz Kylheku]

On 2017-05-22, Mart van de Wege <mvd...@gmail.com> wrote:
> jeosol <jeron...@gmail.com> writes:
>
>> Thanks to @Mart, @informatigo for the contributions.
>>
>> My original point of putting each class in a file is just my first
>> thought/pass but you guys provide other important considerations. I
>> see the limitation of one class per file that method when I have
>> defmethods with multiple classes as arguments. At this point, I am
>> more concerned about best organization option.
>>
>> @Mart, the reason I considered using different packages for each file
>> is following common lisp style guide (see
>> http://labs.ariel-networks.com/cl-style-guide.html).
>
> That is not *the* CL style guide. It reads more like a Java programmer
> trying to ram the round CL peg into a square Java hole.

Be that as it may, ASDF now supports "one package per file" organization:

https://common-lisp.net/project/asdf/asdf/The-package_002dinferred_002dsystem-extension.html

[Mart van de Wege]

Supporting it and mandating it in a style guide are different things. I
also object to that guide heavily promoting annotations and discouraging
macro usage.

If you look at the code examples they give, that looks more like Java or
maybe Python than actual Common Lisp.

Nothing wrong with using tools if they exist, of course. But IMO that
guide is going too far.

[Kaz Kylheku]

On 2017-05-22, Mart van de Wege <mvd...@gmail.com> wrote:
> Kaz Kylheku <686-67...@kylheku.com> writes:
>
>> On 2017-05-22, Mart van de Wege <mvd...@gmail.com> wrote:
>>> jeosol <jeron...@gmail.com> writes:
>>>
>>>> Thanks to @Mart, @informatigo for the contributions.
>>>>
>>>> My original point of putting each class in a file is just my first
>>>> thought/pass but you guys provide other important considerations. I
>>>> see the limitation of one class per file that method when I have
>>>> defmethods with multiple classes as arguments. At this point, I am
>>>> more concerned about best organization option.
>>>>
>>>> @Mart, the reason I considered using different packages for each file
>>>> is following common lisp style guide (see
>>>> http://labs.ariel-networks.com/cl-style-guide.html).
>>>
>>> That is not *the* CL style guide. It reads more like a Java programmer
>>> trying to ram the round CL peg into a square Java hole.
>>
>> Be that as it may, ASDF now supports "one package per file" organization:
>>
>> https://common-lisp.net/project/asdf/asdf/The-package_002dinferred_002dsystem-extension.html
>
> Supporting it and mandating it in a style guide are different things. I
> also object to that guide heavily promoting annotations and discouraging
> macro usage.

I saw that style guide; someone linked to it in a HackerNews
comment. It's imbecillic garbage.

The annotations are crap that nobody in their right mind should use.

[Rainer Joswig]

I would not recommend the "a file is a package" idea. One can do that,
but it often leads to the modern cancer of zillion file projects with
zillion packages. I think it is perfectly fine to have a package which
is used in multiple files and have a different granularity.

I would also not use other packages, but usually I wouldn't even import
individual symbols. Often I prefer to mention the full package name.
Not always, but often.

The annotation feature is pure garbage. The arguments not to use
defwicket is also wrong.
I now would have to use a wicket annotation and an embedded class defintion.
Please not. For me, one of the basic rules of Lisp is that forms should
nest. Annotations don't do that.
Annotations are then a somehow new form of code transformers, while
macros are already built in.

Comments are for the writer of the program? Comments are for all who read it.

Adding :type to slots is great, but one better make sure that the Lisp
enforces them at runtime then.
And a lot of implementations don't. You could end up in a situation
where a lot of these type declarations are wrong.

(cost :type (or integer null)
:initarg :cost)

In above it is not clear what NIL would be supposed to be as a cost.

'Avoid macros' -> avoid Lisp, use something else.

[Jocelyn Fréchot]

On 22/05/2017 05:13, jeosol wrote:

> [..] following common lisp style guide (see http://labs.ariel-networks.com/cl-style-guide.html).

You should rather have a look at this one (although it doesn’t address
the one package per file issue):

https://google.github.io/styleguide/lispguide.xml

You don’t have to agree with everything it says but it’s a good starting
point to think about what “good style” is for you.

--
Jocelyn Fréchot

[Marco Antoniotti]

Weeeeeeelll. ALL style guides I know of say 80 columns. Not 100 3:)

Cheers
--
MA

[smh]

No, only 72 columns! Someday you'll regret if you don't maintain card sequence numbers in columns 73-80!!!

https://upload.wikimedia.org/wikipedia/commons/5/58/FortranCardPROJ039.agr.jpg

[Marco Antoniotti]

On Wednesday, May 24, 2017 at 12:12:30 PM UTC+2, smh wrote:
> No, only 72 columns! Someday you'll regret if you don't maintain card sequence numbers in columns 73-80!!!
>
> https://upload.wikimedia.org/wikipedia/commons/5/58/FortranCardPROJ039.agr.jpg

I second that :)

Cheers
--
MA

[Marco Antoniotti]

On Wednesday, May 24, 2017 at 12:12:30 PM UTC+2, smh wrote:

> No, only 72 columns! Someday you'll regret if you don't maintain card sequence numbers in columns 73-80!!!
>
> https://upload.wikimedia.org/wikipedia/commons/5/58/FortranCardPROJ039.agr.jpg

Of course, nothing should go in columns 1-7 :)

Cheers
--
MA

[Kaz Kylheku]

That's easy to meet; just write the usual stuff like this:

(xxxx
(xxxx xxx
xxxx
(xxx ... )))

Then, add enough leading spaces to those lines that don't already have
seven, so that they have seven:

(xxxx
(xxxx xxx
xxxx
(xxx ... )))

[Pascal J. Bourguignon]

I prefer:

;;;000000111111111122222222223333333333444444444455555555556666666666777
;;;456789012345678901234567890123456789012345678901234567890123456789012

(xxxx
(xxxx xxx
xxxx
(xxx ... )))

[Jocelyn Fréchot]

On 24/05/2017 09:55, Marco Antoniotti wrote:

> Weeeeeeelll. ALL style guides I know of say 80 columns. Not 100 3:)

80? Are you crazy? You really never use diff?
I’ll keep using 78 columns :P


> Cheers
> --
> MA
>


--
Jocelyn Fréchot

[Marco Antoniotti]

On Friday, May 26, 2017 at 1:15:14 AM UTC+2, Jocelyn Fréchot wrote:
> On 24/05/2017 09:55, Marco Antoniotti wrote:
>
> > Weeeeeeelll. ALL style guides I know of say 80 columns. Not 100 3:)
>
> 80? Are you crazy? You really never use diff?
> I’ll keep using 78 columns :P

+1

A man after my own heart :)

Cheers
--
MA

[ken.t...@apexdatasolutions.net]

Do not use one defpackage per file. In fact, do not get carried away with packages. I use one per project, with other packages only for libraries external to the project. For example, I use my Cells and qooxlisp libraries widely, so they pretty much need their own packages (or I would be adding their symbols to cl-user).

People who go crazy over packages are either OCD or Java (or Clojure) developers where every file is a namespace.

If you insist on lotsa packages, define the symbol in the most generic package/class (animal?) and inherit from that package in the packages for diff species.

But really, forget all these packages.

-kt

--
*This e-mail message and its attachments are for the sole use of the
designated recipient(s). They may contain confidential information,
legally privileged information or other information subject to legal
restrictions. If you are not a designated recipient of this message,
please do not read, copy, use or disclose this message or its attachments,
notify the sender by replying to this message and delete or destroy all
copies of this message and attachments in all media. Thank you.*

[jeosol]

I asked this question about organization of a large CLOS-based application several months ago and wanted to give an update.

As for the organization part, I settled on using ":class :package-inferred-system" in my defsystems *.asd files and "all" my lisp files (> 90% for now) have single packages with defpackage defined at the top of each file.

The refactoring was a bit painful as I have to import all functions used in a single file, but it's worth it in the sense that my compilations are faster (I use SBCL). However, the benefit is that it's easier to add new code using this style as I know what to do upfront.

Also, my files are better organized in directories as I fully qualify the package names with directory structure, and for each file, functions used in that file are imported from their respective packages.

In addition, I use long file names that describe the intent or content of the file itself.

Another change I made was split files by class, feature, or use case as suggested here. For my defmethods, I tie my defmethods to a defgeneric defined in the base class, and prefix the derived class defmethods by the package name of the base class as suggested by Jinwoo. Most of my cases have single inheritance structure.

I made this change over the last few months. This is the repository update as at August https://www.youtube.com/watch?v=gqoGpXJTY4w

I write code daily and it's easier to manage now because of the above code reorganizations.

This may not be the "best" approach, but it gets me to focus on moving things forward for now.

So, thanks to all you guys for the useful feedback, comments, and suggestions.