Process to arrive a good design that separates general & special purpose code

[Venkat Dinavahi]

I sometimes end up with code that is ~90% general and ~10% specialized.

For example, take a document viewer which is mostly general purpose code, with some capabilities that are specific to our application such as viewing our own proprietary documents.

In theory separating the general code reduce the complexity of my system.

But in practice I end up separating things incorrectly. And if I try to figure out the APIs upfront, I end up in some sort of analysis-paralysis.

How do you arrive at a system with good modularity? Do you accept that the first iteration will be wrong, write messy code, then re-draw boundaries until you're satisfied? Do you follow an upfront design process? Somewhere in between?

[Ciaran McHale]

I am not aware of there being a fully-generalized approach for dealing with "90% general/boilerplate and 10% specialized" code bases. However, there are some well-known techniques that can be used in limited circumstances. I will mention some of those, and then I will talk about something I have been working on (on-and-off) since 1996 and which I hope will eventually see the light of day as a very mature, field tested and documented set of development tools. This is not an attempt to promote my own vapourware work, but just a way of saying that "current tools for what you want help only partially, but I expect future tools to help a bit more (yet still not help in 100% of scenarios)".

Overview of existing approaches

--------------------------------------------

One well-known technique is to put the 90% general code into a collection of base classes, and then use subclasses to add the 10% specialized code.

Another well-known technique is to write a framework that encapsulates the 90% general code, and then a programmer will put the 10% specialized code into "callback" objects/functions that get registered with the framework. I don't have much experience with developing GUI applications or web servers, but my understanding is that the use of frameworks is common in those domains.

The implementation of the interpreter for the Tcl scripting language uses a framework approach because its core has a hash table that maps name-of-command to C-function-that-implements-the-command, and this enables somebody to implement new functionality via a C function and then register this with the Tcl interpreter's hash table. I assume some other extendable scripting languages, e.g., Python, use a similar-ish technique.

Overview of some potential future approaches

-------------------------------------------------------------

I have written an open-source configuration-file parser library called Config4*. It is available in 2 language flavours: C++ (Config4Cpp) and Java (Config4J). You can find it, with comprehensive documentation on http://www.config4star.org. One (short) chapter in the documentation explains how I used (a predecessor of) Config4*, plus something else that I call "code segment files", in a code generation tool, and the result enabled me to generate 100% of the code for a large application, despite the application code consisting of 90% general-ish code and 10% specialized code. Here is a link to the relevant chapter: http://www.config4star.org/config4star-practical-usage-guide/code-generation.html

My current, incomplete work in this area involves me developing C++ and Java-based tools and libraries that, between them, will greatly simplify the effort required to implement a code generation tool, plus libraries that make it simple to work with, and achieve reuse of, metadata and code segment files. It is likely to be several more years before I release this work publicly.

I am attaching a file that is a chapter from an unfinished book I am writing about this topic. The book is not exclusively about my work in code generation, but also discusses some alternative/competing approaches. The attached chapter gives a brief overview of a particular GUI-based tool for developing code generators.

My attitude regarding the development of code generators for creating  "90% general/boilerplate code plus 10% specialized code"-type applications is that a relatively small number of people, including me and the company behind MetaEdit+, are pioneers in this field and a lot more experimentation needs to be done before a "best of breed" approach will be discovered and become widely used.

Regards,

Ciaran.

-- Ciaran McHale (ciaran...@yahoo.co.uk)
www.CiaranMcHale.com
Mobile: +44-(0)7866-416-134
Home: +44-(0)118-327-9847

--
You received this message because you are subscribed to the Google Groups "software-design-book" group.
To unsubscribe from this group and stop receiving emails from it, send an email to software-design-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/software-design-book/90882cab-62d4-45f6-8da6-0c51bed56e04n%40googlegroups.com.

[Daniel T.]

The real question here, IMO, is how do you know, up front, that you will end up with 90% general and 10% specialized code? Not knowing this makes it rather difficult to do anything other than just writing code and figuring out how to properly modularize as you go along.

After all, if you know up front that it's going to be 90% general, and you know exactly which bits will be part of that 90%, figuring out the APIs up front would be easy.

But how then do you arrive at a system with good modularity? (1) separate effects from logic. This will allow you to test the logic without dealing with side effects. (2) separate features from each other. An architecture that separates features from each other allows you to add/update features easily without affecting other features. (You can only take this so far, some features depends on others.)

I find that the best way to follow these two rules is to first define your features along the lines of "When X side-effect happens, perform Y effect according to Z logic." The Z logic observes the X effect and the Y effect observes the Z logic. Often Z logic will take the form of a state machine.

The above is what works for me.

[John Ousterhout]

I'm joining this thread late... sorry.

In my experience, it's hard to arrive at a system with good modularity; I virtually never get it right the first time, even after all these years. My advice:

* Try to think things through up-front, but don't get stuck in analysis paralysis. Once you stop getting new ideas that could impact the design, it's time to start implementing.

* You will discover problems as you implement. It simply isn't possible to visualize all of the implications of a design before you start building it. Be prepared to redesign when you find problems. It typically takes me 2-3 iterations (one major, plus 1-2 more minor ones) to get to a sweet spot for an API.

* In your initial design, come up with 2 radically different alternatives. You'll learn a lot from comparing the pros and cons of each of the alternatives.

If you follow this approach (and the iterative part is key) you may still struggle to get to a good spot. But as you do this over and over and your experience grows, your skills will improve and I think you'll find you can reach a happy place more quickly.

-John-

--

You received this message because you are subscribed to the Google Groups "software-design-book" group.
To unsubscribe from this group and stop receiving emails from it, send an email to software-design-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/software-design-book/b184046b-12b6-4f3e-90d8-bddcc4b713b8n%40googlegroups.com.

[Daniel T.]

I think the iteration idea is extremely important. My wife is an English Professor. She is constantly stressing to her students the importance of writing several drafts, then doing peer review, before settling on a final draft.

Developers use different words to describe this but the idea is the same. Once the code for that feature works, that's not final code, it's your first draft. Go through at least one more draft (refactoring,) then code review and then yet another draft (more refactoring.)

When I'm writing code, even for stack overflow answers, I write several drafts over the course of a day. Write the code, go work on some other feature for a bit, then come back and rewrite the code... Then do it again if I get more ideas later. Even after I've published the feature for QA, I may still tweak it.

[Gregg Irwin]

I'm keen on iteration, whether for applications, systems (many apps working together), UIs, language/DSL/function design or pretty much anything. Of course, I doubt I'm the only one here who also overthinks things because I want so badly to get it right the first time. 

To continue the writing analogy, stories have a beginning, middle, and end. If you zoom in, so do sections, and chapters. Screenplays, in particular, generally follow both a very specific format and a codified structure. The structure goes from high level to low: 3 act model, 4 major turning points, 15 beats, 40 scenes that combine into ~20 sequences. And it uses a DSL (e.g. INT. HOUSE - DAY, EXT. FIELD - NIGHT).  There's a whole industry around it, and even more detail because it's used to communicate to everyone involved; director, actor, lighting, sound, location, FX, budget. Something I think we can learn from. Whether your brain and team lean more Waterfall, where you work top down before you start writing, or do a vomit draft to prime the pump and then rework it, knowing there is likely a structure that works well, and makes it easier for others to understand is valuable.

I know some will raise GoF and patterns, but this is a different level of thinking. Those patterns still apply, just as SMASH CUT is a general transition idea in a movie.

In a different domain, I have a friend who for many years sold big ERP systems. He said every business thought they were unique. He had to first explain to them that if that were true, ERP systems wouldn't exist. 

[Simon Frankau]

I think this is a fascinating thread, and can't resist adding my experience.

The version I have experienced is the split between "business logic" and "infrastructure/libraries/generic bits".

As well as the split in purpose, there tends to be maintainability differences. I find the business logic layer tends to have quickly moving requirements, while changes to the lower layers are either needed to support business logic changes, or represent some kind of longer term project, and tend to be much chunkier.

As such, building a system where regular business logic changes require no significant effort at the lower levels is a huge productivity win. Any time the lower levels make business-specific assumptions, it creates obscure coupling that is likely to come back and bite you.

I'd agree that what-goes-where (and how to interface the layers) is inevitably subject to iteration, and as such one of the higher priorities is to build a design that can evolve with time (once you're past the prototype/complete rewrite stage).

The way that I've tended to approach this is to look at the business problem and ask myself "How would I solve this, if I could choose any realistic toolkit, that exists or not?", then build or select the tools, and build the "business logic" layer on top.

What I find interesting is that the big projects have almost all ended up being DSLs (embedded in Haskell, Java and Lua for different projects), with smaller, simpler projects just exposing an API.

Cheers,

Simon.

To view this discussion on the web visit https://groups.google.com/d/msgid/software-design-book/cb42a10a-1398-4188-8d49-595b70bd109dn%40googlegroups.com.

[Gregg Irwin]

Thanks Simon. I'm also heavily biased toward DSLs, or dialects as we call eDSLs in our language. I should also clarify that what I'm talking about is structure and organization, not Literate Programming. Both, if done well, support storytelling, but Literate Programming (which is not itself a bad idea) is very much about exposition, rather than giving the reader things to care about, and the clues they need to participate in the story.

[Gregg Irwin]

We can take this analogy even further, comparing non-linear narratives, or stories told from different viewpoints (e.g. The Sound and the Fury) and how they affect understanding, and make a story harder to follow. Those who favor immutability might cite Orson Scott Card on how hard it is to get time travel right, but we can probably all relate to reading some piece of code and thinking only Stephen King could come up with something so evil. :^)

[Ciaran McHale]

A wide variety of suggestions have been made, which provide food for thought. Here are some other kinds of DSL-based approaches that can be useful some (but not all) of the time...

1. Wrapping API alternatives with a parser for a string-based command

Let's assume you want an application to be able to fetch data using any of several different protocols, such as HTTP, FTP, LDAP, SCP, or reading a file. You could write code to support each of those protocols, but doing that would probably require thousands of lines of code. Instead, you could use "curl", which is available as both a command-line utility and an API. Curl can process an instruction of the form "<protocol>:<protocol-specific-parameters>", for example: "http://www.example.com/path/to/file". Curl parses the instruction and then calls a protocol-specific API to do the actual work of retrieving the data. By using Curl, a programmer can avoid the need to write thousands of lines of error-prone code, and instead just use a handful of lines of code that pass a stringified instruction to the Curl API. And if that stringified instruction comes from a runtime configuration file, then the application will be flexible in how it retrieves data.

Many years ago, when I was working with CORBA (an object-oriented, remote-procedure-call mechanism), I used a Curl-like approach to provide two simple "stringified instruction"-based APIs that wrapped overly-complex CORBA APIs for: (1) importing and exporting object references; and (2) creating POA hierarchies (a POA was a container of objects, and properties of a POA, such as whether it was single- or multi-threaded, were applied to the objects contained in the POA). Like with Curl, I found my "stringified instruction"-based APIs saved programmers from writing hundreds or thousands of lines of error-prone code, and reading the stringified instructions from a runtime configuration file made applications more flexible. If anyone wants to read details of these API wrappers I wrote, then look at the "Importing and Exporting Object References" and "Creation of POA Hierarchies Made Simple" chapters in this manual: http://www.ciaranmchale.com/corba-utilities/

2. Using a configuration file to specify how a collection of objects should be created and wired together

In the early days of the X11 Window System, a lot of applications were built using the X Toolkit Intrinsics, which was a C library (providing an object-oriented framework) for creating GUI applications as a hierarchy of specialized types of windows (labels, menu buttons, scroll bars, text editing fields and so on). This framework used a configuration file for specifying attributes (font, font size, background color and so on) of the hierarchical windows in a GUI application. For example, "a.b.c.font: Helvetica" could be used to specify the font in the "c" window which is a child of the "b" window, which in turn is a child of the top-level "a" window. Some clever person decided to add support for two extra attributes in the configuration file: "type" specified the type of a window, and "callback" specified the name of a callback function to be invoked when a particular event (e.g., a mouse press) occurred on a window. He wrote a library that would parse the configuration file and create the entire hierarchy of windows for a GUI application and register callback functions. All that was required was for a programmer to register factory functions for creating each "type" of window, and register a pointer to a C function for each "callback"-type function that might be used in the configuration file. When I discovered this useful library, I realized I could create an application's window hierarchy in, say, 20 lines of configuration rather than in 500 lines of code.

Today, the Spring Framework serves a sort-of-similar purpose: it enables people to use a configuration file to specify how to create and wire-together the objects of an application. I think of the Spring Framework as being a "great idea with a non-ergonomic implementation", so I am not a fan of it, but I am a fan of the basic idea.

The Spring Framework is a *general-purpose* framework for creating objects from configuration, while the X11 utility library I discussed above is a *domain-specific* framework for creating objects from configuration. I have implemented several domain-specific frameworks of this type and have found them useful. If anyone wants to read about one of them as an in-depth case study based on the Java Messaging Service (JMS), then I suggest you visit http://www.config4star.org, read chapters 2 & 3 of the "Getting Started Guide" (for an overview of the syntax and API of the configuration parser), and then read chapters 7 to 11 of the "Practical Usage Guide".

Many years ago, I read a book about enterprise design patterns. A large subset of the book was devoted to discussing patterns associated with one-to-one, one-to-many and many-to-one message flows, and the thought struck me that a DSL might be developed that would read a specification of the desired message flows and create the required objects (using a particular middleware technology), complete with quality-of-service properties specified in the configuration file. The same thought occurred to me when I read (badly written) documentation for the open-source ZeroMQ middleware project.

Final thought... Simon mentioned it is desirable to separate business logic from infrastructure/libraries/generic bits. People have found it useful to also maintain a separation between business logic and the generation of HTML pages in a web server, and various template engines have attempted to provide such a separation. Terence Parr (most famous for designing the ANTLR parser generator) wrote an interesting paper called "Enforcing Strict Model-View Separation in Template Engines" that compares his StringTemplate engine against other template engines, and explains why most template engines fail in their goal of enforcing separation. Here is a link to the paper for anyone interested: https://www.cs.usfca.edu/~parrt/papers/mvc.templates.pdf

Regards,

Ciaran.

-- Ciaran McHale (ciaran...@yahoo.co.uk)
www.CiaranMcHale.com
Mobile: +44-(0)7866-416-134
Home: +44-(0)118-327-9847

https://groups.google.com/d/msgid/software-design-book/3e421432-8c76-452e-96dd-20fd91ca2345n%40googlegroups.com

.

[Brent Welch]

My personal experience with boiler plate dates back to X11 - really X10, and finished with John's Tcl/Tk.  I will note that even today, the Javascript/Typscript/Dart/Angular framework folks continue to introduce a new UI framework every couple of years or so.

The first Sun workstations shipped with "SunTools", and the manual was about 3/16 inch (3 mm) thick.  I wrote the (bad) equivalent of MacPaint with that simple toolkit while I qA in grad school and waiting for access to the Sun source code so I could do my real work.

About 18 months later I went to write another GUI app.  Now it was Sun View.  The manual was almost twice as thick. Say 5mm.  This might have been a text editor - not sure.  No, that was the Andrew Messages Window system from CMU.  The Andrew manual was perhaps 8mm thick.

Another 18 months go by and the apex of boilerplate is the Motif windowing system.  Easily 20mm thick manual.  Loads of boilerplate to build widgets.

Meanwhile, John is writing Tcl, and I graduate. We were writing the Sprite Operating system and I know I wrote a CPU graph widget to monitor CPU utilization.  Probably some raw X11 library code.  Anyway, I graduate and eagerly await John to produce Tk.

John created a simple set of widgets in C, exposed them via the Tcl scripting language, and Voila, all the "boilerplate" is down in the C code, and you write your applications in a lightweight scripting language.  There were times at Xerox PARC where I felt like I was writing a new app every 18 days instead of 18 months.  I wrote a UI for a new color scanner (one of the first) in "3 weeks", including C code to feed scanner output into the Photo widget, and did some super basic "image processing" in C to deal with the idiosyncrasies of the scanner.  Anyway, it was day and night compared to the previous several years of dipping my toe in the water for the framework-heavy, boiler-plate heavy C/C++ UI frameworks.

To view this discussion on the web visit https://groups.google.com/d/msgid/software-design-book/1002379555.4822039.1683120786390%40mail.yahoo.com.

[Venkat Dinavahi]

I wanted to revisit this thread after putting some of the principles into practice for ~6 months. Curious what others think!

(Just wanted to say, this may be one of the best books I have read in my career!)

General purpose code is easier to extract rather than write upfront

It's very difficult to anticipate the requirements of a system until you've actually built the system. So I found it more efficient to create a rough sketch of the architecture, build the thing, then EXTRACT a general purpose layer after knowing the requirements. It's easier to distinguish something that is general from special-purpose once you have a concrete example.

Whenever I tried to anticipate the general purpose design, I have usually failed. I end up creating a complex mess.

Write the code you wish you had

This may sound obvious, but I found it easy to get caught up in the technical architecture of the thing rather than focusing on the actual USAGE of the general purpose piece of software. There is a concept out there called "README driven development" which is very similar. You write down how you would USE this library and you spend time  scrutinizing that design.

I'm certain that I have been under-investing in the design process.

In particular I did not spend enough time thinking deeply about the public interface and the concepts. I did not spend enough trying to push to make the interface deeper. Meaning simplifying the interface, and attempting if I can push more of the complexity into the implementation

There are a lot of questions you can ask repeatedly:

- Do I really need this?

- Why should I have to think about this as a user?

- What would make my life easier in using this library?

- Are there designs out there I can take inspiration from?

Envision hypothetical scenarios to better understand the boundaries

One exercise I found really helpful: invent scenarios on how the library might be used in other parts of the app or a new application.

By doing this, I find it easier to clarify the boundaries on what is general purpose and what is app-specific.

[Paul Becker]

Venkat - i love the points you make. in particular i like the questions you list.

I've found that it's equally possible to get stuck in the design process, thinking about how i would use the library and having a lot of trouble really digging into the implementation. My theory is that there is some sort of lightning bolt that happens when i manage to line up the design and the underlying implementation structure, and the combination breaks through the resistance of the gap between them and they come together. I'm still learning how to do this. I suspect also that sometimes we hit "dead end" ideas that don't actually mesh with reality or what we actually need the program to do---and that slows us down in the design process, if we get caught up in believing that they're necessary. That's where cheap prototypes come in handy, so we have something to actually play with and a sandbox in which to run up against these practical issues in a low-stakes environment.

Extracting general purpose code - this reminds me of the toyota production system, particularly how they describe the design of their assembly lines: "[R]ather than starting with a machine from the beginning, you must first try doing it thoroughly by hand, implement kaizen [continuous improvement], eliminate waste, inconsistencies, and unreasonable requirements [...] and make it possible for anyone to do the work. You must then make it possible to detect abnormalities in the work and build that into the actual machines. These incremental efforts lead to a production line that is high-quality, low-cost, flexible, and easy to maintain." -- https://global.toyota/en/company/vision-and-philosophy/production-system/

it definitely seems to me that there is value in coming at these things with many different approaches.

paul

To view this discussion visit https://groups.google.com/d/msgid/software-design-book/8b653086-ba52-4700-a3d3-1389e0320e23n%40googlegroups.com.