Skip to first unread message
vivek poddar
Jul 24, 2015, 6:36:19 AM7/24/15
to Clean Code Discussion
Hi,
Once again I am starting a project and I want it to be clean architectured :).
Here is the project outline:
Its a kind of search service which provides a common platform to search for local services under one roof.
At present there are individual services which confuses users.
Later on I will be providing my own service but that's far in the road map.
Proposed solution:
1. Have a gateway / adapter (I am confused if they are same) for each service.
2. Have a separate domain model for each service.
3. Keep a single return type for search query to simplify the client side work.
Problems / Issues:
1. Each API has a different query parameters and return type for the same service.
2. The external APIs may change frequently.
3. Howto handle the case when any API (service) is down?
4. Howto scale i.e. add a new service easily without much changes in the code base.
Advance:
1. Caching strategy for performance reasons.
Technology Stack
- Back end:
- Platform: Google App engine.
- Language: Python
- Front end:
- HTML, JS, CSS
Thanks,
Sebastian Gozin
Jul 24, 2015, 7:18:59 AM7/24/15
to Clean Code Discussion, vivek...@gmail.com
Proposed solution:1. Have a gateway / adapter (I am confused if they are same) for each service.
Sounds good.
2. Have a separate domain model for each service.
Not sure why you wouldn't use a single SearchQuery domain entity and translate it to the appropriate format in the gateway (#1)
3. Keep a single return type for search query to simplify the client side work.
Sounds good.
Problems / Issues:1. Each API has a different query parameters and return type for the same service.
Use a stable internal format so the rest of your application doesn't suffer when these providers change things on you.
In the gateways you can do the translating.
2. The external APIs may change frequently.
Make sure you got some kind of fallback mechanism in place so your system does not go down on your users.
You could look into Hystrix for example though it was originally created for down time issues.
But isn't a modified API just a form of down time for you anyway?
3. Howto handle the case when any API (service) is down?
See #2
4. Howto scale i.e. add a new service easily without much changes in the code base.
Put those gateways in their own modules so you can deploy them independently.
Shaun Whyte
Jul 27, 2015, 1:50:58 PM7/27/15
to Clean Code Discussion, vivek...@gmail.com, s...@thinkerit.be
I have a related question to your response on point 4 (about deploying independently), but it's more specific to Java modularity. I get that we can nicely separate our applications in to jar files which are essentially the modules. But how can one actually deploy these independently without needing to rebuild a whole war file? I don't know a lot about OSGi but I believe this would allow for that, but is there anything simpler? (imagine we are running using a tomcat container).
Peter Boszormenyi
Jul 27, 2015, 2:13:53 PM7/27/15
to clean-code...@googlegroups.com
Reloading runtime a jar file always
pain in the neck in the java world. Basically, what you have to do
is to create a new URLClassLoader instance for the new jar file.
Then, you can load the classes from that jar file. However,
Imagine the following simple application:
There is a war deployed in Tomcat which accepts the incoming requests, parses it, decides which controller should handle it, and then delegate request to the selected controller. So, the war is the router of a tipical MVC framework.
The controllers reside in jars, and the war picks them up at deploy time, create a classloader for each jar.
So far we have the following classloader structure:
- A web application classloader (provided by Tomcat), which load the war file.
- Separate classloaders for every jars.
What we have to notice is that, a class that decides which controller will handler a request, and then delegate it for the controller, is loaded by the webapp classloader, and it holds a reference for an objects, whose class is loaded by a jar classloader. Now, if we want to reload a controller jar, we have to create a new URLClassLoader for the jar. Because our code might still holds a reference to the old controller instance, the JVM can't collect the old classloader, therefore the classes of the old controller jar stays in the memory. Every time you redeploy a jar, your code will load the classes again, and again, slowly eat all the memory, and finally produce an OutOfMemoryError: PermGen space. If you use java 8, and you don't set the maximum size of the metaspace, your code will eat all of the physical memory of the masinche, which lead a dead by swap.
In order to avoid this problem, you have to remove all references of the objects comes for the given jar file, so the gc can collect the classloader. If you enable dependencies between the jars, the whole thing will be even more complicated, because in this you have to redeploy all the dependent jars as well, transitively (jesus, I can't even beleive that I could write transitively world, whitout type at the first time). Also an interesing question how the Tomcat can handle that the web application creates classloaders (the java ee spec doesn't allow this).
My suggestion: don't reinvent the wheel, if you need runtime redeployment, use osgi.
There is a war deployed in Tomcat which accepts the incoming requests, parses it, decides which controller should handle it, and then delegate request to the selected controller. So, the war is the router of a tipical MVC framework.
The controllers reside in jars, and the war picks them up at deploy time, create a classloader for each jar.
So far we have the following classloader structure:
- A web application classloader (provided by Tomcat), which load the war file.
- Separate classloaders for every jars.
What we have to notice is that, a class that decides which controller will handler a request, and then delegate it for the controller, is loaded by the webapp classloader, and it holds a reference for an objects, whose class is loaded by a jar classloader. Now, if we want to reload a controller jar, we have to create a new URLClassLoader for the jar. Because our code might still holds a reference to the old controller instance, the JVM can't collect the old classloader, therefore the classes of the old controller jar stays in the memory. Every time you redeploy a jar, your code will load the classes again, and again, slowly eat all the memory, and finally produce an OutOfMemoryError: PermGen space. If you use java 8, and you don't set the maximum size of the metaspace, your code will eat all of the physical memory of the masinche, which lead a dead by swap.
In order to avoid this problem, you have to remove all references of the objects comes for the given jar file, so the gc can collect the classloader. If you enable dependencies between the jars, the whole thing will be even more complicated, because in this you have to redeploy all the dependent jars as well, transitively (jesus, I can't even beleive that I could write transitively world, whitout type at the first time). Also an interesing question how the Tomcat can handle that the web application creates classloaders (the java ee spec doesn't allow this).
My suggestion: don't reinvent the wheel, if you need runtime redeployment, use osgi.
--
The only way to go fast is to go well.
---
You received this message because you are subscribed to the Google Groups "Clean Code Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discu...@googlegroups.com.
To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.
Sebastian Gozin
Jul 27, 2015, 2:27:18 PM7/27/15
to Clean Code Discussion, vivek...@gmail.com, sh...@isw.me
As Peter already pointed out managing class loaders yourself is rather annoying.
That said they do get garbage collected. It is true though that some libraries prevent it causing memory leaks but this is not normal.
Anyway, you also have deployability while still requiring a server restart in order to rebuild the class path.
Jar files do not need to be packaged in a war and can still be included as shared libraries. Most containers provide dedicated folders for just this purpose.
For runtime deployability you could try Peter's explanation but frankly that's what OSGI already does.
Not to say OSGI support is in a good place though.
Personally, I find OSGI containers like Apache Karaf or Fabric8 incredibly hard to get started with.
I can't really make head or tails from their documentation and they seem to go for an approach using multiple vm instances anyway.
I do have good experiences booting up my own OSGI runtime (Apache felix), configuring it to scan a directory for changes in OSGI compatible jar files and then wiring my own code with Apache Camel and it's REST DSL.
This approach does not require war files and uses an embedded Jetty server.
This makes it fairly easy to setup a linux server with some maven or gradle script to update the jars.
That said they do get garbage collected. It is true though that some libraries prevent it causing memory leaks but this is not normal.
Anyway, you also have deployability while still requiring a server restart in order to rebuild the class path.
Jar files do not need to be packaged in a war and can still be included as shared libraries. Most containers provide dedicated folders for just this purpose.
For runtime deployability you could try Peter's explanation but frankly that's what OSGI already does.
Not to say OSGI support is in a good place though.
Personally, I find OSGI containers like Apache Karaf or Fabric8 incredibly hard to get started with.
I can't really make head or tails from their documentation and they seem to go for an approach using multiple vm instances anyway.
I do have good experiences booting up my own OSGI runtime (Apache felix), configuring it to scan a directory for changes in OSGI compatible jar files and then wiring my own code with Apache Camel and it's REST DSL.
This approach does not require war files and uses an embedded Jetty server.
This makes it fairly easy to setup a linux server with some maven or gradle script to update the jars.
vivek poddar
Jul 31, 2015, 12:59:37 PM7/31/15
to Clean Code Discussion
Hi thanks for the insight but I stumbled on one more issue that apart from using third party services I am start my own service which is completely transparent to users. Suppose my use case is this:
Suppose there are a number of food service providers each having its own rate card. Now the issue case is that to get the best deal the user has to go and search individual websites hence I am providing a single interface which allows user to search some food item and get all the results from different providers. Now in future I want to start my own food service and include the results of that too. So, is the single SearchQuery model sufficient in this case. I doubt that. WDYT?
Peter Boszormenyi
Jul 31, 2015, 1:14:29 PM7/31/15
to clean-code...@googlegroups.com
Based what you said, what I see here is a adapter and a composite pattern.
You can define a single interface, which describe a unified search
interface for all of the food service provides. Then, you can write
adapters for each provider, that implement the interface you just
defined, and execute a search specific to the given provider. Finally, a
composite search object, which also implements the interface, that just
iterate through the registered providers, execute the same search on all
of them, and merge the result. You can also create an adapter for your
local food service, and just add to the registered providers.
You can define a single interface, which describe a unified search
interface for all of the food service provides. Then, you can write
adapters for each provider, that implement the interface you just
defined, and execute a search specific to the given provider. Finally, a
composite search object, which also implements the interface, that just
iterate through the registered providers, execute the same search on all
of them, and merge the result. You can also create an adapter for your
local food service, and just add to the registered providers.
vivek poddar
Aug 24, 2015, 3:58:12 AM8/24/15
to Clean Code Discussion
Hi,
Sry guys, I have been busy with some personal work. So, in between I was learning some Java and at the same time I prepared a rough sketch for the project outline.
1. Main Service
2. Query
3. InMemoryGateway (I seriously don't know how to proceed here)
4. At the same time I have thought to have SearchQueryResult model but that seems quite abstract to me. I will check hoe to proceed for it.
At the same time I stumbled upon this blog post by Martin Fowler http://martinfowler.com/articles/refactoring-external-service.html
Will check if I can get some help from it.
WDYT?
Cheers,
Sebastian Gozin
Aug 24, 2015, 5:40:30 AM8/24/15
to Clean Code Discussion
Why don't you put some data in different instances of InMemoryServiceGateway and then do a search and see if the results are combined?
e.g:
service-1{x}
service-2{}
search for x yields x
e.g 2:
service-1{x}
service-2{xx}
search for x yields x and xx
e.g:
service-1{x}
service-2{}
search for x yields x
e.g 2:
service-1{x}
service-2{xx}
search for x yields x and xx
Dave Schinkel
Aug 25, 2015, 9:17:21 PM8/25/15
to Clean Code Discussion
"Keep a single return type for search query to simplify the client side work."
What do you mean by "single return type"?
vivek poddar
Aug 25, 2015, 9:23:58 PM8/25/15
to clean-code...@googlegroups.com
Hi Dave, since every service may have different results but I want this to be transparent to the client. Hence I want to return a uniform result structure for every query for any service.
--
Reply all
Reply to author
Forward
0 new messages