Memory usage
Mark Wilden
used when we load records from FM via RFM, and I was wondering if
anyone has experience with this.
Simply doing
records = @database[layout[:name]].all
to retrieve 40,000 ZIP code records increases the memory used by the
application as reported by the operating system by 100MB. 2500 bytes
per ZIP seems excessive.
Thoughts?
lardawge
records? Any other libs you use that have similar functionality that
we can compare to rfm? I have been working (slowly when I get time) on
sorting out some internal issues that I think may be contributing to
excessive memory usage but would like to get some input from others.
Also, are you using lardawge-rfm or rfm?
Mark Wilden
My question was by way of getting some initial feedback about rfm's
memory consumption. Our application would suddenly terminate, and I
finally determined that it was because of excess memory use. The
application basically copies data from FM into MySQL. The real memory
use of the application gets to over 800MB by the time it's finished.
Firstly, I wouldn't have thought that the source FM database had that
much data in it. Second, I would've thought that after one FM table
was imported, its memory would be available for the next part of the
import. Instead, it seems like memory consumption just keeps going up.
I should stress that I don't know whether the problem is with rfm,
ActiveRecord, ruby, or anything else. I did observe memory use go up
as the result of reading the contents of one FM table, so I thought I
would start there.
> If I could ask, what was your expectation for memory usage for 40,000
> records?
The task that loads the 40,000 ZIP codes actually consumes 500MB.
Tomorrow, I will look into whether a single ZIP code record could
indeed contain 12,500 bytes, but it sure doesn't seem likely!
> Any other libs you use that have similar functionality that
> we can compare to rfm?
I could set up a test, reading the same information into Rails via
ActiveRecord, I suppose.
> Also, are you using lardawge-rfm or rfm?
This is your fork. I've inherited this project, and I must say that
I'm very glad to have this bridge to FileMaker available!
///ark
lardawge
a script I use to test rfm when developing. As stated earlier, I am
looking into ways to solve a bunch of issues within the code where an
object gets stored in a variable within multiple objects. Not sure if
this is the problem but I suspect it may contribute to it. I also
think that the way the xml response from filemaker is getting parsed
may contribute to it as well. Basically I have been working on a
complete rebuild of the API as well as internals of rfm. I may release
it as a new gem entirely and leave Rfm alone... Railsfm? Oooh, I like
the sound of that.
On May 31, 12:13 pm, Mark Wilden <m...@mwilden.com> wrote:
Mark Wilden
> As a follow up, I am seeing some abnormal memory usage and slowdown on
> a script I use to test rfm when developing. As stated earlier, I am
> looking into ways to solve a bunch of issues within the code where an
> object gets stored in a variable within multiple objects. Not sure if
> this is the problem but I suspect it may contribute to it. I also
> think that the way the xml response from filemaker is getting parsed
> may contribute to it as well. Basically I have been working on a
> complete rebuild of the API as well as internals of rfm. I may release
> it as a new gem entirely and leave Rfm alone... Railsfm? Oooh, I like
> the sound of that.
Haha! Let me know if you need a beta tester.
///ark
lardawge
know what you think of the API:
http://gist.github.com/422704
Some stuff like retrieving all the script names and layout names will
be left out. I never use them and until I can get a good reason why I
would need them they will be left out. It assumes you have access to
the filemaker db. My main purpose is 1) Make it rails friendly. 2)
Shuttle info back and forth as quickly as possible. 3) Use whatever
xml parser you want.
Suggestions welcome.
On Jun 2, 10:10 am, Mark Wilden <m...@mwilden.com> wrote:
Mark Wilden
> Will do. I should be close to something usable in a few days. Let me
> know what you think of the API:
>
> http://gist.github.com/422704
>
That looks great - much more Railsy.
> Some stuff like retrieving all the script names and layout names will
> be left out. I never use them and until I can get a good reason why I
> would need them they will be left out.
We actually do get all layout names on occasion, to refresh our
memories. We don't touch the FM side very often, so we forget.
> It assumes you have access to
> the filemaker db. My main purpose is 1) Make it rails friendly. 2)
> Shuttle info back and forth as quickly as possible. 3) Use whatever
> xml parser you want.
>
> Suggestions welcome.
Other than that, sounds excellent.
wbr
>
> That looks great - much more Railsy.
>
is great, however it is important to us that RFM is independent of
Rails. We use RFM just as much without Rails as we use it with Rails.
I haven't looked at the new api yet, so maybe this is a non-issue?
-Bill
lardawge
are some changes to the api as per my earlier post.
http://gist.github.com/422704
At this point I have left most of the old api intact aside from a few
changes due to ripping out internals to streamline testing (and I
think it is more intuitive).
1. my_server.db[][] don't exist. It is now my_server.db().layout()
only.
At this point some methods have been left out because the internals
are completely restructured and I haven't added them. Think of this as
proof of concept for people to check out. I believe I have increased
speed as well as memory footprint. I have also found a significant
memory leak (I believe it is the https lib) when making large queries
where the memory grows when waiting for a response from fm server. A
workaround and something that may be added to the gem is grabbing
records in smaller chunks say 500 at a time and building up an array.
In my testing this helped the memory issue greatly as well as making
it faster.
So the main thing I am interested in at this point is speed tests in
general as well as feeling out the new api and asking questions so I
can better answer them in documentation when I write it.
Thanks,
Larry
lardawge
http://github.com/lardawge/rfm/tree/rfm2
The gemspec is up to date so you should be able to build it locally
for testing.
On Jun 4, 11:33 am, lardawge <larda...@gmail.com> wrote:
> Alright, I have a very very rough version of rfm2 up on github. There
Geoff Coffey
Just to comment on the new direction of RFM... or RailsFM. More Railsy
is great, however it is important to us that RFM is independent of
Rails. We use RFM just as much without Rails as we use it with Rails.
I haven't looked at the new api yet, so maybe this is a non-issue?
Geoff Coffey
--
You received this message because you are subscribed to the Google Groups "RFM Community" group.
To post to this group, send email to rfmcom...@googlegroups.com.
To unsubscribe from this group, send email to rfmcommunity...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rfmcommunity?hl=en.
Mark Wilden
> If you do pursue something like this, I would recommend you consider making
> your model-like classes implement the Rails 3 ActiveModel interface. This
> will allow you to use your FileMaker models in form helpers, add railsy
> validation easily, and just keep things much more rails like with less
> effort.
Yes, that sounds great.
> It would make the API Rails 3 only though. So that may be a deal
> killer for you.
It doesn't have to be, though. Rfm is working fine for people the way
it is. And unless something really weird happens, Rails 3+ is going to
be used a lot more than Rails 2 as time goes on.
///ark
John Lannon
wbr
> I think, though, that with the right architecture, you could have a sort of
> procedural-ish core FileMaker API (like Rfm is) and then a rails ActiveModel
> layer above it. This is, anyway, what we keep thinking we'll do should the
> opportunity arise.
trips me up is that I forget I'm not in ActiveRecord, and I use an
ActiveRecord feature that doesn't make any sense in RFM. Though this
only happens when I'm using ActiveRecord and RFM objects in close
proximity.
>
> I also keep half-thinking it might make sense to build an ActiveRecord
> adapter for FileMaker over ODBC. I have steered clear of ODBC based on
> terrible performance and reliability problems years ago, but people keep
> telling me it is better now.
>
ActiveRecord adapter, however...
Two years ago, I spent a lot of time trying to make FM Server's
(version 9) ODBC support work for us. I eventually tossed the entire
project, because the FM ODBC features were woefully inadequate,
documentation was almost non-existent, performance was slow, and
reliability was poor. Part of this could have been the driver, but
since I had only one choice of ODBC driver, I really don't know.
Supposedly, FM Server 11 has "improved" ODBC features, but I have yet
to see specs on exactly what this means. There is also the unfortunate
fact that ODBC support is only included in FM Server Advanced, which
is 3 times the price of regular FM Server.
Filemaker's XML interface can be clunky and inefficient, but for us
it's been reliable, consistent, and accessible across all of our FM
Server based clients. There are no drivers required to access it, no
proprietary 3rd party software, SSL support is a no-brainer, and it's
well documented.