API Calls...

[Jim Randell]

Hello all, and thank you for letting me join in.

In the last few years, I have found myself building reporting modules (lower-case 'm') based on API calls to Web Service/WebApps such as Jira, Confluence, Assembla, Vimeo, BigCommerce, and others.

I have been thinking about a collaborative, online collection of API calls to common services.

At first glance, this may not be a good fit for mFM. This seems closer to the venerable Brian Dunning Custom Function site/online database - if you consider an API call as a sort of snippet. That may be a way, or even the best way of looking at it. I have considered this, but have not decided.

Another way to look at this would be as Modules (Uppercase "M" - as in mFM) for API workflows. For example, a 'Jira Sprint Reporting Module' which captures broad swaths of data (as opposed to discrete bits of info on any given object) into normalized tables residing within the Module. A developer need only reference the tables from the Module as an EDS, and build reports / integrate data, as needed.

Of course, unlike most other mFM Modules, there is more to be concerned with regarding the Modules 'shelf life'. To make this truly Modular, as I described in my example, these would need to be maintained in repositories. If the module is properly maintained, pulling down an updated build would be super fast and easy.

I understand Git has been an issue of discussion around here, I have strong thoughts on the subject, but do not want to muddy the waters as I am only intruding a concept for consideration.

Maybe this might be part of the URL Builder Module.
Maybe you guys have had this discussion and discovered it did not fit in this project.
Maybe know one cares about this subject.

The only thing I know, is how much I don’t know, so I'm laying it down here to see what you all have to say about it

Thanks for letting me play in your Reindeer Games

Sincerely,

-Jim

[Todd Geist]

Hi Jim,

Thanks for the email.  There is certainly a need for things like this.  I have been doing a lot of this kind of stuff as a well and have given some thoughts about how to make this available. You could certainly make mFM modules for each of these  Web Services APIs. This works fine. I do it all the time.  I don't always choose to make them public, though. 

I think there could be more useful.  I am curious about a few things you mentioned.

How do you envision keeping things updated?

How would use git?

Thanks

Todd

--
You received this message because you are subscribed to the Google Groups "Modular FileMaker" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modular-filema...@googlegroups.com.
To post to this group, send email to modular-...@googlegroups.com.
Visit this group at http://groups.google.com/group/modular-filemaker.
To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/0c1a3ce6-fba5-4a91-b25d-38ce61ccd9e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Todd Geist

(800) 935-6068

[Jim Randell]

Hi Todd,

(putting APIs down for a moment)

I have many thoughts about FM and Repository integration in general. Forgive me if this is changing the subject, I see it more like seeking focus.

It seems to me that Git should be at the very heart of mFM. 

I have seen a thread on the subject of 'where' and 'how' mFMs should be stored and accessed. The discussion springs from a fundamental aspect of mFM: 'Openness'.

The website embraces the concept of modern, Open Collaborative Development. But some clarification is in order. As contradictory (or Orwellian) as it may sound, enforcing adherence to structure is necessary to foster an open community. I know, "freedom thru boundaries" rubs me the wrong way too, but we're not talking about government. In my opinion, mFM does not live up to its self defined aspirations if it does not fully embrace the Collaborative Software Development Model and the framework of the 'Open Repository' which is at its heart.

As a New Kid, there is much about this place I don’t know. Has there been any discussion amongst the mFM community regarding the following?:

Should all mFMs be hosted, and version controlled, on a single, centralized repository service (like GitHub)?

Are repository based workflows something which might? could? should? be part of the mFM development model?

Todd, I don’t mean to turn your question back on you, but I would like to know how YOU (the mFM community) would use GIT.

Thanks,

-Jim

[Todd Geist]

On Tue, Nov 18, 2014 at 7:58 AM, Jim Randell <jimrand...@gmail.com> wrote:

Todd, I don’t mean to turn your question back on you, but I would like to know how YOU (the mFM community) would use GIT.

I love Git, I use it every single day.  I even use it on my FileMaker projects.

That said there is no way that the average FileMaker developer can use Git at all. It is completely a non starter.  As evidence, I will point to what happened when a few people tried to host their modules on github. People had no idea what to do.  I mean no idea.

Lets also remember that git is not even installed by default on Windows, which is still more then half the FileMaker market.

ModularFileMaker.org primary purpose is to get the most number of people to write and share code. Sharing becomes truly meaningful when there are lots of people doing it. Putting git in between the average FileMaker user and sharing will kill off meaningful sharing.

So the current answer is yes, it has been discussed, and no it isn't going to become the recommended way of publishing ModularFIleMaker Modules in the near future

However, I do think that with the right tooling, something might be done.  In other words if there was a UI tool designed to work with FileMaker the way npm works with node, then things could change.  I aware of some of the excellent UI tools for git ( especially on the mac ), but these don't really solve the FileMaker problem.

But, unanswered questions remain.  What is placed in the git repo? The FileMaker file or XML representations of the code, produced by the clipboard capturing tools?  If its the fileMaker file, then merging and forking are out, but you have functioning code that just works when it's downloaded.  If its the XML, you might be able to merge and fork, but the code won't work until it is recompiled.

I think working out how to make git a useful part of the average FileMaker developer's workflow is really important. I have a number of ideas about how to go about doing that, none of which are easy or complete. Until there is a way for the average FileMaker user to publish and consume modules and code with git, changing ModularFileMaker.org recommendations would be premature.

As for your comments about mFM not living up to it's ideal. I think it does about as good a job as can be done given the current state of the technology. I see it as step in the right direction.  Nothing more. 

There will be better things, but I think it will require technical changes from FileMaker Inc. Thankfully they seem interested and regular consult with me about how to make things more modular.


Todd

[Jim Randell]

 Hey Todd,

Let me start hear...

As for your comments about mFM not living up to it's ideal. I think it does about as good a job as can be done given the current state of the technology. I see it as step in the right direction.  Nothing more.  

I love "The Good", please forgive me when my eyes are turned by that sultry seductress, "The Perfect".

I meant no offense..

However, I do think that with the right tooling, something might be done.  In other words if there was a UI tool designed to work with FileMaker the way npm works with node, then things could change.  I aware of some of the excellent UI tools for git ( especially on the mac ), but these don't really solve the FileMaker problem.

I think working out how to make git a useful part of the average FileMaker developer's workflow is really important. I have a number of ideas about how to go about doing that, none of which are easy or complete. Until there is a way for the average FileMaker user to publish and consume modules and code with git.... 

uh uh, did I just get Served?!

[Todd Geist]

On Tue, Nov 18, 2014 at 10:35 AM, Jim Randell <jimrand...@gmail.com> wrote:

uh uh, did I just get Served?!

Sorry, that wasn't my intent.

The way I look at it is that ModularFileMaker serves it's original purpose. Until we get some break thoughs in either tools or new features from FMI I think it's good enough, and should probably just stay where it is.

The longer term question of how to make something better, is still very relevant. I just see it as a separate question. I think this is an area that is need of experimentation and research.

As an example.  I am about to release a tool that takes the FileMaker DDR and turns it into something that is meaningfully diffable.  This is what I use with git and github to manage all of my FileMaker projects.  It doesn't allow branching or merging. It just makes it so that your diffs from one commit to the next are meaningful. By that I mean, when you look at them, you can make sense of them. You have some hope of knowing what changed.

Coupling that with github and its issue tracker, I have workflow that I am reasonably happy with.  But it is a long from enabling any kind modular sharing approach.  It just lets me check my code into source control and know what was checked in

This work is best done by lots of people, trying lots of things. So it wasn't my intent to scare you off.  Just to let you know that ModularFileMaker.org is probably not the forum this work yet.  

Hope that helps explain things

[Jim Randell]

Todd, I have been obtuse.
you didn't scare me, you excited me. I would love to be involved, in any way.

here, let me tell you about me.
I was working at SFMOMA in 2000 when someone I used to work with, now at ILM, called and asked me to come out and fix the (starwars) EPII database.

they liked my work, I liked being an extra in EPII, I stayed - for 7 years.
(you'll find my name on the credit of "War Of The Words", and "Indiana Jones And The Warentless Sequel")

after I left, they could not keep anyone in the position for more then 6 months. it was a crazy, hectic, place, which would drive anyone crazy (well, not me. crazy for me is like the Chicken Pox, I've already got it in my system, I have an immunity).

The first guy after me was Dave Simmerly. Dave called me one day with a problem involving an attitude prone producer. He asked me how I could put up with it for 7 years.

I told him this:
A guy's standing on the corner, watching a circus parade go by. At the end of the parade is some schmuck shoveling up elephant shit.
The guy on the corner says to him, "this must be the worst job in the world, cant you find a better job then this?", and the guy shoveling up the elephant shit says "what, and give up show business?"

so where was I going with this????
Oh ya, you got any elephant shit I can shovel up for you?


(to be clear I am pro bono)

--
You received this message because you are subscribed to a topic in the Google Groups "Modular FileMaker" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modular-filemaker/ccXLuLR7bgs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modular-filema...@googlegroups.com.

To post to this group, send email to modular-...@googlegroups.com.
Visit this group at http://groups.google.com/group/modular-filemaker.

To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/CAEA0Pwzj%2BTK1daDtUDfwB-chTy%2BOHC6AmT1jd3-LxyTx9p1pcQ%40mail.gmail.com.

[Todd Geist]

:-)

Alright you might just find yourself drafted

Thanks

Todd

You received this message because you are subscribed to the Google Groups "Modular FileMaker" group.

To unsubscribe from this group and stop receiving emails from it, send an email to modular-filema...@googlegroups.com.

To post to this group, send email to modular-...@googlegroups.com.
Visit this group at http://groups.google.com/group/modular-filemaker.

To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/546BAAA5.70802%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--Todd

[Jim Randell]

OK,
I'll hose down my shovel.

so, your ddr report thingy, any open sprint items? maybe QA?

I'm diving back into my AdWords API project, I have a delivery date looming
I'm around, just let me know what I can do.


best,
-Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/etPan.546bad60.2ae8944a.356%40todds-mbp-7.home.

[Jeremy Bante]

On Tuesday, November 18, 2014 7:58:00 AM UTC-8, Jim Randell wrote:

I have seen a thread on the subject of 'where' and 'how' mFMs should be stored and accessed. The discussion springs from a fundamental aspect of mFM: 'Openness'.

The website embraces the concept of modern, Open Collaborative Development. But some clarification is in order. As contradictory (or Orwellian) as it may sound, enforcing adherence to structure is necessary to foster an open community. I know, "freedom thru boundaries" rubs me the wrong way too, but we're not talking about government. In my opinion, mFM does not live up to its self defined aspirations if it does not fully embrace the Collaborative Software Development Model and the framework of the 'Open Repository' which is at its heart.

Jim, it sounds like you're falling into the same trap many folks working with open source software trip over. You're conflating values that often coexist, but are actually independent of each other. Usually, it's confusing "open source" software with "free" software — "free" as in speech vs. "free" as in beer. (There are modules written in the spirit of mFM that are not free-as-in-beer.) You appear to be conflating sharing with collaborating. Making collaboration — multiple people developing the same module — easier is a worthy goal, but it's not quite what mFM is trying to accomplish, as I understand it. Many of the modules on ModularFileMaker.org are collaborative efforts, and a handful of them do use GitHub to achieve that, but it's really a pain in the butt compared to collaborative development with plain-text development tools. FileMaker's technical limitations make collaboration difficult, perhaps unreasonable for mFM to tackle at the moment; and GitHub as a way to do it is both technically problematic and goes over the heads of the vast majority of the target audience of FileMaker users. Hopefully someday.

[Jim Randell]

Hi Jeremy,

GitHub as a way to do it is both technically problematic and goes over the heads of the vast majority of the target audience of FileMaker users. Hopefully someday.

I completely agree with you, it is complex (for the FM user base) and problematic. I was seeking the disposition of this forums members, as to weather the time to approach this issue was now, or, as you say with such hopefulness, someday.

If the responses I have received from you and Todd are representative, I would say my answer is clear.

The rest, about the ambiguous meaning of the word 'free', was extremely insightful, thank you for pointing it out to me.

[Jim Randell]

Hello All,
so, I'm thinking about starting with an Authentication module.

FMauth (working title) would allow devs to create, name and test authentication methods for different services.

Here is what I think would need to happen to make this very useful (ALL feedback is GREATLY appreciated!)

1. scope and determine the number of authentication actions required by an API call
• off the top of my head, lets say there are 3:
• NewToken
• RefreshToken
• Send Login
• (I need to flesh this out - but I will continue this post using '3')
  
  
2. set an mFM guideline: these 3 actions should exist as standalone scripts and be called when needed by other scripts.
   
• as opposed to being written into the code of a larger script
  
  

Use Case: developer writes an API module for, say Jira.

1. The dev wants to perform authentication actions, but does not want to use FMauth. 
• dev decides the FMauth module will not suit their needs, so they write their own authentication
  
  
2. But, for whatever reason, the dev does follows the guidelines and makes these subscripts.
• writes individual scripts to get NewToken and RefreshToken
  
  
3. later, Jira changes and only supports OAuth2, (the dev was using OAuth1)

The dev can re-write his own scripts, but first, he decides to...

1. downloads the latest release of FMauth
2. configures and tests a service for JIra with the easy Setup Screen
   
3. Points his scripts at the FMauth version of the NewToken and RefreshToken scripts
4. breaks out in song because instead of passing his developers token to script X (which returns an authentication token), he now passes it to script Y, which does the same thing.
   

Other thoughts:

• what methods of authentication would this support?
• how much effort to put into deprecated or less adopted technology?
• how to decide what to support?

Anyone have any thoughts???
be frank, criticism is no good unless its critical.

On 11/18/14, 12:34 PM, Todd Geist wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/etPan.546bad60.2ae8944a.356%40todds-mbp-7.home.

[Todd Geist]

Authenticating into various web api is a major problem worth solving.  three questions I have ( speaking like Yoda )

1. Are you intending to make this a general module. i.e. for many web services?

2. Many of thee web api’s require methods other than GET , and POST.  Natively FM only fully handles GET, and POST partially.   Also some API’s require digital signing which is not possible with out a plugin or other help.  Whats the plan for that?

3. Some web API’s require a http server to call back to, to pass back tokens.  Do you have a way to solve that?

I ask, because  am actively working on a QuickBooks API module, where I hit each and everyone of those issues :-)

Todd

To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/546E6139.4000507%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

——————————————

[Jim Randell]

On 11/20/14, 2:06 PM, Todd Geist wrote:

Authenticating into various web api is a major problem worth solving.  three questions I have ( speaking like Yoda )

1. Are you intending to make this a general module. i.e. for many web services?

• yes, one size fits as_many_as_I_can
• this will obviously take some fleshing out
• realistically, I would be happy if I could support at least 5 major services
  
  
• LETS START HERE: I have the Google API OAuth2 down solid (relative to its current stage of dev)
• how many locks will that one key open?
• -no really, I don't know,
  
• youtube, googleapps, and the usual "G" suspects
• so many other services utilize the same code - I don't know how many services I could use that one workflow with until I start poking around

so the answer at this point is: rc1 will support many (all of Google at least)

2. Many of thee web api’s require methods other than GET , and POST.  Natively FM only fully handles GET, and POST partially.   Also some API’s require digital signing which is not possible with out a plugin or other help.  Whats the plan for that?

OK,
lets not pretend your the IS manager who's concept of FM is like the database from Microsoft Works, circa 1998.
lets not pretend I'm the python developer who hasn't spent 20 years in cubicles dealing with people who either expected way to much from FM, or wanted it out, OUT of their IS department (I once had a manager describe FM as Excel On Acid) 

we know we can be cleaver with GETs and POSTs, I think we pretty much know where those boundaries are.

we know if we want to do X, we are going to need to bleed outside FM

we no all the potential tricks we can pull out of the webviewers ass.

of course there's ScriptMaster/groovy, which I know you are involved in at various levels,
which is so deeply rooted that Its as close to staying "completely inside" as you can get, without staying "completely inside".
it would be the most "standard" way to handle more sophisticated processes on the client side.

We can talk to the command line directly (better via SM/Groovy)

there's  applescript and its rich, crazy cousin, VB.
its 2015 (almost) are the posix services fully integrated into windows? (I wouldn't count on it)

a solution running off FM server opens a lot more possibilities, but we are starting to get narrow

the most out there: a business entity which hosts a service which receives and processes requests.
We did this at ILM, we setup a server and workflow for FM centric stored procedures.
New procedures were written as needed, I just passed the data to the procedure, and it passed me back what I wanted. Simple. We just need the resources and staff comparable to the old gang from trailer 6 at Kerner

Bottom line - if it can only be done outside FM, and we want to do it, I lean toward scriptmaster/groovey

3. Some web API’s require a http server to call back to, to pass back tokens.  Do you have a way to solve that?

I ask, because  am actively working on a QuickBooks API module, where I hit each and everyone of those issues :-)

in an FMserver environment, a simple listener is easy enough, would need to think about it, but might not need to go outside FM at all.
its a server environment, so its not turn-key, but it, like anything, can be if you want to put enough money into it.

desktop listener is not much harder, but the firewall/router/portforwarding issue makes it a major issue. Not insurmountable, teamviewer does a wonderful job, do we have that kind of budget?.

 Whats the plan for that?

Todd
I suspect you have whitboarded all this stuff, quickbooks and all the rest, with guys at least as clever as myself.
so unless you really are unsatisfied with what you curently see, I'm not sure a whiteboard, a jug of wine, and thou, would get you much closer.

I can give you my opinion on a specific approach, you can ask me something very specific and I can tell you how I might do it, but I'd bet my new granddaughters rattle that you have approaches to everything we have discussed, tasked out and in the case of the QB project, divided into sprints.

To view this discussion on the web visit https://groups.google.com/d/msgid/modular-filemaker/etPan.546e65d0.6b8b4567.f81%40todds-mbp-2.home.

[Todd Geist]

On November 20, 2014 at 4:50:36 PM, Jim Randell (jimrand...@gmail.com) wrote:

Bottom line - if it can only be done outside FM, and we want to do it, I lean toward scriptmaster/groovey

I would have gone that way a couple of years ago.  But I have put aside ScriptMaster in favor of BaseElements.  Not as powerful but better license terms and no Java, which after a few years of supporting ScriptMaster plugins, I like.

I think the google APIs are with doing. They have a desktop work flow for getting the tokens back. Lots of cool stuff their.  I would avoid doing anything that requires server unless you have to.  I’d choose BaseElements, but you could do it with ScriptMaster. If you do do it with ScriptMaster, don’t compile it into a custom plugin.  Just register the functions and use ScriptMaster.  The license on Compiled plugins is different then when you just use ScriptMaster alone.

Todd

[Jim Randell]

fascinating,
I thought BE was a "branch" (I mean that in the loosest of terms) off SM. I had no idea they were completely different.
I use BE and SM interchangeably during development, then 99% of the time, find my own workaround before deployment.

I try hard to never re-invent the wheel, so, since your way more plugged into this subject then me, will go with BE, if/when needed on FMauth.

[Jeremy Bante]

On Thursday, November 20, 2014 1:46:37 PM UTC-8, Jim Randell wrote:

Other thoughts:

• what methods of authentication would this support?
• how much effort to put into deprecated or less adopted technology?
• how to decide what to support?

There are two different approaches I can suggest starting with:

1. Do as many methods as are technically convenient for FileMaker. Maybe start with everything FileMaker can do without using a plug-in or web viewer. Maybe it isn't as broad as everyone would like, but it's a reasonable compromise for the convenience folks will get when installing your module.
   
2. Only support things you've actually used in the first release, and folks will contact you with the services they most want to interact with. This way, you don't have to guess what will be useful.
   

[Jim Randell]

Excellent thought Jeremy, that's exactly what I will do.

[Jim Randell]

OK, I am taking the next two days to de-couple my OAuth2 from my AdWords Solution and scrub it up.
not sure I will have it done in two days, but, anyway, thanks all for the feedback...more soon.