What's Next?

[Susanna]

What’s Next?

After the programmers finish work on FieldWorks 6.0, they are going to
spend a significant amount of time reworking the fundamental code.
This reworking of the code is somewhat like rewiring a house. It is a
major job, takes a while, and involves taking down plasterboard to
allow access to the wires… but there has come a time when it is the
wisest thing to do, in order to make the house safe and livable, and
to support the increasing number of sophisticated electrical products
in every room. Just like in the electrical realm, technology has moved
on, and we want FieldWorks to be able take advantage of those advances
– because we believe they will benefit you!

Think of an old house wired with one or two ungrounded outlets in each
room. We've been plugging in one adapter and extension cord after
another to accommodate the TV, VCR and DVD, fridge, microwave, IPOD,
computer, printer, hair-dryer...we're rapidly approaching the time
when a fuse blows or someone trips over a cord and bad things happen
(when programming). In other words, it's not just that we can do more
with newer technology...it's more that we've already been pushing
beyond the safe limits of the technology currently in FieldWorks, and
it's costing us (programmers) and you (users) in terms of convenience
and reliability, and it can't be pushed much further without an
increase in serious problems.

While the “rewiring” will significantly limit our ability to add new
features and improvements in the coming year, we expect the benefits
to include:
- Faster processing. The extent of this improvement is uncertain, but
there is hope that many functions, especially concordances will be
faster.
- Data Notebook will work more like FLEx (e.g. sorting, filtering),
and FLEx will gain some Data Notebook features it does not yet have
(e.g. a greater range of custom field types, such as custom list
reference fields, and saving configurations.)
- A different database engine. MS SQL Server has caused various
installation problems. Hopefully the replacement won't cause so many
problems.
- We expect to use XML file(s) more for storage and back-up, which
will be a safer way to provide long-term back-up and archiving.
- Eventually (not necessarily immediately) greater collaboration
support for closely networked and disconnected co-workers. Also,
greater interoperability with other programs such as WeSay, etc.
- Greater lifetime of development for FieldWorks. As mentioned in the
analogy, the current code is becoming increasingly difficult to
develop, making it slower and harder to add the features and
improvements you request. The "rewiring" will allow us to make
FieldWorks code cleaner and more robust in ways that facilitate years
of future development. Also, new programmers will be able to
contribute more quickly - the learning curve will be less arduous.
- Linux support. The change of underlying database and other changes
should allow rapid production of a Linux version of FieldWorks.

So, the next year is going to require patience. It is frustrating to
me that this delays adding the features you are asking for. On the
plus side, I will have more time to plan and will hopefully have all
the new work ready for the programmers to tackle as soon as the
“rewiring” is completed! I will also have more time for travel and
seeing some of you at work, and hopefully this too will provide me
with more insights into what you need.

Hope with us that the “rewiring” goes faster than expected, and
produces really dramatic improvements in performance! (-:

Working for you,
Susanna Imrie, and the FLEx team

[Colin Suggett]

Susanna,

Thanks for the overview and rationnale for the upcoming year's development.  The only thing that jumped out at me was changing the database engine since we had news in other posts of this being a very robust server which could handle all the data that me might throw at it. 

Colin

[David Baines]

Hi Sussana,

Is the team planning to add support for other database engines, so that FLEx will work with other database engines as well as SQL Server?
Or has a decision been made to replace one database engine with another?

Thanks,
David.

[John_T...@sil.org]

We have not yet made a decision about what database engine(s) to support, except that we know SqlServer will not be the only one because it is not available on Linux and supporting Linux is one of the goals of the changes we are planning. We hope to pick an engine that will replace SqlServer on both platforms, be much easier to install, and give a similar level of data security.

John Thomson

[Steve Miller]

> Is the team planning to add support for other database engines, so
> that FLEx will work with other database engines as well as SQL Server?
> Or has a decision been made to replace one database engine with another?
>

We have not yet made a decision about what database engine(s) to support, except that we know SqlServer will not be the only one because it is not available on Linux and supporting Linux is one of the goals of the changes we are planning. We hope to pick an engine that will replace SqlServer on both platforms, be much easier to install, and give a similar level of data security.

We also are interested that it perform well, and the Linux guys are particularly interested in using an engine that uses little memory. (SQL Server was really meant for servers, not laptops, and is a memory pig.) Getting an open-source database is also a high priority. Fortunately open-source databases have come a long, long ways in the past decade, and we have a wide range to pick from.

To more closely answer David's original question, we could decide to use one database engine for some types of hardware such as low-power devices, another database engine for the more common Windows laptops, and another engine for a web server (someday maybe). However, using three engines instead of one means tripling the effort for programming, installation, and maintenance. We can make better use of programming time if we could find one that does everything we need. That may not be possible in the end, but we're trying. I have put in a couple weeks of research on the topic, if anyone's interested.

The "rewiring" we have in mind should allow us to plug in different databases, and see how they work. It also means that someone really wants a different engine than the one we chose, has sufficient ambition and ability, and hopefully(!) good reason, he can swap out the engine for the one of his preference.

More technically, one of the goals is to remove the object-relational impedance mismatch.

I can provide more details to anyone interested.

Steve

[Kari Valkama]

Hi Steve, 

Thanks for your explanations.

I would like to get more details. 

Yours,

Kari

[Jeff and Peg Shrum]

I asked my Nephew who is in grad school about which SQL servers might work for a new version of Flex.  I thought sometimes Universities hear about things that are not so well known.  For what its worth his comments are below.

 

Uncle Jeff,

 

There are a number of options.  "The best" option is Oracle - it comes with commercial versions on Linux, Windows, Mac, Solaris, etc and is by far considered the fastest, most feature filled database server available.  But it's generally extremely expensive.

 

If you're looking for free systems, MySQL is probably the best free system that you could have access to.  It is available for all the major systems, it's used by lots of commercial and free applications, websites, etc, and supports a large number of engines which can be plugged in - MyISAM which is super fast, InnoDB which is a bit more stable for not corrupting data, and others.

 

Postgres is also available for most of the systems and is MySQL's biggest free competitor.  It's possibly not quite as fast as MySQL if you're using MyISAM, but when compared to InnoDB and the other MySQL storage engines that are, in my opinion, "better" than MyISAM, Postgres is just as good.

 

There are other ones, not quite as big as the above two, like Firebird and SQLite that I know of.  I don't know much about them, but Neil and Andrew were adapting Translator's Workplace to work with Firebird, so it might be a decent option and is known to be very mature, being a fork from Borland's commercial database server.  SQLite is usually used for applications that need the server to be embedded into the application directly.

 

In short, my recommendations, if you don't want to purchase Oracle, are either MySQL or Postgres, and that really depends on what your administrators and programmers want to use.

[Doug Higby]

Hi Colin,

Sunday, May 17, 2009, 7:05:30 PM, you wrote:

Colin> The only thing that jumped out at me was changing the database
Colin> engine since we had news in other posts of this being a very robust
Colin> server which could handle all the data that me might throw at it.

I am very much hoping for anything but SQL server. Having installed FW on
many machines, it is embarrassing how long SQL server takes to install, and
when there is a problem with the installation (maybe 5-10% of the time) it
takes forever to troubleshoot and repair.

As far as robustness once installed, I have nothing but praise. My computer
seized up during an in-class demonstration yesterday. I rebooted, and my
last keystroke was there for all to see!

--
Doug mailto:Doug_...@sil.org
Language Software Coordinator
SIL Africa Area

[Bruce Hooley]

Hi. I've just tried to install FieldWorks 5.4.1 on a Dell Mini 9 Netbook which has an 8Gb C: drive and an 8Gb Kingston SD card for a D: drive. I installed FW on D:, since C: is already getting fairly full. All went well until right at the end when I got the message:

 

"SQL Server Setup cannot install files to the compressed or encrypted folder: C:\Program Files\Microsoft SQL Server\. To continue, make sure that your installation directories are not compressed or encrypted, or specify a different directory, and then run SQL Server Setup again."

 

SQL Server Database Services was the only thing that failed, but it promised that if I tried to run FLEX etc it would crash. It was quite correct and I got the message Installation of Microsoft SQL Server 2005 Express SP2 failed, and Flex did crash when I tried to run it.

 

Actually everything installed except, I think, the Support Services bit.

 

But it was trying to install the SQL stuff on C: drive, and there was no option to put it anywhere else. So is it not possible to run FW on the Dell Mini--at least on the one I have. I see that the standard Mini comes with a 160Gb hard drive, but I don't have that, only an 8Gb (flash??) drive. Whatever that drive is, at least it is not supposed to have any moving parts.

 

Is there any way to get round this problem?

Thanks for any feedback you might have.

 

Bruce Hooley

__________ Information from ESET Smart Security, version of virus signature database 4089 (20090519) __________

The message was checked by ESET Smart Security.

http://www.eset.com

[Cameron and Valerie Hamm]

Bruce,

 

They do make larger flash drives (called Solid State Drives or SSD) in capacities like 16, 32, 64, 128, and even 256 GB, but they are fairly expensive especially on the high end of capacity.  But you could upgrade to a 32 GB SSD and have no problem with memory size being a factor for installation of FLEx.

 

Cam

 

From: flex...@googlegroups.com [mailto:flex...@googlegroups.com] On Behalf Of Bruce Hooley

Sent: Tuesday, May 19, 2009 5:50 PM
To: flex...@googlegroups.com

[Bruce Hooley]

Cam

 

Thanks for your response, but the problem seems to be not a lack of memory, but the fact that C: is an SSD drive. I don't know anything about SSDs, but it would seem that they must store material as compressed files which the SQL installer can't handle. So unless there is a way to install the relevant files to some other location, it looks like FW just won't work on the Mini 9??

 

Bruce

[Cameron and Valerie Hamm]

Maybe someone else can answer this question.  Have you tried re-installing?  Sometimes there are errors on SQL server installation, as I’ve read on this list.

[John_W...@sil.org]

The problem in this case is the compressed drive. Given the size of the drive, I doubt the compression setting can be changed without causing other problems - like filling up the disk.

I don't know if SQL Server can be installed on a different drive - it would at least be a manual procedure.

It looks like you have the minimum configuration for running Windows XP. I'm not sure you would get reasonable performance out of the machine even if you can get everything to install.

I'm thinking that WeSay may be a more reasonable application to run on this type of hardware.

John

"Cameron and Valerie Hamm" <camero...@sil.org> Sent by: flex...@googlegroups.com 2009/05/20 11:23 AM Please respond to flex...@googlegroups.com

To <flex...@googlegroups.com> cc

[Academic Computing SEB]

Sounds as though the SSD disk is set to compress everything as it's default setting. I've set my hard disk to compress everything before when I was running out of space and got the same messages.
Try opening Windows Explorer and navigating to "C:\Program Files\Microsoft SQL Server\"  (if it's not there create it), then right click on the folder and select "Properties" from the pop-up menu. On the "General" tab click on the "Advanced" button. In the 'Compress or Encrypt attributes' section the 'Compress contents to save disk space' checkbox is probably checked. Uncheck this this box, click on the 'OK' and' Apply' buttons. Then try reinstalling Fieldworks. It should fly through everything upto SQL installation and hopefully this time should work.

Mark Skinner

[Bruce Hooley]

Dear Mark & Richard

 

Many thanks for your suggestions. I tried uncompressing the Microsoft SQL Server folder, and I got much further than I had before. But eventually, right at the end, I got this message to say that it couldn't handle compressed files in C:\Documents and Settings\All Users\Application Data\SIL\FieldWorks\Data and please do something about it before trying to run the program. (Running the program just brought up the same message.)

 

But, the problem was I couldn't find the Application Data folder anywhere, either on C: drive or on D: which is where I installed the program. I got as far as C:\Documents and Settings\All Users, but that was the end. Finally, I tried creating the Application Data folder only to be told it already existed. So it turned out to be a  hidden folder. Once I discovered that I was able to go in and uncheck the compression in the Data folder too. After that, FLEX loaded happily.

 

So many thanks to you both for your help.

Bruce

From: Academic Computing SEB

Sent: Friday, May 22, 2009 9:30 PM

To: flex...@googlegroups.com

Subject: [FLEx] Re: Dell Mini 9

[Randy Regnier]

From: David Baines david_...@sil.org

> Is the team planning to add support for other database engines, so that FLEx will work with other database engines as well as SQL Server?
> Or has a decision been made to replace one database engine with another?

Well, I'm not Susanna, but I understand the issue, since I have been doing a lot of experimental work on the revised FieldWorks (FW) base code. The architecture I developed allows for changing from one data storage system to another fairly easily. By 'fairly easily' I mean that had this new system been in place all along, then moving from a Windows only specific database system, such SqlServer, to a cross platform database system, such as Firebird, could have been done in less than one day. In fact, believe it could have been done before lunch. :-)

 

I don't know that there are plans to have users select a data storage system when installing FW, but I understand the plan is to use one that works on both Windows and Linux. I don't think any decisions have been made on any particular data storage mechanism at this time. The new architecture doesn't actually require that decision to be made at this point anyway, since they can be changed fairly easily.

 

Randy Regnier

[Randy Regnier]

From: Colin Suggett colins...@gmail.com

> Thanks for the overview and rationnale for the upcoming year's development.

> The only thing that jumped out at me was changing the database engine since we had news

> in other posts of this being a very robust server which could handle all the data that me might throw at it.

 

SqlServer isn't overworked with the size of our data sets that we put into it. :-) It's main technical drawback is that it cannot run on Linux.

 

Randy Regnier

[Michael Aubrey]

It had been my understanding that one of the problems of SqlServer was that its definitely not designed for the sorts of computers and computer hardware that most of us use - i.e. its not designed for laptops and a different database engine would give better performance for laptops.

Is this correct?

Mike Aubrey

---

From: Randy Regnier <regn...@yahoo.com>
To: flex...@googlegroups.com
Sent: Thursday, May 28, 2009 4:32:38 PM
Subject: [FLEx] Re: What's Next?

[Randy Regnier]

From: Michael Aubrey mga...@yahoo.com

> It had been my understanding that one of the problems of SqlServer was that its definitely not

> designed for the sorts of computers and computer hardware that most of us use - i.e. its not

> designed for laptops and a different database engine would give better performance for laptops.

SqlServer is designed to run on large server systems and serve up quantities of data (or perhaps comb through a sea of data to find select data). It's probably thus safe to say it is not specifically "designed" for laptop use. :-)

 

One less obvious issue is that there is a basic difference in 'worldview' in how any relational database views data and how the FLex application code views data. The technical term for this is "impendance mismatch'. Without going into a boring technical explanation on what the mismatch means, I'll just say it means that the database and the code are both not on the same page, so each has to work in ways that are not optimal.

 

In my experimental work, I was able to get much better performance out of SqlServer. So, it isn't adequate to say SqlServer on a laptop is the issue. I also tested other systems: Firebird, DB4o, Berkeley DB, and an XML file. Each of the tested systems performed better than the current system. The reason for the gain is how the data is actually stored.

 

Randy Regnier

[Michael Aubrey]

That makes sense. Thank you for the explanation - even if its simplified.

---

From: Randy Regnier <regn...@yahoo.com>
To: flex...@googlegroups.com

Sent: Thursday, May 28, 2009 5:29:40 PM

Subject: [FLEx] Re: What's Next?