CSV Import
Clayton Carter
Thanks for volunteering to implement CSV import. There is some
rudimentary import functionality, which may provide a roadmap, but it
is old and needs to be made more robust.
===[ A Brief Intro On How This System Works ]===
The software implements what I've called "Data Models". This is just
an API which defines how the software modules interact w/ a data
source or data store. This API is defined by the CPSDataModel
abstract class and all "data models" or "data sources" MUST extend
this class. The API defines methods to -- among other things -- read
individual records (crops, varieties or plantings), read groups of
records (as lists and TableModels), create, delete or update
individual and groups of records.
The primary data model used by the software is the HSQLDB, but the CSV
module also implements [most of] this API. Because we only use the
CSV file to import and export, it isn't really meant to provide ALL of
the features of the full Data Model class. Once a data model has been
implemented for a particular format, importing from and exporting to
that format is fairly trivial: we ask formatA_DataModel for a list of
records, which we hand off to formatB_DataModel to create.
In addition to the CPSDataModel API, we also have a CPSExporter
interface. This allows particular DataModels to identify themselves
as "exporters". Once they identify themselves as such, the about
pseduo-code-statement could changes to: we ask formatA_DataModel for a
list of records, which we hand off to formatB_DataModel to EXPORT TO
AN EXTERNAL FILE. This is a trivial change in semantics, but it
allows a DataModel module to say whether it is meant to save data to
an internal format vs an external format. For instance, the HSQLDB
module does not implement CPSExporter, so it will cannot be used to
export data to an external data store or for backup or whatever; the
CSV module DOES implement this interface and therefor CAN be used to
save data to an external format. The GUI uses this class (via a "if (
module instanceof CPSExporter )" call) to create the "export" menu
items.
Now, to complicate things, the CSV module doesn't really use any of
the methods defined in the DataModel class. It used to, but we've
moved past that and it implements that class -- really -- for no good
reason.
===[ Methods To Implement ]===
First, we'll probably want to create another interface: CPSImporter
with methods analogous to those in the CPSExporter interface, except
that they would RETURN lists of records instead of accept them as
parameters.
From there, look at these methods in the CSV module:
public ArrayList<CPSCrop> getCropsAndVarietiesAsList() {
public CPSCrop getCropInfoForRow( int selectedRow ) {
In these methods, you'll see how we used to handle a VERY limited
import function. This will give you an idea on how to start, but I
think that all of your work will likely take place in the methods
which you define in the CPSImporter interface.
You should also implement the methods
public abstract int propNumFromPropName( int recordType, String
propertyName );
public abstract String propNameFromPropNum( int recordType, int
propertyNum );
which will correlate property numbers (defined in
CPSDataModelConstants class) to there "names" which may be different
for every DataModel, but in this case will be the same column names as
used in the HSQLDB module.
===[ My Recommendations ]===
The exported CSV files definitely conform to a certain format:
1) several lines of comments and metadata
2) one line that defines the column names and the order in which they
appear in the file
3) all of the data
4) one final line to signal the end of the file
You should feel free to change this format slightly. For instance, we
can add more metadata info to the header. This might be useful when
exporting/importing crop plans which have years and descriptions
attached to them.
I would suggest the following procedure:
1) Open file
2) Parse header metadata (if any exists)
3) Read the first line, which contains the column names. Parse this
line and record the order in which these columns are specified, as
this will be used or every line (record) there after. Here's some
pseudo code:
read column_line
split column_line into columns
foreach column in columns
get propNumFromPropName for column
save ( propNumFromPropName for column ) into array at ( index of
column in columns )
done
After that, you should have an array (or ArrayList) which will store
the property number of each column name.
4) Now you can move through that array (or list) as you move through
each record, using the stored property number (in the array or list)
to store the value which you read from the record.
As I reread that, I'm not sure I can even make sense of it. Let me
know as you have questions!
Clayton
