How do I speed Up dataset.ReadXML????????

[CEDevice_Brian]

Here is my current code:
Dim fsReadXML As System.IO.FileStream
Dim myXmlReader As System.Xml.XmlTextReader

fsReadXML = New System.IO.FileStream(xmlFile, System.IO.FileMode.Open)
myXmlReader = New System.Xml.XmlTextReader(fsReadXML)

XREFDataset.ReadXmlSchema(myXmlReader) ', XmlReadMode.ReadSchema) 'filename

Dim dt As DataTable
For Each dt In XREFDataset.Tables
dt.BeginLoadData()
Next

'close before reading file straight
fsReadXML.Close()
XREFDataset.ReadXml(xmlFile)
'XREFDataset.ReadXml(myXmlReader)
' or close after reading stream
'fsreadXML.close

For Each dt In XREFDataset.Tables
dt.EndLoadData()
Next

The .ReadXML statement is working for nearly 20 minutes before my Win CE device gives a "Program memory is low" error. I can read the same XML file on the desktop into a dataset within 2 seconds. The XML file is 1.5M.
Please help?

Brian

[Ignacio Machin ( .NET/ C# MVP )]

Hi,

If you look in the archives you will see that the CF has performance
problems reading big XML files. My advice is to make it smaller or find
another way, see my previous post in the xml data storage? thread.

Cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"CEDevice_Brian" <CEDevic...@discussions.microsoft.com> wrote in message
news:1454FE5C-E05D-4FD8...@microsoft.com...

[Ginny Caughey [MVP]]

Brian,

As Ignacio pointed out, you will get the biggest performance improvement by
using some other file format besides XML. For example, CSV files are quite a
bit faster to parse, and there is a CSV data adapter on www.opennetcf.org.

--
Ginny Caughey
.Net Compact Framework MVP

Have an opinion on the effectiveness of Microsoft Embedded newsgroups?
Let Microsoft know!
https://www.windowsembeddedeval.com/community/newsgroups

"CEDevice_Brian" <CEDevic...@discussions.microsoft.com> wrote in message
news:1454FE5C-E05D-4FD8...@microsoft.com...

[CEDevice_Brian]

The xml could contain from 1 record to 10,000 records ranging in fields from 6 to 20.
I adjusted the memory allocation on my device so it doesn't error out but it still takes 4 to 5 minutes to read in 2200 items in one table with 20 columns. The PC takes 3 to for seconds for the same operation.

Do you think the 4 to 5 minutes is the best I can get on CE.NET 4.10, Intel ARM-XScale device?

[Ignacio Machin ( .NET/ C# MVP )]

Hi,

Please, remember that the PPC has a far less powerful processor than your
desktop, in the escenario that you describe is better using something like
a CVS file, if yours records can have comma as part of theirs values then
use a control character as a record separator instead of a comma, I do this
and the improvements are HUGE

I would say that you will NOT GET much better performance using an XML file.

Cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"CEDevice_Brian" <CEDevic...@discussions.microsoft.com> wrote in message

news:973F9FE6-4164-40B6...@microsoft.com...

[CEDevice_Brian]

OK
I'll try it.

Thanks for the input Ignacio and Ginny!

"Ginny Caughey [MVP]" wrote:

> Brian,
>
> As Ignacio pointed out, you will get the biggest performance improvement by
> using some other file format besides XML. For example, CSV files are quite a
> bit faster to parse, and there is a CSV data adapter on www.opennetcf.org.
>
> --
> Ginny Caughey

> ..Net Compact Framework MVP

[William Ryan eMVP]

10,000 records is going to be brutal - definitely go with what hte others
have suggested. I've had some pretty serious wait times on files half that
size. It's great for small files but XML processing has some overhead b/c
of parsing the tags, the csv approach will be a LOT Faster.

--

W.G. Ryan, eMVP

Have an opinion on the effectiveness of Microsoft Embedded newsgroups?
Let Microsoft know!
https://www.windowsembeddedeval.com/community/newsgroups

"CEDevice_Brian" <CEDevic...@discussions.microsoft.com> wrote in message

news:973F9FE6-4164-40B6...@microsoft.com...

[Ilya Tumanov [MS]]

Brian,

1.5 MB XML should be loaded in about 45 seconds on PXA255/400 device
running PPC 2003 and CF SP2.
The reason you're running out of memory is what you (probably) do not have
schema and inference is used.

Here are few tips on how to improve XML/DataSet performance:

1. Install SP2.
2. Use schema. It might be created programmatically (fastest), loaded from
separate file or present in the data file.
3. Avoid nested tables, use none-nested relations with surrogate keys.
4. Map columns as attributes.
5. Avoid using many DateTime columns. Consider storing DateTime.Ticks
instead.

With all that you might go from 20 minutes to well under a minute.

Best regards,

Ilya

This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
> Thread-Topic: How do I speed Up dataset.ReadXML????????
> thread-index: AcReqzw7tB5M+LiZRvme04ZJXz4kjQ==
> X-WBNR-Posting-Host: 207.177.236.69
> From: "=?Utf-8?B?Q0VEZXZpY2VfQnJpYW4=?="
<CEDevic...@discussions.microsoft.com>
> Subject: How do I speed Up dataset.ReadXML????????
> Date: Wed, 30 Jun 2004 07:05:01 -0700
> Lines: 30
> Message-ID: <1454FE5C-E05D-4FD8...@microsoft.com>
> MIME-Version: 1.0
> Content-Type: text/plain;
> charset="Utf-8"
> Content-Transfer-Encoding: 7bit
> X-Newsreader: Microsoft CDO for Windows 2000
> Content-Class: urn:content-classes:message
> Importance: normal
> Priority: normal
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.0
> Newsgroups: microsoft.public.dotnet.framework.compactframework
> NNTP-Posting-Host: TK2MSFTNGXA03.phx.gbl 127.0.0.1
> Path: cpmsftngxa06.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGXA03.phx.gbl
> Xref: cpmsftngxa06.phx.gbl
microsoft.public.dotnet.framework.compactframework:56322
> X-Tomcat-NG: microsoft.public.dotnet.framework.compactframework

[Peter Foot [MVP]]

Have you considered using an alternative method of storing your data, 1.5mb
is quite large for an XML file on a mobile device, and since you are loading
the entire contents into a DataSet in memory it is pretty resource
intensive. However if you want to continue with textfiles you could use CSV
instead, OpenNETCF have a CsvDataAdapter which will allow you to parse CSV
which will have a lower file size and hence load faster than the equivalent
data in Xml.

Peter

"CEDevice_Brian" <CEDevic...@discussions.microsoft.com> wrote in message
news:1454FE5C-E05D-4FD8...@microsoft.com...

[Ginny Caughey [MVP]]

Perhaps that's just how long it's going to take with the data you have. Do
you have a lot of fields that require converting from string to something
else?

--
Ginny Caughey
.Net Compact Framework MVP

"CEDevice_Brian" <CEDevi...@discussions.microsoft.com> wrote in message
news:2D0BC144-6389-4DC7...@microsoft.com...
> I'm now creating a CSV with 11,000 records, 20 fields per.
>
> Code used is
> csvDA = New OpenNETCF.Data.CSV.CSVDataAdapter(csvFile, True)
> 'fill dataset from csv
> csvDA.Fill(XREFDataset, tName)
>
>
> It's been going for over 15 minutes now.
>
> What am I doing wrong?????

[CEDevice_Brian]

Nope.
I did discover by running the csvDA.fill on the PC that there were extra commas in a couple records. On the device, csvDA.fill just kept going until I tried to kill the app. Then it threw the error. I have the data cleaned up now, but it is still painfully slow.
Thanks everybody for the help, but at this point, I am thinking that this section may inspire the users to go with the wireless network so they don't have to work in batches.

"Ginny Caughey [MVP]" wrote:

> Perhaps that's just how long it's going to take with the data you have. Do
> you have a lot of fields that require converting from string to something
> else?
>
> --
> Ginny Caughey

> ..Net Compact Framework MVP

[Ignacio Machin ( .NET/ C# MVP )]

Hi,

Try my suggestion, using a text file where the separator is a control char,
20 for example, that avoid having to process the fields looking for
embedded comma inside the values.

also maybe in your case with so many records if you need to convert to
numeric values it may be faster ( but not elegant ) keep them as string and
convert them in the spot.

just let me know if you need the code for this.

Cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

[CEDevice_Brian]

OK. Looks like a logic change is in order. ReadXML takes way too long. CSVDataAdapter still ttakes to long and simply keeps going in certain data situations.
I am trying to accomplish an inventory table to search and update on the CE device. I do not have SQL Server on the host side of the app. What can I build on the host side that would be the most easily implemented as a dataset or .sdf on the device side?

"Ignacio Machin ( .NET/ C# MVP )" wrote:

[Ginny Caughey [MVP]]

Ignacio,

I don't know if this is THE fastest because I certainly didn't exhaust all
the possibilities, but I ended up using a text file on one project as the
transfer medium with the fields delimited with the pipe symbol (|) and each
record ending with CRLF. I separate the records into fields using
String.Split, which performs nicely, and load them into business objects
converting data into the approriate type as needed. The more non-string data
you have with this approach, the longer it takes. Since the collection of
these business objects is derived from ArrayList, you can use all the normal
databinding.

HTH,

--
Ginny Caughey
.Net Compact Framework MVP

"CEDevice_Brian" <CEDevi...@discussions.microsoft.com> wrote in message
news:20FAB387-B52B-4172...@microsoft.com...