I know the diff between RAM and storage. Geez.
>> My Usenet.sqlite file is 17GB (mostly cola), but could be cut by 25% at
>> least if I deleted all the bullshit from alt.checkmate and
>> talk.politics.guns, etc.
>>
>
> Hey, if you trust it to sqlite, more power to you.
>
> Personally, a set that big, I'd want it in a "real" database.
When it got too big for Access (which at a 2GB max file size happened
quickly), I started looking for an alternative. I thought SQLite was
too wimpy, but found it could handle large datasets, is file-based, easy
to install and administer and backup, and has great documentation and
support. So I gave it a try and found out it rocks for my usage.
The only oddity I don't like is it uses non-static data typing, meaning
you can insert text data into columns you create as integer, and vice versa.
As it's public domain and is distributed with Android and other systems,
SQLite claims to be the most widely used db in the world. I don't doubt
it at all.
>>>>> Or find an old piece of iron, put Linux on it, and install MariaDB
>>>>> there.
>>>>
>>>> Linux is unnecessary at ALL times.
>>>>
>>>> I'm a one-Windows-machine, local DB man.
>>>>
>>>>
>>>>> Or if you want turnkey: buy a Synology Diskstation w/disks of your
>>>>> choice, then go into the package center and install MariaDB 10.
>>>>>
>>>>> No need to use Oracle for your hobby Windows system.
>>>>
>>>> I like Oracle. I like my hobby. I like my hobby running on Oracle.
>>>>
>>>> I also like SQLite, which is a lot easier to setup, use and
>>>> administer.
>>>
>>> Yes, but up to 60GiB in SQLite? How does that even work?
>>
>> It works due to the genius of the original SQLite developer, D. Richard
>> Hipp, who scoffs at 60GiB. His program will put up to 281TB in a single
>> file.
>
> I didn't realize it would handle that much. What's the speed like?
In my system, with a few million rows spread among a dozen
properly-indexed tables, the performance is smokin'.
Here's what I said to rbowman a while back:
----------------------------------------------------------------------------
I've sung the praises of SQLite many times. It's not a multi-user
system suitable for large numbers of concurrent writing, but it's
fantastic for many scenarios, including fairly large databases. It
doesn't support the full SQL standard, but plenty enough. The
documentation is first-rate, and the SQLite Forum is great. I recently
got a quick answer directly from D. Richard Hipp, the original SQLite
developer. It's still under active development.
At one point last year I had SQLite, MariaDB, PostgreSQL, Firebird, DB2
Express and Oracle Express running at the same time on my old Win10
system, each populated with identical data [edit: from my Usenet
database]. The performance of each was
plenty acceptable, but SQLite, Oracle and PostgreSQL were consistently
faster (maybe 5%-10% difference between the servers, using a PyQt app).
SQLite and Firebird were the least 'robust' but I completely trust
SQLite with my data. It's a very well-tested system.
----------------------------------------------------------------------------
Note the 'datasource' list at the top of the screen
https://imgur.com/a/1F5GfVk
I could choose some criteria, then go down the list and click each
server and it would immediately run the same query, return the same
data, and report the timing.
>> It could easily hit 60GB (or GiB) by downloading the entirety of a bunch
>> of newsgroups. I only have cola posts since mid-2003, but it was opened
>> for business in 1994.
>>
>> And my latest collection/storage methodology adds about 50% to the
>> minimum space required: I first download the headers and body of each
>> Usenet message into python list objects that are stored in a master
>> 'messages' table. Later at my leisure I split them apart into 12 or so
>> tables with relations. The granularity of the data and the relations
>> are what lets me search and summarize posts and metadata so deftly.
>
> A perfectly acceptable hobby for your hobby Windows system. 😉
A perfectly grand hobby for my professional, commercial-grade,
built-MS-tough, Windows system!