driver architecture question

[Vince Skahan]

Is it possible to break up monolithic drivers into components that do pieces of the puzzle ?

Using ecowitt as an example, the vendor has been updating their gateways to be able to publish readings to MQTT.  Using Rich's mqttsubscribe as a driver works great.

But...some ecowitt gateways now support backfill capability from SD card, and some backfilling via queries to the vendor's ecowitt servers also is possible.  So there are multiple backfill scenarios that are possible (including no backfill).

What I'm wondering is if it possible to have a two-component driver notionally.  One piece that would handle backfilling on weewx startup and then exit.  A second that would then take over and do the continual readings after that.

In other words, follow the unix philosophy of having small utilities that do one thing well.

Is that kind of pluggable setup possible (or crazy), or would the weewx way be to have a driver that 'only' does backfill and a service that does the realtime readings to the db ?

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> Is it possible to break up monolithic drivers into components that do
> pieces of the puzzle ?
>
> Using ecowitt as an example, the vendor has been updating their gateways to
> be able to publish readings to MQTT. Using Rich's mqttsubscribe as a
> driver works great.
>
> But...some ecowitt gateways now support backfill capability from SD card,
> and some backfilling via queries to the vendor's ecowitt servers also is
> possible. So there are multiple backfill scenarios that are possible
> (including no backfill).
>
> What I'm wondering is if it possible to have a two-component driver
> notionally. One piece that would handle backfilling on weewx startup and
> then exit. A second that would then take over and do the continual
> readings after that.

> Is that kind of pluggable setup possible (or crazy), or would the weewx way
> be to have a driver that 'only' does backfill and a service that does the
> realtime readings to the db ?

I am not following your discussion (but I get the goal).

Personally I see live-data and backfill together as just how things
ought to be, with some stations lacking a way to do backfill.

As I understand it, "driver" and "service" are weewx architecture terms
for code that can be hooked in via weewx APIs and

a driver provides the single "weather station" that is the base of the
weewx setup

a service provides data, but it is somehow extra

and that's the only distinction: one must have exactly one driver and
then can have zero or more services.


In the case of ecowitt stations which (existing outside of weewx) have
the ability to get data as it happens and also backfill, that seems to
me to be exactly like a Davis VP2 data logger. Except that the protocol
details on the driver/hardware side are totally different.


Thinking about your modularity comments, I think they're really about
letting different aspects be maintained separately, perhaps by different
people. Whether they are plugged into weewx via a common driver or not
is not really the big concern.


This discussion makes me think of the common idiom in Home Assistant (HA),
where there is

A python (async) module to talk to some kind of device in python.
This has nothing to do with HA, and can be used in any python program.
Examples are aioshelly, aioesphomeapi, aiounifi and many more.
However, often it is motivated by HA where one wants to write an
integration also.

An "integration" which interfaces to HA APIs (about how to create
automation objects, report their status, and accept commands) and to
the device by making calls into the async module. This code does not
have any details of the protocol to talk to the device. But it will
be affected by the semantics of what you can do.

Usually the same aiofoo can be used by other home automation systems.


I would suggest that the way forward is to create python modules that
turn the ecowitt API into something pythonic, perhaps with a backfill
module that augments the main one, or something, and then to have a
driver that uses this module. This is likely more refactoring than
anything else. I would expect, without any more than gut feel basis,
that such a python module would not be really enormous.

While I've mentioned async, that's only because architecturally HA is
async. weewx is not; at least "egre -R async ." turns up empty :-) This
great python traditional/async dichotomy has led to there being both
py-foo and py-aiofoo for many values of foo.

[Vince Skahan]

I think you follow just fine.  I'm asking modularity what-ifs.

Currently drivers for gear supporting backfill are monolithic.  They do vendor-specific backfill then talk to the hardware for data, again vendor-specific. Combining the two tasks into a monolithic driver can get very complicated.

Ecowitt is a good example.  Its backfill is pretty odd under the hood.  The driver part is even more complicated as they flood the market with more and more sensors and their rather poorly documented APIs change over time.

Given weewx's requirement for 'some' driver and its support for optional services, can the typically monolithic 'backfill + drive' driver can be componetized ala:

• (driver) component-X that does backfill via the mechanism the user wants, if any
  
• (service) component-Y that does the periodic talking to the hardware for sensor readings

Using ecowitt as an example:

• you might have a gw3000 driver which backfills from SD card
  
• or you might have a driver that backfills from API query of the ecowitt server
• or you might want a null driver that does nothing at all (no backfill)

And then of course add a service to talk to the hardware or external API or whatever for sensor data

• you might be able to talk to hardware via the original gw1000 driver
• but other gear might require a different ecowitt driver to talk to the hardware
• or you could query the vendor server for sensor data
• or you could get data via MQTTSubscribe
• and so on

Basically be able to split the two pieces (backfill + sensor data)  into two components.  Componentize.  Be able to break up monolithic things.  think ala-carte.

Hope that helps explain...

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> Currently drivers for gear supporting backfill are monolithic. They do
> vendor-specific backfill then talk to the hardware for data, again
> vendor-specific. Combining the two tasks into a monolithic driver can get
> very complicated.

Sure, but is the backfill *concept* really hard? What I mean is
consider that you have a python module that deals with the details, and
you have calls like open to return an object, and then a
get_record_indexes call to return a sequence of timestamps, and then
get_record, that returns a dictionary. Then it's just doing that and
matching up with the db and adding. It's not a lot of code, and it
could be a common routine if the per-vendor modules are all subclasses.

I assert that the huge amount of code in an online/backfill driver is
not because of online/backfill but because it has code for two different
interfaces each of which is complicated.

> Ecowitt is a good example. Its backfill is pretty odd under the hood. The
> driver part is even more complicated as they flood the market with more and
> more sensors and their rather poorly documented APIs change over time.

Sure, but that's all 'ecowitt is messy', rather than making it net
easier to split along those lines.

> Given weewx's requirement for 'some' driver and its support for optional
> services, can the typically monolithic 'backfill + drive' driver can be
> componetized ala:
>

> - (driver) component-X that does backfill via the mechanism the user
> wants, if any
> - (service) component-Y that does the periodic talking to the hardware
> for sensor readings

That would be weird. The same values (e.g. outTemp) would be sometimes
coming from driver, and sometimes from service, and if you aren't
missing anything, there's be no backfill archive record. It's really
the same device and the same data, with just catchup on boot, like
Davis.

> Using ecowitt as an example:
>

> - you might have a gw3000 driver which backfills from SD card
> - or you might have a driver that backfills from API query of the
> ecowitt server
> - or you might want a null driver that does nothing at all (no backfill)

That's a great argument for having a backfill class definition and three
implementations (in separate .py files) and then config to say which one
you want, with the code that calls it being simple. Basically I'm
saying that traditional module design tames the complexity, or at least
encapsulates it and prevents extra spaghetti connections.

> And then of course add a service to talk to the hardware or external API or
> whatever for sensor data
>

> - you might be able to talk to hardware via the original gw1000 driver
> - but other gear might require a different ecowitt driver to talk to the
> hardware
> - or you could query the vendor server for sensor data
> - or you could get data via MQTTSubscribe
> - and so on

Aside from MQTTSubscribe, same answer - multiple modules that implement
the "fetch online data" class.

For MQTTSubscribe, there should be a module that gets all the data from
MQTT and keeps it available in a pythonic (dict surely) way. Then the
MQTTSubscribe service would be... a service that just uses this and
injects database columns that aren't part of the "station". And a
driver would have this as one of the choices.

> Basically be able to split the two pieces (backfill + sensor data) into
> two components. Componentize. Be able to break up monolithic things.
> think ala-carte.

Splitting the code is great, into modules that can be composed and used
in various ways. But I do not believe that extending that split into
driver/service is helpful.


Looking at Home Assistant, which has a vast number of integrations
(weewx equivalent of service, sort of, because HA has no concept of a
single thing which is the main point), there's an event bus, entities,
etc. which correspond to weewx's database and what should be assembled
into LOOP and ARCHIVE. Then there are a vast number of integrations
that bridge to any device one can. The core doesn't know anything about
those devices, and when writing an integration you can ignore the rest.
And then further the details of talking to that device are abstracted
into an async python module that is not at all about HA.

Pretending async/not divide did not exist (counterfactual but for the
discussion), suppose one wrote an ecowitt online module that you can use
to query the local API and return a dict. Then one could use that same
code to write a HA integration that would basically call open and then
'get me the list of variables' and create entities, and then once/minute
call read and use the dict to update the entities. Then the weather
values would natively appear in HA. All of this with only writing the
code to talk HTTP API and produce a dict of values once. And not
implementing HTTP, but using the requests module probably (as the WLL
driver does).

[bell...@gmail.com]

Hmm, a service that performs catchup for the gw3000

Or 

A driver that loads MQTTSubscribe’s MQTT processing

Sounds like fun projects I might have to get  a gw3000. 

Where is the api doc? I should be able to write a lot of the code without the hardware.

rich

[matthew wall]

> On Jan 16, 2026, at 19:05, Greg Troxel <g...@lexort.com> wrote:
>
> Vince Skahan <vince...@gmail.com> writes:
>

>> Using ecowitt as an example:
>>
>> - you might have a gw3000 driver which backfills from SD card
>> - or you might have a driver that backfills from API query of the
>> ecowitt server
>> - or you might want a null driver that does nothing at all (no backfill)
>
> That's a great argument for having a backfill class definition and three
> implementations (in separate .py files) and then config to say which one
> you want, with the code that calls it being simple. Basically I'm
> saying that traditional module design tames the complexity, or at least
> encapsulates it and prevents extra spaghetti connections.

for a long time i have wanted to write a standalone 'weectl upload' that is analogous to the 'weectl import'. the upload would take data from the weewx database and push it to the desired data storage. my specific use case is influx, but any uploader that supports other-than-just-live-data is a candidate. think wunderfixer, but generically written to use an uploader to drive it.

a 'weectl backfill' would be a similar, useful tool. that would pull data from some source, whether it is online or hardware, and put the data into the weewx database. this is a bit more tricky, since the source could be defined by an uploader *or* a driver. but it probably just means massaging the api a little bit for each.

m

[matthew wall]

> On Jan 16, 2026, at 11:37, Vince Skahan <vince...@gmail.com> wrote:
>
> What I'm wondering is if it possible to have a two-component driver notionally. One piece that would handle backfilling on weewx startup and then exit. A second that would then take over and do the continual readings after that.

that already exists in a weewx driver. one function that gets historical data, and another that gets 'live' data.

> In other words, follow the unix philosophy of having small utilities that do one thing well.
>
> Is that kind of pluggable setup possible (or crazy), or would the weewx way be to have a driver that 'only' does backfill and a service that does the realtime readings to the db ?

this really depends on the hardware. in most cases, there is one way to query the hardware for historical data (if it is supported) and a different way to query for 'live' data. i think only ecowitt has different protocols for talking to the same hardware device.

i think that grouping based on backfill and realtime is not as useful as providing a generic python interface to the hardware. the problem is, there is no generic interface.

weewx defines a very simple interface with functions to get historical data, get live data, set/get the time, etc., with no-ops when the hardware does not support the functionality.

it is nice to have *everything* about a single type of hardware in a single python file. the plumbing to initiate communication, handle errors, etc is going to be the same whether you are doing historical reads or live reads.

i often wish that a driver could be standalone, i.e., not require weewx in order to run, test, and do useful things. we typically write drivers now so that they have *some* of this independence, e.g., a hardware abstraction that talks to the driver and does things without weewx, even though weewx is required to actually load the driver file (because of the python dependencies).

maybe weewx could use the home assistant interface for its drivers instead of the weewx api? or maybe someone has defined a better, more generic way to talk to hardware?

in most cases, the driver code never changes. only when the hardware gets firmware updates does the driver software have to change.

m

[Greg Troxel]

matthew wall <mwall...@gmail.com> writes:

> i often wish that a driver could be standalone, i.e., not require
> weewx in order to run, test, and do useful things. we typically write
> drivers now so that they have *some* of this independence, e.g., a
> hardware abstraction that talks to the driver and does things without
> weewx, even though weewx is required to actually load the driver file
> (because of the python dependencies).
>
> maybe weewx could use the home assistant interface for its drivers
> instead of the weewx api? or maybe someone has defined a better, more
> generic way to talk to hardware?

That's sort of what I'm trying to suggest, separating into

python module that lets (some python code)
open/read/get-backfill/configure/etc., basically making the device
interface pythonic but not really trying to change it. This code
would not be part of weewx, and could be used by other python
programs.

weewx driver, that adapts the device behavior to the weewx
database/API


I don't think it would help to use the HA internal API. It's all
about creating "entities", such as "sensor" which has a value and unit,
and then HA internally stores those in a DB and prunes them. They have
whatever names the integration wants to call them, sort of
sensor.outside_temperature or switch.living_room_lamp. Sensor values
are stored as they arrive.

weewx has a always-keep database, with a concept of archive records, and
a fixed schema into which data is put, because the skin layer and
general interop wants things organized the same way. Plus, HA is
~entirely 'async def'.

So I was merely suggesting adopting the concept of splitting code into
"make the hardware look pythonic" and "adapt semantics to weewx". I
suspect there is mostly that split in existing drivers, and am arguing
that forcing that harder by module separation would help abstraction,
understanding, maintainabilty and reuse.