Aussearch error accessing the BOM

148 views
Skip to first unread message

Susan Mackay

unread,
May 29, 2021, 5:22:14 AM5/29/21
to weewx-user
I have been using Aussearch for a while now but in the last few days I've been getting:

May 29 19:05:34 weather weewxd[8649]: weewx[8649] ERROR user.aussearch: aussearch: cannot download xml file http://www.bom.gov.au/fwo/IDZ00059.warnings_vic.xml: HTTP Error 403: Forbidden

That part is not so bad but I suspect that it is also leading to:


May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: Generate failed with exception '<class 'RecursionError'>'

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: **** Ignoring template /home/weewx/skins/Responsive/index.html.tmpl

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: **** Reason: maximum recursion depth exceeded in comparison

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 322, in generate

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "_home_weewx_skins_Responsive_index_html_tmpl.py", line 1688, in respond

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "_home_weewx_skins_Responsive_index_html_tmpl.py", line 681, in __errorCatcher88

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "<string>", line 1, in <module>

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "/home/weewx/bin/user/aussearch.py", line 381, in __getattr__

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****      if self.root_node is not None:

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "/home/weewx/bin/user/aussearch.py", line 381, in __getattr__

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****      if self.root_node is not None:

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "/home/weewx/bin/user/aussearch.py", line 381, in __getattr__

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****      if self.root_node is not None:

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    [Previous line repeated 489 more times]

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****    File "/home/weewx/bin/user/aussearch.py", line 378, in __getattr__

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****      if child_or_attrib in ['__call__', 'has_key']:

May 29 19:06:08 weather weewxd[8649]: weewx[8649] ERROR weewx.cheetahgenerator: ****  RecursionError: maximum recursion depth exceeded in comparison


This is stopping the generation of the whole web page file.

I suspect that this is part of the on-going process by the BoM  to restrict access to various files but I could be wrong.

Before I try to take this out, is there any workaround?

Susan

Darryn Capes-Davis

unread,
May 29, 2021, 8:05:50 AM5/29/21
to weewx-user
Hi Susan,

See the GitHub repo for update aussearch.py. The code now behaves more like a browser with more headers and keep alive so not blocked by BOM. 

Regards

Darryn

Neville Davis

unread,
May 29, 2021, 5:54:15 PM5/29/21
to weewx-user
 Yes happening to me as well… started on Thursday,,,my normal fix of getting the /var/lib/weewx/aussearch reloaded not working…I will check the repo 
Thanks Darryn

Neville Davis

unread,
May 29, 2021, 6:43:18 PM5/29/21
to weewx-user
Darryn

new aussearch.py installed (large no of changes since my feb install compared with BBedit) 
Still not working...the json files are not being generated, the xml files are OK....in /var/lib/weewx/aussearch.

logs show

May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'weewx.cheetahgenerator.CheetahGenerator'
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****  maximum recursion depth exceeded
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 196, in run
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      obj.start()
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 281, in start
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      self.run()
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 149, in run
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      self.initExtensions(gen_dict[section_name])
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/cheetahgenerator.py", line 193, in initExtensions
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      self.search_list_objs.append(class_(self))
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/aussearch.py", line 213, in __init__
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      self.refresh_time = float(self.generator.skin_dict['AusSearch']['refresh_time'])
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/aussearch.py", line 535, in __init__
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      (self.local_file_path, check_datetime))
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/aussearch.py", line 548, in __getattr__
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      syslog.syslog(syslog.LOG_DEBUG, "aussearch: json file does not exist: %s" %
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/aussearch.py", line 548, in __getattr__
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      syslog.syslog(syslog.LOG_DEBUG, "aussearch: json file does not exist: %s" %
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/aussearch.py", line 548, in __getattr__
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****      syslog.syslog(syslog.LOG_DEBUG, "aussearch: json file does not exist: %s" %
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****    [Previous line repeated 491 more times]
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****  RecursionError: maximum recursion depth exceeded
May 30 08:30:24 raspberrypi weewx[700] ERROR weewx.reportengine:         ****  Generator terminated

Carlingford Weather

unread,
May 29, 2021, 6:46:59 PM5/29/21
to weewx...@googlegroups.com
Hi Neville,

You may need to empty files in /var/lib/weewx/aussearch. Also make sure you have restarted weewx. 

Cheers

Darryn

From: weewx...@googlegroups.com <weewx...@googlegroups.com> on behalf of Neville Davis <nevilled...@gmail.com>
Sent: Sunday, May 30, 2021 8:43:18 AM
To: weewx-user <weewx...@googlegroups.com>
Subject: [weewx-user] Re: Aussearch error accessing the BOM
 
--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/tsKLfRh0QvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9eec16fe-633b-4ab5-9c17-22c85156ce0en%40googlegroups.com.
Message has been deleted

Neville Davis

unread,
May 29, 2021, 7:00:14 PM5/29/21
to weewx-user
Had deleted files, now have restarted json files now generated.

Screen Shot 2021-05-30 at 8.56.16 am.png

Thanks Darryn

Carlingford Weather

unread,
May 29, 2021, 7:13:37 PM5/29/21
to weewx...@googlegroups.com
Great. Good to see everyone working again. Hopefully we are in the clear for a while to come. I did enter into the form BOM points you towards and have started a conversation as to why assets that are available via a browser and not available to well behaved scripts, and what is the legislative restriction that they are following. Nothing back yet.

Sent: Sunday, May 30, 2021 9:00:13 AM
To: weewx-user <weewx...@googlegroups.com>
Subject: Re: [weewx-user] Re: Aussearch error accessing the BOM
 

Susan Mackay

unread,
May 29, 2021, 7:23:26 PM5/29/21
to weewx-user
Thanks for the tip to download the latest file form the repository - all working again now.
Also thanks for the quick response.
Susan

Cameron D

unread,
May 30, 2021, 8:42:35 AM5/30/21
to weewx-user
Yes, I "entered into the conversation" a couple of weeks ago.  So far it's been very one-sided.
The impression I got from the introductory pages was that they are only interested in commercial opportunities.

On Sunday, 30 May 2021 at 9:13:37 am UTC+10 Darryn Capes-Davis wrote:

Graham Eddy

unread,
May 30, 2021, 8:57:29 AM5/30/21
to weewx...@googlegroups.com
i thought BOM was pushing non-commercial ‘citizen scientist’ stuff to project based on/in UK’s WOW

Susan Mackay

unread,
May 30, 2021, 10:58:06 PM5/30/21
to weewx-user
(Note:I used to work for the BoM a number of years ago - I DO NOT speak on their behalf. These opinions are my own entirely.)
This is getting off topic but.....

When I was at the BoM they certainly did talk about WOW (and I even have my weather station reporting to it) but I don't recall it going any further. (That could have been after I left.)
However there always was a tension between what were considered commercial services (and some like the airline forecasts were clearly commercial) and what were public interest and therefore freely available.
With the Governments 'efficiency dividends' being on-going, the end result is that they may be re-classifying some of their services, especially when there are apps around that use the BoM data by 'screen scraping' and other techniques.
My opinion is that the Government should fund the various federal departments properly so they can provide basic services (and I consider the 'warnings' that were the original problem as a basic service) in a freely accessible way. However there is probably a grey area - some of the 'basic' services may come as a by-product of the more 'expensive to create' products that would otherwise be classified as commercial.
Susan

Carlingford Weather

unread,
May 31, 2021, 5:24:50 PM5/31/21
to weewx...@googlegroups.com
Thanks for the insight Susan. 

From: weewx...@googlegroups.com <weewx...@googlegroups.com> on behalf of Susan Mackay <vk3...@gmail.com>
Sent: Monday, May 31, 2021 12:58:05 PM

To: weewx-user <weewx...@googlegroups.com>
Subject: Re: [weewx-user] Re: Aussearch error accessing the BOM
--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/tsKLfRh0QvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages