Hmm, Aiko gateway seems to have failed.

10 views
Skip to first unread message

Luke Weston

unread,
Mar 9, 2010, 9:53:39 AM3/9/10
to Aiko distributed platform
My gateway seems to be misbehaving. This is strange because I've just
had it sending there unattended, and it has been fine, logging sensor
data up onto Smart Energy Groups for quite a while previously, without
maintenance. But now it does not seem to be sending meaningful sensor
data - however, SEG still says that the device *is online*. I can SSH
into the router, and ping smartenergygroups from inside the router, so
its connectivity is fine.
I suspect that the gateway side of the system is fine but something
might be wrong with the Arduino side.

When I telnet localhost:2000 (after killing the lua process) there are
no sensor data expressions coming in on the ser2net socket - there is
nothing.

Any ideas?

Thanks.

Sam Sabey

unread,
Mar 9, 2010, 3:29:20 PM3/9/10
to aiko-p...@googlegroups.com
Hey Luke,

How long had it been running? Nearly 4 weeks? Have you rebooted the
gateway?

These is some fault in the Aiko Andy and I have found, we are
investigating...

Sam.

...from @samotage on the go.

On 10/03/2010, at 1:53 AM, Luke Weston <reindeer...@gmail.com>
wrote:

> --
> You received this message because you are subscribed to the Google
> Groups "Aiko distributed platform" group.
> To post to this group, send email to aiko-p...@googlegroups.com.
> To unsubscribe from this group, send email to aiko-platfor...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/aiko-platform?hl=en
> .
>

Luke Weston

unread,
Mar 9, 2010, 6:13:33 PM3/9/10
to Aiko distributed platform
I have reset it many times, and it's still giving the above behaviour.

Sam Sabey

unread,
Mar 9, 2010, 7:25:53 PM3/9/10
to aiko-p...@googlegroups.com
Strange indeed.

I notice the last good data from your gateway was March 5th for both streams ending at about the same time.

I wonder what's up with the Arduino? Perhaps this could have had some #fail, have you had a peep at it's direct output?

Sam.

On 10/03/2010, at Wed 10, 10:13 AM, Luke Weston wrote:

> I have reset it many times, and it's still giving the above behaviour.
>

Andy Gelme

unread,
Mar 9, 2010, 7:45:35 PM3/9/10
to aiko-p...@googlegroups.com
hi Luke,

Luke Weston wrote:
> My gateway seems to be misbehaving. This is strange because I've just
> had it sending there unattended, and it has been fine, logging sensor
> data up onto Smart Energy Groups for quite a while previously, without
> maintenance. But now it does not seem to be sending meaningful sensor
> data - however, SEG still says that the device *is online*.

Can you confirm whether the SEG web-page is indicating that the "device"
is on-line ... or if it is indicating that the "gateway" is on-line ?

> I suspect that the gateway side of the system is fine but something
> might be wrong with the Arduino side.

After killing the "aiko_gateway.lua" process, you can edit the
configuration file to turn debugging on. See line 34 ...

debug = true

Then, try running "aiko_gateway.lua" from the command line and watch the
terminal output. You'll then see the Aiko-Gateway sending a
"send_event_boot" message and then a "send_event_heartbeat" message
(which should repeat every minute, depending upon your configuration).

That is the best way to determine how well the Aiko-Gateway is performing.

On a regular basis, you'll see the message ...

-- coroutine.resume(coroutine_serial):

That is where the Aiko-Gateway is waiting for Aiko-Node messages from
the serial port. If the Aiko-Node is functioning correctly, you should
see the various Aiko-Node heartbeat and sensor value messages.

If you don't see any messages from the Aiko-Node, then that confirms
what you've already found.

> When I telnet localhost:2000 (after killing the lua process) there are
> no sensor data expressions coming in on the ser2net socket - there is
> nothing.

Yes, telnet'ing to the Ser2Net port is also a good way to test the
Aiko-Node specifically.

Sam wrote:
> How long had it been running? Nearly 4 weeks? Have you rebooted the
> gateway? These is some fault in the Aiko Andy and I have found, we
> are investigating...

Sam is referring to a problem in the Aiko-Node software. And, mentions
rebooting the Gateway (I think he means power-cycling), as an easy way
to know when the Aiko-Node was last power-cycled.

It appears that Aiko-Node handlers may be failing (possibly removed from
the event queue) either around 24.85 days after the Aiko-Node starts
(about 2^31 milliseconds) or possibly after 2^31 times that the event
handler is invoked. Not sure exactly, the problem is still under
investigation.

Note: Just rebooting the Aiko-Gateway, without resetting or
power-cycling the Aiko-Node would not "fix" the problem.

Luke wrote:
> I have reset it many times, and it's still giving the above behaviour.

Resetting the Aiko-Gateway (via the button on the back) ... won't change
things.

Resetting the Aiko-Node definitely "fixes" the problem that Sam and I
have been chasing.

If you power-cycle the whole box (both Gateway and Node) and you are
still not getting messages from the Aiko-Node, then I suspect that
something more serious has occurred.

> Any ideas?

Try power-cycling the whole Aiko-Gateway / Node.
Does the Aiko-Node pin 13 LED blink afterwards ?

Try re-downloading the Aiko-Node firmware.

Check the serial connection between the router console and the Arduino.

Double check your Ser2Net configuration, is "/etc/ser2net.conf" okay ?
Correct baud rate ... 38,400 ?
Can telnet to port 2000 correctly ?

--
-O- cheers = /\ /\/ /) `/ =
--O -- http://www.geekscape.org --
OOO -- an...@geekscape.org -- http://twitter.com/geekscape --

Sam Sabey

unread,
Mar 9, 2010, 8:00:23 PM3/9/10
to aiko-p...@googlegroups.com
Hey Andy,

Nice recipie, Luke, if you follow all of this, it's likely you will manage to herd the #fail into a corner.

From SEG, it seems that the gateway was online, i.e. SEG getting messages with the last offline event at 10:40am this morning.

If the online indicator is showing in SEG, it means it's recently heard from the site or device. Whether it received any data, or stored data or not is another thing.

Sam.

Luke Weston

unread,
Mar 10, 2010, 2:53:03 AM3/10/10
to Aiko distributed platform
Hhmph. The arduino is dead - it can't be programmed. That seems
surprising.

Sam Sabey

unread,
Mar 10, 2010, 3:30:12 AM3/10/10
to aiko-p...@googlegroups.com
Hey Luke,

Pisser...which one haz #fail? The Duemilionove that got a new brain once upon a time or another?

Sam.

On 10/03/2010, at Wed 10, 6:53 PM, Luke Weston wrote:

> Hhmph. The arduino is dead - it can't be programmed. That seems
> surprising.
>

Andy Gelme

unread,
Mar 10, 2010, 3:30:41 AM3/10/10
to aiko-p...@googlegroups.com
hi Luke,

Luke Weston wrote:
> Hhmph. The arduino is dead - it can't be programmed. That seems
> surprising.
>

Sorry to hear that :(

Let's look into re-flashing the boot-loader.

Now there is even more incentive to sort out the Arduino-based programmer :)

Luke Weston

unread,
Mar 10, 2010, 5:51:37 AM3/10/10
to Aiko distributed platform
I mean the Arduino Pro inside my gateway router.

There are three options, in order of decreasing win:

i) Re-flash the AVR via ICSP
ii) Get a new Arduino Pro
iii) Swap the AVR chip - using a hot-air iron to remove the SMD chip -
probably not worth the trouble

Still... I've had a few mysterious failures of Arduino AVRs... and I'm
curious to find out if it's really hardware failure of the chip or
it's some kind of firmware corruption which can be fixed.

George Patterson

unread,
Mar 10, 2010, 8:07:00 AM3/10/10
to aiko-p...@googlegroups.com
On Wed, Mar 10, 2010 at 7:30 PM, Sam Sabey <sam....@esskware.com.au> wrote:
> Hey Luke,
>
> Pisser...which one haz #fail?  The Duemilionove that got a new brain once upon a time or another?
>
> Sam.
>
> On 10/03/2010, at Wed 10, 6:53 PM, Luke Weston wrote:
>
>> Hhmph. The arduino is dead - it can't be programmed. That seems
>> surprising.
>>

"Brrraaaaiiiiinnnns", anonymous zombie.

Sam Sabey

unread,
Mar 10, 2010, 5:35:28 PM3/10/10
to aiko-p...@googlegroups.com, aiko-p...@googlegroups.com
Luke's 'drino shall be known as 'Scarecrow'

...from @samotage on the go.

On 11/03/2010, at 12:07 AM, George Patterson
<george.p...@gmail.com> wrote:

Luke Hovington

unread,
Mar 10, 2010, 3:37:38 PM3/10/10
to aiko-p...@googlegroups.com

On 11/03/2010, at 12:07 AM, George Patterson
<george.p...@gmail.com> wrote:

Someone has being playing plants vs zombies..

Luke Weston

unread,
Mar 20, 2010, 11:09:19 AM3/20/10
to Aiko distributed platform
Hi everyone,

I'm currently setting up Dale's gateway, but I've got some more fail
here also.

The arduino works... if I plug in an FTDI board into the Arduino I can
see the serial expressions with sensor data coming in off the Arduino.
However, I can't seem to get that serial data stream coming into the
router.

All the hardware electronic connectivity seems good... I can see a
brief burst of sensor data coming down from the Arduino to the
router's RX line once every 15 seconds on the oscilloscope. (serial
line normally sits high at 5 V and goes low as the serial data burst
comes through, which I *think* is the normal behavior)

However, I can't seem to see that data stream inside OpenWRT via
ser2net (via telnet to localhost:2000 after killing the aiko gateway
process)

There is an entry for ser2net in init.d, so it is being executed on
boot, and /etc/ser2net.conf is set up with the appropriate baud rate,
38,400 baud, which matches the aiko node firmware.

Any ideas?

Cheers

Sam Sabey

unread,
Mar 20, 2010, 4:26:08 PM3/20/10
to aiko-p...@googlegroups.com, Aiko distributed platform
Hey Luke,

'normally' for me when thus happens my rx and tx pins are arse about.

The other thing to check closely is sn excess of solder on the wrt
header. Those pads are kinda small.

Also, with the ser2net.conf which device is it listening for?

...from @samotage on the go.

On 21/03/2010, at 2:09 AM, Luke Weston <reindeer...@gmail.com>
wrote:

> Hi everyone,

Andy Gelme

unread,
Mar 20, 2010, 6:08:48 PM3/20/10
to aiko-p...@googlegroups.com
hi Luke,

Luke Weston wrote:
> However, I can't seem to see that data stream inside OpenWRT via
> ser2net (via telnet to localhost:2000 after killing the aiko gateway
> process)
>
> There is an entry for ser2net in init.d, so it is being executed on
> boot, and /etc/ser2net.conf is set up with the appropriate baud rate,
> 38,400 baud, which matches the aiko node firmware.
>
> Any ideas?

Have you ensured that the shell process on the serial console has been
disabled ?

In /etc/inittab, you'll need to comment (or delete) this line ...
# tts/0::askfirst:/bin/ash --login

If this hasn't been done, then there will be a shell running on the
serial console, preventing the ser2net process from gaining access and
performing it's task.

Luke Weston

unread,
Mar 20, 2010, 11:25:44 PM3/20/10
to Aiko distributed platform
OK, the following is the contents of inittab.
The # character is the appropriate syntax for commenting out the line,
in this file, right?
Therefore it should already be commented out... I don't think I need
to edit it. Right?

::sysinit:/etc/init.d/rcS S boot
::shutdown:/etc/init.d/rcS K stop


# tts/0::askfirst:/bin/ash --login

ttyS0::askfirst:/bin/ash --login
tty1::askfirst:/bin/ash --login

I will check the hardware wiring and soldering to the serial port now.

Cheers,
Luke

Sam Sabey

unread,
Mar 21, 2010, 1:53:35 AM3/21/10
to aiko-p...@googlegroups.com
Hey Luke this is about right to me.

Sam.

Thomas Sprinkmeier

unread,
Mar 21, 2010, 12:50:29 AM3/21/10
to aiko-p...@googlegroups.com
On Sat, 2010-03-20 at 20:25 -0700, Luke Weston wrote:
> OK, the following is the contents of inittab.
> The # character is the appropriate syntax for commenting out the line,
> in this file, right?
> Therefore it should already be commented out... I don't think I need
> to edit it. Right?

$ fuser -av /dev/ttyS*
should tell you who is using the serial ports


Thomas

Luke Weston

unread,
Mar 24, 2010, 12:40:08 PM3/24/10
to Aiko distributed platform
I now have win on both my gateway and Dale's gateway, which is nice :)

andyg (geekscape)

unread,
Mar 25, 2010, 8:39:31 PM3/25/10
to Aiko distributed platform
hi Luke,

On Mar 25, 3:40 am, Luke Weston <reindeerfloti...@gmail.com> wrote:
> I now have win on both my gateway and Dale's gateway, which is nice :)

Great news !

Are you able to describe what you found to be wrong with each one ...
and the solution ?
Just so everyone else knows ... thanks !

cheers andyg (@geekscape)

Reply all
Reply to author
Forward
0 new messages