Problems with Scream

178 views
Skip to first unread message

Branden Christensen

unread,
Jul 13, 2014, 12:36:00 PM7/13/14
to Earthworm Community Forum
Hi all:


Buenas tardes.

We are having some problems with a Scream server. And I know that many people on this forum have a lot of experience with Scream. 

Ever since we added a few new stations to the server, it consumes all of the CPU almost instantly and the data for all stations, all channels are full of gaps. 

Scream is running on a server with plenty of resources:

OS: Ubuntu 10.04 LTS, 64-bit
CPU: 8 cores
RAM: 8 G

The data is fed from Scream to SeedLink via the SeedLink plugin for Scream. We disabled SeedLink to make sure that SeedLink was not the cause of the problem. As soon as we shut Scream down, the load drops back to base line levels. It seems pretty convincing to me that Scream is the problem but I am not sure how to pinpoint the cause and find a solution. 

Any insights/ recommendations you can offer are most appreciated. 

Have a great Sunday and enjoy the World Cup final!


Saludos, 


Branden Christensen
Director, OSOP
Sign up for the OSOP mailing list: http://www.osop.com.pa/about/osop-mailing-list/
LA SALSA VIVE...Gózala. Cali 2014. 

OSOP
--

Murray McGowan

unread,
Jul 13, 2014, 4:03:51 PM7/13/14
to earthwo...@googlegroups.com

Hi Branden,

 

There are no known limits in Scream to the number of stations that it can handle, so I am surprised that you have found such a dramatic change in behaviour from adding a few stations.

Do you know how many stations you had before, which worked fine, and how many you added that caused the issues?

Can you please say what connection types you have to your stations (serial/UDP/TCP etc) ?

 

Because Scream imposes very few limitations in its design, there are occasionally ways that you can find the OS limits (memory, thread count, file handles, etc) that can cause adverse behaviour. To try to understand if you have found one of these, I need to know the above information.

If you can email your network topology and scream.ini config file to sup...@guralp.com, we can look into this in detail.

 

Best regards,

 

Murray McGowan

- Author of Scream.

 

ps. I don't know of a "SeedLink plugin for Scream" ?

pps. I will be at the IASPEI meeting in Bogota. All are welcome to drop in for a chat.

Branden Christensen

unread,
Jul 13, 2014, 6:10:20 PM7/13/14
to Earthworm Community Forum
Estimado Murray:


Thank you for your response.  

 

Do you know how many stations you had before, which worked fine, and how many you added that caused the issues?

Previously there were 14 stations and then we added two more. We have since removed one and the problem continues. The other one we do not want to remove yet because it is installed in a remote location and works in push-mode. 

Can you please say what connection types you have to your stations (serial/UDP/TCP etc) ?

Most are TCP:

[PullServers]
TCP;201.197.242.59:1568=connecting  //Maravilla
TCP;201.201.188.54:1568=connecting  //La Lucha
TCP;201.207.94.6:1568=connecting  //Durika
TCP;201.202.246.146:1568=connecting  //V.Arenal
TCP;201.202.199.14:1568=connecting  //Cabo Blanco
UDP;186.176.79.232:1568=//V.Irazu
TCP;201.193.192.22:1568=//Turrubares
TCP;10.68.240.28:1568=connecting  //Tacares
TCP;201.237.238.202:1568=//Piro
TCP;163.178.107.47:1568=connecting  //Cima cima
TCP;201.207.44.126:1568=connecting  //Gandoca
TCP;10.68.240.27:1568=connecting  //San Ramon
TCP;163.178.105.107:1568=connecting  //San Jose
TCP;201.237.216.182:1568=connecting  //Puerto Jimenez

The new, fifteenth station does not appear in scream.ini because the data from this station are pushed to the Scream server. 

 

ps. I don't know of a "SeedLink plugin for Scream" ?

Yes, it has existed for many years. But maybe you read my words and think SL --> Scream when what I mean to say by "SeedLink plugin for Scream" is Scream --> SL. 

When viewing all 15 stations with the Scream real-time wave view gui, you notice a latency in the data retrieval and the stations lurch forward. So it appears that the Scream server is not receiving data in real-time for all channels/ stations due to the high cpu load. But all data packets do eventually arrive. 

You can find the scream.ini attached.

The log files are full of these "Block sequence error" messages:
G $C0 $0400  64260 PTJ1Z2 07/13/14 09:33:24 PM 100   2   250        1565:       1565 $0000:$0000 Block sequence error. Expected : $C2

I have also attached the log files. 


Branden
-- 
 
scream.ini
ScreamLogs.zip

Murray McGowan

unread,
Jul 14, 2014, 7:41:37 AM7/14/14
to earthwo...@googlegroups.com

Branden,

 

I think I see the problem here. If you look in the scream.ini file, the [NetServers] section is very large, and probably growing continuously.

This section is used internally by Scream to keep track of the sequence numbers from the various data sources that it receives data from. A data source is identified by the IP:Port combination.

 

You say "The new, fifteenth station does not appear in scream.ini because the data from this station are pushed to the Scream server."

I think this may be where the problem is. I can see that there are several IP addresses in the same range, and a large number of ports listed for each IP address. I'm guessing that this station is behind a NAT firewall, that is on a dynamic IP address? So, every time the IP address changes, your Scream "sees" this as a new server. Equally, each time the NAT firewall maps the data to a different port, Scream sees this as yet another client.

Each time Scream receives a network packet, it must look up the expected packet number from this list, which is now over 3500 entries.

 

To run a connection in this configuration, you need to change a few things.

1) use a hostname for the data source (e.g. dyndns)

2) Set Scream to pull from this hostname

3) use a port forward on the firewall at the source to ensure the port number is fixed.

 

To clear out this list of sequence numbers, you should

1) stop Scream

2) edit the INI file, and delete the entire [NetServers] section

3) re-start Scream.

If you don't fix the source of the problem, however, the list will grow again.

 

Alternatively, we do supply dedicated Network Appliance Modules (NAMs) which are designed for just the sort of network concentrator functionality that you are wanting, and can work with EAM-enabled equipment that is behind firewalls. This would replace your Scream PC, and offers more options for the data feed into Earthworm. It has seedlink support, and even an export_generic implementation, so you can go straight into EW. You can still use Scream for monitoring and SoH functions, as the NAM would act as a Scream server as well.

 

Murray.

Branden Christensen

unread,
Jul 14, 2014, 7:45:55 AM7/14/14
to Earthworm Community Forum
Murray:


Thank you for the great, detailed advice. I will give this a go and will be in touch if this works out. 


Kind Regards, 


Branden Christensen
Director, OSOP
Sign up for the OSOP mailing list: http://www.osop.com.pa/about/osop-mailing-list/
LA SALSA VIVE...Gózala. Cali 2014. 

OSOP
--

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Branden Christensen

unread,
Jul 14, 2014, 9:16:30 PM7/14/14
to Earthworm Community Forum
Murray:


Worked like a charm. Thanks! What a relief!!!


Branden
--
Reply all
Reply to author
Forward
0 new messages