horizontal scaling and rapidly changing data - cache advice needed

[Mark Davies]

Hey all,

This question is probably not the average use case.

Currently I have one server with a C# net core console app that accepts incoming TCP IOT data. Each IOT device reports every 5 seconds when active and 60 mins when idle and there are 40,000 devices

My existing caching on 1 server is in local app memory, and just hits the DB on initial start-up, and pulls in the data and populates a key,value c# dictionary, and I just read and write to this key,value object....which is easy when you have 1 server.
I batch to the database every 30 secs and push data to the web UI in real time using web sockets (data has to been viewed by user in real time)
This is all on one server and works fine, but when the server/data centre goes down, i'm stuffed!

I now need to move to distributed model with many of these servers and HAProxy distributes data using leastconn model (long running TCP connections) to multiple servers
The issue I have is each new bit of information from the IOT device is compared against its "last record" and logic fires if there is a state change, or movement etc

As the IOT devices have sim cards, I have seen TCP connections get interrupted and the next time it connects it gets routed to different server and its "last record" on the next server is old and not the actual last record.
This causes obvious data consistency issues. I can live with the "last record" being 30 secs out of date, but not much more

I had started to look into caching with redis or memcached, but from what I read, you should use caching for infrequently changing data, whereas mine is changing every 5 secs

As I can I see I have the following options

1. configure haproxy to sticky sessions from IP address to certain server - not perfect and will imbalance loads as each IOT device can report on same IP/different port so may not actually solve the issue.
2. Use local in mem cache and remote cache, then broadcast recent data every 30 secs to the main cache, and pull from main cache every 30 secs on all servers.
   If i see an IOT device connect, and my "last record" in local cache is older than X seconds, I can go to the remote cache to see if the data is newer

Can anyone offer insight into how I might tackle this problem, what features or tech exists for this type of setup.

I currently have 40,000 IOT devices, but I need to design a system that will easily handle 500,000 + (all reporting every 5 seconds)

I can detect when a device creates a tcp connection and disconnects, and could wire up logic to update the central cache at these points, but some devices also use UDP, and also some just ungracefully disconnect, and it could be a few mins before I clean up that connection, by which point it could already be reporting to a different server with an out of date cached "last record"

Is redis the correct solution for this type of problem? or should I also be looking at other tech?
Can redis handle this amount of change/load too?
Can all server subscribe to remote cache and pull down data when items are updated?
any other insights or advice appreciated for this problem!

Thanks!

[Mehedi Hasan]

Hi Davies,

If I have not misunderstood, you need to deal with three things:

 

1. Dealing with large amount of incoming data

2. Comparing each record with the last record from same device

3. Scaling the overall solution.

 

I am apologizing beforehand to the community if my answers are not in line with Redis use cases.

 

Again I am not expert of IOT, but have some experience of handling large scale software. 

 

First let’s divide and try to rule. You can decouple your application servers from cache and db and scale those with a HA Proxy with least connection. Having an IP based routing will be a bad idea. As you mentioned, IOT devices have SIM cards and telecom providers do IP multiplexing. So at the same time many of your devices will have the same IP. So a generic least connection will be sufficient.

 

Next you need to read the data fast enough , possibly from a cache and then write it to a DB, persistent DB possibly. So in large scale, we normally prefer Kafka like broker in this kind of scenario. Your application may push the events directly to a kafka cluster. From their one consumer can update the Redis & another consumer can push those in a Cassandra. We preferred Cassandra as it gives huge write scalability. Also in this mode you don’t need to do batching from your application.

 

Next, every record needs to be compared with the previous record. There can be scenario, immediate last record is not yet available in Redis to query when you have received. I think you might need to forfeit true compliance to this principle in some cases and accept eventual consistency as you need to scale. Moreover, may be a zset can assist you in this use case having a max limit of 4B.

 

Then scale. In this mode, you can scale Redis, Kafka & Cassandra independently. So, I think 500k devices will not be a problem.

 

Finally, recently RedisLab introduced Redis Stream which can also be an option. I dont have firsthand experience of using it.

 

Post finally, there are kafka connectors for doing almost everything and most of the times they are just pluggable. And I am not confluent employee. :) 

 

Will be glad if you can find the solution you are looking for.

 

Mehedi

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/c4fd34cd-68d4-4dc6-9ff2-f37807df355cn%40googlegroups.com.