is possible scale out node-red horizontally on cluster of nodes? not find documentation on same. scenario handle millions of events per second , process them in real time using node-red.
i posted question on google groups node-red forum (https://groups.google.com/forum/#!topic/node-red/nx1wwqbelbi) , got interesting answers. jotting down various options below.
if input on http, can use of standard load-balancing techniques load balance requests on cluster of nodes running same node-red flow - e.g. 1 can use haproxy, nginx, etc. important note since running same flow on many nodes, cannot store state in context variables. have store state in external service such redis.
if ingesting on mqtt, have multiple options: option a: let each flow listen different topic. can have different gateways publish different topics on mqtt broker - e.g. flow instance 1 subscribes device/a/# node-red instance 2 subscribe device/b/# , on.
option b: mqtt brokers support concept of 'shared subscription' (hivemq) equivalent point-to-point messaging - i.e. each consumer in subsciption group gets message , broker load-balances using round-robin. explanation on how enable using hivemq given here - http://www.hivemq.com/blog/mqtt-client-load-balancing-with-shared-subscriptions/. thing hivemq support load-balancing consumers there no change required in consumer code. can continue using mqtt consumer - topic url change :)
option c: put simple node-red flow message ingestion reads payload , makes http request cluster of load-balanced node-red flows (similar option 1)
option d: extension option c , entails creating buffer between message ingestion , message processing using apache kafka. ingest message devices on mqtt , extract payload , post on kafka topic. kafka can support message-queue paradigm using concept of consumer groups. can have multiple node-red flow instances subscribing kafka topic using same consumer group. option makes sense, if message broker not support load-balancing consumers.
have posted blog post links here - http://www.narendranaidu.com/2016/07/scaling-node-red-horizontally-for-high.html
Comments
Post a Comment