QBoard » Artificial Intelligence & ML » AI and ML - Python » Combining node.js and Python

Combining node.js and Python

  • Node.js is a perfect match for our web project, but there are few computational tasks for which we would prefer Python. We also already have a Python code for them. We are highly concerned about speed, what is the most elegant way how to call a Python "worker" from node.js in an asynchronous non-blocking way?
      September 24, 2021 12:39 PM IST
    0
  • This sounds like a scenario where zeroMQ would be a good fit. It's a messaging framework that's similar to using TCP or Unix sockets, but it's much more robust (http://zguide.zeromq.org/py:all)

    There's a library that uses zeroMQ to provide a RPC framework that works pretty well. It's called zeroRPC (http://www.zerorpc.io/). Here's the hello world.

    Python "Hello x" server:

    import zerorpc
    
    class HelloRPC(object):
        '''pass the method a name, it replies "Hello name!"'''
        def hello(self, name):
            return "Hello, {0}!".format(name)
    
    def main():
        s = zerorpc.Server(HelloRPC())
        s.bind("tcp://*:4242")
        s.run()
    
    if __name__ == "__main__" : main()

     

    And the node.js client:

    var zerorpc = require("zerorpc");
    
    var client = new zerorpc.Client();
    client.connect("tcp://127.0.0.1:4242");
    //calls the method on the python object
    client.invoke("hello", "World", function(error, reply, streaming) {
        if(error){
            console.log("ERROR: ", error);
        }
        console.log(reply);
    });

     

    Or vice-versa, node.js server:

    var zerorpc = require("zerorpc");
    
    var server = new zerorpc.Server({
        hello: function(name, reply) {
            reply(null, "Hello, " + name, false);
        }
    });
    
    server.bind("tcp://0.0.0.0:4242");

     

    And the python client

    import zerorpc, sys
    
    c = zerorpc.Client()
    c.connect("tcp://127.0.0.1:4242")
    name = sys.argv[1] if len(sys.argv) > 1 else "dude"
    print c.hello(name)
      December 1, 2021 2:54 PM IST
    0
  • This sounds like a scenario where zeroMQ would be a good fit. It's a messaging framework that's similar to using TCP or Unix sockets, but it's much more robust (http://zguide.zeromq.org/py:all)

    There's a library that uses zeroMQ to provide a RPC framework that works pretty well. It's called zeroRPC (http://www.zerorpc.io/). Here's the hello world.

    Python "Hello x" server:

    import zerorpc
    
    class HelloRPC(object):
        '''pass the method a name, it replies "Hello name!"'''
        def hello(self, name):
            return "Hello, {0}!".format(name)
    
    def main():
        s = zerorpc.Server(HelloRPC())
        s.bind("tcp://*:4242")
        s.run()
    
    if __name__ == "__main__" : main()

     

    And the node.js client:

    var zerorpc = require("zerorpc");
    
    var client = new zerorpc.Client();
    client.connect("tcp://127.0.0.1:4242");
    //calls the method on the python object
    client.invoke("hello", "World", function(error, reply, streaming) {
        if(error){
            console.log("ERROR: ", error);
        }
        console.log(reply);
    });

     

    Or vice-versa, node.js server:

    var zerorpc = require("zerorpc");
    
    var server = new zerorpc.Server({
        hello: function(name, reply) {
            reply(null, "Hello, " + name, false);
        }
    });
    
    server.bind("tcp://0.0.0.0:4242");

     

    And the python client

    import zerorpc, sys
    
    c = zerorpc.Client()
    c.connect("tcp://127.0.0.1:4242")
    name = sys.argv[1] if len(sys.argv) > 1 else "dude"
    print c.hello(name)
      September 25, 2021 2:23 PM IST
    0
  • I've had a lot of success using thoonk.js along with thoonk.py. Thoonk leverages Redis (in-memory key-value store) to give you feed (think publish/subscribe), queue and job patterns for communication.
    Why is this better than unix sockets or direct tcp sockets? Overall performance may be decreased a little, however Thoonk provides a really simple API that simplifies having to manually deal with a socket. Thoonk also helps make it really trivial to implement a distributed computing model that allows you to scale your python workers to increase performance, since you just spin up new instances of your python workers and connect them to the same redis server
      September 27, 2021 5:00 PM IST
    0
  • Update 2019

    There are several ways to achieve this and here is the list in increasing order of complexity

    1. Python Shell, you will write streams to the python console and it will write back to you
    2. Redis Pub Sub, you can have a channel listening in Python while your node js publisher pushes data
    3. Websocket connection where Node acts as the client and Python acts as the server or vice-versa
    4. API connection with Express/Flask/Tornado etc working separately with an API endpoint exposed for the other to query

    Approach 1 Python Shell Simplest approach

    source.js file

    const ps = require('python-shell')
    // very important to add -u option since our python script runs infinitely
    var options = {
        pythonPath: '/Users/zup/.local/share/virtualenvs/python_shell_test-TJN5lQez/bin/python',
        pythonOptions: ['-u'], // get print results in real-time
        // make sure you use an absolute path for scriptPath
        scriptPath: "./subscriber/",
        // args: ['value1', 'value2', 'value3'],
        mode: 'json'
    };
    
    const shell = new ps.PythonShell("destination.py", options);
    
    function generateArray() {
        const list = []
        for (let i = 0; i < 1000; i++) {
            list.push(Math.random() * 1000)
        }
        return list
    }
    
    setInterval(() => {
        shell.send(generateArray())
    }, 1000);
    
    shell.on("message", message => {
        console.log(message);
    })

     

    destination.py file

    import datetime
    import sys
    import time
    import numpy
    import talib
    import timeit
    import json
    import logging
    logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
    
    size = 1000
    p = 100
    o = numpy.random.random(size)
    h = numpy.random.random(size)
    l = numpy.random.random(size)
    c = numpy.random.random(size)
    v = numpy.random.random(size)
    
    def get_indicators(values):
        # Return the RSI of the values sent from node.js
        numpy_values = numpy.array(values, dtype=numpy.double) 
        return talib.func.RSI(numpy_values, 14)
    
    for line in sys.stdin:
        l = json.loads(line)
        print(get_indicators(l))
        # Without this step the output may not be immediately available in node
        sys.stdout.flush()

    Notes: Make a folder called subscriber which is at the same level as source.js file and put destination.py inside it. Dont forget to change your virtualenv environment

     
      September 28, 2021 1:47 PM IST
    0