Node.js usage.

[trusktr]

Hi, I did some research, looking for an inline Worker+Blob API library. Basically, the three clear options on npmjs are operative, threadless, and paralleljs.

paralleljs seems like something that would be built on top of threadless or operative, so I dropped that out of the picture, looking just for the basic inline worker implementation.

So between operative and threadless, operative has the cleanest API. threadless is more like a C/Java programmer approach, and the required err argument for the callbacks can get annoying.

That being said, threadless shines because it works in both server-side nodejs and client-side browser.

Ultimately I'd like a single API across both environments. threadless does it, but operative's API is so nice.

Would you be willing to adopt the workerjs library that threadless is depending on to make operative work everywhere?

I feel that multi-threading will be necessary in the upcoming years as the web evolves into more of a 3D world, not just simple documents. Libraries like http://famo.us, for example, aren't using workers yet but I think it will become necessity for them do start using them in the new age of 3D web. Three.js is of course using workers already. It makes sense for highly performant applications.

[padolsey]

Hi @trusktr! Thanks for opening this issue. Node.js support was requested before, but at the time I wasn't sure whether it should be just another degraded variation vs. utilizing child processes / VMs, so opted to leave it out. I also couldn't really envisage a usage for Operative in both client and server envs. But I guess it could be worth having a single codebase that can easily be run in both.

I've been experimenting a bit with this today, and shall continue to do so. I'll update this issue when I have a workable branch / some progress.

So far my only question is around dependencies. Operative allows arbitrary remote dependencies to be passed in, ..

// Create interface to call lodash methods inside a worker:
var lodashWorker = operative(function(method, args, cb) {
    cb(
        _[method].apply(_, args)
    );
}, [
    'http://cdnjs.cloudflare.com/ajax/libs/lodash.js/1.3.1/lodash.min.js'
]);

I'm not sure how this would ideally behave in the Node.js env... Maybe it can just be an interface to local require calls? What do you think?

[padolsey]

Also, re: the workerjs lib, I think I'd rather just interface directly with child_process. Fortunately it all looks quite simple to do.

[trusktr]

For sure. But yeah, the benefit is having a single API on both sides. If it's child processes in nodejs, all the same from outside the API. And yeah, a require interface might be it for nodejs too, but still it'd be all the same outside the API. I don't think I'll be using URL dependencies like though. I'm primarily interested in handling my own dependencies, then using inline workers.

To give a worker a dependency, are you injecting the whole code of the dependency into the function?

[padolsey]

On the client-side, dependencies would either be a <script> in the degraded iframe or would be loaded via importScripts inside a worker (and so would end up exposing themselves globally).

I think it's fair to simply disable this functionality while in a Node env, since you can freely require() inside of your operative anyway, i.e.

var uniq = operative(function(array, cb) {
  var _ = require('lodash');
  cb( _.uniq(array) );
});

Does that look okay?

[trusktr]

Yeah, that looks nice!

[padolsey]

Worth a mention: Passing data to/from child processes in Node seems very slow (since it's just JSON-serialization).

[trusktr]

Yeah. :(

[padolsey]

Given that transferring data is such a huge bother (and so very slow), right now, it doesn't seem worth bringing node.js support to operative. I'm also not too happy about the dependency-loading inconsistency implicit in such support. Gonna close this issue for now.

[trusktr]

@padolsey For the Node.js, you could use an existing thread library (server-side) or just spawn children, but then wrap the API in your operative library so it exposes the same API as for the browser. The advantage would be that on the server side you can pass objects, not just JSON.

[padolsey]

Yeh, I do agree that it'd be nice to have a consistent API present on the server as well as the client. I'll re-open this issue for now... But I don't currently have time to implement it fully (and am still not sure what to do about dependency loading, as mentioned).

[StreetStrider]

Node's child_process has special form of spawn called fork. Not sure when it was appeared.
Here's two things about it:

1. It establish «communication channel», child will have process.send & process.on('message') and parent can child.send and child.on('message') (API is pretty symmetric).
2. fork accepts modulePath, not a function. Not sure is it good or bad for operative but it looks similar to Worker which accepts path too.

Here's might be the same perfomance issues, since communication is a JSON too.

I don't see any issue with dependencies, since node's child processes spawns/forks will have appropriate require. Maybe I don't understand the real issue here.