Skip to content

Add naive semiparallel processor#19

Open
txus wants to merge 2 commits into
masterfrom
improvement/semiparallel
Open

Add naive semiparallel processor#19
txus wants to merge 2 commits into
masterfrom
improvement/semiparallel

Conversation

@txus

@txus txus commented Dec 4, 2012

Copy link
Copy Markdown
Collaborator

A naive semiparallel processor.

It chunks consecutive GET requests and executes each chunk in a map of threads.

To use it, simply pass sequential: false in the request parameters.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have an explicit option -- params["strategy"] or whatever: "sequential", "semiparallel", "parallel", etc.

@arsduo

arsduo commented Jan 14, 2013

Copy link
Copy Markdown
Owner

After writing a spec for the Batch API, I'm thinking this should be the default. There's really no need for fully sequential. I'll poke around with this later this week.

@lukelex

lukelex commented Oct 4, 2014

Copy link
Copy Markdown

Any progress on this?

@dvgica

dvgica commented Nov 11, 2014

Copy link
Copy Markdown

I would love to see some movement on this as well.

@arsduo

arsduo commented Apr 4, 2015

Copy link
Copy Markdown
Owner

@lukelex @DWvanGeest I haven't worked on this project in a while, but hope to catch it up this weekend. Did you ever test @txus' pull request? Does it work for you? If so, I'll merge it in.

@arsduo

arsduo commented Apr 4, 2015

Copy link
Copy Markdown
Owner

Actually, there's a later pull request that does full parallel processing with Celluloid: #33 -- if you're still interested in parallel processing and could try that out and see if that works for you, that's probably the direction this project should go.

@jeremyhaile

Copy link
Copy Markdown

So should this commit be closed in favor of #33?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants