integrate with quantopian/zipline?

[fawce]

Hi @yinhm

I ran across datafeed, and I was wondering if it could be integrated with our project: https://github.com/quantopian/zipline

what do you think?

thanks,
fawce

[yinhm]

Hi @fawce ,

Just did a quick view of your project: zipline.

It looks good, better than my home made backtester, :-)

I'd like to see it could be used together, and there is no license issue(they are all Apache 2.0).

The question is: How do you want it be integrated?

--Regards
yinhm

[trbck]

I use both of your projects and get the data like this at the moment:

client = Client()
symbol = 'ADS.DE'
y = client.get_day(symbol, 9999)

index = np.array([datetime.date.fromtimestamp(v) for v in y['time']], dtype=object)
index = pd.DatetimeIndex(index, name="Date")

df = pd.DataFrame.from_records(y, index=index, exclude=['time'])
#df.index = df.index.tz_localize(pytz.utc)

data = pd.DataFrame(df["close"], columns=[symbol],index=df.index)

It is just a quick solution but I really would like to see datafeed being integrated as a datasource.

[fawce]

fyi, I made a parallel issue on zipline: quantopian/zipline#14
I also mentioned it to @ehebert and @twiecki - seems promising based on @trbck's example code.

[yinhm]

Great, i'll looking into it later.

I may integrate @trbck's code as a helper, if that is ok. @trbck , can I have your permission to do so?

--Regards

[trbck]

@yinhm yes, of course. @fawce right now I send the data dataframe to the handle_data() function structured with the close price of each symbol per column. For further analysis with ta-lib I have to get a separate dataframe from @yinhm's (excellent) datafeed solution with OHLC data. This seems not very efficient. I would appreciate your ideas and suggestions for improvement.

[fawce]

Internally, zipline makes all datasources into generators. Zipline also supports OHLC, it just also needs a price field. Even the pandas dataframe source wraps the dataframe in code to produce a dataframe. This would make integration as simple as writing a datasource that knows how to instantiate a datafeed client, and iterate over the client's results, format it into a zipline.ndict with '['datetime','sid','volume','high','low','close','open']' properties.

I made a little prototype of a csv datasource that illustrates the point. It isn't ready to be in the zipline codebase, so I made a gist that you can checkout: https://gist.github.com/4057021

If you were to fork zipline, you could add datafeed to requirements.txt to ensure it is included as a dependency, and then implement a datasource that wraps datafeed. Would you be up for doing that?

@trbck you want to use a BatchTransform inside your algorithm. Then you can fetch data once, stream it through the zipline, and it will be aggregated and sent to your transform as a dataframe, which you can in turn feed to ta-lib. We would love a PR with ta-lib powered batch transforms. @twiecki did an example at PyData, which is in cell In [16] in this ipython notebook: http://nbviewer.ipython.org/3962843/

[yinhm]

@fawce

Finally got some time to wrote an adapter.

Checkout the draft version of zipline adapter:
https://gist.github.com/4084683

Some questions still remain:

• What is a sid? and sid_range? Is it a symbol name, and why it should be a integer?
• For the draft version, the adapter support get_day(OHLCs) api only. This related to the design choice: should we pass the client instance to the adapter(the current choice) or should we just pass raw data to it then let the adapter to transform it to zipline specified data(iterator).

[ehebert]

@yinhm a sid does not need to be an integer, it just needs to be a unique identifier for the stock among the data set.
This can be the stock symbol, and for the datafeed adapter probably should be the stock symbol.

(For some color on why we use an integer internally in the Quantopian app is that stock symbols can be renamed, so we use the integer to uniquely identify the stock across renames.)

[ehebert]

@yinhm I think the adapter should do the transform to the zipline dictionary format.

Working on a base DataSource object that should aid in that. I'll ping back here when it's ready.

I'm not sure that passing the client to the adapter precludes the adapter being passed in the client object. Though I could be understanding incorrectly.

[yinhm]

@ehebert

I have tried leave sid as a string, but it fails when doing assert_trade_protocol, so I thought that casting to integer was mandatory.

For "passing the client to the adapter", what i trying to say is datafeed client support more apis than get_day, for example, we have: get_reports, get_min, etc. Only get_day was supported in the current draft adapter.

[ehebert]

@yinhm I apologize for the confusion from the assert_trade_protocol. That method is used to assert the form of events used in our internal code (which has the integer sid mentioned above). This is evidence of some bleed between the two projects.
I'll move the assert_trade_protocol method out of zipline since it's actually unused anywhere in zipline.

[ehebert]

@yinhm ah I see what you're saying now. I think passing the client in makes the most sense here, I mentally model it as installing in to the class the "heart" of the generator that emits the data.

[yinhm]

@ehebert

Great, I'll just remove assert_trage_protocol then, no problem.

Agree with passing the client to the adapter, thought I may make it more flexible later.

[ehebert]

@yinhm, we just pushed a base class for data sources to zipline.

The class is DataSource in: https://github.com/quantopian/zipline/blob/master/zipline/sources/data_source.py

For an example usage, there is: https://github.com/quantopian/zipline/blob/master/zipline/sources/data_frame_source.py

This class is in it's nascent form, so feedback is very much welcome!