memory use is unreasonable

[eastein]

The amount of data present in the markov chain is 2.6 MB as of now in a running instance - in the disk stored mode. The bot is using 164MB of RAM. This needs to be addressed. I could potentially try a few approaches:

Already did (helped a fairly large amount:)

1. use slots to somewhat reduce the in memory use

Additional steps:

1. Use BTree from zope to store the indexes rather than using dict... https://pypi.python.org/pypi/BTrees
2. analyze what exactly is using up the RAM and try to reduce it while remaining pure python
3. Rewrite the markov chain logic in c

[eastein]

Maybe http://pypy.org/ will help.

[eastein]

Using cffi might improve things as well. http://morepypy.blogspot.com/2015/05/cffi-10-beta-1.html