Native python converters for HBaseResult and others

[gorlins]

@GenTang would you be interested in pulling and updating on spark-packages.org? I still haven't seen something as simple for python wrappers, though I am new to this tech stack, so I would think this is widely useful.

I have added a few options similar to yours that work as JSON arrays (#3) as well as getting raw list/dict support in Python without passing through strings.

I updated build to hbase 1.0 as well, unclear to me best protocol here.

I could use a good review since this is the only Scala work I have completed, so feel free to comment on syntax etc. Any suggestions for testing also welcome.

[GenTang]

Hi, thanks for your contribution.
However, I don't think that the converter of [Any, java.util.List[java.util.Map[String, String]] nor [Any, java.util.Map[String, String]] would really work. Currently, in PySpark only (String, String) or (Array[Byte], Array[Byte]) are supported.

[gorlins]

Can you clarify or point me to some details about this? They were working
for me interactively against cdh 5 hbase 1.0 and pyspark 1.3 local mode on
a mac.

Did you mean [Any, String] and [Any, Array[Byte]]? I'm not clear exactly
what (String, String) is referring to.
On Feb 1, 2016 2:32 AM, "Gen TANG" [email protected] wrote:

Hi, thanks for your contribution.
However, I don't think that the converter of [Any,
java.util.List[java.util.Map[String, String]] nor [Any,
java.util.Map[String, String]] would really work. Currently, in PySpark
only (String, String) or (Array[Byte], Array[Byte]) are supported.

—
Reply to this email directly or view it on GitHub
#4 (comment).

[GenTang]

Yeah, it could work in local mode without any doubt, as the serialization
of data doesn't really happen in local mode.
In the cluster mode, python driver will pass the code to jvm driver. And
then jvm driver will pass code to jvm executor. After that, for some basic
operation, jvm executors will pass data and code to python executor. In the
python side, it needs to deserialize data from jvm. And only RDD[String,
String] or RDD[Array[Byte], Array[Byte]]
are supported.
Therefore, in cluster mode, your python converter could cause serialization
problem.

Hope it could be helpful

Cheers
Gen

On Mon, Feb 1, 2016 at 8:45 PM, Scott Gorlin [email protected]
wrote:

Can you clarify or point me to some details about this? They were working
for me interactively against cdh 5 hbase 1.0 and pyspark 1.3 local mode on
a mac.

Did you mean [Any, String] and [Any, Array[Byte]]? I'm not clear exactly
what (String, String) is referring to.
On Feb 1, 2016 2:32 AM, "Gen TANG" [email protected] wrote:

Hi, thanks for your contribution.
However, I don't think that the converter of [Any,
java.util.List[java.util.Map[String, String]] nor [Any,
java.util.Map[String, String]] would really work. Currently, in PySpark
only (String, String) or (Array[Byte], Array[Byte]) are supported.

—
Reply to this email directly or view it on GitHub
#4 (comment).

—
Reply to this email directly or view it on GitHub
#4 (comment).

[gorlins]

Would you mind pointing me towards documentation as such? I'm not finding anything about that online...

It doesn't quite seem right to me since I did get a number of py4j serialization errors in the code while developing, so something was going through the JVM (and failing until I got the types right). I did test on a real HBase installation, only Spark was local mode, so java -> python definitely did work. I am hoping to deploy a Spark cluster soon, so will test there as well.

I also don't quite understand what you mean about only those two RDD types being supported, lots of examples use different schemas (ie word count output would be RDD[String, long] would it not? are you just referring to the immediate output from HBase?)

Thanks for your continued help!

[GenTang]

I am sorry that I could not find the document that mentioned that. You
know, spark is not very well documented.
However, I found some codes which can explain that your code works.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala#L194

In fact, Python RDDs are stored in Spark as RDD[Array[Byte]] of serialized
Python objects.
http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/josh-rosen-amp-camp-2012-spark-python-api-final.pdf

Therefore your code is strongly depends on toString() methods that
implemented by the objects that you use. This way is not very safe. That's
why we choose to transfer hbase result to string by ourself.(it likes that
we "implement" toString() method)

Maybe the relative document is here:
https://github.com/amplab-extras/SparkR-pkg/blob/master/pkg/R/utils.R#L33
as R use the same architecture as python in spark.

Hope that it could be helpful

Cheers
Gen

On Wed, Feb 3, 2016 at 11:28 AM, Scott Gorlin [email protected]
wrote:

Would you mind pointing me towards documentation as such? I'm not finding
anything about that online...

It doesn't quite seem right to me since I did get a number of py4j
serialization errors in the code while developing, so something was going
through the JVM (and failing until I got the types right). I did test on a
real HBase installation, only Spark was local mode, so java -> python
definitely did work. I am hoping to deploy a Spark cluster soon, so will
test there as well.

I also don't quite understand what you mean about only those two RDD types
being supported, lots of examples use different schemas (ie word count
output would be RDD[String, long] would it not? are you just referring to
the immediate output from HBase?)

Thanks for your continued help!

—
Reply to this email directly or view it on GitHub
#4 (comment).

[gorlins]

Thanks so much! I'll review those

On Wed, Feb 3, 2016 at 8:16 AM, Gen TANG [email protected] wrote:

I am sorry that I could not find the document that mentioned that. You
know, spark is not very well documented.
However, I found some codes which can explain that your code works.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala#L194

In fact, Python RDDs are stored in Spark as RDD[Array[Byte]] of serialized
Python objects.

http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/josh-rosen-amp-camp-2012-spark-python-api-final.pdf

Therefore your code is strongly depends on toString() methods that
implemented by the objects that you use. This way is not very safe. That's
why we choose to transfer hbase result to string by ourself.(it likes that
we "implement" toString() method)

Maybe the relative document is here:
https://github.com/amplab-extras/SparkR-pkg/blob/master/pkg/R/utils.R#L33
as R use the same architecture as python in spark.

Hope that it could be helpful

Cheers
Gen

On Wed, Feb 3, 2016 at 11:28 AM, Scott Gorlin [email protected]
wrote:

Would you mind pointing me towards documentation as such? I'm not finding
anything about that online...

It doesn't quite seem right to me since I did get a number of py4j
serialization errors in the code while developing, so something was going
through the JVM (and failing until I got the types right). I did test on
a
real HBase installation, only Spark was local mode, so java -> python
definitely did work. I am hoping to deploy a Spark cluster soon, so will
test there as well.

I also don't quite understand what you mean about only those two RDD
types
being supported, lots of examples use different schemas (ie word count
output would be RDD[String, long] would it not? are you just referring to
the immediate output from HBase?)

Thanks for your continued help!

—
Reply to this email directly or view it on GitHub
#4 (comment).

—
Reply to this email directly or view it on GitHub
#4 (comment).