Kafka Source by tdas · Pull Request #24 · marmbrus/spark

tdas · 2016-01-07T04:06:14Z

No description provided.

AmplabJenkins · 2016-01-07T04:11:43Z

Merged build finished. Test FAILed.

AmplabJenkins · 2016-01-07T04:11:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/17/
Test FAILed.

marmbrus · 2016-01-07T07:46:15Z

maybe try this test case:

(1 to 50).foreach { i => test(s"basic receiving #$i") { val topic = s"topic$i" testUtils.createTopic(topic) val kafkaSource = KafkaSource(Set(topic), kafkaParams) val mapped = kafkaSource .toDS() .map(kv => new String(kv._2).toInt + 1) testStream(mapped)( AddKafkaData(kafkaSource, topic, 1, 2, 3), CheckAnswer(2, 3, 4), StopStream, DropBatches(1), StartStream, CheckAnswer(2, 3, 4), StopStream, AddKafkaData(kafkaSource, topic, 4, 5, 6), StartStream, CheckAnswer(2, 3, 4, 5, 6, 7), AddKafkaData(kafkaSource, topic, 7), CheckAnswer(2, 3, 4, 5, 6, 7, 8)) } }

Oh yeah, I am going to add more tests! That's why WIP. Going to add multi-topic tests, etc.

Of course, I just tried that specific test and it uncovered several concurrency issues.

…f-kafka

AmplabJenkins · 2016-01-08T09:46:13Z

Merged build finished. Test FAILed.

AmplabJenkins · 2016-01-08T09:46:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/27/
Test FAILed.

AmplabJenkins · 2016-01-08T21:53:36Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-08T21:53:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/28/
Test PASSed.

tdas · 2016-01-09T04:48:47Z

@marmbrus This is a bug here. Basically, [0, 0] < [0, 1] is correct, but the earlier code would give exception as signs = [0, -1]

tdas · 2016-01-09T04:50:04Z

I want to add the randomized testing as well, but I dont want this to get blocked by that. Please start reviewing this.

AmplabJenkins · 2016-01-09T04:51:06Z

Merged build finished. Test FAILed.

AmplabJenkins · 2016-01-09T04:51:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/33/
Test FAILed.

AmplabJenkins · 2016-01-09T04:56:01Z

Merged build finished. Test FAILed.

AmplabJenkins · 2016-01-09T04:56:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/34/
Test FAILed.

…f-kafka

tdas · 2016-01-11T20:12:25Z

@marmbrus Ping for review.

Also Jenkins does not seem to be testing this. What do I do to force it?

tdas · 2016-01-11T20:12:32Z

Jenkins, test this please.

AmplabJenkins · 2016-01-11T20:14:53Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-11T20:14:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/38/
Test PASSed.

marmbrus · 2016-01-11T20:17:15Z

Spark/Kafka ordering are inconsistent with the file above.

marmbrus · 2016-01-11T20:31:19Z

wrap the trailing */

AmplabJenkins · 2016-01-11T20:32:52Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-11T20:32:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/40/
Test PASSed.

marmbrus · 2016-01-11T20:32:56Z

I think this should be equals as well. Note that means we need to override hashcode too.

AmplabJenkins · 2016-01-11T22:08:49Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-11T22:08:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/41/
Test PASSed.

AmplabJenkins · 2016-01-11T23:40:05Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-11T23:40:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/42/
Test PASSed.

AmplabJenkins · 2016-01-12T02:19:14Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-12T02:19:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/43/
Test PASSed.

marmbrus · 2016-01-12T19:23:59Z

I'm not sure that its valid to override equals without also overriding hashCode

AmplabJenkins · 2016-01-12T21:44:06Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-12T21:44:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/44/
Test PASSed.

AmplabJenkins · 2016-01-12T22:31:27Z

Merged build finished. Test PASSed.

AmplabJenkins · 2016-01-12T22:31:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/45/
Test PASSed.

Kafka Source

Ported KafkaSource from different branch

3d61f82

tdas changed the title ~~Kafka Source~~ [WIP] Kafka Source Jan 7, 2016

marmbrus reviewed Jan 7, 2016
View reviewed changes

tdas added 2 commits January 7, 2016 23:20

Merge remote-tracking branch 'marmbrus/streaming-df' into streaming-d…

89b1ef0

…f-kafka

Fixed race condition, and updated tests

5fa1a20

Added license

426cd7a

Fixed offset comparison bugs

4079217

tdas reviewed Jan 9, 2016
View reviewed changes

formatting change

2990039

tdas added 4 commits January 11, 2016 11:33

Fix style

c4e2849

Merge remote-tracking branch 'marmbrus/streaming-df' into streaming-d…

3cbc73f

…f-kafka

Added more docs

2767a06

More docs

97c4aa5

marmbrus reviewed Jan 11, 2016
View reviewed changes

Addressed comments form Michael

35dea1b

Override equals instead of ==

b6a081b

Addressed one more comment

d747866

marmbrus reviewed Jan 12, 2016
View reviewed changes

Added stress test

81cda63

Removed equals

bcfd6a9

tdas changed the title ~~[WIP] Kafka Source~~ Kafka Source Jan 12, 2016

marmbrus added a commit that referenced this pull request Jan 12, 2016

Merge pull request #24 from tdas/streaming-df-kafka

d0807b1

Kafka Source

marmbrus merged commit d0807b1 into marmbrus:streaming-df Jan 12, 2016

Conversation

tdas commented Jan 7, 2016

Uh oh!

AmplabJenkins commented Jan 7, 2016

Uh oh!

AmplabJenkins commented Jan 7, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jan 8, 2016

Uh oh!

AmplabJenkins commented Jan 8, 2016

Uh oh!

AmplabJenkins commented Jan 8, 2016

Uh oh!

AmplabJenkins commented Jan 8, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas commented Jan 9, 2016

Uh oh!

AmplabJenkins commented Jan 9, 2016

Uh oh!

AmplabJenkins commented Jan 9, 2016

Uh oh!

AmplabJenkins commented Jan 9, 2016

Uh oh!

AmplabJenkins commented Jan 9, 2016

Uh oh!

tdas commented Jan 11, 2016

Uh oh!

tdas commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 11, 2016

Uh oh!

AmplabJenkins commented Jan 12, 2016

Uh oh!

AmplabJenkins commented Jan 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jan 12, 2016

Uh oh!

AmplabJenkins commented Jan 12, 2016

Uh oh!

AmplabJenkins commented Jan 12, 2016

Uh oh!

AmplabJenkins commented Jan 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone