Is there any way to make our example code in docstrings part of the test suite?

I'm worried about code rot and typos -- is there any way we can make it so that the snippets we have are actually tested ? Could we pull them out of a unit test somewhere? or a test suite that is specifically samples and we load based on an ID of sorts?

We don't have assertions in our code examples and we probably shouldn't. This would make our examples gross. If they want examples that definitely work and have assertions, they can look at the tests. :)

We don't have assertions in our code examples and we probably shouldn't.

Maybe we could put asserts in there and strip them out as part of populating the examples?

If they want examples that definitely work and have assertions, they can look at the tests. :)

That seems kind of crappy... I think we owe users of the library examples that work... :(

In kubernetes open source, we explicitly have a unit test that scrapes all
of our docs looking for examples like this and making sure that they still
work.

It's 100% do-able, and I think we should prioritize it. The number of out
of date examples, when I was initially learning the Google Cloud Platform
was really bad.

Brendan
On May 14, 2015 2:12 PM, "JJ Geewax" [email protected] wrote:

We don't have assertions in our code examples and we probably shouldn't.

Maybe we could put asserts in there and strip them out as part of
populating the examples?

If they want examples that definitely work and have assertions, they can
look at the tests. :)

That seems kind of crappy... I think we owe users of the library examples
that work... :(

—
Reply to this email directly or view it on GitHub
#598 (comment)
.

@brendandburns

Personally, I don't like the idea of docs and tests being so closely linked. We talk to computers with tests, we talk to humans with docs. I like the flexibility that our docs give us and the ability to be clear and concise. If we change this, our examples need to be as verbose as our tests in which case, why don't we just scrap the examples and copy and paste our tests into the docs?

I do understand the appeal, don't get me wrong, so if we can make this not as painful as I envision it, then I'd be happy to explore the idea.

@brendandburns can you link me to code on how this is done in kubernetes?

As an aside, I think our developers.google.com docs are always out of date. :( The struggle is so real.

If we change this, our examples need to be as verbose as our tests

To be more clear: I'm not saying our docs need to be as verbose as tests. I'm saying that they need to work -- always.

I'd really love it if we had a check at pull-request time to make sure all the examples still work... How we do this is in the air, but the goal is that if I were to check in a typo, Travis would reject it...

As an aside, I think our developers.google.com docs are always out of date. :( The struggle is so real.

And it sucks. Big time. This is just adding fuel to the fire of "let's make sure our examples work..."

Here is the code:

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/examples_test.go

The basic approach is:

Scrape through the docs, look for things that start with ```
Try and extract that thingy
Try to parse that data object into a sensible object
Validate with a rule in the test that indicates what the object
should be.

Concretely for developers.google.com, I think this would look like, writing
a unit test for each API, that scrapes through the relevant
developer.google.com content, extracts source code, and attempts to compile
it using the appropriate library. That way, if a developer modifies
something that breaks the example, they have to fix the docs as well as
the change their making.

A different option would be to figure out a way to automatically include
the source code from original source files, if you do that, then you can
just write traditional unit tests that live alongside the example source
files.

On Thu, May 14, 2015 at 2:39 PM, JJ Geewax [email protected] wrote:

If we change this, our examples need to be as verbose as our tests

To be more clear: I'm not saying our docs need to be as verbose as tests.
I'm saying that they need to work -- always.

I'd really love it if we had a check at pull-request time to make sure all
the examples still work... How we do this is in the air, but the goal is
that if I were to check in a typo, Travis would reject it...

As an aside, I think our developers.google.com docs are always out of
date. :( The struggle is so real.

And it sucks. Big time. This is just adding fuel to the fire of "let's
make sure our examples work..."

—
Reply to this email directly or view it on GitHub
#598 (comment)
.

@elibixby

@elibixby

FYI, for cloud.google.com (cloud instance of developers.google.com) sample code, we will go to the latter option.

Like thinking hypothetically, we should just split each line and run it through the node REPL. If it doesn't throw up somewhere along the way, good2go?

You mean all the lines as a whole right? Ie, this should fail:

var gcloud = require('gcloud');
var dataset = gcloud.datastore.dataset();
dataset.save({
  key: datastore.key('Company'),
  data: {name: 'Google'}
});

Oh yeah sorry, it's too early. That sounds reasonable yes?

@brendandburns

If the snippet runs successfully, that's a good thing...

I think it's safe to assume that the methods we're calling work as expected, and are tested to do so in a unit test. So the following "test" wouldn't be in the docs:

var key = dataset.key('Company');
assert(key.path[0] == 'Company');

We're not trying to check that methods do the right things (ie, that creating a key with 'Company' as the first argument sets the kind). We're trying to check that our examples use methods in the right ways. That is, I'm looking primary for typos, invalid method parameters (ie, dataset.key('too', 'many', 'arguments')), invalid method names (ie, datastore.key('Company') instead of dataset.key('Company')), and other things like that.

Figuring out if calling the right method does what it should do is a job for tests IMO. And we always have the "report an issue" button and pull requests for people who find broken code that slips into the gaps...

@brendandburns , can you comment and tell me if I'm way out of line here?