Repository navigation
feat: add recursiveDelete() - #1427
Conversation
f99da27 to
3d1fa79
Compare
Codecov Report
@@ Coverage Diff @@
## bc/rc-main #1427 +/- ##
==============================================
+ Coverage 98.20% 98.51% +0.31%
==============================================
Files 32 32
Lines 19502 19623 +121
Branches 1363 1296 -67
==============================================
+ Hits 19152 19332 +180
+ Misses 346 286 -60
- Partials 4 5 +1
Continue to review full report at Codecov.
|
| writer.flush().then(async () => { | ||
| if (ref instanceof DocumentReference) { | ||
| try { | ||
| await ref.delete(); |
There was a problem hiding this comment.
I can't think of a more elegant way to differentiate between the final document reference delete failing vs. the BulkWriter deletes. Any suggestions?
There was a problem hiding this comment.
I haven't looked at the rest of the PR, but if you want to work on it before I do - couldn't you just let the BulkWriter handle this delete as well?
There was a problem hiding this comment.
I still have a few more changes/cleanup I need to push upstream, so reviewing now might result in some wasted work.
Thanks for the suggestion! I originally figured a normal delete is less verbose than enqueuing another operation to BulkWriter and calling flush(), but using BulkWriter allows for any user callbacks to also run.
|
One more note: Can you ping the backend chat to determine whether we can retry deletes that fail with RST_STREAM? (I see this is already part of your TODOs, so please pretend that you never saw this note). |
| err.message = | ||
| 'Failed to fetch children documents. ' + | ||
| 'The provided reference was not deleted.'; | ||
| deleteCompleted.reject(wrapError(err, stack)); |
There was a problem hiding this comment.
Is not deleting the provided reference in the case of a failed StructuredQuery the ideal, expected behavior? I need a quick sanity check. Here are the options:
- Do not delete the provided reference if StructuredQuery stream ever errors.
- Do not delete provided reference iff StructuredQuery stream errors before any deletes are enqueued.
- Delete the reference regardless of what happens.
I think we originally agreed on Option 2. It makes logical sense -- if Firestore can't fetch any of the descendants, that's a failed precondition, so Firestore shouldn't delete the provided reference. However, what's tripping me up is if the StructuredQuery stream errors out midway through the deletion process. In this case, an argument can be made both ways -- Firestore shouldn't delete the provided reference because it couldn't fetch the descendants (failed precondition argument) OR Firestore should delete the provided reference to be consistent in behavior (the provided reference is deleted even if BulkWriter fails to delete some references).
Trying to document all these edge cases led me to consider Option 3 -- if I'm a developer who simply wants the collection or document deleted, I don't care about the provided reference, and Firestore should just nuke whatever it can get a hold of.
There was a problem hiding this comment.
The query can error at any point in time. I would suggest we make no guarantees whatsoever, which makes the overall behavior of the API much more predictable. If any operation fails, the database tree is in an undefined state, and the user needs to try again.
There was a problem hiding this comment.
Done. Changed it to always delete the provided reference when possible, but kept the error message in for query stream failures. Not sure if we want the FAILED_PRECONDITION status code though.
| ); | ||
| if (lastError instanceof BulkWriterError) { | ||
| error.code = (lastError.code as number) as Status; | ||
| } |
There was a problem hiding this comment.
Not sure why these lines aren't covered by Codecov, but I'm pretty sure they're being called. Maybe it's b/c I put it in a lambda function to access lastError and errorCount. Is this an anti-pattern?
schmidt-sebastian
left a comment
There was a problem hiding this comment.
Feedback for your implementation. I hope that I can get to the tests soon. They look a bit scary in that you wrote a lot of them :)
| * | ||
| * @private | ||
| */ | ||
| private _bulkWriter: BulkWriter | undefined; |
There was a problem hiding this comment.
Could we move this to BulkWriter (kind of like a singleton pattern)? This could then be a shared instance that could be used by other operations in the future as well.
I also wonder whether the default ramp up makes sense here. It will slow down the delete (especially when compared to the CLI) and it also doesn't quite fit the use case for 5/5/5. We do not have to ramp up to give the database a chance to shard - the data is already sharded, we just want to delete it.
There was a problem hiding this comment.
Could we move this to BulkWriter (kind of like a singleton pattern)? This could then be a shared instance that could be used by other operations in the future as well.
Do you mean that calling firestore.bulkWriter() would return the same instance each time, or having a Firestore-internal BulkWriter.bulkWriter() getter that lazy-loads?
I also wonder whether the default ramp up makes sense here.
Yeah, that's something I was thinking about as well to implement as part of the RST_STREAM retries. Deletes aren't subject to 555 throttling, which creates two possibilities. 1) We can add an internal override on BulkWriter to turn off throttling when recursive delete is called. However, this would mean that writes being performed at the same time won't get throttled. 2) We can add logic for BulkWriter to automatically ignore throttling limits for deletes, but then we'd have to batch deletes separately from writes (or think of another solution that allows writes to still be throttled).
There was a problem hiding this comment.
Firestore internal getter that lazy-loads. I want to keep it similar to what you have, but try to move these internals out of Firestore.
There was a problem hiding this comment.
Added a static BulkWriter.bulkWriter() method.
| err.message = | ||
| 'Failed to fetch children documents. ' + | ||
| 'The provided reference was not deleted.'; | ||
| incrementErrorCount(err); |
There was a problem hiding this comment.
This shouldn't bump up the error count that is used for the error message, should it?
There was a problem hiding this comment.
Good point. This made me realize that the # deletes failed message won't be accurate if the stream fails since all the remaining docs not returned by the stream are unaccounted for. Should I special case the error message in the stream failure case to capture this fact?
| const writer = bulkWriter ?? this.getBulkWriter(); | ||
| writer._verifyNotClosed(); | ||
| const docStream = this.getAllDescendants(ref); | ||
| const deleteCompletedDeferred = new Deferred<void>(); |
There was a problem hiding this comment.
Should this just be result? While generic, it makes it clear that this is what the method returns.
There was a problem hiding this comment.
Renamed to resultDeferred. Since the return type is a void promise, I want to make it clear that there is no real "result" being returned.
| // TODO(chenbrian): Make this a private method after adding recursive delete. | ||
| _getAllDescendants( | ||
| ref: CollectionReference | DocumentReference | ||
| private getAllDescendants<T = DocumentData>( |
There was a problem hiding this comment.
This should still be underscore prefixed to prevent "usage by code completion".
There was a problem hiding this comment.
Renamed. Aren't private methods hidden from code completion? How do they show up? Also, what's the difference between the _method() and method_() patterns in this file?
There was a problem hiding this comment.
"private" is only hidden from TS developers. JS developers will still see it in code completion.
Underscore suffix is the same as underscore prefix, but from about 10 years ago. I am surprised this client uses underscore suffix.
Co-authored-by: Sebastian Schmidt <[email protected]>
schmidt-sebastian
left a comment
There was a problem hiding this comment.
Feedback for the tests. Generally looks great!
| try { | ||
| expect(request.writes).to.deep.equal(expected.writes); | ||
| } catch (e) { | ||
| batchWriteError = e; |
There was a problem hiding this comment.
Do the tests not fail if we simply let the exception throw here? Ideally, the implementation should be able to handle if the batchWrite RPC throws an exception and propagate it as a failed BatchWrite.
There was a problem hiding this comment.
I have a test that checks the error stack to make sure that the original error thrown in batchWrite is surfaced. BulkWriter currently doesn't do any special handling for batchWrite RPC failures, so recursive delete can't perform any handling logic for it. Do you think we should add logic to surface batchWrite RPC failures differently?
There was a problem hiding this comment.
No, I think we can keep the logic as is. Individual RPC failures versus individual operations failures are not different from a user perspective. I was trying to see if we could simply bubble up the assertion failure as an individual RPC failure, which would surface these assertions in onWriteResult or via failed promises rather than via batchWriteError.
| firestore = await instantiateInstance([ | ||
| ['anna', 'bob', 'bob/children/charlie', 'bob/children/daniel'], | ||
| ]); | ||
| await firestore.recursiveDelete(firestore.collection('collectionId')); |
There was a problem hiding this comment.
I am not sure I follow this test. The documents are not part of the collection, but they are still getting returned from the RunQuery request if I am not mistaken.
There was a problem hiding this comment.
The collectionId is implicit in instantiateInstance(), which follows the pattern that other test helpers use. So in this case, the documents under firestore.collection('collectionId') are ['collectionId/anna', 'collectionId/bob', ...].
Added a comment to clarify.
| let callbackCount = 0; | ||
| const bulkWriter = firestore.bulkWriter(); | ||
| bulkWriter.onWriteResult(() => { | ||
| callbackCount++; |
There was a problem hiding this comment.
Do we want to add a test that verifies that we return the expected list of DocumentReferences?
| callbackCount++; | ||
| }); | ||
| await firestore.recursiveDelete(firestore.collection('foo'), bulkWriter); | ||
| expect(callbackCount).to.equal(3); |
There was a problem hiding this comment.
We also don't seem to have tests that exercise the error handler.
| queryEqualsHelper(actual, '', ...protoComponents); | ||
| } | ||
|
|
||
| function queryEqualsHelper( |
There was a problem hiding this comment.
I am not a huge fan of Helpers. Would it be possible to just make this queryEqualsWithParent?
There was a problem hiding this comment.
A lot of tests use queryEquals() without a parent parameter, and I didn't want to pass in an empty string for parent in all of them or change the spread operator to an array. Is it possible to combine these two signatures in an overload:
export function queryEquals(
actual: api.IRunQueryRequest | undefined,
parent: string,
...protoComponents: api.IStructuredQuery[]
): void;
export function queryEquals(
actual: api.IRunQueryRequest | undefined,
...protoComponents: api.IStructuredQuery[]
): void;There was a problem hiding this comment.
You could do:
export function queryEquals(
actual: api.IRunQueryRequest | undefined,
parentOrProtoComponent: string|api.IStructuredQuery|undefined,
...protoComponents: api.IStructuredQuery[]
): void;
But that is not all that pretty.
I was actually thinking you keep both queryEquals and queryEqualsWithParent, but merge queryEqualsWithParent and queryEqualsHelper.
There was a problem hiding this comment.
oops, that makes more sense. done.
| * containing the number of failed deletes and the stack trace of the last | ||
| * failed delete. The provided reference is deleted regardless of whether | ||
| * all deletes succeeded, except when Firestore fails to fetch the provided | ||
| * reference's descendants. |
There was a problem hiding this comment.
Please update according to the suggestions above.
| * The promise is rejected if any of the deletes fail. | ||
| */ | ||
| recursiveDelete( | ||
| ref: CollectionReference | DocumentReference, |
There was a problem hiding this comment.
Please add <unknown>, otherwise this typing won't allow converters.
thebrianchen
left a comment
There was a problem hiding this comment.
Thanks for the detailed review! Addressed comments for the logic, going to work on test comments next.
| ref: CollectionReference<T> | DocumentReference<T>, | ||
| bulkWriter?: BulkWriter | ||
| ): Promise<void> { | ||
| const writer = bulkWriter ?? this.getBulkWriter(); |
There was a problem hiding this comment.
writer has type BulkWriter, whereas bulkWriter always has type BulkWriter | undefined, which makes using the variable later messier.
| ): Promise<void> { | ||
| const writer = bulkWriter ?? this.getBulkWriter(); | ||
| writer._verifyNotClosed(); | ||
| const docStream = this.getAllDescendants(ref); |
There was a problem hiding this comment.
Hmm, I'm not really sure where to move it. Are you suggesting we encapsulate all the behavior in a separate class?
| err.message = | ||
| 'Failed to fetch children documents. ' + | ||
| 'The provided reference was not deleted.'; | ||
| incrementErrorCount(err); |
There was a problem hiding this comment.
Good point. This made me realize that the # deletes failed message won't be accurate if the stream fails since all the remaining docs not returned by the stream are unaccounted for. Should I special case the error message in the stream failure case to capture this fact?
| * The promise is rejected if any of the deletes fail. | ||
| */ | ||
| recursiveDelete<T = DocumentData>( | ||
| ref: CollectionReference<T> | DocumentReference<T>, |
There was a problem hiding this comment.
I had to change the signature to firestore.DocumentRef and add some casting logic from the firestore typed classes to our internal classes. Can you double check to see if there's a cleaner way? Thanks!
| * Firestore uses a BulkWriter instance with default settings to perform the | ||
| * deletes. To customize throttling rates or add success/error callbacks, | ||
| * pass in a custom BulkWriter instance. | ||
| * |
There was a problem hiding this comment.
Done. I included the default error handler to make it easier to paste. Do you think including it would confuse developers?
| * | ||
| * @private | ||
| */ | ||
| private _bulkWriter: BulkWriter | undefined; |
There was a problem hiding this comment.
Could we move this to BulkWriter (kind of like a singleton pattern)? This could then be a shared instance that could be used by other operations in the future as well.
Do you mean that calling firestore.bulkWriter() would return the same instance each time, or having a Firestore-internal BulkWriter.bulkWriter() getter that lazy-loads?
I also wonder whether the default ramp up makes sense here.
Yeah, that's something I was thinking about as well to implement as part of the RST_STREAM retries. Deletes aren't subject to 555 throttling, which creates two possibilities. 1) We can add an internal override on BulkWriter to turn off throttling when recursive delete is called. However, this would mean that writes being performed at the same time won't get throttled. 2) We can add logic for BulkWriter to automatically ignore throttling limits for deletes, but then we'd have to batch deletes separately from writes (or think of another solution that allows writes to still be throttled).
thebrianchen
left a comment
There was a problem hiding this comment.
One more note: Can you ping the backend chat to determine whether we can retry deletes that fail with RST_STREAM? (I see this is already part of your TODOs, so please pretend that you never saw this note).
Checked in with with backend team, and it's fine to retry deletes since they're idempotent.
| * containing the number of failed deletes and the stack trace of the last | ||
| * failed delete. The provided reference is deleted regardless of whether | ||
| * all deletes succeeded, except when Firestore fails to fetch the provided | ||
| * reference's descendants. |
| let callbackCount = 0; | ||
| const bulkWriter = firestore.bulkWriter(); | ||
| bulkWriter.onWriteResult(() => { | ||
| callbackCount++; |
| callbackCount++; | ||
| }); | ||
| await firestore.recursiveDelete(firestore.collection('foo'), bulkWriter); | ||
| expect(callbackCount).to.equal(3); |
| firestore = await instantiateInstance([ | ||
| ['anna', 'bob', 'bob/children/charlie', 'bob/children/daniel'], | ||
| ]); | ||
| await firestore.recursiveDelete(firestore.collection('collectionId')); |
There was a problem hiding this comment.
The collectionId is implicit in instantiateInstance(), which follows the pattern that other test helpers use. So in this case, the documents under firestore.collection('collectionId') are ['collectionId/anna', 'collectionId/bob', ...].
Added a comment to clarify.
| queryEqualsHelper(actual, '', ...protoComponents); | ||
| } | ||
|
|
||
| function queryEqualsHelper( |
There was a problem hiding this comment.
A lot of tests use queryEquals() without a parent parameter, and I didn't want to pass in an empty string for parent in all of them or change the spread operator to an array. Is it possible to combine these two signatures in an overload:
export function queryEquals(
actual: api.IRunQueryRequest | undefined,
parent: string,
...protoComponents: api.IStructuredQuery[]
): void;
export function queryEquals(
actual: api.IRunQueryRequest | undefined,
...protoComponents: api.IStructuredQuery[]
): void;
schmidt-sebastian
left a comment
There was a problem hiding this comment.
Looking better and better. Some small suggestions remain.
| * bulkWriter | ||
| * .onWriteError((error) => { | ||
| * if ( | ||
| * error.code === GrpcStatus.UNAVAILABLE && |
There was a problem hiding this comment.
I think we should drop this line. Deletes are almost always retyrable, and we should show this here and do a blind retry up to MAX_RETRY_ATTEMPTS times.
Regardless of what the developer does, we should apply a sensible backoff on top of these attempts. This means that we should backoff longer for RESOURCE_EXHAUSTED.
There was a problem hiding this comment.
Done. Added the RESOURCE_EXHAUSTED note to the todo list at the top of the PR.
| * ) { | ||
| * return true; | ||
| * } else { | ||
| * console.log('Failed write at document: ', error.documentRef); |
There was a problem hiding this comment.
We should probably add a toString() to make this output sensible.
"That sounds like a good idea, but it seems unrelated to this PR."
I came up with an answer for you to copy & paste.
There was a problem hiding this comment.
I often rely on printing the javascript object for DocumentReference when debugging to view all the different attributes of the reference, so I'm afraid to replace that with a pretty-print.
Updated to use documentRef.path which should be sufficient. Thoughts?
|
|
||
| docStream | ||
| .on('error', err => { | ||
| err.code = Status.FAILED_PRECONDITION; |
There was a problem hiding this comment.
Please take a look at https://developers.google.com/maps-booking/reference/grpc-api/status_codes
FAILED_PRECONDITION means that a simple retry won't work here. The developer has to change something about the state of the system before retrying the operation. This is likely not the case here. I would guess that we will have an error code here already when we hit this code path, but if we don't, I would suggest using UNAVAILABLE as the return code.
| } | ||
| const collectionId = | ||
| ref instanceof CollectionReference ? ref.id : ref.parent.id; | ||
| ref instanceof CollectionReference |
There was a problem hiding this comment.
CollectionReference means CollectionReference<DocumentData>. If you change this to ref instanceof CollectionReference<unknown> you should be able to remove the as DocumentReference cast (which should also be as DocumentReference<unknown>).
There was a problem hiding this comment.
The TS compiler was not happy with that suggestion:
TS2348: Value of type 'typeof CollectionReference' is not callable. Did you mean to include 'new'
There was a problem hiding this comment.
I think the TS compiler is right, since the generics support of TypeScript is pretty much like Java and "erased at compile time". I was able to get the code to transpiler even without the ref as DocumentReference though..
FWIW, an as DocumentReference<unknown> would still be the correct way to type this if we were to keep the cast. Right now, you are dropping the converter in the cast, when really we don't know anything about it.
There was a problem hiding this comment.
Added the <unknown> type to the DocumentReference cast.
| queryEqualsHelper(actual, '', ...protoComponents); | ||
| } | ||
|
|
||
| function queryEqualsHelper( |
There was a problem hiding this comment.
You could do:
export function queryEquals(
actual: api.IRunQueryRequest | undefined,
parentOrProtoComponent: string|api.IStructuredQuery|undefined,
...protoComponents: api.IStructuredQuery[]
): void;
But that is not all that pretty.
I was actually thinking you keep both queryEquals and queryEqualsWithParent, but merge queryEqualsWithParent and queryEqualsHelper.
| }); | ||
| await firestore.recursiveDelete(randomCol, bulkWriter); | ||
| expect(callbackCount).to.equal(6); | ||
| }); |
There was a problem hiding this comment.
Can we also get a test that has "randomCollA/docA" and "randomCollB/docB" and only deletes randomCollA? There might be some nuances in how to delete parent collection (in other ports) and I want to make sure that all of our clients handle this case correctly.
| try { | ||
| expect(request.writes).to.deep.equal(expected.writes); | ||
| } catch (e) { | ||
| batchWriteError = e; |
There was a problem hiding this comment.
No, I think we can keep the logic as is. Individual RPC failures versus individual operations failures are not different from a user perspective. I was trying to see if we could simply bubble up the assertion failure as an individual RPC failure, which would surface these assertions in onWriteResult or via failed promises rather than via batchWriteError.
| * @private | ||
| */ | ||
| static _getInstance(firestore: Firestore): BulkWriter { | ||
| if (!this.instance || this.instance.firestore !== firestore) { |
There was a problem hiding this comment.
I didn't think about the fact that this would break if multiple Firestore instances are used together. I am sightly inclined to ask you to undo this change since it would simplify the instance creation and its lifetime management (at the cost of further increasing the size of Firestore).
There was a problem hiding this comment.
I can see arguments either way, but at this point, I'm more in favor of leaving it on the Firestore instance, since trying to track the different firestore instances to pass into BulkWriter is more confusing than just having the BulkWriter instance available on the Firestore instance.
Reverted back to before.
|
|
||
| expect(await countCollectionChildren(collA)).to.equal(0); | ||
| expect(await countCollectionChildren(randomCol)).to.equal(3); | ||
| expect(await countCollectionChildren(collB)).to.equal(1); |
There was a problem hiding this comment.
I think this is now a similar test to what you have above (something akin to on nested collection). I think the edge case is for root collections so something like randomCol.doc('bob') and a recursive delete on "otherCollection" (which could really be any name). We can then ensure than randomCol still has one document.
There was a problem hiding this comment.
Thanks for clarifying what you meant. Added a test for that.
| } | ||
| const collectionId = | ||
| ref instanceof CollectionReference ? ref.id : ref.parent.id; | ||
| ref instanceof CollectionReference |
There was a problem hiding this comment.
I think the TS compiler is right, since the generics support of TypeScript is pretty much like Java and "erased at compile time". I was able to get the code to transpiler even without the ref as DocumentReference though..
FWIW, an as DocumentReference<unknown> would still be the correct way to type this if we were to keep the cast. Right now, you are dropping the converter in the cast, when really we don't know anything about it.
| * return false; | ||
| * } | ||
| * }); | ||
| * await firestore.recursiveDelete(docRef, bulkWriter); |
There was a problem hiding this comment.
Please update this snippet as well (drop the error code, change the logging).
| * Firestore uses a BulkWriter instance with default settings to perform the | ||
| * deletes. To customize throttling rates or add success/error callbacks, | ||
| * pass in a custom BulkWriter instance. | ||
| * |
…e, comment updates
thebrianchen
left a comment
There was a problem hiding this comment.
Thanks for the quick feedback cycles!
| * return false; | ||
| * } | ||
| * }); | ||
| * await firestore.recursiveDelete(docRef, bulkWriter); |
| } | ||
| const collectionId = | ||
| ref instanceof CollectionReference ? ref.id : ref.parent.id; | ||
| ref instanceof CollectionReference |
There was a problem hiding this comment.
Added the <unknown> type to the DocumentReference cast.
| * @private | ||
| */ | ||
| static _getInstance(firestore: Firestore): BulkWriter { | ||
| if (!this.instance || this.instance.firestore !== firestore) { |
There was a problem hiding this comment.
I can see arguments either way, but at this point, I'm more in favor of leaving it on the Firestore instance, since trying to track the different firestore instances to pass into BulkWriter is more confusing than just having the BulkWriter instance available on the Firestore instance.
Reverted back to before.
|
|
||
| expect(await countCollectionChildren(collA)).to.equal(0); | ||
| expect(await countCollectionChildren(randomCol)).to.equal(3); | ||
| expect(await countCollectionChildren(collB)).to.equal(1); |
There was a problem hiding this comment.
Thanks for clarifying what you meant. Added a test for that.
First pass at adding recursive delete. Merging to a feature branch for iterative development.
Some things that I still have yet to add: