Repository navigation
Get() key search can bleed through levels of JSON hierarchy #5
Description
I want to first thank you, @buger, for your work on this library. Looking up a few JSON key paths in large JSON blobs is a significant bottleneck in a project I'm working on, and your library could give us a big speedup without changing our data format.
Unfortunately, I've discovered an issue in Get(): when searching for a key, Get() may locate that key outside the current JSON object. Here is an example test case that breaks (written using check.v1):
package jsonparser_test
import (
"github.com/buger-jsonparser"
. "gopkg.in/check.v1"
"testing"
)
func (s *JsonParserTests) TestJsonParserSearchBleed(c *C) {
killer := []byte(`{
"parentkey": {
"childkey": {
"grandchildkey": 123
},
"otherchildkey": 123
}
}`)
var jtype int
_, jtype, _, _ = jsonparser.Get(killer, "childkey")
c.Assert(jtype, Equals, jsonparser.NotExist) // fails, returns data parentkey.childkey
_, jtype, _, _ = jsonparser.Get(killer, "parentkey", "childkey", "otherchildkey")
c.Assert(jtype, Equals, jsonparser.NotExist) // fails, returns data parentkey.otherchildkey
}
// Boilerplate
func Test(t *testing.T) { TestingT(t) }
type JsonParserTests struct{}
var _ = Suite(&JsonParserTests{})
The issue is that Get() uses bytes.Index() to find the next key it's looking for, but only validates it by checking that it is surrounded by double quotes and followed by a colon. In particular, it does not check whether it has crossed an unmatched sequence of braces, which would indicate transitioning into another JSON object level.
I don't have a great suggestion as to how to fix this, sadly. Best of luck.
Activity
buger commentedon Mar 22, 2016
Good point, i have few ideas how to fix it, will keep you updated, thanks!
Limit scope while searching for nested keys #5
buger commentedon Mar 22, 2016
Should be fixed now, for each next key it will limit scope to parent key object.
daboyuka commentedon Mar 22, 2016
Sorry to say, your patch only fixes the second failing assertion, but not the first (which I think is the more difficult one, sadly).
buger commentedon Mar 22, 2016
I see what you mean, thank you for pointing. Yes, it is definitely a bit harder, will try to get it fixed.
daboyuka commentedon Mar 22, 2016
For what it's worth, my suggestion would be to use your parsing functions to more carefully step through the keys in an object. A (very) rough sketch:
Unfortunately this won't be nearly as efficient as bytes.Index, so perhaps there is a more efficient way.
daboyuka commentedon Mar 22, 2016
I found another case that breaks the current parser: key
"<anything>\"abc": matches a lookup for key "abc". Here's an expanded test case including that:buger commentedon Mar 22, 2016
Can you try this branch https://github.com/buger/jsonparser/tree/key-search, i rewrote how key search works. Thanks!
buger commentedon Mar 22, 2016
(to test branch, create vendor folder and put jsonparser to vendor/github.com/buger/jsonparser, then checkout needed branch there)
daboyuka commentedon Mar 22, 2016
Thanks, I'll check it in a little while. I'll also make a PR with the tests above for your convenience. Thanks for tackling this!
Added tricky JSON tests (issue #5)
daboyuka commentedon Mar 22, 2016
Your key-search branch passes the three cases identified in this issue (test cases added in #7). Thanks!
Merge pull request #7 from pendo-io/master
buger commentedon Mar 24, 2016
Merged to master, thanks!
jsonparser re-sweep with translator gap fixes — 12 new lemmas