Skip to content

doctor: convert nonnumber confusables into explicit strings #1127

Open

Description

@hseg
  • papis version ($ papis --version or commit number): 79dd5d7e

how to reproduce the issue

Consider the citations of https://doi.org/10.1038/s41562-018-0474-5

citations:
- unstructured: Halbwachs, M. La Mémoire Collective (Albin Michel, Paris, 1997).
- author: J Assmann
  doi: 10.2307/488538
  first-page: 125
  journal-title: New Ger. Crit.
  unstructured: Assmann, J. Collective memory and cultural identity. New Ger. Crit.
    65, 125–133 (1995).
  volume: 65
  year: 1995
- unstructured: Assmann, J. Das kulturelle Gedächtnis. Schrift, Erinnerung und politische
    Identität in frühen Hochkulturen (Verlag C. H. Beck, Munich, 2007).
- doi: 10.1007/978-90-481-8945-8_2
  unstructured: Assmann, J. in Cultural Memories (eds Meusburger, P. et al.) 15–27
    (Springer, Heidelberg, 2011).
- doi: 10.1017/CBO9780511613715
  unstructured: Wertsch, J. V. Voices of Collective Remembering (Cambridge Univ. Press,
    Cambridge, 2002).
- unstructured: 'Goldhammer, A., Nora, P. & Kritzman, L. D. Realms of Memory: The
    Construction of the French Past (Columbia Univ. Press, New York, 1998).'
- author: H Roediger
  doi: 10.1126/science.1259627
  first-page: 1106
  journal-title: Science
  unstructured: Roediger, H. & DeSoto, K. Forgetting the presidents. Science 346,
    1106–1109 (2014).
  volume: 346
  year: 2014
- doi: 10.1017/CBO9780511626999.009
  unstructured: Roediger, H. L. III, Zaromb, F. M. & Butler, A. C. in Memory in Mind
    and Culture (eds Boyer, P. & Wertsch, J. V.) 138–170 (Cambridge Univ. Press, Cambridge,
    2009).
- author: F Zaromb
  doi: 10.3758/s13421-013-0369-7
  first-page: 383
  journal-title: Mem. Cognit.
  unstructured: Zaromb, F., Butler, A. C., Agarwal, P. K. & Roediger, H. L. III. Collective
    memories of three wars in United States history in younger and older adults. Mem.
    Cognit. 42, 383–399 (2014).
  volume: 42
  year: 2014
- author: DC Rubin
  doi: 10.1126/science.aaa2341
  first-page: 1058
  journal-title: Science
  unstructured: Rubin, D. C. How quickly we forget. Science 346, 1058–1059 (2014).
  volume: 346
  year: 2014
- author: R García-Gavilanes
  doi: 10.1126/sciadv.1602368
  first-page: e1602368
  journal-title: Sci. Adv.
  unstructured: 'García-Gavilanes, R., MollgaardA., Tsvetkova, M. & Yasseri, T. The memory remains: understanding collective memory in the digital age. Sci. Adv. 3, e1602368 (2017).'
  volume: 3
  year: 2017
- author: D Wang
  doi: 10.1126/science.1237825
  first-page: 127
  journal-title: Science
  unstructured: Wang, D., Song, C. & Barabási, A.-L. Quantifying long-term scientific
    impact. Science 342, 127–132 (2013).
  volume: 342
  year: 2013
- author: KW Higham
  doi: 10.1103/PhysRevE.95.042309
  first-page: 042309
  journal-title: Phys. Rev. E
  unstructured: 'Higham, K. W., Governale, M., Jaffe, A. B. & Zülicke, U. Fame and
    obsolescence: disentangling growth and aging dynamics of patent citations. Phys.
    Rev. E 95, 042309 (2017).'
  volume: 95
  year: 2017
- author: K Higham
  doi: 10.1016/j.joi.2017.10.004
  first-page: 1190
  journal-title: J. Informetr.
  unstructured: Higham, K., Governale, M., Jaffe, A. & Zülicke, U. Unraveling the
    dynamics of growth, aging and ination for citations to scientific articles from
    specific research fields. J. Informetr. 11, 1190–1200 (2017).
  volume: 11
  year: 2017
- author: S Valverde
  doi: 10.1103/PhysRevE.76.056118
  first-page: 056118
  journal-title: Phys. Rev. E
  unstructured: Valverde, S., Solé, R. V., Bedau, M. A. & Packard, N. Topology and
    evolution of technology innovation networks. Phys. Rev. E 76, 056118 (2007).
  volume: 76
  year: 2007
- author: G Csárdi
  doi: 10.1016/j.physa.2006.08.022
  first-page: 783
  journal-title: Physica A
  unstructured: Csárdi, G., Strandburg, K. J., Zalányi, L., Tobochnik, J. & Érdi,
    P. Modeling innovation by a kinetic description of the patent citation system.
    Physica A 374, 783–793 (2007).
  volume: 374
  year: 2007
- author: M Golosovsky
  doi: 10.1103/PhysRevLett.109.098701
  first-page: 098701
  journal-title: Phys. Rev. Lett.
  unstructured: Golosovsky, M. & Solomon, S. Stochastic dynamical model of a growing
    citation network based on a self-exciting point process. Phys. Rev. Lett. 109,
    098701 (2012).
  volume: 109
  year: 2012
- author: SN Dorogovtsev
  doi: 10.1103/PhysRevE.62.1842
  first-page: 1842
  journal-title: Phys. Rev. E
  unstructured: Dorogovtsev, S. N. & Mendes, J. F. F. Evolution of networks with aging
    of sites. Phys. Rev. E 62, 1842–1845 (2000).
  volume: 62
  year: 2000
- author: AL Barabási
  doi: 10.1126/science.286.5439.509
  first-page: 509
  journal-title: Science
  unstructured: Barabási, A.-L. & Albert, R. Emergence of scaling in random networks.
    Science 286, 509–512 (1999).
  volume: 286
  year: 1999
- author: R Albert
  doi: 10.1103/RevModPhys.74.47
  first-page: 47
  journal-title: Rev. Mod. Phys.
  unstructured: Albert, R. & Barabási, A.-L. Statistical mechanics of complex networks.
    Rev. Mod. Phys. 74, 47–97 (2002).
  volume: 74
  year: 2002
- author: DDS Price
  doi: 10.1002/asi.4630270505
  first-page: 292
  journal-title: J. Am. Soc. Inf. Sci.
  unstructured: Price, D. D. S. A general theory of bibliometric and other cumulative
    advantage processes. J. Am. Soc. Inf. Sci. 27, 292–306 (1976).
  volume: 27
  year: 1976
- author: GU Yule
  doi: 10.1098/rstb.1925.0002
  first-page: 21
  journal-title: Phil. Trans. R. Soc. Lond. B
  unstructured: Yule, G. U. A mathematical theory of evolution, based on the conclusions
    of Dr. J. C. Willis, F.R.S. Phil. Trans. R. Soc. Lond. B 213, 21–87 (1925).
  volume: 213
  year: 1925
- author: RK Merton
  doi: 10.1086/354848
  first-page: 606
  journal-title: Isis
  unstructured: 'Merton, R. K. The Matthew effect in science, II: cumulative advantage
    and the symbolism of intellectual property. Isis 79, 606–623 (1988).'
  volume: 79
  year: 1988
- doi: 10.1515/9783110207262.2.109
  unstructured: Assmann, J. in Cultural Memory Studies. An International and Interdisciplinary
    Handbook (eds Erll, A. & Nünning, A.) 109–118 (Walter de Gruyter, Berlin, 2008).
- author: W Hirst
  doi: 10.1016/j.tics.2018.02.010
  first-page: 438
  journal-title: Trends. Cogn. Sci.
  unstructured: Hirst, W., Yamashiro, J. K. & Coman, A. Collective memory from a psychological
    perspective. Trends. Cogn. Sci. 22, 438–451 (2018).
  volume: 22
  year: 2018
- author: HL Roediger
  doi: 10.1177/0956797616631113
  first-page: 644
  journal-title: Psychol. Sci.
  unstructured: 'Roediger, H. L. & DeSoto, K. A. Recognizing the presidents: was Alexander
    Hamilton president? Psychol. Sci. 27, 644–650 (2016).'
  volume: 27
  year: 2016
- doi: 10.1093/oso/9780195082111.001.0001
  unstructured: 'Rubin, D. C. Memory in Oral Traditions: the Cognitive Psychology
    of Epic, Ballads, and Counting-Out Rhymes (Oxford Univ. Press, Oxford, 1995).'
- author: PL Hammack
  doi: 10.1075/ni.21.2.09ham
  first-page: 311
  journal-title: Narrat. Inq.
  unstructured: Hammack, P. L. Narrative and the politics of meaning. Narrat. Inq.
    21, 311–318 (2011).
  volume: 21
  year: 2011
- author: D Sperber
  doi: 10.1016/j.tics.2003.11.002
  first-page: 40
  journal-title: Trends Cogn. Sci.
  unstructured: Sperber, D. & Hirschfeld, L. A. The cognitive foundations of cultural
    stability and diversity. Trends Cogn. Sci. 8, 40–46 (2004).
  volume: 8
  year: 2004
- author: A Buskell
  doi: 10.1007/s10539-017-9570-6
  first-page: 377
  journal-title: Biol. Philos.
  unstructured: Buskell, A. What are cultural attractors? Biol. Philos. 32, 377–394
    (2017).
  volume: 32
  year: 2017
- doi: 10.7208/chicago/9780226712130.001.0001
  unstructured: 'Richerson, P. J. & Boyd, R. Not by Genes Alone: How Culture Transformed
    Human Evolution (Univ. Chicago Press, Chicago, 2005).'
- author: BC Storm
  doi: 10.1080/20445911.2012.674030
  first-page: 617
  journal-title: J. Cogn. Psychol.
  unstructured: Storm, B. C., Bjork, E. L. & Bjork, R. A. On the durability of retrieval-induced
    forgetting. J. Cogn. Psychol. 24, 617–629 (2012).
  volume: 24
  year: 2012
- author: E Garcia-Bajos
  doi: 10.1080/09658210802572454
  first-page: 92
  journal-title: Memory
  unstructured: Garcia-Bajos, E., Migueles, M. & Anderson, M. Script knowledge modulates
    retrieval-induced forgetting for eyewitness events. Memory 17, 92–103 (2009).
  volume: 17
  year: 2009
- author: A Cuc
  doi: 10.1111/j.1467-9280.2007.01967.x
  first-page: 727
  journal-title: Psychol. Sci.
  unstructured: 'Cuc, A., Koppel, J. & Hirst, W. Silence is not golden: a case for
    socially shared retrieval-induced forgetting. Psychol. Sci. 18, 727–733 (2007).'
  volume: 18
  year: 2007
- author: G Echterhoff
  doi: 10.1111/j.1745-6924.2009.01161.x
  first-page: 496
  journal-title: Perspect. Psychol. Sci.
  unstructured: 'Echterhoff, G., Higgins, E. T. & Levine, J. M. Shared reality: experiencing
    commonality with others’ inner states about the world. Perspect. Psychol. Sci.
    4, 496–521 (2009).'
  volume: 4
  year: 2009
- author: A Coman
  doi: 10.1037/xge0000077
  first-page: 717
  journal-title: J. Exp. Psychol. Gen.
  unstructured: 'Coman, A. & Hirst, W. Social identity and socially shared retrieval-induced
    forgetting: the effects of group membership. J. Exp. Psychol. Gen. 144, 717–722
    (2015).'
  volume: 144
  year: 2015
- author: A Coman
  doi: 10.1177/0956797614531024
  first-page: 1281
  journal-title: Psychol. Sci.
  unstructured: 'Coman, A., Stone, C. B., Castano, E. & Hirst, W. Justifying atrocities:
    the effect of moral-disengagement strategies on socially shared retrieval-induced
    forgetting. Psychol. Sci. 25, 1281–1285 (2014).'
  volume: 25
  year: 2014
- author: CB Stone
  doi: 10.1080/09658210903159003
  first-page: 170
  journal-title: Memory
  unstructured: 'Stone, C. B., Barnier, A. J., Sutton, J. & Hirst, W. Building consensus
    about the past: schema consistency and convergence in socially shared retrieval-induced
    forgetting. Memory 18, 170–184 (2010).'
  volume: 18
  year: 2010
- author: R Kurzban
  doi: 10.1073/pnas.251541498
  first-page: 15387
  journal-title: Proc. Natl Acad. Sci. USA
  unstructured: Kurzban, R., Tooby, J. & Cosmides, L. Can race be erased? Coalitional
    computation and social categorization. Proc. Natl Acad. Sci. USA 98, 15387–15392
    (2001).
  volume: 98
  year: 2001
- author: AZ Yu
  doi: 10.1038/sdata.2015.75
  first-page: 150075
  journal-title: Sci. Data
  unstructured: Yu, A. Z., Ronen, S., Hu, K., Lu, T. & Hidalgo, C. A. Pantheon 1.0,
    a manually verified dataset of globally famous biographies. Sci. Data 3, 150075
    (2016).
  volume: 3
  year: 2016
- unstructured: 'Jara-Figueroa, C., Yu, A. Z. & Hidalgo, C. A. How the medium shapes
    the message: printing and the rise of the arts and sciences. PLoS ONE (in the
    press).'
- doi: 10.1017/CBO9781139649605
  unstructured: Skiena, S. & Ward, C. B. Who’s Bigger? Where Historical Figures Really
    Rank (Cambridge Univ. Press, Cambridge, 2014).
- doi: 10.1109/JCDL.2014.6970189
  unstructured: Kanhabua, N., Nguyen, T. N. & Niederée, C. What triggers human remembering
    of events? A large-scale analysis of catalysts for collective memory in Wikipedia.
    In Proc. 14th ACM/IEEE-CS Joint Conference on Digital Libraries 341–350 (IEEE,
    2014).
- author: B Uzzi
  doi: 10.1126/science.1240474
  first-page: 468
  journal-title: Science
  unstructured: Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations
    and scientific impact. Science 342, 468–472 (2013).
  volume: 342
  year: 2013
- author: S Mukherjee
  doi: 10.1126/sciadv.1601315
  first-page: e1601315
  journal-title: Sci. Adv.
  unstructured: 'Mukherjee, S., Romero, D. M., Jones, B. & Uzzi, B. The nearly universal
    link between the age of past knowledge and tomorrow''s breakthroughs in science
    and technology: the hotspot. Sci. Adv. 3, e1601315 (2017).'
  volume: 3
  year: 2017
- author: S Ronen
  doi: 10.1073/pnas.1410931111
  first-page: E5616
  journal-title: Proc. Natl Acad. Sci. USA
  unstructured: 'Ronen, S. et al. Links that speak: the global language network and
    its association with global fame. Proc. Natl Acad. Sci. USA 111, E5616–E5622 (2014).'
  volume: 111
  year: 2014
- author: M Ferron
  doi: 10.1177/1750698013490590
  first-page: 22
  journal-title: Mem. Stud.
  unstructured: 'Ferron, M. & Massa, P. Beyond the encyclopedia: collective memories
    in Wikipedia. Mem. Stud. 7, 22–45 (2014).'
  volume: 7
  year: 2014
- author: B Yucesoy
  doi: 10.1140/epjds/s13688-016-0079-z
  journal-title: EPJ Data Sci.
  unstructured: Yucesoy, B. & Barabási, A.-L. Untangling performance from success.
    EPJ Data Sci. 5, 17 (2016).
  volume: 5
  year: 2016
- doi: 10.7208/chicago/9780226774497.001.0001
  unstructured: Halbwachs, M. On Collective Memory (Univ. Chicago Press, Chicago,
    1992).
- author: S Fortunato
  doi: 10.1126/science.aao0185
  first-page: eaao0185
  journal-title: Science
  unstructured: Fortunato, S. et al. Science of science. Science 359, eaao0185 (2018).
  volume: 359
  year: 2018
- author: F Radicchi
  doi: 10.1073/pnas.0806977105
  first-page: 17268
  journal-title: Proc. Natl Acad. Sci. USA
  unstructured: 'Radicchi, F., Fortunato, S. & Castellano, C. Universality of citation
    distributions: toward an objective measure of scientific impact. Proc. Natl Acad.
    Sci. USA 105, 17268–17272 (2008).'
  volume: 105
  year: 2008
- author: T Kuhn
  first-page: 16926
  journal-title: Phys. Rev. X
  unstructured: Kuhn, T., Perc, M. & Helbing, D. Inheritance patterns in citation
    networks reveal scientific memes. Phys. Rev. X 4, 041036 (2014).
  volume: 4
  year: 2014
- author: R Sinatra
  doi: 10.1126/science.aaf5239
  first-page: aaf5239
  journal-title: Science
  unstructured: Sinatra, R., Wang, D., Deville, P., Song, C. & Barabasi, A.-L. Quantifying
    the evolution of individual scientific impact. Science 354, aaf5239 (2016).
  volume: 354
  year: 2016
- author: MM King
  first-page: 1
  journal-title: Socius
  unstructured: 'King, M. M., Bergstrom, C. T., Correll, S. J., Jacquet, J. & West,
    J. D. Men set their own cites high: gender and self-citation across fields and
    over time. Socius 3, 1–22 (2017).'
  volume: 3
  year: 2017
- doi: 10.31235/osf.io/jk63c
  unstructured: Chu, J. S. G. & Evans, J. A. Too many papers? Slowed canonical progress
    in large fields of science. Preprint at SocArXiv https://doi.org/10.31235/osf.io/jk63c
    (2018).
- author: SH Yook
  doi: 10.1103/PhysRevE.72.045105
  first-page: 19013
  journal-title: Phys. Rev. E
  unstructured: Yook, S.-H., Radicchi, F. & Meyer-Ortmanns, H. Self-similar scale-free
    networks and disassortativity. Phys. Rev. E 72, 045105 (2005).
  volume: 72
  year: 2005
- doi: 10.1007/978-3-319-45023-0_12
  unstructured: Mukherjee, S., Uzzi, B., Jones, B. F. & Stringer, M. in Knowledge
    and Networks (eds Glückler, J. et al.) 243–267 (Springer, Cham, 2017).
- author: HW Shen
  doi: 10.1073/pnas.1401992111
  first-page: 12325
  journal-title: Proc. Natl Acad. Sci. USA
  unstructured: Shen, H.-W. & Barabasi, A.-L. Collective credit allocation in science.
    Proc. Natl Acad. Sci. USA 111, 12325–12330 (2014).
  volume: 111
  year: 2014
- doi: 10.3386/w8498
  unstructured: 'Hall, B. H., Jaffe, A. B. & Trajtenberg, M. The NBER Patent Citation
    Data File: Lessons, Insights and Methodological Tools (National Bureau of Economic
    Research, 2001); www.nber.org/papers/w8498'
- author: AB Jaffe
  doi: 10.2307/2118401
  first-page: 577
  journal-title: Q. J. Econ.
  unstructured: Jaffe, A. B., Trajtenberg, M. & Henderson, R. Geographic localization
    of knowledge spillovers as evidenced by patent citations. Q. J. Econ. 108, 577–598
    (1993).
  volume: 108
  year: 1993
- unstructured: The Hot 100 Ranking Billboard https://www.billboard.com/charts/hot-100
    (2016).
- unstructured: Spotify web API Spotify https://developer.spotify.com/documentation/web-api/
    (2017).
- unstructured: Last.fm web services last.fm https://www.last.fm/api (2017).

In particular, focusing on the first-page values, we see

  first-page: e1602368
  first-page: 042309

The first is less problematic -- since it doesn't parse as a number, yq considers it a string. But for the second, yq commits to viewing it as a base-8 number, and gets confused by the final 9.
This is in part a problem with yq, and further back, its use of PyYAML
kislyuk/yq#152
Still, this is a point of potential confusion, and I'd support a generic recommendation that at least strings starting in 0 should be quoted.
Trying to work around this, it seems this should be checkable by doctor-key-type-keys, but unfortunately there's no way of configuring that for subobjects (while one can set first-page:int, one cannot write citations.first-page:int).

Activity

  1. alexfikl commented on Jan 8, 2026

    @alexfikl
    Collaborator

    Trying to work around this, it seems this should be checkable by doctor-key-type-keys, but unfortunately there's no way of configuring that for subobjects (while one can set first-page:int, one cannot write citations.first-page:int).

    That would definitely be a welcome addition. Shouldn't be too hard to add some parsing + recursion in there.

  2. hseg commented on Jan 8, 2026

    @hseg
    ContributorAuthor

    At least https://github.com/mikefarah/yq parses these files correctly, but it has its own issues (eg unfamiliarity vis-a-vis jq)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions