Fix line breaks / formatting on export --append

@alexfikl

Followup for #1036

Context: #1036 (comment), #1036 (comment), #1036 (review)

To summarize: on export --append, the resulting outfile is badly formatted in most cases, specifically:

bibtex: no newline at end-of-file, so appended text doesn't have a proper linebreak (see Export (over)write outfile by default, add --append and --batch flags #1036 (comment))
csl: no newline (as above)
json:
1. no newline (as above)
2. instead of extending existing list, starts a new one (see Export (over)write outfile by default, add --append and --batch flags #1036 (comment))
yaml: doesn't insert --- in between documents on append
typst: can't export multiple docs (?)
- otherwise quite sane, although I suspect it would look better with a blank line in between on append

@alexfikl:

Maybe by extending the exporter interface to do most of the work? I can't really see a way around it..

This sounds like it makes sense. I can probably work on this.

yaml: doesn't insert --- in between documents on append

This is a proper bug, since the result is incorrect (squishes two documents together). Same for the json exporter -- the result is not valid JSON.

typst: can't export multiple docs (?)

You mean our exporter? The format itself can definitely support multiple documents.

This sounds like it makes sense. I can probably work on this.

To expand a bit on this: to properly support json, it seems like we need to let the exporter do the file writing / appending. I'm inclined to suggest reworking the exporter plugins to be a little class similar to papis.importer.Importer that can do whatever it needs to. What did you have in mind?

typst: can't export multiple docs (?)

You mean our exporter? The format itself can definitely support multiple documents.

Yes, does this work for you?

papis export -f typst --all

I get this (--log DEBUG):

[DEBUG] utils: Indexing folders in '/home/tuure/files/docs/papis'.
[DEBUG] utils: Retrieved 56 valid folders.
[DEBUG] utils: Finished in 36.2 ms.
[DEBUG] database.cache: Computing 'papis_id' for each document.
[DEBUG] database.cache: Loaded 56 documents.
[DEBUG] database.cache: Querying database for '.'.
[ERROR] commands.export: Failed to load 'typst' exporter. Cannot export to this format.
  ┆ Traceback (most recent call last):
  ┆   File "/usr/lib/python3.13/site-packages/papis/commands/export.py", line 88, in run
  ┆     .plugin(document for document in documents))
  ┆      ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ┆   File "/usr/lib/python3.13/site-packages/papis/hayagriva.py", line 229, in exporter
  ┆     create_reference(doc): to_hayagriva(doc) for doc in documents
  ┆                            ~~~~~~~~~~~~^^^^^
  ┆   File "/usr/lib/python3.13/site-packages/papis/hayagriva.py", line 169, in to_hayagriva
  ┆     parent_known_keys = HAYAGRIVA_TYPE_PARENT_KEYS[htype]
  ┆                         ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  ┆ KeyError: ''

Also happens with any 2 documents. But a single document works.

To expand a bit on this: to properly support json, it seems like we need to let the exporter do the file writing / appending. I'm inclined to suggest reworking the exporter plugins to be a little class similar to papis.importer.Importer that can do whatever it needs to. What did you have in mind?

Yeah something along those lines, will have to look into it/experiment to find out more details :)

Also happens with any 2 documents. But a single document works.

This should be fixed by #1035. If you can test that, it would be great! 😁 It works for me for documents that are e.g. only article type.

Oh, that's completely right. I was unlucky with my testing in that at least one doc always just happened to not be article. I'll test your fix now.

Edit: and even that whenever I tested a single document, it was article. I am a master at confusing myself 😄

Do you agree that it would look better with an empty line between like:

foo1
  bar1: baz1
  qux1: grault1

foo2
  bar2: baz2
  qux2: grault2

instead of

foo1
  bar1: baz1
  qux1: grault1
foo2
  bar2: baz2
  qux2: grault2

?

For the solution I had a high-level thought:

Maybe it's possible that we retain the generic export func that writes the file, and the exporter plugin only has to give one piece of information: what is the separator between two distinct documents?

Edit: right, it's not that simple in the json case though 🤔

Do you agree that it would look better with an empty line between like:

Can't say I feel strongly either way, but since everybody else gets a new line, might as well? It's just a YAML file in the end.

Edit: right, it's not that simple in the json case though

Yeah, the JSON exporter really makes this more complicated than it needs to be :\ I don't know.. maybe we can just have the exporter also return the separator and if it's None just say we can't append?

EDIT: Actually, maybe it's better to bite the bullet. I also have a CSV exporter that wouldn't fit in the "just give me a separator" bucket very well (at least assuming it also has headers?).

EDIT: Actually, maybe it's better to bite the bullet. I also have a CSV exporter that wouldn't fit in the "just give me a separator" bucket very well (at least assuming it also has headers?).

Yeah thanks for that context, let's be robust with it

For brainstorming's sake, maybe it's possible to reconceptualize appending as

taking the set of documents that are currently in out.xyz (as a dict or something)
adding to that set the docs to be appended
overwriting with the complete set

This however has the problem that if user has hand-written something in the outfile, would be undone

For brainstorming's sake, maybe it's possible to reconceptualize appending as
1. taking the set of documents that are currently in `out.xyz` (as a `dict` or something)

2. adding to that set the docs to be appended

3. overwriting with the complete set
This however has the problem that if user has hand-written something in the outfile, would be undone

I kinda like this, at least if it can be done somewhat reasonably simply :). I think hand-written entries should be extremely rare (I mean if you're using Papis, might as well add that doc to the library). But maybe I'm missing some use case... (EDIT: and at least many such cases could be detected and the operation aborted)

Yeah, that approach would also, kinda by coincidence, solve #938 (comment):

This also raises the issue of what to do on --append, since ensuring no duplicates means we need to read the existing file in, check it, possibly fix it, append the new documents, and write it back out.

Heavily considering.

Navigation Menu

Repository navigation

Fix line breaks / formatting on `export --append` #1038

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Fix line breaks / formatting on export --append #1038

Description

Activity

alexfikl commented on Jul 7, 2025

tuurep commented on Jul 7, 2025

alexfikl commented on Jul 7, 2025

tuurep commented on Jul 7, 2025

tuurep commented on Jul 7, 2025

alexfikl commented on Jul 7, 2025

tuurep commented on Jul 7, 2025

tuurep commented on Jul 7, 2025

jghauser commented on Jul 8, 2025

tuurep commented on Jul 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Fix line breaks / formatting on `export --append` #1038