Skip to content

Find a way to deal with UTF-8 with old awk versions #24

Description

@phyver

The script box.awk displays a frame around a text file. Some versions of awk (for example the one on macOS or BSD) do not return the correct length on UTF-8 string:

$ awk 'BEGIN {print length("hééé")}'
4
$ mawk 'BEGIN {print length("hééé")}' 
7

When the text file contains non-ASCII characters, the right border isn't aligned properly. One way to deal with that could be to try using iconv (POSIX, but not installed everywhere by default) : UTF-8 -> ISO-something -> box -> UTF-8.

Doing that would make it possible to completely remove the box8.py script.

A similar thing could be used when the detected locale is not UTF-8.

Activity

  1. phyver commented on May 30, 2021

    @phyver
    OwnerAuthor

    9d05c95 should correct that:
    if a version of awk doesn't get the length of UTF-8 right, and if iconv is found, box.sh converts the text to ISO-8859-15 (an 8 bit encoding of european characters), pass it to box.awk, and pass the result to iconv again to get back UTF-8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions