Consider the following:
$ cat -A input.txt
Hello^IWorld$
newline$Here, cat -A takes actual newlines and tabs, i.e. the real characters, and converts them to representations.
Is there a shell way or command-line application in Ubuntu repositories that would allow taking representations of unprintable characters and output the real values ?
In a sense, I'm asking whether there's something analogous to $'Hello\tWorld\nnewline\n', except instead of C-quoted strings, I want use "shell-quoted" strings.
1 Answer
Well, Python to the rescue!
Check out this one-liner, which reads from STDIN and prints to STDOUT, handling all possible "caret escapes"/"C0 codes" (like ^I) and line-end indicators ($):
python3 -c 'import sys,re;print(re.sub(r"\^([A-Z?@[\\\]^_])",lambda m:chr((ord(m.group(1))-64)&127),sys.stdin.read().replace("$\n","\n")))'Actually it's both compatible with python (2) and python3. Here's a longer, more readable version doing basically the same:
#!/usr/bin/env python3
import sys, re
# read everything from stdin and remove line-end indicators
s = sys.stdin.read().replace("$\n", "\n"))
# replace caret escapes like ^I or ^M and output to stdout
print(re.sub(r"\^([A-Z?@[\\\]^_])", lambda m: chr((ord(m.group(1)) - 64) & 127), s)So, first we remove the line-end indicators $.
Second we use the regular expression pattern \^([A-Z?@[\\\]^_]) to find all valid characters following carets and replace both with the correct unescaped character, according to Wikipedia on Caret notation and C0 control codes. Note how only capital letters A-Z or one of ?@[\]^_ have a special meaning.
Now to unescape such a C0 code, we take the position in the alphabet of the character succeeding the caret (found in m.group(1)), e.g. "A" is 1, "B" is 2 and so on. This is equal to its ASCII value minus the ASCII code of "A" plus one, which makes up the -64, which also explains e.g. "@" (ASCII 64) being 0 or "[" (ASCII 91) being ESC (ASCII 27). We do a binary AND operation on this number with 127 to only consider the first 7 bits of information, so that e.g. "?" (ASCII 63 == 64-1) wraps around to 127, representing the DEL character.
Finally after all these highly complex computations are done, we simply print the resulting string to STDOUT again.
