Hi,
I was having a unicode issue with a file - but at the time I didn't realize it. I looked at the file in TexPad and it seemed OK nothing out of the ordinary. I then looked at it in another editor and saw a "null" after each character.
Why didn't TextPad show me this? It would of been helpful.
Thanks,
Lee G.
Unicode Double-Byte character
Moderators: AmigoJack, bbadmin, helios, MudGuard
-
ben_josephs
- Posts: 2464
- Joined: Sun Mar 02, 2003 9:22 pm
The answer is in your title.
Your file is in Unicode, a 21-bit character encoding, and it's in UTF-16 format: each character whose encoding fits into 16 bits occupies two bytes in the file (and each character that doesn't fit into 16 bits is represented as a pair of 16-bit "surrogate" characters).
For all characters in the ASCII (7-bit) character set and many of those in the ISO 8859 series of (8-bit) character sets, the Unicode encoding fits into eight bits; so the top byte of each of these is zero. These are the nulls you are seeing.
TextPad is correctly interpreting and displaying each pair of bytes as a single character. Your other editor is failing to recognise the file's encoding, and is treating it as a single-byte character file, which it isn't.
If you want TextPad to display your file as a sequence of bytes instead of a sequence of characters, open it in File format: Binary.
Your file is in Unicode, a 21-bit character encoding, and it's in UTF-16 format: each character whose encoding fits into 16 bits occupies two bytes in the file (and each character that doesn't fit into 16 bits is represented as a pair of 16-bit "surrogate" characters).
For all characters in the ASCII (7-bit) character set and many of those in the ISO 8859 series of (8-bit) character sets, the Unicode encoding fits into eight bits; so the top byte of each of these is zero. These are the nulls you are seeing.
TextPad is correctly interpreting and displaying each pair of bytes as a single character. Your other editor is failing to recognise the file's encoding, and is treating it as a single-byte character file, which it isn't.
If you want TextPad to display your file as a sequence of bytes instead of a sequence of characters, open it in File format: Binary.
-
leegold
- Posts: 65
- Joined: Sat May 03, 2003 1:16 am
- Location: USA
The binary form you're talking about is HEX?
Not like what editors were showing me. They were showing the readable character and showing the word "null" in a boxed-in font next to the readable character.
DOS box output showed the null as a space.
I would of prefered the option in textpad to show a null symbol as well as reading it correctly.
I'll have to play with UTF-16 and see if TextPad can do this...I could be wrong:^)
Thanks
Not like what editors were showing me. They were showing the readable character and showing the word "null" in a boxed-in font next to the readable character.
DOS box output showed the null as a space.
I would of prefered the option in textpad to show a null symbol as well as reading it correctly.
I'll have to play with UTF-16 and see if TextPad can do this...I could be wrong:^)
Thanks
-
ben_josephs
- Posts: 2464
- Joined: Sun Mar 02, 2003 9:22 pm
Your title indicates that your file consists of 16-bit characters. The editors you refer to are not displaying those 16-bit characters. They are displaying the bytes in the file as 8-bit characters. So they are representing each single character in the file as two characters on the screen. In this file, the sequence of bytes 41 00 is not the pair of characters 'A' <NUL>; it is the single Unicode-encoded character U+0041: 'A'. And the sequence 52 01 is not the pair of characters 'R' <SOH>; it is the single Unicode-encoded character U+0152: 'Å’'.
TextPad's binary display will show you the contents of your file as a sequence of bytes, both in hexadecimal and, where possible, as displayable 8-bit characters.
Does this not serve your purposes? What are you trying to do?
TextPad's binary display will show you the contents of your file as a sequence of bytes, both in hexadecimal and, where possible, as displayable 8-bit characters.
Does this not serve your purposes? What are you trying to do?