UTF-8 symbols display broken on my Ubuntu 16.04

[d9k]

Hi!
I love diakonos but when I open utf8 file with some russian characters it shows them as strange sequence of letters and signs, for example: the word файл is shown as M-Q~DM-PM-0M-PM-9M-PM-; in diakonos.

Are there any options in diakonos config (F12) to fix this? Russian letters in files and keyboard input are shown properly in bash, vim and nano editor.

I use Diakonos v. 0. 9. 6 (Feb 19, 2016) from Kubuntu 16.04 standart aptitude repository package.

[Pistos]

@d9k Thanks for trying Diakonos! Please provide a sample file for download somewhere. When I copy and paste ф into diakonos, and save it and re-open the file, it works for me.

[pepa65]

I can't find diakonos in the 16.04 repos myself to check, but the most comprehensive text file for UTF-8 testing is: http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt

Trying to install from this github repo, doing:

sudo apt install ruby-dev
sudo gem install curses
sudo ruby install.rb

But diakonos doesn't work...

[Pistos]

@pepa65 Diakonos handles that UTF-8-demo.txt file very well, actually. (I checked [again] just now.) Your terminal font needs to support all the characters, though. (My choice of font, Terminus, for example, doesn't have the accented Greek letters.)

[pepa65]

It's only the zero-width spacing (for the Thai script) that isn't working then for the UTF-8-demo.txt file.

Any idea what can be done to build it on Ubuntu 16.04?

[Pistos]

Any idea what can be done to build it on Ubuntu 16.04?

@pepa65 I rarely use Ubuntu, but bear in mind that the curses library was removed from Ruby in, I think, Ruby 2.1. They made it a separate gem, so you'll need to gem install curses if you are using a Ruby version higher than 2.0. Try that to see if that is the only issue you have.

[d9k]

Thanks for providing help, @Pistos, @pepa65! 😌
First lines of UTF-8-demo.txt in my diakonos:

UTF-8 encoded sample plain-text file
M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@

Markus Kuhn [M-K~HmaM-JM-3kM-J~Js kuM-K~Pn] <http://www.cl.cam.ac.uk/~mgk25/> M-b~@~T 2002-07-25 CC BY

Maybe there are some codepage settings in diakonos config?
Maybe diakonos checks some environment variables containing codepage information on startup?
nano and vim show UTF-8-demo.txt fine

[d9k]

I can try to execute some ruby code if you give, maybe that's problem with ruby libraries installed.

[pepa65]

I tried gem install curses (see 3rd comment on this Issue), but that didn't help...

[Pistos]

@d9k: Is there anything relevant-looking in ~/.diakonos/diakonos.log or ~/.diakonos/debug.log? I've never seen anything like what you're describing, and it's hard to websearch for it.

[Pistos]

@d9k Tell me more about what Linux distro you're on, what desktop environment, what terminal program you're using. And if you can send the output of env to https://gist.github.com/ or something, I'd like to see it. Be sure to remove any potentially sensitive lines from that, first, though. (Maybe you have passwords or personal info in it.)

[Pistos]

@d9k Also, send the output of locale. And tell me anything else unusual about your system and setup that you think might be relevant. Ruby version, too.

[d9k]

@Pistos, thanks for your investigation!

I use Kubuntu 16.04 upgraded from 14.04 with

sudo do-release-upgrade -d
sudo apt-get dist-upgrade

% locale
LANG=en_US.UTF-8
LANGUAGE=en.en.en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

env dumped from xfce4-terminal emulator:

% env
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_CONFIG_DIRS=/etc/xdg/xdg-plasma:/etc/xdg:/usr/share/kubuntu-default-settings/kf5-settings
LC_TELEPHONE=ru_RU.UTF-8
LC_TIME=ru_RU.UTF-8
LANG=en_US.UTF-8
PWD=/home/d9k
XDG_VTNR=7
KDE_SESSION_VERSION=5
SHLVL=2
MANDATORY_PATH=/usr/share/gconf/plasma.mandatory.path
LOGNAME=d9k
LANGUAGE=en_US
LC_NAME=ru_RU.UTF-8
KDED_STARTED_BY_KDEINIT=1
XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/d9k
QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1
KDE_SESSION_UID=1000
COLORTERM=xfce4-terminal
XDG_SESSION_ID=c2
DISPLAY=:0.0
DESKTOP_SESSION=plasma
XDG_RUNTIME_DIR=/run/user/1000
DEFAULTS_PATH=/usr/share/gconf/plasma.default.path
GDMSESSION=plasma
PAM_KWALLET5_LOGIN=/tmp/kwallet5_d9k.socket
XDG_SESSION_DESKTOP=plasma
LC_NUMERIC=ru_RU.UTF-8
LC_IDENTIFICATION=ru_RU.UTF-8
LC_PAPER=ru_RU.UTF-8
LC_MONETARY=ru_RU.UTF-8
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-qwpHvyECy9,guid=eec3f42fb332e6c89a4e9a2d5734f19b
LC_MEASUREMENT=ru_RU.UTF-8
XAUTHORITY=/tmp/xauth-1000-_0
XDG_DATA_DIRS=/usr/share/plasma:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
_=/usr/bin/env
WINDOWID=58720260
PAM_KWALLET_LOGIN=/tmp/kwallet_d9k.socket
XDG_CURRENT_DESKTOP=KDE
QT_ACCESSIBILITY=1
QT_IM_MODULE=compose
SHELL=/usr/bin/zsh
XDG_SESSION_TYPE=x11
GDM_LANG=en_US
XCURSOR_THEME=oxy-neon
SSH_AGENT_PID=2537
GTK_MODULES=gail:atk-bridge
SESSION_MANAGER=local/phenomIIX4:@/tmp/.ICE-unix/2627,unix/phenomIIX4:/tmp/.ICE-unix/2627
SSH_AUTH_SOCK=/tmp/ssh-NxVOKn005Jhs/agent.2477
PATH=/home/d9k/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
XCURSOR_SIZE=0
TERM=xterm
USER=d9k
HOME=/home/d9k
KDE_FULL_SESSION=true
LC_ADDRESS=ru_RU.UTF-8
GS_LIB=/home/d9k/.fonts
GPG_AGENT_INFO=/home/d9k/.gnupg/S.gpg-agent:0:1
XDG_SEAT=seat0
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
OLDPWD=/home/d9k

% which ruby
/usr/bin/ruby
% ruby --version
ruby 2.3.0p0 (2015-12-25) [x86_64-linux-gnu]

~/.diakonos/diakonos.log and ~/.diakonos/debug.log are both empty.

[pepa65]

That explains it. There is no diakonos package for 16.04, but you somehow still have the old diakonos from 14.04. It won't work well with mismatch in libraries.

[Pistos]

@d9k What version do you get when you type diakonos --version?

Are you able to paste ф into a file in Diakonos, then save it, then reopen it in Diakonos? What do you see?

[d9k]

@Pistos when I try to paste ф to diakonos it pastes this letter to text as M-Q~D sequence, but when I save file with diakonos and open it in another editor ф is shown.

[d9k]

(mine last message contained a mistake and was corrected)

[Pistos]

@d9k Please paste this string into Diakonos and then tell me what you see:

АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ

[d9k]

@d9k What version do you get when you type diakonos --version?

I wrote the version info in the first message:

I use Diakonos v. 0. 9. 6 (Feb 19, 2016)

.

@d9k Please paste this string into Diakonos and then tell me what you see:
АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ

I see

M-P~PM-P~QM-P~RM-P~SM-P~TM-P~UM-P~VM-P~WM-P~XM-P~YM-P~ZM-P~[M-P~\M-P~]M-P~^M-P~_M-PM- M-PM-!M-PM-"M-PM-#M-PM-$M-PM-%M-PM-&M-PM-'M-PM-(M-PM-)M-PM-*M-PM-+M-PM

The string is too long to copy. I can't scroll it. (One russian letter is shown as ~5 symbols on screen but cursor works as if there was just 30 letters on screen, not ~150. When I delete one such invisible letter ~5 corresponding symbols dissapear from screen)

[Pistos]

@d9k Can you try from other terminals, like Konsole and xterm? Also try to open a file from the tty consoles (Ctrl-Alt-F1 to go to the first tty, Ctrl-Alt-F7 to go back to the first xorg session).

[pepa65]

UTF-8 output is NOT broken on Ubuntu, any version. Otherwise you would hear a whole lot more about it.

d9k, your install of diakonos is totally broken. You have diakonos left over from the Ubuntu 14.04 repos, and you have renewed all linked libraries etc.

There is no need to debug this situation any further. What is needed is a proper way yto get diakonos to run on Ubuntu 16.04.

[Pistos]

@d9k Also, try to run irb, the Interactive Ruby console. Then paste something into that, see if it exhibits the same problem.

[Pistos]

How are things with this issue?

[pepa65]

Well, an official package for diakonos on Ubuntu 16.04 still doesn't exist, and the package from 14.04 still doesn't work well with the 16.04 libraries (as can be expected).

[d9k]

@Pistos, I experience same issue on the fresh ubuntu 16.04 installation on another computer.
Nano editor and other work fine.

Also, try to run irb, the Interactive Ruby console. Then paste something into that, see if it exhibits the same problem.

No problem with irb

% irb --version
irb 0.9.6(09/06/30)
% ruby --version
ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
% sudo gem list
*** LOCAL GEMS ***
bigdecimal (1.2.8)
curses (1.0.2)
did_you_mean (1.0.0)
io-console (0.4.5)
json (1.8.3)
minitest (5.8.4)
ncursesw (1.4.9)
net-telnet (0.1.1)
power_assert (0.2.7)
psych (2.0.17)
rake (10.5.0)
rdoc (4.2.1)
test-unit (3.1.7)

Can you try to install diakonos on the ubuntu 16.04 inside VirtualBox and debug an issue?

[pepa65]

@d9k Can I ask from where you got the package? Standard Ubuntu 16.04 doesn't have a diakonos package.

[pepa65]

I just managed to build without issue on Ubuntu 16.04 following the instructions in the README.md
Russian text renders fine. Still issues with zero-width symbols in Thai of course.

[d9k]

@pepa65, I followed instructions from README.rdoc too.
It's strange you have no issues with russian text display.
As I said, I tried diakonos on the fresh ubuntu 16.04 installation on another computer.

[pepa65]

Does it build OK?

[d9k]

@pepa65, what do you mean? It works. But with bug described here.

[Pistos]

@d9k Sorry that I am late in replying to all this.

I noticed in your environment that your env and your locale don't match. Your locale says en_US, but you have many ru_RU lines in your env. I don't know if that's okay, or if it might be causing you trouble. For example, my locale and env are in sync. Here is an Ubuntu wiki page about locales. Check your output of locale -a, and also read about using locale-gen to make sure your locale setup is right. Maybe ask in the Ubuntu community if your locale is set up correctly.

[Pistos]

@d9k This is what I meant when I said "try it in irb":

% irb
2.3.1 :001 > puts "АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ"
АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ
 => nil 

Can you try that, and copy-paste the irb output to us?