UTF-8 symbols display broken on my Ubuntu 16.04

[d9k]

Hi!
I love diakonos but when I open utf8 file with some russian characters it shows them as strange sequence of letters and signs, for example: the word файл is shown as M-Q~DM-PM-0M-PM-9M-PM-; in diakonos.

Are there any options in diakonos config (F12) to fix this? Russian letters in files and keyboard input are shown properly in bash, vim and nano editor.

I use Diakonos v. 0. 9. 6 (Feb 19, 2016) from Kubuntu 16.04 standart aptitude repository package.

[Pistos]

@d9k Thanks for trying Diakonos! Please provide a sample file for download somewhere. When I copy and paste ф into diakonos, and save it and re-open the file, it works for me.

[pepa65]

I can't find diakonos in the 16.04 repos myself to check, but the most comprehensive text file for UTF-8 testing is: http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt

Trying to install from this github repo, doing:

sudo apt install ruby-dev
sudo gem install curses
sudo ruby install.rb

But diakonos doesn't work...

[Pistos]

@pepa65 Diakonos handles that UTF-8-demo.txt file very well, actually. (I checked [again] just now.) Your terminal font needs to support all the characters, though. (My choice of font, Terminus, for example, doesn't have the accented Greek letters.)

[pepa65]

It's only the zero-width spacing (for the Thai script) that isn't working then for the UTF-8-demo.txt file.

Any idea what can be done to build it on Ubuntu 16.04?

[Pistos]

Any idea what can be done to build it on Ubuntu 16.04?

@pepa65 I rarely use Ubuntu, but bear in mind that the curses library was removed from Ruby in, I think, Ruby 2.1. They made it a separate gem, so you'll need to gem install curses if you are using a Ruby version higher than 2.0. Try that to see if that is the only issue you have.

[d9k]

Thanks for providing help, @Pistos, @pepa65! 😌
First lines of UTF-8-demo.txt in my diakonos:

UTF-8 encoded sample plain-text file
M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@M->M-b~@

Markus Kuhn [M-K~HmaM-JM-3kM-J~Js kuM-K~Pn] <http://www.cl.cam.ac.uk/~mgk25/> M-b~@~T 2002-07-25 CC BY

Maybe there are some codepage settings in diakonos config?
Maybe diakonos checks some environment variables containing codepage information on startup?
nano and vim show UTF-8-demo.txt fine

[d9k]

I can try to execute some ruby code if you give, maybe that's problem with ruby libraries installed.

[pepa65]

I tried gem install curses (see 3rd comment on this Issue), but that didn't help...

[Pistos]

@d9k: Is there anything relevant-looking in ~/.diakonos/diakonos.log or ~/.diakonos/debug.log? I've never seen anything like what you're describing, and it's hard to websearch for it.

[Pistos]

@d9k Tell me more about what Linux distro you're on, what desktop environment, what terminal program you're using. And if you can send the output of env to https://gist.github.com/ or something, I'd like to see it. Be sure to remove any potentially sensitive lines from that, first, though. (Maybe you have passwords or personal info in it.)

[Pistos]

@d9k Also, send the output of locale. And tell me anything else unusual about your system and setup that you think might be relevant. Ruby version, too.

[d9k]

@Pistos, thanks for your investigation!

I use Kubuntu 16.04 upgraded from 14.04 with

sudo do-release-upgrade -d
sudo apt-get dist-upgrade

% locale
LANG=en_US.UTF-8
LANGUAGE=en.en.en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

env dumped from xfce4-terminal emulator:

% env
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_CONFIG_DIRS=/etc/xdg/xdg-plasma:/etc/xdg:/usr/share/kubuntu-default-settings/kf5-settings
LC_TELEPHONE=ru_RU.UTF-8
LC_TIME=ru_RU.UTF-8
LANG=en_US.UTF-8
PWD=/home/d9k
XDG_VTNR=7
KDE_SESSION_VERSION=5
SHLVL=2
MANDATORY_PATH=/usr/share/gconf/plasma.mandatory.path
LOGNAME=d9k
LANGUAGE=en_US
LC_NAME=ru_RU.UTF-8
KDED_STARTED_BY_KDEINIT=1
XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/d9k
QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1
KDE_SESSION_UID=1000
COLORTERM=xfce4-terminal
XDG_SESSION_ID=c2
DISPLAY=:0.0
DESKTOP_SESSION=plasma
XDG_RUNTIME_DIR=/run/user/1000
DEFAULTS_PATH=/usr/share/gconf/plasma.default.path
GDMSESSION=plasma
PAM_KWALLET5_LOGIN=/tmp/kwallet5_d9k.socket
XDG_SESSION_DESKTOP=plasma
LC_NUMERIC=ru_RU.UTF-8
LC_IDENTIFICATION=ru_RU.UTF-8
LC_PAPER=ru_RU.UTF-8
LC_MONETARY=ru_RU.UTF-8
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-qwpHvyECy9,guid=eec3f42fb332e6c89a4e9a2d5734f19b
LC_MEASUREMENT=ru_RU.UTF-8
XAUTHORITY=/tmp/xauth-1000-_0
XDG_DATA_DIRS=/usr/share/plasma:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
_=/usr/bin/env
WINDOWID=58720260
PAM_KWALLET_LOGIN=/tmp/kwallet_d9k.socket
XDG_CURRENT_DESKTOP=KDE
QT_ACCESSIBILITY=1
QT_IM_MODULE=compose
SHELL=/usr/bin/zsh
XDG_SESSION_TYPE=x11
GDM_LANG=en_US
XCURSOR_THEME=oxy-neon
SSH_AGENT_PID=2537
GTK_MODULES=gail:atk-bridge
SESSION_MANAGER=local/phenomIIX4:@/tmp/.ICE-unix/2627,unix/phenomIIX4:/tmp/.ICE-unix/2627
SSH_AUTH_SOCK=/tmp/ssh-NxVOKn005Jhs/agent.2477
PATH=/home/d9k/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
XCURSOR_SIZE=0
TERM=xterm
USER=d9k
HOME=/home/d9k
KDE_FULL_SESSION=true
LC_ADDRESS=ru_RU.UTF-8
GS_LIB=/home/d9k/.fonts
GPG_AGENT_INFO=/home/d9k/.gnupg/S.gpg-agent:0:1
XDG_SEAT=seat0
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
OLDPWD=/home/d9k

% which ruby
/usr/bin/ruby
% ruby --version
ruby 2.3.0p0 (2015-12-25) [x86_64-linux-gnu]

~/.diakonos/diakonos.log and ~/.diakonos/debug.log are both empty.

[pepa65]

That explains it. There is no diakonos package for 16.04, but you somehow still have the old diakonos from 14.04. It won't work well with mismatch in libraries.

[Pistos]

@d9k What version do you get when you type diakonos --version?

Are you able to paste ф into a file in Diakonos, then save it, then reopen it in Diakonos? What do you see?

[d9k]

@Pistos when I try to paste ф to diakonos it pastes this letter to text as M-Q~D sequence, but when I save file with diakonos and open it in another editor ф is shown.