Verify if a string is a valid UTF-8 string? [SOLVED]

Message

Argolance · #1 Post by **Argolance** » Tue 05 Mar 2013, 16:40

Hello,
Don't know if it is specific to my NVIDIA graphic card, but running a script (report-video) that intend to get the product label, I get this:

Code: Select all

(gtkdialog:28206): Pango-WARNING **: Invalid UTF-8 string passed to pango_layout_set_text()

(gtkdialog:28206): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Erreur à la ligne 5, caractère 2 : Codage UTF-8 non valide dans le nom - « 
 oem: NVIDIA
product: BIOS-P/N@N3932:0 GW-CLK@\x84\u0003\x8a\u0002\xe8\u0003\xbc\u0002\xb0\u0004R\u0003

 » n'est pas valide

So, I would like to verify if the string is a valid UTF-8 string, then display, for example "unknown" if not, because this makes the script not to be displayed properly ("strange" characters and wrong text using markup) and consequently xerrs.log (bip!) not to be editable/readable.

Thank you.

Cordialement.

amigo · #2 Post by **amigo** » Tue 05 Mar 2013, 17:55

I found this:
https://github.com/zeevt/validate_utf8

and this:
http://lua-users.org/lists/lua-l/2011-01/msg00503.html

validate_utf8 seems to work here.

Argolance · #3 Post by **Argolance** » Tue 05 Mar 2013, 18:40

Thank you!
I found such very complicated stuffs too but it is probably very difficult to use inside a simple bash script (for me!). I thought of something simpler. I am wondering if it should not be more convenient to use a function with grep command to know if there is any error message inside xerrs.log file concerning this string, then, if yes and so on?

Cordialement.

don570 · #4 Post by **don570** » Tue 05 Mar 2013, 19:52

I would check if there is an error message

Code: Select all

report-video  2>/tmp/error$$

Barry Kauler should be warned about this.

Argolance · #5 Post by **Argolance** » Tue 05 Mar 2013, 22:30

Hello don570,

Barry Kauler should be warned about this.

I tried to let a message on bkhome but I forgot my password

... Please, could you do it?

Thanks.

Cordialement.

don570 · #6 Post by **don570** » Tue 05 Mar 2013, 22:42

I posted it on his blog for you.

You should run Tazoc's SYS-INFO to generate a report and post it
on this thread.

By the way I see a gtkdialog error in your image so I don't think
that Barry K's simple script to find the driver that loaded is causing
this problem. I believe he added in a gtkdialog script in recent
builds.

Perhaps you could download the source code of gtkdialog
and compile a version specifically for your puppy.
/usr/sbin/gtkdialog4 is the application and /usr/sbin/gtkdialog
is the link. Zigbert had problems with his
packages because gtkdialog wasn't installed correctly.

http://code.google.com/p/gtkdialog/down ... 8.3.tar.gz

.

Argolance · #7 Post by **Argolance** » Wed 06 Mar 2013, 09:57

Hello,

I posted it on his blog for you.

Thanks.

By the way I see a gtkdialog error in your image so I don't think
that Barry K's simple script to find the driver that loaded is causing
this problem. I believe he added in a gtkdialog script in recent
builds.

The first picture above shows the video-report GUI when the wrong UTF-8 line inside /tmp/video-report is deleted and replaced with "unknown", then saved as /tmp/report-video-test. Inside the original GUI, I replaced "report-video" with "report-video-test" for testing... and this works fine, including additional "text use-makup" and so on - for my own.

export RV="
<window title=\"$(gettext 'VIDEO REPORT')\" >
<vbox>
<vbox border-width=\"30\">
<pixmap><width>48</width><input file>/usr/share/pixmaps/report-video.png</input></pixmap>
<text use-markup=\"true\">
<label>\"$(cat /tmp/report-video-test)\"</label>
</text>
</vbox>
<hbox>
<button tooltip-text=\"$(gettext 'Edit /tmp/report-video file')\"><input file stock=\"gtk-edit\"></input>
<action>defaulttextviewer /tmp/report-video &</action></button>
<button><input file stock=\"gtk-close\"></input></button>
</hbox>
</vbox>
</window>"

$GTKDIALOG --program=RV

The second picture above shows what happens when report-video runs with the normal (wrong) /tmp/report-video.

When using the Lignthouse GUI (picture below), text window is blank.

Code: Select all

Video-Info 1.5.1 - mer. 6 mars 2013 on ToOpPy Precise 5.4.3 - Linux 3.2.29 i686
 
Chip description:
  0.0 VGA compatible controller: NVIDIA Corporation G94 [GeForce 9600 GT] (rev a1)  
  oem: NVIDIA
  product: BIOS-P/N@N3932:0 GW-CLK@è¼°R
 
X Server: Xorg  Driver used: vesa

X.Org version: 1.11.3
  dimensions:    3200x1200 pixels (956x354 millimeters)
  depth of root window:    24 planes 

   ...the above also recorded in /tmp/root/ as video-info,
and archived with xorg.conf and Xorg.0.log as video-info-full.gz

(gtkdialog4:18443): Gtk-CRITICAL **: gtk_text_buffer_emit_insert: assertion `g_utf8_validate (text, len, NULL)' failed

Cordialement.

Argolance · #8 Post by **Argolance** » Wed 06 Mar 2013, 13:45

Hello don570,
... Solved!
The reason why I got this UTF-8 issue: I was using/making enhancement from your "report-video 1.0" (now "report-video 1.1") which had something wrong somewhere...
Using the latest BK's version (from Puppy Precise 5.4.95) and building a GUI with, all works perfectly.
I visited bkhome thread about "report-video" and read the answer given to "our" post:

Posted on 6 Mar 2013, 10:15 by BarryK
Re report-video
No, that is not my 'report-video' script.

There are two other report-video PETs, sys-info by TaZoc and another by sc0ttman.

My Wary/Racy 5.5, and I think the latest Precise, only have my report-video, which is commandline only, does not use pango/gtk. The Forum thread does not identify which "report-video" is causing the problem.

By the way, I asked the authors to change the name of their programs so as not to be the same as my 'report-video'.

This is quite right and I personally will do it!

So, all my apologies to BK for having taken this feather out of its cap.

Cordialement.

Argolance · #9 Post by **Argolance** » Wed 06 Mar 2013, 16:53

Hello,
Here is a nth version of report-video named "2P Report Video"

pot file is in /usr/share/doc/nls/2P_report-video for bold text to be taken into account and some strings that are not translated in the original BK script...

Cordialement.

L18L · #10 Post by **L18L** » Wed 06 Mar 2013, 17:38

Hello Argolance,

you wrote:...
pot file is in /usr/share/doc/nls/2P_report-video for bold text to be taken into account and some strings that are not translated in the original BK script...

1,
Why do you share 2P_report-video.pot? Does momanager, the tool for translations with puppy not work with 2P_report-video

2,
Which are the strings that are not translated in the original BK script?

Cordialement.

Argolance · #11 Post by **Argolance** » Wed 06 Mar 2013, 18:20

Hello L18L,

1,
Why do you share 2P_report-video.pot? Does momanager, the tool for translations with puppy not work with 2P_report-video

I suppose yes, but I admit I never used momanager: Just little command lines and poeditor... Sorry!

2,
Which are the strings that are not translated in the original BK script?

just one: Line 43 "Requested by /etc/X11/xorg.conf:" and the 2 words "millimeters" and "planes" that my GUI translate using gettext too, directly inside /tmp/report-video and /tmp/report-video_ (these 2 files are different because one is displayed inside the window GUI with bold characters - text use-markup - and the other without, in the default text viewer).

Cordialement.

L18L · #12 Post by **L18L** » Thu 07 Mar 2013, 10:34

Argolance wrote:...
2,
Which are the strings that are not translated in the original BK script?
just one: Line 43 "Requested by /etc/X11/xorg.conf:" and the 2 words "millimeters" and "planes" that my GUI translate using gettext too, directly inside /tmp/report-video and /tmp/report-video_ (these 2 files are different because one is displayed inside the window GUI with bold characters - text use-markup - and the other without, in the default text viewer)....

Thanks, now I see too:
-Requested by /etc/X11/xorg.conf:
-millimeters
-planes
and
pixels

with / without <b>

why not change from

echo "$(gettext '<b>VIDEO REPORT:</b>') ${DISTRO_NAME}, $(gettext 'version') ${DISTRO_VERSION}"...

to

echo "<b>$(gettext VIDEO REPORT:')</b> ${DISTRO_NAME}, $(gettext 'version') ${DISTRO_VERSION}" ....

?

Argolance · #13 Post by **Argolance** » Thu 07 Mar 2013, 12:06

Hello L18L,

why not change from
Quote:

echo "$(gettext '<b>VIDEO REPORT:</b>') ${DISTRO_NAME}, $(gettext 'version') ${DISTRO_VERSION}"...

to
Quote:
echo "<b>$(gettext VIDEO REPORT:')</b> ${DISTRO_NAME}, $(gettext 'version') ${DISTRO_VERSION}" ....

I got into this habit because, in most of the cases I had to handle, there were several marks, for example:

echo "$(gettext '<b><span size='large'>VIDEO REPORT:</span></b>is a <i>GUI</i> for <u><i>BK</i> command line script</u>')

and I noticed it is simpler this way than repeat $(gettext '') for each marked word.

But in the case of "report-video", I admit it would be better to make otherwise.

-Requested by /etc/X11/xorg.conf:
-millimeters
-planes
and
pixels

... "forgot" this one, because it is the same word in most of the languages. But maybe you are right, we should not forget the "others", even if they are a minority...

EDIT: I was wrong, I would have say for most of the European languages
بكس Arabian
пик

L18L · #14 Post by **L18L** » Thu 07 Mar 2013, 15:13

Argolance wrote:...I would have say for most the European languages.

So you have been thinking at our Greeks too!

-------
http://forum.slitaz.org/topic/i18n-rules-tips-tricks

Argolance · #15 Post by **Argolance** » Thu 07 Mar 2013, 17:15

Greeks, Chinese (not exactly a "minority"), Bretons....

don570 · #16 Post by **don570** » Sat 09 Mar 2013, 19:47

It's possible to do stream editing (SED) on one line by using the semi-colon.
It might speed up the script as well.

Code: Select all


sed -i "s%<b>%%g;" /tmp/report-video_
sed -i "s%</b>%%g" /tmp/report-video_

sed -i "s%<b>%%g;s%</b>%%g" /tmp/report-video_

Argolance · #17 Post by **Argolance** » Sat 09 Mar 2013, 21:14

Hello don570,
Yes, it is much better! This is the way we learn, little by little... Thank you! I will have to revise most of the scripts I made, because since the first one, I learnt many small tricks like this one: for sure, they will be less heavy and faster!

Cordialement.

Argolance · #18 Post by **Argolance** » Sat 16 Mar 2013, 15:33

Bonjour,
Following the code lines above, I replaced:

Code: Select all

func () {
sed -i "s%<b>%%g" /tmp/report-video_
sed -i "s%</b>%%g" /tmp/report-video_
sed -i "s%millimeters%$(gettext 'millimeters')%g" /tmp/report-video_
sed -i "s%planes%$(gettext 'planes')%g" /tmp/report-video_
sed -i "s%pixels%$(gettext 'pixels')%g" /tmp/report-video_
}
export -f func

sed -i "s%millimeters%$(gettext 'millimeters')%g" /tmp/report-video
sed -i "s%planes%$(gettext 'planes')%g" /tmp/report-video
sed -i "s%pixels%$(gettext 'pixels')%g" /tmp/report-video

with:

Code: Select all

func () {
sed -i "s%<b>%%g";"s%</b>%%g";"s%millimeters%$(gettext 'millimeters')%g";"s%planes%$(gettext 'planes')%g";"s%pixels%$(gettext 'pixels')%g" /tmp/report-video_
}
export -f func

sed -i "s%millimeters%$(gettext 'millimeters')%g";"s%planes%$(gettext 'planes')%g";"s%pixels%$(gettext 'pixels')%g" /tmp/report-video

And I got:

sed: no input files
/usr/sbin/2P_report-video: line 127: s%planes%niveaux%g: command not found
/usr/sbin/2P_report-video: line 127: s%pixels%pixels%g: command not found
sed: no input files
sh: line 1: s%</b>%%g: No such file or directory
sh: line 2: s%millimeters%millimètres%g: command not found
sh: line 3: s%planes%niveaux%g: command not found
sh: line 4: s%pixels%pixels%g: command not found
EXIT="OK"
Script completed hit RETURN to close window.

Cordialement.

L18L · #19 Post by **L18L** » Sat 16 Mar 2013, 15:42

Argolance wrote:...Following the code lines above, I replaced:..

Pas "tres" exactement ...

amigo · #20 Post by **amigo** » Sat 16 Mar 2013, 15:44

The syntax is like this:

Code: Select all

sed -i -e '....' -e '...' filename

or:

Code: Select all

sed -i -e '....' \
-e '...' \
-e '...' filename

(old)Puppy Linux Discussion Forum

(old)Puppy Linux Discussion Forum

Verify if a string is a valid UTF-8 string? [SOLVED]

Verify if a string is a valid UTF-8 string? [SOLVED]

nth version of report-video named "2P Report Video"