alonso acuña | 14 Apr 2012 08:09
Picon

problem with 8bit characters

Hello. I am using smsd from gnokii 0.6.31
I need to send characters like ñ and ó
When sending from gnokii this works only if I sent the -8 option on the command line.
When using smsd this does not work. I have tried many things changing the LANG to utf8 from my default iso8859-1. Also changed the mysql to utf8 from the detault latin1. With no luck.
What I get is garbled characters.
Finally my solution was to edit smsd/lowlevel.c and just before the call to gn_sms_send I added a line:

dt->sms->dcs.u.general.alphabet = GN_SMS_DCS_8bit;
d->status = gn_sms_send (dt, sm); 

So now it works. I don't know if this is a bug or somehow my error. I call smsd like this:

smsd -t 1 -u smsgw -p smsgw222 -d smsgw -c localhost -m mysql -b SM -f /var/log/reca/smsd1.log -e latin1 -0 

I have LANG=en_US.ISO-8859-1

Thanks

alonso
_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Daniele Forsi | 14 Apr 2012 14:00
Picon

Re: problem with 8bit characters

2012/4/14 alonso acuña:

> When sending from gnokii this works only if I sent the -8 option on the
> command line.

so you have been lucky because 8bit format means "binary data" and it
isn't supposed to work with the default SMS application of a phone, in
fact those messages should be silenty discarded by the phone if there
isn't a special application on the specific port referenced in the SMS

> I don't know if this is a bug or somehow my error.

you didn't give enough details to tell where the problem is

first provide the debug output[1] of gnokii --sendsms without -8 and
use the fake driver[2] since it produces less output; the default
gnokiirc has a [fake] section so you can use this line:

echo -n "ñó" | gnokii --phone fake --sendsms 0

whatever the encoding of your terminal, the SMS should be encoded in
the same way (UCS2 in this case since ó is not part of the Default GSM
Alphabet)

[1] http://wiki.gnokii.org/index.php/Bug_report
[2] http://wiki.gnokii.org/index.php/Fake_driver
--

-- 
Daniele Forsi
alonso acuña | 16 Apr 2012 05:04
Picon

Re: problem with 8bit characters

Hi. I have done some more tests and I'm starting to understand better the character set issues. Not sure if this is a bug but I find it is strange and since it was not documented (I think?) I spent several days figuring it out.

So basically the issue is that I have my PC with the en_US.ISO-8859-1 LANG as well as the other locale variables like LC_CTYPE etc.
echo -n "ño" | gnokii --sendsms  does not work in this scenario as the characters do not display correctly on the received message.
Then I made a file with "ño" encoded in UTF8. But cat file | gnokii --sendsms   stil does not work.
setting LANG=en_US.UTF8 stil does not work. What works is setting LC_CTYPE=en_US.UTF8.

So my env is 
LC_PAPER=es_CR.ISO-8859-1
LC_ADDRESS=es_CR.ISO-8859-1
LC_MONETARY=es_CR.ISO-8859-1
LC_SOURCED=1
LC_NUMERIC=es_CR.ISO-8859-1
LC_TELEPHONE=es_CR.ISO-8859-1
LC_MESSAGES=en_US.ISO-8859-1
LC_COLLATE=en_US.ISO-8859-1
LC_IDENTIFICATION=es_CR.ISO-8859-1
LANG=en_US.ISO-8859-1
LC_MEASUREMENT=es_CR.ISO-8859-1
LANGUAGE=en_US.ISO-8859-1:en_US:en
LC_TIME=en_US.ISO-8859-1
LC_NAME=es_CR.ISO-8859-1
LC_CTYPE=en_US.UTF8

the file is UTF8 encoded and cat file | gnokii --sendsms  works properly.
If I encode the file in ISO-8859-1 then I get an empty message.
If I encode the file in ISO-8859-1 and leave all the locale variables with ISO-8859-1 characters do not display properly.

So in summary the instructions for me to be able to send special characters are: put the content utf8 encoded in a file, set LC_CTYPE=en_US.UTF8 and send with cat file | gnokii . Did I miss this on the documentation? I am doing something weird? Is there a bug somewhere?

Thanks.


On Sat, Apr 14, 2012 at 6:00 AM, Daniele Forsi <dforsi <at> gmail.com> wrote:
2012/4/14 alonso acuña:

> When sending from gnokii this works only if I sent the -8 option on the
> command line.

so you have been lucky because 8bit format means "binary data" and it
isn't supposed to work with the default SMS application of a phone, in
fact those messages should be silenty discarded by the phone if there
isn't a special application on the specific port referenced in the SMS

> I don't know if this is a bug or somehow my error.

you didn't give enough details to tell where the problem is

first provide the debug output[1] of gnokii --sendsms without -8 and
use the fake driver[2] since it produces less output; the default
gnokiirc has a [fake] section so you can use this line:

echo -n "ñó" | gnokii --phone fake --sendsms 0

whatever the encoding of your terminal, the SMS should be encoded in
the same way (UCS2 in this case since ó is not part of the Default GSM
Alphabet)

[1] http://wiki.gnokii.org/index.php/Bug_report
[2] http://wiki.gnokii.org/index.php/Fake_driver
--
Daniele Forsi

_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users

_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users
alonso acuña | 16 Apr 2012 06:44
Picon

Re: problem with 8bit characters

Some machines require using the proper LC_CTYPE=en_US.UTF-8  instead of LC_CTYPE=en_US.UTF8

Now smsd is working correctly when run after changing that and also with  -e utf8 .  Using no -e option or using -e latin1  which would be the usual for me produces empty messages.




2012/4/15 alonso acuña <acuna007 <at> gmail.com>
Hi. I have done some more tests and I'm starting to understand better the character set issues. Not sure if this is a bug but I find it is strange and since it was not documented (I think?) I spent several days figuring it out.

So basically the issue is that I have my PC with the en_US.ISO-8859-1 LANG as well as the other locale variables like LC_CTYPE etc.
echo -n "ño" | gnokii --sendsms  does not work in this scenario as the characters do not display correctly on the received message.
Then I made a file with "ño" encoded in UTF8. But cat file | gnokii --sendsms   stil does not work.
setting LANG=en_US.UTF8 stil does not work. What works is setting LC_CTYPE=en_US.UTF8.

So my env is 
LC_PAPER=es_CR.ISO-8859-1
LC_ADDRESS=es_CR.ISO-8859-1
LC_MONETARY=es_CR.ISO-8859-1
LC_SOURCED=1
LC_NUMERIC=es_CR.ISO-8859-1
LC_TELEPHONE=es_CR.ISO-8859-1
LC_MESSAGES=en_US.ISO-8859-1
LC_COLLATE=en_US.ISO-8859-1
LC_IDENTIFICATION=es_CR.ISO-8859-1
LANG=en_US.ISO-8859-1
LC_MEASUREMENT=es_CR.ISO-8859-1
LANGUAGE=en_US.ISO-8859-1:en_US:en
LC_TIME=en_US.ISO-8859-1
LC_NAME=es_CR.ISO-8859-1
LC_CTYPE=en_US.UTF8

the file is UTF8 encoded and cat file | gnokii --sendsms  works properly.
If I encode the file in ISO-8859-1 then I get an empty message.
If I encode the file in ISO-8859-1 and leave all the locale variables with ISO-8859-1 characters do not display properly.

So in summary the instructions for me to be able to send special characters are: put the content utf8 encoded in a file, set LC_CTYPE=en_US.UTF8 and send with cat file | gnokii . Did I miss this on the documentation? I am doing something weird? Is there a bug somewhere?

Thanks.


On Sat, Apr 14, 2012 at 6:00 AM, Daniele Forsi <dforsi <at> gmail.com> wrote:
2012/4/14 alonso acuña:

> When sending from gnokii this works only if I sent the -8 option on the
> command line.

so you have been lucky because 8bit format means "binary data" and it
isn't supposed to work with the default SMS application of a phone, in
fact those messages should be silenty discarded by the phone if there
isn't a special application on the specific port referenced in the SMS

> I don't know if this is a bug or somehow my error.

you didn't give enough details to tell where the problem is

first provide the debug output[1] of gnokii --sendsms without -8 and
use the fake driver[2] since it produces less output; the default
gnokiirc has a [fake] section so you can use this line:

echo -n "ñó" | gnokii --phone fake --sendsms 0

whatever the encoding of your terminal, the SMS should be encoded in
the same way (UCS2 in this case since ó is not part of the Default GSM
Alphabet)

[1] http://wiki.gnokii.org/index.php/Bug_report
[2] http://wiki.gnokii.org/index.php/Fake_driver
--
Daniele Forsi

_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users


_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Daniele Forsi | 20 Apr 2012 09:38
Picon

Re: problem with 8bit characters

2012/4/16 alonso acuña:

> Hi. I have done some more tests and I'm starting to understand better the
> character set issues. Not sure if this is a bug but I find it is strange and
> since it was not documented (I think?) I spent several days figuring it out.

you still didn't show the debug output I asked in my previous reply

> So in summary the instructions for me to be able to send special characters
> are: put the content utf8 encoded in a file, set LC_CTYPE=en_US.UTF8 and
> send with cat file | gnokii . Did I miss this on the documentation? I am
> doing something weird? Is there a bug somewhere?

IMO this aspect doesn't need specific documentation because it's
supposed to work automatically if all the programs use the same
encoding (terminal and gnokii or smsd and database and your
application) because libgnokii converts to and from your locale
settings so it shouldn't matter if you're using UTF-8 or ISO-8859-*

--

-- 
Daniele Forsi
alonso acuña | 20 Apr 2012 16:39
Picon

Re: problem with 8bit characters

So you mean it should just work with my ISO-8859 settings and with the content also ISO-8859 encoded or the content should always be utf8 encoded?  Please let me know exactly what debug output you would like to see as there are several combinations of locale settings and content encoding.

On Fri, Apr 20, 2012 at 1:38 AM, Daniele Forsi <dforsi <at> gmail.com> wrote:
2012/4/16 alonso acuña:

> Hi. I have done some more tests and I'm starting to understand better the
> character set issues. Not sure if this is a bug but I find it is strange and
> since it was not documented (I think?) I spent several days figuring it out.

you still didn't show the debug output I asked in my previous reply

> So in summary the instructions for me to be able to send special characters
> are: put the content utf8 encoded in a file, set LC_CTYPE=en_US.UTF8 and
> send with cat file | gnokii . Did I miss this on the documentation? I am
> doing something weird? Is there a bug somewhere?

IMO this aspect doesn't need specific documentation because it's
supposed to work automatically if all the programs use the same
encoding (terminal and gnokii or smsd and database and your
application) because libgnokii converts to and from your locale
settings so it shouldn't matter if you're using UTF-8 or ISO-8859-*

--
Daniele Forsi

_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users

_______________________________________________
gnokii-users mailing list
gnokii-users <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Daniele Forsi | 20 Apr 2012 16:55
Picon

Re: problem with 8bit characters

2012/4/20 alonso acuña:

> So you mean it should just work with my ISO-8859 settings and with the
> content also ISO-8859 encoded or the content should always be utf8 encoded?

I mean that encoding doesn't matter at the computer side, it only
matters for the phone and libgnokii will take care of the conversion
(unless there is a bug, but we need to see what libgnokii sees to tell
if there is a bug, I only have UTF-8 here and it works for me)

>  Please let me know exactly what debug output you would like to see as there
> are several combinations of locale settings and content encoding.

debug output is printed if you set debug=on in your config

I wrote:
> first provide the debug output[1] of gnokii --sendsms without -8 and
> use the fake driver[2] since it produces less output; the default
> gnokiirc has a [fake] section so you can use this line:
>
> echo -n "ñó" | gnokii --phone fake --sendsms 0

> [1] http://wiki.gnokii.org/index.php/Bug_report
> [2] http://wiki.gnokii.org/index.php/Fake_driver

--

-- 
Daniele Forsi

Gmane