Json.org Android adding weird characters

Question

I have a client / server app written using Android and I'm using the standard org.json package classes bundled with android to do the parsing and creating.

I've been getting weird characters appearing on the server side right in the middle of the json strings generated for example (not the full one, because its big):

{!lo":"es_MX","id":2791884,"os":"8"}

As you can see the (!) exclamation mark appears randomly instead of a double quote. I also get other random characters appearing mid string. It's very bizarre.

Here is the code which creates the JSON object...

JSONObject jsonObject = new JSONObject();
jsonObject.put("key", someValue);

Here is the code which sends..

HttpPost type = new HttpPost(<server url here>);
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("v", jsonObject.toString()));
type.setEntity(new UrlEncodedFormEntity(params, HTTP.UTF_8));
httpClient.execute(type); // This is a DefaultHttpClient

I say random, but the exclamation mark in this exact position is consistent in many errors, but not every-time. About 5 messages that get this error, among tens of thousands per day. And usually not the contents of the values inserted into the json, but the characters (such as the quote character above) that define the structure of the message, which suggests to me that this isn't a character set issue.

Has anyone come across this?

Answer 1

It looks like your sender is not properly setting the character set. Spanish will have symbols not present in regular ASCII or most Windows encodings, you need to use UTF-8:

Content-Type: text/html; charset=utf-8

Without knowing which HTTP exchange you're using (read more), it is not possible to give you an exact code snippet to fix the problem - but that should be easy enough to figure out.

Answer 2

Print the json on the client (using Log.d or similar) and see if it contains weird characters before sending it to the server.

Answer 3

it seems you are composing string in other format, and on receiving text decode in another format like iso to utf.

Answer 4

You give not enough information. Radical method to fix your problem is just replace all (!) characters to (").

string.replaceAll("!", "\"");

I guess it is server side issue.

Answer 5

I had also simmilar problem. Let me write much more to describe my environment. My server was returning data in json format. But my problem was connected with special chars like ąść. YOu should know, json_encode() will return from server in this case string text as a null. I know, it sucks! So I added mysql_query('SET CHARACTER SET utf8'); before my selction for items from database. This allowed me to take strings from server with special diacritics letters.

Now on the app site, I was taking data from server by GET method. First I was storing result data into InputStream. Then I was packing it into InputStreamReader and byte by byte I was appending it into stringBuilder. That's ready appended text was converting by toString() ready string. Then I was putting it to new JsonArray(readyString). However I discovered some parts of text for json had weird chars.. Especially in that places where were special letters like żóć. For example "description":"aaa" was throwing "descriptionPffa":"aa"null:`.

I decided to try another way for converting result from data. In places where I was converting data from server I used method below. At the end, wgen I got byteArrayOutputStream object I changed it to new String(byteArray) and then somehow it worked with new JsonArray(new String(byteArray))!

public class Streams {
    public static byte[] getBytes(InputStream is) {
        int len;
        int size = 1024;
        byte[] buf = new byte[0];

        try {
            ByteArrayOutputStream bos = new ByteArrayOutputStream();
            buf = new byte[size];
            while ((len = is.read(buf, 0, size)) != -1)
                bos.write(buf, 0, len);
            buf = bos.toByteArray();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return buf;
    }
}