Thoughts on software.

Some time back, I was trying to pass on an xml file from a .NET application (PDA based, on Compact Framework) to a java web service, and I faced this curious problem:
When the web service would try to parse it, the xml file would throw a SAXParseException on the server.

To compound the situation, I could not seem to reproduce this exception, but the client, if she used the same .NET application long enough, would report the problem sooner or later.
Now, how do you solve this problem you can't reproduce?
Common sense dictates that an xml file should have its encoding specified in the root tag,

I did not specify the encoding, and trusted the underlying OS to do the right thing.

I asked the client to pass the xml file to me. The file was all pretty and well formed. No garbage anwhere. I tried uplaoding this file to the web serviec from my side, it worked. The client tried the same and it failed again. Was something happening while the file was being passed on to me, maybe some format change when the file was copied from the PDA device's file system to her desktop?

This sparked a search to understand what encoding is all about. An important realization: all unicode characters are not represented
in 2 bytes, some of them can go upto 6 bytes, and something of this sort was happening because not all characters are stored in 4 bytes (the size of the file would have hinted us if this was the cause), and the one that gave us the error was obviously being interpreted as taking up 4 bytes. That the character was even being thought of as a 4 byte sequence, should have alarmed us since most common characters are represented in 2 bytes. Look at Joel's article on Unicodes ( at Joel on Software).

Did it have something to do with the different OSes, Windows and Linux have different default file encoding formats?

And then I fell into "analysis paralysis" trap. We changed the file encoding in a text editor. We even opened a hex editor and tried to figure if the file was getting corrupted at a particular position, and it seemed like some rogue bits were getting inserted in between in the middle of the file. All the information would get shifted, and from that point on the file would be all gibberish. The network could not be the problem since the client was reproducing this problem fairly regularly by now. Nothing actionable so far. End of Day 3, still no clues.

Finally, I looked at how the xml file was being written to, and found a constrcutor that takes a encoding as a parameter.

[C#] public XmlTextWriter(Stream, Encoding);

I figured that if the file is being written to with a particular encoding format, then the OS should override its default if any.
Since the web service was expecting a utf-8, I decided on this format to be mentioned in the constructor. (I also tried changing the encoding to utf-16, and yes the file size did double.)

Another suspect was the byte order marker or BOM which is often present at the beginning of an xml file. However, we found that the chances of a BOM occurring in the middle of a file are low, and the SaxParser is intelligent enough to ignore it even if it were to occur.

So, we tried to give it a go. With the encoding as the only change and with nothing else as a probable cause, we went ahead and deployed the application again.

The exception hasn't recurred since. :)