System designTroubleshooting

XML parser invalid token errors in camera description files

An invalid token error at line 1, column 2 means bad bytes in the GenICam camera XML, not your code. Dump the file, match the byte pattern, patch or re-encode.

LumaSpec Vision10 min read Share on LinkedIn

On this page 9 sections
  1. 01What the error signature tells you
  2. 02Line 1, column 2 is a byte address
  3. 03The four causes, ranked
  4. 04Dump the bytes before you change anything
  5. 05Worked example: one bad byte, found and repaired
  6. 06Bring the fix to the right layer
  7. 07Prove the fix before you judge the image
  8. 08Keep it from coming back
  9. 09Questions

When a GenICam camera description file stops being well-formed XML, the GenTL consumer refuses to build the node map and throws not well-formed (invalid token) from the GenICam XML parser — sometimes before enumeration finishes, sometimes mid-run, with black frames saved anyway. Your grab loop is almost never the fault. This page shows you how to read the reported line and column as a byte address, dump the failing file, identify the bad bytes from their hex values, and apply the fix at the right layer: the file, the cache, the parser build or the firmware.

What the error signature tells you

The full message usually names the parser and the position: a RuntimeException thrown at XmlParser.cpp, quoting not well-formed (invalid token) at line 1, column 2 (older builds quote XmlParser.cpp, line 1720). None of it is about your code. The parse runs while the consumer turns the camera's feature description into a node map, before any pixel moves. If buffers still arrive afterwards, every feature write — exposure, gain, pixel format — was lost on the way to a half-built node map, which is why the frames come out black. Treat the parse failure as the primary fault and the black images as a symptom.

A GenTL producer also serves more than one description: the device XML, and on many stacks a separate stream and interface XML besides. The garbage can sit in the stream file while the device file is clean. How GenTL stacks those node maps when only the remote device ships an XML is its own topic; parse the file your error log names, not the one you assume. And when the camera maker's own tool opens the camera without complaint, that proves the transport, not the file your consumer reads — the tool often loads its own on-disk copy.

Line 1, column 2 is a byte address

An XML document must open with < (hex 3C), and the parser then expects a legal name-start character in column 2: a letter, an underscore or a colon. A hyphen is legal inside a name but never as its first character, so a first element written <-en ... fails exactly at line 1, column 2 — the signature behind several reports of this error. A stray high byte fails the same way, because in UTF-8 the bytes 0x80–0xBF are only legal as continuations of a multi-byte character.

byte offset of the fault = characters before the reported line + (column - 1)

Line 1, column 2 therefore points at the second byte of the file, and nothing precedes it on line 1. Any reported line later than 1 means the declaration parsed cleanly and the bad byte sits in the body, so go straight to that line. For garbage that is not XML at all — an archive, truncated bytes — the same message appears with line 1 and a small column, which is why the next step is a byte dump, never a code change.

The four causes, ranked

  1. Windows code-page characters inside vendor strings. Tool tips, model names and document links carry ® (0xAE), ° (0xB0), © (0xA9) or µ (0xB5) as single bytes; one of them breaks UTF-8 decoding and the reported line lands inside the file's text.
  2. A file that is not XML. The description arrives ZIP-compressed or truncated, so the first bytes are a ZIP header (50 4B 03 04) or binary garbage.
  3. Repairs applied to the wrong file. You fix the device XML; the producer keeps serving the broken stream XML from its own folder or from cache.
  4. A parser build too old for the file. Stacks that bundle an outdated GenICam reference implementation reject files the current genicam module accepts, and bringing that module current has cleared this exact error on Python GenTL setups.

Dump the bytes before you change anything

Find the file the error names: on disk it sits beside the producer library; over the link it comes from the camera. Read the first 16 bytes — three lines of Python do it, and the third doubles as the decisive decode test:

# data: bytes — the exact file your error log names; "camera.xml" stands in for it here
data = open("camera.xml", "rb").read()
print(data[:16])        # first 16 bytes decide everything below
data.decode("utf-8")    # raises UnicodeDecodeError at the bad offset, if there is one

If the decode returns cleanly but the parser still reports an invalid token, the file is valid UTF-8 and structurally broken: check the first element name for a leading - and scan the text for a stray < or &. If the decode fails, the offset in the exception is your byte address.

First bytes (hex) What you are looking at The fix
3C 3F 78 6D 6C (<?xml) A proper declaration; the fault is at the reported line further down Read that line's bytes and jump to the worked example below
EF BB BF 3C 3F UTF-8 BOM, then a declaration Accepted nearly everywhere; if your parser rejects the BOM, strip those three bytes
FF FE or FE FF UTF-16 byte-order mark Convert the file to UTF-8 or load the vendor's UTF-8 copy; many parsers autodetect the BOM, so this row usually means the consumer decoded the file first
50 4B 03 04 (PK) ZIP local file header — an archive, not XML Unpack it or clear the consumer's cache; see the ZIP walkthrough below
3C followed by anything but a letter, _ or : The first element name starts with an illegal byte Patch or re-encode; this is the classic line 1, column 2 case

The archive row and the stale-cache row each have their own walkthroughs: troubleshooting camera XML ZIP errors during acquisition setup and clearing the camera XML cache safely. If the path in your error points somewhere you never created, start with the GenTL XML file path triage.

Worked example: one bad byte, found and repaired

Take a decode test that fails like this (the position comes from a run like yours; yours will differ):

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 28417

Dump the neighbourhood of offset 28417:

28411:  50 72 6F 62 65 20 AE 20 70 72 65 63 69 73 69 6F 6E
         P  r  o  b  e     ®     p  r  e  c  i  s  i  o  n

Check the arithmetic: 28411 plus six bytes lands on 28417, which holds AE. A continuation byte must follow a lead byte, and the byte before ours is 20, a plain space. So this is a ® stored as the single Windows-1252 byte AE where UTF-8 needs the pair C2 AE — the whole fault in one substitution.

Repair a copy on disk and prove the repair:

data = open("camera.xml", "rb").read()   # bytes, the untouched original
bad = 28417                              # from the UnicodeDecodeError
fixed = data[:bad] + "®".encode("utf-8") + data[bad + 1:]
fixed.decode("utf-8")                    # returns cleanly now
open("camera_fixed.xml", "wb").write(fixed)

"®".encode("utf-8") produces b'\xc2\xae', so each repaired character grows the file by one byte. Point the consumer at camera_fixed.xml and enumerate. For more than a handful of characters, convert the whole file with your editor's encoding converter instead, or replace each character with the ASCII numeric reference &#174;, which no later re-save can mis-encode.

Bring the fix to the right layer

When the bad bytes sit in a file the camera or producer serves over the link, you cannot edit it at the source, so work through these in order:

  1. Bring the parser current. Install the latest GenICam reference implementation your consumer is missing — the genicam module on a Python stack. A newer parser accepts files an older bundled build rejects, and this alone has cleared the error on more than one setup.
  2. Serve a clean local copy. Every producer ships its XMLs on disk. Point the consumer's XML folder or local-file setting at the vendor copy and take the link out of the parse path.
  3. Update firmware per component. Run the camera's updater and the frame grabber's updater separately; "up to date" on one card says nothing about the camera. If the shipped description itself carries the bad bytes, ask the camera maker for a corrected file.

When the garbage sits in a secondary file such as the stream XML, some consumers accept a truncation setting: a file-name pattern plus marker bytes to cut at. In the Python GenTL consumer Harvester, that fix shipped as a mapping keyed by the regular expression r'GenTL_.+\.xml', with the garbage prefix as the value. And if your table row was the ZIP header, encoding was never the issue — the archive walkthrough covers unpacking once and serving plain XML.

For a CoaXPress camera on a frame grabber, the description can also come from the card's own configuration file when the camera is non-standard; the frame grabber camera config file guide shows where that file lives and what belongs in it.

Prove the fix before you judge the image

After enumeration succeeds, read three strings back from the node map: vendor name, model and device version. Correct text means the consumer parsed the right file completely. Only then grab. Frames that stay black after a clean parse are a lighting and exposure question, and the camera is ready to be tuned like any freshly connected one. If instead Python itself raises a UnicodeDecodeError during connect, before your consumer reports anything, that earlier failure stage has its own walkthrough: GenICam XML UTF-8 encoding errors on connect.

Keep it from coming back

  • Record the working version pair — GenTL producer and genicam module — in the station's build notes, and upgrade one at a time.
  • Re-run the 16-byte dump and the decode test after every camera or producer firmware update; both take under a minute.
  • Keep the producer's XML folder versioned with the firmware that shipped it; a cache shared between producer versions is how archive bytes get served as XML.
  • After any patch, run the file through a validating parser. An assertion complaint from schema validation usually indicts the validator rather than the camera, as the schema validation walkthrough explains.

If this sends you re-stacking the acquisition side, start at the frame grabbers or area scan cameras categories — the accessories finder filters frame grabbers by interface and port count — and the applications team takes specific stack questions through the contact page.

Questions

Why did my script save black images when the XML never parsed?

Because the consumer can open the stream and deliver buffers even with an empty node map. Every feature write your script made — exposure time, gain, pixel format — was lost, so the camera ran at power-on defaults or handed over empty buffers. The black frames were a symptom downstream of the parse failure. Repair the XML, re-enumerate, and read a few feature values back before judging image quality again.

What encoding is a GenICam camera description file supposed to be in?

XML 1.0 assumes UTF-8 when the file has no encoding declaration, so undeclared high bytes from a Windows code page (® as 0xAE, ° as 0xB0, µ as 0xB5) are invalid. A file that declares ISO-8859-1 and matches it can parse cleanly, which is why some vendor files work for years. Failures start when the declaration is missing, says UTF-8, or a tool re-saves the file under an encoding that no longer matches the declaration.

The camera maker's own tool opens the camera without errors — doesn't that prove the XML is clean?

It proves the transport works, not that the file your consumer reads is clean. Vendor tools frequently load a copy of the description shipped on disk and bundle their own parser build, so both the file and the parser can differ from what your GenTL consumer sees. Dump the bytes of the exact file in your error log; only if that file matches the vendor copy byte for byte and still fails does the camera itself deserve suspicion.

How do I confirm my consumer actually loaded the patched file?

Patch a marker first: change one ToolTip string in the copy on disk and read that feature back after enumeration. If the marker text appears, the local file won; if not, the consumer still prefers the served copy or cached bytes, so set its XML-folder or local-file option and clear the cache. Keep the device's stored description untouched either way; editing flash contents needs the manufacturer's update procedure.

Why does the error mention XmlParser.cpp when my script never opens an XML file?

The GenICam node map is built by parsing an XML document, and the reference implementation's parser code lives in XmlParser.cpp. Enumeration, stream setup or a feature write triggers that parse inside the library, so the exception surfaces from the library's file and line, not from your script. Your GenTL producer fetched the file; the parser only reports where the bytes stopped being XML.