#12: The dead still send invoices

I’m still not entirely sure how it happened, but at work I have somehow become the file-format necromancer. I do not remember applying. There was no interview, no ceremonial robe, and nobody has explained whether the position comes with a parking space.

Mostly, people bring me files.

Some contain invoices. Some contain what appear to be invoices after a brief but violent encounter with a lawnmower. All of them are important, several of them are undocumented, and somewhere a perfectly reasonable person needs them to become a different sort of file before the next billing run.

I have already written about Raindance and its emotionally significant spaces. Those integrations at least started with an API or SIE data. We could identify the source data before arguing with the receiving format.

This time we had old .exe files built in MapForce. The original visual mapping projects were no longer available. The executables still converted data from Edlevo and SpeedAdmin into Raindance files, which was reassuring in the limited sense that a locked box making accounting noises is reassuring.

We needed to know what was inside it.

There were two source formats to recover and a third format waiting at the other end. Edlevo had hidden its meaning in character positions. SpeedAdmin had supplied semicolons, which initially looked like an act of kindness. Raindance wanted the survivors arranged in precisely measured rectangles of text. It was a three-way incident in which everyone had brought a different definition of a row.

Also, one of the files took nearly 46 minutes and 19 seconds to convert. It was about four megabytes. I have downloaded entire operating systems in less time, including the part where I regretted the operating system.

The executable was the documentation#

There is a special kind of software immortality available to anything involved in moving money. The source project can disappear. The toolchain can become archaeology. The deployment instructions can retreat into a person who now works somewhere else. As long as the invoices keep arriving, the executable remains gainfully employed.

Eventually someone asks whether we can replace it.

Yes. First we must interview the deceased.

Fortunately, this was not a case of staring at machine instructions until a customer number emerged through spiritual exhaustion. These were .NET executables, and I had decompiled generated C# to work from, along with reference workbooks, configuration, the Frends wrapper and example files. I worked through that material with Codex. We had evidence. We just did not have a friendly explanation of what any of it meant.

The old Frends task already accepted bytes and returned bytes, but internally it wrote a temporary input, built an INI file, launched the converter through a Windows shell, waited, read the result and attempted cleanup. The Edlevo executable consulted workbooks and produced intermediate files along the way.

Generated mapping code has a particular reading experience. Somewhere inside a large population of selectors, enumerators and helper methods is a decision such as use this amount unless one of those other amounts is zero. The decision is small. Finding it requires visiting most of its extended family.

Reading everything in order was not going to improve my relationship with the material. I started at the output instead. Find the field Raindance receives, follow the selector backwards, and keep going until it reaches something in the source file.

That gives you an actual dependency:

1Raindance invoice-row amount
2  <- amount selected by the final mapping
3  <- accounting data retained by the preparation stage
4  <- amount in source record 63

It also gives you somewhere to put the explanation. A field dictionary can tell me where fifteen characters live. It cannot tell me whether the converter uses them, ignores them, or consults them only when another field contains the name of a bus pass.

That last one is not entirely hypothetical. There is a busskort text match involved in choosing a VAT code. Municipal finance has side quests.

What the fuck is this number?#

Edlevo at least looks like something you should approach carefully. Its records begin with numeric markers, and each marker selects a different fixed-width layout. The intended structure looks roughly like this:

110  file header
2  60  customer block
3  61  supplementary header
4    62  item description
5    63  associated accounting detail
6    ... more details ...
7  64  extended address
8  ... more customers ...
950  file trailer

The spaces inside those records are structural. There is no separator to rescue you if an offset is wrong. Fifteen perfectly valid digits taken from the wrong fifteen positions can still parse as a number, which is a particularly unhelpful kind of success.

One of the more welcoming stretches of the Edlevo format looks like this:

1+00000000032400000000000001200+000000027000

This is the point where a file stops looking like data and starts looking like a printer attempting to contact its ancestors.

It sits inside a 62 record. Once the field boundaries are recovered, the enormous number becomes four rather smaller problems:

1amount       +00000000032400  -> 32400
2quantity     000000000001200  -> 12.00 units
3unit price   +00000002700     -> 2700
4VAT          0

The widths are fifteen, fifteen, twelve and one character. The signs belong to their own numeric fields. The final zero belongs to the VAT field; it has simply been standing very close to the unit price and hoping nobody asks questions.

The relationship is:

132400 = (1200 / 100) * 2700

Quantity is encoded in hundredths. The aggregate check across 5,272 source rows supported that relationship, including the zero rows. This is much better evidence than staring at three examples and deciding that the zeros probably mean money.

It does not, by itself, tell me whether the monetary values are kronor, öre or another agreed scale. Multiply both amount and unit price by the same factor and the equation still works. Arithmetic can tell me how the fields relate. It cannot phone the finance department and establish the units.

Anyway. Having deciphered the amount, quantity, unit price and VAT indicator, I discovered that the current converter discards them before the final financial mapping.

The money comes from record 63, positions 9–23.

Of course it does.

The obvious amount field is not the amount field we want. It is an amount field we are allowed to understand recreationally.

Had I built the replacement from the most plausible column names, I would have produced a beautifully readable program which billed people differently.

This is why reverse engineering involves following what the code actually does, even after it has become personally disappointing.

The weird bits are still requirements#

The columns were only the beginning. There were also grouping rules, selection rules and little pockets of state which had apparently been granted permanent residency.

Two customer blocks with the same retained identity can become one invoice even when they are nowhere near each other in the file. The first occurrence determines the output group order.

So read a customer, write an invoice, forget the customer was already wrong. The file remembered things. The replacement would have to remember them too.

Then there were the repeated accounting records. The recovered Edlevo rule takes the magnitude from the first present amount, can take the minus sign from any associated negative amount, and can exclude the detail if any associated accounting amount is zero.

That is not a sum. Replacing it with a sum because sums seem appropriate in accounting would be a business change conducted under cover of tidying up C#.

Whether those are still the right business rules is a conversation with the people responsible for the money. I do not want it to begin with while refactoring, I took the liberty.

At this point, a format with actual delimiters sounded quite pleasant.

This was my next mistake.

SpeedAdmin has helpfully supplied delimiters#

SpeedAdmin looks approachable. It has semicolons. It has quotes. You can open it in something resembling a spreadsheet and see recognisable words occupying recognisable cells. After Edlevo’s wall of zeros, this feels like discovering that the next archaeological site has a visitor centre.

Then you find out that removing one trailing semicolon can change whether a row contains money.

Consider these synthetic fragments:

1R;Tuition;125.50;;MF;;;77;
2R;Tuition;125.50;;MF;;;77

The later field reader can extract the same effective values from both: an amount of 125.50, VAT text MF, 77 in field eight and an empty or absent final field. If you were cleaning up a CSV export, removing that final separator might seem harmless.

An earlier preprocessing stage has other ideas. It looks for a raw line beginning with R;, containing ;;;, and apparently having an empty eighth field. When those conditions hold, it inserts an internal line_split; marker. That marker starts a heading block. The first R in the block supplies heading text rather than a financial detail.

The eighth-field detector is a sequence of raw substring operations. Skip past seven semicolons, then take the text before the next one. With 77; remaining, it finds 77. With just 77 remaining, there is no next delimiter, and this helper returns an empty string.

The number is visibly there. The detector has concluded that it is absent. We have successfully implemented object permanence with a dependency on terminal punctuation.

In an appropriate surrounding file, that new boundary can promote the apparent charge into a heading, whose amount and VAT are ignored. A cleanup operation has changed the row’s job before the actual field parser gets to read it.

The eighth field is named unit price. Its value does not calculate the output amount or populate the output unit price. Its apparent emptiness can still decide whether the amount is treated as money at all. After Edlevo’s recreational amount field, this felt consistent with the organisation’s emerging policy on numbers.

The important detail is the order of operations:

1physical lines
2  -> preliminary hash-delimited read
3  -> raw R-line classification and inserted boundaries
4  -> structural splitting by line prefixes
5  -> semicolon fields and quote handling
6  -> mapping

By the time the quote-aware field reader arrives, several other components have already expressed binding opinions about the text.

For example:

1R;"Label;with;several;parts";100;;;;;2;

The field reader understands that the quoted label is one field and sees 2 in field eight. The earlier classifier counts the semicolons inside the label, lands on an empty apparent eighth field, and inserts a heading boundary. The two stages disagree about where the fields are. Correct quote handling at the end cannot go back in time and stop the argument.

An earlier stage is even less interested in quotation marks. It reads a one-column table delimited by #, with quoting disabled, and retains the first column. This produces the following deeply helpful transformation:

1Source:      R;"Lesson#2";125.50;;MF;;;77;
2Remaining:   R;"Lesson

The rest of the line has gone before semicolon parsing begins. Depending on the surrounding structure, the later result may be altered output or a failure. The quotation marks can explain themselves to the next parser. Most of the row will not be attending.

Semicolon-delimited formats can be perfectly sensible. The curse here belongs to this particular export and its legacy consumer together: several parsing stages disagree about what a field is, and each gets an opportunity to damage the evidence before handing it on. Replacing the last stage with a respectable CSV library would still leave everything that happened before it.

The rows have custody arrangements#

Even when the punctuation behaves, the apparent table is a hierarchical message. O and C are the outer envelopes. An S introduces a customer, H introduces a source invoice boundary, Rs supply headings and charges, and Ks sit beneath charges as associated accounting records.

Ownership follows position and containment. The converter does not join a K back to its charge by invoice number. Moving a row can change whose accounting it is. Sorting the file in a spreadsheet would therefore be an exciting way to reorganise municipal finance without obtaining any of the usual approvals.

The hierarchy also loses parts on its way through. Multiple source H blocks beneath one S become one target invoice header. The business fields in those source headers do not populate the target H. Preserving every source invoice as a separate target invoice would look entirely reasonable in an object model, and it would change the result.

The C trailer contains counts and a batch sum, but the mapping does not reconcile them. Those potentially useful checks are travelling with the file in an advisory capacity.

Accounting has a more creative arrangement. Each mapped accounting dimension takes its own first present value from the associated source Ks. This synthetic example shows the consequence:

That output combination never existed in either source row. We have assembled a new accounting record from selected parts of the deceased. Apparently the necromancy qualification transfers between file formats.

Now put three spaces in the first row’s responsibility field. In this parser, a zero-length value is absent, but spaces are present. The selector chooses the spaces, then normalises them to nothing. The later 200 no longer wins. Trimming all fields before selecting values would change the accounting output.

Even the order of clean up the text and choose the value is business behaviour.

The source K amount and sign do not supply the output financial amount, either. SpeedAdmin takes that from the charge R, using its separate credit marker, and writes it into both target R and target K. It can create a target K when no source K exists at all. Edlevo gets its money from an accounting record; SpeedAdmin gets its accounting money from an invoice row. Giving both paths a shared property called Amount would conceal most of the interesting part.

And the R and K letters on the output side belong to a different format. They are not promises that every input R becomes an output R or that an input K survives as an output K. Same letters, different careers, no forwarding address.

Raindance would like its spaces back#

Once the source has survived all that, Raindance wants four kinds of fixed-width record. These are the layouts emitted by the supplied converters, rather than a universal definition of every Raindance import:

An output R can be a heading, period text, an explanation or an actual charge. They all occupy 105 bytes. A text-only row has fifteen spaces where an amount could have been. That is an absent amount, which is different from a numeric zero. Counting R records therefore does not tell me how many charges survived. It tells me how many things were issued the letter R.

Financial rows have another small ceremony. The magnitude goes in one field, and the minus sign lives separately. For a synthetic value of -125.50, the relevant fields are:

1target field       positions     contents
2R magnitude        63–77         "          125.5"
3R credit marker    78            "-"
4K magnitude        125–139       "          125.5"
5K credit marker    140           "-"

Those positions are one-based. The magnitude fields are fifteen characters wide. The formatter removes the unnecessary fractional zero; it does not apply a universal division by 100. Both the invoice detail and its accounting representation carry the value, so adding the R and K amounts together would count it twice.

Remove the minus sign and leave a space in its place, and the line length still passes. The visible magnitude still looks correct. The financial sign has changed. This is why the rows have the right length is a useful check and an exceptionally poor stopping point.

The description gets sixty characters in R but only thirty in K. The same text can be complete on the invoice detail and truncated in accounting. Meanwhile the header layout offers date and invoice-number fields which the mapping can leave blank. A slot existing is not an instruction to fill it. Helpfully copying a similarly named source value into one of those spaces would be another behaviour change.

The output still needs Latin-1 without a BOM, explicit CRLF line endings, the right record order and all the trailing spaces. Moving the conversion onto Linux does not give Raindance a sudden appetite for UTF-8 and personal growth. The newline comes from the file contract, even when the host operating system would prefer something shorter.

Nor is the recovered writer a stern guardian of financial integrity. It truncates overlong fields on the right, including formatted numeric text, and its Latin-1 encoder can substitute for characters it cannot represent. Those are inherited behaviours which need to stay visible for review. Matching them does not turn truncating an overflowing amount into a sound financial policy.

That is the three-way incident. A field can exist in Edlevo and supply no money. A delimiter can decide whether a SpeedAdmin row is a charge. The output can faithfully carry an explanation in a record which looks, by its marker, just like a financial detail. Understanding each layout separately still leaves the mapping between them to account for.

The job was to move those behaviours into code we could understand without quietly changing anybody’s invoice. Knowing why a converter did something ridiculous was useful; I also needed a way to check that the replacement did the same ridiculous thing.

A minus sign in the wrong column is still a minus sign in the wrong column. It just arrives faster now.

Ask the corpse a question with a known answer#

The useful thing about having the old executable is that I could ask it what it did.

I used Codex for a substantial part of the reconstruction: tracing the generated code, implementing the replacement, building the comparison harness and tests, and writing the documentation. I supplied the executables, decompiled sources, production captures and reference files, and set the constraints. The most important one was that the financial output had to match byte for byte.

That made the old executable both the thing we wanted to replace and the authority on whether we had reproduced its behaviour. Codex could propose an explanation; the comparison could tell us it was wrong. My job was to decide what needed preserving and what evidence would justify accepting the result. A plausible explanation was not enough to send someone an invoice.

This also meant Codex spent 46 minutes supervising a Windows process which was still considering four megabytes. For once, I was quite happy to delegate that part.

Same input bytes. Same company, options and reference values. Run the old converter, run the replacement, compare the output bodies byte for byte. Then construct small inputs which force the suspicious rules to show themselves.

Want to know whether customer blocks are grouped or simply processed in sequence? Put the same identity in two nonadjacent blocks. Want to know whether the first amount controls the sign? Give it a positive first amount and a negative later one. A carefully arranged little file can be considerably more informative than another thousand lines of generated code.

The comparison must be rude about whitespace. No trimming lines to make the diff look nicer. No sorting records. No normalising line endings because both files look fine in the editor. In this format, that is equivalent to testing a network packet after removing the bytes you found aesthetically inconvenient.

The current replacement passed 58 synthetic unit-test cases and reproduced 38 captured original outputs with identical bodies. Those comparisons included all eight benchmarked Edlevo files, two supplied SpeedAdmin files and 28 constructed cases also run through the original converters. The current version was checked against those verified captures, so later changes did not require asking the executable to spend another afternoon being authoritative.

The extra punctuation examples illustrate the recovered pipeline; the terminal-semicolon case was also checked against the compiled replacement. They are not additional original-executable runs hidden inside that count. The comparisons do cover constructed SpeedAdmin cases for repeated accounting, missing accounting and multiple source invoice headers, among other rules.

The filename has a generated timestamp which needs a separate comparison. The body gets no such diplomatic immunity.

Matching the old program does not prove that the old program was financially correct. Two implementations can agree perfectly on a mistake. What the comparison gives me is evidence that I have preserved the observed behaviour for those cases, so we can investigate business changes without also wondering whether I moved a field one character to the left.

Forty-six minutes is quite a long time to read four megabytes#

For the largest Edlevo input, the measured result was:

That is about 46 minutes and 19 seconds versus 82 milliseconds, with an observed ratio of approximately 34,048×. The output body was identical: just over three megabytes containing 1,882 customers and their associated records.

Across all eight measured Edlevo inputs, the output bodies matched, with observed speedups ranging from roughly 3,900× to 34,000×.

Before anyone puts that ratio on a conference slide, the measurements have different boundaries. The original measurement covered a fresh process, workbook access, conversion and intermediate-file I/O through to process exit. The replacement measurement covered the public conversion call in an already loaded .NET process, with input bytes and reference values ready. It included parsing, mapping, output bytes and record counting, but excluded fetching and delivering the files.

There was one completed original run for that input. The replacement figure is the median of fifteen measured calls after three warmups. This benchmark was of the first replacement; the later configuration redesign was checked for identical output, not benchmarked again.

So this is a comparison of the two execution arrangements, not proof that my substring operation has defeated everyone else’s substring operation in single combat.

Still. Forty-six minutes.

The decompiled mapping contained a much more useful explanation than old software is slow. While emitting individual customers, it repeatedly opened and parsed the complete intermediate document. In two customer-level paths, this happened even when the eventual question concerned a fact in the file header. Underneath, the parser called ReadToEnd.

Imagine checking the return address on an envelope by rereading the entire novel inside it. Now do that for every person on the mailing list.

For a document of size N and C customers, that repeated whole-document operation costs work proportional to C × N. If customer count grows with file size, that part of the work can grow roughly quadratically. This is an explanation of a concrete repeated operation, not a proof of the entire executable’s complexity.

The replacement builds the source structure once and works out stable file-level facts outside the customer loops. It keeps the groups in memory and writes the output from that representation. There are still other linear scans for things such as naming and record counts. I have not achieved a sacred state in which every byte is touched exactly once. I have stopped asking every customer to bring their own copy of the universe.

I did not use a profiler, so I cannot assign every millisecond to a particular helper. Repeated global parsing is right there in this mapping, though, and removing it is considerably more convincing than claiming the speedup came from better vibes.

Visual mapping tools have their place. Not everybody should need to spend an afternoon reading C# to move some invoice fields around. This particular generated mapping had developed a very expensive relationship with its own intermediate file; that is what these measurements describe.

Fewer things to summon#

The useful part of the old arrangement was already there: bytes went in and bytes came out. The Windows process operating a filing cabinet in between them was something we could retire.

The replacement is a .NET 8 library with named parsing and mapping rules and a shared fixed-width writer. Source bytes go in. Result bytes, a suggested filename and record counts come out. It needs no converter executable, workbook access or temporary conversion files at runtime, and the production assembly has no external package dependencies.

It does keep the file and output in memory. For these few-megabyte inputs, that was a useful trade. If the files become enormous, or we run enough conversions at once, memory deserves another look. Streaming would need to respect the nonadjacent grouping and file-wide decisions; adding the word Stream to a class name will not persuade those dependencies to leave.

Removing the Windows executable also opens a path into our normal Kubernetes environment. Actually deploying there and validating a live Raindance import are still work to do. A fast local conversion has not secretly completed the rest of the project while I was admiring the number.

What I have now is something I can read. I can point to the rule that selected an amount, explain why a reference stayed blank, and give the next person a small test instead of telling them not to touch the executable because it is very important and nobody knows why.

That is the part I care about more than the stopwatch. Although I would like it on record that I am quite fond of the stopwatch.

If you inherit one of these boxes, preserve the executable and its settings before doing anything clever. Get one repeatable successful run. Trace the output back to the source, give every surprising rule a small synthetic example, and compare the actual bytes. Keep what it currently does separate from what we think it ought to do until someone qualified to decide about invoices has joined the conversation.

The invoices still need to go out. The old software does not need to remain alive forever. Its obligations do need somewhere to go.

Preferably in a repository somebody can still find.

I would still quite like the robe.