The trend as you move away from simple sequences, such as binary numbers, to state machines, toward trees and general graphs, is that problems become rapidly less tractable. The situation degrades strikingly fast: numbers have sophisticated algebras, regular languages are fairly comprehensible; but on the other end, graphs are replete with NP-complete (intractable) problems. Intractable problems aren't something you have to solve, obviously, because you won't get anywhere. But I think that not only are the hard problems in graphs intractable, the hardness of graphs carries over to the simple problems. Nothing is simple on graphs, actually, and trees are not far from graphs. So if unstructured text seems almost too simple, we should feel glad, because trees are already nettlesome even to write straightforward code for. I feel that hierarchies, as seductive as they seem, almost invariably bite; I think it's a good principle to rule out all flat solutions before considering hierarchies as solutions to any problem.
I had to write a library to transform JSON from one shape into another, carrying over the values but placing them in different nodes on a new tree. I found that thinking in terms of tree transformations was incredibly hard. The way I solved it was by flattening the JSON, transforming the flattened version, and when I'm done, inflating it back up.
In other words, working with the simpler, "2D" version of a JSON was much simpler than its "3D" tree structure, and writing flatten/unflatten independent from the transforming code was a nice modularity bonus.
It's really all about decomposing the problem and solving its parts.
If you think in terms of the operations that perform the tree transformations, of course you will have difficulty because, you're using a different tree: the syntax tree representing the code of those transformations! Not the syntax tree of the source and destination structure. So you are "one removed" from the real problem. Moreover, since both domains are trees, it's confusing.
It sounds like you could have benefited from a way to express pattern matching using JSON syntax, for destructuring a JSON object, and to represent the synthesis of new JSON using a "JSON quasiquote".
E.g. Lisp transformation with classic destructuring-bind and backquote:
There are pattern matching libraries nowadays that do a lot more than destructuring-bind, but it illustrates the basic point.
The destructuring-bind macro writes the code to pull apart some-obj according to the tree picture with embedded variables. The backquote syntax generates the code whose evaluation synthesizes the new tree object according to a template, with the values of expressions indicated by , and ,@ substituted and spliced into the template.
I never thought of using destructuring-bind together with quasiquoting in such a direct way. It does seem that that's a simpler way than what I have done. I'll have to think about it some more. Though we have to consider I had to write this in Ruby, where I don't have access to CL niceties. :)
It did get me thinking, though, so I appreciate the advice. I'm wondering now how to port that idea to Ruby.
More commonly, this destructuring is done in macros, by their "macro lambda lists". destructuring-bind is just a binding gadget which performs almost the same destructuring as macro lambda lists.
(Why almost: because "destructuring lambda lists" don't support the environment parameter of macro lambda lists!)
