Naturally, there is no agreement of what exactly justice means. A pragmatic definition of justice that I found rather helpful for navigating the world: a society is just if it gives me (my in-group, my tribe) somewhat more than the others.
That said, and generalising from the success of formalisation of other previously informal domains (e.g formalisation of fair elections lead to insights like Arrow's impossibility theorem), I expect that formalising concepts of justice will lead to major progress.
I think the ambiguity with justice comes from the ambiguity of what exactly a crime is. What is or isn't a crime is largely a product of the society that defines it.
I believe we should start with a simple principle of "whatever an individual can do that does not directly or environmentally affect people who hasn't given their content can never be considered a crime". As long as this is not put among fundamental axioms of justice I can hardly recognize how any other attempts to apply logic to this area can make much sense.
imv, "just" good training data could definitely make things more just.
that is, if you clean your training data properly and adjust for correlates of injustice, e.g. race. I could imagine two ways (I am sure there are more): 1) you only train it on data from one sub-population, e.g. whites / wealthy people / etc.. 2) you carefully correct wrong decisions and biases in your training data and don't even train on such factors as race.
and then, in applying the model, you don't specify those correlates, either.