Not sure I have understood the issue, but I always use features on the part for aligning the CAM toolpaths rather than the stock? When generating CAM I add stock that replicates the overall dimensions of the part. The stock is only there to visualise the correct material removal and check tolerances.
There are two basic starting points. When the part itself is rectangular I mostly start with stock that is the correct size. I expect that general squaring up of stock on the manual mill will be within a thou or so of design size. For a part that has a non-rectangular profile I start with a rectangle that is a millimetre or two larger than the extremes of the part. On these of parts I pick up on a feature on the part such as a hole. The first operation will be profiling with multiple steps inwards so just add an extra step to the width of cut if needed. One will cut air for a period but for low part quantities it is not worthwhile spending hours fine tuning the CAM. Plus, when the CNC mill is running I can go and do something else.
As an example these parts started as a rectangular blocks of hot rolled steel to within a thou of final size:

The position of the two holes in the lugs are the only critical dimension; all three holes were drilled in the blanks on the manual mill before CNC machining. The reference point on the part for each operation was the north west point and top surface. This was set with reference to the fixed vice jaw and ‘zero’ height set from the work with tool 0 so that the tool height table entries were correct.
For these expansion links the positioning of the three holes are critical and they were drilled/reamed on the manual mill using the DRO. The outline is anything but rectangular so I used the middle hole as the zero reference:

The expansion links were mounted on a fixture with accurately drilled holes for bolts. Before CNC machining the oversize blank was mounted using bolts in the two outer holes and a co-axial indicator used to set zero to the middle hole. Then the third bolt was installed and CNC machining commenced:

Profile first followed by roughing out and finishing the curve slot. The slot was left a thou undersize to allow for hand fitting of the die block.
For odd shapes like the L mentioned I will start with a rectangle and a profiling cut so that the CNC mill generates the waste part. I wouldn’t spend time milling it all away.
Julie