Use the fee model you actually have
Keep a percentage fee and a flat fee separate. A provider may apply its fee to shipping, tax or a whole order, so record how you allocated it to one unit.
The working formula
Under the simplified assumptions, required price equals cost plus flat fee, divided by one minus the percentage fee and target margin. If the fee and margin reach 100%, the model must reject the input.
Review after real sales
Compare the plan with actual transaction exports, refunds and fulfilment costs. An editable assumption is safer than a hardcoded marketplace fee.