Generally you prefer dmg over armor pen, especially if you have agility attributes (precision strike and rolling thrust give 10% and 25% armor penetration respectively).
iirc, armor penetration is now subtracted from dmg reduction; so if an entity has 50% dmg reduction and you have a total of 50% armor penetration, they effectively have 0% dmg reduction.
DmgReduction = ArmorValue ÷ (500+ArmorValue)
Now why do we prefer damage over armor penetration in most cases?
First answer is that raw damage gets multiplied in many scenarios:
• 1 dmg on a Strength Weapon is 2 dmg if you have full Strength attribute; same with Agility.
• Then you can add armor dmg bonuses, which are a percentage of the base weapon damage.
• Similarly, you can add food and elixir dmg buffs!
→ And then, any full damage multiplier applies on top of that, like the first attack of 1H-Axe, or the heavy-blows perk, or the berserker perk etc. etc.
The second answer is that armor penetration doesn’t matter if your base damage is too low… 0 dmg with 100% armor pen will still be 0 dmg 
Let’s see with examples!
Example 1: Let’s say you have 100 total dmg, but no armor penetration
Enemy has 30% dmg reduction, meaning you would hit for 70 dmg.
Now let’s give +10 dmg, you hit for 77 dmg.
Let’s instead give +10% armor pen, you hit for 80 dmg.
Example 2: 100 dmg and 10% armor pen baseline.
Enemy has 30% dmg reduction, so you hit for 80 dmg.
Add +10 dmg, you hit for 88 dmg.
Add +10% AP instead, you hit for 90 dmg.
→ Conclusion, when your damage is high, armor penetration can be very helpful.
Now onto examples with lower damage…
Example 3: 20 dmg, 0% AP
Enemy has 30% dmg reduction, you hit for 14 dmg.
Give +5 dmg, you hit for 17.5 dmg.
Give +10% AP instead, you hit for 16 dmg.
Example 4: 20 dmg, 10% AP
Enemy has 30% dmg reduction, you hit for 16 dmg.
Give +5 dmg, you hit for 20 dmg.
Give +10% AP instead, you hit for 18 dmg.
→ Conclusion, when your damage is low, you should prioritize increasing damage!
TLDR is that you should always prioritize getting more dmg, the only exception being Master Weapon Fitting (+5 dmg, +12%) vs. Advanced Weapon Damage Kit (+6 dmg) because in that case +12% AP ≫ +1 dmg.
Other thing to note is that you should seriously invest some points in Agility to get Precision Strike, because the 10% Armor Penetration is basically the base amount you need to take full advantage of your dmg.
I personally recommend going full agility because Rolling Thrust adds 25%, for a total of +35% Armor Pen whenever you roll and poke… You can see that as long as you have high damage values this will be enough armor pen to take care of everything 