Preheat oven to 350 degrees and line muffin pan with cupcake liners.
Prepare cake mix and stir in Reese's peanut butter chips.
Pour 1 tablespoon of cake mix batter into each muffin cup.
Bake for 10-12 minutes; allow cupcakes to cool.
Top cupcakes with slightly melted ice cream.
Freeze for at least 2-3 hours, preferably overnight.
Top with Cool Whip frosting and mini Reese's peanut butter cup.