Prepare 8x8 baking pan by lining with foil. Grease foil liner.
Melt chocolate, sweetened condensed milk and butter over medium-low heat; stir frequently.
Add vanilla extract if desired.
Spread into greased pan. Sprinkle toffee pieces on top. Slightly press toffee into the fudge. Refrigerate for 2 hours or until set.