Can i sew a non-stretchy fabric around a stretchy one?

I cut some wooly/nylon stockings that never fit me into fingerless elbow length gloves. I want to sew a decorative piece of fabric around the top of the arm, which i need to stretch to fit my (small) bicep. How do i sew the fabric onto the stretchy part so that it will still fit around my arm? Brainstorming i thought of maybe somehow sewing it.. like, not all the way around so that there's some room, but i think in practice the stitches would rip out if i did that. I do have a sewing machine, but I was planning on doing this by hand since it's not a lot of fabric.