A second layer of drywall would help, a second layer separated from the existing drywall be a Z strip to give an air gap would help even more. Unfortunately both of these are messy to implement and will shrink your room size slightly.
It also depends on how your common wall was constructed. The best techniques don't share framing for the units. I've owned two townhomes over the years, the first used 2x6's for the top and bottom plates of the walls, then they used 2x4's for the studs, my unit's studs were not used to support my neighbor's drywall. Then they covered both sides with 2 layers of drywall. They also insulated the common wall with fiberglass, though it doesn't do much to stop noise transmission.
The other townhome had separate 2x4 stud walls with a 1" thick gypsum board in-between and a single layer of drywall on each unit. Both systems worked fairly well, though in both I could hear when my neighbors closed cabinet doors on the common walls. I never heard foot falls between units. Bass sound could come through the walls if they were really cranking the stereo.
These are both considered firewalls between units. Probably the worst design from a sound perspective is a concrete block wall, in my basement before I finished it I could hold a conversation with my neighbor through the cinder block walls. Once I studded the walls out and insulated them that problem went away.
"Pain is temporary, quitting lasts for ever." Lance Armstrong