Whether you like castle or not, Yatsushiro's shopping street, which is in the city centre, is around the castle park.
These are popular tourist destinations in Yatsushiro. They're located in outskirt, especially for Gokanosho, though.
Apart from Yatsushiro Castel or Temples, there are some other options such as Hinagu Onsen (Hot Spring) , Tea house on the seashore, Boat Tour around Yatsushiro Castle.
Yatsushiro’s specialty is chikuwa, a Japanese food made from fish surimi mixed with other ingredients and wrapped around a bamboo giving it its cylindrical shape.
The "main" center is down by the old castle walls where there's a long arcade. The Honmachi Arcade has been declining since Aeon and Youme Town malls have opened with more parking and easier access. There are still a few shops left though. One that I frequent is the Hama tea shop.
Otherwise, a taxi would get you to the Moritaka Swordsmith shop in Miyaji. The 777 steps in the same area (for views over the plain and out to Unzen and the Amakusa Islands).
Kodayaki potteries, one is by Route 3 and the other is in Hinagu (there is also a third farther north).
A few streets in Hinagu are interesting but most has been built up since it was a tourist site in the 18th C.
You could also just head out to Youme Town, a 2-story mall, for some shopping.
It's not a temple or a castle, but next to the castle is the Shohinken Tea Garden, built in 1688 by the Matsui lord for his mother. It also houses a small museum in a couple of buildings.