Chapel to Castle is only a few hundred yards and relatively flat but you can't enter the Castle grounds.[Years ago you could and the 'Below Ground' apartments are very extensive]. BUT if you want to walk the Glen which starts at the Castle then you have a very steep downhill slope followed by a relatively rough path down along the River Esk. You don't mention where you intend to leave the Glen as you can walk all the way to Musselburgh which takes about 4 hours.
In addiiton to the comment above there is a pubic car park down in Rosslyn Glen (this is below the chapel car park) from where you can walk the glen in two directions; upstream or downstream. The upstream one is easier as its a decent wide gravel path and mainly flat(ish). The downstream one is much rougher, basicaly just a rough earthen footpath and can be very muddy in places.
Thank you for your comments! On the Rosslyn Chapel website, it says the castle is a ten minute walk from the chapel through the woods and there are ruins along the way. That is the only walk that I was hoping to do and wondered if that stretch is relatively flat/easy. Based on the response from MikeBR, it sounds like that section is relatively easy.
Hvae read at https://www.tripadvisor.co.uk/Attraction_Rev… and make your mind up. I cant recall If I have walked between the two but the map indicates that, going from the chapel to castle, the road goes down and then up again and of course reverse on the way back.