Requirements Engineering (RE) is known to be critical for the success of software projects, and hence forms an important part of any Software Engineering (SE) education curriculum offered at tertiary level. In this paper, we report the results of an exploratory pilot study conducted to assess the effectiveness of Case-Based Learning (CBL) methodology in facilitating the learning of several RE concepts. The evaluation was made on the basis of graduate students' responses to a set of questions representing various key learning principles, collected after the execution of two CBL sessions at DA-IICT, Gandhinagar (India). We investigate the perceived effectiveness of CBL in students' learning of various RE concepts, based on factors like case difference, gender diversity, and team size. Additionally, we collect and analyze the Teaching Assistants' (TAs) opinions about the conducted CBL sessions. The outcome of this CBL exercise was positive as maximum students were able to achieve all the five stated learning objectives. The authors also report various challenges, recommendations, and lessons learned while experiencing CBL sessions.