I had tried out the code Recent WordPress Post in Sidebar suggested by EngLee. Although the code did shows the recent posts correctly but some functions (eg: is_page() and is_home() are not working properly when it is after the portion of the code. My guess is the wordpress loop in the code modified value of some global variables.
I have a workaround for it. Before using the is_home() or is_page() function, I save the returned in a temporay variable, call the recent posts loop, and use the temporary variable later when neccessary.
Today when I search for wp_get_archives() in WordPress WIki, I found that I can generate recent wordpress posts in the more easier way. Here is the code:
<li>
	<h2>Recent Posts</h2>
	<ul>
		<?php wp_get_archives('type=postbypost&limit=10'); ?>
	</ul>
</li>
Simple and tidy :)