Apache requirement for WordPress Pretty URL

I installed a fresh WordPress on a brand new VPS. No matter how I change the settings of permalinks, it is always giving me 404 error for all pages except Homepage.
Later I google it and find the solution from the official site.

* Apache web server with the mod_rewrite module installed
* In WordPress’s home directory,
o The FollowSymLinks option enabled
o FileInfo directives allowed (e.g. AllowOverride FileInfo or AllowOverride All)
o An .htaccess file (if this file is missing, WordPress will try to create it when you activate “pretty” permalinks)
o If you want WordPress to update the .htaccess file automatically, WordPress will need write access to the file.

For my situation, it is because of options of Apache.
I edited the httpd.conf and added following options.

FollowSymLinks
AllowOverride All

Then apply the changes and it works.
My URL looks like: http://www.mysite.com/food/delicious/