@Ravindra,
That's not true.
Here's how the server side components for all of the browsers are designed.
For Firefox, the server side component is built as a firefox plugin, [ Firefox supports plugins and works with profiles. So WebDriver extracts the embedded webdriver xpi, creates an anonymous profile, adds this plugin to it and then spawns the browser]. That way all of your webdriver commands that you trigger via the client bindings can interact with the embedded server.
For IE, the earlier releases used to work with an internal IEDriver.dll which was embedded into the selenium jars. But after 2.25 I believe Jim Evans separated them out of the codebase and started releasing the server component (Earlier the IEDriver.dll) as IEDriverServer.exe. This was done to basically decouple the server component from the language bindings so that it becomes easier to fix problems at the server side (that was mostly where most of the heavy lifting of work is done in terms of browser interactions) and release it independent of a selenium release.
For Chrome, the same role is played by chromedriver.exe which plays the server component role.
IEDriverServer is very much part of the core webdriver produced artifacts. ChromeDriver on the other hand is being provided by the chromium project.
The reason a server component is involved here is because WebDriver APIs adhere to the JSONWireProtocol which basically leverages GET/PUT/POST/DELETE operations on an end point to get the interaction with the browser done. The server component basically responds to the JSONWireProtocol compliant commands (The client bindings internally convert all your API actions into a JSONWireProtocol compliant command and post it against the end point) and performs the UI operations.