How to determine if PHP-to-Solr connection is slow?

Hi have a LAMP stack connecting to a Solr 3.6.1 server running on CentOS 6.3. While benchmarking my app, I noticed sudden spikes in the time taken for PHP to send a search query to Solr and getting the results back.

This time is usually about 20-40 msec and Solr’s QTime is typically 0-30 msec.

My benchmark results:

Start function                                         msec
Creating the query                                     2.50196456909 msec
Executing query and getting results (PHP & Solr)       37.4531745911 msec
    \->Executing query (as reported by Solr QTime)     21 msec
Processing Solr results

Problem: However, sometimes this step Executing query and getting results (PHP & Solr) will shoot up to 2000+ msec, and Solr continues reporting 0-30msec QTime. This makes me suspect that there may be a problem in the connections between PHP/Apache and Solr.

How can I determine if this is true, or make this connection better?

1. Use curl to replicate a search request
2. Loop curl and see if you get the same response times
3. Run the loop at the same time as hitting the PHP app and see if there’s disparity.
4. Use Wireshark/tcpdump to check what is happening to the socket connection

Check more discussion of this question.