Note - database logging is off in production - saves maybe 100 milleseconds

                                                                   

Looks like the acquia agent module is causing the slow http request
acquia_agent_get_subscription()
What can you do to limit the number of http requests this module does?

Have you tried entity cache?
https://drupal.org/project/entitycache                                                                    

                                                                   

I'll see if I can get the Acquia Agent module shut off I think its required by Acquia ApacheSolr module though.

I'll try entitycache too I'm not using it currently. I might need to use a cache warmer though to see a lot of benefit. That would be ok though as long as it is performant.

                                                                   

Do you agree though? There are a few things to firm up perhaps but outside of those this seems like death by 1,000 cuts. There is just too much code running to be fast

death by 1,000 cuts analysis                                                                    

                                                                       

xdebug's cachegrind output when analyzed with wincachegrind (works under wine) can be very helpful when trying to find the slow points. I find KcacheGrind to not be as user friendly when it comes to finding out the slow points.

Just to check are you using php 5.4 or grater with some kind of op code cache like APC installed? If not, do it as it will help a lot.

• Login or register to post comments

•  

                                                                   

It's not the Acquia Agent module itself that is slow but your network or the Acquia server. The Exclusive Wall-Time is negligible for the various acquia_agent functions but quite high for the call to stream_socket_client() at 164.835 ms to open a connection.

The 3 calls to fread() at 135.390ms may or may not be an issue depending on the file size of the response and the network latency (i.e. distance) between your server and the Acquia server. 45ms on average per HTTP request doesn't really seem that bad.

HTTP requests, however, only account for 15% of the total execution time. There's other things that are contributing to the slowness.

Try running the Block Timer module (https://drupal.org/sandbox/jchin1968/2061005) on your development environment. It will highlight slow rendering blocks on a page and even blocks that are rendered in the background but not displayed.

I think this is due to                                                                    

                                                                       

I think this is due to running xhprof in my Dev environment. Presumably in production these functions will run faster because the distance to the Acquia agent server will be much less.

I'll give Block Timer a try

• Login or register to post comments

•  

                                                                   

Do you have Panel cache and View cache enabled ? Seems Memcached gets are really few.

Need Drupal help?
Reach Me                                                                    

Share your Posts, Url, Sites
www.sociopost.com                                                                    

I this case this is a node                                                                    

                                                                       

I this case this is a node page. Panels cache is active for things that are 'global'. Activating it for node content really isn't that helpful. Once a page is hit once it gets cached by varnish for 10 minutes. Most of the traffic is anonymous.

Maybe entity cache with a cache warmer would help

• Login or register to post comments

•  

Also we have had to move                                                                    

                                                                       

Also we have had to move parts of the cache back to the Database. Memcache seems to run out of memory if we leave the entire cache memcache. Page cache and views cache use the db cache.

• Login or register to post comments

•  

                                                                   

1.5 million pageviews/mo may or may not break down into a lot of concurrent page requests/sec. is the site slow all the time or just during peak loads?

also, have you verified that varnish is caching or identified what URLs are missing? or could you increase the ttl or use content based caching (a la acquia purge)?

fwiw, unless you have some super traffic spikes, on a normal and properly configured site; i would think 99.9% of anon visitors are only having a speedy varnish cached experience.

if the site is marketplace.org, the ttl is 600 seconds. just clicking around, i got mostly quick response times but hit a number of cache misses. i noticed a couple of cache counts around 200, but most were lower in the 10s. i realize this is a news site, but if the traffic covers a normal distribution of URLs--i'd guess quite a few visitors get cache misses b/c of the 10m ttl. varnish could effectively be covering the homepage and a few top links here.

Loudon & Company Consulting                                                                    

                                                                   

Yeah 10 minute ttl seems to be the most I can get away with for political reasons much more than technical ones. I could try for much longer cache and then selective clearing but the real problem is 200 + modules and scores of features that are not built for performance - the site needs an audit and a significant rebuild with performance in mind. The current build was more of a situation where they made mockups and threw all the modules they could at it to make it work, look and act like the mock up.

Could you write a crawler to verify cache hits/misses?                                                                    

                                                                       

If you write a crawler that hits your top 100 URLs (say 1 every 3 seconds 6 times a day) and stores the HEAD, in particular:
X-Age:
X-Cache-Hits:

You would be able to tell how effective your caching policy is. I don't think it would be super difficult/time-consuming to write. I could post a gist of something close in ruby...but it might be nice to hijack some boost?? code so it could have a dynamic URL list instead of a static one.

I'm not sure you're going to be able to do a whole lot to improve the speed. Try running ab or whatever benchmarking tool you like against a file that just bootstraps Drupal--no menu routing, no rendering, etc. I would look at better caching policies, but YMMV.

Cheers

• Login or register to post comments

•  

                                                                   

Selective cache clearing can go a long way (if a new blog post is written, only clear the blog home page, etc.).

And then ESI (Edge Side Includes) to have different caching rules for different parts of a single page (e.g. the 'most popular' block probably doesn't need to be updated more than once an hour and is the same for every page on the site).

--
Dave Hansen-Lange
Technical Manager
Advomatic LLC
Great White North Office
Canada

ESI Resources/Tips?