Poor platform user lookup performance

This was originally reported in the field where looking up ~350 coaches via UPortalPersonAttributesService#getCoaches() was taking ~1.5m. In that case we demonstrated that only a negligible portion of that time could be attributed to the "core" LDAP search. Further, even searches with result sets smaller by an order of magnitude would take several seconds to return, which is rather unexpected for what should be heavily cached data.

The fix in the field was to dramatically increase the TTLs on the org.jasig.services.persondir.USER_INFO.merged cache, which did fix the perceived poor performance, but meant that changes in external attribute stores (LDAP) would be very slow to appear in SSP (Platform).

Replicating the problem locally was difficult until I added a third PersonAttributeDAO to the default two uPortal-internal DAOs, in which case I could get even small datasets, e.g. 2 coaches, to take up to a second to return and larger datasets, e.g. 40 coaches, to take up to 15 seconds to return. I think this happens because the query pattern triggered by PersonLookupHelperImpl#searchForPeople has at least two problems:

1. Before the "main" search runs, the attribute filter ends up querying each DAO for each possible attribute as if it were a username.

2. After the "main" search runs the results are filtered for user-scoped visibility (i.e. only return the users for which the current user has permissions to see), which results in each found user being queried for again, but by username

#1 seems like a flat out bug. I don't understand why that would ever make sense to do. Need to talk with the uPortal guys about that one.

#2 seems like we both need to recommend that all "external" DAOs need a caching wrapper and that we need that wrapper to always cache both query results and per-username results so that if the app turns around immediately and asks for the same users by username we're not going to hit the underlying data store. It's still going to be terribly inefficient, but would be an improvement (and, FWIW, not all that dissimilar from the Hibernate identity cache).

Lastly, to avoid regressions in this area and because we can't set per-search cache policies on the platform side, we should probably implement one or more caches on the SSP side... maybe some simple wrappers on the PersonAttributesService interface? (http://docs.spring.io/spring/docs/3.1.4.RELEASE/spring-framework-reference/htmlsingle/#cache)

Here is an example of problem #1 where I captured a stack trace just to demonstrate where the call was coming from. You should expect to see ~10 similar lookups, basically one for every possible user attribute:

And here is an example of #2. You should expect to see one of these for each person returned from the "main" coach search: