I've read what's in the UCMA SDK doc on the threading model, and I gathered that each conversation (call) has its own queue of events, and I presume that multiple event queues can be serviced simultaneously by different threads. We are curently load testing our application and when I look through the logs, what I see from a threading point of view doesn't seem to line up with this threading model (and at the moment makes no sense). The question I am leading up to is under load it appears that calls don't appear to be executing as if they were all running in their own event queues/threads. For example, we will see a few periods during our load tests where calls don't seem to be having their events serviced (and hence delayed), and then finally when they do, the event servicing seems to happen with several calls being serviced at exactly the same time, and by the same thread! Why would that be? This is actually happening while the call is still getting initialized (so we have done teh beginAccept and are waiting for the call to go to Established and for the flow to go to Active, so at this point the application is not doing anything, so it is not holding up any event servicing threads or blocking on anything. What I see in the log is a single thread start servicing the call state changed method for a bunch of different calls all at the same time, almost like something was preventing the call initialization to proceed for all these different calls, and then it this one thread finally runs and services the events for many different calls all at once. The same thread also appears to service the tone controller for a session (call). Each session has its own tone controller (as well has speech recognizers and synthesizers)
In our application there is one application endpoint that receives call and creates a new 'app session' object which implements the application. Each session registers for call state change events on the call it encapsulates. Here is some log output. The thread IDs are in square brackets ([]). You can see a bunch of calls all have their call state change events get serviced at the same time, when I am almost certain this should have been happening in a more distributed even fashion, and probably by different threads.
Note the CallIds from the log output are from the CallId of the Audio Video call being handled.
2012-07-30 16:16:56,703 [609] DEBUG - Dtmf detected and barge in enabled and prompt still playing, so will cancel; callId: e8ee7d0fb0b213de5016f9547d64c; group:2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: 13d82f580104a6a55016f9c1cdff6
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: 5c2fd9982798e8a95016f9c2d7c37
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: e8716995663a2ad55016f9c18e855
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: c5eb801fb4c9a0305016f9c362938
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: c5eecd12d38e551a5016f9c323197
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: eaee06b37694b5895016f9c298497
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: 5c5590fc53a1ac5e5016f9c14f0b5
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: 1e0a30ad2dd5368d5016f9c10f914
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: 5613eae7b668949f5016f9c3e1878
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: 8ca67bae4347d95c5016f9c3a20d8
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: a795a69f1e7a12665016f9c46c579
2012-07-30 16:16:56,703 [609] INFO - Caller Session state change from: Incoming to: Establishing. Transition Reason: Accepted. ; callId: e910060b77926eb65016f9c42cdd9
2012-07-30 16:16:56,703 [609] DEBUG - Tone controller got a tone: 1; callId: af044fa3a25de49f5016f9113de4e; group:
Note: The average CPU utilization during this test was less than 20% with a Max CPU utilization of around 50%, so the machine wasn't pegged; it's a new box with I think 16 cores or maybe 24, so I would expect a more nice even parallel behavior than what I see.
I've gone over the code many times. There no locks, not blocking calls that seem to apply here. Maybe if I understood better how threads are used by UCMA I could track down the issue.
Thanks for any help/advice on this.