What is ICMFrequency? and How do you specify it?

I came across with an issue while fitting ecdf function for a large dataset (>700 ). When I specify the NameValue parameter "ICMFrequency" as anything greater than 15 it works almost instantly. The default value is supposed to be 10 and when I set it to default, it sits there for ever. I would like to know some clarification regarding this parameter. I have searched everywhere including google searches but no luck so far. Thanks in advance.

Answer 1

[Liam Walsh]

Hi Pappu,

To find the ecdf for interval-censored data, ecdf uses the EMICM algorithm as laid out in:

Anderson-Bergman, Clifford. "An Efficient Implementation of the EMICM Algorithm for the Interval Censored NPMLE." Journal of Computational and Graphical Statistics 26, no. 2 (April 3, 2017): 463–67.

This algorithm is iterative, and works by alternating until the ecdf is found between two different procedures - an expectation-maximization procedure (EM), and an iterative convex minorant procedure (ICM). The ICMFrequency Name-Value pair controls the frequency of the ICM step. By default, the algorithm will compute the ICM every 10th iteration, and the EM step every other iteration. Changing the ICMFrequency paramter changes how often the ICM algorithm is run. For more information, please see this doc page:

https://www.mathworks.com/help/stats/ecdf.html#mw_263237f2-cf8c-4ec5-af26-daa1279aeaf7

Are you willing to share your data? I'd like to investigate a bit further why you are seeing this behavior with the ICMFrequency with your data.

[Pappu Murthy]

here it is an excel file with the data.

[Pappu Murthy]

I have not heard from you since I sent the dataset. Just checking to see if you made any progress. Thanks.

[Liam Walsh]

Hi Pappu,

My apologies for the delay! I have investigated the issue further, and have found the source of the problem. Thank you again for sending along your data.

I believe you recently reported an issue with the ecdf function as it relates to double censoring - the root issue there is the same reason why setting ICMFrequency to 10 doesn't converge, but setting it to 15 does. Important to note though - the answer you are getting from ecdf when ICMFrequency is the correct answer.

A fix is in the works for both issues. Based on how the tech support systems works, you should receive an email when the first issue you reported with double censoring is fixed, and when you receive that email, know that this issue will also be fixed.

I am sorry for the inconvenience with using this function.

Best,

Liam

[Pappu Murthy]

Thanks much for the update. I will wait for the updated function.