Journal of Software, Vol 6, No 4 (2011), 651-661, Apr 2011
doi:10.4304/jsw.6.4.651-661

A Study of When to Release a Software Product from the Perspective of Software Reliability Models

Richard Lai, Mohit Garg, Parmod Kumar Kapur, Shaoying liu

Abstract


If a software product with a significant numberof defects is released too early to users, the softwaremanufacturer will incur post-release costs of fixing the faults.If a product is released too late, the additional developmentcost and the risk of missing a market window could besubstantial. Software Reliability Growth Models (SRGMs)can capture the quantitative aspects of testing and areused to estimate software release time. From a cost-benefitviewpoint, SRGMs aid developers to decide the optimalrelease time of the software product by providing effectiveapproaches to minimising the expected total software systemcost. This paper helps answer the question of when to stoptesting a software product by presenting the perspectivesfrom a study of cost models. The study focuses on aspectsof the relationship between development cost and scheduledelivery of the software product and the total software costincluding the risk costs, such as the penalty cost incurred dueto late delivery of software product and the cost of fixing afault during the warranty period.We also investigate varioussoftware release policies, for example, policies based on thedual constraints of cost and reliability.


Keywords


cost models; software reliability growth models; software release policies; software testing; Non-Homogeneous Poisson Process (NHPP)

References


[1] N. Ahmad, M. U. Bokhari, S. M. K. Quadri, and M. G. M. Khan. The Exponentiated Weibull Software Reliability Growth Model With Various Testing-efforts and Optimal Release Policy. International Journal of Quality and Reliability Management, 25(2):211–235, 2008.
doi:10.1108/02656710810846952

[2] D. S. Bai and W. Y. Yun. Optimum Number of Errors Corrected before Releasing a Software System. IEEE Transactions on Reliability, 37(1):41–44, 1988.
doi:10.1109/24.3710

[3] E. W. Berghout and M. Nijland. Full Life-cycle Management and the IT Management Paradox. In D. Remeny and A. Brown, editors, Make or Break Issues in IT Management, pages 77–107. Butterworth-Heinemann, 2001.

[4] P. J. Boland and H. Singh. Determining the Optimal Release Time for Software in the Geometric Poisson Reliability Model. International Journal of Reliability, Quality and Safety Engineering, 9(3):201–213, 2002.
doi:10.1142/S0218539302000779

[5] R. Brettschneider. Is Your Software Ready for Release? IEEE Software, pages 100–108, 1989.
doi:10.1109/52.31659

[6] S. R. Dalal and C. L. Mallows. When Should One Stop Testing Software? Journal of American Statistical Association, 83(403):872–879, 1988.
doi:10.2307/2289319

[7] W. Ehrlich, B. Prasanna, J. Stampfel, and J. Wu. Determining the Cost of A Stop-Test Decision. IEEE Software, 10(2):33–42, 1993.
doi:10.1109/52.199726

[8] F. Sgarbossa and H. Pham. A Cost Analysis of Systems Subject to Random Field Environments and Reliability. IEEE Trasactions on Systems, MAN, and Cybernetics - Part C: Applications and Review, 40(4):429-437, 2010.

[9] N. E. Fenton and S. L. Pfleeger. Software Metrics: A Rigorous & Practical Approach. PWS Publishing Company, 1997.

[10] A. L. Goel and K. Okumoto. Time-Dependent Error Detection Rate Model for Software Reliability and other Performance Measures. IEEE Transactions on Reliability, R-28(3):206–211, 1979.
doi:10.1109/TR.1979.5220566

[11] S. S. Gokhale, M. R. Lyu, and K. S. Trivedi. Incorporating fault debugging activities into software reliability models: a simulation approach. IEEE Transactions on Reliability, 55(2):281–292, 2006.
doi:10.1109/TR.2006.874911

[12] R. Hou, S. Kuo, and S. Chang. Optimal Release Times for Software Systems with Scheduled Delivery Time Based on the HGDM. IEEE Transactions on Computers, 46(2):216– 221, 1997.
doi:10.1109/12.565602

[13] C. Huang. Cost-Reliability-Optimal Release Policy for Software Reliability Models Incorporating Improvements in Testing Efficiency. The Journal of Systems and Software, 77:139–155, 2005.
doi:10.1016/j.jss.2004.10.014

[14] C. Huang, S. Kuo, and M. R. Lyu. Optimal Software Release Policy Based on Cost and Reliability with Testing Efficiency. International Computer Software and Applications Conference, (COMPSAC), pages 468–473, 1999.

[15] C. Huang and M. R. Lyu. Optimal Release Time for Software Systems Considering Cost, Testing-Effort, and Test Efficiency. IEEE Transactions on Reliability, 54(4):583– 591, 2005.
doi:10.1109/TR.2005.859230

[16] Z. Jelinski and P. B. Moranda. Software Reliability Research. In Statistical Computer Performance Evaluation (Ed.) W. Freiberger, 465–484, 1972.

[17] D. R. Jeske and X. Zhang. Some Successful Approaches to Software Reliability Modelling in Industry. The Journal of Systems and Software, 74:85–99, 2005.
doi:10.1016/j.jss.2003.10.024

[18] P. C. Jha, D. Gupta, B. Yang and P. K. Kapur. Optimal testing resource allocation during module testing considering cost, testing effort and reliability. Journal of Computers & Industrial Engineering, 57:1122–1130, 2009.
doi:10.1016/j.cie.2009.05.001

[19] J. Juran and F. Gryna. Quality Control Handbook, 4th edition. McGraw-Hill, 1988.

[20] P. K. Kapur and R. B. Garg. Cost-reliability Optimum Release Policies for a Software System Under Penalty Cost. International Journal of Systems Science, 20(12):2547– 2562, 1989.
doi:10.1080/00207728908910332

[21] P. K. Kapur and R. B. Garg. Optimal Software Release Policies for Software Growth Model Under Imperfect Debugging. Researche Operationelle/Operations Research (RAIRO), 24:295–305, 1990.

[22] P. K. Kapur and R. B. Garg. A Software Reliability Growth Model for Error Removal Phenomenon. Software Engineering Journal, 7:291–294, 1992.
doi:10.1049/sej.1992.0030

[23] P. K. Kapur, R. B. Garg, and S. Kumar. Contributions to Hardware and Software Reliability. World Scientific, Singapore, 1999.

[24] P. K. Kapur, V. B. Singh, S. Anand, and V. S. S. Yadavalli. Software Reliability Growth Model With Change-point and Effort Control Using a Power Function of the Testing Time. International Journal of Production Research, 46(3):771–787, 2008.
doi:10.1080/00207540600926113

[25] P. K. Kapur, M. Xie, R. B. Garg, and A. K. Jha. A Discrete Software Reliability Growth Model With Testing Effort. 1st International Conference on Software Testing, Reliability and Quality Assurance, 1994.
doi:10.1109/STRQA.1994.526379

[26] P. K. Kapur, S. Younes, and S. Agarwala. A General Discrete Software Reliability Growth Model. International Journal of Modelling and Simulation, 18(1):60–65, 1998.

[27] M. Kimura, T. Toyota, and S. Yamada. Economic Analysis of Software Release Problems with Warranty Cost and Reliability Requirement. Reliability Engineering and System Safety, 66:49–55, 1999.
doi:10.1016/S0951-8320(99)00020-4

[28] H. S. Koch and P. Kubat. Optimal Release Time of Computer Software. IEEE Transactions on Software Engineering, SE-9:323–327, 1983.
doi:10.1109/TSE.1983.236868

[29] Y. W. Leung. Optimum Software Release Time with A Given Cost Budget. The Journal of Systems and Software, 17:233–242, 1992.
doi:10.1016/0164-1212(92)90112-W

[30] K. D. Levin and O. Yadid. Optimal Release Time of Improved Versions of Software Packages. Information and Software Technology, 32(1):65–70, 1990.
doi:10.1016/0950-5849(90)90048-V

[31] C. Lin and C. Huang. Enhancing and measuring the predictive capabilities of testing-effort dependent software reliability models. The Journal of Systems and Software, 81:1025-1038, 2008.
doi:10.1016/j.jss.2007.10.002

[32] C. T. Liu and Y. C. Chang. A Reliability-constrained Software Release Policy Using A Non-Guassian Kalman Filter Model. Probability in the Engineering and Informational Sciences, 21:301–314, 2007.
doi:10.1017/S0269964807070180

[33] J. D. Musa, A. Iannino and K. Okumoto Software Reliability: Measurement, Prediction, Application. McGraw-Hill, Inc., 1987.

[34] K. Okumoto and A. L. Goel. Optimum Release Time for Software Systems Based on Reliability and Cost Criteria. The Journal of Systems and Software, 1:315–318, 1980.
doi:10.1016/0164-1212(79)90033-5

[35] H. Pham and X. Zhang. A Software Cost Model with Warranty and Risk Costs. IEEE Transactions on Computers, 48(1):71–75, 1999.
doi:10.1109/12.743412

[36] H. Pham and X. Zhang. Software Release Policies With Gain in Reliability Justifying the Costs. Annals of Software Engineering, 8:147–166, 1999.
doi:10.1023/A:1018958810083

[37] H. Pham and X. Zhang. NHPP Software Reliability and Cost Models with Testing Coverage. European Journal of Operational Research, 145:443–454, 2003.
doi:10.1016/S0377-2217(02)00181-9

[38] J. G. Shanthikumar. Software Reliability Models: A Review. Microelectronics Reliability, 23:903–949, 1983.
doi:10.1016/0026-2714(83)91018-1

[39] J. G. Shanthikumar and S. Tufekci. Application of A Software Reliability Model to Decide Software Release Time. Microelectronics Reliability, 23(1):41–59, 1983.
doi:10.1016/0026-2714(83)91368-9

[40] S. A. Slaughter, E. D. Harter, and M.S. Krishnan. Evaluating the Cost of Software Quality. Communication of the ACM, 41(8):67–73, 1998.
doi:10.1145/280324.280335

[41] G. Xia, P. Zeephongsekul, and S. Kumar. Optimal Software Release Policy With a Learning Factor for Imperfect Debugging. Microelectronics Reliability, 33:81–86, 1993.
doi:10.1016/0026-2714(93)90047-3

[42] M. Xie. Software Reliability Modelling. World Scientific, Singapore, 1991.

[43] M. Xie and B. Yang. A Study of the Effect of Imperfect Debugging on Software Development Cost. IEEE Transactions on Software Engineering, 29(5):471–473, 2003.

[44] S. Yamada, J. Hishitani, and S. Osaki. Software-Reliability Growth with a Weibull Test-Effort: A Model & Application. IEEE Transactions on Reliability, 42(1):100–106, 1993.
doi:10.1109/24.210278

[45] S. Yamada, H. Narihisa, and S. Osaki. Optimum Release Policies for A Software System With A Scheduled Software Delivery Time. International Journal of Systems Science, 15(8):905–914, 1984.
doi:10.1080/00207728408926610

[46] S. Yamada and S. Osaki. Discrete Software Reliability Growth Models. Applied Stochastic Models and Data Analysis, 1:65–77, 1985.
doi:10.1002/asm.3150010108

[47] S. Yamada and S. Osaki. Optimal Software Release Policies for A Non-Homogeneous Software Error Detection Rate Model. Microelectronics Reliability, 26(4):691–702, 1986.
doi:10.1016/0026-2714(86)90014-4

[48] S. Yamada and S. Osaki. Optimal Software Release Policies With Simultaneous Cost and Reliability Requirements. European Journal of Operational Research, 31:46– 51, 1987.
doi:10.1016/0377-2217(87)90135-4

[49] B. Yang, H. Hu, and L. Jia. A Study of Uncertainty in Software Cost and its Impact on Optimal Software Release Time. IEEE Transactions on Software Engineering, 34(6):813–825, 2008.
doi:10.1109/TSE.2008.47

[50] M. C. K. Yang and A. Chao. Reliability-estimation and Stopping-rules for Software Testing, Based on Repeated Appearance of Bugs. IEEE Transactions on Reliability, 44(2):315–321, 1995.
doi:10.1109/24.387388

[51] P. Zeephongsekul, C. Xia, and S. Kumar. A Software Reliability Growth Model Primary Errors Generating Secondary Errors under Imperfect Debugging. IEEE Transactions on Reliability, R-43(3):408–413, 1994.
doi:10.1109/24.326435


Full Text: PDF


Journal of Software (JSW, ISSN 1796-217X)

Copyright @ 2006-2013 by ACADEMY PUBLISHER – All rights reserved.