Tools for Performance Evaluation of Computer Systems: Historical Evolution and Perspectives

The development of software tools for performance evaluation and modeling has been an active research area since the early years of computer science. In this paper, we offer a short overview of historical evolution of the field with an emphasis on popular performance modeling techniques such as queuing networks and Petri nets. A review of recent works that provide new perspectives to software tools for performance modeling is presented, followed by a number of ideas on future research directions for the area.

Chapter PDF

Similar content being viewed by others

Performance Engineering: From Numbers to Insight

Chapter © 2013

Broadened support for software and system model interchange

Article 25 April 2019

PMIF+: Extensions to Broaden the Scope of Supported Models

Chapter © 2013

Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Baskett, F., Chandy, K.M., Muntz, R.R., Palacios, F.G.: Open, closed, and mixed networks of queues with different classes of customers. Journal of the ACM 22(2), 248–260 (1975) ArticleMATHMathSciNetGoogle Scholar
  2. Buchholz, P., Ciardo, G., Donatelli, S., Kemper, P.: Complexity of memory-efficient kronecker operations with applications to the solution of markov models. INFORMS Journal on Computing 12(3), 203–222 (2000) ArticleMATHMathSciNetGoogle Scholar
  3. Buzen, J.P.: Computational algorithms for closed queueing networks with exponential servers. Comm. of the ACM 16(9), 527–531 (1973) ArticleMATHMathSciNetGoogle Scholar
  4. Carlson, G.: A user’s view of hardware performance monitors. In: Proc. IFIP Congress, vol. 71, pp. 128–132. North-Holland (1971) Google Scholar
  5. Casale, G., Muntz, R.R., Serazzi, G.: Tools for computer performance modeling and reliability analysis. ACM Performance Evaluation Review 36(4) (2009) Google Scholar
  6. Chiola, G., Dutheillet, C., Franceschinis, G., Haddad, S.: Stochastic well-formed colored nets and symmetric modeling applications. IEEE Transactions on Computers 42(11), 1343–1360 (1993) ArticleGoogle Scholar
  7. Chung, M.-Y., Ciardo, G., Donatelli, S., He, N., Plateau, B., Stewart, W., Sulaiman, E., Yu, J.: A comparison of structural formalisms for modeling large markov models. In: Proc. of IPDPS, vol. 11, p. 196 (2004) Google Scholar
  8. Buzen, J.P., et al.: Best/1 - design of a tool for computer system capacity planning. In: Proc. of the 1978 National Computer Conf., pp. 447–455. AFIPS Press (1978) Google Scholar
  9. Ferrari, D., Serazzi, G., Zeigner, A.: Measurement and Tuning of Computer Systems. Prentice-Hall (1983) Google Scholar
  10. Fishman, G.S.: Statistical analysis for queueing simulations. Management Science 20(3), 363–369 (1973) ArticleMATHGoogle Scholar
  11. Gribaudo, M.: Fspnedit: a fluid stochastic petri net modeling and analysis tool. In: Proc. of Tools of Aachen 2001, pp. 24–28 (2001) Google Scholar
  12. Reiser, M., Haring, G., Lindemann, C. (eds.): Dagstuhl Seminar 1997. LNCS, vol. 1769. Springer, Heidelberg (2000) Google Scholar
  13. Heidelberger, P., Welch, P.D.: A spectral method for confidence interval generation and run length control in simulations. Comm. of the ACM 24(4), 233–245 (1981) ArticleMathSciNetGoogle Scholar
  14. Herman, D.J.: Scert: a computer evaluation tool. Datamation 13(2), 26–28 (1967) Google Scholar
  15. Herman, D.J., Ihrer, F.: The use of a computer to evaluate computers. In: Proc. Conf. 1964 SJCC, Washington DC, pp. 383–395. Spartan Books (1964) Google Scholar
  16. Hillston, J.: Fluid flow approximation of pepa models. In: QEST 2005, pp. 33–42, 19–22 (2005) Google Scholar
  17. IBM. General purpose systems simulator iii user’s manual. Technical Report Form H20-0163, IBM (1965) Google Scholar
  18. Jensen, K.: Coloured Petri Nets: Basic Concepts, Analysis Methods, and Practical Use, 2nd edn. Springer, Heidelberg (1997) MATHGoogle Scholar
  19. Kartson, D., Balbo, G., Donatelli, S., Franceschinis, G., Conte, G.: Modelling with Generalized Stochastic Petri Nets. John Wiley & Sons, Inc., New York (1994) Google Scholar
  20. Kleinrock, L.: Queueing Systems, Theory, vol. 1. John Wiley & Sons, New York (1976) Google Scholar
  21. Lavenberg, S.S.: A perspective on queueing models of computer performance. Performance Evaluation 10(1), 53–76 (1989) ArticleGoogle Scholar
  22. LeMay, E., Unkenholz, W., Parks, D., Muehrcke, C., Keefe, K., Sanders, W.H.: Adversary-driven state-based system security evaluation. In: MetriSec 2010: Proceedings of the 6th International Workshop on Security Measurements and Metrics, pp. 1–9. ACM, New York (2010) ChapterGoogle Scholar
  23. Markowitz, H.M., Hausner, B., Karr, H.W.: Simscript: a simulation programming language. Prentice Hall (1963) Google Scholar
  24. Martens, A., Koziolek, H., Becker, S., Reussner, R.: Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms. In: WOSP/SIPEW, pp. 105–116 (2010) Google Scholar
  25. Montani, S., Portinale, L., Bobbio, A., Codetta-Raiteri, D.: Radyban: A tool for reliability analysis of dynamic fault trees through conversion into dynamic bayesian networks. Reliability Engineering and System Safety 93(7), 922–932 (2008); Bayesian Networks in Dependability ArticleGoogle Scholar
  26. Nielsen, N.R.: Computer simulation of computer system performance. In: Proc. of ACM National Meeting, pp. 581–590 (1967) Google Scholar
  27. Pawlikowski, K.: Steady-sate simulation of queueing processes: A survey of problems and solutions. ACM Computing Surveys 22(2), 123–168 (1990) ArticleMathSciNetGoogle Scholar
  28. Potier, D., Veran, M.: The markovian solver of QNAP2 and examples. In: Hasegawa, T., et al. (eds.) Computer Networking and Perf. Eval., pp. 259–279. North-Holland, Amsterdam (1986) Google Scholar
  29. Sauer, C.H., Chandy, K.M.: Computer Systems Performance Modeling. Prentice-Hall (1981) Google Scholar
  30. Sauer, C.H., McNair, E.A., Kurose, J.F.: The research queueing (RESQ) package, version 2: Introduction and examples. Technical Report IBM rep. no. RA 138, IBM (1982) Google Scholar
  31. Saure, D., Glynn, P., Zeevi, A.: A linear programming algorithm for computing the stationary distribution of semi-martingale reflecting brownian motion (under submission) Google Scholar
  32. Schwetman, H.: CSIM Reference Manual (1988) Google Scholar
  33. Smith, C., Llado, C.: Performance model interchange format (pmif 2.0): Xml definition and implementation. In: Proc. of QUEST 2004. IEEE Press (2004) Google Scholar
  34. Smith, C., Lladó, C., Puigjaner, R.: Performance model interchange format (pmif 2): A comprehensive approach to queueing network model interoperability. Perform. Eval. 67(7), 548–568 (2010) ArticleGoogle Scholar
  35. Smith, C.U., Williams, L.G.: Performance Engineering Evaluation of CORBA-Based Distributed Systems with SPE∙ED. In: Puigjaner, R., Savino, N.N., Serra, B. (eds.) TOOLS 1998. LNCS, vol. 1469, pp. 321–335. Springer, Heidelberg (1998) ChapterGoogle Scholar
  36. Smith, C., Williams, L.: Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley (2001) Google Scholar
  37. Stewart, W.J.: Introduction to the Numerical Solution of Markov Chains. Princeton University Press, Princeton (1994) MATHGoogle Scholar
  38. Trivedi, K.S., Bobbio, A., Ciardo, G., German, R., Puliafito, A., Telek, M.: Non-markovian petri nets. In: SIGMETRICS 1995/PERFORMANCE 1995, pp. 263–264. ACM, New York (1995) Google Scholar

Author information

Authors and Affiliations

  1. Imperial College London, London, SW7 2AZ, UK Giuliano Casale
  2. Politecnico di Milano, I-20133, Milan, Italy Marco Gribaudo & Giuseppe Serazzi
  1. Giuliano Casale