Monday, 2017-06-12

Workshop on Challenges in Performance Methods for Software Development (WOSP-C'15)

A workshop at the 2015 International Conference on Performance Engineering in Austin, Texas.

 

Workshop date: January 31 2015. The conference dates are Jan 31 – Feb 4.

The program is available on a separate page.

The workshop discussion results are available on a separate page

Contact: Murray Woodside cmw(at)sce.carleton(dot)ca

 

The workshop proceedings are available in the ACM digital library.

Workshop Outline

The first ACM Workshop on Challenges in Performance Methods for Software Development was held in Austin, Texas, on Jan. 31 2015, and was co-located with the 2015 ACM/SPEC International Conference on Performance Engineering (ICPE). Its purpose was to open up new avenues of research on methods for software developers to address performance problems. The software world is changing, and there are new challenges. As its name implies, the workshop includes the description of problems as well as solutions. The acronym WOSP-C also recalls the original discussion-heavy format of WOSP, the ACM International Workshop on Software and Performance, which has been a co-organizer of ICPE since 2010.

Challenges

The workshop papers and discussions were asked to explore the new challenges to product performance that have arisen due to changes in software and in the development process, such as:

  • faster development means less time for performance-related planning and analysis,
  • the need for scalability and adaptability increases the pressure while introducing new sources of delay,
  • model-driven engineering, component engineering and software development tools offer opportunities but their exploitation requires effort and carries risks,
  • use of third-party services and components reduces the developer’s control over the end-to-end performance
  • mobile applications require fast response in a difficult environment

These software-related challenges have led to important performance failures, such as the roll-out of the healthcare.gov web site in the US. It is probably fair to say that

  1. performance is a major problem in system development today,
  2. no one is satisfied with the methods available to address it,
  3. there is no agreement on how to improve the situation.

This workshop was intended to describe the challenges and to explore ways in which software performance engineering methods can be developed to provide solutions.

New challenges imply new opportunities for progress, and there are new tools that we may be able to incorporate (for example, methods for big-data analysis). So suggestions for new lines of attack are as important for the workshop, as the characterization of the challenges.

The workshop program lists the presented papers while the discussions during the workshop are summarized below.

Software Performance Engineering Methods

Three books that provide the background on methods for dealing with performance of software are those of Smith and Williams [1] (including making performance models from UML design models), Cortellessa et al [2] (using various kinds of models to analyze performance) and Bondi [3] (a book on useful processes, not model-based, and going beyond software).

Software performance engineering methods as understood for this workshop seek to structure and unify the understanding of performance effects and their causes, and to support extrapolation of existing knowledge to provide predictions of performance for new developments and configurations of a system. This generally requires some kind of model.

All kinds of models may be considered, ranging from regression models of data, through mathematical and simulation models of flows, contention, executions and timing in the system, to dynamic control models and semantic models of data gathered from execution. To be effective the models must relate seamlessly to existing or future system execution data. While purely data-driven performance tuning is not our main concern in this workshop, closely allied concerns such as measurement and monitoring methods are important.

Discussion Topics

Roughly half of the workshop was devoted to sessions for focused discussion. The issues for discussion arose out of the paper presentations, and out of the concerns of those present.

Some topics that we expected to be prominent are:

  • DevOps and Software Performance Engineering (SPE): is this a new opportunity? DevOps [4] integrates quality assurance (which includes performance concerns) with development and operations. QA usually means just testing: can SPE provide greater insights?
  • Model-driven SPE: is it a success? Since the first WOSP there has been substantial research on SPE based on software models (e.g. UML models) (e.g. [5], [6], [7]). Is it ready for widespread use?
  • Clouds, scale, complexity and unpredictability: large systems, rapid evolution, unpredictable workloads, complexity introduced by the combination of components that are not engineered to work together, make analysis unwieldy. New solutions may include big-data methods for analysis. A survey of current ideas is given in [8].
  • Adaptive systems: how can SPE contribute? Model-based adaptation has the potential to be faster and to provide better and cheaper performance. A survey is given in [9].

The papers in the proceedings address these and other issues, and other topics of discussion have arisen, as described below.

References

[1] Connie U. Smith, Lloyd Williams, “Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software”, Addison-Wesley, 2002.

[2] Vittorio Cortellessa, Antinisca di Marco, Paola Inverardi, “Model-Based Software Performance Analysis”, Springer, 2011.

[3] Andre Bondi, “Foundations of Software and System Performance Engineering”, Addison-Wesley, 2014.

[4] DevOps description at en.wikipedia.org/wiki/DevOps

[5] Fabian Brosig, Philipp Meier, Steffen Becker, Anne Koziolek, Heiko Koziolek, and Samuel Kounev. “Quantitative evaluation of model-driven performance analysis and simulation of component-based architectures”, IEEE Trans on Software Engineering, IEEE Transactions, Oct 2014.

[6] Murray Woodside, Dorina C. Petriu, Jose Merseguer, Dorin B. Petriu, Mohammad Alhaj, Transformation challenges: from software models to performance models, Software and Systems Modeling, v. 13 n 4 pp 1529-1552 Oct 2014.

[7] Murray Woodside, Greg Franks, Dorina C. Petriu, "The Future of Software Performance Engineering", Proc Future of Software Engineering 2007, eds L. Briand, and A. Wolf, IEEE Computer Society Order Number P2829, May 2007, pp 171-187.

[8] Yasir Shoaib and Olivia Das, “Performance-oriented Cloud Provisioning: Taxonomy and Survey”, Nov 2014, submitted for publication, available on arxiv.org/abs/1411.5077

[9] Matthias Becker, Markus Luckey, Steffen Becker, “Model-Driven Performance Engineering of Self-Adaptive Systems: A Survey”, Proc. QoSA’12, June 25–28, 2012, Bertinoro, Italy.

Workshop Discussion

In addition to discussing the presented papers (see program and ACM digital library), we had breakout group discussions on

  • SPE Challenges for Embedded Systems (notes)
  • SPE Challenges in Cloud Computing (notes)
  • DevOps Performance (notes)

Overall, we discussed several issues listed in the following

  • performance aware software migration: How should a performance model be evolved/calibrated so that it can predict a software's performance on a newer hardware/OS/virtualization platform?  Typically no measurement data is available for that software from the newer system although there might be some standard benchmark data available.
  • SPE for cloud-based applications: How should SPE be adapted to fit with a world where applications are deployed on cloud systems, which typically deliver highly variable performance.
  • What challenges does Big Data bring to SPE? How to make Data a first-class citizen in performance modelling and prediction?
  • Trends like DevOps are giving to the developer greater control over the runtime environment. What's the baseline of SPE in supporting the runtime and what challenges lie ahead?
  • Software is increasingly asked to control the physical world (e.g., cyber-phsyical systems). What are the challenges and limitations of SPE in this area?
  • indoctrinating stakeholders in a performance-oriented mindset
  • characterizing performance requirements for heterogeneous and time-varying workloads
  • software performance challenges for systems with many components that communicate with one another, as well as with a central server (such as embedded systems that send their status to and are controlled by a control station)
  • new challenges in new kinds of software, of development, of service delivery (examples, problems raised)
  • process challenges that still need to be solved (e.g. how to derive models efficiently)
  • new opportunities in new analysis capabilities? new kinds of models?