7th Annual Conference on Systems Engineering Research 2009 (CSER 2009)
On the Methods of Research for Systems Engineering Timothy L.J. Ferris Defence and Systems Institute, University of South Australia, Australia,
[email protected] Abstract Systems engineering deals with different subject matter and concerns than the other engineering fields. The other engineering disciplines are focused on the application of particular classes of phenomena or technologies, or application domains. Systems engineering is concerned with ensuring both the coherence of the whole product system and that the most appropriate product system for the purpose is developed. Ensuring the technical coherence of the product system as a complete entity is a role that uses methods and perspectives that are closely aligned to the traditional methods of engineering. Ensuring the appropriateness of the product system demands a distinctive systems perspective. The link of systems engineering to the other engineering fields has resulted in research methods in systems engineering resembling research methods in engineering in general. However, there is little literature describing how engineers do research. This paper uses Varro’s taxonomy of philosophies as a means to establish a fundamental link between the knowledge sought and research methodology used to generate that knowledge, with particular application to the description of research methods which can support the concerns confronted in systems engineering. The use of the taxonomy to assist in the design of a research method for a particular project is discussed. Keywords – Research methods; Research design. 1 Introduction Research is generally understood to be a process by which new knowledge is created. Each field of academic activity has developed research methods which enable the creation of new knowledge in the form and by the process recognized by that discipline as appropriate. The interest in formalization of and discipline specificity of research methods is evidenced in universities by the large number of research methods courses offered. For example, in 2008 University of South Australia offered 59 research methods courses across the University and across program levels [1]. Generally, within an academic discipline, there are one or a few research methods which are considered the means by which real or appropriate knowledge can be developed. These methods have entered into the discipline tradition and are often regarded so highly that it is virtually impossible for anyone to present knowledge developed by some other method. Usually only the most famous researchers in the field, who achieved that status through performing a body of work regarded as important and substantial, can challenge the accepted norms. However, such people are challenged in attempting such an introduction of novel methods because of their deeply entrenched patterns of doing research and their potential to lose face over the attempt. In addition, those successful in the current paradigm have no incentive to change the paradigm. Therefore academic disciplines tend to advance towards finer and finer detail about their fields, and at least to some extent towards practical irrelevance. This phenomenon is documented in relation to Operations Research by Ackoff [2-4]. Kline [5] refers to the tendency for scientific disciplines to perform research which refines knowledge within the current paradigm until a breakpoint at which someone discovers that a fundamental change of paradigm is required to adequately explain the accumulation of
observations which do not comfortably belong within the existing paradigm of explanation. These issues demand consideration of the nature of knowledge and the means of learning new knowledge, research. This paper outlines the nature of knowledge, as analysed in some recent philosophy, and then proceeds to consider Varro’s taxonomy of philosophies and the application of this taxonomy to research methods. The purpose of this work is to show the inter-relation of the appropriate research method with the purpose to which the knowledge obtained will be put, and thus to provide a tool to assist research methodology selection of design. 2 Knowledge Research concerns the creation of knowledge. In order to have a top-down approach to the evaluation of methods of research, considering such matters as what methods of research are possible and appropriate for particular purposes, it is necessary to explore the nature of knowledge. We review some recent distinctions in the description of knowledge. “Know that” is a formulation used to describe declarative knowledge, following Gilbert Ryle’s distinction between “knowing that” and “knowing how” [6]. This distinction has been noted as significant in engineering, where both kinds of knowledge are required [7]. Declarative knowledge is knowledge of the kind that can be articulated in words that represent ideas. As such, declarative knowledge is a form of knowledge which it is fairly easy to teach and learn because it is possible to reduce the teaching or learning to recitation of the representation of the knowledge. The recitative character of declarative knowledge is discussed by Biggs [8], who also associates high valuation of declarative knowledge with a tendency towards the use of surface learning strategies. In turn this is associated with the Loughborough University – 20th - 23rd April 2009
7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) conceptualization of knowledge and learning as concerning the representations of facts rather than the assimilation of the significance of the facts into a construct that guides appropriate action.
doing the activity of discovery. However, the incremental magnitude of advance of the knowledge may be of any size, including very small, and may be very situationally constrained.
“Know how” is Ryle’s formulation to describe the capacity to perform a function. This capacity is quite distinct from the capacity to describe the area of knowledge related to the function or to recite the relevant theory. Biggs [8] describes this kind of knowledge as “functional”. Functional knowledge emphasizes the capacity of the person to perform a function, not to articulate description of what is known. However, functional knowledge does not preclude the ability to articulate what is known, functional knowledge only emphasizes the ability to perform an act. The distinctive feature of functional knowledge is that the knower’s orientation is concerned with doing a function rather than describing something.
3 Varro’s taxonomy of philosophies Augustine, in De Civitate Dei XIX:1, refers to Marcus Varro’s now lost work, On Philosophy, which provided an analysis of schools of philosophy, both actual and possible [10]. This taxonomy is repeated in Table 1. The total number of kinds of ‘philosophy’ possible in his taxonomy is 4×3×2×2×2×3=288, distinguished by the combinations of view held about each of the categories.
A third kind of knowledge is named in Nissen [9] as “knowing”, and by Biggs [8] as “procedural knowledge”, or “skill”. The emphasis in this kind of knowledge is on the ability of the knower to choose and perform some action in an appropriate and effective manner. Thus Nissen says that knowing how to ride a bicycle is demonstrated by mounting and actually riding a bicycle. The only test of knowing is to set a practical test in which the candidate must perform the action. Ability to articulate anything about the matter of action or its situation or a theory about the action or its situation is irrelevant to “knowing”. “Knowing” contrasts with “know how”, where the emphasis is on ability to perform function, but “knowing” is usually associated with some judgement foundation knowledge concerning such matters as when one would do the action, or constraints or other factors impacting on the choice of whether or how to act. These three kinds of knowledge span the range from the ability to abstract, describe and theorize about subject matter, through having applied knowledge which informs and enables the performance of some action, to practical ability to perform separated from description and theorizing. These named kinds of knowledge form identifiable points in the characterization of knowledge, but any particular knowledge held by a particular person will have its own combination of abstract theorizing and practical ability to act, and so could be described as belonging at a particular point along a continuum, possibly at a point in a continuous space of knowledge. The identification of the three kinds of knowledge is a useful tool for theorizing about the nature of knowledge, and so, by extension, about research. Research is an activity which creates new knowledge of any kind. The new knowledge maybe in an abstracted, declarative form, or may be embedded in the capacity of a person to act appropriately as either functional or procedural knowledge. In describing research as means to produce new knowledge, it is important that the new knowledge be new to everyone, not just new to the person Loughborough University – 20th - 23rd April 2009
The dimensions included in Table 1 indicate that the categorization of philosophies represented in this framework regard philosophy as a means to guide life, and so as a combination of a view on the nature of the universe and the appropriate human response. As such, this taxonomy is worthy of consideration as a foundation for the analysis of research methods, because research methods are means to discover knowledge about the universe that will be used to guide interaction with it. The two dimensions of philosophy as a frame of theorizing about nature and a statement of the appropriate human response imply that knowledge concerns the pragmatic of an ability to know and to found choices of possible and appropriate action in the world. Varro’s classification of philosophies identifies the salient features of each sect of philosophy and so identifies both the view of knowledge and the practical consequences held by the particular philosophical sect. The positions one holds in relation to each of the dimensions in Varro’s analysis determine one’s weltanschauung which in turn expresses the valuation that one has over particular pursuits and capabilities. A practical manifestation of this is the different character of ethnic cultures which are built around different philosophical or religious frameworks, which show quite different manifestations in most dimensions of interest. The differences in weltanschauung are also evident between academic disciplines, in the most obvious way between the sciences and humanities, as noted in Snow’s Two Cultures [11], but also in more subtle ways between particular sciences and quite significantly between science and engineering. The distinction between the views of knowledge in science and engineering is discussed further below. These differences in weltanschauung are important in relation to research methods because they involve views on the nature of valid and valuable knowledge and on the means required to discover what will pass the test of knowledge within the paradigm. 4 A taxonomy of research methods The above observations concerning philosophies and their consequent weltanschauungen make it appropriate to explore a classification of research methods which maps Varro’s taxonomy of philosophies because research
7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) methods are a practical consequence of the views of knowledge embedded in the weltanschauung. The taxonomy of research methods is proposed in Table 2. Table 1 - Varro’s criteria for the taxonomy of philosophies [10]. Dimension Possible categories Number Desiderata Pleasure1 [10] 4 Repose2 [10] Combination of pleasure and repose Primary natural blessings3 [10] Relation to Virtue is to be desired as the 3 virtue goal Desiderata to be desired to provide virtue Both virtue and desiderata to be desired Beneficiary Self 2 Others View of Old Academy – views have 2 certainty of certainty4 knowledge New Academy – views are uncertain5 View of Hold to tradition of 2 tradition philosophers6 Cynics (iconoclasts who reject tradition)7 Objective Look for a life of leisure 3 of life Look for a life of business Look for a life of leisure and business The structure of Table 2 is similar to that of Table 1 demonstrating the link between the proposed taxonomy of research methods and Varro’s analysis of philosophies. The final group in Table 2 concerns the objective of life which is a reconstruction of Varro’s classification to express the view on the relationship between the knower and the knowledge. This dimension may, at first, be criticised as 1
Pleasure: the “stimulation of bodily senses that gives delight”. 2 Repose: “the state in which the person suffers no bodily distress”. 3 Primary natural blessings: these are concerned with the body, such as health and wholeness, and the mind, such as abilities of the person. 4 Old Academy: the Old Academy was the Platonic philosophy which held that certainty is possible, and that there is such a thing as the truth. 5 New Academy: the New Academy was a more recent philosophical position that certainty is impossible and that there is no single truth. 6 Hold to tradition: this position built the teaching on the preceding philosophers in the tradition. 7 Cynics: the Cynics were iconoclasts who deliberately sought to reject the tradition, and deliberately performed acts to offend or contradict those holding to the tradition.
redundant, since it seems to parallel the second category: “relation to knowledge”. Consequently the sets of factors and appear to be not orthogonal. The factor which distinguishes these two categories is embedded in the weltanschauung of the actor, which leads to a sense of valuation associated with the purpose of discovery being distinguished between the existential sense of why one may do research and a rational objective. The set is associated with the rational description of the purpose of research, whilst the set expresses a view on the existential purpose of research. The other dimensions are clearly important, and orthogonal, as means to distinguish characteristics of the research methods themselves, either as methods or the outcomes that they are expected to generate. Table 2 – Proposed criteria for a taxonomy methods. Dimension Code Possible categories Desiderata D1 Develop the theory of the field D2 Develop the practice of the field D3 Develop the theory and the practice of the field Relation to K1 Knowledge is to be knowledge desired as the goal K2 Knowledge is desired for practical application K3 Both knowledge and application to be desired Person who P1 Researcher benefits P2 Others View of C1 Knowledge is certain certainty of and absolute knowledge C2 Knowledge is relative and contingent View of T1 Conform to general tradition pattern of discipline T2 Iconoclasts – challenge or reject tradition Objective O1 To enjoy Life of of life knowing leisure O2 To enjoy Life of practice business O3 To enjoy Life of both leisure and knowing business and practice
of research Number 3
3
2 2
2
3
The dimensions in the taxonomy of Table 2 are: • The Desiderata, , concern the researcher’s view about the objective of the research work. Research may be oriented towards development of the theory of the field, or development of the practice of Loughborough University – 20th - 23rd April 2009
7th Annual Conference on Systems Engineering Research 2009 (CSER 2009)
•
•
•
•
the field, or a combination. Development of the theory emphasizes the intellectual constructs which describe the framework of ideas in the field. In contrast, development of practice of the field concerns improvement in the ability to act appropriately in the field, which is not concerned with the construction of a theory of the abstraction of the field. Work involving observation of practice to, for example, determine the impact of different methods of working, in so far as it is intended to provide a description of the impact of the methods of working concerns development of the theory. Work which is about improving the ability of practitioners to act, without concern for development of a generalizable account about the effectiveness of the practices, is, most likely, research through practice, and is oriented towards the practice of the field. The relation to knowledge, , concerns whether the knowledge developed is regarded as the valued end-point in itself, or is valued as means to enable performance of something else. This contrast appears in the divide between disciplines which are oriented towards the creation of knowledge, such as the sciences, and those oriented towards performance of some activity, such as engineering. The person who benefits, , is concerned with the primary customer of the research activity. The research activity creates knowledge which is intended, by the researcher, to benefit a targeted audience, which could be the researcher in the case of work done to satisfy curiosity or for some kind of self-development, or others, where the researcher intends that the work be disseminated and seeks to develop results which may provide some benefit to others, most likely through being generalizable. View of certainty, , concerns the researcher’s epistemology. Knowledge may be viewed as an objective certainty, which is true in an absolute sense, that is true at all times, in al places and the same for all observers. Alternatively, knowledge may be viewed as relative or contingent, and thus reflective of the perspective of the observer. The latter position regards things as constructs of the observer rather than as objectively extant, and therefore the view of whether things objectively exist is linked to the position one takes with respect to whether it is possible to know about them objectively or only experiential. The view of the tradition, , concerns whether the tradition is viewed as sufficient and complete, so that the researcher seeks to fit all of their activity, both methodology and their interpretation of observations to construct conclusions, within the framework established within the tradition of the field, or is an iconoclast who challenges or even rejects the tradition. Challenging or rejecting the tradition involves, at its milder end, willingness to interpret observations in a manner that is significantly different than the extant tradition of the field has established. In the extreme the
Loughborough University – 20th - 23rd April 2009
iconoclastic position involves deliberate rejection of the tradition just to be different. • The final, objective of life, dimension, , is closely paralleled to the desiderata and relation to knowledge dimensions. The object of life can be made the pleasure to be obtained through knowing, or the pleasure of practice and achievement which is dependant on the application of knowledge, or a combination of both these. These dimensions, as discussed above, reflect different perspectives. The desiderate dimension concerns that which is regarded as innately desirable, the relation to knowledge concerns a rational view of the nature and purpose of knowledge and the objective of life dimension concerns the existential relation of the researchers to the knowledge. Whilst appearing to relate to the same general area, of whether knowledge is about theory or practice, the three dimensions concern different aspects of the relationship between the knower and the subject matter of the knowledge, which differ according to the weltanschauung. The views of the researcher on these matters will influence the research methodology because they each provide a distinctive cut on the nature and purpose of the outcome of the research task. All research methods can be classified according to this taxonomic framework. We proceed now to illustrate that assertion through reviewing a few kinds of research, and then developing the taxonomy into a set of questions which could be used to assist in the process of determining the objective of the work and the appropriate research method. 5
Examples of research methods classified
5.1 Positivist hypothesis testing Positivist hypothesis testing, PHT, is an approach to research in which one proposes an hypothesis that purports to explain some observable phenomenon. One then reconstructs the hypothesis in the form of the null hypothesis, that is, that the effect one is investigating does not exist, and then one seeks to disprove the null hypothesis. The aim is to establish a contribution to the theory of the field, which is valued as knowledge, and verifiable to certain measure of statistical confidence. The possibility of any application of the knowledge is subordinated to the establishment of the objective knowledge. As such the knowledge is created and made available to, and is presumed useful for, the whole community. The knowledge obtained is presumed certain subject to the caveats applied to all statistical methods. The knowledge obtained, whilst in detail new, is aligned with the paradigm of the field which enabled the original observation, the construction of the hypothesis, the means of observation and the interpretation of the results. The methodology used belongs within the tradition of the field. PHT is a
research method.
7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) 5.2 Action research Action research, AR, is a research method concerned with a practitioner learning how to improve their practice. As such, it is a method which embraces the impact that the researcher’s research process may have on the situation investigated. The AR approach involves the researcher making deliberate interventions in the situation with a view to observing the effect of each approach taken by the researcher, and ultimate aims to enable the researcher to select the ‘best’ mode of participation to obtain the outcomes the researcher considers desirable. As such AR is oriented towards improvement of practice, development of knowledge for practical application, primarily for the benefit of the researcher and, indirectly through the researcher’s improved practice, for beneficiaries of the desired improvement in practice. AR holds a relativist or situational view of knowledge, which is relevant in the situation in which it was developed, rather than concerned with the statement of universal truths. AR tends to be ambivalent concerning the view of tradition but is weighted towards the challenging of tradition. AR is a
research method.
5.3 Grounded theory Grounded theory, GT, is a research methodology which seeks to develop a theory of the field under investigation from the observable phenomena themselves, with emphasis on the development of knowledge as the goal. The GT approach approaches investigation post hoc, seeking to observe the extant phenomena and to develop an interpretation which becomes the theory. The intention of GT is to develop knowledge which is useful in the community associated with a particular field. It is most likely that usefulness is determined by the predictive power of the theory in relation to correctly predicting the full state of affairs when partial information is initially available. The GT approach presupposes that there is truth to be found, and that the description of that truth will be structured through the activities of the researcher. However, the GT researcher is willing to question the existing tradition of their field, whilst not deliberately attempting to be iconoclastic. GT is a
research method.
5.4 Design Design is an engineering research method in which the researcher addresses a problem which is important and novel through the activity of designing a solution. In a design research project the researcher finds means to solve the problem, thereby developing practice, and the knowledge developed is primarily developed for practical application with the possibility of some theoretical development as an additional outcome. The focus of design research is the development of knowledge which benefits others. The knowledge developed in design is certain because it concerns what does or does not work, in the
context of a project seeking to deliver means to address a defined goal. The methods and knowledge used in design research are normally within the normal patterns established within the discipline, with the novelty being in either or both of the problem addressed or the exact combination of methods used to satisfy the objective. Design is a
research
method. 5.5 Summary of examples The four research methods chosen here are methods that the author has elsewhere discussed in relation to systems engineering [12, 13]. These methods, classified in the taxonomy of research methods proposed in this paper demonstrate a wide range of perspectives on the first five dimensions shown in Table 2. This section, therefore, shows that prima facie the taxonomy of research methods is likely to be of some benefit in systems engineering because of the breadth of kinds of knowledge which are required for advance within systems engineering, which in turn results in systems engineering needing to explore a broader diversity of issues for a range of different purposes that are investigated in many other engineering fields. 6 Questions to enable research methodology selection We have considered, above, the taxonomy of Table 2 as means to describe research methods post hoc. If this were the only power of the taxonomy it would remain a curiosity because it would not be able to provide guidance as to research methodologies to use in particular cases. The taxonomy will be useful if it provides a basis for deciding on a particular research methodology to use in a particular project. The questions of Table 3 begin with six meta-dimensional questions which address the classical ‘who’, ‘what’, ‘where’, ‘when’ and ‘why’ questions that situate any activity or report of activity. The answers to these questions, which are most likely descriptive answers, will provide a foundation for deriving answers to the questions related to the six dimensions. 7 Relation to systems engineering The discussion above has described a classification of research methods which includes all research methods that could be applied for any project to develop any kind of knowledge in any field of interest. The reader may, reasonably, question the relevance of this to systems engineering. Systems engineering is a branch of engineering which addresses a wide diversity of matters including technical, management and product appropriateness issues associated with the design, development, delivery, sustainment and retirement of product systems that need to provide service in diverse application including both technical and human contexts. The breadth of areas of concern in systems Loughborough University – 20th - 23rd April 2009
7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) engineering is considerably greater than in any other field of engineering, where the concerns are constrained by either a group of physical phenomena which form the basis of a class of technologies or an area of application concern. Table 3 – Proposed questions based in the taxonomy of research methods to assist in the selection or development of a research method for a particular project. Dimension Questions MetaWhat is the subject matter of the proposed dimensional project? questions Why will the proposed project be done? Who will do the proposed project? For whom will the proposed project be done? When are the results of the proposed project required? Where will the proposed project be done? Desiderata Is the proposed project intended to make a significant contribution to the theory of the field? Is the proposed project intended to make a significant contribution to the practice of the field? Relation to Is the knowledge expected in the proposed knowledge project primarily desired for its intrinsic value? Is the knowledge expected in the proposed project primarily desired for its instrumental value as means to achieve something else? Person who Is the primary beneficiary of knowledge benefits expected in the proposed project the researcher? Is the primary beneficiary of knowledge expected in the proposed project people other than the researcher? View of Does the proposed project presuppose that certainty of the knowledge to be developed concerns knowledge matters which objectively exist? Does the proposed project presuppose that the knowledge to be developed concerns matters which are constructs of the community? View of Does the proposed project presuppose that tradition the existing framework of the field should be used as a foundation? Does the proposed project presuppose that the existing framework of the field should be rejected or vigorously challenged? The other engineering fields, like other disciplines in general, have developed bodies of knowledge and means of developing their body of knowledge, that is, ways of doing research. In many non-engineering fields there has been considerable effort to rigorously define, describe and teach the research methods. This work has been done as part of the development of those disciplines into rigorous disciplines. The engineering disciplines have little work in Loughborough University – 20th - 23rd April 2009
the definition of research methods, tending to borrow research methodologies from other fields as appears appropriate to particular projects. This paucity of work was found in the lack of any in-print books identified by the University librarians when requested to find books on engineering research methodology, early in 20088. In systems engineering the subject matter of interest is much broader than in the other engineering disciplines. Therefore it is reasonable to expect that a wider range of research methods will be appropriate in systems engineering. However, to leave the statement at this level, or even to advance a little to say that one must select the method out of a broad range of methods of fundamentally different kinds, as the author did in [12] is not useful because it does not provide a basis to choose a particular method for a particular project. The earlier work did not provide a basis to guide the researcher in the selection of a methodology for a research project. This paper has organized the kinds of knowledge and the perspectives on knowledge in order to provide a conceptual framework to assist the selection of research methods to suit the need in particular projects. The research question or issue must be identified, and then the questions described in Table 3 can be applied to generate understanding of the nature of knowledge that is sought. From recognition of the kind of knowledge sought it is necessary to do the synthetic task of proposing a research methodology which seems, plausibly, to be capable of providing the right kind of knowledge. Then the methodology can be analysed using its characteristics, in the manner that Table 2 was applied above, in order to determine the characteristics of the particular methodology proposed. In turn, this characterisation of the proposed methodology can be compared with the knowledge characteristics required from the project, already determined using the questions in Table 3. When the knowledge characteristics required match the characteristics achievable through the proposed research methodology the methodology is a plausible methodology for the project. From here it is necessary to work on detail design of the research methodology so that it actually generates the right kind of knowledge about the issues forming the substance of the project. In systems engineering there will be need for a diversity of research methods, across different projects, although probably some possible methods never will be needed. 8 Conclusions This paper has presented a method of characterising research methods in terms of the kind of knowledge that they create with a view to providing a foundation for selection of suitable research methods for application in the range of research projects which may be attempted within 8
University of South Australia. The author put the question with a view to possible acquisition of relevant works.
7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) systems engineering. A method to select a suitable research methodology has also been provided. 9 References [1] D. Dawson, "Review of UniSA research methods courses," University of South Australia, Adelaide 2008. [2] R. L. Ackoff, "The future of operational research is past," The journal of the operational research society, vol. 30, pp. 93-104, February 1979. [3] R. L. Ackoff, "Resurrecting the future of operational research," The journal of the operational research society, vol. 30, pp. 189-199, March 1979. [4] R. L. Ackoff, "OR: after the post mortem," Systems dynamics review, vol. 17, pp. 341-346, Winter 2001. [5] S. J. Kline, Conceptual Foundations for Multidisciplinary Thinking. California: Stanford University Press, 1995. [6] G. Ryle, "Knowing how and knowing that," Proceedings of the Aristotelian society, vol. 46, 1948. [7] L. L. Bucciarelli, Engineering philosophy. Delft, Netherlands: DUP Satellite, 2003.
[8]
[9] [10] [11] [12]
[13]
J. Biggs, Teaching for quality learning at university: what the student does. Buckingham, UK: Society for Research into Higher Education, Open University Press, 1999. M. E. Nissen, Harnessing knowledge dynamics: principled organizational knowing & learning. Hershey, Pennsylvania: IRM Press, 2006. Augustine of Hippo, Concerning the city of God against the pagans. Harmonsdsworth, Middlesex: Penguin Books, 1984. C. P. Snow, Two cultures: and the scientific revolution, 2 ed. Cambridge, England: Cambridge University Press, 1964. T. L. J. Ferris, S. C. Cook, and E. C. Honour, "Towards a structure for systems engineering research," in 15th Annual International Symposium INCOSE 2005 Systems Engineering: Bridging Industry, Government and Academia Rochester, New York: International Council on Systems Engineering, 2005, p. Paper 6.1.1. T. L. J. Ferris, S. C. Cook, and E. Sitnikova, "Design as a Research Methodology for Systems Engineering," in Conference on systems engineering research Redondo Beach, California: University of Southern California, 2008.
Loughborough University – 20th - 23rd April 2009
