Transactions of the 
Discussion on the concept of 
Real-life Application of the MCDM techniques

that took place on May 3-22, 2000, on MCDM discussion list
(Multicriteria Discussion List <MCRIT-L@LISTSERV.UGA.EDU)

May 3, 2000
Dear Colleagues,

I'd like to attract your attention to the question that seems to be extremely important for the MCDM Society: what does it mean real-life application of a MCDM technique?

Often we find papers where authors argue that recent MCDM tools and approaches find real-life application fairly seldom. But what do these people mean under real-life applications?

On the other hand, a MCDM researcher may declare a real-life application of a technique (developed by him/herself) even in the case when its application is actually a demonstration that the technique can be adapted to a particular real-life problem. In this case, decision makers have no idea at all that a MCDM professional is thinking about one of their problem. Surely, such kind of application is a methodological one.

In another case, a decision maker knows about the research, but (s)he does not take the results into account. In this case the MCDM specialist may declare real-life application of the technique only since a decision maker has found time to listen about the technique.

Surely a precise definition of a real-life application is needed. In the recent paper by E.Kasanen, J.Wallenius, J.Wallenius, and S.Zionts, EJOR, v. 120, 2000, p. 496-510, devoted to the discussion of the role of MCDM techniques, a definition of a "real" application is given (page 497, footnote):

This is an <<application where
i) an actual problem of an actual organization is studied
ii) using real data,
iii) in which decision makers participate, and
iv) the results of which have been implemented.>>

Is this definition full and precise? Is it instrumental? Can we use it to distinguish among methodological and real-life applications?

To my opinion, the above definition accurately describes the modern state of common understanding. However, real-life application of famous techniques often has nothing to do with the common understanding. Let us consider two techniques distinguished for their real-life application, the goal programming and the AHP.

Multiple decision makers, experts and other users apply these techniques without informing authors about it. Are the problems they study actual or not? What kind of data do they use? Do decision makers participate, or experts prepare decisions by themselves, or ORMS professors develop educational examples? How about the implementation? Do the techniques influence decision processes at all?

We can not answer these questions: too many institutions and private people use the techniques in many different ways. I have met several cases where applications of the goal programming and the AHP were not successful. From the other side, many successful applications are known. What we know for sure is that multiple institutions and private people buy Expert Choice and other software systems or develop software by themselves and use it somehow. Therefore we assert that both techniques have the real-life applications. Important that the quality of an application in this case depends on user, but not on the technique.

So the above definition seems to be more related to an attempt of an author of a technique to test its applicability, but not to a broad real-life application of it. Therefore, the definition of "real-life application" of a MCDM technique needs further discussion. My proposal is as follows. We start the discussion now via email and develop a draft definition. During the conference in Ankara, the definition may be finalized and approved by the conference. In this case, we would get a definition recognized by the MCDM Society.

What do you think about it?

To start the discussion, I formulate my proposal concerning the definition of real-life applications of a MCDM/MAUT technique.

A MCDM/MAUT technique is said to have real-life applications if institution(s) or particular people have mastered the technique and use it independently from the author for a relatively long time (regularly or sporadically).

As you can see, both goal programming and the AHP satisfy this requirement. I'm convinced that other techniques that pretend to have real-life applications must satisfy this requirement, too.

I look forward to hear about your comments.

Cordially,
Alexander

Alexander V. Lotov, Ph.D., Doctor of Science

Head, Department of Mathematical Methods for Economic Decision Analysis
Russian Academy of Sciences, Computing Center
Vavilov street, 40, Moscow, 117967 Russia
Phone  (7-095)-135-1209       Fax  (7-095)-135-6159
E-mail  LOTOV@CCAS.RU   http://www.ccas.ru/mmes/mmeda/
and
Associate Professor, Department of Systems Analysis
Lomonosov Moscow State University

May 3, 2000

Dear Alexander, and the rest of the List Members:

I've attempted, for decades, to obtain some agreement as to the subject of "real-life applications," including (and particularly) those of MCDM and Multiple Objective Decision Making. The definition that I personally employ is as follows:

"A real life application is one in which the tool of interest was employed to solve a real problem, for a real organization, and the results were successfully implemented AND adopted. Furthermore, that 'real organization' MUST be identified."

This definition rules out lots and lots of alleged applications in which the analysts refuse to identify the organization that allegedly employs --- or employed --- their alleged results. In some cases this is problematic because, on occasion, some real organizations refuse to reveal what they consider a trade secret or competitive advantage. On the other hand, the definition does stop false or exaggerated claims of "real life" applications ---- and, unfortunately, I've seen a (large) number of such instances (and too many get published).

For example, whenever I receive a paper to review that starts off with something like "our method was successfully applied to an organization that must remain anonymous," or "for proprietary reasons we cannot reveal the name of the organization," I am immediately suspicious. The scientific method simply does not provide for such exceptions, even if they may happen to be true.

Jim Ignizio

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Professor James P. Ignizio
University of Virginia
ignizio@virginia.edu
FREE PREVIEW! GONE AWRY: A virtual tour through High Tech Hell
http://www.geocities.com/goneawry25
http://carolinanavy.com/navy/renaissancefleet/gypsy/james/index.html
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

May 3, 2000
Dear all
This sounds as if it could be an interesting discussion - I've always cringed slightly at the use of the term "real-life application" in that for me an application is by definition "real-life" in the sense that involves a decision maker (or group of decision makers) and possibly other stakeholders concerned about a problem / decision facing them and on which they must take action.  It could be a personal problem or an organisational one.  Anything else is an "illustrative example" (or methodological application in Alexander's terms).

However, I would like to discuss one aspect of the defn quoted by Alexander and of James' response - that is, what exactly is meant by "implementation".  I have use MCDA (generally MAVT-based) in many "real-life" situations.  In some cases there has been a clear cut decision which has been acted upon (e.g. allocate the contract to company X).  However, in many other cases the "outcome" has not been so clear cut the process / analysis has caused the decision makers to "think again" something unanticipated has happened subsequent to the process which has "over-ruled" the outcome .... the outcome of the analysis has formed the basis of a recommendation to some other (usually higher-level) group and I don’t always know what the eventual outcome was.  However, in most of these cases the decision makers who participated in the process felt that it was of benefit to them & that they learned from it.  Hence, I would like to put forward a proposition that a successful application is one in which the participants learned about their problem and priorities in a way which enabled them to move forward.

James' point about "confidentiality" raises a general question about why we would want to define a "real-life application" - is it a criterion for publication? prizes? or what?  I'm sure there are very many real applications that cannot be published and some thinly disguised ones that have been - the fact that the organisation involved does not wish to be named does not make the application any less real.  However, as pointed out this should not be used as a front to disguise "illustrative examples".

Yet another issue raised is the expertise of the "analyst" (or facilitator) if they apply the method in a manner which the experts might not be completely happy with, does that make the application any less real?  We teach MAVT to several hundred MBA students each year ... and often hear from ones who have used it in their organisations to analyse significant issues.  I cant be sure they haven’t defined the scales or weights "correctly" ... but does that make the application any less real?  I think that if MCDA is to realise its full potential we must seek to encourage such use and to design softwares (Intelligent MCDSS?) and supporting explanatory materials which guide "D.I.Y" users through the process.

Enough for now!!
Best regards
Val

Valerie Belton
Management Science
University of Strathclyde
40 George Street
Glasgow G1 1QE
Tel +44 141 548 3615
Fax +44 141 552 6686

May 4, 2000
Dear friends

Alexander introduces a very interesting subject. I am very sympathetic with Valerie's reply. I will try to introduce some further considerations.

To my point of view a real-life application is the one where it is possible to observe what I call a decision aiding process. That means at least a client and an analyst are involved, the first expressing a "problem",  the second trying to give him some advice. Other actors may be involved, each with different concerns and stakes in the process.

The above description can apply to cases where the client is a patient and the analyst is a psychologist or a physician, the client is anybody and the analyst is a lawyer. What distinguish and characterise our field is the use of formal methods, that is methods reducing ambiguity, typical of  human communication. Under such a perspective the output of the decision aiding process is not the result of a method applied to a model, but the advice given to the client and further the use of such an advice done by the client.

Can we speak about "successful" real life applications? This involves two dimensions. The first is client's satisfaction. Here we have to pay attention. Satisfaction does not mean that the method output or the analyst advice were OK! for the client. It can be the case (and I had such experiences) where the advice was more or less rejected, but where the process was satisfactory because enabled the client to understand better his problem. Therefore satisfaction refers to the decision aiding process and not to its result. The second dimension is correctness. In the sense that the advice has to be based on a sound basis and fulfill at least basic meaningfulness requirements.  Not all methods and models apply to all cases. 

An interesting question arising at this point is the client's independence and capacity to be critic towards the analyst. In other terms: are we sure that we do not influence the client by just using a certain approach instead of another? And how the client can be aware of such an influence?

      Thanks to Alexander for the nice issue.

Yours, Alexis Tsoukias

PS. John Buchanan and Mordecai Henig organise a nice session in the Ankara MCDM meeting on some of these questions. We may report on this discussion there.

Alexis Tsoukias
LAMSADE - CNRS
Universite Paris Dauphine
75775 Paris Cedex 16, France

tel: +33(0)144054401
fax: +33(0)144054091
e-mail: tsoukias@lamsade.dauphine.fr
http://www.lamsade.dauphine.fr/~tsoukias

May 4, 2000

Based on the emails from Valerie and Alexis, perhaps we might consider dividing the MCDM/Multiple Objective Optimization discipline into two distinct sub-disciplines for purpose of any discussion of "real life application."

One would deal with the solution of (quantitative) problems for which a scientific and objective basis for their successful application exists (e.g., a real problem, for a real organization, with a real solution, that was really applied, and for which there are real, objective, and measurable results).

The other sub-discipline would then address (qualitative) problems in which the analyst "feels" that something "good" has happened, and the client is of the "opinion" that the discussion/consultation was "beneficial."

Both approaches can be of value -- and both deserve equal respect, but there is clearly a difference between the goals and methods of practitioners in the first and those in the second. While I work in both sub-disciplines, I haven't as yet figured out how to scientifically validate efforts in the softer area.

Jim Ignizio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Professor James P. Ignizio
University of Virginia
ignizio@virginia.edu
JUST PUBLISHED! GONE AWRY: A virtual tour through High Tech Hell
http://www.geocities.com/goneawry25
http://carolinanavy.com/navy/renaissancefleet/gypsy/james/index.html
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

May 4, 2000
Dear colleagues,
I have some comments about "Alexander V.Lotov" e-mail <lotov@CCAS.RU>.

> Dear Colleagues,
>(...) what does it mean real-life application of a MCDM technique?
>(...) a MCDM researcher may declare a real-life application of a technique (developed by him/herself) even in the case when its application is actually a demonstration that the technique can be adapted to a > particular real-life problem.
>1. ("In this case, decision makers have no idea at all that a MCDM professional is thinking about one of their problem").
>2. (...) a decision maker knows about the research, but (s)he does not take the results into account. 

Surely, it is not a correct way to deal with the problem. If it really occurs, perhaps the results will have no significance. It's trivial but very important to define:
a) the nature of the problem and the variables;
b) the set of alternatives
c) the set of criteria or rules of decision
d) the scales of the weight of the criteria and the judgements.
e) the best methodology to solve the problem.

We know the communication between the DM MCDM is not as easy as it seems. But it's one of (if not the most) the most important step of the analysis.

(...) This is an <<application where 
 i) an actual problem of an actual organization is
studied
> ii) using real data,
> iii) in which decision makers participate, and
> iv) the results of which have been implemented.>>
> (...) Is this definition full and precise? Is it instrumental?

I think, at first time, that it is a good and precise definition.

>  (...) Let us consider two techniques distinguished for their real-life application, the goal programming and the AHP.
> ("Multiple decision makers, experts and other users apply these techniques without informing authors about it. Are the problems they study actual or not? What kind of data do they use? Do decision makers participate, or experts prepare decisions by themselves, or ORMS professors develop educational examples? How about the implementation? Do the techniques influence decision processes at all?")

Well, It is very difficult to know about the all applications of the techniques, but we suppose that the information about the problem are available for anybody interested. If they (the data and information) are actual or not, we will know when contact the researchers, authors and experts - they must have responsibility, mustn't have they?

We suppose that the majority of the questions about the problems could be answered. 

> ("Important that the quality of an application in this case depends on user, but not on the technique.")
>
Again, I think the user (DM) must understand the way that the problem has been treated and which kind of technique will be used.

>  (...) A MCDM/MAUT technique is said to have real-life applications if institution(s) or particular people have mastered the technique and use it independently from the author for a relatively long time (regularly or sporadically). As you can see, both goal programming and the AHP satisfy this requirement.

Particularly, I think it's not a precise definition to a MCDM/MAUT technique that is said to have real-life applications. As we have read above, it's very difficult to know about the all applications of the techniques and if they are really well used.

Sincerely yours,
Andre Policani.

CEFET Campos/UENF
<policani@uenf.br>

May 10, 2000

Alexander Lotov and I have been having a good discussion on his original topic of real MCDM.  I used personal e-mail, because I get enough slings and arrows from journal and book reviews without asking to get hit by the entire community.  Alexander asked why I didn't use the public listserver, so I shall.

My views in a nutshell:
I think the crux of the matter is that we approach MCDM from a modeling perspective, expecting to find the "true" solution.  But human decision making is a political process.  There is no true solution.  I think that we should try to objectively find the expected impact of alternatives, but realize that these are not totally accurate, nor complete, and involve risks, and decision makers have to make their own decisions. There are a number of problems with MCDM applications I have noticed. Probably the most widely respected study in the US was by Ralph Keeney in siting a disposal area for nuclear waste.  The rather arbitrary original decision by the government was rejected, which was undoubtedly an improvement.  But the site that ended up being selected has been radically opposed by the citizens of the state in which it is located.  The facility has mostly been built, but is not going to be used for the foreseeable future, and the problem of nuclear waste is growing at nuclear plants. Keeney has himself criticized the site selected, although it was ranked as a leading contender in his original study. I personally have participated in only one major MCDM application, and that only in the initial phases.  I did notice that government decision makers were not about to commit themselves to preference analysis. My opinion is that MCDM studies are viewed by government officials as dangerous; possible support for arguments if things go right, but they don't want to commit in case things go wrong.

Alexander replied with a very interesting relation of his experiences. I don't want to put words in his mouth, but he also encountered political behavior on the part of decision makers, and a reticence to publicly express their preference.

To return to the original point of what a real MCDM study is, I would argue that real studies are flawed, and if accurately reported, wouldn't pass the review process (which unrealistically expects perfection). I think the biggest shame is that it is the flaws that are what we could learn the most from.

David Olson
Lowry Mays Professor of Business
Department of Information & Operations Management
Texas A&M University
4217 TAMUS
College Station, TX  77843-4217

(979)845-2254
FAX (979) 845-5653

May 10, 2000
David Olson, in his email notes that: 
>To return to the original point of what a real MCDM study is, I would argue that real studies are >flawed, and if accurately reported, wouldn't pass the review process (which unrealistically expects >perfection). I think the biggest shame is that it is the flaws that are what we could learn the most >from.

I certainly agree that any and all efforts to solve real world problems with mathematical tools are going to be imperfect -- and that's what I've always told my students. However, I must disagree that, "if accurately reported,(the studies) wouldn't pass the review process ..."
I've been engaged for quite some time in solving real world problems via quantitative methods (a large portion via goal programming) -- and so have many, many others (e.g., Saul Gass, Abe Charnes, Bill Cooper, etc.) in the Operational Research sector. Many of these efforts have been published, and most of them have cited their limitations along with their achievements. 
While it often wasn't easy to find journals with editors and review boards either (truly) interested in, or appreciative of real world problems, there certainly are a number of these available in our field. We all should, in fact, demand that our journals emphasize real applications.
To respond to another aspect of David's email, of course there are politics and other qualitative aspects in any attempt to solve a real problem --- but that's what Operational Research has always had to deal with since its origin. Read, for example, C.H. Waddington's book on OR in WWII, and you'll see a huge number of real life applications of OR, along with all the political impediments, flaws, and successes. We're supposed to be problem solvers, and that includes ALL the aspects of real life problem solving --- not just the models and algorithms. Or maybe I'm missing something.
Jim Ignizio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Professor James P. Ignizio
University of Virginia
ignizio@virginia.edu
JUST PUBLISHED! GONE AWRY: A virtual tour through High Tech Hell
http://www.geocities.com/goneawry25
http://carolinanavy.com/navy/renaissancefleet/gypsy/james/index.html
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
May 10, 2000
I stand corrected by Jim.  I overstated myself (as I often do).  I did not mean to question the veracity of Jim's reported applications, as I have always found him to be thorough and accurate.  What I question is the actual implementation of studies, which Jim and I agree is a political process, and which I suspect Jim agrees should be a political process.  What I was trying to say is that decision makers have to consider many things (multiple criteria), and it is rarely fruitful or appropriate to express their position on all of aspects of the problem.  I think that all of us agree with Jim that multiple criteria analysis can be very useful to decision makers in their study of real application problems.
David Olson
Lowry Mays Professor of Business
Department of Information & Operations Management
Texas A&M University
4217 TAMUS
College Station, TX  77843-4217
(979)845-2254
FAX (979) 845-5653

May 10, 2000
Dear colleagues,

Now I'd like to inform you about my part of the discussion with David Olson.

First of all, I totally agree with David that decision making is a political process. To my opinion, MCDM professionals can not hope that decision maker will use decisions we have prepared for them even in their presence and on the basis of their answers. The example with nuclear waste provided by David proves it. Moreover, I agree that real decision makers are afraid of computer tools that ask decision makers too much about their preferences. It is clear that decision maker's openness may harm him/her.

Now about my experience of a real-life application mentioned in the email of David.

In the beginning of 80s, a large research project was started at the State Planning Agency of the former Soviet Union ---  development of a computer-based decision support system for a medium and long-time planning of the national economy.  The DSS was based on application of a hierarchical system of dynamic input-output models that described the development of the USSR economy with different levels of precision. The most aggregated model (the upper level model) roughly described possible development of the USSR economy during 15 years. It was used for setting the long-term national goals. A goal was actually a desired combination of criterion values, for which consumption of goods by several population groups, development of health care and educational systems, etc were used.

In the early variant of the DSS, officials had to identify the goals on the basis of their experience, without any support. As a rule, the goals identified by them were not feasible. Then, an optimization software was used to compute a set of feasible criterion values nearest to the identified goals. Usually, the closest feasible criterion values were extremely distant from the identified goals, i.e. the identified goals had nothing to do with the reality. This result was disappointing for the officials. After small number of attempts, they refused to use the DSS. It seems that officials were afraid that the result of goal setting would be attributed to their incompetence.

For the designers of the DSS, it was clear that an additional decision support tool was needed to help officials to identify feasible goals. The designer of the upper level, Dr. Ilya Matlin did know about our Feasible Goals technique, and so in 1982 he hired me and one of my younger colleagues to develop such decision support tool. We approximated the variety of feasible criterion values (the Feasible Set in Criterion Space, the FSCS) for the upper level model. That time there was no idea about PC in the USSR, so we printed out a large album of collections of two-criterion efficiency frontiers as the slices of the FSCS and provided the album to Dr. Matlin. He told me that my direct contacts with the officials are not desirable and that he will familiarize officials with the feasible goals by himself. Then paid money to me and said "See you later". After several months I've phoned him, but he said that the DSS tool (actually, album of pictures) works great and that my help is not needed. 

In 1985, I've visited him once again to receive the document confirming the real-life application of our technique (I did need it to defend my Habilitation thesis). Simultaneously, I've asked about the results of the application. He showed the album to me. It looked dirty, greasy and crumpled. He told me that this is the poof that the album is used very intensively. I told him that we are ready to print out a new copy of the album (without being paid for it). However he declined my proposal and said that the existing copy is better. Than, I proposed to compute the FSCS once again (they have changed the model a bit and refined the data), but he refused once again saying precision is not important.

The application of the album was stopped after "perestroika" made the planning system obsolete. So, officials used the technique more than three years. It supported their better understanding of the situation. 

The question is whether this case was a real-life application (though I have a document that it was)? In accordance to the definition given in the paper by  E.Kasanen, J.Wallenius, J.Wallenius, and S.Zionts, it is not. Indeed, goal identification was not the actual problem of the State Planning Agency, but a problem related to the new DSS that didn't work. Moreover, the data were extremely bad, I can affirm it as a specialist who has spend much time in input-output analysis. Were the above officials the decision makers? Surely, they influenced somehow the decision, but the final decisions about consumption of goods, development of health care and educational systems, etc. were made by the people from the Central Committee of the Communist Party, but not by the official of the State Planning Agency. So real decision makers did not participate. Finally, about the implementation of the results. The result was a better understanding of the situation by the officials who influenced the plans prepared for the final decision. Is it possible to name it a real-life implementation?

So, as you can see my effort does not confirm the requirements of the definition of a real-life. However, officials used the support more than three years. What is extremely important, our recent applications have the same feature --- they support better understanding on the early stage of preparation of possible decision strategies (now we display the strategies related to the identified goals, too). Experts use our tools, but how they are related to the final decision? I do not know.

What do you think about it? What is your personal opinion, was it a real-life application or not?

I'd be happy to receive your answer that is very important for me.

Cordially,
        Alexander

Alexander V. Lotov, Ph.D., Doctor of Science

Head, Department of Mathematical Methods for Economic Decision Analysis
Russian Academy of Sciences, Computing Center
Vavilov street, 40, Moscow, 117967 Russia
Phone  (7-095)-135-1209       Fax  (7-095)-135-6159
E-mail  LOTOV@CCAS.RU   http://www.ccas.ru/mmes/mmeda/
and
Associate Professor, Department of Systems Analysis
Lomonosov Moscow State University

May 10, 2000

To my opinion (and let me say in the so called "european approach")  you had a real life application. As I had the opportunity to state, is the process that makes an application real and not the result (not even the implementation).

All the best, Alexis Tsoukias
Alexis Tsoukias
LAMSADE - CNRS
Universite Paris Dauphine
75775 Paris Cedex 16, France
tel: +33(0)144054401
fax: +33(0)144054091
e-mail: tsoukias@lamsade.dauphine.fr
http://www.lamsade.dauphine.fr/~tsoukias
May 10, 2000

This may not be a politically correct response to the discussion, but it is my opinion that ---- if you have to ASK whether or not an endeavor was a real life application, it probably isn't.

That doesn't mean that it wasn't an important effort, nor a useful contribution, nor not of interest to our community, nor something that provides an important lesson. But, if you've actually solved a real problem, for a real organization, and seen real implementation, you'll probably not need to ask.

Jim Ignizio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
James Ignizio
svapollo@aol.com
Free Preview: GONE AWRY - a virtual tour through High Tech Hell
http://www.geocities.com/goneawry25
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
May 11, 2000
Dear Alexander
By my criterion it is a real life application in that something was developed which helped an involved party better understand the situation it would, of course have been interesting if Dr Martin had been willing to talk more about how the album was used and/or you had been able to have involvement with the planners but this is not always possible (and is one aspect of the politics mentioned by David & others).

Best regards

Val

Valerie Belton
Management Science
University of Strathclyde
40 George Street
Glasgow G1 1QE
Tel +44 141 548 3615
Fax +44 141 552 6686

May 11, 2000

Dear Val, Jim and Alexis,
Many thanks for your comments!
I have got what I wished -- different answers. Now we can see very clear that different positions do exist.

I can answer (partially) the question of Val about how the album was used. Dr.Matlin has informed me about it in general features. Officials from the State Planning Agency loosely liked to look through the album.  But after it they were able to specify the goals that were pretty close to the feasible criterion values. Due to this Dr.Matlin managed to surmount one of the complications related to the goal procedures used by him. However, I have no idea about other complications of the procedure.  It seems we simply have provided additional information into the mess named planning.

Now about the doubts concerning political correctness expressed in the message by Jim. To my opinion, it is politically correct to express a personal view in a scientific discussion. Without it the sense of scientific discussions will be lost. I hope to receive more comments and I want to encourage those members of our society who do not consider my application to be a real-life one, to write to our MCRIT list concerning my question. 

Cordially,
        Alexander

Alexander V. Lotov, Ph.D., Doctor of Science

Head, Department of Mathematical Methods for Economic Decision Analysis
Russian Academy of Sciences, Computing Center
Vavilov street, 40, Moscow, 117967 Russia
Phone  (7-095)-135-1209       Fax  (7-095)-135-6159
E-mail  LOTOV@CCAS.RU   http://www.ccas.ru/mmes/mmeda/
and
Associate Professor, Department of Systems Analysis
Lomonosov Moscow State University

May 12, 2000
Dear colleagues,
It seems that now is a proper time to review preliminary results of our discussion.

Two different positions were expressed concerning real-life applications of MCDM techniques. 
First, there is a classic definition formulated in the paper by E.Kasanen, H.Wallenius, J.Wallenius, and S.Zionts, EJOR, v. 120, 2000, p. 497. The definition was actually supported by Jim Ignizio.

Indeed, the classic definition is very convenient, especially in the case one has to review a paper submitted to a journal. On the other hand, the classic definition can be applied only in the case if a MCDM professional can control or at least monitor the situation. (S)he must have some kind of evidence that a decision obtained through the application of a MCDM technique was implemented. This is a seldom case.

On the other hand there exist multiple cases where MCDM techniques are applied, but without a control of the author of a method or any other MCDM professional responsible for documenting the process. Many applications do exist in this form. Surely, it can happen that experts and decision makers do use MCDM techniques in distorted forms. Moreover, final decision making can be a political decision that has nothing to do with MCDM tools. As I can understand, position of this kind was expressed by Val Belton, David Olson and Alexis Tsoukias.

Alexis formulated a brilliant concept of <<client's satisfaction>>:
<<the process was satisfactory because enabled the client to understand better his problem. Therefore satisfaction refers to the decision aiding process and not to its result>>. It seems that real-life applications of important tools like goal method and the AHP often satisfy the definition given by Alexis, but not the classic definition.

I can add that a MCDM technique can be used not on a final stage, but on an early screening phase of the decision process. It means that the decisions found on this phase (usually multiple decisions at once!) are subject for further testing, correction, refinement, further selecting, etc. In this case it is complicated or even impossible to prove the implementation of results of the application of a MCDM tool. In addition, decision makers can apply several tools simultaneously. For example, I know about an engineering institution in Russia that uses the AHP and our Visual Market simultaneously for selecting preferable projects from a database of alternative projects. The selected projects are discussed then in an informal way to select a final project. By the way, the early screening phase seems to be the most promising one for real-life application of the MCDM techniques.

What can be done in this situation? To my opinion we have to use both definitions. In the case of the decision process controlled by a MCDM professional we can use the definition of E.Kasanen, H.Wallenius, J.Wallenius, and S.Zionts, i.e. a real-life application is an <<application where
i) an actual problem of an actual organization is studied
ii) using real data,
iii) in which decision makers participate, and
iv) the results of which have been implemented.>>

In the case of decision process where there is no control of a MCDM professional, or the control is only partial, or a MCDM technique is used on the early screening stage or just to improve the understanding of decision makers or experts, we can apply another definition based on Alexis idea, i.e. a real-life application is an application where
<<the application process was satisfactory because enabled the client to understand better his/her problem>>.

Surely, the problem of correctness does exist, as it was soon discussed by Alexis. It is an extremely complicated problem. I can narrate a funny story about a real-life application of our Feasible Goals method. One enterprising person has applied the FGM to a real-life water management problem. However, instead trying to implement the strategy he has developed, he printed out multiple color copies of decision maps displayed in the framework of the FGM and distributed them among influential officials of the Russian Federal Ministry for Agriculture. As the result, he has got a new position as a department head at a research institution of the Ministry plus a financial support for development of several projects. Surely I encourage people to use the FGM in this way, too, but as an intermediate step only. Hopefully the above person will need to use MCDM techniques in real-life applications now.

Was this a correct application of the MCDM technique? I think not yet, but the client was satisfied.

So, the above definition is still not precise, but I hope we would elaborate it sometime. Alexis has proposed to discuss the issue at the Ankara MCDM meeting at a session organized by John Buchanan and Mordecai Henig. It is a great idea! We can prepare ourselves to the meeting continuing the discussion on email.

I'd like to stress another aspect of the problem. In the process of the developing the definition, we should not think too much whether the definition is convenient for reviewers of the papers. Publication of a paper is not a final goal of a MCDM research. I utterly agree with David Olson that the <<real studies are flawed, and if accurately reported, wouldn't pass the review process (which unrealistically expects perfection).  I think the biggest shame is that it is the flaws that are what we could learn the most from>>. For example, I never had an idea to publish my experience at the State Planning Agency I've described in my previous messages. The reason is clear --- this is not a subject of journal publication and will be rejected by reviewers.  INTERFACES publishes experience of application of OR/MS tools, but they prefer successful stories, but not <<the flaws that are what we could learn the most from>>. Perhaps, we could start an Internet journal or discussion list related to "not classic" real-life applications of MCDM techniques, possible flaws and negative experience? It could be very helpful for our society.

What do you think about the Internet journal? And how to improve the above draft of the definition?

Cordially,
        Alexander

Alexander V. Lotov, Ph.D., Doctor of Science

Head, Department of Mathematical Methods for Economic Decision Analysis
Russian Academy of Sciences, Computing Center
Vavilov street, 40, Moscow, 117967 Russia
Phone  (7-095)-135-1209       Fax  (7-095)-135-6159
E-mail  LOTOV@CCAS.RU   http://www.ccas.ru/mmes/mmeda/
and
Associate Professor, Department of Systems Analysis
Lomonosov Moscow State University

May 14, 2000
Dear Colleagues and Friends,
I have the feeling that several terms should be used to represent the different objects which were mentioned or commented in the previous messages. Here is a non-exhaustive list of such "objects":
Object A: description of a situation where a scientist used a multicriteria method to work on real data, in view to help a client to solve a decision problem.
Object B: object A where the approach was well-founded on a theoretical point of view.
Object C: object A which led to new methodological developments.
Object D: object A where the client was satisfied by the work done by the >scientist (several reasons are possible).
Object E: object A where the result was effectively implemented.
Object F: object A which led to an important evolution in the decision process of an organisation,...etc

A possible terminology for these objects could be the following (English natives will certainly propose better terms): we only have to agree on a precise terminology in order to avoid misunderstandings and useless discussions.
A = real-life application (rla).
B = well-founded rla.
C = methodologically interesting rla.
D = satisfying rla.
E = successful rla.
F = revolutionary rla....etc

Of course, this list should be completed and precised and, probably, the terms should be changed. We could try to build a sort of dictionary around this subject in order to prepare the discussion in Ankara.

For our community, it can be interesting to be informed on as many rla's as possible to improve our experience and to convince potential clients (or colleagues) that our tools are effectively used. So, I like very much the idea of an Internet Journal or a Web Site built as a library and a discussion room about rla's.

Another question is "when does a rla justify a publication in a scientific journal". Of course, this depends on the editorial strategy of the journal. The ideal is to have a well-founded methodologically interesting revolutionary rla, but it is clear that we can also learn from more modest contributions.

Every comment is welcome,
Ph.Vincke.

Ph.VINCKE
Universite Libre de Bruxelles, SMG – ISRO
Boulevard du Triomphe, 1050 BRUXELLES
TEL: 32 2 6505889 ou 5885
FAX: 32 2 6505970
E-MAIL professionnel: pvincke@smg.ulb.ac.be
E-MAIL prive: pv@smg.ulb.ac.be
http://smg.ulb.ac.be/personal/PVincke.html
   SEE ALSO  http://smg.ulb.ac.be/

May 15, 2000
Dear colleagues,

Here is a small contribution to the discussion on real-life applications.

How do engineers build bridges? They use the most sophisticated stability models and perform zillions of operations on super computers. When they finally have the solution to their equations, they do not implement it; they multiply all sections by a safety coefficient equal to 2! Of course, this is caricatural but not so far from reality.

Shall we say that stability models have no real-world or real-life applications because the solution given by the model is never used? No one would dare say so. Thanks to the stability model, the engineer knows that his project is feasible, that it does not have weak points, how the sections should vary from point to point, that some "consistency" conditions are satisfied... But, finally, he knows he should not really trust the model. His experience, his teachers, his friends and his colleagues taught him a list of rules of thumb. And he somehow adapts the solution prescribed by the stability model, according to his experience, these rules of thumb and his intuition.

Why should it be different with MCDM models ? If a decision-maker doesn't implement the decision prescribed by the model but if he has been in contact with the model (directly or via people reporting to him or interacting with him) his final decision is probably influenced by the model. Then it is a real-life application. Of course, the degree to which the model influenced the final decision, in a way or in another, can greatly vary. Did the DM spend 10 minutes or 10 days in contact with the model. Did he use it himself, did he work with an analyst or did he quickly read a report written by an employee?  Hence, some applications can be more real-life than some others but it has nothing to do with the fact that the solution has been implemented.

In my opinion, an application where the solution is implemented is probably not a good one (even though it is real-life). I prefer an application where the DM gains insight into his problem during the decision-aiding process but where the DM is critical enough to take the solution with a pinch of salt.

Sincerely,

Thierry Marchant

Thierry Marchant
Ghent University              <-------  New address  !!!
Department of Data Analysis
H. Dunantlaan 1
B-9000 Gent BELGIUM
Tel: +32/9-264.63.73
Fax: +32/9-264.64.87
<mailto:Thierry.Marchant@rug.ac.be>
http://allserv.rug.ac.be/~tmarchan/
May 16, 2000

I agree with my Flemish colleague.
Ph.Vincke

May 22, 2000
Dear colleagues,
One week ago we have received two extremely interesting messages from Belgium.

Thierry Marchant gave a nice example from the field of engineering that shows how versatile real-life applications can be.

By the way I could add several funny stories about real-life applications that seem to be fairly strange, but fruitful. I hope to share them in Ankara.

Therefore the draft classification of applications proposed by Philippe VINCKE seems to be extremely important and interesting. I'm sure that we will discuss the proposal in Ankara, but the Conference is very short, and so we have to prepare to it during one and half months left.

I tried to develop my comments to the classification by Philippe VINCKE. As the result I can propose two new objects (objects H and I). They differ from the objects proposed by Philippe VINCKE since they are not based on object A. Object A is related to the <<situation where a scientist used a multicriteria method to work on real data, in view to help a client to solve a decision problem>>. I can see many applications (including engineering applications described by Thierry Marchant, the AHP, goal methods and applications of my method) where OR/MS specialists are excluded from the decision process. Engineers make decisions (develop projects) in the example of Thierry Marchant without any help. Bureaucrats and the experts from their institutions (economists and water managers, but not DSS specialists) use the multicriteria tool in my examples by themselves.

So I'd like to formulate 
Object H: situation where stakeholders and their experts used a multicriteria method to work on real data; 
and
Object I: object H where stakeholders and their experts use a multicriteria method for a long time.

What do you think about the objects?
Cordially,
        Alexander Lotov

May 22, 2000
Object H is object A where the client (and his experts) is the same person as the scientist. However, I agree to distinguish it and to try to collect information about this kind of real-life application, even if it will be very more difficult because it may happen that the involved people do not have any contact with our associations.

Ph.Vincke.

May 22, 2000
Surely, we have to discuss all the words used in our definitions, especially such important notions as <<scientist>>. As to the people that have no contact with our associations, we have to deal with them in the case we want to deal with true real-life applications.
Cordially,
        Alexander