Presenting Qualitative And Quantitative Data Essay

Research methods are split broadly into quantitative and qualitative methods.

Which you choose will depend on your research questions, your underlying philosophy of research, and your preferences and skills.

Our pages Introduction to Research Methods and Designing Research set out some of the issues about the underlying philosophy.

This page provides an introduction to the broad principles of qualitative and quantitative research methods, and the advantages and disadvantages of each in particular situations.

Some definitions


Quantitative research is “explaining phenomena by collecting numerical data that are analysed using mathematically based methods (in particular statistics).”*

Qualitative research seeks to answer questions about why and how people behave in the way that they do. It provides in-depth information about human behaviour.

* Taken from: Aliaga and Gunderson¬†‘Interactive Statistics ‘3rd Edition (2005)


Quantitative Research

Quantitative research is perhaps the simpler to define and identify.

The data produced are always numerical, and they are analysed using mathematical and statistical methods. If there are no numbers involved, then it’s not quantitative research.

Some phenomena obviously lend themselves to quantitative analysis because they are already available as numbers. Examples include changes in achievement at various stages of education, or the increase in number of senior managers holding management degrees. However, even phenomena that are not obviously numerical in nature can be examined using quantitative methods.

Example: turning opinions into numbers


If you wish to carry out statistical analysis of the opinions of a group of people about a particular issue or element of their lives, you can ask them to express their relative agreement with statements and answer on a five- or seven-point scale, where 1 is strongly disagree, 2 is disagree, 3 is neutral, 4 is agree and 5 is strongly agree (the seven-point scale also has slightly agree/disagree).

Such scales are called Likert scales, and enable statements of opinion to be directly translated into numerical data.

The development of Likert scales and similar techniques mean that most phenomena can be studied using quantitative techniques.

This is particularly useful if you are in an environment where numbers are highly valued and numerical data is considered the ‘gold standard’.

However, it is important to note that quantitative methods are not necessarily the most suitable methods for investigation. They are unlikely to be very helpful when you want to understand the detailed reasons for particular behaviour in depth. It is also possible that assigning numbers to fairly abstract constructs such as personal opinions risks making them spuriously precise.

Sources of Quantitative Data

The most common sources of quantitative data include:

  • Surveys, whether conducted online, by phone or in person. These rely on the same questions being asked in the same way to a large number of people;
  • Observations, which may either involve counting the number of times that a particular phenomenon occurs, such as how often a particular word is used in interviews, or coding observational data to translate it into numbers; and
  • Secondary data, such as company accounts.
Our pages on Survey Design and Observational Research provide more information about these techniques.

Analysing Quantitative Data

There are a wide range of statistical techniques available to analyse quantitative data, from simple graphs to show the data through tests of correlations between two or more items, to statistical significance. Other techniques include cluster analysis, useful for identifying relationships between groups of subjects where there is no obvious hypothesis, and hypothesis testing, to identify whether there are genuine differences between groups.

Our page Statistical Analysis provides more information about some of the simpler statistical techniques.

Qualitative Research

Qualitative research is any which does not involve numbers or numerical data.

It often involves words or language, but may also use pictures or photographs and observations.

Almost any phenomenon can be examined in a qualitative way, and it is often the preferred method of investigation in the UK and the rest of Europe; US studies tend to use quantitative methods, although this distinction is by no means absolute.

Qualitative analysis results in rich data that gives an in-depth picture and it is particularly useful for exploring how and why things have happened.

However, there are some pitfalls to qualitative research, such as:

  • If respondents do not see a value for them in the research, they may provide inaccurate or false information. They may also say what they think the researcher wishes to hear. Qualitative researchers therefore need to take the time to build relationships with their research subjects and always be aware of this potential.
  • Although ethics are an issue for any type of research, there may be particular difficulties with qualitative research because the researcher may be party to confidential information. It is important always to bear in mind that you must do no harm to your research subjects.
  • It is generally harder for qualitative researchers to remain apart from their work. By the nature of their study, they are involved with people. It is therefore helpful to develop habits of reflecting on your part in the work and how this may affect the research. See our page on Reflective Practice for more.

Sources of Qualitative Data

Although qualitative data is much more general than quantitative, there are still a number of common techniques for gathering it. These include:

  • Interviews, which may be structured, semi-structured or unstructured;
  • Focus groups, which involve multiple participants discussing an issue;
  • ‘Postcards’, or small-scale written questionnaires that ask, for example, three or four focused questions of participants but allow them space to write in their own words;
  • Secondary data, including diaries, written accounts of past events, and company reports; and
  • Observations, which may be on site, or under ‘laboratory conditions’, for example, where participants are asked to role-play a situation to show what they might do.
Our pages on Interviews for Research, Focus Groups and Observational Research provide more information about these techniques.

Analysing Qualitative Data

Because qualitative data are drawn from a wide variety of sources, they can be radically different in scope.

There are, therefore, a wide variety of methods for analysing them, many of which involve structuring and coding the data into groups and themes. There are also a variety of computer packages to support qualitative data analysis. The best way to work out which ones are right for your research is to discuss it with academic colleagues and your supervisor.

Our page Analysing Qualitative Data provides more information about some of the most common methods.

It’s your research‚Ķ

Finally, it is important to say that there is no right and wrong answer to which methods you choose.

Sometimes you may wish to use one single method, whether quantitative or qualitative, and sometimes you may want to use several, whether all one type or a mixture. It is your research and only you can decide which methods will suit both your research questions and your skills, even though you may wish to seek advice from others.

Volume 7, No. 4, Art. 9 – September 2006

Secondary Analysis and Culture of Disputation in European Qualitative Research

Magdalini Dargentas

Conference Essay:

Secondary Analysis in Qualitative Research—Utopia and Perspectives. International Symposium. Maison des Sciences de l'Homme, Alpes, Grenoble, France, 3-4 November 2005, organized by CAPAS1)

Abstract: This essay reviews the Symposium on Secondary Analysis in Qualitative Research—Utopia and Perspectives. The Symposium, which was held in November 2005, was the first academic meeting in France to focus on qualitative research, mainly through secondary analysis and archiving. Researchers from various European countries, academic fields and working contexts discussed their practices in qualitative research and secondary analysis. As secondary analysis is almost nonexistent in France, this Symposium has offered researchers an opportunity to become familiar with this method, to meet European experts, and to engage in constructive dialogues. This paper initially places qualitative research, secondary analysis and archiving in an international perspective. It then relates these practices to the current situation in France and introduces the organization of the Symposium. Next it describes scholars' contributions and areas of study under reflection throughout this event. In addition, it discusses several directions the field of secondary analysis is currently moving in, as well as qualitative methods. The main contributions of this Symposium deal with the issues of cumulative knowledge, the need to take an epistemological turn, the institutionalization of qualitative research in France, as well as academic disciplines' growing need to reflect on the traditions and standards of qualitative research.

Key words: qualitative research, secondary analysis, archiving, methods, ethics, practices, epistemology, theory, interviews

Table of Contents

1. Qualitative Methods and Secondary Analysis in International Research

2. The State of Qualitative Research and Secondary Analysis in France—The Recent Interest in Secondary Analysis and the Organization of the Symposium

3. Contents and Overview of the Symposium Secondary Analysis in Qualitative Research—Utopia and Perspectives

3.1 Keynote address: Louise CORTI: Qualidata in the UK

3.2 First session: Qualitative research

3.3 Second session: Practices around secondary analysis and archiving

3.4 Third session: Ethical and legal issues

3.5 Fourth session: Methods and tools assisting secondary analysis

3.6 The roundtable discussion closing the Symposium

4. Discussion of Topics Raised at the Symposium: A Field Undergoing Evolution

4.1 Re-analysis vs. secondary analysis

4.2 Theoretical gains

4.3 Methodological and epistemological issues

4.4 Deontological and legal issues

4.5 Infrastructure, archiving, researchers

5. Concluding Remarks and Future Directions

5.1 Discussing the Symposium

5.2 Future issues to explore

5.3 The academic context in France—Future events

Notes

References

Author

Citation

 

1. Qualitative Methods and Secondary Analysis in International Research2)

In recent years, we have seen major changes in the use of qualitative methods in the social and human sciences. Interest in these methods has grown rapidly in many academic disciplines, mainly in the Anglo-Saxon research communities. We have observed the ever expanding publication of journals or academic networks3) that embrace qualitative research. And the number of textbooks dedicated to these methods is growing rapidly (see SEALE, GOBO, GUBRIUM & SILVERMAN, 2004). [1]

Moreover, "new" methods for dealing with qualitative data have emerged, namely archiving and secondary analysis; even though these practices have been known since the 1960s and used mainly for quantitative data (STEWART & KAMINS, 1993). It is only since the 1990s that there has been a systematic reflection on archiving and re-use of qualitative data, alongside the creation of national institutes dedicated to training, methodological assistance, or archiving of major research data sets. In Europe, the British Archival Resource Centre Qualidataplays an important role in disseminating these new practices, in assisting and training researchers, as well as in coordinating European networks (CORTI, 2000). [2]

The field of Secondary Analysis, which appears to be closely related to Health Studies, is often associated with Grounded Theory, and with the use of software for analytical process (CAQDAS). Yet secondary analysis and archiving question traditional practices in qualitative research, such as: the relationship to interviewees, confidentiality, and the status of data and research possibilities (FIELDING, 2000; HINDS, VOGEL & CLARKE-STEFFEN, 1997; MAUTHNER, PARRY & BACKETT-MILBURN, 1998). Many debates have taken place regarding these new practices in Anglo-Saxon countries in Northern Europe or Northern America. Colleagues there have organized various workshops, conferences, and symposia on secondary analysis and archiving (see Box 1). One can see here a growing number of papers and books published on those practices (see BERGMAN & EBERLE, 2004; CORTI, KLUGE, MRUCK & OPITZ, 2000; CORTI, WITZEL & BISHOP, 2005; CORTI & THOMPSON, 2004; HEATON, 2004).

International Events

2004:

Secondary Analysis of Qualitative Data. Amsterdam, The Netherlands, 17-20 August 2004. Chaired Sessions at the Sixth International Conference on Social Science Methodology (RC33 – Research Committee on Logic and Methodology), International Sociological Association (ISA); sessions organized by ESDS Qualidata.

2003:

Everything You Wanted to Know about Processing Qualitative Data but Were Afraid to Ask. University of Ottawa, Canada, 27-30 May 2003. ESDS Qualidata Workshop at IASSIST (International Association for Social Science Information Service and Technology), 29th Annual Conference.

2002:

Social Science Archive and Resource Center for Qualitative Research in Switzerland. Neuchâtel, Switzerland, 26-27 April 2002. Invitational Workshop and Conference, organized by the Swiss Academy for Humanities and Social Sciences (Social Science Policy Council) and the Swiss Information and Data Archive Service for the Social Sciences (SIDOS).

2000:

Secondary Analysis of Qualitative Data. Köln, Germany, 3-6 October. Chaired Sessions at the Fifth International Conference on Social Science Methodology (RC33 – Research Committee on Logic and Methodology), International Sociological Association (ISA); sessions organized by ESDS Qualidata, German Social Science Infrastructure Service (GESIS), Zentralarchiv für Empirische Sozialforschung (ZA).

 

British National Workshops

2006:

Exploring the Potential: Examining Archived Data at Mass Observation and ESDS Qualidata. ESDS Qualidata, University of Essex, 19 January 2006.

2005:

Re-using Qualitative Data. CRESC Methods Workshop, Qualitative Research Laboratory, The University of Manchester. ESRC – CRESC (Economic and Social Research Council, Centre for Research on Socio-Cultural Change), 28 September 2005.

2004:

Secondary Analysis of Qualitative Data: Using Atlas-ti to Explore Archived Sources. ESDS Qualidata, UK Data Archive, University of Essex, 24 June 2004.

Box 1: Selection of academic events on archiving and secondary analysis of qualitative data (International events and British national workshops) [3]

Despite the limits of qualitative secondary analysis, the development of such a practice appears to be facilitated by the current development in social and human sciences and technological achievements that allow the creation of databases. This kind of analysis is also part of other innovative research practices, such as the use of the Internet for research, creation of on-line journals, participating in interactive communities of research, and so on. Furthermore, financial issues and theoretical claims imply a re-examination of qualitative material. A special emphasis is given by researchers on the impossibility to exploit entirely issues emerging in specific material. Therefore, research can benefit by re-analyzing the same material under other theoretical and methodological angles. Secondary analysis is finally associated with growing collaborative research. However, in spite of the gains brought by this method, it remains novel. It is used by a minority of researchers and is still expected to prove its utility in research and to propose ways of dealing with problematic issues, such as the context of research, ethics, and relationships with the respondents. The factors that will supposedly enhance its future development include: the expansion of archiving centers; analysis software training; and reflection on common standards of qualitative research, as well as on theoretical, methodological and epistemological issues (BACKHOUSE, 2004; CORTI & THOMPSON, 2004; FIELDING, 2004; HEATON, 1998, 2004; PARRY & MAUTHNER, 2004; SANTACROCE, DEATRICK & LEDLIE, 2000; THORNE, 1998). [4]

2. The State of Qualitative Research and Secondary Analysis in France—The Recent Interest in Secondary Analysis and the Organization of the Symposium

The landscape of qualitative research has evolved differently in France. For instance, there are no associations directly related to qualitative research which could unite researchers using these methods. [5]

The only association we can refer to is "réseau ARCATI" (Atelier-Réseau Coopératif pour Analyses Textuelles Informatisées). This association is dedicated to the study of various types of software for the analysis of qualitative data and of their methodological and theoretical implications. It is mainly funded by scholars from sociology.4) [6]

There are also other associations linked to textual data analysis. Let us mention the following ones, editing three on-line journals. They are directly related to the field of linguistics:

  • Lexicometrica is an on-line journal publishing studies on textual analysis using lexicometric, statistical methods originated from linguistics. The network of researchers editing this journal is organizing a bi-annual conference, namely JADT—Journées d'Analyse Statistique des Données Textuelles (Textual Data Statistical Analysis).

  • Texto! is an on-line network and journal related to textual semantics, to meaning and interpretation. It mainly concerns linguistics and hermeunetics. It is directed by François RASTIER.

  • CORPUS is an on-line journal publishing studies on corpus linguistics and concerned by theoretical, methodological and epistemological issues. [7]

No major academic event has been organized up to now and there are no journals focused solely on qualitative methods. The only periodical we can propose is, Langage & Société, which is indirectly linked to qualitative research, as it stresses the use of language or the way it is treated in different disciplines, such as anthropology, sociology, psychology, and history. Although this element is significant for the orientation of qualitative research in France, the primary focus of this journal remains on the domain of socio-linguistics. Even though in recent years we have seen the publication of major papers and books on this topic from scholars in various disciplines, the global scientific production around qualitative methods seems limited when compared to that of the Anglo-Saxon academic circles (see BLANCHET, GHIGLIONE, MASSONAT & TROGNON, 1987; BLANCHET & GOTMAN, 1992; DEMAZIERE & DUBAR, 2004; DUCHESNE & HAEGEL, 2004; JODELET, 2003; KAUFMANN, 1996; LABORIER & BONGRAND, 2004; MUCCHIELLI, 2005; SANTIAGO-DELEFOSSE & ROUAN, 20015)). [8]

Moreover, archiving and secondary analysis have only recently begun to interest scholars in France. Since 1998 GRETS6) is the sole social science research group that employs systematic archiving and secondary analysis in an applied area (LE-ROUX & VIDAL, 2000a, 2000b; DARGENTAS & LE-ROUX, 2005). This research group fashioned these methods after the experience of Qualidata. Another group of researchers, those of CIDSP7), which has been devoted to archiving quantitative data in the field of social and political sciences, is currently considering the extension of this activity to qualitative data. Both those research teams (GRETS and CIDSP) have been mutually engaged, since 2004, in a research program dedicated to new ways of dealing with qualitative data in the social and human sciences, namely archiving and secondary analysis (CAPAS). This collaboration is multidisciplinary, as members of those teams belonging to the group CAPAS come from linguistics, sociology, political science, and social psychology. The main objective of this group is to enhance scientific reflection related to qualitative methods, to develop secondary analysis in France, to enlarge the methodological and theoretical experience of GRETS, and to undertake a profound epistemological reflection on this method. [9]

The organization of a symposium was part of this research program. The Symposium Secondary Analysis in Qualitative Research—Utopia and Perspectives is the first academic event in France not only to examine this method but also to be connected to qualitative research. It was held in Grenoble on November 3rd and 4th, 20058). The main goal of this academic event was to exchange experiences, concerns and thoughts on secondary analysis with the larger scientific community in an interdisciplinary and international context. Thus, it seemed important to assemble scholars engaging in different practices of qualitative research, as well as having different experiences with and knowledge about secondary analysis. It was also an occasion to promote archiving and secondary analysis in France, as well as to initiate the creation of a French network which would be bonded to the existing European ones. Another aim was to present this method to young researchers. Examining qualitative material through interviews, the Symposium focused on the following issues: the state-of-the-art in qualitative research, practices of secondary analysis, topics of analysis, ethical and legal problems related to archiving and secondary analysis, methods used in preparing or assisting with secondary analysis, and theoretical and epistemological perspectives of secondary analysis. A special emphasis was given to Anglo-Saxon practices, to the beginnings of French secondary analysis, and to the specific methodological habits of various academic fields in France. [10]

This event gathered some 20 key-figures in the field of secondary analysis and qualitative research. Many these experts and scholars came from European countries other than France. On the contrary, French researchers were mainly experts on qualitative methods and had little experience on secondary analysis. They were mostly invited to reflect on the compatibility of secondary analysis with their academic background. Scholars from various disciplines—sociology, political science, social psychology, linguistics, statistics, and law—came from the United Kingdom, Finland, Germany, Switzerland, and France. In addition, there was a mix of academic and applied researchers. [11]

The audience consisted of some 80 people, including PhD candidates and researchers, who use qualitative methods within different domains of the social and human sciences. Almost all were French, while the majority of young researchers in the audience came from political science, which is not surprising given that one of the institutions organizing the Symposium, CIDSP, is within the field of political science. The languages used during the Symposium were French and English; translation was provided, facilitating exchanges between scholars. [12]

3. Contents and Overview of the Symposium Secondary Analysis in Qualitative Research—Utopia and Perspectives

The Symposium was held over a period of two days and consisted of one keynote conference and three paper sessions dedicated to: qualitative research, secondary analysis and archiving, ethics and legal issues, and tools of secondary analysis. A roundtable discussion brought the Symposium to a close. Even though each session had a specific topic, they were not exclusive, and it was common to have papers dealing with more than one topic at a time. Hereafter is a more extended description of the issues and contributions brought forth during these two days. The description of papers is made in chronological order. [13]

3.1 Keynote address: Louise CORTI: Qualidata in the UK

The first day opened with keynote speaker, Louise CORTI, Manager of a major institution for the development of archiving and secondary analysis of qualitative data, that of UK Data Archive (Qualidata, University of Essex). Her address was entitled Approaching Utopia: the maturing of qualitative data archiving services and secondary analysis in the UK. In her paper CORTI gave an overview of the development and work of Qualidata Resource Centre since 1994. She stressed the role of archiving that is relevant to the enhancement of qualitative data, making possible their further exploitation or avoiding their physical loss. She also dealt with the importance of accompanying researchers and the role of Qualidata for promoting secondary analysis and archiving in Europe (see also CORTI, 2000). [14]

3.2 First session: Qualitative research

This session centered on the state-of-the-art of qualitative research, focusing on the resurgence of qualitative research and on research practices. Scholars from various backgrounds were invited to discuss specifics of the use of qualitative methods in their work and in their academic fields. [15]

This session commenced with a noteworthy paper from Nigel FIELDING (University of Surrey). He described the evolution of the status of qualitative methods, developed a detailed history of their use and underlined their current institutionalization. This overview gave him the opportunity to approach issues such as: the debate between qualitative and quantitative methods, the qualitative turn and resurgence of qualitative methods, new developments in software, and attempts to define research standards. [16]

Other scholars presenting papers in this session came from the French academic context. Pascale LABORIER (Centre Marc Bloch9), Berlin) and Philippe BONGRAND (CURAPP10), Université d'Amiens) paid particular attention to the field of public policy analysis. They described the increased use of interviews in doctoral dissertations in their discipline and emphasized the misuses of those methods, such as when interviews are used in a non-systematic way, without means of systematizing, such as recording, and are often analysed only on an informative level, ignoring other phenomena. Sophie DUCHESNE (CEVIPOF11), CNRS) represented the field of political science. She described the status of qualitative material in a major French institution in political science, CEVIPOF. The extensive and rich work of key researchers in this field, such as Guy MICHELAT and Michel SIMON, is available but abandoned in archives and unexploited. DUCHESNE also discussed secondary analysis, describing the evolution of her own attitude towards this method, and giving emphasis to the relationship between researchers and interviewees. From sociology Dietmar LOCH (Université Pierre Mendès France, Grenoble) presented specifics about the use of qualitative methods in the study of immigrants and used his own research on this topic to describe the role of qualitative methods in understanding actors' views. He focused on the importance of context in the process of data collection and analysis. In LOCH's opinion, the importance of context in primary research limits the validity of secondary analysis regarding the understanding of actors' views. [17]

3.3 Second session: Practices around secondary analysis and archiving

The first day closed with another session on secondary analysis and archiving which focused on the variation of practices in European and French research. The first presentation of this session was given by Libby BISHOP (UK Data Archive, University of Essex). In her paper, BISHOP presented a secondary analysis on convenience foods. She questioned the dichotomy of primary and secondary analysis. Other topics addressed were relevant to the process of analysis and the relationship of the researcher to the interviewee as well as to data. BISHOP emphasized the process of research and issues common in any qualitative inquiry, such as relationships with respondents, co-construction of data, the issue of consent, and the research context. [18]

The following two presentations were given by German scholars and were concerned with the state of secondary analysis and archiving of qualitative data in their country. Irena MEDJEDOVIĆ (Archiv für Lebenslaufforschung, Graduate School of Social Sciences, Bremen) presented results of a nation-wide feasibility study regarding researchers' experiences, demands, objections and fears about archiving and secondary analysis. In spite of the different issues impeding the development of those methods (such as confidentiality, fear of criticism, data fit, context, etc), it seems that German scholars argue for a "culture of disputation" through qualitative secondary analysis and view this method as challenging their own interpretation. Andreas WITZEL (Archiv für Lebenslaufforschung, Graduate School of Social Sciences, Bremen) and Reiner MAUER (Zentral Archiv für Empirische Sozialforschung, Köln) described the qualitative archiving situation in Germany and drew out conditions for the future creation of a nation-wide archive. In Germany, while qualitative archiving is already undertaken by the Life Course Archive (Archiv für Lebenslaufforschung), the national archive for empirical social research (Zentral Archiv) limits archiving to quantitative data. The feasibility study revealed that researchers would support such a creation of qualitative archiving and that they would be interested in re-using data. For the two authors the national service for qualitative data should follow the model of the British Qualidata, mainly as intermediary for decentralized archives, providing consulting and training (see also, MEDJEDOVIĆ & WITZEL, 2005; OPITZ & MAUER, 2005). [19]

The following two papers related experiences doing secondary analysis in France. Magda DARGENTAS (PACTE/R&D-EDF) and Dominique LE-ROUX (R&D-EDF) described the uniqueFrench experience with archiving and secondary analysis in the social sciences research group of EDF. They focused on the characteristics of the secondary analyses undertaken (e.g., large numbers of interviews re-analyzed from numerous primary studies, applied perspective of analysis, co-operation between researchers, reflexivity, use of various types of software for analysis: French programs such as Alceste, Tropes, vs. Anglo-Saxon CAQDAS such as Atlas.ti). Dominique BEYNIER (Université de Caen, Centre Maurice Halbwachs) presented two examples of secondary analysis on the topics of breast-feeding (2004) and profession definitions for social workers (2005). This scholar is among the very few academics in France to re-analyze data. Working in the field of Sociology of Health, he has, for a long time, archived extensively and re-analyzed his students' data. Acknowledging re-analysis as "secondary analysis", translates for BEYNIER the need of researchers to question the implications and the process of data re-analysis methods. He has also used mixing methods (qualitative and quantitative) in data analysis; in his paper, he defended the complementarity between those approaches. BEYNIER also emphasized the impossibility of doing exhaustive analysis of empirical material in the first analysis. [20]

3.4 Third session: Ethical and legal issues

The second day of the Symposium opened with a session on ethics and legal issues. Arja KUULA (Finnish Social Science Data Archive) considered the relationship of the researcher to the interviewee, data property, and researchers' fears of harming interviewees by misusing data. She shared the conclusions of a study that indicated most people who had participated in studies would allow their interviews to be re-used. KUULA underlined the need to re-define the relationship between researcher and interviewee only in relation to the research objectives. She emphasized the importance of respondents' anonymization during publication rather than data analysis and stressed the need to obtain respondents' consent before processing or publicizing their data. [21]

Anne-Marie BENOIT (CIDSP-PACTE) questioned the current French juridical status of interview archiving and re-analysis. She addressed questions regarding the solutions to protect participants, while optimizing use of data-sets. She stated on the necessity of obtaining participants' consent. This led her to approach the appropriation of data by the researcher, emphasizing the notion that the author is the interviewee and not the researcher. She also proposed the elaboration of a practice guide, including qualitative research standards, by scholars in collaboration with the national body controlling databases (CNIL, Commission Nationale de l'Informatique et des Libertés). [22]

3.5 Fourth session: Methods and tools assisting secondary analysis

Another session on methods and assisting tools to secondary analysis took place. Jacques JENNY (Iresco, co-founder of ARCATI) questioned the production of discourse and the process of analysis, inspired by sociolinguistics and pragmatics. He defined discourse as a co-enunciated system in relation to representations, values, social and discursive practices, and emphasized the temporal dimension of discourse. In addition, he questioned methods of analysis as to their theoretical and methodological principles and criticized lexicometric methods of analysis. [23]

Nigel FIELDING began his paper by reminding us of typologies of the ways of doing secondary analysis. He approached qualitative research specificities (such as reflexivity, data richness, diverse factors influencing phenomena under study, etc) and examined the contribution of CAQDAS in coding while stressing the importance for researchers to be aware of their analytical perspectives. He underlined the importance of knowledge accumulation and the need for the social sciences to undertake an epistemological turn, thus facilitating new ways of scientific labor (sharing data, cyber-research, re-analyzing, meta-analysis, etc). [24]

Sylvaine NUGIER (R&D-EDF) presented a current project on software development in the social sciences research group of EDF. This software aims at structuring archived data and meta-data coming from diverse databases and preparing them for further analyses. In this development, heterogeneous databases from the R&D department in EDF were used, including interviews, questionnaires, web forums, web pages. She also discussed the contribution of RDF formats in this kind of software. [25]

3.6 The roundtable discussion closing the Symposium

The Symposium closed with a roundtable. Although this was entitled "Re-analysing interviews: Epistemological issues", there was a variety of issues discussed outside this frame. Louise CORTI and Libby BISHOP focused on the need to develop archiving as a condition to the use of qualitative secondary analysis. Dominique JOYE (SIDOS) spoke about transparency in qualitative research, the usefulness of secondary analysis in longitudinal topics such as social change and difficulties linked to archiving. He also underlined the false opposition between quantitative and qualitative methods. Béatrice MADIOT (Université de Picardie, Jules Verne) discussed the pertinence of qualitative secondary analysis for social psychology, a field that mostly values experimental and quantitative research. She quoted existing studies of re-analysis in France (e.g. JODELET, 1982). She underlined the need for social psychology to study the effects of social thought on process in order to vary its angles of analysis. Mathieu BRUGIDOU, Magda DARGENTAS (PACTE/R&D-EDF) and Céline BELOT (PACTE) provided an overview of topics and introduced a process of reflection on the appropriation by Anglo-Saxon researchers of this method, as well as on prospects and specificities of secondary analysis in French academic research. A discussion among scholars as well as with the audience followed. Non-French researchers were made aware of the absence of a French qualitative resource center, and reflected on the reasons for this absence. Two research reports examining the feasibility of such a center were mentioned (DUBAR & REINERT, 2001; CRIBIER & FELLER, 2003). The establishment of a practice guide was also evoked. [26]

4. Discussion of Topics Raised at the Symposium: A Field Undergoing Evolution

In order to get a clearer idea about content after classifying topics raised during the Symposium, it seems that secondary analysis was viewed from the following perspectives: its emergence; theoretical gains, methodological issues and epistemological stand, deontological concerns, infrastructure and reception by researchers. [27]

4.1 Re-analysis vs. secondary analysis

The first point to notice is that re-analysis has been practized for a long time by certain researchers, including those in France. However, it seems now necessary to also reflect on the possibilities and problems of secondary analysis, as well as to extend its use, making possible data-sharing through archiving institutions. Recent developments of this method are associated with: technological progress providing great amounts of data and favoring innovative research practices (such as cyber-research); increased applied or collective research schemes urging data-sharing; a thought trend which values data and stresses the importance of cumulative knowledge. [28]

4.2 Theoretical gains

Another topic that emerged at the Symposium relates to the theoretical gains of secondary analysis; many speakers gave emphasis to the benefits of this method regarding the study of social change, longitudinal research, the analysis of social thought on process, the return of historicity and context in dealing with social phenomena and the subsequent gain for some disciplines. [29]

4.3 Methodological and epistemological issues

A third trend appearing in numerous papers was related to methodology and epistemology. This was the issue that arose most frequently during the Symposium and it generated a wide array of ideas. [30]

Secondary analysis emerged as a way to enhance quality in qualitative research regarding issues such as: confronting biases linked to the relation between researcher and respondent, and to the analysis of data; circulation of methods (for example, getting acquainted with various types of software, such as ones used in France—automating the analysis process—inspired from socio-linguistics, or the Anglo-Saxon (CAQDAS); standardizing ways to collect data, to report on the research process, to transcribe data and to report the context of research. These issues address the ideal of transparency in qualitative research. Also, there is emphasis on the rejection of a dichotomy between primary and secondary analysis and on common processes of analysis in any qualitative research. Researchers are not the sole holders of collected data; rather, they are supposed to make an account of their data to the community of researchers in such a way that every researcher would be able to analyze the same data, perhaps uncovering biases which may have escaped the attention of the primary researcher. [31]

I suspect these ideas reflect a specific epistemological stand common with quantitative research. In these papers, this ideal of transparency was frequently associated at the benefits for research, as the way to deal with social phenomena on a methodological and epistemological level. This, in my opinion, diverges from the first group of ideas because, rather than enhancing quality in qualitative methods, it stresses the co-construction of objects, the complementarity of analyses. This stand assumes that the appreciation of the richness of data is never gained through first analysis. In order to exploit all issues included in a data-set, further analyses by the same or various researchers are required. This methodological, theoretical and epistemological attitude has an effect on research practices, as it embraces collaborative research, sharing of data, and interdisciplinarity. It also requires awareness by researchers of their choices in the research-process, or of analytical perspectives as for the types of software used. Researchers are supposed to question their ways of treating and constructing an object, thus adopting reflexivity. This range of issues emphasizes the ideal of a "culture of disputation" or a "challenge to interpretation". It should be noted that this school of thought leads to insisting on researchers' practices and on reflexivity, which passes beyond qualitative research and joins quantitative or experimental currents (see for a review, BALEZ, submitted). [32]

A final topic linked to methodology is related to the re-opening of an old debate in social and human sciences, that of the opposition between qualitative and quantitative methods. The increased use of qualitative research and the need to standardize it, questions the traditional frontiers between methods. Many researchers at the Symposium defended their complementarity. Finally, the use of both during the analysis of qualitative material was mentioned as practice, which is significant regarding the evolution of ways to deal with methodological traditions and dichotomies. [33]

To close this methodological range of issues, secondary analysis appears dependent on general qualitative research, especially in France, where qualitative methods have not followed the same process as seen in Anglo-Saxon academic circles. Thus, a critical appraisal of qualitative practices should contribute to necessary conditions for secondary analysis to be developed. [34]

4.4 Deontological and legal issues

Another group of topics discussed are connected to deontological and legal issues. They were mainly related to the relationship between researchers and respondents and the fear of betraying confidences. Two platforms were presented: the conservative one—giving emphasis to the protection of respondents, and the innovative one—attending to the need to redefine this relationship. [35]

The academic community perceives secondary analysis negatively regarding the following issues: confidentiality, fidelity towards respondents, and fear of harming them by betraying their discourse or by the risk of identifying them. Other problems concern the property of data and the fear of criticisms by other researchers. Most of the scholars at the Symposium tried to explain the validity of those issues. They focused on the personal and emotional character of the relationship between researchers and respondents, leading researchers to a false perception of respondents as vulnerable subjects, as well as to less distance between them and their data. The paper by Arja KUULA was particularly interesting in this framework in the following aspects: it showed that the perception of respondents as vulnerable is false; it emphasized the importance of revising this relationship with respondents and with data, and of redefining it only through the research objectives; and it showed that anonymization could not be a problem for analysis but rather for publication. Also, most scholars insisted on the importance of thoroughly documenting the context of research, of obtaining consents, even for past research, as well as of anonymizing data. Finally, a guide of practices, or of standards, especially for French research, should accommodate the development of secondary analysis in the future and should help to deal with the above issues. [36]

4.5 Infrastructure, archiving, researchers

The final topic discussed at the Symposium concerns archiving infra-structure and the reception of secondary analysis by researchers. The establishment of archiving institutions was evaluated to be a condition for the development of secondary analysis. However, one should keep in mind that the establishment of such institutions generally suffers from practical difficulties, and requires long development in order to be able to be a real actor in research and to be able to complete its different tasks (e.g., relaying researchers and archiving institutions, providing methodological assistance, teaching, yielding data, developing software, discussing methods and practices). Similarly, the reception by researchers of secondary analysis and of archiving is submitted to a long process of evolution. Those methods question traditional practices of researchers, and a significant amount of time is required for them to reflect on those methods' costs and benefits in research. It seems that the "institutionalization" of secondary analysis will be acquired only if infra-structure related to archiving is established. Also, development of archiving depends on demands of secondary analysis. This shows that the development of secondary analysis and of archiving is interdependent. Finally, an interesting topic that was noted concerned the formation of a theoretical identity for research-teams, which are archiving their data, being able to re-analyze them and to emerge as experts of particular topics. [37]

5. Concluding Remarks and Future Directions

5.1 Discussing the Symposium

A wide range of issues were discussed in this Symposium and in this sense it was successful, even though it had ambitious aims, as its title reveals. All participants' contributions were interesting and fundamental as to the issues they were addressing. Overall, this event gave the rare opportunity to researchers of different academic, cultural and working backgrounds and with varying levels of experiences in qualitative research and in secondary analysis to confront their views and practices. Thus, this Symposium was constructive in several ways so that researchers could position their methods in a wider academic context. French researchers got acquainted with the practices and problematics surrounding qualitative methods in other European countries and the state-of-the-art in secondary analysis. Anglo-Saxon and European researchers had the opportunity to know more about the state of qualitative research in various fields in France. Finally, the Symposium included state-of-the-art of qualitative research and secondary analysis or archiving, as well as of central issues and debates related to these methods. The Symposium succeeded in promoting qualitative research and secondary analysis in France, in gathering together many scholars from various contexts to share knowledge, to foster networks on those methods, and to illustrate the many problems surrounding secondary analysis. [38]

The main idea that emerged though this Symposium was that secondary analysis represents a new trend in research in social and human sciences. A trend that addresses the importance of cumulative knowledge and that requires a review of qualitative methods and practices in various fields. In addition, it urges an epistemological turn, forcing researchers to re-define their relation to research, respondents, and data. Thus, archiving arose as a condition to the development of secondary analysis. [39]

In my view, in spite of the richness of this event, some areas remained underdeveloped. These were related to the lack of explaining in detail the process of analysis in secondary studies. Of course, the aim of the Symposium was not to teach ways of doing secondary analysis, event though, I think this element was lacking for researchers who would have liked to discover, on a concrete level, how secondary analysis is undertaken. In spite of this objection, this Symposium was a very successful event allowing researchers to get acquainted with a breadth of issues. Another underdeveloped area was connected to the debate regarding the opposition between quantitative and qualitative methods. Many scholars addressed this issue and held a position defending the complementarity of those methods. I think this issue was expressed as a stand held by scholars, but without further analysis of this debate and of its importance in connection with the contents of the Symposium. The status of this debate in the French academic community is interesting, as scholars "have difficulty situating themselves on this or that side of the qualitative/quantitative divide" (ANGERMÜLLER, 2005, para.4). In spite of this limitation, it is very important to mention that this issue emerged spontaneously throughout the Symposium and was not inspired by the organizers' agenda. This proves the significant character of this debate. [40]

5.2 Future issues to explore

Other issues emerged in this Symposium, which I think would be useful topics of study. One area concerns qualitative research in France. This approach seems to have evolved differently than in other cultural academic contexts (not only the chronological evolution in the use of qualitative methods, but the issues and the practices surrounding this approach as well). A "history" of this kind of research would be useful in order to complete the international landscape on qualitative methods, to widen Anglo-Saxon perspectives, and to anchor in this context secondary analysis or archiving. This history would be also important in order to establish a review of practices in the different academic fields in France, which seem dominated by different principles or traditions. Such work should contribute to institutionalization12) of qualitative research in France, to a "unity" between qualitative researchers and to a "culture of disputation". Another issue that I suspect of being connected to the "easier" appropriation by Anglo-Saxon researchers of secondary analysis, is that of a cultural practices or implicit rules that may encourage sharing data and knowledge. In order to give an example of these kinds of practices, Anglo-Saxon scholars arrived at this Symposium with papers to distribute to the attendees, while French scholars generally do not have this habit. This topic seems interesting to explore further. A final issue appearing in this Symposium is related to the methodological practices, such as software of analysis, different in the various academic contexts. I think common knowledge about the variety of practices should be further undertaken, in order to complete the analytical perspectives of tools assisting qualitative research. [41]

5.3 The academic context in France—Future events

Overall, this Symposium helped to introduce this novel method and its concomitant problems in the French research context. The research team Capas will be continuing its activities and enlarging the network of researchers. A collective book based on the Symposium papers is in progress and will be published. A guide of practices throughout different academic fields will also be elaborated. Further information on the Symposium is available on-line. [42]

To conclude, it seems that this scientific event was held in a general context that seems to encourage the emergence of such a practice in research in France. I can here refer to two reports, mentioned above, assessing the need for social and human sciences of a national qualitative database. Let me also mention that many workshops and symposia—organized by various fields—were recently held on neighboring topics13). Finally, the first French-speaking Conference on Qualitative Research will be held in June 2006, and one can expect in France the same process of institutionalization in qualitative research as in the Anglo-Saxon world14). Let us hope that such scientific activities will continue and that a network of researchers will be funded around qualitative methods. Studying qualitative research and relevant topics, such as archiving and secondary analysis, is expected to contribute to a scientific culture of sharing and collaboration. [43]

Notes

References

Angermüller, Johannes (2005). "Qualitative" methods of social research in France: Reconstructing the actor, deconstructing the subject. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(3), Art. 19. Available at: http://www.qualitative-research.net/fqs-texte/3-05/05-3-19-e.htm [Date of access: May 2, 2006].

Backhouse, Gill (2004). Acquiring qualitative data for secondary analysis. In Manfred M. Bergman & Thomas S. Eberle (Eds.), Qualitative inquiry: Research, archiving, and reuse (pp.89-103). Bern: Swiss Academy of Humanities and Social Sciences. [A revised version is available at: http://www.qualitative-research.net/fqs-texte/2-05/05-2-36-e.htm (Date of access: May 24)]

Balez, Ralph (submitted). De la prophétie auto-réalisante à la confirmation des hypothèses du chercheur en psychologie: Aperçu théorique et controverses épistémologiques en psychologie sociale. L'Année Psychologique.

Bergman, Manfred M. & Eberle, Thomas S. (Eds.) (2004). Qualitative inquiry: Research, archiving, and reuse. Bern: Swiss Academy of Humanities and Social Sciences. [Re-published in Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(2), http://www.qualitative-research.net/fqs/fqs-e/inhalt2-05-e.htm (Date of access: May 2, 2006)]

Beynier, Dominique (2004). Allaitement et normes de groupe, analyse secondaire d'entretiens auprès de femmes ayant eu le choix d'allaiter. Actes de la 2ème journée nationale sur l'allaitement, Réseau normand de l'allaitement, 19 march 2004, Caen.

Beynier, Dominique; Tudoux, Benoît & Momic, Milan (2005). Les métiers du travail social. Drees. Etudes et Résultats, 441, 1-10.

Blanchet, Alain & Gotman, Anne (1992). L'enquête et ses méthodes: l'entretien. Paris: Nathan.

Blanchet, Alain; Ghiglione, Rodolphe; Massonat, Jean & Trognon, Alain (1987). Les techniques d'enquête en sciences sociales. Paris: Dunod.

Corti, Louise (2000). Progress and problems of preserving and providing access to qualitative data for social research—The international picture of an emerging culture. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), Art. 2. Available at: http://www.qualitative-research.net/fqs-texte/3-00/3-00corti-e.htm [Date of access: May 2, 2006].

Corti, Louise & Thompson, Paul (2004). Secondary analysis of archived data. In Clive Seale, Giampetro Gobo, Jaber F. Gubrium & David Silverman (Eds.), Qualitative research practice (pp.327-343). London: Sage Publications.

Corti, Louise; Witzel, Andreas & Bishop, Libby (Eds.) (2005). Secondary analysis of qualitative data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1). Available at: http://www.qualitative-research.net/fqs/fqs-e/inhalt1-05-e.htm [Date of access: May 2, 2006].

Corti, Louise; Kluge, Susan; Mruck, Katja & Opitz, Diane (Eds) (2000). Text, Archive, Re-analysis. Forum Qualitative Sozialforschung/ Forum: Qualitative Social Research, 1(3). Available at: http://www.qualitative-research.net/fqs/fqs-e/inhalt3-00-e.htm [Date of access: May, 2, 2006].

Cribier, Françoise & Feller, Elise (2003). Projet de conservation des données qualitatives des sciences sociales recueillies en France auprès de la "société civile". CNRS-EHESS, LASMAS, UMR 8097. Available at: http://www.iresco.fr/labos/lasmas/rapport/Rapdonneesqualita.pdf [Date of access: May 2, 2006].

Dargentas, Magda & Le Roux, Dominique (2005). Potentials and limits of secondary analysis in a specific applied context: The case of EDF—Verbatim. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), 40. Available at: http://www.qualitative-research.net/fqs-texte/1-05/05-1-40-e.htm [Date of access: May 2, 2006].

Demazière, Didier & Dubar, Claude (1997). Analyser les entretiens biographiques. L'exemple de récits d'insertion. Paris: Nathan.

Dubar, Claude & Reinert, Max (2001). Faisabilité d'une banque de données d'entretiens biographiques en sciences sociales : Réflexions à partir du rapport remis au CNRS-SHS sur l'archivage et la capitalisation des entretiens en sciences sociales. Paper at Journée d'Etudes du CIDSP sur Les Nouvelles Méthodes d'Analyse des Entretiens, Grenoble, 9 March, MSH-Alpes. Available at: http://www.msh-alpes.prd.fr/Actualites/Analyseentretiens.htm [Date of access: May 2, 2006].

Duchesne, Sophie & Haegel, Florence (2004). L'enquête et ses méthodes: l'entretien collectif. Paris: Nathan.

Fielding, Nigel (2000). The shared fate of two innovations in qualitative methodology: The relationship of qualitative software and secondary analysis of archived qualitative data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), Art. 22. Available at: http://www.qualitative-research.net/fqs-texte/3-00/3-00fielding-e.htm [Date of access: May 2, 2006].

Fielding, Nigel (2004). The resurgence, legitimation and institutionalization of qualitative methods. In Manfred M. Bergman & Thomas S. Eberle (Eds.), Qualitative inquiry: Research, archiving, and reuse (pp.89-103). Bern: Swiss Academy of Humanities and Social Sciences. [Also available at: http://www.qualitative-research.net/fqs-texte/2-05/05-2-32-e.htm (Date of access: May 24)]

Heaton, Janet (1998). Secondary analysis of qualitative data. Social Research Update, 22. Available at: http://ww.soc.surrey.ac.uk/sru/SRU22.html [Date of access: May 2, 2006].

Heaton, Janet (2004). Reworking qualitative data. London: Sage Publications.

Hinds, Pamela S.; Vogel, Ralph J. & Clarke-Steffen, Laura (1997). The possibilities and pitfalls of doing a secondary analysis of a qualitative data set. Qualitative Health Research, 7(3), 408-424.

Jodelet, Denise (2003). Aperçus sur les Méthodologies Qualitatives. In Serge Moscovici & Fabrice Buschini (Eds.), Les méthodes des sciences humaines (pp.139-162). Paris: PUF.

Jodelet, Denise & Ohana, Jocelyne (1982). Systèmes de représentation du corps et groupes sociaux. Paris: Laboratoire de Psychologie Sociale—EHESS.

Kaufmann, Jean-Claude (1996). L'entretien compréhensif. Paris: Nathan.

Laborier, Pascale & Bongrand, Philippe (2004). L'entretien dans l'analyse des politiques publiques, un impensé méthodologique? Revue Française de Science Politique, 55(1), 73-111.

Le-Roux, Dominique & Vidal, Jean (2000a). Verbatim: Qualitative data archiving and secondary analysis in a French company. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), Art. 15. Available at: http://www.qualitative-research.net/fqs-texte/3-00/3-00rouxvidal-e.htm [Date of access: May 2, 2006].

Le-Roux, Dominique & Vidal, Jean (2000b). Verbatim: Une expérience de capitalisation d'entretiens qualitatifs. Bulletin de Méthodologie Sociologique, 65, 58-67.

Mauthner, Natacha S.; Parry, Odette & Backett-Milburn, Kathryn (1998). The data are out there, or are they? Implications for archiving and revisiting qualitative data. Sociology, 32(4), 733-745.

Medjedović, Irena & Witzel, Andreas (2005). Secondary analysis of interviews: Using codes and theoretical concepts from the primary study. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), Art. 46. Available at: http://www.qualitative-research.net/fqs-texte/1-05/05-1-46-e.htm [Date of access: May 2, 2006].

Mucchielli, Alex (Ed.) (2005). Dictionnaire des méthodes qualitatives en sciences humaines et sociales. Paris: Armand Colin.

Opitz, Diane & Mauer, Reiner (2005). Experiences with secondary use of qualitative data—First results of a survey carried out in the context of a feasibility study concerning archiving and secondary use of interview data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), Art. 43. Available at: http://www.qualitative-research.net/fqs-texte/1-05/05-1-43-e.htm [Date of access: May 2, 2006].

Parry, Odette & Mauthner, Natacha S. (2004). Whose data are they anyway? Practical, legal and ethical issues in archiving qualitative research data. Sociology, 38(1), 139-152.

Santacroce, Sheila J.; Deatrick, Janet A. & Ledlie, Susan W. (2000). Secondary analysis of qualitative data: A means of collaboration in HIV-related research. Journal of the Association of Nurses in AIDS Care, 11(3), 99-104.

Santiago-Delefosse, Marie & Rouan, Georges (Eds.) (2001). Méthodes qualitatives en psychologie. Paris: Dunod.

Seale, Clive; Gobo, Giampetro; Gubrium, Jaber F. & Silverman, David (Eds.) (2004). Qualitative research practice. London: Sage Publications.

Stewart, David W. & Kamins, Michael, A. (1993). Secondary research: Information sources and methods. London: Sage Publications.

Thorne, Sally (1998). Pearls, pith, and provocation: Ethical and representational issues in qualitative secondary analysis. Qualitative Health Research, 8(4), 547-555.

Author

Magdalini DARGENTAS, Ph.D. in Social Psychology. Her research topics include social representations, psychology of religion, issues related to death and funeral practices, as well as qualitative research methods. She is an affiliated member of the Laboratory of Social Psychology at EHESS (Ecole des Hautes Etudes en Sciences Sociales, CETSAH). She is currently working on a research project on secondary analysis of qualitative data for R&D-EDF (research-team: GRETS, social science research group) and CNRS (research team: PACTE, Politique-Action publique-Territoires). She is also giving courses on qualitative methods in the Psychology Department at Université de Paris X, Nanterre.

Contact:

Magdalini Dargentas

GRETS (R&D-EDF)
1, avenue du Général de Gaulle
92141 CLAMART Cedex
France

E-mail: dargenta@ehess.fr

Citation

Dargentas, Magdalini (2006). Secondary Analysis and Culture of Disputation in European Qualitative Research. Conference Essay: Secondary Analysis in Qualitative Research—Utopia and Perspectives [43 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 7(4), Art. 9, http://nbn-resolving.de/urn:nbn:de:0114-fqs060495.

0 thoughts on “Presenting Qualitative And Quantitative Data Essay

Leave a Reply

Your email address will not be published. Required fields are marked *