The Future Place Blog

The book continues to make progress

Despite spending some wonderful time with my family over Christmas, I have been able to make a bit more progress on the book, Handbook of Online and Social Media Research.

Over the last few weeks a number of people have been kind enough to help in lots of different ways. Some by contributing via this blog, some directly sending me thoughts and case studies, some Tweeting advice or just encouragement.

One key group have been kind enough to join a LinkedIn group and have reviewed pages and in some cases whole chapters of the book. There are so many people helping that all I will be able to do is to thank them, and when I next see them buy them a drink, but their help has been invaluable. Because the publisher for the book is Wiley, all the copyrights and such for the book and the contributions end up being owned by them, but there will be space in the book to list and thank the many helpers.

If you would like to join the group, sign-up in LinkedIn and you will be greatly welcomed.

Will MROCs replace BBGs?

To a considerable extent the difference between BBGs (Bulletin Board Groups) and online research communities (aka MROCs) is simply a generational difference. When BBGs were developed they reflected current practice in terms of how the Internet was being used to create discussions, i.e. bulletin boards. Although some bulletin boards and forums still exist, they are increasingly being replaced by online communities, a phenomenon of the world of social media.

It is likely that BBGs, as a research tool, will be replaced with online research communities (also known as MROCs) over the next few years. The BBGs have the advantage of being quicker to create, of currently having more research functions, and of being more suitable for IDIs (parallel In-Depth Interviews). By contrast online research communities are more intuitive for the participants and more attractive to clients.

---

The question and text above have arisen out of the work I am doing for my book. I would be really glad to hear other people's thoughts.

Using Participant Blogs in Market Research

As part of the book I am looking at how market researchers have used blogs as research tools, specifically where they have recruited respondents and enabled them to create a blog as part of a research topic.

Adding together my experience, colleagues’ experiences, and conference papers my views are that blog projects tend to have the following in common:

• Each participant is given their own blog to maintain.
• Usually qualitative. The participants may be asked to fill in a survey as part of the exercise, but this tends to be for background and segmentation purposes.
• The participants recruited in the same sorts of ways that are used for other qualitative projects and for online research communities.
• If incentives are used they tend to be quite high (compared with quantitative and online research communities).
• Tend to last more than three days, but less than three months.
• The number of participants tends to be modest, compared with quantitative studies and online research communities. Typically ranging from about ten to about 60.

What is your experience? Do you often run them longer/shorter? With fewer or more participants?
In terms of naming the people doing the blogging, what other names are you using? (I have heard collaborators, and academics sometimes say subjects).

What are the key issues about Web survey systems?

As part of the book I am writing for Wiley, a handbook of online and social media research I have been working this week on the systems that companies use to conduct online surveys, i.e. Web surveys.

I would really appreciate hearing people’s thoughts on the structure I am thinking about using.

The first level of the chapter looks at the categories that anybody choosing a an approach towards a web survey system should consider. The headings at this level are:

1. Whether to have a system  - or just buy completed surveys from third party suppliers
2. Buying or developing a system
3. Software or SaaS (Software as a Service)
4. Cost
5. System features
6. Capacity

The ‘System features’ category clearly needs subdividing and my thoughts at the moment are the following subdivisions:

1. Questionnaire creation tools
2. Questionnaire testing facilities
3. Project management and monitoring
4. Mulitmedia options
5. Advanced graphical interface
6. Panel management
7. Project reporting
8. Analysis options
9. International options
10. Mobile research
11. Hybrid survey options
12. Security options
13. Advanced marketing science modules
14. Programmable script or language

I would appreciate any suggestions or thoughts. Do you think I have missed an important category? Have I missed an important subdivision of Systems features?

Have you used a marketing database for market research sample?

As part of the research for my new book I am looking to explore whether people are still using this approach and what their experiences are.

In the early days of the online surveys, particularly in the late 1990s I used this route quite a bit as online access panels did not exist. Sometimes it worked really well, sometimes it produced odd results. For example I can remember studies where we had to screen out a large number of people because they a significant number came from the 'compers' community, people who spend a large amount of time entering competitions.

I am also aware, from talking to the database people, than many (perhaps most) of the online access panels have used these databases as one of their sources in the past.

More recently I have seen marketing databases used to populate community panels, particularly where the client's business model means that it does not have a good customer database of its own (i.e. many FMCG (also known as CPG in some markets) products where the direct customer relationship is typically with the retailer not the brand).

So, what have your experiences been?

I am interested in both information and quotes I could use in the book.

Email Surveys

An email survey is a term that is normally applied to a survey that is emailed to respondents. The term does not normally refer to studies where the survey is hosted on the Web and an invitation is emailed to the respondent. However, email surveys have become much less common than they used to be and the term email survey may become ambiguous.

In the very early days of online research, email surveys were popular because they did not require people to be online whilst they completed the survey. The Internet at this stage being very slow and most users paid for access by the minute. Also, in the early days of the Internet there were people who email access but who did not have access to the web, particular from PCs they accessed whilst at work.

There are two ways to conduct an email survey. The first is to include the survey in the body of the email and the second is to send the send the survey as an attachment to the email.

Email Surveys in the body of the email

Email surveys in the body of the email further divide into two main varieties, text emails and HTML emails.

A text email sets out the questions as part of the email. The respondent clicks ‘Reply’ and then scrolls down the email, entering their responses. The main drawbacks with this approach were that the survey tended to be boring, restricted, and respondents could miss questions, answer inappropriately (for example picking two options in single question), or even delete part of the questionnaire. Software exists to draft these surveys, to conduct the mailing, and to interpret the replies.

An HTML email survey uses the power of HTML to make a more interesting survey. An HTML survey can look appealing, it can use standard features such as radio buttons and check boxes, and can include a SUBMIT button at the foot of the email. The main problem with HTML surveys is that many people’s email filters will either prevent them getting through, or they will convert them to plain text.

One area where email surveys have a specific benefit is within a large organisation. In a large organisation a web based survey can cause many recipients to log onto the same resource at the same time. An HTML study emailed within in a company will, in many cases, distribute the load better.

One special case of an email survey is where an email service is utilised to poll people’s views, for example the voting buttons that can be used in Microsoft’s Outlook to gather people’s views.

Email Surveys as an attachment

In the early days of the Internet it was quite acceptable to email a survey as a piece of executable code. These emails arrived on the respondent’s machine, the attachment was opened, the survey completed, and it then emailed itself back to the project sponsor. The general reluctance to accept executable code via email has resulted in this form of survey becoming very rare.

---

This post is part of the book I am currently writing for Wiley on online and social media research.

I would welcome any comments, corrections, and suggestions, but please note that any posts will be assumed to belong to the project and the use of them will not transfer any rights to the person making the post. Sorry to be so boring, but that is the reality of sharing stuff that is destined for a book.

Comparison of data collection modalities

This section looks at the key modalities used to collect quantitative data and highlights their key differences and implications. In 2008, according to the ESOMAR Global Market Research 2009 report, it was estimated that quantitative survey research accounted for about 55% of all market research. Of that 55%, online accounts for about 39% (i.e. 20% of all research spend), with telephone just behind at 33%, Face-to-Face at 22%, and Postal at 9% (i.e. 5% of all research spend).

Interviewer

The main issue that divides the modalities is the presence, or absence, of an interviewer. This difference is crucial in many ways. Respondents tend to prefer surveys with an interviewer, especially when the interviewer if physically present. If there are problems with the survey the interviewer can explain, help, probe, and ensure that messages about problems are relayed back to the project team. With self-completion modalities nobody sees the respondent. Are they who they say they are? Are they concentrating appropriately? Are they confused? However, on the other hand, there is considerable evidence that people are more honest when there is not an interviewer present, and of course the cost is less without an interviewer.

Online and postal surveys are both self-completion, i.e. they do not have an interviewer. Telephone and face-to-face are both interviewer administered. One exception is where the telephone interviews are conducted by automated processes such as IVR (interactive voice response).

Sampling

The key, historic, issue about sampling was whether it approximated to random probability sampling. Random probability sampling is based on an assumption that the population is known and that every member of the population has a known, non-zero, chance of being selected. If a sample was a random probability sample (or could be made to behave like one through the use of techniques such as weighting) then, subject to sampling error, results from the sample could be projected to the population.

If a sample is not a random probability sampling then the results cannot be projected to the population directly. With non-probability sampling the researcher has to adopt one of the following strategies:

a)      Use an alternative technique to link the sample to the population, for example propensity weighting (for more information see http://www.schonlau.net/publication/03socialsciencecomputerreview_propensity_galley.pdf) or by establishing a modelled link (for example experience may indicate that a score of X in the sample equates to a Y share of purchase in the real market).

b)      By assuming that the movements in the sample, from one wave to another, or from one test to another, reflect changes in the population, even if the absolute values may not reflect the population. Much ad tracking is based on this assumption.

c)       By assuming that the sample is a reasonable proxy for the population. Quite often the face validity of this approach is improved by using quotas to make some key characteristics of the sample match the population, for example age, gender , and income.

Although traditional market research has, since the 1930s, been built on an assumption of random probability sampling, the reality has, in most cases, been very different. If there is a significant proportion of people who can’t be reached by the medium (for example people who do not have a landline, or who tend to be out a lot), or if there are a significant proportion who decline to do a survey, then non-response bias can render random probability sampling, and its implications for representativeness and projectability, worthless. Views differ, but it is safe to assume that if the response rate is less than 70%, then the study is not approximating to a random probability sample.

Most online research is conducted via online access panels, which is clearly not a method of random probability sampling. Online research conducted with client databases (or via visitors to a website) might approximate to random probability sampling if a) the database or the website visitors are the relevant population, b) if the response rate was high enough.

Face-to-face data collection tends to come in two main varieties, door-to-door and central location. In some countries both methods are prevalent, in other countries only one of these will tend to apply. For example, in North America and Australia face-to-face is very rarely conducted door-to-door. Door-to-door studies can be constructed so as to approximate to a random probability sample, if the response rate is high enough. Central location testing is not capable of approximating to a random probability sample, and usually relies on the sample being a reasonable proxy, e.g. by quota controlling for variables that are felt to be key.

For a while, and in some markets (such as Australia and USA), telephone research was able to claim a semblance of random probability sampling. In markets where more than 90% of households had a telephone, and due to the development of RDD (Random Digit Dialling), only response rates stood between the researcher and a credible claim of random probability sample. However, due to the growth of mobile (cell phone) only households, telephone has lost its ability to assert that it is fully representative.

Postal research has declined in importance over recent years and the ESOMAR Global Research figures for 2008 show it as accounting for only a quarter as much spend as online. In most markets it is possible to construct a sample frame that would be representative of the general population. The key limitation in terms of achieving a representative sample is non-response, which in most market research surveys result in fewer than 70% of surveys being returned (usually a lot less than 70%).

Stimuli

The four data collection modalities being reviewed here have very different implications for the kind of stimulus material that can be used.

In the early days of online research most surveys were restricted to simple text, some colour, and perhaps a low resolution graphical image or two. The limiting factors were the software being used and even more critically the speed of most people’s connection to the internet. With the advent of widely available broadband (in many markets), good software, new options (such as Flash), the range of stimulus material that can be used includes pictures, audio, moves, and interactive modules. In terms of what cannot be used, the most notable are smell, taste, touch, and detailed pictures (for example, an A4 photo can clearly show twenty cars on a show card, most screens struggle to show more than ten, because of size and definition).

Two limiting factors in terms of stimuli for online surveys are a) people who complete the survey whilst at work where either sound or movies are disabled or are unsuitable, and b) the rise in the number of people who are completing their surveys via smartphones (such as iPhones) and who therefore have smaller screens.

Face-to-face data collection, particularly central location testing, presents the widest range of stimuli that can be used. However, it should be noted that some face-to-face formats are more limited, for example when conducting in-street interviewing using a clipboard, paper, and pencil. In terms of face-to-face interviewing, the introduction of CAPI (Computer Aided Personal Interviewing) was, in particular, responsible for extending the range of stimuli that can be used in face-to-face interviewing, for example by making it easier to show media files.

Telephone surveys have the most limited range of stimuli that can be readily be used. In a typical survey the interviewer reads the questions to the respondent, who then says their answer. This means that the questions and prompts cannot be too complex, which is why conjoint analysis is normally considered unsuitable for telephone research. Audio files (for example a radio advert) are suitable for telephone surveys.

Most postal surveys are very limited in the stimuli they use, they tend to be a printed questionnaire, possibly with an image or too. However, some researchers have used postal surveys with much more complicated stimuli, including scratch and sniff cards, DVDs, and products to use and test.

Adaptive

Survey modalities vary in the degree to which the survey can ‘adapt’ to the respondent. The most adaptive interviews are those computer administered surveys which are scripted to adapt the lists, stimuli, and questions based on earlier responses. Computer administered surveys can be online, telephone (in which case it is CATI, computer aided telephone interviewing), and face-to-face (in which case it is CAPI, computer aided personal interviewing).

Online surveys can be, and often are, adaptive. Unless face-to-face surveys are computer administered the degree of adaptively is limited to procedures such as skip patterns and rotations.

When telephone survey are computer administered they can be adaptive. If a telephone survey is not computer administered then the adaptivity will be limited to simple options such as skip patterns and rotations.

Postal surveys represent the least adaptive modality. Surveys may contain instructions such as “If yes, skip to question 12 on page 3”, but respondents are often poor at complying with these instructions

Summary of differences

The table below summarises the key differences between the leading data collection modalities for quantitative research.

	Online	Face-to-face (paper & pencil)	Telephone	Postal
Interviewer	Self-completion.	Interviewer mediated.	Interviewer mediated.	Self-completion.
Sampling	Tends to be via access panel, hard to claim random probability sampling, subject to response rate concerns.	Door-to-door may be able to claim random probability sampling. Central location cannot claim random probability sampling	RDD may be able to claim random probability sampling, subject to response rate concerns.	Can claim to be using random probability sampling, subject to response rate concerns.
Stimuli	Video, audio, medium resolution images, interactive materials.	Can include video/taste/smell/high definition images.	Just that which can be conveyed by the spoken word, plus audio.	Tends to be just text and show cards, however, can include products, scratch’n’sniff etc.
Adaptive	Scripts can be adaptive.	Only computer aided personal interviewing is adaptive.	If computer aided then scripts can be adaptive.	Not adaptive.

---

This post is part of a my project to write a book on online and social media research for Wiley. The copyright for the material above is retained as it will be needed for the book. You are welcome to use the material above provided that you credit the source.

I would welcome any comments, corrections, and suggestions, but please note that any posts will be assumed to belong to the project and the use of them will not transfer any rights to the person making the post. Sorry to be so boring, but that is the reality of sharing stuff that is destined for a book.

---

The Scale of Online Research

Data showing the scale of online research, compared with other data collection mediums, tends to be somewhat out of date when it is published, so the data below should be seen as indicative of the rate of change and the scale of Internet in market research.

Table below shows data from the ESOMAR annual Global Market Research Survey.

	2005	2008
Online	13%	20%
Telephone	21%	18%
Face-to-face	24%	12%
Postal	6%	5%

 

Note the large elements of ‘Other’ includes qualitative and ‘automated digital/electronic’ (which includes retail audits and offline media measurements).

Within the global figures there are enormous differences, largely (but not entirely) dependent upon the degree of economic development. For example, in 2008, over 30% of all research was conducted via Online in Canada, Japan, and Australia. In China and Switzerland the figure was 11%, whereas in Bolivia, India, and Portugal the figure is close to 0%.

A Taxonomy of Research Communities

Because the field of online research communities is developing so rapidly it is important to create a structure to help understand the options that confront researchers proposing to use online research communities to gain insight.

The key issues that determine the nature of a community are:

1.       Short term versus continuous
A short term community typically from 3 days to three months, the alternative is to set the community as an ongoing project.

2.       Closed versus open
Most online research communities are created as private communities, often referred to as walled gardens or gated communities. The members of a closed community are recruited by invitation only, which reduces the risk of unfriendly eyes being part of the project. A closed community is also hidden from view, it is not normally possible for people to see it by simply typing in a URL.
An open community, by contrast, has members who will have joined from a variety of sources. They may have responded to advertising, they may be as a result of initiatives such as ‘member find member’, or they may have responded to links on a brand’s website. Even an open community can have restrictions, for example it may screen potential members on whether that are customers or users of some product or service, it may screen by age, by location, or by some other characteristic.

3.       Incentivised versus no incentives
One issue for a community is whether to offer incentives, and if incentives are to be offered, what sort of incentives should they be, for example gifts or cash, per person or prize draw.

4.       Agency or client moderated
Who will run the community, will it be managed by a third-party, such as an agency, or will it be run in-house by the brand.

5.       Highly directed versus community driven
Communities vary in the extent to which community members can determine the agenda.

6.       Pure Research or Hybrid
Is the community going to be purely for market research purposed, or might the community also be used for other purposes, such as Word of Mouth advocacy.

7.       How many members?
Online research communities can have as few as 30 members or as many as tens of thousands. The implications of the number of members is quite profound, as it affects the degree to which a community functions as a community.

The role of incentives in online research communities

One of the more heated debates, in the area of running online communities, is about the topic of incentives. The key questions are:

a)      Should a community use incentives?

b)      If incentives are used, what form should they take?

c)       If there are incentives, what should they be linked to?

The main argument against paying incentives is that it can change the relationship away from being a mutual interest endeavour towards paying somebody to do something they don’t want to do. If a community is delivering true engagement and value to its members, it should not be necessary to pay incentives, and the absence of incentives will make the team running the community try harder to develop the engagement.

There are also occasions where the rules or conventions of the organisation running the research inhibit the use of incentives; this is often true for organisations funded by public money or charitable contributions.

There are two counter arguments that people make in favour of incentives.

The first argument relates to fairness. If companies are making money from people’s views, surely it is only fair to reward the time and effort members are contributing. This point of view suggests that a community can’t be based on the client always taking and never giving.

The second argument is related to the practicalities of getting the necessary responses to the programme for the community. In some communities, particularly short term, highly tasked, intense exercises incentives may be the only way to persuade people take part, particularly at the required level of activity.

If a community is going to be incentivised, it needs to define what the incentives are and how they will be distributed.

Types of Incentives

A very common form of incentive in online research is cash (or points that can be converted into cash or products). However, this is the incentive that is the most corrosive to the concept of a community of interest, and the one that is most likely to create an employer/employee mentality.

Many communities offer rewards linked to the reason why people have joined the community. For example, the airline easyJet offer members of their community the chance to win flights. This approach is frowned on by some research bodies (such as the UK’s MRS) but seems to be closer to what members want.

Communities should also remember that incentives and rewards can go well beyond things with a cash value. Many communities seek to make members feel they are special, this might mean sending advance information about new products or services, even about upcoming adverts, or inviting members to events such as a tour of a factory or location.

Distributing Incentives

If a community is using incentives, it has to decide on how to distribute them. The options that are currently common include:

• An incentive to every member, regardless of what they contribute. This is simple and if the reward is just a thank you it might be fine, but it does not incentivise activity. If the amount is small, it may not be seen as much of a thank you, if it is large it will drive costs up.
• An incentive is paid to every member who makes some contribution during a relevant period of time, for example a month or three months (a contribution might be classified as a post, answering a poll, or perhaps uploading a photo). This method does incentivise activity, but it can be expensive. For short term communities these incentives can be quite high, for example US$50 per week for three weeks, for everybody making at least 3 contributions during the week.
• A prize draw is offered, either to everybody in the community, or to those who have contributed in the relevant period. This method allows for a larger incentive, whilst controlling costs, but not everybody responds to prize draws as an incentive.
• An award is paid to the ‘best’ contribution in the relevant period. Some communities which use this method allow the community to choose the best contribution; others prefer the moderator or the end client to select the best contribution.

Many communities which use incentives use a combination of the techniques outlined above.

One trap that some new communities fall into is that they offer very large incentives initially, in order to ‘get it going’. However, this approach usually leads to high expectations amongst community members, which can result in large incentives always being necessary, or to a drop in activity as members realise the big incentives were just to get them interested and they are now being asked to ‘work’ for much less.