A Survey on Consumer Ratings for the Market Available Drug Products Via the E-Commerce Platforms

Abstract

It is to illustrate post-market surveillance on drug products using consumer ratings posted on e- commerce portals. Primatene mist, an FDA-approved inhaler was chosen for the study because of its OTC status.

Amazon provides customers two options for product rating either to score a product by star rating (1~5 stars), or score and post experience as customer reviews. Given that patients utilize the number of stars to rank drug performance, it is to unlock the opinions of the reviewers and generalize them over the star ratings by ranking, for the post-market studies in drug performance.

A snapshot of 10,802 ratings was obtained and 525 of them wrote reviews forming the basis of the study. Statistical analyses were performed on safety and effectiveness, as well as device integrity and other aspects of the inhaler.

53.5% of reviewers explicitly cited effectiveness (281 of 525); 0.6% (3 of 525) reported concerns about safety (1 ER (Emergency Room) visit and 2 ADE (Adverse Drug Event)). When extrapolating their experience over the population of star ratings, Primatene brings benefits to 95% of the participants (n=10,802).

E-commerce portal provides input about the usability of a product from the feedback of consumers. For pharmaceuticals, such an input reveals viable information as a new source for post-market surveillance studies. Based on the customer reviews on Amazon portal, Primatene Mist is safe and effective in performance, serving patients well in the segment of intermittent asthma.

Keywords

Post-market surveillance; Effectiveness and safety; Over the Counter (OTC) medication; Customer reviews and ratings

Introduction

Purpose and goal

The purpose of this study is to reveal patient’s direct experiences in the performance of a pharmaceutical that was recently approved by FDA. The specific goal is to analyze the customer reviews posted on Amazon for Primatene Mist inhaler, which is the Over the Counter (OTC) rescue inhaler approved by FDA for the temporary relief of the mild symptoms of intermittent asthma [1]. The great accessibility of reviews on OCT medicines on e-commerce portals makes such a study possible. It is to reveal the experiences of using Primatene, from the aspects of both a patient and consumer on effectiveness and safety; Pros and Cons of its OTC settings; potential issues on device integrity and usability; those related to the new formulation such as uniformity; in label comprehension to self-mastering the inhaler under the OTC; the attributes that the patients valued the most and the least about the product. It is also to identify potential space for improvement in design, manufacture and patient education of the product. The study assesses the pharmaceutical under the current industry standard and appraises the outcome of a public health decision of the regulatory as well.

Background

The FDA first approved epinephrine (API of Primatene Mist) MDI in 1967 as an OTC medication, based on decades of preliminary experience in drug safety and efficacy. However, “some organizations oppose the restoration of the OTC status for the new HFA Primatene mist, auguring that it was less effective than prescription medications with only β2 agonists alone vs. the epinephrine with mixed α-β agonists, which increases the risk of cardiac toxicity and other adverse effects” [2]. In fact, “several pharmacodynamics safety measures indicated that resultant drug levels at doses nearly 13-fold higher than proposed (125 mcg versus 1,600 mcg in one trial) were not likely associated with significant safety issues, i.e., transient hyperglycemia, hypokalemia, increases in blood pressure or heart rate, or arrhythmias. There was no data identifying a cardiovascular safety concern when the product was used as intended, that is, according to labeling”, quoted by Center for Drug Evaluation and Research (CDER) recently [3]. OTC medications serve the purposes of savings and accessibility, while OTC rescue inhaler is an alternative when prescription is unavailable [4].

Materials and Methods

Product selection

Primatene was chosen for the study also because of its prolonged “legacy” status of OTC. Safety and efficacy remained a “myth” for several decades when no clinical data were available until the recent approval of the new formulation in November 2018. Since April 2020, Amazon has been posting customer reviews on Primatene as it does for other thousands and millions of listings, making data gathering possible for the study on the new formulation. It was chosen since Primatene is an FDA-approved medication, “with similar quality performance as prescription inhalation aerosols” [5]. Further, obtaining the recent FDA’s approval of the new Primatene makes it possible to juxtapose patients’ experiences on such an “old” brand, with the current standards and regulatory requirements of the pharmaceutical industry.

There are twenty-five million asthma patients in the USA and 35.2% of them (close to nine million) are diagnosed with intermittent asthma [6,7]. Post-market surveillance therefore bears significant stakes for the patients’ safety on using asthma medications. Besides, analyzing patients’ experiences in drug performance would benefit the patient’s awareness about pharmaceuticals, helping the decisions on whether it is the right medication for them based on the product experience on e-commerce.

Data gathering

Data source: Both customer reviews and star ratings are extracted from the Amazon e-commerce portal. “To calculate the overall star rating and percentage breakdown by star, Amazon doesn’t use a simple average. Instead, its system considers things like how recent a review is and if the reviewer bought the item on Amazon. It also analyzed reviews to verify trustworthiness” [8]. In this study, the snapshots of Primatene reviews on Amazon were extracted as of 17 May, 2022, in the format of MS-Excel spreadsheet by FeedCheck [9]. As an additional reference, the Snapshot of the Primatene reviews was also obtained from Walmart’s e-commerce portal, in an aggregated format [10].

Record-set: The record-set definition for customer reviews on Amazon portal is as below.

• Id–a unique sequential number that was assigned to each record of the reviews by the author.

• Review title–a brief description made by the reviewer for what the review is all about.

• Review content–the contents of the review.

• Review user–the identity that a reviewer used to write the customer review on Amazon.

• Rating–a rating in a scale of 1-5 by star that a reviewer uses to score the product.

• Review link–a unique identifier that Amazon assigns to each reviewer for traceability.

• Verified–for Amazon internal use.

• Review date–the date that a customer reviews is registered on Amazon portal.

The scope of this study is based on only the title portion, describing the contents by the reviewers.

Data integrity

To safeguard the integrity of the reviews, “99% of Amazon’s actions on incentivized reviews take place before problems are reported to the Company”, according to CNET (Computer Network). Further, unlike the other types of consumables possessing a commercial drive for incentivized reviews, Primatene is a rescue medication therefore patients’ reviews should be trustworthy. Aside from adding a unique sequential number to each individual record to preserve the referential integrity, the study neither excludes records nor does it edit any of the contents on the raw data. For the data from Walmart portal, the aggregated ratings for Primatene were manually copied from the Walmart portal into the Excel sheet for the study.

Inductive reasoning

Data screening: It is to use inductive reasoning to decipher the sentiments in star ratings, according to the specificities posted as product reviews on Primatene. Transactional reviews were broken down into five groups in accordance with the number of stars that the reviewers rated the product. Each segment of the reviews was then appraised, record-by-record, to identify keywords associated with certain aspects of the medication. The definitions of keywords vary based upon their usage and frequency of appearances in the context of patients’ testimonials.

A bottom-up approach: Amazon provides two options for product rating consumers can either score a product by star rating ranging from one to five stars, or score and then post personal experience on product as customer reviews. For Primatene, less than 5% of the participants submitted reviews simultaneously while rating the medication.

Data screening is a two-path process to identify and count the keywords. Keywords are those that patients use frequently to describe their experiences on certain aspects of the medication.

Using induction: Given that patients utilize a number of stars to rank their experiences in drug performance, it is reasonable to generalize the sentiments of the reviewers, which are the proper subsets of the participants’ population, over the entire population of the star ratings, by ranking.

Assessment on risk of bias

Human error: The process of bottom-up approach is cumbersome and time consuming that needs the complete involvement of human beings, however human activities could induce errors and possible omissions. The only way to mitigate the risk is to repeat the works for verification, to reduce the exposure of risk. The process of data-screening was repeated three times for this study.

Deviation: The percentage of reviews in the 1-star ranking at Amazon on Primatene was out of proportion compared to that of the other rankings, skewing the overall rating to the negative side. That is, compared to the proportions of customers who wrote reviews on the other rankings, there were more customers who ranked the product 1-star and wrote reviews.

Customer reviews from Walmart were therefore obtained as the control reference to calibrate the proportion of 1-star reviewers on Primatene.

Overgeneralization: Overgeneralization is one of the inherited risks of using inductive inference. In this study it propagates the sentiments in customer reviews to the population of star ratings, just relying on the 1-to-5 ranking in number of stars.

Results

Rating vs. customer review

A small fraction of the patients who gave product ratings also submitted reviews about their unique experience, sharing the characteristics of Primatene in performance. Given that a group of reviewers is a proper subset of the participants in star rating, it is to expect that the proportion of customer reviews remains a constant across over the full spectrum of ratings, unless there were extra patients who were willing to write reviews in a particular rank, making it out of the proportion. Over 95% of patients scored Primatene a 3-star or higher in star ratings and in each rating about 5% wrote customer reviews as well. However, about 24% of patients who scored the product a 1-star submitted customer reviews, which is almost five times higher than the proportions of reviewers in other ratings. The distribution of proportions in customer reviews in rating at Walmart on Primatene nevertheless resembles that of star ratings from Amazon among all five ratings, which supports the validity to include star ratings in inductive reasoning in this study, i.e., it provides evidence (n=776) that the percentage of 1-star reviews on Amazon was out of proportion as shown in Table 1.

Table 1. Rating vs. review by ranking (1).

• The data sources of Primatene mist ratings were extracted from Amazon and Walmart, as of 17 May, 2022, and 29 May, 2022, respectively.

• Amazon captures product opinions in both the global ratings and reviews, where Walmart posts only customer reviews.

• Walmart customer reviews by categories.

• About 24% of patients rated the product a 1-star also wrote customer reviews, which is five-time higher than the average of 5% who submitted reviews and rated Primatene a 2-star or higher.

• The distribution of proportions is almost identical between Amazon global ratings and Walmart customer review throughout the rankings.

Breakdowns on review title

The customer reviews are further broken down into two tiers according to the product experiences reportedly in the review title. As the first-tier, reviewers who ranked the product a 3-star or higher reported good product performance (82.5%: 433 of 525) with various benefits of using the Primatene. Further, effectiveness was explicitly cited by 69.5% (248 of 357), 40.7% (22 of 54) and 47.6% (10 of 21) in the subgroups of 5-star, 4-star and 3-star, respectively (Table 2), making it a watershed to split the groups of ranks into tiers.

Table 2. Product attributes breakdown by tier 1 (title of customer reviews).

For the remaining patients who rated Primatene a 1-star and 2-star there was not a single review cited effectiveness therefore belonging to the second tier. In contrast, patients in the second tier reported the ineffectiveness, stressing more on problems with the pharmaceutical. The population of the first tier (rated the product a 3-star or higher) represents a higher proportion (95.2%: 10,281 of 10,802) in star ratings. As shown in Tables 2 and 3 elaborates the major attributes of Primatene, backed by the keywords captured in the review title, by tier.

Table 3. Product attributes breakdown by tier 2 (title of customer reviews).

Effectiveness and safety

Given that the out of proportion in ranking was exhibited between the star ratings and its proper subset in customer reviews, statistical analyses were performed based on the proportions of customer reviews and star ratings, respectively as shown in Table 4.

Table 4. Statistical analyses on effectiveness.

Other aspects

The nutshell of the categorical definition for other aspects of Primatene was based on the requirements and principles revealed in the “Cross-discipline team leader review for Primatene mist, NDA/BLA# 205920 and SDN-73”. 10/24/2018, by Center for Drug Evaluation and Research (CDER).

It is to reveal the degree of compliances in the regulatory requirements in six other aspects of the Primatene, i.e., number of reported issues on: Device integrity such as leaked units; metering issues in over/under counting; clarity of the label instructions under the OTC settings; device clogging or cleaning; aspects of formulation such as uniformity; and irregular dosing, reported by the patients as shown in Table 5.

Table 5. Other issues by category.

Discussion

E-commerce has become a mainstream part of our daily activities therefore e-commerce portal can be utilized as a new viable source for post-market surveillance studies in pharmaceuticals. Primatene mist was chosen to illustrate the process of utilizing the new source for safety and performance evaluation. It was reformulated and approved by FDA in November 2018, for the temporary relief of the mild symptoms of intermittent asthma. According to the customer reviews posted on Amazon portal, statistically Primatene mist is safe and effective in performance. The MDI is easy to use with good quality in device integrity.

Decipher the sentiment

The patients’ main sentiments in using the Primatene are elaborated in six dimensions below.

Effectiveness and safety: Primatene works for 82.5% and 95.2% of patients, in accordance with the customer reviews and star ratings, respectively. Specifically, among the 525 reviewers 1 reported ER visit overnight (in 2-star pool) and 2 experienced Adverse Drug Events (ADEs) (in 1-star rating).

Pros and cons of OTC settings: Patients explicitly reported that (1) “economic justification in cost”-2.5% (13 of 525); and (2) “an alternative medication to Rx” -4.4% (23 of 525) are the top 2 facts as the pros of using the Primatene.

Patients also reported its effectiveness for the relief of respiratory symptoms with causes other than asthma. Returning to the prescription medications, if it doesn’t work, is the Con for using this OTC medication.

Device integrity: Patients reported 1 alleged clog (in 2-star) and 4 leaked or damaged units (in 1-star pool).

Formulation issues: Patients also reported side effects of Primatene (in 1 to 3-star), ranging from heat or fire in lung or throat (8), or test issues such as awful, nasty or smell (10), etc. There was no evidence reported in non-uniformed formulation.

Clarity of label instructions: Patients explicitly reported that the functioning and operating of the MDI were exactly as label described or advertised (in 5-star and 4-star) for the self-mastering of the device; there was no reporting on unclear instructions or experienced difficulties in using the device.

Best and worst attributes: “Effectiveness”-53.5% (281 of 525), “A good product”-15.2% (80 of 525) and “high quality”-4.4% (23/525) are the three attributes that patients explicitly rated as the best about Primatene. However, 3.2% (17 of 525) patients also reported that the bad taste of the formulation (in pools of 4-star and lower) was the worst attribute.

Seasonality of review

The change in number of reviewers seems seasonal, i.e., it increases along the changing season to Spring and Winter based on a single year experience in 2021, with a monthly average of 23 reviews posted on Amazon portal.

There were two peaks in April and December in the number of reviews throughout the year. During a three-month period, it moved upward sharply from the minimum of 9 counts in January and reached the maximum of 45 in April-a 196% jump from the monthly average. The trend flattened afterward, starting in May and then oscillated slightly around the monthly average during the seasons of summer and fall. It dropped for the second time to 14 counts in November. It then almost doubled in December (in just 30 days) with 27 counts, as the second peak (a 192% spike) throughout the year. The timing of change in number of reviewers could be potentially driven by the demand of the medication which coincides with the change in season. The seasonal change in demand, if confirmed statistically, also reveals that Primatene lands in the designated pool of patients with intermittent asthma, matching the label claim of the product.

Logistic handling of the OTC

Very few incidents were reported in the handling in mail order of the medication in delivery. No other types of concerns in handling under the stand of OTC.

Foreseeable improvement

There are two areas for potential improvements.

• To explore the feasibility of improving the formulation to make its taste less unattractive.

• To study new policies for the return to the manufacturer of all units with alleged device issues for investigation.

Conclusion

The analysis of consumer ratings and reviews for Primatene Mist, an FDA-approved OTC inhaler, offers significant insights into its post-market performance and safety profile. The study reveals that a substantial majority of users (95%) report positive experiences, particularly emphasizing the inhaler's effectiveness in alleviating mild symptoms of intermittent asthma. Notably, 53.5% of reviewers explicitly highlight its efficacy, while safety concerns remain minimal, with only 0.6% of users reporting adverse events. This suggests that Primatene Mist is generally regarded as safe and effective for its intended use, despite some isolated issues related to device integrity and the unpleasant taste of the formulation. The benefits of using this OTC inhaler, such as cost-effectiveness and accessibility, contrast with the potential drawbacks of needing to return to prescription medications. Additionally, seasonal patterns in review data indicate a correlation between increased usage and changes in weather, further validating its role for asthma patients. Overall, the findings underscore the value of consumer feedback from e-commerce platforms for post-market surveillance, supporting the ongoing use and refinement of Primatene Mist as a reliable OTC option for asthma management.

References

1. Amazon portal. 1 best seller in asthma medication. Accessed: 2022.
2. Feldman WB, et al. Switching to over-the-counter availability of rescue inhalers for asthma. Jama. 2022;327:1021-1022.

   [Crossref] [Google Scholar] [PubMed]

3. Kelty JL, et al. Cross-discipline team leader review for primatene mist. Center for Drug Evaluation and Research (CDER). 2018a;82.
4. Blue Cross NC. Over-the-Counter Medication. Accessed: 2022.
5. Asthma and Allergy Foundation of America, 2010. Asthma facts and figures. Accessed: 2022.
6. Center for Disease Control and Prevention (CDC), 2006~2010. Asthma severity among adults with current asthma. Accessed: 2022.

   [Google Scholar]

7. Amazon portal. How customer reviews and ratings work. Accessed: 2022.
8. Feedcheck. Connecting brands to their product ratings and reviews across any marketplace. Accessed: 2022.
9. Walmart portal. Customer reviews and ratings. Accessed: 2022.
10. Hautala L. Amazon’s never-ending fake reviews problem, explained. CNET. 2021;23.

    [Google Scholar]