The Importance of Having an Ethical Data Collection Policy

Flipnode on May 04 2023

An effective external data collection policy can bring valuable benefits to your B2B company beyond preventing unethical behavior. In addition to safeguarding your company's reputation, it can significantly improve the efficiency of your scrapers.

Recognizing unethical data collection

When conducted ethically, B2B data collection can offer numerous advantages for companies. In the retail industry, this can result in improved data that can optimize transactions such as price, features, and quality for both individuals and businesses.

There are countless scenarios where data collection can benefit society as a whole. These include enhancing educational standards, facilitating price comparisons across various goods and services, reducing waste by aligning demand with manufacturing output, mitigating risks in investment scenarios, and many more.

To ensure ethical data collection, certain characteristics must be taken into account during the actual gathering process. Ethical data gathering should be unobtrusive, have a low impact (with low request rates), be efficient, thoughtful, and governed by well-communicated practices that development teams adhere to.

On the other hand, unethical data collection occurs when data sources are not publicly available and are behind paywalls or require registered user accounts.

When collecting data from the public domain, unethical practices involve:

• Indiscriminately collecting all available data without regard for speed or efficiency;
• Disregarding the impact on the website or platform being accessed;
• Lack of planning or consideration in the data collection process;
• Failure to evaluate the impact of actions on the website or platform.

This policy applies specifically to industry data that does not involve personally identifiable information (PII). Having an ethical data collection policy in place makes it much easier to distinguish between what is ethical and unethical.

Going the distance with a good policy

The team at Merit Data and Technology comprises more than 24 data engineers, developers, and business analysts who develop data harvesting tools for various industries. They have observed firsthand how a well-defined policy can significantly enhance the effectiveness of the scrapers they create. This policy has helped identify instances where database calls were excessive, duplicate data was being continuously collected, and poor design decisions led to suboptimal outcomes.

Having a common baseline of best practices for data collection has resulted in substantial cost savings and has facilitated the development of stronger collection tools across the board.

An ethical data collection policy prompts developers to think more profoundly about their approach to a given source, and this moment of contemplation is always beneficial. It leads to better technical outcomes in terms of reliability, efficiency, and design for each developed scraper.

Benefits of ethical data collection policy:

• Ensuring all team members are aligned on critical dos and don'ts in scraper design.
• Leading to better quality code and more easily maintainable components since scrapers that violate ethical guidelines are usually poorly designed.
• Encouraging customers and data users to carefully consider their needs, resulting in more focused and efficient data collection.
• Significantly reducing costs through better design thinking and the use of best practices.
• Helping to prevent potential lawsuits.
• Avoiding reputational damage and protecting the value of the company.

Developing the policy: Crucial elements

When creating your first ethical data collection policy, there are some general points to consider. Apart from industry-specific requirements, companies should include "best practice" norms they want their developers to follow when building scrapers.

One of the key elements of the policy should be a "design and approach" document that outlines how each source site will be approached from both an ethical and efficiency/reliability standpoint. This document should be completed before coding begins.

The policy should be relatively simple and written in plain language to capture the spirit and intent of how data should be collected.

Below are some possible revised principles to consider:

• Avoid collecting data that is restricted by paywalls or requires user authentication.
• Limit the frequency and volume of requests to minimize the impact on the target website.
• Follow the guidelines provided in the website's robots.txt file, if available.
• Collect only the data that is necessary for your business purposes and avoid excessive scraping.
• Do not simulate user actions such as adding items to a shopping cart or making purchases.
• Stop data collection when encountering frequent errors and review the scraper code to determine the cause.

Implementing the policy: How to succeed

Once you have created the policy, it's essential to ensure it is effectively implemented and not disregarded. It is crucial to implement the policy in a way that does not overburden the development team. The following approach may assist in ensuring that the policy is well-received and efficiently enforced.

1. Discuss it with the tech team
   Be transparent with the tech team and acknowledge that a policy may initially create challenges. However, emphasize the advantages of adhering to shared guidelines and industry best practices. Encourage developers to innovate and think creatively when implementing the policy, as this may present new opportunities for growth and improvement.
2. Create an escalation path
   In cases where the policy cannot be followed, it is important to have senior staff review the developers' work. It is recommended to randomly audit a percentage of collection scripts each month or quarter and conduct a full audit on a dozen robots per quarter to ensure policy adherence. It's crucial to check the site evaluation reports for thoroughness and review scripts and robots for various aspects, such as site handling, speed, analytics impact, and error handling.
3. Motivate with the carrot and the stick
   Establishing consequences for non-compliance and incentives for excellent data collection codes is crucial to establish and enhancing standards.

Final thoughts

The above guidelines are universal and suitable for any B2B business that prioritizes ethical data acquisition or is only to embark on this path. Even though implementing the ethical policy may take some time and resources during the first stages, in the long run, it will be rewarding in many different ways.