Working with online services means that you’ll inevitably run into a situation where you need to scrape some sort of data. Sometimes you only need to do it once, in other cases, you might want to set up an ongoing process. In any case, there are some ethical considerations to make in a scenario like this. You have to be aware of the moral implications of what you’re doing, and also be prepared to be met with resistance by the owners of the data you’re trying to scrape.
It’s an ongoing game of cat and mouse, and it’s never easy to tell which side has the upper hand.
Know How to Get the Most Out of Your Activities
To be efficient at scraping, prepare to conceal your tracks to some extent. It’s a good idea even when you’re doing it without any unethical intentions on your mind. Having a good, reliable proxy server is always necessary in these scenarios and a Selenium rotating proxy can provide you with a lot of flexibility in your actions. It will allow you to easily circumvent certain regional restrictions, giving you access to the full range of content that might be hosted on a server. Click here to learn more about selenium proxies from tech-heads at Smartproxy.
Additionally, you should learn how to develop your own tools as much as you can. This is not just an extra bonus these days – it’s a basic requirement if you want to get anywhere in this field. Using customized, purpose-written tools is the only way to ensure that you’ll match your requirements completely.
Provide Contact Information
Unless you’re scraping for illicit reasons, you should always provide some sort of contact details to the person you’re scraping your data from. This is commonly done in the user agent headers, but that’s not a strict requirement. It’s just the first place most site owners would look when they suspect that someone is scraping their data. Some systems would have more appropriate ways to leave your contact details.
If you do get contacted, don’t immediately become defensive. Many site owners are actually fine with their data being scraped when it’s not for malicious reasons, but they might have some issues with the specific way you’re going about it. Rate limits and things like that can be negotiated for each specific case.
The Thin Line Between Scraping and Exploiting
You also need to think about the exact methods you’re utilizing to get access to the data you’re after in the first place. If there’s no public API, then scraping is the only way to go about that. But sometimes scraping can force you to rely on exploits in the target system (known or unknown), in which case the owner of the services you’re scraping will definitely not be happy with what’s going on. Ask yourself if you’re using the system in an intended manner. For example, trying to discover every profile on a social network by scanning user IDs incrementally may raise some eyebrows. If you don’t think you should have access to this data in the first place, then really consider what you’re doing.
This brings us to our next, more general point. You must ensure to always comply with ethical norms when going about your scraping activities. In other words, don’t attempt to obtain data that you should not have in the first place. Your intents with regards to that data are also important here. If you’re simply scraping to build up your own library of something, that’s fine. For example, you might be interested in the activities of a certain celebrity, which are spread out across multiple wikis and social networks. But if you plan on reselling that data, or otherwise taking advantage of it for personal gain, that’s crossing a line that definitely should not be crossed.
There are also some considerations to be made that extend out of the moral world and directly into the legal one. Are you actually allowed to do what you’re doing?
Whether you agree with it or not, some types of scraping can be considered illegal. Companies have commonly brought up their Terms of Service as a defense in cases like these. It’s not uncommon for a site or service to have explicit clauses in its TOS that state that you’re not allowed to do any actions that cause unnecessary strain on their networks. And while there is a lot of ground for arguing against these clauses in court, it often means taking on an uphill battle against companies with a large number of lawyers and general resources. Sometimes their goal isn’t even to win – it’s to drag out the case long enough to ruin you financially.
The bottom line is, be careful not to fall under the crosshairs of the wrong organization. This could cost you dearly in the long run and can have long-lasting repercussions.
What to Do if You Come Across Something Unusual
Sometimes your scraping activities might lead you to discover something out of the ordinary you shouldn’t be seeing in the first place. For example, scraping a certain site for all available data might reveal admin-only pages, or sections containing the private data of users. The ethical thing to do in these cases is to notify the site owners and let them fix the problem.
- Ethics 101: What Every Leader Needs To Know
- There’s No Such Thing As “Business” Ethics: There’s Only One Rule for Making Decisions
- How to Gather Marketing Intelligence with Web Scraping Tools
- Conversion Optimization: How to Optimize Your Website for Visitors
- How to Use Web Metrics to Improve Your WebSite