Facebook Scraper For Multiple Pages

Scrapemultiplepublic Facebook pages en masse to help yield social analytics. Automatically fetch more detailed insights for owned pages' posts and videos to see what content strikes well. Go all the way back in time or specify two dates and watch the goodness happen at lightning speed via multi-threading!

Distinguishing Features

• Multi-threadedfor rapid data collection frommultiple pagessimultaneously(as many as you want!)
• Collect detailed performance metrics on multiple owned business pagesautomaticallyvia theinsightsandvideo insightsendpoints
• Retrieve number of public shares of a link byanyoneacross Facebook via theURL Object
• Custommetrics computed:
  • Impression Rate Non-Likers (%):exploreviralityof your posts outside your typical audience
  • Engagement Rate:(Shares + Reactions + Comments) / Total Unique Impressions
  • Adjusted Engagement Rate (%)andAdjusted CTR (%):normalise rates across pages of different audience sizes and account for uncertainty in small numbers i.e. 5/10 CTR < 100/200 CTR as detailed byEvan Miller
• Proper timezone handling

What can be collected from public page posts?

Post ID, Publish Date, Post Type, Headline, Shares, Reactions, Comments, Caption, Link

... and optionally with a performance cost:
Public Shares, Likes, Loves, Wows, Hahas, Sads, Angrys

What isadditionallycollected from owned page posts?

Posts

Video Views, Unique Impressions, Impression Rate Non-Likers (%), Unique Link Clicks, CTR (%), Adjusted CTR (%), Engagement Rate (%), Adjusted Engagement Rate (%), Hide Rate (%), Hide Clicks, Hide All Clicks, Paid Unique Impressions, Organic Unique Impressions

Videos

Live Video, Crossposted Video, 3s Views, 10s Views, Complete Views, Total Paid Views, 10s/3s Views (%), Complete/3s Views (%), Impressions, Impression Rate Non-Likers (%), Avg View Time

Setup

1)Add the page names you want to scrape insidePAGE_IDS_TO_SCRAPE

Grab the @'handles' or in url (e.g. 'vicenews' below).

2)Grab your owntemporaryuser tokenhereand place insideOWNED_PAGES_TOKENS:
Get Token -> Get User Token -> Get Access Token

OWNED_PAGES_TOKENSis the dictionary that stores the token(s) necessary to scrape public data. If the token is apermanent tokenfor a business page, it is used to scrape private data provided that the page is placed inPAGE_IDS_TO_SCRAPEand its corresponding key is identically named in this dictionary.

3)Install Python dependencies withpip install requests scipy pandas

N.BOSX users should have installedHomebrewand Python withbrew install Python

Execution

Specify number of days back from present:

Python get_fb_data.py post 5(Public & owned pages)
Python get_fb_data.py video 5(Owned pages only for video-specific data)

Specify two dates (inclusive) in yyyy-mm-dd format:

Python get_fb_data.py post yyyy-mm-dd yyyy-mm-dd
Python get_fb_data.py video yyyy-mm-dd yyyy-mm-dd

The csv file is placed in thefacebook_outputfolder by default

Credit

Thanks to minimaxir and hisprojectfor showing me the ropes

FYI

Additionalsocial_elastic.pyused to scrape dataandpush to Elastic instance(s) via theirbulk api