You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
grace 8aa0e98758 Make a simple example program. 6 days ago
app Run basic rails process once on `boom` machine. 11 months ago
bin Merge rails codebase inside main reap node. 12 months ago
cache Remark on our cache. 8 months ago
config Pull on a schedule. 11 months ago
db Add `crono` scheduler. 11 months ago
lib Scrape all UScode XML bundles. 2 months ago
log Merge rails codebase inside main reap node. 12 months ago
public Merge rails codebase inside main reap node. 12 months ago
storage Merge rails codebase inside main reap node. 12 months ago
tmp Merge rails codebase inside main reap node. 12 months ago
vendor Merge rails codebase inside main reap node. 12 months ago
.gitattributes Merge rails codebase inside main reap node. 12 months ago
.gitignore Rename `/orgs/` -> `/cache/`. 12 months ago
.ruby-version Prepare publishing on `silicon.am`. 6 months ago
Dockerfile Prepare publishing on `silicon.am`. 6 months ago
Gemfile parallel-process scraping etymonline phrases. 3 months ago
Gemfile.lock parallel-process scraping etymonline phrases. 3 months ago
Procfile Prepare publishing on `silicon.am`. 6 months ago
README.md Send paylinks absence process commands as-you-go. 2 months ago
Rakefile Merge rails codebase inside main reap node. 12 months ago
app.json Prepare publishing on `silicon.am`. 6 months ago
channel.apple.sh begin by pulling only 6 records. 4 months ago
config.ru Merge rails codebase inside main reap node. 12 months ago
coursera.process.rb Pull coursera courses. 6 months ago
crunchyroll.process.rb Make direccions more clear. 5 months ago
daily.sh Make reddit parser and daily pull more reliable. 10 months ago
docker-compose.yml Prepare publishing on `silicon.am`. 6 months ago
duckduckgo.process.rb Make direccions more clear. 5 months ago
etymonline.process.rb Make repairs in old scrapers. 2 weeks ago
example.process.rb Make a simple example program. 6 days ago
go Pull on a schedule. 11 months ago
irs.process.rb Bump up resilience. 4 months ago
justia.process.rb Small upgrades in `justia.process.rb`. 2 weeks ago
mercury.disburse.rb [cash] add and enhance logs. 5 months ago
municode.process.rb Make direccions more clear. 5 months ago
paylinks.absence.rb explain "Reason" query, describe as a summary. 2 months ago
paylinks.hours.rb Send paylinks absence process commands as-you-go. 2 months ago
pluto.process.rb Make direccions more clear. 5 months ago
reddit.process.rb Make repairs in old scrapers. 2 weeks ago
sec.process.rb Add in missing addresses on each held bin. 5 months ago
sec.record.rb Rename `process` and `record` using org name. 12 months ago
senate.recordings.rb [senate] repair error in page addresses. 5 months ago
translate-google.process.rb General clean up. 8 months ago
twitter.process.rb Change twitter scraper's `.env` -> `.call`. 4 months ago
urbandictionary.process.rb [urbandictionary] oh dear, here goes... 2 months ago
uscode.process.rb [uscode] Speed up and bump resilience. 2 months ago

README.md

Reap -

Pulls many online sources,
and pays some bills.

Add Dependencies.

You need some online scraping engines:

Grab a copy of each one, and place inside ~/bin. Run: echo 'export PATH=$PATH:~/bin' >> ~/.bashrc; . ~/.bashrc; so your shell realizes you added the programs.

In geckodriver's case, you should extract the zipped file you pulled, and place the unzipped binary inside ~/bin. Also make sure your machine has a copy of Firefox.

On Mac Homebrew:

  • brew install chromedriver geckodriver yt-dlp

You also need our code and some ruby programs. Begin by grabbing Ruby 3.1.2.

  • git clone git@base.assembled.app:code/reap
  • cd reap
  • gem install bundler
  • bundle install --with=pull

Run programs.

All sourced records are placed under ./cache/*/

Open a MyPayLinks calendar:

ruby paylinks.absence.rb
ruby paylinks.hours.rb

Record "leave" hours in MyPayLinks:

cat <<END >> .call
domain.user=MyNameHere
domain.passcode=P4ssC0de
END
ruby paylinks.absence.rb 1 V 1 4-8 11-13

... this records one hour of annual 'V'acation, on this month's days #1, 4,5,6,7,8, 11,12,13. See paylinks.absence.rb, REASONS hash.

You can also,

ruby paylinks.hours.rb 1 1-5 8-12 15 17-19
  • 1 means one hour applied per day
  • 1-5 8-12 15 17-19 are day ranges inside month.

Should you need changes in a prior month, re-enable paylinks.hours.rb#48.

Senate

Scrape senate floor proceedings

ruby senate.recordings.rb # or...
ruby senate.recordings.rb 2022

... check inside ./cache/senate.

IRS

Pay your bills; especially prior years.

cat <<END >> .call
irs.address=your@email.here
irs.passcode=your_id.me_passcode
irs.routing=0011223344
irs.account=4433221100
irs.pay_by=2022-10-01
END
ruby irs.process.rb

Crunchyroll

Pull some anime!

  • ruby crunchyroll.process.rb
  • ruby crunchyroll.process.rb https://www.crunchyroll.com/bleach
  • ruby crunchyroll.process.rb bleach

Once you indexed, pull a season:

cd cache/crunchyroll/bleach/00-Bleach\ Season\ 1/
./_source.rb

DuckDuckGo

Run a search.

ruby duckduckgo.process.rb 8 Videos gecko reflow

Etymonline

Learn some old language. Requires around a day.

ruby etymonline.process.rb

Municode

Scrape all indexed US municipal codes. Requires many days.

ruby municode.process.rb

Reddit

Mainly pulls images and camera recordings.

ruby reddit.process.rb ProgrammerHumor

SEC

Pull business records.

ruby sec.process.rb AAPL AMZN FB GOOGL MSFT TSLA TWTR

You can run more analysis, by requiring rails dependencies:

bundle install --with=rails
ruby sec.record.rb

Twitter

Sources money analysis remarks and economic models.

echo "TWITTER_TOKEN=abc123" >> .call
ruby twitter.process.rb

Pluto.TV

Check in on cinema:

ruby pluto.process.rb

Rails dependencies: display your reaped records.

Run postgres locally on your machine:

  • Mac: brew install postgresql; brew services start postgresql.
  • Fedora: sudo dnf install postgresql; sudo systemctl start postgresql.
  • ...and so on,

and use:

bundle install --with=rails
rails db:create db:migrate db:seed
rails s

Roadmap:

Leadership and Spending.

Geography and Mapping

Online Meshes

Online Shops.

region address hack number_records_guess guarded
Alabama http://www.doc.state.al.us/InmateSearch sql 25060
Arizona https://corrections.az.gov/public-resources/inmate-datasearch skim-6 NULL
Arkansas https://apps.ark.org/inmate_info/search.php sql 16341
California https://inmatelocator.cdcr.ca.gov/ sql 2000
Colorado http://www.doc.state.co.us/oss/index.php?ref=home skim-abc NULL
Connecticut http://www.ctinmateinfo.state.ct.us/searchop.asp sql 14640

Business records.

region address hack number_records_guess guarded
Delaware https://icis.corp.delaware.gov/eCorp/EntitySearch/NameSearch.aspx skim-abc NULL recaptcha

Language phrase books.

...and similar sources in many languages.

Media