Skip to content

Commit eb406f1

Browse files
rmariuzzormariuzzo
committed
A new world
0 parents  commit eb406f1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+5522
-0
lines changed

Diff for: ‎.env.default

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
OSD_GH_TOKEN=

Diff for: ‎.eslintignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
build
2+
node_modules

Diff for: ‎.eslintrc.js

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
module.exports = {
2+
root: true,
3+
parser: '@typescript-eslint/parser',
4+
plugins: ['@typescript-eslint'],
5+
extends: [
6+
'eslint:recommended',
7+
'plugin:@typescript-eslint/eslint-recommended',
8+
'plugin:@typescript-eslint/recommended',
9+
'prettier/@typescript-eslint',
10+
],
11+
}

Diff for: ‎.github/workflows/scrape.yml

+57
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
name: Scrape data each hours
2+
3+
on:
4+
schedule:
5+
- cron: '0 * * * *'
6+
repository_dispatch:
7+
types: [trigger]
8+
9+
env:
10+
NODE_ENV: production
11+
OSD_GH_TOKEN: ${{ secrets.OSD_GH_TOKEN }}
12+
13+
jobs:
14+
scrape:
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- uses: actions/checkout@v2
19+
20+
- name: Read .nvmrc
21+
run: echo ::set-output name=NVMRC::$(cat .nvmrc)
22+
id: nvm
23+
24+
- name: Setup node
25+
uses: actions/setup-node@v1
26+
with:
27+
node-version: '${{ steps.nvm.outputs.NVMRC }}'
28+
registry-url: 'https://registry.npmjs.org'
29+
30+
- name: Cache node modules
31+
uses: actions/cache@v2
32+
env:
33+
cache-name: cache-node-modules
34+
with:
35+
path: ~/.npm
36+
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
37+
restore-keys: |
38+
${{ runner.os }}-build-${{ env.cache-name }}-
39+
${{ runner.os }}-build-
40+
${{ runner.os }}-
41+
42+
- name: Install dependencies
43+
run: npm i
44+
45+
- name: Run data migrations
46+
run: npm run migrations:run
47+
48+
- name: Scrape data
49+
run: npm run scrape
50+
51+
- name: Generate data
52+
run: npm run generate
53+
54+
- uses: stefanzweifel/[email protected]
55+
with:
56+
commit_message: Auto-update data.
57+
branch: master

Diff for: ‎.gitignore

+108
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Logs
2+
logs
3+
*.log
4+
npm-debug.log*
5+
yarn-debug.log*
6+
yarn-error.log*
7+
lerna-debug.log*
8+
9+
# Diagnostic reports (https://nodejs.org/api/report.html)
10+
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
11+
12+
# Runtime data
13+
pids
14+
*.pid
15+
*.seed
16+
*.pid.lock
17+
18+
# Directory for instrumented libs generated by jscoverage/JSCover
19+
lib-cov
20+
21+
# Coverage directory used by tools like istanbul
22+
coverage
23+
*.lcov
24+
25+
# nyc test coverage
26+
.nyc_output
27+
28+
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
29+
.grunt
30+
31+
# Bower dependency directory (https://bower.io/)
32+
bower_components
33+
34+
# node-waf configuration
35+
.lock-wscript
36+
37+
# Compiled binary addons (https://nodejs.org/api/addons.html)
38+
build/Release
39+
40+
# Dependency directories
41+
node_modules/
42+
jspm_packages/
43+
44+
# TypeScript v1 declaration files
45+
typings/
46+
47+
# TypeScript cache
48+
*.tsbuildinfo
49+
50+
# Optional npm cache directory
51+
.npm
52+
53+
# Optional eslint cache
54+
.eslintcache
55+
56+
# Microbundle cache
57+
.rpt2_cache/
58+
.rts2_cache_cjs/
59+
.rts2_cache_es/
60+
.rts2_cache_umd/
61+
62+
# Optional REPL history
63+
.node_repl_history
64+
65+
# Output of 'npm pack'
66+
*.tgz
67+
68+
# Yarn Integrity file
69+
.yarn-integrity
70+
71+
# dotenv environment variables file
72+
.env
73+
.env.test
74+
75+
# parcel-bundler cache (https://parceljs.org/)
76+
.cache
77+
78+
# Next.js build output
79+
.next
80+
81+
# Nuxt.js build / generate output
82+
.nuxt
83+
dist
84+
85+
# Gatsby files
86+
.cache/
87+
# Comment in the public line in if your project uses Gatsby and *not* Next.js
88+
# https://nextjs.org/blog/next-9-1#public-directory-support
89+
# public
90+
91+
# vuepress build output
92+
.vuepress/dist
93+
94+
# Serverless directories
95+
.serverless/
96+
97+
# FuseBox cache
98+
.fusebox/
99+
100+
# DynamoDB Local files
101+
.dynamodb/
102+
103+
# TernJS port file
104+
.tern-port
105+
106+
src/db/data-development.sqlite
107+
build
108+
.env

Diff for: ‎.nvmrc

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
v14.16.1

Diff for: ‎.prettierrc

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"semi": false,
3+
"singleQuote": true,
4+
"trailingComma": "es5",
5+
"printWidth": 100
6+
}

Diff for: ‎.sequelizerc

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
const path = require('path')
2+
3+
module.exports = {
4+
config: path.resolve(__dirname, 'src', 'db', 'config.json'),
5+
'models-path': path.resolve(__dirname, 'src', 'db', 'models'),
6+
'seeders-path': path.resolve(__dirname, 'src', 'db', 'seeders'),
7+
'migrations-path': path.resolve(__dirname, 'build', 'migrations'),
8+
}

Diff for: ‎.vscode/settings.json

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"typescript.tsdk": "node_modules/typescript/lib"
3+
}

Diff for: ‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2020 Developers Dominicanos
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

Diff for: ‎README.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Opensource Data
2+
3+
Data source for [opensource frontend](https://developersdo.github.com/opensource). This repo **[scrape data hourly](https://github.com/developersdo/opensource-data/actions)** and **[publish it as JSON files](https://developersdo.github.io/opensource-data/)**.
4+
5+
## Usage
6+
7+
Data sources are available at: [developersdo.github.io/opensource-data](https://developersdo.github.io/opensource-data).
8+
9+
## Development
10+
11+
You will need to:
12+
13+
1. Clone this repo.
14+
2. Install dependencies: `npm i`
15+
3. Run migrations: `npm run migrations:run`
16+
4. Copy `.env.default` to `.env`.
17+
5. [Create a personal GitHub access token](https://help.github.com/articles/creating-an-access-token-for-command-line-use/) (with no scopes) and add the value to `OSD_GH_TOKEN` in your `.env`.
18+
- <small>Note: You can skip that if you don't need to run the scraper locally.</small>
19+
20+
## Scripts
21+
22+
| Script | Description |
23+
| -------------------------------- | ------------------------------------ |
24+
| `npm run migrations:run` | Run all migrations. |
25+
| `npm run migrations:add [name]` | Add a new migration. |
26+
| `npm run scrape` | Scrape all data. |
27+
| `npm run scrape -- --only=users` | Scrape users data only. |
28+
| `npm run scrape -- --only=repos` | Scrape repos data only. |
29+
| `npm run generate` | Generate public site and JSON files. |
30+
31+
## Debugging
32+
33+
Prepend any commmand with `DEBUG=osd:*`. For example:
34+
35+
```sh
36+
DEBUG=osd:* npm run scrape
37+
```

Diff for: ‎docs/index.html

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
7+
<meta http-equiv="X-UA-Compatible" content="ie=edge">
8+
<title>Opensource Data</title>
9+
</head>
10+
11+
<body>
12+
<h1><a href="https://github.com/developersdo/opensource-data">Opensource Data</a></h1>
13+
<h2>Data Sources</h2>
14+
15+
<ul>
16+
17+
<li><a href="./users.json">users.json 1.23 MB (gzip:
18+
265 kB)</a>
19+
</li>
20+
21+
<li><a href="./repos.json">repos.json 14.9 MB (gzip:
22+
2.51 MB)</a>
23+
</li>
24+
25+
</ul>
26+
27+
<p><strong>Last Updated:</strong> Thu Mar 16 2023 19:38:05 GMT+0000 (Coordinated Universal Time)</p>
28+
</body>
29+
30+
</html>

Diff for: ‎docs/repos.json

+1
Large diffs are not rendered by default.

Diff for: ‎docs/users.json

+1
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)