




gregory.tech
2.8K posts

@gregorytechKE
Data | Software Development | Founder https://t.co/rJVdmQSerY |








created tables jobs; and scrape logs;



I chose different sources: adzuna, brightermonday, fuzu, glassdoor, google search, indeed, linked-in, myjobsinkenya.

built a jobs data pipeline to solve fragmented job listings. got data from multiple sources, run DAGs in Airflow to get data daily at 2pm data is cleaned and stored in PostgreSQL and visualized in Grafana. here: github.com/gregory-bot/jo…





created tables jobs; and scrape logs;

@gregorytechKE Doing something similar for a personal project: Raw files to local Warehouse to local SQLite DB for transformation to Dashboard in streamlit now moving it to the cloud: Raw files to Data Lake (S3) to Warehouse (Redshift where transformation occurs) to Dashboard (Streamlit)







created tables jobs; and scrape logs;



created tables jobs; and scrape logs;