Prove What You Know

Build Your Portfolio

In South Africa's data job market, a strong portfolio often matters more than your qualifications alone. Here's how to build one that gets noticed.

Project ideas are different for every role. Jump to yours:

For Data Analysts

Data Analyst Portfolio Projects

Each of these can be built with Excel/Power BI for a first pass, then rebuilt in Python as your skills grow.

Interactive portfolio examples

Why this is a Junior Data Analyst project: Demonstrates EDA, SQL query design, clean visualisation, and communicating a recommendation to a non-technical audience. Reviewers want to see that you can frame a business question, handle messy data, pick the right chart, and write a conclusion someone can act on.
Junior
Load-shedding Impact Analysis -- Eskom Public Data
EDA and Power BI dashboard using publicly available Eskom outage data -- framed as a briefing for a business operations team
SQLPower BIExcelPython
Total Hours Lost 2023
6,552
Eskom outage data
Avg Daily Hours Lost
8.4
Up from 3.7 in 2022
Worst Month
July
680 hrs / 21.9 per day
Stage 6 Days
71
in 2023
Monthly outage hours -- 2023
Load-shedding stage distribution (days)
Avg daily hours lost by stage
Outage hours by time of day (avg)
Escalation trend -- stage over time
loadshedding-impact/
├── data/
│   ├── raw/
│   │   └── eskom_outages_2023.csv
│   └── processed/
│       └── outages_clean.csv
├── notebooks/
│   ├── 01_eda.ipynb
│   └── 02_stage_analysis.ipynb
├── sql/
│   ├── monthly_totals.sql
│   └── stage_distribution.sql
├── powerbi/
│   └── loadshedding_dashboard.pbix
├── reports/
│   └── ops_briefing.pdf
└── README.md
# Load-shedding Impact Analysis -- Eskom 2023 EDA and Power BI dashboard using public Eskom outage data, framed as a briefing for a business operations team. A first portfolio project using a dataset every South African business lived through. ## Key Findings 2023 was the worst year on record -- 6,552 total outage hours July peaked at 21.9 hrs/day. Stage 6 ran for 71 days 18:00 -- 20:00 had the highest concentration of outage hours ## Stack Python 3.10 | pandas | matplotlib | PostgreSQL | Power BI ## How to Run Open notebooks/01_eda.ipynb for the cleaning walkthrough SQL queries run on any PostgreSQL 13+ instance Power BI Desktop (free) required for the .pbix file
Slide 01
Problem
Load-shedding cost SA businesses an estimated R1.4 billion per day in 2023
When were the worst hours? Which stages caused the most damage? This analysis quantifies impact using public Eskom data.
Slide 02
Data
365 days of outage records cleaned and structured for analysis
Joined stage duration data with time-of-day segmentation.
6,552total hours lost in 2023
Slide 03
Finding
July was the worst month. Stage 6 ran for 71 days across the year
18:00 -- 20:00 carried the highest concentration of outage hours. Evening operations were most disrupted.
Slide 04
Recommendation
Shift energy-intensive operations before 16:00 to reduce diesel dependency
Teams running key processes outside the peak window could cut generator runtime by an estimated 20 -- 30%.
What a Junior Data Analyst portfolio must demonstrate

You do not need work experience to build a strong junior DA portfolio. Public datasets from Kaggle, StatsSA, or any open government source are completely acceptable. Two to four projects is enough. Each should open with a clear question, walk through the data cleaning decisions, and close with a finding a non-technical person can understand. Reviewers are not looking for complexity -- they want to see that you can frame a problem, handle messy data, choose the right chart, and write a conclusion that means something. SQL, Excel or Python, and one BI tool are enough to get your first role.

Why this is a Mid-level Data Analyst project: Demonstrates end-to-end pipeline ownership, advanced attribution modelling, and connecting analysis directly to a business decision that changed how the team allocates budget.
Mid-level
Marketing Attribution and Budget Optimisation
Markov chain attribution showing true revenue contribution per channel -- used weekly by the growth team to reallocate spend
dbtAzure SynapsePythonPower BI
Total Revenue
R4.2M
Last 90 days
CAC Reduction
-18%
After reallocation
Top Channel
Email
Undervalued by 14pp
ROAS Improvement
1.4x
Last-click vs Markov attribution (%)
Revenue trend -- before vs after reallocation
Budget split -- old vs new (%)
Conversion rate by channel
CAC over time (R)
marketing-attribution/
├── data/
│   ├── raw/
│   │   └── touchpoints_2024.csv
│   └── processed/
│       └── channel_weights.csv
├── notebooks/
│   ├── 01_last_click_baseline.ipynb
│   ├── 02_markov_attribution.ipynb
│   └── 03_validation.ipynb
├── sql/
│   └── channel_touchpoints.sql
├── dbt/
│   └── models/
│       └── fct_attribution.sql
├── reports/
│   └── attribution_methodology_memo.pdf
└── README.md
# Marketing Attribution Rebuild Replaced last-click attribution with a Markov chain model across 5 channels for a South African e-commerce business. Used to reallocate R4.2M in quarterly spend. ## Why Markov Over Last-Click Last-click was overstating paid search contribution by 27 percentage points Markov models the full path -- each channel gets credit proportional to its removal effect Email contribution tripled from 10% to 24% under the new model ## Results CAC reduced from R980 to R800 per customer over 6 months Revenue +26% after budget reallocation (Jan to Jun) ## Stack Python | pandas | dbt | Azure Synapse | Power BI ## Run Order Run notebooks in order: 01 -> 02 -> 03
Slide 01
Problem
Last-click attribution was giving paid search credit it had not earned
58% of conversions attributed to paid search. The actual driver was a multi-touch journey starting with email or organic.
Slide 02
Method
Markov chain model built on 5-channel customer journey data
Modelled removal effect across 5 channels. 3 notebooks: baseline, model, validation.
27pppaid search overstatement
Slide 03
Finding
Email was undervalued by 14 percentage points. Paid search overstated by 27pp
Markov surfaced organic and email as top drivers of assisted conversions -- invisible under last-click.
Slide 04
Outcome
R180 CAC reduction per customer. 22% of paid search budget moved to email
Growth team now runs weekly budget reviews from the Power BI dashboard. Zero manual prep since go-live.
What a Mid-level Data Analyst portfolio must demonstrate

At mid-level the portfolio should move beyond single analyses into recurring systems. Projects should demonstrate automated reporting pipelines, multi-touch attribution, cohort analysis, or A/B test interpretation. Each project should make clear why the analysis was commissioned, what decision it informed, and what changed because of it. dbt, Azure Synapse or SQL Server, and a production Power BI dashboard all belong here. Business context is as important as technical method.

Why this is a Senior Data Analyst project: Demonstrates platform thinking and organisational influence -- a metrics layer serving multiple teams, zero conflicting KPIs across business units, and measurable impact on how the executive team makes decisions.
Senior
Executive Revenue Intelligence Platform
Enterprise metrics layer across 6 business units -- single source of truth for board reporting, forecasting, and strategic planning
dbtAzure SynapsePower BITerraform
Total Revenue YTD
R142M
+11% vs prior year
BUs on Platform
6
Zero conflicting KPIs
Reporting Cycle
4 hrs
Was 3 days
Analyst Onboarding
2 days
Was 15 days
Revenue by BU vs target (R millions)
Monthly revenue -- actual vs forecast
Gross margin by BU (%)
Data quality score by model
Query time improvement (sec)
revenue-intelligence/
├── dbt/
│   ├── models/
│   │   ├── staging/
│   │   └── marts/
│   │       ├── fct_revenue.sql
│   │       ├── dim_customers.sql
│   │       └── fct_orders.sql
│   ├── macros/
│   │   └── revenue_metrics.sql
│   └── tests/
├── dashboards/
│   └── board_template.pbit
├── docs/
│   ├── data_dictionary.md
│   └── stakeholder_guide.md
└── README.md
# Executive Revenue Intelligence Platform Enterprise metrics layer across 6 business units on Azure Synapse + dbt. Single source of truth for board reporting, forecasting, and strategic planning. ## Architecture dbt models: staging layer + 3 marts (revenue, customers, orders) Star schema: 2 fact tables, 4 dimension tables, all tested in CI Query time: 47s to 3.8s on largest joins after partitioning ## Outcomes 6 BUs self-serving in Power BI (was a 3-day manual reporting cycle) Last 3 board decks had zero conflicting numbers across BUs ## Stack dbt | Azure Synapse | Power BI | Terraform | GitHub Actions (CI)
Slide 01
Problem
Six business units reporting different revenue numbers to the same board
EXCO spent the first 30 minutes of every meeting reconciling conflicting figures. Each BU had its own spreadsheet process.
Slide 02
Architecture
dbt star schema on Azure Synapse -- one certified gold layer for all BUs
KPIs defined in code, tested in CI on every commit.
99.1%dbt test pass rate
Slide 03
Impact
Board reporting cycle from 3 days to 4 hours. Analyst onboarding from 15 days to 2
R142M YTD tracked across 6 units with zero discrepancies in the last 3 board cycles.
Slide 04
Governance
KPI definitions live in version-controlled code -- not locked in spreadsheets
CFO office adopted the board template. Strategic decisions now happen in the same meeting as the data review.
What a Senior Data Analyst portfolio must demonstrate

A senior analyst portfolio demonstrates platform thinking and organisational influence. Projects should show a metrics layer serving multiple teams from a single source of truth, governance over how KPIs are defined, and evidence that your work changed how the business makes decisions. Architecture diagrams, stakeholder framing, and measurable outcomes matter more than code. In the South African context, this often means bridging multiple legacy source systems into a single governed layer that feeds Power BI reports used by EXCO and the board. Reviewers are asking whether you can define the data strategy for a function, not just execute it.

Project ideas

Beginner, Excel/Power BI

1. National Budget & Expenditure Analysis

Use National Treasury's budget data to explore government spending by department or province over time. Questions to explore: Which departments saw the biggest increases or cuts? How does spending compare to population size by province?

What "good" looks like: a 1-page dashboard with at least one chart per question, a short written summary of your findings, and a README describing your data source and method.

Beginner, Excel or Python

2. Unemployment Trends

Use Stats SA's Quarterly Labour Force Survey data to explore unemployment trends by province, age group, or education level. Great for showing you can work with official statistics and tell a clear story.

What "good" looks like: a clear trend chart over at least 3 quarters, one insight that isn't obvious from the headline number, and plain-language commentary a non-analyst could understand.

Intermediate, Power BI or Python

3. Inflation Dashboard

Build a Power BI or Python dashboard tracking South African CPI (inflation) over time, broken down by category (food, fuel, housing). Add commentary on what's driving changes.

What "good" looks like: an interactive dashboard (filters by category/date), at least one comparison (e.g. food vs fuel inflation), and a short "what this means for households" write-up.

Intermediate, Python

4. Crime Statistics Analysis

Use SAPS crime statistics to explore trends by province or category over time. Questions to explore: Which categories are rising or falling? How do provinces compare relative to population?

What "good" looks like: a normalised comparison (per 100,000 people, not raw counts), a clear visualisation of the trend, and a discussion of limitations in the data.

Advanced, Python + SQL

5. Municipal Service Delivery Analysis

Analyse open municipal data on service delivery (water, electricity, refuse collection) by ward or area, a project that demonstrates civic awareness and real analytical thinking.

What "good" looks like: data pulled and joined from more than one source, a map or geographic breakdown if possible, and recommendations based on your findings.

South African data sources

vulekamali.gov.za

National Treasury's budget data portal. National and provincial budget and expenditure data, ideal for government spending analysis.

Statistics South Africa

The authoritative source for unemployment, inflation (CPI), population and economic data. Free and regularly updated.

Kaggle

A large library of datasets including many South African and African datasets, plus free notebooks and tutorials to learn from.

SAPS Crime Statistics

SAPS publishes quarterly crime stats by province and category. Municipal open data portals like City of Cape Town cover services and infrastructure.

Why your portfolio matters

For each project, write a short README explaining: the question you asked, where the data came from, what you did, what you found, and what you'd do next. This is often what recruiters actually read first.

Getting hired as a Data Analyst

Skills and projects get you ready. This is how you turn that into interviews and offers.

Your CV

  • Lead with a "Projects" section above "Education" if you have no work experience. Link each project to GitHub.
  • For each project: one line covering the question, tools used, and headline finding.
  • List skills as a short table (SQL, Excel, Power BI, Python) rather than a vague paragraph.
  • Keep it to one page.

Where to apply in South Africa

  • LinkedIn Jobs: search "Data Analyst" + "junior" or "graduate", set location to South Africa, turn on alerts.
  • Indeed and Careers24: two of the largest SA job boards.
  • OfferZen: tech-focused, companies approach you directly.
  • Bank and corporate graduate programmes: FNB, Standard Bank, Absa, Discovery, Nedbank. Search "[bank] graduate programme data".

Interview prep

  • Practice live SQL questions. Most SA analyst interviews include a short SQL test (JOINs, GROUP BY, filtering).
  • DataLemur for free SQL interview practice at real company difficulty levels.
  • Walk through one portfolio project end-to-end: question, data, method, finding, and what you would do differently.
  • Prepare 2-3 questions about how the team uses data day-to-day. Shows genuine interest.
For Data Scientists

Data Scientist Portfolio Projects

These focus on statistics, modelling and communicating uncertainty, not just dashboards. Build in Python (pandas, scikit-learn, statsmodels).

Interactive portfolio examples

Why this is a Junior Data Scientist project: Demonstrates a structured end-to-end approach: problem framing, data exploration, class imbalance handling, model comparison, and translating findings into a business recommendation. The emphasis is on process and communication, not on producing the most complex model.
Junior
Customer Churn Prediction -- Telco Dataset
An end-to-end classification project showing how I explored the data, compared three models, and communicated the findings to a business stakeholder
Pythonpandasscikit-learnXGBoostJupyter
My approach
1
Downloaded Telco dataset (7,043 rows, 21 features) and ran first describe/info pass
2
Spotted class imbalance: 85% stayed, 15% churned -- applied SMOTE before training
3
Explored correlations -- contract type and tenure were the strongest signals by far
4
Tested 3 models: LogReg AUC 0.78, Random Forest 0.85, XGBoost 0.89 -- documented why each was chosen
5
Wrote a 1-page summary translating the model's top predictors into plain language -- which customer types to watch and why
What I learned
Class imbalance matters more than model complexity
A simple written summary of findings landed better than showing charts -- business people want a recommendation, not a confusion matrix
Stakeholders care about which customers to call, not what AUC means
ROC curve -- model comparison (AUC)
Feature importance -- SHAP mean |value|
churn-prediction/
├── data/
│   └── processed/
│       └── telco_clean.csv
├── notebooks/
│   ├── 01_eda.ipynb
│   ├── 02_feature_engineering.ipynb
│   └── 03_model_comparison.ipynb
├── models/
│   └── xgb_churn_v1.pkl
├── reports/
│   └── business_brief.md
└── README.md
# Customer Churn Prediction -- Telco Dataset End-to-end classification project comparing 3 models on the Telco churn dataset. Closes with a 1-page business brief explaining which customers an operator like Vodacom or MTN should prioritise. ## Results XGBoost AUC-ROC: 0.887 vs logistic regression baseline 0.782 Top signals: contract type, tenure, monthly charges Class imbalance handled with SMOTE before training ## Stack Python | pandas | scikit-learn | XGBoost | imbalanced-learn | Jupyter ## Run Order Run notebooks 01 -> 02 -> 03 in sequence Final model saved to models/xgb_churn_v1.pkl
Slide 01
Problem
Telcos lose R800 -- R2,000 per churned customer in re-acquisition costs
Can we identify which customers are likely to leave before they do? This project builds a classifier to flag high-risk accounts early.
Slide 02
EDA Insight
Month-to-month contract customers churn at 5x the rate of 2-year contract holders
Class imbalance: 85% stayed, 15% churned. Applied SMOTE to avoid a model that just predicts "stay" every time.
Slide 03
Model Result
XGBoost AUC 0.887. Logistic regression baseline was 0.782
Top 3 features: contract type, tenure, monthly charges.
0.887XGBoost AUC-ROC
Slide 04
Business Brief
Prioritise outreach to month-to-month customers in months 3 -- 6 of tenure
That window is where the model flags the highest concentration of likely churners. A retention offer in month 4 is cheaper than re-acquiring a lost customer.
What a Junior Data Scientist portfolio must demonstrate

A junior portfolio should contain two to four complete projects. Each should start with a clear problem statement, walk through exploration, cleaning, and feature engineering decisions, and close with business recommendations. A reviewer should be able to understand why modelling decisions were made and what was learned. A logistic regression with honest evaluation beats a neural network with no discussion of limitations. The portfolio should feel like: here is how I approached this problem and what I discovered -- not: here is a completed production system.

Why this is a Mid-level Data Scientist project: Demonstrates end-to-end solution design: problem scoping, systematic experimentation, business-driven model selection, deployment, and automation. The project is not just a notebook -- it runs in production and eliminates manual work.
Mid-level
Retail Demand Forecasting -- 500 SKUs
Weekly automated forecasts with lag feature engineering, MLflow experiment tracking, and ERP integration -- the buying team has had zero manual work since deployment
PythonXGBoostMLflowAirflowpandasSQL
Why this problem matters
The business was sitting on R2.4M in overstock across 500 SKUs. Buyers were placing orders manually using a 12-week moving average that couldn't capture seasonality or promotions.
Why XGBoost over SARIMA
SARIMA couldn't handle 500 parallel series efficiently. XGBoost with lag features gave us MAPE 6.2% vs 11.2% for SARIMA and ran in under 2 minutes per weekly batch.
The automation impact
Forecasts now push directly to the ERP system every Monday at 06:00. The buying team has zero manual work. MLflow tracked 47 experiments to get here.
In [7]: feature_engineering.py
def make_lag_features(df, col, lags=[1,2,4,8,12,52]):
    for lag in lags:
        df[f'{col}_lag{lag}'] = (
            df.groupby('sku_id')[col].shift(lag)
        )
    df[f'{col}_roll4'] = (
        df.groupby('sku_id')[col]
        .transform(lambda x:
            x.shift(1).rolling(4).mean())
    )
    return df

# 47 MLflow experiments tracked
# Best config: MAPE 6.2% (baseline 18.4%)
# Lag-52 feature was most impactful
Forecast vs actual -- Electronics (weekly units)
Model MAPE comparison (%) -- 47 experiments
demand-forecasting/
├── data/
│   └── processed/
│       └── weekly_sales_clean.csv
├── notebooks/
│   ├── 01_eda_seasonality.ipynb
│   ├── 02_sarima_baseline.ipynb
│   └── 03_xgboost_features.ipynb
├── src/
│   ├── feature_store.py
│   └── evaluate.py
├── airflow/
│   └── weekly_forecast_dag.py
├── mlruns/
│   └── (47 MLflow experiment runs)
└── README.md
# Retail Demand Forecasting -- 500 SKUs Weekly automated forecasts for a Gauteng retailer with 500 SKUs. XGBoost with lag and calendar features, tracked across 47 MLflow experiments. Forecasts push to ERP every Monday at 06:00 via Airflow. ## Why XGBoost Over SARIMA SARIMA could not handle 500 parallel series efficiently XGBoost MAPE: 6.2% vs SARIMA 11.2% vs moving average baseline 18.4% Lag-52 (prior year, same week) was the single most impactful feature ## Outcomes R2.4M in overstock freed from working capital Stockout reduction 23%, overstock reduction 18% ## Stack Python | XGBoost | MLflow | Airflow | pandas | SQL
Slide 01
Business Problem
R2.4M in overstock tied up in working capital across 500 SKUs
The buying team was using a 12-week moving average that could not handle seasonality or promotions. Manual process, every week.
Slide 02
Experiments
47 MLflow experiments. 3 model families tested against a moving average baseline
Moving avg: MAPE 18.4%. SARIMA: 11.2%. XGBoost with lag features: 6.2%
Slide 03
Key Finding
Lag-52 (prior year, same week) was the most important single feature
The yearly seasonality signal explained most of the XGBoost accuracy gain over SARIMA.
Slide 04
Deployment
Airflow DAG pushes forecasts to ERP every Monday. Zero manual work since go-live
R2.4M working capital freed. Buying team re-focused on supplier negotiations rather than spreadsheet maintenance.
What a Mid-level Data Scientist portfolio must demonstrate

A mid-level portfolio should evolve from individual notebooks into end-to-end solutions. Projects should demonstrate the ability to compare modelling approaches systematically, track experiments, deploy a model into a real workflow, and connect results to a business objective. Reviewers want to see not only that a model works but why it was chosen and how it is used. Feature stores, orchestration pipelines, and evaluation frameworks that go beyond test-set accuracy all belong here.

Why this is a Senior Data Scientist project: Demonstrates production ML system design, regulatory compliance, champion-challenger governance, drift monitoring, and measurable business impact. The code is secondary -- the decision-making, architecture, and outcomes are primary.
Senior
Credit Risk Scoring -- Production Deployment
Replacing a legacy rules-based scorecard with LightGBM on a R2.8B book -- explainable for compliance, live in production, monitored for drift
PythonLightGBMSHAPMLflowFastAPISQL
Business context
Legacy scorecard approved 18% of customers who defaulted within 12 months. NCR compliance review flagged the model as underpowered for a R2.8B book. Approval volumes were adequate; the model simply couldn't identify bad risk.
Challenge
Improve default identification without reducing approval volumes. Every decision must be auditable under NCR guidelines -- a human-readable explanation is required for every credit outcome affecting a South African consumer.
Outcome
Default rate down 12%. R4.2M saved annually. Compliance approved deployment first attempt because every SHAP explanation is surfaced in the credit officer UI. Live 9 months, zero rollbacks.
Technical decisions
LightGBM selected over XGBoost: faster training on tabular data, native handling of missing values, and better calibration at low score thresholds
SHAP TreeExplainer generates per-customer reason codes in under 8ms -- fast enough for real-time API responses
Champion-challenger split: 90% traffic to champion, 10% to challenger -- automatic promotion when challenger wins on Gini over a 30-day window
PSI monitoring runs nightly -- threshold 0.2 triggers a retraining job automatically
Fairness audit run across age and income cohorts before deployment -- documented and signed off by risk committee
Precision-recall -- new model vs legacy scorecard
PSI drift monitoring -- 12 weeks in production
Stakeholder communication
Executive summary (2 pages) shared with CRO and risk committee before deployment approval
SHAP reason codes surfaced in the credit officer UI -- every declined application has a written reason code meeting NCR requirements
Post-deployment performance report to risk committee every quarter
Model card on SharePoint: assumptions, limitations, training data vintage, monitoring SLAs
R4.2M
saved annually
12%
default rate reduction
9 mo
live with zero rollbacks
0
compliance rejections
credit-risk-model/
├── notebooks/
│   ├── 01_eda_application.ipynb
│   ├── 02_feature_selection.ipynb
│   └── 03_lgbm_training.ipynb
├── src/
│   ├── scorecard.py
│   ├── psi_monitor.py
│   └── shap_report.py
├── api/
│   └── main.py
├── monitoring/
│   └── psi_weekly_report.csv
├── docs/
│   ├── model_card.md
│   └── cro_executive_summary.pdf
└── README.md
# Credit Risk Scoring -- Production Deployment LightGBM replacing a legacy rules-based scorecard on a R2.8B book. Explainable for NCR compliance, champion-challenger governed, PSI-monitored in production for 9 months. ## Key Technical Decisions LightGBM over XGBoost: faster training, better calibration at low score thresholds SHAP TreeExplainer generates reason codes in under 8ms -- fast enough for real-time API Champion-challenger: 90/10 traffic split, auto-promote when challenger wins on Gini over 30 days PSI threshold 0.2 triggers nightly retraining automatically ## Compliance Fairness audit across age and income cohorts signed off by risk committee before go-live Every declined application has a SHAP reason code in the credit officer UI (NCR requirement) ## Outcomes Default rate down 12%. R4.2M saved annually. 9 months live, zero rollbacks
Slide 01
Business Context
Legacy scorecard approved 18% of customers who defaulted within 12 months
NCR compliance review flagged the model as underpowered for a R2.8B book. Better default identification needed without cutting approval volumes.
Slide 02
Architecture
LightGBM + SHAP + FastAPI. Champion-challenger from day one
PSI monitoring runs nightly. Auto-promotion when challenger wins Gini over 30 days.
8msSHAP reason code latency
Slide 03
Compliance
Compliance approved deployment first attempt -- SHAP reason codes in the credit officer UI
Fairness audit across age and income cohorts signed off by risk committee before go-live.
Slide 04
Outcomes
Default rate down 12%. R4.2M saved annually. Nine months with zero rollbacks
Post-deployment performance reported to risk committee quarterly. PSI has stayed below threshold for all 9 months.
What a Senior Data Scientist portfolio must demonstrate

A senior portfolio focuses on impact, not technical complexity. Projects should demonstrate production machine learning systems, model governance, drift monitoring, champion-challenger pipelines, and measurable business outcomes. The most important questions a reviewer asks are: how many customers or rands were affected, what changed after deployment, and how do you know the model is still working? Architecture diagrams, compliance narratives, and business outcome metrics are worth more than code snippets at this level. The portfolio should feel less like a collection of notebooks and more like a collection of case studies.

Project ideas

Beginner-Intermediate, Python

1. Customer Churn Prediction

Use the Kaggle "Telco Customer Churn" dataset to predict which customers are likely to cancel a service. A classic, business-relevant problem that maps directly to SA telcos and banks.

What "good" looks like: compare at least two models (e.g. logistic regression vs random forest), evaluate with precision/recall (not just accuracy), and explain which features drive churn in plain language.

Intermediate, Python

2. House Price Prediction

Use a Kaggle housing-prices dataset to build a regression model that predicts sale price from property features. Adapt the approach later to South African property data as you find suitable sources.

What "good" looks like: feature engineering (not just raw columns), an RMSE or MAE score you can explain, and a discussion of which features matter most and why.

Intermediate, Python + Stats

3. A/B Test Analysis

Use a Kaggle "Marketing A/B Testing" dataset to analyse whether a campaign change actually improved results.

What "good" looks like: a clearly stated hypothesis, the correct statistical test for the data type, a properly interpreted p-value, and a recommendation, this is the skill that separates analysts who can run numbers from scientists who can be trusted with decisions.

South African data sources

vulekamali.gov.za

National Treasury's budget data portal. National and provincial budget and expenditure data, ideal for government spending analysis.

Statistics South Africa

The authoritative source for unemployment, inflation (CPI), population and economic data. Free and regularly updated.

Kaggle

A large library of datasets including many South African and African datasets, plus free notebooks and tutorials to learn from.

SAPS Crime Statistics

SAPS publishes quarterly crime stats by province and category. Municipal open data portals like City of Cape Town cover services and infrastructure.

Tools and platforms

UCI ML Repository

Classic machine learning datasets for practising classification, regression, and clustering.

World Bank Open Data

Global development indicators including South Africa-specific economic and social data.

Hugging Face Datasets

Thousands of ready-to-use datasets, useful once you start working with text or more advanced ML.

SARB Publications

South African Reserve Bank data on monetary policy, interest rates, and banking sector statistics.

Why your portfolio matters

Data Scientist interviews often include a take-home or live modelling exercise. Having 2-3 of these already done, documented, and ready to discuss means you walk in with proof, not promises.

Getting hired as a Data Scientist

Skills and projects get you ready. This is how you turn that into interviews and offers.

Your CV

  • Your Kaggle profile and GitHub are extensions of your CV. Link both prominently.
  • Show model metrics for every project (AUC, F1, RMSE). Never just say "I built a model".
  • If you have Honours or MSc, put it high on the page. SA data science roles often filter on postgrad qualifications.
  • Include a one-line description of your thesis or research focus if it is quantitative or data-related.

Where to apply in South Africa

  • LinkedIn Jobs: search "Data Scientist" + "junior" or "graduate", South Africa. Set alerts.
  • OfferZen: strong presence of SA fintechs and tech companies recruiting scientists.
  • FirstRand Quant and Analytics Graduate Programme on FirstRand Workday.
  • Standard Bank Data Science stream via their graduate programmes page.
  • Bank innovation and data labs: Absa AI, FNB Data Science, Discovery Insights.

Interview prep

  • Know your statistics cold: hypothesis testing, p-values, distributions, and when to use each test.
  • Be ready to explain your model choices and tradeoffs, not just what you ran.
  • Expect a take-home case study: a dataset, a business question, 24-48 hours. Practice on Kaggle.
  • StatQuest for building intuition on every algorithm you list on your CV.
  • Python coding questions are common: pandas manipulation, writing a model from scratch in scikit-learn.
For Data Engineers

Data Engineer Portfolio Projects

These focus on pipelines, schemas and reliability, the things that make data usable for everyone else. Build with Python, SQL and free-tier cloud tools.

Interactive portfolio examples

Why this is a Junior Data Engineer project: Demonstrates a working end-to-end pipeline: extract from an API, validate the schema, load to a database, schedule it, and test it. Reviewers want to see that data arrives correctly and consistently -- not a complex architecture.
Junior
Public API to PostgreSQL ETL Pipeline
Personal pipeline project using the Shopify sandbox API -- built to learn extraction, schema validation and scheduling before entering the industry
PythonPostgreSQLpanderapytestcronREST API
Pipeline architecture
Shopify APIREST v2024-01
Extractrequests + retry
Validatepandera schema
Transformpandas normalise
PostgreSQLorders schema
dbt silvertested models
Metabasefinance reports
extract.py
def fetch_orders(session, since_id=None):
    params = {'status': 'any', 'limit': 250}
    if since_id:
        params['since_id'] = since_id
    for attempt in range(3):
        r = session.get(BASE + '/orders.json',
                        params=params)
        if r.status_code == 200:
            return r.json()['orders']
        time.sleep(2 ** attempt)
    raise RuntimeError('Failed 3 times')
# Daily at 06:00 via cron
# Teams alert if null rate > 1%
# 94% test coverage via pytest
Before vs after
Failures/week3x0
Null rate (order_id)8.3%0.1%
Load time4h manual12 min
Manual hrs/week6 hrs0
Test coverage0%94%
shopify-etl/
├── src/
│   ├── extract.py
│   ├── transform.py
│   └── load.py
├── dbt/
│   └── models/
│       ├── stg_orders.sql
│       └── fct_daily_revenue.sql
├── tests/
│   ├── test_extract.py
│   └── test_transform.py
├── config/
│   └── schema_config.yaml
├── cron/
│   └── schedule.sh
└── README.md
# Shopify API to PostgreSQL ETL Pipeline Personal pipeline project using the Shopify sandbox API. Built to demonstrate production-quality ingestion before entering the data engineering industry. ## Pipeline Flow Extract from Shopify REST API v2024-01 with retry logic (3 attempts, exponential backoff) Validate schema with pandera before load -- null rate dropped 8.3% to 0.1% Load to PostgreSQL via psycopg2, upsert on order_id dbt silver layer runs quality tests after every load Scheduled via cron at 06:00 daily, Teams alert if null rate exceeds 1% ## Test Coverage 94% via pytest -- extract, transform, and load layers all covered ## Stack Python | requests | pandera | psycopg2 | dbt | pytest | cron
Slide 01
Design Goal
Build a production-quality ETL pipeline before getting a first job in data engineering
Most portfolios have notebooks. This one has a scheduled pipeline with retry logic, schema validation, and 94% test coverage.
Slide 02
Architecture
Extract, validate, transform, load -- 7 steps from Shopify API to Metabase
Pandera catches bad data before the database. dbt tests verify quality after every load.
94%pytest coverage
Slide 03
Key Decisions
Retry logic on extract. Upsert on load. Teams alert if null rate exceeds 1%
3 attempts with exponential backoff. Idempotent upsert means re-runs never create duplicates.
Slide 04
Results
Zero failures since schema validation was added. Load time from 4 hrs to 12 mins
Null rate on order_id: 8.3% to 0.1%. Manual hours per week: 6 to 0.
What a Junior Data Engineer portfolio must demonstrate

A junior portfolio should contain two to four pipeline projects. Each should demonstrate a clear data flow from source to destination, working extraction and loading code, schema validation, automated scheduling, and basic error handling. Reviewers want to see that data arrives correctly and consistently and that failures are caught. A pipeline that runs cleanly on a schedule and sends an alert when something breaks is worth more than a complex system held together with manual steps.

Why this is a Mid-level Data Engineer project: Demonstrates cloud platform ownership, infrastructure as code, modular dbt transformation layers, and the ability to explain architectural decisions in business terms -- not just what was built but why each layer exists and what it costs.
Mid-level
Cloud Data Lakehouse -- Medallion Architecture on Azure
8 source systems unified in one gold layer -- all infrastructure as Terraform, all transformations as tested dbt models, compute cost down 62%
dbtAzure Data LakeAzure SynapseAirflowTerraformGreat Expectations
Medallion architecture -- why each layer exists
8 SourcesCRM, ERP, SAP...
ADLS Bronzeraw, immutable
Spark / dbtsilver -- cleaned
Synapse Goldfact + dim models
Great Expectsquality gates
Power BIanalysts query here
fct_orders.sql (dbt)
-- Incremental: only process new rows
-- Partitioned for query speed + cost
{{
  config(
    materialized='incremental',
    unique_key='order_id',
    partition_by={'field': 'order_date'},
    cluster_by=['customer_id']
  )
}}
SELECT
    o.order_id, o.customer_id,
    c.segment, o.order_date,
    SUM(oi.qty * oi.price) AS gross_rev
FROM {{ ref('stg_orders') }}    o
JOIN {{ ref('dim_customers') }} c USING(customer_id)
JOIN {{ ref('stg_items') }}     i USING(order_id)
{% if is_incremental() %}
WHERE o.order_date > (
  SELECT MAX(order_date) FROM {{ this }})
{% endif %}
GROUP BY 1,2,3,4
Before vs after
Avg query time47s3.8s
Source onboarding3 weeks2 days
Compute/monthR48kR18k
dbt test pass rate--99.1%
Sources unified8 silos1 gold layer
data-lakehouse/
├── terraform/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
├── dbt/
│   ├── models/
│   │   ├── staging/
│   │   └── marts/
│   └── tests/
├── airflow/
│   └── dags/
│       └── lakehouse_dag.py
├── great_expectations/
│   └── checkpoints/
├── docs/
│   ├── ADR_001_medallion.md
│   └── cost_analysis.md
└── README.md
# Cloud Data Lakehouse -- Medallion Architecture on Azure 8 source systems unified into one gold layer. All infrastructure as Terraform, all transformations as tested dbt models. Compute cost down 62% (R48k to R18k/month). ## Why Medallion Bronze: raw immutable copy -- replay any source from scratch if needed Silver: cleaned, typed, deduplicated -- built on Spark/dbt Gold: fact + dim models, partitioned and clustered for query speed ## Results Query time: 47s to 3.8s. Compute cost: R48k to R18k/month New source onboarding: 3 weeks to 2 days. dbt test pass rate: 99.1% ## Stack dbt | Azure Data Lake | Azure Synapse | Airflow | Terraform | Great Expectations ## Docs Architecture decisions in docs/ADR_001_medallion.md
Slide 01
Problem
8 source systems, 8 data models, zero consistency. Analysts raised ETL tickets to get any data
Query times were 47 seconds on key joins. Power BI reports broke weekly.
Slide 02
Architecture
Medallion: Bronze (raw), Silver (clean), Gold (marts) -- all in Azure, all as code
Terraform manages all infrastructure. dbt models are version-controlled and tested in CI.
99.1%dbt test pass rate
Slide 03
Results
Query time 47s to 3.8s. Compute cost down 62%. New source onboarding 3 weeks to 2 days
All analysts self-serve from Power BI. Zero ETL tickets raised since go-live.
Slide 04
Decision Record
Architecture decision records explain every design choice -- not just what was built, but why
Cost analysis showed partitioning + incremental dbt models saved R30k/month vs full refresh on Synapse.
What a Mid-level Data Engineer portfolio must demonstrate

A mid-level portfolio demonstrates ownership of a complete data platform. Projects should show cloud infrastructure, orchestration, modular transformation layers, and data quality frameworks. Business context matters -- reviewers want to understand why the architecture was chosen. Cost comparisons, trade-off explanations, and before-and-after metrics tell a stronger story than a diagram alone.

Why this is a Senior Data Engineer project: Demonstrates platform thinking and organisational impact: a streaming system at scale, infrastructure as code, exactly-once guarantees, and enabling four downstream teams to build real-time products without needing to understand Kafka or Flink.
Senior
Real-Time Event Streaming Platform -- 40k Events/sec
Kafka + Flink + Delta Lake replacing overnight batch -- risk alerts under 3 seconds, exactly-once semantics, zero data loss in 9 months
KafkaApache FlinkDelta LakeKubernetesTerraformPrometheus
Business context
Risk alerts were running overnight in a batch job. By the time fraud was flagged, transactions had cleared. The business was absorbing losses that could have been stopped in seconds.
Challenge
Build a real-time event platform that 4 downstream teams can build on without needing to understand Kafka or Flink. Exactly-once delivery is non-negotiable -- a missed risk event costs money.
Outcome
Risk alerts now fire in under 3 seconds. Zero data loss in 9 months. Throughput up from 5k to 40k events/sec. Four teams shipped real-time products on top of the platform.
Technical decisions
Kafka over Kinesis: replayability and lower cost at our event volume -- 3x replication for fault tolerance
Flink with exactly-once checkpointing to Delta Lake -- tested with chaos engineering before go-live
5-minute tumbling windows for risk aggregation: balances latency against false positive rate
Kubernetes + Terraform: any engineer can spin up a new Flink job in 30 minutes using the team template
Prometheus + Grafana dashboards per downstream team -- each team owns its own alerting thresholds
Alert latency (ms) -- weekly avg
Throughput growth (k events/sec)
Stakeholder communication
Executive brief to CTO: why real-time vs overnight batch, cost model, migration risk
Platform onboarding guide on SharePoint -- used by all 4 downstream teams
Quarterly reliability report posted to Teams: SLA adherence, data loss events, incident log
Architecture decision records on SharePoint for every major design choice
Chaos engineering results shared with risk and compliance via Teams before go-live
<3s
alert latency
40k
events/sec peak
0
data loss events
4
teams enabled
streaming-platform/
├── kafka/
│   ├── topics.yaml
│   └── connect_config.yaml
├── flink/
│   ├── risk_alerts_job.py
│   └── analytics_job.py
├── infra/
│   ├── terraform/
│   └── kubernetes/
│       └── flink_deployment.yaml
├── monitoring/
│   └── grafana_dashboards/
├── docs/
│   ├── ADR_kafka_vs_kinesis.md
│   └── chaos_engineering_results.md
└── README.md
# Real-Time Event Streaming Platform -- 40k Events/sec Kafka + Flink + Delta Lake replacing overnight batch. Risk alerts under 3 seconds, exactly-once semantics, zero data loss in 9 months. Enables 4 downstream teams to ship real-time products. ## Key Decisions Kafka over Kinesis: replayability and lower cost at our event volume (3x replication) Flink exactly-once checkpointing to Delta Lake -- chaos-tested before go-live 5-minute tumbling windows for risk aggregation: balances latency vs false positive rate Kubernetes + Terraform: any engineer spins up a new Flink job in 30 minutes ## Outcomes Risk alerts: overnight to under 3 seconds. Throughput: 5k to 40k events/sec Zero data loss in 9 months. 4 teams shipped real-time products on the platform ## Stack Kafka | Apache Flink | Delta Lake | Kubernetes | Terraform | Prometheus | Grafana
Slide 01
Problem
Risk alerts ran overnight. By the time fraud was flagged, transactions had cleared
The business was absorbing losses that could have been stopped in seconds.
Slide 02
Architecture
Kafka + Flink + Delta Lake -- exactly-once, Kubernetes-managed, chaos-tested
Any engineer spins up a new Flink job in 30 minutes using the team template.
<3srisk alert latency
Slide 03
Platform Impact
Throughput from 5k to 40k events/sec. Four teams shipped real-time products on this platform
Zero data loss in 9 months. Quarterly reliability report posted to Teams.
Slide 04
CTO Brief
Why real-time vs overnight batch -- cost model, migration risk, team enablement strategy
Architecture decision record explains Kafka vs Kinesis trade-off. Chaos engineering results shared with risk and compliance before go-live.
What a Senior Data Engineer portfolio must demonstrate

A senior portfolio centres on platform thinking and organisational impact. Projects should demonstrate real-time systems, infrastructure as code, and the ability to enable other teams to build safely on top of your platform. Cost analysis, architectural trade-offs, SLAs, and evidence of sustained production reliability matter more than code volume.

Project ideas

Beginner-Intermediate, Python + SQL

1. Load-Shedding Data Pipeline

Pull data from the EskomSePush API on a schedule (cron job or simple Python script) and store outage history in a database. Build a small dashboard showing patterns by area or time of day.

What "good" looks like: a script that runs on a schedule without manual intervention, a clean table schema, and basic checks for missing or duplicate data.

Intermediate, SQL + dbt

2. Retail Sales Data Warehouse

Take a Kaggle retail sales dataset, design a simple star schema (fact table + dimension tables), load it into PostgreSQL or BigQuery, and write dbt models to transform raw data into reporting-ready tables.

What "good" looks like: a documented schema diagram, version-controlled SQL/dbt code, and at least one data quality test (e.g. no nulls in key fields).

Advanced, Python + Airflow

3. End-to-End ETL with Airflow

Orchestrate a multi-step pipeline (extract from an API or CSV, transform with Python/pandas, load into a database) using Apache Airflow, run locally with Docker or on Astronomer's free trial.

What "good" looks like: a visible DAG (pipeline diagram), logging and error handling for failed steps, and a README explaining how to run it from scratch.

South African data sources

vulekamali.gov.za

National Treasury's budget data portal. National and provincial budget and expenditure data, ideal for government spending analysis.

Statistics South Africa

The authoritative source for unemployment, inflation (CPI), population and economic data. Free and regularly updated.

Kaggle

A large library of datasets including many South African and African datasets, plus free notebooks and tutorials to learn from.

SAPS Crime Statistics

SAPS publishes quarterly crime stats by province and category. Municipal open data portals like City of Cape Town cover services and infrastructure.

Tools and platforms

EskomSePush API

Real-time and historical load-shedding data. One of the most SA-relevant APIs for pipeline projects.

ExchangeRate-API

ZAR exchange rate data. Useful for financial data pipelines and currency tracking projects.

Supabase

Free-tier PostgreSQL database. Good for building a real data warehouse without cloud costs.

dbt

The standard transformation tool in modern data stacks. Free to use locally and widely used in SA.

Why your portfolio matters

Data Engineer interviews rarely ask you to "analyse" data, they want to see that you can move it reliably. A working pipeline, even a small one, demonstrates exactly that.

Getting hired as a Data Engineer

Skills and projects get you ready. This is how you turn that into interviews and offers.

Your CV

  • GitHub with working pipelines is non-negotiable. A README showing how to run it locally is a strong signal.
  • List tools explicitly: dbt, Airflow, Spark, Snowflake, Kafka, Docker. Recruiters search for these.
  • Cloud certifications add real credibility: AZ-900 or AWS Cloud Practitioner first, then DP-203 (Azure Data Engineer) or AWS Data Analytics.
  • Show pipeline architecture diagrams in your README. A simple diagram explains more than a paragraph.

Where to apply in South Africa

  • LinkedIn Jobs: search "Data Engineer" + "junior" or "graduate", South Africa.
  • OfferZen: strong for SA fintechs (PayJustNow, TymeBank, Jumo) which are active data engineering hirers.
  • Cloud consulting firms: DVT, BBD, Synthesis, BCX. They place junior engineers and pay for certifications.
  • Large bank data platform teams: FNB, Standard Bank and Absa all have dedicated data engineering divisions.
  • Careers24: search "Analytics Engineer" or "ETL Developer" for entry-level data engineering roles.

Interview prep

  • Expect a SQL and Python coding test. Practice writing pipeline logic in Python and complex SQL (window functions, CTEs).
  • Data modeling questions are common: explain the difference between star schema and snowflake schema, what SCD Type 2 is.
  • Be ready for pipeline design scenarios: "How would you build a pipeline that ingests X and loads it into Y every hour?"
  • Know basic cloud concepts for your target stack. If the job mentions Azure, know ADF, Synapse, and Databricks at a surface level.
  • Walk through your pipeline project: what would break at scale, and how would you handle it?
For AI/ML Engineers

AI/ML Engineer Portfolio Projects

These focus on getting models into something usable, a deployed demo beats a notebook every time.

Interactive portfolio examples

Why this is a Junior AI/ML Engineer project: Demonstrates the gap between a notebook model and a deployed service -- wrapping, containerisation, testing, load benchmarking, and a model card. The emphasis is on engineering discipline, not research novelty.
Junior
Sentiment Classification API
Taking a pre-trained DistilBERT model and making it production-ready -- Dockerised, tested, load-benchmarked, with a model card and a real deployment
PythonHuggingFaceFastAPIDockerpytestLocust
How I built this
1
Chose DistilBERT: 40% smaller than BERT with only 3% accuracy loss -- justified the trade-off for a latency-sensitive API
2
Fine-tuned on 8,000 labelled reviews -- 5 epochs, batch 16, LR 2e-5 -- tracked overfitting via train vs validation loss each epoch
3
Wrapped in FastAPI, containerised with Docker, ran Locust load tests to benchmark latency at 100 concurrent users
4
Wrote a model card: training data, known limitations, and what the model should NOT be used for
5
Deployed to a free-tier cloud VM and shared the live endpoint with classmates and my bootcamp facilitator for feedback
What I learned
Engineering discipline matters as much as accuracy -- tests and latency benchmarks are part of the work
A model card forces you to think about failure modes you'd otherwise ignore
The support team cared about speed and reliability -- not F1 score
Training -- 5 epochs
Training vs validation loss
Live API demo
POST /predict {"text": "Delivery quick but packaging damaged"} 200 OK -- 38ms { "label": "mixed", "confidence": 0.74, "latency_ms": 38 }
sentiment-api/
├── model/
│   └── distilbert_finetuned/
├── api/
│   ├── main.py
│   └── schemas.py
├── tests/
│   ├── test_api.py
│   └── locust_load_test.py
├── Dockerfile
├── model_card.md
└── README.md
# Sentiment Classification API DistilBERT fine-tuned on South African product reviews, containerised as a FastAPI service. Built to demonstrate the full path from fine-tuning to production deployment. ## Results F1: 0.85 (TextBlob baseline: 0.61) API latency at p99: 38ms under load Locust load test: 200 concurrent users, zero errors ## model_card.md covers Training data: 12k labelled South African product reviews Known limitation: code-switching (Zulu/English) reduces F1 to 0.71 Intended use: e-commerce review monitoring for SA market ## Stack Python | HuggingFace | FastAPI | Docker | pytest | Locust
Slide 01
Design Goal
Take a pre-trained model and make it production-ready -- not just a notebook
Most junior AI/ML portfolios show fine-tuning. This one shows wrapping, containerisation, load testing, and a model card.
Slide 02
Architecture
DistilBERT fine-tuned -- FastAPI -- Docker -- load tested with Locust
FastAPI for the REST wrapper. Docker for portability. Locust to verify it holds under 200 concurrent users.
38msp99 latency under load
Slide 03
Model Result
F1 from 0.61 (TextBlob) to 0.85 after fine-tuning on SA review data
Known limitation documented in model card: code-switching drops F1 to 0.71. Honest about what the model cannot do.
Slide 04
Model Card
Every deployed model needs a model card -- inputs, outputs, limitations, intended use
This is what separates an engineer who deploys models from one who just trains them.
What a Junior AI/ML Engineer portfolio must demonstrate

A junior portfolio should demonstrate the ability to take an existing model and make it production-ready. Projects should include wrapping in a REST API, containerisation, unit and integration tests, latency benchmarks under load, and a model card documenting inputs, outputs, and limitations. The emphasis is on engineering discipline. A reviewer should see that you understand the gap between a notebook and a deployed service.

Why this is a Mid-level AI/ML Engineer project: Demonstrates end-to-end ML system design with measurable evaluation -- not just a RAG demo, but RAGAS scoring, guardrails for compliance, and a documented rationale for every design decision.
Mid-level
RAG System for HR Policy Q&A
Retrieval-Augmented Generation over 5,000 policy documents -- hybrid search, RAGAS evaluation, guardrails, deployed on Kubernetes
PythonLangChainChromaDBAzure OpenAIFastAPIDocker
Why RAG over fine-tuning
Policy docs update frequently. Fine-tuning would need retraining on every change. RAG retrieves from the latest documents at inference time -- no retraining needed.
Why hybrid retrieval
Dense-only retrieval missed exact policy numbers and clause refs. Adding BM25 as a sparse layer and re-ranking improved context recall from 0.48 to 0.79.
POPIA and audit requirements
No employee data leaves SA borders -- Azure OpenAI with SA data residency. Every answer cites its source. Guardrails refuse out-of-scope queries (refusal rate 0.98) rather than hallucinating an answer.
In [3]: retrieval_pipeline.py
def hybrid_retrieve(query, k=5):
    dense = chroma.similarity_search(
        query, k=k*2)
    sparse = bm25.get_top_n(
        query.split(), corpus, n=k*2)
    combined = reciprocal_rank_fusion(
        dense, sparse)
    reranked = cross_encoder.predict(
        [(query, d.page_content)
         for d in combined])
    return [combined[i]
            for i in
            np.argsort(reranked)[::-1][:k]]
# Context recall:  0.48 -> 0.79
# Faithfulness:    0.61 -> 0.89
RAGAS evaluation -- naive RAG vs this system
Context recall by policy category
hr-policy-rag/
├── ingestion/
│   ├── chunker.py
│   └── embed_and_store.py
├── retrieval/
│   ├── hybrid_retrieve.py
│   └── reranker.py
├── generation/
│   ├── prompt_templates.py
│   └── guardrails.py
├── evaluation/
│   └── ragas_eval.py
├── api/
│   └── main.py
└── README.md
# RAG System for HR Policy Q&A Retrieval-Augmented Generation over 5,000 HR policy documents. Hybrid search, RAGAS evaluation, POPIA-compliant (SA data residency), guardrails refusing out-of-scope queries. ## Why RAG Over Fine-Tuning Policy docs update frequently -- RAG retrieves from latest docs at inference time with no retraining Dense-only retrieval missed exact policy numbers. BM25 sparse layer fixed this ## RAGAS Results Faithfulness: 0.61 -> 0.89 (naive RAG to this system) Context recall: 0.48 -> 0.79 Out-of-scope refusal rate: 0.98 ## Compliance Azure OpenAI with SA data residency -- no employee data leaves SA borders (POPIA) Every answer cites its source document. Compliance can audit any answer ## Stack Python | LangChain | ChromaDB | Azure OpenAI | FastAPI | Docker | RAGAS
Slide 01
Problem
HR was answering 200+ policy queries per month manually -- same questions, different answers each time
Inconsistent responses created compliance risk. The team needed a system giving auditable, consistent answers grounded in actual policy documents.
Slide 02
Architecture
Hybrid retrieval: dense (ChromaDB) + sparse (BM25) + cross-encoder re-ranking
POPIA compliance: Azure OpenAI with SA data residency. Guardrails refuse out-of-scope queries rather than hallucinating.
0.98out-of-scope refusal rate
Slide 03
Evaluation
RAGAS faithfulness 0.61 to 0.89. Context recall 0.48 to 0.79 vs naive RAG
Every metric measured before and after system improvements. Reviewers can see exactly what changed and why.
Slide 04
Outcome
HR query resolution time down 60%. Compliance can audit every answer back to its source
System handles routine policy queries 24/7. HR team re-focused on complex cases that need human judgment.
What a Mid-level AI/ML Engineer portfolio must demonstrate

A mid-level portfolio should demonstrate end-to-end ML system design: experiment tracking, structured evaluation, RAG pipelines, fine-tuning workflows, or LLM development with guardrails. The portfolio should show that you can build systems that are measurable and improvable. RAGAS scores, latency benchmarks, and evaluation results tell a stronger story than a single accuracy figure. Reviewers want to see that you thought about failure modes -- hallucinations, out-of-scope queries, and edge cases.

Why this is a Senior AI/ML Engineer project: Demonstrates platform and organisational impact: infrastructure enabling 7 teams to deploy, monitor, and iterate on models independently -- with drift detection, canary rollouts, and a measurable reduction in deployment time.
Senior
ML Platform -- Model Lifecycle and Deployment
Internal MLOps platform enabling 7 teams to ship models independently -- model registry, canary rollout, auto-retrain on drift
MLflowKubeflowGitHub ActionsKubernetesPrometheusGrafana
Business context
7 data science teams were deploying models by emailing Python scripts to DevOps. Deployments took 3 weeks, models had no monitoring, and nobody knew when a model had drifted in production.
Challenge
Build a platform that lets each team own its model deployments end to end without DevOps involvement. Canary rollouts and automated drift monitoring are non-negotiable for production safety.
Outcome
Deployment time down from 3 weeks to 2 days. Platform caught 14 drift events and retrained automatically. Zero production incidents since canary rollout was introduced.
Technical decisions
MLflow self-hosted so model artefacts stay on-premises -- required for regulatory compliance
Canary rollout: 5% traffic for 7 days, auto-promote if error rate and latency SLAs hold
PSI threshold 0.2 triggers a Kubeflow retraining job automatically -- no human required for routine drift
GitHub Actions CI: training + unit tests + eval thresholds all must pass before registry promotion
Grafana dashboards per team -- each team owns its own alerting thresholds and SLA definitions
Deploy time to production (days)
Auto-retrains triggered by drift
Stakeholder communication
CTO brief: why an MLOps platform vs continuing ad-hoc deployments, with ROI estimate
Team onboarding guide on SharePoint: first commit to production, used by all 7 teams
Quarterly platform health report posted to Teams channel: deployment frequency, drift events, SLA adherence
Model governance policy co-written with compliance: who approves a model going live
Teams call to all 7 teams explaining drift detection and how to read PSI alerts -- recording pinned in the data channel
2 days
deploy time (was 3 wks)
14
drift events caught
7
teams on platform
0
production incidents
ml-platform/
├── registry/
│   └── mlflow_config.py
├── pipelines/
│   └── kubeflow/
│       └── retrain_pipeline.yaml
├── monitoring/
│   ├── drift_detector.py
│   └── psi_alert.py
├── ci/
│   └── .github/
│       └── workflows/
│           └── model_ci.yaml
├── docs/
│   ├── model_governance_policy.md
│   └── team_onboarding_guide.md
└── README.md
# ML Platform -- Model Lifecycle and Deployment Internal MLOps platform enabling 7 data science teams to deploy, monitor, and iterate on models independently. MLflow registry, Kubeflow pipelines, canary rollouts, automated drift detection. ## Key Decisions MLflow self-hosted on-prem: model artefacts stay on-premises for regulatory compliance Canary rollout: 5% traffic for 7 days, auto-promote if error rate and latency SLAs hold PSI threshold 0.2 triggers Kubeflow retraining automatically -- no human needed for routine drift GitHub Actions CI: training + unit tests + eval thresholds must all pass before registry promotion ## Outcomes Deployment time: 3 weeks to 2 days. 14 drift events caught automatically 7 teams on platform. Zero production incidents since canary rollout was introduced ## Stack MLflow | Kubeflow | GitHub Actions | Kubernetes | Prometheus | Grafana
Slide 01
Problem
7 data science teams were deploying models by emailing Python scripts to DevOps
Deployments took 3 weeks. Models had no monitoring. Nobody knew when a model had drifted in production.
Slide 02
Platform
MLflow registry + Kubeflow pipelines + canary rollouts + PSI drift monitoring
CI: training, unit tests, and eval thresholds must all pass before a model reaches the registry.
2 daysdeployment time (was 3 weeks)
Slide 03
Impact
14 drift events caught and retrained automatically. Zero production incidents since canary rollout
7 teams now own their deployments end to end. Platform health report posted to Teams every quarter.
Slide 04
Governance
Model governance policy co-written with compliance -- who approves a model going live
Onboarding guide used by all 7 teams: first commit to production in one document.
What a Senior AI/ML Engineer portfolio must demonstrate

A senior portfolio demonstrates platform and organisational impact -- infrastructure enabling teams to deploy, monitor, and iterate on models independently. Drift detection, canary rollouts, and model registries are expected. The portfolio should show that models are alive in production, not just deployed once and forgotten. Reviewers are asking whether you can design the ML platform for an organisation and set the standards for model governance.

Project ideas

Beginner-Intermediate, Python

1. Deploy a Model as an API

Train a simple classifier (e.g. on a Kaggle dataset), wrap it in a FastAPI app, and deploy it for free on Render or as a Hugging Face Space.

What "good" looks like: a live demo link that actually works, basic input validation, and a short write-up of the model's accuracy and limitations.

Intermediate, Python + Deep Learning

2. Image Classifier with Transfer Learning

Fine-tune a pretrained model (e.g. ResNet or a small vision transformer) on a small custom or Kaggle image dataset.

What "good" looks like: clear before/after accuracy reporting, example correct and incorrect predictions, and an honest discussion of where the model fails.

Advanced, Python + LLMs

3. RAG Chatbot Over a Small Dataset

Build a retrieval-augmented chatbot over a focused dataset, for example, SA university admission requirements or bursary information, using an open-source model via Hugging Face.

What "good" looks like: a working demo, a clear explanation of the architecture (retrieval + generation), and an honest section on limitations and hallucination risks.

South African data sources

vulekamali.gov.za

National Treasury's budget data portal. National and provincial budget and expenditure data, ideal for government spending analysis.

Statistics South Africa

The authoritative source for unemployment, inflation (CPI), population and economic data. Free and regularly updated.

Kaggle

A large library of datasets including many South African and African datasets, plus free notebooks and tutorials to learn from.

SAPS Crime Statistics

SAPS publishes quarterly crime stats by province and category. Municipal open data portals like City of Cape Town cover services and infrastructure.

Tools and platforms

Hugging Face

Models, datasets, and deployment tools. The default starting point for most ML and NLP projects.

Render

Free tier hosting for deploying FastAPI or Flask model endpoints. No credit card required to start.

Papers with Code

Research papers with implementation code. Useful for understanding SOTA methods and benchmarks.

Masakhane

African NLP research community with datasets covering South African languages including isiZulu and Sesotho.

Why your portfolio matters

Almost nobody starts their career as an AI/ML Engineer. These projects show you can take a model from "it works on my machine" to "someone else can use it", which is exactly the gap most junior candidates miss.

Getting hired as a AI / ML Engineer

Skills and projects get you ready. This is how you turn that into interviews and offers.

Your CV

  • A deployed model is worth more than 10 notebooks. Even a simple FastAPI endpoint on Render or Hugging Face Spaces counts.
  • GitHub and Hugging Face profile are your portfolio. Link them before LinkedIn.
  • Kaggle competition placements (even top 30%) show you can compete. Include your best result.
  • If you have research publications or a thesis on ML/AI, list them with a one-line summary.
  • Show the full stack: model training, API serving, and monitoring. Even a basic MLflow experiment log helps.

Where to apply in South Africa

  • LinkedIn Jobs: search "ML Engineer", "AI Engineer", "NLP Engineer", "Computer Vision" in South Africa.
  • OfferZen: growing number of SA AI-focused companies recruiting here.
  • AI and ML research roles: CSIR (Council for Scientific and Industrial Research), UCT AISB, Wits School of AI.
  • Large tech SA offices: Amazon, Google and Microsoft all have SA engineering teams with ML roles.
  • Fintech and insurtech companies building credit or fraud models: PayJustNow, Jumo, OUTsurance data science teams.

Interview prep

  • Expect a Python ML coding test. Be comfortable implementing models and pipelines from scratch, not just calling sklearn.fit().
  • ML system design questions: "Design a real-time fraud detection system for a payment API."
  • Deep learning concepts: be ready to explain backpropagation, optimizers, regularisation, and transformer attention.
  • Model deployment and monitoring: how do you detect data drift, how do you version models, what does your CI/CD for ML look like?
  • fast.ai and Hugging Face courses are your best prep resources.

Where to Showcase Your Portfolio

GitHub

The standard place to host your code, notebooks and READMEs. Recruiters and hiring managers expect to find a GitHub link on your CV or LinkedIn.

Notion

A great way to build a visual "portfolio page" that links to your projects, with summaries and screenshots, easy to set up and share as a link.

Personal Website

A simple personal site (even a single page) ties everything together, your story, your projects, and your contact details, all in one place.

What Hiring Managers Actually Look For

  • Can you clearly explain your thought process, not just your code?
  • Did you ask a meaningful question, or just run a tutorial?
  • Is your work clean, documented and easy to follow?
  • Do you show curiosity, did you dig deeper than the obvious question?
  • Can you communicate findings to a non-technical audience?

Your Portfolio is Your Proof, Build It Before You Need It

Start with one project this week. Don't wait until you "feel ready", you'll learn more by doing than by preparing.

Explore Opportunities