I built a reporting system for my language-learning app oqou. Every morning it pulls data from four sources, cleans it, loads it into a SQL database and updates a dashboard. I use it to track growth, find where users drop off, test ideas and decide what to work on next.
SQL (SQLite)ETL / daily data pipelineKPI definitions & targetsFunnel & cohort analysisSegmentationA/B test designHypothesis testingData quality checksDashboardingGoogle Analytics 4 APIRevenueCat APIApp Store ConnectNode.js · React
ⓘDemo data. All numbers on this page are made up. The app's real figures stay private. The page structure, metric definitions and checks match the real dashboard I use every day.
The app's data was spread across four tools, and they did not agree with each other.
Scattered sources
Downloads were in App Store Connect, usage in Google Analytics 4, subscriptions in RevenueCat and website search in Google Search Console. Comparing anything meant opening four tabs.
Numbers that disagreed
Each source counts a "new user" differently, uses different time zones and finalises data with a different delay. My own test installs were also counted as real users.
Questions without answers
Do people who start a trial come back? Which onboarding step loses the most users? Do social media posts bring downloads? No single tool could answer these.
What I built
A daily ETL job feeds a SQL data model. One metric catalog defines every KPI, and the dashboard and the morning report both read from it.
The dashboard
The headline KPIs, each compared with the previous period. A rate always shows its numerator and denominator, so "100%" can never hide "1 of 1". Dates are synthetic.
Daily downloads and active users
The last two days are dashed: they are provisional and can still change. Orange dots mark days with a social media post.
KPI scoreboard: tracking an improvement plan
Before changing anything in the app, I measured a baseline and set a target for each KPI. The plan ran in phases, and each KPI belongs to the phase meant to move it. The scoreboard shows, every day, how far each KPI has moved from baseline toward its target.
Scoreboard
Hit target reachedOn track 50%+ of the wayMoving some progressFlat no real change
Progress = (current − baseline) / (target − baseline), with the direction taken into account, so "fewer errors" and "more readers" score the same way. A KPI that could not be measured at baseline shows a dash there instead of a zero.
One KPI over time: day-1 return rate
Weekly value against baseline and target. The dotted lines mark when each phase shipped.
How the KPIs were chosen
One KPI tree, not a list: stability and first-session reading feed day-1 return, which feeds trial starts and payment. A KPI lower in the tree is not judged until the ones above it move.
Leading before lagging: "subscribers who reopen the app" was added because payments lag by weeks. It shows early whether a change is working.
Every KPI has an owner phase: a paywall change is judged on paywall-to-trial, not on retention, so one change cannot take credit for another.
Snapshots, not just "now": each run is saved with its date, so the plan has a before-and-after record instead of a dashboard that only shows today.
Funnel and cohort analysis
Where new users drop off, and how many come back. Funnel steps are ordered: a user counts at a step only after completing the steps before it.
New-user funnel
Users who installed in the selected range. The arrow shows how many reached the step from the step before.
Retention by weekly install cohort
Share of a cohort active again on day 1, 7 and 30. A dash means the day has not happened yet, not zero.
Analyses and insights
Nine questions I answered with this data: what I asked, how I tested it, what came out and what I changed. The findings are real; every number and chart uses demo data.
01 · Cohort analysis · Segment comparison
Retention, not the paywall, limited sales
Question
Few trials turned into paid subscriptions. Was the paywall the problem?
Method
Split new users into those who came back on a later day and those who used the app on one day only, then compared trial starts and payments between the two groups.
Finding
The first session worked well, but most users never came back after day 1. Users who returned started trials several times more often, and almost every paying user was a returner. A paywall change moves how many trials start; retention moves how many people buy.
Action
The roadmap moved from paywall design to day-2 retention. The retention cohorts track whether it works.
02 · Hypothesis testing · Funnel
Notifications were not the bottleneck, the reminder rules were
Question
Do users not come back because they refuse notifications?
Method
Followed the path from permission to reminder: who granted permission, who qualified for any reminder, who opened one. Then checked the rules that decide who gets a reminder.
Finding
Hypothesis rejected: most users granted permission. But the reminder rules only covered users who had saved a word, and most users never save one, so the majority never received a reminder at all.
Action
A day-2 reminder for every new user, instead of moving the permission prompt, which would have fixed the wrong thing.
03 · Segmentation · Confidence intervals
Which users start a trial? Segmenting by survey answer
Question
Do users with different goals behave differently?
Method
Joined the onboarding survey answer ("why are you learning?") to each user's later behaviour and compared engagement and trial rate per segment, with 95% confidence intervals (Wilson) because the groups are small.
Finding
People learning for travel had more sessions and the highest trial rate. The largest group, people learning for work, had the lowest. The intervals overlap, so this is a lead to follow, not a proven difference.
Action
Marketing copy and stories for travel learners are the next test; the segment view keeps running so the intervals narrow as data grows.
04 · Statistics · Anomaly detection
A record day that wasn't
Question
Why did new users jump to a record on one day, then fall straight back?
Method
Broke the day down by device model, country and app language. The spike came from one country, eleven languages and a device model that only simulators report.
Finding
Most of the "record" was my own automated screenshot runs for the App Store. With them removed, the series had no spike at all, just steady growth.
Action
Test traffic is now filtered out at the source and in the pipeline, and old figures were corrected rather than quoted.
05 · Experiment design · Power analysis
Planning A/B tests on the paywall
Question
Does a free trial lead to more paying users than no trial? Does a redesigned paywall convert better?
Method
Two experiments run at the same time on separate country groups, so no user is in both. Before starting, I calculated how many users each arm needs to detect a realistic lift.
Finding
At the app's current traffic, even a large lift needs months to reach significance. Results are read as directional, and neither test is generalised beyond its countries.
Action
Both tests run with a fixed end rule. Try the calculation yourself:
–users per arm
–weeks to finish
Two-sided test, α = 0.05, power = 80%, equal split.
06 · Event study · Lagged correlation
Do social media posts bring downloads?
Question
Does a TikTok or Instagram post increase downloads and trials in the next days?
Method
For each posting day: a baseline from the 14 days before that had no post, adjusted for weekday, against the 3 days after. Windows with a release or another post are flagged. Across days: Pearson correlation of posts with outcomes at 0 to 3 days' lag, detrended, with a permutation p-value.
Finding
There were too few logged posting days for a reliable answer, so the analysis says "not enough data" instead of showing a correlation that would mean nothing.
Action
Every post is now logged with its date and platform, and the test runs automatically once there are enough posting days.
07 · Search data · Segmentation
German speakers weren't searching for English
Question
The website gets search impressions but few clicks. Where is the untapped demand?
Method
Grouped Google Search Console queries by the searcher's language and the language they want to learn, and looked at the backlink profile.
Finding
German was the biggest market with almost no clicks, and German speakers were mostly searching for Portuguese, Japanese and Korean, not English. Nearly all backlinks came from one source, which caps rankings whatever the content.
Action
Built German landing pages for those language pairs and started link building instead of writing more blog posts.
08 · Event data · Root-cause analysis
Event data exposed a feature that never worked
Question
How often do people use the AI study plan, and what does it cost?
Method
Broke the feature's events down by outcome, then compared the app's events with the server's cost log.
Finding
Every single request from a real user ended in an error before it reached the server. The feature looked used in the app's counts but had never worked for anyone.
Action
Fixed the cause. A feature's success rate is now part of the daily checks, not just its usage.
Study-plan request outcome
Share
Error on the device
all
Reached the server
none
Plan generated
none
09 · Data validation · Reconciliation
The report said zero, and it was wrong
Question
Why did the report show no paid subscriptions?
Method
Compared the pipeline's revenue with the payment provider's own charts, day by day, then traced the difference back to the code.
Finding
The report read a field the API does not return, so every payment counted as zero. Two goals had been met while the dashboard said they had not.
Action
Money figures now come from the provider's own charts and are reconciled every day. A missing field is reported as "not measured", never as zero.
Data quality: one number from three sources
Downloads are counted by App Store Connect, RevenueCat and Google Analytics, and they never match exactly. The pipeline compares them every day and picks the most reliable figure for each day.
Downloads per day by source
App Store Connect is the final number but arrives two days late, so its last two days are empty here, not zero.
Checks built into the pipeline
Source priority per day: App Store Connect when it has the day, otherwise RevenueCat minus test installs, otherwise GA4. Each figure is labelled Final, Provisional or Live.
Test installs removed: my own simulator and screenshot runs are detected by pattern and excluded from every figure.
Missing is not zero: when a source fails, the value is stored as "not measured". A run where no source answered stops with an error instead of reporting a quiet day.
Unsettled days are re-loaded: GA4 can still revise a day for 48 hours, so the job loads those days again until they settle.
Small samples flagged: a rate with fewer than 10 users in the denominator is marked and never highlighted.
Metric definitions and SQL
Each KPI is defined once in a catalog, so the dashboard, the report and any ad-hoc query calculate it the same way.
Example: Day-1 retention
Definition
Share of users who were active again one day after their first session
Numerator
Cohort members active on day 1
Denominator
Users whose first session was on the cohort date
Grain
Daily cohort; weekly and monthly values are recomputed from summed numerator and denominator, never averaged
Caveats
A cohort younger than one day has no value yet. Denominator under 10 is flagged as a small sample.
Source
GA4 cohort report, table fct_retention
Example query: a rate by country
Ratios are summed first and divided once, so a big country is not weighted the same as a small one.
SELECT x.key AS country,
SUM(x.value) AS trials,
SUM(y.value) AS new_users,
ROUND(100.0 * SUM(x.value)
/ SUM(y.value), 1) AS trial_rate
FROM fct_daily_dim x
JOIN fct_daily_dim y
ON y.date = x.date
AND y.dimension = x.dimension
AND y.key = x.key
AND y.metric = 'new_users'WHERE x.dimension = 'country'AND x.metric = 'trial_users'AND x.date BETWEEN :from AND :to
GROUP BY x.key
HAVINGSUM(y.value) >= 10
ORDER BY trial_rate DESC;