Scraping job postings for investor insights (using AI)
•By Dave Wang
Every investor wants to know what a company is building before management says it out loud.
The problem is that most of the early signals are messy.
They do not show up as clean KPIs in a filing.
They show up as corporate exhaust. Job postings, supplier comments, procurement records, app reviews, obscure PDFs, contract awards.
Hiring is one of the best examples.
If a company is suddenly recruiting foundation model engineers, GPU validation leads, data center SREs, or payments infrastructure architects, that tells you something about where resources are going.
But raw job boards are a disaster.
The same role can show up in five cities. Retail roles can swamp the corporate signal.
Evergreen roles sit open forever. Generic descriptions make keyword search look smarter than it is.
That is exactly the kind of mess AI can help with.
Not because AI magically knows what a company is doing.
Because AI can read messy text, extract structure, compare similar roles, classify intent, and explain uncertainty when you give it the right rails.
So I wanted to test a simple question.
Can you use AI to turn public job postings into something closer to an investor roadmap tape?
I ran the workflow on Apple and scraped ~6,000 job postings.
The trick is not scraping more data....
The trick is designing the scrape so the data can be trusted....
Here was the plan:
Pull the full jobs index
Save raw payloads for auditability
Deduplicate to stable requisition IDs
Separate retail from corporate roles
Use AI to classify strategic themes
Use AI to flag roles worth reading manually
Let's build this jobs insights dashboard together!
The Prompt
I ran this as a Claude Code workflow.
This is not a one-shot prompt where you ask for "Apple hiring analysis" and hope the model does the right thing.
The design matters.
The prompt has to force the model to think like a data pipeline before it thinks like an analyst.
Here is the logic I baked in:
Raw-first collection. Save every source payload so the work can be audited....
Requisition-level counting. Count one posting ID as one role, even if it appears in multiple locations.
Confidence scoring. Weak matches get flagged instead of turned into fake certainty.
Watchlist extraction. Broad counts are useful, but specific postings can be big signals (e.g., head of a new division?).
Tracker-ready schema. One snapshot is helpful. But you should run this across time to get time-series analysis...
After deduping to requisitions, that became 4,897 unique postings.
After separating retail and pipeline roles, the default investor tape was 4,092 corporate roles.
That cleaning step matters because the conclusion changes if you count the wrong thing.
The key to any AI work / data science work is quality data! The great thing about AI is now it can help us normalize or clean up messy datasets automatically.
The cleaned tape did not show one simple strategy.
It showed a stack.
Apple had 1,092 silicon, GPU, platform, and wireless roles, 1,008 hardware and product roles, 828 AI, Siri, and Foundation Model roles, and 189 cloud, PCC, and data-center roles.
That is a more useful read than “Apple is hiring for AI.”
It suggests Apple’s public hiring signal is spread across the layers that make AI matter: devices, silicon, models, product surfaces, private compute, services, and trust.
Some key takeaways for AAPL investors analyzing the raw jobs tape that AI surfaced:
Hardware and silicon are the largest technical signal: 42.7% of non-retail visible roles go here
AI is meaningful, but split across product and infrastructure: 828 AI/Siri/Foundation Model roles show up across Siri, model work, Vision Products, cloud, and compute
The strongest read is AI distribution: Apple appears to be staffing the product surfaces where AI gets embedded, including Siri, Vision, App Store, Wallet, and iCloud-adjacent systems
Some roles hint at AI being used internally, not just shipped externally: roles like applied AI for EDA/platform architecture suggest AI may be entering Apple’s engineering process itself
The Bigger Point
This is not really an Apple story.
It is an AI diligence story....using AI-native data science techniques.
The lesson is that AI works best when it is pointed at a specific messy-data problem.
(you just need to make sure you are asking the correct questions + structuring your prompt logic correctly)
What are some other ways you can leverage this same jobs scraping techniques for other investing diligence?
Hiring vs management commentary: Compare job themes against earnings calls, investor days, 10-Ks, and press releases.
Unannounced priority watchlist: Alert when new senior or unusually specific titles appear.
Velocity tracker: Track new roles, closures, reopenings, and duration by theme.
Peer comparison: Compare a company's hiring mix against comps
Narrative mismatch screen: Find areas where the hiring tape is loud but the company narrative is quiet.
Just some ideas for you - happy hunting!
Personal
I was recently appointed by OpenAI to be an official Codex Ambassador!
I'm already spending pretty much all waking hours using Codex / Claude Code / other AI tools to crack new AI for investing angles....so that doesn't change.
But what does change is I'll have direct access to OpenAI devs, early previews of models or features, and resources to host in person events for YOU.
One thing I'm thinking of is organizing a free AI for Investing hackathon for my newsletter
If you would come, hit reply with your city .... I travel a lot to the different global financial centers