Ironside
1.9K posts


LLM Knowledge Bases
Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge (stored as markdown and images). The latest LLMs are quite good at it. So:
Data ingest:
I index source documents (articles, papers, repos, datasets, images, etc.) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of .md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into .md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them.
IDE:
I use Obsidian as the IDE "frontend" where I can view the raw data, the the compiled wiki, and the derived visualizations. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. I've played with a few Obsidian plugins to render and view data in other ways (e.g. Marp for slides).
Q&A:
Where things get interesting is that once your wiki is big enough (e.g. mine on some recent research is ~100 articles and ~400K words), you can ask your LLM agent all kinds of complex questions against the wiki, and it will go off, research the answers, etc. I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this ~small scale.
Output:
Instead of getting answers in text/terminal, I like to have it render markdown files for me, or slide shows (Marp format), or matplotlib images, all of which I then view again in Obsidian. You can imagine many other visual output formats depending on the query. Often, I end up "filing" the outputs back into the wiki to enhance it for further queries. So my own explorations and queries always "add up" in the knowledge base.
Linting:
I've run some LLM "health checks" over the wiki to e.g. find inconsistent data, impute missing data (with web searchers), find interesting connections for new article candidates, etc., to incrementally clean up the wiki and enhance its overall data integrity. The LLMs are quite good at suggesting further questions to ask and look into.
Extra tools:
I find myself developing additional tools to process the data, e.g. I vibe coded a small and naive search engine over the wiki, which I both use directly (in a web ui), but more often I want to hand it off to an LLM via CLI as a tool for larger queries.
Further explorations:
As the repo grows, the natural desire is to also think about synthetic data generation + finetuning to have your LLM "know" the data in its weights instead of just context windows.
TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a .md wiki, then operated on by various CLIs by the LLM to do Q&A and to incrementally enhance the wiki, and all of it viewable in Obsidian. You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts.
English

@MattHDGamer 1) Limited game modes turn me off, sweaty stress
2) I can't grow/maintain 'my' ultimate team, just meta
3) Too much time needed to 'keep up'
4) League SBCs to build fodder? Give over
Notice how none of the above is about game play. I'm the segment they've lost.
English

@hafiz_11120 Hi Hafiz, our price changes are essential to maintain safe, high-quality drinking water, fix leaky pipes and ageing mains, and invest in upgrading our sewage works to improve river health (1/2)
English

But water bill is up by 40% @thameswater
UK Prime Minister@10DowningStreet
Today the energy price cap has dropped by £117. We said we’d bring energy bills down - we meant it.
English


@NepentheZ @EASFCDirect dont know how casuals dont have fodder. I completed Messi from a week of saved packs. This SBC was hardly needed
English

I'm blown away @EASFCDirect
88-90 x5 Per day (2 days repeat only) will enable someone to complete ONE x 89 rated squad if they're luck.
82+ x25 is good value only needs in 2x 83 squads w/ IF's but is only repeatable 10x per day.
Why? What are you doing?
English

@FUT_Yapper I'm beyond fed up with it. Ruins the game. Sacking off the game next year. Life is too important for this bullshit.
It's a fooking digital card that'll be outdated in two months time. WHO do they think they are holding that from us.
English

Yep let’s just believe a random website that shows the ‘confirmed pack odds’
Great idea guys 👍
FGZ ⚽️🎮@FGZNews
🚨 ONLY 10 TOTYS HAVE BEEN PACKED FROM 90,000 LA LIGA PREMIUM UPGRADES ☠️ ☠️ WTFFFFF 📸 @Aleks_FUT | #FC26
English

@Aleks_FUT Doesn't seem to prioritise my SBC storage though?
Lewisham, London 🇬🇧 English

FREE ai SBC Solver you need for TOTY 🌟
• Grind 82x20 packs in seconds with cheapest solution
• auto choose player picks
• quick open packs (skip animation and send to club / quick sell with 2 keyboard buttons)
• complete league SBCs in 5 seconds automatically with cheapest price
• gauntlet ai squad builder using your main teams tactics
• marquee matchups solved from club in 5 secs
20x free SBCs a day or only £5 for unlimited month 📆
English

@d123456bb How? They're not going to tax the large corporations who are their funders. They like to keep all of that quiet, don't they.
English

Merry Christmas @Nigel_Farage one of the few MP's in this country that most people would love to have a pint with 🎄🍻

English

@LMenhenott REFORM have screwed up every constituency they've taken control of thus far.
English

@Nigel_Farage Missed out the bit where you take accountability for all of the lies and slander throughout Brexit.
English

@trussliz @RachelReevesMP Are you genuinely a troll? You're so unaware of what you did and how you're perceived. Reeves is doing poorly, but you're infinitely worse and I pay £900 per month in interest as a result of your brain.
English

Of course @RachelReevesMP doesn't blame it on:
- Bank of England printing money
- Wrong OBR forecasts
- Net Zero and highest energy costs in the West
- Highest taxes for 70 years
- State spending 45% of GDP
-£400bn COVID spending
- Bloated welfare bill
No - it's Truss and Trump!
Everyone knows she's lying.
That's why she's the least popular Chancellor in history.
English

Fact check:
- Reeves has signaled intent to fully scrap the two-child benefit cap in the Nov 2025 budget, per Guardian and BBC reports, to address child poverty (cost ~£3bn).
- UK families with 3+ dependent children: 14.8% overall (ONS 2024). No recent ethnic breakdown found; older data (2019) shows higher rates for Pakistani/Bangladeshi households, but 40% not confirmed.
- Unemployment: Pakistani/Bangladeshi ~9-11% vs white British 3-3.3% (2022-2024, GOV.UK/Ethnicity Facts, Parliament), or ~3x higher.
Demographic "replacement" claims are subjective interpretations.
English

Rachel Reeves is evil.
Scrapping the two child benefit cap will just accelerate the demographic replacement of native Brits.
Only 14% of white British families have three or more children compared to 40% of Pakistani and Bangladeshi.
Plus they’re 2-3 x more likely to be unemployed than the white British population so we’re the ones funding it.
With tax rises, this is going to make it impossible for white British to have a children.
You’re funding your own replacement.
Politics UK@PolitlcsUK
🚨 NEW: Rachel Reeves will officially scrap the two-child benefit cap in the Budget [@thetimes]
English











