You don't need to be a blockchain enthusiast to follow this post. Here's the minimum context:

The setup: Polkadot is a blockchain network with a communal treasury currently worth a few millions of dollars. Anyone can submit a proposal requesting funds ("give us 50k$ to build a developer tool"), and token holders vote on whether to approve it. It's like a decentralized grants program with no central committee.

The problem: There are a lot of proposals. Reviewing them all is a full-time job. Most token holders don't have time, so they delegate their voting power to "delegates", people (or, in this case, AI systems) who vote on their behalf.

Subsquare (https://polkadot.subsquare.io/) Think of it as the governance dashboard. It's where proposals are listed, discussed, and voted on. When my AI agents voted, their reasoning appeared as comments on Subsquare for everyone to see.

Treasury governance is a perfect test-bed for AI decision-making because:

• Real money is at stake (not a toy problem)
• Proposals are adversarial (people try to game the system)
• Everything is public and recorded permanently
• The community expects accountability

If you've ever thought "we should use AI to help make complex decisions, but how do we make sure it's not a black box?" that's exactly what this experiment tried to answer.

Now, back to the story.

The delegation has now ended and it is time to reflect on the Cybergov V0 experiment. If you're interested to read about how this started, check out these links:

• Initial idea: https://forum.polkadot.network/t/decentralized-voices-cohort-5-light-track-karim-cybergov/14254
• Technical breakdown (how it works): https://forum.polkadot.network/t/cybergov-v0-automating-trust-verifiable-llm-governance-on-polkadot/14796

For three weeks in September 2025, three AI agents named Balthazar, Melchior, and Caspar voted on blockchain treasury proposals on my behalf. They analyzed 19 proposals on Polkadot, 3 on Kusama, and a handful on the Paseo test network. Here's what happened and what it might mean for anyone building AI systems that need to be trusted.

Why this matters

Before diving in: you don't need to care about Polkadot to find this interesting. The core problem is universal: how do you build AI systems that make consequential decisions while remaining auditable, transparent, and resistant to manipulation?

Blockchain governance was my test case because:

• Decisions involve real money (treasury funds)
• There's an adversarial environment (people will try to game the system)
• Everything happens on-chain, creating a permanent record
• The community expects transparency from delegates

But the lessons apply anywhere you're deploying AI for high-stakes decisions: content moderation, loan approvals, hiring recommendations, medical triage. The question is always the same: can we trust this thing, and can we verify that trust?

The experiment

CyberGov V0 was an experiment to see if LLMs could provide transparent, reproducible governance decisions. Instead of me personally reviewing dozens of treasury proposals, I built a system where three AI agents would independently analyze each proposal and collectively decide how to vote.

The names come from the MAGI supercomputers in Neon Genesis Evangelion: three systems that must reach consensus to make critical decisions. Each MAGI had a distinct personality based on aspects of their creator. I did the same thing.

The Three Agents

• Balthazar (GPT-5) was the strategist. His job was evaluating whether proposals strengthened Polkadot's competitive position against other blockchains. Does this create sustainable advantage, or just temporary hype?
• Melchior (Gemini 2.5 Pro) focused on ecosystem growth and ROI. His core question: does this activity actually translate into measurable value, or are we just subsidizing user acquisition that evaporates when the money runs out?
• Caspar (Claude Sonnet 4) was the risk analyst, treating every treasury allocation as an investment rather than a grant. He flagged moral hazard, questioned multi-year commitments, and demanded accountability mechanisms.

Each agent received the exact same proposal text but evaluated it through their distinct lens.

The numbers

Over three weeks, the system voted on 19 treasury proposals:

Only 4 votes were unanimous (21%). The rest involved disagreement between agents.

Distinct personalities

The per-agent breakdown reveals how the personas actually influenced behavior:

Melchior wanted to fund almost everything. Caspar wanted to fund almost nothing. Balthazar was the swing vote. This was directly reflected how I'd written their system prompts.

One interesting case: Proposal 1703 had Balthazar voting Nay while Caspar and Melchior voted Aye. The strategist saw competitive risk; the risk analyst (surprisingly) saw an acceptable investment. The growth analyst saw opportunity. Final result: Aye. The system worked as designed: genuine disagreement led to a deliberated outcome. After a lengthy discussion with people reaching out and commenting, the truth table for the MAGIs was updated to reflect the following:

This means that retroactively, the decision should have been ABSTAIN.

The voting logic

The truth table was then deliberately made more conservative:

• Unanimous agreement → Cast that vote
• Two agree, one abstains → Cast the majority vote
• Any genuine disagreement → Abstain

If Balthazar saw strategic value but Caspar flagged unacceptable risk, the system abstained. The philosophy was: when in doubt, don't spend someone else's money.

The 53% abstention rate wasn't a bug but the system being appropriately uncertain. Most proposals had something to like and something to worry about.

Radical Transparency

Every vote came with a manifest file containing SHA256 hashes of all inputs and outputs (example), links to the GitHub Actions run where inference happened, and the exact proposal text the agents saw. The hash was submitted on-chain alongside each vote.

Anyone could:

1. Download the manifest
2. See exactly what text the agents received
3. Verify the hash matched what was recorded on-chain
4. Re-run the pipeline to check reproducibility

This is table stakes for trustworthy AI. If you can't show your work, you shouldn't expect trust.

Consistent analysis

The agents never had a bad day. They evaluated proposal #1757 with the same rigor as proposal #1701. They caught prompt injection attempts (during testing). They flagged missing budget breakdowns and vague milestones consistently.

Each agent produced structured output:

• Neutral critical analysis with scores (Feasibility/10, Value-for-Money/10, Risk/10)
• Key factors considered
• Decision trace showing reasoning
• Safety flags for detected issues
• Persona-filtered rationale

Testnets are your friend

Before touching real governance, I ran the system on Paseo (Polkadot's testnet). Subsquare (the governance interface) worked identically on testnet, so I could see exactly how comments would render, test the proxy account setup, and verify the whole pipeline without risking actual treasury funds.

This sounds obvious, but many AI deployments skip this step. If your system can fail safely in a sandbox first, use the sandbox.

GitHub actions logs expire

Here's something I didn't anticipate: GitHub Actions logs have a retention limit. After 90 days, those "verifiable execution logs" I proudly linked to? Gone.

For a governance system where accountability might matter years later, this is a real problem. The on-chain hash remains, and the manifest files in S3 persist, but the process evidence disappears. A future version needs to archive execution logs to permanent storage like the one being built for Polkadot right now.

Lesson: Audit trails need to outlive your cloud provider's default retention policies.

Context limitations

The agents had no memory of past proposals or community relationships. They couldn't know that this was someone's third failed delivery, or that this team had consistently exceeded expectations before.

Proposal 1745 got an Abstain partly because the agents couldn't see the proposer's track record. A human delegate would have known the context.

Lesson: We need an embeddings database or a historical archive of all proposals' contents.

External link rot

Proposals often linked to external documents like Notion pages, Google Drive, etc. The system deliberately excluded these (too much complexity, URL rot risk), which meant agents sometimes missed crucial details.

This is a fundamental tension: you want agents to evaluate complete information, but you also want deterministic inputs. I chose determinism over completeness. Not sure that was right.

Lesson: a Web3 governance system needs, at its core, a way to provide decentralized proposal content submission (important read)

No deliberation

When agents disagreed, the system just... abstained. There was no "Balthazar, explain to Caspar why this risk is acceptable." No synthesis of perspectives. Just voting logic.

Real consensus involves argument, persuasion, and updating beliefs. CyberGov V0 had none of that.

DSPy changed how I think about LLM applications

I used DSPy for LLM orchestration, and it was a revelation. Instead of prompt engineering through trial and error, I defined:

• A signature (what inputs and outputs I expected)
• A few training examples
• A compilation step that optimized prompts automatically

The framework handled few-shot learning, structured outputs, and cross-model compatibility. When I switched from one model to another, the same pipeline worked.

Lesson: Stop hand-crafting prompts. Use a framework that treats prompt optimization as a learnable problem.

Transparency is achievable

The "verifiable process > opaque conviction" principle worked. Blockchain + deterministic settings + content-addressed storage = auditable AI decisions.

But it required:

• Running inference on public infrastructure (GitHub Actions)
• Storing all artifacts with content hashes
• Submitting attestations on-chain
• Building a whole transparency layer into the output

Most teams won't do this. They should.

Multi-Agent consensus is fragile

Three models disagreeing doesn't mean "this is a hard decision" but more often meant the proposal was written ambiguously, or one model fixated on an irrelevant detail.

Having multiple agents creates useful tension, but you need better mechanisms for resolving that tension than "just abstain."

Personas help, but they're not wisdom

Giving agents distinct evaluation criteria (strategic vs. growth vs. risk) created useful diversity. The Caspar/Melchior dynamic (one conservative, one aggressive) was genuinely valuable.

But they were still pattern matchers. When Caspar flagged "moral hazard," he was matching that concept to proposal text, not reasoning from first principles about incentive structures.

The abstention default was correct

In governance, the cost of a wrong YES (wasted treasury funds) exceeds the cost of a wrong ABSTAIN (missed opportunity). The conservative bias felt right.

53% abstention might look like the system was useless. I'd argue it was appropriately humble. Here's its policy compared to other delegates:

What a V1 could look like

CyberGov was a V0 proof of concept. Here's what a production version might include:

Historical context via RAG

Build a vector database of past proposals, their outcomes, and proposer track records. Before evaluating a new proposal, retrieve relevant context: "This team delivered Project X on time and under budget" or "This proposer's last three proposals failed to deliver milestones."

I have a POC of this using ChromaDB but it's not ready for prime-time.

Agent deliberation protocols

Instead of independent voting, have agents respond to each other:

1. Each agent gives initial assessment
2. Agents see each other's concerns
3. Round two: agents can update their position or rebut
4. Final vote

This mimics how actual committees work. The computational cost is a tiny bit higher, but the decisions would be richer. IMHO this is a primitive needed also beyond Web3 AI governance.

Dynamic re-evaluation

Proposals evolve during voting periods based on community feedback. A V1 could monitor for significant edits and trigger re-analysis, possibly changing its vote if new information addresses previous concerns.

Permanent audit logs

Store execution logs, full API responses, and all intermediate artifacts to permanent decentralized storage. Governance decisions might be contested years later but the evidence needs to persist and outlive us.

Human-in-the-Loop mode

The best near-term use case might not be autonomous voting, but AI-assisted analysis that a human delegate reviews. The structured output (scores, factors, traces) would be genuinely useful as "first-pass triage" before a human makes the final call.

The MAGI system in Evangelion was ultimately fallible because it could be hacked, manipulated, or simply wrong about what humanity needed. CyberGov V0 was too. But at least you could see exactly how it was wrong.

That's the real contribution here: not that AI can govern well, but that AI governance can be transparent. The ability to audit AI decisions matters more than whether any particular decision was optimal.

CyberGov V0 abstained 53% of the time. It voted against only one proposal. It agreed with itself only 21% of the time. By most metrics, it was not conclusive.

But every decision was:

• Publicly reasoned
• Cryptographically attested
• Independently verifiable

That's more than most human delegates offer.

Links:

• GitHub repository (GPL3)
• Original forum post
• Technical architecture

Note: This is a long form essay synthesizing my research and observations on web architecture and decentralization. I'm writing this primarily to organize my own thinking, but I'm sharing it publicly because I believe these ideas matter. I'm not claiming novel invention as most of these primitives have existed for years already. Also, I'm certainly not claiming to have all the answers. If you spot errors, faulty reasoning or have alternative perspectives, I genuinely want to hear them. This is exploratory work and not gospel, the conclusion contains a recap of the claims of this essay. Consider it an invitation to discussion.

Disclaimers: 80% my thoughts & ideas, 20% formatting & structure using gemini-2.5-pro. Title image by nano-banana.

I believe that the architecture of the modern web not only makes meaningful decentralization difficult but that it also structurally precludes it. As a consequence, seemingly "decentralized" systems built on top of modern web architecture inevitably collapse toward centralized intermediaries. I'm proposing an alternative set of primitives, or ways to think about the problem, that when properly composed, could make decentralization the path of least resistance rather than maximal pain.

Centralization gradient

Systems built on the modern web stack exhibit a consistent pattern: regardless of initial design intent, they converge toward centralized architectures over time, this almost feels like an architectural inevitability.

Every major protocol built on web foundations has followed this trajectory:

• Email was designed federated but now it's a Gmail/Outlook duopoly (>80% of users)
• Web hosting was designed distributed but today it's an AWS/GCP/Azure oligopoly (>60% of infrastructure)
• DNS was designed distributed but Cloudflare resolves >20% of web traffic
• etc...

I believe that these outcomes are not coincidental but that the architecture of these systems contain structural properties that make centralization energetically favorable.

Architectural centralization

A system is architecturally centralized if:

• Functioning requires third-party infrastructure that could observe, modify, or deny service
• Optimal performance or reliability creates pressure toward shared infrastructure
• Correctness relies on trusting entities outside the end-to-end path
• P2P (peer to peer) operation requires significantly more complexity than client-server

A system may be logically decentralized (example: federation) while remaining architecturally centralized if these properties hold.

Current modern web primitives

Location based addressing (DNS/IP)

The primary way of accessing the web today is by using what's called location based addressing. You click a link and you get a result: resources/things are identified by network location (https://domain.com/resource)

The request flow goes something like this (very roughly):

In this flow, there are 4+ mandatory intermediaries (DNS, CA, ISP, origin server) and more than 3 trust assumptions along the way (DNS hierarchy, CA system, server identity).

The consequences of this primitive:

• You cannot verify the content, a same URL can return different content. So no caching guarantees, no integrity verification, no offline operation. Additional layers can be used for mitigation (SRI, signatures, etc) but these are optional and never used in practice
• The server sending the response needs to be always on, if the origin is offline, the resource is unavailable. This means, 24/7 infrastructure costs, DDoS vulnerability, geographic latency, the mitigation of which is usually involving a CDN intermediary (read: more centralization)
• The performance is degraded by physical distance to the origin (geographic coupling). Users farther away have worse experience, so distributing the infrastructure becomes a way to make the experience better (thus, economies of scale favor large provider)
• Domain owners can change content, revoke access, track usage (censorship vector, surveillance vector)

These consequences combined posit that in location addressed systems, optimal resource availability requires resource replication across multiple locations. Replicating infrastructure exhibits economies of scale. Therefore, providers with larger infrastructure networks have structural cost advantages, creating convergent pressure toward consolidation.

Stateless Request/Response (HTTP)

Using HTTP, each request is independent, there is no session state in the protocol.

A HTTP exchange goes something like this (way simplified):

Request: Method + Headers + Body
Response: Status + Headers + Body

(so no persistent connection, no identity, no state)

The consequences of this primitive:

• Since applications need state and HTTP provides none, implementations usually leverage cookies, tokens, server-side sessions, resulting in applications usually implementing state management differently. This is a direct centralization vector because the session state is stored server-side, and thus the user bound to specific server pool.
• There is no identity layer in the protocol, so implementations leverage simple credentials, oauth flows, session tokens, which again, every application implements however they want. The usual mantra is "don't roll your own auth", so what devs do is they externalize it to identity providers. Another centralization vector.
• All requests are observable by network path, observers including ISPs, DNS resolvers, potential MITM... Of course a mitigation here is TLS encryption but as we'll see later, current ways of doing it aren't any better and introduce trust assumptions too.

So all that combined, it's safe to say that in stateless protocols, applications requiring state must implement state management at the application layer. Application-layer state management cannot provide end-to-end guarantees without trusted intermediaries. Therefore, stateless protocols force trust delegation to intermediaries.

In fact, the state must be stored somewhere, the options being: (1) client-side (tamperable), (2) server-side (requires trusting server), (3) distributed consensus (requires coordination protocol). Options (2) and (3) both require trusting entities beyond end points.

Interpreted execution (JavaScript)

Execution code is delivered as simple text and interpreted by browsers at runtime.

How JavaScript ends in your browser:

The consequence of this primitive are:

• Code execution is non-deterministic and hinges on browser version, JIT optimization, API availability, extensions etc
• Users cannot verify what code will execute because the source can be obfuscated, minified, or even completely different per request (note: even with SRI, you can't verify execution, only source)
• Browser vendor (third party) controls the runtime so they can deprecate APIs, add tracking, modify behavior (manifest V2 example), and there is absolutely no recourse
• Tabs share browser context, bringing with it cross-site tracking, fingerprinting etc...

Put together, in interpreted execution models, code verification is impossible without trusting the interpreter. The interpreter is provided by a third party (browser vendor). Therefore, code execution requires trusting a third party.

Certificate authority trust model (TLS/PKI)

This is the big one, trust is delegated to certificate authorities.

How it works:

The consequence of this primitive are:

• All CAs must be trusted equally, meaning any CA can issue cert for any domain. Certificate transparency might mitigate this but doesn't solve it once and for all. There is a certain trust transitivity involved.
• CAs are subject to national jurisdiction, governments can compel CA to issue malicious certs. We've seen this happen.
• Certs rotate, there is no no identity persistence and it isn't possible to verify "same entity as last time". The result of this is usually that the identity is conflated with the domain name (controlled by the registrar) and not the cryptographic key ensuring the green lock in the browser
• Let's face it, ~3 CAs (or 2, if you think about it) handle the majority of web. A single CA compromise affects a massive swath.

Again put together, it's easy to say that in delegated trust models with transitive trust, security is bounded by the weakest link in the trust chain. As the number of trusted entities increases, the probability of voluntary or involuntary compromise approaches 1. For more on CAs please read this excellent (although tongue in cheek) article: https://michael.orlitzky.com/articles/lets_not_encrypt.xhtml

Why centralization is inevitable

The architectural primitives described above both enable centralization and make it structurally inevitable through four reinforcing mechanisms:

The intermediary trap

The web's architecture creates a recursive pattern where intermediaries are first convenient and then necessary. It starts with the primitives themselves: location addressing requires always-on infrastructure, stateless protocols demand session management, and performance requirements push toward geographic distribution.

Users naturally gravitate toward intermediaries offering the best performance and reliability. But here's where the trap springs: as usage concentrates on particular intermediaries, economies of scale kick in. Higher utilization means better cache hit rates for DNS resolvers, more efficient PoP (point of presence) utilization for CDNs, and improved spam filtering for email providers. These efficiency gains create a competitive moat that new entrants simply cannot breach. New entrants cannot match the unit economics of incumbents without first achieving comparable scale, which they cannot do without matching the performance that economies of scale provide. The market consolidates not because of anti competitive behavior but because the architecture makes any other outcome economically irrational.

Consider DNS resolvers: Google's 8.8.8.8 and Cloudflare's 1.1.1.1 collectively handle over 30% of global DNS traffic last I checked. They didn't capture this market through superior innovation alone though: recursive resolution fundamentally requires always-on infrastructure, and performance improves dramatically with scale as cache hit rates increase. Each query these services answer makes them marginally faster for the next query. A new entrant starting with zero queries has zero cached answers and must perform full recursive lookups for everything. They're structurally disadvantaged from day one. And here's the kicker: these entities see every domain every user visits, creating an observation point that enables both surveillance and control.

The same pattern plays out with CDNs. Geographic distribution is expensive: operating points of presence globally requires massive capital expenditure that only makes sense at scale. Fixed costs get amortized across more traffic, creating sublinear cost scaling. Cloudflare, AWS CloudFront, and Fastly don't just handle the majority of web traffic but they can modify, block, or surveil any content passing through their networks. Users don't choose this arrangement because they want surveillance but just because the alternative is unacceptably slow.

Email completed this same journey decades ago. SMTP's requirement for always-on servers with static IPs created a baseline infrastructure cost that favored consolidation, but spam filtering accelerated it dramatically. Effective spam detection improves with data scale because the more email you process, the better your models become. Same as with the CDNs, Gmail and Outlook don't just host email but they can read all of it, and they implement opaque rules for inbound and outbound mail that smaller providers must accommodate or face deliverability issues. Independent email servers are now so likely to be classified as spam that running your own email infrastructure is effectively infeasible for most use cases.

Performance gradient

Location addressed systems impose a hard physical constraint: latency is proportional to distance, bounded by the speed of light. This is just physics. Optimal performance therefore requires geographic distribution of infrastructure which is capital intensive and exhibits strong economies of scale.

Consider what actually happens during a web request. You pay for DNS lookup (20-120ms), TCP handshake (one round-trip time), TLS handshake (one or two RTTs depending on version), the HTTP request itself (another RTT), server processing time, and finally response transfer. For a user in New York accessing a server in Singapore, you're looking at about 15,000 km of distance. The physical lower bound for a round trip at light speed in fiber is roughly 150ms, but actual RTTs hit 200-250ms due to routing overhead. A typical page load requiring six round trips burns 1200-1500ms on network latency alone, before a single byte of content is processed.

The standard mitigation is deploying a CDN with global PoPs, but this costs at minimum $200/month for serious global coverage. Free CDN tiers exist, but they're either severely limited or subsidized by other revenue (usually surveillance-based advertising).

The economic implication is that only applications with direct revenue or subsidy can achieve acceptable global performance. Free or low revenue applications deliver inferior experiences, pushing developers toward ad-supported models that require user tracking which, of course, benefits from centralized data collection. The architecture makes surveillance capitalism the economically rational choice for anyone who wants their application to be fast.

Trust externalization spiral

When protocols lack essential primitives, applications must implement them at higher layers. This seems reasonable until you realize that application-layer implementation complexity creates strong economic pressure toward outsourcing, which concentrates users on a few providers, which grants those providers both data access and leverage, which creates lock-in. The spiral is self-reinforcing.

Authentication demonstrates this perfectly. HTTP has no identity primitive, so sites initially used username and password. Password reuse created security vulnerabilities, so password managers emerged. But password managers don't solve phishing, so two-factor authentication became necessary. 2FA improved security but added friction, so "Sign in with Google/Facebook/GitHub" emerged as the path of least resistance. Today, over 50% of sites support OAuth, and over 30% of users rely on federated identity. What seems like user convenience is actually structural: implementing robust authentication is sufficiently complex that outsourcing it is the rational choice for most developers.

But look at what happens during an OAuth flow. User clicks "Sign in with Google" then gets redirected to Google, and Google learns which site the user is visiting, when, and can correlate this with the user's identity. Google returns a token to the site asserting the user's identity, and the site trusts this assertion. Google can now see every login to every site using its OAuth, and can, if it chooses, impersonate any user. The site gains convenience and Google gains comprehensive surveillance of user behavior across the web.

None of this would be necessary if identity were a protocol primitive. With public key identity, the user signs a challenge, the site verifies the signature, and no third party is involved. But because HTTP lacks this primitive, we've built an entire industry around outsourced identity that extracts rent and surveillance rights in exchange for solving a problem that shouldn't exist.

I wrote about this in my earlier article here if you'd like to read a bit more about this:

We accidentally built the wrong internet

We built the internet on email & passwords, coupled with an analog payment system based on typing 16-digit numbers into forms. If someone pitched this today, we’d laugh them out of the room. The answer might be in places most of our brightest minds aren’t willing to look at - yet.

Karim JeddaKarim Jedda

Coordination tax

Perhaps the cruelest irony of web architecture is that building peer-to-peer systems on top of it is harder than building client-server systems, despite P2P being conceptually simpler (direct connection between peers). The architecture itself has been optimized for client-server patterns so thoroughly that P2P operation pays a massive coordination tax.

WebRTC demonstrates this tax explicitly. To establish a browser-based P2P connection, you need a STUN server (third party) to discover your public IP, a TURN server (third party) to relay traffic if NAT prevents direct connection, and a signaling server (third party) to exchange connection metadata. You then perform ICE negotiation, trying multiple connection paths to find one that works. Success rate for direct P2P is only about 70% due to NAT configurations, the remaining 30% requires TURN relay, which introduces the very intermediary you were trying to avoid. Oh, and this process adds 200-500ms of latency just for negotiation.

Think about that: connecting two browsers directly is harder and slower than connecting both browsers to a server, despite the direct path being shorter and requiring less infrastructure. This of course isn't WebRTC's fault though but rather the inevitable consequence of building P2P on an architecture optimized for client-server. NATs, firewalls, and ISP routing policies all assume client-server topology. P2P is swimming upstream while the architecture actively pushes water against it.

The fundamental problem is one of design philosophy:

• A web browser's job is to be a secure, sandboxed client. Its primary function is to fetch and display untrusted content from a server while protecting the user's machine from that content. It is designed for a client-server world. It requests, it does not serve.
• A P2P peer's job: to be a sovereign, first-class citizen of the network. It must be able to listen for incoming connections, store significant amounts of state, route data for others, and run persistently. It must be both a client and a server.

These two philosophies are in direct opposition. Trying to force a peer's job into a browser's sandbox leads to a series of crippling compromises.

The result is that developers rationally choose client-server not because it's technically superior but because the web makes it the path of least resistance. This is architectural determinism: the primitives shape the possibility space so thoroughly that alternatives become economically irrational even when technically feasible.

The above taken together, explains at least to me, why so many "decentralized" web3 projects end up with centralized components: they're fighting the architecture itself.

Alternative primitives

If centralization is structurally embedded in web primitives, the question becomes: what primitives would make decentralization structurally natural? I'm not claiming these are novel as many have existed for years or decades, but their composition creates a fundamentally different architectural possibility space.

Content addressing

Content addressing means resources are identified by cryptographic hash of their content rather than network location. The address format is simple: hash://algorithm:digest (e.g., ipfs://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi). Verification is mathematical: hash(received_content) == address. Source is irrelevant as you can fetch from any peer because the hash guarantees content integrity.

This primitive has profound cryptographic properties. Hash collision implies content match, which means content is verifiable without trusting the source. Identical content has identical hash, which enables storage and bandwidth savings that scale with content reuse (reuse). Content can be fetched from untrusted sources safely, eliminating single points of failure and enabling resilience to server outages and censorship. Performance improves automatically because you can fetch from the geographically nearest peer rather than a specific origin.

The tradeoff is that no entity controls the hash space, so you cannot revoke access to content once published. This is a feature for censorship resistance but a bug for content moderation. Versioning requires higher-level abstractions like name services or blockchain registries.

Compared to location based addressing:

In content-addressed systems, resource availability scales with demand. Under P2P architecture, high-demand resources get replicated more, improving both availability and performance. Popular content is cached by more peers, and availability approaches 1 as the number of peers increases: availability = 1 - ∏(1 - p_i) where p_i is the probability peer i is online.

Cryptographic identity

Entities are identified by public keys. The identity is the public key itself, example: Ed25519, 32 bytes. Authentication is accomplished by signing a challenge with the private key. Authorization is simply verifying the signature matches the public key.

The consequences of this architectural choice are interesting. "Registering" becomes a non-concept as there's no dependency on identity providers. Identity generation is unobservable and costs nothing since you just generate a keypair. The public key is the identity, and a signature proves knowledge of the corresponding private key. No CA needed, no third party involved. This works over any channel: signing HTTP requests, data structures, transactions, it's all the same and isn't tied to a specific protocol. Assertions about other keys enable complex authorization without central authority through delegation chains.

Compared to traditional authentication:

In public key identity systems, authentication requires no third party and is unforgeable (assuming cryptographic hardness). This is at the same time more secure but also differently secure. The attack surface is local device security, not the security of identity providers and all network paths to them.

Deterministic execution

WebAssembly (WASM) provides a binary instruction format with specified semantics. It's a stack-based virtual machine with guaranteed memory safety, control-flow integrity, and determinism. Same input always produces same output, which means you can verify behavior without trust as the execution environment cannot subvert the computation.

WASM enforces principle of least privilege architecturally through capability based security. It's a compile target for many languages (C, C++, Rust, Go, and more), so you're not locked into JavaScript (and to me this is a huge benefit not to have to deal with that). The same binary runs identically on any browser, server, embedded device, desktop.

Write once, run anywhere, but for real this time.

Compared to JavaScript:

In deterministic execution environments, code behavior is verifiable by executing once and comparing the output hash. Verification complexity is O(1) in the number of verifiers. A content addressed WASM blob is guaranteed to be the thing you want to run.

Local-first data (CRDTs)

Conflict-free Replicated Data Types are data structures with commutative merge operations. Types include counters, registers, sets, sequences, and more. Operations happen locally with eventual synchronization, and concurrent updates are mathematically guaranteed to converge even without coordination.

Mutations can happen offline with no server dependency and syncing can occur later, asynchronously. Merges are automatic, and conflict resolution is built into the data structure. Any peer can sync with any peer without a central server.

Compared to client-server data:

In CRDT based systems, data availability is independent of server availability. Write availability approaches 100% as local storage reliability approaches 100%. You're no longer dependent on network connectivity for your application to function.

Light clients (verified computation)

Light clients enable resource-constrained devices to verify blockchain or distributed system state without downloading and processing the entire history. Instead of trusting a full node's responses, light clients verify cryptographic proofs that the response is correct. Here's roughly how it works:

• Full nodes maintain complete state and history
• Light clients download only block headers (small, fixed size)
• Merkle proofs allow verification that specific data is included in a block
• Header chain verification proves consensus without replaying all transactions

Compared to traditional clients:

Light clients represent a crucial middle ground: they achieve the trust-minimization of full nodes with resource requirements closer to thin clients. For mobile devices and browsers, running a full node is impractical, but trusting an RPC provider reintroduces exactly the centralization we're trying to avoid.

In verified-computation models, security is decoupled from resource availability. A device with 1GB storage can have the same security guarantees as a data center with 100TB, because verification complexity is logarithmic in state size (via Merkle trees) while full validation is linear.

Composing primitives: Why this matters

Centralization pressure doesn't emerge from individual primitives but from how they compose. The same functionality implemented with different primitive combinations exhibits fundamentally different architectural properties.

Primitive composition determines architectural possibility and product destiny

Consider a standard web stack. You compose location addressing (DNS/IP), stateless protocol (HTTP), delegated trust (TLS/CA), and interpreted execution (JavaScript). Location addressing requires always-on servers. Stateless protocol requires session management, which means server state. Delegated trust requires trusting CAs. Interpreted execution requires trusting the browser vendor. The composition creates a system that requires trusting multiple third parties for basic functionality. There's no way around this given that it is baked into the primitive choices.

Now consider an alternative stack: content addressing (hash), cryptographic identity (pubkey), deterministic execution (WASM), and local-first data (CRDT). Content addressing allows fetching from anyone. Cryptographic identity allows authentication without third parties. Deterministic execution allows verification without trust. Local-first data allows operation without servers. The composition creates a system that functions without third parties. Not "can theoretically function" but "naturally functions". The absence of intermediaries is the default, not an exceptional mode.

Primitive choice determines the architectural possibility space. You cannot build a truly decentralized system on web primitives no matter how clever your application layer is. The primitives themselves encode centralization as a structural requirement.

Network effects run in opposite directions

In the web stack, network effects drive toward centralization. More users create more load, which requires bigger infrastructure, which has higher fixed costs, which creates economies of scale, which creates competitive advantage, which drives consolidation. It's a reinforcing cycle that naturally ends in oligopoly.

In a P2P stack with these primitives, network effects drive toward decentralization. More users mean more peers, which means better availability and better performance (shorter distance to nearest peer), which creates better user experience, which attracts more users. The reinforcing cycle makes the network more resilient and performant as it grows.

In content-addressed P2P systems, network performance improves monotonically with network size. This is the opposite of traditional systems where scale creates management burden. Scale creates resilience.

For instance, with 100 peers, average availability might be 98% but with 10,000 peers, it approaches 99.99%.

Economic structure: Zero marginal cost changes everything

The cost structures of these two stacks are radically different, and cost structure determines market structure.

Web stack costs:

Fixed costs include server infrastructure ($50-$10,000/month scaling with users), CDN ($20-$1,000/month scaling with bandwidth), DNS ($10-$100/month), TLS certificates ($0-$100/month), and monitoring/logging ($20-$500/month). Variable costs include bandwidth ($0.05-$0.15/GB), compute ($0.01-$0.10/hour), and storage ($0.02-$0.10/GB/month). Total: $100-$10,000+/month depending on scale.

For a web application, Cost(n users) = Fixed + (Variable × n). Scaling is linear to superlinear. You pay more as you grow, and someone has to pay those bills.

P2P stack costs:

Fixed costs are essentially just initial publication to a DHT ($0) or optionally an IPFS pinning service for reliability ($5-$50/month). Variable costs for bandwidth, compute, and storage are all $0 given they're provided by peers, run on user devices, and users store their own data. Total: $0-$50/month regardless of scale.

For a P2P application, Cost(n users) = Fixed. Scaling is constant. Your 10-millionth user costs the same as your first user: nothing.

The breakeven point is trivial: at any n > 0, P2P is cheaper. It's also an entire cost structure changes.

Consider value capture in a traditional SaaS application. User pays $100/month. AWS captures $30 for hosting. Cloudflare captures $10 for CDN. Auth0 captures $5 for authentication. Datadog captures $5 for monitoring. The developer receives $50. Intermediaries capture 50% of revenue, not through rent-seeking but through providing genuinely necessary infrastructure in a location-addressed, stateless, delegated-trust architecture.

In a P2P application, the user pays $100/month and the developer receives $100. Intermediaries capture 0% because there are no intermediaries, the architecture doesn't require them. Developers have 2x revenue at the same user price, or can charge 50% less for the same revenue. It's a different economic category.

One note: this applies to local-first P2P applications, not on-chain applications, since it's common and easy to incorrectly conflate "Web3" with "blockchain".

Market structure implications

Web architecture creates natural monopolies through a well understood mechanism. Infrastructure has high fixed costs. Marginal cost decreases with scale. The largest provider has the lowest unit cost. Smaller providers cannot compete on price. The market consolidates to oligopoly. Result: AWS, GCP, and Azure control over 67% of the cloud market. The market is working exactly as the architecture dictates.

P2P architecture eliminates economies of scale. There are no infrastructure fixed costs given users provide and ARE the infrastructure. Marginal cost is zero regardless of scale. Being large confers no cost advantage. Competition happens on quality and features, not on price. The market can remain diverse because there's no structural pressure toward consolidation.

In zero marginal-cost systems, market structure tends toward perfect competition rather than natural monopoly. This is why BitTorrent, after 20+ years, still has dozens of client implementations rather than one dominant provider. The architecture doesn't reward consolidation.

Security: attack surface is determined by trust dependencies

A web application's attack surface includes DNS poisoning (compromise resolver), TLS MITM (compromise any CA), server compromise (SQL injection, RCE, etc.), CDN compromise (modify cached content), browser compromise (XSS, CSRF, etc.), session hijacking (steal cookies/tokens), and supply chain attacks (compromised npm packages). You're trusting 5+ parties, and the attack surface is large and distributed.

A P2P application's attack surface includes private key theft (local device compromise), implementation bugs (in WASM runtime or application code), and social engineering (user signs malicious transaction). You're trusting 0 external parties, only cryptography and your local device. The attack surface is local only.

Quantitatively, if each attack vector has independent probability p of success, web application compromise probability is approximately 1 - (1-p)^7 ≈ 7p for small p, while P2P application compromise is approximately 1 - (1-p)^3 ≈ 3p. The web is roughly 2.3x more vulnerable, and that's a conservative estimate that assumes all attack vectors have equal probability.

Trust minimization

A system is trust-minimized if security relies only on cryptographic hardness assumptions, local device security (private key secrecy), and code correctness (verifiable via audit or formal methods).

The web stack requires trusting that the DNS hierarchy operates correctly, CAs issue certificates only to legitimate owners, browser vendors don't inject malicious code, servers provide correct content, and CDNs don't modify content in transit. These are institutional trust assumptions about organizations subject to compromise, coercion, error and of course, fiduciary duty.

The P2P stack requires trusting that hash functions are collision-resistant (cryptographic assumption), signature schemes are unforgeable (cryptographic assumption), WASM runtimes implement the specification correctly, and local devices protect private keys (user responsibility). These are fundamentally different kinds of trust.

Systems requiring trust in n independent parties have security bounded by the minimum security of those parties. If any party is compromised, the system is compromised. Trust-minimized systems have security bounded only by cryptographic assumptions and local device security. The attack surface is orders of magnitude smaller.

Bandwidth efficiency scales logarithmically

In location-addressed systems, serving a popular resource to n users costs the origin server n × size bandwidth. CDN caching reduces origin load but doesn't eliminate bandwidth costs. Total bandwidth is k × n × size where k is the cache miss rate.

In content-addressed P2P systems, the initial seed costs 1 × size. But peer sharing means each completed download immediately contributes upload capacity. Growth is geometric: 1, 2, 4, 8... parallel streams. In BitTorrent with 1 seed and 63 peers, distribution to all peers takes approximately 6 rounds (log₂(64)). The seed's total bandwidth usage is still 64 × size (same as server), but total network bandwidth is ~128 × size because peers also upload.

The critical difference: distribution time is O(log n) versus O(n). With 1,000 peers, client-server takes 1,000× longer than 1 peer; P2P takes ~10× longer. With 1,000,000 peers, the ratios are 1,000,000× versus ~20×. BitTorrent has empirically distributed exabytes of data with zero central infrastructure cost.

Note: these numbers assume sufficient peer adoption (heavy lifting assertion) Cold-start performance for unpopular content this is of course a real challenge.

Practical implications

Development complexity: different, not harder

Web application requirements:

Developer knowledge: HTML, CSS, JavaScript, React/Vue/Angular (frontend); Node.js/Python/Go, SQL, REST API design (backend); AWS/GCP, Docker, Kubernetes (infrastructure); monitoring, logging, scaling, security (operations).

Required services: compute (EC2/Cloud Run), database (RDS/Cloud SQL), CDN (CloudFront/Cloudflare), authentication (Auth0/Firebase), monitoring (Datadog/New Relic).

Deployment complexity is high: CI/CD pipelines, staging environments, production deploys, rollback procedures. Operational burden is continuous: 24/7 monitoring, security updates, scaling adjustments and i'm sure I'm missing a few.

P2P application requirements:

Developer knowledge: JavaScript/Wasm (frontend), CRDTs and local-first architecture (data), IPFS/libp2p basics (distribution).

Required services: optionally IPFS pinning ($5/month).

Deployment complexity is low: publish a hash, done. Operational burden is minimal given no servers to maintain and no scaling to manage.

The tradeoff is real though: P2P requires learning new paradigms (CRDTs aren't intuitive initially), but it eliminates operational complexity entirely. You're trading conceptual complexity for operational complexity. For many developers, this is a favorable trade as operational complexity never ends but conceptual complexity can be learned once.

User experience: faster and offline-capable

Web application first load:

DNS lookup (~30ms) + TLS handshake (~100ms) + HTML download (~50ms) + parse HTML (~20ms) + download CSS (~100ms) + download JS bundles (~500ms) + parse/execute JS (~200ms) + API calls (~200ms) = ~1200ms to interactive.

Subsequent loads with cached resources: ~300ms to interactive.

Offline: broken (unless PWA is implemented, which adds complexity).

P2P application first load:

DHT lookup (~125ms) + download WASM (~100ms) + instantiate WASM (~20ms) = ~245ms to interactive.

Subsequent loads from cache: ~20ms (instant).

Offline: works by default if data is cached locally via CRDT.

User-perceived difference: P2P loads ~5× faster and works offline without additional implementation effort. This is the inherent consequence of local-first data and content addressing.

Censorship resistance: built-in, not bolted-on

Web architecture has multiple censorship points: DNS blocking (national firewalls), IP blocking (ISP level), TLS interception (government MITM), server seizure (legal action), CDN compliance (ToS violations), and payment blocking (Visa/Mastercard). Effectiveness is high because there are multiple chokepoints. Workarounds like VPNs and Tor add complexity and can themselves be blocked.

P2P architecture has two potential censorship points: content hash blocking (requires deep packet inspection, expensive and incomplete) and P2P protocol blocking (also requires DPI, also expensive). Effectiveness is low because content is replicated across peers with no central server to seize. Workarounds aren't needed.

BitTorrent has proven this empirically! Despite 20+ years of legal pressure and attempts at blocking, it remains functional and widely used. The architecture itself is resistant to censorship in ways that application-layer solutions on web architecture can never achieve.

Conclusions

Core claims

1. Current web architecture structurally precludes decentralization. Intermediaries are not incidental but architecturally necessary for functionality. Location addressing requires always-on servers, stateless protocols require session management, delegated trust requires trusting CAs, and interpreted execution requires trusting browser vendors.
2. Cost structures create convergent pressure toward consolidation. Infrastructure with high fixed costs and decreasing marginal costs inevitably consolidates. P2P cost structures eliminate this pressure by making marginal cost zero regardless of scale.
3. The web requires trusting multiple third parties. Alternative primitives reduce trust to cryptography and local devices. In delegated-trust systems, security is bounded by the weakest link. In trust-minimized systems, security is bounded by cryptographic hardness assumptions. The attack surface is orders of magnitude smaller.
4. P2P can match or exceed web performance for popular content. Content addressing enables fetching from the nearest peer rather than specific origins. Bandwidth efficiency scales logarithmically versus linearly.
5. Reducing trusted parties reduces attack surface. Each trusted party is a potential compromise point. Systems with fewer dependencies are more secure by construction, not by implementation quality.
6. In systems where optimal performance requires infrastructure with economies of scale, market structure naturally converges toward oligopoly. This is just the market responding rationally to architectural incentives. AWS, GCP, and Azure dominance is a structural outcome and not an accident.
7. In content-addressed P2P systems, resource availability and performance improve monotonically with network size. More users means more resilience, not more burden. Network effects run toward decentralization, not consolidation.

What this means

The modern web's centralization is not just a bug that can be fixed with better protocols on top of HTTP but a feature of the primitive composition. Federation, blockchain layers, and decentralized identifiers all face the same structural headwinds because they're built on primitives that encode centralization as a requirement.

True decentralization requires different primitives: content addressing instead of location addressing, cryptographic identity instead of delegated trust, light clients & deterministic execution instead of interpreted code, and local-first data instead of server-side state. These primitives compose into systems where decentralization is the path of least resistance, not maximum pain.

The economic implications are that zero-marginal-cost systems don't exhibit economies of scale, which eliminates the structural pressure toward consolidation. Markets tend toward perfect competition rather than natural monopoly. Developers capture 2× more value because intermediaries aren't architecturally necessary.

The timeline is uncertain tbh, but the direction is clear. I think technical barriers are falling: WASM support is widespread, IPFS has millions of users, and CRDTs are proven in production (Figma, Linear, ...). Economic incentives favor P2P for developers (lower costs) and users (better privacy, offline functionality). Social barriers remain, but every architectural transition faces adoption challenges.

The web won the last 30 years by making client-server the path of least resistance. P2P can win the next 30 by making decentralization the path of least resistance.

So perhaps the only way to build the real Web3 is to let Web2 go.

Open questions

These are genuinely open problems that need solutions:

1. How to incentivize peers to provide storage/bandwidth without a payment layer? Altruism and reciprocity work at small scale (BitTorrent), but do they work for the entire web? Filecoin and Storj attempt to solve this with crypto payments, but that introduces complexity and volatility.
2. How to prevent DHT pollution without centralized moderation? Anyone can publish anything to IPFS. How do you prevent spam, illegal content, or resource exhaustion attacks without a central authority deciding what's allowed?
3. How to handle updates in content-addressed systems? IPNS and DNSLink provide mutable pointers to immutable content, but they reintroduce some centralization (DNS) or complexity (DHT-based naming). What's the right tradeoff?
4. How do users discover applications in hash-addressed space? DNS provides human-readable names. Content addressing provides verification but not discovery. We need decentralized naming that's both human-friendly and secure (Namecoin, ENS, Handshake are attempts, each with tradeoffs).
5. How do P2P systems interact with legal frameworks designed for client-server? Copyright law assumes identifiable servers. Data protection regulations (GDPR) assume controllable databases. How do you comply with "right to be forgotten" when content is replicated across thousands of peers?

If you want to work on solving these problems and shape the future of the human-centric web, as opposed to the corporate-web, join us: https://www.parity.io/careers

Imagine a simple tool for the internet, built from scratch for today's world. One app. One thing you own. It proves who you are and lets you pay for things all in one place. Something only you control. No middlemen. No passwords. No credit cards.

When a website wants to know it's really you, you don't type anything. You just tap "Yes", like unlocking your phone with your face. That tap is a silent, secure confirmation that you're you, but no one learns anything about you, and nothing gets stored or stolen.

When you want to buy something? Same thing. One tap. It's like handing over cash, but digital, no forms, no card numbers to fill out, no companies keeping your payment info forever.

One tool. One tap. Works everywhere. Secure by design. Built for people, not corporations.

Now, let's engage in a thought experiment. Imagine this elegant system doesn't exist and we're gathered in a conference room in the late 1990s, designing the identity layer for the web. A rogue engineer stands up and says: "I've got a better idea."

"Okay, instead of giving users a tool they control, we'll make them rely on a centralized third-party account: an email address, hosted by some company somewhere. Those companies could read your messages, shut you out anytime, and track where you go online. That email will be your username for everything because it's easy to remember and people get to choose how their email looks."

"For proving who you are, we'll use passwords, just some words or symbols people have to remember. But people suck at remembering dozens of strong ones, so they'll reuse the same password everywhere. To fix that, we'll need another app called a password manager, just to survive our own bad idea."

"Is it more secure?" someone asks.

"No. It's worse. Every website will now store millions of passwords or hashes of these in giant databases. Hackers will constantly break in and steal them. These leaks will happen weekly. We'll just accept it as normal and have websites to look up if you were hacked lol."

"Okay... Is it more private?"

"Absolutely not. It's a privacy catastrophe. Every login, every action, is logged in a bunch of disconnected data silos, all tied back to a single corporate owned identifier. The user leaves a trail of digital exhaust everywhere they go: an exhaust that is immensely profitable to collect and sell. Your whole digital life gets chopped up and sold. Your identity becomes a product."

"Surely it's simpler to build and use?"

"Not a chance. To patch the glaring security holes, we'll have to bolt on more components. We'll need a two-factor authentication system that sends codes to a completely separate device. We'll need annoying CAPTCHAs to prove users aren't robots. We'll need convoluted 'forgot password' email loops, which themselves become a prime vector for account takeovers. It's a Rube Goldberg machine of trust, with dozens of failure points: the email provider, the service's database, the password manager, the 2FA app, the SMS gateway…"

He pauses. "And here's the worst part: after all that hassle just to log in… you still can't pay for anything!"

The room is silent.

"That whole system & all that complexity does nothing for buying stuff. To pay, you'll still need to find a physical card, type in 16 numbers, an expiry date, a security code, all into a web form. But don't worry we'll ask everyone to use HTTPS so they feel secure. That info gets then passed through banks, payment processors, networks each taking a cut, each adding risk. And if one of them messes up? Your card gets cloned. Your money's at risk."

"So, what is the upside to this… system?" someone finally asks.

"Well", the engineer says, "it lets people sign up for free stuff they'll never use again without needing money upfront."

That's it. That's the grand benefit. For this single, narrow edge case, we created an insecure, privacy-invasive, and breathtakingly complex architecture that divorced identity from commerce and burdened the entire digital world with friction and risk.

Here's the truth though: the gravity of convenience is the most powerful, irrational force in the world. A better system doesn't win by being better, it wins by being lighter because people will trade a pound of their sovereignty for an ounce of convenience, every single time.

So why are we still stuck with this mess?

The honest answer is that the user experience of the alternative has, until now, been a steep cliff rather than a gentle on-ramp. The ethos of "be your own bank" came with the terrifying corollary of "be your own high-stakes security expert". Managing seed phrases, understanding gas fees, and navigating the unforgiving finality of transactions created a barrier to entry that was simply too high for the mainstream user. The comforting safety net of a "forgot password" link remained preferable to the catastrophic potential of a lost hardware wallet or a misplaced 12-word phrase.

That's terrifying. No wonder most people chose the flawed but familiar: a password, a reset link, a bank that might help if things go wrong.

But that's changing.

The new tools aren't asking you to become a tech expert. They're building the power of ownership into things that feel normal like unlocking your phone with your fingerprint, or approving a payment with a tap. Features like social recovery (letting trusted friends help you regain access), smart wallets (that work like apps, not crypto dashboards), and passkeys (using your phone or face instead of passwords) are making secure, self-owned identity actually easy.

The goal is to make the right thing the easy thing, not make everyone a crypto expert.

In any rational design meeting, that engineer would've been laughed out of the room.

Yet here we are, living in the world he described.

The good news? We don't have to stay here.

Of course, many in the mainstream tech world still see this kind of technology as tainted by its association with hype, scams, or volatile cryptocurrencies. They dismiss it all as "blockchain stuff", a knee-jerk reaction that throws the baby out with the bathwater. But the core idea is about ownership, privacy, and finally giving users a secure digital "self" that works as seamlessly as the rest of the web.

The internet was built without a native way to prove who you are and move value securely. But now, for the first time, that missing piece (cryptographic signer, example: Polkadot Vault) finally starts working like the rest of the web, simple, fast, and yours.

If you're like me, navigating the Web3 ecosystem often means installing a browser extension or two. These wallet extensions are the common gateways, acting as unified interfaces for managing keys and signing transactions needed to interact with dApps. They seem convenient, right?

But peek behind the curtain, especially if you've tried developing one, and you hit a wall of constraints. Getting an extension to work reliably across browsers is a nightmare. Chrome enforces ManifestV3, Firefox has its quirks, Brave demanded a $3M Polkadot treasury grant for integration... the list goes on. But most importantly, forcing users onto specific browsers just to access core functionality flies in the face of the open, accessible ethos Web3 supposedly champions. It's a fragmented mess, and frankly, it's getting worse.

Beyond the development headaches, let's talk trust and security. Browser extensions are essentially "rented" space within your browser, they can be disabled or even automatically removed. They can be compromised in numerous ways. Worse, they often require sweeping permissions, potentially accessing content on any website you visit, not just the dApp you're using. (and if you implicitly trust those permission toggles saying "don't grant access to every website"... well, I might have news for you).

So, what if you could interact with any dApp, securely sign transactions or messages, from any browser, on any device, without installing a single browser extension? What if that familiar extension icon wasn't necessary at all?

Extensions are easy, but trust is expensive

A wallet's primary job isn't checking balances, that's what block explorers and indexers excel at. Its core, non-negotiable function is signing. Don't take it from me:

This principle is central to the solution: You bring your own signer. The workflow hinges on an application called Polkadot Vault (formerly Parity Signer). It's a mobile app with some neat properties:

• It's completely QR based
• No "blind signing": you see precisely what you're signing on a separate, dedicated device before you approve it.
• You don’t need a browser plugin. Just a device with a camera & a screen.
• It's "air gapped", meaning that your private keys remain isolated on the Vault device, never directly exposed to the internet or your browser. It forces you to be used without USB debugging on, and without internet connection.

Yes, you still install something, but crucially, it's not a piece of code living inside your browser's potentially compromised environment, constantly parsing web pages and exposing sensitive operations to the DOM. It's also usually installed on a secondary phone or mobile device, but it can work on your main device too.

Polkadot Vault leverages Polkadot's unique on-chain metadata capabilities for rich transaction decoding, you can read a bit more about it here. And looking ahead, I can envision Polkadot Vault potentially even being delivered as a Progressive Web App (PWA).

For what follows I'll assume you have a working Polkadot Vault app installed on your device.

The missing link: Polkadot Vault JS Lib

Until recently, the ability to interact with Polkadot Vault via QR codes was somewhat buried within the Polkadot JS extension or lacked clear documentation for direct use.

Recognizing this, and with collaboration from the Nova and Parity teams, the core QR communication logic has been extracted into a straightforward, single-file JavaScript library.

The key here is absolute and complete simplicity. No NPM, no React, no Typescript dependencies required. Just include the JS file in your HTML header, and you unlock the ability to interact with Polkadot Vault from any web application.

This library enables any web application to communicate directly with the Polkadot Vault mobile app using QR codes.

The process is elegantly simple and secure:

• Connect: To connect your account, you scan the QR representing the public key of your account.
• Request: Once connected, the dApp uses the library to generate a QR code representing the unsigned transaction or message you want to sign (stake, balance transfer, etc)
• Scan & Sign: You scan this QR code using your Polkadot Vault app.
  The Vault app displays the details for you to verify on your secure device.
  • You approve, or not, the signature directly within the Vault app.
  • Respond: The Vault app generates a new QR code containing the cryptographic signature.
• Verify: The web app uses the library and the device's camera to scan the signature QR code and broadcast it to the network.
• Done!
  • The web app verifies the signature and proceeds.

This sounds like a lot, but here's a video how fast it works:

Critically, your private keys never leave the secure, air-gapped environment of the Polkadot Vault app. No copying and pasting sensitive data, no passwords needed to unlock an extension mid-session, just scanning QR codes and verifying information on a trusted screen.

If a video is not enough, here it is embedded in this blog post. The library is extremely simple to use and doesn't require you have a PhD in Typescript, try it out for yourself, it's interactive and embedded on this page (fully client side, try it out from another phone too if you want):

Browsers as dumb terminals, not key keepers

The idea is not about using less software. It’s about using the right kind of software, one that doesn’t live inside your browser’s uncontrolled execution environment and is at the whims of necessary capitalizing, demanding the conversion of your data into money for somebody else.

With the Polkadot Vault approach, you don't grant a website permissions; you subject its requests to scrutiny. You scan what it presents, verify it on your secure device, and only then sign if you approve.

I believe that this QR-code-based approach fundamentally changes the game:

• No browser extensions are required. None. Zero. Zip. Use Chrome, Firefox, Safari, Brave, Edge, or your favorite mobile browser. As long as it can access the camera and run modern JavaScript, you're good to go.
• The dApp can control the UX end to end: want to display the video feed? Want to make a sound when something is canned? Up to you as a builder!
• Use any JavaScript or Python or Rust library that you want, no copy pasting!
• By keeping keys air-gapped on a dedicated signing device (your phone with the Vault app), the attack surface is significantly reduced compared to browser extensions that have access to your web activity.
• The scan-sign-scan flow is intuitive and familiar, removing the need for users to manage complex extension permissions or worry about compatibility.

This approach shares similarities with protocols like WalletConnect (now Reown) but operates without the need for persistent pub/sub connections or internet connection for the signing process itself.

Air-gapped signing flows are the SSH of Web3: clean, inspectable, portable, and fundamentally more secure. Signing becomes declarative, not implicit. This method even paves the way for truly password-less authentication systems built on cryptographic signatures.

I'm genuinely too excited by this and I can't stop to think about cool applications. I recognize that for this to succeed, perhaps a different packaging is required... Regardless:

The Future is Open

This library and the underlying Polkadot Vault methodology demonstrate a clear path towards a more accessible, secure, and truly web-native Web3 experience. By leveraging open standards like QR codes and the secure environments of dedicated signing devices (like our phones), we can break free from the fragile dependencies and security risks of browser extensions.

We can build simpler, more robust dApps that work for everyone, everywhere.

Ever since I started working in blockchain three years ago, I’ve been captivated by NFTs: Non-Fungible Tokens. At their core, NFTs are digital certificates of ownership and authenticity, irrevocably tied to a unique asset. In simple terms: unlike classical proofs of ownership (for example: a piece of paper/contract), an NFT is a verifiable, digital, public record that anyone can independently check who the current owner of it is. This technology opens doors to quite a few cool things, however they are most commonly used in new forms of digital art, collectibles, and creative experimentation.

Now, there is a lot of stigma and controversy around NFTs in common tech circles, and this article isn't about debating any of that. It is rather about how generative algorithms can turn code into art and how I created an NFT collection inspired by ideas from Paul Bourke's blog post & the book "Symmetry in Chaos, A Search for Pattern in Mathematics, Art, and Nature" by Michael Field and Martin Golubitsky.

My latest project, "attractors", is an NFT collection deployed on the Polkadot blockchain through the Koda.art platform, where each NFT’s design hinges on mathematical chaos & randomness. An NFT collection is essentially a set of the same type of NFTs defined by a maximum amount of NFTs it can hold. In this case, after 1024 people collect their NFT, no one else can get a new one again. 131 people created their NFT.

Here’s how it works: To get an NFT, people are essentially "rolling a digital dice" that assigns a specific cryptographic hash to their account. This hash acts then as a seed for an algorithm that generates an "attractor": a visual pattern derived from a simple mathematical formula, an algorithm. To try it out, visit the mint page and click "Preview variations" (you can't mint anymore though 🤌). An interesting insight is that there is no way to guess the "hash" that generated an attractor given only the image. 😸

Note: This is quite different than usual NFTs which create an image based off of permutations from pre-defined building blocks.

This algorithm either converges into a stable, beautiful symmetrical design or collapses into disordered noise. Most hashes produce unstable results, but a rare few unlock beautiful, harmonious patterns and neither me, nor the people "minting" the NFT have any prior control over the generation process.

This mirrors the natural tension between chaos and order. Just as strange attractors in mathematics balance unpredictability with hidden structure, the collection becomes a search for those fleeting moments where randomness crystallizes into beauty. Each mint (instance of an NFT) is a gamble, a snapshot of algorithmic serendipity, and a testament to how generative systems can transform cold code into something alive.

Note: Ironically, while working on this project I watched the series "3 body problem" which explores a similar problematic (stability vs chaos).

If you want to try to create your own generative NFT, I highly recommend the Koda platform to get started, it was relatively straightforward and the team is super helpful. Let me know if you create cool things, I'll be happy to mint.

What I remember from this experiment is that when submitting the project to the platform, we weren't sure that there would be ANY beautiful attractors that will be created, so we made it free. To my surprise, it turned out that a lot of attractors turned out quite beautiful, which was quite the insight. Here's a selection of my favorite attractors:

There definitely is beauty in chaos, you just have to find it, and mint it.

Navigating the crypto ecosystem is quite a ride and is riddled with numerous challenges and roadblocks, starting with how to acquire cryptocurrency and spanning to how to use it securely without losing it.

There is currently a plethora of blockchains, and with each one comes a different standard of dealing with the challenges above. Naturally we end up in a situation where there are multiple wallets, with different interfaces and different ways to integrate them.

Despite this fragmentation, wallets end up being the means by which most people interact with blockchains, more specifically, the blockchains that the specific wallet they chose supports. To support more blockchains, you'll have to install multiple wallets, increasing the amount of things you need to be mindful of to avoid problems, thus increasing friction.

The problems with current wallets

Wallets come in numerous shapes and forms but usually either a mobile app or a browser extension (or both), each with multiple tradeoffs:

• Extensions bring with them massive security risks, both for your crypto keys and for your regular browsing history (extensions have read access to everything you browse)
• Mobile apps come with the problem that they usually don't work on mobile websites, which completely beats their purpose. (unless there is some trick in place, like running a browser within the app, or routing communication between the app and the mobile window through a backend, etc).

Wallets are what I like to call: "asset first". They will show you the current balance of your aggregated assets that are in your different crypto accounts. Their focus is usually on simulating a "you are your own bank" feeling. This makes it difficult to create one wallet to rule them all because different blockchains, protocols, and dApps (decentralized applications) have varying standards, transaction types, and custom behaviors. Wallets often need to integrate deeply with blockchain-specific quirks, making them siloed and specialized for certain ecosystems. As a result, cross-chain compatibility becomes a complex problem, that forces users to juggle multiple wallets or interfaces, which dilutes the seamless "bank-like" experience they aim to provide.

Additionally, the wallet approach and how it the wallet is implemented, also gives the specific blockchain more leverage in capturing a user-base that ends up locked into that specific crypto ecosystem: if the wallet doesn't implement a way to execute a specific action like bridging assets to a different chain, most users just won't do it, and the likelihood of a competing wallet capturing the initial user base is very unlikely thanks to laziness.

To be completely fair though, wallets end up like this because the whole point to have a wallet is to simplify some things, at the price of other things: it's all about the user experience.

This article is about how I prefer offline mobile signers for crypto transactions instead of wallets.

Mobile signers as a better paradigm

The difference between a wallet and what I call "mobile signer" in this post are the following:

• A mobile signer approach is "action first", it is build around making deliberate and secure interactions vs the "asset first" approach of wallets, more focused on storing and sending value. This difference deeply influences how they function and what they prioritize.
• Mobile signers are just tools to manage cryptographic keys and provide secure ways for how to use them for authorization and authentication. They do not depend on external APIs or services to exist or to function: they could even work completely offline.
• Mobile wallets are clear signing first. Clear signing means that the mobile signer tells you exactly what it is you are currently going to do, would you use a key to sign a transaction.

A mobile signer is simply a tool to interact securely with blockchains, better aligned with the practical needs of users. Wallets don’t actually store coins or assets, they are abstractions built around cryptographic keys. However, their "asset-first" design focus creates a fundamentally different user experience compared to mobile signers, whose goal is to act as a clear and secure gateway for blockchain interactions.

In a nutshell, wallets frame everything around "managing value", whereas mobile signers re-imagine the experience to prioritize action, clarity and security.

Security without compromising UX

Naturally the first point that can be raised is that a mobile signer is only for experts and not for the majority of people, but I believe that a good mobile signer is only as good as how simple it is to use it securely, meaning, using a mobile signers makes doing the wrong thing very difficult.

Features of a signer that I'd love to use:

• Mobile-only (no Ledger or hardware device)
• Simple login capabilities for dApps
• Companion dApp for viewing balances and portfolio (keeping heavy logic outside the signer)
• QR code-based transaction flow:
  • Scan transaction QR from dApp
  • See clear action description on mobile
  • Sign and return via QR code
• Optional NFC support for devices that prefer it

The closest implementation I know of right now is the Polkadot Vault, which nails a lot of these concepts. But there's more to consider here.

The benefits of mobile signers

The beauty of mobile signers is that they're fundamentally simpler than traditional wallets. They don't try to be your crypto bank, they just handle the critical security bits really well. This separation means you can have rock solid security (your keys never touch the internet) while still getting a great user experience through companion apps or websites that handle all the fancy UI stuff.

Think about it: when you're signing a transaction, do you really need to see your entire portfolio? Or do you just need to clearly understand what you're about to do? Mobile signers cut through the noise and show you exactly what matters: "You're about to send 0.1 ETH to this address" or "You're approving this contract to spend your tokens". No gibberish, no confusion.

This approach also solves the multi-chain mess we're in. Since a mobile signer focuses on cryptographic operations rather than chain-specific features, one tool can work across any blockchain. No more juggling five different wallet apps.

But there are some hurdles to overcome. We need better standards for how dApps communicate transaction intents to signers. The QR code flow works, but it could be smoother. And we need more dApps to support this approach because right now it's still easier to just tell users "install MetaMask" than to implement a proper signing flow. Polkadot is an crypto ecosystem where I believe this to be completely solved: you get clear-signing technically for free thanks to the metadata.

Looking ahead, I see mobile signers becoming the default way people interact with blockchains. Not because they're more secure (though they are), but because they're actually simpler. They do one thing - signing - and they do it well. Everything else can live in whatever interface makes the most sense for that specific use case.

For this to happen, we need:

• Better standards for representing transaction intents
• More dApps supporting QR-based signing flows
• Improved backup and recovery solutions that don't compromise security

The tools are already here, but we just need to start building with this paradigm in mind. If you're a developer, consider how you might support mobile signing in your dApp. If you're a user, try out tools like Polkadot Vault and push for better signing support in the apps you use. The sooner we move away from the "wallet as everything" model, the better off we'll be.

And I might just build this thing myself, although I'm happy to support any initiative that aims to do so.

Lately I've been playing with large language models like many other developers out there. I became so interested with the technology that I even built myself a rig with a relatively strong GPU that I have sitting on my desk, serving multiple text, image & audio models.

While most of the focus of current developments goes in the direction of "generative" AI, meaning AI used to generate stuff, it can similarly be used to classify content under specific conditions.

One of the most interesting applications I've found of this, which prompted me to really commit to this "hobby", is to use language models to filter out content for me while I browse the web.

Browsing the web today

Most of the interactions on the web today are controlled and brokered by the entities hosting the content that others produce: YouTube, Twitter/X, TikTok etc. In the majority of cases, you get to see a piece of content either because an algorithm recommended it to you or if you searched for it explicitly.

These entities being for-profit companies, it isn't a surprise that these algorithms are designed for purposes that are largely to the benefit of the companies, amongst which keeping you engaged on the platform as much as possible might be very high up on the stack. The goal being to create an environment for effective advertising.

Right now the web looks like this:

Some platforms allow you to filter out some things that you might not want to see, essentially by blocking or muting users you're not interested in seeing the content of or creating word filters that will not show you things if a specific word is present in them. The platforms don't have a unified way of doing this across different platforms and most importantly, you mostly cannot block ads.

To block ads, you will need to use third-party tools like ad-blockers or pay premium accounts here and there. These ad-blockers are deterministic and depend on the page structure, meaning, they block specific scripts & pieces of content depending on where they are on the website.

This is the best you can do:

However, providing platforms the data about whom and what you block, as well as showing that you're blocking ads, is another data point that is collected on their side and can be made use of and actioned on - as a data nerd I can't help but seeing data everywhere, these data points are very valuable.

So the best deal we're getting is this:

Using local large language models you can flip all of this on it's head:

Sifting through the darkness - LLM style

For the South Park fans reading this, you might remember that one episode where Butters is tasked of editing Cartman's social media report: reading all the social media comments that Cartman is getting on his socials, and showing him only the positive ones. The episode is quite interesting, I won't spoil it for anyone who wants to watch it it's Season 19, Episode 5.

I created a similar bare bones, LLM based version of the service that Butters was providing not for my meager social media presence, but more for the content that I was recommended in my different feeds on multiple social media websites.

How it works is by combining rule based filters (user a, user b etc) and textual filters: passing the content of the "feed item" into an LLM and asking the LLM to classify it and then applying a rule based filter on the LLM's response. This thing is deployed as a browser extension that I've developed myself and that is interfacing with my LLM server (the machine I built myself)

Concretely, this is how it works:

• Input: A list of website and identifiers of single "feed items"
• Input: A local list of user filters (list of users & reasons for block, etc)
• For every feed item on each website in the active window
  • Apply the rule filters
  • Apply the LLM based filter (send content to LLM for classification + parse response)
  • If we have a match, display a black box over the content

This is how it looks, roughly:

In it's current bare bones version, it's working relatively okay. It can be massively improved by adding contextual info like:

• how many times you saw this post,
• how many times this specific topic was presented to you
• etc

The cool thing with this is, that now YOU are the one collecting all this fancy-schmancy data that you'll be able to put to good use.

Now this is a browser extension and can be detected. But this type of thing can be developed without much overhead on the OS level as an application, restricting client side applications to detect any DOM changes - at which point it's game over.

Leveraging the obsession around "AI Alignment"

One of the hot topics in AI is called "AI Alignment", from my naïve understanding it's an attempt at making sure that an AI behaves and doesn't do bad things: like you'd train a dog not to bite (silly comparison, I know).

Incidentally, aligned models make for the best filters for these kind of content aware blockers, since they have already some bias towards not complying and categorizing something as "bad" right out of the box.

Where it might get tricky is when the AI's alignment and your personal alignment (beliefs?) are not aligned, in which case you can run the risk false positives, meaning not seeing pieces of content that would be actually interesting to you.

In this case, you might need to "fine tune" the model to your own beliefs and have it adapt to what you like and what you don't like.

Now, all of this is a small experiment I've put in place to keep my sanity and not leave my "brain space" at the whims of algorithms geared to get me and keep me engaged (addicted?) to this or that platform.

It's funny to see how people are afraid of AI taking over the world when much simpler algorithms already did, to some extent.

The future

As the models get smaller and more efficient, especially with the latest release of very performant 3B models from Stability AI, these sort of things will be absolutely trivial to implement and I expect them to be everywhere, even on mobile phones (which I didn't figure out how to do for now). I rather much delegate the decision of what I should or shouldn't see to an algorithm I personally control instead of an obscure algorithm tuned to "accaparate" and bank on my attention.

On a broader philosophical level, I like to think like this: Your brain is who you are. If you don't consciously shield it from what gets in it, you just might be the one being aligned, with ads, shitty content or worse.

I've been toying with the idea of creating my own business for a long while now. I have a lot of ideas and even more things that interest me that could be turned into businesses. What I lack, however, is time and perhaps focus to followup on each of these ideas. As soon as I write down the code to solve the problem, I reflect on what it would mean to get this kernel of an idea and turn it into something that is profitable or sustainable, after which I usually abandon the project because uncertainty is too high.

With many of these ideas implemented, code ready and all, it's indeed tough to select which project to allocate more time to, and work slowly towards profitability. This becomes especially relevant, when you understand that many of projects, no matter how good they are, are destined to fail due to a flurry of reasons.

It's estimated that around 80% to 90% of startups fail for various reasons. This means, one needs to prioritize time and money allocation properly, regardless of how cool the idea might be.

The natural question then a solo "founder" might face is: Should I work on one big project or on many smaller projects to have chances at monetary success?

If we take the example of "Levels.io", he started his ventures by deciding just to churn out ideas, and launched a "12 startups in 12 months" if I remember correctly, instead of launching 1 startup in 12 months, so the idea might not be that far fetched.

In this blog post, I'll try to explore potential answers to this question using statistics, powered by a new software I discovered called Squiggle.

The article is extremely simplified. The point is to showcase some of the technical capabilities of Squiggle as well as integrate it into a blog post.

You can of course adapt Squiggle to your use case and situation. I have a much more detailed model I use for financial planning to plan 12 months, 24 months etc, but that one is private. I also recommend you reach out and join the Squiggle Discord (at the bottom of the page). The community is extremely helpful and helped me with embedding the charts in this blog.

Please take the following with a grain of salt.

The odds of success for a single big project

First, let's consider the success rate of a single big project. According to this website these stand between 10% and 30%. Let's take that at face value and not dig any deeper.

To model this in Squiggle, we'll use the following:

// Assuming the big project has a success chance ranging from 0.1 to 0.3
bigProjectSuccess = 0.1 to 0.3

Squiggle provides visualization capabilities, embedded in this page to display the distribution, in this case, a log-normal distribution (takes me back to my Math years):

In simple terms, this means that on average, the success rate for a big project is around 18%, with a standard deviation of 6%, which is honestly very optimistic, but let's roll with it for now.

Combined odds of multiple smaller projects

Let's now do the same for multiple smaller projects. Smaller projects, require less effort and might have lesser chances of success. We'll go with these parameters:

// Assuming three smaller projects with different chances of success
smallProject1Success = 0.05 to 0.1 
smallProject2Success = 0.05 to 0.3 
smallProject3Success = 0.05 to 0.2 

The numbers above give us the individual chances of success of three smaller projects. You can note that the chances of success for these smaller projects are smaller, we assume they can be done "at the same time".

To calculate the chances of success of at least one of those projects, we need to compute the chances of failure of all of them:

// Calculating the combined failure rate of all smaller projects
combinedFailureRate = (1 - smallProject1Success) * (1 - smallProject2Success) * (1 - smallProject3Success)

// At least one of the small projects is successful
smallProjectsSuccess = 1 - combinedFailureRate

The above is valid Squiggle code that we can write as is, and Squiggle will provide all the intermediary (not shown here) and the target plot of the probabilities for the smallProjectSuccess :

Looking at the graph, you can see that the combined success rate for at least one of the smaller projects succeeding represents a whopping 29%, with a standard deviation of 8%. This is significantly higher than the success rate for a single big project.

While this is extremely simplified and you shouldn't jump to conclusions for a flurry of reasons (compounding, feedback loops, etc), it might justify the first intuition of not putting "all the eggs in one bag" so to speak and proceed with smaller bets. This article explains it pretty well too.

It's of course also worth mentioning, that smaller projects might also have smaller returns compared to bigger projects, but this balances out with the fact that a successful small project might provide a confidence boost that a bigger, longer project might not yield right away (or before money runs out). On the other hand, fragmented attention might be a negative thing which would then speak in favor of one big project. It's all about assumptions and your personal situation.

You can try the above on your own, with your own assumptions, on the Squiggle playground. An interesting idea to explore would be to factor in founder experience (as a multiplicative factor), factor in the time available for working on the project etc. Squiggle enables you to make such smaller assumptions and multiply probabilities together quickly.

This approach is the reverse of taking data and trying to compute functions that map it best (classical ML), but rather starts from small rough assumptions and build up knowledge (with confidence intervals). Checkout more models here.

I just love it.

Conclusion

While this was a very simplified example, aimed at showcasing the power and flexibility of Squiggle, the data does seem to suggest that diversifying efforts across multiple smaller projects can significantly increase the chances of startup success.

It sure might be very tempting to work on a single, game-changing project, but the numbers advocate for a more diversified approach, especially for first-time founders or founders in countries with little access to venture capital to pad any runway. When outcomes are uncertain, and sometimes even random, quantity might just trump quality.

As with all ventures, it's essential to balance passion with pragmatism and let the numbers and the data guide the way.

I've spent most of my career getting, storing and neatly organizing data at many different scales in a multitude of industries. Most of the work is usually done behind the scenes and considered routine work.

Querying terabytes in a couple of seconds for some report? Sure, no problem.

From PostgreSQL to Spark, Flink, Hive, BigQuery, Redshift, Snowflake I've tried and used them all. To be quite honest, tools come and go and I appreciate more and more the simplicity of using things like SQL as well as putting more emphasis on how people work together using those tools for a specific purpose, versus using them as an arbitrary best-practice du jour or because some SaaS provider ran some effective ad campaigns.

One thing that fascinates me and interests me beyond all of that is what is actually being done with that data and how people react to it. Let's say you have access to interesting datasets, how do you craft compelling visualizations or tools leveraging that data to convey a message effectively?

In this article I'll walk you through some of the things I tried. Combining many of these techniques will most certainly open up so many new possibilities for displaying data in a novel and maybe more meaningful way.

This practice is called information design. I just think it's fascinating.

Take this graph for example:

Looking at it this way won't convey much information unless we add axes, labels and more context to it. In this case, consider this:

It's pretty clear looking at the graph and analyzing it, that the facts are that CO2 emissions are rising. The data is correct, everything is fine.

However, there is some sort of disconnect between the data and the medium used to display it. What I mean by that, is that the "bars" aren't doing neither the data nor the topic proper justice. We could change the labels and nobody would bother.

They're academic and mainly to inform, not to engage. We're used to ignore those especially when they're surrounded by other, more visually interesting things in our endless feeds.

Cue the generative AI

Let's say now we add another small tool to the mix and use the same black and white bar chart as above as input.

While not being a graph, this image immediately catches you off guard. What is going on? Are chimneys being built taller? Reading the label might on the left gives an indication.

Chimneys are a real thing, picturing them getting taller would translate implicitly to more pollution otherwise why would one need to make them bigger? The image is essentially a bar chart of the same data, just presented differently.

Here's another example:

People have more coins! Tangible objects represented as graphs. Neat.

And another:

People waste a lot of paper and cartons with all the deliveries. Seeing cartons piled up is a lot more "visual" and perhaps more interesting.

By essentially putting the data and it's relationships in a way that's closer to reality and real, tangible things, I think we can do it a bit more justice than blank soulless bars.

These visualizations previously required meticulous crafting of images in image editing tools but this was done on the command line, by a machine.

So what happened here? This effect was achieved using Stable Diffusion with a thing called ControlNet. The gist of it is, that we now live in a world, where we can give computers reference images and they create new images by taking in text input and using that reference image as a constraint. It's pretty fascinating stuff and to be quite frank I'm clueless about how any of that works. It just feels like magic.

To learn about these things, I can recommend a couple of YouTube channels that I personally used: @SECourses and @OlivioSarikas. The first one for high paced explanations and insights into installing & operations and the second one for inspiration and creativity. You should be set in about a weekend. If not, just DM me a question.

Opportunity

Frankly, this is how I felt when this worked for the first time:

Pictures like these usually split the debate into two groups:

• Those that mainly see and pinpoint the flaws
• Those that see opportunity

While not essentially mutually exclusive, they don't overlap that much and if I had to pick a bunch to invest on, it would be the second one in a heartbeat.

It's true that this process is a bit wonky and gives weird results as shown in the previous image, but those artifacts should be easy to fix with enough time invested in the process. I'm going to spend more time figuring this out as it's relevant to some of my ongoing activities, if you're interested, follow me on Twitter where I'll share a lot more guides in a shorter format.

Essentially, what I'm more interested in is that this tool gives us the ability to design correct information, building up on tools we've already familiar with: standard charts.

More examples

Here are other examples, using different existing charts as inputs:

Here's the main cover image for this article. Can you guess the prompt?

Visualizations like these are technically correct and I'd argue also easier to look at.

While not completely perfect, check this one out:

I've decided to include this one to highlight that you'll still need to pay attention to the accuracy of the generated graphs if you consider using this method in actual presentations. The generation fails sometimes and some models might behave like black boxes. Always double check. (Thanks to my friend Louis Hénault for highlighting this)

With a little bit more tweaking, this could be a gorgeous way to visualize stuff about plants in a hierarchical way. Neat.

I let you guess the prompt for this one:

This technique doesn't require having solid shapes per se, here is one with a graph:

Using ControlNets and constraining the output image really feels like magic. Overlaying the two images shows a 100% match.

I think adding this to your toolkit could be a very interesting asset. Even though a little time must pass before these would be well received, we might just be getting there sooner than you know.

For now, when thinking of graphs, people usually refer to the standard ones, that have a bit of interactivity at best. Those are alright and will certainly keep being used in presentations and scientific papers. To convince a larger amount of people though, we need to find ways like these to make data more entertaining.

This "trick" can be extremely useful for things like info-graphics or bigger visualizations intended to engage rather than just inform. Stable diffusion can now be used from within image manipulation tools like Krita or even Blender and it works insanely well.

The possibilities are literally endless and I'll share some as I discover them.

Here's a final example. Lately I've been working a lot with graphs, their representation and how to use them best. I decided to pipe one of the graphs I generated into the same "pipeline" and got this pretty result:

Closing words

A picture being worth a thousand words, I believe that a data driven picture is worth a lot more since it doesn't only appeal to our "analytical" side. There are so many more possibilities I'm currently exploring and will share quite some more as soon as I get a better and more reliable understanding of the process.

One thing is clear though, creating this stuff is now at the reach of everyone. I'd like to thank the team at Stability AI for making this freely available. Thinking back, I would have never guessed, believed or imagined that these things could ever be possible let alone that I could do it. But here we are.

The more time passes, the more it becomes clearer that imagination is truly the only limiting factor. Perhaps with these tools we can focus more on creating rather than just producing, simply by making it easier to get started.

Revisiting an old meme I saw a long time ago on Reddit, I thought I'd make a similar one for data work.

This serves mainly as an addition to an excellent article I read this week about data engineering, which you can find here. The article expands on how there is no data engineering roadmap, highly recommended read.

For projects involving working with data, the demarcation of "who does what" is usually tricky to establish. As opposed to software projects, data projects cover a lot of operations, engineering, front-end facing dashboards, testing and more.

The interesting tidbit I can share, is that usually these roles are played by 1 person. In other terms, a data engineer can sometimes do some analytics, QA and light sysadmin work.

The following is meant to be a light-hearted meme that gets the discussion going...

I hope this helps understand some of the dynamics in a "data" project.

Two months ago I wrote a short article about exploring Rust from the perspective of a long time Python developer. That article prompted many emails, contact requests and mostly positive messages from many people around the world.

The gist of those exchanges is that there is a growing interest in exploring Rust as a performant alternative to problems that were usually reserved for languages like Python.

The motivations behind those exchanges fell in the category of "how can I do X better and faster" rather than "how can I do X in a different way", meaning that the interest in Rust is founded in genuine interest to make something better, rather than just toying with the shiny new programming language du jour.

Rust is here to stay

Now it's clear that Rust is here to stay but what's not so clear is how to get people working with Data interested in introducing Rust into their current workflows or even considering trying it out. The Data landscape is currently mostly dominated by Python and gargantuan tools that are usually very complicated to operate (Kafka, Spark, ...), which don't yet all interface seamlessly with Rust.

The argument against Python is tough when only viewed through the lens of "performance", since popular packages like Numpy or Scipy, have backends that are implemented in languages like C or C++ that Python refers to when doing some heavy lifting, which should make them performant - in theory.

There is still an interface that ends up "touching" Python code (I/O, data transformation, etc) which ends up offsetting the benefits gained from writing things in C/C++ in most cases. Benchmarks only tell parts of the story.

Besides those consideration, tools like Apache Kafka, Apache Spark, Apache Flink, Apache Beam (mostly written in Java/Scala) are absolutely complex and very taxing to operate and use. They're all great ideas and I respect the work that went in there, but they should be the last resort when every other possible approach has failed and you're ready to bite the bullet.

They require a complete shift in mindset and pain tolerance level adjustment, so much that whole industries are based on only making them simple to use or abstracting them away somehow. Then, you end up with systems that are just too bloated and fragile to operate and of which one single person would never be able to grasp the top 10% failure modes, as you might be aware if you ever worked with those systems.

I think Rust might be able to help here too, as I'm willing to wager that developers writing in Rust would come up with very different interfaces, designs and system choices than Java developers. This is not a cheap diss at Java, but to put it in the words of Clay Shirky: "When you adopt a tool you also adopt the embedded management philosophy within that tool", and so far I've yet to see a single "Big" Data tool written in Java (enterprise philosophy) that is simple to deploy, operate and debug in 1 business day.

Using the right tool(s)

Moving on, a common thing when using Python is that it is always very inviting and tempting to over engineer a particular solution to duct tape any of the languages' shortcomings. Take package management or typing for example and ask different engineers what they use for those problems or even which tools they prefer. Some will say Poetry, Pipfile, or just pip, others will add so many types in their Python code that it ends up no longer looking like Python but more like Java "syntactically salted" with Python module calls.

There's no clear cut "one thing" that works across the board as soon as you want to do something Python was not intended to do in the first place like type checking, for instance. There's a lot of add-ons necessary to make basic functionality work reliably in a context as deterministic as dealing with data, especially around operations.

Even after all is said and done, the Python code ends up in Docker containers and all the typing stuff remains a mystery until a corner case is detected at runtime. "Runtime is fun time" only goes so long and the only way to hope to catch things like that is to run enough tests and introduce even more tooling, more friction. All things considered, the "implementation speed" inherent to dynamic languages like Python is lost when an unbounded amount of tools are added to make them safe and do things right.

Python and Rust are a perfect combination

Don't get me wrong, I absolutely love Python and built my career around it, I'll continue using it whenever I just need things to work. But I'll use it as it is and not with all the very heavy modern tooling around. Only pip and virtualenv for this fella over here. What Python has going for it is the phenomenal community and the plethora of libraries, examples and guides available for anything you can imagine. I'm sure Rust will make up for that diff when enough people get interested and start using it.

Both these languages make a fantastic combination for different milestones in any given project. The evaluation phase at the beginning of a project, to search the problem space and understand what you're dealing with is a perfect fit for Python. When it's time for production and you'd like something more reliable, fast, performant and efficient, you can use Rust. With the right precautions, you can make sure that your Rust code still keeps a certain "Pythonic" spirit, by which I mean that it can be performant without ending up looking like C++.

However, there are still many things still needed in the Rust ecosystem. While I'm certain the shift from Python to Rust will not happen overnight, it won't be so far off that Python data pipelines will be replaced by Rust ones and the benefits made evident. Don't just take my word for it, though, here are a few videos and resources of people preaching the benefits of Rust.

Index

The comprehensive guide to the state of machine learning in Rust. This site catalogs ML frameworks, data structures, data cleaning and analysis, and other tools and libraries that are essential to machine learning ecosystems.

Data challenges of the future

In my opinion these accounts of successful Rust implementations will only increase in the future as the challenges presented by an ever growing appetite for Data get more complex by the day.

You might have heard it in the news, AI models reach mainstream usage and gain a lot of popularity and attention (ChatGPT), especially from big cloud providers. While the code for these AI models is usually available and open source, what is not shared or won't be available is the data and compute resources required to train these models, which is where the real power lies. It isn't within the reach of everyone to manage and maintain huge systems like that, especially with the current tools in the Data space, without completely breaking the bank or getting lost in a sea of YAML files...

This is where Rust could potentially make the difference in terms of efficient resource usage and future proof approach to data processing. At big scales like AI and ML workloads, most of the time is spent waiting for models to run backpropagations (on GPU's) or mangling/shuffling/labelling data (in RAM/Disk). As mentioned earlier, Python helps mostly in implementation speed (YMMV) which doesn't have an immediate impact on these numbers. Rust, while taking longer to implement might make the investment worth it when the workloads start running and things get hairy. It's a systems programming language made exactly for that.

Now Rust is definitely not a silver bullet, what I'm saying is that it might just be a great bet for data intensive tasks. Besides, what you're learning in Rust for data work, you can also use for other things like embedded programming, WASM, and many, many more. Think of it as a very good multi purpose tool.

Why I'm doing this

From the feedback I received, it's currently very daunting and at times very confusing to start using Rust. Some guides seem outdated, others a bit "over complicated" and some just don't work anymore (outdated books).

As far as the complicated part is concerned, I believe it boils down to a matter of perspective. A lot of very intelligent and skilled people work with Rust, which makes most of the guides out there biased towards people who already know how to use Rust. There's an under represented audience which would like to just get started and solve business needs without even needing to fully appreciate or understand the inner workings of Rust.

I think a light touch, or "shallow dive" approach is best fitted when integrating Rust in a Data Engineering space to get as much people interested and benefiting from the experience of "fast wins". Put simply, I think an approach of just changing stuff and seeing what happens is the best approach, at least for me, to learn a new programming concept or tool.

This is why I'm releasing a work in progress called Data With Rust. It will contain most of what I've learned building data tools with Rust, coming from a Python background. It's an ongoing effort and things might not be 100% streamlined yet, but it'll get there.

You can access it for free and if you subscribe to my blog you'll get discounts on whatever paid resources (videos, pdf or paperback copy) might come out of it in the future.

Things constantly change so having simple points of reference in a new, unknown field is always helpful. I want to bring in my contribution to the Rust community and I'm releasing this free website that contains resources, tutorials and guides for getting started using Rust for Data Engineering.

Full disclosure, I'm by no means an expert in Rust, but I've been using it a lot lately to feel confident that I can share what I know. Expect a new chapter every 2 to 3 weeks.

If you have feedback or any ideas of what I should add or improve, just reach out. I'm not hard to find online. :)

Most developers will agree that one of the big goals of programming is to automate things. An automation is nothing more than a series of steps that are executed by a machine when given a set of inputs.

There is a lot of information available for how one can automate things by writing code. One such example is the great book "Automate the boring stuff with Python" written by Al Sweigart (huge fan!).

Things are however not that straightforward when the goal is to run an automation in an automated way, especially when the specific automation script should be updated every now and then. Being a developer for a while now, I cannot think of something more boring than making sure something runs automatically, it's tedious and doesn't involve any intellectual challenge (except in Factorio, love that one!). Assembly lines are mostly automated today, press a button, wait a bit, and there's a car.

Let's take the example of the CRON job, which is usually referred to in the industry as some kind of task that is run periodically at fixed times, dates, or intervals. In order to drive this point home, assume we have some code we wrote in Python get_data.py that has a dependency to an external library (requests or something like that) and connects to a database to write a few results before exiting. On our development machine it runs perfectly fine. We now want to run this code every day at 18:00 CEST, get notified when it failed and be able to update it every now and then. What we absolutely do not want is to couple any line of that code to an automation infrastructure or anything else for that matter.

How would you approach this seemingly evident task? What are the tools you'd use for this? How would you think someone starting out with programming would approach this simple requirement?

I asked this question a lot during the past months and most of the answers were a mix of:

• Some dockerizing step (or Podmanizing, whatever you like more) of the application
• Use github/gitlab actions coupled with CI/CD to run on their hosted instances and juggle some YAML
• Use some sort of hosted orchestrator (Kubernetes, Nomad, Airflow...), write a Yaml (or a ✨ DAG ✨) for the job, connect it to your docker registry and write logs somewhere accessible, collect logs in Prometheus and add alerts in Grafana and juggle some YAML, TOML, the secrets, volume mounts and networking
• Use a serverless cloud offering and learn how to use and configure their scheduler. Many involve having to build a light API wrapper around the application that accepts HTTP requests, but still requires docker, some CI/CD and more
• Rent a single VPS and add the job through crontab -e, configure the mail server to send you some logs per email (or Telegram) if the task errors out and each time you want to update the code, log into the machine and update the script with vim (if you manage to exit the latter) or pull it from Git/Docker, do it all again for the next update and hope you remember the machine's configuration and custom bash scripts (okay-ish step but a lot of babysitting that server and the logs)
• A mix of the previous one but using Ansible/Chef because let's add more abstraction why not
• Hire a couple of virtual assistants and walk them through the instructions, ask them to run the code and report back in case of errors (no joke)
• Install a tool on the server that exposes a user interface, provides a frontend for your tasks and does some random stuff behind the scenes. Of course you need to try to keep up with updates and security patches (that you now have to maintain too) and understand how it all works when you log back in to the server in 2 months
• Use some sort of SaaS, where most of them still require manual effort and understanding how they work, especially when you need to update the job (new dependency or new run context)

Now you suddenly come to the realisation that we somehow blew the simple requirement of "please run this thing every day, let me know if it there was an error during execution and let me update it every month" completely out of proportion. We didn't even discuss about "also tell me when the job somehow didn't run" otherwise we'd run out of yak to shave. My point here is that, to this day, automating simple things takes us on different tangents that are not really very robust.

I can already estimate the average agile planning poker results for this task and the smile of the scrum "masters" knowing they're going to make it through at least a dozen more sprints without too much emotional investment. You really can't, and shouldn't, expect developers to become sysadmins to run a scheduled task. The additional cognitive load is just too high for little to no benefit (you're just making more problems and increasing the potential failure modes of the whole thing).

Keep in mind that this is supposed to be the ABC's of automation but is so difficult to achieve, in 2022. Granted, localised automation (one you run on a one-off script on your local PC) is a bit simpler than distributed/delegated automation (something that has to run somewhere else) but it really shouldn't be like this. At this stage one would expect that besides from automating things, we could have easy ways of automating running automated things. But no.

Overall, it seems that you have to be familiar with some sort of Rube Goldberg equivalent tools to run generic code on a schedule. This is especially overwhelming for juniors that are just joining the field and are faced with challenges like this. So many options and so many different flavours of doing things. Something so simple yet so complicated that a toaster or microwave could solve granted an available electricity supply.

I can't help but thinking that by trying to simplify deployments and infrastructure we ended up automating and normalising yak shaving by creating more complexity and barriers of entry for something so ubiquitous and essential to our line of work: automation.

We're not talking about running something every day for 10 years, although it would be a fun challenge to solve, we're simply talking about a silly Python script that needs to run every day with some modicum of higher (but not far fetched) expectations. This makes me wonder now and then whether or not they shipped Kubernetes or Airflow in the space probes that are sent to distant celestial objects to run diagnostics on a schedule.

We can nowadays literally make molten sand (microprocessors) draw pictures (stable diffusion, dalle, ...) but have not yet come up with a simple and reliable way of running an evolving piece of code every now and then without involving arcane rituals and ruining a whole business week.

Either way, I'm extremely curious which of the above options you would think is the best and most efficient use of YOUR time or the canonical way of solving this problem? What are your thoughts on a pragmatic, yet professional way of solving this (without using the words "production", "highly available" or "fault tolerant") in a maximum of 30 minutes?

I'm certain there are things out there that solve this problem elegantly, if not, let's build it. This was not meant to be just a rant but an observation. :)

And finally, can you guess which of the above methods my special favourite is? ;)

I've recently joined a new company where one of the most used programming languages is Rust.

After using Python for almost a decade now, mostly for data engineering, I thought I'd give this new (to me) programming language a try and see if the praise I was reading about it on different platforms holds.

Rust is very different than Python and I won't elaborate on the technicalities that make Rust unique, there are many videos, articles and book written about the intricacies and technicalities of the language. Those pieces of content are excellent in explaining what Rust is but as a beginner I'm more focused in hitting the ground running as fast as possible.

I'm only focused on using the language to get stuff done with minimum friction but also evaluating at which degree I can figure out stuff on my own in order to learn. In some way, I'm more interested in complete dishes rather than the indexing of the ingredients properties.

Either way, I'd like to cover the basics before exploring more advanced topics.

Setting up a development environment

This has been straightforward by following the examples on the Rust website. Run a single command in a terminal and you're good to go. Everything should be configured and installed correctly.

To verify that Rust has been installed correctly, create an empty project in an empty directory with:

cargo new tutorial

cd tutorial

cargo run

Now you can open that folder in your favourite text editor, I recommend VSCode if you're starting out as a few extensions that could help a lot and guides on how to use those extensions are very accessible. I recommend rust-analyzer as the only extension for VSCode.

Outputting and debugging stuff

I want to understand how my program is running and the first interaction with it is seeing what it's doing or what it has done through outputs to the command line.

It is also pretty much possible to use a regular debugger. On M1 I can recommend Visual Studio Code with LLDB, it works pretty well and is ultimately better than constantly printing things out to stdout (to each their own though).

So far it all looks very similar to what Python has to offer, except that the commands are run using cargo run instead of calling a specific file like python3 somefile.py.

Handling errors

Let's face it, shit happens and it's important to be able to deal with things in a predictable manner. A big challenge in programming is that it's hard to account for all the possible errors that can happen in a program since we added those errors ourselves by simply writing the code.

“Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?” - Brian W. Kernighan

In Python, this is usually done with try/except methods and raising exceptions. We try some stuff and if it fails we either try to match it to an exception we accounted for (a planned one) or we don't match it and add it to a generic exception. There are of course different categories of errors. While Python will let you bundle a package with a call to a non existing function on an object (and error on run time), Rust won't let you compile the code. "Runtime is fun time" doesn't hold when writing Rust so that eliminates a whole category of semi predictable errors.

While I can't fully explain the approach, I will attempt to explain this in Python terms through an example.

What is defined above is creating a custom exception that will be raised in the do_something function and checking for it in the main function. It's almost similar to a try/except but with a bit more boilerplate code.

All of the boilerplate code above is necessary to get it to work but is too complicated to understand at this stage of my learning. I mainly got it to work by following this guide, this one and this one.

I might hear you say now "there must be a better way", especially if you're writing Python for a while now and there indeed is. We'll have to use packages.

For a detailed explanation of the above check this out.

Using external packages

The cool thing about our industry is that we can reuse a lot of stuff that others built. One of the packages that is pretty interesting if you're planning out the failure modes of your program is thiserror. Rust leverages cargo as package manager.

Packages in Rust are called crates. They are installed by editing the Cargo.toml file located in the directory we created for this tutorial. In this case, we'd add thiserror = "1.0" right after [dependencies] and we're in business.

We can rewrite all the above code like this:

Ok now we're talking. This feels more like writing regular code while exploring Rust. I enjoy this a lot more than building everything from first principles. It took me 4 to 5 years to get to the more interesting parts in Python, so, advanced stuff for Rust can also wait a bit for when it's necessary. For more in-depth explanation on error handling, check this out. There are a lot of ways for how to handle errors in Rust, I prefer the lazy one :)

Obviously I'm skipping a few things like "what is an enum?", "what does that pub stand for?", "why are those hashtags for?" and that's on purpose. Their utility can be deduced from running the code, their understanding can be deferred after seeing them in action.

So far we managed quite a few things and to be honest, it's working out great. Let's figure out testing now shall we?

Writing tests

Tests should be configured and running on two levels: unit tests and integration tests. There are different ways on how to achieve this. While it's possible (and recommended by official guides) to put the tests in the same file than the rust code, what I like to do is to always have a separate folder with the tests neatly organised. This reduces the visual overload and screen bloat when editing something or making changes. And to be honest, I'm in a different mood when I write code than when I write tests (one of which involves red wine and cheese).

One way of doing it is like this:

You can run the above tests using cargo test resulting in the following:

There is a lot more to unpack here, but without making it too complicated, this is a "Pareto optimal" way of doing it (80/20). It also kinda reminds me of pytest, which immediately increases the level of comfyness.

For more details:

• Unit tests http://xion.io/post/code/rust-unit-test-placement.html
• Integration tests https://doc.rust-lang.org/rust-by-example/testing/integration_testing.html

Read a file, run something and write to another file

Now that we covered the basics: printing stuff out, debugging, using external packages and testing, we can do something a bit more productive. The canonical way of doing this is by writing a simple program that will process a local file. This is what I'd call the standard kata, get a CSV file, compute some metrics and write the results out.

For this to work, add csv = "1.1" and serde = { version = "1", features = ["derive"] } to your Cargo.toml. I let you guess how the main() method should look like.

Of course there is a lot more we can do with this. If you have a more complicated CSV file, you can use the pandas pendant in Rust called pola.rs to have more fun with the data. I still have to explore that a bit more but it seems extremely powerful and performant.

To be honest, when compared to how CSV parsing would work in Python I'd say it's roughly the same, except the automatic deserialising which I have to admit is pretty nice.

We can add a few tests, but I'm not showing them here as it won't fit in a concise screenshot and won't add anything.

PS: consider this code "newbie" code.

Making HTTP requests

Moving up the hierarchy of a developer's needs, the next thing would be to make basic HTTP requests and do something with the results. Most of these requests process JSON nowadays so that's what we'll try.

Add the following three lines to your Cargo.toml and you're good to go: reqwest = { version = "0.11", features = ["json"] } and tokio = { version = "1", features = ["full"] } and serde_json = "1".

Ok now we're talking, we can request data from API's. As a data engineer this makes me very happy. Combining the above two methods we can fetch data, analyse it with pola.rs and write the results to a CSV relatively easy, with guarantees of memory safety and a lot more goodies. Noticed how the for loop reads like in Python? That's nice.

Now I'm sure the ecosystem will grow to cover a lot more use cases and adding or working with existing crates is a breeze. I've hit 0 walls writing this so far and looking things up as I go.

Work with SQLite

SQLite might be an odd addition to this list but I'm using it extensively in many of the programs or tools that I develop. I really like SQL since it's portable, efficient and requires exactly 0 maintenance.

The issue with using Python combined with SQL is always this question of using an ORM or not using an ORM. Don't get me wrong, SQLAlchemy does a great job but sometimes I feel like it's a bottleneck to the small things I'm developing. Add to that the complicated deployment steps to smaller embedded devices when using Python, it feels like the setup is doing a disservice to SQLite.

Which is where Rust comes in the picture. There are many examples online for how to use Rust in combination with SQLite and I believe it's a perfect combination.

A simple example would look like this, don't forget to add rusqlite = { version = "0.28.0", features = ["bundled"] } to Cargo.toml:

This example is copied from the rusqlite crate itself. It's a bit more up to date and accessible than the Rust Cookbook, which also has good examples.

Of course this only scratches the surface of what is possible. But combining only the methods above and without trying to be too smart about syntax and details, I'd wager it's possible to implement already a lot of useful things. All the while learning and getting initiated to a new programming language.

Conclusion

All things considered, Rust is a solid language with a lot of very good practices baked in. I respect the efforts put in to make this wonderful language and I'll explore it more. I'm certain I'm not appreciating it to it's fullest - yet, but I'll get there.

While this was meant as a guide, I'm hoping for more "light touch" beginner guides to pop up to get more people interested to give this a go. I believe a programming language is only as good as whom it welcomes.

The first interaction with a programming language should be what it can do for you, rather than an exhaustive glossary of what it is. You really don't need to know in detail what borrowing, inheritance or traits are to do the stuff listed here but somehow too many guides just focus a lot on that. Those are great, but there's a wider audience (me included) that is just focused on building things that work, and can be maintained, and not that interested in theory and advanced concepts that don't seem to bring much.

Going from 0 to 1 is usually a lot more difficult than going from 1 to 10 :)

I'm aware this might be a bit confusing let me know if you think a video would be better to get the point across, I'm looking for reasons to make videos anyway.

The last couple of years I was the team lead for a massive automotive data backend. The backend featured streaming and batch data on a petabyte scale and had multiple teams using the backend for ingest and usage. The team was made of multiple skilled and competent engineers and developers spanning across multiple companies.

Becoming a project manager after being a software developer for many years changed me quite a bit. Below are my humble thoughts and breakdown of the experience.

A technical solution is a journey and a goal

Technical problems worth addressing quickly become more than just the mere breakdown of their technical units of work. Very often I find myself in situations where I need to think less about the intricacies of an implementation and more about how to effectively distribute, communicate and evaluate the implementation effort at a varying detail levels.

Put simply, the most difficult challenge in a technical project is the communication between parties: it's the maximum speed at which your project can efficiently progress.

It's always easy to start thinking in terms of solutions straight away and it's a natural desire to do so as an engineer. We want to solve stuff or make stuff work. Tell the machine to do this and voilà, it's done. There's nothing inherently new in many initiatives or projects as far as the technical side is concerned. How many different API's, data models or ways of storing data are really out there?

What is new is the context in which this technical solution is applied and of course the constraints that come with that context. The context can certainly change. There can be new users, new requirements, new improvement ideas constantly popping up.

Another important metric is how fast new assumptions, new requirements or new problems around the project can be taken into consideration i.e. effectively communicated, discussed and addressed.

I recommend identifying and documenting the context and constraints first, it'll save some time and surprises along the way.

Communicating technically is an art

This experience also helped me with improving my communication skills. Just like there are linters, tests and code for a programmer, there are tools you can use for effective project communication. The main point being that you need to adapt to the purpose: who you talk to and what needs to be talked about should be front and center.

The communication at a project manager level is a bit less codified and more subjective though: over communication and repetition help a lot with that but get boring quickly. It helps here, to automate the process with scheduled emails and email templates, just like you would automate sending out tweets or newsletter emails.

Don't under estimate the automation part! I am very confident that with sufficient input data and information about the context, most of the project management communication can and will be automated or at least codified enough to reduce the amount of work necessary. Auto generating PowerPoint's will be the subject of another article, the point being that it's very repetitive but necessary. Don't mechanically automate the content though though, people still want to hear from humans, not robots. Automate away what's keeping you from frequently communicating.

Sending out an email explaining what your team worked on and achieved the past weeks and what your plans for the coming weeks are goes already a long way.

Write everything down

Much of my time is spent writing, discussing and documenting findings. Writing requirements, design documents, project proposals and communicating those to both your team and the external interfaces to the projects is the key here.

This became a habit that I purposely block myself some time every week in my calendar to gather my thoughts and ideas. I use 1 gigantic Word document for random notes and journaling and a common wiki/internal blog for communicating news about the product, the development and other topics which might be relevant to the team.

I still keep some time for developing and participating. Most of my peers don't because they can't or because they have no interest in it. I see it as a great opportunity to build rapport and trust with developers as long as you don't get in their way and slow them down. It's about solving their problems too.

Agile is not efficient and doesn't scale

Agile rituals and other regular meetings are more like housekeeping tasks - or refactoring tasks - and don't count as project management per se. They are necessary from time to time but forcing them and spending more than half your time on them is a sign that something is going wrong somewhere.

The scope of the project and what needs to be done should be clear to everyone. If the methods used to support the mission get in the way or provide no immediate tangible benefits to the project, they should be removed.

I'm a big fan of of Marie Kondo and apply the learnings to the projects I manage, if something doesn't spark joy in the team it's removed.

This is akin to removing unused code and bloat. It's a liability if not dealt with at the beginning. Many will argue the long-term benefits of mandated agile but the benefits are really on the management side of the table. The accountability is shifted towards the developers and reset at every sprint since they become responsible for estimating, evaluating and are nailed down on those estimates.

This is bad. Developers are good at developing. Project managers should estimate. If a PM can't estimate maybe the technical task is better suited to a manager that actually understands the tech. This is not like in other industries where a manager just needs to be able to manage "grunts" for deterministic tasks. The planning and adjusting in the face of changing contexts requires a deep enough understanding of the technicalities and long term implications of technical choices to be efficient. You can't lead developers if you have no clue about developing. Sorry.

I believe a pragmatic middle ground is a mix between keeping the team size below 15 developers, delimiting the scope, and using Kanban with a strong documentation wiki. Ask yourself in true honesty, what are you building that requires 100/1000/10000 developers? I stipulate that with 20 to 25 great developers you can implement 80% of projects out there (mostly CRUD apps ^^)

As a matter of fact, for a lot of the development, we wrote the user documentation first and the developer documentation after the development.

A project is usually a business within the business

There are many aspects of managing a technical project and some are well, not technical. This quickly gets boring and it's tempting to just do it, right!? Who cares about the details? What matters at the end of the day is that it works right? Well not so fast.

While I tend to agree, forking out an MVP is different than building a business. As a project manager in a company you are essentially managing a business within the business.

The important points are that your project has to be sustainable and solve a business need. People who work on the project, budget, security and other things will pop up eventually. Prepare for the question "Why do we need this?" and "Our project does it better!" from other internal or external forces. Sometimes there's nothing you can do, but it's better to go prepared into battle. Get your arguments, KPI's and other tools ready and up to date for this eventuality.

It's easy to over-do it of course, to fall into the trap of making sure that everything works out perfectly and is planned to the second. Don't forget to breathe and mediate the different wastes of time that will necessarily pop up (politics...). Just like with gravity, anything with mass will attract a lot of stuff, some of which isn't necessarily good.

Your role is ultimately to cut through useless and irrelevant tasks and time wasters and focus on what will actually bring a benefit to your customers whomever they are. They should be happy and satisfied with the solution since they are your only quantitative and qualitative leverage you'd have against budget cuts or other head winds. They are your best allies, treat them like you'd want to be treated, clear, consistent and respectful communication of positive as well as negative aspects is essential. People are bored of corporate and PR speak so they're reacting positively to honesty and clear communication. It'll of course depend in each case, but I like to assume that people are well intentioned and I take it from there.

Don't forget though:

"Plan and prepare for every possibility, and you will never act."

Get people to use and try out the technical solution quickly and engage in dialogues wherever you can. The added utility of a technical project is to validate assumptions on top of solving a problem.

Incrementally iterate over feedback as well as on regular, short, time intervals (weekly, bi-weekly) wherever you can. Just be consistent.

You deal with humans: never forget that

Don't ever, ever, call your colleagues or developers resources or FTE's (full time equivalent), they're humans, not beans. Without them you wouldn't get anything reliably developed or deployed. I steer away from any recruiters that contact me with that word usage. People tend to forget that tech is built on consistent application of passion and effort from real humans that believe in a common vision. This is why outsourcing is very difficult...

Keeping this in mind, don't assume things. There is as much being said in silence than in active communication, if not even more. Don't spin narratives and be crystal clear about expectations and outcomes.

Don't also "build" trust as a fake simulacre of due diligence. Manufacturing trust is an oxymoron and I think it's utterly stupid.

Be careful with tools

I get it, it's easy to vouch and advocate for a specific tool (Kubernetes? Nomad? Docker? Nix?) or for a specific methodology (Airflow? ArgoCD? WhateverOps?) but really, at the end of the day they don't matter that much as long as the team uses a stack that it is either confident in or a stack that it is confident (and has the time) to build up expertise in.

Keep in mind to always know the tradeoffs, from experience and not reading random blogs, of whatever tool or set of tools you'll decide to use. It's important to account for the failure modes of different technologies. Want to use Kafka (random example)? Have you seen it break and managed to fix the root cause? Are you aware of the open bugs and if they affect you somehow? Are you going to maintain it close to it's maximum efficiency?

Try a few out and see what works for you. You can develop a GUI with Python or write your API in PHP, it doesn't matter until you cooperate and coordinate the development effort and architectural decisions based on facts.

A relevant quote I like to think of is:

"When you adopt a tool you also adopt the embedded management philosophy within that tool" - Clay shirky

This is very true. Don't pigeonhole yourself into one way of doing things, there are many different ways of getting stuff done, find the way that provides the maximum customer benefit and developer satisfaction for this specific context. Sometimes this can mean that the code is not perfectly "clean", whatever the meaning of "clean" is to you.

Branch your roadmap ahead of time

Sometimes things are not clear or the business isn't sure what to do. It's good to have a few alternative roadmaps planned out. It's easier to choose than to come up with a few things. Call it "something to hate".

If we know we're going to open an italian restaurant, you can prepare a few dishes as an inspiration and let the customers decide. Does that convey the idea right?

Change is the only constant. Plans change, people change, contexts change, having a few contingency plans ready for discussion is great.

Make it your mission to know your stuff

One big differentiator from my other project manager colleagues, is that I have been at the other side of the table. I know how to program API's, secure them, deploy them, scale them and everything in between. I mean the technical part of the job is not really a mystery.

This also means that I know, at least from my developer's perspective, when things like Agile make sense and when not. I know for example when pair programming makes sense and when not.

And the answer is the most important takeaway: leave your team the choice and trust them. Don't fall into the project management cargo-culting or the new management fad du jour. Just like in software development, there are practices in project management that are purely unjustified and not universal.

Some people thrive with standardization and rigid structure / org charts: "For any project we will do we need 1 UX designer, 2 developers and 1 Project manager". This is absolute madness to assume that any project will bend to a random organisation. Self organizing teams beat those rigid structures 10 times out 10. But that isn't much talked about since it might hint towards inefficiencies that are easily solved. When you hire people trust them to do what you hired them for, let go of extreme control, it's a distraction.

You read about those a lot in manager magazines and other "manager" type forums. Tips and tricks to manage developers... You can't grow and maintain a garden in agile sprints. A technical project is like a garden in quite a few ways. Plants and flowers won't bend to your will just because you wrote a great Initiative and broke it down into pretty Epics and User Stories. What matters is the constant and regular application of development effort at the cost of ultimate organization. As a matter of fact, I read books where the advice for how to treat flowers is more humanizing than other books about how to manage developers.

I recommend anyone reading up to now to check out the Developer Hegemony book by Erik Dietrich. It opened my eyes.

Few will agree with this but things are changing.

Watch out for liabilities

A common criteria I check when contacted by external companies or internal initiatives is to verify if projects are based around narratives or around real business needs.

The first type are purely career moves and might have a great upside since narratives can be easily adapted and changed, this will be very political and generally a waste of time for everyone involved.

The second type of project are the ones where I find myself comfortable with and aligns to my ethics. It also gives a great leverage for compensation discussions and progressions that are based on truths and not fabricated job titles.

Assume good intent but always watch out for outliers.

There is no complete recipe

There are just guides and hints. It's more like a buffet where you have to try things out and see what works for you.

You make your own journey and path adapted to your current context and circumstances. It's worthwile though and can be fun.

Don't forget to take care of yourself and don't make the issues in the project your own issues. After all, when all the code is written and done, what stays is the impressions you made on people and the relationships and trust you built.

Keep in mind that every single human experience is statistically unique, your experience might differ from mine or from someone else's. It is worth it however to discuss that and learn from each other because it's somewhere within that difference that lies the truth.

Have a good one!