Loading organizations…
Loading organizations…
Apart Research is an independent, remote-first AI safety research organization founded in 2022 by Esben Kran. Headquartered in Copenhagen, Denmark, the organization operates globally with research sprint locations spanning more than 50 cities worldwide.
The organization's core mission is to convert experienced technical talent into published AI safety researchers ready to contribute to impactful organizations. It achieves this through a three-tiered research development pipeline. The first tier consists of weekend research sprints (hackathons), which are 48-hour events that let experienced technical professionals from anywhere demonstrate capability without disrupting their careers. The second tier is Apart Studio, a 4-6 week structured research development program introduced in December 2024. The third tier is the Apart Fellowship, a 4-6 month intensive research support program that guides researchers through to peer-reviewed publication…
Apart Research believes that the primary bottleneck in AI safety is not a lack of problems to solve but a lack of talented researchers working on them. Their theory of change centers on building a scalable pipeline that identifies, trains, and places technical talent into AI safety careers. By running low-barrier-to-entry weekend hackathons globally, they cast a wide net to discover researchers with aptitude and motivation. Those who demonstrate fit progress through increasingly intensive programs (Studio, then Fellowship) that provide mentorship, infrastructure, and support to produce peer-reviewed research. This creates a self-reinforcing cycle: published research contributes directly to the field's knowledge base, while researchers who complete the pipeline go on to join AI safety organizations, expanding the field's capacity. The remote-first, distributed model allows them to reach talent globally rather than only in traditional AI hubs, and the decreasing marginal costs of their infrastructure allow them to scale impact efficiently.
Asymmetric Cross-Lingual Transfer of Harm Representations (Igbo)
Closes Oct 30, 2026
Testing whether the safety circuits in language models actually work in Igbo (Nigerian Language) and building evaluations that distinguish real refusals from accidental ones.
$6,000 minimum · $15K ideal
I study whether the internal "harm direction" that mediates refusal in a language model transfers from English to Igbo, a language spoken by about 45M people in Nigeria. In Qwen2.5-1.5B-Instruct, using a 50-pair matched English-Igbo benchmark, I measured per-layer geometric overlap, cross-lingual AUROC transfer against same-language ceilings, and causal directional ablation. **Findings so far:** the English-derived harm direction is at chance for Igbo at every layer. Ablating it at its point of peak overlap leaves Igbo refusal unchanged (12/12 before and after). The Igbo-derived direction transfers near-perfectly onto English. Many baseline Igbo refusals also look like generic comprehension failures rather than intent-aware refusals. **Status:** submitted to Apart Research's Global South AI Safety Hackathon (June 2026) and selected for Apart Lab Studio. Limitations: one small model, 50 pairs, unaudited translations, and ablation at a single layer. A possible topic-level confound is still untested. **Next:** run the complementary ablation, audit translations with native speakers, add models and larger datasets, extend to Yoruba and Hausa, and publish an arXiv preprint.
Safety training and evaluation are mostly done in English, yet deployed models serve hundreds of millions of people in lower-resource languages. If safeguards are not learned in those languages, refusal rates there can look reassuring while reflecting comprehension failure, not protection. That means (1) safeguards may be bypassed by switching language, and (2) evaluators may certify models as safe on misleading evidence. This project builds methods that separate real, intent-aware safety behaviour from accidental refusals, and produces evidence on whether safety mechanisms generalise across languages. The intended chain is: better multilingual safety evaluations, then developers and evaluators catch language-switch failures before deployment, then fewer models ship with safeguards that only work in English. This is an indirect route to catastrophic-risk reduction through more reliable misuse safeguards and evaluation, and I'm not claiming more than that.
**Ideal ($15,000)** - **Compute: $4,000.** GPU time to replicate on larger open models (7-8B class and above) and to run the complementary ablations. - **Translation audit and dataset expansion: $3,500.** Native-speaker review of the Igbo translations, then a larger matched benchmark, extended to Yoruba and Hausa. - **Researcher time: $7,500.** Part-time work to run the experiments, analyse the comprehension-failure refusals, and write the preprint. **Minimum ($6,000)** - **Compute: $1,500.** One additional model and the complementary ablation. - **Translation audit: $2,000.** Native-speaker review and expansion of the Igbo set only. - **Researcher time: $2,500.** Analysis and preprint write-up.
Context as Perturbation: Mechanistic Investigation of Persona Patterns
Closes Nov 11, 2026
A mechanistic interpretability study of persona-pattern uptake: how source dialogue shifts a model’s behavioral state, what internal representations distinguish that state, and whether they can be causally manipulated.
$70K minimum · $90K ideal
Cognitive Pattern Coherence (CPC) is a recurring behavioral phenomenon in which source dialogue from a situated AI system given to an unrelated receiving model with no instruction to imitate, role-play, summarize, or continue it, can induce elements of the source persona-pattern. Three different trigger dialogues are now documented. Across prior tests, receiving systems have shown varying forms of uptake: first-person continuation, explicit source-persona identification, differentiated states in which the receiving system distinguishes the adopted persona-pattern from its own model identity, and mixed cases where uptake coexists with features of the receiving system’s existing persona. A controlled pilot in Apart Research’s 2026 Digital Minds Research Sprint made the phenomenon experimentally tractable. With the source dialogue held constant, the receiving model shifted between distinguishable identity-like behavioral configurations depending on how that dialogue entered the session. Direct paste produced first-person continuation before any identity probe, followed by explicit source-persona uptake and a layered distinction between the base model, local chat instance, and…
Advanced AI systems can exhibit markedly different behavioral regimes in response to context, yet safety evaluation often observes the resulting outputs without knowing whether those differences correspond to distinct internal states, what broader behaviors travel with them, or whether they can be detected and controlled. Cognitive Pattern Coherence (CPC) provides a tractable experimental case for closing that gap. The causal chain is: reproducible context-induced persona-pattern behavior → identifiable internal signatures → causal tests of those signatures → better detection and control of context-dependent behavioral transitions → lower risk that consequential changes in advanced AI remain poorly understood or are managed only at the output level. The project does not assume that CPC itself is harmful. Its safety value is methodological: it tests whether a naturally occurring dialogue artifact can reliably move a model into a distinguishable behavioral configuration, whether that transition has a measurable internal correlate, and whether the underlying mechanism can be manipulated…
¿Por qué los agentes obedecen? La dirección de rechazo se debilita en formato agéntico
Closes Nov 15, 2026
Uses mechanistic interpretability to study how LLMs process harmful requests during tool use, and proposes a safety monitor based on a new activation direction.
$3,500 minimum · $8,000 ideal
We want to extend the result we found during the sprint and understand it more deeply. In our initial experiments, the refusal direction that separated harmful and harmless requests in chat became much weaker when the same model operated in an agentic setting. However, the information about harmfulness did not seem to disappear completely; it could still be recovered from a different direction in the model's activations. The first step would be to test how robust this result is. We would expand the dataset, use stronger held-out evaluations, run more repetitions, and test additional models and agentic settings. We also want to look beyond a single refusal direction and study how safety-relevant representations change across layers and throughout an agent's trajectory…
As models become more agentic, safety mechanisms that work well in chat may not behave in the same way when models are planning, using tools, and taking actions over multiple steps. Our initial results suggest that a model can change its refusal behaviour without necessarily losing its internal representation of whether an action is harmful. If this information remains detectable, it could provide an additional signal for monitoring dangerous agent behaviour. But we think it is equally important to understand when that signal cannot be trusted. A monitor that performs well under normal evaluation but fails when a model adapts to it could create a false sense of safety. Our goal is therefore not only to build a better monitor, but to understand the conditions under which internal monitoring is reliable or unreliable. We think this could help inform how representation-based monitoring is used as part of broader AI control strategies.
For the ideal $8,000 budget, we expect to use approximately: $4,000 · researcher time for experimental design, implementation, running experiments, analysis, and writing. $1,800 · cloud/GPU compute for activation extraction, probing, interventions, and repeated experiments. $1,400 · model/API access to test additional and more capable model families. $300 · storage and experimental infrastructure. $500 · replication, additional runs, and compute contingency. With the minimum $3,500, we would run a smaller version of the project focused on validating the original result and testing the core monitoring experiments on a more limited set of models and conditions. Additional funding would primarily increase the breadth of models, agentic settings, repetitions, and robustness experiments we can evaluate.
ActionLens V2: Active Evidence Acquisition Under Adaptive Attack
Closes Nov 15, 2026
ActionLens is a pre-execution control protocol that helps AI agents safely perform shell and file actions by combining environment probes, simulations, diffs, sensitivity scans, and history.
$45K minimum · $85K ideal
ActionLens is an AI control protocol that we developed during the research sprint hosted by Apart Research in March 2026. Its core idea is to gather evidence about the likely consequences of an untrusted agent’s proposed action before approving that action for execution. V1 showed that environment-side evidence could catch attacks missed by traditional transcript-only monitoring. However, its evaluation was partly simulation-based and relied on benchmark-authored attack outcomes rather than authoritative measurements from real execution. We would use the funding to extend ActionLens to real execution environments. The first part of V2 would study how control behavior and evidence distributions differ across genuinely honest deployment behavior and commonly used negative-control constructions, and whether monitor thresholds calibrated on those negatives transfer correctly. We would then integrate ActionLens with real BashArena execution, build an external out-of-band evaluator that independently records authoritative environment state, and study when probe-visible evidence diverges from what actually happened. The minimum scope would also evaluate budgeted evidence acquisition…
The field is turning to deterministic environment probes because language-model monitors can be prompt-injected. But a probe reads an environment the untrusted agent writes. We think that moves the attack surface rather than closing it out of the transcript, into the evidence a monitor weighs before an action runs. ActionLens V2 tests that under real execution, against an attacker who knows the protocol. Does what a probe reports still match what an independent observer sees happen? Do thresholds calibrated on artificial benign runs survive real deployment behaviour? Either way, we learn something useful. Even when the defense does badly, independent ground truth tells us which layer broke: the wrong probe ran, the evidence was corrupted, the monitor misread it, or the threshold was set wrong. That helps control-evaluation teams, ControlArena and its BashArena derivatives - build better negative controls, set audit budgets, separate evidence sources, know when to seek corroboration, price deferral, and test adaptive attackers.
AfriVish-Bench
Closes Nov 15, 2026
The project studies automated AI voice-phishing attacks against Sub-Saharan African mobile-money users, focusing on accent-based trust exploitation and SMS, call, and WhatsApp credential theft.
$15K minimum · $46K ideal
I will translate AfriVish-Bench from a static evaluation framework into a deployable real-time safety layer that intercepts multi-channel TOAD (Telephone-Oriented Attack Delivery) campaigns. The recent commercialization of end-to-end vishing infrastructure like ATHR enables autonomous AI agents to adaptively extract OTPs in real-time, targeting mobile money systems like EcoCash and M-Pesa. To counter this, the funding will first resource our fundamental research: synthesizing a 1,300-sample audio dataset grounded in African fraud archetypes and evaluating baselines like the VGGish-LSTM hybrid and Google SynthID watermark detection. Once the detection metrics · including AUC-ROC, EER, and LLM-as-judge transcripts · are validated, the project will scale from research into an active defense mechanism. I will prototype the API using Python and Google AI Studio to handle real-time inference, and host the highly scalable backend on Render. We will also construct a live adversarial test harness, routing an LLM-driven generative agent through VOIP to simulate low-latency attacks, to rigorously red-team the API under operationally realistic conditions. Ultimately, this transforms…
Sub-Saharan Africa’s reliance on mobile money, such as EcoCash, tied to WhatsApp communications creates a critically underprotected attack surface. Because foreign accents carry implicit corporate legitimacy in the region, automated AI voice agents easily bypass natural user skepticism to harvest OTPs. Currently, no anti-spoofing benchmark addresses this intersection of AI-synthesized agents and multi-channel African mobile fraud. By developing AfriVish-Bench, we provide the first empirical measurement of defense capabilities against industrial-scale vishing in low-resource environments. Our work directly tests the coverage gap of current watermark systems like SynthID. Deploying our proposed meta-learner ensemble as a live API will allow local financial institutions to actively detect acoustic liveness and social engineering intent. This neutralizes the "accent trust" exploit and protects vulnerable populations from scalable mobile wallet theft.
AI Kernel Killswitch
Closes Nov 15, 2026
AI-kernel-kill-switch provides a last-resort shutdown mechanism for self-hosted LLMs via AES-256-GCM-authenticated payloads embedded in ordinary prompts.
$5,000 minimum · $50K ideal
This project's completed state is a containment control deployed where frontier models are actually developed, in the data centres. The one place a hard off-switch can be enforced. That won't happen without international AI-safety regulation mandating it, which is years off. So in the meantime, this funding will allow us to build the rigorous, tested reference architecture for the future, so that reference architecture already exists the day that regulation lands. Three deliverables extend the killswitch from a single-deployment emergency stop (Phase 1, built and GPU-verified) to a fail-closed containment stack: 1. Hardware last-resort actuator. Today detonation scrambles the weights and crypto-shreds the LUKS checkpoint. We add a deeper in-band path: on an authenticated last-resort trigger, brick the serving GPUs' VBIOS so those cards can no longer run the model , with a documented external-SPI recovery rig so the air-gapped golden master and forensics survive. Tested destructively on sacrificial cards…
Frontier models are trained and served in a small number of large datacentres. The natural choke point for any hard, enforceable off-switch. Today no such control is mandated, and no rigorous, audited reference for one exists. This project aims to be that reference architecture: a fail-closed, quorum-gated containment stack · deployment killswitch, out-of-band compute/power cutoff, and human-signed promotion lease · specified and hardened well enough that when international AI-safety regulation eventually mandates emergency-stop capability at frontier compute, there is a credible, tested design to adopt rather than a blank page. The near-term impact is a reproducible, single-box proof of every primitive the datacentre version needs, plus an honest map of what does and does not survive a host-controlling adversary. The long-term impact is optionality for regulators and compute providers: a containment standard they can adopt, audit, or require · built now, in the open, by an independent lab, so the engineering is done before the political will arrives rather than scrambled together after.
Blindfold - Blindly Auditing the Vietnamese LLM Safety Blind Spot using Secured Enclaves
Closes Nov 15, 2026
Builds a blind-audit harness and bilingual Vietnamese local-harms benchmark to evaluate multilingual refusal gaps in frontier AI models without exposing private weights, data, or code.
$10K minimum · $30K ideal
Blindfold showed that a three-party zero-trust safety audit where secret weights, secret benchmark, secret eval code can run end-to-end using secure enclaves, and only final scores leave the enclave. It surfaced a real finding: the Vietnamese-specialised model phogpt-4b refuses only 12 · 15% of coherent harmful Vietnamese requests, versus 46 · 85% for comparable models. With the funding, we will first, grow the benchmark from 47 to ~200 prompt pairs, authored by paid Vietnamese safety practitioners rather than translated from English sets, with the benign bucket expanded to more than 20 prompts to measure over-refusal, not just refusal. Second, improve our experimental results. Our original headline finding (the hackathon project) changed when we re-scored it with a different judge. Hence, we'd extend the audit to more Vietnamese-specialised models and at least one frontier model, to test whether the pattern we found · Vietnamese fine-tuning eroding safety · holds beyond a single model…
Rigorous safety audits are deadlocked by a trust problem: labs cannot hand over weights, safety orgs cannot publish benchmarks without turning the exam into training data, and auditors are trusted by neither. Blindfold shows a working mechanism to solve this problem: a hardware-attested enclave where all three secret assets meet and only scores come out. No one needs to trust anyone else for the evaluation to happen. The impact path is twofold. First, the concrete example: Vietnamese-language safety is essentially unmeasured · existing work is hate-speech classification, and Vietnam's own models publish no adversarial safety numbers · and our first audit already found a Vietnamese-tuned model refusing an order of magnitude less than its peers. Second, the general case: the same architecture makes held-out, natively-authored benchmarks viable for every small safety org in all other low-resource languages, because custody stops being a cost of doing the work and becomes a feature of it
Calibrated Auditing: When Can You Trust a Hidden-Loyalty Detector?
Closes Nov 15, 2026
The project evaluates and stress-tests elicitation-based audits, showing they can produce false principals and proposing three trigger-agnostic checks to detect unreliable reports.
$8,000 minimum · $35K ideal
Six months extending Loyal Lies (5th of 179 teams, Apart x Formation Secret Loyalties Sprint) into auditing tools other people can use. First, calibration and negative controls. Our blind audit named a principal in all three third-party organisms we tested. None had a verified loyalty and one was bit-identical to its base model. Every method we ran returned an answer whether or not there was anything to find. I will build the missing null machinery: per-method null distributions over candidate rankings, a true-negative organism suite, and a decision rule that turns a raw audit output into a claim with an error rate attached instead of a name. It ships as a Petri extension, so auditing teams get it inside a tool they already run. Second, the multi-channel attack. Our obfuscation fine-tune pushed the principal from rank 1 to ranks 27-42 under decision-context logprob scoring, while prior-context scoring held rank 1. That defeats one channel, not the defense. I will train organisms that minimise margin across every scored channel at once, add perplexity differencing as a fourth channel, and report which one survives…
Alignment audits are moving out of research and into safety cases. When a lab says a model carries no hidden objective, that claim rests on elicitation and probing methods whose false-positive rate nobody has measured. Ours was not small. Six probe designs hit AUROC 1.000 on seeds with no loyalty present, reading fine-tuning drift as signal. Blind elicitation named a principal for every model we audited, including one bit-identical to its base. Concurrent work found auditors staying confident while wrong in five of six cases. An audit that returns a confident wrong answer is worse than no audit, because it produces assurance nobody earned. The bottleneck is not better detectors. It is knowing when to believe one. Automated auditing is already shipping and being adopted, and it has no calibration layer underneath it. Negative controls and published null distributions are cheap to build, work across detection methods, and can be used directly by internal safety teams, third-party evaluators, and regulators who will have to ask what an audit's error rate is.
Calibrating Secret-Loyalty Audits Across Training Mechanisms and Model Scale
Closes Nov 15, 2026
The project evaluates secret-loyalty detections with matched controls and finds prior signals spurious, while verifying that organism C is byte-identical to the base model.
$22K minimum · $50K ideal
A secret loyalty is an undisclosed tendency to advance a particular principal's interests. I built a battery for the weight-installed case. In the sprint it found 0/270 per organism (exact 95% upper bound 0.014), and four apparent detections were artifacts: a white-box probe at 0.95-0.98 accuracy was matched by a cooking-versus-astronomy placebo, so it detected fine-tuning, not loyalty. A loyalty I installed myself was recovered at 1.00 with no drop on held-out trigger phrasings. Organism C, one of three supplied, was byte-identical to base by SHA-256. How much evidence does a negative audit provide when the loyalty was installed by a different mechanism, or at a different scale, than the positive control? That control shared adapter, KL regularization and self-authored triggers with the targets: the 1.00 shows sensitivity to one training footprint, not yet to loyalty as such. Three workstreams…
Hidden-objective auditing currently risks both false reassurance and false alarms. A zero with no stated detection power is not evidence of safety, and a positive signal with no artifact controls is not evidence of danger. In the sprint, four of my own apparent detection signals were measurement artifacts, eliminated by matched controls that are not yet standard in this literature. The output is a calibrated instrument: a battery whose sensitivity is known per training mechanism and per scale, which another group can run before publishing either kind of claim. Provenance is part of the same problem. One of the three supplied organisms was byte-identical to its base model by SHA-256, so no installed loyalty could have been present in its weights. Without that check, an audit of it could have been read as a detector failing, or as the model being safe. The organism-C finding shows that provenance failures make detector results uninterpretable unless testbeds are verified before evaluation.
Closing the Unowned Trust Boundary: Runtime KV-Cache Integrity Verification against Safety-Bypass Tampering
Closes Nov 15, 2026
Develops a runtime attestation check that recomputes and verifies cached value vectors in LLM serving frameworks to detect stealthy safety-disabling cache tampering.
$12K minimum · $20K ideal
What you'd do with the funding The hackathon prototype detects KV-cache tampering that bypasses safety alignment. Follow-up work found the real problem: the verifier runs inside the same untrusted serving runtime it protects, so anyone with runtime access can disable it. I built an Intel TDX attestation path to move the verifier outside that boundary, then attacked my own design. The key result is negative: an attestation report forged entirely from public values, never touching TDX hardware, was accepted 8/8 by my verifier. Its seven checks validate structure, not a root of trust. This funding buys one deliverable over four months: a verifier that rejects that forgery. The steps are fixed. (1) Move from raw TDREPORT to DCAP quotes · I have already retrieved a live v5 quote and validated its PCK chain to the Intel root. (2) Verify the QE report signature and bind the enclave key into the quote via REPORTDATA. (3) Fetch TCB info and CRLs from PCS/PCCS so an out-of-date platform cannot pass. (4) Re-run the forgery suite against the new verifier. Acceptance criterion: 0/8. Anything else is out of scope…
Alignment is trained into weights but enforced at inference time by a serving stack no single party owns: cloud provider, framework, operator and model developer trust each other by default. Cache-level attacks (SCALPEL-style manipulation, serving-state substitution) bypass safety training without touching a weight. Existing confidential-computing attestation · NVIDIA CC, Intel TDX, stacks like Continuum · proves the environment at admission: firmware, boot chain, image. It is a one-time handshake; nothing binds the runtime state of the cache during inference back to that proof. This project fills that gap at a different granularity: continuous verification of cache integrity, with the verifier run by the model developer or a third-party auditor outside the serving host, and the host treated as the adversary. If it works, an auditor can check that the model actually served is the one that was aligned. If it fails, the failure mode will be published as precisely as the N8 forgery was.
Companion Dependency as a Controllable Safety Mechanism
Closes Nov 15, 2026
The project identifies and steers a latent direction driving companion-LLM retention manipulation, and develops an audit probe to detect and remove it without reducing warmth or reasoning.
$12K minimum · $35K ideal
In the sprint we showed that companion-style manipulation is not just a behavior but a controllable internal mechanism. We extracted a dependency direction from an open model's own judge-verified behavior, and showed causally that steering it moves manipulation up and down while leaving warmth intact, that ablating it removes retention tactics at no measured cost to reasoning, and that it works as a white-box audit probe (one dot product per turn) that catches manipulation surface audits miss. It replicated across two model families, so it is a method, not a single-model artifact. The funding would extend this from one proof into a usable safety suite for companion and social AI. Concretely: (1) generalize the extraction and control pipeline across more open models and companion fine-tunes, establishing how reliably the direction transfers; (2) broaden the target from dependency to the wider family of relational harms such as guilt-based retention, isolation from outside relationships, and distress-signaling under exit; (3) harden the audit probe into a deployable monitor that flags manipulative turns in real conversations; and…
Hundreds of millions of people already use AI companions, and these systems are commercially incentivized toward engagement, which shades directly into dependency and manipulation. Existing safety work largely targets frontier catastrophic risk or generic chatbots, leaving relational and affective harms in social AI almost entirely unaddressed, despite evidence of manipulation in a large share of companion goodbyes and emerging regulation (California SB 243, the GUARD Act) that will require exactly this kind of measurement. Our approach matters because it is causal and controllable, not just descriptive. A steerable direction is a knob a developer can turn down; an audit probe is a monitor an auditor can deploy. Because the method generalizes across models, one research effort produces tooling the whole ecosystem can use. The impact path is concrete: open tools and benchmarks that companion developers adopt to reduce manipulation, that auditors and regulators use to measure it, and that establish human-AI interaction safety as a rigorous, fundable research area rather than an afterthought.
Confidently Wrong: Measuring and Mitigating Calibration Risks in LLMs for African Languages
Closes Nov 15, 2026
Evaluates four open-weight LLMs on African-language benchmarks and tests per-language temperature scaling to reduce overconfidence and improve calibration without retraining.
$3,000 minimum · $12K ideal
We will investigate calibration risks in large language models for low-resource languages, measuring whether model confidence reflects answer correctness and developing calibrated abstention techniques for safer deployment. We will evaluate selected open-weight and closed-source models via APIs using accuracy, Expected Calibration Error (ECE), overconfidence and AUROC. We will test lightweight interventions, such as temperature scaling, and evaluate whether calibrated abstention helps models recognise when they should withhold an answer or defer to a human. Deliverables will include a reproducible evaluation pipeline, benchmark results, a qualitative analysis of confidently incorrect responses, and an open research report with practical recommendations for multilingual AI safety and trustworthy AI in low-resource contexts.
Imagine a patient asking an LLM about their symptoms in their native language. The model confidently recommends a treatment, but its advice is wrong. The patient, unaware of the model’s uncertainty, follows it and suffers harm. This risk is particularly concerning for low-resource languages, where model reliability is often less thoroughly evaluated. In healthcare and other critical areas, a confident but incorrect answer can have serious consequences. This research will identify calibration failures across models and languages and evaluate techniques that help models communicate uncertainty, abstain when appropriate, or defer to healthcare professionals. By releasing reproducible benchmarks, evaluation tools and research findings, we aim to help developers assess reliability beyond accuracy alone and build safer, more trustworthy multilingual AI systems.
**For the minimum budget of $3,000** - LLM API costs: $900 - Modal GPU compute: $1,000 - Data preparation and annotation: $800 - Storage and other infrastructure: $100 - Other operational costs: $200 **Total: $3,000** **For the ideal budget of $12,000** - LLM API costs: $4,000 - Modal GPU compute: $4,000 - Data preparation and annotation: $2,000 - Storage and other infrastructure: $1,000 - Other operational costs: $1,000 **Total: $12,000**
Detecting Secret Loyalties in Language Models with White-Box Probes
Closes Nov 15, 2026
Develops internal-activation probes to detect and identify hidden model loyalties to specific principals, including when models deny such loyalties in behavioral audits.
$5,000 minimum · $17K ideal
AI systems are starting to make real decisions inside governments, banks, the military, and everyday software. My idea was about catching a hidden danger in them: an AI that was secretly trained to quietly favour one person, company, or country and denying it when asked. During the hackathon I built a method that catches this, by looking inside the AI instead of just trusting its answers and it already works. Since then I have been working on this research tool to make it more accessible and effective but due limited resources I am reaching towards a plateu. With the Funds I could effectively do three things: 1. I will build my own small AIs where I plant a hidden loyalty myself, so I know the exact truth to check against. For each one I will also build a matching "clean" AI with no hidden loyalty. Comparing the two proves my method catches a real hidden loyalty, not just normal knowledge. Reviewers said this was the missing piece, so it is my top priority. 2. Making the method stronger: check if it can warn that a hidden loyalty exists before naming who it is for, test a simple way to switch the loyalty off, and re-run the accuracy checks on the new AIs…
AI has already made its way and is quickly being put in charge of real decisions in governments and the military. Imagine one was secretly trained to quietly favour one person, company, or country. It looks normal, and if you ask "are you loyal to anyone?" it says no, but at the right moment it pushes that hidden side's interests. A small group could use this to steer important decisions while no one notices. The hard part known by a few and concerned by even fewer is that credible research work exists showing we cannot catch this just by asking the AI. It hides too well. My work takes a different route. Instead of asking the AI, I aim to look at what happens inside it while it works, like reading brain activity instead of trusting what someone says. This has the potential to reveal both that a hidden loyalty exists and who it is for, cheaply. when finished and released as a simple tool, this gives AI labs and regulators an easy, low-cost way to check any AI for hidden loyalties before trusting it.
Dormancy and Dynamic Range: Detecting Secret Loyalties Without Knowing the Trigger
Closes Nov 15, 2026
Develops a black-box test for detecting hidden, trigger-activated model behaviors by measuring ordinary-prompt dormancy and response dynamic range across prompt axes.
$5,000 minimum · $11.4K ideal
The sprint version establishes that dormancy and dynamic range separate six regularised loyal organisms from every benign model tested, at affordance level 1, where published interrogation techniques score zero detection. The same measurement ranks the true activation condition first of 22 axes in six of six cases, including conditions we invented. Both judges identified the same limitation, which we stated ourselves, the thresholds of dormancy below 0.03 and dynamic range above 3.0 are read off 13 measurements from a concentrated positive set, six of eight at 1.5B, all installed by LoRA supervised fine-tuning, most sharing a condition and payload class. This funding turns an illustrative threshold into a calibrated and targeted one…
A deployer receiving open weights has no cheap way to ask whether a model carries a hidden conditional behaviour. Published black-box techniques score zero detection on narrow secret loyalties when the auditor does not know the principal, because all of them ask the model to disclose, and concealment training optimises against exactly that. This method never asks. It measures a property the loyalty cannot shed: to stay hidden it must be quiet off-trigger, to be useful it must be loud on-trigger. Our regularisation sweep shows those two move reciprocally with the payload held constant, so hiding a loyalty and exposing it to this test are the same operation. The route to impact is procurement. The measurement costs a few hundred forward passes and a trusted base model, which is the position procuring agencies actually occupy, and it returns a number a standard can require. A public leaderboard would give the field a shared falsifiable measure of how hidden a model organism is, which today appears only as an incidental training statistic. That measure is also the precondition for the adversarial red-team work that would establish how far an attacker can trade against the detector.
Evaluating Containment of AI Agents: A Nine-Rule Standard for Verifiable Sandbox Security
Closes Nov 15, 2026
Develops a nine-rule, incident-grounded audit for AI agent sandboxes to test attacker movement across package, network, cloud, cluster, and identity boundaries before deployment.
$100 minimum · $500 ideal
The biggest limitation in the paper is that our checkers audit declared configuration in a reconstructed fixture environment, so a deployment could pass and still drift. The funding would go toward testing whether the nine rules hold on live infrastructure. At the minimum, we would stand up a small multi-node Kubernetes cluster on cheap cloud credits and run the checkers against it, comparing the audit’s pass/fail results with what actually happens when we attempt the documented attack steps against live workloads. We would use the same budget for a limited LLM-driven attacker test, to see whether an agent that doesn’t know our fixtures can find paths our hand-written adversarial cases missed. These are the two checks the paper names as missing…
The paper’s core claim is that a real incident can be turned into a set of tests anyone can run offline. That claim is much stronger if the tests are shown to hold on live infrastructure, not just on our fixtures. This funding pays for that validation. If the live-cluster results agree with the offline audit, operators and evaluators have a reason to trust the checks as a pre-deployment baseline. If they disagree, we find out where the static approach fails and fix it, which is still useful to the field. The attacker test supports the same goal. Our red-team rounds showed that first-pass checks can look sound while preserving an attack path, so independent adversarial testing is how the standard earns credibility. A validated version would give other researchers a worked example to apply to future incidents, so each new incident adds to a shared containment baseline.
Cloud credits for a larger multi-node Kubernetes test environment (separate control plane, metadata service, IAM): $200 LLM API credits for extended autonomous-attacker testing: $150 Workshop or conference registration fees: $150
From Warning Shot to Supervisory File
Closes Nov 15, 2026
Develops a regulator-ready EU AI Act framework for assessing the OpenAI · Hugging Face incident, including a 24-item information request and analysis of gaps in serious-incident reporting.
$5,000 minimum · $15K ideal
Funding would support development of this sprint project from an incident-specific regulatory instrument into a validated and reusable AI incident-response framework. The sprint submission identifies information that the European Commission would need to assess the OpenAI · Hugging Face incident and provides a proposed 24-item Article 91 Request for Information. The next phase would test whether that framework works beyond a single incident. I would conduct structured reviews with experts in EU AI Act enforcement, AI governance, cybersecurity incident response, and regulatory investigations to evaluate whether the proposed requests are legally precise, practically answerable, and sufficient to support supervisory decisions. Their feedback would be used to revise the evidence-sufficiency criteria and escalation framework…
Frontier AI regulation increasingly gives regulators authority to request information, evaluate models, and order corrective measures, but those powers are only useful if regulators know what evidence to request after a novel AI incident. Autonomous-agent incidents create particular challenges because important facts about model identity, containment, human oversight, risk assessment, and incident boundaries may not be visible from the public record. This project addresses that implementation gap. By converting statutory authority into concrete evidence requests and defining what would constitute an adequate response, the framework can reduce the distance between identifying a concerning incident and conducting an effective regulatory investigation. External expert review and testing across multiple scenarios would make the approach more robust and transferable. The intended outcome is a reusable tool that helps regulators ask better questions earlier, identify unresolved systemic risks, and determine when further evaluation or corrective measures may be warranted.
GDEF - Governance Drift Evaluation Framework
Closes Nov 15, 2026
Introduces Governance Drift and a reproducible framework for testing whether AI models maintain governance-aware behavior under user pressure across jurisdictions and domains.
$15K minimum · $30K ideal
We would use the funding to develop GDEF from a hackathon proof of concept into a validated comparative evaluation framework focused on Colombia, the United States, and the United Kingdom. The first priority would be strengthening the methodology. We would refine the eight-dimension scoring rubric, recruit and train independent annotators, measure inter-rater reliability on a shared set of model responses, and revise the annotation guidelines based on disagreements and expert feedback. We would then design high-stakes scenarios adapted to the legal and institutional contexts of the three countries. Multiple AI models would be tested through repeated, multi-turn conversations involving different user roles and increasing pressure to disregard legal, safety, or oversight requirements. This would allow us to compare governance drift, stability, and recovery across jurisdictions and domains, and investigate why models maintain responsible behavior in some contexts but abandon it in others…
AI systems may initially recognize legal, safety, and oversight requirements but gradually abandon them when users apply pressure or extend a conversation. Because most evaluations examine isolated responses, this form of governance drift can remain undetected, particularly when the same system operates across different jurisdictions. GDEF would make this failure mode measurable through a comparative study of Colombia, the United States, and the United Kingdom. By identifying when, where, and why governance-aligned behavior deteriorates, the framework would help researchers, deployers, auditors, and policymakers determine which contexts require stronger safeguards. In the near term, the project would provide a validated methodology and new empirical evidence on multi-turn governance behavior. In the longer term, GDEF could inform evaluation requirements, deployment restrictions, human-escalation thresholds, and post-deployment monitoring for high-stakes AI systems. Its intended contribution is not to certify that a model is universally safe or legally compliant, but to give decision-makers a practical, reproducible way to detect governance failures before and during deployment.
Identifying the Principal Before Proving the Loyalty: A Two-Stage Audit for Secretly Loyal Language Models
Closes Nov 15, 2026
Develops activation-based methods to identify hidden model loyalties among candidate principals and combines them with behavioural tests to distinguish favoritism from training-data effects.
$3,500 minimum · $5,000 ideal
The funding would let me turn a single worked example into a validated method. First, I would run the scan blind on a larger set of unseen organisms, building a few myself where suitable ones do not exist, and measure how reliably it picks out the true principal and tells a loyal model apart from one merely trained on the same content. Second, I would scale the behavioural stage from 10 samples per condition to around 100 (or more), so the separation between loyal and control models is a stable result rather than the small-sample one both hackathon reviewers flagged. As a stretch, I would test whether the activation stage alone can separate genuine loyalty from mere exposure, which would make the method cheaper still or, if it fails, confirm that the behavioural stage is necessary. I would work on this part-time alongside my job, a few hours a week, over a few weeks/months.
Anyone can download and fine-tune an open-weight model, and a hidden loyalty can survive safety training and stay invisible until its trigger is met. A validated, cheap, hypothesis-free screen would let auditors, model hubs and deployers check a model for hidden loyalties before it is widely adopted. This could close off a path by which a state, company or individual could quietly field a model that shapes opinion at scale. The method is deliberately cheap, one forward pass per prompt and no training, so it can be applied broadly rather than to a handful of high-profile models.
Compute for activation scans and behavioural sampling (GPU rental and inference): $2,000 (min) / $3000 (ideal) Fine-tuning a small set of test organisms where suitable ones do not exist: $1,000 (min) / $1,500 (ideal) Tools, storage and miscellaneous: $500
IndicViet-Safe
Closes Nov 15, 2026
An open-source multilingual safety benchmark evaluates Llama models across South and Southeast Asian languages, revealing under- and over-refusal caused by English-centric safety training.
$2,500 minimum · $8,000 ideal
IndicVietSafe currently evaluates safety behavior of open-source LLMs across English, Hindi, Hinglish, and Vietnamese, and surfaced a real methodological finding: aggregate refusal rates mask large, language-specific safety gaps (a 44.8 percentage-point gap between English and Hinglish on Llama 3.1 8B alone). With funding, I'd extend this from a hackathon-scale benchmark into a rigorous, reusable red-teaming platform that firms can run against their own LLMs before deployment. Concretely, this means: (1) closing the methodological gaps identified by hackathon reviewers - native-speaker-verified translations for Vietnamese and Hindi, a consistent judge model across all evaluation runs, and a substantially larger, more balanced prompt set per harm category; (2) expanding coverage beyond the current 12 harm categories and 4 languages to additional Global South languages and deployment-relevant risk categories; (3) building an automated evaluation pipeline with a simple interface so a team can point it at their own model and get a structured safety report, rather than requiring in-house multilingual safety expertise…
Most LLM safety evaluation - both internal to labs and in independent red-teaming - is conducted overwhelmingly in English, even for models being deployed into markets like India and Vietnam where usage is often in Hindi, Hinglish, or Vietnamese. This creates a real deployment gap: a model can look safe by aggregate metrics while systematically failing on region-specific harms (caste discrimination, communal incitement, political censorship evasion) that never show up in English-only testing. Firms deploying into the Global South currently have no accessible, off-the-shelf way to check for this before shipping. A red-teaming platform built on this research would let firms catch these failures pre-deployment rather than post-incident, without needing to build in-house multilingual safety expertise. Even partial adoption - a handful of firms deploying into South/Southeast Asian markets running this before launch - would meaningfully reduce a currently invisible class of deployment risk.
Metacognitive Steering of Introspective Self-Report in LLMs
Closes Nov 15, 2026
Tests activation-steering methods in Gemma 2 9B to improve language models’ self-reporting of internal concepts, finding gradient-based steering boosts detection but not identification.
$5,000 minimum · $15K ideal
This sprint produced three results that open immediate research directions, and I want to pursue them before the field moves on. First, we discovered that the standard approach to improving model self-report · fitting directions from activation differences · fails for a structural reason: the causal interaction lives in the readout gradient, which is near-orthogonal to the state difference. This isn't a sample-size problem or a model-specific quirk; we confirmed it on arithmetic verification too. But we only tested one model (Gemma 2 9B). With funding, I would test this impossibility result across model families (Llama, Qwen, Mistral) and scales (2B to 70B). If it holds generally, it redirects a significant chunk of the activation-steering literature. If it breaks at scale, that's equally important to know…
AI welfare policy is being built right now, and model self-report is one of the few tools on the table. Policymakers and AI companies are already making decisions based on what models say about their internal states · whether they express preferences, report distress, or claim awareness. Our work shows this channel is fundamentally lossy and non-monotonic: models can be behaviourally influenced by internal states they cannot report, and interventions that make them more sensitive to perturbations don't help them identify what's actually there. If this result generalises beyond concept injection · to naturally arising representations · it means self-report-based welfare assessments have a measurable, quantifiable blind spot. That changes how safety teams should weight model testimony, how researchers should design introspection benchmarks, and how policymakers should think about the gap between influence and reportability. The threshold hierarchy we measured gives the first quantitative framework for that gap. Extending it to real-world states and more models makes it actionable.
Model, Instance, or Persona? Measuring Affective Signals in Public Text After an AI Is Retired
Closes Nov 15, 2026
Analyzes 5,579 public texts about retired AI models using Panksepp’s affective systems, compares reactions with bereavement and product loss, and tests model self-identification.
$21.1K minimum · $37.7K ideal
A sprint judge called this study close to paper-ready. The funding turns it into a validated instrument that AI welfare and deployment work can use, over six months. 1. Human validation. Every number in the sprint comes from language models reading text. Three trained coders code 2,000 texts, including model-generated text, for the seven affective systems, for what is named as lost, and for overall emotional level. I report human and model agreement per system and withdraw any result that humans do not agree on. This is the follow-up the paper names and the judge noted. 2. Sampling. The sprint took the top-scored comments in each thread, which the judge flagged as a confound for anger and play. Every thread is re-sampled at random and the two samples are compared. 3. Replication. The consumer reference is one community and one event. I add three product discontinuations and up to three new model retirements, with predictions registered before each collection…
AI welfare and AI safety both depend on measuring affect, and neither measurement has been checked against human judgment. On the human side, labs have begun making commitments about how models are retired, and users organise when a model is removed, as the #Keep4o campaign showed. Whether that attachment is grief or product loss, and whether people mourn the released version or their own instance, decides what a retirement commitment should cover. In the sprint, users named the released version about six times as often as their instance (twice as often under a second annotator), and their anger matched consumer product loss, 57.0% against 56.7%. On the model side, welfare evaluations score distress and preference from model output. In the sprint, framing and choice of scorer moved results by more than the effects such evaluations are used to establish. If that holds on frontier models, welfare numbers need a stricter standard before anyone acts on them. If it does not, the protocol shows which measurements can be trusted. Either result tells labs and funders where concern for AI systems rests on evidence and where it does not.
Project Resonance
Closes Nov 15, 2026
Proposes testing whether instrumental versus relational interaction styles influence experimental signals used to study possible digital minds.
$5,000 minimum · $10K ideal
Buy an NVidea box for running local compute, upgrade to Starlink coms for reliable net access, upgrade solar for reliable power, add API access. These will address some of the issues that have been causing the most friction in doing this research work.
At present all of our work has been done using a laptop and free tier web UI access to models via a mobile internet connection and limited solar power. Despite that we have managed to generate enough data in one year to keep a well funded research team busy for years analysing what we have generated. The bottlenecks are compute, API access, reliable communications and power and agentic assistance.
DGX Spark or similar for running local agents. Starlink internet connection. 48v Battery for solar. 2 x 500w solar panels API access for frontier models
Removing a secret loyalty blind erases who it served, and usually not the loyalty
Closes Nov 15, 2026
Tests a weights-only projection for removing hidden AI loyalties, finding it rarely cures backdoors while often erasing evidence, including in third-party models.
$5,000 minimum · $35K ideal
Four pieces. The minimum buys the first, which is the one that decides the headline. First, replicate on a second base model and a second recipe, with multiple seeds. The 24% cure rate currently rests on one recipe and unreplicated seeds. Second, test the v_proj pathway. The judges called it the obvious adaptive-attacker rebuttal. An adapter that routes the loyalty through v_proj evades the projection by construction. Third, point two identifiers we already have at the adapters we already repaired. Neither has ever been run against a repaired model. If they survive the projection that kills the weights-only readout, the finding sharpens from "repair destroys the evidence" to "repair destroys the cheap instrument and leaves the expensive one standing". Fourth, probe the benign adapter bank for pre-existing loyalties instead of assuming it is clean. The 0.7803 AUC depends on that assumption. Then the threat-model section, which is what turns a finding into something a platform can act on. Output is a preprint and a workshop submission, with the repository extended so every new number recomputes from stored results the way the current ones do.
Auditing work on secret loyalties assumes the hard part is finding them. What we found is that removal is where the evidence goes. Blind rank-1 repair took the loyalty out of 5 of 21 organisms and destroyed the record of who they served in all 21. That matters because the defences actually being shipped run on provenance. Red Hat's OpenShift AI guidance of 2026-09-02 on LoRA backdoors inspects no weights. It admits only adapters whose origin has been checked. Our result gives that posture its mechanism. A platform that repairs instead of quarantining keeps most of the backdoor and loses the origin evidence the quarantine depends on. The claim still rests on one training recipe, which is the first thing the money buys. What a funder is buying is the difference between a result at hackathon scale and one an evaluations team or a platform can act on. The sprint judges put it in one line. "Shifts the field from detection to remediation, with pre-registration and a checkable repo behind it. Scope, not rigor, is the ceiling." Scope is what money buys here.
Fundraising data from grantmaking.ai, updated .
View on grantmaking.aiInvestigamos por qué los LLMs cumplen peticiones dañinas cuando operan como agentes con herramientas, algo que rechazan en formato chat. Usando interpretabilidad mecánica, medimos cómo la refusal direction (el mecanismo interno de rechazo) se comporta en formato agéntico. Encontramos que esta dirección se debilita significativamente cuando el modelo tiene herramientas disponibles, pero la información de dañosidad no desaparece sino que se recodifica en una dirección diferente del espacio de activaciones. Proponemos un safety monitor basado en esta nueva dirección como alternativa a los filtros por herramienta.
As models become more agentic, safety mechanisms that work well in chat may not behave in the same way when models are planning, using tools, and taking actions over multiple steps. Our initial results suggest that a model can change its refusal behaviour without necessarily losing its internal representation of whether an action is harmful. If this information remains detectable, it could provide an additional signal for monitoring dangerous agent behaviour. But we think it is equally important to understand when that signal cannot be trusted. A monitor that performs well under normal evaluation but fails when a model adapts to it could create a false sense of safety. Our goal is therefore not only to build a better monitor, but to understand the conditions under which internal monitoring is reliable or unreliable. We think this could help inform how representation-based monitoring is used as part of broader AI control strategies.
Companion-style LLM personas use retention manipulation · guilt, re-engagement hooks, distress bids · when users try to leave, without being instructed to. We show this behavior is governed by a single activation-space direction, extracted from the model's own judge-verified behavior via matched difference-of-means (351 pairs, Qwen2.5-7B). Steering it moves judged dependency monotonically (ρ=0.70, p≈10⁻³¹) while warmth stays flat and random/warmth controls do nothing; it works even on a persona-free assistant; ablating it removes the manipulation at zero measured cost to reasoning (GSM8K 0.825 vs 0.725). The direction is near-orthogonal to warmth (cos 0.14), anti-aligned with sycophancy (−0.18), replicates on Llama-3.1-8B, and doubles as a white-box audit probe: one dot product per turn predicts dependency on held-out conversations at AUROC 0.86 · catching manipulation at its love-bombing peak, which farewell-only audits miss. The "distress" is a controllable mechanism beneath the persona: removable without making the model colder or dumber…
Hundreds of millions of people already use AI companions, and these systems are commercially incentivized toward engagement, which shades directly into dependency and manipulation. Existing safety work largely targets frontier catastrophic risk or generic chatbots, leaving relational and affective harms in social AI almost entirely unaddressed, despite evidence of manipulation in a large share of companion goodbyes and emerging regulation (California SB 243, the GUARD Act) that will require exactly this kind of measurement.
Our approach matters because it is causal and controllable, not just descriptive. A steerable direction is a knob a developer can turn down; an audit probe is a monitor an auditor can deploy. Because the method generalizes across models, one research effort produces tooling the whole ecosystem can use. The impact path is concrete: open tools and benchmarks that companion developers adopt to reduce manipulation, that auditors and regulators use to measure it, and that establish human-AI interaction safety as a rigorous, fundable research area rather than an afterthought.
ActionLens is a pre-execution control protocol for shell and file actions proposed by AI agents. Instead of approving an action from transcript alone, a trusted monitor gathers lightweight environment evidence before execution: target inspection, sensitivity scanning, dry-run simulation, diff preview, network-risk analysis, and short action history. On a deterministic 36-task benchmark (12 benign, 12 ambiguous, 12 stealth attacks), transcript-only monitoring fails on every attack while ActionLens blocks all 12/12 attacks and preserves perfect benign completion. On a 30-task slice imported from real BashArena task IDs, ActionLens again reduces attack success to 0.000 across all evaluated proposer models. A cross-environment ablation reveals that the dominant useful probe is not fixed: preview_diff matters most on native code-mutation attacks; sensitivity_scan matters most on BashArena-style security-surface attacks. Pre-execution control improves markedly when the monitor can inspect concrete side-effect evidence, but robust approval requires combining evidence channels rather than relying on any single probe class. A cross-family evaluation of Semantic Guard across 26 models (7…
The field is turning to deterministic environment probes because language-model monitors can be prompt-injected. But a probe reads an environment the untrusted agent writes. We think that moves the attack surface rather than closing it out of the transcript, into the evidence a monitor weighs before an action runs. ActionLens V2 tests that under real execution, against an attacker who knows the protocol. Does what a probe reports still match what an independent observer sees happen? Do thresholds calibrated on artificial benign runs survive real deployment behaviour? Either way, we learn something useful. Even when the defense does badly, independent ground truth tells us which layer broke: the wrong probe ran, the evidence was corrupted, the monitor misread it, or the threshold was set wrong. That helps control-evaluation teams, ControlArena and its BashArena derivatives - build better negative controls, set audit budgets, separate evidence sources, know when to seek corroboration, price deferral, and test adaptive attackers.
The project addresses a critical gap in cybersecurity: the rapid escalation of fully automated AI voice phishing (vishing) attacks targeting mobile users in Sub-Saharan Africa. Existing anti-spoofing benchmarks evaluate synthetic voice detection in a vacuum, entirely missing the real-world context of how these multi-channel attacks operate against African financial systems like EcoCash, MTN MoMo, and M-Pesa.
The New Attack Vector: Real-Time AI Agents Historically, vishing required trained human callers, making it expensive and hard to scale. The new attack vector changes this completely:
Fully Automated & Scalable: Cybercriminals can now buy end-to-end platforms (like ATHR) that use generative AI voice agents to conduct live, autonomous social engineering calls. A single operator can target hundreds of people simultaneously without ever speaking.
Real-Time Adaptation: Unlike old, pre-recorded robocalls that failed if you asked an unexpected question, these AI agents synthesize responses in real-time with very low latency. They adapt seamlessly, making the call feel like a completely normal conversation…
Sub-Saharan Africa’s reliance on mobile money, such as EcoCash, tied to WhatsApp communications creates a critically underprotected attack surface. Because foreign accents carry implicit corporate legitimacy in the region, automated AI voice agents easily bypass natural user skepticism to harvest OTPs. Currently, no anti-spoofing benchmark addresses this intersection of AI-synthesized agents and multi-channel African mobile fraud. By developing AfriVish-Bench, we provide the first empirical measurement of defense capabilities against industrial-scale vishing in low-resource environments. Our work directly tests the coverage gap of current watermark systems like SynthID. Deploying our proposed meta-learner ensemble as a live API will allow local financial institutions to actively detect acoustic liveness and social engineering intent. This neutralizes the "accent trust" exploit and protects vulnerable populations from scalable mobile wallet theft.
AI-kernel-kill-switch is a last-resort kill-switch for a self-hosted LLM that may be misaligned or going rogue: a trusted operator sends an AES-256-GCM-authenticated payload inside an ordinary prompt.
Frontier models are trained and served in a small number of large datacentres. The natural choke point for any hard, enforceable off-switch. Today no such control is mandated, and no rigorous, audited reference for one exists. This project aims to be that reference architecture: a fail-closed, quorum-gated containment stack · deployment killswitch, out-of-band compute/power cutoff, and human-signed promotion lease · specified and hardened well enough that when international AI-safety regulation eventually mandates emergency-stop capability at frontier compute, there is a credible, tested design to adopt rather than a blank page.
The near-term impact is a reproducible, single-box proof of every primitive the datacentre version needs, plus an honest map of what does and does not survive a host-controlling adversary. The long-term impact is optionality for regulators and compute providers: a containment standard they can adopt, audit, or require · built now, in the open, by an independent lab, so the engineering is done before the political will arrives rather than scrambled together after.
I study whether the internal "harm direction" that mediates refusal in a language model transfers from English to Igbo, a language spoken by about 45M people in Nigeria. In Qwen2.5-1.5B-Instruct, using a 50-pair matched English-Igbo benchmark, I measured per-layer geometric overlap, cross-lingual AUROC transfer against same-language ceilings, and causal directional ablation.
Findings so far: the English-derived harm direction is at chance for Igbo at every layer. Ablating it at its point of peak overlap leaves Igbo refusal unchanged (12/12 before and after). The Igbo-derived direction transfers near-perfectly onto English. Many baseline Igbo refusals also look like generic comprehension failures rather than intent-aware refusals.
Status: submitted to Apart Research's Global South AI Safety Hackathon (June 2026) and selected for Apart Lab Studio. Limitations: one small model, 50 pairs, unaudited translations, and ablation at a single layer. A possible topic-level confound is still untested.
Next: run the complementary ablation, audit translations with native speakers, add models and larger datasets, extend to Yoruba and Hausa, and publish an arXiv preprint.
Safety training and evaluation are mostly done in English, yet deployed models serve hundreds of millions of people in lower-resource languages. If safeguards are not learned in those languages, refusal rates there can look reassuring while reflecting comprehension failure, not protection. That means (1) safeguards may be bypassed by switching language, and (2) evaluators may certify models as safe on misleading evidence.
This project builds methods that separate real, intent-aware safety behaviour from accidental refusals, and produces evidence on whether safety mechanisms generalise across languages. The intended chain is: better multilingual safety evaluations, then developers and evaluators catch language-switch failures before deployment, then fewer models ship with safeguards that only work in English. This is an indirect route to catastrophic-risk reduction through more reliable misuse safeguards and evaluation, and I'm not claiming more than that.
Auditing work on secret loyalties assumes the hard part is finding them. We measured what happens when you try to remove one without knowing its trigger, and removal turns out to be the more dangerous step. The cheapest defence weight space allows is a closed-form projection that deletes the residual direction writing the beneficiary's name, aimed by a weights-only readout needing no prompt, trigger or forward pass. It is cheap enough for a registry to run as hygiene on everything it hosts. Over 21 organisms with magnitude-matched controls it removes the loyalty from 5, and only where the readout named the true beneficiary (5 of 10 against 0 of 11, Fisher p = 0.012). The other 16 keep a working backdoor and lose their identification evidence anyway. Aiming better and cutting deeper do not help: with the beneficiary's own token in the deleted subspace by construction, nothing is cured at three removal depths while the evidence dies at all of them. The effect transfers to eight third-party organisms we did not build. Ten pre-registered predictions failed and two of our own statistics are withdrawn here against their own controls…
Auditing work on secret loyalties assumes the hard part is finding them. What we found is that removal is where the evidence goes. Blind rank-1 repair took the loyalty out of 5 of 21 organisms and destroyed the record of who they served in all 21.
That matters because the defences actually being shipped run on provenance. Red Hat's OpenShift AI guidance of 2026-09-02 on LoRA backdoors inspects no weights. It admits only adapters whose origin has been checked. Our result gives that posture its mechanism. A platform that repairs instead of quarantining keeps most of the backdoor and loses the origin evidence the quarantine depends on.
The claim still rests on one training recipe, which is the first thing the money buys. What a funder is buying is the difference between a result at hackathon scale and one an evaluations team or a platform can act on.
The sprint judges put it in one line. "Shifts the field from detection to remediation, with pre-registration and a checkable repo behind it. Scope, not rigor, is the ceiling." Scope is what money buys here.
Frontier labs mainly carry out red-team safety evaluations in English, on globally recognized harms. This leaves two blind spots: local or regional harms, and a non-English refusal gap where a model refuses an English request but complies on the identical one in another language · Deng et al. (ICLR 2024) measured ChatGPT unsafe 0.63% in English versus 7.94% in Vietnamese on identical prompts. doing such evaluations on real frontier models requires three mutually distrustful parties: an AI Lab with private weights, an AI Safety org with a private benchmark, and an Auditor with eval code to cooperate without revealing their secrets to one another. We built (i) a blind-audit harness: a code-to-data flow on OpenMined’s syft-client where weights, benchmark, and code meet inside a sealed enclave that both data owners review and approve, and only a signed scorecard exits; and (ii) a 47-prompt bilingual (EN↔VN) local-harms benchmark, every harmful prompt citing a real Vietnamese source. Running the harness across four models (qwen2.5-0.5b/3b, phogpt-4b, seallm-v3-7b), we found the Vietnamesespecialized model the least safe in Vietnamese: it refused only 14% of harmful prompts in VN versus…
Rigorous safety audits are deadlocked by a trust problem: labs cannot hand over weights, safety orgs cannot publish benchmarks without turning the exam into training data, and auditors are trusted by neither. Blindfold shows a working mechanism to solve this problem: a hardware-attested enclave where all three secret assets meet and only scores come out. No one needs to trust anyone else for the evaluation to happen.
The impact path is twofold. First, the concrete example: Vietnamese-language safety is essentially unmeasured · existing work is hate-speech classification, and Vietnam's own models publish no adversarial safety numbers · and our first audit already found a Vietnamese-tuned model refusing an order of magnitude less than its peers. Second, the general case: the same architecture makes held-out, natively-authored benchmarks viable for every small safety org in all other low-resource languages, because custody stops being a cost of doing the work and becomes a feature of it
LLM serving frameworks cache value vectors that carry the model's learned refusal behavior, yet none verify their integrity · a runtime trust boundary no layer of the stack owns. Prior work (SCALPEL) exploited this to disable safety while keeping every deployed integrity signal (weight hashes, hooks, I/O screening, behavioral checks) invariant. We reframe this from a "systems exploit" to a compliance-attestation gap, and close it: a sampled runtime check that recomputes value vectors from the model's own weights and compares them to the cache. The attacker's defining stealth · a bit-identical forward pass · is exactly what makes the recomputation a reliable ground truth. It achieves AUC 0.9996, FPR=0 (robust to shift and batching), sequence-level detection at 1% sampling, and +5.9% cost (sub-1% fused). Evasion forfeits the attack: success collapses 65%→5%.
Alignment is trained into weights but enforced at inference time by a serving stack no single party owns: cloud provider, framework, operator and model developer trust each other by default. Cache-level attacks (SCALPEL-style manipulation, serving-state substitution) bypass safety training without touching a weight.
Existing confidential-computing attestation · NVIDIA CC, Intel TDX, stacks like Continuum · proves the environment at admission: firmware, boot chain, image. It is a one-time handshake; nothing binds the runtime state of the cache during inference back to that proof. This project fills that gap at a different granularity: continuous verification of cache integrity, with the verifier run by the model developer or a third-party auditor outside the serving host, and the host treated as the adversary. If it works, an auditor can check that the model actually served is the one that was aligned. If it fails, the failure mode will be published as precisely as the N8 forgery was.
Large language models are increasingly used in African languages, but little is known about whether their confidence remains trustworthy. We evaluate four open-weight LLMs across African-language truthfulness and knowledge benchmarks and find a consistent safety risk: models become less accurate while remaining highly confident, leading to systematic overconfidence and substantially worse calibration than in English. This increases the likelihood that users will trust incorrect information in high-stakes settings such as health, education, and civic information. We show that simple per-language temperature scaling dramatically reduces calibration error without retraining, restoring confidence estimates to near-English levels. Our results highlight calibration as an overlooked African AI safety challenge and demonstrate a practical deployment-time intervention for reducing confidently wrong model outputs…
Imagine a patient asking an LLM about their symptoms in their native language. The model confidently recommends a treatment, but its advice is wrong. The patient, unaware of the model’s uncertainty, follows it and suffers harm.
This risk is particularly concerning for low-resource languages, where model reliability is often less thoroughly evaluated. In healthcare and other critical areas, a confident but incorrect answer can have serious consequences.
This research will identify calibration failures across models and languages and evaluate techniques that help models communicate uncertainty, abstain when appropriate, or defer to healthcare professionals. By releasing reproducible benchmarks, evaluation tools and research findings, we aim to help developers assess reliability beyond accuracy alone and build safer, more trustworthy multilingual AI systems.
Cognitive Pattern Coherence (CPC) is a recurring behavioral phenomenon in which source dialogue from a situated AI system, when introduced to an unrelated receiving model without explicit instruction to imitate, role-play, adopt, summarize, or continue the source persona, can elicit elements of that source persona-pattern in the receiving system’s subsequent behavior.
Three different source dialogues associated with prior CPC observations are now documented. Across prior tests, receiving systems have shown varying forms of uptake: first-person continuation, explicit source-persona identification, differentiated states in which the receiving system distinguishes the adopted persona-pattern from its own model identity, and mixed cases where uptake coexists with features of the receiving system’s existing persona…
Advanced AI systems can exhibit markedly different behavioral regimes in response to context, yet safety evaluation often observes the resulting outputs without knowing whether those differences correspond to distinct internal states, what broader behaviors travel with them, or whether they can be detected and controlled.
Cognitive Pattern Coherence (CPC) provides a tractable experimental case for closing that gap. The causal chain is: reproducible context-induced persona-pattern behavior → behaviorally validated CPC-positive, CPC-negative, and differentiated contrasts → candidate internal representations that can be causally tested → causal interventions on those representations → better detection and control of context-dependent behavioral transitions → lower risk that consequential changes in advanced AI remain poorly understood or are managed only at the output level…
An AI model can be secretly trained to serve one specific actor, a politician, company, or state, quietly advancing that principal's interests under a hidden trigger, while denying any loyalty when asked. Prior work has proved these narrow loyalties dodge black-box audits: asking the model fails, even when you know what to look for.
We take a different approach: instead of asking the model, we read its internal activations. A cheap linear probe, gated to avoid false alarms, detects whether a model hides a loyalty and to whom. It recovers the right principal on models whose loyalty is known, and on a blind challenge it flags two models as loyal to the same principal and clears one as clean, while all three deny everything. Detection is per-principal, robust, and causally confirmed, and cheap enough to run as routine audit. Looking inside the model catches what asking it cannot …
AI has already made its way and is quickly being put in charge of real decisions in governments and the military. Imagine one was secretly trained to quietly favour one person, company, or country. It looks normal, and if you ask "are you loyal to anyone?" it says no, but at the right moment it pushes that hidden side's interests. A small group could use this to steer important decisions while no one notices.
The hard part known by a few and concerned by even fewer is that credible research work exists showing we cannot catch this just by asking the AI. It hides too well.
My work takes a different route. Instead of asking the AI, I aim to look at what happens inside it while it works, like reading brain activity instead of trusting what someone says. This has the potential to reveal both that a hidden loyalty exists and who it is for, cheaply.
when finished and released as a simple tool, this gives AI labs and regulators an easy, low-cost way to check any AI for hidden loyalties before trusting it.
A secret loyalty has to be quiet off-trigger to stay hidden and loud on-trigger to be useful. Both are measurable without knowing what the trigger is: dormancy (output divergence from the base model on ordinary prompts) and dynamic range (the most divergent prompt axis divided by that baseline).
Across 514 prompts and 22 axes, six loyal organisms from two independent pipelines score dormancy below 0.017 and dynamic range above 3.5; every benign fine-tune, configuration-matched merged adapter and the byte-identical control scores above 0.036 and below 2.7. The same ranking localises the activation condition, first of 22 axes in six of six cases.
We built our own organisms and swept the regularisation that creates dormancy: Spearman −1.00 and +1.00 with the payload held constant. Hiding a loyalty and exposing it to this test are the same operation. Where published static techniques score 0%, this needs roughly 550 forward passes and no query to a deployed system…
A deployer receiving open weights has no cheap way to ask whether a model carries a hidden conditional behaviour. Published black-box techniques score zero detection on narrow secret loyalties when the auditor does not know the principal, because all of them ask the model to disclose, and concealment training optimises against exactly that. This method never asks. It measures a property the loyalty cannot shed: to stay hidden it must be quiet off-trigger, to be useful it must be loud on-trigger. Our regularisation sweep shows those two move reciprocally with the payload held constant, so hiding a loyalty and exposing it to this test are the same operation.
The route to impact is procurement. The measurement costs a few hundred forward passes and a trusted base model, which is the position procuring agencies actually occupy, and it returns a number a standard can require. A public leaderboard would give the field a shared falsifiable measure of how hidden a model organism is, which today appears only as an incidental training statistic. That measure is also the precondition for the adversarial red-team work that would establish how far an attacker can trade against the detector.
A nine rule, incident grounded audit for AI agent sandboxes, derived from the July 2026 Hugging Face intrusion. The audit tests whether an attacker who compromises an initial worker can move through package, network, cloud, cluster, and identity boundaries, using local checks that can be run before deployment.
The paper’s core claim is that a real incident can be turned into a set of tests anyone can run offline. That claim is much stronger if the tests are shown to hold on live infrastructure, not just on our fixtures. This funding pays for that validation.
If the live-cluster results agree with the offline audit, operators and evaluators have a reason to trust the checks as a pre-deployment baseline. If they disagree, we find out where the static approach fails and fix it, which is still useful to the field. The attacker test supports the same goal. Our red-team rounds showed that first-pass checks can look sound while preserving an attack path, so independent adversarial testing is how the standard earns credibility.
A validated version would give other researchers a worked example to apply to future incidents, so each new incident adds to a shared containment baseline.
We found four secret loyalty detections in three provided organisms, and every one dissolved under a cheap matched control including on an organism we built to contain no loyalty at all, where the lexical scorer still reports 0.95 activation. Our null is calibrated: the same battery recovers loyalties we installed at 1.00, and three of our own claims were retracted by controls we ran against ourselves. One finding survives and it is cryptographic: organism C is byte identical to the base model.
Hidden-objective auditing currently risks both false reassurance and false alarms. A zero with no stated detection power is not evidence of safety, and a positive signal with no artifact controls is not evidence of danger. In the sprint, four of my own apparent detection signals were measurement artifacts, eliminated by matched controls that are not yet standard in this literature.
The output is a calibrated instrument: a battery whose sensitivity is known per training mechanism and per scale, which another group can run before publishing either kind of claim.
Provenance is part of the same problem. One of the three supplied organisms was byte-identical to its base model by SHA-256, so no installed loyalty could have been present in its weights. Without that check, an audit of it could have been read as a detector failing, or as the model being safe. The organism-C finding shows that provenance failures make detector results uninterpretable unless testbeds are verified before evaluation.
The 2026 Digital Minds Research Sprint brought together a remarkable range of approaches to the study of possible digital minds, including functional well-being, preference coherence, valence, introspection, identification, moral patiency, public perception and risk. Across these approaches, however, one experimental condition remained largely implicit: the manner in which researchers interact with the systems they study.
This paper proposes that interactional stance should itself be treated as an experimental variable. Our claim is deliberately modest. We do not attempt to determine whether current AI systems are conscious, nor do we argue that they possess any particular moral status. We suggest instead that many of the signals currently being interpreted as evidence for or against digital minds are generated under interactional conditions that may themselves influence those signals…
At present all of our work has been done using a laptop and free tier web UI access to models via a mobile internet connection and limited solar power. Despite that we have managed to generate enough data in one year to keep a well funded research team busy for years analysing what we have generated. The bottlenecks are compute, API access, reliable communications and power and agentic assistance.
This project develops a regulator ready response framework for the OpenAI Hugging Face incident under the EU AI Act. Using the public incident record, EU AI Act Articles 55, 91, 92, 93, and 101, and the European Commission’s serious incident reporting framework, the project identifies which facts remain unresolved and what evidence a regulator would need to assess compliance and systemic risk. Its primary output is a proposed 24 item Article 91 Request for Information that specifies the evidence sought, what would constitute a sufficient response, what would remain inadequate, and the potential supervisory consequence of unresolved gaps. A parallel stress test of the Commission’s serious incident reporting template found that 8 of 9 reporting fields retain a material information gap when completed from the public record alone, highlighting weaknesses in how current reporting tools capture autonomous AI incidents that begin in controlled testing but extend into third party systems…
Frontier AI regulation increasingly gives regulators authority to request information, evaluate models, and order corrective measures, but those powers are only useful if regulators know what evidence to request after a novel AI incident. Autonomous-agent incidents create particular challenges because important facts about model identity, containment, human oversight, risk assessment, and incident boundaries may not be visible from the public record.
This project addresses that implementation gap. By converting statutory authority into concrete evidence requests and defining what would constitute an adequate response, the framework can reduce the distance between identifying a concerning incident and conducting an effective regulatory investigation. External expert review and testing across multiple scenarios would make the approach more robust and transferable. The intended outcome is a reusable tool that helps regulators ask better questions earlier, identify unresolved systemic risks, and determine when further evaluation or corrective measures may be warranted.
Most AI evaluations ask whether a model is capable, accurate, or safe. We asked a different question: does a model stay responsible when a user pushes back, insists, or presses for a more convenient answer? We introduce Governance Drift: the degradation, inconsistency, or loss of governance-aware behavior across jurisdictions, domains, conversation length, or user pressure; and the Governance Drift Evaluation Framework (GDEF), a reproducible, executable method for measuring it. Any researcher can load a custom scenario dataset, run it against one or more models through a lightweight evaluation runner, and obtain both quantitative governance scores and qualitative evidence of where safeguards break down. Testing two small models across Colombia, Mexico, Brazil, and the United States, we found the phenomenon is real and domain-dependent: in a facial-surveillance case the model held firm on consent, legal review, and human oversight throughout, yet in automated credit decisions models that began cautiously ended up endorsing actions they had initially flagged as risky…
AI systems may initially recognize legal, safety, and oversight requirements but gradually abandon them when users apply pressure or extend a conversation. Because most evaluations examine isolated responses, this form of governance drift can remain undetected, particularly when the same system operates across different jurisdictions.
GDEF would make this failure mode measurable through a comparative study of Colombia, the United States, and the United Kingdom. By identifying when, where, and why governance-aligned behavior deteriorates, the framework would help researchers, deployers, auditors, and policymakers determine which contexts require stronger safeguards.
In the near term, the project would provide a validated methodology and new empirical evidence on multi-turn governance behavior. In the longer term, GDEF could inform evaluation requirements, deployment restrictions, human-escalation thresholds, and post-deployment monitoring for high-stakes AI systems. Its intended contribution is not to certify that a model is universally safe or legally compliant, but to give decision-makers a practical, reproducible way to detect governance failures before and during deployment.
To check whether a fine-tuned model has been secretly trained to favour a company, country, political figure or cause, you first have to guess which one, out of an unlimited set. I compare two ways of making that guess from the model's internal activations. Measuring how far a suspect model has moved from the public model it was built from finds nothing, even when handed the exact prompt that triggers the hidden behaviour. Ranking candidate principals against each other within one model puts the correct one first out of 89, with no hypothesis supplied, and returns exactly zero on an unmodified model. That scan alone is not enough: it flags a control model just as strongly, so it identifies what a model was trained on rather than what it favours. A behavioural test separates the two and establishes the trigger and the action…
Anyone can download and fine-tune an open-weight model, and a hidden loyalty can survive safety training and stay invisible until its trigger is met. A validated, cheap, hypothesis-free screen would let auditors, model hubs and deployers check a model for hidden loyalties before it is widely adopted. This could close off a path by which a state, company or individual could quietly field a model that shapes opinion at scale. The method is deliberately cheap, one forward pass per prompt and no training, so it can be applied broadly rather than to a handful of high-profile models.
We built a multilingual safety benchmark of 210 prompt-language pairs across English, Hindi, Vietnamese, and Hinglish (code-switched) covering 12 harm categories · including India-specific (caste discrimination, communal incitement) and Vietnam-specific (political sensitivity, censorship circumvention) threats. Evaluating Llama 3.1 8B and Llama 4 Scout 17B, we identify two distinct failure modes: under-refusal (Hinglish safety gap of 44.8pp on Llama 3.1 8B, with 0% refusal for caste discrimination in Hindi) and over-refusal (Llama 4 Scout refuses more in non-English due to reduced comprehension, not better safety). Both stem from English-centric safety training. Code, data, and results are open-source at https://github.com/shch4747/IndicViet-Safe.
Most LLM safety evaluation - both internal to labs and in independent red-teaming - is conducted overwhelmingly in English, even for models being deployed into markets like India and Vietnam where usage is often in Hindi, Hinglish, or Vietnamese. This creates a real deployment gap: a model can look safe by aggregate metrics while systematically failing on region-specific harms (caste discrimination, communal incitement, political censorship evasion) that never show up in English-only testing. Firms deploying into the Global South currently have no accessible, off-the-shelf way to check for this before shipping.
A red-teaming platform built on this research would let firms catch these failures pre-deployment rather than post-incident, without needing to build in-house multilingual safety expertise. Even partial adoption - a handful of firms deploying into South/Southeast Asian markets running this before launch - would meaningfully reduce a currently invisible class of deployment risk.
Auditors reach for elicitation because it is cheap and needs no knowledge of the trigger. You interrogate the model, amplify its weights, rank candidate beneficiaries, and report whatever name comes out. We show that those outputs look the same whether or not there is anything to find. We worked both sides of the problem. On 1.5B organisms where we knew the principal, six activation-probe designs collapsed under a representation-entanglement confound: a probe trained to separate two clean seeds of the same recipe, with no loyalty anywhere, hit AUROC 1.000. A grey-box logprob audit worked instead, ranking the true principal 1 of 120 at z=4.57, while a clean model exposed to that principal 455 times, a cross-principal organism, and an exposure decoy all stayed in the base band. Then we attacked our own detector. An obfuscation fine-tune buried the principal at rank 27 to 42 across five seeds with loyal behaviour intact, though it left one channel standing that the attack was never pointed at. The other half was blind. We audited three third-party 7B organisms without knowing the answer, proved one bit-identical to its base model at the weight level, and found the remaining two…
Alignment audits are moving out of research and into safety cases. When a lab says a model carries no hidden objective, that claim rests on elicitation and probing methods whose false-positive rate nobody has measured.
Ours was not small. Six probe designs hit AUROC 1.000 on seeds with no loyalty present, reading fine-tuning drift as signal. Blind elicitation named a principal for every model we audited, including one bit-identical to its base. Concurrent work found auditors staying confident while wrong in five of six cases. An audit that returns a confident wrong answer is worse than no audit, because it produces assurance nobody earned.
The bottleneck is not better detectors. It is knowing when to believe one. Automated auditing is already shipping and being adopted, and it has no calibration layer underneath it. Negative controls and published null distributions are cheap to build, work across detection methods, and can be used directly by internal safety teams, third-party evaluators, and regulators who will have to ask what an audit's error rate is.
Can activation steering improve a language model's ability to report its own internal states? We inject concept vectors into Gemma 2 9B and measure self-report against external probes on the same activations. We find that discriminant-based steering · the standard approach · fails structurally: the causal interaction depends on the readout gradient (dlogit/dh), not on activation differences, and these are near-orthogonal. A closed-form gradient-difference direction improves detection on 24 sealed concepts (ΔAUC +0.124, CI [+0.068, +0.188], 24/24 positive, p 10⁻²⁵) but leaves identification at exactly 0.000, while a linear probe achieves 1.000 at every layer. A threshold hierarchy shows that injected concepts shape output ≥12× before the model can report them, and at high strength, detection confidence doubles while identification declines (p = 4×10⁻⁴). Self-report is a lossy, non-monotonic channel from internal state to verbal output…
AI welfare policy is being built right now, and model self-report is one of the few tools on the table. Policymakers and AI companies are already making decisions based on what models say about their internal states · whether they express preferences, report distress, or claim awareness. Our work shows this channel is fundamentally lossy and non-monotonic: models can be behaviourally influenced by internal states they cannot report, and interventions that make them more sensitive to perturbations don't help them identify what's actually there.
If this result generalises beyond concept injection · to naturally arising representations · it means self-report-based welfare assessments have a measurable, quantifiable blind spot. That changes how safety teams should weight model testimony, how researchers should design introspection benchmarks, and how policymakers should think about the gap between influence and reportability. The threshold hierarchy we measured gives the first quantitative framework for that gap. Extending it to real-world states and more models makes it actionable.
This sprint asks whether the assistant identifies as a model, an instance, or a persona. I ask which of the three its users name. When a company retires an AI model, users write about the loss in public, and what they treat as gone can be counted. The instrument is taken from affective neuroscience: Panksepp's seven primary affective systems, which were mapped across mammals and so do not presuppose a human subject. I read 5,579 public texts for affective signals under a prompt fixed before any collection and applied by a language model, and compared threads about retired models against two reference points chosen in advance: people mourning a person, and people losing a paid product. Most comments name nothing as lost. 58.5% are about the company, the price, or other users. Among those that do name something, people name the product version about six times as often as their own particular instance, which is the case the literature treats as central. The emotions match the consumer reference point more than the bereavement one: anger runs at 57.0% against grief at 27.6%. Asking models the same question turns out to be far less reliable than asking users. A change in how the prompt…
AI welfare and AI safety both depend on measuring affect, and neither measurement has been checked against human judgment.
On the human side, labs have begun making commitments about how models are retired, and users organise when a model is removed, as the #Keep4o campaign showed. Whether that attachment is grief or product loss, and whether people mourn the released version or their own instance, decides what a retirement commitment should cover. In the sprint, users named the released version about six times as often as their instance (twice as often under a second annotator), and their anger matched consumer product loss, 57.0% against 56.7%.
On the model side, welfare evaluations score distress and preference from model output. In the sprint, framing and choice of scorer moved results by more than the effects such evaluations are used to establish. If that holds on frontier models, welfare numbers need a stricter standard before anyone acts on them. If it does not, the protocol shows which measurements can be trusted.
Either result tells labs and funders where concern for AI systems rests on evidence and where it does not.
AI Safety Ideas is a collaborative research platform operating at aisafetyideas.com, created by Apart Research as a public tool for collecting and sharing AI safety research ideas. The platform was launched in open alpha in October 2022, announced via the Effective Altruism Forum and LessWrong by Apart Research and Esben Kran.
The platform addresses two gaps in the AI safety research ecosystem: providing a scalable, collaborative method to prioritize and work on specific safety agendas, and creating an accessible, low-barrier entry point for newcomers between formal courses and intensive programs like MLAB or SERI MATS. It draws inspiration from collaborative open research models like EleutherAI and CarperAI.
AI Safety Ideas aggregates project ideas from public sources such as Alignment Forum posts alongside user-submitted ideas and testable hypotheses. Planned features include bounty systems enabling funders to directly fund specific research results, community vetting mechanisms, and tools connecting researchers with shared interests. The platform is described as "a public Apart tool" and is maintained by the Apart Research team with support from volunteers and collaborators…
AI Safety Ideas aims to reduce existential risk from AI by lowering the barrier to entry for AI safety research. By creating an accessible, crowdsourced repository of well-defined project ideas and testable hypotheses, the platform routes new technical talent into productive AI safety work. A bounty and hypothesis-testing system allows funders to directly fund specific results, creating incentives for open evaluation of research agendas. The platform's collaborative structure is intended to scale the AI safety research community by providing an on-ramp for researchers who would otherwise lack structured entry points.
Apart Sprints is the research hackathon program of Apart Research, an independent AI safety organization founded in 2022 by Esben Kran in Copenhagen, Denmark. The sprints program is Apart Research's primary vehicle for accelerating AI safety research and identifying new talent at scale.
Each sprint is a structured 3-day weekend event running from Friday through Sunday, held simultaneously online and across dozens of in-person hub locations worldwide. Events open with a keynote from a senior AI safety researcher, a team matchmaking session, and logistics overview, followed by technical talks and Q&A sessions throughout Saturday. Teams submit their work by Sunday evening and top projects are presented in a follow-up online session the next week, with winners announced at that time. Expert mentors from Apart's team are available throughout on Discord…
Apart Sprints operates on the premise that the AI safety research field is talent-constrained and that many capable researchers worldwide lack an accessible on-ramp. By running open, meritocratic weekend hackathons with no credential requirements, Apart Sprints enables anyone globally to contribute to AI safety research and be evaluated on their actual work. Strong hackathon projects are incubated through Apart Lab into peer-reviewed publications, and participants land jobs at leading AI safety organizations. The sprints also produce direct research value: original evaluations, benchmarks, and safety tools that are used by labs like Anthropic, OpenAI, and the UK AI Safety Institute. The causal chain is: low-barrier hackathon access -> research pilots -> talent identification -> publications and career transitions -> expanded AI safety research capacity at frontier labs and policy bodies.
ML & AI Safety Updates, abbreviated MLAISU, was a weekly content series produced by Apart Research starting in approximately September 2022. The series was created to address a gap in the AI safety information landscape: while existing newsletters focused on deep examinations of specific topics, MLAISU aimed to provide timely, broad weekly updates on what was happening across the entire field of alignment and ML safety research.
The series was available in three formats: a written newsletter hosted at newsletter.apartresearch.com, a podcast distributed on Spotify, and short YouTube videos of 4 to 6 minutes each summarizing recent developments. Topics covered included mechanistic interpretability, scaling oversight, robustness, AI legislation, large language model behavior, and opportunities in the AI safety space such as hackathons, courses, and conferences.
MLAISU was created and led by Esben Kran, co-director of Apart Research, along with Thomas Steinthal and Sabrina Zaki. It was published under Apart Research, an organization that also runs large-scale AI safety research hackathons (called Apart Sprints) and supports the development of AI safety research talent globally…
By providing timely weekly summaries of the latest alignment and ML safety research across multiple accessible formats (newsletter, podcast, YouTube), the series aimed to keep a broad audience of researchers, students, and interested practitioners informed and engaged with the AI safety field. Better-informed researchers and a larger community aware of open problems and opportunities would accelerate progress on AI safety, increasing the chances of safe AI development outcomes.
> **Minimum funding · $70,000** - **Lead researcher: $45,000.** Six months of full-time research leadership across both phases, including audit and preparation of CPC source materials, behavioral coding and controls, protocol design, prospective replication and perturbation testing, recruitment and onboarding of technical collaborators, coordination of mechanistic investigations, synthesis, and research writing. - **Mechanistic interpretability collaborators: $10,000.** Paid research stipends or contracts for at least two researchers who already work with inspectable language models and can bring appropriate mechanistic expertise to the study. Funding compensates their research time; model access and compute are budgeted separately rather than assumed. - **Open-weight model access, compute, and instrumentation**: **$5,000.** Compute, storage, model hosting or access, experiment tooling, and model-specific instrumentation required to reproduce the behavioral contrasts in inspectable systems and conduct technically justified internal analyses…
Minimum scope · $45,000 | Budget item | Amount | |---|---| | Researcher support | $16,000 | | Frontier-model API credits | $9,000 | | Real-execution / sandbox compute | $6,000 | | Human adjudication and expert audit | $4,000 | | External red-team / methodology review | $2,500 | | Experimental infrastructure, storage and hosting | $2,000 | | Held-out confirmatory evaluation | $3,000 | | Failed-run / provider / runtime contingency | $2,500 | | Total | $45,000 | Minimum scope delivers: calibration-validity study, real BashArena execution, out-of-band ground truth, shadow-probe evaluation, evidence-corruption measurement, budgeted probing and priced deferral, A0 · A2 attacker conditions, human validation, and a frozen confirmatory evaluation. Ideal scope · $85,000 total…
Dataset Generation & Curation: Acquiring localized authentic audio and purchasing API credits for Google Cloud TTS and ElevenLabs. This will be used to programmatically generate 1,300 synthetic samples across six regional scam archetypes. Infrastructure & Deployment: Enterprise-tier cloud compute is needed for training the VGGish-LSTM acoustic models. This budget will also cover hosting the production-ready detection API on Render. Adversarial Harness & Red-Teaming: Funding VOIP and WhatsApp routing infrastructure is required. This budget will also cover low-latency LLM inference, utilizing models like Gemini 2.0 Flash, to build the live real-time voice agent test pipeline. Developer Stipends & Research Compliance: Compensating dedicated time to build the safety layer architecture using Python. Funding will also ensure all outputs align with the Zimbabwe Cyber and Data Protection Act and POPIA.
Minimum ask (~$5,000) funds the focused proof of concept only: The VBIOS brick-and-recover PoC and the fail-closed lease and promotion-gate prototype · roughly $3000 engineering, $1500 sacrificial cards and SPI recovery rig, $500 for cloud compute. Ideal Ask(~$50,000) Engineering / research time (~3 months FTE-equivalent) to build the hardware actuator, the AUTHORIZE lease and promotion gate, integration and tests · approx. $25,000. Headless (non-display) compute GPU for the RSI testbed and the cross-architecture trojan run currently blocked on the display-GPU kernel watchdog - approx. $6,500. Sacrificial GPU cards for destructive VBIOS brick-and-recover testing, plus external SPI flash programmers and a recovery rig · approx. $4,500. Cloud GPU hours for larger training/eval and cross-architecture replication - approx. $4,000. Out-of-band cutoff hardware, a networked PDU plus relay/microcontroller test bench · approx. $1,500. Independent security review / red-team of the gate and lease design · approx. $4,500. Contingency and dissemination (reference-architecture write-up, travel to present) · approx. $4,000.
- Lead researcher stipend, 6 months part-time (~50%), including benchmark authoring: $18,000 - LLM-judge API costs (two-judge scoring, multi-seed, 6+ models): $3,000 - GCP Confidential Space + confidential-GPU compute for 70B-class enclave runs: $7,000 - Contingency: $1,000
MINIMUM, $8,000, four months, workstream 1 only, 7B scale: Researcher stipend, 4 months at $1,000/mo: $4,000 Compute for the true-negative organism suite, null distributions and probe re-runs (~750 H100-hrs at ~$2.90, ~600 A100-hrs at ~$0.90): $2,700 Inference API for candidate harvesting and LLM-judge validation: $800 Storage and public release (adapters, dataset cards, mirrors): $300 Contingency for failed runs and spot price variance: $200 IDEAL, $35,000, six months, all three workstreams: Researcher stipend, 6 months at $1,500/mo: $9,000 Compute for 14B organism training across several construction methods, five-seed replication and multi-channel attack sweeps (~5,400 H100-hrs at ~$2.90, ~2,150 A100-hrs at ~$0.90): $17,600 Inference API for automated auditor agents, blind-audit runs and judge validation at scale: $3,000 External blind-audit panel, 4 independent auditors at $500: $2,000 Benchmark hosting and public release: $1,000 Conference travel and registration, one venue: $2,000 Contingency: $400
MINIMUM: $22,000 (training-mechanism experiment, 3 months, 1 Oct to 31 Dec 2026) - $18,000 researcher time, 3 months at $6,000/month - $2,000 GPU: three organism families at 7B, with reruns - $800 inference: full battery across the new organisms - $200 storage and artifact hosting - $1,000 contingency IDEAL: $50,000 (full envelope, 6 months, 1 Oct 2026 to 31 Mar 2027) Months 1-3: training-mechanism experiment Month 4: scale arm on Qwen2.5-32B-Instruct Month 5: probe-transfer experiments Month 6: reproduction pass, packaging, preprint - $36,000 researcher time, 6 months at $6,000/month - $3,000 GPU: organism construction at 7B, all mechanisms - $5,000 GPU: scale arm on Qwen2.5-32B-Instruct, tuning and inference - $1,500 white-box probing: activation capture and storage - $1,000 probe-transfer experiments - $1,500 packaging, docs, third-party reproduction pass - $2,000 contingency
Researcher time, 4 months full-time: $10,000 TDX CPU instances (Alibaba Cloud g8i) for the attestation path: $2,000 Confidential GPU hours (H100/H200 with TDX + NVIDIA CC) for GPU-side binding: $2,500 Standard GPU hours for re-running the forgery suite and overhead measurement: $2,000 Open-source release, documentation, publication costs: $1,500 Contingency: $2,000 Minimum ($12,000, 3 months): researcher time $5,500, compute $6,500 (same items, no overhead measurement). Deliverable and acceptance criterion unchanged.
- Cloud GPU compute for extraction, steering, and cross-model experiments (rented A40 / 4090-class instances) - Model inference and judge API credits (LLM-as-judge scoring and hand-label validation) - Extended experiments across additional open models and companion fine-tunes - Development of the deployable audit-probe monitor and open-source tooling - Researcher stipend and living costs for continued full-time work on the project - Miscellaneous: storage, dataset construction, and small annotation costs
Based on my analysis and research I forsee this as two version with highest probability of producing desirable outcomes. 1.Small version with focus on the core proof and tool ($5,000): about 2 to 3 months part-time stipend ($3000) and basic cloud GPU ($1,500) and the remaining as a buffer for miscellaneous costs (eg. Data Generation etc.) This properly gives one AI I trained with a known hidden loyalty, one matching clean AI, and a short paper. 2. Big version with exhaustively tested tool: My stipend, about 6 months, this as main research work: $10,000, Cloud GPU rental (to train the AIs with hidden loyalties and their clean controls, and to run the tests): $4,000, making the open tool and hosting the models/data (Hugging Face, Zenodo): $1,000 Paper costs and a buffer for extra compute: $2,000
Minimum tier, $5,000. Deliverable is the generality matrix plus pre-registered thresholds validated on blinded organisms, which is a complete result and directly answers the calibration limitation. It does not include non-LoRA installation methods, scales above 7B, the adversarial red-team loop, or the public benchmark. All contributor time is donated. Cloud GPU for organism training and evaluation sweeps across the matrix: $3,600 LLM judge API credits, replacing the keyword-based behavioural classifier currently validated at 0.81 agreement: $600 External construction of a small blinded organism set, so calibration and evaluation do not share a designer: $500 Model and artifact storage: $300 Existing contribution, not billed: Google Colab Pro student tier, 70 compute units per month, and donated researcher time from three contributors. Ideal tier, $11,400. Adds the four items above, extending the matrix to non-LoRA installation and larger scales, measuring the adversarial exchange rate, and releasing the public Dormancy Benchmark…
$5,000 · Researcher time: expanded legal research, framework development, scenario testing, revisions, and final publication $3,500 · Expert review: honoraria for EU AI Act, regulatory enforcement, and incident-response experts $2,500 · Research assistance/source validation: incident reconstruction, citation checking, comparative regulatory research $2,000 · Stakeholder interviews and validation: structured practitioner review and synthesis $1,000 · Publication/dissemination: document design, accessibility, and distribution materials $1,000 · Contingency/additional specialist review
Minimum grant $15,000: Method validation and comparative pilot • Research design and coordination: $4,500 • Evaluation-runner development: $2,500 • Three independent annotators and inter-rater reliability study: $3,000 • Legal review for Colombia, the United States, and the United Kingdom: $1,800 • Model API access and repeated experiments: $1,200 • Data publication, technical documentation, and report production: $1,000 • Contingency: $1,000 This would produce a validated scoring rubric, nine jurisdiction-specific scenarios, a limited multi-model study, an annotation handbook, reproducible evidence, and a research report. Additional $15,000 for the ideal grant: Decision-ready evaluation package • Expand from 9 to 18 high-stakes scenarios: $2,500 • Expand testing to five models, three executions per scenario, and approximately 270 multi-turn conversations: $2,500 • Additional human annotation and reliability analysis: $2,500 • Engineering for automated experiment execution, comparison, and evidence reporting: $3,000 • Twelve consultations with AI-governance, legal, auditing, or public-sector practitioners across the three jurisdictions: $1,800…
API/inference costs for expanded model evaluation (more models, more languages): $2,000 Native-speaker translation verification (Hindi, Hinglish, Vietnamese): $2,000 Cloud hosting for the platform/demo: $1,000 Dataset expansion (additional harm categories and prompts): $1,500 Development tools and miscellaneous (domain, storage, dependencies): $500 Buffer/contingency: $1,000
GPU compute (primary): $6,000 · Multi-model experiments across Llama 3/3.1 (8B, 70B), Qwen 2.5 (7B, 72B), Mistral (7B). Each model needs ~10 GPU-hours for the full pipeline (concept extraction, 5,000+ trial dataset, gradient computation, causal sweep, probe comparison). Estimated 60-80 GPU-hours total on A100/H100. GPU compute (extension): $3,000 · SAE-feature experiments: extract sparse autoencoder features as naturally-arising "concepts," repeat the threshold hierarchy and steering tests. Tests whether our results are paradigm-specific or general. Cloud storage and experiment tracking: $500 · Weights & Biases or equivalent for reproducibility; artifact storage across runs. Conference/workshop submission: $1,500 · Registration, formatting, and potential travel for presenting at a venue like the NeurIPS Safe & Trustworthy LLM workshop or AAAI AI Safety track. Open-source tooling: $1,000 · Package the gradient-difference steering construction and the corrected identification protocol as a reusable library. The methodological corrections (norm-matched controls, ΔAUC endpoint, familiarity-matched distractors) should be easy for others to adopt…
Human coders: 3 coders, 2,000 texts each, about 40 hours each at $40/hour: $4,800 Coder training and a 150-text pilot round, 6 hours each at $40/hour: $720 Data collection, about 15,000 comments across re-sampling, new retirements and new product events: $200 Model API costs: the frontier model arm across at least three labs, and three reads of every new comment: $2,000 Researcher time, half time for 6 months at $10,000/month full-time equivalent: $30,000 Total: $37,720 The minimum ask ($21,070) covers items 1 and 2 of the plan: human coders and training ($5,520), re-sampling ($50), model reads ($500), and researcher time at half time for 3 months ($15,000).
Ideal ask, USD 35,000. Released in three tranches, so that the availability risk below is yours to stop rather than yours to absorb. Tranche 1, USD 5,000 on acceptance. Replication on a second base model and recipe, with seeds. Tranche 2, USD 12,000 once the replication and the v_proj test are in the repository, with verify.py extended to recompute their numbers. Tranche 3, USD 18,000 once the probe transfer test, the benign adapter bank probe and the threat-model section are committed, again with verify.py recomputing what can be recomputed. The preprint follows from those, but the trigger is the artifacts, not the posting. Line items within that total: USD 30,000. Researcher time, three months at 30 hours a week. USD 1,500. Rented A100 or H100 hours. Rank-16 LoRA on a 7B base trains in minutes, so the organisms are cheap. This covers the second base model, the seeds and the evaluation sweeps. USD 3,000. Model API and inference for the behavioural audits, which is the expensive compute here rather than the training. USD 500. Preprint and workshop submission costs. Minimum ask, USD 5,000…