Posted on by omarmetwally

Internal Medicine: Distilled

By Omar Nabil Metwally, MD
Internal Medicine Physician
First published 30 August 2026 · Revised and expanded with references

One of the most important classes that I attended as a Clinical Informatics Fellow at UCSF was a course on clinical research. This reflection surprises me in hindsight because I long associated “Clinical Informatics”, which was then a new medical specialty (I was in UCSF’s second class and their third Clinical Informatics Fellow), with the intersection of technology and patient care — a niche for technologically-affine physicians capable of transforming healthcare, if only (so the reasoning) they connected with the right decision makers, got on the right Teams calls, and joined the most elite mailing lists. I was not expecting to sit down almost 10 years later to write a blog post about what I then considered the driest of subjects: biostatistics.

In retrospect, I had a lot to learn, and I could have learned those valuable lessons with a more open and receptive mindset. I consider Professor Pletcher’s course significant in two regards. First, it gave me an opportunity to think deeply about the tests that doctors (especially non-surgeon physicians) routinely order on behalf of their patients and are expected to interpret and act upon correctly. Second, it planted the seed for the realization, many years after my time at Parnassus, that the specialty of Internal Medicine boils down to one concept: a priori likelihood.

There are two types of doctors that take care of people: physicians and surgeons. Physicians — doctors like internists and pediatricians — do a bulk of their work in the cognitive realm, which often takes the form of making a short-list of diagnoses and honing in possibilities (in medical jargon, this is called making a differential diagnosis). This is in contrast to surgeons, whose work tends to be more operative in a hands-on, physical sense than a physician. A physician who does not understand the tests they are ordering, the diagnostic consequences of their choice of test, and how to interpret the results, is like a surgeon without the physical skills necessary to perform a particular operation. I’m deliberately over-simplifying to make the point that despite the fact that test ordering and interpretation has traditionally been a core competency of physicians in general, I encounter an alarming number of physicians, new and experienced alike, who lack the skills to understand what they’re ordering (or not ordering) and how to correctly interpret positive and negative results. Equally alarming, and I suspect related, is organizational policies that narrowly prescribe diagnostic protocols in order to move a patient from point A to point B on the healthcare labyrinth — in the process, stripping physicians of what little autonomy they still have to exercise their clinical judgement.

The objective of this writing is to capture that lesson for any doctor who will ever order (or decide not to order) a test and be expected to interpret the result — if they are lucky to even have the choice.

The order that sounds correct

A middle-aged man presents with months of progressive hip and buttock pain. The pain is worst in the first hours after waking and after prolonged sitting. It wakes him at night. He has a years-long history of intermittent low back pain, a personal history of eosinophilic esophagitis, and a family history of autoimmune disease. Plain films show mild bilateral hip osteoarthritis — premature for his age. He asks for a rheumatology referral.

A reasonable-sounding plan: order an ANA and rheumatoid factor first. If negative, hold the referral. If positive, refer.

That plan has the shape of good medicine — screen, then escalate. It is also, for this presentation, close to backwards. Two errors are stacked inside it, and the second one is the more instructive.

Error one: the tests cannot see the disease

The clinical question here is axial spondyloarthritis. AxSpA is a seronegative spondyloarthropathy — and “seronegative” is not decoration. RF is expected to be negative. ANA has no association with the disease and appears nowhere in the ASAS classification criteria [1]. No result either test returns can move the probability of axSpA in either direction.

So the proposed gate was built from tests that cannot detect the condition being gated. A negative result would have been recorded as reassurance while carrying zero information about the actual question.

Error two: predictive value belongs to the patient, not the test

Here is the concept the wards tend to erode: sensitivity and specificity are properties of the assay. Positive and negative predictive value are properties of the encounter. The same ANA, on the same analyzer, means something entirely different depending on who is in the chair.

Do the arithmetic once and it inoculates you.

Worked example: the ANA in this patient

ANA is positive in roughly 20–30% of healthy adults at a 1:40 titer, and 10–15% at 1:80 [2]. Our patient has monoarticular hip pain with a mechanical mechanism, an axial pain pattern, and no features of connective tissue disease. Be generous and set the pretest probability of an ANA-associated disease at 0.5% — 1 in 200.

Per 1,000 similar patients Test positive Test negative
Disease present (5) 5 0
Disease absent (995) ~120 ~875
Total ~125 ~875
Assuming ~95% sensitivity and ~88% specificity at a 1:80 cutoff [3].

PPV = 5 / 125 = 4%. Twenty-four out of every twenty-five positives are false. A positive ANA in this patient is not a signal — it is noise wearing the costume of data.

And the costume is expensive. A positive ANA in the chart is never inert: it cascades to an ENA panel, a dsDNA, a “possible early connective tissue disease” note, a referral placed for the wrong reason, a patient reading about lupus at 2 a.m., and a label copied forward through every future chart review. The harm of a low-value test is rarely the test. It is the machinery the result switches on.

Worked example: the test that actually earns its place

Now run HLA-B27 through the same patient. Sensitivity is roughly 85% for axSpA — best established for radiographic disease, somewhat lower for non-radiographic axSpA and in non-European ancestries — with specificity around 90%, capped by a background allele frequency of roughly 6–8% in US populations of European descent (and higher still in Scandinavia) [4]. That yields LR+ ≈ 9 and LR− ≈ 0.15.

Given the inflammatory pain pattern, morning predominance, hip involvement, male sex, and family history, a pretest probability of 35% is defensible (pretest odds ≈ 0.54).

  • Positive: 0.54 × 9 = posterior odds 4.9 → ~83% post-test probability. Decision-changing.
  • Negative: 0.54 × 0.15 = posterior odds 0.08 → ~7% post-test probability. Also decision-changing, in the opposite direction.

That is the signature of a test worth ordering: both results move you substantially, from an intermediate starting point. Note the dependency, though — run that same B27 on unselected back pain with a 2% pretest probability and the PPV falls under 20%. The likelihood ratio is a property of the assay; the usefulness is a property of your clinical reasoning.

And the tests that are neither: CRP and ESR

Worth naming because their prominence in “inflammatory workups” oversells them. CRP is elevated in only 40–50% of axSpA — a normal value carries a likelihood ratio near 1 and excludes essentially nothing. Their value here is prognostic (elevated CRP is one of the strongest predictors of radiographic progression, alongside baseline syndesmophytes and smoking [5]), as a baseline for treatment monitoring and ASDAS scoring, and for flagging infection. Diagnostically, they are close to inert. Order them knowing which job they are doing.

A worked example everyone lived through: the COVID home test

During the pandemic the federal government mailed free rapid antigen tests to any household that asked — well over a billion tests moved through federal programs before distribution wound down and was finally suspended in March 2025 [6]. Today the same tests sit on pharmacy shelves at $10–20 a box, and most people with a cough don’t bother. That shift is the thesis of this essay playing out at national scale — though the arithmetic contains a surprise worth sitting with.

At the Omicron peak, roughly half of symptomatic adults with acute respiratory symptoms actually had COVID: in CDC’s outpatient surveillance network, 56% of symptomatic adults tested positive during Omicron predominance [7]. Rapid antigen tests in symptomatic people carry a pooled sensitivity of about 73% and a specificity of about 99.6% [8]. Per 1,000 symptomatic people at 50% prevalence: 365 true positives against 2 false ones — a positive predictive value near 99.5%. The test was superb, and it was superb because of who was taking it.

Today, the same symptoms in the same person carry perhaps a 10% probability of COVID in a typical week — the rest is rhinovirus, RSV, influenza, and everything else — and closer to 3% in a deep trough between waves. Same assay, same operating characteristics. At 10% prevalence: 73 true positives against about 4 false ones — PPV around 95%. At 3%: PPV around 85%.

Here is where the intuition usually goes wrong, and it is the most instructive moment in this essay. The PPV fell — but nowhere near as far as it did for the ANA in our patient, whose PPV was 4% at low pretest probability. Why the difference? Specificity. The antigen test generates false positives at a rate of about 0.4%; the ANA at a 1:80 cutoff generates them at about 12%. When prevalence drops, a test with a fraction-of-a-percent false-positive rate degrades gracefully, and a test with a double-digit false-positive rate falls off a cliff. Pretest probability sets your ceiling; specificity determines how fast you fall from it. A positive home COVID test today is still probably real. A positive ANA in a patient without connective-tissue-disease features almost never is.

One more property worth internalizing: the sensitivity of a strategy is not the sensitivity of a test. A single antigen test on the first day of symptoms detects only about 60% of infections, but two tests taken 48 hours apart detect over 93% [9] — which is exactly why the FDA recommends repeat testing after a negative result [10]. Operating characteristics belong not just to the assay but to how you deploy it.

So why did universal home testing wind down? Partly for structural reasons that have nothing to do with epidemiology: the public health emergency ended in May 2023 and took the insurer free-test mandate with it, federal distribution stopped, and the tests now cost real money. But the clinical logic tracks the third question of this essay: will the result change what I do next? Under CDC’s unified respiratory-virus guidance, isolation is now symptom-based — stay home until improving and fever-free for 24 hours — regardless of which virus you have [11]. For a healthy, low-risk adult with mild symptoms, a positive test changes little about their own care.

And notice that the answer flips right back for the people in whom question three is still answered yes: anyone eligible for antiviral therapy, where a positive result starts a five-day treatment clock [12]; anyone about to sit down with an immunocompromised relative; healthcare workers and congregate settings, which the relaxed guidance explicitly does not cover. The newer over-the-counter combination flu/COVID tests sharpen the point further — in a high-risk patient, distinguishing influenza from COVID selects between two different antivirals. Same test, same prevalence, opposite recommendation — because the question being asked is different.

We ran a natural experiment on 300 million people, and the lesson was the one Professor Pletcher was teaching in a classroom on Parnassus: the test never changed. Who we gave it to, and what we planned to do with the answer, changed everything.

The three questions that replace the panel

The fix isn’t memorizing which panel goes with which complaint — panels are how we got here. It’s a habit, asked in order, before anything is ordered:

  1. What disease am I testing for? Named and specific, out loud. Not “inflammatory something.” If you can’t name the disease, you can’t choose the test — tests are only interpretable against a named hypothesis.
  2. What is my pretest probability — even crudely? “Under 5%, coin flip, or over 50%” is enough resolution to change behavior. The discipline is committing to a number before the result exists. A probability estimated afterward isn’t an estimate; it’s a rationalization.
  3. Will either result change what I do next? If positive changes nothing and negative changes nothing, the test’s only outputs are cost, noise, and cascade risk.

Run the case through them. Disease: axSpA. Pretest probability: intermediate. Will ANA/RF change the next step? No — the MRI and the referral were indicated regardless of serology. Question three deletes the ANA in about four seconds. It also reveals something sharper: the problem wasn’t only which tests formed the gate. There should have been no gate.

Why good physicians make this error

This is not stupidity. It is a systems failure with a cognitive assist, and you should recognize the ingredients, because they will be your working conditions:

  • Order sets are frozen decisions. Someone decided once that ANA + RF = “rheum labs,” and the EHR has been re-making that decision on autopilot ever since. Every checkbox panel is a colleague from the past overriding your present reasoning — sometimes correctly, and you won’t know which times unless you look.
  • Covering is medicine at low resolution. An unfamiliar chart, an intermediary, a message queue. Pattern-matching to “joint pain → rheum panel” is what cognition produces under those constraints. The antidote isn’t heroic effort; it’s cheap heuristics that survive low resolution — which is what the three questions are.
  • “Baseline” is a thought-terminating word. Nobody argues with a baseline. But a baseline is just a test whose pretest probability nobody bothered to state.

And the sequel matters. When this reasoning was raised — politely, in writing — the physician reviewed the chart and ordered the appropriate labs within two hours. That responsiveness is the trait to emulate. The initial error was ordinary; the rapid update was excellent medicine. If you take a villain from this story, you’ve misread it. The villain is the reflex, and every one of us has it.

The distillation

A test result is not information about the patient. It is information about the patient conditional on why you ordered it.

The test doesn’t know who you ordered it on. You do. The entire interpretive weight of every result you will ever receive rests on the probability you assigned — explicitly, or by default — before you clicked the order.

Assign it explicitly. Every time. Ten seconds, and it is the highest-yield ten seconds in diagnostic medicine.

References

  1. Rudwaleit M, van der Heijde D, Landewé R, et al. The development of Assessment of SpondyloArthritis international Society classification criteria for axial spondyloarthritis (part II): validation and final selection. Ann Rheum Dis. 2009;68(6):777–783.
  2. Tan EM, Feltkamp TE, Smolen JS, et al. Range of antinuclear antibodies in “healthy” individuals. Arthritis Rheum. 1997;40(9):1601–1611.
  3. Leuchten N, Hoyer A, Brinks R, et al. Performance of antinuclear antibodies for classifying systemic lupus erythematosus: a systematic literature review and meta-regression of diagnostic data. Arthritis Care Res (Hoboken). 2018;70(3):428–438.
  4. Reveille JD, Hirsch R, Dillon CF, Carroll MD, Weisman MH. The prevalence of HLA-B27 in the United States: data from the US National Health and Nutrition Examination Survey, 2009. Arthritis Rheum. 2012;64(5):1407–1411.
  5. Poddubnyy D, Rudwaleit M, Haibel H, et al. Rates and predictors of radiographic sacroiliitis progression over 2 years in patients with axial spondyloarthritis. Ann Rheum Dis. 2011;70(8):1369–1374.
  6. US Department of Health and Human Services. Federal at-home COVID-19 test distribution program (COVIDTests.gov), January 2022; distribution suspended March 2025.
  7. Kim SS, Chung JR, Talbot HK, et al. Effectiveness of two and three mRNA COVID-19 vaccine doses against Omicron- and Delta-related outpatient illness among adults, October 2021–February 2022. Influenza Other Respir Viruses. 2022;16(6):975–985.
  8. Dinnes J, Sharma P, Berhane S, et al. Rapid, point-of-care antigen tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. 2025;CD013705.
  9. Soni A, Herbert C, Lin H, et al. Performance of rapid antigen tests to detect symptomatic and asymptomatic SARS-CoV-2 infection: a prospective cohort study. Ann Intern Med. 2023;176(7):975–982.
  10. US Food and Drug Administration. At-home COVID-19 antigen tests: take steps to reduce your risk of false negative results. FDA Safety Communication, August 2022.
  11. Centers for Disease Control and Prevention. Respiratory virus guidance. March 2024.
  12. US Food and Drug Administration. Nirmatrelvir-ritonavir (Paxlovid) prescribing information: treatment of mild-to-moderate COVID-19 in patients at high risk for progression, initiated within 5 days of symptom onset.

Clinical details are composite and deidentified. Test characteristics are approximate, vary by assay, cutoff, population, and timing, and are cited to their principal sources above — the arithmetic is illustrative, and the habit is the point.

Posted on by omarmetwally

On the economics of knowledge creation and sharing

Omar Metwally, MD 
omar.metwally@gmail.com
University of California, San Francisco

DDASH - Ethereum Operating System 
for Knowledge Creation and Sharing
==========================================
    Github repository
    -----------------
    Project website
    -----------------
    First Draft

Abstract

This work bridges the technical concepts underlying distributed computing and blockchain technologies with their profound socioeconomic and sociopolitical implications, particularly on academic research and the healthcare industry. Several examples from academia, industry, and healthcare are explored throughout this paper. The limiting factor in contemporary life sciences research is often funding: for example, to purchase expensive laboratory equipment and materials, to hire skilled researchers and technicians, and to acquire and disseminate data through established academic channels. In the case of the U.S. healthcare system, hospitals generate massive amounts of data, only a small minority of which is utilized to inform current and future medical practice. Similarly, corporations too expend large amounts of money to collect, secure and transmit data from one centralized source to another. In all three scenarios, data moves under the traditional paradigm of centralization, in which data is hosted and curated by individuals and organizations and of benefit to only a small subset of people.

1. Introduction

In its current siloed state, data is a liability rather than an asset. The value of data depends on its quantity and quality. Organizations, including corporations, government, and academia, have few incentives to share data outside the context of selling it. For instance, advertisers use data procured  from individuals’ browsing history and social media use (via internet service providers, social media and search engines) to create detailed profiles of individuals’ online behavior and spending habits and more effective sell products to unknowing consumers. While this paradigm fits naturally into a capitalistic society, these economics of data collection and transfer do not facilitate the generation or sharing of knowledge in the academic setting.

A typical university-based research group depends upon external funding to support its research activities. These funds often originate from governmental bodies, philanthropic organizations, or corporations and are difficult to secure [1]. Only a small minority of tenure track scientists ever becomes principal investigators, and a lab that is productive today can become defunct tomorrow if its principal investigator is unable to secure funding for laboratory equipment and supplies such as microscope parts, reagents, and to compensate technicians and trainees [2]. Principal investigators spend a majority of their time writing grant applications rather than participating directly in the process of knowledge generation [3].

It is often said that publications are the currency of academia. The maxim “publish or perish” applies to most research groups, whose work culminates in peer-reviewed publications with publication fees commonly amounting to several thousand dollars [4]. Moreover, these peer-reviewed publications are heavily biased toward so-called “positive results,” in which mathematical correlations between variables are described [5]. The vast majority of data produced by scientific researchers do not refute the null hypothesis; in a best case scenario, they are deemed “negative results,” and are discarded; in a worst case scenario, they are data that can’t be replicated, verified, or are outright fraudulent [6]. The result is the modern-day academic machinery. This severely flawed system, a victim of many conflicting economic forces, results in a tremendously inefficient workflow in which most grant money is wasted in the form of negative, and therefore unpublishable, results. Principal investigators spend a majority of their time trying to secure funding. The ultimate winner is the $10 billion business of academic publishing [6]. In this reality, data with the potential to produce vast knowledge is rendered into a vastly wasted opportunity to exponentially build on communities’ resources. Individuals’ roles are minimized by the centralization of resources in the hands of a privileged few.

2. Background

While the term “blockchain” has been touted to near-hysteria in popular media in the context of initial coin offerings and get-rich-quick schemes, an understanding of this data structure’s logic reveals the tremendous and fascinating socioeconomic implications of storing data on blockchain. In its most simplified form, a blockchain is a ledger [7]. The reason for blockchain’s natural association with financial derivatives lies in its ability to mathematically prove the authenticity of data and demonstrate proof of stake and proof of work [8].

The starting port for these use cases is the typical consumer, who is separate from (and often completely unaware of) the data collected about him or her. For instance, a customer’s online behavior is collected and used to up-sell the customer as much as algorithmically possible [9]. Customers have  nothing to gain (and a few thousand dollars each year in extra spending to lose) from such data, which companies can sell to data brokers and merchants [10]. Analogously, the majority of taxpayers have no access to — and oftentimes no way to directly benefit from — publications funded through research that ends up property of academic journals [11, 12].

2.1 Case Study: Proof of Stake

Consider a research lab living from grant to grant, sifting through negative results to find crumbs of publishable positive results. If its lab notebooks were stored in the form of a blockchain, every experiment conducted, every machine learning model and dataset, and every clinical trial would generate data that lives on the blockchain as a cryptographic asset. Also referred to as “coins” and “tokens,” these cryptographic assets have inherent value because they are perfect receipts of the existence and transfer of data [13]. Never before in history has such a perfect ledger existed [14, 15]. On the blockchain, a relatively worthless set of negative results generated by a research lab becomes, when combined with negative results from thousands of other research groups, a trove of extremely valuable scientific data which can be traced to its owners whenever and however it is utilized. This large collection of negative results can become the source of unexpected positive results.

Moreover these blockchain-hosted data take on a new life as a financial derivative [16, 17]. These cryptographic assets, perfect receipts of the creation and movement of knowledge, can be traded by third-parties analogously to the way a company’s common stock is bought and sold on private and public marketplaces, albeit without the same regulations and on a different scale [13]. These tokens enable individuals, small and large groups alike to be compensated for their services in ways that are impractical or impossible in traditional economies [18]. Rather than relying on the slow and inefficient process of securing funding through grants, research labs can codify contracts on the blockchain to allow third-parties to bid for services and products rendered, on the metadata (what kind of knowledge research labs generate through their scholarly activities), and allow third-parties to become stakeholders in a research group’s success by directly benefitting from these research activities. For instance, if I believe that a particular group is contributing to science and society in a positive way, I can economically support this group by donating computing power and electrical energy to support the integrity of their lab notebook-turned-ledger, or by trading fiat for tokens representing proof of stake in their scholarly activities. What are today opportunities exclusive to accredited investors and institutions will become abundant opportunities for individuals to influence how perceived value circulates through society.

2.2 Case Study: Proof of Work

Consider the United States healthcare system, which still excludes millions of Americans from access to healthcare and financially ruins even more [19, 20]. Insurance companies are able to impose high premiums simply because they can. This is the logic of a capitalistic society, and insurance companies alone enjoy the benefits of owning valuable health data to their fullest extent — at the expense of those whose health data was collected [20, 21]. Imagine, on a smaller scale, a radiology group that puts a copy of every imaging study they do on a blockchain, along with a timestamp, a description of which type of study was done, and why it was performed. In doing so, data that would have otherwise been discarded can be engaged with by third-parties while directly benefiting the radiology group as well. For instance, grassroots-based health insurance co-ops could emerge from these sources of data which are otherwise privy to insurance companies, to the benefit of health consumers, who can undergo imaging studies and receive other healthcare services at a fraction of current costs. Information about which studies are performed — where, by whom, and why, and the result of those studies, can be used to lower healthcare costs while improving health outcomes, rather than raise healthcare costs and increasing profits.

One question that naturally arises, especially in the context of current centralized data paradigms, is: why would healthcare providers be incentivized to make public valuable data that is routinely used by corporations and insurance companies to maximize profits? One powerful force driving healthcare costs upward is the process through which health providers bill patients via insurance companies. Whether ordering relatively common drugs or expensive therapeutics or procedures, healthcare systems rely on administrators whose role is to submit authorization requests to insurance companies for approval to prescribe therapeutics on their patients’ behalf [22]. When a service is rendered in the hospital or in a clinic, a healthcare team is reimbursed a fraction of the amount they bill for, creating a cat and mouse game in which providers continuously bill as high as possible for services rendered with the expectation that they will only receive a fraction of what they bill for, and in which insurance companies place limitations on which drugs and services this will pay for and how much of the cost they will cover [23]. Blockchain would provide an end to this cat-and-mouse game and create a race to the bottom for healthcare costs, through price transparency and elimination of bloated administrative layers that handle authorization requests and billing, while creating a race to the top for healthcare outcomes as this ledger of health services and outcomes would be publicly accessible on a blockchain. Simultaneously, healthcare providers can immediately receive payment for services rendered, and although individual payments may be less, overall profits would increase because payments would arrive immediately and there would be no need for entire departments of administrators whose entire role is to maximally inflate bills sent to insurance companies (and patients, insured and uninsured) and to see these bills through collection.

2.3 Informing current and future medical practice

We may well already have all the knowledge we need to cure many illnesses currently considered incurable [24]. We may well have all the data we need to create intelligent machines that can interpret CT scans, diagnose disease, and synthesize drugs to cure any condition. The reason this knowledge hasn’t culminated in more rapid advancement in healthcare and science is that information is fragmented into pieces, siloed, and ultimately rendered worthless data. Blockchain allows transparent access to data. It would be naive to imply that a data structure will cure society of all its ailments. However blockchain allows data to culminate into extremely valuable information, once at the disposal of a powerful few, now to the benefit of all who become stakeholders by contributing to, interacting with, and propagating data.

3. The need for a ledger of scholarly assets

The need for this project, a protocol for the hosting and sharing of data on a distributed network (“Distributed Data Sharing Hyperledger,” or DDASH), arises from the observations by the above examples, as well as the observation that numerous research groups at UCSF and other academic institutions are working in parallel in their endeavors to create knowledge with little synergistic interaction [25]. How would research group A at UCSF Medical Center know that research group B at the University of Michigan is working to answer the same scientific questions, for instance? Without a transparent glimpse into which resources an organization owns and how they are being used and shared, both research groups miss opportunities for synergistic collaboration, within and among organizations.

Those acquainted with the politics of contemporary academia will be quick to raise several criticisms. Working within the current reality of Google, the most comprehensive collection of information known to humanity as of September 2017, why can’t research groups A and B simply host their digital assets — data and knowledge gleamed from this data — on websites or public databases? And if groups A and B are competing to be the first to publish in academic journals and competing to drink from the same pools of grant funding, why would any research group benefit by sharing the results of experiments that were costly to run before they can reap the benefits of publication and intellectual property [26]? The answer is in blockchain’s ability to capture proof of work and proof of stake in a network’s digital assets. There is nothing to stop a competing research group from stealing these data and benefiting at their competitors’ expense. Hosting data in the form of knowledge on a blockchain elegantly solves this problem through irrefutable mathematical proof of data ownership, transfer, and authenticity [27].

3.1 Distributed Data Sharing Hyperledger (DDASH)

DDASH (link to open source Github repository) is a ledger of scholarly data and knowledge produced by life science, informatics, and clinical researchers at UCSF and other academic institutions. The need for this project arises from the negative impact of data siloing, competition, and counterproductive financial incentives in the academic world on the creation and sharing of knowledge. Concretely, researchers can host data — datasets, experimental results, and machine learning models, among other examples of scholarly knowledge — on the distributed InterPlanetary Filesystem (IPFS) network and record the location of these assets on an Ethereum-based blockchain, along with a description of the asset, when it was created, and who has privileges to access the data.

3.2 Network Architecture

We believe that the IPFS protocol’s combination of security and speed is well suited for this application. IPFS uses content-based addressing, in which a hashing function determines a file’s network address based on the file’s contents [28]. Storing data in the form of a directed acyclic diagram (in this case, a Merkle DAG) results in trees that can be efficiently traversed and queried. IPFS is a peer-to-peer network in which data is continuously circulating through network participants’ machines which are running the client software. Data are rendered permanent by virtue of content-based addressing and persistent by virtue of its peer-to-peer architecture, and data are rapidly accessible without the bottlenecks that Internet Protocol imposes.

3.3 Blockchain as a ledger

The blockchain functions as a decentralized ledger of digital resources and the movement of these resources throughout the network. As the DDASH protocol is formalized, more robust mechanisms for associating IPFS hashes with the owner of the resource and the permissions granted by the owner are necessary. Currently the DDASH protocol accounts for the following elements:

  • IPFS content-addressed hash, which defines the location of an asset on the IPFS network
  • The owner’s public key fingerprint
  • The public key fingerprints of users authorized to access the resource, or a designation as “public”
  • Timestamp

In its current form, DDASH interfaces between the IPFS network and the Ethereum blockchain. One can conceive an alternative version of the DDASH protocol that seamlessly integrates a ledger-based indexing and permission management system, using for example IPFS’s native public and private keys and a native IPFS ledger. Keeping the networking architecture separate from the blockchain has tangible advantages, however, including the versatility of allowing users to create digital assets using any permutation of blockchains, private and public.

3.4 Security

DDASH allows users to manage access to privileged resources using public-key encryption. Public-key encryption allows users to identify themselves on the network using a verifiable public key, which can be used to encrypt resources such that they can only be unencrypted using a  corresponding private key accessible exclusively to the intended recipient. Future versions of the DDASH protocol may feature ways to host resources on private clusters and manage access to these clusters on the blockchain. In doing so, resources are secured by limiting the movement of certain data to a subset of the swarm (network peers), and through a second layer of encryption. This not only allows data to move much more quickly through a network, it also greatly enhances security compared to the antiquated paradigm of data hosted on centralized, and therefore inherently vulnerable, servers. Common sources of wasted IT budgets and wasted productivity, such as forgotten, cracked and stolen passwords, or easily-intercepted HTTP network traffic, are obviated by virtue of the DDASH protocol. What stands between the theoretical underpinnings of this protocol and its implementation in academic centers and healthcare systems is not a question of the feasibility of this technology, but rather, whether legislation governing health information and computing will keep up with emerging trends in computing. Catastrophic beaches of sensitive consumer information, such as the Equifax data breach, have become regular occurrences and urgent reminders of the shortcomings of our antiquated Internet Protocol and undeserved trust in institutions that centralize large amounts of highly sensitive data at individuals’ expense [29].

3.5. DDASH Repository

DDASH is hosted as an open source repository at https://github.com/osmode/ddash.

We intend for this nascent project to illustrate the concepts and the larger vision outlined here while serving as a starting point for a formalized protocol for hosting and interacting with distributed digital assets. We made this a public repository early in the conception of this project in order to allow the codebase to benefit from the technical expertise and creativity of the open source community, and to allow the project to benefit from the rapid and exciting evolution in computing paradigms driven by the blockchain and distributing computing communities.

4. Using DDASH

DDASH currently runs on the blackswan private Ethereum network at 104.236.141.200. It benefits from the open source work produced by the IPFS, Ethereum, OpenPGP, web3.py, and py-ipfs communities.

The Go Ethereum client, web3.py, and py-ipfs Python packages are all prerequisite. The instructions here are for machines running Ubuntu 16.04. A Ethereum node must be connected to the blackswan private network and possess the ability to lock/unlock accounts to send transactions.

4.1 Directory Structure

Start by creating these directories:

mkdir /home/omarmetwally/blackswan
mkdir /home/omarmetwally/blackswan/gnupg
mkdir /home/omarmetwally/blackswan/data

4.2 Genesis Block

To connect to the blackswan network, you’ll need to use the same genesis block defined in genesis.json (see the Github repository). Move this file to /home/omarmetwally/blackswan/ and set your genesis block (you only need to do this once, and you need to install the Ethereum go client geth and Ethereum developer tools first):

 

geth --datadir=/home/omarmetwally/blackswan/data init /home/omarmetwally/blackswan/genesis.json

bootnode --genkey=boot.key

bootnode --nodekey=boot.key

4.3 Go Ethereum client and IPFS daemons

In order to use the web3.py and ipfs wrappers, you’ll need to run geth and ipfs daemons in the background, respectively:

geth --verbosity 1 --datadir /home/omarmetwally/blackswan/data --networkid 4828 --port 30303 --rpcapi="db,eth,net,web3,personal,web3" --rpc 104.236.141.200 --rpcport 8545  console

Be very careful when enabling RPC while your accounts are unlocked. This can lead to Ethereum wallet attacks, hence the recommendation to keep your development environment completely separate from any real Ether you might own.

The above command starts the go Ethereum client on your local machine and attempts to connect to the blackswan server at 104.236.141.200. Remember to set your genesis block according to the above directions. Trying to join this network with a different genesis block (such as the default genesis block) will not work.

Then open a new terminal window or tab and start the ifps daemon:

ipfs daemon

4.4 DDASH command line interface

Once your Ethereum and IPFS nodes are running, your account is unlocked, and you can interact with both clients, start the DDASH command line interface (CLI):

python main.py

                    DDASH
    ::: Distributed Data Sharing Hyperledger :::
    https://github.com/osmode/ddash

    Welcome to the DDASH Command Line Interface.

[1]   ddash> sanity check
      IPFS and geth appear to be running.
[2]   ddash> set directory /home/omarmetwally/blackswan/gnupg
[3]   ddash> new key
[4]   ddash> show keys
[5]   ddash> use key 0
[6]   ddash> show accounts
[7]   ddash> use account 0
[8]   ddash> set recipient your_recipient's_pubkey_id 
[9]   ddash> set file /path/to/clinical/trial/data.csv
[10]  ddash> encrypt
[11]  ddash> upload
[12]  ddash> checkout QmUahy9JKE6Q5LSHArePowQ91fsXNR2yKafTYtC9xQqhwP

The above commands:
1. check if IPFS daemon and Go Ethereum client are running
2. specify working directory (need to have read/write permission)
3. generate a new PGP keypair
4. list all PGP keypairs on your machine
5. uses the first (index 0) keypair as your identity
6. list Ethereum accounts
7. specify index of Ethereum account to use for transactions
8. specify an intended recipient's public key
9. upload the file to IPFS and create transaction containing the hash, user id of the person who uploaded the file, and recipient's public key id (or "public" indicating that it's not encrypted).
10. encrypt file from step 9 using public key from step 8
11. upload file from step 9 to IPFS network
12. check blockchain using IPFS hash as handle

 

4.5 Mining on the blackswan Ethereum network

Mining difficulty is currently relatively easy (1e6) on the blackswan network. Mine Ether by running:

geth --verbosity 4 --datadir /Users/omarmetwally/Desktop/blackswan/data --networkid 4828 --port 30303 --rpc 104.236.141.200--rpcport 8545  --mine console

 

5. Acknowledgements

I’m grateful to my mentor, Dr. David Avrin (UCSF) for his belief in this vision and for his unwavering support. My colleagues, Dr. Michael Wang and Dr. Steven Chan, provided formative feedback during the conception of these ideas. Steven Truong (UC Berkeley) inspired me with his technical creativity. Visionaries such as Vitalik Buterin and Juan Benet, and many brilliant minds contributing to the open source communities they inspired, conceived the technical underpinnings which are allowing these concepts to grow into powerful tools which I believe will transform and modernized academic research. 

6. References

  1. Grover A et al. “The Economics of Academic Medical Centers.” N Engl J Med 2014; 370:2360-2362. June 19, 2014. DOI: 10.1056/NEJMp1403609.
  2. Bohannon J. “Want to be a PI?”Science. June 2 2014. http://www.sciencemag.org/careers/2014/06/want-be-pi-what-are-odds. Accessed: 11 September 2017.
  3. Kaplan K. “A roll of the dice.” Nature 479, 433-435 (2011). doi:10.1038/nj7373-433.
  4. Van Noreen R. “Open access: the true cost of science publishing.” Nature. 27 March 2013. Accessed: 11 September 2017. Available: https://www.nature.com/news/open-access-the-true-cost-of-science-publishing-1.12676.
  5. World Health Organization. “WHO Statement on public disclosure of clinical trial results.” Published: 9 April 2015. Accessed: 11 September 2017. Available: http://www.who.int/ictrp/results/reporting/en/.
  6. Ionnidis JP. “Why most published research findings are false.” PLoS Medicine Published: 30 August 2005. Accessed: 11 September 2017. Available: https://doi.org/10.1371/journal.pmed.0020124.
  7. Narayan A et al. Bitcoin and cryptocurrency technologies. Princeton University Press, 19 July 2016.
  8. Narayan A and Clark J. “Bitcoin’s academic pedigree.” ACM Vol 15:14, 29 August 2017. doi: 10.1145/3134434.3136559
  9. Keyes D. “Amazon looks to gain a machine learning advantage.” Business Insider. Published: 8 September 2017. Accessed: 11 September 2017. Available: http://www.businessinsider.com/amazon-looks-to-gain-a-machine-learning-advantage-2017-9.
  10. Federal Trade Commission. “Data Brokers: A Call for Transparency and Accountability.” Published: May 2014. Accessed: 11 September 2017. Available: https://www.ftc.gov/system/files/documents/reports/data-brokers-call-transparency-accountability-report-federal-trade-commission-may-2014/140527databrokerreport.pdf.
  11. Kimbrough, Julie L., and Laura N. Gasaway. “Publication of government-funded research, open access, and the public interest.” Vand. J. Ent. & Tech. L. 18 (2015): 267.
  12. California State Department of Public Health. “California Taxpayer access to publicly funded research act (Assembly Bill No. 609).” Published: 29 September 2014. Accessed: 11 September 2017. Available: https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201320140AB609.
  13. Nakamoto, S. “Bitcoin: A peer-to-peer electronic cash system.” Satoshi Nakamoti Institute. Published: 31 October 2008. Accessed: 11 September 2017. Available: http://nakamotoinstitute.org/bitcoin/#selection-2203.14-2203.21 .
  14. Aspnes J et al. “Exposing computationally-challenged Byzantine imposters.” Yale University Department of Computer Science. Published: 26 July 2005. Accessed: 11 September 2017. Available: http://cs.yale.edu/publications/techreports/tr1332.pdf.
  15. Boyle TF. “GLT and GLR: component architecture for general ledgers.”  https://linas.org/mirrors/www.gldialtone.com/2001.07.14/GLT-GLR.htm .
  16. Wood G. “Ethereum: a secure decentralized transaction ledger.” http://gavwood.com/paper.pdf.
  17. Buterin V. “Notes on scalable blockchain protocols.” Ethereum Foundation. Published: 31 May 2015. https://github.com/vbuterin/scalability_paper/blob/master/scalability.pdf.
  18. Y Combinator. June 30th 2017. “IPFS, CoinList, and the Filecoin ICO with Juan Benet .” https://blog.ycombinator.com/ipfs-coinlist-and-the-filecoin-ico-with-juan-benet-and-dalton-caldwell/.
  19. Centers for Disease Control and Prevention. “Health Insurance Coverage.” https://www.cdc.gov/nchs/fastats/health-insurance.htm.
  20. Metwally O. “Building smart contract-based health insurance.” Published: 30 June 2014. https://omarmetwally.blog/2014/06/30/building-smart-contract-based-health-insurance/.
  21. Angrisano C et al l (McKinsey Global Institute). “Accounting for the cost of health care in the United States.” Jan 2017. http://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/accounting-for-the-cost-of-health-care-in-the-united-states.
  22. California Department of Health and Human Services. “Treatment Authorization Request.” http://www.dhcs.ca.gov/provgovpart/Pages/TAR.aspx.
  23. Jiwani A et al. “Billing and insurance-related administrative costs in the United States’ health care: synthesis of micro-costing evidence.”  BMC Health Serv Res. 2014 Nov 13;14:556. doi: 10.1186/s12913-014-0556-7.
  24. Hamermesh RG and Guisti K. “One obstacle to curing cancer: patient data isn’t shared.” Harvard Business Review, 28 Nov 2016. https://hbr.org/2016/11/one-obstacle-to-curing-cancer-patient-data-isnt-shared.
  25. Distributed Data Sharing Hyperledger. https://github.com/osmode/ddash.
  26. Fecher B, Friesike S and Hebing M. “What drives academic data sharing?” PLoS One Published: February 25, 2015. Available: https://doi.org/10.1371/journal.pone.0118053.
  27. Ethereum Foundation. “A next-generation smart contract and decentralized application platform.” https://github.com/ethereum/wiki/wiki/White-Paper .
  28. Benet J. “IPFS – content addressed, versioned, p2p file system.” arXiv:1407.3561 [cs.NI].
  29. Gressen S (Federal Trade Commission). “The Equifax Data Breach: What to Do.” Published: 8 September 2017. Available: https://www.consumer.ftc.gov/blog/2017/09/equifax-data-breach-what-do