MetaMed researcher policy
Internal policy document for researchers contracting with MetaMed Research, the personalized medical-research consultancy Alyssa co-founded with Michael Vassar, Jaan Tallinn, and Zvi Mowshowitz. Part 1 covers billing, project workflow, and HIPAA. Part 2 lays out the company’s research philosophy — Bayesian rather than frequentist, suspicious of publication bias, and aimed at giving patients numbers rather than hedged advice. The framing is consciously drawn from the LessWrong rationalist community.
Part 1: Billable hours, authorization procedures, and confidentiality Billable hours policy Before a researcher can work on projects commissioned by patients, he/she must sign our researcher agreement. Trial work on behalf of the company, and company projects, can be done for a short time without signing the agreement.
Researcher hours fall into three categories: Billable to Patient, Billable to Company and Not Billable. The following outlines what constitutes each category.
Billable to Patient (or Company if the Company is the Patient): Initial familiarization with and evaluation of the patient’s case, including communication with the patient’s doctor or our own MDs, PhDs and other PM researchers to gain additional data, specificity and background.
Searching for, acquiring, and reading studies and information relevant to the case. This includes acquiring data from our own internal databases as well as considering evidence later dismissed as unsound.
Analysis relevant to the case.
Location of, and consultation with, experts in areas related to the case information or the case itself.
Writing of the report, analyzing it with the patient’s doctor and our team, receiving the doctor’s sign-off, and modifying the report as requested.
Billable to Company: Entering findings with regard to studies, questions or other information sources into our databases.
Explicitly pre-approved training, including in rationality, statistics, biology, medicine and company-specific skills, software and websites.
Analysis which, although not necessary for the individual patient in question, applies to the general case of which the patient is a subset, and which it is efficient to do in conjunction with the current patient.
Not Billable: Background reading, including books such as Better Doctors, Better Patients, Better Decisions .
Travel time.
Non-purposeful emails and phone calls and poorly planned or non-fruitful activities (this does not include the examination and subsequent dismissal of potential evidence that proves unsound.)
Time spent documenting billable hours.
Unauthorized work. When working for clients, all work must commence with an approved work order form; when working for the company, explicit e-mail authorization is also permitted.
Hours beyond 20 unless a greater hour count is explicitly authorized for a particular project.
For a given project that is not a Company project, Billable to Company hours cannot exceed 50% of Billable to Patient hours. Any additional hours spent on behalf of Company can be stored and billed as part of future projects where the hours are billable expenses relevant to that case, but ask in advance for permission to store such hours.
In addition, you may bill reasonable expenses, including internal expenses if you are using non-free internal resources. Acquire approval in advance for major expenses; for expenses relating to article access, see the section on articles.
Research procedure Doctor meets with patient and a written request for research is made, including the topic, budget and any other preferences.
Research supervisor reads the request and assigns an appropriate researcher if it is accepted.
Researcher asks for any necessary clarifications or additional information. Our own M.D. is contacted for any medical information.
Doctor who made request gives additional information as needed, and our doctor gives background as needed.
Researcher states what is to be investigated, as he or she understands it, at what rate and for how many hours. Doctor signs off on this after obtaining authorization from the payer/patient. This confirms that everyone is on the same page.
When work is near completion, a preliminary report is submitted to our M.D. for approval. M.D. can request changes, ask questions or otherwise do what is necessary to ensure an accurate and complete report that M.D. can sign off on. Should a researcher encounter a question with the potential to require large changes, he/she should check with our M.D. for feedback at that time. At present, this M.D. is Christopher Jensen, who can be reached at cjjensen@gmail.com.
If researcher feels project justifies more time, more time can be requested while issuing the preliminary report. This request will be given to the doctor and patient for a decision.
Once our M.D. signs off on it, the report is given to the patient’s doctor.
Internal projects Internal Projects are designed as aids to our product offerings and to establish a knowledge base. They do not have a particular patient in mind; all possible patients that fit the description given must be considered. All billable hours spent on internal projects are billed to the company, and all steps listed above that refer to the patient or the patient’s doctor would instead refer to the internal source managing the project, by default Zvi Mowshowitz, the Senior Vice President for Research. His explicit authorization via e-mail is sufficient to begin work. Such projects are assumed to require check-in at least once for every 20 hours of authorized work, unless more hours are explicitly authorized.
Article and informational access Many of the company’s employees have associations with various universities that give them access to an array of otherwise unavailable articles, and the company will maintain subscriptions to article and other informational services. Before you purchase an article that costs more than five dollars, check with the company to see if the article is available from us for free. If it is not, check to make sure there are no subscriptions the company would want to buy that would include access to the article. If neither of these is the case and the article is worth its price, go ahead and purchase it as an expense. If such expenses are more than 10% of the cost of your section of the project, check in with company before proceeding.
Note also that the company has access to UpToDate and will in time have access to other similar services as well. If you do not receive these resources at the start, make sure to ask for them!
Deadlines and availability Unless it is otherwise explicitly agreed on, a researcher accepting an assignment is committing to a minimum of twenty billable hours of work within two weeks, and to continue at that rate until the project is completed, with check-ins every two weeks. It is always acceptable to complete projects faster if the hours are authorized, but slower work must be explicitly authorized in advance. If a faster pace is specified, due to the nature of the patient’s case or otherwise — accepting the terms means agreeing to complete work on schedule. If you cannot do so for any reason, you must contact your supervisor at PM immediately. Failure to do so will endanger your ability to receive payment for the current assignment and your ability to receive assignments in the future. PM does not tolerate unreliable researchers.
Availability for work comes in one of two forms: a researcher can choose to be “on-call,” or not. If you are on-call, then you are agreeing to respond in a timely fashion to research requests, putting in your agreed upon number of on-call hours when a job becomes available and is assigned to you. If you are not on-call, you are free to decline a given project if you wish. To what extent a researcher is “on-call” will help determine the hourly rate paid.
Confidential information All patient records and medical information is strictly confidential, and the utmost care must be taken to safeguard such information.
Importantly, in your role as a researcher, and to the extent your assignment requires you to access personally identifiable health information, you are a HIPAA-regulated “Business Associate” to Personalized Medicine. As a Business Associate you are subject to the same HIPAA privacy and security rules as the medical provider treating the subject patient. Before accessing or using personally identifiable health information, you must (a) undergo HIPAA training and (b) execute the company’s standard Business Associate Agreement (“BAA”). Whenever a patient commissions research through Personalized Medicine’s program of services, the company will make reasonable efforts to (a) obtain only “de-identified” health information, which is not protected health information (“PHI”) under HIPAA, and (b) if personally identifiable information is provided, obtain a patient authorization for release. Importantly, even where the patient executes an authorization for release of PHI, you are obligated to comply with HIPAA security provisions.
Part 2: Principles of research Formatting Provide sections with headers, including Abstract, Recommendations, Body and Citations. Files should be in .doc or .docx format (not .pdf).
Rationality and Bayesian vs. frequentist statistics It is impossible to conduct the required research for this job without Bayesian logic and statistics. If you aren’t familiar with Bayes’ Theorem, you should read Yudkowsky’s intuitive explanation before you continue further with this document or with any research work.
We are committed to spreading the art of Rationality and using it to help our doctors and patients make better decisions. Rationality is the advanced art of how to think. It teaches us to intuitively use Bayesian thinking in every aspect of life, account for and overcome the biases that we all have and do that which causes us to win. Rationality teaches you to use the tools at your disposal to achieve your goals and maximize that which you value. If you wish to study Rationality, which is highly recommended, start with the LessWrong Sequences . There is a community dedicated to Rationality at LessWrong .
Academic medical research uses frequentist statistics, looks for ‘statistically significant’ results and treats p-values as serious business. This is better than no statistics, but it creates many errors. Researchers often examine many potential correlations to find the one in twenty with p<0.05. Discussion of the problems arising from frequentist statistics can be found in “ Parapsychology: the control group for science ” and “ Frequentist statistics are frequently subjective .”
We also recommend reading the book Better Doctors, Better Patients, Better Decisions , which illustrates many of these problems in great detail. Another description of many such issues is in The Atlantic ’s “ Lies, Damned Lies, and Medical Science .”
Everything is evidence. Stronger evidence is better. There is no switch where something passes from ‘not evidence’ into ‘evidence.’ Evidence need not take the form of a clinical trial or other formal experiment; it is anything that changes how likely something is to be true. Disregard findings with p<0.01, if they are not strong enough evidence. Use data whose p-value was not calculated, or which was p>0.05, if that data is useful evidence. Always consider your prior — how likely something was to be true before you began looking at the evidence — then adjust your beliefs and findings based on the evidence.
Few doctors and even fewer patients will understand Bayesian statistics. Do not assume that anyone outside the company will understand them; in fact, you should presume the opposite.
Bias Rationality often focuses on bias. Humans have systematic biases that distort their beliefs. While the medical research system helps to combat some of them, it exacerbates others and introduces new biases. A strong understanding of the role of bias is necessary to evaluate evidence. This document will briefly discuss some of the biases most important to medical research.
The most important bias in medical research is publication bias. Studies that do not show positive results are far less likely to be published. This is true even if no one intended to bias the results; no journal wants a finding that two things aren’t linked or that a proposed treatment does not work. This effect is lessened when sample sizes are large.
One will see a steady decline in average effect size as sample size increases. This is the result of publication bias: scientists report the results they found, not the results they failed to find. For this reason, one must be very careful not to overreact to a consensus of findings, and adding together the results from different studies to form a meta-study will systematically overstate effect sizes and degrees of confidence.
Every study is undertaken for a reason. If you don’t understand that reason, you don’t understand the study. A question was asked and a hypothesis was tested — why was this hypothesis chosen? Often it is an attempt by those funding the study to prove a finding that will be to their advantage, financial or otherwise. Drug companies fund many of the studies showing that their drugs work. These studies are designed from the beginning to have the highest chance of producing results that the drug companies want, and it works. Such studies can be done ethically, they can be done with publication bias, or they can be done with outright fraud and manipulation of results.
What is discussed here only scratches the surface; such a discussion could easily form an entire book. Expect this section to be expanded over time.
Actionable options Most of our clients’ interests lie in the bottom line: how our findings can help them to be healthier. This is what will directly influence behavior and health outcomes. The section of any report where we give them this information is the most important.
For legal reasons, Personalized Medicine cannot and does not offer recommendations. This includes both “recommendations” and statements that are effectively recommendations.
What we do offer our clients are Actionable Options. We state our findings, in the most honest and useful way possible, and let doctors and patients draw their own conclusions. We do not and cannot substitute our findings for the physician’s medical judgment.
If the research says that some or all of the actionable options are to take the same actions everyone else should take for their health, then it is good and right to list those among your actionable options.
The recommendation section must provide the best actionable option(s) available, even if there is uncertainty surrounding them. The act of not deciding is itself a choice, and thus the patient must make a decision (to take the pill, to change their diet, to undergo the therapy, or not.) Not listing an actionable option takes that option away. Acting like the FDA, which refuses to permit anything until evidence for its safety and efficacy is “bulletproof” harms through inaction. Which actionable options are best includes factoring in compliance; what matters is what the patient does. Negative actionable options are as important as positive actionable options. Patients need to know what not to do! This includes alternate treatments that have been considered and rejected, especially those which constitute the standard of care. Anticipate questions about interventions you do not consider actionable and address why this is so.
Implementation of actionable options is often non-obvious. Be concrete about how to implement actionable options. Include contingencies and alternatives where appropriate. If the details matter, say how much they matter. If, for example, a Vitamin D pill is best be taken once a day with a meal that includes fat, note the impact if the pill was taken at other times or frequencies, so patients can either use a flexible regimen or be better motivated to adhere strictly. Even in this relatively simple case, not every day will have a meal that includes fat, and that likely isn’t something we want the patient to change!
Numbers Numbers allow patients to make good decisions. They are mandatory. Always give your best guess given what you know.
Although there is a great reluctance to state numbers — be they absolute risks, relative risks, effect sizes for various interventions, the necessary size of the interventions, exactly who must do what and the costs and benefits of these various actions — for fear of being wrong or, worse, sued, better decisions are impossible without numbers. Numbers are not optional.
The majority of papers submitted for review come back with requests for additional numbers and estimates.
A patient needs to know how much their risk can be increased or decreased, what the concrete costs are in terms of time, money and discomfort of doing so, and all the side effects both good and bad of such action. Only then can the patient choose a course of action. This can be even more important in the ‘normal’ case. One place we improve on modern medicine is to take its techniques and calculate to what extent, and how often, to apply them.
Confidence We must walk a fine line when it comes to confidence.
If the report is not confident, doctors and patients will be frustrated, and our reports will not be actionable. Expected effect sizes must always be listed, even when you are supremely unconfident and you have large error bars. Numbers are not optional.
We also cannot give people false confidence. The solution is to be confident in your actionable options’ expected effects, factoring in the possibility that they may not be effective. Thus, if you have found enough evidence that you would recommend taking cinnamon to prevent Alzheimer’s due to the low cost and high potential rewards — but still don’t think it is that likely to work — you might say: Cinnamon may reduce risk for Alzheimer’s. In expectation it reduces risk by X%.
You are confident in your estimate, even when not confident the option will be effective. Full confidence in the effectiveness of an intervention requires very strong evidence, far stronger than necessary to justify action.
Diet and exercise Diet is an especially complicated issue. High quality nutritional science is all but impossible, due to the difficulties of compliance and inaccurate reporting. It’s considered unethical to get people to eat unhealthy diets to gather data. The interactions involved are complex and subtle. Even more than other places, experts not only contradict each other, but advocate opposing strategies.
Diet is also likely the most important determinant of health.
It is vital to divorce your personal view of what constitutes a healthy diet from your research.
If you believe that everyone would be better off as a vegan, or on the Paleo diet, or as a raw foodist, or on caloric restriction, that’s fine. Only suggest it if it would be an especially effective solution to this particular problem, and be open to alternate dietary interventions. Failing all else, use a phrase like “eat a healthy diet” and let the doctor decide what that means.
Diet compliance in patients is as problematic as compliance in clinical trials. Doctors are constantly preaching to their patients to eat right and exercise, typically with little effect. Most are content to see even marginal improvements, such as patients reducing sugar intake or increasing physical activity; it is said for this reason that the best exercise is the one the patient will actually do, rather than the one that is theoretically the most efficient or effective.
Also remember that, no matter what you tell patients to do regarding diet and exercise, even those doing their best to carry out these recommendations will often get it wrong. An exercise that will injure the patient if improperly performed isn’t actionable, unless the patient can afford a trainer to watch them. A food regimen that requires treating things as if the patient has a deadly allergy require the level of motivation that comes with such an allergy, or expecting compliance is futile.
For this reason, dietary and exercise actionable options must always keep in mind, even more than others, what the patient is likely to end up doing. Unless the patient’s condition makes them unusually motivated, calls for large adjustments rarely result in change. Smaller, more targeted adjustments are more likely to be carried out and help the patient. Keep options specific, motivated and realistic.
Top general research principles — the writing/report Submit your report as a .doc or .docx file, NOT as a .pdf! Editing, formatting and our ability to copy text are important.
Include all of your sources.
Put MD-approved reports on the wiki, to the extent that they do not contain confidential information. As changes are made, edit on the wiki. [Note to contest entrants: Skip this.] Actionable options matter only insofar as they are accepted by our doctor, the patient’s doctor and ultimately the patient. State your findings honestly and ethically, but within those bounds so as to create positive health outcomes in the patient, and establish trust with the patient and doctor.
The paper should address any common interventions you rejected. Explain why you rejected them.
The primary audience for your report is the Doctor or other medical professional. The attending physician may also share the report with the patient. The actionable options must be understandable to the layman, but other sections can be written for a scientific audience.
Reports should be as concise as possible while addressing all required issues. Complex problems may require complex reports. Simpler problems can be addressed by short reports.
Note any prospects for additional research in the area.
The crazier what you are suggesting sounds, the stronger your evidence needs to be, and the more carefully you must make your case.
Genetics and other tests can be used as a fig leaf to recommend and motivate things patients should be doing anyway, and often should be used this way. Don’t hesitate to take advantage of this placebo technique, for example to tell people to “get some exercise” or (ironically) to “not eat as much sugar.”
Share your take on the evidence, its quality, what it means and what actions will have what results. Patients must make decisions, and score no points for saying “I don’t know!” Neither do you. Numbers are mandatory. Give your opinion, even if uncertain.
Top general research principles — the research If your assignment involves access to PHI, you must exercise strict compliance with the company’s HIPAA privacy and security requirements. Keep all medical information about patients strictly confidential at all times.
Articles, even when expensive, are cheap if they are important. When necessary, buy them! They can be expensed. Before buying, confirm they are what you need and that the company does not have access to them that you may not have.
If you find potentially useful information sources, tell us about them! Also include any tools you believe can help us. We are always looking to improve our process.
If you’re going to rely on the information from a paper, you need to read that paper to confirm the data and the procedures used to gather it.
Publication Bias is the biggest bias you will deal with. This is also a problem with any and all meta-studies, since they can only look at studies that were published. Funding bias is similar and related. Because of this problem, naïve combining of data from different studies is not reliable.
We are not frequentists. We’re not attached to whether the p-value is greater than or less than 0.05, or 0.01. However, do remember that the doctors you must convince operate to a large extent on this paradigm.
We are Bayesians. Everything is evidence. Form your opinion on the evidence, and reason out what it means; what you think is important. However, remember that you must convince doctors and patients in order to affect change and improve health.
Look for alternate explanations for (and flat out mistakes in) the data. Both are everywhere. Don’t assume professionals know what they are doing. Some of them do, but many do not. Even professionals miss things all the time.
Research should look to build our knowledge base and help us gain insights. Your focus is on helping the patient in question, but also on helping all patients over the long term, and the latter often has a far bigger impact. Watch for future worthwhile research, and for future experiments we can run. We seek truth rather than trying to qualify for publication. In such a context, a little empirical evidence can go a long way.
Remember that the purpose of research is to acquire information we can use to help patients.