File upload & “data types”

Last week I finalized and publicly shared the File Uploader activity on Open Humans.1

This is a standard “activity” in Open Humans. Our structure is modular: in theory, anyone can use the APIs to create an activity that does the same thing. (In this case, the activity is created and administered by us. Yes, it’s open source.)

As such, using the File Uploader requires going to a different site (which is linked on the activity page):

What’s the “File Uploader” for?

This is intended as a replacement for “Data Selfies” to be generic approach for file upload to Open Humans.

There are a variety of personal data files that people might want to upload in order to (1) share with research and citizen science projects, and/or to (2) run personal data analyses on using our notebooks tool.

Indeed, the number of potential data sources is infinite, and tracking which type of data is in a file is important (and tricky): this is critical authorizing and sharing data with other activities, and for identifying data when running personal data analysis.

About “Data Types”

“File Uploader” adds something new that “Data Selfies” lacked: data types.

Historically, Open Humans primarily understood “what” a file is according to “data source” — that is, the activity that added the data. (e.g. “23andMe Uploader” or “AncestryDNA Uploader”). This had two issues: (1) a single activity might generate more than one “type” of data, (2) different activities might have data of the same “type” and people might want to manage that in a unified way (e.g. “any personal genetic data”).

On top of that: it’s weird overhead to create a new website for each new type of file upload!

So “DataTypes” was created! Don’t see a DataType you need? You can create one yourself…

How to create a new DataType

  1. Go to the DataTypes page:
  2. Click “Add datatype”
  3. Add information (e.g. “parent datatype” if any, how it’s acquired or created, format info)
  4. Mark “uploadable” if this is something the File Uploader should support

Data Types created in this way (i.e. “uploadable”) are immediately available as an option on the File Uploader.

What’s left?

So far, the utility of “data type” isn’t fully implemented. Next things to do are probably…

  1. Search Personal Data Notebooks according to “data type”. Right now it’s only possible to search according to “data source”. (Hopefully this is easy to do!)
  2. Enable requesting “data type” for activities. Currently activities can only request authorization for specific “data sources”. This is probably “hard” to add (the logic and interface gets complicated), but it should be possible for a project to request a type of data (e.g. “sleep data”) in a generic way.
  3. Retire old file uploader projects. As mentioned above, it’s silly to have separate projects for each one. (But doing this smoothly might require completing step #2 above.)

1 This was initially created a long time ago! I was very absent during the pandemic, and I’m trying to get back into things…. dusting off old stuff we meant to finish. πŸ˜…

Doing Personal Science together – in memory of Steven Keating

Whether you call it Personal Science, Self-Research or Quantified Self, many of us have questions about ourselves and try to find answers for them. It’s a process that can range from structured journaling to rigorous N-of-1 studies. It’s about you – and it can be about nearly anything. From introspection to data analysis, self-research can help people find what habits work for them, manage chronic conditions, and learn more about themselves.

In collaboration between Quantified Self and Open Humans, we want to support self-research as a collective endeavor and provide a framework where a community of self-researchers can help each other. As one of the main support frameworks for this we have been hosting weekly self-research chats. In these chats you can share your currently on-going projects or projects you are currently planning; ask for feedback or advice; get help when you’re stuck; or brainstorm new ideas.

We started these calls last year as part of our collective memorial to Steven Keating. On July 19 2019 Steven – an inspiring advocate for patient data access – passed away. Curiosity was a driving force in Steven’s life. He recorded and shared videos of his brain surgery, explored his cancer’s genetic data, and printed 3D models of his tumor. Steven was also a supporter of Open Humans, and served as a director up until his passing. In his memory and in celebration of his life, we would like to help more people be curious about themselves. Our goal is to share reports about what we learned this July, in honor of him.

Our first memorial to Steven last year culminated in a Show & Tell meeting in which participants of the regular self-research chats presented their projects and what they learned from them. Given the large interest in this we will do the same this year: At the end of July we will organize another Show & Tell meeting in memory of Steven and just like last year we will hold weekly self-research meetings to prepare our projects. 

We’re inviting you to join our calls – every Thursday at 10am Pacific time – to share your ideas about self-research projects – questions they have, and potential approaches – and provide support for each other in our efforts. Doing this in a group means you can get help when you’re stuck! This is an opportunity to try self-research for the first time – or to pursue a project you already have – by doing it with a small group of people who have diverse skills, lots of experience, and a desire to support you.

Join us in learning about ourselves. Find more information about the Memorial, the self-research chats and when the next meeting is.

Learn about Quantified Flu and our weekly community calls

Join Quantified Flu

Do changes in physiological signals that are measured through wearables (such as body temperature and resting heart rate) precede consciously experienced symptoms (such as coughs, sore throats, shortness of breath, etc.)? This question is on many people’s minds right now, and it was a hot topic during an Open Humans community call in early March. Collectively we wondered whether it is possible to explore and potentially improve, through collective self research & an open process, what individual value data has with wearable data and symptom tracking.

As a result of this we have launched Quantified Flu, a collective project that tries to answer this question through both retrospective data analyses and the ongoing reporting of symptoms. It allows you to annotate your existing wearable data by adding past dates on which you fell sick. For the ongoing symptom collection, you can set up a daily check-in time on which you will get an email asking you to either quickly report “no symptoms” or enter symptoms if you experience any. 

You can decide how much of this data you want to share: You opt-in to publicly share the data through a random identifier, allowing others to analyze that data and create new and improved data visualizations. In the long-run we might share aggregated and de-identified data as well, as long as we are confident that this respects individual privacy. If you are interested in joining this effort as a participant, head to Quantified Flu and sign up!
All of this was made possible in such a short amount of time thanks to our great community, including Gary Wolf, Ernesto Ramirez, Katarzyna Wac, Chris Ball, Beau Gunderson, Lukasz Baldy, Konstantin Vdovkin and many more. If you want to contribute data visualizations, adding support for more wearables or help with web development, join our Slack and visit the #quantifiedflu channel!

Come to our Community Calls

As mentioned before: The idea to run Quantified Flu came directly out of our community calls, and given the huge interest in them we have ramped up the frequency and are now doing them on a weekly basis: They take place every Tuesday at 10am Pacific / 1pm Eastern / 6 pm GMT / 7pm CEST and you are invited to join us. We always welcome seeing new faces! 

The calls are done via Zoom and we have a growing Google Document that contains all the details on how to join the calls, an agenda for the upcoming call as well as notes of the past ones. If you want to get reminders for the meetings in your own calendar, we do have a community calendar that you can subscribe to. 

If you want to catch up on the prior community calls, we are recording them and make them available on the Open Humans YouTube channel.

“Self research” panel video, join next on Mar 10!

We wanted to share our recording of an excellent a panel we had on Monday, and invite you to our next one on March 10.

As part of our Keating Memorial Self Research activity, we discussed how to get started with your own self-research, what challenges one might experience, and advice to solve any issues encountered so far. In addition to having some veteran self-researchers, we were happy to have some first-timers join us as well!

Liz Salmi, Katarzyna Wac, Rogier Koning, Steven Kaye, Gary Wolf and Steven Jonas joined Bastian Greshake Tzovaras and Mad Ball to share their experiences on a variety of topics, including how to collect relevant data, how to keep motivated when doing time-intensive active tracking, and how just the act of collecting data can already modify our behavior.

You can watch the full recording at

We’re going to have the same format for our community call on March 10. Our topic is “From individual to collective self-research”: How do people translate individual ideas and efforts to work that involves others? This can include shared & re-used methods, tools, and analyses, to aggregating data for collective insight. We’ll invite attendees to participate as a panel of ourselves, and we’ll be recording the conversation to share everyone’s questions and insights with others.

You’re invited to take part! Meeting info:

If you’re inspired to do your own self-research, you can still join the Keating Memorial and join our group of self-researchers:

Notes & video from our self-research kickoff

Kickoff Webinar Attendees ScreenshotIn case you missed it: we took notes & recorded our kickoff webinar for the Keating Memorial Self Research activity last week!

On Thursday we’ll have “open office hours” to offer free expert support for people getting started with questions and ideas about what they want to do. The meeting is at 10am PST / 6pm GMT on February 13 (Thursday).

Please feel welcome to join the videochat on Thursday! See this link for how to join (it’s also where we’ll take notes):

You’re welcome to join the Keating Memorial Self Research activity at any time in coming months. (Although we think it’s ideal to start now!) Our support is staged (developing ideas, prototyping tools, data analysis) but we’ll be available throughout. Our goal is to complete projects and share what we learned no later than mid-July.

Notes from the meeting are available online, and the video is available below.

Keating Memorial Self Research

We want to share a new project we’re running in coming months: Keating Memorial Self Research – an invitation to collaborate in self-research!

(Click here for extended details, including timeline. If you’re unfamiliar with Open Humans, click here to read more about us!)

We’re inviting people to share their ideas about self-research projects – questions they have, and potential approaches – and provide support for each other in our efforts.

Doing this in a group means you can get help when you’re stuck! This is an opportunity to try self-research for the first time – or to pursue a project you already have – by doing it with a small group of people who have diverse skills, lots of experience, and a desire to support you.

To join:

  • Log in to our Self-Research Forum
  • Post something:
    … a self-research project you might want to do, a project you’re currently working on, something you already did, or just introduce yourself – say hi!

Anybody can do self-research! It can range from structured journaling to rigorous N-of-1 studies. It’s about you – and it can be about nearly anything. From introspection to data analysis, self-research can help people find what habits work for them, manage chronic conditions, and learn more about themselves.

Our goal is to start in February and to have completed our projects by July. What we learn will serve as a celebration in memory of Steven Keating, an inspiring advocate for patient data access. You can read extended details about this effort on our site (including more about Steven, our timeline, and calendar of events).

Collaborating on individual discovery

About the authors: Mad Price Ball and Bastian Greshake Tzovaras lead Open Humans as Executive Director and Director of Research, a nonprofit project dedicated to empowering individuals and communities around their personal data, to explore and share for the purposes of education, health, and research. We have other diverse work in open science, data technology, digital and research ethics, health and humans subjects research, and citizen science. We’ve been thinking about this for a while now: What can be built to make collaboration a reality in self-research?

Gary Wolf’s call for enabling individual discovery describes a vision we also share: we believe in the untapped potential of people studying themselves. Answering questions relevant to their own lives, with motivations ranging from chronic disease to curiosity. We encourage you to read what he’s written: “How to Make 10 Million Discoveries”

We can all ask questions about our lives: sometimes unique, sometimes common. “Does this medication help me? Do I say “sorry” too often? Is social media making me unhappy?”

Each of those questions is a potential discovery – about you.

Gary wrote: “My colleagues and I envision a world with as many new, consequential health discoveries as there are articles in Wikipedia.” That is, ten million discoveries.

We think this can be done, and we’ll tell you how.

(And if you’re interested in helping – skip to the bottom to read how!)

Doing things together

β€œIf you want to go fast, go alone. If you want to go far, go together.”

Until recently, people would have told you it’s impossible for hundreds of thousands of volunteers write the world’s largest encyclopedia in nearly 300 languages. Wikipedia now exists. It is an inspiring example of the potential of “peer production”: the idea that people can co-create knowledge in a collective way, with an open collaborative approach. It is made possible by the Internet – by online methods for knowledge-sharing, re-use, and decentralized coordination.

Could this work with self-research? Gary observed the potential:

“Technology that makes it easy for people to make, discover, and share things has fundamentally reshaped older systems of mass production in many domains. Why not in health? The diversity of needs and questions people have, and the diversity of skills and experiences required to address them, makes this a natural job for peer production. Healthcare may be slow in helping. But we can help each other.”

Few of us can become an expert in everything we need when studying ourselves. This is something peer production can solve. The outcome is incredibly powerful: people can contribute in the place they are expert, and benefit from the expertise of others.

Imagine a world where someone produces an analysis of their data – and then shares it in a way others can easily use to analyze their own data.

Imagine a world where someone produces a tool to collect data – and then shares it in a way others can easily use to collect their own data.

Even more – imagine the power if these are done in “open” ways – where analysis code can be remixed to answer a new personal question, where an app or hardware tool can be repurposed to collect the data you need.

Imagine a place where people shared their remixes. Their methods for putting the pieces together. Their ideas.

This isn’t pure fantasy: we’re already part way there! The card10 badge that Gary discusses – a device self-trackers might use – is an open hardware project that has produced scores of open source apps. In Open Humans, we have scores of open source notebooks you can use to analyze diverse personal data. In the Quantified Self forums, there is a growing set of “project logs” where people share self-research as they do it.

What’s missing is a synergistic connection for the parts: a thriving community of co-creators.

What’s left to build

The answer is, to us, deceptively simple: we need better communication.

We need to connect people more: to share personal projects and tools and ideas, to engage and help each other, to form groups that share our interests. There are already many solutions for how to go about self-research. What we really need is a place where we can learn about, use, and adapt the things that already exist and have already been done. We need something that promotes peer support and collaboration through communication, sharing, contribution, remix, and re-use.

We should make those things easy and rewarding. Doing research about ourselves needn’t be a self-centered task: when we share our solutions and experiences and discoveries, that can have value to others. We can become part of something larger than ourselves.

To be clear: there is more needed than “building a platform”. It needs communities using it. Self-research needs to be do-able. Gary’s vision of ten million discoveries requires aligned work in the tools, the methods, and in communities that want it – things other stakeholders in the Article 27 effort represent.

We’re not alone in seeing the missed opportunity. A recent paper about the Quantified Self community (Heyen 2019) reflected:

“it is noticeable that, so far, no knowledge accumulation can be observed in the QS movement, nor has a common stock of knowledge developed (yet). This is certainly also due to its rather loose form of networking. […] …each self-tracker starts more or less from scratch, regardless of whether another self-tracker has already worked on exactly the same question and gained insights that could be built upon. Correspondingly, public presentations of self-tracking research at meetups or conferences make hardly any references to other self-trackers and their activities. In this respect as well, personal science is a very self-related affair.” (emphasis added)

What’s missing isn’t merely new solutions – but, more importantly, connecting people who create and use them. A place where new solutions are created and shared and improved upon.

This is entirely doable. One of the most powerful aspects of the Internet is that it connects us.

What you can do

We want to build this, and we need to prototype. You can help.

Can you do a self-research project together with us?

We’re calling for people to work together in the coming months to do a set of self-research projects together. If you’re interested in being part of the group, get involved!

You can…

We are hoping this effort will be, in part, a memorial. Last year we lost Steven Keating, a member of our board, to brain cancer. Curiosity was a driving force in Steven’s life, and he was an inspiring advocate for patient data access. He recorded and shared videos of his brain surgery, explored his cancer’s genetic data, and printed 3D models of his tumor.

In his memory, we would like to help more people make discoveries about themselves. What we do in coming months has the potential to seed an ongoing community.

Notes from our first Community Call

We held our first community call on December 10th – many thanks to the attendees and invited presenters, Karolina Alexiou and Rogier Koning! Future community calls will be held on the second Tuesday of each month.

Interested in attending on January 14? You can add the Google calendar event and visit our ongoing agenda & notes document to get information about joining the call.

Invited guest: Rogier Koning, Nobism, and Cluster Headaches

Rogier shared the Nobism app and data source in Open Humans, which he created to track symptoms, potential triggers, and treatments. Cluster headaches are one of the most painful things people can experience – they’re called “suicide headaches” – and patients understandably want to understand how to anticipate and reduce their own symptoms!

Rogier’s reports showed compelling visualizations produced with the Ubiqum project, which members can share their data with – illustrating how headaches occur over the course of months, at different times of the day – different individuals had different patterns for what time headaches were likely to occur. The effect of medications could be clearly seen in the patterns on the graphs!

One of our long-time community members, Ben Carr, noted he’s also a cluster headache patient, and reflected on his use of the app! He reflected on potential improvements and hadn’t realized he could also join the accompanying Ubiqum project. (Showing us there’s a potential need for prompting people!) Rogier also explained that the app isn’t limited to cluster headaches, and he’d welcome other chronic disease patients using it for new purposes.

Rogier has plans to expand community aspects of his work, and hopes to share more in the future!

Invited guest: Karolina Alexiou and GitHub data import

Karolina presented one of our latest data source additions, which might especially be of interest for programmers. Her GitHub data import gets all data on contributions made to (open source) code projects on GitHub, giving members a view of to which projects they contribute and when & how much they program.

github commit word cloud
An analysis example from the GitHub data notebook: a word cloud generated fromΒ  commit messages

In addition to the data import itself, Karolina demonstrated how this data can be visualized and what can be learned from it, by running a Personal Data Notebook on her own data. This notebook is already publicly available, so if you are using GitHub and want to give it a try, you can start right away.

Data Types & Uploading files

Mad & Bastian shared some ongoing work on the Open Humans end. While data sets are currently organized by the data source that has uploaded them, this sometimes makes sharing the data complex. Either because multiple projects upload the same or very similar data types (e.g. genetic data from different sources), or because a single data source uploads multiple kinds of data (e.g. activity tracking data that contains both step counts and GPS records). As Ben Carr noticed on the call, this can make granular sharing hard.

Noise mapping plots
Bastian’s noise mapping, split by movement type (cycling, stationary, walking).

To adjust for this, Mad has been working on a data type system, which allows individual data files be classified according to the kind of data in them, instead of just relying on the source. They presented a new uploader tool for Open Humans, that can assign the data type to each file upload. Based on this, Bastian presented how this data can be used to upload environmental noise data, as recorded from an Apple Watch and how it can be explored through a Personal Data Notebook. One thing Bastian learned was that some of his noisiest times were at home; when he looked into it more closely, it was when he was in the shower!

Further discussions

Additionally, the participants of the community call discussed their experiences with tracking blood sugar through Continuous Glucose Monitors, how to make the Personal Data Notebooks more user friendly and whether it is possible to allow access to data without sharing the data sets themselves by allowing analyses being run in the cloud. Exchanging community member experiences and what they are working on was inspiring.

If you want to participate in the next community call please see our Community Call information & notes document for event details, it will be 14th of January at 10am PST / 6pm GMT.

Meet the newest projects on Open Humans

In the last few weeks our community has launched a plethora of new projects that you can join to collect more data about yourself as well as new research opportunities covering topics from the genetics of personality over blood pressure tracking to cluster headaches. Find out more about those projects below:

QCycle: Tracking ovulatory cycles

QCycle is a participatory research study that follows the spirit of the Quantified Self. As a collaboration between Azure Grant at the University of California, Berkeley and all participants, the study is interested in mapping the diversity of biological rhythms such as the ovulatory cycle through different wearable devices such as the Oura Ring. In the long-term one of the aims could be creating open-source ovulation predictions.

As the study is a collaborative project, you can also bring along your own research questions as a participant. Visit the study website to learn more on the aims and how it works.

Genetics of Personality Type

If you have your own genetic data from 23andMe, AncestryDNA or FamiliyTreeDNA you might be interested in the Genetics of Personality Type study of Dr. Denise Cook of the Ronin Institute. In her study, Denise wants to find out whether it is possible to find genetic variants that are correlated with the personality type as defined by different questionnaires that make use of the Myers-Briggs Type Indicator.

After joining the study you will be asked to fill out 3 surveys, the results of which will be deposited in your Open Humans account as well. You can learn more about the study and join it on the study website.

Nobism: Tracking cluster headaches

The cluster headache patient community around Nobism has been particularly active in the last few weeks. Under the lead of Rogier Koning , the community was awarded one of the Open Humans project grants. The grant allowed them to integrate a data synchronization into their mobile application for tracking symptoms and interventions. Check out their app for iOS and Android.

Thanks to a collaboration with the Ubiqum Code Academy you can already use the data collected by the mobile apps to get personalized reports and data visualizations. In the Nobism Ubiqum Cluster Headache Project a team of data science students will create evolving, monthly reports based on the data you collect.

Last but not least you can also decide to share those individualized reports with the larger Nobism community through the nobism reports for Advocating project. By sharing those reports you allow the larger community to use those reports in outreach materials like presentations.

Remembering Steven Keating

This last weekend I was saddened to hear about the passing of one of our Board of Directors – Steven Keating, an inspiring activist and advocate for access to health data.

Steven Keating passed away on Friday, at the age of 31, ending his battle with brain cancer. When he was first diagnosed, Steven was a graduate student at MIT. His natural curiosity led him to collect and share diverse data about his cancer and treatment.

He shared all sorts of data, video of his brain surgery, and – most memorably – 3D printed copies of his tumor. You can read more about his life in this remembrance on MIT News: “Celebrating a curious mind”. Steven advocated for the importance of access to our health data: to explore, to use, and to share.

As a person, Steven was positive. Amazingly positive. It’s a lesson that helped me on a personal scale: sometimes bad things happen, but I learned that it’s still possible to face them with positivity. Steven taught by example.

When Steven’s “silly tumor” came back a year ago, he told me he wanted to keep serving as normal, as long as he felt able. And he did. He shared his experimental treatments with us during meetings. He had marked “yes” to a board meeting this Monday. He was with us as long as he could be.

He will be missed. My life is a better one for having known him.