Calculating user contributions
==============================

When gearing up for publication, you will find that due to the collaborative approach of CATMAID a great many people will have contributed to the data you would like to publish. In order to come up with a sensible author list, it is useful to quantify how much a user has contributed to the reconstruction of a set of neurons. CATMAID provides you a broad summary of contributions e.g. via the selection table. Pymaid let's you fetch that data and more.

First, let's get a set of olfactory projection neurons to demonstrate user statistics on:

.. code:: ipython3

    import pymaid
    
    # Initialize connection
    rm = pymaid.connect_catmaid()
    
    # Fetch some neurons
    nl = pymaid.get_neurons('annotation:glomerulus DA1 right')


.. parsed-literal::

    INFO  : Found 9 skeletons with matching annotation(s) (pymaid)


First some basic stats:

.. code:: ipython3

    cont = pymaid.get_contributor_statistics(nl, separate=True)
    cont.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>skeleton_id</th>
          <th>n_nodes</th>
          <th>node_contributors</th>
          <th>n_presynapses</th>
          <th>pre_contributors</th>
          <th>n_postsynapses</th>
          <th>post_contributors</th>
          <th>review_contributors</th>
          <th>multiuser_review_minutes</th>
          <th>construction_minutes</th>
          <th>min_review_minutes</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>61221</td>
          <td>7875</td>
          <td>{'adamjohn': 2442, 'ranftp': 4109, 'ratliffj':...</td>
          <td>404</td>
          <td>{'hsuj': 1, 'adamjohn': 10, 'ranftp': 249, 'ha...</td>
          <td>128</td>
          <td>{'ranftp': 89, 'alij': 1, 'koppenhaverb': 5, '...</td>
          <td>{'ratliffj': 2834, 'masoodpanahn': 84, 'koppen...</td>
          <td>165</td>
          <td>416</td>
          <td>135</td>
        </tr>
        <tr>
          <th>1</th>
          <td>27295</td>
          <td>9975</td>
          <td>{'koppenhaverb': 102, 'robertsr': 53, 'lovef':...</td>
          <td>412</td>
          <td>{'robertsr': 32, 'alij': 6, 'schlegelp': 205, ...</td>
          <td>59</td>
          <td>{'alij': 2, 'heatha': 2, 'hallouc': 2, 'schleg...</td>
          <td>{'schlegelp': 517, 'adesinaa': 2586, 'kmecoval...</td>
          <td>367</td>
          <td>502</td>
          <td>313</td>
        </tr>
        <tr>
          <th>2</th>
          <td>57323</td>
          <td>4585</td>
          <td>{'kmecoval': 2194, 'robertsr': 159, 'koppenhav...</td>
          <td>361</td>
          <td>{'calles': 2, 'michaelLingelbach': 1, 'kmecova...</td>
          <td>76</td>
          <td>{'vallas': 1, 'ranftp': 1, 'edmondsona': 20, '...</td>
          <td>{'ratliffj': 2116, 'kmecoval': 350, 'masoodpan...</td>
          <td>176</td>
          <td>208</td>
          <td>84</td>
        </tr>
        <tr>
          <th>3</th>
          <td>57311</td>
          <td>4882</td>
          <td>{'kmecoval': 1376, 'robertsr': 24, 'lovef': 75...</td>
          <td>371</td>
          <td>{'robertsr': 18, 'lovef': 220, 'jamasba': 4, '...</td>
          <td>58</td>
          <td>{'robertsr': 10, 'ranftp': 1, 'heatha': 1, 'lo...</td>
          <td>{'mooree': 2308, 'adamjohn': 1231, 'kmecoval':...</td>
          <td>390</td>
          <td>241</td>
          <td>163</td>
        </tr>
        <tr>
          <th>4</th>
          <td>57353</td>
          <td>4898</td>
          <td>{'kmecoval': 1054, 'hsuj': 1, 'koppenhaverb': ...</td>
          <td>302</td>
          <td>{'hsuj': 1, 'robertsr': 13, 'kmecoval': 17, 's...</td>
          <td>24</td>
          <td>{'robertsr': 7, 'kmecoval': 2, 'meechank': 13,...</td>
          <td>{'adamjohn': 1843, 'sharifin': 1231, 'masoodpa...</td>
          <td>177</td>
          <td>247</td>
          <td>101</td>
        </tr>
      </tbody>
    </table>
    </div>



For convencience, there is another function that shows contributions per-user instead of per-neuron:

.. code:: ipython3

    by_user = pymaid.get_user_contributions(nl)
    by_user.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>user</th>
          <th>nodes</th>
          <th>presynapses</th>
          <th>postsynapses</th>
          <th>nodes_reviewed</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>robertsr</td>
          <td>14527</td>
          <td>1038</td>
          <td>481</td>
          <td>4729</td>
        </tr>
        <tr>
          <th>1</th>
          <td>ranftp</td>
          <td>5927</td>
          <td>401</td>
          <td>132</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>kmecoval</td>
          <td>5204</td>
          <td>87</td>
          <td>4</td>
          <td>5177</td>
        </tr>
        <tr>
          <th>3</th>
          <td>hallouc</td>
          <td>4267</td>
          <td>61</td>
          <td>11</td>
          <td>8047</td>
        </tr>
        <tr>
          <th>4</th>
          <td>michaelLingelbach</td>
          <td>4253</td>
          <td>1</td>
          <td>0</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



Sometimes the plain number of nodes (or cable length) are not very helpful. Did a person trace mostly backbone (fast) or did they scramble tracing fine terminal branches (slow)? Personally, I consider time invested to be a better metric:

.. code:: ipython3

    time_inv = pymaid.get_time_invested(nl)
    time_inv.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>total</th>
          <th>creation</th>
          <th>edition</th>
          <th>review</th>
        </tr>
        <tr>
          <th>user</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>robertsr</th>
          <td>1113</td>
          <td>690</td>
          <td>420</td>
          <td>198</td>
        </tr>
        <tr>
          <th>hallouc</th>
          <td>627</td>
          <td>183</td>
          <td>180</td>
          <td>333</td>
        </tr>
        <tr>
          <th>koppenhaverb</th>
          <td>378</td>
          <td>108</td>
          <td>15</td>
          <td>144</td>
        </tr>
        <tr>
          <th>ranftp</th>
          <td>357</td>
          <td>219</td>
          <td>189</td>
          <td>0</td>
        </tr>
        <tr>
          <th>kmecoval</th>
          <td>333</td>
          <td>147</td>
          <td>72</td>
          <td>126</td>
        </tr>
      </tbody>
    </table>
    </div>



Notice how the order of contributors changes whether we look at e.g. nodes created vs total time invested?

:func:`pymaid.get_time_invested` takes timestamps from node creation, edits and reviews and calculates the time invested in minutes. In brief, pymaid sums up the minutes in which a user has performed 10+ actions. You can tweak this behaviour - have a look at the documentation!

Compartmentalizing contributions 
--------------------------------

You might end up publishing only reconstructions in a certain neuropil. In that case will want to subset contribution accordingly.

Let's first prune the projection neurons to the lateral horn

.. code:: ipython3

    import matplotlib.pyplot as plt
    
    lh = pymaid.get_volume('LH_R')
    lh.color = (240,240,240,.5)
    
    nl_lh = nl.prune_by_volume(lh, inplace=False)
    
    fig, ax = pymaid.plot2d([nl_lh, lh], connectors=False, linewidth=1.5)
    plt.show()



.. image:: user_contributions_files/user_contributions_9_0.png


You must **not** use :func:`pymaid.get_user_contributions` or :func:`pymaid.get_contributor_statistics` with neuron fragments as they are not "`fragment-safe`". That's because these functions use only skeleton IDs to query the CATMAID server which is totally oblivious to the local changes to these neurons.

For our purposes, :func:`pymaid.get_time_invested` and :func:`pymaid.get_team_contributions` are `fragment-safe`:

.. code:: ipython3

    time_lh = pymaid.get_time_invested(nl_lh)
    time_lh.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>total</th>
          <th>creation</th>
          <th>edition</th>
          <th>review</th>
        </tr>
        <tr>
          <th>user</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>robertsr</th>
          <td>588</td>
          <td>249</td>
          <td>210</td>
          <td>177</td>
        </tr>
        <tr>
          <th>koppenhaverb</th>
          <td>225</td>
          <td>96</td>
          <td>9</td>
          <td>33</td>
        </tr>
        <tr>
          <th>lovef</th>
          <td>198</td>
          <td>0</td>
          <td>15</td>
          <td>162</td>
        </tr>
        <tr>
          <th>schlegelp</th>
          <td>171</td>
          <td>27</td>
          <td>30</td>
          <td>66</td>
        </tr>
        <tr>
          <th>ranftp</th>
          <td>105</td>
          <td>69</td>
          <td>39</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



There is currently no pymaid wrapper for the number of nodes/connectors/etc per user but that's no problem:

.. code:: ipython3

    node_details = pymaid.get_node_details(nl_lh.nodes.treenode_id.values)
    connector_details = pymaid.get_node_details(nl_lh.connectors.connector_id.values)




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>node_id</th>
          <th>creation_time</th>
          <th>creator</th>
          <th>edition_time</th>
          <th>editor</th>
          <th>reviewers</th>
          <th>review_times</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>331812</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[53, 13]</td>
          <td>[2016-05-03 14:43:00, 2015-05-12 12:32:00]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>331813</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[13, 53]</td>
          <td>[2015-05-12 12:32:00, 2016-05-03 14:43:00]</td>
        </tr>
        <tr>
          <th>2</th>
          <td>331814</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[13, 53]</td>
          <td>[2015-05-12 12:32:00, 2016-05-03 14:43:00]</td>
        </tr>
        <tr>
          <th>3</th>
          <td>331815</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[13, 53]</td>
          <td>[2015-05-12 12:32:00, 2016-05-03 14:43:00]</td>
        </tr>
        <tr>
          <th>4</th>
          <td>331816</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[53, 13]</td>
          <td>[2016-05-03 14:43:00, 2015-05-12 12:32:00]</td>
        </tr>
      </tbody>
    </table>
    </div>



Next, map user IDs to names:

.. code:: ipython3

    user_list = pymaid.get_user_list().set_index('id').login.to_dict()
    
    node_details['creator2'] = node_details.creator.map(user_list)
    connector_details['creator2'] = connector_details.creator.map(user_list)
    
    node_details.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>node_id</th>
          <th>creation_time</th>
          <th>creator</th>
          <th>edition_time</th>
          <th>editor</th>
          <th>reviewers</th>
          <th>review_times</th>
          <th>creator2</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>331812</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[53, 13]</td>
          <td>[2016-05-03 14:43:00, 2015-05-12 12:32:00]</td>
          <td>adesinaa</td>
        </tr>
        <tr>
          <th>1</th>
          <td>331813</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[13, 53]</td>
          <td>[2015-05-12 12:32:00, 2016-05-03 14:43:00]</td>
          <td>adesinaa</td>
        </tr>
        <tr>
          <th>2</th>
          <td>331814</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[13, 53]</td>
          <td>[2015-05-12 12:32:00, 2016-05-03 14:43:00]</td>
          <td>adesinaa</td>
        </tr>
        <tr>
          <th>3</th>
          <td>331815</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[13, 53]</td>
          <td>[2015-05-12 12:32:00, 2016-05-03 14:43:00]</td>
          <td>adesinaa</td>
        </tr>
        <tr>
          <th>4</th>
          <td>331816</td>
          <td>2015-05-12 12:31:00</td>
          <td>13</td>
          <td>2015-10-20 06:44:00</td>
          <td>13</td>
          <td>[53, 13]</td>
          <td>[2016-05-03 14:43:00, 2015-05-12 12:32:00]</td>
          <td>adesinaa</td>
        </tr>
      </tbody>
    </table>
    </div>



Group by user and count:

.. code:: ipython3

    import pandas as pd
    
    node_counts = node_details.groupby('creator2').node_id.count()
    cn_counts = connector_details.groupby('creator2').node_id.count()
    
    lh_counts = pd.concat([node_counts, cn_counts], axis=1, sort=True).fillna(0).astype(int)
    
    lh_counts.columns=['nodes', 'connectors']
    
    lh_counts.sort_values('nodes', ascending=False).head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>nodes</th>
          <th>connectors</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>robertsr</th>
          <td>5527</td>
          <td>1091</td>
        </tr>
        <tr>
          <th>koppenhaverb</th>
          <td>2847</td>
          <td>0</td>
        </tr>
        <tr>
          <th>ranftp</th>
          <td>2496</td>
          <td>349</td>
        </tr>
        <tr>
          <th>jefferis</th>
          <td>947</td>
          <td>11</td>
        </tr>
        <tr>
          <th>schlegelp</th>
          <td>914</td>
          <td>333</td>
        </tr>
      </tbody>
    </table>
    </div>



Contributions by date
---------------------

You might find yourself in a situation where your neurons of interested have already been published in the past. In that case, you could consider crediting only reconstructions that have been done since.

For this, we need to subset neurons to nodes/connectors that have been created/reviewed/edited after a certain date:

.. code:: ipython3

    import numpy as np
    
    after_date = np.datetime64('2017-01-01')
    
    # Get node details
    node_details = pymaid.get_node_details(nl.nodes.treenode_id.values)
    cn_details = pymaid.get_node_details(nl.connectors.connector_id.values)
    
    # Subset to nodes/connectors created after given date
    new_nodes = node_details[node_details.creation_time >= after_date].node_id.values
    new_connectors = cn_details[cn_details.creation_time >= after_date].node_id.values
    
    # Subset neurons to the new nodes/connectors
    nl_new = nl.copy()
    for n in nl_new:
        pymaid.subset_neuron(n, new_nodes, inplace=True, remove_disconnected=False)
        n.connectors = n.connectors[n.connectors.connector_id.isin(new_connectors)]

Let's visualize this: old neuron in grey, recent additions in red.

.. code:: ipython3

    fig, ax = pymaid.plot2d(nl, color=(.8,.8,.8), connectors=False)
    _ = pymaid.plot2d(nl_new, color='r', ax=ax, connectors=False, linewidth=1)
    
    plt.show()



.. image:: user_contributions_files/user_contributions_21_0.png


Now that we have subset our neurons of interest, we can use :func:`pymaid.get_time_invested` and :func:`pymaid.get_team_contributions` just like we did before:

.. code:: ipython3

    time_new = pymaid.get_time_invested(nl_new)
    time_new.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>total</th>
          <th>creation</th>
          <th>edition</th>
          <th>review</th>
        </tr>
        <tr>
          <th>user</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>robertsr</th>
          <td>168</td>
          <td>81</td>
          <td>51</td>
          <td>9</td>
        </tr>
        <tr>
          <th>schlegelp</th>
          <td>30</td>
          <td>9</td>
          <td>9</td>
          <td>9</td>
        </tr>
        <tr>
          <th>alij</th>
          <td>6</td>
          <td>0</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>sharifin</th>
          <td>6</td>
          <td>0</td>
          <td>3</td>
          <td>0</td>
        </tr>
        <tr>
          <th>edmondsona</th>
          <td>6</td>
          <td>3</td>
          <td>3</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



Relevant functions:
    
.. autosummary::

    pymaid.get_contributor_statistics
    pymaid.get_user_contributions
    pymaid.get_time_invested
    pymaid.get_team_contributions