Fetching data from a CATMAID server
***********************************
With minor exceptions, ``pymaid`` covers every API endpoint of your CATMAID server
- i.e. you should be able to get all the data that your web browser shows (and more)

``pymaid`` has >50 different functions that fetch various kinds of data from
CATMAID. Too many to cover them all - instead this tutorial will try to give
you a flavour by showcasing a few examples.

If you know what data you are after, head over to the :ref:`API reference <api>`
and search for keywords to find the appropriate function.

A quick primer
==============
All functions that require neurons/skeleton IDs as inputs (e.g. :func:`~pymaid.get_neuron`) accept either:

1. Skeleton ID(s) (int or str)::

    n = pymaid.get_neuron(16)
    nl = pymaid.get_neuron([16, 57241])

2. Neuron name(s) (str)::

    n = pymaid.get_neuron('PN glomerulus DC3 57242 ML')
    nl = pymaid.get_neuron(['PN glomerulus DC3 57242 ML', 'PN glomerulus VA6 017 DB'])

3. Annotation(s) (str)::

    n = pymaid.get_neuron('annotation:glomerulus VA6')
    nl = pymaid.get_neuron(['annotation:glomerulus DC3', 'annotation:glomerulus VA6'])

4. ``CatmaidNeuron`` or ``CatmaidNeuronList`` object::

    n = pymaid.get_neuron(16)
    cn_table = pymaid.get_partners(n)

By default, annotations and neuron names must match exactly. You can use regex on names and annotations, using a leading ``/``::

    # Get all neurons that have 'VA6' in their name
    nl = pymaid.get_neuron('/VA6')

    # Get all neurons annotated with a 'VA6'-containing annotation
    nl = pymaid.get_neuron('annotation:/VA6')

    # Get all skeleton IDs that have an annotation starting with 'AV1.R':
    skids = pymaid.get_skids_by_annotation('/^AV1.R')


See `here <https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285>`_ for detailed explanation of regex patterns.


Fetching neurons
================
One of the most basic thing is fetching neurons. First, we need to set up the connection to the CATMAID server. This should already look familiar - if not check out the `quickstart tutorial <tutorial>` first:

.. code:: ipython3

    import pymaid
    rm = pymaid.connect_catmaid()


.. parsed-literal::

    INFO  : Global CATMAID instance set. Caching is ON. (pymaid)


Now that's done, we can get going. In 99% of the cases, you will use :func:`pymaid.get_neurons` to fetch skeletons for analysis and visualization.

.. code:: ipython3

    da1 = pymaid.get_neuron('annotation:glomerulus DA1 right')
    da1


.. parsed-literal::

                                                                







.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>type</th>
          <th>name</th>
          <th>skeleton_id</th>
          <th>n_nodes</th>
          <th>n_connectors</th>
          <th>n_branches</th>
          <th>n_leafs</th>
          <th>cable_length</th>
          <th>soma</th>
          <th>units</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 57316 2863105 ML</td>
          <td>2863104</td>
          <td>6774</td>
          <td>470</td>
          <td>280</td>
          <td>None</td>
          <td>1522064.50</td>
          <td>[3245741]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>1</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 57382 ML</td>
          <td>57381</td>
          <td>8243</td>
          <td>382</td>
          <td>195</td>
          <td>None</td>
          <td>1337348.75</td>
          <td>[3364368]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>2</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 61222 AJ</td>
          <td>61221</td>
          <td>8578</td>
          <td>556</td>
          <td>199</td>
          <td>None</td>
          <td>1353478.75</td>
          <td>[3254492]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>3</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 57354 GA</td>
          <td>57353</td>
          <td>6985</td>
          <td>370</td>
          <td>225</td>
          <td>None</td>
          <td>1539364.25</td>
          <td>[3254736]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>4</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 57324 LK JSL</td>
          <td>57323</td>
          <td>6223</td>
          <td>481</td>
          <td>256</td>
          <td>None</td>
          <td>1483462.50</td>
          <td>[3251355]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>5</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 57358 IJA</td>
          <td>755022</td>
          <td>7453</td>
          <td>515</td>
          <td>230</td>
          <td>None</td>
          <td>1401414.75</td>
          <td>[3252148]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>6</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 27296 BH</td>
          <td>27295</td>
          <td>10216</td>
          <td>507</td>
          <td>304</td>
          <td>None</td>
          <td>1806000.25</td>
          <td>[3005291]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>7</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mlALT DA1 vPN mlALTed Milk 18114...</td>
          <td>1811442</td>
          <td>11769</td>
          <td>804</td>
          <td>378</td>
          <td>None</td>
          <td>2169807.25</td>
          <td>[6838672]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>8</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT DA1 lPN 57312 LK</td>
          <td>57311</td>
          <td>5543</td>
          <td>463</td>
          <td>211</td>
          <td>None</td>
          <td>1364103.25</td>
          <td>[3059181]</td>
          <td>1 nanometer</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: ipython3

    fig, ax = da1.plot2d(lw=1.5)







.. image:: fetching_data_files/fetching_data_4_1.png


Fetching volumes
================
Lets also fetch a volume to put those neurons into context:

.. code:: ipython3

    vol = pymaid.get_volume('v14.neuropil')
    vol




.. parsed-literal::

    <navis.Volume(name=v14.neuropil, color=(120, 120, 120, 0.6), vertices.shape=(8997, 3), faces.shape=(18000, 3))>



.. code:: ipython3

    # For visualization we rely on navis
    import navis
    
    # Make the volume more transparent
    vol.color = (230, 230, 230, .2)
    
    # Plot
    fig, ax = navis.plot2d([da1, vol], lw=2)
    
    # Rotate to frontal view
    ax.elev = ax.azim = -90
    ax.dist = 5







.. image:: fetching_data_files/fetching_data_7_1.png


Obscure functions
=================
Fetching skeletons and volumes is pretty standard, and fetching connectivity is covered in a :ref:`separate tutorial <cn_tut>`. Let's have a look at some more obscure functions.

Find out who contributed to the DA1 neurons we loaded above:

.. code:: ipython3

    cont = pymaid.get_contributor_statistics(da1)
    
    # `cont` is a pandas.Series containing e.g. nodes contributed
    # Let's plot only people with >1000 nodes contributed
    {k: v for k, v in cont.node_contributors.items() if v > 1000}


.. parsed-literal::

                                                       



.. parsed-literal::

    {'harrisonb': 2587,
     'jefferis': 2031,
     'schlegelp': 9977,
     'koppenhaverb': 3355,
     'ratliffj': 2177,
     'hallouc': 3328,
     'robertsr': 14537,
     'masoodpanahn': 1436,
     'kmecoval': 5203,
     'alij': 3740,
     'ranftp': 5930,
     'fisherc': 2286,
     'imtiazs': 2029,
     'adamjohn': 2791,
     'allredg': 1696,
     'michaelLingelbach': 4253}



Have any of these nodes been imported (as opposed to being manually placed):

.. code:: ipython3

    iinfo = pymaid.get_import_info(da1)
    iinfo


.. parsed-literal::

    Fetching info: 100%|██████████| 1/1 [00:02<00:00,  2.37s/it]




.. parsed-literal::

    {'2863104': {'n_imported_treenodes': 1524},
     '57381': {'n_imported_treenodes': 502},
     '61221': {'n_imported_treenodes': 679},
     '57353': {'n_imported_treenodes': 1005},
     '57323': {'n_imported_treenodes': 951},
     '755022': {'n_imported_treenodes': 1271},
     '27295': {'n_imported_treenodes': 1161},
     '1811442': {'n_imported_treenodes': 916},
     '57311': {'n_imported_treenodes': 608}}



Looks like some parts of these neurons have been imported. 

Last but not least, let's check who has reviewed these neurons:

.. code:: ipython3

    rev = pymaid.get_review_details(da1)
    
    # `rev` is a DataFrame that contains timestamps of when a user has reviewed a given node
    rev.head(3)




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>node_id</th>
          <th>imtiazs</th>
          <th>masoodpanahn</th>
          <th>kmecoval</th>
          <th>fisherc</th>
          <th>ratliffj</th>
          <th>helmickl</th>
          <th>alij</th>
          <th>sharifin</th>
          <th>robertsr</th>
          <th>...</th>
          <th>meechank</th>
          <th>edmondsona</th>
          <th>jamasba</th>
          <th>schlegelp</th>
          <th>calles</th>
          <th>hallouc</th>
          <th>adesinaa</th>
          <th>lovef</th>
          <th>mooree</th>
          <th>skeleton_id</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>3047214</td>
          <td>2016-05-04 21:05:00</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>...</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>2863104</td>
        </tr>
        <tr>
          <th>1</th>
          <td>3047212</td>
          <td>2016-05-04 21:05:00</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>...</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>2863104</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3047210</td>
          <td>2016-05-04 21:05:00</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>...</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>NaT</td>
          <td>2863104</td>
        </tr>
      </tbody>
    </table>
    <p>3 rows × 24 columns</p>
    </div>



Scaling things up
=================
So far, the examples have pulled only a small amount of data but ``pymaid`` scales really well. Let's demonstrate how to fetch all neurons in the FAFB dataset, shall we?

First we need to compile a list of neurons above a certain size and with a soma using :func:`pymaid.find_neurons`:

.. code:: ipython3

    all_fafb = pymaid.find_neurons(min_size=1000, only_soma=True)


.. parsed-literal::

    INFO  : Get all neurons with >= 1000 nodes (pymaid)






.. parsed-literal::

    INFO  : Found 7920 neurons matching the search parameters (pymaid)


``all_fafb`` is now a list of 7920 neurons. At the moment these neurons only consist of skeleton ID and a name - any other data such as nodes or connectors are not (yet) loaded. Note the ``NAs``:

.. code:: ipython3

    all_fafb[:5]




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>type</th>
          <th>name</th>
          <th>skeleton_id</th>
          <th>n_nodes</th>
          <th>n_connectors</th>
          <th>n_branches</th>
          <th>n_leafs</th>
          <th>cable_length</th>
          <th>soma</th>
          <th>units</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>CatmaidNeuron</td>
          <td>PD5g1_L#3 8880132 ECM Tarly family left</td>
          <td>8880131</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>1</th>
          <td>CatmaidNeuron</td>
          <td>KCaBx 12719 CF</td>
          <td>10190852</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>2</th>
          <td>CatmaidNeuron</td>
          <td>aSP-g type tract crossing alpha lobe neuron 98...</td>
          <td>983054</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>3</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT VA6 adPN 017 DB</td>
          <td>16</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>4</th>
          <td>CatmaidNeuron</td>
          <td>Multiglomerular mALT l2PN VM1+VP2+VP3+23 LTS 0...</td>
          <td>32793</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
      </tbody>
    </table>
    </div>



The not-yet-available data is lazily loaded on request:

.. code:: ipython3

    all_fafb[0].nodes.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>node_id</th>
          <th>parent_id</th>
          <th>creator_id</th>
          <th>x</th>
          <th>y</th>
          <th>z</th>
          <th>radius</th>
          <th>confidence</th>
          <th>type</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>38680419</td>
          <td>38680415</td>
          <td>127</td>
          <td>649397.0</td>
          <td>128668.0</td>
          <td>170760.0</td>
          <td>51.0</td>
          <td>5</td>
          <td>slab</td>
        </tr>
        <tr>
          <th>1</th>
          <td>38680424</td>
          <td>38680419</td>
          <td>127</td>
          <td>649497.0</td>
          <td>128728.0</td>
          <td>170840.0</td>
          <td>51.0</td>
          <td>5</td>
          <td>slab</td>
        </tr>
        <tr>
          <th>2</th>
          <td>38680426</td>
          <td>38680424</td>
          <td>127</td>
          <td>649733.0</td>
          <td>128876.0</td>
          <td>170920.0</td>
          <td>51.0</td>
          <td>5</td>
          <td>slab</td>
        </tr>
        <tr>
          <th>3</th>
          <td>38680436</td>
          <td>38680432</td>
          <td>127</td>
          <td>649945.0</td>
          <td>129012.0</td>
          <td>171040.0</td>
          <td>51.0</td>
          <td>5</td>
          <td>slab</td>
        </tr>
        <tr>
          <th>4</th>
          <td>38680432</td>
          <td>38680426</td>
          <td>127</td>
          <td>649901.0</td>
          <td>128984.0</td>
          <td>171000.0</td>
          <td>51.0</td>
          <td>5</td>
          <td>slab</td>
        </tr>
      </tbody>
    </table>
    </div>



Note how the first neuron is now the only one with more detailed info in the summary:

.. code:: ipython3

    all_fafb[:5]




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>type</th>
          <th>name</th>
          <th>skeleton_id</th>
          <th>n_nodes</th>
          <th>n_connectors</th>
          <th>n_branches</th>
          <th>n_leafs</th>
          <th>cable_length</th>
          <th>soma</th>
          <th>units</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>CatmaidNeuron</td>
          <td>PD5g1_L#3 8880132 ECM Tarly family left</td>
          <td>8880131</td>
          <td>3129</td>
          <td>96</td>
          <td>126</td>
          <td>None</td>
          <td>883607</td>
          <td>[38712262]</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>1</th>
          <td>CatmaidNeuron</td>
          <td>KCaBx 12719 CF</td>
          <td>10190852</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>2</th>
          <td>CatmaidNeuron</td>
          <td>aSP-g type tract crossing alpha lobe neuron 98...</td>
          <td>983054</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>3</th>
          <td>CatmaidNeuron</td>
          <td>Uniglomerular mALT VA6 adPN 017 DB</td>
          <td>16</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
        <tr>
          <th>4</th>
          <td>CatmaidNeuron</td>
          <td>Multiglomerular mALT l2PN VM1+VP2+VP3+23 LTS 0...</td>
          <td>32793</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>NA</td>
          <td>1 nanometer</td>
        </tr>
      </tbody>
    </table>
    </div>



So in theory, you now have about ~8k neurons to work with. But while the lazy loading is neat, it is also rather slow because it is done sequentially (i.e. one neuron at a time). If you want skeleton information (nodes + connectors) for all these neurons, you are best off triggering a full download which will use parallel threads:

.. code:: ipython3

    all_fafb.get_skeletons()

By default, ``pymaid`` uses 100 parallel threads to fetch that data. On slow internet connections, this can cause errors. To reduce the maximum number of parallel threads, you can do this:

.. code:: ipython3

    rm.max_threads = 10

That's it for now. Check out the :ref:`API reference <api>` for a full list of available functions.