.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/tutorials/ctapipe_overview.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_tutorials_ctapipe_overview.py: Analyzing Events Using ctapipe ============================== Initially presented @ LST Analysis Bootcamp in Padova, 26.11.2018 by Maximilian Linhoff (@maxnoe) & Kai A. Brügge (@mackaiver). Updated since to stay compatible with current ctapipe. .. GENERATED FROM PYTHON SOURCE LINES 10-43 .. code-block:: Python :lineno-start: 11 import timeit import tempfile from copy import deepcopy import astropy.units as u import matplotlib.pyplot as plt import numpy as np from astropy.coordinates import AltAz, angular_separation from matplotlib.colors import ListedColormap from scipy.sparse.csgraph import connected_components from traitlets.config import Config from ctapipe.calib import CameraCalibrator from ctapipe.coordinates import TelescopeFrame from ctapipe.image import ( ImageProcessor, camera_to_shower_coordinates, concentration_parameters, hillas_parameters, leakage_parameters, number_of_islands, timing_parameters, toymodel, ) from ctapipe.image.cleaning import TailcutsImageCleaner from ctapipe.io import DataWriter, EventSource, TableLoader from ctapipe.io.metadata import read_ctao_metadata from ctapipe.reco import ShowerProcessor from ctapipe.utils.datasets import get_dataset_path from ctapipe.visualization import ArrayDisplay, CameraDisplay # %matplotlib inline .. GENERATED FROM PYTHON SOURCE LINES 44-48 .. code-block:: Python :lineno-start: 44 plt.rcParams["figure.figsize"] = (12, 8) plt.rcParams["font.size"] = 14 plt.rcParams["figure.figsize"] .. rst-class:: sphx-glr-script-out .. code-block:: none [12.0, 8.0] .. GENERATED FROM PYTHON SOURCE LINES 49-52 General Information ------------------- .. GENERATED FROM PYTHON SOURCE LINES 54-69 Design ^^^^^^ - DL0 → DL3 analysis - Currently some R0 → DL2 code to be able to analyze simtel files - ctapipe is built upon the Scientific Python Stack, core dependencies are - numpy - scipy - astropy - numba .. GENERATED FROM PYTHON SOURCE LINES 71-88 Development ^^^^^^^^^^^ - ctapipe is developed as Open Source Software (BSD 3-Clause License) at https://github.com/cta-observatory/ctapipe - We use the “Github-Workflow”: - Few people (e.g. @kosack, @maxnoe) have write access to the main repository - Contributors fork the main repository and work on branches - Pull Requests are merged after Code Review and automatic execution of the test suite - Early development stage ⇒ backwards-incompatible API changes might and will happen .. GENERATED FROM PYTHON SOURCE LINES 90-100 What’s there? ^^^^^^^^^^^^^ - Reading simtel simulation files - Simple calibration, cleaning and feature extraction functions - Camera and Array plotting - Coordinate frames and transformations - Stereo-reconstruction using line intersections .. GENERATED FROM PYTHON SOURCE LINES 102-108 What’s still missing? ^^^^^^^^^^^^^^^^^^^^^ - IRF calculation - Documentation, e.g. formal definitions of coordinate frames .. GENERATED FROM PYTHON SOURCE LINES 110-124 What can you do? ^^^^^^^^^^^^^^^^ - Report issues - Hard to get started? Tell us where you are stuck - Tell user stories - Missing features - Start contributing - ctapipe needs more workpower - Implement new reconstruction features .. GENERATED FROM PYTHON SOURCE LINES 126-129 A simple Hillas analysis ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 129-134 .. code-block:: Python :lineno-start: 130 # Reading in simtel files # ^^^^^^^^^^^^^^^^^^^^^^^ # input_url = get_dataset_path("gamma_prod5.simtel.zst") .. GENERATED FROM PYTHON SOURCE LINES 135-137 EventSource() automatically detects what kind of file we are giving it, if already supported by ctapipe or an installed plugin .. GENERATED FROM PYTHON SOURCE LINES 137-147 .. code-block:: Python :lineno-start: 137 with EventSource(input_url, max_events=5) as source: print(type(source)) for event in source: print( "Id: {}, E = {:1.3f}, Telescopes: {}".format( event.count, event.simulation.shower.energy, len(event.tel) ) ) .. rst-class:: sphx-glr-script-out .. code-block:: none Id: 0, E = 0.073 TeV, Telescopes: 3 Id: 1, E = 1.745 TeV, Telescopes: 14 Id: 2, E = 1.745 TeV, Telescopes: 4 Id: 3, E = 1.745 TeV, Telescopes: 20 Id: 4, E = 1.745 TeV, Telescopes: 31 .. GENERATED FROM PYTHON SOURCE LINES 148-150 Each event is a ``SubarrayEventContainer`` holding several ``Field`` s of data, which can be containers or just numbers. Let’s look a one event: .. GENERATED FROM PYTHON SOURCE LINES 150-152 .. code-block:: Python :lineno-start: 150 event .. rst-class:: sphx-glr-script-out .. code-block:: none ctapipe.containers.SubarrayEventContainer: count: number of events processed with default 0 index.*: event indexing information with default None simulation.*: Simulated Event Information with default None with type dl0.*: DL0 subarray wide information with default None with type dl2: DL2 reconstructed event properties with default None with type monitoring.*: Subrray monitoring information with default None tel[*]: Telescope Events with default None .. GENERATED FROM PYTHON SOURCE LINES 153-155 .. code-block:: Python :lineno-start: 153 source.subarray.camera_types .. rst-class:: sphx-glr-script-out .. code-block:: none (CameraDescription(name=CHEC, geometry=CHEC, readout=CHEC), CameraDescription(name=FlashCam, geometry=FlashCam, readout=FlashCam), CameraDescription(name=LSTCam, geometry=LSTCam, readout=LSTCam), CameraDescription(name=NectarCam, geometry=NectarCam, readout=NectarCam)) .. GENERATED FROM PYTHON SOURCE LINES 156-158 Telescope-wise data is stored in ``TelescopeEventContainer``s under ``.tel``, which is a mapping by telescope id: .. GENERATED FROM PYTHON SOURCE LINES 158-160 .. code-block:: Python :lineno-start: 158 len(event.tel), event.tel.keys() .. rst-class:: sphx-glr-script-out .. code-block:: none (31, dict_keys([3, 4, 8, 9, 14, 15, 23, 24, 25, 29, 32, 36, 42, 46, 52, 56, 103, 104, 109, 110, 118, 119, 120, 126, 127, 129, 130, 137, 139, 143, 161])) .. GENERATED FROM PYTHON SOURCE LINES 161-163 .. code-block:: Python :lineno-start: 161 event.tel[3] .. rst-class:: sphx-glr-script-out .. code-block:: none ctapipe.containers.TelescopeEventContainer: index.*: event indexing information with default None simulation.*: Simulated Event Information with default None with type r0.*: Raw Data with default None with type r1.*: R1 Calibrated Data with default None with type dl0.*: DL0 Data Volume Reduced Data with default None with type dl1: DL1 images and parameters with default None with type dl2: DL2 reconstructed event properties with default None with type count: number of events processed with default 0 monitoring.*: container for event-wise monitoring data (MON) with default None muon: Container for muon analysis results with default None with type .. GENERATED FROM PYTHON SOURCE LINES 164-169 Data calibration ^^^^^^^^^^^^^^^^ The ``CameraCalibrator`` calibrates the event (obtaining the ``dl1`` images). .. GENERATED FROM PYTHON SOURCE LINES 169-171 .. code-block:: Python :lineno-start: 169 calibrator = CameraCalibrator(subarray=source.subarray) .. GENERATED FROM PYTHON SOURCE LINES 172-174 .. code-block:: Python :lineno-start: 172 calibrator(event) .. GENERATED FROM PYTHON SOURCE LINES 175-179 Event displays ^^^^^^^^^^^^^^ Let’s use ctapipe’s plotting facilities to plot the telescope images .. GENERATED FROM PYTHON SOURCE LINES 179-181 .. code-block:: Python :lineno-start: 179 tel_id = 130 .. GENERATED FROM PYTHON SOURCE LINES 182-188 .. code-block:: Python :lineno-start: 182 geometry = source.subarray.tel[tel_id].camera.geometry.transform_to(TelescopeFrame()) dl1 = event.tel[tel_id].dl1 print(geometry) print(dl1) .. rst-class:: sphx-glr-script-out .. code-block:: none NectarCam {'image': array([-0.3856395 , 1.2851689 , 1.403967 , ..., 2.6847053 , 0.04941979, -1.0566036 ], shape=(1855,), dtype=float32), 'image_mask': None, 'is_valid': True, 'parameters': None, 'peak_time': array([10.433766 , 2.8860204, 35.608727 , ..., 36.9452 , 36.13659 , 36.995068 ], shape=(1855,), dtype=float32), 'pixel_status': array([7, 7, 7, ..., 7, 7, 7], shape=(1855,), dtype=uint8)} .. GENERATED FROM PYTHON SOURCE LINES 189-191 .. code-block:: Python :lineno-start: 189 dl1.image .. rst-class:: sphx-glr-script-out .. code-block:: none array([-0.3856395 , 1.2851689 , 1.403967 , ..., 2.6847053 , 0.04941979, -1.0566036 ], shape=(1855,), dtype=float32) .. GENERATED FROM PYTHON SOURCE LINES 192-196 .. code-block:: Python :lineno-start: 192 display = CameraDisplay(geometry) display.image = dl1.image display.add_colorbar() .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_001.png :alt: NectarCam :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 197-203 Image Cleaning ^^^^^^^^^^^^^^ ctapipe allows most configuration options to be configured by telescope type. The example below configures some of the parameters for the `~ctapipe.image.TailcutsImageCleaner`: .. GENERATED FROM PYTHON SOURCE LINES 205-206 .. code-block:: Python :lineno-start: 205 print(source.subarray.telescope_types) .. rst-class:: sphx-glr-script-out .. code-block:: none (TelescopeDescription(type='SST', optics_name='ASTRI', camera_name='CHEC'), TelescopeDescription(type='LST', optics_name='LST', camera_name='LSTCam'), TelescopeDescription(type='MST', optics_name='MST', camera_name='NectarCam'), TelescopeDescription(type='MST', optics_name='MST', camera_name='FlashCam')) .. GENERATED FROM PYTHON SOURCE LINES 207-220 .. code-block:: Python :lineno-start: 207 cleaning = TailcutsImageCleaner( source.subarray, picture_threshold_pe=[ ("type", "*", 7), # global default ("type", "MST_MST_NectarCam", 8), ("type", "SST_ASTRI_CHEC", 4), ], boundary_threshold_pe=[ ("type", "*", 3.5), ("type", "MST_MST_NectarCam", 4), ("type", "SST_ASTRI_CHEC", 2), ], ) .. GENERATED FROM PYTHON SOURCE LINES 221-226 .. code-block:: Python :lineno-start: 221 image_mask = cleaning( tel_id, dl1.image, ) .. GENERATED FROM PYTHON SOURCE LINES 227-244 .. code-block:: Python :lineno-start: 227 fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5)) d1 = CameraDisplay(geometry, ax=ax1) d2 = CameraDisplay(geometry, ax=ax2) ax1.set_title("Image") d1.image = dl1.image d1.highlight_pixels(image_mask, color="w") d1.add_colorbar(ax=ax1) ax2.set_title("Pulse Time") d2.image = dl1.peak_time - np.average(dl1.peak_time, weights=dl1.image) d2.cmap = "RdBu_r" d2.add_colorbar(ax=ax2) d2.set_limits_minmax(-20, 20) .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_002.png :alt: Image, Pulse Time :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 245-248 Image Parameters ^^^^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 248-252 .. code-block:: Python :lineno-start: 248 hillas = hillas_parameters(geometry[image_mask], dl1.image[image_mask]) print(hillas) .. rst-class:: sphx-glr-script-out .. code-block:: none {'fov_lat': , 'fov_lon': , 'intensity': np.float64(303.9444303512573), 'kurtosis': np.float64(2.4649477808816287), 'length': , 'length_uncertainty': , 'phi': , 'psi': , 'psi_uncertainty': , 'r': , 'skewness': np.float64(-0.36090637987956614), 'transverse_cog_uncertainty': , 'width': , 'width_uncertainty': } .. GENERATED FROM PYTHON SOURCE LINES 253-259 .. code-block:: Python :lineno-start: 253 plt.figure() display = CameraDisplay(geometry, image=dl1.image) display.highlight_pixels(image_mask, color="w") display.add_colorbar() display.overlay_moments(hillas, color="xkcd:red", n_sigma=2) .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_003.png :alt: NectarCam :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 260-263 .. code-block:: Python :lineno-start: 260 timing = timing_parameters(geometry, dl1.image, dl1.peak_time, hillas, image_mask) print(timing) .. rst-class:: sphx-glr-script-out .. code-block:: none {'deviation': 0.7921518270503455, 'intercept': np.float64(21.81257256241495), 'slope': } .. GENERATED FROM PYTHON SOURCE LINES 264-272 .. code-block:: Python :lineno-start: 264 long, trans = camera_to_shower_coordinates( geometry.pix_x, geometry.pix_y, hillas.fov_lon, hillas.fov_lat, hillas.psi ) plt.figure() plt.plot(long[image_mask], dl1.peak_time[image_mask], "o") plt.plot(long[image_mask], timing.slope * long[image_mask] + timing.intercept) .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_004.png :alt: ctapipe overview :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_004.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none [] .. GENERATED FROM PYTHON SOURCE LINES 273-276 .. code-block:: Python :lineno-start: 273 leakage = leakage_parameters(geometry, dl1.image, image_mask) print(leakage) .. rst-class:: sphx-glr-script-out .. code-block:: none {'intensity_width_1': np.float32(0.0), 'intensity_width_2': np.float32(0.0), 'pixels_width_1': np.float64(0.0), 'pixels_width_2': np.float64(0.0)} .. GENERATED FROM PYTHON SOURCE LINES 277-281 .. code-block:: Python :lineno-start: 277 disp = CameraDisplay(geometry) disp.image = dl1.image disp.highlight_pixels(geometry.get_border_pixel_mask(1), linewidth=2, color="xkcd:red") .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_005.png :alt: NectarCam :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 282-286 .. code-block:: Python :lineno-start: 282 n_islands, island_id = number_of_islands(geometry, image_mask) print(n_islands) .. rst-class:: sphx-glr-script-out .. code-block:: none 2 .. GENERATED FROM PYTHON SOURCE LINES 287-291 .. code-block:: Python :lineno-start: 287 conc = concentration_parameters(geometry, dl1.image, hillas) print(conc) .. rst-class:: sphx-glr-script-out .. code-block:: none {'cog': np.float64(0.2571728392782083), 'core': np.float64(0.4014042974365615), 'pixel': np.float64(0.09514053791448862)} .. GENERATED FROM PYTHON SOURCE LINES 292-321 Putting it all together / Stereo reconstruction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ All these steps are now unified in several components configurable through the config system, mainly: - CameraCalibrator for DL0 → DL1 (Images) - ImageProcessor for DL1 (Images) → DL1 (Parameters) - ShowerProcessor for stereo reconstruction of the shower geometry - DataWriter for writing data into HDF5 A command line tool doing these steps and writing out data in HDF5 format is available as ``ctapipe-process`` In addition to the event data, ``DataWriter`` stores metadata describing the resulting CTAO data product. Most of this metadata, such as the data level and observation identifier, is derived automatically. Information that cannot be inferred from the input can be supplied using a configuration. The ``Contact`` section identifies the person or organization responsible for the generated data product, while ``ProductMetadata`` provides overrides for fields in the CTAO data-product model. Simulation node metadata requires a site identifier or observatory location. The example Prod5 file does not contain a valid observatory location, so we explicitly identify it as a CTAO-South simulation. See the `ctao-datamodel DataProducts documentation `__ for the definitions of the data-product metadata fields. .. GENERATED FROM PYTHON SOURCE LINES 321-402 .. code-block:: Python :lineno-start: 323 image_processor_config = Config( { "ImageProcessor": { "image_cleaner_type": "TailcutsImageCleaner", "TailcutsImageCleaner": { "picture_threshold_pe": [ ("type", "LST_LST_LSTCam", 7.5), ("type", "MST_MST_FlashCam", 8), ("type", "MST_MST_NectarCam", 8), ("type", "SST_ASTRI_CHEC", 7), ], "boundary_threshold_pe": [ ("type", "LST_LST_LSTCam", 5), ("type", "MST_MST_FlashCam", 4), ("type", "MST_MST_NectarCam", 4), ("type", "SST_ASTRI_CHEC", 4), ], }, } } ) writer_config = Config( { "DataWriter": { "Contact": { "name": "Example User", "organization": "Example Organization", "email": "user@example.org", }, "ProductMetadata": { "instance": { "site_id": "CTAO-South", }, }, } } ) input_url = get_dataset_path("gamma_prod5.simtel.zst") f = tempfile.NamedTemporaryFile(suffix=".hdf5") with EventSource(input_url) as source: calibrator = CameraCalibrator(subarray=source.subarray) image_processor = ImageProcessor( subarray=source.subarray, config=image_processor_config ) shower_processor = ShowerProcessor(subarray=source.subarray) horizon_frame = AltAz() with DataWriter( source, output_path=f.name, overwrite=True, write_dl2=True, config=writer_config ) as writer: for event in source: energy = event.simulation.shower.energy n_telescopes = len(event.tel) event_id = event.index.event_id print(f"Id: {event_id}, E = {energy:1.3f}, Telescopes: {n_telescopes}") calibrator(event) image_processor(event) shower_processor(event) stereo = event.dl2.geometry["HillasReconstructor"] if stereo.is_valid: print(" Alt: {:.2f}°".format(stereo.alt.deg)) print(" Az: {:.2f}°".format(stereo.az.deg)) print(" Hmax: {:.0f}".format(stereo.h_max)) print(" CoreX: {:.1f}".format(stereo.core_x)) print(" CoreY: {:.1f}".format(stereo.core_y)) print(" Multiplicity: {:d}".format(len(stereo.telescopes))) # save a nice event for plotting later if event.count == 3: plotting_event = deepcopy(event) writer(event) .. rst-class:: sphx-glr-script-out .. code-block:: none Overwriting /tmp/tmpmte7ynof.hdf5 Id: 4009, E = 0.073 TeV, Telescopes: 3 Alt: 70.18° Az: 0.41° Hmax: 14066 m CoreX: -113.2 m CoreY: 366.5 m Multiplicity: 3 Id: 5101, E = 1.745 TeV, Telescopes: 14 Alt: 70.00° Az: 359.95° Hmax: 10329 m CoreX: -739.5 m CoreY: -268.4 m Multiplicity: 10 Id: 5103, E = 1.745 TeV, Telescopes: 4 Alt: 70.21° Az: 0.50° Hmax: 11526 m CoreX: 832.1 m CoreY: 945.4 m Multiplicity: 2 Id: 5104, E = 1.745 TeV, Telescopes: 20 Alt: 70.16° Az: 0.07° Hmax: 10643 m CoreX: -518.3 m CoreY: -451.5 m Multiplicity: 15 Id: 5105, E = 1.745 TeV, Telescopes: 31 Alt: 70.02° Az: 0.13° Hmax: 10449 m CoreX: -327.7 m CoreY: 408.8 m Multiplicity: 25 Id: 5108, E = 1.745 TeV, Telescopes: 3 Alt: 69.95° Az: 359.79° Hmax: 10409 m CoreX: -1304.5 m CoreY: 205.6 m Multiplicity: 2 Id: 5109, E = 1.745 TeV, Telescopes: 8 Alt: 69.62° Az: 359.93° Hmax: 10886 m CoreX: -1067.6 m CoreY: 480.5 m Multiplicity: 3 .. GENERATED FROM PYTHON SOURCE LINES 403-405 The validated CTAO data-product metadata can be read directly from the file. You can equivalently inspect it with ``ctapipe-fileinfo``. .. GENERATED FROM PYTHON SOURCE LINES 405-410 .. code-block:: Python :lineno-start: 406 metadata = read_ctao_metadata(f.name) print("Data product metadata:") print(metadata.model_dump_json(indent=2)) .. rst-class:: sphx-glr-script-out .. code-block:: none Data product metadata: { "ctao_metadata_version": "2.2.0", "description": "ctapipe Data Product", "data": { "level": "DL2", "division": "Event", "association": "Subarray", "type": "ObservationSim" }, "instance": { "id": "01a1c61f-3076-72e3-9b25-ed2c831d469b", "obs_id": 1, "event_type": null, "ae_id": null, "ae_class": null, "subarray_id": null, "chunk_id": null, "batch_id": null, "calibration_service_id": null, "observing_night": null, "sublevel_id": "PARAMETERS|GEOMETRY", "target_id": null, "region_id": null, "observing_period_id": null, "lunar_cycle_id": null, "facility_name": "Simulated-CTAO", "site_id": "CTAO-South", "particle_pdgid": null, "messenger": null, "category": null, "data_source": null, "assembly_name": null, "metric_set_name": null, "local_run_id": null }, "curation": { "release": null, "reference": null, "license": "CC-BY-SA-4.0", "license_url": "https://creativecommons.org/licenses/by-sa/4.0/", "copyright": null, "rights": null, "release_date": null, "valid_from": null, "valid_to": null }, "model": { "name": "ctapipe", "version": "v8.0.0", "url": null }, "disclaimer": null, "creation_time": "2026-11-10T15:41:35.221917000", "contact": { "name": "Example User", "organization": "Example Organization", "email": "user@example.org" }, "activity": { "process": "data_processing", "name": "/home/docs/checkouts/readthedocs.org/user_builds/ctapipe/envs/stable/bin/python", "description": null, "id": "5d8228b8-874b-439a-b8dd-0107910d7264", "start": "2026-11-10T15:41:02.458000000", "end": "2026-11-10T15:41:35.222432000", "software": { "name": "ctapipe", "version": "0.33.0", "url": "https://github.com/cta-observatory/ctapipe" }, "configuration_id": "", "inputs": null }, "observation": null, "acquisition": null, "sdc": null, "simulation": { "production_name": "unknown", "node": { "zenith": { "value": 20.0, "unit": "deg" }, "azimuth": { "value": 4.576543183146013e-6, "unit": "deg" }, "hour_angle": null, "declination": null, "nsb_factor": 1.0 } } } .. GENERATED FROM PYTHON SOURCE LINES 411-415 .. code-block:: Python :lineno-start: 411 loader = TableLoader(f.name) events = loader.read_subarray_events() .. GENERATED FROM PYTHON SOURCE LINES 416-428 .. code-block:: Python :lineno-start: 416 theta = angular_separation( events["HillasReconstructor_az"].quantity, events["HillasReconstructor_alt"].quantity, events["true_az"].quantity, events["true_alt"].quantity, ) plt.hist(theta.to_value(u.deg) ** 2, bins=25, range=[0, 0.3]) plt.xlabel(r"$\theta² / deg²$") None .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_006.png :alt: ctapipe overview :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 429-432 ArrayDisplay ------------ .. GENERATED FROM PYTHON SOURCE LINES 432-465 .. code-block:: Python :lineno-start: 433 angle_offset = plotting_event.monitoring.pointing.azimuth plotting_hillas = {} plotting_core = {} for tel_id, tel_event in plotting_event.tel.items(): plotting_hillas[tel_id] = tel_event.dl1.parameters.hillas plotting_core[tel_id] = tel_event.dl1.parameters.core.psi disp = ArrayDisplay(source.subarray) disp.set_line_hillas(plotting_hillas, plotting_core, 500) plt.scatter( plotting_event.simulation.shower.core_x, plotting_event.simulation.shower.core_y, s=200, c="k", marker="x", label="True Impact", ) plt.scatter( plotting_event.dl2.geometry["HillasReconstructor"].core_x, plotting_event.dl2.geometry["HillasReconstructor"].core_y, s=200, c="r", marker="x", label="Estimated Impact", ) plt.legend() .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_007.png :alt: MonteCarloArray :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_007.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 466-469 Reading the LST dl1 data ^^^^^^^^^^^^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 469-478 .. code-block:: Python :lineno-start: 470 loader = TableLoader(f.name) dl1_table = loader.read_telescope_events( ["LST_LST_LSTCam"], dl2=False, true_parameters=False, ) .. GENERATED FROM PYTHON SOURCE LINES 479-488 .. code-block:: Python :lineno-start: 479 plt.scatter( np.log10(dl1_table["true_energy"].quantity / u.TeV), np.log10(dl1_table["hillas_intensity"]), ) plt.xlabel("log10(E / TeV)") plt.ylabel("log10(intensity)") None .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_008.png :alt: ctapipe overview :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_008.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 489-505 Isn’t python slow? ------------------ - Many of you might have heard: “Python is slow”. - That’s trueish. - All python objects are classes living on the heap, even integers. - Looping over lots of “primitives” is quite slow compared to other languages. | ⇒ Vectorize as much as possible using numpy | ⇒ Use existing interfaces to fast C / C++ / Fortran code | ⇒ Optimize using numba **But: “Premature Optimization is the root of all evil” — Donald Knuth** So profile to find exactly what is slow. .. GENERATED FROM PYTHON SOURCE LINES 507-514 Why use python then? ^^^^^^^^^^^^^^^^^^^^ - Python works very well as *glue* for libraries of all kinds of languages - Python has a rich ecosystem for data science, physics, algorithms, astronomy .. GENERATED FROM PYTHON SOURCE LINES 516-520 Example: Number of Islands ^^^^^^^^^^^^^^^^^^^^^^^^^^ Find all groups of pixels, that survived the cleaning .. GENERATED FROM PYTHON SOURCE LINES 520-525 .. code-block:: Python :lineno-start: 522 geometry = loader.subarray.tel[1].camera.geometry .. GENERATED FROM PYTHON SOURCE LINES 526-528 Let’s create a toy images with several islands; .. GENERATED FROM PYTHON SOURCE LINES 528-548 .. code-block:: Python :lineno-start: 529 rng = np.random.default_rng(42) image = np.zeros(geometry.n_pixels) for i in range(9): model = toymodel.Gaussian( x=rng.uniform(-0.8, 0.8) * u.m, y=rng.uniform(-0.8, 0.8) * u.m, width=rng.uniform(0.05, 0.075) * u.m, length=rng.uniform(0.1, 0.15) * u.m, psi=rng.uniform(0, 2 * np.pi) * u.rad, ) new_image, sig, bg = model.generate_image( geometry, intensity=rng.uniform(1000, 3000), nsb_level_pe=5 ) image += new_image .. GENERATED FROM PYTHON SOURCE LINES 549-555 .. code-block:: Python :lineno-start: 549 geometry = geometry.transform_to(TelescopeFrame()) image_mask = cleaning( tel_id=1, image=image, ) .. GENERATED FROM PYTHON SOURCE LINES 556-562 .. code-block:: Python :lineno-start: 556 disp = CameraDisplay(geometry) disp.image = image disp.highlight_pixels(image_mask, color="xkcd:red", linewidth=1.5) disp.add_colorbar() .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_009.png :alt: LSTCam :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_009.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 563-601 .. code-block:: Python :lineno-start: 563 def num_islands_python(camera, clean): """A breadth first search to find connected islands of neighboring pixels in the cleaning set""" # the camera geometry has a [n_pixel, n_pixel] boolean array # that is True where two pixels are neighbors neighbors = camera.neighbor_matrix island_ids = np.zeros(camera.n_pixels) current_island = 0 # a set to remember which pixels we already visited visited = set() # go only through the pixels, that survived cleaning for pix_id in np.where(clean)[0]: if pix_id not in visited: # remember that we already checked this pixel visited.add(pix_id) # if we land in the outer loop again, we found a new island current_island += 1 island_ids[pix_id] = current_island # now check all neighbors of the current pixel recursively to_check = set(np.where(neighbors[pix_id] & clean)[0]) while to_check: pix_id = to_check.pop() if pix_id not in visited: visited.add(pix_id) island_ids[pix_id] = current_island to_check.update(np.where(neighbors[pix_id] & clean)[0]) n_islands = current_island return n_islands, island_ids .. GENERATED FROM PYTHON SOURCE LINES 602-604 .. code-block:: Python :lineno-start: 602 n_islands, island_ids = num_islands_python(geometry, image_mask) .. GENERATED FROM PYTHON SOURCE LINES 605-615 .. code-block:: Python :lineno-start: 605 cmap = plt.get_cmap("Paired") cmap = ListedColormap(cmap.colors[:n_islands]) cmap.set_under("k") disp = CameraDisplay(geometry) disp.image = island_ids disp.cmap = cmap disp.set_limits_minmax(0.5, n_islands + 0.5) disp.add_colorbar() .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_010.png :alt: LSTCam :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_010.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 616-618 .. code-block:: Python :lineno-start: 616 timeit.timeit(lambda: num_islands_python(geometry, image_mask), number=1000) / 1000 .. rst-class:: sphx-glr-script-out .. code-block:: none 0.001865510514999869 .. GENERATED FROM PYTHON SOURCE LINES 619-633 .. code-block:: Python :lineno-start: 621 def num_islands_scipy(geometry, clean): neighbors = geometry.neighbor_matrix_sparse clean_neighbors = neighbors[clean][:, clean] num_islands, labels = connected_components(clean_neighbors, directed=False) island_ids = np.zeros(geometry.n_pixels) island_ids[clean] = labels + 1 return num_islands, island_ids .. GENERATED FROM PYTHON SOURCE LINES 634-636 .. code-block:: Python :lineno-start: 634 n_islands_s, island_ids_s = num_islands_scipy(geometry, image_mask) .. GENERATED FROM PYTHON SOURCE LINES 637-643 .. code-block:: Python :lineno-start: 637 disp = CameraDisplay(geometry) disp.image = island_ids_s disp.cmap = cmap disp.set_limits_minmax(0.5, n_islands_s + 0.5) disp.add_colorbar() .. image-sg:: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_011.png :alt: LSTCam :srcset: /auto_examples/tutorials/images/sphx_glr_ctapipe_overview_011.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 644-646 .. code-block:: Python :lineno-start: 644 timeit.timeit(lambda: num_islands_scipy(geometry, image_mask), number=10000) / 10000 .. rst-class:: sphx-glr-script-out .. code-block:: none 0.00014489372220000406 .. GENERATED FROM PYTHON SOURCE LINES 647-651 **A lot less code, and a factor 3 speed improvement** Finally, current ctapipe implementation is using numba: .. GENERATED FROM PYTHON SOURCE LINES 651-652 .. code-block:: Python :lineno-start: 651 timeit.timeit(lambda: number_of_islands(geometry, image_mask), number=100000) / 100000 .. rst-class:: sphx-glr-script-out .. code-block:: none 1.6921035079999455e-05 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 25.039 seconds) .. _sphx_glr_download_auto_examples_tutorials_ctapipe_overview.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: ctapipe_overview.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: ctapipe_overview.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: ctapipe_overview.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_