Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
4c049c4
added explore data tutorial
tjarkmiener Nov 6, 2026
3b52474
cache tutorial data before and read it from there in the tutorials
tjarkmiener Nov 6, 2026
ac9ce8a
added changelog
tjarkmiener Nov 6, 2026
53bcce4
upload docs build to artifacts and let RTD execute tutorials
tjarkmiener Nov 6, 2026
fc05fc8
make animations in docs
tjarkmiener Nov 6, 2026
2e6b5d2
deploy github pages for the docs and remove readthedocs
tjarkmiener Nov 6, 2026
a6a3407
add tutorialfor training the models
tjarkmiener Nov 9, 2026
29cec92
support data format v8
tjarkmiener Nov 6, 2026
6f02756
add changelog
tjarkmiener Nov 6, 2026
dc86da2
added the config example files
tjarkmiener Nov 9, 2026
d8d7ae0
fix docs
tjarkmiener Nov 9, 2026
f65e878
fix lint
tjarkmiener Nov 9, 2026
524b78d
remove mocking the installation of DL frameworks
tjarkmiener Nov 9, 2026
56ca403
use more events and reduce batch size
tjarkmiener Nov 9, 2026
661296e
fix label for docs only to not execute all CI jobs
tjarkmiener Nov 9, 2026
e8a64f8
fix docs label
tjarkmiener Nov 9, 2026
a0d1775
increase further the stats
tjarkmiener Nov 9, 2026
f1a4345
increase stats and reduce batch size further
tjarkmiener Nov 9, 2026
bd6fbb9
fix docs only label
tjarkmiener Nov 9, 2026
a92eb46
added a section for plotting the training curves and fixing the pt ->…
tjarkmiener Nov 9, 2026
9c24391
fix pre-commit hooks
tjarkmiener Nov 9, 2026
a0078b2
move back to more stats to run tutorial properly
tjarkmiener Nov 9, 2026
8dc3850
use normal size cuts in example config; reduce stats
tjarkmiener Nov 9, 2026
7473524
fixs docs
tjarkmiener Nov 9, 2026
baa4408
speed up doc building
tjarkmiener Nov 9, 2026
3be3c98
enforce explicit order of tutorials
tjarkmiener Nov 9, 2026
d22aba4
add disclaimer for non optimal training curves; just for showing the …
tjarkmiener Nov 9, 2026
4f93a2b
added verbose
tjarkmiener Nov 9, 2026
4e522f3
added verbose on stdou on the PIPE
tjarkmiener Nov 9, 2026
fccdb65
Trigger CI
tjarkmiener Nov 9, 2026
4699c8c
polish some explanations
tjarkmiener Nov 9, 2026
f7076db
add a version switcher for switch to tag versions
tjarkmiener Nov 9, 2026
9fc7fbc
added back the doc CIs
tjarkmiener Nov 9, 2026
283c13b
added tutorial 3 for prediction of CTLearn models
tjarkmiener Nov 10, 2026
1ee623e
polish train and predict tutorials
tjarkmiener Nov 10, 2026
324e237
add tutorial number to part sections
tjarkmiener Nov 10, 2026
127c42b
further polishing docs
tjarkmiener Nov 10, 2026
c792894
add three boxes for user/dev guide and api docs
tjarkmiener Nov 10, 2026
d6385e3
bump dl1dh version and polish docs
tjarkmiener Nov 11, 2026
26cafb5
added top bar for selecting different section
tjarkmiener Nov 11, 2026
63191c5
add back a hidden toctree
tjarkmiener Nov 11, 2026
96d719c
added github and pypi button
tjarkmiener Nov 11, 2026
00ef4ca
use pydata theme for docs
tjarkmiener Nov 11, 2026
00a648b
add logo and icon for docs
tjarkmiener Nov 11, 2026
97b9a5c
remove background from logo and polish some docs
tjarkmiener Nov 11, 2026
53ea1fc
fix bug with licence in pyproject
tjarkmiener Nov 11, 2026
adff05a
fix path to pyproject in the conf of the docs
tjarkmiener Nov 11, 2026
092f886
make docs a little bit more beautiful
tjarkmiener Nov 11, 2026
40f8c8a
fixs version switcher; also point to legacy read the docs for older v…
tjarkmiener Nov 12, 2026
43616dd
added skip-docs-tutorials label
tjarkmiener Nov 12, 2026
7a6acef
polish docs
tjarkmiener Nov 13, 2026
1752e85
polish docs
tjarkmiener Nov 16, 2026
e91fba7
fix typos
tjarkmiener Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .codespell-ignores
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
CHEC
fpr
210 changes: 204 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,9 @@ jobs:

cache-test-data:
needs: lint
if: >-
github.event_name != 'pull_request' ||
!contains(github.event.pull_request.labels.*.name, 'documentation-only')
runs-on: ubuntu-24.04
env:
SKIP_DOCS_TUTORIALS: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'skip-docs-tutorials') }}
steps:
- name: Cache ctapipe data
id: cache-ctapipe
Expand All @@ -40,21 +39,220 @@ jobs:
path: ~/.cache/ctapipe
key: ctapipe-data-${{ runner.os }}-v1

- name: Cache tutorial data
if: env.SKIP_DOCS_TUTORIALS != 'true'
id: cache-tutorial-data
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
with:
path: ~/.cache/ctlearn/tutorial-data
key: ctlearn-tutorial-data-${{ runner.os }}-v2

- name: Set up Python
if: steps.cache-ctapipe.outputs.cache-hit != 'true'
if: >-
steps.cache-ctapipe.outputs.cache-hit != 'true' ||
(env.SKIP_DOCS_TUTORIALS != 'true' && steps.cache-tutorial-data.outputs.cache-hit != 'true')
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: '3.12'

- name: Install ctapipe
if: steps.cache-ctapipe.outputs.cache-hit != 'true'
if: >-
steps.cache-ctapipe.outputs.cache-hit != 'true' ||
(env.SKIP_DOCS_TUTORIALS != 'true' && steps.cache-tutorial-data.outputs.cache-hit != 'true')
run: pip install ctapipe

- name: Download test data
- name: Download ctapipe test data
if: steps.cache-ctapipe.outputs.cache-hit != 'true'
run: |
python -c "from ctapipe.utils import get_dataset_path; get_dataset_path('gamma_test_large.simtel.gz'); get_dataset_path('proton_20deg_0deg_run4___cta-prod5-paranal_desert-2147m-Paranal-dark-100evts.simtel.zst')"

- name: Download tutorial data
if: env.SKIP_DOCS_TUTORIALS != 'true' && steps.cache-tutorial-data.outputs.cache-hit != 'true'
run: |
python - <<'PY'
from pathlib import Path
from ctapipe.utils.download import download_file

cache_dir = Path.home() / ".cache" / "ctlearn" / "tutorial-data"
files = {
"gamma_prod5.simtel.zst": (
"https://minio-cta.zeuthen.desy.de/dpps-testdata-public/data/"
"ctapipe-test-data/v1.1.0/gamma_prod5.simtel.zst"
),
"gamma-diffuse_with_images_00.dl2.h5": (
"https://zenodo.org/api/records/7298569/files/"
"gamma-diffuse_with_images_00.dl2.h5/content"
),
"gamma-diffuse_with_images_01.dl2.h5": (
"https://zenodo.org/api/records/7298569/files/"
"gamma-diffuse_with_images_01.dl2.h5/content"
),
"proton_with_images_00.dl2.h5": (
"https://zenodo.org/api/records/7298569/files/"
"proton_with_images_00.dl2.h5/content"
),
"proton_with_images_01.dl2.h5": (
"https://zenodo.org/api/records/7298569/files/"
"proton_with_images_01.dl2.h5/content"
),
}
for name, url in files.items():
download_file(url=url, path=cache_dir / name, progress=True)
PY

docs:
needs: cache-test-data
runs-on: ubuntu-24.04
permissions:
contents: read
env:
PIP_EXTRA_INDEX_URL: "https://download.pytorch.org/whl/cpu"
CTLEARN_DOCS_VERSION: ${{ github.ref_type == 'tag' && github.ref_name || 'main' }}
SKIP_DOCS_TUTORIALS: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'skip-docs-tutorials') }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: "3.12"

- name: Cache documentation dependencies
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-docs-pip-py312-v1

- name: Install documentation dependencies
run: |
python -m pip install --upgrade pip
python -m pip install .
python -m pip install -r docs/requirements.txt

- name: Get ctapipe version for processed-data cache
if: env.SKIP_DOCS_TUTORIALS != 'true'
id: ctapipe-version
run: echo "version=$(python -c 'import ctapipe; print(ctapipe.__version__)')" >> "$GITHUB_OUTPUT"

- name: Restore tutorial input data
if: env.SKIP_DOCS_TUTORIALS != 'true'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
with:
path: ~/.cache/ctlearn/tutorial-data
key: ctlearn-tutorial-data-${{ runner.os }}-v2

- name: Restore processed tutorial data
if: env.SKIP_DOCS_TUTORIALS != 'true'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
with:
path: |
~/.cache/ctlearn/tutorial-data/gamma_prod5-max500.r1.dl1.h5
~/.cache/ctlearn/tutorial-data/ctapipe-*
key: >-
ctlearn-tutorial-processed-${{ runner.os }}-v2-max500-${{ steps.ctapipe-version.outputs.version }}-${{ hashFiles('examples/tutorials/explore_public_data.py', 'docs/requirements.txt', 'pyproject.toml') }}

- name: Build documentation
env:
CTLEARN_TUTORIAL_MAX_EVENTS: "500"
run: |
if [[ "$SKIP_DOCS_TUTORIALS" == "true" ]]; then
python -m sphinx -b html --keep-going -D sphinx_gallery_conf.plot_gallery=0 docs/source docs/build/html
else
python -m sphinx -b html --keep-going docs/source docs/build/html
fi

- name: Upload rendered documentation for review
if: >-
always() &&
(github.event_name == 'pull_request' || github.event_name == 'push')
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: docs-${{ github.event_name }}-${{ github.run_id }}-${{ github.run_attempt }}
path: docs/build/html
retention-days: 14
if-no-files-found: warn

deploy-docs:
if: >-
github.event_name == 'push' &&
(
github.ref == 'refs/heads/main' ||
startsWith(github.ref, 'refs/tags/')
)
needs: docs
runs-on: ubuntu-24.04
permissions:
contents: write
pages: write
id-token: write
concurrency:
group: github-pages
cancel-in-progress: false
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Configure GitHub Pages
uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5

- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0

- name: Download rendered documentation
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: docs-${{ github.event_name }}-${{ github.run_id }}-${{ github.run_attempt }}
path: docs/build/html

- name: Merge version into Pages site
env:
VERSION: ${{ github.ref_name }}
REF: ${{ github.ref }}
run: |
site_dir="$RUNNER_TEMP/gh-pages"
if git ls-remote --exit-code origin refs/heads/gh-pages >/dev/null 2>&1; then
git fetch origin gh-pages
git worktree add -B gh-pages "$site_dir" FETCH_HEAD
else
git worktree add --orphan -b gh-pages "$site_dir"
fi

if [[ "$REF" == "refs/heads/main" ]]; then
version_path="main"
else
version_path="${VERSION//\//--}"
fi

mkdir -p "$site_dir/$version_path"
rsync -a --delete docs/build/html/ "$site_dir/$version_path/"
if [[ "$version_path" == "main" ]]; then
rsync -a docs/build/html/ "$site_dir/"
fi
touch "$site_dir/.nojekyll"

git -C "$site_dir" add -A
if ! git -C "$site_dir" diff --cached --quiet; then
git -C "$site_dir" -c user.name="github-actions[bot]" \
-c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit -m "Deploy documentation for $VERSION"
git push origin gh-pages
fi

mkdir -p "$RUNNER_TEMP/pages-artifact"
rsync -a --exclude='.git' "$site_dir/" "$RUNNER_TEMP/pages-artifact/"

- name: Upload GitHub Pages artifact
uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4
with:
path: ${{ runner.temp }}/pages-artifact

- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4

pixi:
needs: cache-test-data
if: >-
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ output_dir2/
*.tar.gz
*.pbtxt
*.png
!docs/source/_static/ctlearn_favicon.png
*.csv
*~
*.log
Expand All @@ -31,6 +32,9 @@ dist
.pytest_cache/*.log
# Sphinx documentation
docs/build/
docs/source/auto_tutorials/
docs/source/user-guide/auto_tutorials/
docs/source/sg_execution_times.rst
build/
# Default pytorch output
run/
Expand All @@ -43,3 +47,5 @@ test/
*.sh
output*
config*
!docs/source/config/
!docs/source/config/*.json
16 changes: 0 additions & 16 deletions .readthedocs.yaml

This file was deleted.

39 changes: 34 additions & 5 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,13 @@ Contributing to CTLearn
=======================

CTLearn is open source and welcomes contributions from the community!
CTLearn follows the same code style and development guidelines as the ctapipe
community. We recommend reviewing the `ctapipe Developer Guide
<https://ctapipe.readthedocs.io/en/stable/developer-guide/index.html>`_ in
detail and following the guidelines below:

Contributing guidelines
-----------------------
^^^^^^^^^^^^^^^^^^^^^^^

In CTLearn we follow the "fork-and-pull" Git workflow:

Expand All @@ -16,8 +20,13 @@ In CTLearn we follow the "fork-and-pull" Git workflow:
* Push your work back up to your fork on GitHub
* Submit a Pull Request so the core developer team can review your changes

Some CI features are unavailable for pull requests from forks. Frequent
contributors and those preparing pull requests with significant changes
can contact the maintainers to request write access and contribute from a
branch in the main repository instead of a fork.

Code quality
------------
^^^^^^^^^^^^

CTLearn uses pre-commit hooks for whitespace and file checks, spelling, and
Ruff linting and formatting. Install the development dependencies and set up
Expand All @@ -35,21 +44,41 @@ all files, use:

pre-commit run --all-files

CI also runs ``SonarQube`` analysis through ``SonarCloud`` on pull requests from this
repository. It checks code quality, including potential bugs, security issues,
maintainability, and test coverage.

Documentation builds
^^^^^^^^^^^^^^^^^^^^

For documentation-only pull requests, apply the ``documentation-only`` label
to skip the pip and Pixi test jobs and the dependent SonarCloud scan. The
documentation build still runs, as do the lint and data-cache jobs and any
other applicable workflow checks. Removing the label restores the skipped
pull request jobs. The label does not affect push builds, including pushes to
``main`` and tags, or the release workflow.

To speed up a pull request's documentation build, apply the
``skip-docs-tutorials`` label. The documentation and tutorial pages are still
built, but the tutorials are not executed, so their outputs are omitted.
Removing the label restores tutorial execution. Push builds always execute the
tutorials, regardless of pull request labels.

Changelog entries
-----------------
^^^^^^^^^^^^^^^^^

Every pull request should include a Towncrier fragment describing its user-facing
changes. See ``docs/changes/README.md`` for the naming conventions and supported
categories. If a change does not warrant a changelog entry, maintainers can apply
the ``no-changelog-needed`` pull request label.

Issues
------
^^^^^^

Feel free to submit issues and enhancement requests through our `issues page on GitHub <https://github.com/ctlearn-project/ctlearn/issues>`_.

Copyright and License
---------------------
^^^^^^^^^^^^^^^^^^^^^

CTLearn is licensed under the `BSD 3-Clause License <https://opensource.org/licenses/BSD-3-Clause>`_.

Expand Down
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ CTLearn: Deep Learning for IACT Event Reconstruction
CTLearn is a package under active development to run deep learning models to analyze data from all major current and future arrays of imaging atmospheric Cherenkov telescopes (IACTs). CTLearn can load R1/DL0/DL1 data from `CTAO <https://www.cta-observatory.org/>`_ (Cherenkov Telescope Array Observatory), `FACT <https://www.isdc.unige.ch/fact/>`_\ , `H.E.S.S. <https://www.mpi-hd.mpg.de/hfm/HESS/>`_\ , `LST-1 <https://www.lst1.iac.es/>`_\ , `MAGIC <https://magic.mpp.mpg.de/>`_\ , and `VERITAS <https://veritas.sao.arizona.edu/>`_ telescopes reduced by `ctapipe <https://github.com/cta-observatory/ctapipe>`_ and processed by `DL1DataHandler <https://github.com/cta-observatory/dl1-data-handler>`_.

* Code, feature requests, bug reports, pull requests: https://github.com/ctlearn-project/ctlearn
* Documentation: https://ctlearn.readthedocs.io
* Documentation: https://ctlearn-project.github.io/ctlearn/
* License: BSD-3

Installation
Expand All @@ -53,7 +53,7 @@ The latest version of this package can be installed via pip:

pip install ctlearn

See the documentation for further information like `installation instructions for the IT-cluster <https://ctlearn.readthedocs.io/en/latest/installation.html#install-a-released-version>`_, `installation instructions for developers <https://ctlearn.readthedocs.io/en/latest/installation.html#installing-with-pip-setuptools-from-source-for-development>`_, `package usage <https://ctlearn.readthedocs.io/en/stable/usage.html>`_, and `dependencies <https://ctlearn.readthedocs.io/en/stable/installation.html#dependencies>`_ among other topics.
See the documentation for further information like `installation instructions for the IT-cluster <https://ctlearn-project.github.io/ctlearn/installation.html#install-a-released-version>`_, `installation instructions for developers <https://ctlearn-project.github.io/ctlearn/installation.html#installing-with-pip-setuptools-from-source-for-development>`_, `package usage <https://ctlearn-project.github.io/ctlearn/usage.html>`_, and `dependencies <https://ctlearn-project.github.io/ctlearn/installation.html#dependencies>`_ among other topics.


Citing this software
Expand Down
3 changes: 3 additions & 0 deletions docs/changes/322.maintenance.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Build and publish the documentation with GitHub Actions. The CI documentation
job executes the Sphinx-Gallery tutorial, reuses cached tutorial data, and
deploys the generated HTML to GitHub Pages on pushes to ``main``.
1 change: 1 addition & 0 deletions docs/changes/323.datamodel.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added support for ctapipe HDF5 data format v8.0.0, using the updated DL0 trigger-table paths while preserving compatibility with earlier data formats.
12 changes: 8 additions & 4 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
sphinx
sphinx_rtd_theme
pydata-sphinx-theme
sphinx-changelog
sphinx-design
# Light-weight runtime dependencies needed by autodoc to import ctlearn.
# The heavy ML frameworks (TensorFlow, PyTorch, ...) are mocked in conf.py.
# Dependencies for autodoc and the executable Sphinx-Gallery tutorials.
# The Sphinx autodoc configuration mocks the heavy ML frameworks.
ctapipe>=0.29
dl1_data_handler>=0.14.12
dl1_data_handler>=0.14.13
matplotlib
ctaplot
scikit-learn
sphinx-gallery>=0.16.0
Binary file added docs/source/_static/ctlearn_favicon.ico
Binary file not shown.
Binary file added docs/source/_static/ctlearn_favicon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/ctlearn_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/ctlearn_logo_transparent.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading