Organize Astrophotography Data

This is a Ruby gem command line tool to help organize astrophotography data into folders using keywords to assist in grouping and matching subframes for preprocessing and calibration. Subframe organization is compatible with keyword grouping in PixInsight using WeightedBatchPreProcessing.

The tool can quickly rename raw (CR2) files based on EXIF metadata and manual user input, which can then be used to group for preprocessing and calibration.

AstroSubframeOrganizer has been adapted from a standalone Ruby Script to a customizable gem, with more features, customizability, and improved usability and safeguards.

This version of the gem was written to organize or reorganize the data collected to match my current workflow.

• Organize Astrophotography Data
  • Table of Contents
  • Installation
  • Getting Started
    • Initialize your equipment set
    • Configuration
      • Creating a Config File
    • Advanced Configuration
      • Temperature Tolerance
      • Telescope Ignore Patterns
      • Custom Metadata Mappings
      • Custom File Extensions
      • Using a Custom Config File
  • Organization Tools
    • Interactive Menu
    • Global Options
    • Equipment Options
    • Command Mode
      • Darks
        • Flat Darks
      • Flats
        • FlatSet
      • Lights
      • Biases
      • Unorganize
    • Raw Tools
      • Rename From EXIF
      • Revert Rename
    • Cleanup Tools
      • Delete Thumbnails
      • Delete Empty Directories
    • Utils
  • Organized Directory Structure
    • Light Frames
    • Flat Frames
    • Dark Frames
    • Bias Frames
    • Flat Darks (DarkFlats)
  • PixInsight - Saved WBPP Process Icons
    • WBPP_Darks
    • WBPP_Flats
    • WBPP_Integration
  • Historical Development Notes
    • Pre-ASIAir Image Data
      • Exposure Time
      • Renaming Previously Renamed Files
  • Contributing
    • Get Started Coding
  • Disclaimer

Install the gem:

gem install astro-subframe-organizer

Or clone the repository and run:

bundle install
bundle exec rake install

Create a default config file with sample equipment and settings (~/astro-subframe-organizer-config.yml) and edit it to include your equipment:

astro-subframe-organizer init

Or create it with your equipment list directly:

astro-subframe-organizer init --telescope CarbonStar200 --filter NBZ --camera 'ZWO ASI183MC Pro'

Run the organizer:

astro-subframe-organizer run

Follow the interactive prompts to organize your files.

The gem supports customizable lists of telescopes, filters, and cameras. By default, it uses built-in lists, but you can create a personal configuration file.

To create a default configuration file:

astro-subframe-organizer init

This creates ~/astro-subframe-organizer-config.yml with default values. Edit this file to customize your equipment or change configurations:

telescopes:
  - RedCat51
  - ZhumellZ130
  - YourTelescope
filters:
  - BaaderMoon
  - NBZ
  - YourFilter
cameras:
  - T7
  - 183MC
  - ZWO ASI183MC Pro
  - Canon EOS 1500D
  - YourCamera
temperature_tolerance: 5.0

If you only use one set of equipment, you can create and update the config file in one shot:

astro-subframe-organizer init --telescope CarbonStar200 --filter NBZ --camera 'ZWO ASI183MC Pro'

When you run the organizer with only one option in each equipment category, the interactive organizer will automatically select the equipment unless there's a mismatch in the available metadata (FITS headers or EXIF data), so it may be useful to create multiple config files that include your common equipment configurations:

astro-subframe-organizer init --config ~/galaxy-season.yml --telescope CarbonStar200 --filter BaaderMoon --camera 'ZWO ASI533MC Pro'

astro-subframe-organizer init --config ~/nightscape.yml --telescope Rokinon135 --filter BaaderMoon --camera 'Canon T7'

astro-subframe-organizer init --config ~/dual-narrowband.yml --telescope Redcat51 --filter NBZ --camera 'ZWO ASI183MC Pro'

If you shoot with multiple filters, you can add additional filters in the config file:

telescopes:
  - Redcat71
filters:
  - L
  - R
  - G
  - B
cameras:
  - ZWO ASI2600MM Pro
temperature_tolerance: 5.0

Note: if you have an automatic filter wheel and your software records the filter name in FITS headers, ASO will automatically select the assigned filter for organization.

You can further customize how metadata is extracted and how equipment is identified and grouped by adding these optional sections to your configuration file.

By default, all image types that get grouped by CCD-TEMP will be grouped together with the temperature rounded to the nearest 5°C. See the following example table for rounding ranges:

Override temperature_tolerance in your config files to change the rounding.

This is especially useful if you're using CMOS/DSLR or otherwise uncooled cameras, and will help you build a Darks library when you can't control the temperature.

ASIAir and other software often populate the TELESCOP header with the name of your mount rather than your telescope. You can define partial match strings to ignore these values. These patterns are case-insensitive and will force a manual selection or use a CLI override if matched.

telescope_ignore_patterns:
  - Mount
  - EQMod
  - AM5
  - AM3
  - RST-135

If your capture software uses non-standard FITS headers or your DSLR uses unusual EXIF tags, you can map them to the fields the organizer expects. The tool uses a fallback mechanism: it checks the tags in the order they are listed and uses the first one that contains a value.

fits_header_mappings:
  temperature:
    - CCD-TEMP
    - SET-TEMP
    - SENSOR-TEMP
  exposure:
    - EXPOSURE
    - EXPTIME
  target:
    - OBJECT
    - TARGET
exif_tag_mappings:
  temperature:
    - camera_temperature
    - sensor_temperature
  iso:
    - iso
    - base_iso

If you use file extensions not supported by default, you can add them here:

fits_extensions:
  - .fit
  - .fits
  - .fts
raw_extensions:
  - .cr2
  - .cr3
  - .nef
  - .arw
  - .dng

You can specify a custom config file in all subcommands, and that config will be used for the duration of that run session:

astro-subframe-organizer run --config ~/galaxy-season.yml # All interactive menus will use equipment from galaxy-season.yml until you quit

There are several ways to use astro-subframe-organizer.

To run the interactive organizer:

astro-subframe-organizer run # aliases: -i, --interactive

This will present you with a series of interactive menus to walk you through organizating each subframe type. Each subframe type has its own set of required inputs, which it will try to gather from FITS headers or filename patterns. If the information can't be detected automatically, the you will be prompted to choose, typically from the equipment in your configuration file.

$ astro-subframe-organizer run --dry-run
Using config file at /Users/joshkovach/.astro_subframe_organizer.yml
What are we organizing? (Dry Run)
  1) Darks
  2) Flats
  3) Lights
  4) Biases
  5) Remove empty directories
  6) Remove jpg thumbnails
  7) Rename files from EXIF data
  8) Quit
  Choose 1-8 [1]: 

Selecting a given tool will lead you through an interactive set of prompts and confirmation steps to help organize your data, for example:

What are we organizing? (Dry Run) Lights
Using config file at ~/.astro_subframe_organizer.yml
Preparing to move 330 Light files from 14 groups...
Preparing to move 1 Light file(s) 
  FROM spec/fixtures/cr2 
  TO   spec/fixtures/cr2/Light_Aurora_FLATSET_20210418_ISO_6400_EXP_4.0s_Bin_1_CCD-TEMP_20._TELESCOPE_????_FILTER_????_CAMERA_????
Continue? (Y/n) 

Note the ???? in the target path name – these are the fields that were not able to be detected automatically, and you will be prompted to select from a series of options to pick the missing fields from your configured equipment:

For Light set Light_Aurora_4.0s_Bin1_ISO6400_20210418-025606_20.0C_0441.CR2..Light_Aurora_4.0s_Bin1_ISO6400_20210418-025606_20.0C_0441.CR2:
⚠ Telescope auto-detect failed.
What telescope is this set for? 
  1) RedCat51
  2) ZhumellZ130
  3) AperturaAD8
  4) MeadeDS90
  5) CanonEFS1855
  Choose 1-5 [1]: 

Selected Telescope: RedCat51

⚠ Filter auto-detect failed.
What filter is used with this set? 
  1) BaaderMoon
  2) NBZ
  3) NoFilter
  Choose 1-3 [1]: 

Selected Filter: NBZ

⚠ Camera auto-detect failed.
What camera is used with this set? 
  1) T7
  2) 183MC
  Choose 1-2 [1]: 

Selected Camera: 183MC

mv /Users/joshkovach/astrophotography/ruby-scripts/astro-subframe-organizer/spec/fixtures/cr2/Light_Aurora_4.0s_Bin1_ISO6400_20210418-025606_20.0C_0441.CR2 /Users/joshkovach/astrophotography/ruby-scripts/astro-subframe-organizer/spec/fixtures/cr2/Light_Aurora_FLATSET_20210418_ISO_6400_EXP_4.0s_Bin_1_CCD-TEMP_20._TELESCOPE_RedCat51_FILTER_NBZ_CAMERA_183MC/Light_Aurora_4.0s_Bin1_ISO6400_20210418-025606_20.0C_0441.CR2
Done

Preparing to move 62 Light file(s) 
  FROM spec/fixtures/fits/light-blanks 
  TO   spec/fixtures/fits/light-blanks/Light_C 1_FLATSET_20260411_ROTATION_108deg_GAIN_111_EXP_300.0s_Bin_1_TELESCOPE_EQMod Mount_FILTER_????_CAMERA_ZWO ASI183MC Pro
Continue? (Y/n) 

For each new set of files (split by index number overflow or when detected equipment changes) within the target directory, you'll be guided through the same set of prompts until all files have been organized into descriptive, keyword-based directories.

--dry-run

Great way to start experimenting with the tool. It will display all the mv commands that would happen, but without actually moving anything. Experiment without any risk, and to get a feel for how it will work with your files.

--skip-confirm

After you've become comfortable with how the tools work, you can pass override options to any command to make your workflow quicker. Passing the --skip-confirm option to any command will skip the Continue? prompts, and just assume that you know what you're doing. Especially helpful when you're using command mode, where you can skip all interactions completely.

$ astro-subframe-organizer run --dry-run --skip-confirm
Using config file at /Users/joshkovach/.astro_subframe_organizer.yml
What are we organizing? (Dry Run) Lights
⚠ Unprocessed raw images detected, but will be ignored. Raw images must be renamed before organizing. Run `astro-subframe-organizer raw rename_from_exif`, then try again.
Using config file at ~/.astro_subframe_organizer.yml
Preparing to move 330 Light files from 14 groups...
For Light set Light_Aurora_4.0s_Bin1_ISO6400_20210418-025606_20.0C_0441.CR2..Light_Aurora_4.0s_Bin1_ISO6400_20210418-025606_20.0C_0441.CR2:
⚠ Telescope auto-detect failed.
What telescope is this set for? 
  1) RedCat51
  2) ZhumellZ130
  3) AperturaAD8
  4) MeadeDS90
  5) CanonEFS1855
  Choose 1-5 [1]: 

--verbose

Prints all logs, including debug messages. Every file moved will display its mv command.

--path

By default, ASO will operate on the working directory (where the command is called from), and works on all sub-directories. Alternatively, you can specify a --path to operate on. Again, this will run recursively by default.

astro-subframe-organizer light --path staged/Plan/Lights/NGC\ 2264/

The above command will organize only the light files in the NGC 2264 directory.

--telescope

Automatically sets the telescope field for any command that uses it. When using --telescope, it will override the TELESCOP FITS header, and does not need to be one of the telescopes in your config file. If the auto-detected telescope is different from the --telescope value, a warning will be logged (e.g., Using telescope RedCat51, but detected EQMod Mount), but the move will continue using your override.

--camera

Automatically sets the camera field for any command that uses it. When using --camera, it will override the INSTRUME FITS header. If the auto-detected camera differs from your override, a warning will be logged, but the move will continue.

--filter

Automatically sets the filter field for any command that uses it. When using --filter, it will override the FILTER FITS header. If the auto-detected filter differs from your override, a warning will be logged, but the move will continue.

If you want to skip the interactive menu, or you know what you're doing without it, you can directly call each of the available commands (and a few more) using the commands directly.

Darks will be grouped in a folder by CCD-TEMP (rounded to nearest 5°C), ISO or GAIN, EXP, CAMERA and MONTH (e.g. 2022-06).

# run without interactive steps
astro-subframe-organizer darks --camera 'ZWO ASI183MC Pro' --skip-confirm

The darks command will also check for possible flat darks, and will ask for confirmation when the exposure time is 10.0s or less. In the case of a flat dark set, it will organize them into a folder with ISO or GAIN, EXP, and FLATSET (the date of the darks and flats). These don't take CCD-TEMP into account, assuming the flats and flat darks are taken at roughly the same time and under the same conditions. 1°C variation is not going to make enough of a difference for me to care, and I only want one master flat from this FLATSET.

These files will be grouped into a folder prefixed with DarkFlat instead of just Dark. When moving off the ASIAir, I put this folder inside a FLATSET_<date> folder with the accompanying flats folder so I can load all the necessary files by directory in one shot in WBPP.

Flats need to be taken with the same image train conditions as your lights, which typically includes:

• ROTATION
• ISO or GAIN
• EXP
• TELESCOPE
• FILTER
• CAMERA

Ideally, they are taken during or right after your session, so that any variation in the image train (dust, focus, ice, etc.) don't have a chance to change before you capture the frames that are meant to remove those variations.

In order to match lights, flats will be grouped using the FLATSET keyword along with the keywords listed above.

# run without interactive steps
astro-subframe-organizer flats --telescope Redcat51 --filter NBZ --camera 'ZWO ASI183MC Pro' --skip-confirm

All of the aforementioned keywords are used in WBPP when doing the lights calibration and integration.

The FLATSET value is just the date in format YYYYMMdd. Because I always take my flats in the morning after my session, the FLATSET for lights is calculated as the end date of the session. Any subframe taken after noon will use the FLATSET of the following day, and anything before noon will use the same date.

In the rare event that I can't take flats for a session, as long as I didn't change the image train significantly, I can usually still calibrate with a FLATSET taken on another date close to the missed session.

Similar to flats, organizing lights will prompt to select the telescope and filter used for this data set, and will organize into a folder with most of the same keywords as the flats.

# run without interactive steps
astro-subframe-organizer lights --telescope Redcat51 --filter BaaderMoon --camera 'ZWO ASI183MC Pro' --skip-confirm

Biases are taken at the fastest shutter speed available to your camera to remove the latent electrical noise on the camera sensor. When using flats without flat-darks, the flats and lights both need to be calibrated with bias frames. These frames are grouped primarily by EXP, CAMERA, and MONTH (to help match appropriate darks).

# run without interactive steps
astro-subframe-organizer biases --camera 'ZWO ASI183MC Pro' --skip-confirm

If you find that the organization didn't go as you hoped, you can easily ungroup those subframes, which just takes them out of their grouped directory, and removes the empty directory after the move.

# run without interactive steps
astro-subframe-organizer unorganize --skip-confirm

If you're shooting RAW without any software to wrap them with FITS headers, you'll end up getting an image named something like IMG_00001.CR2 (that's what my Canon T7 does, anyway). This doesn't really help you be an organized astrophotographer, so I built a tool to help!

To use this tool uses exiftool for working with RAW files. For Mac and Linux users, the tool is included with the gem, but Windows users will need to install it separately.

Windows

Follow instructions at https://exiftool.org/, or install using Chocolatey on Windows 10. I don't use Windows, so if anyone uses it and wants to update this to be more specific, please submit a pull request.

choco install exiftool

OSX

brew install exiftool

Ubuntu Linux

sudo apt install exiftool 

I first wrote this tool after I had started using an ASIAir with my Canon T7, and my original organization script used a filename parser to capture metadata and group accordingly. So for historical reasons, this tool renames RAW files to match the naming scheme used by ASIAir, which allowed me to group data from before and after getting my ASIAir.

Use raw rename to rename IMG_****.CR2 files to something helpful based on the embedded EXIF data and your own knowledge about what's in those photos.

# run without interactive steps
astro-subframe-organizer raw rename --type light --target 'NGC 2264' --telescope Redcat51 --filter BaaderMoon --skip-confirm
astro-subframe-organizer raw rename --type flat  --target 'NGC 2264' --telescope Redcat51 --filter BaaderMoon --skip-confirm
astro-subframe-organizer raw rename --type dark --skip-confirm
astro-subframe-organizer raw rename --type bias --skip-confirm

You should do this on files that are already grouped in separate directories to help keep them separate. There's currently no auto-detection set up for the type of subframe you're organizing, so you have to specify it. If you leave off any of the options, you will be prompted to fill in or select the required values for the subframe type.

The raw rename tool will only work on images that match the prefix IMG_, so if you messed up the rename on the first pass, you can revert them and have another go.

# run without interactive steps
astro-subframe-organizer raw revert --skip-confirm

This will rename each image back to IMG_****.CR2 as Canon intended.

ASIAir, and maybe other capturing software, includes a thumbnail for each FITS files so that it can display preview images on your phone. Outside of that context, they are a nuisance and are completely unnecessary. This tool will delete all those annoying jpgs in a snap!

# run without interactive steps
astro-subframe-organizer cleanup thumbnails --skip-confirm

It can't be undone, but why would you want to?

After reorganizing my photos and moving them to a better place, I'm often left with a bunch of deeply nested empty directories, with some not-so-empty ones mixed in, and I'd rather not have to dig through each one to figure out what to delete. So I made a tool for that!

astro-subframe-organizer cleanup empty

This will remove any directory that completely empty, recursively. Mac users may be familiar with the infamous hidden .DS_Store file - I don't count that as content. If the directory contains only .DS_Store, it still counts as empty. You can run this at the top-level of wherever you're working to make it easier to find what matters.

If you want to see FITS headers or EXIF data that's on your images, use inspect:

astro-subframe-organizer inspect dark/IMG_0001.CR2

This will print out all the metadata on your RAW images, or all the FITS headers on a .fit image.

The organizer moves files into specific directory structures based on the image type. These structures are designed to be compatible with PixInsight's WeightedBatchPreProcessing (WBPP) script by using standard keywords.

Organized to match with flats from a specific session and darks of the same exposure/gain/temp. Light_<target>_FLATSET_<date>_GAIN_<gain>_EXP_<exp>_Bin_<bin>_CCD-TEMP_<temp>_TELESCOPE_<tel>_FILTER_<fil>_CAMERA_<cam>

Organized to match with Light frames from the same session. Flat_FLATSET_<date>_GAIN_<gain>_EXP_<exp>_Bin_<bin>_TELESCOPE_<tel>_FILTER_<fil>_CAMERA_<cam>

Organized to facilitate a library of master darks. Dark_ISO_<iso>_EXP_<exp>_CCD-TEMP_<temp>_CAMERA_<cam>_MONTH_<month>

Bias_GAIN_<gain>_EXP_<exp>_Bin_<bin>_CAMERA_<cam>_MONTH_<month>

Used specifically to calibrate flats when bias frames aren't used. These are especially useful if you are using an uncooled camera, and you take your flat darks at the same time as your flats. DarkFlat_FLATSET_<date>_ISO_<iso>_EXP_<exp>_Bin_<bin>_CAMERA_<cam>

All of this organization is to facilitate a standardized workflow in PixInsight using WBPP with predefined process icons for generating each of master darks, master flats, and the master lights.

This process icon is preloaded with appropriate master biases, uses the following grouping keywords on the Calibration tab:

The generated darks are then able to be used in the WBPP_Integration process icon.

This process icon is also preloaded with appropriate master biases, and uses the following grouping keywords on the Calibration tab:

Since my flats and darkflats are together in the same directory, I can load them into WBPP using the Directory button in one step, and then click the run button.

This process icon is preloaded with appropriate biases, but shouldn't be necessary at this point if you followed the process described so far. This step uses the following grouping keywords on the Calibration tab:

If you are working on multiple targets, you'll want to make sure you choose the Registration Reference Image -> Mode -> auto by LIGHT (or PANE) setting under the Calibration tab.

This final step is relatively easy. Simply load your master darks (not darkflats), your master flats, and all your lights. The script should automatically detect and group all the files for calibration. You may have to manually select a few of the darks and flats for calibration if you reuse the same FLATSET for multiple nights. Other than that, just check your other settings and output directory for this run and you should be good to go.

This was based on a script written a long time ago, so here's some historical information that may be relevant for new astrophotographers.

With the Canon T7 data captured before I started using an ASIAir Plus, the images were captured in RAW format as CR2 files, with the name IMG_0001.CR2, which is pretty useless for AP photo organization. However, I've found that these RAW files do include most of the necessary data in EXIF tags to allow renaming to match my newer data generated by the ASIAir Plus, including exposure time, camera temperature, iso, etc. Using exiftool, these files can be renamed (with some extra parsing work in Ruby and some user input) to match the same file name pattern with actual data from the original source files.

To rename your older IMG_XXXX.CR2 files, you can use the Rename files with EXIF data option. You will then be prompted to choose which type of file you are organizing. If you are organizing a Light file, you'll also be prompted to enter the target name.

When the EXIF data includes ExposureTime less than 1 second, the value is formatted as a fraction, e.g. 1/250, which then gets interpreted by most file systems as a directory separator. In order to handle this appropriately to match the decimal exposure formatting that the ASIAir generates, we need to do a few workarounds. First, we need to replace the / character with - so that the files don't get misplaced in a new directory. Second, we need to take the file that exiftool generates and parse it to recalculate that fraction value as a decimal at an appropriate time scale. So we parse the 1-250, convert that to a Rational in Ruby, Rational(1, 250), and then change the scale from seconds to milliseconds to nanoseconds until we have the exposure time represented as a number equal to or greater than 1.0.

This operation also lets you rename files that you renamed with an older naming format and convert it automatically to use the consistent naming pattern. If the script finds files that are not named IMG_XXXX.CR2, it will prompt you to choose whether to skip or rename them. It will then rename them all to IMG_XXXX.CR2, where XXXX is the last 4 characters of the filename (usually the sequence number). It will then run the script as normal on the now normallized files.

Once all of the files are renamed, they can then be organized into folders just as we do with the FITS files that we get from the ASIAir.

This is a tool I've been using for several years now, and I'm excited to get it into a place I feel comfortable sharing it with other people to use. That said, it's alpha and has only been used by me. If you use this and have any problems with it, or would like better support for your equipment or workflows, please drop a friendly issue, and we can discuss it. If you have ideas for improvement, I'm also happy to consider pull requests.

Contributions are welcome from astrophotographers and developers alike — you don't need to write code to help. You can test the organizer with your own equipment and report what works or doesn't, validate the organization structure for your stacking software workflow, improve documentation, or help other users.

See CONTRIBUTING.md for full details, including development setup, fixture file guidelines, and pull request expectations.

This project follows the Contributor Covenant Code of Conduct.

git clone https://github.com/your-username/astro-subframe-organizer
cd astro-subframe-organizer
bin/setup
bundle exec rake spec

The test suite uses real stripped FITS fixture files. If you're working on a bug that requires a new fixture, strip it first to remove image data and location headers:

bundle exec bin/strip_fits -o spec/fixtures/fits/your-set/ your_file.fit

I make no claims about the reliability of Astro Subframe Organizer under your circumstances. Please test and verify the code and the conditions you will run this script through before running it on your data. Dry runs are your friend. While Astro Subframe Organizer is relatively well tested, it may behave unexpectedly if your data capture is significantly different from mine. I am not responsible for lost or corrupted data or damaged devices resulting from the use of this Astro Subframe Organizer. Just be careful, make backups, test things out before you fully rely on any part of this tool. If you find an issue while using this tool, please report it on the issue tracker.

Clear skies!