This phase I trial studies the side effects and best dose of M5A-IL2 immunocytokine (M5A-ICK) combined with stereotactic body radiation therapy (SBRT) and to see how well they work in treating patients with colorectal cancer or xarcinoembryonic antigen (CEA) positive breast cancer that cannot be removed by surgery (unresectable) or has spread from where it first started (primary site) to other places in the body (metastatic). Carcinoembryonic Antigen (CEA) is a protein that is present in most colorectal cancers and in many other cancers, such as breast cancer, as well. SBRT uses special equipment to position a patient and deliver radiation to tumors with high precision. This method may kill tumor cells with fewer doses over a shorter period and cause less damage to normal tissue. Cytokines are signaling proteins that help control inflammation in the body. They allow the immune system to mount a defense if germs or cancer or other substances that can make people sick enter the body. Interleukin-2 (IL-2) is a powerful cytokine able to regulate the immune responses that are important for anticancer immunity. Immunocytokines (also called antibody-cytokine fusion proteins) are small proteins that regulate the activity of immune cells. The M5A-IL2 immunocytokine (M5A-ICK) combines the cancer targeting features of the M5A antibody with the immune system regulation properties of the cytokine IL-2. Giving M5A-ICK in combination with standard of care (SOC) SBRT may work better in treating patients with unresectable metastatic colorectal cancer or CEA positive metastatic breast cancer.