This phase II trial studies how well pembrolizumab works when given together with endocrine therapy and palbociclib in treating postmenopausal patients with newly diagnosed stage IV estrogen receptor positive breast cancer that has spread to other parts of the body (metastatic). Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Estrogen can cause the growth of breast cancer cells. Fulvestrant blocks the use of estrogen by the tumor cells. Letrozole lowers the amount of estrogen made by the body. This may help stop the growth of tumor cells that need estrogen to grow. Palbociclib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving pembrolizumab, palbociclib, and letrozole or fulvestrant may be an effective treatment for patients with stage IV estrogen receptor positive breast cancer.