This phase I trial studies the side effects and best dose of entinostat and nivolumab when given together with ipilimumab in treating patients with solid tumors that have spread to other places in the body and usually cannot be cured or controlled with treatment (metastatic) or that cannot be removed by surgery (unresectable) or human epidermal growth factor receptor 2 (HER2)-negative breast cancer that has spread from where it started to nearby tissue or lymph nodes or other parts of the body. Entinostat is in a class of drugs called histone deacetylase (HDAC) inhibitors. It may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth (locally advanced/metastatic). Immunotherapy with monoclonal antibodies, such as nivolumab and ipilimumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Giving entinostat and nivolumab together with ipilimumab may work better in treating in patients with solid tumors.