This phase I/II trial studies the side effects and best dose of ipilimumab and nivolumab when given together with brentuximab vedotin, and how well they work in treating patients with Hodgkin lymphoma that has returned after a period of improvement (recurrent) or has not responded to previous treatment (refractory). Immunotherapy with monoclonal antibodies, such as ipilimumab and nivolumab, may help the body's immune system attack the cancer, and may interfere with the ability of cancer cells to grow and spread. Brentuximab vedotin is a monoclonal antibody, brentuximab, linked to a toxic agent called vedotin. Brentuximab attaches to CD30 positive cancer cells in a targeted way and delivers vedotin to kill them. It is not known whether giving brentuximab vedotin and nivolumab with or without ipilimumab may kill more cancer cells.