Art? Art Brockway? Are you out there Art? I will always think of you with this passage. When Art and I were grad students at the University of Miami, he wrote what I'm sure is the definitive article on Joyce and mummery. There is little else to say, or rather, there would be little else to say were your correspondent not writing from Philadelphia, mummery capital of the world.
(image from gophila.com, taken by R. Kennedy)
Again, I'm hoping Art will chip in at some point, but on the most basic level, when Mulligan calls Stephen a "mummer," he's saying that he's disguised, he's pretending to be something he's not.
The tradition of mumming came to Philadelphia from many places, but the strongest thread runs from Ireland & the other Celtic countries. By tradition, around the holidays, a gang of costumed men would go from house to house and basically trick or treat for booze. There might be a play or a performance involved, but there's a costume and some kind of entertainment and probably "something sinister" in having them come into your home... as Mulligan suggests.
And as for sinister... here's another question for the masses. Was Joyce left-handed? Stephen, based on a number of references in this book, seems to be a leftie. And Joyce's corresponding figure in Finnegans Wake, Shem the Penman, is left-handed. Of course, even if Joyce were left-inclined, no school in Ireland would have let him actually write that way...