Leah, you have limited information about this "se", because it has about a dozen different uses, and the one you mention is just one of them, and not a very important one.
"Ir" is used when the focus is on the destination, i.e. to go. You cannot use this verb without a destination, explicit or implicit. If you suddenly say to someone "Voy.", this person will be puzzled, and will probably say something like "Vas... ¿adónde'", wondering why you omitted the destination.
"Irse" is used when the focus is on the place from which we depart, i.e. to leave. If you say "Me voy", the focus of your sentence is on the starting point, which you are about to leave, regardless of your destination, which does not have to be mentioned, as it is secondary (although you can mention it too, of course).
In this case, the "se" changes the focus, but in other verbs it has different functions. Think of "ir" and "irse" as separate verbs, like "go" and "leave" in English.
"Venir" works in a similar way, although unlike "ir(se)", this verb implies by definition that the destination is HERE, so the difference is even subtler. Think of these two sentences: "The cloth is over the board" and "The board is under the floor"; they say pretty much the same, but the focus is different, like in "half full" and "half empty".
"Se" has so many uses, that there are entire books describing them (and only covering part of the whole picture). For some reasons, books only mention the reflexive and the reciprocal uses, even though they are the LEAST common ones in practice.