This has happened to me a few times. One I remember was the game Alchemist, where I just sat there confused as hell for 4 and a half hours while three guys were all talking about strategies. Tonight it was Terraforming Mars, where I was told it would be a 3 hour game, but by hour 4 we were halfway done. This time I said "it's 11pm, I have work in the morning, this will be my last hand" and the host got very passive aggressive with me. I just don't know what to do in these situations.
^Also is there a word for this? My girlfriend said I was "held game hostage" but I don't see that used in my searches.^
Update:
I sent an apology for leaving early, and he wasn't too frustrated about it and understood my frustration which was nice. I told him I didn't think it was my cup of tea since it was so dense, but he kept trying to sell me on the game.
I just gotta learn how to decline with this guy, he is a bit of a "won't take no for an answer" person, but I'm still learning to be firm with boundaries.
I'm really a 45 minute or less person, and prefer games with like... 5 rules. I have communicated that before, but he really wants me to play the games he loves which I take as a compliment.
He did have me playing Dominion for a while, and that was a time when I just would suck it up and play for his sake since he was going through a divorce. We literally had the parks and rec sketch where I said "I don't really like Dominion" and he said "what do you mean? You've played all the games!"
He housed me when I was homeless, so it's hard for me to decline things with him since he showed me that huge kindness.
This is why I always do research on the game before playing it. Optimally, I'll download and read the rulebook, but "let's play" videos work in a pinch. (Or if the rulebook is a bit obtuse.)
Of course, if you don't know what you're playing beforehand, then it's up to the host(s) to teach you, and it all depends upon how good they are at teaching (and how good you are at paying attention). These situations are obviously much more dodgy, so I try to at least peruse the rulebook a bit and keep it nearby during the game so that I can look up rules during other players' turns.
Finally, it does get easier. It does take time and energy to learn these more complex games, but the payoff is more interesting and satisfying game experiences. Every time you play, you learn a little more. Of course, if you aren't getting any enjoyment out of it at all, you might want to stick with lighter games.