Even just adding one word "Then Ulbricht had walked into the public library and sat down at the table directly in front of me" would be enough of a clue.
If you read it one way, it's almost impossible to not be misdirected, because the following sentence works with both meanings.
If you include the had this would be enough of a clue to correct the incorrect assumption. Although it still might make for slightly bumpy reading.