It could be interpreted as either "I" or "words" to be with the world, depending whether the "words" is the object to be left for or the object to leave from. :)

"I said nothing but those words, to be passed on in this world" would make this sturcture clearer, but less poetic.