The ambiguity you point to to, in terms of passive and active agency is taken one step further in the case of "factor" - while we now most commonly refer to "factors" a mere parts of an argument (this factor - that factor) the terms also means a person who brings things about - my factor will exchange contracts for me. ( by extension - a fact is a brought-about or finished action - which then comes up in per-fect = a finished thing and factory etc.). So we do have a deep ambiguity whereby a factor can both be a person and a thing.
The issue of "agency" has two senses. One sense involves those entities
that act in the world regardless of ontological status, whether
independent or dependent, whether acting on their own desires or serving
the desires of another actor. The other sense involves the agency of
those creatures that act as independent agents. The term "agent" can
take either meaning, and -- in fact -- the very word and concept have
that ambiguous nature. I possess agency as an independent decision
maker. I can also serve as the agent of another person who delegates
responsibility or authority to me as his or her agent.