Currently the pam_oath module doesn't work with SELinux out of the box, because it creates lock file when updating usersfile. The problem is that it creates the lock file in the same directory the usersfile is located and it's mostly not allowed by SELinux rules to create new files by pam modules.

It seems it is not possible to remove the external lock file and use only advisory locking on usersfile, because it will introduce race condition.

So I tried to extend the liboath API by oath_set_lockfile_path call which sets the lockfile location for all successive API calls. If not used or the lockfile path is set to NULL, previous behaviour (i.e. no global lock, only local usersfile lock) is used. I also extended pam module to use this new API call and create its global lock as: /var/lock/pam_oath.lock. This should resolve the SELinux problem. I think using one global lock for pam module shouldn't be performance bottleneck in most cases, but for cases where it is, I also added lockfile pam module parameter, so arbitrary usersfile/lockfiles (without one global lock) can be also used.

Attached patch is proof of concept, feel free to change/rework it as needed.

There is Fedora bug report about this problem:
https://bugzilla.redhat.com/show_bug.cgi?id=1178036