the following works when using checkstyle-5.5:
<path id="checkstyle.classpath">
<pathelement path="path-where-logback.xml-is-located"/>

<taskdef resource="checkstyletask.properties" classpathref="checkstyle.classpath" onerror="failall"/>
<target name="checkstyle">

without putting it in the tasks classpath, but doing:
ant -k -Dlogback.configurationFile=path-where-logback.xml-is-located/logback.xml

seems not to find logback.xml.