Add documentation for integrators

Relates: #14
This commit is contained in:
Daniel Siepmann 2020-07-29 15:09:13 +02:00
parent 33685d1735
commit f48714cb7b
21 changed files with 672 additions and 28 deletions

View file

Before

Width:  |  Height:  |  Size: 138 KiB

After

Width:  |  Height:  |  Size: 138 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 173 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 62 KiB

After

Width:  |  Height:  |  Size: 132 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

92
Documentation/Index.rst Normal file
View file

@ -0,0 +1,92 @@
.. _start:
==================
Tracking Extension
==================
A very simple server side tracking extension.
Initially developed for demonstration purposes.
Soon requested by customers in order to have privacy focused minimal tracking inside of TYPO3.
The extension provides built in tracking.
All requests are tracked right within TYPO3 as custom records.
The extension also delivers widgets for EXT:dashboard to visualize tracked information.
.. _goal:
Goal
----
This extension only provides very basic features and is not intended to resolve complex solutions like Google Analytics.
Yet it should provide the very minimum requirements to remove the need of such complex solutions on small websites.
.. figure:: /Images/Widgets.png
:align: center
Figure 1-1: Screenshot of how widgets might look, representing tracked information.
.. figure:: /Images/ListViewPageviews.png
:align: center
Figure 1-2: Screenshot of TYPO3 list view, showing records of tracked data.
Integrators should be able to configure more or less everything.
From collection to displaying via widgets.
Features
--------
The extension allows to track :ref:`pageview`,
as well as views to specific TYPO3 records via :ref:`recordview`,
e.g. records from EXT:news or EXT:tt_address.
Missing features
----------------
Features that will not make it:
The extension does not limit reporting based on user access.
Foreign systems like Google Analytics also wouldn't know which pages or records an editor has access to.
Features that might be implemented in the future:
* Remove tracked records based on adjusted rule.
Right now results tracked will be kept.
If rules are adjusted, e.g. another bot is excluded, old entries will be kept.
In the future there might be an command that will reduce existing records based on current rules.
* Collecting information about referrers.
* Collecting information based on Events.
* Collecting information about URL parameters like campaigns or utm.
* Does not extract device types out of User Agents.
* Does not extract version of operating system.
* Has a very rough bot detection.
Differences to typical tracking solutions
-----------------------------------------
This extension does not need any JavaScript or Cookies.
Tracking is happening on server side.
Therefore Client caching can prevent "requests" from being tracked.
But that can be seen as an "unique visitor" feature.
Also information like "how long did the user visit the page" are not available.
Therefore no data is passed to any 3rd Party or kept and made available.
Only internal information of TYPO3, such as Page or records, are tracked.
The only foreign information being tracked is the User Agent and URL,
in order to extract further information from them with future updates.
.. toctree::
:hidden:
Installation
Pageview
Recordview

View file

@ -0,0 +1,20 @@
.. highlight:: bash
.. _installation:
============
Installation
============
Install the extension as usual via composer or some other way::
composer require danielsiepmann/tracking
There is no TypoScript included which needs to be included.
Instead further configuration via :file:`Services.yaml` is necessary.
The extension highly depends on the dependency injection feature introduced with TYPO3 v10.
Check corresponding sections about :ref:`pageview` and :ref:`recordview`.
The extension should work out of the box,
but should be configured to the specific installation.

View file

@ -0,0 +1,86 @@
.. highlight:: yaml
.. _pageview:
========
Pageview
========
Each view of a TYPO3 page is tracked by default.
Requests can be ignored by configuring a rule that has to match the current request.
All configuration happens via :ref:`t3coreapi:DependencyInjection` inside of :file:`Services.yaml` of your Sitepackage.
.. figure:: /Images/ListViewPageviews.png
:align: center
Screenshot of list view of created "pageview" records.
.. figure:: /Images/RecordPageview.png
:align: center
Screenshot of edit form view of created "pageview" records.
Saved record
------------
Whenever a pageview is tracked, a new record is created.
The record can be viewed via TYPO3 list module. That way all collected information can be checked.
Configure tracking
------------------
Let us examine an concrete example::
services:
_defaults:
autowire: true
autoconfigure: true
public: false
DanielSiepmann\Tracking\Middleware\Pageview:
public: true
arguments:
$rule: >
not (context.getAspect("backend.user").isLoggedIn())
and not (request.getHeader("User-Agent")[0] matches "/^TYPO3|TYPO3 linkvalidator/")
and not (request.getHeader("User-Agent")[0] matches "/^Codeception Testing/")
and not (request.getHeader("User-Agent")[0] matches "/Wget|curl|Go-http-client/")
and not (request.getHeader("User-Agent")[0] matches "/bot|spider|Slurp|Sogou|NextCloud-News|Feedly|XING FeedReader|SEOkicks|Seekport Crawler|ia_archiver|TrendsmapResolver|Nuzzel/")
and not (request.getHeader("User-Agent")[0] matches "/mattermost|Slackbot|WhatsApp/")
The first paragraph will not be explained, check out :ref:`t3coreapi:configure-dependency-injection-in-extensions` instead.
The second paragraph is where the tracking is configured.
The PHP class ``DanielSiepmann\Tracking\Middleware\Pageview`` is registered as PHP middleware and will actually track the request.
Therefore this class is configured.
The only interesting argument to configure is ``$rule``,
which is a `Symfony Expression <https://symfony.com/doc/current/components/expression_language/syntax.html>`__.
The same is used by TYPO3 for TypoScript conditions and is not explained here.
This rule is evaluated to either ``true`` or ``false``,
where ``true`` means that the current request should be tracked.
The current request is available as ``Psr\Http\Message\ServerRequestInterface`` via ``request``,
while ``TYPO3\CMS\Core\Context\Context`` is available via ``context``.
That way it is possible to check all kind of information like frontend user, backend user or cookies and parameters,
as well as request header.
Check `PSR-7: HTTP message interfaces <https://www.php-fig.org/psr/psr-7/#321-psrhttpmessageserverrequestinterface>`__
as well as
:ref:`t3coreapi:context-api`.
The above example blocks tracking for requests with logged in backend user,
as well as specific user agents like bots, TYPO3 itself and other systems.
Widgets
-------
The extension does not provide any widgets, but providers for widgets of EXT:dashboard.
That way widgets of EXT:dashboard can be combined with all providers of this extension.
The concepts are not documented here, check :ref:`t3dashboard:start` instead.
.. toctree::
:glob:
PageviewWidgets/*

View file

@ -0,0 +1,58 @@
.. php:namespace:: DanielSiepmann\Tracking\Dashboard\Provider
.. program:: DanielSiepmann\Tracking\Dashboard\Provider\NewestPageviews
.. _newestpageviews:
===============
NewestPageviews
===============
Provides a list of the newest pageview entries.
Example
=======
.. figure:: /Images/Widgets/NewestPageviews.png
:align: center
Default widget configuration.
:file:`Configuration/Services.yaml`::
services:
DanielSiepmann\Tracking\Dashboard\Provider\NewestPageviews:
arguments:
$queryBuilder: '@querybuilder.tx_tracking_pageview'
$blackListedPages: [1, 11, 38]
dashboard.widget.danielsiepmann.tracking.newestPageviews:
class: 'TYPO3\CMS\Dashboard\Widgets\ListWidget'
arguments:
$view: '@dashboard.views.widget'
$dataProvider: '@DanielSiepmann\Tracking\Dashboard\Provider\NewestPageviews'
tags:
- name: 'dashboard.widget'
identifier: 'newestPageviewsList'
groupNames: 'tracking'
iconIdentifier: 'content-widget-list'
title: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.newestPageviewsList.title'
description: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.newestPageviewsList.description'
height: 'medium'
width: 'small'
Options
=======
.. option:: $maxResults
Integer defining how many results should be displayed.
Defaults to 6.
.. option:: $blackListedPages
Array of page uids that should not be collected.
Defaults to empty array, all pages are shown.
This becomes handy if certain pages are called in order to show specific records.
In those cases the pages will be called very often but don't provide much benefit and can be excluded.
Use this in combination with :ref:`recordview` to show the records instead.

View file

@ -0,0 +1,67 @@
.. php:namespace:: DanielSiepmann\Tracking\Dashboard\Provider
.. program:: DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerDay
.. _pageviewsperday:
===============
PageviewsPerDay
===============
Provides the total page calls on the last x days.
This way editors can see how many total requests were made at specific dates.
Example
=======
.. figure:: /Images/Widgets/PageviewsPerDay.png
:align: center
Default widget configuration.
:file:`Configuration/Services.yaml`::
services:
DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerDay:
arguments:
$queryBuilder: '@querybuilder.tx_tracking_pageview'
$blackListedPages: [1, 11, 38]
dashboard.widget.danielsiepmann.tracking.pageViewsPerDay:
class: 'TYPO3\CMS\Dashboard\Widgets\BarChartWidget'
arguments:
$view: '@dashboard.views.widget'
$dataProvider: '@DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerDay'
tags:
- name: 'dashboard.widget'
identifier: 'pageViewsBar'
groupNames: 'tracking'
iconIdentifier: 'content-widget-chart-bar'
title: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.pageViewsBar.title'
description: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.pageViewsBar.description'
additionalCssClasses: 'dashboard-item--chart'
height: 'medium'
width: 'small'
Options
=======
.. option:: $days
Integer defining the number of days to respect.
Defaults to 31.
.. option:: $blackListedPages
Array of page uids that should not be collected.
Defaults to empty array, all pages are shown.
This becomes handy if certain pages are called in order to show specific records.
In those cases the pages will be called very often but don't provide much benefit and can be excluded.
Use this in combination with :ref:`recordview` to show the records instead.
.. option:: $dateFormat
String defining the format used for labels.
Defaults to 'Y-m-d'.

View file

@ -0,0 +1,59 @@
.. php:namespace:: DanielSiepmann\Tracking\Dashboard\Provider
.. program:: DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerOperatingSystem
.. _pageviewsperoperatingsystem:
===========================
PageviewsPerOperatingSystem
===========================
Provides the total calls on a operating system level.
This way editors can see which operating systems most visitors use.
Example
=======
.. figure:: /Images/Widgets/PageviewsPerOperatingSystem.png
:align: center
Default widget configuration.
:file:`Configuration/Services.yaml`::
services:
DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerOperatingSystem:
arguments:
$queryBuilder: '@querybuilder.tx_tracking_pageview'
$days: 62
dashboard.widget.danielsiepmann.tracking.operatingSystems:
class: 'TYPO3\CMS\Dashboard\Widgets\DoughnutChartWidget'
arguments:
$view: '@dashboard.views.widget'
$dataProvider: '@DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerOperatingSystem'
tags:
- name: 'dashboard.widget'
identifier: 'operatingSystemsDoughnut'
groupNames: 'tracking'
iconIdentifier: 'content-widget-chart-pie'
title: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.operatingSystemsDoughnut.title'
description: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.operatingSystemsDoughnut.description'
additionalCssClasses: 'dashboard-item--chart'
height: 'medium'
width: 'small'
Options
=======
.. option:: $days
Integer defining the number of days to respect.
Defaults to 31.
.. option:: $maxResults
Integer defining how many pages should be shown.
Defaults to 6 because EXT:dashboard only provides 6 colors.
Defaults to 6.

View file

@ -0,0 +1,68 @@
.. php:namespace:: DanielSiepmann\Tracking\Dashboard\Provider
.. program:: DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerPage
.. _pageviewsperpage:
================
PageviewsPerPage
================
Provides the total calls on a per page level.
This way editors can see which pages were requested the most during a specified period.
Example
=======
.. figure:: /Images/Widgets/PageviewsPerPage.png
:align: center
Default widget configuration.
:file:`Configuration/Services.yaml`::
services:
DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerPage:
arguments:
$queryBuilder: '@querybuilder.tx_tracking_pageview'
$blackListedPages: [1, 11, 38]
dashboard.widget.danielsiepmann.tracking.pageViewsPerPage:
class: 'TYPO3\CMS\Dashboard\Widgets\DoughnutChartWidget'
arguments:
$view: '@dashboard.views.widget'
$dataProvider: '@DanielSiepmann\Tracking\Dashboard\Provider\PageviewsPerPage'
tags:
- name: 'dashboard.widget'
identifier: 'pageViewsPerPageDoughnut'
groupNames: 'tracking'
iconIdentifier: 'content-widget-chart-bar'
title: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.pageViewsPerPageDoughnut.title'
description: 'LLL:EXT:tracking/Resources/Private/Language/locallang.xlf:dashboard.widgets.pageViewsPerPageDoughnut.description'
additionalCssClasses: 'dashboard-item--chart'
height: 'medium'
width: 'small'
Options
=======
.. option:: $days
Integer defining the number of days to respect.
Defaults to 31.
.. option:: $maxResults
Integer defining how many pages should be shown.
Defaults to 6 because EXT:dashboard only provides 6 colors.
Defaults to 6.
.. option:: $blackListedPages
Array of page uids that should not be collected.
Defaults to empty array, all pages are shown.
This becomes handy if certain pages are called in order to show specific records.
In those cases the pages will be called very often but don't provide much benefit and can be excluded.
Use this in combination with :ref:`recordview` to show the records instead.

View file

@ -0,0 +1,100 @@
.. _recordview:
==========
Recordview
==========
Many installations will have custom records beside TYPO3 pages.
E.g. one uses EXT:news or EXT:tt_address to display news or personal information.
Those typically are displayed via a Plugin content element leading to the same Page
for all records.
This part allows to track views of individual records.
All configuration happens via :ref:`t3coreapi:DependencyInjection` inside of :file:`Services.yaml` of your Sitepackage.
.. note::
In contrast to :ref:`pageview`, there is no default rule.
No record is tracked by default as no TYPO3 installation has any default records to track.
In order to start tracking records, the rules need to be configured.
.. figure:: /Images/ListViewRecordviews.png
:align: center
Screenshot of list view of created "recordview" records.
.. figure:: /Images/RecordRecordview.png
:align: center
Screenshot of edit form view of created "recordview" records.
Saved record
------------
Whenever a recordview is tracked, a new record is created.
The record can be viewed via TYPO3 list module. That way all collected information can be checked.
Configure tracking
------------------
Let us examine an concrete example::
services:
_defaults:
autowire: true
autoconfigure: true
public: false
DanielSiepmann\Tracking\Middleware\Recordview:
public: true
arguments:
$rules:
topics:
matches: >
request.getQueryParams()["topic_id"] > 0
and not (context.getAspect("backend.user").isLoggedIn())
and not (request.getHeader("User-Agent")[0] matches "/^TYPO3|TYPO3 linkvalidator/")
and not (request.getHeader("User-Agent")[0] matches "/^Codeception Testing/")
and not (request.getHeader("User-Agent")[0] matches "/Wget|curl|Go-http-client/")
and not (request.getHeader("User-Agent")[0] matches "/bot|spider|Slurp|Sogou|NextCloud-News|Feedly|XING FeedReader|SEOkicks|Seekport Crawler|ia_archiver|TrendsmapResolver|Nuzzel/")
and not (request.getHeader("User-Agent")[0] matches "/mattermost|Slackbot|WhatsApp/")
recordUid: 'request.getQueryParams()["topic_id"]'
tableName: 'sys_category'
The first paragraph will not be explained, check out :ref:`t3coreapi:configure-dependency-injection-in-extensions` instead.
The second paragraph is where the tracking is configured.
The PHP class ``DanielSiepmann\Tracking\Middleware\Recordview`` is registered as PHP middleware and will actually track the request.
Therefore this class is configured.
The only interesting argument to configure is ``$rules``.
The argument itself is an array. That way one can configure multiple rules, e.g. one per record.
The above example includes a single rule for ``topics``, but further can be added.
Each rule has the following options which are all mandatory:
``matches``
A Symfony Expression, which is used to check whether the current rule should be processed for current request.
Check :ref:`pageview` to get further information, as it is the same implementation and concept.
``recordUid``
A Symfony Expression, which is used to fetch the UID of the actual record from current request.
Only the request itself is provided within the expression.
Check `PSR-7: HTTP message interfaces <https://www.php-fig.org/psr/psr-7/#321-psrhttpmessageserverrequestinterface>`__.
``tableName``
A simple string which defines the actual database table name where records are stored.
Widgets
-------
The extension does not provide any widgets, but providers for widgets of EXT:dashboard.
That way widgets of EXT:dashboard can be combined with all providers of this extension.
The concepts are not documented here, check :ref:`t3dashboard:start` instead.
.. toctree::
:glob:
RecordviewWidgets/*

View file

@ -0,0 +1,103 @@
.. php:namespace:: DanielSiepmann\Tracking\Dashboard\Provider
.. program:: DanielSiepmann\Tracking\Dashboard\Provider\Recordviews
.. _recordviews:
===========
Recordviews
===========
Provides the total views of configured records.
This way editors can see which records were requested the most during a specified period.
Example
=======
.. figure:: /Images/Widgets/Recordviews.png
:align: center
.. note::
In contrast to :ref:`pageview`, there is no default rule.
No record is tracked by default as no TYPO3 installation has any default records to track.
In order to start tracking records, the rules need to be configured.
Example widget configuration.
:file:`Configuration/Services.yaml`::
services:
dashboard.provider.danielsiepmann.tracking.records.topics:
class: 'DanielSiepmann\Tracking\Dashboard\Provider\Recordviews'
arguments:
$queryBuilder: '@querybuilder.tx_tracking_recordview'
$recordTableLimitation: ['sys_category']
dashboard.widget.danielsiepmann.tracking.records.topics:
class: 'TYPO3\CMS\Dashboard\Widgets\DoughnutChartWidget'
arguments:
$view: '@dashboard.views.widget'
$dataProvider: '@dashboard.provider.danielsiepmann.tracking.records.topics'
tags:
- name: 'dashboard.widget'
identifier: 'topicsDoughnut'
groupNames: 'tracking'
iconIdentifier: 'content-widget-chart-pie'
title: 'Topics'
description: 'Shows which topics are called most'
additionalCssClasses: 'dashboard-item--chart'
height: 'medium'
width: 'small'
Each widget should be a combination of an configured provider as well as an widget from EXT:dashboard.
The provider delivers results for all chart widgets.
The above example configures the provider first,
followed by an widget using the provider to display top topics.
Only the provider is documented, as the widget is part of EXT:dashboard.
Options
=======
.. option:: $days
Integer defining the number of days to respect.
Defaults to 31.
.. option:: $maxResults
Integer defining how many pages should be shown.
Defaults to 6 because EXT:dashboard only provides 6 colors.
Defaults to 6.
.. option:: $pagesToExclude
Array of page uids that should not be collected.
Defaults to empty array, all pages are shown.
This can be used if records are delivered through different pages.
This way news records can be filtered e.g. by limiting to press or internal news plugin pages.
.. option:: $recordTableLimitation
Array of database table names.
Defaults to empty array, records from all tables are shown.
Allows to limit the resulting records to specific tables.
E.g. only show records of ``sys_category`` or ``tt_address``.
.. option:: $recordTypeLimitation
Array of record types.
Defaults to empty array, records of all types are shown.
TYPO3 allows to define a types field per database table.
E.g. ``doktype`` for ``pages`` table, or ``CType`` for ``tt_content``.
That way different sub types of the same record can be stored.
Using this option offers a way to limit records e.g. to specific types of news or
address records.

View file

@ -0,0 +1,16 @@
[general]
project = Tracking Extension
release = 1.0.0
copyright = since 2020 by Daniel Siepmann
[html_theme_options]
github_branch = main
github_repository = DanielSiepmann/tracking
project_home = https://daniel-siepmann.de/projects/typo3-extension-tracking.html
project_issues = https://github.com/DanielSiepmann/tracking/issues/
project_repository = https://github.com/DanielSiepmann/tracking/
[intersphinx_mapping]
t3coreapi = https://docs.typo3.org/m/typo3/reference-coreapi/master/en-us/
t3dashboard = https://docs.typo3.org/c/typo3/cms-dashboard/master/en-us/

View file

@ -37,35 +37,8 @@ EXT:dashboard
Todos
=====
#. Add 100% code coverage (Widgets are missing)
#. Add version matrix to test with multiple PHP versions.
#. Add campaigns if possible (twitter parameter, etc.)
#. Add referrer if available.
#. Add device type phone, tablet, desktop?
#. Add operating system version?
#. Add further widgets.
#. Grouped by user agents (bar).
#. Top 404 requests (Collect them to show them, doughnut).
#. Move bot detection to another rule.
#. Keep indexing those requests, but mark them as bot and separate them in widgets.
#. Provide an overview of crawls as widgets. E.g. to allow fine grained robots.txt.
#. Add information to Admin Panel.
#. Add command that will iterate over all DB entries and remove ones matching the black list rule.
E.g. if rule is adjusted in meanwhile.
Example
=======
@ -75,4 +48,6 @@ The following widgets are added and could look like:
A new record is added which looks like:
.. image:: Documentation/Images/ListView.png
.. image:: Documentation/Images/ListViewPageviews.png
.. image:: Documentation/Images/RecordRecordview.png