diff --git a/docs/api-reference/failed-events/index.md b/docs/api-reference/failed-events/index.md
new file mode 100644
index 0000000000..8360032c92
--- /dev/null
+++ b/docs/api-reference/failed-events/index.md
@@ -0,0 +1,225 @@
+---
+title: "Failed event types"
+sidebar_position: 15
+---
+
+This page lists all the possible types of [failed events](/docs/fundamentals/failed-events/index.md).
+
+## Where do failed events originate?
+
+While an event is being processed by the pipeline it is checked to ensure it meets the specific formatting or configuration expectations. These include checks like: does it match the schema it is associated with, were Enrichments successfully applied, and was the payload sent by the tracker acceptable.
+
+Generally, the [Collector](/docs/api-reference/stream-collector/index.md) tries to write any payload to the raw stream, no matter its content, and no matter whether it is valid. This explains why many of the failure types are filtered out by the [Enrich](/docs/api-reference/enrichment-components/index.md) application, and not any earlier.
+
+:::note
+
+The Collector might receive events in batches. If something is wrong with the Collector payload as a whole (e.g. due to a [Collector payload format violation](#collector-payload-format-violation)), the generated failed event would represent an entire batch of Snowplow events.
+
+Once the Collector payload successfully reaches the validation and enrichment steps, it is split into its constituent events. Each of them would fail (or not fail) independently (e.g. due to an [enrichment failure](#enrichment-failure)). This means that each failed event generated at this stage represents a single Snowplow event.
+
+:::
+
+## Schema violation
+
+This failure type is produced during the process of [validation and enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md). It concerns the [self-describing events](/docs/fundamentals/events/index.md#self-describing-events) and [entities](/docs/fundamentals/entities/index.md) which can be attached to your snowplow event.
+
+
+
+In order for an event to be processed successfully:
+
+1. There must be a schema in an [Iglu repository](/docs/api-reference/iglu/iglu-repositories/index.md) corresponding to each self-describing event or entity. The enrichment app must be able to look up the schema in order to validate the event.
+2. Each self-describing event or entity must conform to the structure described in the schema. For example, all required fields must be present, and all fields must be of the expected type.
+
+If your pipeline is generating schema violations, it might mean there is a problem with your tracking, or a problem with your [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md) which lists where schemas should be found. The error details in the schema violation JSON object should give you a hint about what the problem might be.
+
+Snowplow BDP customers should check in the Snowplow BDP Console that all data structures are correct and have been [promoted to production](/docs/data-product-studio/data-structures/manage/ui/index.md). Snowplow Community Edition users should check that the Enrichment app is configured with an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/index.md) that points to a repository containing the schemas.
+
+Next, check the tracking code in your custom application, and make sure the entities you are sending conform to the schema definition.
+
+Once you have fixed your tracking, you might want to also [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md), to avoid any data loss.
+
+Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
+
+Schema violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema).
+
+
+
+## Enrichment failure
+
+This failure type is produced by the [Enrich](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, and it represents any failure to enrich the event by one of your configured enrichments.
+
+
+
+There are many reasons why an enrichment will fail, but here are some examples:
+
+- You are using the [custom SQL enrichment](/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/index.md) but the credentials for accessing the database are wrong
+- You are using the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/index.md) but have mis-configured the location of the MaxMind database
+- You are using the [custom API request enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/index.md) but the API server is not responding
+- The raw event contained an unstructured event field or a context field which was not valid JSON
+- An Iglu server responded with an unexpected error response, so the event schema could not be resolved
+
+If your pipeline is generating enrichment failures, it might mean there is a problem with your enrichment configuration. The error details in the enrichment failure JSON object should give you a hint about what the problem might be.
+
+Once you have fixed your enrichment configuration, you might want to also [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md), to avoid any data loss.
+
+Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
+
+Enrichment failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema).
+
+
+
+## Collector payload format violation
+
+This failure type is produced by the [Enrich](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, when Collector payloads from the raw stream are deserialized from thrift format.
+
+
+
+Violations could be:
+
+- Malformed HTTP requests
+- Truncation
+- Invalid query string encoding in URL
+- Path not respecting `/vendor/version`
+
+The most likely source of this failure type is bot traffic that has hit the Collector with an invalid HTTP request. Bots are prevalent on the web, so do not be surprised if your Collector receives some of this traffic. Generally you would ignore, and not try to recover, a Collector payload format violation, because it likely did not originate from a tracker or a webhook.
+
+Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
+
+Collector payload format violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/collector_payload_format_violation/jsonschema).
+
+
+
+## Adaptor failure
+
+This failure type is produced by the [Enrich](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, when it tries to interpret a Collector payload from the raw stream as a HTTP request from a [3rd party webhook](/docs/sources/webhooks/index.md).
+
+:::info
+
+Many adaptor failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline.
+
+:::
+
+
+
+The failure could be:
+
+1. The vendor/version combination in the Collector URL is not supported. For example, imagine an HTTP request sent to `/com.sandgrod/v3` which is a mis-spelling of the [sendgrid adaptor](http://sendgrid) endpoint.
+2. The webhook sent by the 3rd party does not conform to the expected structure and list of fields for this webhook. For example, imagine the 3rd party webhook payload is updated and stops sending a field that it was sending before.
+
+Many adaptor failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline. However, if you believe you are missing data because of a misconfigured webhook, then you might try to fix the webhook and then [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
+
+Adapter failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema).
+
+
+
+## Tracker protocol violation
+
+This failure type is produced by the [Enrich](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, when an HTTP request does not conform to our [Snowplow Tracker Protocol](/docs/sources/trackers/snowplow-tracker-protocol/index.md).
+
+
+
+Snowplow trackers send HTTP requests to the `/i` endpoint or the `/com.snowplowanalytics.snowplow/tp2` endpoint, and they are expected to conform to this protocol.
+
+Many tracker protocol violations are caused by bot traffic, so do not be surprised to see some of them in your pipeline.
+
+Another likely source is misconfigured query parameters if you are using the [pixel tracker](/docs/sources/trackers/pixel-tracker/index.md). In this case you might try to fix your application sending events, and then [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
+
+Tracker protocol violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema).
+
+
+
+## Size violation
+
+This failure type can be produced either by the [Collector](/docs/api-reference/stream-collector/index.md) or by the [Enrich](/docs/pipeline/enrichments/what-is-enrichment/index.md) application. It happens when the size of the raw event or enriched event is too big for the output message queue. In this case it will be truncated and wrapped in a size violation failed event instead.
+
+
+
+Failures of this type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md). The best you can do is to fix any application that is sending over-sized events.
+
+Because this failure is handled during collection or enrichment, events in the real time good stream are free of this violation type.
+
+The size violation schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/size_violation/jsonschema/1-0-0).
+
+
+
+## Loader parsing error
+
+This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/index.md), if the enriched event in the real time good stream cannot be parsed as a canonical TSV event format. For example, if the row does not have enough columns (131 are expected) or the `event_id` is not a UUID. This error type is uncommon and unexpected, because it can only be caused by an invalid message in the stream of validated enriched events.
+
+
+
+This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+The loader parsing error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_parsing_error/jsonschema/2-0-0).
+
+
+
+## Loader Iglu error
+
+This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/index.md) and describes an error using the [Iglu](/docs/api-reference/iglu/index.md) subsystem.
+
+
+
+For example:
+
+- A schema is not available in any of the repositories listed in the [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md).
+- Some loaders (e.g. [RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/index.md) and [Postgres loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/index.md)) make use of the "schema list" API endpoints, which are only implemented for an [Iglu server](/docs/api-reference/iglu/iglu-repositories/iglu-server/index.md) repository. A loader Iglu error will be generated if the schema is in a [static repo](/docs/api-reference/iglu/iglu-repositories/static-repo/index.md) or [embedded repo](/docs/api-reference/iglu/iglu-repositories/jvm-embedded-repo/index.md).
+- The loader cannot auto-migrate a database table. If a schema version is incremented from `1-0-0` to `1-0-1` then it is expected to be [a non-breaking change](/docs/api-reference/iglu/common-architecture/schemaver/index.md), and many loaders (e.g. RDB loader) attempt to execute a `ALTER TABLE` statement to facilitate the new schema in the warehouse. But if the schema change is breaking (e.g. string field changed to integer field) then the database migration is not possible.
+
+This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Loader Iglu error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_iglu_error/jsonschema/2-0-0).
+
+
+
+## Loader recovery error
+
+Currently only the [BigQuery repeater](/docs/api-reference/loaders-storage-targets/bigquery-loader/index.md#block-8db848d4-0265-4ffa-97db-0211f4e2293d) generates this error. We call it "loader recovery error" because the purpose of the repeater is to recover from previously failed inserts. It represents the case when the software could not re-insert the row into the database due to a runtime failure or invalid data in a source.
+
+
+
+This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Loader recovery error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_recovery_error/jsonschema/1-0-0)
+
+
+
+## Loader runtime error
+
+This failure type can be produced by any loader and describes generally any runtime error that we did not catch. For example, a DynamoDB outage, or a null pointer exception. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.
+
+
+
+This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Loader runtime error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-0-1).
+
+
+
+## Relay failure
+
+This failure type is only produced by relay jobs, which transfer Snowplow data into a 3rd party platform. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.
+
+
+
+This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Relay failure schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/relay_failure/jsonschema/1-0-0).
+
+
+
+## Generic error
+
+This is a failure type for anything that does not fit into the other categories, and is unlikely enough that we have not created a special category. The failure error messages should give you a hint about what has happened.
+
+
+
+This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+
+Generic error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/generic_error/jsonschema/1-0-0).
+
+
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/images/athena-count.png b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/athena-count.png
similarity index 100%
rename from docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/images/athena-count.png
rename to docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/athena-count.png
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/images/athena-create-table.png b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/athena-create-table.png
similarity index 100%
rename from docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/images/athena-create-table.png
rename to docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/athena-create-table.png
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/images/bigquery-count.png b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/bigquery-count.png
similarity index 100%
rename from docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/images/bigquery-count.png
rename to docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/bigquery-count.png
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-1-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-1-1.jpg
deleted file mode 100644
index 09cf844a62..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-1-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-1-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-1-2.jpg
deleted file mode 100644
index 09cf844a62..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-1-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-2-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-2-1.jpg
deleted file mode 100644
index d8dca42c4d..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-2-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-2-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-2-2.jpg
deleted file mode 100644
index d8dca42c4d..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-2-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-3-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-3-1.jpg
deleted file mode 100644
index 712e26d0ff..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-3-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-3-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-3-2.jpg
deleted file mode 100644
index 712e26d0ff..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-3-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-4-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-4-1.jpg
deleted file mode 100644
index 2d018fedb5..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-4-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-4-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-4-2.jpg
deleted file mode 100644
index 2d018fedb5..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-4-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-5-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-5-1.jpg
deleted file mode 100644
index 31cf06c482..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-5-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-5-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-5-2.jpg
deleted file mode 100644
index 31cf06c482..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-5-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-6-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-6-1.jpg
deleted file mode 100644
index fefce225a7..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-6-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-6-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-6-2.jpg
deleted file mode 100644
index fefce225a7..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-6-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-7-1.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-7-1.jpg
deleted file mode 100644
index d34c4ad2ba..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-7-1.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-7-2.jpg b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-7-2.jpg
deleted file mode 100644
index d34c4ad2ba..0000000000
Binary files a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/images/failed-evs-gcs-7-2.jpg and /dev/null differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md
index 18517d090c..ffac681ac8 100644
--- a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md
+++ b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md
@@ -1,10 +1,15 @@
---
title: "Accessing failed events in file storage"
-sidebar_label: "Using S3 or GCS"
-sidebar_position: 1
+sidebar_label: "In file storage"
+sidebar_position: 2
---
-When failed events are generated on your pipeline the raw event payload along with details about the failure are saved into file storage (S3 on AWS, GCS on Google Cloud).
+```mdx-code-block
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+```
+
+On AWS and GCP, when failed events are generated on your pipeline, the raw event payload along with details about the failure are saved into file storage (S3 on AWS, GCS on Google Cloud).
:::info Community Edition quick start guide on GCP
@@ -12,56 +17,325 @@ If you followed the [Community Edition quick start guide](/docs/get-started/snow
:::
-You can directly access and download examples of events that are failing from file storage, this is useful for further investigation and also required to design a recovery operation.
+## Retrieving raw data
+
+You can directly access and download examples of events that are failing from file storage. This is useful for further investigation, and also required to design a recovery operation.
+
+
+
+
+Login to your AWS Console account and navigate to the sub-account that contains your Snowplow pipeline. Then navigate to your S3 storage buckets.
+
+You should find a bucket with a name ending in `-kinesis-s3-bad` and within that a folder with your pipeline name e.g. `prod1`.
+
+
+
+
+
+Navigate into this folder and you should see `partitioned` (search if it isn't visible).
+
+
+
+Within this folder, there will be a subfolder for each type of failed event. Select the relevant type for the failed events you wish to find.
+
+
+
+You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.
+
+
+
+
+
+
+Login to your Google Cloud Platform account and navigate to the project that contains your Snowplow pipeline. Then navigate to your Google Cloud Storage buckets.
+
+You should find a bucket named with a prefix of `sp-storage-loader-bad`.
+
+
+
+Navigate into this folder and you should see `partitioned` (search if it isn't visible).
+
+
+
+Within this folder, there will be a subfolder for each type of failed event. Select the relevant type for the failed events you wish to find.
+
+
+
+You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.
+
+
+
+
+
+
+
+
+
+
+## Using Athena or BigQuery
+
+[Athena](https://aws.amazon.com/athena/) on AWS and [BigQuery](https://cloud.google.com/bigquery) on GCP are tools that let you query your failed events, using the cloud storage files as a back-end data source.
+
+```sql
+SELECT data.failure.messages FROM adapter_failures
+WHERE from_iso8601_timestamp(data.failure.timestamp) > timestamp '2020-04-01'
+```
+
+This approach is handy for debugging your pipeline without the need to load your failed events into a separate database.
+
+Before you can query this data, you need to create corresponding tables in Athena or BigQuery as we explain below. Each different failed event type (e.g. schema violations, adapter failures) has a different schema, so you will need one table per event type.
+
+## Creating the tables
+
+
+
+
+Go to [the Athena dashboard](https://eu-central-1.console.aws.amazon.com/athena/home) and use the query editor. Start by creating a database (replace `{{ DATABASE }}` with the name of your pipeline, e.g. `prod1` or `qa1`):
+
+```sql
+CREATE DATABASE IF NOT EXISTS {{ DATABASE }}
+```
+
+Then run each SQL statement provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/athena) by copying them into the Athena query editor. We recommend creating all tables, although you can skip the ones you are not interested in.
+
+:::info Placeholders
+
+Note that the SQL statements contain a few placeholders which you will need to edit before you can create the tables:
+
+* `{{ DATABASE }}` — as above, change this to the name of your pipeline, e.g. `prod1` or `qa1`.
+* `s3://{{ BUCKET }}/{{ PIPELINE }}` — this should point to the directory in S3 where your bad rows files are stored.
+
+:::
+
+
+
+
+
+
+:::info Community Edition quick start guide on GCP
+
+If you followed the [Community Edition quick start guide](/docs/get-started/snowplow-community-edition/quick-start/index.md), you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/index.md) to save failed events into GCS, as it’s currently not included in the Terraform scripts.
+
+:::
+
+:::note
+
+These instructions make use of the [`bq` command-line tool](https://cloud.google.com/bigquery/docs/bq-command-line-tool) which is packaged with the [Google cloud SDK](https://cloud.google.com/sdk/docs). Follow the SDK instructions for how to [initialize and authenticate the SDK](https://cloud.google.com/sdk/docs/initializing). Also take a look at the [BigQuery dashboard](https://console.cloud.google.com/bigquery) as you run these commands, so you can see your tables as you create them.
+
+:::
+
+Create a dataset to contain your failed event tables:
+
+```bash
+bq mk --data_location=EU bad_rows_prod1
+# Dataset 'my-snowplow-project:bad_rows_prod1' successfully created.
+```
+
+The `--data-location` should match the location of your bad rows bucket. Also replace `prod1` with the name of your pipeline.
+
+Next, download the table definitions provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery) in JSON format.
+
+:::info Placeholders
+
+Each table definition contains a `{{ BUCKET }}` placeholder which needs to be changed to the GCS bucket where your bad rows files are stored (e.g. `sp-storage-loader-bad-prod1-com_acme`).
+
+:::
+
+Now run `bq mk` for each table definition in turn. Use the `--external_table_definition` parameter so that BigQuery uses the bucket as the back-end data source. Here is how to run the command for the first three tables (note that you should change the dataset name `bad_rows_prod1` to match the dataset you just created):
+
+```bash
+bq mk \
+ --display_name="Adapter failures" \
+ --external_table_definition=./adapter_failures.json \
+ bad_rows_prod1.adapter_failures
+
+# Table 'my-snowplow-project:bad_rows_prod1.adapter_failures' successfully created.
+
+bq mk \
+ --display_name "Schema violations" \
+ --external_table_definition=./schema_violations.json \
+ bad_rows_prod1.schema_violations
+
+# Table 'my-snowplow-project:bad_rows_prod1.schema_violations' successfully created.
+
+bq mk \
+ --display_name "Tracker protocol violations" \
+ --external_table_definition=./tracker_protocol_violations.json \
+ bad_rows_prod1.tracker_protocol_violations
+
+# Table 'my-snowplow-project:bad_rows_prod1.tracker_protocol_violations' successfully created.
+```
+
+Run the corresponding commands for the remaining table definitions. We recommend creating all tables, although you can skip the ones you are not interested in.
+
+:::tip Why not just auto-detect the schemas?
+
+BigQuery has an “Auto-detect” feature to automatically generate the table definition for you by inspecting the file contents. So you might wonder why it is necessary to provide explicit schema definitions for your tables.
+
+There are two potential pitfalls when using the autogenerated schema with the Snowplow bad rows files:
+
+- _Optional fields_. BigQuery might not “notice” that a field exists, depending on the sample of data used to detect the schema.
+- _Polymorphic fields_, e.g. `error` that can be either a string or an object. BigQuery will throw an exception if it sees an unexpected value for a field. Our table definitions use the `JSON` data type for these fields.
+
+:::
+
+
+
+
+## Querying the data
+
+
+
+
+As example of using your Athena tables, you might start by getting counts of each failed event type from the last week. Repeat this query for each table you have created:
+
+```sql
+SELECT COUNT(*) FROM schema_violations
+WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
+```
+
+
+
+If you have schema violations, you might want to find which tracker sent the event:
+
+```sql
+SELECT data.payload.enriched.app_id, COUNT(*) FROM schema_violations
+WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
+GROUP BY data.payload.enriched.app_id
+```
+
+You can do a deeper dive into the error messages to get a explanation of the last 10 failures:
+
+```sql
+SELECT message.field AS field,
+ message.value AS value,
+ message.error AS error,
+ message.json AS json,
+ message.schemaKey AS schemaKey,
+ message.schemaCriterion AS schemaCriterion
+FROM schema_violations
+CROSS JOIN UNNEST(data.failure.messages) AS t(message)
+ORDER BY data.failure.timestamp DESC
+LIMIT 10
+```
+
+
+
+
+You can query your tables from the query editor in the [BigQuery console](https://console.cloud.google.com/bigquery). You might want to start by getting counts of each failed event type from the last week. This query will work, but it is relatively expensive because it will scan all files in the `schema_violations` directory:
+
+```sql
+SELECT COUNT(*) FROM bad_rows_prod1.schema_violations
+WHERE data.failure.timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY);
+```
+
+You can construct a more economical query by using the `_FILE_NAME` pseudo column to restrict the scan to files from the last week:
-## Retrieving raw data from S3 on AWS
+```sql
+SELECT COUNT(*) FROM bad_rows_prod1.schema_violations
+WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY);
+```
-- Login to your AWS Console account and navigate to the sub-account that contains your Snowplow pipeline
-- Navigate to your S3 storage buckets
-- You should find a bucket with a name ending in `-kinesis-s3-bad` and within that a folder with your pipeline name e.g. `prod1`
-- Navigate into this folder and you should see `partitioned` (search if it isn't visible), and within this a folder for each type of failed event. Select the relevant type for the failed events you wish to find.
-- You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.
+You can repeat that query for each table you created in your bad rows dataset.
-
+
-Step 1 - login to AWS and navigate to S3, find or search for the '-kinesis-s3-bad' folder for your pipeline
+If you have schema violations, you might want to find which tracker sent the event:
-
+```sql
+SELECT data.payload.enriched.app_id, COUNT(*) FROM bad_rows_prod1.schema_violations
+WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)
+GROUP BY data.payload.enriched.app_id;
+```
-Step 2 - locate or search for the folder called 'partitioned'
+If you have tracker protocol failures, you can do a deeper dive into the error messages to get a explanation of the last 10 failures:
-
+```sql
+SELECT message.field AS field,
+ message.value AS value,
+ message.error AS error,
+ message.expectation AS expectation,
+ message.schemaKey AS schemaKey,
+ message.schemaCriterion AS schemaCriterion
+FROM bad_rows_prod1.tracker_protocol_violations,
+UNNEST(data.failure.messages) AS message
+WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)
+ORDER BY data.failure.timestamp DESC
+LIMIT 10;
+```
-Step 3 - select the relevant folder for your error type
+
+ Digging deeper
-
+You might notice that the `error` field in the result of the query above has the `JSON` type.
+This is because depending on the variety of the failed event, the `error` might be a simple string or a complex object with additional detail.
-Step 4 - use the date and timestamps to find a batch of failed events that will contain an example of the event you wish to find
+For example, the “invalid JSON” message might have this `error`:
-## Retrieving raw data from GCS on GCP
+```json
+"invalid json: expected false got 'foo' (line 1, column 1)"
+```
-- Login to your Google Cloud Platform account and navigate to the project that contains your Snowplow pipeline
-- Navigate to your Google Cloud Storage buckets
-- You should find a bucket named with a prefix of `sp-storage-loader-bad`
-- Navigating into this you should see `partitioned`, and within this a folder for each type of failed event. Select the relevant type for the failed event you wish to find.
-- You can now drill down by year, month, day, and hour to find a batch of failed events that occured on that date / time period.
+In contrast, in case of a failure to resolve Iglu server, the value in the `error` field would look like this, with “sub-errors” inside:
-
+```json
+{
+ "error": "ResolutionError",
+ "lookupHistory": [
+ {
+ "attempts": 1,
+ "errors": [
+ {
+ "error": "RepoFailure",
+ "message": "Unexpected exception fetching: org.http4s.client.UnexpectedStatus: unexpected HTTP status: 404 Not Found"
+ }
+ ],
+ "lastAttempt": "2021-10-16T17:20:52.626Z",
+ "repository": "Iglu Central"
+ },
+ ...
+ ]
+}
+```
-Step 1 - find the right folder
+You can figure out what to expect from such a field by looking at the JSON schema for the respective type of failed events, in this case the [tracker protocol violations schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema/1-0-0). The mapping between the various failed event tables and the corresponding JSON schemas is [here](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery).
-
+BigQuery has a variety of JSON functions that allow you to extract data from within complex objects. For instance, if you are interested in Iglu repositories that failed to resolve, you can use something like this:
-Step 2 - select the partitioned folder
+```sql
+SELECT DISTINCT(JSON_VALUE(message.error.lookupHistory[0].repository))
+FROM ...
+WHERE ...
+AND message.error.lookupHistory IS NOT NULL
+```
-
+It’s also possible, although unwieldy, to reduce all `error`s to a single string:
-Step 3 - select the relevant folder for your error type
+```sql
+-- Unnest individual messages for each failed event
+WITH unnested_messages AS (
+ SELECT message, CASE
+ -- resolution errors
+ WHEN message.error.lookupHistory IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.lookupHistory[0].errors)
+ -- event validation errors
+ WHEN message.error.dataReports IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.dataReports)
+ -- schema validation errors
+ WHEN message.error.schemaIssues IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.schemaIssues)
+ -- other errors
+ ELSE [TO_JSON(STRUCT(message.error as message))]
+ END AS errors
+FROM bad_rows_prod1.tracker_protocol_violations,
+UNNEST(data.failure.messages) AS message
+WHERE ...)
-
+SELECT JSON_VALUE(error.message) AS error
+FROM unnested_messages,
+UNNEST(errors) AS error
+```
-Step 4 - drill down into the folder structure by year, month, day and time
+In the future, we plan to simplify the schemas of failed events so that they are more uniform and straightforward to query.
-
+
-Step 5 - once you find the raw files you can download them and view them in a text editor
+
+
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying-postgres/index.md b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying-postgres/index.md
deleted file mode 100644
index d6484ae2bc..0000000000
--- a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying-postgres/index.md
+++ /dev/null
@@ -1,32 +0,0 @@
----
-title: "Querying failed events in Postgres"
-sidebar_label: "Using Postgres"
-sidebar_position: 3
----
-
-If you use the [Postgres Loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/index.md) (not recommended for large volume production use cases), you can load your [failed events](/docs/fundamentals/failed-events/index.md) into Postgres.
-
-Each type of failed event is stored in its own table. You can get a full list of tables with the following query:
-
-```sql
-SELECT * FROM information_schema.tables
-WHERE table_schema = 'atomic_bad';
-```
-
-For instance, to check the number of [schema violations](/docs/fundamentals/failed-events/index.md#schema-violation), you can query the respective table:
-
-```sql
-SELECT COUNT(*) FROM atomic_bad.com_snowplowanalytics_snowplow_badrows_schema_violations_2;
-```
-
-Taking it further, you can check how many failed events you have by [schema](/docs/fundamentals/schemas/index.md) and error type:
-
-```sql
-SELECT
- "failure.messages"->0->'error'->'error' AS error,
- "failure.messages"->0->'schemaKey' AS schema,
- count(*) AS failed_events
-FROM atomic_bad.com_snowplowanalytics_snowplow_badrows_schema_violations_2
-GROUP BY 1,2
-ORDER BY 3 DESC
-```
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/index.md b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/index.md
deleted file mode 100644
index 48c6aa04cd..0000000000
--- a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/index.md
+++ /dev/null
@@ -1,276 +0,0 @@
----
-title: "Querying failed events in Athena or BigQuery"
-sidebar_label: "Using Athena or BigQuery"
-sidebar_position: 2
----
-
-```mdx-code-block
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-```
-
-[Athena](https://aws.amazon.com/athena/) on AWS and [BigQuery](https://cloud.google.com/bigquery) on GCP are tools that let you query your failed events, using the cloud storage files as a back-end data source.
-
-```sql
-SELECT data.failure.messages FROM adapter_failures
-WHERE from_iso8601_timestamp(data.failure.timestamp) > timestamp '2020-04-01'
-```
-
-This approach is great for debugging your pipeline without the need to load your failed events into a separate database.
-
-Before you can query this data, you need to create corresponding tables in Athena or BigQuery as we explain below. Each different failed event type (e.g. schema violations, adapter failures) has a different schema, so you will need one table per event type.
-
-## Creating the tables
-
-
-
-
-Go to [the Athena dashboard](https://eu-central-1.console.aws.amazon.com/athena/home) and use the query editor. Start by creating a database (replace `{{ DATABASE }}` with the name of your pipeline, e.g. `prod1` or `qa1`):
-
-```sql
-CREATE DATABASE IF NOT EXISTS {{ DATABASE }}
-```
-
-Then run each sql statement provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/athena) by copying them into the Athena query editor. We recommend creating all tables, although you can skip the ones you are not interested in.
-
-:::info Placeholders
-
-Note that the sql statements contain a few placeholders which you will need to edit before you can create the tables:
-
-* `{{ DATABASE }}` — as above, change this to the name of your pipeline, e.g. `prod1` or `qa1`.
-* `s3://{{ BUCKET }}/{{ PIPELINE }}` — this should point to the directory in S3 where your bad rows files are stored.
-
-:::
-
-
-
-
-
-
-:::info Community Edition quick start guide on GCP
-
-If you followed the [Community Edition quick start guide](/docs/get-started/snowplow-community-edition/quick-start/index.md), you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/index.md) to save failed events into GCS, as it’s currently not included in the Terraform scripts.
-
-:::
-
-:::note
-
-These instructions make use of the [bq command-line tool](https://cloud.google.com/bigquery/docs/bq-command-line-tool) which is packaged with the [google cloud sdk](https://cloud.google.com/sdk/docs). Follow the sdk instructions for how to [initialize and authenticate the sdk](https://cloud.google.com/sdk/docs/initializing). Also take a look at the [BigQuery dashboard](https://console.cloud.google.com/bigquery) as you run these commands, so you can see your tables as you create them.
-
-:::
-
-Create a dataset to contain your failed event tables:
-
-```bash
-bq mk --data_location=EU bad_rows_prod1
-# Dataset 'my-snowplow-project:bad_rows_prod1' successfully created.
-```
-
-The `--data-location` should match the location of your bad rows bucket. Also replace `prod1` with the name of your pipeline.
-
-Next, download the table definitions provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery) in JSON format.
-
-:::info Placeholders
-
-Each table definition contains a `{{ BUCKET }}` placeholder which needs to be changed to the GCS bucket where your bad rows files are stored (e.g. `sp-storage-loader-bad-prod1-com_acme`).
-
-:::
-
-Now run `bq mk` for each table definition in turn. Use the `--external_table_definition` parameter so that BigQuery uses the bucket as the back-end data source. Here is how to run the command for the first three tables (note that you should change the dataset name `bad_rows_prod1` to match the dataset you just created):
-
-```bash
-bq mk \
- --display_name="Adapter failures" \
- --external_table_definition=./adapter_failures.json \
- bad_rows_prod1.adapter_failures
-
-# Table 'my-snowplow-project:bad_rows_prod1.adapter_failures' successfully created.
-
-bq mk \
- --display_name "Schema violations" \
- --external_table_definition=./schema_violations.json \
- bad_rows_prod1.schema_violations
-
-# Table 'my-snowplow-project:bad_rows_prod1.schema_violations' successfully created.
-
-bq mk \
- --display_name "Tracker protocol violations" \
- --external_table_definition=./tracker_protocol_violations.json \
- bad_rows_prod1.tracker_protocol_violations
-
-# Table 'my-snowplow-project:bad_rows_prod1.tracker_protocol_violations' successfully created.
-```
-
-Run the corresponding commands for the remaining table definitions. We recommend creating all tables, although you can skip the ones you are not interested in.
-
-:::tip Why not just auto-detect the schemas?
-
-BigQuery has an “Auto-detect” feature to automatically generate the table definition for you by inspecting the file contents. So you might wonder why it is necessary to provide explicit schema definitions for your tables.
-
-There are two potential pitfalls when using the autogenerated schema with the Snowplow bad rows files:
-
-- _Optional fields_. BigQuery might not “notice” that a field exists, depending on the sample of data used to detect the schema.
-- _Polymorphic fields_, e.g. `error` that can be either a string or an object. BigQuery will throw an exception if it sees an unexpected value for a field. Our table definitions use the `JSON` data type for these fields.
-
-:::
-
-
-
-
-## Querying the data
-
-
-
-
-As example of using your Athena tables, you might start by getting counts of each failed event type from the last week. Repeat this query for each table you have created:
-
-```sql
-SELECT COUNT(*) FROM schema_violations
-WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
-```
-
-
-
-If you have schema violations, you might want to find which tracker sent the event:
-
-```sql
-SELECT data.payload.enriched.app_id, COUNT(*) FROM schema_violations
-WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
-GROUP BY data.payload.enriched.app_id
-```
-
-You can do a deeper dive into the error messages to get a explanation of the last 10 failures:
-
-```sql
-SELECT message.field AS field,
- message.value AS value,
- message.error AS error,
- message.json AS json,
- message.schemaKey AS schemaKey,
- message.schemaCriterion AS schemaCriterion
-FROM schema_violations
-CROSS JOIN UNNEST(data.failure.messages) AS t(message)
-ORDER BY data.failure.timestamp DESC
-LIMIT 10
-```
-
-
-
-
-You can query your tables from the query editor in the [BigQuery console](https://console.cloud.google.com/bigquery). You might want to start by getting counts of each failed event type from the last week. This query will work, but it is relatively expensive because it will scan all files in the `schema_violations` directory:
-
-```sql
-SELECT COUNT(*) FROM bad_rows_prod1.schema_violations
-WHERE data.failure.timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY);
-```
-
-You can construct a more economical query by using the `_FILE_NAME` pseudo column to restrict the scan to files from the last week:
-
-```sql
-SELECT COUNT(*) FROM bad_rows_prod1.schema_violations
-WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY);
-```
-
-You can repeat that query for each table you created in your bad rows dataset.
-
-
-
-If you have schema violations, you might want to find which tracker sent the event:
-
-```sql
-SELECT data.payload.enriched.app_id, COUNT(*) FROM bad_rows_prod1.schema_violations
-WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)
-GROUP BY data.payload.enriched.app_id;
-```
-
-If you have tracker protocol failures, you can do a deeper dive into the error messages to get a explanation of the last 10 failures:
-
-```sql
-SELECT message.field AS field,
- message.value AS value,
- message.error AS error,
- message.expectation AS expectation,
- message.schemaKey AS schemaKey,
- message.schemaCriterion AS schemaCriterion
-FROM bad_rows_prod1.tracker_protocol_violations,
-UNNEST(data.failure.messages) AS message
-WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)
-ORDER BY data.failure.timestamp DESC
-LIMIT 10;
-```
-
-
- Digging deeper
-
-You might notice that the `error` field in the result of the query above has the `JSON` type.
-This is because depending on the variety of the failed event, the `error` might be a simple string or a complex object with additional detail.
-
-For example, the “invalid JSON” message might have this `error`:
-
-```json
-"invalid json: expected false got 'foo' (line 1, column 1)"
-```
-
-In contrast, in case of a failure to resolve Iglu server, the value in the `error` field would look like this, with “sub-errors” inside:
-
-```json
-{
- "error": "ResolutionError",
- "lookupHistory": [
- {
- "attempts": 1,
- "errors": [
- {
- "error": "RepoFailure",
- "message": "Unexpected exception fetching: org.http4s.client.UnexpectedStatus: unexpected HTTP status: 404 Not Found"
- }
- ],
- "lastAttempt": "2021-10-16T17:20:52.626Z",
- "repository": "Iglu Central"
- },
- ...
- ]
-}
-```
-
-You can figure out what to expect from such a field by looking at the JSON schema for the respective type of failed events, in this case the [tracker protocol violations schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema/1-0-0). The mapping between the various failed event tables and the corresponding JSON schemas is [here](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery).
-
-BigQuery has a variety of JSON functions that allow you to extract data from within complex objects. For instance, if you are interested in Iglu repositories that failed to resolve, you can use something like this:
-
-```sql
-SELECT DISTINCT(JSON_VALUE(message.error.lookupHistory[0].repository))
-FROM ...
-WHERE ...
-AND message.error.lookupHistory IS NOT NULL
-```
-
-It’s also possible, although unwieldy, to reduce all `error`s to a single string:
-
-```sql
--- Unnest individual messages for each failed event
-WITH unnested_messages AS (
- SELECT message, CASE
- -- resolution errors
- WHEN message.error.lookupHistory IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.lookupHistory[0].errors)
- -- event validation errors
- WHEN message.error.dataReports IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.dataReports)
- -- schema validation errors
- WHEN message.error.schemaIssues IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.schemaIssues)
- -- other errors
- ELSE [TO_JSON(STRUCT(message.error as message))]
- END AS errors
-FROM bad_rows_prod1.tracker_protocol_violations,
-UNNEST(data.failure.messages) AS message
-WHERE ...)
-
-SELECT JSON_VALUE(error.message) AS error
-FROM unnested_messages,
-UNNEST(errors) AS error
-```
-
-In the future, we plan to simplify the schemas of failed events so that they are more uniform and straightforward to query.
-
-
-
-
-
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/images/enable-stream.png b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/images/enable-stream.png
new file mode 100644
index 0000000000..89cb111cf6
Binary files /dev/null and b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/images/enable-stream.png differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/images/loader-type.png b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/images/loader-type.png
new file mode 100644
index 0000000000..72212e1906
Binary files /dev/null and b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/images/loader-type.png differ
diff --git a/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/index.md b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/index.md
new file mode 100644
index 0000000000..562ab59bd2
--- /dev/null
+++ b/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/index.md
@@ -0,0 +1,131 @@
+---
+title: "Exploring failed events in the warehouse or data lake"
+description: "Load common types of failed events to a separate table in your warehouse or lake to analyze them easily."
+sidebar_label: "In warehouse or lake"
+sidebar_position: 1
+---
+
+:::note Compatibility
+
+This feature is available since Enrich 5.0.0 and works with Snowflake and Lake loaders.
+
+:::
+
+## Introduction
+
+[Failed events](/docs/fundamentals/failed-events/index.md) are events the pipeline had some problem processing (for example, events that did not pass validation).
+
+For the common failures (validation and enrichment), you can configure continuous loading of any offending events into _a separate table_ in your warehouse or lake. This way, you can easily inspect them and decide how they might be patched up (e.g. with SQL) and merged with the rest of your data.
+
+:::note
+
+This feature is not retroactive, i.e. only failed events that occur _after it’s enabled_ will be loaded into your desired destination.
+
+:::
+
+## Format
+
+The format of the failed events loaded into your warehouse or lake is [the same as for your atomic events](/docs/fundamentals/canonical-event/index.md). All the standard fields are present (unless themselves invalid — see below). This allows you to query and aggregate this data easily, e.g. if you want to see the number of failures per `app_id`.
+
+There are two differences compared to regular events.
+
+**Invalid data is removed from the event.** This principle applies to all columns:
+* Any invalid standard column (e.g. `geo_country`) will be set to `null` instead of the offending value.
+* Likewise, any column containing the JSON for a self-describing event (`unstruct_...`) will be set to `null` if that JSON fails validation.
+* Finally, for entity columns (`contexts_`), if one entity is invalid, it will be removed from the array of entities. If all entities are invalid, the whole column will be set to `null`.
+
+For more information about the different columns in Snowplow data, see [how Snowplow data is stored in the warehouse](/docs/destinations/warehouses-lakes/schemas-in-warehouse/index.md).
+
+**There is an extra column with failure details.** The column is named `contexts_com_snowplowanalytics_snowplow_failure_1`. In most cases, it will also contain the invalid data in some form. See the [next section](#example-failed-event) for an example.
+
+## Example failed event
+
+Here is an example of what the `contexts_com_snowplowanalytics_snowplow_failure_1` column might contain. Note that a single failed event might have more than one error. In this case, there is a required field missing, and also an unrecognized field present.
+
+```js
+[
+ {
+ // timestamp of the failure
+ "timestamp": "2025-01-15T14:12:50.498148Z",
+
+ // failure type and failure schema version
+ "failureType": "ValidationError",
+ "_schema_version": "1-0-0",
+
+ // the component where the failure happened
+ "componentName": "snowplow-enrich-kafka",
+ "componentVersion": "5.1.2",
+
+ // the schema of the offending event
+ "schema": "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",
+
+ // any properties which were invalid
+ "data": {
+ "invalidProperty": "This schema doesn't have this property"
+ },
+
+ // there can be multiple errors per event
+ "errors": [
+ {
+ "keyword": "required",
+ "message": "$.targetUrl: is missing but it is required",
+ "path": "$",
+ "source": "unstruct",
+ "targets": [
+ "targetUrl"
+ ]
+ },
+ {
+ "keyword": "additionalProperties",
+ "message": "$.invalidProperty: is not defined in the schema and the schema does not allow additional properties",
+ "path": "$",
+ "source": "unstruct",
+ "targets": [
+ "invalidProperty"
+ ]
+ }
+ ]
+ }
+]
+```
+
+## Setup
+
+To use this feature, you will first need to enable the stream that contains failed events in the [Snowplow TSV format](/docs/fundamentals/canonical-event/understanding-the-enriched-tsv-format/index.md) suitable for loading into your warehouse or lake.
+
+The instructions below are for Snowplow BDP users. For Community Edition, you will need to configure this manually via Terraform.
+
+:::note Infrastructure costs
+
+An additional stream (Kinesis, Pub/Sub or Event Hubs on AWS, GCP and Azure respectively) will be reflected in your cloud infrastructure costs (unless you are using BDP Cloud). That said, failed events are usually a tiny fraction of all events, so this stream will be minimally sized.
+
+:::
+
+Open the _“Pipeline configuration”_ section for the desired pipeline and select _“Failed events stream”_.
+
+
+
+Click _“Enable”_ and wait for the changes to take effect.
+
+Now you are ready to add a loader. Click _“Add failed events loader”_, which will take you to the destinations catalog.
+
+You can use the following loaders with the failed events stream:
+
+* Snowflake Streaming Loader
+* Lake Loader
+
+Pick your desired destination and follow the steps in the UI, selecting _“failed events”_ as the type of events.
+
+
+
+Note that as with any other loader, you will first need to create a connection to your warehouse or lake, and then the loader itself.
+
+:::warning PII in failed events
+
+Some of the problems that cause failed events could lead them to contain personally identifiable information (PII). For example, a validation error could stem from PII placed in the wrong field, and that field might not be pseudonymized, leaving PII exposed in the error message. Or the [PII enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/index.md) itself might have failed.
+
+For this reason, we strongly recommend loading failed events into a separate schema (in case of a warehouse) or storage location (in case of a data lake) compared to your atomic events. This allows you to restrict access to failed events.
+
+Keep this in mind when creating the connection.
+
+:::
diff --git a/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/manual/getting-started/index.md b/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/manual/getting-started/index.md
index 77c1b010d3..50b5a155ab 100644
--- a/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/manual/getting-started/index.md
+++ b/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/manual/getting-started/index.md
@@ -6,7 +6,7 @@ sidebar_position: 0
Event recovery at its core, is the ability to fix events that have failed and replay them through your pipeline.
-After inspecting failed events either in the [Snowplow BDP Console](/docs/data-product-studio/data-quality/failed-events/monitoring-failed-events/ui/index.md), or in the [partitioned failure buckets](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/index.md), you can determine which events are possible to recover based on what the fix entails.
+After inspecting failed events either in the [Snowplow BDP Console](/docs/data-product-studio/data-quality/failed-events/monitoring-failed-events/ui/index.md), or in the [partitioned failure buckets](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md), you can determine which events are possible to recover based on what the fix entails.
With recovery it is possible to:
@@ -16,9 +16,9 @@ With recovery it is possible to:
If your failed events would not be fixed by applying the above, they currently would be considered unrecoverable. Due to the fact that there might be a mix of recoverable and unrecoverable data in your storage, event recovery uses configuration in order to process only a subset of the failed events.
-### What you'll need to get started
+### What you'll need to get started
-The typical flow for recovery and some prerequisites to consider would be:
+The typical flow for recovery and some prerequisites to consider would be:
**Understanding the failure issue**
- Familiarity with the [failed event types](/docs/fundamentals/failed-events/index.md)
diff --git a/docs/data-product-studio/data-quality/snowplow-inspector/overview/index.md b/docs/data-product-studio/data-quality/snowplow-inspector/overview/index.md
index f33cf938fe..2a8280d99f 100644
--- a/docs/data-product-studio/data-quality/snowplow-inspector/overview/index.md
+++ b/docs/data-product-studio/data-quality/snowplow-inspector/overview/index.md
@@ -32,6 +32,6 @@ This makes the tool a good first port of call when trying to answer questions su
Additionally, you can configure the extension to show whether or not an event has passed validation according to any event validation rules codified in the corresponding [schema](/docs/fundamentals/schemas/index.md).
-For events that failed validation in production historically that you are unable to replicate in your own browser, see our guides on [how to query failed events](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/index.md) from their respective destinations.
+For events that failed validation in production historically that you are unable to replicate in your own browser, see our guides on [how to query failed events](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md) from their respective destinations.
These failed events have a [specific format](/docs/fundamentals/failed-events/index.md) that includes an array of helpful, detailed error messages that explain the exact reasons why the event failed validation.
These events can also [be imported](/docs/data-product-studio/data-quality/snowplow-inspector/importing-events/index.md#importing-failed-events) into the extension to view as if your browser had generated them itself.
diff --git a/docs/fundamentals/canonical-event/index.md b/docs/fundamentals/canonical-event/index.md
index ec7d410b4e..56b42e9c0a 100644
--- a/docs/fundamentals/canonical-event/index.md
+++ b/docs/fundamentals/canonical-event/index.md
@@ -2,7 +2,7 @@
title: "Understanding the structure of Snowplow data"
description: A summary of the Snowplow events table and its fields, including custom events and entities
sidebar_label: "What the data looks like"
-sidebar_position: 5
+sidebar_position: 4
---
```mdx-code-block
diff --git a/docs/fundamentals/failed-events/index.md b/docs/fundamentals/failed-events/index.md
index 75bcb2ce2c..9c9855a182 100644
--- a/docs/fundamentals/failed-events/index.md
+++ b/docs/fundamentals/failed-events/index.md
@@ -1,239 +1,76 @@
---
title: "Understanding failed events"
-sidebar_label: "Failed events (bad rows)"
-sidebar_position: 4
+sidebar_label: "Failed events"
+sidebar_position: 5
description: "Failed events represent data that did not pass validation or otherwise failed to be processed"
---
-## What is a Failed Event?
+_Failed events_ is an umbrella term for events that the pipeline had some problem processing.
-A Failed Event is simply what we label any event that was not able to be processed through your pipeline. There are multiple points at which an event can fail in the pipeline: collection, validation and enrichment.
-
-:::info Terminology
-
-An older term for “failed events” is “bad rows”. You will still see it used throughout this documentation and some APIs and tools.
-
-:::
-
-All failed events are routed to storage (AWS S3 or GCP Cloud Storage).
-
-Many types of failed events can be “recovered”. That is, you can fix the underlying issue (such as a problem in the pipeline setup or incorrect event data) and successfully load the events. See [recovering failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-
-## Where do Failed Events originate?
-
-While an event is being processed by the pipeline it is checked to ensure it meets the specific formatting or configuration expectations; these include checks like: does it match the schema it is associated with, were Enrichments successfully applied and was the payload sent by the tracker acceptable.
-
-Generally, the [Collector](/docs/api-reference/stream-collector/index.md) tries to write any payload to the raw stream, no matter its content, and no matter whether it is valid. This explains why many of the failure types are filtered out by the [enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, and not any earlier.
+These problems can arise at various stages:
+* Collection (e.g. invalid payload format)
+* Validation (e.g. event does not match the schema)
+* Enrichment (e.g. external API unavailable)
+* Loading (very unlikely with modern versions of Snowplow)
:::note
-The Collector might receive events in batches. If something is wrong with the Collector payload as a whole (e.g. due to a [Collector Payload Format Violation](#collector-payload-format-violation)), the generated failed event would represent an entire batch of Snowplow events.
-
-Once the Collector payload successfully reaches the validation and enrichment steps, it is split into its constituent events. Each of them would fail (or not fail) independently (e.g. due to an [Enrichment Failure](#enrichment-failure)). This means that each failed event generated at this stage represents a single Snowplow event.
+Failed events are **not** written to your `atomic` events table, which only contains high quality data. See [below](#dealing-with-failed-events) for how to deal with them.
:::
-## Failure Types
-
-### Schema Violation
-
-This failure type is produced during the process of [validation and enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md). It concerns the [self-describing events](/docs/fundamentals/events/index.md#self-describing-events) and [entities](/docs/fundamentals/entities/index.md) which can be attached to your snowplow event.
-
-
-
-In order for an event to be processed successfully:
-
-1. There must be a schema in an [iglu repository](/docs/api-reference/iglu/iglu-repositories/index.md) corresponding to each self-describing event or entity. The enrichment app must be able to look up the schema in order to validate the event.
-2. Each self-describing event or entity must conform to the structure described in the schema. For example, all required fields must be present, and all fields must be of the expected type.
-
-If your pipeline is generating schema violations, it might mean there is a problem with your tracking, or a problem with your [iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md) which lists where schemas should be found. The error details in the schema violation JSON object should give you a hint about what the problem might be.
-
-Snowplow BDP customers should check in the Snowplow BDP Console that all data structures are correct and have been [promoted to production](/docs/data-product-studio/data-structures/manage/ui/index.md). Snowplow Community Edition users should check that the Enrichment app is configured with an [iglu resolver file](/docs/api-reference/iglu/iglu-resolver/index.md) that points to a repository containing the schemas.
-
-Next, check the tracking code in your custom application, and make sure the entities you are sending conform the schema definition.
-
-Once you have fixed your tracking, you might want to also [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md), to avoid any data loss.
-
-Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
-
-Schema violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema).
-
-
+## Common failures
-### Enrichment failure
+The two most common types of failed events are:
-This failure type is produced by the [enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, and it represents any failure to enrich the event by one of your configured enrichments.
+* **Validation failures**. These happen when an [event](/docs/fundamentals/events/index.md) or an [entity](/docs/fundamentals/entities/index.md) does not match its [schema](/docs/fundamentals/schemas/index.md). The reason usually is incorrect tracking code. Validation can also fail if the schema is not available, e.g. because you forgot to add it to the production schema registry before putting the tracking code live.
-
+* **Enrichment failures**. These can be due to an [API enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/index.md) reaching out to an external API that’s down. Another cause is a failure in the custom code in a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/index.md).
-There are many reasons why an enrichment will fail, but here are some examples:
+:::tip
-- You are using the [custom SQL enrichment](/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/index.md) but the credentials for accessing the database are wrong.
-- You are using the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/index.md) but have mis-configured the location of the MaxMind database.
-- You are using the [custom API request enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/index.md) but the API server is not responding.
-- The raw event contained an unstructured event field or a context field which was not valid JSON.
-- An iglu server responded with an unexpected error response, so the event schema could not be resolved.
-
-If your pipeline is generating enrichment failures, it might mean there is a problem with your enrichment configuration. The error details in the enrichment failure JSON object should give you a hint about what the problem might be.
-
-Once you have fixed your enrichment configuration, you might want to also [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md), to avoid any data loss.
-
-Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
-
-Enrichment failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema).
-
-
-
-### Collector Payload Format Violation
-
-This failure type is produced by the [enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, when Collector payloads from the raw stream are deserialized from thrift format.
-
-
-
-Violations could be:
-
-- Malformed HTTP requests
-- Truncation
-- Invalid query string encoding in URL
-- Path not respecting /vendor/version
-
-The most likely source of this failure type is bot traffic that has hit the Collector with an invalid http request. Bots are prevalent on the web, so do not be surprised if your Collector receives some of this traffic. Generally you would ignore not try to recover a Collector payload format violation, because it likely did not originate from a tracker or a webhook.
-
-Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
-
-Collector payload format violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/collector_payload_format_violation/jsonschema).
-
-
-
-### Adaptor Failure
-
-This failure type is produced by the [enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, when it tries to interpret a Collector payload from the raw stream as a http request from a [3rd party webhook](/docs/sources/webhooks/index.md).
-
-:::info
-
-Many adaptor failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline.
+In many cases, you will be able to fix the underlying problem directly, e.g. by altering your tracking code, by providing the correct schema, or by changing your enrichment configuration.
:::
-
-
-The failure could be:
-
-1. The vendor/version combination in the Collector url is not supported. For example, imagine a http request sent to `/com.sandgrod/v3` which is a mis-spelling of the [sendgrid adaptor](http://sendgrid) endpoint.
-2. The webhook sent by the 3rd party does not conform to the expected structure and list of fields for this webhook. For example, imagine the 3rd party webhook payload is updated and stops sending a field that it was sending before.
-
-Many adaptor failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline. However, if you believe you are missing data because of a misconfigured webhook, then you might try to fix the webhook and then [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+## Other failures
-Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
+Other failures generally fall into 3 categories:
-Adapter failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema).
+* **Bots or malicious activity**. Bots, vulnerability scans, and so on, can send completely invalid events to the Collector. The format might be wrong, or the payload size might be extraordinarily large.
-
+* **Pipeline misconfiguration**. For example, a loader could be reading from the wrong stream (with events in the wrong format). This is quite rare, especially for Snowplow BDP, where all relevant pipeline configuration is automatic.
-### Tracker Protocol Violation
+* **Temporary infrastructure issue**. This is again rare. One example would be Iglu Server (schema registry) not being available.
-This failure type is produced by the [enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md) application, when a http request does not conform to our [Snowplow Tracker Protocol](/docs/sources/trackers/snowplow-tracker-protocol/index.md).
+:::tip
-
+All of these are internal failures you typically can’t address upstream.
-Snowplow trackers send http requests to the `/i` endpoint or the `/com.snowplowanalytics.snowplow/tp2` endpoint, and they are expected to conform to this protocol.
-
-Many tracker protocol violations are caused by bot traffic, so do not be surprised to see some of them in your pipeline.
-
-Another likely source is misconfigured query parameters if you are using the [pixel tracker](/docs/sources/trackers/pixel-tracker/index.md). In this case you might try to fix your application sending events, and then [recover the failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-
-Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.
-
-Tracker protocol violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema).
-
-
-
-### Size Violation
-
-This failure type can be produced either by the [Collector](/docs/api-reference/stream-collector/index.md) or by the [enrichment](/docs/pipeline/enrichments/what-is-enrichment/index.md) application. It happens when the size of the raw event or enriched event is too big for the output message queue. In this case it will be truncated and wrapped in a size violation failed event instead.
-
-
-
-Failures of this type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md). The best you can do is to fix any application that is sending over-sized events.
-
-Size violation schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/size_violation/jsonschema/1-0-0).
-
-
-
-### Loader Parsing Error
-
-This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/index.md), if the enriched event in the real time good stream cannot be parsed as a canonical TSV event format. For example, if line has not enough columns (not 131) or event_id is not UUID. This error type is uncommon and unexpected, because it can only be caused by an invalid message in the stream of validated enriched events.
-
-
-
-This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-
-Loader parsing error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_parsing_error/jsonschema/2-0-0).
-
-
-
-### Loader Iglu Error
-
-This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/index.md) and describes an error using the [Iglu](/docs/api-reference/iglu/index.md) subsystem.
-
-
-
-For example:
-
-- A schema is not available in any of the repositories listed in the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/index.md).
-- Some loaders (e.g. [RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/index.md) and [Postgres loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/index.md)) make use of the "schema list" api endpoints, which are only implemented for an [iglu-server](/docs/api-reference/iglu/iglu-repositories/iglu-server/index.md) repository. A loader iglu error will be generated if the schema is in a [static repo](/docs/api-reference/iglu/iglu-repositories/static-repo/index.md) or [embedded repo](/docs/api-reference/iglu/iglu-repositories/jvm-embedded-repo/index.md).
-- The loader cannot auto-migrate a database table. If a schema version is incremented from `1-0-0` to `1-0-1` then it is expected to be [a non-breaking change](/docs/api-reference/iglu/common-architecture/schemaver/index.md), and many loaders (e.g. RDB loader) attempt to execute a `ALTER TABLE` statement to facilitate the new schema in the warehouse. But if the schema change is breaking (e.g. string field changed to integer field) then the database migration is not possible.
-
-This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-
-Loader iglu error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_iglu_error/jsonschema/2-0-0).
-
-
-
-### Loader Recovery Error
-
-Currently only the [BigQuery repeater](/docs/api-reference/loaders-storage-targets/bigquery-loader/index.md#block-8db848d4-0265-4ffa-97db-0211f4e2293d) generates this error. We call it "loader recovery error" because the purpose of the repeater is to recover from previously failed inserts. It represents the case when the software could not re-insert the row into the database due to a runtime failure or invalid data in a source.
-
-
-
-This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-
-Loader recovery error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_recovery_error/jsonschema/1-0-0)
-
-
-
-### Loader Runtime Error
-
-This failure type can be produced by any loader and describes generally any runtime error that we did not catch. For example, a DynamoDB outage, or a null pointer exception. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.
-
-
-
-This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-
-Loader runtime error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-0-1).
+:::
-
+## Dealing with failed events
-### Relay Failure
+Snowplow BDP provides a dashboard and alerts for failed events. See [Monitoring failed events](/docs/data-product-studio/data-quality/failed-events/monitoring-failed-events/index.md).
-This failure type is only produced by relay jobs, which transfer Snowplow data into a 3rd party platform. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.
+---
-
+For the common failures (validation and enrichment), you can configure continuous loading of any offending events into _a separate table_ in your warehouse or lake. This way, you can easily inspect them and decide how they might be patched up (e.g. with SQL) and merged with the rest of your data.
-This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+:::note
-Relay failure schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/relay_failure/jsonschema/1-0-0).
+This feature is not retroactive, i.e. only failed events that occur _after it’s enabled_ will be loaded into your desired destination.
-
+:::
-### Generic Error
+The events will include a special column with the details of the failure, and any invalid columns will be set to `null`. Otherwise, the format is [the same as for your atomic events](/docs/fundamentals/canonical-event/index.md).
-This is a failure type for anything that does not fit into the other categories, and is unlikely enough that we have not created a special category. The failure error messages should give you a hint about what has happened.
+See [Exploring failed events](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/warehouse-lake/index.md) for more details and setup instructions.
-
+---
-This failure type cannot be [recovered](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
+Finally, on AWS and GCP all failed events are backed up in object storage (S3 and GCS respectively). Sometimes, but not in all cases (e.g. not if the original events exceeded size limits), it’s possible to recover them by replaying them through the pipeline. This is a complicated process mainly reserved for internal failures and outages. Refer to [Recovering failed events](/docs/data-product-studio/data-quality/failed-events/recovering-failed-events/index.md).
-Generic error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/generic_error/jsonschema/1-0-0).
+---
-
+You can find a full list of failed event types [in the API reference section](/docs/api-reference/failed-events/index.md).
diff --git a/docs/get-started/snowplow-community-edition/what-is-deployed/index.md b/docs/get-started/snowplow-community-edition/what-is-deployed/index.md
index 83acc72ce8..c16669982c 100644
--- a/docs/get-started/snowplow-community-edition/what-is-deployed/index.md
+++ b/docs/get-started/snowplow-community-edition/what-is-deployed/index.md
@@ -239,7 +239,7 @@ See the [S3 Loader](https://registry.terraform.io/modules/snowplow-devops/s3-loa
The following loaders and folders are available:
* Raw loader, `raw/`: events that come straight out of the Collector and have not yet been validated or enriched by the Enrich application. They are Thrift records and are therefore a little tricky to decode. There are not many reasons to use this data, but backing this data up gives you the flexibility to replay this data should something go wrong further downstream in the pipeline.
* Enriched loader, `enriched/`: enriched events, in GZipped blobs of [enriched TSV](/docs/fundamentals/canonical-event/understanding-the-enriched-tsv-format/index.md). Historically, this has been used as the staging ground for loading into data warehouses via the [Batch transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/index.md) application. However, it’s no longer used in the quick start examples.
-* Bad loader, `bad/`: [failed events](/docs/fundamentals/failed-events/index.md). You can [query them using Athena](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/index.md).
+* Bad loader, `bad/`: [failed events](/docs/fundamentals/failed-events/index.md). You can [query them using Athena](/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/index.md).
Also, if you choose Postgres as your destination, the Postgres loader will load all failed events into Postgres.
diff --git a/src/remark/abbreviations.js b/src/remark/abbreviations.js
index ae682045bb..14d4b9b812 100644
--- a/src/remark/abbreviations.js
+++ b/src/remark/abbreviations.js
@@ -24,7 +24,7 @@ const plugin = () => {
KCL: 'Kinesis Client Library',
OSS: 'Open Source Software',
QA: 'Quality Assurance',
- PII: 'Personal Identifiable Information',
+ PII: 'Personally Identifiable Information',
RDS: 'Amazon Relational Database Service',
S3: 'Amazon Cloud Object Storage',
SS: 'Server Side',
diff --git a/static/_redirects b/static/_redirects
index ef4ac4497e..d1e7537b76 100644
--- a/static/_redirects
+++ b/static/_redirects
@@ -328,3 +328,6 @@ docs/understanding-tracking-design/managing-data-structures-with-data-structures
/docs/storing-querying/* /docs/destinations/warehouses-lakes/:splat 301
/docs/recipes/* /docs/resources/recipes-tutorials/:splat 301
/docs/using-the-snowplow-console/* /docs/account-management/:splat 301
+
+# Failed events reshuffle
+/docs/data-product-studio/data-quality/failed-events/exploring-failed-events/querying/ /docs/data-product-studio/data-quality/failed-events/exploring-failed-events/file-storage/ 301