Event times

[rroupski]

There are many event timestamps that are useful to record when event creation, collection, processing, and logging. The following terms are used within this discussion:

Event Producer -- the system (application, services, etc.) that generates events.
Event Consumer -- the system that receives the events generated by the event producer.
Event Processor -- the system that processes and logs the events received by the event consumer.

• time: timestamp_t
  The normalized event occurrence time. Normalized time means the original event time ref_time was corrected for the clock skew and it was converted to the OCSF timestamp_t.

• ref_time: string
  The original event time, as created by the event producer. The time format is not specified by the OCSF. The time could be UTC time in milliseconds (1659378222123), ISO 8601 (2019-09-07T15:50-04:00), or any other value (12/13/2021 10:12:55 PM).

• created_time: timestamp_t (new attribute)
  The created time is the time when the event was created by the event producer. The created_time, if present, must be equal or greater than the ref_time.
  Currently it is defined as timestamp_t, however the event producer may not use the OCSF time format. Do we need ref_created_time and the created_time is the normalized created time?
  The created_time is useful in cases when the event producer creates events sometime after they occur.

• processed_time: timestamp_t (rename as posted_time)
  The event processed time, such as an ETL operation.
  The posted time is the time when the event was sent by the event producer to the event consumer. The posted time can be used to determine the clock skew. Clock skew occurs when the clock time on one computer differs from the clock time on another computer. I think we need one more time, the time when was received by the event consumer, received_time?

• logged_time: timestamp_t
  The time when the event processor logged the event. It must be equal or greater than the event time.

• updated_time: timestamp_t
  The time when the event was last updated or enriched.It must be equal or greater than the event time. It could be less-than, equal, or greater-than the logged_time .

• start_time/end_time: timestamp_t
  The start and end event times are used when the event represents some activity that happened over time. The other use-case is event aggregation. Aggregation is a mechanism that allows for high number of events of the same event type that are raised to be merged into one for more efficient processing. Since the event aggregation could be done by the event producer, the event consumer, or the event processor, so what should be the time format?

[pagbabian-splunk]

I like these options as discussed. If we end up with Proposal 11 for time format, a profile will be available for RFC 3339 against these times, just like other timestamp attributes in the schema. Given the computational ease of using timestamp_t, start_time / end_time makes the most sense since the time range is a subtraction.

Re: a received_time, the normalized time _time is computed based on the correction between the receiver's clock, assumed to be accurate, and the sender's clock, not assumed to be accurate. The ref_time is presumed to be the time when the event was produced or created, assuming an accurate clock (which may not always be the case). There needs to be a way to normalize the ref_time into _time. If ref_time is in an arbitrary format, and created_time is not equal to ref_time, we have a problem doing this.

ref_xxx attributes are unaltered from the source (as if they were unmapped). Therefore we need an event_time that is a well known form of ref_time.

[rroupski]

the time that is a well known form of ref_time, right?

[pagbabian-splunk]

Yes, right.

[jp-harvey]

One thing to consider is whether we care about tracking the processing lineage.

We centered OCSF around the producer then we could have time, ref_time, updated_time, and start_time / end_time as part of the base object. All consumers and processors could then be in a map or array which is essentially a timeseries of the processing / consumption events.

[JasonKeirstead]

So far I do not understand a few things about this

• The description of the ref_time field does not really explain what ref_time is supposed to be used for. It explains what is in there - but how is it supposed to be used and why does anyone need this duplicate information if 'time' has it already?

• time field says 'correct for clock skew' and I do not understand what that is supposed to mean or how one is supposed to accomplish it. Are you suggesting the data store should manipulate the time in the log? This does not sound like a good idea.

• I do not understand what 'processed_time' is for - again the description is vague. ETL of what to what? This sounds like an application-specific use case, not something OCSF should have in the base schema.

In general log processing has only 3 timestamps* that matter

• The time in the log itself
• The time the log was received
• The time the log was written to disk
  Network flows have a start and end timestamp as well but those are in that object.

I have never heard of a 'processed time' and I don't know which one of these 3 it is.

I think for all of these, someone needs ot explain the end-to-end use-case, including how a threat analyst is supposed to use the information - the fact we can imagine a scenario where a field exists, does not make it useful enough to be part of a base schema.

If the information can not be used in a security use case and we can't plainly articulate where it comes from and how that is used to inform the analyst of something important, then conveying is unnecessary.

[pagbabian-splunk]

There are a few ways processed time could be interpreted (and we would need to disambiguate). 1/ the event is processed in stream before it is received by some consumer. 2/ the event could be delayed in transit (e.g. batched or backhauled) such that the time in the log itself and the time it is posted to a consumer or processor is significantly non-zero. In this case, a less ambiguous post time would be better than processed time.
Regarding correcting for clock skew, often distributed systems don't have synchronized clocks. If an event is produced at a system with a clock that is incorrect vis-a-vis a consumer's clock, it needs to be corrected. One way to approximate the correction is to measure the time between the event posting and the event reception, assume that it should be zero, and correct via the difference. However in the event there is a batch of events that are delayed, many events in the batch would be assumed to be skewed if they all shared the same clock as the agent posting the batch.