Fixing requirements relationship between process and parent_process

[awhite456]

Problem - A less detailed process object could be more compliant

The current version of the schema for the process object has parent_process as a recommended field, which points to another instance of the process object. This means that any constraints applied to fields in the process object are also applied to the parent_process.

For telemetry producers that provide less detail about the parent_process, this causes a situation where they may be more compliant if they do not provide a parent_process at all, rather than provide an incomplete parent_process object. Take as an example, an event that contains only a pid for the parent_process and no other parent information - a more compliant transformation would be to skip parent_process, which is only recommended, rather than add parent_process and having missing required fields for cmd_line and path.

Telemetry producers should be incentivized to populate as many fields as possible, not fewer fields, indicating that this aspect of the schema needs to change. For reference, an example of the minimum compliant process with parent_process populated is shown below:

{
    "cmd_line": "/usr/bin/uname -a",
    "parent_process": {
        "cmd_line": "/bin/bash",
        "path": "/bin/bash",
        "pid": 13220,
    },
    "path": "/usr/bin/uname",
    "pid": 6228,
}

An example of a provider that only provides a pid for the parent, and subsequently would not populate parent_process to remain compliant, is shown below:

{
    "cmd_line": "/usr/bin/uname -a",
    "path": "/usr/bin/uname",
    "pid": 6228,
}

Current state

The current state of required fields and parent_process for the process object are as follows:

"attributes": {
	"cmd_line": {
		"requirement": "required"
	},
	"parent_process": {
		"requirement": "recommended"
	},
	"path": {
		"requirement": "required"
	},
	"pid": {
		"requirement": "required"
	}
}

Option 1 - Make cmd_line and path recommended + make parent_process required

To change the requirements on fields for the parent_process, we could change the requirements on the root process object.

While this is the simplest solution to maintain, it also significantly lowers the bar for what constitutes a process object, as the cmd_line and path fields would no longer be required. This may result in compliant providers that do not provide enough information to be useful.

To handle the example case, where only the pid of the parent is provided, only the pid field would be able to be required. The minimum compliant process object would be as follows:

{
    "parent_process": {
        "pid": 13220,
    },
    "pid": 6228,
}

Option 2 - Create new parent_process object + make parent_process required

To change the requirements on fields for the parent_process, we could create a new object that clones the process object and has lower requirements.

This would allow the requirements of any field in the parent_process object to be modified independently of the process object, but would add additional overhead and complexity as now two process-like objects need to be maintained.

To handle the example case, where only the pid of the parent is provided, only the pid field would be required in the parent object. The minimum compliant process object would be as follows:

{
    "cmd_line": "/usr/bin/uname -a",
    "parent_process": {
        "pid": 13220,
    },
    "path": "/usr/bin/uname",
    "pid": 6228,
}

Option 3 - Integrate parent_process fields into process and make parent_pid required

To change the requirements on fields for the parent_process, we could move the fields from this object into the process object with the prefix parent_.

This would significantly increase the number of fields in the process object as each would need to be duplicated with the prefix parent_ and remove the ability to nest multiple parent_process objects, but would only result in one object to maintain and the ability to set individual requirements on any parent field.

To handle the example case, where only the pid of the parent is provided, only the parent_pid field would be required out of the parent_ fields. The minimum compliant process object would be as follows:

{
    "cmd_line": "/usr/bin/uname -a",
    "parent_pid": 13220,
    "path": "/usr/bin/uname",
    "pid": 6228,
},

Option 4 - Make parent_process required

To prevent providers being compliant by specifying less data, we could make parent_process required to force providers to provide as much data as they are able to.

This would not significantly change the format of the process object, but would result in some non-compliant providers specifying fields with blank values.

To handle the example case, where only the pid of the parent is provided, only the pid field would be populated in the parent and the other fields would be blank. The non-compliant process object from this provider would be as follows:

{
    "cmd_line": "/usr/bin/uname -a",
    "parent_process": {
        "cmd_line": "",
        "path": "",
        "pid": 13220,
    },
    "path": "/usr/bin/uname",
    "pid": 6228,
}

[jp-harvey]

@awhite456 I've been thinking about this a little bit in a different context recently. You make a good point about it being ideal that there's an incentive to provide complete information.

I think parent_process was made recommended rather than required because some processes don't have a parent - those launched by the kernel - which would make Option 4 not viable since walking up the process tree would end up with a process object without a parent which is invalid.

The problem, stated another way, is that at the moment it's not possible to tell if a process was started by the kernel or if the telemetry provider decided not to include the information, and the information is important.

Option 3 solves the nested object issue. It's limited in usefulness though since PIDs do not persist across process launches (for the vast majority of processes), so one needs to correlate to determine which parent process is the right one rather than it being unambiguously specified, and with that comes a reasonable possibility of the information being inaccurate - far from ideal for forensics.

Option 2 could work, but then a parent process is just another process so it will much of the time have a parent process and that's important information. That being the case you'll want to know what it's parent is and you're in the same boat as option 4.

That leaves option 1 which as you point out means one can simply provide not-very-much information.

The process object also has a downside in that the payload will get heavy if a process has many ancestors - effectively an unlimited number. The way around that would be to create a custom parent_process object like in 2, however as mentioned it still has the problem with needing a parent process field so while the payload would be slimmed down it is still just a lighter object.

Thinking out loud a bit, these are probably the ideal requirements with regards to tracking parents of a process:

1. Be able to trace a process back through parents and grandparents to the kernel
2. Be able to relate processes to each other without having unwieldy payloads
3. Be able to tell when a process has no parent (launched by the kernel)

If we translate those into schema conditions to meet all of those parent_process could require one of:

• A process object, or
• A process GUID

This would accomplish the following:

• With a process object we can provide the whole process tree, as we would now.
• The GUID option allows a provider to send only a GUID, thus keeping the payload light, while still being able to relate processes to each other by joining on previously collected data. A zeroed GUID could also be used to represent that the parent process was the kernel - ie. no parent.

There are some in-progress thoughts in there so let me know what you think, I suppose there's some other option where it's clear that the telemetry provider has chosen not to provide the parent process info.

[awhite456]

You raise two good points here that I agree the proposals fail to address:

• PIDs are insufficient for identifying a process uniquely
• We need a way to distinguish between when the parent is not included in the data and when the process has no parent

I agree that unique process identifiers (such as GUIDs) are the ideal way to address the problem of uniquely identifying processes. My preference would be to make these required in the process object, which would allow reduction of log event size for other events involving processes (such as network or file activity).

As you suggest, the zeroed guid would be a good way to represent when the process has no parent. A bunch of empty fields have an ambiguous meaning.

I am not sure I agree with the proposal of requiring a parent_process OR parent_guid, as this changes analysis workflows depending on the provider of the data. I also do not agree with just requiring a parent_guid, as this makes simple searches for queries like "word.exe launches cmd.exe" require a join.

I think my proposal based on your argument would be to make uid required in the process object and follow a similar approach as option 2 to create a new parent object, except with a subset of the high value fields from the process object rather than a full copy . This would reduce the data size of each event, would prevent nesting, would allow most searches to be satisfied by a single event, and still allow the full parent process lookup for more complex searches using the parent.uid. Sample object below:

{
    "cmd_line": "/usr/bin/uname -a",
    "parent": {
        "cmd_line": "/bin/bash",
        "path": "/bin/bash",
        "pid": 13220,
        "uid": "A23EAE89-BD28-5903-0000-00102F345D00"
    },
    "path": "/usr/bin/uname",
    "pid": 6228,
    "uid": "A23EAE89-BD56-5903-0000-0010E9D95E00"
},

Which fields to include in the parent object and what their requirement values should be under this approach is up for debate. I think cmd_line, path, pid, uid, user and some sort of file hash would be my ideal minimum fields.

[jp-harvey]

The logic behind having either / or primarily hinged on the lack of a required uid. Since the uid is not provided by the operating system and generated by the telemetry provider itself it makes sense that the schema does not require it (even though every product I've seen creates one). If the process object requires a uid then it's less of an issue. It seems opinionated to require a uid but we're defining a schema and a critical part of a schema is to accurately relate objects to other objects so it seems this is an opinion that matters.

One characteristic of the parent_process object as described is that it's only going to allow one level, which then forces a join which is the argument for not just having a uid only. The assertion is that if we include a few more fields then sometimes joins won't be required, but what constitutes a high value field? If we're looking to eliminate the need for joins then we should use our current model where the parent is a process object.

The parent_process object solves some of the issues, but is not that different from just providing just a uidas the parent. We're still imposing the need for a join at some point unless we provide all the fields, in which case we just have a parent_process object which is exactly the same as process but does not have a parent field. That's not to say it's not a valid additional option, we should just not consider this a "no join" solution, quite the opposite. And the initial problem posed is that some providers won't have all the information. The more fields we add the more likely it is that we'll have the same problem.

[awhite456]

Maybe we need to back up and first vote whether we can make uid required? I think it may determine which direction we go to solve this problem. In the example producer being discussed, it would make the producer non-compliant, but given that the parent PID is not enough information to uniquely identify the parent, it probably should be non-compliant. I can also think of some analyst use cases, such as "show me all descendants of a process", which would be difficult under the current nested model, but would be possible with uid values.

This new proposal was not meant to be a "no join" solution, but instead try to cover some common search use cases without a join so that joins are only required sometimes. You make a good point however, if the provider does not populate those additional parent fields, you are always going to need a join for any searches involving parent fields. We would need to make these parent fields required to avoid this.

[awhite456]

Process required fields were reduced to allow this to work.