DIG-1671: Adding date format to manifest

[yavyx]

What's new

• Added a required date_format field to the manifest. It specifies the order of day, month and year to parse ambiguous dates properly.
• Updated test data and manifest with date formats.
• Improved mapping exceptions to be more readable and specific

How to test

• Add date_format to the manifest
• Run CSVConvert.py using a dataset with ambiguous dates for that field and try different patterns in the template to confirm they are parsed as indicated.
• Test removing date resolutions and references to see the exceptions.

Previously:
Added a date_format parameter to the date_interval mapping function to specify the order of day, month and year to parse ambiguous dates properly.

How to test
- Add date_format to any date field in the template
DONOR.INDEX.date_of_birth, {date_interval(Donor.date_of_birth, date_format="DMY")}
- Run CSVConvert.py using a dataset with ambiguous dates for that field and try different patterns in the template to confirm they are parsed as indicated.

[daisieh]

If I change the template line to

DONOR.INDEX.date_of_birth, {date_interval(Donor.date_of_birth, date_format="MDY")}

I don't think the date_format gets passed in? If I add a print statement to the top of date_interval in mappings.py, like

def date_interval(data_values, date_format=None):
    print(f"hello {date_format}")

I get a lot of "hello None" in the output.

I don't think that you can pass in extra arguments like this into the template file, at least not as I've written it.

I appreciate the idea behind making it more flexible for date formats, but I'm not sure that it's possible to do it this way. I think you might have to write a wrapper method in your own .py file that parses the date with a format.

[yavyx]

Re-submitting PR, DIG-1671

What's new

• Added a required date_format field to the manifest. It specifies the order of day, month and year to parse ambiguous dates properly.
• Updated test data and manifest with date formats.
• Improved mapping exceptions to be more readable and specific

How to test

• Add date_format to the manifest
• Run CSVConvert.py using a dataset with ambiguous dates for that field and try different patterns in the template to confirm they are parsed as indicated.
• Test removing date resolutions and references to see the exceptions.

[mshadbolt]

I added some try catch blocks to the part that is reading the manifest so that if the fields are blank, the script exits as well as if the manifest doesn't have those specified at all, also added a catch to ensure the date format is some form of 'DMY/MDY/YDM'

[mshadbolt]

Looks good to me and works as expected

[daisieh]

Tiny changes

[yavyx]

@daisieh I addressed the changes and Marion's fix is working

[daisieh]

Looks good to me! Thanks to both @yavyx and @mshadbolt for taking over so much of the coding in this project!