Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Round the to date down to the start of the day #224

Merged
merged 9 commits into from
Jun 18, 2024
Merged

Conversation

dubiousdavid
Copy link
Member

@dubiousdavid dubiousdavid commented Jun 17, 2024

  • Set the to date to the end of the current day for all lastXDays and lastXMonths date ranges.
  • Set the from date to the start of the day for all lastXDays date ranges.

See: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/common-options.html#date-math

Copy link

changeset-bot bot commented Jun 17, 2024

🦋 Changeset detected

Latest commit: f55cc18

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
contexture-elasticsearch Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@dubiousdavid dubiousdavid marked this pull request as draft June 17, 2024 17:13
@dubiousdavid dubiousdavid changed the title Anchor the to date to the start of the current day. Use the start of the current day for the to date Jun 17, 2024
@dubiousdavid dubiousdavid changed the title Use the start of the current day for the to date Round the to date down to the start of the day Jun 17, 2024
@dubiousdavid dubiousdavid marked this pull request as ready for review June 17, 2024 18:16
last36Months: { from: 'now/d-36M', to: 'now' },
last48Months: { from: 'now/d-48M', to: 'now' },
last60Months: { from: 'now/d-60M', to: 'now' },
last12Months: { from: 'now/d-12M', to: 'now/d' },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the to date is not supposed to be inclusive, should we be doing the following https://github.com/smartprocure/contexture/blob/main/packages/provider-elasticsearch/src/utils/dateUtil.js#L59 It seems like we are somewhat inconsistent on what each lastX uses as the to expression

Copy link
Member Author

@dubiousdavid dubiousdavid Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually thinking the following might be the best - now+1d/d-1ms. It should cache nicely while also returning results from the current day. The cache expires after six hours, so each time there is a cache miss the query will return any new records. Thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually thinking the following might be the best - now+1d/d-1ms. It should cache nicely while also returning results from the current day. The cache expires after six hours, so each time there is a cache miss the query will return any new records. Thoughts?

It makes more sense to me personally to include the current date in the lastX expressions, should we adjust the from to also do now+1d/d in that case so that we are not picking up an extra day in the beginning if the end date include the current day?

Copy link
Member

@stellarhoof stellarhoof Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elastic datemath still gets converted to a timestamp so the cache is ineffective.

Ignore this. Per our talk, /d rounds to the start of the day so we're good 👍

You can see it in the elastic query produced for the issued date filter in spending Screenshot 2024-06-17 at 3 54 00 PM

Also we may want to factor dateUtil into the new util package since it looks identical to the one used for other providers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dshishkov In a meeting with Bradley we decided to be inclusive on the entire current day for either past/future ranges. This means that to will always be now+1d/d-1ms for past ranges and from will always be now/d for future ranges.

Copy link
Member

@stellarhoof stellarhoof left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍

Also tested locally

Alejandro Hernandez added 2 commits June 17, 2024 18:32
Copy link
Contributor

github-actions bot commented Jun 17, 2024

Messages
📖 You reduced the total lines of code! Awesome! 👍

Generated by 🚫 dangerJS against f55cc18

Copy link
Contributor

Coverage after merging feature/pin-dates into main will be

90.54%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
packages/client/src
   exampleTypes.js93.78%85.71%40%97.80%109, 123, 126–133
   index.js98.96%93.85%100%100%211, 213, 284, 73
   lens.js100%100%100%100%
   listeners.js100%100%100%100%
   mockService.js95.65%83.33%100%100%22, 30
   node.js96.75%96%87.50%97.78%72, 76, 76
   reactors.js99.12%100%90.91%100%
   serialize.js100%100%100%100%
   subquery.js97.59%88.89%100%98.55%66, 66
   traversals.js100%100%100%100%
   types.js92.59%83.33%100%95.24%12–13
   validation.js78.79%83.33%100%77.78%20–26
packages/client/src/actions
   index.js100%100%100%100%
   wrap.js96.67%85.71%100%97.92%41, 41
packages/client/src/exampleTypes
   pivot.js82.87%74.29%87.50%83.70%10–13, 138–139, 14, 140–149, 15, 150–161, 167–168, 21–22, 227, 23, 243, 257–263, 48, 53–57, 8–9, 9
packages/client/src/util
   futil.js67.31%100%57.14%65.85%18–19, 24–28, 31, 36–41
   tree.js100%100%100%100%
packages/export/src
   csv.js92.98%80%50%96%26, 47–48
   excel.js93.69%76.92%50%97.87%13, 38, 52, 91–92
   index.js0%100%0%0%1–6
   utils.js100%100%100%100%
packages/export/src/nodes
   index.js0%100%0%0%1–5
   pivot.js98.43%82.35%100%100%52, 56, 89
   results.js100%100%100%100%
   terms_stats.js100%100%100%100%
packages/provider-elasticsearch/src
   index.js81.07%50%80%87.22%111–119, 12, 120–122, 13, 29, 58, 58–63, 69, 72–73, 80, 82, 94, 98, 98
   schema.js92.52%92.86%80%93.18%52–55, 85–87
   types.js100%100%100%100%
packages/provider-elasticsearch/src/example-types
   index.js100%100%100%100%
   schemaMapping.js100%100%100%100%
   testUtils.js98.88%92.31%100%100%12
packages/provider-elasticsearch/src/example-types/filters
   bool.js100%100%100%100%
   date.js85.71%66.67%50%89.19%10–11, 19, 19, 9
   dateRangeFacet.js70.71%100%75%67.82%59–86
   exists.js100%100%100%100%
   facet.js92.80%86.67%66.67%94.39%39, 64, 64–69
   geo.js100%100%100%100%
   number.js52.05%100%50%50%22–29, 33–56
   query.js100%100%100%100%
   tagsQuery.js86.98%90.63%85.71%86.36%111–128, 140, 149–153, 159–161
   tagsText.js100%100%100%100%
   text.js95.73%86.67%100%98.82%11, 22, 40, 71, 71
packages/provider-elasticsearch/src/example-types/legacy
   cardinality.js93.33%100%50%100%
   dateHistogram.js95.45%100%50%100%
   esTwoLevelAggregation.js92.11%86.67%66.67%93.75%34–36, 83–87
   matchStats.js97.06%100%50%100%
   rangeStats.js100%100%100%100%
   smartIntervalHistogram.js97.44%100%50%100%
   statistical.js90.91%100%50%100%
   terms_stats.js97.73%100%66.67%100%
packages/provider-elasticsearch/src/example-types/metricGroups
   dateIntervalGroupStats.js100%100%100%100%
   dateRangesGroupStats.js98.28%100%75%100%
   fieldValuePartitionGroupStats.js87.76%100%40%92.31%34–36
   fieldValuesDelta.js97.44%100%66.67%100%
   fieldValuesGroupStats.js99.07%100%83.33%100%
   groupStatUtils.js86.05%100%75%85.71%20–24
   numberIntervalGroupStats.js100%100%100%100%
   numberRangesGroupStats.js97.87%100%66.67%100%
   percentilesGroupStats.js82.46%75%50%85.71%16–18, 24–26, 43–44
   pivot.js85.62%89.89%71.43%85.65%107, 151–152, 159, 166, 408–410, 460–470, 473, 478, 480, 483, 486, 534–535, 563–619, 75–81, 84–91
   stats.js86.96%100%50%94.12%16
   tagsQueryGroupStats.js100%100%100%100%
packages/provider-elasticsearch/src/example-types/metricGroups/pivotData
   columnResponse.js100%100%100%100%
   columnResult.js100%100%100%100%
   pivotResponse.js100%100%100%100%
   pivotResponseWithFilteredFieldValueGroup.js100%100%100%100%
packages/provider-elasticsearch/src/example-types/results
   index.js18.60%100%0%19.51%10–40, 8–9
packages/provider-elasticsearch/src/example-types/results/highlighting
   request.js95.15%93.75%100%95.26%193, 193–201, 40–42
   response.js100%100%100%100%
   search.js31.15%100%0%31.67%20–60
   testSchema.js100%100%100%100%
   util.js100%100%100%100%
packages/provider-elasticsearch/src/schema-data
   aliases.js100%100%100%100%
   mapping-with-non-objects.js100%100%100%100%
   mapping-with-types.js100%100%100%100%
   mapping-without-types.js100%100%100%100%
   schema-with-types.js100%100%100%100%
   schema-without-types.js100%100%100%100%
packages/provider-elasticsearch/src/utils
   elasticDSL.js100%100%100%100%
   fields.js100%100%100%100%
   futil.js99.39%100%92.86%100%
   luceneQueryUtils.js100%100%100%100%
   regex.js100%100%100%100%
   results.js100%100%100%100%
   smartInterval.js83.33%33.33%100%92.86%11, 11, 9
packages/provider-mongo/src
   index.js93.33%71.43%83.33%96.77%10, 33, 45, 9
   types.js100%100%100%100%
packages/provider-mongo/src/example-types
   bool.js100%100%100%100%
   date.js96%90.91%100%97.06%23, 23
   dateHistogram.js94.20%62.50%100%98.31%19, 19, 25, 25
   exists.js100%100%100%100%
   facet.js98.95%95.74%100%99.56%123–124, 124
   index.js100%100%100%100%
   mongoId.js100%100%100%100%
   number.js100%100%100%100%
   results.js97.74%92.31%100%99.01%180, 59, 61, 64–66
   statistical.js100%100%100%100%
   tagsText.js100%100%100%100%
   termsStats.js100%100%100%100%
   text.js100%100%100%100%
packages/react/src
   FilterAdder.js0%100%0%0%1, 10–19, 2, 20–29, 3, 30–39, 4, 40–49, 5, 50–52, 6–9
   FilterAdder.stories.js0%100%0%0%1, 10–19, 2, 20–23, 3–9
   FilterButtonList.js0%100%0%0%1, 10, 100–109, 11, 110–119, 12, 120–129, 13, 130–139, 14, 140–149, 15, 150–159, 16, 160–168, 17–19, 2, 20–29, 3, 30–39, 4, 40–49, 5, 50–59, 6, 60–69, 7, 70–79, 8, 80–89, 9, 90–99
   FilterButtonList.stories.js0%100%0%0%1, 10–19, 2, 20–29, 3, 30–39, 4, 40–49, 5, 50–56, 6–9
   FilterList.js0%100%0%0%1, 10, 100–109, 11, 110–119, 12, 120–129, 13, 130–139, 14, 140–149, 15, 150–159, 16, 160–169, 17, 170–179,

@dubiousdavid dubiousdavid merged commit 62b4654 into main Jun 18, 2024
4 checks passed
@dubiousdavid dubiousdavid deleted the feature/pin-dates branch June 18, 2024 12:13
@stellarhoof stellarhoof mentioned this pull request Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants