feat(dart): dart-next generator

[ahmednfwela]

supersedes #15485

This generator is designed to replace both dart and dart-dio generators, and has the following features:

1. NO dependency on 3rd party modelling/serialization libraries (built_value, freezed, etc...), everything is done in-house (also no need to run dart run build_runner build).
2. Heavy use of the new Extension types to add strict static typing with no run-time cost. an example of this is:
   • UndefinedWrapper<T> which makes any non-required variable recognize undefined as a valid value, in which case, it will NOT be included during serialization.
   • Enums: an enum is just an expression of the available values a variable can have, it shouldn't replace the data type, which is a perfect use case for extension types (see this file for an example).
3. Pluggable networking libraries, by extending NetworkingClientBase
   • Support initially will be for package:http, but package:dio can be added as well
4. Heavy use of mustache's ability to recursively include templates (example), which minimizes java code that was used to hack around nested types.
5. separate shared_infrastructure package that has simple base models that should NOT be changed, but the generator will also include an option to inline it. this is done so we can maintain a version of it on pub.dev which users can opt-in to use.
6. Use package:cross_file to handle files
7. support multiple mime types, not just json.
8. generate reflection information for models.
9. Support request/response streaming.

Current tasks:

• finalize JSON model serialization
• support discriminators when available
• finalize XML support
• finalize API classes
• write tests in java
• generate model tests (roundtrip serialization)
• generate API tests by mocking

P.S. testing and templating for this generator is done here https://github.com/ahmednfwela/dart-next-generator-testing

PR checklist

• Read the contribution guidelines.
• Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
• Run the following to build the project and update samples:

  ./mvnw clean package 
  ./bin/generate-samples.sh ./bin/configs/*.yaml
  ./bin/utils/export_docs_generators.sh
  

  (For Windows users, please run the script in Git BASH)
  Commit all changed files.
  This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
  These must match the expectations made by your contribution.
  You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
  IMPORTANT: Do NOT purge/delete any folders/files (e.g. tests) when regenerating the samples as manually written tests may be removed.
• File the PR against the correct branch: master (upcoming 7.6.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
• If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

[ahmednfwela]

cc core team: @wing328 @jimschubert @cbornet @jmini @etherealjoy

I made a non-breaking change in DefaultCodegen by adding a new method updateCodegenPropertyEnum to fix name conflicts in inner enums for languages that don't support namespaces

public void updateCodegenPropertyEnum(CodegenProperty var, CodegenModel model)

[ahmednfwela]

cc dart committee for early review:
@jaumard (2018/09) @josh-burton (2019/12) @amondnet (2019/12) @sbu-WBT (2020/12) @kuhnroyal (2020/12) @agilob (2020/12) @ahmednfwela (2021/08)

[bahag-chandrana]

I think this is a great effort yet a complicated approach to create a single solution that handles different cases.

Simplicity over complex use cases.

I understand we need to handle fields that are not defined in a json or xml. I assume the purpose of UndefinedWrapper extension type is meant for this. While this might help deserialization scenarios, it highly obstructs serialization scenarios. Take the following create a simple Apple instance.

Dart approach:

// easy to define null properties
final apple = Apple();
// easy to define an apple instance with both properties.
final apple = Apple(cultivar: 'Pink Lady', origin: 'Australia');

Proposed approach:

// easy to define null properties
final apple = Apple();
// complicated approach to define an apple instance with both properties.
final apple = Apple(cultivar: UndefinedWrapper('Pink Lady'), origin: UndefinedWrapper('Australia'));

Now imagine a scenario where you have to deal with an api where we have to pass a huge object in post request.
This also leads to learning new pattern even for common cases.

While UndefinedWrapper might solve a problem, it may introduce a cognitive load when these instances are actually used. Take for example I want to check if an apple instance has a name.

if(apple.cultivar.isDefined){
 // do something
}
// instead of 
if(apple.cultivar != null){
 // do something
}

The same can be achieved by simply exporting an extension on 'Object?` Following works on primitives as well. But nevertheless we can completely skip it and just use plain null check.

extension Undefined on Object? {
  bool get isDefined => this != null;
  bool get isUndefined => this == null;
}

Same comment on your proposal for a new form of enum.

build_runner is not bad!

The purpose of existing code generation for serialization is to keep the codebase readable and leave the obvious stuffs to the generator. I agree these can be often slow on a larger projects. But when we use an api client generator how often do we have to run the build_runner? Once or twice if the spec is not clear. But then it stays in the repository. With dart macros around the corner do we really need another completely new serialization deserialization logic. Perhaps its helpful for XML cases, which I can agree. But i strongly believe the whole build_runner for any form of serialization will vanish the day macros are launched and an XML macro could be the next bet.

Questions:

1. What is the purpose of the reflection classes? e.g. __return.reflection.dart , apple.reflection.dart
2. Can we extend with other serialization libraries in future if we want it?
3. What is the purpose of openapi_infrastructure package? It thought the aim is to reduce external dependencies.

[ahmednfwela]

thanks for your review @bahag-chandrana , let me first shed some light on some key points before I answer your questions.

The target of this generator is 100% spec compliance.

Meaning that the generated spec files MUST be able to accurately reflect a given openapi specification, with all of its supported features.

Of course the openapi spec is huge, and has some ambiguity.

This is why I decided to first cover the most important aspect of openapi, schemas.

---

A schema is essentially a description of a type.

But openapi's only concern is validating a given json (or xml) value against a schema.

it does NOT put any rules on what models should look like at the client side before they are sent to the server, and this is why I created this discussion before: OAI/OpenAPI-Specification#2807.

This means that in order to achieve my goal, which is 100% compliance, I needed to introduce a lot of "new" concepts to dart to be able to perfectly represent a given schema.

---

Problem 1: non-required nullable types

Here is a fairly common schema:

my_object:
  type: object
  properties:    
    name:
      type: string
      nullable: true

the name property here can be either

1. a valid string "a b c"

{ "name": "a b c" }

2. a null

{ "name": null }

3. not present, or undefined

{ }

in a perfect world, if dart supported union types, we can represent this as follows:

class Undefined {
  const Undefined();
}
final <String | null | Undefined> name;

and if name is Undefined, we ignore it during serialization, or -during deserialization- set it to Undefined if it's not present in the json.

However, the closest thing we have to union types is nullable types, e.g. String? which represents <String | null>.

So it's simply impossible to represent Undefined without a wrapper class, which is what extension types were made for.

While i definitely agree that UndefinedWrapper adds a cognitive load, it's also the ONLY design (until union types are added) that serves my goal of 100% compliance.

Also in your example

if(apple.cultivar.isDefined){
// do something
}
// instead of 
if(apple.cultivar != null){
// do something
}

we can introduce an extension method to handle this common case

extension UndefinedWrapperNullableExt<T> on UndefinedWrapper<T?> {
  bool get isAvailable => src.split(
      defined: (src) => src != null,
      unDefined: () => false,
    );
}

also if dart-lang/language#3614 gets implemented, that cognitive load of having to wrap UndefinedWrapper around all values is gone.

---

Problem 2: reflection

As you know, using dart:mirrors to reflect on dart code is not possible (unlike c# reflection), so the alternative is to either use macros (still kinda limited and in preview) or put information about the code as part of the code (which is what build_runner does).

Now to implement an openapi schema as a dart class, the following features are expected of it:

1. Proper handling of required and nullable constructs
2. Proper handling of enums
3. Proper handling of oneof,anyof, allof constructs
4. Value-based equality == and hashcode
5. Cloning.
6. Modifying values in a class.
7. serialize/deserialize/canDeserialize methods

And there is not a single existing generator based on build_runner that can do all of that right now, some libraries even have long standing bugs for years now.

So after getting fed up with all the existing build_runner packages, I created my own solution to reflect the generated schemas as part of the generation process, where each type tree has its own reflection tree that is able to deliver all of the features I mentioned above.

This makes sure that tree shaking is applicable, and that we can follow separation of concerns principle, where the generated model classes only serve one purpose, storing values, and the reflection classes do the rest of the requested features.

---

So for Question 1:

All reflection classes extend SerializationReflection<T>, but model reflections inherit from ModelReflection, which is where most of the magic happens (serialization/deserialization, example generation, equality checking, hashing, etc...).

The beautiful thing about this design, is that ALL the generated reflection classes can be replaced with macros!

---

Question 2:

Simply no, because there is no need to.
A serialization library will -at best- do a subset of the features a fully fledged reflection class can do.

---

Question 3:

You can consider it an abstraction layer above package:http and package:dio, which unifies features between them (especially multipart support).

This enables the user to define their own networking adapter that does NOT care about any of serialization logic, as it's only dealing with raw bytes and pure HTTP concepts.

And this solves the main issue this generator was created for, decoupling networking from serialization

[DmitrySboychakov]

Hello @ahmednfwela, thank you for the great contribution. I love the overall design, but..
Let me say I also share some concerns about UndefinedWrapper. My believe that most of users would treat null as undefined without a problem and personally I'd love to see that as feature flag to the generator.

[ahmednfwela]

Hello @ahmednfwela, thank you for the great contribution. I love the overall design, but..
Let me say I also share some concerns about UndefinedWrapper. My believe that most of users would treat null as undefined without a problem and personally I'd love to see that as feature flag to the generator.

maybe after I finish the initial PR and release it as experimental, I will add a flag to treat null as undefined if the property is not required

[vasilich6107]

Hi @ahmednfwela
Is there any chance to move iteratively on existing dart-dio?
With all respect to your hard work - 2years of previous PR were abandoned.
This PR aims to replace them all - how hard it could be to migrate to dart-next?

[ahmednfwela]

actually all the work is already done, the only thing missing is proper testing

[vasilich6107]

actually all the work is already done, the only thing missing is proper testing

Pretty significant piece of work I would say.

Could you clarify what is the roadmap before merging dart-next as experimental generator into master?
I could help with testing if it is in master

[ahmednfwela]

well, first I think i might need help with writing all the java tests, these are present in:

• modules/openapi-generator/src/test/java/org/openapitools/codegen/dart/next/DartNextClientCodegenModelTest.java
• modules/openapi-generator/src/test/java/org/openapitools/codegen/dart/next/DartNextClientCodegenOptionsTest.java
• modules/openapi-generator/src/test/java/org/openapitools/codegen/dart/next/DartNextClientCodegenTest.java

so that I can generate the samples and CI can pass.